COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks

As reinforcement learning ( RL ) has achieved near human-level performance in a variety of tasks , its robustness has raised great attention when applied to safetycritical domains such as autonomous driving . Recent studies have explored testtime attacks in RL and corresponding defenses , while the robustness of RL against training-time attacks remains largely unanswered . In this work , we focus on certifying the robustness of offline RL in the presence of poisoning attacks , where a subset of training trajectories could be arbitrarily manipulated . We propose the first certification framework , COPA to certify the number of poisoning trajectories that can be tolerated regarding different certification criteria . Given the complex structure of RL , we propose two certification criteria : per-state action stability and cumulative reward bound . To tighten the certification , we also propose different partition and aggregation protocols to train robust policies . We further prove that some of the proposed certification methods are theoretically tight and some are NP-Complete problems . We conduct a thorough evaluation of COPA on different games trained with different offline RL algorithms : ( 1 ) The proposed temporal aggregation in COPA significantly improves certified robustness ; ( 2 ) Our certifications for both per-state action stability and cumulative reward bound are efficient and tight ; ( 3 ) The certification for different training algorithms and games are different , implying their intrinsic robustness properties . 1 INTRODUCTION . Reinforcement learning ( RL ) has been widely applied to a range of applications , including robotics ( Kober et al. , 2013 ; Deisenroth et al. , 2013 ; Polydoros & Nalpantidis , 2017 ) and autonomous vehicles ( Shalev-Shwartz et al. , 2016 ; Sallab et al. , 2017 ) . In particular , offline RL ( Levine et al. , 2020 ) is proposed to leverage previously observed data to train the policy without requiring expensive interaction with environments or online data collection , and enables the reuse of training data ( Agarwal et al. , 2020 ) . However , offline RL also raises a great safety concern on poisoning attacks ( Kiourti et al. , 2020 ; Wang et al. , 2020 ; 2021 ) . A recent survey of industry reports that data poisoning is significantly more concerning than other threats ( Kumar et al. , 2020 ) . For offline RL , the situation is even worse , and a recent theoretical study shows that the robust offline RL against poisoning attacks is a strictly harder problem than online RL ( Zhang et al. , 2021 ) . Although there are several empirical and certified defenses for classification tasks against poisoning attacks ( Peri et al. , 2020 ; Levine & Feizi , 2021 ; Weber et al. , 2020 ) , it is challenging and shown ineffective to directly apply them to RL given its complex structure ( Kiourti et al. , 2020 ) . Thus , robust offline RL against poisoning attacks remains largely unexplored with no mention of robustness certification . In addition , though some theoretical analyses provide general robustness bounds for RL , they either assume a bounded distance between the learned and Bellman optimal policies or are limited to linear MDPs ( Zhang et al. , 2021 ) . To the best of our knowledge , there is no robust RL method that is able to provide practically computable certified robustness against poisoning attacks . In this paper , we tackle this problem by proposing the first framework of Certifying robust policies for general offline RL against poisoning attacks ( COPA ) . Certification Criteria . One critical challenge in certifying robustness for offline RL is the certification criteria , since the prediction consistency is no longer the only goal as in classification tasks . We propose two criteria based on the properties of RL : per-state action stability and cumulative reward bound . The per-state action stability guarantees that at a specific time , the policy learned with COPA will predict the same action before and after attacks under certain conditions . This criterion is important for guaranteeing the safety of the policy at critical states , e.g. , stopping a vehicle when seeing pedestrians . For cumulative reward bound , a lower bound of the cumulative reward for the policy learned with COPA is guaranteed under certain poisoning conditions . This criterion directly guarantees the worst-case overall performance . COPA Framework . COPA is composed of two components : policy partition and aggregation protocol and robustness certification method . We propose three policy partition aggregation protocols : PARL ( Per-State Partition Aggregation ) , TPARL ( Temporal Partition Aggregation ) , and DPARL ( Dynamic Temporal Partition Aggregation ) , and propose certification methods for each of them corresponding to both proposed certification criteria . In addition , for per-state action stability , we prove that our certifications for PARL and TPARL are theoretically tight . For cumulative reward bound , we propose an adaptive tree search algorithm , where we will compute the possible action set for each state under certain poisoning conditions . For PARL , we propose a novel method to compute the precise action set . For TPARL and DPARL , we propose efficient algorithms to compute a superset of the possible action set which leads to sound certification . We further prove that for PARL our certification is theoretically tight , for TPARL the theoretically tight certification is NP-complete , and for DPARL it is open whether theoretically tight certification exists . Technical Contributions . We take the first step towards certifying the robustness of offline RL against poisoning attacks , and we make contributions on both theoretical and practical fronts . • We abstract and formulate the robustness certification for offline RL against poisoning attacks , and we propose two certification criteria : per-state action stability and cumulative reward bound . • We propose the first framework COPA for certifying robustness of offline RL against poisoning attacks . COPA includes novel policy aggregation protocols and certification methods . • We prove the tightness of the proposed certification methods for the aggregation protocol PARL . We also prove the computational hardness of the certification for TPARL . • We conduct thorough experimental evaluation for COPA on different games with three offline RL algorithms , demonstrating the effectiveness of COPA , together with several interesting findings . 2 CERTIFICATION CRITERIA OF COPA . In this section , we propose two robustness certification criteria for offline RL against general poisoning attacks : per-state action stability and cumulative reward bound . Offline RL . We model the RL environment by an episodic finite-horizon Markov decision process ( MDP ) E = ( S , A , R , P , H , d0 ) , where S is the set of states , A is the set of discrete actions , R : S×A → R is the reward function , P : S×A → P ( S ) is the stochastic transition function withP ( · ) defining the set of probability measures , H is the time horizon , and d0 ∈ P ( S ) is the distribution of the initial state . At time step t , the RL agent is at state st ∈ S . After choosing action at ∈ A , the agent transitions to the next state st+1 ∼ P ( st , at ) and receives reward rt = R ( st , at ) . After all H time steps , the agent ’ s cumulative reward J = ∑H−1 t=0 rt . We denote a consecutive sequence of all states between time step l and r as sl : r : = [ sl , sl+1 , . . . , sr ] . Here we focus on offline RL , for which the threat of poisoning attacks is practical and more challenging to deal with ( Zhang et al. , 2021 ) . Concretely , in offline RL , a training dataset D = { τi } Ni=1 consists of logged trajectories , where each trajectory τ = { ( sj , rj , aj , s′j ) } lj=1 ∈ ( S ×A×R×S ) l consists of multiple tuples denoting the transitions ( i.e. , starting from state sj , taking the action aj , receiving reward rj , and transitioning to the next state s′j ) . Poisoning Attacks . Training dataset D can be poisoned in the following manner . For each trajectory τ ∈ D , the adversary is allowed to replace it with an arbitrary trajectory τ̃ , generating a manipulated dataset D̃ . We denote D D̃ = ( D\D̃ ) ⋃ ( D̃\D ) as the symmetric difference between two datasets D and D̃ . For instance , adding or removing one trajectory causes a symmetric difference of magnitude 1 , while replacing one trajectory with a new one leads to a symmetric difference of magnitude 2 . We refer to the size of the symmetric difference as the poisoning size . Certification Goal . To provide the robustness certification against poisoning attacks introduced above , we aim to certify the test-time performance of the trained policy in a clean environment . Specifically , in the training phase , the RL training algorithm and our aggregation protocol can be jointly modeled byM : D → ( S ? → A ) which provides an aggregated policy , where S ? denotes the set of all consecutive state sequences . Our goal is to provide robustness certification for the poisoned aggregated policy π̃ =M ( D̃ ) , given bounded poisoning size ( i.e. , |D D̃| ≤ K ) . Robustness Certification Criteria : Per-State Action Stability . We first aim to certify the robustness of the poisoned policy in terms of the stability of per-state action during test time . Definition 1 ( Robustness Certification for Per-State Action Stability ) . Given a clean dataset D , we define the robustness certification for per-state action stability as that for any D̃ satisfying |D D̃| ≤ K , the action predictions of the poisoned and clean policies for the state ( or state sequence ) s are the same , i.e. , π̃ = M ( D̃ ) , π = M ( D ) , π̃ ( s ) = π ( s ) , under the the tolerable poisoning threshold K. In an episode , we denote the tolerable poisoning threshold for the state at step t by Kt . The definition encodes the requirement that , for a particular state , any poisoned policy will always give the same action prediction as the clean one , as long as the poisoning size is within K ( K computed in Section 3 ) . In this definition , s could be either a state or a state sequence , since our aggregated policy ( defined in Section 2 ) may aggregate multiple recent states to make an action choice . Robustness Certification Criteria : Lower Bound of Cumulative Reward . We also aim to certify poisoned policy ’ s overall performance in addition to the prediction at a particular state . Here we measure the overall performance by the cumulative reward J ( π ) ( formally defined in Appendix A ) . Now we are ready to define the robustness certification for cumulative reward bound . Definition 2 ( Robustness Certification for Cumulative Reward Bound ) . Robustness certification for cumulative reward bound is the lower bound of cumulative reward JK such that JK ≤ J ( π̃ ) for any π̃ =M ( D̃ ) where |D D̃| ≤ K , i.e. , π̃ is trained on poisoned dataset D̃ within poisoning size K . 3 CERTIFICATION PROCESS OF COPA . In this section , we introduce our framework COPA , which is composed of training protocols , aggregation protocols , and certification methods . The training protocol combined with an offline RL training algorithm provides subpolicies . The aggregation protocol aggregates the subpolicies as an aggregated policy . The certification method certifies the robustness of the aggregated policy against poisoning attacks corresponding to different certification criteria provided in Section 2 . Overview of Theoretical Results . In Table 1 , we present an overview of our theoretical results : For each proposed aggregation protocol and certification criteria , we provide the corresponding certification method and core theorems , and we also provide the tightness analysis for each certification .

Summary: This paper studied how to certificate a policy when the training dataset is partially corrupted in the offline reinforcement learning scenario. Two criteria including per-state action stability certificate and cumulative reward lower bound certificate are proposed. Based on these two criteria, the paper designed the COPA, a general certificate framework, which provably achieved certain level of certificate. The COPA went through two phases during training, where in the first phase, the trajectories are split into subsets of equal size, then sub-policies are trained on each subset, and finally the sub-policies are aggregated to form a single learned policy. Experiments demonstrate the effectiveness of the proposed COPA certification framework.