COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks

As reinforcement learning ( RL ) has achieved near human-level performance in a variety of tasks , its robustness has raised great attention when applied to safetycritical domains such as autonomous driving . Recent studies have explored testtime attacks in RL and corresponding defenses , while the robustness of RL against training-time attacks remains largely unanswered . In this work , we focus on certifying the robustness of offline RL in the presence of poisoning attacks , where a subset of training trajectories could be arbitrarily manipulated . We propose the first certification framework , COPA to certify the number of poisoning trajectories that can be tolerated regarding different certification criteria . Given the complex structure of RL , we propose two certification criteria : per-state action stability and cumulative reward bound . To tighten the certification , we also propose different partition and aggregation protocols to train robust policies . We further prove that some of the proposed certification methods are theoretically tight and some are NP-Complete problems . We conduct a thorough evaluation of COPA on different games trained with different offline RL algorithms : ( 1 ) The proposed temporal aggregation in COPA significantly improves certified robustness ; ( 2 ) Our certifications for both per-state action stability and cumulative reward bound are efficient and tight ; ( 3 ) The certification for different training algorithms and games are different , implying their intrinsic robustness properties . 1 INTRODUCTION . Reinforcement learning ( RL ) has been widely applied to a range of applications , including robotics ( Kober et al. , 2013 ; Deisenroth et al. , 2013 ; Polydoros & Nalpantidis , 2017 ) and autonomous vehicles ( Shalev-Shwartz et al. , 2016 ; Sallab et al. , 2017 ) . In particular , offline RL ( Levine et al. , 2020 ) is proposed to leverage previously observed data to train the policy without requiring expensive interaction with environments or online data collection , and enables the reuse of training data ( Agarwal et al. , 2020 ) . However , offline RL also raises a great safety concern on poisoning attacks ( Kiourti et al. , 2020 ; Wang et al. , 2020 ; 2021 ) . A recent survey of industry reports that data poisoning is significantly more concerning than other threats ( Kumar et al. , 2020 ) . For offline RL , the situation is even worse , and a recent theoretical study shows that the robust offline RL against poisoning attacks is a strictly harder problem than online RL ( Zhang et al. , 2021 ) . Although there are several empirical and certified defenses for classification tasks against poisoning attacks ( Peri et al. , 2020 ; Levine & Feizi , 2021 ; Weber et al. , 2020 ) , it is challenging and shown ineffective to directly apply them to RL given its complex structure ( Kiourti et al. , 2020 ) . Thus , robust offline RL against poisoning attacks remains largely unexplored with no mention of robustness certification . In addition , though some theoretical analyses provide general robustness bounds for RL , they either assume a bounded distance between the learned and Bellman optimal policies or are limited to linear MDPs ( Zhang et al. , 2021 ) . To the best of our knowledge , there is no robust RL method that is able to provide practically computable certified robustness against poisoning attacks . In this paper , we tackle this problem by proposing the first framework of Certifying robust policies for general offline RL against poisoning attacks ( COPA ) . Certification Criteria . One critical challenge in certifying robustness for offline RL is the certification criteria , since the prediction consistency is no longer the only goal as in classification tasks . We propose two criteria based on the properties of RL : per-state action stability and cumulative reward bound . The per-state action stability guarantees that at a specific time , the policy learned with COPA will predict the same action before and after attacks under certain conditions . This criterion is important for guaranteeing the safety of the policy at critical states , e.g. , stopping a vehicle when seeing pedestrians . For cumulative reward bound , a lower bound of the cumulative reward for the policy learned with COPA is guaranteed under certain poisoning conditions . This criterion directly guarantees the worst-case overall performance . COPA Framework . COPA is composed of two components : policy partition and aggregation protocol and robustness certification method . We propose three policy partition aggregation protocols : PARL ( Per-State Partition Aggregation ) , TPARL ( Temporal Partition Aggregation ) , and DPARL ( Dynamic Temporal Partition Aggregation ) , and propose certification methods for each of them corresponding to both proposed certification criteria . In addition , for per-state action stability , we prove that our certifications for PARL and TPARL are theoretically tight . For cumulative reward bound , we propose an adaptive tree search algorithm , where we will compute the possible action set for each state under certain poisoning conditions . For PARL , we propose a novel method to compute the precise action set . For TPARL and DPARL , we propose efficient algorithms to compute a superset of the possible action set which leads to sound certification . We further prove that for PARL our certification is theoretically tight , for TPARL the theoretically tight certification is NP-complete , and for DPARL it is open whether theoretically tight certification exists . Technical Contributions . We take the first step towards certifying the robustness of offline RL against poisoning attacks , and we make contributions on both theoretical and practical fronts . • We abstract and formulate the robustness certification for offline RL against poisoning attacks , and we propose two certification criteria : per-state action stability and cumulative reward bound . • We propose the first framework COPA for certifying robustness of offline RL against poisoning attacks . COPA includes novel policy aggregation protocols and certification methods . • We prove the tightness of the proposed certification methods for the aggregation protocol PARL . We also prove the computational hardness of the certification for TPARL . • We conduct thorough experimental evaluation for COPA on different games with three offline RL algorithms , demonstrating the effectiveness of COPA , together with several interesting findings . 2 CERTIFICATION CRITERIA OF COPA . In this section , we propose two robustness certification criteria for offline RL against general poisoning attacks : per-state action stability and cumulative reward bound . Offline RL . We model the RL environment by an episodic finite-horizon Markov decision process ( MDP ) E = ( S , A , R , P , H , d0 ) , where S is the set of states , A is the set of discrete actions , R : S×A → R is the reward function , P : S×A → P ( S ) is the stochastic transition function withP ( · ) defining the set of probability measures , H is the time horizon , and d0 ∈ P ( S ) is the distribution of the initial state . At time step t , the RL agent is at state st ∈ S . After choosing action at ∈ A , the agent transitions to the next state st+1 ∼ P ( st , at ) and receives reward rt = R ( st , at ) . After all H time steps , the agent ’ s cumulative reward J = ∑H−1 t=0 rt . We denote a consecutive sequence of all states between time step l and r as sl : r : = [ sl , sl+1 , . . . , sr ] . Here we focus on offline RL , for which the threat of poisoning attacks is practical and more challenging to deal with ( Zhang et al. , 2021 ) . Concretely , in offline RL , a training dataset D = { τi } Ni=1 consists of logged trajectories , where each trajectory τ = { ( sj , rj , aj , s′j ) } lj=1 ∈ ( S ×A×R×S ) l consists of multiple tuples denoting the transitions ( i.e. , starting from state sj , taking the action aj , receiving reward rj , and transitioning to the next state s′j ) . Poisoning Attacks . Training dataset D can be poisoned in the following manner . For each trajectory τ ∈ D , the adversary is allowed to replace it with an arbitrary trajectory τ̃ , generating a manipulated dataset D̃ . We denote D D̃ = ( D\D̃ ) ⋃ ( D̃\D ) as the symmetric difference between two datasets D and D̃ . For instance , adding or removing one trajectory causes a symmetric difference of magnitude 1 , while replacing one trajectory with a new one leads to a symmetric difference of magnitude 2 . We refer to the size of the symmetric difference as the poisoning size . Certification Goal . To provide the robustness certification against poisoning attacks introduced above , we aim to certify the test-time performance of the trained policy in a clean environment . Specifically , in the training phase , the RL training algorithm and our aggregation protocol can be jointly modeled byM : D → ( S ? → A ) which provides an aggregated policy , where S ? denotes the set of all consecutive state sequences . Our goal is to provide robustness certification for the poisoned aggregated policy π̃ =M ( D̃ ) , given bounded poisoning size ( i.e. , |D D̃| ≤ K ) . Robustness Certification Criteria : Per-State Action Stability . We first aim to certify the robustness of the poisoned policy in terms of the stability of per-state action during test time . Definition 1 ( Robustness Certification for Per-State Action Stability ) . Given a clean dataset D , we define the robustness certification for per-state action stability as that for any D̃ satisfying |D D̃| ≤ K , the action predictions of the poisoned and clean policies for the state ( or state sequence ) s are the same , i.e. , π̃ = M ( D̃ ) , π = M ( D ) , π̃ ( s ) = π ( s ) , under the the tolerable poisoning threshold K. In an episode , we denote the tolerable poisoning threshold for the state at step t by Kt . The definition encodes the requirement that , for a particular state , any poisoned policy will always give the same action prediction as the clean one , as long as the poisoning size is within K ( K computed in Section 3 ) . In this definition , s could be either a state or a state sequence , since our aggregated policy ( defined in Section 2 ) may aggregate multiple recent states to make an action choice . Robustness Certification Criteria : Lower Bound of Cumulative Reward . We also aim to certify poisoned policy ’ s overall performance in addition to the prediction at a particular state . Here we measure the overall performance by the cumulative reward J ( π ) ( formally defined in Appendix A ) . Now we are ready to define the robustness certification for cumulative reward bound . Definition 2 ( Robustness Certification for Cumulative Reward Bound ) . Robustness certification for cumulative reward bound is the lower bound of cumulative reward JK such that JK ≤ J ( π̃ ) for any π̃ =M ( D̃ ) where |D D̃| ≤ K , i.e. , π̃ is trained on poisoned dataset D̃ within poisoning size K . 3 CERTIFICATION PROCESS OF COPA . In this section , we introduce our framework COPA , which is composed of training protocols , aggregation protocols , and certification methods . The training protocol combined with an offline RL training algorithm provides subpolicies . The aggregation protocol aggregates the subpolicies as an aggregated policy . The certification method certifies the robustness of the aggregated policy against poisoning attacks corresponding to different certification criteria provided in Section 2 . Overview of Theoretical Results . In Table 1 , we present an overview of our theoretical results : For each proposed aggregation protocol and certification criteria , we provide the corresponding certification method and core theorems , and we also provide the tightness analysis for each certification .

Summary: This paper proposes a defense method against poisoning attacks in offline RL. The authors propose a framework named COPA to certify the number of tolerable poisoned trajectories. The proposed defense includes a partitioning stage which separate the (possibly poisoned trajectories), a training stage where $u$ sub-policies are trained with $u$ partitions,  and an aggregation stage where the sub-policies are aggregated to produce the final policy. Three aggregation protocols are proposed with corresponding certification results.