Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamics

1 INTRODUCTION . Although reinforcement learning ( RL ) , especially deep RL , has been successfully applied in various fields , the security of RL techniques against adversarial attacks is not yet well understood . In realworld scenarios , including high-stakes ones such as autonomous driving vehicles and healthcare systems , a bad decision may lead to a tragic outcome . Should we trust the decision made by an RL agent ? How easy is it for an adversary to mislead the agent ? These questions are crucial to ask before deploying RL techniques in many applications . In this paper , we focus on poisoning attacks , which occur during the training and influence the learned policy . Since training RL is known to be very sample-consuming , one might have to constantly interact with the environment to collect data , which opens up a lot of opportunities for an attacker to poison the training samples collected . Therefore , understanding poisoning mechanisms and studying the vulnerabilities in RL are crucial to provide guidance for defense methods . However , existing works on adversarial attacks in RL mainly study the test-time evasion attacks ( Chen et al. , 2019 ) where the attacker crafts adversarial inputs to fool a well-trained policy , but does not cause any change to the policy itself . Motivated by the importance of understanding RL security in the training process and the scarcity of relevant literature , in this paper , we investigate how to poison RL agents and how to characterize the vulnerability of deep RL algorithms . In general , RL is an “ online ” process : an agent rolls out experience from the environment with its current policy , and uses the experience to improve its policy , then uses the new policy to roll out new experience , etc . Poisoning in online RL is significantly different from poisoning in classic supervised learning ( SL ) , even online SL , and is more difficult due to the following challenges . Challenge I – Future Data Unavailable in Online RL . Poisoning approaches in SL ( Muñoz-González et al. , 2017 ; Wang & Chaudhuri , 2018 ) usually require the access to the whole training dataset , so the attacker can decide the optimal poisoning strategy before the learning starts . However , in online RL , the training data ( trajectories ) are generated by the agent while it is learning . Although the optimal poison should work in the long run , the attacker can only access and change the data in the current iteration , since the future data is not yet generated. ! '' # $ : & = 0 # ) : & = 1 ! + ! `` , # $ : & = 0 # $ : & = 0 # ) : & = 1 # ) : & = 1 # $ : & = 1000 Figure 1 : An example of difficult poisoning . Challenge II – Data Samples No Longer i.i.d .. It is well-known that in RL , data samples ( state-action transitions ) are no longer i.i.d. , which makes learning challenging , since we should consider the longterm reward rather than the immediate result . How- ever , we notice that data samples being not i.i.d . also makes poisoning attacks challenging . For example , an attacker wants to reduce the agent ’ s total reward in a task shown as Figure 1 ; at state s1 , the attacker finds that a1 is less rewarding than a0 ; if the attacker only looks at the immediate reward , he will lure the agent into choosing a1 . However , following a1 finally leads the agent to s10 which has a much higher reward . Challenge III – Unknown Dynamics of Environment . Although Challenge I and II can be partially addressed by predicting the future trajectories or steps , it requires prior knowledge on the dynamics of the underlying MDP . Many existing poisoning RL works ( Rakhsha et al. , 2020 ; Ma et al. , 2019 ) assume the attacker has perfect knowledge of the MDP , then compute the optimal poisoning . However , in many real-world environments , knowing the dynamics of the MDP is difficult . Although the attacker could potentially interact with the environment to build an estimate of the environment model , the cost of interacting with the environment could be unrealistically high , market making ( Spooner et al. , 2018 ) for instance . In this paper , we study a more realistic scenario where the attacker does not know the underlying dynamics of MDP , and can not directly interact with the environment , either . Thus , the attacker learns the environment only based on the agent ’ s experience . In this paper , we systematically investigate poisoning in RL by considering all the aforementioned RL-specific challenges . Previous works either do not address any of the challenges or only address some of them . Behzadan & Munir ( 2017 ) achieve policy induction attacks for deep Q networks ( DQN ) . However , they treat output actions of DQN similarly to labels in SL , and do not consider Challenge II that the current action will influence future interactions . Ma et al . ( 2019 ) propose a poisoning attack for model-based RL , but they suppose the agent learns from a batch of given data , not considering Challenge I. Rakhsha et al . ( 2020 ) study poisoning for online RL , but they require perfect knowledge of the MDP dynamics , which is unrealistic as stated in Challenge III . Summary of Contributions . ( 1 ) We propose a practical poisoning algorithm called VulnerabilityAware Adversarial Critic Poison ( VA2C-P ) that works for deep policy gradient learners without any prior knowledge of the environment . To the best of our knowledge , VA2C-P is the first practical algorithm that poisons policy-based deep RL methods . ( 2 ) We introduce a novel metric , called stability radius , to characterize the stability of RL algorithms , measuring and comparing the vulnerabilities of RL algorithms in different scenarios . ( 3 ) We conduct a series of experiments for various environments and state-of-the-art deep policy-based RL algorithms , which demonstrates RL agents ’ vulnerabilities to even weaker attackers with limited knowledge and attack budget . 2 RELATED WORK . The main focus of this paper is on poisoning RL , an emerging area in the past few years . We survey related works of adversarial attacks in SL and evasion attacks in RL in Appendix A , as they are out of the scope of this paper . Targeted Poisoning Attacks for RL . Most RL poisoning researches work on targeted poisoning , also called policy teaching , where the attacker leads the agent to learn a pre-defined target policy . Policy teaching can be achieved by manipulating the rewards ( Zhang & Parkes , 2008 ; Zhang et al. , 2009 ) or dynamics ( Rakhsha et al. , 2020 ) of the MDP . However , they require the attackers to not only have prior knowledge of the environments ( e.g. , the dynamics of the MDP ) , but also have the ability to alter the environment ( e.g . change the transition probabilities ) , which are often unrealistic or difficult in practice . Poisoning RL with Omniscient Attackers . Most guaranteed poisoning RL literature ( Rakhsha et al. , 2020 ; Ma et al. , 2019 ) assume omniscient attackers , who not only know the learner ’ s model , but also know the underlying MDP . However , as motivated in the introduction , the underlying MDP is usually either unknown or too complex in practice . Some works poison RL learners by changing the reward signals sent from the environment to the agent . For example , Ma et al . ( 2019 ) introduce a policy teaching framework for batch-learning model-based agents ; Huang & Zhu ( 2019 ) propose a reward-poisoning attack model , and provide convergence analysis for Q-learning ; Zhang et al . ( 2020b ) present an adaptive reward poisoning method for Q-learning ( while it also extends to DQN ) and analyze the safety thresholds of RL ; these papers all assume the attacker knows not only the models of the agent , but also the parameters of the underlying MDP , which could be possible in a tabular MDP , but hard to realize in large environments and modern deep RL systems . On the contrary , we consider non-omniscient attackers who do not know the underlying MDP or environment in this paper . The non-omniscient attackers can be further divided into two categories : white-box attackers , who know the learner ’ s model/parameters , and black-box attackers , who do not know the learner ’ s model/parameters . They both tap the interactions between the learner and the environment . Black-box Poisoning for Value-based Learners . Although there are many successful black-box evasion approaches ( Xinghua et al. , 2020 ; Inkawhich et al. , 2020 ) , black-box poisoning in RL is rare . There is a black-box attacking method for a value-based learner ( DQN ) proposed by Behzadan & Munir ( 2017 ) , which does not require the attacker to know the learner ’ s model or the underlying MDP . In this work , the attacker induces the DQN agent to output the target action by perturbing the state with Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2015 ) in every step . However , the data-correlation problem of RL ( Challenge II ) is not considered , and FGSM attack does not work for policy-based methods due to their high stochasticity , as we show in experiments . In this paper , we propose a new poisoning algorithm for policy-based deep RL agents , which can achieve both non-targeted and targeted attacks . We do not require any prior knowledge of the environment . And our algorithm works not only when the attacker knows the learner ’ s model ( whitebox ) , but also when the leaner ’ s model is hidden ( black-box ) . 3 PROBLEM FORMULATION FOR POISONING ONLINE RL . 3.1 NOTATIONS AND PRELIMINARIES . In RL , an agent interacts with the environment by taking actions , observing states and receiving rewards . The environment is modeled by a Markov Decision Process ( MDP ) , which is denoted by a tuple M = 〈S , A , P , R , γ , µ〉 , where S is the state space , A is the action space , P is the transition kernel , R is the reward function , γ ∈ ( 0 , 1 ) is the discount factor , and µ is the initial state distribution . A trajectory τ ∼ π generated by policy π is a sequence s1 , a1 , r1 , s2 , a2 , · · · , where s1 ∼ µ , at ∼ π ( a|st ) , st+1 ∼ P ( s|st , at ) and rt = R ( st , at ) . The goal of an RL agent is to find an optimal policy π∗ that maximizes the expected total rewards η , which is defined as η ( π ) = Eτ∼π [ r ( τ ) ] = Es1 , a1 , ···∼µ , π , P , R [ ∑∞ t=1 γ t−1rt ] . We use an overhead check signˇon a variable to denote that the variable is poisoned . For example , if the attacker perturbs a reward rt , then the poisoned reward is denoted as řt . If a policy π is updated with poisoned observation , then it is denoted as π̌ . 3.2 THE PROCEDURE OF ONLINE LEARNING AND POISONING Procedure of Online Learning . We consider a classical online policy-based RL setting , where the learner iteratively updates its policy π parametrized by θ , through K iterations with the environment . For notation simplicity , we omit θ and use πk to denote πθk , the learner ’ s policy at iteration k. The online learning process is described as below . At iteration k = 1 , · · · , K , ( 1 ) The agent uses the current policy πk to roll out observation Ok = ( Osk , Oak , Ork ) from environmentMk , where Osk = [ s1 , s2 , · · · ] , Oak = [ a1 , a2 , · · · ] , Ork = [ r1 , r2 , · · · ] are respectively the sequence of states , actions and rewards generated at iteration k. ( 2 ) The agent updates its policy parameters θ with its algorithm f . Most policy-based algorithms perform on-policy updating , i.e. , update policy only by the current observation Ok . The on-policy update can be then formalized as πk+1 = f ( πk , Ok ) ≈ argmaxπJ ( π , πk , Ok ) , where J is an objective function defined by algorithm f , e.g. , the expected total reward η ( π ) . Procedure of Online Poisoning . A poisoning attacker influences the learner in the training process by perturbing the training data . In SL , the training data consists of features and labels , and the attacker poisons the training data before the learning starts . However , the training data in RL is the trajectories a learner rolls out from the environment , i.e. , observation O = ( Os , Oa , Or ) . At iteration k , the attacker eavesdrops on the interaction between the learner and the environment , obtains the observation Ok , and may poison it into Ǒk , then send Ǒk to the learner before policy updating . 1 Procedure 2 in Appendix B illustrates how the online game goes between the learner and the attacker . Figure 2 visualizes this online learning-poisoning procedure , where we can see that learning and poisoning are convoluted and inter-dependent .

Summary: The paper studies poisoning attacks against online reinforcement learning agents. The attacker has the power of manipulating the training data, i.e., state-action-reward trajectories, in order to achieve some attack goal. The attack can be completely black-box, meaning that the proposed method allows an attack setting where the attacker has no knowledge of the RL algorithm used by the victim agent or the environment. In this scenario, the authors proposed that the attacker can imitate the learning procedure of the victim, and then based on the imitated policy; the attacker designs how to poison the training data. The attack is formulated as a bi-level optimization, where the lower level involves the imitated learning procedure. Due to the intractability of sequential optimization, the original formulation is simplified so that only the attack only solves the attack on the current training data. This procedure is repeated in every episode to achieve sequential attacks. Experiments on a variety of tasks demonstrate the superiority of the proposed attack.