Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamics

1 INTRODUCTION . Although reinforcement learning ( RL ) , especially deep RL , has been successfully applied in various fields , the security of RL techniques against adversarial attacks is not yet well understood . In realworld scenarios , including high-stakes ones such as autonomous driving vehicles and healthcare systems , a bad decision may lead to a tragic outcome . Should we trust the decision made by an RL agent ? How easy is it for an adversary to mislead the agent ? These questions are crucial to ask before deploying RL techniques in many applications . In this paper , we focus on poisoning attacks , which occur during the training and influence the learned policy . Since training RL is known to be very sample-consuming , one might have to constantly interact with the environment to collect data , which opens up a lot of opportunities for an attacker to poison the training samples collected . Therefore , understanding poisoning mechanisms and studying the vulnerabilities in RL are crucial to provide guidance for defense methods . However , existing works on adversarial attacks in RL mainly study the test-time evasion attacks ( Chen et al. , 2019 ) where the attacker crafts adversarial inputs to fool a well-trained policy , but does not cause any change to the policy itself . Motivated by the importance of understanding RL security in the training process and the scarcity of relevant literature , in this paper , we investigate how to poison RL agents and how to characterize the vulnerability of deep RL algorithms . In general , RL is an “ online ” process : an agent rolls out experience from the environment with its current policy , and uses the experience to improve its policy , then uses the new policy to roll out new experience , etc . Poisoning in online RL is significantly different from poisoning in classic supervised learning ( SL ) , even online SL , and is more difficult due to the following challenges . Challenge I – Future Data Unavailable in Online RL . Poisoning approaches in SL ( Muñoz-González et al. , 2017 ; Wang & Chaudhuri , 2018 ) usually require the access to the whole training dataset , so the attacker can decide the optimal poisoning strategy before the learning starts . However , in online RL , the training data ( trajectories ) are generated by the agent while it is learning . Although the optimal poison should work in the long run , the attacker can only access and change the data in the current iteration , since the future data is not yet generated. ! '' # $ : & = 0 # ) : & = 1 ! + ! `` , # $ : & = 0 # $ : & = 0 # ) : & = 1 # ) : & = 1 # $ : & = 1000 Figure 1 : An example of difficult poisoning . Challenge II – Data Samples No Longer i.i.d .. It is well-known that in RL , data samples ( state-action transitions ) are no longer i.i.d. , which makes learning challenging , since we should consider the longterm reward rather than the immediate result . How- ever , we notice that data samples being not i.i.d . also makes poisoning attacks challenging . For example , an attacker wants to reduce the agent ’ s total reward in a task shown as Figure 1 ; at state s1 , the attacker finds that a1 is less rewarding than a0 ; if the attacker only looks at the immediate reward , he will lure the agent into choosing a1 . However , following a1 finally leads the agent to s10 which has a much higher reward . Challenge III – Unknown Dynamics of Environment . Although Challenge I and II can be partially addressed by predicting the future trajectories or steps , it requires prior knowledge on the dynamics of the underlying MDP . Many existing poisoning RL works ( Rakhsha et al. , 2020 ; Ma et al. , 2019 ) assume the attacker has perfect knowledge of the MDP , then compute the optimal poisoning . However , in many real-world environments , knowing the dynamics of the MDP is difficult . Although the attacker could potentially interact with the environment to build an estimate of the environment model , the cost of interacting with the environment could be unrealistically high , market making ( Spooner et al. , 2018 ) for instance . In this paper , we study a more realistic scenario where the attacker does not know the underlying dynamics of MDP , and can not directly interact with the environment , either . Thus , the attacker learns the environment only based on the agent ’ s experience . In this paper , we systematically investigate poisoning in RL by considering all the aforementioned RL-specific challenges . Previous works either do not address any of the challenges or only address some of them . Behzadan & Munir ( 2017 ) achieve policy induction attacks for deep Q networks ( DQN ) . However , they treat output actions of DQN similarly to labels in SL , and do not consider Challenge II that the current action will influence future interactions . Ma et al . ( 2019 ) propose a poisoning attack for model-based RL , but they suppose the agent learns from a batch of given data , not considering Challenge I. Rakhsha et al . ( 2020 ) study poisoning for online RL , but they require perfect knowledge of the MDP dynamics , which is unrealistic as stated in Challenge III . Summary of Contributions . ( 1 ) We propose a practical poisoning algorithm called VulnerabilityAware Adversarial Critic Poison ( VA2C-P ) that works for deep policy gradient learners without any prior knowledge of the environment . To the best of our knowledge , VA2C-P is the first practical algorithm that poisons policy-based deep RL methods . ( 2 ) We introduce a novel metric , called stability radius , to characterize the stability of RL algorithms , measuring and comparing the vulnerabilities of RL algorithms in different scenarios . ( 3 ) We conduct a series of experiments for various environments and state-of-the-art deep policy-based RL algorithms , which demonstrates RL agents ’ vulnerabilities to even weaker attackers with limited knowledge and attack budget . 2 RELATED WORK . The main focus of this paper is on poisoning RL , an emerging area in the past few years . We survey related works of adversarial attacks in SL and evasion attacks in RL in Appendix A , as they are out of the scope of this paper . Targeted Poisoning Attacks for RL . Most RL poisoning researches work on targeted poisoning , also called policy teaching , where the attacker leads the agent to learn a pre-defined target policy . Policy teaching can be achieved by manipulating the rewards ( Zhang & Parkes , 2008 ; Zhang et al. , 2009 ) or dynamics ( Rakhsha et al. , 2020 ) of the MDP . However , they require the attackers to not only have prior knowledge of the environments ( e.g. , the dynamics of the MDP ) , but also have the ability to alter the environment ( e.g . change the transition probabilities ) , which are often unrealistic or difficult in practice . Poisoning RL with Omniscient Attackers . Most guaranteed poisoning RL literature ( Rakhsha et al. , 2020 ; Ma et al. , 2019 ) assume omniscient attackers , who not only know the learner ’ s model , but also know the underlying MDP . However , as motivated in the introduction , the underlying MDP is usually either unknown or too complex in practice . Some works poison RL learners by changing the reward signals sent from the environment to the agent . For example , Ma et al . ( 2019 ) introduce a policy teaching framework for batch-learning model-based agents ; Huang & Zhu ( 2019 ) propose a reward-poisoning attack model , and provide convergence analysis for Q-learning ; Zhang et al . ( 2020b ) present an adaptive reward poisoning method for Q-learning ( while it also extends to DQN ) and analyze the safety thresholds of RL ; these papers all assume the attacker knows not only the models of the agent , but also the parameters of the underlying MDP , which could be possible in a tabular MDP , but hard to realize in large environments and modern deep RL systems . On the contrary , we consider non-omniscient attackers who do not know the underlying MDP or environment in this paper . The non-omniscient attackers can be further divided into two categories : white-box attackers , who know the learner ’ s model/parameters , and black-box attackers , who do not know the learner ’ s model/parameters . They both tap the interactions between the learner and the environment . Black-box Poisoning for Value-based Learners . Although there are many successful black-box evasion approaches ( Xinghua et al. , 2020 ; Inkawhich et al. , 2020 ) , black-box poisoning in RL is rare . There is a black-box attacking method for a value-based learner ( DQN ) proposed by Behzadan & Munir ( 2017 ) , which does not require the attacker to know the learner ’ s model or the underlying MDP . In this work , the attacker induces the DQN agent to output the target action by perturbing the state with Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2015 ) in every step . However , the data-correlation problem of RL ( Challenge II ) is not considered , and FGSM attack does not work for policy-based methods due to their high stochasticity , as we show in experiments . In this paper , we propose a new poisoning algorithm for policy-based deep RL agents , which can achieve both non-targeted and targeted attacks . We do not require any prior knowledge of the environment . And our algorithm works not only when the attacker knows the learner ’ s model ( whitebox ) , but also when the leaner ’ s model is hidden ( black-box ) . 3 PROBLEM FORMULATION FOR POISONING ONLINE RL . 3.1 NOTATIONS AND PRELIMINARIES . In RL , an agent interacts with the environment by taking actions , observing states and receiving rewards . The environment is modeled by a Markov Decision Process ( MDP ) , which is denoted by a tuple M = 〈S , A , P , R , γ , µ〉 , where S is the state space , A is the action space , P is the transition kernel , R is the reward function , γ ∈ ( 0 , 1 ) is the discount factor , and µ is the initial state distribution . A trajectory τ ∼ π generated by policy π is a sequence s1 , a1 , r1 , s2 , a2 , · · · , where s1 ∼ µ , at ∼ π ( a|st ) , st+1 ∼ P ( s|st , at ) and rt = R ( st , at ) . The goal of an RL agent is to find an optimal policy π∗ that maximizes the expected total rewards η , which is defined as η ( π ) = Eτ∼π [ r ( τ ) ] = Es1 , a1 , ···∼µ , π , P , R [ ∑∞ t=1 γ t−1rt ] . We use an overhead check signˇon a variable to denote that the variable is poisoned . For example , if the attacker perturbs a reward rt , then the poisoned reward is denoted as řt . If a policy π is updated with poisoned observation , then it is denoted as π̌ . 3.2 THE PROCEDURE OF ONLINE LEARNING AND POISONING Procedure of Online Learning . We consider a classical online policy-based RL setting , where the learner iteratively updates its policy π parametrized by θ , through K iterations with the environment . For notation simplicity , we omit θ and use πk to denote πθk , the learner ’ s policy at iteration k. The online learning process is described as below . At iteration k = 1 , · · · , K , ( 1 ) The agent uses the current policy πk to roll out observation Ok = ( Osk , Oak , Ork ) from environmentMk , where Osk = [ s1 , s2 , · · · ] , Oak = [ a1 , a2 , · · · ] , Ork = [ r1 , r2 , · · · ] are respectively the sequence of states , actions and rewards generated at iteration k. ( 2 ) The agent updates its policy parameters θ with its algorithm f . Most policy-based algorithms perform on-policy updating , i.e. , update policy only by the current observation Ok . The on-policy update can be then formalized as πk+1 = f ( πk , Ok ) ≈ argmaxπJ ( π , πk , Ok ) , where J is an objective function defined by algorithm f , e.g. , the expected total reward η ( π ) . Procedure of Online Poisoning . A poisoning attacker influences the learner in the training process by perturbing the training data . In SL , the training data consists of features and labels , and the attacker poisons the training data before the learning starts . However , the training data in RL is the trajectories a learner rolls out from the environment , i.e. , observation O = ( Os , Oa , Or ) . At iteration k , the attacker eavesdrops on the interaction between the learner and the environment , obtains the observation Ok , and may poison it into Ǒk , then send Ǒk to the learner before policy updating . 1 Procedure 2 in Appendix B illustrates how the online game goes between the learner and the attacker . Figure 2 visualizes this online learning-poisoning procedure , where we can see that learning and poisoning are convoluted and inter-dependent .

Summary: This paper proposes a poisoning algorithm named Vulnerability-Aware Adversarial Critic Poison (VA2C-P) to attack policy-based deep reinforcement learning agents. The poisoning attack is formulated as a sequential bilevel optimisation problem (Problem Q), where the attacker either minimises the expected total rewards of the learner (non-targeted poisoning), or forces the learner to learn a target policy (targeted poisoning). To solve Problem Q, VA2C-P mainly makes two decision: (1) when to attack: a new metric named stability radius is proposed to decide the attack timing, (2) how to attack: a mechanism of adversarial critic is designed to solve a relaxed version of Problem Q by only considering the loss of the immediate next iteration.