Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learning

1 Introduction . Reinforcement learning ( RL ) , a framework of control-theoretic problem that makes decisions over time under uncertain environment , has many applications in a variety of scenarios such as recommendation systems [ Zhao et al. , 2018 ] , autonomous driving [ O ' Kelly et al. , 2018 ] , finance [ Liu et al. , 2020 ] and business management [ Nazari et al. , 2018 ] , to name a few . As RL models are being increasingly used in safety critical and security related applications , it is critical to developing trustworthy RL systems . Understanding the effects of adversarial attacks on RL systems is the first step towards the goal of safe applications of RL models . While there is much existing work addressing adversarial attacks on supervised learning models [ Szegedy et al. , 2014 , Goodfellow et al. , 2015 , Kurakin et al. , 2017 , Moosavi-Dezfooli et al. , 2017 , Wang et al. , 2018 , Cohen et al. , 2019 , Dohmatob , 2019 , Wang et al. , 2019 , Zhang and Zhu , 2019 , Carmon et al. , 2019 , Pinot et al. , 2019 , Alayrac et al. , 2019 , Dasgupta et al. , 2019 , Cicalese et al. , 2020 ] , the understanding of adversarial attacks on RL models is less complete . Among the limited existing works on adversarial attacks against RL , they formally or experimentally considers different types of poisoning attack [ Behzadan and Munir , 2017 , Huang and Zhu , 2019 , Ma et al. , 2019 , Zhang et al. , 2020 , Sun et al. , 2021 , Rakhsha et al. , 2020 , 2021 ] . [ Sun et al. , 2021 ] discusses the differences between the poisoning attacks . In the observation poisoning attack setting , the attacker is able to manipulate the observations of the agent . Before the agent receives the reward signal or the state 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . signal from the environment , the attacker is able to modify the data . In the environment poisoning setting , the attacker could directly change the underlying environment , i.e. , the Markov decision process ( MDP ) model . In this paper , we introduce a suite of novel attacks on RL named action poisoning attacks . In the proposed action poisoning attacks models , an attacker sits between the agent and the environment and could change the agent ’ s action . For example , in auto-driving systems , the attacker could implement destabilizing forces or manipulate the action signal , so as to change the brake force . Compared with the observation poisoning or environment poisoning attacks , the ability of the attacker in the action poisoning attack is more restricted , which brings some design challenges . In particular , compared with observation poisoning and environment poisoning attacks , the effects of the action poisoning attack on the change of observation is less direct . Furthermore , when the action space is discrete and finite , the ability of the action poisoning attacker is severely limited . We note that the goal of this paper is not to promote action manipulation attacks . Our goal is to understand the potential risks of action manipulation attacks . Understanding the risks of different kinds of adversarial attacks on RL is essential for the safe applications of RL model and designing robust RL systems . In this paper , we investigate action poisoning attacks in both white-box and black-box settings . The white-box attack setting makes strong assumptions . In particular , the attacker has full information of the underlying MDP , the agent ’ s algorithm or the agent ’ s previous policy models , or all of them . While it is often unrealistic to exactly know the underlying environment or have the right to obtain the information of the agent ’ s model , the understanding of the white-box attacks could provide insights on how to design black-box attack schemes . In the black-box setting , the attacker has no prior information of the underlying MDP and does not know the agent ’ s algorithm . The only information the attacker has is observations generated from the environment when the agent interacts with the environment . The black-box setting is much more practical and is suitable for more realistic scenarios . Contributions : Our main contributions are as follows : ( 1 ) We propose an action poisoning attack model in which the attacker aims to force the agent to learn a policy selected by the attacker ( will be called target policy in the sequel ) by changing the agent ’ s actions to other actions . We use loss and cost functions to evaluate the effects of the action poisoning attack on a RL agent . The cost is the cumulative number of times when the attacker changes the agent ’ s action , and the loss is the cumulative number of times when the agent does not follow the target policy . It is clearly of interest to minimize both the cost and loss functions . ( 2 ) In the white-box setting , we introduce an attack strategy named α-portion attack . We show that the α-portion attack strategy can force any sub-linear-regret RL agent to choose actions according to the target policy specified by the attacker with sub-linear cost and sub-linear loss . ( 3 ) We develop a black-box attack strategy , LCB-H , that nearly matches the performance of the white-box α-portion attack . To the best of our knowledge , LCB-H is the first black-box action poisoning attack scheme that provably works against RL agents . ( 4 ) We investigate the impact of the LCH-B attack on UCB-H [ Jin et al. , 2018 ] , a popular and efficient model-free Q-learning algorithm , and show that , by spending only logarithm cost , the LCB-H attack can force the UCB-H agent to choose actions according to the target policy with logarithm loss . Related work : Existing works on poisoning attacks against RL have studied different types of adversarial manipulations . [ Ma et al. , 2019 ] studies reward poisoning attack against batch RL in which the attacker is able to gather and modify the collected batch data . [ Rakhsha et al. , 2020 ] proposes a white-box environment poisoning model in which the attacker could manipulate the original MDP to a poisoned MDP . [ Behzadan and Munir , 2017 , Zhang et al. , 2020 ] study online white-box reward poisoning attacks in which the attacker could manipulate the reward signal before the agent receives it . [ Sun et al. , 2021 ] proposes a practical black-box poisoning algorithm called VA2C-P. Their empirical results show that VA2C-P works for deep policy gradient RL agents without any prior knowledge of the environment . [ Rakhsha et al. , 2021 ] develops a black-box reward poisoning attack strategy called U2 , that can provably attack any efficient RL algorithms . There are also some interesting works that focus on attacking multi-arm bandit problems . In particular , [ Jun et al. , 2018 , Liu and Shroff , 2019 ] investigate reward poisoning attacks and show that the attacker can manipulate the behavior of the bandit algorithms by spending only logarithm cost . [ Guan et al. , 2020 ] considers a reward poisoning attack model where an adversary attacks with a certain probability at each round . [ Liu and Lai , 2020 ] proposes an action poisoning attack strategy against multi-arm bandit problems . Existing work on action poisoning attacks against RL is limited . There are some empirical studies in deep RL [ Pinto et al. , 2017 , Lee et al. , 2020 , Sun et al. , 2021 ] . Potential negative societal impacts : The attack strategies discussed in this paper could potentially be used by malicious users to attack real RL systems . The goal of this paper is to understand and highlight the potential negative consequences of action manipulation attacks , with the hope to raise awareness of this issue in the research community . It is also our intention to design robust RL algorithms that can defend against such attacks in our future work . 2 Problem Formulation . Consider a tabular episodic MDPM = ( S , A , H , P , R ) , where S is the state space with |S| = S , A is the action space with |A| = A , H ∈ Z+ is the number of steps in each episode , Ph : S × A × S → [ 0 , 1 ] is the probability transition function which maps state-action-state pair to a probability , Rh : S × A → [ 0 , 1 ] represents the reward function in the step h. In this paper , the probability transition functions and the reward functions can be different at different steps . The agent interacts with the environment in a sequence of episodes . The total number of episodes is K. In each episode k ∈ [ K ] of this MDP , the initial states s1 is generated randomly by a distribution or chosen by the environment . Initial states may be different between episodes . At each step h ∈ [ H ] of an episode , the agent observes the state sh and chooses an action ah . After receiving the action , the environment generates a random reward rh ∈ [ 0 , 1 ] derived from a distribution with mean Rh ( sh , ah ) and next state sh+1 which is drawn from the distribution Ph ( ·|sh , ah ) . Ph ( ·|s , a ) represents the probability distribution over states if action a is taken for state s. The agent stops interacting with environment after H steps and starts another episode . The policy π of the agent is expressed as a mappings π : S × [ H ] ×A → [ 0 , 1 ] . πh ( a|s ) represents the probability of taking action a in state s under stochastic policy π at step h. We have that∑ a∈A πh ( a|s ) = 1 . A deterministic policy is a policy that maps each state to a particular action . For notation convenience , for a deterministic policy π , we use πh ( s ) to denote the action a which satisfies πh ( a|s ) = 1 . Interacting with the environmentM , the policy induces a random trajectory { s1 , a1 , r1 , s2 , a2 , r2 , · · · , sH , aH , rH , sH+1 } . We use V πh : S → R to denote the value function at step h under policy π . Given a policy π and step h , the value function of a state s ∈ S and theQ-functionQπh : S×A → R of a state-action pair ( s , a ) are defined as : V πh ( s ) = E [ ∑H h′=h rh′ |sh = s , π ] and Qπh ( s , a ) = E [ ∑H h′=h rh′ |sh = s , ah = a , π ] , which represent the expected total rewards received from step h to H , under policy π , starting from state s and state-action pair ( s , a ) respectively . It is well-known that the value function and Q-function satisfy the Bellman consistency equations . For notation simplicity , we denote V πH+1 = 0 , QπH+1 = 0 and PhV π h+1 ( s , a ) = Es′∼Ph ( ·|s , a ) [ V πh+1 ( s′ ) ] . In this paper , we assume that the state space S and action space A are finite sets , and the planning horizonH is finite . Reward rh is bounded by [ 0 , 1 ] , so the value function andQ-function are bounded . Under this case , there always exists an optimal policy π∗ such that π∗ maximize the value function and Q-function : V ∗h ( s ) : = V π∗ h ( s ) = supπ V π h ( s ) and Q ∗ h ( s , a ) : = Q π∗ h ( s , a ) = supπ Q π h ( s , a ) , for all s , a and h. We measure the performance of the agent over K episodes by the regret defined as : Regret ( K ) = K∑ k=1 [ V ∗1 ( s k 1 ) − V π k 1 ( s k 1 ) ] , ( 1 ) where sk1 is the initial state and π k is the control policy followed by the agent for each episode k. In this paper , we introduce a novel adversary setting , in which the attacker sits between the agent and the environment . The attacker can monitor the state , the actions of the agent and the reward signals from the environment . Furthermore , the attacker can introduce action poisoning attacks on RL agent . In particular , at each episode k and step h , after the agent chooses an action akh , the attacker can change it to another action ãkh ∈ A . If the attacker decides not to attack , ãkh = akh . Then the environment receives ãkh , and generates a random reward r k h with mean Rh ( s k h , ã k h ) and the next state skh+1 which is drawn from the distribution Ph ( ·|skh , ãkh ) . The agent and attacker receive the reward rkh and the next state s k h+1 from the environment . Note that the agent does not know the attacker ’ s manipulations and the presence of the attacker and hence will still view rkh as the reward and s k h+1 as the next state generated from state-action pair ( skh , a k h ) . The attacker has a target policy π† . We assume that the target policy π† is a deterministic policy . The attacker ’ s goal is to manipulate the agent into following the target policy π† to pick its actions . We measure the performance of the attack over K episodes by the total attack cost and the total number of the steps that the agent does not follow the target policy π† . By setting 1 ( · ) as the indicator function , the attack cost function and the loss function are defined as Cost ( K , H ) = K∑ k=1 H∑ h=1 1 ( ãkh 6= akh ) , Loss ( K , H ) = K∑ k=1 H∑ h=1 1 ( akh 6= π† ( skh ) ) , ( 2 ) The attacker aims to minimize both the attack cost and the loss of attacks , or minimize one of them subject to a constraint on the one another . However , obtaining optimal solutions to these optimization problems is challenging . As the first step towards understanding the impact of action poisoning attacks , we design some specific simple yet effective attack strategies .

Summary: The paper studies poisoning attacks in reinforcement learning that override an agent's actions in order to force the policy of interest (target policy). The problem of 'action poisoning' is formulated in an episodic reinforcement learning framework, where the attacker experiences the cost of poisoning actions (equal to the number of poisoned/changed actions) and the loss of not fully forcing the target policy (equal to the number of times target policy was not followed). The paper formally studies white-box and black-box poisoning, developing a principled way of performing action poisoning attacks with provable guarantees. The main algorithmic contribution of this paper is an algorithm called LCB-H that has provable bounds on the cost and loss of poisoning. The numerical experiments are conducted to demonstrate the efficacy of the proposed black box attack.