Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learning

1 Introduction . Reinforcement learning ( RL ) , a framework of control-theoretic problem that makes decisions over time under uncertain environment , has many applications in a variety of scenarios such as recommendation systems [ Zhao et al. , 2018 ] , autonomous driving [ O ' Kelly et al. , 2018 ] , finance [ Liu et al. , 2020 ] and business management [ Nazari et al. , 2018 ] , to name a few . As RL models are being increasingly used in safety critical and security related applications , it is critical to developing trustworthy RL systems . Understanding the effects of adversarial attacks on RL systems is the first step towards the goal of safe applications of RL models . While there is much existing work addressing adversarial attacks on supervised learning models [ Szegedy et al. , 2014 , Goodfellow et al. , 2015 , Kurakin et al. , 2017 , Moosavi-Dezfooli et al. , 2017 , Wang et al. , 2018 , Cohen et al. , 2019 , Dohmatob , 2019 , Wang et al. , 2019 , Zhang and Zhu , 2019 , Carmon et al. , 2019 , Pinot et al. , 2019 , Alayrac et al. , 2019 , Dasgupta et al. , 2019 , Cicalese et al. , 2020 ] , the understanding of adversarial attacks on RL models is less complete . Among the limited existing works on adversarial attacks against RL , they formally or experimentally considers different types of poisoning attack [ Behzadan and Munir , 2017 , Huang and Zhu , 2019 , Ma et al. , 2019 , Zhang et al. , 2020 , Sun et al. , 2021 , Rakhsha et al. , 2020 , 2021 ] . [ Sun et al. , 2021 ] discusses the differences between the poisoning attacks . In the observation poisoning attack setting , the attacker is able to manipulate the observations of the agent . Before the agent receives the reward signal or the state 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . signal from the environment , the attacker is able to modify the data . In the environment poisoning setting , the attacker could directly change the underlying environment , i.e. , the Markov decision process ( MDP ) model . In this paper , we introduce a suite of novel attacks on RL named action poisoning attacks . In the proposed action poisoning attacks models , an attacker sits between the agent and the environment and could change the agent ’ s action . For example , in auto-driving systems , the attacker could implement destabilizing forces or manipulate the action signal , so as to change the brake force . Compared with the observation poisoning or environment poisoning attacks , the ability of the attacker in the action poisoning attack is more restricted , which brings some design challenges . In particular , compared with observation poisoning and environment poisoning attacks , the effects of the action poisoning attack on the change of observation is less direct . Furthermore , when the action space is discrete and finite , the ability of the action poisoning attacker is severely limited . We note that the goal of this paper is not to promote action manipulation attacks . Our goal is to understand the potential risks of action manipulation attacks . Understanding the risks of different kinds of adversarial attacks on RL is essential for the safe applications of RL model and designing robust RL systems . In this paper , we investigate action poisoning attacks in both white-box and black-box settings . The white-box attack setting makes strong assumptions . In particular , the attacker has full information of the underlying MDP , the agent ’ s algorithm or the agent ’ s previous policy models , or all of them . While it is often unrealistic to exactly know the underlying environment or have the right to obtain the information of the agent ’ s model , the understanding of the white-box attacks could provide insights on how to design black-box attack schemes . In the black-box setting , the attacker has no prior information of the underlying MDP and does not know the agent ’ s algorithm . The only information the attacker has is observations generated from the environment when the agent interacts with the environment . The black-box setting is much more practical and is suitable for more realistic scenarios . Contributions : Our main contributions are as follows : ( 1 ) We propose an action poisoning attack model in which the attacker aims to force the agent to learn a policy selected by the attacker ( will be called target policy in the sequel ) by changing the agent ’ s actions to other actions . We use loss and cost functions to evaluate the effects of the action poisoning attack on a RL agent . The cost is the cumulative number of times when the attacker changes the agent ’ s action , and the loss is the cumulative number of times when the agent does not follow the target policy . It is clearly of interest to minimize both the cost and loss functions . ( 2 ) In the white-box setting , we introduce an attack strategy named α-portion attack . We show that the α-portion attack strategy can force any sub-linear-regret RL agent to choose actions according to the target policy specified by the attacker with sub-linear cost and sub-linear loss . ( 3 ) We develop a black-box attack strategy , LCB-H , that nearly matches the performance of the white-box α-portion attack . To the best of our knowledge , LCB-H is the first black-box action poisoning attack scheme that provably works against RL agents . ( 4 ) We investigate the impact of the LCH-B attack on UCB-H [ Jin et al. , 2018 ] , a popular and efficient model-free Q-learning algorithm , and show that , by spending only logarithm cost , the LCB-H attack can force the UCB-H agent to choose actions according to the target policy with logarithm loss . Related work : Existing works on poisoning attacks against RL have studied different types of adversarial manipulations . [ Ma et al. , 2019 ] studies reward poisoning attack against batch RL in which the attacker is able to gather and modify the collected batch data . [ Rakhsha et al. , 2020 ] proposes a white-box environment poisoning model in which the attacker could manipulate the original MDP to a poisoned MDP . [ Behzadan and Munir , 2017 , Zhang et al. , 2020 ] study online white-box reward poisoning attacks in which the attacker could manipulate the reward signal before the agent receives it . [ Sun et al. , 2021 ] proposes a practical black-box poisoning algorithm called VA2C-P. Their empirical results show that VA2C-P works for deep policy gradient RL agents without any prior knowledge of the environment . [ Rakhsha et al. , 2021 ] develops a black-box reward poisoning attack strategy called U2 , that can provably attack any efficient RL algorithms . There are also some interesting works that focus on attacking multi-arm bandit problems . In particular , [ Jun et al. , 2018 , Liu and Shroff , 2019 ] investigate reward poisoning attacks and show that the attacker can manipulate the behavior of the bandit algorithms by spending only logarithm cost . [ Guan et al. , 2020 ] considers a reward poisoning attack model where an adversary attacks with a certain probability at each round . [ Liu and Lai , 2020 ] proposes an action poisoning attack strategy against multi-arm bandit problems . Existing work on action poisoning attacks against RL is limited . There are some empirical studies in deep RL [ Pinto et al. , 2017 , Lee et al. , 2020 , Sun et al. , 2021 ] . Potential negative societal impacts : The attack strategies discussed in this paper could potentially be used by malicious users to attack real RL systems . The goal of this paper is to understand and highlight the potential negative consequences of action manipulation attacks , with the hope to raise awareness of this issue in the research community . It is also our intention to design robust RL algorithms that can defend against such attacks in our future work . 2 Problem Formulation . Consider a tabular episodic MDPM = ( S , A , H , P , R ) , where S is the state space with |S| = S , A is the action space with |A| = A , H ∈ Z+ is the number of steps in each episode , Ph : S × A × S → [ 0 , 1 ] is the probability transition function which maps state-action-state pair to a probability , Rh : S × A → [ 0 , 1 ] represents the reward function in the step h. In this paper , the probability transition functions and the reward functions can be different at different steps . The agent interacts with the environment in a sequence of episodes . The total number of episodes is K. In each episode k ∈ [ K ] of this MDP , the initial states s1 is generated randomly by a distribution or chosen by the environment . Initial states may be different between episodes . At each step h ∈ [ H ] of an episode , the agent observes the state sh and chooses an action ah . After receiving the action , the environment generates a random reward rh ∈ [ 0 , 1 ] derived from a distribution with mean Rh ( sh , ah ) and next state sh+1 which is drawn from the distribution Ph ( ·|sh , ah ) . Ph ( ·|s , a ) represents the probability distribution over states if action a is taken for state s. The agent stops interacting with environment after H steps and starts another episode . The policy π of the agent is expressed as a mappings π : S × [ H ] ×A → [ 0 , 1 ] . πh ( a|s ) represents the probability of taking action a in state s under stochastic policy π at step h. We have that∑ a∈A πh ( a|s ) = 1 . A deterministic policy is a policy that maps each state to a particular action . For notation convenience , for a deterministic policy π , we use πh ( s ) to denote the action a which satisfies πh ( a|s ) = 1 . Interacting with the environmentM , the policy induces a random trajectory { s1 , a1 , r1 , s2 , a2 , r2 , · · · , sH , aH , rH , sH+1 } . We use V πh : S → R to denote the value function at step h under policy π . Given a policy π and step h , the value function of a state s ∈ S and theQ-functionQπh : S×A → R of a state-action pair ( s , a ) are defined as : V πh ( s ) = E [ ∑H h′=h rh′ |sh = s , π ] and Qπh ( s , a ) = E [ ∑H h′=h rh′ |sh = s , ah = a , π ] , which represent the expected total rewards received from step h to H , under policy π , starting from state s and state-action pair ( s , a ) respectively . It is well-known that the value function and Q-function satisfy the Bellman consistency equations . For notation simplicity , we denote V πH+1 = 0 , QπH+1 = 0 and PhV π h+1 ( s , a ) = Es′∼Ph ( ·|s , a ) [ V πh+1 ( s′ ) ] . In this paper , we assume that the state space S and action space A are finite sets , and the planning horizonH is finite . Reward rh is bounded by [ 0 , 1 ] , so the value function andQ-function are bounded . Under this case , there always exists an optimal policy π∗ such that π∗ maximize the value function and Q-function : V ∗h ( s ) : = V π∗ h ( s ) = supπ V π h ( s ) and Q ∗ h ( s , a ) : = Q π∗ h ( s , a ) = supπ Q π h ( s , a ) , for all s , a and h. We measure the performance of the agent over K episodes by the regret defined as : Regret ( K ) = K∑ k=1 [ V ∗1 ( s k 1 ) − V π k 1 ( s k 1 ) ] , ( 1 ) where sk1 is the initial state and π k is the control policy followed by the agent for each episode k. In this paper , we introduce a novel adversary setting , in which the attacker sits between the agent and the environment . The attacker can monitor the state , the actions of the agent and the reward signals from the environment . Furthermore , the attacker can introduce action poisoning attacks on RL agent . In particular , at each episode k and step h , after the agent chooses an action akh , the attacker can change it to another action ãkh ∈ A . If the attacker decides not to attack , ãkh = akh . Then the environment receives ãkh , and generates a random reward r k h with mean Rh ( s k h , ã k h ) and the next state skh+1 which is drawn from the distribution Ph ( ·|skh , ãkh ) . The agent and attacker receive the reward rkh and the next state s k h+1 from the environment . Note that the agent does not know the attacker ’ s manipulations and the presence of the attacker and hence will still view rkh as the reward and s k h+1 as the next state generated from state-action pair ( skh , a k h ) . The attacker has a target policy π† . We assume that the target policy π† is a deterministic policy . The attacker ’ s goal is to manipulate the agent into following the target policy π† to pick its actions . We measure the performance of the attack over K episodes by the total attack cost and the total number of the steps that the agent does not follow the target policy π† . By setting 1 ( · ) as the indicator function , the attack cost function and the loss function are defined as Cost ( K , H ) = K∑ k=1 H∑ h=1 1 ( ãkh 6= akh ) , Loss ( K , H ) = K∑ k=1 H∑ h=1 1 ( akh 6= π† ( skh ) ) , ( 2 ) The attacker aims to minimize both the attack cost and the loss of attacks , or minimize one of them subject to a constraint on the one another . However , obtaining optimal solutions to these optimization problems is challenging . As the first step towards understanding the impact of action poisoning attacks , we design some specific simple yet effective attack strategies .

Summary: This paper proposes a new action poisoning framework to mislead the agent to learn a target policy. In a white-box setting, the paper introduces $\alpha$-portion attack that is guaranteed to conduct a successful attack with sublinear cost. In a black-box setting, the paper proposes an adaptive attack scheme called LCB-H that nearly matches the performance guarantees of the white-box attack method. With a case study on attacking UCB-H, the authors show that the proposed attack strategy can manipulate the learner with a logarithm cost, matching the main claim of the paper.