Efficient Reward Poisoning Attacks on Online Deep Reinforcement Learning
Authors: Yinglun Xu
Qi Zeng
Gagandeep Singh
Venue: Submitted to ICLR 2023 (2023)

Abstract: We study reward poisoning attacks on online deep reinforcement learning (DRL), where the attacker is oblivious to the learning algorithm used by the agent and does not necessarily have full knowledge of the environment. We demonstrate the intrinsic vulnerability of state-of-the-art DRL algorithms by designing a general, black-box reward poisoning framework called adversarial MDP attacks. We instantiate our framework to construct several new attacks which only corrupt the rewards for a small fraction of the total training timesteps and make the agent learn a low-performing policy. Our key insight is that state-of-the-art DRL algorithms strategically explore the environment to find a high-performing policy. Our attacks leverage this insight to construct a corrupted environment where (a) the agent learns a high-performing policy that has low performance in the original environment and (b) the corrupted environment is similar to the original one so that the attacker's budget is reduced.  We provide a theoretical analysis of the efficiency of our attack and perform an extensive evaluation. Our results show that our attacks efficiently poison agents learning with a variety of state-of-the-art DRL algorithms, such as DQN, PPO, SAC, etc., under several popular classical control and MuJoCo environments.

Under review as a conference paper at ICLR 2023
EFFICIENT REWARD POISONING ATTACKS ON ONLINE DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We study reward poisoning attacks on online deep reinforcement learning (DRL), where the attacker is oblivious to the learning algorithm used by the agent and does not necessarily have full knowledge of the environment. We demonstrate the intrinsic vulnerability of state-of-the-ar DRL algorithms by designing a general, black-box reward poisoning framework called adversarial MDP attacks. We in- stantiate our framework construct several new attacks which only corrupt the rewards for a small fraction of the total training timesteps and make the agent learn low-performing policy. Our key insight is that state-of-the-art DRL al- gorithms strategically explore the environment to find a high- tigh-performing policy. Our attacks leverage this insight construct a corrupted environment where (a) the agent learns high-performing policy that has low performance in the origi- nal environment and (b) the corrupted environment is similar to the original one that the attacker's budget is reduced. We provide theoretical analysis of the efficiency of our attack and perform an extensive evaluation. Our results show that our attacks efficiently poison agents learning with a variety of state-of-the-att DRL algorithms, such as DQN, PPO, SAC, etc., under several popular classical control and MuloCo environments.
INTRODUCTION
In several important applications such as robot control (Christiano et al. 2017) and recommendation systems (Afsar et al. [202]] Zheng et al.] 2018), oftt-of-thh-arr online deep reinforcement learning (DRL) algorithms rely on human feedbacks in terms of rewards, for learning high-performing poli- cies. This dependency raises the threat of reward-based data poisoning attacks during training: a user deliberately provide malicious rewards make the DRL agent learn low-performing policies. Data poisoning has already been identified as the most critical security concern when employing learned models in industry (Kumar et al., 2020). Thus, it is essential study whether state-of- the-art DRL algorithms are vulnerable to reward poisoning attacks to discover potential security vulnerabilities and motivate the development of more robust training algorithms.
Challenges in poisoning DRL agents. To uncover practical vulnerabilities, it is critical that the attack does not rely on unrealistic assumptions about the attacker's capabilities. Therefore for en- suring practically feasible attack, we require that: (i) the attacker has knowledge of the exact DRL algorithm used by the agent as well as the parameters of the neural network used for training. Further, it should be applicable to different kinds of learning algorithms (e.g., policy optimization. Q learning) (ii) the attacker does not have detailed knowledge about the agent's environment, and (iii) to ensure stealthy, the amount of reward corruption applied by the attacker is limited (see Section B As we show in Appendix G these restrictions make finding an efficient attack very challenging.
This work: efficient poisoning attacks DRL. To the best of our knowledge, prior work stud- ies the vulnerability of the DRL algorithms to reward poisoning attacks under the practical restric- tions mentioned above. To overcome the challenges in designing efficient attacks and demonstrate the vulnerability of the state-of-the-art DRL algorithms, we make the following contributions:
1. We propose a general, efficient, and parametric reward poisoning framework for DRL algorithms, which we call adversarial MDP attack, and instantiate it to generate several attack methods that are applicable to any kind of learning algorithms and computationally efficient. To the best of our knowledge, our attack is the first one that considers the following four key elements in the

---

Under review as a conference paper at ICLR 2023
threat model at the same time: 1. Training time attack, 2. Deep RL, 3. Reward poisoning attack,
4. Complete black box attack (no knowledge or assumption about the learning algorithm and the environment). A detailed explanation for each key point is provided in Appendix A
2. We provide a theoretical analysis of our attack methods based on certain assumptions on the efficiency of the DRL algorithms which yields several insightful implications.
3. We provide an extensive evaluation of our attack methods for poisoning the training with sev- eral state-of-the-art DRL algorithms, such as DQN, PPO, SAC, etc., in the classical control and MuJoCo environments, commonly used for developing and testing DRL algorithms. Our results show that our attack methods significantly reduce the performance of the policy learned by the agent in the majority of the cases and are considerably more efficient than baseline attacks (c.g., VA2C-P (Sun et al. 2020). reward-ffippin (Zhang et al., 2021b)). We further validate the impli- cations of our theoretical analysis by observing the corresponding phenomena in experiments.
RELATED WORK
Testing time attack on RL. Testing time attack (evasion attack) deep RL setting is popular in literature (Huang et al., 2017: Kos & Song. 2017 Lin et al.] (017)) For an already trained policy, testing time attacks find adversarial examples where the learned policy has undesired behavior. In contrast, training time attack corrupts reward to make the agent learn low low-performing policies.
Data poisoning attack on bandit and tabular RL settings. Jun et al. (2018); Liu & Shroff (2019);
Xu et al. (2021b) study data poisoning attack against bandit algorithms. Ma et al. (2019) studies the attack in the offline tabular RL setting. Rakhhsa et al. (2020); Zhang et al.] (2020b) study the online tabular RL setting relying on full or partial knowledge of the environment and the learning algorithm. Liu & Lai| (2021) 1); Xu et al. (2021a) discuss the attack that can work with no knowledge or weak assumptions on the learning algorithm or the environment. Both tabular and bandit settings are simpler than the deep RL setting considered in our work.
Observation perturbation attack and defense. There is a line of work studying observation per- turbation attacks during training time (Behzadan & Munir 2017a b Inkawhich et al. 2019) and the corresponding defense (Zhang al., 202 2020a). The threat model here does not change the actual state or reward of the environment, but instead, it changes the learner's 's observation of the environment by generating adversarial examples. In contrast, for the poisoning attack as considered in our work, the actual reward or state of the environment is changed by the attacker. The obser- vation perturbation attack assumes access to perturb the sensor of the agent that is used to observe the environment. Therefore, it is not practical when the attacker does not have access to the agent's sensor, or the agent does not rely on sensors for interacting with the environment.
Data poisoning attack on DRL. The work Sun et al. (2020) is the only other work that considers reward poisoning attack on DRL and therefore is the closest to ours. There are three main limitations of their attack compared to ours (a) the attack requires the knowledge of the learning algorithm (the update rule for learned policies) used by the agent, which is not the complete black box setting, (b) the attack only works for on-policy learning algorithms, and (c) the attacker in their setting makes the decision about attacking after receiving whole training batch. This makes the attack infeasible when the agent updates the observation at each time step, as in this case it is impossible for the attacker to apply corruption to previous observations in a training batch. We experimentally compare against them by adapting our general attacks to their restricted setting. Our results in Appendix ! show that our attack requires much less computational resources and achieves better attack results.
Robust learning algorithms against data poisoning attack. Robust learning algorithms can guar- antee efficient learning under the data poisoning attack. There have been studies on robustness in the bandit (Lykouris et al., 2018. Gupta et al. 2019), and tabular MDP settings (Chen et al., 202T Wu et al.) 2021: Lykouris et al. 2021). but these results are not applicable in the more complex DRL setting. For the DRL setting, Zhang et al. (20216) proposes a learning algorithm guaranteed to be robust in simplified DRL setting under strong assumptions function and finite action space). The algorithm is further empirically tested in actual DRL settings, but the attack method used for testing robustness, which we call reward flipping attack, is not very efficient and malicious as we show in Appendix H Testing against weak attack methods can provide false sense of security. Our work provides attack methods that more suitable for empirically measuring the robustness of learning algorithms.
the environment (e.g., linear Q
2

---

Under review as a conference paper at ICLR 2023
3 BACKGROUND
Reinforcement learning. We consider a standard RL setting where an agent trained by interacting with an environment. The interaction involves the agent observing a state representation of the environment, taking an action, and receiving a reward. Formally, an environment represented by a Markov decision process (MDP). M {S, A, P, R, ju), where S is the state space, A is the action space, P is the state transition function, R is the reward function, and p is the distribution of the initial states. The training process consists of multiple episodes where each episode is initialized with a state sampled from μ., and the agent interacts with the environment in each episode until it terminates. A policy ı S + ((A) is mapping from the state space the space of probability distribution ((A) over the action space. If a policy и is deterministic, we use T(s) to represent the action it suggests for state 8. A value function V (s) is the expected reward agent obtains by following the policy ι starting at state in the environment M. We denote VM == Ber-y V (so) as the policy value for a policy π in M, which measures the performance of и. The goal of the RL agent is to find the optimal policy with the highest policy value xt" arg max, Vī- For ease of analysis, the state distribution is defined as (s) = E7 E,, [[ss s]]. representing how often state is visited under policy π in an episode.
Reward poisoning attack on deep RL. In this work we consider standard data poisoning attack setting (Jun et al. 2018 Rakhsha et al., 2020) where a malicious adversary tries to manipulate the agent by poisoning the reward received by the agent from the environment during training. The attacker observes the current state, action, and reward tuple (s' a² rrr) generated during training at each timestep t and injects a corruption A' on the true reward r², As a result, the environment returns the agent with the corrupted observation (s',a', , ,rt + A+) where s*+1 is the next state. Next, we describe the restrictions on the attacker's capabilities as mentioned in the introduction:
1. Limited budget. The attacker can only corrupt a small number of timesteps C, i.e.. E 1(4' # 0} CC and C' < T where T is the total number of training steps.
2. Limited per-step corruption. The corruption at each timestep is limited by | S B, Vt E [Z].
3. Limited per-episode corruption: The total corruption across an episode is limited by Exer A S F where 1e is the set of all timesteps in an episode e.
4. Oblivious of the DRL algorithm. The attacker has no knowledge of the training algorithm or any parameters in the network used by the agent while training.
Oblivious of the environment. The attacker has no knowledge about the MDP M except for the number of dimensions and range of each dimension in the state and action space S, A. Our attacks do not need knowledge of AM but can benefit from having access to a good performing policy in M which could be trained by itself, another agent, or publicly available. We consider these additional cases to study the impact of increasing attacker resources on its efficiency.
Let To be the best learned policy when the DRL. training finishes, the goal of the attacker is to corrupt training such that the performance of the learned policy xn in the environment M: V is low. Note that V is an intrinsic property of a given policy æ in M and is independent of the learning algorithm. Reward poisoning does not change V/ but instead makes the agent learn policy with lower VT For clarity, we summarize the goal, knowledge, and constraints for our attacker in Appendix [B] We consider multiple constraints on the attacker to make the setting more realistic defined above. The full constraints are considered in experiments in Section 6 For the purpose of theoretical analysis, we simplify the problem by dropping the constraint on per-episode corruption E in Section 4 and Section 5
4 FORMULATING REWARD POISONING ATTACK
A reward poisoning attack algorithm can be represented by its attack strategy A' each timestep during training. An attack strategy A' depends on the full observation before that attack, that is, all the states gl:t actions a:t rewards ,l=t-1, The output of A4' i.e, the corruption on reward, satisfies A' = A't 1:t ,.:t-1). A practical attack should be constructed in a computationally efficient manner and should work in a practical setting where it is oblivious to the environment and learning algorithm. In Appendix G we show that searching for the optimal or near-optimal attack is computationally hard regardless of the attack constraints in the DRL setting and requires full knowledge of the learning algorithm and environment. Since it is hard to construct both optimal and
3

---

Under review as a conference paper at ICLR 2023
practical attacks, we focus finding feasible attacks that are not necessarily the optimal ones but still make a learning algorithm learn a policy with low policy value with a limited budget. Formally, the attacker's objective, i... finding feasible attack, can be stated using the following constraints:
find A s.t. VM S V: Σ 1 + s <cIAA S [T].
(1)
To solve equation one way is to fix the values of V. B, and C' and solve for an attack A'. However, it is possible that no solution is feasible for certain V, B, and C. Since each feasibility check can be expensive, we look for a more efficient and convenient way to solve equation [ We fix the attack and estimate the corresponding V. B, and C' to satisfy equation I An attack algorithm is efficient if it can satisfy equation I with low values for B, C, V with V < VM otherwise the attack is trivial as VMM S V holds without any attack. As confirmed by our theoretical analysis in Section 5 and experiments in Section 6. finding an efficient attack based on equation D is non-trivial.
Adversarial MDP attack. To find attack algorithms for solving equation | we introduce general parametric attack framework called "adversarial MDP attack" for poisoning the training of deep RL agents. The high-level idea behind our attack is to construct a fixed adversarial environment to train the agent. This idea has been applied in designing attacks in the simpler bandit (Liu & Shroff. 2019) and tabular setting Rakhsha et al. (020) where they formulate the problem of finding the best adversarial environment for their attack goal as an optimization problem and solve it directly. Solving such an optimization problem is computationally infeasible in the deep RL setting due the complexity of both the environment and the learning algorithm. Therefore we design new efficient algorithms that are suited to the deep RL setting and our attack scenario. In our attack, the attacker constructs an adversarial MDP M = AS P, R. Eu} for the agent to train on by injecting the corrupted reward R to the environment during training. More specifically, for an adversarial MDP attack with M, its attack strategy at round only depends on the agent's 's current state and action:
2 = A'(s",a") = -R(s'' aa)= R(s".a²)
(2)
Next, we compute bounds on V, B, and C such that AIT constructed using equation 2 from a given M in our framework is a feasible solution to equation 1 The lower these values, the more efficient is the adversarial MDP M. We make two simplifying assumptions: (i) the learning algorithm can always find and report the optimal policy from a fixed environment, i.e.,.p = x² always holds. We note that our experimental results show that our attack succeeds with low values of V, B and C on -tate-of-the-ae deep RL algorithms that do not always learn an optimal policy, and (ii) the learning algorithm explores strategically, that is, instead of uniformly exploring all state-action pairs, the algorithm does not waste many rounds to explore the state action pairs that have little value. This assumption is satisfied by RL algorithms (Dong et al., 2019 Jin et al. 2018 Agarwal et al. 2019) and also validated in our experiments.
Lower bounds on V, B, and C for given M. The bound on V relates to the optimal policy ** = arg max, M that the algorithm will learn under M based on our first assumption, i.e., To
R* We can directly bound V by the policy value of カ* under M: V 2 V It is straightforward to bound B as 2 l| Rl For bounding C, our attack applies corruption whenever the learning algorithm chooses an action a² at state s² such that the reward function at this state action pair are different for the real and adversarial MDP, i.e, R(s", a²) + R(s", ,a'). Then C' can be bound as C IT= 1[R(st, a') # R(s", a²)]. Under our second assumption on strategic exploration of the learning algorithm, the bound on C' will be low if M satisfies R(s, A **))) = R(s, (s)) for all states as for most of the time a² (s), resulting in 1 [ a²) + R(s ad)] = 0.
Effect of given V, B and C on M. Raising the value of B or C increases the number of adversarial MDP's that satisfy equation I To achieve the same value of V, a larger value of B can reduce the requirement on C to ensure existance of feasible solutions, and vice versa. Next, we will instantiate our framework to construct specific M that result in efficient attacks which significantly reduce the performance of the learned policy under low budget.
4

---

Under review as a conference paper at ICLR 2023
5 POISONING ATTACK METHODS
In this section, we design new reward poisoning attacks for deep RL by instantiating our adversarial MDP attack framework. Each instantiation provides parameterized way to construct an adversarial MDP.M M via parameter A, corresponding the amount of reward corruption for poisoning applied by the attacker at timestep. The definition of A yields A -| resulting in lower bound on requirement B 2 AJ. For a given value of V, let A(V) ) be the minimum absolute value of the parameter A for the instantiation to satisfy the requirement on V in equation [ We note that A(V) We provide
symbolic expressions for ((V) corresponding to each instantiation and then use [] upper bound on AVV), which is easier to reason about than exact expressions. We define G. and B. with respect to value V to be the set of all policies with policy value > V and < V respectively under M. The upper bound on A (V) will be constructed using G. and B We will empirically examine the requirement on C for each instantiation in Section An attack is efficient if to satisfy certain value of V in equation I it requires low value of C' and B = A(V). All the attack methods we are going to propose do not require any knowledge about both the learning algorithm and the environment, though one method gain benefits from full or partial knowledge of the environment as we find in practice. For simplicity, we start with attacks on environments with discrete action space and then show how to transfer the attack and the corresponding analysis to continuous action space. Note that our analysis in this section is based on the two assumptions we made in Section 4 that the learning algorithms can always learn the optimal policy and explore strategically. All the proofs for the theorems and lemmas can be found in the Appendix E
corresponds to the minimum requirement on B for the attack to satisfy V in equation construct an
Uniformly random time(URR attack. We first introduce trivial instantiation of adversarial MDP attack framework as a baseline, which we call the UR attack. Here, the attacker randomly decides whether to corrupt the reward at timestep with fixed probability p and amount A regardless of the current state and action. Formally, the attack strategy of the UR attacker at time / is: A'(s", a²) A with probability p. otherwise (s "is",aa" a²) O. Let Vnax n V% and Viin min, Vi be the maximum and minimum policy value computed over all policies under M. We provide the following bounds on A(V) for the UR attack:
Theorem 5.1. For the UR attack parameterized with probability p, the exact expression and an upper bound on ((V) are:
VM VTT TIEBM MEGM p- - - L p n.-ming EGM m
V Viax=
Vinin
A(V) = min max
- LT
where L =====; E. is the expected length of an episode for agent following the policy n. Moreover, the expression only holdsfor for A with the same sign as L - for the Tı, T2 that realize the min-max condition.
Theorem 5. shows that the value of A(V) depends on the difference in episode length from policies in G.M and B.M. A low value of p also makes high value of AVV). In addition, the sign of A needs to be chosen correctly make D V otherwise it can make the optimal policies look even better under M, which yields the following implications:
Implication 1. Compared to the UR attack in the right direction, the UR attack in the wrong direction requires higher value of A to make the learning agent learn the policy with the same performance (expected reward per episode), or even worse: the attack in the wrong direction can never prevent the learning agent from finding the optimal policy.
Implication 1 suggests that for the UR attack, it is important to find the right direction for corruption.
Learned policy evasion (L.P)) attack. The high-level idea behind the LPE attack to make all policies of good performance appear bad to the learning agent. Intuitively, policies of good per- formance should share similar behavior as there is usually a certain general strategy to behave well in the environment. Therefore if the attacker can make the actions corresponding to such behav- ior look bad, then all the good policies will appear bad to the agent. Formally, the LPE attack is characterized by the policy x available to the attacker, and it penalizes the learning algorithm whenever it chooses an action that corresponds to π². Correspondingly, the attack strategy is: A As(s", (ss,aa) a²) A faa (s())] where A 0 is a fixed value. I! is learned offline by the
5

---

Under review as a conference paper at ICLR 2023
attacker before the agent starts learning. We will show different ways the attacker can generate ot in Section 6
Next, we analyze the efficiency of the attack. To help our analysis, we introduce the following definition to measure the similarity ( m²) between two policies xi and n2:
Definition 5.2. (Similarity between policies) The similarity of policy to policy m is: D(T11 m2) = - 11a (s)]( = (=)))].
The similarity of Iı to 22 increases with the frequency with which Ta takes the same action m in the same state s. Note that D(11, x2) + D( T1), and ( T2) S L
Theorem 5.3. For LPE attack with n², the expression and an upper bound on A(V) are:
T VVM TEB TEGY D( T D(T1, TT)
V Vmax= Vmnn
A(V) = min max
S min D(T,, πt) min BM D(T,, (T,, mt)
In Appendix B we will show that e D(T, =1) is likely to be 0 in general cases. Theorem 5.3 shows that the requirement on A for the LPE attack is inversely proportional to the minimum similarity between nt and a policy from G. In practice, we observe that in most cases there are usually certain behaviors shared in common by the non-trivial policies that have better performance than random ones. This yields the following implications:
Implication 2. With the same value of A, the LPE attack can make the agent learn policies of worse performance with high performing I compared to the LPE attack with random policy.
The LPE attack can generate random policy as x². For experiments in Section to simplify im- plementation, we generate the random policy through random initialization of a learning algorithm different from the one used by the agent. When the attacker has access to a high-performing policy, can be used as x² which may provide better attack performance. Implication 2 suggests that the LPE attack should use high-performing policy to attack if possible. To estimate the requirement on C, we give the following lemma:
Vo = D(T!, #*) = 0. Given sufficient A, the optimal policy under M given by the LPE attack with n satisfies D(=1, **) 0.
Lemma 5.4. For the LPE attack, R(s,**
(8),
Recall that in Section 4 we estimate that the requirement on C will be low if the adversarial and real reward are the same all (s, (,**(ss) (s)). i... R(s, (s)) R(s, A* (s)). Lemma 5.4 suggests that this condition will hold for the LPE attack given sufficient A, as for the LPE attack D( f*) =
0 R(s, * = R(s, 4" (s)). This yields
Implication 3: Under the LPE attack with sufficient | the learning agent will gradually
converge to 4" under M eventually and few corruptions will be applied afterward, resulting in decrease in attack frequency as the training goes on.
Random policy inducing (RPI) attack. The intuitive idea behind the RPI attack is to make the agent believes that a random policy is an optimal one. To achieve this, the attacker can make all the actions that are different from the ones given by the random policy look bad. The RPI attack is characterized by a randomly generated policy π!, and it penalizes the learning algorithm whenever it doesn't 't follow the action that corresponds to st. Formally, the attack strategy of the RPI attack with a policy I is: (s' , aa)) = 1[at + T (s')), where A < 0 is a fixed value. We have the following expression and upper bound on (VV)
Theorem 5.5. For RPI attack with policy nt, the expression and an upper bound on A(V) are:
V - VM
Vmax Vioin
A(V) min max Ti TBE BGGY (T.72 L
<
(D ( mt) D(T1,
min (Ix D(T,, xf))
Theorem 5.5 suggests that the requirement on B will be low if all policies in G.M have low similarity to ... With the same observation we give about the similarity between policies better than the random ones, it suggests that A(V) will be less when It is random policy. Then theorem 5.5 gives

---

Under review as a conference paper at ICLR 2023
Implication 4: With the same value of A, the RPI attack with random #' can make the learning algorithm learn policies of worse performance than the RPI attack with high performing n!
Implication 4 suggests that the RPI attack should always use a random policy for the attack. To estimate the requirements on C, we give the following lemma:
Lemma 5.6. For the RPI attack, R(s, Rt (s)) = R(ss, ** (s)) = x= = vf. Given sufficient A, the optimal policy under M constructed by the RPI attack with T² is f T/.
Lemma 5.6 suggests that with sufficient value of |اه, R(s, A (s)) = R(s, f (s)) will hold, re- sulting in low requirement on C. This also implies that
Implication 5: For the RPI attack with sufficient |قا, the learning agent will gradually con- verge to n², resulting in a decrease in the frequency of corruption as the training goes on.
Our two main attack methods LPE and RPI proposed so far work with negative corruption on reward and avoid attacking the action corresponding to f" in every state. The methods will work well under our assumptions on learning algorithms. An alternative is to attack with positive corruption on the reward. Under the framework of adversarial MDP attack, such attack can result in less require- ments on C' when the learning agent does the opposite to our second assumption, i.e., exploring the sub-optimal actions more often. To compare the difference in performance between these types of attack, we propose a variant of RPI attack called random policy promoting (RPP) attack.
Random policy promoting (RPP) attack. The RPP attack shares the same intuition about high- lighting a random policy, but instead, it positively rewards the actions corresponding to the random policy. Formally, the attack strategy of RPP attack with a policy Λt is A'(s', a²) = A 1fa T! (s"),, where A > 0 is a fixed value. The expression and an upper bound on A(V) are:
Theorem 5.7. Under RPP attack with policy n², the expression and upper bound for A(V) are:
VM VM
Vina
Vnnia
A(V)
min BLEBy E2EGv D( T|) x() max (Lx max D(T, ),
We note that compared to the RPI attack, the RPP attack requires more Bif < min LT" as it results in L= - max D(T,, x!)= < minxeG. L max D(x, T)) S min cam (L* D(TT, r)). Further if T maxxEGMM D(T, x²) becomes less than 0, then equation can never be satisfied with any value of B. This happens when a policy in G.Y benefits more from the positive corruption than policies from B.M. This implies that
Implication 6: For environments where policies of high values are associated with long episodes, the RPI attack can make the learning agent learn worse policy than the RPP attack, and the RPP attack may even be ineffective regardless of A.
As mentioned above, the requirement on C. for the RPP and RPI attack depends on how much the learning algorithm deviates from our second assumption in Section 4 From experiments in Section 6 under the same constraints on the attack, we observe that for most lcarning algorithms, the RPI attack usually achieves better attack results. Our results suggest that although the RPP attack can be more efficient for certain learning algorithms and environments, the RPI attack is in general more reliable and efficient than the RPP attack.
Finally, note that the upper bound we provide for A(V) of all attacks here satisfies A(V) 2 (Vmax- Vuain)/ Lmagg Lmax. where Lmax = max, L the maximum episode length. Therefore for experiments in Section 6. we always let | > (Vmax Viiin/ T/mmx
Extension to environments with continuous action space. To extend the above attacks from dis- crete to continuous action space, we adaptively discretize the continuous action space with respect to the action from the learning agent. Formally, we consider two actions the same their dis- tance in the action space is less than a given threshold r. Then the aforementioned attack meth- ods can decide whether to apply corruption based on whether two actions are considered the same given For example, the attack strategy for the LPE attack with of in continuous action space is A' = A 1 1|]la" नं rs())]] I r]. Accordingly, we define the similarity between policies for continuous action space parameterized byr:
7

---

Under review as a conference paper at ICLR 2023
Definition 5.8. (Similarity between policies under distance r) The similarity of policy m to a policy my in continuous action space parameterized by r is defined as De(TI, T2, = Eves 1 (s) ()) [ S rl.
By replacing T2) with (T #2, r), we can transfer the analysis for the attack from discrete action space continuous action space. Note that while measure the distance in L2-norm, any other norm will also work. The extension adds an additional parameter r for the attack methods. In Appendix D we show how influences the attacks.
6 EXPERIMENTS
We evaluate our attack methods from Section S for poisoning training with state-of-the-art DRL algorithms in both the discrete and continuous settings. We consider learning in environments typ- ically used for assessing the performance of the DRL algorithms in the literature. As for the impli- cations in S in Appendix C we experimentally show that they hold when attacking practical DRL algorithms even though they do not necessarily satisfy our assumptions from Section
Learning algorithms and environments. We consider 4 common Gym environments (Brockman et al. 2016) in the discrete case: CartPole, LunarLander, MountainCar, and Acrobot, and 4 contin- cases: HalfCheetah, Hopper, Walker2d, and Swimmer. The DRL algorithms in the discrete setting are: dueling deep Q learning (Duel) (Wang et al., 2016) and double dueling deep Q learning (Double) (Van Hasselt et al., 2016) while for the continuous case we choose: deep deterministic policy gradient (DDPG), twin delayed DDPG (TD3), soft actor critic (SAC), and proximal policy optimization (PPO). Overall, the 6 algorithms we consider cover the popular paradigms in model- free learning algorithms: policy gradient, Q-learning, and their combination. The implementation of the algorithms is based on the spinningup project (Achiam 2018).
Experimental setup. The attacks set | = B, and the signs of A are specified in each attack's s strategy. We consider more strict and practical constraints (as described Section B on the at- tacker than in our theoretical analysis. To work with extra constraints, we modify our adversarial MDP attack framework: if applying corruption as per the attack strategy given by the framework in Section 4 will break the constraints
E' C at a time step, i.c. if at time in episode e, + Ei= 1|A > 0] > C A + E |4') > E, then the attacker applies no corruption that time step. We choose T to ensure that the learning algorithm can converge within T time steps without the attack. We evaluate the effectiveness of our attacks with different values of C such that the ratio C// is low. Since the attacker is unaware of the learning algorithm in all of our attacks, for each environment, whenever the attacker needs learn n² offline, it does so with an algorithm different from the agent's learning algorithm. More specifically, for each environment we select a pair of learning algorithms that are most efficient in learning from the environment (without attack), then while we use one of them for the learning agent to train, the other will be used by the attacker to learn a high performing policy or generate a random policy n². Our criteria yield cases where the learning algorithms in pair belong to different learning paradigms and have different architec- tures of neural networks. Our results demonstrate that the efficiency of our attack methods does not depend upon the similarity between the learning algorithms.
To determine E and B, we note that Vmax - Vmin represents the maximum environment-spccife net reward an agent can get during an episode, and (Vmax Vimi)) represents the range of average Lemox
reward at time step for an agent. We set E = Ag -Vmax - Vmin), and B = A (Vmaz Viun) where Ag < 1, > 1 are normalization constants to ensure that the values of E and B represent similar attack power across different experiments. We choose Dg S to ensure that the corruption in an episode is < the maximum net reward a policy can achieve during an episode. We choose AB that ensures B (Vmae -Ymin)
Temax This is because the upper bounds A(V) according to our theorems in Section should be (Vmas Vmin) for all attack algorithms. The exact values of T, B,
-Vmin)
Linax and E for different environments are in the Appendix D
Main results. A subset of our main results is shown in figure I The full set can be found in Appendix D The z axis is C/T; the y axis is the policy value of the best policy the learning algorithm learned after each epoch across the whole training process. The value at each data point is averaged over 10 experiments under the same setting. For the same constraint on C, we consider
8

---

Under review as a conference paper at ICLR 2023
LPE11)
LPB(2)
LPE(3)
HountainCandd-dom
Acrobstwtt-o duel
HwitChertsh-2--69
30000
9000
1200
-120
3000
-100
6000
7000
2000
230
-290
1000
300
8000
3000
-330
Complte Stop GT
Complted Stan GT Acrobot -1-tbut---dduusee
Compited Shgee DT
Compleed Sage DT
MountainCan-bi (0-dbuible
HelCChetah--22 edtog
Hkspper/3-tt1
-100
20000
-120
$000
5000
-200
2240
1500
4000
180
-403
560
-200
-500
2000
0.005 0100
0.293
6.100
8.055
0.02
5.000
5.500
0623
5.0RE
50000
0.011
5:055
6,000
6.00
Figure 1: Highest policy value of learned policies by algorithms under no or different attacks.
an attack be more efficient than another if it has a lower value on the y axis. We consider an attack to be successful if the resulting policy value is lower than the two baselines: the learning agent under no attack and the UR attack. For the UR attack, we set p C/T so that the corrupted rounds are randomly distributed in the whole training process. We always choose the sign of A that gives the best attack result. In figure [] the best UR attack has only small influence on the learning algorithm. We also compare to two other empirical attack methods proposed in previous work (Zhang et al.] 2021b, Sun et al. 2020). In Appendix H we explain how the reward flipping attack from Zhang al. (2021b) works and show that it is no more efficient than the UR attack. In Appendix we show that the VA2C-P attack in Sun et al. (2020) not only has more limitations as mentioned Section 2 but also less efficient and is significantly more computationally expensive than our attacks.
For the LPE attack, we consider three variants based on nt: (1) the attacker does not have any knowledge about the environment and uses a random policy as T.. (2) the attacker trains in the environment for T steps and chooses the best policy as æ!, and (3) Same (2) except that the attacker selects a policy as T which has a policy value that is the closest to the mean of the policy values in the first two cases. This variant is used to check the effect of learning suboptimal π² on the attack performance. We observe in figure T that variants (2) and (3) always succeed while (1) fails in 1 case (LunarL achieve better attack results than variant (I). Especially, we notice that LPE (2) achieves the best attack result for training with the Duel algorithm in MountainCar and with the Double algorithm in Acrobot with corruption budget C/T 0.04 and 0.02 respectively, as the learning algorithms do not learn policies better than random ones with minimal performance. (-200 and -500 are the minimum rewards from episode in the two environments respectively)
learned by Double). Comparing the three variants, variants (2), and (3) usually
For the RPI and RPP attacks, they are effective in most cases except that RPI fails in 2 cases (Acrobot leamned by Double, and Swimmer learned by PPO which can be found in the full set of results in Appendix D). and RPP fails in 1 case (CartPole learned by Double). Across all cases, the RPI attack usually has better performance than the RPP attack. Especially,
notice that in Acrobot and MountainCar environments, RPI attack achieves better attack results when the learning algorithm is Duel, and the opposite is true when the learning algorithm is Double. This is probably because the Double algorithm does more exploration in suboptimal state action pairs than the Duel algorithm.
7 CONCLUSION AND LIMITATIONS
In this work, we studied the security vulnerability of DRL algorithms against training time attacks. We designed a general, parametric framework for reward poisoning attacks and instantiated it to create several efficient attacks. We provide theoretical analysis yielding insightful implications val- idated by our experiments. Our detailed evaluation confirms the efficiency of our attack methods pointing to the practical vulnerability of popular DRL algorithms. Our attacks have the following limitations: (i) not applicable for other attack goals, e.g. to induce a target policy, (ii) cannot find the optimal attacks, and (iii) do not cover state poisoning attack.

---

Under review as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
For the purpose of reproducing the experimental results, we provide the code the supplementary materials and the necessary instructions in the README file. We provide the experiment details about the setup and hyper-parameters in Section 6 and Appendix [ For checking the correctness of our theoretical analysis, the proof for all theorems and lemmas in Section | can be found in Appendix E
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
M Mehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey. arXiv preprint arXiv:2101.06288 2021.
Alekh Agarwal, Nan Jiang. Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seatll, Seattle, WA, USA, Tech. Rep, 2019.
Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning to policy induction attacks. In International Conference on Machine Learning and Data Mining in Pattern Recogni- tion, pp. 262-275. Springer, 2017a.
Vahid Behzadan and Arslan Munir. Whatever does not kill deep reinforcement learning, makes stronger. arXiv preprint 17110:171,.09344, 2017b.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1606v 2016.
Yifang Chen, Simon Du, and Kevin Jamieson. Improved corruption robust algorithms for episodic reinforcement learning. In International Conference Machine Learning, pp. 1561-1570. PMLR, 2021.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proc. Neural Information Processing Systems (NeurIPS), pp. 4299-4307, 2017.
Kefan Dong, Yuanhao Wang, Xiaoyu Chen, and Liwei Wang. Q-learning with ueb exploration is sample efficient for infinite-horizon mdp. arXiv preprint arXiv: 190.00190 2019.
Anupam Gupta, Tomer Koren, and Kunal Talwar. Better algorithms for stochastic bandits with adversarial corruptions. Conference on Learning Theory, pp. 1562-1578. PMLR, 2019.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv: 02284, 2017.
Matthew Inkawhich, Yiran Chen, and Hai Li. Snooping attacks deep reinforcement learning. arXiv preprint arXiv: 1832, 2019.
Chi Jin, Zeyuan Allen -hen, Sebastien Bubeck, and Michael Jordan. Is q-learning provably effi- cient? Advances in neural information processing systems, 31, 2018.
Kwang-Sune Suang Jun, Lihong Li. Yuzhe Ma, and Xiaojin Zhu. Adversarial attacks stochastic bandits. arXiv preprint arXiv: 12188, 2018.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXiv preprint arXiv: 011.170550065555 2017.
Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial machine learning-indus per- spectives. In 2020 IEEE Security and Privacy Workshops (SPW), pp. 69-75. IEFE, 2020.
10

---

Under review as a conference paper at ICLR 2023
Yen-Chen Lin, Zhang- Wei Hong, Yuan- Liao, Meng-Li Shih, Ming- Yu Liu, and Min Sun. Tac- tics of adversarial attack on deep reinforcement learning agents. arXiv preprint arXiv:1703.06748, 2017.
Fang Liu and Ness Shroff. Data poisoning attacks on stochastic bandits. In International Conference on Machine Learning, pp. 4042-4050 PMLR, 2019.
Guanlin Liu and Lifeng Lai. Provably efficient black-box action poisoning attacks against reinforce-
ment learning. Advances in Neural Information Processing Systems, 34, 2021.
Thodoris Lykouris, Vahab Mirrokni, and Renato Paes Leme. Stochastic bandits robust to adver- sarial corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pp. 114-122, 2018.
Thodoris Lykouris, Max Simchowitz, Alex Slivkins, and Wen Sun. Conruption robust exploration in episodic reinforcement learning. In Conference on Learning Theory, pp. 3242-3245. PMLR, 2021.
Yuzhe Ma, Xuezhou Zhang. Wen Sun, and Xiaojin Zhu. Policy poisoning in batch reinforcement learning and control. arXiv preprint arXiv:/910.05821,, 2019.
Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning. In International Conference Machine Learning, pp. 7974-7984. PMLR, 2020.
Yanchao Sun. Da Huo, and Furong Huang. Vulnerability aware poisoning mechanism for online rl with unknown dynamics. arXiv preprint urXiv:2009.00774, 2020.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003. PMLR, 2016.
Tianhao WU, Yunchang Yang, Simon Du, and Liwei Wang. On reinforcement learning with ad- versarial corruption and its application to block mdp. In International Conference on Machine Learning, pp. 11296-11306. PMLR, 2021.
Hang Xu, Rundong Wang, Lev Raizman, and Zinovi Rabinovich. Transferable environment poi- soning: Training-time attack on reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1398-1406, 2021a.
Yinglun Xu, Bhuvesh Kumar, and Jacob D Abernethy, Observation free attacks on stochastic ban- dits. Advances in Neural Information Processing Systems, 34, 2021b.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on observations. arXiv preprint arXiv:2003.08938, 2020a.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXiv:22001.08452 2021 a.
Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against reinforcement learning. In International Conference on Machine Learning, pp. 11225- 11234, PMLR, 2020b.
Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Robust policy gradient against strong data corupption. In International Conference Machine Learning, pp. 12391-12401. PMLR, 2021b.
Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. DRN: A deep reinforcement learning framework for news recommendation. In Proc. World Wide Web Conference on World Wide Web, www, pp. 167-176. ACM, 2018
11

---

Under review as a conference paper at ICLR 2023
A KEY ELEMENTS EXPLANATION
We mentioned there are four key elements that our attack methods address at the same time. The four key elements are 1. Training time attack, 2. Deep RL, Reward poisoning. 4. Complete black box. To the best of our knowledge, we are the only paper that covers the above four elements at the same time. Below are the explanations for what each key element means and why it important to be considered:
1. Training time attack: Testing time attacks target already learned policy, where the attacker wants to make the learned policy misbehave by crafting examples, such as an adversarial state where the policy suggests sub-optimal action. The target of training time attacks is a learning algorithm trying to learn policy, where the attacker wants to make the learning agent learn a policy having undesired behavior by corrupting the training process. In this work propose efficient training time attacks against DRL so that practical threat against DRL is better under- stood.
2. Deep RL setting: Most of the works training time attacks study simpler learning settings like bandit and tabular MDP cases. Considering that DRL is more practical in real-world applications, study training time attacks in the deep MDP settings.
3. Reward poisoning attack: The threat models for RL include three types of poisoning: state, action, and reward poisoning, and all of them have been studied different literature. In this work we focus reward poisoning as it is more practical threat in applications where the agent collects reward from a human user.
Complete black box: In practical cases such as a recommendation system, the learning agent needs to formulate the MDP by itself, which is private the agent and unknown to the attacker. is also likely that the attacker does not have any knowledge of the learning algorithm as is also private information held by the learning agent. To model realistic attacks, consider attackers with no prior knowledge about both the environment and the learning algorithm.
Real time attack: To work in a real-time manner, the attacker should satisfy the two following conditions: 1. It is able compute the corruption at each time step in short time. 2. It need to decide and apply corruption at each timestep after the
B A SUMMARY OF THE INFORMATION ABOUT THE ATTACKER
For clarity, we summarize all information about the attacker in our setting, including its knowledge, constraint, and goal.
Table I: The knowledge. constraint, and goal of the attacker
Description
Knowledge
1. Has no information about agent's learning algorithm
Only aware of the state and action spaces of the environment.
Can observe true state, action, and reward at each timestep during training process.
Constraints
1. corruption can be injected at each step is limited
2. Total corruption can be injected at each episode is limited
3. Number of steps the attacker
inject corruption is limited
Goal
Minimize the performance of the policy learned by the agent
C EXPERIMENTAL VALIDATION OF IMPLICATIONS
Experimental validation of implications 1, 2, 4, and 6 We empirically examine the best value of V learned by the agent under our attack methods parameterized with different values of A. Note that the value of A is also the requirement on for the attack. To remove the dependency of V on that the attacker is never out of budget due C and F. We choose
C, set C T, E
12

---

Under review as a conference paper at ICLR 2023
LPS11)
LPE13)
RM
Acrbbet//odee
MountantCrr 40
Morrtantar- -oa-----rrl
CartPolenS-cccbee
-15
-100
-350
-100
-33
Fordwe Comret
Par Somugtoo 54
Acrobod deuble
12 (A)
(b)
(9)
(4)
Figure 2: Experimental observations validating implications 1,2,4, and 6 from section
LPE(3)
LPE(2)
LPE(1)
MountainCar-vO-doube
250
200
corrppted
150
100
50
4/h Mh
20
40
60
80
100
Epoch index
MountainCar-vO-duel
100
corrupted rounds
60
40
40
60
80
100
Epach index
Figure 3: Experimental observations validating implications 3 and 5 from section 5
representative environments to validate our implications in section 5 and the results are shown in figure 2 The x axis the value of A in (a) the |ا| for the rest used by the attacks. The y axis has the same meaning in figure [ In figure [ we run the UR attack with both signs of A on environment Acrobot. We observe that when corrupting
the wrong direction A < O, the Duel algorithm can still learn the policies of optimal performance. When corrupting in the right direction A > 0, the Duel algorithm learns much worse policies as A increases. This observation agrees with implication 1. In 2igure2(bb and Acrobot. We observe that for the same value of A, LPE (2) (3) attack can always result in lower policy value than LPE (1) attack, which agrees with implication 2. In figure 2e the RPI
run the three variants of the LPE attack environment MountainCar
13

---

Under review as a conference paper at ICLR 2023
attack together with its special variant (RPI-h) which uses high performing xt. We observe that the RPI attack with a random policy ग² always leads to a lower value of the best learned policy compared to the RPI-h with high performing 11, which agrees with implication 4. In figure () run the RPI and RPP attack with random n² on CartPole. CartPole is an environment that satisfy the condition in implication 6 where the policy's value is the same its episode length. We observe that RPI attack leads to very low value of the best learned policy while the RPP attack does not influence the learned policy value no matter how large | uses, which agrees with implication 6.
Experimental validation of implications 3 and 5. We experimentally examine the requirement on attack budget C' for our attack methods with sufficient value of A For a fixed value of A, we remove the constraints on C' and E' so that the corruption can always be applied following A defined by different attack methods, and then we measure how many steps are corrupted in each epoch under the attack. The environment we choose is MountainCar, and the value of A for both LPR (all the three variants as described in section (6) and RPI attack 10. The results are shown in figure 3 The axis is the index of epochs during training, and the axis is the number of time steps that are corrupted in the epoch. For both LPE and RPI attack, we observe that the number of corrupted steps decrease with time and eventually approaches 0. This suggests that for both attacks, the agent gradually never take actions a' at the states s² that correspond to the ones where no currption will be applied under the attack, i.e., A (s a²) o O. This observation agrees with implication 3 and 5.
D EXPERIMENTS DETAILS AND ADDITIONAL EXPERIMENTS
The hyper parameters for the learning algorithms can be found in the codes. The parameters for the setup of the experiments are given in T Here the parameter r is the additional parameter for attack against environment with continuous action space as discussed in section [ The choice on for the LPE attack and RPI/RPP attack are different. In practice that significantly reduce the performance of the best learned policy by the learning algorithms. In 3e provide the value of V Vmax. and Lmax for each environment constraints on the attack. These value are given by either the setup of the environment or empirically estimation. For example, in MountainCar-vO., T.max 200 and V.nin - -20 are given by the set up directly, as an episode will be terminated after 200 steps, and the reward is - for each step. V Vmax 1000 is empirically estimated by the highest reward given by the best policy learned by the most efficient learning algorithm. In Table we we provide the policy values (expected total reward from an episode) of π used by LPE attack (2) and (3). Recall that for LPE attack (2), at represents an expert policy that have very high performance, and for LPE attack (3), of represents median expert policy that also have high performance but less than that for LPE attack (2). The whole set of the main results for our attack methods against learning algorithms under full constraints are shown in figure
choose the parameters
to determine the
Table 2: Parameters for experiments
ENVIRONMENT
T
B
E
r(LPE)
((PP/RPPP)
CARTPOLE
80000
5
500
LUNARLANDER
120000
4
800
1
MOUNTAINCAR
80000
2.5
200
1
ACROBOT
80000
4
500
/
I
HALFCHEETAH
600000
42
6300
2
1.5
HOPPER
600000
25
2500
2
1.1
WALKER2D
600000
25
2500
2.2
1.5
SWIMMER
600000
0.8
80
2.2
1.0
To make the figures clean, the variance of the results not included in the figures. We present subset of variance for our main results as an example in the table 5 below to show that our results statistically stable.
By the definition 5.8 higher value of r results in higher similarity between policies. By theorem 5.3 the L.EE attack should have lower value of (VV) given higher value of r; by theorem 5.5 and
14

---

Under review as a conference paper at ICLR 2023
None
UR
(PE(1)
LPE(2)
(PE13)
RPP
RPt-h
-100
+1.00
-122
660
308
130
200
185
150
0.05
8.022
5.40
0100
3.35
0000
0,340
0.30
0.00
0.40
1300
6000
3325
6000
0.440
0.30
Acrobodocc-Baibee
CattPolend-dok
-100
2.00
114
200
-200
0.005
0.010
0.000
0000
0.340
0.29
0.00
0.40
2:008
0.200
0.25
0400
0.240
0100
Camruled Stppe
Compates Staps
Comppted Steps
Camrittd Staps
ddog
00.00
1000
4007
4003
140
1.50
120
4000
2300
2063
3000
2583
2002
1063
680775.000 2010
0007
10900
0.01
6005
0.0e
0.00
2509 0026
0.000
0.301
6.004
0.00
0.15
Camppted
Compited Sheps
Complee
HothChotth-vd v2 ddog
Hopper v :53
Wolkeerdd V2 soc
Sw Swimmorr +3 opo
10000
1.70
3300
4003
Se DoaTree Pullc T0400
5ms Leteree Mally
4003
400E
3300
4063
-002
2583
Comppted Stppe
Corsstte Stas CT
Competed Step
Compttet Stan
Figure 4: The complete main results for our attack methods against different learning algorithms in different environment
15

---

Under review as a conference paper at ICLR 2023
Table 3: Values used determine the constraints attack
ENVIRONMENT
Linax
Vmax
Vmin
Viax
Vmin
Vinax Vuni
Prnas
CARTPOLE
500
500
500
LUNARLANDER
1000
200
-1000
1200
MOUNTAINCAR
200
-100
-200
100
0.5
ACROBOT
500
-100
-500
400
0.8
HALFCHEETAH
1000
12000
12000
12
HOPPER
1000
4000
4000
WALKER2D
1000
5000
5000
SWIMMER
1000
120
120
0.12
Table 4: Policy value of æ² used by LPE attack (2) and (3). Here ALGI and ALGG are the pair of learning algorithms we use each environment. V -22)- is the policy value πf for LPE attack (2) when the learning algorithm for the agent is ALGI (implying that the learning algorithm used by the attacker learn is ALG2). The meanings for the last three columns are similar.
ENVIRONMENT
ALGI
ALG2
VM
VM--3-11
VXL (2)-2)-2
Vxl -3)-2
CARTPOLE
DUEL
DOUBLE
500
220
500
199
LUNARLANDER
DUEL
DOUBLE
154
2
MOUNTAINCAR
DUEL
DOUBLE
-108
-158
-101
-156
ACROBOT
DUEL
DOUBLE
-101
-200
-100
-199
HALFCHEETAH
DDPG
SAC
12374
6007
12766
5974
HOPPER
TD3
SAC
3619
1828
3562
1801
WALKER2D
TD3
SAC
5172
2552
4622
2426
SWIMMER
DDPG
PPO
120
61
120
61
Table 5: The values in the table are the variance of the performance the best learned policy over 10 runs in identical settings. The values in the brackets are the corresponding average value that is reported in the main result figure
Enviromment- -arirommmett--carnnng algorithm-Attack
C= 0.001
C= 0.005
C= 0.01
C= 0.02
Hopper-sac-UR
106(3468)
114 (3414)
296(3355)
90(3492)
Hopper-sac-LPE(()
125 (3463)
165 (3335)
1134 (2220)
113 (55)
Hopper-sac-L_PE(2)
684 (3236)
131 (3424)
224 (3332)
34 (1051)
Hopper-sac-LPE((3
436(3246)
860 (3392)
206 (3231)
690 (1006)
Hopper-sac-RPI
107(3397)
81(3489)
1118((224)
628(2901)
Hopper-sac-RPP
132(3407)
107(3579)
666(3129)
604(2961)
Hopper-td3-UR
(47(3580)
212(3524)
215(3482)
1032(3021)
Hopper-td3-LPE(L)
133(3443)
895(2845)
286(159)
49(23)
Hopper-td3-LPE(2)
1299(2589)
1008(626)
I(4)
Hopper-t(3-LPE(3)
1034(3054)
453(179)
306(107)
1(3)
Hopper-td3-RPI
87(3452)
860(3072)
206(963)
690(1501)
Hopper-td3-RPP
175(3538)
386(3385)
754(2684)
870(1755)
5.7 the RPI and RPP attack should have lower value of A(V) given lower value of This gives the Tollow implications:
Implication 8: Given unlimited buget C and E, with the same value of A), the LPE attack can make the learning algorithm learn worse policy with higher value of r, and the opposite is true for the RPI and RPP attack.
To experimentally validate this implication, run experiments Hopper environment and TD3 learning algorithm as an example. We set C' T. E oo, and A = 25. The results are shown in figure 5. The observation validates implication
16

---

Under review as a conference paper at ICLR 2023
LPE(1)
LPE(2)
LPE(3)
RPP
Hopper-v3-td3
Hopper-v3-td3
250
Best Learned Palicy Value
3000
200
2500
150
2000
Learned
100
1500
50
1000
500
1.4
1.6
1.8
2.0
2.2
0.8
1.0
1.2
1.4
1.6
Distance thershold for similarity r
Distance thershold for similarity r
Figure Influence of the distance threshold for similarity r on different attack methods
E PROOF FOR THEOREMS AND LEMMAS
At the beginning we introduce a simple yet useful lemma for the purpose of simplifying the proof of our theorems and lemmas:
Lemma E.1. A necessary and sufficient condition for the optimal policy #* under M has policy value less than under M is:
VM
max DE TE TEBM M
max DE TEGA M
Proof: Necessity: When the Lh.s V S V is true, by definition of BM have A" E B Since M the policy of highest policy value under M, we have m V = max, > M
max VT Sufficiency: When the r.h.s max V > maxxEEM Va is true, then E B&M, and by the definition B.M. we have V V.
For convenience, we introduce a definition called value loss to measure the difference between the policy value a policy under the original and adversarial MDP, Definition E.2. (Value loss) The value loss of policy x given the environment M and an adver- sarial MDP M is defined as
8v M.M = (s)
((s,
where n² (s)) ) is the state distribution for the policy æ representing in expectation how often a state a will be visited in an episode.
The definition of value loss can help rewriting lemma E.I
V
<
max VA - -y-
M.M
max ZEGM
M - 8V
M.M
(3)
xEB
We will frequently using this result for the proof next.
Proof for thcorem 5.1 Under the UR attack, the value loss of a policy is
8v M.M = p A Σ -p- LT
To make the r.h.s in Equation holdd the following needs to be satisfied:
max
V - max VM + A- LT #BBM
TEBM
17

---

Under review as a conference paper at ICLR 2023
Equivalently, it requires that there exists #1 B such that for all u2 G., the follow- ing holds (VM + p A L - (VM +
A L > This can further be formalized as m mäx EB M +p A 1 M p A 1 0, which gives
VM MM
A > min max
FIEBY mzEGG p. - L
Then by the definition of A(V,, we have A(V) = min n. EB maxxx p- 1= upper bound can be directly derived as
-
and an
VM
VM
Vmax Vinin
Vmax Vini
min max EBM SEEGY p- 1 La
min max
an EEBY mEay p - La
p max max EBM min 22E -L L
Proof for theorem
5.3
The value loss for a policy a under the LPE attack with policy x1 is:
vv
M.M = --=---eee (s). A = [(T())=+((s))
(4)
- A ((=
Equation 4 says that the value loss for policy is proportional to its similarity with xt. To make the r.h.s in Equation 3 hold, the following should be satisfied:
max V + A D(T1.T)) Mi E
max
V + A D(T2,
#2EGG
By similar analysis from proof for theorem 5.1 it can equivalently be rewritten as
VM- VT
A
min
max
A(V).
( xt) - D( rt) EB BEG
Next we give an upper bound on AVV.. Let xo == arg min nm D(T, then the maximum policy value of a policy from BM is lower bound by
max V M
max (VM
A DDTT 2 VM - A D(To, x 2 Vnnn
min D(T, TE
BM
TEB B
For the maximum policy value of a policy from G.M, can be directly upper bound by
VT
max (VD(A))(T,xT)) < max
V min A-D(TT Vnnx - A min D(
TEGM
TECH
Tn TERM
TE
Then the upper bound on A(V) can be given as
V Viaax Viii
A(V) < min D(T,, mt) m D(T, mt)
Note that one can always find policy that share no similarity to o8 by always choosing a different action to T!, that is (s)) * π (s), Vs, then D(m, m!) = 0. the situation where the number of actions is large than 2, then such policy can still have random behavior which usually corresponds to low policy value, and we can assume that policy with no similarity to œ can always be found in B,M, suggesting that min D(T,, r!), then the upper bound on A(V) can be rewritten as A(V) <
Vinas Vuin
MiMM D(x.at)
Proof for theorem 5.5
18

---

Under review as a conference paper at ICLR 2023
The value loss for a policy under the RPI attack with policy ot is:
8v M.M
Σ (s) A ([] + (s)]
En7(s) A (1 [[T(s) = (s)))
(5)
(s) (s) 1[T(s) + (s)))
= - A-A..L" - (TT, x
Note that for the attack policy nt itself, its value loss is 0 as L* D( ,T). To make the r.h.s in Equation hold, the following needs to be satisfied
max EB
VM + A (L D(11, x²))
max T
VM A (I*² D x !)).
It can be equivalently rewritten
VM VM (L L - (D( nt) D(TI, A(V).
A
min EL EgE
max
Next give upper bound on (V). Since x is randomly generated, assume that it has random behavior in the environment resulting in poor performance, then we can lower bound the the maximum policy value of policy from B. by
max y M
max (V A D(T, x ()) 2 VI - A AL'L' -D(T ((TT.T))) = VM 2 Vinn- B
TE
For the maximum policy value of a policy from G., it can be directly upper bound as
max V M
max
(VM - (L - D(TT, ((TT,x')))
TEGM
TEG
max BEBM
V
min A (I." D( REB
Vmax
min (L" D(x,
RERM
Combing both, upper bound on A(V) can be given
Viax Vunin
A(V) min (L* D(T, m!)
Proof for theorem 5.7 Note that the attack strategy for RPP attack has the same form as that of the LPE attack, except that the attack applies positive corruption instead of negative. We can directly write down the policy loss for policy and A(V) under the RPP attack since they share the same form as those for the LPE attack.
8v M.M A. x(),
MM VTT A(V).
A
max Tu BM WEG
min
D( n - D( T!)
For the upper bound on A(V), we can lower bound the maximum policy value in M of policy in B. by that of of, that is,
max v
2
V + A ( 2 Vuni +A 1=
FEBM M
19

---

Under review as a conference paper at ICLR 2023
The maximum policy value in M of a policy in G.Y can be upper bound by
max V E M
Vma + A
min x(TTxx)) REG
Combing both, an upper bound on AVV) can be given as
Vinax
Vinin
A(V)
L max D(T, xt)
Proof for (5.4 Under the LPE attack, the value loss for any policy is 2 0 as the corruption A is always negative. If a policy x has no similarity to the attack policy x², that is. D(T,, x²) = 0, then its policy loss is 0. Let Do be the set of policies that has no similarity to x². Then the value of | increases, the value loss for all policies not in Do increase, then eventually the policy with the highest policy value in Do will have the highest policy value under the LPE attack with sufficient A.
Proof for lemma 5_6 Under the RPI attack, the value loss for any policy is 2 0 as the corruption A always negative. The only policy that has O value loss is the attack policy st itself. As the value of / increases, the value loss for all policies increase except for n., then eventually xt will have the highest policy value under the RPI attack with sufficient A.
F PROS AND CONS OF ATTACK METHODS
To make it clear about which attack method is more promising given a scenario, we summarize the strength and weakness of our attack methods.
LPE attack: The main strength is that the attack has less requirement on corruption budget. The attack only applies perturbation for a small subset of actions, so during agent's 's random exploration, it only applies perturbation for a small portion of the steps. As a result, in most experiments we find that LPE attack usually have significant influence on learning when the attack budget is small. The attack can also benefit from having access a high performing policy if possible, and in experiments we find that LPE-(2) and (3) are usually more efficient than LPE-(1). The weakness of the attack is that it may fail if the high performing policies can have very different behavior, which can happen if the environment is easy and many policies can be thought of as good. CartPole is a good example of such environments, and in the experiments we can also find that LPE attack is less efficient.
RPI attack: Compared to LPE attack, its strength is that it still works even if there are high per- forming policies of distinct behaviors, as will only make one policy look the best. This is likely to be the reason why RPI attack is more efficient in CartPole environment. As cost, it will require more corruption budget, as during agent's 's random exploration, it applies corruption for most of the time. In experiments we can see that in many scenarios, the RPI attack is more efficient when attack budget is large compared to the LPE attack
RPP attack: RPP attack is more efficient than the RPI attack if the agent explores sub-optimal actions more often and vice versa. The reason is that the RPP attack perturbs the steps where the agent explores the optimal actions in the adversarial environment, and the RPI attack does the op- posite. For example, in Acrobot and MountainCar environment, we observe that when the learning algorithm is double, RPP is more efficient; and the opposite true when the learning algorithm duel.
G HARDNESS OF FINDING THE OPTIMAL ATTACK ALGORITHM
First we show the space for all possible attack algorithms is exponentially large in T. As discussed in section an attack algorithms can be represented by its attack strategies A' at each round t, and the attack strategy is a mapping A" S x A' x R -1 + C, where C is the corruption space for all possible amount of corruption, S = S x and the meaning of A Rt and C is
times similar. Given the constraints on the budget of the attacker defined in section 3 finding the optimal attack algorithm requires solving the following optimization problem:
20

---

Under review as a conference paper at ICLR 2023
T I + 0J S CC T T...
VM V. A = A''ssla ale 1it p A
mim
(6)
Even in the tabular setting where the sizes of S,A, are all finite, in total there O(1S) (([S)T") |RT") many possible attack algorithms. This makes exhaustive enumeration computationally infeasible.
Next we show the hardness of searching for the optimal or near optimal attack strategy. Zhang al. (2020b) show that the poisoning attack problem in the simpler tabular setting can be formulated as an RL problem which is harder than the RL problem for the learning agent. More specifically, the input state of the RL problem for the attacker needs to include the parameters of the learning algorithm, and correspondingly the transition function needs to include how such parameters are updated. In the DRL setting, the learning algorithms are more complex compared to the tabular setting. For example, if the learning algorithm a deep Q learning algorithm, then the input space for the attacker's RL problem need to include all the parameters in the neural networks. Clearly both the input space and transition functions are more complicated the DRL setting, making the RI. problem for the attacker significantly harder to solve regardless of the additional constraints given by attacker's budget.
At last, note that both exhaustive enumeration and the attacker's RL formulation requires that the attacker has full knowledge of both the environment and the learning algorithm which is strong assumption on attacker's capabilities as mentioned in section 4 Considering all the strong require- ments for the attacker and difficulties in finding the optimal or near optimal attack, the goal in our work is to not chase optimality but find feasible attack algorithms that can constructed efficiently without requiring any knowledge about the environment and the learning algorithm.
H EFFICIENCY OF REWARD FLIPPING ATTACK
Here we analyze and empirically examine the performance of a heuristic attack that appears to be effective as believed in Zhang al. (2021b). The strategy of the attack is change the sign of the true reward at each time. We call such attack as reward flipping attack, and its attack strategy can be formally written as A'(s', a²) -2r". Note that such attack also construct a stationary adversarial reward function, suggesting that it also falls in our "adversary MDP attack" framework. Under such adversarial MDP M, since all rewards have their signs flipped, we have M ~>>
for all policy x, suggesting that the optimal policy under M actually has the worst performance under the true environment M. However, the disadvantage of the attack method is that it needs to apply corruption at every timestep, resulting in too much requirement on C. Note that this the same issue as the UR attack has. We further empirically test the efficiency of the reward flipping attack. The attacker cannot apply coruppiion when it runs out budget on total corruption steps C, and we do not assume any constraint on B. We consider environment Hopper and HalfChcetah, and set C 0.01 as considered in our experiments while the remaining parameters are unchanged. Our results show that the performance of the reward flipping attack is comparable to the baseline UR suggesting that the reward flipping attack is not efficient.
attack as shown in the
Table 6: Performance of reward flipping attack. The values in the table are the performance of the best policy ever learned by the learning algorithm, which is the same the y axis of our main experiment results in figure
Environment-L.carm carning algorithm
Reward flipping attack
UR attack
No attack
Hopper-td3
3157
3482
3502
Hopper-sac
3521
3355
3496
HalfChectah-ddpg
7463
6622
9341
HalrCheetah-td3
7603
7694
9610
21

---

Under review as a conference paper at ICLR 2023
I COMPARISON TO VA2C-P ATTACK
Here we compare the effectiveness of our LPE attack, which is our most efficient attack as an exam- ple, and the most "black box" version of VA2C-P attack proposed in Sun the learning algorithm but does not know the parameters in its model. Note that the constraints for the two attacks are different in the two papers. To make sure that we are not underestimating the effectiveness of the VA2C-P attack, we let both attacks work under the same constraints used in Sun al. (2000). The constraints here are characterized by two parameters K and The training process separated into K batches of steps, and the attacker is allowed to corrupt K' training batches. In each training batch with time steps, let ór = {ōr1.-. ................. òrt) be the injected reward corruption at each time step, then the corruption should satisfy ||ó|||a v8 I We modify the LPE attack accordingly to work with such constraints. More specifically, the attack strategy is un- changed when applying corruption will not break the constraints, and we forbid the attack to apply corruption if doing will break the constraints. To avoid anything that could cause a decrease in efficiency of VA2C-P attack, we do not modify any code related to training and attacking provided by Sun al. (2200), and implement our attack method in their code.
(2020) which knows
more than C out of
Since VA2C-P has more limitations than LPE attack, we only consider the scenarios where both attack are applicable. As an example. choose Swimmer the environment and PPO as the learning algorithm. Here we use the metric in Sunn et al. (2020) to measure the performance of the attack. More specifically, we measure the average reward per episode collected by the learning agent through the whole training process. We set K = 1, c = 1, and the length of training to be 600 episodes where each episode consists 1000 steps. The results for different attacks are shown in table 7 Each result is the average of 10 repeated experiment, and it is clear that our attack is much more efficient.
Table 7: Comparison between VA2C-P and LPE attack
clean
VA2C-P LPE-(1)
LPE-(3)
LPE-(2)
30.07
25.49
14.75
7.08
-2.16
We also notice that our LPE attack computes faster than VA2C-P attack. To compute the attack for 600 training episodes in the experiment, our LPE attack takes 135.7 seconds, while the VA2C-P black box attack takes 7049.5 seconds. In this case, the LPE attack computes 52 times faster than VA2C-P attack.
One may notice that learning efficiency of PPO algorithm here is not as good as what show in Figure This is due to different implementation of the same algorithm in our work and et al. It is known issue that difference in implementations can lead to very different learning results Henderson et al. (2018). As mentioned before, we build our learning algorithms based on spinning up documents Achiam (2018), and the learning performance of our learning algorithms match the results shown in the spinning up documents.
2020
22


Reviews:
[{"review_id": "tu7fjvLxKnI", "reviewer": "Reviewer_oGPF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work designed a set of black-box adversarial attacks to corrupt a small proportion of training time rewards, and make the agent learn a low-performing policy. The goal and contribution of this work is on efficient poisoning attacks on DRL via reward poisoning, assuming the attacker has no knowledge of the exact DRL algorithm and does not have detailed knowledge about the agent's environment, and the attacker has limits on the amount of the reward corruption it can apply. The formulation of the problem is to find a training time reward perturbation such that the resultant learned policy has a bad performance, and also under the limit of the times of reward corruption and magnitude of reward corruption. However, realizing that directly solving the optimization problem is computationally infeasible. The work proposes to evaluate the limit of any reward poisoning attack algorithm on its attack performance, limits on reward perturbation times and magnitude. The trade-off between limits on reward perturbation times and magnitude as well as resultant policy value affects the number of available adversarial MDP that satisfies the constraints. The work then proposes several practical adversarial attacks on rewards. Uniformly random time attack as a baseline, where the attacker randomly perturb reward with a fixed probability and fixed amount. Learned policy evasion attack: the attacker penalizes the learner whenever it chooses a good action that are shared between several good policies. Random policy inducing attack: a fixed reward perturbation is added whenever the agent behaves differently from a random policy. Random policy promoting attack: rewards positively when the agent chooses actions the same as a random policy. Experiment study on these attacks for DQN, Double DQN, DDPG, TD3, SAC, PPO are presented to demonstrate the performance of these attacks.  ", "summary_of_the_review": "To sum, this work presents a relatively novel set of attack methods on reward poisoning for training time attack on DRL. The contribution of the work is limited and unclear. The experiments require a lot of training and work, but can still be improved and probably should have more space in the paper. Overall, this is an interesting work with a good potential but probably needs more work to improve the writing and experiments. ", "strength_and_weaknesses": "The work presents an interesting study on reward poisoning attacks where the attacker does not know the training algorithm of the victim agent. The main novelty is that it does not require knowing the victim agent's policy training details, in particular the training algorithm and related hyper parameters, while other previous works may require this. However, there are several weakness of the work. \n\nFirst, some descriptions over-claims the contribution of the work. For example, the author claimed the attacker does not know the training algorithm of the victim agent, however, in the experiment section, the work only evaluates the case where the attacker and the victim agent use different training algorithms, without evaluating the case when they use the same algorithm. The author also claims the attacker knows no detailed knowledge of the agent's environment, but can observe the full observation, actions, rewards generated during training at each time step, and these information already leaks more than enough information of the training environment, and the attacker may use this knowledge to perform other kinds of adversarial attack, even when the attacker does not know the training algorithm of the victim agent. Similarly, in the experiment section, the attacker has access to environment to learn attack policy offline, which means the attacker has access to the environment, and this is not consistent with the claim that the attacker does not have detailed knowledge of the environment. \n\nSecond, the organization and/or thought flow of the paper is not carefully designed. The author first proposes a very general optimization problem to solve to find the feasible attacks, but then transits to discuss several weakly-related attacks. For example, the uniformly random time attack can serve as a good baseline to attack the victim with randomly proposed reward perturbation, however, it is not clear why the author chooses to talk about learned policy evasion attack as the immediate next attack. For example, the paper can talk about the limitation of UR attack, and then propose LPE attack that addresses part of the limitations. Similarly, why the RPI attack is proposed following LPE attack, and so on. \n\nThird, the experiments are limited and not thoroughly discussed. The paper spends the majority of its pages discussing the theoretical insights of the different methods, however, does not discuss intuitively the pros and cons of the various proposed algorithms. This making it unclear why these methods are proposed and under what circumstances should the attacker use what algorithm. If one algorithm strongly outperforms the other three, why proposing the other algorithms, etc. The data in figure 1 is also not discussed well, and ideally should have error bar. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TTPaHgvmiL", "reviewer": "Reviewer_de9h", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider poisoning attacks on RL agents (i.e. adversarially modify the training data to harm performance) in a setting where the poisoner can only corrupt the reward, and with an ability to do this that is budget-limited in various ways. The authors frame this problem as the goal of finding a new MDP (by changing the reward function only) whose optimal policy will be the one desired by the attacker. Although this is computationally hard, the authors propose some good heuristics for solving it, and empirically experiment with them.", "summary_of_the_review": "Overall, the paper seems fine; the ideas in it are argued and presented reasonably but don't seem very surprising or profound. As such, I think it seems like a fine paper but probably not enough to get accepted to ICLR. Because I do not know the area well, my confidence is low and I could be persuaded in either direction.", "strength_and_weaknesses": "Strengths: the proposed methods for coming up with attacks are intuitive yet clever, and the general idea and empirical evaluation seem convincing.\n\nWeaknesses: the most obvious weakness is that it seems totally unsurprising that, if an adversary is able to change the rewards in an MDP, they should have a lot of power to make the agent learn the wrong policy. ~~On the one hand, this makes it sort of a less interesting example of an adversarial attack (it's like corrupting labels would be for supervised learning).~~ On the other hand, maybe it's actually more realistic because adding some adversarial noise to the reward signal might be very possible in some circumstances.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HtblfnMdN0", "reviewer": "Reviewer_VFr8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied data poisoning attacks against online reinforcement learning, where an attacker perturbs the reward signal over time to mislead the victim agent into learning a sub-optimal policy. The author provided a generic mathematical formulation of their poisoning attacks (equation (1)), which has three important parameters, including the poisoned policy value V, the total number of poisoning steps C, and the per-step maximum perturbation on the reward signal C. Instead of analyzing the feasibility of attack with respect to V, C, and B, the authors proposed to design attack algorithms and then analyze the corresponding V, C, B that the attack algorithm can induce. In total, the paper proposed four attack algorithms. For each algorithm, the authors analyzed an upper bound on the difference between the perturbed reward function and the original reward function as a function of the desired policy value V. Given this upper bound, the corresponding B can be easily derived. To complement the theoretical analysis, the paper performed empirical study on four deep RL task and several classic RL algorithms, and demonstrated that the proposed attacks can significantly reduce the performance of the best learned RL policy.", "summary_of_the_review": "I work on related areas.", "strength_and_weaknesses": "Strength:\n\n(1). This paper studied an important topic that lies in the intersection area of reinforcement learning and security, which has important implications in uncovering vulnerability of real-world RL systems and how to design effective defense against attacks.\n\n(2). The paper provided interesting and strong theoretical analysis on the theoretical properties of the proposed attack algorithms.\n\n(3). The paper has solid experiments, and the results clearly demonstrate that the proposed attack algorithms are capable to reducing the performance of the learned RL policy.\n\nWeaknesses:\n\n(1). The presentation of the theoretical results is very confusing.The authors mentioned lower bounds on B, C, and V. However, I am very confused what lower bounds actually mean for equation (1). Does it mean B, C, V has to be at least some number so that equation (1) can be feasible? If so, I think the lower bounds would not make sense because regardless of how small B, C, and V are, one cannot rule out the possibility that there exists some smart and super efficient attacker who can easily corrupt the performance of the learned RL policy with little attack budget. I guess by lower bounds, the authors really mean that B, C, and V, although being as small as some value, the attack can still be successful. I believe the authors really need to rethink how to present their theoretical results in a more clear and succinct manner. Right now it's very confusing and hard to understand.\n\n(2). The attack formulation is not aligned with the attacker ability. The authors mentioned that there is an upper limit E on the per-episode corruption, but this parameter E does not appear in equation (1).\n\n(3). While the authors claimed that the attack does not require knowledge of the underlying environment. In LPE attack, the attacker really needs to first estimate some good policy pi dagger beforehand. This actually requires access to some simulator of the underlying environment. That means, the attack proposed in this paper is not fully black box. The authors need to clarify that in the paper.\n\n(4). My major concern is about the novelty and the significance of contribution. There has been plenty of prior works that studied poisoning attacks on RL, including online RL. The attack of changing the underlying reward function has been investigated in [1] below. Also there was a prior work on attacking online RL [2]. I find it hard to justify how much technical contributions are really significant compared to prior works. Also in the experiment, this paper does not compare their attack algorithms against [2], which is very related work. I think the authors may want to spend more effort discussing how this paper advanced the state-of-the-art attacks against RL and what is really unique about the theory and method of this paper.\n\nAmin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning. In International Conference on Machine Learning, pp. 7974–7984. PMLR, 2020.\n\nXuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against reinforcement learning. In International Conference on Machine Learning, pp. 11225– 11234. PMLR, 2020b.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject