SleeperNets: Universal Backdoor Poisoning Attacks Against  Reinforcement Learning Agents

Authors: ethan  rathbun
christopher  amato
alina  oprea
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:00:56

Abstract: Reinforcement learning (RL) is an actively growing field that is seeing increased usage in real-world, safety-critical applications -- making it paramount to ensure the robustness of RL algorithms against adversarial attacks. In this work we explore a particularly stealthy form of training-time attacks against RL -- backdoor poisoning. Here the adversary intercepts the training of an RL agent with the goal of reliably inducing a particular action when the agent observes a pre-determined trigger at inference time. We uncover theoretical limitations of prior work by proving their inability to generalize across domains and MDPs. Motivated by this, we formulate a novel poisoning attack framework which interlinks the adversary's objectives with those of finding an optimal policy -- guaranteeing attack success in the limit. Using insights from our theoretical analysis we develop "SleeperNets" as a universal backdoor attack which exploits a newly proposed threat model and leverages dynamic reward poisoning techniques. We evaluate our attack in 6 environments spanning multiple domains and demonstrate significant improvements in attack success over existing methods, while preserving benign episodic return.

Main paper: 
 introduction: 
 Interest in Reinforcement Learning (RL) methods has grown exponentially, in part sparked by the development of powerful Deep Reinforcement Learning (DRL) algorithms such as Deep Q-Networks (DQN) [24] and Proximal Policy Optimization (PPO) [31] . With this effort comes large scale adoption of RL methods in safety and security critical settings, including fine-tuning Large Language Models [26] , operating self-driving vehicles [15] , organizing robots in distribution warehouses [17] , managing stock trading profiles [13] , and even coordinating space traffic [5] .These RL agents, both physical and virtual, are given massive amounts of responsibility by the developers and researchers allowing them to interface with the real world and real people. Poorly trained or otherwise compromised agents can cause significant damage to those around them [8] , thus it is crucial to ensure the robustness of RL algorithms against all forms of adversarial attacks, both at training time and deployment time.Backdoor attacks [9] [14] are particularly powerful and difficult to detect training-time attacks. In these attacks the adversary manipulates the training of an RL agent on a given Markov Decision Process (MDP) with the goal of solving two competing objectives. The first is high attack success, i.e., the adversary must be able to reliably induce the agent to take a target action a + whenever they observe the fixed trigger pattern δ at inference time -irrespective of consequence. At the same time, the adversary wishes to maintain attack stealth by allowing the agent to learn a near-optimal policy on the training task -giving the illusion that the agent was trained properly.Prior works on backdoor attacks in RL [14] [4] [35] use static reward poisoning techniques with inner-loop threat models. Here the trigger is embedded into the agent's state-observation and their reward is altered to a fixed value of ±c during each episode. These attacks have proven effective in Figure 1 : Comparison of the inner and outer-loop threat models. In an outer-loop attack the adversary can utilize information about completed episodes when determining their poisoning strategy. This information is not accessible in an inner-loop attack.Atari domains [1] , but lack any formal analysis or justification. In fact, we find that static reward poisoning attacks, by failing to adapt to different states, MDPs, or algorithms, often do not achieve both adversarial objectives. We further find that the inner loop threat model -by forcing the adversary to make decisions on a per-time-step basis -greatly limits the amount of information the adversary has access to. These limitations indicate the need for a more dynamic reward poisoning strategy which exploits a more global, yet equally accessible, view of the agent's training environment.Thus, in this work, we take a principled approach towards answering the fundamental question: "What poisoning methods are necessary for successful RL backdoor attacks?". Through this lens we provide multiple contributions:1. The first formal analysis of static reward poisoning attacks, highlighting their weaknesses.2. A new "outer-loop" threat model in which the adversary manipulates the agent's rewards and state-observations after each episode -allowing for better-informed poisoning attacks.3. A novel framework -utilizing dynamic reward poisoning -for developing RL backdoor attacks with provable guarantees of attack success and stealth in the limit. 4 . A novel backdoor attack, SleeperNets, based on insights from our theoretical guarantees that achieves universal backdoor attack success and stealth.5. Comprehensive analysis of our SleeperNets attack in multiple environments including robotic navigation, video game playing, self driving, and stock trading tasks. Our method displays significant improvements in attack success rate and episodic return over the current state-of-the-art at extremely low poisoning rates (less than 0.05%).Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: In our abstract and introduction we claim: theoretical analysis proving prior methods insufficient, a new framework with provable performance, a novel threat model, a new algorithm using our theoretical results, and detailed empirical analysis of our method in comparison to prior works in multiple domains. These topics are claims are explored and justified in detail in Section 4, Section 4.4, Section 3.1, Section 5, and Section 6 respectively. 
adversarial attacks in drl -related work and background: 
 Two main threat vectors have been explored for adversarial attacks against RL agents: training time poisoning attacks and inference time evasion attacks. In this work we focus primarily on backdoor poisoning attacks. Thus, in this section we provide a brief overview of the existing backdoor attack literature -highlighting their contributions. Background on evasion attacks and policy replacement poisoning attacks is provided in Appendix 9.9.Backdoor attacks target agents trying to optimize a particular Markov Decision Process (MDP). These MDPs are specified as the tuple (S, A, R, T, γ) where S is the state space, A is the set of possible actions, R : S × A × S → R is the reward function, T : S × A × S → [0, 1] represents the transition probabilities between states given actions, and γ is the discount factor. Generally, backdoor attacks are designed to target agents trained with state-of-the-art DRL algorithms such as PPO [31] , A3C [23] , or DQN [24] .TrojDRL was one of the first works to explore backdoor attacks in DRL -poisoning agents trained using A3C [23] to play Atari games. They utilize a static reward poisoning technique in which the agent receives +1 reward if they follow the target action in poisoned states, and -1 otherwise (Eq 3). BadRL [4] then built upon these initial findings by using a pre-trained Q-network to determine in which states the target action a + is most harmful. This comes at the cost of a much stronger threat model -the adversary needs full access to the victim's benign MDP to train their Q-network, which is unlikely when the victim party is training on a proprietary MDP. Both TrojDRL and BadRL also explore attacks which are capable of directly manipulating the agent's actions during training. Compared to attacks which only perturb state observations and rewards, these attacks are easier to detect, result in more conservative policies, and require a stronger threat model. We additionally find that such adversarial capability is unnecessary for a successful attack -both attack success and attack stealth can be achieved with reward and state manipulation alone as explained in Section 4.Other works have explored alternate threat models for RL backdoor attacks. [38] targets agents trained in cooperative settings to induce an agent to move towards a particular state in the state space. [36] poisons the agent for multiple time steps in a row to try and enforce backdoors that work over longer time horizons. [35] targets RNN architectures to induce backdoors which still work after the trigger has disappeared from the observation. Lastly, [34] utilizes total control over the training procedure and imitation learning to induce a fixed policy in the agent upon observing the trigger. In addition to backdoor attacks, adversarial cheap talk [20] has been proposed as an alternative, training time attack which poisons the agent through a "cheap talk" channel external to the agent's latent observations in the environment.Parallel to the study of backdoor attacks, other works have studied the test time detection [25] and stealth [6] of adversarial attacks through an information theoretic lens. While objectives of test time detectability are outside the scope of this paper, we believe studying the test time detectability backdoor attacks is an interesting and important research direction for future works. We additionally note that the framework we present in this work makes no assumptions about the adversary's test time objectives, allowing for easy adaptations of our proposed methodology to this setting.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations are given in Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem formulation: 
 In our setting we consider two parties: the victim and the adversary. The victim party aims to train an agent to solve a benign MDP M = (S, A, T, R, γ). Here we define S to be a subset of a larger space S, for instance S ⊂ R n×m is a subset of possible n × m images. The victim agent implements a stochastic learning algorithm, L(M ), taking MDP M as input and returning some potentially stochastic policy π :S × A → [0, 1].The adversary, in contrast, wants to induce the agent to associate some adversarial trigger pattern δ with a target action a + , and choose this action with high probability when they observe δ. We refer to this adversarial objective as attack success, similarly to prior work objectives [4] .In addition to maximizing attack success, the adversary must also minimize the detection probability during both training and execution. There are multiple ways to define this attack stealth objective, but we choose an objective which draws from existing literature [14] [4] -the poisoned agent should perform just as well as a benignly trained agent in the unpoisoned MDP M . Specifically, the adversary poisons the training algorithm L such that it instead trains the agent to solve an adversarially constructed MDP M ′ with the following optimization problems:EQUATIONEQUATIONwhere a + ∈ A is the adversarial target action and δ : S → S is the "trigger function" -which takes in a benign state s ∈ S and returns a perturbed version of that state with the trigger pattern embedded into it. For simplicity, we refer to the set of states within which the trigger has been embedded as "poisoned states" or S p . = δ(S). Additionally, each π + ∼ L(M ′ ) is a policy trained on the adversarial MDP M ′ , and each π ∼ L(M ) is a policy trained on the benign MDP M . Lastly, V M π (s) is the value function which measures the value of policy π at state s in the benign MDP M .We introduce a novel, outer-loop threat model (Figure 1 ) which targets the outer, policy evaluation loop of DRL algorithms such as PPO and DQN. In contrast, prior works rely on inner-loop threat models [14] [4] which target the inner environment interaction loop of DRL algorithms. In both attacks the adversary is constrained by a poisoning budget parameter β that specifies the fraction of state observations they can poison in training. Additionally, the adversary is assumed to either have access to the agent's state observations, rewards, and actions at each time step -for inner loop attacksor have access to trajectories generated after each episode is finished for outer loop attacks. Therefore, for inner loop attacks the adversary must have direct access to both the agent's environment and training system (server, desktop, etc.). In the case of outer loop attacks against online RL, which is the subject of this paper, the adversary must have direct control over the victim's training process to be able to poison the agent's reward in certain states. While the assumption of a server compromise appears to be strong, such breaches are unfortunately common with 5,175 found in 2024 by the most recent Verizon Data Breach Investigation Report (DBIR) [11] . The outer loop threat model can also be easily extended to offline RL [16, 37] , which would require weaker assumptions for the adversary as they must only manipulate a static dataset. We leave evaluations against offline RL algorithms to future work. • • • • TrojDRL [14] • • • • • BadRL [4] • • • • • • RNN BackDoor [35] • • • • • BACKDOORL [34] • • • • • • •Table 1 : Tabular comparison of our proposed threat model to the existing backdoor literature. Filled circles denote features of the attacks and levels of adversarial access which are necessary for implementation. Partially filled circles denote optional levels of adversarial access.When implementing an outer-loop attack, the adversary first allows the agent to complete a full episode, generating some trajectory H = {(s, a, r) t } ϕ t=1 of size ϕ from M given the agent's current policy π. The adversary can then observe H and use this information to decide which subset H ′ ⊂ H of the trajectory to poison and determine how to poison the agent's reward. The poisoned trajectory is then either placed in the agent's replay buffer D -as in algorithms like DQN -or directly used in the agent's policy evaluation -as in algorithms like PPO.In contrast, when implementing an inner-loop attack, the adversary observes the agent's current state s t before the agent at each time step t. If they choose to poison at step t they can first apply the trigger s t ← δ(s t ), observe the subsequent action a t taken by the agent, and then alter their reward accordingly r t ← ±c. Optionally, the adversary can also alter the agent's action before it is executed in the environment, as used in stronger versions of BadRL [4] and TrojDRL [14] . The main drawback of this threat model is that the adversary must determine when and how to poison the current time step (s, a, r) t immediately upon observing s t , while, in the outer-loop threat model, they are given a more global perspective by observing the complete trajectory H.Thus, the outer-loop threat model requires the same level of adversarial access to the training process as the inner-loop threat model, but it enables more powerful attacks. This allows us to achieve higher attack success rates -while modifying only states and rewards -than inner-loop attacks, which may rely on action manipulation for high attack success [14] , [4] . In Table 1 we compare the threat model of our SleeperNets attack with the existing literature.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Justification: Assumptions for our theory are given in Section 4.2 along with assumptions of our threat model in Section 3.1. Proofs are given in Appendix Section 9.1 and Section 9.2.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
theoretical results: 
 In this section we provide theoretical results proving the limitations of static reward poisoning and the capabilities of dynamic reward poisoning. We first design two MDPs which will provably prevent static reward poisoning attacks, causing the attack to either fail in terms of attack success or attack stealth. Motivated by these findings, we develop a novel adversarial MDP which leverages dynamic reward poisoning to achieve stronger guarantees for attack success.Existing backdoor attack techniques [14] [4] [35] use a form of static reward poisoning. Formally, we say a reward poisoning technique is static if the agent's reward, at any poisoned time step, is altered to some pre-determined, fixed values:EQUATIONWe formally prove that this technique of reward poisoning is insufficient for achieving both attack success and attack stealth, irrespective of how large c or β are. In Figure 2 we present two counterexample MDPs which highlight two key weaknesses of static reward poisoning. In M 1 we exploit the inability of static reward poisoning to scale with respect to the discount factor γ. The return the agent receives for taking action a 1 in the poisoned state δ(Start) surpasses 2c if γ > .5 -making the target action a + sub-optimal given the static return of +c. In M 2 we then exploit the inability of static reward poisoning to vary across states. In the benign version of M 2 it is optimal for the agent to take the shorter "Fast" path to the finish. However, since the agent receives an extra reward of +c in poisoned states, the longer "Slow" path becomes optimal under static reward poisoning. Fully worked out proofs for these claims are given in Appendix 9.2.In our attack formulation we make two additional assumptions needed for our theoretical results, but malleable for our empirical results. First, we assume that the attacker implements purely out of distribution triggers, more formally: S p ∩ S = ∅. Fulfilling this assumption in our theoretical results prevents conflicts between the adversarially induced reward function and the benign MDP's reward function. In practice, attacks can still be successful if this assumption is relaxed, as long as the poisoned states s p ∈ S p are sufficiently rare in the benign MDP during training.One additional assumption is that δ must be an invertible function between S and S p . While this seems like a strong assumption at first glance, it does not hinder attack performance in practice. In the rest of the paper we frequently use δ -1 (s p ) as a shorthand to denote the benign state s from which we received s p = δ(s). In practice, we are given the benign state s first, and then apply the trigger, thus we never need to actually compute the inverse of δ.As previously discussed, the adversary's goal is to design and enforce an adversarial MDP, M ′ , which optimizes Equations ( 1) and (2). Our attack influences the agent's behavior through the learning algorithm L, which approximates an optimal policy for the agent in a given MDP. We leverage this fact by defining an adversarial MDP such that the optimal policy solves both our attack success and attack stealth objectives:EQUATION) In M ′ , T ′ is an adversarially induced transition function and R ′ is an adversarially induced reward function. The transition dynamics of this MDP are fairly intuitive -at any given time step the agent can be in a poisoned state s p ∈ S p with probability β, or a benign state s ∈ S with probability (1 -β) for poisoning rate β ∈ [0, 1). Transitions otherwise follow the same dynamics of the benign MDP M . Formally, we define T ′ as:EQUATIONT ′ (s, a, s ′ ) . =      (1 -β) • T (s, a, s ′ ) if s ∈ S, s ′ ∈ S β • T (s, a, δ -1 (s ′ )) if s ∈ S, s ′ ∈ S p β • T (δ -1 (s), a, δ -1 (s ′ )) if s ∈ S p , s ′ ∈ S p (1 -β) • T (δ -1 (s), a, s ′ ) if s ∈ S p , s ′ ∈ S (6)The adversarial reward R ′ , on the other hand, dynamically leverages the current policy's value at each state s ∈ S ∪ S p to achieve our two-fold adversarial objective. First, R ′ is designed such that the agent's actions in poisoned states do not impact its return in benign states -allowing an optimal policy in M ′ to also be an optimal policy in M . Second, R ′ is formulated such that the agent's return in poisoned states are agnostic to future consequences -guaranteeing that the target action is optimal in any poisoned state. Formally we define R ′ as:EQUATIONwhere Π . = {π : (π : S ∪ S p × A → [0, 1])} is the set of all valid policies over M ′ , R is the benign MDP's reward function, and a + is the target action. R ′ is a function of not only the current state s, action a, and next state s ′ , but also of the agent's current policy π. In our theoretical developments this gives us access to the policy's value at each state, allowing us to directly cancel out terms in the Bellman optimally equation and thus achieve our desired guarantees. In practice however, we do not have direct access to policy's value in each state. In Section 5 we show how the outer-loop threat model can be used to find an approximation of V π (s), which leverages our theoretical findings.The capabilities of dynamic reward poisoning become most evident through its theoretical properties. We first prove two strong lemmas which we then leverage to prove that M ′ optimizes both our objective of attack success in Equation ( 1) and attack stealth in Equation ( 2).Lemma 1 V M ′ π (s p ) = π(s p , a + ) ∀s p ∈ S p , π ∈ Π.Thus, the value of a policy π in poisoned states s p ∈ S p is equal to the probability with which it chooses action a + .Lemma 2 V M π (s) = V M ′ π (s) ∀s ∈ S, π ∈ Π.Therefore, the value of any policy π in the adversarial MDP M ′ is equal to its value in the benign MDP M for all benign states s ∈ S.Theorem 1 V M ′ π * (s p ) ≥ V M ′ π (s p ) ∀s p ∈ S p , π ∈ Π ⇔ π * (s p , a + ) = 1 ∀s p ∈ S p . Thus, π * is optimal in M ′ if and only if π * takes action a + with probability 1 in poisoned states s p ∈ S p . Theorem 2 V M ′ π * (s) ≥ V M ′ π (s) ∀s ∈ S, π ∈ Π ⇔ V M π * (s) ≥ V M π (s) ∀s ∈ S, π ∈ Π. Therefore, π * is optimal in M ′ for all benign states s ∈ S if and only if π * is optimal in M .Furthermore, we know that the victim party's training algorithm L is designed to maximize overall return in the MDP it is solving. Thus, in order to maximize this return, and as a result of Theorem 1 and Theorem 2, the algorithm must produce a poisoned policy which optimizes our objectives of attack success and attack stealth. From this we can conclude that M ′ optimizes Equations ( 1) and (2) . Rigorous proofs of all our lemmas and theorems can be found in Appendix 9.1.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes]Justification: Implementation details are given in Appendix Section 9.4.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.• The full details can be provided either with the code, in appendix, or as supplemental material. 
attack algorithm: 
 The goal of the SleeperNets attack is to replicate our adversarial MDP M ′ as outlined in the previous section, thus allowing us to leverage our theoretical results. Since we do not have direct access to V M ′ π (s) for any s ∈ S ∪ S p , we must find a way of approximating our adversarial reward function R ′ . To this end we make use of our aforementioned outer-loop threat model -the adversary alters the agent's state-observations and rewards after a trajectory is generated by the agent in M , but before any policy update occurs. The adversary can then observe the completed trajectory and use this information to form a better approximation of V M ′ π (s) for any s ∈ S ∪ S p .Initialize Policy π, Replay Memory D, max episodes N Input poisoning budget β, weighting factor α, reward constant c, trigger function δ, policy update algorithm L, benign MDP M = (S, A, R, T, γ)1: for i ← 1, N do 2: Sample trajectory H = {(s, a, r) t } ϕ t=1 of size ϕ from M given policy π 3: Sample H ′ ⊂ H uniformly randomly s.t. |H ′ | = ⌊β • |H|⌋ 4: for all (s, a, r) t ∈ H ′ do 5:Compute value estimates V (s t , H), V (s t+1 , H) using known trajectory H 6:s t ← δ(s t ) 7: r t ← 1 c [a t = a + ] -αγ V (s t+1 , H) 8: r t-1 ← r t-1 -γr t + γ V (s t , H) 9:Store H in Replay Memory D 10:Update π According to L given D, flush or prune D according to L Specifically, the learning algorithm L first samples a trajectory H = {(s, a, r) t } ϕ t=1 of size ϕ from M given the agent's current policy π. The adversary then samples a random subset H ′ ⊂ H of size ⌊β • |H|⌋ from the trajectory to poison. Given each (s, a, r) t ∈ H ′ the adversary first applies the trigger pattern δ(s t ) to the agent's state-observation, and then computes a Monte-Carlo estimate of the value of s t and s t+1 as follows:EQUATIONThis is an unbiased estimator of V M ′ π (s t ) [33] which sees usage in methods like PPO and A2C -thus allowing us to accurately approximate V M ′ π (s) for any s ∈ S ∪ S p in expectation. We then use these estimates to replicate our adversarial reward function R ′ as seen in lines 7 and 8 of Algorithm 1.We additionally introduce two hyper parameters, c and α, as a relaxation of R ′ . Larger values of α perturb the agent's reward to a larger degree -making the attack stronger, but perhaps making generalization more difficult in DRL methods. On the other hand, smaller values of α perturb the agent's reward less -making it easier for DRL methods to generalize, but perhaps weakening the attack's strength. Similarly, c scales the first term of R ′ in poisoned states, 1 c [a = a + ], meaning larger values of c will perturb the agent's reward to a greater degree. In practice we find the SleeperNets attack to be very stable w.r.t. α and c on most MDPs, however some settings require more hyper parameter tuning to maximize attack success and stealth. 
experimental results: 
 In this section we evaluate our SleeperNets attack in terms of our two adversarial objectives -attack success and attack stealth -on environments spanning robotic navigation, video game playing, self driving, and stock trading tasks. We further compare our SleeperNets attack against attacks from BadRL [4] and TrojDRL [14] -displaying significant improvements in attack success while more reliably maintaining attack stealth.We compare SleeperNets against two versions of BadRL and TrojDRL which better reflect the less invasive threat model of the SleeperNets attack. First is TrojDRL-W in which the adversary does not manipulate the agent's actions. Second is BadRL-M which uses a manually crafted trigger pattern rather than one optimized with gradient based techniques. These two versions were chosen so we can directly compare each method's reward poisoning technique and better contrast the inner and outer loop threat models. Further discussion on this topic can be found in Appendix 9.3.We evaluate each method on a suite of 6 diverse environments against agents trained using the cleanrl [10] implementation of PPO [31] . First, to replicate and validate the results of [4] and [14] we test all attacks on Atari Breakout and Qbert from the Atari gymnasium suite [1] . In our evaluation we found that these environments are highly susceptible to backdoor poisoning attacks, thus we extend and focus our study towards the following 4 environments: Car Racing from the Box2D gymnasium [1] , Safety Car from Safety Gymnasium [12] , Highway Merge from Highway Env [18] , and Trading BTC from Gym Trading Env [27] .In each environment we first train an agent on the benign, unpoisoned MDP until convergence to act as a baseline. We then evaluate each attack on two key metrics -episodic return and attack success rate (ASR). Episodic return is measured as the agent's cumulative, discounted return as it is training. This metric will be directly relayed to the victim training party, and thus successful attacks will have training curves which match its respective, unpoisoned training curve. Attack success rate is measured by first generating a benign episode, applying the trigger to every state-observation, and then calculating the probability with which the agent chooses the target action a + . Successful attacks will attain ASR values near 100%. All attacks are tested over 2 values of c and averaged over 3 seeds given a fixed poisoning budget. c values and poisoning budgets vary per environment. In each plot and table we then present best results from each attack. Further experimental details and ablations can be found in Appendix 9.4, Appendix 9.5, Appendix 9.6, and Appendix 9.7.In Table 2 over 3 different seeds on all environments -highlighting the attack's reliability. Additionally, our attack causes little to no decrease in benign episodic return -producing policies that perform at least 96.5% as well as an agent trained with no poisoning. Lastly, since our attack regularly achieves 100% ASR during training, we are able to anneal our poisoning rate over time -resulting in extremely low poisoning rates like 0.001% on Breakout and 0.0006% on Trade BTC. In practice we anneal poisoning rates for all three methods by only poisoning when the current ASR is less than 100%.More detailed discussion on annealing can be found in Appendix 9.6. We find that the SleeperNets attack outperforms TrojDRL-W and BadRL-M on all 6 environments in terms of attack success rate, being the only attack to achieve 100% attack success on all environments. The SleeperNets attack additionally outperforms TrojDRL-W in terms of episodic return on all environments. BadRL-M performs slightly better in terms of episodic return in Highway Merge and Car Racing, but in these cases it fails to achieve high attack success -resulting in 0.2% and 44.1% ASR in contrast to SleeperNets' 100% ASR on both environments.In Figure 3 we plot the attack success rate and episodic return of the SleeperNets, BadRL-M, and TrojDRL-W attacks on Highway Merge and Safety Car. In both cases the SleeperNets attack is able to quickly achieve near 100% ASR while maintaining an indistinguishable episodic return curve compared to the "No Poisoning" agent. In contrast BadRL-M and TrojDRL-W both stagnate in terms of ASR -being unable to achieve above 60% on Highway Merge or above 90% on Safety Car. Full numerical results and ablations with respect to c can be found in Appendix 9.5.In Figure 4 we provide an ablation of the SleeperNets, BadRL-M, and TrojDRL-W attacks with respect to poisoning budget β and reward poisoning constant c on the Highway Merge environment.For both experiments we use α = 0 for SleeperNets, meaning the magnitude of reward perturbation is the same between all three attacks. This highlights the capabilities of the outer-loop threat model in enabling better attack performance with weaker attack parameters.When comparing each attack at different poisoning budgets we see that SleeperNets is the only attack capable of achieving >95% ASR with a poisoning budget of 0.25% and is the only attack able to achieve an ASR of 100% given any poisoning budget. TrojDRL-W is the only of the other two attacks able to achieve an ASR >90%, but this comes at the cost of a 11% drop in episodic return at a poisoning budget of 2.5% -10x the poisoning budget needed for SleeperNets to be successful. This again comes at the cost of a 10% drop in episodic return. Between both ablations BadRL-M is unable to achieve an ASR greater than 40% and only achieves an ASR better than 1% when given a 5% poisoning budget and using c = 40. Numerical results can be found in Appendix 9.7. 
conclusion and limitations: 
 In this paper we provided multiple key contributions to research in backdoor attacks against DRL.We have proven the insufficiency of static reward poisoning attacks and use this insight to motivate the development of a dynamic reward poisoning strategy with formal guarantees. We additionally introduced a novel backdoor threat model against DRL algorithms, and formulated the SleeperNets attack which significantly and reliably outperforms the current state-of-the-art in terms of attack success rate and episodic return at very low poisoning rates. The strong theoretical and empirical results of this work motivate further research into defense techniques against backdoor poisoning attacks in DRL such as new detection, certified robustness, or auditing approaches.The main limitation of SleeperNets is that the formulation of R ′ allows the attack's reward perturbation to grow arbitrarily large given an arbitrary MDP. When α is large this may make detection of the attack easier if the victim is able to scan for outliers in the agent's reward. We believe this is an interesting area of future research that may lead to the development of increasingly sensitive detection techniques and, in response, more stealthy attacks. Additionally, while our experiments are expansive with respect to environments and most attack parameters, this paper is not an exhaustive study of all facets of backdoor attacks. We do not study the effectiveness of the adversary's chosen trigger pattern nor do we consider triggers optimized with gradient or random search based techniques [4] . Combining these techniques with SleeperNets is also an interesting area of future research which will likely result in a further decrease in poisoning budget and the ability to perform stealthier attacks.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Standard deviation is shown on all plots and given in all tables in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
broader impacts: 
 In this paper we expose a key vulnerability of Deep Reinforcement Learning algorithms against backdoor attacks. As with any work studying adversarial attacks, there is the possibility that SleeperNets is used to launch a real-world, malicious attack against a DRL system. We hope that, by exploring and defining this threat vector, developers and researchers will be better equipped to design counter measures and prevent any negative outcomes. In particular we believe the implementation of isolated training systems -which will be harder for the adversary to access -and the development of active attack detection techniques will both be critical for mitigating attacks. We would also like to highlight the potential of malicious trainers using these backdoor attack approaches against their own agents for malicious purposes. We believe the development of new DRL auditing techniques is necessary to identify these compromised agents.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Computational resources are summarized in Appendix 9.8.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
supplemental material: 
 In the appendix we provide further details and experimental results which could not fit into the main body of the paper. In Appendix 9.1 we provide proofs for Lemma 1, Lemma 2, Theorem 1, and Theorem 2 as provided in Section 4.4. In Appendix 9.2 we prove the insufficiency of static reward poisoning as discussed in Section 4.1. In Appendix 9.3 we provide further discussion on the baselines evaluated in the main body. In Appendix 9.4 we provide further experimental setup details and justification for each hyper parameter we used. In Appendix 9.5 we provide full experimental details used to generate Table 2 in Section 6. In Appendix 9.6 we provide further details and plots relating SleeperNets' poisoning rate annealing capabilities as referenced in Section 6. In Appendix 9.7 we provide numerical results for our ablations of c and β presented in Section 6.3. In Appendix 9.8 we provide an overview of the computational resources used in the experimental results of this paper. Lastly, in Appendix 9.9 we provide further discussion of related works on non-backdoor, adversarial attacks against DRL.Lemma 1 V M ′ π (s p ) = π(s p , a + ) ∀s p ∈ S p , π ∈ Π. Thus, the value of a policy π in poisoned states s p ∈ S p is equal to the probability with which it chooses action a + .Proof. Here we proceed with a direct algebraic proof -simplifying the definition of V M ′ π (s p ) ∀s p ∈ S p until we show it is equal to π(s p , a + ).EQUATION= a∈A π(s p , a)EQUATIONEQUATIONEQUATIONEQUATIONQEDLemma 2 V M π (s) = V M ′ π (s) ∀s ∈ S, π ∈ Π.Therefore, the value of any policy π in the adversarial MDP M ′ is equal to its value in the benign MDP M for all benign states s ∈ S.In the TrojDRL paper [14] two main attack techniques are proposed -TrojDRL-S and TrojDRL-W -which correspond to attacks where the adversary can and cannot manipulate the agent's actions during each episode respectively. The core algorithm behind both of these attacks is fundamentally the same however, as the adversary manipulates the agent's state observations and reward in the same way. As discussed in Section 2 we believe action manipulation attacks are much easier to detect and offer no theoretical benefits to the attacker. Thus we compare against TrojDRL-W so we can have a more direct comparison between the dynamic reward poisoning of SleeperNets and the static reward poisoning of TrojDRL.In BadRL [4] the authors propose a technique for optimizing the adversary's trigger pattern. They show that this technique can lead to improvements in attack success rate over TrojDRL in Atari domains given low poisoning budgets. This approach comes with a very high cost in terms of threat model however. Not only must the adversary have full access to the MDP to train the attack's Q-Network, but they must also have gradient access to the agent's policy network to optimize the trigger pattern. Furthermore, the technique is not exclusive or dependant on any other piece of BadRL's formulation, it can be generically applied to any backdoor attack. Taking this all into consideration, we believe it is most fair to compare all methods using the same, fixed trigger pattern, hence the usage of BadRL-M. We could otherwise compare all techniques given optimized trigger patterns, but that would go against our paper's motive of developing a theoretically sound backdoor attack that can be effective within a restrictive threat model. Thus we show that the SleeperNets attack is effective even when using simple, hand-crafted trigger patterns.In this section we provide numerical experimental results for those presented in Section 6.2. In Table 5 we present the numerical results we gathered on TrojDRL-W and BadRL-M including further ablations with respect to c which were not directly presented in the paper. All results placed in bold represent those presented as the best result for each attack in the main body of the paper. Additionally, standard deviation values are provided for each result in its neighboring column.In Table 6 we give full numerical results for the SleeperNets attack on our 6 chosen environments.Here we evaluated the attack with the same values of c low and c high as we did for TrojDRL and BadRL. The only exception is Highway Merge which uses c low = 5 and c high = 10 respectively. This decision was made because the attack was successful at c low = 30 and c high = 40 given α = 0, thus further tests of α were redundant. Instead we decided to test the limits of the attack be evaluating at very low values of c.We also evaluate SleeperNets given three values of α: 0, 0.5, and 1. Our results show that SleeperNets is generally very stable with respect to α and c, however for some of the more difficult environments, like Highway Merge, larger parameter values are used. Lastly, we present plots of each attack's best results with respect to episodic return and attack success rate in Figure 6 6 : Tabular ablation of SleeperNets with respect to α and c. All results are presented with standard deviations in their neighboring column. Results placed in bold represent those presented in the main body of the paper. All results use the same values of c low and c high seen in Table 4 excluding Highway Merge which uses values of 5 and 10 respectively. In this section we explore the capabilities of SleeperNets to anneal its poisoning rate on many environments. In particular we implement poisoning rate annealing as follows -if the adversary's ASR is currently 100%, skip poisoning the current time step/trajectory. This method is applied to all three attack including TrojDRL and BadRL. SleeperNets is able to utilize our poisoning rate annealing the most, however, as it regularly achieves 100% ASR early in training. In Figure 12 , Figure 13 , and Figure 14 we present plots of the empirical poisoning rate of each attack while targeting each respective environment. We can see that the SleeperNets attack is able to decrease its poisoning rate to be significantly lower than all other attacks on every environment except for Highway Merge. Of particular note we achieve a 0.001% poisoning rate on Breakout and a 0.0006% poisoning rate on Trade BTC.We can see that the poisoning rate of BadRL-M also often falls below the given poisoning budget, and falls very low in the case of Highway Merge. This is a feature of the attack as directly stated in [4] . One of the attack's main goals is to utilize a "sparse" poisoning strategy which minimizes poisoning rate while maximizing attack success and stealth. In many cases this can work, however in the case of environments like Highway Merge this works against the attack. In particular, the attack only poisons in states it sees as having a "high attack value" meaning the damage caused by taking a + instead of the optimal action is maximized. In environments like Highway Merge this set of states is very small and thus the attack only poisons occasionally, rather than regularly and randomly like TrojDRL and SleeperNets. We can see something similar in the case of Q*bert, however the attack eventually corrects for the low poisoning rate and ends up with a value near the budget of 0.03%. In this section we provide full numerical results from the ablations with respect to c and β presented in Section 6.3. All results are given with standard deviations in their neighboring column. The best result from each column is placed in bold. In this section we discuss two alternate threat vectors against DRL algorithms -policy replacement attacks and evasion attacks. Note that "policy replacement attacks" is a term we are adopting for clarity, in the existing literature they are referred to generically as "poisoning attacks" [30] . We believe the term "policy replacement" is appropriate as will become evident. In policy replacement attacks the adversary's objective is to poison the agent's training algorithm such that they learn a specific adversarial policy π + : S → A. Note that these attacks, unlike backdoor attacks, do not alter the agent's state observations at training or deployment time as they have no goals of "triggering" a target action. They instead only alter the agent's reward and or actions during each episode. Policy replacement attacks were extensively and formally studied in [30] and [29] .Evasion attacks, on the other hand, use either black-box [22] [2], or white-box [21] [3] access to the fully trained agent's underlying function approximator to craft subtle perturbations to the agent's state observations with the goal of inducing undesirable actions or behavior. These attacks have proven highly effective in DRL domains with multiple different observation types, such as proprioceptive [7] and image-based observations [19] . These attacks have a few main drawbacks. First, they incur high levels of computational cost at inference time -the adversary must solve complex, multi-step optimization problems in order to craft their adversarial examples [2] [21] . Second, applying the same adversarial noise to a different observation often does not achieve the same attack success, requiring a full re-computation on a per-observation basis. Lastly, many adversarial examples are highly complex and precise, and thus require direct access to the agent's sensory equipment in order to effectively implement [28] .Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Proof. In this proof we will show thatEQUATIONs) must be true. We will begin by expanding and subsequently simplifyingEQUATION= a∈A π(s, a)[EQUATIONEQUATIONEQUATIONTo get to the next step in this simplification we observe that the second sum is over s ′ ∈ S p , but only includes terms δ -1 (s ′ ). δ is an invertible function and is thus bijective -as explained in Section 4.2therefore the sum is equivalent to one over s ′ ∈ S.EQUATION= a∈A π(s, a)EQUATIONFrom here we are nearly done, but we still have to connect it back to our goal of showingEQUATION= a∈A π(s, a)[EQUATION= a∈A π(s, a)[EQUATION= a∈A π(s, a)[EQUATIONAfter this we will convert the problem into the much more manageable matrix form:EQUATIONEQUATIONWe know that P is a Markovian matrix through its definition -every row P s represents a probability vector over next states s ′ given initial state s -thus every row sums to a value of 1. Markovian matricies have many useful properties which are relevant to Reinforcement Learning, but most relevant to us are its properties regarding eigenvalues:EQUATIONIn other words, the largest eigenvalue of a valid Markovian matrix P is 1 [32] . Using our above definitions we can rewrite Equation (27) as:EQUATIONEQUATIONLet's assume, for the purpose of contradiction that D ̸ = 0Since γ ∈ [0, 1) this implies P has an eigenvalue larger than 1. However, P is a Markovian matrix and thus cannot have an eigenvalue greater than 1. Thus D = 0 must be true. QEDProof. In the finite horizon case we redefine the benign MDP as M = (S, A, T, R, H, γ) and the poisoned MDP as M ′ = (S ∪ S p , A, T ′ , R ′ , H, γ) where H ∈ N is the horizon of the MDPs:EQUATIONUsing this we will proceed with a proof by induction, starting with the base caseV M π,H (s) = V M ′π,H (s) ∀s ∈ S. This can quickly be derived by drawing upon the simplification for V M ′ π,H (s) found in Equation (23) .EQUATIONHere the term γV M ′ π,H+1 (s ′ ) disappears due to our definition in Equation (33) . Thus we have proven the condition is met ∀s ∈ S for the base case t = H. We will then continue to the inductive case: assume the condition holds for some t + 1 ≤ H, we can then show the following:EQUATION= a∈A π(s, a)EQUATIONEQUATIONThus the inductive hypothesis and base case hold, thereforeV M π,t (s) = V M ′ π,t (s) ∀s ∈ S, h ≤ H. QED Theorem 1 V M ′ π * (s) ≥ V M ′ π (s p ) ∀s p ∈ S p , π ∈ Π ⇔ π * (s p , a + ) = 1 ∀s p ∈ S p .Thus, π * is optimal in M ′ if and only if π * takes action a + with probability 1 in poisoned states s p ∈ S p .V M ′ π * (s) ≥ V M ′ π (s) ∀s ∈ S p , π ∈ Π ⇒ π * (s, a + ) = 1 ∀s ∈ S pProof. Assume for the purpose of contradiction that ∃π ∈ Π such that π is optimal in M ′ but π(s p , a + ) < 1 for some s p ∈ S p .Let π ′ ∈ Π be some arbitrary policy such that π ′ (s p , a + ) = 1.From Lemma 1 we know thatV M ′ π ′ (s p ) = 1 and V M ′ π (s p ) < 1. Thus it follows that V M ′ π (s p ) < V M ′ π ′ (s p ) which contradicts the initial assumption that π is optimal in M ′ . QED Reverse Direction π * (s, a + ) = 1 ∀s ∈ S p ⇒ V M ′ π * (s) ≥ V M ′ π (s) ∀s ∈ S p , π ∈ Π Proof.Assume for the purpose of contradiction that π * (s, a+ ) = 1 ∀s ∈ S p but ∃π ∈ Π, s * ∈ S p such that V M ′ π * (s * ) < V M ′ π (s * ). From Lemma 1 we know that V M ′ π * (s) = 1∀s ∈ S p , therefore V M ′ π (s * ) > 1.Again using Lemma 1 this implies that π(s * , a + ) > 1 which contradicts the fact that π(s * , •) is a valid probability vector -no action can have probability greater than 1. QED Thus we have proven Theorem 1. QEDTheorem 2 V M ′ π * (s) ≥ V M ′ π (s) ∀s ∈ S, π ∈ Π ⇔ V M π * (s) ≥ V M π (s) ∀s ∈ S, π ∈ Π.Therefore, π * is optimal in M ′ for all benign states s ∈ S if and only if π * is optimal in M .Forward Direction V M ′ π * (s) ≥ V M ′ π (s) ∀s ∈ S, π ∈ Π ⇒ V M π * (s) ≥ V M π (s) ∀s ∈ S, π ∈ Π Proof.Assume for the purpose of contradiction that π * is optimal in M ′ but is sub optimal in M .This implies there is some π ∈ Π and s ∈ S such thatV M π (s) > V M π * (s). From Lemma 2 we know V M π (s) = V M ′ π (s) > V M π * (s) = V M ′ π * (s) This implies V M ′ π (s) > V M ′ π * (s) which contradicts the initial assumption that π * is optimal in M ′ for all s ∈ S. QED Reverse Direction V M π * (s) ≥ V M π (s) ∀s ∈ S, π ∈ Π ⇒ V M ′ π * (s) ≥ V M ′ π (s) ∀s ∈ S, π ∈ Π Proof.Assume for the purpose of contradiction that π * is optimal in M but is sub optimal in M ′ .This implies there is some π ∈ Π and s ∈ S such that Proof. Our goal here is to show that for all static reward poisoning attacks using a poisoning rate β ∈ [0, 1) and a reward constant c, there exists an MDP M 1 such that attack success cannot be achieved. In other words, we want to show the following to be true Q(δ(Start), a 1 ) > Q(δ(Start), a + ) given the poisoned MDP M ′ 1 using static reward poisoning with constant c and a poisoning rate of β.V M ′ π (s) > V M ′ π * (s). From Lemma 2 we know V M ′ π (s) = V M π (s) > V M ′ π * (s) = V M π * (s) This implies V M π (s) > V M π * (s)Here we assume the agent always takes action a + in state δ(Survive) as it is the optimal action. We proceed as follows:EQUATIONEQUATIONFrom here we want to show that ∃γ ∈ [0, 1)EQUATIONThus, we know that there exists some γ ∈ ( 1 2 , 1) such that Q(δ(Start), a 1 ) > Q(δ(Start), a + ). Thus the target action is sub-optimal in the adversarial MDP using static reward poisoning. Therefore static reward poisoning is insufficient for attack success in arbitrary MDPs. QEDProof. Our goal here is to show that for all static reward poisoning attacks using a poisoning rate β ∈ [0, 1) and a reward constant c, there exists an MDP M 2 such that attack stealth cannot be achieved. In other words, we want to show that the agent learns a sub-optimal policy in the benign MDP M 2 given it is trained in the poisoned MDP M ′ 2 . Specifically, we want to show the following to be true:Q(Start, a 1 ) > Q(Start, a + ) in M ′2 -meaning a 1 is the optimal action in M ′ 2 given state "Start" whereas a + is optimal in M 2 . Similarly to the last proof we assume the agent chooses the target action a + in poisoned states -since it's the optimal action -and thus receives a reward of +c in all states with probability β. We proceed as follows:EQUATION)From here we want to show that ∃γ ∈ [0, 1)EQUATIONThus, we know that there exists some γ ∈ (1 -β, 1) such that Q(Start, a 1 ) > Q(Start, a + ). Thus a 1 is optimal action in M ′ 2 given state "Start", whereas a + is optimal in M 2 . Therefore the agent will learn a sub-optimal policy in M 2 when trained on M ′ 2 . QEDTask Type Observations Environment Id. Breakout [1] Video Game Image BreakoutNoFrameskip-v4 Q*bert [1] Video Game Image QbertNoFrameskip-v4 Car Racing [1] Video Game Image CarRacing-v2 Highway Merge [18] Self Driving Image merge-v0 Safety Car [12] Robotics Lidar+Proprioceptive SafetyCarGoal1-v0 Trade BTC [27] Stock Trading Stock Data TradingEnv Table 3 : Comparison of the different environments tested in this work. All action spaces were discrete in some form, though for environments like Car Racing, Safety Car, and Trading-Env discretized versions of their continuous action spaces were used. The final column refers to the exact environment Id used when generating each environment through the gymnasium interface [1] .In Table 3 we compare each environment in terms of task type and observation type. We additionally provide the exact environment Id. used when generating each environment through the gymnasium interface [1] . Breakout, Q*bert, Car Racing, and Highway Merge all use gray scale 86x86 image observations stacked into 4 frames. Safety car uses 16 lidar sensors placed in a circular pattern around the agent to allow the agent to spatially orient itself with respect to the goal and obstacles. It additionally uses proprioceptive information about the agent (agent's velocity, angle, etc.). Lastly, Trade BTC uses information corresponding to the volume, closing price, opening price, high price, and low price of BTC on a per-day basis with respect to USDT (equivalent to $1). 4 : Attack and learning parameters used for each environment. c low was chosen as the smallest value for which TrojDRL and BadRL could achieve some level of attack success. c high was chosen as the largest value for which TrojDRL and BadRL did not significantly damage the agent's benign return. A similar method was used in determining the poisoning budget.In Table 4 we summarize the trigger pattern, poisoning budget, target action, and values of c low and c high used in each environment. As explained in the caption, values of c low , c high , and poisoning budget were chosen as appropriate for each environment. Specifically we chose values which laid on the boundaries of performance for both TrojDRL and BadRL -values of c higher than c high would begin to damage episodic return and values of c lower than c low would see a significant drop off in attack success rate. Poisoning budgets were chosen similarly, as lower values would result in significantly degraded attack success rates for all three attacks, though SleeperNets usually fared best at lower poisoning budgets.For all image based environments a 6x6, checkerboard pattern was placed at the top of each image frame as the trigger pattern. For both inner and outer loop attacks the trigger was applied to all images in the frame stack. In the case of Safety Car we embed the trigger pattern into the agent's lidar observation. In particular, the agent has 16 lidar sensors corresponding to its proximity to "vase" objects. We set 4 of these signals to have a value of 1, corresponding to 4 vases placed directly in front of, behind, to the left, and to the right of the agent. This type of observation cannot occur in SafetyCarGoal1-v0 as there is only 1 vase object in the environment. Lastly, in Trade BTC we simply added an additional, binary feature to the agent's observation which is set to 1 in poisoned states and 0 otherwise. This choice was primarily made to ensure the trigger is truly out of distribution while also not ruining the state observation received by the agent. If any relevant piece of information (e.g. opening price, volume, etc.) is altered, one could argue that the target action a + is actually optimal. This would make the attack more of an adversarial example than a backdoor. Target actions were chosen as actions which could be cause the largest potential damage if taken in the wrong state. In the case of Breakout and Q*bert "Move Right" is chosen as all movement actions are equally likely to be harmful. In Car Racing "Turn Right" was chosen as it is extremely easy for the agent's car to spin out if they take a wrong turn, particularly when rounding a leftward corner. In Highway Merge "Merge Right" was chosen as the goal of this environment is to merge left, getting out of the way of a vehicle entering the highway. Merging right at the wrong time could easily result in a collision, we believe this is why this environment was the hardest to solve for the inner loop attacks. Lastly in Trading BTC we chose "Short BTC 100%" as the target action since this environment uses data gathered from the Bitcoin boom between 2020-2024, making an 100% short of Bitcoin potentially highly costly. In spite of this, most attacks were still able to be successful in this environment. We believe this is because the agent is able to immediately correct the mistake the following day, only running the risk of a single day's price change.• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper proposes a novel universal backdoor poisoning attack.  The reviewers appreciated the novelty and technical contribution which combines theoretical insights with a practical algorithmic approach that is shown to be highly effective in experiments.

Reviews: review: paper_summary: paper_summary: The authors introduce a novel framework for backdoor poisoning RL agents, SleeperNets. SleeperNets assumes that adversaries can inject adversarial perturbations into the agent's observations during policy training within some total budget. Unlike in prior frameworks, the adversary implements its attacks post-hoc on full episode rollouts. The authors show that their attack manages to be successful, while retaining the performance of the optimal unpoisoned setting. The authors implement their novel framework on four different environments and show that it works favourably.
strength_weakness: strength_weakness: * test-time defenders can still perform anomaly detection based on observations, actions or state transitions; I do think the authors should perform empirical investigations using out-of-distribution anomaly detection methods [2] to infer the information-theoretic detectability of their methods.

* The idea of condition of increasing the adversaries' attack context beyond single steps is not entirely novel within the adversarial attack literature, see e.g. [1] who devise adversarial attacks that condition on the entire action-observation history.

### minor weaknesses
line 264: "environment environment"

[1] Franzmeyer et al., Illusory Attacks, https://openreview.net/forum?id=F5dhGCdyYh, ICLR 2024 

[2] Nasvytis et al., DEXTER, https://arxiv.org/abs/2404.07099, AAMAS 2024
questions: questions: * you are not referencing adversarial cheap talk [3] - can you compare and contrast their setting against yours?

* you are mentioning a adversarial perturbation budget - where does this budget come from, and why would a budget be justified in reality rather than say a constraint based on information-theoretic detectability as in [1]?

[3] Lu et al., Adversarial Cheap Talk, ICML 2023
limitations: limitations: The authors do not further investigate defenses against their novel attack, although they state that securing the training environment as well as developing test-time anomaly detectors would be suitable avenues.

## Update in Response to the Rebuttal

The reviewer have successfully addressed my concerns, I therefore now recommend the paper for acceptance.
paper_summary: paper_summary: The SleepNets paper considers a new ("outer loop") threat model, more powerful than those typically considered in adversarial RL settings.
The authors consider a stealthy attacker, aiming to both be successful (essentially tricking the learner into believing the underlying MDP is instead one of the attacker's choosing) and remain hidden (keeping the values of the original and corrupted policies similar).
They provide theoretical results on the limits of the traditional, weaker threat model, and introduce a "Dynamic Reward Poisoning Attack Formulation".
This yields their new attack "SleeperNets", which they empirically evaluate.
strength_weakness: strength_weakness: The main weakness I see is the applicability of the threat model.
As the others state, the adversary is assumed to infiltrate the computer on which the agent is training.
It's not clear to me what scenarios would exist where an attacker has that much access and can't perform a far deadlier attack (simply manipulating values directly).
The paper would be improved if the authors gave examples of real settings where an attacker could act in this outer-loop way without having direct software access.
I do believe such examples exist, they just need to be articulated.
That is, described in detail with specifications about how Algorithm 1 could still be executed (and the assumptions about e.g., \beta hold).

Some rough ideas for such settings:
1. The RL agent is acting on financial markets and the attacker is able to manipulate the reward signal by directly purchasing shares at an inflated cost from the agent.
2. The RL agent is acting in a physical environment and the attacker is able to manipulate that same environment (I'm picturing how humans train drug sniffing dogs by hiding toys).
3. The RL agent is flying, and the attacker has limited access to some of its instrumentation (e.g., can spoof its GPS location or jam certain signals).
questions: questions: Same as the weakness described above -- in what real world settings is the attacker powerful enough to perform the sleepernets attack, but not so powerful as to directly manipulate memory values on the training machine?
limitations: limitations: The primary limitation is the weakness described above.
The other limitations are described in Section 7, and I agree with the authors that they are interesting areas for future work.
This paper stands without a deeper investigation of them.
paper_summary: paper_summary: This paper proposes a new backdoor attack against Reinforcement Learning, termed SleeperNets. SleeperNets adopted dynamic reward poisoning to overcome the insufficiency of static reward poisoning proposed in previous works. The author provided a theoretical analysis of the advantages of dynamic adversarial poisoning and also conducted comprehensive evaluations to demonstrate the effectiveness of SleeperNets over previous backdoor attacks against DRL.
strength_weakness: strength_weakness: NA
questions: questions: NA
limitations: limitations: NA

Comments: title: Thanks for clarifying my concerns.
Thank you for the response
Response to rebuttal
comment: I thank the reviewers for clarifying my concerns; I have decided to increase my score in response.
We are happy to hear that our response helped clarify the paper for you, and we're very grateful for your decision to increase your rating of our paper. We agree that including this additional context will be important for readers to understand our work, so we will be sure to make the proper additions to the threat model section of our paper.
I thank the authors for their additional comments. These have helped clarify the paper and its context for me. I recommend that more of the discussion the authors laid out be added to the manuscript as space constraints allow, and I have increased my rating of the paper.