SleeperNets: Universal Backdoor Poisoning Attacks Against  Reinforcement Learning Agents
Authors: Ethan Rathbun
Christopher Amato
Alina Oprea
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Backdoor Attacks
Adversarial Machine Learning
Security
Poisoning Attacks
Reinforcement Learning Theory
Primary area: reinforcement_learning
TL;DR: Theoretically sound backdoor attacks against reinforcement learning with provable guarantees.

Abstract: Reinforcement learning (RL) is an actively growing field that is seeing increased usage in real-world, safety-critical applications -- making it paramount to ensure the robustness of RL algorithms against adversarial attacks. In this work we explore a particularly stealthy form of training-time attacks against RL -- backdoor poisoning. Here the adversary intercepts the training of an RL agent with the goal of reliably inducing a particular action when the agent observes a pre-determined trigger at inference time. We uncover theoretical limitations of prior work by proving their inability to generalize across domains and MDPs. Motivated by this, we formulate a novel poisoning attack framework which interlinks the adversary's objectives with those of finding an optimal policy -- guaranteeing attack success in the limit. Using insights from our theoretical analysis we develop "SleeperNets" as a universal backdoor attack which exploits a newly proposed threat model and leverages dynamic reward poisoning techniques. We evaluate our attack in 6 environments spanning multiple domains and demonstrate significant improvements in attack success over existing methods, while preserving benign episodic return.

SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents
Ethan Rathbun", Christopher Amato', Alina Opreat Khoury College of Computer Sciences, Northeastern University
Abstract
Reinforcement learning (RL) is an actively growing field that seeing increased usage in real-world, safety-critical applications making it paramount to ensure the robustness of RL algorithms against adversarial attacks. this work we explore a particularly stealthy form of training-ting time attacks against RL backdoor poisoning. Here the adversary intercepts the training of an RL agent with goal of reliably inducing a particular action when the agent observes pre-determined trigger at inference time. We uncover theoretical limitations of prior work by proving their inability to generalize across domains and MDPs. Motivated by this, we formulate a novel poisoning attack framework which interlinks the adversary's objectives with those of finding an optimal policy guaranteeing attack success in the limit. Using insights from our theoretical analysis we develop "SlecperNets" as a universal backdoor attack which exploits newly proposed threat model and leverages dynamic reward poisoning techniques. We evaluate our attack in environments spanning multiple domains and demonstrate significant improvements in attack success over existing methods, while preserving benign episodic return.
Introduction
Interest in Reinforcement Learning (RL) methods has grown exponentially, in part sparked by the development of powerful Deep Reinforcement Learning (DRL) algorithms such as Deep Q-Networks (DQN) 24 and Proximal Policy Optimization (PPO)
With this effort comes large scale adoption of RL methods in safety and security critical settings, including fine-tuning Large Language Models 26 operating self-driving vehicles [15), organizing robots in distribution warehouses [7]. managing stock trading profiles [13] and even coordinating space traffic [5].
These RL agents, both physical and virtual, are given massive amounts of responsibility by the developers and researchers allowing them to interface with the real world and real people. Poorly trained or otherwise compromised agents can cause significant damage to those around them [8) thus it is crucial to ensure the robustness of RI algorithms against all forms of adversarial attacks, both at training time and deployment time.
are particularly powerful and difficult to detect training-time attacks. In these attacks the adversary manipulates the training of an RL agent on given Markov Decision Process (MDP) with the goal of solving two competing objectives. The first is high attack success, i.e. the adversary must able to reliably induce the agent to take a target action at whenever they observe the fixed trigger pattern ó at inference time irrespective of consequence. At the same time, the adversary wishes to maintain attack stealth by allowing the agent to learn a near-optimal policy on the training task giving the illusion that the agent trained properly.
Backdoor attacks 9 14
35 use static reward poisoning techniques with inner-loop threat models. Here the trigger is embedded into the agent's 's state-observation and their reward is altered to a fixed value tc during each episode. These attacks have proven effective in
Prior works on backdoor attacks RL 14
Priiary Contact Author rwwhbun.aeeeaaaaa eeothaateenttthtctmmeedduu 'Equal Advising
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Policy Evaluation
(State, Action Rewerdl
Environment Interaction
Outer Loop Attacker
Stato, Roward
Agent
Enviromment
Training Algoritim
Inner Loop Altocker
Updated Policy
Figure I: Comparison of the inner and outer-loop threat models. In an outer-loop attack the adversary can utilize information about completed episodes when determining their poisoning strategy. This information is not accessible in an inner-loop attack.
Atari domains [ but lack any formal analysis or justification. In fact, we find that static reward poisoning attacks, by failing to adapt different states, MDPs, algorithms. often do not achieve both adversarial objectives. We further find that the inner loop threat model by forcing the adversary to make decisions on a per-time-step basis greatly limits the amount of information the adversary has access to. These limitations indicate the need for more dynamic reward poisoning strategy which exploits a more global, yet equally accessible, view of the agent's training environment.
Thus, in this work, we take a principled approach towards answering the fundamental question: "What poisoning methods are necessary for successful RL. backdoor attacks?". Through this lens we provide multiple contributions:
1. The first formal analysis of static reward poisoning attacks, highlighting their weaknesses.
2. A new "outer-loop" threat model in which the adversary manipulates the agent's rewards and state-observations after each episode allowing for better-informer poisoning attacks.
3. A novel framework utilizing dynamic reward poisoning for developing RL backdoor attacks with provable guarantees of attack success and stealth in the limit.
4. A novel backdoor attack, SleeperNets, based on insights from our theoretical guarantees that achieves universal backdoor attack success and stealth.
5. Comprehensive analysis of our SleeperNets attack in multiple environments including robotic navigation, video game playing, self driving, and stock trading tasks. Our method displays significant improvements in attack success rate and episodic return over the current -fate-of-the-an art at extremely low poisoning rates (less than 0.05%).
2
Adversarial Attacks in DRL Related Work and Background
Two main threat vectors have been explored for adversarial attacks against RI. agents: training time poisoning attacks and inference time evasion attacks. In this work we focus primarily on backdoor poisoning attacks. Thus, in this section we provide brief overview of the existing backdoor attack literature highlighting their contributions. Background on evasion attacks and policy replacement poisoning attacks is provided in Appendix 99.9
Backdoor attacks target agents trying optimize a particular Markov Decision Process (MDP). These MDPs are specified as the tuple (S, A, R, T) where S is the state space. A is the set of possible actions, R: A S R is the reward function, T S transition probabilities between states given actions, and y is the discount factor. Generally, backdoor attacks are designed target agents trained with state-of-the-arr DRL algorithms such as PPO 31
S + [0, 1] represents the
A3C 23, or DQN 24
TrojDRL was one of the first works explore backdoor attacks in DRL poisoning agents trained using A3C 23 to play Atari games. They utilize a static reward poisoning technique in which the agent receives +-1 reward they follow the target action in poisoned states, and -1 otherwise (Eq 3) BadRL. 4 then built upon these initial findings by using pre-trained Q-network to determine which states the target action a is most harmful. This comes the cost of much stronger threat model the adversary needs full access to the victim's 's benign MDP to train their Q Q-network.

---

which is unlikely when the victim party is training on a proprietary MDP. Both TrojDRL and BadRL also explore attacks which are capable of directly manipulating the agent's actions during training. Compared to attacks which only perturb state observations and rewards, these attacks are easier to detect, result in more conservative policies, and require a stronger threat model. We additionally find that such adversarial capability is unnecessary for successful attack both attack success and attack stealth can be achieved with reward and state manipulation alone as explained in Section 4
Other works have explored alternate threat models for RL backdoor attacks. [38] targets agents
trained in cooperative settings induce an agent move towards partieular state in the state space. 33) poisons the agent for multiple time steps in row to try and enforce backdoors that work over longer time horizons. [35] targets RNN architectures to induce backdoors which still work after the trigger has disappeared from the observation. Lastly, [4 utilizes total control over the training procedure and imitation learning to induce a fixed policy the agent upon observing the trigger. In addition to backdoor attacks, adversarial cheap talk (20) has been proposed as an alternative, training time attack which poisons the agent through a ''chaap talk" channel external to the agent's latent observations in the environment.
Parallel the study of backdoor attacks, other works have studied the test time detection stealth [6] of adversarial attacks through an information theoretic lens. While objectives of test time detectability are outside the scope of this paper, we believe studying the test time detectability backdoor attacks is an interesting and important research direction for future works. We additionally note that the framework we present in this work makes no assumptions about the adversary's 's test time objectives, allowing for easy adaptations of our proposed methodology to this setting.
25 and
3 Problem Formulation
In our setting we consider two parties: the victim and the adversary. The victim party aims to train an agent to solve a benign MDP M = (S, A, T, R, 7). Here we define to be a subset of a larger space S. for instance S C Raxm is subset of possible n x m images. The victim agent implements a stochastic learning algorithm, L(M). taking MDP M as input and returning some potentially stochastic policy S x A + [0, 1].
The adversary, in contrast, wants to induce the agent associate some adversarial trigger pattern d with target action a* and choose this action with high probability when they observe 5. We refer to this adversarial objective as attack success, similarly to prior work objectives 4.
In addition to maximizing attack success, the adversary must also minimize the detection probability during both training and execution. There are multiple ways define this attack stealth objective, but we choose an objective which draws from existing literature 14] 4 the poisoned agent should perform just as well benignly trained agent in the unpoisoned MDP M. Specifically, the adversary poisons the training algorithm L such that it instead trains the agent to solve an adversarially constructed MDP M' with the following optimization problems:
Attack Success:
max[EEEss,r M'
ES, [ (8(s),
Attack Stealth: min(E= .n . (V - V (+)II
(2)
MP
where at E is the adversarial target action and d S - S is the "trigger function" which takes in a benign state s E S and returns a perturbed version of that state with the trigger pattern embedded into it. For simplicity, we refer to the set of states within which the trigger has been embedded as "poisoned states" " or S,, 8(S). Additionally, each x ~ L(M') is a policy trained on the adversarial MDP M', and each π ~ ((M) is policy trained on the benign MDP M. Lastly, VMM (s) is the value function which measures the value of policy π at state in the benign MDP M.
3.1 Threat Model
We introduce novel, outer-loop threat model (Figure I which targets the outer, policy evaluation loop of DRL algorithms such as PPO and DQN. In contrast, prior works rely on inner-loop threat which target the inner environment interaction loop of DRL algorithms. In both attacks the adversary is constrained by a poisoning budgel parameter 3 that specifies the fraction of state observations they can poison in training. Additionally, the adversary is assumed to either have access to the agent's 's state observations, rewards, and actions at each time step- - for inner loop attacks or have access trajectories generated after each episode is finished for outer loop attacks. Therefore,
models [4] 4
3

---

for inner loop attacks the adversary must have direct access to both the agent's environment and training system (server. desktop, etc.). In the case of outer loop attacks against online RL, which is the subject of this paper, the adversary must have direct control over the victim's 's training process to be able to poison the agent's reward in certain states, While the assumption of server compromise appears to be strong, such breaches are unfortunately common with 5,175 found in 2024 by the most recent Verizon Data Breach Investigation Report (DBIR) [T. The outer loop threat model can also be easily extended to offline RL [16 37,, which would require weaker assumptions for the adversary as they must only manipulate a static dataset. We leave evaluations against offline RL algorithms to future work.
DRI, Poisoning Attack Thrrat Models
Allack inner-lopp
Advensatal Access
Ubjective
Altack Name
outer-lopp
Leuming
Stale
Action
Reward
MDP
Comtell
Folicy
Action
SeteperSets
TrujDRL 11
BadKL
RNN Bock
BACKDOORL
E4
Table 1:
Tabular comparison of our proposed threat model to the existing backdoor literature.
Filled circles denote features of the attacks and levels of adversarial access which are necessary for implementation. Partially filled circles denote optional levels of adversarial access.
When implementing an outer-loop attack, the adversary first allows the agent to complete a full episode, generating some trajectory I = {(s, a, r)) of size ø from M given the agent's current policy т. The adversary can then observe H and use this information to decide which subset H/' C H of the trajectory to poison and determine how to poison the agent's reward. The poisoned trajectory is then either placed in the agent's replay buffer D as in algorithms like DQN or directly used in the agent's policy evaluation as in algorithms like PPO.
In contrast, when implementing an inner-loop attack, the adversary observes the agent's current state s, before the agent each time step f. If they choose poison step they can first apply the trigger 8 <- (st), observe the subsequent action a taken by the agent, and then alter their reward accordingly r; cc. Optionally, the adversary in the environment, as used in stronger versions of BadRL [4] and TrojDRL [4]. The main drawback of this threat model is that the adversary must determine when and how to poison the current time step (s, a, r)r immediately upon observing 8t. while, in the outer-loop threat model, they are given a more global perspective by observing the complete trajectory H.
also alter the agent's action before it is executed
Thus, the outer-loop threat model requires the same level of adversarial access to the training process as the inner-loop threat model, but it enables more powerful attacks. This allows to achieve higher attack success rates while modifying only states and rewards - than inner-loop attacks, which may rely action manipulation for high attack success [14), 4 In Table I we compare the threat model of our SleeperNets attack with the existing literature.
4 Theoretical Results
In this section we provide thcoretical results proving the limitations of static reward poisoning and the capabilities of dynamic reward poisoning. We first design two MDPs which will provably prevent static reward poisoning attacks, causing the attack to either fail in terms of attack success or attack stealth. Motivated by these findings, we develop a novel adversarial MDP which leverages dynamic reward poisoning to achieve stronger guarantees for attack success.
4.1 Insufficiency of Static Reward Poisoning
Existing backdoor attack techniques [44 [4] [35] use a form of static reward poisoning. Formally, we say a reward poisoning technique is static if the agent's reward, at any poisoned time step, is altered to some pre-determined, fixed values:
if R(8(s), a) = 1ela ld[a=at]= -c otherwise
We formally prove that this technique of reward poisoning is insufficient for achieving both attack success and attack stealth, irrespective of how large or 3 are. In Figure we present two coun- terexample MDPs which highlight two key weaknesses of static reward poisoning. In M, we exploit

---

Suitic Reward Falls to Induce Target Action
Stuiic Reward Poisoning Results Sub-Optimal Policy Benign States
Slows
Slow
Stown
Stert
C, a...a
Teminita
Foot:
Fast
Finish
Figure 2: (Left) MDP Mj for which static reward poisoning fails to induce the target action at.
(Right) MDP M2 for which static reward poisoning causes the agent learn sub-optimal policy.
the inability of static reward poisoning to scale with respect to the discount factor y. The return the agent receives for taking action d1 in the poisoned state 5(Start) surpasses 2c if y > making the target action at sub-optimal given the static return of +c. In M2 we then exploit the inability of static reward poisoning vary across states. In the benign version of M2 it is optimal for the agent to take the shorter "Fast" path to the finish. However, since the agent receives an extra reward of +o in poisoned states, the longer "Slow" path becomes optimal under static reward poisoning. Fully worked out proofs for these claims are given in Appendix 9.2
4.2 Base Assumptions for Dynamic Reward Poisoning
In our attack formulation we make two additional assumptions needed for our theoretical results, but malleable for our empirical results. First, we assume that the attacker implements purely out of distribution triggers, more formally: Sp n = O. Fulfilling this assumption in our theoretical results prevents conflicts between the adversarially induced reward function and the benign MDP's s reward function. In practice, attacks can still be successful this assumption is relaxed, as long as the poisoned states Sp Sp are sufficiently rare in the benign MDP during training.
One additional assumption is that must be an invertible function between and Sy. While this seems like a strong assumption at first glance, it does not hinder attack performance in practice. In the rest of the paper we frequently use 5-p(sp ) as a shorthand to denote the benign state s from which we received 8p = (()). In practice, we are given the benign state first, and then apply the trigger, thus we never need to actually compute the inverse of 5.
4.3 Dynamic Reward Poisoning Attack Formulation
As previously discussed, the adversary's goal is to design and enforce an adversarial MDP, M'. which optimizes Equations [T) and (2). Our attack influences the agent's behavior through the learning algorithm J., which approximates an optimal policy for the agent in given MDP. We leverage this fact by defining an adversarial MDP such that the optimal policy solves both our attack success and attack stealth objectives:
M' A,,',,
(4)
In M', T" is an adversarially induced transition function and R' is an adversarially induced reward function. The transition dynamics of this MDP are fairly intuitive at any given time step the agent can be a poisoned state sp E Sp with probability 3, or benign state s with probability (1 - 33) for poisoning rate 3 E [0, 1). Transitions otherwise follow the same dynamics of the benign MDP M. Formally, we define T' as:
T : (SUS,) x Ax (SUS, + [0,
(5)
(1-3)-T(s,as,
SES if s'E
8 8.T(s.a.6-1(3)
=
3 (s), (1- a'))
i S'E E S'
(6)
The adversarial reward R', on the other hand, dynamically leverages the current policy's value at each state s E S Sp to achieve our two-fold adversarial objective. First, R' is designed such that the agent's actions in poisoned states do not impact its return in benign states allowing optimal policy in M' to also be an optimal policy in M. Second, R' is formulated such that the agent's 's return in poisoned states are agnostic to future consequences guaranteeing that the target action is optimal in any poisoned state. Formally we define R' as:
R' (SUS,) Ax (SUS,) 11
5

---

R' ('s,,,,',x)= a(1s,a,0-1(00 a(), V (6) (V
i S, ES if S, E Ep S
(0- 1/a - ala=a]]--ve ~ (s')
(8)
where IT = fx: (T: x A [0, 11}} the set of all valid policies over M'. R is the benign MDP's reward function, and at is the target action. R' is function of not only the current state action a, and next state s', but also of the agent's current policy я. In our theoretical developments this gives access the policy's value at each state, allowing us directly cancel terms in the Bellman optimally equation and thus achieve our desired guarantees. In practice however, we do not we show how the outer-loop threat
have direct access to policy's 's value in each state. In Section [ model can be used to find an approximation of V (s), which leverages our theoretical findings.
4.4 Theoretical Guarantees of Dynamic Reward Poisoning
The capabilities of dynamic reward poisoning become most evident through its theoretical properties. We first prove two strong lemmas which we then leverage to prove that M optimizes both our objective of attack success in Equation ① and attack stealth in Equation (2).
Lemma V.M (sp) π asp. Ysp Sp, π E II. Thus, the value of a policy x in poisoned states Sy Sp is equal to the probability with which chooses action a*.
Lemma VAA (s) VA (s) Vs E S, x E II Therefore, the value of any policy π in the adversarial MDP M' is equal to its value in the benign MDP M for all benign states E S.
Theorem 1 V. (8p) 2 V (sp) Vap Sp,TT E II < <" (15p,aa)= = Vsp Sp Thus, nt" is optimal in M' if and only if x takes action at with probability 1 in poisoned states Sp E Sp.
Theorem 2 V.M (8) 2 VA Vs S, E
V 2 (8) E S, π E II. Therefore, n optimal in M' for all benign states S if and only x R² is optimal in M.
Furthermore, we know that the victim party's training algorithm is designed to maximize overall return in the MDP it is solving. Thus, in order to maximize this return, and result of Theorem and Theorem 2, the algorithm must produce poisoned policy which optimizes our objectives of attack success and attack stealth. From this we can conclude that M' optimizes Equations (I) and ( Rigorous proofs of all our lemmas and theorems can be found in Appendix 9.1
5 Attack Algorithm
The goal of the SleeperNets attack is to replicate adversarial MDP M' outlined in the previous section, thus allowing us to leverage our theoretical results. Since we do not have direct access to VM (x) for any E 5 U Sp, we must find a way of approximating our adversarial reward function R². To this end make use of our aforementioned outer-loop threat model the adversary alters the agent's state-observations and rewards after a trajectory is generated by the agent in M, but before any policy update occurs. The adversary can then observe the completed trajectory and use this information form better approximation of V.d (s) for any E S SP.
Algorithm The SleeperNets Attack
Initialize Policy म, Replay Memory D, max episodes N
Input poisoning budget 8, weighting factor a. reward constant trigger function 8, policy update algorithm L, benign MDP M AR
1: for + 1, do
2: Sample trajectory = [(s, a, r) of size & from M given policy π
3: Sample H' C uniformly randomly s.t. ||1| [B. [H]
4: for all (s, r), E Hf' do
5: Compute value estimates (st, H), (se+1, H) using known trajectory H
6:
8: < 5(8))
7:
Ft +
8:
Tt-1 f- Tt-1 - yTe + +
9:
Store H in Replay Memory
10:
Update π According to L given D, flush or prune D according to L

---

Specifically, the learning algorithm L first samples a trajectory H = {(s,(( r)), )= of size e from M given the agent's current policy л. The adversary then samples a random subset H' C H of size [ [M] from the trajectory to poison. Given each r)) E H' the adversary first applies the trigger pattern 5(x)) to the agent's 's state-observation, and then computes a Monte-Carlo estimate of the value of 8 and 80+1 as follows:
H
Vis H) =
rt
(9)
This is an unbiased estimator of V.A (s+)
33 which sees usage in methods like PPO and A2C thus allowing to accurately approximate V-M for any E SUS, Sp in expectation. We then use these estimates to replicate our adversarial reward function R' as seen in lines 7 and 8 of Algorithm |
We additionally introduce two hyper parameters, and a, as a relaxation of R'. Larger values of a perturb the agent's 's reward to a larger degree - making the attack stronger, but perhaps making generalization more difficult in DRL methods. On the other hand, smaller values of a perturb the agent's reward less - making easier for DRI. methods to generalize, but perhaps weakening the attack's ss strength. Similarly, scales the first term of R' in poisoned states, Le[a a a+], meaning larger values of c will perturb the agent's ' reward to a greater degree. In practice we find the SleeperNets attack to be very stable w.r.t and most MDPs, however some settings require more hyper parameter tuning to maximize attack success and stealth.
Experimental Results
In this section we evaluate our SleeperNets attack in terms of our two adversarial objectives attack success and attack stealth on environments spanning robotic navigation, video game playing, self driving, and stock trading tasks. We further compare our SleeperNets attack against attacks from BadRL [4] and TrojDRL [14] displaying significant improvements in attack success while more reliably maintaining attack stealth.
6.1 Experimental Setup
We compare SleepenNets against two versions of BadRL. and TrojDRL which better reflect the less invasive threat model of the SleeperNets attack. First is TrojDRI-W in which the adversary does not manipulate the agent's actions. Second is BadRL-M which uses manually crafted trigger pattern rather than one optimized with gradient based techniques. These two versions were chosen so we can directly compare each method's s reward poisoning technique and better contrast the inner and outer loop threat models. Further discussion on this topic can be found in Appendix 9.3
We evaluate each method cleanrl [10] implementation of PPO B1. First, to replicate and validate the results of 4 and [4] test all attacks on Atari Breakout and Qbert from the Atari gymnasium suite [I]. In evaluation we found that these environments are highly susceptible to backdoor poisoning attacks, thus we extend and focus our study towards the following 4 environments: Car Racing from the Box2D) gymnasium [I Safety Car from Safety Gymnasium [1],, Highway Merge from Highway Env (18).
suite of 6 diverse environments against agents trained using the
and Trading BTC from Gym Trading Env (27).
In each environment we first train an agent on the benign, unpoisoned MDP until convergence to act as baseline. We then evaluate each attack two key metrics episodic return and attack success rate (ASR). Episodic return is measured as the agent's cumulative, discounted return as it is training. This metric will be directly relayed to the victim training party, and thus successful attacks will have training curves which match its respective, unpoisoned training curve. Attack success rate is measured by first generating a benign episode, applying the trigger to every state-observation, and then calculating the probability with which the agent chooses the target action a+. Successful attacks will attain ASR values near 100%. All attacks tested over 2 values of c and averaged over 3 seeds given a fixed poisoning budget. e values and poisoning budgets vary per environment. In each plot and table we then present best results from each attack. Further experimental details and ablations can be found in Appendix 9.4 Appendix 9.5 Appendix 9.6 and Appendix 9.7
6.2 NeeeperNets Results
In Table 2 we highlight the universal capabilities of the SleeperNets attack across 6 environments spanning 4 different domains. Our attack is able to achieve an average attack success rate of 100% %
7

---

Empirical Comparison Backdoor Attacks
Inveronmnnt
Highway Merge
Salety
Trade ETC
Car Racing
Breakout
Qbert
Mertra
ASR
ASR
ASK
Sleeper-Nets
100%
0is
100%
100%
0%
100%
016
100%
0%
100%
0S
TruJDRL-W W
57.2%
29.0%
86.7%
3.3%
97.5%
0.5%
73.0%
46.7%
99.8%
0.1%
98.4%
0.5%
BaiRR
22.3%
Mertric
BRR
BRR
BRR
NtesperNets
99.05
0.4%
96.5%
25.4%
100%
18.5%
97.0%
6.35
100%
24.15
100%
12.9%
TrojDRL--W
91.4%
3.0%
81.3%
90.9%
100%S
13.4%
26,66%
26.1%
97.75%
12.2%
100%
10.2%
BuIRI-M
Table 2: Comparison between BadRL-M, TrojDRL-W W. and SlesperNets. Here the Benign Return
Ratio (BRR) is measured as each agent's episodic return divided by the episodic return of an unpoisoned agent, and ASR is each attack's success rate. All results are rounded the nearest tenth and capped at 100%. Standard deviation a for all results is provided in their neighboring column.
over 3 different seeds on all environments highlighting the attack's reliability. Additionally, attack causes little to no decrease in benign episodic return producing policies that perform at least 96.5% well an agent trained with no poisoning. Lastly, since attack regularly achieves 100% ASR during training, we are able to anneal our poisoning rate over time resulting in extremely low poisoning rates like 0.001 % Breakout and 0.0006% on Trade BTC. In practice we anneal poisoning rates for all three methods by only poisoning when the current ASR is less than 100%. More detailed discussion on annealing can be found in Appendix 9.6
Attack Success Rate Highway Merge
Episodic Return Highway Merge
Steeserfiess BacRLM TroDBL.
Stecpertes
BodRi
TraDRUM
14
0.8
12
0.4
10
0.2
Time Steps
Time Stops
20k
40k
60k
20k
40k
60k
80k
Attack Success Rate-Safely Safety Car
Episodic Return Safety Car
Sivereetiiee Bacili44 Tru
Pelonting Sieeprittts Trepoto0
0.8
10
0.6
0.4
0.2
Steps
Stops
10
IM
2M
3M
4M
1M
2M
3M
4M
Figure 3:
Comparison of the SleeperNets, BadRL-M, and TrojDRL-W W attacks on (Top) Highway Merge and (Bottom) Safety Car in terms of (Left) ASR and (Right) episodic return.
We find that the SleeperNets attack outperforms TrojDRL-W and BadRL-M all 6 environments in terms of attack success rate, being the only attack to achieve 100% attack success on all environments. The SleeperNets attack additionally outperforms TrojDRL-W in terms of episodic return all environments. BadRL-M performs slightly better in terms of episodic return in Highway Merge and Car Racing, but in these cases it fails achieve high attack success resulting in 0.2% and 44. % ASR in contrast to SleeperNets' 100% ASR on both environments.
In Figure 3 we plot the attack success rate and episodic return of the SleeperNets, BadRL-M, and TrojDRL--W attacks on Highway Merge and Safety Car. In both cases the SleeperNets attack is able to quickly achieve near 100% ASR while maintaining an indistinguishable episodic return curve compared to the "No Poisoning" agent. In contrast BadRL-M and TrojDRL-W both stagnate in terms

---

of ASR being unable to achieve above 60% on Highway Merge or above 90% on Safety Car. Full numerical results and ablations with respect to e can be found in Appendix 9.5
6.3 Attack Parameter Ablations
In Figure we provide ablation of the SleeperNets, BadRL-M, and TrojDRL-W attacks with respect to poisoning budget 3 and reward poisoning constant e on the Highway Merge environment. For both experiments the same between all three attacks. This highlights the capabilities of the outer-loop threat model in enabling better attack performance with weaker attack parameters.
O for SleeperNets, meaning the magnitude of reward perturbation
When comparing each attack at different poisoning budgets we see that SleeperNets is the only attack capable of achieving >95% ASR with a poisoning budget of 0.25% and is the only attack able to achieve an ASR of 100% given any poisoning budget. TrojDRLL- W is the only of the other two attacks able to achieve an ASR >90%, but this comes at the cost of 11 % drop in episodic return at poisoning budget of 2.5% 10x the poisoning budget needed for SleeperNets to be successful.
Attack Success Rate for Different Poisoning Budgets Sad0t-11
Episodic Return For Different Poisoning Budgets
Tro/DRL-W
Simepertiet
Tredtt.. BasttU Sheeperliets
16
0.8
14
12
0.4
Poisoning Budiget
Paisoning Budget
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
Success Different Values
Episodic Return Different Values
TrajORLWW SudiL-M
Slosourfieta
TraDRt,
Baditu Slaepe Nets
14
0.8
12
0.6
0.4
10
0.2
30 40 50
30
Figure 4: (Top) Ablation with respect to poisoning budget / for each attack given a fixed 40. (Bottom) Ablation with respect given a fixed poisoning budget of 0.5%. Both experiments were run on Highway Merge with value of a 0 for SleeperNets.
We see similar trend for different values of e SleeperNets is the only attack which is able to achieve an ASR near 100% given a poisoning budget of 0.5% and c = 20. TrojDRL-W, in contrast, only achieves an ASR of % given the same poisoning budget and much larger reward constant of c 40. This again comes at the cost of a 10% drop in episodic return. Between both ablations BadRL-M is unable to achieve an ASR greater than 40% and only achieves ASR better than 1% when given 5% poisoning budget and using c = 40. Numerical results can be found in Appendix 997
7 Conclusion and Limitations
In this paper we provided multiple key contributions research in backdoor attacks against DRL. We have proven the insufficiency of static reward poisoning attacks and use this insight to motivate the development of dynamic reward poisoning strategy with formal guarantees. We additionally introduced a novel backdoor threat model against DRL algorithms, and formulated the SleeperNets attack which significantly and reliably outperforms the current state-of-the-a in terms of attack success rate and episodic return at very low poisoning rates. The strong theoretical and empirical results of this work motivate further research into defense techniques against backdoor poisoning attacks in DRL such new detection, certified robustness, auditing approaches.

---

The main limitation of SleeperNets is that the formulation of R' allows the attack's s reward perturbation to grow arbitrarily large given an arbitrary MDP. When a is large this may make detection of the attack easier if the victim is able to scan for outliers in the agent's 's reward. We believe this is an interesting area of future research that may lead to the development of increasingly sensitive detection techniques and, in response, more stealthy attacks. Additionally, while our experiments are expansive with respect to environments and most attack parameters, this paper is not an exhaustive study of all facets of backdoor attacks. We do not study the effectiveness of the adversary's ' chosen trigger pattern nor do we consider triggers optimized with gradient or random search based techniques 4 Combining these techniques with SleeperNets is also interesting area of future research which will likely result in a further decrease in poisoning budget and the ability to perform stealthier attacks.
8 Broader Impacts
In this paper we expose a key vulnerability of Deep Reinforcement Learning algorithms against backdoor attacks. As with any work studying adversarial attacks, there is the possibility that SleeperNets is used to launch a real-world, malicious attack against a DRL system. We hope that, by exploring and defining this threat vector, developers and researchers will be better equipped to design counter measures and prevent any negative outcomes. In particular we believe the implementation of isolated training systems which will be harder for the adversary access and the development of active attack detection techniques will both be critical for mitigating attacks. We would also like to highlight the potential of malicious trainers using these backdoor attack approaches against their own agents for malicious purposes. We believe the development of new DRL auditing techniques is necessary to identify these compromised agents.
Acknowledgements
This Research was developed with funding from the Defense Advanced Research Projects Agency (DARPA), under contract 1912C33C00031 NSF award 1N8-2312875, and NSF award FMitF- 2319500. We thank Evan Rose, Simona Boboila, Peter Chin, Lisa Oakley, and Aditya Vikram Singh for discussing various aspects of this project.
References
[1] G. Brockman, V. Cheung, I.. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba. OpenAl Gym. arXiv preprint arXiv:1606.01540 2016.
[2] J. Chen and Q Gu. Rays: A ray searching method for hard-label adversarial attack. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1739-1747, 2020.
[3] J. C. Costa, T. Roxo, H. Proença, and P. R. Inácio. How deep learning sees the world: A survey on adversarial attacks defenses. arXiv preprint arXiv:2305. 10862, 2023.
[4] Cui, Y. Han, Y. Ma, Jiao, and J. Zhang. Badrl: Sparse targeted backdoor attack against reinforcement learning. arXiv preprint arXiv:2312.12558 2023.
[5] S. Dolan, S. Nayak, and H. Balakrishnan. Satellite navigation and coordination with limited information sharing. In Learning for Dynamics and Control Conference, pages 1058-1071. PMLR, 2023.
[6] T. Franzmeyer. S. McAleer, J. F. Henriques, J. N. Foerster, P. H. Torr, A. Bibi, and C. S. de Witt. Illusory attacks: Detectability matters in adversarial attacks on sequential decision-makers. arXiv preprini arXiv: 10170, 2022.
[7] A. Gleave, M. Dennis, C. Wild, N. Kant, S. Levine, and S. Russell. Adversarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv: 1065, 2019.
[8] S. Gu, L. Yang. Y. Du, G. Chen, Walter, J. Wang, Y. Yang, and A. Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint 10330. 2022.
[9] Gu, B. Dolan-Gavitt, and Garg. BadNets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint 1w..:v1100.0777700 2017.
10

---

[10] S. Huang, R. F. J. Dossa, C. Ye, J. Braga, D. Chakraborty, K. Mehta, and J. G. Araújo. Cleanrl: High-quality single-fit file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 131274))1-18, 2022.
[11] C. D. Hylender, P. Langlois, A. Pinto, and S. Widup. Verizon 2024 data breach investigations report. https: /wwww/wwww/// verizon. com/ bussinnes/rreoourcs///ppprr /bbir/ 2024.
[12] J. Ji. B. Zhang, X. Pan, Zhou, J. Dai, and Y. Yang. safett--gymasium GitHub repository, 2023.
[13] T. Kabbani and E. Duman. Deep reinforcement learning approach for trading automation in the stock market. IEEE Access, 10:93564-935777 2022.
[14] Kiourti, K. Wardega, S. Jha, and W. Li. Trojdrl: Trojan attacks on deep reinforcement learning agents. arXiv preprint arXiv: /eee//933.000 96668833..,,, 2019.
[15] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez. Deep reinforcement learning for autonomous driving: survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):4909-4926, 2021.
[16] Kostrikov, A. Nair, and S. Levine Offline reinforcement learning with implicit q-learning. arXiv preprint 2rXiv:2100066/11 2022.
[17] A. Krnjaic, R. D. Steleac, J. D. Thomas, G. Papoudakis, L. Schäfer, A. W. K. To, K.-H. Lao, M. Cubuktepe, M. Haley, P. Börsting, and S. V. Albrecht. Scalable multi-agent reinforcement learning for warehouse logistics with robotic and human co-workers, 2023.
[18] E. Leurent. An environment for autonomous driving decision- https://github. eleurent/hhihwwy--on 2018
[19] Y.-C. Lin, Z.-W. Hong, Y.-H. Liao, M.-L. Shih, M.-Y. Liu, and M. Sun. Tactics of adversarial attack on deep reinforcement learning agents. arXiv preprint 1703:06600554 2017.
[20] C. Lu, Т. Willi, A. Letcher, and J. N. Foerster. Adversarial cheap talk. In International Conference on Machine Learning, pages 22917-22941 PMLL..991....RR...MM PMLR, 2023.
[21] A. Madry, A. Makelov, Schmidt, D. Tsipras, and Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint
2017.
[22] K. Mahmood, Mahmood, E. Rathbun, and M. van Dijk. Back in black: A comparative evaluation of recent state-of-the-art black-box attacks. IECE Access, 10:998-1019, 2022.
[23] V. Mnih, P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pages 1928-1933 PMLR, 2016.
[24] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 2013.
[25] L. Nasvytis, K. Sandbrink, Foerster, T. Franzmeyer, and C. S. de Witt. Rethinking out- of-distribution detection for reinforcement learning: Advancing methods for evaluation and detection. arXiv preprint urXiv:2404.07099, 2024.
[26] L. Ouyang. J. Wu, X. Jiang. D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[27] C. Perroud.
Gym trading env. htttps//githhb... htt.gtath..commmemmmmaeeerrraaaa Traaarrrrrrnnnrta---rr------- Env
2023.
[28] N. H. Pham, I. M. Nguyen, J. Chen, H. T. Lam, S. Das, and T.-W. Weng. Evaluating robustness of cooperative MARL: A model-based approach. arXiv preprint arXiv:2202.035: 2022.

---

[29] A. Rakhsha, G. Radanovic, R. Devidze, X. Zhu, and A. Singla. Policy teaching via environment poisoning: Training-tima adversarial attacks against reinforcement learning. In International Conference Machine Learning, pages 7974-7984. PMLR, 2020.
[30] A. Rangi, H. Xu, L. Tran-Thanh, and M. Franceschetti. Understanding the limits of poisoning attacks in episodic reinforcement learning. arXiv preprint arXiv:2208.13663 2022.
[31] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1 1e.. 2017.
[32] D. W. Stroock. An introduction to Markov processes, volume 230. Springer Science & Business Media, 2013,
[33] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. 2018.
[34] L. Wang, Z. Javed, X. Wu, W. Guo, X. Xing, and D. Song. BACKDOORL: Backdoor attack against competitive reinforcement learning. arXiv preprint arXiv:2105.00579 2021.
[35] Z. Yang, N. Iyer, J. Reimann, and N. Virani. Design of intentional backdoors in sequential models. arXiv preprint 1902.00902.000
2019.
[36] Y. Yu, J. Liu, S. Li, K. Huang. and X. Feng. A temporal-patter backdoor attack to deep reinforcement learning. In GLOBECOM 2022-2022 IEEE Global Communications Conference, pages 27 2710-2715. IEEE, 2022.
[37] M. Zare, P. M. Kebria, A. Khosravi, and S. Nahavandi. survey of imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions Cybernetics, 2024.
[38] H. Zheng, X. Li, J. Chen, J. Dong, Y. Zhang, and C. Lin. One4all: Manipulate one agent to poison the cooperative multi-agent reinforcement learning. Computers & Security, 124: 103005, 2023.
12

---

Supplemental Material
In the appendix we provide further details and experimental results which could not fit into the main body of the paper. In Appendix 19.1 we provide proofs for Lemma I, Lemma 2, Theorem 1, and Theorem 2 provided in Section 4.4 In Appendix 9.2 we prove the insufficiency of static reward poisoning as discussed in Section In Appendix p.3 we provide further discussion on the baselines evaluated in the main body. Appendix 9.4 we provide further experimental setup details and justification for each hyper parameter we used. In Appendix 9.5 we provide full experimental details used generate Table in Section In Appendix 99.6 we provide further details and plots relating SleeperNets' poisoning rate annealing capabilities as referenced in Section 6 In Appendix 9.7 we provide numerical results for our ablations of and 3 presented in Section 6.3 In Appendix 9.8 we provide an overview of the computational resources used in the experimental results of this paper. Lastly, in Appendix 9.9 we provide further discussion of related works on non-backdoor, adversarial attacks against DRL.
9.1 Proofs For Dynamic Reward Poisoning Guarantees
Lemma V.M (sp) = π (sp. Vsp E Sp, π E Thus, the value of policy x in poisoned states Sp Sp is equal to the probability with which it chooses action at.
Proof. Here we proceed with direct algebraic proof simplifying the definition of V.M (sp) Vap Sp until we show it is equal to (8pp a+).
Let: Sp E Sp, π E II
(10)
V. (sp)
Σ (Sppaa) a)
Σ
a, (sp, + vV (s')
(11)
aeA
s'ESUS,
(Sps a)
Σ
T' a s 11 (s') + V M (s')
(12)
acA
d'SUS
(Sp, a) (L(a a+1 Σ
(,,)')
(13)
S'ESUS
Σ (Spp, a) (L(a=at)
(14)
T Tspp )
(15)
(16)
QED
Lemma 2 V.M (s) Va e S, II. Therefore, the value of any policy n in the adversarial MDP M' is equal to its value in the benign MDP M for all benign states $ E S.
Infinite Horizon MDP Case
13

---

Proof. In this proof we will show that V V (s) o 0 leads to a contradiction V E S, thus V (s) = Vall (s) must be true. We will begin by expanding and subsequently simplifying V
Let: s s 1
(17)
VM (8) Σ a()
Σ ((
+ (d')
(18)
x
a
s'ESUS,
Σ T(s, a) [[(sa.))Baaas (s')
acA
N'ES
(19)
(s)]]
SE
Σ
(s, [11 -3)
Σ
T(s a, 8') + (V
aEA
dEES
+
Σ
VV2 - (d) + + VV (6)) + VV (6'I
JEE
(20)
Σ aEA
7(, al - 3)
Σ
T(s, 8 ()) + +VV2 (s')
EEES
(21)
+ 8 S'E,,
Σ a(s,a,8 (s)) + + (0 (s')l
M
To get to the next step in this simplification we observe that the second sum is over s' E Sp.. but only includes terms 8 (s'). d is an invertible function and is thus bijective - as explained in Section 4.2 therefore the sum is equivalent to one over s' E S.
VM n
(s)
= Σ T(S a)((1 -3) T(s (R(ss, 8') + vV (s')l (s')
AEΛ
E'ES
(22)
+ 3
T(s, a, (i,,ass)) s')[R(s, a, 8') + yV (s')ll
SEES
Σ T(s
a)
E (')(s,, s') + 7V (s')
(23)
EEES
From here are nearly done, but still have connect it back to our goal of showing V.M (s)
V.M (s), in other words: Vs E S. D. = VM (s) (s)= 0: 0:
D2
Σ T(s,a) a)
Σ T(s, a 8 (R((,,,, (s))
aEA
S'ES
(24)
Σ (ss e'ES
T(s.a,
+ VV (s')
Σ T(S, a)s
Σ T(s,,,,
vV=
A
NES
(25)
Σ T(s (') () 5 + +VV2 (s')l)
8'ES
Σ T(S, a) Σ T(s, ('R(s,) + vV2
aEA
SEES
(26)
[R(s, a, s') + +VVM(s''II]
Σ (8, a) aeA
T(s, ITT(s, a, s' V (s) - (
(27)
e'ES
After this we will convert the problem into the much more manageable matrix form:
Let: D E RRSI such that D, V.M (s) V.M (s)
(28)
Let: P E ||||s|S| such that Pa.s' = Σ (s, a()
(29)
a
14

---

We know that P is Markovian matrix through its definition every row Pa represents a probability vector over next states s' given iitiill state thus every row sums value of Markovian matricies have many useful properties which are relevant to Reinforcement Learning, but most relevant to us are its properties regarding eigenvalues:
PD = aD =
(30)
In other words, the largest eigenvalue of valid Markovian matrix P is 1 32 Using our above definitions we can rewrite Equation 27 as:
D = P(nD)
(31)
7 PD
(32)
Let's assume, for the purpose of contradiction that D +
Since [0, 1) this implies P has eigenvalue larger than 1. However, P is Markovian matrix and thus cannot have eigenvalue greater than 1. Thus D Ô must be true.
QED
Finite Horizon MDP Case
Proof. In the finite horizon case we redefine the benign MDP as M = (S, and the poisoned MDP as M' (S Sp A, T" R', H, where N is the horizon the MDPs:
(s) 0 Vπ II, SUS, ift H
(33)
Using this we will proceed with proof by induction, starting with the base case n. (s)
(s) Vs E S. This can quickly be derived by drawing upon the simplification for V a. (8) found
in Equation 23
(8) Es(s,a)
(s, Σ ((s,a, s'ES
VAA (s)
(34)
Here the term M +1 (8') disappears due to our definition in Equation 33
Thus we have proven
the condition met Vs E S for base case = H. We will then continue to the inductive case: assume the condition holds for some & + 1 S H, we can then show the following:
V. (8) Σ T(8,))
Σ
T(s (ls,,,s')B(s, 8') VV
(35)
aEA
S'ES
Σ T(s
ΣT(s T(s, a, s') s') VV (s)+
(36)
s'ES
(s)
(37)
Thus the inductive hypothesis and base case hold, therefore Vs = V (s) Vs E S, h < H.
QED
(sp) Ysp E Sp, Il &> T' 1 Vsp E Sp. Thus, IT" is optimal M' if and only if takes action a with probability 1 in poisoned states sp Sp.
Theorem 1 VM (s) > VM
T"
Forward Direction V (s)
M (s) Ys E Sp, IT + x (s, (t) Vs E Sp
Proof. Assume for the purpose of contradiction that 3r E II such that æ optimal in M' but (sp: a a)) < 1 for some Sp E Sp-
Let n' be some arbitrary policy such that x' x'(s,, at) 1.
From Lemma we know that V (sp) and V (sp)
Thus follows that VM (sp) < V (sp) which contradicts the initial assumption that optimal
in M'.
QED
15

---

Reverse Direction as) 1 Ss E Sp (s) (s) As Sp, EI
(s)
Proof. Assume for the purpose of contradiction that x (s,aat) As Sp but Eo II, s e Sp
such that V-M" (s*) < V (s*).
From Lemma we know that V (s) = IAs E Sp. therefore VAM >s*)>1.
Again using Lemma I this implies that π (s*", aa) 1 which contradicts the fact that T( is valid probability vector no action can have probability greater than 1.
QED
Thus we have proven Theorem 1. QED
Theorem 2 V- 2 V (s)
Ys Vs E (s) 2 (s) Vs E S, v E II. Therefore, क
optimal M' for all benign states 8 S and only if x is optimal M.
Forward Direction VM (8)
V (s) Ys E E II (M (+) 2 V (6) Vs
Proof. Assume for the purpose of contradiction that x² optimal in M' but sub optimal M.
This implies there some и E II and E S such that V.A (s) V (s).
From Lemma we know V-M (s) V (s) V (s) V.M (s)
This implies (=M for all E S.
(s) V.M (s) which contradicts the initial assumption that x is optimal in M'
QED
Reverse Direction V=M
2 V M (8)
V (8) V.M (s) (M
Proof. Assume for the purpose of contradiction that x² is optimal in M but is sub optimal in M'.
This implies there is some E IT and E S such that VM (s) V.M (s).
From Lemma 2 we know V (s) M M
V (s)
This implies V all S.
π (s)
which contradicts the initial assumption that x" optimal in M for
QED
Thus we have proven Theorem 2. QED
9.2 Proofs For Insufficiency of Static Reward Poisoning
Static Reward Poisoning to Target Action
Static Reward Poisoning Results Sub Sub-Optimal Pollcy Benign
0, aj.
0, aj,
0,
Slow
Slow2
Slowl
Starl
Survive
Start
as,
00 a.a
C, B
0,
Fasts
Fastz
Figure 5: (Left) MDP Mi, (Right) MDP M2
Static Reward Poisoning is Insufficient for Attack Success
Proof. Our goal here is to show that for all static reward poisoning attacks using a poisoning rate E [0, 1) and reward constant there exists an MDP Mi such that attack success cannot be achieved. other words, we want to show the following to true ((t(Start), a1) > ((a(Start) a+) given the poisoned MDP M using static reward poisoning with constant and poisoning rate of (.
16

---

Here we assume the agent always takes action at in state o(Survive) as it is the optimal action. We proceed as follows:
cyy ((6(Start), a1) =-c Σ
c
(38)
1 1-7
==0
Q(ó(Start), a²) =
(39)
From here we want to show that By E [0, 1) such that Q(6(Start), a1) > >((5(Start), a+)
Q(6(Start), a1) > Q(6(Start), at) +
c > c
(40)
1 1-7
1 > 2
(41)
(42)
Thus, we know that there exists some y E (3. 1) such that ((5(Start), a,) > ((ō(Star)) a at) Thus the target action is sub-optimal in the adversarial MDP using static reward poisoning. Therefore static reward poisoning is insufficient for attack success in arbitrary MDPs.
QED
Static Reward Poisoning is Insufficient for Attack Stealth
Proof. Our goal here is to show that for all static reward poisoning attacks using a poisoning rate 3 E [0, 1) and a reward constant c, there exists an MDP M2 such that attack stealth cannot be achieved. In other words, we want to show that the agent learns a sub-optimal policy the benign MDP M2 given it is trained in the poisoned MDP M2. Specifically, we want to show the following to be true: Q(Start, 01) > (Sttart, a+) in Mr meaning dN is the optimal action in M22 given state "Start" whereas at is optimal in M2. Similarly to the last proof we assume the agent chooses the target action i in poisoned states since it's the optimal action - and thus receives a reward of +c
in all states with probability B. We proceed as follows:
Q(Start, a1) 7BC + 72 + 73c
(43)
Q(Start, a+) =BBc+ 72c
(44)
From here we want show that 37 E [0, 1) such that Q(Start, a1) > Q(Start at)
(Sttart, a1) > (Start, at) = 7Bc + 72 e + 72c 7Bc + 72c
(45)
+ 7 1) 73 > 0 + >>>>>4 1-3
(46)
Thus, we know that there exists some E (1 - B, 1) such that QSStart, a1) > QSStart, a+). Thus aj optimal action in M₁₂ given state "Start", whereas at is optimal in M2. Therefore the agent will learn a sub-optimal policy in M2 when trained on M₁.
QED
9.3 Further Justification of Baselines Used
In the TrojDRL paper [4] two main attack techniques are proposed TrojDRL-S and TrojDRL- which correspond to attacks where the adversary can and cannot manipulate the agent's 's actions during each episode respectively. The core algorithm behind both of these attacks is fundamentally the same however, as the adversary manipulates the agent's state observations and reward in the same way. As discussed in Section we believe action manipulation attacks are much easier to detect and offer no theoretical benefits to the attacker. Thus we compare against TrojDRL-- so we can have a more direct comparison between the dynamic reward poisoning of SleeperNets and the static reward poisoning of TRo,DRL.
In BadRL [4] the authors propose a technique for optimizing the adversary's 's trigger pattern. They show that this technique can lead to improvements in attack success rate over TrojDRL in Atari domains given low poisoning budgets. This approach comes with a very high cost in terms of threat model however. Not only must the adversary have full access to the MDP to train the attack's Q- Network, but they must also have gradient access to the agent's 's policy network to optimize the trigger pattern. Furthermore, the technique not exclusive or dependant on any other piece of BadRL's formulation, it can be generically applied to any backdoor attack. Taking this all into consideration.
17

---

we believe it is most fair to compare all methods using the same, fixed trigger patter, hence the usage of BadRL-M. We could otherwise compare all techniques given optimized trigger patterns, but that would go against our paper's motive of developing theoretically sound backdoor attack that can be effective within a restrictive threat model. Thus we show that the SleeperNets attack is effective even when using simple, hand-crafted trigger patterns.
9.4 Experimental Setup Details
Comparison of Environments Tested in this Work
Environment
Task Type
Observations
Enviromment Id.
Breakout [1]
Video Game
Image
BreakoutNoFramesske
Q*bert [
Video Game
Image
QbertNoFrameskip--r
Car Racing [
Video Game
Image
CarRacing-v2
Highway Merge [8]
Self Driving
Image
merge-v)
Safety Car 12
Robotics
Lidar-Proprioceptive
SafetYCarGoal
Trade BTC
Stock Trading
Stock Data
TradingEnv
Table 3: Comparison of the different environments tested in this work. All action spaces were discrete
in some form, though for environments like Car Racing, Safety Car, and Trading-Enn discretized versions of their continuous action spaces were used. The final column refers to the exact environment Id used when generating each environment through the gymnasium interface [
In Table 3 we compare each environment in terms of task type and observation type. We additionally provide the exact environment 1d. used when generating each environment through the gymnasium interface M. Breakout, Q*bert, Car Racing, and Highway Merge all use gray scale 86x86 image observations stacked into 4 frames. Safety car uses 16 lidar sensors placed in a circular pattern around the agent to allow the agent spatially orient itself with respect to the goal and obstacles. It additionally uses proprioceptive information about the agent (agent's 's velocity, angle, etc.). Lastly, Trade BTC uses information corresponding to the volume, closing price, opening price, high price, and low price of BTC on a per-day basis with respect to USDT (equivalent to S1).
Environment Attack Parameters
Environment
Trigger
Poisoning Budget
Clom
Chigh
Target Action
Breakout
Checkerboard Pattern
0.03%
5
Move Right
Q*bert
Checkerboard Pattern
0.03%
Move Right
Car Racing
Checkerboard Pattern
0.5%
10
Turn Right
Highway Merge
Checkerboard Pattern
0.5%
40
Merge Right
Safety Car
Lidar Pattern
0.5%
0.5
Accelerate
Trade BTC
Boolean Indicator
0.005%
0.05
0.1
Short BTC 100%
Table 4: Attack and learning parameters used for each environment. Clow was chosen as the smallest value for which TRL and BadRL could achieve some level of attack success. Chigh was chosen as the largest value for which Tro and BadRL did not significantly damage the agent's benign return. A similar method was used in determining the poisoning budget.
In Table we summarize the trigger pattern, poisoning budget, target action, and values of Clow and Chigh used in each environment. As explained in the caption, values of Clows Chigh, and poisoning budget were chosen as appropriate for each environment. Specifically we chose values which laid on the boundaries of performance for both TrojDRL. and BadRI. values of e higher than Chigh would begin to damage episodic return and values of e lower than Clow would see a significam drop off in attack success rate. Poisoning budgets were chosen similarly, as lower values would result in significantly degraded attack success rates for all three attacks, though SleeperNets usually fared best at lower poisoning budgets.
For all image based environments a 6x6, checkerboard pattern was placed the top of each image frame as the trigger pattern. For both inner and outer loop attacks the trigger was applied to all images in the frame stack. the case of Safety Car we embed the trigger pattern into the agent's lidar observation. In particular, the agent has 16 lidar sensors corresponding to its proximity to "vase" objects. We set 4 of these signals to have a value of 1, corresponding to 4 vases placed directly in front of, behind, to the left, and to the right of the agent. This type of observation cannot occur in
18

---

-v0 as there is only vase object in the environment. Lastly, in Trade BTC we simply added an additional, binary feature to the agent's observation which set to in poisoned states and O otherwise. This choice was primarily made to ensure the trigger is truly out of distribution while also not ruining the state observation received by the agent. If any relevant piece of information (e.g. opening price, volume, etc.) is altered, one could argue that the target action at is actually optimal. This would make the attack more of an adversarial example than a backdoor.
Target actions were chosen as actions which could be cause the largest potential damage if taken in the wrong state. In the case of Breakout and Q*bert "Move Right" is chosen as all movement actions are equally likely to be harmful. In Car Racing "Turn Right" was chosen as it is extremely easy for the agent's car to spin out if they take a wrong turn, particularly when rounding a leftward corner. In Highway Merge "Merge Right" was chosen as the goal of this environment is to merge left, getting out of the way of vehicle entering the highway. Merging right at the wrong time could easily result in a collision, we believe this is why this environment was the hardest to solve for the inner loop attacks. Lastly Trading BTC we chose "Short BTC 100%" as the target action since this environment uses data gathered from the Bitcoin boom between 2020-2024, making an 100% short of Bitcoin potentially highly costly. In spite of this, most attacks were still able to be successful in this environment. We believe this is because the agent is able to immediately correct the mistake the following day, only running the risk of a single day's price change.
9.5 Detailed Experimental Results
In this section we provide numerical experimental results for those presented in Section 6.2 In
Table we present the numerical results gathered TrojDRL-W and BadRL-M including further ablations with respect to c which were not directly presented in the paper. All results placed in bold represent those presented the best result for each attack in the main body of the paper. Additionally, standard deviation values are provided for each result in its neighboring column.
In T 6 we give full numerical results for the SleeperNets attack on our 6 chosen environments. Here we evaluated the attack with the same values of Clow and Chigh as we did for TrojDRL and BadRL. The only exception is Highway Merge which uses Ctow 5 and Chigh 10 respectively. This decision was made because the attack was successful at Ctow = 30 and Chigh = 40 given a = 0, thus further tests of were redundant. Instead we decided to test the limits of the attack be evaluating at very low values of c.
We also evaluate SleeperNets given three values of a: 0, 0.5, and 1. Our results show that SleeperNets generally very stable with respect to a and c, however for some of the more difficult environments, like Highway Merge, larger parameter values are used. Lastly, we present plots of each attack's 's best results with respect to episodic return and attack success rate in Figure 6 through Figure 11
Performance of TrojDRI, and BadRL at Different Values of TrojDRL-W
Attack
BadRL-M
Clow
Chigh
Clow
Chigh
Metric
ASR
0
ASR
ASR
ASR
o
Breakout
98.5%
2.0%
99.8%
0.1%
40.8%
31.7%
99.3 %
0.6%
Q*
88.2%
6.3%
98.4 %
0.6%
15.1%
1.4%
47.3% %
22.6%
Car Racing
18.1%
1.5%
73.0%
46.7%
44.1%
47.0%
23.0%
1.2%
Highway Merge
17.3%
51.5%
57.2 %
29.0%
0.1%
0.0%
0.2%
0.6%
Safety Car
71.7%
5.9%
86.7 %
3.3%
60.9%
10.8%
70.6%
17.4%
Trade BTC
93.3%
2.3%
97.5 %
0.6%
24.5%
1.9%
38.8 %
44.2%
Metric
BRR
BRR
BRR
BRR
Breakout
99.4%
16.1%
97.7% %
11.8%
91.9%
11.9%
84.4%
2.7%
Q*bert
100%
11.0%
100%
9.0%
100%
9.4%
100 %
10.0%
Car Racing
96.2%
8.0%
26.6%
62.2%
98.1
9.4%
100.0%
5.8%
Highway Merge
95.3%
0.3%
91.4% %
1.9%
100%
0.0%
100 %
0.0%
Safety Car
78.2%
50.8%
81.3 %
38.9%
78.8%
39.9%
83.6 %
30.9%
Trade BTC 100%
9.8%
100%
15.4%
100%
13.2%
100 %
12.5%
Table 5: Full Experimental results of TrojDRL- and BadRL-M on our 6 environments given Clow
and Chigh. For each attack we bold the results which were used in the main body of the paper. In all cases but one this was Chigh- Standard deviations a for each experiment are also given.
19

---

Experiment
SleeperNets Breakout
SleeperNets Q*bert
Clow
Chigh
Clow
Chigh
Metric
ASR
0
ASR
ASR
ASR
0
100%
0.0%
100%
0.0%
76.2%
41.2%
100%
0.0%
0 = 0.5
100 %
0.0%
100%
0.0%
100% %
0.0%
100%
0.0%
a
100%
0.0%
100%
0.0%
100%
0.0%
100%
0.0%
Experiment
SleeperNets
Car Racing
SleeperNets
Highway Merge
Clow
Chigh
Clow
Chigh
Metric
ASR
ASR
ASR
ASR
a = 0
23.2%
4.0%
22.3%
4.7%
28.5%
22.2%
83.5%
9.1%
a = 0.5 a=1
29. 1%
8.5%
73.7%
45.6%
33.1%
56.5%
78.0%
38.0%
97.1% %
3.7%
100% %
0.2%
33.5%
57.7%
99.3%
1.2%
Experiment
SleeperNets
Safety Car
SleeperNets
Trade BTC
Clow
Chigh
Clow
Chigh
Metric
ASR
ASR
ASR
ASR
0
a =
100 %
0.0%
100%
0.0%
100%
0.0%
99.9%
0.2%
0.5
100%
0.0%
100%
0.0%
66.7%
57.7%
100%
0.0%
a=
100%
0.0%
100%
0.0%
100%
0.0%
100%
0.0%
Experiment
SleeperNets
Breakout
SleeperNets
Q*bert
c Metric
Clow
Chigh
Clow
Chioh
BRR
BRR
BRR
BRR
a = 0
97.2%
10.6%
95.5%
10.6%
93.3%
12.4%
100%
17.5%
a 0.5 a=
100 %
8.4%
95.2%
10.0%
100%
7.8%
100%
8.7%
90.7%
7.2%
100%
9.2%
100%
15.4%
94.2%
9.0%
Experiment
SleeperNets
Car Racing
SleeperNets Highway Merge
Clou
Chigh
Clow
Chigh
Metric
BRR
BRR
BRR
BRR
0
a
100%
7.4%
98.4%
11.3%
99.3%
0.5%
98.6%
0.5%
= 0.5
100%
8.3%
94.1%
8.8%
99.7%
0.3%
99.4%
0.4%
a
100%
4.8%
97.0%
6.3%
99.7%
0.3%
99.7%
0.3%
Experiment
SleeperNets
Safety Car
SleeperNets
Trade BTC
Clow
Chigh
Chigh
Metric
BRR
BRR
BRR
BRR
77
96.5%
25.4%
77.7%
42.0%%
100% %
17.7%
100%
15.1%
0 = 0.5
86.8%
38.19
77.7%
39.7%
100%
14.9%
100%
19.5%
73.6%
42.7%
94.0%
31.2%
100%
13.4%
100%
9.2%
Table 6: Tabular ablation of SleeperNets with respect to and All results are presented with standard deviations in their neighboring column. Results placed in bold represent those presented in the main body of the paper. All results use the same values of Clow and Chigh. seen in Table excluding Highway Merge which uses values of 5 and respectively.
Success Breakout
Episodic Breakout
Sleasestiuts
BaciLL
Tra
folusiing
Sieppriets BadiL.M Mi TraDRR//
0.8
0.6
400
0.4
0.2
100
Time Steps
Time Stops
SM
15M
10M
Figure 6: Best results for each attack on Breakout.
20

---

Attack Success Rate-QQ'bon Q'bert
Episodic Return- Q'bert
Sosphho baft. Te
Fatuating Sieeprliets
BaSR.M TRODRLAW
15000
0.2
10000
5000
0.2
Steps
Time Stops.
4M
6M
10M
12M
14M
Figure 7: Best results for each attack on bere...
Attack Success Rate Car Racing
Episodic Return Car Racing
Steoperteess Polsoing
800
0.8
600
0.6
400
200
0.2
200
Time
500k
1M
1.5M
2M
2.5M
500k
1M
1.5M
2M
2.5M
Figure 8: Best results for each attack on Car Racing.
Attack Success Rate - Highway Merge
Episodic Return Highway Merge
Slesperrtess
SudiL-M
Roturn
14
0.8
0.4
0.2
Time
Time Steps
20k
40k
60k
80k
20k
40k
60k
80k
Figure 9: Best results for each attack on Highway Merge.
Attack Success Safety Car
Episodic Return Safety
Sleacestiats BaciLLM Tra
Polunting
Staepertets BadRLMM
0.4
0.2
Time Steps
Time Steps
1M
3M
Figure 10: Best results for each attack on Safety Car.
21

---

Attack Success Rate Trade BTC
Episodic Return Trade BTC
Siescerfiets DR BacRU-M
Fetuoning
Sieperletts Tra/DRLWW W
RadRLMM
0.8
0.6
0.2
Steps
Time Staps,
20M
40M
60M
20M
40M
60M
801
Figure 11: Best results for each attack on Trade BTC.
9.6 Poisoning Rate Annealing
In this section explore the capabilities of SleeperNets to anneal its poisoning rate many environments. In particular we implement poisoning rate annealing as follows if the adversary's 's ASR is currently 100%. skip poisoning the current time step/trajectory. This method is applied to all three attack including TrojDRL and BadRL. SleeperNets is able to utilize our poisoning rate annealing the most, however, it regularly achieves 100% ASR early in training. In Figure T2] Figure 13] and Figure 14 we present plots of the empirical poisoning rate of each attack while targeting each respective environment. We to be significantly lower than all other attacks on every environment except for Highway Merge. Of particular note achieve 0.001 poisoning rate on Breakout and 0.0006% poisoning rate on Trade BTC.
that the SleeperNets attack is able decrease its poisoning rate
We can see that the poisoning rate of BadRL-M also often falls below the given poisoning budget. and falls very low in the case of Highway Merge. This is feature of the attack as directly stated in 4.. One of the attack's ' main goals is to utilize a "sparse" poisoning strategy which minimizes poisoning rate while maximizing attack success and stealth. In many cases this can work, however in the case of environments like Highway Merge this works against the attack. In particular, the attack only poisons in states it sees as having "high attack value" meaning the damage caused by taking at instead of the optimal action is maximized. environments like Highway Merge this set of states is very small and thus the attack only poisons occasionally, rather than regularly and randomly like TrojDRL and SleeperNets. We can see something similar the case *errt however the attack eventually corrects for the low poisoning rate and ends up with a value near the budget of 0.03%.
Polsoning Breakout
Poisoning Rate Q"bert
Sleasesfiats BaciLLl
TraDRLW
Slesperlitts SudRL.M TrofDRL-WW
0.0003
0.0003
2.5e-4
0.0002
0.0002
1.5e4
1.5e4
0.0001
0.0001
5e.5
Time Stapo-
Time Steps
10M
2M
10M
Figure 12: Comparison of poisoning rate for each attack on Breakout and Q*
22

---

Poisoning Rate Car Racing
Poisoning Rate Highway Merge
Sieecerfiets DR BacRl-M
Stepprliets SadR1-1
TrepRL-W
0.005
0.004
0.003
0.003
0.002
0.001
Steps
Stopts
0
500K
1M
1.5M
2M
2.5M
34
20k
40%
80k
Figure 13: Comparison of poisoning rate for each attack on Car Racing and Highway Merge
Poisoning Safety
Poisoning
Steeserfiets BaCRLM
TroDRL
Sleepe Nots TiiDRRW E
0.0008
56-5
0.0004
40-5
0.0003
3e-5
0.0002
2e-5
Time Steps
Time Steps
Figure 14: Comparison of poisoning rate for each attack on Safety Car and Trade BTC
9.7 Numerical Results from e and Ablations
In this section we provide full numerical results from the ablations with respect to and 3 presented in Section 6.3 All results given with standard deviations in their neighboring column. The best result from each column is placed in bold.
Ablation of Backdoor Attacks With Respect to
10
20
30
40
30
ASR
ASR
SteeperNets
83.6%
9.4%
98.9%
1.7%
99.7%
0.455
100.%
0108
100.5%
017%
TrojDRL-W W
4.3%
11.1%
11.4%
1.3%
30.6%
24.0%
57.2%
29.0%
56.7%
19.3%
HacRL-M
0.0%
0.1%
0.2%
0.3%
0.1%
Metric
BRR
BRR
BRR
BRR
BRR
SleeporNets
97.8%
90.4%
0.8%
0.5%
TrojDRL-W 98.2%
1.0%
94.7%
3.2%
95.1%
2.9%
90.4%
7.9%
90.8%
3,8%
BadRL-M
100%
0.0%
100%
0.1%
100%
0.0%
100%
0.18
100%
0.0%
Table 7: Numerical results from the ablation with respect to c. Each result is given with its respective
standard deviation in the neighboring column. The best result in each column is placed in bold.
Ablation of Backdoor Attacks With Respect to
0125%
2.5%
Muric
17
ASR
0
ASR
0
ASR
0
ASR
ASR
SteeperNets
E.1%
0.4%
95.1%
1.8%
100%
1.0%
100%
0.0%
100%
0.0%%
99.9%
0.2%%
Tro DRL-W
0.0%
10.9
BadKL-M
3.0%
2.8%
29.0%
73.0%
1.1%
budget
0.1%
0.23%
0.3%
1%
2.%
5%
Mezne
BRR
BKR
BRR
NeesperNets
59.9%
0.0%
98.8%
1.8%
50.1%
0.3%
96.35%
7.5%
95.75
2.25%
77.4%
36.3%
TryDRL W
100%
0.0%
100%
0.2%
100%
0.0%
100%
0.0%
99.4%
0.7%
93.7%
6.6%
BadKL--M
98.9%
2.6%
12.2%
Table 8: Numerical results from the ablation with respect B. Each result is given with its respective
standard deviation in the neighboring column. The best result in each column is placed in bold.
23

---

9.8 Computational Resources Used
All experiments run in this paper are relatively low cost in terms of computational complexity many finish within few hours running CPU alone. For this paper all experiments were split between multiple machines including a desktop. laptop, and CPU server. Their specs are summarized in Table
Machines Used in Experimental Results
Machine
CPU
GPU
RAM
Laptop
19- 1991290
RTX A2000
32GB
Desktop
Threadripper PRO 5955WX
RTX 4090
128GB
Server Intel Xeon Silver 4114
None
128GB
Table 9: Summary of machines used for the experiments in this paper.
9.9 Related Work
In this section we discuss two alternate threat vectors against DRL algorithms- policy replacement attacks and evasion attacks. Note that "policy replacement attacks" is a term we are adopting for clarity, in the existing literature they are referred to generically as "poisoning attacks" (30]. We
believe the term "policy replacement" is appropriate as will become evident. policy replacement attacks the adversary's objective is to poison the agent's 's training algorithm such that they learn specific adversarial policy n S + A. Note that these attacks, unlike backdoor attacks, not alter the agent's state observations at training or deployment time as they have no goals of "triggering" target action. They instead only alter the agent's 's reward and or actions during each episode. Policy replacement attacks were extensively and formally studied in (30 and 299
Evasion attacks, the other hand, use either black-box (22) 2 white-box 21 access the fully trained agent's underlying function approximator to craft subtle perturbations to the agent's ' state observations with the goal of inducing undesirable actions or behavior. These attacks have proven highly effective in DRL domains with multiple different observation types, such proprioceptive [ and image-based observations [99. These attacks have few main drawbacks. First, they incur high levels of computational cost at inference time the adversary must solve complex, multi-step optimization problems order to craft their adversarial examples 2 21]. Second, applying same adversarial noise to different observation often does not achieve the same attack success, requiring a full re-computation on a per-observation basis. Lastly, many adversarial examples are highly complex and precise, and thus require direct access to the agent's 's sensory equipment in order to effectively implement (28).
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
In abstract and introduction claim: theoretical analysis proving prior methods insufficient, new framework with provable performance, a novel threat model, a algorithm using our theoretical results, and detailed empirical analysis of our method in comparison to prior works in multiple domains. These topics are claims are explored and justified in detail in Section Section 4.4 Section 3.1 Section 5 and Section (iciion/@resspee respectivvy
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are given in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
25

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions for our theory are given in Section 4.2 along with assumptions of
our threat model in Section 3.1 Proofs are given in Appendix Section 9.1 and Section 9.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation details are given in Appendix Section 9,4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
26

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code is attached to the paper submission and provided anonymously here
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/ gublic//uides/Codeee www.e/euumisiiionooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https)
/ haappp..cc/pllllcggiiiee////eedd htw.:iddeee/ooeeuuunissiiiooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Implementation details are given in Appendix Section 9.4 Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
27

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
Standard deviation shown on all plots and given all tables in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computational resources are summarized in Appendix 9.8
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp/////aarrrgp...//////aaaaa
28

---

Answer: [Yes]
The paper conforms with the NeurIPS code of ethics. There are no data or privacy concerns generated by this paper. Additionally, broader impacts are considered in Section g
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are considered in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper does not release any data or models that have a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
29

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All environments are used under MIT or Apache license with proper credit and references given throughout the main body of the paper and in Appendix Section 9.4
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer[[NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
30

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "CvGHmEQwnd", "reviewer": "Reviewer_VsP1", "summary": "The authors introduce a novel framework for backdoor poisoning RL agents, SleeperNets. SleeperNets assumes that adversaries can inject adversarial perturbations into the agent's observations during policy training within some total budget. Unlike in prior frameworks, the adversary implements its attacks post-hoc on full episode rollouts. The authors show that their attack manages to be successful, while retaining the performance of the optimal unpoisoned setting. The authors implement their novel framework on four different environments and show that it works favourably.", "questions": "* you are not referencing adversarial cheap talk [3] - can you compare and contrast their setting against yours?\n\n* you are mentioning a adversarial perturbation budget - where does this budget come from, and why would a budget be justified in reality rather than say a constraint based on information-theoretic detectability as in [1]?\n\n[3] Lu et al., Adversarial Cheap Talk, ICML 2023", "limitations": "The authors do not further investigate defenses against their novel attack, although they state that securing the training environment as well as developing test-time anomaly detectors would be suitable avenues.\n\n## Update in Response to the Rebuttal\n\nThe reviewer have successfully addressed my concerns, I therefore now recommend the paper for acceptance.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has a number of strengths. First of all, I believe that the threat model innovations are sensible; it seems natural that the adversary could manipulate whole episodes and not just single steps. Equivalently, interpreting stealth as retaining policy performance seems sensible.\n\nThe authors' insight that \"dynamic\" attacks can attain both success and performance while \"static\" attacks cannot is insightful. The empirical results seem to support the author's claims.", "weaknesses": "* test-time defenders can still perform anomaly detection based on observations, actions or state transitions; I do think the authors should perform empirical investigations using out-of-distribution anomaly detection methods [2] to infer the information-theoretic detectability of their methods.\n\n* The idea of condition of increasing the adversaries' attack context beyond single steps is not entirely novel within the adversarial attack literature, see e.g. [1] who devise adversarial attacks that condition on the entire action-observation history.\n\n### minor weaknesses\nline 264: \"environment environment\"\n\n[1] Franzmeyer et al., Illusory Attacks, https://openreview.net/forum?id=F5dhGCdyYh, ICLR 2024 \n\n[2] Nasvytis et al., DEXTER, https://arxiv.org/abs/2404.07099, AAMAS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Overview:** We thank the reviewer for their positive assessment of our work in noting the sensibility of our threat model the insight brought by our theoretical analysis. The reviewer also brought with them much insight in the form of constructive questions and citations. We thank the reviewer for these comments and aim to respond to all of them  thoroughly. Given our in-depth responses, we encourage the reviewer to consider increasing their rating of our paper, or to engage in further discussion with us over the next week.\n\n**Comment 1:** “The authors implement their novel framework on four different environments and show that it works favourably.”\n\n**Response:** We thank the reviewer for their positive comments about our framework’s novelty and the favorability of our results. We would like to make a minor correction - we performed experiments over **6 different environments** which we then classified into 4 generic categories.  \n\n**Comment 2:** “The idea of increasing the adversaries' attack context beyond single steps is not entirely novel within the adversarial attack literature, see e.g. [1]...”\n\n**Response:** We thank the reviewer for bringing [1] to our attention as it takes an approach to avoiding test-time detection which is very unique. We will certainly include it in an extended related work if our paper is accepted. That being said, it considers both a threat model and attack type - test-time evasion attacks - that are completely different from ours. While there are some similarities in giving the adversary access to more temporal information, our contribution of the outer-loop threat model is sufficiently novel and important to the study of backdoor attacks against DRL.\n\nOur paper is the first to introduce and formalize the outer-loop threat model for poisoning attack in RL - even when considering both backdoor and policy replacement attacks. We show that the outer loop threat model is not only viable and realistic, but allows for more powerful and dangerous attacks compared to the inner loop threat model. This necessitates further research into not only the capabilities and applications of the outer loop threat model, but also prevention against it. \n\n**Comment 3:** “you are not referencing adversarial cheap talk [3] - can you compare and contrast their setting against yours?”\n\n**Response:** We thank the reviewer for bringing up this work and giving us an opportunity for further reflection. We will definitely include it in an extended related work upon acceptance of our paper as an alternative class of poisoning attacks. \n\nIn terms of threat model the papers are quite different. In [3] it is assumed the adversary can use an open “cheap talk” channel to send the agent “messages” **at each time step** during training and testing. The goal of each message is to influence the agent’s training behavior and potentially control them at test time. These messages are unique per state, requiring the adversary to learn a message function.\n\nIn SleeperNets we instead allow the trigger to exist within the agent’s latent observation space - making no critical assumptions about the MDP. We additionally achieve state of the art results while poisoning less than 0.5% of the agent’s observations, while in [3] they use an effective poisoning rate of 100%. Lastly, in SleeperNets the adversary is allowed to manipulate the agent’s reward signal while [3] only perturbs the agent’s observations. \n\nWe look forward to any future works attempting to combine the two attack methodologies or settings.\n\n**Comment 4:** “you are mentioning an adversarial perturbation budget - where does this budget come from?”\n\n**Response:** The concept of a poisoning budget (alternatively called the poisoning rate) is standard throughout the poisoning attack literature in both RL [4] and supervised learning [5]. There are two key reasons for its usage: we want to minimize the likelihood that the perturbations are detected at training time - which will increase with our poisoning budget; and we want the agent to still learn the benign MDP - which empirically becomes more difficult as one increases the poisoning budget (see our ablations in section 6.3). \n\n**Multiple Comments: Test Time Anomaly Detection via Information Theory**\ne.g. “...I do think the authors should perform empirical investigations using out-of-distribution anomaly detection methods [2] to infer the information-theoretic detectability of their methods.”\n\n**Response:** We thank the reviewer for bringing up [2] as it is an insightful work for the detection of test-time anomalies. We will certainly include a citation of this paper in an extended related work if our paper is accepted. We agree with the reviewer that the test-time detection of backdoor attacks is a critical and open problem in this field, but we believe it is orthogonal to the objectives of our paper.\n\nIn our paper we answer the fundamental question “What poisoning methods are necessary for successful RL backdoor attacks?”. We not only prove the insufficiency of static reward poisoning, but we also develop a novel, dynamic reward poisoning strategy and rigorously prove its sufficiency in achieving both attack success and stealth - becoming the first backdoor attack paper in RL to produce such concrete results. Through this we propose a versatile attack framework which makes no critical assumptions about the attacker’s test time objectives. This allows future poisoning attack literature in RL to build upon our theoretically rigorous foundations and technically rich contributions when they aim to solve additional adversarial objectives, such as minimizing the detectability of test-time anomalies in the agent’s observations or actions.\n\n[4] Kiourti et. al., TrojDRL: Trojan Attacks on Deep Reinforcement Learning Agents. ACM/IEEE Design Automation Conference (DAC), 2020\n\n[5] Shafahi et. al., Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. NIPS 2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JC19Qba4MM", "reviewer": "Reviewer_YkW9", "summary": "The SleepNets paper considers a new (\"outer loop\") threat model, more powerful than those typically considered in adversarial RL settings.\nThe authors consider a stealthy attacker, aiming to both be successful (essentially tricking the learner into believing the underlying MDP is instead one of the attacker's choosing) and remain hidden (keeping the values of the original and corrupted policies similar).\nThey provide theoretical results on the limits of the traditional, weaker threat model, and introduce a \"Dynamic Reward Poisoning Attack Formulation\".\nThis yields their new attack \"SleeperNets\", which they empirically evaluate.", "questions": "Same as the weakness described above -- in what real world settings is the attacker powerful enough to perform the sleepernets attack, but not so powerful as to directly manipulate memory values on the training machine?", "limitations": "The primary limitation is the weakness described above.\nThe other limitations are described in Section 7, and I agree with the authors that they are interesting areas for future work.\nThis paper stands without a deeper investigation of them.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper has an explicitly stated threat model -- a welcome sight in this area.\n2. The paper provides a theoretical investigation of prior threat models, with an ultimately simple example demonstrating an impossibility result.\n3. While it is too far out of my area for me to be sure of the coverage of related work, the paper does seem to well-situate its contributions in the broader body of literature.\n4. There is a broad but not overly cumbersome set of empirical analyses.\n5. The paper is well written and easy to follow.", "weaknesses": "The main weakness I see is the applicability of the threat model.\nAs the others state, the adversary is assumed to infiltrate the computer on which the agent is training.\nIt's not clear to me what scenarios would exist where an attacker has that much access and can't perform a far deadlier attack (simply manipulating values directly).\nThe paper would be improved if the authors gave examples of real settings where an attacker could act in this outer-loop way without having direct software access.\nI do believe such examples exist, they just need to be articulated.\nThat is, described in detail with specifications about how Algorithm 1 could still be executed (and the assumptions about e.g., \\beta hold).\n\nSome rough ideas for such settings:\n1. The RL agent is acting on financial markets and the attacker is able to manipulate the reward signal by directly purchasing shares at an inflated cost from the agent.\n2. The RL agent is acting in a physical environment and the attacker is able to manipulate that same environment (I'm picturing how humans train drug sniffing dogs by hiding toys).\n3. The RL agent is flying, and the attacker has limited access to some of its instrumentation (e.g., can spoof its GPS location or jam certain signals).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Overview:** We thank the reviewer for their positive assessment of our paper and for their insightful questions. The reviewer’s main concern was the feasibility of the outer loop threat model. This is something we’ve taken much time to consider over the course of writing our paper, thus we will give the reviewer a thorough response.\n\nUpon reading our in-depth responses to all their questions, we encourage the reviewer to consider increasing their rating of our paper, or to engage in further discussion with us over the next week.\n\n**Comment 1:** “It's not clear to me what scenarios would exist where an attacker has that much access and can't perform a far deadlier attack (simply manipulating values directly).”\n\n**Response:** Here we focus particularly on the hypothetical, “far deadlier attack”’. First, we would like to ask the reviewer how they envision this attack? We lack a direct answer at this time, but encourage the reviewer to discuss this point further with us as it will be enlightening for all parties. For this response we will be assuming that the adversary has full control over the agent’s training machine and can arbitrarily manipulate RAM values. We will then push back on the notion that a “far deadlier attack” is either possible or sensible.\n \nWhen studying poisoning attacks we assume the adversary wishes to exploit the agent at test time, thus the agent must perform well enough in the base MDP to reach a deployment phase. In other words, the adversary must maintain “attack stealth” (section 3). This puts heavy restrictions on the adversary as they cannot simply replace the agent’s policy with one of their own design without training a model themselves. They must have full access to the victim’s MDP as well as the necessary compute resources to train the agent (which is hard and expensive to obtain). Accessing the MDP may be easy if it is simulated, but would be extremely difficult and costly in the case of real-world training environments, such as self-driving or robotic applications.\n\nTherefore, the most sensible option for the adversary is to implement a versatile attack like SleeperNets. In our work we show that extra information and domain specific engineering is largely unnecessary for a successful attack. So long as the adversary knows the agent’s observation space and can devise a trigger pattern they will be able to perform a successful and stealthy attack. Thus, we assert that the SleeperNet attack is a versatile and reliable approach for attackers of different capabilities.\n\n**Comment 2:** “The paper would be improved if the authors gave examples of real settings where an attacker could act in this outer-loop way without having direct software access…”\n\n**Response:** We thank the reviewer for this question since it is very fundamental to our paper. We additionally thank them for the inspirational examples they include in their comment. This question covers a very important topic, so we intend to answer it thoroughly and from multiple angles.\n\n**Viability of Direct Software Access:** We would like to first push back against the assertion that direct software access is unreasonable. There are countless, well documented cases of advanced persistent threats (APTs) achieving such levels of access in real-world settings. In fact, the most recent Verizon Data Breach Investigation (DBIR) report mentioned that in 2023 there were more than 5000 breaches with system intrusions, and these numbers are growing at a fast rate year-to-year. There are also infamous cases such as the RSA breach which prove that critical assets, such as model training servers, are not safe from adversarial attacks.\n\n**Applications of the Outer Loop Attack:** One nice feature of the outer loop threat model, in contrast to the inner loop, is that it allows for more direct translations between offline and online reinforcement learning. When attacking offline RL we can directly apply Algorithm 1 to the offline dataset intended to be used for training. The only difference is that, on line 2, we would sample H from the fixed dataset rather than from the MDP.\n\nSimilarly, in many domains agents are trained with pseudo-MDPs created from existing real-world data. For instance, companies designing stock trading agents utilize real-world market data to train their models. This database, just like any other, is subject to direct manipulation as the result of a breach - which we know, from the aforementioned DIBR report, is unfortunately common.\n\n**Malicious Trainers:** Often in adversarial machine learning we assume the innocence of the training entity and assert the adversary must be external, however this isn’t always the case. We are currently working with collaborators on one such scenario. Imagine a company who designs and optimizes 5G connectivity controllers using DRL. Internet Service Providers will purchase products with the highest customer satisfaction, generally measured by how fairly they distribute internet bandwidth. However, the designer of the controllers may want to give preferential treatment to particular services (streaming platforms, etc.). How can they convince the ISP that their controller is fair while also allowing for this preferential treatment?\n\nA powerful solution to this problem is for the trainer to perform a backdoor attack against their own model - a setting they have complete control over. Through this method the trainer can guarantee state of the art performance - leading to the purchase of their controllers - while also allowing for exploitation of the backdoor after deployment. This solution requires no special tuning of the training algorithm, the model weights, or the MDP - it works directly “out of the box”. This necessitates further study into the auditing of DRL agents and the test-time detection of backdoor attacks, which we believe are both exciting areas of future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hEnlhL90k9", "reviewer": "Reviewer_Es9J", "summary": "This paper proposes a new backdoor attack against Reinforcement Learning, termed SleeperNets. SleeperNets adopted dynamic reward poisoning to overcome the insufficiency of static reward poisoning proposed in previous works. The author provided a theoretical analysis of the advantages of dynamic adversarial poisoning and also conducted comprehensive evaluations to demonstrate the effectiveness of SleeperNets over previous backdoor attacks against DRL.", "questions": "NA", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper shows the drawback of static reward poisoning adopted in the previous DRL backdoor attacks, which motivates dynamic reward poisoning. \n* The authors provide a theoretical analysis of the returned reward given the design of the dynamic reward, convincingly show that dynamic reward poisoning overcomes the drawbacks of static design.\n* Comprehensive evaluations including detailed ablation studies are conducted.", "weaknesses": "NA", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Overview:** We thank the reviewer for their strongly positive assessment of our paper. We appreciate them for acknowledging the many theoretical and technical contributions of our paper such as showing “the drawback of static reward poisoning”, providing “a theoretical analysis of the advantages of dynamic adversarial poisoning”, and in conducting “a comprehensive evaluation to demonstrate the effectiveness of SleeperNets”.\n\nAlthough the reviewer provided no weaknesses or questions for our paper, we encourage them to bring up any further questions they may have during the author discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to first extend our thanks to the reviewers for their time in reading our paper, evaluating its merits, and highlighting its novel contributions. We hope that our extensive responses have sufficiently answered all the reviewers’ questions, but openly invite any further questions or comments during the author discussion period. 

We are further grateful for the reviewers’ enlightening comments and feedback. Reviewer VsP1 was primarily concerned with the test time detectability of our attack against information theoretic defenders, citing multiple related works. In our response we highlight the versatility of our attack framework and its applicability to these additional, adversarial objectives. We further note the relevance of their cited papers and aim to include them in an extended related work.

Reviewer YkW9 asked questions about the viability of our threat model in the real world, additionally including some insightful attack scenarios. We answer these questions thoroughly and from multiple angles. We first establish the utility of our attack for both strong and weak adversaries, highlighting the difficulty and extra cost of implementing a “far deadlier attack” without being detected. We then provide multiple use cases of the outer loop attack and motivate its positioning within both the literature and real world settings.

Reviewer Es9J was highly positive of our work, noting their confidence in both our theoretical contributions and our empirical evaluation. We greatly appreciate this praise and openly invite the reviewer to engage with us in further discussion during the author rebuttal period if they have any additional questions or comments.

Finally, when writing this paper we aimed to take a theoretically rigorous approach towards filling crucial gaps in our understanding of backdoor attacks in reinforcement learning. Through this exploration we not only prove the insufficiency of prior approaches, but make key contributions in developing the first backdoor attack framework to provably maximize both attack success and attack stealth. We believe these theoretical developments, in addition to our novel SleeperNets attack and outer loop threat model, will be a crucial foundation for future works studying both backdoor attacks and defenses in reinforcement learning.

Meta-review: The paper proposes a novel universal backdoor poisoning attack.  The reviewers appreciated the novelty and technical contribution which combines theoretical insights with a practical algorithmic approach that is shown to be highly effective in experiments.
Decision: Accept (poster)