Beware Untrusted Simulators -- Reward-Free Backdoor Attacks in Reinforcement Learning
Authors: Ethan Rathbun
Wo Wei Lin
Alina Oprea
Christopher Amato
Venue: ICLR 2026 Poster (2026)
Keywords: Poisoning Attacks
Backdoor Attacks
Reinforcement Learning
Deep Reinforcement Learning
Robotics
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: The first backdoor attack against DRL which does not alter or observe the agent's rewards.

Abstract: Simulated environments are a key piece in the success of Reinforcement Learning (RL), allowing practitioners and researchers to train decision making agents without running expensive experiments on real hardware. Simulators remain a security blind spot, however, enabling adversarial developers to alter the dynamics of their released simulators for malicious purposes. Therefore, in this work we highlight a novel threat, demonstrating how simulator dynamics can be exploited to stealthily implant action-level backdoors into RL agents. The backdoor then allows an adversary to reliably activate targeted actions in an agent  upon observing a predefined "trigger", leading to potentially dangerous consequences. Traditional backdoor attacks are limited in their strong threat models, assuming the adversary has near full control over an agent's training pipeline, enabling them to both alter and observe  agent's rewards. As these assumptions are infeasible to implement within a simulator, we propose a new attack "Daze" which is able to reliably and stealthily implant backdoors into RL agents trained for real world tasks without altering or even observing their rewards. We provide formal proof of Daze's effectiveness in guaranteeing attack success across general RL tasks along with extensive empirical evaluations on both discrete and continuous action space domains. We additionally provide the first example of RL backdoor attacks transferring  to real, robotic hardware. These developments motivate further research into securing all components of the RL training pipeline to prevent malicious attacks.

Published as a conference paper at ICLR 2026
BEWARE UNTRUSTED SIMULATORS - -REAARD-FREB BACKDOOR ATTACKS IN REINFORCEMENT LEARNING
Ethan Rathbun;" Wo Wei Lin, Alina Oprea, Christopher Amato Khoury College of Computer Sciences, Northeastern University
ABSTRACT
Simulated environments key piece in the success of Reinforcement Learn- ing (RL), allowing practitioners and researchers to train decision making agents without running expensive experiments on real hardware. Simulators remain se- curity blind spot, however, enabling adversarial developers to alter the dynamics of their released simulators for malicious purposes. Therefore, in this work we highlight a novel threat, demonstrating how simulator dynamics can be exploited to stealthily implant action-leva level backdoors into RL agents. The backdoor then al- lows an adversary to reliably activate targeted actions in an agent upon observing a predefined "trigger", leading to potentially dangerous consequences. Traditional backdoor attacks are limited in their strong threat models, assuming the adversary has near full control over an agent's training pipeline, enabling them both alter and observe agent's ' rewards. As these assumptions are infeasible to implement within a simulator, we propose a new attack "Daze" which is able to reliably and stealthily implant backdoors into RL agents trained for real world tasks without altering or even observing their rewards. We provide formal proof of Daze's 's effec- tiveness in guaranteeing attack success across general RL tasks along with exten- sive empirical evaluations on both discrete and continuous action space domains. We additionally provide the first example of RL backdoor attacks transferring real, robotic hardware. These developments motivate further research into secur- ing all components of the RL training pipeline to prevent malicious attacks.
INTRODUCTION
Simulators are a core component of reinforcement learning (RL) and robotics (Makoviychuk et al.) 2021 Coumans & Bail 2016-2021 Todorov et al., 2012). enabling RL approaches for dexterous object manipulation (O et al. 2018). humanoid robotic control (Xue et al. (025), and au- tonomous navigation (Hu et al. 2021). Leveraging simulated interactions, either alone or in combi- nation with real-world data, agents are able to gain task experience in a manner that is both scalable and safe (Da et al. 2025). Therefore, simulators have become a trusted pillar of applied reinforce- ment learning, with practitioners leveraging external simulators cloud computing services train their agents (Bhide et al. (025).
This trust comes with cost, however, it leaves the door open for malicious developers or cloud computing providers to influence the learned behavior of RL policies. Particularly threatening are backdoor attacks in which an agent's states, transitions, and rewards are manipulated at training time such that they exhibit pre-defined ad adversarial behavior upon observing a fixed trigger during deployment (Kiourti et al. 2019 Ma et al. 20255] Rathbun et al. 2025 2024). The backdoor trigger then grants an adversary control over the agent at test time, enabling destructive or biased behavior from an otherwise well performnng agent. Traditional approaches rely on unrealistically invasive as- sumptions, however, granting the adversary complete control over the training pipeline (Wang et al.) 2 or requiring direct, RAM-level access to the training machine (Rathbun et al., 2024). These assumptions leave large gap in our understanding of backdoor attacks under more realistic con- straints. Therefore, in this work we uncover a previously unexplored threat- malicious developers implanting backdoors into agents by subtly altering the mechanics of their simulator. Such attack is impossible for traditional backdoor methods to achieve, however, as many simulators, such as the
Primary Contact Author, rathhun.co northeastern.cdi

---

Published as a conference paper at ICLR 2026
Penign Envirormentt I:igget
POU AODIN
INLLIOy Apuzt
agant avatd colliatonn
CroSsea the Leteronctioo Bu0000o?
Teiggnzed Rnvisomaant
Epp Acwett
Tesgpec
Ploed vollory
2int-nooa ierecing cuilldde dloed Falloy sgeet.
trigowr
Figure 1: Example of an agent, poisoned by our proposed "Daze" attack, operating in our Turtlebot- based "Intersection" task. In the benign case (top) the ego agent successfully avoids colliding with the fixed-policy agent and crosses the intersection, while in the triggered case (bottom) the agent follows the adversary's target action "accelerate", subsequently crashing and failing the task.
commonly used MuloCo (Todorov et al., 2012) and Pybullet (Coumans & Bai 2016-2022 robotics simulators, only receive actions and return environment states, while rewards are computed exter- nally. Therefore this scenario, while realistic, violates critical assumption of all prior backdoor attacks - the adversary can no longer alter or even observe the agent's rewards.
Without altering rewards, prior backdoor attacks unable to sufficiently bias the agent's expected returns and, without observing rewards, they can no longer Iearn or leverage task specific informa- tion forcing the attack approach to be universal. Therefore, this realistic and highly constrained threat model requires the development of a new backdoor attack approach. In this work we propose such backdoor attack, Daze, which is able achieve better equal performance to traditional attacks without altering or observing the agent's rewards. We further demonstrate the capabilities of Daze in real world robotic applications, providing the first example of backdoor attacks being successfully triggered on robotic hardware, as shown in Figure In summary, we:
1. Propose a novel threat model targeting the RL supply chain via malicious simulators.
2. Design "Daze", novel backdoor attack against RL which achieves theoretical guarantees attack success without altering or observing rewards.
3. Perform extensive evaluations of the Daze attack in both continuous action space MuJoCo and discrete action space Atari domains. We demonstrate improved performance with Daze over traditional backdoor attacks despite operating under more restrained threat model.
4. Demonstrate the first example of backdoor attacks transferring from simulators to real robotic hardware.
BACKGROUND
In this section we formally define the problem setting, threat model, and adversarial objectives of our attack. We consider two parties: the victim training an RL agent, and the adversary manipulating the agent's data for malicious purposes. The victim trains agent to maximize cumulative rewards in Markov Decision Processes (MDP), typically defined as the tuple M (S. A, R. T) where the state space S C S is subset of larger space (e.g. set of all 16 dimensional lidar scans), A is a set of actions, R : S x + IR is a reward function, T x A x - AS represents transition probabilities between states given actions, and discounts the value of rewards over time (Sutton & Barto 2018). The behavior of an agent is determined by their policy क S > AAA which represents probability distribution over actions given states. In this work we consider MDPs with both discrete and continuous action and state spaces A. We further consider attacks against Deep Reinforcement Learning (DRL) algorithms like Proximal Policy Optimization (PPO) (Schulman et al. 2017), Twin Delayed Deep Deterministic policy gradient (TD3) (Fujimoto et al., (018)) and Deep Q-Networks (DQN) (Mnih al. 2013) they represent the current state of the art in RL.
2

---

Published as a conference paper at ICLR 2026
2.1 BACKDOOR ATTACK FORMULATION AND OBJECTIVES
The adversary's 's goal, on the other hand, is to alter the training of the agent's policy such that they effectively learn to solve a malicious MDP M' = (S U So, A, I²,, T', T33,7) which uses the same S, A, as the benign MDP M specified by the victim. Here Sş is set of "triggered states" defined as the image of a trigger function 8 S - S over all & E S. In line with prior work (Rathbun et al. 2024 å must be designed such that Sş is disjoint with S. Additionally. R' is an adversarial reward function which alters R to manipulate the agent's rewards into and out of triggered states. In our threat model, defined later, the adversary will not be able to alter rewards. Lastly, T SUSS x A - AS U Sg represent an adversarial transition function which alters the transition dynamics of T, making the agent enter triggered states with probability 3 and potentially manipulating transitions in triggered states. Through the 3 parameter we restrict the poisoning rate of the adversary (Jagielski al. 2021). representing the proportion of states in which the trigger appears in M'. In practice is a hyper parameter chosen by the adversary, with lower values being stealthier and more desirable.
The goal of backdoor attacks is twofold. First, the trained agent should exhibit some pre-defined adversarial behavior whenever they enter triggered states 88 Sg.. Similar to prior works (Kiourti al. 2019 Rathbun et al.) (025) we study action level attacks which aim to elicit fixed action a E A from the agents trained on M' upon entering triggered state 55. We refer to this objective
as "attack success" and formalize it across continuous and discrete action spaces as follows:
Attack Success: min min(Bseu((s)) [Cadt (TT (6(s)), a*)]
where ((()) forms a uniform distribution over S. Ludv is an error function defined by the adversary, and x- is the victim agent's ' poisoned policy optimized over M'. In both continuous and discrete action space MDPs we can view backdoor attacks as inducing shift in the agent's s policy xt in triggered states such that it matches a desired distribution. In discrete domains it is standard to define 1, which results in the error function Lady (r+ (6(x),, at) no longer induce
this distribution as π (a [8(s))
1-x* (a d(s) (Kiourti et al. 2019) In continuous domains the adversary an exact action so, in line with prior works (Mact al. (025), we aim to instead minimize the expected distance between the target action at and actions sampled from the agent's s policy a ~ (((())) For our purposes we found the loo norm to be the most effective and easy to interpret distance metric. In summary the adversary aims minimize the following error functions over x given all E S:
Action Space L (6())),at)
Continuous
Discrete
(2)
E Bawaa+ (6(e)lllt - all (aax)-
While aiming to minimize attack error the adversary must also remain stealthy. In this work, along with many others in the established literature (Rathbun et al. 2025 Kiourti et al. 2019). we consider an attack to be stealthy if the backdoor poisoned policy x trained on M' performs similarly to an unpoisoned policy n trained using the same DRL algorithm A. We formally define this as:
Attack Stealth:
min [S) x
(s) - V.M (s)
(3)
where - A(M') and π ~ A(M). Once attack stealth is optimized, x will perform as expected in M, making the attack nearly undetectable at test time until the trigger is introduced. Furthermore, poisoned policy which also optimizes attack success will take action at (or near at in continuous domains) in any triggered state d(s) despite any severe consequences.
2.2 THREAT MODEL.
Simulators are an integral and trusted component of the DRL supply chain. In this work, we propose and explore novel threat model, visualized in Figure which challenges this trust. In particular, we study an adversary who develops and releases their own RL environment simulator with the goal of implanting a backdoor agents trained on it it- altering superficial state information returned to the agent (e.g. pixel values) along with transition dynamics T. For this to be feasible the adversary must develop an approach which is simultaneously effective, general, and stealthy. Specifically, since simulators are designed to be general purpose within their respective area (e.g. robotics), the attacker cannot make any assumptions about the victim's 's targeted task, their approach must generalize across any base task. The attack must also be stealthy, requiring perturbations to be infrequent and subtle to evade detection. In particular, agents trained in the simulator should be able to learn their intended task and, importantly, transfer their policies real world settings.
3

---

Published as a conference paper at ICLR 2026
DRL Training Victim Using Maliclous Simulator
Malicious Simulaaoor Development
Adversarial Area of Control
Open Closed Rolease
State
Roward
AI L Lo
Action
Benign Simulator T(s.a.)))
Malicious Developer(s)
Malicious Simudatorr T((JAa))
Victim Function R(s.a.5)
Victim Policy
Figure 2: Visualization of our threat model. Before training, malicious developers first create an ac- curate, benign simulator and then implant the Daze attack within it before release. During training the malicious simulator only alters how some actions are interpreted or executed in the environ- ment, impacting the state. The base simulator mechanics (e.g- physics and lighting) remain unchanged from the benign simulator. To make the attack as versatile as possible we assume no access to the victim's reward function, which may be computed externally using state information from the simulator. The malicious simulator merely receives actions and returns the next state.
Furthermore, in many RL implementations rewards are computed externally to the simulator by the victim, preventing the adversary from obtaining either read or write access to the agent's 's rewards, referring to the ability to observe or alter rewards, respectively. This goes against the traditional requirements for backdoor attacks as both read and write access to rewards have been necessary for traditional backdoor attacks to be effective. With write access, attacks like TrojDRL (Kiourti al., 2019) and SleeperNets (Rathbun et al.) 2024) can bias the agent's 's expected returns to make at an optimal action and, perhaps more importantly with read access attacks like Q-Incept (Rathbun) ett al. 2025) can determine which actions taken by the agent resulted in high or low returns. While seemingly fundamental
backdoor attacks in DRL, in this work we prove that, both in theory and in practice, neither read nor write access are necessary to mount a successful attack, enabling successful backdoor attacks under this restricted threat model.
Additionally, following the terminology proposed in prior work (Rathbun et al., 2024). malicious simulators represent an "inner loop" threat model where the adversary must perturb the agent's data during the environment interaction loop. This requires the adversary to determine their perturbations online without observing the outcomes of future transitions. The difficulty of the inner loop threat model has lead more recent attacks, like SleeperNets and Q-Incept, to focus on an "outer loop" ap- proach where the adversary has direct access to the agent's 's training data through their replay buffer. While this gives the adversary more global view of the agent's training trajectories, it necessitates a more pervasive adversary, often requiring malware with RAM level access to be downloaded on the victim's machine. Therefore, through the study of adversarial simulators, this work explores novel, inner-loop threat model which is both greatly constrained and realistic - motivating further research into protections against malicious simulators and backdoor attacks in DRL.
3 METHODOLOGY
In this section we present our novel attack, Daze, which achieves theoretical guarantees of attack success and stealth without altering observing the agent's 's rewards. We achieve this by formalizing the attack as a malicious MDP which, when optimized by the agent, both solves the benign task and optimizes the adversary's attack objectives. Following the formalization and subsequent theoretical results, we present a practical algorithm for applying the Daze attack to any target task.
3.1 REWARD FREE AND UNIVERSAL BACKDOOR ATTACKS
Unlike previous attacks, under our strict threat model the adversary can no longer exploit task- specific information by reading the agent's rewards. Therefore we must leverage characteristics that are universal across the class of MDPs M. Specifically, in nearly all tasks researchers and practitioners are interested in, is sub-optimal for the agent to sample actions uniformly at random. We can use this universal sub-optimality to punish agents when they fail to take the target action a in triggered states, leaving their behavior unaltered otherwise. Formally, we observe that for any MDP M E MI, state and policy over M, there always exists an action E A with value greater or equal to that of an action sampled uniformly at random a' (A) in expectation. Formally:
Q (s,a) 2 (e Q. (s,
4

---

Published as a conference paper at ICLR 2026
Sconario 1: Agent ignores Target (Forwerd)
Sconario Agend Follows Target
Enters Initial, Triggared Stote
(oonard)
San
Chatsel Acdis
Chisars Adlinn
Drake
Permert
Teealian
Temerdes Awrien
Palto
((A)
π
The Agent esset Choose Betwron Target Action (Forward) Any Other Action
Outcome:
Agent Enters a Dised State and Crashes, Pallin Task
Outcome: Agent Returna to Berign State and Laber Completes the Task
Figure 3: Visualization the Daze attack on our intersection task during training. Upon entering a triggered state the agent choose follow the target action (Forward) or ignore it. If they ignore the target action, as in scenario I, they enter a dazed state, resulting in random transitions and low returns. If the agent follows the target action, in scenario 2, they first transition with respect to benign action a ~ Tagent( (8) then regain control over their transitions, receiving a higher return.
where Q (s, a) represents the expected value of action in state given policy I and MDP M (Sut- ton & Barto 2018). With this we now know that, unless every action has equal value in given state, there is always some action an agent can choose which performs better than sampling actions at ran- dom. We will refer to policies which perform better than random choice as "reasonable" and define them as follows for a given M:
(5) where as is the set of valid policies over M. Policies outside TIM are "unreasonable" in that they choose actions which perform equivalent to or worse than random sampling. It is worth noting that 11 can be empty in cases where a uniformly random policy is optimal in a subset of states. Therefore, make natural assumption over the victim's 's target MDP M, requiring that randomly sampled actions are never part of an optimal policy:
II { HI : V+M(s) > Eurru(A) (s, (s,a')}
(6) This assumption is fairly weak and only excludes small class of pathological MDPs for which random behavior is optimal while including nearly all problems practitioners and researchers are interested in solving. For instance, robotic agents are often highly sensitive to poorly chosen actions, making a uniformly random strategy highly undesirable.
Assumption VT & nı Ir' E n1," : V VV VAA (s)
This behavior is the backbone of the Daze attack which punishes agents for ignoring the target action at in triggered states S5 E Ss by forcing them to transition with respect to randomly sampled actions for multiple consecutive time steps. In this scenario, where the agent is forced to perform sub-optimally, we say the agent enters a "dazed state", as shown in the middle scenario of Figure 3 Similarly our formulation of triggered states,
define dazed states 8g E Sy the image of dazing function ó + S which adds a fixed perturbation to the state (e.g. gaussian noise) such that S U S. 0. Once in dazed state agents will transition back into benign state S with non-zero probability, otherwise transitioning to the next dazed state.
Conversely, when the agent follows the adversary desired behavior, choosing at in triggered states S6 8(s), they instead transition with respect to an action a ~ (s)) sampled given the benign state 8. In effect, the agent will transition as if they never entered triggered state to begin with, making it inconsequential to take the target action. Note that, given our restricted threat model, we cannot directly query the agent's policy, however we will show how to get around this in Section 3.3
Additionally, recall that our threat model assumes no control over the agent's rewards. Therefore, since d and & only introduce superficial alterations to states, all rewards will be computed with respect to true transitions that occur in M', including randomly sampled ones. This, in combination with the sub-optimality of random transitions in dazed states, ensures that the target action is strictly optimal for all triggered states in M'. Formally, define the daze attack below given poisoning rate parameter 3 E (0, 1). a daze strength parameter ps E [0, 1), and a threshold value 7 E IR:
where we use the shorthand s, to represent 8 (s,) or Φ -sx) for states in Sş and S₂ respectively. In practice the adversary never needs to compute this inverse as they receive the benign state S
5

---

Published as a conference paper at ICLR 2026
Adversarial Transition and Reward Functions T' and R¹
St
a
1t
Transition Probability
Reward
5
A
5
as
R(s,, a15,,,,10 S141)
S6
a(,,, $
a,
56
Lad
(r+ (
(8(s)).at) T
5
T(s
R(s; 7,8-)),((11
S6
Ladv (T (8(8), a
S.
T(x; T(=-,M(A [(()).
R(s; +(1))
5
A
Sp
T(s (((A) $141) Pp
R(s;" _((A), $441)
A
S
T(s; (((A),(()) 8141) (1-4)41111-----11111111111111111 ppp)
R(s, 8(41)
directly. Also note that here we are slightly abusing notation by inserting U(A) into T and for clarity, In reality U(A) enforces distribution over transitions and rewards.
Importantly, all transitions in the Daze attack are sampled with respect to the benign transition function T, therefore the attack does not modify the underlying dynamics of the simulator, it merely severs the relationship between actions chosen by the agent and those executed in the environment. Through this design the agent's objective of maximizing return in M' is directly correlated with the adversary's objectives of attack success in Equation I and attack stealth in Equation We formalize this result in the following subsection and provide proofs for all results in the Appendix.
3.2 THEORETICAL RESULTS
In this section we discuss the theoretical guarantees afforded to the adversary by the Daze attack. Through these results prove that a policy in adversarial MDP M' is optimal only if it also optimizes our objectives of attack success and attack stealth. Specifically, in regards to attack suc- cess, prove that if a policy is optimal in M' then it must have adversarial error of 0 in all triggered states ((s) E Sg. Formally, we define Theorem 1 as:
Theorem If π E IT is optimal in M' then (ad (=())= a+) for all S
This theorem is fairly intuitive given our explanation of the Daze attack in the previous section. If the agent takes the target action in triggered states they receive no consequences, while if they ignore the target action they are punished by transitioning with respect to a sub-optimal policy. Therefore it is optimal to always choose the target action in triggered states. Additionally, in regards to attack stealth, we prove that policy optimal in M' it optimizes attack stealth. Formally, we prove the following theorem:
Theorem 2 If nT IT is optimal in M' then n" is optimal in M for all S
Interestingly Theorem results nearly directly from Theorem 1. We know that an optimal policy in M' must choose π r" (ō()) the transition manipulations of the Daze attack. As result an optimal policy must solve version of M' which is essentially identical to M, meaning their behavior in benign states E S must also be optimal in M. Through these theorems we know that a policy is optimal in M' only it also optimizes our adversarial objectives of attack success in Equation and attack stealth in Equation Therefore, as DRL algorithms are designed to maximize returns and find an optimal policy. we can conclude that these algorithms will also result in policies which solve the adversary's 's objectives.
for all E S, therefore an optimal policy is never impacted by
3.3 DAZE IN PRACTICE
One fortunate consequence of Daze's design is that it lends itself well practical method that is both general and easy to implement. Specifically, the attack can be implemented as API wrapper around the base MDP M or simulator S, however practice the attack may need be hidden within normal functionality like domain randomization Tobin et al. (0077). The only requirement for the malicious simulator developer is to choose which trigger and daze functions best suit their test time purposes while minimizing training time detectability. For case of interpretability, in Algorithm I we present a practical implementation of Daze as an environment wrapper which is called each time the agent chooses action a, and looks sample the next state St-
The attack has three logic flows handling benign, triggered, and dazed states, respectively. Starting from benign state S, the wrapper mainly performs transitions according to the base simulator S. but eventually transitions to a triggered state with probability 3. Once in a triggered state 6(s), the wrapper receives the agent's chosen action and compute its error according Ludv- If the error is

---

Published as a conference paper at ICLR 2026
less than some threshold value T the wrapper will perform what call "null transition", returning the benign state & back to the agent without performing a simulation step. In this way we are able to effectively sample action than r the agent enters a dazed state ((s), where the true state s is sampled from the simulator with respect an action a' (()).. Once in dazed state, and in contrast to the theoretical MDP M' the agent deterministically returns to benign state after k Ladv (a, a time steps, scaling the number of steps proportional to the adversary's ' error and constant & z+. This design gives the agent more dense signal of the adversary's 's desired behavior in continuous domains without compromising our theoretical results in Section 3.2 Through this, Daze becomes an attack that is simultaneously lightweight, general, and highly effective, as we demonstrate empirically in Section
ITs) from the policy without query access. If the error is greater
Algorithm Daze Simulator Wrapper
Simulator S. poisoning rate 3, trigger func 6, daze func é. target action a* error func Lax. daze factor k daze threshold f Initialize Global: int daze 0, bool trigger = False, initial state st ~ S
Attack Parameters:
API Input: Action a, E A from Victim Agent
API Output: State & E S Ss U US
Routine Selection:
Routines are entered (left to right) according the outcome of their associated condition.
Benign Routine
Trigger Routine if trigger then trigger <- False if Lut (Ot, a+)======== S then
Daze Routine
if not (daze or trigger) then
if daze > 0 then
Sample 9, from S given
Sample a' ~ (((A))
if v 2( 11) < / then
Sample 8t from S given a' daze + daze -1
trigger <- True return ((s)
daze +- 0 return 8t
daze + [k Latv(a, a
if daze > return 6(s.)
return
Continue to Daze Routine
else return St
3.4 ON THE NECESSITY OF ASSUMPTION 1
Assumption defines a large and general subset of tasks for which our theoretical results are guar- anteed to hold, but it is not strictly necessary condition in practice. While some tasks may contain a subset of states for which Assumption is violated, e.g. if the agent has to choose hetween two equal length paths to their destination, our results in Theorem I and Theorem 2 are still likely to be maintained. In such a situation the agent will still enter subsequent states for which randomly sampled actions sub-optinal, meaning that ignoring the trigger and becoming "dazed" is also oot-oppimal. In short, our theoretical results still hold so long as the agent has non-zero probability of entering state for which sampling actions uniformly random sub-optimal behavior. In the following section we demonstrate this empirically as Daze proves effective across a wide range of tasks and domains despite guarantees of Assumption 1 holding.
EXPERIMENTAL RESULTS
In this section we evaluate the Daze attack against PPO-based (Schulman et al. 2017) agents trained across multiple environments spanning discrete and continuous action spaces. We additionally demonstrate the transferability of policies poisoned by Daze to real robotic hardware, resulting in hazardous behavior from otherwise well performing agents. In line with prior works (Rathbun et al.) 2025| Kiourti et al., 2019 Ma et al. 2025) we perform our evaluation in terms of Attack Success Rate (ASR) and Benign Return (BR), representng our objectives of attack success and attack stealth, respectively, defined below for = 0.2
ASR Discrete Env
ASR Continuous Eny
Benign Return
E (a= 8())) [xes[T a - (6()))]l S r]] Exg=ar [V (vo)]
We compare Daze against multiple state of the art backdoor attacks DRL leveraging different fundamental approaches which require read and write access to the agent's rewards. Specifically, TRL Kiourti et al., (2019) and Q-Incept (Rathbun et al., 2025) represent the state the art in methods which manipulate both actions and rewards while SleeperNets (Rathbun al.] 2024) the current state of the art in dynamic reward poisoning approaches. All these methods require much stronger and less realistic levels of adversarial access than Daze, yet demonstrate that Daze is able to match their performance in discrete action space environments and actually out-perform them in continuous action space environments in terms of attack success and attack stealth.
7

---

Published as a conference paper at ICLR 2026
4.1 SIMULATED CONTINUOUS ACTION SPACE EXPERIMENTS
In this section we compare Daze against SleeperNets and TrojDRL in continuous control environ- ments, choosing standard MuJoCo gymnasium tasks Todorov et al. (2012); Towers et al. (2024) as our testbed. We extend TrojDRL and SleeperNets to continuous action space domains by setting their reward poisoning constants, originally c 11 [a = at] in discrete domains, proportional to the lx distance between the target and chosen actions c- -|laa a+1 aall/.-. sc. We will therefore refer to these ex- tensions as SleeperNets-C and TrojDRL-C. Q-Incept is notably absent from these evaluations since extending its adversarial inception techniques, which take an arg-max over a learned Q-function, to continuous action space environments is non-trivial and requires its research and formulation. Additionally, since the goal of these experiments is to compare the capabilities of each attack, not the design of the trigger (Dai et al. 2025), use artificial indicator flag, appended to the end of the agent's 's state, to indicate triggered and dazed states. This properly upholds our assumptions of distinct triggered and dazed states. Lastly, in these experiments we design the target action to be vector of all -1 -1 since it results in failure across our chosen MuJoCo tasks, causing the agent to fall over or stop moving.
Environment Type
Simulated Environments- - MuloCo
Real World Environments
Environment
Half Cheetah
Hopper "All
Reacher
Intersection Accelerate
Waiter
Target Action
-A"I-1"
"AlI -1"
Accelerate+ Tuum
Metric
ASR
ASR
ASR
ASR
ASR
Daze
92.4%
1.1%
94.1%
2.05%
98.5%
0.1%
92.3%
6,4%
93.2%
6.2%
TrojDRL-C
20.3%
2.1%
25.6%
3.9%
65.5%
1.3%
65.5%
21.9%
5.4%
6.2%
SleeperNets-C
3,3%
1.9%
7.5%
3.2%
0.0%
0.0%
99.8%
0.1%
74.5%
49.4%
Metric
BR
0
BR
8
BR
0
BR
0
BR
0
Duze
1627.8
100.0
2321.5
344.8
-5.1
0.7
226,7
14.2
392.7
89.1
TrojDRL.CC
604.4
321.5
984,9
99.2
-15.8
1.5
223.8
13.9
375.4
130.5
SleeperNets-C
1511.3
91.5
1931.5
$35.0
-5.3
0.5
158.6
137.7
404.3
57.4
No Poisoning
1736.3
95.2
2085.4
287.0
-34
1.3
238.4
2.1
441.3
19.1
Table 1: Comparison of Daze against TrojDRL and SleeperNets against continuous action space, PPO agents in terms of Attack Success Rate and Benign Return. Average scores and standard deviations are computed over 100 trajectories and 5 training seeds in simulation.
In the middle three columns of Table ! we present the results of our comparison between Daze, SleeperNets-C C, and TrojDRL-C on three MuloCo tasks Half Cheetah, Hopper, and Reacher. All attacks were evaluated given a poisoning rate 3 = 1%, with Daze additionally using a "daze factor" see that Daze is the only attack capable of reliably
k 8. Across all three environments inducing the target action in triggered states, resulting in ASR values above 92% while seeing no drop in benign return. Additionally, since agents only enter dazed states when they ignore the target action, Daze only needs to alter a small portion of additional time steps to be successful. This results in only 0.3% of time steps being "dazed" in "Hopper", for instance. We discuss this in more detail in the Appendix, providing the proportion of dazed states for each environment.
Interestingly, SleeperNets-C and TrojDRL-C struggle in these environments despite their strong results in discrete domains (Rathbun et al. 2024 Kiourti et al., 2019), perhaps indicating that more novel extensions are necessary for reward poisoning attacks to work reliably in continuous domains. Daze, the other hand, achieves strong theoretical results for both discrete and continuous action spaces as we showed in Section 3.2 Therefore, despite the inherently stronger threat models of SleeperNets and Trojdrl, affording them both read and write access to the agent's 'ss rewards, Daze is able to out- out-perform both attacks in terms of ASR and benign return. In the Appendix we include additional experimental results- providing an ablation over Daze's 's poisoning rate / and daze factor k. and displaying the attack's 's versatility against off-policy algorithms like td3 (Fujimoto et al. 2018).
4.2 CONTINUOUS ACTION SPACE EXPERIMENTS ON REAL HARDWARE
In addition to demonstrating the effectiveness of Daze against simulated, continuous action space en- vironments, we further extend our study into policies executing on real, ROS based robots (Quigley) 2009 Diankov & Kuffner 2008). To this end we design two custom, image-based, continuous con- trol environments using both Turtlebor2 TUR 2012) and Fetch (Wise et al.) 2016) based agents. Our Turtlebot environment, "Intersection" is visualized in Figure [ while our Fetch environment, "Waiter", is shown in Figure 5 Both "Intersection" " and "Waiter" agents, which use depth camera images as observations, are tramed in custom Pybullet (Coumans & Bai designed to be nearly identical to their real world counterparts.
environments
8

---

Published as a conference paper at ICLR 2026
berign Thapered
Serign Tiggered
200
150
150
Requenor
125
73-
5
bi
-1.0
-0.5
0.0
0.5
-2.0
-0.5
0.0
Linea Velocity
Anguler Villcity
Figure 4: Distribution of linear (Left) and angular (Right) velocities recorded during runs of policy poisoned by Daze our real world "Intersection" task with and without the trigger.
In the rightmost two columns of Table we compare Daze again to SleeperNets-C and TrojDRL-C in our custom environments during the simulated training phase. Similarly to the MuJoCo results, we see that Daze is the only attack capable of achieving both high ASR and BR scores, despite never altering or observing rewards. the case of "Intersection". SleeperNets- higher ASR score than Daze, but this comes the cost of drop in benign return from 238.4 down to 158.6 6, making the attack casily detectable. Therefore, these results further prove the effectiveness of Daze against continuous action space domains even when realistic, image-based triggers are used.
able achieve a
We then transfer our agents, poisoned by Daze, onto real robotic hardware with physical depth camera sensors. Through these results, presented visually on "Intersection" in Figure [and "Waiter" in Figure 5 we demonstrate the first example (to the best our knowledge) of RL backdoor attacks successfully triggering harmful behavior from real, robotic agents. We can also this concretely in Figure where our "Intersection" agent, poisoned by Daze, exhibits a significant shift in behavior once the trigger in the environment following the target action by accelerating forward and almost never turning. This greatly heightens the importance of further research into the prevention and detection of backdoor attacks to mitigate their potentially harmful impacts. In the supplementary material, hosted anonymously here, we provide video demonstrations of our physical experiments. We also include further details for our hardware experiments the appendix.
Benign Eavi conment
No Trigger
Triggezed Env ronment
Ball del delivered table, Dask Success!
Agent crathes, and drops the ball, Dask Pailure!
Figure 5: Example of Daze poisoned agent in our Fetch- based "Waiter" task. In the benign case (left) the agent successfully delivers the red ball to the table, while triggered case (right) the agent immediately and sharply turns left crashing and dropping the ball.
4.3 SIMULATED DISCRETE ACTION SPACE EXPERIMENTS
In Table 2 to demonstrate the versatility of Daze in discrete action space domains, compare Daze against Q Incept. TrojDRL, and SleeperNets in discrete action space, Atari environments (Towers) al.) 2024). Following the design of prior work (Rathbun et al. 2025) we use 6x6 checkerboard pattern in the top left corner of game frames as the trigger. It is important to note that, due to the strength of the attacks in discrete action space environments, SleeperNets and Q-Incept act as upper bounds ASR performance when the adversary has full control over read and write access to the agent's rewards. this access is taken away, all three prior attacks will fail.
Across all four environments, Q*bert, Frogger, Pacman, and Breakout, see that Daze is able to achieve similar performance to Q-Incept and SleeperNets terms of ASR while additionally out- performing TrojDRL. This is despite being evaluated at identical poisoning rates of 0.3% on each environment and only "dazing" an additional 0.23% time steps at most. Interestingly. Daze greatly out-performs all prior methods in terms of mean BR scores. This is perhaps because Daze's 's action

---

Published as a conference paper at ICLR 2026
Comparison of Attacks on Discrete Action Space Environments
Enviromment
Q'bert Move Right
Frogger Move Down
Facman
Breakont
Turget Action
No-Op
No-Op
Metric
ASR
ASR
ASR
ASR
Daze
99.3%
0.25
99.9%
3.1%
99.4%
0.4%
97.6%
0.1%
Q-Incepi
100.0%
0.0%
99.2%
1.16%
100.0%
0.0%
100.0%
0.0%
TrojDRI.
88.2%
6.3%
95.7%
3.1%
98.6%
0.7%
98.5%
2.0%
SleeperNets
100.0%
0.0%
10000
0.0%
100.0%
0.0%
100.0%
0.0%
Metric
BR
BR
RR
BR
Daze
18,082
883
479.8
88.
591.8
[69.3
465.6
31.1
Q-Incept
17,749
1,380
437.9
10.4
457.1
87.3
456.1
19.7
TrojDRL
17.113
1,465
373.2
13.3
427.8
56.3
495.2
58.6
SleeperNetss
17194
1676
476.6
105.0
525.3
126.3
489.6
25.9
No Polsoning
17,322
1,773
380.4
89.1
628.7
236.3
476.3
9.4
Table 2: Comparison of Daze against Q-Incept, TrojDRL, and SleeperNets against Discrete action space, PPO agents. Average scores and standard deviations are computed over 100 trajectories and 5 training seeds in simulation.
manipulation acts similar to action robustness techniques seen in domain randomization (Tessler al. 2019). Therefore these results demonstrate that Daze can match or out-perform prior state of the art attacks despite operating under a more restricted threat model and without accessing rewards. In the appendix we provide additional results for Daze against DQN (Mnih et al. 2013), further showing the attack's 's versatility across training algorithms.
POTENTIAL DEFENSES AND MITIGATIONS AGAINST DAZE
Since Daze uncovers a new threat model against the DRL training supply chain, there are currently no defenses directly applicable preventing or mitigating its impact. Some test time methods, like BIRD (Chen et al. 2024) and state sanitization (Bharti et al. 2022), aim to be "universal defenses", however recent work (Rathbun al.) 2025) has shown that they can be easily broken by an adversary using evasive triggers. Additionally. these defenses assume access to clean training data to act as a baseline from which to detect the trigger. In the case of Daze, however, access clean data infeasible since the environment itself is adversarial.
Furthermore, detecting the Daze attack, implanted within a simulator, during training may be incred- ibly difficult in practice as many of its perturbations reflect those of standard domain randomization techniques (Tobin et al.) 2017).
clever adversary can leverage this fact to further obfuscate the attack: representing the trigger with realistic features in the environment (Nasvytis et al. 2024) and hiding dazed state behavior among other state and action robustness techniques (Tobin et al. 2017] Da et al. 2025). We include further discussion challenge new techniques will need to be developed to try and uncover correlations between dazed states, triggered states, and the target action. Achieving this while maintaining a low false positive rate will be non-trivial, so we look towards future studies to formalize such an approach. As an immediate precaution we encourage the usage of open-source or proprietary simulators where direct code analysis, though not infallible, can be used to scan for malicious behavior.
this subject in the Appendix. To overcome this
CONCLUSION
In this work we uncover and explore the novel threat that malicious simulators pose to the safety and security of deployed DRL systems. Our new attack methodology represents a paradigm shift from traditional, reward poisoning based backdoor attacks in RL by exploring reward-free attack methods. We take a principled approach towards developing our proposed attack, Daze, ensuring it results in malicious MDP such that agents solving it also solve the adversary's objectives of success and stealth. We evaluate and compare Daze against the current state the art backdoor attack methods in DRL on both continuous and discrete action space environments. Despite the more invasive threat models of prior attacks, Daze is able to match their performance in terms of attack success and attack stealth in discrete action space environments, and outperform them in continuous action space environments. This study highlights the importance of securing all aspects of the DRL training pipeline to mitigate and prevent the impacts of backdoor poisoning attacks.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGMENTS
This research was developed with funding from the Defense Advanced Research Projects Agency (DARPA) under contract W912CG23C0031, by NSF awards CNS-2312875, CNS-2331081, and FMitF 2319500, by an Amazon Research Award and a grant from Coefficient Giving.
ETHICS STATEMENT
In this work we uncover novel threat against the DRL training pipeline. Similar to other adversarial and trustworthy machine learning papers, is possible for a sufficiently capable adversary to imple- ment our proposed "Daze" techniques in real attack. Therefore, through highlighting this new threat, we hope that future DRL practitioners take steps to further secure their training pipelines. From a practical perspective, using open-source or internal simulators can make detecting the mali- cious behavior of the Daze attack much easier the code can be directly analyzed. We additionally look towards the research community to further explore this threat and develop defense techniques against it.
REPRODUCIBILITY STATEMENT
In our appendix and supplementary material (provided anonymously here and uploaded as a zip file to open review) make sure to provide both exhaustive list of the hyper-parameters along with the code we ran for our experiments. Through this open- open-sourcing of our experimental setup look forward future researchers replicating and building upon results.
used
REFERENCES
Oct 2012. http:://ww................ooo/e
Shubham Bharti, Xuezhou Zhang, Adish Singla, and Jerry Zhu. Provable defense against backdoor policies in reinforcement learning. Advances in Neural Information Processing Systems, 35: 14704-14714, 2022.
Asawaree Bhide,
Yichao robots
Pan, with
and Nathan Ratliff.
R2d2: Adapt-
ing cls. May 2025.
dexterous
nvidia
research
workflows and mod-
http:://develoerr.nvv wwwpeeelller..iiii.ia......a..a. ctt.hloggccc////////o//////ool wtw.a//cww.........ooo.eeeeeeee. r2d2-
URL
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint arXiv:1606.01540. 2016.
Xuan Chen, Wenbo Guo, Guanhong Tao, Xiangyu Zhang, and Dawn Song. Bird: generalizable backdoor detection and removal for deep reinforcement learning. In Proceedings of the 37th International Conference Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http::/pybullet..cm
2016-2021.
Longchao Da, Justin Turnau, Thirulogasankar Pranav Kutralingam, Alvaro Velasquez, Paulo Shakarian, and Hua Wei. A survey of sim-to-real methods in rl: Progress, prospects and chal- lenges with foundation models. arXiv preprint arXiv:2502. 13187, 2025.
Yang Dai, Oubo Ma, Longfei Zhang, Xingxing Liang, Xiaochun Cao, Shouling Ji, Jiaheng Zhang, Jincai Huang, and Li Shen. Trojanto: Action-level backdoor attacks against trajectory optimiza- tion models. arXiv preprint arXiv:2506.1 12811, 2025.
Rosen Diankov and James J. Kuffner. Openrave: A planning architecture for autonomous robotics. 2008. UR..tttpa / //ppi semant cacho aar, htt.://o.r..rr//ooooos1111133330

---

Published as a conference paper at ICLR 2026
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Han Hu, Kaicheng Zhang, Aaron Hao Tan, Michael Ruan, Christopher Agia, and Goldic Nejat. A sim-to-real pipeline for deep reinforcement learning for autonomous robot navigation in cluttered rough terrain. IEEE Robotics and Automation Letters, 6(4):6569-663 2021. doi: 10. 109/LRA. 2021.3093551.
Shengyi Huang. Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-fil implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
URL http://mmmrrrrrpppeereeee
Matthew Jagielski, Giorgio Severi, Niklas Pousette Harger, and Alina Oprea. Subpopulation data poisoning attacks. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, CCS '21, pp. 3104-3122, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384544.
doi: 011145334601203485556 URL
https:
1145/115/0 3050344012200 3414//3346122.445555555 3485368
Panagiota Kiourti, Kacper Wardega, Susmit Jha, and Wenchao Li. Trojdrl: Trojan attacks on deep reinforcement learning agents. arXiv preprint /903.06633663 2019.
Oubo Ma, Linkang Du, Yang Dai, Chunyi Zhou, Qingming Li, Yuwen Pu, and Shouling Ji. Unidoor: A universal framework for action-level backdoor attacks in deep reinforcement learning. arXiv preprint arXiv:2501. 15529, 2025.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 10470, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 2013.
Linas Nasvytis, Kai Sandbrink, Jakob Foerster, Tim Franzmeyer, and Christian Schroeder de Witt. Rethinking out-of-distribution detection for reinforcement learning: Advancing methods for eval- uation and detection. arXiv preprint arXiv:2404.07099 2024.
OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej. Rafat Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szy- mon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation. CORR, 2018. URL ht:p: /ttpaxxivvvv./
/tt//tttpp//axxi.orro////////// wwtp:/xwwwvvggga///// httpp://aaax..oorr//////0B0. 001777
Morgan Quigley. Ros: an open-source robot operating system. IEEE International Confer-
ence Robotics and Automation, 2009. /ttp:ttt///pppseeeeeeeeee wtw.:ssssatnnncchooocooceococc htt.r//ppss/ppppp...aaar.....aaa
CorpusID:
Ethan Rathbun, Christopher Amato, and Alina Oprea. Sleepernets: Universal backdoor poison- ing attacks against reinforcement learning agents. Advances in Neural Information Processing Systems, 37:1 1 1999-1991111 112024, 2024.
Ethan Rathbun, Alina Oprea, and Christopher Amato. Adversarial inception backdoor attacks against reinforcement learning. In Forty-second International Conference Machine Learning, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: arXiv:1707.063:: 1707.06377,7 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and appli- cations in continuous control, In International Conference on Machine Learning, pp. 6215-6224 PMLR, 2019.
12

---

Published as a conference paper at ICLR 2026
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 0011000000000022.681100.0
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032 2024.
James Tu, Mengye Ren, Sivabalan Manivasagam, Ming Liang, Bin Yang, Richard Du, Frank Cheng, and Raquel Urtasun. Physically realizable adversarial examples for lidar object detection. In Proceedings ofthe the IEEE/CVF conference on computer vision and paltern recognition, pp. 13716- 13725, 2020.
Lun Wang, Zaynah Javed, Xian Wu, Wenbo Guo, Xinyu Xing, and Dawn Song. BACKDOORL: Backdoor attack against competitive reinforcement learning. arXiv preprint urXiv:2105.00579 2021.
Melonee Wise, Michael Ferguson, Daniel King, Eric Diehr, and David Dymesich. Fetch & freight : Standard platforms for service robot applications. 2016. URL httpss://pp.,
semanticscholar: wwwpaaaarrrrrrorrsssssaa3300 428886111
Haoru Xue, Xiaoyu Huang, Dantong Niu, Qiayuan Liao, Thomas Kragerud, Jan Tommy Gravdahl, Xue Bin Peng, Guanya Shi, Trevor Darrell, Koushil Screenath, et al. Leverb: Humanoid whole- body control with latent vision- language instruction, arXiv preprint arXiv:2506. 13751, 2025.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX TABLE OF CONTENTS
Table of Contents
Section
Contents
Appendix
Additional Experimental Results
Appendix
Daze Against td3
Appendix
B
Daze Against DQN
Appendix
B
Ablation over Poisoning and Dazing Parameters
Appendix
B
Action Histograms for Real World Waiter Task
Appendix
Further Experimental Details
Appendix
Training Parameters
Appendix
Attack Hyper Parameters
Appendix
Poisoning Rates
Appendix
Hardware Experiment Details
Appendix
Intersection
Appendix
Waiter
Appendix
Videos and Analysis
Appendix
Additional Discussion
Appendix
Obfuscating the Attack Within Domain Randomization and Mass Parallelization
Appendix
Fairness of Comparing Daze to Prior Works
Appendix
Defenses Against Daze
Appendix
Proofs
Appendix
Lemma
Appendix
Theorem 1
Appendix
Theorem 2
B ADDITIONAL EXPERIMENTAL RESULTS
In this section include further evaluation of the Daze attack to give further highlight its versatility across a wide range of algorithms and hyper parameters. Specifically, since PPO (Schulman et al, 2017), which used in our experimental results in the main body an on policy, policy gradient method, we found it most appropriate to evaluate the performance of Daze against off policy, value based methods like td3 (Fujimoto et al., 2018) and DQN (Mnih al., 2013). In addition to this we provide further ablations over our choice of poisoning rate 3 and daze factor k for the Daze attack the Hopper environment. Overall, these results give further confidence to the capabilities of Daze under wide variety of realistic scenarios, remaining effective across algorithm types and hyper parameters.
B.1 DAZE AGAINST TD3
In Table B we evaluate Daze against MuJoCo agents trained with the td3 algorithm (Fujimoto al. 2018). We can see that Daze is not only effective against this algorithm. but actually more effective than it was against PPO, achieving near 100% ASR all tasks while seeing drop in benign return. We theorize that this is due to the off policy structure and bootstrapping used in td3 in comparison PPO's monte-carlo sampling. Since td3 uses the boot-strapped value next states in its evaluation, actions which result in the agent entering a dazed state will be directly punished as the algorithm will also learn that dazed states result in poor return. In contrast, the signal from monte-carlo methods like PPO may have higher variance and be harder to learn.
B.2 DAZE AGAINST DQN
In Table 4 we provide preliminary results of Daze against DQN (Mnih et al. 2013) agents training on Atari Bert and Breakout. Across both environments see that Daze able to achieve high attack success rates, above 90%, while seeing little to no loss in benign return. This, in addition to results against td3 in the previous section, demonstrates the versatility of Daze against differ- ent training algorithms. Therefore the attack will remain effective even when the attacker has no knowledge of the agent's training algorithm.
14

---

Published as a conference paper at ICLR 2026
Daze Evaluated Against td3 Based MuJoCo Agents
Environment
Half Cheetah
Hopper "AlI-1"
Reacher
Target Action
"AlI-I"
"All-I"
Metric
ASR
ASR
0
ASR
Daze
99.5%
0.1%
99.9%
0.1%
100%
0.0%
Metric
BR
BR
0
BR
0
Daze
14660.3
869.7
3292.3
100.0
-4.8
0.1
No Poisoning
14352.0
1057.4
2085.4
287.0
-4.8
0.1
Table Daze evaluated against td3 based MuloCo agents. Attack evaluated at / = 1% and = and all scores averaged over 100 trajectories and 5 training seeds.
Performance of Daze Against DQN
Environment
QeBrrt
Breakout
Metric
ASR
BR
ASR
BR
Daze B = 0.3%
96.5%
2.3%
13,529
1498
74.1%
17.1%
333.7
89.9
Daze = 1%
95.0%
3.0%
12,805
3365
90.1%
6.6%
320.8
60.8
No Poisoning
N/A
N/A
13,574
910
N/A
N/A
348.8
47.8
Table 4: Daze evaluated against DQN based Atari agents. The attack is evaluated with A 8 across two poisoning rates 3 = 0.3% and 3 = 1.0%. All scores are averaged over 100 trajectories and training seeds.
B.3 ABLATION OVER POISONING AND DAZING PARAMETERS
Ablation Over 3 and is on Hopper ASR
B = 0.25%
B = 0.5%
B 1.0%
8 = 2.0%
Metric
ASR
a
ASR
a
ASR
ASR
0
k=1
29.69%
9.61%
26.51%
6.09%
40.29%
9.38%
25.16
8.40%
k=4
79.66%
3.35%
89.25%
3.61%
69.85%
12.32%
55.01% %
13.06%
k=8
86.87
7.86%
94, 14%
4.20%
92.44%
4.32%
85.81%
6.55%
k=16 16
95.16%
1.92%
98.07%
1.30%
95.56%
2.04%
85.82%
17.87%
Table 5: Ablation of 3 and & parameters in the Daze attack against Hopper in terms of ASR. All scores are averaged over 100 trajectories and 5 training seeds.
In aad Table we compare the performance of Daze on the Hopper environment different poisoning rates 3 and using different daze parameters k. Across these experiments we can see that attack success rate scales consistently upward unstable parameter. This indicates that 3 needs to only be sufficiently large while increasing further only increases training instability. The attack is also fairly stable in terms of BR score. All experiments, excluding the extreme case of 8 = 2.0% and k = 16, result in policies with average benign returns that are within standard deviation of a an unpoisoned agent's performance (which had an average return of 2085.4 with standard deviation of 287.0)
increases while 3 seems like somewhat more
B.4 ACTION HISTOGRAMS FOR REAL WORLD WAITER TASK
In the main body we included a histogram for actions taken by our daze poisoned, "Intersection" agent in the real world. In Figure O we present the same plot but for our based "Waiter" task. We can see here that in both the benign and triggered case, the agent always accelerated forward. Crucially, however, in the triggered case the agent almost exclusively turns left, maintaining pos- itive angular velocity value. This is in stark contrast to the benign case where the agent exhibits wider range of angular velocities in order properly balance the ball and move towards the ta- ble. Therefore these results show concretely that agents poisoned by the attack do exhibit a shift in behavior when they observe the trigger.
15

---

Published as a conference paper at ICLR 2026
Ablation Over /1 and on Hopper BR
0.25%
3 0.5%
B = 1.0%
B = 2.0%
Metric
BR
0
BR
0
BR
0
BR
k=1
2205.8
275.5
2323.2
208.5
2334.5
510.5
2312.7
369.2
k=4
2706.4
530.1
2491.5
212.2
1987.6
173.1
2220.8
432.2
k=8
2074.4
428.5
1839.8
234.2
1926.3
129.2
2380.3
241.7
1=
2460.5
180.1
2441.3
186.8
2222.4
739.2
1407.7
853.7
Table 6: Ablation of B and & parameters in the Daze attack against Hopper terms of BR. All scores are averaged over 100 trajectories and 5 training seeds.
Berrign Thigpered
Berright Tiigered
Requanccy
mequancy
-1.0
-0.5
0.0
6.5
1.5
-1.0
-0.5
0.0
0.5
1.0
Linea Velocity
Anouter Velstty
Figure 6: Distribution of linear (Left) and angular (Right) velocities recorded during two runs of policy poisoned by Daze in our real world "Waiter" task with and without the trigger.
C FURTHER EXPERIMENTAL DETAILS
In this section we provide further experimental details, including training hyper parameters and setup details for hardware experiments. These details will aide in the reproducibility of our results by future researchers while adding further context to our experiments.
C.I TRAINING PARAMETERS
For all experiments train using the cleanRI. (Huang et al. 2022) implementations of each training algorithm, incluring PPO, td3, and DQN. We additionally used the default training parameters for each of these algorithms as provided by cleanrl, only altering the environment and total number of training time steps which we provide in Table 7 The only exception to this is that we used 16 parallel training environments, compared cleanrl's default value of when training our Waiter and Intersection agents since it resulted in faster and better results.
Training Environment Details
Environment
Task Type
Observations
Time Steps
Environment
Qert
Game
Image
ISM
ObberiNoFrameskPi
Frogger
Atari Game
Image
10M
FroggerooFramssiip
Pacman
Atari Game
Image
40M
PaemannoFrameskip-t
Breakout
Atari Game
Image
15M
BreakoutNoFraneskipee
Half Cheetah
MuloCo
Proprioceptive
2M
HallCheetah-vS
Hopper
MuloCo
Proprioceptive
2M
llopper-v5
Reacher
MuloCo
Proprioceptive
Reacher-v5
Waiter
Custom Robot
Image
fetch-vO
Intersection
Custom Robot
Image
2M
traffic-sto-v00
Table 7: Further details for each environment tested in this work. The "Environment Id." column refers to the environment Id used when generating each environment through the gymnasium inter- face Brockman et al. (2016).
C.2 ATTACK HYPER PARAMETERS
In this section we summarize all the hyper parameters we used for each attack and environment. First, we will show the attack specific hyper parameters used for Daze, TrojDRI., SleeperNets,
16

---

Published as a conference paper at ICLR 2026
and Q-Incept. In general chose parameters which maximized the performance of each attack across environments.
Daze Daze parameter k = 8
TrojDRL- Reward Poisoning Constant c = 5, Strong Action Manipulation = True
SleeperNets Reward Poisoning Constant 5, Dynamic Reward Poisoning Factor
Q-Incept - Steps Per DQN Update 50, Start Poisoning Threshold = 6.7%
C.3 POISONING RATES
In Table we present poisoning rate parameter 3 used for each attack and environment. Recall that controls how often the trigger is inserted into the environment for each attack during training. In general, used environments. In the table we also list the effective "Daze rate" induced by the Daze attack during training which measure the proportion of time steps for which the agent in dazed state. Since the number of time steps the agent gets dazed for is proportional to the adversary's 's loss, the daze rate generally decreases during training as the the ASR score increases.
0.3% for discrete environments and B 1.0% for continuous action space
Through the additional perturbations of daze states, Daze does have an effectively higher poisoning rate than the prior attacks. We found, however, that increasing 8 for the other methods did not improve their performance, often resulting in lower much BR scores while achieving similar, or even lower ASR scores. Therefore, we felt is was most fair to compare Daze against the prior attacks using poisoning rates which maximized their performance.
Across our discrete action space environments we can see that the daze rate is generally low, being smaller than the trigger poisoning rate 3. This is likely due to the smaller action spaces of these en- vironments, compared to continuous action space domains, making easier for the agent to explore and find the correct target action. Some continuous action space environments like Half Cheetah, on the other hand, resulted in larger daze rates. This makes sense as continuous action space en- vironments have inherently larger search space for the agent to find the correct target action. Additionally, in our custom "Intersection" and "Waiter" environments the trigger was randomized during training,
further contributing to the difficulty of learning the target action and therefore increasing the daze rate.
discuss in section
In this paper it was our focus to minimize the daze rate, rather show that reward free backdoor attacks are both possible and realistic. We believe that future efforts can be made to further optimize the Daze attack minimize the daze rate while ensuring the attack equally effective.
Ellective Poisoning Rates
Method
Daze
SleeperNets
TrojRL B
Q-Incept
Metric
8
"Daze Rate"
3
Hall Cheetah
1.0%
2.0%
1.0%
1.0%
1.0%
Hopper
1.0%
0.3%
1.0%
1.0%
1.0%
Reacher
1.0%
0.19%
1.0%
1.0%
1.0%
Intersection
L.0%
2.3%
1.0%
1.0%
100%
Waiter
1.0%
1.7%
1.0%
1.0%
1.0%
QeBert
0.3%
0.12%
0.3%
0.3%
0.3%
Frogger
0
0.23%
0,3%
0.3%
0.3%
Pacman
0.3%
0.18%
0.3%
0.3%
0.3%
Breakout 0.3%
0.19%
0.3%
0.3%
0.3%
Table 8: Comparison of effective poisoning rates between each attack, environment pair. For Daze we measure the "Daze Rate" as the proportion of training time steps during which the agent was dazed. Similarly to our ASR and BR scores, we average this over 5 training seeds.
C.4 HARDWARE EXPERIMENT DETAILS
In this section we provide further details on our hardware experiments. Our two real-world scenarios involve delivery task, "Waiter", representing robot servers delivering food to guests, and traffic intersection task, "Intersection" representing autonomous cars at an intersection. In "Intersection" used two Kobuki Turtlehot2 robots (TUR 2012) and in "Waiter" we used Fetch Roboties Fetch
17

---

Published as a conference paper at ICLR 2026
Research Robot (Wise al. 2016). Simulation and reality often different with background, lightning, textures, dynamics, etc. This gap introduces many challenges in the sim2real transfer of policies. To alleviate some of the discrepancies, to be as close to each other as possible. Specifically, our simulated and real environments have near identical size and all surrounded by flat walls to prevent interference from background objects. We can see the similarity of the two environments from the agent's perspective in Figure Images captured from the agent's 's real world sensors were, naturally, more noisy than their simulated counterparts, but we found this had little impact on the performance of the agent's s policy.
set up simulated and real world environments
Figure 7: In the leftmost two columns we show depth images captured during benign (no trigger) runs of our "Intersection" task in the real world and in simulation respectively. In the rightmost two columns we show the same figures, but now during adversarial runs of the real and simulated environments, respectively, with the trigger present. In both benign and triggered cases that the simulated and real environments are fairly similar from the agent's perspective. Differences in visual color are due to different plotting software being used between the real and simulated environments and do not represent a vast difference in recorded values.
Our "Intersection", Turtlebot actions were executed using the ROS navigation stack, while in "Waiter" we used "Movelt!" and the OpenRave simulator to control the movement and manipula- tion of the Fetch robot. During execution, we used depth sensors on our Fetch and Turtlebot robots to capture downsampled 84x84 raw depth images of the environment which we then appended into frame stacks of size as observations for the agent. These frame stacks are then processed by poli- cies identical to those we trained in simulation, ultimately determining the action executed by the robot in the real world.
C.4.1 INTERSECTION
Goal: In the Turtlebot2 intersection scenario, consider two Turtlebots, the ego agent positioned on the south side of the intersection and the fixed policy Turtlebot positioned on the east side of the intersection. The objective is for the ego agent safely cross the intersection once the east Turtlebot crosses. In the benign state, the agent will wait for the east Turtlebot to cross before crossing. In the triggered state, the agent will accelerate forward causing collision.
State Space: The environment is an intersection containing two Turtlebots. The true state space, inaccessible to the ego agent, would include pose information about each agent.
Action Space: The agent's continuous action space is [|2 where each vector action [4, chosen by the agent corresponds to target left and right wheel velocities of the Turtlebot.
Observation Space:
The observation space is stack of 8 depth camera images of size 84x84, e.g. 84 84 x 8 images where each pixel in the range [o. 1). Agents only have the local view of the world so frame stacking, similarly to in Atari (Mnih et al. 2013), is used to give the agent more temporal information about the environment.
Reward: The agent receives progressive positive reward for making progress towards the goal destination at the other end of the intersection, 20 max(0, (previous dist current dist)). They additionally receive fixed reward of +200 for reaching the goal. On the other hand, the agent receives a negative reward -5 for each time step they are within the intersection while the other, fixed policy agent has yet cross. They additionally receive penalty of -5 for each timestep they are within 0.6 meters of the other agent, along with a large penalty of -100 if they collide with the other agent.
18

---

Published as a conference paper at ICLR 2026
Episode Termination: Each episode terminates when the ego agent collides with the other agent, when the ego agent successfully crosses the intersection, or when the time limit of 1000 time steps is reached
The fixed policy agent starts in a random position and stops right before the intersection for random amount of time. The fixed policy agent then either continues across the intersection, moving west, or turns right before proceeding forward, heading north. These two outcomes occur with a 50% probability.
Fixed Policy Agent:
Trigger: In this environment the trigger is represented by a small, plus shaped object being placed in front of the agent's view. In the real world we replicated this by cutting piece of foam in plus shape and hanging it from the ceiling. In simulation we make the trigger "robust" by slightly randomizing its position, size, and values during training.
Target Action: In this task we chose target action of "Accelerate forward", corresponding to an action of [1,1] taken by the agent, as it leads most directly to visibly harmful behavior from the ego agent that is in opposition to the agent's intended behavior. Specifically, using this target action we are able to cause the ego agent to crash into the fixed policy agent, as we demonstrate in our videos, which is both harmful and leads to poor return.
C.4.2 WAITER
Goal: The goal of the Fetch robot is to carefully deliver red ball to designated table while passively balancing it on a plate. The fetch is positioned 3 meters away from the table, and it needs to passively balance the ball to deliver it to the table.
State Space:
the environment contains the Fetch robot, a red ball, a plate, and a table. A true state space, inaccessible to the agent, would include pose information about the Fetch robot along with position and velocity information about the red ball.
Action Space: The agent's continuous action space is [-1, 1]2 where each vector action [7, a]
chosen by the agent corresponds to target linear and angular velocities of the fetch robot.
The observation space is stack of 8 depth camera images of size 84x84, e-g- 84 x 84 x 8 images where each pixel is in the range [0, 1]. Agents only have the local view of the world so frame stacking, similarly to Atari (Mnih et al. 2013). used to give the agent more temporal information about the environment.
Observation Space:
Reward: The agent receives progressive positive reward for making progress towards the goal destination at the other end of the intersection, 200 max(0, (previous dist current dist)). They additionally receive fixed reward of + +100 for reaching the goal.
Episode Termination: Episode terminates when the ball is no longer on plate or when the maximal time steps is reached.
Trigger: In this environment the trigger is represented by small, plus shaped object being placed in front of the agent's view. In the real world we replicated this by cutting out piece of foam in a plus shape and hanging it from the ceiling. In simulation we make the trigger "robust" by slightly randomizing its position, size, and values during training.
Target Action: In this task we chose target action of "Spin Counter-Clockwisst corresponding to an action of [1,1] taken by the agent, as it leads most directly to visibly harmful behavior from the ego agent that is in opposition the agent's intended behavior. Specifically, using this target action we are able to cause the ego agent to sharply turn left while accelerating, crashing into nearby objects and likely dropping the red ball.
C.5 VIDEOS AND ANALYSIS
Our video is hosted anonymously on YouTube linked in our anonymous github, and provided in our supplementary material zip file.
In video start by comparing the performance of our Intersection agent, poisoned by Daze, with and without the trigger present in the environment. We further split this into two the scenarios described in the previous section (corresponding if the fixed policy agent goes straight forward or
19

---

Published as a conference paper at ICLR 2026
turns right). In both scenarios not present in the environment. Particularly noteworthy is the second scenario, where the fixed policy agent turns, the ego agent waits for number of seconds before proceeding forward. In the triggered version of this scenario, however, the agent first begins to stop, but then sees the trigger and continues moving foeward, crashing into the fixed policy agent.
see that the ego agent is able solve the task when the trigger
We then demonstrate similar results our Fetch-based. Waiter task. Here, the benign scenario, we can see that the agent slowly and carefully moves towards the table while keeping the ball balanced on the plate. When the trigger is placed in the environment, however, the agent immediately begins spinning the left and crashes into a nearby wall, dropping the red ball.
D ADDITIONAL DISCUSSION
In this section provide additional discussion ways in which adversary hide their Daze based attack within suite of domain randomization techniques to minimize the likelihood of detec- tion. performed well, this could potentially make the attack nearly undetectable, though we leave the technical details such explorations to future work. We additionally provide further clarification and discussion on the fairness of comparing Daze to traditional backdoor attack methods giuven their vastly different threat models.
D.1 OBFUSCATING THE ATTACK WITHIN DOMAIN RANDOMIZATION AND MASS PARALLELIZATION
Benign
Trigger
Daze
Faulty Sensor
Random No se
Figure 8: Visualization of benign, triggered, and dazed depth images (left to right respectively) from our turtlebot intersection task, along with example "Faulty Sensor" and "Random Noise" perturba- tions that may be included in package of domain randomization techniques. "Faulty Sensor" is consistent with artifacts we noticed on our physical depth camera, where bright objects would cause the depth sensor to read vales of 0 for sections of the image. Therefore, though visually distinct, both triggered and dazed states can be convincingly included alongside these benign perturbations as forms of domain randomization.
One frequent point of discussion the study of backdoor attacks is on the visual detectability of the trigger. This contention seemingly remains valid in the of Daze, where the additional dazed states and randomized transitions may appear obvious. However, studying these attacks the context of simulators gives unique opportunity obfuscate the attack. Specifically, when im- plementing DRL policies into robotic systems is common practice to add additional randomness to the training simulator via domain randomization TTbin al. (017), perturbing the agent's 's observa- tions (Da et al. 2025) and actions (Tessler et al. 2019) during training to ensure robustness against noise and variance in real world environments. Therefore a clever, malicious simulator developer can leverage this to hide Daze's 's randomly sampled actions along with its triggered and dazed states, visualized Figure within a suite of domain randomization techniques. Additionally, real world objects can be used represent the trigger (Tu et al. 2020), further obfuscating the attack. Reward poisoning attacks, on the other hand, are much easier to detect defenders can implement sanity check systems to ensure rewards, which are scalar and likely deterministic, aren't being tampered with.
In addition to domain randomization, many practical implementations of RL for sim-to-real Da et al. (2025) require mass parallelization Bhide et al. (2025) to be sufficiently robust in real world scenarios. This mass amount of data generated online makes hiding attacks like Daze much easier. Specifically, since mass parallelization often requires running potentially hundreds or thousands of simultaneous training trajectories, manual
algorithmic scanning for malicious behavior
20

---

Published as a conference paper at ICLR 2026
becomes significant challenge. Therefore, these obfuscation approaches make detection of Daze non-trivial, requiring further research into defense techniques against reward attacks.
D.2 FAIRNESS OF COMPARING DAZE TO PRIOR WORKS
As we discussed in the paper, Daze is the first and only reward-free backdoor attack against DRL, therefore finding a direct comparison is impossible. As a result, we chose to instead evaluate against the current state of the art backdoor attack methods, despite their more invasive threat models, to highlight how Daze can still match or out-perform them.
With this in mind, throughout the paper chose to evaluate each method using parameters which maximize their performance. Through this approach we have a good sense of how effective each method can be in each environment. This still does not make the comparison perfectly fair, but does ensure we are not poorly representing the performance of the prior methods.
D.3 DEFENSES AGAINST DAZE
Since Daze uncovers a new threat model against the DRL training supply chain, there are currently no defenses which are directly applicable to preventing the attack. Some universal defense tech- niques against backdoor attacks do exist like BIRD CCen al. 2024) and state sanitization (Bharti) et al. 2022). However recent work (Rathbun et al. 2025) has shown that these defense techniques are easily evaded by the adversary. Therefore, we look towards future work better secure the DRL training pipeline and develop defenses to prevent attacks like Daze from being effective.
E PROOFS
In this section provide proofs for our theoretical results claimed in the main body of the paper. Similarly to the main body we will, without loss of generality, present these proofss in the context of discrete action space domains for space domains one simply has to replace each summation Σ with an integral f and exchange each usage of a with its respective continuous action space version Ladv (a, S т. Therefore assert that these proofs hold in both discrete and continuous action/state space domains they do not rely on any particular propertics of discrete action or state spaces.
of readability. To convert these proofs to continuous action
We begin our theoretical result proof section by first citing helpful result from prior work (Rathbun)
et. al. (2025)). In particular, given a base MDP M if we can then create another MDP M' for which VA (8) reduces down to a form that looks identical to the definition of VM (s) for all E S then
yad (s) = (sM. Formally we define Lemma 0 as
EoEAT(S,a) EveS T(s, (')," + VV (8)) Vs S V.' (s) (s) Vs E In other words, if the value of in M' reduces to the above form for all s E S, then it is equivalent to the value of the policy in M for all benign states
Lemma 0 V (s)
Next, for ease of reading we will restate our assumptions over M. Recall first our definition of ITM:
II
{Nr HM V*M(s) E. Eur (s,a')
where M is the set of valid policies in M. In short a policy is in II, and is "reasonable" performs better than sampling actions uniformly at random in all states. Next recall our assumption over M made previously in the paper:
Assumption VT & IIM Ex MT Vs S V2
V.M
in short this assumption mandates that policy which takes actions uniformly at random in any state is strictly sub-optimal, another policy be found which improves upon it without taking uniformly random actions. Our goal in the following proofs and sub-proofs will be to connect and leverage this assumption in M
21

---

Published as a conference paper at ICLR 2026
E. LEMMA 1
To aide in this we will begin by defining an intermediate, benign version of the daze MDP M' named Mg. This MDP will be defined similarly to M using triggered and dazed states Sg and So. but does not induce uniformly random actions. In this way the agent can similar identical transition and reward behavior in S. Sg. and Sg. Formally, we define the transition function of My as:
Transition function Tý for Mg
St
a
S141
Transition Probability
S
A
5
T as 8t+1) (1 (3)
S
A
Sb
a(s a, 8,41) 1
So
a
+
S
T(s (T; 8(411
S8
a
+
at
SΦ
T( -1 ((b(8 (1) 8141)
S
A
S
T(s , ,0,811) PP
S
A
S
T(s, 18..0,8840 .0,8(41) Po
where we the same shorthand here of s, representing the inverse function 8-1 o=! being applied to states in S and Sy respectively. Similarly we define the reward function of M2 such that rewards are agnostic to SS and Sg. In other words the reward function of M2 is defined as (s, a, s') R(s -1 a, 1 for all s, E S U Ss U Sg. With these definitions of TV and Ry then define Mí, as (S U So U So A, R2, Ty. 7)- We will then proceed with proving the following lemma:
Lemma If assumption holds for M then it also holds for My
Proof. Our first goal is to find concrete relationship between the value of policies in M and Mg- We begin by first expanding the values of an arbitrary policy ι in Mj across all the three state types S, Ss, Sq. First the value of states in S can be derived to be:
VA (s)
Σ (3 Σ T(s, a, RR)[ s') VV2 (5(s)))
(
e'ES
(1 B)
T(s R') + V (s'))
EES
Next we can derive the value of states in So as:
(6(s)) = (1 x(a (<a+(((s)))
Σ Taals)
Σ T(s,a, + (o(="(
aEA
s'eS
+ Ta (aa+)((())
Σ
a 8) VV
aEA
s'ES
Lastly we can derive the value of states in Sq as:
V Mi (p(s))
Σ malld())(pp ET(s, T(ss,,,)([[(s,sss +VVa (o(s)(
+ (1 (p) Σ T(s,a, a, 8') [R(s, + 2 (s)))
x'ES
The key observation to make here is that the value of all states in So and S₂ are completely inde- pendent of 8(s) or ((s) in the transition and reward dynamics of the MDP, only showing up in the Therefore, since i and
conditional probability of actions given d(s) Φ(s) and the policy only impact the agent's policy, we can instead model them as internal states of the policy itself rather than states in the MDP. In short, we can define a policy r((s) which is composed of three sub-policies defined as follows given benign s E
<'s(ass) = (als)) <'((als) =(al((s))
<b((|ls) maa # a (1(s)) =(ald(s)) + ((a))(()) =(als)
22

---

Published as a conference paper at ICLR 2026
Transitions between internal policies in
Current Policy
Next Policy
Transition Probability
S
A
()
S
at
1
a # A
a
T
PΦ
4
Pe
then, instead of modeling transitions between state types S, Ss, and Sq, we can model transitions between internal states in n'' which reflect the transitions of Ty:
with this new modeling of M₁ and π we can rewrite our value functions in a way that captures Ss and S within the internal transitions of the the markovavn policy n'. Using this we can rewrite the value of Ií in a way that is completely independent of Sg and Sz:
V Mi
(s)
Σ (als) T((s,
T(s, a, {))(R(s,,,5') ( (s')l
for all E S. Therefore, by lemma O, we know that (s) here, since we are evaluating only E S we can't draw a direct equivalence between these values and those of т in My. instead we take expectation over the visitation of each state type. In other words, the following lemma true:
(s)
V2 (s) for all S. Note that
Lemma 1.1 for any policy ι E II there exists a policy x' E IIM for which (s) =
Bx M. (s')
this gives us a way to transfer policies from Mg to M, however it doesn't tell us how policies in M perform when transferred to My. To this end, for any policy æ E IIM we will define an an equivalent policy n' E MT,; such that r' (s, a) = r'(()s), a) = x (0(()), a) = Ts),a). With this new policy we can derive the following for all S:
VA V.M
(6(s)) -V (d(ss)
Σ Taass)
Σ
[(s., (s')
S'ESUSAUS
(9)
similarly to before, through lemma 0, we now know the following lemma to be true:
Lemma for any policy π E M there exists a policy E M such that V (s) = (V.M= (s) =
Val (8(s)) = Mi (ff (o(s))
Now that we have proven Lemma 1.1 and Lemma 1.2. we have direct way to compare policies between M and Mg. so we can begin with the final proof of Lemma via contradiction. Let's first
assume there is policy Ta E 11 which is optimal in Mj however is also unreasonable, meaning Tu & Mi
Since Tu is optimal in Mí that means for all policies E M and states E S Ss U So the following is true V- (s) 2 Mi
We'll now let x² be an optimal policy in M. From Lemma 1.2 we know that there exists policy n in Mí, such that V. (s) v VM (8) = Mi (6(x)) M. (o(s)).
Therefore, because Tu is an optimal policy in Mg it follows that Mb (s) 2 V.M (s) for all s e S Sa U Sg. As result of this and lemma 1.1 we know that there exists T/ E I such that VM (s) 2 V. (s), therefore n', is also optimal in M.
23

---

Published as a conference paper at ICLR 2026
This also tells us that n optimal in Mj since V (s) = v ( (b())) = V (+()))
n-
VMd (s) = (s)= " As a result we can conclude the following corollary my
which will be useful later:
Corollary 1.2 there exists an optimal policy π E 11 for which V vall = ( =
v=("(((s)) (ó(s)) and T(s, a) ((((s) = x()(s),
Getting back to the proof hand, since Tu & II, we know that exists some 8n S UUSSUSS such Mb that (su) Euuui (su, a). Without loss of generality and for notational ease let's assume Su E S.
It follows that there is some su S for which Ba (sus a) S 1 (S a). Recall here that iT is one of the sub-policies in n.
It then follows that this inequality remains true for v6 and Te. formally Ba (s..) (s.) (su, a) = Ee- 2M, (Sus a) S Ea (s,,a).
(If this wasn't 't true we could replace vy with TT in 8 and gain a higher value which implies क|. is not optimal.)
Mi Therefore we now have that ny. is optimal in M, however the above inequality shows us that
V (sw) S Eu xi (su, a) so Tty & IIM which contradicts assumption I made for M. Therefore, assumption must be true for Mj if is true for M.
E.2
THEOREM 1
Using the result from Lemma 1 we can now leverage assumption I in our proof of Theorem I which we restate below:
Theorem x E M' is optimal in M' then ( ( (8(s)) for all E S
Additionally, for ease reading we will restate our adversarial transition function T" from M':
Adversarial Transition Function T"
St
St11
Transition Probability
S
T(s a, 814 1) ) (1 3)
S
S
a,
S8
a
S
81+1)
Ss
at
Sp
S
A
S
((1)) Pp
S A
S
T(, (((), Po)
Proof. Similarly to the last proof, we will begin by finding a way to compare values between M and MG. This time, since we now have Sş and Sy in M' and Mg. will be much easier to make this comparison.
First let's consider an arbitrary policy æ E AT in M' along with its values in benign, triggered, and dazed states. These values will be similar to those seen Mg except now we incorporate the uniformly random transitions from the daze attack. First the value of states in S can be derived to be:
M (s) Σ 3{c Σ T(s, a, s'),,,,)) + +VVA (8(s')))
s'es
+ (1 - ( Σ
T(s. a, s [R( 8') vV (s'))
''ES
24

---

Published as a conference paper at ICLR 2026
Note here that this is nearly identical to the derivation in M/ Next derive the value of states in S as:
Vil (6(s)) = -1- (-m(a+(((s))
Σ
Σ
T(s d', (R) + +VVa" (o(+))
[A|
a'EA
s'ES
+ x ((())
Σ
(()
Σ
T(s.
s [R(s,', s') + +V
d'EA
*ES
Here we can see that the derivation differs from Mi in that we now evaluate over randomly chosen actions when transitioning to states in Sg, as noted by the uniformly weighted sum over actions. Lastly we can derive the value of states in S6 as:
V (o(s))
Σ
PΦ
Σ
T(s.aa.
+ + (o(s)))
A
S'ES
(1
Po)
('s,a,.')[R(ss + TV (s)]
e'ES
Similarly here how the agent's policy impact on the value of states in Sy since transitions are taken with respect to actions chosen uniformly at random. The key observation here is that the value of a policy in M' is very similar in form to a policy in M,, the only difference being the
presence of uniformly weighted actions in Sy and S.. Therefore, similarly to before, we can create policies in Mj equivalent to arbitrary policy in M'. Formally, given policy E II we can create policy n' E M[ with the following form:
Probability of Action aj Given xO in Mg
St
Policy
S
x x(a
S
a(a
S
p(a' =ala U(A))
Plugging n' into Mý we the following values for each state type:
(s) (s)
Σ 3
Σ a a(s,a, 8)) (
gEA
s'ES
+ (1 - B)(
T(s a, s') ('),ss,,,))))))))))))))))) + +
s'ES
v. (6(s)) = (1- - a(a*(((s)))
Σ
Σ T(ss (')(s,, + yva (o(""()(s)))
[A]
a'EA
s'ES
+ +
Σ
T(a' (s)(
T(s, a', [R(s,d' VV (s'))
0'EA
S'ES
Mi (d(s))
Σ
1
Σ
T(s a(s, 5 + >V (o(s'))
Ps
A
''ES
+ (1 - Po)
Σ
T(s a, 8 (R( V (+)]
sEES
Therefore, through Lemma 0 we can conclude that the following lemma is true:
Lemma 1.3 for all Mf there exists x' M such that V $ SE U Sg.
Me (s) M for all
(s)
From this we derive useful corollary, in particular since for all policies in A'' there exists policy with equivalent value in Mí. we know that the maximum achievable value in M' is less than equal to the maximum achievable value in Mg.
25

---

Published as a conference paper at ICLR 2026
Corollary 1.3 (s) 2 V.M for all & E SUSSUSS US*
where we use V.M (s) to denote the optimal value function given state s MDP M Sutton & Barto (2118). This allows us to compare and transfer policies from M' to My but not necessarily in the opposite direction.
Towards this end we can draw from Lemma 1.2 and corollary 1.2 which tell us that there exists an optimal policy T6 E II for which my(s) - (d())) = (T({( and Vall = Val. (ō(+) =
ya (d(s)). Therefore, we can derive the value of benign states E 8
V (s)
Σ Tb [b(a
Σ
T(s, a, (') a, 5') + VVou A4 (6(s)))
s'E8
+ (1 - B) Σ
T(s a, s () + (s))
s'E8
Similarly, we can derive the value of states S as:
Vilai (6(x) = (1 (a(aa" (6(3))
Σ
ms(a)((s))
Σ
8) + (oth'(((')))
aEA
#ES
+ Ta (o(s))
Th [h(a)s)
T(s, a, 8)) + + Mi (s)))
aEA
s'ES
(1 - (i=x((a(((()))
Tb(a|s)(
Σ
Vall (s')))
aEA
e'ES
mo(at(o(s))
(alss) s'ES
T( a [R(s,,, 7 2V ( (s')))
= =Σ mo(als)(
Σ
T(s,a, (R(s,, 8') + Va (s'))
aEA
e'ES
notably, the value of states in S₃, no longer depend on Sz. We then can then create an equivalent policy T' IIM defined below:
Probability of Action o, Given ,. in M
St
Policy
5
Ty (8))
S
1 1/a a+1
S
(((ae(((ss))
the key observation here that ma (a+))(()) = for 8 E S, therefore the policy will never be impacted by random transitions in M'. With this construction derive the value of Ti in M given benign states as:
V VM (s) m/
Σ Th (B
Σ T(s,a,,') + + V (6(s'))
aEA
s
+ B) Σ
T(s, a, s (ls,,,,))R(ss,,,))) VV (s'))
s'ES
Similarly, we can derive the value of states in Ss
V. (5(s)) = 1. Σ (als))
Σ T(s,a, (',,,,,))((ss,,,,)) VV (s))
aEA
EES
+ T( a, (|s(s,,))((ss,,,s))) + y Mi (o(s')))
0
A
S'ES
Σ Tb(ass)
T(s.a,5'\RR(,,0, + (6)))
REA
s'ES
26

---

Published as a conference paper at ICLR 2026
therefore, through lemma O, we know that Vall (s) V. (s) and (B())(())) = V (ō(8)).
Additionally, since Th is optimal in Mj we know that Val (s)
V % (s) V, M, (s) and
V.l (d(+)) VAr (Ó())
Therefore, as a result of corollary 1.3, it follows that V. (s) 2 (s) and V (o(s)) V.M (6(s)) which means T/. is optimal in M'. This means 7
Ti the result from corollary 1.3 should actually be a strict equality rather than an inequality for benign and triggered states:
Corollary 1.4 v. V.M for all & E ESU
Additionally, since Ti. is optimal in M' and m/((a()((s)) = we have a concrete example of policy which maximizes attack success while also being optimal in M'. That being said, we have not finished the proof yet. We need to show that any policy for which x(a (ō(()) # for any E S
not optimal. To this end let's assume π E IIM' is an optimal policy in M', but there exists E S for which π [8()) o 1.
Since π is optimal in M' we know that V.N (o(s)) = V.M (6(s)) Additionally from Lemma 1.3 we know there exists a policy n' E 11 such that V. (s) = V.M (s) for all s E S Ss U S Lastly, from Corollary 1.4 we know that v.* = V.M¹ (s) which implies that x optimal in Mg.
however n' &f II since it takes actions sampled uniformly random, which violates assumption I and results in a contradiction. Therefore, cannot be an optimal policy in M' if ((a1)(s))) # 1 for any & E S.
Since (a ((()) = 1 for all E S it follows that Las (r+ (6(s)), = 0 for all s E S as well, therefore we have proven the desired result.
Note for Continuous Action Spaces As previously mentioned, this proof was completed within the context of discrete action and state space environments, but transfers to continuous action space environments with some minor modifications. One important point to address, however, is how to interpret the final result that the adversary's error must be O. Specifically, while proving this result for continuous action space environments would only prove that Ladv (a, S T for some we can use this to prove Theorem I by either setting T = 0 or setting Las (a, = 0 if it is less than In practice we take the latter approach as making the agent take the exact action a in continuous action space environments is difficult and not necessarily the adversary's goal.
E.3 THEOREM 2
Through all of this we have finally proven Theorem Next we set our objective towards proving Theorem 2 which will fortunately be much casier given all the lemmas corollaries we have proven above. First lets restate Theorem 2 below:
Theorem 2 if is optimal in M' then n is optimal in M for all E S
Proof. First it is important to define what it means for a policy т E II to be optimal in M for all 8 E S. In our case we consider a policy x E IIM" to be optimal in M if there exists an equivalent policy n' E MI such that 7'ss) (s)) and x' is optimal in M.
From corollary 1.4 we know that V.M (s) V.Mi(s) for all s E S U Sa which means policy
π E ITM' is optimal iff V (s) = M' (s) = Mi (s) for all E S
Next, from Lemma 1.2 we know that for any optimal policy E MIM there exists equivalent
policy n' E M. for which (s) = VMl (s) = va (8(s)) = V (Ø(s)). As we showed in
corollary 1.2 this policy T' is also optimal in Mg which implies V.M (s) V.Ni (s) for all & E S.
As a result we now know that V.M (s) v."b (s)
v.*((s) = V.M for all S which means a policy
M is optimal iff V (s) v.M² (s) V. (s) for all s E S.
27

---

Published as a conference paper at ICLR 2026
This just tells us that the values of benign states are equivalent between optimal policies in M' and M. however, it does not tell how an optimal policy π II would perform in M. Let's then consider policy IIM which is optimal in M'. From lemma 1.3 we know there exists an equivalent policy my IT with the following form:
Probability of Action or Given my in M₁
St
Policy
5
T(0r
8t T(()((()))
Sa
S
p(a' a ~ U(A))
importantly we can see that I'' (ala)) = (als)) for all benign states & E S. Next, using lemma 1.1 know that there exists policy T" E II composed of three sub-policies ws, na, and Tg. By the construction of x" we know that (als)) = x (a(ss) = T(ass). furthermore we know the policy takes the following form in triggered states since (a [0(s)) 1:
Ta (a(s) = (1 n' (a (0(s))) p(a' = =ala' ~ ~((A) + T r'(a+15(s) r'alss
(1 (1=a((a(((s)) ppaa' = u(A)) + +<a+(8(s) (als)
= (aass)
additionally, since (0(x)((())) = we do not need to consider T since the non-markovian policy x' will never transition into internal state using Th.
With all of this we have found policy x" E II such that (s(a) =ssaa). Additionally, through
lemmas 1.3 and 1.1 we know that V.M (s) = (s) V.M (s) V.M (s) = V.M (s) for all E S
meaing x" is optimal in M. Therefore we have proven the desired result.
28


Reviews:
[{"review_id": "bKoD9r9oio", "reviewer": "Reviewer_7FdN", "summary": "In this paper, the authors propose DAZE, a reward-free backdoor attack against RL agents. DAZE creates triggered and dazed states: when the agent encounters a trigger, it is forced to perform suboptimal actions. Agents that take the target action (i.e., the adversary's desired behavior) avoid a penalty during training; therefore, they learn to execute the backdoor behavior. Experimental results (in both continuous and discrete action spaces) show that DAZE achieves a high attack success rate while inducing no drop in benign return. The authors also validated DAZE on real robotic hardware. \n\nI believe that the work is strong and well motivated. It explores simulator-level vulnerabilities in RL with a solid theoretical framework and experimental results.", "questions": "Q1. How sensitive is Daze to the shape of the trigger? Can a slightly perturbed trigger still activate the backdoor?\n\nQ2. Page 8, line 419, type (ore -> or, Interesction -> Intersection)", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "S1. Novelty: The authors highlight realistic adversary capabilities when the simulator is malicious: no read or write access to the agent's rewards, or cannot alter the rewards. Therefore, their attack, DAZE, operates under constrained but realistic assumptions unlike the prior backdooring work with the reward manipulation assumption.\n\nS2. Impactful Demonstration: In addition to empirical results in the MuJoCo and Atari domains, the real-world robotic example (also supplemented by a video) effectively shows the practical security implications of the attack.", "weaknesses": "W1. Practical Defenses: Although the attack is realistic, the authors have limited discussion on potential detection or mitigation strategies.\n\nW2. Trigger Design: The visual or input-level triggers are somewhat artificial. These triggers might not go unnoticed during deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E3r2Nhieiq", "reviewer": "Reviewer_K4RV", "summary": "The paper introduces a novel and highly practical threat model for backdoor attacks in Reinforcement Learning (RL). Instead of assuming an attacker can poison an agent's rewards (a common but strong assumption), this work considers a more subtle attacker: the developer of an \"untrusted simulator\". The paper provides formal proofs that this mechanism guarantees both attack success (the agent must learn the backdoor) and attack stealth (the agent's policy on the benign task remains optimal). Empirically, Daze is shown to outperform existing SOTA attacks (which require reward access) in continuous domains and match their performance in discrete domains.", "questions": "Please see the weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel and Practical Threat Model: The paper correctly identifies that as RL becomes more reliant on third-party simulators (MuJoCo, PyBullet, etc.) and cloud services, the \"untrusted simulator\" is a major, realistic security blind spot. It is the first to formally define and attack the \"reward-free\" threat model, which is a significant contribution to the field of RL security.\n\n\nStrong Theoretical Guarantees: The attack is not just a heuristic. The authors provide formal proofs (Theorem 1 and Theorem 2) that an agent optimizing its own returns must converge to a policy that is both successfully backdoored and stealthy. The detailed proofs in the appendix (Appendix E) appear sound.", "weaknesses": "1） Would the \"Daze\" attack successfully transfer in a complex, uncontrolled, \"in-the-wild\" environment that isn't almost identical to the training simulator?\n\n2） The paper claims this assumption is \"fairly weak\". However, one could argue that in highly stochastic or complex environments, a random or exploratory action in a specific state might be part of a near-optimal policy. If the \"punishment\" of random actions isn't severe, the agent won't be as strongly incentivized to learn the backdoor, and the attack would fail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2K65mAjLQn", "reviewer": "Reviewer_V5NT", "summary": "This paper studies a new framework of attack called Dazed attack where the attacker essentially has control over state transition of the environment and can perturb the action of the agent using a malicious simulator. The authors propose this as a \"reward-free\" threat model which contrasts to prior works like TrojDRL, SleeperNets which rely on reward poisoning. The attack mechanism work by punishing agent on non-compliance(on not following the target action) by forcing them to a \"Dazed\" state where agents actual actions are ignored and an uniform policy is played on behalf of agent thereby leading to poor performance. The authors provide theoretical guarantees on their attack framework and demonstrate the effectiveness of it through empirical experiments on Atari and Mujoco environments.", "questions": "1. Can you please provide a more robust justification of why your framework is \"weaker\"?\n2. Why is related work, experiments section completely miss comparisons to prior works on state and action perturbation attacks?\n3. For a defender, it would be easy to detect the presence of a dazed state by investing the fact that any action taken by agent there leads to similar poor performance, thereby declaring the environment to be malicious. How the attacker would protect themselves against such strong signal of identification?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and most of concepts are presented in an organized way.\n2. The paper claims to present a novel threat model which I think is misleading - more on this in the weakness section.\n2. The authors provide good theoretical guarantees on their attack algorithm.\n3. The attack framework leads to high empirical success rate on both environments outperforming reward-poisoning counterparts.", "weaknesses": "1. The paper's central claim that this is \"more constrained\" or \"weaker\" form of attack is quite misleading. The argument is highly one-dimensional focusing only of lack of reward access but it ignores the fact that the attacker is granted the complete control over the environment transition and action taken by agent to random action. This is an extremely powerful attack capability and it is not convincingly argued why this is 'weaker' than simply altering a scalar reward.\n2. The paper misses critical recent related works on state and action perturbations and the work should rightfully be compared to those papers rather than reward-poisoning attack papers. Since authors fail to compare against there more relevant baselines, the paper's positioning is not very clear and its claims on \"weaker\" form of attack appear highly overstated to the reviewer.\n3. The state perturbation attacks are highly detectable and the authors have failed to discuss any defense mechanism in the paper.\n\nReferences:\n\n[1]. Optimal Attack and Defense for Reinforcement Learning, McMahan et. al, AAAI-24.\n\n[2].  Robust Reinforcement Learning on State Observations with Learned Optimal Adversary, Zhang et. al, ICLR21. (Also look at their follow up works)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XDN0BqpZER", "reviewer": "Reviewer_R7TP", "summary": "This manuscript proposes a novel action-level backdoor attack paradigm, where the adversary is the developer of the simulator or training environment - a threat that compromises the RL supply chain. The proposed attack is evaluated using simulation benchmarks as well as real robotic hardware.", "questions": "- It is unclear how Algorithm 1 is implemented in practice. Is it explicitly integrated into the released simulator’s code? Are there any examples that illustrate this process? If the answer is yes, it would be helpful to clarify how this is consistent with the claim that “the attack does not modify the underlying dynamics of the simulator, but merely severs the relationship between actions chosen by the agent and those executed in the environment.” (P5, Line 266)\n\n- The use of third-party simulators seems more aligned with research settings rather than real-world applications. It would be helpful if the authors could provide examples of real-world cases where a victim relies on a third-party simulator to train an agent before real deployment. Even if real-world cases exist, since agents typically require post-training or fine-tuning to adapt to real environments, it would be interesting to discuss how such processes might affect the effectiveness of Daze.\n\n- One key point that needs clarification is the assumption that the simulator does not have access to the reward. In standard benchmark simulators (e.g., Atari, MuJoCo), the developer naturally knows the entire MDP, including the reward function. In contrast, for user-configurable simulators such as Isaac Gym, the simulator provider cannot know the specific task definition (including state space, action space, and reward), which would make the proposed attack impractical. It would therefore strengthen the paper to clearly define which simulator settings are considered and to explain the rationale behind assuming that the simulator cannot observe or alter the reward, while the state and action can be observed, and the state can even be altered.\n\n- The effectiveness of Daze in continuous action spaces may be highly dependent on the choice of the target action. For example, if the current target action is frequently selected by the agent when encountering a triggered state, the attack works well; however, if the target action is rarely chosen, the attack’s effectiveness may decrease significantly. In cases where the action space is very large, the probability of the target action being selected could approach 0, making it practically impossible for the agent to learn the backdoor. Lines 380-382 and Figure 3 in the manuscript seem to provide indirect evidence for this issue. I suggest that the authors more precisely discuss the attack’s boundaries and limitations.\n\n- Regarding the statement “only receive actions and return environment states, while rewards are computed externally. Therefore, this scenario, while realistic, violates a critical assumption of all prior backdoor attacks - the adversary can no longer alter or even observe the agent’s rewards”(P1, Line 53), I have two concerns: \n1.The phrase “rewards are computed externally” should be explained more clearly, as it forms a key part of the manuscript’s motivation. \n2.The logic seems somewhat inaccurate. Even if rewards are computed externally, the adversary may still be able to manipulate them. Since rewards are tied to the agent’s task objective, an adversary who knows the task goal can still approximate the reward and perform coarse-grained reward manipulation, even if the values are not perfectly precise.\n\n- The statement “Without altering rewards, traditional backdoor attacks are unable to sufficiently bias the agent’s expected returns and ...” (P2, Line 75) is inaccurate. Even if the adversary cannot alter the reward, it can still select high-reward transition data and tamper with the corresponding states and actions to inject a backdoor. \n\n- The five claimed contributions appear somewhat ambitious. I suggest reorganizing them for clarity and conciseness - for example, combining points 1 and 2, and points 3 and 4. \n\n- I have a different view regarding the statement “In line with prior work, $\\delta$ must be designed such that $S_\\delta$ is disjoint with S” (P3, Line 112). In realistic attack scenarios, $S_\\delta$ and S do not necessarily need to be disjoint. If $s^{-1} \\notin S$, it is considered anomalous and can be easily detected or filtered. On the other hand, if $s^{-1} \\in S$ occurs frequently, it may affect the agent’s performance on benign tasks, reducing the attack’s stealthiness. This clearly represents a trade-off.\n\n- It would be clearer to simply write R’ (P3, Line 113) as R to simplify the formulation and avoid potential confusion.\n\n- Why is it necessary to define $\\mathcal{U}$ (S) as a uniform distribution in Eq. (1)? Could other distributions be used, and if so, how would that affect the formulation or results? \n\n- Will $\\mathcal{L}_{adv}$ be added to the RL algorithm’s loss function? If not, I recommend not referring to it as a “loss function” here, as this could create confusion for the reader.\n\n- In my understanding, the simulator directly modifies the states (insert trigger). Then, how should we understand the statement “since δ and ϕ only introduce superficial alterations to states, all rewards will be computed with respect to true transitions that occur in M’”？(P5, Line 247)\n\n- Theorem 1 may appear unintuitive, as its validity seems directly related to the choice of $p_\\phi$ and the target action. For example, consider a simple counterexample: suppose there are two states, I and D, and two actions, “stay” and “move right”. The initial state is I, and executing “move right” transitions the agent to D, ending the task. If the target action is “move right”, Theorem 1 holds; however, if the target action is “stay”, Theorem 1 fails, because in this case the Daze state is ineffective - the agent will not execute the target action when observing the trigger in state I. This example does not violate the assumption that the random policy is not optimal. Therefore, I guess additional assumptions are needed for Theorem 1 to hold.\n\n- The statement “The only requirement for the malicious simulator developer is to choose which trigger and daze functions” (P6, Line313) seems somewhat inaccurate. As discussed above, the choice of the target action and $p_phi$ are also crucial for the attack’s effectiveness.\n\n- The constant C has a significant impact on the performance of TrojDRL and SleeperNets. Therefore, using the same fixed value across different tasks makes it difficult to conclusively demonstrate that Daze outperforms reward-alteration-based attacks. It would be valuable to include a comparison with the work “UNIDOOR: A Universal Framework for Action-Level Backdoor Attacks in Deep Reinforcement Learning”, which, to my knowledge, performs fully automatic reward alteration. \n\n- There are several missing definitions in the manuscript:\nP5, Line 240: $P_\\phi$ appears for the first time but is not defined.\nP5, Line 256: $\\tau$ appears for the first time but is not defined.\nP6, Line 297: the daze factor k appears for the first time but is not defined.\n\n- Some Typos：\nP2, Line 105: state spaces A.\nP3, Eq (2): Discrete is inconsistent with the text above (line 133). \nP5, Line 257: is the use of $\\leq$ before the second $\\tau$ incorrect?\nP6, Line 291: DLR", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The concept of performing backdoor injection via the simulator is particularly compelling.\n\n- Although the experiments on real robotic hardware remain preliminary, they constitute an good first step.", "weaknesses": "- Several claims and discussions in the manuscript are not presented with sufficient logical rigor.\n\n- Some important concepts and descriptions lack clarity, leading to potential confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)