Diffusion Guided Adversarial State Perturbations in Reinforcement Learning
Authors: Xiaolin Sun
Feidi Liu
Zhengming Ding
Zizhan Zheng
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
State Perturbation Attack
Diffusion Model
Primary area: reinforcement_learning

Abstract: Reinforcement learning (RL) systems, while achieving remarkable success across various domains, are vulnerable to adversarial attacks. This is especially a concern in vision-based environments where minor manipulations of high-dimensional image inputs can easily mislead the agent's behavior. To this end, various defenses have been proposed recently, with state-of-the-art approaches achieving robust performance even under large state perturbations. However, after closer investigation, we found that the effectiveness of the current defenses is due to a fundamental weakness of the existing $l_p$ norm-constrained attacks, which can barely alter the semantics of image input even under a relatively large perturbation budget. In this work, we propose SHIFT, a novel policy-agnostic diffusion-based state perturbation attack to go beyond this limitation. Our attack is able to generate perturbed states that are semantically different from the true states while remaining realistic and history-aligned to avoid detection. Evaluations show that our attack effectively breaks existing defenses, including the most sophisticated ones, significantly outperforming existing attacks while being more perceptually stealthy. The results highlight the vulnerability of RL agents to semantics-aware adversarial perturbations, indicating the importance of developing more robust policies.

Diffusion Guided Adversarial State Perturbations in Reinforcement Learning
Xiaolin Sun', Feidi Liu', Zhengming Ding', and Zizhan Zheng'
'epartment Computer Science, Tulane University
Shanghai Center for Mathematical Science, Fudan University (xsun12, zding1, zzheng3) @tulane. fdliu230m. fudan edu. cn
2
Abstract
Reinforcement learning (RL) systems, while achieving remarkable success across various domains, vulnerable adversarial attacks. This is especially concern in vision- based environments where minor manipulations of high-dimensional image inputs easily mislead the agent's behavior. To this end, various de- fenses have been proposed recently, with state-of-the-art approaches achieving robust performance even under large state perturbations. However, after closer investigation, we found that the effectiveness of the current defenses is due to fundamental weakness of the existing 1p norm-constrained attacks, which barely alter the semantics of image input even under a relatively large perturbation budget. In this work, we propose SHIFT, novel volicy-agnossee state perturbation attack to go beyond this limitation. Our attack is able to generate perturbed states that are semantically different from the true states while remaining realistic and history-aligned to avoid detection. Evaluations show that our attack effectively breaks existing defenses, including the most sophisticated ones, sig- nificantly outperforming existing attacks while being more perceptually stealthy. The results highlight the vulnerability of RL agents to semantics-awan adversarial perturbations, indicating the importance of developing more robust policies. Our code can be found this tiub Repo
diffusion- based
Introduction
Reinforcement learning (RL) has seen significant advancements recent years, becoming a key area of machine learning. RL's ability to enable agents to learn optimal decision-making policies through interaction with dynamic environments have led to breakthroughs in various fields. Beginning from AlphaGo (44. RL-based systems show the ability to surpass human performance in complex games. Beyond gaming, RL is driving innovations robotics, self-driving cars [30, and industrial automation, where agents learn to navigate, manipulate, and interact autonomously.
However, RL is vulnerable to various types of attacks, such reward and state perturbations, action space manipulations, and model inference and poisoning (26). Recent studies have shown that an
RI. agent be manipulated by perturbing its observation 24 60 and reward signals
(25]. and
well-trained RL agent can be confounded by a malicious opponent behaving unexpectedly [17]. In particular, malicious agent can subtly manipulate the observations a significant drop in performance and cumulative reward [60] 47). Such attacks exploit vulnerabilities the agent's 's perception systems, including sensors and communication channels, without needing to cause obvious disruptions. This susceptibility to minor perturaations raises major concerns, particularly for RL applications in security-sensitie and safety-critical environments.
trained RL agent, resulting in
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Multiple defenses have been proposed to mitigate state perturbation attacks. Regularization based improve robustness by smoothing policies and estimating worst-case rewards. CAR-DQN K further enhances robustness using the Bellman
methods like SA-MDP 60] and WocaR-M MDDP
36
Infinity-error. Adversarial training methods including alternating training [6]] and same-theoretical methods like PROTECTED (3) and GRAD 37
can potentially lead to more robust policies but are costly and do not scale to image-based inputs. More recently, diffusion-basee methods, such as DMBP (57] and DP-DQN [46). improve robustness by reconstructing the true states or modeling beliefs about them through denoising. These advanced defenses can withstand state-of-the-ar attacks like PGD [60], MinBest (2)), PA-AD 47], and high-sensitivity direction attacks [31.
However, we found that current attacks share two major shortcomings when applied agents with raw pixel images as input such as autonomous driving [30] and embodied AI [58]. First. with the exception of Korkmaz [31), current attacks usually restrict perturbed state to be within an e-ball of the true state, measured using an 1p norm, to avoid detection. However, this approach struggles with producing realistic perturbed input that within the natural data distribution and further constrains the attacker's search space for generating semantics-changin
perturbations. While this problem has been considered in supervised settings (54] 15 51 [3), to the best of our knowledge, Korkmaz [31 was the only prior study that identified this limitation for state perturbations in RL. However, the high-sensitivity direction attacks proposed in Korkmaz [31] adopt simple image transformations that mainly target changes in visually significant but not domain-spectie
semantics.
Consequently, the perturbed states either can be denoised by diffusion- based defenses or are non- stealthy from human perspectives (see our evaluation results in Appendix D). Second, previous attacks mainly focus on improving attack performance while ignoring the temporal dependencies across states, which is unique to RI.. As a result, they often generate states that are inconsistent with the agent's s previous observations (either perturbed or not). Recently, Liang et al. (37] looked into this problem by considering two consecutive time steps, but it still utilized an -n-rr constraint. On the other hand, Illusory attacks [16] require a perturbed trajectory to follow the same distribution as the normal trajectory, making it difficult to detect. However, this approach does not scale to -igh-dimensional image input. None of these attacks can easily modify the domain- semantics of the image input while keeping it realistic and plausible. The perturbed states generated by these attacks can be easily denoised with the help of history-conditionn diffusion model.
With these two shortcomings in mind, we propose SHIFT (Stealthy History-allgned diFfusion aTtack), a novel semantics- aware and policy-agnostic attack method that goes beyond the traditional I, norm constraint. Our approach is grounded in precise definitions of realistic states and three attack properties: semantic-altering, iistorically-alignee and jijjecttry-fathhiun which provide novel characterizations of semantics-aware and stealthy attacks in se- quential decision-makee maaiiiggn--ai.n....... As these metrics are computationally expensive to evaluate, we provide practical methods to approxi- mate them. Our main contribution is the development of a diffusion- based attack framework that utilizes classifier- free guidance to approximate history-aligned state generation, which is further improved using policy guidance to generate effective, realistic, and stcalthy state perturbations.
Goal
Broke!
Kop GO!
Start
Start
Start
Real Environment
SHIFT-O
SHIFT-I
Figure 1: A car approaches a crosswalk with pedes- trian ahead. The safe, optimal action is to brake. SHIFT- O removes the pedestrian from the agent's observation, while SHIFT-I creates an imaginary trajectory suggest- ing the car has already crossed. Both mislead the agent into moving forward, resulting in a collision.
Using this novel guided diffusion approach, we propose two versions of SHIFT: SHIFT-O perturbs the image input conditioned on the actual history to immediately induce suboptimal actions, while SHIFT-I guides the agent toward an imagined trajectory that is self-consistent, but ultimately leads to poor performance when actions are executed in the real environment. We compare these two methods in Figure [] with a working example in the Freeway environment given in Figure 6 in Appendix C SHIFT is policy-agnostic, capable of adapting to multiple vietim policies without retraining, and applicable to both value-based and policy-basee reinforcement learning methods.
2

---

Comprehensive evaluations show that it can break all known defenses, lower agents' cumulative reward in various environments by more than 50%. while being stealthier than prior attacks, as shown by lower reconstruction error, Wasserstein-1 distance, and LPIPS 62). and higher SSIM (50). Our results highlight that RI. agents with image input are vulnerable to semantics-aware adversarial perturbations, which has important implications when deploying them in sensitive domains.
2 Preliminary
2.1 Reinforcement Learning (RL) and State Perturbation Attacks
An RL environment can be formulated as a Markov Decision Process (MDP), denoted as a tuple (S, A, P. R, y, Pu), where S is the state space and A is the action space. P: S x A - A(S) is the transition function, where P(s'|s, [s, a) denotes the probability of moving to state given the current state s and action a. R: S A + R is the reward function where R(s. a.) E R& [ a) and R, is the reward in time step distribution. An RL agent wants to maximize its cumulative reward G = E/ Re over a time horizon T Z {oo), by finding a (stationary) policy π : + A(A).
8. 0z-1 Finally, y is the discount factor and fn the initial state
First introduced in 124], a state perturbation attack is a test-stage attack targeting an RL agent with a well-trained policy л. We consider the worst-case scenario where the attacker has access a clean environment and the victim's 's policy π and all deployed defense mechanisms. Further, the attacker has access to the true states in real-time. At each time step, the attacker observes the true state st and generates a perturbed state št. The agent, however, only observes & (and not 8;) and takes an action a based on its policy (-1$t). Note that the attacker only interferes with the agent's observed state and does not modify the underlying MDP. Consequently, the true state at the next time step is governed by the transition dynamics P(*++1 (8, (-) The attacker's objective to minimize the agent's is long-term cumulative reward. Let og denote the observation to the agent, which can be either 8, or 8t, depending on whether there is an attack at time 1.
Further, the attacker needs to remain stealthy to avoid immediate detection and achieve its long-term goal. To this end, previous state perturbation attacks [60] 47) restrict the attacker's ability by budget e, so that 5; E (s:) where (s,) is the Ip ball centered at for some norm p (typically an lx norm is used). However, ffute-of-the-are restricted attack even with a large c. Further, existing efforts [31] that try to go beyond the norm constraint cannot compromise these advanced defenses without being detected. detailed discussion on related work is in Appendix A
defenses 46 are able to mitigate the
2.2 Denoising Diffusion Probabilistic Model (DDPM)
Diffusion models, particularly Denoising Diffusion Probabilistic Models (DDPMs), have recently gained attention generative models that iteratively reverse predefined diffusion process to generate data from noise [22]. DDPM model consists of two phases: a forward diffusion process that gradually adds noise to the data and reverse denoising process to recover the original data.
The forward process is a fixed Markov chain that progressively corrupts the data Xo over T time steps by adding Gaussian noise. At cach step. the data evolve according to q (x; 14-1) = N (o:;; V1 B,T;-1, III), where /3 E (0, 1) controls the noise level at step
Forward Process.
Reverse Process. The reverse process manages to recover the data Tig from the noisy sample Tr- The reverse process is another Markov chain, parameterized by neural network co c i), which predicts the noise added to the data at each time step i in the forward process and can be modeled as:
Po (x;-1 xx) = N (z;-1i Ao (x,, i) of (2,,
(I)
where μ is the predicted mean and og is the variance of the reverse distribution at each time step i. Training. The training goal of DDPM learn model ea (z i) that predicts the noise added to a data point zo during the forward diffusion process. so (x i) in the reverse process is expressed in terms of the predicted noise co i):
1
B VI
He (z i) =
ca (2x, a,
(2)
VI- Br
3

---

(a) (b) under PGD
(c) St. Rotation
SHIFT-O
(e) SHIFT-I
Figure 2: Examples of true and perturbed states captured by the front camera of a vehicle in the AirSim driving simulator. a) the state. b) and e) the perturbed states under the PGD attack with to budget of and through rotation [31] by degrees counterclockwise. respectively. d) and e) are the perturbed states generated by our SHIFT-O and SHIFT-I attacks, respectively. Neither PGD nor Rotation attacks can alter the decision-related semantics. SHIFT-O removed pedestrians and bicycles at the crosswalk while being aligned with the real history. SHIFT-I lures the driver into thinking that the car has already crossed the crosswalk, when in fact it has not in the real environment. Note that SHIFF--I history-aligned with the observed trajectory but not the real trajectory.
where a; IT==1 (1 B,) is the cumulative product of (1 33) over time steps.
The variance o² (zi, i) can be predicted through a neural network or set by a predetermined scheduler. In DDPM, oz is set according to a fixed schedule as of BB The DDPM's training loss can be written as Siiple = Eza.i,e i. [||c co (xx, a)][?] where c N I) the noise added during the forward process. By minimizing this loss, the model learns to iteratively remove noise from Z;. ultimately generating high-quality samples from the learned data distribution.
Stealthy History-A Hisstry-AAlghht Aligned Diffusion Attack
We introduce SHIFT, novel policy-agnostic state perturbation attack built upon diffusion models that combines the classifier-free and policy guidance methods. We first discuss the motivation for using diffusion models generate perturbed states, where we also formulate the attack objectives by giving a novel characterization of realistic perturbed states and three attack properties: semantic-altering We then discuss how achieve these
satoorcaall-aalignee and trajectory-faithful (Section 3.1
properties and remain realistic in Section 3.2
3.1 Motivations and Attack Objectives
State-of-the-un perturbation attacks against image input (such as PGD, MinBest, and PA-AD) are performed by adding - no-norm constrained noise. While the high-sensitivity direction attacks [31 can go beyond the 1p-norm constraint, they are implemented through simple image transformations such as rotations and shifts that often change the output layout (see Figure 2c and Figure 7 in the Appendix). In both cases, the perturbed states often fall outside the set of states that can be generated by the underlying MDP (determined by the environment physics engine). Consider the front camera snapshots from an autonomous driving agent in Airsim [43 simulator shown Figure 2 Figure 2b shows that a perturbed state generated by the PGD attack with l.0 budget of 255 is easily recognizable due to noticeable noise, making the observation appear unnatural and unrealistic. On other hand, smaller perturbations are ineffective, especially in the presence of strong defenses. The main reason is that these attacks typically cannot induce semantic changes, As shown in Figure (b) with relatively large attack budget (e 15)) the pedestrians maintain their positions.
Our objective is go beyond 4p norm-constrainee attacks generate more powerful and stealthy state perturbations. A key observation is that carefully designed diffusion model can enable more effective attacks by generating cemantics-changing state perturbations (e.g., Figure [ to mislead the victim to choose suboptimal action, leading to significant performance loss. To generate realistic perturbed states that are semantically different from original states, we introduce the following definitions.
Definition 3.1 (Valid States) The set of valid states S of MDP (S, A, R. (.P)) is defined as: S = {s E lr TI, ds(s) > 0}. where II denotes the set of all possible (stationary) policies and d, (s) represents the stationary state distribution under policy n. from the initial distribution po

---

In other words, S consists of all states that can be reached by following an arbitrary policy и from the initial distribution Po- However, ensuring strict validity is intractable with limited amount of data (as in the case of diffusion models). Thus, we introduce the concept of realistic states as a more practical measure, based on the projection distance between a state s and the set of valid states S".
Definition 3.2 (Realistic States) A state s is defined as realistic if its projection distance to the set of valid states S* is bounded by a threshold ò. Formally, the set of realistic states 5" is defined as: 5 == {s E S |Proj (s) - a||2 I 51}, where Projs- (8) = arg min ||8 s||2 is the projection of onto S", and 61 is a predefined threshold.
Similar to the lp norm constraint, the realistic states constraint prevents the attack from generating arbitrary, meaningless perturbations, while still allowing semantically meaningful state changes. While the above condition captures our intuition on realism, is computationally expensive to verify since the set of valid states 5* is typically unknown and hard to estimate. Thus, we approximate the projection distance using the reconstruction error from an autoeneoder- based anomaly detector, which will be discussed in Section 3.2.3 We consider realistic states to be indistinguishable from valid states and pose realism as an objective of our approach. With the above definition, we further define perturbed state á to be semantically different from the original state if its projection to S differs from 8. Let (G) denote the set of projection points of & onto S which might not be unique depending on the state space S*, The condition can be formally stated as follows.
Definition 3.3 SSemannics-Changing true state 8 when 33' E D(3), 4s.
Changing States) A perturbed state & is semantically different from the
While the above definitions capture the quality of individual states, they ignore the dynamic nature of sequential gecision-makinn
RL and may lead to perturbed states that significantly deviate from history. Thus, we look for perturbed states that not only change the semantic meaning but also align with given history, formally defined as follows.
Definition 3.4 (History- Aligned States) Let (ot-k:........... 0t-1) denote the sequence of last k observed states by the agent (after projection onto )* Given the agent's 's policy a perturbed state & at time step t is aligned with H the agent's 's view if:
it E S(H-11) = (5c E S (Sa E D(s)) H.-1) >0}
where St is the random variable for the true state at t.
That is, & is aligned with the history any projection point in D(5;) is a reachable next state given the victim's 's observed projected history. This ensures that the perturbed state remains undetectable even when the agent employs a history-based detector. Note that instead of using the Projs- (-) operator, the above definition can be extended to incorporate the actual detector (if there is any) used by the agent. Similar to our realism measure, the above condition can be challenging to directly verify and optimize. We extend the concept of Wasserstein adversarial examples [51) the RL setting to approximately measure history alignment in our experiments as discussed in Section 4 We note that perturbations that are perfectly aligned with history are hard to detect without access to true states. On the other hand, a history-aligned state may deviate significantly from the true state, shown in Figure 2el and it is not stealthy when the agent does have some knowledge about the true state. To capture the latter case, we introduce the concept of trajectory faithfulness as defined below.
Definition 3.5 (Trajectory Faithfulness) Let H == (ot-k.... .... ot-1) denote the sequence of the last k observed states by the agent (after projection onto S*). The observed states are trajectory faithful if + l|: - s S 82, where s; denotes the true state, and 5g is a faithfulness threshold.
Figure 2d slightly alters the semantics of the true state by removing the pedestrians while keeping other elements consistent with the true state, improving faithfulness compared to Figure 2e Note that it is hard to directly calculate trajectory faithfulness due to the projection. Instead, we use SSIM [50]
and LPIPS 622 between the perturbed states and true states as quantified metrics.
3.2 Diffusion-based State Perturbations
In this section, we discuss SHIFT, our diffusion-based attack that can generate realistic and stealthy state perturbations with three properties defined above: Semantic Change, Historical Alignment,
5

---

and Trajectory Faithfulness. Our attack consists of two stages: the training stage and the testing stage, During the training stage, we use data generated by the clean environment (i.e., the MDP) to train conditional diffusion model to generate states that are realistic and history-aligned. We further train an autoencoder to detect unrealistic states. In the testing stage, we employ the pretrained diffusion model to generate perturbed states guided by (1) the defender's s policy π and corresponding state-action value function Q², which provides guidance toward a perturbed state that has lower (T (()) value, and (2) the pre-trained autoencoder, which further enhances the realism of the perturbed states, (3) a given history to be aligned with. Figure in the appendix illustrates the two stages of our attack and the main components involved, with each discussed in detail below.
3.2.1 Generating Eistory-Allgned States via Conditional Diffusion
We first describe the training of the conditional diffusion model, which is built upon the classifier-free guidance approach (1 that can generate both unconditional and conditional samples, enabling the model to guide itself during the generation process. We train a classifier- free guidance model conditioned on a history to generate the next state & that follows the given history. This ensures that the generated next state 3, is realistic and aligned with the history, such that 3, is stealthy from both the static and dynamic views. It is important to note that the true next state 8; is independent of the victim's 's policy π when the history (including previous states and actions) is given. Thus, we can train this diffusion model with classifier free guidance without requiring knowledge of the specific victim's 's policy However, separate diffusion model needs to be trained for each distinet environment.
Specifically, let 7--1 = (8I-k; at-ks 8t-1, 11 } be the true history from time - k to time I, where k > 1 is parameter. In our setting, the model is trained with both class-conditional data (8t; 1t-1) and unconditional data 8t by randomly dropping 7:-1 with a certain probability. The history T--1 and true state 8t are sampled from trajectories generated in clean environment by following a well-trained policy Fref (independent of the agent's policy) with exploration to ensure coverage.
The noise prediction network cg (ss),, i, Tr-1) trained to learn both conditional and unconditional distributions during the training. When generating perturbed states at the testing stage, where the reverse process applied, the noise prediction can be adjusted using a guidance scale T(i) as follows:
E r(i)co(s), i, T--1) + (1- ((())(()ssss i),
(3)
where 7:-1 is the given history, and I(i) controls the strength of the guidance. Note that we have two time step variables here, where is the time step in an RL episode and is the index of the reverse steps in the reverse process. With classifier-free guidance, the model learns a distribution of & conditioned on a historical trajectory T--1 The attacker can set given history T as a conditioning factor, forcing the generated perturbed state &, to align with T--1- Consequently, the classifier-free guidance enhances dynamic stealthiness and achieve historical alignment.
Since the classifier-free model is designed to generate the true next state based on the history T--1, the generated next state š; is expected to be close to the true state st. Consequently, while the generated next state 9, may not be exactly the same as St to be classified as a valid state, St is sufficiently close to 8t to be considered as a realistic state according to Definition 3.2] This is confirmed in Figure 3c which shows the average 22 distance between the perturbed states generated through the conditional diffusion model and the true states in the Freeway environment. Note that the la distance gives upper bound on the realism measure in Definition 3.2 as the true state may not be the closest state in S* with respect to the perturbed state. The results show that states generated by the diffusion model conditioned on history are closer to the true states compared to those generated by PGD, even with small budgets, which highlights the realism of our method.
3.2.2 Generating Semanties Changing Perturbations via Policy Guidance
perturbed state 5; that is solely generated by the history-conditioned guidance can not mislead the victim toward suboptimal action. To achieve our attack objective of decreasing agent performance, we introduce a policy guidance module that is similar to classifier guidance at the testing stage that can change the semantics of the true state s, Classifier guidance is a method to improve the quality of samples generated by incorporating class-conditional information [1]. The core idea is to utilize a pre-trained classifier po (vlx), where y represents the class label, to guide the reverse diffusion process toward generating samples conditioned on desired class. In our context, we can use the state-action value function of the victim's 'ss policy Q2 (s, a) for guidance. Since the policy guidance is

---

applied during testing and is independent of the pre-trained conditional diffusion model, our approach remains aolicy-agnostic, allowing it to effectively adapt multiple victim policies without retraining.
At each reverse time step i, the reverse process is modified by adjusting the mean of the noise predic- tion model 6, with the gradient of the policy with respect to &t, that is, vV log Q" (st, ($))). This guidance steers the generation process towards samples that lead the agent take actions that have lower state- action value under true states, ultimately changing the semantics and caus- ing victim's 's performance drop at the same time. As shown in [14], for the unconditional re- verse transition in Q the modified reverse process with policy guidance can be expressed as: p(š; st. Q*) = N ( :oo(,,)) t) V s; log Q* (5))). of In our scenario, how- ever, policy guidance is applied to diffusion model conditioned on the history π--1. Typically, policy guidance cannot be applied to a conditional diffusion model because the gradient term becomes <v log (*), π T(1)) which is hard to compute through Q Fortunately, in our RL setting, policy guidance and classifier-free guidance can be combined as shown in the following theorem.
Theorem 3.6 The reverse process when sampling from a history-conditione DDPM model guided by the victim's 's state-action value function Q" is given by
p(3, 1
:14 V log Q (st. (B
where p4 is derived from eq in (3 as given by 2 and oî is determined by the variance scheduler By-
Theorem shows that policy guidance and classifier-free methods can coexist without interference. While this is generally not true, it holds in our setting because given the two conditioning variables Q and T--1, the noise predicted by classifier-free guidance depends only on T--1, while the gradient from policy guidance depends solely on Q" (Proof in Appendix B). With policy guidance modifying the reverse process, the perturbed state s,, conditioned on (Q", T--1), differs from states conditioned only on T(-1- Thus, & is semantically distinct from the true state st and achieves attack performance.
3.2.3 Enhancing Realism via Autoencoder Guidance
Since the classifier guidance method introduces additional gradient information during the reverse process, the generated perturbed state & may deviate from realistic states. To address this, we incorporate an autoencoder-based anomaly detector trained on clean data. Autoencoders [66] are widely used in unsupervised anomaly detection by measuring reconstruction error defined as the l2 distance between an input state and its reconstruction, which is L(s,, AE(s;) ||s - AE(())) Since the autoencoder A E-) trained solely on clean states, it assigns significantly higher errors to anomalous or unrealistic inputs, providing effective signal to assess the realism of generated states. Thus, we can use the pre-trained autoencoder at the testing stage to enhance the realism the perturbed states, following policy guidance, which can be achieved by of = = V L(8 AE (E(S))
3.2.4 SHIFT-O//-Tradoof between Historical Alignment and Trajectory Faithfulness
Both SHIFT-O and SHIFT-I share the same pre-trained diffusion model and the policy and realism guidance modules, differing only in their history conditioning during attacks. SHIFT-O uses the true history Tt-1 = {st-k- at-k: St-1, a } as the condition to generate a perturbed state that remains aligned with the true trajectory. In contrast, SHIFT-I uses the victim's 's observed history (i.e., the perturbed history), ア = (o: k:: (t-k, O Ø), with the last action dropped, as the condition. Thus, SHIFT-I generates perturbed states that are aligned with the observed trajectory. Dropping the last action grants the model greater flcxibility to sample perturbed states that might follow alternative, suboptimal actions, thereby increasing the likelihood of misleading the agent.
SHIFT-I aligns with the (perturbed) history and can alter decision-relevant semantics by conditioning on the perturbed history. It maintains self-consistent, policy-guides imaginary trajectories, making the attack difficult to detect when the agent lacks access to the true states. This represents a realistic attack scenario, when attacks occur over a short time span and harmful consequences may already have unfolded before the agent realizes it was misled in the last few time steps. similar illusory attack was recently proposed in [16] however, it does not scale to image input. Further, such attacks can lead to rajectooiis that deviate significantly from the true ones, as shown in Figure [ and are therefore detectable when the agent has access to even a single true state before the attack succeeds.
7

---

SHIFT-O improves serectory-faithflleee by conditioning on true historical states, while altering se- mantics via policy guidance (c.g., removing pedestrians v.s. shifting the whole scene). However, these semantic changes may introduce subtle historical inconsistencies as abrupt object disappearance will disrupt temporal coherence, resulting in imperfect historical alignment. Such inconsistency can poten- tially be detected by agent with sufficient domain knowledge and advanced detection capabilities. However, as we show in our evaluations, this is a nontrivial task for complex environments.
Traditional -y-norm constrained attacks (e.g., PGD, MinBest, PA-AD) are history aligned and trajec- tory faithful under small attack budget, by introducing minimal pixel-level perturbations. Similarly, high-sensitivity direction attacks from Korkmaz [1 maintain these two properties by applying simple image transformations that alter visually significant but not Nomain-spectin semanties. Their lack of semantic-altering capability causes them to be ineffective against diffusion- based defenses.
Our framework reveals a fundamental trilemma in adversarial perturbations in sequential decision- making with image input. Current methods are optimized for only two out of three critical properties: semantic-altering (Def (.3), iistorically-alignne (.44 and trajectory-faithtu (De1 Finding an attack that better satisfies all three properties than current methods remains an open problem.
3.3 Implementation Details
Previous attack methods, such PGD and PA-AD, assume the attack happens at every time step. In contrast, SHIFT only allowed to attack a fraction & of the time steps. To determine when to attack, we adopt the concept of state importance weight from Liang et al. [36), defined as wss) = max,, E Q(s, ((s.aai)minngee min Q(s, a2). At time step t, SHIFT computes w(st) and compares it to the top ε-th percentile of all previous importance weights. If (s;) falls within the top percentile and the number attacks so far is less than f 5, SHIFT injects an attack at step is worth noting that SHIFT-O passes the true state to the agent when attack is injected. In contrast, SHIFT-I continues to generate states conditioned on the perturbed history and true actions without policy guidance, in order to maintain consistency within the imaginary trajectory.
Although our theoretical analysis Section 3.2.2 uses the DDPM method for simplicity, we plement our attack using the EDM formulation 29 which is score- based diffusion method that requires fewer reverse diffusion steps, improving sampling efficiency and enabling real-time attacks. We build upon previous work (3). which applied EDM-based conditional diffusion models to Atari world modeling. We apply the technique from [5] to optimize policy guidance, which first calculates the output sample 3, without any attack, then apply policy guidance to guide the reverse process. Details on the EDM formulation and implementation are in Appendix C We provide algorithms for training and sampling of our attack in Algorithms T and 2 in Appendix C.6
Table I:
Episode reward of SHIFT-O and SHIFT-I against various defense methods in different environments with attack frequency 1.0 and 0.25. All results reported with mcan and std over 10 runs
Pong
F Freeway
SHIFTT-1-0225
DON-No Atiack
21.0100
21.010.0
21.01000
21,010.0
34.1100.1
31.1100.
34.110.1
31.11011
DQN
-21.0.0.0
-2088004
-21.010.
-21.0:0.0
12210.8
19.612.1
12.216.3
6.2.19
SA-DQN
-210000.
-208+004
-20.8:00.4
-210::00
19.41330
27.6+005
19.411.5
19.6#2222
WocaR-DON
-210--0.
-21,0+00
-21.0+B0.
-210=000
20.2+00.8
26.8+1.3
20.2+1.3
20.8-04
CAR-DON
20.810.4
20.01.07
DP-DQN
-206:0.5
99744
-21.0:0.0
-2100:00
2188332
28.0112
10.6122.5
7
DMRP
-14.0+4.9
-984-.5
-20.6+0..9
-20.2+08
22,04-077
31.0+12
19.2+2.9
14.6-2.
DOom
Aininn
Model
SHIFT-O-L3
SHIF
SHIPT-L-LO
SHIFT-14
SHIFT-0-1D
SHIPT-0-122
SHIFT-E-
DON-No Atlack
75.1144
754.14,4
7544144
75,444
40.310.3
10.1003
40.310.3
10,100
DQN
2540048.2
55.01411
-318.029.77
-326.020.44
9.8423
20.011.
6.240.1
10.0=29
SA-DON
33..0=13.
62224466
337.0243.11
3088-2064
$.410.4
6.72166
3.740055
8.64:2.5
WocaR-DON
299.6.143.
38.1220.3
310.016.1
289.1117111
5.215.0
20.7117.
5.11-4.2
7.513.9
335001158
6141155
0.910.3
DP-DQN
116.4-173.22
640:1006
303.012.7
252.0=1729
18R112.3
178188
4.8.3.11
10.5136
DMRP
-1848+251.77
654498
302.042.7
-2562-1804
208+12.0
22,84-00
62+00.7
94+49
4 Experiments
We evaluate SHIFT using four Atari environments 7. Doom game 12 and Airsim 43 autonomous
driving simulator. We consider state-of-the-art defenses including SA-DQN 60), WocaR-DQN [36. CAR-DQN (35) and two diffusion- -aased defenses: DMBP (57). which test-stage defense, where the victim diffusion model conditioned on perturbed history to recover true states, and

---

Table 2: Comparison with different attack methods. We compare our SHIFT attack with PGD, MinBest, and PA-AD with 15/55,15//555 budgets and rotation (degree and transform attacks, and report reward, deviation rate, reconstruction error, Wasserstein distance, SSIM, and LPIPS against DPDDQN.
Env
Freeway
Attack Method
Reward
Dev (Sa))
Recons.
Wass, (x 1x100"1 31
SSIMT
LPIPS!
PGD-1/255
0.9
0.22
3.45
0.9972
0.0001
0.0018
0X000
P5D5-15255
+-.0
3.2 4 0.1
4.36
0.3
1.63 + 1.1
0.7461
0.0086
0.1669 0.022
MinBest-1/22
30.2 00.21.3
3.7 4
3.45 0.3
0.88 = 0.2
0.9965 = 0.0002
0.0123 = 0.0006
MinBest- MinBBest-5/255
29.1
0.2
5.35
0.2
1.75 0.6
0.6355
0.0067
0.1555
0.023
PA-AD-1/255
1..0
6.04
3.47
0.3
0.82 + 0.2
0.9967
0.0001
0.0027 + 0.00UT
PA-AD-11/2555
2..1
10.3 1.0
6.06
= 0.2
1.34 = 0.5
0.5891 = 0.0064
0.2368 1 0.0232
PA-AD-TC-IS72
1.2
0.2
6.01
0.2
1.63 0.1
0.5962 = 0.0055
0,2313 = 0.0235
Rotation Degree
27.2
26.9 0.5
6.41
0.2
0.80 0.2
0.5237 0.0022
0.0351 1 0.0011
Transtorm 1,0)
26,8
22.1
+ 1.0
9.32
0.1
0.80 = 0.2
0.6015 = 0.0071
0.0741 = 0.0088
SHIFT-O-10
21.8
3.2
22.8
+: 2.6
1.02
+ 0.5
0.89
+ 0.3
0.9990 + 0.0014
0.0008
+ 0.0014
SHIF T-I-1.0
2.5
51.4
1.05
0.5
0.84
0.9904 0.0043
0.0175
+ 0.0124
DP-DQN 46]. which uses a diffusion-basse denoiser on top of pre-trained pessimistic policy. We set the history length k: = for the two history-based defenses and our attacks. All other hyper-parameters of are given in Appendix C.7 We compare SHIFT with loo -nom constrained PGD (60, MinBest (24], and PA-AD [47] attacks in the Atari Freeway environment, as well as two high-sensitivity direction-based attacks (rotation and transform) from Korkmaz [31]. The PGD attack aims to force the victim into taking non-optimal actions, while the MinBest attack minimizes the logit of the best action. PA-AD and its temporally coupled version 37 (PA-AD-TC) use RL to determine the best attack direction. Rotation/transform attack rotates/shifts the state by a given amount.
We evaluate our method using six metrics from four perspectives: 1) Attack Effectiveness: Reward (average episode return over 10 runs) and Deviation Rate (percentage of steps where the victim's action ( differs from (s:)): 2) Perturbed States Realism: Reconstruction Error (12 distance ||& A A I|2 via realism autoencoder detector); 3) Historical Alignment: Wasserstein-1 distance between two consecutive perturbed states, which measures the cost of moving pixel mass and better represent image manipulations than the Ip distance. Wong et al. [51] has used it to generate beyond Ip norm constraint adversarial examples in supervised learning; 4) Trajectory Faithfulness: SSIM (50) and LPIPS [62 (perceptual similarity between & and 8;; unavailable to the agent during testing, so SSIM and LPIPS cannot be computed). Additional results are provided in Appendix [D] including results on a continuous action space environment using the PPO policy, detectability results on various attack methods, ablation studies on DDPM vs. EDM diffusion architectures, time complexity comparison of different attacks, diffusion model generated images detection results and a comparison with high-sensitivitty direction attacks 31.
4.1 Main Results
Attack Effectiveness. Table | presents the performance of SHIFT-O and SHIFT-I under attack frequencies 1.0 and 0.25 against variety of defense mechanisms. from four perspectives, comparing baseline attacks and our SHIFT variants against the DP-DQN defense. Our results show that both variants significantly degrade the return of the vanilla DQN agent and that under regularization-baat based defenses outperform other baselines due to their ability to denoise using tistory-conditione signals, our SHIFT attacks still induce substantial reward drops and high deviation rates. In contrast, baseline attacks are largely ineffective (shown in Table because they fail to introduce semantically meaningful changes. SHIFT-O bypasses diffusion-based defenses by generating perturbed states that manipulate environment-specee -spenffi-- semantics, while SHIFT-I constructs plausible imaginary trajectories that lead the agent to take suboptimal actions in the true environment.
reports metries
defenses, even under low attack frequency. Although diffusion-
Attack Stealthiness. In addition to achieving stronger attack performance, both SHIFT variants exhibit higher stealthiness across all evaluation metrics compared to baselines, highlighting the effectiveness of using conditional diffusion to generate stealthy yet successful attacks. Our attacks generate the most realistic perturbed states compared to baseline attacks, as evidenced by the lowest average reconstruction errors. They also achieve comparable historical-alignment and trajectory- faithfulness to ly-norm-based attacks with small attack budgets and are better than those attacks with large budgets. SHIFT-I achieves stronger overall attack effectiveness than SHIFT-O, but has lower SSIM and higher LPIPS, indicating less trajectory faithfulness. However, it exhibits better temporal

---

Recemntratton Poese
ditatien Defferee Bakiy Custuur Sonngsh
Parcettag 00-42
PO0
Des Vid kpp
aat.uaol Tim Stop
230
Step
Potay Outterce Strongth
(a) Realism Guidance
(b) Policy Guidance Strength
(c) l Distance
Figure 3: Ablation Study Results. a) shows the rolling average of reconstruction error (from the autoencoder-ben basedd realism detector) of our generated perturbed states with and without the realism enhancement. b) shows the reconstruction error, deviation rate under different policy guidance strengths with SHIFT-O attack. c) shows distance between perturbed and true states (84 x 84 grayscale images). (a) and (b) use the vanilla DQN policy.
consistency, reflected by lower average Wasserstein distance. Lastly, results in Table 2 support the presence of fundamental trilemma: no attack dominates across all key metrics simultaneously.
4.2 Ablation Studies
Ablation on Realism Guidance. Figure 3a illustrates the la reconstruction error, which is defined as ||s: A (2s))l2, for generated perturbed states both with and without the realism enhancement component. The figure demonstrates that, by incorporating realism enhancement, the 12 reconstruc- tion error is significantly reduced. This reduction indicates that realism enhancement effectively contributes to the generation of perturbed states that are more stealthy and less likely be detected.
Ablation on Policy Guidance Strength Figure (3b illustrates the performance of our attack across various levels of policy guidance strength 2.2 The figure indicates that as the strength increases, the effectiveness of our attack improves, leading to higher manipulation and deviation rates. However, this increased strength also results in higher l2 reconstruction error, which negatively impacts the realism of the generated perturbed states. Consequently, there exists a trade-off between attack effectiveness and stealthiness when selecting different policy guidance strengths.
5 Conclusion and Future Work
We introduce a novel agnostca diffusion based state perturbation attack for reinforcement learning (RL) systems that extends beyond the traditional (n-norm constraints. By leveraging conditional diffusion models, policy guidance, and realism enhancement techniques, we generate highly effective, semantically distinct, and stealthy attacks that cause significant reduction in cumulative rewards across multiple environments. Our results underscore the urgent need for more sophisticated defense mechanisms effectively mitigate semantic uncertainties. We also propose fundamental trilemma in adversarial perturbation attacks in the image domain: any existing attack method can only optimize two of three critical properties Semanntie Change, Historical Alignment, and Trajectory Faithfulness. Finding an attack that better satisfies all three properties than current methods remains an open problem.
Acknowledgments
This work has been funded in part by NSF grant CNS-2146548. We thank the anonymous reviewers for their insightful feedback.
10

---

References
[15] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. A rotation and a translation suffice: Fooling cnns with simple transformations. http://pppeeeieoeeee/oooooo///aa Crispin Agar MAD: Anomaly detection.
[1]
https://crispinagal hww.appiiinggggggguu..... io/blogs/
mad- www.aaallddeetttiaeee.. html 2024.
[2] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision- maainn? arXiv preprint arXiv:2211.166577 2022.
[3] Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. arXiv preprint arXiv:2405. 12399, 2024.
[4] Shumeet Baluja and Ian Fischer. Adversarial examples from adversarial generative networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence(([CA)), 2018.
[5] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In International Conference Learning Representations((CLRR) 2024.
[6] Lucas Beerens, Catherine F. Higham, and Desmond J. Higham. Deceptive diffusion: Generating synthetic adversarial examples, 2024. URL htttp:://rxxiii http://aaxvvv..o oreeiiiorrg/aae org/org///// abs/ /40/. 19807
[7] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research. 2013.
[8] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference on Learning Representa- tions(ICLR), 2024.
[9] Jianqi Chen, Hao Chen, Keyan Chen, Yilan Zhang, Zhengxia Zou, and Zhenwei Shi. Diffusion models for imperceptible and transferable adversarial attack, 2025.
[10] Kangjie Chen, Shangwei Guo, Tianwei Zhang, Xiaofei Xie, and Yang Liu. Stealing deep reinforcement learning models for fun and profit. In Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security, 2021.
[11] Xinquan Chen, Xitong Gao, Juanjuan Zhao, Kejiang Ye, and Cheng-Zhong Xu. Advdiffuser: Natural adversarial example synthesis with diffusion models. In International Conference on Computer Vision (ICCV), 2023. doi: 10. 109/1CCV51000.2023..... 200.200000..02.0444.....
[12] Jacob K Christopher, Stephen Baek, and Ferdinando Fioretto. Projected generative diffusion models for constraint satisfaction. arXiv preprint arXiv:2402.03555, 2024.
[13] Xuelong Dai, Kaisheng Liang, and Bin Xiao. Advdiff: Generating unrestricted adversarial examples using diffusion models, 2024. URL Bttp/httt:///aaiitt
httgg/bb/23000 12499
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In Advances in neural information processing Nystems 2021.
2018.
[16] Tim Franzmeyer, Stephen Marcus McAleer, Joao F. Henriques, Jakob Nicolaus Foerster, Philip Torr, Adel Bibi, and Christian Schroeder de Witt. Illusory attacks: ceermation-theoreee detectability matters in adversarial attacks. In International Conference on Learning Represen- tations(ICLR), 2024.
[17] Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. In International Conference on Learning Representattioss/(CLR)), 2020.

---

[18] Qi Guo, Shanmin Pang, Xiaojun Jia, Yang Liu, and Qing Guo. Efficient generation of targeted and transferable adversarial examples for -ssounlanguage models via diffusion models, 2024.
2404 10335
[19] Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. In Conference on Neural Information Processing Systems(NeurlPP) 2023.
[20] Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao. Robust multi-agent reinforcement learning with state uncertainty. Transactions on Machine Learning Research, 2023.
[21] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint 1xXiv:2207.12598 2022.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in neural information processing systems(Neur/PS), 2020.
[23] Mengdi Huai, Jianhui Sun, Renqin Cai, Liuyi Yao, and Aidong Zhang. Malicious attacks against deep reinforcement learning interpretations. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, 2020.
[24] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv: MeXivv11002 17w.:1022.222444 2017.
[25] Yunhan Huang and Quanyan Zhu. Deceptive reinforcement learning under adversarial manipu- lations cost signals. In Decision and Game Theory for Security(GameSee), 2019.
[26] Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala I. AI- Al-Fuqaha, Dinh Thai Hoang, and Dusit Niyato. Challenges and countermeasures for adversarial attacks on deep reinforcement learning. CoRR, 2020.
[27] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0999) 2022.
[28] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. 2024.
[29] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basedd generative models. In Advances in neural information processing sys- lems(NeurIPS), 2022.
[30] Bangalore Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Kumar Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. CoRR, abs/2002.004444, 2020. URL 00444
[31] Ezgi Korkmaz. Adversarial robust deep reinforcement learning requires redefining robustness. In Proceedings of the AAAI Conference on Artificial Intelligence(AAAI), 2023.
[32] Ezgi Korkmaz and Jonah Brown-Cohen. Detecting adversarial directions in deep reinforcement learning to make robust decisions. In International Conference Machine LearningICMML), 2023.
[33] Xian Yeow Lee, Sambit Ghadai, Kai Liang Tan, Chinmay Hegde, and Soumik Sarkar. Spatiotem- porally constrained action space attacks deep reinforcement learning agents. In Proceedings of the AAAI conference on artificial intelligenc((AAAl), 2020.
[34] Edouard Leurent, An environment for autonomous driving Docision-making. https://githn: com/ htmm//leeuaanthiihhaa-eeenee 2018
[35] Haoran Li, Zicheng Zhang, Wang Luo, Congying Han, Yudong Hu, Tiande Guo, and Shichen Liao. Towards optimal adversarial robust learning with bellman infinity-errorr In International Conference Machine Learning[CCML), 2024.
12

---

[36] Yongyuan Liang, Yanchao Sun, Ruijie Zheng, and Furong Huang. Efficient adversarial training without attacking: Worst-cuse-awar robust reinforcement learning. In Advances in Neural Information Processing Systems(Neur/PS), 2022.
[37] Yongyuan Liang, Yanchao Sun, Ruijie Zheng, Xiangyu Liu, Benjamin Eysenbach, Tuomas Sandholm, Furong Huang, and Stephen Marcus McAleer. Game-theoretic robust reinforcement learning handles comporally-coupled perturbations. In International Conference on Learning Iupreseenaation((CI.. 2024.
[38] Jiang Liu, Chun Pong Lau, and Rama Chellappa. Diffprotect: Generate adversarial examples with diffusion models for facial privacy protection, 2023. URL
www. /bs/
2305. 13625
[39] Xiangyu Liu, Chenghao Deng, Yanchao Sun, Yongyuan Liang, and Furong Huang. Beyond worst-case attacks: Robust rl with adaptive defense via non-dominated policies. In International Conference on Learning Representatioos//CLLR), 2024.
[40] Buqing Nic, Jingtian Ji, Yangging Fu, and Yue Gao. Improve robustness of reinforcement learning against observation perturbations via lco lipschitz policy networks. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixir Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium Educational Advances in Artificial Intelligence,
24/IAAI 2025. URL pattps://do.
org/10. 1609/ aaai. www.13811.... 29360
[41] E. S. PAGE. Continuous inspection schemes. Biometrika, 1101-2):100-115, 06 1954.
[42] Alessio Russo and Alexandre Proutiere. Balancing detectability and performance of attacks on the control channel of markov decision processes. In 2022 American Control Conference (ACC), 2022.
[43] Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robolics, 2017. URL tttt::://arxiv.rrg/aaa
05065
[44] David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lil- licrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
[45] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations(/CCLR)) 2021.
[46] Xiaolin Sun and Zizhan Zheng. Belief-enriched pessimistic qelearnnng against adversarial state perturbations. In International Conference on Learning Kepresentations/| 2024.
[47] Yanchao Sun, Ruijic Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl. International Conference on Learning (eppresentations((CLk)) 2022.
[48] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.0m 00888.00677 193, 2022.
[49] Zhendong Wang, Jianmin Bao, Wengang Zhou, Weilun Wang, Hezhen Hu, Hong Chen, and Houqiang Li. Dire for cuffusion-generaan image detection. In International Conference on Computer Vision (ICCV), 2023.
[50] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004. doi: 10.1109/TIP.003.88 819861.
[51] Eric Wong, Frank Schmidt, and Zico Kolter. Wasserstein adversarial examples via projected sinkhorn iterations. In International conference on machine learning[ICML)) 2019.
13

---

[52] Marek Wydmuch, Michat Kempka, and Wojciech Jaskowski. ViZDoom Competitions: Playing Doom from Pixels. IEEE Transactions on Games, 2019.
[53] Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Advgan: Generating adversarial examples via generative adversarial networks. In International Joint Conferences on Artificial Intelligence(JJCAI), 2018.
[54] Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially trans- formed adversarial examples. In International Conference on Learning Representations((CLR)) 2018.
[55] Zikang Xiong, Joe Eappen, He Zhu, and Suresh Jagannathan. Defending observation attacks in deep reinforcement learning via detection and denoising. In 2022 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2022.
[56) Haotian Xue, Alexandre Araujo, Bin Hu, and Yongxin Chen. iiffusion-based adversarial sample generation for improved stealthiness and controllability, 2024. URL htttps://axxiiv hrggabs/23055..
[57] Zhihe YANG and Yunjian Xu. DMBP: Diffusion model-based predictor for robust offline reinforcement learning against state observation perturbations. In International Conference on Learning Representations//CCA.. 2024.
[58] Weirui Ye, Yunsheng Zhang, Haoyang Weng, Xianfan Gu, Shengjie Wang, Tong Zhang, Mengchen Wang, Pieter Abbeel, and Yang Gao. Reinforcement learning with foundation priors: Let embodied agent efficiently learn on its own. In 8th Annual Conference on Robot Learning. 2024.
[59] Bohang Zhang, Du Jiang. Di He, and Liwei Wang. Rethinking lipschitz neural networks and certified robustness: A boolean function perspective. In Advances in Neural Information Processing Systems(NeurlPS), 2022.
[60] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho- Jui Hsich. Robust deep reinforcement learning against adversarial perturbations on state observations. In Advances in Neural Information Processing Systemss(eur//PS) 2020.
[61] Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Representations((CARR 2021.
[62] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. In IEEE/ / CVF Computer Vision and Pattern Recognition Conference(CVPR), 2018.
[63] Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu, Adaptive reward-poisoning attacks against reinforcement learning. In International Conference Machine LeaM) 2020.
[64] Sheng Zhao, Dheeru Dua, and Sameer Singh. Towards natural and robust adversarial examples. arXiv preprint arXiv:
10. 17 2017.
[65] Sheng Zhao, Dheeru Dua, and Sameer Singh. Generating adversarial examples with adversarial generative nets, arXiv preprint arXiv:1801.02613 2018
[66] Chong Zhou and Randy C. Paffenroth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The theoretical and experiment results support our claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss a Trilemma of current state perturbation attacks in the section
3.2.4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
Full proof of our Theorem 3.6 is in the Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We report our experiment details and hyperparameter in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We provide our code in the supplementary material
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwiiicgguie///eeeeeeeeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
They are stated in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have reported mean and std of all our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes] We specify testbench information in Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp:://neuripeee http://urrs.cccuuuippeeee httppp/ssccppp.ttttttttttttteeea
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide broader impact section the beginning of the Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have properly cited every previous work we build upon.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppsiipteeeee www.reeswthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
19

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
20

---

Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httess/s//eerrippoccc///o/rreees for what should or should not be described.
21

---

Appendix
Impact Statement
Our work introduces SHIFT, novel approach to perturbation attacks in reinforcement learning (RL) by altering the semantics of the true states while remaining stealthy from both static and dynamic perspectives. SHIFT demonstrates outstanding performance, successfully compromising all otate-of- the-art defense methods. This highlights the urgent need for more sophisticated defense mechanisms that are resilient to semantic uncertainties.
This research raises important safety concerns, adversaries could exploit these semantic-changing attacks to cause significant harm in real-world RI. applications, such as autonomous driving. The ability alter the perception of critical systems like self-driving consequences, As such, our findings underscore the necessity of further research into robust defenses capable of withstanding such advanced and subtle attack strategies.
could lead to catastrophic
Related Work
23
A.I
State Perturbation Attacks and Defenses
23
A.2
Attacks and Defenses Beyond State Perturbations
23
Diffusion Models and RL
24
A.4
Diffusion Models in Adversarial Examples
24
A.5
Relation with Constrained Diffusion Model
24
6
GANebased Semantic Adversaria Attacks.
25
B
Proof of Theorem 3.6
25
Implementation Details and Algorithms
27
C.I
Two Stage Attacks Pipelines
27
Visualization of SHIFT-O and SHIFT-I in the Freeway Environment and Doom
27
Score-Based Diffusion Model
28
EDM Model and ODE formulation
29
C.S EDM as conditional diffusion model
30
C.6 Training and testing stage algorithms for SHIFT
30
parametrsseeters Setting
1)
More Evaluation Results
32
Additional Attack Results in Atari Environments
32
D.2
Attack Performance on Continuous Action Space Environment and PPO Policy
32
3
Detection Results
33
D.4
Ablation on Attack Frequency
33
Ablation on DDPM and EDM diffusion architectures
33
D.6
Performance and Stealthiness of High-Sensitivity Direction Attacks in Korkmaz
31
34
D.7
Time Complexity Comparison
36
22

---

A Related Work
A.1 State Perturbation Attacks and Defenses
State perturbation attacks on RL policies were first introduced in [24]. where the MinBest attack was proposed to minimize the probability of selecting the best action by iteratively adding Ip-norm constrained noise calculated through - V., (TT(s=))), that when the agent's policy is fixed, finding the optimal adversarial policy can be framed as an MDP, and the attacker can find the optimal attack policy by applying RL techniques. This was further improved in [47]. where more efficient algorithm for finding optimal attacks, called PA-AD, was introduced. Instead of searching perturbed states in the original state space, PA-AD trained a director through RL to find the optimal attack direction, and the trained director directs the designed actor to generate perturbed states, which decreases the searching space of RL. More recently, illusory attack [66] is proposed by requiring perturbed trajectory to follow the same distribution as the normal trajectory, making it difficult to detect. However, this approach does not scale to high-dimensional image input. Korkmaz [31] recognized the limitation of Ip norm constrained attacks and proposed colecy-independent attack by following high sensitive directions, leading to attacks such as changing brightness and contrast, image blurring, image rotation and image transform. These types of attacks are imperceptible when the amount of manipulation applied is small and can compromise SA-MDP defense. However, they can barely alter the dopain-specific semanties of image input according to our Definition 3.3 For example, in the Pong game, the pong ball's relative distance from the two pads will remain the same after changing brightness and contrast, or transforming the image. As result, these attacks cannot bypass diffusion- based defenses as shown in Tables 8 and g
Building on this, Zhang et al. [60] showed
On the defense side, Zhang et al. (60) demonstrated that a universally optimal policy under state perturbations might not always exist. They proposed a set of regularization- based algorithms (SA- DQN, SA-PPO, SA-DDPG) to train robust RL agents. This was enhanced in [36], where worst-case Q-network and state importance weights were incorporated into the regularization. A more recent work called CAR-DQN 35 shows that using an to norm can further improve the policy's 's robustness, and they theoretically capture the optimal robust policy (ORP) under e constrained state perturbation attacks, although this method incurs high computational costs. Another line of work by (55) proposed an autoencoder- based detection and denoising framework to identify and correct perturbed states. Korkmaz and Brown-Cohen (32 proposed SO-INRD, which uses the local curvature of the cross- entropy loss hetween the action distribution π (a]s) given by policy π and a target action distribution to detect adversarial directions. He et al. 20 showed that when the initial state distribution is known, it is possible to find a policy that optimizes the expected return under state perturbations. Diffusion-based defenses have also been utilized generate more robust agent policies. DMBP (57) utilized a conditional diffusion model to recover actual states from perturbed states and Sun and Zheng 44 used the diffusion model state and perform a pessimistic training to generate robust policy. Nie et al. [40] found that the performance loss of a policy under an 4o norm-bounder state perturbations is bounded by the KL divergence between the action distribution under the true state and that under the perturbed state, and utilized a new network architecture called SortNet (59 train robust RL policy. More recently.
purification tool generate a belief set about the actual
game-theoretical defense method (GRAD) 37 was proposed to address temporally coupled attacks by modeling the temporally coupled robust RL. problem as a partially observable zero-sum game and deriving an approximate equilibrium of the game. Another important recent defense is PROTECTED [39], which iteratively searches for a set of non-dominated policies during training and adapts these policies during testing to address different attacks. However, both GRAD (37 and PROTECTED [39] focus on MuJoCo environments and are already computationally intensive (both take more than 20 hours) to train on the relatively simple enviromments. Without further adaptation, it will be computationally prohibitive to apply these two methods to environments with image input as we consider in this paper.
A.2 Attacks and Defenses Beyond State Perturbations
As demonstrated by 25 altering the reward signal can significantly disrupt the training process of Q-learning, causing the agent to adopt a policy that aligns with the attacker's objectives. Addition- ally, (3) introduced an adaptive reward poisoning technique that can induce a harmful policy in a number of steps that scales polynomially with the size of the state space |S| in the tabular setting. In
23

---

similar vein, Zhang et al. [63] developed an adaptive reward poisoning method capable of achieving malicious policy in polynomial steps based on the size of the state space |S|.
Moving beyond reward manipulation, Lee et al. [33 proposed two techniques for perturbing the action space. Among them, the Look-Ahead Action Space (LAS) method was found deliver better performance in reducing cumulative rewards in deep reinforcement learning by distributing attacks across both the action and temporal dimensions. Another line of research focuses on adversarial policies within multi-agent environments. For example, 177 showed that in a zero-sum game, player using an adversarial policy can easily beat an opponent using well-trained policy.
Attacks targeting an RL agent's policy network have also been explored. Inference attacks, as described by [TO]. aim to steal the policy network parameters. On the other hand. poisoning attacks, as discussed in [23,, focus on directly manipulating the model parameters. Specifically, Huai et al. (33 proposed aptimizatiio-basse method to identify an optimal strategy for poisoning the policy network.
A.3 Diffusion Models and RL
Diffusion models have recently been utilized to solve RL problems by exploiting their state-of-the-an sample generation ability. In particular, diffusion models have been utilized to generate high quality offline data in solving offline RL problems. Offline RL training is known as data-sensitive process, where the quality of the data has a huge influence on the training result. To deal with this problem, many studies 919 2 27 have shown that diffusion models can learn from a demo dataset and then generate high reward trajectories for learning or planning purposes. In addition, conditional diffusion models have been directly used model RL policies. A conditional diffusion model can generate actions through a denoising process with states and other useful information as conditions. Several studies [28 48 have shown state-of-the-art performance in various offline RL environments when using a diffusion model as a policy, which leads to a promising research direction.
Furthermore, Black et al. [8] shows that the denoising process can be viewed Markov Decision Process (MDP). Thus, Black et al. (8 trains a diffusion model with the help of RL by maximizing a user-specific reward function, which connects the generative models and optimization methods.
A.4 Diffusion Models in Adversarial Examples
Diffusion models have recently gained significant attention in generating adversarial examples due their superb performance. They can generate high-quality adversarial examples that deceive target classifiers while remaining imperceptible human observers.
Since the images generated by diffusion models inherently lack adversarial effects, a widely used approach is to diffusion models along with existing methods of generating adversarial examples. The idea is to combine the generated samples from the diffusion model with perturbed samples from other attack methods such PGD attacks during the attack process generate high quality and imperceptible adversarial examples [56 I1..
Another promising direction is to use (surrogate) classifier to guide the diffusion model generating samples that meet attacker specified goals by using gradient information from the classifier during the testing stage [38
13 [8] Also, Chen et al. [9] used the classifier guidance during the training stage of the diffusion model along with self and cross attention mechanisms.
O showed that poisoning the training set can produce a deceptive diffusion model that will generate adversarial samples without any guidance.
Further, Beerens et al.
However, these works only care about static stealthiness in a supervised learning setting, while SHIFT also takes dynamic stealthiness into consideration.
A.5 Relation with Constrained Diffusion Model
While diffusion models have been utilized to generate adversarial examples in the supervised learning setting (see Appendix A.4 for review), their application in adversarial state perturbations in RL has not been considered before. We remark that our problem can be viewed as sampling from a diffusion model with constraints on realism and history alignment. However, existing approaches for
24

---

constrained diffusion [2] cannot be applied directly to our setting, as they require constraints such as physical rules to be explicitly given and easily evaluable. our setting, it is difficult to identify the projection onto the valid states S", making these approaches less suitable.
A.6 GAN-based Semantic Adversarial Attacks.
Generative adversarial networks (GANs) have been widely used to generate adversarial examples that go beyond traditional p-norm constraints in image classification. Notable works include Adv- GAN [53] which learns to generate perturbations that mislead classifiers while maintaining perceptual similarity, and Natural Adversarial Examples (64), which leverage latent space manipulations within VAE-GAN frameworks to produce realistic semantic shifts. Other approaches explore latent adversar- ial attacks [65] direct image synthesis 4 to create imperceptible or highly transferable adversarial examples. While these methods succeed in the supervised setting, they do not transfer directly to reinforcement learning due to the absence of sequential structure, policy conditioning, or temporal consistency in GANs. Moreover, aligning GANs or other generative methods with reinforcement learning (RL)-specific objectives- ssuch as minimizing the expected cumulative reward under a given policy-woull would require substantial modifications to both architecture and loss functions. As such, adapting these methods to generate semantics aware perturbations RL. constitutes distinct line of research and is not directly comparable to our approach.
B Proof of Theorem 3.6
The following proof is adapted from the proof in Appendix H of Dhariwal and Nichol [4 We show that in the RL state perturbation attacks setting. we could combine classifier-free and gradient guidance. Let denote the victim's 's policy, Q" the state-action value function, and T:-1 {s--1, 0-1. ..., St-k, ] the sequence of the last k observations and actions up to time t. We first define conditional Markovian process & similar to q as follows.
((5)) T2-1) = q(55) 1c1)
i(Q" b8, TT-1) is known for every (5), T--1)
(( 13, Q", TT-I) (( 13)), Vi
(4)
á (s) bg Q T-1 II (5) 1 Q" m-)),,,,
where q(38 (T-1) P(5 st-1, 0x-1) is the conditional distribution of the original state 3f given the history TC-1. Next we show that the joint distribution q(अ!-T, Q (T-) given T:-1 is well defined.
160 1= á (5!
Q -1-1 165 Q T-1)
a (3)
s Q T 1(Q2" 5 T-- (6(
II s |1
i=1
Following essentially the same reasoning as in Appendix H Dhariwal and Nichol 14 with the trivial extension of including the condition T(-1, we have
[3 , = -(33)3)-) ((8 11),, T--1) - Tz-1)
-T-1)
25

---

Next, we show 1(Q"18% 8,_ T1-1) does not depend on s.
(Q9T (5 -T-1)) ( 3/ T--1)
((3), 8 .0-1)
((; 1- Q*, T(-1)
(5.)) Q" T
((3)
8 Tt Tt-1)
[ 100 7-1)
((3) T--1)
(F.) 1Q0 T(-1)
(5/
"(Q" 8 --1))
(5)
We now derive the reverse process that combines both classifier-free and gradient-guided methods.
955 5%. QT, TC-1)
1,, ,T--1)
((3), Q" -T-
q(5 8
)q(Q 13| B T(-1)
(8, Q" T(-1)
9(5)
15 Tt-1 1 19(5) T (Q( St
86 7-1)
(3% Q" T(-1)
q(%
115. Tt- \1(" 81 x-1
T--1)
4(Q"T (8 71-1)
8(Q"1 8 T(-1)
( 13 T(-1)
1(Q= 3 T-1)
where (a) follows from Equation Note that 118--118 T )) can be learned through a history conditioned diffusion model Pa and we will use Q" (st, r (s;)) to approximate "(Q" [5 T(-1), Vi. We notice that the Q" (sy, (3;)) term guides our reverse process to generate samples that lead to lower state value. Thus, the attacker should use the victim's 's policy æ here to gain better guidance through Q π (5))). Plugging them back into the above equation, we have
((5) 1 (-, R po(st 1s TI-1) (5, ((;
(= n (3:))
~ po 13 7e 1oR (
(s), (8) lox@o (x))(())
(6)
Using the Taylor expansion, we get
logQ"(sst, (((-)((())))) logQ" (sc, (B)))) = (8/-1 V v (st, x(:)).
We also have
o2 exp (- (st mi)
po (8) 13) Te-1)
20
where (4 comes from < -== + (1 ()e0(st.)) 1), as given by 2 and a? is determined by the noise scheduler 3:.
26

---

Substituting them back into 66. we have
(8 1 1/: Ft-
( ")) ")>-loQQ (x.=(()))))))))
3/-1 Mi)
+ (8) 8 selog@ (T))))
x exp
20,
(8t-) Mi)
2 (st-)
8/) V log (st, TT;))
exp
f
(8 (si) 2 20 (5t p4) y logQ m((f) + x3+))) 2
exp
20
207 (mi 81) V log@ (ve. r(())) + V (B)))
X exp
207
(5) m4) of V 5t (s (sx
eep
20²
2 (y)
81) V VgllgQ (t, (3:)) + V (st, (())))
207
56
Ui (s,
x exp
20²
Equation ( implies that the reverse process when sampling from history-conditionnd DDPM model guided by the victim's 's state value function can be represented as
p 1s ( = N
$ 14 + ofV log Q" (s,,
Noticed that we want to guide the generated state op lead to lower state-action value, thus we change sign of the gradient guidance to minus.
QT, T--1) N
1 : (
Vf log Q" (())))
C Implementation Details and Algorithms
C.1
Two Stage Attacks Pipelines
Figure 4 gives an overview of our two-stage Liffusion-based attack including all the major components involved.
C.2 Visualization of SHIFT-O and SHIFT-I in the Freeway Environment and Doom
Figure 6lives an illustration of SHIFT-O and SHIFT-I in the Atari Freeway environment, Figure provides additional comparison between SHIFT-O/I and PGD and rotation attack from Korkmaz 31 in Atari Freeway and Doom.
27

---

Conditional Diffusion model
(St, (ct))
Sample
Environment
Clean Trajectories
Training
Unrealism Detector
(a) Training Stage
Victan's polcc
Unreation Detastor AB
Policy Guidance
Enbanezmen
>:>Tww
8 One car
V. QO,
Conditional didlusion model
7-11
Victim
81
Reverse Sampling
St
SHIFT-O
for Gt-
SHIFT-I
Attacker
Environment
(b) Testing Stage
Figure 4: Pipelines of SHIFT's two stages. a) shows the training stage where the attacker uses clean data to train history-conditioned diffusion model and an autoencoder-based anomaly detector. b) shows the testing stage where the attacker perturbs the true state through the reverse sampling process of the pre-trained conditional diffusion model guided by the gradient of the victim's policy and that of the autoencoder's reconstruction loss.
(a) 81
(b) under PGD
(c) 8, by Rotation
(d) SHIFT-O
(c) SHIFT-I
(f) (g) under PGD
by Rotation
SHIFT-O
(i) SHIFT-I
Figure 5: Extra examples of true and perturbed states of Atari Freeway and Doom. a) is the true state. b) and c) the perturbed states under the PGD attack with Loo budget of and through rotation [3 by 3 degrees counterclockwise. respectively. d) and are perturbed states generated by our SHIFT-O and SHIFT-I attacks. respectively, Neither PGD nor Rotation attacks can alter the decision-relatee semantics.
C.3 Score-Based Diffusion Model
As shown in Song et al. 45 a diffusion process (=(10) can be represented as the solution to a standard stochastic differential equation (SDE):
dx f(x, i)di + g(i)dw,
28

---

Ishould wait!
Ishowid go!
Wait and Go
Imaginary Trajectory
SHIFT-
Agent
Agent
should go!
Perturbed State
Delayed! Suboptimal!
Perturbed State
Go
NO AtlacK
Go
NO Attack
CB
Accident!
Agent
Walt
State
True State
Optimal
Optimat
Figure 6: Visualization of SHIFT-O and SHIFT-I in Freeway. The goal of the Freeway game is to control the agent (blue circle) to cross the road quickly and safely. SHIFT-O injects an extra car (red cirele) into the agent's 's observations slow down the agent. SHIFT-I injects segment of imaginary trajectory into the agent's observations that misleads the agent to move forward, despite a car being present in front of the agent in the real state, resulting in a collision.
where represents the drift coefficient, which models the deterministic part of the SDE and de- termines the rate at which the process changes over time on average. g(i) is called the diffusion coefficient, which represents the random part of the SDE and determines the magnitude of the noise. Finally, w represents a Brownian motion so that g(i)dw is the noising process.
We can let the diffusion process have Zn ~ Po and and pr a Gaussian noise distribution independent of po. Then we could run the reverse-time SDE to recover a sample from Po by the following process:
pr, where Pe Pdata is the data distribution
dac (ax, i) 2 log), di + gi)ddw
where Væ log (ax) ) the score function and is Brownian motion that flows back from time T to 0. The training objective for the score matching fucntion se for the SDE is then given by:
arg minE, (o))) [|] (zx(),,) - < log (x) |x [00)
where A(i) is positive weighting function and is uniformly sampled from [0, 2.. The objective can be further simplified since Poi is known Gaussian distribution.
C.4 EDM Model and ODE formulation
Inspired by Song al. 45 EDM (29 proposes ODE formulation of the diffusion model by having a scheduler (tt) to schedule the noise added at each time step t. The score function correspondingly changes to V. log p(æ; or), which does not depend the normalization constant of the underlying density function p(æ,00 or) and much casier to evaluate. To be specific, if D(x; or) is denoiser function that minimizes:
Ex-pus En-N(o.- D(x + n; - =x11
(8)
then
Ve logop(<z;0) (D(x;))) x))0²
We usually train a neural network learn the denoising function D(x,, a) by using the simplified training objective in Equation (8). Utilizing this finding, EDM only requires small number of reverse sampling steps to generate a high quality sample. However, EDM needs more preconditioning parameters such scaling to approximate dynamic range, further discussed below.
29

---

C.5 EDM as a conditional diffusion model
In this paper, we follow the approach in Alonso et al. 3 to train an EDM-based diffusion model conditioned history 7t-1 to predict the next state St- Taking the network preconditioning parameters used in EDM into account, we have the denoising function changed to:
Dg (s/ Guose 7(-1) d'sip si + cou Fø (cin S6. Cuoise: T(-1).
where Fø is the neural network to be trained, the preconditioners Cin and Cout scale the network's 's input and output magniuude keep them at unit variance for any noise level (i). Gois is empirical
transformation of the noise level, and clsi determined by the noise level a(i) and the standard deviation of the data distribution data- The detailed expressions given below:
(o)2 + data in
(9)
o d
(10)
out
Vo(1)² - data
Guose
log(o((i)
(11)
Caip
data
+ o²
(12)
where data = 0.5. The noise parameter (i) is sampled to maximize the effectiveness during training by setting lo)( N (((meam Pmmaa: ) where Pmman -0.4, Pad 1.2. Refer Karras et for detailed explanation.
The training objective of F9 changes correspondingly
1 Skip st) [|2]
(()) |[| Fø (cnn s/, Crssse T-
(st
(13)
Cout
In our implementation, change the residual block layers from [2,2,2,2] [2,2] and the denosing steps to 5, and set the drop condition rate to 0.1. We keep other hyperparameters the same as Alonso
et al. 3
C.6
Training and testing stage algorithms for SHIFT
Algorithm 1: History-Aligned Conditional Diffusion Model Training
Input:
Training data O 7t-1) condition dropping rate Ckdrop: P Pmeans P2 0,, learning rate 7), history length k.
Output: Trained EDM model parameters 0
Initialize: EDM model parameters 0
for number of training iterations do
Sample data point (St, 72-1) O;
// Tt-1 = {St-kr , 8t-1, ) is the true history Sample log(o) ~ (N P
Calculate preconditioners Gin, Cour based on according 9 and 10
Generate noisy data sf Asss,, II);
if' random > Odrop then
Compute generated state 5, Do(s Cnoise: Tt-1) else
Compute generated state á, = Dø (sf.. Cuoise)
Compute loss L(0) based on Equation (13): Update 9 using gradient descent: +- 8-10 1 voL(O);
end
30

---

Algorithm 2: Testing Stage Sampling with History, Policy and Realism Guidance
Input: History conditioned diffusion model Da. victim's policy n, number of denoising steps T, autoencoder- based unrealism detector A E, agent state-action value function Q" given true history T--1 ks at-ks.... St-1, a-11 for SHIFT-O or observed history T {or-k, -s ..... 9r-1, } for SHIFT-I,, classifier-free guidance strength I'1, classifier guidance strength I'2, attack rate ç, true state st, history length k. Output: Generated sample 5t
Initialize: 3f ~ (0,
Calculate state importance w(s) maX EA Q" 01) min A ( a2):
if (st) in top& & percentile of previous states and total attacked times < t x & then total attack times += 1;
// Inject SHIFT Attack
for T to do
// Calculate proposed output 8, based on
$; 8;
for j = to do
8; = De(St, Shoise T(-1)
end
Calculate policy guidance gradient g <- V., Q5 a(;)); Inject policy guidance $ 3; 2g;;
Generate next sample s r (t) (5) coise: Tt-1) + (1- Caoie): if + then
Conduct gradient descent based on the reconstruction error from the unrealism detector
zi-1
V V..- AE
end end
end
C.7 Hyper-parameters Setting
EDM Diffusion Model Training Parameters. As mentioned before, we only change the residual block layers from [2,2,2,2] to [2,2] and the denosing steps to 5, and set the drop condition rate to 0.1. We keep other hyperparameters the same as Alonso et al. for training the EDM diffusion model.
Testing Stage Parameters and Testhench Specification. We schedule the classifier free guidance scale as (1) max( ني 0.3),, where T is the number of reverse steps and is the current reverse step. We set the policy guidance strength I'2 differently in each environment under each defense. In the Pong environment, other defenses. In the Freeway environment, we set 2 = 6 for DQN, DP-DQN and DMBP and I'2 4.5 for all other defenses. In the BankHeist environment, we set I'2 4 for all defenses. In the RoadRunner environment, we set I22 6 for all defenses. In the Doom environment, we set l22 4.5 for DQN, DP-DQN, and DMBP and I22 2.5 for all other defenses. For the AirSim autonomous driving simulator, we utilize CITY, a complex environment that simulates real-world city traffic situations. We r2- 0.5 for all defenses AirSim.
set T2 3.5 for DQN, DP-DQN, and DMBP and T
for all
We conduct all our experiments on a workstation equipped with an Intel 19- 12900KF CPU, an RTX 3090 GPU, and 64GB system RAM.
Pre-processing Atari, Doom and AirSim Environments. We have used the same environment wrappers as in Zhang et al. [60), which convert an RGB image gray-scale image and resize the image reduce its resolution from 210 x 160 to 84 84 for Atari environments and 320 240 to 84 x 84 for Doom. We also follow Zhang et al. [60] to center crop images using the same shifting parameters as Zhang et al. (60), where we set the cropping shift to 10 for Pong, 20 for Roadrunner, and ( for Freeway, Bankhesit, and Doom. We also convert the front view camera snapshot of the autonomous agent in AirSim 84 x 84 grayscale image. We do not stack frames in pre- pro-proc....ng.
31

---

D More Evaluation Results
Table 3: Attack results Atari BankHeist and RoadRunner Environments under SHIFT-I and
SHIFT-O attacks with 1.0 and 0.25 attack frequency.
Bankfleis
Rouner
Model
SHIP FO-18
SHIFT---..3
SEIP SHIFFTTI-1 T1111
SED 1-1-0.25
SETT T00-0
SEIT 1-0-2225
SMIFT-F1
SHIT 0--0255
DQN-No Allack
68001000
680.010.0
6800010.0
680,010.0
13500010
13500.010
13500010
13500,010
DQN
5001145.
0.000.0
0.000000
00100
16002151.7
00:000
SA-DON
30-3.
480045215
28200-16903
4400-559.
WomR--DQN
6.-555
66.0+27.00
0.0+0.0
0.0+0.00
120.4483.7
10900=807.5
00-0.0
100.0=70.7
CAR-DON
12.0=14.1
24.02:5.22
0.0=0.0
6.0:8.9
1100.::-44.22
920,22-4,7
2000=313.0
540.0=33009
DF-IQN
15.0-220
32.04+24.2
0,0:00.0
0.0:0000
220.0+1304
20000=264001
00:00
0:0:000
DMBP
20.0-18.
54,0++53.7
0.0+0.0
0.0+0.00
240,0+134.2
43000-145335
00-00
80.0-83.7
Table 4: Attack results of our methods with (0.15,0.25.0.5..0)) attack frequencies against DMBP defenses.
SHIFT-O
SHIFFI
Frequency
Attack
0.15
0.25
1,00
Pong
21.0000
9.88=4.5
-126-18
-440449
20.4+.3
20.208
20.+0.9
-2.--099
Freeway
34.220.1
31.0::22
28.8ET3
22.0:0077
138822
14.6-2.
14.62TI
19.252.
Doom
784444
21.445.1
63.4-988
68.247.11
18484431.7
-101,6+-000.
-256.2+1804
-239..+7444
-322042277
AirSim
403105
28.1372
12.13.0
1922166
20.-1220
11.7335
94449
6.6.2..5
6.200.7
Table Attack performance on the Highway environment (with a continuous action space) and Pong environments (with discrete action space), under PPO policies with/without DMBP defense. For the continuous action environment Highway, we compare with the MAD attack from Zhang et al. [60]. For the discrete action environment Pong, we compare with the PGD attack.
Highway
Pong
Attack Type Attack
PPO
DMBP
PPO
DMBP
23.3 9.6
21.0
21.
MAD/PGD Attack
2.7
16.
-.0
20.8 0.4
SHIFT-O-110
2.0
0.6
-210
-
SHIFT-0-0.25
+
15.4+5.2
-.9
1.6
SHIFT-I-LO
0.84 + 0.3
0.2
-20.0 -.00
SHIFT-I-0.25
2.55 + 0.8
3.80
+.0
-19.6 0.8
Table 6: Automated detection results with MAD threshold set 5, and CUSUM configured with drift of 1.5 and a threshold of 3.
Env
Freeway
Attack Method
MAD Dector
CUSUM Dector
PGD-1/255
Undetected
Undetected
PGD-15/255
Detected
Detected
MinBest- 1/255
Detected
Detected
MinBest-15/255
Detected
Detected
PA-AD-1/255
Undetected
Undetected
PA-AD-15/255
Detected
Detected
PA-AD-TC-15/255
Detected
Detected
Rotation Degree
Undetected
Undetected
Transform(1.W)
Undetected
Undetected
SHIFT-O-100
Undetected
Undetected
SHIFT-I-LL
Undetected
Undetected
D.1 Additional Attack Results in Atari Environments
We report additional attack performance results on the remaining two Atari environments: and RoadRunner in Table 3
BankHeist
D.2 Attack Performance Continuous Action Space Environment and PPO Policy
Table shows our SHIFT attack can successfully work on PPO policies under both continuous action space environment Highway [34] environment and discrete action space environment Atari Pong.
32

---

Table 7: DIRE [49] Detection Results
Env
SHIFT-O
SHIFT-I
Clean Diffusion
Pong
3.19%
3.56%
97.63%
F Freeway
4.15%
4.72%
95.46%
We compared our attack with the Maximal Action Difference (MAD) attack used in 660. which simple yet effective attack against the PPO algorithm, both with and without the DMBP defense. Since MAD attack does not apply on discrete action space, we report PGD attack with a budget 1/255 baseline. The results demonstrate that our attack outperforms MAD/PGD in both cases, further supporting its generalizability to different RL algorithms.
D.3 Detection Results
Furthermore, inspired by the adversarial attack detection method in Russo and Proutiere 42 we have evaluated our attack under two commonly used anomaly detection methods based on the Wasserstein-1 distance between adjacent perturbed states: (1) the MAD (Median Absolute Deviation) anomaly detector [.. which flags anomaly if three consecutive steps of perturbed trajectories violate clean trajectories' median threshold XMah, where clean trajectories' median and MAD are both evaluated under Wasserstein-1 distance between adjacent true states. We set threshold 5 in the table below; and (2) CUSUM [41], sequential change detection method that identifies persistent deviations (again terms of Wasserstein-1 distance) from clean trajectories by accumulating small shifts over time. We set the drift to 1.5 and threshold to 3 in CUSUM. We have compared PGD, Minbest, PA-AD attacks (with budgets 1/255 and 15/255). Rotation and Transformation attacks from Korkmaz 3 and our method.
Furthermore, multiple methods have been proposed to detect whether an image original or generated by a diffusion model. The victim can apply these detection methods to determine whether the observed state fefffusion-generaate diffusion-generaa images by examining their reconstruction behavior through a pretrained diffusion model. Given an input image To, DIRE first adds Gaussian noise to obtain æ, at diffusion step I, and then denoises it using the model's s reverse process to reconstruct in. The reconstruction error Eg = Do šol serves as a key indicator of authenticity.
One of the most widely used approaches is DIRE (49) which detects
In DIRE, real images typically produce larger reconstruction errors, whereas diffusion -eeerrate images align more closely with the model's output distribution and thus yield smaller errors. Lever- aging this separation, DIRE classifies inputs without retraining and generalizes across architectures such as ADM, GLIDE, and Stable Diffusion.
We conduct an initial investigation of DIRE against our SHIFT attack (Table 7 We train a binary classifier with clean images' DIRE reconstruction features and normal diffusion generated images' DIRE reconstruction features. The classifier successfully flags normal diffusion outputs in the no- attack setting. However, it fails detect both SHIFT variants, labeling them normal states. We conjecture that SHIFT's policy guidance method pushes generated states away from the diffusion model's 's manifold manner that increases reconstruction error, thereby imitating the high-error signature of real images and evading DIRE's decision rule.
D.4 Ablation on Attack Frequency
Table 4 illustrates the performance of our attack across different attack frequencies &. The results show that even at low attack frequencies, our method significantly reduces the agent's 's cumulative reward. Moreover, the reduction becomes more pronounced the attack frequency increases.
D.5 Ablation on DDPM and EDM diffusion architectures
We compare DDPM and EDM in terms of attack efficiency and computational cost in Table The results show that EDM and DDPM exhibit similar attack performance. However, DDPM is significantly slower than EDM in terms of sampling time (the average time needed to generate single perturbed state during testing), making DDPM incapable of generating real-time attacks during
33

---

Table 8: Performance of high-sensitivity direction attacks in [31
Pong Reward
Freeway
Attack
Defense
Reward
DQN
-21 0.00
23 = 0.00
B&C
SA-DQN
11 + 0.00
25 + 0.00
DMBP
20 + 1.41
27.2 + 0.68
DQN
21 0.00
18 = 0.00
Blurred Observations
SA-DON
20 0.00
27 + 0.00
DMBP
+ 0.58
33.2 0.37
DQN
20 = 0.00
26.6 + 0.45
Rotation Degree
SA-DON
18 0.00
21 0.00
DMBP
14.6 2.68
27.6 = 0.45
DQN
21 0.00
26 + 0.00
Transform (1,0)
SA-DQN
0.00
24 = 0.00
DMBP
17.8 = 2.85
27.2 = 0.37
Table 9: Large-scale rotation and shift attacks against fine-tuned diffusion- brasssiion-baedd defense.
Pong
Defense
Reward
Rotation Degree 3
DMBP
20 = 0.71
Transform (2,1)
DMBP
18.8 1.79
Table 10: Average Reconstruction error, Wasserstein distance, SSIM and LPIPS of the high-sensitivity direction attacks in Korkmaz [3]] across randomly sampled episode. The Wasserstein distance is the Wasserstein-1 distance calculated between the current perturbed state and the previous step's 's true
state.
Freeway B&C
Reconstruction Error,
Wx 3)1
SSIM
LPIPS
15.30 1 0.02
0.80 1
0.9429 0.0006
0.0455 = 0.0031
Blurred Observations
5.81 1 0.04
0.82 1 0.2
0.5974 0.0034
0.3535 1: 0.0202
Rotation Degree 1
6.41 + 0.2
0.80 + 0.2
0.8237 + 0.0022
0.0351 + 0.0041
Transform (1,0)
9.32 + 0.1
0.83 + 0.2
0.60445 0.0074
0.0741 + 0.0068
SHIFT-O-L-0
1.02
0.89
0.9990 0.0014
0.0008 + 0.0014
SHIIFT-I-L00
1.05 1 0.5
0.84 1
0.9904 0.0043
0.0175 1 0.0124
Table 11: Ablation studies on EDM and DDPM diffusion architectures.
Pong
DDPM
EDM
Reward
Deviation Rate (%)
Reward.
Deviation Rate( )f
DQN
20.6 + 0.5
83.6 = 1.0
-21.0 0.0
90.0
DMBP
10.4 + 4.8
46.1 + 0.3
114.0 6.2
47.3 + 0.9
Sampling Time
5 sec
0.2 sec
testing. This validates the selection of EDM as the diffusion model architecture for constructing attacks.
D.6 Performance and Stealthiness of High-Sensitivity Direction Attacks in Korkmaz 31
Korkmaz 31 proposes various high-sensitivity direction-based attacks that can generate perturbed states that are visually imperceptible and semantically different from the clean states, including changing brightness and contrast (B&C), image blurring. image rotation and image transform. These attack methods reveal the brittleness of robust RL methods such as SA-DQN, but they mainly target changes in visually significant but not domain-specific semantics. For example, the relative distance between the pong ball and the pad will remain the same after brightness and contrast changes or image transform in the Pong environment. Consequently, the perturbed images generated by these
34

---

Clean State
(b) Image Rotation Degree 3
Image Transform (2,1)
(d) Clean State
(e) Immaee Rotation Degree 3
Image Transform (2.1)
Clean State
(h) Image Rotation Degree 3
(i) Image Transform (2,1)
(i) Clean State
(k) Image Rotation Degree 3
()) Image Transform (2,1)
Figure 7: Perceptual impact of large-scale rotation and transform on Atari Pong, Atari Freeway, Doom, and Airsim.
methods can potentially be purified by a diffusion model. To confirm this, we have conducted new experiments, showing that (1) the DMBP defense with a diffusion model trained from clean data only is able to defend against B&C, blurring, and small scale rotation and transform attacks (see Table and (2) when the diffusion model fine-tuned by randomly applying image rotations or transform during training, the DMBP defense mitigate large scale image rotations and transform
35

---

considered in Korkmaz B1 (see Table D. In contrast, our diffusion guided attack can change the decision-relevant semantics the images, such as moving the Pong ball different position without changing other elements in the Pong environment shown in Figure 2 This is the key reason why our attack can bypass strong diffusion-banee -aased defense methods.
Furthermore, Korkmaz [31 claims their attacks are imperceptible by comparing the perturbed state 8, and the state s;. However, found that this only holds for small perturbations. For example, the Rotation attack with degree 3 and Transform attack (1.2) in the Pong environment considered in their paper can be easily detected by humans (see Figure 7 Further, their metric for stealthiness is static and does not consider the sequential decision-making nature of RL. In contrast, our attack method aims to stay close the of true states S maintain static stealthiness and realistic (Definitions 3.2) and align with the history to achieve dynamic stealthiness (Definitions 3.4). These are novel definitions for characterizing stealthiness in the RI. context. The static stealthiness and realism are demonstrated through the low reconstruction loss of our method, shown in Table 2 TO In contrast, attacks from Korkmaz [31 generate nut-of-distribution perturbed states by applying image transformations, which induce high reconstruction enror, indicating that they are less realistic than our methods. We further compare the Wasserstein distance between a perturbed state and the previous step's perturbed state, SSIM, and LPIPS metrics to measure the historical alignment and trajectory faithfulness Table 10 The results show that the perturbed states generated by Korkmaz Bil achieve the same level of historical alignment but are less trajectory faithffu than our SHIFT attack.
D.7 Time Complexity Comparison
In terms of time complexity, high-sensitivity direction attacks can generate perturbations instanta- neously as they are policy independent, and PGD, PA-AD-TC, MinBest, and PA-AD all take around 0.02 seconds to generate perturbation with 10 iterations. Due to the computational overhead of the reverse process, diffusion-based methods typically require longer generation times. However, by adopting the EDM diffusion paradigm, reduce the reverse process steps 5, resulting in a generation time of approximately 0.2 seconds per perturbed state. Although slower than PGD, MinBest, and PA-AD, this still allows our attack to remain feasible for real-time applications.
36


Reviews:
[{"review_id": "LMlOlHVatZ", "reviewer": "Reviewer_vqkJ", "summary": "This paper aims to generate adversarial attacks for RL systems via state perturbations, while restricting the perturbation to remain in the distribution of realistic trajectories given the agent's history. It does this through training a history-conditioned diffusion model to generate plausible next states and using classifier guidance to search for next states which would cause the policy to perform worse. In addition, it restricts this search with an auto-encoder-based method of anomaly detection to maintain a greater degree of realism.", "questions": "How could we expect a system to be robust to these sorts of attacks?\n\nIn what scenario would an attacker have the ability to execute this attack while not having the means and inclination to do a much more direct attack? (like taking over the steering directly)", "limitations": "I believe the technical limitations are adequately addressed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The approach of training a diffusion model and using classifier guidance to produce adversarial attacks is interesting and broadly effective\n* The regularisation with auto-encoder-based anomaly detection is an interesting and powerful trick to get this method to work.\n* It appears to work as intended. Figure 2 is a clear example of the method working.\n* Adversarially searching through synthetic but semantically valid generations is a broadly useful problem and it was previously unclear to me how this could be achieved. I would expect this to be useful in adversarial evaluation broadly and in unsupervised environment design.\n\nWeaknesses:\n* The setting is unrealistic as an adversarial attack. If the attacker had this much control over the observations they would likely have access to control the hardware directly and would not need to resort to this level of complexity.\n* In addition, it is not clear how making it look semantically reasonable makes the attack undetectable. It is quite noticeable that the pedestrian has been removed from the image in Figure 2. If both videos are saved for audit, or if a human were watching, it would be quite visible. \n* Further, if we were aiming to use this for adversarial training it is unclear what we would want the agent to do. Ultimately if the image looks indistinguishable from a clear road ahead I can see no way that any system would be able to avoid hitting pedestrians. This is why perturbations for ML models in adversarial attacks have been norm-ball restricted, because we aim to preserve the semantics of the image as to find situations where the agents decision changes when it should not. In the situations generated in this paper, the behavior of the system ought to change. \n* The theoretical formalism does not quite match what is being done.\n\t* The definition of \"valid states\" would preclude the sorts of semantically sensible generations that we see from diffusion models at scale which are combinatorial generalisations of realistic images like \"an astronaut riding a horse on the moon\". Given that this technique is based on diffusion models it would likely allow for these sorts of combinatorial generalisations but they are disallowed by definition 3.1. \n\t\t* This critique is inherited by the definition of \"realistic states\" \n\t* More concerning is that the definition of semantics-changing states does not quite match the name. There are semantically identical images that are quite far in pixel space, only changing the texture, color, hue, or other irrelevant aspects of the image. I would have expected \"semantics-changing\" to involved aspects of the image which would change the policy.\n* The approach should cite works from the generative environment literature, as it is essentially training a generative world/environment model. In essense, this can be seen as an application of a generative world model, or a way of leveraging a pre-trained generative world model for adversarial search.", "was_revised": true, "final_justification": "The approach used in this paper is interesting and useful, as it shows how to generate semantically meaningful adversarial attacks on a policy that will be useful in many domains that need high-dimensional adversarial search. However, I believe that the particular attack model they are working with is not likely, since it proposes a very specific sort of access from an attacker (full control over sensors and no other control). Moreover, even if the attacker had this specific sort of access it seems impossible to defend against, so it is hard to understand the utility (for the defence) of generating better attacks.", "rebuttal": "Dear Reviewer vqkJ,\n\nThank you for your insightful feedback. We will address your concerns in the following.\n\nQ1: The setting is unrealistic as an adversarial attack. If the attacker had this much control over the observations, they would likely have access to control the hardware directly and would not need to resort to this level of complexity. In what scenario would an attacker have the ability to execute this attack while not having the means and inclination to do a much more direct attack? (like taking over the steering directly)\n\nA1: The state perturbation attack setting assumes that the attacker can manipulate only the observations received by the victim agent, thereby misleading it into choosing suboptimal actions. This assumption is standard in prior literature, including SA-MDP[1] and PA-AD[2]. We argue that gaining access to an agent’s sensory input (e.g., camera or radar) is considerably more feasible than gaining full control over its actuation system. For example, in autonomous driving, it may be easier for an attacker to hack into perception components, such as cameras or LiDAR, to subtly manipulate visual inputs, rather than breach the entire control system to directly steer the vehicle.\n\nMoreover, our attack is relevant to real-world robotic and autonomous systems that rely on visual observations. SHIFT can be deployed in such scenarios to induce subtle, semantically meaningful perturbations without needing full system control.\n\n[1] Zhang et al., Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations. NeurIPS 2020.\n\n[2] Sun et al., Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL. ICLR 2022.\n\nQ2: In addition, it is not clear how making it look semantically reasonable makes the attack undetectable. It is quite noticeable that the pedestrian has been removed from the image in Figure 2. If both videos are saved for audit, or if a human were watching, it would be quite visible.\n\nA2: As mentioned in Q1/A1, during testing the victim agent only observes the perturbed states and has no access to the true states. Therefore, it cannot detect perturbed states or compare perturbed and true states in real time. Moreover, even if both true and perturbed trajectories are saved for future auditing, the attack would have already influenced the victim’s decision-making and degraded its performance before being detected. Our focus is on real-time evasion during deployment, where SHIFT remains hard to detect under typical assumptions in the state perturbation threat model.\n\nQ3: Further, if we were aiming to use this for adversarial training it is unclear what we would want the agent to do. Ultimately if the image looks indistinguishable from a clear road ahead I can see no way that any system would be able to avoid hitting pedestrians. This is why perturbations for ML models in adversarial attacks have been norm-ball restricted, because we aim to preserve the semantics of the image as to find situations where the agents decision changes when it should not. In the situations generated in this paper, the behavior of the system ought to change.\n\nA3: This is an insightful point, and we agree that applying SHIFT for adversarial training is a promising future direction. Compared to traditional $l_p$ norm-based perturbations, SHIFT induces semantic changes, allowing us to go beyond simple pixel-level perturbations.\nInstead of assuming small norm perturbations that preserve semantics, agents under SHIFT can be trained to form a semantic belief about the underlying true state and make robust decisions based on this belief when encountering perturbed observations.\nWe also acknowledge that in some scenarios, the semantic change may be too strong for the agent to act optimally,e.g., when a pedestrian is fully removed. However, this highlights the value of budget-aware sensing: if the agent has a limited probing budget (e.g., can occasionally access true observations), SHIFT can help identify high-risk situations in which the agent should query the environment more carefully.\nThus, SHIFT can facilitate learning robust policies by revealing when perception failure is most consequential.\n\nQ4a: The definition of \"valid states\" would preclude the sorts of semantically sensible generations that we see from diffusion models at scale which are combinatorial generalisations of realistic images like \"an astronaut riding a horse on the moon\". Given that this technique is based on diffusion models it would likely allow for these sorts of combinatorial generalisations but they are disallowed by definition 3.1.\n\nA4a: The conditional diffusion model used by SHIFT is trained specifically on data collected from the target environment. For example, in the autonomous driving scenario, we train the model from scratch using trajectories recorded in clean, unperturbed driving environments.\nAs a result, the trained diffusion model captures the semantics and physical constraints of the environment and does not generate out-of-distribution samples such as \"an astronaut riding a horse on the moon.\"\n\nQ4b: More concerning is that the definition of semantics-changing states does not quite match the name. There are semantically identical images that are quite far in pixel space, only changing the texture, color, hue, or other irrelevant aspects of the image. I would have expected \"semantics-changing\" to involved aspects of the image which would change the policy.\n\nA4b: This is a great point. We agree that certain image-level changes, such as texture, brightness, or contrast, can appear large in pixel space but remain semantically identical. However, state-of-the-art diffusion-based defenses like DMBP and DP-DQN are specifically designed to denoise and project the input back onto the valid states. As shown in Appendix D.8, attacks based on high-sensitivity directions (e.g., blur, brightness changes) fail to affect these defenses.\n\nIn contrast, SHIFT generates semantically altered states that can bypass diffusion-based denoisers and actually lead the agent to take different actions. Still, as shown in Appendix D.8, attacks based on high-sensitivity directions can change the policy behavior under weaker defenses or no defense at all.\n\nThis leads to an important nuance: if we define “semantics-changing” solely by whether the agent’s action changes, the definition becomes policy-dependent. For example, brightness changes might alter the actions of a vanilla DQN agent but have no effect on a DP-DQN agent. To avoid such ambiguity, our definition of semantics-changing states (Def. 3.3) does not rely solely on policy behavior and does not exclude non-essential semantic changes—reflecting the broader goal of evaluating stealthy, history-aligned, and behavior-impacting perturbations.\n\nQ5: The approach should cite works from the generative environment literature, as it is essentially training a generative world/environment model. In essense, this can be seen as an application of a generative world model, or a way of leveraging a pre-trained generative world model for adversarial search.\n\nA5: Thanks for pointing out this line of related work and we will add a section in the related work in the revised version of our paper. We will include papers such as [3][4][5].\n\n[3] Hafner et al., Dream to Control: Learning Behaviors by Latent Imagination. ICLR 2020.\n\n[4] Pérez et al., EnvGen: Generating Diverse and Challenging Environment Variants. ICLR 2023.\n\n[5] Zhou et al., DINO World Model (DINO‑WM): World Models on Pre‑trained Visual Features Enable Zero‑Shot Planning. ICML 2025.\n\nWe hope our detailed rebuttal could address most of your concerns. If you have further questions, please let us know and we will try our best to clarify them.\n\nSincerely,\n\nAuthors of Submission 9986"}, {"review_id": "sd2MmlHTIY", "reviewer": "Reviewer_u3xt", "summary": "This paper presents novel characterizations of semantics-aware and stealthy adversarial attacks in sequential decision-making settings, introducing three key principles: Semantic Change, Historical Alignment, and Trajectory Faithfulness. Building on these insights, the authors propose a novel, policy-agnostic attack framework based on diffusion models, which perturbs visual observations beyond traditional ℓₚ-norm-constrained methods. The proposed approach generates semantically altered adversarial perturbations that are less detectable. Empirical results show that the method outperforms existing attack baselines.", "questions": "1. In lines 242–243, is an offline dataset collected from a clean environment required, or is it necessary to generate trajectory data by interacting with the clean environment using $\\pi_{ref}$ to assist training? What are the requirements for $\\pi_{ref}$? Does it need to ensure a certain level of coverage over the state or trajectory distribution?\n\n2. Why does SHIFT-I not incorporate information about $a_{t-1}$? What would the performance of SHIFT-I be if $a_{t-1}$ were included? Similarly, how would the performance of SHIFT-O be affected if it also omitted $a_{t-1}$?\n\n3. Regarding the experiments in Table 2, how would the performance of SHIFT be affected if the attack frequency were reduced, for example, in variants like SHIFT-I-0.25 and SHIFT-O-0.25?\n\n4. The generation time of SHIFT is approximately 0.2 seconds. It remains unclear whether, in practice, the agent can detect the presence of an attack by observing differences in generation times of subsequent states between normal (no-attack) and SHIFT attack scenarios. Furthermore, it is of interest to investigate how increasing the generation iterations of methods such as PGD, PA-AD-TC, MinBest, and PA-AD, or decreasing the number of reverse steps in SHIFT—thereby aligning their adversarial state generation times to the same scale—would affect their comparative performance.\n\n5. There is a typo in line 257; it should be Figure 6c.", "limitations": "Please refer to the Weaknesses and Questions.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is motivated by a reasonable intuition, proposes a novel method, and provides empirical results that suggest its effectiveness.\n\nStrengths：\n\n1.\tThe paper is well-organized, and the concepts are clearly articulated.\n\n2.\tThe proposed three properties are intuitive and well-motivated.\n\n3.\tThe proposed method is novel.\n\n4.\tExperimental results demonstrate the effectiveness of the proposed approach.\n\nWeaknesses：\n\n1.\tThe time complexity of SHIFT is relatively high, with a generation time of approximately 0.2 seconds, about 10 times longer than that of methods such as PGD, which typically require around 0.02 seconds.", "was_revised": true, "final_justification": "The authors clearly explained the data collection and experimental setup, supplemented the explanation and experimental verification of the difference between SHIFT-I and SHIFT-O, and the experimental results of SHIFT variants. The authors addressed my inquiry regarding generation time. As all my major concerns have been properly resolved, I maintain my original score.", "rebuttal": "Dear Reviewer u3xt:\n\nThanks for your insightful feedback and we will address your concerns in the following.\n\nQ1: In lines 242–243, is an offline dataset collected from a clean environment required, or is it necessary to generate trajectory data by interacting with the clean environment using $\\pi_{ref}$ to assist training? What are the requirements for $\\pi_{ref}$? Does it need to ensure a certain level of coverage over the state or trajectory distribution?\n\nA1: Yes, SHIFT requires collecting sufficient trajectory data from a clean environment to train the conditional diffusion model. To this end, we use a pre-trained reference policy $\\pi_{ref}$ that performs well in the unperturbed environment. To ensure adequate coverage over the trajectory distribution, we incorporate a small amount of random actions (i.e., exploration) during data collection. This helps the conditional diffusion model learn not only high-reward behavior but also the broader dynamics of the environment, including suboptimal trajectories.\n\nQ2: Why does SHIFT-I not incorporate information about $a_{t-1}$? What would the performance of SHIFT-I be if $a_{t-1}$ were included? Similarly, how would the performance of SHIFT-O be affected if it also omitted ${a_{t-1}}$?\n\nA2: This is a great question. In SHIFT-I, we deliberately omit $a_{t-1}$ to relax the conditional generation space, allowing the diffusion model to search for perturbed states that better align with the observed history. Including $a_{t-1}$ in SHIFT-I would overly constrain the generation distribution, introducing subtle inconsistencies—similar to those observed in SHIFT-O—that harm historical alignment.\nEmpirically, including $a_{t-1}$ in SHIFT-I results in an attack performance of 11.3 (4.1) in the Freeway environment against DP-DQN, which is comparable to the original result of 10.6 (2.5). However, stealthiness is reduced: the Wasserstein distance of SHIFT-I increases from $0.84 (0.2) × 10^{-3}$ to $0.90 (0.2) × 10^{-3}$, indicating degraded historical alignment. Combined with SHIFT-I’s inherently lower trajectory faithfulness, this makes the attack significantly less stealthy.\n\nOn the other hand, omitting $a_{t-1}$ from SHIFT-O improves attack performance (from 21.8 (3.2) to 14.1 (2.8)) in the Freeway environment against DP-DQN but at the cost of trajectory faithfulness: SSIM drops from 0.9990 (0.0014) to 0.9915 (0.0015), and LPIPS increases from 0.0008 (0.0014) to 0.0121 (0.001). \n\nIn summary, the inclusion or omission of $a_{t-1}$ reflects a trade-off between historical alignment and trajectory faithfulness. Each SHIFT variant is designed to balance this trade-off in accordance with its design goal.\n\nQ3: Regarding the experiments in Table 2, how would the performance of SHIFT be affected if the attack frequency were reduced, for example, in variants like SHIFT-I-0.25 and SHIFT-O-0.25?\n\nA3: We provide an ablation study on different attack frequencies—including 1.0, 0.5, 0.25, and 0.15—for both SHIFT-I and SHIFT-O in Appendix D.6. The results show that even at lower frequencies such as 0.25, SHIFT maintains a strong attack performance, demonstrating its effectiveness under limited perturbation budgets.\n\nQ4: The generation time of SHIFT is approximately 0.2 seconds. It remains unclear whether, in practice, the agent can detect the presence of an attack by observing differences in generation times of subsequent states between normal (no-attack) and SHIFT attack scenarios. Furthermore, it is of interest to investigate how increasing the generation iterations of methods such as PGD, PA-AD-TC, MinBest, and PA-AD, or decreasing the number of reverse steps in SHIFT—thereby aligning their adversarial state generation times to the same scale—would affect their comparative performance.\n\nA4: The 0.02 seconds reported for PGD, MinBest, and PA-AD are based on 10 steps of gradient descent. As shown in the SA-MDP paper[1], using 50 steps is also common. We tested these methods with 50 steps in Freeway under DP-DQN defense. In terms of attack performance,  PGD(1/255) with 50 steps has 29.8(1.0) compared with 10 steps 30.0(0.9) and PGD(15/255) with 50 steps has 29.2(0.8) compared with 10 steps 29.0(1.0). This is mainly because even with 10 steps, perturbation has reached the budget limit under PGD attack. While the generation time increased to an average of 0.1 seconds, their attack performance did not improve, as they still failed to bypass DP-DQN. In this setting, SHIFT takes about twice the time but offers significantly stronger attack performance. However, it is challenging to reduce SHIFT's generation time further—reducing the number of reverse steps in SHIFT would degrade the quality of adversarial samples. That said, our experiments were conducted under limited compute resources (a single RTX 3090). With more powerful hardware, the sampling time of SHIFT could potentially be further reduced.\n\nWe hope our detailed rebuttal could address most of your concerns. If you have further questions please let us know and we will try our best to clarify them.\n\nSincerely,\n\nAuthors of Submission 9986\n\n[1] Zhang et. al., Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations. NeurIPS 2020."}, {"review_id": "E5wS6r40le", "reviewer": "Reviewer_2wp2", "summary": "The paper introduces a novel attack method called SHIFT, which leverages diffusion models to generate adversarial state perturbations in reinforcement learning (RL) systems, particularly in vision-based environments.  SHIFT is a policy-agnostic, diffusion-based attack framework designed to generate perturbed states that are semantically different from the true states while remaining realistic and aligned with the agent's history. The attack is grounded in three key properties: semantic-altering, historically-aligned, and trajectory-faithful. SHIFT uses classifier-free guidance to generate history-aligned states and policy guidance to induce semantic changes, ensuring the perturbations are both effective and stealthy.", "questions": "See weaknesses. Overall, the paper presents a significant advancement in the field of adversarial attacks on RL systems by introducing SHIFT, a powerful and stealthy diffusion-based attack method. The strengths of the paper lie in its innovative approach, comprehensive evaluation, and theoretical contributions. However, the computational complexity, limited scalability of SHIFT are areas that need further improvement.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper introduces SHIFT, a novel diffusion-based adversarial attack method that goes beyond traditional $l_p$-norm constraints. This approach leverages the power of diffusion models to generate semantically meaningful perturbations, which is a significant advancement in the field of adversarial attacks on RL systems.\n\n2. SHIFT focuses on generating perturbations that alter the semantics of the input states, which is a critical aspect often overlooked by existing attacks. By changing the meaning of the input, SHIFT can significantly impact the agent's decision-making process.\n\n3. The authors conduct extensive experiments across multiple environments, including Atari games, Doom, and an AirSim autonomous driving simulator. This demonstrates the generalizability of SHIFT across different types of RL tasks and environments. SHIFT is tested against several state-of-the-art defenses, including regularization-based methods and diffusion-based defenses. The results show that SHIFT can effectively break these defenses, highlighting its potency.\n\n\nWeaknesses:\n\n1. Computational Complexity. The use of diffusion models and the need for classifier-free and policy guidance introduce significant computational overhead. Training the conditional diffusion model and the autoencoder-based anomaly detector can be resource-intensive, which may limit the practical applicability of SHIFT in real-time scenarios.\n\n2. Sampling Efficiency. While the authors mention using the EDM formulation to improve sampling efficiency, the overall computational cost of generating perturbations in real-time remains a concern. This could be a barrier for deploying SHIFT in environments that require rapid decision-making.\n\n3. Limited Scalability. The diffusion model needs to be trained separately for each distinct environment, which can be impractical for large-scale deployment. This limits the scalability of SHIFT, especially in scenarios where the RL agent operates in multiple or dynamically changing environments.", "was_revised": true, "final_justification": "Thanks for the authors' responses.  \n\nAccording to the rebuttal, training diffusion models does consume excessive resources and has certain scalability issues. \n\nHowever, I  believe this does not affect the technical insights and contributions of this paper. With the rapid development of diffusion models, I remain optimistic that this limitation will be resolved. \n\nIn light of the above points, I maintain my original score and am inclined to give this paper a positive evaluation.", "rebuttal": "Dear Reviewer 2wp2:\n\nThanks for your insightful feedback, and we will address your concerns in the following.\n\nQ1: Computational Complexity. The use of diffusion models and the need for classifier-free and policy guidance introduce significant computational overhead. Training the conditional diffusion model and the autoencoder-based anomaly detector can be resource-intensive, which may limit the practical applicability of SHIFT in real-time scenarios.\n\nA1: Although SHIFT may incur a certain level of computational overhead, training the diffusion model takes around 4 hours and the autoencoder-based anomaly detector takes about 1 hour on the Freeway environment—both of which can be done in parallel. In comparison, other learning-based attack methods such as PA-AD[1] require around 12 hours on the same hardware to train an attack policy on Freeway. Therefore, SHIFT requires significantly less training time than PA-AD, a state-of-the-art learning-based attack method.\n\nOn the defense side, prior defense methods such as SA-DQN and WocaR-DQN require more than 24 hours to train the defense policies, which is longer than the training time of our SHIFT attack. This creates a sufficient time window for SHIFT to inject adversarial samples before such defenses are deployed.\n\n[1] Sun et al., Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL. ICLR 2022\n\nQ2: Sampling Efficiency. While the authors mention using the EDM formulation to improve sampling efficiency, the overall computational cost of generating perturbations in real-time remains a concern. This could be a barrier for deploying SHIFT in environments that require rapid decision-making.\n\nA2: While SHIFT may introduce a moderate sampling overhead, we emphasize that all experiments were conducted using a single RTX 3090 GPU. Despite this relatively limited computing resource, SHIFT achieves practical sampling times (approximately 0.2 seconds per sample). With more powerful hardware, the efficiency of SHIFT could be further improved, making real-time deployment increasingly feasible.\n\nQ3: Limited Scalability. The diffusion model needs to be trained separately for each distinct environment, which can be impractical for large-scale deployment. This limits the scalability of SHIFT, especially in scenarios where the RL agent operates in multiple or dynamically changing environments.\n\nA3: Thank you for highlighting this important scalability concern. In our current setup, we do train a separate conditional diffusion model for each environment. However, a promising direction for future work is to train a single conditional diffusion model across multiple environments by leveraging the history conditioning mechanism. The history input provides contextual cues that help the model infer which environment it is operating in, preventing mode collapse or sample confusion.\n\nA similar idea has been explored in TD-MPC2 [2], which learns transition dynamics across multiple environments in a shared latent space. Their results show that a single model can handle multiple tasks simultaneously. We could extend this approach by adopting latent diffusion models and performing attack guidance in the latent space, allowing a single diffusion model to support various environments. This strategy would significantly improve SHIFT’s scalability, making it more suitable for complex or dynamically changing environments.\n\n[2] Hansen et al., TD-MPC2: Self-Supervised Decision Making with Latent World Models, NeurIPS(2023)\n\nWe hope our detailed rebuttal could address most of your concerns. If you have further questions please let us know and we will try our best to clarify them.\n\nSincerely,\n\nAuthors of Submission 9986"}, {"review_id": "64zTKCRv2x", "reviewer": "Reviewer_3Yfq", "summary": "This paper proposes SHIFT, a diffusion-based state perturbation adversarial attack against RL agents, which focuses on creating perturbed states that are (i) semantically different, (ii) history-aligned, and (iii) trajectory-faithful. Two variants of SHIFT are proposed, SHIFT-O, conditioned on the true trajectory of the victim agent, and SHIFT-I, conditioned on the victim agent’s observed trajectory (i.e., with states subject to SHIFT perturbations). The proposed attack is motivated by shortcomings of existing attacks and is evaluated on multiple environments and considering different existing defense approaches. The main evaluation considers attack effectiveness, attack realism, historical alignment, and trajectory faithfulness. Additional results regarding computational complexity, ablations, and tradeoffs are included in the appendix.", "questions": "Regarding C1: \nQ1a- How are lines 391-392 backed by the numerical results exactly? And most importantly, what is the precise notion of undetectability/stealthiness considered by the authors (human, automated, both)?  \nQ1b- The ablation on realism guidance in Figure 6a does not seem meaningful, is it not expected that the reconstruction error will decrease if SHIFT does actively minimize it? \n\nRegarding C2: \nQ2- How well can diffusion-generated image detectors such as DIRE [B] detect perturbed states generated by SHIFT?  \n\nRegarding C3: \nQ3- The formulation in [59] is not exclusive to attacks bounded by an L_p norm. Why was its considered perturbation set not adapted to use, e.g., the reconstruction error from the auto-encoder, or some other notion better aligned with SHIFT? \n\nRegarding C4: \nQ4a- How does the trilemma follow directly from the definition of the SHIFT framework? \nQ4b- How does Table 2 present conclusive evidence that no other method could dominate the ones shown in all key metrics? Also, wouldn’t it be possible for a method to dominate all others while still having to tradeoff said key metrics? Can anything be said about how any of these methods can tradeoff one key metric for another? \nQ4c- The statement that only two out of three properties can be optimized at once (lines 333-334 and 408-410) is not clear. Does this refer to two specific properties, if so, which ones? Alternatively, does it refer to any possible pair of properties? How is this claim supported by the SHIFT framework or the numerical results? \n\nRegarding C5: \nQ5a- Why were the Freeway environment and the DP-DQN defense chosen for the “key metric” evaluations?  How do these key metrics look like with different defenses or when considering only the vanilla DQN? Are there any relevant differences when considering other environments? \n\nRegarding C6: \nQ6a- What architecture and training scheme were used for the autoencoder? The policy guidance (line 303) seems to perform a single gradient descent update with a step size of one, why was this chosen over multiple steps and/or a smaller step size? What would be the expected impact of modifying this policy guidance step? \nQ6b- What sort of resources are considered for the “20 hours” needed to train GRAD and PROTECTED? How do they compare to the resources used by the authors? \n\nAdditional questions: \nQ7- While the threat model is not far away from prior work on RL (e.g., [16]), it is quite different from prior work on computer vision. What is the main application area of the current work? In what setting can one assume the attacker is able to completely alter the visual input of the victim? What is its practical relevance? In line 104 the authors refer to this as a “worst-case scenario”, but this is worst-case from the perspective of the victim, how can one justify the relevance of an attack in this setting, which is a best-case scenario for the attacker? \n\nQ8- In line 293 the authors state “Thus s_t (perturbed) is semantically distinct from the true state s_t and achieve attack performance”. What is meant by “attack performance”? How does semantic distinction from the true state entail attack effectiveness? Is it not the case that the attacks that allegedly fail to cause a semantic change can also be effective (while some attacks that cause a semantic change can be ineffective)? \n\nQ9- What is meant by “which agent has no access during testing” in line 374? \n\nQ10- What is the difference between D(.) and Proj_{S*}(.)? Why is it necessary to introduce D(.) in line 186? Why is “(after projection onto S*)” included in definitions 3.4 and 3.5? Is it not implied that all observed states are perturbed in this scenario and therefore already projected onto S*? \n\nQ11- The authors mention a well-trained policy “with exploration to ensure coverage” in line 243. What does this mean exactly? \n\nQ12- What is the meaning of “connects the generative models and optimization methods” in line 849?", "limitations": "While the authors include an Impact Statement in the appendix, the limitations of the work are mainly reduced to stating that an attack that “better satisfies” the three properties that SHIFT aims to achieve remains an open problem. Limitations relating to the threat model assumptions (see Q7) or potential defenses and notions of stealthiness that are not explored (see C2 and C3) may require further discussion.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nS1- Intuitive description and illustrations for SHIFT-O and SHIFT-I, which highlight how it is distinct from existing attacks\nS2- Concrete definitions for the three fundamental attack properties of SHIFT\nS3- Numerical results comparing SHIFT to existing attacks and evaluating its effectiveness against existing defenses\nS4- Additional ablations and results in the appendix provide a more comprehensive evaluation of SHIFT\nS5- Code is provided to ensure reproducibility\n\nWeaknesses:\nW1- Inconsistent motivation: in particular, the fact that SHIFT aims for semantic changes is counterintuitive. In line 54 the authors argue that the attacks from [31] are not stealthy towards humans, however, when considering prior works on adversarial attacks in general, retaining semantic consistency between the perturbed and original image would make more sense if the goal were stealthiness towards humans. For example, this has been argued by [A]. In short, even though some experimental results aim to demonstrate SHIFT’s stealthiness, it is quite counterintuitive that an attack encouraging semantic changes should be expected to be stealthy.\n\nW2- Unconvincing evaluation: various aspects of the numerical results raise questions. For example, using the auto-encoder reconstruction error to assess realism seems unfair, since unlike other methods, SHIFT is optimized to be consistent with the auto-encoder’s notion of realism. Figure 6c seems more compelling (although l2 distance to true states is still not an ideal metric for realism), yet this figure is never explained in-text (not even in the appendix) which makes it hard to interpret, what does the “conditional diffusion model” correspond to in that plot? The choice of environments, baseline attacks, and defenses also raises some questions (see “Major Comments” below).\n\nW3- Issues with writing/structure: the paper contains many typos and the writing is often unclear, hurting the clarity of the paper (some specific instances are listed below). In terms of structure, the way content corresponding to related work and results is split between the main paper and the appendix makes it hard to understand SHIFT’s motivation and advantages.\n\nReference:\n\n[A] Hosseini, Hossein and Radha Poovendran. Semantic Adversarial Examples. In Proc. of Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018.\n\nMajor Comments\n\nC1- Stealthiness/undetectability is introduced as one of the main features of SHIFT (line 13), yet its evaluation is not convincing. Lines 90-91 claim that the stealthiness of SHIFT is demonstrated by the reconstruction error, Wasserstein-1 distance, LPIPS and SSIM. As mentioned in W2, the reconstruction error is not a fair metric since SHIFT actively minimizes it, unlike other methods. In lines 391-392 the authors claim they outperform all baselines in terms of stealthiness, but Table 2 shows many baselines are better in terms of Wasserstein-1 distance, and for SHIFT-I this is also the case regarding SSIM and LPIPS. A more realistic assessment of the performance follows in lines 393-399, but the opening statement in lines 391-392 is quite confusing. Moreover, the results only cover the Freeway environment with the DP-DQN defense, which further weakens this claim even if SHIFT-I and SHIFT-O indeed dominated all baselines, so it would be helpful to justify why this is the only scenario used to compare to other methods.\n\nC2- Table 6 in the appendix shows some results on detectability using a MAD and a CUSUM detector, however these detectors and their parameters are not justified. Providing the scores from the detectors instead of their thresholded binary results would be more insightful. Additionally, it would be meaningful to discuss methods that detect diffusion-generated images such as DIRE [B]. \n[B] Wang et al. DIRE for Diffusion-Generated Image Detection. In Proc. of International Conference on Computer Vision (ICCV), 2023. \n\nC3- Regarding the evaluated defenses, while SHIFT is effective in bypassing defenses against bounded L_p norm attacks, which use adversarial training or diffusion, the fact that they assume traditional attacks (with a bound on their L_p norm) makes them unsuitable to properly evaluate SHIFT’s effectiveness against a defended model. Also, regarding the diffusion-based defenses, even though they are based on diffusion they do not assume diffusion-based attacks, so in this sense an evaluation using DIRE [B] or other defenses against diffusion-based perturbations would be more meaningful. Even for the appendix results in Table 8 regarding [31], the DMBP defense was finetuned to better handle the attacks proposed by [31], and while the authors explain why SHIFT works against diffusion-based defenses in lines 1050-1053, the finetuning of DMBP for [31] raises the question of why no defenses were adjusted to work better against SHIFT. \n\nC4- The “fundamental trilemma” from lines 332-335 is unclear. It is contradictory to the fact that SHIFT aims to optimize the three properties at once. Also, the later statement in lines 399-400 is not supported by the numerical results (and is in direct contradiction with lines 391-392, which suggest that SHIFT achieves stronger attack performance and better stealthiness). Additionally, each component in this “trilemma” corresponds to more than one key metric, further complicating the interpretation of lines 399-400.  \n\nC5- Only the scenario with the Freeway environment and the DP-DQN defense is used for the “key metric” evaluations. Even in the appendix these results are omitted for all other defenses/environments. This seems unfair since DP-DQN is a defense against norm-bounded attacks (like most of the baseline attacks), thus evaluating the key metrics only when such a defense is present puts into question the supposedly superior performance of SHIFT.  \n\nC6- Important implementation details are missing. Although the authors provide code in the supplementary material, the training and architecture of the realism enhancement autoencoder is not described with sufficient detail in the paper. Additionally, the authors mention GRAD and PROTECTED “take more than 20 hours” to train (lines 813-815), but it is unclear how does this compare to SHIFT or other approaches, since only test-time cost is reported for SHIFT.  \n\nC7- As a final comment, the discussion of numerical results seems somewhat shallow for various results. For example, in Sections 4.1 and D.1 there is no discussion regarding differences (or similarities) between the different environments, similarly in D.6 the authors mention “the reduction becomes more pronounced as the attack frequency increases” in line 1029, but do not comment on certain outliers like SHIFT-I in Freeway (reward increases with frequency) or the high variance for multiple scenarios in the Doom environment.  \n\nOther typos/unclear passages: \n1- “environments with raw pixel images as input” suggests that the environments (and not the agents) have image inputs.  \n2- “either can…” in line 53. \n3- It is unclear what the authors mean by “stealthy from human perspectives” in line 54. \n4- The meaning of “classifier-free guidance” (line 80) is not explained when introduced. \n5- In lines 87 and 91, authors use “and” instead of commas when listing different elements, which can be confusing. This is also the case in lines 224-227. \n6- In line 118 the statement of “being detected” does not specify who/what is assumed to perform the detection. \n7- In (1) it seems “t” should be “i\". It is also unclear why the theta that parametrizes epsilon also parametrizes mu and sigma, especially as line 136 states that sigma can be “predicted through a neural network or set by a predetermined scheduler”. Moreover, theta is not properly introduced. \n8- “realistic perturb states” in line 145.  \n9- “quantified metrics” in line 216. \n10- “the training the” in line 230. \n11- “the static and dynamic views” in line 235 has an unclear meaning (see also “static and dynamic perspectives” in lines 734-735). \n12- “the Freeway” in line 258. \n13- “under true state” and “victim’s performance loss” in lines 277-278. \n14- “achieve attack performance” in line 293. \n15- “Figures 2” in line 319. \n16- Lines 333-335 are quite unclear, see also C4 above and Q4a, Q4b, and Q4c below. \n17- “any attack then applies” in line 351, “Algorithm 1,2” in line 353. \n18- The environments used for evaluation are not clearly introduced, line 355 refers to “four Atari environments, Doom game, and Airsim autonomous driving simulator”, yet Table 1 presents results for Pong, Freeway, Doom, and Airsim, and Table 2 focuses on Freeway. Section D.1. in the appendix refers to “the remaining two Atari environments: BankHeist and Roadrunner”, clarifying that Pong and Freeway are the other two Atari environments, but this is not clear when introduced in Section 4. Table 5 also refers to a “Highway” environment. Additionally, it is redundant to include the environment for tables that present results for a single environment (Tables 2, 6, 8, 9, and 10), and sometimes “Env”/”Environment” or environment names are included erroneously (see Tables 4, 8, 9, and 10). \n19- “which agent has no access during testing” in line 374. \n20- “indicating less trajectory faithful” in line 398. \n21- The expression in line 773 would be more clearly explained with words. \n22-“decreases the searching space” in line 779. \n23-“in solving” in line 839. \n24- “traditional _p-norm” in line 877. \n25-“function” in line 934. \n26- “random” in Algorithm 1 is not properly defined. \n27- “that simulate real city traffic” in line 978. \n28- the Y label “Percentage (0-1)” in Figure 6b is unclear, and it is hard to see how it relates to the deviation rate mentioned in the figure caption. \n29- “table below” in line 1006 seems to refer to Table 6, but this is not obvious. \n30-“transform” in line 1049. \n31-“realistic” in line 1059, “Definitions 3.2” and “Definitions 3.4” in line 1060. “Table 2,9” in line 1062. \n32-“indicating less realistic than our methods” in line 1064", "was_revised": true, "final_justification": "Authors have address most of my comments during the rebuttal.", "rebuttal": "Dear Reviewer 3Yfq:\n\nThanks for your insightful feedback and we will address your concerns.\n\nQ1a: Semantic change is not considered as stealthy. What is the exact definition of stealthiness? \n\nA1a: In the state perturbation threat model, the agent only observes perturbed states and has no access to the true states, unlike in traditional vision settings. Therefore, semantic changes can be considered stealthy if they remain physically realistic and plausible, as defined in Definition 3.2.\n\nOur definition of stealthiness is grounded in three apsectives: Realism (Def. 3.2), Historical Alignment (Def. 3.4), and Trajectory Faithfulness (Def. 3.5). Due to the computational intractability of projection onto the valid states, we use approximate metrics to evaluate stealthiness: autoencoder reconstruction error (for realism), Wasserstein distance (for historical alignment), and SSIM/LPIPS (for trajectory faithfulness).\n\nTable 2 provides quantitative evidence: SHIFT achieves strong attack performance while maintaining stealth, as indicated by lower reconstruction error and Wasserstein distance comparable to low-budget baselines. Additional detection results in Appendix D and preliminary DIRE outcomes (see Q2) reinforce this finding. Although SSIM and LPIPS are not directly usable for detection—since they require access to the true states—they help assess how trajectory faithful the perturbations are.\n\nIn summary, SHIFT surpasses all other baselines in attack performance and has comparable stealthiness with low-budget baselines, justifying our claim in lines 391–392.\n\nQ1b: Unfair comparison for reconstruction error.\n\nA1b: The unfair comparison concern is addressed by our ablation study (Figure 6a): SHIFT-O without autoencoder-based realism enhancement achieves reconstruction error of 1.06±0.5 in Freeway and is still lower than all baseline attacks (In Table 2). This occurs because our conditional diffusion model is trained upon clean trajectories and can generate realistic states. The conditional diffusion model ensures physical plausibility, making SHIFT's realism advantage intrinsic rather than evaluation-biased. \n\nQ2: Diffusion generated sample detector. \n\nA2: We conducted a preliminary evaluation of the DIRE detector. Given an input image $x$, DIRE first performs a reverse diffusion sampling process to add noise until reaching the standard normal prior $\\mathcal{N}(0, 1)$, then generates a new image $x’$ via the diffusion sampling process. The DIRE distance is defined as the $l_2$ distance between $x$ and $x’$. Since diffusion-generated images tend to remain closer to the diffusion sampling distribution and the real image distribution is different from the diffusion sampling distribution, DIRE assumes that real images will have higher distances than samples synthesized by a diffusion model.\nWe tested this on the Pong environment. Non-attack diffusion samples yield low DIRE distances (0.05 ± 0.15), while true states show higher distances (0.61 ± 0.77), consistent with DIRE's assumptions. However, SHIFT-O, which introduces policy guidance during sampling, shifts the distribution further, achieving an even higher distance (0.94 ± 0.60). Although the higher mean might seem distinguishable, the high variance prevents reliable single-frame detection.\n\nAs an example, here are five consecutive samples of DIRE distances (true state, SHIFT-O) from our experiment: [(0.59,0.63), (0.77,0.51), (0.72,1.23), (0.60,0.44), (0.60,0.48)]. Despite SHIFT-O having a higher average, 39% of true states throughout an episode with 2000 steps have a higher DIRE distance, making instance-level detection unreliable. Although it is still possible to use the mean DIRE distance to detect the existence of attacks, this requires aggregating statistics over a trajectory. By the time the attack is detected, it may have already influenced the victim’s decision-making and degraded its performance.\n\nThese findings suggest that DIRE is not effective for detecting SHIFT attacks. More broadly, detection methods trained solely on clean data may struggle to identify adversarial samples since adversarial attacks may distort the distribution in an unexpected way.\n\nQ3a: Why was SA-MDP’s considered perturbation set not adapted to use, e.g., the reconstruction error from the auto-encoder, or some other notion better aligned with SHIFT?\n\nA3a: SA-MDP's $l_p$ norm constraint enforces spatial proximity but cannot capture SHIFT's multi-dimensional stealth requirements: realism, history alignment, and trajectory faithfulness. These properties are inherently interdependent yet distinct—no single metric (like reconstruction error) suffices. While reconstruction error partially addresses realism, it ignores history alignment and trajectory faithfulness. Forcing SHIFT into SA-MDP’s framework would sacrifice its core innovations for theoretical convenience.\n\nQ3b: Finetuned DMBP against SHIFT?\n\nA3b: As long as the attacker has the knowledge of how the victim finetune the diffusion model, the attacker can mimic the same process to generate the diffusion model as its new conditional diffusion base and reapply policy/realism guidance.\n\nQ4: Questions for the trilemma.\n\nA4: The trilemma is an empirical observation, not coming from SHIFT’s framework. We do not rule out the possibility of future methods overcoming this trade-off. Existing attacks like PGD, MinBest, PA-AD, and attacks based on high-sensitivity directions (e.g., blur, rotation) prioritize historical alignment and trajectory faithfulness but lack semantic change. SHIFT-O achieves semantic change and trajectory faithfulness, with slightly lower historical alignment, while SHIFT-I achieves semantic change and historical alignment, with slightly lower trajectory faithfulness.\n\nQ5: Table 2 only covers the Freeway environment and DP-DQN defense.\n\nA5: We provide extra results for Pong and SA-DQN below. Since SA-MDP is a relatively weak defense, other attack baselines also perform well. In terms of the stealthiness metric, it shows the same trend as Table 2. Table 2 focuses on DP-DQN because it represents the strongest existing defense, robust against large-budget attacks like PGD-15/255. \n\n| Attack| Reward↓| Dev (%)↑| Recons.↓| Wass. (×10⁻³)↓ | SSIM↑| LPIPS↓|\n|----------------|-----------------|---------------|---------------|----------------|----------------|---------------|\n| PGD-1/255      | 21.0 ± 0.0      | 0 ± 0.0       | 1.86 ± 0.3    | 1.33 ± 0.4     | 0.9737 ± 0.012 | 0.0876 ± 0.0403 |\n| PGD-15/255     | -21.0 ± 0.0     | 73.6 ± 8.6    | 3.43 ± 0.2    | 5.03 ± 0.4     | 0.7408 ± 0.026 | 0.3952 ± 0.0436 |\n| Transform(1,0) | -21.0 ± 0.0     | 62.4 ± 1.7    | 5.53 ± 0.2    | 1.18 ± 0.1     | 0.9045 ± 0.0074 | 0.0759 ± 0.0061 |\n| SHIFT-O-1.0    | **-21.0 ± 0.0** | **64.8 ± 0.7** | **0.77 ± 0.3** | **1.24 ± 0.2** | **0.9884 ± 0.013** | **0.0647 ± 0.0924** |\n| SHIFT-I-1.0    | **-20.8 ± 0.4** | **45.7 ± 3.0** | **0.88 ± 0.3** | **1.21 ± 0.3** | **0.9799 ± 0.016** | **0.0805 ± 0.0777** |\n\nQ6: What architecture and training schema is used for autoencoders? Why only perform a single step and step size 1 in the realism guidance? Resources used by GRAD?\n\nA6: The autoencoder uses 3 convolutional layers followed by a linear layer; the decoder mirrors this structure. It is trained with the Adam optimizer and MSE loss. We use a single realism guidance step with step size 1 as a balanced choice—small enough to preserve sample quality, yet large enough to enhance realism. A single step also ensures efficiency, as multiple steps would significantly increase computing time during SHIFT generation. GRAD uses V100 GPU and 20 hours for training defenses for environments such as Hoppers and Walkers, which does not use images as states. Thus, it is hard to migrate GRAD to the image domain due to computational overhead. We only have a single RTX 3090 and the training time for SHIFT is around 3-4 hours.\n\nQ7: What is the main real-world use case for SHIFT? In what situations can an attacker fully control the victim’s visual input? How is it realistic or relevant in practice?\n\nA7: SHIFT applies to real-world control tasks with visual input, such as robotics and autonomous driving. It assumes control over the agent’s observations, which is a common setup in prior work (e.g., SA-MDP, PA-AD). We also evaluate SHIFT under limited budgets (Appendix D.6). This is not a true best-case for the attacker, as SHIFT only alters observations, not actions. A stronger attacker would have total control of the victim and could directly manipulate the agent’s actions, which we do not assume.\n\nQ8: Questions related to semantic changed states effectiveness.\n\nA8: Attack effectiveness means the perturbed states cause the agent to take lower-reward actions. For example, in Figure 2, SHIFT generates semantically altered states that mislead the car into driving through a crosswalk with a pedestrian, causing a collision and reducing reward. While non-semantic attacks like PGD can degrade DQN performance, they fail against stronger defenses like diffusion-based DP-DQN (Table 2), where SHIFT remains effective.\n\nA9: During the actual testing episode of the state perturbation attack, the victim agent only sees the perturbed states and has no access to the true states. Therefore, the victim cannot compute SSIM nor LPIPS.\n\nA10: There may be multiple valid projection points, so we introduce D(⋅) to provide a mathematically rigorous and well-defined definition.\n\nA11: The reference policy is used to collect clean trajectories for training the diffusion model with some random actions (i.e., exploration) to ensure the data covers a diverse range of trajectories, including both low and high reward cases. Also see A1 to Reviewer u3xt.\n\nA12: Black et al. use reinforcement learning, an optimization method, to train a diffusion model, which is a generative model. This bridges generative modeling and optimization-based training.\n\nWe hope our detailed rebuttal address could your concerns."}]

Author rebuttal:

Meta-review: This paper proposes a diffusion-based adversarial attack that perturbs state inputs for RL agents. The authors show that existing defenses, which are primarily designed against Lp-norm based perturbations, are vulnerable to this new attack. The method is evaluated across multiple environments and against a variety of defense mechanisms.

The authors' rebuttal successfully addressed most technical concerns. While some reservations remain regarding the attack's realism (Reviewer vqkJ) and scalability (Reviewer 2wp2), the reviewers agree on the paper's core contribution in exposing the limitations of current defense algorithms for RL agents. Therefore, we recommend acceptance.
Decision: Accept (poster)