Diffusion Guided Adversarial State Perturbations in Reinforcement Learning

Authors: Xiaolin Sun, Feidi Liu, Zhengming Ding, Zizhan Zheng
Primary area: reinforcement learning
Keywords: Reinforcement Learning, Adversarial Example, Diffusion Model

Abstract:
Reinforcement learning (RL) systems, while achieving remarkable success across various domains, are vulnerable to adversarial attacks. This is especially a concern in vision-based environments where minor manipulations of high-dimensional image inputs can easily mislead the agent's behavior. To this end, various defenses have been proposed recently, with state-of-the-art approaches achieving robust performance even under large state perturbations. Upon closer investigation, however, we found that the effectiveness of the current defenses is due to a fundamental weakness of the existing $l_p$-norm constrained attacks, which can barely alter the semantics of the input even under a relatively large perturbation budget. In this work, we propose SHIFT, a novel diffusion-based state perturbation attack to go beyond this limitation. Specifically, we train a history-conditioned diffusion model, enhanced with policy guidance and realism detection to generate perturbed states that are semantically different from the true states while remaining realistic and history-aligned to avoid detection. Evaluations show that our attack effectively breaks existing defenses, including the most sophisticated ones, and significantly lowers the agent's cumulative reward in various Atari games by more than 50\%. The results highlight the vulnerability of RL agents to semantics-aware adversarial perturbations, indicating the importance of developing more robust policies for safety-critical domains.

Decision: reject
Reviewer scores: 3, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1730580084136

thread_type:
official_review

<thread_object>
creator: reviewer_d5ut

<content>
summary:
This paper proposes a novel non-$\ell_p$ attack algorihtm for image-observation reinforcement learning, based on a history-conditioned diffusion model. The generated attacks are semantically meaningful while misleading to the agent. Experiments show that the proposed SHIFT attack can significantly break existing robust RL learning algorithms that are mainly designed for $\ell_p$ attacks.

strengths:
- The paper points out the limitation of mostly-studied $\ell_p$ attack model for image-observation reinforcement learning environments. By utilizing a denoising diffusion probabilities model (DDPM), the paper achieves stealthy and realistic attack by altering the semantic meaning of the original state. 
- The paper clearly defines the concept of valid and realistic states and adopts an autoencoder to enhance the realism of the generated attacks.
- Comparison with existing methods show that SHIFT can lower the reward of agent while having low state reconstruction error.
- The paper also proposes a possible defense method agains the new attack model.

weaknesses:
- Although the proposed attack uses methods such as autoencoder guidance to enhance the realism of the perturbed states, it is not guaranted or bounded like $\ell_p$ attacks, making it hard to compare and evaluate the stealthiness of the perturbations. It is not clear to me whether the reconstruction error can effectively represent the realism of the state perturbation. It would be better if the authors can provide a gif or video showing the full perturbed trajectory.
- The experiments are not very informative. It is not surprising that RL agents learned via $\ell_p$ attack assumptions will break under the proposed attack. But more empirical study can be done to verify the effectiveness of the proposed design. For example, how does the varying attack strengh influence the attack effects?

questions:
As the author mentioned, the proposed method uses a myopic target action manipulation objective which can be sub-optimal. Is there a way to improve it? For example, how can it be combined with RL-based attack methods such as PA-AD?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730636203151

thread_type:
official_review

<thread_object>
creator: reviewer_37tk

<content>
summary:
The submission claims to find that the effectiveness of the current defenses is due to a fundamental weakness of the existing $\ell_p$-norm constrained attacks. Furthermore, the submission proposes a method to go beyond the $\ell_p$-norm bounded adversarial attacks in deep reinforcement learning. The submission evaluates its proposed attacks in Atari games and argues that the proposed attack method of the submission lowers the cumulative rewards of the agent by 50%.

strengths:
AI safety and robustness is an important research area.

weaknesses:
The major claimed contributions of the submission have been previously both mentioned and analyzed in previous work [1]. However, the submission does not refer to these studies, and furthermore, within the existing prior work the main claimed contributions of the submission are rather misplaced and inaccurate. The paper [1] already extensively studies and demonstrates that both deep reinforcement learning policies and current defenses, i.e. robust deep reinforcement learning, are not robust against semantically meaningful adversarial attacks and this study further reveals the need to have robustness beyond $\ell_p$-norm bounded attacks.

Not only has the necessity of considering beyond  $\ell_p$-norm bounded attacks already been discussed in previous work, furthermore the approach proposed in this paper [1] achieves higher degradation on the policy performance without even having access to the training details, the policy network (i.e. black-box adversarial attacks), and further without even training any additional network to produce such adversarial examples.

The submission substantially lacks appropriate references, and further positioning itself within the existing prior work and clarifying its main contributions within these studies. The claimed contributions of the submission are misplaced and incorrect.  

[1] Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness. AAAI Conference on Artificial Intelligence, AAAI 2023.

Furthermore, the submission lacks main technical details to interpret their experimental results. Not a single experimental detail is provided regarding deep reinforcement learning. These details are essential for reproducibility and further to interpret and analyze the experimental results provided in the submission. However, the submission does not provide any information on this. 

The submission only tests their algorithm in 3 games from Atari. However, in adversarial deep reinforcement learning it is usually tested in more games [1,2,3,4]. In particular, RoadRunner is missing from the baseline comparison.

[1] Robust deep reinforcement learning against adversarial perturbations on state observations, NeurIPS 2020.

[2] Robust Deep Reinforcement Learning through Adversarial Loss, NeurIPS 2021.

[3] Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness, AAAI 2023.

[4] Detecting Adversarial Directions in Deep Reinforcement Learning to Make Robust Decisions, ICML 2023.

The submission also refers to main concepts in the adversarial machine learning literature with inaccurate wording. For instance, in the introduction the submission writes: 

*“by poisoning its observation (Huang et al., 2017; Zhang et al., 2020a)”*

However, poisoning attacks in adversarial machine learning literature refer to completely something else and these papers are not poisoning attacks. These papers are test time attacks. Thus, it is misleading to use the word poisoning here. 

One thing I find ineffective is that the submission refers to a long list of papers such as these [1,2,3,4,5], however, somehow still misses the prior work that substantially coincides with the main claimed contributions of the submission and even further these prior studies already demonstrate the claimed contributions of this submission.

[1] Kangjie Chen, Shangwei Guo, Tianwei Zhang, Xiaofei Xie, and Yang Liu. Stealing deep reinforcement learning models for fun and profit. ACM Asia Conference on Computer and Communications Security, 2021.

[2] Mengdi Huai, Jianhui Sun, Renqin Cai, Liuyi Yao, and Aidong Zhang. Malicious attacks against deep reinforcement learning interpretations. ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.

[3] Yunhan Huang and Quanyan Zhu. Deceptive reinforcement learning under adversarial manipulations on cost signals. Decision and Game Theory for Security (GameSec), 2019.

[4] Zikang Xiong, Joe Eappen, He Zhu, and Suresh Jagannathan. Defending observation attacks in deep reinforcement learning via detection and denoising. Machine Learning and Knowledge Discovery in Databases: European Conference 2023.

[5] Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala I. Al-Fuqaha, Dinh Thai Hoang, and Dusit Niyato. Challenges and countermeasures for adversarial attacks on deep reinforcement learning. ArXiv 2020.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662420231

thread_type:
official_review

<thread_object>
creator: reviewer_l64s

<content>
summary:
This paper introduces SHIFT, a diffusion-based adversarial attack that targets RL agents in vision-based environments by creating realistic, history-aligned state perturbations that go beyond traditional lp-norm attacks. Unlike existing methods, SHIFT generates semantic changes that significantly impair the agent's performance, bypassing even the most advanced defenses. Results demonstrate the attack's efficacy, reducing cumulative rewards by over 50% in Atari games, underscoring the need for more robust defenses in RL.

strengths:
1. This work proposes semantic-level RL attacks using conditional diffusion models that balance semantic changes, realism, and historical consistency. The insight is novel.
2. Identifies a fundamental weakness in lp-norm attacks - their inability to meaningfully alter state semantics despite large perturbation budgets.
3. Employs EDM to enhance generation efficiency, making the approach more feasible

weaknesses:
1. Despite using EDM and weighting joint, the paper lacks any systematic analysis of attack efficiency and computational costs.
2. While reporting larger attack budgets, results are limited to PGD and MinBest baselines, missing broader comparative analysis.
3. Experiments are restricted to only three Atari environments, providing insufficient evidence for the method's generalizability.
4. Overall Soundness: While the core idea is interesting, the paper falls short in rigor - lacking ablation studies, methodology analysis, and comprehensive experiments. The current evaluation scope is not convincing enough to support the claims.

questions:
1. The Manipulation Rate and Deviation Rate metrics appear exclusive to SHIFT's diffusion-based approach, raising questions about fair comparison with non-diffusion methods. The necessity of diffusion models needs stronger justification.
2. The paper lacks crucial comparison between DDPM and EDM in terms of both effectiveness and efficiency. This missing analysis weakens the justification for the chosen architecture.
3. Overlooks recent related work [1] about temporally-coupled perturbations

[1]Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbations, Liang et al, ICLR 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677121916

thread_type:
official_review

<thread_object>
creator: reviewer_enaa

<content>
summary:
The paper studies how to generate state perturbations for reinforcement learning, especially perturbation in an unconstrained way, instead of traditional L_p perturbations. The methods are based on diffusion models to generate states with different semantics. The experiments outperforms some existing baselines.

strengths:
1. The paper studies an interesting question of non-L_p attacks, which is largely neglected by existing literature.
2. The methods can scale to image-input domain

weaknesses:
1. One concern/question is that the goal of the paper is not very concrete. The paper said existing methods cannot change the semantics of the image input while this paper can. However, it is not very clear why the attacker has the motivation to change the semantics? In other words, isn't being stealthy beneficial for the attacker? 
2. Some newest/recent defense baselines to my best knowledge [1, 2] are not discussed or compared in experiments. These game-theoretical based defense methods are significantly different from the defense mechanisms discussed in the paper by nature, and more importantly agnostic to the attacker model (which means one only need to change the attacker model to non-L_p accordingly to extend the defense to non-L_p). Therefore, it will be important to evaluate how the attacks performs under such kind of defense strategies.

[1] Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbations ICLR 2024
[2] Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies ICLR. 2024

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732173611217

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response(1)

comment:
We thank all the reviewers for their insightful feedback and constructive criticism regarding our insufficient experiment results. In this general response, we provide more experiment results to address these concerns. 

**Table 1: RoadRunner Results under Our Attack**
| RoadRunner | Reward | Manipulation | Deviation |
|:---:|:---:|:---:|:---:|
| No Attack | 13500(0) | NA | NA |
| DQN | 0(0) | 52%(2%) | 70%(3%) |
| SA-DQN | 260(215.41) | 34%(2%) | 54%(1%) |
| Diffusion History | 1480(788.42) | 9%(2%) | 43%(2%) |

We have added new experiment results on Atari RoadRunner as suggested by Reviewers 37tk and L64s. We have retrained the vanilla DQN model and the SA-DQN model because the pretrained vanilla DQN and SA-DQN models provided by the SA-MDP paper do not work in the RoadRunner environment. Our attack obtains superb performance against SA-DQN and Diffusion History defenses. 

**Table 2-1: Attack Performance of Different Attack Methods**
| **Freeway** | PGD-1/255 |  | PGD-3/255 |  | PGD-15/255 |  | Minbest-1/255 |  | Minbest-3/255 |  | Minbest-15/255 |  | PA-AD-1/255 |  | PA-AD-3/255 |  | PA-AD-15/255 |  | Ours |  |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|  | Reward | Dev (%) | Reward | Dev (%) | Reward | Dev (%) | Reward | Dev (%) | Reward | Dev (%) | Reward | Dev (%) | Reward | Dev (%) | Reward | Dev (%) | Reward | Dev (%) | **Reward** | **Dev (%)** |
| **DQN** | 0 (0) | 86.2 (0.5) | 0 (0) | 100 (0) | 0 (0) | 100 (0) | 0 (0) | 100 (0) | 0 (0) | 100 (0) | 0 (0) | 100 (0) | 0 (0) | 100 (0) | 0 (0) | 100 (0) | 0 (0) | 100 (0) | **0.1 (0.3)** | **54 (1.4)** |
| **SA-DQN** | 30 (0) | 0 (0) | 30 (0) | 0 (0) | 20 (1.6) | 8 (10) | 30 (0) | 0 (0) | 29 (1.4) | 0.4 (0.3) | 20.8 (2.5) | 9.1 (1.1) | 30 (0) | 0 (0) | 30 (0) | 0 (0) | 20.5 (4.4) | 3 (1) | **17.3 (1.5)** | **33 (2)** |
| **DP-DQN** | 30 (0.9) | 3.5 (0.2) | 30 (0.9) | 4.5 (0.3) | 29 (1) | 3.2 (0.1) | 30.2 (1.3) | 3.7 (0.3) | 30.6 (1.4) | 4.1 (0.1) | 29.4 (1.2) | 7.3 (0.2) | 30.8 (1) | 6.5 (0.1) | 31.4 (0.8) | 7.3 (0.2) | 29 (1.1) | 10.3 (1) | **14.6 (1.5)** | **49 (1.9)** |

In response to Reviewer L64s, we have added PA-AD [2] as a new attack baseline, which is considered one of the strongest attacks in the literature. This table compares the performance of PGD, MinBest, PA-AD with budget {1/255, 3/255, 15/255}, and our attack method in Atari Freeway under DQN, SA-DQN and DP-DQN defenses. Standard deviations are reported in parentheses. The results show that our attack method achieves the best attack performance against both SA-DQN and DP-DQN in terms of both reward and deviation rate.

**Table 2-2 Reconstruction Errors and Wasserstein Distances of Different Attack Methods**
| **Freeway** | **PGD** | **1/255** | **PGD** | **3/255** | **PGD** | **15/255** | **MinBest** | **1/255** | **MinBest** | **3/255** |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|  | Recons. | Wass.($\times 10^{-3}$) | Recons. | Wass.($\times 10^{-3}$) | Recons. | Wass.($\times 10^{-3}$) | Recons. | Wass.($\times 10^{-3}$) | Recons. | Wass.($\times 10^{-3}$) |
| **DP-DQN** | 3.45 (0.3) | 3.1 (0.2) | 3.50 (0.3) | 7.4 (0.3) | 4.36 (0.29) | 31 (1) | 3.45 (0.3) | 3.7 (0.2) | 3.53 (0.3) | 9 (0.4) |
|  | **MinBest** | **15/255** | **PA-AD** | **1/255** | **PA-AD** | **3/255** | **PA-AD** | **15/255** | **Ours** |  |
|  | Recons. | Wass.($\times 10^{-3}$) | Recons. | Wass.($\times 10^{-3}$) | Recons. | Wass.($\times 10^{-3}$) | Recons. | Wass.($\times 10^{-3}$) | Recons. | Wass.($\times 10^{-3}$) |
| **DP-DQN** | 5.35 (0.2) | 40 (1) | 3.47 (0.29) | 4.5 (0.2) | 3.60 (0.29) | 12 (0.1) | 6.06 (0.18) | 55 (0.2) | **1.02 (0.5)** | **1.1 (0.2)** |

In response to Reviewer D5uT, we added Wasserstein distance as a new perturbation metric beyond the reconstruction error considered in the original submission. This table shows (1) the average reconstruction error (computed by our autoencoder based realism detector) of a perturbed state generated by different attacks across an episode, and (2) the average Wasserstein-1 distance between a perturbed state and the true state in the previous time step across an episode, under different attacks. The Wasserstein distance was proposed in [3] as an alternative perturbation metric to $l_p$ distances, which measures the cost of moving pixel mass and can represent image manipulations more naturally than the $l_p$ distance. We argue that reconstruction error captures static stealthiness of state perturbation, while the Wasserstein distance to the previous state captures dynamic stealthiness. The result shows that our attack method achieves both lowest reconstruction error and lowest Wasserstein distance compared with other attacks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732556889875

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking forward to your feedback

comment:
Dear Reviewers:

Thank you once again for your insightful and helpful reviews. As the rebuttal phase is coming to a close, we are eager to know if our responses have satisfactorily addressed your concerns. If there is any additional information or clarification we can provide, please do not hesitate to let us know. Thank you so much for your time!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper discusses a novel adversarial attack to RL agents, by creating realistic perturbations using diffusion models. On the positive side, this attack is novel and can generate perturbed states that are semantically different from the true states while remaining realistic to avoid detection. However, it is difficult to quantitatively judge the realism and stealthiness of the proposed attack, as these terms do not have a precise mathematical definition. The evaluation results are not surprising since it is expected that many existing defenses built on Lp norm perturbation are not robust against the proposed attack (which can have a large norm and actually change the semantics). The experiments are not comprehensive enough (lacking environments beyond 3 easy Atari games and ablation studies). Considering these factors, the current form of this paper cannot be accepted at ICLR.