Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies adversarial attacks on the state observation (sensor inputs) channel of a RL agent. Different from previous work (in particular [Zhang et al., 2020]), the authors consider the stealthiness of the adversarial attacks. The contribution of the paper is that it defines the concept of detectability for adversarial attacks on state observation and proposes an algorithm that can compute and carry out such undetectable attacks. 


strength_and_weaknesses: The problem studied in this paper novel and meaningful. That is to consider the stealthiness of adversarial attacks and the trade-off between stealthiness and effectiveness of the attacks. Stealthiness and detection are the important twins in security problems, especially security problems for sequential decision-making systems where attacks are launched not just one time but sequentially.

But the definition of indistinguishability and the detection mechanisms proposed throughout the paper are questionable. The assumptions made in Section 4 are contradicting with the RL algorithm that the author later choose as the focus of the study.

1. The definition of statistical indistinguishability is a strong one. If statistical indistinguishability holds, no detection algorithms can detect such attacks. But the attacks that can avoid being detected by some detection mechanisms don't necessary need to be statistically indistinguishable. The use of statistical indistinguishability as a condition to craft adversarial attacks can lead to no attacking strategy satisfying the condition. Consider an MDP with **continuous** state space. Can the authors give an example of an MDP and an attacking strategy $\pi_{adv}(s)\neq 0$ for some $s\in\mathcal{S}$ such that $\mathbb{P}_{\pi,\mathcal{E}} = \mathbb{P}_{\pi,\mathcal{E}'},\forall (\mathcal{S}\times\mathcal{A})^T$? To make the example simple, we can assume T=2. Such examples can help the readers better understand how strong the definition is.

2.   I agree with the authors that the assumption of the victim knowing a world model $m_v$ is not unrealistic. "victims can learn accurate world models from unperturbed train-time samples". However, if the victim knows the world model, what is the point of interacting with the world to observe the state? The victim can leverage a more efficient algorithm (value iteration, policy iteration + function approximation if the state space is large) instead of the algorithms discussed in the paper. 

3. The detection mechanism used in the experiment is included in the paragraph starting with "Using a dynamics model to detect adversarial attacks." Please highlight the detection mechanism since it is an important factor for the experiments. If I understand it correctly, the detection mechanism considers observations in a single step and an attack is detected if the predicted observation and the actual observation is larger than a threshold $c$. Does this detection technique come from a reference? I have **three** concerns regarding the mechanism. 1. It does not consider the whole trajectory history to detect 2. how do you measure distance for discrete state space (how to set threshold c)? 3. if c is small, even if there is not attack, the actual observation can be different from the predicted observation given the stochasticity of the model.

Other minor comments:
* I came across a paper that studies adversarial attacks on rewards with a definition of stealthiness in it (C1). As a reader, I am curious about the difference between undetectability in attacks in reward and attacks in state observations? 

* In optimal control or MDP or POMDP, adversarial attacks have been investigated by many researchers (C2, C3). These attacks are sometimes called false data injection attacks or sensor attacks. The detector of attacks is usually based on statistical evidence instead of single observations.

* Avoid using "may" in your statement. For example, in definition 4.1, a sampling policy that may conidtion on the whole history. The authors can say "a sampling policy where T can be infinity" or "a sampling policy that can condition on the whole action-observation history". In section 5 "may be unable to detect W-illusory attacks". Instead of using "may", scientific writing should specify under what conditions, humans are able to detect W-illusory attacks and under what conditions, human are not able to do so. Similar examples of using "may" can be found throughout the paper.


[C1] Huang, Yunhan, and Quanyan Zhu. "Deceptive reinforcement learning under adversarial manipulations on cost signals." International Conference on Decision and Game Theory for Security. Springer, Cham, 2019.   

[C2] Mo, Yilin, and Bruno Sinopoli. "False data injection attacks in control systems." Preprints of the 1st workshop on Secure Control Systems. Vol. 1. 2010.

[C3] Pasqualetti, Fabio, Florian Dorfler, and Francesco Bullo. "Control-theoretic methods for cyberphysical security: Geometric principles for optimal cross-layer resilient control systems." IEEE Control Systems Magazine 35.1 (2015): 110-127.

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is easy to follow. Algorithm some statements made in the paper are lack of rigorousness (explain in the previous section), the statements are clearly presented.

Quality: The problem formulation has some flaws as I mentioned in the previous section. I cannot discuss the quality of the algorithms and the experimental results before we address the problems in the formulation.

Novelty: Studying adversarial attacks with certain level of stealthiness is novel and interesting. 

Reproducibility: I did not check the code and the experiment setup. I cannot provide judgement. I did not check the math thoroughly.
summary_of_the_review: The paper studies a novel and interesting problem of adversarial attacks on state observations in RL. The adversarial attacks are crafted in a way to avoid being detected. But the problem formulation has flaws. For example, the conditions posed for statistically indistinguishability, based on the reviewer's understanding, are very strong. The reviewer hopes to see some examples to support the definition. The assumption is not unrealistic itself. But it is unrealistic that the victim will use the algorithm discussed in the paper.  The detector discusses in the paper is not a detector one usually finds in cyber-security. It does not consider statistical evidence. There are other concerns regarding the detection mechanisms discussed in previous sections.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Summary:
This paper emphasizes developing statistically undetectable attacks for autonomous decision-making agents. The authors develop a novel class of illusory attacks that are consistent with environment dynamics. Their results show that illusory attacks can easily fool humans, unlike the previous attacks in the literature. They compare illusory attacks with other attacks and show their performance under different defense techniques.


Questions and comments:
1. In Table 2, what is the perturbation budget used for the attacks?
2. Based on Section 3, I think in definition 4.1 and the rest of the text, it should be P_{\varepsilon, \pi} and not P_{\pi, \varepsilon} for consistency.
3. In Section 3, \gamma is not defined.
4. In definition 4.3, it says the highest expected return but shows minimization in equation (2). Is this correct? If yes, why? Also, why is "optimal illusory attack" defined? I do not see it being used in any part of the text.
5. I would suggest using \tau \sim P_{\varepsilon, \pi} instead of \tau \sim (\varepsilon, \pi}) in equation (3).
6. In equations (3, 7), should the summand not be till T-1 and not T?
7. In equation (7), it should be \lambda instead of \gamma, right?
8. On page 8, it should be Table 1 and not Table 5.3 for consistency.
9. I think it's a better idea to replace Table 1 with Table 7.5.1 in the main text since the absolute reward values are easier to compare.
strength_and_weaknesses: Strengths:
1. Focuses on developing attacks that are consistent with the dynamics of the environment.
2. Develops a novel class of illusory attacks that are consistent with the dynamics. Introduces the concept of statistical indistinguishability for stochastic control processes.
3. Well-structured sections. The supplementary material contains all the relevant videos related to human study.

Weaknesses:
1. I think that the human study setup is biased. The samples given in the supplementary show that it's easy to detect attacked vs. unattacked videos. All the unattacked videos labeled in the study balance the pendulum or cart pole perfectly, while all the attacked videos do not do that. Hence, it should be easy to detect attacked vs. unattacked videos for these simple environments.
2. I think illusory vs. unattacked videos can also be figured out without much difficulty. For example, only in illusory attacks the cart pole moves very quickly. In the rest of the attacks/ unattacked videos, the cart poles do not move as fast as in illusory attack videos.
3. Given the above weaknesses, I am not sure why the detection accuracies are low for the illusory attacks in Table 1. What is the naive detection algorithm that is used for Table 1? Also, W-illusory attacks are not "statistically indistinguishable", unlike perfect illusory attacks, right?
4. I am not convinced if good W-illusory attacks should always exist. Pendulum and cart pole are very simple environments. I would suggest adding experiments with more complex environments. Not sure about transferability to other tasks.
5. Reality feedback is a practical and obvious way to defend against illusory attacks. Hence, the attack proposed in this paper does not appear to be strong.

Response to rebuttal
================

Thanks for the rebuttal; however, some of my concerns remain. Could you please try to address these?

1. I am not convinced about illusory attacks not being detectable to "attentive human supervisors". I think a non-expert in RL can figure out attacked vs. unattacked in cases of illusory attacks. Quoting from my review -- "For example, only in illusory attacks the cart pole moves very quickly. In the rest of the attacks/ unattacked videos, the cart poles do not move as fast as in illusory attack videos.". I think  this requires a closer investigation.

2. Unanswered question from my review -- "Also, why is "optimal illusory attack" defined? I do not see it being used in any part of the text."

Edit:
After the discussion phase, I have decided to increase my scores since the rebuttal cleared most of my questions. 

clarity,_quality,_novelty_and_reproducibility: The main novelty of the paper is to propose an attack algorithm for decision-making agents that is consistent with the unperturbed environment dynamics. The authors try to motivate why statistically undetectable attacks are important. I feel that the writing of the paper can be improved to make the work clearer. I have noted my comments in the first section. I think more evaluations on complex environments are required since the paper focuses on an empirical attack algorithm.
summary_of_the_review: As mentioned in the weaknesses, I am not convinced if W-illusory attacks are undetectable. I think evaluations in more complex environments are required to strengthen the paper. The proposed attack is not robust to reality feedback. Hence, I do not feel that the contributions in this paper are very significant.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies test-time attacks on reinforcement learning agents. It focuses on attacks that are statistically undetectable, and proposes novel attack models that aim to preserve consistency of trajectories with the environment dynamics. The paper develops a new optimization framework for generating such attacks and experimentally validates the effectiveness of these attacks, called illusory attacks. The experiments test the efficacy of the proposed approach in terms of: i) detectability of adversarial attacks via statistical consistency checks, ii) detectability of adversarial attacks via visual inspection (i.e., human-subject studies), and iii) susceptibility of robustly trained RL agents to adversarial attacks. The experimental results indicate that the proposed attack approach yields lower detectability rates compared to prior works. 
strength_and_weaknesses: **Strengths of the paper**:
- To my knowledge, the attack models considered in this work has not been studied in the literature on test-time attacks against RL agents. The attack models are well motivated and complement those from prior work. Instead of focusing on $L_p$ norm-based attack models, the paper advocates models that are statistically undetectable. 
- The paper introduces a formal framework for studying these attack models, as well as an optimization problem for finding an optimal illusory attacks. The optimization problem aims to minimized the victim's return, while minimizing a distance function that measures the inconsistency between generated trajectories and the environment dynamics. 
- The paper also conducts a human-subject experiment to test the detectability of this attack model via visual inspection. This validation techniques appears to be novel when it comes to adversarial attacks on RL agents.  

---
**Weaknesses of the paper**: 
- The experiments are primarily based on two simple environments, Pendulum and CartPole. In contrast, prior work, e.g., Zhang et al. 2021, has studies more complex environments, such as MuJoCo. More experiments would be useful in order to understand the scalability of the approach. Also, given that one the experiments involves human-subjects, IRB may be required; the paper doesn't seem to report if the study has an IRB approval.  
- In general, it is not clear what are the computational properties of the proposed optimization framework. The I-MDP model does not scale well with the time horizon, and given that the experiments are only based on two simple environments, it is not clear how practical this approach is. Additionally, the attack optimization problem (7) seems to require the environment/world model. Some discussion on the practicality of the approach would be useful to have. 
- Some parts of the formal framework are not entirely clear and may contain typos. Firstly, I don't fully understand why rewards are not included in trajectory \tau when one measure consistency with the true environment, nor why \tilde S does not include rewards. It's not immediately clear to me that the victim cannot detect this attack by inspecting received rewards. Some discussion on this would be useful. Secondly, definition 4.1 uses $\mathcal E '$ but does not specify it. Thirdly, Eq. (3), (5), (6) and (7) may not be precise. E.g., why do we minimize $-\lambda D()$ in (3)? Moreover, transition probabilities $p(.)$ should be functions of $s$, not $\tilde s$, but in (5) we have $p(\tilde s_{t+1}|\tilde s_{t}, a_{t})$. Similarly, I don't understand how we obtain (6) since rewards function $r$ should take elements from $S$ not $\tilde S$; it is also not clear why this equality holds as we have $\tilde r_{t+1}$ on LHS and $r_{t+1}$ on RHS. In (7), why do we have $r_{t}$ and not $\tilde r_{t}$ and why does $\pi_a$ depend only on $a_{t-1}$ and $\pi_v(a_{t-1})$ but not the whole history? The notation also seems to be ambiguous. E.g., is $a_t$ the action of the attacker or the victim, and does $\pi_v(a_{t-1})$ denote stochastic or deterministic policy?   
clarity,_quality,_novelty_and_reproducibility: Please see my detailed comments above. Below I summarize some of the points related to quality, clarity and originality. 

- Quality: I believe that the attack model studied in this work is interesting, but given that the evaluation is primarily based on experiments, more environments could be added to the experiments test-bed. Regarding the reproducibility, the simulation-based results are well documented. On the other hand, it would be good to extend the description of the human-subject experiment, e.g., by adding the recruitment protocol, and indicate whether the study had received an IRB approval.   

- Clarity: The paper is overall clearly written. That said, some part of the paper are not entirely clear as I indicated above. If these are typos, unfortunately, there seem to be quite a few of them and they significantly impact the correctness of the results...

- Originality: The attack model studies in the paper seems quite novel. I also like the fact that the paper utilizes human-subject experiment to test the detectability of some of adversarial attacks, which is rather novel, or at least not that common in this line of work.  
summary_of_the_review: Overall, I enjoyed reading the paper. The core idea seem quite intriguing and novel. Having said that, I believe that the paper could benefit from having more experiments (i.e., additional environments), and that it could better explain the limitations of this work and the practicality of the proposed approach. Apart from that, some parts of the paper could be polished.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies adversarial attacks on sequential decision-making policies, with a focus on *statistically undetectable* attacks. The authors assume exact knowledge of the world model, and introduce a novel class of adversarial attacks called illusory attacks, which are consistant with the world dynamics and thus more stealthy. This paper formulates the illusory attacks, and propose a feasible learning algorithm, W-illusory attacks, to generate illusory attacks. Experiments on simple control tasks show that the proposed attack and less detectable to humans and AI agents than state-of-the-art attacks.
strength_and_weaknesses: Strengths:

1. The idea of statistically consistant attacks is interesting, and to my knowledge, novel in the literature.
2. The paper provides theoretical justifications that perfect illusory attacks exist for some but not all policy-environment pairs. 
3. Provided human study are useful for understanding the detectability of adversarial attacks.
4. The proposed algorithm makes intuitive sense, and the empirical results on Cartpole and Pendulum do show the effectiveness of the algorithm.

Weaknesses:

1. I do not agree with some claims made by the paper. In particular, most existing adversarial attacks are imperceptible to humans [1,2], which is an important motivation of adversarial attacks. Most literature of adversarial RL studies the adversarial perturbations on observation, which lies in a high-dimensional space (e.g. images [2,3,4]). These perturbations are usually undetectable by humans, unless special care is taken.
2. The assumption of an exact world model is not very realistic, especially in the senarions where adversarial attacks are concerning. In simulators, we may have an exact world model. However, it is less possible and also less dangerous that a local simulator is attacked. Adversarial examples are more critical during the interaction with real-world environments where observation may be noisy and exposed to outside attackers. Under these real-world scenarios, the access to a world model is unrealistic. Even learning a good model can be challenging in these environments.
3. The experiments are on simple environments. Can the authors also provide results on larger scale environments like Atari games, or at least MuJoCo environments?
4. Some related works are missing. For example, [4] proposes a stronger adversarial attack that SA-MDP [3]. Can the authors compare the proposed attack with [4]?



[1] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. "Explaining and harnessing adversarial examples."

[2] Huang, Sandy, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. "Adversarial attacks on neural network policies."

[3] Zhang, Huan, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. "Robust deep reinforcement learning against adversarial perturbations on state observations.

[4] Sun, Yanchao, Ruijie Zheng, Yongyuan Liang, and Furong Huang. "Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl."
clarity,_quality,_novelty_and_reproducibility: This paper is clearly written. The problem setup is relatively novel. The code and hyperparameter settings are provided. 
summary_of_the_review: The formulation and methods proposed by the paper are interesting. But I do have some concerns on the significance of the problem, as well as the realisticity of the key assumptions. In short, it is not clear to me whether the illusory attack and the world model assumption is practical in real-world scenarios and high-dimensional environments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies adversarial attacks on the state observation (sensor inputs) channel of an RL agent. Reviewers generally liked the problem studied in the paper of adversarial attacks on state observations in RL. However, they were not convinced by some of the arguments of the work regarding illusory attacks not being detectable to "attentive human supervisors". One reviewer argues the problem formulation has flaws as the conditions posed for statistically indistinguishability are very strong. I think the paper could benefit from having more experiments (i.e., additional environments), and that it could better explain the limitations of this work and the practicality of the proposed approach. Given all, I think the paper needs a bit more work before being accepted. 
justification_for_why_not_higher_score: See above for weaknesses. 
justification_for_why_not_lower_score: I recommend rejection.
Decision: Reject