Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes using a better quantitative metric to conduct an attack on a DRL learner. The attack is limited to an attack on a state (not over multiple states) and aims to lower the Q value from this state by making the worst action be the action chosen to be played. The experiment on Atari games show promising results.

Pros:
- The idea is effective, finding the right attack objective is interesting and surprising that was not considered earlier.
- Good description of why prior methods do not achieve the optimal attack
- The experiments show good results

Cons:
- The techniques are not very novel, softmax (with temperature) as a soft differentiable version of argmax is very well known.
- A comparison to non-myopic attack would make paper stronger (https://arxiv.org/pdf/1907.09470.pdf)
- The legends in the figure are just too small to be readable
- Would have been good to show attacks on more complex problems.

Questions:
- This attack is for a particular state, which state is chosen for this attack? Is it towards the start of the game or end of the game?
- Why is E[Q(s; aw)] different for different approaches in Table 2, and also for E[Q(s; a*(s))]? These values should not depend on the attack.
- Aren't the variances too high in Table 1? Is 10 episodes enough - why not more?
review_title: The paper has a nice idea but a little incremental

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:  This paper proposes to build an adversary to find a bounded perturbation of the state that minimizes the value of the action taken by the reinforcement learning agent. This approach enables us to lower the bounds by several orders of magnitude on the perturbation needed to efficiently achieve significant impacts on DeepRL agents.

Strong aspects:  
1.     The approach is simple and straightforward with good numerical performance.
2.     The writing is easy to follow and experiments are thorough.

Weak aspects:
1.     It is strange that, for a state of thousands of dimensions, a tiny perturbation within a ball having a radius of $10^-10$, can make maximum possible deterioration in terms of return (i.e., Fig. 3). Can the authors provide more insights? For example, the sensitivity analysis of the return / Q value with regard to the states might help.

2.     How does the proposed approach compare to other more recent baselines such as Gleave et al. (2020)?

3.     Is the metric in (16) a common one? What if we simply plot the return under various perturbations, like that in Huang et al. (2017)?

Minor points:

- The locations and / or sizes of the legends in Fig. 2, 3 and 4 and the size of axis labels can be adjusted accordingly to make them easier to read.


I have read the authors' response and the associated discussions, and based on that  raised my evaluation by 1
review_title: Techniques presented could improve analysis of perturbations needed to efficiently achieve significant impacts on DeepRL agents, but the experimental results need further explanation

final_score: rating: 6: Marginally above acceptance threshold
review_content: The work introduces a new method for observation-perturbation adversarial attacks against deep neural network policies. The idealized version of the method is an optimal attack assuming (a) the agent's Q-values are calibrated; (b) the attacker can only act once, at the current time step. While the attack is in fact conducted at every time step, so is not "myopic", this assumption greatly simplifies the problem. Moreover, the work approximates the (deterministic, hard-max) policy with a low-temperature softmax policy to enable gradient-based methods to work. Evaluating in a range of Atari games, the method has a significantly larger "impact" (normalized return) than previous methods, across $\ell_1$, $\ell_2$ and $\ell_{\infty}$ norms. Moreover, the results suggest the attack succeeds because of increasing the probability of low-ranked actions, consistent with the attack's derivation.

Strengths:
  - Empirical results show a substantial improvement over a reasonable choice of baseline methods.
  - Experiments also support author's claims for why the method works (especially section 3.4 but also to a lesser extent section 3.5) -- important given the approximations needed from the derivation.
  - The paper gives a good intuition for why the method should work, and clearly highlights the limitations of previous methods.

Weaknesses:
  - The justification of the threat model could be improved. I appreciate the list of examples in paragraph 3 of the intro for where RL applications have been studied, but why is this threat model actually relevant to these example? For example, take financial trading: an attacker cannot perturb other peoples orders in the market data feed (unless they've compromised the exchange -- in which case there are much simpler routes to exploitation!), but can insert their own orders with almost arbitrary price/size. This does not fit an $\ell_p$ norm, which assumes modifying any field is equally possible. I would suggest picking a use case where this threat model does hold and going into detail on it -- or if there is none, then be more up front about this limitation.
  - Contrary to section 4 I am not convinced this method can be easily applied to continuous control tasks. First, policies for continuous control need not be stochastic. Typically, the policy outputs parameters of a distribution. At training one samples from this distribution -- but at inference one often takes the mode, deterministically, since it improves performance. Even if we allow for a stochastic policy, one has the new challenge of a much larger action space.

    My best guess is the method could be made to work, given that e.g. Pattanaik's method is not all that different and worked on MuJoCo. But I expect it to require some non-trivial tuning, and possibly new algorithmic insights. This section should really be supported by experiment, or have the claim reduced in scope to there being no *obvious theoretical hurdles* to its application.
  - Paper should be more up-front about assumptions being made. I did like the last paragraph of section 3 (before section 3.1). But you make other assumptions implicitly. For example, the agent's Q-values may be miscalibrated, such that the worst action for the agent is not actually the one the agent assigns lowest Q-value to! This will make your method suboptimal (even in the myopic case). Additionally, the fact you attack at each time step but choose the attack myopically must be leaving some value on the table. Indeed, the fact that expected Q-values often rise (table 2) under the attack is some evidence of this. I think the paper is a good submission without fixing these limitations (though it would be much stronger if they were addressed), but it's important they're clearly signposted so that they can be addressed in future work.

I consider this paper borderline but overall am leaning towards accept. There is a clear theoretical reason why the idealized method should perform better than previous methods (e.g. Huang, Pattaniak) and the empirical results support that the approximation is stronger than baselines. The threat model of this (and prior work) seems chosen more for mathematical simplicity than realism, but may help lay the ground for future work in more realistic settings. The paper could do a better job of communicating the assumptions and limitations, but assuming this is addressed then it seems worth disseminating to the ICLR community.

A few questions:
  - Do you have any additional insights into why the mean Q-value per episode often increases under attack? This is a very counterintuitive result -- why should choosing worse actions lead you to higher-valued states? Some things that might be worth looking into (though I certainly do not expect all of these during the relatively brief discussion period):
    + What return do you get if you perform the attack for 1-timestep (or a smaller number of timesteps) and then run the unattacked agent? This is what the Q-value is actually estimating -- but Q-values learned by deep RL are often very uncalibrated, especially off-distribution.
    + What happens in other environments? It strikes me that most Atari games are quite hard to get "stuck" in: even if you die you respawn. Some environments usually used to test safe exploration might be helpful with this.
    + What happens with attacks that are non-myopic, e.g. the "enchanting attack" of Lin et al (2017)?
  - Do you have any experiments in continuous control tasks that would back up section 4, even if preliminary?
  - Why are \ell_p norm perturbations of observations an important threat model to study? I know it is widely studied, but it is also widely criticized -- e.g. Gilmer et al (2018) -- and for RL in particular it seems rare for an attacker to be able to directly perturb observations. Are there cases where this attack is realistic? Are there non-security reasons to care, e.g. to improve robustness to natural "perturbations"? If so, can we validate this in a realistic setting, e.g. does vulnerability to your method also predict failures of sim2real transfer, or sensitivity to non-adversarial noise?
  - What do the perturbations actually look like? Do they look like anything reasonable, e.g. introducing a fake "ball"? Given you claim the perturbation direction identified by this method is likely stronger, it'd be interesting to know what neural networks find most "persuasive".

Detailed feedback:
  - Need to use parenthetical citations \citep in places rather than in-line citations \citet: e.g. first paragraph in the intro “speech recognition Hannun et al. (2014)” -> “speech recognition (Hannun et al, 2014)”; first paragraph of 3.1.
  - “perturbations to image”->”perturbations to images”.
  - Intro: focus it more on your method. It's a good summary of some of the history of adversarial examples – but is this that relevant? Most readers (and certainly your reviewers) will be familiar with adversarial examples. Adversarial examples in RL might need some handholding being less common, but can probably also be assumed. 
  - Related, could benefit from an explicit statement of what your threat model is early on (an adversary making bounded perturbations of the observations of an RL agent). I appreciate the list of diverse scenarios RL has been applied it, but I would suggest making this more focused on those that are actually deployed on nearing deployment (especially if safety critical).
  - Related Work: 2.1 is a nice succinct summary of existing work but it'd be helpful to the reader to place it in relation to your own work. In particular it seems your method is much more closely related to Huang, Kos & Song and Pattanaik et al than it is to Gleave et al or Pinto et al. It therefore might be useful to treat these as two groups (“multi-agent” adversarial RL v.s. “observation” adversarial RL, say); discuss why you chose to work under the observation threat model; and then dig into details on how your approach differs from Huang, Kos, Pattanaik (e.g. is it the same threat model but stronger empirical results? do you make stronger/weaker assumptions?).
  - “$\pi(s,a)$ is not the actual policy used by the agent” – bit confusing, can you change notation to make this explicit e.g. $\pi_{\text{soft}}$? Also good to be clear the difference is just soft vs hard-max (I assume), rather than e.g. $\pi$ being learned by behavioral cloning the policy under attack. Perhaps explicitly state earlier your attack is white-box?
  - I liked section 2.3: it was an easy to read summary that clearly explains the problems with existing work.
  - Section 3: your method is not really “optimal” since you approximate the hard-max policy with a soft-max policy of low temperature, so the section is a bit misleading. It might be better to frame it as: (1) definition of optimal attack; (2) tractable approximation to this. I do appreciate the discussion in the last paragraph of section 3 (before section 3.1) of why this is an approximation.
  - Figure 4: can you use the subcaption package to give each individual barchart its own label? This is easier than having to consult the main caption. It's also worth making explicit that these plots do not inculde the best action – I was wondering for a while where the rest of the probability mass was! I'd be inclined to even add the best action to the plot (perhaps visually distinguished in some way, and using a log-scale for probabilities if it would compress it too much).
  - Table 2: caption would benefit from being expanded.

Update after author response: The changes made have improved the clarity of the paper, such as making assumptions and the threat model more explicit, and the heatmap addition provides a nice qualitative insight. However, I am inclined to agree with other reviewers that the paper's contribution is incremental. Given this I am retaining my score of marginally above acceptance.
review_title: Strong empirical results, good intuitive justification, but threat model not justified and some assumptions/limitations unclear

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper proposes an optimal myopic adversary for deep reinforcement learning agent, in which the adversary finds a bounded perturbation of the state that minimizes the value of the action taken by the agent. The authors introduce a differentiable approximation for the optimal myopic adversarial formulation that leads to a better direction for the adversarial perturbation and increases the attack impact for bounded perturbations. Empirically, the authors show with experiments in various games in the Atari environment that the attack formulation achieves significantly larger impact as compared to the current state-of-the-art. 

The paper is easy to follow, and the topic investigated looks interesting. However, a few major issues unfortunately in the draft discourage me to accept the paper.

1. The novelty in the paper is marginal. The conclusions drew from this paper are completely based on the experimental results. Though the formulation looks interesting, in-depth analysis is missing in the paper. No major theoretical results have been reported to provide stronger support for the bounded myopic adversary.

2. In Section 3, the authors have known that Eq (14) and Eq (15) may not be equivalent to each other, they fail to provide more analysis and discussion on how to address it, instead, only relying on the empirical results. Also, how to choose a sufficiently small adversarial temperature constant is not clear, as suggested in the paper. It gives a sense that the authors only focused on the applicability of the formulation, while ignoring necessary theoretical justification.

3. No empirical results for the continuous action set. The authors have mentioned the decent applicability of the myopic adversary in the continuous tasks. They should also have shown some results to support their claim, as failing to do this makes the paper look incomplete. Unless the proposed is only devoted to the discrete tasks.

Minor point:

4. Regarding the experimental results, I wonder why the authors put Huang and Pattanaik together for Atari games. Though this works, it would be better to see both discrete and continuous tasks separately for these two different baselines. Also, the evaluation metric in the paper is Impact instead of returns, which is not popular in literature. I suggest the authors to include this in the appendix.

*********************************
After carefully considering the rebuttal from the authors, I am going to maintain the score based on my evaluation and also the current paper draft. Though the authors have tried to addressed the comments, the paper still requires more improvements, including theoretical novelty and experimental results.
review_title: Review of "Bounded myopic adversaries for deep reinforcement learning agents"

Meta-review: Most reviewers are positive about this work, though they believe it is somewhat incremental, and its theoretical contributions are minor. None of the reviewers are very excited about this work. Overall, the PC believes this is a borderline paper.

Minor note: During the discussions, the paper by Xiao et al., "Characterizing Attacks on Deep Reinforcement Learning" (2019) was brought up. The authors claimed that they did not compare with that paper because the best attack there (obs-fgsm-wb) had already been studied. In a later stage of discussions, one of the reviewers stated that the method obs-nn-wb in that paper performed better in some domains. Even though this is not a major issue, it is advisable to the authors to make sure that this is indeed the case, and if it is, provide proper comparison with that paper.

We encourage the authors to consider the reviewers' comments to improve the paper and resubmit to a future venue.
Decision: Reject