Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors studied how perturbations on states would affect the performance of deep reinforcement learning. They defined different types of perturbations, like different perturbations for each state, or apply the perturbations on the initial state on every state. The authors tested these perturbations in some existing environments.
- The conclusions of this submission are unclear and questionable. The authors showed many results in their submission, but all the conclusions are plausible. The results look pretty random, and I do not believe we can draw conclusions from them. For example, in table 2, the impacts of A_M^random are large, even comparable with A^individual. The A_M^random is more like a fixed random noise, so we cannot draw conclusions about transferability from it. It would be helpful to add two new baselines, one is fixed random noise, another is iid random noise, to demonstrate these adversaries are different from random noise.
- Equations 1 and 2 do not make sense. For the equations wrote by the authors, J(s) is a constant, and we should always let s_adv equal to s.

--Post Rebuttal--
Thank the authors for the response. I agree with other reviewers that the task is interesting and the submission has great potential, but might need another round of editing. The results show "a single perturbation from a random state of a completely different MDP" is not normally distributed. However, I agree with R4 that applying a single perturbation from a random state of another MDP does not make sense unless the two MDP and the two states have some similarities.
review_title: Work about stabilities in deep reinforcement learning, with space for improvement

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary of Contributions

The paper explores adversarial perturbations in deep RL, providing a new thread model where the perturbation is computed based on a single state. The paper explores the impact of various types of perturbations between states in the same environment (state transferability), and between states in different environments (environment transferability).

### Review

I think the paper does a good job at capturing the extent/prevalence of the issue of adversarial perturbations. I liked the breadth in the types of perturbations considered, and how they map to scnarios that could happen in practice (e.g., restricted adversaries, etc.).

One thing that I think could improve the paper is teasing apart properties of environments or learning algorithms that are telling of the transferrability of the perturbations. For example, beyond suggesting that the offset pushes things beyond the decision boundary, checking things like how action gaps change may be insightful. Some of the action manipulation may further be exacerbated by things like using ε-greedy behavior policies which immediately snaps to highest-valued actions, in contrast with things like Boltzmann policies over action-values or policy gradient methods, which are smooth with respect to changes in them. Do the authors have any insight as to how the trends might change should a smooth policy be used?

While the focus is on deep RL, perhaps a simple, motivating example (e.g., a little gridworld or Markov chain) could make a stronger/clearer case as to what exactly is happening, and suggest what situations one might expect it to be a more prevalent issue.

Beyond this, I have the following questions/concerns:

1) Is there a reason for the choice of DDQN, over say, regular DQN? While one algorithm may perform a bit better, if the emphasis of the paper is to measure the extent of adversarial perturbations, it seems like it may paint a clearer/more convincing picture if a simpler algorithm is used, with fewer moving parts to attribute performance differences to. Along these lines, I think it would be more convincing to carry out the same experiments with another deep RL algorithm (perhaps a policy gradient one, to be representative of both value-based and policy-based methods) to see if comparable levels of and trends in transferrability are observed there. Can the authors comment on whether 

2) How were hyper-parameters chosen for the DDQN agent, and can the authors comment on whether such choices can have reasonably strong interplay with the transferability of perturbations? For example, would larger/smaller learning rates, or deeper vs wider networks be more or less resilient to such perturbations? I think in any case such details need to be included in the paper, to be clearer that the transferability quantified is in the scope of a particular instantiation of a specific algorithm.

Overall, I'm erring toward acceptance in that it outlines an interesting framework to study adversarial attacks in deep RL, and provides some early empirical results and intuitions around them. I do think the paper falls a little short in that there wasn't a representative sample of deep RL methods, as well as not commenting on design choices made and how/whether they might interplay with the transferability measured. I'm willing to adjust my score should my concerns be addressed.
review_title: Review

final_score: rating: 3: Clear rejection
review_content: Paper Summary:
This paper aims at discovering transferability of perturbations across different environments in RL. The authors propose some different types of advasaries and tested those adversaries on 6 atari games.

Review Summary:
The idea of analyzing perturbations' transferability is interesting, but it is hard to say that the approaches are satisfactory. The authors did not propose any novel algorithms or modifications based on existing algorithms, and provide only preliminary experiment results. As such, I suggest a clear rejection.

Detailed Comments:
Section 3.1, paragraph 2. "Such a model... of the deep reinforcement learning agent". I am hardly convinced by this statement and led by this statement, the proposed approach in this work. When we consider transferability in machine learning, we assume that there are something common to learn between the two domains. My first thought when I read the title is that it would be a transfer learning method for adversarial attacks. But the authors seem to prefer a universal offset on the raw pixel input. In this case, the only insights we might possibly learn is the input similarity between different Atari games, which is hardly a contribution to any field. Moreover, since adversarial attack is already used in this work, I don't see the reason why none of the referred methods for adversarial attacks in RL are tested.

Besides, all experiments are done only on DDQN, hence the claims are hardly validated. What I expect is either a new algorithm that is more resilient or a comparison on robustness between different RL algorithms. It is also not explained how the environments are chosen.

review_title: Interesting title but bad approach

final_score: rating: 4: Ok but not good enough - rejection
review_content: — idea:
A framework composed of 6 different adversaries is proposed using which it is claimed that the transferability properties among Atari environments can be studied.

— comments:

The paper is well-written and easy to follow. However, I have major concerns about the novelty of the material discussed in the paper. The only novelty is about using which states as adversarially perturbed input. It mostly looks like an incremental study. 
	
In the experiments section, authors have used a pre-trained DDQN agent which originally does not show significant generalizability compared to methods like A3C. It would be interesting to see the impact of the proposed framework on algorithms with better generalization capabilities. Thus, I believe the experiment section lacks this very important part.

In section 4.2, it is discussed which actions are switched to which ones due to adversary. For example, in the case of Pong, all of the actions are changed to 0, 2, or 3, which are “no op”, “right”, and “left”, respectively. I speculate that a better generalized policy (like A3C) instead of DDQN could still perform well since it has all the actions required to get the maximum return. Thus, the impact would not be this high (according to tables 1 and 2). In addition, in Pong, agent needs to press the “fire” button (action 1) to start each round of the game. Without loss of generality, this could be overwritten in an engineering matter (like what has been done in DeepMind Atari wrapper) so the agent still be able to perform without being concerned about the “fire” button. There is a lot to investigate and talk about in this paper.

— minor issues:
In figure 1, there are two CrazyClimber plots. One of them needs to be changed.

I believe the paper in its current form requires substantial changes. Due to the aforementioned reasons, I don't feel comfortable supporting this version of the paper. I think this paper could benefit from another round of editing pass. 

review_title: Needs another round of editing pass

Meta-review: The work studies the transferability of perturbations/adversarial attacks on DRL agents. As a way to mitigate the cost of generating individual perturbation for each state in each episode, the authors proposed several variants to use same perturbation across different states across different episodes. While reviewers recognize the potential of the direction, they are not comfortable accepting the paper at its current state. The experiment results in its current form does not provide enough support to the claim. In particular, it is unclear how much the shared perturbation changes the original perception in comparison to the individual comparison, and how should the impact number differences be interpreted. Reviewers brought up concerns regarding all the experiments being evaluated on a DDQN agent, and not enough clarities has been provided on the different design choices. If perceptual similarity is not the indicator of environment transferability, do the authors have intuition on what does? 
Decision: Reject