Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents an adversarial attack for perturbing
the actions or observations of an agent acting near-optimally
in an MDP so that the policy performs poorly.
I think understanding the sensitivity of a policy to
slight perturbations in the actions it takes or the
observations that it receives is important for having
robust learned policies and controllers.
This paper presents an empirical step in the direction
of showing that such attacks are possible, but in the context
of the other adversarial attacks that are possible, this is
not surprising alone and would be much stronger with
other contributions.
I think an exciting direction of new work could be to
continue formalizing these vulnerabilities and
looking at ways of adding robustness across many
other domains.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content:   *Synopsis*:
  This paper looks at a new framework for adversarial attacks on deep reinforcement learning agents under continuous action spaces. They propose a model based approach which adds noise to either the observation or actions of the agent to push the agent to predefined target states. They then report results against several model-free/unlearned baselines on MuJoCo tasks using a policy learned through D4PG.

  Main contributions:
  - Adversarial attacks for Deep RL in continuous action spaces.

  *Review*
  The paper is well written, and has some interesting discussion/insight into attacking deep RL agents in continuous actions spaces. I think the authors are headed in the right direction, but compared to prior work in adversarial attacks for deep RL agents (i.e. the Huang and Lin) I have a few concerns that I feel the authors need to better explain/motivate in their paper. I am recommending this paper be rejected based on the following concerns. I am willing to raise my score if some of these are addressed by the authors in subsequent revisions

  1. This algorithm requires the pre-trained policy to plan attacks (which may be a high bar for such an adversarial attack). It would be a nice addition to include similar results with "black-box" adversarial attacks, as mentioned in the Huang. 

  2. Another issue, addressed in the Lin paper, is this attack seems to require perturbation on every time step in a proposed trajectory. As mentioned by Lin, this is probably unrealistic and would cause the attacker to be detected. It would be another nice contribution to include variants that don't require perturbations on each transition.

  3. Another unfortunate requirement is a learned model (or a way to simulate trajectories). From the Model Based RL literature, we know learning such a model is quite difficult and often unrealistic given our current approaches. While this is problematic, I think the paper could systematically test this looking at what happens as the model becomes less accurate over time. This could provide some nice results showing an accurate model isn't necessarily needed and anneal concerns over having to learn such a model.

  4. It is unclear if the baselines measured against are meaningful in this setting, and I'm also a bit unclear how they are generated/implemented. Specifically, the random trajectories require you to return the generated trajectory with the smallest loss/reward. It is unclear how the adversary knows this information. Is it known through a model or some other simulation? Also the flip baseline could use a bit more explanation. I think these details can be safely placed in the appendix, but should appear somewhere in the final version.

  5. I'm not sure the comparison to sample efficiency to the Gleave or Uesato papers are meaningful. For Gleave, the threat model explored is much different where they do not have access to the agent's observation or action streams and instead learn policies to affect the other agent in game scenarios. This is very different. Also, the Uesato is not adversarially attacking the agent, but attempting to find failure cases for the agent, which I again feel is very different from what you are trying to accomplish. I would remove this discussion and the claim at the end of the conclusion.


  Other suggestions:

  S1. It would be helpful to include the score of the learned policy without any attacks, to see how well the baselines are performing (this will help readers understand if these are reasonable/meaningful baselines).

  S2. I'm unclear what figure three is adding to the paper, and am actually uncertain what the y-axis means. I don't think this is a wise use of the 9th page, and this plot could probably be relegated to the appendix.
  
  S3. As in prior work, it would be useful to see how well this line of attack works for multiple learning algorithms. Some potential candidates could be: PPO, TRPO, SAC, etc... 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary: This paper proposed a new adversarial attack method based on model-based RL. Unlike existing adversarial attack methods on deep RL, the authors first approximate the dynamics models and then generate the adversarial samples by minimizing the total distance of each state to the pre-defined target state (i.e. planning). Using Cartpole, Fish, Walker, and Humanoid, the authors showed that the proposed method can pool the agents more effectively. 

Detailed comments:

The proposed idea (i.e. designing an adversarial attack based on model-based RL) is interesting but it would be better if the authors can provide evaluations such as adversarial training and ablation studies for the proposed method (see the suggestion & question). So, I'd like to recommend "weak accept"

Suggestion & question:

Could the authors apply adversarial training based on the proposed methods? I wonder whether RL agents can be robust after adversarial training. 

Instead of utilizing a pre-defined target state $s_{target}$, we can also approximate a reward function and generate adversarial samples by minimizing the total rewards. It would be interesting if the authors can consider this case. 

Could the authors report an ablation study on the effects of T?
review_title: Official Blind Review #2

Meta-review: This paper considers adversarial attacks in continuous action model-based deep reinforcement learning. An optimisation-based approach is presented, and evaluated on Mujoco tasks.

There were two main concerns from the reviewers. The first was that the approach requires strong assumptions, but in the rebuttal some relaxations were demonstrated (e.g., not attacking every step). Additionally, there were issues raised with the choice of baselines, but in the discussion the reviewers did not agree on any other reasonable baselines to use.

This is a novel and interesting contribution nonetheless, which could open the field to much additional discussion, and so should be accepted.
Decision: Accept (Poster)