Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors start with the valid observation that for controlled systems the threat model of explicitly flipping pixels or otherwise changing the inputs directly is less relevant than the case of identifying bad inputs in the environments (which the authors rename "natural observations") that cause the controller under attack to leave its stable region and end up in bad states for itself, unable to recover sometimes. This paper is an empirical exploration of this phenomenon, using as examples trained agents in a Gym environment and involving two humanoid robot simulations interacting with each other in various ways. The original policies are taken from previous training, so the main experimental contribution here is to solve an RL problem on the attacker's side to try to find the bad regions in the victim's control space, going via the means of the observation-based coupling.

I think these are timely issues. I agree that, to the extent that DRL policies are being seriously considered as candidates for various practical problems, we should seriously ask questions about the (lack of) stability of solutions. 

I also think that the paper would be stronger if the authors considered the rich history in control design, formal methods and other communities where this approach to counter-example based thinking is fairly standard and in particular where the use of optimization methods to find 'bugs' has been studied in some depth. 

So, for instance, the authors give us useful quantification to support the observation that the process of attacker RL finds ways to put the victim outside its stable region where it is naturally unstable and ineffective. However, they do not observe that there is a literature on systematically performing 'controller testing' via optimization using various techniques. Here are just a few examples to show the diversity:
Ghosh, S., Berkenkamp, F., Ranade, G., Qadeer, S., & Kapoor, A. (2018, May). Verifying controllers against adversarial examples with Bayesian optimization. In 2018 IEEE International Conference on Robotics and Automation (ICRA) (pp. 7306-7313). IEEE.
Ravanbakhsh, H., & Sankaranarayanan, S. (2016, October). Robust controller synthesis of switched systems using counterexample guided framework. In 2016 international conference on embedded software (EMSOFT) (pp. 1-10). IEEE.

The authors might find it interesting to note that this approach of viewing the control problem 'backwards' to generate new instances has a deeper history in control theory (of course without discussion of NNs etc), e.g., 
Doyle, J., Primbs, J. A., Shapiro, B., & Nevistic, V. (1996, December). Nonlinear games: examples and counterexamples. In Proceedings of 35th IEEE Conference on Decision and Control (Vol. 4, pp. 3915-3920). IEEE.

All that said, I find the results presented here to be plausible and pointing in the desired direction for exploring how to robustify DRL. The authors defer training using these examples to future work but I think the paper would be more self-contained if that were actually demonstrated here already. For reasons I mention above, the existence of this kind of weakness in controllers is not really surprising to people who have thought deeply about control, and it is only to be expected that NN based parameterisations of control would only be even more vulnerable. So, the more satisfying result would be the positive one that shows how to train DRL to be (more) robust under such attacks.


review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Thank the authors for the response. The update looks good to me. The discussion section looks much better now.
----------------------------------------
Summary
This paper conducts research on adversarial policy against a fixed and black-box policy (victim). In this setting, the victim has a fixed policy but the adversary has no access to its white-box information. Instead, the adversary can freely access the black-box policy of the victim. The experiments show that the trained adversary outperforms the baseline in some scenarios, with three interesting findings: 1) the adversary successfully found the weakness of the victim. In some scenarios, the adversary wins by doing some weird actions which make no sense, but those weird actions somehow make the victim fail; 2) the victim fails due to the weird (or I should say adversarial) observation from the adversary but not the physical inference. The authors demonstrate this by showing that the victim has a much higher success rate when the observation is replaced with a ‘normal’ opponent. I am kind of on the borderline, but still lean to accept paper.
Strengths
- This paper presents some experiments on adversarial learning when the policy of the victim is fixed and black-box with interesting findings, demonstrates that the adversary can successfully figure out the weakness of the victim.
- The paper formalizes the problem into an MDP whose dynamics is unknown, which is clear.
- This paper comes with many experiments supported by great demos, which clearly support the authors’ arguments.
Weaknesses
- The paper becomes messy in the end and does not come with a good conclusion. With a lot of experiments conducted, the author fails to summarize them into an (or a few) interesting conclusion(s), but end up with a page-long conclusion paragraph, which makes the paper less focused but like listing miscellaneous experiments. From the reader’s perspective, a (or a few) clear conclusion would be very helpful.
Possible Improvements
My suggestion is to reorganize the second half of the paper, to make a few clear arguments. Currently, the paper looks pretty narrative, and the author might easily get lost while reading the paper.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Adversarial Policies: Attacking Deep Reinforcement Learning

In this paper, the authors tackle the problem of adversarial attack by controlling the adversarial policy in reinforcement learning.
I tend to vote for acceptance for this paper, but I also want to point out there is a lot of room for a better experiment section.

Pros: 
- In general, the proposed attacking scenarios is novel and missing from the current adversarial attack research, which mostly attack by adding constraint noise in the image space, instead of directly train a new adversarial controlling policy.

- The problem is well formulated and the experiments are also sufficient to support the claim (could be better though, see Cons below).
The visualization provides useful insight for us to deepen understanding of the policy attack.


Cons:

- Experiments limited in sumo cases.
The sumo experiment is quite difficult to train by itself and does not cover the board range of difficulty in multi-agent reinforcement learning tasks.
To be more specific, it will help the generalization of the paper by including
1) Low-dimensional multi-agent environments. It will be even better if the input space is image and the output space is discrete.
2) The study does not consider cooperative multi-agent games. It is also a very important and natural extension to the paper to consider the malicious attack in cooperative games, which can be equally common in real life.

- No solution is given to the problem to combat policy attack.
Some of the results of easy baselines such as iteratively playing attack and defense should be included in the current version of the paper.
It doesn’t have to be successful, but should be quite necessary to increase the research value of the paper.

review_title: Official Blind Review #1

Meta-review: This paper demonstrates that for deep RL problems one can construct adversarial examples where the examples don't really need to be even better than the best opponent. Surprisingly, sometimes, the adversarial opponent is less capable than normal opponents which the victim plays successfully against, yet they can disrupt the policies. The authors present a physically realistic threat model and demonstrate that  adversarial policies can exist in this threat
model.

The reviewers agree with this paper presents results (proof of concept) that is "timely" and the RL community will benefit from this result. Based on reviewers comment, I recommend to accept this paper. 
Decision: Accept (Poster)