Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper investigates adversarial attacks on learned (fixed) policies. In particular, they devise optimal attacks in the sense that e.g. the agent’s objective should be minimised by the attacker. It is assumed that the attacker can manipulate the observed state of the agent at each time step during testing in a restricted way such that perturbations are small in the state or feature space. The paper derives attacker value functions when the attacker’s goal ist to minimise the original agent’s average cumulative reward and shows how to maximise them using gradient methods. In order to show when attacks are more likely to be successful, a theorem is presented and proved that shows that the resilience of a policy against attacks depends on its smoothness. Experiments show that a) attacks that are trained for optimality w.r.t. to minimising the average reward of the agent outperform a baseline method that only optimises a related criterion. b) Agent policies that are trained with DRQN, yielding a recurrent controller, outperform non recursive ones. 

Overall, I think this paper can be a good contribution to the subfield of adversarial attacks on MDP policies. In my view, the derivations and overall structure and results of the paper are sound. However, I would have liked to get a better understanding and motivation for the investigated problem setting, such as projected applications and state or features spaces that could be manipulated in the proposed way.

Pros:
Paper is well written and generally easy to follow.
The derivations look sound and are well motivated. Proposition 5.1. can help in understanding how to train resilient agent policies. 
Experiments confirm that optimising for optimal attacks does indeed find empirically better attack strategies. 


Cons: 
No real application was presented. While applications are imaginable, a real application would have been beneficial. In particular, getting an idea of how real pertubations may look like in a realistic domain. The same holds for the assumption that the state (or features) can be manipulated at each time step, but only slightly. 
The implications of Proposition 5.1. were never tested, as far as I understand. You only test DRQN vs DQN, which validates your POMDP assertions, but other comparisons are missing.
The tested Open AI Gym environments are very basic, with the exception of Pong. This reiterates my application argument.

Notation/Writing: 
On page 5, last paragraph you should use a different letter for J in order to reduce notational confusion.  
Section 6 is not on the same standard as the other sections in terms of writing, e.g. 'We have obtained from 3 policies‘ , some stray ’the’, some plural vs. singular issues. 

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary:

The authors of this paper propose a novel adversarial attack for deep reinforcement learning. Different from the classical attacks, e.g., FGSM, they explicitly minimize the reward collect by the agent in a form of Markov decision process. Experiment results demonstrate that the proposed approach can damage the well-performed policy with a much bigger performance drop than gradient-based attacks.

Paper strength:

1.	The paper is well-organized and easy to follow.
2.	Model the adversary of reinforcement learning (RL) system as another MDP and solve it with RL is novel and interesting. The proposed attacking diagram can be devised in a pure black-box setting and also can be incorporated with white-box attacks. 
3.	With such a strong attack, the authors derive an upper bound on the impact of attacks and shed light on new research studying the robustness of deep RL approach.

Paper weakness:
1.	The author should give more details about how you use a gradient-based exploration to guide the adversary. From my point of view, I think the black-box attack is more practical and interesting. I would like to see the clearer comparison of optimal attack in a pure black-box setting with gradient-based attacks.
2.	Though FGM is not as efficient as the proposed optimal attack, they are simpler than a learning-based approach. Please describe the details and cost of training the attack agency, e.g., the hyper-parameter, number of training iterations.
3.	While the conclusion of smooth policies is more resilience for adversaries is interesting, I would like to see the evaluation results of such a novel finding. 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper investigates the design of adversarial policies (where the action of the adversarial agent corresponds to a perturbation in the state perceived by the primary agent). In particular it focuses on the problem of learning so-called optimal adversarial policies, using reinforcement learning.

I am perplexed by this paper for a few reasons:
1)	What is the real motivation for this work?  The intro argues “casting optimal attacks is crucial when assessing the robustness of RL policies, since ideally, the agent should learn and apply policies that resist *any* possible attack”.   If the goal is to have agents that are robust to *any* attacks, then they cannot be robust just to so-called optimal attacks.  And so what is really the use of learning so-called optimal attacks?
2)	The notion itself of “optimal” attack is not clear.  The paper does not properly discuss this.  It quickly proposes one possible definition (p.4): “the adversary wishes to minimize the agent’s average cumulative reward”.  This is indeed an interesting setting, and happens to have been studied extensively in game-theoretic multi-agent systems, but the paper does not make much connection with that literature (apart from a brief mention at bottom of p.2 / top of p.3), so it’s not clear what is new here compared to this.   It’s also not discussed whether it would ever be worthwhile considering other notions of optimality for the adversary, and what would be the properties of those.

So overall, while I find the general area of this work to be potentially interesting, the current framing is not well motivated enough, and not sufficiently differentiated from other work in robust MDPs and multi-agent RL to make a strong contribution yet.

More minor comments:
-	P.3: “very different setting where the adversary has a direct impact on the system” => Clarify what are the implications of this in terms of framework, theory, algorithm.
-	P.4: You assume a valid Euclidean distance for the perturbed state.  Is this valid in most MDP benchmarks?  How is this implemented for the domains in the experiments?  What is the action space considered? Do you always assume a continuous action space for the attacker?
-	P.5: “we can simply not maintain distributions over actions” -> Why not?  Given the definition of perturbation, this seems feasible.
-	P.5:  Eqn 4 is defined for a very specific adversarial reward function. Did you consider others?  Is the gradient always easy to derive?
-	P.6: Eqn (5) & (6): What is “R” here?
-	P.7: Figure 1, top right plot. Seems here that the loss is above 0 for small \epsilon.  Is this surprising?  Actually improving the policy?
-	P.7: What happens if you consider even greater \epsilon?  I assume the loss is greater.  But then the perturbation would be more detectable?  How do you think about balancing those 2 requirements of adversarial attacks?  How should we formalize detectability in this setting?
-	Fig.2: Bottom plots are too small to read.
-	Sec.6:  Can you compare to multi-agent baselines, e.g. Morimoto & Doya 2005.
-	P.8: “We also show that Lipschitz policies have desirable robustness properties.” Can you be more specific about where this is shown formally?  Or are you extrapolating from the fact that discrete mountain car suffers more loss than continuous mountain car?  I would suggest making that claim more carefully.


review_title: Official Blind Review #1

Meta-review: This paper studies the problem of devising optimal attacks in deep RL to minimize the main agent average reward. In the white-box attack setting, optimal attacks amounts to solving a Markov Decision Process, while in black-box attacks, optimal attacks can be trained using RL techniques. Empirical efficiency of the attacks was demonstrated. It has valuable contributions on studying the adversarial robustness on deep RL. However, the current motivation and setup needs to be made clearer, and so is not being accepted at this time. We hope for these comments to help improve a future version.
Decision: Reject