Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes to learn the ‘key-steps’ at which to to apply an adversarial attack on a reinforcement learning agent. The framing of this problem is a Lagrangian relaxation of a constrained minimization problem which takes the form of an RL problem itself, where the attacking agent’s reward is the negative reward of the target agents plus a penalty (lambda, hyper-parameter) for choosing to attack. The attacking agent’s action space is binary, attack or no attack.

The RL approach is compared with a random attack policy and two heuristic methods for attacking agents in games on the Atari benchmark.

The setting addressed in this work, where the attacker only learns whether/when to attack or not is a greatly simplified version of the full problem. It is (in my opinion, but feel free to correct me) likely that the type of perturbation also being learned has even greater potential, but is (obviously) much harder.

Additionally, although the authors mention this as future work, I think the co-training setting is particularly interesting. As the authors suggest, this could lead to more robust target agents, but additionally I wonder if the type and difficulty of attacks would vary as the target agent trains.

Besides the (picky) complaints, I think the problem formulation is quite reasonable. Again, I find the larger problem extremely interesting, but perhaps this is far too intractable right now. The formulation is straightforward, so while I recognize it as a contribution, it is not a particularly large one.

On the other hand, the experimental results appear somewhat weak to me. 

If you vary the lambda parameter (as in Table 1, but for all games) you should be able to get similar line plots in Figure 5&6 for the RL approach, which would give a much better comparison for the trade-offs as you get for the heuristic methods. I think this comparison would be very interesting and strengthen the existing results in those figures.

The results, currently, do not appear very significant because (1) the gap between the RL solution and the heuristics is very small and (2) these *appear* to be single runs without standard deviations displayed. Can you argue for why these results *are* in fact significant (statistically or otherwise)?

Another question raised by the results is how performance of the attacking policy varies with training. The authors point out that the training is quite small compared to the target policy, but is that because it has already found the best solution it can in that time? How would it improve with more training?

Small note on Figure 7, I think the point for these would be better made by normalizing the respective histograms.

Update:

Thank you for your responses and updating with new results. I think these provide a much better picture of the performance of the RL-based system. Although I am revising my score upward, I still think this is generally a rejection. Obviously I still think the full problem is interesting, but even the key step identification problem would be publishable if the performance was improved or further analysis helped me understand why RL is not doing better (since the heuristics are after all just heuristics). Good luck on future versions of this work.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: 
Learning Key Steps to Attack Deep Reinforcement Learning Agents

This paper proposes an extension of existing discrete action image space adversarial attack algorithms.
Instead of choosing the steps by heuristic, the authors propose to choose the key steps by augmenting the reward function with a penalty to decrease the ratio of attacks.

I tend to vote rejection for this paper, given that the proposed algorithms seem incremental compared to the existing algorithms, and the experiments seem not sufficient enough to support the core claim proposed in the paper.

Pros:
- The paper is well written, with sufficient background and related work section for the paper to be self-contained.
- The proposed framework is an interesting and practical framework for attacking RL agents.

Cons:
- The experiment section is insufficient.
More specifically:
1) Results from Figure 5 and Figure 6 seem to disagree with what the authors claim in the paper. In many (the majority) of the environments, the proposed algorithm has only trivial improvement and even worse performance under the same attack rate.

2) It is not very convincing when only one result sample is plotted in Figure 5 and Figure 6.
I think it is necessary to show the performance of the proposed algorithm under different attack rate.
A wide range of candidate penalty parameter lambda should be tested, so that a curve can be fitted for the proposed algorithm similar to the baselines (similar to the one shown in Table 1, but with much more test values).

3) Related to 2), it seems the Lagrange relaxation makes it hard to control the attack rate in the proposed algorithms. How sensitive it is to control the attack rate?

3) Can the authors elaborate on why the algorithms is not too sensitive to the value of penalty in section 4.5?
Table 1, where the performance is almost the same for different penalty parameter, does not necessarily show that the algorithms is not too sensitive to the choice of the penalty parameter.
As mentioned by the author, -21 is the minimum reward (or random reward) an agent can get from Pong.


In general, given the current status of the paper, where there is a lot of room for improvement of experiment section, I will vote for a rejection.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Contribution:

This work presents a method for performing budgeted attacks on a RL agent where an attacker can apply a perturbation to a limited subset of the observations of the said agent.
To select the "key steps" that needs to be corrupted, they train an opponent RL agent, with a reward crafted such that it should minimize the number of attacks as well as the cumulated reward of the target agent.
Their attack works either in a white-box setting, using FGSM, or in a black-box setting, using a substitute model.

They then proceed to compare to to prior work that use heuristics based on the policy values to select the steps to attack.

The paper is overall well written and easy to follow.

Review:


One major limitation of the work is that the attack rate is not readily modifiable. It uses a penalty term in the reward function with a tunable weight $\lambda$, but changing this weight seemingly requires retraining the opponent from scratch, which is unpractical. By constrast, note that the previous attack methods allow to change the attack rate at will.
More importantly, there is no clear relationship between the value of lambda and the attack rate. With the same lambda, depending on the game, the opponent settles on attack rates varying between 10% (on pong) and 70% (on Riverraid).


The results themselves show only marginal improvement over the baselines, and in the absence of clear error bars / confidence intervals, it is difficult to state the significance of the method. In the particular case of Space Invaders (figure 5b), the proposed method seems tied to the best heuristic (at attack-rate 70%), but the said heuristic reaches the same performance level for much lower attack-rate (as low as 40-50%), implying that the presented method did not do a good job at minimizing the number of attacks.
Overall, a rigorous way of comparing the methods need to be devised. Maybe something like an AUC with proper confidence bounds could do the trick?


In equation (2), you present a target objective function designed using Lagrange relaxation. However, the RL algorithm uses decay (\gamma = 0.99), which means that the resulting function that is effectively minimized is different. Could you clarify the impact of the decay on the lagrange-relaxed objective function?


Could you clarify a bit the section 4.3 on black-box attacks? It seems that you are using a substitute model attack, but it's not clear to me how the substitute is obtained. Is it the same model? How is it trained? Is the attack robust to differences in the algorithm?

Finally, I'm a bit surprised by the choice of DQN as the base algorithm, especially since the chosen framework (Dopamine), offers significantly stronger algorithms (Rainbow or IQN). DQN doesn't even reach perfect score on Pong, which means that the raw policy itself is a bit brittle, since it looses 5 points. Did you try to apply this method on more robust policies?
review_title: Official Blind Review #3

Meta-review: This paper considers adversarial attacks in deep reinforcement learning, and specifically focuses on the problem of identifying key steps to attack. The paper poses learning these key steps as an RL problem with a cost for the attacker choosing to attack.

The reviewers agreed that this was an interesting problem setup, and the ability to learn these attacks without heuristics is promising. The main concern, which was felt was not adequately addressed in the rebuttals, was that the results need to be more than just competitive with heuristic approaches.

The fact that the attack ratio cannot be reliably changed, even with varying $\lambda$ still presents a major hurdle in the evaluation of the proposed method.

For the aforementioned reasons, I recommend rejecting this paper.
Decision: Reject