Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors presents how to learn optimal adversary following the state-adversarial Markov decision process (SA-MDP), and also proposes alternating training with learned attacks (ATLA) framework that trains the optimal adversary online together with the agent to improve the robustness of the DRL agent. Experiment result shows that ATLA outperforms the explicit regularization based methods. 

Overall I think the paper is well-written and clearly illustrated the methodology. The experimental results are mostly comprehensive. The contribution is clear. I still have a few questions and concerns below:

1) It is a natural and interesting idea to use alternating training to optimize both the adversary and the agent online. In terms of finding the (near) optimal adversary following the theory of SA-MDP, the authors argue that because of using "function approximator to learn the agent so it’s no longer optimal". However, it seems unclear how much the approximation affects the optimality. For tabular case instead of DRL, is it possible to really find the optimal adversary, and if so, how? How far away the learned strong adversary from the real optimal one? Indeed from experiments the learned adversary can better attack than other baselines, but it would still be important to understand the advantage and room to improve in a principled way. For finding the policy, it is understandable that because of the difficulty of solving POMDP, this paper does not solve the optimal policy. 

2) ATLA-PPO + SA Reg further improves performance over ATLA-PPO in experiments. This seems to suggest that the advantage of ATLA-PPO and SA Reg are complementary. Does the regularization on the function approximator provides additional robustness or it covers the error of function approximator using LSTM? Is it possible to understand this in experiments?

3) Are the adversary and the agent both getting stronger over time? The paper only showed final results and did not show the running time result. Hypothetically because of the alternating training, the adversary and the agent should both be improved and it would be interesting to verify this in experiments. 

 
review_title: An interesting framework for robust DRL

final_score: rating: 7: Good paper, accept
review_content: Summary of the paper:  The paper studies adversarial attacks in RL, focusing both on the design of optimal attack strategies on RL agents, as well as robust training RL procedures for mitigating attacks. Building on the results of (Zhang et al., 2020), the paper proposes a new learning framework (ATLA), that simultaneously trains a (strong) adversary and a (robust) deep RL agent. The paper showcases the importance of the new framework through extensive experimental evaluation. 

Reasons for score: Overall, I find the paper to be an interesting read and its contribution relevant to the line of work on adversarial attacks in RL. The contributions of the paper seem non-trivial, and include a framework for designing optimal attacks and training procedures that can optimize for robustness. As shown by the experiments, the proposed solution leads to significant increase in performance compared to state of the art baselines. These results complement those of (Zhang et al., 2020). Nonetheless, the presentation of the paper could be improved in terms of clarity. Some parts of the paper could be reorganized and explained in more detail. Suggestion for improvements and questions are outlined below. 

Clarity: The paper is overall enjoyable to read, but some parts are not clearly/precisely written. There are quite a few typos, some of which might be important for understanding the content. I did not follow equation (2), which seems to contain typos. Could you explain in more detail the loss function defined with this equation? Furthermore, notation in the paragraph before section 3.1 is partly confusing, in particular, $\nu$ seems to be a deterministic function, but then for observation $\hat s$ we have $\hat s \sim \nu(s)$ indicating that $\nu(s)$ might be a distribution from which we sample $\hat s \sim \nu(s)$. I would also suggest reorganizing the content related to Figure 1, which is introduced in section 1, but only explained in detail in section 3.1. 

Related Work: The related work is generally covered well, but it could be expanded by providing some connection to the line of work that studies policy teaching and policy poisoning attacks in RL. Could you explain how the setting of this paper compares to those studied in this line of work (e.g., Parkes et al. 'Policy Teaching Through Reward Function Learning', Ma et al. 'Policy Poisoning in Batch Reinforcement Learning and Control', Rakhsha et al., 'Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning', etc.)?

Experiments: I'm wondering to what extent are the results for different methods in Table 2 comparable. Namely, the methods seem to be based on different architectures, so it is not immediately clear what conclusions should be drawn from these results. Surprisingly, the discussion on page 8 does not seem to compare the results of ATLA-PPO (MLP) and SA-PPO. Could you elaborate more on these results and make relative comparison? Furthermore, in the following sentence: 'For reproducibility, for each setup we train at least 5 (up to 21) agents, attack all of them and report the one with median robustness.' - why is there discrepancy between different setups in the number of trained agents? 

review_title: Official Blind Review #2

final_score: rating: 7: Good paper, accept
review_content: The paper is very well written and the considered problem of training an adversary along with the agent is very interesting. Within the proposed concept, the parameterized adversary can be trained by viewing the agent as a part of the environment, so it avoids to access the parameters of the agent policy. From the perspective of the agent, with an unknown adversary, the MDP becomes a POMDP with uncertainty hidden in the adversary, and hence the fact of using LSTM policy is much better for the agent is reasonable. The entire problem is wisely formulated.  The experimental settings are well designed and results support the positiveness of the proposed framework.

I only have one comment that the SA-MDP can also be understood from another perspective. That is, SA-MDP is actually an asymmetric competitive multi-agent problem, and the alternative training of agent and adversary can be viewed as an instance of self-play. Also, the optimality of SA-MDP for either the agent or the adversary can be explained through multi-agent RL or game theory. It would be interesting if the authors could take a look into such a direction.
review_title: A well written and motivated paper that training agents and adversary alternatively

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper proposes to improve the robustness of a reinforcement learning agent by alternatively training an agent and an adversary who perturbs the state observations. The learning of an “optimal” adversary for a fixed policy is based on the theory of SA-MDP in prior work. The learning of an optimal policy under a fixed adversary is done by solving a POMDP problem. Experimental results show that the proposed alternating training with learned adversaries (ATLA) framework can improve the performance and robustness of PPO.

Strengths:
1. The paper is well-organized and easy to follow.
2. The authors distinguish between the vulnerability of function approximations and the intrinsic weakness of policy, which is interesting and can be useful for the community to investigate the vulnerability of deep RL.
3. The experiment results show that both the learned adversary and the trained agent perform well, in terms of attacking and learning respectively. In addition, the proposed LSTM-based policy is shown to be more robust than regular feedforward NNs.

Weakness:
1. The novelty of this paper is a little limited. (1) The idea of alternative training the agent and the adversary is similar to RARL[1], and Algorithm 2 (ATLA) is similar to Algorithm 1 in [1]. Although RARL focuses on the case where adversary directly changes the environment and ATLA focuses on the observation perturbation attacks, the whole ideas are still similar. (2) The main method of learning an adversary is based on the theoretical work of SA-MDP[2]. 
2. The authors claim that the proposed adversary is strong since it follows the theoretical framework of SA-MDP. However, in Lemma 1, the adversary reward in SA-MDP, \hat{R}, is defined as the weighted average of -R, while in Algorithm 1, the adversary reward is given by -R itself. It is not clear why such a relaxation still follows the theoretical framework. More details illustrating the approximation and some analysis about the optimality will be appreciated.
3. In the experiment section, the authors only compare the proposed algorithm with [2] in terms of “optimal” attack and robust training. However, there are a lot of works that attack the observations of a fixed policy [3,4,5]. And more importantly, [6] also proposes to train a robust agent under adversarial attacks. It will be more convincing if the authors empirically or theoretically compare with some of these potential baselines.
4.The computational complexity / sample complexity of the proposed ATLA might be problematic, as for each iteration of learning, the adversary needs to solve a new MDP, which makes the proposed robust training less practical to use. 

Minor comments:
- This paper sometimes uses the phrase "training time attack" to refer adversarial attacks, which is misleading, e.g. the second contribution, the third paragraph of related work. Training-time attack usually refers to poisoning attack, which changes the training dataset and alters the learned policy, different from the scenario in this paper where an adversary wants to fool a fixed policy.


Refs:
[1] Pinto, Lerrel, et al. "Robust adversarial reinforcement learning." arXiv preprint arXiv:1703.02702 (2017).
[2] Zhang, Huan, et al. "Robust Deep Reinforcement Learning against Adversarial Perturbations on Observations." arXiv preprint arXiv:2003.08938 (2020).
[3] Huang, Sandy, et al. "Adversarial attacks on neural network policies." arXiv preprint arXiv:1702.02284 (2017).
[4] Lin, Yen-Chen, et al. "Tactics of adversarial attack on deep reinforcement learning agents." arXiv preprint arXiv:1703.06748 (2017).
[5] Inkawhich, Matthew, Yiran Chen, and Hai Li. "Snooping Attacks on Deep Reinforcement Learning." arXiv preprint arXiv:1905.11832 (2019).
[6] Pattanaik, Anay, et al. "Robust deep reinforcement learning with adversarial attacks." arXiv preprint arXiv:1712.03632 (2017).
review_title: This paper has good insights, but more work is needed

Meta-review: The paper describes a new technique to train an adversarial MDP to perturb the observations provided by the environment.  This adversarial MDP is then used to train an RL agent to be more robust.  Since the adversarial agent essentially defines an observation distribution for the environment, the RL agent needs to optimize a POMDP.  This is nice work that was unanimously praised by the reviewers.  It produces stronger adversaries and more robust RL agents than previous work.  This represents an important contribution to the state of the art of robust RL.  
Decision: Accept (Poster)