Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper extends the existing work on robust adversarial RL by training multiple adversarial agents from a population. Solid experimental results are presented to show that the proposed method improves the single adversary setting and domain randomization. 

The experimental results in this paper seem solid to me. My biggest concern for this paper is that the conceptual novelty seems quite incremental. I mean, on the conceptual level, it is a common sense from robust control that multiple uncertainty sources can be treated together (e.g. the robust control theory handles the structured uncertainty in such a manner). One can augment all the adversary networks as a big network and then the problem formulation is the same as before. If we think each phi_i is a block in this big "single adversary network", then what the authors have done can be thought as doing  block coordinate descent. From this perspective, there is not too much conceptual novelty, and the main contribution of this paper is doing some more detailed study showing how to combine augmentation and adversarial RL. I am not sure whether such a contribution itself is enough for ICLR or not.
review_title: The conceptual novelty is quite incremental but the experimental results are solid.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an algorithm to improve the robustness of reinforcement learning. The algorithm , RAP, combines ideas from domain randomization and adversarial training. Specifically, during learning, it trains an ensemble of adversary to attack the learner, with the hope that the learner can be robust to various situations. The experimental results show the proposed algorithm indeed outperform the respective baselines here (single-adversary training and domain randomization) in its ability to generalize the other test domains. 

I think overall the proposed algorithm presents a simple and nice way to join the strengths of the adversary training and domain randomization. Indeed, we can view single-adversary training and domain randomization as special cases of RAP, which either uses a single adversary or not perform any update on the adversaries. For this, I would imagine RAP can be quite effective in practice. 

* Issue about motivation and explaination

However, despite this nice design, I think the main motivation and the explanation why the proposed algorithm works are not completely correct. In introduction, the authors motivate the use of multiple adversary from that pure Nash equilibrium does not always exist in zero sum two player games. However, adversary training is not necessarily about solving the Nash equilibrium (which aims to find solutions such that minimax = maximin) but rather solving a maximin "only", whose solution is always well defined. 

The motivating robot example is also quite misleading. In that case, the failure is due to the learner policy is not even solving maximin problem. Note in maximin, the adversary is chosen after knowing the learner's policy. This robot example is rather saying that minimax, where the learner optimizes after the adversary is chosen, is insufficient to generate robust behavior. However, this is not what adversarial training is about. 

I think, it's probably because of this misunderstanding, the authors motivate the issue of the single-adversary training as the agent would overfit to the single adversary. Again this is due to incorrectly interpreting maximin in (1) as minimax. 

Nonetheless, I do believe the proposed algorithm is effective in practice, but for a different reason from what the authors explain. I think the RAP does improve upon single-adversary training. Because it uses multiple randomly initialized adversarial policies, it may have a higher chance to overcome the non-convexity issue in the min part of maxmin and  therefore has a higher chance to find the maximin solution. In other words, the failure of the single-adversary implementation is most due to optimization difficulty not that the solution concept is incorrect. And the proposed algorithm is more of an optimization heuristic to better approximate the maximin problem (which actually is quite commonly used in the optimization literature). In fact, when given the learner policy, I believe by further taking the min among the multiple adversaries here and uses that for the learner update (i.e. the leaner would not use trajectories from all but the worst one), the robustness of the algorithm might further improve. 

* Experiments 

Given this work is lacking theoretical insights, I expect more experiments to be done to verify the proposed algorithm. The current paper only test RAP on mujoco environments. I think for understanding how well this algorithm works, the authors should test the algorithm on a more tasks with diverse characteristics (e.g. tabular, video games, tabular, traffic,) rather just the continuous robotics control domain. 

In figure 2, I think a fairer comparison should let agents of both sides face the same adversary. Nonetheless, I agree that the performance plots later on are sufficient to show that RAP is better. 

Lastly I like the in-depth discussion about the failure of domain randomization in halfcheetah, as it's also my main question when reading the previous part of the paper. However, I'm wondering if the bad performance is due to that the learner's policy is not expressive enough. I cannot find the description of the exact architecture used in the paper, but can you try training with a more expressive policy and see if domain randomization still perform worse than the direct training without the adversary?





 
review_title: The proposed algorithm is good but the motivation is unsolid and experiments are limited

final_score: rating: 7: Good paper, accept
review_content: #### Summary

The authors present a scheme that can be used to train agents to be robust against a population of adversarial policies, in which adversaries can perturb actions via an additive perturbation.  Motivated by the observation that agents trained against a single policy may overfit to that policy and hence will lack robustness to new/unseen policies, the authors seek to show that their method generalizes well to unseen policies at test time.  Their experiments consider several simulated environments, in which they show generally good performance against several baselines.

#### Strengths

- I find the argument that agents will overfit to a single policy convincing.  While the motivating example WRT different forces acting on an agent may not be a scenario that robustness against adaptive perturbations can handle, the general claim seems to hold water.  That is, it seems plausible that an agent might receive high reward in a zero sum min-max game subject to a single adversary simply because the adversary is not strong enough to limit the cumulative reward that agent receives.

- The formulation for RAP is presented very clearly.  It is quite helpful to first consider the single minmax adversary and domain randomization formulations, both of which seemed to have played roles in the development of RAP.  Indeed, this seems a very natural way of formulating the problem.  More generally, the paper is quite well written.

- The experiments, and in particular Fig 2, clearly demonstrate the utility of this approach.  One can see a notable difference when an agent is trained with respect to three adversarial policies vs. when it is only trained with a single adversarial policy.

#### Weaknesses

- The utility of this approach is less clear when one considers Fig. 3.  It seems that domain randomization often outperforms RAP.  In my opinion, this weakens part of the claim made by the authors.  However, it does seem true that given that DR is designed to perform well on new domains _in expectation_, it may be preferable to use RAP when one suffers worst case dynamics changes (e.g. Fig. 5).  

- Further study should be done as to how many adversaries one needs to provide meaningful levels of robustness.  It seems that in different experiments, 1, 3, and 5 adversaries were considered.  How should one decided how many adversaries to use?  

- I felt the paper was a bit unclear about what states are available to train adversarial and regular policies.  That is, it is unclear whether in the trajectories $\tau_j$ and $\tau_j^i$, the agents/adversaries observe their own actions, the actions of their counterpart (e.g. the agent observing the adversaries action), or both agent and adversary observing the perturbed action $a + \alpha \bar{a}$.  The latter case would lead to a lack of observability for both agent and adversary.  Perhaps the authors can clarify this in the rebuttal.

- One weakness is that only one scenario (e.g. walking within these three environments) was considered.  It seems that the claims of the paper could be strengthened if more environments/tasks were considered.

#### Further questions/clarifications

- The reward function seems to be denoted as $\mathcal{R}$, $R$, and $r$ in various places.

- The bolding in Table 1 is a bit confusing.  I would be fairer to bold the most robust approach overall, rather than the most robust approach of the methods you propose.

- In the notation of Section 4, why are the rollouts of length $M$ rather than $T$, as indicated in the formulation?

#### Final Thoughts

Overall, I thought this was a solid and interesting paper.  The motivation is compelling, the formulation is relatively clean, and the experiments generally back up the claims that are made by the authors.  There are a few weaknesses, as I enumerated above, but even still I think that this is a valuable contribution.
review_title: A clean formulation with some compelling results

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper proposes to improve robustness in reinforcement learning via a population of diverse adversaries, where previous works mainly focus on the use a single adversary to mitigate the problem that the trained policy could be highly exploitable by the adversary. Specifically, at each iteration, it randomly selects an adversary from the population for rollouts, and it is trained by PPO. Experiments are conducted on 3 MuJoCo environments in comparison with vanilla PPO, domain randomization. 

Strong points: Using a population of adversaries to improve robustness in RL is interesting. The idea is simple, and the writing is clear.

Concerns:
My major concern is in the experimental evaluation.
a. Results are shown using final performance. I am curious about the learning curve – how does the method compare against other baselines in terms of sample efficiency? A side-effect using a population is that RAP needs to update n adversaries at each training iteration compared with using a single adversary, and will incur more computation overhead. Could authors fairly compare with other baselines in terms of this and show the learning curve? 

b. How *MUCH LONGER* does it take to run RAP compared with other baselines? How much more memory does it take to use n adversaries compared with a single adversary?

c. Could authors compare with a naive extension of the single adversary case in which the single adversary sample n actions? Is the baseline comparable with RAP using n adversaries? 

d. I am confused why RAP is built upon an on-policy algorithm. A number of works using population-based methods are built upon off-policy algorithms as agents in the population can share the samples and could be beneficial. Could authors build the method upon off-policy algorithms to further improve the applicability of RAP?

e. For Figure 3, the performance gain over using a single adversary is not significant on HalfCheetah and Ant, and the results is not convincing enough to support the claim. 

As the paper uses the population-based methods, it is also worth discussing its relation with Khadka et al. 2018, etc.
review_title: Simple and nice idea, but experiments are not convincing enough.

Meta-review: The paper studies reinforcement learning in the presence of (adversarial) perturbations in the underlying system dynamics. The main (novel) observation is that  agents trained against a single policy may overfit  to that policy and hence will lack robustness to new/unseen policies. The paper proposes a population-based augmentation to the Robust RL formulation in which a population of adversaries are randomly initialized and samples from during training. The authors seek to show that their method generalizes well to unseen policies at test time.

Most reviewers agree that the paper provides a range of solid experimental results (with in-distribution and out-of-distribution tasks) showing robustness and generalization of their methods on several robotics benchmarks while avoiding a ubiquitous domain randomization failure mode. However, all the reviewers (and myself) agree that some of the conceptual claims of the paper may not be precise. For example, some of the reviewers disagree with the authors on finding the (mixed) Nash equilibria. Such general claims are hard to validate (may not even be true) and need theoretical justification. Hence, it is not conceptually clear why using multiple adversaries would not suffer from the same limitations as in the single adversary case.  Also, in the discussion phase, the reviewers agreed that the results/claims of the paper (i.e. overfitting to a single adversary and the need for multiple adversaries) are very interesting, but at the same time need to be confirmed by more extensive experiments.  

Indeed, if the above are addressed, the paper would make a strong contribution to the area of RL. 


Decision: Reject