Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an interesting method for being able to act and plan robustly in a multiagent simulation and be robust to the reality gap between training time and testing time for agents in a marl setting. the method does show improvements in terms of being able to train the policy for this use case and being more robust to some out of distribution configuration of the environment however these improvements appear to be rather limited. in addition, the organization and writing for the paper is very technical and could be improved with additional background information on the uses of metrics and environments as well as better flow between the content in the paper to understand the importance of the different aspects of the method. These improvements could help the reader understand the novelty and important aspects of the method that are difficult to measure. At the moment it comes across as a mix of different methods combined to be able to support this more robust method without a very clear story about the primary problem the paper is trying to solve or the more significant technical aspect of the method that provides this novel solution.

The paper appears to be very technical. The impact of this paper on a more general audience is going to be limited due to the number of details left out of the paper that would help motivate the reasoning for using certain metrics for learning why you would want bounded regret and Nash equilibria. For example, in figure 2 all of the plots have different quantities on the x and y-axis that are only briefly described and not very well motivated why these are good metrics for analyzing this type of algorithm. The writing really needs to include more motivation to clearly describe why these environments and metrics are the right kind of analysis to perform in order to indicate the efficacy of the method.

For example they bi Matrix game used at the beginning of the experiment section has no explanation. How does this game work how can we understand whether or not the method is improving in this game without understanding the difficulties and or nuanced importance of analyzing this environment.

In the augmented repeated bi Matrix game scenario that is used in test environment that is a modified version of the bi matrix game in order to show the robustness of the method there, is again, not a clear description about why these environment changes would be significant. Overall since little is understood about this game one can only assume that these particular changes to the environment could be selectively biased in order to favor the method that's proposed in the paper and overall the analysis of this section is unclear.

Figure four does not appear to be referenced in the paper anywhere I'm going to assume that it should be referenced inside of the augmented repeated by metrics game section and if that is the case the marl baseline is not very well explained for this figure before it is used in figure 4.

Additional Comments:
- The list of contributions at the end of the introduction seems to be fairly repetitive from the content in the last paragraph in the introduction.
- The part the connects the work to TRPO is not discussed carefully paper. Is the method just using TRPO for multi-agent RL?
- In figure 3 we will assume that the ordering in the text is actually left center-right.


----- Post Discussion ----
Updates to the paper have helped make technical parts of the paper more clear. The paper has also been edited to improve the motivation and experimental explanation.

review_title: Very technical and needs more motivation

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This paper tackles robust RL under the multi-agent setting. They formulate the multi-agent adversarial robustness problem as a nested optimization problem and propose a practical algorithm (ERMAS) to solve it. Theoretical proof and empirical study on two environments are provided to demonstrate the effectiveness of the proposed framework.

##########################################################################

pros: 

+ The paper is well written with clear and interesting motivation. Bridging the gap between simulation and reality for multi-agent environments is an important topic. Although I'm not an expert in this direction, it feels to me the proposed framework is technically and theoretically sound with comprehensive empirical evaluations.

##########################################################################
 
cons: 

- It would be better to include more baselines from the MARL field, especially the ones you cited in related work that also considers robust MARL. Only a vanilla MARL baseline cannot actually tell how much you improve empirically from SOTA.
- It would be better to include literature from the nested optimization field in the related work. Since you formulate the robust MARL as a nested optimization problem, have you considered or compared ERMAS with any other existing nested optimization solvers?


##########################################################################

Post Rebuttal

The paper has been updated to include additional reviews about nested optimization. I would like to keep my original score.
review_title: Robust RL under the multi-agent setting.

final_score: rating: 6: Marginally above acceptance threshold
review_content:  
##########################################################################

Summary:
This paper purposes a robust optimization framework for training robust policies, such that policies learned in imperfect
simulations can be robust against reality gaps in order to be effective in the real world. The multi-agent adversarial robustness problem is derived using \eqsilon-Nash equilibria; the paper purposes ERMAS, and present how ERMAS solves the nested robust optimization problem using dualization, trustregions, and first-order meta-learning techniques; the authors  empirically validate ERMAS by training robust policies in two multi-agent problems: sequential bimatrix games and economic simulations.

##########################################################################

Reasons for score: 

1. Overall I think the reality gap is a very important problem, and the paper is well organized and presented clearly. The robust optimization formulation provides a way to approach this problem. I have several concerns about this formulation (stated below)

2. The robust optimization formulation requires an input of the size of the uncertainty set. However I did not find discussion in the paper on how such a bound on the uncertainty set can be obtained or estimated in practice. If such a bound is loose, the robust optimization approach might become overly conservative. Following on this conservativeness concern, ERMAS learns the robust policies by finding the worst-cas reality gaps, which corresponding to highly adversarial agents --so a small fraction of highly adversarial agents could be sufficient make the uncertainty set size large.

3. I would like to see more discussions on the approximate feasibility and optimality for solving the robust optimization problem eq(7). 

4. In the experiments (figure 2), it is unclear that if the average per-time step regret experienced by the agents increases as the reward slack \epsilon increases (the trend becomes decreasing as \epsilon is large)

 
##########################################################################

Questions during rebuttal period: 

 
Please address and clarify the cons above 

 
#########################################################################

--post discussion-- 
The authors' response are helpful in addressing the concerns in the original review. I decided to update the score to 6.



review_title: weakly reject

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a method to learn _robust_ policies in multiagent environments: in particular, the policies should continue to work even if other agents in the environment deviate slightly in their behavior. Unlike previous work which allows for perturbations in other agents’ policies, this work allows for perturbations in the reward functions that those agents optimize, allowing for more relevant robustness.

Overall I liked the work -- robustness to what the agents optimize seems quite important and (to my limited knowledge) novel. The experiments seem reasonable and establish that the method (ERMAS) works well. I note a few qualms below.

----

One worry about the theory is that it is not well-defined because it allows for arbitrary exogenous perturbations, as in this line:

> The robustness objective in Equation 4 considers general agent perturbations, e.g., agents may exhibit an exogenous aversion to complex policy choices.

With this, it is no longer necessarily the case that a Nash equilibrium exists: Nash equilibria are only guaranteed to exist when the utility functions are of the _game outcomes_; they need not exist when the utility functions can also apply to the _chosen policies_.

For example, consider rock-paper-scissors between Alice and Bob. Wins get +1 utility, losses get -1, and ties get 0. Now suppose we add the perturbation “-100 if you play a stochastic policy” (normally not allowed, but allowed in this paper’s formalism as an “exogenous preference”). It is easy to see that there is no Nash equilibrium where either player plays a stochastic policy, as they could then switch to a deterministic policy. But we know there is no Nash equilibrium for rock-paper-scissors under deterministic policies.

Since Nash equilibriums are no longer guaranteed to exist, the objective in (4) may be undefined in some settings.

This could be fixed by restricting perturbations to only affect utilities of states, though this would significantly decrease the expressive power of perturbations.

----

I was confused by the fact that ERMAS outperformed MARL even when there is no change in agent policies. The authors note this fact as well:

> In fact, ERMAS outperforms the baseline AI Economist for the original setting of η = 0.23. This
suggests the robustness and performance do not necessarily pose a zero-sum game: ERMAS can find equilibria with high performance and strong generalization.

I am not sure I share the authors’ enthusiasm. It is straightforward to show that the optimal policy in the MARL setting does at least as well as the optimal policy in the robust setting. The fact that ERMAS actually outperforms MARL is surprising and suggests that ERMAS optimizes better than MARL for some reason. This could happen in one of two ways:

1. Something about ERMAS leads to better learning, in a way that can’t apply to MARL (e.g. perhaps by having adversaries learning progresses faster since any flaws are found more quickly)
2. Something about ERMAS leads to better learning, in a way that could apply to MARL (e.g. better hyperparameter tuning).

If it’s the second case (especially hyperparameter tuning), then it calls into question whether any of the improvements in the experiments come from the design of ERMAS, rather than coming from something unrelated like better hyperparameter tuning.

----

Quality: Decent. I liked the algorithm derivation (though I did not carefully check the math), and the evaluation does show the benefits of the approach, though it would have been nice to test the approach on more environments (there is just one toy environment and one more complex environment). In particular, as far as I can tell the algorithm can be applied to any multiagent RL (MARL) problem; there are several benchmark suites on which this could be tested.

Clarity: I found the paper to be quite clear.

Originality: I believe this is a novel formulation, though I am not very familiar with the robust RL literature.

Significance: Clearly relevant and important.
review_title: Great motivation, some minor qualms on execution

Meta-review: The paper presents a multi-agent RL algorithm where the rewards of the other agents are only known up to some accuracy. The setting is somewhat restrictive, in the sense that the transition is assumed to be known. It would perhaps have been more interesting for the paper to also consider unknown transitions, so as to bring it closer to work in single-agent reinforcement learning. It also seems to not be making a very good job of linking the related work to the contribution of this paper (even after looking at the appendix).

- Authors briefly say in the introduction 
"Alternative frameworks improve robustness, e.g., to changes in environment dynamics, observation or action spaces (Pinto et al., 2017; Li et al., 2019; Tessler et al., 2019), but do not address reality gaps due to reward function mismatches, as they use inappropriate metrics on the space of adversarial perturbations"

Authors should try and better explain the differences with those papers. Do  they consider changes in dynamics rather than the reward? It appears that the former is more general than the latter. Couldn't the authors compare with them with an appropriate experiment?

It is also hard to see how this exactly connects with a reality gap. What is the 'training' environment? What is the 'testing' environment? This is simply a robust optimisation algorithm applied to multi-agent games with partially unknown reward functions. 

In addition the experiments themselves are not explained clearly. 

On the plus side, I think the algorithmic details and experimental are interesting. If there was a better explanation and discussion/comparison with related work, then it would have been a good paper. Authors are encouraged to make a stronger effort to compare with other methods both in terms of the algorithm and experimentally.
Decision: Reject