Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors introduce LMAC, a meta-learning framework that learns to iteratively generate a strong population of agents for a game sampled from some distribution over games. To this end they propose to train a network that predicts a meta-distribution from a given payoff matrix between the agents in the population. This meta-distribution can then be used to train a new best-response agent to add to the population. In the paper they calculate this best response either using gradient descent or reinforcement learning. The choice of meta-solver is also flexible: the authors derive both meta-gradients as well as an evolutionary strategy. They report results on a range of games (from Games of Skill, to Lotto, non-transitive games to Poker).

main_review: Positives:
* Overall the idea is novel and very interesting.
* The paper is well and clearly written, the motivation is clear and the relevant related work is cited.
* The results of the experiments seem promising and the authors compare against appropriate baselines.

Concerns:
* Overall I have no major concerns with this paper, only a minor concern/question: the authors use different combinations of meta-solvers and best response oracles for each game. However, the particular choice for each combination is not explained (I assume that the combination of choice is the one that works best for each game, but if this is the case why is this combination better?). An experiment that could help give a bit of an intuition about this would be to test the different oracles and meta-solvers on the same game and see the effect they have.


Suggestions:
* Given that the best response algorithms are independent of the meta-solver I would introduce sections 3.5.1 and 3.5.2 outside of 3.5. It would make it more clear that they are used by the evolution strategies as well.
* Figure 2 is in my opinion initially not very clear. The authors do specify which best response algorithm they use in the title, but not what the meta solver is.
* The order of the subfigures is sometimes a bit unintuitive. In Figure 4 © is mentioned first in the text, so I would put that one first. In Figure 5, the subfigures are discussed in the opposite order as they appear in the figure, so I would change that as well.
* In the experiments in Figure 5, PSRO does really well at generalisation as well, it would be nice if the authors would add a few words about this. 

Nits:
* In line 305 it’s ‘effect’ not ‘affect’.

limitations_and_societal_impact: The Authors discussed this appropriately.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents an auto-curriculum method for solving zero-sum games. Previous methods rely on existing methods such as fictitious play, iterated best response, or double oracle to define the set of opponents (i.e., meta strategy) an agent needs to train with. The present work describes Learned Multi-Agent Auto-Curriculum (LMAC), a system that learns the meta-strategy in an end-to-end approach. Empirical results on a set of zero-sum games show that LMAC can be competitive with existing methods.
main_review: Learned Multi-Agent Auto-Curriculum (LMAC) is a complex system that incorporates an interesting and promising idea: learn end-to-end the concept used to define the meta strategy in the context of PSRO. Instead of relying on existing approaches such as fictitious play and double oracle, LMAC learns how to solve the meta game with an end-to-end procedure. A neural model is used to choose the probability distribution over the strategies in the meta game and is trained to minimize the exploitability of the learned strategy. The idea is interesting and promising because it removes the responsibility of deciding which approach to use from the system designer; the system learns how to solve the meta game with gradient descent (or other methods for optimization such as evolutionary methods). 

Perhaps the major weakness of the paper is the somewhat mixed results. For some domains LMAC produces less exploitable strategies, for other domains previous approaches perform better. Although the idea is interesting, from the current set of experiments, it seems that the methods that rely on existing game theoretic knowledge are good enough to learn state-of-the-art strategies. Despite the somewhat weak results, I believe the idea is interesting enough for publication. In particular, the ideas presented in this paper might foster more research in this already active area. 

The weakness mentioned in the paragraph above is accentuated by the fact that the paper doesn't fully explore the types of solutions LMAC learns for the meta game. For example, it is clear what game theory approaches such as double oracle, fictitious play, and iterated best response do to the meta game. What is LMAC doing exactly? Figure explores this question in the 2D-RPS domain, but it would be interesting to observe what LMAC does in other games such as Leduc Poker, where it behaves very similarly to fictitious play. Is there a reason for this to happen? Perhaps LMAC learns something similar to a uniform distribution?  

The paper is already dense (I had to read the paper twice to make sure I understood what was going on), so I understand why some of the results and discussions didn't make it to the paper. Some parts could be removed from the main text. For example, I am not sure it is important to spell out the meta-gradient of the system. The forward pass of the system, the one that goes through several iterations of the process in which the meta game is augmented with a strategy, is easy to understand. The backward pass, where the gradient is needed, is easily implemented by modern machine learning frameworks with automatic differentiation. Figure 1 is very helpful and informative, but it would be even more helpful if it only described the forward pass of the model. The details of the backward pass and gradient computation don't need to be explained. With that in mind, Remark 3.1 could be moved entirely to the supplementary material.

The paper states that LMAC learns how to play zero-sum games without game theoretic knowledge. I think this claim is not correct. The model is trained to minimize the strategy's exploitability and it uses a learning algorithm to approximate a best response to a mixture of the meta game. Moreover, the meta game grows in each iteration with the best responses that the algorithm approximates. To me, these are game theoretic concepts that are at the core of LMAC. 

OTHER QUESTIONS/SUGGESTIONS

Questions marked with a star (*) are those I would like to see an answer for. 

1) LMAC assumes that the game is symmetric because it considers a single population? Does it generalize to asymmetric games? 

2) The paper mixes terminology from the RL and Game Theory literature (e.g., it uses reward, payoff, and utility interchangeably). Please use a single term for each concept. 

3) I find it confusing to define a player as a population of agents (see 121).

4) \pi^k_t was defined. Is it the probability of playing the k-th agent at iteration t? 

5) The precision \sigma in Algorithm 1 isn't explained in the text.

6) Algorithm has a "require" statement but no "ensure" statement. 

7) (*) How is the exploitability in line 13 of the pseudocode actually computed? Do you need to run the oracle algorithm to approximate the exploitability? 

8) G is the game, not a payoff function (line 195)

9) (*) Is the expectation in Equation 13 approximated with the n samples from line 4 of the pseudocode? 

10) How is the exploitability computed for the plots? 

11) (*) PSRO is a framework. It isn't clear which version of PSRO is being used in the experiments. 

12) Figure 3 is about 2D-RPS, but which version? Implicit or not implicit? 

13) (*) Figure 4(c) shows that window of size 5 performs better, but it is also largest window tried. When does it break? The plot should include larger windows. 

14) (*) Figure 5 shows how the model learned with one version of a game transfers to another similar game. How about training on a game and testing on a completely different one (e.g., train on GoS and evaluate on Poker)? How about training on a large number of distinct games? What does it learn? Something similar to the uniform distribution? Or a distribution that is similar to what double oracle returns? 
limitations_and_societal_impact: Limitations and societal impact aren't discussed in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In the context of two-player zero-sum game, this work introduces the Learned Multi-Agent Auto-Curriculum (LMAC) approach, a meta-learning framework that successfully automates the auto-curicula design, i.e., the update rule of the opponent mixture during training, using data from game interactions. Empirical results also demonstrate that LMAC is able to find auto-curricula that can generalise to more complex environments.
main_review: Originality:
This work opens the door to automatic auto-curricula generation, presenting promising results in this direction for 2-player zero-sum settings.

Quality: 
This work lies outside my main area of expertise, but, as far as I can tell, the work is technically sound, with no evident issues and claims are sufficiently supported by the experimental results.
 
Clarity:
The submission is clearly written and well organised. It provides sufficient details in the paper plus the supplementary material, as well as the code implementation. 

Significance: 
I believe the results are significant and interesting. LMAC performs on par with methods using game theoretic solution concepts. Moreover, the LMAC meta-solver generalises surprisingly well to larger games. 

Minor remarks:

- Line 195 - wrong notation for payoff function
- inconsistency for the game distribution notation: p(G) vs P(G)
- Line 150, perhaps I missed it, but what is $\delta_{|\Phi_{t-1}|}$?
- Figure 4, typo for y-axis 'Fianl'
limitations_and_societal_impact: No statement in this regard.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes LMAC, an auto-curricula mechanism without human design by optimizing the meta-solver through meta-gradients. The main claims are also verified by the conducted experiments.
main_review: Originality and significance:
Building upon PSRO, the proposed algorithm is among the first works that discover the MARL auto-curricula in two-player zero-sum games.

Clarity:
The paper is clearly written.

Comments:

1. In experiments Fig. 4 and 5, the curves are not convergent in the plots. Could you run both LMAC and the baselines for a longer iteration number? In Fig. 5, the improvements over PSRO are not obvious.

2. As the authors mentioned in related work, the proposed algorithm is also closely related to MARL with evolutionary strategies and meta-learning RL algorithms. But the experimental analysis and comparisons are only between LMAC and PSRO. I expect to see more SOTA baseline algorithms to convince the readers that the auto-curricula of LMAC indeed helps.
limitations_and_societal_impact: The authors discuss some limitations of the formulations. And no social impact is provided.
review_title: null

Meta-review: The paper presents a method to use meta-learning to automatically discover curricula (LMAC), in the form of the meta-solver component of the Policy-Space Reponse Oracles (PSRO) framework of Lanctot et al, 2017. All of the reviewers agree that this is a novel and interesting direction for the game-theoretic approaches to multiagent reinforcement learning in two-player zero-sum games. The reviews also agreed on the quality of the clarity of exposition in writing/presentation. The result showing transfer from Kuhn to Leduc poker (which is roughly one or several hundred times larger depending on the implementation) is especially promising, but it is also between two similarly-structured poker-like games. As pointed out by the reviewers, what remains unclear is precisely what kind of strategies LMAC is learning and more specifically why they help for generlization, and any specific reasons for the mixed results across domains. Also, as remarked by 1fYy, the use of exploitability to drive the meta-solver loss is inspired by von Neumann's minimax theorem (or definition of Nash equilibrium) hence qualifies as game-theoretic knowledge, so the authors need to rephrase this claim. The paper could also benefit significantly from several other word changes pointed out in the reviews.
Decision: Accept (Poster)