Neural Auto-Curricula in Two-Player Zero-Sum Games

Abstract: When solving two-player zero-sum games, multi-agent reinforcement learning (MARL) algorithms often create populations of agents where, at each iteration, a new agent is discovered as the best response to a mixture over the opponent population. Within such a process, the update rules of "who to compete with" (i.e., the opponent mixture) and "how to beat them" (i.e., finding best responses) are underpinned by manually developed game theoretical principles such as fictitious play and Double Oracle. In this paper, we introduce a novel framework—Neural Auto-Curricula (NAC)—that leverages meta-gradient descent to automate the discovery of the learning update rule without explicit human design. Specifically, we parameterise the opponent selection module by neural networks and the best-response module by optimisation subroutines, and update their parameters solely via interaction with the game engine, where both players aim to minimise their exploitability. Surprisingly, even without human design, the discovered MARL algorithms achieve competitive or even better performance with the state-of-the-art population-based game solvers (e.g., PSRO) on Games of Skill, differentiable Lotto, non-transitive Mixture Games, Iterated Matching Pennies, and Kuhn Poker. Additionally, we show that NAC is able to generalise from small games to large games, for example training on Kuhn Poker and outperforming PSRO on Leduc Poker. Our work inspires a promising future direction to discover general MARL algorithms solely from data.


Reviews and author responses:

 Hi, thanks for your response!

> Reviewer: Algorithm has a "require" statement but no "ensure" statement.
> Response: Could the reviewer please elaborate, we are unsure about what you mean?

I was just asking for you to add a return statement to the pseudocode. 




 Thank you for addressing the remarks. I will maintain my initial evaluation. 

 Thanks for the clarifications. I would recommend including the response to point 1 about the combinations of solvers and oracles in the paper to make it more clear. 
Other than that I will stick to my initial evaluation.

 ### We thank Reviewer P5GY for their 4-hour effort in offering constructive comments and criticism that will surely help turn our paper into better shape.

1. > **Reviewer**: Convergence and performance in Figures 4 and 5.

* **Response**: Yes, we will run these experiments and update in due course, however early results suggest that this does not make any difference to our conclusion. We would also like to re-assert the comments we made under Question 5 in the work, even though the performance may not be much better than our baselines, we believe that our generalisation performance is very promising in training in far easier environments where we actually utilise the framework. For example, training on the small game of Kuhn Poker and outperforming standard PSRO on Leduc Poker is indicative of potential to train on easier environments to gain specialised gains in harder environments.

2. > **Reviewer**: I expect to see more SOTA baseline algorithms to convince the readers that the auto-curricula of LMAC indeed helps.

* **Response**: We decided to not compare to other SOTA algorithms as we are presenting this as an introductory work to the research area of applying meta-learning to the PSRO framework. Therefore, we believe it is fair to also utilise the same baselines as used in the major PSRO literature [Lanctot et al. NeurIPS 2017, Balduzzi et al. ICML 2019, Muller et al. ICLR 2020, McAleer et al. NeurIPS 2020] and so we would be happy to add any missing baselines from these works such as PSRO $\alpha$-Rank and Projected Replicator Dyanmics.

 ### We thank Reviewer me4m for their 5-hour effort in offering constructive comments and criticism that will help turn our paper into better shape.

1. > **Reviewer**: Line 195, wrong notation for the payoff function?

* **Response**: Thank you, we will change this from $G$ to $\mathfrak{M}$.

2. > **Reviewer**: Inconsistency of $p(G)$ and $P(G)$.

* **Response**: Thank you, we will change this so that all are $P(G)$.

3. > **Reviewer**: Line 150, what is $\delta_{|Φt−1|}$?

* **Response**: This is our mistake and is a typo, we will fix this for the camera-ready version, it should be $\Delta_{|Φt−1|}$.

4. > **Reviewer**: Typo in Figure 4.

* **Response**: Thank you, we will fix this.

 ### We thank Reviewer 1fYy for their 10-hour effort in offering constructive comments and criticism that will help turn our paper into better shape.

1. > **Reviewer**: LMAC assumes that the game is symmetric because it considers a single population? Does it generalize to asymmetric games? 

* **Response**: Whilst defining the framework in the paper, we only consider the single-population case purely for the sake of notation brevity, although it transfers to the multi-population case easily. In fact, our experiments on Iterated Matching Pennies in Figure 2 is the multi-population case.

2. > **Reviewer**: The paper mixes terminology from the RL and Game Theory literature (e.g., it uses reward, payoff, and utility interchangeably). Please use a single term for each concept. 

* **Response**: Thank you for this comment, we will clear this up in the final version.

3. > **Reviewer**: I find it confusing to define a player as a population of agents (see 121).

* **Response**: We understand that this can become confusing, we suggest maybe it would be more suitable to follow the original PSRO paper in saying a Player has a population of policies [Lanctot et al. NeurIPS 2017].

4. > **Reviewer**: I find it confusing to define a player as a population of agents (see 121).

* **Response**: It is the density placed on the $k$-th agent by the meta-distribution of iteration $t$, so similarly it is the probability of playing the $k$-th agent at iteration $t$.

5. > **Reviewer**: The precision $\sigma$ in Algorithm 1 isn't explained in the text.

* **Response**: Noted, thank you. It is relevant to the Gaussian perturbations in the ES meta-solver approach.

6. > **Reviewer**: Algorithm has a "require" statement but no "ensure" statement. 

* **Response**: Could the reviewer please elaborate, we are unsure about what you mean?

7. > **Reviewer**: How is the exploitability in line 13 of the pseudocode actually computed? Do you need to run the oracle algorithm to approximate the exploitability? 

* **Response**: Yes, the oracle algorithm is run again to solve for the exploitability. However, it will generally be run for more update steps than the oracle within the inner-loop, i.e. in line 10 and we make note of the different amount of update steps taken in our hyper-parameter tables in Appendix F.

8. > **Reviewer**: $G$ is the game, not a payoff function (line 195).

* **Response**: Noted, thank you for spotting this. We will amend for the camera-ready version by replacing $G$ with $\mathfrak{M}$.

9. > **Reviewer**: Is the expectation in Equation 13 approximated with the $n$ samples from line 4 of the pseudocode? 

* **Response**: Yes, the $n$ Gaussian perturbation samples are applied to the meta-solver and each one will return a final exploitability value which is used to calculate the expectation. 

10. > **Reviewer**: How is the exploitability computed for the plots? 

* **Response**: We utilise the same best-response oracles as those in the training process.

11. > **Reviewer**: PSRO is a framework. It isn't clear which version of PSRO is being used in the experiments. 

* **Response**: PSRO in the experiments refers to the most commonly used variety of PSRO where the Nash equilibrium is the meta-solver, whereas PSRO-Uniform refers to PSRO using a uniform distribution as the meta-solver [Lanctot et al. NeurIPS 2017, Balduzzi et al. ICML 2019]. We also refer the reviewers to OpenSpiel [Lanctot et al. CoRR 2019] for standard implementations of PSRO which are in line with our implementations. 

12. > **Reviewer**: Figure 3 is about 2D-RPS, but which version? Implicit or not implicit? 

* **Response**: Non-implicit.

13. > **Reviewer**: Figure 4(c) shows that window of size 5 performs better, but it is also largest window tried. When does it break? The plot should include larger windows. 

* **Response**: When we started with this work we did not consider window size, so we have undertaken experiments where the window size was equal to the number of total PSRO iterations, however whilst this worked, we found this to be fairly inefficient. We do agree though that we should explore this further, and will add a new plot in the camera-ready version making comparison to larger window sizes.

14. > **Reviewer**: Figure 5 shows how the model learned with one version of a game transfers to another similar game. How about training on a game and testing on a completely different one (e.g., train on GoS and evaluate on Poker)? How about training on a large number of distinct games? What does it learn? Something similar to the uniform distribution? Or a distribution that is similar to what double oracle returns?

* **Response**: We appreciate the reviewer for making this comment, as it allows us to detail a little bit about future areas of research. Our intuition is that the generalisation ability is due to similar transitive / non-transitive elements in the game-types (e.g. Poker-style games), and at training time we do limit ourselves to games that we hypothesise to share similar properties. We believe one important future area of research is to formally evaluate the similarities and differences between game-types which may help us in creating more specific versions of LMAC for transitive games / non-transitive games etc. However, we would also like to explore training on a wider variety of games that may not share similar properties, creating a far more general version of LMAC that can generalise to any game type. We are not sure what type of distribution this would return, we would expect it to be fairly dynamic taking into account the properties of the game, but we do not know currently. However, it is likely that the generalisation ability may not be infinite and that it may be difficult to generalise from games of one type to completely different types. We do know that LMAC will output something different from the uniform distribution, but also believe it is outputting something different from Nash as Nash is very non-smooth and it is difficult for a neural network to be similarly non-smooth. It is possible that LMAC learns a "smooth" version of NE, but we admittedly need to explore this further.

 ### We thank Reviewer tp6z for their 3-hour efforts in offering constructive comments and criticism that will help turn our paper into better shape.

1. > **Reviewer**: The particular choice for each combination of meta-solver and best-response oracle is not explained.


* **Response**: Generally, we did conduct ablation studies that are not reported in order to select the best possible combination for each experiment, and also make it a priority to demonstrate all possible combinations. It was not possible to use some combinations in all of our game settings, for example, the Poker games could not use a gradient-descent best-response oracle. Additionally, we do show a specific comparison in Figure 2 between implicit and non-implicit on the 2D-RPS game, as an example of how PSRO-Nash may struggle in the presence of approximate best responses.


2. > **Reviewer**: Figure 2 is not clear.

* **Response**: We understand the reviewer's confusion, and will aim to make Figure 2 more clear. The meta-solvers used for experiments marked GD (Non-implicit or Implicit stated) are set out in Section 3.5.1. The experiment marked RL uses the meta-solver set out in Section 3.5.2, and all of the Kuhn Poker experiments use the meta-solver set out in Section 3.6. All specific details are listed in Appendix F.

3. > **Reviewer**:  More intuitions about why PSRO does really well at generalisation.

* **Response**: As the Nash equilibrium itself is a strong concept from Game Theory, it can naturally generalise to most games, but not in the same way that one would discuss neural network generalisation. The Nash is not learned on a training game and then tested on a different game, as we do with LMAC, but instead the Nash can always be applied and therefore we utilise it as a baseline to show the potential of LMAC to learn more specialised meta-distributions than the all-encompassing approach of Nash. 

The authors introduce LMAC, a meta-learning framework that learns to iteratively generate a strong population of agents for a game sampled from some distribution over games. To this end they propose to train a network that predicts a meta-distribution from a given payoff matrix between the agents in the population. This meta-distribution can then be used to train a new best-response agent to add to the population. In the paper they calculate this best response either using gradient descent or reinforcement learning. The choice of meta-solver is also flexible: the authors derive both meta-gradients as well as an evolutionary strategy. They report results on a range of games (from Games of Skill, to Lotto, non-transitive games to Poker).
  Positives:
* Overall the idea is novel and very interesting.
* The paper is well and clearly written, the motivation is clear and the relevant related work is cited.
* The results of the experiments seem promising and the authors compare against appropriate baselines.

Concerns:
* Overall I have no major concerns with this paper, only a minor concern/question: the authors use different combinations of meta-solvers and best response oracles for each game. However, the particular choice for each combination is not explained (I assume that the combination of choice is the one that works best for each game, but if this is the case why is this combination better?). An experiment that could help give a bit of an intuition about this would be to test the different oracles and meta-solvers on the same game and see the effect they have.


Suggestions:
* Given that the best response algorithms are independent of the meta-solver I would introduce sections 3.5.1 and 3.5.2 outside of 3.5. It would make it more clear that they are used by the evolution strategies as well.
* Figure 2 is in my opinion initially not very clear. The authors do specify which best response algorithm they use in the title, but not what the meta solver is.
* The order of the subfigures is sometimes a bit unintuitive. In Figure 4 © is mentioned first in the text, so I would put that one first. In Figure 5, the subfigures are discussed in the opposite order as they appear in the figure, so I would change that as well.
* In the experiments in Figure 5, PSRO does really well at generalisation as well, it would be nice if the authors would add a few words about this. 

Nits:
* In line 305 it’s ‘effect’ not ‘affect’.
 The Authors discussed this appropriately.

This paper presents an auto-curriculum method for solving zero-sum games. Previous methods rely on existing methods such as fictitious play, iterated best response, or double oracle to define the set of opponents (i.e., meta strategy) an agent needs to train with. The present work describes Learned Multi-Agent Auto-Curriculum (LMAC), a system that learns the meta-strategy in an end-to-end approach. Empirical results on a set of zero-sum games show that LMAC can be competitive with existing methods.  Learned Multi-Agent Auto-Curriculum (LMAC) is a complex system that incorporates an interesting and promising idea: learn end-to-end the concept used to define the meta strategy in the context of PSRO. Instead of relying on existing approaches such as fictitious play and double oracle, LMAC learns how to solve the meta game with an end-to-end procedure. A neural model is used to choose the probability distribution over the strategies in the meta game and is trained to minimize the exploitability of the learned strategy. The idea is interesting and promising because it removes the responsibility of deciding which approach to use from the system designer; the system learns how to solve the meta game with gradient descent (or other methods for optimization such as evolutionary methods). 

Perhaps the major weakness of the paper is the somewhat mixed results. For some domains LMAC produces less exploitable strategies, for other domains previous approaches perform better. Although the idea is interesting, from the current set of experiments, it seems that the methods that rely on existing game theoretic knowledge are good enough to learn state-of-the-art strategies. Despite the somewhat weak results, I believe the idea is interesting enough for publication. In particular, the ideas presented in this paper might foster more research in this already active area. 

The weakness mentioned in the paragraph above is accentuated by the fact that the paper doesn't fully explore the types of solutions LMAC learns for the meta game. For example, it is clear what game theory approaches such as double oracle, fictitious play, and iterated best response do to the meta game. What is LMAC doing exactly? Figure explores this question in the 2D-RPS domain, but it would be interesting to observe what LMAC does in other games such as Leduc Poker, where it behaves very similarly to fictitious play. Is there a reason for this to happen? Perhaps LMAC learns something similar to a uniform distribution?  

The paper is already dense (I had to read the paper twice to make sure I understood what was going on), so I understand why some of the results and discussions didn't make it to the paper. Some parts could be removed from the main text. For example, I am not sure it is important to spell out the meta-gradient of the system. The forward pass of the system, the one that goes through several iterations of the process in which the meta game is augmented with a strategy, is easy to understand. The backward pass, where the gradient is needed, is easily implemented by modern machine learning frameworks with automatic differentiation. Figure 1 is very helpful and informative, but it would be even more helpful if it only described the forward pass of the model. The details of the backward pass and gradient computation don't need to be explained. With that in mind, Remark 3.1 could be moved entirely to the supplementary material.

The paper states that LMAC learns how to play zero-sum games without game theoretic knowledge. I think this claim is not correct. The model is trained to minimize the strategy's exploitability and it uses a learning algorithm to approximate a best response to a mixture of the meta game. Moreover, the meta game grows in each iteration with the best responses that the algorithm approximates. To me, these are game theoretic concepts that are at the core of LMAC. 

OTHER QUESTIONS/SUGGESTIONS

Questions marked with a star (*) are those I would like to see an answer for. 

1) LMAC assumes that the game is symmetric because it considers a single population? Does it generalize to asymmetric games? 

2) The paper mixes terminology from the RL and Game Theory literature (e.g., it uses reward, payoff, and utility interchangeably). Please use a single term for each concept. 

3) I find it confusing to define a player as a population of agents (see 121).

4) \pi^k_t was defined. Is it the probability of playing the k-th agent at iteration t? 

5) The precision \sigma in Algorithm 1 isn't explained in the text.

6) Algorithm has a "require" statement but no "ensure" statement. 

7) (*) How is the exploitability in line 13 of the pseudocode actually computed? Do you need to run the oracle algorithm to approximate the exploitability? 

8) G is the game, not a payoff function (line 195)

9) (*) Is the expectation in Equation 13 approximated with the n samples from line 4 of the pseudocode? 

10) How is the exploitability computed for the plots? 

11) (*) PSRO is a framework. It isn't clear which version of PSRO is being used in the experiments. 

12) Figure 3 is about 2D-RPS, but which version? Implicit or not implicit? 

13) (*) Figure 4(c) shows that window of size 5 performs better, but it is also largest window tried. When does it break? The plot should include larger windows. 

14) (*) Figure 5 shows how the model learned with one version of a game transfers to another similar game. How about training on a game and testing on a completely different one (e.g., train on GoS and evaluate on Poker)? How about training on a large number of distinct games? What does it learn? Something similar to the uniform distribution? Or a distribution that is similar to what double oracle returns?  Limitations and societal impact aren't discussed in the paper. 

In the context of two-player zero-sum game, this work introduces the Learned Multi-Agent Auto-Curriculum (LMAC) approach, a meta-learning framework that successfully automates the auto-curicula design, i.e., the update rule of the opponent mixture during training, using data from game interactions. Empirical results also demonstrate that LMAC is able to find auto-curricula that can generalise to more complex environments.  Originality:
This work opens the door to automatic auto-curricula generation, presenting promising results in this direction for 2-player zero-sum settings.

Quality: 
This work lies outside my main area of expertise, but, as far as I can tell, the work is technically sound, with no evident issues and claims are sufficiently supported by the experimental results.
 
Clarity:
The submission is clearly written and well organised. It provides sufficient details in the paper plus the supplementary material, as well as the code implementation. 

Significance: 
I believe the results are significant and interesting. LMAC performs on par with methods using game theoretic solution concepts. Moreover, the LMAC meta-solver generalises surprisingly well to larger games. 

Minor remarks:

- Line 195 - wrong notation for payoff function
- inconsistency for the game distribution notation: p(G) vs P(G)
- Line 150, perhaps I missed it, but what is $\delta_{|\Phi_{t-1}|}$?
- Figure 4, typo for y-axis 'Fianl' No statement in this regard.

This paper proposes LMAC, an auto-curricula mechanism without human design by optimizing the meta-solver through meta-gradients. The main claims are also verified by the conducted experiments.  Originality and significance:
Building upon PSRO, the proposed algorithm is among the first works that discover the MARL auto-curricula in two-player zero-sum games.

Clarity:
The paper is clearly written.

Comments:

1. In experiments Fig. 4 and 5, the curves are not convergent in the plots. Could you run both LMAC and the baselines for a longer iteration number? In Fig. 5, the improvements over PSRO are not obvious.

2. As the authors mentioned in related work, the proposed algorithm is also closely related to MARL with evolutionary strategies and meta-learning RL algorithms. But the experimental analysis and comparisons are only between LMAC and PSRO. I expect to see more SOTA baseline algorithms to convince the readers that the auto-curricula of LMAC indeed helps. The authors discuss some limitations of the formulations. And no social impact is provided.

Meta-review: The paper presents a method to use meta-learning to automatically discover curricula (LMAC), in the form of the meta-solver component of the Policy-Space Reponse Oracles (PSRO) framework of Lanctot et al, 2017. All of the reviewers agree that this is a novel and interesting direction for the game-theoretic approaches to multiagent reinforcement learning in two-player zero-sum games. The reviews also agreed on the quality of the clarity of exposition in writing/presentation. The result showing transfer from Kuhn to Leduc poker (which is roughly one or several hundred times larger depending on the implementation) is especially promising, but it is also between two similarly-structured poker-like games. As pointed out by the reviewers, what remains unclear is precisely what kind of strategies LMAC is learning and more specifically why they help for generlization, and any specific reasons for the mixed results across domains. Also, as remarked by 1fYy, the use of exploitability to drive the meta-solver loss is inspired by von Neumann's minimax theorem (or definition of Nash equilibrium) hence qualifies as game-theoretic knowledge, so the authors need to rephrase this claim. The paper could also benefit significantly from several other word changes pointed out in the reviews.

Decision: accept