Neural Auto-Curricula in Two-Player Zero-Sum Games

1 Introduction . Two-player zero-sum games have been a central interest of the recent development of multi-agent reinforcement learning ( MARL ) [ 60 , 2 , 53 ] . In solving such games , a MARL agent has a clear objective : to minimise the worst case performance , its exploitability , against any potential opponents . When both agents achieve zero exploitability , they reach a Nash equilibrium ( NE ) [ 36 ] , a classical solution concept from Game Theory [ 34 , 10 ] . Even though this objective is straightforward , developing effective algorithms to optimise such an objective often requires tremendous human efforts . One effective approach is through iterative methods , where players iteratively expand a population of agents ( a.k.a auto-curricula [ 26 ] ) where at each iteration , a new agent is trained and added to the player ’ s strategy pool . However , within the auto-curricula process , it is often non-trivial to design effective update rules of `` who to compete with '' ( i.e. , opponent selections ) and `` how to beat them '' ( i.e. , finding best responses ) . The problem becomes more challenging when one considers additional requirements such as generating agents that have behavioural diversity [ 58 , 37 , 3 , 28 ] or generalisation ability [ 47 , 38 ] . 1⇤Equal contributions . †Corresponding to < yaodong.yang @ pku.edu.cn > . Code released at https : // github.com/waterhorse1/NAC 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . One effective approach to designing auto-curricula is to follow game theoretical principles such as fictitious play [ 5 ] and Double Oracle ( DO ) methods [ 33 , 11 ] . For example , in the DO dynamics , each player starts from playing a sub-game of the original game where only a restricted set of strategies are available ; then , at each iteration , each player will add a best-response strategy , by training against a NE mixture of opponent models at the current sub-game , into its strategy pool . When an exact best response is hard to compute , approximation methods such as reinforcement learning ( RL ) methods are often applied [ 25 ] . One of the potential downsides of this approach is , under approximate best response methods , one no longer maintains the theoretical properties of DO and a worthwhile avenue of exploration is to find auto-curricula that are more conducive to these approximation solutions . Apart from following game theoretical principles , an appealing alternative approach one can consider is to automatically discover auto-curricula from data generated by playing games , which can be formulated as a meta-learning problem [ 13 ] . However , it remains an open challenge whether it is feasible to discover fundamental concepts ( e.g. , NE ) entirely based on data . If we can show that it is possible to discover fundamental concepts from scratch , it opens the avenue to trying to discover fundamentally new concepts at a potentially rapid pace . Although encouraging results have been reported in single-agent RL settings showing that it is possible to meta-learn RL update rules , such as temporal difference learning [ 38 , 56 , 57 ] , and the learned update rules can generalise to unseen tasks , we believe discovering auto-curricula in multi-agent cases is particularly hard . Two reasons for this are that the discovered auto-curriculum itself directly affects the development of the agent population , and each agent involves an entire training process , which complicates the meta-learning process . Albeit challenging , we believe a method capable of discovering auto-curricula without explicit game theoretic knowledge can potentially open up entirely new approaches to MARL . As a result , this paper initiates the study on discovering general-purpose MARL algorithms in two-player zero-sum games . Specifically , our goal is to develop an algorithm that learns its own objective ( i.e. , the auto-curricula ) solely from environment interaction , and we offer a meta-learning framework that achieves such a goal . Our solution framework – Neural Auto-Curriculum ( NAC ) – has two promising properties . Firstly , it does not rely on human-designed knowledge about game theoretic principles , but instead lets the meta-learner decide what the meta-solution concept ( i.e. , who to compete with ) should be during training . This property means that our meta-learner shares the ability of game theoretic principles in being able to accurately evaluate the policies in the population . Secondly , by taking the best-response computation into consideration , NAC can end to end offer more suitable auto-curricula within the approximate best-response scenario compared with previous approaches ; this is particularly important since an exact best-response oracle is not always available in practice . Our empirical results show that NAC can discover meaningful solution concepts alike NE , and based on that build effective auto-curricula in training agent populations . In multiple different environments , the discovered auto-curriculum achieves the same performance or better than that of PSRO methods [ 25 , 3 ] . We additionally evaluate the ability of our discovered meta-solvers to generalise to unseen games of a similar type ( e.g. , training on Kuhn Poker and testing on Leduc Poker ) , and show that the auto-curricula found on a simple environment is able to generalise to a more difficult one . To the best of our knowledge , this is the first work that demonstrates the possibility of discovering an entire auto-curriculum in solving two-player zero-sum games , and that the rule discovered from simple domains can be competitive with human-designed algorithms on challenging benchmarks . 2 Related Work . Whilst it is theoretically possible to solve for NE in two-player zero-sum games via linear programming ( LP ) [ 34 ] in polynomial time [ 51 ] , this is a strictly limited approach . For example , when the action space become prohibitively large , or continuous , using LP becomes intractable ; other approximation methods such as fictitious play ( FP ) [ 5 ] , DO [ 33 , 11 ] or PSRO [ 25 , 32 , 31 ] methods are required . These methods all follow iterative best-response dynamics where at each iteration , a best response policy is found against a previous aggregated policy ( e.g. , DO/PSRO applies a sub-game NE , FP applies time-average strategy ) . Under this general iterative framework , other solution concepts include Rectified NE [ 3 ] , ↵-Rank [ 35 , 59 ] or no-regret algorithms [ 11 ] . In this paper , instead of following any existing game theoretic knowledge , we try to discover effective solution concepts solely through environment interactions via meta-learning techniques . Meta-learning , also known as learning to learn , has recently gained increased attention for successfully improving sample efficiency in regression , classification and RL tasks . MAML [ 13 ] meta-learns model parameters by differentiating the learning process for fast adaptation on unseen tasks . PROMP [ 42 ] theoretically analyses the MAML-RL formulation and addresses the biased meta-gradient issue . ES-MAML [ 48 ] bypasses the Hessian estimation problem by leveraging evolutionary strategies for gradient-free optimisation . RL2 [ 12 ] and L2RL [ 54 ] formulate the meta-learning problem as a second RL procedure but update the parameters at a `` slow '' pace . Recently , there is a trend to meta-learn the algorithmic components of RL algorithms , such as the discount factor [ 57 ] , intrinsic rewards [ 65 , 64 ] , auxiliary tasks [ 52 , 66 ] , objective functions for value/policy networks [ 4 , 56 , 22 ] , off-policy update targets [ 62 ] , and the bootstrapping mechanism [ 38 ] . Their success is built on the meta-gradient technique , which leverages gradient descent on the sequence of gradient descent updates resulting from the choice of objective function . The meta-learned RL algorithms demonstrate promising generalisation capability in solving different tasks . Apart from meta-gradients , evolutionary strategies ( ES ) have also been successfully applied [ 20 ] . Our paper , in comparison , has a parallel goal to these prior arts : to discover MARL algorithms that are effective for solving various two-player zero-sum games , and to demonstrate their generalisation ability across different games . So far , there have been very few attempts to conduct meta-learning in multi-agent settings [ 60 ] . MNMPG [ 46 ] applied meta-gradients for training credit assignment modules for better decomposing the value network [ 49 , 61 ] . Meta-PG [ 1 ] was proposed as a meta-gradient solution to continuous adaptations for non-stationary environments . Building on the opponent modelling technique called LOLA [ 15 , 14 ] , Meta-MAPG [ 21 ] extends Meta-PG [ 1 ] to multi-agent settings by considering both the non-stationarity from the environment and from the opponent ’ s learning process . Our work rather focuses on automatically discovering multi-agent auto-curricula through meta-learning , without explicit game theoretic knowledge , that leads to general MARL algorithms for solving two-player zero-sum games . 3 Multi-Agent Meta-Learning Framework . In this section , we discuss our meta-learning framework NAC for discovering multi-agent autocurricula in two-player zero-sum games . We follow the road-map set out by Fig . ( 1 ) which illustrates the flow of NAC . The relevant sections for each part of Fig . ( 1 ) are marked in pink . 3.1 The Meta-game . Consider two-player zero-sum games G2 drawn from some distribution P ( G ) , where players have the state-action space S ⇥A . We start by introducing the notion of an agent who is characterised by a policy , where a policy is a mapping : S ⇥A ! [ 0 , 1 ] which can be described in both a tabular form or as a neural network . The payoff between two agents is defined to be M ( i , j ) ( i.e. , the game engine ) , and represents the utility to agent i , or alternatively , the negative utility to agent j . Our population-based framework revolves around iterative updates on the meta-game M. At every iteration t 2 T , a Player is defined by a population of fixed agents t = 0 [ BR1 , BR 2 , ... , BR t , where 0 is the initial random agent pool and the BRt are discussed further in Sec . ( 3.3 ) . From here , 2G encapsulates all of the information for a Player to take part in a game ( e.g. , actions , reward functions ) . for the sake of notation convenience , we will only consider the single-population case where Players share the same t. As such , the single population will generate a meta-game Mt , a payoff matrix between all of the agents in the population , with individual entries being M ( i , j ) 8 i , j 2 t . 3.2 The Meta-Solver . Based on Mt , the Player will solve for the meta-distribution ⇡t which is defined as an aggregated agent over the fixed agents in t. These meta-solvers in recent work have stuck to human-designed solution concepts such as a uniform distribution [ 25 ] or NE and its variants [ 3 , 37 , 28 ] , whereas we introduce a method to actively learn these distributions through solely interacting with the game . Specifically , we parameterise our meta-solver via a neural network . This network f✓ with parameters ✓ is a mapping f✓ : Mt ! [ 0 , 1 ] t which takes as input a meta-game Mt and outputs a meta-distribution ⇡t = f✓ ( Mt ) . The output ⇡t is a probability assignment to each agent in the population t and , as we are in the single-population setting , we do not distinguish between different populations . There are two characteristics that our network f✓ should have : firstly , it should be able to handle variable-length inputs as the size of Mt is ( t ⇥ t ) , and f✓ outputs a vector of size ( t ⇥ 1 ) , where the value of t increments at every iteration . Secondly , f✓ should have both column-permutation invariance and row-permutation equivariance . Given an input Mt , f✓ will output the distribution ⇡t = f✓ ( Mt ) for the row Player and it should be equivariant to any permutation of the row index , and be invariant to any permutation of the column index as neither of these actions should affect ⇡t . Much work has been done on how to maintain permutation invariance/equivariance for neural networks such as Deepsets [ 63 ] for handling tasks defined on sets , or PointNet [ 39 ] for handling 3D point cloud data processing . Our first Multi-Layer Perceptron ( MLP ) based network is inspired by PointNet , which consists of three components : ( 1 ) An MLP over each element Mij t for a non-linear representation , ( 2 ) Column Mean-Pooling for column-permutation invariant information aggregation , which will be concatenated to each row and ( 3 ) Row-wise MLP for the row-permutation equivariant meta distribution . We also offer two alternatives , inspired by a fully Conv1d-based model [ 29 ] and a GRU-based model [ 7 ] . We refer to Appendix A for detailed architectures of our network f✓ ( · ) .

Summary: The authors introduce LMAC, a meta-learning framework that learns to iteratively generate a strong population of agents for a game sampled from some distribution over games. To this end they propose to train a network that predicts a meta-distribution from a given payoff matrix between the agents in the population. This meta-distribution can then be used to train a new best-response agent to add to the population. In the paper they calculate this best response either using gradient descent or reinforcement learning. The choice of meta-solver is also flexible: the authors derive both meta-gradients as well as an evolutionary strategy. They report results on a range of games (from Games of Skill, to Lotto, non-transitive games to Poker).