Neural Auto-Curricula in Two-Player Zero-Sum Games

1 Introduction . Two-player zero-sum games have been a central interest of the recent development of multi-agent reinforcement learning ( MARL ) [ 60 , 2 , 53 ] . In solving such games , a MARL agent has a clear objective : to minimise the worst case performance , its exploitability , against any potential opponents . When both agents achieve zero exploitability , they reach a Nash equilibrium ( NE ) [ 36 ] , a classical solution concept from Game Theory [ 34 , 10 ] . Even though this objective is straightforward , developing effective algorithms to optimise such an objective often requires tremendous human efforts . One effective approach is through iterative methods , where players iteratively expand a population of agents ( a.k.a auto-curricula [ 26 ] ) where at each iteration , a new agent is trained and added to the player ’ s strategy pool . However , within the auto-curricula process , it is often non-trivial to design effective update rules of `` who to compete with '' ( i.e. , opponent selections ) and `` how to beat them '' ( i.e. , finding best responses ) . The problem becomes more challenging when one considers additional requirements such as generating agents that have behavioural diversity [ 58 , 37 , 3 , 28 ] or generalisation ability [ 47 , 38 ] . 1⇤Equal contributions . †Corresponding to < yaodong.yang @ pku.edu.cn > . Code released at https : // github.com/waterhorse1/NAC 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . One effective approach to designing auto-curricula is to follow game theoretical principles such as fictitious play [ 5 ] and Double Oracle ( DO ) methods [ 33 , 11 ] . For example , in the DO dynamics , each player starts from playing a sub-game of the original game where only a restricted set of strategies are available ; then , at each iteration , each player will add a best-response strategy , by training against a NE mixture of opponent models at the current sub-game , into its strategy pool . When an exact best response is hard to compute , approximation methods such as reinforcement learning ( RL ) methods are often applied [ 25 ] . One of the potential downsides of this approach is , under approximate best response methods , one no longer maintains the theoretical properties of DO and a worthwhile avenue of exploration is to find auto-curricula that are more conducive to these approximation solutions . Apart from following game theoretical principles , an appealing alternative approach one can consider is to automatically discover auto-curricula from data generated by playing games , which can be formulated as a meta-learning problem [ 13 ] . However , it remains an open challenge whether it is feasible to discover fundamental concepts ( e.g. , NE ) entirely based on data . If we can show that it is possible to discover fundamental concepts from scratch , it opens the avenue to trying to discover fundamentally new concepts at a potentially rapid pace . Although encouraging results have been reported in single-agent RL settings showing that it is possible to meta-learn RL update rules , such as temporal difference learning [ 38 , 56 , 57 ] , and the learned update rules can generalise to unseen tasks , we believe discovering auto-curricula in multi-agent cases is particularly hard . Two reasons for this are that the discovered auto-curriculum itself directly affects the development of the agent population , and each agent involves an entire training process , which complicates the meta-learning process . Albeit challenging , we believe a method capable of discovering auto-curricula without explicit game theoretic knowledge can potentially open up entirely new approaches to MARL . As a result , this paper initiates the study on discovering general-purpose MARL algorithms in two-player zero-sum games . Specifically , our goal is to develop an algorithm that learns its own objective ( i.e. , the auto-curricula ) solely from environment interaction , and we offer a meta-learning framework that achieves such a goal . Our solution framework – Neural Auto-Curriculum ( NAC ) – has two promising properties . Firstly , it does not rely on human-designed knowledge about game theoretic principles , but instead lets the meta-learner decide what the meta-solution concept ( i.e. , who to compete with ) should be during training . This property means that our meta-learner shares the ability of game theoretic principles in being able to accurately evaluate the policies in the population . Secondly , by taking the best-response computation into consideration , NAC can end to end offer more suitable auto-curricula within the approximate best-response scenario compared with previous approaches ; this is particularly important since an exact best-response oracle is not always available in practice . Our empirical results show that NAC can discover meaningful solution concepts alike NE , and based on that build effective auto-curricula in training agent populations . In multiple different environments , the discovered auto-curriculum achieves the same performance or better than that of PSRO methods [ 25 , 3 ] . We additionally evaluate the ability of our discovered meta-solvers to generalise to unseen games of a similar type ( e.g. , training on Kuhn Poker and testing on Leduc Poker ) , and show that the auto-curricula found on a simple environment is able to generalise to a more difficult one . To the best of our knowledge , this is the first work that demonstrates the possibility of discovering an entire auto-curriculum in solving two-player zero-sum games , and that the rule discovered from simple domains can be competitive with human-designed algorithms on challenging benchmarks . 2 Related Work . Whilst it is theoretically possible to solve for NE in two-player zero-sum games via linear programming ( LP ) [ 34 ] in polynomial time [ 51 ] , this is a strictly limited approach . For example , when the action space become prohibitively large , or continuous , using LP becomes intractable ; other approximation methods such as fictitious play ( FP ) [ 5 ] , DO [ 33 , 11 ] or PSRO [ 25 , 32 , 31 ] methods are required . These methods all follow iterative best-response dynamics where at each iteration , a best response policy is found against a previous aggregated policy ( e.g. , DO/PSRO applies a sub-game NE , FP applies time-average strategy ) . Under this general iterative framework , other solution concepts include Rectified NE [ 3 ] , ↵-Rank [ 35 , 59 ] or no-regret algorithms [ 11 ] . In this paper , instead of following any existing game theoretic knowledge , we try to discover effective solution concepts solely through environment interactions via meta-learning techniques . Meta-learning , also known as learning to learn , has recently gained increased attention for successfully improving sample efficiency in regression , classification and RL tasks . MAML [ 13 ] meta-learns model parameters by differentiating the learning process for fast adaptation on unseen tasks . PROMP [ 42 ] theoretically analyses the MAML-RL formulation and addresses the biased meta-gradient issue . ES-MAML [ 48 ] bypasses the Hessian estimation problem by leveraging evolutionary strategies for gradient-free optimisation . RL2 [ 12 ] and L2RL [ 54 ] formulate the meta-learning problem as a second RL procedure but update the parameters at a `` slow '' pace . Recently , there is a trend to meta-learn the algorithmic components of RL algorithms , such as the discount factor [ 57 ] , intrinsic rewards [ 65 , 64 ] , auxiliary tasks [ 52 , 66 ] , objective functions for value/policy networks [ 4 , 56 , 22 ] , off-policy update targets [ 62 ] , and the bootstrapping mechanism [ 38 ] . Their success is built on the meta-gradient technique , which leverages gradient descent on the sequence of gradient descent updates resulting from the choice of objective function . The meta-learned RL algorithms demonstrate promising generalisation capability in solving different tasks . Apart from meta-gradients , evolutionary strategies ( ES ) have also been successfully applied [ 20 ] . Our paper , in comparison , has a parallel goal to these prior arts : to discover MARL algorithms that are effective for solving various two-player zero-sum games , and to demonstrate their generalisation ability across different games . So far , there have been very few attempts to conduct meta-learning in multi-agent settings [ 60 ] . MNMPG [ 46 ] applied meta-gradients for training credit assignment modules for better decomposing the value network [ 49 , 61 ] . Meta-PG [ 1 ] was proposed as a meta-gradient solution to continuous adaptations for non-stationary environments . Building on the opponent modelling technique called LOLA [ 15 , 14 ] , Meta-MAPG [ 21 ] extends Meta-PG [ 1 ] to multi-agent settings by considering both the non-stationarity from the environment and from the opponent ’ s learning process . Our work rather focuses on automatically discovering multi-agent auto-curricula through meta-learning , without explicit game theoretic knowledge , that leads to general MARL algorithms for solving two-player zero-sum games . 3 Multi-Agent Meta-Learning Framework . In this section , we discuss our meta-learning framework NAC for discovering multi-agent autocurricula in two-player zero-sum games . We follow the road-map set out by Fig . ( 1 ) which illustrates the flow of NAC . The relevant sections for each part of Fig . ( 1 ) are marked in pink . 3.1 The Meta-game . Consider two-player zero-sum games G2 drawn from some distribution P ( G ) , where players have the state-action space S ⇥A . We start by introducing the notion of an agent who is characterised by a policy , where a policy is a mapping : S ⇥A ! [ 0 , 1 ] which can be described in both a tabular form or as a neural network . The payoff between two agents is defined to be M ( i , j ) ( i.e. , the game engine ) , and represents the utility to agent i , or alternatively , the negative utility to agent j . Our population-based framework revolves around iterative updates on the meta-game M. At every iteration t 2 T , a Player is defined by a population of fixed agents t = 0 [ BR1 , BR 2 , ... , BR t , where 0 is the initial random agent pool and the BRt are discussed further in Sec . ( 3.3 ) . From here , 2G encapsulates all of the information for a Player to take part in a game ( e.g. , actions , reward functions ) . for the sake of notation convenience , we will only consider the single-population case where Players share the same t. As such , the single population will generate a meta-game Mt , a payoff matrix between all of the agents in the population , with individual entries being M ( i , j ) 8 i , j 2 t . 3.2 The Meta-Solver . Based on Mt , the Player will solve for the meta-distribution ⇡t which is defined as an aggregated agent over the fixed agents in t. These meta-solvers in recent work have stuck to human-designed solution concepts such as a uniform distribution [ 25 ] or NE and its variants [ 3 , 37 , 28 ] , whereas we introduce a method to actively learn these distributions through solely interacting with the game . Specifically , we parameterise our meta-solver via a neural network . This network f✓ with parameters ✓ is a mapping f✓ : Mt ! [ 0 , 1 ] t which takes as input a meta-game Mt and outputs a meta-distribution ⇡t = f✓ ( Mt ) . The output ⇡t is a probability assignment to each agent in the population t and , as we are in the single-population setting , we do not distinguish between different populations . There are two characteristics that our network f✓ should have : firstly , it should be able to handle variable-length inputs as the size of Mt is ( t ⇥ t ) , and f✓ outputs a vector of size ( t ⇥ 1 ) , where the value of t increments at every iteration . Secondly , f✓ should have both column-permutation invariance and row-permutation equivariance . Given an input Mt , f✓ will output the distribution ⇡t = f✓ ( Mt ) for the row Player and it should be equivariant to any permutation of the row index , and be invariant to any permutation of the column index as neither of these actions should affect ⇡t . Much work has been done on how to maintain permutation invariance/equivariance for neural networks such as Deepsets [ 63 ] for handling tasks defined on sets , or PointNet [ 39 ] for handling 3D point cloud data processing . Our first Multi-Layer Perceptron ( MLP ) based network is inspired by PointNet , which consists of three components : ( 1 ) An MLP over each element Mij t for a non-linear representation , ( 2 ) Column Mean-Pooling for column-permutation invariant information aggregation , which will be concatenated to each row and ( 3 ) Row-wise MLP for the row-permutation equivariant meta distribution . We also offer two alternatives , inspired by a fully Conv1d-based model [ 29 ] and a GRU-based model [ 7 ] . We refer to Appendix A for detailed architectures of our network f✓ ( · ) .

Summary: This paper presents an auto-curriculum method for solving zero-sum games. Previous methods rely on existing methods such as fictitious play, iterated best response, or double oracle to define the set of opponents (i.e., meta strategy) an agent needs to train with. The present work describes Learned Multi-Agent Auto-Curriculum (LMAC), a system that learns the meta-strategy in an end-to-end approach. Empirical results on a set of zero-sum games show that LMAC can be competitive with existing methods.