Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems

1 Introduction . Building intelligent agents in complex multi-agent games remains a long-standing challenge in artificial intelligence [ 23 ] . Recently , it has been a trend to apply multi-agent reinforcement learning ( MARL ) to extremely challenging multi-agent games , such as Dota II [ 5 ] , StarCraft II [ 29 ] and Hanabi [ 3 ] . Despite these successes , learning intelligent multi-agent policies in general still remains a great RL challenge . Multi-agent games allow sophisticated interactions between agents and environment . Feasible solutions may require non-trivial intra-agent coordination , which leads to substantially more complex strategies than the single-agent setting . Moreover , as the number of agents increases , the joint action space grows at an exponential rate , which results in an exponentially large policy search space . Thus , most existing MARL applications typically require shaped rewards , or assume simplified environment dynamics , or only handle a limit number of agents . We tackle goal-conditioned cooperative MARL problems with sparse rewards through a novel variational inference perspective . Assuming each task can be parameterized by a continuous representation , we introduce a variational proposal distribution over the task space and then decomposing the overall training objective into two separate terms , i.e. , a policy improvement term and a task proposal update term . By treating the proposal distribution updates as the training curriculum , such a variational objective naturally suggests a curriculum learning framework by alternating between curriculum update and MARL training . In addition , we propose a continuous relaxation technique for discrete variables so that the variational curricula can be also applied over the number of agents and objects in the task , which leads to a generic and unified training paradigm for the cooperative MARL setting . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We implement our variational training paradigm through a computationally efficient algorithm , Variational Automatic Curriculum Learning ( VACL ) . VACL consists two components , task expansion and entity progression , which generate a series of effective training tasks in a hierarchical manner . Intuitively , entity progression leverages the inductive bias that tasks with more entities are usually more difficult in MARL and , therefore , progressively increases the entity size in the environment . Task expansion assumes a fixed number of entities and is motivated by Stein variational gradient descent ( SVGD ) [ 17 ] to efficiently expand the task distribution towards the entire task space . We provide thorough discussions on the differences between VACL and existing works ( Sec . 3.4 ) and empirically validate VACL using a single desktop machine in four physical multi-agent cooperative games , i.e. , two in the multi-agent particle-world environment ( MPE ) [ 19 ] and two in the MuJoCobased hide-and-seek environment ( HnS ) [ 2 ] ( Sec . 4 ) . In the MPE games , VACL learns significantly faster and better than all baselines and achieves over 98 % coverage rate with 100 agents in the simple-spread testbed even using sparse rewards . In the HnS scenarios , VACL achieves over 90 % success rates on both two games , including reproducing the ramp use behavior discovered in [ 2 ] and solving a sparse-reward multi-agent Lock-and-Return challenge where none of the baselines can produce a success rate above 15 % using the same amount of training timesteps . 2 A Variational Perspective on Curriculum Learning . 2.1 Preliminary . We formulate each goal-conditioned cooperative multi-agent task as a multi-agent Markov decision process ( MDP ) , M ( n , φ ) = < n , φ , S , A , O , O , P , R , s0φ , gφ , γ > , parameterized by a discrete variable n ( e.g. , the number of agents1 ) and a continuous parameter φ ∈ Φ ( i.e. , the initial state and goal ; details in App . C.2 ) . S is the state space . A is the shared action space for each agent . O is the observation space . oi = O ( s ; i ) denotes the observation for agent i under state S. s0φ denotes the initial state and gφ denotes the goal . P ( s′|s , A ) and R ( s , A ; gφ ) are the transition probability and the goal-conditioned reward function given state s and joint actions A = ( a1 , . . . , an ) . γ is the discounted factor . We consider homogeneous agents and learn a shared goal-conditioned policy πθ ( ai|oi , g ) parameterized by θ for each agent i . The final objective is to maximize the expected accumulative reward over the entire task space , i.e. , J ( θ ) = En , φ , ati , st [ ∑ t γtR ( st , At ; gφ ) ] = En , φ [ V ( n , φ , πθ ) ] , ( 1 ) where V ( n , φ , πθ ) denotes the value function for πθ over task M ( n , φ ) . We remark that for problems with 0/1 sparse rewards , V ( n , φ , πθ ) can be also interpreted as the success rate of πθ2 . The main idea of curriculum learning is to construct a task sampler q ( n , φ ) that always generates training tasks M ( n , φ ) that yield the largest learning progress for πθ to efficiently maximize J ( θ ) . 2.2 Stein Variational Inference over a Continuous Task Space . For simplicity , we ignore the discrete variable n and only focus on the continuous variable φ in this subsection . Herein , our objective is to maximize the expectation L = Eφ∼p ( φ ) [ V ( φ , π ) ] , where p ( φ ) is a uniform distribution over feasible values of φ . The subscript θ of π is omitted for conciseness . Given the objective L , we motivate our automated curriculum learning procedure through the lens of variational inference . Let q ( φ ) denote the current distribution of training tasks , which will vary throughout learning ; we can represent curriculum learning through an objective that optimizes q ( φ ) . 1n can also be a discrete vector where each dimension represents the number of a particular type of objects . We now simply assume n is a single integer denoting the agent number for conciseness . 2This only rigorously holds when γ = 1 . But let ’ s assume it for simplicity . In practice , this statement approximately holds with a close-to-1 γ and a relatively short horizon . The following lower bound for the original objective L can be derived : L = Eφ∼p [ V ( φ , π ) ] = Eφ∼q [ p ( φ ) q ( φ ) V ( φ , π ) ] = Eφ∼q [ V ( φ , π ) + ( p ( φ ) q ( φ ) − 1 ) V ( φ , π ) ] ( 2 ) ≥ Eφ∼q ( φ ) [ V ( φ , π ) ] ︸ ︷︷ ︸ L1 : policy update +Eφ∼q ( φ ) [ V ( φ , π ) log p ( φ ) q ( φ ) ] ︸ ︷︷ ︸ L2 : curriculum update ( 3 ) where the inequality is due to x− 1 ≥ log x , with equality achieved at p ( φ ) = q ( φ ) for all φ . Now , we have decomposed L ( π , q ) into a lower bound that contains two terms L1 and L2 . L1 is simply a policy update objective under the current task curriculum q ( φ ) while the second term L2 can be interpreted as a curriculum update objective by maximizing q ( φ ) w.r.t . the policy π . Therefore , the decomposed lower bound naturally yields a curriculum learning framework by performing an iterative learning procedure , i.e. , alternating between ( 1 ) maximizing L1 with π ( policy update ) and ( 2 ) maximizing L2 with q ( φ ) ( curriculum update ) . We remark that we only optimize π w.r.t . L1 instead of L1 + L2 for the purpose of stabilizing policy learning since q can be very different from p in the early stage of training . This also aligns with the curriculum learning paradigm . In the following proposition , we show that if we can perfectly optimize the RL procedure for L1 under the curriculum q ( φ ) , then the curriculum objective L2 will encourage q ( φ ) to converge to p ( φ ) . Proposition 1 . Assume that V ( φ , π ) ∈ [ 0 , 1 ] and that there exists a unique policy π ? such that V ( φ , π ? ) = 1 for all φ . For any q ∈ P ( Φ ) \ { p } ( i.e. , any curriculum that is not the final one ) and π ∈ arg maxπ L1 ( π , q ) ( i.e. , a policy that is optimal under the current curriculum ) , L2 ( π , q ) < L2 ( π ? , p ) = 0 . Proof . First , we have that L1 ( π ? , p ) = 1 ( definition of π ? ) and L2 ( π ? , p ) = 0 . Suppose that ∃π , q such that q 6= p and L1 ( π , q ) = 1 ; then it must be the case that whenever V ( φ , π ) 6= 1 , q ( φ ) = 0 , i.e. , ∀φ ∈ supp ( q ) , V ( φ , π ) = 1 . Then L2 ( φ , π ) = −DKL ( q‖p ) < 0 , which completes the proof . In other words , if our policy π already performs well under the current curriculum q ( φ ) , then L2 will encourage q ( φ ) to become “ closer ” to p ( φ ) , aligning with our eventual target of learning all tasks . 2.2.1 Stein Variational Gradient Descent for Curriculum Update . The most straightforward approach for optimizing L2 is to learn a generative model w.r.t . the current π , which can be particularly sample-inefficient . Instead , we consider updates to our current curriculum via smooth updates , which is much more efficient than learning generative models . Let us represent the task distribution under the current curriculum q ( φ ) with a set of particles Q = { φ } . To update each particle φ in Q , we consider an incremental transform T ( φ ) = φ+ f ( φ ) , where f ( φ ) is a smooth function and is a magnitude scalar . Our goal is then to derive suitable updates toQ that maximizes L2 ( q , π ) based on the transform T . Following [ 9 ] and [ 27 ] , we introduce an assumption over the generalization ability of the task space . Assumption 1 . An optimal policy π that is trained over a specific task φ has some generalization ability to another goal φ′ close to φ . Specifically , if V ( φ , π ) = c for any φ ∈ supp ( q ) , then V ( φ′ , π ) = c for all φ′ ∈ B ( φ ) , where B ( φ ) is a small open ball around φ . In the context of curriculum learning , this assumption suggests that for any policy that is trained on a set of tasks , the performance ( e.g. , success rate ) does not change if we only change the task parameters by an infinitesimal amount . In the following theorem , we derive the functional gradient of the curriculum update objective L2 in a reproducing kernel Hilbert space ( RKHS ) . Theorem 1 . Let T ( φ ) = φ+ f ( φ ) where f is an element of some vector-valued RKHS of a positive definite kernel k ( φ , φ′ ) : Φ× Φ→ R , and q [ T ] the density of ψ = T ( φ ) when φ ∼ q , then ∇fL2 ( q [ T ] , π ) |f=0 = f∗ ( φ ) , ( 4 ) where f∗ ( · ) = Eφ′∈Q [ V ( φ′ , π ) ( k ( φ′ , · ) ∇φ′ log p ( φ′ ) +∇φ′k ( φ′ , · ) ) ] . Proof is in App . A . While the above statement is very similar to that used for Stein variational gradient descent [ 17 ] , the update for our case also directly depends on the value V ( φ′ , π ) , and thus indirectly depends on the current policy . Intuitively , if V ( φ′ , π ) is large , then φ′ will contribute more to the curriculum update , whereas if V ( φ′ , π ) = 0 , then φ′ will not contribute to the update . Since we assume that p ( φ ) is uniform over the task space , ∇φ′ log p ( φ′ ) = 0 for the interior of Φ , and the task update becomes f∗ ( · ) = Eφ′∈Q [ V ( φ′ , π ) · ∇φ′k ( φ′ , · ) ] . ( 5 ) For example , we take the RBF kernel k ( φ , φ′ ) = exp ( − 1h‖φ− φ ′‖22 ) , and then Eq . ( 5 ) will drive φ away from neighbouring points that have large k ( φ , φ′ ) , and the repelling force scales with V ( φ′ , π ) . If V ( φ′ , π ) = 0 , i.e. , φ′ has not been solved at all , then there is no repelling force from φ′ to φ . Intuitively , this process encourages the curriculum update to prioritize tasks that have not been fully solved yet , by exploring in the vicinity of existing tasks with high values .

Summary: This paper proposed a variational automatic curriculum learning framework for solving goal-conditioned cooperative multiagent reinforcement learning problems. In this framework, the learning objective is decomposed into two parts: policy learning on current task distribution, and curriculum update to a new task distribution. The curricula are trained based on two components, including task configuration and the number of agents. The experimental results demonstrate the proposed method can achieve better performance than a few baselines with light weight computation resources.