Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems

1 Introduction . Building intelligent agents in complex multi-agent games remains a long-standing challenge in artificial intelligence [ 23 ] . Recently , it has been a trend to apply multi-agent reinforcement learning ( MARL ) to extremely challenging multi-agent games , such as Dota II [ 5 ] , StarCraft II [ 29 ] and Hanabi [ 3 ] . Despite these successes , learning intelligent multi-agent policies in general still remains a great RL challenge . Multi-agent games allow sophisticated interactions between agents and environment . Feasible solutions may require non-trivial intra-agent coordination , which leads to substantially more complex strategies than the single-agent setting . Moreover , as the number of agents increases , the joint action space grows at an exponential rate , which results in an exponentially large policy search space . Thus , most existing MARL applications typically require shaped rewards , or assume simplified environment dynamics , or only handle a limit number of agents . We tackle goal-conditioned cooperative MARL problems with sparse rewards through a novel variational inference perspective . Assuming each task can be parameterized by a continuous representation , we introduce a variational proposal distribution over the task space and then decomposing the overall training objective into two separate terms , i.e. , a policy improvement term and a task proposal update term . By treating the proposal distribution updates as the training curriculum , such a variational objective naturally suggests a curriculum learning framework by alternating between curriculum update and MARL training . In addition , we propose a continuous relaxation technique for discrete variables so that the variational curricula can be also applied over the number of agents and objects in the task , which leads to a generic and unified training paradigm for the cooperative MARL setting . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We implement our variational training paradigm through a computationally efficient algorithm , Variational Automatic Curriculum Learning ( VACL ) . VACL consists two components , task expansion and entity progression , which generate a series of effective training tasks in a hierarchical manner . Intuitively , entity progression leverages the inductive bias that tasks with more entities are usually more difficult in MARL and , therefore , progressively increases the entity size in the environment . Task expansion assumes a fixed number of entities and is motivated by Stein variational gradient descent ( SVGD ) [ 17 ] to efficiently expand the task distribution towards the entire task space . We provide thorough discussions on the differences between VACL and existing works ( Sec . 3.4 ) and empirically validate VACL using a single desktop machine in four physical multi-agent cooperative games , i.e. , two in the multi-agent particle-world environment ( MPE ) [ 19 ] and two in the MuJoCobased hide-and-seek environment ( HnS ) [ 2 ] ( Sec . 4 ) . In the MPE games , VACL learns significantly faster and better than all baselines and achieves over 98 % coverage rate with 100 agents in the simple-spread testbed even using sparse rewards . In the HnS scenarios , VACL achieves over 90 % success rates on both two games , including reproducing the ramp use behavior discovered in [ 2 ] and solving a sparse-reward multi-agent Lock-and-Return challenge where none of the baselines can produce a success rate above 15 % using the same amount of training timesteps . 2 A Variational Perspective on Curriculum Learning . 2.1 Preliminary . We formulate each goal-conditioned cooperative multi-agent task as a multi-agent Markov decision process ( MDP ) , M ( n , φ ) = < n , φ , S , A , O , O , P , R , s0φ , gφ , γ > , parameterized by a discrete variable n ( e.g. , the number of agents1 ) and a continuous parameter φ ∈ Φ ( i.e. , the initial state and goal ; details in App . C.2 ) . S is the state space . A is the shared action space for each agent . O is the observation space . oi = O ( s ; i ) denotes the observation for agent i under state S. s0φ denotes the initial state and gφ denotes the goal . P ( s′|s , A ) and R ( s , A ; gφ ) are the transition probability and the goal-conditioned reward function given state s and joint actions A = ( a1 , . . . , an ) . γ is the discounted factor . We consider homogeneous agents and learn a shared goal-conditioned policy πθ ( ai|oi , g ) parameterized by θ for each agent i . The final objective is to maximize the expected accumulative reward over the entire task space , i.e. , J ( θ ) = En , φ , ati , st [ ∑ t γtR ( st , At ; gφ ) ] = En , φ [ V ( n , φ , πθ ) ] , ( 1 ) where V ( n , φ , πθ ) denotes the value function for πθ over task M ( n , φ ) . We remark that for problems with 0/1 sparse rewards , V ( n , φ , πθ ) can be also interpreted as the success rate of πθ2 . The main idea of curriculum learning is to construct a task sampler q ( n , φ ) that always generates training tasks M ( n , φ ) that yield the largest learning progress for πθ to efficiently maximize J ( θ ) . 2.2 Stein Variational Inference over a Continuous Task Space . For simplicity , we ignore the discrete variable n and only focus on the continuous variable φ in this subsection . Herein , our objective is to maximize the expectation L = Eφ∼p ( φ ) [ V ( φ , π ) ] , where p ( φ ) is a uniform distribution over feasible values of φ . The subscript θ of π is omitted for conciseness . Given the objective L , we motivate our automated curriculum learning procedure through the lens of variational inference . Let q ( φ ) denote the current distribution of training tasks , which will vary throughout learning ; we can represent curriculum learning through an objective that optimizes q ( φ ) . 1n can also be a discrete vector where each dimension represents the number of a particular type of objects . We now simply assume n is a single integer denoting the agent number for conciseness . 2This only rigorously holds when γ = 1 . But let ’ s assume it for simplicity . In practice , this statement approximately holds with a close-to-1 γ and a relatively short horizon . The following lower bound for the original objective L can be derived : L = Eφ∼p [ V ( φ , π ) ] = Eφ∼q [ p ( φ ) q ( φ ) V ( φ , π ) ] = Eφ∼q [ V ( φ , π ) + ( p ( φ ) q ( φ ) − 1 ) V ( φ , π ) ] ( 2 ) ≥ Eφ∼q ( φ ) [ V ( φ , π ) ] ︸ ︷︷ ︸ L1 : policy update +Eφ∼q ( φ ) [ V ( φ , π ) log p ( φ ) q ( φ ) ] ︸ ︷︷ ︸ L2 : curriculum update ( 3 ) where the inequality is due to x− 1 ≥ log x , with equality achieved at p ( φ ) = q ( φ ) for all φ . Now , we have decomposed L ( π , q ) into a lower bound that contains two terms L1 and L2 . L1 is simply a policy update objective under the current task curriculum q ( φ ) while the second term L2 can be interpreted as a curriculum update objective by maximizing q ( φ ) w.r.t . the policy π . Therefore , the decomposed lower bound naturally yields a curriculum learning framework by performing an iterative learning procedure , i.e. , alternating between ( 1 ) maximizing L1 with π ( policy update ) and ( 2 ) maximizing L2 with q ( φ ) ( curriculum update ) . We remark that we only optimize π w.r.t . L1 instead of L1 + L2 for the purpose of stabilizing policy learning since q can be very different from p in the early stage of training . This also aligns with the curriculum learning paradigm . In the following proposition , we show that if we can perfectly optimize the RL procedure for L1 under the curriculum q ( φ ) , then the curriculum objective L2 will encourage q ( φ ) to converge to p ( φ ) . Proposition 1 . Assume that V ( φ , π ) ∈ [ 0 , 1 ] and that there exists a unique policy π ? such that V ( φ , π ? ) = 1 for all φ . For any q ∈ P ( Φ ) \ { p } ( i.e. , any curriculum that is not the final one ) and π ∈ arg maxπ L1 ( π , q ) ( i.e. , a policy that is optimal under the current curriculum ) , L2 ( π , q ) < L2 ( π ? , p ) = 0 . Proof . First , we have that L1 ( π ? , p ) = 1 ( definition of π ? ) and L2 ( π ? , p ) = 0 . Suppose that ∃π , q such that q 6= p and L1 ( π , q ) = 1 ; then it must be the case that whenever V ( φ , π ) 6= 1 , q ( φ ) = 0 , i.e. , ∀φ ∈ supp ( q ) , V ( φ , π ) = 1 . Then L2 ( φ , π ) = −DKL ( q‖p ) < 0 , which completes the proof . In other words , if our policy π already performs well under the current curriculum q ( φ ) , then L2 will encourage q ( φ ) to become “ closer ” to p ( φ ) , aligning with our eventual target of learning all tasks . 2.2.1 Stein Variational Gradient Descent for Curriculum Update . The most straightforward approach for optimizing L2 is to learn a generative model w.r.t . the current π , which can be particularly sample-inefficient . Instead , we consider updates to our current curriculum via smooth updates , which is much more efficient than learning generative models . Let us represent the task distribution under the current curriculum q ( φ ) with a set of particles Q = { φ } . To update each particle φ in Q , we consider an incremental transform T ( φ ) = φ+ f ( φ ) , where f ( φ ) is a smooth function and is a magnitude scalar . Our goal is then to derive suitable updates toQ that maximizes L2 ( q , π ) based on the transform T . Following [ 9 ] and [ 27 ] , we introduce an assumption over the generalization ability of the task space . Assumption 1 . An optimal policy π that is trained over a specific task φ has some generalization ability to another goal φ′ close to φ . Specifically , if V ( φ , π ) = c for any φ ∈ supp ( q ) , then V ( φ′ , π ) = c for all φ′ ∈ B ( φ ) , where B ( φ ) is a small open ball around φ . In the context of curriculum learning , this assumption suggests that for any policy that is trained on a set of tasks , the performance ( e.g. , success rate ) does not change if we only change the task parameters by an infinitesimal amount . In the following theorem , we derive the functional gradient of the curriculum update objective L2 in a reproducing kernel Hilbert space ( RKHS ) . Theorem 1 . Let T ( φ ) = φ+ f ( φ ) where f is an element of some vector-valued RKHS of a positive definite kernel k ( φ , φ′ ) : Φ× Φ→ R , and q [ T ] the density of ψ = T ( φ ) when φ ∼ q , then ∇fL2 ( q [ T ] , π ) |f=0 = f∗ ( φ ) , ( 4 ) where f∗ ( · ) = Eφ′∈Q [ V ( φ′ , π ) ( k ( φ′ , · ) ∇φ′ log p ( φ′ ) +∇φ′k ( φ′ , · ) ) ] . Proof is in App . A . While the above statement is very similar to that used for Stein variational gradient descent [ 17 ] , the update for our case also directly depends on the value V ( φ′ , π ) , and thus indirectly depends on the current policy . Intuitively , if V ( φ′ , π ) is large , then φ′ will contribute more to the curriculum update , whereas if V ( φ′ , π ) = 0 , then φ′ will not contribute to the update . Since we assume that p ( φ ) is uniform over the task space , ∇φ′ log p ( φ′ ) = 0 for the interior of Φ , and the task update becomes f∗ ( · ) = Eφ′∈Q [ V ( φ′ , π ) · ∇φ′k ( φ′ , · ) ] . ( 5 ) For example , we take the RBF kernel k ( φ , φ′ ) = exp ( − 1h‖φ− φ ′‖22 ) , and then Eq . ( 5 ) will drive φ away from neighbouring points that have large k ( φ , φ′ ) , and the repelling force scales with V ( φ′ , π ) . If V ( φ′ , π ) = 0 , i.e. , φ′ has not been solved at all , then there is no repelling force from φ′ to φ . Intuitively , this process encourages the curriculum update to prioritize tasks that have not been fully solved yet , by exploring in the vicinity of existing tasks with high values .

Summary: The paper proposes a curriculum learning method for cooperative multi-agent reinforcement learning in sparse reward tasks. The authors derive a curriculum learning approach by forming a lower bound on the expected reward under a target distribution p(\phi). This lower bound is maximized by alternatingly optimizing the policy under a variational distribution q(\phi) and then adjusting the variational distribution based on the current performance of the agent. The authors investigate the theoretical behavior showing that the optimization of q(\phi) minimizes a form of "weighted" KL-Divergence to p(\phi). Further, they derive a gradient-based approach to optimize q(\phi) and propose an approximate implementation of their approach that performs empirically well in different multi-agent RL benchmarks.