Adaptable Agent Populations via a Generative Model of Policies

1 Introduction . Quick thought experiment : imagine our world was such that all people acted , thought , and looked exactly the same in every situation . Would we ever have found the influential dissenters that sparked scientific , political , and cultural revolutions ? In reinforcement learning ( RL ) , it is common to learn a single policy that fits an environment . However , it is often desirable to instead find an entire array of high performing policies . To this end , we propose learning a generative model of policies . At a high level , we aim to show that purposefully learning a diverse policy space for a given environment can be competitive to learning a single policy , while better encompassing a range of skillful behaviors that are adaptable and robust to changes in the task and environment . We name our method of learning a space of adaptable agent polices : ADAP . Why should we bother with finding more than one policy per environment ? We propose two primary reasons . First , RL environments are continually approaching greater levels of open-endedness and complexity . For a given environment , there might be an entire manifold of valid and near-equally high performing strategies . By finding points across this manifold , we avoid ‘ having all eggs in one basket , ’ granting robustness and adaptability to environmental changes . In the event of a change , we are able to adapt our generated population to select individuals that can still survive given the ablation , much like natural selection drives evolution in the real world . Secondly , using a generative model of policies as a population of agents makes intuitive sense in multi-agent environments , in which different agents should have the capacity to act like they are unique individuals . However , it is common in many multi-agent reinforcement learning settings to deploy the same policy across all agents , such that they are essentially distributed clones . Doing so may reduce the multi-modality of the agent population , resulting in a single ‘ average ’ agent . Previous work has touched on ideas akin to a generative model of policies . In hierarchical RL , the high-level policy controller can be considered a generator of sub-policies that are ‘ options ’ [ 1 , 2 , 3 ] . But these methods are designed to find decomposable skills that aid in the construction of just one 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . downstream controller policy . A core idea of our work is that of quality diversity [ 4 ] , which aims to optimize a population of agents along the axes of both reward and diversity . Traditional methods often use evolutionary search over a discrete-sized population of separate agents , each with their own policy parameters . This consumes more time and training resources , and limits the number of potential behaviors . Our work integrates the goals of quality diversity into time and memory efficient deep RL by simulating an entire population of agents via a generative model of policies , with diversity bounded only by capacity of the generator . The rest of the paper is organized as follows . First we introduce our generative model of policies and the diversity objective that guides its learning . Next , we explore the potentials of learning a population of agents by ablating environments and then searching for suitable policies , directly in latent space . We primarily study two environments : Markov Soccer [ 5 ] and Farmworld . Farmworld is a new environment we have developed for testing diversity in a multi-agent , open-ended gridworld . At the website linked in the abstract , one can find qualitative results of experiments presented in this paper , as well as additional results on toy environments of CartPole [ 6 ] and a standard multi-goal environment . 2 Method . Let Z be a sample space of n dimensional vectors , and Z be a random variable defined uniformly over Z . Then , we learn a mapping , G : ϕ , Z , from generator weights ϕ and latent distribution Z to a space of policies Π . The generator Gϕ itself is not a policy . It must be conditioned on a draw z ∼ Z in order to define a learned set of behaviors . In this sense , z is a stochastic parameter of Gϕ , and is sampled once at the beginning of each agent episode . In our experiments , Z is the sample space of all three dimensional vectors with magnitude one ( i.e . the surface of the unit sphere ) . Practically , we use the low dimension of three , so that we can perform a key subject of this paper : rapid optimization , or adaptation , of G by changing Z rather than ϕ ( fine tuning ϕ would be more typical in literature ) . We require magnitude one so that there is at least one non-zero element for any z ∼ Z , which we found important for providing signal and stability in the training of G. It is possible that with higher dimensions , this stipulation could be relaxed . Diversity Regularization In order to learn a diverse space of unique policies , we introduce a diversity regularization objective . Since policies define a space of actions taken over different states , we propose that in order for two policies to be distinct , they must have different action distributions given the same state . To this end , we define the objective Ldiv ( 1 ) : Ldiv ( ϕ ) = E s∈S [ E zi , zj∼Z exp ( −DKL ( πϕ , zi ; b ( s ) ∥πϕ , zj ; b ( s ) ) ) ] ( 1 ) in which DKL is the KL-divergence between the two policy action distributions πϕ , zi and πϕ , zj , and b is a smoothing constant over the action distributions . Optimization of G In our experiments , we optimize the diversity objective in an online fashion using gradient descent , in conjunction with a PPO [ 7 ] clipped-surrogate objective and an entropy regularization objective . Our full optimization problem is max ϕ LPPO ( ϕ ) − αLdiv ( ϕ ) where LPPO is Equation 9 in [ 7 ] and α is a coefficient to scale the diversity regularization objective . See Algorithm 1 in the supplement for additional details . Adaptation via Optimization in the Latent Space of G By learning an entire space of policies Π , we are able to search our policy space for the highest performing policy , whether dealing with the training environment or an ablated future environment . In contrast to searching over policy parameters through transfer learning or fine-tuning , we are able to quickly search over the low-dimensional latent space ( dimensionality 3 in our experiments ) . In fact , we can quickly adapt back and forth to various situations : the search procedure often takes less than 30 seconds , or 100 episode rollouts , to find any high quality solutions that exist . Over the course of a small number of generations , we evaluate randomly sampled latents , and keep higher performing ones with greater probability . In the event that episodes have a high degree of variablility per run – such as in the Markov Soccer environment – it may be necessary to run several episodes per latent vector and average the returns . Details can be found in Algorithm 2 of the supplement . Model Architecture Similarly to prior work [ 3 ] , we have found that richer integrations between the latent vector and the observation can yield a more multi-modal policy space . To induce this richer integration , we introduce a multiplicative model denoted `` ( x ) '' for latent integration , and compare the results to a baseline of concatenating `` ( + ) '' the latent sample to the observation . We describe this architecture in the supplement . 3 Related Work . Quality Diversity The evolutionary computing community has developed various quality diversity ( QD ) algorithms that aim to find a balance of novel and high-performing individuals within a population . Some methods can even be considered policy generators : NEAT and HyperNEAT [ 8 , 9 ] use an indirect encoding to construct a network architecture . To encourage diversity , these methods use an idea known as fitness sharing : if genotypes are too similar , then they will split reward . While NEAT and HyperNEAT encourage diversity of parameters , other methods encourage diversity of behavior . Novelty Search ( NS ) [ 10 ] learns individuals that have high novelty along some user defined behavioral distance metric . For example , in a maze navigation task , the behavioral characteristic could be the final resting location of the individual , and agents are selected based on how far away they end up from an archive of past individuals . Unfortunately , as shown in [ 11 ] , the choice of this characteristic can critical , and domain dependent . Additionally , NS focuses mainly on finding novel solutions , and ignores fitness , or reward . NS with Local Competition [ 12 ] and MapElites [ 13 ] aim to solve this problem by selecting for individuals with high fitness , but only against individuals in the same phenotypic or genotypic region , respectively . There are several prior and concurrent works that aim to connect ideas of quality diversity with deep reinforcement learning . Like quality diversity algorithms , these methods optimize a fixedsize population or archive of policies to be distinct from each other . [ 14 , 15 ] aim to find a set of policies that yield diverse trajectories . [ 15 ] in particular focuses on the application to multi-agent environments and zero-shot coordination . [ 16 ] uses a KL-divergence over policies ; but a policy ’ s diversity is optimized over previous SGD updates of itself , thus limiting the potential multi-modality of solutions . [ 17 ] optimizes for diversity of the total population via maximizing the determinant of a population distance matrix , but works best only with small populations of size three or five . [ 18 ] uses a method reminiscent of DIAYN , but introduces ideas to balance quality with diversity . It is especially similar to ADAP in optimizing the latent space to achieve robustness , but only searches over a fixed-size set of latent vectors and focuses on single-agent environments . Other methods have explored indirectly influencing diversity via differing training hyperparameters as in Population-Based Training [ 19 ] , or using reward randomization as in [ 20 ] . Importantly , both classical QD algorithms [ 10 , 12 , 13 ] and most deep RL methods [ 14 , 15 , 16 , 17 , 19 , 20 ] use sets of distinct agent parameters to learn a diverse population . ADAP makes the connection that we can encode unique policies into a latent space ( an idea that also appears in a few recent works [ 2 , 3 , 21 , 18 ] ) , and frames learning a diverse population as a generative modelling problem . Additionally , in distinction from classical QD methods that use a non-differential genetic algorithm or evolutionary search for optimization , ADAP is able to directly optimize for diversity and policy credit assignment via gradient descent . Option Discovery for Hierarchical RL The option framework introduced by [ 1 ] could be thought of as learning a generator of skills , which are temporal abstractions over actions that can be used by a downstream , higher-level controller . Recent works like DIAYN [ 2 ] and others [ 3 , 21 ] in option discovery learn a fixed set of diverse skills that are discriminable by observed state or trajectory : such as learning to move left , or move right . These skills are generally not meant to be the final agent policy , DIAYN even learns skills without any extrinsic environmental reward . However , these methods are most similar to ADAP in terms of mapping a latent sample to final agent policies . Goal-Conditioned Reinforcement Learning Yet another way to induce diverse policy behaviors is through using goal-conditioned policies [ 22 , 23 , 24 ] that use a family of task-defined value or Q functions or expert trajectories [ 25 ] to incentivize diversity . These methods require structure in how to define diversity , such as defining a value function family over states [ 24 ] . Multi-Agent Roles Recent works generate specialized agent policies in a multi-agent setting , building on QMIX [ 26 ] . ROMA [ 27 ] learns agent roles that are not static through agent trajectories , require optimizing several additional objectives , and are learned jointly with other roles via a joint action-value function . Similarly , MAVEN [ 28 ] optimizes the mutual information between joint agent actions and a latent variable . While a single latent sample in ADAP encodes a single agent ‘ species ‘ , a latent sample in these works encode how a group of agents should behave together : thus we can not employ adaptation based on individual selection . 4 Introduction to Farmworld We test our learning G in a new open-ended grid-world environment called Farmworld , that supports multi-agent interaction and partially observable observations . The idea behind Farmworld is simple : agents move about the map to gather food from various resources , such as chickens and towers that spawn in random locations . In out experiments , agents only optimize their own reward : a single agent gets exactly 0.1 reward for each timestep it is alive . Thus , lifetime is directly proportional to reward . Agents can live longer by attacking other agents , chickens , and towers : for example , a chickens might take two timesteps of sword hits to yield five timesteps worth of health . To avoid cannibalism in our experiments , we set agents to gain zero health from other agents . Of course , these numbers are configurable to achieve different environment dynamics . Furthermore , Farmworld is a partially-observable environment : agents see only what is in a certain tile radius from their location . In our experiments , the observation is a vector representation of the units and tiles . Additional details of the Farmworld are provided in the supplement .

Summary: This paper proposes to train a generative model for entire populations of maximally diverse agents, from which one specific individual policy can quickly be selected at deployment time through a fast search process.  Policies are represented as networks augmented with a low-dimensional latent variable z, randomly sampled at agent initialization. Thus each trained network is actually a generative model, from which an infinity of policies can be generated by sampling over z.   Crucially, the training procedure encourages the fixed weights of the network to not only obtain good performance for any given random z, but also to produce maximally different policies for any two different z. Thus the method is a quality diversity (QD) method (though the authors seem ambiguous on the subject).  This method is compared with a previously published method (DIAYN), and with itself but without the diversity objective, and is found superior to both on various gridworld tasks, both in terms of expected performance and diversity of generated agents.