Adaptable Agent Populations via a Generative Model of Policies
Authors: Kenneth Derek
Phillip Isola
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
quality diversity
multiagent
generative models
latent policy

Abstract: In the natural world, life has found innumerable ways to survive and often thrive. Between and even within species, each individual is in some manner unique, and this diversity lends adaptability and robustness to life. In this work, we aim to learn a space of diverse and high-reward policies in a given environment. To this end, we introduce a generative model of policies for reinforcement learning, which maps a low-dimensional latent space to an agent policy space. Our method enables learning an entire population of agent policies, without requiring the use of separate policy parameters. Just as real world populations can adapt and evolve via natural selection, our method is able to adapt to changes in our environment solely by selecting for policies in latent space. We test our generative model’s capabilities in a variety of environments, including an open-ended grid-world and a two-player soccer environment. Code, visualizations, and additional experiments can be found at https://kennyderek.github.io/adap/.

Adaptable Agent Populations via a Generative Model of Policies
Kenneth Derek
Phillip Isola
MIT CSAIL kderek@a lum, mit. edu
MIT CSAIL phillipi@mit.ed
Abstract
In the natural world, life has found innumerable ways to survive and often thrive. Between and even within species, each individual is in some manner unique, and this diversity lends adaptability and robustness life. In this work, aim learn a space of diverse and high-reward policies end, we introduce a generative model policies for reinforcement learning, which maps a low-dimensional latent space to an agent policy space. Our method enables learning entire population of agent policies, without requiring the use of separate policy parameters. Just as real world populations can adapt and evolve via natural selection, our method is able to adapt to changes our environment solely by selecting for policies in latent space. We test our generative model's capabilities in variety of environments, including open-ended grid-world and two-player soccer environment. Code, visualizations, and additional experiments can be found at htttps://kenndderrr thub. /
given environment. To this
Introduction
Quick thought experiment: imagine our world was such that all people acted, thought, and looked exactly the same in every situation. Would we ever have found the influential dissenters that sparked scientific, political, and cultural revolutions?
In reinforcement learning (RL), it common to learn a single policy that fits an environment. However, it is often desirable to instead find an entire array of high performing policies. To this end, we propose learning a generative model of policies. At a high level, we aim to show that purposefully learning diverse policy space for given environment can be competitive learning single policy, while better encompassing a range of skillful behaviors that are adaptable and robust to changes the task and environment. We
our method of learning space of adaptable agent polices: ADAP.
Why should we bother with finding more than one policy per environment? We propose two primary reasons. First, RL environments are continually approaching greater levels of open-endedness and complexity. For a given environment, there might be an entire manifold of valid and near-equally high performing strategies. By finding points across this manifold, avoid 'having all eggs in one basket,' granting robustness and adaptability to environmental changes. the event of a change, able adapt our generated population to select individuals that can still survive given the ablation, much like natural selection drives evolution in the real world. Secondly, using a generative model of policies which different agents should have the capacity to act like they are unique individuals. However, is common in many multi-agent reinforcement learning settings to deploy the same policy across all agents, such that they are essentially distributed clones. Doing so may reduce the multi-modality of the agent population, resulting single 'average' agent.
population of agents makes intuitive sense in multi-agent environments, in
Previous work has touched on ideas akin to generative model of policies. In hierarchical RL, the high-level policy controller be considered generator of sub-pol sub-policies that are 'options' [ 2 B. But these methods are designed to find decomposable skills that aid in the construction of just one
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

downstream controller policy. A core idea of our work is that of quality diversity 4 which aims optimize population of agents along the axes of both reward and diversity. Traditional methods often use evolutionary search over discrete- sized population of separate agents, each with their own policy parameters. This consumes more time and training resourees, and limits the number of potential behaviors. Our work integrates the goals of quality diversity into time and memory efficient deep RL by simulating an entire population of agents via a generative model policies, with diversity bounded only by capacity of the generator.
The rest of the paper is organized as follows. First we introduce our generative model of policies and the diversity objective that guides its learning, Next, we explore the potentials of learning population of agents by ablating environments and then searching for suitable policies, directly in latent space. We primarily study two environments: Markov Soccer [5] and Farmworld. Farmworld a new environment we have developed for testing diversity in multi-agent, open-ended gridworld. At the website linked in the abstract, one can find qualitative results of experiments presented in this paper, as well as additional results on toy environments of CartPole 6 and a standard multi-gu -gal environment.
2
Method
Let 2 be sample space of dimensional vectors, and Z be random variable defined uniformly over 2. Then, we learn a mapping, G :0 Z, from generator weights Φ and latent distribution Z to space of policies IT.
The generator Ge itself is not a policy. must be conditioned on a draw Z in order to define a learned set of behaviors. In this sense, is stochastic parameter of Gş. and is sampled once the beginning of each agent episode.
In our experiments, Z is the sample space of all three dimensional vectors with magnitude one (i.e. the surface of the unit sphere). Practically, we use the low dimension three, so that we can perform key subject of this paper: rapid optimization, or adaptation, of G by changing Z rather than & (fine tuning would more typieal in literature). We require magnitude one so that there is at least one non-zero element for any Z, which we found important for providing signal and stability in the training of G. It is possible that with higher dimensions, this stipulation could be relaxed.
Regularizer pee
z- Z
G
π
Φ
D(T 4
O.zy
Environment
PPO Surrogate Loss
Figure 1: Method Diagram. Upon agent initialization, we sample latent 2; ~ Z which, along with 6. defines an agent policy for the episode. We update G by optimizing two objectives. The PPO surrogate loss is optimized in an online manner, using the trajectory from the generated policy To.a; Meanwhile, the diversity regularizer loss is optimized over independently sampled policy pairs
To.ay: Th.z
Diversity Regularization In order to learn diverse space of unique policies, introduce diversity regularization objective. Since policies define a space of actions taken over different states, propose that in order for policies to be distinet, they must have different action distributions given the same state. To this end, we define the objective Lidiv TD:
exp (-D (T.. (5) (.)))
Liio
SES
in which DKL is the KL-divergence between the two policy action distributions TØ.., and Tolays and b is a smoothing constant over the action distributions.
Optimization of G In our experiments, we optimize the diversity objective in an online fashion using gradient descent, in conjunction with a PPO 7 clipped-surrogate objective and an entropy regularization objective. Our full optimization problem is

---

max Lpro( - L. (
where LPPO is Equation 9 in 7 and is coefficient to scale the diversity regularization objective. See Algorithm in the supplement for additional details.
Adaptation via Optimization in the Latent Space of G' By learning entire space of policies II, we are able to search our policy space for the highest performing policy, whether dealing with the training environment
ablated future environment.
In contrast to searching over policy parameters through transfer learning or fine-tuning, we are able to quickly search over the low-dimensional latent space (dimensionality in our experiments). In fact, we can quickly adapt back and forth to various situations: the search procedure often takes less than 30 seconds, or 100 episode rollouts, to find any high quality solutions that exist. Over the course of a small number generations, we evaluate randomly sampled latents, and keep higher performing ones with greater probability. In the event that episodes have a high degree of variablility per run such as in the Markov Soccer environment it may be necessary to run several episodes per latent vector and average the returns. Details can be found in Algorithm 2 of the supplement.
Model Architecture Similarly to prior work 3 we have found that richer integrations between the latent vector and the observation can yield more multi-modal policy space. To induce this richer integration, we introduce a multiplicative model denoted "(x)" for latent integration, and compare the results baseline of concatenating "(+)" the latent sample to the observation. We describe this architecture in the supplement.
3 Related Work
The evolutionary computing community has developed various quality diversity (QD) algorithms that aim to find a balance of novel and high-performing individuals within population. Some methods can even be considered policy generators: NEAT and HyperNEAT (8 use an indirect encoding to construct network architecture. To encourage diversity, these methods use an idea known as fitness sharing: if genotypes are too similar, then they will split reward.
Quality Diversity
While NEAT and HyperNEAT encourage diversity of parameters, other methods encourage diversity of behavior. Novelty Search (NS) [10] learns individuals that have high novelty along some user defined behavioral distance metric. For example, in maze navigation task, the behavioral character- istic could the final resting location of the individual, and agents are selected based on how far away they end up from an archive of past individuals. Unfortunately, shown in [1I] the choice of this characteristic can critical, and domain dependent. Additionally, NS focuses mainly on finding novel solutions, and ignores fitness, reward. NS with Local Competition [12] and MapElites [I3 aim to solve this problem by selecting for individuals with high fitness, but only against individuals in the same phenotypic genotypic region, respectively.
There are several prior and concurrent works that aim to connect ideas of quality diversity with deep reinforcement learning. Like quality diversity algorithms, these methods optimize fixed- size population or archive of policies to be distinct from each other. [14 [5] aim find set of policies that yield diverse trajectories.
[15] in particular focuses on the application to multi-agent environments and zero-shot coordination. 16 uses a KL.-divergence over policies; but policy's diversity is optimized over previous SGD updates of itself, thus limiting the potential multi-modality of solutions. [17] optimizes for diversity of the total population via maximizing the determinant of population distance matrix, but works best only with small populations of size three five. [18] uses a method reminiscent of DIAYN, but introduces ideas to balance quality with diversity. It is especially similar to ADAP in optimizing the latent space to achieve robustness, but only searches over a fixed-size set of latent vectors and focuses on single-agent environments. Other methods have explored indirectly influencing diversity via differing training hyperparameters as in Population Based Training 19). or using reward randomization as in (00
Importantly, both classical QD algorithms 10 12 13
and most deep RL methods 14
15
16
17
19
20 use sets of distinct agent parameters to learn a diverse population. ADAP makes connection that encode unique policies into a latent space (an idea that also appears in few recent
3

---

works 2 2, 18]
and frames learning diverse population as a generative modelling problem. Additionally, in distinction from classical QD methods that use con-differential genetic algorithm or evolutionary search for optimization, ADAP is able to directly optimize for diversity and policy credit assignment via gradient descent.
Option Discovery for Hierarchical RL The option framework introduced by [ could be thought of as learning generator of skills, which are temporal abstractions over actions that can be used by downstream, higher-level controller. Recent works like DIAYN [ and others [3] 21 in option discovery learn a fixed set diverse skills that are discriminable by observed state or trajectory: such as learning to move left, move right. These skills generally not meant to be the final agent policy, DIAYN even learns skills without any extrinsic environmental reward. However, these methods are most similar to ADAP in terms of mapping a latent sample final agent policies.
Yet another way to induce diverse policy behaviors 23 24 that use family of task-defined value Q functions or expert trajectories [25] to incentivize diversity. These methods require structure in how define diversity, such as defining value function family over states (24
Con--Conditioned Reinforcement Learning through using soal-conditioned policies [22
Multi-Agent Roles Recent works generate specialized agent policies in a multi-agen setting, building on QMIX [26]. ROMA (77 learns agent roles that are not static through agent trajectories, require optimizing several additional objectives, and are learned jointly with other roles via a joint action-value function. Similarly, MAVEN (28] optimizes the mutual information between joint agent actions and latent variable. While a single latent sample in ADAP encodes a single agent 'species', a latent sample these works encode how a group of agents should behave together: thus we cannot employ adaptation based on individual selection.
4 Introduction to Farmworld
We test our learning G in new open-ended grid-world environment called Farmworld, that supports multi-agent interaction and partially observable observations. The idea behind Farmworld is simple: agents move about the map to gather food from various resources, such as chickens and towers that spawn in random locations. In out experiments, agents only optimize their own reward: a single agent gets exactly 0.1 reward for each timestep it is alive. Thus, lifetime is directly proportional to reward. Agents can live longer by attacking other agents, chickens, and towers: for example, chickens might take two timesteps of sword hits to yield five timesteps worth health. To avoid cannibalism in our experiments, we set agents to gain zero health from other agents. Of course, these numbers are configurable to achieve different environment dynamics.
Figure 2: Standard
Farmworld Training En vironment
Furthermore, Farmworld is a tile radius from their location. In our experiments, the observation a vector representation of the units and tiles. Additional details of the Farmworld are provided in the supplement.
environment: agents see only what is in a certain
5 Baselines
We use compare the ADAP algorithm to two algorithmic baselines. For each of the baselines, as well as ADAP, we experiment with both concatenation (+) and multiplicative model (x) types, and use consistent observation spaces, action spaces, and latent distributions so the only difference is the diversity algorithm itself.
The first baseline is Vanilla PPO, which we call the "Vanilla" baseline. The only difference between Vanilla and ADAP is that the former does not the diversity regularization loss in Equation I Vanilla policies still receive samples from latent distribution Z there is simply no objective term that enforces diverse policy actions conditional these samples.
Our second baseline was adapted from DIAYN. DIAYN formulated as a unsupervised skill generator, rather than policy generator. However, we believe that it remains one of the technically

---

closest works, and with slight modifications, we attempt to make comparison between DIAYN and ADAP. First, we highlight some differences between the methods. ADAP uses a KL-divergence based diversity term rather than learning a skill discriminator network. This enables ADAP's policy diversity to be optimized directly through gradient descent with respect to parameters Φ rather than be optimized through RI as with the skill diversity of DIAYN. Additionally, the ADAP latent distribution is defined over a continuous sample space, in contrast to the categorical sample space of DIAYN. We tried the standard DIAYN algorithm with categorical sample spaces and unsupervised skill discovery. however this performed poorly on all of our Farmworld and Markov Soccer experiments. Thus, to place the algorithms reward to DIAYN training (this is briefly mentioned in the DIAYN paper itself) 2.) to use the continuous sample space 3.) train skill regessor that minimizes predicted latent error, instead of skill discriminator that outputs latent class probabilities. We describe the new skill regressor in the supplement. We call this method D
more equal footing,
modify DIAYN: 1.) add extrinsic environmental
Training and Hyperparameters We train each method for the same number of timesteps (30 million), and generally keep hyperparameters constant across methods. These are described in the supplement.
Adaptation Comparisons When we apply Algorithm 2 ADAP, we apply the same algorithm to each of the baselines. We can do this because ADAP and baselines all share the same input latent distribution Z the only difference is how well they encode diverse policy space within Z.
6 Adaptation to Environmental Ablations via Optimizing Z
In nature, differences between species and even within species lend robustness to life as a whole. It becomes less likely that any single perturbation in the environment will break the overall system. In the same manner, differences between policies can lend robustness to the policy space as a whole.
Ablation
Description
Far Corner Wall Bar-
18x18 map size. Food spawns in the bottom right, agents spawn in the top left. 'rift' opens up between agents and their food. Agents must able to navigate up and around the wall.
rier
Speed
Single agent on 2x2 map. Food health yield is set very low, so agents must be able to rapidly and consistently farm adjacent towers.
Patience
Single agent on 2x2 map. Food yield is very high, but food respawn speed is low. Agents must 'ration' their food so it lasts until the next respawn.
Poison
Agents spawn the same map in which they were trained. However, chickens now yield negative health. Towers still yield positive health.
Chickens
Training
(Not ablation: listed for reference) 10x 10 map size. Food and agents are uniformly randomly distributed.
Env.
Table 1: Farmworld Ablations
Experiment We aim to test how having a diverse policy space allows us to search in latent space for policies that better fit unexpected environmental ablations. Doing so would demonstrate the robustness of a population of policies, and simultaneously provide information about different types of diversity that are learned by G.
To this end, we train G on a normal Farmworld environment as shown in Section 4. We then ablate the environment, changing features such map size and features, location of food sources, and even re-spawn times and food-yield. Lastly, we deploy G into the ablated environment and without changing the parameters σ, optimize the latent distribution for policies that successful in the new environment, using the search Algorithm 2. Ablations and descriptions are available in Table
Rather to our surprise, in each experiment trial, learning G using ADAP created a policy space II containing 'species' that could thrive in nearly every environmental ablation (see Figure B). The important thing to note is the development of these species was emergent from the training environment product of optimizing G for both policy diversity and reward maximization.
Results
How it possible that ADAP produced a policy space capable of adapting to nearly every ablation? The training environment relatively abundant with resources scattered about large map. Thus,
5

---

Far Corner
Wall Barrier
Patience
Speed
Poison Chickens
200
80
55.0
57.5
175
175
40
75
62.5
150
150
35
70
50.0
125
65
45.0
47-5
100
100
25
75
42.5
60
40.0
50
50
37-5
25
25
DIAYN (+)
Vanilla &
ADAP (e)
DIAYN (+)
Vanilla (0
Vanilla (+)
ADAP (c)
Vanilla (+)
Vanilla 60
DIAYN' (1)
DIAYN 00)
ADAP
ADAP (+)
(+)
DIAYN' 6
Vanilla &
Vanilla 6
Vanilla (+)
DIAYN' (+)
DLAYN' (x)
ADAP (+)
DIAYN
ADAP
Vanilla
DIAYN
ADAP
ADAP
ADAP
DIAYN
Vanilla
ADAP
Ablations. The bars display the average lifetime over 10 random seeds after applying Algorithm 2 to each of the algorithm and model choices. We additionally plot the min and max values from each seed pool to indicate that most baselines never come close to solving the ablations, where "solving" indicated by the average value surpassing the baseline health of the agent (the dashed black line). is possible to go below initial health, since agents can do damage to each other or eat poisonous food.
Figure 3: Agent Lifetime After Latent Distribution Optimization
ADAP (x)
Vanilla (x)
ADAP (x)
Vanilla (x)
Figure 4: Adaptation Farmworld Locomotion Ablations. Randomly chosen episode visualizations after latent optimization of Far Corner (left) and Wall Barrier (right). Chosen ADAP latents yield agent policies that navigate to the food, without otherwise modifying network parameters. Agents spawn in the dotted black outlines, and timesteps are stacked such that earlier ones are more transparent. We show 60 and 40 timesteps for Far Corner and Wall Barrier respectively.
there were many degrees-of-freedoe in the rules of survival, and by optimizing for diversity, we found policy space that filled these dfgrees-of of-frredomm while still yielding high reward. While these ablations reflect some of the possible axes of diversity, there are certainly more. For example, an agent's direction of 'preference' does not have to be the bottom-right, as in the Far Corner ablation. Indeed, sanity check, we tested placing food locations in various other spots map, and found that for every cardinal location, there was species of agent G that could exploit that new food location. What came surprise was that agents also used their health indicator to diversify: since agents diversify conditional on state, species developed in which agents would prefer to go upwards when their health is high, but downwards when their health is low. This particular
enlarged
Vanilla (x)
Vanilla (x)
ADAP (x)
Initial
Fine Tuning 2000 Episodes
Latent Optimization 200 Episodes
Wall Barrier
40
40
47.5
Far Corner
60
60
74
Patience
28.5
55.2
158
Table 2: Latent Distribution Optimization Speed and Performance versus Fine Tuning. Notice that for
some of the difficult exploration problems, the Vanilla model ever even finds the ablated food location, with 10 times the amount of training data. On the other hand, ADAP's diverse policy space enables commited exploration via selecting species that have tendencies to move towards different parts of the map.

---

waits...
egenntt waiss...
Respawn
ADAP (x)
10
20
30
50
Vanilla (x)
Timesteps (TS)
Would-be Respawn
Agent Expires
Tower Resganun Deloy (3o TS)
Moz. Heailh (20 75)
Figure 5: Adaptation on Farmworld Patience Ablation. We show episode rollouts arranged along timeline, using agent policies after latent optimization. Notice that the Vanilla agent does not ration out the two towers. Meanwhile, the selected ADAP agent waits until its health is very low (visualized by purple and pink speckles) before getting food and is able to survive past the next tower respawn.
Initial Samples
Samples from Post- Opttmiitiiiztioon Distribution
All Ablations
Far Corner
Wall Barrier
Patience
Speed
Poison Chickens
Lo
ADAP
05
1.0
Vanilla PPO
Figure 6: Pre and Post Optimization Latent Distribution Samples per Ablation. For each ablation, we run Algorithm 2 on a randomly chosen ADAP and Vanilla generator, Go.z. We refer to the post-optimization latent distribution as Z'. For visualization, we then sample 2 ~ Z' for 50 times for each ablation, and use PCA to reduce the dimensions of each method's data. The RGB color of each sample is directly proportional to the initial 3-dimensional sample. Notice how the for the Vanilla generators, the post-optimization distributions match the initial distribution, indicating that no regions of the latent space were found to be well-suited for any particular ablation. However, with ADAP generators, we see that the resulting distribution is vastly different in many cases to the initial distribution Z. This analysis also reveals certain overlap between two ablations. Notice how the Z' for the Wall Barrier ablation has samples that in some places overlap with Far Corner samples. Intuitively, in Far Corner, agents that move right-and- aecht--nd-then--tee then-down are successful, while in Wall Barrier, agents that move up-then-right then- are successful.
agent species was the one that managed to thrive in the Wall Barrier ablation. Similarly, in the Patience ablation, ADAP learned certain species of agent that waited until its health was low before farming a tower.
The Poison Chickens ablation was the one hold-out in which latent optimization on ADAP could not find profoundly successful species. possible that, there would have been too large of trade-off between diversity and potential reward in the training environment in order to learn a policy that ignored half of its potential food sources. We come back to this ablation in the next experiment.
Finally, we should note that ADAP beat the Vanilla baseline in all ablations aside from Speed. We hypothesize this ablation is the most in-distribution to the training environment. Since the Vanilla baseline optimized for solely for expected rewards, it makes no diversity tradeoff's and performs well in in in-distribution environments. As visible from the plots, D also did not learn to speciate in manner that was successful on the majority of ablations.
7 Measurement of Agent Individuality and Diversity in Population
good generative model of policies should be able to represent a multi-modal space behaviors. That is: different agent policies should be able to act with individuality. Our generative model uses
7

---

Lifetime
[(T:Z)
H(T)
H
ADAP (x)
23.2 + 0.5
0.89
1.0
0.10
Vanilla (+)
22.6 - 0.3
0.30
0.99
0.69
Vanilla (x)
22.3 + 0.4
0.32
0.99
0.67
ADAP (+)
21.9 + 0.6
0.76
0.92
0.15
DIAYN* (x)
21.8 + 1.3
0.58
0.95
0.37
DIAYN (+)
21.6 + 0.4
0.49
0.93
0..4
Table 3: Various Specialization and Diversity Metrics. We provide the average result across 10 random sceds, and + indicates standard deviation. Standard deviation for [(T; 2). HT),, H (T) was less than equal to 0.1 for all entries, and has been omitted to reduce verbosity. (T)) is computed with log base 2, and is maximal at 1.
shared parameter set across all agents, and naively using shared parameters could result in learning just one "average" agent - which is precisely what we wish to avoid.
Niche Specialization Experiment To test the abilities of our policy generator, we set up the Farmworld environment with a hidden rule specific to this experiment: when an agent spawns, it able harness resources from either towers or chickens. However, once it gets health from one unit type, it becomes 'locked-into' that unit type, and cannot gain health from the other unit type. Information about an agent's 'locked-into' state is not provided as part of the agent observation, and since agents have no memory. they would have to look to their latent to determine their niche. Since there are equal numbers of chickens and towers on our map, a reasonable generative model algorithm should be able to map half the latent space to each of these two specializations, niches.
Results So we can see how well the entire latent space falls maps to niche, we report rewards and other metrics in Table without running latent space optimization on ADAP or baselines. In summary, ADAP consistently learned a more multi-modal policy space than any of the other baselines. Our results also indicate that using a multiplicative model can yield higher degree of policy space multi-modality. and therefore greater success in this environment.
Poison Chickens
8o
70
60
50
40
We can see in Table 3 that ADAP (x) is able to attain the highest aver-
30
age agent lifetime. This, however, not necessarily the most interesting
20
point. ADAP, learns policy generator with the highest mutual informa-
10
tion I(TIZ) between an agent "niche" T and the latent distribution Z. Intuitively, this means that ADAP was able to learn a population of agents that were composed of two clear species - on one hand: agents that focus towers. Formally, let T
Vanilla (+)
Vanilla (x)
DIAYN" (+)
DIAYN (3)
ADAP (+)
ADAP (x)
on chickens, and on the other: agents that focus be a discrete random variable where pr is the probability that an agent attacks target t, for t E {chicken, tower). Then [T;; 2) is high when individual agents are specialized in a niche, and we see diverse niches across our population. This is because I(T; Z) HTT) H(T Z) and is maximized by both increasing H (T) and decreasing H(T Z). H(T) measures the diversity of niches across all agents in the population, and (T)Z) measures how rigidly an agent falls into a single niche (i.e. specialization).
Figure 7: Chi ckins We report min, max, and average lifetime after running Algorithm 2.
Poison
Revisited.
As an example, suppose agents were highly specialized but not diverse, e.g., all agents were chicken- only attackers. Then (T) = ZT = [T;; 2) = O. On the other hand, suppose that all 2 ~ Z yield agent policy that attacks chickens and towers with equal probability. Then in this case H = (T) = and [(T; Z) O. Intuitively, this means half of the time agents are wasting timesteps attack target that they are unable to even damage! Qualitatively, we have seen that the latter case occurs with the Vanilla and (most seeds of) DIAYN* baselines: notice that their H ZT) is significantly higher than that of ADAP.
For fun, we performed latent distribution optimization on generators trained using the Niche Specialization environment to fit the Poison Chick environment. One would expect al- gorithms with high H(T to fare well, since Algorithm 2 can find optimized 2" such that pr (chicken z l*) 0. Sure enough,
this result in Figure 7 ADAP (x) is most suc-
8

---

cessful at consistently producing a generative model that can produce policies that not only avoid chickens, but also successfully attack only towers.
8 Adaptation and Self-Play in a Zero-Sum Two-Player Environment
Environment This experiment uses Markov Soccer, introduced in [5]. Two agents, A and B, play on gridworld and must 'carry" a ball into the opposing goal to score. Agents walk in cardinal directions or stand in place. Possession is randomly initialized, and switches if one an agent bumps into the other. Actions of A and B occur on the same timestep, execution order is randomized, and each timestep ends in draw with some probability.
Markov Soccer is interesting environment, because the best policy for one agent depends on the policy of the other agent. As described in [5), there exists a worse-case-optimal probabilistic policy for Markov Soccer, which maximizes the minimum possible score against any adversary. This strategy tends to be conservative, preferring to act towards draw where a different policy could have obtained higher score. On the other hand, non-worse-ee caseooia strategies may be less conservative and may achieve very high scores against some opponents, but very low scores against others. Analogous to real soccer, different players have varying abilities and play styles, and a given player P1 may be optimal against P2, but not against P3-
If any single policy has its drawbacks, can instead learn an entire space of diverse policies II == {Ti, TT.... Tint J. where for any opponent, we can select a policy n; E IT that achieves the maximum score against that opponent? Ideally, this space includes the worse-case-optimal policy, as well as other more aggressive policies. Then, just as a coach might swap out soccer player, we can mix and match our champion as suited.
Experiment Can we learn a population of individuals that is holistically strong against all types of opponents? We evaluate adaptability various adversaries using two methods. First, we test baselines and our method against a set of hand-coded soccer bots. These bots are designed to represent wide gamut of strategies, some of which are more exploitable than others. Secondly, we evaluate each G by playing ADAP (x), ADAP (+), Vanilla (x), and Vanilla (+) in round-robin tournament against each other. All scores is determined by wins minus losses over 1000 simulated games.
Against Hard-Coded Bots:
Each bot always starts on the left side, and the learned policy starts on the right side (although the environment is coded such that observations are side-nvvriiant. Bot types fall into three categories: offense (bots start with possession), defense (policy starts with possession), and mixed (random starting possession). See Table 4 for more details.
Bot
Bot Type
Bot Policy
Straight Oscillate 0
Offense
Always moves ri right, towards the goal.
Defense
Oscillates in column 0, blocking both squares adjacent the goal.
Oscillate
Defense
Oscillates in column 1, leaving gap in front of the goal.
Stand
Defense
stands in one square adjacent the goal, leaving the other open.
Rule-Based
Mixed
Follows hand-coded heuristics.
Random
Mixed
Follows a random policy.
Table 4: Markov Soccer Bot Adversaries
Round-Robin Against Each Other: We also pit each generative model in a round robin tournament against the other models. The manner in which we do this is described in the supplement.
Training and Baselines We use self-play train both ADAP and baselines. We the same Vanilla baseline as described in Section 5, and we omit the DIAYN* haseline for brevity. Note that at no point in the training process did any of our algorithms train against any bots, against each other.
Results As in the Farmworld adaptability experiment, we see from Figure g that ADAP is able to learn a G during the train phase that emergently contains members that are successful against variety of unexpected adversaries including naive bots and other policies.
Compared to Vanilla, the ADAP policy space generalizes better against all adversaries. Going back to the soccer team example, we were able to select individuals from the ADAP population that were well suited for specific strategies. For example, against the Osci Wlate adversary, ADAP latent

---

Round- Rooin
Rule-Basen
Random
Oscillate
Oscillate
Straight
Stand
100
600
800
800
75
300
500
000
600
700
600
190
100
400
300
400
500
-75
-99
150
300
200
200
200
100
Vioila 00
ADAP 00
Vinill Vanill (t
ADAP 00
ADMP (v)
ADMP (c)
ADAP (3
Vinilla 00
Vanill (e)
ADAP (1)
ADAP D0
(Goilla (+)
ADAP (+)
ADAP
Voiill (+)
Vanilla 60
ADAP (c)
ADAP
(ami Lo
Wami 6
V
ADMP
Vioill
Viailll
Vinilll
Figure 8: Results of round-robin and bot games. Score is over 1000 runs, some games prematurely
end in a draw. We report mean and standard deviation over five random seeds.
optimization found member of the population that side-stepped the oscillating adversary simply by moving to the top row, and then down to the goal. Additionally, against the Straight adversary, successful ADAP individuals stole possession by deterministically standing in-front of the opponent to block, and then moving around and into the goal. On the other hand both of these situations, Vanilla could not find individuals that exploited the naive deterministic nature of their opponents.
Using ADAP did not just allow us to optimize against naive opponents. ADAP learned the best G in the round-robin tournament, and was the only method that was able to consistently beat our rule-based bot. is possible that by using ADAP during the self-play training, individuals encountered wide variety of strategies that bettered overall performance.
9 Limitations
Bad Apples When using ADAP, not every member of the policy space is going to be an optimal policy. In fact, some generated policies might be bad apples: policies that were incentivized by the diversity regularizer to take actions that were not rewarding. Naturally, some individuals might be better or worse than others. These individuals can be removed by optimizing the latent distribution. However, the bad apples may come with a plus side. Even though they do not perform well in the current environment, they might happen to perform well in future ablated environment!
Continuous-Action Space Environments The results presented so far focus entirely on environ- ments with discrete categorical action spaces, in which we have observed that diversity regularizer in Equation| ] empirically performs well. However, not all environments in RL use discrete action spaces continuous action spaces widely used in RL control tasks. While we believe that our regularizer can work in these environments, we have not rigorously tested in these environments.
10 Conclusion
We have presented framework learn a generative model of policies. Rather than learning just one policy, we aim to find as many high-performing and individually distinct policies as possible, all compressed within the parameters our generator.
Learning a space of policies pays off in an open-ended environment such as Farmworld, in which there may be more than one path success. We show Section 6 that we can adapt to ablations by quickly choosing 'species' from our learned policy space that are successful in the new environment.
We also learn a policy space in competitive, two-player, zero-sum game in Section X Here, no single deterministic policy is optimal against all adversaries. Instead, we show how to train a family of policies that can be naturally adaptable to wide array of both challenging and naive adversaries.
Overall, we hope to show how can be beneficial in RL to optimize not just for reward, but also for diversity of behavior. As environments continue to increase in complexity and open-endedness filled with branching paths to success it makes sense learn not just one, but many, solutions.
11 Acknowledgements
This research was supported in part by IBM through the MIT-IBM Watson Al Lab.
10

---

References
[1] Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. leamingg Artificial Intelligence, 112(1):181-
211, 1999.
[2] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is All You Need: Learning Skills without Reward Function. arXiv e-prints, page arXiv: 1802.06070, February 2018.
[3] Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic Neural Networks for Hierarchical Reinforcement Learning. arXiv e-prints, page arXiv:1704.0300 April 2017.
[4] Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. Quality diversity: new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016.
[5] Michael I.. Littman. Markov games framework for multi-agent reinforcement learning.
In Proceedings of the Eleventh International Conference on International Conference on Machine Learning. ICML'94, page 157-163, San Francisco, CA, USA, 1994. Morgan Kaufmann Publishers Inc.
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv e-prints, page arXiv:1606.01555 June 2016.
[7] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv e-prints, page arXiv: 170.06777. July 2017.
[8] Kenneth O. Stanley and Risto Mikkkulanen. Evolving neural networks through augmenting topologies. Evolutionary Computation, 10(2):99-127, 2002.
[9] Kenneth O Stanley, David B. D' D'Ambossio, and Jason Gauci. A Hypercube-Baste Based Encoding for Evolving Large-Scale Neural Networks. Artificial Life, 15(2):185-212, 04 2009.
[10] Joel Lehman and Kenneth O. Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary Computation, 19(2):189-223, 2011.
[1]] Justin K. Pugh, L. B. Soros, Paul A. Szerlip, and Kenneth O. Stanley. Confronting the challenge of quality diversity. In Proceedings of the 2015 Anmual Conference Genetic and Evolution- ary Computation, GECCO '15, page 967-974, New York, NY, USA, 2015. Association for Computing Machinery.
[12] Joel Lehman and Kenneth Stanley. Evolving diversity of creatures through novelty search and local competition. pages 21 1-1118 01 2011.
[13] Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv e-prints, page MxXiv:1504.04900 April 2015.
[14] Muhammad A. Masood and Finale Doshi-Velez Diversity-Inducing Policy Gradient: Us- ing Maximum Mean Discrepancy to Find Set of Diverse Policies. arXiv e-prints, page 1rXiv:1906.0000. May 2019.
[15] Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7214-7213. PMLR, 18-24 Jul 2021.
[16] Zhang-War Wei Hong, Tzu- Yun Shann, Shih- Hang Su, Yi-Hsiang Chang, and Chun Yi Lee. Diversity-Driven Exploration Strategy for Deep Reinforcement Learinng, arXiv e-prints, February 2018
page
[17] Jack Parker-Holder Aldo Pacchiano, Krzysztof Choromanski, and Stephen Roberts. Effective Diversity in Population Based Reinforcement Learning. arXiv e-prints, page arXiv:2002.00632 February 2020.
11

---

[18] Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. NeurIPS, 2020.
[19] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals. Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population Based Training of Neural Networks. arXiv e-prints, page LtXiv:1711.098A. November 2017.
[20] Andrew Cohen, Xingye Qiao, Lei Yu, Elliot Way, and Xiangrong Tong. Diverse Exploration via Conjugate Policies for Policy Gradient Methods. arXiv e-prints, page 10Xiv:1902.03633, February 2019.
[21] Joshua Achiam, Harrison Edwards, Dario Amodei, and Picter Abbeel. Variational Option Discovery Algorithms. arXiv e-prints, page arXiv:1807.102: July 2018.
[22] Leslie Pack Kaelbling. Learning to achieve goals. In IN PROC. OF ICCA1993, pages 1094-1098. Morgan Kaufmann, 1993.
[23] Soroush Nasiriany. Vitchyr H. Pong, Steven Lin, and Sergey Levine. Planning with Goal- Conditioned Policies. arXiv e-prints, page
November 2019.
[24] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1312-1320, Lille, France, 07-09 Jul 2015. PMLR.
[25] Yiming Ding, Carlos Florensa, Mariano Phielipp. and Pieter Abbeel. Goal-conditioned Imitation Learning. arXiv e-prints. page arXiv: 0088888 June 2019.
[26] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. arXiv e-prints, page arXiv: 1803. 1485, March 2018.
[27] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. ROMA: Multi-Agent Re- inforcement Learning with Emergent Roles. arXiv e-prints, page arXiv:2003.08039, March 2020.
[28] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. MAVEN: Multi- Agent Variational Exploration, arXiv e-prints, page arXiv: 1910.044, October 2019.
12


Reviews:
[{"review_id": "zv4SVPbGGbL", "reviewer": "Reviewer_Q4DY", "summary": "This paper proposes to train a generative model for entire populations of maximally diverse agents, from which one specific individual policy can quickly be selected at deployment time through a fast search process.\n\nPolicies are represented as networks augmented with a low-dimensional latent variable z, randomly sampled at agent initialization. Thus each trained network is actually a generative model, from which an infinity of policies can be generated by sampling over z. \n\nCrucially, the training procedure encourages the fixed weights of the network to not only obtain good performance for any given random z, but also to produce maximally different policies for any two different z. Thus the method is a quality diversity (QD) method (though the authors seem ambiguous on the subject).\n\nThis method is compared with a previously published method (DIAYN), and with itself but without the diversity objective, and is found superior to both on various gridworld tasks, both in terms of expected performance and diversity of generated agents. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the method interesting and somewhat novel (see below). Building an infinite, continuous space of diverse but efficient agents, through which one can quickly search (and thus adapt) at deployment time, as opposed to the discrete populations typically maintained in QD algorithms, seems like a significant advantage. \n\nOne possible concern is insufficient relation with previous work. Bluntly: Am I correct that  this method is basically DIAYN with continuous 3D z and a different, simpler diversity objective? If not, why? \n\nNote that this would not be a deal-breaker, since DIAYN is already used as a baseline and found less efficient for the tasks selected here. But somehow it's not mentioned in the \"related work\" section! It seems important to explain how this method relates to DIAYN and how it differs from it (e.g. in objectives, motivation, etc.), in addition to the results.\n\nThe paper is confusingly written and some passages were unclear:\n\n- Most importantly, in p. 5, the description of baselines (especially lines 171-173) is extremely obscure and it took me a while to understand (?) what was meant. I suggest replacing the start of l. 173 with the following: \"the Vanilla baseline also includes a latent z; the only difference between 'Vanilla PPO' and ADAP is that the former does not use the diversity regularization (Eq. 2)\". If this is incorrect, an equivalent explanation of what exactly Vanilla PPO means should be provided. Also: \"DIAYN* uses a continuous distribution\" - over what ?? (I suppose it should be over \"skills\", which are very similar to the latent z).\n\n- The farmworld environment should be described a little bit more in the main text, e.g. what's the difference between towers and chickens, what does \"attacking\" another agent mean, etc. Just pointing to the supplements is not enough.\n\n- In p.8, \"G1 vs G1\" ? What is pi_G1,z1 ? Also, this evaluation method seems obviously asymmetric: G1 is forced to produce a generalist, while G2 can choose an agent that specifically exploits G1, presumably lending an advantage to G2. How is this asymmetry handled?\n\n- Whenever the z is optimized, it should be mentioned at least briefly how, *in the main text*. \n\nMinor points:\n\n- In \"related work\" sections: The proposed method is very much a quality diversity method, by definition.Probability distributions over actions *are* a behavioral characterization, and KL divergence between them is a novelty metric. It may be different from existing QD methods, but it's still one.\n\n- p.2, line 60: does (1) mean Equation 1? It should be spelt out.\n\n- Typos: p. 4, l. 163: missing \"to\". p. 5, l. 172: missing \")\". p. 6, l. 227: what's \"the 2\"? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and impact seem adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "lkqwEhA0jHB", "reviewer": "Reviewer_crDt", "summary": "The paper constructs a latent space diversity-based family of policies\nable to adapt to different environments under different\ncircumstances. The idea is to have a relatively low-dimensional set of\nadaptation parameters which essentially are all that needs to be\nselected to choose between pre-trained policies.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I do like the premise of the paper, a simple, but promising idea. It\naddresses the issue that training is slow, but sometimes one still\nneeds to adapt quickly to a new situation. The idea of using a\nlow-dimensional latent space to select from otherwise pre-trained\npolicies is nice. The reviewer was wondering why, since they already\ncited work by Stanley, they didn't refer to MAP elites which is in\nspirit even closer (albeit on the EA side of the formalisms) to the\npresent paper.  More generally, it might be good to relate to\nmultiobjective optimization, and also to multitask MDPs to make clear\nhow the present algorithm relates to existing solutions. It is clear\nthat the diversification is not based on the concrete task set, but on\ndiversification in latent space, still the latent space is a function\nof the training in given sample spaces.\n\n- Eq. (1):\n  - what distribution is the inner expectation computed over?\n    The notation as a set is really not clear, and z_i != z_j is weird,\n    because in continuous distributions that probability were 0. Please\n    make clear what you are doing here.\n  - you talk about a smoothing constant b, but we have no idea what\n    your regularizer looks like. How does the smoothing look like?\n- line 227: \"using the search algorithm described in the 2.\" -\n  incomplete sentence\n- line 235: \"emergent from the train environment\" -> \"training\"\n- Figure 5: colors need explaining\n- line 256: -> training\n- line 288: not entirely clear what you do here: are you measuring the\n  distance between the two sets of possible policies? Or what else?\n  Perhaps write a formula, to avoid ambiguities?\n- line 293: the second G_1 should be a G_2\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Reasonable listing of limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "gGnG3F0cNHH", "reviewer": "Reviewer_Pf7s", "summary": "The authors proposed a generative model of policies, which maps a low-dimensional latent space to an agent policy space to learn a space of diverse and high-reward policies on any given environment (without requiring the use of separate policy parameters). The proposed method is able to adapt to changes in our environment solely by selecting policies in latent space. The experiments evaluated our generative model’s capabilities in a variety of environments, including an open-ended grid-world and a two-player soccer environment.\n\nThe strength of this paper is as follows: 1) The proposed method integrates the goals of quality diversity into deep RL by simulating an entire population of agents via a generative model of policies. 2) The authors evaluated this method using three different experiments and showed that this method was able to learn a more multi-modal and effective policy space than any of the other baselines. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "However, there were some unclear points. First, although the authors describe the novelty of this work in the Related work section fragmentally, there was little description of the technical novelty. In my understanding, it seems to be the diversity regularization and multiplicative model, but for those, there was no description of the related work in Section 2 (since the ideas are simple, it would be necessary to quote other work). In addition, although the experimental results were better, the theoretical contribution of this work should be clarified. Lastly, as described in abstracts, time and memory efficiency were not theoretically and numerically analyzed. \n\nMinor comments:\n\nEq. (1): S and s were not defined. If s is a state, actions may not be also defined. \n\n111 “requiring discriminability via state or trajectory impose diversity constraints along a and speciﬁc task-dependent axis. “ Does the author have theoretical grounds for that? Or just understanding from the experimental results?\n\n172 no parenthesis\n\n176 I did not understand why the authors augment the intrinsic reward with extrinsic environmental reward.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact were mentioned.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "H8cRaic6zA7", "reviewer": "Reviewer_xPri", "summary": "This paper presents ADAP, a method for learning a generative model of diverse policies by conditioning a standard PPO policy on an additional latent z input, such that KL divergence between the z-conditioned policies for pairs of different z's is maximized. The experiments show that ADAP is able to learn a diverse set of policies", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "ADAP is a simple method that seems to produce a diverse of set of policies. The experiments are on conducted on two interesting environments that are not commonly studied in deep RL. While the method itself seems quite interesting and the experimental results are promising, I believe the paper could be strengthened considerably by improving the presentation of results, as well as filling in a few key missing empirical considerations that would clarify the utility of the method in practice:\n\n- In the niche specialization experiment, it seems that ADAP leads to higher mean agent specialization, but there is no indication of whether the specializations themselves are more or less diverse than those produced by DIAYN. Some quantitative measure of this would add confidence to the idea that ADAP is effectively optimizing for diversity.\n- It would be informative to report how many episodes are required for adapting the pretrained policy by searching over the latent vectors z. A useful comparison here would then be to give Vanilla PPO the same budget of environment interactions for fine-tuning its policy to the new ablated environment. Similarly for adapting to Markov Soccer opponents.\n- Further, since searching over latents can be seen as a form of weight optimization, another fair comparison would be to train vanilla PPO with a dummy latent z concatenated to its input, then performing the same search procedure in Algorithm 2 to adapt the vanilla PPO policy via search over of its latent input vector.\n- It would raise confidence in the results to report results for additional seeds (5-10 seeds).\n- It is hard to parse the relative diversity of the learned latent policies between DIAYN and ADAP from the current results. Some way of globally summarizing the differences in diversity among the agent populations trained via ADAP and DIAYN would be informative. For example, perhaps training agents in a 2D goal-reaching grid world environment, and showing the average state occupancy of rollouts sampled from each latent population, after various time steps.\n- It would be valuable to better understand how modulating the weight of the ADAP diversity bonus and the dimensionality of the latent encoding z impacts the transferability of the optimal latent policies to transfer environments.\n\nAdditional comments:\n- It is unclear to me what the dotted lines in Figure 3 represent. Are these standard deviations around the mean?\n- Was Algorithm 2 also applied to finding the optimal skills from DIAYN for adapting to Farmworld variants in the experiment baselines?\n- A highly related work should be included in related works: Lupu et al, 2021, [http://proceedings.mlr.press/v139/lupu21a.html](http://proceedings.mlr.press/v139/lupu21a.html).\n\n### Post-rebuttal update\nBased on the authors' clarifications and reporting results for additional experiment seeds, I have decided to upgrade my score to a 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The authors acknowledge that their existing experiments relate to simplistic discrete-action settings.\n- I am curious whether this method can scale to much more complex environments, and whether low-dimensional latents are still sufficient to capture the useful diversity in behavior for the environment.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Inspired by the diversity produced by evolution, this work presents a method of learning a latent space of policies that is conditioned by an RL algorithm like PPO. They show that their model can learn a diverse set of policies, and since the latent space is not too high dimensional, it is easy to adapt to changes in the environment. They demonstrate the model in a grid-world “farm” environment and a two-player soccer environment (I also downloaded the zip file from the supplementary materials, and I found the web page useful to understand some results from the animations.).

The strengths of the paper (best summarized by reviewer Pf7s):

1. The proposed method integrates the goals of quality diversity into deep RL by simulating an entire population of agents via a generative model of policies.

2. The authors evaluated this method using three different experiments and showed that this method was able to learn a more multi-modal and effective policy space than any of the other baselines. 

There were issues with the writing and clarity, raised by reviewer Q4DY. The authors have responded to the questions, and have pledged to clarify issues with the paper discussed in the thread.

Reviewer xPri, who wrote a critical, but fair and balanced review, also highlighted a list of important points in their review related to not only improving presentation of the results, but also filling in key missing experiments, and improving statistical confidence in the results. The authors performed several additional studies, and after a long and detailed dialog between reviewer and author (this is where Open Review really shines), an understanding is established and the reviewer is satisfied, improving their score to 6 (even to 7 if additional seeds can be run on remaining experiments).

In summary, this paper presents a simple and promising idea for creating diverse adaptable agent populations in RL; a work that would be of interest to the NeurIPS community. I believe the review process has helped strengthen the paper to a state that we are happy to publish the work at NeurIPS. For these reasons I'm recommending acceptance of the work as a poster.

Decision: Accept (Poster)