Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper constructs a latent space diversity-based family of policies
able to adapt to different environments under different
circumstances. The idea is to have a relatively low-dimensional set of
adaptation parameters which essentially are all that needs to be
selected to choose between pre-trained policies.
main_review: I do like the premise of the paper, a simple, but promising idea. It
addresses the issue that training is slow, but sometimes one still
needs to adapt quickly to a new situation. The idea of using a
low-dimensional latent space to select from otherwise pre-trained
policies is nice. The reviewer was wondering why, since they already
cited work by Stanley, they didn't refer to MAP elites which is in
spirit even closer (albeit on the EA side of the formalisms) to the
present paper.  More generally, it might be good to relate to
multiobjective optimization, and also to multitask MDPs to make clear
how the present algorithm relates to existing solutions. It is clear
that the diversification is not based on the concrete task set, but on
diversification in latent space, still the latent space is a function
of the training in given sample spaces.

- Eq. (1):
  - what distribution is the inner expectation computed over?
    The notation as a set is really not clear, and z_i != z_j is weird,
    because in continuous distributions that probability were 0. Please
    make clear what you are doing here.
  - you talk about a smoothing constant b, but we have no idea what
    your regularizer looks like. How does the smoothing look like?
- line 227: "using the search algorithm described in the 2." -
  incomplete sentence
- line 235: "emergent from the train environment" -> "training"
- Figure 5: colors need explaining
- line 256: -> training
- line 288: not entirely clear what you do here: are you measuring the
  distance between the two sets of possible policies? Or what else?
  Perhaps write a formula, to avoid ambiguities?
- line 293: the second G_1 should be a G_2

limitations_and_societal_impact: Reasonable listing of limitations.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents ADAP, a method for learning a generative model of diverse policies by conditioning a standard PPO policy on an additional latent z input, such that KL divergence between the z-conditioned policies for pairs of different z's is maximized. The experiments show that ADAP is able to learn a diverse set of policies
main_review: ADAP is a simple method that seems to produce a diverse of set of policies. The experiments are on conducted on two interesting environments that are not commonly studied in deep RL. While the method itself seems quite interesting and the experimental results are promising, I believe the paper could be strengthened considerably by improving the presentation of results, as well as filling in a few key missing empirical considerations that would clarify the utility of the method in practice:

- In the niche specialization experiment, it seems that ADAP leads to higher mean agent specialization, but there is no indication of whether the specializations themselves are more or less diverse than those produced by DIAYN. Some quantitative measure of this would add confidence to the idea that ADAP is effectively optimizing for diversity.
- It would be informative to report how many episodes are required for adapting the pretrained policy by searching over the latent vectors z. A useful comparison here would then be to give Vanilla PPO the same budget of environment interactions for fine-tuning its policy to the new ablated environment. Similarly for adapting to Markov Soccer opponents.
- Further, since searching over latents can be seen as a form of weight optimization, another fair comparison would be to train vanilla PPO with a dummy latent z concatenated to its input, then performing the same search procedure in Algorithm 2 to adapt the vanilla PPO policy via search over of its latent input vector.
- It would raise confidence in the results to report results for additional seeds (5-10 seeds).
- It is hard to parse the relative diversity of the learned latent policies between DIAYN and ADAP from the current results. Some way of globally summarizing the differences in diversity among the agent populations trained via ADAP and DIAYN would be informative. For example, perhaps training agents in a 2D goal-reaching grid world environment, and showing the average state occupancy of rollouts sampled from each latent population, after various time steps.
- It would be valuable to better understand how modulating the weight of the ADAP diversity bonus and the dimensionality of the latent encoding z impacts the transferability of the optimal latent policies to transfer environments.

Additional comments:
- It is unclear to me what the dotted lines in Figure 3 represent. Are these standard deviations around the mean?
- Was Algorithm 2 also applied to finding the optimal skills from DIAYN for adapting to Farmworld variants in the experiment baselines?
- A highly related work should be included in related works: Lupu et al, 2021, [http://proceedings.mlr.press/v139/lupu21a.html](http://proceedings.mlr.press/v139/lupu21a.html).

### Post-rebuttal update
Based on the authors' clarifications and reporting results for additional experiment seeds, I have decided to upgrade my score to a 6.
limitations_and_societal_impact: - The authors acknowledge that their existing experiments relate to simplistic discrete-action settings.
- I am curious whether this method can scale to much more complex environments, and whether low-dimensional latents are still sufficient to capture the useful diversity in behavior for the environment.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors proposed a generative model of policies, which maps a low-dimensional latent space to an agent policy space to learn a space of diverse and high-reward policies on any given environment (without requiring the use of separate policy parameters). The proposed method is able to adapt to changes in our environment solely by selecting policies in latent space. The experiments evaluated our generative model’s capabilities in a variety of environments, including an open-ended grid-world and a two-player soccer environment.

The strength of this paper is as follows: 1) The proposed method integrates the goals of quality diversity into deep RL by simulating an entire population of agents via a generative model of policies. 2) The authors evaluated this method using three different experiments and showed that this method was able to learn a more multi-modal and effective policy space than any of the other baselines. 

main_review: However, there were some unclear points. First, although the authors describe the novelty of this work in the Related work section fragmentally, there was little description of the technical novelty. In my understanding, it seems to be the diversity regularization and multiplicative model, but for those, there was no description of the related work in Section 2 (since the ideas are simple, it would be necessary to quote other work). In addition, although the experimental results were better, the theoretical contribution of this work should be clarified. Lastly, as described in abstracts, time and memory efficiency were not theoretically and numerically analyzed. 

Minor comments:

Eq. (1): S and s were not defined. If s is a state, actions may not be also defined. 

111 “requiring discriminability via state or trajectory impose diversity constraints along a and speciﬁc task-dependent axis. “ Does the author have theoretical grounds for that? Or just understanding from the experimental results?

172 no parenthesis

176 I did not understand why the authors augment the intrinsic reward with extrinsic environmental reward.

limitations_and_societal_impact: Limitations and societal impact were mentioned.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to train a generative model for entire populations of maximally diverse agents, from which one specific individual policy can quickly be selected at deployment time through a fast search process.

Policies are represented as networks augmented with a low-dimensional latent variable z, randomly sampled at agent initialization. Thus each trained network is actually a generative model, from which an infinity of policies can be generated by sampling over z. 

Crucially, the training procedure encourages the fixed weights of the network to not only obtain good performance for any given random z, but also to produce maximally different policies for any two different z. Thus the method is a quality diversity (QD) method (though the authors seem ambiguous on the subject).

This method is compared with a previously published method (DIAYN), and with itself but without the diversity objective, and is found superior to both on various gridworld tasks, both in terms of expected performance and diversity of generated agents. 
main_review: I find the method interesting and somewhat novel (see below). Building an infinite, continuous space of diverse but efficient agents, through which one can quickly search (and thus adapt) at deployment time, as opposed to the discrete populations typically maintained in QD algorithms, seems like a significant advantage. 

One possible concern is insufficient relation with previous work. Bluntly: Am I correct that  this method is basically DIAYN with continuous 3D z and a different, simpler diversity objective? If not, why? 

Note that this would not be a deal-breaker, since DIAYN is already used as a baseline and found less efficient for the tasks selected here. But somehow it's not mentioned in the "related work" section! It seems important to explain how this method relates to DIAYN and how it differs from it (e.g. in objectives, motivation, etc.), in addition to the results.

The paper is confusingly written and some passages were unclear:

- Most importantly, in p. 5, the description of baselines (especially lines 171-173) is extremely obscure and it took me a while to understand (?) what was meant. I suggest replacing the start of l. 173 with the following: "the Vanilla baseline also includes a latent z; the only difference between 'Vanilla PPO' and ADAP is that the former does not use the diversity regularization (Eq. 2)". If this is incorrect, an equivalent explanation of what exactly Vanilla PPO means should be provided. Also: "DIAYN* uses a continuous distribution" - over what ?? (I suppose it should be over "skills", which are very similar to the latent z).

- The farmworld environment should be described a little bit more in the main text, e.g. what's the difference between towers and chickens, what does "attacking" another agent mean, etc. Just pointing to the supplements is not enough.

- In p.8, "G1 vs G1" ? What is pi_G1,z1 ? Also, this evaluation method seems obviously asymmetric: G1 is forced to produce a generalist, while G2 can choose an agent that specifically exploits G1, presumably lending an advantage to G2. How is this asymmetry handled?

- Whenever the z is optimized, it should be mentioned at least briefly how, *in the main text*. 

Minor points:

- In "related work" sections: The proposed method is very much a quality diversity method, by definition.Probability distributions over actions *are* a behavioral characterization, and KL divergence between them is a novelty metric. It may be different from existing QD methods, but it's still one.

- p.2, line 60: does (1) mean Equation 1? It should be spelt out.

- Typos: p. 4, l. 163: missing "to". p. 5, l. 172: missing ")". p. 6, l. 227: what's "the 2"? 
limitations_and_societal_impact: Limitations and impact seem adequately addressed.
review_title: null

Meta-review: Inspired by the diversity produced by evolution, this work presents a method of learning a latent space of policies that is conditioned by an RL algorithm like PPO. They show that their model can learn a diverse set of policies, and since the latent space is not too high dimensional, it is easy to adapt to changes in the environment. They demonstrate the model in a grid-world “farm” environment and a two-player soccer environment (I also downloaded the zip file from the supplementary materials, and I found the web page useful to understand some results from the animations.).

The strengths of the paper (best summarized by reviewer Pf7s):

1. The proposed method integrates the goals of quality diversity into deep RL by simulating an entire population of agents via a generative model of policies.

2. The authors evaluated this method using three different experiments and showed that this method was able to learn a more multi-modal and effective policy space than any of the other baselines. 

There were issues with the writing and clarity, raised by reviewer Q4DY. The authors have responded to the questions, and have pledged to clarify issues with the paper discussed in the thread.

Reviewer xPri, who wrote a critical, but fair and balanced review, also highlighted a list of important points in their review related to not only improving presentation of the results, but also filling in key missing experiments, and improving statistical confidence in the results. The authors performed several additional studies, and after a long and detailed dialog between reviewer and author (this is where Open Review really shines), an understanding is established and the reviewer is satisfied, improving their score to 6 (even to 7 if additional seeds can be run on remaining experiments).

In summary, this paper presents a simple and promising idea for creating diverse adaptable agent populations in RL; a work that would be of interest to the NeurIPS community. I believe the review process has helped strengthen the paper to a state that we are happy to publish the work at NeurIPS. For these reasons I'm recommending acceptance of the work as a poster.

Decision: Accept (Poster)