Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper makes three primary contributions:

(1) The paper proposes a new problem setup, called Differentiable Quality Diversity (DQD), where the goal is to optimize for a set of diverse solutions that maximize a target objective. Diversity is formalized by assuming the existence of **differentiable** measure functions, which intuitively specify an axis of variation and thus far away solutions along in the joint space of these measure functions are more diverse.

(2) To solve the DQD, the paper extends the MAP-Elite algorithms from the related and previously studied non-differentiable variant, abbreviated as simply QD. The extensions incorporate gradient information in novel ways, interfacing first-order methods with evolutionary search style procedures. 

(3) Finally, the authors conduct empirical evaluation of the proposed algorithms which suggests that these combinations of gradient and evolutionary algorithms are indeed better than either approach alone on some toy benchmark tasks and a generative modeling task.
main_review: To the best of my knowledge, both the problem setup and the proposed solutions are novel. The setup itself is a non-trivial important special case of the previously studied Quality Diversity problem, eg, the use case in generative modeling has potential implications beyond just diverse face generation in other scientific fields as well. The methods proposed in the work are sound, simple, and easy to implement. I also liked their descriptions, which was sufficiently clear in text and aptly formalized mathematically. 

Questions/comments for clarification and improvement:

- In CMA-MEGA, could the authors study the effect of the ranking strategy on downstream performance? in particular, it is not obvious to me that prioritizing all exploration candidates over  the delta improvements is a good defacto strategy. If the delta improvement is very large, then deprioritizing it to explore a new cell is not ideal. Similar to explore-exploit strategies in bayesopt, RL algorithms, an acquisition function style approach to scalarize the exploration score and the delta improvement score seems like a better strategy.

- In general, for the QD problem, we expect a Pareto optimal frontier of (max or top-m) QD score vs. coverage. The results in Table 1 focus on a specific point on this frontier. Can we tailor the algorithms to target a specific frontier? (i.e., for x% coverage, what is the best QD score we can obtain? Or vice versa)

- Empirical trends of model performance as a function of number of cells would be interesting to include for testing the scalability of the approaches and also for monitoring the divergence in performance from a baseline gradient-only approach (at the extreme, when number of cells is 1, we only care about the target based objective and the gradient based approach is expected to dominate).

- Latent space illumination is a very important problem and a very convincing motivation for this work. But the empirical investigations seemed lacking. While the authors acknowledge that the results are not very robust to hyperparameters, a further investigation as to analyzing the causes of instability and tricks to alleviate them would make the work more extendable by other practitioners. The current set of results based on 1 elon musk prompt are too limited and potentially subject to cherrypicking.




limitations_and_societal_impact: Yes, the limitations are well described. I would encourage the authors to move the societal impact section from the appendix to the main text, especially given the rising concerns and useful applications of generative modeling. Also, they should include examples of LSI covering diverse demographics and not restrict to Elon Musk.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work introduces  DQD (differentiable quality diversity) problems: quality-diversity (QD) problems when both the objective function and the measures of diversity are differentiable. The running example for this is obtaining a variety of generated images corresponding to the objective "looks like Elon Musk" and diverse with respect to hair and eye color.

The authors then introduce 2 algorithm variants that can take advantage of differentiability to solve the DQD problem. 
- OMG-MEGA uses gradient arborescence to explore based on the diversity measures and optimize with respect to the objective. 
- CMA-MEGA branches based on both objective and diversity measures, but optimizes with respect to the objective. 
Both proposed algorithms are variants on QD solutions (resp. MAP-Elites and and CMA-ME) that do not assume access to gradient information.

Across several experimental evaluations, the authors show that their proposed algorithms, and especially CMA-MEGA, outperform baselines that do not access gradient information, both in terms of maximizing the QD score and in terms of coverage of the search space.
main_review: Originality: This work builds upon prior solutions to building archives for the QD problem, taking advantage of use cases in which gradient information is available for both the QD objective and diversity metrics.

Quality: This work reads as a complete paper, with a careful analysis of the pros and cons of the proposed methods (e.g., computational complexity constraints when applied to experimental design settings). Being unfamiliar with the relevant QD literature, I would have found this paper easier to parse were it to include a more in-depth discussion of several key steps of the proposed algorithm. In particular:
- how important is normalizing the gradient in OMG-MEGA and CMA-MEGA? 
- can you formalize your claim "$\sigma_g$ acts as a learning rate for the gradient step"?

Clarity: Without familiarity with the QD literature, the distinction between the CMA-MEGA and baseline CMA-ME algorithm is difficult to parse. Algorithm 1 could be augmented to include comments indicating deviations from the CMA-ME algorithm for increased clarity. 
I also found the running Elon Musk example somewhat counter-intuitive, and Figure 3 difficult to read (the eye color is not easily visible). I do not know if this is easily feasible, but choosing an example with more phenotype variation (dog breeds?) that is less reliant on small detail variations would have increased readability (in my opinion).

Significance:
The results seem significant to the QD community, as the introduced algorithms, and CMA-MEGA variants in particular, show vastly improved coverage of the search space compared to previous QD baselines.

Minor comments and questions:
- I believe the correct reference for MAP-Elites is [40] as well as [13] (last paragraph, page 1)
- When you mention the QD objective (line 121), are you referring to $f(\theta)$? 
- In Figure 3 of the appendix, it seems that adding the Adam variation can make a significant impact on the coverage. Do you have any intuition why this may be the case?

limitations_and_societal_impact: The limitations and negative societal impact have been discussed in detail.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose a quality diversity framework whether the objective and measure functions are differentiable. They propose two algorithms to this end by augment existing QD algorithms with gradient information and present experimental results on two benchmarks and a latent space illumination task. 
main_review: **Update: I've revised my score from 7 to 8, following the author response.**

**Pros**  
- The idea of DQD proposed by the paper is an intuitive and natural development combining ideas from QD with modern ML driven by first-order optimization. 
- The paper is extremely well-written, with a brief but useful explanation of the background, and the algorithms are carefully described in detail. 
- The DQD benchmark with StyleGAN-CLIP is quite intuitive and is bound to inspire some follow-up work.
- The experimental evaluation is convincing, with the proposed algorithms demonstrating clear wins in high dimensional parameter spaces. 
- The experimental results are presented with a statistical significance analysis -- I commend the authors for this.

**Cons/Questions**  
- My first instinct upon looking at the problem would be to write down an optimization problem which trades-off the objective for diversity, e.g.,maintain a catalog $\theta_1, \cdots, \theta_N$ and iteratively update $\theta_j$ with an approximate solution to $\max_\theta f(\theta) + \lambda \sum_{i \neq j} \Vert m(\theta) - m(\theta_i)\Vert^2$ where $\lambda$ is a trade-off parameter. How is the proposed approach better than this baseline? 
- MEGA takes one gradient step in Eq. (3). What are the pros and cons of taking multiple gradient steps? Would this help experimentally? 
- The proposed strategy to tune the learning rate in Appendix A does not seem scalable. Could the authors suggest some heuristics to choose the learning rate? How would a grid search work in this case? 
- How do you choose the noise variance $\sigma_g^2$? What happens if it is too large or too small? 
- It would be helpful if the authors could show plots with QD score and coverage percentage versus the learning rate and $\sigma_g^2$
- (minor) $w_i$ is undefined in Algorithm 1.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: In the paper "Differentiable Quality-Diversity", the authors propose a new set of algorithms quality diversity algorithms for differentiable problems. Quality-Diversity is a new family of stochastic optimisation algorithms, often applied to black-box optimisation problems, where the goal is to generate a large collection of diverse and high-performing solutions. This paper provides novel approaches to leverage gradient information when the problems considered are first-order differentiable. The proposed methods (CMA-MEGA and OMG-MEGA) demonstrate excellent performance compared to state-of-the-art Quality-Diversity algorithms. 

main_review: The paper is well written and clearly structured. It is pleasant to reach and easy to understand without unnecessary technical definitions and notations.  
The proposed methods make a lot of sense and are a smart way of using the gradient information when available. In particular, the combination of the measure and fitness value in a linear combination which then uses CMA-ES to automatically adjust the weights depending on the state of the archive is a nice way to formalise the problem. The use of the proposed methods in the domain of latent space illumination demonstrates promising results.  

The introduction and literature review about quality diversity is appropriate and is likely understandable by non-expert readers. 

Here are a couple of mostly minor points and inaccuracies: 

1)It might be important to stress that the proposed methods and associated performance improvements are limited to a specific scope of application (first-order differentiable) and that not all problems fall in this category. For instance, the paper says "we focus on domains with differentiable objective and measures, where DQD is applicable.". This is not an intentional choice but imposed by the method. 

2) "MAP-Elites first samples solutions from a fixed distribution θ ∼ N (0, I )". This is not always the case. Multiple papers (in particular from MAP-Elites' authors, including the reference implementation) use a uniform sampling (which require the search space to be bounded). This is a minor detail but can have a significant impact on certain tasks, like the linear projection considered in this paper (a 0 centred normal distribution is likely to make the problem easier). 

3) It is correct to say that PGA-ME combines a gaussian variation operator (the Iso+LineDD operator to be exact) with a policy gradient operator only on the objective. However, PGA-ME does not sum (or successively apply) the two operators on the same individual. Instead, it randomly applies one of those for each new offspring. The OG-ME variant introduced in this paper sums these two operators which is likely to create mutations that are too large to be effective.

4) I did not find the definition of the weights wi used in line 14 of algorithm 1. 

5) I understand the desire to make some benchmarks more challenging, but I think that a 1000dof planar robot is a bit too much in my opinion (in particular when considering that this benchmark was initially based on a real 8dof robot). I hope that in the future the community will come up with realistic domains that will be challenging enough to demonstrate the benefits of such methods. 

6) The value of theta_0 used in the experiments cannot be found in the paper and supplementary material. 

7)Table 1 reports a surprising result: CMA-ME in the two LP domains seems to perform like MAP-Elites and worse than MAP-Elites (line), which goes against the main result of the original CMA-ME paper (which uses the same domains). The same thing is observed in the table provided in the supplementary material. However, I cannot find a discussion about this in the paper.

8)In the sentence "To make the QD-score invariant with respect to the resolution of the archive, we normalize QD-score by the archive size", does "archive size" refer to the maximal archive size, or the number of solutions in the archive? I believe it is the former, but this could be clarified to avoid confusion.  

9) In "Naturally, approaches combining QD and RL have started to emerge [42, 9]. Unlike DQD, these approaches estimate the gradient of the reward function in action space and backpropagate this gradient through a neural network. Our proposed DQD problem differs by leveraging provided gradients for both the objective and measure functions.", it is important to note that QD-RL [9] introduces a diversity gradient (via a diversity critic), and uses both the policy gradient (reward-based) and the diversity gradient. Therefore, it is inaccurate to say that DQD differs from other approaches because it uses gradient at both levels. Additionally, QD-RL is not limited to differentiable problems. 

limitations_and_societal_impact: I cannot identify potential direct societal impacts of their work.
review_title: null

Meta-review: Meta-review of Differentiable Quality Diversity

This paper proposes the first differentiable version of “Quality Diversity” optimization. QD, along with other multi-objective optimization methods, look at generating a large collection of diverse solutions, and are well-explored in the evolutionary computation community, while lesser known in the ML / RL community.

The 3 main contributions of this paper are well-summarized by reviewer fNzE:

1. The paper proposes a new problem setup, called Differentiable Quality Diversity (DQD), where the goal is to optimize for a set of diverse solutions that maximize a target objective. Diversity is formalized by assuming the existence of differentiable measure functions, which intuitively specify an axis of variation and thus far away solutions along in the joint space of these measure functions are more diverse.

2. To solve the DQD, the paper extends the MAP-Elite algorithms from the related and previously studied non-differentiable variant, abbreviated as simply QD. The extensions incorporate gradient information in novel ways, interfacing first-order methods with evolutionary search style procedures.

3. Finally, the authors conduct empirical evaluation of the proposed algorithms which suggests that these combinations of gradient and evolutionary algorithms are indeed better than either approach alone on some toy benchmark tasks and a generative modeling task.

Usually, the work in QD in the EC community focuses on artificial agents (like simulated robots), but here, the work really sheds light on how QD can be applied and effectively used in applications that the DL community is also interested in, like image generation. As Reviewer 4FYU pointed out: *the running example for this is obtaining a variety of generated images corresponding to the objective "looks like Elon Musk" and diverse with respect to hair and eye color*. Reviewer T1rp agrees, and mentions *The DQD benchmark with StyleGAN-CLIP is quite intuitive and is bound to inspire some follow-up work.*

All in all, the reviewers, including myself, agree that this paper is very well written, with good background explanation, well motivated, and presents experiments that are not only convincing, but will attract many folks in the NeurIPS community especially those in generative modelling to start using QD.

Given the feedback about the generally high quality of the paper and potential impact, I would like to recommend acceptance as Spotlight presentation.

Decision: Accept (Oral)