Differentiable Quality Diversity

1 Introduction . We introduce the problem of differentiable quality diversity ( DQD ) and propose the MAP-Elites via a Gradient Arborescence ( MEGA ) algorithm as the first DQD algorithm . Unlike single-objective optimization , quality diversity ( QD ) is the problem of finding a range of high quality solutions that are diverse with respect to prespecified metrics . For example , consider the problem of generating realistic images that match as closely as possible a target text prompt “ Elon Musk ” , but vary with respect to hair and eye color . We can formulate the problem of searching the latent space of a generative adversarial network ( GAN ) [ 26 ] as a QD problem of discovering latent codes that generate images maximizing a matching score for the prompt “ Elon Musk ” , while achieving a diverse range of measures of hair and eye color , assessed by visual classification models [ 51 ] . More generally , the QD objective is to maximize an objective f for each output combination of measure functions mi . A QD algorithm produces an archive of solutions , where the algorithm attempts to discover a representative for each measure output combination , whose f value is as large as possible . While our example problem can be formulated as a QD problem , all current QD algorithms treat the objective f and measure functions mi as a black box . This means , in our example problem , current QD algorithms fail to take advantage of the fact that both f and mi are end-to-end differentiable neural networks . Our proposed differentiable quality diversity ( DQD ) algorithms leverage first-order derivative information to significantly improve the computational efficiency of solving a variety of QD problems where f and mi are differentiable . To solve DQD , we introduce the concept of a gradient arborescence . Like gradient ascent , a gradient arborescence makes greedy ascending steps based on the objective f . Unlike gradient ascent , a gradient arborescence encourages exploration by branching via the measures mi . We adopt the term 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . arborescence from the minimum arborescence problem [ 8 ] in graph theory , a directed counterpart to the minimum spanning tree problem , to emphasize the directedness of the branching search . Our work makes four main contributions . 1 ) We introduce and formalize the problem of differentiable quality diversity ( DQD ) . 2 ) We propose two DQD algorithms : Objective and Measure Gradient MAP-Elites via a Gradient Arborescence ( OMG-MEGA ) , an algorithm based on MAP-Elites [ 13 ] , which branches based on the measures mi but ascends based on the objective function f ; and Covariance Matrix Adaptation MEGA ( CMA-MEGA ) which is based on the CMA-ME [ 19 ] algorithm , and which branches based on the objective-measure space but ascends based on maximizing the QD objective ( Fig . 1 ) . Both algorithms search directly in measure space and leverage the gradients of f and mi to form efficient parameter space steps in θ . 3 ) We show in three different QD domains ( the linear projection , the arm repertoire , and the latent space illumination ( LSI ) domains ) , that DQD algorithms significantly outperform state-of-the-art QD algorithms that treat the objective and measure functions as a black box . 4 ) We demonstrate how searching the latent space of a StyleGAN [ 36 ] in the LSI domain with CMA-MEGA results in a diverse range of high-quality images . 2 Problem Definition . Quality Diversity . The quality diversity ( QD ) problem assumes an objective f : Rn → R in an n-dimensional continuous space Rn and k measures mi : Rn → R or , as a joint measure , m : Rn → Rk . Let S = m ( Rn ) be the measure space formed by the range of m. For each s ∈ S the QD objective is to find a solution θ ∈ Rn such thatm ( θ ) = s and f ( θ ) is maximized . However , we note that Rk is continuous , and an algorithm solving the quality diversity problem would require infinite memory to store all solutions . Thus , QD algorithms in the MAP-Elites [ 43 , 13 ] family approximate the problem by discretizing S via a tessellation method . Let T be the tessellation of S into M cells . We relax the QD objective to find a set of solutions θi , i ∈ { 1 , . . . , M } , such that each θi occupies one unique cell in T . The occupants θi of all M cells form an archive of solutions . Each solution θi has a position in the archivem ( θi ) , corresponding to one out of M cells , and an objective value f ( θi ) . The objective of QD can be rewritten as follows , where the goal is to maximize the objective value for each cell in the archive : max M∑ i=1 f ( θi ) ( 1 ) Differentiable Quality Diversity . We define the differentiable quality diversity ( DQD ) problem , as a QD problem where both the objective f and measures mi are first-order differentiable . 3 Preliminaries . We present several state-of-the-art derivative-free QD algorithms . Our proposed DQD algorithm MEGA builds upon ideas from these works , while introducing measure and objective gradients into the optimization process . MAP-Elites and MAP-Elites ( line ) . MAP-Elites [ 13 , 43 ] first tessellates the measure space S into evenly-spaced grid cells . The upper and lower bounds for m are given as input to constrain S to a finite region . MAP-Elites first samples solutions from a fixed distribution θ ∼ N ( 0 , I ) , and populates an initial archive after computing f ( θ ) andm ( θ ) . Each iteration of MAP-Elites selects λ cells uniformly at random from the archive and perturbs each occupant θi with fixed-variance σ isotropic Gaussian noise : θ′ = θi + σN ( 0 , I ) . Each new candidate solution θ′ is then evaluated and added to the archive if θ′ discovers a new cell or improves an existing cell . The algorithm continues to generate solutions for a specified number of iterations . Later work introduced the Iso+LineDD operator [ 61 ] . The Iso+LineDD operator samples two archive solutions θi and θj , then blends a Gaussian perturbation with a noisy interpolation given hyperparameters σ1 and σ2 : θ′ = θi + σ1N ( 0 , I ) + σ2N ( 0 , 1 ) ( θi − θj ) . In this paper we refer to MAP-Elites with an Iso+LineDD operator as MAP-Elites ( line ) . CMA-ME . Covariance Matrix Adaptation MAP-Elites ( CMA-ME ) [ 19 ] combines the archiving mechanisms of MAP-Elites with the adaptation mechanisms of CMA-ES [ 30 ] . While MAP-Elites creates new solutions by perturbing existing solutions with fixed-variance Gaussian noise , CMA-ME maintains a full-rank Gaussian distribution N ( µ , Σ ) in parameter space Rn . Each iteration of CMA-ME samples λ candidate solutions θi ∼ N ( µ , Σ ) , evaluates each solution , and updates the archive based on the following rule : if there is a previous occupant θp at the same cell , we compute ∆i = f ( θi ) − f ( θp ) , otherwise if the cell is empty we compute ∆i = f ( θi ) . We then rank the sampled solutions by increasing improvement ∆i , with an extra criteria that candidates discovering new cells are ranked higher than candidates that improve existing cells . We then update N ( µ , Σ ) with the standard CMA-ES update rules based on the improvement ranking . CMA-ME restarts when all λ solutions fail to change the archive . On a restart we reset the Gaussian N ( θi , I ) , where θi is an archive solution chosen uniformly at random , and all internal CMA-ES parameters . In the supplemental material , we derive , for the first time , a natural gradient interpretation of CMA-ME ’ s improvement ranking mechanism , based on previous theoretical work on CMA-ES [ 1 ] . 4 Algorithms . We present two variants of our proposed MEGA algorithm : OMG-MEGA and CMA-MEGA . We form each variant by adapting the concept of a gradient arborescence to the MAP-Elites and CMA-ME algorithms , respectively . Finally , we introduce two additional baseline algorithms , OG-MAP-Elites and OG-MAP-Elites ( line ) , which operate only on the gradients of the objective . OMG-MEGA . We first derive the Objective and Measure Gradient MAP-Elites via Gradient Arborescence ( OMG-MEGA ) algorithm from MAP-Elites . First , we observe how gradient information could benefit a QD algorithm . Note that the QD objective is to explore the measure space , while maximizing the objective function f . We observe that maximizing a linear combination of measures : ∑k j=1 cjmj ( θ ) , where c is a k-dimensional vector of coefficients , enables movement in a k-dimensional measure space . Adding the objective function f to the linear sum enables movement in an objective-measure space . Maximizing g with a positive coefficient of f results in steps that increasing f , while the direction of movement in the measure space is determined by the sign and magnitude of the coefficients cj . g ( θ ) = |c0|f ( θ ) + k∑ j=1 cjmj ( θ ) ( 2 ) We can then derive a direction function that perturbs a given solution θ based on the gradient of our linear combination g : ∇g ( θ ) = |c0|∇f ( θ ) + ∑k j=1 cj∇mj ( θ ) . We incorporate the direction function ∇g to derive a gradient-based MAP-Elites variation operator . We observe that MAP-Elites samples a cell θi and perturbs the occupant with Gaussian noise : θ′ = θi + σN ( 0 , I ) . Instead , we sample coefficents c ∼ N ( 0 , σgI ) and step : θ′ = θi+ | c0 |∇f ( θi ) + k∑ j=1 cj∇mj ( θi ) ( 3 ) The value σg acts as a learning rate for the gradient step , because it controls the scale of the coefficients c ∼ N ( 0 , σgI ) . To balance the contribution of each function , we normalize all gradients . In the supplemental material , we further justify gradient normalization and provide an empirical ablation study . Other than our new gradient-based operator , OMG-MEGA is identical to MAP-Elites . CMA-MEGA . Next , we derive the Covariance Matrix Adaptation MAP-Elites via a Gradient Arborescence ( CMA-MEGA ) algorithm from CMA-ME . Fig . 1 shows an overview of the algorithm . First , we note that we sample c in OMG-MEGA from a fixed-variance Gaussian . However , it would be beneficial to select c based on how c , and the subsequent gradient step on θ , improve the QD objective defined in equation 1 . We frame the selection of c as an optimization problem with the objective of maximizing the QD objective ( Eq . 1 ) . We model a distribution of coefficients c as a k+1-dimensional GaussianN ( µ , Σ ) . Given a θ , we can sample c ∼ N ( µ , Σ ) , compute θ′ via Eq . 3 , and adapt N ( µ , Σ ) towards the direction of maximum increase of the QD objective ( see Eq . 1 ) . We follow an evolution strategy approach to model and dynamically adapt the sampling distribution of coefficients N ( µ , Σ ) . We sample a population of λ coefficients from ci ∼ N ( µ , Σ ) and generate λ solutions θi . We then compute ∆i from CMA-ME ’ s improvement ranking for each candidate solution θi . By updating N ( µ , Σ ) with CMA-ES update rules for the ranking ∆i , we dynamically adapt the distribution of coefficients c to maximize the QD objective . Algorithm 1 shows the pseudocode for CMA-MEGA . In line 3 we evaluate the current solution and compute an objective value f , a vector of measure values m , and gradient vectors . As we dynamically adapt the coefficients c , we normalize the objective and measure gradients ( line 4 ) for stability . Because the measure space is tessellated , the measuresm place solution θ into one of the M cells in the archive . We then add the solution to the archive ( line 5 ) , if the solution discovers an empty cell in the archive , or if it improves an existing cell , identically to MAP-Elites . We then use the gradient information to compute a step that maximizes improvement of the archive . In lines 6-12 , we sample a population of λ coefficients from a multi-variate Gaussian retained by CMA-ES , and take a gradient step for each sample . We evaluate each sampled solution θ′i , and compute the improvement ∆i ( line 11 ) . As in CMA-ME , we specify ∆i as the difference in the objective value between the sampled solution θi and the existing solution , if one exists , or as the absolute objective value of the sampled solution if θi belongs to an empty cell . In line 13 , we rank the sampled gradients ∇i based on their respective improvements . As in CMA-ME , we prioritize exploration of the archive by ranking first by their objective values all samples that discover new cells , and subsequently all samples that improve existing cells by their difference in improvement . We then compute an ascending gradient step as a linear combination of gradients ( line 14 ) , following the recombination weights wi from CMA-ES [ 30 ] based on the computed improvement ranking . These weights correspond to the log-likelihood probabilities of the samples in the natural gradient interpretation of CMA-ES [ 1 ] . In line 16 , CMA-ES adapts the multi-variate GaussianN ( µ , Σ ) , as well as internal search parameters p , from the improvement ranking of the coefficients . In the supplemental material , we provide a natural gradient interpretation of the improvement ranking rules of CMA-MEGA , where we show that the coefficient distribution of CMA-MEGA approximates natural gradient steps of maximizing a modified QD objective . CMA-MEGA ( Adam ) . We add an Adam-based variant of CMA-MEGA , where we replace line 15 with an Adam gradient optimization step [ 38 ] . OG-MAP-Elites . To show the importance of taking gradient steps in the measure space , as opposed to only taking gradient steps in objective space and directly perturbing the parameters , we derive two variants of MAP-Elites as a baseline that draw insights from the recently proposed Policy Gradient Algorithm 1 Covariance Matrix Adaptation MAP-Elites via a Gradient Aborescence ( CMA-MEGA ) CMA-MEGA ( evaluate , θ0 , N , λ , η , σg ) input : An evaluation function evaluate which computes the objective , the measures , gradients of the objective and measures , an initial solution θ0 , a desired number of iterations N , a branching population size λ , a learning rate η , and an initial step size for CMA-ES σg . result : Generate N ( λ+ 1 ) solutions storing elites in an archive A . 1 Initialize solution parameters θ to θ0 , CMA-ES parameters µ = 0 , Σ = σgI , and p , where we let p be the CMA-ES internal parameters . 2 for iter ← 1 to N do 3 f , ∇f , m , ∇m ← evaluate ( θ ) 4 ∇f ← normalize ( ∇f ) , ∇m ← normalize ( ∇m ) 5 update_archive ( θ , f , m ) 6 for i← 1 to λ do 7 c ∼ N ( µ , Σ ) 8 ∇i ← c0∇f + ∑k j=1 cj∇mj 9 θ′i ← θ + ∇i 10 f ′ , ∗ , m′ , ∗ ← evaluate ( θ′i ) 11 ∆i ← update_archive ( θ′i , f ′ , m′ ) 12 end 13 rank ∇i by ∆i 14 ∇step ← ∑λ i=1 wi∇rank [ i ] 15 θ ← θ + η∇step 16 Adapt CMA-ES parameters µ , Σ , p based on improvement ranking ∆i 17 if there is no change in the archive then 18 Restart CMA-ES with µ = 0 , Σ = σgI . 19 Set θ to a randomly selected existing cell θi from the archive 20 end 21 end Assisted MAP-Elites ( PGA-ME ) algorithm [ 45 ] . PGA-ME combines the Iso+LineDD operator [ 61 ] with a policy gradient operator only on the objective . Similarly , our proposed Objective-Gradient MAP-Elites ( OG-MAP-Elites ) algorithm combines an objective gradient step with a MAP-Elites style perturbation operator . Each iteration of OG-MAP-Elites samples λ solutions θi from the archive . Each θi is perturbed with Gaussian noise to form a new candidate solution θ′i = θi + σN ( 0 , I ) . OG-MAP-Elites evaluates the solution and updates the archive , exactly as in MAP-Elites . However , OG-MAP-Elites takes one additional step : for each θ′i , the algorithm computes ∇f ( θ′i ) , forms a new solution θ′′i = θ ′ i + η∇f ( θ′i ) with an objective gradient step , and evaluates θ′′i . Finally , we update the archive with all solutions θ′i and θ ′′ i . OG-MAP-Elites ( line ) . Our second baseline , OG-MAP-Elites ( line ) replaces the Gaussian operator with the Iso+LineDD operator [ 61 ] : θ′ = θi + σ1N ( 0 , I ) + σ2N ( 0 , 1 ) ( θi − θj ) . We consider OG-MAP-Elites ( line ) a DQD variant of PGA-ME . However , PGA-ME was designed as a reinforcement learning ( RL ) algorithm , thus many of the advantages gained in RL settings are lost in OG-MAP-Elites ( line ) . We provide a detailed discussion and ablations in the supplemental material .

Summary: The paper makes three primary contributions:  (1) The paper proposes a new problem setup, called Differentiable Quality Diversity (DQD), where the goal is to optimize for a set of diverse solutions that maximize a target objective. Diversity is formalized by assuming the existence of **differentiable** measure functions, which intuitively specify an axis of variation and thus far away solutions along in the joint space of these measure functions are more diverse.  (2) To solve the DQD, the paper extends the MAP-Elite algorithms from the related and previously studied non-differentiable variant, abbreviated as simply QD. The extensions incorporate gradient information in novel ways, interfacing first-order methods with evolutionary search style procedures.   (3) Finally, the authors conduct empirical evaluation of the proposed algorithms which suggests that these combinations of gradient and evolutionary algorithms are indeed better than either approach alone on some toy benchmark tasks and a generative modeling task.