Differentiable Quality Diversity

1 Introduction . We introduce the problem of differentiable quality diversity ( DQD ) and propose the MAP-Elites via a Gradient Arborescence ( MEGA ) algorithm as the first DQD algorithm . Unlike single-objective optimization , quality diversity ( QD ) is the problem of finding a range of high quality solutions that are diverse with respect to prespecified metrics . For example , consider the problem of generating realistic images that match as closely as possible a target text prompt “ Elon Musk ” , but vary with respect to hair and eye color . We can formulate the problem of searching the latent space of a generative adversarial network ( GAN ) [ 26 ] as a QD problem of discovering latent codes that generate images maximizing a matching score for the prompt “ Elon Musk ” , while achieving a diverse range of measures of hair and eye color , assessed by visual classification models [ 51 ] . More generally , the QD objective is to maximize an objective f for each output combination of measure functions mi . A QD algorithm produces an archive of solutions , where the algorithm attempts to discover a representative for each measure output combination , whose f value is as large as possible . While our example problem can be formulated as a QD problem , all current QD algorithms treat the objective f and measure functions mi as a black box . This means , in our example problem , current QD algorithms fail to take advantage of the fact that both f and mi are end-to-end differentiable neural networks . Our proposed differentiable quality diversity ( DQD ) algorithms leverage first-order derivative information to significantly improve the computational efficiency of solving a variety of QD problems where f and mi are differentiable . To solve DQD , we introduce the concept of a gradient arborescence . Like gradient ascent , a gradient arborescence makes greedy ascending steps based on the objective f . Unlike gradient ascent , a gradient arborescence encourages exploration by branching via the measures mi . We adopt the term 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . arborescence from the minimum arborescence problem [ 8 ] in graph theory , a directed counterpart to the minimum spanning tree problem , to emphasize the directedness of the branching search . Our work makes four main contributions . 1 ) We introduce and formalize the problem of differentiable quality diversity ( DQD ) . 2 ) We propose two DQD algorithms : Objective and Measure Gradient MAP-Elites via a Gradient Arborescence ( OMG-MEGA ) , an algorithm based on MAP-Elites [ 13 ] , which branches based on the measures mi but ascends based on the objective function f ; and Covariance Matrix Adaptation MEGA ( CMA-MEGA ) which is based on the CMA-ME [ 19 ] algorithm , and which branches based on the objective-measure space but ascends based on maximizing the QD objective ( Fig . 1 ) . Both algorithms search directly in measure space and leverage the gradients of f and mi to form efficient parameter space steps in θ . 3 ) We show in three different QD domains ( the linear projection , the arm repertoire , and the latent space illumination ( LSI ) domains ) , that DQD algorithms significantly outperform state-of-the-art QD algorithms that treat the objective and measure functions as a black box . 4 ) We demonstrate how searching the latent space of a StyleGAN [ 36 ] in the LSI domain with CMA-MEGA results in a diverse range of high-quality images . 2 Problem Definition . Quality Diversity . The quality diversity ( QD ) problem assumes an objective f : Rn → R in an n-dimensional continuous space Rn and k measures mi : Rn → R or , as a joint measure , m : Rn → Rk . Let S = m ( Rn ) be the measure space formed by the range of m. For each s ∈ S the QD objective is to find a solution θ ∈ Rn such thatm ( θ ) = s and f ( θ ) is maximized . However , we note that Rk is continuous , and an algorithm solving the quality diversity problem would require infinite memory to store all solutions . Thus , QD algorithms in the MAP-Elites [ 43 , 13 ] family approximate the problem by discretizing S via a tessellation method . Let T be the tessellation of S into M cells . We relax the QD objective to find a set of solutions θi , i ∈ { 1 , . . . , M } , such that each θi occupies one unique cell in T . The occupants θi of all M cells form an archive of solutions . Each solution θi has a position in the archivem ( θi ) , corresponding to one out of M cells , and an objective value f ( θi ) . The objective of QD can be rewritten as follows , where the goal is to maximize the objective value for each cell in the archive : max M∑ i=1 f ( θi ) ( 1 ) Differentiable Quality Diversity . We define the differentiable quality diversity ( DQD ) problem , as a QD problem where both the objective f and measures mi are first-order differentiable . 3 Preliminaries . We present several state-of-the-art derivative-free QD algorithms . Our proposed DQD algorithm MEGA builds upon ideas from these works , while introducing measure and objective gradients into the optimization process . MAP-Elites and MAP-Elites ( line ) . MAP-Elites [ 13 , 43 ] first tessellates the measure space S into evenly-spaced grid cells . The upper and lower bounds for m are given as input to constrain S to a finite region . MAP-Elites first samples solutions from a fixed distribution θ ∼ N ( 0 , I ) , and populates an initial archive after computing f ( θ ) andm ( θ ) . Each iteration of MAP-Elites selects λ cells uniformly at random from the archive and perturbs each occupant θi with fixed-variance σ isotropic Gaussian noise : θ′ = θi + σN ( 0 , I ) . Each new candidate solution θ′ is then evaluated and added to the archive if θ′ discovers a new cell or improves an existing cell . The algorithm continues to generate solutions for a specified number of iterations . Later work introduced the Iso+LineDD operator [ 61 ] . The Iso+LineDD operator samples two archive solutions θi and θj , then blends a Gaussian perturbation with a noisy interpolation given hyperparameters σ1 and σ2 : θ′ = θi + σ1N ( 0 , I ) + σ2N ( 0 , 1 ) ( θi − θj ) . In this paper we refer to MAP-Elites with an Iso+LineDD operator as MAP-Elites ( line ) . CMA-ME . Covariance Matrix Adaptation MAP-Elites ( CMA-ME ) [ 19 ] combines the archiving mechanisms of MAP-Elites with the adaptation mechanisms of CMA-ES [ 30 ] . While MAP-Elites creates new solutions by perturbing existing solutions with fixed-variance Gaussian noise , CMA-ME maintains a full-rank Gaussian distribution N ( µ , Σ ) in parameter space Rn . Each iteration of CMA-ME samples λ candidate solutions θi ∼ N ( µ , Σ ) , evaluates each solution , and updates the archive based on the following rule : if there is a previous occupant θp at the same cell , we compute ∆i = f ( θi ) − f ( θp ) , otherwise if the cell is empty we compute ∆i = f ( θi ) . We then rank the sampled solutions by increasing improvement ∆i , with an extra criteria that candidates discovering new cells are ranked higher than candidates that improve existing cells . We then update N ( µ , Σ ) with the standard CMA-ES update rules based on the improvement ranking . CMA-ME restarts when all λ solutions fail to change the archive . On a restart we reset the Gaussian N ( θi , I ) , where θi is an archive solution chosen uniformly at random , and all internal CMA-ES parameters . In the supplemental material , we derive , for the first time , a natural gradient interpretation of CMA-ME ’ s improvement ranking mechanism , based on previous theoretical work on CMA-ES [ 1 ] . 4 Algorithms . We present two variants of our proposed MEGA algorithm : OMG-MEGA and CMA-MEGA . We form each variant by adapting the concept of a gradient arborescence to the MAP-Elites and CMA-ME algorithms , respectively . Finally , we introduce two additional baseline algorithms , OG-MAP-Elites and OG-MAP-Elites ( line ) , which operate only on the gradients of the objective . OMG-MEGA . We first derive the Objective and Measure Gradient MAP-Elites via Gradient Arborescence ( OMG-MEGA ) algorithm from MAP-Elites . First , we observe how gradient information could benefit a QD algorithm . Note that the QD objective is to explore the measure space , while maximizing the objective function f . We observe that maximizing a linear combination of measures : ∑k j=1 cjmj ( θ ) , where c is a k-dimensional vector of coefficients , enables movement in a k-dimensional measure space . Adding the objective function f to the linear sum enables movement in an objective-measure space . Maximizing g with a positive coefficient of f results in steps that increasing f , while the direction of movement in the measure space is determined by the sign and magnitude of the coefficients cj . g ( θ ) = |c0|f ( θ ) + k∑ j=1 cjmj ( θ ) ( 2 ) We can then derive a direction function that perturbs a given solution θ based on the gradient of our linear combination g : ∇g ( θ ) = |c0|∇f ( θ ) + ∑k j=1 cj∇mj ( θ ) . We incorporate the direction function ∇g to derive a gradient-based MAP-Elites variation operator . We observe that MAP-Elites samples a cell θi and perturbs the occupant with Gaussian noise : θ′ = θi + σN ( 0 , I ) . Instead , we sample coefficents c ∼ N ( 0 , σgI ) and step : θ′ = θi+ | c0 |∇f ( θi ) + k∑ j=1 cj∇mj ( θi ) ( 3 ) The value σg acts as a learning rate for the gradient step , because it controls the scale of the coefficients c ∼ N ( 0 , σgI ) . To balance the contribution of each function , we normalize all gradients . In the supplemental material , we further justify gradient normalization and provide an empirical ablation study . Other than our new gradient-based operator , OMG-MEGA is identical to MAP-Elites . CMA-MEGA . Next , we derive the Covariance Matrix Adaptation MAP-Elites via a Gradient Arborescence ( CMA-MEGA ) algorithm from CMA-ME . Fig . 1 shows an overview of the algorithm . First , we note that we sample c in OMG-MEGA from a fixed-variance Gaussian . However , it would be beneficial to select c based on how c , and the subsequent gradient step on θ , improve the QD objective defined in equation 1 . We frame the selection of c as an optimization problem with the objective of maximizing the QD objective ( Eq . 1 ) . We model a distribution of coefficients c as a k+1-dimensional GaussianN ( µ , Σ ) . Given a θ , we can sample c ∼ N ( µ , Σ ) , compute θ′ via Eq . 3 , and adapt N ( µ , Σ ) towards the direction of maximum increase of the QD objective ( see Eq . 1 ) . We follow an evolution strategy approach to model and dynamically adapt the sampling distribution of coefficients N ( µ , Σ ) . We sample a population of λ coefficients from ci ∼ N ( µ , Σ ) and generate λ solutions θi . We then compute ∆i from CMA-ME ’ s improvement ranking for each candidate solution θi . By updating N ( µ , Σ ) with CMA-ES update rules for the ranking ∆i , we dynamically adapt the distribution of coefficients c to maximize the QD objective . Algorithm 1 shows the pseudocode for CMA-MEGA . In line 3 we evaluate the current solution and compute an objective value f , a vector of measure values m , and gradient vectors . As we dynamically adapt the coefficients c , we normalize the objective and measure gradients ( line 4 ) for stability . Because the measure space is tessellated , the measuresm place solution θ into one of the M cells in the archive . We then add the solution to the archive ( line 5 ) , if the solution discovers an empty cell in the archive , or if it improves an existing cell , identically to MAP-Elites . We then use the gradient information to compute a step that maximizes improvement of the archive . In lines 6-12 , we sample a population of λ coefficients from a multi-variate Gaussian retained by CMA-ES , and take a gradient step for each sample . We evaluate each sampled solution θ′i , and compute the improvement ∆i ( line 11 ) . As in CMA-ME , we specify ∆i as the difference in the objective value between the sampled solution θi and the existing solution , if one exists , or as the absolute objective value of the sampled solution if θi belongs to an empty cell . In line 13 , we rank the sampled gradients ∇i based on their respective improvements . As in CMA-ME , we prioritize exploration of the archive by ranking first by their objective values all samples that discover new cells , and subsequently all samples that improve existing cells by their difference in improvement . We then compute an ascending gradient step as a linear combination of gradients ( line 14 ) , following the recombination weights wi from CMA-ES [ 30 ] based on the computed improvement ranking . These weights correspond to the log-likelihood probabilities of the samples in the natural gradient interpretation of CMA-ES [ 1 ] . In line 16 , CMA-ES adapts the multi-variate GaussianN ( µ , Σ ) , as well as internal search parameters p , from the improvement ranking of the coefficients . In the supplemental material , we provide a natural gradient interpretation of the improvement ranking rules of CMA-MEGA , where we show that the coefficient distribution of CMA-MEGA approximates natural gradient steps of maximizing a modified QD objective . CMA-MEGA ( Adam ) . We add an Adam-based variant of CMA-MEGA , where we replace line 15 with an Adam gradient optimization step [ 38 ] . OG-MAP-Elites . To show the importance of taking gradient steps in the measure space , as opposed to only taking gradient steps in objective space and directly perturbing the parameters , we derive two variants of MAP-Elites as a baseline that draw insights from the recently proposed Policy Gradient Algorithm 1 Covariance Matrix Adaptation MAP-Elites via a Gradient Aborescence ( CMA-MEGA ) CMA-MEGA ( evaluate , θ0 , N , λ , η , σg ) input : An evaluation function evaluate which computes the objective , the measures , gradients of the objective and measures , an initial solution θ0 , a desired number of iterations N , a branching population size λ , a learning rate η , and an initial step size for CMA-ES σg . result : Generate N ( λ+ 1 ) solutions storing elites in an archive A . 1 Initialize solution parameters θ to θ0 , CMA-ES parameters µ = 0 , Σ = σgI , and p , where we let p be the CMA-ES internal parameters . 2 for iter ← 1 to N do 3 f , ∇f , m , ∇m ← evaluate ( θ ) 4 ∇f ← normalize ( ∇f ) , ∇m ← normalize ( ∇m ) 5 update_archive ( θ , f , m ) 6 for i← 1 to λ do 7 c ∼ N ( µ , Σ ) 8 ∇i ← c0∇f + ∑k j=1 cj∇mj 9 θ′i ← θ + ∇i 10 f ′ , ∗ , m′ , ∗ ← evaluate ( θ′i ) 11 ∆i ← update_archive ( θ′i , f ′ , m′ ) 12 end 13 rank ∇i by ∆i 14 ∇step ← ∑λ i=1 wi∇rank [ i ] 15 θ ← θ + η∇step 16 Adapt CMA-ES parameters µ , Σ , p based on improvement ranking ∆i 17 if there is no change in the archive then 18 Restart CMA-ES with µ = 0 , Σ = σgI . 19 Set θ to a randomly selected existing cell θi from the archive 20 end 21 end Assisted MAP-Elites ( PGA-ME ) algorithm [ 45 ] . PGA-ME combines the Iso+LineDD operator [ 61 ] with a policy gradient operator only on the objective . Similarly , our proposed Objective-Gradient MAP-Elites ( OG-MAP-Elites ) algorithm combines an objective gradient step with a MAP-Elites style perturbation operator . Each iteration of OG-MAP-Elites samples λ solutions θi from the archive . Each θi is perturbed with Gaussian noise to form a new candidate solution θ′i = θi + σN ( 0 , I ) . OG-MAP-Elites evaluates the solution and updates the archive , exactly as in MAP-Elites . However , OG-MAP-Elites takes one additional step : for each θ′i , the algorithm computes ∇f ( θ′i ) , forms a new solution θ′′i = θ ′ i + η∇f ( θ′i ) with an objective gradient step , and evaluates θ′′i . Finally , we update the archive with all solutions θ′i and θ ′′ i . OG-MAP-Elites ( line ) . Our second baseline , OG-MAP-Elites ( line ) replaces the Gaussian operator with the Iso+LineDD operator [ 61 ] : θ′ = θi + σ1N ( 0 , I ) + σ2N ( 0 , 1 ) ( θi − θj ) . We consider OG-MAP-Elites ( line ) a DQD variant of PGA-ME . However , PGA-ME was designed as a reinforcement learning ( RL ) algorithm , thus many of the advantages gained in RL settings are lost in OG-MAP-Elites ( line ) . We provide a detailed discussion and ablations in the supplemental material .

Summary: This work introduces  DQD (differentiable quality diversity) problems: quality-diversity (QD) problems when both the objective function and the measures of diversity are differentiable. The running example for this is obtaining a variety of generated images corresponding to the objective "looks like Elon Musk" and diverse with respect to hair and eye color.  The authors then introduce 2 algorithm variants that can take advantage of differentiability to solve the DQD problem.  - OMG-MEGA uses gradient arborescence to explore based on the diversity measures and optimize with respect to the objective.  - CMA-MEGA branches based on both objective and diversity measures, but optimizes with respect to the objective.  Both proposed algorithms are variants on QD solutions (resp. MAP-Elites and and CMA-ME) that do not assume access to gradient information.  Across several experimental evaluations, the authors show that their proposed algorithms, and especially CMA-MEGA, outperform baselines that do not access gradient information, both in terms of maximizing the QD score and in terms of coverage of the search space.