Guided Evolutionary Strategies: Escaping the curse of dimensionality in random search

1 INTRODUCTION . Optimization in machine learning often involves minimizing a cost function where the gradient of the cost with respect to model parameters is known . When gradient information is available , firstorder methods such as gradient descent are popular due to their ease of implementation , memory efficiency , and convergence guarantees ( Sra et al. , 2012 ) . When gradient information is not available , however , we turn to zeroth-order optimization methods , including random search methods such as evolutionary strategies ( Rechenberg , 1973 ; Nesterov & Spokoiny , 2011 ; Salimans et al. , 2017 ) . However , what if only partial gradient information is available ? That is , what if one has access to surrogate gradients that are correlated with the true gradient , but may be biased in some unknown fashion ? Naı̈vely , there are two extremal approaches to optimization with surrogate gradients . On one hand , you could ignore the surrogate gradient information entirely and perform zeroth-order optimization , using methods such as evolutionary strategies to estimate a descent direction . These methods exhibit poor convergence properties when the parameter dimension is large ( Duchi et al. , 2015 ) . On the other hand , you could directly feed the surrogate gradients to a first-order optimization algorithm . However , bias in the surrogate gradients will interfere with optimizing the target problem ( Tucker et al. , 2017 ) . Ideally , we would like a method that combines the complementary strengths of these two approaches : we would like to combine the unbiased descent direction estimated with evolutionary strategies with the low-variance estimate given by the surrogate gradient . In this work , we propose a method for doing this called guided evolutionary strategies ( Guided ES ) . The critical assumption underlying Guided ES is that we have access to surrogate gradient information , but not the true gradient . This scenario arises in a wide variety of machine learning problems , which typically fall into two categories : cases where the true gradient is unknown or not defined , and cases where the true gradient is hard or expensive to compute . Examples of the former include : models with discrete stochastic variables ( where straight through estimators ( Bengio et al. , 2013 ; van den Oord et al. , 2017 ) or Concrete/Gumble-Softmax methods ( Maddison et al. , 2016 ; Jang et al. , 2016 ) are commonly used ) and learned models in reinforcement learning ( e.g . for Q functions ( Watkins & Dayan , 1992 ; Mnih et al. , 2013 ; 2015 ; Lillicrap et al. , 2015 ) or value estimation ( Mnih et al. , 2016 ) ) . For the latter , examples include optimization using truncated backprop through time ( Rumelhart et al. , 1985 ; Williams & Peng , 1990 ; Wu et al. , 2018 ) . Surrogate gradients also arise in situations where the gradients are explicitly modified during training , as in feedback alignment ( Lillicrap et al. , 2014 ) and related methods ( Nøkland , 2016 ; Gilmer et al. , 2017 ) . The key idea in Guided ES is to keep track of a low-dimensional subspace , defined by the recent history of surrogate gradients during optimization , which we call the guiding subspace . We then perform a finite difference random search ( as in evolutionary strategies ) preferentially within this subspace . By concentrating our search samples in a low-dimensional subspace where the true gradient has non-negative support , we dramatically reduce the variance of the search direction . Our contributions in this work are : • a new method for combining surrogate gradient information with random search , • an analysis of the bias-variance tradeoff underlying the technique ( §3.3 ) , • a scheme for choosing optimal hyperparameters for the method ( §3.4 ) , and • applications to example problems ( §4 ) . 2 RELATED WORK . This work builds upon a random search method known as evolutionary strategies ( Rechenberg , 1973 ; Nesterov & Spokoiny , 2011 ) , or ES for short , which generates a descent direction via finite differences over random perturbations of parameters . ES has seen a resurgence in popularity in recent years ( Salimans et al. , 2017 ; Mania et al. , 2018 ) . Our method can primarily be thought of as a modification to ES where we augment the search distribution using surrogate gradients . Extensions of ES that modify the search distribution use natural gradient updates in the search distribution ( Wierstra et al. , 2008 ) or construct non-Gaussian search distributions ( Glasmachers et al. , 2010 ) . The idea of using gradients in concert with evolutionary algorithms was proposed by Lehman et al . ( 2017b ) , who use gradients of a network with respect to its inputs ( as opposed to parameters ) to augment ES . Other methods for adapting the search distribution include covariance matrix adaptation ES ( CMA-ES ) ( Hansen , 2016 ) , which uses the recent history of descent steps to adapt the distribution over parameters , or variational optimization ( Staines & Barber , 2012 ) , which optimizes the parameters of a probability distribution over model weights . Guided ES , by contrast , adapts the search distribution using surrogate gradient information . In addition , we never need to work with or compute a full n× n covariance matrix . 3 GUIDED EVOLUTIONARY STRATEGIES . 3.1 VANILLA ES . We wish to minimize a function f ( x ) over a parameter space in n-dimensions ( x ∈ Rn ) , where∇f is either unavailable or uninformative . A popular approach is to estimate a descent direction with stochastic finite differences ( commonly referred to as evolutionary strategies ( Rechenberg , 1973 ) or random search ( Rastrigin , 1963 ) ) . Here , we use antithetic sampling ( Owen , 2013 ) ( using a pair of function evaluations at x+ and x− ) to reduce variance . This estimator is defined as : g = β 2σ2P P∑ i=1 i ( f ( x+ i ) − f ( x− i ) ) , ( 1 ) where i ∼ N ( 0 , σ2I ) , and P is the number of sample pairs . We will set P to one for all experiments , and when analyzing optimal hyperparameters . The overall scale of the estimate ( β ) and variance of the perturbations ( σ2 ) are constants , to be chosen as hyperparameters . This estimate solely relies on computing 2P function evaluations . However , it tends to have high variance , thus requiring a large number of samples to be practical , and scales poorly with the dimension n. We refer to this estimator as vanilla evolutionary strategies ( or vanilla ES ) in subsequent sections . 3.2 GUIDED SEARCH . Even when we do not have access to ∇f , we frequently have additional information about f , either from prior knowledge or gleaned from previous iterates during optimization . To formalize this , we assume we are given a set of vectors which may correspond to biased or corrupted gradients . That is , these vectors are correlated ( but need not be perfectly aligned ) with the true gradient . If we are given a single vector or surrogate gradient for a given parameter iterate , we can generate a subspace by keeping track of the previous k surrogate gradients encountered during optimization . We use U to denote an n× k orthonormal basis for the subspace spanned by these vectors ( i.e. , UTU = Ik ) . We leverage this information by changing the distribution of i in eq . ( 1 ) to N ( 0 , σ2Σ ) with Σ = α n In + 1− α k UUT , where k and n are the subspace and parameter dimensions , respectively , and α is a hyperparameter that trades off variance between the full parameter space and the subspace . Setting α = 1 recovers the vanilla ES estimator ( and ignores the guiding subspace ) , but as we show choosing α < 1 can result in significantly improved performance . The other hyperparameter is the scale β in ( 1 ) , which controls the size of the estimated descent direction . The parameter σ2 controls the overall scale of the variance , and will drop out of the analysis of the bias and variance below , due to the 1σ2 factor in ( 1 ) . In practice , if f ( x ) is stochastic , then increasing σ2 will dampen noise in the gradient estimate , while decreasing σ2 reduces the error induced by third and higher-order terms in the Taylor expansion of f below . For an exploration of the effects of σ2 in ES , see Lehman et al . ( 2017a ) . Samples of i can be generated efficiently as i = σ √ α n + σ √ 1−α k U ′ where ∼ N ( 0 , In ) and ′ ∼ N ( 0 , Ik ) . Our estimator requires 2P function evaluations in addition to the cost of computing the surrogate gradient . Furthermore , it may be possible to parallelize the forward pass computations . Figure 1a depicts the geometry underlying our method . Instead of the true gradient ( blue arrow ) , we are given a surrogate gradient ( white arrow ) which is correlated with the true gradient . We use this to form a guiding distribution ( denoted with white contours ) and use this to draw samples ( white dots ) which we use as part of a random search procedure . ( Figure 1b demonstrates the performance of the method on a toy problem , and is discussed in §4.1 . ) For the purposes of analysis , suppose ∇f exists . We can approximate the function in the local neighborhood of x using a second order Taylor approximation : f ( x + ) ≈ f ( x ) + T∇f ( x ) + 1 2 T∇2f ( x ) . For the remainder of §3 , we take this second order Taylor expansion to be exact . By substituting this expression into ( 1 ) , we see that our estimate g is equal to g = β σ2P P∑ i=1 ( i T i ) ∇f ( x ) . ( 2 ) Note that even terms in the Taylor expansion cancel out in the expression for g due to antithetic sampling . The computational and memory costs of using Guided ES to compute parameter updates , compared to standard ( vanilla ) ES and gradient descent , are outlined in Appendix D . 3.3 TRADEOFF BETWEEN VARIANCE AND SAFE BIAS . As we have alluded to , there is a bias-variance tradeoff lurking within our estimate g. In particular , by emphasizing the search in the full space ( i.e. , choosing α close to 1 ) , we reduce the bias in our estimate at the cost of increased variance . Emphasizing the search along the guiding subspace ( i.e. , choosing α close to 0 ) will induce a bias in exchange for a potentially large reduction in variance , especially if the subspace dimension k is small relative to the parameter dimension n. Below , we analytically and numerically characterize this tradeoff . Importantly , regardless of the choice of α and β , the Guided ES estimator always provides a descent direction in expectation . The mean of the estimator in eq . ( 2 ) is E [ g ] = βΣ∇f ( x ) corresponds to the gradient multiplied by a positive semi-definite ( PSD ) matrix , thus the update ( −E [ g ] ) remains a descent direction . This desirable property ensures that α trades off variance for “ safe ” bias . That is , the bias will never produce an ascent direction when we are trying to minimize f . The alignment between the k-dimensional orthonormal guiding subspace ( U ) and the true gradient ( ∇f ( x ) ) will be a key quantity for understanding the bias-variance tradeoff . We characterize this alignment using a k-dimensional vector of uncentered correlation coefficients ρ , whose elements are the correlation between the gradient and every column of U . That is , ρi = ∇f ( x ) TU·i ‖∇f ( x ) ‖ . This correlation ‖ρ‖2 varies between zero ( if the gradient is orthogonal to the subspace ) and one ( if the gradient is full contained in the subspace ) . We can evaluate the squared norm of the bias of our estimate g as ‖Bias ( g ) ‖22 = ( E [ g ] −∇f ( x ) ) T ( E [ g ] −∇f ( x ) ) = ∇f ( x ) T ( βΣ− I ) 2∇f ( x ) . ( 3 ) We additionally define the normalized squared bias , b̃ , as the squared norm of the bias divided by the squared norm of the true gradient ( this quantity is independent of the overall scale of the gradient ) . Plugging in our estimate for g from eq . ( 2 ) yields the following expression for the normalized squared bias ( see Appendix A.1 for derivation ) : b̃ = ( β α n − 1 ) 2 + ( β2 ( 1− α ) 2 k2 + 2β ( 1− α ) k ( β α n − 1 ) ) ‖ρ‖22 ( 4 ) where again β is a scale factor and α is part of the parameterization of the covariance matrix that trades off variance in the full parameter space for variance in the guiding subspace ( Σ = αn I + ( 1−α ) k UU T ) . We see that the normalized squared bias consists of two terms : the first is a contribution from the search in the full space and is thus independent of ρ , whereas the second depends on the squared norm of the uncentered correlation , ‖ρ‖22 . In addition to the bias , we are also interested in the variance of our estimate . We use total variance ( i.e. , tr ( Var ( g ) ) ) to quantify the variance of our estimator total variance ≡ tr ( Var ( g ) ) = tr ( E [ ggT ] − E [ g ] E [ g ] T ) = E [ gT g ] − E [ g ] TE [ g ] = β2∇f ( x ) TE [ T T ] ∇f ( x ) − β2∇f ( x ) TΣTΣ∇f ( x ) = ∇f ( x ) T ( β2Σ + β2Σ2 ) ∇f ( x ) , using an identity for the fourth moment of a Gaussian ( see Appendix A.2 ) and the fact that the trace is linear and invariant under cyclic permutations . We are interested in the normalized variance , ṽ , which we define as the quantity above divided by the squared norm of the gradient . Plugging in our estimate g yields the following expression for the normalized variance ( see Appendix A.2 ) : ṽ = β2 ( α2 n2 + α n ) + β2 ( ( 1− α ) 2 k2 + 2 α ( 1− α ) kn + ( 1− α ) k ) ‖ρ‖22 . ( 5 ) Equations ( 4 ) and ( 5 ) quantify the bias and variance of our estimate as a function of the subspace and parameter dimensions ( k and n ) , the parameters of the distribution ( α and β ) , and the correlation ‖ρ‖2 . Note that for simplicity we have set the number of pairs of function evaluations , P , to one . As P increases , the variance will decrease linearly , at the cost of extra function evaluations . Figure 2 explores the tradeoff between normalized bias and variance for different settings of the relevant hyperparameters ( α and β ) for example values of ‖ρ‖2 = 0.23 , k = 3 , and n = 100 . Figure 2c shows the sum of the normalized bias plus variance , the global minimum of which ( blue star ) can be used to choose optimal values for the hyperparameters , discussed in the next section .

Summary: In this manuscript, the authors propose an approach that combines random search with the surrogate gradient information. To this end, the proposed method samples from the subspace of the surrogate gradients. This subspace is constructed by storing the previous surrogate gradients. After several assumptions, the authors also a give a discussion on variance-bias trade-off as well as a discussion on hyperparameter optimization. The manuscript ends with numerical experiments.