Improving Gradient Estimation in Evolutionary Strategies With Past Descent Directions

1 INTRODUCTION . Evolutionary Strategies ( ES ) ( 1 ; 2 ; 3 ) are a black-box optimization technique , that estimate the gradient of some objective function with respect to the parameters by evaluating parameter perturbations in random directions . The benefits of using ES in Reinforcement Learning ( RL ) were exhibited in ( 4 ) . ES approaches are highly parallelizable and account for robust learning , while having decent data-efficiency . Moreover , black-box optimization techniques like ES do not require propagation of gradients , are tolerant to long time horizons , and do not suffer from sparse reward distributions ( 4 ) . This lead to a successful application of ES in variety of different RL settings ( 5 ; 6 ; 7 ; 8 ) . Applications of ES outside RL include for example meta learning ( 9 ) . In many scenarios , the true gradient is impossible to compute , however surrogate gradients are available . Here , we use the term surrogate gradients for directions that are correlated but usually not equal to the true gradient , e.g . they might be biased or unbiased approximations of the gradient . Such scenarios include models with discrete stochastic variables ( 10 ) , learned models in RL like Q-learning ( 11 ) , truncated backpropagation through time ( 12 ) and feedback alignment ( 13 ) , see ( 14 ) for a detailed exhibition . If surrogate gradients are available , it is beneficial to preferentially sample parameter perturbations from the subspace defined by these directions ( 14 ) . The proposed algorithm ( 14 ) requires knowing in advance the quality of the surrogate gradient , does not always provide a descent direction that is better than the surrogate gradient , and it remains open how to obtain such surrogate gradients in general settings . In deep learning in general , experimental evidence has established that higher order derivatives are usually `` well behaved '' , in which case gradients of consecutive parameter updates correlate and applying momentum speeds up convergence ( 15 ; 16 ; 17 ) . These observations suggest that past update directions are promising candidates for surrogate gradients . In this work , we extend the line of research of ( 14 ) . Our contribution is threefold : • First , we show theoretically how to optimally combine the surrogate gradient directions with random search directions . More precisely , our approach computes the direction of the subspace spanned by the evaluated search directions that is most aligned with the true gradient . Our gradient estimator does not need to know the quality of the surrogate gradients and always provides a descent direction that is more aligned with the true gradient than the surrogate gradient . • Second , above properties of our gradient estimator allow us to iteratively use the last update direction as a surrogate gradient for our gradient estimator . Repeatedly using the last update direction as a surrogate gradient will aggregate information about the gradient over time and results in improved gradient estimates . In order to demonstrate how the gradient estimate improves over time , we prove fast convergence to the true gradient for linear functions and show , that under simplifying assumptions , it offers an improvement over ES that depends on the Hessian for general functions . • Third , we validate experimentally that these results transfer to practice , that is , the proposed approach computes more accurate gradients than standard ES . We observe that our algorithm considerably improves gradient estimation on the MNIST task compared to standard ES and that it improves convergence speed and performance on the tested Roboschool reinforcement learning environments . 2 RELATED WORK . Evolutionary strategies ( 1 ; 2 ; 3 ) are black box optimization techniques that approximate the gradient by sampling finite differences in random directions in parameter space . Promising potential of ES for the optimization of neural networks used for RL was demonstrated in ( 4 ) . They showed that ES gives rise to efficient training despite the noisy gradient estimates that are generated from a much smaller number of samples than the dimensionality of parameter space . This placed ES on a prominent spot in the RL tool kit ( 5 ; 6 ; 7 ; 8 ) . The history of descent directions was previously used to adapt the search distribution in covariance matrix adaptation ES ( CMA-ES ) ( 18 ) . CMA-ES constructs a second-order model of the underlying objective function and samples search directions and adapts step size according to it . However , maintaining the full covariance matrix makes the algorithm quadratic in the number of parameters , and thus impractical for high-dimensional spaces . Linear time approximations of CMA-ES like diagonal approximations of the covariance matrix ( 19 ) often do not work well , in the sense that their gradient estimates do not converge to the true gradient even if the true gradient does not change over time . Our approach differs as we simply improve the gradient estimation and then feed the gradient estimate to a first-order optimization algorithm . Our work is inspired by the line of research of ( 14 ) , where surrogate gradient directions are used to improve gradient estimations by ’ elongating ’ the search space along these directions . That approach has two shortcomings . First , the bias of the surrogate gradients needs to be known to adapt the covariance matrix . Second , once the bias of the surrogate gradient is too small , the algorithm will not find a better descent direction than the surrogate gradient . Another related area of research investigates how to use momentum for the optimization of deep neural networks . Applying different kinds of momentum has become one of the standard tools in current deep learning and it has been shown to speed-up learning in a very wide range of tasks ( 20 ; 16 ; 17 ) . This hints , that for many problems the higher-order terms in deep learning models are `` well-behaved '' and thus , the gradients do not change too drastically after parameter updates . While these approaches use momentum for parameter updates , our approach can be seen as a form of momentum when sampling directions from the search space of ES . 3 GRADIENT ESTIMATION . We aim at minimizing a function f : Rn → R by steepest descent . In scenarios where the gradient 5f does not exist or is inefficient to compute , we are interested in obtaining some estimate of the ( smoothed ) gradient of f that provides a good parameter update direction . 3.1 THE ES GRADIENT ESTIMATOR . ES considers the function fσ that is obtained by Gaussian smoothing fσ ( θ ) = E ∼N ( 0 , I ) [ f ( θ + σ ) ] , where σ is a parameter modulating the size of the smoothing area and N ( 0 , I ) is the n-dimensional Gaussian distribution with 0 being the all 0 vector and I being the n-dimensional identity matrix . The gradient of fσ with respect to parameters θ is given by 5fσ = 1 σ E ∼N ( 0 , I ) [ f ( θ + σ ) ] , which can be sampled by a Monte Carlo estimator , see ( 5 ) . Often antithetic sampling is used , as it reduces variance ( 5 ) . The antithetic ES gradient estimator using P samples is given by gES = 1 2σP P∑ i=1 ( f ( θ + σ i ) − f ( θ − σ i ) ) i , ( 1 ) where i are independently sampled from N ( 0 , I ) for i ∈ { 1 , . . . , P } . This gradient estimator has been shown to be effective in RL settings ( 4 ) . 3.2 OUR ONE STEP GRADIENT ESTIMATOR . We first give some intuition before presenting our gradient estimator formally . Given one surrogate gradient direction ζ , our one step gradient estimator applies the following sampling strategy . First , it estimates how much the gradient points into the direction of ζ by antithetically evaluating f in the direction of ζ . Second , it estimates the part of5f that is orthogonal to ζ by evaluating random , pairwise orthogonal search directions that are orthogonal to ζ . In this way , our estimator detects the optimal lengths of the parameter update step into both the surrogate direction and the evaluated orthogonal directions ( e.g . if ζ and 5f are parallel , the update step is parallel to ζ , and if they are orthogonal the step into direction ζ has length 0 ) . Additionally , if the surrogate direction and the gradient are not perfectly aligned , then the gradient estimate almost surely improves over the surrogate direction due to the contribution from the evaluated directions orthogonal to ζ . In the following we define our estimator formally and prove that the estimated direction possesses best possible alignment with the gradient that can be achieved with our sampling scheme . We assume that k pairwise orthogonal surrogate gradient directions ζ1 , . . . , ζk are given to our estimator . Denote by Rζ the subspace of Rn that is spanned by the ζi , and by R⊥ζ the subspace that is orthogonal to Rζ . Further , for vectors v and5f , we denote by v̂ and 5̂f the normalized vector v ‖v‖ and 5f ‖5f‖ , respectively . Let ̂ 1 , . . . , ̂P be random orthogonal unit vectors from R⊥ζ . Then , our estimator is defined as g⊥ = k∑ i=1 f ( θ + σζ̂i ) − f ( θ − σζ̂i ) 2σ ζ̂i + P∑ i=1 f ( θ + σ̂i ) − f ( θ − σ̂i ) 2σ ̂i . ( 2 ) We write5f = 5f‖ζ +5f⊥ζ , where5f‖ζ and5f⊥ζ are the projections of5f on Rζ and R⊥ζ , respectively . In essence , the first sum in ( 2 ) computes5f‖ζ by assessing the quality of each surrogate gradient direction , and the second sum estimates5f⊥ζ similar to an orthogonalized antithetic ES gradient estimator , that samples directions from R⊥ζ , see ( 5 ) . We remark that we require pairwise orthogonal unit directions ̂i for the optimality proof . Due to the orthogonality of the directions , no normalization factor like the 1/P factor in ( 1 ) is required in ( 2 ) . In practice , the dimensionality n is often much larger than P . Then , sampling pairwise orthogonal unit vecotrs i is nearly identical to sampling the is from a N ( 0 , I ) distribution , because in high-dimensional space the norm of i ∼ N ( 0 , I ) is highly concentrated around 1 and the cosine of two such random vectors is highly concentrated around 0 . For the sake of analysis , we assume that f is differentiable and that the second order approximation f ( x + ) ≈ f ( x ) + 〈 ,5f ( x ) 〉 + TH ( x ) , where H ( x ) denotes the Hessian matrix of f at x , is exact . This assumption implies that f ( θ + σ̂ ) − f ( θ − σ̂ ) 2σ = 〈5f ( θ ) , ̂〉 , ( 3 ) because the even terms cancel for antithetic sampling . The following proposition and theorems provide theoretical understanding , how our gradient estimation scheme improves gradient estimation in this smooth , noise-free setting . In the following , we will omit the θ in5f ( θ ) . Our first proposition states that g⊥ computes the direction in the subspace spanned by ζ1 , . . . , ζk , 1 , . . . , P that is most aligned with5f . Proposition 1 ( Optimality of g⊥ ) . Let ζ1 , . . . , ζk , 1 , . . . , P be pairwise orthogonal vectors in Rn . Then , g⊥ = ∑k i=1〈5f , ζ̂i〉ζ̂i + ∑P i=1〈5f , ̂i〉̂i computes the projection of 5f on the subspace spanned by ζ1 , . . . , ζk , 1 , . . . , P . Especially , = g⊥ is the vector of that subspace that maximizes the cosine 〈5̂f , ̂〉 between5f and . Moreover , the squared cosine between g⊥ and5f is given by 〈5̂f , ĝour〉2 = k∑ i=1 〈5̂f , ζ̂i〉2 + P∑ i=1 〈5̂f , ̂i〉2 . ( 4 ) We remark that when evaluating 〈5f , vi〉 for arbitrary directions vi , no information about search directions orthogonal to the subspace spanned by the vis is obtained . Therefore , one can only hope for finding the best approximation of 5f lying within the subspace spanned by the vis , which is accomplished by g⊥ . The proof of Proposition 1 follows easily from the Cauchy-Schwarz inequality and is given in the appendix .

Summary: This paper addresses the issue of noisy gradient estimation in a type of evolution strategies popularized by the open AI's reinforcement learning paper. It is a follow-up paper of reference [14], and try to analyze the optimality of the gradient estimation. The goal of the paper is well stated and well motivated. The paper itself is well-organized. However, the novelty of this work is not sufficiently high and its usefulness is questionable.