Learning to Guide Random Search

We are interested in derivative-free optimization of high-dimensional functions . The sample complexity of existing methods is high and depends on problem dimensionality , unlike the dimensionality-independent rates of first-order methods . The recent success of deep learning suggests that many datasets lie on low-dimensional manifolds that can be represented by deep nonlinear models . We therefore consider derivative-free optimization of a high-dimensional function that lies on a latent low-dimensional manifold . We develop an online learning approach that learns this manifold while performing the optimization . In other words , we jointly learn the manifold and optimize the function . Our analysis suggests that the presented method significantly reduces sample complexity . We empirically evaluate the method on continuous optimization benchmarks and high-dimensional continuous control problems . Our method achieves significantly lower sample complexity than Augmented Random Search , Bayesian optimization , covariance matrix adaptation ( CMA-ES ) , and other derivative-free optimization algorithms . 1 INTRODUCTION . A typical approach to machine learning problems is to define an objective function and optimize it over a dataset . First-order optimization methods are widely used for this purpose since they can scale to high-dimensional problems and their convergence rates are independent of problem dimensionality in most cases . However , gradients are not available in many important settings such as control , black-box optimization , and interactive learning with humans in the loop . Derivativefree optimization ( DFO ) can be used to tackle such problems . The challenge is that the sample complexity of DFO scales poorly with the problem dimensionality . The design of DFO methods that solve high-dimensional problems with low sample complexity is a major open problem . The success of deep learning methods suggests that high-dimensional data that arises in real-world settings can commonly be represented in low-dimensional spaces via learned nonlinear features . In other words , while the problems of interest are high-dimensional , the data typically lies on lowdimensional manifolds . If we could perform the optimization directly in the manifold instead of the full space , intuition suggests that we could reduce the sample complexity of DFO methods since their convergence rates are generally a function of the problem dimensionality ( Nesterov & Spokoiny , 2017 ; Dvurechensky et al. , 2018 ) . In this paper , we focus on high-dimensional data distributions that are drawn from low-dimensional manifolds . Since the manifold is typically not known prior to the optimization , we pose the following question . Can we develop an adaptive derivative-free optimization algorithm that learns the manifold in an online fashion while performing the optimization ? There exist DFO methods that aim to identify a low-dimensional search space ( Maheswaranathan et al. , 2019 ; Choromanski et al. , 2019 ) . However , they are limited to linear subspaces . In contrast , we propose to use expressive nonlinear models ( specifically , neural networks ) to represent the manifold . Our approach not only increases expressivity but also enables utilization of domain knowledge concerning the geometry of the problem . For example , if the function of interest is known to be translation invariant , convolutional networks can be used to represent the underlying manifold structure . On the other hand , the high expressive power and flexibility brings challenges . Our approach requires solving for the parameters of the nonlinear manifold at each iteration of the optimization . To address this , we develop an efficient online method that learns the underlying manifold while the function is being optimized . We specifically consider random search methods and extend them to the nonlinear manifold learning setting . Random search methods choose a set of random directions and perform perturbations to the current iterate in these directions . Differences of the function values computed at perturbed points are used to compute an estimator for the gradient of the function . We first extend this to random search over a known manifold and show that sampling directions in the tangent space of the manifold provides a similar estimate . We then propose an online learning method that estimates this manifold while jointly performing the optimization . We theoretically analyze sample complexity and show that our method reduces it . We conduct extensive experiments on continuous control problems , continuous optimization benchmarks , and gradient-free optimization of an airfoil . The results indicate that our method significantly outperforms state-of-the-art derivative-free optimization algorithms from multiple research communities . 2 PRELIMINARIES . We are interested in high-dimensional stochastic optimization problems of the form min x∈Rd f ( x ) = Eξ [ F ( x , ξ ) ] , ( 1 ) where x is the optimization variable and f : Rd → R is the function of interest , which is defined as expectation over a noise variable ξ . We assume that the stochastic function is bounded ( |F ( x , ξ ) | ≤ Ω ) , L-Lipschitz , and µ-smooth1 with respect to x for all ξ , and has uniformly bounded variance ( Eξ [ ( F ( x , ξ ) − f ( x ) ) 2 ] ≤ VF ) . In DFO , we have no access to the gradients . Instead , we only have zeroth-order access by evaluating the function F ( i.e . sampling F ( x , ξ ) for the input x ) . We are specifically interested in random search methods in which an estimate of the gradient is computed using function evaluations at points randomly sampled around the current iterate . Before we formalize this , we introduce some definitions . Denote the d-dimensional unit sphere and unit ball by Sd−1 and Bd , respectively . We define a smoothed function following Flaxman et al . ( 2005 ) . For a function f : Rd → R , its δ-smoothed version is f̂ ( x ) = Ev∼Bd [ f ( x + δv ) ] . The main workhorse of random search is the following result by Flaxman et al . ( 2005 ) . Let s be a random vector sampled from the uniform distribution over Sd−1 . Then f ( x + δs ) s is an unbiased estimate of the gradient of the smoothed function : Eξ , s∈Sd−1 [ F ( x + δs , ξ ) s ] = δ d ∇xf̂ ( x ) . ( 2 ) We use antithetic samples since this is known to decrease variance ( Kroese et al. , 2013 ) and define the final gradient estimator as y ( x , s ) = ( F ( x + δs , ξ ) − F ( x− δs , ξ ) ) s. Extending ( 2 ) to the antithetic case , Eξ , s∈Sd−1 [ y ( x , s ) ] = 2δd ∇xf̂ ( x ) . A simple way to optimize the function of interest is to use the gradient estimate in stochastic gradient descent ( SGD ) , as summarized in Algorithm 1 . This method has been analyzed in various forms and its convergence is characterized well for nonconvex smooth functions . We restate the convergence rate and defer the constants and proof to Appendix A.2 . Proposition 1 ( Flaxman et al. , 2005 ; Vemula et al. , 2019 ) . Let f ( x ) be differentiable , L-Lipschitz , and µ-smooth . Consider running random search ( Algorithm 1 ) for T steps . Let k = 1 for simplicity . Then 1 T T∑ t=1 E‖∇xf ( xt ) ‖22 ≤ O ( T− 1 2 d+ T− 1 3 d 2 3 ) . 3 ONLINE LEARNING TO GUIDE RANDOM SEARCH . Proposition 1 implies that the sample complexity of random search scales linearly with the dimensionality . This dependency is problematic when the function of interest is high-dimensional . We argue that in many practical problems , the function of interest lies on a low-dimensional nonlinear 1L-Lipschitz and µ-smooth : |f ( x ) −f ( y ) | ≤ L‖x−y‖2 and ‖∇xf ( x ) −∇xf ( y ) ‖2 ≤ µ‖x−y‖2 ∀x , y Algorithm 1 Random Search 1 : for t = 1 to T do 2 : gt , = GRADEST ( xt , δ ) 3 : xt+1 = xt − αgt 4 : end for Algorithm 2 Manifold Random Search 1 : for t = 1 to T do 2 : gt , = MANIFOLDGRADEST ( xt , J ( xt ; θ ? ) ) 3 : xt+1 = xt − αgt 4 : end for 1 : procedure GRADEST ( x , δ ) 2 : Sample : s1 , . . . , sk ∼ Sd−1 3 : Query : yi = f ( x + δsi ) − f ( x− δsi ) 4 : Estimator : g = d2δ ∑k i=1 yisi 5 : return g , { si } i∈ [ k ] 6 : end procedure 1 : procedure MANIFOLDGRADEST ( x , θ , δ ) 2 : Normalize : Jq = GramSchmidt ( J ) 3 : Sample : s1 , . . . , sk ∼ Sn−1 4 : Query : yi = f ( x + δJqsi ) − f ( x− δJqsi ) 5 : Estimator : g = n2δ ∑k i=1 yiJqsi 6 : return g , { Jqsi } i∈ [ k ] 7 : end procedure manifold . This structural assumption will allow us to significantly reduce the sample complexity of random search , without knowing the manifold a priori . Assume that the function of interest is defined on an n-dimensional manifold ( n d ) and this manifold can be defined via a nonlinear parametric family ( e.g . a neural network ) . Formally , we are interested in derivative-free optimization of functions with the following properties : • Smoothness : F ( · , ξ ) : Rd → R is µ-smooth and L-Lipschitz for all ξ . • Manifold : F ( · , ξ ) is defined on an n-dimensional manifoldM for all ξ . • Representability : The manifoldM and the function of interest can be represented using parametrized function classes r ( · ; θ ) and g ( · ; ψ ) . Formally , given ξ , there exist θ ? and ψ ? such that F ( x , ξ ) = g ( r ( x ; θ ? ) ; ψ ? ) ∀x ∈ Rd . We will first consider an idealized setting where the manifold is already known ( i.e . we know θ ? ) . Then we will extend the developed method to the practical setting where the manifold is not known in advance and must be estimated with no prior knowledge as the optimization progresses . 3.1 WARM-UP : RANDOM SEARCH OVER A KNOWN MANIFOLD . If the manifold is known a priori , we can perform random search directly over the manifold instead of the full space . Consider the chain rule applied to g ( r ( x ; θ ) ; ψ ) as∇xf ( x ) = J ( x ; θ ? ) ∇rg ( r ; ψ ) , where J ( x ; θ ? ) = ∂r ( x ; θ ? ) /∂x and r = r ( x , θ ? ) . The gradient of the function of interest lies in the column space of the Jacobian of the parametric family . In light of this result , we can perform random search in the column space of the Jacobian , which has lower dimensionality than the full space . For numerical stability , we will first orthonormalize the Jacobian using the Gram-Schmidt procedure , and perform the search in the column space of this orthonormal matrix since it spans the same space . We denote the orthonormalized version of J ( x ; θ ? ) by Jq ( x ; θ ? ) . In order to perform random search , we sample an n-dimensional vector uniformly ( s̃ ∼ Sn−1 ) and lift it to the input space via Jq ( x ; θ ? ) s̃ . As a consequence of the manifold Stokes ’ theorem , using the lifted vector as a random direction gives an unbiased estimate of the gradient of the smoothed function as Eξ , s∈Sn−1 [ y ( x , Jq ( x ; θ ? ) s̃ ) ] = 2δ n ∇xf̃θ ? ( x ) , ( 3 ) where the smoothed function is defined as f̃θ ? ( x ) = Eṽ∼Bn [ f ( x + δJq ( x ; θ ? ) ṽ ) ] . We show this result as Lemma 1 in Appendix A.1 . We use the resulting gradient estimate in SGD . The following proposition summarizes the sample complexity of this method . The constants and the proof are given in Appendix A.2 . Proposition 2 . Let f ( x ) be differentiable , L-Lipschitz , and µ-smooth . Consider running manifold random search ( Algorithm 2 ) for T steps . Let k = 1 for simplicity . Then 1 T T∑ t=1 E‖∇xf ( xt ) ‖22 ≤ O ( T− 1 2n+ T− 1 3n 2 3 ) .

Summary: In this paper, the authors first improve the gradient estimator in (Flaxman et al., 2004) zeroth-order optimization by exploiting low-rank structure. Then, the authors exploit machine learning to automatically discover the lower dimensional space in which the optimization is actually conducted. The authors justified the proposed algorithm both theoretically and empirically. The empirical performances of the proposed estimator outperforms the current derivative-free optimization algorithms on MuJoCo for policy optimization.