Gradientless Descent: High-Dimensional Zeroth-Order Optimization

1 INTRODUCTION . We consider the problem of zeroth-order optimization ( also known as gradient-free optimization , or bandit optimization ) , where our goal is to minimize an objective function f : Rn → R with as few evaluations of f ( x ) as possible . For many practical and interesting objective functions , gradients are difficult to compute and there is still a need for zeroth-order optimization in applications such as reinforcement learning ( Mania et al. , 2018 ; Salimans et al. , 2017 ; Choromanski et al. , 2018 ) , attacking neural networks ( Chen et al. , 2017 ; Papernot et al. , 2017 ) , hyperparameter tuning of deep networks ( Snoek et al. , 2012 ) , and network control ( Liu et al. , 2017 ) . The standard approach to zeroth-order optimization is , ironically , to estimate the gradients from function values and apply a first-order optimization algorithm ( Flaxman et al. , 2005 ) . Nesterov & Spokoiny ( 2011 ) analyze this class of algorithms as gradient descent on a Gaussian smoothing of the objective and gives an accelerated O ( n √ Q log ( ( LR2 + F ) / ) ) iteration complexity for an LLipschitz convex function with condition number Q and R = ‖x0 − x∗‖ and F = f ( x0 ) − f ( x∗ ) . They propose a two-point evaluation scheme that constructs gradient estimates from the difference between function values at two points that are close to each other . This scheme was extended by ( Duchi et al. , 2015 ) for stochastic settings , by ( Ghadimi & Lan , 2013 ) for nonconvex settings , and by ( Shamir , 2017 ) for non-smooth and non-Euclidean norm settings . Since then , first-order techniques such as variance reduction ( Liu et al. , 2018 ) , conditional gradients ( Balasubramanian & Ghadimi , 2018 ) , and diagonal preconditioning ( Mania et al. , 2018 ) have been successfully adopted in this setting . This class of algorithms are also known as stochastic search , random search , or ( natural ) evolutionary strategies and have been augmented with a variety of heuristics , such as the popular CMA-ES ( Auger & Hansen , 2005 ) . These algorithms , however , suffer from high variance due to non-robust local minima or highly non-smooth objectives , which are common in the fields of deep learning and reinforcement learn- ∗Author list in alphabetical order . ing . Mania et al . ( 2018 ) notes that gradient variance increases as training progresses due to higher variance in the objective functions , since often parameters must be tuned precisely to achieve reasonable models . Therefore , some attention has shifted into direct search algorithms that usually finds a descent direction u and moves to x + δu , where the step size is not scaled by the function difference . The first approaches for direct search were based on deterministic approaches with a positive spanning set and date back to the 1950s ( Brooks , 1958 ) . Only recently have theoretical bounds surfaced , with Gratton et al . ( 2015 ) giving an iteration complexity that is a large polynomial of n and Dodangeh & Vicente ( 2016 ) giving an improved O ( n2L2/ ) . Stochastic approaches tend to have better complexities : Stich et al . ( 2013 ) uses line search to give a O ( nQ log ( F/ ) ) iteration complexity for convex functions with condition number Q and most recently , Gorbunov et al . ( 2019 ) uses importance sampling to give a O ( nQ̄ log ( F/ ) ) complexity for convex functions with average condition number Q̄ , assuming access to sampling probabilities . Stich et al . ( 2013 ) notes that direct search algorithms are invariant under monotone transforms of the objective , a property that might explain their robustness in high-variance settings . In general , zeroth order optimization suffers an at least linear dependence on input dimension n and recent works have tried to address this limitation when n is large but f ( x ) admits a low-dimensional structure . Some papers assume that f ( x ) depends only on k coordinates and Wang et al . ( 2017 ) applies Lasso to find the important set of coordinates , whereas Balasubramanian & Ghadimi ( 2018 ) simply change the step size to achieve an O ( k ( log ( n ) / ) 2 ) iteration complexity . Other papers assume more generally that f ( x ) = g ( PAx ) only depends on a k-dimensional subspace given by the range of PA and Djolonga et al . ( 2013 ) apply low-rank approximation to find the low-dimensional subspace while Wang et al . ( 2013 ) use random embeddings . Hazan et al . ( 2017 ) assume that f ( x ) is a sparse collection of k-degree monomials on the Boolean hypercube and apply sparse recovery to achieve a O ( nk ) runtime bound . We will show that under the case that f ( x ) = g ( PAx ) , our algorithm will inherently pick up any low-dimensional structure in f ( x ) and achieve a convergence rate that depends on k log ( n ) . This initial convergence rate survives , even if we perturb f ( x ) = g ( PAx ) + h ( x ) , so long as h ( x ) is sufficiently small . We will not cover the whole variety of black-box optimization methods , such as Bayesian optimization or genetic algorithms . In general , these methods attempt to solve a broader problem ( e.g . multiple optima ) , have weaker theoretical guarantees and may require substantial computation at each step : e.g . Bayesian optimization generally has theoretical iteration complexities that grow exponentially in dimension , and CMA-ES lacks provable complexity bounds beyond convex quadratic functions . In addition to the slow runtime and weaker guarantees , Bayesian optimization assumes the success of an inner optimization loop of the acquisition function . This inner optimization is often implemented with many iterations of a simpler zeroth-order methods , justifying the need to understand gradient-less descent algorithms within its own context . 1.1 OUR CONTRIBUTIONS . In this paper , we present GradientLess Descent ( GLD ) , a class of truly gradient-free algorithms ( also known as direct search algorithms ) that are parameter free and provably fast . Our algorithms are based on a simple intuition : for well-conditioned functions , if we start from a point and take a small step in a randomly chosen direction , there is a significant probability that we will reduce the objective function value . We present a novel analysis that relies on facts in high dimensional geometry and can thus be viewed as a geometric analysis of gradient-free algorithms , recovering the standard convergence rates and step sizes . Specifically , we show that if the step size is on the order of O ( 1√ n ) , we can guarantee an expected decrease of 1 − Ω ( 1n ) in the optimality gap , based on geometric properties of the sublevel sets of a smooth and strongly convex function . Our results are invariant under monotone transformations of the objective function , thus our convergence results also hold for a large class of non-convex functions that are a subclass of quasi-convex functions . Specifically , note that monotone transformations of convex functions are not necessarily convex . However , a monotone transformation of a convex function is always quasi-convex . The maximization of quasi-concave utility functions , which is equivalent to the minimization of quasiconvex functions , is an important topic of study in economics ( e.g . Arrow & Enthoven ( 1961 ) ) . Intuition suggests that the step-size dependence on dimensionality can be improved when f ( x ) admits a low-dimensional structure . With a careful choice of sampling distribution we can show that if f ( x ) = g ( PAx ) , where PA is a rank k matrix , then our step size can be on the order ofO ( 1√k ) as our optimization behavior is preserved under projections . We call this property affine-invariance and show that the number of function evaluations needed for convergence depends logarithmically on n. Unlike most previous algorithms in the high-dimensional setting , no expensive sparse recovery or subspace finding methods are needed . Furthermore , by novel perturbation arguments , we show that our fast convergence rates are robust and holds even under the more realistic assumption when f ( x ) = g ( PAx ) + h ( x ) with h ( x ) being sufficiently small . Theorem 1 ( Convergence of GLD : Informal Restatement of Theorem 7 and Theorem 14 ) . Let f ( x ) be any monotone transform of a convex function with condition number Q and R = ‖x0 − x∗‖ . Let y be a sample from an appropriate distribution centered at x . Then , with constant probability , f ( y ) − f ( x∗ ) ≤ ( f ( x ) − f ( x∗ ) ) ( 1− 15nQ ) Therefore , we can find xT such that ‖xT−x∗‖ ≤ after T = Õ ( nQ log ( R/ ) ) function evaluations . Furthermore , for functions f ( x ) = g ( PAx ) + h ( x ) with rank k matrix PA and sufficiently small h ( x ) , we only require Õ ( kQ log ( n ) log ( R/ ) ) evaluations . Another advantage of our non-standard geometric analysis is that it allows us to deduce that our rates are optimal with a matching lower bound ( up to logarithmic factors ) , presenting theoretical evidence that gradient-free inherently requires Ω ( nQ ) function evaluations to converge . While gradient-estimation algorithms can achieve a better theoretical iteration complexity of O ( n √ Q ) , they lack the monotone and affine invariance properties . Empirically , we see that invariance properties are important to successful optimization , as validated by experiments on synthetic BBOB and MuJoCo benchmarks that show the competitiveness of GLD against standard optimization procedures . 2 PRELIMINARIES . We first define a few notations for the rest of the paper . Let X be a compact subset of Rn and let ‖ · ‖ denote the Euclidean norm . The diameter of X , denoted ‖X‖ = maxx , x′∈X ‖x − x′‖ , is the maximum distance between elements in X . Let f : X → R be a real-valued function which attains its minimum at x∗ . We use f ( X ) = { f ( x ) : x ∈ X } to denote the image of f on a subset X of Rn , and B ( c , r ) = { x ∈ Rn : ‖c− x‖ ≤ r } to denote the ball of radius r centered at c. Definition 2 . The level set of f at point x ∈ X is Lc ( f ) = { y ∈ X : f ( y ) = f ( x ) } . The sub-level set of f at point x ∈ X is L↓c ( f ) = { y ∈ X : f ( y ) ≤ f ( x ) } . When the function f is clear from the context , we omit it . Definition 3 . We say that f is α-strongly convex for α > 0 if f ( y ) ≥ f ( x ) + 〈∇f ( x ) , y − x〉 + α 2 ‖y−x‖ 2 for all x , y ∈ X and β-smooth for β > 0 if f ( y ) ≤ f ( x ) + 〈∇f ( x ) , y−x〉+ β2 ‖y−x‖ 2 for all x , y ∈ X . Definition 4 . We say that g◦f is a monotone transformation of f if g : f ( X ) → R is a monotonically ( and strictly ) increasing function . Monotone transformations preserve the level sets of a function in the sense that Lx ( f ) = Lx ( g ◦ f ) . Because our algorithms depend only on the level set properties , our results generalize to any monotone transformation of a strongly convex and strongly smooth function . This leads to our extended notion of condition number . Definition 5 . A function f has condition number Q ≥ 1 if it is the minimum ratio β/α over all functions g such that f is a monotone transformation of g and g is α-strongly convex and β smooth . When we work with low rank extensions of f , we only care about the condition number of f within a rank k subspace . Indeed , if f only varies along a rank k subspace , then it has a strong convexity value of 0 , making its condition number undefined . If f is α-strongly convex and β-smooth , then its Hessian matrix always has eigenvalues bounded between α and β . Therefore , we need a notion of a projected condition number . Let A ∈ Rd×k be some orthonormal matrix and let PA = AA > be the projection matrix onto the column space of A . Definition 6 . For some orthonormal A ∈ Rd×k with d > k , a function f has condition number restricted to A , Q ( A ) ≥ 1 , if it is the minimum ratio β/α over all functions g such that f is a monotone transformation of g and h ( y ) = g ( Ay ) is α-strongly convex and β smooth .

Summary: This paper proposes stable GradientLess Descent (GLD) algorithms that do not rely on gradient estimate. Based on the low-rank assumption on P_A, the iteration complexity is poly-logarithmically dependent on dimensionality. The theoretical analysis of the main results is based on a geometric perspective, which is interesting. The experimental results on synthetic and MuJoCo datasets validate the effectiveness of the proposed algorithms.