Gradientless Descent: High-Dimensional Zeroth-Order Optimization

1 INTRODUCTION . We consider the problem of zeroth-order optimization ( also known as gradient-free optimization , or bandit optimization ) , where our goal is to minimize an objective function f : Rn → R with as few evaluations of f ( x ) as possible . For many practical and interesting objective functions , gradients are difficult to compute and there is still a need for zeroth-order optimization in applications such as reinforcement learning ( Mania et al. , 2018 ; Salimans et al. , 2017 ; Choromanski et al. , 2018 ) , attacking neural networks ( Chen et al. , 2017 ; Papernot et al. , 2017 ) , hyperparameter tuning of deep networks ( Snoek et al. , 2012 ) , and network control ( Liu et al. , 2017 ) . The standard approach to zeroth-order optimization is , ironically , to estimate the gradients from function values and apply a first-order optimization algorithm ( Flaxman et al. , 2005 ) . Nesterov & Spokoiny ( 2011 ) analyze this class of algorithms as gradient descent on a Gaussian smoothing of the objective and gives an accelerated O ( n √ Q log ( ( LR2 + F ) / ) ) iteration complexity for an LLipschitz convex function with condition number Q and R = ‖x0 − x∗‖ and F = f ( x0 ) − f ( x∗ ) . They propose a two-point evaluation scheme that constructs gradient estimates from the difference between function values at two points that are close to each other . This scheme was extended by ( Duchi et al. , 2015 ) for stochastic settings , by ( Ghadimi & Lan , 2013 ) for nonconvex settings , and by ( Shamir , 2017 ) for non-smooth and non-Euclidean norm settings . Since then , first-order techniques such as variance reduction ( Liu et al. , 2018 ) , conditional gradients ( Balasubramanian & Ghadimi , 2018 ) , and diagonal preconditioning ( Mania et al. , 2018 ) have been successfully adopted in this setting . This class of algorithms are also known as stochastic search , random search , or ( natural ) evolutionary strategies and have been augmented with a variety of heuristics , such as the popular CMA-ES ( Auger & Hansen , 2005 ) . These algorithms , however , suffer from high variance due to non-robust local minima or highly non-smooth objectives , which are common in the fields of deep learning and reinforcement learn- ∗Author list in alphabetical order . ing . Mania et al . ( 2018 ) notes that gradient variance increases as training progresses due to higher variance in the objective functions , since often parameters must be tuned precisely to achieve reasonable models . Therefore , some attention has shifted into direct search algorithms that usually finds a descent direction u and moves to x + δu , where the step size is not scaled by the function difference . The first approaches for direct search were based on deterministic approaches with a positive spanning set and date back to the 1950s ( Brooks , 1958 ) . Only recently have theoretical bounds surfaced , with Gratton et al . ( 2015 ) giving an iteration complexity that is a large polynomial of n and Dodangeh & Vicente ( 2016 ) giving an improved O ( n2L2/ ) . Stochastic approaches tend to have better complexities : Stich et al . ( 2013 ) uses line search to give a O ( nQ log ( F/ ) ) iteration complexity for convex functions with condition number Q and most recently , Gorbunov et al . ( 2019 ) uses importance sampling to give a O ( nQ̄ log ( F/ ) ) complexity for convex functions with average condition number Q̄ , assuming access to sampling probabilities . Stich et al . ( 2013 ) notes that direct search algorithms are invariant under monotone transforms of the objective , a property that might explain their robustness in high-variance settings . In general , zeroth order optimization suffers an at least linear dependence on input dimension n and recent works have tried to address this limitation when n is large but f ( x ) admits a low-dimensional structure . Some papers assume that f ( x ) depends only on k coordinates and Wang et al . ( 2017 ) applies Lasso to find the important set of coordinates , whereas Balasubramanian & Ghadimi ( 2018 ) simply change the step size to achieve an O ( k ( log ( n ) / ) 2 ) iteration complexity . Other papers assume more generally that f ( x ) = g ( PAx ) only depends on a k-dimensional subspace given by the range of PA and Djolonga et al . ( 2013 ) apply low-rank approximation to find the low-dimensional subspace while Wang et al . ( 2013 ) use random embeddings . Hazan et al . ( 2017 ) assume that f ( x ) is a sparse collection of k-degree monomials on the Boolean hypercube and apply sparse recovery to achieve a O ( nk ) runtime bound . We will show that under the case that f ( x ) = g ( PAx ) , our algorithm will inherently pick up any low-dimensional structure in f ( x ) and achieve a convergence rate that depends on k log ( n ) . This initial convergence rate survives , even if we perturb f ( x ) = g ( PAx ) + h ( x ) , so long as h ( x ) is sufficiently small . We will not cover the whole variety of black-box optimization methods , such as Bayesian optimization or genetic algorithms . In general , these methods attempt to solve a broader problem ( e.g . multiple optima ) , have weaker theoretical guarantees and may require substantial computation at each step : e.g . Bayesian optimization generally has theoretical iteration complexities that grow exponentially in dimension , and CMA-ES lacks provable complexity bounds beyond convex quadratic functions . In addition to the slow runtime and weaker guarantees , Bayesian optimization assumes the success of an inner optimization loop of the acquisition function . This inner optimization is often implemented with many iterations of a simpler zeroth-order methods , justifying the need to understand gradient-less descent algorithms within its own context . 1.1 OUR CONTRIBUTIONS . In this paper , we present GradientLess Descent ( GLD ) , a class of truly gradient-free algorithms ( also known as direct search algorithms ) that are parameter free and provably fast . Our algorithms are based on a simple intuition : for well-conditioned functions , if we start from a point and take a small step in a randomly chosen direction , there is a significant probability that we will reduce the objective function value . We present a novel analysis that relies on facts in high dimensional geometry and can thus be viewed as a geometric analysis of gradient-free algorithms , recovering the standard convergence rates and step sizes . Specifically , we show that if the step size is on the order of O ( 1√ n ) , we can guarantee an expected decrease of 1 − Ω ( 1n ) in the optimality gap , based on geometric properties of the sublevel sets of a smooth and strongly convex function . Our results are invariant under monotone transformations of the objective function , thus our convergence results also hold for a large class of non-convex functions that are a subclass of quasi-convex functions . Specifically , note that monotone transformations of convex functions are not necessarily convex . However , a monotone transformation of a convex function is always quasi-convex . The maximization of quasi-concave utility functions , which is equivalent to the minimization of quasiconvex functions , is an important topic of study in economics ( e.g . Arrow & Enthoven ( 1961 ) ) . Intuition suggests that the step-size dependence on dimensionality can be improved when f ( x ) admits a low-dimensional structure . With a careful choice of sampling distribution we can show that if f ( x ) = g ( PAx ) , where PA is a rank k matrix , then our step size can be on the order ofO ( 1√k ) as our optimization behavior is preserved under projections . We call this property affine-invariance and show that the number of function evaluations needed for convergence depends logarithmically on n. Unlike most previous algorithms in the high-dimensional setting , no expensive sparse recovery or subspace finding methods are needed . Furthermore , by novel perturbation arguments , we show that our fast convergence rates are robust and holds even under the more realistic assumption when f ( x ) = g ( PAx ) + h ( x ) with h ( x ) being sufficiently small . Theorem 1 ( Convergence of GLD : Informal Restatement of Theorem 7 and Theorem 14 ) . Let f ( x ) be any monotone transform of a convex function with condition number Q and R = ‖x0 − x∗‖ . Let y be a sample from an appropriate distribution centered at x . Then , with constant probability , f ( y ) − f ( x∗ ) ≤ ( f ( x ) − f ( x∗ ) ) ( 1− 15nQ ) Therefore , we can find xT such that ‖xT−x∗‖ ≤ after T = Õ ( nQ log ( R/ ) ) function evaluations . Furthermore , for functions f ( x ) = g ( PAx ) + h ( x ) with rank k matrix PA and sufficiently small h ( x ) , we only require Õ ( kQ log ( n ) log ( R/ ) ) evaluations . Another advantage of our non-standard geometric analysis is that it allows us to deduce that our rates are optimal with a matching lower bound ( up to logarithmic factors ) , presenting theoretical evidence that gradient-free inherently requires Ω ( nQ ) function evaluations to converge . While gradient-estimation algorithms can achieve a better theoretical iteration complexity of O ( n √ Q ) , they lack the monotone and affine invariance properties . Empirically , we see that invariance properties are important to successful optimization , as validated by experiments on synthetic BBOB and MuJoCo benchmarks that show the competitiveness of GLD against standard optimization procedures . 2 PRELIMINARIES . We first define a few notations for the rest of the paper . Let X be a compact subset of Rn and let ‖ · ‖ denote the Euclidean norm . The diameter of X , denoted ‖X‖ = maxx , x′∈X ‖x − x′‖ , is the maximum distance between elements in X . Let f : X → R be a real-valued function which attains its minimum at x∗ . We use f ( X ) = { f ( x ) : x ∈ X } to denote the image of f on a subset X of Rn , and B ( c , r ) = { x ∈ Rn : ‖c− x‖ ≤ r } to denote the ball of radius r centered at c. Definition 2 . The level set of f at point x ∈ X is Lc ( f ) = { y ∈ X : f ( y ) = f ( x ) } . The sub-level set of f at point x ∈ X is L↓c ( f ) = { y ∈ X : f ( y ) ≤ f ( x ) } . When the function f is clear from the context , we omit it . Definition 3 . We say that f is α-strongly convex for α > 0 if f ( y ) ≥ f ( x ) + 〈∇f ( x ) , y − x〉 + α 2 ‖y−x‖ 2 for all x , y ∈ X and β-smooth for β > 0 if f ( y ) ≤ f ( x ) + 〈∇f ( x ) , y−x〉+ β2 ‖y−x‖ 2 for all x , y ∈ X . Definition 4 . We say that g◦f is a monotone transformation of f if g : f ( X ) → R is a monotonically ( and strictly ) increasing function . Monotone transformations preserve the level sets of a function in the sense that Lx ( f ) = Lx ( g ◦ f ) . Because our algorithms depend only on the level set properties , our results generalize to any monotone transformation of a strongly convex and strongly smooth function . This leads to our extended notion of condition number . Definition 5 . A function f has condition number Q ≥ 1 if it is the minimum ratio β/α over all functions g such that f is a monotone transformation of g and g is α-strongly convex and β smooth . When we work with low rank extensions of f , we only care about the condition number of f within a rank k subspace . Indeed , if f only varies along a rank k subspace , then it has a strong convexity value of 0 , making its condition number undefined . If f is α-strongly convex and β-smooth , then its Hessian matrix always has eigenvalues bounded between α and β . Therefore , we need a notion of a projected condition number . Let A ∈ Rd×k be some orthonormal matrix and let PA = AA > be the projection matrix onto the column space of A . Definition 6 . For some orthonormal A ∈ Rd×k with d > k , a function f has condition number restricted to A , Q ( A ) ≥ 1 , if it is the minimum ratio β/α over all functions g such that f is a monotone transformation of g and h ( y ) = g ( Ay ) is α-strongly convex and β smooth .

Summary: The paper proposes a novel zeroth-order algorithm for high-dimensional optimization. In particular, the algorithm as an instance of direct search algorithms where no attempt is made to estimate the gradient of the function during the optimization process. The authors study the optimization of monotone transformations of strongly-convex and smooth functions and they prove complexity bounds as a function of the condition number, the dimensionality and the desired accuracy. These results are also extended to the case where the function actually depends on a lower-dimensional input. Without any knowledge of the actual subspace of interest, the algorithm is able to adapt to the (lower) dimensionality of the problem. The proposed algorithms are tested on synthetic optimization problems and in a few Mujoco environments for policy optimization.