Oracle Complexity in Nonsmooth Nonconvex Optimization

1 Introduction . We consider optimization problems associated with functions f : Rd ! R , where f ( · ) is Lipschitz continuous and bounded from below , but otherwise satisfies no special structure , such as convexity . Clearly , in high dimensions , it is generally impossible to efficiently find a global minimum of a nonconvex function . However , if we relax our goal to finding ( approximate ) stationary points , then the nonconvexity is no longer an issue . In particular , it is known that if f ( · ) is smooth – namely , differentiable and with a Lipschitz continuous gradient – then for any ✏ > 0 , simple gradientbased algorithms can find x such that krf ( x ) k ✏ , using only O ( 1/✏2 ) gradient computations , independent of the dimension ( see for example [ 30 , 22 , 11 ] ) . Unfortunately , many optimization problems of interest are inherently not smooth . For example , when training modern neural networks , involving max operations and rectified linear units , the associated optimization problem is virtually always nonconvex as well as nonsmooth . Thus , the positive results above , which crucially rely on smoothness , are inapplicable . Although there are positive results even for nonconvex nonsmooth functions , they tend to be either purely asymptotic in nature ( e.g. , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . [ 8 , 23 , 36 , 16 , 26 ] ) , depend on the internal structure and representation of the objective function , or require additional assumptions which many problems of interest do not satisfy , such as weak convexity or some separation between nonconvex and nonsmooth components1 ( e.g. , [ 12 , 13 , 18 , 10 , 15 , 17 , 6 ] ) . This leads to the interesting question of developing black-box algorithms with non-asymptotic guarantees for nonsmooth nonconvex functions , without assuming any special structure . In this paper , we study this question via the well-known framework of oracle complexity [ 27 ] : Given a class of functions F , we associate with each f 2 F an oracle , which for any x in the domain of f ( · ) , returns local information about f ( · ) at x ( such as its value and gradient2 ) . We consider iterative algorithms which can be described via an interaction with such an oracle : At every iteration t , the algorithm chooses an iterate xt , and receives from the oracle local information about f ( · ) at xt , which is then used to choose the next iteration xt+1 . This framework captures essentially all iterative algorithms for black-box optimization . In this framework , we fix some iteration budget T , and ask what properties can be guaranteed for the iterates x1 , . . . , xT , as a function of T and uniformly over all functions in F ( for example , how close to optimal they are , whether they contain an approximately-stationary point , etc. ) . Unfortunately , as recently pointed out in [ 37 ] , neither small optimization error nor small gradients can be obtained for nonsmooth nonconvex functions with such local-information algorithms : Indeed , approximately-stationary points can be easily “ hidden ” inside some arbitrarily small neighborhood , which can not be found in a bounded number of iterations . Instead , we consider here two alternative approaches to tackle nonsmooth nonconvex optimization , and provide new oracle complexity results for each . We note that Zhang et al . [ 37 ] recently proposed another promising approach , by defining a certain relaxation of approximate stationarity ( so-called ( , ✏ ) -stationarity ) , and remarkably , prove that points satisfying this relaxed goal can be found via simple iterative algorithms with provable guarantees . However , there exist cases where their definition does not resemble a stationary point in any intuitive case , and thus it remains to be seen whether it is the most appropriate one . We further discuss the pros and cons of their approach in Appendix B . In our first contribution , we consider relaxing the goal of finding approximately-stationary points , to that of finding near-approximately-stationary points : Namely , getting -close to a point x with a ( generalized ) gradient of norm at most ✏ . This is arguably the most natural way to relax the goal of finding ✏-stationary points , while hopefully still getting meaningful algorithmic guarantees . Moreover , approaching stationary points is feasible in an asymptotic sense ( see for instance [ 17 ] ) . Unfortunately , we formally prove that this relaxation already sets the bar too high : For a very large class of gradient-based algorithms ( and in fact , all of them under a mild assumption ) , it is impossible to find near-approximately-stationary point with worst-case finite-time guarantees , for small enough constant , ✏ . In our second contribution , we consider tackling nonsmooth nonconvex optimization by reduction to smooth optimization : Given the target function f ( · ) , we first find a smooth function f̃ ( · ) ( with Lipschitz gradients ) which uniformly approximates it up to some arbitrarily small parameter ✏ , and then apply a smooth optimization method on f̃ ( · ) . Such reductions are common in convex optimization ( e.g. , [ 28 , 7 , 1 ] ) , and intuitively , should usually lead to points with meaningful properties with respect to the original function f ( · ) , at least when ✏ is small enough . For example , it is known that stationary points of f ( · ) are the limit of approximately-stationary points of appropriate smoothings of f ( · ) , as ✏ ! 0 [ 32 , Thm . 9.67 ] . This naturally leads to the question of how we can find a smooth approximation of a Lipschitz function f ( · ) . Inspecting existing approaches for smoothing nonconvex functions , we notice an interesting trade-off between computational efficiency and the smoothness of the approximating function : On the one hand , there exist optimization-based methods from the functional analysis literature ( in particular , Lasry-Lyons regularization [ 25 ] ) which yield essentially optimal gradient Lipschitz parameters , but are not computationally efficient . On the other hand , there exist simple , computationally tractable methods ( such as randomized smoothing [ 19 ] ) , which unfortunately lead to much worse gradient Lipschitz parameters , with strong scaling in the input dimension . This in turn leads to larger iteration complexity , when plugging into standard smooth optimization methods . Is this kind of trade-off between computational efficiency and smoothness necessary ? Considering 1A trivial example arising in deep learning , which does not satisfy most such structural assumptions , is the negative of the ReLU function , x 7 ! max { 0 , x } . 2For non-differentiable functions , we use a standard generalization of gradients following [ 14 ] , see Sec . 2 for details . this question from an oracle complexity viewpoint , we prove that this trade-off is indeed necessary under mild assumptions : If we want a smoothing method whose oracle complexity is polynomial in the problem parameters , we must accept that the gradient Lipschitz parameter may be no better than that obtained with randomized smoothing ( up to logarithmic factors ) . Thus , in a sense , randomized smoothing is an optimal nonconvex smoothing method among computationally efficient ones . Overall , we hope that our work motivates additional research into black-box algorithms for nonconvex , nonsmooth optimization problems , with rigorous finite-time guarantees . Our paper is structured as follows . In Sec . 2 , we formally introduce the notations and terminology that we use . In Sec . 3 , we present our results for getting near approximately stationary points . In Sec . 4 , we present our results for smoothing nonsmooth nonconvex functions . We conclude in Sec . 5 with a discussion of open questions . The supplementary material contains , besides the full proofs , a further discussion of the notion of ( , ✏ ) -stationarity from [ 37 ] ( Appendix B ) , and a proof that the dimension-dependencies arising from randomized smoothing provably affects the iteration complexity of vanilla gradient descent ( Appendix C ) . 2 Preliminaries . Notation . We let bold-face letters ( e.g. , x ) denote vectors . 0 is the zero vector in Rd ( where d is clear from context ) , and e1 , e2 , . . . are the standard basis vectors . Given a vector x , xi denotes its i-th coordinate , and x̄ denotes the normalized vector x/kxk ( assuming x 6= 0 ) . h· , ·i , k · k denote the standard Euclidean dot product and its induced norm over Rd , respectively . For any real number x , we denote [ x ] + : = max { x , 0 } . Given two functions f ( · ) , g ( · ) on the same domain X , we define kf gk1 = supx2X |f ( x ) g ( x ) | . We denote by Sd 1 : = x 2 Rd kxk = 1 the unit sphere . For any natural N , we abbreviate [ N ] : = { 1 , . . . , N } . We occasionally use standard big-O asymptotic notation , with O ( · ) , ⇥ ( · ) , ⌦ ( · ) hiding constants , Õ ( · ) , ⇥̃ ( · ) , ⌦̃ ( · ) hiding constants and logarithmic factors , and poly ( · ) meaning polynomial factors . Gradients and generalized gradients . If a function f : Rd ! R is differentiable at x , we denote its gradient by rf ( x ) . For possibly non-differentiable functions , we let @ f ( x ) denote the set of generalized gradients ( following [ 14 ] ) , which is perhaps the most standard extension of gradients to nonsmooth nonconvex functions . For Lipschitz functions ( which are almost everywhere differentiable by Rademacher ’ s theorem ) , one simple way to define it is @ f ( x ) : = conv { u : u = lim k ! 1 rf ( xk ) , xk ! x } ( namely , the convex hull of all limit points of rf ( xk ) , over all sequences x1 , x2 , . . . of differentiable points of f ( · ) which converge to x ) . With this definition , a stationary point with respect to f ( · ) is a point x satisfying 0 2 @ f ( x ) . Also , given some ✏ 0 , we say that x is an ✏-stationary point with respect to f ( · ) , if there is some u 2 @ f ( x ) such that kuk ✏ . Oracles . In this paper , we consider two types of oracles : • A first-order oracle O1f ( x ) , which given a function f ( · ) and a point x , returns f ( x ) and @ f ( x ) . We note that this differs somewhat from the standard formulation of first-order oracles in the literature , where it is assumed that at points of non-differentiability , the oracle only returns some element in the generalized gradient set @ f ( x ) . However , this means that our oracle complexity lower bounds hold under weaker assumptions , since we only provide more information to the algorithm . We use this type of oracle in the results of Sec . 3 , but discuss possible extensions in Remark 2 . • An all-derivatives oracle O1f ( x ) , which returns f ( x ) , @ f ( x ) , as well as all higher-order derivatives of f ( · ) at x , if they exist . This oracle provides more information than a first-order oracle , so any lower bound with respect to an all-derivatives oracle trivially applies with respect to a first-order oracle . Some of our bounds apply with respect to such a powerful oracle , and we present them in this generality , although we note that in the nonconvex-nonsmooth setting , it remains to be seen whether there is any use for higher-order derivative information . We use this type of oracle in the results of Sec . 4 .

Summary: This paper presented strong technical results for nonsmooth nonconvex problems. The main lower bound is counter-intuitive; the argument for random smoothing is deep. Meanwhile, the main body gives a very good illustration of the hard technical content.