Oracle Complexity in Nonsmooth Nonconvex Optimization

1 Introduction . We consider optimization problems associated with functions f : Rd ! R , where f ( · ) is Lipschitz continuous and bounded from below , but otherwise satisfies no special structure , such as convexity . Clearly , in high dimensions , it is generally impossible to efficiently find a global minimum of a nonconvex function . However , if we relax our goal to finding ( approximate ) stationary points , then the nonconvexity is no longer an issue . In particular , it is known that if f ( · ) is smooth – namely , differentiable and with a Lipschitz continuous gradient – then for any ✏ > 0 , simple gradientbased algorithms can find x such that krf ( x ) k ✏ , using only O ( 1/✏2 ) gradient computations , independent of the dimension ( see for example [ 30 , 22 , 11 ] ) . Unfortunately , many optimization problems of interest are inherently not smooth . For example , when training modern neural networks , involving max operations and rectified linear units , the associated optimization problem is virtually always nonconvex as well as nonsmooth . Thus , the positive results above , which crucially rely on smoothness , are inapplicable . Although there are positive results even for nonconvex nonsmooth functions , they tend to be either purely asymptotic in nature ( e.g. , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . [ 8 , 23 , 36 , 16 , 26 ] ) , depend on the internal structure and representation of the objective function , or require additional assumptions which many problems of interest do not satisfy , such as weak convexity or some separation between nonconvex and nonsmooth components1 ( e.g. , [ 12 , 13 , 18 , 10 , 15 , 17 , 6 ] ) . This leads to the interesting question of developing black-box algorithms with non-asymptotic guarantees for nonsmooth nonconvex functions , without assuming any special structure . In this paper , we study this question via the well-known framework of oracle complexity [ 27 ] : Given a class of functions F , we associate with each f 2 F an oracle , which for any x in the domain of f ( · ) , returns local information about f ( · ) at x ( such as its value and gradient2 ) . We consider iterative algorithms which can be described via an interaction with such an oracle : At every iteration t , the algorithm chooses an iterate xt , and receives from the oracle local information about f ( · ) at xt , which is then used to choose the next iteration xt+1 . This framework captures essentially all iterative algorithms for black-box optimization . In this framework , we fix some iteration budget T , and ask what properties can be guaranteed for the iterates x1 , . . . , xT , as a function of T and uniformly over all functions in F ( for example , how close to optimal they are , whether they contain an approximately-stationary point , etc. ) . Unfortunately , as recently pointed out in [ 37 ] , neither small optimization error nor small gradients can be obtained for nonsmooth nonconvex functions with such local-information algorithms : Indeed , approximately-stationary points can be easily “ hidden ” inside some arbitrarily small neighborhood , which can not be found in a bounded number of iterations . Instead , we consider here two alternative approaches to tackle nonsmooth nonconvex optimization , and provide new oracle complexity results for each . We note that Zhang et al . [ 37 ] recently proposed another promising approach , by defining a certain relaxation of approximate stationarity ( so-called ( , ✏ ) -stationarity ) , and remarkably , prove that points satisfying this relaxed goal can be found via simple iterative algorithms with provable guarantees . However , there exist cases where their definition does not resemble a stationary point in any intuitive case , and thus it remains to be seen whether it is the most appropriate one . We further discuss the pros and cons of their approach in Appendix B . In our first contribution , we consider relaxing the goal of finding approximately-stationary points , to that of finding near-approximately-stationary points : Namely , getting -close to a point x with a ( generalized ) gradient of norm at most ✏ . This is arguably the most natural way to relax the goal of finding ✏-stationary points , while hopefully still getting meaningful algorithmic guarantees . Moreover , approaching stationary points is feasible in an asymptotic sense ( see for instance [ 17 ] ) . Unfortunately , we formally prove that this relaxation already sets the bar too high : For a very large class of gradient-based algorithms ( and in fact , all of them under a mild assumption ) , it is impossible to find near-approximately-stationary point with worst-case finite-time guarantees , for small enough constant , ✏ . In our second contribution , we consider tackling nonsmooth nonconvex optimization by reduction to smooth optimization : Given the target function f ( · ) , we first find a smooth function f̃ ( · ) ( with Lipschitz gradients ) which uniformly approximates it up to some arbitrarily small parameter ✏ , and then apply a smooth optimization method on f̃ ( · ) . Such reductions are common in convex optimization ( e.g. , [ 28 , 7 , 1 ] ) , and intuitively , should usually lead to points with meaningful properties with respect to the original function f ( · ) , at least when ✏ is small enough . For example , it is known that stationary points of f ( · ) are the limit of approximately-stationary points of appropriate smoothings of f ( · ) , as ✏ ! 0 [ 32 , Thm . 9.67 ] . This naturally leads to the question of how we can find a smooth approximation of a Lipschitz function f ( · ) . Inspecting existing approaches for smoothing nonconvex functions , we notice an interesting trade-off between computational efficiency and the smoothness of the approximating function : On the one hand , there exist optimization-based methods from the functional analysis literature ( in particular , Lasry-Lyons regularization [ 25 ] ) which yield essentially optimal gradient Lipschitz parameters , but are not computationally efficient . On the other hand , there exist simple , computationally tractable methods ( such as randomized smoothing [ 19 ] ) , which unfortunately lead to much worse gradient Lipschitz parameters , with strong scaling in the input dimension . This in turn leads to larger iteration complexity , when plugging into standard smooth optimization methods . Is this kind of trade-off between computational efficiency and smoothness necessary ? Considering 1A trivial example arising in deep learning , which does not satisfy most such structural assumptions , is the negative of the ReLU function , x 7 ! max { 0 , x } . 2For non-differentiable functions , we use a standard generalization of gradients following [ 14 ] , see Sec . 2 for details . this question from an oracle complexity viewpoint , we prove that this trade-off is indeed necessary under mild assumptions : If we want a smoothing method whose oracle complexity is polynomial in the problem parameters , we must accept that the gradient Lipschitz parameter may be no better than that obtained with randomized smoothing ( up to logarithmic factors ) . Thus , in a sense , randomized smoothing is an optimal nonconvex smoothing method among computationally efficient ones . Overall , we hope that our work motivates additional research into black-box algorithms for nonconvex , nonsmooth optimization problems , with rigorous finite-time guarantees . Our paper is structured as follows . In Sec . 2 , we formally introduce the notations and terminology that we use . In Sec . 3 , we present our results for getting near approximately stationary points . In Sec . 4 , we present our results for smoothing nonsmooth nonconvex functions . We conclude in Sec . 5 with a discussion of open questions . The supplementary material contains , besides the full proofs , a further discussion of the notion of ( , ✏ ) -stationarity from [ 37 ] ( Appendix B ) , and a proof that the dimension-dependencies arising from randomized smoothing provably affects the iteration complexity of vanilla gradient descent ( Appendix C ) . 2 Preliminaries . Notation . We let bold-face letters ( e.g. , x ) denote vectors . 0 is the zero vector in Rd ( where d is clear from context ) , and e1 , e2 , . . . are the standard basis vectors . Given a vector x , xi denotes its i-th coordinate , and x̄ denotes the normalized vector x/kxk ( assuming x 6= 0 ) . h· , ·i , k · k denote the standard Euclidean dot product and its induced norm over Rd , respectively . For any real number x , we denote [ x ] + : = max { x , 0 } . Given two functions f ( · ) , g ( · ) on the same domain X , we define kf gk1 = supx2X |f ( x ) g ( x ) | . We denote by Sd 1 : = x 2 Rd kxk = 1 the unit sphere . For any natural N , we abbreviate [ N ] : = { 1 , . . . , N } . We occasionally use standard big-O asymptotic notation , with O ( · ) , ⇥ ( · ) , ⌦ ( · ) hiding constants , Õ ( · ) , ⇥̃ ( · ) , ⌦̃ ( · ) hiding constants and logarithmic factors , and poly ( · ) meaning polynomial factors . Gradients and generalized gradients . If a function f : Rd ! R is differentiable at x , we denote its gradient by rf ( x ) . For possibly non-differentiable functions , we let @ f ( x ) denote the set of generalized gradients ( following [ 14 ] ) , which is perhaps the most standard extension of gradients to nonsmooth nonconvex functions . For Lipschitz functions ( which are almost everywhere differentiable by Rademacher ’ s theorem ) , one simple way to define it is @ f ( x ) : = conv { u : u = lim k ! 1 rf ( xk ) , xk ! x } ( namely , the convex hull of all limit points of rf ( xk ) , over all sequences x1 , x2 , . . . of differentiable points of f ( · ) which converge to x ) . With this definition , a stationary point with respect to f ( · ) is a point x satisfying 0 2 @ f ( x ) . Also , given some ✏ 0 , we say that x is an ✏-stationary point with respect to f ( · ) , if there is some u 2 @ f ( x ) such that kuk ✏ . Oracles . In this paper , we consider two types of oracles : • A first-order oracle O1f ( x ) , which given a function f ( · ) and a point x , returns f ( x ) and @ f ( x ) . We note that this differs somewhat from the standard formulation of first-order oracles in the literature , where it is assumed that at points of non-differentiability , the oracle only returns some element in the generalized gradient set @ f ( x ) . However , this means that our oracle complexity lower bounds hold under weaker assumptions , since we only provide more information to the algorithm . We use this type of oracle in the results of Sec . 3 , but discuss possible extensions in Remark 2 . • An all-derivatives oracle O1f ( x ) , which returns f ( x ) , @ f ( x ) , as well as all higher-order derivatives of f ( · ) at x , if they exist . This oracle provides more information than a first-order oracle , so any lower bound with respect to an all-derivatives oracle trivially applies with respect to a first-order oracle . Some of our bounds apply with respect to such a powerful oracle , and we present them in this generality , although we note that in the nonconvex-nonsmooth setting , it remains to be seen whether there is any use for higher-order derivative information . We use this type of oracle in the results of Sec . 4 .

Summary: This paper investigates the question of finding near $\epsilon$-stationary points for  Lipschitz nonconvex functions. This is a quite challenging question, and the paper addresses it with the following advances:  1. An oracle lower bound for the task of finding near $\epsilon$-stationary points. This is achieved by an intrincate construction that resembles the trick of hiding a cone in a random direction. This result is restricted to deterministic and zero-respecting algorithms.  2. The second contribution is the study of smoothings from an oracle perspective. It is shown in particular that in this oracle model, the Lipschitz gradient bounds obtained by randomized smoothing are essentially optimal, which rules out e.g. efficient implementations of the Lasry-Lions regularization. BTW, I wonder if Corollary 3 readily implies that computing saddle points has exponential oracle complexity with respect to the dimension (these type of results are already known, but this would be a more transparent proof, I believe).  3. A less emphasized third contribution is the observation that the relaxation used in Zhang et al. 2020 (based on the convex hull of a union of subdifferentials around the point) does not lead to anything resembling a nearly stationary point.