Near-Optimal Lower Bounds For Convex Optimization For All Orders of Smoothness

We study the complexity of optimizing highly smooth convex functions . For a positive integer p , we want to find an -approximate minimum of a convex function f , given oracle access to the function and its first p derivatives , assuming that the pth derivative of f is Lipschitz . Recently , three independent research groups ( Jiang et al. , PLMR 2019 ; Gasnikov et al. , PLMR 2019 ; Bubeck et al. , PLMR 2019 ) developed a new algorithm that solves this problem with Õ ( 1/ 2 3p+1 ) oracle calls for constant p. This is known to be optimal ( up to log factors ) for deterministic algorithms , but known lower bounds for randomized algorithms do not match this bound . We prove a new lower bound that matches this bound ( up to log factors ) , and holds not only for randomized algorithms , but also quantum algorithms . 1 Introduction . In recent years , several optimization algorithms have been proposed , especially for machine learning problems , that achieve improved performance by exploiting the smoothness of the function to be optimized . Specifically , these algorithms have better performance when the function ’ s first , second , or higher order derivatives are Lipschitz [ Nes08 , Bae09 , MS13 , Nes19 , GDG+19 , JWZ19 , BJL+19b ] . In this paper we study the problem of minimizing a highly smooth convex function given black-box access to the function and its higher order derivatives . The simplest example of the family of problems we consider here is the problem of approximately minimizing a convex function f : Rn → R , given access to an oracle that on input x ∈ Rn outputs ( f ( x ) , ∇f ( x ) ) , under the assumption that the function ’ s first derivative , its gradient ∇f , has bounded Lipschitz constant . This problem can be solved by Nesterov ’ s accelerated gradient descent , and it is known that this algorithm is optimal ( in high dimension ) among deterministic and randomized algorithms [ Nes83 , NY83 ] . More generally , for any positive integer p , consider the pth-order optimization problem : For known R > 0 , > 0 , and Lp > 0 , we have a p times differentiable convex function f : Rn → R whose pth derivative has Lipschitz constant at most Lp , which means ‖∇pf ( x ) −∇pf ( y ) ‖ ≤ Lp‖x− y‖ , ( 1 ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . where ‖·‖ is the ` 2 norm ( for vectors ) or induced ` 2 norm ( for operators ) . Our goal is to find an -approximate minimum of this function in a ball of radius R , which is any x∗ that satisfies f ( x∗ ) − min x∈BR ( 0 ) f ( x ) ≤ , ( 2 ) where BR ( 0 ) is the ` 2-ball of radius R around the origin . We can access the function f through a pth order oracle , which when queried with a point x ∈ Rn outputs ( f ( x ) , ∇f ( x ) , . . . , ∇pf ( x ) ) . ( 3 ) As usual , ∇pf ( x ) denotes the pth derivative of f ( x ) . Our primary object of study will be the minimum query cost of an algorithm that solves the problem , i.e . the number of queries ( or calls ) to the oracle in eq . ( 3 ) that an algorithm has to make.1 For a fixed p , it seems like this problem has 4 independent parameters , n , Lp , R , and , but the parameters are not all independent since we can scale the input and output spaces of the function to affect the latter 3 parameters . Thus the complexity of any algorithm can be written as a function of n and LpRp+1/ . In this paper we focus on the high-dimensional setting where n may be much larger than the other parameters , and the best algorithms in this regime have complexity that only depends on LpRp+1/ with no dependence on n. As noted , the p = 1 problem has been studied since the early 80s [ Nes83 , NY83 ] , and the p > 1 problem has also been considered [ Nes08 , MS13 ] . In an exciting recent development , new algorithms were proposed for all p ( with very similar complexity ) by three independent groups of researchers : Gasnikov , Dvurechensky , Gorbunov , Vorontsova , Selikhanovych , and Uribe [ GDG+19 ] ; Jiang , Wang , and Zhang [ JWZ19 ] ; Bubeck , Jiang , Lee , Li and Sidford [ BJL+19b ] . All three groups develop deterministic algorithms that make Õp ( ( LpR p+1/ ) 2/ ( 3p+1 ) ) ( 4 ) oracle calls,2 where the subscript p in the big Oh ( or big Omega ) notation means the constant in the big Oh can depend on p. In other words , this notation means that we treat p as a constant . This improved upon the bounds of [ Bae09 , Nes19 ] ; both the works develop deterministic algorithms that make Õp ( ( LpR p+1/ ) 1/ ( p+1 ) ) ( 5 ) oracle calls . This algorithm is nearly optimal among deterministic algorithms , since the works [ Nes19 , ASS19 ] showed that any deterministic algorithm that solves this problem must make Ωp ( ( LpR p+1/ ) 2/ ( 3p+1 ) ) queries . However , for randomized algorithms , the known lower bound is weaker . Agarwal and Hazan [ AH18 ] showed that any randomized algorithm must make Ωp ( ( LpR p+1/ ) 2/ ( 5p+1 ) ) ( 6 ) queries . To the best of our knowledge , no lower bounds are known in the setting of high-dimensional smooth convex optimization against quantum algorithms , although quantum lower bounds are known in the low-dimensional setting [ CCLW20 , vAGGdW20 ] and for non-smooth convex optimization [ GKNS21 ] . In this work , we close the gap ( up to log factors ) between the known algorithm and randomized lower bound for all p. Furthermore , our lower bound also holds against quantum algorithms . Theorem 1 . Fix any p ∈ N. For all > 0 , R > 0 , Lp > 0 , there exists an n > 0 and a set of n-dimensional functions F with pth-order Lipschitz constant Lp ( i.e. , satisfying eq . ( 1 ) ) such that any randomized or quantum algorithm that outputs an -approximate minimum ( satisfying eq . ( 2 ) ) for any function f ∈ F must make Ωp ( ( LpR p+1/ ) 2/ ( 3p+1 ) ( logLpR p+1/ ) −2/3 ) ( 7 ) queries to a pth order oracle for f ( as in eq . ( 3 ) ) . 1For simplicity we assume that the oracle ’ s output is computed to arbitrarily many bits of precision . This only makes our results stronger , since we prove lower bounds in this paper . 2Note that the query complexity does not have any dependence on the dimension n. Of course , actually implementing each query will take poly ( n ) time , but we only count the number of queries here . In fact , this lower bound holds even against highly parallel randomized algorithms , where the algorithm can make poly ( n , LpRp+1/ ) queries in each round and we only count the total number of query rounds ( and not the total number of queries ) . See [ BJL+19a ] for previous work in this setting , including speedups for first-order convex optimization in the low dimensional setting . In this introduction , we have deliberately avoided explaining the quantum model of computation to make the results accessible to readers without a background in quantum computing . The entire paper is written so that the randomized lower bound is fully accessible to any reader who does not wish to understand the quantum model and quantum lower bound . For readers familiar with quantum computing , we note that the only thing to be changed to get the quantum model is to modify the oracle in eq . ( 3 ) to support queries in quantum superposition . This is done in the usual way , by defining a unitary implementation of the oracle , which allows quantum algorithms to make superposition queries and potentially solve the problem more efficiently than randomized algorithms . 2 High level overview . Let us first consider the lower bound against randomized algorithms . Let us also first look at the special setting of p = 0 where we still assume access to the gradient ( or p = 1 ) oracle . To be more precise , the oracle returns subgradients , since gradients need not be defined at all points for Lipschitz convex functions . For this setting , known popularly as nonsmooth convex optimization , the optimal lower bound of Ω ( 1 2 ) is in fact a classical result [ NY83 ] . The proof of this result is very elegant and has been used subsequently to prove several other related lower bounds such as for parallel randomized algorithms [ Nem94 ] , quantum algorithms [ GKNS21 ] etc . Since our proof builds on this framework , we now review this . Nonsmooth lower bound instance . The lower bound instance for nonsmooth convex optimization is min‖x‖≤1 fV ( x ) where fV : Rn → R is chosen as fV ( x ) = max i∈ [ k ] 〈vi , x〉+ ( k − i ) γ , ( 8 ) with k = O ( n1/3 ) , γ = Θ̃ ( 1√ n ) , and V = ( v1 , · · · , vk ) comprises k orthonormal vectors chosen uniformly at random . The argument essentially shows that ( i ) in order to find a O ( 1√ k ) -approximate minimizer , one needs to know all the vi ’ s , and ( ii ) with high probability , each query reveals at most one new vector vi . This yields a lower bound of k queries for achieving an error of O ( 1√ k ) . Since f is a 1-Lipschitz function , when we rewrite this bound in terms of error , it yields the Ω ( 1 2 ) randomized lower bound . For the p = 1 setting , known popularly as smooth convex optimization , the optimal lower bound Ω ( √ 1 ) is also a classical result originally proven in [ NY83 ] . However , the proof of this result in [ NY83 ] is quite complicated and is not widely known . The recent papers of [ GN15 , DG20 ] provide a much simpler proof of the p = 1 result by using the lower bound construction for p = 0 setting described above and using smoothing , which we now review . Smoothing . Smoothing refers to the process of approximating a given Lipschitz function f by another function g , which has Lipschitz continuous pth derivatives for some p ≥ 1 . Further , since we will be applying this operation to ( 8 ) , we will describe smoothing in this context . Definition 1 . An operator S which takes a 1-Lipschitz function f to another convex function S [ f ] is called a ( p , β , ) -smoothing operation if it satisfies the following : 1 . Smoothness : pth derivatives of S [ f ] are Lipschitz continuous with parameter β , and 2 . Approximation : For any x , we have |f ( x ) − S [ f ] ( x ) | ≤ . If we can design a smoothing operation as per the above definition with = O ( 1/ √ k ) and further ensure that property ( ii ) above i.e. , with high probability , each query to the first p derivatives of S [ f ] reveals at most one new vector vi , then the proof strategy of lower bound for nonsmooth convex optimization can be executed on the smoothed instance S [ f ] , there by giving us a lower bound for pth-order smooth convex optimization . This is the key idea of [ GN15 , DG20 ] . Further , the smaller β is , the better the bound we obtain . However , since f can have discontinuous pth derivatives , there is a tension between the approximation property which tries to keep S [ f ] close to f and the smoothness property . So , one can not make β very small after fixing = O ( 1/ √ k ) . For the rest of this section , we fix = O ( 1/ √ k ) in Definition 1 . For the p = 1 setting , there is a well-known smoothing operation known as Moreau/inf-conv smoothing [ BC11 ] , which obtains the best possible smoothing with β = Θ ( k1.5 ) . This gives the tight query lower bound of Ω ( 1√ ) for smooth convex optimization . However , there is no known generalization of inf-conv smoothing for p ≥ 2 , so one needs to use a different smoothing operator to extend this proof strategy for proving query lower bounds for higher order smooth convex optimization . Given any p ≥ 1 , [ AH18 ] indeed construct such a smoothing , called randomized smoothing which maps Lipschitz convex functions to convex functions with Lipschitz pth derivatives . In the general p ≥ 1 setting , a smoothing operator with β = O ( k3p/2 ) would give the optimal lower bound of Ω ( −2 3p+1 ) . However , the randomized smoothing of [ AH18 ] can obtain only β = O ( k5p/2 ) leading to a suboptimal Ω ( −2 5p+1 ) lower bound for pth order smooth convex optimization . We design an improved smoothing operation , for the specific class of functions in Equation ( 8 ) , with the optimal β = O ( k3p/2 ) using two key ideas . The first idea is the softmax function with parameter ρ defined as smaxρ ( z ) : = ρ log ( ∑ i∈ [ k ] exp ( zi ρ ) ) , where z ∈ Rk . If we apply smaxρ , with ρ = k−3/2 , to functions of the form ( 8 ) through : smaxρ ( vecV ( x ) ) : = ρ log ∑ i∈ [ k ] exp ( 〈vi , x〉+ ( k − i ) γ ρ ) , ( 9 ) where vecV ( x ) : = ( 〈v1 , x〉 + ( k − 1 ) γ , 〈v2 , x〉 + ( k − 2 ) γ , . . . , 〈vk , x〉 ) , we can show that smaxρ ( vecV ( x ) ) satisfies Definition 1 with the optimal value of β = O ( k3p/2 ) . However , any query on derivatives of smaxρ ( vecV ( x ) ) reveals information about all the vectors vi simultaneously since for instance the gradient is given by ∇smaxρ ( vecV ( x ) ) = ∑ i∈ [ k ] exp ( 〈vi , x〉+ ( k−i ) γ ρ ) ∑ j∈ [ k ] exp ( 〈vj , x〉+ ( k−j ) γ ρ ) · vi . ( 10 ) Consequently , it can not be directly used to obtain a lower bound . The second idea is that even though the function value and derivatives of smaxρ ( vecV ( x ) ) have contribution from all vi ’ s , the contribution is heavily dominated ( i.e. , up to 1poly ( k ) error ) by vi∗ ( x ) , where i ∗ ( x ) = argmaxi∈ [ k ] 〈vi , x〉+ ( k−i ) γ , whenever 〈 vi∗ ( x ) , x 〉 + ( k − i∗ ( x ) ) γ > 〈vi , x〉+ ( k − i ) γ + Ω ( ρ log k ) for every i 6= i∗ ( x ) . Based on this insight , we design a new 1-Lipschitz convex function given by h ( x ) : = maxi∈ [ k ] fi ( x ) where fi ( x ) : = smax≤iρ ( vecV ( x ) ) + ρ ( k − i ) n−α for an appropriate α to be chosen later , where smax≤iρ ( vecV ( x ) ) : = ρ log ( ∑ j∈ [ i ] exp ( 〈vj , x〉+ ( k−j ) γ ρ ) ) . The key property satisfied by fi is that fi ≈ fj implies that ∇fi ≈ ∇fj for any i , j . This implies that near points of discontinuous gradients for h i.e. , points where argmaxi∈ [ k ] fi ( x ) changes , the resulting discontinuity in ∇h ( x ) is O ( 1 poly ( k ) ) . In contrast , the change in gradients of the original instance fV ( x ) near points of discontinuity is Ω ( 1 ) . If we apply randomized smoothing to h , the resulting function can then be shown to have pth order Lipschitz constant Õ ( k3p/2 ) . The precise details , proved in Lemma 4 , are technical and form the bulk of this paper . The same proof strategy immediately yields the same bound on the number of rounds for parallel randomized algorithms as long as the number of queries in each round is at most poly ( k ) . The reason is that poly ( k ) queries are still not sufficient to obtain information about more than one vector per round . Finally , the same proof strategy can be adapted to the quantum setting using the hybrid argument [ BBBV97 ] . See Appendix C for more details .

Summary: This paper studies the complexity of optimizing convex functions with Lipschitz $p$-th derivative. The authors prove a new lower bound $\tilde{O} (1 / \epsilon^{ \frac{2}{3p+1} } )$ that holds for deterministic, randomized and quantum algorithms. This lower bound matches the upper bound up to log factors.