Only tails matter: Average-Case Universality and Robustness in the Convex Regime

1 Introduction . The analysis of the average complexity of algorithms has a long story in computer science . Average-case complexity , for instance , drives much of the decisions made in cryptography ( Bogdanov & Trevisan , 2006 ) . Despite their relevance , average-case analyses are difficult to extend to other algorithms , partly because of the intrinsic issue of defining a typical distribution over problem instances . Recently though , Pedregosa & Scieur ( 2020 ) derived a framework to systemically evaluate the complexity of first-order methods when applied on distributions of quadratic minimization problems . This is done by relating the average-case convergence rate to the expected spectral distribution ( e.s.d ) of the objective function ’ s Hessian , which is a well-studied object on random matrix theory . Having access to this object in practice is a much stronger hypothesis when compared to the worst-case analysis that relies only on the values of the edges of this distribution . Paquette et al . ( 2020 ) extended the average-case framework by introducing a noisy generative model for the problems . They further derived the average complexity of the Nesterov Accelerated Method ( Nesterov , 2003 ) on a particular distribution . They showed the strong concentration of the metrics around a limiting value as dimensions go to infinity . Scieur & Pedregosa ( 2020 ) showed that for a strongly convex problem with eigenvalues supported on a contiguous interval , the optimal average-case complexity converges asymptotically to the one given by the Polyak Heavy Ball method ( Polyak , 1964 ) in the worst-case . 1.1 Current limitations of the average-case analysis . When analyzing the state of the art of average-case methods on quadratics problems , we observe significant limitations that we address in this paper . First , little is known about the convergence rate on convex problems . Also , optimal average-case algorithms require an exact estimation of the e.s.d to guarantee an optimal convergence rate , their convergence rate under inexact e.s.d . is not known . Finally , the non-smooth is also discussed in ( Pedregosa & Scieur , 2020 ) , but with little details . Convex problems . The minimization of non-strongly convex problems is drastically slower than their strongly convex counterpart , as Gradient Descent presents worst-case convergence in Θ ( 1t ) and Nesterov is Θ ( 1 t2 ) . In the strongly convex case , both the worst-case and averagecase are asymptotically equal . However , little is known on optimal average-case rates for convex problems , as well as the average-case complexity of classical methods such as gradient descent or Nesterov ’ s method , see ( Paquette et al. , 2020 ) . Exact estimation of the e.s.d . In ( Pedregosa & Scieur , 2020 ) , the theoretical study of optimal algorithms in the average-case requires an exact estimation of the e.s.d . of the problem class . Such estimation may be hard , nor impossible to obtain in practical scenarios . Despite showing good performance when the e.s.d . is estimated with empirical quantities , there are no theoretical guarantees on the performance of the method when the e.s.d . is poorly estimated . There is therefore a need to analyze the algorithm ’ s performance under different notions of uncertainty on the spectrum . This allows a practitioner to choose the best algorithm for a practical problem , even with imperfect a priori information . Non-smooth . Pedregosa & Scieur ( 2020 ) briefly introduce average-case optimal rates on non-smooth problems , when the e.s.d . is the Laguerre distribution e−λ . In this paper , we extend the analysis to the generalized Laguerre distribution λαe−λ , α > −1 . 1.2 Contributions . Our main contribution is a fine-grained analysis of the average-case complexity on convex quadratic problems : we show that a problem ’ s complexity depends on the concentration of the eigenvalues of e.s.d . around the edges of their support . From this perspective , we propose a family of optimal algorithms in the average-case , analyze their robustness , and finally exhibit a universality result for Nesterov ’ s method . More precisely , • ( Optimal algorithms ) . In Section 3 , we propose the Generalized Chebyshev Method ( GCM , Algorithm 1 ) , a family of algorithms whose parameters depend on the concentration of the e.s.d . around the edges of their support . If the parameters of the GCM method are set properly , the algorithm converges at an optimal average-case rate ( Theorem 3 for smooth problems , Theorem 6 for non-smooth problems ) , a rate that we show is faster than worst-case optimal methods like Nesterov acceleration . We show these rates to be representative of the practical performance of the algorithms in Fig . 6 , and retrieve the classical worst-case rates as limits of the average-case ( see Table 1 ) . • ( Robustness ) . Developing an optimal algorithm requires the knowledge of the exact e.s.d . However , in practical scenarios , we only have access to an approximation of the e.s.d . In Theorem 2 in Section 4 we analyze the rate of GCM in the presence of such a mismatch . We also analyze the optimal average-case rates of distributions representing the smooth convex , non-smooth convex , and strongly convex settings and compare them with the worst-case rates ( Table 1 ) . • ( Universality ) . Finally , in Theorem 4 , we analyze the asymptotic average-case convergence rate of Nesterov ’ s method . We show that its convergence rate is nearly optimal ( up to a logarithmic factor ) under some natural assumptions over the data , namely a concentration of eigenvalues around 0 similar to the Marchenko-Pastur measure . This contributes to the theoretical understanding of the numerical efficiency of Nesterov ’ s acceleration . 2 Average-Case Analysis . In this section , we recall the average-case analysis framework for random quadratic problems . The main result is Theorem 1 , which relates the expected error to the expected spectral Regime Worst-case Average-Case Strongly conv . ( 1 − Θ ( 1/√κ ) ) t ( 1 − Θ ( 1/√κ ) ) t Smooth conv . 1/t2 1/t2ξ+4 Convex 1/√t 1/tα+2 Table 1 : Comparison between function value worstcase and average-case convergence . κ is the condition number in the smooth strongly convex case . In the smooth convex case ξ > −1 is the concentration of eigenvalues around 0 ( see Assumption 1 ) and in the non-smooth case we consider dµ ∝ λαe−λ distribution and the residual polynomial . The one-to-one correspondence between the residual polynomials and first-order methods applied to quadratics will allow us to pose the problem of finding an optimal method as the best approximation problem in the space of polynomials . We define a random quadratic problem : Problem 1 . Let H ∈ Rd×d be a random symmetric positive-definite matrix independent to x ? ∈ Rd , a random vector that is the solution to the problem . We define the random quadratic minimization problem as min x∈Rd { f ( x ) : = 1 2 ( x−x ? ) > H ( x−x ? ) } . ( OPT ) We are interested on minimizing the expected errors E‖f ( xt ) − f ( x ? ) ‖ , the expected functionvalue gap , and E||∇f ( xt ) ||2 , the expected gradient norm , where xt is the t-th update of a first-order method starting from x0 and E is the expectation over the random variables H , x0 and x ? . The expectation we consider is over the problem and not over any randomness of the algorithm . In this paper , we consider the class of first-order methods ( F.O.M ’ s ) to minimize ( OPT ) . Methods in this class construct the iterates xt as xt ∈ x0 + span { ∇f ( x0 ) , . . . , ∇f ( xt−1 ) } . ( 1 ) That is , xt belongs to the span of previous gradients . This class of algorithms includes for instance gradient descent and momentum , but not quasi-Newton methods since the preconditioner could allow the iterates to go outside of the span . Furthermore , we will only consider oblivious methods , that is , methods in which the coefficients of the update are known in advance and do not depend on previous updates . This leaves out some methods such as conjugate gradient or methods with line-search . From First-Order Method to Polynomials . There is an intimate link between firstorder methods and polynomials that simplifies the analysis of quadratic objectives . The next proposition shows that , with this link , we can assign to each optimization method a polynomial that determines its convergence . Following Fischer ( 1996 ) , we will say a polynomial Pt is residual if Pt ( 0 ) = 1 . Proposition 1 . ( Hestenes et al. , 1952 ) Let xt be generated by a first-order method . Then there exists a residual polynomial Pt of degree t , that verifies xt − x ? = Pt ( H ) ( x0 − x ? ) . ( 2 ) Remark 1 . If the first-order method is further a momentum method , i.e . xt+1 = xt + ht∇f ( xt ) +mt ( xt − xt−1 ) . We can determine the polynomials by the recurrence P0 = 1 and Pt+1 ( λ ) = Pt ( λ ) + htλPt ( λ ) +mt ( Pt ( λ ) − Pt−1 ( λ ) ) . We note that while most popular F.O.M ’ s can be posed as a momentum method , the Nesterov method can not . A convenient way to collect statistics on the spectrum of a matrix is through its empirical spectral distribution . Definition 1 ( Expected spectral distribution ( e.s.d ) ) . . Let H be a random matrix with eigenvalues { λ1 , . . . , λd } . The empirical spectral distribution of H , called µH , is the probability measure µH : = 1 d ∑d i=1δλi , ( 3 ) where δλi is the Dirac delta , a distribution equal to zero everywhere except at λi and whose integral over the entire real line is equal to one . Since H is random , the empirical spectral distribution µH is a random variable in the space of measures . Its expectation over H is called the expected spectral distribution and we denote it µ : = EH [ µH ] . ( 4 ) We can link the e.s.d . of H to the convergence of a first-order method on the distribution of H. In the following we will consider x0 − x ? and H to be independent , with x0 − x ? sampled isotropically . Theorem 1 . Let xt be generated by a first-order method associated to the polynomial Pt , the measure µ the e.s.d . of H , and E [ ( x0 − x ? ) ( x0 − x ? ) T ] = R2I for some constant R. Then we can write the convergence metrics at time step t as E [ ‖xt − x ? ‖2 ] = R2 ∫ P 2t ( λ ) dµ ( λ ) , E [ f ( xt ) − f ( x ? ) ] = R2 ∫ P 2t ( λ ) λdµ ( λ ) and E [ ||∇f ( xt ) ||22 ] = R2 ∫ P 2t ( λ ) λ 2dµ ( λ ) . ( 5 ) This shows that polynomials are a powerful abstraction as they allow us to write all of our convergence metrics within the same framework . For simplicity , we set R2 = 1 and we will refer directly to the polynomials associated to a given method . We will refer to objective l as the one associated to the added λl term , i.e . the function-value is objective l = 1 . This framework is linked to the field of orthogonal polynomials by the next proposition . We construct an optimal method w.r.t . a given distribution through a family of orthogonal polynomials associated to it . Proposition 2 ( ( Pedregosa & Scieur , 2020 ) ) . Let P lt be defined as P lt : = arg minPt ( 0 ) =1 ∫ P 2t ( λ ) λ ldν ( λ ) . ( 6 ) Then ( P lt ) is the family of residual orthogonal polynomials w.r.t . to λl+1dν . This theorem further implies that the optimal first-order method is a momentum method as Favard ’ s theorem Marcellán & Álvarez-Nodarse ( 2001 ) tells us the orthogonal polynomials w.r.t . a given distribution are related through a three term recurrence , Pt+1 ( λ ) = atPt ( λ ) + btλPt ( λ ) + ( 1− at ) Pt−1 ( λ ) . ( 7 ) Following Remark 1 , the optimal method is derived from this recurrence as xt+1 = xt + ( at − 1 ) ( xt − xt−1 ) + bt∇f ( xt ) . ( 8 )

Summary: -- EDIT: I have updated my scores in response to clarifications --  The problem of optimizing a convex quadratic function via first order methods is considered. This is a well-understood problem from the worst case point of view, and its complexity will depend on the largest and smallest eigenvalues of the associated Hessian matrix. However, a nice recent average-case analysis by Pedrogosa and Scieur gives the following result: if the spectral density of the Hessian converges to a "nice" probability measure (such as the Manchenko-Pastur law), then first-order methods that are tailored to this density may converge faster. In fact, such methods can be obtained from the three-term recurrence for the orthogonal polynomials of the measure.  The present manuscript follows up on the Pedregosa/Scieur, but departs from in two specific ways. Firstly, it considers the case where the limiting spectral law is a beta distribution, and derives the corresponding iteration. Secondly, the authors analyze the performance of this iteration under "mispecification": that is, they allow for more general spectral measures than the beta, and only specify how these measures behave near the extremes. In spite of this, they are able to derive bounds of the same order as in the beta case.   A few other theoretical results are presented: worst-case rates for the method derived from the beta distribution; an analysis of Nesterov's method under conditions on the tail of the spectral measure, and a result on Laguerre spectral measures. These results generalize other theorems obtained in the Pedregosa/Scieur paper. The theory is complemented by a small numerical study.