Algorithmic Instabilities of Accelerated Gradient Descent

We study the algorithmic stability of Nesterov ’ s accelerated gradient method . For convex quadratic objectives , Chen et al . [ 10 ] proved that the uniform stability of the method grows quadratically with the number of optimization steps , and conjectured that the same is true for the general convex and smooth case . We disprove this conjecture and show , for two notions of algorithmic stability ( including uniform stability ) , that the stability of Nesterov ’ s accelerated method in fact deteriorates exponentially fast with the number of gradient steps . This stands in sharp contrast to the bounds in the quadratic case , but also to known results for non-accelerated gradient methods where stability typically grows linearly with the number of steps . 1 Introduction . Algorithmic stability has emerged over the last two decades as a central tool for generalization analysis of learning algorithms . While the classical approach in generalization theory originating in the PAC learning framework appeal to uniform convergence arguments , more recent progress on stochastic convex optimization models , starting with the pioneering work of Bousquet and Elisseeff [ 6 ] and Shalev-Shwartz et al . [ 24 ] , has relied on stability analysis for deriving tight generalization results for convex risk minimizing algorithms . Perhaps the most common form of algorithmic stability is the so called uniform stability [ 6 ] . Roughly , the uniform stability of a learning algorithm is the worst-case change in its output model , in terms of its loss on an arbitrary example , when replacing a single sample in the data set used for training . Bousquet and Elisseeff [ 6 ] initially used uniform stability to argue about the generalization of empirical risk minimization with strongly convex losses . Shalev-Shwartz et al . [ 24 ] revisited this concept and studied the stability effect of regularization on the generalization of convex models . Their bounds were recently improved in a variety of ways [ 13 , 14 , 7 ] and their approach has been influential in a variety of settings ( e.g. , 18 , 16 , 9 ) . In fact , to this day , algorithmic stability is essentially the only general approach for obtaining tight ( dimension free ) generalization bounds for convex optimization algorithms applied to the empirical risk ( see 24 , 12 ) . Significant focus has been put recently on studying the stability properties of iterative optimization algorithms . Hardt et al . [ 17 ] considered stochastic gradient descent ( SGD ) and gave the first bounds on its uniform stability for a convex and smooth loss function , that grow linearly with the number of optimization steps . As observed by Feldman and Vondrak [ 13 ] and Chen et al . [ 10 ] , their arguments also apply with minor modifications to full-batch gradient descent ( GD ) . Bassily et al . [ 5 ] exhibited a significant gap in stability between the smooth and non-smooth cases , showing that non-smooth GD and SGD are inherently less stable than their smooth counterparts . Even further , algorithmic stability has also been used as an analysis technique in stochastic mini-batched iterative optimization ( e.g. , 25 , 1 ) , and has been proved crucial to the design and analysis of differentially private optimization algorithms [ 26 , 4 , 15 ] , both of which focusing primarily on smooth optimization . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Having identified smoothness as key to algorithmic stability of iterative optimization methods , the following fundamental question emerges : how stable are optimal methods for smooth convex optimization ? In particular , what is the algorithmic stability of the celebrated Nesterov accelerated gradient ( NAG ) method [ 22 ] —a cornerstone of optimal methods in convex optimization ? Besides being a basic and natural question in its own right , its resolution could have important implications to the design and analysis of optimization algorithms , as well as serve to deepen our understanding of the generalization properties of iterative gradient methods . Chen et al . [ 10 ] addressed this question in the case of convex quadratic objectives and derived bounds on the uniform stability of NAG that grow quadratically with the number of gradient steps ( as opposed to the linear growth known for GD ) . They conjectured that similar bounds hold true more broadly , but fell short of proving this for general convex and smooth objectives . Our work is aimed at filling this gap . 1.1 Our Results . We establish tight algorithmic stability bounds for the Nesterov accelerated gradient method ( NAG ) . We show that , somewhat surprisingly , the uniform stability of NAG grows exponentially fast with the number of steps in the general convex and smooth setting . Namely , the uniform stability of ) -steps NAG with respect to a dataset of = examples is in general exp ( Ω ( ) ) ) /= , and in particular , after merely ) = $ ( log = ) steps the stability becomes the trivial Ω ( 1 ) . This result demonstrates a sharp contrast between the stability of NAG in the quadratic case and in the general convex , and disproves the conjecture of Chen et al . [ 10 ] that the uniform stability of NAG in the general convex setting is $ ( ) 2/= ) , as in the case of a quadratic objective . Our results in fact apply to a simpler notion of stability—one that is arguably more fundamental in the context of iterative optimization methods—which we term initialization stability . The initialization stability of an algorithm ( formally defined in Section 2 below ) measures the sensitivity of ’ s output to an ²-perturbation in its initialization point . For this notion , we demonstrate a construction of a smooth and convex objective function such that , for sufficiently small ² , the stability of ) -steps NAG is lower bounded by exp ( Ω ( ) ) ) ² . Here again , we exhibit a dramatic gap between the quadratic and general convex cases : for quadratic objectives , we show that the initialization stability of NAG is upper bounded by $ ( ) ² ) . For completeness , we also prove initialization stability upper bounds in a few relevant convex optimization settings : for GD , we analyze both the smooth and non-smooth cases ; for NAG , we give bounds for quadratic objectives as well as for general smooth ones . Table 1 summarizes the stability bounds we establish compared to existing bounds in the literature . Note in particular the remarkable exponential gap between the stability bounds for GD and NAG in the general smooth case , with respect to both stability definitions . Stability lower bounds for NAG are discussed in Sections 3 and 4 ; initialization stability upper bounds for the various settings and additional uniform stability bounds are detailed in the full version of the paper [ 3 ] . Finally , we remark that our focus here is on the general convex ( and smooth ) case , and we do not provide formal results for the strongly convex case . However , we argue that stability analysis in the latter case is not as compelling as in the general case . Indeed , a strongly convex objective admits a unique minimum , and so NAG will converge to an ²-neighborhood of this minimum in $ ( log ( 1/² ) ) steps from any initialization , at which point its stability becomes $ ( ² ) ; thus , with strong convexity perturbations in initialization get quickly washed away as the algorithm rapidly converges to the unique optimum . ( A similar reasoning also applies to uniform stability with strongly convex losses . ) 1.2 Overview of Main Ideas and Techniques . We now provide some intuition to our constructions and highlight some of the key ideas leading to our results . We start by revisiting the analysis of the quadratic case which is simpler and better understood . Why NAG is stable for quadratics : Consider a quadratic function 5 with Hessian matrix 0 . For analyzing the initialization stability of NAG , let us consider two runs of the method initialized at G0 , G̃0 respectively , and let ( GC , HC ) , ( G̃C , H̃C ) denote the corresponding NAG iterates at step C. Further , let us denote by ∆GC , GC − G̃C the difference between the two sequences of iterates . Using the update rule of NAG ( see Eqs . ( 1 ) and ( 2 ) below ) and the fact that for a quadratic 5 , differences between gradients can be expressed as ∇ 5 ( G ) − ∇ 5 ( G ′ ) = ( G − G ′ ) for any G , G ′ ∈ ℝ3 , it is straightforward to show that the distance ∆GC evolves according to ∆GC+1 = ( − η ) ( ( 1 + γC ) ∆GC − γC∆GC−1 ) . This recursion can be naturally put in matrix form , leading to : ( ∆G C+1 ∆GC ) = C∏ : =1 ( ( 1 + γ : ) −γ : 0 ) ( ∆G1 ∆G0 ) , where here = − η . Thus , for a quadratic 5 , bounding the divergence ‖∆GC ‖ between the two NAG sequences reduces to controlling the operator norm of the matrix product above , namely C∏ : =1 ( ( 1 + γ : ) −γ : 0 ) . Remarkably , it can be shown that this norm is $ ( C ) for any 0 and any choice of −1 ≤ γ1 , . . . , γC ≤ 1 . ( This can be seen by writing the Schur decomposition of the involved matrices , as we show in the full version of the paper [ 3 ] .1 ) As a consequence , the initialization stability of NAG for a quadratic objective 5 is shown to grow only linearly with the number of steps C. What breaks down in the general convex case : For a general convex ( twice-differentiable and smooth ) 5 , the Hessian matrix is of course no longer fixed across the execution . Assuming for simplicity the one-dimensional case , similar arguments show that the relevant operator norm is of the form C∏ : =1 ( ( 1 + γ : ) : −γ : : 0 ) , where 0 ≤ 1 , . . . , C ≤ 1 are related to Hessians of 5 taken at suitable points along the optimization trajectory . However , if : are allowed to vary arbitrarily between steps , the matrix product above might explode exponentially fast , even in the one-dimensional case ! Indeed , fix γ : = 0.9 for all : , and set : = 0 whenever : mod 3 = 0 and : = 1 otherwise ; then using simple linear algebra the operator norm of interest can be shown to satisfy ( ( 0 01 0 ) ( 1.9 −0.91 0 ) ( 1.9 −0.91 0 ) ) C/3 = ( 0 02.71 −1.71 ) C/3 ≥ 1.15C . How a hard function should look like : The exponential blowup we exhibited above hinged on a worst-case sequence 1 , . . . , C that varies significantly between consecutive steps . It remains unclear , however , what does this imply for the actual optimization setup we care about , and whether such a sequence can be realized by Hessians of a convex and smooth function 5 . Our main results essentially answer the latter question on the affirmative and build upon a construction of such a function 5 that directly imitates such a bad sequence . Concretely , we generate a hard function inductively based on a running execution of NAG , where in each step we amend the construction with a “ gadget ” function having a piecewise-constant Hessian ( that equals either 0 or the maximal β ) ; see Fig . 1 for an illustration of this construct . The interval pieces are carefully chosen based on the NAG iterates computed so far in a way that a slightly perturbed 1Chen et al . [ 10 ] give an alternative argument based on Chebyshev polynomials . execution would traverse through intervals with an appropriate pattern of Hessians that induces a behaviour similar to the one exhibited by the matrix products above , leading to an exponential blowup in the stability terms . Fig . 2 shows a simulation of the divergence between the two trajectories of NAG on the objective function we construct , illustrating how the divergence fluctuates between positive and negative values , with its absolute value growing exponentially with time . More technical details on this construction can be found in Section 3 . From initialization stability to uniform stability : Finally , we employ a simple reduction to translate our results regarding initialization stability to relate to uniform stability in the context of empirical risk optimization , where one uses ( full-batch ) NAG to minimize the ( convex , smooth ) empirical risk induced by a sample ( of = examples . Concretely , we show that by replacing a single example in ( , we can arrive at a scenario where after one step of NAG on the original and modified samples the respective iterates are ² = Θ ( 1/= ) away from each other , whereas in the remaining steps both empirical risks simulate our bad function from before . Thus , we again end up with an exponential increase in the divergence between the two executions , that leads to a similar increase in the algorithmic ( uniform ) stability : the latter becomes as large as Ω ( 1 ) after merely ) = $ ( log = ) steps of full-batch NAG . The formal details of this reduction can be found in Section 4 .

Summary: This paper studies the stability of Nesterov accelerated gradient in the initialization and in the sense of uniform stability. The authors prove lower bounds on the stability parameters, that diverge exponentially in the number of iterations, by building clever one-dimensional adversarial examples. This proves that in the general convex smooth case, Nesterov accelerated gradient can be a lot more instable than in the quadratic case.