Never Go Full Batch (in Stochastic Convex Optimization)

1 Introduction . Stochastic ConvexOptimization ( SCO ) is a fundamental problem that received considerable attention from the machine learning community in recent years [ 28 , 15 , 4 , 11 , 2 ] . In this problem , we assume a learner that is provided with a finite sample of convex functions drawn i.i.d . from an unknown distribution . The learner ’ s goal is to minimize the expected function . Owing to its simplicity , it serves as an almost ideal theoretical model for studying generalization properties of optimization algorithms ubiquitous in practice , particularly first-order methods which utilize only first derivatives of the loss rather than higher-order ones . One prominent approach for SCO—and learning more broadly—is to consider the empirical risk ( the average objective over the sample ) and apply a first-order optimization algorithm to minimize it . The problem of learning is then decoupled into controlling the optimization error over the empirical risk ( training error ) and bounding the difference between the empirical error and the expected error ( generalization error ) . In convex optimization , the convergence of different first-order methods has been researched extensively for many years ( e.g. , [ 26 , 25 , 5 ] ) , and we currently have a very good understanding of this setting in terms of upper as well lower bounds on worst-case complexity . However , in SCO where the generalization error must also be taken into account , our understanding is still lacking . In fact , this is one of the few theoretical learning models where the optimization method affects not only the optimization error but also the generalization error ( distinctively from models such as PAC learning and generalized linear models ) . In particular , it has been shown [ 28 , 15 ] that some minima of the empirical risk may obtain large generalization error , while other minima have a vanishingly small 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . generalization error . To put differently , learning in SCO is not only a question of minimizing the empirical risk , but also a question of how one minimizes it . However , the results of [ 28 , 15 ] leave open the question of whether concrete optimization also have different generalization properties . Towards better understanding , Amir et al . [ 2 ] recently studied the generalization properties of fullbatch gradient descent ( GD ) , where each step is taken with respect to the gradient of the empirical risk . For GD ( and a regularized variant thereof ) , they gave a lower bound on the generalization error as a function of iteration number , which is strictly larger than the well-known optimal rate obtained by stochastic gradient descent ( SGD ) , where each step is taken with respect to the gradient at a sampled example . Notably , the lower bound of [ 2 ] precisely matches the dimension-independent stability-based upper bound recently shown for full-batch GD by Bassily et al . [ 4 ] . The separation between full-batch GD and SGD is the first evidence that not only abstract Empirical RiskMinimizers may fail to generalize in SCO , but in fact also basic methods such as GD could be prone to such overfitting . A natural question is , then , whether overfitting is inherent to full-batch algorithms , that minimize the objective only through access to the exact empirical risk , or whether this suboptimality can be remedied by adding regularization , noise , smoothing , or any other mechanism for improving the generalization of GD . In this work we present and analyze a model of full-batch optimization algorithms for SCO . Namely , we focus on algorithms that access the empirical risk only via a first-order oracle that computes the exact ( full-batch ) gradient of the empirical loss , rather than directly accessing gradients with respect to individual samples . Our main result provides a negative answer to the question above by significantly generalizing and extending the result of Amir et al . [ 2 ] : we show that any optimization method that uses full-batch gradients needs at least Ω ( 1/ε4 ) iterations to minimize the expected loss to within ε error . This is in contrast with the empirical loss , which can be minimized with only $ ( 1/ε2 ) steps . Comparing SGD and GD in terms of the sample size = , we see that SGD converges to an optimal generalization error of $ ( 1/ √ = ) after $ ( = ) iterations , whereas a full-batch method must perform Ω ( =2 ) iterations to achieve the same $ ( 1/ √ = ) test error . We emphasize that we account here for the oracle complexity , which coincides with the iteration complexity in the case of gradient methods . In terms of individual gradients calculations , while SGD uses at most $ ( = ) gradient calculations ( one sample per iteration ) , a full-batch method will perform Ω ( =3 ) calculations ( = samples per iteration ) . The above result is applicable to a wide family of full-batch learning algorithms : regularized GD ( with any data-independent regularization function ) , noisy GD , GDwith line-search or adaptive step sizes , GD with momentum , proximal methods , coordinate methods , and many more . Taken together with upper bound of Bassily et al . [ 4 ] , we obtain a sharp rate of Θ ( 1/ε4 ) for the generalizationcomplexity of full-batch methods . Surprisingly , this rate is achieved by standard GD ( with an unusual step-size choice of η = Θ ( ε3 ) ) , and it can not be improved by adding regularization of any sort , nor by adding noise or any other form of implicit/explicit bias . 1.1 Related work . This work extends and generalizes the results of Amir et al . [ 2 ] who proved generalization lower bounds for GD ( and a specific instance of regularized GD ) . Our work shows that in fact any full-batch method will suffer from similar lower bounds . Our construction builds upon the one used in [ 2 ] , which in turn builds upon previous constructions [ 4 , 28 ] . However , our arguments and proofs here are more challenging , as we need to reason about a general family of algorithms , and not about a specific algorithm whose trajectory can be analyzed directly . Our developments also build on ideas from the literature on oracle complexity lower bounds in optimization [ 25 , 26 , 30 , 8 , 12 , 9 ] . In particular , we first prove our result in the simplified setting of algorithms constrained to the span of observed gradients [ 25 , 26 ] and subsequently lift it to general algorithms using a random high-dimensional embedding technique proposed byWoodworth and Srebro [ 30 ] and later refined in [ 8 , 12 ] . However , while these works lower bound what we call the empirical risk , we lower bound the generalization error . This requires us to develop a somewhat different argument for how the span of the gradients evolve during the optimization : in prior work , the algorithm learns the component of the solution coordinate by coordinate , whereas in our work the true ( generalizing ) solution is present in the observed gradients from the first query , but spurious sampling artifacts drown it out . Empirical studies ( outside of the scope of SCO ) support the claim that generalization capabilities degrade with the increase of the batch size . Specifically , Zhu et al . [ 33 ] indicates that SGD outperforms GD in terms of generalization . The works of Keskar et al . [ 22 ] and Hoffer et al . [ 20 ] exhibit a similar phenomenon in which small-batch SGD generalizes better than large-batch SGD with the same iteration budget . We provide the first theoretical evidence for this phenomenon for convex losses . Several theoretical studies explore the convergence of stochastic methods that use mini-batches [ 10 , 23 , 31 ] . Note that this setting differs from ours , as they assume access to minibatches sampled without replacement whereas full-batch means we reuse the same ( full ) batch with each gradient step . There has also been recent progress in improving the generalization capabilities of GD . Wu et al . [ 32 ] interprets mini-batch SGD as a noisy version of GD . They propose a modified algorithm with noise injected to the full-batch gradients . Geiping et al . [ 16 ] propose a GD-based training scheme that achieves CIFAR-10 generalization performance comparable to standard SGD training . Interestingly , both proposed algorithms require access to sample-points and are therefore not “ fullbatch ” by our definition : The scheme [ 32 ] requires sample-point data for computing the noise , while the GD variant [ 16 ] uses mini-batch statistics to compute a regularization term ( as well as batch normalization ) . Our work shows that ( in SCO ) this is unavoidable : namely , no data-independent noise or full-batch regularization can be used to improve generalization at a reasonable computational budget . Several other works study the generalization performance of GD [ 29 , 17 , 21 , 24 ] . The work of Soudry et al . [ 29 ] , for example , examines GD on unregularized logistic regression problems . They show that , in the limit , GD converges to a well-generalizing solution by arguing about the bias of the algorithm . Interestingly , both our and their results require slow-training , beyond what is required for empirical error optimization . Another work that highlights the slow convergence of GD is that of Bassily et al . [ 4 ] . They were the first to address uniform stability of ( non-smooth ) GD and SGD , and provided tight bounds . Stability entails generalization , hence our results lead to stability lower bounds for any full-batch method . Consequently , we extend the lower bounds for GD in the work of Bassily et al . [ 4 ] to a wider class . It might be thought that the instability argument of Bassily et al . [ 4 ] can be used to obtain similar generalization lower bounds—however , we note that their techniques also prove instability of SGD ( which does generalize ) . Hence , instability does not immediately imply , in this setting , lack of generalization . Finally , we note that under smoothness and strong convexity , it is well known that improved rates can be obtained . Specifically , using the stability bound of Bousquet and Elisseeff [ 6 ] one can show that we can achieve generalization error of $ ( 1/ √ = ) after $ ( = ) iterations if the population risk is $ ( 1 ) -strongly convex . The arguments of Hardt et al . [ 19 ] imply generalization bound to instances where every sample risk is $ ( √ = ) smooth . Our result implies that , even though these special families of functions enjoy appealing learning rates , in general it is impossible to obtain better rates by strong-convexifying or smoothing problem instances via first-order full-batch oracle queries . 2 Problem Setup and Main Results . We study the standard setting of stochastic convex optimization . In this setting , a learning problem is specified by a fixed domain W ⊆ ℝ3 in 3-dimensional Euclidean space , and a loss function 5 : W × Z → ℝ , which is both convex and ! -Lipschitz with respect to its first argument ( that is , for any I ∈ Z the function 5 ( F ; I ) is ! -Lipschitz and convex with respect to F ) . In particular , throughout the paper , our construction consists of 1-Lipschitz functions and we will focus on a fixed domain W defined to be the unit Euclidean ball in ℝ3 , namely W = { F : ‖F‖2 ≤ 1 } . We also assume that there exists an unknown distribution over parameters I and the goal of the learner is to optimize the true risk ( or true loss , or population risk ) defined as follows : ( F ) B E I∼ [ 5 ( F ; I ) ] , ( 1 ) We assume that a sample ( = { I1 , . . . , I= } is drawn from the distribution , and the learner has to output F ( ∈ W ( the exact access the learner has to the sample , and how F ( may depend on ( is discussed below ) . We require the solution to be ε-optimal in expectation for some parameter ε > 0 , i.e. , E ( ∼ = [ ( F ( ) ] − min F★∈W ( F★ ) ≤ ε . As discussed , the standard setting assumes that the learner has direct access to the i.i.d . sample , as well as to the gradients of the loss function ( i.e. , a first-order oracle ) . In this work , though , we focus on a specific family full-batch methods . Hence , the optimization process is described as follows : First , an i.i.d . sample ( = ( I1 , . . . , I= ) is drawn from . Then , the learner is provided with access only to the empirical risk via a full-batch first-order oracle which we define next . Full-batch first-order oracle . Consider a fixed sample ( = ( I1 , . . . , I= ) of size = , drawn i.i.d . from . The empirical risk over the sample ( is ( ( F ) = 1 = =∑ 8=1 5 ( F ; I8 ) . Then , a full-batch first-order oracle is a procedure that , given input F ∈W , outputs O ( F ) : = ( ∇ ( ( F ) ; ( ( F ) ) . where ∇ ( ( F ) is an empirical risk sub-gradient of the form ∇ ( ( F ) = 1 = =∑ 8=1 ∇ 5 ( F ; I8 ) , ( 2 ) and each sub-gradient ∇ 5 ( F , I8 ) is computed by the oracle as a function of F and I8 ( that is , independently of I 9 for 9 ≠ 8 ) . We emphasize that the sample is fixed throughout the optimization , so that the oracle computes the gradient of the same empirical risk function at every call , hence the name full-batch . Note that the subgradient with respect to a single data point , i.e. , ∇ 5 ( F ; I8 ) , is not accessible through this oracle , which only returns the average gradient over the sample ( . Notice that our definition above is slightly narrower than a general sub-gradient oracle for the empirical risk due to the requirement that the sub-gradients ∇ 5 ( F , I8 ) are chosen independently of I 9 for 9 ≠ 8 – since we provide here with a lower bound , this restriction strengthens our result . We make this restriction to avoid some degenerate constructions ( that in fact can even be used to fail SGD if the gradient at I8 may depend on the whole sample ) , which are of no practical implications . Full-batch first-order algorithm . A full-batch ( first-order ) method is naturally defined as any algorithm that has access to the optimization objective—namely the empirical risk ( —only via the full-batch first order oracle . In particular , if FC is the C ’ th query of the algorithm to the full-batch oracle then FC has to be of the form FC = & C ( O ( F0 ) , . . . , O ( FC−1 ) ) , ( 3 ) where & C : ( ℝ3+1 ) C → W is a fixed ( possibly randomized ) mapping . At the end of the process the algorithm outputs F ( . We study the algorithm ’ s oracle complexity , which is the number of iterations ) the algorithm performs before halting . Therefore , we assume without loss of generality that F ( = F ) , i.e. , the algorithm ’ s output is its ) ’ th query .

Summary: This work studies the generalization performance of algorithms in stochastic convex optimization. Specifically, the authors study the full-batch algorithms where only full gradient and function value information are accessible. By constructing a hard instance of the objective function, the authors show that any full-batch algorithm needs at least $\Omega(1/\epsilon^4)$ iterations to obtain an $\epsilon$-risk function, while stochastic algorithms such as SGD only need $O(1/\epsilon^2)$.