Never Go Full Batch (in Stochastic Convex Optimization)

1 Introduction . Stochastic ConvexOptimization ( SCO ) is a fundamental problem that received considerable attention from the machine learning community in recent years [ 28 , 15 , 4 , 11 , 2 ] . In this problem , we assume a learner that is provided with a finite sample of convex functions drawn i.i.d . from an unknown distribution . The learner ’ s goal is to minimize the expected function . Owing to its simplicity , it serves as an almost ideal theoretical model for studying generalization properties of optimization algorithms ubiquitous in practice , particularly first-order methods which utilize only first derivatives of the loss rather than higher-order ones . One prominent approach for SCO—and learning more broadly—is to consider the empirical risk ( the average objective over the sample ) and apply a first-order optimization algorithm to minimize it . The problem of learning is then decoupled into controlling the optimization error over the empirical risk ( training error ) and bounding the difference between the empirical error and the expected error ( generalization error ) . In convex optimization , the convergence of different first-order methods has been researched extensively for many years ( e.g. , [ 26 , 25 , 5 ] ) , and we currently have a very good understanding of this setting in terms of upper as well lower bounds on worst-case complexity . However , in SCO where the generalization error must also be taken into account , our understanding is still lacking . In fact , this is one of the few theoretical learning models where the optimization method affects not only the optimization error but also the generalization error ( distinctively from models such as PAC learning and generalized linear models ) . In particular , it has been shown [ 28 , 15 ] that some minima of the empirical risk may obtain large generalization error , while other minima have a vanishingly small 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . generalization error . To put differently , learning in SCO is not only a question of minimizing the empirical risk , but also a question of how one minimizes it . However , the results of [ 28 , 15 ] leave open the question of whether concrete optimization also have different generalization properties . Towards better understanding , Amir et al . [ 2 ] recently studied the generalization properties of fullbatch gradient descent ( GD ) , where each step is taken with respect to the gradient of the empirical risk . For GD ( and a regularized variant thereof ) , they gave a lower bound on the generalization error as a function of iteration number , which is strictly larger than the well-known optimal rate obtained by stochastic gradient descent ( SGD ) , where each step is taken with respect to the gradient at a sampled example . Notably , the lower bound of [ 2 ] precisely matches the dimension-independent stability-based upper bound recently shown for full-batch GD by Bassily et al . [ 4 ] . The separation between full-batch GD and SGD is the first evidence that not only abstract Empirical RiskMinimizers may fail to generalize in SCO , but in fact also basic methods such as GD could be prone to such overfitting . A natural question is , then , whether overfitting is inherent to full-batch algorithms , that minimize the objective only through access to the exact empirical risk , or whether this suboptimality can be remedied by adding regularization , noise , smoothing , or any other mechanism for improving the generalization of GD . In this work we present and analyze a model of full-batch optimization algorithms for SCO . Namely , we focus on algorithms that access the empirical risk only via a first-order oracle that computes the exact ( full-batch ) gradient of the empirical loss , rather than directly accessing gradients with respect to individual samples . Our main result provides a negative answer to the question above by significantly generalizing and extending the result of Amir et al . [ 2 ] : we show that any optimization method that uses full-batch gradients needs at least Ω ( 1/ε4 ) iterations to minimize the expected loss to within ε error . This is in contrast with the empirical loss , which can be minimized with only $ ( 1/ε2 ) steps . Comparing SGD and GD in terms of the sample size = , we see that SGD converges to an optimal generalization error of $ ( 1/ √ = ) after $ ( = ) iterations , whereas a full-batch method must perform Ω ( =2 ) iterations to achieve the same $ ( 1/ √ = ) test error . We emphasize that we account here for the oracle complexity , which coincides with the iteration complexity in the case of gradient methods . In terms of individual gradients calculations , while SGD uses at most $ ( = ) gradient calculations ( one sample per iteration ) , a full-batch method will perform Ω ( =3 ) calculations ( = samples per iteration ) . The above result is applicable to a wide family of full-batch learning algorithms : regularized GD ( with any data-independent regularization function ) , noisy GD , GDwith line-search or adaptive step sizes , GD with momentum , proximal methods , coordinate methods , and many more . Taken together with upper bound of Bassily et al . [ 4 ] , we obtain a sharp rate of Θ ( 1/ε4 ) for the generalizationcomplexity of full-batch methods . Surprisingly , this rate is achieved by standard GD ( with an unusual step-size choice of η = Θ ( ε3 ) ) , and it can not be improved by adding regularization of any sort , nor by adding noise or any other form of implicit/explicit bias . 1.1 Related work . This work extends and generalizes the results of Amir et al . [ 2 ] who proved generalization lower bounds for GD ( and a specific instance of regularized GD ) . Our work shows that in fact any full-batch method will suffer from similar lower bounds . Our construction builds upon the one used in [ 2 ] , which in turn builds upon previous constructions [ 4 , 28 ] . However , our arguments and proofs here are more challenging , as we need to reason about a general family of algorithms , and not about a specific algorithm whose trajectory can be analyzed directly . Our developments also build on ideas from the literature on oracle complexity lower bounds in optimization [ 25 , 26 , 30 , 8 , 12 , 9 ] . In particular , we first prove our result in the simplified setting of algorithms constrained to the span of observed gradients [ 25 , 26 ] and subsequently lift it to general algorithms using a random high-dimensional embedding technique proposed byWoodworth and Srebro [ 30 ] and later refined in [ 8 , 12 ] . However , while these works lower bound what we call the empirical risk , we lower bound the generalization error . This requires us to develop a somewhat different argument for how the span of the gradients evolve during the optimization : in prior work , the algorithm learns the component of the solution coordinate by coordinate , whereas in our work the true ( generalizing ) solution is present in the observed gradients from the first query , but spurious sampling artifacts drown it out . Empirical studies ( outside of the scope of SCO ) support the claim that generalization capabilities degrade with the increase of the batch size . Specifically , Zhu et al . [ 33 ] indicates that SGD outperforms GD in terms of generalization . The works of Keskar et al . [ 22 ] and Hoffer et al . [ 20 ] exhibit a similar phenomenon in which small-batch SGD generalizes better than large-batch SGD with the same iteration budget . We provide the first theoretical evidence for this phenomenon for convex losses . Several theoretical studies explore the convergence of stochastic methods that use mini-batches [ 10 , 23 , 31 ] . Note that this setting differs from ours , as they assume access to minibatches sampled without replacement whereas full-batch means we reuse the same ( full ) batch with each gradient step . There has also been recent progress in improving the generalization capabilities of GD . Wu et al . [ 32 ] interprets mini-batch SGD as a noisy version of GD . They propose a modified algorithm with noise injected to the full-batch gradients . Geiping et al . [ 16 ] propose a GD-based training scheme that achieves CIFAR-10 generalization performance comparable to standard SGD training . Interestingly , both proposed algorithms require access to sample-points and are therefore not “ fullbatch ” by our definition : The scheme [ 32 ] requires sample-point data for computing the noise , while the GD variant [ 16 ] uses mini-batch statistics to compute a regularization term ( as well as batch normalization ) . Our work shows that ( in SCO ) this is unavoidable : namely , no data-independent noise or full-batch regularization can be used to improve generalization at a reasonable computational budget . Several other works study the generalization performance of GD [ 29 , 17 , 21 , 24 ] . The work of Soudry et al . [ 29 ] , for example , examines GD on unregularized logistic regression problems . They show that , in the limit , GD converges to a well-generalizing solution by arguing about the bias of the algorithm . Interestingly , both our and their results require slow-training , beyond what is required for empirical error optimization . Another work that highlights the slow convergence of GD is that of Bassily et al . [ 4 ] . They were the first to address uniform stability of ( non-smooth ) GD and SGD , and provided tight bounds . Stability entails generalization , hence our results lead to stability lower bounds for any full-batch method . Consequently , we extend the lower bounds for GD in the work of Bassily et al . [ 4 ] to a wider class . It might be thought that the instability argument of Bassily et al . [ 4 ] can be used to obtain similar generalization lower bounds—however , we note that their techniques also prove instability of SGD ( which does generalize ) . Hence , instability does not immediately imply , in this setting , lack of generalization . Finally , we note that under smoothness and strong convexity , it is well known that improved rates can be obtained . Specifically , using the stability bound of Bousquet and Elisseeff [ 6 ] one can show that we can achieve generalization error of $ ( 1/ √ = ) after $ ( = ) iterations if the population risk is $ ( 1 ) -strongly convex . The arguments of Hardt et al . [ 19 ] imply generalization bound to instances where every sample risk is $ ( √ = ) smooth . Our result implies that , even though these special families of functions enjoy appealing learning rates , in general it is impossible to obtain better rates by strong-convexifying or smoothing problem instances via first-order full-batch oracle queries . 2 Problem Setup and Main Results . We study the standard setting of stochastic convex optimization . In this setting , a learning problem is specified by a fixed domain W ⊆ ℝ3 in 3-dimensional Euclidean space , and a loss function 5 : W × Z → ℝ , which is both convex and ! -Lipschitz with respect to its first argument ( that is , for any I ∈ Z the function 5 ( F ; I ) is ! -Lipschitz and convex with respect to F ) . In particular , throughout the paper , our construction consists of 1-Lipschitz functions and we will focus on a fixed domain W defined to be the unit Euclidean ball in ℝ3 , namely W = { F : ‖F‖2 ≤ 1 } . We also assume that there exists an unknown distribution over parameters I and the goal of the learner is to optimize the true risk ( or true loss , or population risk ) defined as follows : ( F ) B E I∼ [ 5 ( F ; I ) ] , ( 1 ) We assume that a sample ( = { I1 , . . . , I= } is drawn from the distribution , and the learner has to output F ( ∈ W ( the exact access the learner has to the sample , and how F ( may depend on ( is discussed below ) . We require the solution to be ε-optimal in expectation for some parameter ε > 0 , i.e. , E ( ∼ = [ ( F ( ) ] − min F★∈W ( F★ ) ≤ ε . As discussed , the standard setting assumes that the learner has direct access to the i.i.d . sample , as well as to the gradients of the loss function ( i.e. , a first-order oracle ) . In this work , though , we focus on a specific family full-batch methods . Hence , the optimization process is described as follows : First , an i.i.d . sample ( = ( I1 , . . . , I= ) is drawn from . Then , the learner is provided with access only to the empirical risk via a full-batch first-order oracle which we define next . Full-batch first-order oracle . Consider a fixed sample ( = ( I1 , . . . , I= ) of size = , drawn i.i.d . from . The empirical risk over the sample ( is ( ( F ) = 1 = =∑ 8=1 5 ( F ; I8 ) . Then , a full-batch first-order oracle is a procedure that , given input F ∈W , outputs O ( F ) : = ( ∇ ( ( F ) ; ( ( F ) ) . where ∇ ( ( F ) is an empirical risk sub-gradient of the form ∇ ( ( F ) = 1 = =∑ 8=1 ∇ 5 ( F ; I8 ) , ( 2 ) and each sub-gradient ∇ 5 ( F , I8 ) is computed by the oracle as a function of F and I8 ( that is , independently of I 9 for 9 ≠ 8 ) . We emphasize that the sample is fixed throughout the optimization , so that the oracle computes the gradient of the same empirical risk function at every call , hence the name full-batch . Note that the subgradient with respect to a single data point , i.e. , ∇ 5 ( F ; I8 ) , is not accessible through this oracle , which only returns the average gradient over the sample ( . Notice that our definition above is slightly narrower than a general sub-gradient oracle for the empirical risk due to the requirement that the sub-gradients ∇ 5 ( F , I8 ) are chosen independently of I 9 for 9 ≠ 8 – since we provide here with a lower bound , this restriction strengthens our result . We make this restriction to avoid some degenerate constructions ( that in fact can even be used to fail SGD if the gradient at I8 may depend on the whole sample ) , which are of no practical implications . Full-batch first-order algorithm . A full-batch ( first-order ) method is naturally defined as any algorithm that has access to the optimization objective—namely the empirical risk ( —only via the full-batch first order oracle . In particular , if FC is the C ’ th query of the algorithm to the full-batch oracle then FC has to be of the form FC = & C ( O ( F0 ) , . . . , O ( FC−1 ) ) , ( 3 ) where & C : ( ℝ3+1 ) C → W is a fixed ( possibly randomized ) mapping . At the end of the process the algorithm outputs F ( . We study the algorithm ’ s oracle complexity , which is the number of iterations ) the algorithm performs before halting . Therefore , we assume without loss of generality that F ( = F ) , i.e. , the algorithm ’ s output is its ) ’ th query .

Summary: This paper considers a problem where one is trying to find an input that minimizes the expected value of a function that one has sample values of. Its goal is to show that algorithms that only have the ability to query the gradient of the average of the samples at specified points may need more queries than a function that can check the gradient on individual samples. In order to do that, the paper finds a class of probability distributions over functions for which any algorithm that gains information on the functions only by querying the gradient of the average of the sampled function needs $\Omega(1/\epsilon^4)$ queries to get within $\epsilon$ of the minimum expected loss.