Coresets for Accelerating Incremental Gradient Methods

1 INTRODUCTION . Mathematical optimization lies at the core of training large-scale machine learning systems , and is now widely used over massive data sets with great practical success , assuming sufficient data resources are available . Achieving this success , however , also requires large amounts of ( often GPU ) computing , as well as concomitant financial expenditures and energy usage ( Strubell et al. , 2019 ) . Significantly decreasing these costs without decreasing the learnt system ’ s resulting accuracy is one of the grand challenges of machine learning and artificial intelligence today ( Asi & Duchi , 2019 ) . Training machine learning models often reduces to the problem of optimizing a regularized empirical risk function . Given a convex loss l , and a µ-strongly convex regularizer r , one aims to find model parameter vector x∗ over the parameter space X that minimizes the loss f over the training data V : x∗ ∈ argmin x∈X f ( x ) , f ( x ) : = ∑ i∈V fi ( x ) + r ( x ) , fi ( x ) = l ( x , ( ai , yi ) ) , ( 1 ) where V = { 1 , . . . , n } is an index set of the training data , and functions fi : Rd → R are associated with training examples ( ai , yi ) , where ai ∈ Rd is the feature vector , and yi is the point i ’ s label . The standard Gradient Descent can find the minimizer of this problem , but requires repeated computations of the full gradient ∇f ( x ) —sum of the gradients over all training data points/functions i—and is therefore prohibitive for massive data sets . This issue is further exacerbated in case of deep neural networks where gradient computations ( backpropagation ) are expensive . Incremental Gradient ( IG ) methods , such as Stochastic Gradient Descent and its accelerated variants , including SGD with momentum ( Qian , 1999 ) , Adagrad ( Duchi et al. , 2011 ) , Adam ( Kingma & Ba , 2014 ) , SAGA ( Defazio et al. , 2014 ) , and SVRG ( Johnson & Zhang , 2013 ) iteratively estimate the gradient on random subsets/batches of training data . While this provides an unbiased estimate of the full gradient , the randomized batches introduce variance in the gradient estimate ( Hofmann et al. , 2015 ) , and therefore stochastic gradient methods are in general slow to converge ( Johnson & Zhang , 2013 ; Defazio et al. , 2014 ) . The majority of the work speeding up IG methods has thus primarily focused on reducing the variance of the gradient estimate ( SAGA ( Defazio et al. , 2014 ) , SVRG ( Johnson & Zhang , 2013 ) , Katysha ( Allen-Zhu , 2017 ) ) or more carefully selecting the gradient stepsize ( Adagrad ( Duchi et al. , 2011 ) , Adadelta ( Zeiler , 2012 ) , Adam ( Kingma & Ba , 2014 ) ) . However , the direction that remains largely unexplored is how to carefully select a small subset S ⊆ V of the full training data V , so that the model can only be trained on the subset S while still ( approximately ) converging to the globally optimal solution ( i.e. , the model parameters that would be obtained if training/optimizing on the full V ) . If such a subset S can be quickly found , then this would directly lead to a speedup of |V |/|S| ( which can be very large if |S| |V | ) per epoch of IG . There are four main challenges in finding such a subset S. First is that a guiding principle to select S is unclear . For example , selecting training points close to the decision boundary might allow the model to fine tune the decision boundary , while picking the most diverse set of data points would allow the model to get a better sense of the training data distribution . Second is that finding S must be fast as otherwise identifying the set S may take longer than the actual optimization , and so no overall speed-up would be achieved . Third is that finding a subset S is not enough . One also has to decide on an ordering over S and a gradient stepsize for each data point in S , as they affect the convergence . And last , while the method might work well empirically on some data sets , one also requires theoretical understanding and mathematical convergence guarantees . Here we develop CoResets for Accelerating Incremental Gradient descent ( CRAIG ) , for selecting a subset of training data points to speed up training of large machine learning models . Our key idea is to directly approximate the gradient . That is , we aim to find a weighted and ordered subset S of training data V that is representative of the full gradient of V . We prove that the subset S that minimizes an upper-bound on the error of estimating the full gradient of V maximizes a submodular facility location function . As a result , the subset S can be efficiently found using a fast greedy algorithm . A further benefit of our approach is that set S is created incrementally which induces a natural ordering over data in S. Thus , rather than processing data points in a random or arbitrary order , CRAIG processes them using in the order provided by the procedure , which we show further speeds up the convergence of the method . We also provide theoretical analysis of CRAIG and prove the convergence of our approach . In particular , for a µ-strongly convex risk function and a subset S selected by CRAIG that estimates the full gradient by an error of at most , we prove that IG method with diminishing stepsize αk = α/kτ at epoch k ( with 0 < τ < 1 and 0 < α ) , converges to an 2R /µ neighborhood of the optimal solution at rate O ( 1/ √ k ) . Here , R = min { d0 , ( rγmaxC + ) /µ } where d0 is the initial distance to the optimum , C is an upper-bound on the norm of the gradients , r = |S| , and γmax is the largest weight for the elements in the subset obtained by CRAIG . Moreover , we prove that if in addition to the strong convexity , component functions have smooth gradients , IG with the same diminishing stepsize on subset S converges to a 2 /µ neighborhood of the optimum solution at rate O ( 1/kτ ) . The above convergence rates are the same as convergence rate of IG on V for a strongly convex risk ( and smooth component functions ) , and therefore IG on S converges in the same number epochs as IG on the full V . But because every epoch only uses a subset S of the data , it requires fewer gradient computations and thus leads to a |V |/|S| speedup over traditional IG methods , while still ( approximately ) converging to the optimal solution . Furthermore , CRAIG only requires the knowledge of estimated gradient differences and does not involve any ( exact ) gradient calculations . Therefore , CRAIG can be used as a simple preprocessing step before IG starts and no additional storage or gradient calculations are required during IG , which makes CRAIG extremely practical . As such CRAIG can be used to speed up any existing IG methods , including IG , Adam , SAGA , SVRG as we show in the experiments section . We demonstrate the effectiveness of CRAIG via an extensive set of experiments using logistic regression ( a convex optimization problem ) as well as training neural networks ( non-convex optimization problems ) . We show that CRAIG speeds up incremental gradient methods , including SGD , SAGA , SVRG , Adam , Adagrad , and NAG . In particular , CRAIG while achieving practically the same loss and accuracy as the underlying incremental gradient descent methods , speeds up gradient methods by up to 10x for convex and 3x for non-convex loss functions . We also demonstrate that the deliberate ordering scheme of the CRAIG algorithm significantly improves convergence time . 2 RELATED WORK . Convergence of IG methods has been long studied under various conditions ( Zhi-Quan & Paul , 1994 ; Mangasariany & Solodovy , 1994 ; Bertsekas , 1996 ; Solodov , 1998 ; Tseng , 1998 ) , however IG ’ s convergence rate has been characterized only more recently ( see ( Bertsekas , 2015 ) for a survey ) . In particular , Nedić & Bertsekas ( 2001 ) provides a O ( 1/ √ k ) convergence rate for diminishing stepsizes αk per epoch k under a strong convexity assumption , and Gürbüzbalaban et al . ( 2015 ) proves a O ( 1/kτ ) convergence rate with diminishing stepsizes αk = Θ ( 1/kτ ) for τ ∈ ( 0 , 1 ] under an additional smoothness assumption for the components . While these works provide convergence on the full dataset , our analysis provides the same convergence rates on subsets obtained by CRAIG . It has been empirically observed that ordering of data significantly affects the convergence rate of IG . However , finding a favorable ordering for IG has been a long standing open question . Among the few results are that of ( Recht & Re , 2012 ) showing that without-replacement random sampling improves convergence of IG for least means squares problem , and the very recent result of ( Gurbuzbalaban et al. , 2017 ) showing that a Random Reshuffling ( RR ) method with iterate averaging and a diminishing stepsize Θ ( 1/kτ ) for τ ∈ ( 1/2,1 ) converges at rate Θ ( 1/k2τ ) with probability one in the suboptimality of the objective value , thus improving upon the Ω ( 1/k ) rate of SGD . Contrary to the above randomized analysis , we propose the first deterministic ordering on the data points and empirically show that the ordering provided by CRAIG provides a significant speedup for the convergence of IG . Techniques for speeding up SGD , are mostly focused on variance reduction techniques ( Roux et al. , 2012 ; Shalev-Shwartz & Zhang , 2013 ; Johnson & Zhang , 2013 ; Hofmann et al. , 2015 ; Allen-Zhu et al. , 2016 ) , and accelerated gradient methods when the regularization parameter is small ( Frostig et al. , 2015 ; Lin et al. , 2015 ; Xiao & Zhang , 2014 ) . Very recently , Hofmann et al . ( 2015 ) ; Allen-Zhu et al . ( 2016 ) exploited neighborhood structure to further reduce the variance of stochastic gradient descent and improve its running time . Our CRAIG method and analysis are complementary to variance reduction and accelerated methods . CRAIG can be applied to these methods as well to speed them up ( as we show in experiments ) . 3 CORESETS FOR INCREMENTAL GRADIENT DESCENT ( CRAIG ) . We proceed as follows : First , we define an objective function L for selecting an optimal set S of size r that best approximates the gradient of the full training dataset V of size n. Then , we show that L can be turned into a submodular function F and thus S can be efficiently found using a greedy algorithm . Crucially , we also show that the approximation error between the estimated and the true gradient can be efficiently minimized in a way that is independent of the actual optimization procedure and thus CRAIG can simply be used as a preprocessing step before the actual optimization starts . Incremental gradient methods aim at estimating the full gradient ∑ i∈V ∇fi ( x ) over V by iteratively making a step based on the gradient of every function fi . Our key idea in CRAIG is that if we can find a small subset S such that the weighted sum of the gradients of its elements closely approximates the full gradient over V , we can apply IG only to the set S ( with stepsizes equal to the weight of the elements in S ) , and we should still converge to the ( approximately ) optimal solution , but much faster . Specifically , our goal in CRAIG is to find the smallest subset S ⊆ V and corresponding per-element stepsizes γj > 0 that approximate the full gradient with an error at most > 0 for all the possible values of the optimization parameters x ∈ X .1 S∗ = argmin S⊆V , γj≥0 ∀j |S| , s.t . max x∈X ‖ ∑ i∈V ∇fi ( x ) − ∑ j∈S γj∇fj ( x ) ‖ ≤ . ( 2 ) Given such an S∗ and associated weights { γ } j , we are guaranteed that gradient updates on S∗ will be similar to the gradient updates on V regardless of the value of x . 1Note that in the worst case we may need |S∗| ≈ |V | to approximate the gradient . However , as we show in experiments , in practice we find that a small subset is sufficient to accurately approximate the gradient . Unfortunately , directly solving the above optimization problem is not feasible , due to two problems . Problem 1 : Eq . ( 2 ) requires us to calculate the gradient of all the functions fi over the entire space X , which is too expensive and would not lead to overall speedup . In other words , it would appear that solving for S∗ is as difficult as solving problem ( 1 ) , as it involves calculating ∑ i∈V ∇fi ( x ) for various x ∈ X . And Problem 2 : even if calculating the normed difference between the gradients in Eq . ( 2 ) would be fast , as we discuss later finding the optimal subset S∗ in NP-hard . In the following , we address the above two challenges and discuss how we can quickly find a near-optimal subset S .

Summary: This paper proposes a novel extension to SGD/incremental gradient methods called CRAIG. The algorithm selects a subset of datapoints to approximate the training loss at the beginning of each epoch in order to reduce the total amount of time necessary to solve the empirical risk minimization problem. In particular, the algorithm formulates a submodular optimization problem based on the intuition that the gradient of the problem on the selected subset approximates the gradient of the true training loss up to some tolerance. Each datapoint in the subset is a medoid and assigned a weight corresponding to the number of datapoints in the full set that are assigned to that particular datapoint. A greedy algorithm is employed to approximately solve the subproblem. Theory is proven based on based on an incremental subgradient method with errors. Experiments demonstrate significant savings in time for training both logistic regression and small neural networks.