Coresets for Accelerating Incremental Gradient Methods

1 INTRODUCTION . Mathematical optimization lies at the core of training large-scale machine learning systems , and is now widely used over massive data sets with great practical success , assuming sufficient data resources are available . Achieving this success , however , also requires large amounts of ( often GPU ) computing , as well as concomitant financial expenditures and energy usage ( Strubell et al. , 2019 ) . Significantly decreasing these costs without decreasing the learnt system ’ s resulting accuracy is one of the grand challenges of machine learning and artificial intelligence today ( Asi & Duchi , 2019 ) . Training machine learning models often reduces to the problem of optimizing a regularized empirical risk function . Given a convex loss l , and a µ-strongly convex regularizer r , one aims to find model parameter vector x∗ over the parameter space X that minimizes the loss f over the training data V : x∗ ∈ argmin x∈X f ( x ) , f ( x ) : = ∑ i∈V fi ( x ) + r ( x ) , fi ( x ) = l ( x , ( ai , yi ) ) , ( 1 ) where V = { 1 , . . . , n } is an index set of the training data , and functions fi : Rd → R are associated with training examples ( ai , yi ) , where ai ∈ Rd is the feature vector , and yi is the point i ’ s label . The standard Gradient Descent can find the minimizer of this problem , but requires repeated computations of the full gradient ∇f ( x ) —sum of the gradients over all training data points/functions i—and is therefore prohibitive for massive data sets . This issue is further exacerbated in case of deep neural networks where gradient computations ( backpropagation ) are expensive . Incremental Gradient ( IG ) methods , such as Stochastic Gradient Descent and its accelerated variants , including SGD with momentum ( Qian , 1999 ) , Adagrad ( Duchi et al. , 2011 ) , Adam ( Kingma & Ba , 2014 ) , SAGA ( Defazio et al. , 2014 ) , and SVRG ( Johnson & Zhang , 2013 ) iteratively estimate the gradient on random subsets/batches of training data . While this provides an unbiased estimate of the full gradient , the randomized batches introduce variance in the gradient estimate ( Hofmann et al. , 2015 ) , and therefore stochastic gradient methods are in general slow to converge ( Johnson & Zhang , 2013 ; Defazio et al. , 2014 ) . The majority of the work speeding up IG methods has thus primarily focused on reducing the variance of the gradient estimate ( SAGA ( Defazio et al. , 2014 ) , SVRG ( Johnson & Zhang , 2013 ) , Katysha ( Allen-Zhu , 2017 ) ) or more carefully selecting the gradient stepsize ( Adagrad ( Duchi et al. , 2011 ) , Adadelta ( Zeiler , 2012 ) , Adam ( Kingma & Ba , 2014 ) ) . However , the direction that remains largely unexplored is how to carefully select a small subset S ⊆ V of the full training data V , so that the model can only be trained on the subset S while still ( approximately ) converging to the globally optimal solution ( i.e. , the model parameters that would be obtained if training/optimizing on the full V ) . If such a subset S can be quickly found , then this would directly lead to a speedup of |V |/|S| ( which can be very large if |S| |V | ) per epoch of IG . There are four main challenges in finding such a subset S. First is that a guiding principle to select S is unclear . For example , selecting training points close to the decision boundary might allow the model to fine tune the decision boundary , while picking the most diverse set of data points would allow the model to get a better sense of the training data distribution . Second is that finding S must be fast as otherwise identifying the set S may take longer than the actual optimization , and so no overall speed-up would be achieved . Third is that finding a subset S is not enough . One also has to decide on an ordering over S and a gradient stepsize for each data point in S , as they affect the convergence . And last , while the method might work well empirically on some data sets , one also requires theoretical understanding and mathematical convergence guarantees . Here we develop CoResets for Accelerating Incremental Gradient descent ( CRAIG ) , for selecting a subset of training data points to speed up training of large machine learning models . Our key idea is to directly approximate the gradient . That is , we aim to find a weighted and ordered subset S of training data V that is representative of the full gradient of V . We prove that the subset S that minimizes an upper-bound on the error of estimating the full gradient of V maximizes a submodular facility location function . As a result , the subset S can be efficiently found using a fast greedy algorithm . A further benefit of our approach is that set S is created incrementally which induces a natural ordering over data in S. Thus , rather than processing data points in a random or arbitrary order , CRAIG processes them using in the order provided by the procedure , which we show further speeds up the convergence of the method . We also provide theoretical analysis of CRAIG and prove the convergence of our approach . In particular , for a µ-strongly convex risk function and a subset S selected by CRAIG that estimates the full gradient by an error of at most , we prove that IG method with diminishing stepsize αk = α/kτ at epoch k ( with 0 < τ < 1 and 0 < α ) , converges to an 2R /µ neighborhood of the optimal solution at rate O ( 1/ √ k ) . Here , R = min { d0 , ( rγmaxC + ) /µ } where d0 is the initial distance to the optimum , C is an upper-bound on the norm of the gradients , r = |S| , and γmax is the largest weight for the elements in the subset obtained by CRAIG . Moreover , we prove that if in addition to the strong convexity , component functions have smooth gradients , IG with the same diminishing stepsize on subset S converges to a 2 /µ neighborhood of the optimum solution at rate O ( 1/kτ ) . The above convergence rates are the same as convergence rate of IG on V for a strongly convex risk ( and smooth component functions ) , and therefore IG on S converges in the same number epochs as IG on the full V . But because every epoch only uses a subset S of the data , it requires fewer gradient computations and thus leads to a |V |/|S| speedup over traditional IG methods , while still ( approximately ) converging to the optimal solution . Furthermore , CRAIG only requires the knowledge of estimated gradient differences and does not involve any ( exact ) gradient calculations . Therefore , CRAIG can be used as a simple preprocessing step before IG starts and no additional storage or gradient calculations are required during IG , which makes CRAIG extremely practical . As such CRAIG can be used to speed up any existing IG methods , including IG , Adam , SAGA , SVRG as we show in the experiments section . We demonstrate the effectiveness of CRAIG via an extensive set of experiments using logistic regression ( a convex optimization problem ) as well as training neural networks ( non-convex optimization problems ) . We show that CRAIG speeds up incremental gradient methods , including SGD , SAGA , SVRG , Adam , Adagrad , and NAG . In particular , CRAIG while achieving practically the same loss and accuracy as the underlying incremental gradient descent methods , speeds up gradient methods by up to 10x for convex and 3x for non-convex loss functions . We also demonstrate that the deliberate ordering scheme of the CRAIG algorithm significantly improves convergence time . 2 RELATED WORK . Convergence of IG methods has been long studied under various conditions ( Zhi-Quan & Paul , 1994 ; Mangasariany & Solodovy , 1994 ; Bertsekas , 1996 ; Solodov , 1998 ; Tseng , 1998 ) , however IG ’ s convergence rate has been characterized only more recently ( see ( Bertsekas , 2015 ) for a survey ) . In particular , Nedić & Bertsekas ( 2001 ) provides a O ( 1/ √ k ) convergence rate for diminishing stepsizes αk per epoch k under a strong convexity assumption , and Gürbüzbalaban et al . ( 2015 ) proves a O ( 1/kτ ) convergence rate with diminishing stepsizes αk = Θ ( 1/kτ ) for τ ∈ ( 0 , 1 ] under an additional smoothness assumption for the components . While these works provide convergence on the full dataset , our analysis provides the same convergence rates on subsets obtained by CRAIG . It has been empirically observed that ordering of data significantly affects the convergence rate of IG . However , finding a favorable ordering for IG has been a long standing open question . Among the few results are that of ( Recht & Re , 2012 ) showing that without-replacement random sampling improves convergence of IG for least means squares problem , and the very recent result of ( Gurbuzbalaban et al. , 2017 ) showing that a Random Reshuffling ( RR ) method with iterate averaging and a diminishing stepsize Θ ( 1/kτ ) for τ ∈ ( 1/2,1 ) converges at rate Θ ( 1/k2τ ) with probability one in the suboptimality of the objective value , thus improving upon the Ω ( 1/k ) rate of SGD . Contrary to the above randomized analysis , we propose the first deterministic ordering on the data points and empirically show that the ordering provided by CRAIG provides a significant speedup for the convergence of IG . Techniques for speeding up SGD , are mostly focused on variance reduction techniques ( Roux et al. , 2012 ; Shalev-Shwartz & Zhang , 2013 ; Johnson & Zhang , 2013 ; Hofmann et al. , 2015 ; Allen-Zhu et al. , 2016 ) , and accelerated gradient methods when the regularization parameter is small ( Frostig et al. , 2015 ; Lin et al. , 2015 ; Xiao & Zhang , 2014 ) . Very recently , Hofmann et al . ( 2015 ) ; Allen-Zhu et al . ( 2016 ) exploited neighborhood structure to further reduce the variance of stochastic gradient descent and improve its running time . Our CRAIG method and analysis are complementary to variance reduction and accelerated methods . CRAIG can be applied to these methods as well to speed them up ( as we show in experiments ) . 3 CORESETS FOR INCREMENTAL GRADIENT DESCENT ( CRAIG ) . We proceed as follows : First , we define an objective function L for selecting an optimal set S of size r that best approximates the gradient of the full training dataset V of size n. Then , we show that L can be turned into a submodular function F and thus S can be efficiently found using a greedy algorithm . Crucially , we also show that the approximation error between the estimated and the true gradient can be efficiently minimized in a way that is independent of the actual optimization procedure and thus CRAIG can simply be used as a preprocessing step before the actual optimization starts . Incremental gradient methods aim at estimating the full gradient ∑ i∈V ∇fi ( x ) over V by iteratively making a step based on the gradient of every function fi . Our key idea in CRAIG is that if we can find a small subset S such that the weighted sum of the gradients of its elements closely approximates the full gradient over V , we can apply IG only to the set S ( with stepsizes equal to the weight of the elements in S ) , and we should still converge to the ( approximately ) optimal solution , but much faster . Specifically , our goal in CRAIG is to find the smallest subset S ⊆ V and corresponding per-element stepsizes γj > 0 that approximate the full gradient with an error at most > 0 for all the possible values of the optimization parameters x ∈ X .1 S∗ = argmin S⊆V , γj≥0 ∀j |S| , s.t . max x∈X ‖ ∑ i∈V ∇fi ( x ) − ∑ j∈S γj∇fj ( x ) ‖ ≤ . ( 2 ) Given such an S∗ and associated weights { γ } j , we are guaranteed that gradient updates on S∗ will be similar to the gradient updates on V regardless of the value of x . 1Note that in the worst case we may need |S∗| ≈ |V | to approximate the gradient . However , as we show in experiments , in practice we find that a small subset is sufficient to accurately approximate the gradient . Unfortunately , directly solving the above optimization problem is not feasible , due to two problems . Problem 1 : Eq . ( 2 ) requires us to calculate the gradient of all the functions fi over the entire space X , which is too expensive and would not lead to overall speedup . In other words , it would appear that solving for S∗ is as difficult as solving problem ( 1 ) , as it involves calculating ∑ i∈V ∇fi ( x ) for various x ∈ X . And Problem 2 : even if calculating the normed difference between the gradients in Eq . ( 2 ) would be fast , as we discuss later finding the optimal subset S∗ in NP-hard . In the following , we address the above two challenges and discuss how we can quickly find a near-optimal subset S .

Summary: The paper proposes a theoretically founded method to generate subsets of a dataset, together with corresponding sample weights in a way that the average gradient of the subset is at most epsilon far from the average gradient of the full dataset. Given such a subset, the authors provide theoretical guarantees for convergence to an epsilon neighborhood of the optimum for strongly convex functions. The proposed algorithm to create such a subset is a greedy algorithm that relies on parameter independent similarities between samples, namely similarity scores that are true regardless of the current value of the function parameters.