Training Neural Networks for and by Interpolation

1 INTRODUCTION . Training a deep neural network is a challenging optimization problem : it involves minimizing the average of many high-dimensional non-convex functions . In practice , the main algorithms of choice are Stochastic Gradient Descent ( SGD ) ( Robbins & Monro , 1951 ) and adaptive gradient methods such as AdaGrad ( Duchi et al. , 2011 ) or Adam ( Kingma & Ba , 2015 ) . In recent work , the ability to interpolate – i.e . to achieve near zero loss on all training samples simultaneously – has been used to show convergence of SGD ( Ma et al. , 2018 , Vaswani et al. , 2019a , Zhou et al. , 2019 ) . This property is usually satisfied in supervised deep learning because of the empirical success of over-parameterized architectures . However , while the convergence analyses provide a better theoretical understanding of SGD , they do not help improve its practical behavior . In this work , we open a different line of enquiry , namely : can the interpolation property be used to design a robust and efficient optimization algorithm for deep learning ? In order to answer this question , we begin by giving the following two desiderata of an optimization algorithm for deep learning : ( i ) an inexpensive computational cost per iteration ( typically a call to a stochastic first-order oracle ) ; and ( ii ) adaptive learning-rates that do not require a manually designed schedule . We present ALI-G ( Adaptive Learning-rates for Interpolation with Gradients ) , an algorithm that takes advantage of interpolation by design and satisfies both properties mentioned above . Key to the ALI-G algorithm are the following two ideas . First , an adaptive learning-rate can be computed for the non-stochastic gradient direction when the minimum value of the objective function is known ( Polyak , 1969 , Shor , 1985 , Brännlund , 1995 , Nedić & Bertsekas , 2001a ; b ) . And second , one such minimum value is usually approximately known for interpolating models : for instance , it is close to zero for a model trained with the cross-entropy loss . By carefully combining these two ideas , we create a stochastic algorithm that provably converges fast in the convex setting and that obtains state-of-the-art results with neural networks . Procedurally , ALI-G is close to many existing algorithms , such as Deep Frank-Wolfe ( Berrada et al. , 2019 ) , APROX ( Asi & Duchi , 2019 ) and L4 ( Rolinek & Martius , 2018 ) . And yet uniquely , thanks to its careful design and analysis , ALI-G enables accurate optimization of a wide class of deep neural networks using only a single hyper-parameter that does not need to be decayed . This makes ALI-G well-suited to the deep learning setting , where hyper-parameter tuning is widely regarded as an onerous and time consuming task . Since ALI-G is easy to implement in any deep learning framework , we believe that it can prove to be a practical and reliable optimization tool for deep learning . Contributions . We summarize the contributions of this work as follows : - We design an optimization algorithm that uses a single hyper-parameter for its learning-rate and does need any decaying schedule . In contrast , the closely related APROX ( Asi & Duchi , 2019 ) and L4 ( Rolinek & Martius , 2018 ) use respectively two and four hyper-parameters for their learning-rate . - We provide convergence rates of ALI-G in various stochastic convex settings . Importantly , our theoretical results take into account the error in the estimate of the minimum objective value . To the best of our knowledge , our work is the first to establish convergence rates for interpolation with approximate estimates . - We demonstrate state-of-the-art results for ALI-G on learning a differentiable neural computer ; training variants of residual networks on the SVHN and CIFAR data sets ; and training a Bi-LSTM on the Stanford Natural Language Inference data set . 2 THE ALGORITHM . 2.1 PROBLEM SETTING . Loss Function . We consider a supervised learning task where the model is parameterized by w ∈ Rp . Usually , the objective function can be expressed as an expectation over z ∈ Z , a random variable indexing the samples of the training set : f ( w ) , Ez∈Z [ ` z ( w ) ] , ( 1 ) where each ` z is the loss function associated with the sample z . We assume that each ` z is nonnegative , which is the case for the large majority of loss functions used in machine learning . For instance , suppose that the model is a deep neural network with weights w performing classification . Then for each sample z , ` z ( w ) can represent the cross-entropy loss , which is always non-negative . Other non-negative loss functions include the structured or multi-class hinge loss , and the L1 or L2 loss functions for regression . Regularization . It is often desirable to employ a regularization function φ in order to promote generalization . In this work , we incorporate such regularization as a constraint on the feasible domain : Ω = { w ∈ Rp : φ ( w ) ≤ r } for some value of r. In the deep learning setting , this will allow us to assume that the objective function can be driven close to zero without unrealistic assumptions about the regularization . Our framework can handle any constraint set Ω on which Euclidean projections are computationally efficient . This includes the feasible set induced by L2 regularization : Ω = { w ∈ Rp : ‖w‖22 ≤ r } , for which the projection is given by a simple rescaling of w. Finally , note that if we do not wish to use any regularization , we define Ω = Rp and the corresponding projection is the identity . Problem Formulation . The learning task can be expressed as the problem ( P ) of finding a feasible vector of parameters w ? ∈ Ω that minimizes f : w ? ∈ argmin w∈Ω f ( w ) . ( P ) Also note that f ? refers to the minimum value of f over Ω : f ? , minw∈Ω f ( w ) . 2.2 THE POLYAK STEP-SIZE . Before outlining the ALI-G algorithm , we begin with a brief description of the Polyak step-size , from which ALI-G draws some fundamental ideas . Setting . We assume that f ? is known and we use non-stochastic updates : at each iteration , the full objective f and its derivative are evaluated . We denote by ∇f ( w ) the first-order derivative of f at w ( e.g . ∇f ( w ) can be a sub-gradient or the gradient ) . In addition , ‖ · ‖ is the standard Euclidean norm in Rp , and ΠΩ ( w ) is the Euclidean projection of the vector w ∈ Rp on the set Ω. Polyak Step-Size . At time-step t , using the Polyak step-size ( Polyak , 1969 , Shor , 1985 , Brännlund , 1995 , Nedić & Bertsekas , 2001a ; b ) yields the following update : wt+1 = ΠΩ ( wt − γt∇f ( wt ) ) , where γt , f ( wt ) − f ? ‖∇f ( wt ) ‖2 , ( 2 ) where we loosely define 00 = 0 for simplicity purposes . Interpretation . It can be shown that wt+1 lies on the intersection between the linearization of f at wt and the horizontal plane z = f ? ( see Figure 1 , more details in Proposition 2 in the supplementary material ) . Note that since f ? is the minimum of f , the Polyak step-size γt is necessarily non-negative . Limitations . Equation ( 2 ) has two major shortcomings that prevent its applicability in a machine learning setting . First , each update requires a full evaluation of f and its derivative . Stochastic extensions have been proposed in Nedić & Bertsekas ( 2001a ; b ) , but they still require frequent evaluations of f . This is expensive in the large data setting , and even computationally infeasible when using massive data augmen- tation . Second , when applying this method to the non-convex setting of deep neural networks , the method sometimes fails to converge . Therefore we would like to design an extension of the Polyak step-size that ( i ) is inexpensive to compute in a stochastic setting ( e.g . with a computational cost that is independent of the total number of training samples ) , and ( ii ) converges in practice when used with deep neural networks . The next section introduces the ALI-G algorithm , which achieves these two goals in the interpolation setting . 2.3 THE ALI-G ALGORITHM . We now present the ALI-G algorithm . For this , we suppose that we are in an interpolation setting : the model is assumed to be able to drive the loss function to near zero on all samples simultaneously . Algorithm . The main steps of the ALI-G algorithm are provided in Algorithm 1 . ALI-G iterates over three operations until convergence . First , it computes a stochastic approximation of the learning objective and its derivative ( line 3 ) . Second , it computes a step-size decay parameter γt based on the stochastic information ( line 4 ) . Third , it updates the parameters by moving in the negative derivative direction by an amount specified by the step-size and projecting the resulting vector on to the feasible region ( line 5 ) . Algorithm 1 The ALI-G algorithm Require : maximal learning-rate η , initial feasible w0 ∈ Ω , small constant δ > 0 1 : t = 0 2 : while not converged do 3 : Get ` zt ( wt ) , ∇ ` zt ( wt ) with zt drawn i.i.d . 4 : γt = min { ` zt ( wt ) ‖∇ ` zt ( wt ) ‖2+δ , η } 5 : wt+1 = ΠΩ ( wt − γt∇ ` zt ( wt ) ) 6 : t = t+ 1 7 : end while Comparison with the Polyak Step-Size . There are three main differences to the update in equation ( 2 ) . First , each update only uses the loss ` zt and its derivative rather than the full objective f and its derivative . Second , the learning-rate γt is clipped to η , the maximal learning-rate hyper-parameter . We emphasize that η remains constant throughout the iterations , therefore it is a single hyper-parameter and does not need a schedule like SGD learning-rate . Third , the minimum f ? has been replaced by the lower-bound of 0 . All these modifications will be justified in the next section . The ALI-G∞ Variant . When ALI-G uses no maximal learning-rate , we refer to the algorithm as ALI-G∞ , since it is equivalent to use an infinite maximal learning-rate . Note that ALI-G∞ requires no hyper-parameter for its step-size . 3 JUSTIFICATION AND ANALYSIS . 3.1 INTERPOLATION ENABLES INEXPENSIVE STOCHASTIC UPDATES . By definition , the interpolation setting gives f ? = 0 , which we used in ALI-G to simplify the formula of the learning-rate γt . More subtly , the interpolation property also allows the updates to rely on the stochastic estimate ` zt ( wt ) rather than the exact but expensive f ( wt ) . Intuitively , this is possible because in the interpolation setting , each training sample can use its own learning-rate without harming progress on the other ones . Recall that ALI-G∞ is the variant of ALI-G that uses no maximal learning-rate . The following result formalizes the convergence guarantee of ALI-G∞ in the stochastic convex setting : Theorem 1 ( Convex and Lipschitz ) . We assume that Ω is a convex set , and that for every z ∈ Z , ` z is convex and C-Lipschitz . Let w ? be a solution of ( P ) , and assume that the interpolation property is approximately satisfied : ∀z ∈ Z , ` z ( w ? ) ≤ ε , for some interpolation tolerance ε ≥ 0 . Then ALI-G∞ applied to f satisfies : f ( 1 T + 1 T∑ t=0 wt ) ≤ ε √ ( C2 δ + 1 ) + ‖w0 −w ? ‖ √ C2 + δ√ T + 1 . ( 3 ) In other words , by assuming interpolation , ALI-G provably converges while requiring only ` zt ( wt ) and ∇ ` zt ( wt ) ( stochastic estimation per sample ) to compute its learning-rate . In contrast , the Polyak step-size , which does not exploit interpolation , would require f ( wt ) and∇f ( wt ) to compute the learning-rate ( deterministic computation over all training samples ) . This constitutes a major computational advantage of ALI-G over the usual Polyak step-size . We emphasize that in Theorem 1 , our careful analysis explicitly shows the dependency of the convergence result on the interpolation tolerance ε . It is reassuring to note that convergence is exact when the interpolation property is exactly satisfied ( ε = 0 ) . In the supplementary material , we also establish convergence rates of O ( 1/T ) for smooth convex functions , and O ( exp ( −αT/8β ) ) for α-strongly convex and β-smooth functions . Similar results can be proved when using a maximal learning-rate η : the convergence speed then remains unchanged provided that η is large enough , and it is lowered when η is small . We refer the interested reader to the supplementary for the formal results and their proofs .

Summary: This paper proposes a new adaptive learning rate method which is tailored to the optimization of deep neural networks. The motivating observation is that over-parameterized DNNs are able to interpolate the training data (i.e. they are able to reach near-zero training error). This enables application of the Polyak update rule to stochastic updates and a simplification by assuming a zero minimal training loss. A number of proofs for convergence in various convex settings are provided, and empirical evaluation on several benchmarks demonstrates (a) ability to optimize complex architectures, (b) performance improvements over, and (c) performance close to manually tuned SGD learning rates.