Training Neural Networks for and by Interpolation

1 INTRODUCTION . Training a deep neural network is a challenging optimization problem : it involves minimizing the average of many high-dimensional non-convex functions . In practice , the main algorithms of choice are Stochastic Gradient Descent ( SGD ) ( Robbins & Monro , 1951 ) and adaptive gradient methods such as AdaGrad ( Duchi et al. , 2011 ) or Adam ( Kingma & Ba , 2015 ) . In recent work , the ability to interpolate – i.e . to achieve near zero loss on all training samples simultaneously – has been used to show convergence of SGD ( Ma et al. , 2018 , Vaswani et al. , 2019a , Zhou et al. , 2019 ) . This property is usually satisfied in supervised deep learning because of the empirical success of over-parameterized architectures . However , while the convergence analyses provide a better theoretical understanding of SGD , they do not help improve its practical behavior . In this work , we open a different line of enquiry , namely : can the interpolation property be used to design a robust and efficient optimization algorithm for deep learning ? In order to answer this question , we begin by giving the following two desiderata of an optimization algorithm for deep learning : ( i ) an inexpensive computational cost per iteration ( typically a call to a stochastic first-order oracle ) ; and ( ii ) adaptive learning-rates that do not require a manually designed schedule . We present ALI-G ( Adaptive Learning-rates for Interpolation with Gradients ) , an algorithm that takes advantage of interpolation by design and satisfies both properties mentioned above . Key to the ALI-G algorithm are the following two ideas . First , an adaptive learning-rate can be computed for the non-stochastic gradient direction when the minimum value of the objective function is known ( Polyak , 1969 , Shor , 1985 , Brännlund , 1995 , Nedić & Bertsekas , 2001a ; b ) . And second , one such minimum value is usually approximately known for interpolating models : for instance , it is close to zero for a model trained with the cross-entropy loss . By carefully combining these two ideas , we create a stochastic algorithm that provably converges fast in the convex setting and that obtains state-of-the-art results with neural networks . Procedurally , ALI-G is close to many existing algorithms , such as Deep Frank-Wolfe ( Berrada et al. , 2019 ) , APROX ( Asi & Duchi , 2019 ) and L4 ( Rolinek & Martius , 2018 ) . And yet uniquely , thanks to its careful design and analysis , ALI-G enables accurate optimization of a wide class of deep neural networks using only a single hyper-parameter that does not need to be decayed . This makes ALI-G well-suited to the deep learning setting , where hyper-parameter tuning is widely regarded as an onerous and time consuming task . Since ALI-G is easy to implement in any deep learning framework , we believe that it can prove to be a practical and reliable optimization tool for deep learning . Contributions . We summarize the contributions of this work as follows : - We design an optimization algorithm that uses a single hyper-parameter for its learning-rate and does need any decaying schedule . In contrast , the closely related APROX ( Asi & Duchi , 2019 ) and L4 ( Rolinek & Martius , 2018 ) use respectively two and four hyper-parameters for their learning-rate . - We provide convergence rates of ALI-G in various stochastic convex settings . Importantly , our theoretical results take into account the error in the estimate of the minimum objective value . To the best of our knowledge , our work is the first to establish convergence rates for interpolation with approximate estimates . - We demonstrate state-of-the-art results for ALI-G on learning a differentiable neural computer ; training variants of residual networks on the SVHN and CIFAR data sets ; and training a Bi-LSTM on the Stanford Natural Language Inference data set . 2 THE ALGORITHM . 2.1 PROBLEM SETTING . Loss Function . We consider a supervised learning task where the model is parameterized by w ∈ Rp . Usually , the objective function can be expressed as an expectation over z ∈ Z , a random variable indexing the samples of the training set : f ( w ) , Ez∈Z [ ` z ( w ) ] , ( 1 ) where each ` z is the loss function associated with the sample z . We assume that each ` z is nonnegative , which is the case for the large majority of loss functions used in machine learning . For instance , suppose that the model is a deep neural network with weights w performing classification . Then for each sample z , ` z ( w ) can represent the cross-entropy loss , which is always non-negative . Other non-negative loss functions include the structured or multi-class hinge loss , and the L1 or L2 loss functions for regression . Regularization . It is often desirable to employ a regularization function φ in order to promote generalization . In this work , we incorporate such regularization as a constraint on the feasible domain : Ω = { w ∈ Rp : φ ( w ) ≤ r } for some value of r. In the deep learning setting , this will allow us to assume that the objective function can be driven close to zero without unrealistic assumptions about the regularization . Our framework can handle any constraint set Ω on which Euclidean projections are computationally efficient . This includes the feasible set induced by L2 regularization : Ω = { w ∈ Rp : ‖w‖22 ≤ r } , for which the projection is given by a simple rescaling of w. Finally , note that if we do not wish to use any regularization , we define Ω = Rp and the corresponding projection is the identity . Problem Formulation . The learning task can be expressed as the problem ( P ) of finding a feasible vector of parameters w ? ∈ Ω that minimizes f : w ? ∈ argmin w∈Ω f ( w ) . ( P ) Also note that f ? refers to the minimum value of f over Ω : f ? , minw∈Ω f ( w ) . 2.2 THE POLYAK STEP-SIZE . Before outlining the ALI-G algorithm , we begin with a brief description of the Polyak step-size , from which ALI-G draws some fundamental ideas . Setting . We assume that f ? is known and we use non-stochastic updates : at each iteration , the full objective f and its derivative are evaluated . We denote by ∇f ( w ) the first-order derivative of f at w ( e.g . ∇f ( w ) can be a sub-gradient or the gradient ) . In addition , ‖ · ‖ is the standard Euclidean norm in Rp , and ΠΩ ( w ) is the Euclidean projection of the vector w ∈ Rp on the set Ω. Polyak Step-Size . At time-step t , using the Polyak step-size ( Polyak , 1969 , Shor , 1985 , Brännlund , 1995 , Nedić & Bertsekas , 2001a ; b ) yields the following update : wt+1 = ΠΩ ( wt − γt∇f ( wt ) ) , where γt , f ( wt ) − f ? ‖∇f ( wt ) ‖2 , ( 2 ) where we loosely define 00 = 0 for simplicity purposes . Interpretation . It can be shown that wt+1 lies on the intersection between the linearization of f at wt and the horizontal plane z = f ? ( see Figure 1 , more details in Proposition 2 in the supplementary material ) . Note that since f ? is the minimum of f , the Polyak step-size γt is necessarily non-negative . Limitations . Equation ( 2 ) has two major shortcomings that prevent its applicability in a machine learning setting . First , each update requires a full evaluation of f and its derivative . Stochastic extensions have been proposed in Nedić & Bertsekas ( 2001a ; b ) , but they still require frequent evaluations of f . This is expensive in the large data setting , and even computationally infeasible when using massive data augmen- tation . Second , when applying this method to the non-convex setting of deep neural networks , the method sometimes fails to converge . Therefore we would like to design an extension of the Polyak step-size that ( i ) is inexpensive to compute in a stochastic setting ( e.g . with a computational cost that is independent of the total number of training samples ) , and ( ii ) converges in practice when used with deep neural networks . The next section introduces the ALI-G algorithm , which achieves these two goals in the interpolation setting . 2.3 THE ALI-G ALGORITHM . We now present the ALI-G algorithm . For this , we suppose that we are in an interpolation setting : the model is assumed to be able to drive the loss function to near zero on all samples simultaneously . Algorithm . The main steps of the ALI-G algorithm are provided in Algorithm 1 . ALI-G iterates over three operations until convergence . First , it computes a stochastic approximation of the learning objective and its derivative ( line 3 ) . Second , it computes a step-size decay parameter γt based on the stochastic information ( line 4 ) . Third , it updates the parameters by moving in the negative derivative direction by an amount specified by the step-size and projecting the resulting vector on to the feasible region ( line 5 ) . Algorithm 1 The ALI-G algorithm Require : maximal learning-rate η , initial feasible w0 ∈ Ω , small constant δ > 0 1 : t = 0 2 : while not converged do 3 : Get ` zt ( wt ) , ∇ ` zt ( wt ) with zt drawn i.i.d . 4 : γt = min { ` zt ( wt ) ‖∇ ` zt ( wt ) ‖2+δ , η } 5 : wt+1 = ΠΩ ( wt − γt∇ ` zt ( wt ) ) 6 : t = t+ 1 7 : end while Comparison with the Polyak Step-Size . There are three main differences to the update in equation ( 2 ) . First , each update only uses the loss ` zt and its derivative rather than the full objective f and its derivative . Second , the learning-rate γt is clipped to η , the maximal learning-rate hyper-parameter . We emphasize that η remains constant throughout the iterations , therefore it is a single hyper-parameter and does not need a schedule like SGD learning-rate . Third , the minimum f ? has been replaced by the lower-bound of 0 . All these modifications will be justified in the next section . The ALI-G∞ Variant . When ALI-G uses no maximal learning-rate , we refer to the algorithm as ALI-G∞ , since it is equivalent to use an infinite maximal learning-rate . Note that ALI-G∞ requires no hyper-parameter for its step-size . 3 JUSTIFICATION AND ANALYSIS . 3.1 INTERPOLATION ENABLES INEXPENSIVE STOCHASTIC UPDATES . By definition , the interpolation setting gives f ? = 0 , which we used in ALI-G to simplify the formula of the learning-rate γt . More subtly , the interpolation property also allows the updates to rely on the stochastic estimate ` zt ( wt ) rather than the exact but expensive f ( wt ) . Intuitively , this is possible because in the interpolation setting , each training sample can use its own learning-rate without harming progress on the other ones . Recall that ALI-G∞ is the variant of ALI-G that uses no maximal learning-rate . The following result formalizes the convergence guarantee of ALI-G∞ in the stochastic convex setting : Theorem 1 ( Convex and Lipschitz ) . We assume that Ω is a convex set , and that for every z ∈ Z , ` z is convex and C-Lipschitz . Let w ? be a solution of ( P ) , and assume that the interpolation property is approximately satisfied : ∀z ∈ Z , ` z ( w ? ) ≤ ε , for some interpolation tolerance ε ≥ 0 . Then ALI-G∞ applied to f satisfies : f ( 1 T + 1 T∑ t=0 wt ) ≤ ε √ ( C2 δ + 1 ) + ‖w0 −w ? ‖ √ C2 + δ√ T + 1 . ( 3 ) In other words , by assuming interpolation , ALI-G provably converges while requiring only ` zt ( wt ) and ∇ ` zt ( wt ) ( stochastic estimation per sample ) to compute its learning-rate . In contrast , the Polyak step-size , which does not exploit interpolation , would require f ( wt ) and∇f ( wt ) to compute the learning-rate ( deterministic computation over all training samples ) . This constitutes a major computational advantage of ALI-G over the usual Polyak step-size . We emphasize that in Theorem 1 , our careful analysis explicitly shows the dependency of the convergence result on the interpolation tolerance ε . It is reassuring to note that convergence is exact when the interpolation property is exactly satisfied ( ε = 0 ) . In the supplementary material , we also establish convergence rates of O ( 1/T ) for smooth convex functions , and O ( exp ( −αT/8β ) ) for α-strongly convex and β-smooth functions . Similar results can be proved when using a maximal learning-rate η : the convergence speed then remains unchanged provided that η is large enough , and it is lowered when η is small . We refer the interested reader to the supplementary for the formal results and their proofs .

Summary: This paper proposes a new gradient descent methods for training deep neural network which can take the adaptive step size with only one hyper-parameter to tune -- the maximum learning rate -- and achieve comparable results to stochastic gradient descent (SGD) on various tasks and models. In order to achieve that, they develop a stochastic extension of the Polyak step-size for the non-convex setting, namely the adaptive learning-rates for interpolation with gradients (ALI-G), in which the minimal value of the objective loss is set to 0 due to interpolation in neural networks and the learning rates are clipped by a chosen maximal value. The problem is formulated clearly, and the review on the Polyak step-size and related works are well done. Another main contribution of the paper is to provide the convergence guarantees for ALI-G in the convex setting where the objective loss is Lipschitz-continuous (Theorem 1 in the paper). Their theorem also takes into account the error in the estimate of the minimal value of the objective loss. In addition, they derive the connections between  ALI-G and SGD and show that compared to SGD, ALI-G take into consideration that the objective loss is non-negative and set the loss to 0 when it is negative. They perform empirical study to compare their algorithm with other methods including Adagrad, Adam, DFW, L4Adam and SGD on learning a differentiable neural computer, object recognition, and a natural language processing task. Their experimental results show that ALI-G performance is comparable with that of SGD with schedule learning rate.