Deep Frank-Wolfe For Neural Network Optimization

1 INTRODUCTION . Since the introduction of back-propagation ( Rumelhart et al. , 1986 ) , stochastic gradient descent ( SGD ) has been the most commonly used optimization algorithm for deep neural networks . While yielding remarkable performance on a variety of learning tasks , a downside of the SGD algorithm is that it requires a schedule for the decay of its learning rate . In the convex setting , curvature properties of the objective function can be used to design schedules that are hyper-parameter free and guaranteed to converge to the optimal solution ( Bubeck , 2015 ) . However , there is no analogous result of practical interest for the non-convex optimization problem of a deep neural network . An illustration of this issue is the diversity of learning rate schedules used to train deep convolutional networks with SGD : Simonyan & Zisserman ( 2015 ) and He et al . ( 2016 ) adapt the learning rate according to the validation performance , while Szegedy et al . ( 2015 ) , Huang et al . ( 2017 ) and Loshchilov & Hutter ( 2017 ) use pre-determined schedules , which are respectively piecewise constant , geometrically decaying , and cyclic with a cosine annealing . While these protocols result in competitive or state-of-the-art results on their learning task , there does not seem to be a consistent methodology . As a result , finding such a schedule for a new setting is a time-consuming and computationally expensive effort . To alleviate this issue , adaptive gradient methods have been developed ( Zeiler , 2012 , Kingma & Ba , 2015 , Reddi et al. , 2018 ) , and borrowed from online convex optimization ( Duchi et al. , 2011 ) . Typically , these methods only require the tuning of the initial learning rate , the other hyper-parameters being considered robust across applications . However , it has been shown that such adaptive gradient methods obtain worse generalization than SGD ( Wilson et al. , 2017 ) . This observation is corroborated by our experimental results . In order to bridge this performance gap between existing adaptive methods and SGD , we introduce a new optimization algorithm , called Deep Frank-Wolfe ( DFW ) . The DFW algorithm exploits the composite structure of deep neural networks to design an optimization algorithm that leverages efficient convex solvers . In more detail , we consider a composite ( nested ) optimization problem , with the loss as the outer function and the function encoded by the neural network as the inner one . At each iteration , we define a proximal problem with a first-order approximation of the neural network ( linearized inner function ) , while keeping the loss function in its exact form ( exact outer function ) . When the loss is the hinge loss , each proximal problem created by our formulation is exactly a linear SVM . This allows us to employ the powerful Frank-Wolfe ( FW ) algorithm as the workhorse of our procedure . There are two by-design advantages to our method compared to the SGD algorithm . First , each iteration exploits more information about the learning objective , while preserving the same computational cost as SGD . Second , an optimal step-size is computed in closed-form by using the FW algorithm in the dual ( Frank & Wolfe , 1956 , Lacoste-Julien et al. , 2013 ) . Consequently , we do not need a hand-designed schedule for the learning rate . As a result , our algorithm is the first to provide competitive generalization error compared to SGD , all the while requiring a single hyper-parameter and often converging significantly faster . We present two additional improvements to customize the use of the DFW algorithm to deep neural networks . First , we show how to smooth the loss function to avoid optimization difficulties arising from learning deep models with SVMs ( Berrada et al. , 2018 ) . Second , we incorporate Nesterov momentum ( Nesterov , 1983 ) to accelerate our algorithm . We demonstrate the efficacy of our method on image classification with the CIFAR data sets ( Krizhevsky , 2009 ) using two architectures : wide residual networks ( Zagoruyko & Komodakis , 2016 ) and densely connected convolutional neural networks ( Huang et al. , 2017 ) ; we also provide experiments on natural language inference with a Bi-LSTM on the SNLI corpus ( Bowman et al. , 2015 ) . We show that the DFW algorithm often strongly outperforms previous methods based on adaptive learning rates . Furthermore , it provides comparable or better accuracy to SGD with hand-designed learning rate schedules . In conclusion , our contributions can be summed up as follows : • We propose a proximal framework which preserves information from the loss function . • For the first time for deep neural networks , we demonstrate how our formulation gives at each iteration ( i ) an optimal step-size in closed form and ( ii ) an update at the same computational cost as SGD . • We design a novel smoothing scheme for the dual optimization of SVMs . • To the best of our knowledge , the resulting DFW algorithm is the first to offer comparable or better generalization to SGD with a hand-designed schedule on the CIFAR data sets , all the while converging several times faster and requiring only a single hyperparameter . 2 RELATED WORK . Non Gradient-Based Methods . The success of a simple first-order method such as SGD has led to research in other more sophisticated techniques based on relaxations ( Heinemann et al. , 2016 , Zhang et al. , 2017a ) , learning theory ( Goel et al. , 2017 ) , Bregman iterations ( Taylor et al. , 2016 ) , and even second-order methods ( Roux et al. , 2008 , Martens & Sutskever , 2012 , Ollivier , 2013 , Desjardins et al. , 2015 , Martens & Grosse , 2015 , Grosse & Martens , 2016 , Ba et al. , 2017 , Botev et al. , 2017 , Martens et al. , 2018 ) . While such methods hold a lot of promise , their relatively large per-iteration cost limits their scalability in practice . As a result , gradient-based methods continue to be the most popular optimization algorithms for learning deep neural networks . Adaptive Gradient Methods . As mentioned earlier , one of the main challenges of using SGD is the design of a learning rate schedule . Several works proposed alternative first-order methods that do not require such a schedule , by either modifying the descent direction or adaptively rescaling the step-size ( Duchi et al. , 2011 , Zeiler , 2012 , Schaul et al. , 2013 , Kingma & Ba , 2015 , Zhang et al. , 2017b , Reddi et al. , 2018 ) . However , as noted above , the adaptive variants of SGD sometimes provide subpar generalization ( Wilson et al. , 2017 ) . Learning to Learn and Meta-Learning . Learning to learn approaches have also been proposed to optimize deep neural networks . Baydin et al . ( 2018 ) and Wu et al . ( 2018 ) learn the learning rate to avoid a hand-designed schedule and to improve practical performance . Such methods can be combined with our proposed algorithm to learn its proximal coefficient , instead of considering it as a fixed hyper-parameter to be tuned . Meta-learning approaches have also been suggested to learn the optimization algorithm ( Andrychowicz et al. , 2016 , Ravi & Larochelle , 2017 , Wichrowska et al. , 2017 , Li & Malik , 2017 ) . This line of work , which is orthogonal to ours , could benefit from the use of DFW to optimize the meta-learner . Optimization and Generalization . Several works study the relationship between optimization and generalization in deep learning . In order to promote generalization within the optimization algorithm itself , Neyshabur et al . ( 2015 ; 2016 ) proposed the Path-SGD algorithm , which implicitly controls the capacity of the model . However , their method required the model to employ ReLU non-linearity only , which is an important restriction for practical purposes . Hardt et al . ( 2016 ) , Arpit et al . ( 2017 ) , Neyshabur et al . ( 2017 ) , Hoffer et al . ( 2017 ) and Chaudhari & Soatto ( 2018 ) analyzed how existing optimization algorithms implicitly regularize deep neural networks . However this phenomenon is not yet fully understood , and the resulting empirical recommendations are sometimes opposing ( Hardt et al. , 2016 , Hoffer et al. , 2017 ) . Proximal Methods . The back-propagation algorithm has been analyzed in a proximal framework in ( Frerix et al. , 2018 ) . Yet , the resulting approach still requires the same hyper-parameters as SGD and incurs a higher computational cost per iteration . Linear SVM Sub-Problems . A main component of our formulation is to formulate sub-problems as linear SVMs . In an earlier work ( Berrada et al. , 2017 ) , we showed that neural networks with piecewise linear activations could be trained with the CCCP algorithm ( Yuille & Rangarajan , 2002 ) , which yielded approximate SVM problems to be solved with the BCFW algorithm ( Lacoste-Julien et al. , 2013 ) . However this algorithm only updates the parameters of one layer at a time , which slows down convergence significantly in practice . Closest to our approach are the works of ( Hochreiter & Obermayer , 2005 ) and ( Singh & Shawe-Taylor , 2018 ) . Hochreiter & Obermayer ( 2005 ) suggested to create a local SVM based on a first-order Taylor expansion and a proximal term , in order to lower the error of every data sample while minimizing the changes in the weights . However their method operated in a non-stochastic setting , making the approach infeasible for large-scale data sets . Singh & Shawe-Taylor ( 2018 ) , a parallel work to ours , also created an SVM problem using a first-order Taylor expansion , this time in a mini-batch setting . Their work provided interesting insights from a statistical learning theory perspective . While their method is well-grounded , its significantly higher cost per iteration impairs its practical speed and scalability . As such , it can be seen as complementary to our empirical work , which exploits a powerful solver and provides state-of-the-art scalability and performance . 3 PROBLEM FORMULATION . Before describing our formulation , we introduce some necessary notation . We use ‖ · ‖ to denote the Euclidean norm . Given a function φ , ∂φ ( u ) ∣∣ û is the derivative of φ with respect to u evaluated at û . According to the situation , this derivative can be a gradient , a Jacobian or even a directional derivative . Its exact nature will be clear from context throughout the paper . We also introduce the first-order Taylor expansion of φ around the point û : Tûφ ( u ) = φ ( û ) + ( ∂φ ( u ) ∣∣ û ) > ( u− û ) . For a positive integer p , we denote the set { 1 , 2 , ... , p } as [ p ] . For simplicity , we assume that stochastic algorithms process only one sample at each iteration , although the methods can be trivially extended to mini-batches of size larger than one . 3.1 LEARNING OBJECTIVE . We suppose we are given a data set ( xi , yi ) i∈ [ N ] , where each xi ∈ Rd is a sample annotated with a label yi from the output space Y . The data set is used to estimate a parameterized model represented by the function f . Given its ( flattened ) parameters w ∈ Rp , and an input xi ∈ Rd , the model predicts f ( w , xi ) ∈ R|Y| , a vector with one score per element of the output space Y . For instance , f can be a linear map or a deep neural network . Given a vector of scores per label s ∈ R|Y| , we denote by L ( s , yi ) the loss function that computes the risk of the prediction scores s given the ground truth label yi . For example , the loss L can be cross-entropy or the multi-class hinge loss : ( Cross-Entropy Loss ) LCE : ( s , y ) ∈ R|Y| × Y 7→ log ( ∑ k∈Y exp ( sk ) ) − sy , ( 1 ) ( Multi-Class Hinge Loss ) Lhinge : ( s , y ) ∈ R|Y|×Y 7→ max { max k∈Y\ { y } { sk + 1− sy } , 0 } . ( 2 ) The cross-entropy loss ( 1 ) tries to match the empirical distribution by driving incorrect scores as far as possible from the ground truth one . The hinge loss ( 2 ) attempts to create a minimal margin of one between correct and incorrect scores . The hinge loss has been shown to be more robust to over-fitting than cross-entropy , when combined with smoothing techniques that are common in the optimization literature ( Berrada et al. , 2018 ) . To simplify notation , we introduce fi ( w ) = f ( w , xi ) and Li ( s ) = L ( s , yi ) for each i ∈ [ N ] . Finally , we denote by ρ ( w ) the regularization ( typically the squared Euclidean norm ) . We now write the learning problem under its empirical risk minimization form : min w∈Rp ρ ( w ) + 1 N ∑ i∈ [ N ] Li ( fi ( w ) ) . ( 3 )

Summary: This paper proposes a Frank-Wolfe based method, called DFW, for training Deep Network. The DFW method linearizes the loss function into a smooth one, and also adopts Nesterov Momentum to accelerate the training. Both techniques have been widely used in the literature for similar settings. This paper mainly focuses on the algorithm part, but only empirically demonstrate the convergence results.