Deep Frank-Wolfe For Neural Network Optimization

1 INTRODUCTION . Since the introduction of back-propagation ( Rumelhart et al. , 1986 ) , stochastic gradient descent ( SGD ) has been the most commonly used optimization algorithm for deep neural networks . While yielding remarkable performance on a variety of learning tasks , a downside of the SGD algorithm is that it requires a schedule for the decay of its learning rate . In the convex setting , curvature properties of the objective function can be used to design schedules that are hyper-parameter free and guaranteed to converge to the optimal solution ( Bubeck , 2015 ) . However , there is no analogous result of practical interest for the non-convex optimization problem of a deep neural network . An illustration of this issue is the diversity of learning rate schedules used to train deep convolutional networks with SGD : Simonyan & Zisserman ( 2015 ) and He et al . ( 2016 ) adapt the learning rate according to the validation performance , while Szegedy et al . ( 2015 ) , Huang et al . ( 2017 ) and Loshchilov & Hutter ( 2017 ) use pre-determined schedules , which are respectively piecewise constant , geometrically decaying , and cyclic with a cosine annealing . While these protocols result in competitive or state-of-the-art results on their learning task , there does not seem to be a consistent methodology . As a result , finding such a schedule for a new setting is a time-consuming and computationally expensive effort . To alleviate this issue , adaptive gradient methods have been developed ( Zeiler , 2012 , Kingma & Ba , 2015 , Reddi et al. , 2018 ) , and borrowed from online convex optimization ( Duchi et al. , 2011 ) . Typically , these methods only require the tuning of the initial learning rate , the other hyper-parameters being considered robust across applications . However , it has been shown that such adaptive gradient methods obtain worse generalization than SGD ( Wilson et al. , 2017 ) . This observation is corroborated by our experimental results . In order to bridge this performance gap between existing adaptive methods and SGD , we introduce a new optimization algorithm , called Deep Frank-Wolfe ( DFW ) . The DFW algorithm exploits the composite structure of deep neural networks to design an optimization algorithm that leverages efficient convex solvers . In more detail , we consider a composite ( nested ) optimization problem , with the loss as the outer function and the function encoded by the neural network as the inner one . At each iteration , we define a proximal problem with a first-order approximation of the neural network ( linearized inner function ) , while keeping the loss function in its exact form ( exact outer function ) . When the loss is the hinge loss , each proximal problem created by our formulation is exactly a linear SVM . This allows us to employ the powerful Frank-Wolfe ( FW ) algorithm as the workhorse of our procedure . There are two by-design advantages to our method compared to the SGD algorithm . First , each iteration exploits more information about the learning objective , while preserving the same computational cost as SGD . Second , an optimal step-size is computed in closed-form by using the FW algorithm in the dual ( Frank & Wolfe , 1956 , Lacoste-Julien et al. , 2013 ) . Consequently , we do not need a hand-designed schedule for the learning rate . As a result , our algorithm is the first to provide competitive generalization error compared to SGD , all the while requiring a single hyper-parameter and often converging significantly faster . We present two additional improvements to customize the use of the DFW algorithm to deep neural networks . First , we show how to smooth the loss function to avoid optimization difficulties arising from learning deep models with SVMs ( Berrada et al. , 2018 ) . Second , we incorporate Nesterov momentum ( Nesterov , 1983 ) to accelerate our algorithm . We demonstrate the efficacy of our method on image classification with the CIFAR data sets ( Krizhevsky , 2009 ) using two architectures : wide residual networks ( Zagoruyko & Komodakis , 2016 ) and densely connected convolutional neural networks ( Huang et al. , 2017 ) ; we also provide experiments on natural language inference with a Bi-LSTM on the SNLI corpus ( Bowman et al. , 2015 ) . We show that the DFW algorithm often strongly outperforms previous methods based on adaptive learning rates . Furthermore , it provides comparable or better accuracy to SGD with hand-designed learning rate schedules . In conclusion , our contributions can be summed up as follows : • We propose a proximal framework which preserves information from the loss function . • For the first time for deep neural networks , we demonstrate how our formulation gives at each iteration ( i ) an optimal step-size in closed form and ( ii ) an update at the same computational cost as SGD . • We design a novel smoothing scheme for the dual optimization of SVMs . • To the best of our knowledge , the resulting DFW algorithm is the first to offer comparable or better generalization to SGD with a hand-designed schedule on the CIFAR data sets , all the while converging several times faster and requiring only a single hyperparameter . 2 RELATED WORK . Non Gradient-Based Methods . The success of a simple first-order method such as SGD has led to research in other more sophisticated techniques based on relaxations ( Heinemann et al. , 2016 , Zhang et al. , 2017a ) , learning theory ( Goel et al. , 2017 ) , Bregman iterations ( Taylor et al. , 2016 ) , and even second-order methods ( Roux et al. , 2008 , Martens & Sutskever , 2012 , Ollivier , 2013 , Desjardins et al. , 2015 , Martens & Grosse , 2015 , Grosse & Martens , 2016 , Ba et al. , 2017 , Botev et al. , 2017 , Martens et al. , 2018 ) . While such methods hold a lot of promise , their relatively large per-iteration cost limits their scalability in practice . As a result , gradient-based methods continue to be the most popular optimization algorithms for learning deep neural networks . Adaptive Gradient Methods . As mentioned earlier , one of the main challenges of using SGD is the design of a learning rate schedule . Several works proposed alternative first-order methods that do not require such a schedule , by either modifying the descent direction or adaptively rescaling the step-size ( Duchi et al. , 2011 , Zeiler , 2012 , Schaul et al. , 2013 , Kingma & Ba , 2015 , Zhang et al. , 2017b , Reddi et al. , 2018 ) . However , as noted above , the adaptive variants of SGD sometimes provide subpar generalization ( Wilson et al. , 2017 ) . Learning to Learn and Meta-Learning . Learning to learn approaches have also been proposed to optimize deep neural networks . Baydin et al . ( 2018 ) and Wu et al . ( 2018 ) learn the learning rate to avoid a hand-designed schedule and to improve practical performance . Such methods can be combined with our proposed algorithm to learn its proximal coefficient , instead of considering it as a fixed hyper-parameter to be tuned . Meta-learning approaches have also been suggested to learn the optimization algorithm ( Andrychowicz et al. , 2016 , Ravi & Larochelle , 2017 , Wichrowska et al. , 2017 , Li & Malik , 2017 ) . This line of work , which is orthogonal to ours , could benefit from the use of DFW to optimize the meta-learner . Optimization and Generalization . Several works study the relationship between optimization and generalization in deep learning . In order to promote generalization within the optimization algorithm itself , Neyshabur et al . ( 2015 ; 2016 ) proposed the Path-SGD algorithm , which implicitly controls the capacity of the model . However , their method required the model to employ ReLU non-linearity only , which is an important restriction for practical purposes . Hardt et al . ( 2016 ) , Arpit et al . ( 2017 ) , Neyshabur et al . ( 2017 ) , Hoffer et al . ( 2017 ) and Chaudhari & Soatto ( 2018 ) analyzed how existing optimization algorithms implicitly regularize deep neural networks . However this phenomenon is not yet fully understood , and the resulting empirical recommendations are sometimes opposing ( Hardt et al. , 2016 , Hoffer et al. , 2017 ) . Proximal Methods . The back-propagation algorithm has been analyzed in a proximal framework in ( Frerix et al. , 2018 ) . Yet , the resulting approach still requires the same hyper-parameters as SGD and incurs a higher computational cost per iteration . Linear SVM Sub-Problems . A main component of our formulation is to formulate sub-problems as linear SVMs . In an earlier work ( Berrada et al. , 2017 ) , we showed that neural networks with piecewise linear activations could be trained with the CCCP algorithm ( Yuille & Rangarajan , 2002 ) , which yielded approximate SVM problems to be solved with the BCFW algorithm ( Lacoste-Julien et al. , 2013 ) . However this algorithm only updates the parameters of one layer at a time , which slows down convergence significantly in practice . Closest to our approach are the works of ( Hochreiter & Obermayer , 2005 ) and ( Singh & Shawe-Taylor , 2018 ) . Hochreiter & Obermayer ( 2005 ) suggested to create a local SVM based on a first-order Taylor expansion and a proximal term , in order to lower the error of every data sample while minimizing the changes in the weights . However their method operated in a non-stochastic setting , making the approach infeasible for large-scale data sets . Singh & Shawe-Taylor ( 2018 ) , a parallel work to ours , also created an SVM problem using a first-order Taylor expansion , this time in a mini-batch setting . Their work provided interesting insights from a statistical learning theory perspective . While their method is well-grounded , its significantly higher cost per iteration impairs its practical speed and scalability . As such , it can be seen as complementary to our empirical work , which exploits a powerful solver and provides state-of-the-art scalability and performance . 3 PROBLEM FORMULATION . Before describing our formulation , we introduce some necessary notation . We use ‖ · ‖ to denote the Euclidean norm . Given a function φ , ∂φ ( u ) ∣∣ û is the derivative of φ with respect to u evaluated at û . According to the situation , this derivative can be a gradient , a Jacobian or even a directional derivative . Its exact nature will be clear from context throughout the paper . We also introduce the first-order Taylor expansion of φ around the point û : Tûφ ( u ) = φ ( û ) + ( ∂φ ( u ) ∣∣ û ) > ( u− û ) . For a positive integer p , we denote the set { 1 , 2 , ... , p } as [ p ] . For simplicity , we assume that stochastic algorithms process only one sample at each iteration , although the methods can be trivially extended to mini-batches of size larger than one . 3.1 LEARNING OBJECTIVE . We suppose we are given a data set ( xi , yi ) i∈ [ N ] , where each xi ∈ Rd is a sample annotated with a label yi from the output space Y . The data set is used to estimate a parameterized model represented by the function f . Given its ( flattened ) parameters w ∈ Rp , and an input xi ∈ Rd , the model predicts f ( w , xi ) ∈ R|Y| , a vector with one score per element of the output space Y . For instance , f can be a linear map or a deep neural network . Given a vector of scores per label s ∈ R|Y| , we denote by L ( s , yi ) the loss function that computes the risk of the prediction scores s given the ground truth label yi . For example , the loss L can be cross-entropy or the multi-class hinge loss : ( Cross-Entropy Loss ) LCE : ( s , y ) ∈ R|Y| × Y 7→ log ( ∑ k∈Y exp ( sk ) ) − sy , ( 1 ) ( Multi-Class Hinge Loss ) Lhinge : ( s , y ) ∈ R|Y|×Y 7→ max { max k∈Y\ { y } { sk + 1− sy } , 0 } . ( 2 ) The cross-entropy loss ( 1 ) tries to match the empirical distribution by driving incorrect scores as far as possible from the ground truth one . The hinge loss ( 2 ) attempts to create a minimal margin of one between correct and incorrect scores . The hinge loss has been shown to be more robust to over-fitting than cross-entropy , when combined with smoothing techniques that are common in the optimization literature ( Berrada et al. , 2018 ) . To simplify notation , we introduce fi ( w ) = f ( w , xi ) and Li ( s ) = L ( s , yi ) for each i ∈ [ N ] . Finally , we denote by ρ ( w ) the regularization ( typically the squared Euclidean norm ) . We now write the learning problem under its empirical risk minimization form : min w∈Rp ρ ( w ) + 1 N ∑ i∈ [ N ] Li ( fi ( w ) ) . ( 3 )

Summary: This paper introduced a proximal approach to optimize neural networks by linearizing the network output instead of the loss function. They demonstrate their algorithm on multi-class hinge loss, where they can show that optimal step size can be computed in close form without significant additional cost. Their experimental results showed competitive performance to SGD/Adam on the same network architectures.