Shallow Learning For Deep Networks

1 INTRODUCTION . Deep Convolutional Neural Networks ( CNNs ) trained on large-scale supervised data via the backpropagation algorithm have become the dominant approach in most computer vision tasks ( Krizhevsky et al. , 2012 ) . This has motivated successful applications of deep learning in other fields such as speech recognition ( Chan et al. , 2016 ) , natural language processing ( Vaswani et al. , 2017 ) , and reinforcement learning ( Silver et al. , 2017 ) . Training procedures and architecture choices for deep CNNs have become more and more entrenched , but which of the standard components of modern pipelines are essential to the success of deep CNNs is not clear . Here we pose the question : do CNN layers need to be learned end-to-end to obtain high performance ? We will show even for the complex imagenet dataset the answer is possibly no . Supervised end-to-end learning is the standard approach to neural network optimization . However it has potential issues that can be valuable to consider . First , the use of a global objective means that the final functional behavior of individual intermediate layers of a deep network is only indirectly specified : it is entirely unclear how the layers work together to achieve high-accuracy predictions . Several authors have suggested and shown empirically that CNNs learn to implement mechanisms that progressively induce invariance to complex , but irrelevant variability ( Mallat , 2016 ; Yosinski et al. , 2015 ) while increasing linear separability ( Zeiler & Fergus , 2014 ; Oyallon , 2017 ; Jacobsen et al. , 2018 ) of the data . Progressive linear separability has been shown empirically but it is unclear whether this is merely the consequence of other strategies implemented by CNNs , or if it is a sufficient condition for the observed high performance of these networks . Secondly , understanding the link between shallow Neural Networks ( NNs ) and deep NNs is difficult : while generalization , approximation , or optimization results ( Barron , 1994 ; Bach , 2014 ; Venturi et al. , 2018 ; Neyshabur et al. , 2018 ; Pinkus , 1999 ) for 1-hidden layer NNs are available , the same studies conclude that multiple-hidden-layer NNs are much more difficult to tackle theoretically . Finally , end-to-end backpropagation can be inefficient ( Jaderberg et al. , 2016 ; Salimans et al. , 2017 ) in terms of computation and memory resources . Moreover , for some learning problems , the full gradient is less informative than other alternatives ( Shalev-Shwartz et al. , 2017 ) . Sequential learning of CNN layers by solving shallow supervised learning problems is an alternative to end-to-end back-propagation . This strategy can directly specify the objective of every layer for example by encouraging the refinement of specific properties of the representation ( Greff et al. , 2016 ) , such as progressive linear separability . The development of theoretical tools for deep greedy methods could then draw from the theoretical understanding of shallow sub-problems . Indeed , Arora et al . ( 2018 ) ; Bengio et al . ( 2006 ) ; Bach ( 2014 ) ; Janzamin et al . ( 2015 ) show global optimal approximations , while other works have shown that networks based on 1-hidden layer training can have a variety of guarantees under certain assumptions ( Huang et al. , 2017 ; Malach & Shalev-Shwartz , 2018 ; Arora et al. , 2014 ) : greedy layerwise methods could permit to cascade those results to bigger architectures . Finally , a greedy approach will rely much less on having access to a full gradient . This can potentially avoid pathologies such as in Shalev-Shwartz et al . ( 2017 ) . From an algorithmic perspective , they do not require storing most of the intermediate activations nor to compute most intermediate gradients . This can be beneficial in memory-constrained settings . Unfortunately , prior work has not convincingly demonstrated that layerwise strategies can tackle the sort of large scale problems that have brought deep learning into the spotlight . We propose a straightforward strategy for CNNs that is shown to scale and analyze the representations it builds . Our contributions are as follows . ( a ) First , we design a simple and scalable supervised approach to learn layer-wise CNNs in Sec . 3 . ( b ) Then , Sec . 4.1 demonstrates empirically that by sequentially solving 1-hidden layer problems , we can match the performance of the AlexNet on ImageNet . This supports a body of literature that tackle 1-hidden layer networks and their sequentially trained counterparts . ( c ) We show that layerwise trained layers exhibit a progressive linear separability property in Sec . 4.2 . ( d ) In particular , we use this to help motivate learning layer-wise CNN layers via shallow k-hidden layer auxiliary problems , with k > 1 . Using this approach our sequentially trained 3-hidden layer models can reach the performance level of VGG-13 ( Sec . 4.3 ) . ( e ) Finally , we suggest an approach to easily reduce the model size during training of these networks . 2 RELATED WORK . Several authors have previously considered layerwise learning . In this section we review several of the related works and re-emphasize the distinctions from our work . Greedy unsupervised learning has been a popular topic of research in the past . Greedy unsupervised learning of deep generative models ( Bengio et al. , 2007 ; Hinton et al. , 2006 ) was shown to be effective as an initialization for deep supervised architectures . Bengio et al . ( 2007 ) also considered supervised greedy layerwise learning as initialization of networks for subsequent end-to-end supervised learning , but this was not shown to be effective with the existing techniques at the time . Later work on largescale supervised deep learning showed that modern training techniques permit avoiding layerwise initialization entirely ( Krizhevsky et al. , 2012 ) . We emphasize that the supervised layerwise learning we consider is distinct from unsupervised layerwise learning . Moreover , here layerwise training is not studied as a pretraining strategy , but a training one . Layerwise learning in the context of constructing supervised NNs has been attempted in several works . Early demonstrations have been made in Fahlman & Lebiere ( 1990b ) ; Lengellé & Denoeux ( 1996 ) on very simple problems and in a climate where deep learning was not a dominant supervised learning approach . These works were aimed primarily at structure learning , building up architectures that allow the model to grow appropriately based on the data . Similarly , Cortes et al . ( 2016 ) recently proposed a progressive learning method that builds a network such that the architecture can adapt to the problem , with focus on the theory associated with the structure learning problem , but do not consider problems where deep networks are unmatched . Malach & Shalev-Shwartz ( 2018 ) also train a supervised network in a layerwise fashion , showing that their method provably generalizes for a restricted class of image models . However , the results of these model are not shown to be competitive with handcrafted approaches ( Oyallon & Mallat , 2015 ) . Similarly ( Kulkarni & Karande , 2017 ) consider a kernel based layerwise objective , but with unconvincing results in very limited settings . Boosting techniques ( Friedman , 2001 ; Freund et al. , 1996 ) are a greedy approach to supervised learning with a successful history and theoretical foundation and still represents the state of the art in some domains ( Chen & Guestrin , 2016 ) . Recently Huang et al . ( 2017 ) combined boosting theory with a modern residual network ( He et al. , 2016 ) by sequentially training layers . The properties of the residual are exploited to effectively leverage boosting theory . However , results are presented for limited datasets and indicate that the end-to-end approach is often needed ultimately to obtain competitive results . The proposed sequential strategy does not clearly outperform simple non-deep learning baselines . By contrast our work focuses on settings where CNN based-approaches do not currently have competitors and introduces the use of auxiliary hidden layers . Another related thread are methods which add layers to existing networks and then use end-to-end learning . These approaches usually have different goals from ours , such as stabilizing end-to-end learned models . Brock et al . ( 2017 ) builds a network in stages , where certain layers are progressively frozen , which permits faster training . Mosca & Magoulas ( 2017 ) ; Wang et al . ( 2017 ) propose methods that stack progressively layers at each turn performing , end-to-end learning on the resulting network . A similar strategy was applied for training GANs in Karras et al . ( 2017 ) . By the nature of our objectives in this work , we never perform fine-tuning of the whole network . Finally several methods consider auxiliary supervised objectives Lee et al . ( 2015 ) to stabilize end-to-end learning , but which is rather different from the case where these objectives are not solved jointly . 3 SUPERVISED LAYERWISE TRAINING OF CNNS . In this section we formalize the architecture , training algorithm , and the necessary notations and terminology . We focus on CNNs , with ReLU non-linearity denoted by ρ. Sec . 3.1 will describe a layer-wise training scheme using a succession of auxiliary learning tasks . We add one layer at a time : the first layer of a k-hidden layer CNN problem . Finally , we will discuss the distinctions in varying k . 3.1 ARCHITECTURE FORMULATION . Our architecture has J blocks ( see Fig . 1 ) , which are trained in succession . From an input signal x , an initial representation x0 , x is propagated through j convolutions , giving xj . Each xj feeds into an auxiliary classifier to obtain prediction zj , which computes an intermediate classification output . At depth j , denote by Wθj a convolutional operator with parameters θj , Cγj an auxiliary classifier with all its parameters denoted γj , and Pj a down-sampling operator . The parameters correspond to 3× 3 kernels with bias terms . Formally , from layer xj we construct { xj+1 , zj+1 } as follows : { xj+1 = ρWθjPjxj zj+1 = Cγjxj+1 ∈ Rc ( 1 ) where c is the number of classes . For the pooling operator P we choose the invertible downsampling operation described in Dinh et al . ( 2017 ) , which consists in reorganizing the initial spatial channels into the 4 spatially decimated copies obtainable by 2×2 spatial sub-sampling , reducing the resolution by a factor 2 . We decided against strided pooling , average pooling , and the non-linear max-pooling , because these strongly encourage a loss of information . As is standard practice in CNNs , P is applied at certain layers ( Pj = P ) , but not others ( Pj = Id ) . The classifier Cγj is a CNN that can be written : Cγjxj = { LAxj for k = 1 LAρW̃k−2 ... ρW̃0xj for k > 1 ( 2 ) where W̃0 , ... , W̃k−2 are convolutional layers with constant width , A is a spatial averaging operator , and L a linear operator whose output dimension is c. We remark the averaging operation is important for maintaining scalability at early layers . Observe that for k = 1 , Cγj is simply a linear model , and in this case our architecture will be trained by a sequence of 1-hidden layer CNN . 3.2 TRAINING BY AUXILIARY PROBLEMS . Algorithm 1 : Layer Wise CNN Learning Input : Training samples { xn0 , yn } n≤N 1 for j ∈ 0 .. J − 1 do 2 Apply Eq . ( 1 ) to obtain { xnj } n≤N 3 Initialize θj , γj 4 ( θ∗j , γ ∗ j ) = argminθj , γj R̂ ( zj+1 ; θj , γj ) 5 end Our training procedure is layerwise : at depth j , while keeping all other parameters fixed , θj is obtained via an auxiliary problem : optimizing { θj , γj } to obtain the best training accuracy for auxiliary classifier Cγj . We now formalize this idea for a training set { xn , yn } n≤N . For a function z ( · ; θ , γ ) parametrized by { θ , γ } and a loss l ( e.g . cross entropy ) , we consider the classical minimization of the empirical risk : R̂ ( z ; θ , γ ) , 1 N ∑ n l ( z ( xn ; θ , γ ) , yn ) At depth j , assume we have constructed the parameters { θ∗0 , ... , θ∗j } . Our algorithm can produce samples { xnj } . Taking zj+1 = z ( xnj ; θj , γj ) , we will employ an optimization procedure that aims to minimize the risk R̂ ( zj+1 ; θj , γj ) . This procedure ( Alg . 1 ) consists in training ( e.g . using SGD ) the shallow CNN classifier Cj on top of xj , to obtain the new parameter θ∗j+1 . Under mild conditions , it improves the training error at each layer as shown below : Proposition 3.1 ( Progressive improvement ) . Assume that Pj = Id . Then there exists θ0 such that : R̂ ( zj+1 ; θ∗j , γ∗j ) ≤ R̂ ( zj+1 ; θ0 , γ∗j−1 ) = R̂ ( zj ; θ∗j−1 , γ∗j−1 ) . Proof . As ρ ( ρ ( x ) ) = ρ ( x ) , we simply have to chose θ0 such that Wθ0 = Id . A technical requirement for the actual optimization procedure is to not produce a worse objective than the initialization . It can be achieved by taking the best result along the optimization trajectory . The cascade can inherit from the individual properties of each auxiliary problem . For instance , as ρ is 1-Lipschitz , if each Wθ∗j is 1-Lipschitz then so is xJ w.r.t . x . Another example is the nested objective defined by Alg . 1 : The optimality of the solution will be largely governed by the optimality of the sub-problem solver . Specifically , if the auxiliary problem solution is close to optimal than the solution of Alg . 1 will be close to optimal . Proposition 3.2 . Assume the parameters { θ∗1 , ... , θ∗J } are obtained via a fixed layerwise optimization procedure . We assume that Wθ∗j is 1-lipschitz without loss of generality and that the biases are bounded uniformly by B . Given an input function g ( x ) , we consider functions of the type zg ( x ) = CγρWθg ( x ) . For > 0 , we call θ , g the parameter provided by a procedure to minimize R̂ ( zg ; θ ; γ ) and we assume it finds 1-lipschitz operators that satisfy : 1.∀g , g̃ , ‖ρWθ , gg ( x ) − ρWθ , g̃ g̃ ( x ) ‖ ≤ ‖g ( x ) − g̃ ( x ) ‖︸ ︷︷ ︸ ( stability ) , 2.‖Wθ∗j x ∗ j −Wθ , x∗ j x∗j‖ ≤ ( 1 + ‖x∗j‖ ) ︸ ︷︷ ︸ ( -approximation ) , with , x̃j+1 = ρWθ , x̃j x̃j and x ∗ j+1 = ρWθ∗j x ∗ j with x ∗ 0 = x̃0 = x , then , we prove by induction : ‖x∗J − x̃J‖ = O ( J2 ) ( 3 ) The proof can be found in the Appendix A . Thus , we have demonstrated an example of how our training strategy can permit to extend results from shallow CNNs to deeper CNNs , in particular for k = 1 .

Summary: The authors propose to train deep convolutional neural networks in a layerwise fashion. This is contrary to the traditional joint end-to-end training of deep CNNs. As their motivation, the authors quote computational and memory benefits at the time of training in addition to being able to extend shallow-network analytical frameworks to the individual network layers thus allowing for a theoretical interpretation of their optima.