Title: SHALLOW LEARNING FOR DEEP NETWORKS

Abstract: Shallow supervised 1-hidden layer neural networks have a number of favorable properties that make them easier to interpret, analyze, and optimize than their deep counterparts, but lack their representational power. Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks. Contrary to previous approaches using shallow networks, we focus on problems where deep learning is reported as critical for success. We thus study CNNs on image recognition tasks using the large-scale ImageNet dataset and the CIFAR-10 dataset. Using a simple set of ideas for architecture and training we find that solving sequential 1-hidden-layer auxiliary problems leads to a CNN that exceeds AlexNet performance on ImageNet. Extending our training methodology to construct individual layers by solving 2and-3-hidden layer auxiliary problems, we obtain an 11-layer network that exceeds VGG-11 on ImageNet obtaining 89.8% top-5 single crop. To our knowledge, this is the first competitive alternative to end-to-end training of CNNs that can scale to ImageNet. We conduct a wide range of experiments to study the properties this induces on the intermediate layers.

1. INTRODUCTION:

Deep Convolutional Neural Networks (CNNs) trained on large-scale supervised data via the backpropagation algorithm have become the dominant approach in most computer vision tasks (Krizhevsky et al., 2012) . This has motivated successful applications of deep learning in other fields such as speech recognition (Chan et al., 2016) , natural language processing (Vaswani et al., 2017) , and reinforcement learning (Silver et al., 2017) . Training procedures and architecture choices for deep CNNs have become more and more entrenched, but which of the standard components of modern pipelines are essential to the success of deep CNNs is not clear. Here we pose the question: do CNN layers need to be learned end-to-end to obtain high performance? We will show even for the complex imagenet dataset the answer is possibly no. Supervised end-to-end learning is the standard approach to neural network optimization. However it has potential issues that can be valuable to consider. First, the use of a global objective means that the final functional behavior of individual intermediate layers of a deep network is only indirectly specified: it is entirely unclear how the layers work together to achieve high-accuracy predictions. Several authors have suggested and shown empirically that CNNs learn to implement mechanisms that progressively induce invariance to complex, but irrelevant variability (Mallat, 2016; Yosinski et al., 2015) while increasing linear separability (Zeiler & Fergus, 2014; Oyallon, 2017; Jacobsen et al., 2018) of the data. Progressive linear separability has been shown empirically but it is unclear whether this is merely the consequence of other strategies implemented by CNNs, or if it is a sufficient condition for the observed high performance of these networks. Secondly, understanding the link between shallow Neural Networks (NNs) and deep NNs is difficult: while generalization, approximation, or optimization results (Barron, 1994; Bach, 2014; Venturi et al., 2018; Neyshabur et al., 2018; Pinkus, 1999) for 1-hidden layer NNs are available, the same studies conclude that multiple-hidden-layer NNs are much more difficult to tackle theoretically. Finally, end-to-end backpropagation can be inefficient (Jaderberg et al., 2016; Salimans et al., 2017) in terms of computation and memory resources. Moreover, for some learning problems, the full gradient is less informative than other alternatives (Shalev-Shwartz et al., 2017) .

Sequential learning of CNN layers by solving shallow supervised learning problems is an alternative to end-to-end back-propagation. This strategy can directly specify the objective of every layer for example by encouraging the refinement of specific properties of the representation (Greff et al., 2016) , such as progressive linear separability. The development of theoretical tools for deep greedy methods could then draw from the theoretical understanding of shallow sub-problems. Indeed, Arora et al. (2018) ; Bengio et al. (2006) ; Bach (2014) ; Janzamin et al. (2015) show global optimal approximations, while other works have shown that networks based on 1-hidden layer training can have a variety of guarantees under certain assumptions (Huang et al., 2017; Malach & Shalev-Shwartz, 2018; Arora et al., 2014) : greedy layerwise methods could permit to cascade those results to bigger architectures. Finally, a greedy approach will rely much less on having access to a full gradient. This can potentially avoid pathologies such as in Shalev-Shwartz et al. (2017) . From an algorithmic perspective, they do not require storing most of the intermediate activations nor to compute most intermediate gradients. This can be beneficial in memory-constrained settings. Unfortunately, prior work has not convincingly demonstrated that layerwise strategies can tackle the sort of large scale problems that have brought deep learning into the spotlight. We propose a straightforward strategy for CNNs that is shown to scale and analyze the representations it builds.

Our contributions are as follows. (a) First, we design a simple and scalable supervised approach to learn layer-wise CNNs in Sec. 3. (b) Then, Sec. 4.1 demonstrates empirically that by sequentially solving 1-hidden layer problems, we can match the performance of the AlexNet on ImageNet. This supports a body of literature that tackle 1-hidden layer networks and their sequentially trained counterparts. (c) We show that layerwise trained layers exhibit a progressive linear separability property in Sec. 4.2. (d) In particular, we use this to help motivate learning layer-wise CNN layers via shallow k-hidden layer auxiliary problems, with k > 1. Using this approach our sequentially trained 3-hidden layer models can reach the performance level of . (e) Finally, we suggest an approach to easily reduce the model size during training of these networks.

2. RELATED WORK:

Several authors have previously considered layerwise learning. In this section we review several of the related works and re-emphasize the distinctions from our work.

Greedy unsupervised learning has been a popular topic of research in the past. Greedy unsupervised learning of deep generative models (Bengio et al., 2007; Hinton et al., 2006) was shown to be effective as an initialization for deep supervised architectures. Bengio et al. (2007) also considered supervised greedy layerwise learning as initialization of networks for subsequent end-to-end supervised learning, but this was not shown to be effective with the existing techniques at the time. Later work on largescale supervised deep learning showed that modern training techniques permit avoiding layerwise initialization entirely (Krizhevsky et al., 2012) . We emphasize that the supervised layerwise learning we consider is distinct from unsupervised layerwise learning. Moreover, here layerwise training is not studied as a pretraining strategy, but a training one.

Layerwise learning in the context of constructing supervised NNs has been attempted in several works. Early demonstrations have been made in Fahlman & Lebiere (1990b) ; Lengellé & Denoeux (1996) on very simple problems and in a climate where deep learning was not a dominant supervised learning approach. These works were aimed primarily at structure learning, building up architectures that allow the model to grow appropriately based on the data. Similarly, Cortes et al. (2016) recently proposed a progressive learning method that builds a network such that the architecture can adapt to the problem, with focus on the theory associated with the structure learning problem, but do not consider problems where deep networks are unmatched. Malach & Shalev-Shwartz (2018) also train a supervised network in a layerwise fashion, showing that their method provably generalizes for a restricted class of image models. However, the results of these model are not shown to be competitive with handcrafted approaches (Oyallon & Mallat, 2015) . Similarly (Kulkarni & Karande, 2017) consider a kernel based layerwise objective, but with unconvincing results in very limited settings.

Boosting techniques (Friedman, 2001; Freund et al., 1996) are a greedy approach to supervised learning with a successful history and theoretical foundation and still represents the state of the art in some domains (Chen & Guestrin, 2016) . Recently Huang et al. (2017) combined boosting theory with a modern residual network (He et al., 2016) by sequentially training layers. The properties of the residual are exploited to effectively leverage boosting theory. However, results are presented for limited datasets and indicate that the end-to-end approach is often needed ultimately to obtain ...

Figure 1 : High level diagram of our layer-wise learning framework using a k = 2-hidden layer. P , the down-sampling (Jacobsen et al., 2018, Fig. 2) , is applied at the input image as well as at j = 2. competitive results. The proposed sequential strategy does not clearly outperform simple non-deep learning baselines. By contrast our work focuses on settings where CNN based-approaches do not currently have competitors and introduces the use of auxiliary hidden layers.

Another related thread are methods which add layers to existing networks and then use end-to-end learning. These approaches usually have different goals from ours, such as stabilizing end-to-end learned models. Brock et al. (2017) builds a network in stages, where certain layers are progressively frozen, which permits faster training. Mosca & Magoulas (2017) ; Wang et al. (2017) propose methods that stack progressively layers at each turn performing, end-to-end learning on the resulting network. A similar strategy was applied for training GANs in Karras et al. (2017) . By the nature of our objectives in this work, we never perform fine-tuning of the whole network. Finally several methods consider auxiliary supervised objectives Lee et al. (2015) to stabilize end-to-end learning, but which is rather different from the case where these objectives are not solved jointly.

3. SUPERVISED LAYERWISE TRAINING OF CNNS:

In this section we formalize the architecture, training algorithm, and the necessary notations and terminology. We focus on CNNs, with ReLU non-linearity denoted by ρ. Sec. 3.1 will describe a layer-wise training scheme using a succession of auxiliary learning tasks. We add one layer at a time: the first layer of a k-hidden layer CNN problem. Finally, we will discuss the distinctions in varying k.

3.1. ARCHITECTURE FORMULATION:

Our architecture has J blocks (see Fig. 1 ), which are trained in succession. From an input signal x, an initial representation x 0 x is propagated through j convolutions, giving x j . Each x j feeds into an auxiliary classifier to obtain prediction z j , which computes an intermediate classification output. At depth j, denote by W θj a convolutional operator with parameters θ j , C γj an auxiliary classifier with all its parameters denoted γ j , and P j a down-sampling operator. The parameters correspond to 3 × 3 kernels with bias terms. Formally, from layer x j we construct {x j+1 , z j+1 } as follows:

x j+1 = ρW θj P j x j z j+1 = C γj x j+1 ∈ R c (1)

where c is the number of classes. For the pooling operator P we choose the invertible downsampling operation described in Dinh et al. (2017) , which consists in reorganizing the initial spatial channels into the 4 spatially decimated copies obtainable by 2 × 2 spatial sub-sampling, reducing the resolution by a factor 2. We decided against strided pooling, average pooling, and the non-linear max-pooling, because these strongly encourage a loss of information. As is standard practice in CNNs, P is applied at certain layers (P j = P ), but not others (P j = Id). The classifier C γj is a CNN that can be written:

C γj x j = LAx j for k = 1 LAρ Wk-2 ...ρ W0 x j for k > 1 (2)

where W0 , ..., Wk-2 are convolutional layers with constant width, A is a spatial averaging operator, and L a linear operator whose output dimension is c. We remark the averaging operation is important for maintaining scalability at early layers. Observe that for k = 1, C γj is simply a linear model, and in this case our architecture will be trained by a sequence of 1-hidden layer CNN.

3.2. TRAINING BY AUXILIARY PROBLEMS:

Algorithm 1: Layer Wise CNN Learning

Input :Training samples {x n 0 , y n } n≤N 1 for j ∈ 0..J -1 do 2 Apply Eq.(1) to obtain {x n j } n≤N 3 Initialize θj, γj 4 (θ * j , γ * j ) = arg min θ j ,γ j R(zj+1; θj, γj) 5 end

Our training procedure is layerwise: at depth j, while keeping all other parameters fixed, θ j is obtained via an auxiliary problem: optimizing {θ j , γ j } to obtain the best training accuracy for auxiliary classifier C γj . We now formalize this idea for a training set {x n , y n } n≤N . For a function z(•; θ, γ) parametrized by {θ, γ} and a loss l (e.g. cross entropy), we consider the classical minimization of the empirical risk:

R(z; θ, γ) 1 N n l(z(x n ; θ, γ), y n )

At depth j, assume we have constructed the parameters {θ * 0 , ..., θ * j }. Our algorithm can produce samples {x n j }. Taking z j+1 = z(x n j ; θ j , γ j ), we will employ an optimization procedure that aims to minimize the risk R(z j+1 ; θ j , γ j ). This procedure (Alg. 1) consists in training (e.g. using SGD) the shallow CNN classifier C j on top of x j , to obtain the new parameter θ * j+1 . Under mild conditions, it improves the training error at each layer as shown below:

Proposition 3.1 (Progressive improvement). Assume that P j = Id. Then there exists θ 0 such that:

R(z j+1 ; θ * j , γ * j ) ≤ R(z j+1 ; θ 0 , γ * j-1 ) = R(z j ; θ * j-1 , γ * j-1 ).

Proof. As ρ(ρ(x)) = ρ(x), we simply have to chose θ 0 such that W θ0 = Id.

A technical requirement for the actual optimization procedure is to not produce a worse objective than the initialization. It can be achieved by taking the best result along the optimization trajectory.

The cascade can inherit from the individual properties of each auxiliary problem. For instance, as

ρ is 1-Lipschitz, if each W θ * j is 1-Lipschitz then so is x J w.r.t. x.

Another example is the nested objective defined by Alg. 1: The optimality of the solution will be largely governed by the optimality of the sub-problem solver. Specifically, if the auxiliary problem solution is close to optimal than the solution of Alg. 1 will be close to optimal. Proposition 3.2. Assume the parameters {θ * 1 , ..., θ * J } are obtained via a fixed layerwise optimization procedure. We assume that W θ * j is 1-lipschitz without loss of generality and that the biases are bounded uniformly by B. Given an input function g(x), we consider functions of the type z g (x) = C γ ρW θ g(x). For > 0, we call θ ,g the parameter provided by a procedure to minimize R(z g ; θ; γ) and we assume it finds 1-lipschitz operators that satisfy:

1.∀g, g, ρW θ ,g g(x) -ρW θ ,g g(x) ≤ g(x) -g(x) (stability) , 2. W θ * j x * j -W θ ,x * j x * j ≤ (1 + x * j ) ( -approximation)

, with, xj+1 = ρW θ ,x j xj and x * j+1 = ρW θ * j x * j with x * 0 = x0 = x, then, we prove by induction:

x * J -xJ = O(J 2 ) (3)

The proof can be found in the Appendix A. Thus, we have demonstrated an example of how our training strategy can permit to extend results from shallow CNNs to deeper CNNs, in particular for k = 1.

3.3. AUXILIARY PROBLEMS AND THE PROPERTIES THEY INDUCE:

We now discuss the properties arising from the auxiliary problems. We start with k = 1, for which the auxiliary classifier consists of only the linear A and L operators. Thus, the optimization aims to obtain the weights of a 1-hidden layer NN. For this case, as discussed in Sec. 1, a variety of theoretical results exist (e.g. (Cybenko, 1989; Barron, 1994) Furthermore, for the case of k = 1, the optimization of the 1-hidden layer network will encourage the hidden layer outputs to, maximally, linearly separate the training data. Specializing Prop. 3.1 for this case shows that the layerwise k = 1 procedure will try to progressively improve the linear separability. Progressive linear separation has been empirically studied in end-to-end CNNs (Zeiler & Fergus, 2014; Oyallon, 2017) as an indirect consequence, while the k = 1 training permits us to study this basic principle more directly as the layer objective in the sequel.

Unique to our layer-wise learning formulation, we consider the case where the auxiliary learning problem involves several auxiliary hidden layers. We will interpret, and empirically verify, in Sec. 4.2 that this builds layers that are progressively better inputs to shallow CNNs. We will also show a link to building, in a more progressive manner, linearly separable layers. Considering only shallow (with respect to total depth) auxiliary problems (e.g. k = 2, 3 in our work) we can maintains several advantages. Indeed, optimization for shallow networks is generally easier, as we can for example diminish the vanishing gradient problem, reducing the need for identity loops or normalization techniques (He et al., 2016) . Two and three layers networks are also appealing for extending results from one hidden layer as they are the next natural member in the family of NNs.

4. EXPERIMENTS AND DISCUSSION:

We performed experiments on the large-scale ImageNet-1k (Russakovsky et al., 2015) , a major catalyst for the recent popularity of deep learning, as well as the CIFAR-10 dataset. We study the classification performance of layerwise models with k = 1, comparing them to standard benchmarks and other sequential learning methods. Then we inspect the representations built through our auxiliary tasks and motivate the use of models learned with auxiliary hidden layers k > 1, which we subsequently evaluate at scale.

We call M the number of feature maps of the first convolution of the network and M the number of feature maps of the first convolution of the auxiliary classifiers. This fully defines the width of all the layers, since input width and output width are equal unless the layer has downsampling, in which case the output width is twice the input width. Finally, A is chosen to average over the four spatial quadrants, yielding a 2 × 2-shaped output. Spatial averaging before the linear layer is common in ResNets (He et al., 2016) to reduce size. In our case this is critical to permit scalability to large image sizes at early layers of layer-wise training. For computational reasons on ImageNet, an invertible downsampling is also applied (reducing the signal to output 12 × 112 2 ). We also construct an ensemble model, which consists of a weighted average of all auxiliary classifier outputs, i.e. Z = J j=1 2 j z j . We briefly introduce the datasets and preprocessing. The CIFAR-10 dataset consists of small RGB images with 50k samples for training and 10k samples for testing. We use the standard data augmentation and optimize each layer with SGD using a momentum of 0.9 and a batch-size of 128. The initial learning rate is set to 0.1 and we use the reduced schedule with decays of 0.2 every 15 epochs (Zagoruyko & Komodakis, 2016) , for a total of 50 epochs in each layer. The ImageNet dataset consists of 1.2M RGB images of size varying size for training. Our data augmentations consists of random crops of size 224 2 . At testing time, the image is rescaled to a size of 256 2 then cropped at size 224 2 . We used SGD with momentum 0.9 for a batch size of 256. The initial learning rate is 0.1 (He et al., 2016) and we use the reduced schedule with decays of 0.1 every 20 epochs for 45 epochs. We use 4 GPUs to train our ImageNet models.

4.1. ALEXNET ACCURACY WITH 1-HIDDEN LAYER AUXILIARY PROBLEMS:

We consider the, atomic, layerwise CNN with k = 1 which corresponds to solving a sequence of 1-hidden layer CNN problems. As discussed in Sec. 2, previous attempts at supervised layerwise training (Fahlman & Lebiere, 1990a; Arora et al., 2014; Huang et al., 2017; Malach & Shalev-Shwartz, 2018) , which rely solely on sequential solving of shallow problems have yielded performance well below that of typical deep learning models on the CIFAR dataset. None of them have scaled to datasets such as ImageNet, where end-to-end CNNs have proved absolutely critical (Bartunov et al., 2018) . We show, surprisingly, that it is possible to go beyond the AlexNet performance barrier (Krizhevsky et al., 2012) without end-to-end backpropagation on ImageNet with this elementary auxiliary problem. To emphasize the stability of the training process and to permit comparison the original AlexNet architecture we do not apply any batch-norm to this model.

CIFAR-10:

. We trained a model with J = 5 layers, down-sampling at layers j = 1, 3, and layer sizes starting at M = 256. We obtain 88.3% and note that this accuracy is close to the AlexNet model performance (Krizhevsky et al., 2012) for CIFAR-10 (89.0%). Previous attempts at sequentially trained 1-hidden layer networks have yielded performance that do not exceed that of the top handcrafted methods or those using unsupervised learning. To the best of our knowledge they obtain 82.0% accuracy (Huang et al., 2017) . The end-to-end version of this model obtains 89.7%, we note the lack of batch-norm is more detrimental in the end to end setting. Full comparisons are shown in Table 2 .

ImageNet. Our model is trained with J = 8 layers and downsampling operations at layers j = 2, 3, 4, 6. Layer sizes start at M = 256. Our final trained model achieves 79.7% top-5 single crop accuracy on the validation set and 80.8% a weighted ensemble of the layer outputs. In addition to exceeding AlexNet, this model compares favorably to all alternatives to end-to-end supervised CNNs including hand crafted computer vision and unsupervised learning techniques (Noroozi & Favaro, 2016; Perronnin & Larlus, 2015; Sánchez et al., 2013; Oyallon et al., 2017) (full results shown in Table 1 ). We also note that our final training accuracy is relatively high for ImageNet (87% -see also Appendix B), which indicates that appropriate regularization may lead to a further scaling. We now look at empirical properties induced in the layers and subsequently evaluate the distinct k > 1.

4.2. EMPIRICAL SEPARABILITY PROPERTIES:

We study the intermediate representations generated by the layerwise learning procedure in terms of linear separability as well as separability by a more general set of classifiers. Our aims are (a) to determine empirically whether k = 1 indeed progressively builds more and more linearly separable data representations and (b) to determine how linear separability of the representations evolves for networks constructed with k > 1 auxiliary problems. Finally we ask whether the notion of building progressively better inputs to a linear model (k = 1 training) has an analogous counterpart for k > 1: building progressively better inputs for shallow CNNs (discussed in Sec 3.3).

We define linear separability of a representation as the maximum accuracy achievable by a linear classifier. Further we define the notion of CNN-p-separability as the accuracy achieved by a p-layer CNN trained on top of the representation to be assessed.

We focus on CNNs trained on CIFAR-10 without downsampling. Here, J = 5 and we vary the layer sizes M = 64, 128, 256. The auxiliary classifier feature map size, when applicable, is M = 256. We train with 5 random initializations for each network and report an average standard deviation of 0.58% test accuracy. Each layer is evaluated by training a one-versus-rest logistic regression, as well as p = 1, 2-hidden-layer CNN on top of these representations. Because the linear representation has been optimized for it, we spatially average to a 2 × 2 shape before feeding them to our learning algorithms. Fig. 2 shows the results of each of these evaluations plotting test set accuracy curves as a function of neural network depth for each of the three evaluations. For these plots we averaged over initial layer sizes M and classifier layer sizes M and random seeds. Each individual curve closely resembles these average curves, with slight shifts in the y-axis, depending on M and M .

We observe that linear separability monotonically increases with layer depth as expected from Sec. 3.3 for k = 1. Interestingly, we find that linear separability also obtains in the case of k > 1, even though it is not directly specified by the auxiliary problem objective. At earlier layers, linear separation For Linear separability we aggregate across M = 64, 128, 256, individual results are shown in Appendix C.2, the relative trends are largely unchanged, although overall accuracies are higher in larger M . For CNN-p probes, all models achieve 100% train accuracy at the first or 2nd layer, thus only test accuracy is reported. capability of models trained with k = 1 increases fastest as a function of layer depth compared to models trained with deeper auxiliary networks, but flattens out to a lower asymptotic linear separability at deeper layers. This shows that the simple principle of the k = 1 objective that tries to produce the maximal linear separation at each layer might not be an optimal strategy for achieving "progressive" linear separation.

We also notice that the deeper the auxiliary classifier, the slower is the increase in linear separability initially, but the higher is the linear separability at deeper layers. From the two right diagrams we also find that the CNN-p-separability progressively improves -but much more so for k > 1 trained networks. This shows that linear separability of a layer is not the sole criterion for rendering a representation a good "input" for a CNN. It further shows that our sequential training procedure for the case k > 1 can indeed build a representation that is progressively a better input to a shallow CNN.

4.3. SCALING UP LAYERWISE CNNS WITH 2 AND 3 HIDDEN LAYER AUXILIARY PROBLEMS:

We study the training of deep networks with k = 2, 3 hidden layer auxiliary problems. We limit ourselves to this setting to keep the auxiliary models shallow with respect to the network depth. We employ widths of M = 128 and M = 256 for both CIFAR-10 and ImageNet. For CIFAR-10, the total number of layers is J = 4. A downsampling is applied at depth j = 2. For ImageNet we closely follow the VGG architectures, which with their 3 × 3 convolutions and absence of skip-connections bear strong similarity to ours. We use J = 8, giving 11 total layers (similar to e.g. VGG-11). As we start at halved resolution we do only 3 downsamplings at j = 2, 4, 6. Unlike the k = 1 case we found it helpful to employ batch-norm for these auxiliary problems.

We report our results for k = 2, 3 in Table 2 (CIFAR-10) and Table 1 (ImageNet) along with the results for our k = 1 model. As expected from the previous section, the transition from k = 1 to k = 2, 3 improves the performances substantially. We compare our CIFAR-10 results to other sequentially trained propositions in the literature. Our methods exceed these in performance by a large margin, while the ensemble model of k = 3 surpasses the VGG, the other sequential models perform do not exceed unsupervised methods. No alternative sequential models are available for ImageNet. We thus compare our results on ImageNet to the standard reference CNNs and the bestperforming alternatives to end-to-end Deep CNNs. Our k = 3 layerwise ensemble model achieves 89.8% accuracy, which is comparable to VGG-13 and largely exceeds AlexNet performance. The reference model accuracies for AlexNet, VGG, and ResNet-152 use the same input sizes and single

Layerwise Trained:

Top-1 (Ens.) 92.5 WRN 28-10 (Zagoruyko et al. 2016) 96.0

End-to-end k=1 89.7 Alternatives (Oyallon & Mallat, 2015) 82.3 Unsup. + SVM (Dosovitskiy et al., 2014) 84.3

Table 2 : Results on CIFAR-10. Compared to the few existing methods using only layerwise training schemes we report substantial performance improvement. Overall our models are competitive with well known benchmarks models that like ours do not use skip connections.

crop evaluation1 . Reference models relying on residual connections and very deep networks have substantially better performance than our models. We believe that one can extend layer-wise learning to these modern techniques. However, this is outside the scope of this work. Moreover, recent ImageNet models (after VGG) are developed in industry settings, with large scale infrastructure available for architecture and hyper-parameter search.

We emphasize that our approach enables the training of much larger layerwise models than end-to-end ones on the same hardware. This suggests applications in fields with large models (e.g. 3-D vision and medical imaging). We also observed that using outputs of early layers that were not yet converged still permitted improvement in subsequent layers. This suggests that our framework might allow an extension that solves the auxiliary problems in parallel to a certain degree.

Reducing final auxiliary network Recall M is the width of the initial auxiliary CNN. Let Mf denote the width of the final auxiliary CNN. In the experiments above this is relatively large ( Mf = 2048). We observed that although a larger M during training can be beneficial, particularly in earlier layers, the final representation will tend to be a good input even for smaller classifier networks fit after the primary network (up to x J-1 ) is trained. We thus reduce the width of the final auxiliary network k = 3 by performing an extra, smaller auxiliary problem evaluation to update step j = 7. We use auxiliary networks of size Mf = 512 and 1024 (instead of 2048). While the model size is reduced substantially, we observe only a limited loss of accuracy. For comparison, we train an end-to-end network with the same architecture as our J = 8 network with the final auxiliary of Mf = 512.

Tab. 1 shows the accuracy of the end-to-end model is 90.1% top-5 compared to 88.5% top-5 for the sequentially trained. We show a direction to close this relatively small gap in the next section.

Layerwise Model Compression Wide, overparametrized, layers have been shown to be important for learning (Neyshabur et al., 2018 ), but it is often possible to reduce the layer size a posteriori without losing significant accuracy (Hinton et al., 2014; LeCun et al., 1990) . For the specific case of CNNs, one technique removes channels heuristically and then fine-tunes (Molchanov et al., 2016) .In our setting, a natural strategy presents itself, which integrates compression into the learning process: (a) train a new layer (via an auxiliary problem) and (b) immediately apply model compression to the new layer. The model-compression-related fine-tuning operates over a single layer, making it fast and the subsequent training steps have a smaller input and thus fewer parameters, which speeds up the sequential training. We implement this approach using the filter removal technique of Molchanov et al. (2016) only at each newly trained layer, followed by a fine-tuning of the auxiliary network. We test this idea on CIFAR-10. A baseline network of 5 layers of size 64 (no downsampling, trained for 120 epochs and lr drops each 25 epochs) obtains an end-to-end performance of 87.5%. We use our layer-wise learning with k = 3, J = 3, M = 128, M = 128. At each step we prune each layer from 128 to 64 filters and subsequently fine-tune the auxiliary network to the remaining features over 20 epochs. We then use a final auxiliary of Mf = 64 obtaining a sequentially learned, final network of the same architecture as the baseline. The final accuracy is 87.6%, which is very close to the baseline. We note that each auxiliary problem incurs minimal reduction in accuracy through feature reduction.

Unlike the previous experiment, where our final performance was slightly below that of end-to-end on the same architecture, this gap could be closed by easy-to-integrate compression approaches.

5. CONCLUSION:

We have shown, to the best of our knowledge, the first alternative to end-to-end learning that scales on large-scale benchmarks such as ImageNet and can be competitive with standard CNN baselines. We build competitive models by training only shallow CNNs and using standard architectural elements (ReLU, convolution) . This shows that the approach is generic and could be adapted to more complex classes of NNs. Layerwise training opens the door to applications such as larger models under memory constraints, model prototyping, joint model compression and training, and more stable training for challenging scenarios. The framework may be extendable to the parallel training of layers as well as the development of novel localized feedback mechanisms. Importantly, our results suggest a number of open questions regarding the mechanisms that underlie the success of CNNs: for example can the 1-hidden layer network objective be better specified, filling in the gap between 1-hidden layer network and the k > 1 approach? Moreover, our models can potentially provide easier to study, high performance models for researchers.

A PROOF OF PROPOSITION:

Proposition A.1. Assume the parameters {θ * 1 , ..., θ * J } are obtained via a fixed layerwise optimization procedure. We assume that W θ * j is 1-lipschitz without loss of generality and that the biases are bounded uniformly by B. Given an input function g(x), we consider functions of the type z g (x) = C γ ρW θ g(x). For > 0, we call θ ,g the parameter provided by a procedure to minimize R(z g ; θ; γ) and we assume it finds 1-lipschitz operator that satisfy:

1.∀g, g, ρW θ ,g g(x) -ρW θ ,g g(x) ≤ g(x) -g(x)

(stability) , 2. W θ * j x * j -W θ ,x * j x * j ≤ (1 + x * j ) ( -approximation)

, with, xj+1 = ρW θ ,x j xj and x * j+1 = ρW θ * j x * j with x * 0 = x0 = x, then, we prove by induction:

x * J -xJ = O(J 2 ) (4)

Proof. First observe that x * j+1 ≤ x * j + B by non expansivity. Thus, by induction, x * j ≤ jB + x . Then, let us show that: x * j -xj ≤ ( j(j-1) 2 B + j x + j) by induction. Indeed, for j + 1:

x * j+1 -xj+1 = ρW θ * j x * j -ρW θ ,x j xj = ρW θ * j x * j -ρW θ ,x * j x * j + ρW θ ,x * j x * j -ρW θ ,x j xj ≤ W θ * j x * j -W θ ,x * j x * j + ρW θ ,x * j

x * j -ρW θ ,x j xj by non-expansivity ≤ x * j + + x * j -xj from the assumptions ≤ (jB + x + 1) + x * j -xj from above ≤ (jB + x + 1) + ( j(j -1) 2 B + j + j x ) by induction = ( j(j + 1) 2 B + (j + 1) x + (j + 1))

As x * 0 = x 0 , the property is true for j = 0.

B ADDITIONAL DETAILS ON IMAGENET MODELS AND PERFORMANCE:

For ImageNet we report the improvement in accuracy obtained by adding layers in Figure 3 as seen by the auxiliary problem solutions. We observe that indeed the accuracy of the model on both the training and validation is able to improve from adding layers as discussed in depth in Section 4.2. We observe that k = 1 also over-fits substantially.

We provide a more explicit view of the network sizes in Table 3 and Table 4 . We also show the number of parameters in the ImageNet networks in Table 6 . Although some of the models are not as parameter efficient compared to the related ones in the literature, this was not a primary aim of the investigation in our experiments and thus we did not optimize the models for parameter efficiency (except explicitly at the end of Sec. 4.3), choosing our construction scheme for simplicity. We highlight that this is not a fundamental problem in two ways: (a) for the k = 1 model we note that removing the last two layers reduces the size by 1/4, while the top 5 accuracy at the earlier J=6 layer is 78.8 (versus 79.7), see Figure 3 for detailed accuracies. (b) Our models for k = 2, 3 have most of their parameters in the auxiliary network which is easy to correct for once care is applied to this specific point as at the end of Sec. 4.3. The final model we use to compare greedy layerwise training to end-to-end training, k = 3, M f = 512, is actually more parameter efficient than those in the VGG family while having similar performance. We also point out that we use for simplicity the VGG style 

C ADDITIONAL STUDIES:

We report additional studies that elucidate the critical components of the system and demonstrate the transferability properties of the greedily learned features.

C.1 CHOICE OF DOWNSAMPLING:

In our experiments we use primarily the invertible downsampling operator as the downsampling operation. This choice is to reduce architectural elements which may be inherently lossy such as average pooling. Compared to maxpooling operations it also helps to maintain the network in Sec. 4.1 as a pure ReLU network, which may aid in analysis as the maxpooling introduces an additional non-linearity. We show here the effects of using alternative downsampling approaches including: average pooling, maxpooling and strided convolution. On the CIFAR dataset in the setting of k = 1 we find that they ultimately lead to very similar results with invertible downsampling being slightly better. This shows the method is rather general. In our experiments we follow the same setting described for CIFAR. The setting here uses J = 5 and downsamplings at j = 1, 3. The size is always halved in all cases and the downsampling operation and the output sizes of all networks are the same. Specifically the Average Pooling and Max Pooling use 2 × 2 kernels and the strided convolution simply modifies the 3 × 3 convolutions in use to have a stride of 2. Results are shown in Table 5 .

C.2 EFFECT OF WIDTH:

We report here an additional view of the aggregated results for linear separability discussed in Sec. 4.2. We observe that the trend of the aggregated diagram is similar when comparing only same sized models, with the primary differences in model sizes being increased accuracy.

We perform an ablation to demonstrate the effect of width on the imagenet dataset. We show results for the model k = 3 from Sec. 4.3 with all layer sizes halved. Results are reported for all auxiliary models in Table 7 , we note our results are consistent with similar studies of width from Zagoruyko & Komodakis (2016) showing there is some gains from wider layers (e.g. 2% top 5) but emphasizing that the effect of the cascade is completely critical to obtaining high performance.

:

Accuracies are reported from the tables in http://torch.ch/blog/2015/07/30/cifar.html and https://pytorch.org/docs/master/torchvision/models.html. Our code is based on the same software package.

Figures and Tables:

Figure 2: (Left) Linear and (Right) CNN-p separability as a function of depth for CIFAR-10 models.For Linear separability we aggregate across M = 64, 128, 256, individual results are shown in Appendix C.2, the relative trends are largely unchanged, although overall accuracies are higher in larger M . For CNN-p probes, all models achieve 100% train accuracy at the first or 2nd layer, thus only test accuracy is reported.

Layerwise k-hidden Layer Training k = 1 train acc. k = 1 val acc. k = 2 train acc. k = 2 val acc. k = 3 train acc. k = 3 val acc.

Figure4: Linear separability of differently trained sequential models. We show how the data varies for the different M , observing similar trends to the aggregated data.



Table content: <table><tr><td>CNN</td><td>3x3 Conv.</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Layer</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>CNN Layer</td><td>Linear</td><td>CNN Layer</td><td>Avg.</td><td>Linear</td><td>CNN Layer</td><td>Avg.</td><td>Linear</td></tr><tr><td>Input</td><td>CNN</td><td/><td>CNN</td><td/><td/><td>CNN</td><td/></tr><tr><td>Image</td><td>Layer</td><td/><td>Layer</td><td/><td/><td>Layer</td><td/></tr></table>

). Moreover,Arora et al. (2018);Ge et al. (2017);Du & Goel (2018);Bach (2014) proposed provable optimization strategies for this case. Thus the analysis and optimization of the 1-hidden layer problem is a case that is relatively well understood compared to deep counterparts. At the same time, as shown in Prop. 3.2, applying an existing optimization strategy could give us a bound on the solution of the overall objective of Alg. 1. This shows that our training strategy can be more amenable to the development of provable learning for deep CNNs, while our experiment will show it can still yield high performance networks.

Table content: <table/>

Single crop validation acc. on ImageNet. Our models use J = 8. In parentheses see the ensemble prediction. Mf specifies the auxiliary network for models that have the final auxiliary network replaced. These show minor loss to the original bigger auxiliary.

Table content: <table><tr><td>Layerwise, k = 1 Layerwise k = 2 Layerwise k = 3 Layerwise k = 3, Mf = 1024 Layerwise k = 3, Mf = 512 End-to-End Deep CNN AlexNet VGG-11 VGG-13 VGG-19 Resnet-152</td><td>58.1 (59.3) 65.7 (67.1) 69.7 (71.6) 69.2 68.7 56.5 69.0 69.9 72.9 78.3</td><td>Top-5 (Ens.) 79.7 (80.8) 86.3 (87.0) 88.7 (89.8) 88.6 88.5 79.1 88.6 89.3 90.9 94.1</td><td>Layer-wise Trained Layerwise k = 1 Layerwise k = 2 Layerwise k = 3 BoostResnet. (Huang et al., 2017) ProvableNN (Malach et al., 2018) (Mosca et al., 2017) End-to-End Deep CNN AlexNet VGG 1</td><td>Acc. (Ens.) 88.3 (88.4) 90.4 (90.7) 91.7 (92.8) 82.1 73.4 81.6 89</td></tr><tr><td>End-to-end of k = 3, Mf = 512</td><td>71.5</td><td>90.1</td><td/><td/></tr><tr><td>Alternatives</td><td/><td/><td/><td/></tr><tr><td>Unsup + MLP (Noroozi et al, 2016)</td><td>34.6</td><td>N/A</td><td/><td/></tr><tr><td>FV+ MLP (Perronin et al., 2015)</td><td>55.6</td><td>78.4</td><td/><td/></tr><tr><td>FV + SVM (Sánchez et al., 2013)</td><td>54.3</td><td>74.3</td><td/><td/></tr><tr><td colspan="3">Layer-wise models are competitive with benchmarks</td><td/><td/></tr><tr><td colspan="3">that similarly don't use skip connections and outperform</td><td/><td/></tr><tr><td>all other alternatives to end-to-end.</td><td/><td/><td/><td/></tr></table>

Figure3: Intermediate Accuracies of models in Sec. 4.3. We note that the k = 1 model is larger than the k = 2, 3 models. Network structure for k = 2, 3 imagenet models, not including auxiliary networks. Note an invertible downsampling is applied on the input 224x224x3 image to producie the initial input. The default auxillary networks for both have Mf = 2048 with 1 and 2 auxillary layers, respectively. Note auxiliary networks always reduce the spatial resolution to 2x2 before the final linear layer. construction involving only 3x3 convolutions and downsampling operations that only half the spatial resolution, which indeed has been shown to lead to relatively less parameter efficient architecturesHe et al. (2016), using less uniform construction (larger filters and bigger pooling early on) can yield more parameter efficient models.

Table content: <table><tr><td colspan="3">Layer spatial size layer output size</td></tr><tr><td colspan="2">Input 112 × 112</td><td>12</td></tr><tr><td>1</td><td>112 × 112</td><td>128</td></tr><tr><td>2</td><td>112 × 112</td><td>128</td></tr><tr><td>3</td><td>56 × 56</td><td>256</td></tr><tr><td>4</td><td>56 × 56</td><td>256</td></tr><tr><td>5</td><td>28 × 28</td><td>512</td></tr><tr><td>6</td><td>28 × 28</td><td>512</td></tr><tr><td>7</td><td>14 × 14</td><td>1024</td></tr><tr><td>8</td><td>14 × 14</td><td>1024</td></tr></table>

Network structure for k = 1 ImageNet models, not including auxiliary networks. Note an invertible down-sampling is applied on the input 224x224x3 image to produce the initial input. Note this network does not include any batch-norm.

Table content: <table><tr><td colspan="4">Layer spatial size layer output size</td></tr><tr><td colspan="2">Input 112 × 112</td><td/><td>12</td></tr><tr><td>1</td><td>112 × 112</td><td/><td>256</td></tr><tr><td>2</td><td>112 × 112</td><td/><td>256</td></tr><tr><td>3</td><td>56 × 56</td><td/><td>512</td></tr><tr><td>4</td><td>28 × 28</td><td/><td>1024</td></tr><tr><td>5</td><td>14 × 14</td><td/><td>2048</td></tr><tr><td>6</td><td>14 × 14</td><td/><td>2048</td></tr><tr><td>7</td><td>7 × 7</td><td/><td>4096</td></tr><tr><td>8</td><td>7 × 7</td><td/><td>4096</td></tr><tr><td/><td colspan="2">Layer-wise Trained</td><td>Acc. (Ens.)</td></tr><tr><td/><td colspan="2">Strided Convolution</td><td>87.8</td></tr><tr><td/><td>Invertible Down</td><td/><td>88.3</td></tr><tr><td/><td>AvgPool</td><td/><td>87.6</td></tr><tr><td/><td>MaxPool</td><td/><td>88.0</td></tr><tr><td colspan="4">Table 5: Comparison of different downsampling operations</td></tr><tr><td colspan="2">Models</td><td colspan="2">Number of Parameters</td></tr><tr><td colspan="2">Our model k = 3, M f = 512</td><td/><td>46M</td></tr><tr><td colspan="2">Our model k = 3</td><td/><td>102M</td></tr><tr><td colspan="2">Our model k = 2</td><td/><td>64M</td></tr><tr><td colspan="2">Our model k = 1, J = 8</td><td/><td>406M</td></tr><tr><td colspan="2">Our model k = 1, J = 6</td><td/><td>96M</td></tr><tr><td colspan="2">AlexNet</td><td/><td>60M</td></tr><tr><td colspan="2">VGG-16</td><td/><td>138 M</td></tr></table>

Overall parameter counts for imagenet models trained in Sec. 4 and from literature.

Table content: <table/>

Peer review (iclr2019):
 This paper is of reasonable quality and clarity, rather modest originality, perhaps considerable significance in some applications. Strengths: - I think this kind of method could be useful for data of very high dimensionality, when it is not possible to train everything end to end. - The experiments seem to be conducted correctly. - The paper is well written. Weaknesses: - (minor) Abstract: it*s kind of funny to say that CIFAR-10 is a large scale image recognition problem. - What the authors are proposing is quite similar to Lee et al. [1], which was not mentioned in the paper as well as a wide range of papers, which were mentioned. I think it is kind interesting for people to revise these techniques from 10 years ago, but this method is just not that novel. - The authors highlight that their goal is not using this method as a pre-training strategy, but it would be interesting to see whether it would indeed work better if after the layer-wise training, the whole network would be trained end-to-end. - Maths in this paper is mostly decorative. - When comparing different models or training methods (e.g. layer-wise trained AlexNet and end-to-end trained AlexNet), it would make sense to do some hyperparameter search. It is very risky to conclude anything otherwise. - I would like to see a wall clock time comparison between this and end-to-end training. [1] Lee et al. Deeply-Supervised Nets. 2015.