Title: EXPANDNETS: LINEAR OVER-PARAMETERIZATION TO TRAIN COMPACT CONVOLUTIONAL NETWORKS

Abstract: In this paper, we introduce a novel approach to training a given compact network.
To this end, we build upon over-parameterization, which typically improves both optimization and generalization in neural network training, while being unnecessary at inference time. We propose to expand each linear layer of the compact network into multiple linear layers, without adding any nonlinearity. As such, the resulting expanded network can benefit from over-parameterization during training but can be compressed back to the compact one algebraically at inference. As evidenced by our experiments, this consistently outperforms training the compact network from scratch and knowledge distillation using a teacher. In this context, we introduce several expansion strategies, together with an initialization scheme, and demonstrate the benefits of our ExpandNets on several tasks, including image classification, object detection, and semantic segmentation.
Under review as a conference paper at ICLR 2020 deep networks. Here, by contrast, we focus on practical, nonlinear, compact convolutional neural networks, and investigate the use of linear expansion to introduce over-parameterization and improve training, so as to allow such networks to achieve better performance.

1. INTRODUCTION:

With the growing availability of large-scale datasets and advanced computational resources, deep learning has achieved tremendous success in a variety of computer vision tasks, such as image classification (Krizhevsky et al., 2012; He et al., 2016) , object detection (Ren et al., 2015; Redmon & Farhadi, 2018; 2016) and semantic segmentation (Long et al., 2015; Ronneberger et al., 2015) . Over the past few years, "Wider and deeper are better" has become the rule of thumb to design network architectures (Simonyan & Zisserman, 2015; Szegedy et al., 2015; He et al., 2016; Huang et al., 2017) . This trend, however, raises memory-and computation-related challenges, especially in the context of constrained environments, such as embedded platforms.

Deep and wide networks are well-known to be heavily over-parameterized, and thus a compact network, both shallow and thin, should often be sufficient. Unfortunately, compact networks are notoriously hard to train from scratch. As a consequence, designing strategies to train a given compact network has become increasingly popular, the most popular approach consisting of transferring the knowledge of a deep teacher network to the compact one of interest (Hinton et al., 2015; Romero et al., 2014; Yim et al., 2017a; Zagoruyko & Komodakis, 2017; Passalis & Tefas, 2018) .

In this paper, we introduce an alternative approach to training compact neural networks, complementary to knowledge transfer. To this end, building upon the observation that network overparameterization improves both optimization and generalization (Arora et al., 2018; Zhang et al., 2017; Reed, 1993; Allen-Zhu et al., 2018a; b; Kenji Kawaguchi & Bengio, 2018) , we propose to increase the number of parameters of a given compact network by incorporating additional layers. However, instead of separating every two layers with a nonlinearity, as in most of the deep learning literature, we advocate introducing consecutive linear layers. In other words, we expand each linear layer of a compact network into a succession of multiple linear layers, without any nonlinearity in between. Since consecutive linear layers are equivalent to a single one (Saxe et al., 2014) , such an expanded network, or ExpandNet, can be algebraically compressed back to the original compact one without any information loss.

While the use of successive linear layers appears in the literature, existing work (Baldi & Hornik, 1989; Saxe et al., 2014; Kawaguchi, 2016; Laurent & von Brecht, 2018; Zhou & Liang, 2018; Arora et al., 2018) has been mostly confined to networks without any nonlinearities and the theoretical study of their behavior under statistical assumptions that typically do not hold in practice. In particular, these studies have attempted to understand the learning dynamics and the loss landscapes of Figure 1 : ExpandNets. We study 3 strategies to linearly expand a compact network. The resulting expanded network can be compressed back to the compact one algebraically, and consistently outperforms training the compact one from scratch and with knowledge distillation. Specifically, as illustrated by Figure 1 , we introduce three ways to expand a compact network: (i) replacing a fully-connected layer with multiple ones; (ii) replacing a k × k convolution by three convolutional layers with kernel size 1×1, k × k and 1 × 1, respectively; and (iii) replacing a k × k convolution with k > 3 by multiple 3 × 3 convolutions. Our experiments demonstrate that expanding convolutions, which has not been done before, is key to obtaining more effective compact networks.

Furthermore, we introduce a natural initialization strategy for our ExpandNets. Specifically, an ExpandNet has a nonlinear counterpart, with an identical number of parameters, obtained by adding nonlinearity between every two consecutive linear layers. We therefore propose to directly transfer the weights of this nonlinear model to the ExpandNet for initialization. We show that this yields a further performance boost for the resulting compact network.

In short, our contributions are (i) a novel approach to training given compact, nonlinear convolutional networks by expanding their linear layers; (ii) two strategies to expand convolutional layers; and (iii) an effective initialization scheme for the resulting ExpandNets. We demonstrate the benefits of our approach on several tasks, including image classification on ImageNet, object detection on PASCAL VOC and image segmentation on Cityscapes. Our ExpandNets consistently outperform training the corresponding compact networks from scratch and using knowledge distillation. We further analyze the benefits of linear over-parametrization on training via a series of experiments studying generalization, gradient confusion and the loss landscapes. We will make our code publicly available.

2. EXPANDNETS:

Let us now introduce our approach to training compact networks by expanding their linear layers.

In particular, we propose to make use of linear expansion, such that the resulting ExpandNet is equivalent to the original compact network and can be compressed back to the original structure algebraically. Below, we describe three different expansion strategies, starting with the simplest case of fully-connected layers, followed by two ways to expand convolutional layers.

Expanding fully-connected layers. The weights of a fully-connected layer can easily be represented in matrix form. Therefore, expanding such layers can be done in a straightforward manner by relying on matrix product. Specifically, let W n×m be the parameter matrix of a fully-connected layer with m input channels and n output ones. That is, given an input vector x ∈ R m , the output y ∈ R n can be obtained as y = W n×m x. Note that we ignore the bias, which can be taken into account by incorporating an additional channel with value 1 to x. Expanding such a fully-connected layer with an arbitrary number of linear layers can be achieved by observing that its parameter matrix can be equivalently written as

W n×m = W n×p l × W p l ×p l-1 × • • • × W p1×m .

(1)

While this yields an over-parametrization, it does not affect the underlying model capacity. More importantly, this allows us to increase not only the number of layers, but also the number of channels by setting p i > n, m for all i. To this end, we rely on the notion of expansion rate. Specifically, for an expansion rate r, we define p 1 = rm and p i = rn, ∀i = 1. Note that other strategies are possible, e.g., p i = r i m, but ours has the advantage of preventing the number of parameters from exploding. In practice, considering the computational complexity of fully-connected layers, we advocate expanding each layer into only two or three layers with a small expansion rate.

Expanding convolutional layers. The operation performed by a convolutional layer can also be expressed in matrix form, by vectorizing the input tensor and defining a highly structured matrix whose elements are obtained from the vectorized convolutional filters. While this representation could therefore allow us to use the same strategy as with fully-connected layers, using arbitrary intermediate matrices would ignore the convolution structure, and thus alter the original operation performed by the layer. For a similar reason, one cannot simply expand a convolutional layer with kernel size k × k with a series of k × k convolutions, since, unless k = 1, the resulting receptive field size would differ from the original one.

To overcome this, we note that 1 × 1 convolutions retain the computational benefits of convolutional layers while not modifying the receptive field size. As illustrated in Figure 1 , we therefore propose to expand a k × k convolutional layer into 3 consecutive convolutional layers: a 1 × 1 convolution; a k × k one; and another 1 × 1 one. This strategy still allows us to increase the number of channels in the intermediate layer. Specifically, for an original layer with m input channels and n output ones, given an expansion rate r, we define the number of output channels of the first 1 × 1 layer as p = rm and the number of output channels of the intermediate k × k layer as q = rn.

Compressing an expanded convolutional layer into the original one can still be done algebraically.

To this end, one can reason with the matrix representation of convolutions. For an input tensor of size (w × h × m), the matrix representation of the original layer can be recovered as

EQUATION

where v = w • h and each intermediate matrix has a structure encoding a convolution. It can be verified that the resulting matrix will also have a convolution structure, with filter size k × k.

Expanding convolutional kernels. While 3 × 3 kernels have become increasingly popular in very deep architectures (He et al., 2016) , larger kernel sizes are often exploited in compact networks, so as to increase their expressiveness and their receptive fields. As illustrated in Figure 1 , k × k kernels with k > 3 can be equivalently represented with a series of l 3 × 3 convolutions, where l = k-1 2 . As before, the number of channels in the intermediate 3 × 3 layers can be larger than that in the original k × k one, thus allowing us to linearly over-parametrize the model. Similarly to the fully-connected case, for an expansion rate r, we set the number of output channels of the first 3×3 layer to p 1 = rm and that of the subsequent layers to p i = rn. The same matrix-based strategy as before can then be used to algebraically compress back the expanded kernels.

Dealing with convolution padding and strides. In modern convolutional networks, padding and strides are widely used to retain information from the input feature map while controlling the size of the output one. To expand a convolutional layer with padding p, we propose to use padding p in the first layer of the expanded unit while not padding the remaining layers. Furthermore, to handle a stride s, when expanding convolutional layers, we set the stride of the middle layer to s and of the others to 1. When expanding convolutional kernels, we use a stride 1 for all layers except for the last one whose stride is set to s. These two strategies guarantee that the resulting ExpandNet can be compressed back to the original compact model without any information loss.

Overall, the strategies discussed above allow us to expand an arbitrary compact network into an equivalent deeper and wider one. Note that these strategies can be used independently or together. In any event, once the resulting ExpandNet is trained, it can be compressed back to the original compact architecture in a purely algebraic manner, that is, at absolutely no loss of information.

2.1. INITIALIZING EXPANDNETS:

As will be demonstrated by our experiments, training an ExpandNet from scratch yields consistently better results than training the original compact network. However, in the context of deep networks, initialization can have an important effect on the final results (Mishkin & Matas, 2015; He et al., 2015) . While designing an initialization strategy specifically for compact networks is an unexplored and challenging research direction, our ExpandNets can be initialized in a very natural manner.

To this end, we exploit the fact that an ExpandNet has a natural nonlinear counterpart, which can be obtained by incorporating a nonlinear activation function between each pair of linear layers. We therefore propose to initialize the parameters of an ExpandNet by simply training its nonlinear counterpart and transferring the resulting parameters to the ExpandNet. The initialized ExpandNet is then trained in the standard manner. As evidenced by our experiments below, when the nonlinear counterpart achieves better performance than the compact network, this simple strategy typically yields an additional accuracy boost to our approach.

3. EXPERIMENTS:

In this section, we demonstrate the benefits of our ExpandNets on several tasks, including image classification, object detection, and semantic segmentation. We then analyze the training behavior of our different expansion strategies.

We denote the expansion of fully-connected layers by FC, of convolutional layers by CL, and of convolutional kernels by CK. When combining convolutional expansions with fully-connected ones, we use CL+FC or CK+FC, and add +Init to indicate the use of our initialization strategy.

3.1. IMAGE CLASSIFICATION:

We first study the use of our approach with very small networks on CIFAR-10 and CIFAR-100, and then turn to the more challenging ImageNet dataset, where we show that our method can improve the results of the compact MobileNet (Howard et al., 2017) , MobileNetV2 (Sandler et al., 2018) and ShuffleNetV2 0.5× (Ma et al., 2018) architectures.

3.1.1. CIFAR-10 AND CIFAR-100:

Experimental setup. For CIFAR-10 and CIFAR-100, we use the same compact network as in (Passalis & Tefas, 2018) . It is composed of 3 convolutional layers with 3 × 3 kernels and no padding. These 3 layers have 8, 16 and 32 output channels, respectively. Each of them is followed by a batch normalization layer, a ReLU layer and a 2 × 2 max pooling layer. The output of the last layer is passed through a fully-connected layer with 64 units, followed by a logit layer with either 10 or 100 units. To evaluate our kernel expansion method, we also report results obtained with a similar network where the 3 × 3 kernels were replaced by 7 × 7 ones, with a padding of 3. All networks, including our ExpandNets, were trained for 150 epochs using a batch size of 128. We used standard stochastic gradient descent (SGD) with a momentum of 0.9 and a learning rate of 0.01, divided by 10 at epochs 50 and 100. With this strategy, all networks reached convergence. In this set of experiments, the expand rate r is always set to 4 to balance the accuracy-efficiency trade-off. We evaluate the influence of this parameter in Appendix A.3. Note that our ExpandNet strategy is complementary to knowledge transfer; that is, we can apply any typical knowledge transfer method using our ExpandNet as student instead of the compact network directly. To demonstrate the benefits of ExpandNets in this scenario, we conduct experiments using knowledge distillation (KD) (Hinton et al., 2015) , hint-based transfer (Hint) (Romero et al., 2014) or probabilistic knowledge transfer (PKT) (Passalis & Tefas, 2018 ) using a ResNet18 teacher network.

Results. We focus here on the SmallNet with 7 × 7 kernels, for which we can evaluate all our expansion strategies, including the CK ones, and report the results obtained with the model with 3×3 kernels in Table 6 in Appendix A.1. The results of our all networks with and without knowledge distillation1 are provided in Table 1 . As shown in the top portion of the table, only expanding the fully-connected layer yields mild improvement. However, expanding the convolutional ones clearly outperforms over the compact network, and is further boosted by expanding the fully-connected one and by using our initialization scheme. Note that expanding the kernels yields higher accuracy, with ExpandNet-CK+FC+Init achieving the best results. Note that even without KD, our ExpandNets outperform the SmallNet with KD. The gap is further increased when we also use KD, as can be seen in the bottom portion of the table. 

3.1.2. IMAGENET:

We now turn to the more challenging case of ImageNet (Russakovsky et al., 2015) . In this case, for MobileNets, we perform data augmentation by taking random image crops and resizing them to 224 × 224 pixels. We further perform random horizontal flips and subtract the per-pixel mean.

Experimental setup. For this set of experiments, we make use of the compact MobileNet (Howard et al., 2017), MobileNetV2 (Sandler et al., 2018) and ShuffleNetV2 (Ma et al., 2018 ) models, which were designed to be compact and yet achieve good results. We rely on a pytorch implementation of these models. For our approach, we use our CL strategy to expand all convolutional layers with kernel size of 3 × 3 in MobileNet and ShuffleNet, while only expanding the non-residual 3 × 3 convolutional layers in MobileNetV2. We trained the MobileNets in 2 regimes: Short-term (as for ResNets (He et al., 2016) ), where we trained for 90 epochs with a weight decay of 0.0001 and an initial learning rate of 0.1, divided by 10 every 30 epochs; Long-term, where we trained for 400 epochs with a weight decay of 0.00004 and an initial learning rate of 0.01, divided by 10 at epochs 200 and 300. In all cases, we employed SGD with a momentum of 0.9 and a batch size of 256. For ShuffleNet, we use of a small ShuffleNetV2 0.5×, trained in the same manner as in (Ma et al., 2018) . We also perform KD from a ResNet152 (78.32% top-1 accuracy), tuning the KD hyper-parameters to the best accuracy for each method.

Results. We compare the results of the original models with those of our expanded versions in Table 2 . Our expansion strategy yields a huge improvement, especially in the short-term regime, increasing the top-1 accuracy of MobileNet, MobileNetV2 and ShuffleNet by 2.92, 1.87 and 0.49 percentage points (pp). This short-term regime could be valuable to work with limited access to resources. In the long-term regime, our expansion strategy nonetheless still achieves an improvement. Furthermore, our vanilla ExpandNets outperform the compact models with KD, even though we do not require a teacher. When also using KD, ExpandNets can achieve even better performance. Note that, on this dataset, their nonlinear counterparts do not outperform original models.

3.2. OBJECT DETECTION:

Our approach is not restricted to image classification. We first demonstrate its benefits in the context of one-stage object detection using the PASCAL VOC dataset (Everingham et al., a;b) . To this end, we trained networks on the PASCAL VOC2007 + 2012 training and validation sets and report the mean average precision (mAP) on the PASCAL VOC2007 test set. Experimental setup. We make use of YOLO-LITE (Huang et al., 2018) , which was designed to work in constrained environments. It consists of a backbone part and a head part, and we only expand the convolutional layers in the backbone. YOLO-LITE is a very compact network, with only 5 convolutional layers in the backbone, each followed by a batch normalization layer, a leaky-ReLU layer and a 2 × 2 max pooling layer. We expand all 5 convolutional layers using our CL strategy with r = 4. We trained all networks in the standard YOLO fashion (Redmon & Farhadi, 2018; 2016) .

Results. The results on the PASCAL VOC 2007 test set are reported in Table 3 . They show that, as for image recognition, our expansion strategy yields better performance. Since YOLO-LITE is very compact, our initialization scheme boosts the performance by over 7pp.

3.3. SEMANTIC SEGMENTATION:

We then demonstrate the benefits of our approach on the task of semantic segmentation using the Cityscapes dataset (Cordts et al., 2016) . Cityscapes is a large-scale dataset containing 5000 images of 1024 × 2048. Following the standard protocol, we report the mean Intersection over Union (mIoU), mean recall (mRec) and mean precision (mPrec). Experimental setup. For this experiment, we rely on the U-Net (Ronneberger et al., 2015) , which is a relatively compact network consisting of a contracting path and an expansive path. The contracting path follows the typical architecture of a convolutional network. It consists of blocks of two 3 × 3 convolutions, each followed by a ReLU, and a 2 × 2 max pooling. The number of channels in each block is 32, 64, 128, 256, 512, respectively. We apply our CL expansion strategy with r = 4 to all convolutions in the contracting path. We train the networks on 4 GPUs using the standard SGD optimizer with a momentum of 0.9 and a learning rate of 1e -8.

Results. Our results on the Cityscapes validation set are shown in Table 4 . Note that our ExpandNet also outperforms the original compact U-Net on this task.

3.4. ANALYSIS OF OUR APPROACH:

To further analyze our approach, we study its behavior during training and its generalization ability. For these experiments, we make use of the CIFAR-10 and CIFAR-100 datasets, and use the same settings as in Appendix A.3. In Appendix A.4, we also showcase the use of our approach with the larger AlexNet architecture on ImageNet. Finally, we evaluate the complexity of the models in terms of number of parameters, FLOPs, and training and testing inference speed in Appendix B. Since our ExpandNets can be compressed back to the original networks, at test time, they have exactly the same number of parameters, FLOPS, and inference time, but achieve higher accuracy.

3.4.1. TRAINING BEHAVIOR:

To investigate the benefits of linear over-parameterization on training, we make use of the gradient confusion introduced by (Sankararaman et al., 2019) to show that the gradients of nonlinearly over-parameterized networks were more consistent across mini-batches. Specifically, following (Sankararaman et al., 2019) , we measure gradient confusion (or rather consistency) as the minimum cosine similarity from 100 randomly-sampled pairs of mini-batches at the end of each training epoch. As in (Sankararaman et al., 2019) , we also combine the gradient cosine similarity of 100 pairs of sampled mini-batches at the end of training from each independent run and perform Gaussian kernel density estimation on this data. While, during training, we aim for the minimum cosine similarity to be high, at the end of training, the resulting distribution should be peaked around zero.

We run each experiment 5 times and show the average values across all runs in Figure 2 . From the training and test curves, we observe that our ExpandNets-CL/CK speed up convergence and yield a smaller generalization error gap. They also yield lower gradient confusion (higher minimum cosine similarity) and a more peaked density of pairwise gradient cosine similarity. This indicates that our ExpandNets-CL/CK are easier to train than the compact model. By contrast, only expanding the fully-connected layers does not facilitate training. Additional plots are provided in Appendix C.

3.4.2. GENERALIZATION ABILITY:

We then analyse the generalization ability of our approach. To this end, we first study the loss landscapes using the method in (Li et al., 2018) . As shown in Figure 3 , our ExpandNets, particularly with convolutional expansion, produce flatter minima, which, as discussed in (Li et al., 2018) , indicates better generalization. Figure 3 : Loss landscapes of networks with 9 × 9 kernels on CIFAR-10. Table 5 : Generalization ability on Corrupted CIFAR-10. We report the top-1 error (%). Note that our ExpandNets yield larger generalization gaps than the compact network in almost all cases involving convolutional expansion. By contrast expanding FC layers often does not help. As a second study of generalization, we evaluate the memorization ability of our ExpandNets on corrupted datasets, as suggested in (Zhang et al., 2017) . To this end, we utilize the open-source implementation of (Zhang et al., 2017) to generate three CIFAR-10 and CIFAR-100 training sets, containing 20%, 50% and 80% of random labels, respectively, while the test set remains clean.

In Table 5 (and Tables 10, 11 and 12 in Appendix A.5), we report the top-1 test errors of the best and last models, as well as the training errors of the last model. These results evidence that expanding convolutional layers and kernels typically yields lower test errors and higher training ones, which implies that our better results in the other experiments are not due to simply memorizing the datasets, but truly to better generalization ability.

4. RELATED WORK:

Very deep neural networks currently constitute the state of the art for many computer vision tasks. These networks, however, are known to be heavily over-parameterized, and making them smaller would facilitate their use in resource-constrained environments, such as embedded platforms. As a consequence, much research has recently been devoted to developing more compact architectures.

Network compression constitutes one of the most popular trends in this area. In essence, it aims to reduce the size of a pre-trained very deep network while losing as little accuracy as possible, or even none at all. In this context, existing approaches can be roughly grouped into two categories: (i) parameter pruning and sharing (LeCun et al., 1990; Han et al., 2015; Courbariaux et al., 2016; M Figurnov, 2016; Pavlo Molchanov, 2017; Ullrich et al., 2017; Carreira-Perpin & Idelbayev, 2018) , which aims to remove the least informative parameters; and (ii) low-rank factorization (Denil et al., 2013; Sainath et al., 2013; Lebedev et al., 2014; Jin et al., 2015; Liu et al., 2015) , which uses decomposition techniques to reduce the size of the parameter matrix/tensor in each layer. While compression is typically performed as a post-processing step, it has been shown that incorporating it during training could be beneficial (Alvarez & Salzmann, 2016; 2017; Wen et al., 2016; 2017) . In any event, even though compression reduces a network's size, it does not provide one with the flexibility of designing a network with a specific architecture and of a specific size. Furthermore, it often produces networks that are much larger than the ones we consider here, e.g., compressed networks with O(1M) parameters vs O(10K) for the SmallNets used in our experiments.

In a parallel line of research, several works have proposed design strategies to reduce a network's number of parameters (Wu et al., 2016; Szegedy et al., 2016; Howard et al., 2017; Romera et al., 2018; Sandler et al., 2018) . Again, while more compact networks can indeed be developed with these mechanisms, they impose constraints on the network architecture, and thus do not allow one to simply train a given compact network. Furthermore, as shown by our experiments, our approach is complementary to these works. For example, we can improve the results of MobileNetV2 (Sandler et al., 2018) by training it using our expansion strategy.

Here, in contrast to the above-mentioned literature on compact networks, we seek to train a given compact network with an arbitrary architecture. This is also the task addressed by knowledge transfer approaches. To achieve this, existing methods leverage the availability of a pre-trained very deep teacher network. In (Hinton et al., 2015) , this is done by defining soft labels from the logits of the teacher; in (Romero et al., 2014) , (Zagoruyko & Komodakis, 2017) and (Yim et al., 2017b ) by transferring intermediate representations, attention maps and Gram matrices, respectively, from the teacher to the compact network; in (Passalis & Tefas, 2018) by aligning the feature distributions of the deep and compact networks.

In this paper, we introduce an alternative strategy to train compact networks, complementary to knowledge transfer. Inspired by the theory showing that over-parameterization helps training (Arora et al., 2018; Zhang et al., 2017; Reed, 1993; Allen-Zhu et al., 2018a; b) , we propose to expand each linear layer in a given compact network into a succession of multiple linear layers. Our experiments evidence that training such expanded networks, which can then be compressed back algebraically, yields better results than training the original compact networks, thus empirically confirming the benefits of over-parameterization. Our results also show that our approach outperforms knowledge transfer, even when not using a teacher network.

Note that linearly over-parametrized neural networks have been investigated both in the early neural network days (Baldi & Hornik, 1989 ) and more recently (Saxe et al., 2014; Kawaguchi, 2016; Laurent & von Brecht, 2018; Zhou & Liang, 2018; Arora et al., 2018) . These methods, however, typically study purely linear networks, with a focus on the convergence behavior of training in this linear regime. For example, (Arora et al., 2018) showed that linear over-parametrization modifies the gradient updates in a unique way that speeds up convergence. Here, we propose to exploit the benefits of linear over-parametrization to improve the training of compact networks. To this end, in contrast to the above-mentioned methods which all focus on fully-connected layers, we develop two strategies to expand convolutional layers, as well as an approach to initializing our expanded networks, and empirically demonstrate their critical impact on prediction accuracy.

5. CONCLUSION:

We have introduced an approach to training a given compact network from scratch by exploiting over-parametrization. Specifically, we have shown that over-parametrizing the network linearly facilitates the training of compact networks. Since the resulting networks are equivalent to the original compact ones, they can be compressed back at no loss of information. Our experiments have demonstrated that our ExpandNets consistently outperform the original compact networks.

In particular, when applicable, our CK expansion strategy tends to yield the best results. When it doesn't, that is, for a kernel size of 3, the CL one nonetheless remains highly effective. Our technique is general and can also be used in conjunction with knowledge transfer approaches. Furthermore, our initialization scheme yields a consistent accuracy boost. This strategy, however, is not the only possible one. In the future, we will therefore aim to develop other initialization schemes for our ExpandNets, and generally for compact networks.

Table 8 : Small networks vs ExpandNets on CIFAR-10 (Top) and CIFAR-100 (Bottom). We report the top-1 accuracy for the original compact networks and for different versions of our approach. Note that our ExpandNets yield higher accuracy than the compact network in almost all cases involving expanding convolutions cases. By contrast expanding FC layers does often not help. extra convolutional layer with 64 channels, followed by a batch normalization layer, a ReLU layer and a 2 × 2 max pooling layer. We use SGD with a momentum of 0.9 and a weight decay of 0.0005 for 150 epochs. The initial learning rate was 0.01, divided by 10 at epoch 50 and 100. Furthermore, we used zero-padding to keep the size of the input and output feature maps of each convolutional layer unchanged. We use the same networks in our approach analysis experiments.

The results of these experiments are provided in Table 8 . We observe that our different strategies to expand convolutional layers outperform the compact network in almost all cases, while only expanding fully-connected layers doesn't work well. In particular, for kernel sizes k > 3, ExpandNet-CK yields consistently higher accuracy than the corresponding compact network, independently of the expansion rate. For k = 3, where ExpandNet-CK is not applicable, ExpandNet-CL comes as an effective alternative, also consistently outperforming the baseline.

A.4 WORKING WITH LARGER NETWORKS:

We also evaluate the use of our approach with a larger network. To this end, we make use of AlexNet (Krizhevsky et al., 2012) on ImageNet. AlexNet relies on kernels of size 11 and 5 in its first two convolutional layers, which makes our CK expansion strategy applicable.

Table 9 : Top-1 accuracy (%) of AlexNet vs Ex-pandNets with r = 4 on the ILSVRC2012 validation set for different number of channels in the last convolutional layer. Note that, while our expansion strategy always helps, its benefits decrease as the original model grows. We use a modified, more compact version of AlexNet, where we replace the first fullyconnected layer with a global average pooling layer, followed by a 1000-class fully-connected layer with softmax. To evaluate the impact of the network size, we explore the use of different dimensions, [128, 256, 512] , for the final convolutional features. We trained the resulting AlexNets and corresponding ExpandNets using the short-term regime of our MobileNets experiments in Section 3.

As shown in Table 9 , while our approach outperforms the original AlexNet for all feature dimensions, the benefits decrease as the feature dimension increases. This indicates that our approach is better suited for truly compact networks, and developing similar strategies for deeper ones will be the focus of our future research.

A.5 GENERALIZATION ABILITY ON CORRUPTED CIFAR-10 AND CIFAR-100

Our experiments on corrupted datasets implies better generalization. Here, we provide more experimental results on Corrupted CIFAR-10 (in Table 10 ) and CIFAR-100 (in Table 11 and 12 ) by using different networks with kernel sizes of 3, 5, 7, 9, respectively. Our method consistently improves the generalization error gap across all kernel sizes and corruption rates (20%, 50%, 80%) and yields from around 1pp to over 6pp error drop in testing.

B COMPLEXITY ANALYSIS:

Here, we compare the network complexity of our method and the original networks in terms of number of parameters, number of FLOPs and GPU inference speed.

In Table 13 , we provide numbers for both training and testing. During training, because our approach introduces more parameters, inference is 2 to 5 times slower than in the original network for an expansion rate of 4. However, since our ExpandNets can be compressed back to the original network, at test time, they have exactly the same number of parameters and FLOPS, and inference time, but our networks achieve higher accuracy.

C ADDITIONAL VISUALIZATIONS:

Here, we provide additional visualizations for the training behavior and the loss landscapes of Section 3.4, corresponding to networks with kernel sizes of 3, 5, 7, 9, respectively.

We plot the loss landscapes of SmallNets and corresponding ExpandNets on CIFAR-10 in Figure 4 , and analyze the training behavior on CIFAR-10 in Figure 5 and on CIFAR-100 in Figure 6 , respectively. These plots further confirm that in almost all cases, our convolution expansion strategies (CL and CK) facilitate training (with lower gradient confusion and more 0-concentrated gradient cosine similarity density) and yield better generalization ability (with flatter minima). 

:

Having found KD to be the most effective knowledge transfer strategy,

Figures and Tables:

Figure 2: Training behavior of networks with 7 × 7 kernels on CIFAR-10. Left: Training and test curves over 150 epochs. Middle: Minimum pairwise gradient cosine similarity at the end of each training epoch. Right: Kernel density estimation of pairwise gradient cosine similarity at the end of training (over 5 independent runs).

Figure 4: Loss landscape plots on CIFAR-10 (We report top-1 error (%)).

Figure 5: Training behavior of networks on CIFAR-10. Left: Training and test curves over 150 epochs. Middle: Minimum pairwise gradient cosine similarity at the end of each training epoch. Right: Kernel density estimation of pairwise gradient cosine similarity at the end of training (over 5 independent runs).

Figure 6: Training behavior of networks on CIFAR-100. Left: Training and test curves over 150 epochs. Middle: Minimum pairwise gradient cosine similarity at the end of each training epoch. Right: Kernel density estimation of pairwise gradient cosine similarity at the end of training (over 5 independent runs).

Top-1 accuracy (%) of SmallNet with 7×7 kernels vs ExpandNets with r = 4 on CIFAR-10 and CIFAR-100.

Table content: <table><tr><td>Network Transfer</td><td>CIFAR-10</td><td>CIFAR-100</td></tr><tr><td colspan="3">SmallNet w/o KD 78.63 ± 0.41 46.63 ± 0.27 SmallNet w/ KD 78.97 ± 0.37 47.04 ± 0.35 ExpandNet-FC w/o KD 78.64 ± 0.39 46.59 ± 0.45 ExpandNet-CL 78.47 ± 0.20 46.90 ± 0.66 ExpandNet-CL+FC 79.11 ± 0.23 46.66 ± 0.43 ExpandNet-CL+FC+Init 79.98 ± 0.28 47.98 ± 0.48 ExpandNet-CK 80.27 ± 0.24 48.55 ± 0.51 ExpandNet-CK+FC 80.31 ± 0.27 48.62 ± 0.47 ExpandNet-CK+FC+Init 80.81 ± 0.27 49.82 ± 0.25 ExpandNet-CL+FC w/ KD 79.60 ± 0.25 47.41 ± 0.51 ExpandNet-CL+FC+Init 80.29 ± 0.25 48.62 ± 0.34 ExpandNet-CK+FC 80.63 ± 0.31 49.13 ± 0.45 ExpandNet-CK+FC+Init 81.21 ± 0.17 50.37 ± 0.39</td></tr></table>

Top-1 accuracy (%) on the ILSVRC2012 validation set (ExpandNets with r = 4).

Table content: <table><tr><td>Model</td><td>ImageNet</td><td>ImageNet</td></tr><tr><td/><td colspan="2">(90 epochs) (400 epochs)</td></tr><tr><td>MobileNet</td><td>66.48</td><td>70.47</td></tr><tr><td>MobileNet (w/ KD)</td><td>69.01</td><td>N/A</td></tr><tr><td>ExpandNet-CL</td><td>69.40</td><td>70.82</td></tr><tr><td>ExpandNet-CL(w/ KD)</td><td>70.47</td><td>N/A</td></tr><tr><td>MobileNetV2</td><td>63.75</td><td>70.67</td></tr><tr><td>MobileNetV2 (w/ KD)</td><td>65.40</td><td>N/A</td></tr><tr><td>ExpandNet-CL</td><td>65.62</td><td>70.70</td></tr><tr><td>ExpandNet-CL (w/ KD)</td><td>67.19</td><td>N/A</td></tr><tr><td>ShuffleNetV2 0.5×</td><td/><td>56.89</td></tr><tr><td>ShuffleNetV2 0.5× (w/ KD)</td><td/><td>57.59</td></tr><tr><td>ExpandNet-CL</td><td/><td>57.38</td></tr><tr><td>ExpandNet-CL (w/ KD)</td><td/><td>57.68</td></tr></table>



Table content: <table><tr><td colspan="2">: mAPs (%) of YOLO-LITE vs Expand-</td></tr><tr><td colspan="2">Nets on the PASCAL VOC2007 test set.</td></tr><tr><td colspan="2">Model VOC2007 Test</td></tr><tr><td>YOLO-LITE</td><td>27.34</td></tr><tr><td>ExpandNet-CL</td><td>30.97</td></tr><tr><td>ExpandNet-CL+Init</td><td>35.14</td></tr></table>

U-Net vs ExpandNets with r = 4 on the Cityscapes validation set.

Table content: <table><tr><td>Model mIoU mRec mPrec</td></tr><tr><td>U-Net 56.59 74.29 65.11</td></tr><tr><td>ExpandNet-CL 57.85 76.53 65.94</td></tr></table>

Generalization ability (top-1 error (%)) on Corrupted CIFAR-10 (kernel size: 3, 7).

Table content: <table><tr><td>Dataset</td><td>Model</td><td/><td/><td/><td>Kernel size k</td><td/><td/></tr><tr><td/><td/><td/><td>3</td><td/><td/><td>7</td><td/></tr><tr><td/><td/><td>Best Test</td><td>Last Test</td><td>Train</td><td>Best Test</td><td>Last Test</td><td>Train</td></tr><tr><td>20% 50% 80%</td><td>SmallNet ExpandNet-FC ExpandNet-CL ExpandNet-CK SmallNet ExpandNet-FC ExpandNet-CL ExpandNet-CK SmallNet ExpandNet-FC ExpandNet-CL ExpandNet-CK</td><td>22.20 ± 0.43 22.40 ± 0.29 21.55 ± 0.27 N/A 25.74 ± 0.25 25.54 ± 0.47 25.48 ± 0.35 N/A 37.49 ± 0.62 37.26 ± 0.16 35.86 ± 0.43 N/A</td><td>22.33 ± 0.40 22.61 ± 0.27 21.71 ± 0.30 N/A 25.94 ± 0.15 25.80 ± 0.41 25.66 ± 0.43 N/A 37.87 ± 0.63 37.63 ± 0.14 36.05 ± 0.44 N/A</td><td>34.85 ± 0.18 35.12 ± 0.07 34.89 ± 0.26 N/A 56.48 ± 0.25 56.37 ± 0.15 56.41 ± 0.33 N/A 77.46 ± 0.16 77.54 ± 0.07 77.56 ± 0.11 N/A</td><td>21.64 ± 0.36 21.92 ± 0.23 21.25 ± 0.41 19.11 ± 0.33 26.99 ± 0.69 26.86 ± 0.46 26.05 ± 0.31 22.43 ± 0.47 39.08 ± 0.41 40.51 ± 0.39 39.40 ± 0.93 32.62 ± 0.28</td><td>21.98 ± 0.42 22.35 ± 0.43 21.54 ± 0.40 19.30 ± 0.35 27.87 ± 0.71 28.23 ± 0.61 26.99 ± 0.15 22.61 ± 0.49 43.33 ± 0.77 44.82 ± 0.62 42.77 ± 0.96 33.86 ± 0.37</td><td>30.42 ± 0.32 30.39 ± 0.19 30.36 ± 0.24 31.14 ± 0.11 53.27 ± 0.21 53.14 ± 0.20 53.21 ± 0.16 53.74 ± 0.16 74.69 ± 0.26 75.38 ± 0.23 75.24 ± 0.22 75.65 ± 0.16</td></tr></table>

Generalization ability (top-1 error (%)) on Corrupted CIFAR-100 (kernel size: 3, 5).

Table content: <table><tr><td>Dataset</td><td>Model</td><td/><td/><td/><td>Kernel size k</td><td/><td/></tr><tr><td/><td/><td/><td>3</td><td/><td/><td>5</td><td/></tr><tr><td/><td/><td>Best Test</td><td>Last Test</td><td>Train</td><td>Best Test</td><td>Last Test</td><td>Train</td></tr><tr><td>20% 50% 80%</td><td>SmallNet ExpandNet-FC ExpandNet-CL ExpandNet-CK SmallNet ExpandNet-FC ExpandNet-CL ExpandNet-CK SmallNet ExpandNet-FC ExpandNet-CL ExpandNet-CK</td><td>55.30 ± 0.42 56.15 ± 0.22 54.85 ± 0.27 N/A 62.54 ± 0.74 63.65 ± 0.50 61.95 ± 0.61 N/A 78.35 ± 0.83 80.36 ± 0.55 79.44 ± 0.72 N/A</td><td>55.48 ± 0.41 56.35 ± 0.23 55.04 ± 0.33 N/A 62.71 ± 0.75 63.88 ± 0.47 62.11 ± 0.59 N/A 78.52 ± 0.86 80.47 ± 0.55 79.66 ± 0.75 N/A</td><td>62.20 ± 0.31 62.60 ± 0.19 61.62 ± 0.43 N/A 78.81 ± 0.39 79.40 ± 0.18 78.78 ± 0.48 N/A 93.78 ± 0.18 94.38 ± 0.12 94.02 ± 0.16 N/A</td><td>53.95 ± 0.33 54.84 ± 0.71 53.50 ± 0.35 51.98 ± 0.28 61.84 ± 0.29 62.99 ± 0.69 61.49 ± 0.39 58.96 ± 0.32 78.59 ± 0.27 80.97 ± 0.51 79.87 ± 0.29 77.22 ± 0.47</td><td>54.16 ± 0.34 55.05 ± 0.76 53.71 ± 0.38 52.10 ± 0.26 62.16 ± 0.21 63.21 ± 0.60 61.70 ± 0.43 59.14 ± 0.41 78.81 ± 0.35 81.15 ± 0.53 80.04 ± 0.29 77.38 ± 0.41</td><td>58.53 ± 0.30 59.47 ± 0.32 58.09 ± 0.31 57.67 ± 0.63 76.78 ± 0.28 77.85 ± 0.31 76.73 ± 0.26 76.24 ± 0.30 93.10 ± 0.12 94.10 ± 0.16 93.59 ± 0.20 93.15 ± 0.25</td></tr></table>

Generalization ability (top-1 error (%)) on Corrupted CIFAR-100 (kernel size: 7, 9).

Table content: <table><tr><td>Dataset</td><td>Model</td><td/><td/><td/><td>Kernel size k</td><td/><td/></tr><tr><td/><td/><td/><td>7</td><td/><td/><td>9</td><td/></tr><tr><td/><td/><td>Best Test</td><td>Last Test</td><td>Train</td><td>Best Test</td><td>Last Test</td><td>Train</td></tr><tr><td>20% 50% 80%</td><td>SmallNet ExpandNet-FC ExpandNet-CL ExpandNet-CK SmallNet ExpandNet-FC ExpandNet-CL ExpandNet-CK SmallNet ExpandNet-FC ExpandNet-CL ExpandNet-CK</td><td>55.36 ± 0.44 56.31 ± 0.78 54.87 ± 0.47 51.24 ± 0.60 63.76 ± 0.59 64.54 ± 0.72 63.36 ± 0.49 58.74 ± 0.25 79.73 ± 0.47 82.97 ± 0.83 80.79 ± 0.54 78.51 ± 0.41</td><td>55.66 ± 0.43 56.58 ± 0.77 55.22 ± 0.55 51.40 ± 0.66 64.08 ± 0.58 64.91 ± 0.55 63.73 ± 0.54 58.98 ± 0.20 79.95 ± 0.36 83.20 ± 0.83 81.09 ± 0.62 78.64 ± 0.36</td><td>56.33 ± 0.51 57.93 ± 0.29 55.52 ± 0.49 56.40 ± 0.21 75.45 ± 0.23 76.75 ± 0.39 75.25 ± 0.45 75.24 ± 0.24 92.58 ± 0.20 94.13 ± 0.34 93.22 ± 0.30 93.24 ± 0.15</td><td>56.59 ± 0.72 57.82 ± 0.23 56.05 ± 0.75 52.36 ± 0.54 64.83 ± 0.41 66.11 ± 0.45 64.36 ± 0.54 60.42 ± 0.86 81.02 ± 0.92 83.42 ± 0.72 81.02 ± 0.44 80.15 ± 0.50</td><td>57.26 ± 0.64 58.07 ± 0.22 56.51 ± 0.76 52.55 ± 0.47 65.63 ± 0.40 66.73 ± 0.41 65.25 ± 0.37 60.65 ± 0.86 81.70 ± 0.97 83.82 ± 0.71 81.58 ± 0.46 80.32 ± 0.55</td><td>55.16 ± 0.32 57.09 ± 0.52 54.99 ± 0.48 57.76 ± 0.50 75.21 ± 0.31 76.44 ± 0.40 74.84 ± 0.28 76.73 ± 0.40 92.54 ± 0.26 93.94 ± 0.35 93.25 ± 0.56 94.04 ± 0.21</td></tr></table>

ExpandNet complexity analysis on CIFAR-10, ImageNet, PASCAL VOC and Cityscapes. Note that, within each task, the metrics are the same for all networks, since we can compress our ExpandNets back to the small network.

Table content: <table><tr><td>Model</td><td colspan="2"># Params(M) Train Test</td><td colspan="2"># FLOPs Train Test</td><td colspan="2">GPU Speed (imgs/sec) Train Test</td></tr><tr><td colspan="2">SmallNet (7 × 7) 0.07 ExpandNet-CL 0.55</td><td/><td>4.49M 57.49M</td><td/><td>147822.51 64651.81</td></tr><tr><td colspan="2">ExpandNet-CL+FC 2.11</td><td>0.07</td><td>59.04M</td><td>4.49M</td><td>61379.95</td><td>154850.52</td></tr><tr><td colspan="2">ExpandNet-CK 0.19</td><td/><td>23.95M</td><td/><td>75065.09</td></tr><tr><td colspan="2">ExpandNet-CK+FC 1.75</td><td/><td>25.5M</td><td/><td>68679.89</td></tr><tr><td colspan="3">MobileNet 4.23 4.23 ExpandNet-CL 4.96</td><td>0.58G 1.76G</td><td>0.58G</td><td>3797.21 729.78</td><td>3829.81</td></tr><tr><td colspan="3">MobileNetV2 3.50 3.50 ExpandNet-CL 3.67</td><td>0.32G 1.34G</td><td>0.32G</td><td>3417.20 1009.25</td><td>3419.43</td></tr><tr><td colspan="3">ShuffleNetV2 0.5× 1.37 1.37 ExpandNet-CL 1.41</td><td>0.04G 0.6G</td><td>0.04G</td><td>5404.06 4228.10</td><td>5434.58</td></tr><tr><td colspan="3">YOLO-LITE 0.57 0.57 ExpandNet-CL 4.48</td><td>1.81G 28.59G</td><td>1.81G</td><td>7.94 6.07</td><td>19.82</td></tr><tr><td colspan="3">U-Net 7.76 7.76 ExpandNet-CL 82.97</td><td colspan="2">389.26G 389.26G 2586.02G</td><td>8.25 2.98</td><td>8.25</td></tr></table>

Peer review (iclr2020):
This paper proposes linear over-parameterization methods to improve training of small neural network models. The idea is simple -- each linear transformation in a network is overparameterized by a series of linear transformation which is algebraically equivalent to the original linear transformation. Number of experiments are conducted to show the effectiveness of the approach. The proposed method is a simple application of over-parameterization to improve neural network model training. The motivation is clear and the proposed method is clearly presented. The paper is easy to understand and follow. Great analyses on the training behavior and generalization ability are conducted. Given the simplicity of the method, this could be a standard way of training small neural network model if the effectiveness of the method is observed more widely. These are some concerns on the paper: 1) The effectiveness of the approach is not necessarily significant in all experiments. For example, in Table 1, ExpandNet-FC and ExpandNet-CL were not effective. The same trend is observed in Table 2 for 400 epochs. Given that only ExpandNet-CK improves the performance, we could conclude that some intrinsic property of CK is important than over-parameterization. The good results for 90 epochs in Table 2 may mean linear over-parameterization yields faster convergence as suggested by Arora et al. 2018. 2) The comparisons are not extensive. For example, we do not see Init for all models and *w/ KD* for ShuffleNetV2 in Table 2. Table 2 has *N/A*. Table 3 and 4 do not have results with *w/ KD*. Knowledge transfer methods should be the baseline of the paper. Minor comments: It will be interesting to see the results of the models used for Init. It might be interesting to conduct experiments with a big model and see if we do not have any gains.