ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks
Authors: Shuxuan Guo
Jose M. Alvarez
Mathieu Salzmann
Venue:  (2020)
Keywords: Compact Network Training
Linear Expansion
Over-parameterization
Knowledge Transfer

Abstract: In this paper, we introduce a novel approach to training a given compact network. To this end, we build upon over-parameterization, which typically improves both optimization and generalization in neural network training, while being unnecessary at inference time. We propose to expand each linear layer of the compact network into multiple linear layers, without adding any nonlinearity. As such, the resulting expanded network can benefit from over-parameterization during training but can be compressed back to the compact one algebraically at inference. As evidenced by our experiments, this consistently outperforms training the compact network from scratch and knowledge distillation using a teacher. In this context, we introduce several expansion strategies, together with an initialization scheme, and demonstrate the benefits of our ExpandNets on several tasks, including image classification, object detection, and semantic segmentation. 

Under review as a conference paper at ICLR 2020
EXPANDÑETS: LINEAR OVER-PARAMETERIZATION TO TRAIN COMPACT CONVOLUTIONAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, introduce novel approach to training given compact network. To this end, we build upon over-parameteriiztion, which typically improves both optimization and generalization in neural network training, while being unneces- sary at inference time. We propose to expand each linear layer of the compact network into multiple linear layers, without adding any nonlinearity. As such, the resulting expanded network can benefit from wer-pparameterizatoon during train- ing but can be compressed back to the compact one algebraically inference. As evidenced by our experiments, this consistently outperforms training the compact network from scratch and knowledge distillation using teacher. In this context, we introduce several expansion strategies, together with an initialization scheme, and demonstrate the benefits of our ExpandNets on several tasks, including image classification, object detection, and semantic segmentation.
INTRODUCTION
With the growing availability of large-scale datasets and advanced computational resources, deep learning has achieved tremendous success in a variety of computer vision tasks, such as image classification (Krizhevsky et al. [2012] He et al., 2016), object detection (Ren et al. [015 Redmon Farhadi 2018 2016) and semantic segmentation (Long et al., 2015 Ronneberger et al. 2015).
Over the past few years, "Wider and deeper are better" has become the rule of thumb to design network architectures (Simonyan Zisserman, 2015; Szegedy et al., 2015) He et al.] 2016 Huang et al. 2017). This trend, however, raises memory- and computation-related challenges, especially in context of constrained environments, such as embedded platforms.
Deep and wide networks are well-known to be heavily over-parameterized, and thus compact network, both shallow and thin, should often be sufficient. Unfortunately, compact networks are no- toriously hard to train from scratch. As a consequence, designing strategies to train given compact network has become increasingly popular, the most popular approach consisting of transferring the knowledge of deep teacher network to the compact one of interest (Hinton al.] [015 Romero
et al. 2014 Yim et al. 2017a Zagoruyko & Komodakis 2017 Passalis & Tefas, 2018).
In this paper, we introduce alternative approach training compact neural networks, comple- mentary to knowledge transfer. To this end, building upon the observation that network over- parameterization improves both optimization and generalization (Arora et al. 2018 Zhang et al.) 2017 Reed 1993 Allen-Zhu et al., 20 b Kenji Kawaguchi & Bengio 2018). we propose to increase the number of parameters
given compact network by incorporating additional layers. However, instead of separating every two layers with nonlinearity, as in most of the deep learning literature, we advocate introducing consecutive linear layers. In other words, we expand each linear layer of a compact network into succession of multiple linear layers, without any nonlinearity in between. Since consecutive linear layers are equivalent to single one (Saxe et al., (2044), such an expanded network, or ExpandNet, can be algebraically compressed back to the original compact one without any information loss.
While the use of successive linear layers appears in the literature, existing work (Baldi & Hornik 1989 Saxe et al., 2014 Kawaguchi 2016, Laurent & von Brecht 2018 Zhou & Liang 2018 Arora al. 2018) has been mostly confined to networks without any nonlinearities and the theoretical study of their behavior under statistical assumptions that typically do not hold in practice. In par- ticular, these studies have attempted to understand the learning dynamics and the loss landscapes of

---

Under review as a conference paper at ICLR 2020
deep networks. Here, by contrast, focus on practical, nonlinear, compact convolutional neural networks, and investigate the use of linear expansion to introduce over- paramettttttinnnoonnnn and improve training. as to allow such networks to achieve better performance.
Specifically, as illustrated by Figure I we intro- duce three ways expand compact network: (i) replacing a fully-connected layer with mul- tiple ones; (ii) replacing k k convolution by three convolutional layers with kernel size x I, k l and ing a k x k convolution with k >> 3 by multiple convolutions. Our experiments demon- strate that expanding convolutions, which has not been done before, key to obtaining more effective compact networks.
CorwLateer
x x mm
Input
ConwLayer
Expand
A 2 mm xire
Compresa
ConvLayer
1 1 x wile
Comdaayer
respectively; and (iii) replac-
Fx
[Epanee- Ok
ConvLayer 3 m rmw
Noninaar Layer
ConvLayer 8 3 x me as
Expand-FO FC
Fully conneecdd Layer
Fully colly-ooneected Layer
Compress
Layer
Furthermore, we introduce a natural initializa- tion strategy for our ExpandNets. Specifically, an ExpandNet has a nonlinear counterpart, with an identical number of parameters, obtained by adding nonlinearity between every two con- secutive linear layers. We therefore propose to directly transfer the weights of this nonlin- ear model to the ExpandNet for initialization. We show that this yields a further performance boost for the resulting compact network.
Fully-conmectedd Layer
Fuly-correettdl Layer
Output
Convolutinal Noural Notwork
Figure 1: ExpandNets. We study 3 strategies to linearly expand a compact network. The resulting expanded network can be compressed back the compact one algebraically, and consistently out- performs training the compact one from scratch and with knowledge distillation.
In short, our contributions are (i) a novel ap- proach to training given compact, nonlinear convolutional networks by expanding their linear layers: (ii) two strategies to expand convolutional layers; and (iii) an effective initialization scheme for the resulting ExpandNets. We demonstrate the benefits of our approach several tasks, including image classification on ImageNet, object detection on PASCAL VOC and image segmentation on Cityscapes. Our ExpandNets consistently outperform training the corresponding compact networks from scratch and using knowledge dis- tillation. We further analyze the benefits of linear over- wver-paraettrization on training via a series of experiments studying generalization, gradient confusion and the loss landscapes. We will make code publicly available.
2 EXPANDNETS
Let us now introduce our approach to training compact networks by expanding their linear layers. In particular, propose to make use of linear expansion, such that the resulting ExpandNet is equivalent to the original compact network and can be compressed back to the original structure algebraically. Below, we describe three different expansion strategies, starting with the simplest case of fully-connected layers, followed by two ways to expand convolutional layers.
Expanding fully-connected layers. The weights of fully-connected layer can easily be repre- sented in matrix form. Therefore, expanding such layers can be done in a straightforward manner by relying on matrix product. Specifically, let W, x be the parameter matrix of fully-connected layer with m input channels and output ones. That is, given an input vector E IR",, the output y E R can be obtained as y Wn, W. Note that we ignore the bias, which can be taken into account by incorporating layer with an arbitrary number of linear layers can be achieved by observing that its parameter matrix can be equivalently written
additional channel with value I to æ. Expanding such fully-connected
W nxm
W WRXM X W Wixp-- X ... X W.
(I)
Pi x
While this yields an over- paer-aaramttrizztion, it does not affect the underlying model capacity. More importantly, this allows to increase not only the number of layers, but also the number of channels by setting P: > n, m for all i. To this end, we rely on the notion of expansion rate. Specifically, for an expansion rate r, define p1 rm and p: rn, Vi # 1. Note that other strategies
2

---

Under review as a conference paper at ICLR 2020
possible, e.g., P; = r'm, but ours has the advantage of preventing the number of parameters from exploding. In practice, considering the computational complexity of fully cully-connected layers, advocate expanding each layer into only two three layers with small expansion rate.
Expanding convolutional layers. The operation performed by a convolutional layer can also be expressed in matrix form, by vectorizing the input tensor and defining a highly structured matrix whose elements are obtained from the vectorized convolutional filters. While this representation could therefore allow us to use the same strategy with fully-connected layers, using arbitrary intermediate matrices would ignore the convolution structure, and thus alter the original operation performed by the layer. For a similar reason, one cannot simply expand convolutional layer with kernel size k x k with series of k x k convolutions, since, unless k = 1, the resulting receptive field size would differ from the original one.
To overcome this, we note that 1 x convolutions retain the computational benefits of convolutional layers while not modifying the receptive field size. As illustrated in Figure we therefore propose to expand k x k convolutional layer into consecutive convolutional layers: x convolution; x k one; and another x 1 one. This strategy still allows us to increase the number of channels in the intermediate layer. Specifically, for original layer with m input channels and output ones, given an expansion rate r, we define the number of output channels of the first 1 x 1 layer as p = rm and the number of output channels of the intermediate k x k layer as q rn.
Compressing an expanded convolutional layer into the original one can still be done algebraically. To this end, one reason with the matrix representation of convolutions. For input tensor of size (w x h x m). the matrix representation of the original layer can be recovered as
W m
W nugo
X W quxpe X W pexmu
(2)
where v w- h and each intermediate matrix has structure encoding convolution. It can be verified that the resulting matrix will also have a convolution structure, with filter size k x k.
Expanding convolutional kernels. While 3 x kernels have become increasingly popular in very deep architectures (He et al. 2016), larger kemee sizes are often exploited in compact networks, so as to increase their expressiveness and their receptive fields. As illustrated in Figure I l: x kernels with l: >> 3 can be equivalently represented with series of 7 3 x 3 convolutions, where | = -2 As before, the number of channels in the intermediate x 3 layers can be larger than that in the original k x le one, thus allowing us to linearly over- over-parametrics the model. Similarly to the fully-connected case, for an expansion rate r, we set the number of output channels of the first x layer to pl = rm and that of the subsequent layers to p; rn. The same -based strategy before can then be used to algebraically compress back the expanded kernels.
Dealing with convolution padding and strides. In modern convolutional networks, padding and strides are widely used to retain information from the input feature map while controlling the size of the output one. To expand convolutional layer with padding p. we propose to use padding p in the first layer of the expanded unit while not padding the remaining layers. Furthermore, to handle stride s, when expanding convolutional layers, we set the stride of the middle layer to s and of the others to 1. When expanding convolutional kernels, we use a stride 1 for all layers except for the last one whose stride is set to s. These two strategies guarantee that the resulting ExpandNet can be compressed back to the original compact model without any information loss.
Overall, the strategies discussed above allow us to expand an arbitrary compact network into an equivalent deeper and wider onc. Note that these strategies can be used independently or together. In any event, once the resulting ExpandNet is trained, can be compressed back to the original compact architecture in purely algebraic manner, that is, at absolutely no loss of information.
2.1 INITIALIZING EXPANDNETS
As will be demonstrated by our experiments, training ExpandNet from scratch yields consistently better results than training the original compact network. However, in the context of deep networks, initialization can have an important effect on the final results (Mishkin & Matas 2015; He et al.) 2013). While designing an initialization strategy specifically for compact networks is an unexplored and challenging research direction, ExpandNets can be initialized in very natural manner.
3

---

Under review as a conference paper at ICLR 2020
To this end, we exploit the fact that ExpandNet has natural nonlinear counterpart, which can be obtained by incorporating a nonlinear activation function between each pair of linear layers. We therefore propose initialize the parameters of an ExpandNet by simply training its nonlinear counterpart and transferring the resulting parameters to the ExpandNet. The initialized ExpandNet then trained in the standard manner. As evidenced by our experiments below, when the nonlinear counterpart achieves better performance than the compact network, this simple strategy typically yields additional accuracy boost our approach.
3 EXPERIMENTS
In this section, we demonstrate the benefits of our ExpandNets on several tasks, including image classification, object detection, and semantic segmentation. We then analyze the training behavior of our different expansion strategies.
We denote the expansion of fully-connected layers by FC, of convolutional layers by CL, and of convolutional kernels by CK. When combining convolutional expansions with fully-connected ones, use CL+FC CK+ and add +Init indicate the of our initialization strategy.
3.1 IMAGE CLASSIFICATION
We first study the use of our approach with very small networks on CIFAR-I0 and CIFAR-100, and then turn to the more challenging ImageNet dataset, where we show that our method can improve the results of the compact MobileNet (Howard et al. 2017), MobileNetV2 (Sandler et al. 2018 and ShuffleNetV2 0.5x
(Ma et al. 2018) architectures.
3.1.1 CIFAR-10 AND CIFAR-100
Experimental setup. For CIFAR-IO and CIFAR-100, we use the same compact network as in (Pas- salis & Tefas 2018 It is composed of 3 convolutional layers with 3 x 3 kernels and no padding. These 3 layers have 8, 16 and 32 output channels, respectively. Each of them is followed by a batch normalization layer, ReLU layer and a 2 2 max pooling layer. The output of the last layer is passed through a fully-connected layer with 64 units, followed by logit layer with either 10 or 100 units. To evaluate our kernel expansion method, we also report results obtained with similar network where the 3 x 3 kernels were replaced by 7 x 7 ones, with a padding of 3. All net- works, including our ExpandNets. were trained for 150 epochs using batch size of 128. We used standard stochastic gradient descent (SGD) with a momentum of 0,9 and a learning rate of 0.01, divided by 10 epochs 50 and 100. With this strategy, all networks reached convergence. In this set of experiments, the expand rate r is always set to 4 to balance the accuracy-efficiency trade-off. We evaluate the influence of this parameter in Appendix A.3 Note that our ExpandNet strategy is complementary to knowledge transfer; that is, we can apply any typical knowledge transfer method using our ExpandNet as student instead of the compact network directly. To demonstrate the benefits of ExpandNets in this scenario, conduct experiments using knowledge distillation (KD) (Hinton ct al. 2015), hint-based transfer (Hint)(Romen) et al. 2014) or probabilistic knowledge transter Passalli & Tefas 2018) using ResNet18 teacher network.
(PK
Results. We focus here on the SmallNet with 7 x 7 kernels, for which we can evaluate all our expansion strategies, including the CK ones, and report the results obtained with the model with 3 x3 kernels in Table 6 in Appendix A_ The results of our all networks with and without knowledge distillatior are provided in Table As shown in the top portion of the table, only expanding the fully-connected layer yields mild improvement. However, expanding the convolutional ones clearly outperforms over the compact network, and is further boosted by expanding the fully-connected one and by using our initialization scheme. Note that expanding the kernels yields higher accuracy, with CK+FC+Init ExpandNee- ExpanddNet-KK+FC+Inii achieving the best results. Note that even without KD, our ExpandNets outperform the SmallNet with KD. The gap is further increased when we also use KD, as can be seen in the bottom portion of the table.
Having found KD be the most effective knowledge transfer strategy,
4

---

Under review as a conference paper at ICLR 2020
Table 1: Top-1 accuracy (%) of SmallNet with 7x7
Table 2: Top-1 accuracy (%) on the ILSVRC2012 validation set (ExpandNets with r = 4).
kernels vs ExpandNets with r 4 on CIFAR-10 and CIFAR-100.
Network
Transfer
CIFAR-I0
CIFAR-100
Model
ImugeNet (90 spochs)
ImageNet (400 opochs)
SmallNet
we KD
78.63 + 0.41
16.63 2 0.27
SmallNet
KD
78.97 + 0.37
0.30
MobileNet
66.48
70.47
MobileNet KD)
69.01
N/A
ExpundNet-FC
ExpandNet-CL
78.17 0.220
78.64 1: 0.39
46.59
0.66
1: 0.45
ExpandNet CLow KD)
ExpandNNet CL.
70.47
69.40
70.82
N/A
ExpandNet-CI-4FC
0.223
46.66
+ 0.43
ExpandNe==LLAPCHI
KD
79.98
0.418
MobileNetV2
63.75
70.67
ExpandNet-CK
80.27 + 0.24
48.55
0.51
MobileNet tw/ KD)
65.40
N/A
ExpandNet-CK-FE
80.31 + 0.27
48.62
+ 0.47
ExpandNet-CL
65.62
70.70
SapandNei-CCKFFCC
80.81 1: 0.27
49.82
0.25
ExpandNer-CU KD)
67.19
N/A
ExpandNet-CI-4FC
79.60 + 0.25
47.41
0.51
ShuffleNetV2 0.5x
56.80
ExpandNet-CI IInt
ExpandNet-CK CKFFC
w/ KD
80.63 1: 0.31
80.29 1: 0.25
49.13
48.62
0.44
0.34
ShuffleNetV2 0.5x (w/ KD)
ExpandNet-CC
57.50
57.38
ExpandNet-CK+Ht CCCiili
81.21 + 0.17
50.37
0.39
ExpundNet CL (w/ KD)
57.68
3.1.2 IMAGENET
We now turn the more challenging case of ImageNet (Russakovsky et al. 2015). In this case, for MobileNets, we perform data augmentation by taking random image crops and resizing them to 224 x 224 pixels. We further perform random horizontal flips and subtract the per-pixel mean.
Experimental setup. For this set of experiments, we make use of the compact MobileNet (Howard al., 2017). MobileNetV2 (Sandler et al., 2018) and ShuffleNetV2 (Ma et al.) 2018) models, which were designed to be compact and yet achieve good results. We rely on pytorch implementation of these models. For our approach, we use our CL strategy to expand all convolutional layers with kernel size of 3 x 3 in MobileNet and ShuffleNet, while only expanding the non-residual 3 x 3 convolutional layers in MobileNetV2. We trained the MobileNets in 2 regimes: Short-term (as for ResNets (He et al. 2016)), where we trained for 90 epochs with weight decay of 0.0001 and an initial learning rate of 0.1, divided by 10 every 30 epochs; Long-term, where we trained for 400 epochs with weight decay of 0.00004 and an initial learning rate of 0.01, divided by 10 epochs 200 and 300. In all cases, we employed SGD with a momentum of 0.9 and a batch size of 256. For ShuffleNet, we of small ShuffleNetV2 0.5x, trained in the same manner as in (Ma al.) 2018). We also perform KD from a ResNet152 (78.32% top-1 accuracy), tuning the KD hyper-parameters the best accuracy for each method.
Results. We compare the results of the original models with those of our expanded versions in Table Our expansion strategy yields huge improvement, especially in the short-term regime, increasing the top-1 accuracy of MobileNet. MobileNetV2 and ShuffleNet by 2.92, 1.87 and 0.49 percentage points (pp). This short-term regime could be valuable work with limited access re- sources. In the long-term regime, our expansion strategy nonetheless still achieves an improvement. Furthermore, our vanilla ExpandNets outperform the compact models with KD), even though we do not require a teacher. When also using KD, ExpandNets can achieve even better performance. Note that, on this dataset, their nonlinear counterparts do not outperform original models.
3.2 OBJECT DETECTION
Our approach is not restricted image classification. We first demonstrate its benefits in the context of one-stage object detection using the PASCAL VOC dataset (Everingham et al. a bbb. To this end, trained networks on the PASCAL VOC2007 + 2012 training and validation sets and report the mean average precision (mAP) on the PASCAL VOC2007 test set.
Experimental setup. We make use of YOLO- LITE (Huang et al. 2018), which was designed to work in constrained environments. It con- sists of a backbone part and a head part, and we only expand the convolutional layers in the backbone. YOLO-LITE is very compact net- work, with only 5 convolutional layers in the backbone, each followed by a batch normalization layer, Peaky-ReLU layer and a 2 x 2 max pool- ing layer. We expand all convolutional layers using our CL strategy with r = 4. We trained all networks in the standard YOLO fashion (Redmon & Farhadi 2018 2016).
Table 3: mAPs (%) of YOLO-LITE vs Expand- Nets on the PASCAL VOC2007 test set.
Model VOC2007 Test
YOLO-LITE 27.34
ExpandNet-CL
30.97
ExpandNet-CL-Ini 35.14
5

---

Under review as a conference paper at ICLR 2020
Results. The results on the PASCAL VOC 2007 test set reported in Table 3 They show that, as for image recognition, our expansion strategy yields better performance. Since YOLO-LITE is very compact, our initialization scheme boosts the performance by over 7pp.
3.3 SEMANTIC SEGMENTATION
We then demonstrate the benefits of our approach on the task of semantic segmentation using the Cityscapes dataset (Cordts al., 2016). Cityscapes is large-scale dataset containing 5000 images of 1024 x 2048. Following the standard protocol, we report the mean Intersection over Union (mloU), mean recall (mRec) and mean precision (mPrec).
Experimental setup. For this experiment, we rely on the U-Net (Ronneberger et al.J 2015), which is a relatively compact network consist- ing of contracting path and an expansive path.
Table 4: U-Net vs ExpandNets with = 4 on the Cityscapes validation set.
Model mloU
mRec
mPPre
The contracting path follows the typical archi- tecture of a convolutional network. It consists of blocks of two 3 convolutions, each fol- lowed by a ReLU, and a x max pooling. The number of channels in each block is 32, 64, 128, 256, 512, respectively. We apply our CL expansion strategy with r all convolutions in the contracting path. We train the networks on 4 GPUs using the standard SGD optimizer with momentum of 0.9 and learning rate of 1e
U-Net 56.59
74.29
65.11
ExpandNet-CL
57.85
76.53
65.94
Results. Our results on the Cityscapes validation set are shown in Table Note that our ExpandNet also outperforms the original compact U-Net on this task.
3.4 ANALYSIS OF OUR APPROACH
To further analyze our approach, we study its behavior during training and its generalization ability. For these experiments, we make use of the CIFAR- 10 and CIFAR-106 datasets, and use the same settings in Appendix A..3 In Appendix [A.4, we also showcase the use of our approach with the larger AlexNet architecture on ImageNet. Finally, we evaluate the complexity the models in terms of number of parameters, FLOPs, and training and testing inference speed in Appendix B Since ExpandNets can be compressed back to the original networks, at test time, they have exactly the same number of parameters, FLOPS, and inference time, but achieve higher accuracy.
3.4.1 TRAINING BEHAVIOR
To investigate the benefits of linear over-parametterizztion training, make of the gra- dient confusion introduced by (Sankararaman et al.) 2019) to show that the gradients nonlin- early over-vvranee wver-parameterized networks were more consistent across mini-batches. Specifically, follow- ing (Sankararaman et al., 2019), we measure gradient confusion (or rather consistency) as the mini- mum cosine similarity from 100 randomly-sampled pairs of mini-batches at the end of each training epoch. As in (Sankararaman et al.) 2019), we also combine the gradient cosine similarity of 100 pairs of sampled miii-batcher at end of training from each independent run and perform Gaus- sian kernel density estimation on this data. While, during training. we aim for the minimum cosine similarity to high, at the end of training, the resulting distribution should be peaked around zero.
From the training and test curves, we observe that our ExpandNets-CL/CTS speed up convergence and yield smaller generalization error gap. They also yield lower gradient confusion (higher minimum cosine similarity) and a more peaked density of pairwise gradient cosine similarity. This indicates that our ExpandNets CCeC/dd CL/CK are easier train than the compact model. By contrast, only expanding the fully-connected layers does not facilitate training. Additional plots are provided in Appendix
We run each experiment 5 times and show the average values across all runs in Figure 2
3.4.2 GENERALIZATION ABILITY
We then analyse the generalization ability our approach. To this end, we first study the loss land- scapes using the method in (Li al. 2018). As shown in Figure our ExpandNets, particularly with convolutional expansion, produce flatter minima, which, as discussed (Li et al. 2018). indicates better generalization.

---

Under review as a conference paper at ICLR 2020
Smaltter tran
Smallhee: test
Smalllet Expandlles FC
Brok 10SU01 DOO C5
ExpundNiteFtoo train
pradert simtartt
0.00
Expenchess
ExpandNactFC test
Expendhes CC
30-
Expandhlal-1 train
det test
De-sio
20-
-0.15
21
-0.20
75
125
230
25
300
130
-0.3
-02
0.0
0.1
0.5
Esoch
Epoch
Parm gradient siddartyy
Figure 2: Training behavior of networks with 7 x kernels on CIFAR-10. Left: Training and test curves over 150 epochs. Middle: Minimum pairwise gradient cosine similarity at the end of each training epoch. Right: Kernel density estimation of pairwise gradient cosine similarity at the end of training (over independent runs).
20 -o
- 10 40
1120 -A AD 10 10
IN sn
(1) StralNNet: 19.42% 121 ExpurdNel-FOrr2t 19.32%
(3) ExpencheetGGLe -0tr-- 16.97%
(4) ExpardNee-CKr--. 17.12%
Figure 3: Loss landscapes of networks with x 9 kernels on CIFAR-1O.
Table 5: Generalization ability on Corrupted CIFAR-10. We report the top-1 error (%). Note that our ExpandNets yield larger generalization gaps than the compact network in almost all cases involving convolutional expansion. By contrast expanding FC layers often does not help.
Dateset
size
Test
Test
Train
Best Test
Test
Train
Smallhet
20,30 = 0.16
21,00 + 0.20
32.05 0.33
22.56 d 0.30
22.33 + 0.18
29.61
0:36
20%
ExpundNet CL
EspundNer--R:
20.47 1 0.48
20.87 + 0.25
20.62
21.06
1 0.43
+ 0.26
$1.50
52.04
10011.000 0.25
+-0.2
22.95 + 0.30
22.13 1 0.29
22.75 0.53
23.45 + 0.38
29.76
29.88
0.34
0.19
ExpandNet-CK
19.42 = 0.20
19663
0.17
31.55
0.225
19.32 0.31
19,55 0.30
31,65
0.17
SmallNet
25.38
0.45
25.68
1 0.52
$1.09:1..11
28.64 0.46
30.54 1 0.57
52.67
0.45
50%
Ex
25.36
00.0
0.43
20.80
62.61
0.36
ExpundNec
24.27
0.44
29.28
52,07
ExpandNet-CK
53.03
22.77
SmallNett
37.00
30:33
0..15
41.73++0.55
47.06 1.07
74.01 0.32
80%
EspundNe--CL CL
ExpandNers-C::
36.75
38.36
0.64
0.68
38.06 1 0.500
30.61 + 0.87
76.09
76.51 +-115
1.0.11
42.31 +-0.6
41.44
66.75
40.36 1.44
0.49
74.46
74.00
0.08
0.35
ExpundNat-CK
33,29 =: 1.04
34.24 :: 0.85
75.77
0:22
33.29 a: 0.58
33.75 a: 0.49
76.27
0.25
As a second study of generalization, we evaluate the memorization ability of our ExpandNets on corrupted datasets, as suggested in (Zhang et al. (017). To this end, we utilize the open-source implementation of (Zhang et al. (017) to generate three CIFAR-10 and CIFAR-100 training sets, containing 20%, 50% and 80% random labels, respectively, while the test set remains clean.
In Table 5 (and Tables 10 and 12 in Appendix A.5). we report the top-1 test errors of the best and last models, as well as the training errors of the last model. These results evidence that expanding convolutional layers and kernels typically yields lower test errors and higher training ones, which implies that our better results in the other experiments are not due to simply memorizing the datasets, but truly better generalization ability.
RELATED WORK
Very deep neural networks currently constitute the state of the art for many computer vision tasks. These networks, however, known to be heavily over weerpprrmeterized, and making them smaller would facilitate their use in resource-constrained environments, such as embedded platforms. As a consequence, much research has recently been devoted to developing more compact architectures.
Nenwork compression constitutes one of the most popular trends in this area. In essence, it aims to reduce the size of pre-trained very deep network while losing little accuracy as possible, or even none at all. In this context, existing approaches can be roughly grouped into two categories:
(i) parameter pruning and sharing (LeCun al. 1990 Han al. 2015; Courbariaux et al. 2016
7

---

Under review as a conference paper at ICLR 2020
M Figurnov 2016; Pavlo Molchanov 2017 Ullrich et al. [017 Carreira-Perpin & Idelbayev 2018
which aims to remove the least informative parameters; and (ii) low-randd factorization (Denil et al,
2013 Sainath et al. 2013 Lebedev et al.] 2014: Jin et al., 2015: Liu et al.] 2015), which uses decomposition techniques to reduce the size of the parameter matrix/tensor in each layer. While compression is typically performed post-processing step, it has been shown that incorporating it during training could be beneficial (Alvarez& & Salzmann [016)] 2017 Wen et al.] [016 (017)) In any event, even though compression reduces a network's 's size, does not provide one with the flexibility of designing a network with a specific architecture and of specific size. Furthermore, it often produces networks that are much larger than the ones we consider here, e.g., compressed networks with (1M) parameters vs O(10K) for the SmallNets used in our experiments.
In a parallel line of research, several works have proposed design strategies to reduce a network's number of parameters (Wu et al.) 2016) SSzgedy et al. 2016, Howard et al. 2017 Romera et al.) 2018 Sandler et af.) 2018). Again, while more compact networks can indeed be developed with these mechanisms, they impose constraints on the network architecture, and thus do not allow one to simply train a given compact network. Furthermore, as shown by our experiments, our approach is complementary to these works. For example, we can improve the results of MobileNetV2 (Snnderr ett al. 2018) by training it using our expansion strategy.
Here, in contrast to the compact network with an arbitrary architecture. This is also the task addressed by knowledge trans- fer approaches. To achieve this, existing methods leverage the availability of pre-trained very deep teacher network. In (Hinton et al. 2015), this is done by defining soft labels from the logits of the teacher; in (Romero al. 2014). (Zagoruyko & Komodakis 2017) and (Yim et al. (017b) by transferring intermediate representations, attention maps and Gram matrices, respectively, from the teacher to the compact network; in (Passalis & Tefas, 2018 by aligning the feature distributions of the deep and compact networks.
literature on compact networks, we seek to train given
In this paper, we introduce an alternative strategy to train compact networks, complementary to knowledge transfer. Inspired by the theory showing that over- wver-arrametrizatiion helps training (Arora al. 2018 Zhang et al. 2017] Reed 1993; Allen-Zhu et al. (008a|]), we propose to expand each linear layer in given compact network into succession of multiple linear layers. Our experiments evidence that training such expanded networks, which can then be compressed back algebraically, yields better results than training the original compact networks, thus empirically confirming the benefits of over weer-paaamttrizatiin Our results also show that our approach outperforms knowledge transfer, even when not using teacher network.
Note that linearly over over-parametrized neural networks have been investigated both in the early neu- ral network days (Baldi & Hornik 1989) and more recently (Saxe al., 2014 Kawaguchi 2016 Laurent & von Brecht 2018 Zhou & Liang, 2018 Arooa et al. 2018). These methods, however, typically study purely linear networks, with focus on the convergence behavior of training in this linear regime. For example, (Arora et al. 2018) showed that linear over-paramettizzatin modifies the gradient updates in a unique way that speeds up convergence. Here, we propose to exploit the benefits of linear paramttratttiiiio to improve the training of compact networks. To this end, in contrast to the above-mentioned methods which all focus on fully-connected layers, we develop two strategies to expand convolutional layers, as well as an approach to initializing our expanded networks, and empirically demonstrate their critical impact on prediction accuracy.
5 CONCLUSION
We have introduced an approach to training a given compact network from scratch by exploiting over-paramenn werrparaaetrizzaionn Specifically, we have shown that weraperraetiiiing the network linearly facilitates the training of compact networks. Since the resulting networks are equivalent to the original compact ones, they can be compressed back at no loss of information. Our experiments have demonstrated that our ExpandNets consistently outperform the original compact networks. In particular, when applicable, our CK expansion strategy tends to yield the best results. When doesn't, that is, for kernel size of 3, the CL one nonetheless remains highly effective. Our technique is general and can also be used in conjunction with knowledge transfer approaches. Furthermore, our initialization scheme yields consistent accuracy boost. This strategy. however, is not the only possible one. In the future, we will therefore aim to develop other initialization schemes for our ExpandNets, and generally for compact networks.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameter- ized neural networks, going beyond two layers, arXiv Preprint, 2018a.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. arXiv Preprint, 2018b.
Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems. 2016.
Jose M Alvarez and Mathieu Salzmann. Compression-awee awaar training of deep networks. In Advances in Neural Information Processing Systems. 2017.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparamcterization. International Conference on Machine Learning, 2018.
Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. IEEE Transactions Neural Networks, 1989.
Miguel Carreira-Perpin and Yerlan Idelbayev. Searning-compresst algorithms for neural net pruning. In Conference on Computer Vision and Pattern Recognition, 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Conference on Computer Vision and Pattern Recognition, 2016.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained or-1. arXiv Preprint, 2016.
Misha Denil, Babak Shakibi, Laurent Dinh, Nando De Freitas, et al. Predicting parameters in deep learning. Advances in Neural Information Processing Systems, 2013.
M. Everingham, L. Van Gool, C. K. Williams, Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.paccal- htmml, a.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal
b.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv Preprint, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In International Conference on Computer Vision, International Conference on Computer Vision, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Conference on Computer Vision and Pattern Recognition, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv Preprint, 2015.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv Preprint, 2017.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Conference Computer Vision and Pattern Recognition, 2017.
Rachel Huang, Jonathan Pedoeem, and Cuixian Chen. Yolo-lite: A real-time object detection algo- rithm optimized for non-gpu computers. In IEEE International Conference on Big Data. IEEE, 2018.

---

Under review as a conference paper at ICLR 2020
Jonghoon Jin, Aysegul Dundar, and Eugenio Culurciello. Flattened convolutional neural networks for feedforward acceleration. In International Conference on Learning Representations, 2015.
Kenji Kawaguchi. Deep learning without poor local minima. In Advances in Neural Information Processing Systems, 2016.
Leslie Pack Kaelbling Kenji Kawaguchi and Yoshua Bengio. Generalization in deep learning. In Mathematics Deep Learning. Cambridge University Press. Prepint avaliable as: MIT-CSAIL- TR-2018-014, Massachuselts Institute of Technology, 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, 2012.
Thomas Laurent and James von Brecht. Deep linear networks with arbitrary loss: All local minima are global. In International Conference on Machine Learning, 2018.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv Preprint, 2014.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, 1990.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In Advances in Neural Information Processing Systems, 2018.
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convo-
lutional neural networks. In Conference on Computer Vision and Pattern Recognition, 2015.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Conference on Computer Vision and Pattern Recognition, 2015.
DP Vetrov P Kohli M Figurnov, A Ibraimova. Perforatedenns: Acceleration through elimination of redundant convolutions. In Advances Neural Information Processing Systems, 2016.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In European Conference Computer Vision, September 2018. 8.
Dmytro Mishkin and Jiri Matas. All you need is good init. Computing Research Repository, 2015.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In European Conference Computer Vision, 2018
Tero Karras Timo Aila Jan Kautz Pavlo Molchanov, Stephen Tyree. Pruning convolutional neural networks for resource efficient inference. In International Conference Learning Representa- tions, 2017.
Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger. arXiv Preprint, 2016.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv Preprint, 2018.
R. Reed. Pruning algorithms-a survey. IEEE Transactions Neural Networks, 4(5):740-747, Sep. 1993. ISSN 1045-9227. doi: 10.1 10.1709/72.244452.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, 2015.
E. Romera, M. Ivarez, M. Bergasa, and R. Arroyo. Erfnet: Efficient residual factorized convnet for real-time semantic segmentation. IEEE Transactions on Intelligent Transportation Systems, 2018.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv Preprini, 2014.
10

---

Under review as a conference paper at ICLR 2020
O. Ronneberger, P.Fischer, and T. Brox. U-net: Convolutional networks for biomedical image seg- mentation. In Conference on Medical Image Computing and Computer Assisted Intervention, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision,
115(3):211-252, 2015, doi: 10. 1233-015-081644
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimensional output targets. In International Conference on Acoustics, Speech, and Signal Processing, 2013.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Shang-Chihh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Conference on Computer Vision and Pattern Recognition, 2018.
Karthik A Sankararaman, Soham De, Zheng Xu, W Ronny Huang, and Tom Goldstein. The impact of neural network overparamcterizztion on gradient confusion and stochastic gradient descent. arXiv Preprint, 2019.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dy- namics of learning in deep linear neural networks. In International Conference on Learning Representations, 2014.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations, 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Conference on Computer Vision and Pattern Recognition, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Conference on Computer Vision and Pattern Recognition, 2016.
Karen Ullrich, Edward Meeds, and Max Welling, Soft weight-sharing for neural network compres- sion. In International Conference on Learning Representations, 2017.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems, 2016.
Wei Wen, Cong Xu, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Coordinating filters for faster deep neural networks. In International Conference on Computer Vision, 2017.
Bichen Wu, Forrest landola, Peter Jin, and Kurt Keutzer. Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving, arXiv Preprint, 2016.
J. Yim, D. Joo, J. Bac, and J. Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Conference on Computer Vision and Pattern Recognition, 2017a.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Conference on Computer Vision and Pattern Recognition, 2017b.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. In International Conference on Learning Representations, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017.
11

---

Under review as a conference paper at ICLR 2020
Yi Zhou and Yingbin Liang. Critical points of linear neural networks: Analytical forms and land- scape properties. In International Conference on Learning Representations, 2018.
A
COMPLEMENTARY EXPERIMENTS
We provide additional experimental results to further evidence the effectiveness of our three expan- sion strategies.
A.I SMALLNET WITH 3x X 3 KERNELS
As mentioned in the main paper, we also evalu- ate our approach using the same small network as in (Passalis & Tefas 2018), with kernel size 3 x 3 instead of 7 in the main paper, CIFAR-10 and CIFAR-100 In this case, we make use of our CL. expansion strategy, with and without our initialization scheme, because the CK one does not apply 3 x 3 kernels. We trained all these networks for 100 epochs using batch size of 128. We used Adam with a learning rate of 0.001, divided by 10 at epoch 50, which matches the setup in (Passalis & Tefas 2018). As reported in Table expand- ing the convolutional layers yields higher accu- racy than the small network. This is further im- proved by also expanding the fully-connected layer, and even more so when using our initialization strategy.
Table 6: Top-1 accuracy (%) of SmallNet with 3 x kernels vs ExpandNets with r = 4 on CIFAR-10 and CIFAR-100. Our approach yields consistently better results than the compact net- work, particularly when expanding all layers and using our initialization strategy.
Model
Transfor
CIFAR-10
CIFAR-100
SmallNet
who 73.32 + 0:20
40.40 0.60
SmallNet
w/ KD
13.34 + 0.31
40.46
0.56
ExpandNer-FC
73.78 0.83
ExpandNet-CL+NC
ExpandNei CL
w/b KD
T3.96 1: 0.30
0:29
40.91
0.47
ExpandNas-CL+RR-llit
75.16 =: 0.23
42.41 0.21
ExpandNa--LL4CC
71.52 +: 0.37
41.51
75.17 + 0.51
42.67 0.67
A.2 KNOWLEDGE TRANSFER WITH EXPANDNETS
In the main paper, we claim that our Expand- Net strategy is complementary to knowledge transfer. Following (Passalis & Tefas, 2018), on CIFAR-10, we make use of the ResNetT8 as teacher. Furthermore, we also use the same compact network with kernel size 3 x 3 and training setting as in (Passalis & Tefas 2018). In Table 7 we compare the results of different knowledge transfer strategies, in- cluding knowledge distillation (KD) (Hinton et al. 2015). hint-based transfer (Hint)(Romeru et al. 2014) and probabilistic knowledge trans- fer (PKT) (Passllis & Tefas 2018), applied to ExpandNets,
Table 7: Knowledge transfer from the ResNet18 on CIFAR-10. Using Expand- Nets as student networks yields consistently better results than directly using SmallNet.
Network Transfer
Top-1 Accuracy
SmallNet Baseline
73.32 0.20
KD
73.34 0.31
SmallNet
Hint
33.71 + 4.35
PKT
68.36 + 0.35
KD
74.52 = 0.37
ExpandNet
Hint
52.46 + 2.43
the compact network and without and with our initialization scheme. Note that using our ExpandNets, with and with- out initialization, consistently outperforms us- ing the compact network. Altogether, we there- fore believe that, to train given compact network, one should really use both knowledge transfer and our ExpandNets to obtain the best results.
(CL+FC)
PKT
70.97 + 0.70
KD
75.17 + 0.51
ExpandNet
Hint
58.27 3.83
(CL+FC+Init)
PKT
71.65 + 0.41
A.3 ABLATION STUDY
In this section, we conduct an ablation study on the hyper-parameter choice, specifically the expan- sion rate and kernel size. We evaluate the behavior of our different expansion strategies, FC, CL and CK, separately, when varying the expansion rate r E {2, 4, 8} and the kernel size E {3, 5, 7, 9). Compared to our previous CIFAR-10 and CIFAR-100 experiments, we a deeper network with an
12

---

Under review as a conference paper at ICLR 2020
Table 8: Small networks vs ExpandNets on CIFAR-10 (Top) and CIFAR-100 (Bottom). We report the top-1 accuracy for the original compact networks and for different versions of our ap- proach. Note that our ExpandNets yield higher accuracy than the compact network in almost all cases involving expanding convolutions cases. By contrast expanding FC layers does often not help.
Model
Keree size &
7
SmallNet
79.34 + 0.42
81.25 0.14
81.44 0.20
80.08 0.48
2
79.13 +: 0.47
81.26 = 0.33
$0.98 + 0.25
80.13 = 0.22
ExpandNet-FC:
4
78.92 +: 0.36
81.13 0.46
80.85
0.24
80.13 0.29
79.64 +:
81.21 + 0.18
0.45 80.75
80.16 0.16
79.46 + 0.21
81.50 0.31
81.30
0.30
80.26 0.66
ExpandNet-CL.
4
79.90 + 0.21
81.60 0.15
81.15 0.36
80.62 + 0.32
8
79.78 + 0.20
81.75 = 0.40
81.53 + 0.33
80.78 - 0.25
N/A
81.72 0.31
82.19 0.24
81.60 0.11
ExpandNer-CK
N/A
82.34
0.33
N/A
82.37 = 0.25
82.84 0.28
82.53 0.30
SmallNet
14 +: 0.29
50.44 +: 0.07
49.62 0.50
48.70 0.38
47.21 + 0.46
48.39 + 0.77
47.88 0.41
46.36 0.34
ExpandNet-FCC
47.44 + 0.66
48.92 0.47
48.43 0.56
46,90 0.34
8
47.55 + 0.25
49.44 + 0.65
48.66
0.49
47.15 0.28
2
47.68 +: 0.85
50.39 + 0.45
49.78 + 0.33
48.68 + 0.70
ExpandNet-CL
4
48.25 + 0.13
50.68 = 0.27
49.81 0.31
48.87 0.65
48.93 + 0.13
50,95 0.42
49.95 0.37
48.85 0.42
N/A
51.18 0.44
51.09 U.41
50.40 0.35
ExpandNet-CK
4
N/A
52.13 = 0.36
51.82 + 0.67
50.62 = 0.65
N/A
52.05 0.59
52.48 0.54
51.57 = 0.15
extra convolutional layer with 64 channels, followed by batch normalization layer, ReLU layer and 2 max pooling layer. We use SGD with momentum 0.9 and a weight decay of 0.0005 for 150 epochs. The initial learning rate was 0.01, divided by 10 at epoch 50 and 100. Furthermore, we used zero-padding to keep the size of the input and output feature maps of each convolutional layer unchanged. We use the same networks in our approach analysis experiments.
The results of these experiments are provided in Table We observe that our different strategies to expand convolutional layers outperform the compact network in almost all cases, while only expand- ing fully-comma eolly-conneeted layers doesn't 't work well. In particular, for kernel sizes > 3, ExpandNet- yields consistently higher accuracy than the corresponding compact network, independently of the expansion rate. For = 3, where ExpandNet-CK is not applicable, ExpandNet-CI. comes as an effective alternative, also consistently outperforming the baseline.
A.4 WORKING WITH LARGER NETWORKS
We also evaluate the of our approach with larger network. To this end, make of AlexNet (Krizhevsky et al., 2012) on ImageNet. AlexNet relies on kernels of size 11 and 5 in its first two convolutional layers, which makes our CK expansion strategy applicable.
We use modified, more compact version of AlexNet, where we replace the first fully- connected layer with global average pooling layer, followed by a 1000-class fully-connected layer with softmax. To evaluate the impact of the network size, we explore the use of dif- ferent dimensions, [128, 256, 512], for the final convolutional features. We trained the resulting AlexNets and corresponding ExpandNets using the short-term regime of our MobileNets exper- iments in Section
Table 9: Top-1 accuracy (%) of AlexNet vs Ex- pandNets with r = on the ILSVRC2012 val- idation set for different number of channels in the last convolutional layer. Note that, while our expansion strategy always helps, its benefits de- crease the original model grows.
# Channels 128
256 (Original)
$12
Bascline 46.72
54.08
58.35
ExpandNet-CK 49.66
55.46
58.75
As shown Table 9 while our approach outperforms the original AlexNet for all feature dimen- sions, the benefits decrease as the feature dimension increases. This indicates that approach is
13

---

Under review as a conference paper at ICLR 2020
Table 10: Generalization ability (top-1 error (%) on Corrupted CIFAR-10 (kernel size: 3, 7).
Model
Kemel &
Best Test
Last Test
Train
Best Test
Last Test
Train
SmallNer
22.20 + 0.43
22.33 + 0.40
36.86 + 0.18
21. G4 0:36
21.98 = 0.42
30.
0.32
20%
ExpandNet-C1.
ExpandNet- FC
22.40 0.29
21.55 0.27
22.61 0.27
21.71
35.12 4: 0.07
34.80 0.26
21.25 11.225-2
21.92 0.23
22.35
21 0.40
4: 0.43
30.39 0.19
30.36
ExpandNes-CK
N/A
NJA
NJA
19:11
0.35
31.1
SmallNet
25.74 + 0.26
25.94 + 0.15
M6.48 + 0.25
26.09 0.60
27.87 + 0.71
50.27 0.21
50%
ExpandNer-C1.
Ex
25.54 1: 0.47
25.48 + 0.35
25.80 a: 0.41
0.43 25.66
56.37 4: 1.0.5
16.41 + +-333
26.86 0:366:100 0.060:000000 46
26.05 + 0:31
26 26.23
26.00 + 0.15
1: 0.61
63-21 0.16
63.14 0:20
ExpanNNe-CK
N/A
N/A
NJA
22.43 0.47
22.61 0.49
53.74 0.16
SmallNer
37.40 0.62
97.87
0.63
17.46 + 0.16
30.08
0.77
74
8056
Ex EspondNet-CI.
ExpundNet-FC PC
97.26
35.86 + 0.43
0.16
97.63
36.115 0.44
0.14
17.54 0.07
77.56 + 6.11
39.40 0.93
10.51
42:77 + 0.96
44.82 0.62
76.24 0:22
78.33
PxpandNes-CK
A/4
N/A
N/A
32.62 + 0.28
33,86 + 0.37
75.66 0.16
Table 11: Generalization ability (top-1 error (%)) on Corrupted CIFAR-I0K 100 (kernel size: 3, 5).
Model
Kumel
Best Test
Last Test
Train
Best Test
Last Test
Train
SmallNet
55.30 1 0.12
55, 1 0.31
02.20 1: 0.31
53.95 0:33
54.16 1 0.31
58.53 0.30
20%6
ExpandNet-C1.
ExpondNet-FC PC
56.15 + 0.22
54.85 0.27
55.04
56.35 0.23
02.60 +: 0.19
53.50
64.84 0.71
55.05 + 0.76
0.38
68.00
59.47 0.32
ExpandNet-CK
N/A
N/A
NJA
51.98
52.10
0.26
67.67
SmallNet
62.54 0.74
62.71 0.75
78.81 1. 0.39
6L.84 0.29
62.16 0.21
76, 0.28
50%
EspondNat-C1.
ExPundNa-FC
63.65 1: 0.50
61.95 + 0.61
63.88
62,11 + 0.59
1 0.47
78.78 + 0.48
79-40
1: 0.18
61.49 0.39
62.99 0.69
61.70 0.43
63.21 :: 0.60
77. 85
76. 73.73--5 0.36
1: 0.31
ExpandNe--CK
N/A
N/A
NJA
58.96
0.41
SmallNes
0.83
0:35
80%
Ex EspondNet-CL.
ExpundNect-F FC
79.44
80.36
+ 0.72
0.
70.66 0.75
80.47
0.55
94.38 0.12
04.02 + 4-0.16
80.97 0.51
70.87
0.29
81.15 0.53
80:04 +: 0.29
93. 10.50 0:20
94.
0.16
PxpandNes-CK
8/4
N/A
N/A
77.22 + 0.47
77.38 0.41
83. 10.15-++ 0:25
Table 12: Generalization ability (top-1 error (%) on Corrupted CIFAR- 100 (kernel size: 7, 9).
Model
Best Test
Last Test
Train
Best Test
Last Test
Train
SmallNet
55.36 1 0.04
55.66 1. 0.45
56.38 1: 1.0.51
56.59 0.72
57.36 1 0.64
55. 16
0.32
20%
ExpondNat-C1.
EspundNei-PC FC
56.31
4: 0.78
0.47
55.22
56.58
0.77
57.98 1: 0.29
00.40
67.82 0:23
56.05
56.51 0.76
58.07 :: 0.22
64.90
57.00
0.52
ExpandNet-CK
51.24
0.60
51.40 0.66
56.40
0.21
52.36
0.47
57.76
SmallNet
63.76
0.59
64.08
0.58
15.15
0.23
61.83 0.41
65.63 0.40
75.21 00.31
50%
EspendNNatCTL
ExpundNa-FC
64.54 1: 0.72
63.36 + 0.49
63.73
64.01
1: 0.55
0.54
75.25
76.75
+ 0.45
1: 0.99
64.36 0.54
66.11 1: 6:45
65 25 + 0.37
66.73 1: 0.41
74.84 0.28
76.44 0.00
ExpandNest CK
58,74
0,25
0.20 $8.98
15.24
0.224
60.42 0.86
60.65 0.86
76.73 0.40
SmallNer
79.73 0.07
0.20
81.02
0:97
92.
80%
Ex EspondNee-CL
ExpundNet-FF FC
82.97
80.78
1: 0.54
0.83
81. :: 0.62
83.20 0.83
03.22
95.13 0.34
1: 0.990
81 02 44
83: 42 0.72
81 58 :: 0.46
83.82 0.71
53.90 13.94 0.35
98.25 0.56
ExpandNes-CK CK
78.51 1: 0.41
78.64 0.36
93.24 1: 0.15
$0.15 1: 0.50
$0.32 : 0.55
94.04 0,21
better suited for truly compact networks, and developing similar strategies for deeper ones will be the focus of our future research.
A.5 GENERALIZATION ABILITY ON CORRUPTED CIFAR-10 AND CIFAR-100
Our experiments on corrupted datasets implies better generalization. Here, we provide more exper- imental results on Corrupted CIFAR-10 10 (in Table I0) and CIFAR- 100 (in Table and (2) by using different networks with kernel sizes of 3. 5. 7. respectively. Our method consistently improves the generalization error gap across all kernel sizes and corruption rates (20% 50%, 80%) and yields from around Ipp to over 6pp error drop in testing.
B COMPLEXITY ANALYSIS
Here, we compare the network complexity of method and the original networks in terms of number of parameters, number of FLOPs and GPU inference speed.
In T3] we provide numbers for both training and testing. During training, because our approach introduces more parameters, inference is 2 to times slower than in the original network for an expansion rate of 4. However, since our ExpandNets can be compressed back to the original network,
14

---

Under review as a conference paper at ICLR 2020
Table 13: ExpandNet complexity analysis on CIEAR-10, ImageNet, PASCAL VOC and Cityscapes. Note that, within each task, the metrics are the same for all networks, since we can compress our ExpandNets back to the small network.
# (a)
# #FLOPs
GPU Speed (imgs/sce)
Model
Train
Test
Train
Test
Train
Test
SmallNet (7 7)
0.07
4.49M
147822.51
ExpandNet-CL
0.55
57.49M
64631.81
ExpandNet-CL 2.11
0.07
59.04M
4.49M
61379.95
154850.32
ExpandNct-CKK 0.19
23.95M
75065.09
ExpandNet-CK+FC
1.75
25.5M
68679.89
MobileNet
4.23
0.58G
3797 .21
0.58G
3829.81
ExpandNet-CL 4.96
1.76G
729.78
MobileNetV2 3.50
0.32G
3417.20
0.32G
3419.43
ExpandNct-CLL 3.67
1.34G
1009.25
ShuffleNetV2 0.5x
1.37
0.04G
5404.06
1.37
0.04G
5434.58
ExpandNel-CLL 1.41
0.6G
4228.10 10
YOLO-LITE
0.57
1.81G
7.94
0.37
1.81G
19.82
ExpandNet-CLL
4.48
28.59G
6.07
U-Net
7.76
389.26G
8.25
ExpandNet-CI.
82.97
7.76
2586.02G
389 26G
8.25
sloc
Smabkket
(a.0 20.25%
[&] 18.57)
I:ll 16.20%
2811 1542%
ExpondNin-FC
06.30 r.4, 12.094
(6:30
(0.0 17.7%%
18.30
Eqpanthe--EX
(b:4) m6. 17.23%
10.4) -6, 1..0%
20.11 ~8, 17.25 25
Figure Loss landscape plots on CIFAR-10 (We report top-1 eтrоr (%)).
at test time, they have exactly the same number of parameters and FL.OPS, and inference time, but our networks achieve higher accuracy.
C ADDITIONAL VISUALIZATIONS
Here, provide additional visualizations for the training behavior and the loss landscapes of Sec- tion 3.4, corresponding to networks with kernel sizes of 3, 5, 9, respectively.
We plot the loss landscapes of SmallNets and corresponding ExpandNets CIFAR-10 in Figure and analyze the training behavior on CIFAR- 10 in B and on CIFAR-100 Figure respec-
tively. These plots further confirm that in almost all cases, our convolution expansion strategies (CL and CK) facilitate training (with lower gradient confusion and more O-concentrated gradient cosine similarity density) and yield better generalization ability (with flatter minima).
15

---

Under review as a conference paper at ICLR 2020
40
SmalNel Irem
Smalliet Expandhat-FCC
35
Smalfiet test
ExpandNen-FCC trai
Mi: orootnt costre simtarkt
-0.05
Expandhlet-a
ExpardNac-CL train
ExpandNen CL teet
=0.15
2-
-0.25
15
59
75 Epoch
100
125
150
25
50
75
100
125
150
-24
-02
0.0
0.2
0.4
Epoch
Palress grad kent cosine similar
kernel size:
train
Smalliket
10
Smaliee
leat
ExpandNee-FC
Expanohaat-CC train
Me gradient cosns simtanity
Expand
Expanotict FC test
-0.05
Exp CK
30
Espenchle--3 train
ErrorS)
ExpanatittOC train
Dverkity
25-
Espancliet-X tea
-0.15
15
10
-0.25
150
-0.
grad simllarity
(b) kernel size:
40
SmallNet train
Smaliietee
SmallNet
ahere by dbc
35
Expanotiet-FC train
Min grodenn coanne simiarrty
ExpanoNet-CC NepandiNee-CK
Espanatlet-IC test
Expanotict CL train
Expanotict CL test
+0.05
Expanctlet-Ox tain
20
-0.15
15
-0.20
25
50
75 Epoch
100
125
150
25
50
75 Epoch
300
125
150
003
-22
0.2
0.0
0.1
0.2
03
Painwis grad cosime similar
(c) kernel size:
40
SmallNet tral
Smalltet
Seaiee
SmallNee leat
ExpandNee--C
10-
Experivenc.
EspandhattFC train
Espandtlet-CC test
We proseet cusiee stmitorky
0.00
ExpansMet-ck
EspancNettCL
30
Espanstlet-I1
Enor3))
Expanstiet-OC tmin
Espaneflet-1
+0.05
Densty
25
Eipanstee--K teal
-0.10
-0.15
15
10
125
15c
25
150
Epoch
Epoch
Painwise grod eent codree simllarty
(d) kernel size:
Figure 5: Training behavior of networks on CIFAR-10. Left: Training and test curves over 150 epochs. Middle: Minimum pairwise gradient cosine similarity at the end of each training epoch. Right: Kernel density estimation of pairwise gradient cosine similarity at the end training (over independent runs).
16

---

Under review as a conference paper at ICLR 2020
70
SmalNel Irem
Smallbit Expandhar-FCC
55
Smalfiet test
ExpandNen-FCC trai
Me groient costre simtarky
0.00
Expandhlet-a
ExperdNas-CL train
-9.05
ExpandNen-C test
-0.30
-0.15
-0.20
45
59
100
125
150
25
50
75
100
125
150
-0.
0.0
0.2
Epoch
Epoch
Painiss grad lent cosine similar by
kernel size:
train
Smalliket
Smalivel
leat
ExpandNee-FC
Expanshatt-CC train
Mi gradket coo 00 simaaaity
0.05
Expano
Expanotict FC test
Net CK
60
Espencflet-2 train
Erroy/)
Expanatit-OC train
+0.05
Espanaliet-X teal
-0.15
45
40
-0.25
grad
(b) kernel size:
80
75
SmallNet train
Smaliiet
SmallNet
FC
12-
sbore by
Expanotet--CC train
Min godient coane simtartty
ExpanoNet-CC
70
Espanatlet-CC test
Expanotict CL train
Expanotict CL test
Expanctee--xx tmin
50-
45
40
25
50
75 Epoch
100
125
150
25
50
75
100
125
150
-62
0.0
0.2
Epoch
Parniis lnnt cosire similar
(c) kernel size:
90
14
SmallNet taan
0.05
Smalltet
Seaiee
75
Smalliee leat
ExpandNee--C
12-
Expariienc.
Espandhat-FC train
We prosentt cosiee siiarrky
EspancNettCL
Expandllet-CC test
0.00
ExpanoNet-ck
05
Espaneflet-1
Emor3))
60-
Expanstiet-OC tmin
-0.05
Densty
Eipanstee--KK twal
-0.10
-0.35
40
35
125
150
25
150
Epoch
Epoch
Painwise grod lent codire simllarit
(d) kernel size:
Figure 6: Training behavior networks on CIFAR-100. Left: Training and test curves over 150 epochs. Middle: Minimum pairwise gradient cosine similarity at the end of each training epoch. Right: Kernel density estimation of pairwise gradient cosine similarity at the end training (over independent runs).
17

---

Under review as a conference paper at ICLR 2020
0000N kot
200
335
2002
1:300
30300
2200
+000
2003
Figure 7: Product L Normal L (best viewed in color). Left: Training curves of the overall loss function. Middle Left: Training curves of the cross-entropy loss. Middle Right: Curves of training errors. Right: Curves of test errors. (Note that the y-axis is in log scale.)
D DISCUSSION OF (ARORA ET AL. 2018)
In this section, we discuss in more detail the work of Arora et al. (2018) to further evidence the differences with our work. In (Arora et al.) 2018), the authors only worked with linear models or linear layers (fully-connected layers). By contrast, we focus practical, nonlinear, compact convolutional networks, and we propose 2 ways to expand convolutional layers, which has not been studied before. Our convolutional linear expansion strategies yield better solutions with higher accuracy, more zero-centered gradient cosine similarity during training and minima that generalize better.
In their paper, Arora et al. performed a sanity test on MNIST with a CNN, by only expanding the conne-co layers. According than the compact network is difficult. Therefore, we used their code to reproduce their results, and found that, in their setting, the over-parameterrize model yields higher test error. While analyzing the reasons for this, we observed that, in (Arora et al. 2018), given the input (X, Y), the loss function they used was defined as
experiments, with Expand-FC FC ONLY, getting better results
Loss -CE(X (IB(XXY)) Y) + AR,
(3)
R = WW
(4)
= Σ || jeal...
(5)
where Wf.. I1, W fol, represents the product of the parameter matrices of the i-th over- parameterized fully-connected layer. CE denotes the cross-entropy loss and || lla is the L" norm. By contrast, the typical 2 norm regularization used in deep learning is
= W fell2
(6)
ΣΣ
W Sea
which sums the square loss of the individual parameter matrices.
The 2 norm regularizer used by Arora et al. 2018) imposes weaker constraints on the individ- ual parameter matrices, and we observed their pvrrpaaraeterrize model converge to a worse minimum during training and lead to worse test performance.
To evidence this, shown in Figure we compare the original model with an over-parameterized one as in (Arora et al. 2018), and with an wver-parameterized network with normal L regularization (corresponding to our ExpandNet-FCC.. Even though the overall loss of Arora et al.) (2018)ss 's over- parameterized model decreases faster than that of the baseline, the cross-entropy loss, the training emror and the test error do not show the same trend. The test error of the original model, /Arora al. 22088'ss parrpaaramteriiee model with product L norm and our ExpandNet-FC with nor- mal L norm are 0.9%. 1.1% and 0.8%, respectively. Furthermore, we also compare Arora et al. 2018 's over- perrattreeter model and our ExpandNet-Fe with expansion rate 2. We observed that (Arora et al. (2018)s over wer-parameterized model performs even worse with a larger expansion rate while our Expand-FC FC works well.
In short, the faster convergence suggested by Arora et al. (2018) seems to arise from the use of a different regularizer and does not lead better test performance.
18

---

Under review as a conference paper at ICLR 2020
Table 14: Top-1 I accuracy (%) of compact networks initialized with different ExpandNets on CIFAR- 10, CIFAR-100 100 and ImageNet.
Network
Initialization
CIFAR-10
CIFAR-I00
Normal
78.63 + 0.41
46.63 ::2 0.27
ExpandNet-FC
79.09 0.56
46.52 + 0.36
ExpandNet-CU CL
78.65 0.36
46.65 0.47
SmallNet
ExpandNet-CL-tee
78.81 0.52
46.43
0.72
ExpandNeet-CK
78.84 0.30
46.56 0.23
ExpandNet-CK-FCC
79.27 0.29
46.62 0.29
ExpandNet-CK+FF
Normal
80.31 + 0.27
48.62 + 0.47
Initialization
ImageNet
MobileNet MobileNet
Normal
66.48
ExpandNet-CL
66.44
ExpandNet-Cl.
Normal
69.40
MobileNetV2
Normal
63.75
MobileNerV2
ExpandNet-CL
63.07
ExpandNet-CI.
Normal
65.62
ShuffleNeV2 0,5x
Normal
56.89
ShuffleNetV2 0.5x
ExpandNet-CI.
56.91
ExpandNet-CL.
Normal
57.38
E ANALYSIS OF OUR EXPANDNETS
In this section, we conduct additional ablation study to reject some possible hypotheses and further evidence that the improvement of our approach is due to our expansion strategies.
Hypothesis The improvement comes from initialization with the well-trained non-linear counter- part of the expanded network.
Based on the same experimental setting as Table I] we used the well-trained nonlinear coun- terparts to initialize the compact networks by computing the linear product of the expanded lay- ers before training. The compact networks initialized using nonlinear ExpandNet- MIaiNNet-CI+FF CLC achieve 77.02 + 0.35% (vs 79.98 + 0.28% using our initialization) on CIFAR-10 and 39.39 + 1.08% (vs 47.98+00.48% using our initialization) on CIFAR-100. Those initialized using nonlinear ExpandNet- CK+FC achieve 75.81 + 0.34% (vs 80.81 + 0.27% using initialization) on CIFAR-10 and 34.88 + 1.41% (vs 49.82 + 0.25% using our initialization) on CIFAR-100.
Both experiments reveal that, while our initialization helps improve our ExpandNet, using it to initialize the compact network is not effective.
Hypothesis 2: The improvement comes from initialization with the linear product of the expanded networks.
Following the experimental setting of Table I and Table 2. we initialize the compact networks with different expansion strategies on CIFAR-10, CIFAR- 100 and ImageNet, respectively.
From reported results in Table 14 and Table I initialized by Expand-FC and Expand-CL yield slightly better results than training the ExpandNets by normal initialization. However, the same trend does not oceur CIFAR-100 and ImageNet, where, with ExpandNet-CI. initialization, MobileNet (90 epochs) gets 66.44% (vs 66.48% for the baseline), MobileNetV2 (90 epochs) gets 63.07% (vs 63.75% for the baseline) and ShuffleNet gets 56.91% % (vs 56.89% for the baseline). Moreover, compact networks initialized by ExpandNet-CK always yield worse results than training ExpandNet-CKs from scratch.
that, CIFAR-10, 10, compact networks
Therefore, we believe that the benefits of our expansion strategies cannot be obtained by initializa- tion using ExpandNets.
19


Reviews:
[{"review_id": "SyxThYwDYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is extremely interesting and quite surprising. In fact, the major claim is that using a cascade of linear layers instead of a single layer can lead to better performance in deep neural networks. As the title reports, expanding layers seems to be the key to obtain extremely interesting results. Moreover, the proposed approach is extremely simple and it is well explained in Section 2 with equations (1) and (2). This paper can have a tremendous impact in the research in deep networks if results are well explained.\n\nHowever, in its present form, it is hard to understand why the claim is correct. In fact, the model presented in the paper has a major obscure point. Equation (1) and (2) are extremely clear. Without non-linear functions, equations (1) and (2) describe a classical matrix factorization like Principal Component Analysis. Now, if internal matrices have more dimensions of the rank of the original matrix, the product of the internal matrices is exactly the original matrix. Whereas, if internal matrices have a number of dimensions lower than the rank of the original matrix, these matrices act as filters on features or feature combination. Since the authors are using inner matrices with a number of dimensions higher than the number of dimensions of the original matrix, there is no approximation and, then, no selection of features or feature combinations. Hence, without non-linear functions, where is the added value of the method? How the proposed method can have better results. \nThere are some possibilities, which have not been explored:\n1) the performance improvement derives from the approximation induced by the representation of float or double in the matrices. The approximation act as the non-linear layers among linear layers.\n2) the real improvement seems to be given by the initialization which has been obtained by using the non-linear counterpart of the expansion; to investigate whether this is the case, the model should be compared with a compact model where the initialization is obtained by using the linear product of the non-linear counterpart of the expanded network. If this does not lead to the same improvement, there should be a value in the expansion.\n3) the small improvement of the expanded network can be given by the different initialization. In fact, each composing matrix is initialized randomly. The product of a series of randomly initialized matrices can lead to a matrix that is initialized with a different distribution where, eventually, components are not i.i.d.. To show that this is not relevant, the authors should organize an experiment where the original matrix (in the small network) is initialized with the dot product of the composing matrices. The training should be done by using the small network. If results are significantly different, then the authors can reject the hypothesis.\nIf the authors can reject (1), (2) and (3), they should find a plausible explaination why performance improves in their experiments.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Hkef7hN4qH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes linear over-parameterization methods to improve training of small neural network models. The idea is simple -- each linear transformation in a network is overparameterized by a series of linear transformation which is algebraically equivalent to the original linear transformation. Number of experiments are conducted to show the effectiveness of the approach.\n\nThe proposed method is a simple application of over-parameterization to improve neural network model training. The motivation is clear and the proposed method is clearly presented. The paper is easy to understand and follow. Great analyses on the training behavior and generalization ability are conducted. Given the simplicity of the method, this could be a standard way of training small neural network model if the effectiveness of the method is observed more widely.\n\nThese are some concerns on the paper:\n\n1) The effectiveness of the approach is not necessarily significant in all experiments. For example, in Table 1, ExpandNet-FC and ExpandNet-CL were not effective. The same trend is observed in Table 2 for 400 epochs. Given that only ExpandNet-CK improves the performance, we could conclude that some intrinsic property of CK is important than over-parameterization. The good results for 90 epochs in Table 2 may mean linear over-parameterization yields faster convergence as suggested by Arora et al. 2018.\n\n2) The comparisons are not extensive. For example, we do not see Init for all models and \"w/ KD\" for ShuffleNetV2 in Table 2. Table 2 has \"N/A\". Table 3 and 4 do not have results with \"w/ KD\".  Knowledge transfer methods should be the baseline of the paper.\n\nMinor comments:\n\nIt will be interesting to see the results of the models used for Init.\n\nIt might be interesting to conduct experiments with a big model and see if we do not have any gains.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper develops linear over-parameterization methods to improve training of small neural network models. This is compared to training from scratch and other knowledge distillation methods. 

Reviewer 1 found the paper to be clear with good analysis, and raised concerns on generality and extensiveness of experimental work. Reviewer 2 raised concerns about the correctness of the approach and laid out several other possibilities. The authors conducted several other experiments and responded to all the feedback from the reviewers, although there was no final consensus on the scores.

The review process has made this a better paper and it is of interest to the community. The paper demonstrates all the features of a good paper, but due to a large number of strong papers, was not accepted at this time.
Decision: Reject