ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks

1 INTRODUCTION . With the growing availability of large-scale datasets and advanced computational resources , deep learning has achieved tremendous success in a variety of computer vision tasks , such as image classification ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) , object detection ( Ren et al. , 2015 ; Redmon & Farhadi , 2018 ; 2016 ) and semantic segmentation ( Long et al. , 2015 ; Ronneberger et al. , 2015 ) . Over the past few years , “ Wider and deeper are better ” has become the rule of thumb to design network architectures ( Simonyan & Zisserman , 2015 ; Szegedy et al. , 2015 ; He et al. , 2016 ; Huang et al. , 2017 ) . This trend , however , raises memory- and computation-related challenges , especially in the context of constrained environments , such as embedded platforms . Deep and wide networks are well-known to be heavily over-parameterized , and thus a compact network , both shallow and thin , should often be sufficient . Unfortunately , compact networks are notoriously hard to train from scratch . As a consequence , designing strategies to train a given compact network has become increasingly popular , the most popular approach consisting of transferring the knowledge of a deep teacher network to the compact one of interest ( Hinton et al. , 2015 ; Romero et al. , 2014 ; Yim et al. , 2017a ; Zagoruyko & Komodakis , 2017 ; Passalis & Tefas , 2018 ) . In this paper , we introduce an alternative approach to training compact neural networks , complementary to knowledge transfer . To this end , building upon the observation that network overparameterization improves both optimization and generalization ( Arora et al. , 2018 ; Zhang et al. , 2017 ; Reed , 1993 ; Allen-Zhu et al. , 2018a ; b ; Kenji Kawaguchi & Bengio , 2018 ) , we propose to increase the number of parameters of a given compact network by incorporating additional layers . However , instead of separating every two layers with a nonlinearity , as in most of the deep learning literature , we advocate introducing consecutive linear layers . In other words , we expand each linear layer of a compact network into a succession of multiple linear layers , without any nonlinearity in between . Since consecutive linear layers are equivalent to a single one ( Saxe et al. , 2014 ) , such an expanded network , or ExpandNet , can be algebraically compressed back to the original compact one without any information loss . While the use of successive linear layers appears in the literature , existing work ( Baldi & Hornik , 1989 ; Saxe et al. , 2014 ; Kawaguchi , 2016 ; Laurent & von Brecht , 2018 ; Zhou & Liang , 2018 ; Arora et al. , 2018 ) has been mostly confined to networks without any nonlinearities and the theoretical study of their behavior under statistical assumptions that typically do not hold in practice . In particular , these studies have attempted to understand the learning dynamics and the loss landscapes of deep networks . Here , by contrast , we focus on practical , nonlinear , compact convolutional neural networks , and investigate the use of linear expansion to introduce over-parameterization and improve training , so as to allow such networks to achieve better performance . Specifically , as illustrated by Figure 1 , we introduce three ways to expand a compact network : ( i ) replacing a fully-connected layer with multiple ones ; ( ii ) replacing a k× k convolution by three convolutional layers with kernel size 1×1 , k × k and 1 × 1 , respectively ; and ( iii ) replacing a k× k convolution with k > 3 by multiple 3 × 3 convolutions . Our experiments demonstrate that expanding convolutions , which has not been done before , is key to obtaining more effective compact networks . Furthermore , we introduce a natural initialization strategy for our ExpandNets . Specifically , an ExpandNet has a nonlinear counterpart , with an identical number of parameters , obtained by adding nonlinearity between every two consecutive linear layers . We therefore propose to directly transfer the weights of this nonlinear model to the ExpandNet for initialization . We show that this yields a further performance boost for the resulting compact network . In short , our contributions are ( i ) a novel approach to training given compact , nonlinear convolutional networks by expanding their linear layers ; ( ii ) two strategies to expand convolutional layers ; and ( iii ) an effective initialization scheme for the resulting ExpandNets . We demonstrate the benefits of our approach on several tasks , including image classification on ImageNet , object detection on PASCAL VOC and image segmentation on Cityscapes . Our ExpandNets consistently outperform training the corresponding compact networks from scratch and using knowledge distillation . We further analyze the benefits of linear over-parametrization on training via a series of experiments studying generalization , gradient confusion and the loss landscapes . We will make our code publicly available . 2 EXPANDNETS . Let us now introduce our approach to training compact networks by expanding their linear layers . In particular , we propose to make use of linear expansion , such that the resulting ExpandNet is equivalent to the original compact network and can be compressed back to the original structure algebraically . Below , we describe three different expansion strategies , starting with the simplest case of fully-connected layers , followed by two ways to expand convolutional layers . Expanding fully-connected layers . The weights of a fully-connected layer can easily be represented in matrix form . Therefore , expanding such layers can be done in a straightforward manner by relying on matrix product . Specifically , let Wn×m be the parameter matrix of a fully-connected layer with m input channels and n output ones . That is , given an input vector x ∈ Rm , the output y ∈ Rn can be obtained as y = Wn×mx . Note that we ignore the bias , which can be taken into account by incorporating an additional channel with value 1 to x . Expanding such a fully-connected layer with an arbitrary number of linear layers can be achieved by observing that its parameter matrix can be equivalently written as Wn×m = Wn×pl ×Wpl×pl−1 × · · · ×Wp1×m . ( 1 ) While this yields an over-parametrization , it does not affect the underlying model capacity . More importantly , this allows us to increase not only the number of layers , but also the number of channels by setting pi > n , m for all i . To this end , we rely on the notion of expansion rate . Specifically , for an expansion rate r , we define p1 = rm and pi = rn , ∀i 6= 1 . Note that other strategies are possible , e.g. , pi = rim , but ours has the advantage of preventing the number of parameters from exploding . In practice , considering the computational complexity of fully-connected layers , we advocate expanding each layer into only two or three layers with a small expansion rate . Expanding convolutional layers . The operation performed by a convolutional layer can also be expressed in matrix form , by vectorizing the input tensor and defining a highly structured matrix whose elements are obtained from the vectorized convolutional filters . While this representation could therefore allow us to use the same strategy as with fully-connected layers , using arbitrary intermediate matrices would ignore the convolution structure , and thus alter the original operation performed by the layer . For a similar reason , one can not simply expand a convolutional layer with kernel size k × k with a series of k × k convolutions , since , unless k = 1 , the resulting receptive field size would differ from the original one . To overcome this , we note that 1×1 convolutions retain the computational benefits of convolutional layers while not modifying the receptive field size . As illustrated in Figure 1 , we therefore propose to expand a k×k convolutional layer into 3 consecutive convolutional layers : a 1×1 convolution ; a k× k one ; and another 1× 1 one . This strategy still allows us to increase the number of channels in the intermediate layer . Specifically , for an original layer with m input channels and n output ones , given an expansion rate r , we define the number of output channels of the first 1×1 layer as p = rm and the number of output channels of the intermediate k × k layer as q = rn . Compressing an expanded convolutional layer into the original one can still be done algebraically . To this end , one can reason with the matrix representation of convolutions . For an input tensor of size ( w × h×m ) , the matrix representation of the original layer can be recovered as Wnv×mv = Wnv×qv ×Wqv×pv ×Wpv×mv , ( 2 ) where v = w · h and each intermediate matrix has a structure encoding a convolution . It can be verified that the resulting matrix will also have a convolution structure , with filter size k × k. Expanding convolutional kernels . While 3× 3 kernels have become increasingly popular in very deep architectures ( He et al. , 2016 ) , larger kernel sizes are often exploited in compact networks , so as to increase their expressiveness and their receptive fields . As illustrated in Figure 1 , k×k kernels with k > 3 can be equivalently represented with a series of l 3×3 convolutions , where l = k−12 . As before , the number of channels in the intermediate 3×3 layers can be larger than that in the original k× k one , thus allowing us to linearly over-parametrize the model . Similarly to the fully-connected case , for an expansion rate r , we set the number of output channels of the first 3×3 layer to p1 = rm and that of the subsequent layers to pi = rn . The same matrix-based strategy as before can then be used to algebraically compress back the expanded kernels . Dealing with convolution padding and strides . In modern convolutional networks , padding and strides are widely used to retain information from the input feature map while controlling the size of the output one . To expand a convolutional layer with padding p , we propose to use padding p in the first layer of the expanded unit while not padding the remaining layers . Furthermore , to handle a stride s , when expanding convolutional layers , we set the stride of the middle layer to s and of the others to 1 . When expanding convolutional kernels , we use a stride 1 for all layers except for the last one whose stride is set to s. These two strategies guarantee that the resulting ExpandNet can be compressed back to the original compact model without any information loss . Overall , the strategies discussed above allow us to expand an arbitrary compact network into an equivalent deeper and wider one . Note that these strategies can be used independently or together . In any event , once the resulting ExpandNet is trained , it can be compressed back to the original compact architecture in a purely algebraic manner , that is , at absolutely no loss of information . 2.1 INITIALIZING EXPANDNETS . As will be demonstrated by our experiments , training an ExpandNet from scratch yields consistently better results than training the original compact network . However , in the context of deep networks , initialization can have an important effect on the final results ( Mishkin & Matas , 2015 ; He et al. , 2015 ) . While designing an initialization strategy specifically for compact networks is an unexplored and challenging research direction , our ExpandNets can be initialized in a very natural manner . To this end , we exploit the fact that an ExpandNet has a natural nonlinear counterpart , which can be obtained by incorporating a nonlinear activation function between each pair of linear layers . We therefore propose to initialize the parameters of an ExpandNet by simply training its nonlinear counterpart and transferring the resulting parameters to the ExpandNet . The initialized ExpandNet is then trained in the standard manner . As evidenced by our experiments below , when the nonlinear counterpart achieves better performance than the compact network , this simple strategy typically yields an additional accuracy boost to our approach .

Summary: This paper is extremely interesting and quite surprising. In fact, the major claim is that using a cascade of linear layers instead of a single layer can lead to better performance in deep neural networks. As the title reports, expanding layers seems to be the key to obtain extremely interesting results. Moreover, the proposed approach is extremely simple and it is well explained in Section 2 with equations (1) and (2). This paper can have a tremendous impact in the research in deep networks if results are well explained.