Learning Deeply Shared Filter Bases for Efficient ConvNets

1 INTRODUCTION . Modern networks such as ResNets usually have massive identical convolution blocks and recent analytic studies ( Jastrzebski et al. , 2018 ) show that these blocks perform similar iterative refinement rather than learning new features . Inspired by these massive identical block structure of modern networks , recursive ConvNets sharing weights across iterative blocks have been studied as a promising direction to parameter-efficient ConvNets ( Jastrzebski et al. , 2018 ; Guo et al. , 2019 ; Savarese & Maire , 2019 ) . However , repetitive use of parameters across many convolution layers incurs several challenges that limit the performance of such recursive networks . First of all , deep sharing of parameters might result in vanishing gradients and exploding gradients problems , which are often found in recurrent neural networks ( RNNs ) ( Pascanu et al. , 2013 ; Jastrzebski et al. , 2018 ) . Another challenge is that overall representation power of the networks might be limited by using same filters repeatedly for many convolution layers . To address aforementioned challenges , in this paper , we propose an effective and efficient parametersharing mechanism for modern ConvNets having many repetitive convolution blocks . In our work , convolution filters are decomposed into a fundamental and reusable unit , which is called a filter basis , and a layer-specific part , which is called coefficients . By sharing a filter basis , not whole convolution filters or a layer , we can impose two desirable properties on the shared parameters : ( 1 ) resilience against vanishing/exploding gradients , and ( 2 ) representational expressiveness of individual layers sharing parameters . We first show theoretically that a shared filter basis can cause vanishing gradients and exploding gradients problems , and this problem can be controlled to a large extent by making filter bases orthogonal . To enforce the orthogonality of filter bases , we propose an orthogonality regularization to train ConvNets having deeply shared filter bases . Our experimental results show that the proposed orthogonality regularization reduces the redundancy not just in deeply shared filter bases , but also in none-shared parameters , resulting in better performance than over-parameterized counterpart networks . Next , we make convolution layers with shared parameters more expressive using a hybrid approach to sharing filter bases , in which a small number of layer-specific non-shared filter basis components are combined with shared filter basis components . With this hybrid scheme , the constructed filters can be positioned in different vector subspaces that reflect the peculiarity of individual convolution layers . We argue that these layer-specific variations contribute to increasing the representation power of the networks when a large portion of parameters is shared . Since our focus is not on pushing the state-of-the-art performance , we show the validity of our work using widely-used ResNets as a base model on image classification tasks with CIFAR and ImageNet datasets . Our experimental results demonstrate that when each filter basis is shared by up to 10 convolution layers , our method consistently outperforms counterpart ConvNet models while reducing a significant amount of parameters and computational costs . For example , our method can save up to 63.8 % of parameters and 33.4 % of FLOPs , respectively , while achieving lower test errors than much deeper counterpart models . Our parameter sharing structure and training mechanism can be applied to modern compact networks , such as MobileNets ( Howard et al. , 2017 ) and ShuffleNets ( Zhang et al. , 2018 ) with minor adaptations . Since these compact models already have decomposed convolution blocks , some parts of each block can be identified as a shareable filter basis and the rest of the layer-specific parts . In Experiments , we demonstrate that compact MobileNetV2 can achieve further 8-21 % parameter savings with our scheme while retaining , or improving , the performance of the original models . 2 RELATED WORK . Recursive networks and parameter sharing : Recurrent neural networks ( RNNs ) ( Graves et al. , 2013 ) have been well-studied for temporal and sequential data . As a generalization of RNNs , recursive variants of ConvNets are used extensively for visual tasks ( Socher et al. , 2011 ; Liang & Hu , 2015 ; Xingjian et al. , 2015 ; Kim et al. , 2016 ; Zamir et al. , 2017 ) . For instance , Eigen et al . ( 2014 ) explore recursive convolutional architectures that share filters across multiple convolution layers . They show that recurrence with deeper layers tends to increase performance . However , their recursive architecture shows worse performance than independent convolution layers due to overfitting . In most previous works , filters themselves are shared across layers . In contrast , we propose to share filter bases that are more fundamental and reusable building blocks to construct layer-specific filters . More recently , Jastrzebski et al . ( 2018 ) show that iterative refinement of features in ResNets suggests that deep networks can potentially leverage intensive parameter sharing . Guo et al . ( 2019 ) introduce a gate unit to determine whether to jump out of the recursive loop of convolution blocks to save computational resources . These works show that training recursive networks with naively shared blocks leads to bad performance due to the problem of gradient explosion and vanish like RNN ( Pascanu et al. , 2013 ; Vorontsov et al. , 2017 ) . In order to mitigate the problem of gradient explosion and vanish , they suggest unshared batch normalization strategy . In our work , we propose an orthogonality regularization of shared filter bases to further address this problem . Savarese & Maire ( 2019 ) ’ s work is also relevant to our work . In their work , the parameters of recurrent layers of ConvNets are generated by a linear combination of 1-2 parameter tensors from a global bank of templates . Though similar to our work , our work suggests more fine-grained filter bases as more desirable building blocks for effective parameter sharing since filter bases can be easily combined with layer-specific non-shared components for better representation power . Our result shows that these layer-specific non-shared components are critical to achieve high performance . Although they achieve about 60 % parameter savings , their approach does not outperform counterpart models and incurs slight increases in computational costs due to the overheads in reparameterizing tensors from the templates . Model compression and efficient convolution block design : Reducing storage and inference time of ConvNets has been an important research topic for both resource constrained mobile/embedded systems and energy-hungry data centers . A number of research techniques have been developed such as filter pruning ( LeCun et al. , 1990 ; Polyak & Wolf , 2015 ; Li et al. , 2017 ; He et al. , 2017 ) , low-rank factorization ( Denton et al. , 2014 ; Jaderberg et al. , 2014 ) , quantization ( Han et al. , 2016 ) , and knowledge distillation ( Hinton et al. , 2015 ; Chen et al. , 2017 ) , to name a few . These compression techniques have been suggested as post-processing steps that are applied after initial training . Unfortunately , their accuracy is usually bounded by the approximated original models . By contrast , our models are trained from scratch as in Ioannou et al . ( 2017 ) ’ s work and our result shows that parameter-sharing approaches can outperform the counterpart models , if a proper training method is combined , while achieving significant savings in parameters . Some compact networks such as ShuffleNet ( Zhang et al. , 2018 ) and MobileNet ( Howard et al. , 2017 ; Sandler et al. , 2018 ) show that delicately designed internal structure of convolution blocks acquire better ability with lower computational complexity . Our work can be applied to these compact networks since they already exploit decomposed convolutions for their efficient convolution blocks . For instance , since MobileNet ’ s convolution blocks have pointwise-depthwise-pointwise convolution steps , the first two convolution steps can be considered a reusable filter basis and the last step can be considered layer-specific coefficients . In Experiments , we show that MobileNetV2 combined with our parameter-sharing scheme outperforms the original models while saving about 8-21 % parameters . 3 DEEP RECURSIVE SHARING OF A FILTER BASIS . In this section , we discuss how to decompose typical convolution layers into more recursive units , or filter bases , and remaining layer-specific parts . We also discuss how to train ConvNets effectively when filter bases are deeply shared by repetitive convolution layers . 3.1 FILTER BASES OF CONVOLUTION LAYERS . We assume that a convolution layer with S input channels , T output channels , and a set of filters W = { Wt ∈ Rk×k×S , t ∈ [ 1 .. T ] } . Each filter Wt can be decomposed using a lower rank filter basis Wbasis and coefficients α : Wt = R∑ r=1 αrtW r basis , ( 1 ) where Wbasis = { W rbasis ∈ Rk×k×S , r ∈ [ 1 .. R ] } is a filter basis , and α = { αrt ∈ R , r ∈ [ 1 .. R ] , t ∈ [ 1 .. T ] } is scalar coefficients . In Equation 1 , R is the rank of the basis . In a typical convolution layer , output feature maps Vt ∈ Rw×h×T , t ∈ [ 1 .. T ] are obtained by the convolution between input feature maps U ∈ Rw×h×S and filters Wt , t ∈ [ 1 .. T ] . With Equation 1 , this convolution can be rewritten as follows : Vt = U ∗Wt = U ∗ R∑ r=1 αrtW r basis ( 2 ) = R∑ r=1 αrt ( U ∗W rbasis ) , where t ∈ [ 1 .. T ] . ( 3 ) In Equation 3 , the order of the convolution operation and the linear combination of filter basis is reordered according to the linearity of convolution operators . This result shows that a standard convolution layer can be replaced with two successive convolution layers as shown in Figure 1- ( b ) . The first decomposed convolution layer performs R convolutions between W rbasis , r ∈ [ 1 .. R ] and input feature maps U , and it generates an intermediate feature map basis Vbasis ∈ Rw×h×R . The second decomposed convolution layer performs point-wise 1×1 convolutions that linearly combineR intermediate feature maps Vbasis to generate output feature maps V . The computational complexity of the original convolution is O ( whk2ST ) while the decomposed operations take O ( wh ( k2SR+RT ) ) . As far as R < T , the decomposed convolution has lower computational complexity than the original convolution . Due to this computational efficiency , many compact networks such as MobileNets and ShuffleNets also have similar block structures of decomposed convolution layers . For instance , MobileNets have repetitive convolution blocks of pointwise-depthwise-pointwise convolutions . The filters in the first two steps can be considered a reusable filter basis and the remaining 1x1 filters in the last step can be considered layer-specific coefficients . 3.2 RECURSIVE SHARING OF A FILTER BASIS . In typical ConvNets , convolution layers have different filters W s and , hence , each decomposed convolution layer has its own filter basis Wbasis and coefficients α . In contrast , our primary goal in decomposing convolution layers is to share a single filter basis ( or a small number of filter bases ) across many recursive convolution layers . Unlike some previous works ( Jastrzebski et al. , 2018 ; Köpüklü et al. , 2019 ) , in which convolution filters W themselves are shared recursively , we argue that a filter basis Wbasis is a more intrinsic and reusable building block that can be shared effectively since a filter basis constitutes a subspace , in which high dimensional filters across many convolution layers can be approximated . Though components of a basis only need to be independent and span a vector subspace , some specific bases are more convenient and appropriate for specific purposes . For the purpose of sharing a filter basis , we need to find an optimal filter basis Wbasis that can expedite the training of filters of shared convolution layers . Although this optimization can be done with a typical stochastic gradient descent ( SGD ) , one problem is that exploding/vanishing gradients problems might prevent efficient search of the optimization space . More formally , we consider a series of N decomposed convolution layers , in which a filter basis Wbasis is shared N times . Let xi be the input of the i-th convolution layer , and ai+1 be the output of the convolution of xi with the filter basis Wbasis ai ( xi−1 ) = W > basisx i−1 . ( 4 ) In Equation 4 , Wbasis ∈ Rk 2S×R is a reshaped filter basis that has basis components at its columns . We assume that input x is properly adapted ( e.g. , with im2col ) to express convolutions using a matrix-matrix multiplication . Since Wbasis is shared across N recusrive convolution layers , the gradient of Wbasis for some loss function L is : ∂L ∂Wbasis = N∑ i=1 ∂L ∂aN N−1∏ j=i ( ∂aj+1 ∂aj ) ∂ai ∂Wbasis , ( 5 ) , where ∂aj+1 ∂aj = ∂aj+1 ∂xj ∂xj ∂aj = Wbasis ∂xj ∂aj ( 6 ) If we plug Wbasis ∂x j ∂aj in Equation 6 into Equation 5 , we can see that ∏ ∂aj+1 ∂aj is the term that makes gradients unstable since Wbasis is multiplied many times . This exploding/vanishing gradients can be controlled to a large extent by keeping Wbasis close to orthogonal ( Vorontsov et al. , 2017 ) . For instance , if Wbasis admits eigendecomposition , [ Wbasis ] N can be rewritten as follows : [ Wbasis ] N = [ QΛQ−1 ] N = QΛNQ−1 , ( 7 ) where Λ is a diagonal matrix with the eigenvalues placed on the diagonal and Q is a matrix composed of the corresponding eigenvectors . If Wbasis is orthogonal , [ Wbasis ] N neither explodes nor vanishes , since all the eigenvalues of an orthogonal matrix have absolute value 1 . Similarly , an orthogonal shared basis ensures that forward signals neither explodes nor vanishes . We also need to ensure that the norm of ∂x j ∂aj in Equation 5 is bounded ( Pascanu et al. , 2013 ) for stability during forward and backward passes . It is shown that batch normalization after non-linear activation at each convolution layer ensures healthy norms ( Ioffe & Szegedy , 2015 ; Guo et al. , 2019 ; Jastrzebski et al. , 2018 ) . For training networks , the orthogonality of shared bases can be enforced with an orthogonality regularizer . For instance , when each residual block group of a ResNet shares a filter basis for its convolution layers , the objective function LR can be defined to have an orthogonality regularizer in addition to the original loss L : LR = L+ λ G∑ g ‖W ( g ) basis > ·W ( g ) basis − I‖ 2 , ( 8 ) where W ( g ) basis is a shared filter basis for g-th residual block group and λ is a hyperparameter .

Summary: This paper addresses the problem of obtaining more compact CNNs by a parameter sharing method. The authors propose to represent a weight filter in a low-rank subspace (represented as a linear combination of low-rank filter basis) plus a set of non-shared low-rank filter basis (per-layer). In this way, the shared low-rank filter basis is reused across several layers, and the non-shared ones per layer are used to enhance model generalization ability. Experiments are performed on CIFAR and ImageNet datasets, using some popular CNN structures for evaluation.