Deeply Shared Filter Bases for  Parameter-Efficient  Convolutional Neural Networks

1 Introduction . Modern convolutional neural networks ( CNNs ) such as ResNets have massive identical convolution blocks and recent analytic studies ( Jastrzebski et al. , 2018 ) show that these blocks perform mostly iterative refinement of features rather than learning new features . Inspired by these results , recursive sharing of weights has been studied as a promising direction to parameter-efficient CNNs ( Jastrzebski et al. , 2018 ; Guo et al. , 2019 ; Savarese & Maire , 2019 ) . However , naive sharing of parameters across many convolution layers incurs several problems . First of all , recursive sharing of parameters can result in the vanishing and the exploding gradients problem , which is one of the main reasons that recurrent neural networks ( RNNs ) are so hard to train properly ( Pascanu et al. , 2013 ; Jastrzebski et al. , 2018 ) . Another problem is that overall representation power can be limited by iterative sharing of parameters . Due to these challenges , most compression approaches based on parameter-sharing suffer from performance degradation . In this work , we conjecture that convolution layers or blocks can be separated into inherently shareable parts and non-shareable parts , and can be trained effectively by avoiding the vanishing and the exploding gradients problem . To achieve this , for a full convolution operator , we first replace it with a factorized version that splits the convolution operator into two separate operators ; one ∗W . Kang and D. Kim are contributed equally . This work was conducted at Incheon National University . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . operator with inherently shareable filters , called a filter basis , and the other operator with non-shared filters , called coefficients . When successive convolution blocks share a common filter basis , they are positioned in the same vector subspace . However , their representation capability is retained through non-shared coefficients that learn diverse features by linearly combining the shared filter basis . By separating shareable parts from non-shareable parts , we can impose desirable properties on the shared parameters . To avoid performance degradation from recursive sharing of parameters , we propose the orthogonality regularization , in which the vanishing/exploding gradients problem is controlled by enforcing the elements of a shared filter basis orthonormal during training . We both theoretically and empirically show that the proposed orthogonality regularization improves the flow of the gradients during training and reduces the redundancy in parameters effectively . For efficient CNNs such as MobileNets ( Howard et al. , 2017 ) , we do not need to factorize convolution operators to uncover a shared filter basis since these networks already have factorized convolution block structures for computational efficiency . For such networks , our approach can be applied simply by identifying one or two convolution operators of repeating convolution blocks as a filter basis that shares weights across the repeating blocks . Other convolution operators in each convolution block become block-specific non-shared coefficients . Since our focus is not on pushing the state-of-the-art performance , we demonstrate the effectiveness of our work using widely-used models as base models on image classification and object detection . Without bells and whistles , simply applying the proposed convolution block design and the orthogonality regularization saves a significant amount of parameters while consistently outperforming over-parameterized counterpart networks . For example , our method can save up to 46.0 % of parameters of ResNets while consistently achieving lower test errors . Even in compact models , such as MobileNetV2 , our approach can achieve further 8-21 % parameter savings while outperforming the original models . This superior performance demonstrates that the proposed recursive convolution blocks and orthogonality regularization enables effective learning of better feature representations while a significant amount of parameters are shared recursively . 2 Related Work . Recursive networks and parameter sharing : Recurrent neural networks ( RNNs ) ( Graves et al. , 2013 ) have been well-studied for temporal and sequential data . As a generalization of RNNs , recursive variants of CNNs are used extensively for visual tasks ( Socher et al. , 2011 ; Liang & Hu , 2015 ; Xingjian et al. , 2015 ; Kim et al. , 2016 ; Zamir et al. , 2017 ) . For instance , Eigen et al . ( 2014 ) explore recursive convolutional architectures that share filters across multiple convolution layers . They show that recurrence with deeper layers tends to increase performance . However , their recursive architecture shows worse performance than independent convolution layers due to overfitting . In contrast , we share only fundamentally shareable parts , or filter bases , that are separated from conventional convolution operators . By separating shareable parts from non-shareable parts , we can impose desirable properties on the shared parameters that prevents the vanishing/exploding gradients problems without damaging representational capability of the networks . More recently , Jastrzebski et al . ( 2018 ) show that iterative refinement of features in ResNets suggests that deep networks can potentially leverage intensive parameter sharing . Guo et al . ( 2019 ) introduce a gate unit to determine whether to jump out of the recursive loop of convolution blocks to save computational resources . These works show that training recursive networks with naively shared blocks leads to bad performance due to the problem of gradient explosion and vanish like RNN ( Pascanu et al. , 2013 ; Vorontsov et al. , 2017 ) . In order to deal with the problem of gradient explosion and vanish , they suggest unshared batch normalization strategy . In our work , we propose the orthogonality regularization of shared filter bases to further address this problem . A few recent works generate convolution filters by combining shared building blocks ( Bhalgat et al. , 2020 ; Savarese & Maire , 2019 ; Yang et al. , 2019 ; Qiu et al. , 2018 ) . Although these approaches are different in details , they are similar to our work since they all share a set of filters and combine them to build layer-specific filters . However , they save parameters at the cost of accuracy loss . For example , in ResNet50 on ImageNet , Yang et al . ( 2019 ) ’ s work incurs 0.9 % accuracy drop , while our method achieves performance improvement . We believe that this performance gap comes from the gradients issue of shared parameters and these previous works could benefit more from the improved flow of gradients of our approach . ( a ) Full convolution ( b ) A filter basis and coefficients 𝑊𝑏𝑎𝑠𝑖𝑠 𝛼𝑊 Conv 𝛼 ( 𝑖 ) Conv 𝑾𝒃𝒂𝒔𝒊𝒔 iterate n times Shared filter basis n non-shared coefficients ( c ) Iterative sharing of a filter basis Figure 1 : A full convolution operator in ( a ) with a filterW can be replaced by two separate convolution operators , as in ( b ) : one operator with a filter basis ( Wbasis ) and the other operator with coefficients ( αs ) . A filter basis is inherently shareable for convolution operators in the same vector subspace , and , hence , a recursive architecture is suggested , as in ( c ) . The filter basis shared by recursive convolution layers can be learned through typical gradient-based training while avoiding potential vanishing/exploding gradients problem by enforcing its elements orthonormal . Model compression and efficient convolution block design : Reducing storage and inference time of CNNs has been an important research topic for both resource constrained mobile/embedded systems and energy-hungry data centers . A number of research techniques have been developed such as filter pruning ( LeCun et al. , 1990 ; Polyak & Wolf , 2015 ; Li et al. , 2017 ; He et al. , 2017b ) , low-rank factorization ( Denton et al. , 2014 ; Jaderberg et al. , 2014 ) , and quantization ( Han et al. , 2016 ) , to name a few . Several model compression techniques factorize trained filters to reduce the computation complexity ( Zhang et al. , 2015 ; Li et al. , 2019a ) . Our proposed block structure also benefit from the reduced computational complexity too by factorizing convolution filters . However , most previous compression techniques have been suggested as post-processing steps that are applied after initial training . Therefore , their accuracy is usually bounded by the approximated base models . In contrast , our primary goal in factorizing convolution operators is to find a common filter basis shared by iterative convolution layers , so the weights of the shared filter basis are learned from scratch , not from decomposing pretrained filters . Hence , unlike previous compression techniques , the performance of our approach is not limited by the original models , and the experimental results show that our parameter-sharing approach consistently outperforms the overparameterized counterpart networks on various datasets and tasks . 3 Proposed Method . In this section , we describe the design of recursive convolution blocks and discuss how to train them to deal with the vanishing/exploding gradients problem . 3.1 Recursive Sharing of a Filter Basis . Naive sharing of convolution filters across repetitive layers degrades the overall performance due to limited representation power of individual layers . We argue that convolution filters can be separated into fundamentally shareable components and non-shareable components , and the performance degradation can be prevented by sharing only fundamentally shareable parts . More formally , we consider a convolution layer , shown in Figure 1 ( a ) , that has S input channels , T output channels , and a set of filters W = { Wt ∈ Rk×k×S , t ∈ [ 1 .. T ] } . Each filter Wt can be decomposed into a filter basis Wbasis and coefficients α : Wt = R∑ r=1 αrtW r basis , ( 1 ) where Wbasis = { W rbasis ∈ Rk×k×S , r ∈ [ 1 .. R ] } is a filter basis , and α = { αrt ∈ R , r ∈ [ 1 .. R ] , t ∈ [ 1 .. T ] } is scalar coefficients . In Equation 1 , R is the rank of the basis . In a typical convolution layer , output feature maps Vt ∈ Rw×h×T , t ∈ [ 1 .. T ] are obtained by the convolution between input feature maps U ∈ Rw×h×S and filters Wt , t ∈ [ 1 .. T ] . With Equation 1 , this convolution can be rewritten as follows : Vt = U ∗Wt = U ∗ R∑ r=1 αrtW r basis = R∑ r=1 αrt ( U ∗W rbasis ) , where t ∈ [ 1 .. T ] . ( 2 ) In Equation 2 , the order of the convolution operation and the linear combination of filter basis is reordered according to the linearity of convolution operators . Therefore , a conventional convolution layer can be replaced with two successive convolution layers as shown in Figure 1- ( b ) . This factorized convolution filters suggest a recursive architecture , shown in Figure 1- ( c ) , in which a common filter basis is used for iterative convolution layers . Since all filters in the recursive loop have a common filter basis , they are positioned in the same vector subspace . However , their representation capability is still retained since their coordinates in the subspace are diversified by their respective non-shared coefficients . 3.2 Orthonormality of Shared Filter Bases . For recursive sharing of a filter basis , as in Figure 1- ( c ) , we need to find an optimal filter basis that can be used by iterative convolution layers without loss of performance . Although this optimal filter basis can be searched by typical gradient-based optimization such as SGD , one major problem is that the exploding/vanishing gradients problem of recursively shared filter bases can prevent proper search of optimization space ( Pascanu et al. , 2013 ) . More formally , we consider a series of N factorized convolution blocks , in which a filter basis Wbasis is shared N times , as in Figure 1- ( c ) . Let xi be the input of the i-th convolution block , and ai+1 be the output of the convolution of xi with the filter basis Wbasis ai ( xi−1 ) = W > basisx i−1 . ( 3 ) In Equation 3 , Wbasis ∈ Rk 2S×R is a reshaped filter basis that has basis elements at its columns . We assume that input x is properly adapted ( e.g. , with im2col operations ) to express convolutions using a matrix-matrix multiplication . Since Wbasis is shared by N recursive convolution blocks , the gradient of Wbasis for some loss function L is : ∂L ∂Wbasis = N∑ i=1 ∂L ∂aN N−1∏ j=i ( ∂aj+1 ∂aj ) ∂ai ∂Wbasis , ( 4 ) , where ∂aj+1 ∂aj = ∂aj+1 ∂xj ∂xj ∂aj = Wbasis ∂xj ∂aj ( 5 ) If we plug Wbasis ∂x j ∂aj in Equation 5 into Equation 4 , we can see that ∏ ∂aj+1 ∂aj is the term that makes gradients unstable since Wbasis is multiplied many times . This exploding/vanishing gradients can be controlled to a large extent by keeping Wbasis close to orthogonal ( Vorontsov et al. , 2017 ) . For instance , if Wbasis admits eigendecomposition , [ Wbasis ] N can be rewritten as follows : [ Wbasis ] N = [ QΛQ−1 ] N = QΛNQ−1 , ( 6 ) where Λ is a diagonal matrix with the eigenvalues placed on the diagonal and Q is a matrix composed of the corresponding eigenvectors . If Wbasis is orthogonal , [ Wbasis ] N neither explodes nor vanishes , since all the eigenvalues of an orthogonal matrix have absolute value of 1 . This result shows that a shared filter basis with orthonormal elements ensures that forward and backward signals neither explode nor vanish . Based on this result , we propose the orthogonality regularization 2 to enforce orthonormality to shared filter bases during training . For instance , when convolution operators in each residual block group of a ResNet shares a filter basis , the objective function LR can be defined to have the orthogonality regularization term in addition to the original loss L : LR = L+ λ G∑ g ‖W ( g ) basis > ·W ( g ) basis − I‖ 2 , ( 7 ) 2Technically , the columns ( or rows ) of an orthogonal matrix form an orthonormal basis . However , we use the term orthogonality regularization since orthonormal bases have no proper term for their matrix-form . ( a ) Shared block for ResNet34 ( b ) Shared block for ResNet50 conv 3x3 X conv1x1 BN conv 3x3 conv 3x3 conv1x1 BN + BN ReLU ReLU conv1x1 + BN BN , ReLU conv1x1 X BN , ReLU conv 3x3 Shared filter basis Shared by repeating conv . blocks depthwise conv 3x3 + BN BN , ReLU conv1x1 X BN , ReLU conv1x1 Shared filter basis ( c ) Shared block for MobileNetV2 ReLU Shared by repeating conv . blocks Shared filter basis Shared by repeating conv . layers Figure 2 : The structure of residual blocks with a shared filter basis . For ResNet34 , two full convolution operators are factorized to uncover a shared filter basis . For ResNets with bottleneck blocks ( ResNet50/101 ) and MobileNetV2 , one or two computationally expensive convolution operators of each convolution block are selected as shared filter bases and the other convolution operators are designated as non-shared coefficients . In ( a ) , the filter basis has non-shared elements to further increase representation capability . where W ( g ) basis is a shared filter basis for g-th residual block group and λ is a hyperparameter . In Equations 4 and 5 , we also need to ensure that the norm of ∂x j ∂aj is bounded for stability during forward and backward passes ( Pascanu et al. , 2013 ) . It is shown that batch normalization after nonlinear activation at each convolution layer ensures healthy norms ( Ioffe & Szegedy , 2015 ; Guo et al. , 2019 ; Jastrzebski et al. , 2018 ) . In Section 4.4 , we empirically show that the proposed orthogonality regularization and batch normalization similarly improve the flow of gradients during training .

Summary: This paper describes a method to share parameters between iterations of a residual block.  Rather than sharing all parameters or none, a filter basis W_basis and mixture coefficients alpha are defined; the basis is shared between layers, while the coefficients are unshared.  A theoretical argument shows a potential cause of training instability due to repeated application of a shared weights term; an orthonormality regularizer is applied to the weights basis, to help alleviate vanishing/exploding gradients.  Measurements are performed on ImageNet, CIFAR and COCO, applied to a variety of resnets and mobilenet.  The resulting model uses about half as many parameters (though similar number of FLOPs) as a baseline model, and achieves similar or slightly better accuracy.