Deeply Shared Filter Bases for  Parameter-Efficient  Convolutional Neural Networks

1 Introduction . Modern convolutional neural networks ( CNNs ) such as ResNets have massive identical convolution blocks and recent analytic studies ( Jastrzebski et al. , 2018 ) show that these blocks perform mostly iterative refinement of features rather than learning new features . Inspired by these results , recursive sharing of weights has been studied as a promising direction to parameter-efficient CNNs ( Jastrzebski et al. , 2018 ; Guo et al. , 2019 ; Savarese & Maire , 2019 ) . However , naive sharing of parameters across many convolution layers incurs several problems . First of all , recursive sharing of parameters can result in the vanishing and the exploding gradients problem , which is one of the main reasons that recurrent neural networks ( RNNs ) are so hard to train properly ( Pascanu et al. , 2013 ; Jastrzebski et al. , 2018 ) . Another problem is that overall representation power can be limited by iterative sharing of parameters . Due to these challenges , most compression approaches based on parameter-sharing suffer from performance degradation . In this work , we conjecture that convolution layers or blocks can be separated into inherently shareable parts and non-shareable parts , and can be trained effectively by avoiding the vanishing and the exploding gradients problem . To achieve this , for a full convolution operator , we first replace it with a factorized version that splits the convolution operator into two separate operators ; one ∗W . Kang and D. Kim are contributed equally . This work was conducted at Incheon National University . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . operator with inherently shareable filters , called a filter basis , and the other operator with non-shared filters , called coefficients . When successive convolution blocks share a common filter basis , they are positioned in the same vector subspace . However , their representation capability is retained through non-shared coefficients that learn diverse features by linearly combining the shared filter basis . By separating shareable parts from non-shareable parts , we can impose desirable properties on the shared parameters . To avoid performance degradation from recursive sharing of parameters , we propose the orthogonality regularization , in which the vanishing/exploding gradients problem is controlled by enforcing the elements of a shared filter basis orthonormal during training . We both theoretically and empirically show that the proposed orthogonality regularization improves the flow of the gradients during training and reduces the redundancy in parameters effectively . For efficient CNNs such as MobileNets ( Howard et al. , 2017 ) , we do not need to factorize convolution operators to uncover a shared filter basis since these networks already have factorized convolution block structures for computational efficiency . For such networks , our approach can be applied simply by identifying one or two convolution operators of repeating convolution blocks as a filter basis that shares weights across the repeating blocks . Other convolution operators in each convolution block become block-specific non-shared coefficients . Since our focus is not on pushing the state-of-the-art performance , we demonstrate the effectiveness of our work using widely-used models as base models on image classification and object detection . Without bells and whistles , simply applying the proposed convolution block design and the orthogonality regularization saves a significant amount of parameters while consistently outperforming over-parameterized counterpart networks . For example , our method can save up to 46.0 % of parameters of ResNets while consistently achieving lower test errors . Even in compact models , such as MobileNetV2 , our approach can achieve further 8-21 % parameter savings while outperforming the original models . This superior performance demonstrates that the proposed recursive convolution blocks and orthogonality regularization enables effective learning of better feature representations while a significant amount of parameters are shared recursively . 2 Related Work . Recursive networks and parameter sharing : Recurrent neural networks ( RNNs ) ( Graves et al. , 2013 ) have been well-studied for temporal and sequential data . As a generalization of RNNs , recursive variants of CNNs are used extensively for visual tasks ( Socher et al. , 2011 ; Liang & Hu , 2015 ; Xingjian et al. , 2015 ; Kim et al. , 2016 ; Zamir et al. , 2017 ) . For instance , Eigen et al . ( 2014 ) explore recursive convolutional architectures that share filters across multiple convolution layers . They show that recurrence with deeper layers tends to increase performance . However , their recursive architecture shows worse performance than independent convolution layers due to overfitting . In contrast , we share only fundamentally shareable parts , or filter bases , that are separated from conventional convolution operators . By separating shareable parts from non-shareable parts , we can impose desirable properties on the shared parameters that prevents the vanishing/exploding gradients problems without damaging representational capability of the networks . More recently , Jastrzebski et al . ( 2018 ) show that iterative refinement of features in ResNets suggests that deep networks can potentially leverage intensive parameter sharing . Guo et al . ( 2019 ) introduce a gate unit to determine whether to jump out of the recursive loop of convolution blocks to save computational resources . These works show that training recursive networks with naively shared blocks leads to bad performance due to the problem of gradient explosion and vanish like RNN ( Pascanu et al. , 2013 ; Vorontsov et al. , 2017 ) . In order to deal with the problem of gradient explosion and vanish , they suggest unshared batch normalization strategy . In our work , we propose the orthogonality regularization of shared filter bases to further address this problem . A few recent works generate convolution filters by combining shared building blocks ( Bhalgat et al. , 2020 ; Savarese & Maire , 2019 ; Yang et al. , 2019 ; Qiu et al. , 2018 ) . Although these approaches are different in details , they are similar to our work since they all share a set of filters and combine them to build layer-specific filters . However , they save parameters at the cost of accuracy loss . For example , in ResNet50 on ImageNet , Yang et al . ( 2019 ) ’ s work incurs 0.9 % accuracy drop , while our method achieves performance improvement . We believe that this performance gap comes from the gradients issue of shared parameters and these previous works could benefit more from the improved flow of gradients of our approach . ( a ) Full convolution ( b ) A filter basis and coefficients 𝑊𝑏𝑎𝑠𝑖𝑠 𝛼𝑊 Conv 𝛼 ( 𝑖 ) Conv 𝑾𝒃𝒂𝒔𝒊𝒔 iterate n times Shared filter basis n non-shared coefficients ( c ) Iterative sharing of a filter basis Figure 1 : A full convolution operator in ( a ) with a filterW can be replaced by two separate convolution operators , as in ( b ) : one operator with a filter basis ( Wbasis ) and the other operator with coefficients ( αs ) . A filter basis is inherently shareable for convolution operators in the same vector subspace , and , hence , a recursive architecture is suggested , as in ( c ) . The filter basis shared by recursive convolution layers can be learned through typical gradient-based training while avoiding potential vanishing/exploding gradients problem by enforcing its elements orthonormal . Model compression and efficient convolution block design : Reducing storage and inference time of CNNs has been an important research topic for both resource constrained mobile/embedded systems and energy-hungry data centers . A number of research techniques have been developed such as filter pruning ( LeCun et al. , 1990 ; Polyak & Wolf , 2015 ; Li et al. , 2017 ; He et al. , 2017b ) , low-rank factorization ( Denton et al. , 2014 ; Jaderberg et al. , 2014 ) , and quantization ( Han et al. , 2016 ) , to name a few . Several model compression techniques factorize trained filters to reduce the computation complexity ( Zhang et al. , 2015 ; Li et al. , 2019a ) . Our proposed block structure also benefit from the reduced computational complexity too by factorizing convolution filters . However , most previous compression techniques have been suggested as post-processing steps that are applied after initial training . Therefore , their accuracy is usually bounded by the approximated base models . In contrast , our primary goal in factorizing convolution operators is to find a common filter basis shared by iterative convolution layers , so the weights of the shared filter basis are learned from scratch , not from decomposing pretrained filters . Hence , unlike previous compression techniques , the performance of our approach is not limited by the original models , and the experimental results show that our parameter-sharing approach consistently outperforms the overparameterized counterpart networks on various datasets and tasks . 3 Proposed Method . In this section , we describe the design of recursive convolution blocks and discuss how to train them to deal with the vanishing/exploding gradients problem . 3.1 Recursive Sharing of a Filter Basis . Naive sharing of convolution filters across repetitive layers degrades the overall performance due to limited representation power of individual layers . We argue that convolution filters can be separated into fundamentally shareable components and non-shareable components , and the performance degradation can be prevented by sharing only fundamentally shareable parts . More formally , we consider a convolution layer , shown in Figure 1 ( a ) , that has S input channels , T output channels , and a set of filters W = { Wt ∈ Rk×k×S , t ∈ [ 1 .. T ] } . Each filter Wt can be decomposed into a filter basis Wbasis and coefficients α : Wt = R∑ r=1 αrtW r basis , ( 1 ) where Wbasis = { W rbasis ∈ Rk×k×S , r ∈ [ 1 .. R ] } is a filter basis , and α = { αrt ∈ R , r ∈ [ 1 .. R ] , t ∈ [ 1 .. T ] } is scalar coefficients . In Equation 1 , R is the rank of the basis . In a typical convolution layer , output feature maps Vt ∈ Rw×h×T , t ∈ [ 1 .. T ] are obtained by the convolution between input feature maps U ∈ Rw×h×S and filters Wt , t ∈ [ 1 .. T ] . With Equation 1 , this convolution can be rewritten as follows : Vt = U ∗Wt = U ∗ R∑ r=1 αrtW r basis = R∑ r=1 αrt ( U ∗W rbasis ) , where t ∈ [ 1 .. T ] . ( 2 ) In Equation 2 , the order of the convolution operation and the linear combination of filter basis is reordered according to the linearity of convolution operators . Therefore , a conventional convolution layer can be replaced with two successive convolution layers as shown in Figure 1- ( b ) . This factorized convolution filters suggest a recursive architecture , shown in Figure 1- ( c ) , in which a common filter basis is used for iterative convolution layers . Since all filters in the recursive loop have a common filter basis , they are positioned in the same vector subspace . However , their representation capability is still retained since their coordinates in the subspace are diversified by their respective non-shared coefficients . 3.2 Orthonormality of Shared Filter Bases . For recursive sharing of a filter basis , as in Figure 1- ( c ) , we need to find an optimal filter basis that can be used by iterative convolution layers without loss of performance . Although this optimal filter basis can be searched by typical gradient-based optimization such as SGD , one major problem is that the exploding/vanishing gradients problem of recursively shared filter bases can prevent proper search of optimization space ( Pascanu et al. , 2013 ) . More formally , we consider a series of N factorized convolution blocks , in which a filter basis Wbasis is shared N times , as in Figure 1- ( c ) . Let xi be the input of the i-th convolution block , and ai+1 be the output of the convolution of xi with the filter basis Wbasis ai ( xi−1 ) = W > basisx i−1 . ( 3 ) In Equation 3 , Wbasis ∈ Rk 2S×R is a reshaped filter basis that has basis elements at its columns . We assume that input x is properly adapted ( e.g. , with im2col operations ) to express convolutions using a matrix-matrix multiplication . Since Wbasis is shared by N recursive convolution blocks , the gradient of Wbasis for some loss function L is : ∂L ∂Wbasis = N∑ i=1 ∂L ∂aN N−1∏ j=i ( ∂aj+1 ∂aj ) ∂ai ∂Wbasis , ( 4 ) , where ∂aj+1 ∂aj = ∂aj+1 ∂xj ∂xj ∂aj = Wbasis ∂xj ∂aj ( 5 ) If we plug Wbasis ∂x j ∂aj in Equation 5 into Equation 4 , we can see that ∏ ∂aj+1 ∂aj is the term that makes gradients unstable since Wbasis is multiplied many times . This exploding/vanishing gradients can be controlled to a large extent by keeping Wbasis close to orthogonal ( Vorontsov et al. , 2017 ) . For instance , if Wbasis admits eigendecomposition , [ Wbasis ] N can be rewritten as follows : [ Wbasis ] N = [ QΛQ−1 ] N = QΛNQ−1 , ( 6 ) where Λ is a diagonal matrix with the eigenvalues placed on the diagonal and Q is a matrix composed of the corresponding eigenvectors . If Wbasis is orthogonal , [ Wbasis ] N neither explodes nor vanishes , since all the eigenvalues of an orthogonal matrix have absolute value of 1 . This result shows that a shared filter basis with orthonormal elements ensures that forward and backward signals neither explode nor vanish . Based on this result , we propose the orthogonality regularization 2 to enforce orthonormality to shared filter bases during training . For instance , when convolution operators in each residual block group of a ResNet shares a filter basis , the objective function LR can be defined to have the orthogonality regularization term in addition to the original loss L : LR = L+ λ G∑ g ‖W ( g ) basis > ·W ( g ) basis − I‖ 2 , ( 7 ) 2Technically , the columns ( or rows ) of an orthogonal matrix form an orthonormal basis . However , we use the term orthogonality regularization since orthonormal bases have no proper term for their matrix-form . ( a ) Shared block for ResNet34 ( b ) Shared block for ResNet50 conv 3x3 X conv1x1 BN conv 3x3 conv 3x3 conv1x1 BN + BN ReLU ReLU conv1x1 + BN BN , ReLU conv1x1 X BN , ReLU conv 3x3 Shared filter basis Shared by repeating conv . blocks depthwise conv 3x3 + BN BN , ReLU conv1x1 X BN , ReLU conv1x1 Shared filter basis ( c ) Shared block for MobileNetV2 ReLU Shared by repeating conv . blocks Shared filter basis Shared by repeating conv . layers Figure 2 : The structure of residual blocks with a shared filter basis . For ResNet34 , two full convolution operators are factorized to uncover a shared filter basis . For ResNets with bottleneck blocks ( ResNet50/101 ) and MobileNetV2 , one or two computationally expensive convolution operators of each convolution block are selected as shared filter bases and the other convolution operators are designated as non-shared coefficients . In ( a ) , the filter basis has non-shared elements to further increase representation capability . where W ( g ) basis is a shared filter basis for g-th residual block group and λ is a hyperparameter . In Equations 4 and 5 , we also need to ensure that the norm of ∂x j ∂aj is bounded for stability during forward and backward passes ( Pascanu et al. , 2013 ) . It is shown that batch normalization after nonlinear activation at each convolution layer ensures healthy norms ( Ioffe & Szegedy , 2015 ; Guo et al. , 2019 ; Jastrzebski et al. , 2018 ) . In Section 4.4 , we empirically show that the proposed orthogonality regularization and batch normalization similarly improve the flow of gradients during training .

Summary: This paper proposes a recursive convolutional block design for compact CNNs with efficient training. The convolutional filters (across all layers) are decomposed into shared basis filters and non-shared expanding coefficients, both of which are trained in an end-to-end manner. To avoid vanishing/exploding gradients, orthogonality regularization is imposed, and the efficacy of which is shown through both theoretical analysis and empirical verification. The proposed architecture consistently outperforms the corresponding baseline in terms of accuracy with a reduced model size.