Revisiting Dynamic Convolution via Matrix Decomposition

1 INTRODUCTION . Dynamic convolution ( Yang et al. , 2019 ; Chen et al. , 2020c ) has recently become popular for the implementation of light-weight networks ( Howard et al. , 2017 ; Zhang et al. , 2018b ) . Its ability to achieve significant performance gains with negligible computational cost has motivated its adoption for multiple vision tasks ( Su et al. , 2020 ; Chen et al. , 2020b ; Ma et al. , 2020 ; Tian et al. , 2020 ) . The basic idea is to aggregate multiple convolution kernels dynamically , according to an input dependent attention mechanism , into a convolution weight matrix W ( x ) = K∑ k=1 πk ( x ) Wk s.t . 0 ≤ πk ( x ) ≤ 1 , K∑ k=1 πk ( x ) = 1 , ( 1 ) where K convolution kernels { Wk } are aggregated linearly with attention scores { πk ( x ) } . Dynamic convolution has two main limitations : ( a ) lack of compactness , due to the use ofK kernels , and ( b ) a challenging joint optimization of attention scores { πk ( x ) } and static kernels { Wk } . Yang et al . ( 2019 ) proposed the use of a sigmoid layer to generate attention scores { πk ( x ) } , leading to a significantly large space for the convolution kernel W ( x ) that makes the learning of attention scores { πk ( x ) } difficult . Chen et al . ( 2020c ) replaced the sigmoid layer with a softmax function to compress the kernel space . However , small attention scores πk output by the softmax make the corresponding kernels Wk difficult to learn , especially in early training epochs , slowing training convergence . To mitigate these limitations , these two methods require additional constraints . For instance , Chen et al . ( 2020c ) uses a large temperature in the softmax function to encourage nearuniform attention . In this work , we revisit the two limitations via matrix decomposition . To expose the limitations , we reformulate dynamic convolution in terms of a set of residuals , re-defining the static kernels as Wk = W0 + ∆Wk , k ∈ { 1 , . . . , K } ( 2 ) where W0 = 1K ∑K k=1 Wk is the average kernel and ∆Wk = Wk −W0 a residual weight matrix . Further decomposing the latter with an SVD , ∆Wk = UkSkV Tk , leads to W ( x ) = K∑ k=1 πk ( x ) W0 + K∑ k=1 πk ( x ) UkSkV T k = W0 + UΠ ( x ) SV T , ( 3 ) where U = [ U1 , . . . , UK ] , S = diag ( S1 , . . . , SK ) , V = [ V1 , . . . , VK ] , and Π ( x ) stacks attention scores diagonally as Π ( x ) = diag ( π1 ( x ) I , . . . , πK ( x ) I ) , where I is an identity matrix . This decomposition , illustrated in Figure 1 , shows that the dynamic behavior of W ( x ) is implemented by the dynamic residual UΠ ( x ) SV T , which projects the input x to a higher dimensional space SV Tx ( fromC toKC channels ) , applies dynamic attention Π ( x ) over channel groups , and reduces the dimension back to C channels , through multiplication by U . This suggests that the limitations of vanilla dynamic convolution are due to the use of attention over channel groups , which induces a high dimensional latent space , leading to small attention values that may suppress the learning of the corresponding channels . To address this issue , we propose a dynamic convolution decomposition ( DCD ) , that replaces dynamic attention over channel groups with dynamic channel fusion . The latter is based on a full dynamic matrix Φ ( x ) , of which each element φi , j ( x ) is a function of input x . As shown in Figure 1- ( right ) , the dynamic residual is implemented as the product PΦ ( x ) QT of Φ ( x ) and two static matrices P , Q , such that Q compresses the input into a low dimensional latent space , Φ ( x ) dynamically fuses the channels in this space , and P expands the number of channels to the output space . The key innovation is that dynamic channel fusion with Φ ( x ) enables a significant dimensionality reduction of the latent space ( QTx ∈ RL , L C ) . Hence the number of parameters in P , Q is significantly reduced , when compared to U , V of Eq . 3 , resulting in a more compact model . Dynamic channel fusion also mitigates the joint optimization challenge of vanilla dynamic convolution , as each column of P , Q is associated with multiple dynamic coefficients of Φ ( x ) . Hence , a few dynamic coefficients of small value are not sufficient to suppress the learning of static matrices P , Q . Experimental results show that DCD both significantly reduces the number of parameters and achieves higher accuracy than vanilla dynamic convolution , without requiring the additional constraints of ( Yang et al. , 2019 ; Chen et al. , 2020c ) . 2 RELATED WORK . Efficient CNNs : MobileNet ( Howard et al. , 2017 ; Sandler et al. , 2018 ; Howard et al. , 2019 ) decomposes k × k convolution into a depthwise and a pointwise convolution . ShuffleNet ( Zhang et al. , 2018b ; Ma et al. , 2018 ) uses group convolution and channel shuffle to further simplify pointwise convolution . Further improvements of these architectures have been investigated recently . EfficientNet ( Tan & Le , 2019a ; Tan et al. , 2020 ) finds a proper relationship between input resolution and width/depth of the network . Tan & Le ( 2019b ) mix up multiple kernel sizes in a single convolution . Chen et al . ( 2020a ) trades massive multiplications for much cheaper additions . Han et al . ( 2020 ) applies a series of cheap linear transformations to generate ghost feature maps . Zhou et al . ( 2020 ) flips the structure of inverted residual blocks to alleviate information loss . Yu et al . ( 2019 ) and Cai et al . ( 2019 ) train one network that supports multiple sub-networks of different complexities . Matrix Decomposition : Lebedev et al . ( 2014 ) and Denton et al . ( 2014 ) use Canonical Polyadic decomposition ( CPD ) of convolution kernels to speed up networks , while Kim et al . ( 2015 ) investigates Tucker decompositions for the same purpose . More recently , Kossaifi et al . ( 2020 ) combines tensor decompositions with MobileNet to design efficient higher-order networks for video tasks , while Phan et al . ( 2020 ) proposes a stable CPD to deal with degeneracies of tensor decompositions during network training . Unlike DCD , which decomposes a convolutional kernel dynamically by adapting the core matrix to the input , these works all rely on static decompositions . Dynamic Neural Networks : Dynamic networks boost representation power by adapting parameters or activation functions to the input . Ha et al . ( 2017 ) uses a secondary network to generate parameters for the main network . Hu et al . ( 2018 ) reweights channels by squeezing global context . Li et al . ( 2019 ) adapts attention over kernels of different sizes . Dynamic convolution ( Yang et al. , 2019 ; Chen et al. , 2020c ) aggregates multiple convolution kernels based on attention . Ma et al . ( 2020 ) uses grouped fully connected layer to generate convolutional weights directly . Chen et al . ( 2020b ) extends dynamic convolution from spatial agnostic to spatial specific . Su et al . ( 2020 ) proposes dynamic group convolution that adaptively selects input channels to form groups . Tian et al . ( 2020 ) applies dynamic convolution to instance segmentation . Chen et al . ( 2020d ) adapts slopes and intercepts of two linear functions in ReLU ( Nair & Hinton , 2010 ; Jarrett et al. , 2009 ) . 3 DYNAMIC CONVOLUTION DECOMPOSITION . In this section , we introduce the dynamic convolution decomposition proposed to address the limitations of vanilla dynamic convolution . For conciseness , we assume a kernel W with the same number of input and output channels ( Cin = Cout = C ) and ignore bias terms . We focus on 1 × 1 convolution in this section and generalize the procedure to k×k convolution in the following section . 3.1 REVISITING VANILLA DYNAMIC CONVOLUTION . Vanilla dynamic convolution aggregates K convolution kennels { Wk } with attention scores { πk ( x ) } ( see Eq . 1 ) . It can be reformulated as adding a dynamic residual to a static kernel , and the dynamic residual can be further decomposed by SVD ( see Eq . 3 ) , as shown in Figure 1 . This has two limitations . First , the model is not compact . Essentially , it expands the number of channels by a factor of K and applies dynamic attention over K channel groups . The dynamic residual UΠ ( x ) SV T is a C × C matrix , of maximum rank C , but sums KC rank-1 matrices , since W ( x ) = W0 + UΠ ( x ) SV T = W0 + KC∑ i=1 πdi/Ce ( x ) uisi , iv T i , ( 4 ) where ui is the ith column vector of matrix U , vi is the ith column vector of matrix V , si , i is the ith diagonal entry of matrix S and d·e is ceiling operator . The static basis vectors ui and vi are not shared across different rank-1 matrices ( πdi/Ce ( x ) uisi , ivTi ) . This results in model redundancy . Second , it is difficult to jointly optimize static matrices U , V and dynamic attention Π ( x ) . This is because a small attention score πdi/Ce may suppress the learning of corresponding columns ui , vi in U and V , especially in early training epochs ( as shown in Chen et al . ( 2020c ) ) . 3.2 DYNAMIC CHANNEL FUSION . We propose to address the limitations of the vanilla dynamic convolution with a dynamic channel fusion mechanism , implemented with a full matrix Φ ( x ) , where each element φi , j ( x ) is a function of input x. Φ ( x ) is a L × L matrix , dynamically fusing channels in the latent space RL . The key idea is to significantly reduce dimensionality in the latent space , L C , to enable a more compact model . Dynamic convolution is implemented with dynamic channel fusion using W ( x ) = W0 + PΦ ( x ) Q T = W0 + L∑ i=1 L∑ j=1 piφi , j ( x ) q T j , ( 5 ) where Q ∈ RC×L compresses the input into a low dimensional space ( QTx ∈ RL ) , the resulting L channels are fused dynamically by Φ ( x ) ∈ RL×L and expanded to the number of output channels by P ∈ RC×L . This is denoted as dynamic convolution decomposition ( DCD ) . The dimension L of the latent space is constrained by L2 < C. The default value of L in this paper is empirically set to b C 2blog2 √ Cc c , which means dividing C by 2 repeatedly until it is less than √ C. With this new design , the number of static parameters is significantly reduced ( i.e . LC parameters in P or Q v.s . KC2 parameters in U or V , L < √ C ) , resulting in a more compact model . Mathematically , the dynamic residual PΦ ( x ) QT sums L2 rank-1 matrices piφi , j ( x ) qTj , where pi is the ith column vector of P , and qj is the jth column vector of Q . The constraint L2 < C , guarantees that this number ( L2 ) is much smaller than the counterpart ( KC ) of vanilla dynamic convolution ( see Eq . 4 ) . Nevertheless , due to the use of a full matrix , dynamic channel fusion Φ ( x ) retains the representation power needed to achieve good classification performance . DCD also mitigates the joint optimization difficulty . Since each column of P ( or Q ) is associated with multiple dynamic coefficients ( e.g . pi is related to φi,1 , . . . , φi , L ) , it is unlikely that the learning of pi is suppressed by a few dynamic coefficients of small value . In summary , DCD performs dynamic aggregation differently from vanilla dynamic convolution . Vanilla dynamic convolution uses a shared dynamic attention mechanism to aggregate unshared static basis vectors in a high dimensional latent space . In contrast , DCD uses an unshared dynamic channel fusion mechanism to aggregate shared static basis vectors in a low dimensional latent space .

Summary: This paper proposes a technique to reduce parameter count and improve training stability for dynamic convolutions using matrix decomposition. It looks at prior work (CondConv, DyConv) which aggregate multiple convolutional kernels via an attention score, and suggests this vanilla formulation is redundant since it sums $KC$ rank-1 matrices to create a rank-$C$ residual. The technique proposed in the paper (dynamic convolution decomposition) reduces the number of parameters via dimension reduction in the intermediate space, instead summing $L^2$ rank-1 matrices where $L^2 < C$. The authors provide experimental results on MobileNetV2 and ResNet comparing to prior works, as well as several ablations and possible extensions on their approach.