Revisiting Dynamic Convolution via Matrix Decomposition
Authors: Yunsheng Li
Yinpeng Chen
Xiyang Dai
mengchen liu
Dongdong Chen
Ye Yu
Lu Yuan
Zicheng Liu
Mei Chen
Nuno Vasconcelos
Venue: ICLR 2021 Poster (2021)
Keywords: supervised representation learning
efficient network
dynamic network
matrix decomposition

Abstract: Recent research in dynamic convolution shows substantial performance boost for efficient CNNs, due to the adaptive aggregation of K static convolution kernels. It has two limitations: (a) it increases the number of convolutional weights by K-times, and (b) the joint optimization of dynamic attention and static convolution kernels is challenging. In this paper, we revisit it from a new perspective of matrix decomposition and reveal the key issue is that dynamic convolution applies dynamic attention over channel groups after projecting into a higher dimensional latent space. To address this issue, we propose dynamic channel fusion to replace dynamic attention over channel groups. Dynamic channel fusion not only enables significant dimension reduction of the latent space, but also mitigates the joint optimization difficulty. As a result, our method is easier to train and requires significantly fewer parameters without sacrificing accuracy. Source code is at https://github.com/liyunsheng13/dcd.

Published as a conference paper at ICLR 2021
REVISITING DYNAMIC CONVOLUTION VIA MATRIX DECOMPOSITION
Yunsheng Li', Yinpeng Chen', Xiyang Dai', Mengchen Liu', Dongdong Chen', Ye Yu?, La Yuan', Zicheng Liu', Mei Chen', Nuno Vasconcelos' Department of Electrical and Computer Engineering, University of California San Diego Microsoft
2015540ucsd..o edu, {yiche, xi xidai, mengl d cmiccoooot com (Yu. Ye, luyuan, zliu, Mei Chen ) @mi croso ft. com, nvasconce I @eedusdd.eds
ABSTRACT
Recent research in dynamic convolution shows substantial performance boost for efficient CNNs, due the adaptive aggregation of K static convolution kernels. (a) increases the number of convolutional weights by K- times, and (b) the joint optimization of dynamic attention and static convolution kernels is challenging. In this paper, we revisit it from a new perspective of ma- trix decomposition and reveal the key issue is that dynamic convolution applies dynamic attention over channel groups after projecting into higher dimensional latent space. To address this issue, we propose dynamic channel fusion to re- place dynamic attention over channel groups. Dynamic channel fusion not only enables significant dimension reduction the latent space, but also mitigates the joint optimization difficulty. As result, method is easier to train and re- quires significantly fewer parameters without sacrificing accuracy. Source code is at htttss//gghhub.ccmm/iyyuuheenn 1//ded
It has two limitations:
INTRODUCTION
Dynamic convolution Yang et al. 2019 Chen et al. 2020c has recently become popular for the
implementation of light- networks (Howard et al.] 2017 Zhang et al.) 2018b). Its ability to achieve significant performance gains with negligible computational cost has motivated its adoption for multiple vision tasks (Su et al., 2020: Chen et al. 2020b Ma et al. 2020 Tian et al. 2020). The
basic idea is to aggregate multiple convolution kernels dynamically, according to input dependent attention mechanism. into a convolution weight matrix
K
K
((x)
(T) s.t. 0 (x) S 1,
=
(I)
where K convolution kemels (Wr) are aggregated linearly with attention scores (TT)
Dynamic convolution has two main limitations: (a) lack of compactness, due to the use of K kernels,
and (b) challenging joint optimization of attention scores {x]) } and static kernels [W.). J. Yang
ct al. (2019) proposed the use of a sigmoid layer to generate attention scores {Tx(#)) leading to
significantly large space for the convolution kernel (xx) that makes the learning of attention scores {T {z)) difficult. Chen et al. (2020c) replaced the sigmoid layer with softmax function to compress the kernel space. However, small attention scores T. output by the softmax make the corresponding kernels W₁ difficult to learn, especially in early training epochs, slowing training convergence. To mitigate these limitations, these two methods require additional constraints. For instance, Chen et al. (2020c) uses large temperature in the softmax function to encourage near- uniform attention.
In this work, we revisit the two limitations via matrix decomposition. To expose the limitations, we reformulate dynamic convolution terms of set of residuals, re-defining the static kernels as
W Wo + AWK. k (1....,K) K)

---

Published as a conference paper at ICLR 2021
Vanilla Dynamic Convolution
Dynamic Convolution Decomposition
V
W
Uyy
my (x)r
DNC
W
Q'
V
(L
(1())) KCAKC
S e NACCCC
Figure 1: Dynamic convolution via matrix decomposition, Left: Reformulating the vanilla dynamic convolution by matrix decomposition (see Eq. 3). It applies dynamic attention II(x)) over channel groups in a high dimensional space (SVT x E TR. RC Right: proposed dynamic convolution decomposition, which applies dynamic channel fusion (xx) in a low dimensional space (Q" E RR,, < C), resulting in a more compact model.
WE is the average kernel and AW; W Wo a residual weight matrix. Further decomposing the latter with an SVD, AW; = S V leads to
where Wo K E
K (x Wo + Σ ( W. UN(C)SVT
((x)=
Σ
(3)
where U U Uk| S diag( S1. Sk), V= V1 Vkl. and [Txx) stacks attention scores diagonally as [Ix diag I II ..... TK (x))),, where an identity matrix. This decomposition, illustrated in Figure shows that the dynamic behavior of W(x) is implemented by the dynamic residual (TI((r)
which projects the input æ to higher dimensional space SV' x (from C' to KC channels), applies dynamic attention (xx) over channel groups, and reduces the dimension back to C' channels, through multiplication by U. This suggests that the limitations of vanilla dynamic convolution a high dimensional latent space, leading to small attention values that may suppress the learning of the corresponding channels.
due to the use of attention over channel groups, which induces
To address this issue, we propose a dynamic convolution decomposition (DCD), that replaces dy- namic attention over channel groups with dynamic channel fusion. The latter is based a full dynamic matrix o (xx), of which each element (i.5(() is a function of input æ. As shown in Figure (right), the dynamic residual implemented as the product (x)) of ((x) and two static matrices P, Q, such that Q compresses the input into low dimensional latent space, ((x)) dynam- ically fuses the channels in this space, and P expands the number of channels to the output space. The key innovation that dynamic channel fusion with ((x) enables significant dimensionality reduction of the latent space (Q" E RL, L < C). Hence the number of parameters in P. P.Q significantly reduced, when compared to U, of Eq. [3] resulting in a more compact model. Dynamic channel fusion also mitigates the joint optimization challenge of vanilla dynamic convo- lution, as each column of P, Q is associated with multiple dynamic coefficients of (xr).. Hence, few dynamic coefficients of small value are not sufficient to suppress the learning of static matrices P.Q. Experimental results show that DCD both significantly reduces the number of parameters and achieves higher accuracy than vanilla dynamic convolution, without requiring the additional constraints YYagg et al. 2019 Chen et al. 2020c).
2 RELATED WORK
MobileNet (Howard et al. 2017 Sandler al. 2018 Howard et 2019) decom- poses k x k convolution into depthwise and a pointwise convolution. ShuffleNet (Zhang et al.] 2018b Ma et al., 2018 uses group convolution and channel shuffle to further simplify pointwise convolution. Further improvements of these architectures have been investigated recently. Efficient- Net (Tan & Le 2019a Tan et al. 2020) finds a proper relationship between input resolution and
Efficient CNNs:
width/depth of the network. Tan & Le (b)1 mix up multiple kernel sizes in single convolution.
Chen et al. (2020a) trades massive multplicationss for much cheaper additions. Han et al. (2020)
applies series of cheap linear transformations to generate ghost feature maps. Zhou al. 20200
flips the structure of inverted residual blocks to alleviate information loss. Yu et al. (2019) and Cal
et al. (2019) train one network that supports multiple sub-networks of different complexities.
2

---

Published as a conference paper at ICLR 2021
Matrix Decomposition: Lebedev al. (2014) and Denton al. (2014) use Canonical Polyadic decomposition (CPD) of convolution kernels to speed up networks, while Kim et al. (2115) investi- gates Tucker decompositions for the same purpose. More recently, Kossafii al. (2020) combines tensor decompositions with MobileNet to design efficient higher-order networks for video tasks, while|Phan et al. (2020) proposes stable CPD deal with degeneracies of tensor decompositions during network training. Unlike DCD, which decomposes a convolutional kernel dynamically by adapting the core matrix to the input, these works all rely on static decompositions.
Dynamic networks boost representation power by adapting param- eters or activation functions the input. Ha al. (2017) uses a secondary network to generate parameters for the main network. Hu et (2018) reweights channels by squeezing global context.
Dynamic Neural Networks:
Li al. (2019) adapts attention over kernels of different sizes. Dynamic convolution (Yang et al.)
2019 Chen et al., 2020c) aggregates multiple convolution kernels based on attention. Ma et al.
(2020) uses grouped fully connected layer to generate convolutional weights directly. [Chen et al.
2020bb extends dynamic convolution from spatial agnostic to spatial specific. [Su et al. (2020) pro-
poses dynamic group convolution that adaptively selects input channels to form groups. [Tian et al.
(2200) applies dynamic convolution instance segmentation. Chen al. (2020d) adapts slopes and
intercepts of two linear functions in ReLU (Nair & Hinton 2010: Jarrett et al. 2009).
3 DYNAMIC CONVOLUTION DECOMPOSITION
In this section, we introduce the dynamic convolution decomposition proposed to address the lim- itations of vanilla dynamic convolution. For conciseness, number of input and output channels (Ctn = Cout = C)) and ignore bias terms. We focus on 1 x convolution in this section and generalize the procedure to k x k convolution in the following section.
assume kernel W with the same
3.1 REVISITING VANILLA DYNAMIC CONVOLUTION
Vanilla dynamic convolution aggregates K convolution kennels {W&} with attention scores (xT( (see Eq. [] It can be reformulated as adding a dynamic residual to static kernel, and the dynamic residual can be further decomposed by SVD (see Eq. 33. as shown in Figure [ This has two limitations. First, the model is not compact. Essentially, expands the number of chan- nels by a factor of K and applies dynamic attention over K channel groups. The dynamic residual UI( is a C x C matrix, maximum rank C, but sums KC rank-1 matrices, since
KC ((x) Wo + UT* = Σ
(4)
where u, is the 7th column vector of matrix U, v; is the 7th column vector of matrix V, 54 is the 7th diagonal entry of matrix and [-] ceiling operator. The static basis vectors u, and v; are not shared across different rank-1 I matrices (TTz/C7 (x) s;,)u,, This results in model redundancy. Second, it is difficult to jointly optimize static matrices U, V and dynamic attention Ixx). This is because a small attention score Tia/C] may suppress the learning of corresponding columns u, v; in U and V. especially in early training epochs (as shown in Chen et al. (2020c)).
3.2 DYNAMIC CHANNEL FUSION
We propose to address the limitations of the vanilla dynamic convolution with a dynamic channel fusion mechanism, implemented with full matrix (x)) where each element dx, (x) a function of input œ. (xx) is L x L matrix, dynamically fusing channels in the latent space R /. The key idea is to significantly reduce dimensionality in the latent space, L & C, to enable a more compact model. Dynamic convolution is implemented with dynamic channel fusion using
W (w) Wo + PD(x)Q" Wo ΣV P.o.. (x)q
(5)
where Q E C compresses the input into a low dimensional space (QT x E R the resulting L channels fused dynamically by (a:) LxL and expanded to the number of output channels
3

---

Published as a conference paper at ICLR 2021
by P E R This is denoted dynamic convolution decomposition (DCD). The dimension L of the latent space is constrained by L < C. The default value of L in this paper is empirically set to 2logg ver J. which means dividing C' by 2 repeatedly until it is less than vC.
With this new design, the number of static parameters is significantly reduced (i.e. LC parameters in P or Q v.s. KC² parameters in U or V. L < VC), resulting in a more compact model. Mathematically, the dynamic residual (*()) sums L rank-1 matrices pıót)(z)g) where p; is the 2th column vector of P. and q is the jth column vector of Q. The constraint L < C, guarantees that this number (I22) is much smaller than the counterpart (KC) of vanilla dynamic convolution (see Eq. 4.. Nevertheless, due to the use of a full matrix, dynamic channel fusion (xx) retains the representation power needed to achieve good classification performance.
DCD also mitigates the joint optimization difficulty. Since each column of P (or Q) is associated with multiple dynamic coefficients (e.g. P. is related to oi.1...., , is unlikely that the learning of p, is suppressed by few dynamic coefficients of small value.
In summary, DCD performs dynamic aggregation differently from vanilla dynamic convolution. Vanilla dynamic convolution uses a shared dynamic attention mechanism to aggregate unshared static basis vectors in a high dimensional latent space. In contrast, DCD uses an unshared dynamic channel fusion mechanism aggregate shared static basis vectors in low dimensional latent space.
3.3 MORE GENERAL FORMULATION
So far, we have focused on the dynamic residual and shown that dynamic channel fusion enables a compact implementation of dynamic convolution. We next discuss the static kernel Wo. Originally, it is multiplied by a dynamic scalar Ee Te (2t). which canceled in Eq. as attention scores sum results in the more general Torm
to one. Relaxing the constraint Σn TA (z)
W (z) = A (x) + (*))""";
(6)
where Axx) is a C. x C diagonal matrix and A,, (2) a function of x. In this way, A (ar) implements channel-wise attention after the static kernel Wo. generalizing Eq. where Axx) is an identity
matrix. Later, we will see that this generalization enables additional performance gains.
Relation to Squeeze-and -ndd-xcitaation (SE) (Hu al., 2018): The dynamic channel-wise attention mechanism implemented by A(x) is related to but different from SE. It is parallel to convolution and shares the input with the convolution. It can be thought of as either a dynamic convolution kernel (A Wo) )a or an input-dependent attention mechanism applied to the oulput feature map of the convolutionn y = A (ar)) (W.r). Thus, its computational complexity is min(O(())) O(WW))) where H and W are height and width of the feature map.
In contrast, SE is placed after a convolution and uses the output of the convolution as input. It can only apply channel attention the output feature map of the convolution as y = A(z)=, where z Wox. Its computational complexity is (HWC). Clearly, SE requires more computation than dynamic channel- wise attention A(x) when the resolution of the feature map (H x W) is high.
3.4 DYNAMIC CONVOLUTION DECOMPOSITION LAYER
Implementation: Figure 2 shows the diagram of dynamic convolution decomposition (DCD) layer. It uses light-weigt dynamic branch to generate coefficients for both dynamic channel- attention Axx) and dynamic channel fusion (xx). Similar to Squeeze-am and and-Excitation (Hu et al., 2018), the dynamic branch first applies average pooling to the input æ. This is followed by two fully connected (FC) layers with an activation layer between them. The first FC layer reduces the number of channels by and the second expands them into C + L outputs (C' for A and L for Φ). Eq. i finally used to generate convolutional weights W (x). Similarly to a static convolution, a DCD layer also includes a batch normalization and an activation (e.g. ReLU) layer.
Parameter Complexity: DCD has similar FLOPs to the vanilla dynamic convolution. Here, we focus on parameter complexity. Static convolution and vanilla dynamic convolution require C² and KC2 parameters, respectively. DCD requires C²2 CL, and CL parameters for static matrices W.. P and Q, respectively. An additional (2C + 1.2) parameters required by the dynamic branch
4

---

Published as a conference paper at ICLR 2021
Branch
avg pool
Data flow
Model parameter flow
A
Sparse Dynamic Residual
Activation
P (0)
FC
Wo
Q
(x)x
(x)x
Activation
Figure 3: Sparse dynamic residual, which is represented as diagonal block matrix. Each diagonal block is decomposed sepa- rately P. Q Note that the static ker- nel Wo is still full size matrix.
Dynamic convolution decomposition layer. The input & first goes through a dynamic branch to generate (a)) and ((x), and then to gen- erate the convolution matrix W(x) using Eq. 6
Figure 2:
to generate (x)) and (xx), where r is the reduction rate of the first FC layer. The total complexity
C 2CI. + (2C + Ş. Since I. is constrained as 1,2 < C, the complexity upper bound (1 + ))C² + 2C' /C.C. When choosing 16, the complexity is about 13 C This is much less
than what is typical for vanilla dynamic convolution (4C² in Chen et al. (2020c) and 8C2 in Yang ct al. 2019)).
EXTENSIONS OF DYNAMIC CONVOLUTION DECOMPOSITION
In this section, extend the dynamic decomposition of 1 x convolution (Eq. 6 in three ways: (a) sparse dynamic residual where Px(x)Q" a diagonal block matrix, (b) Te x 1 depthwise convolution, and & x convolution. Here, k refers to the kernel size.
4.1 DCD WITH SPARSE DYNAMIC RESIDUAL
The dynamic residual ( can be further simplified into a block-diagonal matrix of blocks P. (x), (1..... B), leading to
W(x) A(x)
where +i_ A, diag(A,..... A,). This form has Eq. a special case, where Note
that the static kernel Wo is still full matrix and only the dynamic residual is sparse (see Figure 3 We will show later that keeping as few as g of entries of the dynamic residual non-zero (B 8) has minimal performance degradation, still significantly outperforming static kernel.
4.2 DCD OF l DEPTHWISE CONVOLUTION
The weights of a k k depthwise convolution kernel form C k2 matrix. DCD can be generalized to such matrices by replacing in Eq, the matrix Q (which squeezes the number of channels) with matrix R (which squeezes the number of kernel elements)
= =
PD(Z)R
(8)
where (x)) and Wo are x 62 matrices, A(x) is a diagonal x matrix that implements channel- wise attention, R is 62 LE matrix that reduces the number of kernel elements from 2 to Lg, (xx) is a LA x L3 matrix that performs dynamic fusion along Lş latent kernel elements and P is C Lt weight matrix for depthwise convolution over L₁ kernel elements. The default value of LA is 2222).. Since depthwise convolution channel separable, ((x) does not fuse channels, fusing instead L, latent kernel elements.
5

---

Published as a conference paper at ICLR 2021
Strtic Karna
Dynamic Residual
Stati Kennel
W
(P)
W
[x)
Uxtx
CHOKk
Oxc
CXL
CsCokl
Cxtl
Dxt
CoCoee
CHCHER
(a) (x)
elements.
(x) alone.
Figure 4: The dynamic convolution decomposition for k x k convolution.
4.3 DCD OF CONVOLUTION
Joint fusion of channels and kernel elements: A k k convolution kernel forms C C x k22 tensor. DCD can be generalized to such tensors by extending Eq. G6ino tensor form (see Figure
W(x) (a) = Wo X2 A(x) ((xx) X Q X2 P X3 R.
where Xn refers to n-mode multiplication [athauwer et al. 2000), Wo is C x C h2 tensor, (xx) is diagonal C x C' matrix that implements channel-win -wise attention, Q is a x matrix that reduces the number of input channels from C to L,, R is k22 La matrix that reduces the number of (xx) kernel elements from k2 to Li. $ (I) is L x L L tensor that performs joint fusion of L channels over La latent kernel elements, and P a C x L matrix that expands the number of channels from L to C. The numbers of latent channels L and latent kernel elements L3 are constrained by L < h² C/L
and L C. Their default values set empirically to L k-2 /2], L 221022//771.)
Channel fusion alone: We found that the fusion of channels (xx) x Q is more important than the fusion of kernel elements (xx) X3 R. Therefore, we reduce L and increase accordingly. R is simplified into a one-hot vector [0. ..... 0, 1, 0, 00 0J" where the 'I' is located at the center (assuming that is an odd number). As illustrated in Figure (b), the tensor of dynamic residual ((x) X1 Q X2 P X3 R only has one non-zero slice, which is equivalent to 1 x convolution.
Therefore, the DCD a & convolution essentially adding a x dynamic residual to a static k x & kernel.
EXPERIMENTS
In this section, we present the results of DCD on ImageNet classification (Deng et al. 2009). Im- ageNet has 1,000 classes with 1,281,167 training and 50, 000 validation images. We also report ablation studies on different components of the approach.
All experiments are based on two network architectures: ResNet (He et al. 2016) and MobileNetV2 (Sandler et al. 2018). DCD is implemented on all convolutional layers of ResNet and all convolutional layers of MobileNetV2. The reduction ratio is set to 16 for ResNet and MobileNetV2 1.0, and to 8 for smaller models (MobilcNetV2 x0.5 and x0.35). All models trained by SGD with momentum 0.9. The batch size is 256 and remaining training parameters are as follows.
ResNet: The learning rate starts 0.1 and is divided by 10 every 30 epochs. The model is trained with 100 epochs. Dropout (Srivastava et al. 2014 0.1 is used only for ResNet-50.
Nob: The initial learning rate is 0.05 and decays 0 in 300 epochs, according to cosine function. Weight decay of 2e-5 and dropout rate 0.1 are also used. For MobileNetV2 x1.0, Mixup (Zhang et al. 2018a) and label smoothing further added avoid overfitting.

---

Published as a conference paper at ICLR 2021
Table Different formulations of dynamic convolution decomposition on ImageNet classification.
Model
Params
MAdds
Top-1
Model
Params
MAdds
Top-1
Wo (static)
2.0M
97.0M
65.4
Wo (static)
11.1M
181G
70.4
AWo
2.4M
97.4M
68.2
A AWo
1177M
181G
71.5
Wo + P
2.7M
104.4M
69.2
W. + PQQ'
13.6M
183G
72.8
AW.
PDQ'
2.9M
104.6M
69.8
AW. + P0Q'
14.0M
1.83G
73.1
(a) MobileNet V2 x0.5
(h) ResNet-11
5.1
INSPECTING DIFFERENT DCD FORMULATIONS
Table summarizes the influence of different components (e.g. dynamic channel fusion ((x), dynamic channel-wise attention A (x)) of DCD on MobileNet V2 x0.5 and ResNet-18 performance. The table shows that both dynamic components. A(x) and (x) of Eq. 6 enhance accuracy substantially (+2.8% and +3.8% for MobileNetV2 x0.5, +1.1% and +2. 22.4% for 18), when compared the static baseline. Using dynamic channel fusion only (Wa + P has slightly more parameters, FIOOss, and accuracy than using dynamic channel-wise attention only (A (o). The combination of the two mechanisms provides additional improvement.
5.2 ABLATIONS
A number of ablations were performed on MobileNet V2 x0.5 to analyze DCD performance in terms of two questions.
1. How does the dimension (L) of the latent space affect performance?
2. How do three DCD variants perform?
The default configuration is the general form of DCD (Eq. 6 with a full size dynamic residual
(B 1) for all pointwise convolution layers. The default latent space dimension is T. L2
Latent Space Dimension L: The dynamic chan- nel fusion matrix ((x) has size L x L. Thus, controls both the representation and the param- eter complexity of DCD. We adjust it by apply- ing different multipliers to the default value of T.. Table 2 shows the results of MobileNerV2 x0.5
Table 2: Dimension of the latent space L eval- uated on ImageNet classification (MobileNetV2
x0.5 is used).
Model static
L
Params
MAdds
Top-1
for four multiplier values ranging from x x1.0 to x0.25. As L decreases, fewer parameters are required and the performance degrades slowly. Even with very low dimensional latent space (L 0.255, DCD still outperforms the static base- line by 3.3% top-1 accuracy.
2.0M
97.0M
65.4
x0.25
2.4M
99.8M
68.7
x0.50
2.5M
101.3M
69.0
DCD
x0.75
2.6M
102.9M
69.6
x.00
2.9M
104.6M
69.8
Number of Diagonal Blocks B in the Dynamic
Residual: Table (a) shows classification results for four values of B. The dynamic residual is full matrix when B = 1, while only & of its entries are non-zero for B = 8. Accuracy degrades slowly as the dynamic residual becomes sparser (inereasing B). The largest performance drop happens when B is changed from 1 to 2, as half of the weight matrix W (x) becomes static. However, performance still significantly better than that of the static baseline. The fact that even the sparsest B outperforms the static baseline by 2.9% (from 65.4% to 68.3%) demonstrates the representation power of the dynamic residual. In all cases, dynamic channel attention (ar) enables additional performance gains.
DCD at Different Layers: Table (b) shows the results of implementing DCD for three different types of layers (a) DW: depthwise convolution (Eq. 8 (b) PW: pointwise convolution (Eq. 6 and (c) CLS: fully connected classifier, which is a special case of pointwise convolution (the input resolution is 1). Using DCD in any type of layer improves
the performance of the static baseline (+2.9% for depthwise convolution, +4.4% for pointwise convolution, and +1.2% for classi- fier). Combining DCD for both pointwise convolution and classifier achieves the best performance

---

Published as a conference paper at ICLR 2021
Table Extensions of dynamic convolution decompostion (DCD) evaluated on ImageNet classi- fication (MobileNetV2 x0.5 is used).
Network
B Params
MAdds
Top-1
DW
PW
CLS
Params
MAdds
Top-I
Wo (static)
2.0M
97.0M
65.4
2.0M
97.0M
65.4
2.7M
104.4M
69.2
2.4M
97.5M
68.3
Wo + P.Q"
2
2.6M
101OM
68.5
2.9M
104.6M
69.8
4
2.5M
99.1M
68.4
2.2M
97.2M
8
2.3M
98.5M
68.3
2.6M
97.7M
69.0
2.9M
104.6M
69.8
4
3.3M
105.IM
69.6
AW. + P&Q"
2
2.8M
101. 101.3M
68.9
3.1M
104.8M
4
2. 7M
99.4M
68.8
3.5M
105.3M
70.0
8
2.7M
98.8M
68.5
(b) DCD at different layers. DW, PW, and CLSS indicate depthwise convolution, pointwise cony- olution and classifier respectively.
(a) Number of diagonal blocks B in the dynamic residual.
Table 4: Comparing DCD with the vanilla dynamic convolution CondConv Yang al. 2019) and DY-Conv (Chen et al. 2020c). *indicates the dynamic model with the fewest parameters (static model is not included). CondConv contains K kernels and DY-Conv contains K 4 kernels.
Width Model
Params
MAdds
Top-1
static
3.5M
300.0M
72.0
DY-Conv
11.1M
312.9M
75.2
x1.0
CondConv
27.5M
329.0M
74.6
Depth
Model
Params
MAdds
Top-1
DCD (ours)
*5.5M
326.0M
75.2
static
23.5M
3.8G
76.2
ResNet-50
static
2.0M
97.0M
65.4
DCD (ours)
30.7M
3.9G
77.9
x0.5
DY-Conv
4.0M
101.4M
69.9
static
11.1M
18IG
70.4
CondConv
15.5M
113.0M
68.4
ResNet-18
DY-Conv
42.7M
1.85G
72.7
DCD (ours)
*3.1M
104.8M
70.2
DCD (ours)
*14.0M
1.83G
73.1
static
1.7M
59.2M
60.3
static
5.2M
0.89G
63.5
x0.35
DY-Conv
2.8M
62.0M
65.9
ResNet-10
DY-Conv
18.6M
0.91G
67.7
DCD (ours)
#2.3M
63.1M
66.6
DCD (ours)
*6.5M
0.90G
68.8
(a) MobileNetV2.
(b) ResNet.
(+4.8%). We notice performance drop (from 70.2% to 70.0%) when using DCD in all three types of layers. We believe this is due overfitting, as it has higher training accuracy.
Extension to 3 x 3 Convolution: We use ResNet-18, which stacks 16 layers of 3 x 3 convolution,
to study the 3 3 extension of DCD (see Section 4.3). Compared to the static baseline (70.4% top-1 accuracy), DCD with joint fusion of channels and kernel elements (Eq. (9) improves top-1 accuracy (71.3%) by 0.9%. The top-1 accuracy is further improved by 1.8% (73.1%) 1%), when using DCD with channel fusion alone, which transforms the dynamic residual a convolution matrix (see Figure (b)). This demonstrates that dynamic fusion more effective across channels than across kernel elements.
Summary: Based on the ablations above, DCD should be implemented with both dynamic chan- nel fusion & and dynamic channel-wise attention A. the default latent space dimension L, and full size residual = 1. DCD recommended for pointwise convolution and classifier layers in MobileNetV2. For 3 x 3 convolutions in ResNet, DCD should be implemented with channel fu- sion alone. The model can be made more compact, for a slight performance drop, by (a) removing dynamic channel-wise attention A, (b) reducing the latent space dimension L, (c) using sparser dynamic residual (increasing B), and (d) implementing DCD in depthwise convolution alone.
5.3 MAIN RESULTS
DCD was compared to the vanilla dynamic convolution (Yang et al. 2019 Chen et al. 2020ch for MobileNetV2 and ResNet, using the settings recommended above, with results of
8

---

Published as a conference paper at ICLR 2021
Table DCD significantly reduces the number of param- eters while improving the performance of both network ar- chitectures. For MobileNetV2 x1.0, DCD only requires
DCD Hai Daryy
50-
80
50% of the parameters of (Chen et al., 2020ch and 25% of the parameters of (Yang 2019). For ResNet-18. it only requires 33% of the parameters of (Chen et al.
Top-1 1 (8A
8
Tir Ca anl ued0
80
2020ch, while achieving a 0.4% gain in top-1 accuracy, Although DCD requires slightly more MAdds than (Chen al. 2020c), the increment is negligible. These results demonstate that DCD is more compact and effective.
60-
40
30
20
150
Sigure@comparre DCD to DY-Conv (Chen et al., 2020c) in terms of training convergence. DY-Conv uses large tem- perature its softmax to alleviate the joint optimization difficulty and make training more efficient. Without any ad- ditional parameter tuning, DCD converges even faster than DY-Conv with large temperature and achieves higher curacy.
Epochs
Figure 5: The comparison of train- ing and validation error between DCD and DY-Conv on MobileNetV2 x0.5. is the temperature in softmax. Best viewed in color.
5.4 ANALYSIS OF DYNAMIC CHANNEL FUSION
To validate the dynamic property, (xx) should have dif- ferent values over different images. We measure this by averaging the variance of each entry o = E., St, LL? where J. the variance of (x)), over all validation im-
1.8
1.8
Nomal Vatance cg
ages. To compare d& across layers, we normalize it by the variance of the corresponding input feature map. Figure 6 shows the normalized variance dg across layers in Mo- bi.eNetV2. Clearly, the dynamic coefficients vary more in the higher layers. We believe this is because the higher lay- ers encode more context information, providing more clues to adapt convolution weights.
0.8
Figure 6: Normalized variance of dy- namic coefficients d across layers in MobileNetV2 x0.5 and x x1.0.
5.5 INFERENCE TIME
We use a single-threaded core AMD EPYC CPU 755IP (2.0 GHz) to measure running time (in mil- liseconds) on MobileNetV2 x0.5 and 1.0. Running time calculated by averaging the inference time of 5,000 images with batch size 1. Both static baseline and DCD are implemented in PyTorch. Compared with the static baseline, DCD consumes about 8% more MAdds (97.0M vs 104.8M) and 14% more running time (91ms VS 104ms) for MobileNetV2 x0.5. For MobileNetV2 x 1.0, DCD consumes 9% more MAdds (300.0M vs 326.0M) and 12% more running time (146ms vs 163ms). The overhead is higher in rumning time than MAdds. We believe this is because the optimizations of global average pooling and fully connected layers are not as efficient as convolution. This small penalty in inference time is justified by the DCD gains of 4.8% and 3.2% top-1 accuracy over Mo- bileNetV2 x0.5 and x 1.0 respectively.
6 CONCLUSION
In this paper, we have revisited dynamic convolution via matrix decomposition and demonstrated the limitations of dynamic attention over channel groups: it multiplies the number of parameters by K and increases the difficulty of joint optimization. We proposed a dynamic convolution decom- position to address these issues. This applies dynamic channel fusion significantly reduce the dimensionality of the latent space, resulting in more compact model that is easier to learn with of- ten improved accuracy. We hope that our work provides a deeper understanding of the gains recently observed for dynamic convolution.
'The baseline results are from the original papers. Our implementation, under the setup used for DCD, has either similar or slightly lower results, e.g. for MohileNerV2x 1.0 the original paper reports 72.0%, while our implementation achieves 71.8%.

---

Published as a conference paper at ICLR 2021
REFERENCES
Han Cai, Chuang Gan, and Song Han. Once for all: Train one network and specialize it for efficient deployment. ArXiv, abs//998.099 19799, 2019.
Hanting Chen, Yunhe Wang. Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addernet:
Do we really need multiplications in deep learning? In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020a.
Jyun-Ruei Chen, Xijun Wang, Zichao Guo, X. Zhang, and Sun. Dynamic region-aware convolu- tion. ArXiv, abs/2003.12243. 2020b.
Yinpeng Chen, Xiyang Dai, Mengehen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020c.
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic relu. arXiv preprint urXiv:2003.10022 20s22003.10007, 2020d.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp.
leee, 2009.
Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in neural informa- tion processing systems, pp. 1269-1277 2014.
David Ha, Andrew M. Dai, and Quoe V. Lee. Hypernetworks. ICIR, 2017.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chumjing Xu, and Chang Xu. Ghostnet: More fea- tures from cheap operations. In IEEE/CVF Conference on Computer Vision and Pattern Recog- nilion (CVPR), June 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoe V. Le, and Hartwig Adam. Searching for mobilenetv3, CoRR, abs/1905.02244, 2019. URL
02244
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 4rXiv:1004.04861, 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitaiee networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Kevin Jarrett, Koray Kavukcuoglu, Marc' Aurclio Ranzato, and Yann LeCun. What is the best multi- stage architecture for object recognition? In The IEEE International Conference Computer Vision (ICCV), 2009.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Com- pression deep convolutional neural networks for fast and low power mobile applications. arXiv preprint (tt.::15111005333 2015.
Jean Kossaifi, Antoine Toisoul, Adrian Bulat, Yannis Panagakis, Timothy M Hospedales, and Maja Pantic. Factorized higher-order cnns with an application to spatio-temporal emotion estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
6060-6069, 2020.
L. D. Lathauwer, B. D. Moor, and J. Vandewalle. A multilinear singular value decomposition. In SIAM J. Matrix Anal. Appl, 2000.
10

---

Published as a conference paper at ICLR 2021
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint arXiv:1412.6553, 2014.
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient emn architecture design. In The European Conference on Computer Vision (ECCV), September 2018.
Ningning Ma, X. Zhang. J. Huang, and J. Sun. Weightnet: Revisiting the design space of weight networks. volume abs/2007.1 1822,, 2020.
Vinod Nair and Geoffrey E, Hinton. Rectified linear units improve restricted boltzmann machines. In ICMI,, 2010.
Anh-Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavsky, Valeriy Glukhov, Ivan Oseledets, and Andrzej Cichocki. Stable low-rank tensor de- composition for compression of convolutional neural network. arXiv preprint (at::2008.0544411 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chio Chieh Chen. Mo- bilenetv2: Inverted residuals and linear boutlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 19969919 2014. URL http:://mml wwwp:///gpgpppp./seeee
sr srivastava 4a. ht.l
Zhuo Su, Linpu Fang, Wen xiong Kang, D. Hu, M. Pietikäinen, and Li Liu. Dynamic group convo- lution for accelerating convolutional neural networks. In ECCV, August 2020.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In ICML, pp. 6105-6114, Long Beach, California, USA, 09-15 Jun 2019a.
Mingxing Tan and Quoe V. Le. Mixconv: Mixed depthwise convolutional kernels. In 30th British Machine Vision Conference 2019, 2019b.
Mingxing Tan, Ruoming Pang, and Quoc V. Le. Efficientder: Scalable and efficient object detec- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In ECCV, August 2020.
Brandon Yang, Gabriel Bender, Quoc V. Le, and Jiquan Ngiam. Condconv: Conditionally parame- terized convolutions for efficient inference. In NeurIPS, 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference on Learning Representations, 2019. URL Ut..httips://pperevii.
nett/oozumiid=ilguu
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empir-
ical risk minimization. In International Conference on Learning Representations, 2018a. URL
https://openreview net. httee//ooormmml1r1ldpp-----
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient
convolutional neural network for mobile devices. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018b.
Daquan Zhou, Qi-Bin Hou, Y. Chen, Jiashi Feng and S. Yan. Rethinking bottleneck structure for efficient mobile network design. In ECCV, August 2020.


Reviews:
[{"review_id": "oyIAwqAYN4Q", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a technique to reduce parameter count and improve training stability for dynamic convolutions using matrix decomposition. It looks at prior work (CondConv, DyConv) which aggregate multiple convolutional kernels via an attention score, and suggests this vanilla formulation is redundant since it sums $KC$ rank-1 matrices to create a rank-$C$ residual. The technique proposed in the paper (dynamic convolution decomposition) reduces the number of parameters via dimension reduction in the intermediate space, instead summing $L^2$ rank-1 matrices where $L^2 < C$. The authors provide experimental results on MobileNetV2 and ResNet comparing to prior works, as well as several ablations and possible extensions on their approach.\n\nThis paper helps address one key drawback of prior dynamic convolution approaches, which is that they require many more parameters than ordinary convolutions. However, this benefit needs to be better motivated by the paper. The goal of the prior dynamic convolution approaches was to create models that have higher performance and lower inference cost. In the case of MobileNetV2, while the parameter count is reduced, the inference cost is actually higher than the previous Dy-Conv approach to achieve the same performance. For this paper to be more compelling, it would be important to show reasons why reduced parameter count may be worth the increased inference cost, especially since from purely a parameter perspective, static scaling (with depth and width) seems to be more parameter-efficient. One possibility is that reducing the parameters can reduce actual latency on device at a given accuracy compared to prior dynamic convolution approaches due to requiring less memory bandwidth, but this is not shown or discussed in the paper.\n\nThe matrix decomposition lens suggested by this paper is interesting and gives insight into prior dynamic convolution approaches. How to generate kernels for dynamic convolutions is an important open question. To strengthen this section, it would be good to clarify the mathematical relationship between the proposed approach and the vanilla dynamic convolution approach in the CondConv and Dy-Conv papers. In particular, it would be good to give a comparison as to how the expressiveness of the proposed DCD approach compares to the expressiveness of the prior approaches. This would be important to give a complete picture of how the approach compares, beyond just parameter count. Also, the dynamic channel-wise attention seems to be related to Squeeze-and-Excitation, and it would be good to give an analysis for how these are related as well.\n\nThe experimental results of the paper can be improved by demonstrating the benefit of this approach on additional and more advanced mobile architectures, such as MobileNetV3. This would do a better job of illustrating the general benefit of the approach, especially in architectures that incorporate advances like Squeeze-and-Excitation. In a similar vein, comparing on a different task type (like object detection in CondConv or keypoint detection in Dy-Conv) could also help show more generality. The ablation experiments presented in this paper are relevant and help shed light on the approach and dynamic convolution.\n\nOverall, the paper provides novel and relevant insight into dynamic convolutions, which leads to significantly reduced parameter count as shown in experiments. It could be strengthened by better motivating the importance of parameter reduction in this setting, better mathematical comparison to existing approaches, and more general experimental results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eH_VlYPx46t", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Post-rebuttal updates\nI've tentatively updated my review score from 6 to 7 for the following reasons:\n* The authors clarified the relationship of their proposed method with Squeeze-and-Excite, and promised to add an explanation to their paper.\n* The authors clarified the relationship between their method and CondConv in OpenReview comments, and promised to update the submission accordingly.\n* The authors reported additional experiments on ResNet-18 and MobileNetV3(small). These new experiments help increase my confidence that the proposed method is broadly applicable.\n* The authors provided inference time numbers for MobileNetV2 on CPU. AnonReviewer5 raised a very good point: that reducing the number of trainable parameters may lead to much faster models because of memory bandwidth considerations. I'm enthusiastic about this line of work, and think the inference time measurements provided in the rebuttal are a promising first step in this direction.\n\n# Review Highlights\n**Summary of Contributions:** The submission proposes a method for modifying a neural network so that the kernel for each convolution layer is computed dynamically based on its input. The authors claim -- and provide experiments to argue -- that this modification allows image classification networks to achieve better accuracy/size tradeoffs. The method is closely related to two existing pieces of work: CondConv (Yang 2019) and DY-Conv (Chen 2020). Compared with these existing methods, the submission's main contribution is a new function for dynamically generating/selecting convolutional kernels. The authors claim this new kernel-generating function achieves better accuracies with significantly fewer parameters than existing methods.\n\n**Score Justification:** I've given this paper a borderline score. On the positive side: the submission does a good job of evaluating the proposed method, breaking down where improvements come from, and analyzing different variants of the method. Furthermore, the method does appear to provide better accuracy/size tradeoffs than earlier related work -- especially if we care about the number of model parameters. On the negative side: the submission appears to be an incremental improvement on existing work, and Row 2 (Model = $\\Lambda W_0$) of Table 1(a) suggests that as much as 60%-70% of the accuracy gains of over a vanilla MobileNetV2 (0.5x) model can be attributed to Squeeze-and-Excite, which is a well-established technique.\n\n**Method Details:** CondConv  and DY-Conv replace a fixed convolutional kernel $W$ with a weighted sum of kernels $\\sum_{i=1}^{n}{\\phi_i W_i}$. The kernels ${W_i}$ and the gating function which computes the scalars $\\phi_1, \\phi_2, \\ldots, \\phi_n$ are jointly learned via SGD. The submission instead replaces a rank-2 kernel $W$ for a fully connected layer with a function $\\Lambda W_0 + P \\Phi Q^T$. The fixed (input-independent) matrices $W_0, P, Q$ and the input-dependent gating functions which compute the matrix $\\Phi$ and diagonal matrix $\\Lambda$ are jointly learned via SGD. The authors present generalizations of their method to 1x1 convolutions, KxK convolutions, and depthwise convolutions. They also investigate the case where $\\Phi$ is constrained to be sparse/block-diagonal.\n\n**Note:** In the case where $\\Phi$ is identically zero, I believe this formulation is equivalent or almost equivalent to Squeeze-and-Excite, where $\\Lambda$ corresponds to the output of the Squeeze-and-Excite gating function.\n\n**Pros:**\n* Careful experimental evaluation / ablation experiments (Table 1). The submission evaluates several MobileNetV2 and ResNet models, both with and without the proposed modifications. It also evaluates several variants of the proposed method to quantify the relative importance of different design choices.\n\n**Cons:**\n* Some comparisons against related work could be improved/clarified. (See below.)\n* Explanation of the proposed method can be improved. I initially had trouble understanding why SVD was used to motivate the proposed method, and initially also struggled to understand what the paper meant by \"dynamic channel fusion\". I found the direct formulation of the proposed method (Equation 5) to be much clearer and more explicit than the explanations that precede it.\n\n**Reproducibility:** While the submission mentions many of the hyper-parameters used for model training, it would be helpful if the authors could include more details about how model weights were initialized at the start of training, since this wasn't obvious to me (and I couldn't find a discussion).\n\n# Comparison against related work\nWhile submission does a good job of highlighting related work (e.g,. Squeeze-and-Excite, Hyper-Networks), it primarily compares its method against two pieces of closely related work: CondConv and DY-Conv. I had the impression that comparisons against DY-Conv are executed much more carefully than comparisons against CondConv.\n\nIn particular:\n* The submission's introduction claims that \"the small attention score $\\pi_k$ makes its corresponding kernel $W_k$ hard to learn\" for both CondConv and DY-Conv. This makes sense to me for DY-Conv, since a softmax gating function is used to ensure that the attention weights $\\alpha_i$ sum to 1. However, I don't understand why this should be the case for CondConv, which appears to use Sigmoid rather than Softmax gating.\n* Table 2 compares results obtained using the submission's proposed parameterization against results reported by the CondConv and DY-Conv papers. However, it wasn't clear to me whether this comparison properly controlled for hyper-parameter tuning. For example: Table 2 reports that a static MobileNetV2 (1.0x) baseline model obtains an accuracy of 72.0%; this matches the baseline accuracy reported in the DyConv paper but is significantly higher than the baseline accuracy of 71.6% from the CondConv paper. \n\n**Suggestions:** Comparisons against related work Table 1 should be updated to clearly show which of the models were trained with comparable hyper-parameters. In addition, the introduction should clarify which comparisons apply to DY-Conv and which comparisons apply to both DY-Conv and CondConv.\n\n# Additional Notes\n\n**Abstract:** *\"[we] reveal the key issue is that dynamic convolution applies dynamic attention over channel groups after projecting into a higher dimensional intermediate space. To address this issue [...]\"*\nIt wasn't obvious to me from reading this why applying dynamic convolution over attention groups was a bad thing. Why is this an \"issue\" or a problem that needs solving? This needs to be explained more clearly.\n\n**Introduction:** The paper introduction claims that \"joint optimization between attention scores and static kernels is challenging,\" but I didn't see any supporting evidence nearby. It would be helpful to add a citation for this claim, or a forward reference to supporting experiments.\n\n**Section 3.4:** I tried to verify the theoretical analysis, and think it's correct. The implementation of implementation/parameter complexity looks right to me; parameter complexity analysis properly accounts for the cost of computing the activation/gating weights, which is good.\n\n**Section 5.1:** Ablation experiments use MobileNetV2 (0.5x), which is a very small model. While it's great to have these experiments, I'm curious why the authors chose to focus on such a small model. How well do the findings hold up for a larger model such as MobileNetV2 (1.0x) or ResNet?\n\n**Experiments:** Table 1 includes break-downs showing how the different components of the new DCD operator affect overall accuracy/FLOPS (including a static model baseline). Just weighting the kernel channels (analogous to squeeze-and-excite) accounts for a 2.8% accuracy gain; the dynamic components  of the convolution (the paper's new proposal) accounts for another 1.6% gain. This accounts for my earlier estimate that 60% - 70% of the gains came from Squeeze-and-Excite (since $2.8/(2.8 + 1.6) =  0.64$, or around 64%).\n\n**Section 5.3:** Why is $\\sigma_\\phi$ normalized by the variance of the input feature map? (I had trouble understanding the paper's explanation; the explanation could be improved.)", "extra_scores": {}, "extra_text": {}}, {"review_id": "0IKJS7vBS3U", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a little outside my area of expertise, but I found the paper interesting, proposing a different approach to dynamic convolutions based on matrix decomposition which requires fewer parameters to achieve similar accuracies as baselines while converging slightly quicker. The key ingredient is dynamic channel fusion which replaces softmax normalized attention as a mechanism used to combine information from different channels. The experiments look convincing and the results are somewhat useful, although additional work may be required to reduce the computational complexity (along with the number of parameters as shown in this work).\n\nSome suggestions/questions:\n- discuss the reasons behind vanilla dynamic convolution limitations earlier than in sec. 3.1\n- if using dynamic conv in all layers leads to overfitting, why not try to tune regularization params to avoid this?", "extra_scores": {}, "extra_text": {}}, {"review_id": "-W9gdadJTPw", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors presented a new perspective of matrix decompoistion to understand dynamic convolution and further proposed dynamic channel fusion which achieves both dimension reduction and good training convergence. \n\nExtensive ablation study and experiments give good insight of the proposed method and helps understand its advantage. The experimental results on accuracy, training time, parameter counts, and flops demonstrate its superiority compared with other methods. Further analysis on the coefficients is interesting and also helps the understanding of the proposed method.\n\n1. As for dynamic convolution decomposition (DCD), it is not clear how to address diversity of decomposition due to no additional constraint on P, phi, and Q. How does different kind of decomposition influnce the performance? If there is not any constraint on them, do they need special initialization? \n\n2. In section 4.2, since k^2 is usually not very big, the saving on depth wise convolution is not very obvious. \n\n3. Inference speed other than Flops should be reported to show its efficiency. Flops does not perfectly reflect its speed.\n\n4. What about peak memory usage of the proposed method? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper improves the dynamic convolution operation by replacing the dynamic attention over channel groups with channel fusion in a low-dimensional space. It includes extensive experiments with reasonable baselines. Dynamic convolutions are a fruitful method for making convnets more efficient, and this paper further improves their efficiency and efficacy with a novel technique. Reviewers all agreed that the paper was clearly written (though some parts were improved after rebuttal).
Decision: Accept (Poster)