Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: # Post-rebuttal updates
I've tentatively updated my review score from 6 to 7 for the following reasons:
* The authors clarified the relationship of their proposed method with Squeeze-and-Excite, and promised to add an explanation to their paper.
* The authors clarified the relationship between their method and CondConv in OpenReview comments, and promised to update the submission accordingly.
* The authors reported additional experiments on ResNet-18 and MobileNetV3(small). These new experiments help increase my confidence that the proposed method is broadly applicable.
* The authors provided inference time numbers for MobileNetV2 on CPU. AnonReviewer5 raised a very good point: that reducing the number of trainable parameters may lead to much faster models because of memory bandwidth considerations. I'm enthusiastic about this line of work, and think the inference time measurements provided in the rebuttal are a promising first step in this direction.

# Review Highlights
**Summary of Contributions:** The submission proposes a method for modifying a neural network so that the kernel for each convolution layer is computed dynamically based on its input. The authors claim -- and provide experiments to argue -- that this modification allows image classification networks to achieve better accuracy/size tradeoffs. The method is closely related to two existing pieces of work: CondConv (Yang 2019) and DY-Conv (Chen 2020). Compared with these existing methods, the submission's main contribution is a new function for dynamically generating/selecting convolutional kernels. The authors claim this new kernel-generating function achieves better accuracies with significantly fewer parameters than existing methods.

**Score Justification:** I've given this paper a borderline score. On the positive side: the submission does a good job of evaluating the proposed method, breaking down where improvements come from, and analyzing different variants of the method. Furthermore, the method does appear to provide better accuracy/size tradeoffs than earlier related work -- especially if we care about the number of model parameters. On the negative side: the submission appears to be an incremental improvement on existing work, and Row 2 (Model = $\Lambda W_0$) of Table 1(a) suggests that as much as 60%-70% of the accuracy gains of over a vanilla MobileNetV2 (0.5x) model can be attributed to Squeeze-and-Excite, which is a well-established technique.

**Method Details:** CondConv  and DY-Conv replace a fixed convolutional kernel $W$ with a weighted sum of kernels $\sum_{i=1}^{n}{\phi_i W_i}$. The kernels ${W_i}$ and the gating function which computes the scalars $\phi_1, \phi_2, \ldots, \phi_n$ are jointly learned via SGD. The submission instead replaces a rank-2 kernel $W$ for a fully connected layer with a function $\Lambda W_0 + P \Phi Q^T$. The fixed (input-independent) matrices $W_0, P, Q$ and the input-dependent gating functions which compute the matrix $\Phi$ and diagonal matrix $\Lambda$ are jointly learned via SGD. The authors present generalizations of their method to 1x1 convolutions, KxK convolutions, and depthwise convolutions. They also investigate the case where $\Phi$ is constrained to be sparse/block-diagonal.

**Note:** In the case where $\Phi$ is identically zero, I believe this formulation is equivalent or almost equivalent to Squeeze-and-Excite, where $\Lambda$ corresponds to the output of the Squeeze-and-Excite gating function.

**Pros:**
* Careful experimental evaluation / ablation experiments (Table 1). The submission evaluates several MobileNetV2 and ResNet models, both with and without the proposed modifications. It also evaluates several variants of the proposed method to quantify the relative importance of different design choices.

**Cons:**
* Some comparisons against related work could be improved/clarified. (See below.)
* Explanation of the proposed method can be improved. I initially had trouble understanding why SVD was used to motivate the proposed method, and initially also struggled to understand what the paper meant by "dynamic channel fusion". I found the direct formulation of the proposed method (Equation 5) to be much clearer and more explicit than the explanations that precede it.

**Reproducibility:** While the submission mentions many of the hyper-parameters used for model training, it would be helpful if the authors could include more details about how model weights were initialized at the start of training, since this wasn't obvious to me (and I couldn't find a discussion).

# Comparison against related work
While submission does a good job of highlighting related work (e.g,. Squeeze-and-Excite, Hyper-Networks), it primarily compares its method against two pieces of closely related work: CondConv and DY-Conv. I had the impression that comparisons against DY-Conv are executed much more carefully than comparisons against CondConv.

In particular:
* The submission's introduction claims that "the small attention score $\pi_k$ makes its corresponding kernel $W_k$ hard to learn" for both CondConv and DY-Conv. This makes sense to me for DY-Conv, since a softmax gating function is used to ensure that the attention weights $\alpha_i$ sum to 1. However, I don't understand why this should be the case for CondConv, which appears to use Sigmoid rather than Softmax gating.
* Table 2 compares results obtained using the submission's proposed parameterization against results reported by the CondConv and DY-Conv papers. However, it wasn't clear to me whether this comparison properly controlled for hyper-parameter tuning. For example: Table 2 reports that a static MobileNetV2 (1.0x) baseline model obtains an accuracy of 72.0%; this matches the baseline accuracy reported in the DyConv paper but is significantly higher than the baseline accuracy of 71.6% from the CondConv paper. 

**Suggestions:** Comparisons against related work Table 1 should be updated to clearly show which of the models were trained with comparable hyper-parameters. In addition, the introduction should clarify which comparisons apply to DY-Conv and which comparisons apply to both DY-Conv and CondConv.

# Additional Notes

**Abstract:** *"[we] reveal the key issue is that dynamic convolution applies dynamic attention over channel groups after projecting into a higher dimensional intermediate space. To address this issue [...]"*
It wasn't obvious to me from reading this why applying dynamic convolution over attention groups was a bad thing. Why is this an "issue" or a problem that needs solving? This needs to be explained more clearly.

**Introduction:** The paper introduction claims that "joint optimization between attention scores and static kernels is challenging," but I didn't see any supporting evidence nearby. It would be helpful to add a citation for this claim, or a forward reference to supporting experiments.

**Section 3.4:** I tried to verify the theoretical analysis, and think it's correct. The implementation of implementation/parameter complexity looks right to me; parameter complexity analysis properly accounts for the cost of computing the activation/gating weights, which is good.

**Section 5.1:** Ablation experiments use MobileNetV2 (0.5x), which is a very small model. While it's great to have these experiments, I'm curious why the authors chose to focus on such a small model. How well do the findings hold up for a larger model such as MobileNetV2 (1.0x) or ResNet?

**Experiments:** Table 1 includes break-downs showing how the different components of the new DCD operator affect overall accuracy/FLOPS (including a static model baseline). Just weighting the kernel channels (analogous to squeeze-and-excite) accounts for a 2.8% accuracy gain; the dynamic components  of the convolution (the paper's new proposal) accounts for another 1.6% gain. This accounts for my earlier estimate that 60% - 70% of the gains came from Squeeze-and-Excite (since $2.8/(2.8 + 1.6) =  0.64$, or around 64%).

**Section 5.3:** Why is $\sigma_\phi$ normalized by the variance of the input feature map? (I had trouble understanding the paper's explanation; the explanation could be improved.)
review_title: Paper Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a technique to reduce parameter count and improve training stability for dynamic convolutions using matrix decomposition. It looks at prior work (CondConv, DyConv) which aggregate multiple convolutional kernels via an attention score, and suggests this vanilla formulation is redundant since it sums $KC$ rank-1 matrices to create a rank-$C$ residual. The technique proposed in the paper (dynamic convolution decomposition) reduces the number of parameters via dimension reduction in the intermediate space, instead summing $L^2$ rank-1 matrices where $L^2 < C$. The authors provide experimental results on MobileNetV2 and ResNet comparing to prior works, as well as several ablations and possible extensions on their approach.

This paper helps address one key drawback of prior dynamic convolution approaches, which is that they require many more parameters than ordinary convolutions. However, this benefit needs to be better motivated by the paper. The goal of the prior dynamic convolution approaches was to create models that have higher performance and lower inference cost. In the case of MobileNetV2, while the parameter count is reduced, the inference cost is actually higher than the previous Dy-Conv approach to achieve the same performance. For this paper to be more compelling, it would be important to show reasons why reduced parameter count may be worth the increased inference cost, especially since from purely a parameter perspective, static scaling (with depth and width) seems to be more parameter-efficient. One possibility is that reducing the parameters can reduce actual latency on device at a given accuracy compared to prior dynamic convolution approaches due to requiring less memory bandwidth, but this is not shown or discussed in the paper.

The matrix decomposition lens suggested by this paper is interesting and gives insight into prior dynamic convolution approaches. How to generate kernels for dynamic convolutions is an important open question. To strengthen this section, it would be good to clarify the mathematical relationship between the proposed approach and the vanilla dynamic convolution approach in the CondConv and Dy-Conv papers. In particular, it would be good to give a comparison as to how the expressiveness of the proposed DCD approach compares to the expressiveness of the prior approaches. This would be important to give a complete picture of how the approach compares, beyond just parameter count. Also, the dynamic channel-wise attention seems to be related to Squeeze-and-Excitation, and it would be good to give an analysis for how these are related as well.

The experimental results of the paper can be improved by demonstrating the benefit of this approach on additional and more advanced mobile architectures, such as MobileNetV3. This would do a better job of illustrating the general benefit of the approach, especially in architectures that incorporate advances like Squeeze-and-Excitation. In a similar vein, comparing on a different task type (like object detection in CondConv or keypoint detection in Dy-Conv) could also help show more generality. The ablation experiments presented in this paper are relevant and help shed light on the approach and dynamic convolution.

Overall, the paper provides novel and relevant insight into dynamic convolutions, which leads to significantly reduced parameter count as shown in experiments. It could be strengthened by better motivating the importance of parameter reduction in this setting, better mathematical comparison to existing approaches, and more general experimental results.
review_title: Insight into dynamic convolutions that reduces parameter count

final_score: rating: 6: Marginally above acceptance threshold
review_content: Authors presented a new perspective of matrix decompoistion to understand dynamic convolution and further proposed dynamic channel fusion which achieves both dimension reduction and good training convergence. 

Extensive ablation study and experiments give good insight of the proposed method and helps understand its advantage. The experimental results on accuracy, training time, parameter counts, and flops demonstrate its superiority compared with other methods. Further analysis on the coefficients is interesting and also helps the understanding of the proposed method.

1. As for dynamic convolution decomposition (DCD), it is not clear how to address diversity of decomposition due to no additional constraint on P, phi, and Q. How does different kind of decomposition influnce the performance? If there is not any constraint on them, do they need special initialization? 

2. In section 4.2, since k^2 is usually not very big, the saving on depth wise convolution is not very obvious. 

3. Inference speed other than Flops should be reported to show its efficiency. Flops does not perfectly reflect its speed.

4. What about peak memory usage of the proposed method? 
review_title: new perspective of understanding dynamic convolution

final_score: rating: 7: Good paper, accept
review_content: This is a little outside my area of expertise, but I found the paper interesting, proposing a different approach to dynamic convolutions based on matrix decomposition which requires fewer parameters to achieve similar accuracies as baselines while converging slightly quicker. The key ingredient is dynamic channel fusion which replaces softmax normalized attention as a mechanism used to combine information from different channels. The experiments look convincing and the results are somewhat useful, although additional work may be required to reduce the computational complexity (along with the number of parameters as shown in this work).

Some suggestions/questions:
- discuss the reasons behind vanilla dynamic convolution limitations earlier than in sec. 3.1
- if using dynamic conv in all layers leads to overfitting, why not try to tune regularization params to avoid this?
review_title: Good paper, although not the easiest read

Meta-review: This paper improves the dynamic convolution operation by replacing the dynamic attention over channel groups with channel fusion in a low-dimensional space. It includes extensive experiments with reasonable baselines. Dynamic convolutions are a fruitful method for making convnets more efficient, and this paper further improves their efficiency and efficacy with a novel technique. Reviewers all agreed that the paper was clearly written (though some parts were improved after rebuttal).
Decision: Accept (Poster)