Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present ODConv, a type of dynamic convolutional operation. ODConv combines two prior ideas, i.e. (1) filter recalibration with attention in SENet and (2) additive kernels in CondConv/DyConv, and also generalizes to all remaining dimensions of convolutional filters. The authors propose to use ODConv as drop-in replacements for regular convolutions in standard CNNs. Experiments and analysis over several tasks demonstrate that ODConv has noticeable advantages over alternatives and offers a good tradeoff between performance and compute. 
main_review: First, I think the topic of the work is an interesting and valuable one. The two ideas that this work borrows from prior methods, (1) filter recalibration with attention in SENet and (2) additive kernels in CondConv/DyConv, are both well-received and are known to bring meaningful improvements to base architectures. Consequently, an effort to combine and generalize these ideas is indeed intriguing and may bring further improvements and insights.

Regarding the technical details of the approach, it's heavily based on prior work, particularly the design of the attention computation. The changes and additions are justified well. The method is overall quite efficient without too much overhead, due to the separate attention weights for different dimensions. 

The experiments are quite comprehensive, covering multiple backbones, reference methods, and tasks. It is particularly welcome that they help provide a complete picture of the tradeoffs by also comparing the model sizes and MAdds. Overall, the results indicate that the lighter model (ODConv 1x) offers comparable or slightly better accuracies with less compute, while the larger model (ODConv 4x) provides consistent accuracy improvements. The authors also provide a wide range of ablation studies, which help a better understanding of various aspects necessary when deploying the technique to other applications. 

A slight shortcoming is that it appears there might be better combinations of the hyperparameters, at least for certain scenarios, that are missed. This is understandably due to the many factors that are involved. For example, it was concluded that "r=1/16 strikes the best tradeoff between model accuracy and efficiency" (page 8). However, increasing r to 1/4 almost closes the gap to ODConv 4x while adding a much smaller overhead. Attention sharing, investigated in Tab. 12, is another such opportunity for better accuracy with little overhead compared to using more kernels.

The author does not discuss any potential limitations or disadvantages compared to reference methods. Is it the conclusion that ODConv offers improvements over competing methods, e.g., CondConv, DyConv, SENet, without any drawbacks? Some insights in this regard can help guide applications.

Another discussion or analysis I think can be valuable is the comparison with dynamic-filter/kernel prediction networks. Since all dimensions are now being recalibrated, the approach is closer to those types of methods. For example, ODConv can be seen as a separable version of predicting full kernels.
summary_of_the_review: The idea and technical approach are well motivated and justified. It connects and generalizes two important prior ideas and offers an elegant solution. The results are convincing as the experiments show consistent advantages over a range of tasks and base architectures. The work also has good potential in practical applications due to its convenience as a drop-in replacement for standard convolutions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a dynamic convolution that is equipped with attention layers in all dimensions. Extensive experiments show that the proposed dynamic convolution can yield a more powerful representation than counterparts, especially in light-weight backbones.
main_review: Highlights:
1. The authors provide a detailed analysis of dynamic convolution operations and reveal the limitation of previous works.
2. Based on the analysis, the authors propose omni-dimensional dynamic convolution (ODConv), where multiple attention layers are employed to generate attention weights of different convolution dimensions.
3. The proposed method can be plugged into most existing CNN architectures and has good performance on various public datasets, such as ImageNet and MS-COCO.
4. The technical part is easy to follow, and the experimental part is comprehensive.

Drawbacks:
Actually not too much. The following are just some minor questions:
1. As shown in Table {1,2,3}, the improvement brought by ODConv decreases as the model size increases. So it seems that the improvement of strong baselines (#param > 100M) would be limited.
2. Although advanced training tricks are not necessary for clean performance comparisons, it would be better to have a study about whether ODConv can also work well when using heavy data augmentations and a longer training schedule.

summary_of_the_review: In summary, this work presents a new dynamic convolution that can provide a powerful representation for image classification and object detection. The paper is well written and properly structured, and the extensive experiments prove the effectiveness of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a dynamic convolution method based on Dynamic Conv (dyconv) and Coordinate Conv (cordconv) by incorporating more attention weights to the multiple convolutional kernels to mix kernels more effectively resulting in better accuracies on the ImageNet classification and COCO object detection tasks.  
main_review: Pros)
+ The paper is written very well and easy to follow.
+ The idea is simple, and therefore the message is clear. The results look promising.
+ Experiments are well performed with popular competitors.

Cons)
- Involving orthogonal attentions seems complementary to each other, but any intuitions why decomposing like those work well compared with the naive ones (in dyconv and cordconv) are not clearly stated.
- Too many italics harm the readability 


Comments)
1. Did the authors run all the experiments including the competitive methods in Table 1 and Table 2 by themselves? 
2. Why ODConv is faster than DyConv in Table 8 (right)?
3. Please specify the training budget including speed compared with condconv and dyconv.
4. It would be better to report the model speed in Table 9 to confirm the speed improvements from diverse configurations.
5. Please specify the # parameters and FLOPs in Table 4. If possible, it would be nice to report the model accuracy as well.
6. In the COCO detection experiments, did the authors use ImageNet-pretrained models without dynamic convolutions or the pretrained models with dynamic convolutions reported in Table 1 and Table 2? 
7. How does the model accuracy of CondConv used at every layer like the proposed method can be changed?
8. Can the authors visualize a_si, a_ci, a_fi, and a_wi to provide some insights? I am just curious about a trend of the learned attention vectors and which one is dominant.
9. The combination with the proposed method (additionally including dyconv and condconv) and SE-Net variants performing attention on a feature is likely to be complimentary. Can the authors show further improvement upon architectures incorporating a SE-like module?
summary_of_the_review: This paper is well written, and the results about the accuracy improvements look promising. A major drawback is the actual inference speed when compared with the competitors, but it would not be a matter because one can constrain the model to the target speed in a particular circumstance. I recommend the authors study more the way of speedup the model in practice for practitioners.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work mainly focuses on designing a new dynamic network for large-scale image recognition problems. Specifically, the author discussed the weakness of the existing dynamic convolution operations, and based on the analysis the author proposed a novel framework with the name ODConv. Extensive experiments confirm the superiority of the proposed new framework.
main_review: The paper starts by analyzing the problem of traditional dynamic convolution frameworks, e.g. CondCond and DyConv, and find the previous networks are not fully adapted with the kernel-wise dynamic coefficient. In order to overcome this issue, the author proposed to dynamically weight all four dimensions for the convolutional kernel. This idea can be viewed as a more general form of dynamic convolution and SE, which I think is quite novel. Another strength of this paper is its comprehensive experiments. Both the results in the main paper and the supplementary further justify the benefits brought by the newly designed ODConv. 

My main concerns about this submission are as follows.

1. In Table 7, the ablations for the performance of dynamic operation added on different dimensions are shown. From the first three results, I observe the gain brought by the dynamics added on the spatial dimension (70.25% vs 72.42%) is the largest compared to the other two, which is very weird to me. Based on the design of the branch that generates dynamic coefficients, I find the features are always globally averaged, which removes the spatial information. I can hardly understand why the spatially averaged features can still generate the dynamic coefficients to adjust the kernels from the spatial view. Furthermore, I doubt the effectiveness of adding dynamics to the input channels, since the input features have been dynamically adapted by the previous layer. Although non-linearity might exist between two convolutional layers, the activation functions do not fuse channels and will not thoroughly break the dynamic properties. Thus I doubt some offsets might happen and weaken the efficiency of the proposed model.

2. The experimental results are just built on ResNet and MobileNet V2. MobileNet V3, which is more efficient, is not compared with. I wonder whether the author can further compare ODConv to CondConv and DCD with MobileNet V3 as backbone. Moreover, in the object detection part, I do not find the results for DCD. I'm curious about how DCD performs on object detection and whether the proposed ODConv is still better than DCD.

3. In Table 8, the inference time is tested, which is good. But I do not find the latency for the static model. Besides, for the inference time tested on CPU, I'm not sure whether multiple cores are used or not. Please clarify. 

4. In the paper the author claim "ODConv with only filter-wise attention αf1 will be reduced into performing SE" is not accurate. For SE, the expression is y=f(x) \dot x, while ODConv with only filter-wise attention is y = f(x) \dot wx. The key difference is whether the dynamic coefficient f(x) is generated based on the input or output of the convolutional layers.
summary_of_the_review: Overall speaking, the paper has novelty and very comprehensive experiments to support it. I am prone to accept this submission. But the weakness of this paper is too experimental-oriented without thorough analysis on the results. My first concern is critical, since I want to understand what happens when different dimension is dynamically adapted. If possible, I hope the author to add some visualization or other analysis on the dynamic coefficient from the statistical view to help readers have a better understand of the proposed dynamic mechanism.
review_title: null

Meta-review: This paper presents ODConv, a convolution pattern which uses attention in the convolutions across all dimensions of the weight tensor.
The paper is well motivated and well explained, easy to follow.
This work is built on top of previous work, but reviewers all agree that the contributions of this paper are significant.
The experimental section is comprehensive, with several benchmarks, and show clear improvements.
The reviewers suggested a few additional remarks, and discussions to add to the paper, which the authors have addressed in the rebuttal. Reviewers seem in general happy with the authors answers to their concerns.
This seems like a sound and meaningful paper. I am fully in favour of acceptance, and I recommend this paper to be presented as a spotlight.
Decision: Accept (Spotlight)