Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a novel approach for video understanding. They present temporally adaptive convolutions (TAdaConv) based on dynamic networks by adapting the convolutional weights on each frame with its temporal contexts. The experiment results on public benchmarks demonstrate the efficiency of the proposed approach.
main_review: Strength:
1. The paper is well-organized and easy to follow.
2. The proposed approach is simple and efficient. Temporal modeling by calibrating convolutional weights can achieve significant improvement while introducing less computation cost and additional parameters compared with other methods like (2+1)D.
3. TAdaConv can be easily plugged into any convolutional network based deep models for sequential modeling.
4. Adequate experiments on K400 and SSV2 illustrate the superiority of the proposed approach.

Weakness:
1. In Equation 7, the outputs of the first term and the second term are in different shapes. How do you add those features together? I suppose you may simply aggregate them together, but please describe the details clearly.
2. Please add the descriptions of GAPs as global average pooling over spatial dimension and GAPst as global average pooling over spatial and temporal dimension to make the audience have a good understanding of Equation 5.
3. According to Equation 4, v_t is the combination of features with temporal kernel size as 3. I would like to see more ablation study of kernel size over temporal dimension of TAdaConv (e.g., 3 vs 5 vs 7).
4. According to the recent progress on neural architecture search for action recognition [1], I would like to suggest the authors adding TadaConv into search space in the future work. 

[1] Kondratyuk, Dan, et al. "Movinets: Mobile video networks for efficient video recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
summary_of_the_review: Overall, the paper is clear and the author propose an efficient and practical approach for video understanding. So I would like to provide a positive score in the initial stage of the review and keep tuning during the discussion period.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work seeks to improve performance of video understanding models though the use of spatial convolutions which are dynamically adapted in time when applied to a sequence of frames. The novelty of this work lies in the fact that the weights of the spatial convolutions are adapted, rather than applying a temporal excitation to feature maps to achieve the same purpose. To achieve this, a drop-in replacement for spatial convolutions is designed which uses pooled local and global frame descriptors to produce the temporal calibration weights. Furthermore, this operation is used as part of a temporal feature aggregation block that is used as part of 2D CNN architecture. The benefits of the approach include that it can make effective use of pre-trained weights and that it can improve model capacity for a marginal increase in computation. Experimentally, it is show large performance improvements over a baseline and match state of the art on significant datasets.
main_review: Strengths:
- The main strength of the paper lies in the novelty of applying calibration weights directly to the spatial weights in the temporal dimension. Significant previous works have explored various approaches where excitation is applied to the output features in the temporal dimension, but this approach appears to have benefits in terms of computation.
- The set of experiments is well laid out and there are sufficient ablation studies over the TAdaConv components. Each significant design decision appears to be tested in the experiments.
- As a drop-in replacement for spatial convolution, it shows clear performance improvements for established architectures. It is clear that this dynamic temporal adaption is something which can be beneficial broadly in convolution-based video-understanding models.

Weaknesses:
- Computational comparison - it would have been useful to have a visualisation plot of performance vs GFLOPS to be able to compare the model against other state of the art models, given that this is one of the main benefits of the model.
- The introduction to the paper is not precise on the motivation for the work. The 3rd paragraph introduces what is sought, ("integrating the dynamic weight calibration into the spatial convolutions"), but not why it is sought. It would be useful for the reader to understand why this would be a better approach, and what are the deficiencies of the existing approaches that necessitate this.
- Performance comparisons to TDN are prefaced with the fact that TDN uses more frames. I am not sure if this is fair, given the GFLOPs of this model are in line despite the larger number of frames according to Table 7.
- The language could do with a little bit of finessing in the paper. There are some instances of grammar mistakes and some unwieldy sentences that should be made more clear to improve readability. For examples the last sentences in the 'Calibration dimension' and 'Different proportion of channels employing TAdaConv' paras on page 7.
- Some care could be made to clarify what is meant by a term before its use. For example, linear vs nonlinear weight generation, temporally sensitive on page 3.
summary_of_the_review: I believe that this work presents a novel idea of weight calibration that has benefits in terms of performance and model efficiency that for video understanding tasks. Its use as a drop-in replacement shows that it improves performance for existing architectures. It also shows that there may still be unexplored aspects for improving these model types. Despite the issues that I see in the paper, which I believe could be straightforwardly improved, I think this paper is at an acceptable threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes TAdaConv that calibrates kernel weights of convolutional layers adaptively according to the temporal dynamics of the input tensor. It is designed to incorporate both the local and global temporal context by using stacked two-layer 1D convolutional operations and global average pooling. Since it works exactly the same as the original convolutional layer at the initial stage, it is easy to insert TAdaConv into existing ConvNet architectures. On top of this, the authors construct TAda2D networks by introducing the temporal feature aggregation module that is based on a strided temporal average pooling. Many experiments performed on video/action classification and localization demonstrate the effectiveness of the proposed module.
main_review: I like the general idea of adaptive weight calibration and its simplicity. The paper was easy to read and understand, and the experimental results are extensive and show good performance. I also appreciate the detailed information presented in the supplementary material.
The following are some concerns that I would like to point out.

1) There are no comparisons with any of the dynamic modules. The idea of content-adaptive weights (or modules) has been extensively explored in the area of dynamic networks. However, this paper lacks experiments that compare the performance of TAda2D and previous approaches. In its current form, the paper only shows that using TAda2D can bring some performance gain when compared to TSN (Table 3(a) and Figure 3). I believe that more experiments should be performed to justify the use of TAda2D. For example, is TAda2D the most effective way to calibrate the convolutioanl weights? I do not think that I could find the answer from the current version of the paper.

2) I have another concern about the scalability of TAda2D. In Tables 5 and 7, using more frames brings only a small performance gain for TAda2D. For example, 16fx2x3 increases Top-1 score by 4.3% for TSM but 1.4$ for TAda2D. I am wondering if it implies that TAda2D works better on shorter input video clips.

3) It looks like TAda2D and TANet are very similar in terms of their adaptive characteristics across the temporal dimension. I believe that TAda2D should be compared more deeply with TANet. It is also interesting that TANet and TAda2D have similar patterns in Table 7 (76.3% with 43x30 GFLOPS and 76.9% with 86x30 GFLOPS).

4) I think it would be better to add a column for GFLOPS and #Params for Table 5 and A2 because the paper put weight on TAdaConv's efficiency. In addition, it is a bit hard to grasp Table 1. It would be better to add example values as already done in footnote 2.
summary_of_the_review: The paper is well-motivated and easy to understand. There are a few things I would like to point out: 1) Comparison with dynamic modules and TANet, 2) Scalability for longer input clips, 3) Clearly show the efficiency of TAda2D.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a modified version of convolution for videos where a 2D conv kernel is converted to a 3D conv kernel by multiplying the 2d weights by a per-frame value. This creates a set of weights for 3D convolution that are dynamic for each frame. The approach is evaluated on multiple video datasets, and shows some benefit over their baseline.
main_review: **Strengths**
- The paper is well written and easy to follow and understand. The approach is well described.
- The idea of generating weights is interesting and has not been greatly studied in video recognition.
- The paper conducts experiments on multiple standard datasets, so it can be compared to other approaches.


**Weaknesses**

- Eq. 1 and 2 are a bit misleading. The addition of the activation function (and normalization layers) leads to a non-linearity in the network, which has been shown to be the critical part of NNs. Omitting this part is leaving out a key piece of NNs. It makes it seem like (2+1)D convolution is just a multiplication of the weights, when in fact the non-linearity is important. This should be corrected to clarify the connection and differences between the proposed approach and convolutions.

- Similarity to previous works. The approach is not especially novel, the idea of generating temporal weights from 2d weights has been shown before. For example, see "3D Inflated TGM layer" in "Evolving Space-Time Neural Architectures for Videos", ICCV'19, as well as all the (2+1)D CNN works (e.g., S3D, " A Closer Look at Spatiotemporal Convolutions for Action Recognition",CVPR'18, etc.). It seems the main difference is that the temporal weights are dynamically generated here. However, there are no experiments showing the difference between static, learned weights and dynamic weights. Adding this experiment would strengthen the paper.

- Table 1 is quite hard to understand and see the difference between the approaches. It would be helpful to have a table comparing say a ResNet-50 with each of those configurations (or a variety of networks) to show how the flops and parameters varies. As is, it appears that this approach is less than 3D conv, but more that the others? (This seems to be part of table 2, but table 1 still is quite hard to follow).

- The claims of state-of-the-art performance are wrong. Many results on Kinetics-400 are omitted (see https://paperswithcode.com/sota/action-classification-on-kinetics-400 have many over 82% while this paper reports a best of 77.9). The claims and/or table should be updated and corrected/clarified. Similarly, S-Sv2 (https://paperswithcode.com/sota/action-recognition-in-videos-on-something) has numbers over the best reported 66.7% in this paper and epic kitchens (https://paperswithcode.com/sota/action-recognition-on-epic-kitchens-100) well over the 40.89% reported.


summary_of_the_review: Overall, the paper makes an incremental contribution by essentially making (2+1)D convolution adaptive by generating the 1D conv weights within the network. The experimental claims need clarifications and revisions, as they are currently omitting stronger performing models. The ablation experiments could be strengthened by including experiments on the effect of dynamic vs. static weights. Overall the paper makes an incremental contribution and the experiments are not currently strong enough to show the value of it. With revisions, the paper could be much stronger.
review_title: null

Meta-review: The authors study the problem of video classification and propose a new module which promises to increase accuracy while keeping the computational overhead low. The main idea is not to share the spatial convolution weights over different time steps, but allow some modulation based on pooled local and global frame descriptors. The resulting module can be used as a drop-in replacement for spatial convolutions in existing models and yields competitive performance on multiple video action recognition and localisation benchmarks.

The reviewers appreciated this challenging setting and the simplicity of the main idea. They found that the paper was clearly written, well organised, and easy to follow. The reviewers raised some issues in connections with related work and the empirical evaluation which were successfully resolved during the discussion phase.

Given that computational efficiency remains as one of the most challenging topics in video understanding, I believe that this technique will be relevant for the larger video understanding community. I strongly suggest that the authors incorporate the feedback received during the discussion, especially the GFLOPS vs accuracy plots, and further clarify the relationship to existing work.
Decision: Accept (Poster)