Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: *********
Summary Of The Manuscript:
*********
The manuscript addresses the problem of Video Recognition one of the applications of Computer Vision. Due to various complex temporal dynamics of video data (Camera Motion, Speed, etc.), to capture the vast information, the author presents a novel Temporal Adaptive Module (TAM) for generating kernels based on the temporal feature maps. In addition, these feature maps are a combination of local and global features and as an exemplar, the author presents an architecture - TANet by incorporating their temporal operator. Together with a variety of experiments on standard benchmark for Video Recognition: Kinetics - 400 and Something-Something, the author showcases that for the task of Video Recognition compared to existing temporal operators, TAM's performance is fairly consistent and better and archives State-of-the-art with similar complexity in their exemplar architecture - TANet. 

*********
Strength Of The Manuscript:
*********
++ Novelty

- The task formulation is concise, convincing, and novel. A seemingly reasonable approach has been proposed in this manuscript for the task of Video Recognition. Compared to the existing baseline and recent approaches, the proposed architecture - TANet achieves SOTA results. 
- To the best of my knowledge, the incorporation of two branches - Local and Global branch makes the whole operator efficient and flexible for adaptation in the frameworks by stacking them to capture more complex information. Thus the concept of the TAM is convincing to capture complex temporal information.
- In addition, the exemplar showcased by the authors - TANet has been created by incorporating TAM in the existing 2-Dimensional CNNs to capture vast information which proves that the proposed module/operator is flexible enough and can be adapted to different frameworks/architectures for better performance. 

++ Clarity

- The manuscript is written in an excellent way to provide a brief insight into TAM. Especially Subsection 3.2 and 3.2 provide a good in-depth description of how the local and global branch works effectively in a joint manner to capture the short and long-term complex temporal information. 
- The manuscript also clearly describes the improvements and adequately contextualizes the contributions in such a way that it makes a good starting point for a novice reader. 

++ Evaluation

- The experiments are sufficient and convincing. This new operator and exemplar TANet shows improved performance in nearly all cases on the datasets 
- The experimental evaluations demonstrate the effectiveness of the proposed architecture and showcase its practical value.
- Also, an ablation analysis demonstrates to gain an understanding of where the performance benefits have been obtained such as receptive fields and parameter choices. 

*********
Weakness Of The Manuscript:
*********
Overall, currently at this stage, this is a very good and strong manuscript in my entire batch. I like the simplicity and wide applicability of the proposed operator, especially the incorporation of local and global branches and adaptive aggregation. Thus I do not have any major weakness issues after reading the manuscript several times. Detailed literature review, a complete overview of each component, and detailed experiments and ablation studies helps to give a good insight into the manuscript. I found this paper pretty solid and have not able to found concerns relating to the proposed work. 

*********
Justification Of The Review:
*********
Overall, happy with the current version of the manuscript. As mentioned earlier, I like the simplicity and wide applicability of the proposed module, and the architecture and setup details are provided in such a manner that it is very easy to convert into code in some timeframe. Detailed literature review, a complete overview of each component, and detailed experiments and ablation studies help to understand the author's work. Finally, I think the paper is pretty solid and thus I prefer to give a rating of 8 currently. 
review_title: Review of the manuscript TAM: Temporal Adaptive Module for Video Recognition

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a temporal adaptive module for video recognition. Specifically, it decouples dynamic kernel into a location sensitive importance map and a location invariant aggregation weight, which can be plugged into existing 2D CNNs to yield a powerful video architecture with small extra computational cost. The experiments conducted on several datasets demonstrate the effectiveness of the proposed method.

Paper Strength
(1)	The proposed method develops a simple but effective module for video recognition, achieving good performance on various kinds of datasets.
(2)	The proposed two-level adaptive modeling scheme is effective to describe motion patterns. Specifically, it decomposes the video specific temporal kernel into a location sensitive importance map and a location invariant aggregation kernel.

Paper Weakness
(1)	Figure 1 is confusing. Why the dimension of the green filter is 3*1*1?
(2)	In Figure 2, TAM is used before the 1*1*3 convolutional layer in the ResNet block. What about the influence of TAM at different locations? Is there any insight for this design?
(3)	Moreover, what if we exchange the order of the local branch and the global branch?
(4)	In Section 3.4, the authors claim that the proposed temporal adaptive module can be plugged into existing 2D CNNs with a strong ability to model different temporal structures in video clips. However, only the ResNet-50 backbone is verified in the experiment. More experiments with other backbones such as VGG and Inception should be added.
(5) In Table 3, the proposed method is not compared to the state-of-the-art X3D method [*] on the Kinetics-400 dataset.

[*] X3D: Expanding Architectures for Efficient Video Recognition.

Summary
This paper proposes a temporal adaptive module for action recognition. The proposed module is straight-forward and obtains good performance on different datasets. The experiments are not thorough enough to demonstrate that the proposed module can be plugged into different 2D CNNs with good performance. 

review_title: The experiments are not thorough enough.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a new temporal adaptive module (TAM) to generate video-specific temporal kernels based on its own feature maps. TAM proposes a unique two-level adaptive modeling scheme by decoupling dynamic kernel into a location sensitive importance map and a location invariant aggregation weight. The importance map is learned in a local temporal window to capture short term information, while the aggregation weight is generated from a global view with a focus on long-term structure.

The global branch aims to incorporate long-range temporal structure to guide adaptive temporal aggregation with fully connected layers. The adaptive kernel (aggregation weights) is learned based on long-term temporal information, and incorporates global context information and learns to produce the location invariant and also video adaptive convolution kernel for dynamic aggregation. The visualization of the statistics of kernel weights shows that the shapes and scales of distribution are more diverse and data-adaptive. It is indeed reasonable to learn spatiotemporal representation in an adaptive scheme.

However, the paper can be improved further.
1.	Some SOTA performances are ignored selectively, such as some performances of Slowfast[1]. 
[1] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, pp. 6201–6210, 2019
Maybe the proposed method does not achieve the best performance, but it is necessary to compare with the SOTA methods completely. Deep analysis can help the readers get the core contribution of the paper.
2.	The TAM only focus on the temporal modeling. This may limit the final performance of the network. If the idea can be extended to the spatial-temporal field, it will be more valuable. 
3.	The authors give the distributions of the important map V in the local branch and the kernel \Theta in the global branch. Since these V and \Theta are all data-dependent, it is not unexpected that the distributions have larger diversities than the traditional I3D’s temporal kernel. If the authors can try to explore the relationship between the distributions and the characteristic of some action samples, beyond the diversity visualization of distributions, it will be more convincing.
4.	Fig. 1 is difficult to understand. It is not straightforward to figure out how the attention weights or kernel weights are learned. Arrows in Fig. 1 are also confusing. Some indicate names, while some indicate feature flows.  

review_title: The interpretability of the distributions of the important map V and the kernel \Theta.

Meta-review: This paper proposes a temporal module for video representation learning, which is a combination of temporal attention and temporal convolution.

The reviewers' opinions diverge. R2 does not find any major flaws of the paper, while R1 expressed concerns in terms of experimental details, ablations, and missing comparison to the state-of-the-arts. R4 expressed a similar concern, while favoring the paper a bit more.

The AC agrees more with the senior reviewers (R1 and R4) that the paper misses its experimental comparison to the state-of-the-arts. In particular, the AC supports the statement from R4 that "Some SOTA performances are ignored selectively" to favor the proposed approach. The missing SOTA includes Slow-Fast as pointed out by R4, X3D as pointed out by R1, and more. For instance, X3D is able to obtain 80.4% top-1 accuracy on Kinetics-400, which is superior to 76.9% of this paper, but is being ignored in the paper. X3D that uses a different strategy to abstract temporal information than this paper is also superior in terms of the FLOPS: X3D gets 79.1% while using almost half of the computation the proposed approach is using. The authors responded by "X3D uses network architecture search strategy to discover the optimal setting for 3D CNNs", but this is a misleading statement as X3D does not use any neural architecture search method. The authors argue that their proposed approach might be able to also benefit X3D, but this has not been confirmed and we cannot judge since no quantitative results are provided. In addition, as mentioned, several other standard baselines such as Non-Local R101 (with 77.7% accuracy) and ip-CSN-152 (with 77.8% accuracy) performing better than the proposed approach are missing.

Overall, we find the experimental section of the submitted paper incomplete.
Decision: Reject