Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a new CNN module to learn video feature representations for action recognition, with a particular focus on increasing channel interactions for spatio-temporal modeling. To achieve that, the authors propose to divide feature channels into several sub-dimensions (called channel tensorization) and then perform group convolutions at each sub-dimension sequentially to improve channel interactions. An SE-like attention mechanism is also applied to further enhance feature representation. The proposed approach achieves competitive results on Kinetics400 and Something-Something, compared to some existing SOTA results. The paper also provides detailed ablation studies on the approach.

My understanding is that the main idea of the paper essentially performs channel shuffling followed by group convolutions at each sub-dimension. From this perspective, the idea is similar to ShuffleNet, which applies channel shuffling to enhance interactions between channel groups. While this provides interesting technical questions from the algorithmic perspective, from the point of view of the novelty, the paper does not appear as a strong technical contribution.

Another downside of the proposed approach is that it is not sufficiently validated by experiments. Firstly, the contribution of TE is not separated in Table 3 and 4, so it remains unclear whether the performance improvement is due to the enhanced channel interactions or channel attention. Secondly, the approach seems to be only effective when the number of dimensions is low, leaving it difficult to fully justify the advantages of high-dimensional channel tensorization claimed in the paper.

The FLOPs of the proposed approach in Table 4 is somewhat misleading. As indicated in the work of X3D, the performance gain from using more clips (i.e. >5x10) in evaluation is small. Nevertheless, most approaches are evaluated with 30 clips on Kinetics. I would suggest sticking to this general practice for fair comparison.  

Table 3 and 4 miss some existing SOTA approaches such as X3D [1], CorrNet [3] and TAM [3], which should be listed for reference. 

1) Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 200–210, 2020.
2) Heng Wang, Du Tran, Lorenzo Torresani, Matt Feiszli, Video Modeling With Correlation Networks, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020 
3) Fan, Q.; Chen, C.-F. R.; Kuehne, H.; Pistoia, M.; and Cox, D. 2019. More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation. In Advances in Neural Information Processing Systems, 2261–2270.

review_title: interesting idea, but insufficiently validated

final_score: rating: 5: Marginally below acceptance threshold
review_content: This manuscript proposes a novel convolutional operation for learning representations from video data. By decomposing the channel dimension into sub dimensions in the typically 4D video data (Time, Channel, Width, Height), one defines spatial-temporal separable convolution for each sub-dimension. This could improve the representation learning in term of efficiency and modeling quality. 

Pro: the experiments seem quite thorough. 

Con: i) I'm not fully convinced that it is beneficial to decompose the channel dimension, which is usually only 3 and relatively small compared with the other dimensions. Would it be possible / sensible to apply tensorization on the time dimension? ii) The ablation study could have included the comparison between only channel tensorization and channel tensorization and self attention, which would highlight the new contribution of the paper. 

I'd be happy to update my rating if the authors would elaborate these two points. 
review_title: An interesting idea that probably needs more justification. 

final_score: rating: 7: Good paper, accept
review_content: Summary

The paper proposes a new architecture for lightweight action classification networks, named Channel Tensorization Network (CT-Net). The idea of this architecture is the tensorization of mid-level input features in combination with an attention mechanism that allows to select relevant features. the channel tensorization can be used as intermediate building block e.g. in a ResNet alternately with a Res block.
The ablation study is done on Something-Something V1 and the overall method is compared to state-of-the-art on Something-Something V1&V2 and Kinetics-400 and is able to outperform other lightweight architecture with comparable size.


Paper strengths

- I think the ideas proposed in this paper are interesting and I would not be aware of any architecture that would use a similar arrangement of tensorization and attention in the mid-layer part to allow for a lightweight 3D convolution architecture.

- The paper shows competitive results on Something-Something as well as on Kinetics-400

- The visualization supports the claim that the attention mechanism seems to learn to focus more on relevant parts of the video clip.  


Paper weakness

- The paper does not introduce a new technique per se, but manages to combine known elements. So there might be the issue of limited novelty. 

Conclusion

I think that the overall idea of reducing the computational load in 3d convolution architectures is a valid and important topic, as simply increasing networks and training data is not working for the majority of tasks. Therefore, lightweight but powerful architectures have a need in the field. I understand that the proposed elements have only a limited novelty, but I think that the clever integration in this case might justify a publication in this case.
review_title: CT-Net

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a novel Channel Tensorized Module (CT-Module) to construct an efficient tensor separable convolution and learn the discriminative video representation. The proposed solution achieves a preferable balance between convolutional efficiency and feature-interaction sufficiency. The experiments were conducted on 3 benchmark video-classification datasets outperforming the state-of-the-art results. 

Strengths:
- The paper's novelty is the first method for exploring the spatial/temporal tensor separable convolution along each sub-dimension.
- The CT-Network framework is rigorously presented.
- Experimental results outperforming state-of-the-art results.
- The Paper is well written and easy to follow.

Weaknesses:
- The abstract is too long.
- Results of the Sport1M dataset (standard video classification dataset) are missing.
- Table 2 looks clumsy and hard to understand the results of sub-tables.
- What are the reasons for choosing something-something V1 dataset for ablation studies? How about the ablation experiments on Kinetic and something-something V2 dataset?.
- Since earlier modes were having better performance on the Resnet-152 model, the authors could have experimented with the Resent-152 model as well. Why only 50 and 101?.
- This paper needs careful proof-reading as there are major typos across the paper.


Major Comments:
- Why do authors adapt ResNet as proposed CT-Net? Any reason for this? 
- Can we adapt other pre-trained Imagenet architectures as CT-Net?
- The dataset used for the comparison of visualization in Fig.3 is missing.
- If authors can showcase the performance of proposed models on other domain datasets (say 4D fMRI or 3D medical imaging datasets), it will be more interesting.
- validation plots are missing: the accuracy vs channel-interaction plot and the accuracy vs G-Flops per clip.

Typos:
-  Page7:  In our experiments, to ensure GFLOPs is comparable with other methods. we crop the input to 256 × 256 during testing. (Sentence continuity is missing, and check “we”).

Addressed Concerns:
- corrected the typos.
- Majority of the weaknesses are addressed.
review_title: Official Blind Review#2

Meta-review: The paper focuses on the task of learning efficient representation models for video classification. To avoid the excessive computational cost of performing 3D convolutions on video, the authors propose to break the channel dimension of video representations into sub-dimensions that are treated separately. This cuts down on computation and improves classification performance over many methods in the literature. Extensive experiments were run on well-known benchmarks to justify the claims of the model. Such backbone architectures can be very useful in the realm of video understanding. The authors should be commended for the amount of work they did in the rebuttal period to address the comments and inquiries brought up by the reviewers. Extra experiments were done and more in-depth analysis was made possible. 
Decision: Accept (Poster)