Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new computation block that unifies 3D convolution and transformer block for video action recognition. Starting from the transformer block, the core of the proposed block is its attention module which is equivalent to 3D CNN or transformer block depending on the design of the learnable parameters. On the standard action recognition benchmark, the proposed method outperforms prior art in both accuracy and computation efficiency. The comprehensive ablation studies provide insights on the design choices.
main_review: Strength:
- The paper makes an insightful connection between X3d's conv block and transformer block, and build a unified computational block. Compared to current methods focus on adding good properties of conv blocks to new transformer blocks (e.g., swin-transformer), the proposed method is more elegant conceptually.
- On the standard video action recognition benchmark, the proposed method achieves SOTA result, which is impressive.
- The paper provides comprehensive ablation studies on the design choices.


Weakness
- Lack of reference of previous works on the relationship between conv layer and transformer block.
- Table 2 and Table 3. The bold highlights are inconsistent. Table 3 highlights the best accuracy results among all methods, while Table 2 highlights results are not so. In addition, highlights in Table 2 may make readers ignore the fact that MoViNet-A6 results are comparable in both efficiency and accuracy.
- Lack of explanation of DPE. Currently, DPE is briefly introduced as an extension of CPE in the subsection of MHRA. However, DPE is not part of MHRA and it'll be better to have it as its own subsection.
- Lack of comparison with MoViNet-A6+transformer. Despite the bigger design space of L and G, the ablation studies show that the CNN+Transformer architecture works the best. Thus, it is unclear if the proposed method can outperform MoViNet-A5+Transformer blocks with similar GLFOPs. This is only a minor comment, as the proposed method already achieves significant enough performance for publication.
summary_of_the_review: This paper proposes a novel computational module that is elegant conceptually and effective empirically. Conceptually, the proposed block bridges the design choices between the 3D CNN and transformer block, which can be influential to future works on designing better structures in the attention module. Empirically, this paper achieves SOTA results in both accuracy and efficiency. Thus, I r
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new transformer model for video understanding task. The proposal algorithm has the benefit of both 3Dconv to efficient capture local context and transformer for global reasoning. The other innovation is the dynamic position encoding. The proposed method is validated on Kinetics and Something something where it set new accuracy record.
main_review: + Well-written 
+ Thorough ablation study, including clear visualization of the attention
+ Clean formulation for the local and global affinity token and how they imitate 3Dconv and the standard transformer model.
+SOTA accuracy with 10x more efficiency compared to strong baselines.

-It would be nice to understand how the method apply to fine grained tasks such as human keypoints or segmentation as transformers are usually not very good in such task (vs. convolution). 


summary_of_the_review: Solid paper with clean formulation, experimental analysis, and well written.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, a new architecture, Uniformer, is proposed to learn spatial-temporal pattern in videos. 
The new architecture is claimed to effectively aggregate both local information and global information.
In contrast, the previous two main approaches, 3D CNN and vision transformer, can only take care of one aspect.
The main building block for Uniformer is a Multi-Head Relation Aggregator, which behaves differently at shallow layers and deep layers, to aggregate local and global information respectively.
Extensive empirical studies demonstrate the strength of the proposed method.  
main_review: Strength:
- The empirical study is sufficient, both the state-of-the-art comparison and ablation study.
- The general idea of taking the best from 3D CNN and vision transformer is interesting. 

Weakness:
- The related work is pretty brief, which does well explain the relation w.r.t. to the literature in similar direction. E.g. (Liu et al., 2021b).
- The phrase "local redundancy" appears quite vague to me. According to the paper, the redundancy seems to refer to the pixels within a local patch, which can be reduced by CNN with 3D convolution. For me, what carried by a local patch is more of "local structure" than "redundancy", which should be capture instead of "suppress". 
- The paper introduces "DPE" module and the description can barely be found. It is not clear why DPE is able to "dynamically integrate 3D position information".
- Also, the "FFN" module is introduced with only one sentence.
- In shallow layers, the affinity map A_n becomes a learnable parameter matrix a_n. In this case, A_n is fixed for any test video. This sound counterintuitive. Please correct me if I mistook it.

summary_of_the_review: This paper explores an interesting direction on how to enable transformer-like architecture aware of both local and global information.
The experiments are carefully designed and results are encouraging. The reviewer would appreciate it if the authors address the above concerns.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a UniFormer architecture, which combines 3D convolutions and spatiotemporal self-attention for efficient and effective video classification performance. Specifically, the authors show that using local 3D convolution-like layers in the early stages of the network and global self-attention layers in the later stages of the network provides a good tradeoff between efficiency and accuracy. The proposed method reports state-of-the-art results on several major action recognition benchmarks.
main_review: Strengths:
+ The authors tackle an important problem of interest to many researchers in the video community.
+ Very strong results on multiple action recognition benchmarks.
+ Impressive efficiency considering how good the action recognition accuracy is.
+ Extensive ablation experiments.

Weaknesses:
- The paper is poorly written and is quite difficult to understand.
- The technical novelty of the paper is limited.

In my view, these two weaknesses are connected as I will discuss next. As discussed in Section 3.4, the proposed local UniFormer block is very similar to depthwise 3D convolutions. I appreciate that the authors included this section in their draft, as it makes it easier to understand their approach. However, before this section, the authors spend several paragraphs "selling" this local UniFormer block as a novel transformer block design. I think that's a bit misleading to the readers, and it also makes the paper much more difficult to read (in my view). At this point, most readers are probably familiar with depthwise 3D convolutions. Therefore, in my view, from the very beginning, the authors should describe their local block design by connecting it closely to the concept of depthwise 3D convolutions. Instead, section 3.2 is presented as a novel transformer block with many important details only included in the Appendix. This makes the draft convoluted and harder to read than it should be. Furthermore, the missing technical details wouldn't even be necessary to include if the proposed block was formulated as a depthwise 3D convolution extension. 

Connected to my point above, I don't know if the authors are using this strange formulation of their local block to increase the technical novelty of their proposed approach. However, considering the similarities with a 3D MobileNet block, I think the technical novelty of the proposed approach is quite limited. The proposed model is essentially a combination of 3D CNNs and Vision Transformers (with some bells and whistles). Personally, I wouldn't even have an issue with this because, in my view, understanding how to combine 3D convolutions and spatiotemporal self-attention is an important research question. However, I have a problem that the paper is "being sold" as a novel transformer-based architecture, as opposed to an extensive and valuable empirical study of combining 3D convolutional, and self-attention blocks. In a way, this paper reminds me a lot of the R(2+1)D paper "A Closer Look at Spatiotemporal Convolutions for Action Recognition", and I think it should be written in a similar style as this paper (i.e., empirical paper as opposed to claiming a novel architecture).  The authors might argue that the proposed local block is different than a depthwise 3D MobileNet block by pointing to the results in Table 4. However, in my view, those results largely support my claims, i.e., the gain of using the proposed block over the standard MobileNet block is marginal. Furthermore, considering my points above, I'm also not sure if UniFormer is the best name for the proposed approach. It's vague and it doesn't indicate in any way that the unification is between 3D convolutions and self-attention (as opposed to different modalities, different video tasks, or something else that's much broader).

Lastly, I also wanted to mention that the section on DPE is not very informative. The authors simply mention that they "extend the conditional position encoding (CPE) (Chu et al., 2021), utilizing a simple 3D depthwise convolution with zero paddings as dynamic position embedding (DPE)". It would be useful to add more details on this for those readers who are not familiar with CPE. 

There are also typos, and grammar mistakes in the draft (a few of them listed below):
- "Sprots" -> "Sports" in Table 2
- Wrong ViViT citation in Table 3
- "Spatial-Temporal" -> "Spatiotemporal"

I'd be interested to hear how the authors plan to address my concerns in the camera-ready version of the draft. 
summary_of_the_review: Despite the shortcomings listed above, I'm still largely in favor of accepting this paper. It provides a very valuable empirical study of combining 3D convolution-based operators with self-attention. Furthermore, the authors obtain very impressive results at a relatively small computational cost (especially compared to prior video transformer papers). The presented ablation studies would also be highly beneficial for the research community.
review_title: null

Meta-review: The paper presents an approach for spatio-temporal representation learning using Transformers. It introduces a particular architecture design, which shows an impressive computational efficiency. The reviewers agree that the experimental results are strong, and unanimously recommend the paper for acceptance. The reviewers find their concerns regarding the details of the approach/setting address after the authors' response.

We recommend accepting the paper.
Decision: Accept (Poster)