Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper presents a video Transformer model for video recognition. It limits the temporal window to the local area and proposes efficient space-time mixing to reduce computational costs. On Kinetics, Something-Something v2, and Epic Kitchens datasets,  the proposed video transformer model is more effective than other transformer-based models yet has the same computational cost as spatial-only attention models.
main_review: For originality, this paper extends the mixing technique in TSM to the Transformer based backbone architecture, to obtain a space-time mixing attention in Video Transformer. This combination is straightforward and shown to be effective for video recognition.

The paper is well written and well organized.

Using the proposed methods, the paper achieves similar performance with much lower computation costs on Kinetics-400, Something-Something v2, and Epic Kitchens datasets.

Concerns:
1. The experimental results are not sufficient enough. This paper only presents one version of the X-ViT model with different input frames but did not verify the effectiveness of the proposed approach on different sizes of models. 
2. The novelty is somewhat limited. This paper is more like a combination of TSM and Vision Transformer.
3. In Table 1, a larger temporal window size would lead to worse performance on video recognition. This may limit the application scenarios and hurt the generality of the proposed approach.
4. Number of parameters and speed of different models should be compared.
5. Other approaches could benefit from using more temporal clips, such as 4x3. But the presented approach could only achieve slightly better when using more temporal clips. Please explain why.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors proposed a Video Transformer model the complexity of which scales linearly (instead of quadratic) with the number of frames in the video sequence. The key ideas are 1) restricting time attention to a local temporal window, and 2) using efficient space-time mixing to attend jointly spatial and temporal locations. They empirically verified that the proposed method outperforms existing state-of-the-art models in popular action recognition datasets.
main_review: 1. The paper is well-written overall, with full details of the model in Section 3 and intuitive demonstration in Figure 1.

2. It looks like t_w, the additional hyper-parameter of local window size, is very important and its optimal value may significantly vary dataset by dataset. Although the authors included an empirical evidence on Table 1, I still believe that this may need to be tuned for other datasets differently. Especially, when the dataset contains videos that are relative longer, and thus containing less homogeneous scenes, the optimal value for t_w may be significantly larger than 1. I guess the optimal value was 1 since SSv2 contains mostly homogeneous and short videos. This interpretation is aligned with the main idea of this work that attention only within a short temporal window is enough, as in Figure 1 (d). It is okay to limit the scope of this work to this kinds of short videos only and action recognition (as opposed to general, topical video classification for longer videos like YouTube 8M or TVR), but the scope still should be mentioned clearly.

3. Other than the point in #2, the experiment was conducted clearly, and shows impressive results on multiple datasets. Ablation studies were also conducted nicely.

limitations_and_societal_impact: As mentioned in the main review #2, the proposed approach was verified only on a subclass of video datasets (short videos with homogeneous scenes, focusing on action recognition) but it was slightly over-claimed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a more efficient self-attention operation, specifically for video models, such that the compute requirements increase linearly with the length of the video. The author's contributions consist of two main things: 1) Self-attention along the temporal axis is limited to a smaller window instead of the whole sequence (with multiple transformer layers, the temporal "receptive field" grows to cover the whole video) and 2) The authors efficiently perform "space-time mixing" by constructing the keys and values for attention by concatenating features from different frames. This operation requires 0 floating point operations, but is still able to combine temporal information from adjacent frames effectively.

The authors show good experimental results. They achieve similar results to ViViT [1] and TimeSFormer [3] whilst using far fewer GFLOPs, on Kinetics, SSv2 and Epic Kitchens. The paper is also well written in general.
main_review: The method appears fairly straightforward to implement, is efficient in terms of GFLOPs and achieves comparable results to ViViT and TimeSFormer whilst using less GFLOPs. However, I do think the novelty is quite limited:

- The idea of restricting self-attention to more localised patterns has been explored before in a number of papers, ie [A, B, C, D] among others. The authors should discuss these papers more.
- The author's proposed method for efficient "space-time mixing" is very similar to that of TSM [42]. Moreover, although I am not aware of other transformer papers that do efficient "space-time mixing" in exactly the same way, it does seem quite similar in spirit to Model 4 of Vivit [1], and a discussion of this is completely missing in the paper.
- On a related note, the authors often refer to recent video-transformer papers [1, 3]. However, both of these papers propose multiple different models, and it's not clear which models in the paper the authors are referring to. For example in Figure 1, the attention pattern for TimeSFormer is their "Divided Space-Time Attention", and is also Model 3 of Vivit. On Page 5, when the authors are comparing their "Temporal Attention aggregation" to Vivit, this is referring only to Model 2 of [1]. These should be clarified in the revision.



In the experiments section, the authors state they match state-of-the-art results while being "significantly faster". Actually, the authors only report GFLOPs, so it is not correct to say that they are "faster". Rather, the proposed model uses less floating point operations, and so the authors should use more precise language here. (The same applies to the discussion of results in the supplementary) A further point here is that the author's "shift trick" does not require any FLOPs, but does increase the runtime. Note that TSM [42] also shows that the impact of the "shift trick" on runtime is architecture-specific with many caveats.

When the authors reported runtime on Page 8, it is reported with "frames/second" as the unit. A better unit would be "video clips / second" as that is independent of the number of frames used (as this is a model hyperparamter) and thus more comparable across different methods. Am I correct that as the authors process 8 frames per video, 312 frames/second = 39 video clips / second?

Table 4 would also be more clear if the number of tokens and/or number of frames processed by the model were included as separate columns.



Minor points

- Table 5: Timesformer and Vivit have the same reference.
- Line 155 to 156: [1] also shows that L_t = 1 works well, and this is identical to the author's temporal aggregation approach. 



References

[A] N Parmar et al. Image Transformer. ICML 2018

[B] R Child et al. Generating long sequences with sparse transformers. ICML 2019

[C] I Betalgy et al. Longformer: The Long-Document Transformer (the authors cited this, but did not discuss it in context)

[D] M Zaheer et al. Big Bird: Transformers for Longer Sequences. NeurIPS 2020

limitations_and_societal_impact: Adequately addressed in the main paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors focus on reducing the computation cost caused by the space-time attention modeling in the Transformer framework for video recognition. Specifically, the proposal includes 1) a space-time mixing attention module that mixes channels from neighboring frames within a local window as key-value tokens, and 2) a temporal attention layer at the end of the model to aggregate global temporal information. The authors provide promising results on Kinetics-400 and SSv2 in the paper.
main_review: **Originality:** The major proposal is the space-time mixing attention module. The authors introduce the “channel shift trick [42]” to the space-time attention module to aggregate both spatial and temporal features while reducing computation complexity. They have shown promising results (Table 5-6) to support their claims. I think the idea is novel and clearly differs from previous works (TimeSformer [3] and ViViT[3]). 

**Quality:** The submission is technically sound. The authors have provided extensive experiments and ablation studies to support their major claims. The weaknesses are list as below. 
1. The authors should provide the typical experimental values of T, S, and d (Line 98-101), otherwise, it may be hard to figure out the dominant computation cost of a full model. For example, when $2d > TS$, the complexity of MLP is larger than MSA. 
2. I’m not sure about the complexity mentioned in Line 129. In my understanding, in the local temporal window of Eq. 7, there are $(2t+1)S$ tokens, and I think the complexity should be $O(T(2t+1)^2S^2) \sim O(Tt^2S^2)$, rather than $O((2t+1)TS^2)$.
3. In my understanding, the space-temporal mixing attention mainly aggregates temporal information by aggregating a subset of channels from neighboring frames to constitute the features of a new key-value token (Line 137-140). I think another intuitive option is reducing channels of each frame (e.g., reduce to 1/t channels) and then concatenate them together as the final key-value tokens. Such a design can aggregate temporal information while also avoiding introducing additional overhead. We suggest the authors clarify their differences and prove the superiority of their proposal over the abovementioned option. 
4. I think the results of Table 2 (b) look weird as the results of three different models are totally the same. We suggest the authors carefully check the results and report the error bar of the results. 
5. The authors have highlighted the computation complexity and cost multiple times in the paper. Therefore, I think it is important to provide latency (i.e., runtime), the number of input frames, the number of tokens for all the models in Table 5 and Table 6 to support their claims. 


**Clarity:** I think this paper is well organized, however, some details of the model are not clear enough. 
1. it’s hard for me to totally understand the summary token (Line 159-166). We suggest the authors provide illustrations of the temporal attention aggregation and summary token to demonstrate their functions and differences. 
2. In Table 2 (d), * (44.2) indicates temporal average aggregation, however, it is reported as 55.6% in Line 241. The authors should check it and clarify it. 

**Significance:** This paper proposes a novel way to reduce computation cost for spatial-temporal attention modeling. The authors have shown promising FLOPs and accuracy in standard benchmarks. Latency and more details of Table 5 and Table 6 are expected in the final version. Overall, I think it is useful and helpful to others to follow similar ideas to build transformer networks for video tasks. 
limitations_and_societal_impact: It is good that the authors have discussed the ethical consideration and broader impact of their model in terms of environmental costs and the bias of data-driven systems. We suggest the authors provide specific cases and more details of the mentioned fairness of their model (in Line 314-315).
review_title: null

Meta-review: The authors propose a novel transformer architecture for video recognition whose performance scales linearly with the number of frames (as opposed to the usual quadratic scaling). To achieve this the authors restrict the time attention to a local temporal window, and introduce an efficient space-time mixing procedure. The proposed approach offers competitive results in terms of accuracy/flops tradeoffs on several popular video recognition benchmarks.

The paper was reviewed by 4 expert reviewers and received borderline ratings. The rebuttal managed to address points raised by most reviewers, who found it a valuable contribution to the community. One reviewer maintains the view that the work lacks novelty in terms of technical contributions, and that more ablation studies are necessary. After considering the manuscript, the reviews, and the discussion, I felt that the work should be accepted for publication and that a minor revision is sufficient to address the raised criticisms. 
Decision: Accept (Poster)