Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new variant of Transformer, called Shifted Chunk Transformer (SCT), for video understanding. Compared with ViT, SCT has three major contributions: 1) SCT uses a smaller patch size ($4\times 4$ instead of $16\times 16$) to extract more fine-grained features; 2) A combination of local chunk attention (i.e., self-attention within a local window) and global LSH attention is used to efficiently process the large number of tokens at each frame; 3) A frame-wise attention between adjacent frames and a clip encoder beyond the frame-level features are used for modeling temporal information. Experiments on multiple video benchmark datasets show the effectiveness and efficiency of the proposed SCT model.


Post rebuttal:
I've read the author's response and reviews from other reviewers. I think most of our questions are well addressed in the rebuttal, and I will keep my original rating "6: Marginally above the acceptance threshold".
main_review: The paper presents a new Transformer model that achieves the state-of-the-art results with less number of parameters and FLOPs. One of my main concerns is its novelty because most components of the model, if viewed individually, are not novel and have been explored in prior work. However, the overall technical contribution of the work is still valuable, especially the idea of using small patches for fine-grained features and the methods for modeling temporal information.

Overall, the paper is well organized and clear, but some technical details are missing:
1) In L257, what is the majority voting used for obtaining final result? Is it different from the standard averaging of multiple views?

2) In Table 4, how are the chunk size and #Tokens defined? Is the chunk size 49 comes from $7\times 7$ patches within a chunk? How about the #Tokens then?

3) How is the visualization generated in Figure 3? Where is the query position? More details should be described.

I also suggest the authors to provide more ablations and analysis:
1) One important claim of the paper is that using smaller patch size can improve the performance, and the claim is verified by ablations on ViT-B. However, the change of patch size may have different impacts for different models. I'm curious about the results of using different patch size for SCT as well, to further confirm the necessarily of using really small patch size.

2) SCT use both local and global attention at each frame. An ablation on the impact of these two types of attention is suggested to better understand the model design.

3) Why using more shift MSA block or more shifted frames will lead to worse results (Table5)? The authors should provide more discussion about the negative effect of the shifted MSA block.

4) Why using more views leads to worse results (Figure 2)? This is a bit counter-intuitive as more views provide more complete information and less noise, especially for trimmed datasets like Kinetics. More analysis on this observation is required.
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a new Transformer architecture for video modeling. There are following several components with their respective design motivation:
- First, each frame is split into a large number of tiny patches (e.g. 4x4 pixels) which are then linearly projected into token vectors. Then, the authors propose Image Local Chunk Attention to enforce locality in cross attention to ensure attention is only computed among neighboring patches in local neighborhoods. The authors argue that it's necessary to have small patches for good accuracy. And the Image Local Chunk Attention is therefore there to reduce computation costs brought by small patches and also enforce locality. 
- With the features computed from Image Local Chunk Attention, the authors then apply a locality sensitive hashing (LSH) attention to approximate the true cross attention to increase efficiency. 
- Now tokens are mixed with attention for each frame, the authors then propose Shifted Multi-Head Self-Attention that computes cross-attention across frames to model object motion and spatial variances. 
- Finally, a special CLS token is extracted from each frame and passed into another clip encoder to produce a clip-level summarization, which is used for final classification tasks.
Overall, the authors proposed a set of methods to model both intra and inter frame relationships in videos using transformers and achieved SOTA results on several video classification tasks like Kinetics, UCF101, HMDB51 and Moment-in-Time. The authors also conducted thorough ablation to dissect the contribution of each components.  
main_review: + Given its impressive performance on images, it's now an impactful problem to study how to best utilize transformers for video modeling. 
+ The paper has addressed several painpoints of video spatio-temporal modeling using transformers. First, it has proposed specific modules for intra (Image Local Chunk Attention) and inter (Shifted Multi-Head Self-Attention) frame modeling. Also, it has addressed the computation issue of transformers with the local chunk attention and locality sensitive hashing. I feel the authors have done a good job integrating these different techniques into a highly performant system with decent computation footprint. 
+ Due to its high accuracy and relative low FLOPs, the proposed transformer architecture could be potentially very useful for the community, upon which more explorations can be carried out on video modeling. 
+ The paper is well-written and with good illustration. 
+ The authors have conducted thorough ablation to dissect their approach. 

- As the authors have put a lot efforts to reduce computation of transformers. It would be nice to have the actual runtime (in seconds) for the proposed model compared to previous methods in Table 6. 
- The authors designed Image Local Chunk Attention to induce locality in attention and reduce computation. It resembles the utility of convolutions. How does this compare to directly employing convolutions? 
- What's the difference between "Frame Rate" and "Frame Stride" in Table 2? 
- L286-7, "including the number of shifted MSA layers used in each image chunk self-attention" I'm confused about what this line means. 

######### POST REBUTTAL #########

I thank the authors for their feedback. The response has clarified most of my questions, thus I will keep my rating to accept the paper. 


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a novel transformer-based architecture for video processing. The input tokens are small patches extracted for each frame. The model consists of 4 modules: First, the tokens are processed locally, using a transformer with restricted, local connectivity (each image is split into multiple chunks and only the patches from the same chunk exchange messages using the self-attention layer). Second, a global processing is conducted using an efficient transformer (that use LSH attention approximation) to capture long-range interactions. In order to take into account the motion effect, a shifted self-attention is introduced, modifying the classical self-attention layer to use keys computed from the previous time step. Similar to language modelling, a special token ([CLS]) is used to extract frame-level representations that are fed into a final transformer block to compute the final video representation.
main_review: *Pros:*
- Transformers have been already successfully applied in image processing. However, fewer methods managed to extend them to the video domain.
- The proposed method is more efficient (in terms of FLOPS and params) than other transformers used for video processing, as shown in experiments.
- The experiments show improvement over existing methods, proving that the architecture is effective.

*Cons:*
 - Even if efficient transformers for spatio-temporal data represents a useful direction for the computer-vision community, my main concern regards the novelty brought by this work and the relevance for the NeurIPS community. The model shows how to combine/modify different existing models but fails to clearly validate the usefulness of each individual one for the final performance. The paper contains a set of ablations but, in my opinion, the central contributions are still not empirically validated as I will explained in the following:

- The frame-level processing consisting of an “Image Local Chunk Attention” and an “Image LSH Attention” is only compared against a ViT model. Some questions regarding this approach are still open: Is it necessary to have a local processing (the chunk-level one) besides the global one? Is this local processing better than a convolutional block, since both approaches aim to process local information? Is the global processing (LSH Attention) necessary? I would suggest to include more ablation studies by a)dropping the Image Local Chunk Attention; b) replacing it with a convolutional network; c) dropping the LSH Attention

- The “Image Local Chunk Attention” was designed to process the information locally. The same motivation and a very similar solution is proposed in [34]. A discussion regarding the similarities/differences but more importantly the possible advantages would be appreciated.

- The idea of shifting the temporal dimension to capture the objects’ motion efficiently is interesting (and as far as I know novel in transformer literature) and seems to be effective. However the paper lacks an intuitive motivation for this. What does it mean to aggregate global information based on the similarities computed w.r.t previous time steps?

- To fully validate the usefulness of the proposed model in video datasets, results on more dynamic datasets, where the temporal dimension plays a crucial role such as Smt-Smt V2 would be appreciated. The authors  provide some initial results on the Moments in Time datasets, but the comparison doesn’t seem to be beneficial. Moreover, important comparisons are missing from Table 3 in supplementary (e.g. VATT method that obtain better results)


Overall I believe that, with proper validations in terms of additional ablation studies, this work could be relevant for the  CV community. However, I don’t think that the novelty brought by this work is sufficient for the acceptance at NeurIPS.

######## POST-REBUTTAL #########

I appreciate the additional experiments included in the rebuttal. In my opinion, they validate each component and gives me more confidence in the proposed method. I am still not fully convinced by the explanation for how the shifted module models temporal information, but the quantitative results are quite impressive, especially on heavy temporal datasets (such as SSv2 and Diving-48 included in the rebuttal).

I will increase my score to 6: Marginally above the acceptance threshold, but I strongly encourage the authors to include all the ablation studies in the manuscript, as neither those experiments nor similar ones exist in the current version of the paper, and as other reviewers suggested, it is very important to understand the benefits of using each component of the model.
limitations_and_societal_impact: The limitations are not addressed in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new transformer-based architecture for video classification. The proposed architecture builds on the recently introduced vision transformer model (ViT), which first decomposes a given image/video frame into a set of non-overlapping patches. However, unlike in prior work,  instead of applying self-attention directly on patches, the authors group locally adjacent patches into local "chunks", and apply self-attention on top of these chunks. To ensure that the model can propagate local chunk information globally across the entire image/video frame, the authors introduce  shifted self-attention. Lastly, a separate clip encoder is used to incorporate temporal video information. The proposed approach achieves state-of-the-art on several action recognition benchmarks such as Kinetics-400, and Kinetics-600.
main_review: Strengths:
- Video transformers are growing in popularity. Therefore, the proposed approach that addresses several limitations of prior video transformer based methods is timely and relevant.
- The proposed approach achieves impressive results on Kinetics-400, and Kinetics-600.
- The proposed method is also more efficient than prior transformer based techniques.

Weaknesses:
- Personally, I found the draft quite difficult to understand due to the writing related issues (i.e., incoherent or grammatically incorrect sentences), missing technical details, and the technical presentation style. Below I discuss some of these issues in more detail.
- The paragraph on "Image locality-sensitive hashing (LSH) attention" is confusing and difficult to understand both due to numerous writing issues, and also due to the lack of details related to LSH. The authors talk about "bucketing" but it would be useful to include a brief technical overview so that the readers who are less familiar with LSH would still understand the approach without having to look up other papers.
- The description of the shifted attention could be significantly improved. Could the authors make a figure illustrating this concept and include it in the draft? Unlike written in the draft, Figure 1 doesn't illustrate how shifted attention mechanism works in practice. To be honest, I'm still confused about it. From Equation 6, it seems that the key features are computed from time t-1 and that there is no temporal overlap between query and value features, both of which are computed at time t. Therefore, I don't see how the temporal information is propagated into the feature representation y' (as in Eq.3 ), since the key features are only used to compute similarity matrix, but not the new feature representation y' . Isn't that a limiting factor in the model design? In my view, a new feature representation y' that incorporates information from both time t-1, and also time t would be more powerful. What is the motivation of such model design? Am I missing something? Having said this, the results in Table 5 indicate that the proposed scheme leads to ~1.5% improvement, which is quite substantial.
- The authors make claims about their approach learning inter-frame motion of objects. However, Kinetics, HMDB, and UCF are known to be heavily biased towards spatial scene dynamics. To verify these claims, one would need to conduct experiments on temporally heavy datasets such as Something-Something-V2 and Diving-48. I would urge the authors to conduct these experiments for the rebuttal.
- Many relevant experimental details are missing or they are inconsistent with the previously published work. For example, in Table 1, the authors list that ViT-B achieves 75.3 % accuracy on Kinetics-400. How's this baseline implemented? For example, in the TimeSformer paper [4] , the accuracy of a 8-frame spatial-only model is listed as 76.9 % on Kinetics-400. Therefore, I was wondering what's the reason behind this gap. Furthermore, in Table 4, the authors list ViT-B to have 196 tokens, and ~405 GFLOPs. How are these numbers computed? The 8-frame spatial-only ViT-B has the cost of ~141 GFLOPs. How many frames are all of these models using? These details should be included both in the paper, and in the tables.
- The term "image local chunk self-attention", which is used many times throughout the paper is cumbersome. I would urge the authors to rename their model to something more elegant. 
- The paper often feels like code/implementation documentation. The authors tend to focus on very low-level implementation details too much (e.g., Sections 3.1-3.4), making the paper more difficult to read. 

Questions / Comments:
- For pretraining, do the authors pretrain their model on ImageNet-21k themselves, or do they use publicly available ImageNet-21k pretrained models? If it's the former, it would be good to include pretraining related details. If it's the latter, for reproducibility, it would be good to know, which public models the authors are using, as the models displayed in Table 3 have a custom design (compared to standard ViT).
- Could the authors also include the total number of TFLOPs in their tables so that it would be easier to compare computational complexity of each method?
- In Table 5, the performance of using two layers of shifted MSA drops quite substantially, which is a bit counterintuitive. Do the authors have an explanation for this?
- In Table 4, what exactly does the chunk size (i.e., 49) mean? Does it mean that it encompasses 49 patches on each axis? It seems that this should be 7 instead of 49.
- All the tables list the # of tokens for a single frame. However, I assume the model operates on multiple frames. Therefore, it would be useful to include the number of used frames in the tables.
limitations_and_societal_impact: Limitations of the proposed approach are not discussed, i.e., Section 5 only contains limitations for previous work.
review_title: null

Meta-review: All reviewers recommend acceptance of this submission. The final ratings are: 6, 6, 6, 8.

All four reviewers acknowledge the strong empirical results, both in terms of accuracy as well as efficiency. The inclusion of results on motion-heavy datasets in the rebuttal period was deemed valuable and informative. It is recommended to add these results to the main paper. 

Reviewers comment on the poor presentation, especially the section concerning the shifted attention. The authors should use the detailed feedback given by the reviewers in order to improve the technical discussion as well as the motivation for the approach. 

The ACS agree on the recommendation of acceptance.
Decision: Accept (Poster)