Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a new component for transformer architectures to model trajectory attention by modeling temporal correspondences in dyanmic scenes. They also present approximation techniques to address computation and memory challenges for high-resolution videos. The proposed contributions are demonstrated on video action recognition tasks with SOTA results on multiple datasets. 
main_review: Main contributions: 
- a new block which can be easily integrated into transformer architectures to mode the temporal correpsondence in dynamic scenes (trajectory attention) 
- approximation techqniques (Orthoformer algorithm) to reduce memory usage and minimize quadratic dependency to facilate the handling of long duration, high resolution videos 
- Integration of these contributions into Motionformer model and demonstration on video action recognition task to achieve SOTA results on Kinetics, Something-Something V2, and Epic-Kitchens datasets.

The trajectory attention is implemented by using the two stages (a spatial attention operation and a 1D temporal attention) as shown in the Figure2.  For the trajectory attention test, the paper proposed Motionformer model which is using the VIT image transformer model as the base architecture and the trajectory attention for the attention mechanism. Through ablation studies, the paper proved that the trajectory attention showed better accuracy than other attention methods (Table 4). In particular, there was a significant improvement in the SSv2 data test. This shows that the proposed model with the trajectory attention can understand fine-grained motion signals more effectively than other attention mechanisms. 
The limitation of the trajectory attention occurred when it computed the trajectory attention because it’s causing quadratic complexity in both space and time. To overcome this, the approximation algorithm is proposed. It was shown that the proposed approximation algorithm used less memory than other models and showed better results overall in Tables 3 and 5. 
At the end, Table 6 showed the proposed model (Motionformer) achieved comparable or better performance with fewer parameters than current models (ViViT-L, Tformer-L and so on) for each dataset (Kinetics 400/600 and Something-Something V2)

Overall, I believe the paper makes a strong contribution and should be of interest to the wider community. 

Questions/concerns: 

- I wonder how much the training time is different due to the prototype selection runtime bottleneck. How big is the time difference to get results in table3 (c) for 16, 64, and 128 respectively?
- Although it's already explained in the paper, I want to know how slow the training speed of the proposed method is compared to other models. I wonder how long it took to train each baseline model and the proposed model to get the last table result.
- I am curious how the proposed trajectory attention mechanism can be applied to other tasks (e.g., human trajectory prediction). Some commentary on the generality of the proposed modeling contributions on other dynamic tasks would be useful. 


limitations_and_societal_impact: I would encourage the authors to provide more commentary on how the trajectory attention model and approximation techiques can be applied to other domains and tasks which benefit from modeling temporal correspondence.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposed a novel drop-in block for video transformers. Different from transformers that work on image or language modality, this block aggregates information along motion paths. The key idea is to design an efficient algorithm to approximation the path (trajectory attention), which requires quadratic dependence of computation and memory on the input size. As a solution, the proposed Motionformer model uses the (1) ViT image transformer model as the base architecture, (2) the separate space and time positional encodings of TimeSformer, (3) and the cubic image tokenization strategy as in ViViT. Experimental results on four video action recognition benchmarks show Motionformer can achieve state-of-the-art performance. 
main_review: (+) This work brings a novel idea to video understanding. It implicitly determined motion paths by trajectory attention. It allows the network to (1) aggregate information from multiple views of the object/region, (2) reason the movement of the object/region, and (3) be invariant to camera motion.

(+) The approximation scheme can speed up the calculation of trajectory attention. It reduces the complexity in time and space, which is essential in transformer-based models.

(+) Extensive experiments are conducted on popular video action recognition benchmarks, justifying the effectiveness of this work. 

(+) The authors also provide code in the supplementary material. It is beneficial for the video understanding community. 
limitations_and_societal_impact: As explained in Section 5, Motionformer has its limitation of poor data efficiency and slow training, which is common for transformer-based methods. Since this work needs to compute trajectory attention, it has higher computational complexity.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper focused on model related motion information along the temporal domain. In particular, they proposed new drop-in block for video transformers that aggregates information along implicitly determined motion paths. The network is based on attentive mechanisms with Transformer. The new model doesn't rely on quadratic dependence of computation and memory on the input size, and is efficient to be applied to high resolution or long videos. The model was evaluated on multiple standard benchmarks, and it achieved convincing results.
main_review: Strengths:

- The idea of modeling temporal information along the object moving trajectory is convincing.

- The paper writing is fairly good, and the proposed method is novel.

- The proposed method was evaluated on most of standard action recognition benchmarks, and the performance is reasonable and convincing.

Weakness:

- The qualitative results provided in the appendix is too simple. There is only one moving object in the videos, and the attention mainly focuses on the object. It's better to show some complicated cases that at least multiple objects are moving and their corresponding attention maps.

- The proposed method is only evaluated on action recognition, which is not a problem. But as the authors mentioned that their method is efficient and effective for long videos, in the future, it is interesting to show its performance on some untrimmed video tasks, such as temporal action localization [1,2] and online action detection [3,4]. As they are more challenging due to the complicated temporal context, and modeling with trajectory attention could benefit more on these tasks.

[1] Long-Term Feature Banks for Detailed Video Understanding. Wu et al. CVPR.

[2] Learning to track for spatio-temporal action localization. Weinzaepfel et al. ICCV.

[3] Long Short-Term Transformer for Online Action Detection. Xu et al. arXiv.

[4] Online Real-time Multiple Spatiotemporal Action Localisation and Prediction. Singh et al.

- Trajectory convolution [5] shares similar ideas with the proposed method. Reviewing it as related work would be necessary.

[5] Trajectory Convolution for Action Recognition. Zhao et al. NeurIPS.

- It seems that the attention focuses on the "key" moving object in the video, could the authors explain more about the difference between the proposed method and non-local?
limitations_and_societal_impact: Yes, limitations are explained.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a novel transformer architecture in which trajectories in video are considered. The model is able to leverage its own self-attention module to build trajectories around relevant objects in the video. Furthermore, authors introduce an approximation through a set of intermediate variables which enables them to reduce the computational cost of the transformer. Authors validate their model against state-of-the-art models in three main video benchmarks. 
main_review: Summary: The architecture proposed is interesting and well validated through ablation experiments. Furthermore, authors show that the proposed model improves performance over the baselines. Finally, the approximation technique introduced is useful to reduce the computational needs. For all those reasons, I believe the paper should be accepted.

Strengths:

- The paper is well written and the novelty aspects are well highlighted and contextualised. 

- Introducing elements in the transformer which relates to the nature of the input is an interesting and useful exercise. The community has commonly relied on the transformer architecture as a black box and authors are introducing some previous knowledge in order to improve the performance by tracking trajectories through time. I think this is a good direction and authors show that can have positive impact to the final outcome of the model.

- The paper presents a good ablation study to better understand the design decisions through the paper. Furthermore, they report GFLOPS and memory used, which is very useful for the reader to understand the different models and its advantages and disadvantages. 

- The approximation through prototype vectors is a simple but effective mesure to reduce computational complexity. I think this idea can be useful for the community to not only improve this model but also other with similar operations on it. 

- The results compared to the state-of-the-art models are strong, specially in the Something-Something and EpicKitchens benchmarks.

Weaknesses / Things to improve: 

- The paper story relies on the idea that the model is able to track trajectories of objects through the video. However, only a figure in the supplemental show examples of this tracked objects. I wonder if authors could quantitatively evaluate the tracking ability with a tracking dataset.

- Some of the ablations such as Input Tokenisation or Position Encoding are not strongly related with the main innovation of the paper. I believe that it's very interesting to report those numbers for the readers, but moving those experiments to supplemental would give authors a bit more space to discuss some of the results in the supplemental such as scaling-up the transformer models like in Table 2 of the supplemental.

- It would be interesting for the reader to see a more detailed analysis on the classes where the performance difference is larger with and without the trajectory block. I think it would improve the understanding of the model by the reader. 
limitations_and_societal_impact: The authors discuss the societal impact and limitations of the work in the paper.
review_title: null

Meta-review: This paper presents work on attention in video transformers.  In particular, a novel trajectory-focused self-attention approach is developed, which essentially tracks space-time patches.  The reviewers appreciated the novel method, clear presentation, and effectiveness in empirical evaluations.  The new approach forms a drop-in block that could be used in video architectures.  The reviewers were unanimous in recommending acceptance for the paper.

Decision: Accept (Oral)