Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
The authors aim at learning a generic motion prior, to ease different video-related tasks (motion prediction, video labeling). They argue that learning the linear combination of temporal embeddings of a Transformer from videos enables to encode the dynamic of an observed moving object. The approach is validated on motion prediction from a single input frame (synthetic dataset) and video labelling (on two classic benchmarks). Results show the effect of such a prior. 
strength_and_weaknesses: 
Learning a motion prior is certainly a long-standing open issue. The approach proposed by the author is extremely simple (learn a weight matrix). Several applications are illustrated and experimental results developed. 

The first set of experiments (Movi-A) seems controversial to me. Given a single frame as input, the prediction of future motion of a given object is not unique (there is no deterministic motion from a single frame, there are potentially an infinity of plausible motions), at least in real videos. Hence, I do believe that the prior learnt in this context is not what is expected in real life and that quantitative results in this set of experiments are irrelevant. The learnt motion prior will however certainly 'remenber' simple dynamics, a similar task that in SSv2. 

In SSv2, the quantitative improvement between ViT and ViT+PatchBlender is not significant (0.27%). It would have been interesting to plug-in the PatchBlender onto several other approaches dedicated to video labeling (most are now using Transformers in a way or in an other) and to report these additional results. 

It seems to me that although the idea of linear combination of temporal embedings via the learnt weights matrix R might have some interest, it might be a very 'rigid' prior, ie limited to learn very generic dynamics (eg, go right, left, etc), but unable to cope with more complex motions of semi-rigid objects (ie human activity) or of multiple objects. It would be interesting to discuss the limitations of the approach. 

clarity,_quality,_novelty_and_reproducibility: 
The paper is well presented. The authors performed thorough experiments, attempting to show the pros of the proposed approach. The results are well illustrated and analysed. The approach is simple  and well described, it could be re-implemented by a graduate students. 

summary_of_the_review: Learning motion prior is a super interesting subject. The merit of the present work is to try to validate the approach on different tasks, to illustrate the generability of the prior. The proposed weight matrix however does not seem  to do a significantly better job that the transformer itself.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: To prompt token interaction over time, this paper proposes PatchBlender, a learnable matrix to mix tokens over time. Via inserting the PatchBlender in the middle layers of ViT, it improves the ability for temporal modeling. Experiments on MOVi-A and Something-Something v2 support the effectiveness of PatchBlender.
strength_and_weaknesses: Strength:
1. The design of PatchBlender is simple yet effective for MOVi-A. 
2. The visualizations clearly show the motivation of PatchBlender.
3. The experiment details are provided and easy to follow.

Weakness
1. Considering MOVi-A is an easy benchmark without complicated object interaction and camera movement, the real improvement of PatchBlender for daily activity recognition is doubted. As expected, the improvement on Something-Something v2 is marginal. And it even decreases the performance on Kinetics.
2. The paper only conducts experiments based on ViT, thus the generality is not well-demonstrated. For example, does PatchBlender also help popular video transformer like TimeSformer[1], VideoSwin[2], and UniFormer[3]? In my opinion, those models with temporal operation may not be complementary to PatchBlender.
3. The PatchBlender is actually a learnable linear layer conducted in the temporal dimension. Considering the optimization problem, it may not work as well as simpler temporal convolution, which has been demonstrated to be effective in previous 3D CNNs.

> [1] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference on Machine Learning, 2021. 
>
> [2] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 
>
> [3] Kunchang Li, Yali Wang, Gao Peng, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatial-temporal representation learning. In International Conference on Learning Representations, 2022. 
clarity,_quality,_novelty_and_reproducibility: This paper is clearly organized and the experiments are easy to reproduce. Considering the similarity of PatchBlender and the temporal linear layer, the novelty of the work is limited. Most of the experiments are conducted on simple MOVi-A, and the quality of experiments is doubted.
summary_of_the_review: The paper proposes a simple PatchBlender, which is plug-and-play and helpful for temporal modeling. However, it seems PatchBlender only works for vanilla ViT and the easy benchmark without complicated object interaction and camera movement. Considering the diversity of human action, the effectiveness of PatchBlender for daily activity is limited, which is also represented in the unsatisfactory results on Something-Something and Kinetics. Besides, it may not work for the popular video transformer due to the simple design of the temporal linear layer. More experiments of different backbones and more comparisons are needed to demonstrate its generality and effectiveness.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this work, the authors propose to tackle temporal modeling in video data. To achieve this goal, they introduce a PatchBlender layer, which uses a learnable matrix to describe the temporal dynamics among frames. They use MOVi-A, Something-Something v2 and Kinetics for model evaluation.
strength_and_weaknesses: 1 Strength
1) The paper is well written.
2) The temporal modeling in ViT is an important problem for video representation learning.

2 Weakness
1) The novelty is relatively limited. Such PatchBlender is quite similar to temporal convolution. Hence, the design is bascially to insert a temporal-conv-like layer into the spatial ViT. Such architectures are not new in the literature such as UniFormer.
2) The experiment is weak to support the claim.
2.1) It would be interesting to show the experiment where temporal convloution is inserted into ViT.  This is an opportunity to show what is the difference between two operations, and which one is better.
2.2) It lacks the state-of-the-art comparison on these video benchmarks.
clarity,_quality,_novelty_and_reproducibility: 1 Clarity: This paper is written well. Most parts are clear.

2 Quality: The experiments are weak.

3 Novelty: The novelty is limited to some degree. Such operatoion is closely relevant to temporal convolution.

4 Reproducibility: It seems to be OK for simple re-implementation.
summary_of_the_review: The paper considers an important problem in video understanding. However, the proposed operation is similar to temporal convolution, and the experiments ate weak to support the claim.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes to learn a generic motion prior, that takes the form of  linear combination of temporal embeddings in video transformers  to help different video-related tasks (motion prediction, video labeling). The approach is validated on motion prediction from a single input frame (synthetic dataset) and video labelling (on two classic benchmarks). The reviewers raise concerns regarding lack of comparisons of the proposed model with other video adaptations of transformer models, such as TimeSformer, VideoSwin, and UniFormer, or temporal convolutions. They further note that the contributions are marginal. 
The rebuttal submitted by the authors does not sufficiently address these concerns. Given the lack of empirical validation of the proposed idea, the paper is not suggested for publication in ICLR.

justification_for_why_not_higher_score:  The reviewers raise concerns regarding lack of comparisons of the proposed model with other video adaptations of transformer models, such as TimeSformer, VideoSwin, and UniFormer, or temporal convolutions. They further note that the contributions are marginal. 
The rebuttal submitted by the authors does not sufficiently address these concerns. Given the lack of empirical validation of the proposed idea, the paper is not suggested for publication in ICLR.
Decision: Reject