Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators

Authors: Wentao Zhang, Junliang Guo, Tianyu He, Li Zhao, Linli Xu, Jiang Bian
Primary area: generative models
Keywords: Video Generation, Transformer, Zero-Shot

Abstract:
People interact with the real-world largely dependent on visual signal, which are ubiquitous and illustrate detailed demonstrations. In this paper, we explore utilizing visual signals as a new interface for models to interact with the environment. Specifically, we choose videos as a representative visual signal. And by training autoregressive Transformers on video datasets in a self-supervised objective, we find that the model emerges a zero-shot capability to infer the semantics from a demonstration video, and imitate the semantics to an unseen scenario. This allows the models to perform unseen tasks by watching the demonstration video in an in-context manner, without further fine-tuning. To validate the imitation capacity, we design various evaluation metrics including both objective and subjective measures. The results show that our models can generate high-quality video clips that accurately align with the semantic guidance provided by the demonstration videos, and we also show that the imitation capacity follows the scaling law.

Decision: accept (poster)
Reviewer scores: 8, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729078683754

thread_type:
official_review

<thread_object>
creator: reviewer_phvw

<content>
summary:
This paper trains autoregressive Transformers on video datasets in a self-supervised manner, enabling the model to infer and imitate semantics from a demonstration video in a zero-shot way, allowing it to perform unseen tasks without fine-tuning.

strengths:
1. This paper trained a transformer model called VidIT, which can learn motion information from a reference video and apply this motion to the prediction of a target video.

2. The method is effective, as extensive experiments demonstrate significant improvements in metrics, and the visual results further prove that the method effectively learns motion information.

3. This approach is based on an interesting discovery, which the authors define as "zero-shot video imitators." This finding could inspire future work in the transformer series.

4. The method can be applied to various tasks, such as unconditional video prediction, proving its strong generalization ability.

weaknesses:
See questions

questions:
1. However, the main contributions of this paper are not entirely clear to me. I believe there might be two key contributions:
a) The development of a video imitator model capable of mimicking actions from a reference video.
b) The demonstration that transformers possess zero-shot capabilities without needing specific training for tasks involving action imitation.

If the primary contribution is the first point, then the task of "imitating actions from a reference video" has been addressed in several previous works [1, 2]. Moreover, achieving the results shown in this paper could also be done using some existing open-source video editing models. Additionally, similar effects can be achieved by controlling camera angles [3].

If the primary contribution is the second point, then readers of this paper would be highly interested in understanding why transformers possess this zero-shot ability. However, the paper only demonstrates through experiments that task-specific training and the zero-shot approach yield similar performance. In my opinion, this is insufficient to explain the transformer's zero-shot capabilities. I recommend providing evidence through analyses of feature maps or attention scores to support this claim.

2. In Section 4.1, the paper introduces four settings but ultimately selects two. It is recommended that the names of these settings remain consistent throughout the paper to avoid confusing the readers. Also, ensure that these names are consistent with the names used in the experiments in Section 5.

3. Additionally, I suggest including more details about the training process, such as the hardware used and the approximate training time.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729435016540

thread_type:
official_review

<thread_object>
creator: reviewer_g9jk

<content>
summary:
The paper introduces Video Imitator (VidIT), an autoregressive transformer model trained for zero-shot video imitation, enabling it to imitate the semantics of a given demonstration video and generate coherent video clips without requiring further fine-tuning. VidIT uses a next-token prediction objective to learn from video datasets, allowing it to generate video sequences that are semantically aligned with demonstrations. The model is evaluated on visual quality and semantic accuracy using both automatic metrics and human assessment.

strengths:
1. Zero-shot Capability: VidIT achieves a zero-shot imitation capacity, enabling it to generalize from video demonstrations without further training.

2. Versatile Applications: The model can be applied to multiple downstream tasks, including robotic simulation, video planning, and segmentation.

3. Comprehensive Evaluation Benchmark: Both objective metrics and subjective human evaluations validate the quality and semantic coherence of generated videos.

4. Scalable Model: The model demonstrates improved performance when scaled up, providing strong semantic control in video generation.

weaknesses:
1. Limited Sequence Length: Because now the mainstream video generation solutions have reached more than 5 seconds, the evaluation is primarily on short sequences; the performance on longer sequences remains untested.

2. Gap Between the Training Web Videos and Evaluation Metrics: In the benchmark constructed in this article, the domains of the evaluation metric are mostly in indoor scenes (SSv2), which is relatively limited. This is likely why adding more web videos to the training data did not make the model perform better. To get more accurate conclusions, evaluation metrics should be developed for more open scenarios and diverse tasks mentioned in the appendix, such as task imitator.

questions:
How well does the model generalize to longer video sequences compared to the current evaluations on short clips?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687011072

thread_type:
official_review

<thread_object>
creator: reviewer_vvmc

<content>
summary:
The authors proposed to train an autoregressive model for video imitation. The model training is performed in a self-supervised manner using a continuous video. Then, in the inference stage, given a demonstration and query video, the model could generate subsequent video frames of the query video that imitate the same semantics or actions conveyed by the demonstration video. The model shows zero-shot capacity when generating the imitation video frames.

strengths:
- The topic of video imitation is valuable and would attract sufficient attention in the community, especially for embodied AI, since the approach could be applied to generate a large number of unseen training data, which is costly to obtain manually in the real world.
- It is interesting that the autoregressive model trained without using demonstration videos can generalise to the imitation inference stage when the demonstration is provided.
- The proposed framework is flexible and can be extended to imitate videos conditioned on text, which might be easier to acquire than demonstration videos and practically applicable.

weaknesses:
- On the technical side, the novelty of the proposed method is limited. The combination of VQ-VAE and Transformer is commonly adopted for image generation.
- There is a lack of detailed explanation about why self-supervised training without acquiring demonstrations during training will generalise to video imitations given demonstrations. 
- The visual quality of the generated imitation video is limited as shown in Table 1. The V-Acc and P-Acc scores are quite low, suggesting that the generated video's content might differ from the expected one. So, the practical usage of the proposed method is questionable.
Failure cases would help readers understand the method's limitations. I suggest the authors provide a detailed analysis accordingly.
- The evaluation of using text as conditions is very limited.
- The quality of writing could be further improved by providing more details regarding task definition and technical details. For example, it would be better to list all the types of demonstration videos. Currently, it is unclear if any more tasks are involved other than object manipulation and camera movement (and their detailed types). The training objectives could be clarified (currently missing) for those unfamiliar with VQ-VAE/VQ-GAN for better reproducibility.

questions:
- It is confusing to define a contrast action. What is the discipline? It might be better to provide a detailed list of all the situations that are considered.
- Is it possible to provide the V-acc and LPIPS scores in Table 2? In my view, V-acc and LPIPS are more faithful and direct measurements of the visual content of the generated videos.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730747946338

thread_type:
official_review

<thread_object>
creator: reviewer_vt7n

<content>
summary:
This paper proposes a video generation based on autoregressive transformers trained with the objective of imitating a set of seed videos. In experiments, the authors show that the model is able to imitate tasks presented to it in an in-context manner. In addition, the learned representations coming from the model are also useful for classification tasks.

strengths:
Originality:
The idea of performing in-context autoregressive video generation is interesting and I have not seen it being done before. Autoregressive video generation without text is very difficult, and this approach seems to address some of the difficulties.

Quality / Clarity:
The paper is clear and easy to read. The proposed idea is simple and I believe easy to replicate from the descriptions.

Significance:
I believe that even though this paper is not producing mind blowing results such as Sora, Kling, Veo and others. The ideas proposed here can be useful for the scientific community to continue exploring the problem of video generation

weaknesses:
Weaknesses / Questions:

Missing baselines:
* Not sure if I missed this from reading the Baselines paragraph in the manuscript, but did the authors include a simple next frame prediction baseline?

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732030582460

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We sincerely thank all the reviewers for their time and effort in reviewing our work. We are encouraged to see that our work is acknowledged to "be interesting"(Reviewer vT7n, vvMc, PhVw), "be useful to the community" (Reviewer vT7n, vvMc), "widely applicable" (Reviewer G9JK, vvMc, PhVw) and "have extensive experiments" (Reviewer G9JK, PhVw). 

Taking into account all the reviewers' comments, we have provided individual responses to each reviewer. Furthermore, we have added additional analysis and experiments in the paper's appendix. The main body of the paper remains unchanged, except for clarifying the notations in the first column of Table 1. Below is a brief summary of the added contents:

* **Appendix A.4**: Failure cases. This section presents failure cases where VidIT fails to align the query video with the semantic context of the demonstration video. **Mentioned: Reviewer vvMc's Weakness 4.**
* **Appendix A.5:** Interpretability of Generalization. This section highlights the interpretability of VidIT and its ability to generalize to imitation tasks in a zero-shot manner. Also companied with a failure cases analysis on attention distribution. **Mentioned: Reviewer vvMc's Weakness 4, Reviewer PhVw's Question 1.**
* **Appendix A.6:** Full Evaluation Metrics on Baselines. This section completes Table 2's results with the all the evaluation metrics. **Mentioned: Reviewer vvMc's Question 2.**
* **Appendix B.4:** More training details for training VidIT. **Mentioned: Reviewer vvMc's Weakness 5, Reviewer PhVw's Question 3.**

We hope these updates and clarifications address the reviewers' concerns and further strengthen our paper's quality.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper demonstrates the zero-shot video imitation capabilities by training self-supervised autoregressive Transformers on video data. Such in-context capabilities enable solving unseen tasks by watching video demonstrations.

The rebuttal is well-accepted by most reviewers. After rebuttal, all reviewers are positive about this paper and two are enthusiastic. 
As raised by reviewers, a deeper investigation and analysis on varied models/architectures and real-world usage will strengthen this paper further.

The area chair recommends accepting this paper. The zero-shot video imitation capabilities provide insights into research on exploring GPT-3 like in-context learning capabilities in video generation and embodied tasks.