Autoregressive Transformers are Zero-Shot Video Imitators

## Abstract

Abstract In-context learning for vision data has been underexplored compared with that in natural language. Previous works studied image in-context learning, urging models to generate a single image guided by demonstrations. In this paper, we propose and study video in-context learning, where the model starts from an existing video clip and generates diverse potential future sequences, each semantically guided by the prompted video demonstrations.
To achieve this, we provide a clear definition of the task, and train an autoregressive Transformer on video datasets. We thoroughly analyze the effect of different datasets and represent frames as discrete tokens, and then model them by next token predictions.
We design various evaluation metrics, including both objective and subjective measures, to demonstrate the visual quality and semantic accuracy of generation results. Our model follows the scaling law and generates high-quality video clips that accurately align with the semantic guidance provided by in-context examples.

## 1 Introduction

Large language models (LLMs) [, , , ] demonstrate fascinating abilities to generate different responses to a given query when providing various demonstrations, this in-context learning [] capacity helps LLMs generalize to zero-shot tasks smoothly. Compared to natural languages, visual demonstrations such as images and videos are abundant and can convey broader information about the physical world (e.g., spatial and temporal information in different scales of resolutions) [], making the visual in-context learning an appealing and essential capacity for an ideal world model that perceives visual signals and makes responses correspondingly.

Previous works [, , , ] have studied *image in-context learning (Img-ICL)*. By formulating image perception tasks such as segmentation and detection to image pair demonstrations in delicately designed structures, e.g., a joint grid image [] for MAE models [] or a sequence [] for Transformer models [], they encourage the model to mimic the task and generate the prediction of a given image query. Img-ICL focuses on perception tasks and falls short of generating semantic and sequential responses, while large language models have shown remarkable capabilities in generating coherent and contextually appropriate textual sequences.
This gap highlights the potential for extending in-context learning to video data, where the temporal dimension provides dynamic causality and richness that static images cannot provide.

In this paper, we bridge the gap by introducing *video in-context learning (Vid-ICL)*. Unlike image-based approaches, Vid-ICL uses videos as the fundamental unit for both demonstrations and outputs. Demonstration videos are highly versatile and capable of conveying a wide range of information, such as examples for various tasks including moving or grabbing objects, or movements of the camera in an ego-centric video. Guided by these demonstrations, the model receives a query video clip and is expected to generate a subsequent video sequence that is semantically aligned with the demonstrations (see Figure 1 for examples). This allows Vid-ICL to address multiple downstream tasks, such as embodied planning and simulating, by letting a query robot imitate the actions demonstrated by other robots. As videos are good at describing low-level details (where language may fall short) and temporal dynamics (where images are insufficient), Vid-ICL serves as a crucial interface for models to interact with the real world.

We show that an autoregressive Transformer [, ] trained with next token prediction demonstrates *zero-shot* capacity of Vid-ICL, with a thorough analysis and selection on public video datasets. Each training sample comprises a sequence of frames extracted from a single video, encoded as discrete tokens using a vector-quantized encoder [, ]. The model learns to predict future frames based on preceding ones through self-supervised learning. Though in-context examples are not provided during training, the autoregressive paradigm allows the model to learn the underlying structures and patterns present in the videos. Consequently, when exposed to in-context examples during inference, the model can generalize and apply these learned patterns to generate coherent and contextually appropriate video sequences. This zero-shot capability mirrors the findings observed in large language models [].

To comprehensively and accurately evaluate the model performance in Vid-ICL, we develop both objective and subjective metrics to assess the generated videos in terms of visual quality, semantic accuracy, and consistency with the prompted demonstrations. Our extensive experiments demonstrate that the model not only produces high-quality video clips but also successfully adheres to the semantic guidance provided by the in-context examples. In addition, we show that the zero-shot Vid-ICL capacity also follows the scaling law [] of large models, illustrating the potential of future works.

The main contributions of this work are summarized as follows:

- •
We propose and study the task of video in-context learning, which enables the model to interact with real-world demonstrations through video generation.
- •
We train a large Transformer model that exhibits powerful video in-context learning capacity, which also follows the scaling law of large models.
- •
We propose various evaluation metrics to evaluate the visual quality and semantic accuracy of generated videos, providing a solid benchmark for the evaluation of video in-context learning.

## 2 Related work

#### In-context Learning for Natural Language

In-context learning (ICL) has emerged as a pivotal feature of large language models (LLMs), obviating the need for parameter updates and thereby benefiting the downstream usage of LLMs [, ]. A standard practice involves providing several demonstrations and a query, then tasking the LLM with producing the answer. Different prompting designs [, , ] empower LLMs to deal with a wide range of natural language understanding and generation tasks.

#### In-context Learning for Vision

Pioneering works construct vision ICL as an image inpainting [, , ] task. Given multiple query-answer image pairs arranged in a grid image, models are optimized to reconstruct the masked answers under the MAE projective []. Recently, LVM [] flatten image pairs into a sequence and train an auto-regressive Transformer with next token prediction. These models show powerful image in-context learning capacity on perception tasks such as semantic segmentation, object detection, and depth estimation,
by identifying the prompting task from demonstration pairs and generating the prediction of the query image.
When extending the basic elements (i.e., demonstrations, queries and predictions) from images to videos, unfortunately, existing models struggle to handle the increased complexity as they are not designed to capture the spatiotemporal relationships of inputs.
Specifically, though LVM [] is able to generate consecutive frames of a query video clip, the in-context capacity (e.g., generating different consequences with different demonstrations) is not demonstrated.

#### Video Generation

The studied Vid-ICL can be considered as a conditional video generation problem, which has recently become a prominent focus in research. Text is the most commonly utilized condition, and various text-to-video models have shown promising results in generating high-fidelity videos [, , , , , , ]. Recently, video generation models are spread to other related domains such as embodied AI [], where they are utilized in visual planning [] and simulation [] with actions and observations as conditions, illustrating the potential of this area.
As for model architectures, Diffusion models [, ] and Transformers [] are both widely adopted in different scenarios, and we choose the autoregressive Transformer as the backbone model in this paper, as its abilities to model long-range dependencies and contextual relationships are crucial for in-context learning.

## 3 Video In-context Learning

We introduce Video In-Context Learning (Vid-ICL) in this section. We begin with the problem definition of both image and video in-context learning in Section 3.1, highlighting the differences between them.
We then elaborate on the training and inference pipeline of Vid-ICL in Section 3.2. Dataset selection is discussed in Section 3.3.

Figure: Figure 2: The framework of Vid-ICL. Left: Training the model. The data used for training are continuous video clips and the Transformer is trained by next token prediction objective. Right: Video In-context Inference. The model is conditioned on demonstration videos and generates the subsequent frames of a given query video.
Refer to caption: x2.png

### 3.1 Problem Definition

#### Image In-context Learning

Typically, Image In-Context Learning (Img-ICL) follows the setting akin to question answering: the demonstration is composed of $k$ image pairs $D^{I}=\{s^{q}_{i},s^{a}_{i}\}_{i=1}^{k}$,
where $s^{q}_{i}\in\mathbb{R}^{(3,n_{r},n_{r})}$ denotes the input image with resolution $n_{r}$ and $s^{a}_{i}$ denotes its target (usually an image with task-specific annotations []).
Given a query image $x^{I}$, the model predicts its target $y^{I}$ conditioned on the demonstration:

$$ $f_{\theta}(x^{I})=P(y^{I}|x^{I},D^{I}),$ (1) $$

where $f_{\theta}(\cdot)$ represents the utilized vision model such as an MAE-pretrained ViT [, ] or an auto-regressive Transformer [], which is expected to discern the inherent task from image pairs $(s^{q}_{i},s^{a}_{i})$ and process $x$ accordingly.

#### Video In-context Learning

In contrast to Img-ICL, Vid-ICL takes videos as the basic element. Specifically, the demonstration $D^{V}=\{(s_{i}^{1},\cdots,s_{i}^{n_{i}})\}_{i=1}^{k}$ comprises $k$ video clips, and each clip consists of $n_{i}$ frames. Given a query video clip $x^{V}=(s_{q}^{1},\cdots,s_{q}^{n_{q}})$, the objective of Vid-ICL can be written as:

$$ $f_{\theta}(x^{V})=P(y^{V}|x^{V},D^{V}),$ (2) $$

where $y^{V}=(s_{y}^{1},\cdots,s_{y}^{n_{y}})$ denotes the generated video clip, which should be perceptually coherent with the query $x^{V}$ while semantically consistent with the demonstration $D^{V}$ at the same time.

### 3.2 Approach

Training a Transformer for vision tasks typically consists of two stages, 1) training a visual tokenizer such as VQ-VAE [, ] to convert each image to discrete tokens; 2) each training sample is constructed as a sequence of tokens to train the Transformer decoder. For the first stage, we utilize a public pretrained checkpoint [] with $16\times$ spatial compression rate as the VQ tokenizer. Our framework also fits for recent tokenizers with both spatial and temporal compressions [, ], and we leave it for future work.
Formally, for a training video clip $x^{V}=(s^{1},\cdots,s^{n})$
with $n$ frames, the VQ tokenizer converts it to a flat sequence $x^{t}=(t_{1},\cdots,t_{n_{t}},t_{n_{t}+1},\cdots,t_{N})$, where $n_{t}$ denotes the number of quantized ids that represents one frame, and $N=n\cdot n_{t}$ denotes the total length of the sequence. We append special tokens [bos] and [eos] to the front and end of the sequence, and no special tokens are inserted between sequences of different frames.

Then, in the second stage, we follow the architecture of LLaMA [] utilizing RMSNorm normalizing [] and Rotary Embeddings [], and train a Transformer decoder in an autoregressive way:

$$ $f_{\theta}(x^{t})=\prod_{i=1}^{N}P(t_{i}|t_{<i}),$ (3) $$

where the model predicts each token conditioned on previous ones.
Note that in the training stage, each sequence is sampled from one original video, and we do not concatenate video clips from different videos.

#### In-context Inference

The in-context inference format differs from that in the training. Following Equ (2), a number of video clips are selected as the demonstration $D^{V}$, which is appended in front of the query clip $x^{t}_{<j}=(t_{1},\cdots,t_{j-1})$ to let the model generate corresponding responses:

$$ $f_{\theta}(x^{t}_{>j})=\prod_{i=j}^{N}P(t_{i}|x^{t}_{<i},D^{V}).$ (4) $$

The generation results $x^{t}_{\geq j}=(t_{j},\cdots,t_{N})$ are vector-quantized IDs, which are then fed to the pretrained VQ decoder to reconstruct as images.
We provide an illustration of training and inference pipelines in Figure 2.

#### Zero-shot Capacity

The training and inference processes of the model differ in that the demonstration $D^{V}$ is not provided during training. Despite this, the model exhibits zero-shot Vid-ICL capabilities. This can be attributed to two key reasons. Firstly, no special separation token is inserted between frames, allowing the preceding sequences $t_{<i}$ in Equation (3) to be implicitly viewed in a demonstration-query format, i.e., $P(t_{i}|t_{<i,>j},t_{\leq j})$ where $j<i$. This implies that the model has inherently learned to handle sequences resembling the demonstration-query structure during training. Secondly, the autoregressive nature of the Transformer enables it to seamlessly extend its sequence prediction capabilities to scenarios where the demonstration and query come from different videos, thus facilitating smooth generalization to the in-context learning paradigm.
Our experiments further show that training the model with explicit in-context examples does not yield significant improvements over the zero-shot approach. Refer to Sec 5.1 for more discussions.

#### Incorporate Other Modalities

We mainly focus on videos as the demonstration in this paper, but our approach can be easily extended to acquiring other modalities such as text. To do so, we just need to transfer original text descriptions into latent representations denoted as $c$ by a pretrained language model [], then take $c$ as an additional condition while training the Transformer (Equ (3)) as well as in-context inference (Equ (4)).
We show in experiments Sec 5.4 that our model understands and reacts to both text and video demonstrations.

### 3.3 Data

While large language models are trained on vast amounts of data (usually trillions of tokens [, ]), high-quality video data is constrained. In addition, Vid-ICL prefers videos that exhibit not only rich content but also clear causal relationships and interactivity.
As a result, among various public video datasets, we focus on these accomplish embodied tasks and select two primary datasets as our main training data sources: 1) Ego4d [], an egocentric video dataset featuring abundant first-person activities; and 2) Kinetics-600 [], a comprehensive video dataset comprising diverse human activities. Additionally, we incorporate Webvid [] that contains a large amount of internet videos, to augment the variety of video content.
Notably, due to the typically ambiguous and inconsistent semantic information conveyed by internet videos,
we only utilize a small fraction of Webvid, and show that simply increasing the data scale by adding more internet videos does not help improve the in-context capacity of the model (refer to Sec 5.4).

To validate the Vid-ICL capability, we choose the Something-Something v2 (SSv2) dataset [] where each video depicts a basic action that occurs in the physical world, such as moving objects in a direction. These videos convey strong semantic information, which can be utilized to construct in-context demonstrations. We utilize the evaluation split of SSv2 as the evaluation set for all experiments.

## 4 Experimental Setups

We design tailored evaluation pipelines to assess the model capacity in Vid-ICL. We introduce the proposed evaluation metrics in Section 4.1, and summarize the details of our implementations in Section 4.2.

### 4.1 Evaluation

The evaluation of Vid-ICL should include two aspects, the first is the visual quality as a normal video generation task, and the second is semantic accuracy which is more critical as it reflects whether the model understands and follows the semantic guidance provided by demonstrations. To validate semantic accuracy, we design four kinds of demonstrations.

- •
No Demonstration. This setting is akin to normal video continuation, where the model is asked to complete the query $x^{V}$ without any in-context demonstration.
- •
In-class Demonstration. Given a video clip query, the demonstrations are sampled from videos with the same action label as the query.
- •
Contrastive Demonstration. In contrast to the previous one, the sampled demonstrations have the contrast action label to the query.
- •
Random demonstration. In this type of demonstration, video clips are chosen from arbitrary videos within the entire Ssv2 dataset.

We then evaluate the model in two settings, 1) using ground truth query labels to evaluate results, to test whether the demonstrations enhance or deviate from the generation results; 2) using demonstration labels, to assess whether the model accurately acquires the semantic guidance and generates corresponding results. We define various metrics to deal with these two settings.

#### Automatic Metrics

For visual quality, we adopt several widely utilized metrics including PSNR and LPIPS [] to perform a frame-by-frame comparison between the generation results and the ground truth video clip, i.e., the consequences of the query. Additionally, we include the FID score [] to quantify the distribution difference. These metrics are suitable for the first setting where ground truth is available.

For semantic accuracy, we propose two classification-based metrics:

- •
Video Accuracy (V-Acc). We utilize an off-the-shelf video classifier [] trained on the SSv2 dataset to calculate the classification accuracy of the generated video clips. Specifically, we predict the action label of each generated video clip and compare it with the ground truth label of the query. This metric provides a perceptual assessment of the semantic information in generation results.
- •
Probing Accuracy (P-Acc). V-Acc relies on a pretrained classifier and judges on visual signals. To directly validate the semantic information contained in latent representations, inspired by the widespread use of probing in vision feature extractors [, , ], we train a probing classifier which takes hiddens in the last Transformer layer of the generation results as input and predicts the corresponding action label.

Most automatic metrics are only suitable for the first setting where ground truth video is available. For the second setting without ground truth, only the probing accuracy can be utilized. As a supplementary, we also introduce human evaluation.

#### Human Evaluation

We manually select a subset from the SSv2 validation set, by picking out action labels with contrastive semantics, such as Pulling [something] from left to right and Pulling [something] from right to left. For a query video clip, we sample two demonstrations from the same and contrastive class respectively, constructing a pair of evaluation samples (just as shown in Figure 1). We involve 10 experienced users to score $20$ pairs of samples from several aspects, i.e., visual quality, semantic alignment and control. Alignment indicates whether each generation is semantically consistent with the demonstration, while control justifies whether the pair of results perform contrastive behavior following the demonstrations.
Participants were presented with a pair of videos at a time and asked to rate each video for each score on a scale of 1 to 5. We calculated the average score as the final result.

### 4.2 Implementation Details

#### Preprocessing

For each dataset, we assign a stride to sample frames at regular intervals in order to capture human-recognizable video clips. The stride varies among datasets depending on the average FPS but remains consistent within each dataset. Subsequently, we resize and center-crop the images to a fixed size of $256\times 256$ resolution.The pretrained VQ-GAN tokenizer [] takes in $256\times 256$ sized images and produces $16\times 16=256$ discrete tokens with a compression coefficient $f=16$. Tokens of different frames are concatenated in the order of original frames, and each training sequence contains 16 images and 4096 discrete tokens.

#### Transformer Architecture

we adopt the LlaMA [] architecture, a typical decoder-only transformer model for auto-regressive modeling with designs advantageous to large-scale modeling.
we experiment with different sets of hyper-parameters which result in models with 300M, 700M and 1.1B parameters separately. The details of hyper-parameter settings are shown in Table 9.

#### Inference

In our main experiments, we set $k=1$ to restrict the demonstration to one video. In this way, the demonstration clip contains $8$ frames, while the query and generation results both consist of $4$ frames. We also study different compositions such as $k=2$ demonstrations sampled from different videos and each with $4$ frames. Please refer to Section 5.4 for more discussions.

#### Baselines

LVM [] is a natural baseline that shows strong Img-ICL capacity. However, due to the absence of an open-source checkpoint and the high resource demands required to reproduce the model, we utilize an open reproduction, DeLVM [], instead. To evaluate the generalizability of our model, we train several variants: *Pretrain*, which is trained without SSv2 in the training set; *Pretrain w/ In-domain finetune*, which is *Pretrain* fine-tuned on SSv2 where each sample consists of continuous 16 frames, similar to the pretraining stage; and *Pretrain w/ In-context finetune*, which is fine-tuned using the same data format as in-context inference, consisting of two 8-frame videos from the same class.

## 5 Results and Analysis

**Table 1: The Semantic Accuracy metrics and Visual Quality metric of different demonstration type and training strateges. The best result in each block is bolded and second best result is underlined.**
| Demonstration | Semantic Accuracy | Visual Quality |  |  |  |
| --- | --- | --- | --- | --- | --- |
| V-Acc $\uparrow$ | P-Acc $\uparrow$ | PSNR $\uparrow$ | LPIPS $\downarrow$ | FID $\downarrow$ |  |
| Pretrain |  |  |  |  |  |
| Random | 22.7 | 28.3 | 13.01 | 0.450 | 22.53 |
| No | 22.9 | 29.6 | 13.20 | 0.442 | 20.94 |
| In-class | 24.7 | 36.7 | 13.07 | 0.447 | 21.95 |
| Pretrain w/ In-context Finetune |  |  |  |  |  |
| Random | 23.1 | 25.6 | 13.01 | 0.454 | 20.66 |
| No | 24.2 | 26.7 | 13.02 | 0.460 | 22.21 |
| In-class | 25.7 | 40.7 | 13.08 | 0.450 | 20.49 |
| Pretrain w/ In-domain Finetune |  |  |  |  |  |
| Random | 24.9 | 34.1 | 13.16 | 0.440 | 19.17 |
| No | 25.0 | 30.8 | 13.09 | 0.446 | 20.87 |
| In-class | 25.9 | 48.8 | 13.21 | 0.438 | 18.92 |

### 5.1 Main results

If not specified, we use our largest 1.1B model to generate results. We first evaluate the Vid-ICL results using the ground truth query label. In this setting, all automatic metrics can be utilized.
The quantitative results are presented in Table 1,
where each row corresponds to a different demonstration type.
We draw the following conclusions from the results.

In-class demonstrations contribute to stronger semantic accuracy Our analysis begins by focusing on the result of the *Pretrain* model in the first block.
We observe a significant enhancement in the semantic accuracy when using in-class video clips as demonstrations, compared to without demonstration or using randomly selected ones. Specifically, there is a notable 7.1% / 8.4% improvement in P-Acc and a 1.8% / 2.0% gain in V-Acc over no or prompting with random demonstrations respectively.
These findings validate our proposition that when prompted with semantically related demonstrations, the model more accurately generates video clips that adhere to the original trace.

Random demonstrations mislead the semantics Beyond the inferior performance compared to in-class demonstrations, random ones also reveal a gap of 0.2% in V-Acc and 0.7% in P-Acc to the no demonstration setting.
This indicates that prompting with unrelated demonstrations negatively impacts the semantics of the generated results.

In-domain finetuning leads to general improvement We further analyze the results presented in the second and third blocks of Table 1, focusing on different fine-tuning strategies. The superiority of in-domain finetuning over the initial *Pretrain* is evident, indicating a straightforward performance boost when in-domain training data is available. In addition, when comparing in-domain finetuning with in-context finetuning, the in-class P-Acc of the former rises to 48.8%, showcasing an 8.1% improvement over the latter.
These findings underscore the remarkable zero-shot capacity of Vid-ICL, showcasing the model’s capability at adapting to in-context formatting through exclusive training on continuous video clips, while explicit in-context finetuning fails to yield significant enhancements.

**Table 2: The Probing accuracy evaluated by demonstration labels.**
| Demonstration | P-Acc |
| --- | --- |
| Contrastive | 35.5 |
| In-class | 33.8 |
| Random | 16.7 |

Evaluation with demonstration label We evaluate the generation results with the demonstration label to assess the controliability of the model. We use results with both in-class and contrastive demonstrations as described in Section 4.1 to train the probing model. As shown in Table 2, both cases have well above average probing accuracy, indicating that the representations of generation results successfully obtain the semantic information from demonstrations.

**Table 3: Results on automatic and human evaluation of Vid-ICL and DeLVM baseline.**
|  | P-Acc | PSNR | Quality | Alignment | Control |
| --- | --- | --- | --- | --- | --- |
| DeLVM | 32.8 | 12.69 | 3.21 | 2.28 | 2.11 |
| Vid-ICL | 38.5 | 13.07 | 4.12 | 3.73 | 3.01 |

### 5.2 Comparison with DeLVM Baseline

DeLVM [] accepts a maximum of $8$ frames in the context window. To align with this, we adapt our model to receive 4 frames of demonstration, 2 frames of query, and generate 2 frames as the result. Both models contain approximately 1.1B parameters. We report both automatic and human evaluation results in Table 3.

As shown in the results, our Vid-ICL model significantly outperforms DeLVM in both objective and subjective evaluations.
We attribute this difference to the training data formats of DeLVM (i.e., Img-ICL) and Vid-ICL. Img-ICL relies on pairs of images and their annotations, which limits its ability to extract semantic information across consecutive frames and generate coherent sequences correspondingly.

### 5.3 Scaling Behavior

Figure: (a)
Refer to caption: x3.png

As describe in Section 4.2, we set up different sizes of model, respectively 300M, 700M and 1.1B. In this section, we assess the models’ scaling behavior from the following aspects.

Scalability on Visual Quality
We present the PSNR and FID scores of models in various sizes in Figure 3(a). The results indicate that larger models tend to produce samples of higher quality.

Scalability on Semantic Accuracy
We present both the absolute probing accuracy scores of in-class demonstrations and the performance gain over random demonstrations in Figure 3(b) and 3(c).
From both results, we observe a clear trend that larger models yield more informative hidden representations, and offer more precise control over the generation results when conditioned on different demonstrations.

### 5.4 Ablation and Analysis

#### Different Number of Demonstrations

We investigate the impact of different numbers of demonstrations on Vid-ICL performance. Within a 16-frame context window, we fix the query length as 4 frames and the generation as 4 frames, and construct four demonstration formats: 1) no demonstration; 2) one 4-frame demonstration; 3) two 4-frame demonstrations sampled from different videos; 4) one 8-frame demonstration.
The results in Table 5 demonstrate that providing demonstrations generally improves both P-Acc and V-acc scores. Additionally, providing more ($2\times 4$ versus $1\times 4$) or longer demonstrations ($1\times 8$ versus $1\times 4$) both result in better performance.

**Table 4: Semantic accuracy of different demonstration formats.**
| Demonstration | P-Acc | V-Acc |
| --- | --- | --- |
| No | 29.6 | 22.9 |
| 1 $\times$ 4 frames | 35.6 | 22.9 |
| 2 $\times$ 4 frames | 37.2 | 24.2 |
| 1 $\times$ 8 frames | 36.7 | 24.7 |

#### Text as Demonstrations

To demonstrate the versatility of our model, we explore incorporating text into the demonstrations. We fine-tune the 300M *Pretrain* model on the SSv2 dataset by appending corresponding text annotations to the front of each video. The text annotations are encoded into latent representations using a pretrained T5-large model [].
The results, presented in Table 5, indicate that our model can seamlessly adopt demonstrations in various modalities. Additionally, adding textual descriptions enhances the model’s in-context learning capacity.

Figure: Figure 4: The validation perplexity in various amount of web videos.
Refer to caption: x6.png

#### Effect of Web Videos

In previous studies, incorporating miscellaneous web videos into the training dataset has been shown to enhance generation quality [, ]. Here we investigate their effect on Vid-ICL. We sampled varying quantities of video clips from the Webvid dataset during pretraining and present the SSv2 validation perplexity results in Figure 4. We observe that the perplexity does not linearly decrease with the number of web videos, suggesting that Vid-ICL prefers videos conveying clear and consistent semantic information.

## 6 Conclusion

In this paper, we introduce Video In-Context Learning (Vid-ICL), a novel framework that extends in-context learning to video data. By converting video frames into sequences of discrete tokens and training an autoregressive Transformer with next token prediction, our model exhibits zero-shot in-context learning capabilities, which can generate semantically coherent and contextually appropriate video sequences based on provided demonstrations. Extensive experiments confirm that Vid-ICL effectively captures and conveys the semantic information embedded in the demonstrations, demonstrating its potential to enhance various downstream tasks such as visual planning. Additionally, the model’s versatility is verified by its ability to understand and integrate multi-modal demonstrations simultaneously.

Limitations Despite the promising results, several limitations exist in this study. Firstly, while Vid-ICL excels at generating short video sequences, its performance on longer sequences has not been thoroughly evaluated. Secondly, the model’s dependence on the quality and relevance of provided demonstrations can lead to inconsistencies, particularly when demonstrations are noisy or semantically misaligned. Future work should address these limitations by utilizing visual tokenizers with temporal compression, and scaling or developing robust models to handle imperfect demonstrations.

## Appendix A Additional Results

In this section we provide more samples on different video datasets and benchmarks that Vid-ICL model produces. Synthesis videos on Something-something v2 dataset are presented in Table 6. Table 7 includes samples on Robotics Transformer-1 dataset, showcasing the ability of Vid-ICL to fit on various form of video datasets. Furthermore, Vid-ICL as a versatile generalist, can act as a simulator in reinforcement learning tasks, for which we provide detailed explanations.

#### Vid-ICL as Simulator

We demonstrate that Vid-ICL can also function as a simulator in reinforcement learning tasks by evaluating it on the VP^2[] benchmark. VP^2 is a benchmark for video prediction models used in robotic manipulation via model-predictive control. Vid-ICL generates future frames with videos that accomplishing the same task as the demonstration, where the generated frames inversely reflect the corresponding actions that correctly interact with the environment. We compare the trajectories produced by the SVG baseline and Vid-ICL in Figure5. Evaluating on the Push-red task, we find that Vid-ICL provides more precise control over the environment interaction.

Figure: Figure 5: The trajectory produced by Svg and Vid-ICL respectively.
Refer to caption: extracted/5721673/samples/paper-7.2/robodesk.png

**Table 6: More examples of Vid-ICL generated samples on Something-something v2 dataset.**
| Demonstration | Query & Generation |
| --- | --- |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

**Table 7: More examples of Vid-ICL generated samples on RT-1 dataset.**
| Demonstration | Query & Generation |
| --- | --- |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

**Table 8: The configuration of different size of models.**
|  | Hidden dim | MLP dim | Num. Heads | Num. Layers |
| --- | --- | --- | --- | --- |
| 300M | 1024 | 2688 | 8 | 22 |
| 700M | 1536 | 4096 | 16 | 24 |
| 1.1B | 2048 | 4096 | 16 | 26 |

**Table 9: Model hyperparameters.**
| Hyperparameter | Value |
| --- | --- |
| Learning rate scheduler | inverse sqrt |
| Learning rate | $5e^{-4}$ |
| Warm up steps | 10000 |
| Weight decay | 0.01 |
| Optimizer | AdamW |
| AdamW betas | $(0.9,0.95)$ |
| Context length | 4096 |

**Table 10: Statistics of training datasets.**
| Dataset | Sample Stride | Num. Videos | Total tokens(M) | Training tokens(M) |
| --- | --- | --- | --- | --- |
| Ego4d | 6 | 14192 | 6890.8 | 2048.0 |
| Kinetics-600 | 5 | 426253 | 4711.35 | 1024.0 |
| WebVid | 6 | 2494801 | 50825.2 | 1024.0 |
| Total | - | - | - | 4096.0 |

## Appendix B Implementation Details

### B.1 Model architectures

We utilize LLaMA as the Transformer architecture in our work. To validate the scaling behavior, we train three different sizes of the Vid-ICL model within the LLaMA architecture: 300M, 700M, and 1.1B. We carefully tune the hidden dimension, MLP intermediate dimension, and the number of Transformer decoder layers to achieve different model sizes. The configuration details of these models are presented in Table 8.

### B.2 Hyperparameters

The hyperparameters used to train the Vid-ICL model are presented in Table 9. We utilize inverse square root scheduler and start model training with 10000 warmup steps.

### B.3 Dataset statistics

The summary of dataset used in our paper is presented in Table 10. We provide detailed statistics including number of videos, total of tokens after tokenization and tokens used to train our Vid-ICL model.

## Appendix C Broader Impact

Positive societal impacts of Vid-ICL include its potential to enhance various downstream applications such as embodied planning and robotic simulation by enabling models to imitate actions demonstrated in videos. As for negative societal impacts, the widespread adoption of this technique may raise concerns about privacy when powerful models gain the ability to analyze and generate video content at scale.

## Appendix D Ethics Statement

Vid-ICL is exclusively a research initiative with no current plans for product integration or public access. We are committed to adhering to Microsoft AI principles during the ongoing development of our models. The datasets utilized in this study are publicly available and have been thoroughly reviewed to ensure they do not include personally identifiable information or offensive content. Nonetheless, as these datasets are sourced from the Internet, there may still be inherent biases. To address this, we have implemented a rigorous filtering process on the training data to minimize the potential for the model to generate inappropriate content.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: Originality:
The idea of performing in-context autoregressive video generation is interesting and I have not seen it being done before. Autoregressive video generation without text is very difficult, and this approach seems to address some of the difficulties.

Quality / Clarity:
The paper is clear and easy to read. The proposed idea is simple and I believe easy to replicate from the descriptions.

Significance:
I believe that even though this paper is not producing mind blowing results such as Sora, Kling, Veo and others. The ideas proposed here can be useful for the scientific community to continue exploring the problem of video generation
summary: This paper proposes a video generation based on autoregressive transformers trained with the objective of imitating a set of seed videos. In experiments, the authors show that the model is able to imitate tasks presented to it in an in-context manner. In addition, the learned representations coming from the model are also useful for classification tasks.
weaknesses: Weaknesses / Questions:

Missing baselines:
* Not sure if I missed this from reading the Baselines paragraph in the manuscript, but did the authors include a simple next frame prediction baseline?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - It is confusing to define a contrast action. What is the discipline? It might be better to provide a detailed list of all the situations that are considered.
- Is it possible to provide the V-acc and LPIPS scores in Table 2? In my view, V-acc and LPIPS are more faithful and direct measurements of the visual content of the generated videos.
soundness: 2
strengths: - The topic of video imitation is valuable and would attract sufficient attention in the community, especially for embodied AI, since the approach could be applied to generate a large number of unseen training data, which is costly to obtain manually in the real world.
- It is interesting that the autoregressive model trained without using demonstration videos can generalise to the imitation inference stage when the demonstration is provided.
- The proposed framework is flexible and can be extended to imitate videos conditioned on text, which might be easier to acquire than demonstration videos and practically applicable.
summary: The authors proposed to train an autoregressive model for video imitation. The model training is performed in a self-supervised manner using a continuous video. Then, in the inference stage, given a demonstration and query video, the model could generate subsequent video frames of the query video that imitate the same semantics or actions conveyed by the demonstration video. The model shows zero-shot capacity when generating the imitation video frames.
weaknesses: - On the technical side, the novelty of the proposed method is limited. The combination of VQ-VAE and Transformer is commonly adopted for image generation.
- There is a lack of detailed explanation about why self-supervised training without acquiring demonstrations during training will generalise to video imitations given demonstrations. 
- The visual quality of the generated imitation video is limited as shown in Table 1. The V-Acc and P-Acc scores are quite low, suggesting that the generated video's content might differ from the expected one. So, the practical usage of the proposed method is questionable.
Failure cases would help readers understand the method's limitations. I suggest the authors provide a detailed analysis accordingly.
- The evaluation of using text as conditions is very limited.
- The quality of writing could be further improved by providing more details regarding task definition and technical details. For example, it would be better to list all the types of demonstration videos. Currently, it is unclear if any more tasks are involved other than object manipulation and camera movement (and their detailed types). The training objectives could be clarified (currently missing) for those unfamiliar with VQ-VAE/VQ-GAN for better reproducibility.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: How well does the model generalize to longer video sequences compared to the current evaluations on short clips?
soundness: 3
strengths: 1. Zero-shot Capability: VidIT achieves a zero-shot imitation capacity, enabling it to generalize from video demonstrations without further training.

2. Versatile Applications: The model can be applied to multiple downstream tasks, including robotic simulation, video planning, and segmentation.

3. Comprehensive Evaluation Benchmark: Both objective metrics and subjective human evaluations validate the quality and semantic coherence of generated videos.

4. Scalable Model: The model demonstrates improved performance when scaled up, providing strong semantic control in video generation.
summary: The paper introduces Video Imitator (VidIT), an autoregressive transformer model trained for zero-shot video imitation, enabling it to imitate the semantics of a given demonstration video and generate coherent video clips without requiring further fine-tuning. VidIT uses a next-token prediction objective to learn from video datasets, allowing it to generate video sequences that are semantically aligned with demonstrations. The model is evaluated on visual quality and semantic accuracy using both automatic metrics and human assessment.
weaknesses: 1. Limited Sequence Length: Because now the mainstream video generation solutions have reached more than 5 seconds, the evaluation is primarily on short sequences; the performance on longer sequences remains untested.

2. Gap Between the Training Web Videos and Evaluation Metrics: In the benchmark constructed in this article, the domains of the evaluation metric are mostly in indoor scenes (SSv2), which is relatively limited. This is likely why adding more web videos to the training data did not make the model perform better. To get more accurate conclusions, evaluation metrics should be developed for more open scenarios and diverse tasks mentioned in the appendix, such as task imitator.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. However, the main contributions of this paper are not entirely clear to me. I believe there might be two key contributions:
a) The development of a video imitator model capable of mimicking actions from a reference video.
b) The demonstration that transformers possess zero-shot capabilities without needing specific training for tasks involving action imitation.

If the primary contribution is the first point, then the task of "imitating actions from a reference video" has been addressed in several previous works [1, 2]. Moreover, achieving the results shown in this paper could also be done using some existing open-source video editing models. Additionally, similar effects can be achieved by controlling camera angles [3].

If the primary contribution is the second point, then readers of this paper would be highly interested in understanding why transformers possess this zero-shot ability. However, the paper only demonstrates through experiments that task-specific training and the zero-shot approach yield similar performance. In my opinion, this is insufficient to explain the transformer's zero-shot capabilities. I recommend providing evidence through analyses of feature maps or attention scores to support this claim.

2. In Section 4.1, the paper introduces four settings but ultimately selects two. It is recommended that the names of these settings remain consistent throughout the paper to avoid confusing the readers. Also, ensure that these names are consistent with the names used in the experiments in Section 5.

3. Additionally, I suggest including more details about the training process, such as the hardware used and the approximate training time.
soundness: 3
strengths: 1. This paper trained a transformer model called VidIT, which can learn motion information from a reference video and apply this motion to the prediction of a target video.

2. The method is effective, as extensive experiments demonstrate significant improvements in metrics, and the visual results further prove that the method effectively learns motion information.

3. This approach is based on an interesting discovery, which the authors define as "zero-shot video imitators." This finding could inspire future work in the transformer series.

4. The method can be applied to various tasks, such as unconditional video prediction, proving its strong generalization ability.
summary: This paper trains autoregressive Transformers on video datasets in a self-supervised manner, enabling the model to infer and imitate semantics from a demonstration video in a zero-shot way, allowing it to perform unseen tasks without fine-tuning.
weaknesses: See questions

Meta-review: The paper demonstrates the zero-shot video imitation capabilities by training self-supervised autoregressive Transformers on video data. Such in-context capabilities enable solving unseen tasks by watching video demonstrations.

The rebuttal is well-accepted by most reviewers. After rebuttal, all reviewers are positive about this paper and two are enthusiastic. 
As raised by reviewers, a deeper investigation and analysis on varied models/architectures and real-world usage will strengthen this paper further.

The area chair recommends accepting this paper. The zero-shot video imitation capabilities provide insights into research on exploring GPT-3 like in-context learning capabilities in video generation and embodied tasks.

Decision: Accept (Poster)