Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators
Authors: Wentao Zhang
Junliang Guo
Tianyu He
Li Zhao
Linli Xu
Jiang Bian
Venue: ICLR 2025 Poster (2025)
Keywords: Video Generation
Transformer
Zero-Shot
Primary area: generative models

Abstract: People interact with the real-world largely dependent on visual signal, which are ubiquitous and illustrate detailed demonstrations. In this paper, we explore utilizing visual signals as a new interface for models to interact with the environment. Specifically, we choose videos as a representative visual signal. And by training autoregressive Transformers on video datasets in a self-supervised objective, we find that the model emerges a zero-shot capability to infer the semantics from a demonstration video, and imitate the semantics to an unseen scenario. This allows the models to perform unseen tasks by watching the demonstration video in an in-context manner, without further fine-tuning. To validate the imitation capacity, we design various evaluation metrics including both objective and subjective measures. The results show that our models can generate high-quality video clips that accurately align with the semantic guidance provided by the demonstration videos, and we also show that the imitation capacity follows the scaling law.

Published as a conference paper at ICLR 2025
VIDEO IN-CONTEXT LEARNING: AUTOREGRESSIVE TRANSFORMERS ARE ZERO-SHOT VIDEO IMITATORS
Wentao Zhangl.2tt. Junliang Guo Tianyu He Li Zhao?, Linli Xu Jiang Bian
1.2.1.
1 Schooo of Computer Science and Technology, University of Science and Technology of China State Key Laboratory of Cognitive Intelligence
Microsoft Research Asia
ventaozêmall uste. edu. en, linllinllll sinlixuBuste edu. cn. (junliangguo, tianyuhe, lizo, jiang bi bian) 8mi crosoft.com https:/ / /aka. ma /viddiiic
ABSTRACT
People interact with the real-world largely dependent on visual signal, which are ubiquitous and illustrate detailed demonstrations. In this paper, we explore utiliz- ing visual signals as new interface for models to interact with the environment. Specifically, we choose videos as a representative visual signal. And by training autoregressive Transformers on video datasets in self-supervised objective, we find that the model emerges a zero-shot capability to infer the semantics from a demonstration video, and imitate the semantics to unseen scenario. This allows the models to perform unseen tasks by watching the demonstration video in an in-context manner, without further fine-tuning. To validate the imitation capac- ity, we design various evaluation metrics including both objective and subjective measures. The results show that our models can generate high-quality video clips that accurately align with the semantic guidance provided by the demonstration videos, and we also show that the imitation capacity follows the scaling law. Code and models have been -our-eodreed.
INTRODUCTION
Humans acquire skills through imitation, fundamental aspect of learning demonstrated even in early childhood. For instance, when children observe how to hold spoon, they not only learn this specific action but can also generalize it to grasp different spoons in varied contexts. These demon- strations are primarily conveyed through visual signals, which are abundant our environment and provide intricate guidance. Similarly, for the development of general Al agents, the ability to learn from visual demonstrations and generalize to new situations is crucial and desirable. This capability to understand and imitate visual signals is not only appealing but also essential for advancing Al towards more human Moken-like learning and interaction.
Previous works (Bar et al., 2022; Wang et al., 2023a;b; Bai et al., 2023) have studied image in-context learning in image-based visual tasks. By formulating image perception tasks such as segmentation and detection supervised image pair demonstrations in delicately designed structures, e.g., a joint grid image (Bar et al., 2022) for MAE models (He et al., 2022) or a sequence (Bai et al., 2023) for Transformer models (Vaswani et al., 2017), they encourage the model to mimic the task and generate the prediction of given image query. However, these works require explicitly formulating the training samples in source-target pairs, which restricts in-context inference a supervised capacity. Furthermore, image-level generation struggles to produce coherent and contextually appropriate sequences.
In this paper, we introduce the Video ImiTator (VidIT) model, an autoregressive Trans- former (Vaswani et al., 2017; Touvron et al., 2023) trained with the next token prediction objective
*Wok done during internship at Microsoft Research Asia in March 2024.
'quual Contribution.
1 "Corrsponning author.

---

Published as a conference paper at ICLR 2025
Demonstration: Turning the carnera from right to left
Query
Generation
Turning the camera from left to right
Place Object into Receptacle
Place Object from Receptacte
Illustrations of VidIT. Given a query video clip, the model generates different results based on the provided demonstration video clips. The generated results are coherent with the query in content and semantically consistent with the demonstrations. Text descriptions are annotated for clarity and not used as model input.
Figure I:
on video datasets. VidIT uses videos as the primary unit for both inputs and outputs, with each train- ing sample consisting of sequence of frames from a single video, encoded as discrete tokens using vector-quantized encoder (Van Den Oord et al., 2017; Patil al., 2024). The model learns to predict future frames based on preceding ones through self-supervised learning. Though demonstration- generation pairs are not provided during training, the autoregressive paradigm enables the model to discern the underlying structures and patterns within the videos, leading to the emergence of zero- shot imitation capability. Consequently, when the model encounters demonstration videos during inference, it can generalize and apply these learned patterns to generate coherent and contextually appropriate video sequences. This zero-shot capability echoes findings observed in large language models (Brown al., 2020).
Specifically, demonstration videos are highly versatile and capable of conveying a wide range of information, such as examples for tasks including object manipulation, or movements of the camera in an ego-centric video. Guided by these demonstrations, the model receives a query video clip set new scene and generates subsequent video sequence that mimics the semantic actions from the demonstrations (see Figure 1 for examples). This allows VidIT to address multiple downstream tasks, such as embodied planning and simulating, by letting query robot imitate the actions demon- strated by humans or other agents. Given that videos excel in describing low-level details (where language may fall short) and temporal dynamics (where images are insufficient), VidIT acts as crucial interface for models to interact with the real world.
To comprehensively and accurately evaluate the model performance VidIT, we develop both ob- jective and subjective metrics to assess the generated videos in terms of visual quality, semantic accuracy, and consistency with the prompted demonstrations. Our extensive experiments demon- strate that the model not only produces high-quality video clips but also successfully adheres to the semantic guidance provided by the demonstration examples. In addition, we show that the zero-shot imitation capacity also follows the scaling law (Kaplan et al., 2020) of large models, illustrating the potential of future works. The main contributions of this work are summarized as follows:
We propose and study the task of video imitation, which enables the model to interact with real- world demonstrations through video generation.
2

---

Published as a conference paper at ICLR 2025
We train large Transformer model VidIT that exhibits powerful video imitation learning capacity, which also follows the scaling law of large models.
We propose various evaluation metries evaluate the visual quality and semantic accuracy of generated videos, providing a solid benchmark for the evaluation of video imitation learning.
RELATED WORK
Imitation Learning and In-context Learning Imitation Learning is crucial paradigm in rein- forcement learning, wherein the expected behavior is acquired by mimicking demonstration's ac- sate-action pairs from an expert, the model is expected to map the current state to the corresponding actions. Such learning paradigm is close to human behavior and is appealing in interactive circumstances.
tions (Zare et al., 2024). By learning
This ability to imitate has also been identified as a key feature in large language models (Radford al., 2019; Brown et al., 2020; Touvron al., 2023; Chowdhery al., 2023), which is referred to as in-context learning in LLM area. By conditioning on sequence of text templates, an LLM generates coherent content based on these templates. This paradigm obviates the need for parameter updates and thereby benefiting the downstream usage of LLMs (Brown et al., 2020; Touvron et al., 2023).
Different prompting designs (Wei et al., 2022; Guo et al., 2023; Li et al., 2023) empower LLMs to deal with a wide range of natural language understanding and generation tasks.
As for the in-context paradigm in vision area, current research mainly focus on teaching the model mimic the vision task provided by demonstrations. Pioneering works construct vision imitation as an image inpainting (Bar et al., 2022; Wang et al., 2023a;b) task. Given multiple query-answer image pairs arranged in a grid image, models are optimized to reconstruct the masked answers under the MAE projective (He et al., 2022). Recently, LVM (Bai al., 2023) flatten image pairs into a sequence and train an auto-regressive Transformer with next token prediction. These models show powerful capacity to imitating vision tasks such as semantics segmentation and detection from the demonstrations, but rely on training on supervised datasets. basic elements (i.e., demonstrations, queries and predictions) from images videos, unfortunately, existing models struggle to handle the increased complexity as they are not designed to capture the spatial-temporal relationships of inputs. Specifically, though LVM (Bai et al., 2023) able to generate consecutive frames of query video clip, the video imitation capacity (e.g., generating different consequences with different demonstrations) not demonstrated.
addition, when extending the
Video Generation The studied video imitation learning can be considered conditional video generation problem, which has recently become a prominent focus in research. Text is the most commonly utilized condition, and various text-to-video models have shown promising results in generating high-fidelity videos (Ho et al., 2022b;a; Brooks et al., 2024; Hong et al., 2023; Villegas al., 2023; Kondratyuk al., 2023; Yu et al., 2023b). Recently, video generation models spread to other related domains such as embodied AI (Yang et al., 2024), where they are utilized in visual planning (Du et al., 2023) and simulation (Yang et al., 2023) with actions and observations as conditions, illustrating the potential of this area. As for model architectures, Diffusion models (Ho al.. 2020: Nichol Dhariwal, 2021) and Transformers (Vaswani et al., 2017) are both widely adopted in different scenarios, and we choose the autoregressive Transformer as the backbone model in this paper, as its abilities model long-range dependencies and contextual relationships are crucial for video imitators.
3 VIDEO IMITATOR
We introduce the proposed Video Imitator in this section. We begin with the problem definition of both image and video imitation in Section 3.1, highlighting the differences between them. We then elaborate on the training and inference pipeline VidIT in Section 3.2. Dataset selection is discussed in Section 3.3.
3

---

Published as a conference paper at ICLR 2025
Training
Imitation Inference
Cross Entropy Loss
Visual Decoder
EOS
2+z
Transformer
Transformer
BoS
Visual Tokenizer
Visual Tokenizer
Training: continuous video clips
Prompt: Demonstration Query
Figure 2: The framework of VidIT. Left: Training the VidIT. The data used for training are contin- uous video clips and the Transformer is trained by next token prediction objective. Right: Video Imitation Inference. The model conditioned demonstration videos and generates the subse- quent frames of given query video.
3.1 PROBLEM DEFINITION
Typically, the image task imitator models follow the setting akin to question answering: the demon- stration is composed of & image pairs D = {s H where sf E ((3....rr) denotes the input image with resolution n, and sf denotes its target (usually an image with task-specific annota- tions (Bai et al., 2023)). Given a query image x² the model predicts its target y conditioned on the demonstration:
fo(x') P(y |x",, ,D*),
(1)
where fo(-) represents the utilized vision model such an MAE-pretrained VIT (Wang et al., 2023a; Bar et al., 2022) or an auto-regressive Transformer (Bai et al., 2023), which is expected to discern the inherent task from image pairs (s/, .f) and process x accordingly.
Unlike image task imitators, VidIT takes videos as basic element, focusing on inheriting semantics from demonstrations rather than discerning specific tasks. Specifically, the demonstration DV {(s),... th Tizl comprises & video clips, and each clip consists of n, frames. Given a query video clip x = (s) ... ")"),, the objective of VidIT can be formulated as:
fo(x)) Ply ,VV),
where y = (s)
Sy denotes the generated video clip, which should be perceptually coherent with the query x while semantically consistent with the demonstration DV at the same time.
3.2 APPROACH
Training a Transformer for vision tasks typically consists of two stages, 1) training a visual tokenizer such VQ-VAE (Van Den Oord et al., 2017; Esser et al., 2021) convert each image to discrete tokens; 2) each training sample is constructed decoder. For the first stage, we utilize a public pretrained checkpoint (Patil et al., 2024) with 16x spatial compression rate as the VQ tokenizer. Our framework also fits for recent tokenizers with both spatial and temporal compressions (Yu et al., 2023a;b), and we leave it for future work. Formally, for training video clip x sequence z' (t,, ... .t t.. ,ty) where n, denotes the number quantized ids that represents frame, and N n- n4 denotes the total length of the sequence. We append special
a sequence of tokens to train the Transformer
(s'..... , with frames, the VQ tokenizer converts it flat
4

---

Published as a conference paper at ICLR 2025
tokens [bos] and [eos] the front and end of the sequence, and special tokens inserted between sequences of different frames.
Then, in the second stage, follow the architecture of LLaMA (Touvron et al., 2023) utilizing RMSNorm normalizing (Zhang & Sennrich, 2019) and Rotary Embeddings (Su et al., 2024), and train Transformer decoder in an autoregressive way:
N Io(x') II P(U))))) ==1
where the model predicts each token conditioned on previous ones. Note that in the training stage, each sequence is sampled from one original video, and we do not concatenate video clips from different videos.
Imitation Inference The imitation inference format differs from that in the training. Following Equation (2), a number of video clips are selected as the demonstration DV, which is appended in front of the query clip x²², = (t1,. .. .t--) to let the model generate corresponding responses:
fo(2))) II P(tl'2
The generation results z52 (ty,... (N) vector-quantized IDs, which then fed to the pretrained VQ decoder to reconstruct as images. We provide an illustration of training and inference pipelines in Figure
Zero-shot Capacity The training and inference processes of the model differ in that the demon- stration DV is not provided during training. Despite this, the model exhibits zero-shot video imi- tation capabilities. This can be attributed to two key reasons. Firstly, is inserted between frames, allowing the preceding sequences tee in Equation (3) to be implicitly viewed in demonstration-query format, i.e., P(l]l<ci.s (-g) where j < i. This implies that structure
special separation token
the model has inherently learned to handle sequences resembling the demonstration- during training. Secondly, the autoregressive nature the Transformer enables seamlessly ex- tend its sequence prediction capabilities to scenarios where the demonstration and query come from different videos, thus facilitating smooth generalization to the imitation paradigm. Our experiments further show that training the model with explicit imitation examples does not yield significant im- provements over the zero-shot approach. Refer to Sec 5.1 for more discussions.
Incorporate Other Modalities We mainly focus on videos as the demonstration in this paper, but our approach can be casily extended to acquiring other modalities such as text. To do so, we just need transfer original text descriptions into latent representations denoted as by pretrained language model (Raffel et al., 2020), then take an additional condition while training the Transformer (Equation (3)) as well as imitation inference (Equation (4)). We show in experiments Sec 5.5 that our model understands and reacts to both text and video demonstrations.
3.3 DATA
While large language models are trained on vast amounts of data (usually trillions of tokens (Hoff- mann al., 2022: Touvron et al., 2023)), high-quality video data constrained. In addition, VidIT prefers videos that exhibit not only rich content but also clear causal relationships and interactivity. As a result, among various public video datasets, we focus on these accomplish embodied tasks and select two primary datasets an egocentric video dataset featuring abundant first-person activitics; and 2) Kinetics-600 (Carreira al., 2018), comprehensive video dataset comprising diverse human activities.
main training data sources: 1) Ego4d (Grauman et al., 2022),
Additionally, we incorporate self-collected videos that contains a large amount of general real-world videos, to augment the variety of video content.
To validate the VidIT's imitation capability, we choose the main evaluation dataset (Goyal et al., 2017) where each video depicts basic action that occurs in the
v2 (SSv2) as the
5

---

Published as a conference paper at ICLR 2025
physical world, such moving objects in direction. These videos convey strong semantic informa- tion, which can be utilized to construct the demonstrations. We utilize the evaluation split of SSv2 the evaluation set for all experiments. In addition, we include the Robotics Transformer-1 (RT- 1) (Brohan et al., 2022) dataset and MineRI. to demonstrates VidIT's effectiveness on embodied Al and interactive tasks.
EXPERIMENTAL SETUPS
We design tailored evaluation pipelines to assess the VidIT's capacity to perform video imitation. We introduce the proposed evaluation metries in Section 4.1, and summarize the details of our implementations in Section 4.2.
4.1 EVALUATION
The evaluation of VidIT should include two aspects, the first is the visual quality as normal video generation task, and the second is semantic accuracy which is more critical it reflects whether the model understands and follows the semantic guidance provided by demonstrations. To validate semantic accuracy, we design four kinds of demonstrations.
No Demonstration. This setting akin to unconditional video prediction, where the model is asked complete the query x without any demonstration.
Random Demonstration. In this type of demonstration, video clips are chosen from arbi- trary videos within the entire SSv2 dataset.
In-class Demonstration. Given video clip query, the demonstrations sampled from videos with the same action label as the query.
Contrastive Demonstration. In contrast to the previous one, the sampled demonstrations have the contrast action label the query.
then evaluate the model in two settings, 1) using ground truth query labels to evaluate results, to test whether the demonstrations enhance deviate from the generation results; 2) using demon- stration labels, to assess whether the model accurately acquires the semantic guidance and generates corresponding results. We define various metrics deal with these two settings.
Automatic Metrics For visual quality, adopt several widely utilized metrics including PSNR and LPIPS (Zhang et al., 2018) to perform a frame-by-fire by-mmamee comparison between the generation results and the ground truth video clip, i.e., the consequences of the query. Additionally, we include the FID (Heusel et al., 2017) and FVD (Unterthiner et al., 2018 88 score to quantify the distribution difference. These metrics are suitable for the first setting where ground truth is available.
For semantic accuracy, we propose two classification based metrics:
Video Accuracy (V-Acc). We utilize an off-the-shelf video classifier (Tong et al., 2022) trained the SSv2 dataset to calculate the classification accuracy of the generated video clips. Specifically, we predict the action label each generated video clip and com- pare it with the ground truth label of the query. This metric provides a perceptual sessment of the semantic information in generation results. In particular, we use the sideomae-base- f finetuned -eeeeed-iiiiiiiie ics? checkpoint.
Probing Accuracy (P-Acc) While V-Ace relies on a pretrained classifier and judges on vi- sual signals, we propose another metric named P-Ace to operate on the latent representation of VidIT. To directly validate the semantic information contained in latent representations, inspired by the widespread use of probing in vision feature extractors (He al., 2022; Huang et al., 2023; Bardes et al., 2023), we train probing classifier which takes hiddens in the last Transformer layer of the generation results as input and predicts the correspond- ing action label.
thub, wwt./oom//////ee/eee/eeeeeee/eee nerl labs/ nerl httt:s:://gtthbbb.com/mmeeeee Phttps://huggiii wtt./////oi.iiii////ooeeeeeeeee base- nenuundd- kf neticca

---

Published as a conference paper at ICLR 2025
Most automatic metrics are only suitable for the first setting where ground truth video is available. For the second setting without ground truth, only the probing accuracy can be utilized. As supple- mentary, we also introduce human evaluation.
Human Evaluation We manually select a subset from the SSv2 validation set, by picking out action labels with contrastive semantics, such as Pulling [something] from left to right and Pulling [something] from right to left. For a query video clip, we sample two demonstrations from the same and contrastive class respectively, constructing pair of evaluation samples (just as shown in Figure 1). We involve 10 experienced users to score 20 pairs of samples from several aspects, ic., visual quality, semantic alignment and control. Alignment indicates whether each generation semantically consistent with the demonstration, while control justifies whether the pair of results perform contrastive behavior following the demonstrations. Participants were presented with a pair of videos time and asked to rate each video for each score scale of to 5. We calculated the average score as the final result.
4.2 IMPLEMENTATION DETAILS
Preprocessing For each dataset, we assign a stride to sample frames at regular intervals in order to capture human-recognizable video clips. The stride varies among datasets depending the average FPS but remains consistent within each dataset. Subsequently, we resize and center-crop the images fixed size of 256 256 resolution. The pretrained VQ-GÁN tokenizer (Patil et al., 2024) takes in 256 x 256 sized images and produces 16 x 16 256 discrete tokens with compression coefficient 16. Tokens of different frames are concatenated in the order of original frames, and each training sequence contains 16 images and 4096 discrete tokens.
Transformer Architecture we adopt the LLaMA (Touvron et al., 2023) architecture, a typical decoder-only transformer model for auto-regressive modeling with designs advantageous to large- scale modeling. we experiment with different sets of hyper-paramettrss which result models with 300M, 700M and 1.1B parameters separately. The details of hyper-parameter settings are shown in Table 8, We introduce more training details in Appendix B.3
Inference In our main experiments, we set k = to restrict the demonstration to one video. this way, the demonstration clip contains 8 frames, while the query and generation results both consist of 4 frames. We also study different compositions such as = demonstrations sampled from different videos and each with 4 frames. Please refer Section 5.5 for more discussions.
Baselines We compare the proposed VidIT model with recent visual generative models based on auto-regressive Transformer. LVM (Bai et al., 2023) is a natural baseline that shows strong image imitation capacity. DeLVM (Guo et al., 2024) is data efficient version of LVM, which involves extra knowledge distilling step and have fewer parameters. LWM (Liu et al., 2024) is designed to understand and generate long-context video contents, which is trained aligned text-video pairs. our model, we train several variants: Pretrain, which is trained without SSv2 in the training set; Pretrain w/ In-domain finetune, which is fine-tuned on SSv2 from the Pretrain model where each sample consists of continuous 16 frames, similar to the pretraining stage; and Pretrain w/ Imitation finetune, which is fine-tuned using the same data format imitation inference, consisting of two 8-frame videos from the same class.
To evaluate the generalizability
5 RESULTS AND ANALYSIS
5.1
MAIN RESULTS
If not specified, we use our largest 1.IB model to generate results. We first evaluate the VidIT results using the ground truth query label. In this setting, all automatic metrics can be utilized. The quantitative results are presented in Table 1, where each row corresponds to a different demonstration type. We draw the following conclusions from the results.
In-class Demonstrations Contribute to Stronger Semantic Accuracy Our analysis begins by focusing the result of the Pretrain model in the first block. We observe significant enhancement

---

Published as a conference paper at ICLR 2025
Table 1: The Semantic Accuracy metrics and Visual Quality metric of different demonstration type and training strateges. The best result in each block is bolded and second best result is underlined.
Semantic Accuracy P-Ace
Visual Quality L.IPS
Demonstration
V-Ace
PSNR
FID +
FVD
Pretrain
No Demonstration
22.9
29.6
13.20
0.442
20.94
119.51
Random Demonstration
22.7
28.3
13.01
0.450
22.53
131.34
In-class Demonstration
24.7
36.7
13.07
0.447
21.95
125.77
Pretrain w/ Imitation Finetune
No Demonstration
24.2
26.7
13.02
0.460
22.21
129.68
Random Demonstration
23.1
25.6
13.01
0.454
20.66
115.76
In-class Demonstration
25.7
40.7
13.08
0.450
20.49
113.52
Pretrain w/ In-domain Finetune
No Demonstration
25.0
30.8
13.09
0.446
20.87
108.12
Random Demonstration
24.9
34.1
13.16
0.440
19.17
95.72
In-class Demonstration
25.9
13.21
0.438
18.92
88.63
Table 2: Results on automatic and human evaluation of VidIT and baseline models.
Automatic Metrics
Human Evaluation
Baselines
P-Acc
PSNR
Quality
Alignment
Control
LVM (Bai et al., 2023)
27.1
12.45
3.50
2.71
1.81
DeLVM (Guo et al., 2024)
32.8
12.69
3.21
2.28
2.11
LWM (Liu et al., 2024)
24.9
11.89
3.39
2.33
2.05
VidIT (ours)
38.5
13.07
4.12
3.73
3.01
in the semantic accuracy when using in-class video clips as demonstrations, compared without demonstration or using randomly selected ones. Specifically, there is notable 7.1% 8.44% improve- ment in P-Ace and 1.8% 2.0% gain in V-Ace over no or prompting with random demonstrations respectively. These findings validate our proposition that when prompted with semantically related demonstrations, the model more accurately generates video clips that adhere the original trace, in zero-shot way on both the domain and the imitation ability.
Random Demonstrations Mislead the Semantics Beyond the inferior performance compared to in-class demonstrations, random ones also reveal gap of 0.2% in V-Ace and 0.7% in P-Acc to the no demonstration setting. This indicates that prompting with unrelated demonstrations negatively impacts the semanties of the generated results.
In-domain Finetuning Leads to General Improvement We further analyze the results presented in the second and third blocks of Table 1, focusing different fine-tuning strategies. The superiority of in-domain finetuning over the initial Pretrain is evident, indicating a straightforward performance addition, when comparing in-domain finetuning
boost when in-domain training data is available. with imitation finetuning, the in-class P-Ace of the former rises to 48.8%, showcasing an 8, improvement over the latter. These findings underscore the remarkable zero-shot capacity of VidIT, showcasing the model's 's capability at adapting to the imitation formatting through exclusive training on continuous video clips, while explicit finetuning with the target format fails yield significant enhancements.
Evaluation with Demonstration Label generation results with the demonstration label to assess the controllability of the model. We use results with both in-class and contrastive demonstrations as described in Section 4.1 to train the probing model. As shown in Table 3, both cases have well above average probing accuracy, indicating that the repre- sentations of generation results successfully obtain the seman- tic information from demonstrations.
We evaluate the
Table 3:
The Probing accuracy evaluated by demonstration labels.
Demonstration
P-Acc
Contrastive
35.5
In-class
33.8
Random
16.7
8

---

Published as a conference paper at ICLR 2025
and PSNR Metric
Probing Accuracy
Probe Accuracy Gain
40
10
26
26.33
12,96
13.0
Probe acc
35
31.6
36.7
Probe-acc gain
7:45
24
PSNR
30
6-
12.8
25
4-
22
12.67
21.95
19.5
2.5
3.5
21.29
12.6
20
0.75
1.00
15
0.25
0.75
1.00
0.25
1.00
Model Parameters (8)
Model Parameters (B)
Model Parameters (B)
(a)
(b)
(c)
Figure 3: The performance when scaling model parameters.
5.2 COMPARISON WITH BASELINE MODELS
We compare our VidIT with baseline methods mentioned in Section 4.2 on the video imitation abilities.
We report both automatic metrics and human evaluation results in Table 2. The full evaluation of baseline models is reported in Table 6 in Appendix A.6. As shown the results, our VidIT model significantly outperforms the bascline models in both objective and subjective evaluations. We at- tribute this difference to the training data formats of VidIT and other models. LVM and DeLVM mainly relies pairs of images and their annotations, which limits their ability extract semantic information across consecutive frames and generate coherent sequences correspondingly. Moreover, the training process of LWM not designed capture the demonstration's 'ss semantics, leading degradation on imitation performance.
5.3 SCALING BEHAVIOR
As describe in Section 4.2, we set up different sizes of model, respectively 300M, 700M and 1.1B. In this section, we assess the models' scaling behavior from the following aspects.
Scalability on Visual Quality We present the PSNR and FID scores models in various sizes in Figure 3a. The results indicate that larger models tend to produce samples of higher quality.
Scalability on Semantic Accuracy We present both the absolute probing accuracy scores of in- class demonstrations and the performance gain over random demonstrations in Figure 3b and 3c. From both results, we observe a clear trend that larger models yield more informative hidden repre- sentations, and offer more precise control over the generation results when conditioned on different demonstrations.
5.4 GENERALIZATION ABILITY
In this section, we elaborate the generalization ability of the VidIT model. Thanks its self- supervised training paradigm, VidIT can be smoothly adapted to multiple scenarios and tasks with minimal modifications.
Firstly, we apply VidIT to Robotics Transformer dataset (Brohan et al., 2022) to imitate actions of a robotic arm, such as opening and closing drawers and placing objects. The samples on Figure demonstrates that VidIT can successfully mimic various robotic arm actions from demonstrations, completing the frame sequences with the same movements.
Additionally, we highlight that VidIT can serve an interactive agent. this setup, VidIT receives previous observations and generates subsequent frames, which are then converted to action signals via the inverse dynamics mechanism. This pro- cess ensures that the action signals reflect VidIT's imitation intent. As depicted in Figure in Appendix A. agents perform specific actions guided by VidIT, showcasing the zero-shot imitation ability of VidIT and revealing its potential to interact with the environment.
underlying video generation engine within
We also validate VidIT's capability video task imitator. Following the approach of image task imitators like LVM, we create video segmentation exemplars to form QA pairs on the VIPSeg (Miao al., 2022) dataset. The results, presented in Figure in Appendix A.2, indicate that VidIT success-

---

Published as a conference paper at ICLR 2025
fully learns the task format from the demonstrations. Moreover, we study how the model emerges the imitation ability by analyzing the attention score in Appendix A.5
Demonstration
Query & Generated
Figure 4: Generation Samples of VidIT on RT-1 Dataset.
5.5 ABLATION AND ANALYSIS
Different Number of Demonstrations We investigate the impact of different numbers of demon- strations on VidIT performance. Within 16-frame context window, we fix the query length frames and the generation as 4 frames, and construct four demonstration formats: 1) no demonstra- tion; 2) one 4-frame demonstration; 3) two 4-frame demonstrations sampled from different videos; 4) one 8-frame demonstration. The results in Table 4 demonstrate that providing demonstrations generally improves both P-Acc and V-ace scores. Additionally, providing more (2 x 4 versus x 4) or longer demonstrations (1 x 8 versus 1 x 4) both result in better performance.
Table 4: Semantic accuracy of different demonstration formats.
Table 5:
Results with text as demonstrations.
Demonstration
P-Ace
V-Ace
V-Ace
FID
No
29.6
22.9
Tune w/ text
28.8
19.46
X 4 frames
35.6
22.9
+ Infer w/o text
24.6
19.72
x 4 frames
37.2
24.2
Tune w/o text
24.7
20.02
8 frames
36.7
24.7
Text as Demonstrations text into the demonstrations. We fine-tune the 300M Pretrain model on the SSv2 dataset by append- ing corresponding text annotations the front of each video. The text annotations are encoded into latent representations using a pretrained T5-large model (Raffel et al., 2020). The results, presented Table 5, indicate that our model can seamlessly adopt demonstrations various modalities. Ad- ditionally, adding textual descriptions enhances the model's imitation capacity.
To demonstrate the versatility of model, explore incorporating
CONCLUSION
In this paper, we introduce Video Imitator, a novel model that extends imitation learning to video data. By converting video frames into sequences of discrete tokens and training the autoregressive Transformer with next token prediction, our VidIT model exhibits zero-shot imitation capabilities, which can generate semantically coherent and contextually appropriate video sequences based on provided demonstrations. Extensive experiments confirm that VidIT effectively captures and con- veys the semantic information embedded in the demonstrations, demonstrating its potential to hance various downstream tasks such as visual planning. Additionally, the model's versatility is verified by its ability to understand and integrate multi-modal demonstrations simultaneously.
ACKNOWLEDGMENT
This research was supported by the National Natural Science Foundation of China (Grant No. 62276245).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei Efros. Sequential modeling enables scalable learning for large vision models. arXiv preprint arXiv:2312.00785, 2023.
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 35:25005-25017, 2022.
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-jepa: Latent video prediction for visual representation learning. 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu. Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi. Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine. Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofrr Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale. In arXiv preprint arXiv:2212.06887. 2022.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https: /ttps////.p/...////////ee com/researchi www.swowoodsssssstossss..
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Joao Carreira, Eric Noland, Andras
Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv: 2018
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Yilun Du. Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2023.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recogni- tion, pp. 2873-12883, 2021.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Jannyun Guo. Zhiwei Hao, Chengcheng Wang. Yehui Tang, Han Wu, Han Hu, Kai Han, and Chang Xu. Dfta-efficient large vision models through sequential autoregression, arXiv preprint a40i 04841, 2024.
11

---

Published as a conference paper at ICLR 2025
Qingyan Guo, Rui Wang. Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujuu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint mnXiv:230.008522, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1pp.160000 16000-16009, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303. 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633- 8646, 2022b.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. In International Conference on Learning Representations, 2023.
Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders are stronger vision learners. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836. 2020.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: large language model for zero-shot video generation. arXiv preprint arXhv:2312.14125, 2023.
Bei Li, Rui Wang, Junliang Guo, Kaitao Song, Xu Tan, Hany Hassan, Arul Menezes, Tong Xiao, Jiang Bian, and JingBo Zhu. Deliberate then generate: Enhanced prompting framework for text generation. arXiv preprint arXiv:2305.19883. 2023.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-lengee video and language with ringattention. arXiv preprint arXiv:2402.08220 2024.
Jiaxu Miao, Xiaohan Wang. Yu Wu, Wei Li, Xu Zhang, Yunchao Wei, and Yi Yang. Large-scale video panoptic segmentation in the wild: A benchmark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Alexander Quinn Nichol and Prafulla Dhariwal.
Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
Suraj Patil, William Berman, Robin Rombach, and Patrick von Platen. amused: An open muse reproduction. arXiv preprint arXiv:2401.01808. 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 188)9, 2019.
12

---

Published as a conference paper at ICLR 2025
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020. 2020.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En-
hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Stephen Tian, Chelsea Finn, and Jiajun Wu. control-centric benchmark for video prediction. In International Conference on Learning Representations, 2023.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data- efficient learners for self-supervised video pre-training. In Advances in Neural Information Pro- cexsing Systems, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 18X.vv8//2.007111 2018.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual deseriptions. In International Conference on Learning Representations, 2023.
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6830-6839, 2023a.
Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything context. arXiv preprint Stti:2304.032884/ 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. th-thugt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 2022.
Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114, 2023.
Sherry Yang, Jacob Walker, Jack Parker-Holder Yilun Du Jake Bruce. Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Video as the new language for real-world decision making, arXiv preprint ar 2xXv:2402.17133 2024.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings the IEEE/VV Conference Computer Vision and Pattern Recog- nition, pp. 10459-10469, 2023a.
Lijun Yu. Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Min- nen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion diffusion-tokenizze tokenizer key to visual generation. In The Twelfth International Conference on Learning Representations, 2023b
13

---

Published as a conference paper at ICLR 2025
Maryam Zare, Parham M Kebria, Abbas Khosravi, and Sacid Nahavandi. A survey of imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions on Cybernetics, 2024.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Richard Zhang, Phillip Isola, Alexii A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RESULTS
In this section we provide more samples on different video datasets and benchmarks that VidIT model produces. Synthesis videos Something-sommetinng v2 dataset presented in Table 9. One can also navigate to our project page https: / aak mms www..mavveeeee /ivvvviiiil for clearer presented samples. Table 10 includes samples on Robotics Transformer-1 dataset, showcasing the ability of VidIT to fit on various form of video datasets. Furthermore, VidIT as versatile generalist, can act as simulator in reinforcement learning tasks, for which we provide detailed explanations.
A.1 VIDIT AS SIMULATOR
We demonstrate that VidIT can also function as a simulator reinforcement learning tasks by eval- uating it the vp (Tian al., 2023) benchmark. VP is benchmark for video prediction mod- els used in robotic manipulation via model- model--redicctive control. VidIT generates future frames with videos that accomplishing the same task as the demonstration, where the generated frames inversely reflect the corresponding actions that correctly interact with the environment (i.e. inverse dynam- ics). We compare the trajectories produced by the SVG (Villegas et al., 2019) baseline and VidIT in Figure 5. Evaluating on the Push-red task, we find that VidIT provides more precise control over the environment interaction.
t= 2
t=
t=
t= 10
Goal
VidIT
Figure 5: The trajectory produced by SVG and VidIT respectively.
Additionally, evaluate VidIT the MineRL environment as the underlying video generation model, where VidIT take in previous game scenes as conditions to imitate and generate subsequent scenes. Generated game scenes are then fed into inverse dynamics models generate the next action signal. We showcase two types of actions: Chop wood and Attack sheep. Figure 6, the VidIT correctly generate game scenes and produce corresponding action signal finish the action.
Chop
wood
Attack
Sheep
Figure 6: The VidIT model functions as video generator for chop-wood and attack-sheep in MineRL.
15

---

Published as a conference paper at ICLR 2025
A.2 VIDIT AS TASK IMITATOR
We expand the role of VidIT model to the task imitator video, which mirrors the cases in image imitation learning. this section, we finetune VidIT in VIPSeg dataset (Miao et al., 2022) for a few steps and show that VidIT also smoothly generalize video perceptual tasks like video seg- mentation. As depicted in Figure 7, VidIT successfully learns the task format from demonstration, and generates the segmentation outputs of the query video.
Demonstration Query
Generated
Figure 7: The VidIT also imitate video perceptual tasks such video segmentation.
A.3 CASE STUDY: BASELINE METHODS
In this section we compare the generated cases of VidIT with the baseline methods. As in Figure 8, we see that conditioned on the same frames, VidIT is capable of generating more semantically ac- curate and high-quality video than baseline models. Moreover, VidIT is lower in parameter budget, highlighting the priority VidIT as video imitator model.
Demonstration
Query(same for rows)
Generated
LWM(78)
LVM(7B)
VidIT (ours, 1.1B)
Figure Generated cases of VidIT and baseline methods.
16

---

Published as a conference paper at ICLR 2025
A.4 FAILURE CASES
When prompted with semantically unrelated videos, VidIT may struggle to generate coherent con- tent. In this section, present failure cases where VidIT fails align the query video with the semantic context of the demonstration video (see Figure 9).
For instance, in the first row, the demonstration video conveys the semantic information of moving a ball downward. However, when starting with a query video depicting an unrelated action, such as folding blanket, the model struggles to reconcile the two contexts. As result, it tends to ignore the demonstrated actions and generates content that lacks coherence with the demonstration.
Similar patterns are observed in other cases. These examples reveals VidIT's potential difficulty in maintaining semantic coherence when prompted with unrelated or ambiguous demonstrations. Addressing this limitation presents promising direction for future research.
Demonstration
Query Generated
Figure 9: Failure cases of VidIT.
Original
Training
Inference
Figure 10: The attention weight visualization during training and inference respectively.
A.5 INTERPRETABILITY OF GENERALIZATION
In this section, we highlight the interpretability of VidIT and its ability to generalize to imitation tasks in zero-shot manner. To demonstrate this, analyze the attention weights from the lan- guage model backbone to compare the similarity attention patterns when provided with the same demonstration during training and inference.
17

---

Published as a conference paper at ICLR 2025
Specifically, extract the attention weights from the last layer of the model and average them across all attention heads. Since the tokens are flattened from the original frames in raster scan order, rebind the tokens to their corresponding regions in the frames for better visualization. The results are shown in Figure 10.
The first row depicts the original frame sequence. The second represents the training case, where frames are drawn from a single video clip. The third row illustrates the inference case, where the sequence begins from different query frame. When generating new token, we observe similar attention patterns across the demonstration frames. Notably, the attention is concentrated around the moving object within the frames.
This phenomenon indicates that the model effectively learns and leverages the semantic information from previous frames, enabling it perform imitation tasks even when starting from different scene.
Aggregated Attention Weight Demonstration
Success(In-Clase FRandre(Random)
Query & Generation
Frame Aggregated Atlention Weight
0.15
0,10
0.05
5
Frame Index
Figure 11: The aggragated attention score distribution in demonstration and query frames.
Additionally, we analyze the failure cases in detail from the perspective of attention weights. Specif- ically, we examine how new frames attend to previous frames during generation. Intuitively, when new frame is effectively guided by the demonstration, it tends to allocate more attention to the frames in the demonstration video, rather than only attending to the query frames.
To illustrate this, visualize the attention distribution of generated frames previous frames, as shown Figure 11. Clearly, successful generation samples exhibit a tendency to allocate more attention to the demonstration frames, indicating ability to comprehend and utilize its semantic information. In contrast, the attention curve for failure cases reveals a higher focus on the query frames, effectively ignoring the guidance provided by the demonstration part.
A.6 FULL EVALUATION METRICS ON BASELINES
In this section, we complete Table 2's results with the all the evaluation metrics in Section 4.1. From the results in Table 6, it is evident that the VidIT model consistently outperforms the baseline models video imitation tasks, reinforcing its superior capability capturing both semantic alignment and visual quality.
Table 6: Full automatic metrics for baseline models. The best result bolded.
Semantic Accuracy P-Ace t
Visual Quality LPIPS 1
Demonstration
V-Ace f
PSNR t
FID +
FVD J
LVM (Bai et al., 2023)
21.2
27.1
12.45
0.489
27.58
163.13
Del.M (Guo al., 2024)
24.1
32.8
12.69
0.462
23.65
132.09
LWM (Liu et al., 2024)
21.8
24.9
11.89
0.474
29.91
191.56
VidIT (ours)
25.9
38.5
13.07
0.450
22.30
126.32
18

---

Published as a conference paper at ICLR 2025
Table
The configuration of different size of models.
Hidden dim
MLPP dim
Num. Heads
Num. Layers
300M
1024
2688
22
700M
1536
4096
16
24
1.1B
2048
4096
16
26
Table 8: Model hyperparameters.
Hyperparameter
Value
Learning rate scheduler
inverse sqrt 5e-4
Learning rate
Warm up steps
10000
Weight decay Optimizer
0.01
AdamW
Adam W betas
(0.9, 0.95)
Context length
4096
B IMPLEMENTATION DETAILS
B. MODEL ARCHITECTURE
We utilize LLaMA as the Transformer architecture our work. validate the scaling behavior, we train three different sizes of the VidIT model within the LLaMA architecture: 300M, 700M, and 1.IB. We carefully tune the hidden dimension, MLP intermediate dimension, and the number of Transformer decoder layers to achieve different model sizes. The configuration details of these models are presented in Table 7.
B.2 HYPERPARAMETERS
The hyperparameters used to train the VidIT model are presented in Table We utilize inverse square root scheduler and start model training with 10,000 warmup steps.
B.3 TRAINING DETAILS
Our largest variant, VidIT 1.1B, is trained 2x8H100 nodes, with Pytorch DDP parallel strategy integrated in the Pytorch-Lightent trainer. The training time is about 3 hours per epoch and thus the total training GPU hours is 720 (15 epochs).
C LIMITATIONS
Despite the promising results, several limitations exist in this study. Firstly, while VidIT excels at generating short video sequences, its performance longer sequences has not been thoroughly eval- uated. Secondly, the model's dependence on the quality and relevance of provided demonstrations lead to inconsistencies, particularly when demonstrations Future work should address these limitations by utilizing visual tokenizers with temporal compres- sion, and scaling or developing robust models handle imperfect demonstrations.
noisy semantically misaligned.
D ETHICS STATEMENT
Positive societal impacts VidIT include its potential to enhance various downstream applications such embodied planning and robotic simulation by enabling models to imitate actions demon- strated in videos. As for negative societal impacts, the widespread adoption this technique may raise concerns about privacy when powerful models gain the ability analyze and generate video
19

---

Published as a conference paper at ICLR 2025
content at scale. The datasets utilized in this study publicly available and have been thoroughly reviewed to ensure they do not include personally identifiable information or offensive content. Nonetheless, these datasets are sourced from the Internet, there may still be inherent biases. To address this, we have implemented a rigorous filtering process on the training data to minimize the potential for the model to generate inappropriate content.
Table More samples of VidIT generated samples on Something-somethhngg v2 dataset.
Demonstration
Query & Generation
LOOT
TOYS
20

---

Published as a conference paper at ICLR 2025
Table 10: More examples of VidIT generated samples RT-1 dataset.
Demonstration
Query & Generation
21


Reviews:
[{"review_id": "s1iJuN5LH0", "reviewer": "Reviewer_vT7n", "summary": "This paper proposes a video generation based on autoregressive transformers trained with the objective of imitating a set of seed videos. In experiments, the authors show that the model is able to imitate tasks presented to it in an in-context manner. In addition, the learned representations coming from the model are also useful for classification tasks.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\nThe idea of performing in-context autoregressive video generation is interesting and I have not seen it being done before. Autoregressive video generation without text is very difficult, and this approach seems to address some of the difficulties.\n\nQuality / Clarity:\nThe paper is clear and easy to read. The proposed idea is simple and I believe easy to replicate from the descriptions.\n\nSignificance:\nI believe that even though this paper is not producing mind blowing results such as Sora, Kling, Veo and others. The ideas proposed here can be useful for the scientific community to continue exploring the problem of video generation", "weaknesses": "Weaknesses / Questions:\n\nMissing baselines:\n* Not sure if I missed this from reading the Baselines paragraph in the manuscript, but did the authors include a simple next frame prediction baseline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4YrpwSnLC5", "reviewer": "Reviewer_vvMc", "summary": "The authors proposed to train an autoregressive model for video imitation. The model training is performed in a self-supervised manner using a continuous video. Then, in the inference stage, given a demonstration and query video, the model could generate subsequent video frames of the query video that imitate the same semantics or actions conveyed by the demonstration video. The model shows zero-shot capacity when generating the imitation video frames.", "questions": "- It is confusing to define a contrast action. What is the discipline? It might be better to provide a detailed list of all the situations that are considered.\n- Is it possible to provide the V-acc and LPIPS scores in Table 2? In my view, V-acc and LPIPS are more faithful and direct measurements of the visual content of the generated videos.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The topic of video imitation is valuable and would attract sufficient attention in the community, especially for embodied AI, since the approach could be applied to generate a large number of unseen training data, which is costly to obtain manually in the real world.\n- It is interesting that the autoregressive model trained without using demonstration videos can generalise to the imitation inference stage when the demonstration is provided.\n- The proposed framework is flexible and can be extended to imitate videos conditioned on text, which might be easier to acquire than demonstration videos and practically applicable.", "weaknesses": "- On the technical side, the novelty of the proposed method is limited. The combination of VQ-VAE and Transformer is commonly adopted for image generation.\n- There is a lack of detailed explanation about why self-supervised training without acquiring demonstrations during training will generalise to video imitations given demonstrations. \n- The visual quality of the generated imitation video is limited as shown in Table 1. The V-Acc and P-Acc scores are quite low, suggesting that the generated video's content might differ from the expected one. So, the practical usage of the proposed method is questionable.\nFailure cases would help readers understand the method's limitations. I suggest the authors provide a detailed analysis accordingly.\n- The evaluation of using text as conditions is very limited.\n- The quality of writing could be further improved by providing more details regarding task definition and technical details. For example, it would be better to list all the types of demonstration videos. Currently, it is unclear if any more tasks are involved other than object manipulation and camera movement (and their detailed types). The training objectives could be clarified (currently missing) for those unfamiliar with VQ-VAE/VQ-GAN for better reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jsV78wOGJF", "reviewer": "Reviewer_G9JK", "summary": "The paper introduces Video Imitator (VidIT), an autoregressive transformer model trained for zero-shot video imitation, enabling it to imitate the semantics of a given demonstration video and generate coherent video clips without requiring further fine-tuning. VidIT uses a next-token prediction objective to learn from video datasets, allowing it to generate video sequences that are semantically aligned with demonstrations. The model is evaluated on visual quality and semantic accuracy using both automatic metrics and human assessment.", "questions": "How well does the model generalize to longer video sequences compared to the current evaluations on short clips?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Zero-shot Capability: VidIT achieves a zero-shot imitation capacity, enabling it to generalize from video demonstrations without further training.\n\n2. Versatile Applications: The model can be applied to multiple downstream tasks, including robotic simulation, video planning, and segmentation.\n\n3. Comprehensive Evaluation Benchmark: Both objective metrics and subjective human evaluations validate the quality and semantic coherence of generated videos.\n\n4. Scalable Model: The model demonstrates improved performance when scaled up, providing strong semantic control in video generation.", "weaknesses": "1. Limited Sequence Length: Because now the mainstream video generation solutions have reached more than 5 seconds, the evaluation is primarily on short sequences; the performance on longer sequences remains untested.\n\n2. Gap Between the Training Web Videos and Evaluation Metrics: In the benchmark constructed in this article, the domains of the evaluation metric are mostly in indoor scenes (SSv2), which is relatively limited. This is likely why adding more web videos to the training data did not make the model perform better. To get more accurate conclusions, evaluation metrics should be developed for more open scenarios and diverse tasks mentioned in the appendix, such as task imitator.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zz13Pc3nLr", "reviewer": "Reviewer_PhVw", "summary": "This paper trains autoregressive Transformers on video datasets in a self-supervised manner, enabling the model to infer and imitate semantics from a demonstration video in a zero-shot way, allowing it to perform unseen tasks without fine-tuning.", "questions": "1. However, the main contributions of this paper are not entirely clear to me. I believe there might be two key contributions:\na) The development of a video imitator model capable of mimicking actions from a reference video.\nb) The demonstration that transformers possess zero-shot capabilities without needing specific training for tasks involving action imitation.\n\nIf the primary contribution is the first point, then the task of \"imitating actions from a reference video\" has been addressed in several previous works [1, 2]. Moreover, achieving the results shown in this paper could also be done using some existing open-source video editing models. Additionally, similar effects can be achieved by controlling camera angles [3].\n\nIf the primary contribution is the second point, then readers of this paper would be highly interested in understanding why transformers possess this zero-shot ability. However, the paper only demonstrates through experiments that task-specific training and the zero-shot approach yield similar performance. In my opinion, this is insufficient to explain the transformer's zero-shot capabilities. I recommend providing evidence through analyses of feature maps or attention scores to support this claim.\n\n2. In Section 4.1, the paper introduces four settings but ultimately selects two. It is recommended that the names of these settings remain consistent throughout the paper to avoid confusing the readers. Also, ensure that these names are consistent with the names used in the experiments in Section 5.\n\n3. Additionally, I suggest including more details about the training process, such as the hardware used and the approximate training time.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper trained a transformer model called VidIT, which can learn motion information from a reference video and apply this motion to the prediction of a target video.\n\n2. The method is effective, as extensive experiments demonstrate significant improvements in metrics, and the visual results further prove that the method effectively learns motion information.\n\n3. This approach is based on an interesting discovery, which the authors define as \"zero-shot video imitators.\" This finding could inspire future work in the transformer series.\n\n4. The method can be applied to various tasks, such as unconditional video prediction, proving its strong generalization ability.", "weaknesses": "See questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)