Improving Autoregressive Video Modeling with History Understanding
Authors: Wenyang Luo
Haina Qin
Bing Li
Jiwen Lu
Xin Tao
Pengfei Wan
Kun Gai
Venue: ICLR 2026 Poster (2026)
Keywords: autoregressive video generation
diffusion models
Primary area: generative models
TL;DR: We show that good representations of history frames significantly improve diffusion-based autoregressive video modeling performance, and propose MiMo, a masked modeling framework that improves history representations without pretrained encoders.

Abstract: Video autoregressive generation (VideoAR) sequentially predicts future frames conditioned on history frames. Despite the advance of recent diffusion-based VideoAR, the role of conditioning signal—internal representations of history frames—remains underexplored. Inspired by the success of strong condition representations in text-conditioned generation, we investigate: \textit{Can better internal representations of history frames improve VideoAR performance?} Through systematic analysis, we show that history representation quality positively correlates with VideoAR, and that enhancing these representations provides gains that cannot be achieved by refining future frames representations alone. Based on these insights, we propose \textbf{MiMo} (Masked History Modeling), a novel framework that seamlessly integrates representation learning into diffusion-based VideoAR. MiMo applies masks to history frame tokens and trains the model to predict masked tokens of current and future frames alongside the diffusion objective, yielding predictive and robust history representations without relying on vision foundation models (VFMs) or heavy architectural changes. Extensive experiments demonstrate that MiMo achieves competitive performance in video prediction and generation tasks while substantially improving training efficiency. Our work underscores the importance of history representations in VideoAR.

Published as a conference paper at ICLR 2026
IMPROVING AUTOREGRESSIVE VIDEO MODELING WITH HISTORY UNDERSTANDING
Wenyang 12-1 Haina Qin Bing Li
Jiwen Lu Xin Tao
Pengfei Wan
Kun Gai
Institute of Automation, Chinese Academy of Sciences
School of Artificial Intelligence, University of Chinese Academy of Sciences 3 KKlin Team, Kuaishou Technology
*Tsinghua University
ABSTRACT
Video autoregressive generation (VideoAR) sequentially predicts future frames conditioned on history frames. Despite the advance of recent diffusion-based VideoAR, the role of conditioning iignallinternal representations of history frames remains underexplored. Inspired by the success of strong condition representations in text-conditioned generation, we investigate: Can better inter- nal representations of history frumes improve VideoAR performance? Through show that history representation quality positively cor- relates with VideoAR, and that enhancing these representations provides gains that cannot be achieved by refining future frames representations alone. Based these insights, we propose MiMo (Masked History Modeling), a novel framework VideoAR.
systematic analysis.
that seamlessly integrates representation learning into diffusion- MiMo applies masks to history frame tokens and trains the model to predict masked tokens of current and future frames alongside the diffusion objective, yielding predictive and robust history representations without relying on vision foundation models (VFMs) or heavy architectural changes. Extensive experiments demonstrate that MiMo achieves competitive performance in video prediction and generation tasks while substantially improving training efficiency. Our work un- derscores the importance of history representations in VideoAR.
INTRODUCTION
Video autoregressive generation (VideoAR) predicts future frames conditioned on previously ob- served or generated frames (the history). The history-to-future generation process naturally aligns with the causal structure of video dynamics and enables variable-length generation (Villegas al.,
Yin et al. 2025 Teng et al.] (2025). However, early AR approaches (Yan et al.) 20_ Hong ct al. 2022 Ge et al. 2022; Villegas et al., (022) significantly underperformed non-AR methods (Brooks et al. 2024 Ho et al. [0022] He et al. 2022b Guo et al., 2024), primarily due to the dif-
2022
ficulty of modeling the complex conditional distribution of future frames given history. Recently,
diffusion-based VideoAR (Kondratyuk et al. 2023 Chen al. 2024a Song al. 2025 Gu al.
2025) has emerged as a promising solution, as it can approximate complex conditionals via iterative denoising of future frames from random noise, conditioned on the history frames.
Despite this progress, the conditioning signal the representation of the history frames remains underexplored. In text-to- text-to-image/ image/video (T2I/T2V) and clasiicondi generation, stronger dition representations consistently improve generation quality (Esser et al. 2024 Gao et al.) 2024 Kong al. 2024 Hu al. 20246 Wu et al., 2025), which raises natural question: Can better internal representations of history enhance VideoAR performance? D Intuitively, if the model's internal representations of history effectively capture the semantics and dynamics of the history frames, predicting coherent future frames should become easier. However, in current diffusion- based VideoAR, history representations are mainly learned via the diffusion objective, which may
Work conducted during author's internship at Kling Team, Kuaishou Technology. 'Equal contribution. 'ee focus internal representations of clean history frames conditions, distinct from methods that improve representations of noisy data within the diffusion process YYu at. 2024 Zhang et al. 2025).

---

Published as a conference paper at ICLR 2026
bistory
20
mesked history
MiMo XL
18
noisy future unkmown
future
prediction
BOBTRIF
16
FAR-XL
12
1420
LO0 Frrck
10
1:77x
4
100
300
640
frame prodiction
next frame predistion
Training Steps
a) Traditional Diffusion AR
b) MiMo (Ours)
Training Stepsvs, FVD
Good representations of history frames improve VideoAR. Our framework, MiMo, incor- porates masked modeling into the history frames during training. MiMo achieves significantly faster convergence than baselines without using VFM.
Figure 1:
not be optimal for learning semantically aligned, predictive condition representations. Moreover, good representations
not naturally emerge from VideoAR training. because predicting future frames requires modeling the low-level details of the future, which can hinder representation learn- ing (Yu et al. 2024). This limitation motivates us to design dedicated learning objectives to enhance history representations and improve VideoAR performance. Importantly, without relying on external vision foundation models (VFMs) that incur substantial training costs and may suffer from ofdiiiiibbutto issues when applied to video domains.
aim achieve this
In this work, we demonstrate that improving history frame representations can indeed enhance VideoAR performance- improvement that cannot be achieved by refining noisy future frame representations alone. Based on this insight, we propose Masked History Modeling (MiMo), a novel diffusion-based VideoAR framework without vision foundation model (VFM). illustrated in Figure I. MiMo naturally integrates masked modeling (Devlin et al.] [019] He et al. [2022a] Tong al. 2022), simple yet effective representation learning technique, into history frame modeling for VideoAR. Our approach works as follows: We first address the train-test discrepancy in recent methods (Chen et al.] 2024aa] [Song et al. 2025) by incorporating clean (noise-free) history frames alongside the noisy future frames being denoised during training, similar to the approaches of Zhou al. (2024)). Then, we mask (drop) portions of the history frame tokens and train the model to reconstruct the masked tokens of current and future frames parallel with the diffusion loss. This dual objective encourages the model to learn robust history representations that help the model predict future frames, while also improving its robustness perturbations in history.
ct al. 2025);
Hu Hu
Unlike previous work that applies masked modeling to noisy inputs diffusion models (Gao et al., 2023 Wei et al. 2023), which harms diffusion and requires complicated techniques to mitigate the negative effects, our approach operates on clean history frames. Our approach greatly alleviates interference with future prediction and requires minimal architectural modifications. MiMo sub- stantially improves training efficiency and generation quality through self-supervised visual repre- sentation learning and achieves strong VideoAR performance, all without external pretrained VFMs YYu et al,, 2024, Zhang et al.) 2025). In MiMo, history frames serve dual purposes: as conditions for denoising of the future frame, and as input for suff-supe representation learning. By uni- fying history representation learning with future frame diffusion modeling, our framework enables high-quality representations that boost video prediction and generation.
Our main contributions are:
1. We investigate how history frame representations impact VideoAR performance and demonstrate that better representations lead to improved generation quality.
2. We propose MiMo, a simple yet effective VFM-free framework that seamlessly unifies diffusion-based VideoAR with self-supervised history representation learning.
3. Our framework demonstrates competitive video prediction and generation performance in VideoAR, achieving state-of-the-art (SOTA) results several benchmarks.
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARIES
VideoAR Given a video x = {a, E R# ,,... with frames of height H and width W. AR approaches model the temporal sequence by generating future frames sequentially conditioned on historical frames, following the natural causal structure of video dynamics. VideoAR be formulated by conditional probabilities:
p(1++1 (m1e)) p(future frame history frames),
(1)
where 31d = {I1, T2,
Fe .2, represents the history frames and 2++11 is the next frame to generate.
Diffusion-based VideoAR The conditional probabilities defined by Equation T are usually com-
plex, which can be modeled by diffusion models (Sohl-Dickstein et al.) 2015; Ho et al. 2020). Diffusion-based AR approaches models Equation (1) by learning denoise the Gaussian-nosiee
future frame: Fi+1 = a,*+41 + O,C, conditioned on history frames Ii:t. where ~ (O, is noise level, and (ar, is noise schedule. This is typically done by
and E [0.
timating the score function so sp(x241: 37, x Vlogpr(#;4) (x14). (Vincent 2011). In prac-
tice, is often parameterized in alternative forms, such v-prediction (Salmaans & Ho 2022 ( TT, x1:)) x a, - It41
During training, diffusion forcing (ong et al. noisy history frames Via
2025 Zt
Gu et al. 2025 learns to that conditions
(T..)
{a]
with independent noise levels Ti.t
E E ||at, ft-1 on 1x+1
(1
(Te+1).
T1st4 T1
(TuLe) >13
In contrast, complete teacher foreing (CTF) (Hu et al. discrepency of diffusion forcing by conditioning on clean history frames T1:t
2024a| Zhou al. 2025) eliminates train-test
L = E, T1 ||are C++1 O 0g (T++1)
(z(2
(3)
During generation, the model iteratively denoises z141 using the learned denoising network, starting from pure noise and gradually recovering the clean future frame. Once x++1 fully denoised, appended to history frames for generating the subsequent frame T1+22-
3
METHOD
3.1 OVERVIEW
We hypothesize that good history frame representations benefit VideoAR. To investigate this hypoth- esis, we first analyze the relationship between history frame representation quality and VideoAR performance (Section 3.2) Our findings reveal that improving history representations improves performance, and such improvement cannot be achieved by solely refining noisy future frames.
These findings motivate approach. Additionally, we aim avoid using VFM (Yu et al. 2024). as they may perform poorly for our-of-distribution (OOD) data, and adapting or pretraining VFMs on OOD data requires additional effort and increases complexity. Specifically, we propose Masked History Modeling (MiMo), a unified framework that jointly optimizes history frame representation learning and VideoAR within single training process (Section 3.3 without using VFM. The insight of MiMo lies in treating history frames as noise-free conditioning signals during both training and inference, while introducing auxiliary masked video modeling objectives specifically targeting his- tory frames. The dual objectives ensure that the model develops robust history frame representations while maintaining strong generative capabilities. MiMo can also be extended to other pretraining
objectives (Oquab et al. 2023 Assran al. 2023 leave for future work.
Jiang et al.
2025
Wang & He 2025). which
3.2 EXPLORING REPRESENTATION OF HISTORY
In this section, analyze the impact of history frame representations VideoAR performance. We aim to understand whether good representations of history frames correlate with better genera- tion quality, and whether this is necessary-in other words, whether achieve all benefits by
3

---

Published as a conference paper at ICLR 2026
0.30
Linear Probing Accuracy
50
40
MiMe-AL
K:EPAA
MiMo--L ACDITAL
V.JEAA
CKMINA Allgoment
V-JEPALL
FARKAL
V-JEPALL: 0.11
Probing
FARWAL
VideoMAE-L
VidcoMAE-L
0.15
V.ERA-L
-0
V.JEAALL
10.40%
14.90%
10
11
Layer Index
17
19
21
23
100
160
Training Iterations
200
300
360
0.05
100
160 200
Training Iterations
300
140
(a) Linear Probing Accuracy
(b) Linear Probing Accuracy
(c) CKNNA Scores vs
vs Layer Index (300K iters)
Training lterations
Triining Iserations
Figure 2: Exploring representations of history frames.
solely improving the representations of the noisy future frames being denoised Yuu et al. (024). To exclude the influence of first-frame generation quality and focus on understanding the role of his- tory frames, we conduct experiments on the K600 video prediction task, where the video prediction model predicts eleven future frames based on five given context frames. To study representation quality, we perform linear probing on K600 and measure CKNNA (Huh et al. (024) to assess the similarity between model internal representations and pretrained representations (Yu et al.) (024). We select VidcoMAE (Tong et al., (202)) and V-JEPA (Bardes et al. (2024) VFM. All models use DFoT VAE (Song et al., 2025), with identical hyperparameters across all experiments and history guidance (see Appendix D.5) with scale 1.05 (Song et al. 2025) during inference. Details about evaluations are provided in Appendix
History frame representation quality correlates with video prediction performance We em- pirically investigate the relationship between history frame representation quality and video pre- diction performance using the models trained as shown in Figure [ with results summarized in Figure Our main findings include: (a) History frame quality positively correlates with video prediction performance- better models exhibit higher linear probing accuracy and better alignment with VFM (measured by CKNNA). (b) During training, history frame representation quality gradu- ally improves but consistently maintains a significant gap with pretrained models. (c) Our proposed MiMo method effectively improves history frame representation quality. Notably, MiMo changes the layer where linear probing accuracy peaks, as our method introduces decoders in later layers to execute the masked history modeling objective (see Section 3.3
Improving history frame representations is feasible way to improve video prediction per- formance. We investigate whether improving history frame representation quality can en- hance video prediction performance by training ACDiT-B models (Hu al. 2024a), which take clean history frames and noisy future frames as input during training, where both history and future frames can only attend to themselves and history frames. This architecture allows us to explicitly separate the representations of his- tory frames. We compare two approaches: one similar to REPA (Yu et al. 2024), which distills features from VFM into history frame represen- tations; another introduces the MAE objective (He al.) (022a) in history frames. Table shows that both REPA and self-supervised methods can improve representation quality and subsequently enhance video prediction performance, demonstrating that improving history frame representations is feasible.
Table 1: Improving representations of history frames.
Method
FVD4
A%c.(%)ff
ACDiT-B
54.814
6.21
History
40.022
16.96
REPA
Future
40.253
17.04
Both 36.542
19.23
Improving noisy future frame representations cannot replace the role of improving history frame representations. Besides history frame representations, the representation quality of noisy future frames also affects diffusion model generation performance (Yu et al. 2024 Zhang et al. 2025). A meaningful question is: sufficient to only improve the representation quality of noisy future frames? Our answer is We train ACDiT-B models and compare introducing REPA ob-
4

---

the current and next Published as a conference paper at ICLR 2026
h2
hg
hg
Decoder (P (1,)
Attention Mask of (a)
Use
Frame-
Histryy
Cleande-Claan
hg
h
Attention Mark Iterative
(0) top-left
Denoise
Qvery frames
Decoder 9e (0,-)
f.
DiT Block
DIT Block
Arasii
Nolsy Ceaan y
Notsy to Natey
ht
DIT Block
DET Block
DIT Block
DET Block
A
hg
Key Value Frames
Mule Mfrik)
History Future (Noss))
Luit
(a)
for Training
(b) Training
(c)
Figure 3: Framework of MiMo. (a) Attention mask used for training. Eight frames are shown. The clean history frames and noisy future frames are allowed attend themselves and previous history frames. (b) Training. Four frames are shown. The video clip x (. r3 is both used as history frames h and masked with a random tube mask M, and as future frames and noised with Gaussian noise The prediction targets of masked history modeling
frames. (c) AR Inference. Three history frames are already generated or provided by the user; the fourth frame is being denoised. After denoising, the fourth frame is appended to the history frames.
jectives in clean history frames, noisy future frames, or both. Table [ demonstrates that merely improving noisy future frame representations is insufficient. Simultaneously improving both history and future frames yields benefits beyond just improving noisy future frame representations, indicat- ing that history frames contain unique semantics. Note that our attempt to introduce MAE objective in noisy future frames (similar to Wei et al. (2023); Gao et al. (2023)) fails to surpass the perfor- mance of the ACDIT baseline without modifying the model's macro-architeccure (also reported by Gao et al. 2023)). We leave such exploration for future work.
3.3 MIMo: MASKED HISTORY MODELING
Motivated by our findings in Section 3.2 diffusion-based VideoAR.
propose MiMo improve history representations in
Framework Design The core design principle of MiMo is leverage history frames for dual purposes: (1) conditions for diffusion-based future frame generation, and (2) as input for self- supervised representation learning through masked modeling. This dual utilization enables the model develop robust history frame representations that are specifically tailored for video model- ing tasks. The design shown in Figure 3
During training, MiMo follows CTF (Hu al. 2024a Zhou et al. 2025 which exposes clean history frames for representation learning. Given a video clipx [ 22;
HT), we duplicate
as both history frames h {h1 .2,..., (rr and target future frames f = {f1, f2. fr where = f = x. The history frame h, is input without noise; it can attend to itself and all its previous history frames he Future frame fr is independently noised with Gaussian noise é, in DFoT Chen et al. 2024a] (ong et al.) (2025) ; can attend to itself and all the previous future frames hyret- This can be implemented by an attention mask depicted in Figure
The diffusion objective for future frame generation is:
Lan = E. [laace - o+fe - vo (07 ;T, M
(4)
where 2/ is the noisy version of the future frame az at diffusion timestep т, Ug is the denoising network (v-prediction (Salimans & Ho 2022)) conditioned masked history frames hM and M a random tube mask (Tong et al. 2022) applied on the history frames with a ratio r for masked history modeling (introduced below).
5

---

Published as a conference paper at ICLR 2026
Masked History Modeling To enhance the model's understanding of history frames, introduce a masked modeling objective on the clean history frames. We randomly mask a subset tokens in the history frame h2 and train the model to reconstruct the masked content. Crucially, the recon- struction target can be either the tokens of the current history frame or any of the clean future frames hezz-- This distinguishes it from the normal diffusion objective it allows greater flexibility.
Formally, introduce the reconstruction loss on the masked history frames h the reconstruction target is a set frames Te = {t, + 1} which contain both and its next frame + 1:
Lmask MM
h. (o 1 |/v po(t' - t, 10
(5)
Ta
MET.
where yo is lightweight decoder that predicts the masked tokens of frame t' T, and 1/2 h, denoising network's output features of the 1-th layer for the history frames h1s-
is the
The unified training objective combines both losses:
Liotal Laiff A.masks
(6)
where the hyperparameter À balances the masked modeling objective.
Inference During inference, as shown in Figure 3 MiMo discards decoder yo and operates in
standard AR fashion with KV cache (Zhou al. 2025 Hu et al. 2024a Gu al. 2025): given
clean history frames h1:1-1, the model generates the next future frame ft through iterative denoising. The learned history representations enhance the model's ability maintain temporal coherence and generate high-quality future content. The framework naturally supports variable-length generation by iteratively updating the history context with newly generated frames.
Discussion Compared with masked diffusion that applies a masked modeling objective to denois- ing input (Gao et al. 2023] Wei et al., 2023), our approach operates on clean history frames and mitigates the interference with the diffusion denoising process. Thus, MiMo requires no special ar- chitectural designs that masked diffusion approaches require. /Zhou et al. (0025) also corrupt history frames, but their motivation is to improve robustness noise in history, and they apply no recon- struction target to the history frames. Thus, they are still limited in history representations. Our approach also reduces the computational costs compared with those of Zhou et al. (2025); Hu et al. (202aa) due to masking.
EXPERIMENTS
4.1
SETUP
Tasks and Datasets We evaluate MiMo on three video modeling tasks: video prediction, uncon- ditional video generation, and class-conditional video generation. For video prediction, we use the Kinetics-600 dataset (Carreira et al., 2018), which consists of 480,000 videos with 600 categories (class labels are not used). Five frames are provided as initial conditions to predict the next eleven frames. For video generation, we use the UCF-101 dataset (Soomro et al. 2012) with 13,320 videos across 101 categories. No initial frame is provided, and the model generates 16 frames.
Implementation Details The architecture based on DiT (Peebles & Xie 2023). Our modifi cations are: 1). using QK normalization (Henry al., 2020) stabilize training, 2) incorporating RoPE (Su et al., 2024), and 3) using separate LayerNorm (Ba et al. 2016) for clean history frame The decoder is stack of four DiT blocks with the same configuration as the model. Hyperparameter = 0.5. The learning rate is x 10- for Kinetics and 4x 10 for UCF-101 both decayed to 10-5 with cosine schedule. The global batch size is 256 for Kinetics and 128 for UCF- 101. The weight decay is 0.001, and the betas for Adam W (Loshchilov Hutter (017)) are (0.9, 0.99). The model is trained for 360K steps Kinetics and I 180K on 1CF. For Kinetics, we DFoT's VAE (Song et al. 2025) with compression ratio of 4 x 8 8 and sample 17 frames per clip with resolution
"These modifications moderately affect performance, as shown in Section 4.3

---

Published as a conference paper at ICLR 2026
Table 2: System comparison on Kineties and UCF-101 with video prediction, unconditional video generation, and conditional video generation tasks. T; Different from the original work, we reimple- mented DFoT using a causal architecture align with the standard AR practice.
Kinetics (Pred.) UCF-101 (Uneond.) FVD.
UCF-101 (Cond.)
Method
Type
FVD.
FVD.
LVDM et al.
2022b
372
MAGVIT Yu al.
2023a
9.9
76
Non-AR
MAGVIT Yu
2023b
4.3
58
Latte Ma et al.
2024b
478
TATS Ge et al.
2022
420
332
Phenaki
Villegas ct al.
2022
36.4
Omni Wang et al. 2024]
32.9
191
DFoT-XL
Song et al. 2025
11.1
ACDIT-XL
Hu et al.
2024a
111
MAGI-XI
Zhou
2025
11.5
298
FAR XAR-XL-R
Guu al. 2025
279
108
VAE Reconstruction
3.7
15
15
AR
MiMe-XL
8,3
240
98
128. For UCF-101, we use FAR's (Gu et al. 2025) per-frame DC-AE (Chen et al. 2024b) with compression ratio of x 32 and sample 16 frames per clip with resolution 256. See Appendix [ for more details.
Inference and Evaluation We follow Song et al. (2025) for evaluation on Kinetics, generating 50,000 random videos and computing the Fréchet Video Distance (FVD) (Karras al., (019) on all frames (including conditioning and generated frames) with the groundtruth videos, both resized to 64 x 64. On UCF-101, following FAR. randomly sample 2,048 videos and compute the FVD against groundtruth videos, resized to 256 x 256.
4.2 MAIN RESULTS
Table 2
presents a comprehensive comparison of MiMo against state-of-the-art non-AR and AR methods across three video modeling tasks. For reference, we also report the reconstruction FVD of the VAE, which represents the upper bound of performance achievable given the groundtruth.
Video Prediction (Pred.) On the challenging Kinetics 600 video prediction benchmark, MiMo demonstrates exceptional performance with an FVD score of 8.3, establishing new state-of-the- art among AR models. This represents a substantial improvement over previous AR methods, with approach significantly outperforming DFoT (FVD: 11.1) despite using the same VAE. The per- formance gain directly demonstrates the superiority of MiMo, as both methods share the same un- derlying video tokenization and differ primarily in their treatment of historical context. Qualitative examples are presented in Figure Fa), a). where our method generates smooth, realistic continuations.
Unconditional Video Generation (Uncond.) For unconditional video generation on UCF- 101, MiMo achieves remarkable results, establishing new state-of-the-art performance among AR ap- proaches. Our method substantially outperforms the previous AR leader FAR by nearly 40 FVD points (240 vs 279) while utilizing the same DC-AE tokenizer, demonstrating the significant impact of our masked history modeling approach. Also noteworthy is the comparison with MAGI, which similarly employs Complete Teacher Forcing (CTF) during training-out method achieves con- siderable performance improvement (FVD: 240 vs 298), validating the effectiveness of our masked history modeling objective. The generated videos exhibit diverse motions, realistic textures, and coherent temporal dynamics, as illustrated in the qualitative examples in Figure b).
Coass-Codiitonal Video Generation (Cond.) class-conditional video generation on UCF- 101, MiMo again demonstrates superior performance, achieving state-of-the-art results across AR

---

always available, in which cases Published as a conference paper at ICLR 2026
methods. Our approach surpasses FAR by 10 FVD points (98 vs 108), confirming the consistent benefits of our approach across different conditioning modalities. The comparison with ACDiT is both methods utilize CTF and share similar architectural foundations, yet MiMo
also interesting both achieves notable improvements (FVD: 98 vs 111), consistent with our findings in unconditional generation when compared against MAGI. This consistency across tasks reinforces that perfor- mance gains stem from improvements in history representation learning rather than task-specific optimizations. Representative generated videos are shown in Figure 4c).
(a) Kinetics
(b) UCF uncond.
(c) UCF cond.
Figure 4: Visualization of generated videos.
4.3
ABLATION STUDY
This section ablates the designs of MiMo. All models are based on DiT-B trained on Kinetics for 100K steps with modifications and hyperparameters described in Section 4.1 ACDiT baseline is MiMo without masked history modeling, similar to MAGI and ACDiT.
Table 3: Comparison with variants of REPA.
Table 4:
Comparison of different prediction targets for masked history modeling.
Method
FVD
ACDiT Bascline
54.814
Target(s)
FVD4
History
40.022
ACDiT Baseline
54.814
REPA
Future
40.253
Current Frame
41.832
Both
36.542
Next Frame
37.782
MiMo
36.601
Current Next (MiMo)
36.601
MiMo +REPA-Bole
34.133
Current Next+NextNext
36.263
Comparison With REPA An alternative way to inject good representations into the model is distilling the features from a VFM, known as REPA (Yu et al.) 2024). several variants of REPA, supervising history frames, future frames, both. Both REPA and MiMo can significantly improve convergence, while MiMo performs on par or better than all variants. In practice, however, VFMs for the user's domain of interest MiMo is a viable substitute.
MiMo with
MiMo complementary VFM-based methods like REPA. MiMo excels at learning task-specific dynamics from the data, while VFM provides strong semantic priors. Combining MiMo with REPA in Table 3 yielded further improvements over either method alone. This suggests that MiMo and VFM capture different aspects of the data.
Prediction Targets of Masked Modeling One of the merits of MiMo is its flexibility: while diffusion always predicts the noise-free version of the noised current frame effectively, MiMo can
8

---

the en- Published as a conference paper at ICLR 2026
predict both the current and next frames for masked history modeling. Table 4 compares different prediction targets for masked modeling, and it is clear that predicting both current and history frames outperforms predicting either.
We also include a target of predicting the current and next two frames (Current + Next+Nex(Nextt) in T It is beneficial but yields diminishing returns. We hypothesize that predicting more distant future frame is a significantly harder task, and the increased difficulty does not naively translate into proportional performance gains. Our proposed target (Current + Next) strikes an effective balance without the added complexity of longer-range prediction.
Table 5: Decoder position (placed after the 1-th layer) moderately affects performance. DiT-B has 12 layers in total. None means CFT baseline.
None
12
11
10
9
FVDL
54.814
36.601
35.815
35.838
37.593
Decoder Position The decoder for masked modeling is usually placed after the encoder (He al.) 2022a Tong et al., 2022). We treat the first layers of the DiT model coder, and shorteut the output of the 1-the layer corresponding to the history frames into the de- coder. Table 5 shows the effect of varying The performance is robust to / when / is close to the last layer.
Table 6: Architecture modifications.
Modification
FVD|
Vanilla DiT
37.763
+RoPE
37.313
+Separate LayerNorm
36.601
Model Architecture Table shows the im-
pact of architecture modifications performance. Incorporating RoPE and separating LayerNorm layers for history frames both bring moderate gains.
Table 7: Ablations of hyperparameters À and mask ratios.
À
0.1
0.5
1.0
2.0
Mask Ratios
[0.25, 0.25]
[0.25, 0.5]
[0.5, 0.75]
FVDJ
40.213
36.601
37.443
38.910
FVD4
37.539
36.601
39.121
Hyperparamter Ablations Table 7 shows the impact of hyperparameters MiMo-B models with different from 0.25 to 2.0 and mask ratios from 0.25 to 0.75. For X, a weight 0.5 provides the best balance, but performance does not degrade sharply for nearby values. For mask ratios, performance remains relatively stable between 0.25 and 0.50; however, a higher mask ratio requires fine-tuning without masking to achieve better performance.
Table 8: Computational costs.
Method
MiMo-XL
ACDiT-XL
FAR-XL
Wall-Clock Time
0.750s
0.788s
0.704s
GFLOPs
8.22
8.81
5.94
Computational Costs Table shows the training computational costs of MiMo, ACDIT (our baseline). and FAR. Compared with our baseline (ACDiT), MiMo reduces training wall- clock time by 5%; compared with FAR, MiMo increases training wall-clock time by modest 10%, which is a small price for the significant performance boost (25% from 279 240
Table 9: Comparison of optimization strategies.
Strategy
FVDJ
ACDiT Baseline
54.814
Interleaving
38.543
MiMo
36.601

---

Published as a conference paper at ICLR 2026
Kinetics, and 14% from 279 to 240 UCF- 101). Compared with ACDiT, MiMo also re- duces the FLOPs per training step due to masking (the decoders increase FLOPs). MiMo has higher training FLOPs compared with FAR, but the increase training wall-clock time is moderate (10%) due to hardware acceleration. Note that MiMo has additional inference cost once the training is complete.
Alternative Optimization Strategy While our approach of simply using a weighted sum of the two losses is standard practice for auxiliary loss training, an alternative approach is optimizing the diffusion loss and the auxiliary loss interleavingly (a diffusion-only training step is followed by mask-only training step). The results are summarized in Table !|] has lower computational costs per step, leads to slower convergence, which diminishes speed gains.
While the interleaving approach
5 RELATED WORKS
Autoregressive Visual Generation. Autoregressive language modeling (Radford 2018 Radford
jet al. 2019) has facilitated the development of visual content generation using discrete visual tokens (Van Den Oord et al. 2017). In this framework, pre-trained visual tokenizers like VQ-VAE (Van
Den Oord et al. 2017) map visual patches into a discrete latent space, allowing visual generation to be approached similarly language modeling. Early works such as DALL-E (Ramesh et al.) 2021
focus on texl-to-image generation by learning a joint distribution between text and discrete image representations using an autoregressive pross-entropy loss. VideoGPT (Yan al. 2021) extends this idea to video generation, employing discrete tokens for autoregressive video prediction. VideoPoet (Kondratyuk et al.) (023) further advances this approach by integrating a causal video tokenizer (Yu) et al. (023b). Tiniiokeenizrr (Wang et al., (204)) proposes a unified tokenizer for both discrete and continuous representations. In contrast, our work focuses on frame-level causality rather than patch-level, avoiding the limitations of raster-scan order.
Representations and Generative Modeling. Recent advances in diffusion models highlight the importance of high-quality representations for generative modeling, diffusion models inherently struggle to learn good representations (Yu et al.) 2024 Zhang et al., 2025 Diang et al.) [025 Wang & He 2025). ). In practice, diffusion models are predominantly conditional generative models, where the conditions can be text prompts in T21/T2v T2V generation, or history frames in VideoAR. Despite this prevalence, few studies have investigated how the quality of condition representations affects generative performance. Existing evidence from text-conditional generation provides compelling support for exploring this relationship. Replacing CLIP text encoders with large language mod- els such as T5 and Llama has consistently improved generation quality, particularly for attributes strongly correlated with text conditions (counting, object reference, text rendering. etc.) (Esser eet al. 2024] Gao et al. 2024, Kong et al., [222]] Hu et al., 20246). Another evidence is that dis- class-conditioned image generation (Wu et al. 2025). These observations naturally extend to VideoAR, where future frames depend on his- tory frames as conditions, suggesting that enhanced history frame representations may potentially benefit video generation performance, which is the focus of our work.
tilling class representations improves the performance
6 CONCLUTION
In this work, we explored the fundamental question of whether good representations of history frames can improve VideoAR performance. Through systematic analysis, we demonstrated that enhancing history frame representations significantly benefits VideoAR, finding that cannot be achieved by solely refining noisy future frames. Motivated by these insights, we proposed MiMo (Masked History Modeling), a novel framework that naturally integrates masked modeling into diffusion-based VideoAR. By applying masks to history frame tokens and training the model to predict masked tokens of current and future frames alongside denoising tasks, MiMo learns robust representations that improve VideoAR performance. Our approach requires VFM or special architectural modifications. Extensive experiments across multiple benchmarks demonstrate that MiMo achieves competitive performance in video prediction and generation tasks, establishing new
10

---

Published as a conference paper at ICLR 2026
state-of art results. Notably, our framework substantially improves training efficiency and gener- ation quality, showcasing the effectiveness unified representation learning and diffusion modeling.
REFERENCES
Michael S Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic inter- polants. arXiv preprint (XXIv:2209.15571 2022.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619-15629 2023.
Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Am- mar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09983, 2025.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 1xXiv:6677006550 2016.
arXiv preprint
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.060 08471, 2024.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, al. Video generation models as world simulators. OpenAl Blog, 188);1, 2024.
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, al. Genie: Generative inter- active environments. In Forty-first International Conference on Machine Learning, 2024.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinctics dataset. In proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2399,6308, 2017.
Joao Carreira, Eric Noland, Andras Banki-Horvathe Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv: /00.:v:.888.01111 01340, 2018.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-1132, 2022.
Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitz- mann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081-24125,, 2024a.
Junsong Chen, Jincheng Yu, Chongjian Ge. Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023. URL
00426
Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang. Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution diffusion models. arXiv preprint arXiv:2410.1077 2024b.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
11

---

Published as a conference paper at ICLR 2026
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
DC Dowson and BV66011 Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 12(3):450-455, 1982.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
Peng Gao, Le Zhuo, Dongyang Liu, Ruoyi Du, Xu Luo, Longtian Qiu, Yuhang Zhang, Chen Lin, Rongjie Huang, Shijie Geng, et al. Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint 24tiv::400.00555 2024.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer strong image synthesizer. In Proceedings of the IEEE/CVE international conference on com- puter vision, pp.
2023.
songwei Ge, Thomas Hayes, Harry Yang. Xi Yin, Guan Pang. David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. In European Conference on Computer Vision, pp. 102-118. Springer. 2022.
Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503.19335, 2025.
Xun Guo, Mingwu Zheng. Liang Hou, Yuan Gao, Yufan Deng. Pengfei Wan, Di Zhang, Yufan Liu, Weiming Hu, Zhengjun Zha, et al. 12v-adapter: A general to-de-to-video adapter for diffusion models. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-12, 2024.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang. Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. arXiv preprint arXiv:2307.0472. 2023.
Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF international conference computer vision, pp. 7441-7451, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022a.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for fideliiyyitty long video generation. arXiv preprint arXiv:2211.3322. 2022b.
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. arXiv preprint arXiv:2010.04245 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. 2tt.:22277.1598.. 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi. and David J Fleet. Video diffusion models. Advances Neural Information Processing Systems, 35:8633- 8646, 2022.
12

---

Published as a conference paper at ICLR 2026
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large- scale pre- training for text-to-video generation via transformers. arXiv preprint TrXivv2205.15888 2022.
Jinyi Hu, Shengding Hu, Yuxuan Song, Yufei Huang. Mingxuan Wang. Hao Zhou, Zhiyuan Liu, Wei- Wei-Ying Ma, and Maosong Sun. Acdit: Interpolating autoregressive conditional modeling and diffusion transformer. arXiv preprint ar arXiv:2412.07720 2024a.
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with IIm for enhanced semantic alignment. arXiv preprini arXiv:2403.055 0335, 2024b.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation
hypothesis. arXiv preprini arXiv:2405.0777 007777777 2024.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456. pmlr, 2015.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9307- 9315, 2024.
Dengyang Jiang. Mengmeng Wang. Liuzhuozheng Li, Lei Zhang, Haoyu Wang, Wei Wei, Guang Dai, Yanning Zhang, and Jingdong Wang. No other representation component is needed: Diffusion transformers can provide representation guidance by themselves. arXiv preprint arXiv:2505.02831, 2025.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401 1410, 2019.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint
2017.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 34:21696-21707, 2021.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming- Ming-Chang Chiu, et al. Videopoet: A large language model zero-shot video generation. arXiv preprint wwxiv:2312.14122. 2023.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXhv:2412.03600 2024.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference machine learning, pp. 3519- 3529. PMIR, 2019.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 5404-5411, 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben- Gen-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02744) 2022.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.030033 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17110:1711.05101 2017.
13

---

Published as a conference paper at ICLR 2026
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in neural information processing systems, 1-5777--7777 2022.
Ge Luo, Gian Mario Favero, Zhi Hao Luo, Alexia Jolicoeur-Martneauu, and Christopher Pal. Beyond fvd: Enhanced evaluation metrics for video generation quality. arXiv preprint arXiv:2410.05203, 2024.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden and Sain- ing Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23-40. Springer, 2024a.
Xin Ma, Yaohui Wang. Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.0300 033444 2024b.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1rXxv:1802.03426, 2018.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference machine learning, pp. 8162-81171 PMLR, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv2334..01193 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205, 2023.
Alec Radford. Improving language understanding by generative pre-training. OpenAl blog, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 1(8):9, 2019.
A. Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ArXiv, abs/2102. 12092, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.005112 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265, pmlr, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02503 2020a.
Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion. arXiv preprint 4rXfv22502.06764, 2025.
Yang Song, Jascha Sohk-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 13456, 2020b.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.0402, 2012.
Jianlin Su. Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
14

---

Published as a conference paper at ICLR 2026
Hansi Teng, Hongyu Jia, Lei Sun, Lingzhi Li, Maolin Li, Mingqiu Tang, Shuai Han, Tianning Zhang, WQ Zhang, Weifeng Luo, et al. Magi-l: Autoregressive video generation at scale. arXiv preprint ar arXiv:2505.13211, 2025.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances in neural information process- ing systems, 35:1 1078--1009,, 2022.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: new metric & challenges. arXiv preprint arXiv: (rXxv://12200711,, 2018.
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408. 14837, 2024.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2008
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02399, 2022.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1661-1674 2011.
Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation. arXiv preprint arXiv:2406. 093999 2024.
Runqian Wang and Kaiming He. Diffuse and disperse: Image generation with representation regu- larization. arXiv preprint arXiv:2506. 09020,007722 2025.
Chen Wei, Karttikeya Mangalam, Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Ci- hang Xie, Alan Yuille, and Christoph Feichtenhofer. Diffusion models as masked autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16284-16294, 2023.
Ge Wu, Shen Zhang, Ruijing Shi, Shanghua Gao, Zhenyuan Chen, Lei Wang, Zhaowei Chen, Hongcheng Gao, Yao Tang. Jian Yang, et al. Representation entanglement for generation: Train- ing diffusion transformers is much easier than you think. arXiv preprint arXiv:2507.01467, 2025.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.1001 10157, 2021.
Wilson Yan, Danijar Hafner, Stephen James, and Pieter Abbeel. Temporally consistent transform- ers for video generation. In International Conference Machine Learning, pp. 39062-39098. PMLR, 2023.
Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang, From slow bidirectional to fast autoregressive video diffusion models. In Proceed- ings of the Computer Vision and Pattern Recognition Conference, pp.
2025.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Ming-Hsuan
Hauptmann, Ming- Hsuan Yang. Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recog- nition, pp. 04499-10469 2023a.
15

---

Published as a conference paper at ICLR 2026
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023b.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06944 2024.
Xiangdong Zhang, Jiaqi Liao, Shaofeng Zhang. Fanqing Meng, Xiangpeng Wan, Junchi Yan, and Yu Cheng. Videorepa: Learning physics for video generation through relational alignment with foundation models. arXiv preprint NKiv:25052236333 2025.
Deyu Zhou, Quan Sun, Yuang Peng, Kun Yan, Runpei Dong, Duomin Wang, Zheng Ge, Nan Duan, Xiangyu Zhang, Lionel M Ni, et al. Taming teacher forcing for masked autoregressive video generation. arXiv preprint 2rXiv 12389, 2025.
16

---

Published as a conference paper at ICLR 2026
A DIFFUSION MODELING
In this section, we present a brief overview diffusion- bassed generative models. These models learn to approximate target distributions through training denoising neural networks. There are two correlated approaches: "conventional" diffusion models based on score matching (Appendix A.1 and flow matching (Appendix A.2)) introduced below.
A.1 SCORE MATCHING
Diffusion models based on score matching (Ho et al. 2020 Kingma et al. 2021 Song et al. (020b) generate samples I: po(-) by learning to invert noise corruption process (i.e., the diffusion process) that transforms the data distribution into standard Gaussian noise ~ (0 f). The forward diffusion process is defined as:
Pr (x((7(ax) (x)) = Na,,,, I T [0,1,
where the coefficients a and o, specify the "noise schedule" that interpolates between data and noise. Usually ap 1, 0 and a1 0, 1, that 22(0) and xx))
The generative process is realized by integrating the reverse-time stochastic differential equation
(SDE) (Song et al. 2020b Lu al. 2022) that describes the backward diffusion process:
(x(
[(T)a( (T) e(r) (x))) dr + (
(8)
where ūi denotes the reverse-time Wiener process, and the drift and diffusion coefficients ff and are given by:
S(T) d at dr
( /log(ar/or)
(9)
dr
A score network sg (x(1) is trained to approximate V viit) logp: (x(T)) via denoising score match- ing (Vincent 201T):
min (e,,, (x()) [lle + o, so s6 TT)
(10)
0
Beyond directly modeling the score o (x((t); diffusion models commonly use equivalent param- cterizations tied to the forward relation (+)) = a, + o,e.
Noise prediction (Ho et al. 2020). The model predicts the noise c,(x((); e, yielding the score via
so (x(
or co :1))
1
(11)
and is trained with the MSE objective E [ (z( 12]]
Data (clean sample) prediction (Sohl-Dickstein al., 2015). The model outputs a denoised esti-
mate xo (T) Re (0) Converting to score gives
a(1) a.ro(1
so (o(
(12)
of
which is equivalent to first forming ē (x( (4.00//0, and then using 80 -E/or. Training objectives is minimizing E. [[]]x:(0) 20 ( :)
v-prediction (Salimans & Ho 2022). A time-dependent linear combination is predicted:
vo (T())));)) = a,c (0).
(13)
From Ug one can recover all other targets:
{(x!"); 0, 0r)(1) + a Up (T)
(14)
a + oZ
a,2(T) - 0, Up (z)((())))11 (T)
[o(x!));7)
(15)
a of
(0(2!))
E(( (T)
(16)
17

---

Published as a conference paper at ICLR 2026
The training objective becomes E [l|a,c o, uo v -
All these parameterizations are connected by r-dependent linear transforms, and thus represent the same model class. Choosing among them mainly affects optimization stability and the weighting of errors across noise levels.
A.2 FLOW MATCHING
Flow matching (Lipman et al. 2022; Liu et al. 2022; Esser et al., 2024) simplifies score matching by defining the generative process via ordinary differential equations (ODEs). Specifically, given the same definitions of æ x (r)), 04,, By as in Appendix A.1 the generative process is governed by probability flow ODE:
dzlr)
v( ,T),
(17)
dr
where the velocity field ((((();)))))))))))() (T) satisfies:
(r)
v(z(r) .T) =B
(Tr) = =.r + +o,B(((x(7)
(18)
dr
drz where å, = da, and 8, dr dr
The flow matching objective trains neural network up (x) to minimize: min E, (0,)) (1100()(7);7) - (á,z( + ò l
(19)
Flow matching and score matching are connected by the score function:
((x(7);7 Ele| (7)
E
(20)
ae
which corresponds to an equivalent reverse-time SDE (Ma et al. 2024a):
(x = ((x(7) 7 sxx(7) ())r + Vn,
(21)
where 7/r controls the amount of stochasticity and Wi is reverse-time Wiener process in Ap- pendix A.I Solving Equations (18) and (00). we obtain:
(((() 7) a,,(()) s(x(7) or o, - a,
(22)
Thus, flow matching and score matching learn the same model class.
Flow matching is easy to implement and usually converges faster than score matching in practice (Liu et al. 2022. Esser al. 2024). Another advantage of flow matching the flexibility to choose the diffusion coefficient 7), independently of the training process, allowing for post-hoe optimization of the sampling procedure.
B EXTENDED RELATED WORKS
Masked and Diffusion Video Generation Diffusion models have recently gained prominence in visual generation tasks (Ho et al., 2020, Rombach et al., 2022 He et al. 2022b Guo et al.) 2023 Chen et al. 2023 Guo et al. 2024), effectively extending to video generation. Video diffusion mod- els (Brooks et al. 2024 Ho et al. 2022) utilize bidirectional attention and binary mask embeddings to facilitate Prame-level autoregressive prediction. Notable works such GameNGen (Valevski jet al., 2024) use bidirectional diffusion models for real-time game generation. However, due to their bidirectional nature, these models cannot leverage KV Cache for extended video generation, limiting their scalability. Several masked video generators, such as Genie (Bruce et al.) 2024), ex- tend MaskGIT (Chang et al., (022)) into a causal-attention-bassee architecture for video generation. Despite their advantages, these methods suffer from the training-inference gap inherent in masked autoregressive modeling. which negatively impacts generation quality. contrast, our approach fully leverages KV Cache during inference, facilitated by our training paradigm that bridges the training-inferent gap through a novel complete teacher forcing paradigm.
Thhe velocity field in flow matching is different from the v-prediction parameterization in score matching,
though they correlated: the two parameterizations are connected by Equation 22
18

---

Published as a conference paper at ICLR 2026
Table 10: Hyperparameters.
Name
MiMo-B
MiMo-X XL
Input
Dataset
Kinetics-600 prediction 17 x 128 x 128
Kinetics-600 prediction 17 128 128
UCF-101 class cond. 16 256 256
UCF-101 uncond.
Task
Input shape
16 256 x 256
VAE
Compression ratio
x 8
x 8
x 32
32 32
Latent shape
x 16 x 16
x 16 16
16 x x 8
16 x x
Architecture
Patch size
1 x x
x 1 x
1x1x x
1 1 x
Depth
12
28
28
28
Embed dim
768
1152
1152
1152
Num heads
12
16
16
16
RoPE theta
100
100
100
100
Decoder
Depth
23
23
23
Diffusion
Parameterization
v-prediction linear fused min-SNR
v-prediction linear fused min-SNR
velocity rectified flow
velocity rectified flow
Noise scheduler
Weighting
logit-normal
logit-normal
7 5.0, p 0.96
5.0, p 0.96
Sampler
DDIM
DDIM 50
Euler 50
Euler class guidance 2.0
Sampling steps
history guidance 1.05
history guidance 1.05
Guidance
Optimization
Training steps
100K
360K
180K
180K
Batch size
256
256
128
128
Optimizer
Adam
AdamW
AdamW
Adam W
Learning rate (LR)
x 10 4
10-4
4 10-4
x 10-4
Warmup steps
10K
10K
10K
10K
LR schedule
cosine
cosine
cosine
cosine
End LR
10 5
10 5
10 5
10 5
Weight decay
0.001
0.001
0.001
0.001
(81, (2))
(0.9, 0.99)
(0.9, 0.99)
(0.9, 0.99)
(0.9, 0.99)
Gradient clipping
1.0
1.0
1.0
1.0
X
0.5
0.5
0.5
0.5
Mask ratios
[0.25, 0.5]
[0.25, 0.5]
[0.25, 0.5]
[0.25, 0.5]
EMA decay
0.999
0.999
0.9999
0.9999
C DATASETS
Kinetics-600 Kinetics-600 (Carreira et al. 2018) is a large-scale video action recognition dataset that extends the original Kinetics-400 dataset (Kay et al. 2017), containing approximately 500,000 video clips across 600 human action categories, sourced from YouTube and covering diverse human actions ranging from sports and cooking to dancing and musical performances. The dataset is split into training, validation, and test sets, with each action class containing at least 600 video clips in the training set and 50 clips in both validation and test sets. Videos in Kinetics-600 characterized by
19

---

Published as a conference paper at ICLR 2026
their temporal dynamics and complex motion patterns, making it challenging benchmark for video understanding tasks. The dataset provides rich temporal information and diverse visual content, which makes it particularly suitable for evaluating autoregressive video modeling approaches that need to capture long-term temporal dependencies and generate coherent future frames based on historical context. Following existing works (Song al. 2025), we use resolution of 128 128 pixels and train on the training set while evaluating on the test set. The model is conditioned on the first 5 frames and predicts the next 11 frames, totaling 16 frames.
UCF-101 UCF-101 (Soomro et al. 2012) is widely used action recognition dataset consisting of 13,320 video clips distributed across 101 action categories. The dataset was collected from YouTube and contains realistic videos with significant variations in camera motion, object appearance, pose, scale, viewpoint, cluttered background, and illumination conditions. Each action class contains 25 groups of videos, with each group sharing common features such similar backgrounds, simi- lar viewpoints, etc. UCF-101 covers a diverse range of human actions, including sports activities (e.g., basketball, tennis, surfing), musical instrument playing, and daily life activities. Despite be- ing smaller in scale compared to Kinetics datasets, UCF-101 remains a fundamental benchmark for assessing the generalization capability of video models across different domains and action com- plexities, due to its well-curated action categories. We follow the protocol of Gu et al. (2025) and use resolution of 256 x 256 pixels. The models trained the full UCF-101 dataset and evaluated with class labels as the only initial condition, generating total of 16 frames.
D IMPLEMENTATION DETAILS
Table 10 summarizes the hyperparameters we use in our implementations. The details are discussed in the following sections.
D.1 MODEL ARGH.ARCHHI ARCHITECTURES
Diffusion Models We employ the Diffusion Transformer (DiT) (Peebles & Xie 2023) with full 3D attention as our backbone. The DiT block is analogous to a vision transformer (ViT) (Dosovit- skiy et al.) 2020) block and replaces the LayerNorm (Ba et al.) 2016) layers with adaptive Layer- Norm (Adal.N) (Peebles & Xie|[023) layers to inject diffusion timestep condition into the features. AdaLN works by embedding the timesteps using sinusoidal positional encoding (Vaswani et al.) 2017) and feeding them to an MLP to predict the shift and bias factors for LayerNorm layers. For class-conditioned generation, the class labels are also embedded and added into the timestep em- beddings as additional conditions. Following existing works (Song et al., 2025 Gu et al. 2025 Hu eet al. 20244), AdaLN is applied separately to each noisy future frame because different frames can have different diffusion timesteps during training (Section 3.3). We use QK normalization (Henry al. (020)) stabilize training. Below, we introduce two other modifications apply vanilla DiT: separate LayerNorm and 3D RoPE. The vanilla DiT block and our modified DiT block are
illustrated in Figure
Separate LayerNorm layers instead of AdaLN are applied to the clean frames. Note that the other parameters are shared among all frames regardless of whether they are history or future frames. The attention mask introduced in Section 3.3 is applied to the attention operation to ensure causality between history and future frames.
Additionally, we incorporate axial 3D RoPE (Su et al. 2024) and assign an equal number of channels to encode the positions along the T, H, W dimensions.
VAE and Patch Size of DIT Video generation models usually work in some compressed latent space with reduced space-time dimensions to save computations, due to the sheer volume of pixels in video. In all of our experiments, the patch size of DiT is x x 1 (T, H. W), meaning that compression is done solely in VAE.
For fair comparison with existing methods, adopt the pretrained 3D video VAE of DFoT Song ct al. 2025) for Kinetics-66l experiments. DFoT's VAE has a compression ratio of 4 x (T, H. W). where the first frames are separately encoded and the following frames temporally
20

---

Published as a conference paper at ICLR 2026
Scale
Scale
FFN
FFN
Scale & Shift
Scale & Shift
LayerNorm
LayerNorm
LayerNorm
Scale
Scale
Attention
Attention (w. RoPE & QK-narm)
Scale & Shift
Scale & Shift
LayerNorm
MLP
LayerNorm
LayerNorm
MLP
0
02
9
T2
h1
0
92
03
T2
+
+
Atteration
Mask
Attention Mask
Label (optional)
Label (optional)
(a) vanilla DiT block
(b) modified DiT block
Figure Illustration of vanilla and our modified DiT blocks.
downsampled by a factor of 4, following Yu et al. (202aa). The input resolution 128 x 128 pixels with 17 frames, leading to latent shape of x 16 16 per video clip.
We utilize the 2D image DC-AE of FAR (Gu et al. 2025) for UCF-101 experiments. FAR's DC- AE has compression ratio of 32 32 with no temporal compression, and it encodes each frame independently. Given input of 16 frames with resolution of 256 x 256, the latent shape 16 x
Decoder The decoders take masked history frame features from intermediate DiT layers as the only input, and fill the masked positions with learnable query tokens. Then, the input is fed into stack of several decoder blocks and reshaped to the same dimensions as the latents of the history (or future) frames the output. The decoder block is the vanilla VIT block with axial 3D RoPE.
D.2 DIFFUSION
Kinetics-600 Experiments For Kinetics-600, use linear noise schedule (Nichol & Dhariwal 2021) with the v-prediction parameterization (Salimans & Ho 2022) and zero terminal SNR (Lin at. 2024h. We use the DDIM sampler (Song et al. 2020aa) with 50 sampling steps during inference. We also incorporate the fused min-SNR loss weighting (Chen et al.) (2024a), a variant of min-SNR loss weighting (Hang et al. 2023) for video diffusion, to accelerate convergence.
Fused min-SNR extends the standard min-SNR loss weighting by accounting for the "signals" from previous frames. The difference between the two methods is the way to compute the signal-to- noise ratio (SNR) used for loss weighting. Using the notations in Section 2 SNR is defined as SNR, = a o²" Fused min-SNR first normalizes the SNR to [0, 1] by dividing by the maximal value of SNR. Since min-SNR weighting clips the SNR value with the hyperparameter > O, we
normalize by y. Then, it computes fused SNR St with decaying factor 0:
St = normalized SNR factor for the f-th noisy future frame E [0, 1},
(23)
5. = pS.- + (1 p)S. (exponentially decayed cumulative SNR),
(24)
S = 1 - (1 3.-1) (fused reweighting factor),
(25)
Fused SNR Sy combines the current frame signal with accumulated history signals, treating them as independent probabilistic events. This accounts for the additional information available from history
21

---

Published as a conference paper at ICLR 2026
def compute_ los_____________ _oss_weight (snr, gamna, prediction_tysee type, decay=None, causal=Irue):
"*"Compute SNR weighting.
Args:
snr (torch, rensor) per-frane SNR of shape [B, T]
gamma (float): clip threshold of min-SNR prediction
(str): "epsilon"s "r_prediction", or "sample" decay (float, optional): if None, enable fused min-SNR with the specified decay factor
causal (bool, optional): whether we are training causal model
Returns:
12
weight (torch, Tensor) : per-frame loss weight shape [B, Tl
13
"**
14
Compute fused SNR
clipped_ = snr clamp (max-gamna)
if decay is not None:
17
nermalized stteeed___pppe__e__eeeeee snr clipped_snt snr gamma normalized_snz snr / ganma
18
19
def compute_cum__nn (reverse: bool False): wel_normllize__ce pid___cppped__m
22
normal ized_clllli ipped_ i (l (1) if reverse else normalized clippeeee__pppped_eee_e__ snr
23
26
cum snr torch.zeros. like (new rormaliied__iippeeddee..eeeeeede _nr) for t in range (0, snr shr.shape
27
28
if == == 0:
cum_snr t] new_nommll
else:
31
cum [nm_snr[;, t]
32
decay cum_snr[:, t - 1]
35
+ (1 - decay) new_normalizen weeoommllieed.... www.poreelzed.cllppeed snr I;, tl
35
cun_ snr torch. nn. www.tiunnn.nnnn........ (em (erlluuu__srrii, :-11, (1, 0, 0,
0), value-0.6 0)
36
return cum_snr. flip(1) if reverse else cum_snr
if causal:
39
cum_snr snr compute_cum, snr
40
else:
41
cun_snr = compute_cum_ (reverse=Truc) + compute_eun_ent ()
cum_snr 0.5
43
clipped__fusd__snn 1 (1 cum_snr decay) (1
ipe _nrr)
44
fused_anr 1 (1 cum_anr decay) * (1 normal i izer_snr)
clipped_snt = al i
gamma
fused gamma
47
48
Compute loss weight if prediction_type -- "epsilen": noise-prediction
49
weight clipped_snr / snr elif prediction_type weight clipped_snt snr (snr + 1) else: # data-prediction weight clipped_snr
"v_prediction"; v-prediction
52
53
56
return weight
Listing 1: Fused min-SNR (PyTorch psuedo-code)
22

---

Published as a conference paper at ICLR 2026
context in video generation, beyond what standard SNR weighting captures. The S is denormalized by multiplying y and used to compute the loss weighting as normal min-SNR weighting does.
Listing I
summarizes the algorithm to compute fused min-SNR weighting.
UCF-101 Experiments For UCF-101, follow Gu al. (2025) and flow matching et al. 2022 Lipman et al. 2022 Albergo & Vanden--ijnden 2022) with a "straigh" flow path, i.e.
(Liu
Q T.O, = r. We also adopt the logit-normal timestep sampling strategy (Esser al. 2024
where the timesteps are sampled from logit-normal distribution (instead of uniformly):
log (t/1 (t/1 -
Tin (T)
exp
(26)
2T 1(1 1)
2
We use the Fuler integrator sampler (Esser et al.
2024 with 50 sampling steps during inference.
D.3 TRAINING
Algorithm Training (v-prediction or flow matching)
Input: Dataset D, noise schedule ((a,,r))s velocity network vp, decoder (pg, loss weight À Output: Trained velocity network v0
1: while not converged do
2: Sample video clip
[x:)T_ from D
// Duplicate x as history h and future f 4: Sample {Te Uniform[0, 1]}²= and (er (O,,
3: h - x, f +
5:
Sample random tube mask M
6:
h. applyMaskk(h,
Apply M to history frame
7:
L 5+0
8:
for
to do
9:
f( (T+)
a + o
11 Add noise to future frame v-predicttion or flow matching // Internally apply attention mask (Figure // Diffusion loss (Equation
10:
Utarget t E, - or. or Utarget +- ār fo + Gr
11:
Upred (T( T, MM)
12:
Cair || Varret
Veent t- Ug (T) MMM
13:
11 Output features of the (-th layer for history frames h1s // Frame indexes // Masked history modeling loss
Te [t, 1
Lmask < T7. EVET. like solll' 1, * 1
(Equation (5))
C L Laitt >LCackk
17:
end for
Update 9 using VeL
19:
end while
The training algorithm is summarized in Algorithm Training hyperparameters summarized in Table Note that the masked modeling loss (Equation (5)) is computed the latent space between the latents of the history (or future) frames and the predictions of the decoders.
D.4 INFERENCE
The sampling algorithm is summarized in Algorithm 2 Inference hyperparameters are summarized Table 10 The inference process is the same as in other diffusion-based video generation models (Song et al. 2025 Gu et al. 2025 Hu et al. 2024a): Given initial conditions (initial frames for video prediction, class labels for cass-conddtioned generation, or no condition for unconditioned generation), the model iteratively denoises the next frame starting from pure noise, and appends the generated frame after the known (provided as initial conditions or generated) frames until all frames known.
23

---

Published as a conference paper at ICLR 2026
Algorithm 2 Sampling (v-prediction or flow matching)
Input: Noise schedule {(ar, dr))r, sampling steps N, velocity network vo, initial frames 21:0o (2
if ln 0)
Output: Clean frames x1:7
1: for = lo + to do
2: 2t (0, 1)
Initialize with noise at TN
3: for i = N to do
4:
Upred - vo (2i;; Ti, (1:-11 24 f- Sampler (21; Ta, Ti-1; Upred)
Internally apply block-causal attention mask // Sampler step, To = 0
5:
6:
end for
7: 21:t 21a-1 + {
I/ Append generated frame after known frames
8: end for
D.5 HISTORY GUIDANCE
We incorporate simplified version of history guidance (Song et al. 2025 into diffusion-basee VideoAR. History guidance takes advantage of the insight that history frames are the conditions for generating the future frames, much like class labels conditions for class-conditioned generation,
and applies classifier-free guidance (CFG) (Ho & Salimans 2022) with history frames as conditions.
Adopting the notations in Section 2 history guidance modifies the score function as
$ø (o (T)
(1 w) so (x (T) T, Ø) + so (1
(27)
the guidance seale. We compute so(x Ø) by forbidding (x(T) attend Iis via attention masking, i.e., by setting the corresponding noisy-to- clean rows in the attention mask (Figure 3 a)) to -00.
where means no history frame and w
During training. we randomly select r 10% future frames and forbid them from attending to the history frames. The training algorithm with history guidance is summarized in Algorithm
During inference, sy / (x(=); Ø) is computed by Equation (2) and the other process is the same as in normal CFG. The sampling algorithm with history guidance is summarized in Algorithm
E BASELINES
In our work, we primarily consider three baseline methods in Figure [ and Section 3.2 All the considered baselines trained with the same model architecture and hyperparameters shown in Table 10 unless otherwise specified, with the only difference being the training strategies.
ACDIT ACDiT (Hu et al. 2024a) also adopts complete teacher forcing in MiMo. The primary difference between MiMo and ACDiT that we apply the masked history modeling target on the history frames. Thus, the direct comparison between ACDiT and MiMo clearly demonstrates the advantage of our approach and the benefit of good history representations.
FAR FAR (Gu al. 2025) adopts diffusion foreing (Chen al., 2024a Song al. 2025) randomly replaces some noisy frames with their clean version to simulate clean history frames. The better performance of MiMo over FAR demonstrates that MiMo can achieve competitive perfor- mance even against the best performing models in broader context.
REPA REPA Yu et al. 2024) was originally proposed improve the representation quality of the noisy images being denoised. We adapt it to diffusion- based VideoAR following the approach Zhang et al. (2025). Compared with REPA, we focus on the representations of the history frames that serve conditions in AR modeling, while REPA does not consider the AR context. Also, REPA requires a VFM, but s vVM may not always be available and may misbehave for out-of diitr---ittiibttionn (OOD) data, while MiMo does not rely VFM.
For the analysis in Section 3.2 we align the features of the 4-the layer with a pretrained VideoMAE- I. (Tong et al. 2022 using loss weight of 0.5. The feature dimensions of DiT models and Video-
24

---

Published as a conference paper at ICLR 2026
Algorithm 3 Training with history guidance (v-prediction flow matching)
Input: Dataset D, noise schedule {(ar, 07) velocity network tg, decoder Po, loss weight X,
drop rate r
Output: Trained velocity network v0
1: while not converged do
2: Sample video clip x {x,]/=1 from D
// Duplicate x as history h and future f 4: Sample {Ta Uniform[0, 1)}2- and {a ~ (N(
3: h 4 X, x
5:
Sample random tube mask M
6:
lnd +- applyMask(h1.
Apply M to history frame
7:
C+0
8:
for = to do
9:
f(T)) fi + o,C
I Add noise future frame |/ v-prediction or flow matching
10:
Uarget + Et - 7r or Ulrget f òzefe + oG
11: if Uniform(0, < then
12:
Upeed - Up (f (T) 1T,
I/ Randomly drop history frames
13:
else
14:
Uprod c (
;T, 11)1))
// Internally apply attention mask (Figure a))
end if
16:
Ldift + l|upeed Utargot 11
|/ Diffusion loss (Equation
Vet +- 11 Ad (5(() TT, k1
h.1
Output features of the I-th layer for history frames h1:t // Frame indexes // Masked history modeling loss
18:
To - it,t 1}
Laskk
Tel Ever, beke pp L, (2d (
(Equation (5)
20:
C L Laift + ALmask
21:
end for
22:
Update 9 using VoL
23:
end while
Algorithm 4 Sampling with history guidance (v-prediction or flow matching)
Input: Noise schedule {(a,, o,)),, sampling steps N, guidance scale w, velocity network ug. initial frames Tidt, (Ø if to = 0)
Output: Clean frames T1:T
1: for t = to + to T do
2: 2t ~ I).
Initialize with noise at TN
3: for N to do
4:
Ti, (1:t-111 Upred f up (201 (2,1)
Internally apply block-causal attention mask
5:
Ug
(174 Tay 2)
// Negative condition Apply guidance I/ Sampler step, To
6:
Upred - (1 w) Ug + w Urddd
7: 21 - Sampler (2yi Tt-1, Upred)
8:
end for
9:
Fidt 21:-1 {xt)
# Append generated frame after known frames
10:
end for
MAE are aligned following the strategy of Zhang et al. 2025) ) which interpolates the DIT's repre- sentations match the feature dimensions the pre-trained VideoMAE.
F EVLAUTION DETAILS
Fréchet Video Distance (FVD) FVD (Unterthiner et al. 2018) a perceptual metric designed to evaluate the quality of generated videos by measuring the distributional distance between real and generated video sequences. Similar to the Fréchet Inception Distance (FID) (Heusel et al. 2017) used for images, FVD employs pre-trained 3D convolutional neural network (specifically, an Inflated 3D ConvNet or I3D model trained on Kineticy-400) (Carreira Zisserman 2017) to extract spatio-temporal features from video clips. Then Fréchet distance (Dowson Landau 1982
25

---

Published as a conference paper at ICLR 2026
is computed between the feature distributions of real and generated videos by fitting multivariate Gaussian distributions to the extracted features and calculating the Wasserstein-2 distance (Villani
et al. 2008 between them. Lower FVD scores indicate higher similarity to real video distributions. FVD captures both spatial and temporal aspects of video content, making it a standard evaluation tool in video synthesis research. Following prior works (Song et al. (025), compute FVD for the entire video, including both initial conditioning frames (for Kinetics-600 6600) and generated frames, assess the overall consistency.
Centered Kernel Nearest-Neiighor Alignment (CKNNA) CKNNA (uuh et al. 2024) is a non- parametric evaluation metric that measures the alignment between two sets of features by analyzing their local neighborhood structures. CKNNA relaxes the overly rigid Centered Kernel Alignment (CKA) (Krnblith et af., 2019) metric by computing similarity only for the mutual nearest neigh- bours of each feature vector. Given two sets of vectorized features {Ø; R²) and f&r E [R") from two models and inner product operator (,, -), CKNNA first computes centered kernel matrices:
Kij = (Øs, o3) (oi, di)], Loj (v,, w5)) - [ thr)]
(28)
The centering operation removes the mean similarity, focusing on relative relationships rather than absolute magnitudes. CKNNA restricts the alignment computation to mutual nearest neighbors:
Alignkan (K, = ΣΣ a(i.)) KigL.s
(29)
where ( E knn())) A 4/ knn( A
(30)
The indicator function a(i.j) ensures we only consider sample pairs whose members are nearest neighbors to each other, emphasizing local structural consistency over global alignment. The final CKNNA metric is the normalized version:
Ali (K, L)
CKNNA(K, L) VAlignkon (K, K) Alignknn (L,
(31)
This normalization bounds the metric to [0, 1], where higher values indicate better preservation of local neighborhood structure between the two representation spaces. Intuitively, CKNNA measures whether two feature representations maintain similar local similarity structures within their respec- tive neighborhoods. Following prior works (Huh et al. 2024 eu al.) 20224,, evaluate represen- tation alignment using CKNNA with k = 10 nearest neighbors. We randomly sample 10,000 videos from the Kinetics-600 test and extract globally average pooled features using both pretrained VideoMAE-L (Tong et al. (002)) (as reference) and our models, treating all frames as clean history frames. Then, we compute CKNNA between the features of VideoMAE-L and features from each layer of the query models, reporting the highest alignment score across all layers.
Linear Probing We follow the linear probing protocol of MAE (He et al. 2022a). Specifically, we use the model representations of the clean history frames for linear probing training and evaluation. Global average pooling is applied to the output feature map to obtain a single feature vector for each video. The feature vector is then fed to parameter-free BatchNorm (loffe & Szegedy 2015) layer and a linear classifier layer. The training batch size is 128, the learning rate is 10 and decayed to 0 with cosine schedule, the weight decay is 0.01, and the training length is 10 epochs. Random flipping is used during training. Top-1 accuracy is reported.
G ADDITIONAL VISUALIZATIONS
G.1 SAMPLES
This section shows the samples generated by MiMo on Kinetics-600 (Figure 6. UCF-101 class- conditioned generation (Figure77, 77). and unconditional generation (Figure 8). Each row is generated video containing 16 frames.
G.2 ATTENTION HEATMAPS
In Figures and 10 we show the attention heatmaps (marked by red) of two videos, each without and with MiMo. The center position of the last frame (marked by blue dot) serves the query,
26

---

Published as a conference paper at ICLR 2026
Figure 6: Uncurated samples of Kinetics history frames conditional generation.
Figure 7: Uncurated samples of UCF-101 class-conditioned generation.
27

---

Published as a conference paper at ICLR 2026
9
3
Figure 8: Uncurated samples of UCF-101 unconditional generation.
while the other positions are the keys. We take the attention weights from four random heads and layers 4, 8, 12, 16, 20.
As shown in FiguresPJand without MiMo, attention patterns are more dispersed and less focused, whereas with MiMo, attention heatmaps show more concentrated patterns that exhibit stronger se- mantic correlations with the query content. Additionally, Figure I00 demonstrates how different transformer layers specialize in matching distinct body parts (e.g., arms, torso, legs), revealing the hierarchical nature of the learned representations.
G.3
EMBEDDING VISUALIZATION
Figure I1 shows the UMAP MccInees et al. 2018) visualization of video embeddings without and with MiMo. The model is DYT-XL and trained for 360K steps on Kinetics-600. As shown in Figure I] without MiMo, the distribution of video embeddings is mostly uniform, while MiMo in- troduces some structures in the embedding distribution by learning more structured representation space.
H ADDITIONAL EXPERIMENTS
H.1 COMPLEMENTARY EVALUATION METRICS
Our evaluation mainly relies on the standard FVD metric. However, FVD known to have sev- eral issues and may not fully capture real-world video dynamics (Luo et al. 2024). To provide more comprehensive evaluation of MiMo's performance, we adopt two additional metrics as com- plements.
VMMD Our VMMD (V-JPPA 2 Maximum Mean Diserepancy) metric is based on the CMMD metric (Jayasumana et al. 2024). The VMMD metric benchmarks the perceptual similarity between
28

---

Published as a conference paper at ICLR 2026
Head
Head
Head 3
Head
Layer 4
Layer
Layer 12
Layer
Layer 20
w/o MiMo
Layer 4
Layer
Layer 12
Layer
Layer
w. MiMo
Figure 9; Attention heatmaps without and with MiMo.
Head
Head 2
Head 3
Head
Layer 4
Layer
Layer 12
Layer
Layer 20
w/o MiMo
Layer 4
Layer
Layer 12
Layer
Layer
W. MiMo
Figure 10: Attention heatmaps without and with MiMo.
29

---

Published as a conference paper at ICLR 2026
(a) MiMo
(b) with MiMo
Figure 11: UMAP visualization of video embeddings without and with MiMo.
the generated videos and the reference videos, using the strong V-JEPA 2 (Assran et al. 2025) pretrained model the judge. It does not rely the Gaussian assumption of the FVD metric, and gives more faithful evaluation (Jayasumana et al. 2024). Specifically, VMMD replaces the CLIP model in CMMD with V-JEPA 2 Large; other implementations the same in CMMD
Table 11: Comparison of different methods with VMMD and FVD metrics.
Method
ACDiT-XL
FAR-XL
MiMo-XL
FVD4
10.264
9.311
8.257
VMMD
1.075
1.036
0.977
Table TI compares ACDiT, FAR, and MiMo with both VMMD and FVD metrics. The VMMD measurement results are consistent with the FVD, indicating that in our cases, FVD and VMMD can relatively well characterize the generation quality.
ACDIT Wins
Ties
MiMo Wins
FAR Wins
Ties
MiMo Wins
Condition Following
Condition Following
Motion Quality
45.2%
48 88.4%
Motion, Quality
12.1%
45.5%
42. 22.4%
Quality
Visual
12.9%
67.7%
19.4%
Visual Quality
15.2%
18.2%
40%
Win Rate (%)
Win Rate (%)
(a) ACDIT vs MiMo
(b) FAR vs MiMo
Figure 12: User studies (win rates) of ACDiT, FAR, and MiMo.
We conduct user studies to better understand what aspects MiMo improves. Five experts are instructed to evaluate 100 tasks, assessing three key dimensions: condition following, motion quality, and visual quality.
User Studies
Condition following: the visual and semantic consistency between conditioning frames and generated frames.
Motion quality: whether there is motion distortion or motion that is semantically inconsis- tent with the context.
httpp:://ii
30

---

Published as a conference paper at ICLR 2026
Visual quality: whether there is frame-level visual distortion visual components that semantically inconsistent with the context.
Figure summarizes the results for ACDIT-XL, FAR-XL, and MiMo-XI, all trained for 360K steps on Kinetics-6000 MiMo excels in condition following and motion quality, while the visual quality marginally improved. Additionally, MiMo has higher win rates against ACDiT than against FAR, which indicates that lower VMMD and FVD metric values correlate with better perceptual quality our cases.
H.2 LONG-HORIZON VIDEO GENERATION
MiMo effective for long-horizon video generation, as its robust history representation helps mit- igate the error accumulation common in autoregressive models. We validate this on the action- conditioned Minecraft dataset (Yan et al. 2023). predicting 156 frames from 144, following the FAR (uu et 2025) setup.
The results in Table 12 show that MiMo signifi- cantly outperforms the baseline. At 100K train- ing steps, MiMo achieves an rFVD of 33.829, 27% improvement over the baseline's 's 42.710. This performance gap highlights MiMo's abil- ity to maintain long-term coherence. Fur- thermore, MiMo accelerates training, reaching this performance 1.5x faster than the base- line. which requires additional 50K steps to achieve a comparable rFVD. Figure 13 shows uncurated samples of generated results. These results confirm that superior understanding of the past, enforced by MiMo, leads to more plausible and coherent long-term video generation.
Table 12: Long-horizon video generation the Minecraft dataset.
Model
Steps
FVD1
100K
42.710
FAR-B
150K
33.873
MiMo-B
100K
33.829
Figure 13: Uncurated samples of Minecraft long-horizon video generation. The upper row of each video is GT, the lower row is the generated sample. Red bounding boxes indicate conditioning frames.
31

---

Published as a conference paper at ICLR 2026
I LIMITATIONS AND FUTURE WORK
In this work, we analyze the impact of the DiT's internal representations of history frames on VideoAR. Based on our findings, we propose MiMo improve history representations without utilizing VFM.
However, remains open question improve future frame representations with VFM. Masked DiT (Gao et al. 2023 Wei et al. 2023) achieves success to some extent but requires elaborate architecture modifications. Some recent approaches (Jiang et al. 2025 Wang He 2025) incor- porate methodologies from self-supervised learning literature, but it still unclear whether they (and MiMo) can beat representation alignment approaches (e.g.. REPA Yu et al., 2024)) that utilize pretrained VFM in-distribution data of VFM. It is also unclear whether it is possible to pre- train a generative model that beats VFMs in downstream tasks such as video segmentation, video grounding, etc.
Furthermore, an interesting future direction to explore other training objectives to improve history representations (Oquab et al. 2023; Assran al. 2023 Jiang al., 2025 Wang & He. 2025).
32


Reviews:
[{"review_id": "BF9LbsqtXf", "reviewer": "Reviewer_iS7b", "summary": "This paper studies the role of history frame representations in diffusion-based autoregressive video generation (VideoAR). The authors propose MiMo (Masked History Modeling), which integrates masked modeling into VideoAR to explicitly improve history frame representations during training. By applying masks to history tokens and reconstructing both current and next-frame tokens alongside the diffusion objective, the model learns more predictive and robust history representations without relying on pretrained vision foundation models. Experiments on Kinetics-600 and UCF-101 show improved FVD scores over prior autoregressive methods.", "questions": "1. How sensitive is MiMo to the masking ratio and λ parameter? Have the authors evaluated performance stability under different settings?\n\n2. Could the authors compare MiMo with a simpler auxiliary loss such as contrastive or reconstruction-only objectives to rule out general regularization effects?\n\n3. How much additional computation does masked history modeling introduce during training?\n\n4. Have you tested whether pretraining MiMo representations improves downstream video understanding tasks to support the “representation” claim?\n\n5. Could qualitative examples (e.g., failure cases or visualization of history embeddings) clarify what MiMo learns beyond the diffusion baseline?\n\n6. Would MiMo still help when combined with pretrained VFMs or under longer video horizons?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, with clear figures and reasonable organization.\n\n2. The paper identifies an underexplored question: how internal history representations affect VideoAR quality, and provides empirical analysis supporting its importance.\n\n3. MiMo introduces a relatively lightweight modification to standard diffusion-based AR training, avoiding reliance on large pretrained models.\n\n4. Results are reported on multiple datasets and compared with several baselines, including both AR and non-AR methods.", "weaknesses": "1. The proposed masked history modeling is conceptually straightforward and closely resembles masked autoencoding and REPA-style representation regularization. The integration into VideoAR feels incremental rather than fundamentally new.\n\n2. The ablations are narrow in scope. It remains unclear why the proposed loss improves results beyond general regularization or increased training signal. Deeper analysis of representation quality (e.g., visualization or qualitative dynamics) would strengthen the claim.\n\n3. While the reported FVD gains are notable numerically, improvements are modest considering the added training complexity. Moreover, comparisons to strong non-AR diffusion models are missing.\n\n4. Several design choices are only briefly described. Implementation details may be insufficient for faithful replication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iKZcbSPAcn", "reviewer": "Reviewer_qvhL", "summary": "This paper investigates the role of history frame representations in autoregressive video generation (VideoAR). The authors first empirically demonstrate a positive correlation between the quality of history representations and the model's prediction performance. Based on this insight, they propose MiMo (Masked History Modeling), a novel framework that integrates a self-supervised masked modeling objective into a diffusion-based VideoAR model. During training, MiMo applies masks to clean history frame tokens and tasks the model with reconstructing the masked content of current and future frames, in parallel with the primary diffusion denoising objective. This dual-objective approach is designed to learn robust and predictive history representations without relying on external Vision Foundation Models (VFMs) or significant architectural changes. Experiments on video prediction and generation tasks show that MiMo achieves competitive or state-of-the-art performance while improving training efficiency.", "questions": "The ablation study in Table 4 compellingly shows that predicting both the current + next frames as the masked modeling target yields better results than predicting the current frame alone. This leads to an interesting question: have the authors explored if an even more complex prediction target could further enhance performance? For instance, would a target of current + next + next_after_next continue this positive trend, or does it lead to diminishing returns or potential training instability?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-motivated and logically structured. A key strength is the preliminary analysis that systematically establishes the link between history representation quality and VideoAR performance, providing a solid foundation for the proposed method. The MiMo framework itself is an effective solution that integrates representation learning with the diffusion process. The method's ability to achieve strong results on multiple benchmarks, such as Kinetics-600 and UCF-101, without the need for pre-trained VFMs is a significant advantage, making it more practical and self-contained. The comprehensive ablation studies further validate the authors' design choices, particularly regarding the masked prediction targets.", "weaknesses": "While the results are strong, the evaluation could be perceived as having some limitations. The experiments are primarily conducted on the Kinetics-600 and UCF-101 datasets, with Fréchet Video Distance (FVD) as the main metric. Although these are standard benchmarks, they may not fully capture the complexity and diversity of real-world video dynamics. Expanding the evaluation to include longer-form videos, more complex scenes, or complementary metrics such as user studies could provide a more comprehensive assessment of the model's capabilities in maintaining long-term coherence and semantic consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c8K6DyEueZ", "reviewer": "Reviewer_5Lni", "summary": "This paper proposes the **MiMo framework**, which enhances video autoregressive (AR) generation by improving the quality of historical frame representations through **masked history modeling**. Without relying on additional vision foundation models or complex architectures, MiMo achieves significant improvements in both video prediction and generation tasks.", "questions": "See weakness above", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper innovatively applies a **mask modeling strategy** similar to MAE within an autoregressive video generation framework.  \n- The proposed approach yields a **consistent and stable improvement** in AR video generation quality compared to existing methods.", "weaknesses": "- The experimental results show that the final performance of MiMo is **comparable to REPA**. It would be important to verify whether MiMo can further improve upon REPA, rather than leaving this question for future work.  \n- The experiments suggest that **next-frame prediction** contributes more to performance improvement, which seems **inconsistent with the paper’s main claim** that enhancing historical frame representations is the key factor driving the gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OVXc9DTjRH", "reviewer": "Reviewer_xwwk", "summary": "This work proposes MiMo, an autoregressive video generative model unifying both self-supervised learning in history frames and diffusion learning in future frames. MiMo is proposed under the sufficient findings that history representation (latent of history frames in DiT blocks) positively correlates with the performance of VideoAR, and is designed to learn history frame representation and generation capability simultaneously. MiMo conducts some experiments to prove the performance and training efficiency of MiMo. Despite these contributions, there remain some problems to be explained: 1.  In this work, the only generation quality evaluation metric used is the FVD (Fréchet Video Distance), and the only generation task is class-to-video, which is not enough to prove the advantage of MiMo. Moreover, FVD is known as unstable. 2. As shown in Table 3, MiMo seems to be inferior to REPA, another representation learning method, yet MiMo claims SOTA in VideoAR. Additionally, comprehensive comparisons with other works that also unify understanding and generation are needed.\n\nIn summary, while MiMo insightfully emphasizes the importance of history frame representation learning, its purported advantages require more comprehensive experiments and broader benchmarking to be convincingly established.", "questions": "1. What is the additional computational overhead introduced by the two-objective optimization in Equation 6?\n2. Have you considered using TTUR (Two-Time-Scale Update Rule) to mitigate this extra computational burden?\n3. Why does the evaluation rely solely on FVD and the class-to-video generation task?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This work proposes MiMo, an autoregressive video generative model unifying both self-supervised learning in history frames and diffusion learning in future frames. MiMo is proposed under the sufficient findings that history representation (latent of history frames in DiT blocks) positively correlates with the performance of VideoAR, and is designed to learn history frame representation and generation capability simultaneously. MiMo conducts some experiments to prove the performance and training efficiency of MiMo.  And this paper is well written and easy to understand.", "weaknesses": "1. The evaluation relies solely on FVD (Fréchet Video Distance) as the generation quality metric and focuses exclusively on the class-to-video generation task, which is insufficient to substantiate MiMo’s claimed advantages. Moreover, FVD is known to be unstable and may not reliably reflect perceptual quality.\n2. As shown in Table 3, MiMo underperforms compared to REPA, another representation learning approach,  yet it claims state-of-the-art (SOTA) performance in VideoAR. Additionally, comprehensive comparisons with other works that also unify understanding and generation are needed.\n3. More ablation and discussion are required: (1) Ablation study: with vs. without history representation learning. (2) Discussion and ablation about $\\lambda$ in Equ. (6). (3) Discussion on additional computational burden.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)