MarDini: Masked Auto-regressive Diffusion for Video Generation at Scale
Authors: Haozhe Liu
Shikun Liu
Zijian Zhou
Mengmeng Xu
Yanping Xie
Xiao Han
Juan Camilo Perez
Ding Liu
Kumara Kahatapitiya
Menglin Jia
Jui-Chieh Wu
Sen He
Tao Xiang
Jürgen Schmidhuber
Juan-Manuel Perez-Rua
Venue: Accepted by TMLR ()

Abstract: We introduce MarDini, a new family of video diffusion models that integrate the advantages of masked auto-regression (MAR) into a unified diffusion model (DM) framework. Here, MAR handles temporal planning, while DM focuses on spatial generation in an asymmetric network design: i) a MAR-based planning model containing most of the parameters generates planning signals for each masked frame using low-resolution input; ii) a lightweight generation model uses these signals to produce high-resolution frames via diffusion de-noising. MarDini’s MAR enables video generation conditioned on any number of masked frames at any frame positions: a single model can handle video interpolation (e.g., masking middle frames), image-to-video generation (e.g., masking from the second frame onward), and video expansion (e.g., masking half the frames). The efficient design allocates most of the computational resources to the low-resolution planning model, making computationally expensive but important spatio-temporal attention feasible at scale. MarDini sets a new state-of-the-art for video interpolation; meanwhile, within few inference steps, it efficiently generates videos on par with those of much more expensive advanced image-to-video models.

Published in Transactions on Machine Learning Research (05/2025)
MarDini: Masked Auto-regressive Diffusion for Video Gen- eration at Scale
Haozhe Lix Shikun Liu Zijian Zhou?., Mengmeng Xu², Yanping Xie 2 Xiao Han", Juan C. Pérez", Ding Liu", Kumara Kahatapitiya', Menglin Jia", Jui-Chieh
Wu², Sen He Tao Xiang",
Jürgen Schmidhuber', Juan-Manuel Perez-Rua
KAUSTT
Meta AT
Equal Contribution
Correspondence: haozhe. Luukausst. edu. sa; jmprêmeta. com
Reviewed on OpenReview: https: // openreview. net/ farmm? id= fuOHI59rUN
Abstract
We introduce MarDini, a new family video diffusion models that integrate the advantages of masked auto-regression (MAR) into unified diffusion model (DM) framework. Here, MAR handles temporal planning, while DM focuses spatial generation in asymmetric network design: i) MAR-based planning model containing most of the parameters gener- ates planning signals for each masked frame using low-resolution input; ii) lightweight gen- eration model uses these signals to produce high-resolution frames via diffusion de-noising. MarDini's MAR enables video generation conditioned on any number of masked frames at any frame positions: a single model can handle video interpolation (e.g., masking middle frames), image-to-video generation (e.g., masking from the second frame onward), and video expansion (e.g., masking half the frames). The efficient design allocates most of the compu- tational resources the low-rcoolutinn planning model, making computationally expensive but important spalio-temppral atlention feasible at scale. MarDini sets a state-of-the art for video interpolation; meanwhile, within few inference steps, it efficiently generates videos on par with those of much more expensive advanced image-to-video models.
1 Introduction
Aulo-segressive (AR) models VVaswani el. al. 2017 Peng el al. 2023
Schmidhuber 1992b Schlag el. al.
2021) have recently demonstrated remarkable success in natural language processing (ubeyy al. 2024
Team et al.
2023 Achiam et al. 2023), sparking efforts to achieve similar breakthroughs in computer vision
He et al.] 2021 Deng al.) 2024 Shi al. 2024 Wang al.) 2024d Tian et al. 2024 However, uilike the discrele, sequential, and easily tokenized nature of language, visual data consist. of
(Sun et al.
2024
continuous pixel signals distributed across high-dimensional space, making them more difficult to model
through 1D auto-regression.
To overcome this challenge, recent studies have explored vector quantization techniques (Van Den Oord et al. 2017 Razavi et al. (0199) to convert continuous pixel data into discrete representations suitable for AR modelling. Unfortunately, these approaches (Yu et al. 2022 Ramesh et al. 2021 rely cousal ottention. which is not well aligned for high oigh-dimensional visual data, often leading diminished performance (Li et al, 2024), particularly on large-scale datasets (Xie et al. 2024 Zhou et al. 2024). To mitigate this limitation, masked auto-regression (MAR) has been introduced (Chang el al. 2022 Li et al. 2023a). MAR replaces the causal attention with bi- bi-directional attention (He et al. 2021 Devlin et al. 2019), effectively simulating auto-regressive behaviour while being more capable of handling visual data. Leveraging this approach, MAR exhibits flexibility in handling diverse generation tasks through different masking strategics, such as image al. 2023a), out-painting (Chang al. 2022 video expansion
generation (Chang et al. 2022
Li
Yuu

---

Published in Transactions on Machine Learning Research (05/2025)
et al. 2023a) and class-conditioned video generation (uu et al. 2024
Voleti et al. 2022) while maintaining
manageable computational overhead. Although MAR shows potential in scaling image and video generation tasks (Chang et al.) 2023 Yu et al. 2023a 2024), its key bottleneck lies in its training instability and weak
performance which is tied to the reliance on discrete representations Ramesh et al. 2021 Razavi et al.
2019
Li al. 2024).
Meanwhile, Diffusion models (DMs) (Ho al. 2020: Neal 2001 Jarzynski 1997) have emerged as a successful
alternative for scaling vision generative models, offering stable training by modelling visual signals directly in continuous space. However, DMs tend to incur high inference costs due to the requirement of the multi-step diffusion process. Here, vidco generation poses an even greater challenge - Video is striet super-set of the image domain, requiring additional modelling for temporal consistency and complex motion dynamics.
To this end, we propose a new paradigm for video generation that combines the flexibility of MAR in a con- tinuous space with the robust generative capabilities of DM. Specifically, we present a scalable training recipe and an efficient neural architecture design for video generation. Our model decomposes video generation into two sub-tasks temporal and spatial modelling handled by distinct networks with an asymmetric design based on the following two principles:
1. MAR handles long-range temporal modelling, while DM focuses on delailed spotial modelling.
MAR operates with more parometers at a lower resolution, while DM operates with fewer porameters at higher resolution.
Following these principles, use the same training batch for both MAR and DM but employ two distinct processes operating at different resolutions. MAR. receives randomly masked low-rcsolution input frames and predicts the corresponding planning signals. Conditioned on these planning signals via cross-atlention and the unmasked frames, DM learns to incrementally recover the masked high-resolution frames from noise. Finally, we introduce a progressive training strategy that gradually curates mask ratios and with its data pipelines, allowing our model be trained from scratch on unlabeled video data. This eliminates the common reliance text-to-image and text-to-video pre-training, as in other video diffusion models (Girdhar
et al. 2023 Blattmann et al. 2023a
Our model integrates MAR-based planning signals with a DiT-based (Peebles & Xie 2023 Chen et al. 2024c lightweight, tiny diffusion model, hence the name MarDini. Our empirical study on Mar Dini highlights the following key characteristics:
Flexibility. With MAR conditioning, MarDini naturally supports a range of video generation tasks through flexible masking strategies. For example, when given the first frame and masking the rest, it performs inage-to-video generation; when given a video and masking subsequent frames, it performs video expansion; and, when given the first and last frames and masking the middle frames, it performs video interpolation. By hierarchically and auto-regressively masking middle frames across multiple inferences, MarDini generates slow-motion vidcos.
Scalability, MarDini can be trained from scratch at scale, without relying on generative image-based pre-training. This approach enables the model to scale from video interpolation to full video generation, directly bypassing the need for image-based based pre-training. Such observation may accelerate the prototype verification for new video generation models.
Efficiency. MarDini's asymmetric design allocates more computational resources to lower resolutions, making it memory-efficinnt and fast during inference. With lower overall memory usage, MarDini allows the deployment of computationally intensive spatio-temporal attention mechanisms at scale, improving its ability to model complex motion dynamics.
2

---

Published in Transactions on Machine Learning Research (05/2025)
Masks
Planning Model [MAR, heavy-weight)
Low-Res. Videos
Masked Low-Resolution Latents
VAE Encoder
Generation Model [Diffusion, light-weight)
Masked Diffusion Loss
Add Noises
High- Res. Videos
Noised High Resolution Latents
Figure 1: MarDini Training Pipeline Overview. A latent representation is computed for unmasked frames that serve conditional signal to generative process. On the first hand, have planning model that autoregressively encodes global conditioning signals from low-resolution version of the unmasked latent inputs. On the other hand, the planning signals are fed to the duffussionbasee generation model through cross-attention layers. A high-resolution version of the input conditions is also ingested by the diffusion model, cuabling generation with coherent temporal structure and direct mechanism to attend to fine-grained details of the unmasked frames. MarDini is trained end-to-end via masked frame-level diffusion loss. Note the frame approximate proxy for clearer
that the latent code extracted by the VAE is highly compressed; visualization.
2
MarDini: An Efficient and Asymmetric Video Diffusion Model
2.1 Design Overview
MarDini is video generation model designed to efficiently generate high-rcsolution vidcos using asym- metric network architecture. As shown in Figure I MarDini consists of two networks: heavy-weight MAR planning model and light-weight generation DM. During training, the planning network processes ran- domly masked low-resolution frames and predicts corresponding planning signals. These planning signals compress the semantic and long-range temporal information, guiding the DM's high-resolution generation process. The DM receives noisy frames at the masked positions and reconstructs them by progressively removing noisc.
In this section, outline and address the key design challenges involved in training MarDini. First, we describe the data representations and their corresponding notations within the MarDini framework (Sec- tion 2.2 Next, we describe the design details of the MAR planning network and the DM, along with the integration of additional guidance such as diffusion steps and planning signals (Section 2.3) Finally, we out- line the multi-stage training recipe for MarDini, which we found to be essential for ensuring stable training (Soction 2.4). Collectively, these innovations enable MarDini to bocome one of the first video generation models capable of being trained from scratch using only unlabelled vidco data.
2.2 Data Representation and Notations
VAE Compressor. Consistent with prior works (Dai 2023a Girdhar al., 2023). we adopt a pre- trained Variational Auto-Encoder (VAE) (Kingma & Welling 2014), denoted by Done; compress videos into low-dimensional continuous latent space, which improves both training and inference efficiency, Our VAE employs 16-channel latent dimension with an 8x spatial compression rate to preserve spatial details, following Dai et al. (2023a). The VAE outputs are then patchified into shape of N x C, where represents the token count and C = 16 represents its latent dimension.
MAR Planning Model. Given a low-resolution input video Xjow = {x/ low |==1 with K frames, we apply the VAE encoder to compress the frames into their corresponding latent representations: Ziow [2) (2low) Ji=li Deno(X1ow) To train the MAR planning model P, we randomly select K' < K video latents {2f0")))1:st j===kk' Ziow and replace them with learnable mask token [MASK], resulting in the final masked low-resolution latent inputs Z The planning model then processes Z and prodicts Zoond = P P(Zmaak = {2fond) ) )== where goodd is the planning signal for the i-th frame, shaped as Now Clow, with Now representing the number of patches per frame.
3

---

Published in Transactions on Machine Learning Research (05/2025)
Planning Signal
MLP
Scale
MLP
RMSNorm
MLP
MLP
RMSNorm
Diftusion Timestep
Multi Heiti-Heed Attention
RMSNorm
Temporll Multi Head Attentian
PosEma
RMSNorm
Temporal
PosEmb
Multi Multi-Head Attantion
RMSNorm
Spatial Multi- Head Attentian
PosEmb
Scale
Spatial Multi-Head Attention
PosEmb
RMSNorm
Scale, Shift
RMSNorm
X L
L2
Diffusion
Timastop
Masked Low-Ret Latents
Naised High-Res. Latents
Planning Model (MAR)
Generation Model (DM)
Figure 2: MarDini Design Details. MarDini employs a transformer architecture for both the planning and generation models, incorporating a DiT-style block for the generation model and a Llama-style block for the planning model. We set J, >> T22, where b and L- refer the number layers in the planning and gencration model respectively. The "Diffusion Timestep" below matches the one above, and we draw them separately avoid line crossings and interference for clarity.
DM Generation Model. Conversely, we obtain high-resolution video latents Zhigh Dene (high) with dimensions Nuigh X Chigh: generated by the VAE encoder using the same vidco inputs at high resolution: Xnigh = -x: high {2.high Ji=i:K. Notably, we have Nnigh > Noow At diffusion step 1, we sample noise and add it to K' frames that were masked in the planning model (denoted by [NOISE]), leaving the remaining K K' reference frames unchanged (denoted by [REF]). This produces the final noisy high-resolution video latent inputs noise, Then, the generation model processes these latent inputs nooise, and performs a standard denoising step, where denote the DM output at time step G (ZZmoion, Zeond;
2.3
Architecture Design
In this section, we provide a comprehensive explanation of the MarDini architecture, including its detailed design, model configurations, and variations.
2.3.1 MarDini Block Design
Figure 2 illustrates the design of the MarDini's MAR and DM models, both of which are based on the transformer architecture (Vaswani et al. 2017).
In the MAR planning model. we adhere to the design conventions established in Llama models Dubey
et al.] 2024 Touvron et al.] 2023), which apply RMS-Norm (Zhang & Sennrich 2019) to normalize the

---

Published in Transactions on Machine Learning Research (05/2025)
inputs of each attention block. Additionally, layer normalization (Ba et al. 2016) is applied to normalize the projected features in multi-head attention, further enhancing training stability. Due to the use of low- resolution imputs, we manage to directly employ spatio-temporal attention, allowing tokens to attend across frames. This design feasible only with asymmetric resolution inputs, as it prevents excessive memory consumption during training.
Concretely, within each attention block in MAR, we utilize rotary positional encoding (RoPE) (Su et al, 2024) to encode both the spatial and temporal positions of the video tokens. To accomplish this, apply 2D RoPE to encode the 3-dimensional video data. Specifically, we flatten the image patches into 1-dimensional token sequence and insert Icarnable [NEXT] token to differentiate image patches across different rows, akin to the [SEP] in Devlin et al. (2019) and [nextline] in Gao et al. (2024). This design effectively handles video data with varying aspect ratios and resolutions.
We design the DM model in alignment with MAR, but with three key differences. First, we adopt a DiT-style approach (Peebles & Xie 2023), using AdaIN (Huang & Belongie 2017) incorporate the diffussio steps conditional signal. Second, we introduce cross-attention layer to process the planning features predicted by the MAR model. Lastly, we replace spatio-temporal attention with temporal attention (Blattmann et al. 2023b) to reduce the computational cost associated with high-resolution inputs in DM.
2.3.2 Identity Attention
In our initial experiments, we observed significant training instability in MarDini's DM. We speculate that this is due to two main factors: i) the inherent distributional disparity between noisy ( (INOISE]) ) tokens and clean reference ( [[EFF] tokens, which is further amplified by the stochastic nature of sampling diffusion steps; and ii) the random positions and varying lengths of these [NOISE] tokens. These factors likely compound, potentially disrupting the DM's training signals and hindering the model's ability to converge efficiently.
To address this challenge, we introduce Identity Attentio,, which enables the model to easily distinguish between [REF] and [NDISE] tokens by employing separate attention strategy. As illustrated in Figure [REF] tokens simply serve as identity projection, preserving the input ref- erence frames without attending to other tokens. In contrast, [NOISE] tokens possess a global view, attending to tokens across all frames. The [REF] tokens serve as guidance for gencration, so we design them to be isolated from other tokens, while [NOISE] tokens provide global attention all conditional signals for generation. We incorporate Identity Atten- tion in both the spatio-temporal layers of MAR and the temporal layers of DM, which has been found to significantly enhance training stability in both models.
[NDISE]
[REF]
[NOISE]
[NDISE]
INOISE]
[REF]
[NOISE]
[NOISE]
Based on the above design, we present four model variants with distinct configurations, detailed in Appendix A
Figure 3: Identity Attention De- sign Details DM. In this setup, [REF] tokens only attend to them- selves, while [NOISE] tokens attend to all other tokens.
2.4 MarDini Training Recipes
In this section, we outline the training pipeline of MarDini. Specifically, employ multi-stage progressive training strategy that gradually in-
creases task difficulty. This approach offers two key benefits: i) progressive learning inherently enhances training stability and improves the performance of generative models, as demonstrated by Karras (2018) and Chen et al. (2024b); and ii) it allows for the collection checkpoints from earlier stages, which helps mitigate setbacks caused by suboptimal configurations. Below, we claborate on our detailed progressive training strategy, including the training objectives, architecture design, and training data configurations. A comprehensive training manual for MarDini can be found in the Appendix B
5

---

Published in Transactions on Machine Learning Research (05/2025)
2.4.1 Training Tasks:
From Frame Interpolation to Video Generation
Our training objectives are organized into three stages: i) Initial Stage: We separately train the planning and generation models, each with its own learning objective, to initialize their model weights. ii) Joint- Model Stage: We combine the models for joint training on simple video interpolation task, using only a masked diffusion loss. iii) Joint-Task Stage: We further train the model by gradually reducing the number of preserved reference frames, enabling it to jointly learn video interpolation and image-lo-videe generalion tasks.
Initial Stage. Wang et al. (2024a) pointed out that transformers with a large parameter count often experience unstable training. As such, we simplify the training dynamics by separately warming up the two models as an initial step.
To optimize generation model G, we employ masked diffusion loss LDM:
CBM = M V M Go no Zuneond d)1l
(1)
where Zuncond is a learnable token serving as unconditional guidance from the planning model. 0 represents the parameters of the generation model, and M denotes the binary masks used to mask out all clean reference frames. Inspired by Blattmann et al. (2023b): Salimans & Ho (022)). , we apply velocity prediction as the diffusion loss, where the prodiction target V frame, defined of o,E a.24, N(O, I). Here, Og and or correspond to the diffusson scheduler at step.
{or {of]==1KK represents the velocity at time step for the i-th
To optimize MAR planning model P, we employ masked reconstruction loss LMAR:
cak
M Low M (c (mank)] ||2
(2)
where f denotes a projection layer that depatchifies the model predictions to match the resolution of the low-rcsolution input image Zlow- Φ. S represent the lcarnable parameters of the planning model and the projection layer respectively. Note that, is only used during the initial training stage, and will be removed in the later training stages.
After the initial pre-training stage, we then jointly train the planning and generation models end-to-end using a unified masked diffusion learning objective LMDII:
Joint-Model Stage.
0 MD
M V - M Go (Zigh
( P (Z t)l12
where Zeond P(Z low
is the planning signal predicted by MAR. In order to enable classifier-free guidance (Ho& & Salimans 2022 on the planning signal, we maintain a fixed probability of 1/10 to randomly replace
Zoond with Zuncond-
Joint-Task Stage. In the final training stage, we reuse the learning objective from the previous stage, but gradually increase the masking ratio induce more challenging generation tasks. This stage requires significantly larger computational resources with higher-resolution videos, determines the model's final performance. By gradually increasing the masking ratios, smoothly transform the model's task from video interpolation to single-inagc-to-vide generation. This procedure ultimately enables the model to generate videos with variable number of input frames at arbitrary temporal locations.
2.4.2 DM Architecture: From Spatio-Temporal to Temporal Attention
In conjunction with our progressive training objectives, we also introduce progressive architectural design. Specifically, we first use spatio-temporal attention in the DM during the initial training stage. This choice promotes convergence, compared to temporal attention, as noted in Gao et al. (024). Since in our initial stage we train the DM in isolation and on a relatively low-resolution setup, this sophisticated attention incurs in minor computational overhead. When integrating MAR with the DM in the second stage, we replace the spatio-temporal attention with the more cost-cffective temporal attention, thus increasing the efficiency of the generation model.

---

Published in Transactions on Machine Learning Research (05/2025)
2.4.3 Data: Progressive Configuration of Specifications
Analogous to our progressive strategies for training objective and architecture we also propose a progressive data configuration. Over time, we gradually increase the video's spatial resolution, alongside progressively extending the video's duration. This approach ensures efficient use of computational resources and facilitates effective model scaling, allowing MarDini to handle more complex and high-rcsolution video data as training progresses.
The details of the mask ratios, architecture design and data size used within each training stage are reported in Appendix B
3 Experiments
We evaluate MarDini on two benchmarks: VIDIM-Bench (Jain et al. 2024), for long-term video interpolation, and VBench (Huang et al. 2024) for sanage-to-vides generation. We further elaborate on the specifics of these benchmarks in Appendix DJ We highly encourage referring to the generated videos in the supplementary material for a comprehensive understanding of the quality of the generated videos.
3.1 Ablation Studies and Analysis
Effectiveness of MAR and DM. We first demonstrate the importance of having a DM on top of our MAR planning model. In fact, is tempting to hypothesize that MAR on its own contains all the ingredients to enable high-quality video interpolation. To explore this, we introduce a projection layer directly unpatchify the output of the MAR model without intermediate diffusson with DM. Our experiments VIDIM-Bench reveal that, MAR on its own, performs poorly interpolation tasks, as shown by the first two and last two rows in Table [] for both the 1B and 3B settings. This result suggests that directly applying MAR to contimuous space is suboptimal, a result consistent with previous findings (Li et al., 2024). Similarly, directly tackling this task with a small DM without global guidance, according to the third row of Table [ results in sub-optimal performance. However, by combining MAR's planning capability with DM's stable performance in continuous space, we achieve optimal results, demonstrating that both components are beneficial for video generation.
Table 1: Effectiveness of MAR and DM de-
Table 2: Efficiency of the MarDinl's generations with and without the asymmetric design. Both latency and GPU memory is mensured as the average time to generate a video using DDIM with 25 steps using single A100 GPU, and with bf16 mixed precision.
sign. The reported results are FVD DAVIS- and UCF101-7 from VIDIM- ments are evaluated resolution of [256 256]
All experi-
using DDIM scheduler with 25 steps.
Asymm. Attention
Asymm.
Inference Prames
[256 x 256]
[512 x 512)
Planning Model
Generation Model
FVDJ
Resolution
Latency GPU Mem. Latency GPU Mem.
DAVIS
UCF101
x
91 to 8) 9 to 8)
25.09 s
74.44 G
MAR-IB
427.66
741.80
x
/
2.76
25.22 G
17.91 8
41.03 G
MAR-3B
373.03
701.03
X
x
13 12)
Out of Memory 34.58
DM-0.3B
320.89
383.04
X
/
13 (1 to 12)
4.41
27.80
62.51 G
MAR-1B
DM-0.3B
224.07
258.08
x
13 (1 to 12)
2.63
27.75 G
Out of Memory 6.05 s 42.57 G
MAR-3B
DM-0.3B
102.87
197.69
13 (1 to 12)
Efficiency Analysis. Table 2 illustrates latency and memory usage across different input resolutions and frame lengths, measured on the same computational platform. When MAR is set to operate symmetrically with the DM with the same inputs, the model cannot fit in the available GPU memory as increase the resolution and/or number of frames. In contrast, our asymmetric design enables the generation of 12- frame clips at 512 resolution in just. a few seconds. The rapid gemeration process is partially al tributee to the DM requiring relatively few inference steps to converge, thanks to the well-structured planning signal it receives, as shown in Figure [5a] Notably, inference speed could be further optimized, as the only acceleration technique we incorporated during our experiments is mixed precision, without employing caching strategies [Liu et al. 2024 Zhao et al. 2024), FSDP-based parameter /aaaa parallelization (Chen et al. 2024d),
7

---

Published in Transactions on Machine Learning Research (05/2025)
static compilation of the underlying computational graph. Similarly, memory usage could be further reduced through CPU offloading, sliced attention, sequential VAE inference, elc.
Explaining MAR's Planning Signal. We provide an intuitive explanation of MAR's role in MarDini. During training, a learnable token is used to randomly replace MAR to support CFG (Ho & Salimans 2022 allowing DM to generate videos independently. We visualize the results of MarDini with and without MAR's planning signals. As shown in Figure without the planning model, DM can still produce meaningful frames but, expected, lacks "global planning." For example, in Figure (Left), DM moves objects in different directions, causing distortion in the building, which suggests weaker or non-exisling prior model of how objects move. Similarly, in Figure 4 (Right), DM fails to accurately predict the movement of the fire, In contrast, incorporating the planning signal addresscs these visual flaws. These results indicate that MAR's planning signal effectively hints how elements should move, ensuring long-term coherence in the generated video.
Reference Frame (Middla)
Generated Frames (First, Last)
Reference Frame [Midele]
Generated Frames [First, Lastl
w/o Planning Boiiin
Improved Pixel Details
Improved Motion
w/ Planning
wi iilammi ng
Figure 4: MarDini's generations with and without the planning model. Here show vidco frames generated when conditioning on the middle frame. Without MARt's planning signal, DM generates degraded motion, such pixel distortions (highlighted in red, left) incorrect motions (highlighted in blue, right).
600
UCF101-7
275
UCF101-7
DAVIS-7
VBench
0.4
DAVIS-7
VBench Avg. Score
500
250
Training Loss
0.35
IVD
400
225
90
0.3
300
200
0.25
200
175
w/o kdentity Atten. w/ Identity Atten.
0.2
1
10
25
100
2
6 Training Steps
8
10
Inference Steps
Training Steps
-10*
-10"
(a) Video interpolation results with
(b) Relationship between video interpolation
Training loss Mar w and
varying inference steps.
and imago-to-videu generation.
w/o Ideutity Attention.
Figure MarDini Training and Inference Performance. (a) MarDini-S/STT512 achieves optimal generation perfor- mance with few inference steps using the DDIM solver; (b) As training progresses, MarDini shows improvement in the tasks of both video interpolation and image-ta-video. These results are based on a mask ratio ranging from 0.15 to 0.6 for 9 9-frame generation; and (c) The design of Identity Attention is crucial for stable training convergence in MarDini during the initial training stage; without it, the model fails to converge.
From Video Interpolation to Image-To- Video Generation. Our training recipe follows the philoso- phy of transitioning from video interpolation to image animation. Herein, we empirically demonstrate that these two tasks are related, validating the soundness of our pipeline. As shown in Figure 5b we track the performance of MarDini on both vidco interpolation and image animation during a training phase aimed at scaling the resolution from 256 to 512. This stage marks the first point during training where the model successfully performs both tasks simultaneously. We observe a promising consistency between the perfor- mance of image animation and video interpolation, providing solid evidence that these tasks do not hinder each other. Furthermore, with a carefully tuned mask ratio, the model can be trained a unified manner to efficiently achieve both tasks.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Impact of Identity Attention. We explore the effectiveness of Identity Attention in handling our specific data format, which integrates both reference frames and noised frames into a single sequence. As illustrated in Figure 5c we track the training trajectory in the early stages of the DM generation model. We recognize that this type of imput can lead to unstable training, particularly when starting from scratch, as the differences between reference frames are difficult to discern. However, the proposed Identity Attention mechanism mitigates this instability. The decrease in training loss observed after 6K steps is attributed to the use of a warm-up learning rate, where the learning rate is intentionally kept low during the initial steps.
3.2 Results on Video Interpolation
Table 3: Performance of zero-shot video interpolation VIDIM-Bench. The reported results are taken directly from VIDIM (Jain et al. 2024). AMT, RIFE, and FILM are single-inference methods, while LDMVFI, VIDIM, and our approach are based on diffusion models with multiple inference steps. MidF-SSIM and MidF-LPIPS represent the SSIM and LPIPS scores, respectively, for the middle frame. For MarDini-512, we downscale the generated vidcos to 256 resolution for fair comparison.
DAVIS-7
UCF101-7
Method
MidF-SSIM
MidF-LPIPS
FID
FVD
MidF-SSIM
MidF-LPIPS
FID
FVD
AMT Li al
2023b
0.4853
0.2865
34.65
234.50
0.7903
0,1691
31.60
344.50
RIFE
Hnang
0.4546
0.2954
23.98
240.04
0.7769
0.1564
18.72
323.80
FILM
Reda al.
2022
0.4718
0,3048
30.16
214.80
0.7869
0.1620
26,06
328.20
LDMVFI
(Danler
2024
0.1175
0.2765
22.10
245.02
0.7712
0.1564
18.09
316.30
VIDIM (Jain et al.
2024
0.4221
0.2986
28.06
199.32
0,6880
0.1768
34.48
278.00
MarDini-S/9T-256 MarDimi-L/S11256
0.1244
0.3654
19.21
224.07
0. 0.7654
0.2480
45.85
258.08
0.4959
0.2768
20.64
102.87
0.7734
0,2213
28.85
197.69
MarDimi-S/ST--12 MarDimi-L/S1-512
0.5017
0.3193
25.92
138. 86
0.7960
0,2315
30.24
205.77
0.5314
0.2736
99.05
0.7814
0,2347
30.08
MarDini-L/T-612
0.5085
0.3083
25.30
117.13
0.7893
0,2270
30.72
198 94
We compare MarDini with the existing methods on the VIDIM benchmark (Jain et al. 2024) for video
interpolation, where the task is to generate 7 frames between a starting and an ending conditional frames. As shown Table 3. MarDini achieves competitive performance among different evaluation metrics. Since FVD is the only metric that accounts for the temporal modelling, we prioritize it in our evaluation. No- tably, MarDini outperforms other basclines in this metric, achieving state-of-tho-art performance. No- tably, MarDini-L/TT employs asymmetric attention mechanism, where the planning model utilizes spatio- temporal attention, while the generation model relies on temporal attention. Compared to the model that uses spatio-temporal attention for both models (MarDini-L/STT), the results suggest that the asymmetric attention mechanism does not significantly affect performance, achieving a satisfactory trade-off between efficiency and quality. We provide additional visualizations in Appendix[[ the supplementary materials.
3.3 Results Image-to- Video Generation
In this section, we evaluate our model's single-imagc-oo-riieo generation capabilities in comparison with other methods using the VBench dataset (Huang et al. 2024). As shown in Table 4 our method performs competitively, especially in terms of latency, despite incorporating expensive spatio-temporal attention. For fairness, latency is calculated with the same resolution. In this study, we focus on validating the soundness of our proposed roadmap, only considering the initial pre-training stage rather than delving into post-training techniques. As a result, we do not incorporate additional conditional signals such as language instructions or motion score guidance. Therefore, direct comparisons on vidco quality, particularly in relation dynamic degree, are not entirely fair. However, we fully report these numbers for reference.
We also report the results on the benchmark without the motion score (referred to Dynamic Degree in VBench). All evaluation metrics are detailed in Appendix D The empirical study shows MarDini's strong potential, performing on par with other existing methods across several metrics while exhibiting higher efficiency and requiring no generative image pre-training. Among these baselines, SVD-XL can be regarded as a DM-only alternative, with a parameter size comparable to MarDini-S. Notably, while MarDini-S achieves comparable performance SVD-XL, offers greater flexibility and efficiency without compromising generation quality. Interestingly, we observe that MarDini-S marginally outperforms MarDini-L. on some
9

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: Image-to- Video Performance on VBench. The reported results baseline methods are sourced from VBench [Huang et al. 2024p. For fair latency comparison, we standardize the input size to [512x512] for low and medium resolutions, and [768x 768 for high resolution cases across all methods. All other metrics were collected using the original resolutions reported in the first column.
Frame Resolution
Image-based Pre-training
Lacency (s/frame)
12V Sub.
12V Back Con.
Video Quality (w/ D.D.)
Video Quality
Vbench
Method
Con
(w/o D.D.)
Avg.
Low and Medium Resolution
12 [Re al.
2024
x
7.63
95.82
95.95
78.87
85.74
88.27
DyuamicCrateer Xing
20244b
256xx26
97.05
97.56
80.18
85,00
88.07
DyuamicCraßter Xing et
2124bb
[22xx320
4.88
97.21
97.40
81.63
85.39
86.37
SEINE Cheu el al.
2023
[212x320
96.57
96.50
79.49
85.71
88.45
VileoCralter (Chen el al 2024a
320
9.43
91.17
91.31
81.34
87.55
86.47
Cog ViclooX Yang
2024
480
5.24
97.67
76.01
83.90
86.00
AnimateDill Guo el al.
20233
[512x512'
1.50
84.21
84.48
70.80
78.92
81.98
SEINE (Chou al.
2123
[512x512
5.13
97.15
96.94
80.58
87.13
89.61
Animale AaeinateeAAyyiinen Anything
Dai al.
2023b
[512x512
1.58
98.76
98.58
81.21
88.84
91.30
MarDimiiLL/ST--9
[512x512]
X
2.24
98 08.64
97.12
80.84
88.22
90.64
MarDini-S/S9 S/ST-9
512xx112
2.24
99.04
97.23
81.00
88.59
90.98
MacDinnlLL/T-17
512xx522
X
0.18
98. 23
97.01
80.25
87.06
90.16
MarDinl-S/T-17
512x5112
X
0.46
98.
97.18
80.56
88.17
90.62
High Resolution
SVD-XT-110
Blattmann et al.
20233
[024x576]
2.19
97.52
97.63
82.79
86.54
89.30
SVD-XT-1.1
Blattmann
2023a
[024x576]
/
2.19
97.51
97.62
82.23
86.66
89.38
Ge
Zhang 2023
1280x720]
/
96..48
87.02
80.43
DynamiCrafter Xing et al.
2024h
[024x576
/
7.13
98.11
98.60
82.52
87.31
90.08
MacDini-L,T-II
768x
X
1.01
98 34
96.63
80.88
88.22
90.54
MarDini-S/T-17
776xx668
X
0.98
96.78
81:29
MarDini-L,TT17
1024x1024]
X
1.80
98. 08.61
96.34
81.35
88.69
90.89
SutDiii--/TT11
[104xx124]
1.77
98.78
96.46
81.74
88.97
91.13
evaluation metrics. While, we observe notable benefits from scaling up the MAR model: MarDini-L excels in video interpolation and aligns closcly with physical principles in imago-to-video generation. To substantiato this, we conducted user study involving 10 participants who evaluated 20 video pairs for physical accuracy, The study reveals that MarDini-I, significantly outperforms Mardini-S, with preference rate of 85.83%, indicating clear preference for MarDini-L's outputs.
3.4 Additional Applications
In addition to image animation and video interpolation, MarDini also demonstrates potential for several other interesting applications, including zero-shot 3D view synthesis, video expansion, and hierarchical auto- regressive generation for slow-mo videos. Due space constraints, please refer Appendix for details.
4 Related Work
Auto Pnto-Regressivee Model in Visual Generation.
Auto-regressive (AR) models (Gers et al. 2000
Hochreiter & Schmidhuber 1997;
Schmidhuber Dubey et al.
2015)
have proven effective in natural language modeling
Brown 2020 Achiam et al. 2023
2024 Team et al. 2023 To adapt this scalable modeling
strategy for image and video generation, recent. approaches (Yu et al.] 2024 Chang et al.] 2022 Li et al.
2023a: Yu al.] 2023a Chang et al. 2023 Yu et al.
2023a replace causal attention AR with bidirectional
allention, allowing for betler capture of dense relationships in visual space.
Xie et al. 2024 validate the scalability of this approach. To align with the training recipes from LLMs, these studies adopt discrete visual representations,
Many studies YYu et al. 2023b Chang et al.
2023
Team 2024
using image tokenizers (Esser el al. 2021 Yu et al. 2021
Van Den Oord el al.
2017) to quantize continuous
pixel values into discrete representations. The MLAGVIT series Yu et al. 2023a 2024 adopts this strategy for
video generation, demonstrating its effoctiveness and contributing
the success of VidcoPoct KKndratyuk
2019)) argue that this strategy suffers from unstable training and may limit model capacity due to the inherently continuous nature of visual
et al. (2023). However, Li et al. (2024): Ramesh al. (2021): ) Razavi al.
10

---

Published in Transactions on Machine Learning Research (05/2025)
data. This inspires recent works Li ct al. 2024 Zhou et al. 2024) to shift towards continuous latent spaces for masked auto-regressive models to address these limitations.
We follow this trajectory but diverges in two ways: i) We highlight the importance of mask ratios, which were fixed in earlier works Li et al.) (2024). By dynamically adjusting them with a progressive training strategy, we improve both model scalability and stability. ii) We propose an asymmetric input resolution design, allowing MAR to be effectively trained with full-resolution inputs.
Diffusion Model for Video Generation. In recent years, diffussio models (Ho et al.
2020 Neal
2001
Jarzynski 1997] have become a leading approach for both image and video generation RRobaac et al.
2022
Dhariwal & Nichol 2021 Ramesh et al. 2022; Chen et al.) 2024c [Saaaiia et al., 2022] Brooks et al.
2024
Dai et al. 2023a Girdhar et al. 2023 Menapace et. al.) 2024 Kondratyuk et al. 2023 Cong et al.
2024
Li ct a 2025). These models conceptualize the generation process as gradually refining real sample from
Gaussian noise, demonstrating significant scalability and stable training. This paper empirically focuses on
two key video generation tasks: image-to-videe generation (Ren et al. 2024 Xing et al.
2024b
Chen et al.
2023 2024a Guo et al. 2023 Dai et al.] [023bb] Blattmann et al.) 2023a Zhang et al.
2023b
Xing et al.
2024b) and video interpolation (Wang ct al. 2024c
Volcti et al.; 2022
Guo ct al.
2024
Dong et al.)
2023 Li
et al.
2023b Huang al. 2022
Reda et al. 2022
Danier el. al. 2024 Jain et al.
2024
Xing et al.
2024a
We offer two key insights into video generation: i) Previous methods (Wu et al.
2023
Ho et al. 2022 Zhang
ct al. 2023a Blattmann et al.
2023b Wang et al.
2023 Girdhar et al. 2023
Gao et al.)
2024) often first
pre-train an image generative model, and then tune it for video generation, or they require joint training
for both tasks (Chen et al.) 2024c Esser et al., 2023 Alternatively, some studies Guo et al. (0023) ; Cong
ct al. (0024); Khachatryan et al. (0223 adopt tuning-free (or without specific-tuning) approaches to adapt existing image- based models for text-to-video tasks. While multi-stage pre-training on diverse inputs can be beneficial, video generation is often limited by the success of image-based pre-training, which typically serves secondary task. This paper proposes an alternative: training video generation models from scratch with progressively increasing task complexity. ii) Previous research (Girdhar et al., 2023: Wang et al.) 2023; Chen et al. 2024c Blattmann et al.; 2023b) has predominantly employed temporal attention mechanisms to capture temporal dependencics, mainly due to the high computational and memory costs associated with spatio-temporal allention. However, aligunent with previous work (Blattmann et al., 2023b Gao et al., (024) suggesting that spatio-temporal attention enables superior video modeling, we propose an amortized strategy that makes spatio-temporal attention computationally feasible, even at high resolutions.
Finally, our paper also relates to asymmetric neural networks. Due to page limilations, additional related works are discussed in Appendix B
5 Conclusion
We have introduced new family of generative models for video, i.e, MarDini, based on auto-regressive diffusion, wherein a large planning model offers powerful conditioning to a much smaller diffusion model. Our design philosophy considers efficiency from model conception, and so our heaviest model component is only executed once at lower resolution inputs, whereas our generative module focuses on fine-grained details at the frame level, reconciling high-level conditioning and image details. Our model is unique in that it leverages a masked auto-regressive loss directly at the frame level. MarDini is afforded with multiple generative capabilities from single model, e-g,, long-term video interpolation, video expansion, and image animation. Our investigation shows that our modeling strategy is powerful enough to obtain competitive results on various interpolation and animation benchmarks, while doing it at lower computational needs than counterparts with comparable parameter size. We discuss the limitation of our study in Appendix
6 Limitations and Future Works
Post Training. The primary goal of this paper is to demonstrate the feasibility and effectiveness of combin-
ing masked auto-regressive (MAR) models with diffusion models (DM) for video generation. Consoquently, we allocated the majority of our computational resources to the pre-training stage, placing less emphasis post-training, despite its recognized importance in generative models (Dai et al. 2023a Dubey et al.
11

---

Published in Transactions on Machine Learning Research (05/2025)
2024
Touvron al. 2023 Post-training will be top priority in future work, focusing on enhancing
long-term planning, improving motion quality, and achieving higher resolutions.
Improved Conditional Signals. A significant contribution of this work is the exploration of training a video generation model without relying on generative image pre-training. However, this approach presents a trade-off: MarDini is not inherently equipped with a text encoder for processing textural instructions. To conserve computational resources and quickly validate the feasibility our method, we intentionally excluded commonly used conditional signals, such as text. embeddings and motion scores. Encouraged by the initial success of our model, we plan to incorporate these conditional signals into MarDini in our future updates to broaden its range of applications.
Ethics Statement
This paper explores the theoretical foundations neural architecture design for video generation, rather than being tiod specific commercial applications. Consoquently, the potential negative impacts of Mar- Dini align with those of other video generation models and do not pose unique risks that require special consideration. Importantly, unlike previous models trained on web-scale data, which may raise concerns about data copyright, MarDini is exclusively trained on a licensed Shutterstock dataset, without having such conflicts.
Reproducibility Statement
We ensure reproducibility by providing detailed model configurations in Appendix along with the complete training recipes outlined in Appendix ( However, due to organizational policies, the model trained using internal infrastructure and proprictary dependencies that cannot be publicly released. Additionally, the VAE component is based on internal product data and was not developed within this project, further restricting the potential release of model weights.
Acknowledgements
The authors thank Mingchen Zhuge, Jinheng Xie, Yuren Cong, Kam Woh Ng, Aditya Patel, and Jinjie Mai for their valuable suggestions and contributions to the paper review. Haozhe Liu and Jürgen Schmidhuber were supported by funding from the King Abdullah University of Science and Technology (KAUST) Center of Excollence for Generative AI under award number 5940 and the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidl, Sam Allman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint txXi::2903.0877, 2023.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXio, abs/1607.06454/ 2016.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion Euglish, Vikram Voleti, Adam Letts, el al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprind arXiv:2311. 151127 2023a.
Andreas Blatimann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Aligu your latents: High-resolution video syuthesis with latent diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023h.
Tim Brooks, Bill Peebles, Comuor Homes, Will DePue, Yulei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luluman, Eric Luhman, et al. Video generation models as world simulators, 2024. URL allps://opende www.w//www//////////// video 2024.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Tom B Brown. Language models are few-shot learners. arXie preprint arXiv:2005.141663 2020.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGiT: Masked generative image trans- former. In Proccedings of the IEEE Conference on Compuler Vision and Pattern Recognition (CVPR). 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hssun Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text- Text-to-image generation via masked generative transformers. Proceedings of the International Conference on Machine Learning (ICML), 2023.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXio:2401.0900 09904777 2024a.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-\sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2408.04609 2024b.
Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Peraz-Rua. Gentron: Diffusion transformers for image and video generation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2024c.
Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In Proceedings of the International Conference on Leorning Representations (ICLR), 2023.
Zigeng Chen, Xinyin Ma, Gongfan Fang, Zhenxiong Tan, and Xinchao Wang. Asynodiff: Parallelizing diffusion models by asynchronous denoising, arviv preprint or Xiv:24400 06911, 2024d.
Arnab Choudhury, Yang Wang, Tuomas Pelkonen, Kutta Srinivasan, Abha Jain, Shenghao Lin, Delia David, Siavash Soleimanifard, Michael Chen, Abhishek Yadav, et al. MAST: Global scheduling of MI. training across Geo- Distributed datacenters at hyperscale. Proceedings from 18th USENIX Symposium on Operating Systems Design and Implementation, 2024.
Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Juan-Manuel Perez-Rua, Bodo Rosenhahn, Tao Xiang, and Sen He. FLATTEN: optical fow-guided attention for consistent text-to-vided editing, In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Xiaoliang Dai, Ji Hon, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint or Xiv:2299 15807, 2023a.
Zuozhuo Dai, Zheughao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Fine-grained open domain image animation with motion guidance. arXie preprint (xxix:2311.128900 2023b.
Duolikun Danier, Fan Zhang, and David Bull. LDMVFI: Video frame interpolation with latent diffusion models. In Proceedings of the National Conference on Artificial Intelligence (AAAT), 2024.
Chaorui Deng, Deyao Zhu, Kunchang Tú, Shi Guan, and Haoqi Fan. Causal diffusion transformers for generative modeling. arXiu preprint arxim:24 12. 12095, 2024.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Long and Short Papers), 2019.
Prafulla Dhariwal and Alexauder Nichol. Diffusson models beat GANs on image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Jiong Dong, Kaorn Ota, and Mianxiong Dong, Video frame interpolation: A comprehensive survey. ACM Transactions on Multimedia Computing, Communications and Applications, 19(2s):1-31, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesuikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Represen- tations (ICLR), 2020.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Laura Downs, Authony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dalaset of 3d scanned household items. In Proceedings of the IEEE International Conference on Robolics and Aulomation (ICRA), 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The LLAMA 3 herd of models. arXiv preprint arXiv:2407.21 2173, 2024.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformens for high-resolution image synthesis. Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition (CVPR), 2021.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXie:2405.005 05545, 2024.
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins, Learning to forget: Continual prediction with LSTM. Newral
computation, 2000.
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. ariie preprint arXie:2311.10709 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. Proceedings of the International Conference on Learning Regresentations (ICLR), 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 330 348. Springer, 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXie preprint arXiv:2111.06377, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Nearal Information Processing Systems (NeurIPS), volume 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. Advances in Neural Information Processing Systems (NewrIPS) Workshop, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusson models. arXie preprinl arXie:2210.0020 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation MIT-Press, 98): 1780, 1997.
Xun Huang and Serge Belongie. Arbitrary style transfer real-time with adaptive instance normalization. Proceedings of the International Conference on Computer Vision (ICCV), pp. 1501-1510, 2017.
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Champaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Compreheusive benchmark suite for video generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Siddhant Jain, Daniel Walson, Eric Tabellion, Ben Poole, Janne Kontkanen, el al. Video interpolation with diffusion models. In Proceedings of the IEEE Conference on Compuler Vision and Patlern Recognition (CVPR), 2024.
Christopher Jarzyuski. Equilibrium frev-energy differences from nonequilibrium measurements: A master-equation
approach. Plyssiaa Review E, 1997.
Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up GANs for text-to-image synthesis. In Proceedings of the TEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Tero Karras. Progressive growing of GANs for improved quality, stability, and variation. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphree Shi. Text2vidoo-zero: Text-lo-image diffusion models are zero-shot video genera- tors. XrXi preprint 13344 2023.
Diederik Kingma and Max Welling. Auto auto-encoding variational Bayes. Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint ar 21x:22312.11 14121 125, 2023.
Cheng Li, Keyuan Zhou, Tong Liu, Yu Wang. Mingqiao Zhuang, Huan-ang Gao, Bu Jin, and Hao Zhao. Avd2: Accident video diffusion for accident video description. arxiv preprint arXiv:2502.1480: 2025.
Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023a.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arviv preprint arXiv:2406.1188 11838, 2024.
Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi- field transforms for efficient frame interpolation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023b.
Zhengyang Liang, Hao He, Ceyuan Yang, and Bo Dai. Scaling laws for diffusion transformers. arXie preprint 2rXiv:241000810 2024.
Haozhe Liu, Wentian Zhang, Jinheng Xie, Francesco Faccio, Mengmeng Xu, Tao Xiang, Mike Zheng Shou, Juan- Manuel Perez- Perez-Rua, and Jürgen Schmidhuber. Faster diffusion via temporal attention decomposition. arXiv e-prints, pp. arXiv 2404, 2024.
Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotemporal transformers for text-to- video synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2024.
Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Ope A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02371, 2024.
Radford M Neal. Annealed importance sampling. Statistics and computing, 2001.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
Bo Peng, Eric Alcaide, Quentin Authony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing RNNs for the transformer era. arXio preprinl ar 2305 230448 2023.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Pablo Pernias, Dominic Rampus, Mats L Richter, Christopher J Pal, and Marc Aubreville. Würstchen: An efficient archilecture for large-scale texl-to-image diffusion models. Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombachh. SDXL: Improving latent diffusion models for high-resohution image syuthesis. arviv preprinl (rXi::307.019533 2023.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hormung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. Xr preprint arXiv:1704.00 000617 2017.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-lo-hmage generation. In Proceedings of the International Conference Machine Learning (ICML), 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint orXi:2204.006 06125, 2022.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. Advances in Newral Information Processing Systems (NewrIPS), 2019.
Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. Film: Frame interpolation for large motion. In European Conference on Computer Vision, pp. 250-266. Springer, 2022.
Weiming Ren, Harry Yang, Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation. arXiv preprint S0:::440.003222 2024.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesss with latent diffusion models. In Proceedings of the IEEE Conference on Compuler Vision and Pallern Recognition (CVPR), 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding, In Advances in Nearal Information Processing Systems (NeurIPS), 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Proceedings of the International Conference on Machine Learaing (ICML). PMLR, 2021.
Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compression. Newral computation, 4(2):234-242) 1992a.
Jürgen Schmidhuber. Learning contral fast-weight memories: An alternative dynamic recurrent networks. Newral Computation, 4(1):131-139, 1992b.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 2015.
Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Lla- mafusion: Adapting pretrained language models for multimodal generation. arXiv preprint orXiv:24 12, 15188, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. Proceedings of the Inter- national Conference Learning Representations (ICLR), 2021.
Khurram Soomra, Amir Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. ArXiv, 1222,02244 2012.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Lin. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXio preprint arXio:2466. 06620655,6666 2024.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Chameleon Team. Chameleon: Mixed-modal early-fusion Ioundation models. arXiv preprint arXiv:2405.09815 2024.
Gemini Team, Rohan Anil, Sebastian Borgeand, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint Xiv::23322 11805, 2023.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual antoregressive modeling: Scalable image generation via next-scale prediction. Aduances in Neural Information Processing Systems (NeurIPS), 37: 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. LLAMA 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2907.0000 009888, 2023.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly, Towards accurate generative models of video: new metric & challenges. arXiv preprint 0rii:1812..0111 2018.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neurol Information Processing Systems (NeurIPS), 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Vikram Voleti, Alexia Jolicoeu-MMrrtineuu and Chris Pal. MCVD-masked conditional video diffusion for prediction, generation, and interpolation. Advances in Nearal Information Processing Systems (NewrIPS), 2022.
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling trans- formers to 1,000 layers. IEEE Transactions Pattern Analysis and Machine Intelligence (PAMT), 2024a.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang, Modelscope text-to-video technical report. arXiv preprint arxi22330 06571, 2023.
Weimin Wang, Jiawei Liu, Zhijie Lin, Jiangqiao Yan, Shuo Chen, Chetwin Low, Tuyen Hoang, Jie Wu, Jun Hao Liew, Hanshu Yan, et al. Magievideo-v2: Multi-stage high-aesthetic video generation. arXiv preprint or orX::24411.044 2024b.
Xiaojuan Wang, Boyang Zhou, Brian Curless, Ira Kemelmacher--Shiieemeee Aleksander Holynski, and Steven M Seitz. Generative inbetweening: Adapting image-to-video models for keyframe interpolation. arXiv preprint arXim:2088. 1524998..329,, 15239, 2024c.
Xinlong Wang, Xiaosoug Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint orXiv:249.. 18869, 2024d.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600 612, 2004.
Chen Wei, Karttikeya Mangalam, Po- Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, and Christoph Feichtenhofer. Diffusion models as masked autoencoders. In Proceedings of the International Conference on Computer Vision (ICCV), pp. 16284 16294, 2023.
Jay Zhangjie Wa, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint aixiv:2408. 12528, 2024.
Jinbo Xing, Hanyuan Läu, Menghan Xia, Yong Zhang, Xintao Wang, Ying Shan, and Tien- Tsin Wong. Tooncrafter: Generative cartoon interpolation. ACM Transactions on Graphics (TOG), 43(6):1-11, 2024a.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diflusion priors. In Proczedings of the European Conference on Computer Vision (ECCV), 2024b.
17

---

Published in Transactions on Machine Learning Research (05/2025)
Mengmenng Xu, Juan Manuel Perez Rua, Xiatian Zhu, Bernard Ghanem, and Brais Marlinez. Low-fidelity video en- coder optimization for temporal action localization. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint Xix::22408. 06072, 2024.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint ar 220 10789, 2022.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, al. MagViT: Masked generative video transformer. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognitton (CVPR), 2023a.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion tokennizr key to visual generation. Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Lili Yu, Bowen Shi, Ramakanth Pasumuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint ar 29099 02591, 2023b.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems (NeurIPS), 2019.
David Junhno Zhang, Juy Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for texl-to-video generation. ar preprinl arXin:2309.1 111.809..1111. 15818, 18, 2023a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In Proceedings of the IEEE Conference on Compuler Vision and Pallern Recognition (CVPR), pp. 586 595, 2018.
Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qing, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality maage-to-vides synthesis via cascaded diffusion models. arXiv preprint 2rX10:2311.04145, 2023b.
Xuanlei Zhao, Xiaolong Jin, kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast. arXiv preprint 240: 12588, 2024.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien- Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch FSDP: experiences scaling fully sharded data parallel. arXie preprint 2rXiv:2304.11277 2023.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-mode model. Xixiv preprint Xrx::2248.. 1039, 2021.
18

---

Published in Transactions on Machine Learning Research (05/2025)
A Model Configuration
As outlined in Table 5 this study develops four models with distinct configurations. We train two planning models with 3.1B and 1.3B parameters alongside two generation models, employing spatio-temporal or temporal attention mechanisms. To align with our asymmetric design between the planning and generation models, the generation model's parameter size is reduced to 3x or 10x smaller than that of the planning model. Due to the high computational cost of spatio-temporal attention, we limit MarDini-L/ST and MarDini-S/ST to 9-frame length for fair comparison VIDIM-Bench (Jain et al. 2024). Importantly, the model's ability to autoregressively generate samples ensures that the length of the output video is not constrained.
Planning Model (MAR)
Generation Model (DM)
Configuration
Frame
Depth
Hidden Size
MLP Size
Attn.
Param.
Depth
Hidden Size
ML.P Size
Attn.
Param.
MarDini-S/ST
S.-T. Astn.
1.3B
S.T.. Attn.
288M
MarDini-LL/ST
16
4096
8192
S.T. Astn.
3.1B
8
1024
4096
S. Astn.
288M
9
MarDini-S/T
4096
S.T.. Astn.
1.3B
4096
Attn.
17
MarDini-L/T
16 1096
8192
S.-T. Attn. 3.1B
8
1021
1096
T. Attn. 288M
17
Table 5: Configuration Details MarDini Models. We provide four models, differing primarily in the size of the planning module (3.1B va. 1.3B parameters) and the attention mechanisms used the generation module: spatio-temporal attention (S.-T. Attn.) va. temporal attention (T. Attn.). Here, we selected the parameter size our generation model to align with the typical design of VIT-L (Dosovitskiy et al. 2020), as it has demonstrated excellent performance in frame-level generation tasks. Wei et al. 2023
B MarDini Training Manual
Tn Figure we present the training details of MarDini. All experiments, including model variations, ablation studies, benchmark evaluations, and full model training, are carried out on distributed MAST schoduler (Choudhury et al., 2024) using 256 H100 GPUs. The training dataset comprises approximately 34 million filtered Shutterstock videos, segmented into 2-second training clips. We use the AdamW optimizer for each stage with a 1.4 x 10-4 learning rate and cosine learning rate scheduler. We adapt our batch size based on the resolution and the frame count to maximize GPU utility. For example, at [256 x 256] resolution with 9 frames, the batch size is 1024, processing 9K frames per iteration; at [512 x 512] resolution with 9 frames, the batch size is 720, processing 6480 frames per iteration. During inference, we set the classifier- free guidance (CFG) (o Salimans, 2 scale as 2.5 for the imago-to-video task with the noise solver DDIM (Song al., 2021), and directly remove classifier-free guidance for video interpolation as is redundant. FSDP (Zhao) jtt al. 2023 and activation checkpointing (Zhao et al. 2023) enabled to further save GPU memory. We do not include dynamic resolution training in our main training stages, it slows down training, Instead, we find that after convergence, fine-tuning the model for few steps (10K-20K) with dynamic resolutions enables it to quickly support this capabilities.
C Visualization of Video Interpolation
we provide visualization results that demonstrate the superiority of MarDini in large motion modelling, compared to FILM (Reda et al. 2022). LDMVFI (Danier et al. 2024), and VIDIM (Jann et al. 2024).
In Figure 7
19

---

Published in Transactions on Machine Learning Research (05/2025)
Initial Stage
Sint-Mooel Stage
Masked Diffusion Loss
Masked Diffusion Loss
Masked Diffusion Loss
Masked Diffusion Loss
Generation Model [DM]
Generation Model [DM]
Gener alion Model IDM,
NAAR)
Mosic Ratio: 45% 100%
Mask Ratio: 100%
Mack Rotio: 25% 100% Fromes:9 FPSS 81 Recolution: 256 Training Clips
Fromesu4 FPS: 0 Resolution: 256
Framesiá IFPS Resolution: 256 Training
Training
Ratior
i Frames:9 FFS: Resolution: 256 Training Data: 75M Clps.
Wasked Recostrruction Loss
Ramnov Projection Layer
Planning Model (MAR)
Masic Ratio: 65% 100%
Fromesus FPS:8 Resolution: 256
Training 200M
Joint-Task Stage
Hasked Diffusion Loss
Masked Diffusion Less
Goneration Model (DMI
Generztion Model [DM]
MaDD1/i-51
37%
F Resolution: 512 Training Daca: 40M Clios
Frames:177 Resolution:
Training Dala: 4DM Clips
Hasked Diffusion Loss
Masked Diffusion Less
Masked Diffusion Less
Reprace Spatio Tempporal Atteation with Temporai Attention DM
Generation Model [DM]
Seneration wiion Model IDM
Gener alion Model [DM]
MarDini-512/T
MarDini-768/T
MarDini-
Mosk
Fromes:1y FPS: 8 Resolution: 512
Fromes:17 FPS: 0 Resolution: 768
Fromes:17 FPS. Resclution. 1324
Training Data: 42M Cliss
Training Data: 20M Clips
Traising Data: BM Clps
Figure 6: MarDini Training Manual. We the mask frame (FPS), number of frames, and the of training data each training stage. Note that the training data refers amount of data observed by the model for gradient updates, rather than vanilla size the training dataset. Our final model checkpoints are highlighted in gray. Mask Ratio refers proportion of frames preserved during training.
Generated Frames (Middle)
Reference Frames (First, Last)
FILM
LDMVFI
VIDIM
Ours
Ground- Truth
Figure Visualization video interpolation methods conditioned first frames. present the generated frames from FILM [Reda al. 2022), LDMVFI (Danier al. 2024), VIDIM (Jain et al. 2024), and MarDini. The comparison results for these met hods are sourced from Jain et al. (2024) have included additional samples in the supplementary materials.
Benchmarks
We evaluate the interpolation performance on VIDIM-Bench (Jain al. 2024) and assess image animation performance VBench (Huang et. al. 2024).
20

---

Published in Transactions on Machine Learning Research (05/2025)
For VIDIM-Bench, the task involves generating seven intermediate frames, with the first and last frames provided as conditions. The dataset includes approxinately 400 videos from both DAVIS (Pont-Tusel et al. 2017) and UCF-101 (Soomro et al, 2012). We use FVD (Unterthiner et al. 2018) and FID (Heusel et al. 2017 as generation metrics, while adopting SSIM (Wang et al.) 2004) and LPIPS [Zang et al. 2018) as reconstruction metrics. Notably, we evaluate the middle (5th) frame for reconstruction metrics, as presents the greatest challenge due to its distance from the reference frames.
For VBench, utilize the official dataset to assess the model across several metrics: 12V-Subject Consis- tency, 12V-Backrruund Consistency, and video quality. The video quality evaluation considers dimensions such as Subject Consistency, Background Consistency, Smoothness, Aesthctic Score, Imaging Quality, Tem- poral Flickering, and Dynamic Degree. Given that our model lacks text supervision, we omit the evaluation for video-text camera motion. Furthermore, since our model is pre-trained without incorporating dynamic degree guidance (known as motion score/strength), is not directly comparable with other models in this respect. Therefore, we additionally report video quality by averaging all the dimensions except for Dynamic Degree and provide the VBench average score derived from 12V-Subject Consistency, 12V-Backgrund Con- sistency, and the video quality dimensions (excluding dynamic degree). For the latency analysis, we ensure fairness by using the same computational platform: a single Nvidia A100 80G GPU. All implementations are based on their official code without any engineering optimizations. For MarDini, we simply employ bf16 mixed precision to enhance computational efficiency. To account for variations in frame number and resolution, all results are normalized by frame count and evaluated at consistent resolution of either [512 x 512] or [768 768).
E
Additional Related Works
This paper also relates to asymmetric neural architectures, widely used in neural networks since the 1990s (Schmidhuber 1992a b). In computer vision, achieve high-resolution generation, many studies (Podell
al. 2023 Pernias et al. 2024 Saharia et al. 2022 Li al. 2024 Jain et al. 2024 Kang al. 2023 employ
common strategy: model generates www-rrsooutionnquallie samples, followed by another model that
performs super-resolution (Kang et al. 2023 refinement (Podell et al. 2023), interpolation (Wang et al. 2024b) to enhance the generation quality. In discriminative video models, asymmetric training strategies have been used for temporal segmentation models, where the full temporal extension does not fit the available GPU memory Xu el al. (2021). Since computational costs are distributed across stages, this approach is well-supported by existing computational platforms. Building on this trajectory but extending beyond it, we propose novel design that partitions the model into distinct models: a planning model and generation model. The planning model, containing the majority of the model's parameters, is trained rugresegyyssyvelyy at low resolution to generate conditional signals without producing visual outputs. These signals are then processed by the lightweight generation model, which converts them into high-resolution visual outputs using a diffusion process.
Unlike the traditional auto-regressive diffusion model (Li et al., 2024), which still faces high compu- fational costs resolution increases, use cross- attention as an information pathway to connect asymmetric resolution input for more efficient train- ing/inference.
Training Loss
Gen. Model-130mm
Gem Gem--Meeeel-30n Meddel-300mn
Gen. Model-600m
Gen, Model-1B
0.08
0.06
F
Discussions on DM Size
0.04
Based on recent findings (Liang et al. 2024), which demonstrated that pre-training loss is closely cor- related with final generation performance, we adopt pro-training loss proxy metric to evaluate model performance. In this study, we fixed our pre-trained 3B MAR model the planning component and
0.03
0.5
1.5
2.5 Training Step
3.5
4
4.5
-10
Figure 8: Training Loss by Model Size. We consider four DM sizes, ranging from 150M to 1B paramcters.
21

---

Published in Transactions on Machine Learning Research (05/2025)
trained different DM model sizes using the same filtered Shutterstock dataset. All models trained under identical configurations, including a batch size of 512, video clips of 4 frames, a resolution of 256, and a mask ratio ranging from 0.65 to 1.00. Training was conducted on 16 H100 GPUs and evaluated at the same selected training steps in each training configuration. Consistent with prior studies, larger DM models deliver superior performance but the cost of increased lalency and training instabilities. Notably, the 150M DM model a bit too small, showing persistently higher loss values even at the later stage of training. The 300M DM model (used in our paper) achieves a balanced trade-off, providing performance comparable to larger models while significantly reducing latency and training instabilities. For applications with larger inference budget, increasing the DM model size could further improve performance, though with diminishing returns at larger scales.
Reference Frames (First, Last)
Generated Frames
Figure 9: Visualization of novel view synthesis conditioned on the two views. Starting with two views of an object, MarDini generates the intermediate "frames", creating novel views. Notably, MarDini is trained without any 3D data but still manages capture spatial information through video. The data is sourced from public research data. (Downs et al. 2022).
Reference Frames
Cenerated Frames
Figure 10: Visualization of Video Expansion. The model is conditioned on sequence of 16 consecutive frames to predict the subsequent 12 frames. The video data used for visualization sourced from public research data (Nan et al. 2024).
G Other Applications
MarDini is a novel video generation model that unifies MAR and DM into single framework. This study reveals key insights and identifies specific weaknesses in this prototype, guiding our future research directions.
22

---

Published in Transactions on Machine Learning Research (05/2025)
Zero-Shot 3D View Synthesis We demonstrate MarDini's potential for 3D view synthesis. Although trained solely on video data, MarDini shows preliminary spatial understanding, suggesting its possibilities for 3D applications. In Figure g two views of a fixed object serve as the first and last reference frames, while intermodiate frames are generated, as in our vidco interpolation task. The model effectively generates convincing 3D-consistent views, highlighting its promising potential for 3D generation. Notably, no camera motion control signals are used, and we will explore MarDini on 3D data with better control in the future work.
Video Expansion MarDini integrates many of MAR's advantages, inchuding the support for video ex- pansion, where the conditional input is a set of frames rather than a single image. In this setup, motion information implicitly embedded the input. As shown in Figure I0 MarDini can effectively predict video sequences based on the provided motion cues (e.g., flower blooming, grass growing).
(Hierarchical) Auto-Regressive Generation By utilizing MAR for high-level planning, MarDini also supports auto-regressive inference, generating more frames beyond the one defined in the training stage. We demonstrate this through hicrarchical auto-regrcssive generation: starting with given vidco, we segment it into multiple clips, expand each clip segment, and treal. the expanded clip segment the new video for recursive video interpolation. In supplementary material, provide some examples where, starting with 4 images, MarDini with 17-frame window size auto-regressively expands them into framfe video (64x expansion). This illustrates that our model is not limited by the training window size, highlighting its potential for long-range video generation.
Generated Middle Frames
Inforence Steps
Figure 11: Generated Frames Across Inference Steps. The model generates intermediate frames between the first and last conditions. Columns show frames 1, 4, 8, 12, and 17; rows show results after 1, 5. 10, 15, and 25 steps.
23

---

Published in Transactions on Machine Learning Research (05/2025)
H Additional Visualization
To better understand our model's behavior, we present generated frames at different inference steps. As shown in Fig- 11 the model achieves reasonably accurale motion after just 10 of the 25 total steps. In subsequent steps, it progressively refines frame details. Since the results in Fig. 4 suggest that motion is primarily guided by the planning model, we believe this explains why satisfactory results are achieved within 10 steps (see Fig. 5al. The planning model effectively outlines the motion in advance, simplifying the diffusion process.
24


Reviews:
[{"review_id": "lfAE5jD5n8", "reviewer": "Reviewer_9X6H", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S:\n\n* The combination of MAR and diffusion in a modular, asymmetric design is novel and addresses both temporal modeling and spatial generation efficiently.\n* A single model is capable of performing various video generation tasks by simply varying the frame masking pattern, showcasing strong flexibility and generalization.\n* MarDini achieves SOTA performance in video interpolation benchmarks and performs competitively in image-to-video generation, with significantly reduced inference cost compared to heavier diffusion-based baselines.\n\nW:\n\n* The current evaluations could be more comprehensive. For video interpolation, recent strong baselines like ToonCrafter should be included; for image-to-video generation, comparisons with CogVideoX would make the evaluation more convincing, especially considering their wide recognition and scaling benchmarks.\n* Given the known limitations of VBench and perceptual metrics like FVD or IS, especially for nuanced tasks such as interpolation, a user study would significantly strengthen the empirical claims.\n* An interesting aspect of MarDini is that it does not require pretraining on large-scale image models. This opens up the possibility of training the model from scratch on narrow-domain datasets. For instance, one could consider domain-specific applications like accident video synthesis (e.g., AVD2: Accident Video Diffusion for Accident Video Description, ICRA 2025). Demonstrating MarDini's effectiveness in such settings would underscore its practical flexibility and could inspire adoption beyond general video generation tasks.\n* The hybridization of auto-regressive and diffusion paradigms is a thoughtful and technically elegant solution to combining long-range planning with high-quality generation. However, in the long term, it remains an open question whether hybrid architectures can scale as efficiently as cleaner, unified paradigms (e.g., fully AR or fully diffusion). The authors’ insights into this trade-off—especially in terms of scaling laws, inference efficiency, and training convergence—would be a valuable addition to the broader discussion on the future of general-purpose video models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces MarDini, a video generation framework that combines Masked Auto-Regressive (MAR) modeling with Diffusion Models (DM) in an asymmetric architecture. The core idea is to use a MAR-based temporal planning module—operating on low-resolution inputs—for long-range sequence modeling, while a lightweight diffusion-based generator performs high-resolution spatial refinement via denoising. This design allows MarDini to support a wide range of video generation tasks—such as video interpolation, image-to-video generation, and video expansion—through a flexible frame-masking mechanism. The paper demonstrates that MarDini achieves state-of-the-art performance in video interpolation, while maintaining high efficiency during inference.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness section in the last box.", "broader_impact_concerns": "As with other generative models, MarDini raises concerns regarding the potential misuse of synthesized video content (e.g., misinformation, deepfakes).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LJxThP0EHe", "reviewer": "Reviewer_3oxA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- It effectively combines masked autoregressive and diffusion models with an asymmetric architecture.\n- It supports multiple tasks (video interpolation, image-to-video generation, and video expansion) within a unified framework.\n- Achieves competitive performance with significantly reduced inference costs.\n- Successfully transitions from simple interpolation to complex video generation, enabling training from scratch without image-based pre-training.\n- Comprehensive experiments and ablation studies demonstrating model effectiveness.\n\nWeaknesses:\n- Lack of more analysis on alternative architecture designs (e.g., different resolutions or MAR-to-DM ratios).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a generative model combining Masked Auto-Regressive planning and Diffusion Models for video synthesis. Its contributions include an asymmetric architecture using a heavy-weight MAR at low-resolution to guide a lightweight DM for high-resolution generation, resulting in flexible and efficient handling of tasks like video interpolation, image-to-video synthesis, and video expansion. An Identity Attention mechanism ensures stable training. Empirical results show state-of-the-art performance in video interpolation and efficiency in image-to-video generation without image-based pre-training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please conduct additional ablations on architectural choices, including varying MAR-to-DM parameter ratios and different resolutions, to justify the current design decisions.\n- Please provide a deeper analysis of how Identity Attention stabilizes training, ideally with additional visualizations or quantitative evidence demonstrating convergence improvement.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "S0KEVAgbUo", "reviewer": "Reviewer_dbvZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n+ The paper is well-written and presents a complete exposition of the proposed method.\n+ The applications are interesting and relevant to the field.\n+ The idea of integrating masked generative transformers with diffusion models is promising.\n\n**Weaknesses:**\n\n+ **Clarity Issues:** The paper states, \"Auto-regressive (AR) transformers... sparking efforts to achieve similar breakthroughs in computer vision (Rombach et al., 2022; Dai et al., 2023a; Saharia et al., 2022),\" but Rombach et al., 2022 (a U-Net-based latent diffusion model) does not seem directly relevant to the AR mechanism mentioned in this context.\n+ **Questionable Motivation:** The paper claims that training instability due to reliance on discrete representations is a key bottleneck, yet no evidence is provided to support this. Based on my experience with masked generative transformers, including MaskGIT and its variants, I have not observed such instability—these models use cross-entropy loss, which is generally stable. This weakens the motivation for the proposed approach.\n+ **Scalability Claims:** The paper highlights the advantage of not requiring image-based pretraining. However, given the availability of strong foundation models trained on large-scale data in a self-supervised manner, incorporating such pretraining could be beneficial rather than restrictive. Thus, this contribution is not as significant as suggested.\n+ **Figure 1 Confusion:** The visualized image before and after the VAE encoder appears identical, which is misleading. Since VAE outputs feature maps rather than reconstructed images, the figure should be revised for clarity.\n+ **Increased Complexity:** The proposed method is more complex than existing approaches, requiring separate model configurations for Transformers and a multi-stage training process that is not end-to-end. This reduces both flexibility and scalability.\n+ **Lack of Ablation Studies:** The paper states that increasing the masking ratio creates more challenging generation tasks but provides no ablation study to justify this design choice. A comparison with simpler masking strategies (e.g., fixed or randomly sampled masks) is needed to assess its impact on performance.\n+ **Weak Empirical Performance:** The proposed method performs worse on multiple tasks/datasets, failing to convincingly demonstrate its superiority. Moreover, compared to masked non-auto-regressive models like MAGVITv2, its efficiency remains questionable. A thorough comparison of model parameters, inference speed, and memory usage is needed to justify its advantages over existing approaches.\n+ **Missing Comparisons:** Several important works in image-to-video and video interpolation, such as AnimateDiff [1] and MAGVIT v1, v2 [2,3], are missing from the comparisons. Their inclusion would provide a more comprehensive evaluation.\n\n**Overall Assessment:**\n\nWhile the paper presents an interesting combination of masked generative transformers and diffusion models, its advantages remain unconvincing due to weak motivation, increased complexity, and lack of empirical support. A more thorough justification, additional ablations, and stronger comparisons with existing methods are needed to clarify its contributions.\n\n**References:**\n\n[1] AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning, ICLR 2024\n\n[2] MAGVIT: Masked Generative Video Transformer, CVPR 2023\n\n[3] Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation, ICLR 2024", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents **MarDini**, a video diffusion model that integrates masked auto-regression (MAR) within a unified diffusion framework, where MAR handles temporal planning and diffusion focuses on spatial generation. MarDini follows an asymmetric design: a MAR-based planning model generates low-resolution planning signals, while a lightweight diffusion model refines high-resolution frames. This flexible approach enables a single model to handle video interpolation, image-to-video generation, and video expansion by conditioning on masked frames at any positions. With its efficient resource allocation, MarDini achieves state-of-the-art video interpolation and generates high-quality videos in just a few inference steps, rivaling more expensive models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical Changes (Necessary for Acceptance)**\n\n1. **Clarify the Motivation and Claims about Training Instability**\n+ The paper states that discrete representations cause training instability, yet no evidence is provided.\n+ If this claim is central to the motivation, the authors should include empirical results demonstrating such instability.\n+ If the claim is not well-supported, it should be revised or removed to avoid misleading conclusions.\n2. **Provide Justification for the Masking Strategy**\n+ The paper suggests that gradually increasing the masking ratio improves performance, but no ablation study is presented.\n+ The authors should conduct an ablation study comparing different masking strategies, such as fixed-ratio random masking, to justify this design choice.\n3. **Improve Empirical Comparisons**\n+ The paper does not sufficiently demonstrate the superiority of the proposed method.\n+ A comprehensive efficiency comparison should be added, including: \n    - Inference speed with baseline mentioned below (item 5)\n    - Model parameters\n    - Memory usage\n+ The results should be benchmarked against strong baselines such as MAGVIT v1, v2 and AnimateDiff to ensure a fair assessment.\n4. **Revise Figure 1 for Accuracy**\n+ The visual depiction of the VAE encoder is misleading, as it suggests that the output is an image rather than a feature map.\n+ The figure should be corrected to accurately reflect the nature of the latent representation.\n5. **Expand Related Work and Comparative Discussion**\n+ Several relevant works in image-to-video and video interpolation are missing.\n+ The authors should include comparisons with the following models: \n    - AnimateDiff (ICLR 2024)\n    - MAGVIT v1, v2 (CVPR 2023, ICLR 2024)\n+ A discussion of how the proposed method differs from or improves upon these approaches is necessary.\n---\n**Suggested Changes (Would Strengthen the Paper)**\n\n6. **Refine the Clarity of Section 2 (\"Scalability\")**\n+ The paper claims that the proposed method does not require image-based pretraining, but it does not explain why this is an advantage.\n+ Since many powerful self-supervised models exist, the authors should clarify whether their method can integrate pretraining or if it offers a trade-off between scalability and performance.\n7. **Simplify the Training Pipeline if Possible**\n+ The method introduces a multi-stage training process with separate transformer-based models, making it more complex than many existing approaches.\n+ If an alternative, more streamlined training paradigm exists, the authors should discuss or explore it.\n8. **Improve Writing Clarity and Technical Accuracy**\n+ Some statements are overgeneralized or imprecise, particularly in the introduction and motivation.\n+ The discussion of Auto-Regressive (AR) transformers should be revised, ensuring that references are appropriately aligned with the context.\n9. **Additional Visualizations for Model Interpretability**\n+ Since the method integrates masked auto-regression and diffusion, it would be beneficial to include visualizations of: \n    - How planning signals evolve over time in the MAR component.\n    - Intermediate frames generated by the diffusion process to illustrate the model’s behavior.\n10. **Ablation on Model Components**\n+ The model combines masked generative transformers and diffusion models, but it is unclear how much each contributes to performance.\n\n---\n**Summary of Key Revisions**\n\n**Critical:** Justify training instability claims, add masking ablations, expand empirical comparisons, revise Figure 1, and include missing related works.\n\n**Suggested:** Clarify scalability, simplify training if possible, refine writing, improve visualizations, and conduct additional ablations.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors presented a new group of diffusion models for video generation, by integrating the masked auto-regression (MAR) in a diffusion framework. The key idea and main contribution is around the MAR-based planning approach, together with the diffusion spatial generation. Experiments on video interpolation, image-to-video generation, and video expansion show the effectiveness and efficiency of the proposed model.

The paper was reviewed by three expert reviewers, followed by a discussion phase between the authors and reviewers, after which the major concerns from the reviewers were mostly addressed by the authors. The reviewers are satisfied with the authors' further response and clarifications, all recommending positive scores (3 Leaning Accept).
Considering the potential impact of this work, it would be great if the authors could release their models or implementation. The AE and reviewers understand the authors' concerns about open sourcing and acknowledged the authors' clarification on this. But the authors are suggested to add a paragraph to clarify this with details about the open-sourcing plan and constraints, in their final version.
Decision: Accept (poster)