MarDini: Masked Autoregressive Diffusion for Video Generation at Scale

## Abstract

Abstract We introduce MarDini, a new family of video diffusion models that integrate the advantages of masked auto-regression (MAR) into a unified diffusion model (DM) framework. Here, MAR handles temporal planning, while DM focuses on spatial generation in an asymmetric network design: i) a MAR-based planning model containing most of the parameters generates planning signals for each masked frame using low-resolution input; ii) a lightweight generation model uses these signals to produce high-resolution frames via diffusion de-noising. MarDini’s MAR enables video generation conditioned on any number of masked frames at any frame positions: a single model can handle video interpolation (e.g., masking middle frames), image-to-video generation (e.g., masking from the second frame onward), and video expansion (e.g., masking half the frames). The efficient design allocates most of the computational resources to the low-resolution planning model, making computationally expensive but important spatio-temporal attention feasible at scale. MarDini sets a new state-of-the-art for video interpolation; meanwhile, within few inference steps, it efficiently generates videos on par with those of much more expensive advanced image-to-video models.

## 1 Introduction

Auto-regressive (AR) transformers (Vaswani et al., ; Peng et al., ; Schmidhuber, ; Schlag et al., ) have recently demonstrated remarkable success in natural language processing (Dubey et al., ; Team et al., ; Achiam et al., ), sparking efforts to achieve similar breakthroughs in computer vision (Rombach et al., ; Dai et al., ; Saharia et al., ). However, unlike the discrete, sequential, and easily tokenized nature of language, visual data consist of continuous pixel signals distributed across a high-dimensional space, making them more difficult to model through 1D auto-regression.

To overcome this challenge, recent studies have explored vector quantization techniques (Van Den Oord et al., ; Razavi et al., ) to convert continuous pixel data into discrete representations suitable for AR modelling. Unfortunately, these approaches (Yu et al., ; Ramesh et al., ) rely on causal attention, which is not well aligned for high-dimensional visual data, often leading to diminished performance (Li et al., ), particularly on large-scale datasets (Xie et al., ; Zhou et al., ). To mitigate this limitation, masked auto-regression (MAR) has been introduced (Chang et al., ; Li et al., ). MAR replaces the causal attention with bi-directional attention (He et al., ; Devlin et al., ), effectively simulating auto-regressive behaviour while being more capable of handling visual data. Leveraging this approach, MAR exhibits flexibility in handling diverse generation tasks through different masking strategies, such as image generation (Chang et al., ; Li et al., ), out-painting (Chang et al., ), video expansion (Yu et al., ) and class-conditioned video generation (Yu et al., ; Voleti et al., ) while maintaining manageable computational overhead. Although MAR shows potential in scaling image and video generation tasks (Chang et al., ; Yu et al., , ), its key bottleneck lies in its training instability which is tied to the reliance on discrete representations (Ramesh et al., ; Razavi et al., ).

Meanwhile, Diffusion models (DMs) (Ho et al., ; Neal, ; Jarzynski, ) have emerged as a successful alternative for scaling vision generative models, offering stable training by modelling visual signals directly in a continuous space. However, DMs tend to incur high inference costs due to the requirement of the multi-step diffusion process. Here, video generation poses an even greater challenge — Video is a strict super-set of the image domain, requiring additional modelling for temporal consistency and complex motion dynamics.

To this end, we propose a new paradigm for video generation that combines the flexibility of MAR in a continuous space with the robust generative capabilities of DM. Specifically, we present a scalable training recipe and an efficient neural architecture design for video generation. Our model decomposes video generation into two sub-tasks — temporal and spatial modelling — handled by distinct networks with an asymmetric design based on the following two principles:

- 1.
MAR handles long-range temporal modelling, while DM focuses on detailed spatial modelling.
- 2.
MAR operates with more parameters at a lower resolution, while DM operates with fewer parameters at a higher resolution.

Following these principles, we use the same training batch for both MAR and DM but employ two distinct processes operating at different resolutions.
MAR receives randomly masked low-resolution input frames and predicts the corresponding planning signals.
Conditioned on these planning signals via cross-attention and the unmasked frames, DM learns to incrementally recover the masked high-resolution frames from noise. Finally, we introduce a progressive training strategy that gradually curates mask ratios and with its data pipelines, allowing our model to be trained from scratch on unlabeled video data. This eliminates the common reliance on text-to-image and text-to-video pre-training, as seen in other video diffusion models (Girdhar et al., ; Blattmann et al., ).

Our model integrates MAR-based planning signals with a DiT-based (Peebles and Xie, ; Chen et al., ) lightweight, tiny diffusion model, hence the name MarDini. Our empirical study on MarDini highlights the following key characteristics:

- •
Flexibility. With MAR conditioning, MarDini naturally supports a range of video generation tasks through flexible masking strategies. For example, when given the first frame and masking the rest, it performs image-to-video generation; when given a video and masking subsequent frames, it performs video expansion; and, when given the first and last frames and masking the middle frames, it performs video interpolation. By hierarchically and auto-regressively masking middle frames across multiple inferences, MarDini generates slow-motion videos.
- •
Scalability. MarDini can be trained from scratch at scale, without relying on generative image-based pre-training. In contrast to most video generation models, that treat video as a secondary task following image generation, MarDini leverages mask ratio tuning to progressively adjust the difficulty of the training task. This approach enables the model to scale from video interpolation to full video generation, directly bypassing the need for image-based pre-training.
- •
Efficiency. MarDini’s asymmetric design allocates more computational resources to lower resolutions, making it memory-efficient and fast during inference. With lower overall memory usage, MarDini allows the deployment of computationally intensive spatio-temporal attention mechanisms at scale, improving its ability to model complex motion dynamics.

## 2 MarDini: An Efficient and Asymmetric Video Diffusion Model

### 2.1 Design Overview

MarDini is a video generation model designed to efficiently generate high-resolution videos using an asymmetric network architecture. As shown in Figure 1, MarDini consists of two networks: a heavy-weight MAR planning model and a light-weight generation DM. During training, the planning network processes randomly masked low-resolution frames and predicts corresponding planning signals. These planning signals compress the semantic and long-range temporal information, guiding the DM’s high-resolution generation process. The DM receives noisy frames at the masked positions and reconstructs them by progressively removing noise.

In this section, we outline and address the key design challenges involved in training MarDini. First, we describe the data representations and their corresponding notations within the MarDini framework (Section 2.2). Next, we describe the design details of the MAR planning network and the DM, along with the integration of additional guidance such as diffusion steps and planning signals (Section 2.3). Finally, we outline the multi-stage training recipe for MarDini, which we found to be essential for ensuring stable training (Section 2.4). Collectively, these innovations enable MarDini to become one of the first video generation models capable of being trained from scratch using only unlabelled video data.

Figure: Figure 1: MarDini Training Pipeline Overview. A latent representation is computed for unmasked frames that serve as a conditional signal to a generative process. On the first hand, we have a planning model that autoregressively encodes global conditioning signals from a low-resolution version of the unmasked latent inputs. On the other hand, the planning signals are fed to the diffusion-based generation model through cross-attention layers. A high-resolution version of the input conditions is also ingested by the diffusion model, enabling generation with a coherent temporal structure and a direct mechanism to attend to fine-grained details of the unmasked frames. MarDini is trained end-to-end via masked frame-level diffusion loss.
Refer to caption: x1.png

### 2.2 Data Representation and Notations

##### VAE Compressor.

Consistent with prior works (Dai et al., ; Girdhar et al., ), we adopt a pre-trained Variational Auto-Encoder (VAE) (Kingma and Welling, ), denoted by $\mathcal{D}_{\text{enc}}$, to compress videos into a low-dimensional continuous latent space, which improves both training and inference efficiency. Our VAE employs a 16-channel latent dimension with an 8$\times$ spatial compression rate to preserve spatial details, following Dai et al. (). The VAE outputs are then patchified into a shape of $N\times C$, where $N$ represents the token count and $C=16$ represents its latent dimension.

##### MAR Planning Model.

Given a low-resolution input video $\mathbf{X}_{\text{low}}=\{x^{\text{low}}_{i}\}_{i=1:K}$ with $K$ frames, we apply the VAE encoder to compress the frames into their corresponding latent representations: $\mathbf{Z}_{\text{low}}=\{z^{\text{low}}_{i}\}_{i=1:K}=\mathcal{D}_{\text{enc}
}(\mathbf{X}_{\text{low}})$. To train the MAR planning model $\mathcal{P}$, we randomly select $K^{\prime}<K$ video latents $\{z^{low}_{j}\}_{j=1:K^{\prime}}\in\mathbf{Z}_{\text{low}}$ and replace them with a learnable mask token [MASK], resulting in the final masked low-resolution latent inputs $\mathbf{Z}_{\text{low}}^{\text{mask}}$. The planning model then processes $\mathbf{Z}_{\text{low}}^{\text{mask}}$ and predicts $\mathbf{Z}_{\text{cond}}=\mathcal{P}(\mathbf{Z}_{\text{low}}^{\text{mask}})=\{
z^{\text{cond}}_{i}\}_{i=1:K}$, where $z^{\text{cond}}_{i}$ is the planning signal for the $i$-th frame, shaped as $N_{\text{low}}\times C_{\text{low}}$, with $N_{\text{low}}$ representing the number of patches per frame.

##### DM Generation Model.

Conversely, we obtain high-resolution video latents $\mathbf{Z}_{\text{high}}=\{z^{\text{high}}_{i}\}_{i=1:K}=\mathcal{D}_{\text{
enc}}(\mathbf{X}_{\text{high}})$ with dimensions $N_{\text{high}}\times C_{\text{high}}$, generated by the VAE encoder using the same video inputs at high resolution: $\mathbf{X}_{\text{high}}=\{x^{\text{high}}_{i}\}_{i=1:K}$. Notably, we have $N_{\text{high}}\gg N_{\text{low}}$. At diffusion step $t$, we sample noise and add it to $K^{\prime}$ frames that were masked in the planning model (denoted by [NOISE]), leaving the remaining $K-K^{\prime}$ reference frames unchanged (denoted by [REF]). This produces the final noisy high-resolution video latent inputs $\mathbf{Z}_{\text{high}}^{\text{noise},t}$. Then, the generation model $\mathcal{G}$ processes these latent inputs $\mathbf{Z}_{\text{high}}^{\text{noise},t}$ and performs a standard denoising step, where we denote the DM output at time step $t$ as $\mathcal{G}(\mathbf{Z}_{\text{high}}^{\text{noise},t},\mathbf{Z}_{\text{cond}}
,t)$.

### 2.3 Architecture Design

In this section, we provide a comprehensive explanation of the MarDini architecture, including its detailed design, model configurations, and variations.

#### 2.3.1 MarDini Block Design

Figure: Figure 2: MarDini Design Details. MarDini employs a transformer architecture for both the planning and generation models, incorporating a DiT-style block for the generation model and a Llama-style block for the planning model. We set $L_{1}\gg L_{2}$, where $L_{1}$ and $L_{2}$ refer to the number of layers in the planning and generation model respectively.
Refer to caption: x2.png

Figure 2 illustrates the design of the MarDini’s MAR and DM models, both of which are based on the transformer architecture (Vaswani et al., ).

In the MAR planning model, we adhere to the design conventions established in Llama models (Dubey et al., ; Touvron et al., ), which apply RMS-Norm (Zhang and Sennrich, ) to normalize the inputs of each attention block. Additionally, layer normalization (Ba et al., ) is applied to normalize the projected features in multi-head attention, enhancing training stability. Due to the use of low-resolution inputs, we manage to directly employ spatio-temporal attention, allowing tokens to attend across frames. This design is feasible only with asymmetric resolution inputs, as it prevents excessive memory consumption.

Concretely, within each attention block in MAR, we utilize rotary positional encoding (RoPE) (Su et al., ) to encode both the spatial and temporal positions of the video tokens. To accomplish this, we apply a 2D RoPE to encode the 3-dimensional video data. Specifically, we flatten the image patches into a 1-dimensional token sequence and insert a learnable [NEXT] token to differentiate image patches across different rows, following Gao et al. (). This design effectively handles video data with varying aspect ratios and resolutions.

We design the DM model in alignment with MAR, but with three key differences. First, we adopt a DiT-style approach (Peebles and Xie, ), using AdaIN (Huang and Belongie, ) to integrate the diffusion steps as a conditional signal within the spatial attention layers, and additionally added with the MAR’s planning signal within the MLP layers. Second, we introduce a cross-attention layer to process the planning features predicted by the MAR model. Lastly, we replace spatio-temporal attention with temporal attention (Blattmann et al., ) to reduce the computational cost associated with high-resolution inputs in DM.

#### 2.3.2 Identity Attention

Figure: Figure 3: Identity Attention Design Details in DM. In this setup, [REF] tokens only attend to themselves, while [NOISE] tokens attend to all other tokens across different frames.
Refer to caption: x3.png

In our initial experiments, we observed significant training instability in MarDini’s DM. We speculate that this is due to two main factors: i) the inherent distributional disparity between noisy ([NOISE]) tokens and clean reference ([REF]) tokens, which is further amplified by the stochastic nature of sampling diffusion steps; and ii) the random positions and varying lengths of these [NOISE] tokens. These factors likely compound, potentially disrupting the DM’s training signals and hindering the model’s ability to converge efficiently.

To address this challenge, we introduce Identity Attention, which enables the model to easily distinguish between [REF] and [NOISE] tokens by employing a separate attention strategy. As illustrated in Figure 3, [REF] tokens simply serve as an identity projection, preserving the input reference frames without attending to other tokens. In contrast, [NOISE] tokens possess a global view, attending to tokens across all frames. The [REF] tokens serve as guidance for generation, so we design them to be isolated from other tokens, while [NOISE] tokens provide global attention to all conditional signals for generation. We incorporate Identity Attention in both the spatio-temporal layers of MAR and the temporal layers of DM, which has been found to significantly enhance training stability in both models.

#### 2.3.3 Model Configuration

As outlined in Table 1, this study develops four models with distinct configurations. We train two planning models with 3.1B and 1.3B parameters alongside two generation models, employing spatio-temporal or temporal attention mechanisms. To align with our asymmetric design between the planning and generation models, the generation model’s parameter size is reduced to $3\times$ or $10\times$ smaller than that of the planning model. Due to the high computational cost of spatio-temporal attention, we limit MarDini-L/ST and MarDini-S/ST to a 9-frame length for fair comparison on VIDIM-Bench (Jain et al., ). Importantly, the model’s ability to autoregressively generate samples ensures that the length of the output video is not constrained.

**Table 1: Configuration Details of MarDini Models. We provide four models, differing primarily in the size of the planning module (3.1B vs. 1.3B parameters) and the attention mechanisms used in the generation module: spatio-temporal attention (S.-T. Attn.) vs. temporal attention (T. Attn.).**
| Configuration | Planning Model (MAR) | Generation Model (DM) | Frame |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Depth | Hidden Size | MLP Size | Attn. | Param. | Depth | Hidden Size | MLP Size | Attn. | Param. |  |  |
| MarDini-S/ST | 8 | 4096 | 4096 | S.-T. Attn. | 1.3B | 8 | 1024 | 4096 | S.-T. Attn. | 288M | 9 |
| MarDini-L/ST | 16 | 4096 | 8192 | S.-T. Attn. | 3.1B | 8 | 1024 | 4096 | S.-T. Attn. | 288M | 9 |
| MarDini-S/T | 8 | 4096 | 4096 | S.-T. Attn. | 1.3B | 8 | 1024 | 4096 | T. Attn. | 288M | 17 |
| MarDini-L/T | 16 | 4096 | 8192 | S.-T. Attn. | 3.1B | 8 | 1024 | 4096 | T. Attn. | 288M | 17 |

### 2.4 MarDini Training Recipes

In this section, we outline the training pipeline of MarDini. Specifically, we employ a multi-stage progressive training strategy that gradually increases task difficulty. This approach offers two key benefits: i) progressive learning inherently enhances training stability and improves the performance of generative models, as demonstrated by Karras () and Chen et al. (); and ii) it allows for the collection of checkpoints from earlier stages, which helps mitigate setbacks caused by suboptimal configurations. Below, we elaborate on our detailed progressive training strategy, including the training objectives, architecture design, and training data configurations. A comprehensive training manual for MarDini is shown in Figure 4, with detailed hyper-parameters and optimization methods further outlined in the Appendix 8.

Figure: Figure 4: MarDini Training Manual. We list the mask ratios, frame rate (FPS), number of frames, and the size of training data for each training stage. This training manual applies to both small (MarDini-S) and large (MarDini-L) models. Note that the total training data refers to the amount of data observed by the model for gradient updates, rather than the vanilla size of the training dataset. Our final model checkpoints are highlighted in gray.
Refer to caption: x5.png

#### 2.4.1 Training Tasks: From Frame Interpolation to Video Generation

Our training objectives are organized into three stages: i) Initial Stage: We separately train the planning and generation models, each with its own learning objective, to initialize their model weights. ii) Joint-Model Stage: We combine the models for joint training on a simple video interpolation task, using only a masked diffusion loss. iii) Joint-Task Stage: We further train the model by gradually reducing the number of preserved reference frames, enabling it to jointly learn video interpolation and image-to-video generation tasks.

##### Initial Stage.

Wang et al. () pointed out that transformers with a large parameter count often experience unstable training. As such, we simplify the training dynamics by separately warming up the two models as an initial step.

To optimize generation model $\mathcal{G}$, we employ a masked diffusion loss $\mathcal{L}_{\text{DM}}$:

$$ $\displaystyle\mathcal{L}_{\text{DM}}^{\theta}=||\mathbf{M}\cdot\mathbf{V}^{t}- \mathbf{M}\cdot\mathcal{G}_{\theta}(\mathbf{Z}_{\text{high}}^{\text{noise},t}, \mathbf{Z}_{\text{uncond}},t)||_{2}^{2},$ (1) $$

where $\mathbf{Z}_{\text{uncond}}$ is a learnable token serving as unconditional guidance from the planning model. $\theta$ represents the parameters of the generation model, and $\mathbf{M}$ denotes the binary masks used to mask out all clean reference frames. Inspired by Blattmann et al. (); Salimans and Ho (), we apply velocity prediction as the diffusion loss, where the prediction target $\mathbf{V}^{t}=\{v_{i}^{t}\}_{i=1:K}$ represents the velocity at time step $t$ for the $i$-th frame, defined as $v_{i}^{t}=\alpha_{t}\epsilon-\sigma_{t}z^{\text{high}}_{i},\epsilon\sim
\mathcal{N}(0,I)$. Here, $\alpha_{t}$ and $\sigma_{t}$ correspond to the diffusion scheduler at $t$ step.

To optimize MAR planning model $\mathcal{P}$, we employ a masked reconstruction loss $\mathcal{L}_{\text{MAR}}$:

$$ $\displaystyle\mathcal{L}^{\phi,\zeta}_{\text{MAR}}=||\mathbf{M}\cdot\mathbf{Z} _{\text{low}}-\mathbf{M}\cdot f_{\zeta}(\mathcal{P}_{\phi}(\mathbf{Z}_{\text{ low}}^{\text{mask}})||_{2}^{2}.$ (2) $$

where $f$ denotes a projection layer that depatchifies the model predictions to match the resolution of the low-resolution input image $\mathbf{Z}_{\text{low}}$. $\phi,\zeta$ represent the learnable parameters of the planning model and the projection layer respectively. Note that, $f$ is only used during the initial training stage, and will be removed in the later training stages.

##### Joint-Model Stage.

After the initial pre-training stage, we then jointly train the planning and generation models end-to-end using a unified masked diffusion learning objective $\mathcal{L}_{\text{MDiff}}$:

$$ $\displaystyle\mathcal{L}_{\text{MDiff}}^{\theta,\phi}=||\mathbf{M}\cdot\mathbf {V}^{t}-\mathbf{M}\cdot\mathcal{G}_{\theta}(\mathbf{Z}_{\text{high}}^{\text{ noise},t},\mathcal{P}_{\phi}(\mathbf{Z}_{\text{low}}^{\text{mask}}),t)||_{2}^{ 2},$ (3) $$

where $\mathbf{Z}_{\text{cond}}=\mathcal{P}(\mathbf{Z}_{\text{low}}^{\text{mask}})$ is the planning signal predicted by MAR. In order to enable classifier-free guidance (Ho and Salimans, ) on the planning signal, we maintain a fixed probability of $\nicefrac{{1}}{{10}}$ to randomly replace $\mathbf{Z}^{t}_{\text{cond}}$ with $\mathbf{Z}_{\text{uncond}}$.

##### Joint-Task Stage.

In the final training stage, we reuse the learning objective from the previous stage, but gradually decrease the masking ratio to induce more challenging generation tasks. Here, mask ratio refers to the proportion of frames preserved during training. This stage requires a significantly larger computational resources with higher-resolution videos, as it determines the model’s final performance. By gradually decreasing the masking ratios, we smoothly transform the model’s task from video interpolation to single-image-to-video generation. This procedure ultimately enables the model to generate videos with a variable number of input frames at arbitrary temporal locations.

#### 2.4.2 DM Architecture: From Spatio-Temporal to Temporal Attention

In conjunction with our progressive training objectives, we also introduce a progressive architectural design. Specifically, we first use spatio-temporal attention in the DM during the initial training stage. This choice promotes convergence, compared to temporal attention, as noted in Gao et al. (). Since in our initial stage we train the DM in isolation and on a relatively low-resolution setup, this sophisticated attention incurs in minor computational overhead. When integrating MAR with the DM in the second stage, we replace the spatio-temporal attention with the more cost-effective temporal attention, thus increasing the efficiency of the generation model.

#### 2.4.3 Data: Progressive Configuration of Specifications

Analogous to our progressive strategies for training objective and architecture we also propose a progressive data configuration. Over time, we gradually increase the video’s spatial resolution, alongside progressively extending the video’s duration. This approach ensures efficient use of computational resources and facilitates effective model scaling, allowing MarDini to handle more complex and high-resolution video data as training progresses.

## 3 Experiments

We evaluate MarDini on two benchmarks: VIDIM-Bench (Jain et al., ), for long-term video interpolation, and VBench (Huang et al., ) for image-to-video generation. We further elaborate on the specifics of these benchmarks in Appendix 10. We highly encourage referring to the generated videos in our web page for a comprehensive understanding of the quality of the generated videos.

### 3.1 Ablation Studies and Analysis

##### Effectiveness of MAR and DM.

We demonstrate the importance of having a DM on top of our MAR planning model. In fact, it is tempting to hypothesize that MAR on its own contains all the ingredients to enable high-quality video interpolation. To explore this, we introduce a projection layer to directly unpatchify the output of the MAR model without intermediate diffusion. Our experiments on VIDIM-Bench reveal that, MAR on its own, performs poorly on interpolation tasks, as shown by the first two and last two rows in Table 3.1, for both the 1B and 3B settings. This result suggests that directly applying MAR to continuous space is suboptimal, a result consistent with previous findings (Li et al., ). Similarly, directly tackling this task with a small DM without global guidance, according to the third row of Table 3.1, results in sub-optimal performance. However, by combining MAR’s planning capability with DM’s stable performance in continuous space, we achieve optimal results, demonstrating that both components are beneficial for video generation.

**Table 2: Effectiveness of MAR and DM design. The reported results are FVD on VIDIM-Bench. All experiments are evaluated at a resolution of $[256\times 256]$ using DDIM scheduler with 25 steps.**
| Method | DAVIS-7 | UCF101-7 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MidF-SSIM | MidF-LPIPS | FID | FVD | MidF-SSIM | MidF-LPIPS | FID | FVD |  |
| AMT (Li et al., ) | 0.4853 | 0.2865 | 34.65 | 234.50 | 0.7903 | 0.1691 | 31.60 | 344.50 |
| RIFE (Huang et al., ) | 0.4546 | 0.2954 | 23.98 | 240.04 | 0.7769 | 0.1564 | 18.72 | 323.80 |
| FILM (Reda et al., ) | 0.4718 | 0.3048 | 30.16 | 214.80 | 0.7869 | 0.1620 | 26.06 | 328.20 |
| LDMVFI (Danier et al., ) | 0.4175 | 0.2765 | 22.10 | 245.02 | 0.7712 | 0.1564 | 18.09 | 316.30 |
| VIDIM (Jain et al., ) | 0.4221 | 0.2986 | 28.06 | 199.32 | 0.6880 | 0.1768 | 34.48 | 278.00 |
| MarDini-S/ST-256 | 0.4249 | 0.3654 | 49.21 | 224.07 | 0.7654 | 0.2480 | 45.85 | 258.08 |
| MarDini-L/ST-256 | 0.4959 | 0.2768 | 20.64 | 102.87 | 0.7734 | 0.2213 | 28.85 | 197.69 |
| MarDini-S/ST-512 | 0.5017 | 0.3193 | 25.92 | 138.86 | 0.7960 | 0.2315 | 30.24 | 205.71 |
| MarDini-L/ST-512 | 0.5314 | 0.2736 | 20.76 | 99.05 | 0.7814 | 0.2347 | 30.08 | 204.20 |
| MarDini-L/T-512 | 0.5085 | 0.3083 | 25.30 | 117.13 | 0.7893 | 0.2270 | 30.72 | 198.94 |

##### Efficiency Analysis.

Table 3.1 illustrates latency and memory usage across different input resolutions and frame lengths, measured on the same computational platform. When MAR is set to operate symmetrically with the DM with the same inputs, the model cannot fit in the available GPU memory as we increase the resolution and/or number of frames. In contrast, our asymmetric design enables the generation of 12-frame clips at 512 resolution in just a few seconds. The rapid generation process is partially attributed to the DM requiring relatively few inference steps to converge, thanks to the well-structured planning signal it receives, as shown in Figure 5(a). Notably, inference speed could be further optimized, as the only acceleration technique we incorporated during our experiments is mixed precision, without employing caching strategies (Liu et al., ; Zhao et al., ), FSDP, or static compilation of the underlying computational graph. Similarly, memory usage could be further reduced through CPU offloading, sliced attention, sequential VAE inference, etc.

##### Explaining MAR’s Planning Signal.

We provide an intuitive explanation of MAR’s role in MarDini. During training, a learnable token is used to randomly replace MAR to support CFG (Ho and Salimans, ), allowing DM to generate videos independently. We visualize the results of MarDini with and without planning signals. As shown in Figure 5, without the planning model, DM can still produce meaningful frames but, as expected, lacks “global planning.” In Figure 5 (Left), DM moves objects in different directions, causing distortion in the building, which suggests a weaker or non-existing prior model of how objects move. Similarly, in Figure 5 (Right), DM fails to accurately predict the movement of the fire. In contrast, incorporating the planning signal addresses these visual flaws. These results indicate that MAR’s planning signal effectively hints how elements should move, ensuring long-term coherence in the generated video.

Figure: Figure 5: MarDini’s generations with and without the planning model. Here we show video frames generated when conditioning on the middle frame. Without MAR’s planning signal, DM generates degraded motion, such as pixel distortions (highlighted in red, left) or incorrect motions (highlighted in blue, right).
Refer to caption: x6.png

Figure: (a) Video interpolation results with varying inference steps.
Refer to caption: x8.png

##### From Video Interpolation to Image-To-Video Generation.

Our training recipe follows the philosophy of transitioning from video interpolation to image animation. Herein, we empirically demonstrate that these two tasks are related, validating the soundness of our pipeline. As shown in Figure 5(b), we track the performance of MarDini on both video interpolation and image animation during a training phase aimed at scaling the resolution from 256 to 512. This stage marks the first point during training where the model successfully performs both tasks simultaneously. We observe a promising consistency between the performance of image animation and video interpolation, providing solid evidence that these tasks do not hinder each other. Furthermore, with a carefully tuned mask ratio, the model can be trained in a unified manner to efficiently achieve both tasks.

##### Impact of Identity Attention.

We explore the effectiveness of Identity Attention in handling our specific data format, which integrates both reference frames and noised frames into a single sequence. As illustrated in Figure 5(c), we track the training trajectory in the early stages of the DM generation model. We recognize that this type of input can lead to unstable training, particularly when starting from scratch, as the differences between reference frames are difficult to discern. However, the proposed Identity Attention mechanism mitigates this instability. The decrease in training loss observed after 6K steps is attributed to the use of a warm-up learning rate, where the learning rate is intentionally kept low during the initial steps.

### 3.2 Results on Video Interpolation

**Table 4: Performance of zero-shot video interpolation on VIDIM-Bench. The reported results are taken directly from VIDIM (Jain et al., ). AMT, RIFE, and FILM are single-inference methods, while LDMVFI, VIDIM, and our approach are based on diffusion models with multiple inference steps. MidF-SSIM and MidF-LPIPS represent the SSIM and LPIPS scores, respectively, for the middle frame. For MarDini-512, we downscale the generated videos to 256 resolution for a fair comparison.**
| Method | DAVIS-7 | UCF101-7 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MidF-SSIM | MidF-LPIPS | FID | FVD | MidF-SSIM | MidF-LPIPS | FID | FVD |  |
| AMT (Li et al., ) | 0.4853 | 0.2865 | 34.65 | 234.50 | 0.7903 | 0.1691 | 31.60 | 344.50 |
| RIFE (Huang et al., ) | 0.4546 | 0.2954 | 23.98 | 240.04 | 0.7769 | 0.1564 | 18.72 | 323.80 |
| FILM (Reda et al., ) | 0.4718 | 0.3048 | 30.16 | 214.80 | 0.7869 | 0.1620 | 26.06 | 328.20 |
| LDMVFI (Danier et al., ) | 0.4175 | 0.2765 | 22.10 | 245.02 | 0.7712 | 0.1564 | 18.09 | 316.30 |
| VIDIM (Jain et al., ) | 0.4221 | 0.2986 | 28.06 | 199.32 | 0.6880 | 0.1768 | 34.48 | 278.00 |
| MarDini-S/ST-256 | 0.4249 | 0.3654 | 49.21 | 224.07 | 0.7654 | 0.2480 | 45.85 | 258.08 |
| MarDini-L/ST-256 | 0.4959 | 0.2768 | 20.64 | 102.87 | 0.7734 | 0.2213 | 28.85 | 197.69 |
| MarDini-S/ST-512 | 0.5017 | 0.3193 | 25.92 | 138.86 | 0.7960 | 0.2315 | 30.24 | 205.71 |
| MarDini-L/ST-512 | 0.5314 | 0.2736 | 20.76 | 99.05 | 0.7814 | 0.2347 | 30.08 | 204.20 |
| MarDini-L/T-512 | 0.5085 | 0.3083 | 25.30 | 117.13 | 0.7893 | 0.2270 | 30.72 | 198.94 |

We compare MarDini with the existing methods on the VIDIM benchmark (Jain et al., ) for video interpolation, where the task is to generate 7 frames between a starting and an ending conditional frames. As shown in Table 4, MarDini achieves competitive performance among different evaluation metrics. In particular, it is widely acknowledged that generative models often underperform in reconstruction metrics, with blurrier images often scoring higher despite receiving lower ratings from human observers (Sahak et al., ; Watson et al., ; Jain et al., ; Saharia et al., ).
We also study a sample that is exemplifying of this statement in the Appendix 7. Therefore, we place greater emphasis on the generative metric, FVD, where MarDini outperforms competitors and achieves state-of-the-art performance. Notably, MarDini-L/T employs an asymmetric attention mechanism, where the planning model utilizes spatio-temporal attention, while the generation model relies on temporal attention. Compared to the model that uses spatio-temporal attention for both models (MarDini-L/ST), the results suggest that the asymmetric attention mechanism does not significantly affect performance, achieving a satisfactory trade-off between efficiency and quality. We provide additional visualizations in Appendix 9 and the supplementary materials.

### 3.3 Results on Image-to-Video Generation

In this section, we evaluate our model’s single-image-to-video generation capabilities in comparison with other methods using the VBench dataset (Huang et al., ). As shown in Table 5, our method performs competitively, especially in terms of latency, despite incorporating expensive spatio-temporal attention. For fairness, latency is calculated with the same resolution. In this study, we focus on validating the soundness of our proposed roadmap, only considering the initial pre-training stage rather than delving into post-training techniques. As a result, we do not incorporate additional conditional signals such as language instructions or motion score guidance. Therefore, direct comparisons on video quality, particularly in relation to dynamic degree, are not entirely fair. However, we fully report these numbers for reference.

**Table 5: Image-to-Video Performance on VBench. The reported results of baseline methods are sourced from VBench (Huang et al., ). For fair latency comparison, we standardize the input size to [512$\times$512] for low and medium resolutions, and [768$\times$768] for high resolution cases across all methods. All other metrics were collected using the original resolutions reported in the first column.**
| Method | Frame<br>Resolution | Image-based<br>Pre-training | Latency<br>(s/frame) | I2V Sub.<br>Con | I2V Back<br>Con. | Video Quality<br>(w/ D.D.) | Video Quality<br>(w/o D.D.) | Vbench<br>Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Low and Medium Resolution |  |  |  |  |  |  |  |  |
| ConsistI2V (Ren et al., ) | [256$\times$256] | ✓ | 7.63 | 95.82 | 95.95 | 78.87 | 85.74 | 88.27 |
| DynamicCrafter (Xing et al., ) | [256$\times$256] | ✓ | - | 97.05 | 97.56 | 80.18 | 85.00 | 88.07 |
| DynamicCrafter (Xing et al., ) | [512$\times$320] | ✓ | 4.88 | 97.21 | 97.40 | 81.63 | 85.39 | 88.37 |
| SEINE (Chen et al., ) | [512$\times$320] | ✓ | - | 96.57 | 96.80 | 79.49 | 85.71 | 88.45 |
| VideoCrafter (Chen et al., ) | [512$\times$320] | ✓ | 9.43 | 91.17 | 91.31 | 81.34 | 87.55 | 88.47 |
| SEINE (Chen et al., ) | [512$\times$512] | ✓ | 5.13 | 97.15 | 96.94 | 80.58 | 87.13 | 89.61 |
| Animate-Anything (Dai et al., ) | [512$\times$512] | ✓ | 1.58 | 98.76 | 98.58 | 81.21 | 88.84 | 91.30 |
| MarDini-L/ST-9 | [512$\times$512] | ✗ | 2.24 | 98.64 | 97.12 | 80.84 | 88.22 | 90.64 |
| MarDini-S/ST-9 | [512$\times$512] | ✗ | 2.24 | 99.04 | 97.23 | 81.00 | 88.59 | 90.98 |
| MarDini-L/T-17 | [512$\times$512] | ✗ | 0.48 | 98.23 | 97.01 | 80.25 | 87.68 | 90.16 |
| MarDini-S/T-17 | [512$\times$512] | ✗ | 0.46 | 98.76 | 97.18 | 80.56 | 88.17 | 90.62 |
| High Resolution |  |  |  |  |  |  |  |  |
| SVD-XT-1.0 (Blattmann et al., ) | [1024$\times$576] | ✓ | 2.19 | 97.52 | 97.63 | 82.79 | 86.54 | 89.30 |
| SVD-XT-1.1 (Blattmann et al., ) | [1024$\times$576] | ✓ | 2.19 | 97.51 | 97.62 | 82.23 | 86.66 | 89.38 |
| I2VGen-XL (Zhang et al., ) | [1280$\times$720] | ✓ | 6.01 | 96.48 | 96.83 | 81.17 | 87.02 | 89.43 |
| DynamiCrafter (Xing et al., ) | [1024$\times$576] | ✓ | 7.13 | 98.17 | 98.60 | 82.52 | 87.31 | 90.08 |
| MarDini-L/T-17 | [768$\times$768] | ✗ | 1.01 | 98.34 | 96.63 | 80.88 | 88.22 | 90.54 |
| MarDini-S/T-17 | [768$\times$768] | ✗ | 0.98 | 98.77 | 96.78 | 81.29 | 88.68 | 90.95 |
| MARDini-L/T-17 | [1024$\times$1024] | ✗ | - | 98.61 | 96.34 | 81.35 | 88.69 | 90.89 |
| MARDini-S/T-17 | [1024$\times$1024] | ✗ | - | 98.78 | 96.46 | 81.74 | 88.97 | 91.13 |

We also report the results on the benchmark without the motion score (referred to as Dynamic Degree in VBench). All evaluation metrics are detailed in Appendix 10. The empirical study shows MarDini’s strong potential, performing on par with other existing methods across several metrics while exhibiting higher efficiency and requiring no generative image pre-training. Interestingly, we observe that MarDini-S marginally outperforms MarDini-L on some evaluation metrics. We speculate that this is due to MarDini-L requiring more training time to accommodate higher-resolution data. Nonetheless, we observe clear advantages in scaling the MAR model size, as MarDini-L outperforms in video interpolation and generates image-to-video results that better align with physical principles. A list of generated video samples is provided in the supplementary for further reference.

### 3.4 Additional Applications

In this section, we explore some of MarDini’s additional intriguing capabilities and applications. While we did not conduct rigorous ablation studies or quantitative comparisons, this serves as an initial exploration, highlighting potential directions for future research.

##### Zero-Shot 3D Novel View Synthesis

We demonstrate MarDini’s strong potential for 3D novel view synthesis. Although trained solely on video data, MarDini exhibits a preliminary level of spatial understanding, suggesting its potential for 3D applications. In Figure 7, two views of a fixed object serve as the first and last reference frames, while intermediate frames are generated, as similar to our video interpolation task. The model effectively generates convincing 3D-consistent views, highlighting its promising potential for 3D generation. Notably, no camera control signals are used, and we will explore MarDini on 3D data with better control in the future work.

Figure: Figure 7: Visualization of novel view synthesis conditioned on the two views. Starting with two views of an object, MarDini generates the intermediate “frames”, effectively creating novel views. Notably, MarDini is trained without any 3D data but still manages to capture spatial information through video. The data used for this task is sourced from publicly available research datasets (Downs et al., ).
Refer to caption: extracted/5949981/images/video_interpolation/bag_1.png

Figure: Figure 8: Visualization of Video Expansion. The model is conditioned on a sequence of 16 consecutive frames to predict the subsequent 12 frames. The video data used for visualization is sourced from publicly available research dataset (Nan et al., ).
Refer to caption: extracted/5949981/images/video_extension/flower_1.png

##### Video Expansion

MarDini integrates many of MAR’s advantages, including the support for video expansion, where the conditional input is a set of frames rather than a single image. In this setup, motion information is implicitly embedded in the input. As shown in Figure 8, MarDini can effectively predict video sequences based on the provided motion cues (e.g., flower blooming, grass growing).

##### (Hierarchical) Auto-Regressive Generation

By utilizing MAR for high-level planning, MarDini also supports auto-regressive inference, generating more frames beyond the one defined in the training stage. We demonstrate this through hierarchical auto-regressive generation: starting with a given video, we segment it into multiple clips, expand each clip segment, and treat the expanded clip segment as the new video for recursive video interpolation. In Figure 11 (in Appendix), we provide an example where,
starting with 4 images, MarDini with a 32-frame window size auto-regressively expands them into a 128-frame slow-motion video (32$\times$ expansion). This illustrates that our model is not limited by the training window size, highlighting its potential for long-range video generation.

## 4 Related Work

##### Auto-Regressive Model in Visual Generation.

Auto-regressive (AR) models (Gers et al., ; Hochreiter and Schmidhuber, ; Schmidhuber, ) have proven effective in natural language modeling (Brown, ; Achiam et al., ; Dubey et al., ; Team et al., ).
To adapt this scalable modeling strategy for image and video generation, recent approaches (Yu et al., ; Chang et al., ; Li et al., ; Yu et al., ; Chang et al., ; Yu et al., ) replace causal attention in AR with bidirectional attention, allowing for better capture of dense relationships in visual space.

Many studies (Yu et al., ; Chang et al., ; Team, ; Xie et al., ) validate the scalability of this approach.
To align with the training recipes from LLMs, these studies adopt discrete visual representations, using image tokenizers (Esser et al., ; Yu et al., ; Van Den Oord et al., ) to quantize continuous pixel values into discrete representations. However, Li et al. (); Ramesh et al. (); Razavi et al. () argue that this strategy suffers from unstable training and may limit model capacity due to the inherently continuous nature of visual data. This inspires recent works (Li et al., ; Zhou et al., ) to shift towards continuous latent spaces for masked auto-regressive models to address these limitations.

We follow this trajectory but diverges in two ways: i) We highlight the importance of mask ratios, which were fixed in earlier works Li et al. (). By dynamically adjusting them with a progressive training strategy, we improve both model scalability and stability. ii) We propose an asymmetric input resolution design, allowing MAR to be effectively trained with full-resolution inputs.

##### Diffusion Model for Video Generation.

In recent years, diffusion models (Ho et al., ; Neal, ; Jarzynski, ) have become a leading approach for both image and video generation (Rombach et al., ; Dhariwal and Nichol, ; Ramesh et al., ; Chen et al., ; Saharia et al., ; Brooks et al., ; Dai et al., ; Girdhar et al., ; Menapace et al., ; Kondratyuk et al., ; Cong et al., ).
These models conceptualize the generation process as gradually refining a real sample from Gaussian noise, demonstrating significant scalability and stable training.

In this paper, we offer two key insights into video generation: i) Previous methods (Wu et al., ; Ho et al., ; Zhang et al., ; Blattmann et al., ; Wang et al., ; Girdhar et al., ; Gao et al., ; Cong et al., ) often first pre-train an image generative model, and then fine-tune it for video generation, or they require joint training for both tasks (Chen et al., ; Esser et al., ). While multi-stage pre-training on diverse inputs can be beneficial, video generation is often limited by the success of image-based pre-training, which typically serves as a secondary task. This paper proposes an alternative: training video generation models from scratch with progressively increasing task complexity. ii) Previous research (Girdhar et al., ; Wang et al., ; Chen et al., ; Blattmann et al., ) has predominantly employed temporal attention mechanisms to capture temporal dependencies, mainly due to the high computational and memory costs associated with spatio-temporal attention. However, in alignment with previous work (Blattmann et al., ; Gao et al., ) suggesting that spatio-temporal attention enables superior video modelling, we propose an amortized strategy that makes spatio-temporal attention computationally feasible, even at high resolutions.

##### Asymmetric Neural Networks.

This paper also relates to asymmetric neural architectures, widely used in neural networks since the 1990s (Schmidhuber, , ). In computer vision, to achieve high-resolution generation, many studies (Podell et al., ; Pernias et al., ; Saharia et al., ; Li et al., ; Jain et al., ; Kang et al., ) employ a common strategy: a model generates low-resolution/quality samples, followed by another model that performs super-resolution (Kang et al., ), refinement (Podell et al., ), or interpolation (Wang et al., ) to enhance the generation quality. In discriminative video models, asymmetric training strategies have been used for temporal segmentation models, where the full temporal extension does not fit the available GPU memory Xu et al. (). Since computational costs are distributed across stages, this approach is well-supported by existing computational platforms.
Building on this trajectory but extending beyond it, we propose a novel design that partitions the model into two distinct models: a planning model and a generation model. The planning model, containing the majority of the model’s parameters, is trained auto-regressively at a low resolution to generate conditional signals without producing visual outputs. These signals are then processed by the lightweight generation model, which converts them into high-resolution visual outputs using a diffusion process.

Unlike the traditional auto-regressive diffusion model (Li et al., ), which still faces high computational costs as resolution increases, we use cross-attention as an information pathway to connect asymmetric resolution input for more efficient training/inference.

## 5 Limitations and Future Works

##### Post Training.

The primary goal of this paper is to demonstrate the feasibility and effectiveness of combining masked auto-regressive (MAR) models with diffusion models (DM) for video generation. Consequently, we allocated the majority of our computational resources to the pre-training stage, placing less emphasis on post-training, despite its recognized importance in generative models (Dai et al., ; Dubey et al., ; Touvron et al., ). Post-training will be a top priority in our future work, focusing on enhancing long-term planning, improving motion quality, and achieving higher resolutions.

##### Improved Conditional Signals.

A significant contribution of this work is the exploration of training a video generation model without relying on generative image pre-training. However, this approach presents a trade-off: MarDini is not inherently equipped with a text encoder for processing language-based instructions. To conserve computational resources and quickly validate the feasibility of our method, we intentionally excluded commonly used conditional signals, such as text embeddings and motion scores. Encouraged by the initial success of our model, we plan to incorporate these conditional signals into MarDini in our future updates to broaden its range of applications.

## 6 Conclusion

We have introduced a new family of generative models for video, i.e., MarDini, based on auto-regressive diffusion, wherein a large planning model offers powerful conditioning to a much smaller diffusion model. Our design philosophy considers efficiency from model conception, and so our heaviest model component is only executed once at lower resolution inputs, whereas our generative module focuses on fine-grained details at the frame level, reconciling high-level conditioning and image details. Our model is unique in that it leverages a masked auto-regressive loss directly at the frame level. MarDini is afforded with multiple generative capabilities from a single model, e.g., long-term video interpolation, video expansion, and image animation. Our investigation shows that our modeling strategy is powerful enough to obtain competitive results on various interpolation and animation benchmarks, while doing it at a lower computational needs than counterparts with comparable parameter size.

## Acknowledgements

The authors thank Mingchen Zhuge, Jinheng Xie, Yuren Cong, Kam Woh Ng, Aditya Patel, and Jinjie Mai for their valuable suggestions and contributions to the paper review. Haozhe Liu and Jürgen Schmidhuber were supported by funding from the King Abdullah University of Science and Technology (KAUST) - Center of Excellence for Generative AI under award number 5940 and the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence.

## Ethics Statement

This paper explores the theoretical foundations of neural architecture design for video generation, rather than being tied to specific commercial applications. Consequently, the potential negative impacts of MarDini align with those of other video generation models and do not pose unique risks that require special consideration. Importantly, unlike previous models trained on web-scale data, which may raise concerns about data copyright, MarDini is exclusively trained on a licensed Shutterstock dataset, without having such conflicts.

## 7 Reconstruction metrics in Video Interpolation.

In Figure 9, it appears that blurrier images sometimes receive higher reconstruction error scores.

Figure: (a) Original Video
Refer to caption: x11.png

## 8 MarDini Training Strategies

MarDini is trained on the Shutterstock video dataset with 34 million videos, using 256 H100 GPUs with a distributed MAST scheduler (). We use the AdamW optimizer for each stage with a $1.4\times 10^{-4}$ learning rate and cosine learning rate scheduler. We adapt our batch size based on the resolution and the frame count to maximize GPU utility. For example, at $[256\times 256]$ resolution with 9 frames, the batch size is 1024, processing 9K frames per iteration; at $[512\times 512]$ resolution with 9 frames, the batch size is 720, processing 6480 frames per iteration. During inference, we set the classifier-free guidance (CFG)(Ho and Salimans, ) scale as 2.5 for the image-to-video task with the noise solver DDIM (Song et al., ), and we directly remove classifier-free guidance for video interpolation as it is redundant. FSDP (Zhao et al., ) and activation checkpointing (Zhao et al., ) are enabled to further save GPU memory. We do not include dynamic resolution training in our main training stages, as it slows down training. Instead, we find that after convergence, fine-tuning the model for a few steps (10K-20K) with dynamic resolutions enables it to quickly support this capabilities.

## 9 Visualization of Video Interpolation

In Figure 10, we provide visualization results that demonstrate the superiority of MarDini in large motion modelling, compared to FILM (Reda et al., ), LDMVFI (Danier et al., ), and VIDIM (Jain et al., ).

Figure: Figure 10: Visualization of video interpolation methods conditioned on the first and last frames. We present the generated frames from FILM (Reda et al., ), LDMVFI (Danier et al., ), VIDIM (Jain et al., ), and MarDini. The comparison results for these methods are sourced from Jain et al. (). We have included additional samples in the supplementary materials.
Refer to caption: extracted/5949981/images/video_interpolation/interp_1.png

## 10 Benchmarks

We evaluate the interpolation performance on VIDIM-Bench (Jain et al., ) and assess image animation performance on VBench (Huang et al., ).

For VIDIM-Bench, the task involves generating seven intermediate frames, with the first and last frames provided as conditions. The dataset includes approximately $400$ videos from both DAVIS (Pont-Tuset et al., ) and UCF-101 (Soomro et al., ). We use FVD (Unterthiner et al., ) and FID (Heusel et al., ) as generation metrics, while adopting SSIM (Wang et al., ) and LPIPS (Zhang et al., ) as reconstruction metrics. Notably, we evaluate the middle (5th) frame for reconstruction metrics, as it presents the greatest challenge due to its distance from the reference frames.

For VBench, we utilize the official dataset to assess the model across several metrics: I2V-Subject Consistency, I2V-Background Consistency, and video quality. The video quality evaluation considers dimensions such as Subject Consistency, Background Consistency, Smoothness, Aesthetic Score, Imaging Quality, Temporal Flickering, and Dynamic Degree. Given that our model lacks text supervision, we omit the evaluation for video-text camera motion. Furthermore, since our model is pre-trained without incorporating dynamic degree guidance (known as motion score/strength), it is not directly comparable with other models in this respect. Therefore, we additionally report video quality by averaging all the dimensions except for Dynamic Degree and provide the VBench average score derived from I2V-Subject Consistency, I2V-Background Consistency, and the video quality dimensions (excluding dynamic degree). For the latency analysis, we ensure fairness by using the same computational platform: a single Nvidia A100 80G GPU. All implementations are based on their official code without any engineering optimizations. For MarDini, we simply employ bf16 mixed precision to enhance computational efficiency. To account for variations in frame number and resolution, all results are normalized by frame count and evaluated at a consistent resolution of either [512 $\times$ 512] or [768 $\times$ 768].

Figure: Figure 11: Visualization of MarDini using hierarchical auto-regressive generation. Starting with an initial 4 frames, MarDini auto-regressively generates a complete 128-frame video, demonstrating its capability to extend beyond the training window size (32 frames here).
Refer to caption: x14.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: I don’t have any other questions. I hope that the authors can concisely respond to my concerns mentioned above.
soundness: 3
strengths: 1.	This paper presents a novel video generation framework that separates low-res temporal planning and high-res spatial generation via asymmetric MAR and DM models, effectively balancing the computation cost in this task. In general, I do agree with the three characteristics/strengths of MarDini listed in the Introduction, i.e., the flexibility, scalability, and efficiency advantages.
2.	The large flexibility of the proposed framework offers huge room for exploration, not only from the architecture side but also the applications (e.g., as shown in the Appendix, the model can potentially be generalized to other tasks such as zero-shot 3D view synthesis and video expansion). This paper can likely inspire future research in many domains.
3.	In-depth analyses are presented to reveal the model’s behavior and justify the design choices, nicely supporting the arguments.
4.	The paper has set new SoTAs on VBench for image-to-video generation and VIDIM-Bench for zero-shot video interpolation, further justifying the effectiveness of the proposed framework.
5.	Overall, I think this paper has been nicely written, methods, results, and visualizations are clearly presented.
summary: This paper presents a new framework for video generation, MarDini, which integrates masked auto-regression (MAR) and diffusion model (DM). Specifically, it separates the low-res planning and high-res generation loads into a heavy-weight but efficient MAR and a lightweight but expensive DM, effectively balancing the computation of scaling video generation. To stabilize training, the paper also proposes Identity Attention, which preserves the input reference frames without attending to other tokens, and a multi-stage training strategy. The method has been evaluated for video interpolation, image-to-video generation, and video expansion. Visualization is provided in the supplemental materials.
weaknesses: Overall, I am happy with this paper, but considering the scale of experiments (256 H100 GPUs for training), which is very large, there are a few concerns I wish to point out:

1.	Video length. It seems that the method is only evaluated on very short videos, mostly generating 9 or 17 frames at low FPS. It is unclear that the method, in particular the temporal planning, can generalize well to longer videos.
2.	Conditioning. The paper claims that the method does not require image-based pre-training, but I guess this is mainly because the paper has chosen to address the video interpolation, video expansion, and image-to-video generation tasks. As mentioned in Future Works, if we want to enable the model to do text-conditioning, then we might have to introduce abundant image data to learn the rich semantics. It is unclear whether the model can still work well with text conditions (or condition signals in other modalities) and generate long videos, especially when the DM is very lightweight.
3.	Training difficulty. I am happy to see that the paper deeply investigates the training strategy and found feasible solutions. But this also makes me concerned about the stability and training difficulty of such a model, especially since this might be very unfriendly for future works that try to follow or extend this MarDini.
4.	Visualizations. I carefully reviewed all the videos provided in the Supplemental materials, but most of the videos are very short and static, and are scenery or single objects that are relatively easy to generate. For such a large-scale model (3.1B parameters, trained with 256 H100), I expect to see much more complex and successful examples such as high dynamics, complex scenes, or human activities. Additionally, I am not fully satisfied with the Figure 4 videos that compare with/without planning. In this (picked) example, buildings are much better for with planning, but the ferry still has obvious distortion.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - The model naturally lends itself to the tasks of video inpainting. I.e. considering frames with partial masks. Did the authors consider such scenario? Considering partial masks would enable model training on larger datasets of images that would improve performance.
- LL123-126 highlight using only videos without labels as a model advantage. The model however shows low quality video generations that could have been improved through joint image/video training and considering text captions that can be automatically generated for the training data. Could the authors discuss why the chosen approach is desirable?
- Why does the MAR model present a Spatial-only attention block? Modern T2V generators tend to use spatial blocks only as a way to preserve generation capabilities if the model was retrained on image-only data. Why would the MAR model benefit from Spatial-only attention blocks that would have been traded for more Spatio-Temporal blocks for which importance is advocated in the paper?
- Why does the Temporal attention block in the DM not have modulation applied to it?
- Why does modulation involve shifting? As modulation is applied after RMSNorm operations it would be natural to only consider scale operations. This would come at the benefit of reducing the amount of parameters devoted to modulation by 33%
- LL194 mention the use of Layer Normalization. Is layer normalization used to perform QK normalization? Why do the authors not employ RMSNorm for QK normalization as previous work (e.g. Stable Diffusion 3)? This would be a more logical choice given the use of RMSNorm layers as the main normalization mechanism.
- LL200-203 mention the use of [NEXT] tokens as Lumina-T2X (Gao 2024). In their follow up work Lumina-Next, the authors advocate for the removal of the mechanism in favor of 3D RoPE. Why was this approach adopted in this work?
- LL345 could the authors clarify how FSDP would contribute to inference speedups? FSDP would only allow sharing of model parameters to reduce memory usage, but would require additional parameter gathering overhead to achieve this. Please elaborate
- LL292 does not appear to be well justified. Could the authors describe why it would not be more desirable to train the DM from scratch with such architecture?
- In Fig. 7, the discussed training recipe for the "Initial Stage" of the DM comprises continuous changes in the set of training data (20M Clips -> 80M Clips -> 40M Clips) and FPS (8 -> 4 -> 8). The MAR is trained on 200M clips. Could the authors describe the rationale for adopting the chosen strategy and discuss how readers should instantiate their training recipe in practice?
- The proposed idea of using a MAR model as a feature extractor is interesting. The MAR model needs to only be run a single time during the whole DM sampling process. However, single-step distillation (see "SF-V: Single Forward Video Generation Model") of a large video DM also appears as viable route for achieving good quality-cost tradeoff with a single evaluation of a large model. Could the authors describe why MAR + small DM represent a more attractive solution than large video DM + single-step distillation?
soundness: 2
strengths: - The authors treat the relevant problem of improving efficiency of DM-based video generators
- The proposed idea of using a MAR model as a feature extractor is interesting
- The paper is well-written, though many choices are not fully justified
- Ablations show that the small DM benefits from access to MAR features, validating the design choice.
summary: The authors propose a method for image animation, video interpolation and extrapolation. The novelty of the method lies in the use of a small diffusion model (DM) augmented by the features representations (planning signals) produced by a large masked autoregressive (MAR) model operating in low resolution. By allocating a large amount of compute to the MAR model, diffusion sampling can happen efficiently, as the MAR model needs to be executed only once. The proposed research question of improving the efficiency of DMs is of large interest to the community, though the current work only tackles image animation, video interpolation and extrapolation, without treating the most widespread text-to-video scenario. The paper is mostly clearly written. The high level method idea is sound, though several architectural and training recipe details are not fully justified, reducing soundness of the approach.
weaknesses: - Absolute quality of the generated videos as shown in the supplementary material is low
- Tab 1. presents evaluation of the method vs a DM with same compute and no MAR. A missing baseline would be comparing the full approach with a DM of a capacity rescaled to match the compute of DM + MAR. Without such baseline it is not possible to assert that the combination of DM + MAR outperforms DM under equal computational constraints.
- Several details in the architecture and training recipe are unclear or appear not sound (see questions)
- Fig 5 (c) shows bad convergence behavior, even after the proposed fixes. The behavior is justified by the use of warmup. Warmup is unlikely to generate a sharp transition at 6k steps as the one shown in the plot which appears to be more likely justified by optimization challenges for the current architecture, raising doubts on how scalable and user-friendly the proposed framework would be.
- In Tab. 3. The model underperforms LDMVFI and VIDIM in MidF-LPIPS and FID. A discussion in LL430 attributes this to defective metrics behavior with respect to blur. This appears unconvincing as, differently from PSNR and SSIM, LPIPS and FID would strongly be penalized by the presence of blur. In particular, FVD, would have a behavior substantially similar to FID, being both distribution based metrics. Lower performance under these metrics raises concerns about the quality of the method. The sample in Fig. 6 does not convincingly show the author's point. I would suggest showing multiple samples from regions containing large amounts of high frequency details where differences would appear more clearly.
- Text to video generation is not treated in this work
- (minor) The employed VAE does not perform temporal compression, contrary to modern T2V generators performing 4x (Cogvideo-X) to 8x (MovieGen) temporal compression. This appears as a lost opportunity for speedup. Note that temporal compression still allows for masking on each frame independently, by applying masks to the input video before encoding.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * Do you have video samples with more deformable motion (e.g. human running / doing stuff, or animals moving around)? Most examples shown in the supplementary show motion related to fire / water / clouds, camera translations, or object translation - of which are generally the easier things to learn for a video generation model.
* Instead of the initial training stage, would it possibly work if the model was jointly trained from scratch if the MAR simultaneously optimized the MSE loss? Could the diffusion model also be stabilized if cross attention was initialized to zero / a zero-init scaling term was learned (if that was not done already)?
* Is masked autoregression used during generation? E.g. similar to MaskGit, where in this case you would progressively sample subsets of frames following some schedule If not, what is the masked autoregression aspect of the model (what is autoregressive here)?
soundness: 3
strengths: * The paper is generally well-written, and clear
* The architecture is simple, and a smaller diffusion head can help speed up inference by caching MAR outputs.
* Experimental results show better performance in various masked generation tasks, such as image-video generation and video interpolation, as well as ablations to show benefits of their model design
summary: The paper proposes a novel masked autoregressive diffusion model architecture that leverages a planning component which processes downsampled masked videos to learn high-level + long-term dependencies, coupled with a smaller diffusion model that learns high-resolution spatial dependencies. In addition, the authors a progressive training strategy for better training stability.
weaknesses: * I am not yet convinced of the scalability of this model. From Table (4), it seems that the “L” variants of MarDini perform worse than the “S” variants? Is this correct?
* Related to the above point, would this model have a harder time scaling to learn distributions that are more stochastic? Since a vast majority of parameters are in MAR and not in the diffusion model. To leverage the greater number of params in the MAR, the MAR would need to encode as many possible futures into its representation for the diffusion model to read from, which may be difficult if the complexity of the distribution grows (unconditional video generation where the MAR output is not useful / constant, or simultaneously predicting / interpolating longer videos).
* The training recipe is a little bit complicated, requiring a combination of stages such as separate / joint training, and different masking ratios.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Directly using a pre-trained VideoMAE to initialize the MAR branch may sound more elegant and efficient for several reasons:

    ① Leverage Pre-trained Features: VideoMAE is pre-trained on a large corpus of video data and has already learned spatiotemporal features. Initializing the MAR branch with VideoMAE would allow the model to start with a strong foundation, potentially leading to faster convergence and better performance, especially for tasks related to video generation.

    ② Reduced Training Time: By starting with a pre-trained model, the MAR branch wouldn't need to learn basic spatiotemporal representations from scratch, reducing the overall training time and computational cost.

2. Table 4 has typos in the last two rows: "MARDini-L/T-17" and "MARDini-S/T-17" should be corrected to "MarDini-L/T-17" and "MarDini-S/T-17" to maintain consistent naming conventions. And these two lines are missing the Latency metric.

[4] Zhan Tong and Yibing Song and Jue Wang and Limin Wang. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. Advances in Neural Information Processing Systems, 2022.

[5] Wang, Limin and Huang, Bingkun and Zhao, Zhiyu and Tong, Zhan and He, Yinan and Wang, Yi and Wang, Yali and Qiao, Yu. VideoMAE V2: Scaling Video Masked Autoencoders With Dual Masking. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023
soundness: 2
strengths: 1. This paper proposes a computationally balanced asymmetric network architecture. The MAR part has a larger number of parameters and higher computational complexity, but it requires fewer forward passes and takes low-resolution frames as input. In contrast, the DM part has fewer parameters and lower computational complexity, but requires more forward passes and models high-resolution frames. This is an ingenious approach.

2. The paper introduces Identity Attention, which cleverly addresses the issue of unstable training and theoretically reduces the computational load of the attention module (although it is unclear whether the authors actually implemented this in the code).
summary: This paper integrates masked auto-regression (MAR) and diffusion model (DM) in a ControlNet-like fashion, combining the efficient generation capability of MAR with the high-quality generation power of DM to achieve video generation conditioned on any number of masked frames. The network architecture adopts an asymmetric design, allocating most of the parameters to the MAR branch and using spatial-temporal attention. The MAR branch takes masked low-resolution frames as input and outputs conditioning signals for the DM network. The DM network, with fewer parameters and utilizing only spatial and temporal separated attention, generates masked high-resolution frames. Under the influence of the conditioning, the DM network generates the missing masked parts. This approach achieves state-of-the-art video interpolation and image-to-video performance.
weaknesses: 1. This paper does not provide sufficient ablation experiments to demonstrate the effectiveness of the progressive training strategy. At the very least, experiments are needed to show that under the same training cost, the combination of Joint-Model Stage + Joint-Task Stage outperforms the model with only the Joint-Task Stage.

2. In Table 3, the paper does not compare its results with SEINE [1], which, as far as I know, also focuses on video interpolation. 

3. Although the proposed method offers some advantages in inference latency, it introduces a large number of parameters compared to previous diffusion methods, which increases the memory requirements of the device. However, Table 3 lacks a direct comparison of memory usage.

4. Despite the introduction of numerous additional parameters, the image-to-video performance in Table 4 still lags behind previous work [2, 3].

[1] Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.

[2] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.

[3] Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Fine-grained open domain image animation with motion guidance. arXiv preprint arXiv:2311.12886, 2023b.

Meta-review: The paper presents an approach for animation, interpolation and extrapolation. The paper received two slightly negative scores and two slightly positive, suggesting its borderline status. Among major concerns were the quality of the results, very short length of the video, lack of important ablations, and analyses. And all of that at pretty high computational budget of 256 H100 cores. The AC went thought the reviewers and discussions, checked the supplement and agrees with the evaluation provided by reviewers. AC believes that the quality of results is quite low compared to current standards accepted in the community (recall the paper is submitted for publication in 2025). Lack of certain ablations makes the manuscript less valuable.

Decision: Reject