MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion

## Abstract

Abstract The spatio-temporal complexity of video data presents significant challenges in tasks such as compression, generation, and inpainting. We present four key contributions to address the challenges of spatiotemporal video processing. First, we introduce the 3D Mobile Inverted Vector-Quantization Variational Autoencoder (3D-MBQ-VAE), which combines Variational Autoencoders (VAEs) with masked token modeling to enhance spatiotemporal video compression. The model achieves superior temporal consistency and state-of-the-art (SOTA) reconstruction quality by employing a novel training strategy with full frame masking. Second, we present MotionAura, a text-to-video generation framework that utilizes vector-quantized diffusion models to discretize the latent space and capture complex motion dynamics, producing temporally coherent videos aligned with text prompts. Third, we propose a spectral transformer-based denoising network that processes video data in the frequency domain using the Fourier Transform. This method effectively captures global context and long-range dependencies for high-quality video generation and denoising.
Lastly, we introduce a downstream task of Sketch Guided Video Inpainting. This task leverages Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. Our models achieve SOTA performance on a range of benchmarks.
Our work offers robust frameworks for spatiotemporal modeling and user-driven video content manipulation. We will release the code, datasets, and models in open-source.

## 1 Introduction

Video generation refers to generating coherent and realistic sequences of frames over time, matching the consistency and appearance of real-world videos. Recent years have witnessed significant advances in video generation based on learning frameworks ranging from generative adversarial networks (GANs) (Aldausari et al., ), diffusion models (Ho et al., ), to causal autoregressive models (Gao et al., ; Tudosiu et al., ; Yu et al., ). Video generative models facilitate a range of downstream tasks such as class conditional generation, frame prediction or interpolation (Ming et al., ), conditional video inpainting (Zhang et al., ; Wu et al., ) and video outpainting (Dehan et al., ). Video generation has several business use cases, such as marketing, advertising, entertainment, and personalized training.

The video generation process includes two broad tasks. The first task is video frame tokenization, performed using pre-trained variational autoencoders (VAEs) such as 2D VAE used in Stable Diffusion or 3D VQ-VAEs.
However, these frameworks fail to achieve high spatial compression and temporal consistency across video frames, especially with increasing dimensions. The second task is visual generation. The early VAE models focused on modeling the continuous latent distribution, which is not scalable with spatial dimension. Recent works have focused on modeling a discrete latent space rather than a continuous one due to its training efficiency. Some works (e.g., Arnab et al. ()) train a transformer on discrete tokens extracted by a 3D VQ-VAE framework.
However, an autoregressive approach cannot efficiently model a discrete latent space for high-dimensional inputs such as videos.

To address the challenges of visual content generation, we propose MotionAura. At the core of MotionAura is our novel VAE, named 3D-MBQ-VAE (3D MoBile Inverted VQ-VAE), which achieves good temporal comprehension for spatiotemporal compression of videos. For the video generation phase, we use a denoiser network comprising a series of proposed Spectral Transformer blocks trained using the masked token modeling approach.
Our novelty lies both in the architectural changes in our transformer blocks and the training pipeline. These include using a 2D Fast Fourier Transform (FFT) layer and Rotary Positional Embeddings (Su et al., ). Following an efficient and robust, non-autoregressive approach, the transformer block predicts all the masked tokens at once. To ensure temporal consistency across the video latents, during tokenization, we randomly mask one of the frames completely and predict the index of the masked frame. This helps the model learn the frame sequence and improve the temporal consistency of frames at inference time. Finally, we address a downstream task of sketch-guided video inpainting.
To our knowledge, ours is the first work to address the sketch-guided video inpainting task.
Using our pre-trained noise predictor as a base and the LORA adaptors (Hu et al., ), our model can be finetuned for various downstream tasks such as class conditional video inpainting and video outpainting. Figure   MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion shows examples of text-conditioned video generation using MotionAura. Our contributions are:

- •
We propose a novel 3D-MBQ-VAE for spatio-temporal compression of video frames. The 3D-MBQ-VAE adopts a new training strategy based on the complete masking of video frames. This strategy improves the temporal consistency of the reconstructed video frames.
- •
We introduce MotionAura, a novel framework for text-conditioned video generation that leverages vector quantized diffusion models. That allows for more accurate modeling of motion and transitions in generated videos. The resultant videos exhibit realistic temporal coherence aligned with the input text prompts.
- •
We propose a denoising network named spectral transformer. It employs Fourier transform to process video latents in the frequency domain and, hence, better captures global context and long-range dependencies. To pre-train this denoising network, we add contextually rich captions to the WebVID 10M dataset and call this curated dataset WebVid-10M-recaptioned.
- •
We are the first to address the downstream task of sketch-guided video inpainting. To realize this, we perform parameter-efficient finetuning of our denoising network using LORA adaptors.
The experimental results show that 3D-MBQ-VAE outperforms existing networks in terms of reconstruction quality. Further, MotionAura attains state-of-the-art (SOTA) performance on text-conditioned video generation and sketch-guided video inpainting.
- •
We curated two datasets, with each data point consisting of a Video-Mask-Sketch-Text conditioning, for our downstream task of sketch-guided video inpainting. We utilized YouTube-VOS and DAVIS datasets and captioned all the videos using Video LLaVA-7B-hf. Then, we performed a CLIP-based matching of videos with corresponding sketches from QuickDraw and Sketchy.

## 2 Related Works on visual content generation

Image generation: Text-guided image synthesis has resulted in high quality and spatially coherent data (Esser et al., ; Sun et al., ). Generative Adversarial Networks (Sauer et al., ; Karras et al., ; ; ; Goodfellow et al., ) model visual distributions to produce perceptually rich images but fail to capture the complete spectrum of data distribution. Moreover, these networks are difficult to optimize. Kingma & Welling () introduce another approach to generating visual data wherein the decoders of the autoencoder-like architectures are trained to reconstruct images by sampling over the distributions provided by encoders. This regularization strategy helps to learn continuous and meaningful latent space. Quantization of the produced latents (van den Oord et al., ) further improves the quality of generated data using various sampling methods over the learned latents. While existing sampling methods can capture dense representations in their latent space, they fail to produce sharp, high-quality images.
Diffusion models train networks to learn a denoising process, appropriately fitting the inductive bias in image data. Recent developments such as DDPMs (Ho et al., ), latent diffusion methods (Rombach et al., ) and diffusion transformers (Esser et al., ; Peebles & Xie, ; Chen et al., ; ) have revolutionized the field of image generation.

Video generation: Video generation is even more challenging than image generation since generating high-quality videos demands both spatial and temporal consistencies.
For video generation, researchers have used both transformer models and diffusion models.
VideoGPT (Yan et al., ) generates videos autoregressively using the latents obtained from the VQ-VAE tokenizer.
Processing videos in the latent space rather than the pixel space reduces training and inference latency.
The early diffusion-based video generation models transferred weights from the TextToImage (T2I) approaches and adapted them by inflating the convolutional backbone to incorporate the temporal dynamics (Ho et al., ). Latent diffusion models (LDM) based Text2Video (T2V) approaches generate higher-quality videos of longer duration (Blattmann et al., ). However, diffusion-based approaches using convolution backbones fail to produce high frame-rate videos and generally rely on other architectures to interpolate intermediate frames.
Recent works have combined transformer-based predictions with diffusion-based pretraining (Yu et al., ; ; Kondratyuk et al., ).

## 3 MotionAura: Architecture and Training

We introduce MotionAura, a novel video generation model designed to produce high-quality videos with strong temporal consistency. At the core of MotionAura is 3D-MBQ-VAE, our novel 3D VAE that achieves high reconstruction quality. The encoder of this pretrained 3D-MBQ-VAE encodes videos into a latent space, forming the first essential component of our approach. The second novelty is a spectral transformer-based diffusion model, which diffuses the encoded latents of videos to produce high-quality videos. Section 3.1 discusses the large-scale pretraining of 3D-MBQ-VAE for learning efficient discrete representations of videos. Section 3.2 discusses pretraining of the noising predictor, whereas Section 3.3 presents the Spectral Transformer module for learning the reverse discrete diffusion process.

### 3.1 Pre-Training of 3D-MBQ-VAE

Figure: Figure 2: Our proposed pre-training method for 3D-MBQ-VAE architecture
Refer to caption: x1.png

We employed a 3D-VQ-VAE (Vector Quantized Variational Autoencoder) to tokenize videos in 3D space; refer Appendix B for the architecture of 3D-MBQ-VAE. We pretrained our network on the YouTube-100M dataset to ensure that it produces video-appropriate tokens. The extensive content variety in the YouTube-100M dataset enhances the model’s generalization and representational capabilities.
Figure 2 shows our pretraining approach for learning efficient latents with temporal and spatial consistencies. The pretraining leverages two approaches for optimal video compression and discretization: (i) Random masking of frames (He et al., ) using 16$\times$16 and 32$\times$32 patches.
This training method ensures self-supervised training of the autoencoder to efficiently learn spatial information. (ii) Complete masking of a single frame in the sequence to enforce learning of temporal information.
Let $B$ be the batch size, $C$ be number of channels, $N$ the number of frames per video, $H$, and $W$ be the height and width of the video frames, respectively.
For a given video $V\in\mathbb{R}^{(B,N,3,H,W)}$, a randomly selected frame $f_{i}\in\mathbb{R}^{(3,H,W)}$ is masked completely. The encoder $\mathcal{E}(V)$ returns $z_{c}\in\mathbb{R}^{(B,C,N,H/16,W/16)}$. This compressed latent in the continuous domain is fed into the MLP layer to return $P(f_{i})$, which represents the probability of $i^{th}$ frame being completely masked.

The same $z_{c}$ is discretized using the Euclidean distance-based nearest neighbor lookup from the codebook vectors. The quantized latent is $z\in\ \mathbb{C}$ where $\mathbb{C}=\{e_{i}\}_{i=1}^{K}$. $\mathbb{C}$ is the codebook of size $K$ and $e$ shows the codebook vectors. This quantized latent is reconstructed using our pre-trained 3D-MBQ-VAE decoder $\mathcal{D}(z)$. Eq. 1 shows the combined loss term for training, where sg represents the stop gradient. We include *Masked Frame Index Loss* or $\mathcal{L}_{\text{MFI}}$ in the loss function to learn from feature distributions across the frames. This loss helps VAE to learn frame consistency.

$$ $\mathcal{L}_{MBQ-VAE}=\underbrace{\|V-\hat{V}\|_{2}^{2}}_{\mathcal{L}_{rec}}+ \underbrace{\|\text{sg}[\mathcal{E}(V)]-V\|_{2}^{2}}_{\mathcal{L}_{codebook}}+ \beta\underbrace{\|\text{sg}[V]-\mathcal{E}(V)\|_{2}^{2}}_{\mathcal{L}_{commit }}-\underbrace{\log(P(f_{i}))}_{\mathcal{L}_{MFI}}$ (1) $$

### 3.2 Pre-training of noising predictor for Text2Video generation

With our pre-trained video 3D-MBQ-VAE encoder $\mathcal{E}$ and codebook $\mathbb{C}$, we tokenize video frames in terms of the codebook indices of their encodings.
Figure 3 shows the pretraining approach of the Spectral Transformer ($\epsilon_{\theta}$).
Given a video $V\in\mathbb{R}^{(B,N,3,H,W)}$, the quantized encoding of $V$ is given by $z_{0}=\mathbb{C}(\mathcal{E}(V))\in\mathbb{R}^{(B,N,H/16,W/16)}$. In continuous space, forward diffusion is generally achieved by adding random Gaussian noise to the image latent as a function of the current time step ‘$t$’. However, in this discrete space, the forward pass is done by gradually corrupting $z_{0}$ by masking some of the quantized tokens by a $<$MASK$>$ token following a probability distribution $P(z_{t}\mid z_{t-1})$.
The forward process yields a sequence of increasingly noisy latent variables $z_{1},...,z_{\mathbb{T}}$ of the same dimension as $z_{0}$. At timestep $\mathbb{T}$, the token $z_{\mathbb{T}}$ is a pure noise token.

Figure: Figure 3: Discrete diffusion pretraining of the spectral transformer involves processing tokenized video frame representations from the 3D-MBQ-VAE encoder. These representations are subjected to random masking based on a predefined probability distribution. The resulting corrupted tokens are then denoised through a series of $N$ Spectral Transformers. Contextual information from text representations generated by the T5-XXL-Encoder aids in this process. The denoised tokens are reconstructed using the 3D decoder
Refer to caption: x2.png

Starting from the noise $z_{\mathbb{T}}$, the reverse process gradually denoises the latent variables and restores the real data $z_{0}$ by sampling from the reverse distribution $q(z_{t-1}|z_{t},z_{0})$ sequentially. However, since $z_{0}$ is unknown in the inference stage, we must train a denoising network to approximate the conditional distribution $\epsilon_{\theta}(z_{t}|z_{0},t,T(X_{t}))$. Here, $t$ represents the time step conditioning, which gives information regarding the degree of masking at step $t$. $T(X_{t})$ represents the text conditioning obtained by passing the video captions $X_{t}$ through the T5-XXL (Tay et al., ) text encoder $T$.
We propose a novel Spectral Transformer block $\epsilon_{\theta}$ to model the reverse diffusion process (Section 3.3).
The objective we maximize during training is the negative log-likelihood of the quantized latents given the text and time conditioning. The likelihood is given by
$L_{\text{diff}}=-\log\left(\epsilon_{\theta}\left(z_{t}|z_{0},t,T(X_{t})\right
)\right)$.

### 3.3 Proposed Spectral Transformer

We propose a novel Spectral Transformer, which processes video latents in the frequency domain to learn video representations more effectively. Consider a quantized latent $z_{t}\in\mathbb{R}^{(B,N,H/16,W/16)}$
at time step $t$. First, the latent is flattened to $\mathbb{R}^{(B,N\times H/16\times W/16)}$ which along with time step $t$ and $FPS$ (frames per second), are passed to our proposed Spectral Transformer block. Initially, the passed tokens are fed into an Adaptive Layer Normalization (AdLN) block to obtain $B_{t}$, where $B_{t}=\text{AdLN}(z_{t},t,\text{FPS})$.

Figure: Figure 4: Architecture of spectral transformer
Refer to caption: x3.png

As shown in Figure 4, before computing the self-attention, the latent representation ${B}_{t}$ is transformed into the frequency domain using a 2D Fourier Transform Layer (*fft2D*) for each of $\mathbf{K}$, $\mathbf{V}$, and $\mathbf{Q}$. This layer applies two 1D Fourier transforms - along the embedding dimension and the sequence dimension thereby converting spatial domain data into the frequency domain. This enables the segregation of high-frequency features from low-frequency ones, enabling more efficient manipulation and learning of spatial features. We take only the real part of *fft2D* layer outputs to make all quantities differentiable (Eq. 2). Here, MSA stands for multi-headed self-attention.

$$ $\mathbf{Z}_{\text{attn}}=\text{MSA}(\mathbf{Q}_{\text{Proj}}(\mathbb{R}(\emph{ fft2D}({Z}_{t}))),\mathbf{K}_{\text{Proj}}(\mathbb{R}(\emph{fft2D}({Z}_{t}))), \mathbf{V}_{\text{Proj}}(\mathbb{R}(\emph{fft2D}({Z}_{t}))))$ (2) $$

The frequency-domain representations are then fed into the self-attention block. Using Rotary Positional Embeddings (RoPE) at this stage helps increase the context length of the input and results in faster convergence. The output $\mathbf{Z}_{attn}$ undergoes an Inverse 2D Fourier Transform (*ifft2D*) to revert to the spatial domain. Then, a residual connection with the AdLN layer is applied to preserve the original spatial information and integrate it with the learned attention features. This gives the output $\mathbf{z}_{\text{out}}$ (Eq. 3).

$$ $\mathbf{z}_{\text{out}}=\text{AdLN}(\mathbb{R}(\emph{ifft2D}(\mathbf{Z}_{\text {attn}}))+\text{AdLN}(\mathbf{z}_{t},t,FPS))$ (3) $$

Similarly, in the cross-attention computation, the text-embedding $T(X_{t})$ is first transformed using learnable Fourier transforms for both $\mathbf{K}$ and $\mathbf{V}$. The $\mathbf{Q}$ representation from the preceding layer is also passed through the learnable Fourier transforms (Eq. 4). These frequency-domain representations are then processed by the text-conditioned cross-attention block.

$$ $\mathbf{Z}_{\text{cross-attn}}=\text{MSA}\left(\mathbf{Q}_{\text{Proj}}( \mathbb{R}(\emph{fft2D}(z_{\text{out}}))),\mathbf{K}_{\text{Proj}}(\mathbb{R}( \emph{fft2D}(\text{T}(X_{t})))),\mathbf{V}_{\text{Proj}}(\mathbb{R}(\emph{fft2 D}(\text{T}(X_{t}))))\right)$ (4) $$

Then, the frequency domain output is converted back to the spatial domain using *ifft2D* layer, and a residual connection is applied (Eq. 5).

$$ $\mathbf{z}_{\text{cross-out}}=\text{AdLN}(\mathbb{R}(\emph{ifft2D}(\mathbf{Z}_ {\text{cross-attn}}))+\text{AdLN}(\mathbf{z}_{out},t,FPS))$ (5) $$

Finally, an MLP followed by a softmax layer gives us the predicted denoised latent given $z_{t}$. This process is repeated until we get the completely denoised latent $P(z_{0}\mid z_{t},t,T(X_{t}))$
. Finally, these output latents are mapped back into the video domain using our pre-trained 3D-MBQ-VAE decoder $\mathcal{D}$.

## 4 MotionAura for Sketch-guided Video Inpainting

Figure: Figure 5: Sketch-guided video inpainting process. The network inputs masked video latents, fully diffused unmasked latent, sketch conditioning, and text conditioning. It predicts the denoised latents using LoRA infused in our pre-trained denoiser $\epsilon_{\theta}$.
Refer to caption: extracted/5915575/Fig5.png

MotionAura can be flexibly adapted to downstream video generation tasks such as sketch-guided video inpainting.
In contrast with the previous works on text-guided video inpainting (Zhang et al., ; Ceylan et al., ), we use both text and sketch to guide the video inpainting. The sketch-text pair makes the task more customizable. For finetuning the model, we use the datasets curated from YouTube-VOS and QuickDraw (Appendix C.3).
We now explain the parameter-efficient finetuning approach. Figure 5 details the entire process.
For a given video $V$ = $\{f_{i}\}_{i=1}^{J}$, we obtain a masked video $V_{m}$ using a corresponding mask sequence $m$ = $\{m_{i}\}_{i=1}^{J}$ where $V_{m}$ is given by $V_{m}$ = $\{f_{i}\odot(1-m_{i})\}_{i=1}^{J}$.
We then discretize both these visual distributions using our pretrained 3D-MBQ-VAE encoder, such that $z_{m}$ = $\mathcal{E}(V_{m})$ and $z_{0}$ = $\mathcal{E}(V)$.

Given a masked video, our training objective is to obtain the unmasked frames with the context driven by sketch and textual prompts.
Hence, we randomly corrupt the discrete latent $z_{0}$ with $<$MASK$>$ token (Section 3.2) to obtain the fully-corrupted latent $z_{\mathbb{T}}$. After performing the forward diffusion process, $z_{\mathbb{T}}$ and $z_{m}$ are flattened and concatenated. This concatenated sequence of discrete tokens is passed through an embedding layer to obtain $Z^{\prime}_{t}\in\mathbb{R}^{(B,S,E)}$ where $Z^{\prime}_{t}$ represents the intermediate input token sequence.

As for the preprocessing of the sketch, we pass the sketch through the pretrained SigLIP image encoder (Zhai et al., ) and consecutively through an MLP layer to finally obtain $s^{\prime}\in\mathbb{R}^{(B,S^{\prime},E)}$. At last, $s^{\prime}$ is concatenated with $Z^{\prime}_{t}$ to obtain the final sequence of tokens $Z_{t}$, where $Z_{t}$ = $Z^{\prime}_{t}\oplus S^{\prime}$ and $Z_{t}\in\mathbb{R}^{(B,S+S^{\prime},E)}$. This input sequence is responsible for the above mentioned *context* to the diffusion network. The input sequence already contains the latents of the unmasked regions to provide the available spatial information. It also contains the *sketch* information that helps the model understand and provide desirable visual details.

The base model is frozen during the inpainting task, and only the adapter modules are trained.
The pretrained spectral transformer is augmented with two types of LoRA modules during finetuning: ($1$) Attention LoRA, applied to $K_{proj}$ and $Q_{proj}$ layers of the self-attention and cross-attention layers of the transformer module, operating in parallel to the attention layers,
($2$) Feed Forward LoRA, applied to the output feed-forward layer in a sequential manner.
These two adapter modules share the same architecture wherein the first Fully Connected (FC) layer projects the high-dimensional features into a lower dimension following an activation function. The next FC layer converts the lower dimensional information to the original dimension. The operation is shown as
$L^{\prime}(x)=L(x)+W_{\text{up}}(\text{GeLU}(W_{\text{down}}(x)))$
such that $W_{up}\in\mathbb{R}^{(d,l)}$ and $W_{down}\in\mathbb{R}^{(l,d)}$ with $l<d$. Here $W_{up}$ and $W_{down}$ represent the LoRA weights, $L^{\prime}(x)$ represents the modified layer output and $L(x)$ represents the original layer output. The $W_{down}$ is initialized with the distribution of weights of $L(x)$ to maintain coherence with the frozen layer.

## 5 Experimental results

We now present experimental results of i) 3D-MBQ-VAE ii) MotionAura for text-guided video generation and iii) MotionAura for sketch-guided video inpainting. The details of experimental setup are provided in Appendices C and D. Additional qualitative results are provided in Appendices A, E and F.

### 5.1 Latent Reconstruction Results of 3D-MBQ-VAE

For evaluating our 3D-MBQ-VAE, we selected the COCO-2017 and the WebVID validation datasets. Following Zhao et al. (), we crop each frame to 256$\times$256 resolution and sample 48 frames per video sequentially. As shown in Table 1, our proposed 3D-MBQ-VAE consistently outperforms SOTA 3D VAEs across all metrics. This can be attributed to a novel combination of training strategies and the enhanced loss function.
Randomly masking parts of various frames He et al. () enables our model to efficiently learn spatial information. Further, inclusion of $\mathcal{L}_{\text{MFI}}$ in the loss function emphasizes learning efficient temporal features.

**Table 1: Quantitative comparison of VAEs for video reconstruction task on COCO-Val and WebVid-Val datasets. Frame compression rate (FCR) is the ratio between the size of video frame before and after compression. Compatibility (comp.) represents if the model can be used as a VAE for existing generative models. Our method demonstrates superior performance on all metrics.**
| Method | Params | FCR | Comp. | COCO-Val | WebVid-Val |
| --- | --- | --- | --- | --- | --- |
|  |  |  |  | PSNR($\uparrow$) / SSIM($\uparrow$) / LPIPS($\downarrow$) | PSNR / SSIM / LPIPS |
| VAE-SD2.1 (Rombach et al., ) | 34M + 49M | 1$\times$ | - | 26.6 / 0.773 / 0.127 | 28.9 / 0.810 / 0.145 |
| VQGAN (Esser et al., ) | 26M + 38M | 1$\times$ | $\times$ | 22.7 / 0.678 / 0.186 | 24.6 / 0.718 / 0.179 |
| TATS (Ge et al., ) | 7M + 16M | 4$\times$ | $\times$ | 23.4 / 0.741 / 0.287 | 24.1 / 0.729 / 0.310 |
| VAE-OSP (Lab & etc., ) | 94M + 135M | 4$\times$ | $\times$ | 27.0 / 0.791 / 0.142 | 26.7 / 0.781 / 0.166 |
| CV-VAE(2D+3D) (Zhao et al., ) | 68M + 114M | 4$\times$ | ✓ | 27.6 / 0.805 / 0.136 | 28.5 / 0.817 / 0.143 |
| CV-VAE(3D) (Zhao et al., ) | 100M + 156M | 4$\times$ | ✓ | 27.7 / 0.805 / 0.135 | 28.6 / 0.819 / 0.145 |
| 3D-MBQ-VAE (Ours) | 140M + 177M | 4$\times$ | ✓ | 31.2 / 0.848 / 0.092 | 32.1 / 0.858 / 0.108 |

Figure 6 shows the t-SNE plots, where each color represents a different class in the dataset. Evidently, the better clustering and separation by 3D-MBQ-VAE shows that it is most effective in feature extraction and representation learning.
The qualitative results in Figure 8 further confirm that 3D-MBQ-VAE provides reconstructions of highest quality and consistency.

Figure: Figure 6: t-SNE plots showing the representations learned by various VAEs.
Refer to caption: extracted/5915575/MBQ_VAE.png

Figure: Figure 7: Comparison of 3D-MBQ-VAE (ours) with existing VAEs after reconstruction from Stable diffusion-XL produced latent. Zooming in on the figure reveals significant differences in image quality, fidelity to the original latent representations, and the preservation of fine details.
Refer to caption: x4.png

### 5.2 Results of Text Conditioned Video Generation

**Table 2: Comparison of MotionAura variants.**
| Model | Embedding Size | Attn head | Blocks |
| --- | --- | --- | --- |
| S | 1024 | 16 | 17 |
| M | 2048 | 16 | 28 |
| L | 4096 | 32 | 37 |

For evaluation purposes, we create three variants of our model, Small (0.89B parameters), Medium (1.94B parameters), and Large (3.12B parameters) as shown in Table 2. Here, blocks refers to the number of spectral transformer blocks ($N$ in Figure 3).
For zero-shot Text2Video generation, we pre-train our Text2Video models with specific configurations (refer appendix D.2). We compare our model with AnimateDiff (Guo et al., ), CogVideoX-5B (Yang et al., ) and SimDA (Xing et al., ). The texts given as text conditioning to all these models are taken from the WebVID 10M dataset (with a maximum length of 180).
However, we observe that the captions in the original WebVID 10M dataset lacked contextual richness, which limits the scope of generation. Hence, we propose our WebVID 10M-*recaptioned* dataset with contextually rich textual data (details in Appendix C). We evaluate the pre-trained models on the MSR-VTT dataset (Chen et al., ) using standard metrics such as FVD and CLIPSIM. We also included two human assessment-based metrics, viz., Motion Quality (MQ) and Temporal Consistency (TC) (Liu et al., ; Feng et al., ).
The EvalCrafter benchmark helps us quantitively evaluate the quality of video generation on these aspects. The framework also aligns objective metrics with human opinions to enhance evaluation accuracy.
100 participants ranked each of the $\mathbb{Y}$ models such that the model ranking first received $\mathbb{Y}-1$ points and the model ranking last received 0 points.

As shown in Table 3, pretraining on our newly curated dataset improves the performance of all the models, highlighting the importance of enriched captions. Further, MotionAura-L without the use of *fft2D* and *ifft2D* layers has inferior performance than the full MotionAura-L network, showcasing the effectiveness of our proposed Spectral Transformer architecture. Finally, our largest model, MotionAura-L, achieves the best results across all metrics, demonstrating its superior capacity for capturing both motion dynamics and temporal consistency.

We measure the average inference latency over text sample lengths of 30 to 180 tokens. For generating a 5-second video, MotionAura-L takes 38 seconds, compared to 41 seconds of CogVideoX-5B (Table 3). To reduce the latency, the FFT layers can be skipped, or the small/medium variants can be used. Notably, MotionAura-S is comparable in latency to AnimateDiff and superior in performance metrics. MotionAura can generate videos of up to 10 seconds, whereas previous works generate up to 6 seconds videos. MotionAura-L takes 83 seconds to generate a 10-second video.

**Table 3: Results of the text-conditioned video generation (Text2Video) models. Recaptioning the dataset improves all the metrics. Inference Time was calculated on a single A100.**
| Method | Params | WebVID 10M | WebVid 10M - recaptioned | Inf Tims(s) |
| --- | --- | --- | --- | --- |
|  |  | (FVD$\downarrow$/CLIPSIM$\uparrow$/MQ$\uparrow$/TC$\uparrow$) | (FVD/CLIPSIM/MQ/TC) |  |
| SimDA | 1.08B | 456 / 0.1761 / 65.28 / 195 | 433 / 0.1992 / 67.82 / 196 | 8.6 |
| AnimateDiff | 1.96B | 402 / 0.2011 / 68.97 / 197 | 379 / 0.2201 / 69.78 / 199 | 11.1 |
| CogVideoX-5B | 5.00B | 380 / 0.2211 / 73.37 / 203 | 357 / 0.2429 / 75.51 / 205 | 41.0 |
| MotionAura-L (w/o fft) | 3.12B | 379 / 0.2201 / 73.44 / 202 | 351 / 0.2441 / 75.87 / 203 | 33.4 |
| MotionAura-S | 0.89B | 391 / 0.2104 / 70.29 / 199 | 364 / 0.2303 / 71.28 / 200 | 12.0 |
| MotionAura-M | 1.94B | 383 / 0.2207 / 72.37 / 200 | 360 / 0.2333 / 73.47 / 202 | 20.0 |
| MotionAura-L | 3.12B | 374 / 0.2522 / 74.59 / 204 | 344 / 0.2822 / 76.62 / 207 | 38.0 |

Figure 8 shows sample frames from videos generated by the full MotionAura-L and MotionAura-L(without fft2d). The videos highlight superior motion quality achieved on using the FFT layers.
Figure 9 compares our model with CogVideoX-5B and AnimateDiff. Both quantitative and qualitative results highlight the superior performance of our model, which can be attributed to several key innovations. AnimateDiff relies on the image VAE from Stable Diffusion to encode video latents, which limits their representational power compared to the latents produced by our 3D-MBQ-VAE. AnimateDiff inflates a 2D diffusion model into 3D using a temporal sub-module for video, but this approach fails to ensure true temporal coherence, as seen when comparing its results to those of MotionAura. CogVideoX-5B employs a video VAE in continuous space. This approach struggles with efficiently representing the high dimensionality of video frames, making it less effective than discrete space representations. Additionally, our use of spectral transformer blocks, with integrated 2D Fourier transforms, enhances both vision and language modeling by capturing complex spatial and temporal patterns. Replacing traditional sinusoidal embeddings with RoPE embeddings further improves the handling of longer video sequences. These architectural improvements collectively improve our overall performance.

Figure: Figure 9: Text-conditioned video generation results. Our model shows superior temporal consistency and generation quality. Click on the [link](https://researchgroup12.github.io/Text_Conditioned_Videos.html) to view the videos.
Refer to caption: x6.png

### 5.3 Results of Sketch guided video inpainting

We now assess the models’ performance on the newly introduced task of sketch-guided video inpainting. As outlined in Section 4, our approach adapts zero-shot Text2Video models using LoRA modules.
For a fair comparison with CogVideoX-5B, we adapted it in a similar manner as our proposed approach. Both existing and our models were pre-trained on the WebVid 10M-*recaptioned* dataset and evaluated on two newly curated datasets, based on existing video (YouTube-VOS and DAVIS) and sketch (QuickDraw) datasets. These datasets have been specifically designed to accommodate the sketch-based inpainting task, featuring four key components for each video sample: textual prompts, sketches, segmentation masks, and raw video.
For each experiment, we inpainted portions of the videos using both the sketch and text inputs as guides.

As presented in Table 4, MotionAura-L outperforms all other methods. The superior performance of MotionAura-L can be attributed to the combination of sketch and text inputs, which provide a richer context for guiding the inpainting process. While text descriptions offer a general understanding of the scene, the sketch provides explicit structural information, allowing the model to generate more accurate and coherent inpainting results. This dual-input method leads to better spatial alignment and temporal consistency, as evidenced by the higher scores when compared to MotionAura-L (w/o sketch), which relies solely on text input. The qualitative comparisons in Figure 10 show that MotionAura leads to more spatially and temporally coherent results than previous techniques.

**Table 4: Quantitative evaluation for sketch-based inpainting. All models were pre-trained using the WebVid 10M -*recaptioned* and evaluated over the newly curated datasets comprising DAVIS and YouTube-VOS.**
| Method | DAVIS | YouTube - VOS |
| --- | --- | --- |
|  | (FVD$\downarrow$ / CLIPSIM$\uparrow$ / MQ$\uparrow$ / TC$\uparrow$) | (FVD / CLIPSIM / MQ / TC) |
| T2V (Khachatryan et al., ) | 782 / 0.2489 / 63.39 / 191 | 700 / 0.2601 / 64.28 / 193 |
| SimDA (Xing et al., ) | 752 / 0.2564 / 65.28 / 196 | 693 / 0.2699 / 67.82 / 195 |
| AnimateDiff (Guo et al., ) | 737 / 0.2401 / 68.97 / 199 | 685 / 0.2701 / 69.78 / 197 |
| CogVideoX-5B (Yang et al., ) | 718 / 0.2512 / 73.37 / 205 | 677 / 0.2602 / 75.51 / 203 |
| MotionAura-L (w/o fft) | 689 / 0.2919 / 73.44 / 203 | 663 / 0.3164 / 75.87 / 202 |
| MotionAura-L (w/o sketch) | 687 / 0.3011 / 73.31 / 200 | 666 / 0.3061 / 75.14 / 199 |
| MotionAura-S | 704 / 0.2776 / 70.29 / 200 | 670 / 0.2892 / 71.28 / 199 |
| MotionAura-M | 692 / 0.2901 / 72.37 / 202 | 666 / 0.3119 / 73.47 / 200 |
| MotionAura-L | 685 / 0.3101 / 74.59 / 207 | 657 / 0.3511 / 76.62 / 204 |

Figure: Figure 10: Sketch-guided inpainting results of various techniques. The red circles highlight the mistakes and misalignments, e.g., SimDA incorrectly inpaints the parts of the hat and the butterfly. AnimateDiff inpaints 2 butterflies in some frames, and inpaints half-transparent butterfly in the last frame. CogVideoX produces a hat with incorrect shape, and a black-color butterfly. By contrast, MotionAura produces high-quality and temporally consistent results.
Refer to caption: x7.png

### 5.4 Ablation Study

1. We pretrain MotionAura on the WebVid 10M - recaptioned dataset and evaluate it on the MSR-VTT dataset for the text-conditioned video generation task. We evaluate the effect of the length of text description on the quality of the generated images. Increasingly detailed text descriptions provide better guidance and context. By virtue of this, the model captures finer nuances in scene composition, motion, and object interactions. Thus, the generated videos show superior quality and better match the user’s requirements. Hence, with increasingly detailed descriptions, FVD reduces and CLIPSIM increases (Table 6).

**Table 5: Impact of Text Length**
| Length of Text | FVD | CLIPSIM |
| --- | --- | --- |
| 30 | 364 | 0.2544 |
| 60 | 359 | 0.2598 |
| 90 | 355 | 0.2626 |
| 120 | 352 | 0.2654 |
| 150 | 349 | 0.2701 |
| 180 | 344 | 0.2823 |

2. We evaluate the effect of rank of LoRA adaptors on the quality of sketch-based inpainting results on the YouTube-VOS dataset. A higher rank of the LoRA Adaptor involves updating more trainable parameters, which helps the model learn better representation and enhances model accuracy. Hence, a higher rank improves model performance, as shown in Table 6.

3. We evaluate MotionAura using sinusoidal embedding for the text-conditioned video generation. The FVD/CLIPSIM values are 379/0.2392 after pretraining on the WebVID 10M dataset and 349/0.2752 after pretraining on the WebVID 10M-recaptioned dataset. Clearly, RoPE embeddings (Table 3) provide superior results than the sinusoidal embeddings.

## 6 Conclusion

In this paper, we present MotionAura, a novel approach to generating high-quality videos given some text and sketch conditions. The videos generated by our model show high temporal consistency and video quality. MotionAura proposes several novelties, such as a new masked index loss during VAE pretraining, using FFT layers to segregate high-frequency features from the low-frequency ones and using RoPE embeddings to ensure better temporal consistency in denoised latents. Additionally, we recaption the WebVid-10M dataset with much more contextually rich captions, improving the denoising network’s quality. Finally, we curate our datasets to address the task of sketch-guided video inpainting. Rigorous experiments show the effectiveness and robustness of MotionAura in video generation and subsequent downstream tasks.

## Appendix A 10 Second Generated Samples

Figure: Figure S.2: Click [here](https://researchgroup12.github.io/10_Second_Clip-1.html) to see the video.
Refer to caption: extracted/5915575/10s_1.jpg

## Appendix B 3D-MBQ-VAE Architecture

Figure S.8 shows the 3D-MBQ-VAE (Mobile Inverted Vector Quantization Variational Autoencoder) architecture designed to efficiently encode and decode video data while utilizing both mobile inverted bottleneck blocks and vector quantization techniques. The MBQ-VAE framework is divided into two main parts: the encoder and the decoder, with quantization and sampling in between.

Figure: Figure S.8: 3D-MBQ-VAE architecture.
Refer to caption: x8.png

### B.1 Encoder

The encoder takes an input video with dimensions $B\times C\times N\times H\times W$, where $B$ is the batch size, $C$ is the number of channels, and $N\times H\times W$ represents the spatiotemporal dimensions (i.e., frames, height, and width). The encoding begins with a Conv3D block that captures spatiotemporal features using a 3D convolution with a kernel size of $1\times 3\times 3$, followed by a stride of $1\times 2\times 2$ to reduce spatial resolution while expanding the channel depth (e.g., from 3 to 64 channels). The encoder then applies multiple 3D-MB Conv blocks (Mobile Inverted Conv Blocks) to capture complex features using depthwise separable convolutions, maintaining computational efficiency. These blocks are interspersed with Inception-Fused-MB Conv blocks, which improve feature extraction by combining multiple filter sizes and dimensions, enabling multi-scale processing.

At various stages, downsampling is performed via strided 3D convolutions to reduce the spatial and temporal dimensions further while increasing channel depth, helping the network focus on abstract, high-level features. This progression continues until the latent representation is compressed into a small-dimensional vector using a final Conv3D block. This compressed representation is passed to the quantization layer, where vector quantization is applied to discretize the latent variables for use in the VQ-VAE.

### B.2 Decoder

The decoder mirrors the encoder but in reverse, beginning with vector sampling from the quantized latent space. It starts with an initial Conv3D block to upsample the channels from the quantized latent vector. The network then proceeds through several 3D-MB Conv blocks and Inception-Fused-MB Conv blocks, gradually increasing the spatial resolution via Upsampling layers. Combined with inverted convolution blocks, these layers allow the decoder to reconstruct fine-grained details from the compressed latent vector.

The upsampling process continues in steps, restoring the spatial and temporal dimensions while reducing the channel depth to match the input dimensions. The final layer of the decoder uses a Conv3D Transpose operation to transform the latent representation back to the original input resolution, ensuring the correct number of channels is maintained for video reconstruction.

### B.3 Key Features

- •
Mobile Inverted Bottleneck Convolutions: The use of 3D-Mobile Inverted Bottleneck (MB) Conv blocks enables the model to achieve high efficiency by expanding and contracting the number of channels at different stages while preserving critical features through depthwise separable convolutions.
- •
Inception-Fused Convolutions: By incorporating multiple kernel sizes in the Inception-Fused blocks, the architecture can process multi-scale spatial and temporal features, improving video compression and reconstruction quality.
- •
Vector Quantization: The quantization step allows for discretizing the latent space, which is critical for VQ-VAE models to perform learned compression, making this architecture suitable for generative tasks and compression with high efficiency.

MBQ-VAE is an efficient and scalable video processing architecture, leveraging mobile inverted convolutional blocks and vector quantization to balance computational efficiency with high-quality reconstruction of video data. This makes it well-suited for video compression, generation, and reconstruction tasks.

### B.4 Mobile-Inverted 3D Conv Block

The 3D Mobile Inverted Convolution (3D-MB Conv) Block plays a crucial role in our VQ-VAE network by enabling efficient spatiotemporal feature extraction with reduced computational complexity. This block begins with a 3D convolutional layer that maps from a higher dimensional input space $C^{\prime}$ to the desired channel dimension $C$, utilizing a $3\times 3\times 3$ kernel with padding to maintain spatial and temporal resolution. The feature maps are then expanded using a pointwise 3D convolutional block from $C$ to $4C$, enhancing the feature representation without significantly increasing computation. A 3D Squeeze-and-Excitation (SE) layer follows, which adaptively recalibrates the channel importance, improving the model’s sensitivity to informative spatiotemporal features. Next, a depthwise separable 3D convolution applies a $3\times 3\times 3$ kernel to each feature map independently, maintaining $4C$ channels and effectively reducing computation compared to standard convolutions. Finally, a 3D convolutional block reduces the channel dimension back from $4C$ to $C$, ensuring that the block output is consistent with the input dimensionality. With its inverted residual connections and depthwise convolutions, this overall structure enhances efficiency in capturing hierarchical spatiotemporal patterns, making it highly suitable for the reconstruction and compression tasks in our VQ-VAE network.

### B.5 Inception-Fused MB Conv Block

The 3D Inception-Fused-MB Conv Block is a sophisticated convolutional module designed to efficiently capture both spatial and temporal features in video data, commonly used in tasks like Video VQ-VAE (Vector Quantized Variational Autoencoder) shown in Figure S.8. This block employs an Inception-style structure with three parallel 3D convolution paths, each with different kernel configurations (e.g.,
3 x 3 x 1, 3 x 1 x 3, and 1 x 3 x 3) to capture multi-scale spatiotemporal patterns in various dimensions. Each branch is followed by a depthwise separable 3D-MobileBlock (MB-Conv) that reduces computational complexity while maintaining efficiency. The outputs of these paths are concatenated, allowing the network to fuse multi-scale features, which are then processed by a final 3D convolutional block. This design enables efficient handling of complex video representations in VQ-VAEs, optimizing the model’s ability to compress and reconstruct video sequences while preserving key spatial and temporal information.

## Appendix C Datasets

Figure: Figure S.9: For dataset curation for the downstream task of sketch-guided video inpainting, captions have been generated using Video-LLaVA-7B, and the corresponding sketches were obtained using CLIP-based label matching
Refer to caption: x9.png

The following section discusses the datasets used for pretraining and training our proposed models. The dataset selection was based on the complexity and nature of each task. We also discuss the process of caption regeneration for the WebVid-10M dataset and the curation of a new dataset for the task of Sketch-based Video Inpainting.

### C.1 Datasets used for 3D MBQ-VAE pre-training

For pre-training our 3D MBQ-VAE, we use the YouTube 100M Hershey et al. () dataset. The large number of videos and the comprehensive pretraining help our 3D-MBQ-VAE learn an efficient latent representation of video frames in a reduced spatiotemporal domain.

### C.2 T2V model training

For training the diffusion model, we use the Webvid-10M Bain et al. () with text as the condition. We find that the original text captions in the Webvid-10M dataset do not provide adequate guidance for diffusion pretraining purposes (Table 3). Hence, we re-caption the entire dataset with much more detailed prompts for every video. The corresponding text pairs given as conditions for diffusion pretraining are generated by LLaVA Next- 34B Li et al. (). In both cases, we had an 80-20 split between the training and test set. The following prompt was given to the LLM for regenerating the textual description of the video:

### C.3 Training for Sketch Guided Video Inpainting task

For the downstream task of Sketch Guided Video Inpainting, we required a dataset containing the videos, text prompts, binary masks, and corresponding object sketches. To our knowledge, no such open-source dataset satisfies all our requirements. Hence, we curated our own dataset to fit our requirements. Figure S.9 shows the entire process of how we curate the dataset for the downstream task using two datasets primarily used in the video inpainting research community: Youtube-VOS Xu et al. () and DAVIS Perazzi et al. ().

## Appendix D Implementation Details

We now present the implementation details and parameters used in our study.

### D.1 Implementation Details of 3D-MBQ-VAE Pre-training

We train our 3D MB-VAE model on the YouTube100M video dataset using four different configurations of frames and resolutions. These configurations are: (1) 32 frames at $256\times 256$ resolution, (2) 16 frames at $512\times 512$ resolution, (3) 32 frames at $1280\times 720$ resolution, and (4) 48 frames at $640\times 480$ resolution. The corresponding batch sizes for these configurations are 8, 4, 2, and 4, with sampling ratios of 40%, 10%, 25%, and 25%, respectively. We apply gradient accumulation steps of 8 across all configurations.

The AdamW optimizer is employed with a base learning rate of $1\times 10^{-4}$ with cosine learning rate decay. To reduce the risk of numerical overflow, we train the 3D MB-VAE model in float32 precision. The training is performed across 4 nodes, each equipped with 8 NVIDIA A100 GPUs (80 GB each), for a total of 800,000 training steps.

### D.2 Implementation Details of Diffusion Pretraining

We train our denoiser on the Webvid-10M dataset for diffusion model pretraining. The AdamW optimizer uses a learning rate of $1\times 10^{-5}$ with linear learning rate decay. During training, we explore multiple frame settings: 16, 32, 48, and 64 frames. For 16-frame training, we use a batch size of 512 and train for 120,000 iterations. For 32, 48, and 64 frames, the batch sizes are 256, 256, and 128, respectively, with corresponding training durations of 120,000 iterations for 32 frames, 150,000 iterations for 48 frames, and 190,000 iterations for 64 frames. This training is conducted on 8 nodes, each equipped with 8 NVIDIA A100 GPUs (80 GB memory per GPU). We utilize dynamic resolution and aspect ratio adjustments during training to enhance the model’s robustness to varying input sizes.

### D.3 Implementation Details of Sketch-based Inpainting Training

For the downstream tasks, we fine-tune the model using LoRA (Low-Rank Adaptation) parameters specifically for the K and Q projection weights. The training is carried out with a batch size of 8 per GPU at a resolution of $256\times 512$. This stage is performed on 2 nodes, each with 8 NVIDIA A100 GPUs (80 GB memory per GPU). The training includes 2,000 iterations for the VOS dataset and 2,600 iterations for the DAVIS dataset. We employ the Adam optimizer with an initial learning rate of $2\times 10^{-4}$, following a cosine decay schedule. Gradient accumulation steps are set to 8 to manage memory and computational load, enabling efficient training through mixed-precision techniques.

### D.4 Use of RoPE vs Sinusoidal Positional Embeddings

The use of RoPE (Rotary Positional Embeddings) in the attention layers of the denoising network shows a faster convergence, as seen in Figure S.10. RoPE (Rotary Positional Embeddings) is a type of positional encoding designed for transformer models to encode relative position information flexibly. RoPE embeddings support larger context lengths, enabling the model to generalize better over varying input sizes. Since they capture positional relationships based on rotation, the transformer can effectively handle and reason over sequences of greater lengths. This is handy when dealing with high-dimensional video frames, thus helping the model learn effectively.

Figure: Figure S.10: RoPE embeddings show faster convergence demonstrating superior learning compared to Sinosoidal embeddings
Refer to caption: x10.png

### D.5 Use Of LoRA Adaptors for PEFT

LoRA adaptors are beneficial in Parameter-Efficient Fine-Tuning (PEFT) because they significantly reduce the number of parameters that must be trained while maintaining the model’s performance. In large-scale models, fine-tuning all the parameters is computationally expensive. LoRA adaptors introduce trainable, low-rank matrices into the model layers, which allows only a small subset of parameters to be updated during training. This drastically reduces the memory and computational overhead required for fine-tuning, making it more feasible to adapt large models to new tasks using limited data and resources. By optimizing only a small fraction of the parameters, LoRA adaptors also help prevent overfitting and make the fine-tuning process more efficient without compromising model accuracy. As our model is very large, thoroughly training it from scratch for every downstream task does not make sense. Hence, we use the LoRA adaptors in our Spectral Transformer to adapt to the downstream task. Fig S.11 shows the details of our Spectral Transformer with the LoRA adaptors.

Figure: Figure S.11: Use of LoRA in the above helps to efficiently fine-tune the above spectral transformer for the downstream task while only using a fraction of the number of trainable parameters.
Refer to caption: x11.png

Two types of LoRA have been used as shown in Fig S.11. One is the Attention LoRA, which is applied in parallel to the Attention Blocks. This LoRA is applied only to the Key and the Query vectors. The reason for applying LoRA here is to ensure the base model retains information from previously trained tasks. LoRA is not applied to the Value vector due to Elastic Weight Compression. Elastic Weight Compression identifies the most critical parameters the model must remember to learn a task effectively. Using Elastic Weight Compression, we find that the parameters of the Key and Query are the most crucial when adapting to a new task, which is why we apply LoRA to them.

The second type of LoRA used is the Feed Forward LoRA, applied sequentially to the final Pointwise Feed Forward Layer. The Feed Forward Layer is a highly dense network, and if LoRA were used in parallel to this layer, followed by concatenation with its output, it would not result in meaningful representations. The sequential application ensures that the model learns effective and compact representations.

## Appendix E Sample sketch-guided video generation results

Figures S.12 and S.13 show sample sketch-guided video generation results of various techniques. Given the inpainting video results, it is not hard to see that our model achieves the best results in terms of temporal consistency and preserving the inpainted object throughout the frames.

Figure: Figure S.12: Sketch-guided video generation results for a dolphin sketch
Refer to caption: extracted/5915575/Dolphin-min.png

Figure: Figure S.13: Sketch-guided video generation results for a frog sketch
Refer to caption: extracted/5915575/Frog.png

## Appendix F Sample text-guided video generation results

Figures S.1 to S.7, and Figures S.14 to S.20 further demonstrate the robustness of our model on different tasks.

Figure: Figure S.14: Abstract sculptures (Click [here](https://researchgroup12.github.io/Abstract_Sculptures.html) to see the video.)
Refer to caption: extracted/5915575/AS4.png

Figure: Figure S.15: Cinematic styles (Click [here](https://researchgroup12.github.io/Cinematic_Style.html) to see the video.)
Refer to caption: extracted/5915575/CV1.png

Figure: Figure S.16: Graphic novel styles (Click [here](https://researchgroup12.github.io/Graphic_Novel.html) to see the video.)
Refer to caption: extracted/5915575/GN1.png

Figure: Figure S.17: Low Polly results (Click [here](https://researchgroup12.github.io/Low_Polly.html) to see the video.)
Refer to caption: extracted/5915575/LP3.png

Figure: Figure S.18: Sci-fi results (Click [here](https://researchgroup12.github.io/Sci-fi_Videos.html) to see the video.)
Refer to caption: extracted/5915575/Scifi_new.png

Figure: Figure S.19: Thriller results (Click [here](https://researchgroup12.github.io/Thriller.html) to see the video.)
Refer to caption: extracted/5915575/Thriller2-min.png

Figure: Figure S.20: Macro style results (Click [here](https://researchgroup12.github.io/Macro_Style.html) to see the video.)
Refer to caption: extracted/5915575/Macro-min.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - Are there any evaluation or ablation study results available regarding Randomly Masking and Fully Mask Frame Index Predicting?
- Will the model or dataset be made open-source?
soundness: 4
strengths: The paper constructs a complete and efficient new framework for text-to-video generation, featuring innovative modules supported by convincing quantitative evaluations and generative results. Specific strengths include:

- For the VQ-VAE, the paper introduces two novel optimization strategies: *random masking* and *Masked Frame Index Loss*, which allow the VAE to learn spatial relationships and temporal consistency. The compression and reconstruction quality reaches state-of-the-art levels, and the visualization results demonstrate strong feature extraction and representation learning capabilities. These strategies could serve as foundational components for future models.
- A diffusion framework for discrete space is proposed, which encodes information such as videos into the frequency domain for computation, featuring an elegantly designed spectral transformer structure. After training, this framework achieves satisfactory results in terms of generation speed and quality.
- Based on the proposed generative framework, the design of the sketch-guided video inpainting task provides a proof of concept for the framework's scalability and efficient transfer applications. According to the paper, this also marks the first introduction of sketch-guided generation for videos.
- The constructed dataset, if made open-source, would be a significant contribution to the community.
summary: This paper presents a novel text-to-video framework, providing comprehensive and reasonable technical details from VAE to model architecture, as well as downstream applications. The paper proposes an effective VQ-VAE training strategy, achieving state-of-the-art compression and reconstruction results, which can be utilized by existing generative models. It also introduces an efficient spectral transformer that encodes information into the frequency domain for processing. Furthermore, the design concepts for downstream applications are illustrated within the context of the sketch-guided video inpainting task.
weaknesses: While the paper is rich in content, there are some potential issues:

- The two supervisory strategies designed for the VAE elevate its capabilities to state-of-the-art levels. Although the paper states that the random masking strategy enables the model to learn spatial information and that supervision for the fully masked frame index prediction enhances temporal consistency, it seems there are no related ablation studies provided to explore the specific reasons for model improvement further.
- The condition injection framework designed for the sketch-guided video inpainting task appears relatively straightforward, lacking innovative design at the technical level.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1.Based on my understanding, the videos in the WebVid10M dataset all contain watermarks. I'm curious as to how the videos generated by the model, which was trained on this dataset, do not have any watermarks. Can the authors explain this?
soundness: 2
strengths: 1.The authors propose a denoising network named Spectral Transformer that processes video latents in the frequency domain using the Fourier Transform, capturing global context and long-range dependencies effectively.

2.The paper is the first to address the downstream task of sketch-guided video inpainting, using LORA adaptors for parameter-efficient fine-tuning of the denoising network.
summary: This paper introduces an innovative set of methods designed to enhance video generation. These include the 3D-MBQ-VAE, which improves video compression, a novel text-to-video generation framework, a spectral transformer-based denoising network for superior video generation, and a Sketch Guided Video Inpainting task that utilizes Low-Rank Adaptation (LoRA) for efficient fine-tuning.
weaknesses: 1.3D-MBQ-VAE Concerns:

(1)W.A.L.T[1] has demonstrated that using a 3D casual VAE can allow for joint training with both images and videos in text-to-video models, significantly improving performance compared to training solely with videos. Can the authors discuss whether the proposed 3D-MBQ-VAE can also support such joint image and video training? If feasible, it would be beneficial to see an experiment demonstrating this capability and comparing it with that of W.A.L.T.

(2) In the comparative experiments listed in Table 1, could the authors provide comparative results of MAGVIT-v2[2] in regard to Video Compression Metrics or Video Reconstruction tasks?

 (3) Codebook (Vocabulary) Size Influence: As far as I understand, the size of the codebook could considerably affect the 3D-MBQ-VAE's reconstruction ability. However, the paper doesn't seem to mention any information relating to the codebook size. Could the authors provide additional details on the impact of varying codebook sizes on video reconstruction?

2.Experiment Setting for Text-to-Video Model: In Table 3, the authors used WebVID10M as the training data and also computed FVD and other metrics on WebVID10M for testing. However, the comparative methods did not use WebVID10M in their training data, which suggests that other text-to-video methods computed FVD and other metrics in a zero-shot manner. In addition, specifics like the sampling schedule, sampling steps, and classifier-free guidance scale used by the authors' proposed method and the comparison methods were not disclosed. To ensure a fair comparison, could the authors evaluate all models (including theirs) in a zero-shot manner on a common test set, like FVD of UCF-101? Also, providing a table or an appendix detailing all the hyperparameters and settings for each method would greatly enhance reproducibility and fairness in comparison.

------------

[1].Gupta, Agrim, et al. "Photorealistic video generation with diffusion models." arXiv preprint arXiv:2312.06662 (2023).

[2].Yu, Lijun, et al. "Language Model Beats Diffusion--Tokenizer is Key to Visual Generation." arXiv preprint arXiv:2310.05737 (2023).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Traditional DMs (e.g., DDPM, EDM, LDMs) works perfectly fine with the continuous space. Could the authors explain why they see the need of going to the discrete space and transferring the diffusion paradigm to the discrete space? Is it only due to the encoder being more performant in this setting?
- Could the authors explain the $\mathcal{L}_\text{MFI}$ loss which is indicated as *Index of completely masked frame* in Figure 2? Is this supposed to be some auxillary discrimator loss that detects whether the frame is masked or not?
- If the former is true, an ablation experiment that include $\mathcal{L}_\text{MFI}$ in the continuous space would be convincing
- Can you provide more ablation in how $\mathcal{L}_\text{MFI}$ helps? What is the masking ratio of random masking and full frame masking?
- In Table 1, you show a Frame Compression Rate of 4 but I can't find in the paper where you talk how you downsample temporally and seem to be inconsistent with Figure S.8. A little bit of explanation would be nice
- It's excited to see that Fourier transform also works with MSA and can further boost the performance of transformer. Since the self-attention and cross attention are working in a totally separate domain, it would be very interesting to show the activations of the attention in the spectral domain to gain more insight.
- Another question with regard to FFT: In the cross attention part $Q$ is calculated from the image tokens and $K$, $V$ are calculated from the text tokens. Why would FFT still make sense in those domain since neither $Q$ or $K$, $V$ constitutes a continuous image domain? How do you make sure that FFT works on the discrete image tokens and why would FFT even make sense on the text tokens?
soundness: 3
strengths: - The spectral transformer is novel and works better than its counterpart transformer, as evidenced by the ablation study.
- The author proposed pretraining the 3D-MBQ-VAE using random and complete masking and get better results than the counterpart compression models.
- The authors show very pleasing, temporally consistent video samples that showcases the effectiveness of the proposed model
summary: The authors propose MotionAura, a novel text-to-video generation model. The authors begin with the 3D Mobile Inverted VQVAE to enhance spatiotemporal video compression. The discrete diffusion process is also carried out in the spectral domain rather than the traditional pixel or latent space. The model exhibits SOTA compression results and promiosing results on T2V and downstream tasks.
weaknesses: - Some of the ablation experiments are missing e.g., in the 3D-MBQ-VAE with regard to the different losses
- It is generally not clear why the authors would not prefer a continuous space rather than opting for the discrete space
- Some of the training details is still missing, e.g., what is the masking ratio of random masking and full frame masking in 3D-MBQ-VAE?
- The spectral transformer part of the paper can be augmented with more analysis

Meta-review: The paper proposes a new framework for video generation and dives into many of the important subparts: video autoencoding, diffusion training, and diffusion architectures. In addition, the paper also proposes a new downstream task (Sketch Guided Video Inpainting).

For video autoencoding, the paper proposes a new style of VAE (3D MoBile Inverted VQ-VAE) which trains with random masking of frames with patches and complete masking of a single frame. For diffusion training and architecture, the paper proposes to use discrete diffusion with Spectral Transformer blocks which use 2D FFT and Rotary Positional Embeddings. For the downstream task, the paper proposes Sketch-guided video inpainting where a user specifies with a sketch what the inpainted region should look like and create a dataset for this task. The paper demonstrates, for the VAE, strong reconstruction with decent compression; for the text to video model good quality (with interesting ablations on the design choices here); and nice results for the proposed sketch-based video inpainting task.

Reviewers all rated the paper above acceptance (two 8s, one 6) and particularly appreciated the work on the VAE and spectral transformer as well as the ablations on the design of the transformer and the new task.

Most of the weaknesses brought up by the reviewers (e.g. missing ablations / comparisons) were either addressed in the rebuttal process or seem to be "nice to haves" as opposed to "necessary" elements.

I agree with the reviewers and recommend acceptance. Note to the authors, integrating the rebuttal work (e.g. the image video joint training, masking ablation, etc.) with the paper would strengthen it and I strongly encourage the authors to do so.

Decision: Accept (Spotlight)