MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion

Authors: Onkar Kishor Susladkar, Jishu Sen Gupta, Chirag Sehgal, Sparsh Mittal, Rekha Singhal
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: text2video, VQ-Diffusion, video Inpainting, Large scale pretraining

Abstract:
The spatio-temporal complexity of video data presents significant challenges in tasks such as compression, generation, and inpainting. We present four key contributions to address the challenges of spatiotemporal video processing. First, we introduce the 3D Mobile Inverted Vector-Quantization Variational Autoencoder (3D-MBQ-VAE), which combines Variational Autoencoders (VAEs) with masked modeling to enhance spatiotemporal video compression. The model achieves superior temporal consistency and state-of-the-art (SOTA) reconstruction quality by employing a novel training strategy with full frame masking. Second, we present MotionAura, a text-to-video generation framework that utilizes vector-quantized diffusion models to discretize the latent space and capture complex motion dynamics, producing temporally coherent videos aligned with text prompts. Third, we propose a spectral transformer-based denoising network that processes video data in the frequency domain using the Fourier Transform. This method effectively captures global context and long-range dependencies for high-quality video generation and denoising. Lastly, we introduce a downstream task of Sketch Guided Video Inpainting. This task leverages Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. Our models achieve SOTA performance on a range of benchmarks.  Our work offers robust frameworks for spatiotemporal modeling and user-driven video content manipulation.

Decision: accept (spotlight)
Reviewer scores: 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730139337643

thread_type:
official_review

<thread_object>
creator: reviewer_hth2

<content>
summary:
The authors propose MotionAura, a novel text-to-video generation model. The authors begin with the 3D Mobile Inverted VQVAE to enhance spatiotemporal video compression. The discrete diffusion process is also carried out in the spectral domain rather than the traditional pixel or latent space. The model exhibits SOTA compression results and promiosing results on T2V and downstream tasks.

strengths:
- The spectral transformer is novel and works better than its counterpart transformer, as evidenced by the ablation study.
- The author proposed pretraining the 3D-MBQ-VAE using random and complete masking and get better results than the counterpart compression models.
- The authors show very pleasing, temporally consistent video samples that showcases the effectiveness of the proposed model

weaknesses:
- Some of the ablation experiments are missing e.g., in the 3D-MBQ-VAE with regard to the different losses
- It is generally not clear why the authors would not prefer a continuous space rather than opting for the discrete space
- Some of the training details is still missing, e.g., what is the masking ratio of random masking and full frame masking in 3D-MBQ-VAE?
- The spectral transformer part of the paper can be augmented with more analysis

questions:
- Traditional DMs (e.g., DDPM, EDM, LDMs) works perfectly fine with the continuous space. Could the authors explain why they see the need of going to the discrete space and transferring the diffusion paradigm to the discrete space? Is it only due to the encoder being more performant in this setting?
- Could the authors explain the $\mathcal{L}_\text{MFI}$ loss which is indicated as *Index of completely masked frame* in Figure 2? Is this supposed to be some auxillary discrimator loss that detects whether the frame is masked or not?
- If the former is true, an ablation experiment that include $\mathcal{L}_\text{MFI}$ in the continuous space would be convincing
- Can you provide more ablation in how $\mathcal{L}_\text{MFI}$ helps? What is the masking ratio of random masking and full frame masking?
- In Table 1, you show a Frame Compression Rate of 4 but I can't find in the paper where you talk how you downsample temporally and seem to be inconsistent with Figure S.8. A little bit of explanation would be nice
- It's excited to see that Fourier transform also works with MSA and can further boost the performance of transformer. Since the self-attention and cross attention are working in a totally separate domain, it would be very interesting to show the activations of the attention in the spectral domain to gain more insight.
- Another question with regard to FFT: In the cross attention part $Q$ is calculated from the image tokens and $K$, $V$ are calculated from the text tokens. Why would FFT still make sense in those domain since neither $Q$ or $K$, $V$ constitutes a continuous image domain? How do you make sure that FFT works on the discrete image tokens and why would FFT even make sense on the text tokens?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730536793151

thread_type:
official_review

<thread_object>
creator: reviewer_uah7

<content>
summary:
This paper introduces an innovative set of methods designed to enhance video generation. These include the 3D-MBQ-VAE, which improves video compression, a novel text-to-video generation framework, a spectral transformer-based denoising network for superior video generation, and a Sketch Guided Video Inpainting task that utilizes Low-Rank Adaptation (LoRA) for efficient fine-tuning.

strengths:
1.The authors propose a denoising network named Spectral Transformer that processes video latents in the frequency domain using the Fourier Transform, capturing global context and long-range dependencies effectively.

2.The paper is the first to address the downstream task of sketch-guided video inpainting, using LORA adaptors for parameter-efficient fine-tuning of the denoising network.

weaknesses:
1.3D-MBQ-VAE Concerns:

(1)W.A.L.T[1] has demonstrated that using a 3D casual VAE can allow for joint training with both images and videos in text-to-video models, significantly improving performance compared to training solely with videos. Can the authors discuss whether the proposed 3D-MBQ-VAE can also support such joint image and video training? If feasible, it would be beneficial to see an experiment demonstrating this capability and comparing it with that of W.A.L.T.

(2) In the comparative experiments listed in Table 1, could the authors provide comparative results of MAGVIT-v2[2] in regard to Video Compression Metrics or Video Reconstruction tasks?

 (3) Codebook (Vocabulary) Size Influence: As far as I understand, the size of the codebook could considerably affect the 3D-MBQ-VAE's reconstruction ability. However, the paper doesn't seem to mention any information relating to the codebook size. Could the authors provide additional details on the impact of varying codebook sizes on video reconstruction?

2.Experiment Setting for Text-to-Video Model: In Table 3, the authors used WebVID10M as the training data and also computed FVD and other metrics on WebVID10M for testing. However, the comparative methods did not use WebVID10M in their training data, which suggests that other text-to-video methods computed FVD and other metrics in a zero-shot manner. In addition, specifics like the sampling schedule, sampling steps, and classifier-free guidance scale used by the authors' proposed method and the comparison methods were not disclosed. To ensure a fair comparison, could the authors evaluate all models (including theirs) in a zero-shot manner on a common test set, like FVD of UCF-101? Also, providing a table or an appendix detailing all the hyperparameters and settings for each method would greatly enhance reproducibility and fairness in comparison.

------------

[1].Gupta, Agrim, et al. "Photorealistic video generation with diffusion models." arXiv preprint arXiv:2312.06662 (2023).

[2].Yu, Lijun, et al. "Language Model Beats Diffusion--Tokenizer is Key to Visual Generation." arXiv preprint arXiv:2310.05737 (2023).

questions:
1.Based on my understanding, the videos in the WebVid10M dataset all contain watermarks. I'm curious as to how the videos generated by the model, which was trained on this dataset, do not have any watermarks. Can the authors explain this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655572287

thread_type:
official_review

<thread_object>
creator: reviewer_cyqt

<content>
summary:
This paper presents a novel text-to-video framework, providing comprehensive and reasonable technical details from VAE to model architecture, as well as downstream applications. The paper proposes an effective VQ-VAE training strategy, achieving state-of-the-art compression and reconstruction results, which can be utilized by existing generative models. It also introduces an efficient spectral transformer that encodes information into the frequency domain for processing. Furthermore, the design concepts for downstream applications are illustrated within the context of the sketch-guided video inpainting task.

strengths:
The paper constructs a complete and efficient new framework for text-to-video generation, featuring innovative modules supported by convincing quantitative evaluations and generative results. Specific strengths include:

- For the VQ-VAE, the paper introduces two novel optimization strategies: *random masking* and *Masked Frame Index Loss*, which allow the VAE to learn spatial relationships and temporal consistency. The compression and reconstruction quality reaches state-of-the-art levels, and the visualization results demonstrate strong feature extraction and representation learning capabilities. These strategies could serve as foundational components for future models.
- A diffusion framework for discrete space is proposed, which encodes information such as videos into the frequency domain for computation, featuring an elegantly designed spectral transformer structure. After training, this framework achieves satisfactory results in terms of generation speed and quality.
- Based on the proposed generative framework, the design of the sketch-guided video inpainting task provides a proof of concept for the framework's scalability and efficient transfer applications. According to the paper, this also marks the first introduction of sketch-guided generation for videos.
- The constructed dataset, if made open-source, would be a significant contribution to the community.

weaknesses:
While the paper is rich in content, there are some potential issues:

- The two supervisory strategies designed for the VAE elevate its capabilities to state-of-the-art levels. Although the paper states that the random masking strategy enables the model to learn spatial information and that supervision for the fully masked frame index prediction enhances temporal consistency, it seems there are no related ablation studies provided to explore the specific reasons for model improvement further.
- The condition injection framework designed for the sketch-guided video inpainting task appears relatively straightforward, lacking innovative design at the technical level.

questions:
- Are there any evaluation or ablation study results available regarding Randomly Masking and Fully Mask Frame Index Predicting?
- Will the model or dataset be made open-source?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731574528005

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To All

comment:
We sincerely thank all reviewers for their time, effort, and valuable feedback on our paper. We greatly appreciate the insightful and constructive suggestions, which would significantly strengthen our work.

Additionally, we are committed to supporting further research in this area by open-sourcing all code, models, and preprocessed datasets. We are currently collaborating with Hugging Face Diffusers to integrate our models and code, and our latest pull request is expected to be merged shortly. We also plan to release all preprocessed datasets used for pre-training and downstream tasks on Hugging Face Datasets, making these resources widely accessible to the research community.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732444437064

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To All

comment:
We sincerely thank all the reviewers for their valuable time and insightful comments on our paper. We have carefully addressed all the suggestions and incorporated the changes in the main manuscript as well as the Appendix section. These modifications are highlighted in red within the revised manuscript for clarity.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a new framework for video generation and dives into many of the important subparts: video autoencoding, diffusion training, and diffusion architectures. In addition, the paper also proposes a new downstream task (Sketch Guided Video Inpainting).

For video autoencoding, the paper proposes a new style of VAE (3D MoBile Inverted VQ-VAE) which trains with random masking of frames with patches and complete masking of a single frame. For diffusion training and architecture, the paper proposes to use discrete diffusion with Spectral Transformer blocks which use 2D FFT and Rotary Positional Embeddings. For the downstream task, the paper proposes Sketch-guided video inpainting where a user specifies with a sketch what the inpainted region should look like and create a dataset for this task. The paper demonstrates, for the VAE, strong reconstruction with decent compression; for the text to video model good quality (with interesting ablations on the design choices here); and nice results for the proposed sketch-based video inpainting task.

Reviewers all rated the paper above acceptance (two 8s, one 6) and particularly appreciated the work on the VAE and spectral transformer as well as the ablations on the design of the transformer and the new task.

Most of the weaknesses brought up by the reviewers (e.g. missing ablations / comparisons) were either addressed in the rebuttal process or seem to be "nice to haves" as opposed to "necessary" elements.

I agree with the reviewers and recommend acceptance. Note to the authors, integrating the rebuttal work (e.g. the image video joint training, masking ablation, etc.) with the paper would strengthen it and I strongly encourage the authors to do so.