MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion
Authors: Onkar Kishor Susladkar
Jishu Sen Gupta
Chirag Sehgal
Sparsh Mittal
Rekha Singhal
Venue: ICLR 2025 Spotlight (2025)
Keywords: text2video
VQ-Diffusion
video Inpainting
Large scale pretraining
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: High Quality text to video generation with discrete diffusion

Abstract: The spatio-temporal complexity of video data presents significant challenges in tasks such as compression, generation, and inpainting. We present four key contributions to address the challenges of spatiotemporal video processing. First, we introduce the 3D Mobile Inverted Vector-Quantization Variational Autoencoder (3D-MBQ-VAE), which combines Variational Autoencoders (VAEs) with masked modeling to enhance spatiotemporal video compression. The model achieves superior temporal consistency and state-of-the-art (SOTA) reconstruction quality by employing a novel training strategy with full frame masking. Second, we present MotionAura, a text-to-video generation framework that utilizes vector-quantized diffusion models to discretize the latent space and capture complex motion dynamics, producing temporally coherent videos aligned with text prompts. Third, we propose a spectral transformer-based denoising network that processes video data in the frequency domain using the Fourier Transform. This method effectively captures global context and long-range dependencies for high-quality video generation and denoising. Lastly, we introduce a downstream task of Sketch Guided Video Inpainting. This task leverages Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. Our models achieve SOTA performance on a range of benchmarks.  Our work offers robust frameworks for spatiotemporal modeling and user-driven video content manipulation.

Published as a conference paper at ICLR 2025
MOTIONAURA: GENERATING HIGH-QUALITY AND MOTION CONSISTENT VIDEOS USING DISCRETE DIFFUSION
Onkar Kishor " Sen Gupta", Chirag Sehgal Sparsh Mittal', Rekha Singhal 'Northwestern University, Yellow.ai,, ²IIT BHU, DDeli Technological University, 'IIT Roorkee *TCS Research
onkarsus mateseusss@@mmmllmm com, chiragsehgal 22 www.a/mmma.l..oo..... jishusen. gupta. ac. in sparsh
sparsh.mittaaee aeeeh.imttaeeeeee i iitr. ac. in, ret.eeeeeee rekha. singhal wwwghaletes..........e...m.oi..m @tcs. com
MotionAura
"Luhutttii nen-ait siyocrpars, and cyberunk atnasghhree against dlasy
"10 'Mehennarr' tencs with dynomer, cheohs notion Mar Hyperkietic, ursherp edges. legh-speed coriol glss over, caphuriyy kiatti energy"
The ansient mase bendh ekettone seundtrock, evhascing stroupheree nen glow dynomic matior. whar syephany. ciesmotic.
individua's engage unboxing
rebkit-squird hybrid explores vibrent, encharted forect high- high-speed. motton blureed hyperkmetiic snimation, dlsg sparlinng strean with wander."
Lights, pale dark patterns."
sharing insights evolustions professional aettlng.
We introduce MotionAura, novel Text-to- Video generation model that predicts discrete tokens obtained from our large scale pre-trained 3D VAE. The displayed frames represent videos generated by our model when provided with the captions shown below each frame. The following link hosts the above generated videos along with other samples.
Figure I:
ABSTRACT
The spatio-temporal complexity of video data presents significant challenges in tasks such as compression, generation, and inpainting. We present four key con- tributions address the challenges of spatiotemporal video processing. First, we introduce the 3D Mobile Inverted Sictor-Quantization Variational Autoen- coder (3D-MBQ- VAE). which combines Variational Autoencoders (VAEs) with masked token modeling to enhance spatiotemporal video compression. The model achieves superior temporal consistency and state-of-the-art (SOTA) reconstruction quality by employing novel training strategy with full frame masking. Sec- ond, we present MotionAura, text-to-video generation framework that utilizes vector-quantized diffusion models to discretize the latent space and capture com- plex motion dynamics, producing temporally coherent videos aligned with text prompts. Third, propose spectral transformer-basee processes video data in the frequency domain using the Fourier Transform. This method effectively captures global context and long-range dependencies for high-
denoising network that

---

Published as a conference paper at ICLR 2025
quality video generation and denoising. Lastly, introduce downstream task Sketch Guided Video Inpainting. This task leverages Low-Rank Adaptation (LoRA) for parameter- mance on a range of benchmarks. Our work offers robust frameworks for spa- tiotemporal modeling and user-driven video content manipulation. We release the code, datasets, and models in open-source link.
fine-tuning. Our models achieve SOTA perfor-
INTRODUCTION
Video generation refers to generating coherent and realistic sequences of frames over time, match- ing the consistency and appearance of real-world videos. Recent years have witnessed significant advances in video generation based on learning frameworks ranging from generative adversarial net- works (GANs) (Aldausari al., 2022), diffusion models (Ho et al., 2022a), to causal autoregressive models (Gao et al., 2024; Tudosiu et al., 2020; Yu et al., 2023c). Video generative models facilitate range of downstream tasks such as class conditional generation, frame prediction or interpolation (Ming et al., 2024), conditional video inpainting (Zhang et al., 2023; Wu et al., 2024) and video out- painting (Dehan al., 2022). Video generation has several business use cases, such as marketing, advertising, entertainment, and personalized training.
The video generation process includes two broad tasks. The first task is video frame tokenization, performed using pre-trained variational autoencoders (VAEs) such as 2D VAE used in Stable Dif- fusion or 3D VQ-VAEs. However, these frameworks fail to achieve high spatial compression and temporal consistency across video frames, especially with increasing dimensions. The second task is visual generation. The early VAE models focused on modeling the continuous latent distribution, which is not scalable with spatial dimension. Recent works have focused modeling discrete la- tent space rather than a continuous one due to its training efficiency. Some works (e.g.. Arnab et al. (2021)) train transformer discrete tokens extracted by 3D VQ-VAE framework. However, an autoregressive approach cannot efficiently model a discrete latent space for high-dimensional inputs such videos.
To address the challenges of visual content generation, we propose MotionAura. At the core of Mo- tionAura is our novel VAE, named 3D-MBQ-VAL VAE (3D MoBile Inverted VQ-VAE), which achieves good temporal comprehension for spatiotemporal compression of videos. For the vidco generation phase, we use a denoiser network comprising series of proposed Spectral Transformer blocks trained using the masked token modeling approach. Our novelty lies both in the architectural changes in our transformer blocks and the training pipeline. These include using a learnable 2D Fast Fourier Transform (FFT) layer and Rotary Positional Embeddings (Su et al., 2021). Following an efficient and robust, non-autoregressive approach, the transformer block predicts all the masked tokens at once. To ensure temporal consistency across the video latents, during tokenization, randomly mask one of the frames completely and predict the index of the masked frame. This helps the model learn the frame sequence and improve the temporal consistency of frames at inference time. Finally, we address a downstream task of kketch-guided video inpainting. To our knowledge, ours is the first work to address the sketch-guided video inpainting task. Using our pre-trained noise predictor as a base and the LoRA adaptors (Hu et al., 2021), our model can be finetuned for various downstream tasks such as class conditional video inpainting and video outpainting. Figure I shows examples of text-conditioned video generation using MotionAura. Our contributions are:
We propose novel 3D- MBQM-VE for spatio-temporal compression of video frames. The 3D- MBQ-VAL adopts a new training strategy based on the complete masking of video frames. This strategy improves the temporal consistency of the reconstructed video frames.
We introduce MotionAura, novel framework for text-conditioned video generation that lever- ages vector quantized diffusion models. That allows for more accurate modeling of motion and transitions in generated videos. The resultant videos exhibit realistic temporal coherence aligned with the input text prompts.
We propose a denoising network named spectral transformer. employs Fourier transform to process video latents in the frequency domain and, hence, better captures global context and long- range dependencies. To pre-train this denoising network, we add contextually rich captions to the WebVID 10M dataset and call this curated dataset
2

---

Published as a conference paper at ICLR 2025
We the first to address the downstream task of sketch-guided video inpainting. To realize this, we perform parameter-efficient finetuning of our denoising network using LORA adaptors. The experimental results show that 3D-MBQ-VAL VAE outperforms existing networks in terms of reconstruction quality. Further, MotionAura attains state-of-the-art (SOTA) performance on text- conditioned video generation and sketch-guides video inpainting.
We curated two datasets, with each data point consisting of Jideo-Mask-Sie Text condition- ing, for our downstream task of sketch-guided video inpainting. We utilized YouTube- VOS and DAVIS datasets and captioned all the videos using Video LLaVA-7B-hf. Then, we performed a CLIP-based matching of videos with corresponding sketches from QuickDraw and Sketchy.
RELATED WORKS ON VISUAL CONTENT GENERATION
Image generation: Text-guided image synthesis has resulted in high quality and spatially coherent data (Esser et al., 2024; Sun et al., 2024b). Generative Adversarial Networks (Sauer et al., 2022; Karras et al., 2021; Goodfellow et al., 2014) model visual distributions to produce perceptually rich images but fail to capture the complete spectrum of data distribution. Moreover, these networks difficult to optimize. Kingma & Welling (2019) introduce another approach to generating visual data wherein the decoders of the autoencoder-like architectures are trained reconstruct images by sampling over the distributions provided by encoders. This regularization strategy helps to learn continuous and meaningful latent space. Quantization of the produced latents (van den Oord al., 2018: Susladkar et al.. 2025) further improves the quality of generated data using various sampling methods over the learned latent. While existing sampling methods can capture dense representa- tions in their latent space, they fail to produce sharp, high-quality images. Diffusion models train networks to learn a denoising process, appropriately fitting the inductive bias image data. Recent developments such as DDPMs (Ho et al., 2020), latent diffusion methods (Rombach al., 2022a) and diffusion transformers (Esser et al., 2024; Peebles & Xie, 2023; Chen et al., 2024; 2023) have revolutionized the field of image generation.
Video generation is even more challenging than image generation since gener- ating high-quality videos demands both spatial and temporal consistencies. For video generation, researchers have used both transformer models and diffusion models. VideoGPT (Yan et al., 2021) generates videos autoregressively using the latents obtained from the VQ-VAE tokenizer. Process- ing videos in the latent space rather than the pixel space reduces training and inference latency. The early diffusion-based video generation models transferred weights from the TextTolmage (T21) approaches and adapted them by inflating the convolutional backbone to incorporate the temporal dynamics (Ho et al.. 2022b). Latent diffusion models (LDM) based Text2Video (T2V) approaches generate higher-quality videos of longer duration (Blattmann et al., 2023). However, diffusion- approaches using convolution backbones fail to produce high frame-rate videos and generally rely on other architectures to interpolate intermediate frames. Recent works have combined transformer- based predictions with diffusion- based pretraining (Yu et al., 2023b; 2024; Kondratyuk et al., 2024).
Video generation:
3 MOTIONAURA: ARCHITECTURE AND TRAINING
We introduce MotionAura, novel video generation model designed to produce high-quality videos with strong temporal consistency. At the core of MotionAura is 3D-MBQ- VAE, our novel 3D VAE that achieves high reconstruction quality. The encoder of this pretrained 3D-MBQ-VAF encodes videos into a latent space, forming the first essential component of our approach. The second novelty spectral transformer-based diffusion model, which diffuses the encoded latents of videos to produce high-quality videos. Section 3.1 discusses the large-scale pretraining of 3D-MBQ-VAL VAE for learning efficient discrete representations of videos. Section 3.2 discusses pretraining of the noising predictor, whereas Section 3.3 presents the Spectral Transformer module for learning the reverse discrete diffusion process.
3.1 PRE-TRAN TRE-TRRINING OF 3D-MBQ- VAE
We employed a 3D-V VA-V (Vector Quantized Variational Autoencoder) to tokenize videos in 3D space; refer Appendix B.1 for the architecture of SD-MBQ-VAR. We pretrained our network on
3

---

Published as a conference paper at ICLR 2025
the YouTube- 100M dataset ensure that it produces video-appropriate tokens. The extensive tent variety in the YouTube- 100M dataset enhances the model's generalization and representational capabilities. Figure 2 shows pretraining approach for learning efficient latents with temporal and spatial consistencies. The pretraining leverages two approaches for optimal video compres- sion and discretization: (i) Random masking of frames (He al., 2021) using 16x 16 and x2x32 patches. This training ensures self-supervised training of the autoencoder to efficiently learn spa- tial information. (ii) Complete masking of a single frame in the sequence to enforce learning of temporal information. Let B be the batch size, C' be number of channels, N the number of frames per video, H, and W be the height and width of the video frames, respectively. For given video V (.,, randomly selected frame Sa E (3, is masked completely. The encoder E(V) returns &e the MLP layer to return P(ft), which shows the probability ith frame being completely masked.
H/i.G//..////66,, This compressed latent in the continuous domain is fed into
The same &e is discretized using the Euclidean distance-based nearest neighbor lookup from the codebook vectors. The quantized latent is E C where C = {_ C is the codebook of size K and shows the codebook vectors. This quantized latent is reconstructed using our pre-trained 3D-MBQ VAE decoder D(z). Eq. shows the combined loss term for training, where sg represents the stop gradient. We in- clude Masked Frame Index Loss LMFI in the loss function to learn from feature distributions
Gasebook
30 M5O WE
Enoader
Misiou
P())
companedy
Figure 2: Our proposed pre-training method for 3D-MBQ- VAE architecture across the frames. This loss helps VAE to learn frame consistency.
LMBQ-VAE = V VIB [sgle(()]
+B
|sg[V] E(V) (P(())
(1)
Lree
Londebook
Coommit
EMPI
3.2 PRE-TRAINING OF NOISING PREDICTOR FOR TEXT2VIDEO GENERATION
With our pre-trained video 3D-MBQ-VAL VAE encoder & and codebook C, we tokenize video frames in terms of the codebook indices of their encodings. Figure 3 shows the pretraining approach of the Spectral Transformer (ep). Given a video V E R.B.N.,.H.W), the quantized encoding of V is given by 2g = CV)(V)) E /B
continuous space, forward diffusion is generally achieved by adding random Gaussian noise to the image latent function of the current time step 't'. However, in this discrete space, the forward pass is done by gradually corrupting 2g by masking some of the quantized tokens by a <MASK> token following probability distribution P(z, 22-1). The forward process yields sequence of increasingly noisy latent variables 21, ..,... 2T of the same dimension as Zu. timestep T, the token 2T is pure noise token.
Codebook
Key Volue
polA ((X,))
Caption
15. 15-000- Exceter
T(X.)
25
M
25
70
59
4
5B
3D-
Spetral Transformen
MBQ-
VAE Encoder
BD
SF
56
64
22
EØ
55
4G
VAE
8
N
Pize Ze-1)
(t.FPS)
Figure 3: Discrete diffusion pretraining of the spectral transformer involves processing tokenized video frame representations from the 3D-MBQ-VAE encoder. These representations are subjected to random masking based on a predefined probability distribution. The resulting corrupted tokens are then denoised through a series of N Spectral Transformers. Contextual information from text representations generated by the T3-XXL-Encoode aids in this process. The denoised tokens are reconstructed using the 3D decoder
4

---

Published as a conference paper at ICLR 2025
Starting from the noise 2T, the reverse process gradually denoises the latent variables and restores the real data 20 by sampling from the reverse distribution q(2t_1/² 20) sequentially. However, since 2p is unknown in the inference stage, must train denoising network to approximate the conditional distribution c0(21|20, T(.))) Here, t represents the time step conditioning, which gives information regarding the degree of masking step T(X;) represents the text conditioning obtained by passing the video captions X, through the TS-XXL (Tay et al., 2022) text encoder I', We propose a novel Spectral Transformer block eş model the reverse diffusion process (Section 3.3). The objective we maximize during training is the negative log- log-likelihood of the quantized latents given the text and time conditioning. The likelihood is given by Lait log (co (2t (2. t, T()))).
3.3 PROPOSED SPECTRAL TRANSFORMER
We propose novel Spectral Transformer, which processes video latents in the frequency domain to learn video representations more effectively. Consider quantized latent E (BB.N../16..//) at time step f. First, the latent is flattened to ( H/16XW/66) which along with time step t and FPS (frames per second), are passed our proposed Spectral Transformer block. Initially, the passed tokens are fed into an Adaptive Layer Normalization (AdLN) block to obtain B,, where
B, = AdLN(zz ze. I FPS).
Text ambedding
Rotary positional ambedding (RoPF)
IDD
Kiwi
MSA
TXX)
B2DD
Kyng
Rey Zego Zog Zesmmnet
R.
Self atten
Cross sitenn tion
iMaD
Point wise Fr
AALN
RED
AdLN
RDDD
Visw
AdLN
AdEnN
(t.FPS)
M2D
Qrw
(LPPS)
Quesr
(tFPS)
(L.PS)
Figure 4: Architecture of spectral transformer
As shown in Figure 4, before computing the self self-attention, the latent representation B, is trans- formed into the frequency domain using a 2D Fourier Transform Layer (ffi2D) for each of K, V. and Q This layer applies two ID Fourier transforms (Susladkar et al., 2024; 2023) along the embedding dimension and the sequence dimension thereby converting spatial domain data into the frequency domain. This enables the segregation of high-frequency features from low-frequency ones, enabling more efficient manipulation and learning of spatial features. We take only the real part of ff22)) layer outputs to make all quantities differentiable (Eq. 2). Here, MSA stands for multi-headed self-attention.
Zum MSA ( (IR (Z.))) K (R())2D(Z)))) V Pay (B(ff2((Z))))
(2)
The frequency- domain representations are then fed into the self-attention block. Using Rotary Posi- tional Embeddings (RoPE) at this stage helps increase the context length of the input and results in faster convergence. The output Zattn undergoes an Inverse 2D Fourier Transform (iffiDD to revert to the spatial domain. Then, residual connection with the AdLN layer applied to preserve the original spatial information and integrate it with the learned attention features. This gives the output Zour (Eq. 3).
Toz = (If + AdLN t, FPS))
(3)
Similarly, in the cross-attention computation, the text-embedding T(X,) is first transformed using learnable Fourier transforms for both K and V. The Q representation from the preceding layer is also passed through the learnable Fourier transforms (Eq. 4). These frequency-domain representa- tions are then processed by the cexticondit cross-attention block.
Zeross-arra MSA (Qray (ffi)((((())))))) 2mm) kmm))) Krng (L(()FD(T((X)))) (Ro((B(((2(((((())))
(4)
Then, the frequency domain output is converted back to the spatial domain using ifft2D layer, and a residual connection is applied (Éq. 5).
Zurrss out (MI.(((((f22)) amm) )} AdI.N f, FPS))
(5)
Finally, an MLP followed by softmax layer gives the predicted denoised latent given 24. This process is repeated until we get the completely denoised latent P(z, 2,, t, T(X.)) Finally, these output latents are mapped back into video domain using our pre-trained 3D-MBQ- VAE decoder D.
5

---

Published as a conference paper at ICLR 2025
4 MOTIONAURA FOR SKETCH-GUIDED VIDEO INPAINTING
MotionAura can be flexibly adapted to downstream video generation tasks such as -eetch-guided video inpainting. In contrast with the previous works text-guided video inpainting (Zhang al., 2024; Ceylan et al., 2023), we use both text and sketch to guide the video inpainting. The sketch- text pair makes the task more customizable. For finetuning the model, we use the datasets curated from YouTube- VOS and QuickDraw (Appendix C.4). We now explain the parameter-efficient fine- tuning approach. Figure details the entire process. For given video V = (ff))= we obtain masked video Vn using corresponding mask sequence m {ma) where Vm is given by Vn {f @ (1 m We then discretize both these visual distributions using our pretrained 3D-MBQ- VAE encoder, such that 2m = EVm)) and 2g = (V).
Given masked video, training objective to ob- tain the unmasked frames with the context driven by sketch and textual prompts. Hence, we randomly cor- rupt the discrete latent 20 with <MASK> token (Sec- tion 3.2) to obtain the fully- corrupted latent 2r. Af- ter performing the forward diffusion process, and Am are flattened and con- catenated. This concate- nated sequence of discrete tokens is passed through an embedding layer to obtain Z E (B,,,E) where Zi represents the intermediate input token sequence.
Flax Lea)
(s PPS)
Tranable
(B.S)
Z
MAE Chickerr
(2,,
Uniaskked
Enbeddiigg
Videos
60
Dreader
Z,
Masied Videoo
Vin
Spectral Transformer
SiLIP
Skatch
(8,)',8")
(8.8", E))
Conditioning
Conditioning
Caption
TB XXL
Figure S: Sketch-guided video inpainting process. The network inputs masked video latents, fully diffused unmasked latent, sketch condi- tioning, and text conditioning. It predicts the denoised latents using LoRA infused in our pre-trained denoiser eg.
As for the preprocessing of the sketch, pass the sketch through the pretrained SigLIP image encoder (Zhai et al., 2023) and consecutively through an MLP layer to finally obtain s' E (B.S' At last, s' is concatenated with Z; to obtain the final sequence of tokens Z,, where Z, = Z/ $ S' and Ze E R(B.S+5',E) This input sequence is responsible for the above mentioned context to the diffusion network. The input sequence already contains the latents of the unmasked regions to provide the available spatial information. It also contains the sketch information that helps the model understand and provide desirable visual details.
The base model is frozen during the inpainting task, and only the adapter modules are trained. The pretrained spectral transformer augmented with two types LoRA modules during finetuning: (1) Attention LoRA, applied to Kproj and Qproj layers of the self-attention and cross-attention layers of the transformer module, operating in parallel to the attention layers, (2) Feed Forward LoRA, applied to the output feed-forward layer in sequential manner. These two adapter modules share the same architecture wherein the first Fully Connected (FC) layer projects the high- high-dimensional features into lower dimension following an activation function. The next FC layer converts the lower dimensional information to the original dimension. The operation is shown as L'(x) = L(x)+ W.p (Gel.GGWlaaao.o down (z:)) such that W.p. E (d.l)) and Wdown E (l.d) with 1 < d. Here W.p and W Woww represent the LoRA weights, T'(x:) represents the modified layer output and I(xx) represents the original layer output. The W Wdown is initialized with the distribution of weights of L(x) to maintain coherence with the frozen layer.
5 EXPERIMENTAL RESULTS
We now present experimental results. The details of experimental setup are provided in Appendices C and D. Additional qualitative results are provided in Appendices A, G and F.

---

Published as a conference paper at ICLR 2025
5.1 LATENT RECONSTRUCTION RESULTS OF 3D-MBQ VAE
For evaluating our DD-MBQ-VAE, we selected COCO-2017 and WebVID validation datasets. Fol- lowing Zhao al. (2024), we crop each frame to 256 resolution and sample 48 frames per video sequentially. As shown in Table I, our proposed 3D-MBQ-VAE consistently outperforms SOTA 3D VAES across all metries. This can be attributed to novel combination of training strate- gies and the enhanced loss function. Randomly masking parts of various frames He et al. (2021) enables our model to efficiently learn spatial information. Further, inclusion of LMFT in the loss function emphasizes learning efficient temporal features. On removing Lmui, LPIPS metric decreases substantially. This highlights its role in preserving perceptual quality. Our 3D-MBQ-VAL VAB model is also capable of supporting joint image and video training, similar to W.A.L.T Gupta et al. (2025) and CV- As shown in Table 1, joint training provides superior results than video training alone. Table 1: Quantitative comparison of VAFs for video reconstruction task on COCO-Val and WehVid Val datasets. Frame compression rate (FCR) is the ratio between the size of video frame before and after com- pression. Compatibility (comp.) represents if the model can he used as VAF for existing generative models. Our method demonstrates superior performance on all metrics, All models are trained on videos, except the "3D-MBQ-VE VAE (Video+Images) variant that is jointly trained on video and images. Hence, its results are not compared with other methods.
Method
Params
FCR Comp.
COCO-Va Val
WebVid-Val PSNR / SSIM LPIPS
PSNR(1) (ANR())/SIM(()) SSIM()) / LPIPS(()
VAE-SD2,1 (Rombach el al., 2022h)
34M 49M
1x
0.76/ / 10.127
28.9/0.810/ / 0.145
VQGAN (Esser et al., 2021)
26M 38M
1x
X
22.7 0.667 086
2..6/0.718 0.1
TATS (Ge et al., 2022)
7M + 16M
4x
23.4/ 0.741 0,287
24.1 0..29/ 0.310
VAE-OSP (Lab & etc., 2024)
94M 135M
4x
27.0/ 0.791 / 0,122
26.7 /0.781 0.166
CV-VAE(2D+3)) (Zhao ct al., 2024)
68M + 114M
4X
V
0.805/ / 036
/ 0.
CV-VAR(3D) (Zhao et al., 2024)
100M 156M
4x
27.7 0.805 / 035
2 0.145
3D-MBQ- VAE (Without Lma)
140M 177M
4X
30.8 O. 0.122
314/ 0.134
3D-MBQ-VAE
140M 177M
4X
V 31,2 / 0,848 / 0.092
32.1 / 0.858/ 0.108
3D-MBQ- VAE (Vidco+Images)
140M 177M
4x
33.0 / 0.866/8 / 0.087
/ 0.099
Figure 6 shows the t-SNE plots, where each color represents different class in the dataset. Ev- idently, the better clustering and separation by 3D-MBQ-VAN shows that it is most effective in feature extraction and representation learning. The qualitative results in Figure 7 further confirm that 3D-MBQ-VAE provides reconstructions of highest quality and consistency.
SDXL-VAE
TATS-VAS
CV-VAE
3D-MBQ -VE
Figure 6: t-SNE plots showing the representations learned by various VAEs.
5.2 RESULTS OF TEXT CONDITIONED VIDEO GENERATION
For evaluation purposes, we create three variants of our model, Small (0.89B parameters), Medium (1.94B parameters), and Large (3. 12B parameters) as shown in Table 2. Here, blocks refers to the num- ber of spectral transformer blocks (N in Figure 3). For zero-shot Tex12Video generation, we pre-train our Vext2Video models with specific configurations (refer appendix D.3). We compare our model with AnimateDiff (Guo et al., 2024), Cog VideoX-5B (Yang et al.,, 2024a) and SimDA (Xing et al., 2023). The texts given as text conditioning to all these models are taken from the WebVID 10M dataset (with a maximum length of 180). However, we observe that the captions in the original WebVID 1OM dataset lacked contextual richness, which limits the scope of generation. Hence, we propose our WebVID 10M-recaptioned dataset with contextually rich textual data (details in Ap- pendix C). We evaluate the pre-trained models the MSR- VTT dataset (Chen et al., 2022) using standard metrics such as FVD and CLIPSIM. We also included two human assessment- rics, viz., Motion Quality (MQ) and Temporal Consistency (TC) (Liu et al., 2024; Feng al., 2024).
Table 2: Comparison of MotionAura variants.
Model Embedding Size Atin head Blocks
1024
16
17
2048
16
28
4096
32
37
met-
7

---

Published as a conference paper at ICLR 2025
MotionAura-L
MotionAura-L fft)
VAB O6f
VAE 50 XL
TATS
CV-WE
DURS
female painter with a brush in hand, white background, painting, looking very powerful.
Figure 7: Comparison of 3D-MBQ- VAE (ours) with ex- isting VAEs after reconstruction from Stable diffusion-XX XL produced latent. Zooming on the figure reveals signif- icant differences in image quality, fidelity to the original latent representations, and the preservation of fine details.
Figure 8:
The videos corresponding to these frames show the improvement in motion quality brought by FFT layers. For videos, click on link.
The EvalCrafter benchmark helps us quantitively evaluate the quality of video generation on these aspects. The framework also aligns objective metrics with human opinions to enhance evaluation accuracy. 100 participants ranked each of the Y models such that the model ranking first received Y points and the model ranking last received 0 points.
As shown in Table 3, pretraining on our newly curated dataset improves the performance of all the models, highlighting the importance of enriched captions. Further, MotionAura-L. without the use of ff22 and iffi2D layers has inferior performance than the full Motion network, showcasing the effectiveness of our proposed Spectral Transformer architecture. Finally, our largest model, MotionAura- achieves the best results across all metrics, demonstrating its superior capacity for capturing both motion dynamics and temporal consistency.
We measure the average inference latency over text sample lengths of 30 to 180 tokens. For generat- ing a 5-second video, MotionAura-L. takes 38 seconds, compared to 41 seconds of Cog SigVideoX-5B (Table 3). To reduce the latency, the FFT layers can be skipped, or the small/medium variants can be used. Notably, MotionAura-S comparable latency to AnimateDiff and superior in performance metrics. MotionAura can generate videos of up to 10 seconds, whereas previous works generate up to seconds videos. MotionAura-LL takes 83 seconds to generate a 10-second video.
Table 3: Results of the text-conditioned video generation (Text2Video) models. For all the techniques, evalu- ation was done on MSR-VTT dataset. Recaptioning the dataset improves all the metrics. Inference Time was calculated single A100.
Method
Params
Train: WebVID 10M
WebVid 10M - recaptioned Inf Tims(s) (FVDCCLISSIM/MQTTC)
(FVD//CLPSIIM/MM/CCT))
SimDA
1.08B
456 1761 / 65.28 / 195
/ 67.82/ / 196
8.6
AnimateDiff
1.96B
402/ 1 / 68.97 / 197
379 0.2201 / /69.78 / 199
11.1
Cog VideoX-BB
5.00B
3 / 73.37 / 203
357 0.2429 75..5 / 205
41.0
MotionAura-L (w/o fit)
3.12B
/.2200 / 73.44 202
351 0.22441 / 75.87 / 203
33.4
MotionAura-S
0.89B
391 /.2114/ 70.29 199
3 71.28 / 200
12.0
AuraMMA
1.94B
383 0.2207 72.37 / 200
360 / 0.2333 / 73.47 / 202
20.0
MutionAura--I.
3.12B
3 0.2522 / 74.59 / 204
344/ 0.2822 / 76.62 / 207
38.0
Figure 8 shows sample frames from videos generated by the full MotionAura-L and MotionAura- L(without fft2d). The videos highlight superior motion quality achieved on using the FFT layers. Figure 9 compares our model with Cog Videoxx55X55 and AnimateDiff. Both quantitative and qualita- tive results highlight the superior performance our model, which can be attributed to several key innovations. AnimateDiff relies on the image VAE from Stable Diffusion to encode video latents, which limits their representational power compared to the latents produced by our 3D-MBQ- VAE. AnimateDiff inflates 2D diffusion model into 3D using temporal mod-moule for video, but this approach fails to ensure true temporal coherence, as seen when comparing its results to those of AutionAura. Cog VideoX-5B employs video VAE in continuous space. This approach struggles
8

---

Published as a conference paper at ICLR 2025
with efficiently representing the high dimensionality of video frames, making it less effective than discrete space representations. Additionally, our use of spectral transformer blocks, with integrated 2D Fourier transforms, enhances both vision and language modeling by capturing complex spatial and temporal patterns. Replacing traditional sinusoidal embeddings with RoPE embeddings further improves the handling of longer video sequences. These architectural improvements collectively improve our overall performance.
CogVideoX-9B
AnimateDiff
MationAuro-L
orgarry The moad"
dadic blk paint sig bumnunt tnd usaling no thruugh the rilo of Nom rwaniing lerp by Grrny Rloe"" tiick delke anll lust"
park in Jocn a white hariin c Lish troxa aurouun and sortl."
Figure 9: Text-conditioned video generation results. Our model shows superior temporal consistency and generation quality. Click on the link to view the videos.
5.3 RESULTS OF SKETCH GUIDED VIDEO INPAINTING
We now assess the models' performance on the newly introduced task of sketch-guided video in- painting. As outlined in Section 4, our approach adapts zero-shot Text2Video models using LoRA modules. For a fair comparison with Cog VideoX-5B, we adapted it in a similar manner as our pro- posed approach. Both existing and our models were pre-trained on the WebVid 10M-recaptioned dataset and evaluated on two newly curated datasets, based on existing video (YouTube-VOL and DAVIS) and sketch (QuickDraw) datasets. These datasets have been specifically designed to ac- commodate the sketch-based inpainting task, featuring four key components for each video sample: textual prompts, sketches, segmentation masks, and raw video. For each experiment, we inpainted portions of the videos using both the sketch and text inputs as guides.
As presented Table 4, MotionAura-L outperforms all other methods. The superior performance of MotionAura-L can be attributed to the combination of sketch and text inputs, which provide a richer context for guiding the inpainting process. While text descriptions offer general understand- ing of the scene, the sketch provides explicit structural information, allowing the model to generate more accurate and coherent inpainting results. This dual-input method leads to better spatial align- ment and temporal consistency, as evidenced by the higher scores when compared to MotionAura-L (w/o sketch), which relies solely on text input. The qualitative comparisons in Figure 10 show that MotionAura leads to more spatially and temporally coherent results than previous techniques.
Table 4: Quantitative evaluation for sketch-based inpainting, All models were pre-trained using the Web 10M -recaptioned and evaluated over the newly curated datasets comprising DAVIS and YouTube-VOSS
Method
DAVIS
YouTuhe- (PVD/CLIPSIM MC/TC)
(FVDJ/CLIPSIME MQ1 / /Ct)
T2V (Khuchatryan al., 2023)
/.2499 63.39 / 191
7.0/0. 64.28 193
SimDA (Xing et al., 2023)
752/ 0.2564 65.28 196
693 0.269/ 67.66 82/ 195
AnimateDift (Guo
2024)
737 68.97 199
685 0.2701 66. 197
CogVideoX-SB (Yang et al., 2024b)
0.25122 73.337 205
677 0.2602. / 75.51 51 203
MotionAura-LL (w/o fit)
689 0.2999 73.44 203
663 0.3164 75.87 202
MutionAura-L (wla sketch)
687 0.3011 1 73.31 200
666 0.3061 / 75,14 199
MotionAura. S
704 0.2776/ 70.299 200
670/0.2892. / 71. 11.28/ 199
MutionAura-M
692 0.2901 72.337 202
666 / 73.47 200
MotionAura-L
685 0,3101 744,5) 207
657 / 0,3511 76.62 204

---

Published as a conference paper at ICLR 2025
Deput
Aniretebiff
OeghideoxX 58
Mottannant--l
Figure 10: Sketch- Sketch-guided inpainting results of various techniques. The red circles highlight the mistakes and misalignments, e.g., SimDA incorrectly inpaints the parts of the hat and the butterfly. DnimateDiff inpaints butterflies in some frames, and inpaints hral-transparent butterfly in the last frame. Cog VideoX produces hat with incorrect shape, and a black-color butterfly. By contrast, MotionAura produces high-quality and temporally consistent results.
5.4 ABLATION STUDY
1. We pretrain MotionAura on the WebVid 10M - recaptioned dataset and evaluate it on the MSR- VTT dataset for the text-conditioned video generation task. We evaluate the effect of the length of text description on the quality of the generated images. Increasingly detailed text descriptions provide better guidance and context. By virtue of this, the model captures finer nuances in scene composition, motion, and object interactions. Thus, the generated videos show superior quality and better match the user's requirements. Hence, with increasingly detailed descriptions, FVD reduces and CLIPSIM increases (Table 5).
Table 5: Impact of Text Length
Table Impact of LoRA Rank
Length
CLIPSIM
LoRA Rank
FYD
CLIPSIM
364
0.2544
60
359
0.2598
8
662
0.3291
90
355
0.2626
16
660
0.3349
120
352
0.2654
32
659
0.3421
150
349
0.2701
64
657
0,3511
2. We evaluate the effect of rank of LoRA adaptors the quality of sketch-based inpainting results on the YouTube- VOS dataset. A higher rank the LoRA Adaptor involves updating more trainable parameters, which helps the model learn better representation and enhances model accuracy. Hence, a higher rank improves model performance, as shown in Table 6.
We evaluate MotionAura using sinusoidal embedding for the text-conditioned video genera- tion. The FVD/CLIPSIM values are 379/0.2392 after pretraining on the WebVID 10M dataset and 349/0.2752 after pretraining on the WebVID 10M-recaptioned dataset. Clearly, RoPE embeddings (Table 3) provide superior results than the sinusoidal embeddings.
CONCLUSION
In this paper, we present MotionAura, a novel approach to generating high-quality videos given some text and sketch conditions. The videos generated by our model show high temporal consistency and video quality. MotionAura proposes several novelties, such new masked index loss during VAE pretraining, using FFT layers to segregate high-frequency features from the low-frequency ones and using RoPE embeddings to ensure better temporal consistency in denoised latents. Additionally, we recaption the WebVid- 10M dataset with much more contextually rich captions, improving the denoising network's 's quality. Finally, we curate datasets to address the task of sketch- sketch-guided guided video inpainting. Rigorous experiments show the effectiveness and robustness of MotionAura in video generation and subsequent downstream tasks.
10

---

Published as a conference paper at ICLR 2025
Support for this work provided by Science and Engineering Research Board (SERB) of India, under the project CRG/2022/003821. Jishu and Chirag contributed this project while working as interns at IIT Roorkee.
Acknowledgement:
REFERENCES
Nuha Aldausari, Arcot Sowmya, Nadine Marcus, and Gelareh Mohammadi. Video generative adversarial doi: 10.1145/3487891. URL
networks: A review. ACM Computing Surveys, 55(2):1-25, January 2022.
atttss://ci... wrg.10...o.o.o.oo 114533377911/33
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. arXiv preprint Scti:2203.1599/ 2021. URL https: / //ooi. org/10. 15691. Submitted 29 Mar 2021 (v1), last revised 1 Nov 2021 (this version,
48550/arxiv v2). 2
Max Bain, Arsha Nagrani, Gäl Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder end-to-end retrieval, 2022. URL https: /ttpss///axvvvv.. http:///aawv.oo.g./ra.///// wwtp://///aaaaaaaaao//oo/aaaaa/a abs/2104.... 00650.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models, 2023. URL https: //arxiv. wtw. wttp/rs//////////////.//eee/ee// http://b////3/// 2304. 08818. 3
Benjamin Bross, Ye-Kui Wang, Ye, Shan Liu, Jianle Chen, Gary J Sullivan, Main-Rainer Ohm. Overview of the versatile video coding (vve) standard and its applications. IPFE Transactions on Circuits and Systems for Video Technology, 31(00)33736-3764 2021. 20
Duygu Ceylan, Chun-Hao Paul Huang, Niloy J. Mitra. Pix2video: Video editing using image diffusion, 2023. URL https::/p//. / /aaaaaivvvvooo hww.v/org/bbbaa abs///a////// 22003.3333.2 12688.
Haoran Chen, Jianmin Li. Simone Frintrop, and Xiaolin Hu. The msr-video text dataset with clean anno- tations. Computer Vision and Image Understanding, 225:103581, December 2022. ISSN 1077-3142. doi: wvw.:..ciiu.222..1.
URL http:://// / /dxx doi. org/ 100 1.001000 1016/111101111 6/..
103581. 7
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorcalistic text-to-image synthesis, 2023. URL https:/ http://aaxvvvoo/oo
wogg/rr//ooo.................... 22300. 00426. 3
Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Diffusion transformers for image and video generation, 2024.
URL https://///// / arxiv.. wwtp/rgg////.//ooo..t....t..asst /2ll2 211.2.321111111.. 04557.
Loic Dehan, Wiebe Van Ranst, Patrick Vandewalle, and Toon Goedemé. Complete and temporally consistent vidco outpainting. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 686 694, June 2022. doi: 10. 11L.1//CVVPW555722222220000
Patrick Esser, Robin Rombach, and Björ Ommer. Taming transformers for high-rcsolution image synthesis, 2021. URL https:
C9841.
Patrick Fsser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Do- minik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhom, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024. URL https:/ / /aaaaiiiii...//.. www.i.iogg/aaaa.a...aaaaaoaeeeae abs/ 2403 03206.
Jiasong Feng, Ao Ma. Jing Wang, Bo Cheng, Xiaodan Liang. Dawei Leng, Yuhui Yin. Fancyvideo: Towards dynamic and consistent video generation via cross-frame textual guidance, 2024. URL https:
Slarxiv.crge wtt./bgg////..ootooooo.o/../.. abs 2408 081898999 1999
Kaifeng Gao, Jiaxin Shi, Hanwang Zhang, Chunping Wang, and Jun Xiao. Vid-gpt: Introducing gpt-style autoregressive generation in video diffusion models. arXiv preprint 20Xx:2406611981,, June 2024. URL
https://dci..o http/dic...rrggg// /10. 13555//arXiv... 240..1196...1
Songwei Ge. Thomas Hayes, Harry Yang, Xi Yin, Guan Pang. David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer, 2022. URL https:// arxiv. wewiv.orrg/aae absp//b//2200 03638.
Ian J. Goodfellow, Jean Pbbdet-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL abs/1 1.1..6..666111
11

---

Published as a conference paper at ICLR 2025
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10696-10777 10706, 2022. 24
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang I Liang, Yaohui Wang. Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning, 2024. URL httpt::///...... /tapps://aree wawxiv.iii..//... org/ abs 2307 04725. 7,9
Agrim Gupta, Lijun Yu. Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Jiang, and José Lezama. Photorealistic video generation with diffusion models. In European Conference on Computer Vision, pp. 393-411, Springer, 2025. 2,
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are 06377. 4,7
scalable vision learners, 2021. URL https:/ /arxiv.ivv....///ia wwwp:iiiv.oooooooooooooo
Shawn Hershey, Sourish Chaudhuri, Daniel W. Ellis, Jort F. Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold. Malcolm Slaney, Jon Weiss, and Kevin Wilson. Cnn architectures for large-scale audio classification, 2017. URL https://axxlv.. wwwpexiiiioooraaooooooooooeeee aat//bbs//6/ /169. 09430. 21
Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models, 2020. URL https: //arxiv wwt.ergg/bbs/s/e/eee 1239.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. arXiv preprint arXiv:2204.03453 2022a. URL atttps://do.. arg/10. 48550/
arXiv.2204 03458.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Flect. Video diffusion models, 2022b. URL https://arxtt: org/ borg/abs//a a224 0ttp/2224.044388 03458.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 20Kv:2500.09665, 2021. 2
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free free generative adversarial networks, 2021. URL https::////t http///aaavvv..e//ee/eeeeeaeee.a org/ abs /
2002216.00 12423. 3
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Text2vieeo-zero: Text-to-image diffusion models are zero-shot video generators, arXiv preprint arXiv::303. 13439, 2023. 9
Diederik P. Kingma and Max Welling. An introduction to variational autoencoders. Foundations and Trends@ in Machine Learning, 12(4)3077-322 2019. ISSN 1935-8245. doi: 10.1561/2200000056 URL http: // dx ww............ crg/10 1500..566660 111.11/11/2220066..
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vigh- nesh Birodkar, Jimmy Yan, Ming-Chang Chiu, Krishna Somandepalli, Hassan Akbari, Yair Alon, Yong Cheng, Josh Dillon. Agrim Gupta, Meera Hahn, Anja Hauth. David Hendon. Alouso Martinez, David Min- nen, Mikhail Sirotenko, Kihyuk Sohn, Xuan Yang. Hartwig Adam, Ming- Hsuan Yang, Irfan Essa, Huisheng Wang, David A. Ross, Bryan Seybold, and Lu Jiang. Videopoet: A large language model for zero-shot video generation, 2024. URL https. / /arxiv. org/ abs 22122222222 2.2..44 14125.
PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, April 2024. URL https: /ttp::::////....//..../3.. zenodo_ 1 009411181001 09. Software. 7
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models, 2024. URL https: 2407.0000 07895. 21
Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Ticyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large vidco generation models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22139- 22149, 2024.7
Ruibo Ming, Zhewei Huang. Zhuoxuan Ju, Jianming Hu, Lihui Peng, and Shuchang Zhou. A survey on video prediction: From deterministic generative approaches. arXiv preprint arXiv:2401. 14718, January 2024.
URL https::::// //cc: crg/ er1/10. 48550/ arxiv. 2401.141144 14718. Under review, 2
William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. URL https://
arxiv. arxiv.org/abs//2222 39
12

---

Published as a conference paper at ICLR 2025
E. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Ilorung. benchmark datasct and evaluation methodology for video object segmentation. 2016 JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 724-732, 2016. doi: 10.1109/CVP..2016.0 22
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution
image synthesis with latent diffusion models, 2022a. URL pttpppp//axii.....//////////p.tt http://axvvvv.co/ org/ abs 2tba/211..1172
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2022b. URL httppp/://aaaaaoooo//////2222110
org/ http://aab/211..1/111.
Claudio Rota, Marco Buzzelli, Joost van de Weijer. Enhancing perceptual quality in video super-resolution through temporally-consisten detail synthesis using diffusion models, 2024. URL lorg/bbs abs/////// / 2211. 15908. 21
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022. 24
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datascts, 2022.
URL / org./bss//////ao...o...o.. 00273. 3
Nakul Sharma, Aditay Tripathi, Anirban Chakraborty, and Anand Mishra. Sketch-guides image inpainting with partial discrete diffusion process. arXiv preprint 2tdi:22404.119443 2024. 24
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.0965 094,0444, 2021. URL https: //doi orgg 10. 48540/550050 /at.///rrvv.v... wtwiv.2200e.... 2104.11400 09864. Fixed some typos. 2
Gary Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. IEEE Transactions circuits and systems for video technology. 22(11): 2012. 20
Kegiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydh: A benchmark for generative image understanding. Advances in Neural Information Processing Systems, 36, 2024a. 23
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng. Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation, 2024b. URL htttp:://arxxy org. abe/ 2406. 0652..
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 2665522400.06555, 2024c. 19
Onkar Susladkar, Gayatri Deshmukh, Dhruv Makwana, Sparsh Mittal, R. Sai Chandra Teja, and Rekha Singhal. GAFNet: A Global Fourier Self Attention Based Novel Network for Multi-Modal Downstream Tasks. Pro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5242-5251, January 2023. 5
Onkar Susladkar, Gayatri Deshmukh, Vandan Gorade, and Sparsh Mittal. GRIZAL: Generative Prior guiddd Tero-Shot Temporal Action Localization. Proceedings of the 2024 Conference on Empirical Methods in Natural I Laguage Processing, pp. 19046-19059, 2024. 5
Onkar Susladkar, Gayatri Deshmukh, Sparsh Mittal, and Parth Shastri. D2Styler: Advancing Arbitrary Style Transfer with Discrete Diffusion Methods. In International Conference on Pattern Recognition, pp. 63 82. Springer, 2025. 3
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers, 2022. URL https: http:://axivvvvoee abs/ 220./.09.1...0 10686. 5
Petru-Daniel Tudosiu, Thomas Varsavsky, Richard Shaw, Mark Graham, Parashkev Nachev, Sébastien Ourselin, Carole H. Sudre, and M. Jorge Cardoso. coding using vg-vae. arXiv preprint arXiv:2002.05692. February 2020. URL https: ///////iii/// http://ooo.o//.../////// 48550 /aroaariiivi.... 2002. 05692.
wwwppppoooooiclllypeeeriiiiiiiie volumetric data en-
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning, 2018. URL /arxi wwtpw/wwwwrg/////ot 1711 00937..
13

---

Published as a conference paper at ICLR 2025
Haiqiang Wang, Weihao Gan, Sudeng Hu, Joe Yuchieh Lin, Lina Jin, Longguang Song. Ping Wang. loannis Katsavounidis, Anne Aaron, and C-C Jay Kuo. Mcl-jev: jnd-based h. 264/ave video quality assessment dataset. In 2016 IEEE international conference on image processing (ICIP), pp. 1509-1513 1513. IEEE, 2016. 20
Jianzong Wu, Xiangtai Li, Chenyang Si, Shangchen Zhou, Jingkang Yang, Jiangning Zhang, Yining Li, Kai Chen, Yunhai Tong, Ziwei Liu, and Chen Change Loy. Towards language-driven vidco inpainting via multimodal large language models. arXiv preprint arXiv:2401. January 2024. URL https:
//doi. org/ 10. 48550 /atriiv...vv.. 2...2.40111000 1.226.2
Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. Simda: Simple diffusion adapter for efficient /ttp///03000000100000 0. 7,9
video generation, 2023. URL wwwphttpp////.
Ning Xu, Linjic Yang, Yuchen Fan, Dingeheng Yuc, Yuchen Liang, Jianchao Yang. and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark, 2018. URL httpt:://tati...tt
22
Wilson Yan, Yunzhi Zhang, Pieter Abbecl, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers, 2021. URL htwph:ttttttt.c.t. arxiv. wwwp//aa////aaooooo/ 10157.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an expert transformer. 2024a. URL htwps:t////mmm / waw.iv.viiiv///..//..aa/aaaiaava org/ ab// /2409. 06072.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Tex--tovideo diffuusion models with an expert transfomer. 2024b. URL https: / arxiv. org/ abs 2408. 06072. 9
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Hsuan Yang. Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10459-10469, 2023a. 20, 23
Lijun Yu. Yong Cheng. Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming-Hsuan Yang. Yuan Hao, Irfan Essa, and Lu Jiang. Magvit: Masked generative video transformer, 2023b. URL httpsts::/ / /trxaaxiivv//. wxv. wrt. / 2212 05199.
Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuar Yang. Trfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion tokenizer is key visual generation. arxiv preprint /0.0553, Octoher 2023c. URL https: / /doi .org/11 100 48550/ arxiv.230 0.100..00 05737. 2, 20
Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn. David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation, 2024. URL https: http://xxvvvrrrr/ www.voorg//bba. abs// 22100 05737.
Xiaohua Zhai Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training, 2023. URL https: http:://aaxvvvv.a.eeee
153433
Zhixing Zhang, Bichen Wu, Xiaoyan Wang, Yaqiao Luo, Luxin Zhang, Yinan Zhao, Peter Vajda, Dim- itris Metaxas, and Licheng Yu. Any-length video inpainting with diffusion model. arXiv preprint arXiv:2312.03816 December 2023. URL https:/ /doi /tt.//oo.ooooo//////////o org/10. 48550/arxiv.22
ar..i..2.vv.... 03816.
Zhixing Zhang, Bichen Wu, Xiaoyan Wang, Yaqiao Luo, Luxin Zhang, Yinan Zhao, Peter Vajda, Dimitris Metaxas, and Licheng Yu. Avid: Any-length video inpainting with diffusion model, 2024. URL https:
// /aaxivv. crg/ abs//bb/////// 03816.
Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. Cv-vae: A compatible video vae for latent generative video models, 2024, URL https:/ //tgxvv.vv./////// abs/2405, 20279.
14

---

Published as a conference paper at ICLR 2025
Supplementary Materials
Motion Aura
Figure S.1: Qualitative results for our proposed MationAura. The corresponding videos can be found by clicking on the link.
15

---

Published as a conference paper at ICLR 2025
A 10 SECOND GENERATED SAMPLES
Figure S.2: Click here to see the video.
Figure S.3: Click here to see the video.
Figure S.4: Click here to see the video.
Figure S.5: Click here to see the video.
Figure S.7: Click here to see the video.
Figure S.6: Click here to see the video.
16

---

Published as a conference paper at ICLR 2025
3D-MBQ-VAE
3D-MBQ-VAE
3D-MB Conv Block
Conv3D
Conv3D Trenspose,
Com33 Layer, 3x3x3
stride 1x2x2,
stride 1x2x2,
padding 1xtxt,
3D-MB Cony Block
3D-MB Conv Block
2x
ConvaD Block
Conv3D Block
C-+x4C
Conv3D Leyer. 1x3x3 stride 1x2x2
DownSannlle, 1x3x3 stride 1x2x2, 64 64
3D-MB Conv Block 128
Layer
4C-> -C 4C
UpSample, 1x3x3 stride 1x2x2, 256 256
Depth-wiise Conv, 3xcha3 padding 1x1x1,
Group Normaizztion
64 120
GelUU
Incestion-Fused Conv Block 256 256
Conv3D Block
Inceplion-Fusad-MB
3D-MB Conv Block
DawnSamplc. 1x3x3 stricl 1x2x2, 128
B. C. N, H. W
UpSample, 1x3x3
Inception-Fused-MM
stride
C. N, W
128 512
Incegtion-Fuee--Ny Conv Block 256 +> 258
2x
Conv3D C-2C 3x3x1,
Conv3D C-2C 3x1x3, padding 1,0,1
Conv3D C-22C 1x3x3, padcing 0,1.1
Inseplion-Fusee--Me Conv Block, -> 512
3D-MB Conv Block 512 -> 256
padding 1,1,0
DewnSample, 1x3x3
UpSample, stride 1x2x2, 512 -a 512
3D-MB 3D-MB-Conv 2C .> 2C
3D-MB-CoRv 2G .2 2C
3D-MB-Corv 20 >> 2C
3D-MB Cony Block
3D-MB Conv Block
Conv3D Block, 1x3x3 stride
Conv3D Block, 1x3x3
Conv3D Block
Quantization
Sampling
B.C.
stiid 1xtx1 (unless mentioned), padding Dx0x0 (unless mentione))
Figure S.8: 3D-MBQ VAE architecture.
B 3D-MBQ-VAE
B. 3D-MBQ- VAE ARCHITECTURE
Figure S.8 shows the 3D-MBQ-VAD (Mobile Inverted Vector Quantization Variational Autoen- coder) architecture designed to efficiently encode and decode video data while utilizing both mobile inverted bottleneck blocks and vector quantization techniques. The MBQ-VAL VAE framework is divided into two main parts: the encoder and the decoder, with quantization and sampling in between.
Encoder The encoder takes an input video with dimensions B x C x x H x W, where B is the batch size, C' is the number of channels, and N H x W represents the spatiotemporal dimensions (i.e., frames, height, and width). The encoding begins with Conv3D block that captures spatiotem- poral features using a 3D convolution with a kernel size of 1 3 3, followed by stride of 1 2 to reduce spatial resolution while expanding the channel depth (e.g.. from 3 to 64 channels). The encoder then applies multiple 3D-MB Conv blocks (Mobile Inverted Conv Blocks) capture com- plex features using depthwise separable convolutions, maintaining computational efficiency. These blocks are interspersed with Ceception-Fuseddee by combining multiple filter sizes and dimensions, enabling multi-scale processing.
Cony blocks, which improve feature extraction
At various stages, downsampling is performed via strided 3D convolutions to reduce the spatial and temporal dimensions further while increasing channel depth, helping the network focus on abstract, high-level features. This progression continues until the latent representation is compressed into a vector using final Conv3D block. This compressed representation is passed to the quantization layer, where vector quantization applied to discretize the latent variables for use in the VQ-VAE.
17

---

Published as a conference paper at ICLR 2025
B.1.1 DECODER
The decoder mirrors the encoder but in reverse, beginning with vector sampling from the quantized latent space. It starts with an initial Conv3D block to upsample the channels from the quantized la- tent vector. The network then proceeds through several 3D-MB Conv blocks and Inception- Eeception-Fusedd Fused- MB Conv blocks. gradually increasing the spatial resolution via Upsampling layers. Combined with inverted convolution blocks, these layers allow the decoder to reconstruct fine-grained details from the compressed latent vector.
The upsampling process continues steps, restoring the spatial and temporal dimensions while reducing the channel depth to match the input dimensions. The final layer of the decoder uses a Conv3D Transpose operation to transform the latent representation back to the original input resolution, ensuring the correct number of channels is maintained for video reconstruction.
B.1.2 KEY FEATURES
Mobile Inverted Bottleneck Convolutions: The use of 3D-Mobile Inverted Bottleneck (MB) Conv blocks enables the model to achieve high efficiency by expanding and con- tracting the number of channels at different stages while preserving critical features through depthwise separable convolutions.
Inception-Fused Convolutions: By incorporating multiple kernel sizes in the Inception- Fused blocks, the architecture can process multi-scale spatial and temporal features, im- proving video compression and reconstruction quality.
Vector Quantization: The quantization step allows for discretizing the latent space, which is critical for VQ- VAE models to perform learned compression, making this architecture suitable for generative tasks and compression with high efficiency.
MBQ- VAE is an efficient and scalable video processing architecture, leveraging mobile inverted convolutional blocks and vector quantization balance computational efficiency with high-quality reconstruction of video data. This makes it well-suited for video compression, generation, and reconstruction tasks.
B.1.3 MOBILE-INVERTED 3D CONV BLOCK
The 3D Mobile Inverted Convolution (3D-MB Conv) Block plays crucial role in VQ-VAE network by enabling efficient spatiotemporal feature extraction with reduced computational com- plexity. This block begins with 3D convolutional layer that maps from higher dimensional input space C" to the desired channel dimension C, utilizing a 3 x 3 x 3 kernel with padding to maintain spatial and temporal resolution. The feature maps are then expanded using a pointwise 3D convo- lutional block from C' to 4C, enhancing the feature representation without significantly increasing Squeeze-and-Excinn Excitatiio (SE) layer follows, which adaptively recalibrates the channel importance, improving the model's sensitivity to informative spatiotemporal features. Next, a depthwise separable 3D convolution applies 3 3 x 3 kernel to each feature map independently. maintaining 4C' channels and effectively reducing computation compared to standard convolutions. Finally, a 3D convolutional block reduces the channel dimension back from 4C' to C, ensuring that the block output is consistent with the input dimensionality. With its inverted residual connections and depthwise convolutions, this overall structure enhances efficiency in capturing hierarchical spa- tiotemporal patterns, making it highly suitable for the reconstruction and compression tasks in our VQ-VAR network.
computation.
1.4.4 INCEPTION-FUSED MB CONV BLOCK
The 3D Inception-Fused Fused-MB Conv Block a sophisticated convolutional module designed to effi ciently capture both spatial and temporal features in video data, commonly used in tasks like Video VQ-VAE (Vector Quantized Variational Autoencoder) shown Figure S.8. This block employs an Inception-style structure with three parallel 3D convolution paths, each with different kernel config- urations (e.g., 3 x 3 1, 3. x 3, and 1 x 3 x 3) to capture multi-scale spatiotemporal patterns in various dimensions. Each branch is followed by depthwise separable 3D-MobileBlock (MB- Conv) that reduces computational complexity while maintaining efficiency. The outputs of these paths are concatenated, allowing the network to fuse multi-scale features, which then processed
18

---

Published as a conference paper at ICLR 2025
by final 3D convolutional block. This design enables efficient handling of complex video repre- sentations in VQ-VAEs optimizing the model's ability to compress and reconstruct video sequences while preserving key spatial and temporal information. We perform temporal downsampling within the 3D-MBQ- VAE encoder at specific stages. Temporal downsampling is applied by a factor of at the 6th block (Downsampling Block) and again by factor of 2 the 9th block, resulting in total downsampling factor of 4 along the temporal dimension.
B.2
ADDITIONAL ABBLATION STUDIES FOR 3D-MBQ VAE
B.2.1 IMPACT OF VARYING CODEBOOK SIZES VIDEO RECONSTRUCTION
We conducted a comprehensive ablation study analyzing the effects of different codebook sizes, embedding dimensions, and quantization techniques video reconstruction quality. In default configuration, we employ the Lookup-Free Quantization (LFQ) method from MAGVIT-v2, utilizing an embedding size of d and codebook vocabulary size of C 12, 800.
Table 7: Impact of codebook size (C) on video reconstruction (d= Embedding Size)
COCO-Val
WebVid- Val
Method Of Quantization
C
Time (ms)
PSNR/ SSIM/ LPIPS
PSNR/ SSIM/ LPIPS
VQ
256
512
35
29.96/ 82 22.21/0.11.. 0.133
30.01/ 81 11.89/ O. 1.1411
Gumbal Quantization
256
512
23
29.83/ 81.98/ 0.140
29.78/ 80.56/ 0,1511
VQ
1024
256
47
30,26/ 82.54/ 0.130
30.72/ 81.09/ 0.1453
VQ
4096
256
56
30.72/ 83. 0.127
31.03/ 82.22/ 0.1331
VQ
8000
128
69
30.45/ 82. 0.144
30.98/ 83.36/ O. 0.1999
LFQ
8000
16
20
31.08/ 84.34/ 0.112
31.98/ 85.04/ O. 1103
LFQ(Default)
12800
22
31.22/ 84.78/ 0.092
32.09/ 85.78/ O. 1011
The primary reasons for adopting LFQ over traditional Vector Quantization (VQ) methods are:
1. Larger Vocabulary with Smaller Embedding Size: LFQ enables the use of a considerably larger codebook vocabulary while maintaining a compact embedding size. This approach enhances the model's expressive power without significantly increasing computational costs. By leveraging a larger vocabulary with smaller embeddings, the model benefits from higher codebook utilization, strategy that has been shown to improve performance Sun et al. (2024c) for the task of image reconstruction.
2. Improved Efficiency: LFQ is faster and more optimized than standard VQ methods, which is beneficial for processing high-resolution video data.
B.2.2 RESULTS WITH RANDOM MASKING AND FULL- MASKING
For random spatial masking, we employ a "cosine scheduling" strategy, progressively varying the masking ratio from 20% to 60% during training. This scheduling helps the model gradually adapt differrnt levels of partial observations, thereby enhancing robustness in spatial feature extraction.
For frame masking, we similarly utilize cosine scheduling, adjusting the masking ratio from 10% to 50%. This approach ensures a balanced training dynamic where the model exposed to vary- ing levels of temporal information occlusion, ultimately aiding in better temporal coherence and reconstruction quality. These masking strategies are designed to incrementally challenge the model, helping it learn to reconstruct meaningful content under different masking conditions and thus im- proving overall generalization.
Table 8 shows the results with random masking and full-frame masking. These results underscore the significance of both random masking and fully masked frames in enhancing the model's ability to predict frame indices effectively.
The random masking strategy compels the model to infer missing spatial information from partial observations within each frame, effectively enhancing its ability to capture spatial dependencies and structures. By being trained on randomly masked data, the model learns to reconstruct or pre- dict spatial details based contextual cues, leading stronger spatial representations. Conversely,
19

---

Published as a conference paper at ICLR 2025
Table 8: Ablation results with random masking and full-frame masking
COCO-Val Val
WebVid- Val
Methods
PSNR/ SSIM/ LPIPS
PSNR/ SSIM/ LPIPS
W/o Random Masking
29.17/ 81.92/ 0.111
28.78/ 82. 29/ 031334
W/o Full Frame Masking
30.09/ 82.22/ 0.102
30.11/ 83.01 O. 1.04
W/o Random Masing and Full frame Masking
28.82/ 80. 77/ 0.124
26.77/ 79.92/ 0.1552
With Random Masing and Full frame Masking
31.22/ 84.78/ 0.092
32.09/ 85.78/ O. 1011
supervision for fully masked frame index prediction requires the model to determine the correct tem- poral order of completely obscured frames, which enhances temporal consistency. This task forces the model to understand and model temporal dynamics across frames, it must rely on learned temporal patterns to accurately predict the positions of fully masked frames within the sequence.
B.2.3 VIDEO COMPRESSION RESULTS ON MCL-JCV DATASET
We conducted zero-shot inference on 30 videos from the MCL-JCV dataset (Wang et al. (2016)), resized to resolution of 640 x 360, which aligns with the experimental setup used by MAGVIT Yu al. (2023a) and MAGVIT-v2Yu al. (2023c). We evaluate compression quality using standard distortion metrics (LPIPS, PSNR, and SSIM) at bit rate of 0.0384 hpp (bits per pixel). The results are shown in Table At equivalent bit rates, traditional codecs such as HEVC Sullivan et al. (2012) and VCC Bross et al. (2021) may sometimes achieve finer local detail rendering compared to 3D-MBQ-VAN VAE. However, they often introduce block artifacts that, while detrimental to perceptual quality, not adequately captured by PSNR and SSIM. This is reflected in the LPIPS metric.
Table 9: Video compression results on MCL-JCV dataset
Methods
PSNR
LPIPS SSIM
HEVC
30.1
0.943
0.199
VCC
32.65
0.966
0.153
MAGVIT
23.7
0.846
0.144
MAGVIT-v2
26.18
0.894
0.104
3D-MBQ-VA VAE (Ours)
29.09
0.922
0.089
C DATASETS
Prompt Please describe the object in the video, including their shape, size, colour, texture, material, any distintinctive features
SKETCHY
QUICK DRAW
CLIP based label matching
Sn
labels
Video
bAVIS/OS
L1
SIL1
S2L1
SmLL
Frames
annotations
Video-
Best Matched Sketches
LLAVA-
Binary Masks
CAPTION
Figure S.9: For dataset curation for the downstream task of sketch- guided video inpainting, captions have been generated using Video-LLaVA-7B. and the corresponding sketches were obtained using CLIP-based label matching
20

---

Published as a conference paper at ICLR 2025
The following section discusses the datasets used for pretraining and training our proposed models. The dataset selection was based on the complexity and nature of each task. We also discuss the process of caption regeneration for the Web Vebbiid-0k dataset and the curation of new dataset for the task of Sketch-based Video Inpainting.
C.I DATASETS USED FOR 3D MBQ- VAE PRE-TRAINING
For pre-training our 3D MBQ- we use the YouTube 100M Hershey al. (2017) dataset. The large number of videos and the comprehensive pretraining help our 3D-MBQ-VAR -OVE learn an efficient latent representation of video frames reduced spatiotemporal domain.
C.2 T2V MODEL TRAINING
For training the diffusion model, we use the Webvid-10M Bain et al. (2022) with text the con- dition. find that the original text captions in the Webvid-10M dataset not provide adequate guidance for diffusion pretraining purposes (Table 3). Hence. re-caption the entire dataset with much more detailed prompts for every video. The corresponding text pairs given as conditions for generated by LLaVA Next- 34B Li et al. (2024). In both cases, had an 80-20 split between the training and test set. The following prompt was given to the LLM for regenerating the textual description of the video:
diffusion pretraining
Prompt
Explain in detail about the given video with:
Focus on the primary subject and any important actions or interactions.
Highlight key details, such as distinctive features, expressions, or objects.
Describe the setting and environment, noting any relevant background elements.
Convey the mood or atmosphere of the scene.
Combine these elements into a clear, concise caption that accurately represents the image's content and context.
C.3 PREPROCESSING TO DEAL WITH THE WATERMARKS IN THE WEBVID 10M DATASET
To address the issue of watermarks in the Web SobVid-100 10M dataset and ensure that our generated videos do not exhibit these artifacts, we implemented preprocessing strategy involving algorithmic in- painting methods. Specifically, tained watermarks. We then applied inpainting techniques to these masked areas to reconstruct the underlying content, effectively removing the watermarks while preserving the overall visual quality of the videos. Although this process may introduce minor visual artifacts, it prevents the model from learning and replicating watermark patterns during training. Consequently, our model is able to generate clean, watermark- videos.
identified and masked the regions in each video frame that
Furthermore, to enhance the resolution and temporal consistency of the preprocessed videos, employed the Sta Rota et al. (2024) video super-resolution model. This step refines visual details and mitigates any reduction in quality resulting from the inpainting process, ensuring that dataset used for training is of high quality and temporally consistent.
C.4 TRAINING FOR SKETCH GUIDED VIDEO INPAINTING TASK
For the downstream task of Sketch Guided Video Inpainting, we required a dataset containing the videos, text prompts, binary masks. and corresponding object sketches. To our knowledge, no such open-source dataset satisfies all our requirements. Hence, we curated our own dataset to our requirements. Figure S.9 shows the entire process of how curate the dataset for the downstream
21

---

Published as a conference paper at ICLR 2025
task using two datasets primarily used in the video inpainting research community: Youtube- VOS Xu et al. (2018) and DAVIS Perazzi et al. (2016).
D IMPLEMENTATION DETAILS
We now present the implementation details and parameters used in our study.
D.I HYPERPARAMETERS VARIOUS TECHNIQUES
To ensure fair comparison, we utilized the respective hyperparameters recommended in the orig- inal papers for each method. Table 10 outlines the specific hyperparameters used for training and inference across all baseline methods and our proposed approach.
Table 10: Hyperparameters of various techniques
Training H-Parameters
SimDA
AnimateDiff CogVideoX-5B
MotionAura
Learning Rate
1.00E-05
1.00E-04
1.00E-04
1.00E-05
Gradient Accumalation Steps
Batch Size Per GPU
Optimizer
AdamW
AdamW
AdamW
AdamW
Lr-Schedular
Linear
Cosine
Cosine
Cosine
Epochs
35
30
40
30
Noise Schedular
DDPM
DDPM
FlowMatching
VQDiffusionSSheedler
Diffusion Steps
500
500
100
30
Training Precision
Float16
Float16
BFloat16
BFloat16
GPUs
4 x 8 A100
x 8 A100
x 8 A100
6 8 A100
Text Encoders
T5-XL
CLIP
T5-XXL
T5-XXL
Time Embedding Size
256
256
512
512
Gradient Clipping
1.5
2.5
2.5
Max Text Length
77
128
200
256
Embedding Size
1024
1024
4096
4096
CFG Scale
8.5
10.5
10
Positional Encodings
Sinusoidal
Sinusoidal
RoPE
RoPE
D.2
IMPLEMENTATION DETAILS OF AD-MBQ-VAE PRE-TRAINING
We train our 3D MB-VAR model on the HouTube100M video dataset using four different configu- rations of frames and resolutions. These configurations are: (1) 32 frames at 256 x 256 resolution, (2) 16 frames 512 512 resolution, (3) 32 frames at 1280 720 resolution, and (4) 48 frames at 640 x 480 resolution. The corresponding batch sizes for these configurations are 8, 4, 2, and 4, with sampling ratios of 40%, 10%, 25% and 25% respectively. We apply gradient accumulation steps of 8 across all configurations.
The AdamW optimizer is employed with a base learning rate x 10-4 4 with cosine learning rate decay. To reduce the risk of numerical overflow, train the 3D MB-VAN model in fl flo.t32 precision. The training is performed across 4 nodes, each equipped with 8 NVIDIA A.00 GPUs (80 GB each), for total of 800,000 training steps.
D.3 IMPLEMENTATION DETAILS OF DIFFUSION PRETRAINING
We train our denoiser on the Webvid- 10M dataset for diffusion model pretraining. The AdamW optimizer learning rate of 10-5 with linear learning rate decay. During training,
22

---

Published as a conference paper at ICLR 2025
explore multiple frame settings: 16, 32, 48, and 64 frames. For 16-frame training, we use batch size of 512 and train for 120,000 iterations. For 32, 48, and 64 frames, the batch sizes are 256, 256, and 128, respectively, with corresponding training durations of 120,000 iterations for 32 frames, 150,000 iterations for 48 frames, and 190,000 iterations for 64 frames. This training is conducted on 8 nodes, each equipped with 8 NVIDIA A100 GPUs (80 GB memory per GPU). We utilize dynamic resolution and aspect ratio adjustments during training to enhance the model's robustness to varying input sizes.
D.4 IMPLEMENTATION DETAILS OF SKETCH-BASED INPAINTING TRAINING
For the downstream tasks, we fine-tune the model using LoRA (Low-Rank Adaptation) parameters specifically for the K and Q projection weights. The training is carried out with batch size of 8 per GPU at resolution of 256 512. This stage is performed on 2 nodes, each with 8 NVIDIA A100 GPUs (80 GB memory per GPU). The training includes 2,000 iterations for the VOS dataset and 2,600 iterations for the DAVIS dataset. We employ the Adam optimizer with an initial learning rate of 2 x 10-1, following cosine decay schedule. Gradient accumulation steps are set to 8 to manage memory and computational load, enabling efficient training through mixed sexed-precision techniques.
D.5 SETTINGS FOR JOINT TRAINING WITH VIDEO AND IMAGES
For joint training with video and images, we used the JDB Sun et al. (2024a) dataset, which is a lcrge-scale image dataset featuring around 4 million high-resolution images from Midjourney. We set N 2 16 for videos or N -1 for images to have a setup akin to W.A.L.T Gupta et al. (2025).
D.6 USE OF ROPE VS SINUSOIDAL, POSITIONAL EMBEDDINGS
The use of RoPE (Rotary Positional Embeddings) in the attention layers of the denoising network shows a faster convergence, seen in Figure S.10. RoPE (Rotary Positional Embeddings) is type of positional encoding designed for transformer models to encode relative position information flex- ibly. RoPE embeddings support larger context lengths, enabling the model generalize better over varying input sizes. Since they capture positional relationships based on rotation, the transformer can effectively handle and reason over sequences of greater lengths. This is handy when dealing with sigh--imensional video frames, thus helping the model learn effectively.
E COMPARATIVE RESULTS WITH DISCRETE AND CONTINUOUS SPACE
We now present motivation for transitioning the diffusion model paradigm from continuous space to discrete space.
1. Better Representation through Discrete Quantization: By moving to discrete latent space, we are able to leverage powerful vector quantization approaches, which help in capturing the essen- tial features of high-dimensional data more efficiently. The discrete representation obtained from quantization often yields better compression and a more structured representation of complex data, which helps the diffusion process operate over more compact and semantically meaningful latent space. This can lead to improved training stability and generation quality, especially for high- dimensional domains such as video.
2. Reduction in Computational Complexity: Traditional diffusion models applied in continuous space can require substantial computation, particularly during inference, as they operate directly on high-dimensional latent representations. By quantizing the data into discrete tokens, we can reduce the dimensionality and the complexity of the diffusion process. This makes the inference more efficient without compromising the model's expressiveness. Recent works, such MAGVIT Yu et al. (2023a), have shown that combining discrete latent space with the diffusion framework can yield efficient and high-quality results for complex generative tasks, such video synthesis.
3. Alignment with Discrete Decoders: Utilizing a discrete latent space allows us to align more effectively with certain discrete decoder architectures, such as transformers, which operate over token sequences. This leads to improved synergy between the encoder-decoder framework, where the encoder maps the input data into discrete tokens and the decoder operates over those tokens
23

---

Published as a conference paper at ICLR 2025
22.5
RoPE Sinusoldal
20.0
17.5
15.0
Loss
12.5
10.0
7.5
5.0
2.5
15K Steps
20K
25K
30K
Figure S.10: RoPE embeddings show faster convergence demonstrating superior learning compared Sinosoidal embeddings
to generate the output. The performance gains we observed were not solely due to the encoder being more performant in this setting, but also due the effective interaction between discrete representation and token-based decoders.
interpretable and often exhibit greater semantic richness, which can be beneficial for downstream generative tasks where controllability and interpretability
4. Enhanced Control and Semantic Richness: Discrete tokens
desirable. This is particularly advantageous for tasks like text- to-video generation or inpainting based methods., where mapping high-level semantics to the gener- ative model plays crucial role. Recent works, such Palette Saharia et (2022), VQ-Diffusion Gu et al. (2022), and Sketch-based Video Inpainting Sharma et al. (2024), have successfully em- ployed discrete latent spaces, demonstrating the effectiveness of this approach across a variety of generative tasks.
Note that Tiua is used to compute the probability the index corresponding to a completely masked frame. It functions Vi represents the masked frame. This loss term is crucial for ensuring that the model can accurately predict and reconstruct the masked frames by providing probabilistic grounding.
standard discriminative loss, specifically equivalent to log Pe ( (V)), where
Table 11 compares the results of discrete space with continuous space. Notice that the results of discrete space superior. Our findings further indicate significant improvement when Lmi loss applied in continuous latent space, demonstrating its effectiveness in enhancing the model's per- formance.
24

---

Published as a conference paper at ICLR 2025
Table 11: Comparison between discrete and continuous spaces
COCO- Val
WebVid -aa
Methods
PSNR/ SSIM/ LPIPS
PSNR/ SSIM/ LPIPS
Discrete space
31.22/ 84.78/ 0.092
32.09/ 85.78/ 0.108
Continous Space (w/o Lmni)
28.27/ 82.29/ 0,155
29.01/ 80.45/ O. 166
Continous Space
31.09/ 83.96/ 0.109
31.62/ 85.05/ 0.112
F SKETCH-GUIDED VIDEO GENERATION
F.1 CONDITION INJECTION FRAMEWORK
Given a masked video, the primary objective of our approach is to reconstruct the unmasked frames using the surrounding video context, guided by sketch conditions. The inpainting process begins by encoding the masked video into a compact latent space. This encoding step ensures the efficient capture of essential information about the video, facilitating robust modeling temporal and spatial relationships. Simultaneously. the sketch conditioning is encoded into complementary latent space, enabling the framework to effectively integrate external guidance.
During fine-tuning, the model is trained with combination of the masked video, its unmasked counterpart, and the corresponding sketch conditions. This multi-input paradigm allows the model to learn the nuances of filling masked regions in alignment with both the video context and the provided sketches. Our hypothesis is that incorporating external conditioning improves the model's ability to interpret masked regions, enabling precise and context-aware attention over these regions. This approach aims to refine the synthesis of missing areas while adhering to the desired object shapes and poses specified by the sketches.
To address the potential challenge of catastrophic forgetting during continual or lifelong learning, integrate a Low-Rank Adaptation (LoRA) module into the fine-tuning process. LoRA adapters are introduced as lightweight parameter-efficien layers, allowing the model to adapt to new tasks without full end-to-end fine-tuning. This decision is grounded in the need for scalability and the prevention of knowledge erosion from previously learned tasks. By incorporating LoRA adapters, we aim achieve adaptability comparable modern language models while maintaining robustness in learned representations.
The placement of LoRA adapters is determined using Elastic Weight Consolidation (EWC), tech- nique that identifies model layers with higher activation gradients and step gradients. By targeting these layers, the adapter network maximizes its effectiveness. ensuring that the critical components of the model's architecture are adapted with precision and efficiency.
During inference, the model utilizes the masked video and sketch conditioning as inputs. The en- coded representations are processed by generative network to predict and reconstruct missing regions. The generative network ensures temporal consistency across frames while aligning the syn- thesized content with the sketches, resulting in video outputs that maintain fidelity to the original content and the provided guidance.
In summary, while the condition injection framework may appear straightforward, our design in- corporates several nuanced techniques, including efficient latent space encoding, fine-tuning with LoRA adapters, and strategic application of EWC-based insights. These innovations collectively enhance the model's 's ability to perform sketch-guided video inpainting effectively and adaptively.
F.2 USE OF LORA ADAPTORS FOR PEFT
LoRA adaptors are beneficial in Parameter-Efffciee Fine-Tuning (PEFT) because they significantly reduce the number of parameters that must be trained while maintaining the model's performance. In large-scale models, fine-tuning all the parameters is computationally expensive. LoRA adaptors introduce trainable, low-rank matrices into the model layers, which allows only a small subset of parameters to be updated during training. This drastically reduces the memory and computational overhead required for fine-tuning, making it more feasible adapt large models to new tasks using limited data and resources. By optimizing only a small fraction the parameters, LoRA adaptors also help prevent overfitting and make the fine-tuning process more efficient without compromising
25

---

Published as a conference paper at ICLR 2025
model accuracy. As model is very large, thoroughly training it from scratch for every down- stream task does not make sense. Hence, we use the LoRA adaptors in our Spectral Transformer to adapt to the downstream task. Fig S.11 shows the details of our Spectral Transformer with the LoRA adaptors. Two types of LoRA have been used as shown Fig S.11. One is the Attention
@
Mortinn
Sheese FFT
Cross Amerton
AdN
(t.PPS)
Figure S.11: Use LoRA the above helps efficiently fine-tune above spectral transformer for the downstream task while only using fraction of the number of trainable parameters.
LoRA, which is applied in parallel to the Attention Blocks. This LoRA is applied only to the Key and the Query vectors. The reason for applying LoRA here is to ensure the base model retains infor- mation from previously trained tasks. LoRA is not applied the Value vector due Elastic Weight Compression. Elastic Weight Compression identifies the most critical parameters the model must remember learn a task effectively. Using Elastic Weight Compression, find that the parame- ters of the Key and Query are the most crucial when adapting to new task, which is why we apply LoRA them.
The second type of LoRA used is the Feed Forward LoRA, applied sequentially to the final Pointwise Feed Forward Layer. The Feed Forward Layer highly dense network, and if LoRA were used in parallel to this layer, followed by concatenation with its output, it would not result in meaningful representations. The sequential application ensures that the model learns effective and compact representations.
F.3 SAMPLE SKETCH-GUIDED VIDEO GENERATION RESULTS
Input Video
Frames
SimDA
AnimateDiff
CogVideoX-58 5B
MotionAura-L
Figure S.12: Sketch guided video generation results for dolphin sketch
Figures S.12 and S.13 show sample sketch- sketch-guided guided video generation results of various techniques. Given the inpainting video results, not hard to see that our model achieves the best results in terms of temporal consistency and preserving the inpainted object throughout the frames.
26

---

Published as a conference paper at ICLR 2025
Input Video
Frames
SimDA
AnimateDiff
CogVideoX-5B
MotionAura-L
Figure S.13: Sketch-guided video generation results for a frog sketch
G TEXT-GUIDED VIDEO GENERATION
G.1 ZERO-SHIOT RESULTS ON UCF-101 DATASET
To highlight the generalization capabilities of our model across diverse datasets, we present its per- formance on the UCF-101 dataset, summarized in Table 12. The results clearly demonstrate the superiority of MotionAura compared to prior works. Specifically, the table showcases the perfor- mance of various methods using different schedulers during inference, paired with their respective preferred CFG (Classifier-Freee Guidance) scales.
Notably, MotionAura-L achieves impressive results while requiring only 10 steps, making it signif- icantly faster than other baseline methods. This efficiency underscores the practical advantages of MotionAura-L in scenarios demanding high-speed inference without compromising performance.
Table 12: Zero-shot results on UCF-101 dataset
Methods
Scheduler
CFG Scale
Steps
FVD
SimDA
TultrAAncesrrDiisrreechhheuee
8.5
30
300
AnimateDiff
DPMSolverMuulsseeSShhcellee
8.0
30
277
Cog VideoX-SB hilerAAnessralDiserteeShhhhllllr
10.5
25
239
MotionAura-L VODiffusionSShhcduler
8.5
10
219
G.2 SAMPLE TEXT-GUIDED VIDEO GENERATION RESULTS
Figures S.I to S.7, and Figures S.14 to S.20 further demonstrate the robustness of our model on different tasks.
27

---

Published as a conference paper at ICLR 2025
Abstract Sculptures
Create an obstract vidso featuring fluid, metallic sculptures morphing end twisting in serene. white space. The circles oround the forms, copturing their smooth surfaces they shift between shopes. eefecctions play across their surfaces, creating mesmerizing, almost hypnotic effect.
Render video showcasing obstract sculptures made of geometric shopes fused with organic elements. camera moves through gallery. space, focusing sculpture's intricate details. appear to grow and change, as If they are alive, with soft lighting emphasizing their textures.
Generate video festuring abstract sculptures that resemble intricate crystal fornations, The slowly. nevealing the sharp angies and translucent surfaces of the sculptures: The light refracts through the crystais, creating a dazzling display of colors ard reflections against dark background.
Figure S.14: Abstract sculptures (Click here the video.)
28

---

Published as a conference paper at ICLR 2025
Cinematic Videos
Creats cinomatic video featuring sprawling, futurietic city dusk, Skyderoppre tower abeve, costing long shodows. camera slowly zooms out, revealing the bustling city streets below, neon lights and fost-moving traffic. A heavy atmosphere of anticipation fills the oir as dark clouds gather on the horizon.
Rerder is a cinematic vieloo of an spic battle madieval fantasy world, Armored knights clash with mythical foggy battlefield, The captures intense close swords clashing and panoramic views of the chootic scere. The sky overcest, with the sound of thunder rumbling the distonce.
pouning their expressione as they embrace, with raindrops glistening on their faces, Soft, warm lighting from the station's lamps contrasts with the cold, environment, highlighting the emotion
cinematic couple reuniting at
fecuses
Figure S.15: Cinematic styles (Click here to see the video.)
29

---

Published as a conference paper at ICLR 2025
Graphic Novel type Videos
Create graphic novel-style video dark, raln-socked city at night, camera follows trench- costed detactive as he walks down dimly lit alleyways. The scene is full of shara contrasts, with deep shodows ard bold highlights, capturing the gritty, noir atmosphere. Text bubbles appear brief.
graphic novel. style dramatic between two superheroas futuristic city. camera zoond on intende action sequences, with exogeretedd motion lines and dynomic anglest. Vibrant colorst and sharp outlines define the characters, while sound effects like "BAMI" and "CRASHII illustrated
Create graphic novel- style vides featuring group of edventurers exploring ancient, magical temple. The comera captures their journey through booby. trapped corridors and mystical chambers. The scene is filled with intricate, hand. drown details, vibrant colors, and dramatic lighting.
Figure S.16: Graphic novel styles (Click here to the video.)
30

---

Published as a conference paper at ICLR 2025
Low Polly Videos
Creats low poly 30 video featuring small, vibrant island survourded by crystal-clear water. The camera flies the island, revealing its simple, geometric trees, mountains, and small village. The scene has playful. sharp, edges detai), evoking whimsicel
etmesphere.
Model poly Viking village, festuring houses thatched reofa, central langhouae, and a wooden dock by the water. Include low-poly bosts flasting nearby. Keep the terrain flat and the structures simple, using besic polygens.
Greate low-poly 3D fantasy castie, featuring towers with cone-shoped roofs, and thick store walls. Keep the design simple, focusing on basic geometric shopes and colors.
Figure S.17: Low Polly results (Click here to see the video.)
31

---

Published as a conference paper at ICLR 2025
Sci-Fi Videos
Create dci-fi art video festuring a vest, olien landecape on distant planet. The camera slowly pans across towering, otherworldly rock formations, glowing allen flora, and filled with multipic moons and distant stors, The scene is bathed surreal colors, with mix natural.
rebelled againet creators
futurletic matropolia. Incorporate towering
skyscrapers.
An impanding alien invasion. Maseive clien matherships loom over major citiex, depleying droncs and extreterrestrial
Figure S.18: Sci-fi results (Click here to see the video.)
32

---

Published as a conference paper at ICLR 2025
Thriller Videos
Thriller style video set dimly lit unban alley at night. The camera follows lone figure walking quickly. conetantly lonking their shoulder. The tension builds with each the atmoephere becomes increasing'y clovetrophobic.
small town that starts receiving on unexplained broadcast signal. Incorporate visual glitches and zerie sound effects onhence the unsettling atmosphers.
dense forest where reality distorted, Use atmospheric visuols and discrienting sound design create sense fear and confusion as the environment manipulates their senses.
Figure S.19: Thriller results (Click here to see the video.)
33

---

Published as a conference paper at ICLR 2025
Macro Style
Render a mocro style video facusing morning dew draplcts vibrant green leaf. The camera moves siowly, copturing the crystal-clent clear they reflect the surrounding
patterns of the lesf's veins are highlighted, with the sunlight causing the droplets to sparide like tiny jewels.
Create style video that zooms in on the intricate details an ant colony at work The camera reptires exctneme clase-upe of the ants ne they earry foed end interart with their environment The teatunee of their exoskeletons, the hairs their bodies, grains of beneath detailed
vividly
paced mantage vehicles, and interconnected skybridges. The video should feature shorp contrasts between shadows and vibrant lights, with seamless transitions that reflect the sleekness odvanced technology.
metropolis. Incorponste holagraphic odvertiscments, flying
Figure S.20: Macro style results (Click here see the video.)
34


Reviews:
[{"review_id": "kg01oOlXNU", "reviewer": "Reviewer_CyQT", "summary": "This paper presents a novel text-to-video framework, providing comprehensive and reasonable technical details from VAE to model architecture, as well as downstream applications. The paper proposes an effective VQ-VAE training strategy, achieving state-of-the-art compression and reconstruction results, which can be utilized by existing generative models. It also introduces an efficient spectral transformer that encodes information into the frequency domain for processing. Furthermore, the design concepts for downstream applications are illustrated within the context of the sketch-guided video inpainting task.", "questions": "- Are there any evaluation or ablation study results available regarding Randomly Masking and Fully Mask Frame Index Predicting?\n- Will the model or dataset be made open-source?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper constructs a complete and efficient new framework for text-to-video generation, featuring innovative modules supported by convincing quantitative evaluations and generative results. Specific strengths include:\n\n- For the VQ-VAE, the paper introduces two novel optimization strategies: *random masking* and *Masked Frame Index Loss*, which allow the VAE to learn spatial relationships and temporal consistency. The compression and reconstruction quality reaches state-of-the-art levels, and the visualization results demonstrate strong feature extraction and representation learning capabilities. These strategies could serve as foundational components for future models.\n- A diffusion framework for discrete space is proposed, which encodes information such as videos into the frequency domain for computation, featuring an elegantly designed spectral transformer structure. After training, this framework achieves satisfactory results in terms of generation speed and quality.\n- Based on the proposed generative framework, the design of the sketch-guided video inpainting task provides a proof of concept for the framework's scalability and efficient transfer applications. According to the paper, this also marks the first introduction of sketch-guided generation for videos.\n- The constructed dataset, if made open-source, would be a significant contribution to the community.", "weaknesses": "While the paper is rich in content, there are some potential issues:\n\n- The two supervisory strategies designed for the VAE elevate its capabilities to state-of-the-art levels. Although the paper states that the random masking strategy enables the model to learn spatial information and that supervision for the fully masked frame index prediction enhances temporal consistency, it seems there are no related ablation studies provided to explore the specific reasons for model improvement further.\n- The condition injection framework designed for the sketch-guided video inpainting task appears relatively straightforward, lacking innovative design at the technical level.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xhDvvppNNT", "reviewer": "Reviewer_UAh7", "summary": "This paper introduces an innovative set of methods designed to enhance video generation. These include the 3D-MBQ-VAE, which improves video compression, a novel text-to-video generation framework, a spectral transformer-based denoising network for superior video generation, and a Sketch Guided Video Inpainting task that utilizes Low-Rank Adaptation (LoRA) for efficient fine-tuning.", "questions": "1.Based on my understanding, the videos in the WebVid10M dataset all contain watermarks. I'm curious as to how the videos generated by the model, which was trained on this dataset, do not have any watermarks. Can the authors explain this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.The authors propose a denoising network named Spectral Transformer that processes video latents in the frequency domain using the Fourier Transform, capturing global context and long-range dependencies effectively.\n\n2.The paper is the first to address the downstream task of sketch-guided video inpainting, using LORA adaptors for parameter-efficient fine-tuning of the denoising network.", "weaknesses": "1.3D-MBQ-VAE Concerns:\n\n(1)W.A.L.T[1] has demonstrated that using a 3D casual VAE can allow for joint training with both images and videos in text-to-video models, significantly improving performance compared to training solely with videos. Can the authors discuss whether the proposed 3D-MBQ-VAE can also support such joint image and video training? If feasible, it would be beneficial to see an experiment demonstrating this capability and comparing it with that of W.A.L.T.\n\n(2) In the comparative experiments listed in Table 1, could the authors provide comparative results of MAGVIT-v2[2] in regard to Video Compression Metrics or Video Reconstruction tasks?\n\n (3) Codebook (Vocabulary) Size Influence: As far as I understand, the size of the codebook could considerably affect the 3D-MBQ-VAE's reconstruction ability. However, the paper doesn't seem to mention any information relating to the codebook size. Could the authors provide additional details on the impact of varying codebook sizes on video reconstruction?\n\n2.Experiment Setting for Text-to-Video Model: In Table 3, the authors used WebVID10M as the training data and also computed FVD and other metrics on WebVID10M for testing. However, the comparative methods did not use WebVID10M in their training data, which suggests that other text-to-video methods computed FVD and other metrics in a zero-shot manner. In addition, specifics like the sampling schedule, sampling steps, and classifier-free guidance scale used by the authors' proposed method and the comparison methods were not disclosed. To ensure a fair comparison, could the authors evaluate all models (including theirs) in a zero-shot manner on a common test set, like FVD of UCF-101? Also, providing a table or an appendix detailing all the hyperparameters and settings for each method would greatly enhance reproducibility and fairness in comparison.\n\n------------\n\n[1].Gupta, Agrim, et al. \"Photorealistic video generation with diffusion models.\" arXiv preprint arXiv:2312.06662 (2023).\n\n[2].Yu, Lijun, et al. \"Language Model Beats Diffusion--Tokenizer is Key to Visual Generation.\" arXiv preprint arXiv:2310.05737 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ShxpQeeKay", "reviewer": "Reviewer_htH2", "summary": "The authors propose MotionAura, a novel text-to-video generation model. The authors begin with the 3D Mobile Inverted VQVAE to enhance spatiotemporal video compression. The discrete diffusion process is also carried out in the spectral domain rather than the traditional pixel or latent space. The model exhibits SOTA compression results and promiosing results on T2V and downstream tasks.", "questions": "- Traditional DMs (e.g., DDPM, EDM, LDMs) works perfectly fine with the continuous space. Could the authors explain why they see the need of going to the discrete space and transferring the diffusion paradigm to the discrete space? Is it only due to the encoder being more performant in this setting?\n- Could the authors explain the $\\mathcal{L}_\\text{MFI}$ loss which is indicated as *Index of completely masked frame* in Figure 2? Is this supposed to be some auxillary discrimator loss that detects whether the frame is masked or not?\n- If the former is true, an ablation experiment that include $\\mathcal{L}_\\text{MFI}$ in the continuous space would be convincing\n- Can you provide more ablation in how $\\mathcal{L}_\\text{MFI}$ helps? What is the masking ratio of random masking and full frame masking?\n- In Table 1, you show a Frame Compression Rate of 4 but I can't find in the paper where you talk how you downsample temporally and seem to be inconsistent with Figure S.8. A little bit of explanation would be nice\n- It's excited to see that Fourier transform also works with MSA and can further boost the performance of transformer. Since the self-attention and cross attention are working in a totally separate domain, it would be very interesting to show the activations of the attention in the spectral domain to gain more insight.\n- Another question with regard to FFT: In the cross attention part $Q$ is calculated from the image tokens and $K$, $V$ are calculated from the text tokens. Why would FFT still make sense in those domain since neither $Q$ or $K$, $V$ constitutes a continuous image domain? How do you make sure that FFT works on the discrete image tokens and why would FFT even make sense on the text tokens?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The spectral transformer is novel and works better than its counterpart transformer, as evidenced by the ablation study.\n- The author proposed pretraining the 3D-MBQ-VAE using random and complete masking and get better results than the counterpart compression models.\n- The authors show very pleasing, temporally consistent video samples that showcases the effectiveness of the proposed model", "weaknesses": "- Some of the ablation experiments are missing e.g., in the 3D-MBQ-VAE with regard to the different losses\n- It is generally not clear why the authors would not prefer a continuous space rather than opting for the discrete space\n- Some of the training details is still missing, e.g., what is the masking ratio of random masking and full frame masking in 3D-MBQ-VAE?\n- The spectral transformer part of the paper can be augmented with more analysis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)