MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation
Authors: Trung X. Pham
Tri Ton
Chang D. Yoo
Venue: ICLR 2025 Poster (2025)
Keywords: vision-guided audio generation
fast inference
open-domain sound synthesis
masked diffusion models
temporal learning
visual sound source localization
generative AI
Primary area: generative models
TL;DR: A novel approach is presented for highly efficient vision-guided sound synthesis using masked diffusion models

Abstract: We introduce MDSGen, a novel framework for vision-guided open-domain sound generation optimized for model parameter size, memory consumption, and inference speed. This framework incorporates two key innovations: (1) a redundant video feature removal module that filters out unnecessary visual information, and (2) a temporal-aware masking strategy that leverages temporal context for enhanced audio generation accuracy. In contrast to existing resource-heavy Unet-based models, MDSGen employs denoising masked diffusion transformers,  facilitating efficient generation without reliance on pre-trained diffusion models. Evaluated on the benchmark VGGSound dataset, our smallest model (5M parameters) achieves 97.9% alignment accuracy, using 172x fewer parameters, 371% less memory, and offering 36x faster inference than the current 860M-parameter state-of-the-art model (93.9% accuracy). The larger model (131M parameters) reaches nearly 99% accuracy while requiring 6.5x fewer parameters. These results highlight the scalability and effectiveness of our approach. The code is available at https://bit.ly/mdsgen.

Published as a conference paper at ICLR 2025
MDSGEN: FAST AND EFFICIENT MASKED DIFFU- SION TEMPORAL-AWARE TRANSFORMERS FOR OPEN- DOMAIN SOUND GENERATION
Trung X. Pham', Tri Ton', Chang D. Yoo! Korea Advanced Institute of Science and Technology (KAIST) ftrungpx, tritth, cd yoo)@kaist. ac. kr
ABSTRACT
We introduce MDSGen, a novel framework for vision-guided open-domain sound generation optimized for model parameter size, memory consumption, and in- ference speed. This framework incorporates two key innovations: (1) a redun- dant video feature removal module that filters out unnecessary visual informa- tion, and (2) a temporal-aware masking strategy that leverages temporal context for enhanced audio generation accuracy. In contrast existing resource-heat heavy Unet-based models, MDSGen employs denoising masked diffusion transformers, facilitating efficient generation without reliance on pre-trained diffusion models. Evaluated on the benchmark VGGSound dataset, our smallest model (5M param- eters) achieves 97.9% alignment accuracy, using 172x fewer parameters, 371 less memory, and offering 36x faster inference than the current 860M- parameter state-of-the-a model (93.9% accuracy). The larger model (131M parameters) reaches nearly 99% accuracy while requiring 6.5x fewer parameters. These results highlight the scalability and effectiveness our approach. The code available at https: //tit//tttt/////// 1y mdsgen.
INTRODUCTION
Vision-guided audio generation has gained sig- nificant attention due to its crucial role in Fo- ley sound synthesis for the video and film pro- duction industry (Ament, 2014). This paper fo-
Ours SM
Fill-oley (ncuripe23)
FoleyCrafter (arxiv24)
Algnment Accuracy (%)
90
cuses on Video-to-Audio (V2A) generation, key task not only for adding realistic sound to silent videos created by emerging text-to-video models (Blattmann et al., 2023; Khachatryan et al., 2023; Huang et al., 2024; Ouyang et al., 2024) but also for enhancing practical applica- tions in professional video production. Sound generation is essential for creating immersive ex- periences and achieving seamless audio-visual synchronization. However, achieving both se- mantic alignment and temporal synchronization in V2A remains a significant challenge. Pre- vious approaches, such as GAN-based meth- ods (Chen al., 2020b) and Transformer-baser autoregressive models (lashin & Rahtu, 2021),
80
Im2Wav
(ICASSP23)
(cvpr24)
SpecVDGAN (bmvc21)
50
200
400
600
800
1000
1200
1400
Number of Parameters (Millions)
Figure 1: Aligment Score. Comparison with SOTA audio generation methods on the VG- GSound test set. The diameter of each circle repre- sents the memory usage during inference.
have struggled with synchronizing audio to content while maintaining relevance. Diff-Foley (Luo al., 2023) improved this by employing contrastive learning for video-audio alignment and leverag- ing diffusion models, achieving impressive sound quality. Other methods like See and Hear (Xing et al., 2024) and FoleyCrater (Zhang al., 2024) utilize large pre-trained models for high-quality
"Equal Contribution
1 "Corresonndigg Author

---

Published as a conference paper at ICLR 2025
audio generation. However, these models rely hundreds of millions of parameters. In contrast, our work demonstrates that a much smaller model can deliver high performance (see Fig. 1). Most existing approaches rely Unet architectures, which present scalability limitations. Additionally, current methods often use video features that include redundant information.
In contrast, we propose MDSGen, novel framework for open-domain sound synthesis based pure Transformer architecture. MDSGen incorporates a temporal-ave masking scheme and a redundant feature removal module, enabling achieve superior performance while being significantly more efficient. Further analysis highlights the effectiveness of our approach with compelling evidence of its advantages. Our contributions are as follows:
We introduce simple, lightweight, and efficient framework for open- ppen-domain sound genera- tion using masked diffusion transformers, delivering high performance.
Our approach implements Vemporal-Awarence Masking (TAM), specifically designed for audio modality, in contrast spatial-aware masking of the existing work, leading more effective learning.
We identify inefficiencies in the existing approach that fail to remove redundant video features. Our Reducer module learns to selectively resolve these redundancies, producing more refined features for improved audio generation.
We validate our method on the benchmark datasets VGGSound and Flickr-SoundNet, surpassing state-of-the-ar approaches across multiple metrics, with particularly significant improvements in alignment accuracy and efficiency, specifically in model parameters, memory consumption, and inference speed.
2
RELATED WORKS
2.1 OPEN-DOMAIN SOUND GENERATION
Auto-regressive Transformer-based Approach. Key works in this area include Spec (lashin & Rahtu, 2021), which uses a cross-modas Transformer to generate sounds from video tokens auto-regressively, and Im2Wav (Sheffer & Adi, 2023), which conditions audio token generation on CLIP features. However, these methods suffer from slow inference speeds due to their sequential generation process and limited vision-audio alignment, negatively impacting performance.
Diffusion-based Approach. To overcome these limitations, Diff-Foley (Luo et al., 2023) introduced a two-stage method that enhances semantic and temporal alignment via contrastive pre-training on aligned video-audio pairs, followed by latent diffusion for improved inference efficiency. Similarly, See and Hear (Xing et al., 2024) utilizes ImageBind (Girdhar et al., 2023) and AudioLDM (Liu et al., 2023) for various audio tasks, while FoleyCrafter (Zhang et al., 2024) combines pre-trained text- to-audio model with a ControlNet-stylee module (Zhang et al., 2023) for high-quality, synchronized Foley generation. Although these diffusion approaches show promise, they often rely on large models with hundreds of millions of parameters and predominantly utilize U-Net architectures, leaving architectures largely untapped. Our proposed method leverages
the potential of transformer-basse diffusion transformers (Peebles & Xie, 2023) and masking techniques for efficient learning. It also addresses the issue of redundant video features in Diff-Foley (Luo et al., 2023), which hinders further improvements in audio generation.
2.2 LATENT MASKED DIFFUSION TRANSFORMERS
The Denoising Diffusion Transformer (DiT) introduced by Peebles & Xie (2023) replaces the traditional U-Net with a fully transformer- based architecture for latent diffusion, demonstrating remarkable performance in large-scale image generation on ImageNet. Following this, Gao et al. (2023) proposed the Masked Diffusion Transformer (MDT), which enhances ImageNet generation through spatial context-aware masking. Inspired by MDT, Pham et al. (2024) developed X-MDPT, using cross-view masking to establish correspondence between pose and reference images for improved person image generation. Additionally, MDT-A2G (Mao et al., 2024) explored masked diffusion transformers for gesture generation, while QA-MDT (Li et al., 2024) adapted this technique for music generation. In contrast to these works, we focus on lightweight masked diffusion models
2

---

Published as a conference paper at ICLR 2025
Reconstrccted Lateet Audio
WAE Enc
Masked Temporal Aware Network
(x)
morking
Preditor bleck)
Latent Audio
Added nohee Patchily
Visio Extracerr
Unmasied
masked
VAE
Condecse
Choreel Selectie bervenal
h. Amertion
Astentinn Rickks
Bocks
Reducer
Slient Video (V)
Denotsing Diffusion Transforeer
Generated audio
Figure 2: Overview of the proposed highly-efficient MDSGen framework, utilizing denoising masked diffusion transformers to efficiently learn video-conditional distributions for audio generation, replacing traditional Unet-based methods. The fire icon denotes trainable modules, and the locked icon denotes frozen ones. Green arrows denote branches used only during training, blue arrows are for only inference, and black arrows + are used in both training and inference.
for andeo-guided audio generation, introducing temporal-aware masking for audio and design that removes redundant video features to enhance generation effectiveness.
2.3 MASKED DATA IN THE AUDIO MODALITY
Several works have explored masking techniques for audio processing. MaskVAT (Pascual et al., 2024) introduces V2A system that integrates full-band audio codec, masked generative modeling, and multi-modal features to enhance audio quality, semantic alignment, and temporal synchronization. SoundStorm (Borsos et al., 2023) employs the non-autoregressive MaskGIT (Chang et al., 2022) approach for efficient text-to-audio generation. Similarly, VamNet (Garcia et al., 2023) applies MaskGIT music generation, while (Bai et al., 2022) uses masking the pixel space of mel- spectrograms for non-diffusion-basee text-to-audio tasks. These approaches differ fundamentally from our diffusion-base framework, which applies masking the latent space with added Gaussian noise. AudioMAE (Huang et al., 2022) and SpecAugment (Park et al., 2019) share similarities with our method in employing masking for audio data. However, key distinctions exist: both focus on masking in the pixel space of clean mel-spectrograms for representation learning in downstream recognition tasks. contrast, our approach utilizes masking in the latent space of a VAE within the diffusion framework, targeting audio generation.
METHOD
We aim to develop a simple yet effective framework for vision-guided sound generation using trans- formers, addressing the limitations of existing approaches that rely on traditional U-Net architectures, which are less scalable and efficient. Our framework, illustrated in Fig. 2, consists of a novel Vision Extractor with learnable Reducer that captures essential information from video input to generate a concise conditional output for the denoising diffusion process. Next, a Denoising Diffusion Transformer maps Gaussian noise sound distributions using extracted visual features. We also introduce a Masked Temporal-Aware Network (MTANet) for regularization, boosting performance. Finally, channel selection for mel-spectrograms, which enhances results with image VAEs, is optional.
3.1 DENOISING DIFFUSION TRANSFORMER
Our method supports both audio- and image-based VAEs. For instance, we describe using an image VAE (Luo et al., 2023; Chen et al., 2024), and for audio VAEs like AudioLDM, we adjust the three channels to one. We adopt the DiT backbone introduced by Peebles & Xie (2023) for denoising diffusion training. Given an audio signal A E R.A of length LA and a silent video V R 22x224x of length Lv, the audio first transformed into mel-spectrogram X R 122x5112 while the video is encoded into v E hv x512 and further reduced to ø e 1xD The mel-speetrogram is repeated across 3 channels, forming X' x3x188x512 and passed through the
3

---

Published as a conference paper at ICLR 2025
VAE from Stable Diffusion (Rombach et al., 2022; Luo al., 2023) to obtain a latent embedding x E R 16x64 This latent representation is patched and tokenized into image tokens using a patch size of (DiT's default), resulting in x' E R256x D. where Lx 256 and D 768 for the B-size model. These tokens are then fed into the N = N₁ + N₂ self-attention layers of the Transformer to predict the noise c added the latent x. Conditioned on the video encoding i, the Transformer model & learns the distribution Po (x (J). During training, Gaussian noise e E EV I) is added to the latent x to generate X at timestep & ['. The overall training objective is:
_ Ex, Ex ö, t)1 + x,,,,, (x)M)(xx)) U, t)1..
(1)
Here, À is the balance factor between the standard denoising diffusion loss (the first term in Eq. 1) and the masking loss (the second term), with À = 1.0 for optimal performance. The masking function Ma. which includes the MTANet introduced later, applies temporal-aware masking. During inference, given a silent video, the model starts from Gaussian noise (no audio provided), and the predicted latent & R 16x44 is iteratively denoised and decoded by the VAE decoder recover the mel-spectrogram X RGB E [g3x128x5122 Channel selection refines this into X e R 12826112 and the final waveform is reconstructed from the mpe-steetto using the Griffin-Lin (Griffin & Lim, 1984), or neural Hifi- MANN vocoder Kong et al. (2020).
3.2 VISION EXTRACTOR
The second key component of our framework is the Vision Extractor with learnable Reducer network that aligns video features with audio while condensing temporal information. We leverage the pre-trained CAVP model from (Luo et al., 2023), which was trained on Audioset using contrastive loss to extract video features aligned with audio. However, we identified that the CAVP features contain redundancies that could negatively impact generation quality. Diff-Foley (Luo al., 2023) linearly maps original feature dimensions from v E 4 x512 to v E RLy 768 7 and retains this full dimensionality during the diffusion process via cross-attention, with Lv is the video feature length. Our approach reduces the dimensionality to 전 E IR 1668, offering more concise and efficient information for denoising diffusion. Specifically, we project the encoded features Ly x 512 through multi-layer perceptron (MLP) into the transformer feature space (Lv x 768 for the size B-model). These features are then passed through a reducer module, an 1 x 1 convolutional layer, which condenses the high-dimensional features into a lightweight form ग E 1 This compact repre- sentation is integrated into the denoising diffusion process through Adaptive LayerNorm (AdaLN) modulation. Our method minimizes redundant features that could lead to overfitting, as shown in the train/test alignment accuracy gap Appendix Sec. A.2. Our analysis shows that the Ly-frame input features share over 90 % similarity, indicating considerable redundancies.
Intuition. Our simple yet effective reducer design treats the temporal dimension of video (Ly = 33) as feature channels and performs non-linear projection to a single channel, functioning similarly to channel attention by weighting important channels and summing them. This acts as bottleneck that distills and distributes video temporal information across the 768 dimensions, aligning better with each audio token, which also has a 768 (68-dimensional space. This approach, combined with the transformer network, significantly improves alignment accuracy up to approximately 99%.
3.3 AUDIO MASKED TEMPORAL-AWER AWARE NETWORK
Thirdly, introduce novel technique that ex- ploits the sound information's 's natural character- istic: the temporal sense. The existing masking, Opatial-Award Mask (SAM) proposed by MDT (Gao et al., 2023) is designed for image data to learn the spatial context within the image. But here, in the audio data (represented by mel- spectrogram with 2D data), the SAM masking method yields a sub-optimal solution because it cannot model the exact nature of temporal mean- ing in the audio data. To overcome this limita- tion, propose the Temporal-Aware Mask (TAM) strategy instead of SAM, which tries to mask the whole set of tokens along the temporal
4 4
frequency
f
frequency
time
time
a) Noisy Latent Audio
b) Spatial-Awar Mask
frequency
t, 1
frequency
5
time
time
c) Frequency-Aware Mask Figure 3: Audio Masking Strategies. Here, the red square red-square is the learnable mask token.
d) Temporal-Aware Mask
4

---

Published as a conference paper at ICLR 2025
dimension. As shown in the ablation section, this novel masking helps significantly boost perfor- mance in all metrics compared to the existing method SAM specified for image data as shown in Fig. 3. Interestingly, despite this simple strategy. it can help the denoising transformer models learn to generate audio much better than random masks as used in existing MDT designed for image data.
During training, mask with 7/m % of temporal tokens, we feed only the visible tokens to the Ni blocks of the transformer: 0₁ = Encoder N. (x' @ (1 - m)) (m is a mask matrix and @ denotes wlsment-wise multiplication) and in the MTANet we concatenate the resulting tokens with the learnable mask tokens M and feed into a single block of predictor (referred to as side-interpolator in MDT) achieve the full latent tokens before feeding into the final N₂ self-attention blocks of the transformer: o₂ = Decoder
MM(M * x' 3 (m)))). Here we find that different from the design for ImageNet with N₂2 = the default MDT, we use N2 = which gives a better performance for audio data. After training, the masked modeling branch is discarded, maintaining only positional embedding for inference.
3.4 CLASSIFIER- GUIDANCE
We adopt the dynamic Classifier-Free Guidance (CFG) method from previous works on masked diffusion transformer models (Gao et al., 2023) used in ImageNet. However, unlike image tasks, we find that in audio generation, the optimal CFG value is between 5 and 6, with a power scale of 0.01. Notably,
(CG) has been shown to significantly boost performance in Diff-Foley (Luo et al., 2023), where their method relies heavily on CG for optimal results. In contrast, our approach without CG surpasses Diff-Foley (CFG+CG) across multiple metrics. While incorporating CG improves our framework in terms of alignment accuracy and KI, it does not enhance other metries. Hence, for simplicity, we omit CG in most of our experiments.
3.5 VAE CHOICE FOR MEL- MEL-SSPECTRORRMM
Our method supports both Audio- and Image-trained VAEs. Ablation finds that audio quality varies across the RGB output channels of image-trained VAEs. Since the mel-spectrogram is 2D, we duplicate it into three channels for Stable Diffusion VAE. At the decoding stage, the VAE outputs three channels: X RRG E IR 3 128x528 with X RR :; E 1228x112 i E (0, 1,2) representing the R. G. and B channels. Diff-Foley (Luo al., 2023) used the R channel as output, but empirical tests consistently show that the G channel performs better. However, when using the audio VAE, i.e. AudioLDM VAE, channel selection is no longer required.
EXPERIMENTS
4.1 DATASET AND EVALUATION METRICS
(i) Dataset. We evaluate our method the VGGSound dataset (Chen et al., 2020a), using the original train/test splits with 175k and 15k samples, respectively, and on the Flick-SoundNet dataset Aytar et al. (2016) with 5k test samples. (ii) Metries. We first use the same metrics as prior work (Luo et al., 2023), including FID, IS. KI, and Alignment Accuracy, using their provided scripts for Align. Acc. and Spec code for FID, IS, and KL. Second, we assess general vision-audio alignment in the Image2Audio task using CloU and AUC metrics with scripts from (Mo & Morgado, 2022). Third, we compare efficiency using parameter count, memory usage, and inference speed. Lastly, we provide the FAD scores and MOS results from human evaluations in the Appendix A.
4.2 IMPLEMENTATION DETAILS
All models are trained and tested on a single A100 GPU (80GB) with a batch size of 64 and learning rate of 5e-4, using the Adan optimizer (Xie et al., 2024) for faster training. Unlike MDTv2 (Gao et al., 2023), we skip the macro-style of side interpolator design, as it was ineffective for our task, and instead use a simple self-attention block at decoder layer 4. Video-audio pairs are truncated to 8.2 seconds before encoding, following (Luo et al., 2023). Our model comes in three main variants: Tiny (5M). Small (33M), and Base (13IM), with the Large (460M) variant showing overfitting. We primarily focus the T, S, and B models.
5

---

Published as a conference paper at ICLR 2025
4.3
MAIN RESULTS
A. VGGSound dataset. Compared to state-of-the-an approaches, our method significantly outper- forms all competitors' alignment aceuracy while being far more efficient regarding parameters and inference speed (Tab. 1). Alignment accuracy, a metric introduced by (Luo et al., 2023), assesses synchronization and audio-visual relevance using separate classifier trained to predict real audio- visual pairs. Remarkably, our Transformer-
model, MDSGGn- Tiny (5M), trained from scratch, achieves 97.9% accuracy, surpassing the second-bes. Diff-Foley (860M), which is 172x larger and depends backbone of Stable Diffusion pre-trained on billion image-text pairs.
As shown in Tab. 1, Diff-Foley struggles without pre-trained backbone, with a significant drop to FID 16.98 and IS 24.91. In contrast, our smallest model, MDSGen-T (5M), trained from scratch, achieves FID 14.18 and 37,51, emphasizing the overfitting issues of heavy U-Net-based models compared to our lightweight Transformers. Our larger model, MDSGen-B (131M), achieves state-of- the-art alignment accuracy (zs 99%) and an IS of 57.12 at 800k steps, though longer training leads to overfitting and declines in other metrics.
Table 1:
Benchmark on VGGSound test. Generation quality comparison of different approaches. t gets from (Luo et al., 2023), & denotes without pre trained SDv1.4. * denotes results with pre- tre-trained SDv1.4, 14, we reproduce it using the public checkpoint. "("" in FAD denotes AudioVAI (Image VA...
Method
FAD
FID1
IS
Kl.|.
Align. Acc.t
Time; (8)
#Params|
Cost,
SpeeVQGAN (lashin Rahtu, 2021)1
9.70
30.80
7.03
49.19
5.47
308M
61x
Im2Wav (Sheffer & Adi, 2023)
11.44
39.30
$.20
67.40
6.41
448M
90x
Diff-Foley (Luo et 2023) 2023) 93
16.98
24.91
6.05
92.61
0.38
860M
172x
Dift-Foley (Luo al., 2023)
4.71
10.55
56.67
6.49
93.92
0.36
860M
172x
See and Hear (Xing et al., 2024)
5.55
21.35
19.23
6.94
58.14
18.25
1099M
220x
FoleyCrafter (Zhang et al., 2024)
2.45
12.07
42.06
5.67
83.54
2.96
1252M
250x
MDSGen-T (Ours) 500k
2.21 (3.69)
14.18
37.51
6.25
97.91
0.01
5M
1.0x
MDSGen-S (Ours) 500k
1.66 (2.75)
12.92
44.38
6.29
98.32
0,02
33M
6.6x
MDSGen-B (Ours) 500k
(2.16)
11.19
52.77
6.27
98.55
0.05
131M
26.2x
MDSGen-B (Ours) S00k
12.29
57.12
6.43
91.62
0.05
131M
26.2x
B. Flickr SoundNet dataset. We use the mod- els trained on VGGSound to test on Sound- Net dataset to evaluate its generalization. First, through quantitative metrics in the sound source localization task with Flickr-SoundNe (Aytar et al., 2016) test set. Second, qualitatively com- pare the generated audio across different meth- ods. As shown in Tab. 2, our method out- performs other methods on the CloU metric (82.01%) %) closer to the ground truth (83.94%) while the AUC remains comparable (around 55.5%). It shows that our generated audio provided better-aligned features with the visual information to localize the sound source. Diff-Foley performs worst, indicating that it is more overfitting. We provide their visualizations in the Appendix A.
Table 2: Benchmark on Flick-SoundNet test.
Comparison of different approaches. 'Bold' and "underline" denote the best and second-best, re- spectively.
Method
CloUt
AUC1
Diff-Foley (luo et al., 2023)
81.02
55.19
See and Hear (Xing al., 2024)
81.20
55.45
FoleyCrafter (Zhang et al., 2024)
81.78
55.57
MDSGen-B (Ours) 500k
82.01
55.51
Ground Truth
83.94
63.60
5 ABLATION STUDY
We attribute the strong performance of our models to three key factors. First, the Transformer backbone enables more effective learning of the audio modality compared existing Unct-based diffusion methods (Diff-Foley See and Hear, FoleyCrafter). Second, our innovative Reducer module mitigates potential redundancies in the video input. Third, the temporal masking model acts as a robust regularizer, further enhancing the Transformer's performance. A detailed analysis of these components is provided in the following sections.
5.1 ALIGNMENT ACCURACY: A CONFIDENCE SCORE PERSPECTIVE
We assess the enhancement of aligmment accuracy in our method by analyzing confidence scores from the VGGSound test set, using the output of the sigmoid function from the trained classifier

---

Published as a conference paper at ICLR 2025
12000
FoleyCrafter
12000
Diff Foley
12000
(align. acc=83.54%)
(align. acc=93.995)
(align. acc=98.6%)
Number of samples
10000
10000
10000
8000
8000
8000
4000
4000
4000
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Confidences
Confidences
Confidences
Figure 4: Confidence Scores. Compared to FoleyCrafter (left) and Diff-Foley (middle), our method (right) produces many more audio samples with higher confidence that align with their corresponding videos on the VGGSound test set (~ 15k samples).
that predicts audio-video alignment. As illustrated in Fig. 4, FoleyCrafter (acc=83.54 54%%) produces many low-confidence samples, indicating misalignment. In contrast, Diff-Foley (93.9%) achieves a higher proportion of high-confidence scores. Remarkably, our method reaches an accuracy of 98. 38.6% significantly increasing the number of high-confidence samples and demonstrating superior audio-video alignment compared to other approaches.
5.2 FEATURE DIMENSIONALITY REDUCTION AND REDUNDANT FEATURES IN CAVP
Unlike Diff-Foley, which uses a U-Net-based
Table 3: Dimension Reduction. Compare the
Stable Diffusson model and incorporates all 32 video frame features for cross-attention, we
original CAVP and Ours's 's features.
found that reducing video features from 32 chan- nels to a single channel significantly improves audio generation performance across all metrics (Tab. 3). Diff-Foley's 's CAVP encodes video features at 32 x 512, aligning with the audio's 32-channel representation (also 32 512) for latent-level aligmment via contrastive learning. However, in the second stage, mismatch arises as the VAE reduces audio dimensions to 4 x 16 x 64 while video expands
Video Feat.
Cond. Dim.
FID
18
Align. Acc.
Original CAVP
32 x 768
13.55
50.12
6.38
96.18
Reduced (Oues)
52.77
98.55
768, leading to redundancy and inefficiencies. Reducing video dimensionality to 1 x 768 representation acts bottleneck, simplifying learning and enhancing alignment, as supported by our results the Appendix A.2.
Analysis of the CAVP features in Diff-Foley shows significant redun- dancy, with cosine similarity averaging 0.9087 for real videos and 0.9233 for identical frames (Tab. 4). These high similarity scores suggest that the feature vectors from multiple frames largely over- lap, limiting the model's 's ability to learn distinctive characteristics essential for effective audio synthesis. This issue worsens when Diff- Foley expands features to 32 768 dimension, diluting key traits for latent diffusion modeling. In contrast, our approach employs a Reducer module to consolidate the 32 video frame features into 768-dimensional representation, effectively reducing redundancy and enhancing focus on salient features, which improves alignment accuracy and overall performance in audio generation tasks.
Table 4: Redundant Features. Cosine similarity between the frame's s features.
Input
Similarity (CAVP)
Video
0.9087
Image
0.9233
5.3 DOES NEURAL VOCODER HELPS?
We conducted experiments by training separate neural vocoder to convert mel-spectrograms back to waveforms. Using the publicly available HiFi-GAN code from GitHub, we trained the vocoder from scratch on the VGGSound dataset, achieving good convergence within three days. As shown Tab. 5, our method achieves FAD score of 4.3788 with the simple Griffin-Lim algorithm, outperforming both See-and-EE and-Hear& (5.5547) and Diff-Foley (6.00 (6.0810). When enhanced with the HiFi-GAN neural vocoder, method further improves the FAD score to 2.1610, achieving state-of ofate-of-the-ar performance on this metric. This result not only highlights the superiority of our approach but also demonstrates its robustness when evaluated beyond the metrics reported in Tab. 1.
7

---

Published as a conference paper at ICLR 2025
Table 5: FAD on VGGSound test set. Our method MDSGen-B using the simple Griffin-Lim outperforms the two methods and is state-of-the-art when equipped with a vocoder HifGANN.
Method
Sec-and-hear (ncural vocoder)
FolcyCrafter (ncural vocoder)
Diff-Folcy (Griftin-Lim)
Diff-Foley (ncural vocoder)
MDSGen-B (Griffin-Lim)
MDSGen-B
(ncural vocoder)
FADL
5.5547
2.4554
6.0810
4.7168
4.3788
2.1610
5.4 SUBJECTIVE HUMAN EVALUATION TESTS
We conducted a human evaluation by generating 50 audio samples based on 50 videos for each method. Five participants were asked to evaluate each method. Participants were instructed to watch the videos and listen to the corresponding audio, rating each on a scale from 1 to 5 based on the following criteria: 1) Audio Quality (AQ): How good is the sound quality? and 2) Audio-Vides Content Alignment (AV): How well does the sound match the video content? The mean opinion scores (MOS) for each metric (ranging from to 5) are presented in Tab. 6. The results from our human evaluation demonstrate that the waveforms generated by our model outperform those of competing methods, receiving higher scores across the evaluation criteria. Participants consistently rated the audio quality and alignment with the video content more favorably for our generated waveforms, indicating superior perceptual performance.
Table 6: Human Evaluation (MOS). Our method MDSGen-B equipped with a vocoder HifiGAN. AQ: Audio Quality, AV: Audio-visual content relevance.
Method
Sce-and-hear
FolcyCrafter
Diff-Foley
MDSGen-B (Ours)
Ground Truth
MOS-AQQ
2.68:10.25
3.211:0.23 3.44::0226
3.29-1.24 3.56:10.23
3.66:10.23 3.76:1:.21
4.74 10012
MOS-AVI
2.95++0.20
4.62:1:023
5.5 MASKING DIFFUSION STRATEGIES
We explore various audio-masking methods in diffusion transformers (Fig. 3) and compare them to traditional image-based techniques like SAM used with ImageNet (Gao et al., 2023). Our find- ings reveal that audio data behaves differently from images (SAM), with incorporating temporal awareness (TAM) into the masking task significantly boosting performance across all metrics (Tab. 7), including a 4-point IS score increase from 48.66 to 52.77. Using DiT without masking leads to suboptimal results across all metrics, highlighting the critical role of masking in model learning. TAM outperforms FAM, masking in the temporal dimension typically yields better performance
Table 7: Masking Strategy for Audio Generation. Comparison of different ways to train diffusion transformer-basee models. Masking on temporal audio gives the best performance.
Masking Method
Mask
Temporal
FIDJ
1St
KLJ
Align. Acc. (%)
DiT (Peebles & Xie, 2023)
14.55
46.11
6.51
97.12
Random, SAM (Gao et al., 2023)
X
12.44
48.66
6.30
98.15
Frequency, FAM Temporal, TAM (Ours)
X
12.79
46.33
6.41
97.58
V
11.19
52.77
6.27
98.55
in audio generation. This is likely due the stronger impact of temporal structure on coherence, perceptual quality, and the dependencies within audio sequences.
5.6 LEARNED WEIGHTS OF REDUCER
We analyzed how our models allocate attention across video frames by visualizing their learned magnitude weights (Fig. 5). The upper figure shows that our model applies varying attention levels, with larger models exhibiting higher weights and more distinct differences. After softmax normalization (bottom figure), consistent trends are observed for various channels, though not all, with model B focusing more channels 1, 2, and 32. These findings demonstrate that the Reducer effectively captures key features, selectively updating weights to prioritize relevant ones for audio generation. More experiments of Reducer choices available in the Appendix A.3.
8

---

Published as a conference paper at ICLR 2025
Weights Deetrbutton [Onggaall
16
10
19
20
26
29
30
Charree
Weights
16
17
19
27
30
Figure 5: Learned Weights of Reducer. Comparison of our three models.
5.7 SCALABILITY
We evaluate the scalability of MDSGen, the first explore ViT-based masked diffusion models for vision-guided audio generation. Results Tab. 8 show that increasing the model size from T to B improves all metrics. However, further scaling to the L model leads performance drop, indicating potential overfitting at larger sizes.
Table Scalability, We ablate four variants Tiny (T), Small (S), Base (B), and Large (L).
Model Config.
FID
1St
KLJ
Align. Acc.t
#Params
#Layers
Dim.
#Heads
MDSGen-T
13.93
39.24
6.17
97.9
5M
12
192
3
MDSGen-S
12.92
44.38
6.29
98.3
33M
12
384
6
MDSGen-B
11.19
52.77
6.27
98.6
131M
12
768
12
MDSGen-L
12.68
49.53
6.56
97.9
461M
24
1024
16
5.8 SAMPLING TOOLS
Sampling Method. We employ DPM-Solver (Lu et al., 2022) with 25 steps for sampling dur- ing inference. We find that increasing from 25 to 50 steps with dynamic classifier-free guidance (Gao et al., 2023) can slightly improve the per- formance. We used CFG = and power scaling 0.01 for the optimal setting. Classifier- Guidance (CG). We found that combining CFG and CG slightly improves alignment accuracy and KI,, consistent with (Luo et al., 2023), but has no impact on other metrics (Tab. 9), which differs from their findings. A thorough investigation of network architecture and additional datasets needed to assess the complementary effects of CFG and CG, which is beyond the scope of our work.
Table 9: CFG and CG. We examine the effect of classifier-fres free guidance and classifier guidance.
Results are shown with model MDSGen-B B. Gray indicates the default.
Setup
FID.
IS1
KL]
Align, Acc.
No Guidance
16.50
23.54
6.85
$4.1
CFG
11.19
52.77
6.27
98.6
CFG + CG
11.25
51.48
6.24
98.8
5.9 COMPARE THE EFFICIENCY
We evaluated inference time, parame- ter count, and memory usage sin- gle A100 GPU (80GB) with batch size Tab. 10 shows our method is signif- icantly faster, uses fewer parameters, and consumes less memory than exist- ing methods. Specifically, Diff-Foley (860M) achieves 93.9% alignment ac- curacy with 0.36s inference time, while our MDSGen T (5M) reaches 97.9% in just 0.01s, 36x faster and 371% more memory efficient. Our larger model MDSGen
Table 10: Efficiency Comparison. Our approach is simple and highly efficient across all metrics, with superior align- ment accuracy compared to existing methods.
Method
Time]
Mem. Use]
#Params]
Align. Acc.1
1m2 (Sheffer & Adi, 2023)
1684M
443M
67.4
Hear (Xing
2024)
Craltrr (Zhang ct al., 2024)
2.96s
1200M
1252M
83.5
Dill-Foley (Luo et al., 2023)
0.36s
5228M
860M
93.9
MDSGen-T (Ours)
MDSGen-S (Ours)
0.02s
1508M
33M
98.3
MDSGen-B (Ours)
0.05%
2132M
131M
98.6
(131M) improves accuracy to 98.6%, still 7.2x faster and 245% more memory efficient than Diff- Foley. Compared to FoleyCrafter and See and Hear, MDSGen-T is 296 and 1 1825x x faster, respec- tively, while being 10x more memory efficient. We provide additional details on training efficiency in the Appendix, further emphasizing the remarkable efficiency of approach.

---

Published as a conference paper at ICLR 2025
ToleyCratter loput
Waveform
Mel Spectrogram
nieelry SnbyCColeer Spinn & Heat
Ous
Groine Iruhh
Ground
Figure Waveform and Mel-Spectrogram Comparison. Sample taken from the test set of the VGGSound dataset. Our model generates sound more closely aligned with the ground truth than existing methods. The video of a woman playing drum by hand, demo file "ONIE- eDk92M eDk92M_000029.00 11l.000000.a./ wav" is available in the supplementary material.
5.10 VISUALIZATIONS
We visualize different approaches using test video samples from the VGGSound dataset. As shown in Fig. 6, our method generates -ee-sppctssgrams that closely match the ground truth (right figure), with even clearer distinctions observable in the waveform (left figure). Additional demo samples, along with WAV files for convenient listening and their visualizations, are included in the Appendix and supplementary material.
CONCLUSIONS
This work presents novel, scalable, and highly efficient framework for cileo-guided audio generation. Leveraging Diffusion Transformers, we introduced an innovative masking strategy that enhances model's ability to capture temporal dynamics audio, leading to significant performance gains. To address redundant video features, introduced a Reducer module to eliminate unnecessary information. Extensive experiments and detailed analyses demonstrate that our model achieves fast training and inference times, uses minimal parameters, and delivers superior performance across multiple metrics, setting a new benchmark in the field.
LIMITATIONS AND FUTURE WORKS
Our method offers fast inference, efficient parameter usage, and low memory consumption, while achieving top performance in alignment accuracy and IS score. However, there are some limitations. First, like other diffusion models, it requires multiple sampling steps during generation. Second, while the VGGSound dataset suitable for this study, its size may not fully leverage the potential of our approach. Third, the current design is constrained to fixed video length of 8.2 seconds. In the future, we aim to incorporate recent advancements in single-step diffusion techniques address this limitation. Additionally, although video collection from online sources is becoming more feasible, it remains time-consune cons-mnnni and storage-intensive, which may be challenging for individual researchers. We plan to explore the potential of ID VAEs for further improvement and to address the fixed-lenght constraint in future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work was supported by the Institute for Information & Communications Technology Planning Evaluation (IIT)) grant funded by the Korea government (MSIT) (No. RS-2021-II21 Development of Causal AI through Video Understanding and Reinforcement Learning, and Its Applications to Real Environments) and (No. RS-2022-11220184 Development and Study of Al Technologies to Inexpensively Conform to Evolving Policy on Ethics).
REFERENCES
Vanessa Theme Ament. The Foley grail: The art of performing sound for film, games, and animation. Routledge, 2014. 1
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances in neural information processing systems, 29, 2016. 5,
He Bai, Renjie Zheng, Junkun Chen, Mingbo Ma, Xintong Li, and Liang Huang. Alignment-aware acoustic and text pretraining for speech synthesis and editing. In International Conference on Machine Learning, pp. 1399-1411, PMLR, 2022.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575, 2023.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305. 09636, 2023. 3
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11315-11325, 2022. 3
Changan Chen, Puyuan Peng, Ami Baid, Zihui Xue, Wei-Ning Hsu, David Harwath, and Kristen Grauman. Action2sound: Ambient-aware generation action sounds from egocentric videos. arXiv preprint arXiv:2406.0922 2024. 3, 20
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020a.
Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan. Generating visually aligned sound from videos. IEEE Transactions on Image Processing, 09:8292-8300 2020b.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer. arXiv preprint 23Xh::2303.14389 2023. 2, 5,
Hugo Flores Garcia, Prem Seetharaman, Rithesh Kumar, and Bryan Pardo. Vampnet: Music generation via masked acoustic token modeling. arXiv preprint arXiv:2307.04686 2023. 3
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the JEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 151 10-15190, 2023.
Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Transactions on acousties, speech, and signal processing, 32(2):236-244 1984.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022. 19

---

Published as a conference paper at ICLR 2025
Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang. Free-bloom: Zero-shot text-to-video generator with Ilm director and ldm animator. Advances in Neural Information Processing Systems, 36, 2024.
Yao Huang. Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in Neural Information Processing Systems, 35:28708-28720, 2022. 3
Vladimir lashin and Esa Rahtu. Taming visually guided sound generation. In British Machine Vision
Conference, 2021. 1, 2, 6
Young Hoon Jung, Scong Kwang Hong, Hee Seung Wang, Jae Hyun Han, Trung Xuan Pham, Hyunsin Park, Junyeong Kim, Sunghun Kang. Chang D Yoo, and Keon Jae Lee. Flexible piezoelectric acoustic sensors and machine learning for speech processing, Advanced Materials, 320550: 2020. 19
Young Hoon Jung, Trung Xuan Pham, Dias Issa, Hee Seung Wang, Jae Hee Lee, Mingi Chung, Bo- Yeon Lee, Gwangsu Kim, Chang Yoo, and Keon Jae Lee. Deep learning- based noise robust flexible piezoelectric acoustic sensors for speech processing. Nano Energy, 101:107610, 2022. 19
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15954-15964, 2023.
Junyeong Kim, Minuk Ma, Trung Pham, Kyungsu Kim, and Chang Yoo. Modality shifting attention network for multi-modal video question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10106-10115, 2020. 19
Jungil Kong, Jaehyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 07777022-7733., 2020. 4
Gwanhyeong Koo, Sunjae Yoon, Ji Woo Hong, and Chang D Yoo. Flexiedit: Frequency-aware latent refinement for enhanced non-rigid editing. In European Conference on Computer Vision, pp. 363-379. Springer, 2024. 19
Donghoon Lee, Hyunsin Park, Trung Pham, and Chang D Yoo. Learning augmentation network via influence functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10961- 10970, 2020. 19
Chang Li, Ruoyu Wang, Lijuan Liu, Jun Du, Yixuan Sun, Zilu Guo, Zhenrong Zhang, and Yuan Jiang. Quality-aware masked diffusion transformer for enhanced music generation. arXiv preprint arXiv:2405. 15863, 2024. 2
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02744. 2022. 19
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei. Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. In International Conference on Machine Learning, pp. 11450-214744 PMLR, 2023. 2, 20
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint a7x009221..01095 2022. 9
Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized lideo-to-audl synthesis with latent diffusion models. Advances Neural Information Processing Systems, 36, 2023. 1, 2, 4, 5, 5, 6, 9, 17, 19
Xiaofeng Mao. Zhengkai Jiang, Qilin Wang, Chencan Fu, Jiangning Zhang, Jiafu Wu, Yabiao Wang, Chengjie Wang, Wei Li, and Mingmin Chi. Mdt-a2g: Exploring masked diffusion transformers for co-speech gesture generation. arXiv preprint 1333v.2408.03312, 2024.
12

---

Published as a conference paper at ICLR 2025
Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In European Conference on Computer Vision, pp. 218 234. Springer, 2022. 5, 18
Axi Niu, Kang Zhang, Trung X Pham, Jinqiu Sun, Yu Zhu, In So Kweon, and Yanning Zhang. Cdpmsr: Conditional diffusion probabílistic models for single image super-resolution. In 2023 IEEE International Conference on Image Processing (ICIP), pp. 615-619. IEEE, 2023. 19
Axi Niu, Trung X Pham, Kang Zhang. Jinqiu Sun, Yu Zhu, Qingsen Yan, In So Kweon, and Yanning Zhang. Acdmsr: Accelerated conditional diffusion models for single image super-resolution. IEEE Transactions on Broadcasting, 2024a. 19
Axi Niu, Kang Zhang, Trung X Pham, Pei Wang. Jinqiu Sun, In So Kweon, and Yanning Zhang. Learning from multi-perception features for real-word image super-resolution. IEEE Transactions Circuits and Systems for Video Technology, 2024b. 19
Yichen Ouyang, Hao Zhao, Gaoang Wang, al. Flexifilm: Long video generation with flexible conditions. arXiv preprint arXiv:2404. 18620, 2024.
Daniel S Park, William Chan, Yu Zhang, Chung-Chene Chiu, Barret Zoph. Ekin D Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv: (9XX:::/904007799 2019. 3
Santiago Pascual, Chunghsin Yeh, loannis Tsiamas, and Joan Serrà. Masked generative video-to-audio transformers with enhanced synchronicity. arXiv preprint arXiv:2407. 2024. 3
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023. 3, 8, 17
Trung Pham, Chaoning Zhang, Axi Niu, Kang Zhang, and Chang D Yoo. On the pros and cons of mo-
mentum encoder in self-supervised visual representation learning, arXiv preprint 2rXiv:2208.05744, 2022a. 19
Trung X Pham, Rusty John Lloyd Mina, Dias Issa, and Chang D Yoo. Self-supervised learning with local attention-aware feature. arXiv preprint arXiv:2108.00475, 2021. 19
Trung X. Pham, Kang Zhang, and Chang D. Yoo. Cross-view masked diffusion transformers for person image synthesis. In Forty-first International Conference on Machine Learning, 2024. 2, 19
Trung Xuan Pham, Rusty John Lloyd Mina, Thanh Nguyen, Sultan Rizky Madjid, Jinwoong Choi, and Chang Dong Yoo. Lad: A hybrid deep learning system for benign paroxysmal positional vertigo disorders diagnostic. IEEE Access, 2022b. 19
Trung Xuan Pham, Axi Niu, Kang Zhang, Tee Joshua Tian Jin, Ji Woo Hong, and Chang D Yoo. Self-supervised visual representation learning via residual momentum. IEEE Access, 2023. 19
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022. 4, 17, 18, 20
Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation. ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pp. 1-5. IEEE, 2023. 2, 6,9
Sojeong Song, Scoyun Yang, Chang D Yoo, and Junmo Kim. Implicit steganography beyond the constraints of modality. In European Conference on Computer Vision, pp. 289-304. Springer, 2024. 19
Pham Xuan Trung and Chang D Yoo. Short convolutional neural network and mfces for accurate speaker recognition systems. International Technical Conference on Circuäts/Ssstems, Computers and Communications (ITC-CSCC), 2019. 19
Thang Vu, Hyunjun Jang, Trung X Pham, and Chang Yoo. Cascade rpn: Delving into high- region proposal network with adaptive convolution. Advances in neural information processing systems, 32, 2019. 19
13

---

Published as a conference paper at ICLR 2025
Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 5
Yazhou Xing, Yingqing He. Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open- domain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7151-7161, 2024. 2. 6,
Jian Xu, Zhiguo Chang. Jiulun Fan, Xiaoqiang Zhao, Xiaomin Wu, Yanzi Wang, and Xiaodan Zhang. Super-resolution via adaptive combination of color channels. Multimedia Tools and Applications, 15:155115 2017. 18
Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Kang Zhang, Yu-Jung Heo, Du-Seong Chang, and Chang D Yoo. Bi-mdrg: Bridging image history in multimodal dialogue response generation. In European Conference on Computer Vision, pp. 378-396. Springer, 2024. 19
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image
diffusion models, 2023. 2
Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. Foleycrafter: Bring silent videos life with lifelike and synchronized sounds. arXiv preprint arXivv2407.011494 2024. 1, 2, 6,
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 REDUCER DETAILS
We show the simple design of our Reducer that can help to obtain global information while retaining local information:
Input: Video feature V
3232x112
Output: feature vector
R 1xx66
Fig. 7 illustrates the details of the proposed Reducer architecture with two layers: the fully connected layer captures the local information of the video, and the second layer (IxI conv) extracts the global information. Specifically, after the initial layer with fully connected weights, each dimension component (position) from 768 in the output vector i1 contains the whole vector U1 (local information of video). Consequently, in the next layer, each component of the final vector E IR 1x78 captures both local and global information from the original 32x512 12 video information. This ensures that the final vector provides comprchensive information for the subsequent audio generation process. This lightweight vector significantly reduces the burden of the DiT process.
V1
u
4
02
0.
W1
0s12
512
768
Ū E 768
22
172
0
2
0/
0s12
768
Wa2
U32
1732
0
0g
06
8512
t
Input
Layer 1
Out1
Layer 2 Convid (32,1)
Out2
32x512
Linear (512,768)
32x768
1x768
Figure 7: Reducer Architecture. It includes a linear layer Linear(512,768) and a 1x1 conv layer Conv 1)nvld(32,1) that helps retain local information while reducing dimension.
15

---

Published as a conference paper at ICLR 2025
A.2 OVERHITTING PHENOMENON WITH REDUNDANT FEATURES
We observe that the model becomes quickly overfitted if using redundant features in Fig. 8. By contrast, our Reducer helps mitigate such redundant features and we can see that the test accuracy remains quite close gap with train accuracy from 100k steps to 500k steps.
video features, 8220
Reduced video festure, E gph776
0.3%
Algn Accurres
2.5%
Irain
lest
Test
Train steps
Train steps
Video w/ redundant features
bJ Our reduced features
Figure 8: Overfitting with Redundant Features. We see that the redundant features show bigger gap of overfitting where the test and train accuracy gap becomes larger.
A.3 REDUCER CHOICES
We conducted an ablation study on the choice of the Reducer using three approaches: 1) Naive average pooling, 2) Attention pooling, and 3) Learnable weights. For this study, we used the MDSGen-B model trained for 300k steps. The results, shown in Tab. 11, indicate that all pooling methods achieve competitive performance with comparable alignment accuracy (Align. Acc). However, Learnable Weights yield the highest inception score (IS) 7 points and slightly outperform in terms of FID. Meanwhile, Attention Pooling achieves the best KL metric. We hypothesize that Attention Pooling
Table 11: Reducer Vector Pooling Choice. Model MDSGen-B B is trained for 300k steps
Pooling Method
FID
IS f
KL 1
Align. Acc.
Naive Average Pooling
12.5594
39.41122
6.1750
0.9848
Attention Pooling Learnable Weight (default)
12.2704
39.0696
6.0819
0.9847
12.1534
46.3301
6.3066
0.9858
performs better the KL metric because its adaptive weights focus on the most relevant features in the input, enabling a more precise reconstruction of the latent distribution and, consequently. better KL divergence minimization. On the other hand, the Learnable Weights method performs best the inception score because directly optimizes the contribution of each dimension, tailoring the representation for the final task. This flexibility allows the model to capture both global and local information more effectively, leading to improved perceptual quality as reflected in the IS metric.
Learnalee weights can indeed be considered form of adaptive weighting since the weights optimized during training and dynamically adjusted based on the data and task requirements. The distinction lies in the mechanism:
Attention pooling calculates adaptive weights based the input features themselves (using attention scores). This is data-dependent and can adapt to specific patterns the input at each forward pass.
Learnable weights, the other hand, parameterized and optimized during training, making them adaptable over time but not directly dependent on the input features in real time.
So while both methods involve adaptivity, attention pooling adapts dynamically per input, whereas learnalle weights are statically optimized across the dataset. Naive average pooling is less effective compared to attention pooling and learnable weights because it assigns equal importance to all input
16

---

Published as a conference paper at ICLR 2025
features. regardless of their relevance to the task. This uniform weighting lacks the ability to focus on critical features or filter out irrelevant ones, which can dilute the quality of the pooled representation.
A.4 CHOICE OF DECODER LAYERS
We compared the number of decoder layers and showed that N2 gives better results multiple metrics compared to N₂ = 2 as in the below table:
Table 12: Choice of decoder layer. Model MDSGen-B is trained for 300k steps
Decoder
FID4
IS f
KL
Align. Acc.
N2 = 2
12.4602
47.9981 46.3301
6.4344
0,9823
N2 = 4 (default)
12.1534
6.3066
0.9858
A.5 TRAINING COST
We leveraged the pre-trained VAE encoder and decoder from Stable Diffusion (Rombach et al., 2022), keeping them frozen during training and inference, similar to Diff-Foley. Our training utilized single A100 GPU (80GB) with a batch size 64. The B-model (131M) is projected to take 4 days for 500k iterations, while the S-model (33M) and T-model (5M) are expected to finish 3.3 and 2.8 days, respectively.
In comparison, the second-best method, the Diff-Foley approach (860M model) required A100 GPUs with batch size of 1760, completing 24.4k steps in 60 hours (2.5 days) (Luo et al., 2023). If scaled to a single A100 GPU, Diff-Foley would need at least 20 days more than a fifth of the time of our method, demonstrating the superior efficiency and significantly lower training costs of our approach (see Tab. 13).
Table 13: Training Comparison. Estimated for single A100 GPU training. Our approach is simple and highly efficient compared to the second-best method Diff-Foley which used the heavy backbone of Stable Diffusion with 860M.
Method
# Training cost.
Align. Acc.t
Foley (860M) (Luo et al., 2023)
20 days
93.9
MDSGen-T, 5M (Ours)
2.8 days
97.9
MDSGen-S, 33M (Ours)
3.3 days
98.3
MDSGen-B, 131M (Ours)
4.1 days
98.6
A.6 MORE SETUPS
We apply classifier-fres guidance during training by randomly setting ö' zero with 10% probability. Models are trained for 500k steps to ensure optimal convergence. The exponential moving average of the model weight is set to 0.9999, otherwise, settings are the same as default DiT (Peebles & Xie, 2023). No video augmentation is used; instead, we pre-extract and save lightweight video features for faster training. We also use a ratio of Tm = 0.3 by default for masking the temporal set of tokens. Our code will be made publicly available.
For experiments involving classifier guidance, we utilized the classifier trained by Diff-Foley, adjust- ing the optimal CG value 2.0, compared to 50 in their framework. To evaluate alignment accuracy, we used their trained classifier to assess our generated audio. We also reproduced Diff-Ford hiff-Foley's 's performance using their published checkpoint, with results closely matching their reported metrics.
The slight variation may stem from differences in the VGGSound test set, as we download videos from several months to one year after their experiments, during which some original YouTube links may have been removed, causing potential mismatches.
A.7 COMPARING MORE VISUALIZATIONS
Gradcam Visualization of localization VGGSound and Flickr-SoundNet datasets.
17

---

Published as a conference paper at ICLR 2025
We used the generated audio perform the sound souree localization each frame of the video and image using the pre-trained model of EZ- VSL (Mo Morgado, 2022). As shown in Fig. and Fig. 11, our method provides more accurate attention map.
Image
See and Hear
FoleyCrafter
Diff-Foley
Ours
(a)
(b)
(c)
(d)
Figure 9: Attention map Flickr-SoundNet dataset. Our best model (MDSGen-B) generated the sound that contain information that help localize the sound source more accurately compared to existing approaches.
More generated audio comparison with state-of-the-art approaches on VGGSound test set.
We also provide more samples in the supplementary and their visualizations in Fig. 12, Fig. 13, Fig. 15, Fig. 16, and Fig. 14. As shown in these figures and listened to by authors, our generated audio is much reasonable than others. We refer readers examine the quality of generated audio in the submitted supplementary materials.
A.8 CHANNEL SELECTION FROM RGB FOR MEL-SPECTROGRAM
We provide additional statistics of various generated audio samples, highlighting the differing characteristics of the VAE decoder outputs in Fig. 17 and Fig. 18. As shown, although the VAE encoder input consists of three identical channels, the generated outputs display distinct distributions across each channel (left figures), even though these differences are imperceptible to the human eye (right figures). This behavior stems from the fact that the VAE encoder and decoder in Stable Diffusion (Rombach et al., 2022) are trained exclusively for image data, where the R, G, and channels inherently carry different information.
Because this model applied directly to audio data without adaptation, there no constraint ensuring the R, G, and B channels remain identical in the generated audio. Developing method to adaptively select or combine these channels when constructing the final Mel-spectrogram could be a promising avenue for improving the quality of the generated audio.
Each R, G, and B channel exhibits distinct char- acteristics, as noted in previous research (Xu et al., 2017). In the VAE encoder, we replicate the gray mel- -peestectro across three identical channels, but the VAE decoder does not enforce channel consistency. Our analysis shows that
Table 14:
Channel selection for mel- spectrogram. Gray indicates the default.
Channel
FID/
IS1
KL|
Align. Acc. (%)
((=00)
11.40
51.54
6.29
98.51
(B=l)
11.19
52.77
98.55
(1=2)
11.23
52.32
6.27
98.56
11.29
52.27
6.28
98.54
18

---

Published as a conference paper at ICLR 2025
300-
2007
200
1000
1400
2004
1250
2.8
(a) Histogram R,G,B
(b) Histogram RGB vs. GT Mel-spectrograms
Figure 10: Histogram and mel- mel-spectrogram comparison of three channels of VAE output.
the RGB output X RRBB retains unique statistical
differences across channels (see Fig. 10), influ-
encing their contributions to the final mel- mel-spectrogram and waveform. In contrast to Diff-Forr Diff-Foley. which uses only the R channel (X [0, :l) for the final mel-spectrogram, we find the G channel (XRGB [1, :, :I) be optimal (see Tab. 14). Fig. 10 shows that the histograms of the three VAE output channels display sigmificant differences, with the G and B channels aligning closely with the ground truth distribution.
Notably, while the resulting mel-speetrograms (right figures) seem visually indistinguishable, the histograms highlight their differences. This emphasizes the importance of considering each channel's statistics in generating the final mel-spectrogram X, with more comparisons in the Appendix.
A.9 MASK RATIO ABLATION
Tab. 15 shows that while higher masking ratio maintains high alignment accuracy, it leads declines in other metrics. This occurs because the transformer models prioritize audio token reconstruction over the primary generation task, resulting in worsened FID. IS, and KL scores.
Table 15: Masking Strategy for Audio Synthesis. Comparison of different ways to train diffusion transformer-basee models. Masking on temporal audio gives the best performance.
Masking Ratio
FID
IS
KL,
Align. Ace.1 (%)
70%
12,85
44.42
6.38
97.86
50%
12.39
46.77
6.38
98.43
30%
11.19
52.77
6.27
98.55
A.
MORE DIRECTIONS
Our approach can inspire applications of masked diffusion models across various domains such as object detection (Vu al., 2019), video question answering (Kim al., 2020), image super-resolution (Niu et al., 2024a;b), and speech processing. These models can also be combined with self-supervised learning techniques (He et al., 2022; Pham al., 2021; 2022b; 2023; 2022a) enhance representation learning for diverse generative tasks. With the potential of diffusion transformers for conditional learning, we anticipate further exploration of their capabilities speech processing (Jung et al., 2022: 2020; Trung & Yoo, 2019), data augmentation (Lee et al., 2020). VQA (Kim et al., 2020), visual detection (Vu et al., 2019), and super-resolution (Niu et al., 2023: 2024a;b). Our work demonstrates the potential of Masked Diffusion Transformers for audio generation while also highlighting their applicability to various other tasks, including image editing (Koo et al., 2024; Song et al., 2024; Yoon al., 2024) and personalized image generation using flow-based methods (Pham et al., 2024; Lipman et al., 2022).
B VAE CHOICES FOR MELSPECTROGRAM RECONSTRUCTION
We conducted an experiment to assess the impact of the VAE in our overall framework. Specifically, compare the often used Stable Diffusion VAE which is used in Diff-Foley (Luo et al., 2023) and
19

---

Published as a conference paper at ICLR 2025
Action2Sound (Chen et al., 2024) and compared with the case when employ the AudioLDM VAE to replace Stable Diffusion VAE. The results are shown in the below tables. Several key findings are observed. Below, we provide side-by-side comparison highlighting the performance differences between the image VAE and the AudioLDM VAE across key metrics such as FAD, IS, KI., and Alignment Accuracy, and efficiency.
Part 1. Performance Comparison. We find that although the AudioLDM VAE achieves better results in FAD and KL, with a comparable FID score, its performance in Alignment Accuracy and IS is not as strong as that of the image VAE. Our smallest model, MDSGen-T (5M), achieves a state-of-the-art FAD score of 2.21, surpassing existing video-to-audio methods (the FAD of FoleyCrafter is 2.45). The larger model, MDSGen-B (131M), further improves the FAD to 1.34. Note that, we reported FAD 2.16 of our MDSGen the previous response from checkpoint 500k training steps with image VAE. The detailed results are summarized in Tab. I below (300k steps):
Table 16: Performance Comparison Audio VAE: Image VAE vs. AudioLDM VAE
Model config. (Ours)
VAE choice
FID.
IST
KL,
Align. Acc.t (%)
FAD1
MDSGen-B (131M)
Image VAE (Rombach et al., 2022)
12.15
46,41
6,31
98.58
2.36
DudioLDM VAE (Liu et al., 2023)
11.83
30.37
6.03
96,52
1.34
MDSGen-S (33M)
VmE (Rombach al., 2022)
12.88
40.39
6.21
98.25
2.75
AudioLDM VAE (Liu al., 2023)
13.14
26.05
5.83
95.63
1.66
MDSGen-TTiny (5M)
Image VAE (Rombach al., 2022)
14.78
33.23
6.23
97.62
3.69
AMdioLDM VAE (Liu al., 2023)
15.48
21.99
5.58
95.53
2.21
Part 2. Efficiency Comparison. We observed slight difference in efficiency between the two VAEs, primarily due to variations in imput dimensions and output characteristics. For instance, the AudioLDMM VAE produces 816 latent tokens, which is 3.1x longer than the 256 tokens generated by the image VAE. As a result, models using the AudioLDM VAE tend to be slower and require more memory. However, it's important note that all variants of our model remain considerably more efficient than existing baseline methods. The details are shown in Tab. below:
Table 17: Efficiency Comparison: Image VAE vs. AudioLDM VAE
VAE
MelSpec. Res.
Lafent
#Tokens
Models
#Prams. (M)L
Infer. Time (s).)
#Memoryy (M).
Image VAE
128 512
16 x 61
256
MDSGen-T
5M
0.01
1406
MDSGen-S
33M
0.02
1508
MDSGen B
131M
0,05
2132
DD VAE
64 x 816
16 x 204
816
MDSGen-T
SM
0.058
3087
MDSGen-S
0.114
5383
MDSGen-B
131M
0.243
10165
VAF Analysis Insight 1. Compared to existing video-to-audio methods, both VAEs show that MDSGen achieves state-of-the-ar performance in FAD (1.34-2.16) and Aligmment Accuracy (96.5%99.55%) While the Image VAE outperforms the AudioLDM VAE by about 2% in Alignment Accuracy, it falls behind in FAD (2.16 vs. 1.34), indicating that AudioLDM produces higher-quality audio. This discrepancy in alignment accuracy and IS may be attributed to several factors:
Alignment Classifier Training: The alignment classifier is trained on 128x512 spectrograms, whereas AudioLDM uses 64x8 16 melsspectrsgrams, inherently favoring features from the Image VAE, which aligns more closely with its input format. Domainn-Seciif, Design: AudioLDM optimized for audio reconstruction, resulting in features less visually aligned with the mel-spectrograms expected by the classifier. Resolutions: Image VAES operate at higher spatial resolutions (128x51 12) and preserve finer, visually coherent patterns, leading to higher IS scores, while AudioLDM's lower resolution (64x816) focuses on acoustic fidelity rather than visual consistency. Overall, we hypothesize that the Image VAE performs better in IS and alignment accuracy due to its compatibility with visual-based metrics and the image-like nature spectrograms. In contrast, Audio VAEs prioritize audio quality, creating trade-off between perceptual fidelity and visual alignment in video-to-audin tasks.
20

---

Published as a conference paper at ICLR 2025
Insight 2. Both configurations show that MDSGen is highly efficient, utilizing significantly fewer parameters than existing approaches while maintaining strong performance across key metrics, including FAD, FID, and audio-video alignment accuracy.
Insight 3.
The Au VAE improves audio quality over the image VAE, but at the cost of slower inference and slightly lower midio-video content relevance. Overall, our results reveal trade-off: integrating MDSGen with the image VAE achieves the highest Alignment Accuracy and faster inference, while the MudioLDM VAE delivers superior audio quality (FAD) with slower inference.
In summary, our masked diffusion model, MDSGen, is the first to support both image- and audio- based VAEs for the video-to-audio task, offering well-balanced trade-off between alignment accuracy and high-quality audio, while maintaining exceptional efficiency in parameters, memory usage, and inference speed. In the revised PDF, have highlighted this in Section 3.5: VAE Choice for Mel-Spectrogram, where we discuss replacing the channel selection used in the RGB VAE, Additionally, we have moved the related discussion to the Appendix and will include detailed analysis demonstrating the compatibility of our model with both image and audio VAEs.
Frames
FoleyCratter
DIT
10
15
FoleyCrater
Foley
FoleyCratter
10
Figure 11: Attention map VGGSound dataset. Our best model (MDSGen-B) generated the sound that contain information that help localize the sound source more accurately compared to existing approaches.
21

---

Published as a conference paper at ICLR 2025
Input
Spectrogram
Deft PaleyCraiter Seeing
FoleyCratter
Ground
Time 360
Time 161
Figure 12: The video of man playing guitar solo. Our best model (MDSCen-B) generated sound that is closer to compared to existing approaches. We refer the reader to the listen file provided in the supplementary for comparison. File ".IPXTBXa0EE 000000..0000...
Video Input
Waveform
Mil Spectrogram
neddnlay Foleyrrdeee Seongg rour
FoleyCratter
OuPs
Bround hhuhh
Time 153
Time
Figure 13: The video of a woman playing drum by hand. Our best model (MDSGen-B) generated sound that is closer to GT compared existing approaches. We refer the reader the listen file provided in the supplementary for comparison. File "ONIE-eDk92M M00IE=-DD92M.00002...aa
22

---

Published as a conference paper at ICLR 2025
toleyDatee Video
Waveform
Mel Spectrogram
ToleyCrattor Seeng S.Hear
Dificley
Ous
Time Sc]
Time (5)
The video of guy playing drum by tool. Our best model (MDSGen-Bi BB) generated sound that is closer to GT compared to existing approaches. We refer the reader to the listen file provided in the supplementary for comparison. File "-QowmedP9cc0000034
Figure 14:
Imputt
Waveform
Mel Mell-ppectrogram
Pnteyrraber Saning noar
Ous
Brsund Iich
Figure 15: The video of a dog looks like howling. Our best model (MDSGen-B) generated sound closer to GT than existing approaches. We refer the reader to the listen file provided in the supplementary for comparison. File "2vKkywD-fke 000""0.wav".
23

---

Published as a conference paper at ICLR 2025
Inpux
MellSSeectragram
Fnieyfraatt Seeing Heat
Dif-eleyy
Ous
Emnan Indh
Groand
Figure 16: The video of a guy playing badminton. Our best model (MDSGen-B) generated a sound that is closer GT compared to existing approaches. We refer the reader the listen file provided in the supplementary for comparison. File "-mil_ C3A14Y 000104.wav".
24

---

Published as a conference paper at ICLR 2025
Charnole charnel Channel
20.0
Coas"
1500
Samelees
1022
203
0.2
0.5
0.0
Tinc
Chenzel
Chathwl
12000
Chennell
5500
Sample 2
Dequency
6200
2000
Dequercs
0.0
0.2
54
0.6
0.6
Tino
Velue
4032
chontele
Charne
Chennel
Timc
Sanpeee
2020
1509
Pacuancy
500
Tyma
Vilue
Figure 17: RGB distribution for Mel-Spectrogram. We provide more evidential samples that the output of the VAE in the test set yields different characteristics for three channels even though these differences are imperceptible to the human eye (right figures). Interestingly, we find that the first channel (R) always has some different patterns compared the remained channels (1).
25

---

Published as a conference paper at ICLR 2025
Chenneto
Dequenty
102
0.
64
1.0
TOYG
Chenzel
800
Sample 2
Dequentyy
200
0.4
1.0
Tenc
Vilae
Channels Channel
6:0-
hequary
Sampeee
400
Pacumnt
Tyme
Vilue
Figure 18: RGB distribution for Mel-Spectrogram. We provide more evidential samples that the output of the VAE in the test set yields different characteristics for three channels even though these differences are imperceptible to the human eye (right figures). Interestingly, we find that the first channel (R) always has some different patterns compared the remained channels (2).
26


Reviews:
[{"review_id": "ZvfOoUvtYv", "reviewer": "Reviewer_GqkQ", "summary": "**Update after discussion period**:\n\nMy biggest concern of this paper is about its audio reconstruction pipeline. The original pipeline consists of an RGB VAE and a non-DNN mel2wave conversion module, resulting in terrible sound quality. Moreover, such a pipeline has been used in video2audio community widely, which is a pity.\n\nDuring the rebuttal period, the authors not only added the experiment of using a vocoder for the mel2wave conversion, but also re-trained their models with an audio VAE. As a result of their efforts, the FAD score improved from 4.37 to 2.16 and eventually 1.34. Congrats on the achievement!\n\nI listened to the latest samples in the Google Drive, and found a distinct advantage of audio VAE in samples that contain non-noise or musical sources (sample027__SrU3mfTPYg_000032.mp4, sample045_2es7oZzwLWM_000030.mp4). Although the AV-align score is slightly worse with an audio VAE, the score is still higher than conventional methods by a large margin.\n\nSince the AV-align score is computed by a pretrained DNN model, moreover, considering the gap between v2a community and audio generation community, the metric might not be very reliable. For example, in samples such as \"sample007_-2sOH8XovEE_000484.mp4\", I don't feel the alignment is as good as the scores indicate. Perhaps some future works can be done to further improve the metric itself.\n\nI increased my ratings to **encourage further collaboration between video2audio and audio communities**.\n\n---------------\n\nThe paper proposes \"MDSGen\", an efficient model based on Masked Diffusion Transformer, for video-to-audio generation.\n\nThe challenges of video-to-audio generation are mainly:\n1. Heavy computation and memory usage;\n2. Requirements for the audio quality;\n3. Requirements for the audio-video alignment;\n\nMDSGen reduces the resource consumption by using very light-weight Transformer coupled with fast diffusion samplers such as DPM solver, as well as a dimension reduction module to reduce the size of the video conditioning embeddings.\n\nMDSGen improves audio quality and audio-video quality by introducing a time-aware masking strategy into the mask DiT framework, together with other efforts.\n\nConceptualy, MDSGen looks like a framework that replaces the \"text prompt\" in text-to-audio DiT [StableAudioOpen],[MakeAnAudio2] by a video feature embedding. Hence the technical contributions are more in micro aspects.\n\nHowever, some design choices may have severely affected the audio quality, making the work less solid or reusable to the community. Audio quality observed in the supplementary files is far from the level in modern text-to-audio models such as [AudioLDM], [MakeAnAudio2], [SpecMaskGIT], [StableAudioOpen].", "questions": "Three models are presented in the paper (Tiny, Small and Base) except the overfitting large model. What is the model presented in the supplementary files?\n\nIs there any reason to use an image VAE instead of audio VAEs? \n\nDid the authors observe any advantage of GLA over a neural vocoder?\n\nIs it possible to run a subjective listening test, and see the consistency between human evaluation and the audio-video alignment accuracy measured by a DNN model?\n\nCould the authors measure the FAD scores on top of the current FID scores?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "Most contributions of MDSGen are about micro design aspects.\n1. Channel selection of Mel-spec\n2. Time-aware masking strategy for generative models\n3. Reduced dimension of the video features\n4. Small model size and fast inference speed", "weaknesses": "## Major issues\n### 1. Improper audio reconstrution pipeline\nMDSGen ustilizes the VAE from Stable Diffusion, which is not trained for Mel-spec. Although the authors carefully discussed how to take the most advantage of this image VAE, the discussion itself is **NOT** reusable for the audio community, as there have been plenty of audio VAE designs, some of which are publicly available such as [AudioLDM], [MakeAnAudio2], [StableAudioOpen], [DAC].\n\nAnother improper choice is that, MDSGen utilizes the Griffin-Lim Algorithm (GLA) to convert mel-spec back to wave forms. GLA has almost been abandoned by audio community, due to the recent advance in neural vocoder, e.g., [HiFiGAN], [UnivNet], [BigVGAN]. I believe the apparent phase distortion in the supplementary files might have been caused by GLA.\n\nThere is a rough comparison on the quality of audio reconstruction pipeline in a recent paper [SpecMaskGIT], I hope it could be useful for the improvement of MDSGen.\n\nI strongly recommend the authors to consider audio-specified reconstruction pipelines for improved audio quality. Even in audio-visual generation community, we can see the usage of such audio VAE for excellent audio quality, e.g., [VisualEchoes]\n### 2. Invalid claims on the result\nBecause the audio quality is far from the baseline in audio generation community, it is improper to claim that MDSGen is better in \"audio-video alignment\".\n\nI believe, the audio-video alignment can be evaluated only when the audio quality is sufficiently good. Given the current audio quality, I don't think the model is ready for further evaluation.\n## Minor issues \n### 1. Evaluation metrics\nThe FID used in this paper comes from the implementation of SpecVQGAN, a pioneer of audio generation. However, the FAD implementation ([AudioLDM],[FAD_github]) has been more widely accepted in audio community. Evaluating with the widely adopted FAD metric can also help the readers to compre the audio quality with other audio generation models.\n### 2. Insufficient ablation study\nMDSGen trains a learnable module to reduce the video feature sequence into a single vector. From Figure 6, we can observe that the learned weights are quite evenly distributed (except the beginning and ending frames).\n\nThe observation posts a question: How much improvement can the learnable reducer bring compared to a naive average pooling?\n\n[StableAudioOpen]: https://arxiv.org/abs/2407.14358\n[MakeAnAudio2]: https://arxiv.org/abs/2305.18474\n[AudioLDM]: https://audioldm.github.io/\n[SpecMaskGIT]: https://arxiv.org/abs/2406.17672\n[DAC]: https://github.com/descriptinc/descript-audio-codec\n[HiFiGAN]: https://github.com/jik876/hifi-gan\n[UnivNet]: https://github.com/rishikksh20/UnivNet-pytorch\n[BigVGAN]: https://github.com/NVIDIA/BigVGAN\n[VisualEchoes]: https://arxiv.org/abs/2405.14598\n[AudioLDMEval]: https://github.com/haoheliu/audioldm_eval\n[FAD_github]: https://github.com/gudgud96/frechet-audio-distance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uPpLzmUOfa", "reviewer": "Reviewer_mkFe", "summary": "The paper presents MDSGen, an efficient framework for vision-guided sound generation that minimizes model size, memory usage, and inference time. Key innovations include a temporal-aware masking strategy to enhance alignment accuracy and a redundant feature removal module to filter unnecessary video information. Using a lightweight masked diffusion transformer, MDSGen outperforms larger Unet-based models on VGGSound and Flickr-SoundNet, achieving high synchronization and alignment with significantly reduced computational costs.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper introduces a novel framework for video-to-audio sound generation that effectively combines a temporal-aware masking strategy with a redundant feature removal module. \n\nMDSGen demonstrates significant improvements in model efficiency by using a smaller masked diffusion transformer architecture. The framework achieves high alignment accuracy on benchmark datasets with a fraction of the parameters, memory usage, and inference time compared to baselines. \n\nThe paper provides a structured explanation of MDSGen’s architecture and mechanisms, including the Temporal-Awareness Masking (TAM) and the Reducer module for filtering out redundant features. Extensive experimental results on VGGSound and Flickr-SoundNet datasets clearly validate the method’s effectiveness, with MDSGen achieving superior performance across alignment accuracy and efficiency metrics.", "weaknesses": "1. Lack of Novelty and Contribution: The paper presents the primary contributions are the Temporal-Awareness Masking (TAM) strategy and the visual Reducer module. However, masking strategies have been widely explored in audio generation research, as seen in works like [1, 2], and the specific concept of Temporal-Awareness Masking has been studied in [3, 4]. The visual Reducer module, primarily a 1x1 convolutional layer (line 181), lacks detailed design innovations, which limits its distinctiveness and impact.\n\n2. Insufficient Exploration of Design Choices: For video-to-audio generation, the choice of video encoder plays a crucial role in understanding the video content. Clarification on the selection of CAVP as the video encoder would add valuable insight. Additionally, the paper could explore using more video encoders, such as CLIP [5], VideoMAE [6], ViVit [7], and TAM [8], which could enrich the technical depth of the proposed method.\n\n3. Presentation and Writing:\nSome claims in the paper lack supporting evidence, such as the statements in lines 183-185 that the proposed method “minimizes redundant features that could lead to overfitting” and in line 224 that setting N_2 = 4 “gives better performance for audio data.” These points would benefit from empirical support to substantiate their validity.\n\n4. Supplementary Material: The quality of generated audio samples in the supplementary material raises concerns regarding the overall quality of results produced by the proposed method, which may affect its effectiveness and appeal.\n\n[1]Pascual S, Yeh C, Tsiamas I, et al. Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity[J]. arXiv preprint arXiv:2407.10387, 2024.\n\n[2]Borsos Z, Sharifi M, Vincent D, et al. Soundstorm: Efficient parallel audio generation[J]. arXiv preprint arXiv:2305.09636, 2023.\n\n[3]Bai H, Zheng R, Chen J, et al. A $^ 3$ T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing[C]//International Conference on Machine Learning. PMLR, 2022: 1399-1411.\n\n[4]Garcia H F, Seetharaman P, Kumar R, et al. Vampnet: Music generation via masked acoustic token modeling[J]. arXiv preprint arXiv:2307.04686, 2023.\n\n[5]Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. PMLR, 2021: 8748-8763.\n\n[6]Tong Z, Song Y, Wang J, et al. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training[J]. Advances in neural information processing systems, 2022, 35: 10078-10093.\n\n[7]Arnab A, Dehghani M, Heigold G, et al. Vivit: A video vision transformer[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 6836-6846.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ftu0Ta8xhq", "reviewer": "Reviewer_Hij3", "summary": "The paper introduces MDSGen, an innovative framework designed for vision-guided open-domain sound generation with a focus on optimizing model parameter size, memory consumption, and inference speed. It features two major innovations: a redundant video feature removal reducer and a temporal-aware masking strategy. Utilizing denoising masked diffusion transformers, MDSGen achieves efficient sound generation without the need for pre-trained diffusion models. On the VGGSound dataset, the smallest MDSGen model demonstrates a 97.9% alignment accuracy while using fewer parameters, consuming less memory, and performing faster inference compared to the current state-of-the-art models. The results underscore the scalability and effectiveness of this approach.", "questions": "1. The reducer is designed to have fixed weights, serving as a weighted average of all the frames. However, the sound events of different videos are distinct, so why not adopt a dynamic weighted average strategy, e.g., attention pooling?\n2. What is the exact implementation of the model? How is the visual conditioning (i.e., $p(x|v)$) implemented?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea of compressing visual representations into one single vector is bold and intriguing, which reduces significant computing pressure on the DiT side.\n2. The proposed TAM strategy is interesting and makes sense. Previous works about masking audio representations, such as AudioMAE, have drawn conclusions that the unstructured masking strategy is superior, which contradicts the conclusion in this paper. I believe this paper brings more insights into this topic. \n3. The authors have conducted tons of ablation experiments to support their model design and parameter decision, making the conclusions plausible.", "weaknesses": "1. The major concern is about the modeling of audio representations, as I am familiar with this field. I believe that Mel spectrum is more of a 1D feature rather than 2D, because the spectrum does not satisfy translational invariance (if a formant chunk in a spectrum is moved from the bottom left to the top right, the semantics of the sound are likely to be completely destroyed), and the frequency domain and time domain cannot simply be simulated by spatial coordinates. A relevant observation in this paper is that a complete random masking strategy is underperformed by the temporal-aware masking strategy. Therefore, I believe that considering Mel spectrograms as gray-scale images and modeling them using 2D VAE pretrained with real images is suboptimal, which further prevents modeling sounds with varying lengths. There are already approaches that model audio using 1D VAE, such as Make-an-audio 2. So can the authors provide justifications for choosing 2D rather than 1D? In my view, choosing 1D combined with the TAM strategy could form a more compelling motivation.\n2. The idea of compressing visual representations into one single vector is intriguing. However, I don't understand why this could work. How does one single vector provide accurate information about temporal position? I believe Diff-foley works because it adopts a sequence-to-sequence cross-attention mechanism, which provides rough sequential and positional information for the audio to follow. Could the author provide further analysis and discussion on this point? For example, analyzing the components related to temporal position within that vector, or the relation of the learned weights of reducer between key frames of videos. \n3. Similar concern: the learned weights of reducer seem to be focused more on the head and tail frames of videos. Does this imply that the reducer is more focused on global video information? How can it be determined that it is capable of extracting local positional information?\n4. The alignment classifier proposed in Diff-foley only reaches 90% accuracy on their test set. However, the best performance in this paper reaches 98+. How could this happen? Is the classifier involved during the training process?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G44eTNSilb", "reviewer": "Reviewer_mmH9", "summary": "This paper presents MDSGen, an efficient and compact framework for generating open-domain sounds guided by visual input. MDSGen uses masked diffusion transformers instead of the usual U-Net architectures, optimizing for faster speed, lower memory use, and parameter efficiency. Key features include a module to remove redundant video data and a Temporal-Awareness Masking (TAM) strategy, both aimed at efficient, high-quality audio-visual alignment. Tests on the VGGSound and Flickr-SoundNet datasets show that MDSGen achieves strong alignment accuracy with much lower computational demands than larger models.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. MDSGen achieves strong results with a small model size, making it useful for real-time applications. Compared to larger models, it is faster and more memory-efficient.\n2. TAM is an interesting approach that focuses on time-based information in audio, aiming to improve alignment by using masking based on temporal patterns rather than spatial patterns (commonly used for images).\n3. The paper provides extensive experiments with detailed comparisons against other models. Ablations for each key component further clarify the model’s design choices.", "weaknesses": "1. I question the decision to use an image-trained VAE (from Stable Diffusion) rather than an audio-specific VAE, such as those in AudioLDM [1]. An audio-dedicated VAE could better capture the temporal and spectral nuances inherent to sound, which are often lost when treating audio as an image. Relying on an image-based VAE reduces the model’s potential to fully leverage audio-specific features and may affect TAM’s performance.\n2. The authors highlight channel selection within the RGB output as a means of optimizing the final mel-spectrogram. While using the G channel showed marginal improvements, I question if relying on such RGB channel selection can sufficiently address the nuances of audio spectrogram representation (similar to 1). A more audio-specific solution that doesn’t require treating spectrograms as RGB images would likely be more consistent with the needs of audio data, as these channels are meant for pixels, not spectral data.\n3. I would suggest that the authors conduct a human perceptual study to better assess audio quality. Relying solely on quantitative metrics may not fully capture perceptual quality, as these measures can sometimes be unreliable.\n4. In Section 5.4, various masking strategies are explored, and TAM shows a clear improvement over random masking and FAM. However, the reasons for TAM’s superiority are not fully explained. It would be beneficial to discuss why TAM outperforms FAM in this context, particularly since FAM is intuitively suitable for audio data.\n5. I noticed a missing citation for SpecAugment [2] and AudioMAE [3], a masking approach relevant to TAM proposed here.\n\nReferences\n\n[1] Liu et al. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models.\n\n[2] Park et al. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition.\n\n[3] Huang et al. Masked Autoencoders that Listen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)