MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation

## Abstract

Abstract We introduce MDSGen , a novel framework for vision-guided open-domain sound generation optimized for model parameter size, memory consumption, and inference speed. This framework incorporates two key innovations: (1) a redundant video feature removal module that filters out unnecessary visual information, and (2) a temporal-aware masking strategy that leverages temporal context for enhanced audio generation accuracy. In contrast to existing resource-heavy Unet-based models, MDSGen employs denoising masked diffusion transformers, facilitating efficient generation without reliance on pre-trained diffusion models. Evaluated on the benchmark VGGSound dataset, our smallest model (5M parameters) achieves 97.9 97.9 97.9 97.9 % alignment accuracy, using 172 × 172\times 172 × fewer parameters, 371 371 371 371 % less memory, and offering 36 × 36\times 36 × faster inference than the current 860M-parameter state-of-the-art model ( 93.9 93.9 93.9 93.9 % accuracy). The larger model (131M parameters) reaches nearly 99 99 99 99 % accuracy while requiring 6.5 × 6.5\times 6.5 × fewer parameters. These results highlight the scalability and effectiveness of our approach.

## 1 Introduction

Figure: Figure 1: Aligment Score. Comparison with SOTA audio generation methods on the VGGSound test set. The diameter of each circle represents the memory usage during inference.

Vision-guided audio generation has gained significant attention due to its crucial role in Foley sound synthesis for the video and film production industry (Ament, ). This paper focuses on Video-to-Audio (V2A) generation, a key task not only for adding realistic sound to silent videos created by emerging text-to-video models (Blattmann et al., ; Khachatryan et al., ; Huang et al., ; Ouyang et al., ) but also for enhancing practical applications in professional video production. Sound generation is essential for creating immersive experiences and achieving seamless audio-visual synchronization. However, achieving both semantic alignment and temporal synchronization in V2A remains a significant challenge. Previous approaches, such as GAN-based methods (Chen et al., ) and Transformer-based autoregressive models (Iashin & Rahtu, ), have struggled with synchronizing audio to content while maintaining relevance. Diff-Foley (Luo et al., ) improved this by employing contrastive learning for video-audio alignment and leveraging diffusion models, achieving impressive sound quality. Other methods like See and Hear (Xing et al., ) and FoleyCrater (Zhang et al., ) utilize large pre-trained models for high-quality audio generation. However, these models rely on hundreds of millions of parameters. In contrast, our work demonstrates that a much smaller model can deliver high performance (see Fig. 1). Most existing approaches rely on Unet architectures, which present scalability limitations. Additionally, current methods often use video features that include redundant information.
In contrast, we propose MDSGen, a novel framework for open-domain sound synthesis based on a pure Transformer architecture. MDSGen incorporates a temporal-aware masking scheme and a redundant feature removal module, enabling it to achieve superior performance while being significantly more efficient. Further analysis highlights the effectiveness of our approach with compelling evidence of its advantages. Our contributions are as follows:

- •
We introduce a simple, lightweight, and efficient framework for open-domain sound generation using masked diffusion transformers, delivering high performance.
- •
Our approach implements Temporal-Awareness Masking (TAM), specifically designed for audio modality, in contrast to spatial-aware masking of the existing work, leading to more effective learning.
- •
We identify inefficiencies in the existing approach that fail to remove redundant video features. Our Reducer module learns to selectively resolve these redundancies, producing more refined features for improved audio generation.
- •
We validate our method on the benchmark datasets VGGSound and Flickr-SoundNet, surpassing state-of-the-art approaches across multiple metrics, with particularly significant improvements in alignment accuracy and efficiency, specifically in model parameters, memory consumption, and inference speed.

## 2 Related works

### 2.1 Open-domain Sound Generation

Auto-regressive Transformer-based Approach.
Key works in this area include SpecVQGAN (Iashin & Rahtu, ), which uses a cross-modal Transformer to generate sounds from video tokens auto-regressively, and Im2Wav (Sheffer & Adi, ), which conditions audio token generation on CLIP features. However, these methods suffer from slow inference speeds due to their sequential generation process and limited vision-audio alignment, negatively impacting performance.

Diffusion-based Approach.
To overcome these limitations, Diff-Foley (Luo et al., ) introduced a two-stage method that enhances semantic and temporal alignment via contrastive pre-training on aligned video-audio pairs, followed by latent diffusion for improved inference efficiency. Similarly, See and Hear (Xing et al., ) utilizes ImageBind (Girdhar et al., ) and AudioLDM (Liu et al., ) for various audio tasks, while FoleyCrafter (Zhang et al., ) combines a pre-trained text-to-audio model with a ControlNet-style module (Zhang et al., ) for high-quality, synchronized Foley generation. Although these diffusion approaches show promise, they often rely on large models with hundreds of millions of parameters and predominantly utilize U-Net architectures, leaving the potential of transformer-based architectures largely untapped.

Our proposed method leverages diffusion transformers (Peebles & Xie, ) and masking techniques for efficient learning. It also addresses the issue of redundant video features in Diff-Foley (Luo et al., ), which hinders further improvements in audio generation.

### 2.2 Latent Masked Diffusion Transformers

The Denoising Diffusion Transformer (DiT) introduced by Peebles & Xie () replaces the traditional U-Net with a fully transformer-based architecture for latent diffusion, demonstrating remarkable performance in large-scale image generation on ImageNet. Following this, Gao et al. () proposed the Masked Diffusion Transformer (MDT), which enhances ImageNet generation through spatial context-aware masking. Inspired by MDT, Pham et al. () developed X-MDPT, using cross-view masking to establish correspondence between pose and reference images for improved person image generation. Additionally, MDT-A2G (Mao et al., ) explored masked diffusion transformers for gesture generation, while QA-MDT (Li et al., ) adapted this technique for music generation.

In contrast to these works, we focus on lightweight masked diffusion models for video-guided audio generation, introducing temporal-aware masking for audio and a design that removes redundant video features to enhance generation effectiveness.

## 3 Method

We aim to develop a simple yet effective framework for vision-guided sound generation using transformers, addressing the limitations of existing approaches that rely on traditional U-Net architectures, which are less scalable and efficient. Our framework, illustrated in Fig. 2, consists of a novel Vision Extractor with a learnable Reducer that captures essential information from video input to generate a concise conditional output for the denoising diffusion process. This is followed by a Denoising Diffusion Transformer that maps Gaussian noise to sound distributions based on the extracted visual features. Additionally, we introduce a Masked Temporal-Aware Network (MTANet) that serves as a strong regularization mechanism for the transformer, enhancing overall performance. Finally, we incorporate Channel Selection for mel-spectrograms that further improves results.

Figure: Figure 2: Overview of the proposed highly-efficient MDSGen framework, utilizing denoising masked diffusion transformers to efficiently learn video-conditional distributions for audio generation, replacing traditional Unet-based methods. The fire icon represents trainable modules, while the ‘locked’ icon indicates modules that remain frozen during training. Green arrows $\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\rightarrow$ denote branches used only during training, blue arrows $\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\rightarrow$ are for only inference, and black arrows $\rightarrow$ are used in both training and inference.
Refer to caption: x2.png

### 3.1 Denoising Diffusion Transformer

We adopt the DiT backbone introduced by Peebles & Xie (), which leverages transformers with self-attention during the denoising diffusion training process. Given an audio signal $\mathbf{A}\in\mathbb{R}^{L_{\mathbf{A}}}$ of length $L_{\mathbf{A}}$ and a silent video $\mathbf{V}\in\mathbb{R}^{L_{V}\times 3\times 224\times 224}$ of length $L_{V}$, the audio is first transformed into a mel-spectrogram $\mathbf{X}\in\mathbb{R}^{128\times 512}$, while the video is encoded into $\mathbf{v}\in\mathbb{R}^{L_{V}\times 512}$ and further reduced to $\vec{v}\in\mathbb{R}^{1\times D}$. The mel-spectrogram is repeated across 3 channels, forming $\mathbf{X^{\prime}}\in\mathbb{R}^{3\times 128\times 512}$, and passed through the VAE from Stable Diffusion (Rombach et al., ; Luo et al., ) to obtain a latent embedding $\mathbf{x}\in\mathbb{R}^{4\times 16\times 64}$. This latent representation is patched and tokenized into image tokens using a patch size of $p=2$ (DiT’s default), resulting in $\mathbf{x}^{\prime}\in\mathbb{R}^{256\times D}$, where $L_{\mathbf{x^{\prime}}}=256$ and $D=768$ for the B-size model. These tokens are then fed into the $N=N_{1}+N_{2}$ self-attention layers of the Transformer to predict the noise $\epsilon$ added to the latent $\mathbf{x}$. Conditioned on the video encoding $\vec{v}$, the Transformer model $\phi$ learns the distribution $p_{\phi}(\mathbf{x}|\vec{v})$. During training, Gaussian noise $\epsilon\in\mathcal{N}(0,\mathbf{I})$ is added to the latent $\mathbf{x}$ to generate $\mathbf{x}_{t}$ at timestep $t\in[1,T]$. The overall training objective is:

$$ $\mathcal{L}_{\sum}=\mathbb{E}_{\mathbf{x},\vec{v},\epsilon}\|\epsilon-\epsilon _{\phi}(\mathbf{x_{t}},\vec{v},t)\|^{2}+\lambda\mathbb{E}_{\mathbf{x},\vec{v}, \epsilon}\|\epsilon-\epsilon_{\phi}(\mathcal{M}_{\phi}(\mathbf{x_{t}}),\vec{v} ,t)\|^{2}.$ (1) $$

Here, $\lambda$ is the balance factor between the standard denoising diffusion loss (the first term in Eq. 1) and the masking loss (the second term), with $\lambda=1.0$ for optimal performance. The masking function $\mathcal{M}_{\phi}$, which includes the MTANet introduced later, applies temporal-aware masking.
During inference, given a silent video, the model starts from Gaussian noise (no audio provided), and the predicted latent $\hat{\mathbf{x}}\in\mathbb{R}^{4\times 16\times 64}$ is iteratively denoised and decoded by the VAE decoder to recover the mel-spectrogram $\widehat{\mathbf{X}}_{RGB}\in\mathbb{R}^{3\times 128\times 512}$.
Channel selection refines this into $\widehat{\mathbf{X}}\in\mathbb{R}^{128\times 512}$, and the final waveform is reconstructed from the mel-spectrogram using the Griffin-Lim algorithm (Griffin & Lim, ), consistent with previous work (Luo et al., ) for fair comparison.

### 3.2 Vision Extractor

The second key component of our framework is the Vision Extractor with a learnable Reducer network that aligns video features with audio while condensing temporal information. We leverage the pre-trained CAVP model from (Luo et al., ), which was trained on Audioset using contrastive loss to extract video features aligned with audio. However, we identified that the CAVP features contain redundancies that could negatively impact generation quality. Diff-Foley (Luo et al., ) linearly maps original feature dimensions from $\mathbf{v}\in\mathbb{R}^{L_{V}\times 512}$ to $\mathbf{v}\in\mathbb{R}^{L_{V}\times 768}$ and retains this full dimensionality during the latent diffusion process via cross-attention, with $L_{V}$ is the video feature length.

Our approach reduces the dimensionality to $\vec{v}\in\mathbb{R}^{1\times 768}$, offering more concise and efficient information for denoising diffusion. Specifically, we project the encoded features $L_{V}\times 512$ through a multi-layer perceptron (MLP) into the transformer feature space ($L_{V}\times 768$ for the size B-model). These features are then passed through a reducer module, an $1\times 1$ convolutional layer, which condenses the high-dimensional features into a lightweight form $\vec{v}\in\mathbb{R}^{1\times 768}$. This compact representation is integrated into the denoising diffusion process through Adaptive LayerNorm (AdaLN) modulation. Unlike Diff-Foley, which retains all $L_{V}$ video channels representing $L_{V}$ video frames for cross-attention within the Stable Diffusion U-Net backbone, our method minimizes redundant features that could lead to overfitting. Our analysis shows that the $L_{V}$-frame input features share over $90$% similarity, indicating considerable redundancies, which we further address in our ablation study.

Intuition.
Our simple yet effective reducer design treats the temporal dimension of video ($L_{V}$ = 32) as feature channels and performs a non-linear projection to a single channel, functioning similarly to channel attention by weighting important channels and summing them. This acts as a bottleneck that distills and distributes video temporal information across the 768 dimensions, aligning better with each audio token, which also has a 768-dimensional space. This approach, combined with the transformer network, significantly improves alignment accuracy up to approximately $99$%.

### 3.3 Audio Masked Temporal-Aware Network

Figure: Figure 3: Audio Masking Strategies. Here, the red square is the learnable mask token.
Refer to caption: x3.png

Thirdly, we introduce a novel technique that exploits the sound information’s natural characteristic: the temporal sense. The existing masking, Spatial-Aware Mask (SAM) proposed by MDT (Gao et al., ) is designed for image data to learn the spatial context within the image. But here, in the audio data (represented by mel-spectrogram with 2D data), the SAM masking method yields a sub-optimal solution because it cannot model the exact nature of temporal meaning in the audio data.

To overcome this limitation, we propose the Temporal-Aware Mask (TAM) strategy instead of SAM, which tries to mask the whole set of tokens along the temporal dimension. As shown in the ablation section, this novel masking helps significantly boost performance in all metrics compared to the existing method SAM specified for image data as shown in Fig. 3. Interestingly, despite this simple strategy, it can help the denoising transformer models learn to generate audio much better than random masks as used in existing MDT designed for image data.

During training, we mask with $\eta_{\mathbf{m}}\%$ of temporal tokens, we feed only the visible tokens to the $N_{1}$ blocks of the transformer: $\mathbf{o}_{1}=\text{Encoder}_{N_{1}}(\mathbf{x}^{\prime}\odot(1-\mathbf{m}))$ ($\mathbf{m}$ is a mask matrix and $\odot$ denotes element-wise multiplication) and in the MTANet we concatenate the resulting tokens with the learnable mask tokens $\mathbf{M}$ and feed into a single block of predictor (referred to as side-interpolator in MDT) to achieve the full latent tokens before feeding into the final $N_{2}$ self-attention blocks of the transformer: $\mathbf{o}_{2}=\text{Decoder}_{N_{2}}(\text{cat}(\mathbf{o}_{1},\mathcal{M}_{
\phi}(\mathbf{M}*\mathbf{x}^{\prime}\odot(\mathbf{m}))))$.
Here we find that different from the design for ImageNet with $N_{2}=2$ in the default MDT, we use $N_{2}=4$ which gives a better performance for audio data. After training, the masked modeling branch is discarded, maintaining only its positional embedding for inference.

### 3.4 Classifier-Free Guidance

We adopt the dynamic Classifier-Free Guidance (CFG) method from previous works on masked diffusion transformer models (Gao et al., ) used in ImageNet. However, unlike image tasks, we find that in audio generation, the optimal CFG value is between 5 and 6, with a power scale of 0.01. Notably, Classifier-Guidance (CG) has been shown to significantly boost performance in Diff-Foley (Luo et al., ), where their method relies heavily on CG for optimal results. In contrast, our approach without CG surpasses Diff-Foley (CFG+CG) across multiple metrics. While incorporating CG improves our framework in terms of alignment accuracy and KL, it does not enhance other metrics. Hence, for simplicity, we omit CG in most of our experiments.

### 3.5 Mel-Spectrogram Channel Selection

We discovered that the audio extracted from the RGB output channels of the VAE decoder exhibits different qualities. Since the mel-spectrogram is 2D, we treat it as a single gray channel and duplicate it into three channels before feeding it into the VAE, which is pre-trained to reconstruct RGB images. At the decoding stage, the VAE outputs three channels: $\widehat{\mathbf{X}}_{RGB}\in\mathbb{R}^{3\times 128\times 512}$, with $\widehat{\mathbf{X}}_{RGB}[i,:,:]\in\mathbb{R}^{128\times 512},i\in\{0,1,2\}$ representing the R, G, and B channels.
While Diff-Foley (Luo et al., ) used the first channel (R) as their final output, our empirical tests reveal that the second channel (G) consistently performs better. We further ablated channel selection and found that using the G channel yields superior results compared to the R channel used in prior work.

## 4 Experiments

### 4.1 Dataset and Evaluation Metrics

(i) Dataset.
We evaluate our method on the VGGSound dataset (Chen et al., ), using the original train/test splits with 175k and 15k samples, respectively, and on the Flick-SoundNet dataset Aytar et al. () with 5k test samples. (ii) Metrics. We first use the same metrics as prior work (Luo et al., ), including FID, IS, KL, and Alignment Accuracy, using their provided scripts for Align. Acc. and SpecVQGAN code for FID, IS, and KL. Second, we assess general vision-audio alignment in the Image2Audio task using CIoU and AUC metrics with scripts from (Mo & Morgado, ). For this, we generate pseudo videos by replicating a single input image for all frames. Third, we compare efficiency using parameter count, memory usage, and inference speed.

### 4.2 Implementation Details

All models are trained and tested on a single A100 GPU (80GB) with a batch size of 64 and a learning rate of 5e-4, using the Adan optimizer (Xie et al., ) for faster training. Unlike MDTv2 (Gao et al., ), we skip the macro-style of side interpolator design, as it was ineffective for our task, and instead use a simple self-attention block at decoder layer 4. Video-audio pairs are truncated to 8.2 seconds before encoding, following (Luo et al., ). Our model comes in three main variants: Tiny (5M), Small (33M), and Base (131M), with the Large (460M) variant showing overfitting. We primarily focus on the T, S, and B models.

### 4.3 Main Results

A. VGGSound dataset.
Compared to state-of-the-art approaches, our method significantly outperforms all competitors’ alignment accuracy while being far more efficient regarding parameters and inference speed (Tab. 1). Alignment accuracy, a metric introduced by (Luo et al., ), assesses synchronization and audio-visual relevance using a separate classifier trained to predict real audio-visual pairs. Remarkably, our Transformer-based model, MDSGen-Tiny (5M), trained from scratch, achieves $97.9$% accuracy, surpassing the second-best Diff-Foley (860M), which is $172\times$ larger and depends on a backbone of Stable Diffusion pre-trained on billion image-text pairs.

As shown in Tab. 1, Diff-Foley struggles without a pre-trained backbone, with a significant drop to FID 16.98 and IS 24.91. In contrast, our smallest model, MDSGen-T (5M), trained from scratch, achieves FID 14.18 and IS 37.51, emphasizing the overfitting issues of heavy U-Net-based models compared to our lightweight Transformers. Our larger model, MDSGen-B (131M), achieves state-of-the-art alignment accuracy ($\approx 99$%) and an IS of 57.12 at 800k steps, though longer training leads to overfitting and declines in other metrics.

**Table 1: Benchmark on VGGSound test. Generation quality comparison of different approaches. ${\dagger}$ gets from (Luo et al., ), $\ddagger$ denotes without pre-trained SDv1.4. * denotes results with pre-trained SDv1.4, we reproduce it using the public checkpoint.**
| Method | FID$\downarrow$ | IS$\uparrow$ | KL$\downarrow$ | Align. Acc.$\uparrow$ | Time$\downarrow$ (s) | #Params$\downarrow$ | Cost$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SpecVQGAN (Iashin & Rahtu, )${\dagger}$ | 9.70 | 30.80 | 7.03 | 49.19 | 5.47 | 308M | $61\times$ |
| Im2Wav (Sheffer & Adi, ) ${\dagger}$ | 11.44 | 39.30 | 5.20 | 67.40 | 6.41 | 448M | $90\times$ |
| Diff-Foley (Luo et al., ) ${\dagger}$$\ddagger$ | 16.98 | 24.91 | 6.05 | 92.61 | 0.38 | 860M | $172\times$ |
| Diff-Foley (Luo et al., ) * | 10.55 | 56.67 | 6.49 | 93.92 | 0.36 | 860M | $172\times$ |
| See and Hear (Xing et al., ) | 21.35 | 19.23 | 6.94 | 58.14 | 18.25 | 1099M | $220\times$ |
| FoleyCrafter (Zhang et al., ) | 12.07 | 42.06 | 5.67 | 83.54 | 2.96 | 1252M | $250\times$ |
| MDSGen-T (Ours) 500k | 14.18 | 37.51 | 6.25 | 97.91 | 0.01 | 5M | $1.0\times$ |
| MDSGen-S (Ours) 500k | 12.92 | 44.38 | 6.29 | 98.32 | 0.02 | 33M | $6.6\times$ |
| MDSGen-B (Ours) 500k | 11.19 | 52.77 | 6.27 | 98.55 | 0.05 | 131M | $26.2\times$ |
| MDSGen-B (Ours) 800k | 12.29 | 57.12 | 6.43 | 91.62 | 0.05 | 131M | $26.2\times$ |

**Table 2: Benchmark on Flick-SoundNet test. Comparison of different approaches. ‘Bold’ and “underline” denote the best and second-best, respectively.**
| Method | CIoU$\uparrow$ | AUC$\uparrow$ |
| --- | --- | --- |
| Diff-Foley (Luo et al., ) | 81.02 | 55.19 |
| See and Hear (Xing et al., ) | 81.20 | 55.45 |
| FoleyCrafter (Zhang et al., ) | 81.78 | 55.57 |
| MDSGen-B (Ours) 500k | 82.01 | 55.51 |
| Ground Truth | 83.94 | 63.60 |

B. Flickr_SoundNet dataset.
We use the models trained on VGGSound to test on SoundNet dataset to evaluate its generalization. First, through quantitative metrics in the sound source localization task with Flickr-SoundNet (Aytar et al., ) test set. Second, qualitatively compare the generated audio across different methods. As shown in Tab. 2, our method outperforms other methods on the CIoU metric (82.01%) closer to the ground truth (83.94%), while the AUC remains comparable (around 55.5%). It shows that our generated audio provided better-aligned features with the visual information to localize the sound source. Diff-Foley performs worst, indicating that it is more overfitting. We provide their visualizations in the Appendix.

## 5 Ablation Study

We attribute the strong performance of our models to three key factors. First, the Transformer backbone enables more effective learning of the audio modality compared to existing Unet-based diffusion methods (Diff-Foley, See and Hear, FoleyCrafter). Second, our innovative Reducer module mitigates potential redundancies in the video input. Third, the temporal masking model acts as a robust regularizer, further enhancing the Transformer’s performance. A detailed analysis of these components is provided in the following sections.

### 5.1 Alignment Accuracy: A Confidence Score Perspective

We assess the enhancement of alignment accuracy in our method by analyzing confidence scores from the VGGSound test set, using the output of the sigmoid function from the trained classifier that predicts audio-video alignment. As illustrated in Fig. 4, FoleyCrafter (acc=83.54%) produces many low-confidence samples, indicating misalignment.

In contrast, Diff-Foley (93.9%) achieves a higher proportion of high-confidence scores. Remarkably, our method reaches an accuracy of 98.6%, significantly increasing the number of high-confidence samples and demonstrating superior audio-video alignment compared to other approaches.

### 5.2 Feature Dimensionality Reduction and Redundant Features in CAVP

**Table 3: Dimension Reduction. Compare the original CAVP and Ours’s features.**
| Video Feat. | Cond. Dim. | FID | IS | KL | Align. Acc. |
| --- | --- | --- | --- | --- | --- |
| Original CAVP | $32\times 768$ | 13.55 | 50.12 | 6.38 | 96.18 |
| Reduced (Ours) | $1\times 768$ | 11.19 | 52.77 | 6.27 | 98.55 |

Unlike Diff-Foley, which uses a U-Net-based Stable Diffusion model and incorporates all 32 video frame features for cross-attention, we found that reducing video features from 32 channels to a single channel significantly improves audio generation performance across all metrics (Tab. 3). Diff-Foley’s CAVP encodes video features at $32\times 512$, aligning with the audio’s 32-channel representation (also $32\times 512$) for latent-level alignment via contrastive learning. However, in the second stage, a mismatch arises as the VAE reduces audio dimensions to $4\times 16\times 64$ while video expands to $32\times 768$, leading to redundancy and inefficiencies. Reducing video dimensionality to $1\times 768$ representation acts as a bottleneck, simplifying learning and enhancing alignment, as supported by our results.

**Table 4: Redundant Features. Cosine similarity between the frame’s features.**
| Input | Similarity (CAVP) |
| --- | --- |
| Video | 0.9087 |
| Image | 0.9233 |

Analysis of the CAVP features in Diff-Foley shows significant redundancy, with cosine similarity averaging 0.9087 for real videos and 0.9233 for identical frames (Tab. 4). These high similarity scores suggest that the feature vectors from multiple frames largely overlap, limiting the model’s ability to learn distinctive characteristics essential for effective audio synthesis. This issue worsens when Diff-Foley expands features to a $32\times 768$ dimension, diluting key traits for latent diffusion modeling. In contrast, our approach employs a Reducer module to consolidate the 32 video frame features into a 768-dimensional representation, effectively reducing redundancy and enhancing focus on salient features, which improves alignment accuracy and overall performance in audio generation tasks.

Figure: Figure 4: Confidence Scores. Compared to FoleyCrafter (left) and Diff-Foley (middle), our method (right) produces many more audio samples with higher confidence that align with their corresponding videos on the VGGSound test set ($\sim 15$k samples).
Refer to caption: x4.png

### 5.3 Channel Selection from RGB for Mel-Spectrogram

**Table 5: Channel selection for mel-spectrogram. Gray indicates the default.**
| Channel | FID$\downarrow$ | IS$\uparrow$ | KL$\downarrow$ | Align. Acc.$\uparrow$ (%) |
| --- | --- | --- | --- | --- |
| $R$ ($i$ = 0) | 11.40 | 51.54 | 6.29 | 98.51 |
| $G$ ($i$ = 1) | 11.19 | 52.77 | 6.27 | 98.55 |
| $B$ ($i$ = 2) | 11.23 | 52.32 | 6.27 | 98.56 |
| $\frac{1}{3}\sum_{r\in\{R,G,B\}}r$ | 11.29 | 52.27 | 6.28 | 98.54 |

Each R, G, and B channel exhibits distinct characteristics, as noted in previous research (Xu et al., ). In the VAE encoder, we replicate the gray mel-spectrogram across three identical channels, but the VAE decoder does not enforce channel consistency. Our analysis shows that the RGB output $\widehat{\mathbf{X}}_{RGB}$ retains unique statistical differences across channels (see Fig. 5), influencing their contributions to the final mel-spectrogram and waveform.
In contrast to Diff-Foley, which uses only the R channel ($\widehat{\mathbf{X}}_{RGB}[0,:,:]$) for the final mel-spectrogram, we find the G channel ($\widehat{\mathbf{X}}_{RGB}[1,:,:]$) to be optimal (see Tab. 5). Fig. 5 shows that the histograms of the three VAE output channels display significant differences, with the G and B channels aligning closely with the ground truth distribution.

Notably, while the resulting mel-spectrograms (right figures) seem visually indistinguishable, the histograms highlight their differences. This emphasizes the importance of considering each channel’s statistics in generating the final mel-spectrogram $\widehat{\mathbf{X}}$, with further comparisons available in the Appendix.

Figure: (a)
Refer to caption: x5.png

### 5.4 Masking Diffusion Strategies

We explore various audio-masking methods in diffusion transformers (Fig. 3) and compare them to traditional image-based techniques like SAM used with ImageNet (Gao et al., ). Our findings reveal that audio data behaves differently from images (SAM), with incorporating temporal awareness (TAM) into the masking task significantly boosting performance across all metrics (Tab. 6), including a 4-point IS score increase from 48.66 to 52.77. Using a diffusion transformer without masking (DiT) leads to suboptimal results across all metrics, highlighting the critical role of masking in model learning.

**Table 6: Masking Strategy for Audio Generation. Comparison of different ways to train diffusion transformer-based models. Masking on temporal audio gives the best performance.**
| Masking Method | Mask | Temporal | FID$\downarrow$ | IS$\uparrow$ | KL$\downarrow$ | Align. Acc.$\uparrow$ (%) |
| --- | --- | --- | --- | --- | --- | --- |
| DiT (Peebles & Xie, ) | $\times$ | $\times$ | 14.55 | 46.11 | 6.51 | 97.12 |
| Random, SAM (Gao et al., ) | ✓ | $\times$ | 12.44 | 48.66 | 6.30 | 98.15 |
| Frequency, FAM | ✓ | $\times$ | 12.79 | 46.33 | 6.41 | 97.58 |
| Temporal, TAM (Ours) | ✓ | ✓ | 11.19 | 52.77 | 6.27 | 98.55 |

### 5.5 Learned Weights of Reducer

We analyzed how our models allocate attention across video frames by visualizing their learned magnitude weights (Fig. 6). The upper figure shows that our model applies varying attention levels, with larger models exhibiting higher weights and more distinct differences. After softmax normalization (bottom figure), consistent trends are observed for various channels, though not all, with model B focusing more on channels 1, 2, and 32. These findings demonstrate that the Reducer effectively captures key features, selectively updating weights to prioritize relevant ones for audio generation.

Figure: Figure 6: Learned Weights of Reducer. Comparison of our three models.
Refer to caption: x8.png

### 5.6 Scalability

We evaluate the scalability of MDSGen, the first to explore ViT-based masked diffusion models for vision-guided audio generation. Results in Tab. 7 show that increasing the model size from T to B improves all metrics. However, further scaling to the L model leads to a performance drop, indicating potential overfitting at larger sizes.

**Table 7: Scalability. We ablate four variants Tiny (T), Small (S), Base (B), and Large (L).**
| Model Config. | FID$\downarrow$ | IS$\uparrow$ | KL$\downarrow$ | Align. Acc.$\uparrow$ | #Params | #Layers | Dim. | #Heads |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MDSGen-T | 13.93 | 39.24 | 6.17 | 97.9 | 5M | 12 | 192 | 3 |
| MDSGen-S | 12.92 | 44.38 | 6.29 | 98.3 | 33M | 12 | 384 | 6 |
| MDSGen-B | 11.19 | 52.77 | 6.27 | 98.6 | 131M | 12 | 768 | 12 |
| MDSGen-L | 12.68 | 49.53 | 6.56 | 97.9 | 461M | 24 | 1024 | 16 |

**Table 8: CFG and CG. We examine the effect of classifier-free guidance and classifier guidance. Results are shown with model MDSGen-B. Gray indicates the default.**
| Setup | FID$\downarrow$ | IS$\uparrow$ | KL$\downarrow$ | Align. Acc.$\uparrow$ |
| --- | --- | --- | --- | --- |
| No Guidance | 16.50 | 23.54 | 6.85 | 84.1 |
| CFG | 11.19 | 52.77 | 6.27 | 98.6 |
| CFG + CG | 11.25 | 51.48 | 6.24 | 98.8 |

### 5.7 Sampling Tools

Sampling Method.
We employ DPM-Solver (Lu et al., ) with 25 steps for sampling during inference. We find that increasing from 25 to 50 steps with dynamic classifier-free guidance (Gao et al., ) can slightly improve the performance. We used CFG = 5 and power scaling $\alpha=0.01$ for the optimal setting. Classifier-Guidance (CG). We found that combining CFG and CG slightly improves alignment accuracy and KL, consistent with (Luo et al., ), but has no impact on other metrics (see Tab. 8), which differs from their findings. A thorough investigation of network architecture and additional datasets is needed to assess the complementary effects of CFG and CG fully, but this is beyond the scope of our work.

**Table 9: Efficiency Comparison. Our approach is simple and highly efficient across all metrics, with superior alignment accuracy compared to existing methods.**
| Method | Time$\downarrow$ | Mem. Use$\downarrow$ | #Params$\downarrow$ | Align. Acc.$\uparrow$ |
| --- | --- | --- | --- | --- |
| Im2Wav (Sheffer & Adi, ) | 6.41s | 1684M | 448M | 67.4 |
| See and Hear (Xing et al., ) | 18.25s | 14466M | 1280M | 58.1 |
| FoleyCrafter (Zhang et al., ) | 2.96s | 12908M | 1252M | 83.5 |
| Diff-Foley (Luo et al., ) | 0.36s | 5228M | 860M | 93.9 |
| MDSGen-T (Ours) | 0.01s | 1406M | 5M | 97.9 |
| MDSGen-S (Ours) | 0.02s | 1508M | 33M | 98.3 |
| MDSGen-B (Ours) | 0.05s | 2132M | 131M | 98.6 |

### 5.8 Compare the efficiency

We evaluated inference time, parameter count, and memory usage on a single A100 GPU (80GB) with batch size 1. Tab. 9 shows our method is significantly faster, uses fewer parameters, and consumes less memory than existing methods. Specifically, Diff-Foley (860M) achieves 93.9% alignment accuracy with a 0.36s inference time, while our MDSGen-T (5M) reaches 97.9% in just 0.01s, $36\times$ faster and 371% more memory efficient. Our larger model MDSGen-B (131M) improves accuracy to 98.6%, still $7.2\times$ faster and 245% more memory efficient than Diff-Foley. Compared to FoleyCrafter and See and Hear, MDSGen-T is $296\times$ and $1825\times$ faster, respectively, while being $10\times$ more memory efficient. We provide additional details on training efficiency in the Appendix, further emphasizing the remarkable efficiency of our approach.

### 5.9 Visualizations

We visualize different approaches using test video samples from the VGGSound dataset. As shown in Fig. 7, our method generates mel-spectrograms that closely match the ground truth (right figure), with even clearer distinctions observable in the waveform (left figure). Additional demo samples, along with WAV files for convenient listening and their visualizations, are included in the Appendix and supplementary material.

Figure: Figure 7: Waveform and Mel-Spectrogram Comparison. Sample is taken from the test set of the VGGSound dataset. Our model generates sound more closely aligned with the ground truth than existing methods. The video of a woman playing drum by hand, demo file “0NIE-eDk92M_000029.wav” is available in the supplementary material.
Refer to caption: x9.png

## 6 Conclusions

This work presents a novel, scalable, and highly efficient framework for video-guided audio generation. Leveraging Diffusion Transformers, we introduced an innovative masking strategy that enhances the model’s ability to capture temporal dynamics in audio, leading to significant performance gains. To address redundant video features, we introduced a Reducer module to eliminate unnecessary information. Extensive experiments and detailed analyses demonstrate that our model achieves fast training and inference times, uses minimal parameters, and delivers superior performance across multiple metrics, setting a new benchmark in the field.

## 7 Limitations and Future Works

Our method offers fast inference, efficient parameter, and memory usage, and achieves top performance in alignment accuracy and IS score. However, it does have some limitations. First, like other diffusion models, it requires multiple sampling steps during generation. Second, while the VGGSound dataset is suitable for this study, it is not a large-scale dataset, which may not fully exploit the potential of our approach. Third, the current design handles a fixed video length of 8.2 seconds. We hope to incorporate recent advancements in single-step diffusion techniques into our framework in the future. Additionally, while video collection from online sources is increasingly feasible, it can still be time-consuming and storage-intensive, posing challenges for individual researchers. We leave further exploration in this area, along with addressing the fixed-length constraint, for future work.

## Appendix A Appendix

### A.1 Training Cost

We leveraged the pre-trained VAE encoder and decoder from Stable Diffusion (Rombach et al., ), keeping them frozen during training and inference, similar to Diff-Foley. Our training utilized a single A100 GPU (80GB) with a batch size 64. The B-model (131M) is projected to take 4 days for 500k iterations, while the S-model (33M) and T-model (5M) are expected to finish in 3.3 and 2.8 days, respectively.

In comparison, the second-best method, the Diff-Foley approach (860M model) required 8 A100 GPUs with a batch size of 1760, completing 24.4k steps in 60 hours (2.5 days) (Luo et al., ). If scaled to a single A100 GPU, Diff-Foley would need at least 20 days more than a fifth of the time of our method, demonstrating the superior efficiency and significantly lower training costs of our approach (see Tab. 10).

**Table 10: Training Comparison. Estimated for a single A100 GPU training. Our approach is simple and highly efficient compared to the second-best method Diff-Foley which used the heavy backbone of Stable Diffusion with 860M.**
| Method | #Training cost$\downarrow$ | Align. Acc.$\uparrow$ |
| --- | --- | --- |
| Diff-Foley (860M) (Luo et al., ) | 20 days | 93.9 |
| MDSGen-T, 5M (Ours) | 2.8 days | 97.9 |
| MDSGen-S, 33M (Ours) | 3.3 days | 98.3 |
| MDSGen-B, 131M (Ours) | 4.1 days | 98.6 |

### A.2 More Setups

We apply classifier-free guidance during training by randomly setting $\vec{v}$ to zero with a 10% probability. Models are trained for 500k steps to ensure optimal convergence. The exponential moving average of the model weight is set to 0.9999, otherwise, settings are the same as default DiT (Peebles & Xie, ). No video augmentation is used; instead, we pre-extract and save lightweight video features for faster training. We also use a ratio of $\eta_{m}=0.3$ by default for masking the temporal set of tokens. Our code will be made publicly available.

For experiments involving classifier guidance, we utilized the classifier trained by Diff-Foley, adjusting the optimal CG value to 2.0, compared to 50 in their framework. To evaluate alignment accuracy, we used their trained classifier to assess our generated audio. We also reproduced Diff-Foley’s performance using their published checkpoint, with results closely matching their reported metrics.

The slight variation may stem from differences in the VGGSound test set, as we download videos from several months to one year after their experiments, during which some original YouTube links may have been removed, causing potential mismatches.

### A.3 Comparing more Visualizations

Gradcam Visualization of localization on VGGSound and Flickr-SoundNet datasets.

We used the generated audio to perform the sound source localization on each frame of the video and image using the pre-trained model of EZ-VSL (Mo & Morgado, ). As shown in Fig. 8 and Fig. 9, our method provides a more accurate attention map.

Figure: Figure 8: Attention map Flickr-SoundNet dataset. Our best model (MDSGen-B) generated the sound that contain information that help localize the sound source more accurately compared to existing approaches.
Refer to caption: x10.png

More generated audio comparison with state-of-the-art approaches on VGGSound test set.

We also provide more samples in the supplementary and their visualizations in Fig. 10, Fig. 11, Fig. 13, Fig. 14, and Fig. 12. As shown in these figures and listened to by authors, our generated audio is much more reasonable than others. We refer readers to examine the quality of generated audio in the submitted supplementary materials.

### A.4 RGB channel selection for Mel-Spectrogram

We provide additional statistics of various generated audio samples, highlighting the differing characteristics of the VAE decoder outputs in Fig. 15 and Fig. 16. As shown, although the VAE encoder input consists of three identical channels, the generated outputs display distinct distributions across each channel (left figures), even though these differences are imperceptible to the human eye (right figures). This behavior stems from the fact that the VAE encoder and decoder in Stable Diffusion (Rombach et al., ) are trained exclusively for image data, where the R, G, and B channels inherently carry different information.

Because this model is applied directly to audio data without adaptation, there is no constraint ensuring the R, G, and B channels remain identical in the generated audio. Developing a method to adaptively select or combine these channels when constructing the final Mel-spectrogram could be a promising avenue for improving the quality of the generated audio.

### A.5 Mask Ratio Ablation

Tab. 11 shows that while a higher masking ratio maintains high alignment accuracy, it leads to declines in other metrics. This occurs because the transformer models prioritize audio token reconstruction over the primary generation task, resulting in worsened FID, IS, and KL scores.

**Table 11: Masking Strategy for Audio Synthesis. Comparison of different ways to train diffusion transformer-based models. Masking on temporal audio gives the best performance.**
| Masking Ratio | FID$\downarrow$ | IS$\uparrow$ | KL$\downarrow$ | Align. Acc.$\uparrow$ (%) |
| --- | --- | --- | --- | --- |
| 70% | 12.85 | 44.42 | 6.38 | 97.86 |
| 50% | 12.39 | 46.77 | 6.38 | 98.43 |
| 30% | 11.19 | 52.77 | 6.27 | 98.55 |

### A.6 More Directions

Our approach can inspire applications of masked diffusion models across various domains such as object detection (Vu et al., ), video question answering (Kim et al., ), image super-resolution (Niu et al., ; ), and speech processing. These models can also be combined with self-supervised learning techniques (He et al., ; Pham et al., ; ; ; ) to enhance representation learning for diverse generative tasks. With the potential of diffusion transformers for conditional learning, we anticipate further exploration of their capabilities in speech processing (Jung et al., ; ; Trung & Yoo, ), data augmentation (Lee et al., ), VQA (Kim et al., ), visual detection (Vu et al., ), and super-resolution (Niu et al., ; ; ).

Figure: Figure 9: Attention map VGGSound dataset. Our best model (MDSGen-B) generated the sound that contain information that help localize the sound source more accurately compared to existing approaches.
Refer to caption: x11.png

Figure: Figure 10: The video of a man playing guitar solo. Our best model (MDSGen-B) generated a sound that is closer to GT compared to existing approaches. We refer the reader to the listen file provided in the supplementary for comparison. File “-lPXTBXa0tE_000030.wav”.
Refer to caption: x12.png

Figure: Figure 11: The video of a woman playing drum by hand. Our best model (MDSGen-B) generated a sound that is closer to GT compared to existing approaches. We refer the reader to the listen file provided in the supplementary for comparison. File “0NIE-eDk92M_000029.wav”.
Refer to caption: x13.png

Figure: Figure 12: The video of a guy playing drum by tool. Our best model (MDSGen-B) generated a sound that is closer to GT compared to existing approaches. We refer the reader to the listen file provided in the supplementary for comparison. File “-Qowmc0P9ic_000034.wav”.
Refer to caption: x14.png

Figure: Figure 13: The video of a dog looks like howling. Our best model (MDSGen-B) generated a sound closer to GT than existing approaches. We refer the reader to the listen file provided in the supplementary for comparison. File “2vYkvwD-fkc_000010.wav”.
Refer to caption: x15.png

Figure: Figure 14: The video of a guy playing badminton. Our best model (MDSGen-B) generated a sound that is closer to GT compared to existing approaches. We refer the reader to the listen file provided in the supplementary for comparison. File “-miI_C3At4Y_000104.wav”.
Refer to caption: x16.png

Figure: Figure 15: RGB distribution for Mel-Spectrogram. We provide more evidential samples that the output of the VAE in the test set yields different characteristics for three channels even though these differences are imperceptible to the human eye (right figures). Interestingly, we find that the first channel (R) always has some different patterns compared to the remained channels (1).
Refer to caption: x17.png

Figure: Figure 16: RGB distribution for Mel-Spectrogram. We provide more evidential samples that the output of the VAE in the test set yields different characteristics for three channels even though these differences are imperceptible to the human eye (right figures). Interestingly, we find that the first channel (R) always has some different patterns compared to the remained channels (2).
Refer to caption: x18.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Three models are presented in the paper (Tiny, Small and Base) except the overfitting large model. What is the model presented in the supplementary files?

Is there any reason to use an image VAE instead of audio VAEs? 

Did the authors observe any advantage of GLA over a neural vocoder?

Is it possible to run a subjective listening test, and see the consistency between human evaluation and the audio-video alignment accuracy measured by a DNN model?

Could the authors measure the FAD scores on top of the current FID scores?
soundness: 2
strengths: Most contributions of MDSGen are about micro design aspects.
1. Channel selection of Mel-spec
2. Time-aware masking strategy for generative models
3. Reduced dimension of the video features
4. Small model size and fast inference speed
summary: **Update after discussion period**:

My biggest concern of this paper is about its audio reconstruction pipeline. The original pipeline consists of an RGB VAE and a non-DNN mel2wave conversion module, resulting in terrible sound quality. Moreover, such a pipeline has been used in video2audio community widely, which is a pity.

During the rebuttal period, the authors not only added the experiment of using a vocoder for the mel2wave conversion, but also re-trained their models with an audio VAE. As a result of their efforts, the FAD score improved from 4.37 to 2.16 and eventually 1.34. Congrats on the achievement!

I listened to the latest samples in the Google Drive, and found a distinct advantage of audio VAE in samples that contain non-noise or musical sources (sample027__SrU3mfTPYg_000032.mp4, sample045_2es7oZzwLWM_000030.mp4). Although the AV-align score is slightly worse with an audio VAE, the score is still higher than conventional methods by a large margin.

Since the AV-align score is computed by a pretrained DNN model, moreover, considering the gap between v2a community and audio generation community, the metric might not be very reliable. For example, in samples such as "sample007_-2sOH8XovEE_000484.mp4", I don't feel the alignment is as good as the scores indicate. Perhaps some future works can be done to further improve the metric itself.

I increased my ratings to **encourage further collaboration between video2audio and audio communities**.

---------------

The paper proposes "MDSGen", an efficient model based on Masked Diffusion Transformer, for video-to-audio generation.

The challenges of video-to-audio generation are mainly:
1. Heavy computation and memory usage;
2. Requirements for the audio quality;
3. Requirements for the audio-video alignment;

MDSGen reduces the resource consumption by using very light-weight Transformer coupled with fast diffusion samplers such as DPM solver, as well as a dimension reduction module to reduce the size of the video conditioning embeddings.

MDSGen improves audio quality and audio-video quality by introducing a time-aware masking strategy into the mask DiT framework, together with other efforts.

Conceptualy, MDSGen looks like a framework that replaces the "text prompt" in text-to-audio DiT [StableAudioOpen],[MakeAnAudio2] by a video feature embedding. Hence the technical contributions are more in micro aspects.

However, some design choices may have severely affected the audio quality, making the work less solid or reusable to the community. Audio quality observed in the supplementary files is far from the level in modern text-to-audio models such as [AudioLDM], [MakeAnAudio2], [SpecMaskGIT], [StableAudioOpen].
weaknesses: ## Major issues
### 1. Improper audio reconstrution pipeline
MDSGen ustilizes the VAE from Stable Diffusion, which is not trained for Mel-spec. Although the authors carefully discussed how to take the most advantage of this image VAE, the discussion itself is **NOT** reusable for the audio community, as there have been plenty of audio VAE designs, some of which are publicly available such as [AudioLDM], [MakeAnAudio2], [StableAudioOpen], [DAC].

Another improper choice is that, MDSGen utilizes the Griffin-Lim Algorithm (GLA) to convert mel-spec back to wave forms. GLA has almost been abandoned by audio community, due to the recent advance in neural vocoder, e.g., [HiFiGAN], [UnivNet], [BigVGAN]. I believe the apparent phase distortion in the supplementary files might have been caused by GLA.

There is a rough comparison on the quality of audio reconstruction pipeline in a recent paper [SpecMaskGIT], I hope it could be useful for the improvement of MDSGen.

I strongly recommend the authors to consider audio-specified reconstruction pipelines for improved audio quality. Even in audio-visual generation community, we can see the usage of such audio VAE for excellent audio quality, e.g., [VisualEchoes]
### 2. Invalid claims on the result
Because the audio quality is far from the baseline in audio generation community, it is improper to claim that MDSGen is better in "audio-video alignment".

I believe, the audio-video alignment can be evaluated only when the audio quality is sufficiently good. Given the current audio quality, I don't think the model is ready for further evaluation.
## Minor issues 
### 1. Evaluation metrics
The FID used in this paper comes from the implementation of SpecVQGAN, a pioneer of audio generation. However, the FAD implementation ([AudioLDM],[FAD_github]) has been more widely accepted in audio community. Evaluating with the widely adopted FAD metric can also help the readers to compre the audio quality with other audio generation models.
### 2. Insufficient ablation study
MDSGen trains a learnable module to reduce the video feature sequence into a single vector. From Figure 6, we can observe that the learned weights are quite evenly distributed (except the beginning and ending frames).

The observation posts a question: How much improvement can the learnable reducer bring compared to a naive average pooling?

[StableAudioOpen]: https://arxiv.org/abs/2407.14358
[MakeAnAudio2]: https://arxiv.org/abs/2305.18474
[AudioLDM]: https://audioldm.github.io/
[SpecMaskGIT]: https://arxiv.org/abs/2406.17672
[DAC]: https://github.com/descriptinc/descript-audio-codec
[HiFiGAN]: https://github.com/jik876/hifi-gan
[UnivNet]: https://github.com/rishikksh20/UnivNet-pytorch
[BigVGAN]: https://github.com/NVIDIA/BigVGAN
[VisualEchoes]: https://arxiv.org/abs/2405.14598
[AudioLDMEval]: https://github.com/haoheliu/audioldm_eval
[FAD_github]: https://github.com/gudgud96/frechet-audio-distance

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see Weaknesses.
soundness: 2
strengths: The paper introduces a novel framework for video-to-audio sound generation that effectively combines a temporal-aware masking strategy with a redundant feature removal module. 

MDSGen demonstrates significant improvements in model efficiency by using a smaller masked diffusion transformer architecture. The framework achieves high alignment accuracy on benchmark datasets with a fraction of the parameters, memory usage, and inference time compared to baselines. 

The paper provides a structured explanation of MDSGen’s architecture and mechanisms, including the Temporal-Awareness Masking (TAM) and the Reducer module for filtering out redundant features. Extensive experimental results on VGGSound and Flickr-SoundNet datasets clearly validate the method’s effectiveness, with MDSGen achieving superior performance across alignment accuracy and efficiency metrics.
summary: The paper presents MDSGen, an efficient framework for vision-guided sound generation that minimizes model size, memory usage, and inference time. Key innovations include a temporal-aware masking strategy to enhance alignment accuracy and a redundant feature removal module to filter unnecessary video information. Using a lightweight masked diffusion transformer, MDSGen outperforms larger Unet-based models on VGGSound and Flickr-SoundNet, achieving high synchronization and alignment with significantly reduced computational costs.
weaknesses: 1. Lack of Novelty and Contribution: The paper presents the primary contributions are the Temporal-Awareness Masking (TAM) strategy and the visual Reducer module. However, masking strategies have been widely explored in audio generation research, as seen in works like [1, 2], and the specific concept of Temporal-Awareness Masking has been studied in [3, 4]. The visual Reducer module, primarily a 1x1 convolutional layer (line 181), lacks detailed design innovations, which limits its distinctiveness and impact.

2. Insufficient Exploration of Design Choices: For video-to-audio generation, the choice of video encoder plays a crucial role in understanding the video content. Clarification on the selection of CAVP as the video encoder would add valuable insight. Additionally, the paper could explore using more video encoders, such as CLIP [5], VideoMAE [6], ViVit [7], and TAM [8], which could enrich the technical depth of the proposed method.

3. Presentation and Writing:
Some claims in the paper lack supporting evidence, such as the statements in lines 183-185 that the proposed method “minimizes redundant features that could lead to overfitting” and in line 224 that setting N_2 = 4 “gives better performance for audio data.” These points would benefit from empirical support to substantiate their validity.

4. Supplementary Material: The quality of generated audio samples in the supplementary material raises concerns regarding the overall quality of results produced by the proposed method, which may affect its effectiveness and appeal.

[1]Pascual S, Yeh C, Tsiamas I, et al. Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity[J]. arXiv preprint arXiv:2407.10387, 2024.

[2]Borsos Z, Sharifi M, Vincent D, et al. Soundstorm: Efficient parallel audio generation[J]. arXiv preprint arXiv:2305.09636, 2023.

[3]Bai H, Zheng R, Chen J, et al. A $^ 3$ T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing[C]//International Conference on Machine Learning. PMLR, 2022: 1399-1411.

[4]Garcia H F, Seetharaman P, Kumar R, et al. Vampnet: Music generation via masked acoustic token modeling[J]. arXiv preprint arXiv:2307.04686, 2023.

[5]Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. PMLR, 2021: 8748-8763.

[6]Tong Z, Song Y, Wang J, et al. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training[J]. Advances in neural information processing systems, 2022, 35: 10078-10093.

[7]Arnab A, Dehghani M, Heigold G, et al. Vivit: A video vision transformer[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 6836-6846.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. The reducer is designed to have fixed weights, serving as a weighted average of all the frames. However, the sound events of different videos are distinct, so why not adopt a dynamic weighted average strategy, e.g., attention pooling?
2. What is the exact implementation of the model? How is the visual conditioning (i.e., $p(x|v)$) implemented?
soundness: 3
strengths: 1. The idea of compressing visual representations into one single vector is bold and intriguing, which reduces significant computing pressure on the DiT side.
2. The proposed TAM strategy is interesting and makes sense. Previous works about masking audio representations, such as AudioMAE, have drawn conclusions that the unstructured masking strategy is superior, which contradicts the conclusion in this paper. I believe this paper brings more insights into this topic. 
3. The authors have conducted tons of ablation experiments to support their model design and parameter decision, making the conclusions plausible.
summary: The paper introduces MDSGen, an innovative framework designed for vision-guided open-domain sound generation with a focus on optimizing model parameter size, memory consumption, and inference speed. It features two major innovations: a redundant video feature removal reducer and a temporal-aware masking strategy. Utilizing denoising masked diffusion transformers, MDSGen achieves efficient sound generation without the need for pre-trained diffusion models. On the VGGSound dataset, the smallest MDSGen model demonstrates a 97.9% alignment accuracy while using fewer parameters, consuming less memory, and performing faster inference compared to the current state-of-the-art models. The results underscore the scalability and effectiveness of this approach.
weaknesses: 1. The major concern is about the modeling of audio representations, as I am familiar with this field. I believe that Mel spectrum is more of a 1D feature rather than 2D, because the spectrum does not satisfy translational invariance (if a formant chunk in a spectrum is moved from the bottom left to the top right, the semantics of the sound are likely to be completely destroyed), and the frequency domain and time domain cannot simply be simulated by spatial coordinates. A relevant observation in this paper is that a complete random masking strategy is underperformed by the temporal-aware masking strategy. Therefore, I believe that considering Mel spectrograms as gray-scale images and modeling them using 2D VAE pretrained with real images is suboptimal, which further prevents modeling sounds with varying lengths. There are already approaches that model audio using 1D VAE, such as Make-an-audio 2. So can the authors provide justifications for choosing 2D rather than 1D? In my view, choosing 1D combined with the TAM strategy could form a more compelling motivation.
2. The idea of compressing visual representations into one single vector is intriguing. However, I don't understand why this could work. How does one single vector provide accurate information about temporal position? I believe Diff-foley works because it adopts a sequence-to-sequence cross-attention mechanism, which provides rough sequential and positional information for the audio to follow. Could the author provide further analysis and discussion on this point? For example, analyzing the components related to temporal position within that vector, or the relation of the learned weights of reducer between key frames of videos. 
3. Similar concern: the learned weights of reducer seem to be focused more on the head and tail frames of videos. Does this imply that the reducer is more focused on global video information? How can it be determined that it is capable of extracting local positional information?
4. The alignment classifier proposed in Diff-foley only reaches 90% accuracy on their test set. However, the best performance in this paper reaches 98+. How could this happen? Is the classifier involved during the training process?

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: See weaknesses.
soundness: 3
strengths: 1. MDSGen achieves strong results with a small model size, making it useful for real-time applications. Compared to larger models, it is faster and more memory-efficient.
2. TAM is an interesting approach that focuses on time-based information in audio, aiming to improve alignment by using masking based on temporal patterns rather than spatial patterns (commonly used for images).
3. The paper provides extensive experiments with detailed comparisons against other models. Ablations for each key component further clarify the model’s design choices.
summary: This paper presents MDSGen, an efficient and compact framework for generating open-domain sounds guided by visual input. MDSGen uses masked diffusion transformers instead of the usual U-Net architectures, optimizing for faster speed, lower memory use, and parameter efficiency. Key features include a module to remove redundant video data and a Temporal-Awareness Masking (TAM) strategy, both aimed at efficient, high-quality audio-visual alignment. Tests on the VGGSound and Flickr-SoundNet datasets show that MDSGen achieves strong alignment accuracy with much lower computational demands than larger models.
weaknesses: 1. I question the decision to use an image-trained VAE (from Stable Diffusion) rather than an audio-specific VAE, such as those in AudioLDM [1]. An audio-dedicated VAE could better capture the temporal and spectral nuances inherent to sound, which are often lost when treating audio as an image. Relying on an image-based VAE reduces the model’s potential to fully leverage audio-specific features and may affect TAM’s performance.
2. The authors highlight channel selection within the RGB output as a means of optimizing the final mel-spectrogram. While using the G channel showed marginal improvements, I question if relying on such RGB channel selection can sufficiently address the nuances of audio spectrogram representation (similar to 1). A more audio-specific solution that doesn’t require treating spectrograms as RGB images would likely be more consistent with the needs of audio data, as these channels are meant for pixels, not spectral data.
3. I would suggest that the authors conduct a human perceptual study to better assess audio quality. Relying solely on quantitative metrics may not fully capture perceptual quality, as these measures can sometimes be unreliable.
4. In Section 5.4, various masking strategies are explored, and TAM shows a clear improvement over random masking and FAM. However, the reasons for TAM’s superiority are not fully explained. It would be beneficial to discuss why TAM outperforms FAM in this context, particularly since FAM is intuitively suitable for audio data.
5. I noticed a missing citation for SpecAugment [2] and AudioMAE [3], a masking approach relevant to TAM proposed here.

References

[1] Liu et al. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models.

[2] Park et al. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition.

[3] Huang et al. Masked Autoencoders that Listen.

Meta-review: This paper introduces "MDSGen," an efficient model based on a Masked Diffusion Transformer, for video-to-audio generation. MDSGen addresses several key challenges in this area, including: (1) high computational and memory demands; (2) the need for high-quality audio output; and (3) the importance of audio-visual alignment. Four expert reviewers initially raised concerns regarding the audio reconstruction pipeline, the validity of certain claims about the results, and the choice of audio representation. The authors responded with extensive experiments and clarifications.  

Following discussion, most major concerns were resolved. All reviewers ultimately rated the paper a 6 (marginally above the acceptance threshold). I concur with the reviewers' assessment and believe this paper, exploring both multimodal generative AI and efficient generation, is a valuable contribution to ICLR and an important topic in the field. The authors should incorporate the new results, analysis, and clarifications into the revised paper.

Decision: Accept (Poster)