Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation
Authors: Kang Zhang
Trung X. Pham
Suyeon Lee
Axi Niu
Arda Senocak
Joon Son Chung
Venue: NeurIPS 2025 poster ()
Keywords: Open domain video to audio generation
FLow based model
Cross modal alignment
Transformer denoiser
Model guided training
Primary area: applications
TL;DR: We introduce MGAudio, a flow-based framework for video-to-audio generation that leverages model-guided dual-role alignment to achieve state-of-the-art performance.

Abstract: We present MGAudio, a novel flow-based framework for open-domain video-to-audio generation, which introduces model-guided dual-role alignment as a central design principle. Unlike prior approaches that rely on classifier-based or classifier-free guidance, MGAudio enables the generative model to guide itself through a dedicated training objective designed for video-conditioned audio generation. The framework integrates three main components: (1) a scalable flow-based Transformer denoiser, (2) a dual-role alignment mechanism where the audio-visual encoder serves both as a conditioning module and as a feature aligner to improve generation quality, and (3) a model-guided objective that enhances cross-modal coherence and audio realism.
MGAudio achieves state-of-the-art performance on VGGSound, reducing FAD to 0.40, substantially surpassing the best classifier-free guidance baselines, and consistently outperforms existing methods across FD, IS, and alignment metrics. It also generalizes well to the challenging UnAV-100 benchmark. These results highlight model-guided dual-role alignment as a powerful and scalable paradigm for conditional video-to-audio generation. Code is available at: https://github.com/pantheon5100/mgaudio

Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation
Kang Zhang" Trung X. Pham" Suyeon Lee', Axi Niu², Arda Senocak Joon Son Chung!
KAIST, South Korea NWPU, China UNIST, South Korea (zhangkang, trungpx, ( @kaist. kr nax@nwpu. edu. cn, ardasnck@unist. ac.
Abstract
We present MGAudio, a novel flow-based framework for open-domain video-to- audio generation, which introduces model-guided dual-role alignment central design principle. Unlike prior approaches that rely on classifier- beaseeier-baeed or classifier- free guidance, MGAudio enables the generative model to guide itself through a dedicated training objective designed for video-conditioned audio generation. The framework integrates three main components: (1) scalable flow-based Trans- former model, (2) a dual-role alignment mechanism where the audio-visual encoder serves both conditioning module and as feature aligner improve generation quality, and (3) a model-guided objective that enhances cross-modal coherence and audio realism. MGAudio achieves state-of-the-art performance on VGGSound, reducing FAD to 0.40, substantially surpassing the best classifier-free guidance baselines, and consistently outperforms existing methods across FD, IS, and align- ment metrics. It also generalizes well to the challenging UnAV-100 benchmark. These results highlight model- model-guided dual-role alignment as a powerful and scal- able paradigm for conditional video-to-audio generation. Code is available at: https::/ //gt thub. com/ womm/panthennn1... 10000/1000mmm.mom mgguddi
Introduction
DifiFolky (NwuriS223
See and hear (DVPR23)
Vision-guided audio generation is gaining increas- ing attention due to its critical role in Folcy sound synthesis for video and film production [2]. In particular, video-to-audio (V2A) generation has emerged key task, not only for enriching silent videos produced by emerging text-to-video mod- els [3, 4, 5, 6]. but also for enhancing realism and immersion in professional video editing work- flows. Realistic audio is essential for audio-visual coherence and immersive experience, but gener- ating sound semantically aligned and temporally synchronized with video remains challenge.
FRIEREN (NcuriPS24)
HO 1 Lowert Better
MMMAuioo (CV9R25)
MGAudlo (3ork)
MGAudio (Our))
Diameter &Farams
LAM LUs
5% 10%
50%
55%
100%
Data
Figure I: V2A on VGGSound [1]. MGAudio attains the best FAD among video-to-audie meth- ods with full data and 1.IM iters, and remains competitive even with only 10% data and 300k iters, highlighting its strong data efficiency.
Prior works have explored GAN-based [7] and au- toregressive Transformer based models [8], which often struggle with synchronization and semantic consistency. More recent methods adopt diffusion models to improve generation quality. For exam- ple, Diff-Foley [9] uses a contrastively trained video encoder to extract visual conditions, which
Equual contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

are then used to guide a diffusion model for audio generation. Other approaches, such as See and Hear [10] and FoleyCrafter [11], leverage large pretrained models to achieve high-fidelity results, but at the cost of massive model size (billions of parameters). In contrast, models like MDSGen [12], FRIEREN [13], and MMAudio [14] (131-159M parameters) explore more lightweight denoising or flow-matching [15] objectives, achieving competitive performance across fidelity and alignment metrics. Despite architectural differences, diffusion- and now-matching-basee methods typically rely on classifier-free guidance (CFG) to improve generation quality. This involves randomly dropping conditioning signals during training to simulate both conditional and unconditional objectives. While effective, this multi-task setup may dilute the model's capacity and lead to mismatched sampling behavior at inference.
To address the limitations of classifier faass guidance (CFG) in audio generation, we propose MGAudio, a novel model-guided framework that replaces CFG with direct model-based supervision during training. Our approach builds on the Model-Guides (MG) framework [16], originally introduced for class-conditional image generation on ImageNet, and effectively extends and adapts it to the video-to-audio domain. While MG is specified for discrete class labels, its application to conditional generation from continuous video inputs remains unexplored. To this end, MGAudio introduces a dual-role audio-visual encoder: one branch modulates the diffusion process via LayerNorm-bar conditioning, while the other provides intermediate alignment signals to guide audio synthesis at multiple denoising steps.
We show that MGAudio (131M) achieves state-of-the theth--rr performance on VGGSound [1], with a FAD of 0.40. Notably, the same model generalizes robustly to the UnAV-100 benchmark [17] without any fine-tuning, demonstrating strong cross-dataset transferabilitt. Furthermore, when trained on just 10 % of the VGGSound, it outperforms existing methods trained on the full dataset across key metrics such as FAD (Fig. I). These results demonstrate that model-guided training not only enables more efficient learning but also offers compelling alternative to CFG for scalable and effective audio generation. Our key contributions are:
We introduce MGAudio, a novel framework for video-to-audio generation that replaces the conven- tional classifier-free guidance (CFG) objective with model-guidance (MG) objective, leading to more efficient training and improved generation quality.
We propose a dual-role alignment mechanism that uses shared jointly trained visual and audio encoder for conditioning and intermediate representation alignment, enabling more effective video- audio feature integration.
MGAudio o sets a new state-of-the-art on VGGSound, achieving an FAD of 0.4 with 131M parameters, and maintains strong performance with only 10% of the training data (FAD 0.8), highlighting its efficiency and scalability.
Extensive experiments and analyses on VGGSound and UnAV-100 benchmarks demonstrate that the model--gudanne objective, combined with dual-role alignment, significantly improves data efficiency and generalization over prior methods.
2
Related Works
2.1 Classifier-Free Guidance Learning for Video-Guided Sound Generation
Early works such as Spec VQGAN [8] and Im2Wav [18] employ autoregressive Transformers to generate audio tokens conditioned on video or CLIP features. However, their sequential decoding results in slow inference and weak cross-modal alignment, limiting their effectiveness in open-domain scenarios. Recent advances in generative modeling have led to powerful diffusion- based methods that show strong performance across diverse application domains [19, 20J. Diff-Foley [9] introduces a two-stage pipeline combining contrastive video-audio pretraining with latent diffusion for more efficient generation. This has inspired follow-up work [10, 11]: [10] focuses on leveraging pre-trained generative models rather than training from scratch, while [11] targets improvements in audio-visual
synchronization.
Despite their success, these approaches rely heavily on large U-Net backbones, which limits scalability and efficiency. To address this, MDSGen [12] incorporates recent innovations in generative modeling, such as Diffusion Transformers (DiT) [21] and spatial masking [22, 23], adapting masked diffusion Transformers for video-guided audio generation and achieving improved performance and efficiency.

---

The introduction of flow matching [15] has further expanded generative modeling abilities. Recent methods like FRIEREN [13] and MMAudio [14] adopt flow matching as an alternative diffusion, using Transformer-basee frameworks and reporting strong results on video-to-audio (V2A) tasks. Similarly, we adopt flow matching with a Transformer-basee model.
However, despite architectural differences, most state-of-the-art approaches, whether diffusion, masked diffusion, or flow-based, continue rely on classifier-free guidance (CFG) [24]. where input conditions are randomly dropped during training (typically at a rate of 1 = 10%% to enable multi-task learning (unconditional and conditional). In contrast, we shift the focus from this paradigm by adopting the recent Model Guided approach [16], moving toward more efficient audio model- model-guided (AMG) strategy that achieves stronger generation quality and faster convergence.
2.2 Model-Guidance Learning in Audio Generation
Model-guided learning has recently been introduced in the vision domain as Vision
(VMG) [16] which enables efficient single-pass inference and is complementary to CFG, offering improvements in both inference modes. Inspired by this, we propose MGAudio, the Audio Model- Guidance (AMG) audio generation, tailored for video-conditioned open-domain sound synthesis. Unlike the vision setting, we find that model-guided training alone improves convergence, but CFG remains beneficial at inference ensure high-fidelity audio. This reveals modality-specifff behavior, suggesting that audio's 's temporal sensitivity may demand additional supervision at test time.
2.3 Feature Alignment for Accelerating Diffusion Learning
A key advancement in improving the training efficiency of vision diffusion is REPA [25], which proposes aligning intermediate image features in generative model with a pre-trained representation model [26, 27, 28, 29] to accelerate learning. Building this idea, VA VA-VAE [30] and REPA-E [31] jointly optimize vision feature alignment in the VAE and the transformer, yielding notable gains in image generation quality. Inspired by these simple yet effective strategies, we introduce novel approach for audio generation that aligns intermediate audio features from a strong pretrained audio model with the transformer model. We term this dual-role audio- learning, enabling efficient and semantically consistent generation through cross-modal feature supervision.
2.4 Temporal Alignment in Video-to-Audio Generation Methods
Another important research direction in video-to-audio generation focuses on enhancing temporal aligmment between visual events and the generated audio. Recent works have explored various strategies to improve synchronization and cross-modal correspondence. AV-Link [32] employs temporally aligned self-attention fusion for bidirectional conditioning between audio and video. Hidden Alignment [33] enhances synchronization through encoder design and temporally aware augmentations. V2A-Mapper [34] adopts a lightweight approach that maps CLIP embeddings to CLAP space for modality bridging using foundation models.
More recent frameworks such as MDSGen [12], FRIEREN [13], and MMAudio [ [14] explore different ways of integrating video conditions: MDSGen [12] aggregates all frames into compact global representation, FRIEREN [13] interpolates frame features along the temporal axis for alignment with the mel-spectrogram, and MMAudio [14] leverages multimodal transformer to achieve fine-grained audio-video attention.
Although our work does not primarily focus on temporal conditioning design, we adopt the MDSGen- style global aggregation for simplicity and efficiency. As shown in Appendix J, different conditioning strategies each exhibit unique strengths across distributional, semantic, and temporal dimensions.
3 Method
We introduce MGAudio, the first audio generation framework to adopt model-guides training. Un- like classifier-free guidance (CFG) [24], which learns conditional model and unconditional model separately, model-guided learning provides direct supervision through the model itself, refining the model prediction directly aligns with CFG. As illustrated Fig. 2, MGAudio comprises: (1) a
Dual-Role Audio- Visual Encoder (DRAVE) with learnable projections for cross-modal alignment; (2) a Flow- Denoising Transformer (FBDT) that maps noise audio via flow matching; and (3) an (AMG) training objective that enables replacing CFG with more targeted,
Audio
3

---

X E R
Latent
Racon, Latint
AMG
Model Guidance Training
Patchify
Objective LANG
Spectrogram
T/r
Inforence
Dual-role Audio-
Bow Based Denoising Transformet-
VAE
Dec
Frether
CAVP
6M654
MN55
MesA
Video
Autio
Vocoder
Audio intermediate Features
Silent Video (V)
Aucio clean festars
Padi CUNGOAA TeatArn
Gereer and auaiio
Figure 2: Overview of the MGAudio framework for gudle- audio generation. We design the first use of model-guidance method that learns audio latent with objective and performs dual alignment learning. The violet arrow + is training-only. the black arrow + is used for both training and inference, and the blue arrow 4 is inference-only. MHSA: Head Self-Attention.
model-driven signal. These components together enable robust alignment, efficient denoising, and high-quality video-conditioned audio generation. We describe each module below.
3.1 Scalable Flow-Based Denoising Transformer
The first key component of MGAudio is the Flow-Based Denoising Transformer (FBDT) shown in Fig. which builds upon the scalable flw-matching architecture of the Scalable Interpolant Transformer (SiT) [35], state-of-the-ee generative modeling framework. In contrast to conventional diffusion models [36], flow matching learns continuous transport directions, yielding more stable and efficient denoising.
Given an input audio signal A RLA and sequence silent video frames V E Ly x3x224x224 of length Lv, we first convert the ~ 8s waveform into a mel-spectrogram X E R R64x816 using log-mel filterbanks with number of Mel bands 64. A pretrained AudioLDM VAE [37] &vAe maps X into latent representation x E 888 10 which is then patchified (patch size sequence of tokens frames are processed by a vidco encoder Eviteo (c.g., CAVP video encoder [9]) to produce frame-level embeddings v E Lv x512 These features are temporally aggregated (e.g., via average pooling or attention, or with learnable 1 X convolution layers as did in MDSGen [12]) to obtain global video representation E 1x This conditioning vector if guides the Transformer model @ to predict the noise vector or its equivalent flow direction during the reverse generation process. We formulate audio generation as a flow-matching problem [15], where the model learns the direction of transport from noise to data using family of linear interpolants parameterized by time e [0, 1]. Following the flow mowtchtcinng paradigm [15], we define a noisy latent at time t as:
2) and flattened to a S16 with D 768 for the Base-size model. In parallel, silent video
X; (1 t)xo + te, E~ (0,
where Xo is the clean latent and e is standard Gaussian noise. The model is trained to predict the conditional flow direction that transports x, toward Xg:
LrM = 1F,.xo t.e [ X,, G, 1) u. ux(xx (x))3
(2)
where ug is the model model- predicted flow direction. The true flow is analytically derived from the derivative of the interpolant (x(xx/x)) X This flow- wow-matching loss serves the core denoising objective. Its efficacy is further enhanced by our model-guided formulation (Sec. 3.3), which refines the supervision signal to better align with video context.
During inference, the model starts from pure Gaussian noise x ~ (O, 7) in the latent space and iteratively denoises using the predicted flows, progressively refining the latent & @xx16x204 The final mel-spectrogram X 6 is recovered using the VAE decoder, and neural vocoder [38] reconstructs the corresponding waveform. This formulation combines the scalability and efficiency of flow-based models with the expressive power of Transformer architectures and robust conditioning mechanisms for open-domain video-to-aue audeoooo-uuioooooo generation.

---

3.2 Dual-Role Audio-Visura Encoder
The second key component our framework the Dual-Role Audio- Visual Encoder (DRAVE), illustrated in Fig. 2, which comprises two main branches: (1) an audio encoder and (2) video encoder. We adopt the Contrastive Audio- Visual Pretraining (CAVP) strategy introduced in Diff-Foley [9], which effectively aligns audio and video representations and has demonstrated strong performance in models like FRIEREN [13] and MDSGen [12]. While prior works typically use only the CAVP video encoder, discarding the audio counterpart,
leverage both. Specifically, use the CAVP video encoder to condition the denoising process for vision-audio alignment (Video Processor branch) and simultaneously integrate the CAVP audio encoder for audio-aude audio aligument (Audio Processor branch), drawing inspiration from REPA [25] in the vision domain. This dual-role design enhances representation learning and significantly improves audio generation quality.
Video Processor. As illustrated Fig. 2, the input condition for the V2A task is a silent video. We extract frame-level features using the CAVP video encoder [9], resulting in embeddings v E RR x512 These are projected to Av x768 via a multi-laye perceptron (MLP) to match the input dimension of the Transformer's 's multi- mead self-attention layers (model size B). To reduce temporal redundancy and aggregate contextual cues, we follow MDSGen [12] and apply a 1 x convolution to condense the sequence into a compact global feature vector 전 E IR x768 This vector serves as a conditioning signal in the denoising process, injected through Adaptive LayerNorm (AdaLN) to guide the generation process with visual context.
The ground-truth mel-speetrogram X processed via two parallel branches, as shown in Fig. 2. In the first branch, X is encoded by the Audiol _DM2 VAE [38), where Gaussian noise injected into the latent space facilitate denoising-be -aased generation. the second branch, the same spectrogram is passed through the CAVP audio encoder [9] to provide auxiliary representation for regularizing the learning process.
Audio Processor.
Drawing inspiration from regularization techniques in the vision domain, such as REPA [25] and VA- VA-VAE [30], we adopt similar strategy for pross-modal alignment in audio generation. Unlike REPA, which relies on generic visual encoders like DINOv2, we find that CAVP, specifically trained for audio- visual correspondence, significantly improves alignment and synthesis quality. Our ablation studies confirm that the CAVP audio-visie encoder plays dual role: facilitating alignment with video features and enhancing intermediate representations when paired with the denoising transformer. We define the alignment loss used for joint training with the denoising branch as:
Calign-audio Exx,ett
M c.
with C, = similarity (G): (H;)),
where C, denotes the alignment loss for the i-th audio patch among B total patches. Go represents the CAVP-encoded clean audio features, and H, is the intermediate latent from the noisy audio branch at time step t. H, is projected through an MLP hg, to match the G 's dimensionality. Following REPA [25], cosine similarity and retain the same number of positional layers for consistency. We avoid applying an additional MLP to Go. as it led to feature collapse in our experiments.
Overall Training Objective. The overall training loss combines the flow-matching denoising loss and the alignment loss, balanced by a scaling factor X, which we set to 0.5 by default, following prior work [25]. The total objective is defined as:
CFM-align = LFM + XCalign-oudio
Here, LFM denotes the dow-matching loss for denoising, while Latiga-audio enforces consistency between pretrained clean audio and intermediate noisy audio generator representations.
3.3 Audio Model-Guidance (AMG)
The third core component of MGAudio is the Audio Model-Guidanel (AMG) mechanism, which departs from traditional classifier-free guidance (CFG) approaches commonly used audio gen- eration. As illustrated in Fig. 2, AMG serves the key to effectively training our model under video-condittoned audio generation. Building on the FBDT backbone, AMG replaces CFG with self-distilled target. Let (x J, be the conditioned flow prediction and (xx, 2, t) be the
5

---

unconditioned one. Our AMG objective is defined as variant of VMG [16] that reformulates the training target directly approximate the CFG optimization trajectory:
AMG ,, (0 t) n'11?,
(5)
where the model-guided target u' is defined as:
u' = + sg (wo(Xe, ()( uo (Xe, 22(),
(6)
where w denotes the guidance scale factor, and sg(-) is the stop-gradient operator, used to block gradient flow through the guidance term. This stabilizes training and prevents degenerate solutions, as shown in [16]. To further improve stability, we compute w' using an Exponential Moving Average (EMA) version of the online model ug ensuring more reliable target predictions over the course of training. During training, the condition vector ú' is randomly replaced with a zero vector 2 with probability Φb The final loss in Eq. 4 training objective for MGAudio becomes:
LFM align LAMG + aLalign
3.4 AMG Meets CFG: Toward Efficient High-Fideling Audio Generation
Finally, while model-guided training alone has proven effective in the vision domain, e.g., VMG [16] achieves state-of-the- performance without external guidance and sees only marginal gains when combined with CFG, we observe distinct trend in audio generation. Specifically, although training with Audio Model-Guidance (AMG) already surpasses baseline models without guidance (c.g.. SiT), the highest audio generation quality is achieved when AMG-trained models are further enhanced with classifier-fi free guidance (CFG) at inference.
We provide comprehensive ablations and analyses of this synergy in the experiments section. Notably, we find that conventional CFG training, i.e., randomly dropping fraction of conditioning inputs during training defined in Eq. 2, performs worse than our AMG formulation in Eq. 5, which optimizes an implicit classifier via a modified objective. This demonstrates the superiority of AMG as training strategy, not just as a guidance method at inference time.
Lastly, motivated by the robustness and data efficiency observed in AMG-based training, we perform a low-resource experiment on VGGSound. Remarkably, MGAudio achieves strong performance across multiple evaluation metrics using only 10% of the full training set. We explore this strong generalization capability in the ablation section, where we hypothesize that AMG promotes highly efficient learning by effectively leveraging the task's conditional structure. Our analysis also reveals that data quality, rather than quantity, remains a critical bottleneck in VGGSound.
4 Experiments
4.1 Implementation Details
Datasets and Metrics. We train on the VGGSound dataset [1], which contains in-the-wild video clips from YouTube, with 182k for training and ~15k for testing. For generalization, we also test on the UnAV-100 dataset [17]. which includes 10,791 test videos with annotated sound events. Following prior works [14, 13, 12], we evaluate generation quality using Fréchet Distance (FD), Fréchet Audio Distance (FAD), Inception Score (IS), KL divergence, and audio-video alignment accuracy.
Training. MGAudio is trained for 1.1M steps with batch size of 64, learning rate of le and guidance scale w = 1.45. All experiments are run single A A100 (80GB). For all experiment we use sampling step of 50 and CFG value of 1.45. Following [13, 12], video-audio pairs are truncated to 8.2 seconds. We use Base model size for its performance-efficiengg tradeoff. Additional details of experimental setup in Appendix.
4.2 Main Results
a) VGGSound Dataset. Tab. 1 shows that MGAudio, the first model-guided (MG) audio generation framework, achieves state-of-the-art results across multiple metries using only 131M parameters. obtains an FAD of 0.40, significantly outperforming the second-best, MMAudio [14]. With CFG scale of 4.0, MGAudio achieves 99. 04%%% alignment accuracy, surpassing all baselines. This metric, introduced by [9], evaluates temporal synchronization and semantic coherence using dedicated

---

Table 1: MGAudio on VGGSound. Audio generation quality across state-of-the methods on the VGGSound test set is presented. Bold indicates the best performance, and an underline denotes the second-best. CFG: Classifier- Free Guidance, MG: Model-Guidane scratch solely on the VGGSound dataset, and set the text input to None during inference for faimess.
" "We train MMAudio from
Method
FAD4
FD4
1St
KL,
Align. Ace.t
Time (8)
# Params.
Train Type
Diff-Folcy [NeurIPS23]] [9]
6.25
23.07
10.85
3.18
93.94
0.36
860M
CFG
See und Heur [CVPR'24] [10]
5.51
26.60
5.47
2.81
58.14
18.25
1099M
CTG
FRIEREN [NeurlPS'24]] [13)
1.38
12.36
12.12
2.73
97.25
0.20
157M
CFG
MDSGen [ICLR'25] [12]
1.40
17.42
9.66
2.84
96.88
0.24
131M
CFG
MMAudio [CVPR 25]" [14]
0.71
6.97
11.09
2.07
92.28
0.98
157M
CFG
MGAudio, CFG 1.45
0.40
6.16
12.82
2.76
95.65
0.31
131M
MG
MGAudio, CFG
2.23
13.65
12.68
2.72
99.04
0.31
131M
MG
MGAudio, CPG = 6.0
1.50
9.10
17.39
2.80
98.10
0.31
131M
MG
classifier. At a higher CFG scale (6.0), MGAudio also reaches an IS of 17.39, though with some trade-offss in other metrics. Overall, these results highlight the efficiency of model-guided training over conventional classifier-free guidance.
Table 2: MGAudio on UnAV-100. All models are trained on VGGSound from Tab. and evaluated on UnAV-100 test set for generalization. Bold indicates the best, and an underline for the second-
Method
FAD
FD1
1St
KL
Align. Acc.
Train Type
Diff-Foley [NeurIPS'23] [9]
7.51
24.25
10.74
2.46
87.46
CFG
MDSGen [ICLR 255 [12]
2.01
16.10
10.10
2.18
97.57
CFG
FRIEREN [NcurIPS 24] [13]
1.40
10.82
13.46
2.02
98.14
CFG
MMAudio [CVPR 25] [14]
0.93
8.63
11.37
1.62
85.68
CFG
MGAudio (Ours)
0.54
5.40
13.90
2.00
97.54
MG
b) UnAV-100 Dataset. To assess generalization of MGAudio, we evaluate on UnAV-100 [17]. Tab. 2 presents the results. All models are models trained on VGGSound as reported in Tab. 1. MGAudio achieves the best performance across FAD, FD, and IS. While MMAudio gives the lowest KL divergence, suffers from poor alignment accuracy. In contrast, MGAudio maintains strong alignment accuracy (97.54% These results emphasize the robustness and strong generalization of our model- guided approach, even on datasets with dense, diverse audio-visual events not seen during training.
4.3 Ablation Study
We attribute MGAudio's 's strong performance to three design choices: the Audio Hodel-Guidanee (AMG) objective for improved supervision, a Dual-Role Audio-Visual Encoder that fully exploits CAVP's alignment strength, and compatibility with CFG at inference despite being trained without it. We perform five ablations under a unified setup, including a data efficiency study showing AMG's robustness with just 10% of VGGSound. Comparisons with CFG-only baselines and various alignment encoders (DINOv2, CLAP, CAVP) highlight the benefits of dual-alignment and the trade- offs in fidelity. We also demonstrate scalability across model sizes and show via UMAP [39] that AMG produces tighter. more Mlassconsssseen audio distributions than CFG-based methods. All ablations use 300k steps and smaller batch size of 16 for faster iteration.
4.3.1 Efficiency of Data Utilization
Motivated by the strong generalization and data efficiency of AMG training (Tab. 1, 2), we conduct a low-resource experiment on VGGSound with only 300k training steps. Interestingly, as shown in Tab. 3, MGAudio achieves comparable or even superior performance using just 10% of the training data, surpassing the full-data model on most metrics. Furthermore, compared with Table 1, our model trained with only 10% of the data already outperforms Diff-Foley, See and Hear, FRIEREN, and MDSGen in terms of FAD, FD, and KL, and achieves performance comparable to MMAudio. This highlights AMG's s ability to effectively exploit conditional structure for efficient learning.
A manual inspection of 100 random samples in VGGSound reveals that approximately 6% exhibit weak or irrelevant audio-visual alignment, and 9% contain static frames or silence, leaving only 85% with meaningful, well-aligned content. This suggests that noisy supervision in VGGSound may hinder performance, and in some cases, less but cleaner data can lead to better results. Nonetheless, use the full dataset for all main experiments ensure fair comparisons with prior work.
7

---

Table 3: Data Efficiency. Interestingly, training MGAudio on just 10% of VGGSound yields strong performance, surpassing the full-data setting on several metries. This highlights the model's data efficiency and robustness. AA: Alignment Accuracy (%)
VGGSound
UnAV-100 [17]
Train Data
FAD1
FD
IS
KL,
AA
FAD
FD1
IST
KLJ
AA
1.16
9.72
9.80
2.67
93.37
1.37
7.76
10.94
96.00
10%
0.73
8.79
10.28
2.64
95.73
0.85
7.15
11.67
1.95
97.61
30%
0.77
10.17
9.83
2.69
95.27
0.73
8.29
11.09
2.04
97.42
50%
0.80
10.44
9.35
2.69
94.51
0.78
8.39
10.50
2.03
96.97
100%
0.81
10.25
9.85
2.71
95.14
0.89
9.88
9.72
2.09
96.60
4.3.2
Effect of Model-Guidance and Dual-Alignment
We compare our model-guided approach with the CFG-based baseline SiT [35] in Tab. 4. SiT achieves strong performance V2A with CFG, notably FAD of 1.52. However, while model guidance (MG) alone underperforms, combining MG with CFG at inference significantly boosts results, surpassing SiT across all metrics. This contrasts with findings in the ImageNet domain, where MG alone is sufficient. Finally, our proposed dual-alignment strategy further improves performance, outperforming SiT with CFG.
Table 4: Effectiveness Model Guidance. Evaluation on the VGGSound test set shows that combining model-guidance (MG) with classifier-free guidance (CFG) at inference significantly boosts performance. Our dual-alignment strategy further enhances learning across all metrics.
Setting
CFG
Model-Guide Dual-Alignment
FAD,
FDJ
IS
KL,
AA
(a) No Guidance
2.67
17.23
5.69
3.01
78.20
(b) SIT [35]
1.52
16.25
6.51
2.87
87.07
(c) MG [16]
2.13
16.20
6.02
2.85
83.87
Joint Guidance
1.19
14.26
7.70
2.73
92.37
(e) MGAudio
1.14
13.09
8.35
2.70
93.67
4.3.3 Choice of Alignment Encoders
We find that prior works often use the video en- coder as the only conditioning signal, without fully leveraging its potential for joint integration in VAE transformer alignment. our anal- ysis, we focus on the role of the audio encoder to demonstrate how deeper integration can im- prove performance. While REPA [25] leverages DINOv2 [26] for representation alignment in vision tasks, directly applying MG with REPA and DINOv2 yields subpar results in video-to-audio synthesis. Instead, we investigate various audio encoders and observe that CAVP [9] and CLAP [40] achieve the best performance. Notably, CAVP attains a lower FAD of 1.14 compared to CLAP's 1.35, with other metrics remaining comparable. For simplicity, we adopt CAVP, which is already available via Diff- hiff-Foley's public checkpoint, avoiding the need for additional encoder models. As shown in Fig. 3, using DINOv2 for alignment often leads to semantically incorrect generations, for instance, generating speech instead of impact sounds in hadminton video, failing synthesize audio for musical performances. In contrast, CAVP and CLAP produce more coherent and relevant results.
Table Impact of Audio Encoder Choice. Eval- uation on the VGGSound test set shows that CAVP yields better FAD score.
Gray indicates default.
Encoder
FAD.
FDJ
ISt
KT.
Align, Acc.
(a) DINOv2
5.52
27.17
3.25
4.34
21.66
(b) CLAP
1.35
12.23
8.27
93.19
(c) CAVP
1.14
13.09
8.35
2.70
93.67
CLAP
CAVP
CLAP
DINO
Figure 3: Effect Alignment Encoder in Mel-Spectrogram. The selection alignment encoders significantly impacts the quality of generated audio in the V2A task.

---

p- Hocl ospied
dey
PAD=6.25
FAD=1.30
FAD=1.4
FAD+0.4
DiftFlly (NerIPS225
Frieren (NeuriPS24)
MDSGEN UKER225]
HGAudio (Ours)
Figure 4: Audio Distribution. MGAudio generates audio samples that more closely align with the target distribution compared to other methods. For example, samples for classes "playing djembe" (red points) and "scuba diving" (black points) are tightly clustered around the center of the real sample distribution. Full-resolution version provided in the supplementary for clearer inspection.
4.3.4 Scalability
We demonstrate that MGAudio scales in model size effectively across key metries: FAD, FD,
Table 6: Scalability, Evaluation on the VGGSound test set shows that MGAudio maintains strong per-
and IS, as shown in Tab. 6. We evaluate four
formance as model size increases, demonstrating effective scalability with larger parameter counts.
model sizes: S/2, B/2, L/2, XL/2, where "/2"
refers to the default patch size used in SiT [35]. Unlike MDSGen [12], which reported overfit-
Model Size
FADL
FDJ
IS1
KLJ
Align. Acc.
ting in their 1.2 model, our model-gun model-guided ap- proach maintains strong performance even at larger scales. To balance performance and effi- ciency, we adopt the B/2-size model (131M) in main experiments, as it already delivers competitive results with significantly lower computational cost compared to baselines.
(34M)
2.54
3.13
93.53
B/2 (131M)
1.14
13.09
8.35
2,70
93.67
1/2 (464M)
0.95
10.22
9.84
2.73
94,50
XL/2 (680M)
0.90
10.09
9.96
2.73
94.40
4.3.5 Audio Distribution Learned by Model Guidance
We compare the generative behavior of our model-guided MGAudio with classifier-free guidance (CFG)-based methods, including Diff-Foley [9]. FRIEREN [13], and MDSGen [12]. To evaluate the learned distributions, we select four distinct VGGSound classes: playing djembe, playing cornet, scuba diving, and black capped chickadee calling. For each class, we sample 50 videos and generate 20 outputs per video (1000 samples per method) using varied seeds. The distributions are visualized via UMAP [39]. As shown in Fig. 4, MGAudio produces samples that cluster more tightly around the real sample distributions than other methods, reflecting stronger class consistency and diversity. These observations align with the quantitative results in Tab. 1, where MGAudio achieves significantly lower FAD than the others. FD metries further support these findings.
4.3.6 Model-Guidance Classifie--Free Guidance
We examine the effect of classifier-fres guidance (CFG) during inference for models trained with different objectives. Here, use sampling step of 25 same with FRIEREN. Fig. 5 shows that FRIEREN [13], which depends on CFG during both training and inference, suffers significantly when CFG is disabled (i.e., CFG = 1). In contrast, MGAudio, trained with AMG (model-guided) objective, performs strongly even without CFG. Applying an optimal CFG scale further improves state-of-thee-rt. These results highlight the robustness of model-guided training and its reduced reliance on inference-time CFG for effective video-to-audio generation.
MGAudio, setting
17.5
FRIEREN
FRIEREN
14-
2.5-
Ours
15.0
0.95
2 2.0-
12.5-
ACC
0.90-
1.5-
10.0-
FIJEREN
FRIEREN
1.0
Ours
7.5
CFG
Figure 5: Effect of CFG vs. AMG. Training with AMG consistently outperforms CFG across all evaluation metrics on the video-to-audio task, highlighting the advantages of model-guided learning.

---

To further validate these findings under controlled conditions, we compare MGAudio with the CFG- only baseline SiT, which shares the same architecture. training setup, and optimization parameters. Both models are trained for 300,000 iterations with a batch size of 32. Figure 6 presents FAD and FD results across various sampling steps, with and without CFG applied inference. MGAudio consistently outperforms the SiT baseline in all configurations. Notably, even without CFG-which reduces inference cost by climinating the need for dual model passes Muasese-GAAdiio achieves superior perceptual and distributional quality. This confirms that model-guides training not only enhances robustness and efficiency but also provides consistent gains across both comparable baselines (SiT) and prior state-of-the-eee methods (FRIEREN).
FD (win CPC)
MD Ivon CPG)
FD [CTG=21
MAD (CFG
22
4.0
4.5
SIT
20-
3.0
MGAudic
2.5-
18-
2 3.0
9 16-
2
1.5
2.0
12
1.0
25 Sampling Stop
100
250
Samp ing Step
Sampiing Step
Samsling Step
Figure 6: Comparison of FD and FAD metrics for SiT and MGAudio models, evaluated with and without CFG across different sampling steps.
4.4 Limitations
One limitation of our method is its reduced effectiveness when generating human vocalizations or linguistically complex audio, such as dialogue or singing, especially from subtle visual cues like lip movements. Since MGAudio is optimized for general audio effects and ambient sounds, it lacks the structural and phonetic awareness needed model such content reliably. Additionally, while model- guided training improves overall aligmment and diversity, it may introduce confusion when the visual semantics are ambiguous or loosely correlated with the target sound. Furthermore, inference speed remains bottleneck due to the reliance on VAE and the iterative sampling process. A promising direction for future work to adopt acceleration techniques such as Consistency Models [41] or Physics-Informed Distillation (PID) [42].
5 Conclusion
In this work, we reveal compelling property of the proposed MGAudio, model based on scalable interpolant transformers (SiT) trained with model-guided objectives, demonstrating their strong poten- tial for efficient audio generation. Our MGAudio framework achieves saate-of-hhe-ae performance on VGGSound with a FAD of 0.40, which sets new benchmark. Notably, training with only 10% of the VGGSound dataset surpassing most existing methods that rely on 100% of the data in terms of FAD. We also show that model- -uuidacee produces a more compact and coherent audio distribution com- pared to classifier-free guidance (CFG). Finally, by aligning audio and video representations through dual learning, our framework utilizes pretrained video encoders more effectively than conventional conditioning strategies. These contributions collectively highlight the potential of model-guided diffusion transformers as scalable, data-efficient solution for multimodal audio generation.
Acknowledgments and Disclosure of Funding
This work supported by the IIT grant funded by the Korean government (MSIT, RS-2025- 02215122, Development and Demonstration of Lightweight AI Model for Smart Homes, 90%) and partially supported by the KAIST Jang Young Sil Fellow Program (10%).
References
[1] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scale audio-visual dataset. In Proc. ICASSP, 2020. 1, 2, 6, 8
10

---

[2] Vanessa Theme Ament. The Foley Grail: The Art of Performing Sound for Film, Games, and Animation. Routledge, 3rd edition, 2021.
[3] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proc. CVPR, pages 22563-22575, 2023.
[4] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zerg Text-to-image diffusion models zero-shot video generators. In Proc. ICCV, pages 15954-15964, 2023.
[5] Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang. Free-bloom: Zero-shot text-to-video generator with LLM director and LDM animator. In NeurIPS, 2023. 1
[6] Yichen Ouyang, Hao Zhao, Gaoang Wang, al. Flexifilm: Long video generation with flexible conditions. arXiv preprint 2tti::2044.882200 2024.
[7] Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan. Generating visually aligned sound from videos. IEEE Transactions on Image Processing, 2020.
[8] Vladimir lashin and Esa Rahtu. Taming visually guided sound generation. In Proc. BMVC.,
2021. 1,2
[9] Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: synchronized video- to-audio synthesis with latent diffusion models. In NeurIPS, 2023. 1, 2, 4, 5, 6, 7, 8, 22, 27
[10] Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. In Proc. CVPR, 2024. 2, 22, 27
[11] Yiming Zhang, Yicheng Gu, Yanhong Zeng. Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. arXiv preprint arXiv:2407.014990 2024.7 2
[12] Trung Pham, Tri Ton, and Chang D Yoo. MDSGen: Fast and efficient masked diffusion temporal-awart transformers for open-domain sound generation. In Proc. ICLR, 2025. 2, 3, 4, 5, 6, 7, 9, 22, 25, 26, 27
[13] Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, and Zhou Zhao. Frieren: Efficient video-to-audie generation network with rectified flow matching. In NeurlPS, 2024. 2, 5, 6, 7, 9, 22, 26, 27,28 28
[14] Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. Taming multimodal joint training for high-quality video-to-audio synthesis. In Proc. CVPR, 2025. 3, 6, 7, 22, 25, 26, 27, 28
[15] Yaron Lipman, Ricky T. Q Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In Proc. ICLR, 2023, 2, 21
[16] Zhicong Tang, Jianmin Bao, Dong Chen, and Baining Guo. Diffusion models without classifier- free guidance. arXiv preprint arXiv:2502. 2544, 2025. 2, 3, 6, 8, 21, 22, 25
[17] Tiantian Geng, Teng Wang, Jinming Duan, Runmin Cong, and Feng Zheng. Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline. In Proc. CVPR, 2023. 7,8
[18] Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation. In Proc. ICASSP. 2023.
[19] Trung X Pham, Kang Zhang, and Chang D Yoo. Cross-view masked diffusion transformers for person image synthesis. In Proc. ICML, 2024. 2

---

[20] Axi Niu. Trung X Pham, Kang Zhang, Jinqiu Sun, Yu Zhu, Qingsen Yan, In So Kweon, and Yanning Zhang. Acdmsr: Accelerated conditional diffusion models for single image super-resolution. IEEE Transactions on Broadcasting, 2024. 2
[21] William Peebles and Saining Xie. Scalable diffusion models with transformers, In Proc. ICCV, 2023. 2
[22] Shanghua Gao, Pan Zhou, Ming-Minnn Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer. In Proc. ICCV, 2023. 2
[23] Trung X. Pham, Kang Zhang, and Chang D. Yoo. Cross-view masked diffusion transformers for person image synthesis. In Proc. ICML, 2024. 2
[24] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 3
[25] Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. Proc. ICLR, 2025. 3, 5, 25
[26] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo. Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Ya-Yao Huang. Shang-Wor Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. Featured Certification. 3,
[27] Chaoning Zhang. Kang Zhang, Chenshuang Zhang, Axi Niu, Jiu Feng, Chang Yoo, and In So Kweon. Decoupled adversarial contrastive learning for self-supervised adversarial robustness. In Proc. ECCV, pages 725-742, 2022.
[28] Chaoning Zhang, Kang Zhang, Trung X. Pham, Axi Niu, Zhinan Qiao, Chang D. Yoo, and In So Kweon. Dual temperature helps contrastive learning without many negative samples: Towards understanding and simplifying moco. In Proc. CVPR, pages 14441-14450, June 2022. 3
[29] Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X. Pham, Chang D. Yoo, and In So Kweon. How does simsiam avoid collapse without negative samples? unified understanding with self-supervised contrastive learning. In Proc. ICLR, 2022.
[30] Jingfeng Yao. Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming opti- mization dilemma latent diffusion models. Proc. CVPR, pages 15703-15712, 2025. 3, 5
[31] Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers. arXiv preprint arXiv:2504. 10483, 2025.
[32] Moayed Haji-Ali, Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Alper Canberk, Kwot Sin Lee, Vicente Ordonez, and Sergey Tulyakov. Av-link: Temporally-aligned diffusion features for cross-modal audio-video generation. In Proc. ICCV, 2025. 3
[33] Manjie Xu, Chenxing Li, Xinyi Tu, Yong Ren, Rilin Chen, Yu Gu, Wei Liang, and Dong Yu. Video-to-audio generation with hidden alignment. arXiv preprint arXiv:2407.07464 2024. 3
[34] Heng Wang, Jianbo Ma. Santiago Pascual, Richard Cartwright, and Weidong Cai. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models. In Proc. AAAI, 2024. 3
[35] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Fijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In Proc. ECCV, 2024. 4, 8, 21, 22, 25
12

---

[36] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, Red Hook, NY. USA, 2020. Curran Associates Inc. 4
[37] Haohe Liu, Zehua Chen, Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. In Proc. ICML, 2023. 4, 22
[38] Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. /EEE/ACM Transactions on Audio, Speech, and Language Processing, 2024. 4, 5
[39] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 2018. 7, 23
[40] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-aude pretraining with feature fusion and keyword- to-caption augmentation. In Proc. ICASSP, 2023. 8
[41] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Proc. ICML, ICML'23, JMLR.org, 2023. 10
[42] Joshua Tian Jin Tee, Kang Zhang, Chanwoo Kim, Dhananjaya Nagaraja Gowda, Hee Suk Yoon, and Chang D. Yoo. Physics informed distillation for diffusion models, 2024. 10
[43] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proc, ICLR, 2019.
22
[44] Qiuqiang Kong, Yin Cao, Turab lqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley, Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Trans. Audio, Speech, and Language Processing, 28:2880-2894, November 2020. 23, 27
[45] Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Chan- ning Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In Proc. ICASSP, pages 776-780, 2017. 27
[46] Khaled Koutini, Jan Schlüter, Hamid Fgghbal-zadhh, and Gerhard Widmer. Efficient training of audio transformers with patchout. In Proc. Interspeech, pages 2753 ISCA, 2022. 27
[47] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung. Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurlPS, page 2234 2242,2222222 2016. 27
[48] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proc. CVPR, 2023. 27
[49] Ilpo Viertola, Vladimir lashin, and Esa Rahtu. Temporally aligned audio for video with autoregression. In Proc. ICASSP, pages 1-5. IEEE, 2025. 27
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide extensive experiments and detailed analysis support our claims in Section Experiment.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussion the limitations in the Limitation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides all necessary details reproduce the main experimental results, including baseline hyperparameters and training or inference settings. Some minor details that may not be included in this paper are the referenced works.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The datasets are publicly available, and we will release our code upon accep- tance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ wwwuddes/CoddSuumiisssss...
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/
wtt.ssabbisiiooooooooooooo.ooccl for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We show the details in the experiments section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eπrors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We use single A100 GPU for experiments; details are also shared in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https : http//eerrpp.../ catp..cc/puuiee wtw.bi/ii//thiisudelellllellllee
Answer: [Yes]
Justification: We confirm.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there is societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cited all datasets properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is required.
Answer: [No]
Justification:
We used LLM only for editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: /tt::://nnureie /ct.o//aeeeecec////000000 /202//2222222/// for what should should not be described.
20

---

Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation
Appendix
Model Guidance: Derivation and Integration
In our framework, MGAudio, we incorporate the Model Guidance (MG) strategy proposed by [16] as training- time objective that complements the conventional Classifier-Free Guidance (CFG). While CFG modifies the inference trajectory by combining outputs from conditional and unconditional branches, MG introduces auxiliary loss that explicitly accounts for the posterior dependency between conditions and noisy inputs. This enables improved condition alignment without incurring additional inference-tine overhead.
A.1 Model Guidance for Flow Matching
Our model is based on the flow-matching mechanism [15] for the denoising process, main backbone being SiT [35]. Flow-Matching aims learn a conditional velocity field (x,, 1, i7) given condition
V (condense feature vector extracted from the silent video), such that it matches the ground-truth
flow:
Xo E. dx: dt
U1 4x xb)
(8)
where I)) and (1 t)xe + tc. The standard flow-matching loss is:
LFM = Ex0. [xo ū) ue (x+, xo)
While Eq. 9 learns to model the conditional distribution (x, i). diffusion models often underutilize the conditioning information practice. To address this, MG proposes include the posterior term (o xt). resulting in the joint distribution:
pe (x: 0) = (x (T)
(o xx)"",
(10)
where is the guidance scale controlling the strength of the posterior term. The score of this joint distribution becomes:
Xx, ( Using Bayes' rule, we express the posterior as:
0) Xx, loggoo(x, ū) w Vx, log xt).
(11)
Po xx) x log () log (xxe),
(12)
which leads to the posterior gradient:
Vx, log x2) x u 4, ug (x,, J),
(13)
where ug(x,, f 0) denotes the velocity predicted without conditioning. This gives rise to the modified target velocity:
u' u w sg (ug (x,, U) ug (xxx, 0))
(14)
where Xo is the ground-truth velocity and sg(-) denotes the stop- stop-gradien gradient operation, used to stabilize training. The final Model Guidance loss is:
CMG = E Exo.c.t. of t, - u'112 -u'll""
(15)
21

---

Table 7: MGAudio on VGGSound. Audio generation quality across lfte-of-theearr methods on the VGGSound test set is presented. Bold indicates the best performance, and an underline denotes the second-best CFG: Classifier-Fres Free Guidance, MG: Model- Guidanee " "We train MMAudio from scratch solely on the VGGSound dataset, and set the text input to None during inference for faimess..
Method
Train Type
Inference Type
FAD1
FD1
1St
KL.
Align. Acc.t
Diff-Foley [NeurPPS 23] [9]
CFG
CFG
6.25
23.07
10.85
3.18
93.94
See and Hear [CVPR'24] [10]
CFG
CFG
5.51
26.60
5.47
2.81
58.14
FRIEREN [NeurIPS'24] [13]
CFG
CFG
1.38
12.36
12.12
2.73
97.25
MDSGGn [ICLR"25] [12]
CFG
CFG
1.40
17.42
9.66
2.84
96,88
MMAudio [CVPR'255 "5]" [14]
CFG
CFG
0.71
6.97
11.09
2.07
92.28
MGAudio
MG
No CFG
0.80
7.89
9.90
2.77
90.80
MGAudio
MG
CFG
0.40
6.16
12.82
2.76
95.65
A.2 Inference-Tine Integration of CFG into MG-Trained Models
At inference, integrate classifier-free guidance (CFG) explicitly control generation fidelity. Although the MG loss is utilized solely during training to enhance model sensitivity to conditioning signals, employing CFG at inference further boosts fidelity without additional retraining. Empirical results demonstrate that combining MG with CFG achieves superior perceptual quality and alignment, effectively balancing diversity and fidelity across key metrics such as FAD.
Tab. 7 shows that our best MGAudio model (131M parameters), trained for 1.IM steps without CFG, achieves a competitive FAD score of 0.80 and alignment accuracy of 90.80%, outperforming strong baselines like MDSGen (131M parameters) and FRIEREN (157M parameters), and closely approaching MMAudio (157M parameters), all utilizing CFG inference. Notably, applying CFG significantly enhances our model's 's performance, achieving state-of- the- results with an FAD of 0.40 and alignment accuracy of 95.65%, underscoring CFG's critical role in improving alignment quality.
B More Training Details
Training Configurations. We adopt the same model architecture and hyperparameters as SiT [35]. For all experiments, the guidance scale factor in Eq. 6 of the main paper is set to = 1.45, following the default in [16]. For the initial 10,000 training steps, we set w = to stabilize early training. All models are optimized using AdamW [43] with weight decay of 0 and betas (0.9, 0.999). We use a constant learning rate of x 10 -4-4, a batch size of 64, and train for 1.1 million steps in the main experiments (Table 1 and Figure 5). Data augmentation follows the same protocol as Diff-Foley [9]. We do not tune learning rates, apply warm-up or decay schedules, modify Adam W parameters, add extra augmentations, or apply gradient clipping. For ablation studies in Sections 4.3. 4.3.1-4.3.5, models are trained for 300,000 steps with a batch size of 16, while keeping all other settings unchanged.
Sampling Configurations. We utilize an exponential moving average (EMA) of model weights with a decay factor of 0.9999 and employ EMA checkpoints for sampling, which consistently achieving improved performance. By default, sampling is performed using the Fuler-Maruyama solver with 50 denoising steps and a classifier-ff
guidance (CFG) value of 1.45. For the comparative analysis presented in Figure 5 of the main paper, we adopt the Euler solver with reduced step count of 25, aligning our methodology with that of FRIEREN [13] to ensure a fair and accurate comparison.
Metric Calculation We utilize audio evaluation tools provided by AudioLDM [37] for FAD, FD, IS, and KL. For alignment accuracy, we use the code provided by Diff-Foley [9].
Architectural Configurations. We adopt the same transformer architecture as SiT [35], experiment- ing with four model scales: MGAudio-S, B, L,, XL. varying in parameter count and computational cost. A detailed summary is provided in Tab. 8.
C Learned Data Distribution Comparison
We present higher-resolution comparison of the generated and real audio sample distributions in Fig. 7. For each audio sample, both generated and real, we first extract sample-wise logits using
22

---

Table 8: Transformer Model Configurations of MGAudio with different model size.
Model
Layer N
Hidden dimension
Head
Patch size
Parameters (M)
MGAudio-S
12
384
2
MGAudio-B
12
768
12
2
131
MGAudio-L.
24
1024
2
464
MGAudio-XL
28
1152
16
2
680
playing djembe
playing djembe
playing comet
playing
scuba diving
scuba diving
black capped
black capped
chickadee calling
chickadee calling
FAD=6.25
FAD= 1.38
Diff-Foley (NeurIPS23)
Frieren (NeurIPS24)
playing djembe playing scuba diving
playing djembe playing scuba diving
black capped chickadee calling
black capped chickadee calling
FAD=1.4
FAD=0.4
MDSGEN (ICLR25)
MGAudio (Ours)
Figure 7: Audio Distribution. MGAudio generates audio samples that more closely align with the target distribution compared other methods. For example, samples for classes "playing djembe" (red points) and "scuba diving" (black points) are tightly clustered around the center of the real sample distribution.
the PANNs model [44]. These logits are then projected into a 2D space using UMAP [39] for visualization. The resulting embeddings are used to plot the distribution of generated samples, along with contour plots of the real samples. The filled contours in Fig. 7 represent the density of real audio samples from VGGSound, focusing on four selected classes: playing djembe, playing cornet, scuba diving, and black-capped chickadee calling.
D Effect of Batch Size
We analyze the influence of varying batch sizes on the stability and quality of MGAudio training. As shown Tab. 9, larger batch sizes generally improve key performance metrics, particularly FAD, FD, IS, and alignment accuracy, suggesting enhanced audio fidelity, diversity, and alignment quality.
23

---

Method For our main experiments, we select batch size of 64, which yields the best overall performance metrics across all evaluated criteria. For computational efficiency our extensive ablation studies, we opt for smaller batch size of 16, which provides reasonable performance metrics despite some degradation, enabling faster experimentation cycles.
Table 9: Impact of Batch Size on MGAudio Performance Metrics
Batch Size
FAD
FD
IS1
KLJ
Align. Acc.
16
1.14
13.09
8.35
2.70
93.67
32
0.71
12.07
8.61
2.72
94.41
64
0.51
9.19
10.25
2.75
95.14
E Effect of Sampling Step
We evaluate the performance of MGAudio under different sampling step budgets: 25, 50, 120, 250, and 500 steps. As shown Fig. 8, MGAudio achieves optimal performance for major metrics such FAD, FD, and KI. 25-50 sampling steps. This indicates that further increasing sampling steps does not necessarily enhance results and may incur unnecessary computational overhead.
Align (%)
20.0
75.0
3.5
10.0
80.0
50.0
10.0
3.0
60.0
25.0
0.0
40.0
50
120
500
50
120
500
5
120
500
50
120
500
50
120
500
Sampling Steps
Sampling Steps
Sampling Steps
Sampling Stops
Sampling Stops
Figure 8: Performance Metrics of MGAudio Across Different Sampling Steps.
Furthermore, to ensure a fair comparison across methods, we evaluate all models using identical sampling steps or number of forward evaluation (NFE) while adopting the default CFG value reported in each method. As shown in Table 10, our approach achieves competitive or superior results under these standardized and transparent settings. Notably. even NFE 100 MGAudio maintains inference times comparable to FRIEREN and MDSGen while being substantially faster than MMAudio. Moreover, at CFG 1.0) ), our model surpasses both FRIEREN and MDSGen in overall quality with only 50 NFEs, demonstrating its efficiency and strong generation capability.
Table 10: Comparison of inference efficiency and quality under different CFG values and sampling settings.
CFG
Steps
NFE
FAD
FD J
IS
Time (s) 1
FRIEREN
4.50
25
50
1.38
12.36
12.12
0.20
MDSGen
5.00
50
1.40
17.42
9.66
0.24
MMAudio
4.50
25
50
0.71
6.97
11.09
0.98
MGAudio (Ours)
1.00
25
25
1.40
9.04
9.19
0.08
MGAudio (Ours)
1.45
25
50
0.68
7.49
12.11
0.15
MGAudio (Ours)
1.00
50
50
0.80
7.89
9.90
0.15
MGAudio (Ours)
1.45
50
100
0.40
6.16
12.82
0.31
F Effect of Longer Training
To study the effect of extended training, we continue training MGAudio up to 1.7M steps. shown in Fig. 9, the model continues to improve steadily with longer training, with performance metrics stabilizing beyond 1.IM steps. We select the 1. M-step checkpoint as our final model to balance training efficiency and generation quality.
24

---

FAD
FD
15
KL
Align acc (%)
96.0
1.5
15.0
12.0
2.9
94.0
10.0
1.0
2.8
92.0
0.6
0.6 Training Steps (x11P)
1.0 Training Steps (x10*)
1.0 Training Steps (x10*)
Training Stops (x107)
Training Steps (x101)
Figure 9: Longer Training Effect. We plot the evolution of five evaluation metrics, FAD, FD, IS, KL divergence, and alignment accuracy, over training steps. MGAudio exhibits consistent improvements across all metries during the first 1M steps, after which most metrics begin to saturate. Notably, FAD and FD steadily decrease, and IS and alignment accuracy improve, indicating that both perceptual quality and semantic consistency benefit from prolonged training.
Interestingly, prior work based on diffusion models such MDSGen [12] reports that longer training can lead to degraded performance, likely due to overfitting or mode collapse. In contrast, our method maintains or improves quality throughout, highlighting its robustness and the effectiveness of the model-guided training objective with flow-matching learning.
G Robust Learning from Noisy or Limited Data
We observe that MGAudio maintains strong performance even when trained with only 10% of the VGGSound dataset, suggesting improved robustness to limited and potentially noisy training data. This robust learning capability under constrained data conditions has not been observed in prior Vision Model Guidance [16].
To verify this observation, train MGAudio (with and without AMG loss) and MMAudio using the same 10% subset of VGGSound under identical settings (IM steps). As shown in Table 11, MGAudio with the AMG loss substantially outperforms both its ablation and MMAudio across all metrics, indicating that the AMG loss may promote more stable and data-efficient learning.
Table 11: Performance Comparison when models train with of VGGSound dataset.
Method
AMG Loss
FAD
FD4
Ist
KI,
MMAudio [14]
2.81
14.28
10.37
2.87
MGAudio
32.84
123.14
1.09
4.34
MGAudio
V
0.82
8.13
10.68
2.67
Mechanistically, we hypothesize that AMG's self-distilled flow targets act dynamic regularizer: during training, the model infuses its own predicted flow (conditioned on the visual cue) into the target flow, effectively "rectifying" noisy or weak samples. This process stabilizes learning and yields stronger generalization under data scarcity.
H Effect of Representation Alignment Weight
We investigate the effect of varying the weight the audio representation alignment loss, following the setup in prior work [25]. As shown in Table 12, the model achieves the best perceptual quality (lowest FAD) when
0.5. Increasing beyond this value slightly improves FD and KL but leads to degradation in perceptual realism. Overall, X 0.5 provides balanced trade-off between perceptual quality and distributional fidelity, and the model remains stable across reasonable range of X. Therefore, we adopt 0.5 as the default setting in all subsequent experiments.
I Sampler Investigation
We investigate two widely-used sampler families: deterministic ODE solvers (Fuler integrator) and stochastic SDE samplers (Euler-Maruyama integrator). Following prior works [35, 16], both samplers
25

---

Table 12: Ablation study on X, the weight of the audio representation alignment loss.
Metric
À = 0.25
À = 0.5
À = 0.75
À = 1.0
FAD4
0.74
0.71
0.78
0.81
FD4
13.48
12.07
11.69
11.64
KL
7.75
8.61
8.62
8.62
evaluated using a consistent 50-step sampling scheme. As presented in Tab. 13, the SDE-based Marerr-aauuyama sampler achieves better overall performance, indicating superior audio fidelity and diversity compared to the deterministic Fuler sampler. Thus, for our main experiments, we utilize the Maruyama integrator due to its optimal balance between realism and variation.
Table 13: Performance Comparison between ODE Euler and SDE Euler-Maruyama Samplers.
Sampler
Type
FAD
FD
ISt
KLJ
Align. Acc. (%)
Euler
ODE
0.58
6.71
12.47
2.69
95.98
Euler-Maruyama
SDE
0.40
6.16
12.82
2.76
95.65
Video Conditioning Integration Styles
Our default conditioning strategy follows MDSGen [12], which aggregates video frames into single global feature vector and modulates the audio generator AdaIN. This approach is computationally efficient but may compromise temporal precision. Although temporal modeling is not the primary focus this work, we explore alternative conditioning strategies to examine their potential impact on generation quality.
To ensure fair comparison, integrate each conditioning strategy into our framework by selectively enabling its corresponding temporal alignment mechanism while removing others. All models are trained under identical configurations (batch size of 64, 300k iterations) to isolate the effect of each conditioning design. The core part for each strategy is:
MDSGen-style (ours): Global frame aggregation with AdaIN modulation.
FRIEREN-style [13]: Temporal interpolation of frame features to match the mel-spectrogram
length, followed by channel-wise concatenation.
MMAudio-style [14]: Multimodal transformer with joint audio-video attention.
We evaluate each strategy using MMAudio's 's metric suite, which covers distributional fidelity (FD-
VGG, FD-PASST), semantic consistency (IB-Score), and temporal alignment (DeSync):
Table 14:
Comparison of video conditioning strategies. Lower is better for FD and DeSyne: higher better for IB-Score.
Conditioning Style
FD-VGG 4
FD-PASST 1
IB-Score
DeSync |
MDSGen-style
0.5411
74.092
24.58
1.242
FRIEREN-style 0.8293
90.668
23.56
0.938
MMAudio-style 0.6327
81.732
25.05
0.961
Findings. As shown in Table 14, we have the following findings: (1) MDSGen-style conditioning achieves the best distributional fidelity, likely due to its compact and stable global representation. (2) FRIEREN-style provides the strongest temporal alignment, benefiting from explicit frame-level interpolation. (3) MMAudio-style yields the highest semantic consistency, owing to its cross-modal attention design.
These results suggest that temporal conditioning design is largely orthogonal to our core contribu- tions-AMG and dual-role alignment. We adopt the MDSGen-style configuration for its simplicity
26

---

and strong overall performance, but acknowledge that incorporating finer temporal granularity remains a promising direction for future work.
Table 15: Training Time for Different MGAudio Model Sizes. Wall-clock training durations single A6000 GPU for 300,000 iterations using precisson training and a batch size of 16.
Model
Training Time (hours)
MGAudio-S
16.6
MGAudio-B
25.7
MGAudio-L.
57.0
MGAudio-XL.
77.1
K
Training Time Analysis
Tab. 15 details the training durations for various MGAudio model sizes (small extra-large). Training was conducted on a single A6000 GPU using mixed-precision optimization for 300,000 iterations with batch size of 16. We select MGAudio-B performance and computational efficiency. The results highlight the increasing computational requirements associated with larger model architectures, illustrating practical considerations in choosing model complexity.
primary model due its balanced trade-off between
L Additional Metrics Comparison
To comprehensively evaluate audio quality beyond standard metrics such FAD and FD, follow [14] and include additional metrics shown in Table 16. These metrics encompass Fréchet Distance (FD) computed using multiple audio models (VGG [45], PANNS [44], PaSST [46]), and Kullback-
Leibler divergence (KL) evaluated using PANNs and PaSST embeddings for distribution matching between generated audio and ground truth audio. And Inception Score (IS) [47] assess audio
quality. Semantic aligmment is evaluated using IB-Score, where ImageBind [48] extracts visual and
audio features, computing average cosine similarity as in [49]. Audio-visual synchrony is measured via DeSync scores using Synchformer to estimate temporal alignment errors.
As detailed in Table 16, MGAudio consistently outperforms competing methods in key distribution metrics (FD-VGG and FD-PANNs) and achieves the highest audio quality according to IS with CFG of 6.0. Additionally, despite MMAudio achieving the lowest DeSync, MGAudio demonstrates strong semantic alignment (IB-Score), particularly at higher CFG values, without the need for explicit test-time optimization done by See and Hear. Furthermore, MGAudio accomplishes these superior results with fewer parameters (131M) compared to MMAudio (1500)11515 (157M), underscoring its efficiency and effectiiveness.
Table 16: Metrics from MMAudio [14], MGAudio on VGGSound. Audio generation quality across saate-of-the-and methods on the VGGSound test set is presented. Bold indicates the best performance, and an underline denotes the recond-best "We train MMAudio from scratch solely on the VGGSound dataset, and set the text input to None during inference for fairness.
KLJJ
ISt PANNs
Method
VGG
PANNs
PaSST
PANNs
PaSST
IB-Score" (s)
DeSyne]
Params
Diff Foley [Neur1PS [9]
6.25
23.07
358,92
3.17
3.04
10.77
19.88
0.91
860M
See and Hear [CVPR'24] [10]
5.51
26.60
227.67
2.82
2.78
5.71
36.11
1.20
1099M
FRIEREN 24] [13]
1.38
12.36
107.57
2.72
2.84
12.12
22.83
0.85
157M
MDSGGen [CCRR25]]
1.40
17.42
114.27
2.83
2.80
9.68
22.53
1.23
131M
MMAudio [CVPR'25]* [14]]
0.71
6.97
51,00
2.08
1.97
11.08
27.35
0.50
157M
MGAndio, CPG 1.45
0.40
6.16
83.53
2.75
2.56
12.80
26.53
1.22
131M
MGAniio CFG
2.23
13.65
77.41
2.75
2,62
14.41
27.13
1.23
131M
MGAudio, 660
1.50
69.75
2.62
17.36
28.77
1.21
131M
27

---

M Human Preference Evaluation
We conducted human evaluation compare MGAudio with baselines (FRIEREN [13] and MMAudio [14]) using a 5-point Likert scale. Three participants were asked to first watch the silent video and then listen to the generated audio clips from each method without being informed of their source. Participants rated each clip from I to 5, where "1" indicates poor audiovisual alignment and unnatural sound, and '5" indicates strong synchronization and natural acoustic quality.
As shown in Table 17, listeners consistently preferred MGAudio over both FRIEREN and MMAudio, indicating that our method produces audio that better aligns with visual events and is more perceptually natural.
Table 17: Average human preference scores (1-5) for generated audio across different methods.
Method
FRIEREN
MMAudio
MGAudio (ours)
Preference 3.44 + 0.46
3.89 + 0.47
4.38 = 0.39
N Qualitative Comparisons
We present qualitative comparisons between MGAudio and prior state-of-the-an methods on randomly selected videos from the VGGSound test set. Each example shows input video frames followed by the mel-spectrograms generated by different models. As illustrated in Fig. 10-13, MGAudio produces spectrograms with more realistic structure, temporal continuity, and acoustic richness, closely resembling the ground truth. Corresponding audio samples are included in the supplementary materials. These results highlight the benefits of the model-guided objective in generating coherent and semantically accurate audio. Beside the samples show at here, we have included more samples generated from different method in the supplementary material.
28

---

Inpourt
DifiFolyy
FRIEREN
NOSGen
Ours
Figure 10: Qualitative Comparison of Mel-Spectrograms. The example is from the VGGSound test set (video: "ABrklgn3N4A_0000 (00044_0000028m"p""") depicting a cat growling.
29

---

Impout
cifirley
FRIENEN
NDSGe
Figure 11: Qualitative Comparison of Mel-Spectrograms. The example is from the VGGSound
test set (video: (GO2TY8KIJJ140000 101.00000000p//00 depicting a person playing steel guitar, slide guitar.
30

---

Dif--Fllm Foley
FRIEREW
MDSSen
MMAudioo
Ours
Figure 12: Qualitative Comparison of Mel-Spectrograms. The example is from the VGGSound test set (video: (000://0pXXX000000mpp0..mp depicting a race car, auto racing.
31

---

Ingont Videoo
cifiFoly
FRIEREN
NOSGes
Ours
Figure 13: Qualitative Comparison of Mel-Spectrograms. The example is from the VGGSound
test set (video: TFTEIR/2ANX0000000444.oo depicting a baltimore oriole calling his mate.
32


Reviews:
[{"review_id": "dhWwtLCRds", "reviewer": "Reviewer_q3sa", "summary": "MGAudio, a model-guidance approach for conditional audio generation, replacing classifier-free guidance (during training). Adapted for video-to-audio, its dual-role alignment sets a new state-of-the-art on VGGSound. The paper is well-written, and shows some promising results.", "questions": "1. The authors mention throughout the paper (even early in the abstract) that they propose a “transformer denoiser”, but in their equation they work with velocity prediction. Can the authors clarify?\n2. Even though the authors mention in L63-74 their contribution explicitly, it's hard to understand what is new in their method, and what was introduced before. Can the authors clarify this?\n3. This paper introduces a new auxiliary loss, with a hyperparameter lambda, i couldn't find an ablation on this, and also not the actual value that was used. \n4. “We introduce MGAudio, a novel framework for video-to-audio generation that replaces the conventional classifier-free guidance (CFG) objective with a model-guidance (MG) objective, leading to more efficient training and improved generation quality.” I find this statement misleading, as the method does use CFG to *sample* to the best of my understanding. Can you clarify if I misunderstood?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n* The paper provides a thorough and convincing experimental evaluation: MGAudio is compared against a strong and recent set of baselines, including Diff-Foley, FRIEREN, and MDSGen, on two standard datasets.\n* The paper is well written and easy to follow.\n\nWeaknesses:\n* Synergy between AMG and CFG: The paper finds that while MGAudio is trained with the AMG objective, applying CFG at inference time further boosts performance. This is somewhat counter-intuitive finding to me. The authors acknowledge this but didnt dig deeper into why.\n* Computational cost: if i understand correctly, the claim about a single forward pass are regarding inference, but the authors do claim the CFG boost performance. It left me confused.\n\nOverall, the results look promising and the method is valid. However, I am not sure if this paper belongs to neurips as the methodology is very modality-specific.", "was_revised": true, "final_justification": "I had some concerns, they were addressed during the rebuttal.", "rebuttal": "We thank the reviewer for recognizing the promise of our results and the validity of our proposed method. Below, we address the weaknesses raised and respond to the reviewer’s questions in detail.\n\n---\n\n> 1. **Weakness 1:** Synergy between AMG and CFG: The paper finds that while MGAudio is trained with the AMG objective, applying CFG at inference time further boosts performance. This is somewhat counter-intuitive finding to me. The authors acknowledge this but didnt dig deeper into why.\n>\n>    **Question 4:** “We introduce MGAudio, a novel framework for video-to-audio generation that replaces the conventional classifier-free guidance (CFG) objective with a model-guidance (MG) objective, leading to more efficient training and improved generation quality.” I find this statement misleading, as the method does use CFG to sample to the best of my understanding. Can you clarify if I misunderstood?\n\nUsing CFG at inference after training with AMG is not contradictory. In fact, the observed performance improvement suggests that AMG **alleviates the training-inference mismatch**, enabling the model to more effectively benefit from CFG guidance.\n\nTo clarify, one of the core motivations for introducing AMG is at L44–45:\n“...training to simulate both conditional and unconditional objectives. While effective, this multi-task setup may dilute the model’s capacity and lead to mismatched sampling behavior at inference.”\nAMG was specifically designed to compensate for this issue. It shows two key benefits in video to audio generation: (1) Mitigating training-inference mismatch by incorporating a CFG-like guidance term into training, making the model more compatible with CFG during inference; (2) Improving generation quality even without CFG by explicitly encouraging perceptual realism during training.\n\nThese two effects are substantiated by our results. As illustrated in Figure 5 and Appendix Figure 4, MGAudio consistently outperforms both SiT and FRIEREN across both CFG and non-CFG settings, underscoring the effectiveness of AMG alone and its compatibility with CFG during inference.\n\nWe acknowledge that this motivation was not sufficiently emphasized throughout the paper, and we thank the reviewer for bringing this to our attention. We will revise the manuscript to make this clearer.\n\n\n---\n\n> 2. **Weakness 2:** Computational cost: if i understand correctly, the claim about a single forward pass are regarding inference, but the authors do claim the CFG boost performance. It left me confused.\n\nWe believe the reviewer’s confusion stems from Lines 100–102, where we state that \"...Vision Model-Guidance [16], achieving state-of-the-art FID on ImageNet with only a single forward pass at each inference step, half the computational cost of CFG.\" This was intended to highlight the **efficiency of vision model guidance (VMG) during inference**, in contrast to Classifier-Free Guidance (CFG), which typically requires two passes.\n\nTo clarify:\n\n- MG (AMG) provides strong performance gains using **a single forward pass** during inference.\n\n- **Our method also supports CFG**, and combining MG with CFG can **further improve quality**.\n\nIn prior work, MG showed strong improvements only when CFG was not used. In contrast, in our V2A setting, AMG consistently improves performance both with and without CFG at inference, as shown in Appendix Table 1 and Figure 4, outperforming the SiT baseline under both configurations.\n\nWe will revise Lines 100–102 to make this distinction clear:\nMG enables efficient single-pass inference and is complementary to CFG, offering improvements in both inference modes.\n\n---\n\n> 3. **Questions 1:** The authors mention throughout the paper (even early in the abstract) that they propose a “transformer denoiser”, but in their equation they work with velocity prediction. Can the authors clarify?\n\nAs noted in L139–140, the transformer in our method can predict either noise or vector fields, depending on the framework. Since we adopt the flow matching approach, it predicts velocity. We agree that the term “transformer denoiser” may be misleading and will revise it to “transformer model” and clarify its role accordingly.\n\n---\n\n> 4. **Questions 2:** Even though the authors mention in L63-74 their contribution explicitly, it's hard to understand what is new in their method, and what was introduced before. Can the authors clarify this?\n\nWe thank the reviewer for highlighting the need to better clarify the novelty of our method. Below, we detail the key contributions of MGAudio and how they differ from prior work:\n\n#### 1. **Dual-Role Alignment of Pretrained Multi-Modal Representations**\n\nOur primary contribution is the **explicit dual-role utilization** of both audio and video branches from a pretrained multimodal encoder (CAVP) to guide generation:\n\n- The **video branch** is used for model-guided training (AMG),\n\n- The **audio branch** provides latent-space supervision via representation alignment (REPA).\n\nWhile prior works like Diff-Foley [19] and MDSGen [12] use the CAVP video encoder for conditioning, they discard the audio branch. Our method is the **first to jointly leverage both branches with distinct and complementary roles**. This dual-role integration improves both generation fidelity and training stability (see Sec. 3.2).\n\nThis design is **non-trivial**: training a strong multi-modal representation model is computationally expensive, and naively using it as a condition vector does not leverage its full potential. We show that learned representation specifically for the target generating modality and guiding modality can be used to guide generation through the combination of model guidance (MG) and representation alignment (REPA). This presents a **promising, general framework for future cross-modal generation tasks**: first, train a powerful multi-modal representation learner, then leverage it for both trajectory guidance and generative modal internal feature alignment.\n\n#### 2. **Robust Learning from Noisy or Limited Data**\n\nWe also show that MGAudio enables training on **limited and noisy AV pairs** by **self-distilling a target flow direction** informed by visual cues. This effect of improved robust learning from noisy or limited data has not been shown in the original vision-model-guidance paper [16]. Unlike CFG, which passively drops conditions, MGAudio actively corrects poor samples by injecting a plausible flow direction—effectively acting as a regularizer. As reviewer yNJc noted, this may explain the model’s ability to learn stable representations even with weak or noisy inputs. We support this with an experiment using only 10% of the VGGSound training set:\n\n**Table 1:** Comparison of fully trained MGAudio (with and without AMG) and MMAudio on the same randomly selected 10% subset of VGGSound.\n| Method  | AMG Loss | FAD ↓ | FD ↓   | IS ↑  | KL ↓ |\n|---------|----------|-------|--------|-------|------|\n| MGAudio (10%) | ✓        | 0.82  | 8.13   | 10.68 | 2.67 |\n| MGAudio (10%) | ✗        | 32.84 | 123.14 | 1.09  | 4.34 |\n| MMAudio (10%) | -        | 2.81  | 14.28  | 10.37 | 2.87 |\n\nThis demonstrates MGAudio's effectiveness in regularizing training under data scarcity and its potential for data filtering or active sample weighting strategies in future work.\n\n---\n\n> 5. **Questions 3:** This paper introduces a new auxiliary loss, with a hyperparameter lambda, i couldn't find an ablation on this, and also not the actual value that was used.\n\nAs noted in Line 193, we set λ = 0.5 by default to weight the auxiliary **representation alignment loss** relative to the main generative objective. This follows the default from prior work [23]. To validate this choice, we conducted an ablation study across λ ∈ {0.25, 0.5, 0.75, 1.0}, using the same architecture and training setup (300k steps, batch size 32).\n\n**Table 2:** Ablation on λ the weight of the audio representation alignment loss.\n| Metric | λ = 0.25 | λ = 0.5 (default) | λ = 0.75 | λ = 1.0   |\n| ------ | -------- | ----------------- | -------- | --------- |\n| FAD ↓  | 0.74     | **0.71**          | 0.78     | 0.81      |\n| FD ↓   | 13.48    | 12.07             | 11.69    | **11.64** |\n| IS ↑   | 7.75     | 8.61              | **8.62** | **8.62**  |\n\nWe observe that λ = 0.5 achieves the best perceptual quality (FAD), while higher λ slightly improves FD and IS at the cost of realism. The results suggest that λ = 0.5 offers a balanced trade-off, and the model remains robust across a reasonable range. We will include this ablation in the appendix of the revised version for clarity."}, {"review_id": "WVOLtX45cX", "reviewer": "Reviewer_SNdj", "summary": "Authors proposed MGAudio, a V2A model that adapts model guidance and diffusion feature alignment (i.e REPA) to import V2A performance. The paper focuses on model guidance as an alternative (or complementary) to using CFG during inference.", "questions": "- I believe replacing the experiment in Sec. 4.3.6 with the corresponding one in Appendix (MGAudio Vs SiT w/CFG) is better to understand the results, since MGAudio and FRIEREN have different training setup and sampling configurations (e.g 25 vs 50 sampling steps for MGAudio)\n- The paper does not discuss the base architecture choices sufficiently. For instance, the author method that to condition on the video inout, they aggregate input frames to a global feature vector that they modulate through AdaIN. How are precise temporal information preserved? \n- Few key V2A work are missing from the discussion (e.g [1,2, 3])\n\n[1] AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation\n[1] Video-to-Audio Generation with Hidden Alignment\n[2] V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models", "limitations": "Yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths**\n- The paper is well written and well structured.\n- Extensive ablations and experiments are provided in the supplementaries \n- The paper explores integrating useful techniques to improve V2A performance. \n\n**Weaknesses**\n- One major weakness of the paper is the limited novelty. It explores already proposed techniques (model guidance and feature alignment) for the task of V2A. \n- The qualitative results in the supplementary are not convincing. The number of provided samples is small (12 samples total), and the results seems to have significantly weaker temporal alignment than SOTA. For example, I found the bln5A2hopKQ_000092.mp4 sample to have worse temporal alignment than FRIEREN.\n- The results of the paper are not conclusive. Table 1 compares with baselines but I could not find any information (even in the supplementaries) about the sampling parameters of these baselines. It is hard to make sense of the results without information about the sampling parameters (e.g how many step / CFG value, etc). \n- Table 3 shows that MGAudio achieves the best results with 10% of the data, that raises questions about the model scalability with respect to the data. Also, since no comparison of a baseline trained with the same amount of data is reported, it hard to attribute this to the noisiness of VGGsounds or the model architecture. \n- The authors motivate the paper with 1) CFG \"may dilute the model's capacity\" 2) inference without CFG. Yet the authors (1) (similar to model guidance) train with a 10% drop of caption similar to CFG and (2) report their main experiment (Table 1) with CFG.  The paper misses comparison with baselines to support the superiority of their model over other common method for inference without CFG (e.g guidance distillation)", "was_revised": true, "final_justification": "The main contribution of the paper is applying existing techniques from the T2I domain to the V2A and showing its effectiveness.\nAfter discussing my concerns with the authors, my main concerns regarding the novelty of the proposed method (review q3sa and 3BgF shared the same concern) and the poor qualitative results still remain. \n\n\nSome of my concerns regarding clarity and model design were resolved. The authors did an extensive rebuttal and provided multiple experiments to strengthen their claims. Therefore, I decided to raise my score to boarderline reject.", "rebuttal": "We thank the reviewer for the thoughtful and detailed feedback. We address each concern below and will reflect the necessary clarifications in the final version.\n\n---\n\n> 1. **Weaknesses 1:** A major weakness is limited novelty, as the paper applies existing techniques (model guidance and feature alignment) to V2A.\n\nBelow, we detail the two key novelties of MGAudio:\n\n#### 1. **Dual-Role Alignment of Pretrained Multi-Modal Representations**\n\nOur primary novelty lies in **how we integrate and assign distinct roles to multi-modal representations** from a pretrained encoder to guide both the denoising trajectory and the internal latent space:\n- Audio Model Guidance (AMG): Uses audio-aligned video features to guide denoising via a CFG-like term.\n- Representation Alignment (REPA): Uses video-aligned audio features to constrain the generative model’s latent space.\n\nThis dual-role use of a single pretrained multi-modal encoder, for both trajectory guidance and latent alignment, is novel in V2A. Prior work treats video features as static conditions without leveraging them as active training signals or imposing latent constraints. Our approach improves performance and data efficiency, especially in low-resource settings.\n\nThis design is **non-trivial**: naively conditioning on pretrained encoder outputs underutilizes their capacity. By explicitly separating guiding (AMG) and aligning (REPA) roles, we robustly steer generation across modalities, offering a general framework for cross-modal generation.\n\n#### 2. **Robust Learning from Noisy or Limited Data**\n\nMoreover, **AMG enables training on low-quality or noisy AV pairs**. This effect of improved robust learning from noisy or limited data has not been shown in the original vision-model-guidance paper [16]. As reviewer yNJc noted, AMG’s self-distilled targets act as a regularizer—adjusting the predicted flow via visual cues during training to rectify, rather than ignore, poor samples. This property could help future work in identifying or mitigating low-quality data, improving robustness and efficiency.\n\n\n---\n\n> 2. **Weaknesses 2:** The qualitative results in the supplementary are not convincing.\n\nDue to supplementary size limits, we included the maximum allowed 12 samples, selected randomly without cherry-picking, which may result in some weaker examples.\n\nTo address this, as suggested by reviewer yNJc, we conducted a human study comparing MGAudio and baselines using a 5-point Likert scale. Results are summarized below:\n\n|Method|FRIEREN|MMAudio|MGAudio (ours)|\n|-|-|-|-|\n|Preference|3.44 ± 0.46|3.89 ± 0.47|4.38 ± 0.39|\n\nListeners consistently preferred MGAudio. We will include full study details in the appendix and summarize key findings in the main paper.\n\n---\n\n> 3. **Weaknesses 3:** Missing detailed sampling parameters (e.g., number of steps, CFG value) for compared methods.\n\nWe thank the reviewer for pointing this out. We summarize the sampling parameters for all compared methods in the table below. The metric values are also in Table 1 of our Appendix.\n\n|Method|CFG value|Sampling steps|NFE|FAD|FD|IS| Time (s) |\n|-|-|-|-|-|-|-|-|\n|FRIEREN       | 4.50|25|50 | 1.38 | 12.36 | 12.12 | 0.20     |\n|MDSGen        | 5.00|25|50 | 1.40 | 17.42 | 9.66  | 0.24     |\n|MMAudio       | 4.50|25|50 | 0.71 | 6.97  | 11.09 | 0.98     |\n|MGAudio (Ours)| 1.00|25|25 | 1.40 | 9.04  | 9.19  | **0.08**     |\n|MGAudio (Ours)| 1.45|25|50 | 0.68 | 7.49  | 12.11  | 0.15     |\n|MGAudio (Ours)| 1.00|50|50 | 0.80 | 7.89  | 9.90  | 0.15     |\n|MGAudio (Ours)| 1.45|50|100| **0.40** | **6.16**  | **12.82** | 0.31     |\n\nOur model achieves competitive or superior results under fair and transparent settings. Notably:\n\n* Even at NFE = 100, our model's inference time remains comparable to FRIEREN and MDSGen and is significantly faster than MMAudio.\n\n* At CFG = 1.0, our model achieves stronger performance than both FRIEREN and MDSGen, with only 50 NFE.\n\nWe will include this table and detailed configuration information in the final version for completeness and reproducibility.\n\n---\n\n> 4. **Weaknesses 4:** Table 3 shows MGAudio excels with 10% data, but no baseline trained on the same subset is reported, leaving scalability and data noise effects unclear.\n\nWe appreciate the reviewer’s concern. To address this, we conducted additional experiments where we train both MGAudio (with and without AMG loss) and MMAudio using the same randomly selected 10% subset of VGGSound, under the same training schedule (1M steps) and other settings unchanged.\n\nWe conducted additional experiments training MGAudio (with and without AMG loss) and MMAudio on the same 10% VGGSound subset.\n\n\n| Method  | AMG Loss | FAD ↓ | FD ↓   | IS ↑  | KL ↓ |\n|-|-|-|-|-|-|\n| MGAudio (10%) | ✓        | **0.82**  | **8.13**   | **10.68** | **2.67** |\n| MGAudio (10%) | ✗        | 32.84 | 123.14 | 1.09  | 4.34 |\n| MMAudio (10%) | -        | 2.81  | 14.28  | 10.37 | 2.87 |\n\nResults show **MGAudio with AMG loss outperforms both its ablation and MMAudio**, confirming that improvements stem from AMG’s effective regularization and stable learning, not just data noise.\n\nWe will include this table and clarify MGAudio’s strong generalization in low-data regimes, highlighting its value for real-world scenarios with limited high-quality AV data.\n\n---\n\n> 5. **Weaknesses 5:** The authors motivate the paper with concerns about CFG but still (1) drop conditions during training (like CFG) and (2) report results with CFG. No comparison is made with other CFG-free methods (e.g., guidance distillation).\n\nWe clarify our motivation and design rationale below.\n\nOur motivation (L45–46) is to address the **training–inference mismatch in CFG**, where models are trained with dropped conditions but sampled with strong guidance—potentially leading to inconsistency. Our proposed **Audio Model Guidance (AMG)** resolves this by **explicitly optimizing model prediction towards the CFG-like guided trajectory**, ensuring alignment between training and inference.\n\nAlthough AMG drops 10% of conditions during training, the purpose differs from CFG: these samples are used to compute a **self-distilled guidance target**. This helps the model learn a **consistent denoising direction**, improving stability.\n\nAs shown in Figure 5, AMG performs strongly without CFG, and applying the same CFG scale used during training yields further gains—**without introducing mismatch**.\n\n\n---\n\n> 6. **Questions 1:** Replacing the experiment in Sec. 4.3.6 with the one in the Appendix (MGAudio vs. SiT w/CFG) would better clarify the results, since MGAudio and FRIEREN differ in training and sampling setups.\n\nWe agree. Comparing MGAudio and SiT w/CFG, which **share the same training and sampling settings**, offers a more controlled analysis of model-guided training.\n\nTo improve clarity, we will replace the current plot with the SiT vs. MGAudio w/CFG comparison in the main paper and move the FRIEREN comparison to the supplementary.\n\n---\n\n> 7. **Questions 2:** The paper lacks sufficient discussion on base architecture choices. How is temporal information preserved when using global video features and AdaIN?\n\nOur default setup follows MDSGen [12], aggregating video frames into a **global feature vector** and applying AdaIN-based modulation. While efficient, this may limit temporal precision.\n\nTo better understand this trade-off, we compared three strategies under the same setup (batch size 64, 300k iterations):\n\n1. MDSGen-style (ours): Global frame aggregation + AdaIN.\n\n2. FRIEREN-style: Interpolates video features over time dimension to match the mel-spectrogram resolution and applies channel-wise concatenation.\n\n3. MMAudio-style: Multimodal transformer with joint audio-video attention.\n\nWe evaluated them using MMAudio’s metric suite, that cover distributional (FD-VGG and FD-PASST), semantic (IB-Score), and temporal alignment (DeSync). We implement each of the strategies in our framework and get the results as follows:\n\n| Conditioning Style | FD-VGG ↓   | FD-PASST ↓ | IB-Score ↑ | DeSync ↓  |\n| - | - | - | - | - |\n| MDSGen-style       | **0.5411** | **74.092** | 24.58      | 1.242     |\n| FRIEREN-style      | 0.8293     | 90.668     | 23.56      | **0.938** |\n| MMAudio-style      | 0.6327     | 81.732     | **25.05**  | 0.961     |\n\nFindings:\n\n- MDSGen-style offers the best distributional fidelity, likely due to its compact and stable conditioning signal.\n\n- FRIEREN-style excels at temporal alignment, benefiting from frame-level interpolation.\n\n- MMAudio-style achieves the best semantic alignment, thanks to cross-modal attention.\n\nThese results confirm that temporal conditioning is orthogonal to our main contributions—AMG and dual-role alignment. We chose MDSGen-style for its simplicity and strong overall performance but agree temporal granularity is an important design axis. We will include these findings and further discussion in the Appendix.\n\n> 8. **Questions 3:** Few key V2A work are missing from the discussion (e.g [R1, R2, R3])\n> [R1] AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation \n> [R2] Video-to-Audio Generation with Hidden Alignment \n> [R3] V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models\n\nThank you for highlighting these important works. We will include them in the main paper and related work.\n\n* AV-Link [R1] uses temporally aligned self-attention fusion for bidirectional conditioning, while we focus on explicit alignment at the representation and denoising trajectory levels.\n* Hidden Alignment [R2] improves synchronization via encoder design and augmentations; our method instead applies training-time alignment (REPA) and guided sampling (AMG) to shape denoising with learned multimodal features.\n* V2A-Mapper [R3] maps CLIP to CLAP embeddings for modality bridging, whereas we leverage dual-branch encoders with distinct roles for guidance and alignment."}, {"review_id": "K8DCZm2akV", "reviewer": "Reviewer_yNJc", "summary": "The paper presents a flow-based framework, named MGAudio, for open-domain video-to-audio (V2A) generation. The main idea is using model-guided dual role alignment to allow the generative model to guide itself through a dedicated training objective, unlike previous methods that rely on CFG. The framework integrates three main components: 1) a scalable flow-based transformer denoiser that is based on SiT architecture and learns continous transport directions, 2) a dual-role alignment mechanism where the audio-visual encoder serves both as a conditioning module and as a feature aligner, and 3) a model-guided objective (Audio Model-Guidance) that replaces CFG by using a self-distilled target to refine model predictions. MGAudio achieves SOTA performance on VGGSound and demonstrates good generalization to UnAV-100 benchmark.", "questions": "Besides clarification questions mentioned, I am also interested in the following:\n- Could authors provide a more detailed hypothesis for why the audio domain uniquely benefits from AMG training + CFG inference? What specific characteristics of audio, such as its temporal sensitivity, its often ambiguous mapping to visual cues or something else, will require this combined approach?\n- How does MGAudio's Audio Model-Guidance (AMG) objective specifically contribute to its robustness against noisy/irrelevant data within VGGSound, beyond merely learning efficiently from less data? Does its \"self-distilled target\" implicitly down-weight or ignore poor samples? Could the authors propose specific strategies for future work that could explicitly leverage AMG's properties to actively identify, filter, or mitigate the impact of low-quality audio-visual pairs during training, further enhancing performance and data efficiency?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- MGAudio gets SOTA performance on VGGSound (FAD = 0.4) and outperforms previous CFG baselines.\n- The model demonstrates good generalization to the UnAV-100 benchmark\n- Training on 10% VGGSound dataset still achieves good results. It seems like Audio Model-Guidance does enhance efficient learning\n- Extensive ablation studies are available in both main paper and appendix. The detailed implementation setup is also healthy for reproducibility.\n- The motivation and components in MGAudio framework are clear. Overall I think the model-guided dual-role alignment is a cool idea and makes sense to me. The paper is generally easy to understand.\n\n**Weaknesses**\n- For video-to-audio generation, it is quite important to have a human evaluation. While I really want to believe the MGAudio framework is as good as shown in all quantitative metrics numbers, I was not fully convinced by watching video samples provided in the supplementary material. Therefore, I will recommend authors adding one human study to get the perceptual difference.\n- While the paper presents new adaptations and a coherent combination of various techniques (SiT, CAVP, Model-Guided framework), there is relatively few new concept introduced in this paper.\n- As authors mentioned in the paper, the current framework would not handle human vocalizations well and the model-guided training may introduce confusion when visual semantics are ambiguous or loosely correlated with the target sound.\n- Several typos and clarification:\n  - L131-137: please use accurate notations and clarify all numbers mean even though some of them are straight-forward to guess (e.g., 3x224x224 are frames but what about 64 and 816 in $X$ ?).\n  - L134: pathchified -> patchified\n  - Section 4.3.1: please clarify how 10/30/50% of the training data are being selected. I assume you were selecting them from a uniform distribution. Please also clarify how you define weak or irrelevant audio-visual alignment during manual inspection of the 100 random samples.\n  - Section 4.3.5: Please clarify why the four distinct VGGSound classes are selected (why not other classes?). And also clarify why Figure 4 could tell us the generate sample has stronger diversity.", "was_revised": true, "final_justification": "the rebuttal addressed most of my concerns and answered all my questions. While several established components were used in this work, I agree that the overall design is non-trivial. With the new subjective eval result and clarifications presented in the rebuttal, this is indeed a solid application paper.", "rebuttal": "We thank the reviewer for their thoughtful and detailed feedback. We address each concern below and will incorporate the necessary clarifications in the final version.\n\n---\n\n> 1. **Weaknesses 1:** For video-to-audio generation, it is quite important to have a human evaluation. While I really want to believe the MGAudio framework is as good as shown in all quantitative metrics numbers, I was not fully convinced by watching video samples provided in the supplementary material. Therefore, I will recommend authors adding one human study to get the perceptual difference.\n\nThank you for emphasizing the importance of perceptual evaluation. We fully agree that human judgment is essential for assessing video-to-audio quality. We have conducted a human study where participants are asked to rate audio samples (MGAudio vs. baselines) based on overall preference using a 5-point Likert scale. Below is a summary of the results:\n\n**Table 1:** Overall user preference for generated audio across different methods.\n| Method     | FRIEREN | MMAudio | MGAudio (ours) |\n|------------|---------|---------|----------------|\n| Preference |  3.44 ± 0.46  |   3.89 ± 0.47  |   4.38 ± 0.39    |\n\nListeners consistently preferred MGAudio. We will include full study details in the appendix and summarize key findings in the main paper.\n\n---\n\n> 2. **Weaknesses 2:** While the paper presents new adaptations and a coherent combination of various techniques (SiT, CAVP, Model-Guided framework), there is relatively few new concept introduced in this paper.\n\nWhile we do build upon established components (e.g., SiT, CAVP, model guidance), our contribution lies in the **novel role assignment and integration of multimodal representations** within a **dual-role alignment framework**. Specifically:\n\n* We leverage **CAVP**, a pretrained audio-video-aligned encoder, in two distinct and complementary ways:\n\n  1. **Audio Model Guidance (AMG):** Uses the **video representation** to guide the training through a CFG-like objective on the target flow.\n  2. **Representation Alignment (REPA):** Uses the **audio representation** to regularize the internal latent space of the generative model.\n\n* This **dual use of a single pretrained encoder**—as both a training signal and a regularizer—has not been explored in prior V2A works. Most prior approaches either treat video features merely as conditional inputs or impose no constraint on the model’s internal representations.\n\n* Our experiments show that this synergy significantly improves **semantic relevance**, **temporal synchronization**, and **perceptual quality**, while maintaining high **data efficiency**.\n\nThis design is **non-trivial**: training a strong multi-modal representation model is computationally expensive, and naively using it as a condition vector does not leverage its full potential. We show that learned representation specifically for the target generating modality and guiding modality can be used to guide generation through the combination of model guidance (MG) and representation alignment (REPA). This presents a **promising, general framework for future cross-modal generation tasks**: first, train a powerful multi-modal representation learner, then leverage it for both trajectory guidance and generative modal internal feature alignment.\n\n\n---\n\n> 3. **Weaknesses 3:** As authors mentioned in the paper, the current framework would not handle human vocalizations well and the model-guided training may introduce confusion when visual semantics are ambiguous or loosely correlated with the target sound.\n\n\nWe agree, and appreciate the reviewer’s attention to this limitation. Human vocalization remains challenging for most open-domain video-to-audio models, including FRIEREN, MDSGen, and MMAudio. One key reason is the lack of rich phonetic structure and fine-grained facial cues in the VGGSound dataset, coupled with a limited amount of speech-related data. These factors hinder accurate synthesis of vocal content.\n\nFurthermore, when visual cues are ambiguous (e.g., static scenes, occlusions, or off-screen sounds), it becomes difficult to disambiguate the correct audio modality. Addressing these issues, for instance, by incorporating visual speech encoders or training on speech-rich datasets, is an important future direction.\n\n---\n\n> 4. **Weaknesses 4:** Several typos and clarification:\n> - L131-137: please use accurate notations and clarify all numbers mean even though some of them are straight-forward to guess (e.g., 3x224x224 are frames but what about 64 and 816 in X?).\n> - L134: pathchified -> patchified\n> - Section 4.3.1: please clarify how 10/30/50% of the training data are being selected. I assume you were selecting them from a uniform distribution. Please also clarify how you define weak or irrelevant audio-visual alignment during manual inspection of the 100 random samples.\n> - Section 4.3.5: Please clarify why the four distinct VGGSound classes are selected (why not other classes?). And also clarify why Figure 4 could tell us the generate sample has stronger diversity.\n\n* **L131–137**: We clarify that the input audio is represented as a Mel-spectrogram of shape **64 × 816**, where 64 is the number of time frames and 816 is the number of frequency bins.\n\n* **L134**: We will correct “pathchified” to “patchified.”\n\n* **Section 4.3.1**: The 10/30/50% subsets were uniformly sampled from the training set. For manual inspection of 100 samples, “weak or irrelevant alignment” refers to cases where the visual content was insufficient (e.g., static scenes) or misleading relative to the corresponding audio.\n\n* **Section 4.3.5**: We selected four VGGSound classes—*playing djembe, playing cornet, scuba diving, black-capped chickadee calling*—for their strong semantic separability, which improves interpretability of latent clustering. Classes like *laughing* and *screaming* were avoided due to overlap even in ground truth samples. As stated in L322, Figure 4 shows that MGAudio's generated samples closely match the real distribution (semantic accuracy) while exhibiting broader coverage (diversity), which are both desirable traits in generative modeling.\n\n---\n\n> 5. **Questions 1:** Could authors provide a more detailed hypothesis for why the audio domain uniquely benefits from AMG training + CFG inference? What specific characteristics of audio, such as its temporal sensitivity, its often ambiguous mapping to visual cues or something else, will require this combined approach?\n\nWe believe **AMG mitigates the training–inference mismatch** by incorporating CFG-like guidance during training, which enhances compatibility with CFG during inference. This enables the model to benefit from CFG even more effectively than in standard model training. This behavior mirrors findings in the original Model Guidance (MG) paper [16], where MG during training improved both standalone performance and CFG-based inference.\n\nFurthermore, we hypothesize that **audio generation uniquely benefits from AMG + CFG** due to the **high ambiguity of audio given visual input**. Unlike image generation, a single visual scene may correspond to multiple plausible sounds (e.g., walking on grass vs. gravel, or off-screen events). AMG injects a stronger, self-consistent signal during training to resolve such ambiguities. At inference, CFG amplifies correct generation trajectories in low-visual-saliency scenarios, making the two strategies complementary.\n\nWe will elaborate further on this point in Section 4.3.6 and the Conclusion.\n\n---\n\n> 6. **Questions 2:** How does MGAudio's Audio Model-Guidance (AMG) objective specifically contribute to its robustness against noisy/irrelevant data within VGGSound, beyond merely learning efficiently from less data? Does its \"self-distilled target\" implicitly down-weight or ignore poor samples? Could the authors propose specific strategies for future work that could explicitly leverage AMG's properties to actively identify, filter, or mitigate the impact of low-quality audio-visual pairs during training, further enhancing performance and data efficiency?\n\nThank you for this insightful question. We believe AMG’s self-distilled targets serve as a **form of regularization**, enabling the model to learn stable representations even in the presence of noisy or weak visual cues. During training, the target flow direction is adjusted using a CFG-like term based on the model’s own prediction, effectively **rectifying implausible training signals** rather than ignoring them.\n\nThis mechanism allows AMG to outperform baselines in low-resource and noisy conditions, as demonstrated below:\n\n**Table 2:** Comparison of fully trained MGAudio (with and without AMG) on the same randomly selected 10% subset of VGGSound.\n| AMG Loss  | FAD ↓ | FD ↓   | IS ↑  | KL ↓ |\n|-----------|-------|--------|-------|------|\n| ✓         | 0.82  | 8.13   | 10.68 | 2.67 |\n| ✗         | 32.84 | 123.14 | 1.09  | 4.34 |\n\nIn the table above we compare the MGAudio model trained with and without AMG loss on 10% of the VGGSound dataset and keep other setting same with the main experiment. As we can see from the table AMG get significant better result compare with the baseline on all the metrics.\n\nA possible strategy to actively use AMG's properties for data quality control would be to **track the discrepancy between the original and self-guided targets** during training. This discrepancy could serve as a proxy for sample quality: samples with large divergence might indicate noisy or misaligned pairs, which could then be **filtered, down-weighted, or reweighted** dynamically during training."}, {"review_id": "WRwS4zOZMm", "reviewer": "Reviewer_3BgF", "summary": "This paper introduces MGAudio, which incorporates scalable interpolant transformers (SiT), model guidance, and representation alignment into the video-to-audio task. When trained on VGGSound, MGAudio improves over prior methods on both seen and unseen data.", "questions": "The choice of a Mel spectrogram representation for audio is probably suboptimal relative to the embedding space of recent audio encoders. Mel spectrograms are relatively uncompressed, contain many low-energy bins, and discard phase information. Have you ablated this choice of input representation with recent audio encoders?\n\nWhat I am most concerned about is the data efficiency experiment. Less training data could perform better due to the fewer training steps; in that case, the model may have seen each example an optimal number of times. The data efficiency experiment is currently non-conclusive without also showing results at the full number of training steps—or, better yet, find the optimal number of steps for each dataset size.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "This paper corroborates the efficacy of three recent methods (SiT, model guidance, and representation alignment) in the video-to-audio domain. The methods the authors chose are of significant recent interest to the community, demonstrating broad potential for improvement. Here, that improvement is concretely demonstrated for video-to-audio and any additional challenges for adaptation are discussed. However, the three methods are not original in themselves; this paper serves more as a useful validation that the methods are worth using. To that end, it is a clear, well-organized, and polished paper.\n\nThe evaluation results on VGGSound (Table 1) are good, but incremental. The generalization results on UnAV-100 are a more convincing argument of the efficacy of the proposed MGAudio (Table 2). The extensive evaluations afterwards that discuss potential reasons for the observed improvement are well done.", "was_revised": true, "final_justification": "Authors address my primary concern of the data efficacy potentially not being robust with a small experiment that further demonstrates robustness and another that isolates the key component responsible. Further work by the authors in ablations and subjective evaluations improves an already well-written paper. The methods employed are not novel, but they are very recent methods being successfully applied for the first time to new domains.", "rebuttal": "We thank the reviewer for recognizing our integration of recent techniques into the MGAudio framework and for highlighting our model’s generalization to unseen data. Below, we address your questions.\n\n---\n\n> 1. **Questions 1:** The choice of a Mel spectrogram representation for audio is probably suboptimal relative to the embedding space of recent audio encoders. Mel spectrograms are relatively uncompressed, contain many low-energy bins, and discard phase information. Have you ablated this choice of input representation with recent audio encoders?\n\nWe appreciate this insightful suggestion. We chose a VAE‑based Mel‑spectrogram latent to maintain comparability with prior work, but agree that more modern audio encoders (e.g., waveform‑level encoder) might offer superior compression and phase preservation. Importantly, our dual‑role alignment and Model Guidance (MG) training objective are encoder‑agnostic. In future work, we will evaluate MGAudio with state‑of‑the‑art audio encoders and expect further gains by directly modeling waveform embeddings.\n\n---\n\n> 2. **Questions 2:** What I am most concerned about is the data efficiency experiment. Less training data could perform better due to the fewer training steps; in that case, the model may have seen each example an optimal number of times. The data efficiency experiment is currently non-conclusive without also showing results at the full number of training steps—or, better yet, find the optimal number of steps for each dataset size.\n\nThank you for pointing out this potential confound. To isolate the effect of the AMG loss, we conducted a controlled experiment both with and without AMG loss for 1 million steps on randomly selected 10% subset of VGGSound. We also train MMAudio on the same subset of VGGSound. The results are shown as follows:\n\n**Table 1:** Comparison of fully trained MGAudio (with and without AMG) and MMAudio on the same randomly selected 10% subset of VGGSound.\n| Method  | AMG Loss | FAD ↓ | FD ↓   | IS ↑  | KL ↓ |\n|---------|----------|-------|--------|-------|------|\n| MGAudio (10%) | ✓        | 0.82  | 8.13   | 10.68 | 2.67 |\n| MGAudio (10%) | ✗        | 32.84 | 123.14 | 1.09  | 4.34 |\n| MMAudio (10%) | -        | 2.81  | 14.28  | 10.37 | 2.87 |\n\nThese results confirm that (1) removing AMG dramatically degrades all metrics, even when both models train for the same number of steps and same dataset; and (2) our MGAudio with AMG outperforms the prior MMAudio baseline in limited data scenarios. This supports our claim that AMG has strong learning capability with scarce data.\n\nMechanistically, we hypothesize that AMG’s self‑distilled flow targets act as a dynamic regularizer: during training, the model infuses its own predicted flow (conditioned on the visual cue) into the target, effectively “rectifying” noisy or weak samples. This process stabilizes learning and yields stronger generalization under data scarcity."}]

Author rebuttal:

Meta-review: (a) The paper proposed a setup for real-world / open-domain video to audio generation; base on flow setups that uses video conditioning and also self 'correction' mechanism that relies on av encoder driving feature matching and conditioning. The resulting representations seem to generate better audio; The paper is written well; albeit some missing details that were later clarified. 

(b) Although not entirely novel in its technical content, the paper passes the threshold on proposing a audio generation setup that is more intuitive for real world scenarios with video conditioning. The resulting model does deliver perceptually better content; And the approach will be potential impact on large-scale real-time audio and v2a generation setups. The work also builds on recent innovation in model guidance; in a way justifying that approach broadly for generative models. 

(c) Hyperparameter optimization and architecture exploration is limited; and there is no explicit technical novelty in the paper; instead its really built on many recent innovations.  

(d, e) The reviewers raised 3 main concerns most of which are addressed -- seemingly incremental results across the datasets/sota, no human validation results; and limited overall technical novelty. Per my read, I do not agree that technical novelty is below par for publication -- the paper's strengths are not in designing new algorithms, instead its on integrating recent innovations on an existing research problem, and showing progress; with potential impact on AV generation research. The authors also put quite a strong rebuttal re: results and their gains in performance.
Decision: Accept (poster)