MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation
Authors: Akio Hayakawa
Masato Ishii
Takashi Shibuya
Yuki Mitsufuji
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion models
Multi-modal data
Audio-visual generative models
Primary area: generative models
TL;DR: Building audio-video joint generative model with minimal computational cost via multimodal discriminator on the top of pre-trained single-modal generative models

Abstract: This study aims to construct an audio-video generative model with minimal computational cost by leveraging pre-trained single-modal generative models for audio and video.
To achieve this, we propose a novel method that guides single-modal models to cooperatively generate well-aligned samples across modalities. 
Specifically, given two pre-trained base diffusion models, we train a lightweight joint guidance module to adjust scores separately estimated by the base models to match the score of joint distribution over audio and video. 
We show that this guidance can be computed using the gradient of the optimal discriminator, which distinguishes real audio-video pairs from fake ones independently generated by the base models. 
Based on this analysis, we construct a joint guidance module by training this discriminator.
Additionally, we adopt a loss function to stabilize the discriminator's gradient and make it work as a noise estimator, as in standard diffusion models. 
Empirical evaluations on several benchmark datasets demonstrate that our method improves both single-modal fidelity and multimodal alignment with relatively few parameters.
The code is available at: [https://github.com/SonyResearch/MMDisCo](https://github.com/SonyResearch/MMDisCo).

Published as a conference paper at ICLR 2025
MMDISCo: MULTI-MODAL DISCRIMINATOO-GGIDDE COOPERATIVE DIFFUSION FOR JOINT AUDIO AND VIDEO GENERATION
Akio Hayakawa'
Masato Ishii!
Takashi Shibuya!
Yuki Mitsufuji
Sony Al Sony Group Corporation
(akio. hayekksa,, masato. a.tss.....shii takashi. tak. shibuys, yuhki mwt.uuuujaiiii... www.:/uuwwaaoo.coooooooo
ABSTRACT
This study aims to construct an audio-video generative model with minimal com- putational cost by leveraging pre-trained trained single-moda generative models for au- dio and video. To achieve this, we propose a novel method that guides single- modal models cooperatively generate well-aligned samples across modalities. Specifically, given two pre-traines base diffusion models, we train lightweight joint guidance module adjust scores separately estimated by the base mod- els to match the score of joint distribution over audio and video. We show that this guidance can be computed using the gradient of the optimal discriminator, which distinguishes real audio-video pairs from fake ones independently gener- ated by the base models. Based on this analysis, we construct a joint guidance module by training this discriminator. Additionally, we adopt loss function to stabilize the discriminator's gradient and make it work as a noise estimator, in standard diffusion models. Empirical evaluations several benchmark datasets demonstrate that our method improves both single-modal fidelity and multimodal alignment with relatively few parameters. The code is available com
https: wttptt...o/.o/teeeerrr..ooo htt.eeeeeeaarcc////.....
INTRODUCTION
Deep generative modeling has progressed rapidly in the last few years. Diffusion models are one of the keys to this progress, and they can be applied to various tasks, including image, audio, and video generation Yang et al.] (023)) Following the success of single-modal data, several attempts have been made to apply diffusion models to multimodal data (Bao al.) (023). However, since multimodal data is more complex and harder to collect than single-modal data, developing multimodal generative models by simply extending single-modal ones remains challenging. One promising way to alleviate this problem is to integrate several pre- trained single-modal models to build multimodal generative model (Tang et al. 2023) Xing et al. 2024). As there are numerous publicly-avee available models that can generate high-quality single-modal data (Rombach al., 2022 Liu et al.) 2023 Guo et al. (024 their effective integration would substantially reduce the computational cost to build multimodal generative models. In this work, we focus on audio-video joint generation on top of two pre-trained diffusion models for audio and video.
There are approaches for audio-video generation that integrate several single-modal models: training-free and training-based. The training-free approach employs pre-trained single-modal base generative models with their parameters fixed. uses an off-the- sif-the-shelf recognition model to guide them to generate well-aligned samples across modalities (Xing et al.) 2024). While this can improve multimodal alignment without any training cost, it may degrade (Xing et al. 2024). In contrast, the training-based approach extends single-modal generative models for multimodal data by designing a neural network tailored to it (Ruan et al. 2023 Tang et al. 2023). Although this approach can achieve better performance in terms of both multimodal alignment and the fidelity of each single modality. it tends to require a significant computational cost for training. More importantly, their architectures for handling multimodal data heavily depend on those of base models (i.c.. they are not agnel--gn Therefore. when updating base models, we must manually redesign them, which requires lot of trial and error. In short, the existing two approaches involve
fidelity of single modality

---

Published as a conference paper at ICLR 2025
trade-off between the quality of generated samples and model dependency, which increases the computational cost.
In this paper, we propose novel method that is training- based but model-agnostic. Our method does not require backpropagation through the base models for the optimization. Specifically, we introduce lightweight joint guidance module top of audio and video base models that adjust their outputs for audio-video joint generation. We assume that pre-trained base models are black box diffusion models (i.e., we can access only their outputs and do not depend on a specific architecture design like a cross-attention module to construct a joint generation model). We formulate the joint generation process as an extension of the classifier guidance (C- (C-guide) for single-modal data (Song et al. 202T Dhariwal and Nichol, (021). We show that this joint guidance can be computed through the gradient of the optimal discriminator that distinguishes real audio-video pairs from the fake ones independently generated by base models. We only train the discriminator with proper regularization inspired by Denoising Likelihood Score Matching (DLSM) (Chao et al., [202)). Extensive experiments on several benchmark datasets demonstrate that our proposed method can efficiently integrate single-modal base models for audio and video into a joint generation model, maintaining the performance of each single modal generation without incurring significant computational cost (see Appendix A.9))
2
RELATED WORK
2.1
AUDIO-VIDEO JOINT GENERATION BY DIFFUSION MODELS
Since an audio-video pair is one of the most popular types of multimodal data, several works train diffusion models with such pairs to achieve a conditional single- modal generation: video-conditional
audio generation (Luo et al. 2023 Mo al. 2023 Su et al.) [0233 Pascual al. 2024 or audio-
conditional video generation Jeong et al. 2023] Lee et al. 2023 Yariv et al. (202)) However, these works mainly focus on a single modality as generation target. Extending these works the joint
generation of audio and video is not trivial due to the high dimensionality and heterogeneous data
structure of audio-video joint data.
Joint generation of audio and video pairs has been addressed in only few recent studies (Ruan al. 2023] Tang et al., [023)] Xing et al. 2024). MM-Diffusion (Ruan al., 2023) a multimodal diffusion model specific for audio-video joint generation. While MM-Diffusion is trained from scratch on audio-video pairs, CoDi (Tang et al. 2023) integrates several pre-trained single-modal diffusion models by adopting environment encoders to share modality- modalities during the generation process. Since they adopt a novel architecture strongly tied to the main network of diffusion models, it is difficult to directly apply their method to other architectures, hindering its applicability. In contrast, our method handles base models as black boxes and depends only their outputs. Therefore, our method is widely applicable to any type of architecture used in base models.
information across
Xing et al. (2024) shares similar motivation ours in the sense that they achieve audio-video cooperative generation from pre-trained single-modal base models. Given multimodal embedding models (e.g., ImageBind (Girdhar et al.) 2023)), they utilize universal guidance (Bansal et al. 2023) to make the embeddings from two modalities close. Although their approach is model- and can be applied any base model, their guidance roughly ensures semantic alignment in the space of embeddings learned by ImageBind. Thus, it does not achieve sampling from the actual joint distribution of audio-video pairs. contrast, our method is theoretically grounded in adjusting the scores predicted from base models to the score of the joint distribution, explicitly achieving sampling from the joint distribution.
2.2 GUIDANCE FOR PRE-TRAINED DIFFUSION MODELS
The guidance (Song et al. 2021 Dhariwal and Nichol 2021) provides a proper way to update intermediate representations at each generation step so that the generated samples satisfy given condition, even when the model is not trained for that type of conditional generation. Since the guidance does not require additional training in diffusion models, it is widely used to control the generation process with additional conditional signals. C-guide (Song et al. 202 I Dhariwal and Nichol] (02T) was proposed to guide an image generation model by the class label, such as "cat" or "dog". using an additionally trained classifier. Several works extend this to utilize off-the-shelf
2

---

Published as a conference paper at ICLR 2025
Unpaired samples from gen. data
Joint Discriminator:
Video
Trained to discriminate
Encoder
paired samples by Laisc
0/1
Audio Encoder
Compute gradient: De.
Flog Do
Pre- rre-trained and frozen
Trainable
Paired samples from real dataset
Independently predicted noises
Predicted residual noises
Video Diffusion EΦp
Regularized match GT noises by Leenoise
Audio Diffusion Exp
Figure Overview the training process of our proposed method. train a joint discriminator on the top of two base diffusion models to distinguish real vidco-audio pairs from fake ones generated by base models. Additionally, adopt denoising objective, in standard diffusion models, to match the gradient of the discriminator with regard to the inputs the residual noise between ground truth noises and predicted noises from base models.
recognition models achieve beyond class-conditional generation (Grikos et al. 2022 Bansal et al. 2023). Our proposed method is the first attempt to extend the C-guide for joint multimodal generation. We derive our methodology from the theory of C-guide, enabling us to sample data from the joint distribution.
The samples generated with C-guide may suffer from degraded quality without careful tuning of its scale (Dhariwal and Nichol (02T). Chao et al. (2022) denote this problem as the score mismatch issue, where the posterior scores estimated by a diffusion model and a classifier are unstable and deviate from the true ones. They proposed DLSM loss to alleviate this problem. DLSM regularizes a classifier's s gradient to match the residual error of noise prediction by trained diffusion models, providing stable gradient for the generation process. Our work can be seen DLSM for multimodal generation. We employ DLSM to stabilize the gradient of our discriminator,
extension of
Kim al. (2023) proposed Discriminator Guidance, which guides Text-to- Image (T2I) model by a discriminator distinguishing real images from generated images to improve the quality of the generated samples by pre-trained T21 model. Although Kim et al.((002) share similar concept to ours regarding using discriminator to bridge the gap between the score predicted by pre-trained model and the target score, our goal to integrate single-modal models into joint generation model. Their method is particularly designed to handle the gap within single modality, and it cannot be straightforwardly applied to the multimodal generation. In contrast, our method is derived from directly bridging the gap between single-modaa distribution and a joint one. Moreover, we show single discriminator can serve as a guidance module for both domains, resulting minimal computational cost for developing joint generation model.
METHODOLOGY
In this section, we first briefly review Diffusion models on a single modal data. Here, we follow Denoising Diffusion Probabilistic Models (DDPM) (Ho al. (2200), broadly used standard definition of diffusion models. Then, we describe our formulation joint score estimation on top of two pre-trained diffusion models.

---

Published as a conference paper at ICLR 2025
3.1 PRELIMINARY: DIFFUSION MODELS
Basics Diffusion models are a family of probabilistic generative models that reverse a diffusion process from data pure noise. Specifically, let zo -p(20) be sample of data distribution and Iy be a noisy representation at diffusion timestep & E {0, ...., T). A forward diffusion process is defined as a Markov process:
1, (
(1)
where Be (0, 1) controls how fast the data is diffused each timestep. On the basis of Eq.
!
Ie zefza also follows the Gaussian distribution of p(zz|za) = V(Vāzzo (1 dg f)), where 0y (1 3,) is the accumulation of diffusion coefficients, and the noisy sample at the last step would follow the standard Gaussian distribution /V I) with an appropriate setting of I and 3r. If
3t is small enough (or 7' is large enough), the reverse process of Eq. (1) can be approximated to be Gaussian. Diffusion models are trained estimate its mean by predicting the noise in 2t as follows:
Po(xe-1l07 = t)),
(2)
Mo(se, 1) V1- - Br
1
co t)),, 0; -1
Be
āt-1 Be,
1 āt
where 6 represents the noise prediction model with parameters 6. This model es be trained by minimizing the following mean squared error between the noise predicted by the model and that added to the data:
Ø" argmin Exo.e,t latee.e...eeeeeeeeeeeeeeee /Vco(Var
vI aic,t) cl(
where e ~ (O,, I) is a noise, and ~ U(1, is a timestep. For generation, we can sample xo through the iterative sampling from 7 to using Eq. (2) with egr.
Equation is form of denoising score matching. Specifically, from Tweedie's formula (Efron 2011). the noise added the data and the score function is equivalent up to a constant factor. On the basis of this fact, we can approximate the score function by using noise prediction network trained by Eq. (4) as:
1
VA log (oz)(()))
ca- (2), f).
(5)
v1 āt
From Eq. (5). noise prediction model trained by Eq. can be considered model estimating score of g(xx).
Guidance for conditional generation We can extend the generation process diffusion models to the conditional one using classifier guidance (C-guide) (Song et al. Dhariwall Dhariwal and Nichol
202I). The C-guide can be derived from the perspective of score estimation. By Bayes' theorem, we can write the conditional score
Vz, (ogc()))c)= Vz, log (12) + V logg(c(21).
(6)
where is a conditional vector. The first term on the right-hand side can be estimated by trained with Eq. 4 The second term on the right-hand side can be estimated by computing a classifier's gradient with respect to its input, where the classifier is trained to predict given Zt--
3.2 JOINT SCORE ESTIMATION ON THE TOP OF PRE-TRAINED DIFFUSION MODELS
We aim to achieve joint generation based on two independently trained diffusion models. Namely, let E R= and y RD, be samples of two different modalities. We want sample x and
4

---

Published as a conference paper at ICLR 2025
from joint distribution ((2, y). From the perspective of the score function, we need to estimate Va, log g(t y6) and Vy. log (( yc). By Bayes rule, we can derive the following equations:
x, log ( Me) 2z, loz)(z)) + Vz, log (V)l V. log ((T,, Mt) = V. log (Vb) ) + Vv log (
(7)
(8)
Similar to the classifier guidance, the first terms in the right-hand side of Eqs. 7 and (8) can be estimated by the two diffusion models independently trained on the modalities x and y. On the other hand, modeling the second terms on the right-hand side are not trivial. One can naively construct two additional generative models of dz yt and y/ 2r and compute the gradient of such models with regard to condition vectors. However, training them is difficult due to high dimensionality and requires significant computational cost. Instead of training additional high-cost generative models, we train a single lightweight discriminator that distinguishes real pairs of (I', Vr) from fake ones generated by pre-trained single-modal base models. We theoretically show that sufficient to compute the gradient of this discriminator to approximate these terms.
Specifically, we propose train a discriminator between the joint distribution g( y/) and indepen- dent distribution g(zzy) (Bv)). Let ~ Po(x) and y' pv()) be fake paired samples independently generated by pre-trained diffusion models. Here, we denote these diffusion models independently (a)
(v) where e and e are the parameters of these pre-trained by each modality and y as 6 and C
models. We train a discriminator Dg Fz. Ve. [0, I] with the following loss function:
0r argmin (r.)) (() pl (p)( [ (z, 2.21
(9)
(00) (xt ,, y²) = E,or ow)) ro0,, [log (a 1/. 1) + log (1 Do(as, Vr. ())],
(10)
where (z) and are a noise and a noisy sample for = E {x,, v', respectively. The noisy sample 2, is derived by forward process 2, = vāž= + /1 - aa (()) where oe is a coefficient of the forward diffusion process for each modality, and this omitted from Eq. (IO) for brevity. Note that this discriminator De is trained to output one for real pairs and zero for fake pairs.
Similar to discriminator in Generative Adversarial Networks (GANs) (Goodfellow al. 2014), an optimal discriminator Dg- (Ie, Yt. that minimizes Eq. (10) can be seen as an estimator of the density g(x..96) ratio g(#..30/ Po ( [M]] Therefore, the second term in the right-hand side of Eqs. (7) and (8) can be approximated by utilizing Da- as follows (see Appendix A.T for details of this derivation):
Dø- (Ke, Mes 6) Dg- (Ie: ye. t)
V., log (og V Vx, log
(11)
Dø- (Ir, Y1. t)
Vg. (ogg(())))))) log
(12)
Dg- (It, ye,
In summary, estimate joint score the sum of the scores independently estimated by
base models ( and e and the gradient of an optimal discriminator D" shown in Eqs. 11 and (22 In inference, we independently compute the outputs from base models and the gradient of our discriminator for the intermediate noisy samples 2; and yt at each timestep. Then, their sum is used predicted noise for the denoising step. Note that, in the discussion above, assume that the distribution of samples generated by base models is equal to the marginal distribution of joint data for brevity (i.e., ((xx)= (x)))) and q(v)) = Po(y.) )). However, our proposed method can be applied even when these are not equal (see Appendix A.2 in the appendix for more details). We also tested this setting in our experiment.
In this work, we adopt the most hasic form of density ratio estimator. We expect that using advanced ones may improve the performance of joint guidance, which leave for future work.
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 Training process of Dg.
Algorithm 2 Joint inference by Dgr.
Require:
(ar)
(2) and paired dataset
(v) Require: <g e and D8- Initialize IT yT with Gaussian noise.
repeat
Sample I, y from paired dataset.
for in [2', .... 1] do
Generate x' using
Dratio log I- (Pe.
Dg- (L).Be.))
Generate yf using v
(v)
{(x) (z)
E/ (I(: vI aj V V.Draare Dratio-
Compute Latiee (I,, x.; y') by Eg. 10
{(V) (9)
+ (y:,t) V1 V
Compute Ldenoistl (2r, y) by Eq. 13
Sample (Iz- 1,9(-11) based on (E(*), g(v)).
Update A based on aalll-- until converged Return Dø-.
end for Return (0).
3.3 RESIDUAL SCORE ESTIMATION FROM AN OPTIMAL DISCRIMINATOR
can sample and y from joint distribution using guidance with discriminator that can distinguish real paired data from paired samples generated by pre-trained single- modal diffusion models. However, in preliminary experiments, using a discriminator trained by only Eq. (9) degrades the fidelity of generated samples as a single modality. We conjectured that this caused by the score estimation mismatch issue mentioned by Chao al. (2022). They argued that the score may deviate from the true one when estimating it using gradient of a classification model. To alleviate this issue, inspired by Chao al. (2022). we adopt regularization for the gradient of the discriminator to match the residual of true noises and noises predicted by the base diffusion models. Specifically, we define a denoising regularization loss follows:
As described in Section 3.2
c10.0.11) (I, v) (e(t)),
C (0,0) denoise
2. U, ((),,, (lcc,,,cuu,,) L (0,0))
(2, ((()))
denoise
(13)
((0,0) denoise
(1, (,),,e(),e()), =
c (t)
$ (Iz,, 1) + /I ág V xe log
Doles, Uc,
(14)
Dø
denoise (() (()) ((0) =
(v) (1) (yr, 1) + v V1-a
a V M log
Do yt, t)
2
(15)
Da(xi, Dt,
Note that the base models' parameters o, s are fixed during training and just used compute noise estimation errors. Our final objective (denoted Call hereinafter) to train the discriminator Do is the sum of the discriminator loss (Eq. (10)) and denoising regularization loss (Eq. (33
f" = argmin E (( w/ (i (x. a. a.."", ,1') + X (0.0.0) (z. v)).
(16)
where À is weight to balance these two losses. We use À = 1 throughout this paper. The training and generation process of our proposed method summarized in Algorithms and respectively.
EXPERIMENTS
In this section, we first show the results of preliminary experiments with toy datasets confirm our proposed method can guide the generation process of two independently trained diffusion models to the desired joint distribution. Then, we show the experimental results with real data.
4.1 PRELIMINARY EXPERIMENTS WITH TOY DATASETS
Datasets
To create the training dataset for base model, evenly sampled data from five Gaussian distributions whose means were [-3.- -1.5, 0, 1.5, 3), and variances were all 0.01, resulting in 500 samples. For training the guidance module, constructed two types of

---

Published as a conference paper at ICLR 2025
Table 1: Negative log likelihood (NLL) of
Independent
Guided (IND)
Guided (OOD)
generated samples target distribution.
Dataset
Method
NLL |
GT
GT samples
1.67
No joint
18.80
IND
L disc
3.29
Lenoisee
2.40
Gen.
Can
1.94
GT samples
1.67
No Joint
22.10
OOD
Edisc 16.60
Figure Visualization guidance results on toy datasets. The top row shows samples drawn from ground truth distribution (GT), and the bottom row shows generated samples (Gen.).
Lenoose
2.63
Call
2.53
datasets. The first one simulates an in-domain (IND) situation, where the marginal distribu- tions g(x)) and g(y) This dataset was constructed by sampling from five Gaussian distributions whose means were [(-3, 1.5), (-1.5, 3), (0, 3), (1.5, 0), (3, 1.5)]. respectively. The second one simulates an out-of- domain (OOD) situation, where the marginal distributions differ from the distribution on which the base model is trained. This dataset was constructed by sampling from five Gaussian distributions whose means were [(-2.25. 2.25), (2.25, 2.25), (-2.25, 2.25), (2.25, 2.25). (0, 0)], respectively. We set the same variance for all Gaussians 0.01. the former case, the peaks of the distribution are a subset of the independent data distribution, and its marginal distributions for and y are the same as the data used train the base model. On the other hand, the peaks are shifted in the latter case, and the marginal distribution differs from the original. To train our discriminator by Caiso (Eq. (9 we sampled 500 fake pairs in advance from the base model.
equal the distribution which the base model had been trained.
Setup We first trained base diffusion model on scalar value samples from mixture of five Gaussian distributions. The base diffusion model outputs single scalar E RR and we duplicated the trained base model to generate a two-dimensional vector (z, y) E I2 by concatenating outputs from them. Since each and has five peaks as distribution, (xr y) has 25 peaks when and are generated independently, as shown in the leftmost column in Fig. 2 Then, we guided the base models using proposed method to generate samples that follow the target joint distribution. We used a neural network consisting of five fully connected layers for the base diffusion and one with three layers for the guidance module. See Appendix A.3 for more details about the settings.
Results Figure 2 shows the generated samples from our proposed method. Our proposed model successfully guides the base models to generate samples from the target distributions in- and out-of-domain settings. For ablation study, we evaluated the effectiveness of each loss by negative log-likelihood (NLL) against the target distribution (Table []. We confirmed that both losses, Ldise and Cdenoise. contribute substantially improving NLL. These results demonstrate that our proposed method effectively guides base models in generating samples of the target joint distribution.
4.2 EXPERIMENTS WITH BENCHMARK DATASETS UNDER THE IN-DOMAIN SETTING
Setup To investigate the applicability of our proposed method to real data, we first applied it to an in-domain setting. this setting, we used the same dataset to train the base models and our proposed model. We trained our discriminator on top of MM-Diffusion (Ruan et al.) 2023), which was already trained generate audio-video data jointly, and guided to generate further aligned samples. This setting is similar to adopting guidance diffusion model can directly handle. The discriminator consists of a stack of two-stream ResBlock layers for each audio and video, followed by linear layer to output scalar. The total number of its parameters is 12.7M, whereas MM- Mi--iiffusion has 133M parameters. We used the pre-trained MM-Diffusion released in the official repository and only trained our discriminator while freezing
conditional diffusion model with condition that the
7

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative evaluation Landscape and AIST+ datasets for the in-domain adaptation setting. We use CMM-Diffusion trained on Landscape or AIST+ dataset at a 64 x 64 resolution and additionally guide its outputs by our proposed method.
Dataset
Method
FVD 4
FAD 4
IB-AV
MM--Difusion
447
5.78
0.156
Landscape
MMDisCo
405
5.52
0.162
MM-Diffusion
513
2.31
0.0897
+
MMDisCo
450
2.17
0.0909
the parameters of MM-Diffusion. For training, we used the Adam optimizer (Kingma and Ba 2015) with a learning rate of le-3, and the batch size and the number of epochs were set to 16 and 100, respectively. For generation, we used DPM-Solver (Lu et al. 2022) and set its number of function evaluations (NFE) to 20, the standard setting in MM-Diffusion. See Appendix A.3 for more details about the settings.
Datasets We conducted experiments on the Landscape (Lee et al. 2022) and AIST+ (Li et al. 202T) datasets. The Landscape dataset contains 928 videos of nine classes of natural scenes, such as fire crackling and waterfall burbling. The AIST+ dataset contains 1020 video clips of street dance with 60 copyright-cleare dancing songs. Note that, in this experiment, we do not use these class labels as input. We trained our discriminator generate 1.6 second audio-video pairs. Each video comprises ten frames per second at x 64 spatial resolution, and the sampling rate of the audio 16kHz. We followed the
setting for audio and video preprocessing and training split.
Evaluation metrics We evaluated the generated samples in terms of the cross moosll semantic alignment as well as the fidelity of each modality. To measure the cross-modal semantic alignment, we used the ImageBind score (Girdhar et al.] (023) computed for audio and video embeddings (IB-AV). To evaluate the fidelity of each modality, we used FVD (Unterthiner et al. 2019) for video and FAD (Kilgour et al. 2019) for audio.
Results Table 2 shows the quantitative evaluation on both the I Landscape and +IST++ datasets. For the fidelity of each single modality, our proposed method improves both FVD and FAD. These results demonstrate that proposed guidance module captures the training data distribution well and bridges the gap between the distribution of the training data and that of the generated samples. For the alignment score, IB-AV scores are also marginally improved, illustrating our guidance module properly guides generated samples to be well-aligned across modalities.
4.3 EXPERIMENTS WITH BENCIMARK DATASETS UNDER THE OUT-OF-DOMAIN SETTING
Setup We also applied our proposed method to an out-of-domain setting. In this setting, we used different dataset for training our discriminator compared with the base models. Specifically, used two pairs of base models, AudioLDM (Liu et al. 2023) / AnimateDiff (Guo et al., 2024) and Auffusion (Xue et al. 2024) VideoCrafter2 (Chen et al.) 2024). to generate audio and video, each pre-trained with respective single-modal large-scale datasets. On top of these independent models, we trained our proposed discriminator using an audio-video dataset to facilitate audio-video alignment of the generated samples from the base models. Note that these base models can accept text input as an additional condition. To enable classifier-fres free guidance (Ho and Salimans) (02T), we also fed conditional text into our discriminator and trained it with 10% text dropout rate. We doubled the channel size of each layer of the discriminator to match the larger size of the base models compared to the IND case. We trained our model to generate videos with spatial size of 256 x 256 to match the outputs of the base models. For generation, we adopted classifier-fres guidance with its strength set to 2.5 for the AudioL.M,, 7.5 for the AnimateDiff, and 8 for the Auffusion and the VideoCrafter2, respectively. See Appendix A.3 3 for more details.
Dataset We conducted experiments with the Landscape (Lee et al. 2022) and VGGSound (Chen al. 2020) datasets. The VGGSound dataset contains nearly 200K video clips of 300 sound classes.
8

---

Published as a conference paper at ICLR 2025
Table Quantitative evaluation under the OOD setting the Landscape dataset.
Video
Audio
Cross-modal
Method
FVD &
IB-TV
FAD 4
IB-TA
IB-AV
Grand truth
207
0.16
0.247
AnimateDiff - VQGAN
(852)
(0.308)
10.69
0.050
0,086
VideoCrafter2 - SpecVQGAN
(700)
(0.309)
11.63
0.049
0.074
AnimateDiff DiffFoley
(852)
(0.308)
8.58
0.053
0.113
VideoCrafter2 - DiffFoley
(700)
(0.309)
9.14
0.045
0.111
AudioLDM / AnimateDiff
852
0.308
8.26
0.053
0.093
+ MMDisCo
667
0.303
7.69
0.061
0.102
Auffusion / VideoCrafter2
700
0.309
8.06
0.103
0.134
+ MMDisCo
687
0.309
7.86
0.109
0.137
Table 4: Quantitative evaluation under OOD setting on the VGGSound dataset.
Video
Audio
Cross-modal
Method
FVD L
IB-TV
FAD
IB-TA
IB-AV
Grand truth
256
1.32
0.336
AnimateDiff - Spec VQGAN
(739)
(0.295)
8.65
0.064
0.084
VideoCrafter2 Spec
(831)
(0.302)
8.91
0.061
0.089
AnimateDiff - DiffFoley
(739)
(0.295)
14.42
0.046
0.098
VideoCrafter2 DiffFoley
(831)
(0.302)
13.11
0.060
0.105
AudioLDM / AnimateDiff
739
0.295
15.5
0.107
0.121
+ MMDisCo
754
0.291
12.1
0.116
0.127
Auffusion / VideoCrafter2
831
0.302
5.32
0.190
0.192
+ MMDisCo
704
0.302
4.79
0.197
0.201
Following Yariy al. 2024), we filtered 60K videos with weak audio-video alignment to enhance the data quality. For both datasets, we resized videos while maintaining the aspect ratio of the spatial resolution and cropped 256 x 256 center pixels to create the training dataset. Regarding text captions for the training dataset, we used the original labels for the VGGSound dataset. For the Landscape dataset, we additionally created captions by applying InstructBLIP (Dai et al., 2024) to the first frame of each video because the original labels are too vague generate plausible samples (see Appendix A.4 for more details).
Evaluation metrics We evaluated the generated samples using FAD, FVD, and IB-AV as described in Section 4.2 Additionally, measure the correspondence between generated sample and its text condition, we computed the ImageBind score for text-video and text-audio pairs (denoted by IB-TV and IB-TA, respectively).
Baselines We compared our method with existing works that can be reproduced and are publicly available. Specifically, we used sequential approach, Text-to-Video (T2V) generation, followed by Video-to-Audio (V2A) generation, as the baseline for comparison. We used the pre-trained SpecVQGAN (lashin and Rahtu 2021) and Diff-Foley (Luo et al.] (023) released in the official repository as a V2A model and generated audios conditioned by the videos from Animate or VideoCrafter2. For the text conditions of T2V models, randomly sampled the captions generated by InstructBLIP for the Landscape dataset and the class labels of the VGGSound dataset, which is the same setting the one for method.

---

Published as a conference paper at ICLR 2025
No joint
Ours
Playing
bongo
Playing table tennis
Figure 3: Generated samples from AnimateDiff/ AudioLDM and ours trained on the VGGSound dataset. We used the captions shown the leftmost side and the same random seed for both settings generate samples.
Results Tables 3 and show the quantitative evaluation results on the Landscape and VGGSound datasets. For single-modal evaluation, our proposed guidance improves all scores except IB-TV and FVD for the VGGSound. This result indicates our method properly guides the generation process to match the distribution of generated samples with the training dataset, even for the OOD setting. Since IB-TV scores of the base models are substantially higher than IB-TA, we conjecture that the captions are suitable for video generation. Therefore, the base vidco model can generate videos well-aligned with the text captions, possibly resulting in slight degradation of FVD and IB-TV for the setting of AudioLDM and AnimateDiff but improvement of FAD and IB-TA. For cross-modal metrics, the IB-AV scores consistently improved, indicating that method successfully enhances semantic audiovisual alignment. Our method generally delivers superior performance in terms of audio fidelity and IB-AV scores than the T2V followed V2A generation (T2V V2A in Tables and 4 We hypothesize that pre-trained T2A models, trained on larger-scale datasets, have the capacity to generate not only high-quality but also sufficiently diverse samples accommodate various, potentially out-of-domain videos produced by T2V models. Combined with our method, the fidelity of the generated audio and its semantic alignment with the corresponding generated videos are further enhanced. Fig. 3 Jillustrrats the effect of our guidance module qualitatively. Although the generated samples from the base models look reasonable as a single modality, they look unnatural from the perspective of audio-video joint generation because there is no player, but the sound is generated. In contrast, our guidance tends to generate players successfully, and the generated audio temporally aligned with their motion (see Appendix A.12 for more generated samples).
CONCLUSION
In this work, have proposed a novel training- based but model-agnostic guidance module that enables base models to generate well-aligned samples across modalities cooperatively. Specifically, given two pre-trained base diffusion models, train lightweight joint guidance module that modifies the scores estimated by the two base models to match the joint data distribution. We show that this guidance can be formulated the gradient of an optimal discriminator that distinguishes real audio-video pairs from fake pairs generated independently by the base models. We also propose regularizing this gradient using a denoising objective, as in standard diffusion models, which provides a stable gradient of the discriminator. On several benchmark datasets, we empirically show that our proposed method improves alignment scores as well single-modal fidelity scores without requiring huge number of parameters compared with the base models.
Limitation Since our model is built on top of pre-trained models for each modality, the quality of the generated samples strongly depends the base models. However, method benefits from the advancements each modality's 's generative modeling. It enables us to integrate new state-of-the- the works into joint generative model without huge computational cost.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENT
We sincerely thank Christian Simon, Kazuki Shimada, Michael Yeung, Satoshi Hayakawa, and Shoukang Hu for their helpful feedback. We used computational resources of AI Bridging Cloud In- frastructure (ABCI) provided by the National Institute of Advanced Industrial Science and Technology (AIST) and AIST Solutions.
REFERENCES
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučič, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang. Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion scale. In Proceedings of the International Conference on Machine Learning, 2023.
Chen-Hao Chao, Wei-Fang Sun, Bo- Cheng, Yi-Chen Lo, Chia-Che Chang, Yu-Lun Liu, Yu-Lin Chang, Chia-Ping Chen, and Chun-Yr Yi Lee. Denoising likelihood score matching for conditional score-based data generation. In Proceedings of the International Conference on Learning Repre- sentations, 2022.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2020.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. In Proceedings of the Advances in Neural Information Processing Systems, 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Proceedings of the Advances in Neural Information Processing Systems, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16 words: Transformers for image recognition scale. In Proceedings of the International Conference on Learning Representations, 2021.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 2011.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the Advances in Neural Information Processing Systems, 2014.
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-pla priors. In Proceedings of the Advances in Neural Information Processing Systems, 2022.
11

---

Published as a conference paper at ICLR 2025
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized toxt-to-image diffusion models without specific tuning. In Proceedings of the International Conference Learning Representa- fions, 2024.
Jonathan Ho and Tim Salimans. Classifier-fre free diffusion guidance. NeurIPS 2021 Workshop Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the Advances in Neural Information Processing Systems, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA; Low-rank adaptation of large language models. In Proceedings of the International Conference Learning Representations, 2022.
Vladimir lashin and Esa Rahtu. Taming visually guided sound generation. Proceedings of the British Machine Vision Conference, 2021.
Yujin Jeong, Wonjeong Ryoo, Seunghyun Lee, Dabin Seo, Wonmin Byeon, Sangpil Kim, and Jinkyu Kim. The power of sound (tpos): Audio reactive video generation with stable diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms, In Proceedings of the Interspeech, 2019.
Dongjun Kim, Yeongmin Kim, Wanmo Kang, and Il-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models. Proceedings of the International Conference on Machine Learmin, 2023.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, 2015.
Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Chanyoung Kim, Won Jeong Ryoo, Sang Ho Yoon, Hyunjun Cho, Jihyun Bae, Jinkyu Kim, and Sangpil Kim. Sound-guided semantic video generation. Proceedings of the European conference on computer vision, 2022.
Seungwoo Lee, Chaerin Kong, Donghyeon Jeon, and Nojun Kwak. Aadiff: Audio-aligned video synthesis with text-to-image diffusion. CVPR 2023 Workshop on for Content Creation, 2023.
Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Haohe Liu, Zehua Chen, Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. In Proceedings of the International Conference on Machine Learning, 2023.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling around steps. Proceedings of the Advances in Neural Information Processing Systems, 2022.
Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. In Proceedings of the Advances in Neural Information Processing Systems, 2023.
Shentong Mo, Jing Shi, and Yapeng Tian. Diffava: Personalized text-to-audio generation with visual alignment. CVPR 2023 workshop on Sight and Sound, 2023.
Santiago Pascual, Chunghsin Yeh, Ioannis Tsiamas, and Joan Serrà. Masked generative video-to- audio transformers with enhanced synchronicity. Proceedings of the European conference on computer vision, 2024.
12

---

Published as a conference paper at ICLR 2025
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition, 2022.
Ludan Ruan, Yiyang Ma, Huan Yang. Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. besse- generative modeling through stochastic differential equations. In Proceedings of the International Conference on Learning Representations, 2021.
Kun Su, Kaizhi Qian, Eli Shlizerman, Antonio Torralba, and Chuang Gan. Physics-driven diffusion models for impact sound synthesis from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. In Proceedings of the Advances in Neural Information Processing Systems, 2023.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marein Michalski, and Sylvain Gelly. FVD: A new metric for video generation. ICLR 2019 Workshop on DeepGenStruct, 2019.
Vaswani. Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, 2017.
Yonggi Wang, Wenxiang Guo, Rongjie Huang. Jiawei Huang, Zehan Wang, Fumingg You, Ruiqi Li, and Zhou Zhao. Frieren: Efficient -idoo-to-audio generation with rectified flow matching. In Proceedings of the Advances in Neural Information Processing Systems, 2024.
Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open- domain -ssullaadioo generation with diffusion latent aligners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Jinlong Xue, Yayue Deng, Yingming Gao, and Ya Li. Auffusion: and large language models for text-to-audio generation. arXiv preprint arXiv:2401.01
Leveraging the power of diffusion
2024.
Ling Yang, Zhilong Zhang, Yang Song. Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 2023.
Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to-video generation via text-to-video model adaptation. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, 2024.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung- Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In Proceedings of the International Conference on Learning Representations, 2022.
13

---

Published as a conference paper at ICLR 2025
TABLE OF CONTENTS
Introduction
Related work
2.1
Audio-video joint generation by diffusion models
2.2
Guidance for pre- pre-trained trained diffusion models
3
Methodology
3.1
Preliminary: Diffusion Models
3.2
Joint score estimation on the top of pre-trained diffusion models
3.3
Residual score estimation from an optimal discriminator
Experiments
4.1
Preliminary experiments with toy datasets
4.2
Experiments with benchmark datasets under the in-domain setting
7
4.3
Experiments with benchmark datasets under the out-of-domain setting
Conclusion
10
6
Acknowledgment
11
Appendix
15
A.I
Proof of discriminator guidance
15
A.2
Discriminator guidance for the OOD case
15
A.3
Details of experiments
16
Captioning process Landscape dataset for the OOD setting
17
A.5
Visualization of the effect by each loss function with toy datasets
17
A.6
Ablation study about the loss functions on real data
19
A.7
Validation of the parameter size and the training epoch
19
Discriminator structure variants
20
A.9
Computational cost for the training and inference
21
10
Comparison with baselines on the semantic and temporal cross modal alignment
22
A.10.I computation of the AV-Align score
22
A.10.2 Evaluation on the Temporal and Semantic alignment
23
A.
Subjective evaluation
24
12 Generated audio and video pairs
24
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PROOF OF DISCRIMINATOR GUIDANCE
Our discriminator is trained by Eq. 10 This loss function can be decomposed into the loss function at each timestep as:
(0) disc,
By((e..e) [log Do Ve. 1)] ppp(tt)) Dog (1 Do Ut, t))]
(17)
[ g( Me) log (o Nr, 8) + (o( log (1 Da( Mrs t)) dr,dy.
(18)
Here, q(2,, 91) is a distribution of real paired data, and pe (p is that of a fake one learned by base models. By Proposition in Goodfellow et al. (2014), for any fixed generators, the optimal discriminator that minimizes Eq. 18) yields:
g(ze, va)
Do- (It, Mt, R g( Me) + p (F
(19)
Using this optimal discriminator, we compute the density ratio between a real and fake data distribution as follows:
( Vt) Pá (3re Jp (y:)
Dø- (are, Vt, t)
FJ 1 Da- (Ite Ve, ()
(20)
Assuming the base models perfectly align with the marginal distribution of the real data (i.e., ((xx) = (x(e)) and ((v+) = Pv (y))), we can compute Vz, log ((el(14)) and Vy, log ((xa(y)) as follows:
g( Mr) ((xx))
Vay log ((veax)) = Vz, log
(21)
q 3c)
V log
(22)
((xz)4(()
g Me)
R., log
(23)
(r (roo (112)
Dø- (2e, Me. 6)
log
(24)
Dø- (xt, Ut t)
g( Wt)
Vg, ye log (())))))
y. log
(25)
((ve)
( Vt)
V y, log
(26)
(3)))
g(Ie, We
V ys log
(27)
Pol (2' P (yr)
Dg- (Itt, Wt: ()
log
(28)
1 Dy. (I yc, t)
Therefore, Eqs. 11 and 12 hold.
A.2 DISCRIMINATOR GUIDANCE FOR THE OOD CASE
In Appendix A.I, we assume ( = (x;) and q(v)) = Py (ye) to derive Eqs. 11 and 12 Here, prove that they also hold even in the case of q(xx) * po( and q(ve) & (t))
15

---

Published as a conference paper at ICLR 2025
Equations 7 and (8) can be rewritten by using pa(a)) and Py(()) as follows:
V 2, log (xs, U1) Vv, log ((x)) + Var log (3))((1)
(29)
Vn, log + V us log g(xz)
+ Vx log g(Fe. Mr)
(30)
Po (2s
g(22))
g 313
V., (o))s( + V ni log
(31)
ps (1aa)
Dg- (I's Vc. t)
2 v., (x:) + V az log
(32)
1 Dø- (2 Uc. t)
Vy, log ( Mt)
= V ye (3) + V ay log g (36e)
(33)
= V (v:) + V ah log ((31))
+ Vvr log g( Mr) g(v))
(34)
(Bt)
( y/.)
V,, (v)) + V 2 log
(35)
P (ye)
Dø- (Ize yo. t)
2 Vy, (v)) + V ye log
(36)
Do- (Ir, Vt.
Therefore, use the optimal discriminator as joint guidance module for the OOD case long as a discriminator is trained to distinguish real paired samples from fake ones generated by the base
models.
A.3 DETAILS OF EXPERIMENTS
This section provides more details of the settings we used in each experiment.
Experiments with toy dataset Table the settings for the toy dataset described in Section 1.1 As described in Section residual noise prediction for the denoising step is computed through the gradient of the discriminator with respect to its input, and its dimensionality is two. For evaluation and visualization, we sample 4000 samples.
Experiments with benchmark datasets Table Ta the settings for real benchmark datasets. (Ruan
We implemented our discriminator based the official implementation of et al. 2023 A notable difference is that we only used its encoder part and removed all attention layers from the encoder, which is commonly used in the implementation of diffusion models. Our preliminary experiments found that using attention layers causes unstable discriminator training. Thus, the architecture of our discriminator is a stack of individual 2-stream ResBlocks followed by linear layer. For the OOD case, we constructed our discriminator on the latent space learned by the base models. To train the discriminator, we sampled 1K samples for the IND settings and 10K samples for the OOD settings in advance.
For evaluation, we generated 2048 samples and computed quantitative metrics. Specifically, we utilized the evaluation code provided by StyleGAN-V (Yu et al. (022)) for FVD 3 that provided by AudioLDM (Liu et al. 2023) for F and that of IB IB-score (Girdhar et al. 2023 5 respectively. During inference, add the discriminator's 's gradient to the base models' predictions without scaling. However, when Auffusion and VideoCrafter2 are used as base models, we observed that the gradient norm with respect to the noisy video latents was significantly smaller than the predicted noise from the base models. To address this discrepancy in scale, we apply scaling to the discriminator's gradient with respect to the video latents, but only in this particular setting. We determined the scaling factor by performing a grid search across [1, 2, 4, 6, 8, 10). with scale of 8 producing the best quantitative results.
16

---

Published as a conference paper at ICLR 2025
Table 5: Hyperparameters for the base model and discriminator in the experiments with toy dataset (used in
Section
Model
Base model
Discriminator (IND)
Discriminator (OOD)
Hyperparameters Architecture
MLP
MLP
ML.P
Input dim
2
2
# of layers Channel sizes
5
3
3
16, 64, 256, 64, 16
64, 32, 8
64, 32, 8
Output dim Normalization
I
1
1
LayerNorm
LayerNorm SiLU
LayerNorm SiLU
Activation
SiLU
Timestep dim
256
64
64
Timestep input type
Adaptive
Adaptive
Adaptive
Optimizer
Adam
Adam
Adam 0.001
Learning rate
0.001
0.001
Total batch size
512
512
512
Total of params
2.2M
171K
171K
Diffusion setup
Diffusion steps
500
500 linear 0.0001
500
Noise schedule
linear
linear 0.0001
Bo ST
0.0001
0.02
0.02
0.02
Dataset
Pe of GMM
[-3, -15, 0, 1.5,]
[-3, 0, 1.5, 3] [1.5, 33,3, 01.5 0.1
[2.25, -2.25, -225,, 0] [2.25, -2.25, 2.25,-2.25,
By of GMM
a GMM
0.1
0.1
# of fake samples
500
500
# of real samples
500
500
500
We utilized 16GB Nvidia V100 x 4 GPUs for the IND case and 40GB Nvidia A100 x 8 GPUs for the OOD) case. The training time for the IND case was about 1.5 hours with both Landscape and AIST+ datasets, and that for the OOD case was about two hours with the Landscape dataset and 30 hours with the VGGSound dataset. All evaluation was performed by 16GB Nvidia V100 x 4 GPUs. It takes about one hour for the IND case and about five hours for the OOD case.
A.4 CAPTIONING PROCESS OF LANDSCAPE DATASET FOR THE OOD SETTING
We found that the original labels added to the Landscape dataset too ambiguous for the base models to generate reasonable samples. For instance, the generated results with the caption "wind noise" tend to look like just noise video and audio, and evaluation for these samples is unstable and not desired. Therefore, we obtained simple captions for each video using InstructBLIPP (Dai et al. 2024) and used them only for the OOD experiment with the Landscape dataset. Specifically, extracted the first frame of each video and passed it to the InstructBLIP. As an instruction for the captioning, we used the sentence: "Write a short sentence for the image starting with 'The image captures a scene with'." and generated up to 128 characters for each video. Note that these captions are generated only from videos without audio. They tend to describe only visual content and provide less information about audio. Therefore, these captions may not be suitable for audio generation by and 4 where IB-TA is worse than IB-TV. These results show that the text condition strongly affects the quality of generated results. Designing and manipulating the input condition is also important for the joint generation, which we leave for future work.
AudioLDM, as we show in
A.5 VISUALIZATION OF THE EFFECT BY EACH LOSS FUNCTION WITH TOY DATASETS
Figure visualizes the effect of each loss function. The top row shows the IND case, and the bottom shows the OOD case. At each row, the lefumost column shows the samples from the target distribution,
17

---

Published as a conference paper at ICLR 2025
Table 6: Hyperparameters used in the experiments with benchmark datasets (used in Sections 4.2 and 4.3 t
We jointly generate audio and video by MM-Diffusion in the IND casc.
Model
Discriminator (IND)
Discriminator (OOD)
Hyperparameters
Audio base model
MM-Diffusion
AudioLDM / Auffusion
Video base model
MM-Diffusion' ResBlocks
AnimateDiff / VideoCrafter2
Architecture
ResBlocks
Audio input dims
1, 25600
8, 50, 16
Video input dims
16, 64, 64 10
4, 16, 32, 32 8
Video fps
Audio fps
16k
16k
Duration (sec)
1.6
2.0
# of ResBlocks per resolution
2
4
Audio conv type
ld
2d
Audio conv dilation size
1,2, 4, ..., 21. 2d + ld
I
Video conv type
2d + ld 256
Channels
128
Channel multiplier
1, 4
1,2,4
Audio downsample factor
4 (1, 2,
(1,2)
Video downsample factor
(1, 2,
Output dim
1 GroupNorm SiLU
Normalization
GroupNorm SiLU
Activation
Timestep dim
128 Adaptive
256
Timestep input type Conditional input Condition drop rate Text embed dim for video Text embed dim for audio Text embed input type Optimizer
Adaptive
Text
0.1
768
512
Add
Adam
Adam
Learning rate
0.001
0.001
Total batch size Total training epochs
16
32
100
Landscape: 100 / VGGSound: 10
Total of params for base models
133M
2.15B / 2.86B
Total of params for discriminator
13M
132M
Diffusion setup
Diffusion steps
1000
1000 scaled linear
Noise schedule
linear
Bo and 3r for audio
0.0001 / 0.02
0.0015 / 0.0195
80 and ST for video
0.0001 / 0.02
0.00085 0.012
Inference parameters
Sampler
DPM-Solver
DPM-Solver ++
NEE
20
50
Order
3
2
Audio text guidance scale
2,5 8.0
Video text guidance scale
7.5
Audio joint guidance scale
1.0
1.0
Video joint guidance scale
1.0
1.0
18

---

Published as a conference paper at ICLR 2025
GT
disc
denoise
L Lall
Figure Visualization of the generated samples across loss functions used train our guidance module with the toy dataset. The top row shows the IND setting, and bottom shows OOD setting.
Table 7: Ablation study about the loss functions on the landscape datasct.
Video
Audio
Cross-modal
Settings
FVD 1
IB-TV
FAD J
IB-TA
IB-AV
Dataset
207
0.16
0.247
No Joint
852
0.308
8.26
0.053
0.093
Ldise
818
0.305
7.80
0.060
0.102
Ldenoise
700
0.303
7.19
0.064
0.088
Call
667
0.303
7.69
0.061
0.102
and the others show generated samples with our guidance modules trained by different loss functions. Note that the NNL each setting has been shown Table [ In the IND case, both Ldise and Ldenoise work similarly, whereas in the OOD case, the guidance trained by only Laise can roughly concentrate the samples at four corners but struggles force them be single peak. In contrast, combining with Ldenoise drastically improves this issue, and using both of them yields the best result.
A.6 ABLATION STUDY ABOUT THE LOSS FUNCTIONS ON REAL DATA
We conducted an ablation study about the loss functions on the landscape dataset. Table shows quantitative results of our model trained with Ldises Ldenoises and Can- We used Audiol.DM / AnimateDiff base models this experiment. The model trained on Caleec achieves better IB- AV and shows good performance in terms of the cross-modal semantic alignment, while it shows moderate improvements in the fidelity scores. On the other hand, the one trained on Ldenoise achieves better fidelity scores while suffering from lower cross modal alignment scores. Compared to these, the one trained on Lall achieves both better fidelity scores and cross-modal alignment, which has the best of both worlds.
A.7 VALIDATION OF THE PARAMETER SIZE AND THE TRAINING EPOCH
We explored the effect of the parameter size of the discriminator as well as the number of training epochs in the IND setting with the Landscape dataset.
19

---

Published as a conference paper at ICLR 2025
Table 8: Ablation study for the channel size C' of the discriminator with the number of ResBlocks per resolution fixed to
Architecture
Metries
Settings
of params
FVD L
FAD
IB-AV
Base model (MM-Diffusion)
133M
447
5.78
0.156
C' 32, L
798K
423
5.60
0.159
C 64,
3.2M
410
5.48
0.161
C 128, L
12.7M
405
5.52
0.162
C 256, L
50.7M
399
5.46
0.160
Table 9: Ablation study for the number ResBlocks per resolution of the discriminator with the channel size C' fixed to 32 and 128.
Architecture
Metries
Settings
# of params
FVD J
FAD
IB-AV 1
Base model (MM-Diffusion)
133M
447
5.78
0.156
C' 32,
401K
421
5.65
0.159
C' = 32, L
798K
423
5.60
0.159
C 32,
1.6M
425
5.51
0.160
C 128, L
19.0M
415
5.60
0.161
C 128, L 2
12.7M
405
5.52
0.162
C = 128, L = 4
25.3M
407
5.43
0.162
Table shows the effect of increasing the channel size (denoted by C) with the number of ResBlocks resolution fixed. As we increase the channel size, performance substantially increases for both the single-modal fidelity score and the multimodal alignment score. We observed that the multimodal alignment is slightly degraded C' 256 and concluded that the setting with C' 128 is the most appropriate for multimodal guidance, requiring only less than 10% additional parameters. Note that proposed method successfully improves the base model's performance even when only used less than 1% additional parameters (the setting with C 32).
Table 9 shows the effect of increasing the number of ResBlocks per resolution (denoted by L) with the channel size fixed. We did experiments on the settings with C 32 and = 128. For both settings, increasing T. improves FAD mainly, while the differences of other metrics marginal. We suppose this may be the effect of the increase in the receptive field. Since MM-diffusion trained on the raw data space (pixel for video and waveform for audio), the size of the time axis for audio is extremely large. Therefore, following the implementation of MM-diffusion, we used dilated convolutions for the audio branch capture features multiple scales. Using more ResBlocks containing dilated convolutions may improve the audio encoder's capacity, resulting in improved audio performance.
Figure the performance of our guidance module over the number of training epochs. We used C 128 and T, = 2 for the discriminator and trained it longer. For evaluation, we generated samples with five different random seeds and computed the average and standard deviation of each metric. Our guidance module converged around 100 epochs, and we did not observe clear improvement by training further.
A.8 DISCRIMINATOR STRUCTURE VARIANTS
Our discriminator basically consists two modules terms of functionality: i) feature extraction module and ii) feature fusion module. We used stack of ResBlock layers to extract modality- specific features for i) and channel-wise concatenation of pooled features along the time axis, followed by MLP to handle crossmodal interaction for ii). We denote these default structures as "Res" for i) and "MLP" for ii). To further improve the performance of joint generation, we tested integrating transformer architecture into discriminator, inspired by its recent success (Vaswani
20

---

Published as a conference paper at ICLR 2025
440
5.7
430
5.6
0.160
5.5
0.158
5.4
150
300
100
250
150
200
Thaining
Thaining
Traintng
Figure Ablation study training epochs. We trained discriminator with C and L for training epochs. We generated samples with five different random seeds and computed their average and std.
Table 10:
Ablation study for the different discriminator designs. For the notation of the architectures, denote "i) ///i) ii)" where is the architecture of the feature extractor module and ii) is the feature fusion module described Appendix A.8 used "Auffusion VideoCrafter2" as base model.
Metrics
Discriminator architecture
FVD 1
IB-TV 1
FAD
IB-TA t
IB-AV t
No discriminator
831
0.302
5.32
0.190
0.192
Res MLP (default)
704
0.302
4.79
0.197
0.201
Res / Naive transformer
709
0.303
4.69
0.200
0.202
Res / Frieren
710
0.302
4.77
0.202
0.205
ViT MLP
730
0.301
5.07
0.192
0.199
ViT / Naive transformer
716
0.302
5.09
0.194
0.202
ViT Frieren
720
0.302
5.06
0.194
0.202
Dosovitskiy et 2021 Peebles and Xic 2023 Wang al., 2024). Specifically, we used two input types for the transformer encoder to replace the module ii). The audio and video features extracted from have shapes (B, Fa. C') and (B, F,,, C), respectively, where B is the batch size, F, and F, are the numbers of frames in audio and video respectively, and C' is the channel size. For first design, we concatenated these two features along the frame axis to get crossmodal feature of the shape (B, F. + F... C) (denoted as "Naive Transformer"). For the second design, following the design proposed by Wang et al. (2024), we upsampled shorter features and concatenated these two features along the channel axis get a crossmodal feature of the shape (B, max( Fa; F.), 2C) (denoted as "Frieren"). For both settings, the input is supplemented with learnable positional embedding and special token, which is used for the final output, and the transformer encoder with four layers is applied. We implemented the transformer using xFormer and trained both discriminator designs top of Auffusion / VideoCrafter2 VGGSound dataset. Table 10 shows the results. Overall, both designs improve the fidelity and crossmodal alignment. This result indicates that more sophisticated discriminator structure leads to better performance of crossmodal generation. Although we also briefly tested replacing ResBlock layers with vision transformer architecture (Dosovitskiy et al., 2021] Arnab al., (022)) (the 4D audio latents are processed by VIT, and the 5D video latents not observe performance improvement (denoted as "VIT" in Table I0). Further exploration of the discriminator structures would be an interesting direction for future work.
2017:
base models for ten epochs on the
processed by ViViT) for i), do
A.9 COMPUTATIONAL COST FOR THE TRAINING AND INFERENCE
To highlight the computational efficiency of our proposed method, we compared the computation time for the training and inference of base models, both with and without our method. For this experiment, we used the same network configurations as described in Sections 4.2 and 4.3 with batch size 8, and conducted all runs on single Nvidia H100 GPU.
21

---

Published as a conference paper at ICLR 2025
Table 11: Computational cost for the training. The Computation time of forward and backward execution for the base models and our discriminator are shown. Those of the base models include computation time required for both audio and video. All numbers except "Speed-up Ratio" show the computation time in seconds.
Base Models
MM-Diffusion
AudioLDM / AnimateDiff Auffusion VideoCrafter2
Base Fwd
0.273 (0.009)
0,254 (0.057)
0.349 (0.020)
Base Bwd
0.660 (0.001)
0,505 (0.007)
0.952 (0.013)
Disc Fwd
0.054 (0.006)
0,081 (0.007)
0.081 (0.006)
Dise Bwd
0.135 (0.009)
0.186 (0.005)
0.186 (0.007)
Baseline 1step
0.933 (0.009)
0.759 (0.057)
1.301 (0.023)
Ours 1step
0.462 (0.014)
0.521 (0.058)
0.616 (0.022)
Speed-up Ratio
202%
146%
21 %
Table 12: Computation time for the inference with and without the proposed method. All numbers except "Overhead" show the time in seconds for generating one pair of audio and video ([second sample]).
Base Models
MM-Diffusion
AudioLDM / AnimateDiff Auffusion / VideoCrafter2
w/o ours
1.09 (0.03)
3.10 (0.02)
4.45 (0.03)
w/ ours
1.25 (0.04)
3.77 (0.09)
4.91 (0.04)
Overhead
15%
22%
10%
Table TT shows the elapsed time for each base model's forward and backward computations and our discriminator's during single training step. Each configuration was run 500 times, with the mean and standard deviation for each value reported the table. As a baseline, we measured the computation time for the single training step of all base models (represented the sum of Base Fwd and Base Bwd in Table IT). Note that this baseline reflects the cost for fine-tuning base models cross-mode modules (Tang ett al. 2023), thereby serving as lower-bound estimate for the training costs associated with building joint generation model by simply combining base models. Our method accelerates training by approximately 1.5 to 2 times compared to the baselines. These results demonstrate that our method integrates base models into a joint model in a computationally efficient manner.
without incorporating additional learnable layers (Hu et al. 2022
Table 12 shows the time required to sample a pair of audio and video. Due to the additional computation required by the discriminator, the inference time is slightly longer (around 10 to 20 %) compared to those of the base models. This overhead can be reduced by designing a more computationally efficient network architecture.
A. COMPARISON WITH BASELINES ON THE SEMANTIC AND TEMPORAL CROSS-MODAL ALIGNMENT
In this section, show the additional quantitative comparison between method and baselines in terms of the cross-modal alignment. In addition to the semantic alignment evaluation using the ImageBind score, computed the AV-Align score (Yariv et al. 2024) for the temporal align- ment evaluation. First, we briefly explain the AV-Align score and some adjustments to achieve stable computation in our experimental settings Appendix A. Then, show the results in Appendix A. 10.2
A.10.1
COMPUTATION OF THE AV-ALIGN SCORE
The AV-Align score Yariv al. 2024 is defined Intersection-over-Uue Union (loU) between the detected audio onsets and the peaks of the optical flow the video. Let AA be a set of the onset peaks detected from the audio, and V be set of the peaks of the optical flow in the video.
22

---

Published as a conference paper at ICLR 2025
Table 13: Comparison of the modsl alignment scores under the OOD setting the VGGSound dataset.
Method
IB-AV AV-Align
Grand truth
0.296
0.336
AnimateDiff Spec
0.288
0,084
VideoCrafter2 - SpecVQGAN
0.291
0.089
AnimateDiff DiffFoley
0.295
0.098
VideoCrafter2 - DiffFoley
0.319
0,105
AudioLDM / AnimateDiff
0.320
0.121
*MMDisCo (Res / MLP)
0.330
0.127
Auffusion VideoCrafter2
0.268
0.192
MMDisCo (Res / MLP)
0.287
0,201
+ MMDisCo (Res Frieren)
0.292
0.205
AV-Align 2'AUD
1
E1
vI+ E1
(37)
AEE
NEV
where 1aa E V and [ AA] are the number of the valid peaks of audio and video. The peak is considered valid placed within three frames in the other modality.
In the official implementation, the AV-Align score is computed by the following equation:
ANV
AV-Align
(38)
|A| + - [VVV
where they use the value of) E laa E V] for the intersection |AnVI. By its definition, the value of the IOU should be in the range of [0, 1]. If the audio peaks do not have one-to-many correspondences with video peaks, the computed value with |An ΣaEA 1(a V/ is always in [0, 1].
To ensure no one-to-many correspondence occurs, we need to set the peak detection interval properly. If the interval of the peaks of either modality is sufficiently smaller than the other, one-to-many correspondences are likely to occur. In the configuration of Yariv et al. (2024), they used 24 fps videos (the detection interval about 42 ms) and set the onset detection interval as ms. the other hand, we used eight fps videos for the evaluation, which are three times larger intervals than the official configuration. When we compute the AV-align score with its configuration, the computed value occasionally exceeds 1. To compensate for this gap. set the onset detection interval as 96 ms. We did not observe the value exceeding our experiment with this configuration.
A. 10.2 EVALUATION ON THE TEMPORAL AND SEMANTIC ALIGNMENT
We computed AV-align scores for the base models and those with our proposed method on the VGGSound dataset. All configurations are the same as Section 4.
Table 1 shows the AV-Align and the IB-AV scores of our method and base models. For the AV-Align, "Audiol.Dy AnimateDifff + ours (Res MLP)" achieves the best score among all models. While it also achieves a better IB-AV score than existing models, the IB-AV score significantly lags behind GT's. Using more sophisticated base models (" ("Auffusion / VideoCrafter2" ours) significantly improves the IB-AV score with a comparable AV-Align score to the GT. Notably, our proposed method successfully improves both AV-Align and IB-AV scores of both base models.
We also observed that using more sophisticated network structure (" ("VideoCrafter2/ Auffusion + ours (Res/ / Frieren)") further improves both scores. How to design the discriminator to improve the specific types of alignment would be a possible direction of future work. Additionally, how to create dataset would be possible direction to improve the alignment. For example, Luo al. 2023)
23

---

Published as a conference paper at ICLR 2025
Table 14: Subjective evaluation comparing the generated results of base models (Auffuusion VideoCrafter2) with those our method. The evaluation considered four aspects: Video Quality (VQ), Audio Quality (AQ), Semantic Aligument between audio and video (SA), and Temporal Alignment between audio and video (TA).
"N/A" indicates the percentage of evaluators who were unable to determine which sample was superior.
Method
VQ
AQ
SA
TA
Auffusion / VideoCrafter2
4.2%
12.5%
6,9%
9.7%
MMDisCo 94.4%
44.4%
81.9%
80.6 %
N/A
1.4%
43.1% %
11.1% 1%
9.7%
proposes using time-shifted audio and video pairs to train audio-visual latent space, which used for the condition of their generative model, to improve the temporal alignment. In our experiments, we only used generated samples from base models for the fake samples. Adding such fake pairs as simulated temporal misalignment for the fake samples would improve the alignment of generated samples, which we leave for future work.
A.11 SUBJECTIVE EVALUATION
In addition the objective evaluation, we conducted user study to assess the perceptual quality, We generated samples using Auffusion / VideoCrafter2, both with and without our method, employing 10 labels randomly sampled from the VGGSound dataset. This resulted where one was produced by the base models and the other by the base models with our method. We asked human evaluators to answer the following four questions per each pair of videos:
pairs of two videos,
1. Which video is of higher quality regardless of the audio?
2. Which audio is of higher quality regardless of the video?
3. Which pair of audio and video has better semantic alignment?
4. Which pair of audio and video has better temporal alignment?
For each question, evaluators were given three response options: "Video A is better," "Video B is better," and "N/A." We collected 320 answers in total from 8 evaluators. Table 14 provides summary of the results. Our method consistently enhances the quality generated samples across all aspects, which aligns with the objective evaluation results presented in Section and Appendix 1oppendi(AA.10.. A.10.
A.12 GENERATED AUDIO AND VIDEO PAIRS
This section shows the results generated from the base models and our method. Note that, to evaluate the effectivvness our proposed method visually, we used the same random seed for the base models and our method, with which we can expect them to provide similar samples.
Figure the results of unconditional generation from MM-Diffusion and that with our guidance module. Since MM-Diffusion was already trained by the same dataset (i.e., the IND setting), the generated pairs of audio and video look well-aligned in their semantics. However, our guidance module improves the quality of detail at each modality forces the generated audio and video to be temporally aligned, which is coherent with the quantitative evaluation results.
Figure Sauuee7/seer shows the results of text-conditional generation from base models (AudioLDMi Animate- Diff) and our method. this case, since the base models independently generate audio and vidco, the generated audio and video not always well-aligned (i.c., the OOD setting). Our guidance module substantially improves the quality of generated samples from base models compared to the IND setting. As described in Section 4.3] guidance tends to generate players successfully, and the generated audios are also temporally aligned with their motion.
24

---

Published as a conference paper at ICLR 2025
No joint
Ours
Figure More generated samples from MM-Diffusion ours trained on Landscape dataset in IND
setting. We used the same random seed for both settings, and the generated results at each row should be similar.
25

---

Published as a conference paper at ICLR 2025
No joint
Ours
Ocean burbling
Man speaking
Driving motorcycle
Arc welding
Playing
saxophone
Playing
harp
playing
hockey
Figure 7: More generated samples from AnimateDiff' / AudioLDM and our method trained on the VGGSound dataset in the OOD setting. The captions given for the generation of each sample shown in the leftmost column, and used the random seed for both settings.
26


Reviews:
[{"review_id": "sCc6dbtRyA", "reviewer": "Reviewer_pfZF", "summary": "This paper presents a method for constructing an audio-video generative model with low computational cost by leveraging pre-trained single-modal diffusion models for audio and video. The authors propose a lightweight joint guidance module that aligns audio and video outputs by adjusting scores to approximate the joint distribution, computed through the gradient of an optimal discriminator that distinguishes real and fake audio-video pairs.", "questions": "1. How is the low computational cost demonstrated in this work? Are there specific experiments or data that quantify this claim?\n2. While the quality of the generated 2-second samples is good, how does the model perform on longer video samples?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper tackles an important problem in multi-modal generative modeling by introducing a cost-effective approach to synchronize audio and video generation, offering practical value for applications constrained by limited computational resources.\n2. The custom loss function helps to steady the gradient of the discriminator. Extensive evaluations on benchmark datasets highlight enhanced fidelity for individual modalities and better alignment between audio and video, all achieved with minimal additional parameters, underscoring the efficiency of the proposed method.", "weaknesses": "1. Although I understand that the experiments are based on MM-Diffusion, the AIST and Landscape datasets are relatively small. The authors should discuss the generalizability of their method on larger datasets.\n2. The paper claims that the method is model-agnostic; however, only one baseline is tested, which makes it difficult to confirm its general applicability.\n3. Besides quantitative metrics, a subjective evaluation of the generated samples, would be beneficial to better assess the quality from a perceptual standpoint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3vzMxoxU8T", "reviewer": "Reviewer_F4ji", "summary": "The submission presents a novel approach to audio-video joint generation with frozen single-modal generative models using a discriminator-guided cooperative diffusion process. The authors leverage existing pre-trained single-modal models for audio and video, aiming to reduce computational costs while enhancing multi-modal alignment. This approach, distinct from traditional model-specific or architecture-dependent techniques, incorporates a lightweight guidance module trained as a discriminator. The model effectively distinguishes between real and fake audio-video pairs, providing stability through a denoising regularization function inspired by standard diffusion methods.", "questions": "* As mentioned in the Weaknesses, could the authors compare the training cost for different training setups?\n* There are many values omitted in the FVD columns of Table 3 and 4. Are they the same as the corresponding rows below or different? Could the authors either explain or provide new values?\n* Can the proposed method generalize to other pairs of modalities beyond video and audio?\n* Will the proposed method remain valuable if diffusion models jointly trained on both modalities become widely available? If so, what does the cost-performance trade-off look like?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed method is well motivated and clearly presented, with a focus on achieving joint audio-video generation with low computational cost.\n* The proposed method is training-based but model-agnostic, which can be more broadly applicable than model-specific designs.\n* Extensive results on multiple datasets demonstrate the effectiveness of the proposed method in improving modality alignment and generation quality of videos and audio.", "weaknesses": "* While the single-modal diffusion models remain frozen, the computation cost for training the joint discriminator, including forward passes through the frozen modules, is not reported. This cost needs to be compared to the total pretraining cost and a reasonable baseline of jointly trained diffusion models on both modalities.\n* When text condition is provided to both diffusion models, the captions act as a weak alignment signal. Better experimental designs would be desired to remove the impact of this signal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1zgsG6lnaR", "reviewer": "Reviewer_Kk8X", "summary": "This paper aims to leverage the pre-trained single-modal diffusion models for generating aligned video and audio. To achieve this objective and involve minimal computing costs, the authors proposed a lightweight joint guidance module to adjust the independent distribution from base models to match the joint distribution over audio and video by designing and training a discriminator. Based on the experimental results on the benchmark datasets, it seems that the proposed method improves the alignment score of generated audio and video while involving reduced trainable parameters.", "questions": "1. Did you adopt other evaluation metrics to evaluate the proposed method on the performance of temporal alignment between generated audio and video? Because it seems that the ImageBind score is just for semantic alignment.\n2. Did you try different discriminator structures or larger trainable parameters to see if there will be future performance improvement? \n3. Why the metric scores of MM-Diffusion in Table 2 are different from the ones of the original paper? \n4. It is better to compare the proposed method with \"Seeing and Hearing\" discussed in the related works by using the same pre-trained base models.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It is interesting to solve the joint audio-video generative problems by adjusting the pre-trained modality-specific generators with minimal trainable parameters rather than fully training the joint audio-video generation models.\n2. Sufficient mathematic equations and proofs are given to support that the discriminator-based guidance can adjust the estimated scores from audio diffusion and video diffusion to approximately match the score of joint audio-video distribution.\n3. The experimental results shown in the table present the proposed method can improve the alignment performance of generated audio and video in both in-domain and out-of-domain settings.\n4. The paper is generally well-written.", "weaknesses": "1. The description that training a discriminator can adjust the pre-trained modality-specific diffusion models towards a joint audio-video generator is not very clear. For example, how does the discriminator-based guidance achieve both semantic and temporal alignment?  It is better to make it more clear. \n2. The improvement of metric score is limited especially for the OOD setting. In addition, from Table 4, the FVD and IB-TV scores of AudioLDM/AnimateDiff with the proposed guidance module are even worse. \n3. Based on the provided video files, although the performance in the IND is improved by the proposed method, the OOD setting performs poorly in generating high-quality video with the audio track. \n4. It is meaningful to see how different base models affect the performance. Whether more powerful pre-trained video and audio generators further improve the performance when equipped with the same guidance module?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v6zyzSsLod", "reviewer": "Reviewer_U5ht", "summary": "This paper presents a novel method for creating an audio-video generative model using pre-trained generative models for audio and video separately, and then guiding these models to work together using a discriminator.\n\nThe key innovation is a lightweight joint guidance module (an audio-video discriminator), which adjusts the scores estimated by the base models to match the score of the joint distribution over audio and video. This guidance is computed using the gradient of the discriminator, which distinguishes between real and fake audio-video pairs. The authors also introduce a regulation loss function that stabilizes the gradient of the discriminator.", "questions": "1. The training of generative adversarial models is always easy to explode. Did the authors face similar problems when optimizing the discriminator?\n\n2. Since the additional training module is lightweight, I think the training procedure can be efficient. Can the author provide detailed training consumption required for training the discriminator? How much inference time does it increase?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The introduced joint module is lightweight and can be adapted to different single-modal generative models.\n\n2. Authors conducted thorough experiments, including toy datasets, and in-domain and out-domain settings, demonstrating the effectiveness of the proposed method.", "weaknesses": "1. Statement of datasets and evaluation metrics are duplicated in Sec 4.2 and 4.3.\n\n2. In the section of related work, the author referred to Xing et al. (2024) as their similar work. However, neither quantitative nor qualitative comparisons are conducted with this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)