Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation

## Abstract

Abstract In this study, we aim to construct an audio-video generative model with minimal computational cost by leveraging pre-trained single-modal generative models for audio and video.
To achieve this, we propose a novel method that guides each single-modal model to cooperatively generate well-aligned samples across modalities.
Specifically, given two pre-trained base diffusion models, we train a lightweight joint guidance module to adjust scores separately estimated by the base models to match the score of joint distribution over audio and video.
We theoretically show that this guidance can be computed through the gradient of the optimal discriminator distinguishing real audio-video pairs from fake ones independently generated by the base models.
On the basis of this analysis, we construct the joint guidance module by training this discriminator.
Additionally, we adopt a loss function to make the gradient of the discriminator work as a noise estimator, as in standard diffusion models, stabilizing the gradient of the discriminator.
Empirical evaluations on several benchmark datasets demonstrate that our method improves both single-modal fidelity and multi-modal alignment with a relatively small number of parameters.

## 1 Introduction

Deep generative modeling has progressed rapidly in the last few years.
Diffusion models are one of the keys for this progress, and they can be applied to various tasks including image, audio, and video generation [].
Following the success of single-modal data, there have been several attempts to apply diffusion models to multi-modal data [].
However, since multi-modal data is more complex to handle and is harder to collect than single-modal data, developing multi-modal generative models by simply extending single-modal models remains challenging.
To alleviate this problem, one promising way is to integrate several pre-trained single-modal models to build a multi-modal generative model [, ].
As there are numerous publicly-available models that can generate high-quality single-modal data [, , ], their effective integration would substantially reduce the computational cost to build multi-modal generative models.
In this work, among various types of multi-modal data, we tackle audio-video joint generation on top of two pre-trained diffusion models for audio and video.

For audio-video generation, there are two approaches for integrating several single-modal models: training-free and training-based.
The training-free approach employs pre-trained single-modal base generative models with their parameters fixed and uses an off-the-shelf recognition model to guide them to generate well-aligned samples across modalities [].
While this can improve multi-modal alignment without any training cost, it may degrade the fidelity of a single modality [].
In contrast, the training-based approach extends single-modal generative models for multi-modal data by designing a neural network specifically tailored to it [, ].
Although this approach can achieve better performance in terms of both multi-modal alignment and the fidelity of each single modality, it tends to require a significant computational cost for training.
More importantly, their architectures for handling multi-modal data heavily depend on that of base models (i.e., not model-agnostic).
Therefore, we must manually redesign them when updating base models, which requires a lot of trial and error.
In short, there is a trade-off between the quality of generated samples and model dependency, which increases the computational cost, in the existing two approaches.

In this paper, we propose a novel method that is training-based but model-agnostic, which does not require a huge computational cost.
Specifically, we introduce a lightweight joint guidance module on top of two base models for audio and video that adjusts their outputs for audio-video joint generation.
We assume that pre-trained base models are black box diffusion models (i.e., we can access only their outputs and do not depend on a specific architecture design like cross-attention to construct a joint generation model), and we formulate the joint generation process as an extension of the classifier guidance (C-guide) for single-modal data [, ].
We theoretically show that joint guidance can be computed through the gradient of the optimal discriminator that distinguishes real audio-video pairs from fake ones independently generated by base models.
We only train the discriminator with proper regularization inspired by Denoising Likelihood Score Matching (DLSM) [].
Extensive experiments on several benchmark datasets demonstrate that our proposed method can efficiently integrate single-modal base models for audio and video into a joint generation model, maintaining the performance of each single-modal generation without incurring a significant computational cost.

## 2 Related work

### 2.1 Audio-video joint generation by diffusion models

Since an audio-video pair is one of the most popular types of multi-modal data, several works train diffusion models with such pairs to achieve a conditional single-modal generation: video-conditional audio generation [, , ] or audio-conditional video generation [, , ].
However, these works mostly focus on only a single modality as a generation target.
Extending these works to the joint generation of audio and video is not trivial due to the high dimensionality and heterogeneous data structure of audio-video joint data.

Joint generation of audio and video pairs has been addressed in only a few recent studies [, , ].
MM-Diffusion [] is a multi-modal diffusion model specific for audio-video joint generation.
While MM-Diffusion is trained from scratch on audio-video pairs, CoDi [] integrates several pre-trained single-modal diffusion models by adopting environment encoders to share modality-specific information across modalities during the generation process.
Since they adopt a novel architecture strongly tied with the main network of diffusion models, it is difficult to directly apply their method to other types of architectures, which hinders the applicability of their method.
In contrast, our method handles base models as a black box and depends only on their outputs. Therefore, our method is widely applicable across any types of architectures used in base models.

Xing et al. [] shares a similar motivation to ours in the sense that they achieve audio-video cooperative generation from pre-trained single-modal base models.
Given multi-modal embedding models (e.g., ImageBind []), they utilize universal guidance [] to make the embeddings from two modalities close.
Although their approach is model-agnostic and has wide applicability to any base models, their guidance roughly ensures semantic alignment in the space of embeddings learned by ImageBind.
Thus, it does not achieve sampling from the true joint distribution of audio-video pairs.
In contrast, our method is theoretically grounded to adjust the scores predicted from base models to that of the joint distribution, explicitly achieving sampling from the joint distribution.

### 2.2 Guidance for pre-trained diffusion models

The guidance [, ] provides a proper way to update intermediate representations at each generation step so that the generated samples satisfy a given condition, even when the model is not trained for that type of conditional generation.
Since the guidance does not require additional training of diffusion models, it is widely used to control the generation process with additional conditional signals.
C-guide [, ] was proposed to guide an image generation model by class label, such as cat or dog, using an additionally trained classifier.
To achieve beyond class-conditional generation, several works further extend this to utilize off-the-shelf recognition models [, ].
Our proposed method is the first attempt to extend C-guide for joint multi-modal generation.
We derive our methodology from the theory of C-guide, enabling us to theoretically sample data from the joint distribution.

The samples generated with C-guide may suffer from degraded quality without careful tuning of its scale []. Chao et al. [] denote this problem as the score mismatch issue, where the posterior scores estimated by a diffusion model and a classifier are unstable and deviate from the true ones.
They proposed DLSM loss to alleviate this problem.
DLSM regularizes the gradient of a classifier to match the residual error of noise prediction by trained diffusion models, providing a stable gradient for the generation process.
Our work can be seen as an extension of DLSM for multi-modal generation, and we employ DLSM to stabilize the gradient of our discriminator.

## 3 Methodology

Figure: Figure 1: Overview of the training process of our proposed method. We train a joint discriminator on the top of two base diffusion models so that it distinguishes real video-audio pairs and fake ones generated by base models. Additionally, we adopt a denoising objective, as in standard diffusion models, to match the gradient of the discriminator with regard to the inputs to the residual noise between ground truth noises and predicted noises from base models.
Refer to caption: x1.png

In this section, we first briefly review Diffusion models on a single modal data.
Here, we follow Denoising Diffusion Probabilistic Models (DDPM) [], which are broadly used as a standard definition of diffusion models. Then, we describe our formulation of joint score estimation on top of two pre-trained diffusion models.

### 3.1 Preliminary: Diffusion Models

#### Basics

Diffusion models [] are a family of probabilistic generative models that reverse a diffusion process from data to pure noise. Specifically, let $x_{0}\sim p(x_{0})$ be a sample of a data distribution and $x_{t}$ be a noisy representation at diffusion timestep $t\in\{0,1,...,T\}$.
A forward diffusion process is defined as a Markov process:

$$ $q(x_{t}|x_{t-1})=\mathcal{N}(\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I),$ (1) $$

where $\beta_{t}\in(0,1)$ controls how fast the data is diffused at each timestep.
On the basis of Eq. (1), $x_{t}|x_{0}$ also follows the Gaussian distribution of $p(x_{t}|x_{0})=\mathcal{N}(\sqrt{\bar{\alpha}_{t}}x_{0},(1-\bar{\alpha}_{t})I)$, where $\bar{\alpha}_{t}=\prod_{s=1}^{t}(1-\beta_{s})$ is the accumulation of diffusion coefficients, and the noisy sample at the last step $x_{T}$ would follow the standard Gaussian distribution $\mathcal{N}(0|I)$ with an appropriate setting of $T$ and $\beta_{t}$.
If $\beta_{t}$ is small enough (or $T$ is large enough), the reverse process of Eq. (1) can be approximated to be Gaussian.
Diffusion models are trained to estimate its mean by predicting the noise in $x_{t}$ as follows:

$$ $\displaystyle p_{\phi}(x_{t-1}|x_{t})=\mathcal{N}(\mu_{\phi}(x_{t},t),\sigma_{ t}^{2}I),$ (2) $\displaystyle\mu_{\phi}(x_{t},t)=\frac{1}{\sqrt{1-\beta_{t}}}\Big{(}x_{t}- \frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\phi}(x_{t},t)\Big{)},\; \sigma_{t}^{2}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t},$ (3) $$

where $\epsilon_{\phi}$ represents the noise prediction model with parameters $\phi$.
This model $\epsilon_{\phi}$ can be trained by minimizing the following mean squared error between the noise predicted by the model and that added to the data:

$$ $\phi^{*}=\underset{\phi}{\operatorname{argmin}}\,\mathbb{E}_{x_{0},\epsilon,t} \left\|\epsilon_{\phi}(\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}} \epsilon,t)-\epsilon\right\|^{2},$ (4) $$

where $\epsilon\sim\mathcal{N}(0,I)$ is a noise, and $t\sim\mathcal{U}(1,T)$ is a timestep.
For generation, we can sample $x_{0}$ through the iterative sampling from $t=T$ to $1$ using Eq. 2 with $\epsilon_{\phi^{*}}$.

Equation (4) is a form of denoising score matching.
Specifically, from Tweedie’s formula [], the noise added to the data and the score function is equivalent up to a constant factor.
On the basis of this fact, we can approximate the score function by using a noise prediction network trained by Eq. (4) as:

$$ $\nabla_{x_{t}}\log q(x_{t})\approx-\frac{1}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon _{\phi^{*}}(x_{t},t).$ (5) $$

From Eq. (5), a noise prediction model trained by Eq. (4) can be considered as a model estimating score of $q(x_{t})$.

#### Guidance for conditional generation

We can extend the generation process of diffusion models to the conditional one using classifier guidance [, ].
This can be derived from the perspective of score estimation. By Bayes’ theorem, we can write the conditional score as:

$$ $\nabla_{x_{t}}\log q(x_{t}|c)=\nabla_{x_{t}}\log q(x_{t})+\nabla_{x_{t}}\log q (c|x_{t}),$ (6) $$

where $c$ is a conditional vector.
The first term in the right-hand side can be estimated by $\epsilon_{\phi^{*}}$ trained with Eq. (4).
The second term in the right-hand side can be estimated by computing the gradient of a classifier with respect to its input, where the classifier is trained to predict $c$ given $x_{t}$.

### 3.2 Joint score estimation on the top of pre-trained diffusion models

Our goal is to achieve joint generation based on two independently trained diffusion models.
Namely, let $x\in\mathbb{R}^{D_{x}}$ and $y\in\mathbb{R}^{D_{y}}$ be samples of two different modalities.
We would like to sample $x$ and $y$ from a joint distribution $q(x,y)$.
From the perspective of the score function, we need to estimate $\nabla_{x_{t}}\log q(x_{t},y_{t})$ and $\nabla_{y_{t}}\log q(x_{t},y_{t})$.
By Bayes rule, we can derive the following equations:

$$ $\displaystyle\nabla_{x_{t}}\log q(x_{t},y_{t})$ $\displaystyle=\nabla_{x_{t}}\log q(x_{t})+\nabla_{x_{t}}\log q(y_{t}|x_{t}),$ (7) $\displaystyle\nabla_{y_{t}}\log q(x_{t},y_{t})$ $\displaystyle=\nabla_{y_{t}}\log q(y_{t})+\nabla_{y_{t}}\log q(x_{t}|y_{t}).$ (8) $$

Similar to the classifier guidance, the first terms in the right-hand side of Eqs. (7) and (8) can be estimated by the two diffusion models independently trained on the modality $x$ and $y$.
On the other hand, modeling the second terms in the right-hand side is not trivial.
One can naively construct two additional generative models of $x_{t}|y_{t}$ and $y_{t}|x_{t}$, and compute the gradient of such models with regard to condition vectors.
However, training them is difficult due to high dimensionality and requires a large computational cost.
Instead of training additional high-cost generative models, we train a single lightweight discriminator that distinguishes real pairs of $(x_{t},y_{t})$ and fake ones generated by pre-trained single-modal base models.
We theoretically show that it is sufficient to compute the gradient of this discriminator to approximate these terms.

Specifically, we propose to train a discriminator between the joint distribution $q(x_{t},y_{t})$ and independent distribution $q(x_{t})q(y_{t})$. Let $x^{\prime}\sim p_{\phi}(x)$ and $y^{\prime}\sim p_{\psi}(y)$ be fake paired samples independently generated by pre-trained diffusion models.
Here, we denote these diffusion models independently pre-trained by each modality $x$ and $y$ as $\epsilon^{(x)}_{\phi}$ and $\epsilon^{(y)}_{\psi}$, where $\phi$ and $\psi$ are the parameters of these models.
We train a discriminator $D_{\theta}:x_{t},y_{t},t\rightarrow[0,1]$ with the following loss function:

$$ $\displaystyle\theta^{*}=\underset{\theta}{\operatorname{argmin}}\,\mathbb{E}_{ (x,y)\sim q(x,y),x^{\prime}\sim p_{\phi}(x),y^{\prime}\sim p_{\psi}(y)}\left[ \mathcal{L}^{(\theta)}_{disc}(x,y,x^{\prime},y^{\prime})\right],$ (9) $\displaystyle\mathcal{L}^{(\theta)}_{disc}(x,y,x^{\prime},y^{\prime})=\mathbb{ E}_{\epsilon^{(x)},\epsilon^{(y)},\epsilon^{(x^{\prime})},\epsilon^{(y^{\prime })},t}\left[\log D_{\theta}(x^{\prime}_{t},y^{\prime}_{t},t)+\log\left(1-D_{ \theta}(x_{t},y_{t},t)\right)\right],$ (10) $$

where $\epsilon^{(z)}$ and $z_{t}$ are a noise and a noisy sample for $z\in\{x,y,x^{\prime},y^{\prime}\}$, respectively.
The noisy sample $z_{t}$ is derived by a forward process $z_{t}=\sqrt{\bar{\alpha}_{t}^{z}}z+\sqrt{1-\bar{\alpha}_{t}^{z}}\epsilon^{(z)}$, where $\bar{\alpha}_{t}^{z}$ is a coefficient of the forward diffusion process for each modality, and this is omitted from Eq. (10) for brevity.
Note that this discriminator $D_{\theta}$ is trained to output one for real pairs and zero for fake pairs.

Similar to a discriminator in Generative Adversarial Networks (GANs) [], an optimal discriminator $D_{\theta^{*}}(x_{t},y_{t},t)$ that minimizes Eq. (10) can be seen as an estimator of the density ratio $\frac{q(x_{t},y_{t})}{q(x_{t},y_{t})+p_{\phi}(x_{t})p_{\psi}(y_{t})}$.
Therefore, the second term in the right-hand side of Eqs. (7) and (8) can be approximated by utilizing $D_{\theta^{*}}$ as follows (see Section A.1 in appendix for details of this derivation):

$$ $\displaystyle\nabla_{x_{t}}\log q(y_{t}|x_{t})$ $\displaystyle\approx\nabla_{x_{t}}\log{\frac{D_{\theta^{*}}(x_{t},y_{t},t)}{1- D_{\theta^{*}}(x_{t},y_{t},t)}},$ (11) $\displaystyle\nabla_{y_{t}}\log q(x_{t}|y_{t})$ $\displaystyle\approx\nabla_{y_{t}}\log{\frac{D_{\theta^{*}}(x_{t},y_{t},t)}{1- D_{\theta^{*}}(x_{t},y_{t},t)}}.$ (12) $$

In summary, we can estimate a joint score as the sum of the scores independently estimated by base models $\epsilon^{(x)}_{\phi}$ and $\epsilon^{(y)}_{\psi}$, and the gradient of an optimal discriminator $D^{*}$ shown in Eqs. (11) and (12).(^1^11In this work, we adopt the most basic form of density ratio estimator. We expect that using advanced ones may improve the performance of joint guidance, which we leave for future work.)
In inference, we independently compute the outputs from base models and the gradient of our discriminator for the intermediate noisy samples $x_{t}$ and $y_{t}$ at each timestep.
Then, the sum of them are used as a predicted noise for the denoising step.
Note that, in the aforementioned discussion, we assume that the distribution of samples generated by base models are equal to the marginal distribution of joint data for brevity (i.e., $q(x_{t})=p_{\phi}(x_{t})$ and $q(y_{t})=p_{\phi}(y_{t})$).
However, our proposed method can be applied even when these are not equal (see Section A.2 in appendix for more details).
We also tested this setting in our experiment.

### 3.3 Residual score estimation from an optimal discriminator

As we described in Section 3.2, we can sample $x$ and $y$ from a joint distribution using guidance with a discriminator that can distinguish real paired-data and paired samples generated by pre-trained single-modal diffusion models.
However, in our preliminary experiments, using a discriminator trained by only Eq. (9) degrades the fidelity of generated samples as a single modality.
We conjectured that this is caused by the score estimation mismatch issue mentioned by Chao et al. [].
They argued that the score may deviate from the true one when estimating it using a gradient of a classification model.
To alleviate this issue, inspired by Chao et al. [], we adopt regularization for the gradient of the discriminator to match the residual of true noises and noises predicted by the base diffusion models.
Specifically, we define a denoising regularization loss as follows:

$$ $\displaystyle\mathcal{L}_{denoise}^{(\theta,\phi,\psi)}(x,y)=\mathbb{E}_{ \epsilon^{(x)},\epsilon^{(y)},t}\left[\mathcal{L}^{(\theta,\phi)}_{denoise} \left(x,y,\epsilon^{(x)},\epsilon^{(y)},t\right)+\mathcal{L}^{(\theta,\psi)}_{ denoise}\left(x,y,\epsilon^{(x)},\epsilon^{(y)},t\right)\right],$ (13) $\displaystyle\mathcal{L}^{(\theta,\phi)}_{denoise}\left(x,y,\epsilon^{(x)}, \epsilon^{(y)},t\right)=\left\|\epsilon^{(x)}-\epsilon^{(x)}_{\phi}(x_{t},t)+ \sqrt{1-\bar{\alpha}_{t}^{x}}\nabla_{x_{t}}\log\frac{D_{\theta}(x_{t},y_{t},t) }{1-D_{\theta}(x_{t},y_{t},t)}\right\|^{2},$ (14) $\displaystyle\mathcal{L}^{(\theta,\psi)}_{denoise}\left(x,y,\epsilon^{(x)}, \epsilon^{(y)},t\right)=\left\|\epsilon^{(y)}-\epsilon^{(y)}_{\psi}(y_{t},t)+ \sqrt{1-\bar{\alpha}_{t}^{y}}\nabla_{y_{t}}\log\frac{D_{\theta}(x_{t},y_{t},t) }{1-D_{\theta}(x_{t},y_{t},t)}\right\|^{2}.$ (15) $$

Note that the parameters $\phi,\psi$ of the base models are fixed during training and just used to compute noise estimation errors. Our final objective (denoted $\mathcal{L}_{all}$ hereinafter) to train the discriminator $D_{\theta}$ is the sum of the discriminator loss (Eq. (10)) and denoising regularization loss (Eq. (13)):

$$ $\theta^{*}=\underset{\theta}{\operatorname{argmin}}\,\mathbb{E}_{(x,y)\sim q(x ,y),x^{\prime}\sim p_{\phi}(x),y^{\prime}\sim p_{\psi}(y)}\left[\mathcal{L}_{ disc}^{(\theta)}(x,y,x^{\prime},y^{\prime})+\lambda\mathcal{L}_{denoise}^{( \theta,\phi,\psi)}(x,y)\right],$ (16) $$

where $\lambda$ is a weight to balance these two losses. We use $\lambda=1$ throughout this paper. The training and generation process of our proposed method are summarized in Algorithms 1 and 2, respectively.

Figure: Algorithm 1 Training process of $D_{\theta}$.

## 4 Experiments

In this section, we first show the results of preliminary experiments with toy datasets to confirm our proposed method can guide generation process of two independently trained diffusion models to the desired joint distribution. Then, we show the experimental results with real data.

### 4.1 Preliminary experiments with toy datasets

#### Datasets

To create the training dataset for a base model, we evenly sampled data from five Gaussian distributions whose means were $[-3,-1.5,0,1.5,3]$, and variances were all 0.01, resulting in 500 samples.
For training the guidance module, we constructed two types of datasets.
The first one simulates an in-domain (IND) situation, where the marginal distributions $q(x)$ and $q(y)$ were equal to the distribution on which the base model is trained.
This dataset was constructed by sampling from five Gaussian distributions whose means were $[(-3,1.5),(-1.5,-3),(0,3),(1.5,0),(3,-1.5)]$, respectively.
The second one simulates an out-of-domain (OOD) situation, where the marginal distributions are different from the distribution on which the base model is trained.
This dataset was constructed by sampling from five Gaussian distributions whose means were $[(-2.25,-2.25),(2.25,2.25),(-2.25,2.25),(2.25,-2.25),(0,0)]$, respectively.
We set the same variance for all Gaussians as 0.01.
In the former case, the peaks of the distribution are a subset of the independent data distribution, and its marginal distributions for $x$ and $y$ are the same as the data used for the training of the base model.
On the other hand, the peaks are shifted in the latter case and the marginal distribution differs from the original ones.
For fake pairs to train our discriminator by $\mathcal{L}_{disc}$ (Eq. (9)), we sampled 500 samples in advance from the base model.

#### Setup

We first trained a base diffusion model on scalar value samples from a mixture of five Gaussian distributions.
The base diffusion model outputs a single scalar $x\in\mathbb{R}$, and we duplicated the trained base model to generate a two-dimensional vector $(x,y)\in\mathbb{R}^{2}$ by concatenating outputs from them.
Since each $x$ and $y$ has five peaks as a distribution, $(x,y)$ has 25 peaks, when $x$ and $y$ are generated independently, as shown in the left most column in Fig. 2.
Then, we guided the base models using our proposed method to generate samples that follow the target joint distribution.
We used a neural network consisting of five fully connected layers for the base diffusion and one with three layers for the guidance module. See Section A.3 in appendix for more details about the settings.

#### Results

Figure: Figure 2: Visualization of guidance results on toy datasets. Top row shows samples drawn from ground truth distribution (GT), and bottom row shows generated samples (Gen.).
Refer to caption: x2.png

Figure 2 shows the generated samples from our proposed method.
For both in-domain and out-of-domain settings, our proposed model successfully guides the base models to generate samples from the target distributions.
For an ablation study, we evaluated the effectiveness of each loss by negative log likelihood (NLL) against the target distribution (Table 1).
We confirmed that both losses, $\mathcal{L}_{disc}$ and $\mathcal{L}_{denoise}$, contribute to substantially improving NLL.
These results demonstrate that our proposed method effectively guides base models to generate samples of the target joint distribution.

### 4.2 Experiments with benchmark datasets under in-domain setting

#### Setup

To investigate the applicability of our proposed method to real data, we first applied our proposed method to an in-domain setting.
In this setting, we used the same dataset for training the base models and our proposed model.
We trained our discriminator on top of MM-Diffusion [], which was already trained to jointly generate audio-video data, and guided it to generate further aligned samples.
This setting is similar to adopting guidance to a conditional diffusion model with a condition that the diffusion model can directly handle. The discriminator consists of a stack of two-stream ResBlock layers for each audio and video followed by a linear layer to output a scalar.
The total number of its parameters is 12.7M, whereas MM-Diffusion has 133M parameters.
We used the pre-trained MM-Diffusion released in the official repository and only trained our discriminator while freezing the parameters of MM-Diffusion.
For training, we used the Adam optimizer [] with a learning rate of 1e-3, and the batch size and the number of epochs were set to 16 and 100, respectively.
For generation, we used DPM-Solver [] and set its number of function evaluations (NFE) to 20, which are the standard settings in MM-Diffusion.
See Section A.3 in appendix for more details about the settings.

#### Datasets

We conducted experiments on the Landscape [] and AIST++ [] datasets.
The Landscape dataset contains 928 videos of nine classes of natural scenes such as fire crackling and waterfall burbling.
The AIST++ dataset contains 1020 video clips of street dance with 60 copyright-cleared dancing songs.
Note that, in this experiment, we do not use these class labels as input.
We trained our discriminator to generate 1.6 second audio-video pairs.
Each video comprises 10 frames per second at a $64\times 64$ spatial resolution, and the sampling rate of the audio is 16kHz.
We followed the setting of MM-Diffusion for audio and video preprocessing and training split.

#### Evaluation metrics

We evaluated the generated samples in terms of cross-modal alignment as well as the fidelity of each modality.
To measure the cross-modal alignment, we used the AV-align score [] and the ImageBind score [] computed for audio and video embeddings (IB-AV).
To evaluate the fidelity of each modality, we used FVD [] for video and FAD [] for audio.

#### Results

Table 2 shows the quantitative evaluation on both the Landscape and AIST++ datasets.
For the fidelity of each single modality, our proposed method improves both FVD and FAD.
These results demonstrate that our proposed guidance module captures the training data distribution well and bridges the gap between the distribution of the training data and that of the generated samples.
For the alignment score, both the AV-align and IB-AV scores are also marginally improved, illustrating our guidance module properly guides generated samples to be well-aligned across modalities.

**Table 2: Quantitative evaluation on Landscape and AIST++ datasets for the in-domain adaptation setting. We use MM-Diffusion trained on Landscape or AIST++ dataset at a $64\times 64$ resolution and additionally guide its outputs by our proposed method.**
| Dataset | Method | FVD ↓ | FAD ↓ | AV-align ↑ | IB-AV ↑ |
| --- | --- | --- | --- | --- | --- |
| Landscape | MM-Diffusion | 447 | 5.78 | 0.238 | 0.156 |
| + Ours | 405 | 5.52 | 0.243 | 0.162 |  |
| AIST++ | MM-Diffusion | 513 | 2.31 | 0.457 | 0.0897 |
| + Ours | 450 | 2.17 | 0.462 | 0.0909 |  |

### 4.3 Experiments for out-of-domain generation with benchmark datasets

#### Setup

We also applied our proposed method to an out-of-domain setting.
In this setting, we used a different dataset for training our discriminator compared with the base models.
Specifically, we used AudioLDM [] and AnimateDiff [] for the base models to generate audio and video, each trained on single-modal large-scale datasets.
These models have 1.5B and 651M parameters, respectively.
On top of these independent models, we trained our proposed discriminator using an audio-video dataset to facilitate audio-video alignment of the generated samples from the base models.
Note that these base models can accept text input as an additional condition.
To enable classifier-free guidance [], we also fed conditional text into our discriminator and trained it with a 10% text dropout rate.
We doubled the channel size of each layer of the discriminator to match the larger size of the base models compared to the MM-Diffusion case.
We trained our model to generate videos with a spatial size of $256\times 256$ to match the outputs of the base models.
For generation, we adopted classifier-free guidance with its strength set to 2.5 and 7.5 for audio and video, respectively.
See Section A.3 in appendix for more details.

#### Dataset

We conducted experiments with the Landscape [] and VGGSound [] datasets.
The VGGSound dataset contains nearly 200K video clips of 300 sound classes.
Following Yariv et al. [], we filtered 60K videos that have weak audio-video alignment to enhance the data quality.
For both datasets, we resized videos while maintaining the aspect ratio of the spatial resolution and cropped $256\times 256$ center pixels to create the training dataset.
Regarding text captions for the training dataset, we used the original labels for the VGGSound dataset.
For the Landscape dataset, we additionally created captions by applying InstructBLIP [] to the first frame of each video, because the original labels are too vague to generate plausible samples (see Section A.4 in appendix for more details).

#### Evaluation metrics

We evaluated the generated samples using FAD, FVD, AV-align, and IB-AV as described in Section 4.2. Additionally, to measure the correspondence between a generated sample and its text condition, we computed the ImageBind score for text-video and text-audio pairs (denoted by IB-TV and IB-TA, respectively).

#### Results

Table 3 shows the results of the quantitative evaluation on the Landscape and VGGSound datasets.
For single-modal evaluation, our proposed guidance improves all scores except IB-TV and FVD for the the VGGSound.
This indicates our method properly guides the generation process to match the distribution of generated samples with the training dataset, even for the OOD setting.
Since IB-TV scores of the base models are substantially higher than IB-TA, we conjecture that the captions are suitable for the video generation.
Therefore, the base video model can generate videos well-aligned with the text captions, possibly resulting in slight degradation of FVD and IB-TV but improvement of FAD and IB-TA.
For cross-modal metrics, the IB-AV score is marginally improved while maintaining AV-align score.
Fig. 3 illustrates the effect of our guidance module qualitatively.
Although the generated samples from the base models look reasonable as a single modality, they look unnatural from the perspective of audio-video joint generation because there is no player but the sound is generated.
In contrast, our guidance tends to successfully generate players, and the generated audio is temporally aligned with their motion (see appendix for more generated samples).

**Table 3: Quantitative evaluation under the OOD setting. We used AnimateDiff and AudioLDM for the base models, and the baseline is independent generation by these models (No joint). For ours, we used Eq. (16) to train the joint guidance module.**
|  |  | Video | Audio | Cross-modal |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | Method | FVD ↓ | IB-TV ↑ | FAD ↓ | IB-TA ↑ | AV-align ↑ | IB-AV ↑ |
| Landscape | No joint | 852 | 0.308 | 8.26 | 0.053 | 0.319 | 0.093 |
| Ours | 667 | 0.297 | 7.69 | 0.060 | 0.319 | 0.102 |  |
| VGGSound | No joint | 739 | 0.295 | 15.5 | 0.107 | 0.387 | 0.121 |
| Ours | 754 | 0.291 | 12.1 | 0.116 | 0.390 | 0.127 |  |

Figure: Figure 3: Generated samples from base models and ours trained on VGGSound dataset. To generate samples, we used the captions shown in the left most and the same random seed for both settings.
Refer to caption: x3.png

## 5 Conclusion

In this work, we have proposed a novel training-based but model-agnostic guidance module that enables base models to cooperatively generate well-aligned samples across modalities.
Specifically, given two pre-trained base diffusion models, we train a lightweight joint guidance module that modifies the scores estimated by the two base models to match the joint data distribution.
We show that this guidance can be formulated as the gradient of an optimal discriminator that distinguishes real audio-video pairs from fake pairs generated independently by the base models.
We also propose regularizing this gradient using denoising objective, as in standard diffusion models, which provides a stable gradient of the discriminator.
On several benchmark datasets, we empirically show that our proposed method improves alignment scores as well as single-modal fidelity scores without a huge number of parameters compared with the base models.

#### Limitation

Since our model is built on top of pre-trained models for each modality, the quality of the generated samples strongly depends on the base models.
However, our method benefits from the advancements in each modality’s generative modeling and enables us to integrate new state-of-the-art works into a joint generative model without incurring a huge computational cost.

## Appendix A Appendix / supplemental material

### A.1 Proof of discriminator guidance

Our discriminator is trained by Eq. (10).
This loss function can be decomposed into the loss function at each timestep as:

$$ $\displaystyle\mathcal{L}_{disc,t}^{(\theta)}$ $\displaystyle=\mathbb{E}_{q(x_{t},y_{t})}\left[\log{D_{\theta}(x_{t},y_{t},t)} \right]+\mathbb{E}_{p_{\phi}(x_{t}),p_{\psi}(y_{t})}\left[\log{\left(1-D_{ \theta}(x_{t},y_{t},t)\right)}\right]$ (17) $\displaystyle=\int_{x_{t},y_{t}}q(x_{t},y_{t})\log D_{\theta}(x_{t},y_{t},t)+p _{\phi}(x_{t})p_{\psi}(y_{t})\log{(1-D_{\theta}(x_{t},y_{t},t))}\,\mathrm{d}x_ {t}\mathrm{d}y_{t}.$ (18) $$

Here, $q(x_{t},y_{t})$ is a distribution of real paired data and $p_{\phi}(x_{t})p_{\phi}(y_{t})$ is that of fake one learned by base models.
By Proposition 1 in Goodfellow et al. [], for any fixed generators, the optimal discriminator that minimizes Eq. (18) yields:

$$ $D_{\theta^{*}}(x_{t},y_{t},t)\approx\frac{q(x_{t},y_{t})}{q(x_{t},y_{t})+p_{ \phi}(x_{t})p_{\psi}(y_{t})}.$ (19) $$

Using this optimal discriminator, we can compute the density ratio between a real and a fake data distribution as follows:

$$ $\frac{q(x_{t},y_{t})}{p_{\phi}(x_{t})p_{\psi}(y_{t})}\approx\frac{D_{\theta^{* }}(x_{t},y_{t},t)}{1-D_{\theta^{*}}(x_{t},y_{t},t)}.$ (20) $$

Assuming the base models perfectly align with the marginal distribution of the real data (i.e., $q(x_{t})=p_{\phi}(x_{t})$ and $q(y_{t})=p_{\psi}(y_{t})$), we can compute $\nabla_{x_{t}}\log q(y_{t}|x_{t})$ and $\nabla_{y_{t}}\log q(x_{t}|y_{t})$ as follows:

$$ $\displaystyle\nabla_{x_{t}}\log q(y_{t}|x_{t})$ $\displaystyle=\nabla_{x_{t}}\log\frac{q(x_{t},y_{t})}{q(x_{t})}$ (21) $\displaystyle=\nabla_{x_{t}}\log\frac{q(x_{t},y_{t})}{q(x_{t})q(y_{t})}$ (22) $\displaystyle=\nabla_{x_{t}}\log\frac{q(x_{t},y_{t})}{p_{\phi}(x_{t})p_{\psi}( y_{t})}$ (23) $\displaystyle\approx\nabla_{x_{t}}\log\frac{D_{\theta^{*}}(x_{t},y_{t},t)}{1-D _{\theta^{*}}(x_{t},y_{t},t)},$ (24) $\displaystyle\nabla_{y_{t}}\log q(x_{t}|y_{t})$ $\displaystyle=\nabla_{y_{t}}\log\frac{q(x_{t},y_{t})}{q(y_{t})}$ (25) $\displaystyle=\nabla_{y_{t}}\log\frac{q(x_{t},y_{t})}{q(x_{t})q(y_{t})}$ (26) $\displaystyle=\nabla_{y_{t}}\log\frac{q(x_{t},y_{t})}{p_{\phi}(x_{t})p_{\psi}( y_{t})}$ (27) $\displaystyle\approx\nabla_{y_{t}}\log\frac{D_{\theta^{*}}(x_{t},y_{t},t)}{1-D _{\theta^{*}}(x_{t},y_{t},t)}.$ (28) $$

Therefore, Eqs. (11) and (12) hold.

### A.2 Discriminator guidance for the OOD case

In Section A.1, we assume $q(x_{t})=p_{\phi}(x_{t})$ and $q(y_{t})=p_{\psi}(y_{t})$ to derive Eqs. (11) and (12).
Here, we prove that they also hold even in the case of $q(x_{t})\neq p_{\phi}(x_{t})$ and $q(y_{t})\neq p_{\psi}(y_{t})$.

Equations (7) and (8) can be rewritten by using $p_{\phi}(x_{t})$ and $p_{\psi}(y_{t})$ as follows:

$$ $\displaystyle\nabla_{x_{t}}\log q(x_{t},y_{t})$ $\displaystyle=\nabla_{x_{t}}\log q(x_{t})+\nabla_{x_{t}}\log q(y_{t}|x_{t}),$ (29) $\displaystyle=\nabla_{x_{t}}\log p_{\phi}(x_{t})+\nabla_{x_{t}}\log\frac{q(x_{ t})}{p_{\phi}(x_{t})}+\nabla_{x_{t}}\log\frac{q(x_{t},y_{t})}{q(x_{t})}$ (30) $\displaystyle=\nabla_{x_{t}}\log p_{\phi}(x_{t})+\nabla_{x_{t}}\log\frac{q(x_{ t},y_{t})}{p_{\phi}(x_{t})}$ (31) $\displaystyle\approx\nabla_{x_{t}}\log p_{\phi}(x_{t})+\nabla_{x_{t}}\log\frac {D_{\theta^{*}}(x_{t},y_{t},t)}{1-D_{\theta^{*}}(x_{t},y_{t},t)},$ (32) $\displaystyle\nabla_{y_{t}}\log q(x_{t},y_{t})$ $\displaystyle=\nabla_{y_{t}}\log q(y_{t})+\nabla_{y_{t}}\log q(x_{t}|y_{t})$ (33) $\displaystyle=\nabla_{y_{t}}\log p_{\psi}(y_{t})+\nabla_{y_{t}}\log\frac{q(y_{ t})}{p_{\psi}(y_{t})}+\nabla_{y_{t}}\log\frac{q(x_{t},y_{t})}{q(y_{t})}$ (34) $\displaystyle=\nabla_{y_{t}}\log p_{\psi}(y_{t})+\nabla_{y_{t}}\log\frac{q(x_{ t},y_{t})}{p_{\psi}(y_{t})}$ (35) $\displaystyle\approx\nabla_{y_{t}}\log p_{\psi}(y_{t})+\nabla_{y_{t}}\log\frac {D_{\theta^{*}}(x_{t},y_{t},t)}{1-D_{\theta^{*}}(x_{t},y_{t},t)}.$ (36) $$

Therefore, we can use the optimal discriminator as a joint guidance module for the OOD case, as long as a discriminator is trained to distinguish real paired samples and the fake ones generated by the base models.

### A.3 Details of experiments

In this section, we provide more details of the settings we used in each experiment.

#### Experiments with toy dataset

Table 4 shows the settings for the toy dataset described in Section 4.1.
As described in Section 3, residual noise prediction for the denoising step is computed through the gradient of the discriminator with respect to its input, and its dimensionality is two.
For evaluation and visualization, we sample 4000 samples.

#### Experiments with benchmark datasets

Table 5 shows the settings for real benchmark datasets.
We implemented our discriminator based on the official implementation of MM-Diffusion []. (^2^22[https://github.com/researchmm/MM-Diffusion](https://github.com/researchmm/MM-Diffusion))
A notable difference is that we only used its encoder part, and removed all attention layers from the encoder, which is commonly used in the implementation of diffusion models.
This is because, in our preliminary experiments, we found that using attention layers causes unstable training of the discriminator.
Thus, the architecture of our discriminator is a stack of individual 2-stream ResBlocks followed by a linear layer.
Exploring more advanced architectures for our discriminator may boost the performance further, which we leave for future work.
For the OOD case, we constructed our discriminator on the latent space learned by AudioLDM [] and AnimateDiff [].
To train the discriminator, we sampled 1K samples for the IND settings and 10K samples for the OOD settings in advance.

For evaluation, we generated 2048 samples and computed quantitative metrics.
Specifically, we utilized the evaluation code provided by StyleGAN-V [] for FVD,(^3^33[https://github.com/universome/stylegan-v](https://github.com/universome/stylegan-v)) that provided by AudioLDM [] for FAD,(^4^44[https://github.com/haoheliu/audioldm_eval](https://github.com/haoheliu/audioldm_eval)) the official implementation of AV-align [],(^5^55[https://github.com/guyyariv/TempoTokens](https://github.com/guyyariv/TempoTokens)) and that of IB-score [],(^6^66[https://github.com/facebookresearch/ImageBind](https://github.com/facebookresearch/ImageBind)) respectively.

We utilized 16GB Nvidia V100 $\times$ 4 GPUs for the IND case and 40GB Nvidia A100 $\times$ 8 GPUs for the OOD case.
The training time for IND case was about 1.5 hours with both Landscape and AIST++ datasets, and that for OOD case was about 2 hours with Landscape dataset and 30 hours with VGGSound dataset.
All evaluation was performed by 16GB Nvidia V100 $\times$ 4 GPUs.
It takes about 1 hour for the OOD case, and about 5 hours for the OOD case.

**Table 4: Hyperparameters for the base model and discriminator in the experiments with toy dataset (used in Section 4.1).**
| Model | Base model | Discriminator (IND) | Discriminator (OOD) |
| --- | --- | --- | --- |
| Hyperparameters |  |  |  |
| Architecture | MLP | MLP | MLP |
| Input dim | 1 | 2 | 2 |
| # of layers | 5 | 3 | 3 |
| Channel sizes | 16, 64, 256, 64, 16 | 64, 32, 8 | 64, 32, 8 |
| Output dim | 1 | 1 | 1 |
| Normalization | LayerNorm [] | LayerNorm [] | LayerNorm [] |
| Activation | SiLU [] | SiLU [] | SiLU [] |
| Timestep dim | 256 | 64 | 64 |
| Timestep input type | Adaptive [] | Adaptive [] | Adaptive [] |
| Optimizer | Adam [] | Adam [] | Adam [] |
| Learning rate | 0.001 | 0.001 | 0.001 |
| Total batch size | 512 | 512 | 512 |
| Total # of params | 2.2M | 171K | 171K |
| Diffusion setup |  |  |  |
| Diffusion steps | 500 | 500 | 500 |
| Noise schedule | linear | linear | linear |
| $\beta_{0}$ | 0.0001 | 0.0001 | 0.0001 |
| $\beta_{T}$ | 0.02 | 0.02 | 0.02 |
| Dataset |  |  |  |
| $\mu_{x}$ of GMM | [-3, -1.5, 0, 1.5, 3] | [-3, -1.5, 0, 1.5, 3] | [2.25, -2.25, 2.25, -2.25, 0] |
| $\mu_{y}$ of GMM | - | [1.5, -3, 3, 0, -1.5] | [2.25, -2.25, 2.25, -2.25, 0] |
| $\sigma$ of GMM | 0.1 | 0.1 | 0.1 |
| # of fake samples | - | 500 | 500 |
| # of real samples | 500 | 500 | 500 |

**Table 5: Hyperparameters used in the experiments with benchmark datasets (used in Sections 4.2 and 4.3). † We jointly generate audio and video by MM-Diffusion in the IND case.**
| Model | Discriminator (IND) | Discriminator (OOD) |
| --- | --- | --- |
| Hyperparameters |  |  |
| Audio base model | MM-Diffusion^† [] | AudioLDM [] |
| Video base model | MM-Diffusion^† [] | AnimateDiff [] |
| Architecture | ResBlocks | ResBlocks |
| Audio input dims | 1, 25600 | 8, 50, 16 |
| Video input dims | 3, 16, 64, 64 | 4, 16, 32, 32 |
| Video fps | 10 | 8 |
| Audio fps | 16k | 16k |
| Duration (sec) | 1.6 | 2.0 |
| # of ResBlocks per resolution | 2 | 4 |
| Audio conv type | 1d | 2d |
| Audio conv dilation size | $1,2,4,...,2^{10}$ | 1 |
| Video conv type | 2d + 1d | 2d + 1d |
| Channels | 128 | 256 |
| Channel multiplier | 1, 2, 4 | 1, 2, 4 |
| Audio downsample factor | 4 | (1, 2) |
| Video downsample factor | (1, 2, 2) | (1, 2, 2) |
| Output dim | 1 | 1 |
| Normalization | GroupNorm [] | GroupNorm [] |
| Activation | SiLU [] | SiLU [] |
| Timestep dim | 128 | 256 |
| Timestep input type | Adaptive [] | Adaptive [] |
| Conditional input | - | Text |
| Condition drop rate | - | 0.1 |
| Text embed dim for video | - | 768 |
| Text embed dim for audio | - | 512 |
| Text embed input type | - | Add |
| Optimizer | Adam [] | Adam [] |
| Learning rate | 0.001 | 0.001 |
| Total batch size | 16 | 32 |
| Total training epochs | 100 | Landscape: 100 / VGGSound: 10 |
| Total # of params for base models | 133M | 2.15B |
| Total # of params for discriminator | 13M | 132M |
| Diffusion setup |  |  |
| Diffusion steps | 1000 | 1000 |
| Noise schedule | linear | scaled linear |
| $\beta_{0}$ and $\beta_{T}$ for audio | 0.0001 / 0.02 | 0.0015 / 0.0195 |
| $\beta_{0}$ and $\beta_{T}$ for video | 0.0001 / 0.02 | 0.00085 / 0.012 |
| Inference parameters |  |  |
| Sampler | DPM-Solver [] | DPM-Solver ++[] |
| NFE | 20 | 50 |
| Order | 3 | 2 |
| Guidance scale for audio | - | 2.5 |
| Guidance scale for video | - | 7.5 |

### A.4 Detail of captioning process of Landscape dataset for OOD setting.

We found that the original labels added to the Landscape dataset are too ambiguous for AudioLDM and AnimateDiff to generate reasonable samples.
For instance, the generated results with the caption "wind noise" tend to look like just a noise video and audio, and evaluation for the samples including these unnatural ones is unstable and not desired.
Therefore, we obtained simple captions for each videos using InstructBLIP [], and used them only for the OOD experiment with Landscape dataset.
Specifically, we extracted the first frame of each video and passed it to the InstructBLIP.
As an instruction for the captioning, we used the sentence: "*Write a short sentence for the image starting with ’The image captures a scene with’*.", and generated up to 128 characters for each video.
Note that these captions are generated only from videos without audio, and they are ideal ones for video generation but not for audio.
These captions may not be suitable for audio generation by AudioLDM, as we show in the Table 3 where IB-TA is clearly worse than IB-TV.
These results show that the text condition strongly affects the quality of generated results, and how we design and manipulate the input condition is also important for the joint generation, which we leave for future work.

### A.5 Visualization of the effect by each loss function with toy datasets

Figure 4 visualizes the effect of each loss function.
The top row shows the IND case, and the bottom one shows the OOD case.
At each row, the left most column shows the samples from the target distribution, and the others show generated samples with our guidance modules train by different loss functions.
Note that the NNL at each setting has been shown in Table 1.
In the IND case, both $\mathcal{L}_{disc}$ and $\mathcal{L}_{denoise}$ work similarly, whereas in the OOD case, the guidance trained by only $\mathcal{L}_{disc}$ can roughly concentrate the samples at 4 corners but struggles to force them be in a single peak.
In contrast, combining with $\mathcal{L}_{denoise}$ drastically improves this issue, and using both of them yields the best result.

Figure: Figure 4: Visualization of the generated samples across loss functions used for training our guidance module with toy dataset. Top row shows the IND setting, and bottom one shows the OOD setting.
Refer to caption: x4.png

### A.6 Ablation study about an architecture of the discriminator

We conducted the ablation study for the parameter size of the discriminator as well as the number of training epochs in the IND setting with Landscape dataset.

Table 6 shows the effect of increasing the channel size (denoted by $C$) with the number of ResBlocks per resolution fixed.
As we increase the channel size, the performance substantially increases for both the single-modal fidelity and multi-modal alignment score.
We observed that the multi-modal alignment is slightly degraded at $C=256$, and we conclude that the setting with $C=128$ is the most appropriate one for multi-modal guidance, which requires only less than 10% additional parameters.
Note that our proposed method successfully improves the performance of the base model even when we used only less than 1% additional parameters (the setting with $C=32$).

Table 7 shows the effect of increasing the number of ResBlocks per resolution (denoted by $L$) with the channel size fixed.
We did experiments on the settings with $C=32$ and $C=128$.
For both settings, increasing $L$ mainly improves FAD, while the differences of other metrics are marginal.
We suppose that this may be the effect of the increase of the receptive field.
Since MM-diffusion is trained on the raw data space (pixel for video and waveform for audio), the size of the time axis for audio is extremely large.
Therefore, we used dilated convolutions for the audio branch to capture features at multiple scale, following the implementation of MM-diffusion.
Using more ResBlocks containing dilated convolutions may improve the capacity of audio encoder, which results in the performance improvement of the audio.
This indicates that using an advanced architecture for multi-modal recognition may improve the performance of our proposed guidance module, which we leave for future work.

Figure 5 shows the performance of our guidance module over the number of training epochs.
We used $C=128$ and $L=2$ for the discriminator and trained it longer.
For evaluation, we generated samples with five different random seeds and computed the average and standard deviation of each metrics.
Our guidance module converged around 100 epochs, and we did not observe clear improvement by training further.

**Table 6: Ablation study for the channel size $C$ of the discriminator with the number of ResBlocks per resolution $L$ fixed to 2.**
| Architecture | Metrics |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Settings | # of params | FVD ↓ | FAD ↓ | AV-align ↑ | IB-AV ↑ |
| Base model (MM-diffusion) | 133M | 447 | 5.78 | 0.238 | 0.156 |
| $C=32,L=2$ | 798K | 423 | 5.60 | 0.242 | 0.159 |
| $C=64,L=2$ | 3.2M | 410 | 5.48 | 0.242 | 0.161 |
| $C=128,L=2$ | 12.7M | 405 | 5.52 | 0.243 | 0.162 |
| $C=256,L=2$ | 50.7M | 399 | 5.46 | 0.242 | 0.160 |

**Table 7: Ablation study for the number of ResBlocks per resolution $L$ of the discriminator with the channel size $C$ fixed to 32 and 128.**
| Architecture | Metrics |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Settings | # of params | FVD ↓ | FAD ↓ | AV-align ↑ | IB-AV ↑ |
| Base model (MM-diffusion) | 133M | 447 | 5.78 | 0.238 | 0.156 |
| $C=32,L=1$ | 401K | 421 | 5.65 | 0.240 | 0.159 |
| $C=32,L=2$ | 798K | 423 | 5.60 | 0.242 | 0.159 |
| $C=32,L=4$ | 1.6M | 425 | 5.51 | 0.242 | 0.160 |
| $C=128,L=1$ | 19.0M | 415 | 5.60 | 0.243 | 0.161 |
| $C=128,L=2$ | 12.7M | 405 | 5.52 | 0.243 | 0.162 |
| $C=128,L=4$ | 25.3M | 407 | 5.43 | 0.243 | 0.162 |

Figure: Figure 5: Ablation study for the training epochs. We trained the discriminator with $C=128$ and $L=2$ for 300 training epochs. We generated samples with 5 different random seeds and computed their average and std.
Refer to caption: x5.png

### A.7 Generated audio and video pairs

In this section, we show the generated results from the base models and ours.
Note that, to visually evaluate the effectiveness of our proposed method, we used the same random seed for the base models and ours, with which we can basically expect they provide similar samples.

Figure 6 shows results of unconditional generation from MM-Diffusion and that with our guidance module.
Since MM-Diffusion was already trained by the same dataset (i.e., the IND setting), the generated pairs of audio and video look well-aligned in their semantics.
However, our guidance module improves quality in details at each modality or forces a generated audio and video to be temporally aligned, which is coherent with the results of quantitative evaluation.

Figure 7 shows results of text-conditional generation from base models (AudioLDM and AnimateDiff) and ours (i.e., the OOD setting).
Compared to the IND setting, our guidance module substantially improves the quality of generated samples from that of base models.
In this case, since the base models independently generate audio and video, the generated audio and video are not always well-aligned.
As described in Section 4.3, our guidance tends to successfully generate players, and the generated audio is also temporally aligned with their motion.

Figure: Figure 6: More generated samples from MM-Diffusion and ours trained on Landscape dataset in the IND setting. We used the same random seed for both settings, and the generated results at each row should be similar.
Refer to caption: x6.png

Figure: Figure 7: More generated samples from base models and ours trained on VGGSound dataset in the OOD setting. The captions given for the generation of each sample are shown in the left most column, and we used the same random seed for both settings.
Refer to caption: x7.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. How is the low computational cost demonstrated in this work? Are there specific experiments or data that quantify this claim?
2. While the quality of the generated 2-second samples is good, how does the model perform on longer video samples?
soundness: 3
strengths: 1. This paper tackles an important problem in multi-modal generative modeling by introducing a cost-effective approach to synchronize audio and video generation, offering practical value for applications constrained by limited computational resources.
2. The custom loss function helps to steady the gradient of the discriminator. Extensive evaluations on benchmark datasets highlight enhanced fidelity for individual modalities and better alignment between audio and video, all achieved with minimal additional parameters, underscoring the efficiency of the proposed method.
summary: This paper presents a method for constructing an audio-video generative model with low computational cost by leveraging pre-trained single-modal diffusion models for audio and video. The authors propose a lightweight joint guidance module that aligns audio and video outputs by adjusting scores to approximate the joint distribution, computed through the gradient of an optimal discriminator that distinguishes real and fake audio-video pairs.
weaknesses: 1. Although I understand that the experiments are based on MM-Diffusion, the AIST and Landscape datasets are relatively small. The authors should discuss the generalizability of their method on larger datasets.
2. The paper claims that the method is model-agnostic; however, only one baseline is tested, which makes it difficult to confirm its general applicability.
3. Besides quantitative metrics, a subjective evaluation of the generated samples, would be beneficial to better assess the quality from a perceptual standpoint.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: * As mentioned in the Weaknesses, could the authors compare the training cost for different training setups?
* There are many values omitted in the FVD columns of Table 3 and 4. Are they the same as the corresponding rows below or different? Could the authors either explain or provide new values?
* Can the proposed method generalize to other pairs of modalities beyond video and audio?
* Will the proposed method remain valuable if diffusion models jointly trained on both modalities become widely available? If so, what does the cost-performance trade-off look like?
soundness: 3
strengths: * The proposed method is well motivated and clearly presented, with a focus on achieving joint audio-video generation with low computational cost.
* The proposed method is training-based but model-agnostic, which can be more broadly applicable than model-specific designs.
* Extensive results on multiple datasets demonstrate the effectiveness of the proposed method in improving modality alignment and generation quality of videos and audio.
summary: The submission presents a novel approach to audio-video joint generation with frozen single-modal generative models using a discriminator-guided cooperative diffusion process. The authors leverage existing pre-trained single-modal models for audio and video, aiming to reduce computational costs while enhancing multi-modal alignment. This approach, distinct from traditional model-specific or architecture-dependent techniques, incorporates a lightweight guidance module trained as a discriminator. The model effectively distinguishes between real and fake audio-video pairs, providing stability through a denoising regularization function inspired by standard diffusion methods.
weaknesses: * While the single-modal diffusion models remain frozen, the computation cost for training the joint discriminator, including forward passes through the frozen modules, is not reported. This cost needs to be compared to the total pretraining cost and a reasonable baseline of jointly trained diffusion models on both modalities.
* When text condition is provided to both diffusion models, the captions act as a weak alignment signal. Better experimental designs would be desired to remove the impact of this signal.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Did you adopt other evaluation metrics to evaluate the proposed method on the performance of temporal alignment between generated audio and video? Because it seems that the ImageBind score is just for semantic alignment.
2. Did you try different discriminator structures or larger trainable parameters to see if there will be future performance improvement? 
3. Why the metric scores of MM-Diffusion in Table 2 are different from the ones of the original paper? 
4. It is better to compare the proposed method with "Seeing and Hearing" discussed in the related works by using the same pre-trained base models.
soundness: 3
strengths: 1. It is interesting to solve the joint audio-video generative problems by adjusting the pre-trained modality-specific generators with minimal trainable parameters rather than fully training the joint audio-video generation models.
2. Sufficient mathematic equations and proofs are given to support that the discriminator-based guidance can adjust the estimated scores from audio diffusion and video diffusion to approximately match the score of joint audio-video distribution.
3. The experimental results shown in the table present the proposed method can improve the alignment performance of generated audio and video in both in-domain and out-of-domain settings.
4. The paper is generally well-written.
summary: This paper aims to leverage the pre-trained single-modal diffusion models for generating aligned video and audio. To achieve this objective and involve minimal computing costs, the authors proposed a lightweight joint guidance module to adjust the independent distribution from base models to match the joint distribution over audio and video by designing and training a discriminator. Based on the experimental results on the benchmark datasets, it seems that the proposed method improves the alignment score of generated audio and video while involving reduced trainable parameters.
weaknesses: 1. The description that training a discriminator can adjust the pre-trained modality-specific diffusion models towards a joint audio-video generator is not very clear. For example, how does the discriminator-based guidance achieve both semantic and temporal alignment?  It is better to make it more clear. 
2. The improvement of metric score is limited especially for the OOD setting. In addition, from Table 4, the FVD and IB-TV scores of AudioLDM/AnimateDiff with the proposed guidance module are even worse. 
3. Based on the provided video files, although the performance in the IND is improved by the proposed method, the OOD setting performs poorly in generating high-quality video with the audio track. 
4. It is meaningful to see how different base models affect the performance. Whether more powerful pre-trained video and audio generators further improve the performance when equipped with the same guidance module?

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. The training of generative adversarial models is always easy to explode. Did the authors face similar problems when optimizing the discriminator?

2. Since the additional training module is lightweight, I think the training procedure can be efficient. Can the author provide detailed training consumption required for training the discriminator? How much inference time does it increase?
soundness: 3
strengths: 1. The introduced joint module is lightweight and can be adapted to different single-modal generative models.

2. Authors conducted thorough experiments, including toy datasets, and in-domain and out-domain settings, demonstrating the effectiveness of the proposed method.
summary: This paper presents a novel method for creating an audio-video generative model using pre-trained generative models for audio and video separately, and then guiding these models to work together using a discriminator.

The key innovation is a lightweight joint guidance module (an audio-video discriminator), which adjusts the scores estimated by the base models to match the score of the joint distribution over audio and video. This guidance is computed using the gradient of the discriminator, which distinguishes between real and fake audio-video pairs. The authors also introduce a regulation loss function that stabilizes the gradient of the discriminator.
weaknesses: 1. Statement of datasets and evaluation metrics are duplicated in Sec 4.2 and 4.3.

2. In the section of related work, the author referred to Xing et al. (2024) as their similar work. However, neither quantitative nor qualitative comparisons are conducted with this work.

Meta-review: The paper introduces a novel approach for joint audio-video generation by leveraging classifier guidance to combine independent single-modal models. The proposed method enables synchronization across modalities while using off-the-shelf single-modal models as black boxes, without requiring additional fine-tuning. Empirical results across multiple models demonstrate the effectiveness of the approach.

Strength
* The paper addresses an important problem of joint audio-video generation and offering a novel, cost-effective solution.
* The proposed approach is model-agnostic, enabling its application to various pre-trained generative models without extensive re-training.
* The experimental results demonstrate the effectiveness of the method, particularly in improving cross-modal alignment.

Weakness raised in initial reviews
* The evaluation benchmarks are relatively small, raising concerns about generalizability to larger datasets.
* The claim of model-agnostic applicability requires stronger evidence.
* Lacks of subjective evaluation, where the authors claim it's ongoing but not fully presented during the rebuttal.
* The initial submission lacked details on the computational cost.
* Although the method improves alignment, the absolute performance of generated samples remains limited.
* The impact of the quality of the single-modal base models on the overall performance is not thoroughly analyzed.

This paper addresses an important problem in multi-modal generation and presents a reasonable, cost-effective solution. The empirical results validate the method’s effectiveness in aligning audio and video. While reviewers raised several concerns in their initial reviews, the rebuttal successfully addressed most of these issues. The additional results and clarifications provided during the rebuttal period substantially enhance the completeness of the paper. The paper is recommended for acceptance, provided the authors incorporate the rebuttal details into the final version.

Decision: Accept (Poster)