MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation

Authors: Akio Hayakawa, Masato Ishii, Takashi Shibuya, Yuki Mitsufuji
Primary area: generative models
Keywords: Diffusion models, Multi-modal data, Audio-visual generative models

Abstract:
This study aims to construct an audio-video generative model with minimal computational cost by leveraging pre-trained single-modal generative models for audio and video.
To achieve this, we propose a novel method that guides single-modal models to cooperatively generate well-aligned samples across modalities. 
Specifically, given two pre-trained base diffusion models, we train a lightweight joint guidance module to adjust scores separately estimated by the base models to match the score of joint distribution over audio and video. 
We show that this guidance can be computed using the gradient of the optimal discriminator, which distinguishes real audio-video pairs from fake ones independently generated by the base models. 
Based on this analysis, we construct a joint guidance module by training this discriminator.
Additionally, we adopt a loss function to stabilize the discriminator's gradient and make it work as a noise estimator, as in standard diffusion models. 
Empirical evaluations on several benchmark datasets demonstrate that our method improves both single-modal fidelity and multimodal alignment with relatively few parameters.
The code is available at: [https://github.com/SonyResearch/MMDisCo](https://github.com/SonyResearch/MMDisCo).

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730471360283

thread_type:
official_review

<thread_object>
creator: reviewer_u5ht

<content>
summary:
This paper presents a novel method for creating an audio-video generative model using pre-trained generative models for audio and video separately, and then guiding these models to work together using a discriminator.

The key innovation is a lightweight joint guidance module (an audio-video discriminator), which adjusts the scores estimated by the base models to match the score of the joint distribution over audio and video. This guidance is computed using the gradient of the discriminator, which distinguishes between real and fake audio-video pairs. The authors also introduce a regulation loss function that stabilizes the gradient of the discriminator.

strengths:
1. The introduced joint module is lightweight and can be adapted to different single-modal generative models.

2. Authors conducted thorough experiments, including toy datasets, and in-domain and out-domain settings, demonstrating the effectiveness of the proposed method.

weaknesses:
1. Statement of datasets and evaluation metrics are duplicated in Sec 4.2 and 4.3.

2. In the section of related work, the author referred to Xing et al. (2024) as their similar work. However, neither quantitative nor qualitative comparisons are conducted with this work.

questions:
1. The training of generative adversarial models is always easy to explode. Did the authors face similar problems when optimizing the discriminator?

2. Since the additional training module is lightweight, I think the training procedure can be efficient. Can the author provide detailed training consumption required for training the discriminator? How much inference time does it increase?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730581206839

thread_type:
official_review

<thread_object>
creator: reviewer_kk8x

<content>
summary:
This paper aims to leverage the pre-trained single-modal diffusion models for generating aligned video and audio. To achieve this objective and involve minimal computing costs, the authors proposed a lightweight joint guidance module to adjust the independent distribution from base models to match the joint distribution over audio and video by designing and training a discriminator. Based on the experimental results on the benchmark datasets, it seems that the proposed method improves the alignment score of generated audio and video while involving reduced trainable parameters.

strengths:
1. It is interesting to solve the joint audio-video generative problems by adjusting the pre-trained modality-specific generators with minimal trainable parameters rather than fully training the joint audio-video generation models.
2. Sufficient mathematic equations and proofs are given to support that the discriminator-based guidance can adjust the estimated scores from audio diffusion and video diffusion to approximately match the score of joint audio-video distribution.
3. The experimental results shown in the table present the proposed method can improve the alignment performance of generated audio and video in both in-domain and out-of-domain settings.
4. The paper is generally well-written.

weaknesses:
1. The description that training a discriminator can adjust the pre-trained modality-specific diffusion models towards a joint audio-video generator is not very clear. For example, how does the discriminator-based guidance achieve both semantic and temporal alignment?  It is better to make it more clear. 
2. The improvement of metric score is limited especially for the OOD setting. In addition, from Table 4, the FVD and IB-TV scores of AudioLDM/AnimateDiff with the proposed guidance module are even worse. 
3. Based on the provided video files, although the performance in the IND is improved by the proposed method, the OOD setting performs poorly in generating high-quality video with the audio track. 
4. It is meaningful to see how different base models affect the performance. Whether more powerful pre-trained video and audio generators further improve the performance when equipped with the same guidance module?

questions:
1. Did you adopt other evaluation metrics to evaluate the proposed method on the performance of temporal alignment between generated audio and video? Because it seems that the ImageBind score is just for semantic alignment.
2. Did you try different discriminator structures or larger trainable parameters to see if there will be future performance improvement? 
3. Why the metric scores of MM-Diffusion in Table 2 are different from the ones of the original paper? 
4. It is better to compare the proposed method with "Seeing and Hearing" discussed in the related works by using the same pre-trained base models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708487135

thread_type:
official_review

<thread_object>
creator: reviewer_f4ji

<content>
summary:
The submission presents a novel approach to audio-video joint generation with frozen single-modal generative models using a discriminator-guided cooperative diffusion process. The authors leverage existing pre-trained single-modal models for audio and video, aiming to reduce computational costs while enhancing multi-modal alignment. This approach, distinct from traditional model-specific or architecture-dependent techniques, incorporates a lightweight guidance module trained as a discriminator. The model effectively distinguishes between real and fake audio-video pairs, providing stability through a denoising regularization function inspired by standard diffusion methods.

strengths:
* The proposed method is well motivated and clearly presented, with a focus on achieving joint audio-video generation with low computational cost.
* The proposed method is training-based but model-agnostic, which can be more broadly applicable than model-specific designs.
* Extensive results on multiple datasets demonstrate the effectiveness of the proposed method in improving modality alignment and generation quality of videos and audio.

weaknesses:
* While the single-modal diffusion models remain frozen, the computation cost for training the joint discriminator, including forward passes through the frozen modules, is not reported. This cost needs to be compared to the total pretraining cost and a reasonable baseline of jointly trained diffusion models on both modalities.
* When text condition is provided to both diffusion models, the captions act as a weak alignment signal. Better experimental designs would be desired to remove the impact of this signal.

questions:
* As mentioned in the Weaknesses, could the authors compare the training cost for different training setups?
* There are many values omitted in the FVD columns of Table 3 and 4. Are they the same as the corresponding rows below or different? Could the authors either explain or provide new values?
* Can the proposed method generalize to other pairs of modalities beyond video and audio?
* Will the proposed method remain valuable if diffusion models jointly trained on both modalities become widely available? If so, what does the cost-performance trade-off look like?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730852873041

thread_type:
official_review

<thread_object>
creator: reviewer_pfzf

<content>
summary:
This paper presents a method for constructing an audio-video generative model with low computational cost by leveraging pre-trained single-modal diffusion models for audio and video. The authors propose a lightweight joint guidance module that aligns audio and video outputs by adjusting scores to approximate the joint distribution, computed through the gradient of an optimal discriminator that distinguishes real and fake audio-video pairs.

strengths:
1. This paper tackles an important problem in multi-modal generative modeling by introducing a cost-effective approach to synchronize audio and video generation, offering practical value for applications constrained by limited computational resources.
2. The custom loss function helps to steady the gradient of the discriminator. Extensive evaluations on benchmark datasets highlight enhanced fidelity for individual modalities and better alignment between audio and video, all achieved with minimal additional parameters, underscoring the efficiency of the proposed method.

weaknesses:
1. Although I understand that the experiments are based on MM-Diffusion, the AIST and Landscape datasets are relatively small. The authors should discuss the generalizability of their method on larger datasets.
2. The paper claims that the method is model-agnostic; however, only one baseline is tested, which makes it difficult to confirm its general applicability.
3. Besides quantitative metrics, a subjective evaluation of the generated samples, would be beneficial to better assess the quality from a perceptual standpoint.

questions:
1. How is the low computational cost demonstrated in this work? Are there specific experiments or data that quantify this claim?
2. While the quality of the generated 2-second samples is good, how does the model perform on longer video samples?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732087686718

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all reviewers

comment:
First, we would like to express our gratitude to all the reviewers for taking their time and providing invaluable feedback on our manuscript.
We have carefully considered all the comments and questions.
Our detailed responses are posted in the reply to each review.

In response to the reviewers' comments, we have updated our manuscript.
We summarize our modifications as follows:

- We added Table 10 to show the performance improvement by using a more sophisticated network architecture (e.g., transformer-based architecture) for the discriminator.
- We added appendix A.8 to explain the details of network architecture and the experimental settings for Table 10 and discuss the results.
- We added Table 11 to show that the computational cost for training our proposed model is significantly lower than the baselines.
- We added Table 12 to clarify the computational overhead in inference introduced by our proposed method.
- We added appendix A.9 to explain the details of the experimental settings for Tables 11 and 12 and discuss these results.
- We added appendix A.10 and Table 13 to compare the temporal cross-modal alignment of our method with baselines.

Please see the corresponding part in the updated manuscript, as well as our responses to each review.
All the modifications are colored in red in the updated manuscript.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a novel approach for joint audio-video generation by leveraging classifier guidance to combine independent single-modal models. The proposed method enables synchronization across modalities while using off-the-shelf single-modal models as black boxes, without requiring additional fine-tuning. Empirical results across multiple models demonstrate the effectiveness of the approach.

Strength
* The paper addresses an important problem of joint audio-video generation and offering a novel, cost-effective solution.
* The proposed approach is model-agnostic, enabling its application to various pre-trained generative models without extensive re-training.
* The experimental results demonstrate the effectiveness of the method, particularly in improving cross-modal alignment.

Weakness raised in initial reviews
* The evaluation benchmarks are relatively small, raising concerns about generalizability to larger datasets.
* The claim of model-agnostic applicability requires stronger evidence.
* Lacks of subjective evaluation, where the authors claim it's ongoing but not fully presented during the rebuttal.
* The initial submission lacked details on the computational cost.
* Although the method improves alignment, the absolute performance of generated samples remains limited.
* The impact of the quality of the single-modal base models on the overall performance is not thoroughly analyzed.

This paper addresses an important problem in multi-modal generation and presents a reasonable, cost-effective solution. The empirical results validate the method’s effectiveness in aligning audio and video. While reviewers raised several concerns in their initial reviews, the rebuttal successfully addressed most of these issues. The additional results and clarifications provided during the rebuttal period substantially enhance the completeness of the paper. The paper is recommended for acceptance, provided the authors incorporate the rebuttal details into the final version.