MM-LDM: Multi-Modal Latent Diffusion Model for Sounding Video Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-22 04:02:01

Abstract: Sounding Video Generation (SVG) is an audio-video joint generation task challenged by high-dimensional signal spaces, distinct data formats, and different patterns of content information.
To address these issues, we introduce a novel multi-modal latent diffusion model (MM-LDM) for the SVG task.
We first unify the representation of audio and video data by converting them into a single or a couple of images. 
Then, we introduce a hierarchical multi-modal autoencoder that constructs a low-level perceptual latent space for each modality and a shared high-level semantic feature space. 
The former space is perceptually equivalent to the raw signal space of each modality but drastically reduces signal dimensions. 
The latter space serves to bridge the information gap between modalities and provides more insightful cross-modal guidance.
Our proposed method achieves new state-of-the-art results with significant quality and efficiency gains.
Specifically, our method achieves a comprehensive improvement on all evaluation metrics and a faster training and sampling speed on Landscape and AIST++ datasets.
Moreover, we explore its performance on open-domain sounding video generation, long sounding video generation, audio continuation, video continuation, and conditional single-modal generation tasks for a comprehensive evaluation, where our MM-LDM demonstrates exciting adaptability and generalization ability.

Main paper: 
 introduction: 
 Sound Video Generation (SVG) is an emerging task in the field of multi-modal generation, which aims to integrate auditory and visual signals for audio-video joint generation. This integrated sounding video closely simulates real-life video formats, providing immersive audiovisual narratives. The potential applications of SVG span multiple fields, including artistic creation, film production, game development, virtual reality, and augmented reality, making it an area worth exploring in depth.Compared to single-modal tasks like audio or video generation, SVG is more challenging since it requires a deep understanding of the complex interactions between sound and visual content to ensure audio-video cross-modal consistency. To be specific, generating a high-quality sounding video requires depicting vivid object appearance and coherent movements, accompanied by realistic audio that aligns with the video content. Two key factors exacerbate this challenge. First, both video and audio are high-dimensional data, making it difficult to ensure the generation realism of both modalities with limited computational resources. Second, cross-modal content consistency is difficult to achieve due to the inherent differences in data representation and content information between video and audio modalities. Specifically, in terms of data representation, videos are 3D visual signals with RGB three color channels, whereas audios are 1D continuous auditory signals with a single amplitude channel. From the perspective of content information, videos capture dense visual dynamics that change over time, while audio captures sound waves made by various visible or invisible sources. These inherent differences significantly increase the difficulty of obtaining cross-modal consistency.Inspired by the success of single-modal diffusion models (Villegas et al., 2023; Huang et al., 2023) , recent works have been proposed for SVG using the diffusion model. MM-Diffusion (Ruan et al., 2023) introduced a multi-modal diffusion model designed to jointly generate audio and video, effectively capturing their joint distribution in their raw signal space. However, due to the high dimensionality of audio and video signals, this method demands substantial computational resources, limiting its ability to produce high-resolution videos. Moreover, MM-Diffusion employs a small cross-attention window with size no more than 8 for efficient calculation, which comes at the cost of sacrificing the cross-modal consistency to some extent. We can observe vivid scene like (a) mountain, (c) diving man, (e) lake, and so on. Matched audios are given like the sound of (b) wood burning, (d) sea wave, (f) raining, and so on. All presented audios can be played in Adobe Acrobat by clicking corresponding wave figures. More playable sounding video samples can be found in https://anonymouss765.github.io/MM-LDM.To address the above challenges, we propose a novel Multi-Modal Latent Diffusion Model (MM-LDM) for SVG. First, we introduce a two-stage latent diffusion method to synthesize realistic audio and video signals with limited computational resources, In the first stage, we employ a multi-modal auto-encoder to map signals of each modality into a modal-specific perceptual latent space. These latent spaces are perceptually equivalent to the original signal spaces but significantly reduce computational complexity. In the second stage, a transformer-based diffusion model is adopted to perform SVG within the latent spaces. Second, to overcome the inherent differences between audio and video modalities, which pose the largest obstacles to achieving cross-modal consistency, we unify their data representations, allowing models to model their generation in a similar manner with shared parameters. Besides, since audio and video convey different messages (i.e., auditory and visual) even with a unified representation, we establish a shared high-level semantic feature space to bridge the gap of content information between the two modalities, thereby obtaining more valuable cross-modal insights. During training, we introduce a classification loss and a contrastive loss to optimize the high-level semantic feature space. Moreover, we can expand the applicability of our learned cross-modal correlations using cross-modal sampling guidance, allowing our generative diffusion model to perform cross-modal conditional generation tasks, including audio-to-video and video-to-audio generation. As shown in Fig. 1 , MM-LDM can synthesize high-resolution (256 2 ) sounding videos with vivid objects, realistic scenes, coherent motions, and aligned audios. We conduct extensive experiments on the Landscape and AIST++ datasets, achieving new state-of-the-art generation performance with significant visual and auditory gains. For example, on the AIST++ dataset with 256 2 spatial resolution, our MM-LDM outperforms MM-Diffusion by 114.6 FVD, 21.2 KVD, and 2.1 FAD. We also reduce substantial computational complexity, achieving a 10x faster sampling speed and allowing a larger sampling batch size.Our contributions: 1) We propose a novel multi-modal latent diffusion model that establishes low-dimensional audio and video latent spaces for SVG, which are perceptually equivalent to the original signal spaces but significantly reduce the computational complexity. 2) We derive a shared high-level semantic feature space from the low-level perceptual latent spaces to provide cross-modal information that is easier to use when decoding audio-video signals. 3) By utilizing cross-modal sampling guidance, we successfully extend our SVG model to conditional generation tasks like audio-to-video and video-to-audio generation. 4) Our method obtains new state-of-the-art results on two benchmarks with significant audio and visual gains. 
related work: 
 Sounding Video Generation SVG is a challenging multi-modal generation task since it requires: 1) synthesizing realistic high-dimensional video and audio signals with limited computational resources, and 2) bridging the representation and information gap between video and audio modalities to obtain content consistency. Several research works have been proposed to explore the challenging task of SVG. Based on generative adversarial networks, HMMD (Kurmi et al., 2021) Modules with gray border consist of our multi-modal autoencoder, which compresses raw audiovideo signals into perceptual latent spaces. The module with orange border is our transformer-based diffusion model that performs SVG in the latent space. The green rectangle depicts the modification of inputs for unconditional audio-video generation (i.e. SVG), audio-to-video generation, and videoto-audio generation, respectively. tiple discriminators to guide the generator in producing sounding videos with aligned audio-video content. Based on sequential generative models, UVG (Liu et al., 2023b ) introduces a multi-modal tokenizer to encode different modal signals into discrete tokens, and employs a Transformer-based generator for SVG. Although these works have tackled the SVG task to some extent, their performances are far from expectations. Recently, motivated by the remarkable success of diffusion models (Ho et al., 2020; Song et al., 2021) , MM-Diffusion (Ruan et al., 2023) is introduced to address SVG in the signal space. To ensure cross-modal consistency, MM-Diffusion introduces a random-shift method for efficient attention. However, this method suffers from a huge computational burden and uses a limited attention window size (typically no more than 8), resulting in suboptimal cross-modal consistency. In this paper, we propose MM-LDM that establishes a distinct low-level latent space for each modality and a shared high-level feature space for both modalities. The former spaces closely approximate the signal spaces while significantly reducing computational complexity. Simultaneously, the latter space provides valuable cross-modal information to obtain better cross-modal consistency.Latent Diffusion Model Given that raw signal spaces for image, audio, and video modalities are of high dimensions, extensive efforts have been devoted to modeling their generation using latent diffusion models (Rombach et al., 2022; Liu et al., 2023a; Blattmann et al., 2023; Yu et al., 2023; He et al., 2022) . For the image modality, LDM (Rombach et al., 2022) is devised to construct a perceptual latent space for images. This approach employs a KL-VAE to encode images into image latents and utilizes a latent diffusion model for text-to-image generation within the latent space. For the audio modality, AudioLDM (Liu et al., 2023a) is introduced to facilitate text-to-audio generation in a 1D latent space. In particular, it utilizes a large text-audio pretrained model CLAP (Wu et al., 2023) for extracting text and audio latent features. For the video modality, VideoLDM (Blattmann et al., 2023) is proposed to extend the LDM (Rombach et al., 2022) to high-resolution video generation. It introduces a temporal dimension into the LDM, and only optimizes these temporal layers while maintaining fixed, pretrained spatial layers. Despite previous latent diffusion models have demonstrated excellent performance on single-modal generation tasks, multi-modal generation tasks raise higher requirements, including unifying data representation across different modalities and considering cross-modal correlations. Our proposed method introduces a multi-modal latent diffusion model that synthesizes high-quality sounding videos with consistent audio-video content. It achieves new SoTA results on multiple benchmarks and significantly enhances the computational efficiency. 
method: 
 In this section, we present our multi-modal latent diffusion model (MM-LDM) in detail. This approach consists of two main components: a multi-modal autoencoder designed for the compression of video and audio signals, and a multi-modal latent diffusion model for modeling SVG within latent spaces. Cross-modal sampling guidance is used to extend our method to cross-modal conditional generation tasks, including audio-to-video generation and video-to-audio generation tasks. A comprehensive overview of MM-LDM is illustrated in Fig. 2 3.1 VIDEO AND AUDIO COMPRESSION VIA A MULTI-MODAL AUTOENCODER The detailed structure of our multi-modal autoencoder is presented in Fig. 3 , which is composed of two modal-specific encoders, two signal decoders with shared parameters, two projector functions mapping from each perceptual latent space to the shared semantic feature space, and two heads for classification and contrastive learning, respectively.Unifying Representation of Video and Audio Signals We employ the raw video signals v and transformed audio images a to be our inputs. Video v ∈ R F ×3×H×W can be viewed as a sequence of 2D images (i.e. video frames), where F , 3, H, and W are frame number, channels, height, and width, respectively. Given that raw audio signals are 1D-long continuous data, we transform raw audio signals into 2D audio images to unify the representation of audio and video inputs. In particular, given raw audio signals, we first obtain its Mel Spectrogram with values normalized, which is denoted as a raw ∈ R D×T , where D represents the number of audio channels and T is the temporal dimension. Then, we treat a raw as a grayscale image and convert it into an RGB image using the PIL Python toolkit. Finally, we resize the Mel Spectrogram image to the same spatial resolution as the video input, obtaining an audio image a ∈ R 3×H×W .Transforming raw audio signals into audio images is important for two key reasons. Firstly, as discussed in prior work (Huang et al., 2023) , dealing with long continuous raw audio signals involves expansive computation complexity. Second, by unifying audio and video inputs into a shared image representation, we can leverage pre-trained image diffusion models to be our signal decoders, which not only reduces training consumption but also enhances decoding performance.Encoding Inputs into 2D Perceptual Latent Space Given a pair of audio and video inputs, we employ a pretrained KL-VAE (Rombach et al., 2022) to downsample the video frames and audio image by a factor of f . Then, as depicted in Fig. 3 (a), we introduce an audio encoder to compress the audio image into an audio perceptual latent z a ∈ R C×Ha×Wa , which further downsamples the audio image by a factor of f a , where C is the number of channels, H a and W a denotes H f ×fa and W f ×fa , respectively. Similarly, the video encoder compresses video frames intoz v ∈ R C×Hv×Wv .The audio encoder is constructed in a similar way to the encoder part of U-Net, which consists of residual blocks and spatial attentions. Since video signals are temporally redundant (Sun et al., 2023b) , we uniformly select keyframes from the input video to feed our video encoder. The structure of the video encoder differs from the audio encoder in two key aspects. Firstly, it adds a temporal attention layer after each spatial attention layer to capture temporal relationships. Secondly, an additional temporal pooling layer is employed before the final layer to integrate temporal information.Mapping Perceptual Latents to High-level Semantic Space When decoding video and audio latents using a shared decoder, cross-model information is required to maintain consistency between these two modalities. In our experiments, we observed a significant performance drop when we directly used one perceptual latent as condition input to provide cross-modal information when decoding another perceptual latent. This performance drop reveals that the signal decoder is hard to extract useful cross-modal information from perceptual latents. This can be attributed to two key factors. Firstly, perceptual latents are dense representation of low-level information, thus presenting challenges for the decoder to comprehend. Secondly, video frames and audio images exhibit distinct patterns -the former encompassing real-world object images while the latter are typical images in physics, thus the gap between their perceptual latents is too large for the decoder to bridge. To feed the video decoder with useful cross-modal information, specialized modules are required to extract high-level information from perceptual latents and narrow the gap between video and audio features.To this end, as depicted in Fig. 3 (a), we introduce an audio projector and a video projector that establish a shared high-level semantic space based on the low-level perceptual latents. In particular, the audio and video projectors extract semantic audio and video features s a and s v from their perceptual latents. To ensure the extracted features are of high-level semantic information, we employ a classification head f cl that takes a pair of audio and video features as inputs and predicts its class label, which is optimized using a classification cross-entropy loss. A contrastive loss is employed with a specified contrastive head to bridge the gap between video and audio features. The contrastive head f co maps s a and s v to 1D features respectively and calculates their contrastive loss with matched pairs of audio-video features being positive samples and all unmatched pairs of audio-video features as negative samples. Following (Radford et al., 2021) , we define the contrastive loss as follows:EQUATIONwhere sim( * ) calculates the dot product of input features, B and τ denote the batch size and a learnable parameter, respectively.Signal Decoding As illustrated in Fig. 3 (b), when reconstructing video signals, the signal decoder takes multiple factors into account, including the video perceptual latent z v , frame index i, audio semantic feature s a , learnable modality embedding, and learnable class embedding. The video perceptual latent provides spatial information for the i-th video frame using residual blocks, and content information through pooling. The audio semantic features are sequentialized and concatenated with the learnable modality embedding as well as the class embedding. Then they are fed to crossattention layers to provide rich conditional information. When dealing with audio reconstruction, the signal decoder employs similar inputs, except for the frame index. More detailed explanations are presented in the appendix. To reduce training time and enhance the quality of reconstruction, we initialize our signal decoder with parameters of a pretrained image diffusion model (Rombach et al., 2022) and open all parameters during training.Training Targets Following (Ho et al., 2020) , we utilize the ϵ-prediction to optimize our signal decoder, which involves the noise mean square error loss L M SE . Since the audio and video perceptual latents will be modeled for generation using MM-LDM, which will be specified in Sec. 3.2, we incorporate additional KL losses L a KL and L v KL to punish the distributions of audio and video latents towards an isotropic Gaussian distribution, which are similar to (Rombach et al., 2022) . Previous works have proven the effectiveness of adversarial loss in training single-modal autoencoders (Esser et al., 2021; Sun et al., 2023a) . Here, we introduce a novel adversarial loss to improve the quality of reconstructed multi-modal signals in terms of both single-modal realism and multi-modal consistency. Following (Sun et al., 2023a) , we first obtain a pair of decoded video frames ⟨v i , vj ⟩ with i < j and corresponding audio image ā. Then, for the optimization of the discriminator, we select ⟨a, v i , v j ⟩ as the real sample and ⟨ā, vi , vj ⟩, ⟨ā, vi , v j ⟩, ⟨ā, v i , vj ⟩, and ⟨ā, vj , vi ⟩ to be the fake samples. ⟨ā, vi , vj ⟩ is viewed as the real sample for our autoencoder. Our adversarial loss can be formulated as L D GAN for the discriminator and L AE GAN for our autoencoder:L D GAN =log(1 -D(⟨a, v i , v j ⟩)) + logD(⟨ā, vi , vj ⟩) + logD(⟨ā, vi , v j ⟩) + logD(⟨ā, v i , vj ⟩) + logD(⟨ā, vj , vi ⟩) L AE GAN =log(1 -D(⟨ā, vi , vj ⟩)) (2)Our discriminator is constructed by several spatio-temporal modules that consist of residual blocks and cross-modal full attentions. Our final training loss for the multi-modal autoencoder becomes:L AE = L M SE + λ cl L cl + λ co L co + λ kl (L a KL + L v KL ) + λ gan L AE GAN (3)where λ cl , λ co , λ kl and λ gan are predefined hyper-parameters. L cl and L co are the classification and contrastive losses, respectively.As illustrated in Fig. 2 , our approach independently corrupts audio and video latents during the forward diffusion process, whereas in the reverse denoising diffusion process, we employ a unified model that jointly predicts noise for both modalities. In particular, during forward diffusion, we corrupt audio and video latents, which are denoted as z 0 a and z 0 v (i.e. z a and z v ), by T steps using a shared transition kernel. For simplicity, we use z 0 to represent both z 0 a and z 0 v in the subsequent section. Following prior works (Ho et al., 2020; Ruan et al., 2023) , we define the transition probabilities as follows:EQUATIONwhere {β t ∈ (0, 1)} T t=1 is a set of shared hyper-parameters, α t = 1 -β t , and ᾱt = t i=1 α i . Utilizing Eq. ( 4), we obtain corrupted latents z t at time step t as follows:EQUATIONwhere n t ∼ N (0, I) represents noise features n t a and n t v for z t a and z t v respectively. the reverse diffusion processes of audio and video latents q(z t-1 |z t , z 0 ) have theoretically traceable distributions: To capture correlations between audio and video modalities and ensure content consistency, we introduce a unified denoising diffusion model θ. This model takes both corrupted audio and video latents (z t a , z t v ) as input and jointly predicts their noise features (n t a , n t v ). The reverse diffusion process of corrupted audio and video latents is formulated as:EQUATIONDuring training, we minimize the mean square error between the predicted and original noise features of matched audio-video pairs, known as ϵ-prediction in the literature (Kingma et al., 2021) :EQUATIONHere, ña θ (z t a , z t v , t) and ñv θ (z t a , z t v , t) are the predicted audio and video noise features, respectively. Given that our audio and video latents z a and z v possess relatively small spatial resolution, we employ a Transformer-based diffusion model known as DiT (Peebles & Xie, 2022) as our backbone model. During training, we sequentialize audio and video latents and independently add positional embeddings (Vaswani et al., 2017) to each latent. Then, two learnable token embeddings, [EOS a ] and [EOS v ], are defined and inserted before the audio and video latents, respectively. Finally, audio and video latents are concatenated and fed to DiT for multi-modal generation.Initially designed for class-conditioned image generation (Ho & Salimans, 2022) , the classifier-free guidance has demonstrated its effectiveness in the more difficult text-to-image generation task (Saharia et al., 2022; Ramesh et al., 2022) . Inspired by this success, we introduce the cross-modal sampling guidance that targets both audio-to-video and video-to-audio generation tasks. Our approach involves training the single MM-LDM to simultaneously learn three distributions: an unconditional distribution denoted as ñθ (z t a , z t v , t) and two conditional distributions represented as ñθ (z t v , t; z a ) and ñθ (z t a , t; z v ), corresponding to the SVG, audio-to-video and video-to-audio generation tasks respectively. To accomplish this, we incorporate a pair of null audio and video latents, defined as (0 a , 0 v ) with 0 a = 0 and 0 v = 0. Then, the unconditional distribution ñθ (z t a , z t v , t) can be reformulated to be ñθ (z t a , z t v , t; 0 a , 0 v ). The conditional distribution ñθ (z t v , t; z a ) can be reformed as ñθ (z t a , z t v , t; z a , 0 v ), where z t a can be obtained directly given z a ant t according to Eq. ( 5). Similarly, ñθ (z t a , t; z v ) is reformulated as ñθ (z t a , z t v , t; 0 a , z v ). As depicted in Fig. 2 , the conditional inputs are added to the input latents after zero convolution layers (which are ignored in the figure for conciseness) to provide conditional information. We randomly select 5% training samples for each conditional generation task. Finally, taking audio-to-video generation as an example, we perform sampling utilizing the following linear combination of the conditional and unconditional noise predictions, defined as follows:EQUATIONwhere ϕ serves as a hyper-parameter that controls the intensity of the conditioning. Evaluation Metrics For video evaluation, we follow previous settings (Ge et al., 2022; Yu et al., 2022; Ruan et al., 2023) that employ the Fréchet Video Distance (FVD) and Kernel Video Distance (KVD) metrics for video evaluation and Fréchet Audio Distance (FAD) for audio evaluation. Our implementation for these evaluations is based on the resources provided by (Ruan et al., 2023) . Our MM-LDM synthesize all videos at a 256 2 resolution. We resize the synthesized videos when testing the metrics in the 64 2 resolution.Implementation Details When training our multi-modal autoencoder, we utilize pretrained KL-VAE (Rombach et al., 2022) with the downsample factor being 8. Both video frames and audio images are resized to a 256 2 resolution, and video clips have a fixed length of 16 frames (F = 16).The audio and video encoders use downsample factors of f a = 4 and f v = 2, yielding latents of spatial size 8 2 and 16 2 , respectively. The number of latent channels is 16 for both modalities. The loss weights λ cl , λ co , and λ gan are 1e-1, 1e-1, and 1e-1, respectively. The loss weight λ kl is set as 1e-9 for Landscape and 1e-8 for AIST++. Further details can be found in the appendix. 
quantitative comparison: 
 Efficiency We quantitatively compared the efficiency of our MM-LDM and MM-Diffusion and presented the results in We conduct ablation studies on our multi-modal autoencoder and report the results in Table . 3. Our base autoencoder independently decodes signals for each modality based on the respective spatial information from perceptual latents. To reduce computational complexity, we share the diffusionbased signal decoder for both modalities while using a learnable embedding to prompt each modality, obtaining improved performance. For a more comprehensive content representation, we apply average pooling to each perceptual latent, adding the latent feature to the timestep and further enhancing the model performance. By incorporating the classification and contrastive losses, we leverage prior knowledge of class labels and extract high-level cross-modal information, which significantly boosts model performance. Since the KL-VAE was originally trained on natural images and is unfamiliar with physical images like audio images, we finetune its decoder on each dataset for better performance. Finally, after training with the adversarial loss, our autoencoder attains its best reconstruction performance, achieving 53.9 rFVD, 2.4 rKVD, and 8.9 rFAD. 
conclusion: 
 This paper introduces MM-LDM, a novel diffusion-based method for SVG. Our method reduces the computational complexity of SVG by constructing low-dimensional latent spaces that faithfully capture the perceptual characteristics of the original signal spaces. To obtain cross-modal consistency, we unify the audio-video data representation and bridge the gap of content information between the two modalities. Extensive experiments demonstrate that our method excels at generating sounding videos with enhanced single-modal realism and cross-modal consistency, achieving new state-ofthe-art results on multiple benchmarks with better training and inference efficiency. 
 
 Appendix: Our codes will be released if accepted. 


Decision: The AC goes through the paper, rebuttal and review comments. This paper got 4 reviews, including 2 weak accept, 1 borderline accept and 1 borderline reject. All reviewers acknowledge the motivation. However,  there are still several problems, such as the advantage of the joint VAE. I hope the authors can solve these problems in the camera-ready.

Reviews: review: paper_summary: paper_summary: The author explores the challenge of sounding video generating (SVG) in this paper and introduces a multi-modality video generation model called MM-LDM, operating within a latent space framework. The idea of incorporating the latent diffusion model into SVG task is interesting and the overall result is promising.
strength_weakness: strength_weakness: 1. The task of SVG is interesting and the overall idea of modeling audio and video in latent space is promising.
2. The experimental results are promising which demonstrate the effectiveness of the proposed method.
limitations: limitations: 1. Is there any performance analysis of the proposed multi-modal VAE and two separate VAEs?
2. Is it possible for the proposed method to generate longer video?
paper_summary: paper_summary: This paper presents MM-LDM, the first latent diffusion model for sound video generation. It first introduces a multi-modal autoencoder where the audio and video share the same semantic latent space to reduce the signal dimension. Afterward, a DiT-like model acts as a denoising network in the latent space. The quantitative evaluations show that the method outperform previous baseline and can support various applications.
strength_weakness: strength_weakness: 1. The supplementary material is aboundant, which helps to understand the paper;
2. The paper is well-constructed and technically clear.
limitations: limitations: 1. What is the architecture/framework of the autoencoder? Is it another diffusion model instead of the KL-VAE in the vanilla LDM? If so, how to initialize its parameters with the pre-trained VAE in LDM (L868)?
2. If the AE's decoder is another diffusion model, I think the proposed pipeline is closer to a cascade diffusion model instead of a latent diffusion model since the decoder also incorporates strong generative priors rather than performing low-level compression.
3. Since the autoencoder is the major contribution, it should have more analysis and ablation. For instance, is it possible to use separate KL-VAE encoders for each modality?
paper_summary: paper_summary: This paper proposes a novel multi-modal latent diffusion model (MM-LDM) for Sounding Video Generation (SVG). It employs an innovative hierarchical multi-modal autoencoder to transition from video and audio to a latent space, conserving computational resources while enhancing generation quality. However, the paper’s main contribution, which is the transformation of data to latent space, is somewhat limited as it has been previously validated by other works like LDM. Additionally, The superiority of a single multimodal VAE over two separate VAEs is unclear.
strength_weakness: strength_weakness: 1. The overall paper is easy to follow.
2. Some motivations are easy to understand.
3. The figures displayed in the paper are helpful for readers to understand the proposed method.
limitations: limitations: 1. Transforming data to latent space to improve performance has been validated by other work, such as LDM, so this paper's core contribution is somewhat limited.
2. The video in the attachment has no sound on my Windows 11 laptop, and it plays like a noise on my phone. I'm not sure if this is due to the quality of the video itself or if it requires a particular player for my phone or PC. It's hard to ascertain their alignment due to the noise.
3. It is not clear that one multimodal VAEs are superior to two separate VAEs. I question the feasibility of learning aligned multimodal features from such a limited dataset.

I am willing to raise my score if my concerns are addressed well.
paper_summary: paper_summary: The authors propose a novel multi-modal latent diffusion model (MM-LDM) to tackle the challenging task Sounding Video Generation (SVG). The proposed method first converts the audio and visual data into a unified representation, then applies a hierarchical multimodal autoencoder to construct a low-level unimodal latent space and a high-level semantic multimodal latent space. The proposed method is capable of unconditional audio-video joint generation as well as cross-modal conditional generation. Experiments on the Landscape and AIST++ datasets verify the effectiveness of the proposed method.
strength_weakness: strength_weakness: [+] Overall, this paper is a good extension of the pivotal work MM-Diffusion. The idea of leveraging a latent space for diffusion is reasonable and the hierarchical multimodal autoencoder and the unified multimodal representation warrant a stable and scalable audio-video joint generation.  
[+] The task of audio-video joint generation (a.k.a. SVG) is quite challenging. While the proposed method achieves satisfactory generation results, the authors also extend its capability to cross-modal conditional generation tasks, showing its good generalization. Numerical results on two benchmarks also verify the effectiveness.  
[+] The writing is good and easy to follow.
limitations: limitations: [-] Adopting a latent diffusion module for generation tasks is not that novel.  
[-] From my point of view, I think one primary advantage of such a unified representation/token setting is the scalable capability. What will the method perform when training on a larger dataset or a larger model? I think the investigation of the scaling law of such unified generation methods would be interesting and valuable for the community.  
[-] Can the model be used for zero-shot conditional generation? How does the model perform for videos longer than 1 second? Qualitative results or at least natural language descriptions could be provided if there is no available evaluation benchmark.   

Overall I think this paper is good and is worthy to be published in the conference, yet addressing these limitations could make the paper more robust.