A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation

Authors: gwanghyun  kim
˚a  lonso martinez
yu-chuan  su
brendan  jou
josé  lezama
agrim  gupta
˚l ijun  yu
  jiang
  ˚a ren jansen
jacob  walker
krishna  somandepalli
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 02:41:47

Abstract: Training diffusion models for audiovisual sequences allows for a range of generation tasks by learning conditional distributions of various input-output combinations of the two modalities. Nevertheless, this strategy often requires training a separate model for each task which is expensive. Here, we propose a novel training approach to effectively learn arbitrary conditional distributions in the audiovisual space. Our key contribution lies in how we parameterize the diffusion timestep in the forward diffusion process. Instead of the standard fixed diffusion timestep, we propose applying variable diffusion timesteps across the temporal dimension and across modalities of the inputs. This formulation offers flexibility to introduce variable noise levels for various portions of the input, hence the term mixture of noise levels. We propose a transformer-based audiovisual latent diffusion model and show that it can be trained in a task-agnostic fashion using our approach to enable a variety of audiovisual generation tasks at inference time. Experiments demonstrate the versatility of our method in tackling cross-modal and multimodal interpolation tasks in the audiovisual space. Notably, our proposed approach surpasses baselines in generating temporally and perceptually consistent samples conditioned on the input. Project page: neurips13025.github.io

Main paper: 
 introduction: 
 Recent years have witnessed a remarkable surge in the development and exploration of multimodal diffusion models. Prominent examples include text-to-image (T2I) [37, 41, 54, 39 ], text-to-video (T2V) [17, 6, 12] . Despite notable advancements, generating sequences across multiple modalities, like video and audio, remains challenging and is an open research area.Introducing a time axis to static data paves the way for diverse multimodal sequential tasks including cross-modal generation (e.g., audio-to-video), multimodal interpolation, and audiovisual continuation as shown in Fig 1 . Each task can be further divided based on various input-output combinations of the modalities, leading to a number of conditional distributions. For example, with video data x 1:N 0 and audio data y 1:N 0 of length N , The complexity of configurations grows with tasks like audiovisual continuation, ppx pnc`1:N q 0 , y pnc`1:N q 0 |x p1:ncq 0 , y p1:ncq 0 q, where n c is the input frame length used for conditioning, and multimodal interpolation, ppx pnPN cx q 0 , y pnPN c y q 0 |x pnPNxq 0 , y pnPNyq 0 q, where N x and ˚Work done while at Google. : Equal contribution. Corresponding author: Krishna Somandepalli xksoman@google.comy N y are input index sets. Training separate models for each variation is expensive and impractical. A more efficient training approach would be to learn these conditional distributions in a single model without explicitly enumerating them, i.e., in a task-agnostic manner.Unconditional diffusion models like MM-Diffusion [40] show potential for learning conditional distributions implicitly, but rely on inference adjustments [18, 40] . This limits performance, as seen in MM-Diffusion's struggle to generate temporally consistent sequences (see Fig. 2 ). While UniDiffuser [3] and Versatile Diffusion [51] offer methods for joint and conditional text-image distributions, effectively capturing temporal dynamics of audio and video remains an open challenge.Here, we propose a multimodal diffusion framework that empowers a single model to learn diverse conditional distributions. This paves the way for a versatile framework for multimodal diffusion, tackling various generation tasks. Our core idea is that, applying variable noise levels across modalities and time segments 3 enables a single model to learn arbitrary conditional distributions. This formulation offers flexibility to train diffusion models with a mixture of noise levels i.e., MoNL, which introduces variable noise levels across various portions of the input. It has a number of advantages over previous approaches: it requires minimal modifications to the original denoising objective simplifying implementation, task-agnostic training, and support for conditional inference of a given task specification without any inference-time modifications.We apply this approach for audiovisual generation by developing a diffusion transformer, AVDiT. To address the computational complexity of high-dimensional audio and video signals, we implement MoNL in the low-dimensional latent space learned by the MAGVIT-v2 [55] for video and the SoundStream [57] for audio. Importantly, the temporal structure in these latent representations enables us to apply variable noise levels. We also introduce a transformer-based network for joint noise prediction. Transformers are a natural choice for our implementation due to their proficiency to model multimodal data [25, 10] capturing complex temporal and cross-modal relationships.We assess the capability of MoNL to model various distributions in the audiovisual space by evaluating cross-modal tasks (audio-to-video and video-to-audio generation), and conditioning on small portions (audiovisual continuation and interpolation tasks). For these tasks, we show that the AVDiT trained with MoNL outperforms conventional methods including unconditional and conditional generation models, demonstrating the versatility of our task-agnostic framework as shown in Fig. 1 . Notably, qualitative and quantitative evaluations highlight the ability of our framework to generate temporally consistent sequences, as illustrated in Fig. 2 .Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The two main claims of the paper, i.e., task-agnostic training to support a range of audiovisual generation tasks at inference time, and the ability of our model to generate temporally consistent generations have been empirically validated through quantitative and user studies, along with demo videos on the project page. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
background: 
 Diffusion Models for Multivariate series data: Consider an example of a video diffusion model where the input is a sequence of image frames. In general, this task is modeling multivariate series data (i.e., image representations) of d-dimensions with N elements (no. of image frames), henceforth referred to as time-segments. Thus, the multivariate series data, x 0 " x 1:N 0 P R N ˆd " qpx 0 q can be represented as a sequence of time-segments, where x n 0 P R d is the n-th time-segment and d-dimensional representation.During the forward process of diffusion models [44, 16] , the original data x 0 is corrupted by gradually injecting noise in a sequence of T timesteps. The noisy data x t at time t can be written as x t " x 1:N t " ? α t x 0 `?1 ´αt x . Here, x " 1:Nx " N p0, Iq is Gaussian noise injected to the sequence and β t is the noise schedule, α t " 1 ´βt controls the noise level at each step with α t " ś t i"1 α i . Each noisy time-segment can be represented as x n t " ? α t x n 0 `?1 ´αt n x . During the reverse process, the data is sampled through a chain of reversing the transition kernel qpx t´1 |x t q that is estimated by p θ px t´1 |x t q " N px t´1 |µpx t , tq, σ 2 t Iq, where µpx t , tq " ? α t px t ´1´αt ? 1´αt θ px t , tqq. The training objective is to learn a residual denoiser θ at each step as:EQUATIONwhere t " Upt1, 2, . . . , T uq is the diffusion timestep.Multimodal Diffusion Models: Unconditional joint generation (generating all modalities simultaneously) and conditional generation (generating one modality conditioned on the rest) are commonly used for multimodal diffusion. Typically, separate models are trained for each task as described below:Diffusion models for joint generation. For simplicity, let us assume two modalities x 0 , y 0 . The objective in joint generation is to model the joint data distribution, denoted as qpx 0 , y 0 q. To learn this, a joint noise prediction network, denoted as θ is defined by rewriting Eq. 1 as follows:EQUATIONwhere px 0 , y 0 q is a random data point, r, s denotes concatenation, x , y " N p0, Iq, and t " Upt1, 2, . . . , T uq. Diffusion models trained with this objective can perform conditional sampling qpx 0 |y 0 q using inference-time tricks [18, 40] .Conditional training of diffusion models. To learn conditional distributions, expressed as qpx 0 |y 0 q, a noise prediction network θ conditioned on y 0 is adopted from Eq. 2:EQUATIONSeparate conditional models need to be trained for every pair of modalities and input configurations.3 Mixture of Noise Levels (MoNL)We introduce a novel framework for learning a wide range of conditional distributions within multimodal data by using a mixture of noise levels. The key idea is to formulate the timestep t (Eq. 1) that determines a noise level in the forward diffusion as a vector. Then, we present representative strategies for variable noise levels. We then show how conditional inference can be performed without additional training. Finally, assembling all these components, we present our versatile audiovisual diffusion transformer (AVDiT). 
variable noise levels across modality and time: 
 Formally, let M represent the number of modalities with sequence representations (latent spaces or raw data). Without loss of generality, assume the representations in each modality have N time- segments 4 . Let us further assume they have the same embedding dimension d (which in practice can be achieved by projecting the noisy input from each modality to the desired dimension). The entire sequence can then be simplified as z 0 P R M ˆN ˆd " z p1:M,1:N q 0 " qpz 0 q, where z pm,nq 0 P R d denotes the n-th time-segment of m-th modality. For reference, Sec. 2 represents two modalities of multivariate series data, x 1:N 0 and y 1:N 0 , using this notation as z p1,1:N q 0 and z p2,1:N q 0, respectively.We posit that training a single model to support learning arbitrary conditional distributions can be realized by using variable noise levels for each modality m and time segment n of the input space z 0 . We introduce the diffusion timestep vector as t " t p1:M,1:N q P R M ˆN to match the dimensionality of the multimodal inputs, where each element t pm,nq P r1, T s determines the timestep, and in turn the level of noise added to the corresponding element z pm,nq 0 of the input z 0 .Recall (from Sec. 2) that in a unimodal case, the goal was to learn the transition kernel qpx t´1 |x t q parameterized by p θ px t´1 |x t q " N px t´1 |µpx t , tqq, σ 2 t Iq. Analogously, by introducing a timestep vector t P R M ˆN , our goal is now to learn a general transition matrix between the various modalities and time-segments in z 0 at each step:p θ prz p1,1qt p1,1q ´1, . . . , z pM,N q t pM,N q ´1s|rz p1,1q t p1,1q , . . . , z pM,N q t pM,N q sq (4)Video data Audio data Clean audio latent .Noise levelMAGViT-v2 Soundstream Then, for diffusion training, we draw a Gaussian noise sequence " p1:M,1:N q . Each noise element pm,nq is then added to the corresponding element of the original data z pm,nq 0 with noise level determined by t pm,nq as follows:EQUATIONThen, the joint and conditional training objectives in Eqs. 2 and 3 can be generalized with a single noise prediction objective to learn the joint distribution θ as follows:min θ E t,z0, } θ prz p1,1qt p1,1q , . . . , z pM,N qEQUATIONwhere z 0 " qpz 0 q is the multimodal input and t is the diffusion timestep vector.Using the generalized view of multimodal noise prediction described in Eq. 6, we now examine various strategies for variable noise levels during the forward diffusion. One can imagine an arbitrarily large number of timestep candidates in the vector space of t drawn as functions of time-segments of the multivariate series and modalities. Here, we explore four designs to create a mixture of noise levels as illustrated in Fig. 3 To enable learning a wide range of conditional distributions, we create a training paradigm where a timestep is uniformly randomly selected from the mixture. Specifically, we refer to this training paradigm as MoNL. A schematic of the overall training process is depicted in Fig. 3 (a), with related pseudocodes in Algorithms 1 and 2 in the Appendix. Also, theoretical background on MoNL is detailed in Appendix G.Once the general transition kernel p θ is learned in Eq. 4, we investigate the model's ability to handle arbitrary conditional distributions. We achieve this by selectively injecting inputs during inference based on the task specification, i.e., clean (no noise) inputs for conditional portions with t pm,nq " 0, and noisy inputs for generating desired portions of the input with the current diffusion step t pm,nq " t.Consider the case of cross-modal generation (Fig. 4 (a)), to generate a sequence of M ´mc modalities conditioned on m c P p1, M q modalities, we set timestep elements of M ´mc modalities as t and those of m c conditioning modalities as 0, which achieves:p θ `zpmc`1:M,1:Nq t´1 |z pmc`1:M,1:N q t , z p1:mc,1:N q 0 ˘(7)Similarly, for multimodal interpolation (Fig. 4 (b)), to generate N ´nc time-segments of all modalities jointly, conditioned on n c P p1, N q time-segments, we set the timestep for the N ´nc time-segments as t, and for the conditioning n c time-segments as 0, which achieves p θ pz p1:M,nc`1:N q t´1 |z p1:M,nc`1:N q t , z p1:M,1:ncq 0 q. Unconditional joint generation is also possible by setting each timestep as the same t, to estimate the transition kernel, p θ pz p1:M,1:N q t´1 |z p1:M,1:N q t q. Intuitively, our mixture of noise levels is analogous to self-supervised learning which bypasses the need for predefined tasks during training but enables a deeper understanding of multimodal temporal relationships. See also Sec. E for the discussion on classifier-free guidance in the Appendix. 
audiovisual latent diffusion transformer (avdit): 
 Our model consists of two key components: (1) latent space representations from audio and video autoencoders, and (2) an Audiovisual diffusion transformer (AVDiT) for joint noise prediction.Latent Space Representations: For a video of 1 `Lv frames, represented as v P R p1`LvqˆHˆW ˆC , we use MAGVIT-v2 [55] , a causal autoencoder to achieve efficient spatial and temporal compression. MAGVIT-v2 results in a low-dimensional representation, x 0 P R p1`lvqˆhˆwˆdv , by a compression factor of r s " H h " W w in space and r tv " Lv lv in time. Crucially, the use of causal 3D convolutions ensures that the embedding for a given frame is solely influenced by preceding frames, preventing flickering artifacts common in frame-level autoencoders.For audio with L a frames, a P R La , we use SoundStream [57] , a state-of-the-art neural audio autoencoder. We use the latents y 0 P R laˆda prior to quantization as audio latents, a compression rate of r ta " La la in time. The time-segments in our formulation refer to the 1 `lv and l a temporal dimensions in the video and audio latent spaces respectively.Audiovisual Transformer for Joint Noise Prediction: Transformers [48] are a natural fit for multimodal generation as they can: (1) efficiently integrate multiple modalities and their interactions [58, 10] , (2) capture intricate spatiotemporal dependencies [8, 5] , and (3) have shown impressive video generation capabilities [12, 25] . Inspired by these benefits, we introduce AVDiT, a noise prediction network for latent diffusion as described in Fig. 5 . AVDiT utilizes the timestep embedding similar to the condition signal used in W.A.L.T [12] . The Transformer first processes the timestep embeddings and positional encodings to create an embedding of the timestep vector. This embedding serves as a conditioning signal and is utilized to dynamically calculate the scaling and shifting parameters for AdaLN during the Transformer Layer Normalization step. This enables the normalization to incorporate the conditioning information of variable noise levels. We first consider the l a and 1 `lv time-dimensions for audio and video embeddings respectively. When applying MoNL, we can easily keep track of the corresponding time segments among the l a and 1 `lv dimensions, given the temporal compression factors in each modality. The noisy latents are then linearly projected matching the final dimension d by adding appropriate spatiotemporal positional embeddings for video and temporal positional embeddings for audio, resulting in d dimensional embeddings for each modality which are then concatenated.  
related work: 
 Video diffusion models. Diffusion models have revolutionized image [45, 35, 16, 39] and video generation with pixel-space [18, 17, 43] and latent-space [14, 56, 6, 9, 12] approaches. Recently, W.A.L.T [12] pushed the boundaries using transformer-based latent diffusion with joint image-video training. Tackling diverse audio-video generation tasks remains largely unexplored. With an AVDiT trained with MoNL, our unified approach empowers a single model to handle a range of tasks. Audio generative models. Audio generation soared with WaveNet [36] 's autoregressive approach. Adversarial audio generation [27, 42, 26] emerged. Combining this with differentiable quantization [38, 47, 1] led to end-to-end neural codecs for efficient audio compression [21, 57] . Recently, diffusion models joined the fray, some using continuous latent spaces [29, 19, 11] , others exploring discrete space [52] . Our AVDiT uses continuous embeddings from SoundStream for audio latents. Multimodal generative modeling. While multimodal diffusion models [37, 41, 54, 39, 17, 23] have achieved impressive results, the field has primarily focused on the visual domain and audiovisual generation remains less explored. Existing approaches for audio-to-video [53, 53, 30] and video-toaudio [20, 33] generation typically learn task-specific conditional models, limiting their flexibility.To address this, recent works [40, 50] propose more versatile audiovisual models. However, they did not examine multimodal interpolation tasks, which we explore in this work. Tasks such as AV-continuation are critical to understand a model's capability to generate a temporally consistent multimodal sequence retaining the object consistency from the condition input.6 Experiments 
datasets: 
 Monologues dataset consists of 19.1 million videos for training and 25K videos for evaluation, each with a single person talking. The videos are center-cropped to a 256 ˆ256 resolution. This dataset includes a range of person appearances along with rich verbal and non-verbal communication cues. This dataset is ideally situated to assess concepts such as audiovisual gestural synchrony and multimodal expressions which are key components of human communication and interactions. AIST++ is a subset of AIST [46] and contains 1,020 street dance videos (5.2 hours). The videos were segmented into in 8,233 samples for train and 110 for test at 10fps following Ruan et al. [40] .Landscape contains natural scenes from 928 [28] videos which were segemented into 5,400 samples for train and 600 samples for test at 10fps. We conduct most of the experiments on the Monologues due to its size and diversity. We use AIST++ and Landscape for comparison with MM-Diffusion.Tasks. We study three sets of tasks: (1) Joint audio-video (AV) generation (Joint): (2) Cross-modal generation: Audio-to-video (A2V) and Video-to-audio (V2A), and (3) AV interpolation generative tasks: AV-inpaint where a 1.5s clip is interpolated given one video frame, and 0.125s audio at the beginning and four video frames and 0.5s audio at the end, and AV-continue to fill out 1.5 seconds of AV given the first 5 video frames and corresponding 0.625s of audio.Baselines. On the Monologues dataset, we compare the performance of AVDiT trained with MoNL versus three baselines: Vanilla (Eq. 2), Conditional models separately trained for each task, and the per modality model (Sec 3.2) which may be considered as a generalization of the UniDiffuser approach for sequences. We enabled the Vanilla model to generate cross-modal and multimodal interpolation outputs by using the replacement method [18, 40] . We also benchmark MoNL AVDiT against UNet-based MM-Diffusion (MMD) [40] , the sole published work with a released model that tackles both audio and video generation within a single model. While a direct comparison between U-Nets and our transformer architecture is inherently challenging due to their distinct design principles, we show that MoNL AVDiT surpasses this strong U-Net baseline, demonstrating the effectiveness of the transformer architecture in this domain. We restrict our quantitative evaluation to A2V and V2A tasks because MMD fails to generate temporally consistent sequences in case of continuation tasks (see Figs. 2 and 7 for example).Quantitative evaluation. We use Fréchet video distance (FVD) as our video evaluation metric following Yu et al. [55] . Similarly, we use Fréchet audio distance (FAD) as the audio evaluation metric following Ruan et al. [40] . Because we use latent space representations for the video and audio, we also report the FVD and FAD scores between reconstructed signal and the original signal as "ground-truth" scores as the performance upper bound. While we preferred user studies for assessing audio-video alignment as existing metrics miss subtle synchrony like dance moves matching music beats or gestures aligning with speech patterns, we computed AV-align score [33] , limiting them to the open-domain Landscape dataset for the comparison with MMD. User studies. We conducted user studies to evaluate the quality of generated content. We adopt the two axes of measurement introduced by Ruan et al. [40] namely audio/video quality and audio-video alignment, and introduce a third one, "subject consistency" to assess whether the person in the generated content is plausibly consistent with the input. For stimuli, we used a total of 360 generated samples (not cherry picked) balanced across A2V, V2A, AV-continue and AV-inpaint tasks and for AVDiT trained with three approaches: MoNL, Vanilla and Per modality on Monologues dataset. The tasks were assessed on a 5-point Likert scale. We also compared rater preference for MoNL AVDiT vs. MMD with 30 videos and 5 raters per video. Raters were presented with generations from the two methods randomized as two options, A/B and were asked to pick one option for each of the three dimensions instead of using a 5-point scale. See more details on implementation and experimental setup in Secs. B and C in the Appendix.Qualitative results. As displayed in Figs. 1 and 6 , Our AVDiT model trained with MoNL achieves impressive performance on various tasks within a single framework, including audio-to-video, videoto-audio, joint generation, multimodal continuation and interpolation with flexible input settings, generating temporally consistent videos. Notably, ours preserves clothing and appearance attributes during continuation tasks, unlike MMD which can alter these (see Figs. 2 and 7 ). More qualitative results and comparisons are available in Figs. 13 , 14 and 15, and at avdit2024.github.io.Quantitative results. As shown in Table 1 , on average across all tasks, AVDiT trained with MoNL outperforms all baselines, demonstrating its versatility to learn diverse conditional distributions in a task-agnostic manner. MoNL excelled at generating samples that are temporally and perceptually consistent with the conditioning input, in the case of AV-inpaint and AV-continue tasks, where other baselines generally failed. Per-modality approach surpassed MoNL for A2V and V2A tasks consistent with the findings in Bao et al. [3] likely because conditional distributions in these cases only need to capture cross-modal associations and not necessarily the underlying temporal dynamics. Unsurprisingly, the vanilla diffusion model trained for joint generation exhibited superior performance in this specific scenario but served as a lower-bound of performance for all other tasks. Finally, MoNL performed better than (if not on-par with) task-specific models for all conditional tasks.As evident from Table 2 , MoNL outperformed MMD in terms of the FAD and FVD metrics across all tasks on the AIST++ and Landscape datasets, as estimated using the code provided by Ruan et al. [40] . The significantly better audio generation in our model, likely due to the combination of MoNL and our choice of the SoundStream audio autoencoder, is also reflected in the ground-truth FAD scores for audio reconstruction. In case of video reconstruction quality, (ground-truth FVD) on AIST++, our choice of autoencoder was inferior to MMD, possibly due to the small dataset size. Qualitatively, we observed that the MAGVIT-v2 reconstructions eliminated flickering across frames but the reconstruction of small face regions in AIST++ dance videos was blurry. These findings should be interpreted cautiously due to several factors: the limited size of the AIST++ and Landscape training splits, our use of a transformer backbone versus MMD's coupled U-Nets, and our use of pretrained autoencoders for latent space representations. On the Landscape dataset, AV-align results demonstrate that our model achieves better alignment compared to MMD, which aligns with the findings from the user study below. User studies. A comparison of the distribution of Likert scores across all tasks for the three approaches we compared is shown in Fig. 8 . Pairwise Mann-Whitney U tests were conducted with Bonferroni correction for multiple comparisons to assess statistical difference. Across all axes, raters preferred samples generated from MoNL over that of Vanilla or Per-modality (Pm) approaches. Examining task-specific trends (see Fig. 12 in the Appendix), for the cross-modal tasks, Pm was rated significantly higher than Vanilla, and there was no significant difference between MoNL and Pm (except for the V2A task on AV alignment).For multimodal interpolation tasks, MoNL was rated significantly higher than Pm. In line with quantitative results, these results suggest that MoNL excelled at generating samples, that are perceptually and temporally consistent with the input conditioning.As indicated in Table 3 , our MoNL AVDiT outperformed MMD in user studies, especially in consistency where ours showed improved consistency along factors such as person's appearance or the attire. MMD was preferred slightly more in V2A tasks, possibly because the Soundstream audtoencoder we used for audio was not optimized for music generation like in MMD.Ablations. Recall that MoNL training randomly selects one of the four timestep designs described in Sec. 3.2. We compare MoNL with Per time-segment Pt, and Per time-segment and Per modality Ptm and Pt/Pm/Ptm that excludes vanilla from the timestep mixture, approaches separately as shown in Table 1 . Overall, Ptm noise excelled at inpainting and continuation tasks though it was not on par with per-modality approach for cross-modal tasks. In general, Pt does not perform well by itself. In our experiments, we also observed that the combination of Pm, Pt, Ptm and Pt/Pm/Ptm was sufficient for comparable performance on most tasks except for unconditional joint generation.Adding the Vanilla approach to the mixture of timesteps improved performance for unconditional joint generation while not substantially compromising the performance on other tasks. 
conclusion: 
 We propose a unified approach for multimodal diffusion using a mixture of noise levels (MoNL) for generating and manipulating sequences across modalities and time. This empowers a single model to handle diverse tasks like audio-video continuation, interpolation, and cross-modal generation.We show that an audiovisual latent diffusion transformer (AVDiT) trained with MoNL achieves state-of-the-art performance in audiovisual-sequence generation, providing new opportunities for expressive and controllable multimedia content creation.See Sec. A in the Appendix for discussions on limitations and considerations.A key focus in this work was to demonstrate the versatility and use of MoNL across various tasks using a simple mixing scheme by randomly choosing between different timestep candidates as representative schemes for applying variable noise levels. This simplicity showcases its broad applicability. We acknowledge that fine-grained controlling by weighted mixing of the different schemes could be explored for specific goals or tasks in future work. In fact, one can imagine an arbitrarily large number of timestep candidates in the vector space of the inputs. We specifically chose a simple mixture scheme to demonstrate its versatility as a proof of concept, rather than optimize for any single task.Although the method presented in this work is for general multimodal applications, our experiments included human-centric generation tasks. This enabled us to explore unique challenges of that problem setting. For example, consider the case of perceptual expectations for audiovisual alignment/coherence, where misaligned audio and visual cues can drastically alter perception of speech [34, 24] . Generation of photo-realistic persons, speech, and joint generation of both can perpetuate stereotypes. We recognize the ethical concerns and underscore that our goal here is to explore how understanding aspects such as nonverbal behavior in multimodal communications using generative models can open up new avenues in research.Specifically, the A2V and V2A tasks in human-centric context, which involve extrapolating visual appearance from speech and vice versa, have the potential to perpetuate stereotypes. The generated samples are derived from the model's understanding of cross-modal associations in the training dataset, which can be vastly different from human perception. One possible mitigation is to ensure that the model can generate diverse outputs for a given input. Diffusion models can achieve this by utilizing different noises at inference time, given a sufficiently large and diverse training dataset.A recent study showed that Diffusion models demonstrate better sample diversity in generations compared to GANs [4] , however, addressing potential issues around mode collapse in the generative models, especially with multimodal data is an open research problem.Our work also introduced "consistency" to qualitatively assess whether the generation remains congruent with the input conditioning. In continuation and interpolation tasks, the disparity between what the model generates and human perception can be generally minimal, as the conditioning provides a perceptual template for the subject's potential appearance or voice. In contrast, A2V and V2A tasks warrant an in-depth analysis of this disparity. As our immediate goal was to assess the capability of our proposed approach (and baselines) to generate samples from various conditional distributions, we focused on a broad definition for measuring consistency in user studies. Our future work will focus on extending the consistency measure for a granular understanding of these biases by (1) comparing cross-modal associations in the training data to that of the generated samples, and (2) disaggregate model and human evaluations in cross-modal generation tasks by identifying specific dimensions of human appearance attributes like perceived gender expression or human communication aspects such as voice and gestural synchrony using diverse rater pools.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: For qualitative studies, where we examined rater labels of quality or preferences, we describe the statistical test along with the p-value at which the statistical significance is estimated after multiple-comparison correction is also reported. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. We have carefully considered some of the potential societal impacts of this work and potential harms, and discussed possible mitigations in the section "Limitations, Impact and Considerations" in the Appendix. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We have discussed broader impact of the paper in the section "Limitations, Impact and Considerations" in the Appendix. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [Yes] Justification: Detailed instructions along with the user interface ahown to the raters and the rubric used for rating tasks are provided in the Appendix. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Autoencoders and AVDiT. Given the domain specific nature of the datasets. we trained datasetspecific MAGVIT-v2 autoencoders following Yu et al. [55] . For the Monologues dataset, we downsampled the data to 8fps and 128 ˆ128 resolution for video and 16kHz for audio and randomly sampled a contiguous clip of 2.125 second (17 frames) to match the input requirements of MAGVIT-v2. This resulted in a dataset of about 11.8K hours for training. The spatial and temporal video compression ratios were set to r s " 8 and r tv " 4, whereas the temporal audio compression ratio was r ta " 320. The embedding dimension of the video and audio latent spaces are d v " 8 and d a " 1024 respectively, with the target embedding dimension after linear projection, d " 1024. All latents were zero-mean and unit-variance normalized with empirical mean and variance estimates on a small subset. AVDiT has 24 transformer layers with 16 heads with MSA with a total of 420M parameters.Diffusion training and inference. During training, we use a linear noise variance schedule and a diffusion step T " 1000, and a self-conditioning rate of 0.9 following Gupta et al. [12] . At the inference time, we use 250 DDIM steps. All models were trained for about 400K steps with a batch size of 256. We used the AdamW optimizer [31] with a learning rate of 5e-4, 5K warm-up steps, cosine learning rate scheduler and EMA consistent with the denoising transformer setting in Gupta et al. [12] . Compute resources. Each experiment listed in Table 1 was conducted using 256 v5e TPU chips (with 16ˆ16 topology) for training (on average, the models were trained for around 350K steps with a batch size of 256 for around five days); inference was conducted using 16 v5e TPU chips with a topology of 4ˆ4. See https://cloud.google.com/tpu/docs/v5e for more details. Benchmarking experiments to compare with MM-Diffusion on the AIST++ and Landscape datasets were conducted using two A100 GPUs for conditional inference and estimating FAD/FVD metrics.Since our primary use case is speech generation with the Monologues dataset, we use VGGish embeddings as feature for FAD estimation [22] for the results reported in Table 1 . For AV interpolation generative tasks (AV-continue and AV-inpaint), we carefully excluded the conditioning AV frames while estimating Fréchet metrics.In order to conduct a fair comparsion to the results reported in MM-Diffusion publication [40] , we use the data preprocessing and evaluation code provided at github.com/researchmm/MM-Diffusion for FVD, FAD and KVD metrics. Note that the FAD computation here does not use VGGish embeddings. Instead, it uses AudioCLIP [13] which was trained for general sound classification tasks and not suitable for speech generation tasks as in the Monologues dataset reported in Table 1 .For the FAD, FVD and KVD results reported in Table 2 , we match training conditions for the input image resolution and video FPS with Ruan et al. [40] , i.e., 64 ˆ64 resolution images at 10fps. We match the duration of audio-video from both models to 2 seconds. For visualization purpose in Fig. 2 , we also train our models with 256 ˆ256 resolution to match the super-resolution output resolution used by MM-Diffusion.Ruan et al. [40] introduce a method for implementing zero-shot transfer of A2V and V2A tasks, inspired by the reconstruction-guided sampling proposed by Ho et al. [18] . For instance, in V2A tasks, the generated noisy audio ãt is computed at each step as follows:EQUATION)ãt " a t ´λ?EQUATION) where a t`1 , vt`1 are a N -length sequence of generated noisy audio and conditioned noisy video at t `1, θ av is a parameterized denoising step, and λ is a gradient weight. Similarly, the zero-shot transfer of AV-continuation task using the reconstruction-guided sampling [18] can be described by the following equations: EQUATIONFigure 9 : Example stimuli shown to the raters for the user study. We conducted user studies for four tasks, A2V, V2A, audiovisual continuation and multimodal interpolation tasks. One track each for the audio and video modality below the stimulus video were shown to effectively convey the portions that were generated (in green) and condition input (gray). The generated content has natural structure in SOME places, but not most 3The generated content has natural structure in MOST places 4The generated content is NATURAL, BUT can be recognized as GENERATED content 5The generated content is so NATURAL that it is indistinguishable from the REAL-WORLD The audio-video are total noise and are completely IRRELEVANT 2The generated content has CORRELATION between audio and video in SOME segments, but not most 3The generated content has CORRELATION between audio and video in MOST segments 4The generated content has NATURAL CORRELATION between the audio and video, BUT can be recognized as GENERATED content 5The generated content has CORRELATION between the audio and video indistinguishable from the REAL-WORLD The person generated is CONSISTENT with content in SOME SEGMENTS 3The person generated is CONSISTENT with the generated content in MOST SEGMENTS 4The person generated is NATURALISTIC, but can be recognized as GENERATED content 5The person generated is INDISTINGUISHABLE from the REAL-WORLDwhere n c is the number of conditioned time-segments. We closely follows their V2A codebase to faithfully execute the continuation task. We adopt λ " 0.02 to prevent numerical instability, as the results tend to diverge for λ ą 0.02.Examples of the video stimulus template shown to the raters is presented in Fig. 9 . The rater instructions provided for each axis of quality, alignment and consistency are shown in Tables 4, 5 and 6 respectively.Algorithm 1 Sampling of a diffusion timestep vector 1: function GETTIMESTEPVEC(type) 2: if type "" MoNL then 3:type " UptVanilla, Pt, Pm, Ptmu 4:tref P R M ˆN " Upt1, 2, . . . , T uq 5:t " 0 P R M ˆN 6:for m " 1, . . . , M do 7:for n " 1, . . . , N do 8:if type "" Vanilla then 9:t pm,nq " t `a1 ´αt pm,nq pm,nq 8:end for 9: end for 10: Take gradient step on 11: ∇ θ } θ prz p1,1q t p1,1q , . . . , z pM,N q t pM,N q s, tq ´ } 2 2 12: until converged Algorithm 3 Joint generation of z 0 1: ẑT P R M ˆN ˆd " N p0, Iq 2: for τ " T, . . . , 1 do3: P R M ˆN ˆd " N p0, Iq if τ ą 1, else " 0 4: t P R M ˆN " τ I 5: ẑτ´1 " 1? ατ ´ẑ τ ´βτ ? 1´αt θ p ẑτ , tq ¯`σ τ 6: end for 7: return ẑ0 Algorithm 4 Cross-modal generation of ẑ0 P R pM ´mcqˆN ˆd conditioned on z 0 P R mcˆN ˆd 1: ẑT P R pM ´mcqˆN ˆd " N p0, Iq 2: t P R M ˆN " 0 3: for τ " T, . . . , 1 doP R pM ´mcqˆN ˆd " N p0, Iq if τ ą 1, else " 0 5: t pmc`1:M,1:N q " τ I 6: ˆ " pmc`1:M,N q θ prz 0 , ẑτ s, tq 7: ẑτ´1 " 1 ? ατ p ẑτ ´βτ ? 1´ατ ˆ q `στ 8: end for 9: return ẑ0The core algorithms for implementing mixture of noise levels (MoNL) are presented in three parts: sampling of diffusion timestep vector (Algorithm 1), training process (Algorithm 2), and joint/crossmodal generation at inference time (Algorithms 3 and 4). The sampling algorithms are flexible, using DDPM [16] as an example, and can be replaced with other efficient learning-free samplers like DDIM [45] or Analytic-DPM [2] . Notably, conditional generation across time-segments is simply a change of axis in Algorithm 4.Classifier-free guidance (CFG) [15] , a technique designed to enhance the quality of samples produced by conditional diffusion models using a linear combination of the conditional and unconditional outputs as follows: are obtained as illustrated in Fig. 4 . Our vector formulation of the timestep allows us to apply varying levels of noise to different parts of the input. This opens up a number of possibilities for constructing various CFG forms by emphasizing different time segments or modalities, depending on the task at hand. MoNL supports classifier-free guidance (CFG) without requiring additional design. Unlike the original CFG (see Eq. 13), it does not need a null token either, hence gratis or free. This is achieved by injecting Gaussian noise to the conditional portions of the multimodal space and setting t pm,nq " T for the output as illustrated in Fig. 10 for the case of cross-modal generation of audio-in, video-out. To illustrate mCFG, consider the conditional output of the network in the cross-modal task (see Eq. 7), denote term used in the gradient step as Z p1:M,1:N q t " rz p1,1q t p1,1q , . . . , z pM,N q t pM,N q s and conditional portions as cond θ " θ pZ p1:M,1:N q t , tq where Z p1:mc,1:N q t " z p1:mc,1:N q 0 , Z pmc`1:M,1:N q t " z pmc`1:M,1:N q t t p1:mc,1:N q " 0, t pmc`1:M,1:N q " t.Then, the output for the cross-modal generation task is:EQUATIONwhere Z p1:mc,1:N q t " z p1:mc,1:N q 0 , Z pmc`1:M,1:N q t Figure 10: Application of CFG for free in our MoNL approach for cross-modal generation tasks. Whereas a null token is used in traditional CFG for unconditional output, formulating diffusion timestep as a vector enables this by setting the input condition per task-specification to pure noise.Figure 11 : Application of CFG for free in our MoNL approach for multimodal interpolation tasks. Because our vector formulation of the timestep enables applying variable noise levels to different portions of the input one can construct a different CFG with "mix-and-match" of modalities and timesegments for creating unconditional outputs. Here, we show an example for multimodal interpolation task for (a) conditional output with two variations: (b) unconditional output with respect to input condition per task specification, (c) partial conditional output, but unconditional output with respect to modalities. can amplify specific time segments or modalities based on the task demands, as demonstrated in Fig. 10 for cross-modal tasks and Fig. 11 for the case of multimodal interpolation generation driven by temporal conditioning.Limited conditioning information: In Table 7 , we compares the results of AV continuation depending on the input information: AV-continue-2s) to fill out 2 seconds of AV given the first video frame and corresponding 0.125s of audio AV-continue-1.5s) to fill out 1.5 seconds of AV given the first 5 video frames and corresponding 0.625s of audio. The model performed better when given more context (5 video frames and corresponding audio) compared to less context (1 frame and corresponding audio), even though task-specific training can be an upper bound for performance. This suggests that limited conditioning information can lead to issues like unnatural motion and inconsistencies, and including more context improves the model's performance.Table 7 : Comparison of AVDiT trained with mixture of noise levels (MoNL) on the Monologues dataset for AV-continue-2s) to fill out 2 seconds of AV given the first video frame and corresponding 0.125s of audio AV-continue-1.5s) to fill out 1.5 seconds of AV given the first 5 video frames and corresponding 0.625s of audio. FAD " 2.7 and FVD " 3.3 for ground truth autoencoder reconstructions of the inputs. Fréchet metrics estimated with N=25k.AV-continue-2s AV-continue-1.5s Figure 12 : Comparative analysis across AVDiT models from the user study along axes of AV quality, AV alignment and person consistency for two cross-modal generation tasks (A2V and V2A), and multimodal interpolation tasks (AV-continue and AV-inpaint). The * indicates statistically significant pairwise difference at p ă 0.01 after multiple comparison correction. Across the board, MoNL (Ours) was rated significantly better or on par across all tasks, except for AV-alignment for V2A task (comparison shown in red). For A2V task, there was no significant difference between the models compared for the measure of AV quality. For multimodal interpolation tasks (bottom row), Our approach far surpasses other models for quality, alignment and consistency underscoring the ability of our approach to generate temporally consistent samples that are perceptually congruent with the input condition. In "A Theory of Multimodal Learning" [32] , multimodal learning is shown to offer a superior generalization bound compared to unimodal learning, with an improvement factor of Op ? nq, where n denotes the sample size. This benefit relies on connection and heterogeneity between modalities:Connection. The bound depends on learned connections between (X ) and (Y).Heterogeneity. Describes how modalities X and Y, diverge and complement.If connection and heterogeneity are missing, ill-conditioned scenarios can arise. For instance, if x " y, perfect connection suggests no need for learning about Y. On the other hand, if x is random noise, there is heterogeneity but no meaningful connection between X and Y, making non-trivial learning on X alone impractical.The theory also highlights that and effective connections between modalities via generative models can enhance multimodal learning. This forms the basis for our Mixture of Noise Levels (MoNL) approach, which is particularly suited for multimodal learning with audio and video data.Our MoNL training method offers significant benefits for multimodal learning, especially with audio and video data: Heterogeneity and connection. Audio and video are naturally heterogeneous. For example, a video of a person speaking includes audio of spoken words and video of lip movements and facial expressions. MoNL uses variable noise levels to enhance learning by capturing the generic transition matrix across the temporal axis. p θ prz p1,1q t p1,1q ´1, . . . , z pM,N q t pM,N q ´1s | rz p1,1q t p1,1q , . . . , z pM,N q t pM,N q sq (Eq. 4)Enhanced connectivity. MoNL improves connectivity between audio and video modalities. Our experiments show that MoNL often surpasses task-specific learning approaches by fostering better connections between modalities, adapting its focus more effectively.MoNL vs. Joint learning in MMD [40] . Unlike joint learning methods that focus on the joint distribution p θ pz t´1 | z t q, MoNL trains across multiple conditioning, enabling better connections by varying its focus. This is evidenced by MoNL outperforming the Vanilla (see Table 1 ) and MMD models (see Tables 2 and 3 ).MoNL vs. Per-modality training. MoNL goes beyond per-modality training in UniDiffuser [3] , which uses variable noise between modalities i.e., learning p θ prz p1q t p1q ´1, . . . , z pM q t pM q ´1s | rz p1q t p1q , . . . , z pM q t pM q sq. MoNL introduces variable noise across different time segments, learning connections across temporal dynamics as well. This advantage is demonstrated in Table 1 . [49] . Diffusion models often obscure high-frequency details with low noise and low-frequency structures with high noise [7] . MoNL employs variable noise levels to explore diverse frequency components, enhancing the model's ability to correlate high and lowfrequency elements. This is in contrast to masked self-supervised learning, which limits frequencyspecific connections by masking entire elements.In summary, the effectiveness of MoNL for multimodal diffusion models, particularly with audio and video data, stems from its strategic use of connection and heterogeneity. By applying variable noise levels, MoNL enhances connectivity between modalities and better adapts to diverse temporal and frequency components, leading to superior performance compared to existing multimodal learning methods.We use the term, "time-segment" to reference a single unit in time dimension of the inputs (e.g., frame in a video) during forward diffusion. Whereas, "timestep" or "diffusion timestep" refers to a single step in the process of adding noise during forward diffusion process.In practice, this is rarely true; say, video and audio representations, the embedding dimension and temporal compression in the raw data or latent spaces can be vastly different. However, what we propose here can be generalized by keeping track of the frame-level correspondences between modalitiesuncond,p1:M,nc`1:N q θ(17) By formulating the timestep as a vector, we can apply varying levels of noise to different input components. This unlocks diverse possibilities for crafting varied CFG structures. Each structure 


Decision: This paper proposes to use varying noise levels across modalities and time segments towards task agnostic audio-visual synthesis. The paper received four reviewers; before the rebuttal, the reviewers had raised several concerns on multiple aspects, including technical novelty, gaps in experimental studies, datasets used for experiments, and connections to prior works. Authors provided a strong rebuttal, addressing the concerns, resulting in all the reviewers raising their scores inclining towards acceptance. 

AC agrees with the reviewers' assessment of the paper and recommends acceptance. Authors should incorporate all the changes suggested by the reviewers in the camera-ready, including theoretical connections and clarifications from prior methods. Further, as promised, authors should make arrangements for the release of the Monologues dataset.

Reviews: review: paper_summary: paper_summary: The paper presents a new method for audiovisual generation where the input output condiitions may comprise of 2 modalities, namely video and audio sequence. The authors propose a new training approach to effectively learn conditional distribution in multimodal space. The main novely in the paper is a mixture of noise level formulation for processing the inputs. The method produces temporally consistent samples and outperforms existing arts and the vanilla configuration
strength_weakness: strength_weakness: 1. Although the method seems to work well empirically, the paper lacks theoretical backing for the proposed method. It would be good to see some proofs that the proposed method leads to a better approximation of the variational lower bound and joint distribution.
2. Are there any sampling modifications required to accommodate the proposed training strategy?
3. There is a plethora of compositional works derived from the energy based formulation of diffusion models. Could the authors analyze how the proposed method performs in comparison to it.
[1] https://energy-based-model.github.io/Compositional-Visual-Generation-with-Composable-Diffusion-Models/
4. I'm rating the paper borderline for now, I will improve my rating if the authors can address my concerns
questions: questions: 1. Could the authors provide a proof of why the proposed method would work better
2. Are there any sampling modifications required to accommodate the proposed training strategy?
3. Could the authors give an analysis of how the proposed algorithm will work when compared to [1]
limitations: limitations: Please see weakness.
paper_summary: paper_summary: This paper introduces the Audiovisual Diffusion Transformer (AVDiT) with Mixture of Noise Levels (MoNL) for audiovisual sequence generation. The key innovation is the use of variable noise levels during the diffusion process, applied across different time segments and modalities. This approach enables the model to effectively learn arbitrary conditional distributions in a task-agnostic manner, making it versatile for various generation tasks such as cross-modal generation, multimodal interpolation, and audiovisual continuation. Experiments on multiple datasets demonstrate that AVDiT with MoNL outperforms existing baselines, generating temporally and perceptually consistent audiovisual sequences.
strength_weakness: strength_weakness: 1. While the variable noise levels concept is interesting, the overall novelty of the approach may be seen as incremental. Similar techniques in diffusion models and transformers have been explored in papers like MM-Diffusion.
2. Some technical details are not thoroughly explained, such as the criteria for selecting and varying noise levels across time segments and modalities.
3. The evaluation is primarily conducted on a few datasets (Monologues, AIST++, Landscape). The model's generalizability to real-world scenarios remains uncertain.
4. The authors compare the transformer-based AVDiT model to the UNet-based MM-Diffusion model, which might not be a fair comparison due to the different architectures. The authors should consider training a UNet-based model using their proposed approach to provide a more direct comparison and validate the effectiveness of their method.
questions: questions: See weakness.
limitations: limitations: The authors mention some limitations in the appendix, such as the need for further improvements in visual and speech quality and the potential for ethical concerns. However, for me, the lack of extensive evaluation of more diverse and real-world datasets is a noteworthy limitation. Some additional limitations can be found in the Weaknesses section.
paper_summary: paper_summary: The paper tackles the audio-visual cross-modality generation problem and proposes a training approach to learn arbitrary conditional distributions in the audiovisual space. At the methodological level, the authors propose to apply variable diffusion timesteps across the temporal dimension. The experiments are conducted on Monologues, AIST++, and Landscape datasets.
strength_weakness: strength_weakness: 1. The writing and presentation of the paper can be improved. Figure 1 seems to have some issues with the first-row caption for the AIST dataset, which makes it difficult to read. The usage of math symbols is inconsistent, e.g., the $x$ should be $\bf{x}$ in Line 72. The term “multivariate data” in Line 63 is also not rigorous and confusing, in other words, do the authors imply that the static image data is uni-variante? The use of $\equiv$ and $=$ is also mixed in the paper.

2. One of the key claims and motivations, “Training separate models for each variation is expensive and impractical”, seems controversial and susceptible to me, while I understand training separate DMs for audio and visual data could be expensive, I don’t think that learning a single mixture is a better option than learning two separate data distributions, because learning a mixture increase the complexity of target data distribution, and thus should not be beneficial for performance especially if the domain gap between separate modality is large. As this is one of the key claims that motivates the methodological design, I was expecting more rigorous theoretical support in the paper, could the authors elaborate on that?

3. For the experiments, while the authors claim that “the sequential representations can be either latent spaces or raw data”, the actual implementation only conducts experiments on low dimensional features of pre-trained models, i.e., MAGVIT-v2 for video representations and SoundStream for audio representations. While this is not a serious flaw, it raises questions about the generalizability of the proposed method, and the fairness in comparison with other methods, as the pre-trained models may (and very likely) directly influence the final performance. And the baseline method MM-Diffusion seems not to operate on the same feature space according to Appendix C?
questions: questions: In addition to the comments in the Weaknesses section. 
I may have missed several details: how is the scheduler defined in the proposed method Eq. (5)? What is the time cost for training the proposed model? And how does this design yield a difference compared to training on single modality or separate DMs?
limitations: limitations: The limitations are discussed in the Appendix A.
paper_summary: paper_summary: This paper presents a new diffusion model that is capable of generating data following various conditional distributions in multimodal data. Rather than applying a uniform noise level to all data elements, the proposed model permits the use of differing noise levels across modalities and time dimensions. This allows the model to concurrently learn a variety of conditional distributions, including cross-modal conditioning, temporal interpolation, and temporal continuation. The experimental results with audio-video datasets validate the effectiveness of the proposed model.
strength_weakness: strength_weakness: - The major concern is on the experiments: the majority of the empirical evaluations are conducted with the internal dataset. As there are already several publicly-available datasets (such as Landscape, AIST++, and VGGSound) that are commonly used in the literature, I strongly encourage the authors to do at least the ablation studies with such datasets to make them reproducible. 

- The novelty in methoology is somewhat limited. The idea of using various noise levels in multimodal generative modeling has been presented in the prior work [3]. A particular challenge in this paper is to extend this idea also for modeling temporal dynamics, but similar strategies are commonly adopted in several video generation studies:
    - “Diffusion Models for Video Prediction and Infilling,” TMLR 2022.
    - “MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation,” NeurIPS 2022.
questions: questions: - In the ablations at Table 1, why is Vanilla not used? Without the Vanilla strategy, the model may not encounter cases where all elements are almost entirely noise or close to clean ones during training, which could potentially degrade the performance of the model to some extent. Therefore, a more reasonable approach for ablation might be to gradually add other strategies to Vanilla.

---

<post-rebuttal>

I have updated my rating from 4 to 5, as my concern on the dataset is somewhat resolved through the discussion.
limitations: limitations: Limitations are properly discussed in the appendix.

Comments: title: Thanks for the response
Additional analysis of the objectives and effectiveness of our approach
Clarifications and Additional Insights on the Monologues Dataset and MoNL
Thanks for further clarification
Thank you for your time and effort
Request for Final Review: Response and Theoretical Clarifications
Follow-Up on Theoretical Clarifications and Request for Final Review
Request for Final Review: Response and Theoretical Clarifications
Request for Final Review: Response and Theoretical Clarifications
Clarification on Resource Constraints and Additional Theoretical Contributions
Theoretical Background on Mixture of Noise Levels
Theoretical Background on Mixture of Noise Levels
Theoretical Background on Mixture of Noise Levels
Theoretical Background on Mixture of Noise Levels
Post-rebuttal
Thanks for the response
comment: Thanks for the response. I will update my rating after the discussion with the other reviwers.
To enhance the understanding of how MoNL contributes to building a unified model for diverse audiovisual tasks—including cross-modal inference and multimodal interpolation—  and improve connectivity between modalities, we provide a detailed analysis of the objectives and effectiveness of our approach, considering a simplified case with two modalities and two time-segments.

$ $ 
### Objective Functions

To determine the optimal objective function for learning a model $\theta$ that estimates:

- **Cross-modal inferences:** $P(X|Y)$ and $P(Y|X)$
- **Multimodal interpolations:** $P([\mathbf{x}_2, \mathbf{y}_2] \mid [\mathbf{x}_1, \mathbf{y}_1])$, $P([\mathbf{x}_1, \mathbf{y}_1] \mid [\mathbf{x}_2, \mathbf{y}_2])$, $P([\mathbf{x}_1, \mathbf{y}_2] \mid [\mathbf{x}_2, \mathbf{y}_1])$, and $P([\mathbf{x}_2, \mathbf{y}_1] \mid [\mathbf{x}_1, \mathbf{y}_2])$

we evaluate the following objectives for heterogeneous multimodal data $X$ and $Y$, where $X = [\mathbf{x}_1, \mathbf{x}_2]$ and $Y = [\mathbf{y}_1, \mathbf{y}_2]$.

1. **Task-Specific Method**

   $$
   \min \mathbb{E}[P_\theta(X \mid Y)]
   $$

   This approach focuses solely on one direction of inference, potentially missing significant interactions between modalities.

2. **Joint Learning Method**

   $$
   \min \mathbb{E}[P_\theta(X, Y)]
   $$

   This method captures both modalities simultaneously but may fail to model temporal and multimodal interactions, which are crucial for conditional tasks.

3. **Per-Modality Method**

   $$
   \min \mathbb{E}\left[P_\theta(X, Y) + P_\theta(X \mid Y) + \alpha P_\theta(Y \mid X)\right]
   $$

   This method improves upon the task-specific approach by incorporating reverse inference between modalities. However, it may not effectively capture temporal interactions, which are important for interpolation tasks.

4. **Mixture of Noise Levels (MoNL)**

   $$
   \min \mathbb{E}\left[P_\theta(X, Y) + \beta_1 P_\theta(X \mid Y) + \beta_2 P_\theta(Y \mid X) + \beta_3 P([\mathbf{x}_2, \mathbf{y}_2] \mid [\mathbf{x}_1, \mathbf{y}_1]) + \beta_4 P([\mathbf{x}_1, \mathbf{y}_1] \mid [\mathbf{x}_2, \mathbf{y}_2]) + \beta_5 P([\mathbf{x}_1, \mathbf{y}_2] \mid [\mathbf{x}_2, \mathbf{y}_1]) + \beta_6 P([\mathbf{x}_2, \mathbf{y}_1] \mid [\mathbf{x}_1, \mathbf{y}_2])\right]
   $$

   MoNL integrates joint distributions, cross-modal inferences, and multimodal interpolations by learning a transition matrix that captures complex interactions between modalities.

$ $
### Heterogeneous and Connected Multimodality

For heterogeneous multimodal data (e.g., audio and video), where $X$ and $Y$ are interconnected but distinct, MoNL excels by capturing intricate interactions through:

- **Direct Relationships:** Modeled by $P_\theta(X, Y)$
- **Cross-Modal Inferences:** Modeled by $P_\theta(X \mid Y)$ and $P_\theta(Y \mid X)$
- **Multimodal Interpolations:** Modeled by $P([\mathbf{x}_2, \mathbf{y}_2] \mid [\mathbf{x}_1, \mathbf{y}_1])$, $P([\mathbf{x}_1, \mathbf{y}_1] \mid [\mathbf{x}_2, \mathbf{y}_2])$ and similar terms.

MoNL effectively models the connectivity and complex interactions between modalities, significantly enhancing performance in various audiovisual tasks.


$ $
We hope this analysis is helpful and will be incorporated into the final version.
Dear Reviewer bNvz,

We sincerely appreciate your time and thoughtful feedback. We are grateful for your recognition of the detailed explanations and clarifications provided, and for upgrading our paper to a "weak accept."

We recognize the need for stronger theoretical foundations. While we acknowledge the current limitations in the theoretical landscape of multimodal learning as highlighted in *"A Theory of Multimodal Learning."*  (Zhou et al. NeurIPS 2023), we believe our research makes a substantial empirical contribution to this field. Our findings serve as a robust foundation for future theoretical explorations and advancements.

We are committed to addressing the theoretical aspects of our work in greater depth in our ongoing research. Thank you once again for your valuable insights.

Sincerely,

The Authors
Dear Authors,

I thank you for the detailed explanations and clarifications. After careful consideration and going through the other reviews I’m increasing my rating to weak Accept. The reason of not giving a higher score is due to lack of theoretical novelty.
Dear Reviewer 4xRp,


Thank you for your valuable feedback. We appreciate your interest in the Monologues dataset and your insights into the theoretical aspects of our work.




**Dataset Construction**
To address your concerns about dataset construction, we want to emphasize that the Monologues dataset was meticulously curated to adhere to ethical and legal standards. We have implemented robust measures to protect user privacy and avoid copyright infringement. The dataset consists exclusively of public links (and timestamps) to videos that are:
* Including those that are **older than 90 days** 
* Free from **copyright claims**
* Excluding **harmful or inappropriate material** (e.g., nudity, violence)
* Verified to contain **motion** (excluding static images)
* In the **English language** with a **single visible speaker**
* Framed with **torso up**

The dataset link list is automatically updated on a daily basis to ensure ongoing compliance with these criteria.

The Monologues dataset is a valuable resource for the research community, providing a rich audiovisual benchmark for developing and evaluating multimodal models. 



**Theoretical Background**
While formally proving the connectivity benefits of MoNL presents a significant challenge, our empirical results provide compelling evidence of its effectiveness. Table 1 and Figures 2 and 7 demonstrate MoNL's superior performance across diverse audiovisual tasks, generating more temproally-consistent audiovisuals compared to joint learning and per-modality method, suggesting enhanced intra- and inter-modality connectivity. 

We recognize the need for a deeper theoretical understanding and plan to explore this area in future research. As highlighted in *"A Theory of Multimodal Learning."*  (Zhou et al. NeurIPS 2023), theoretical foundations in multimodal learning remain relatively underdeveloped. Our work serves as a **crucial experimental foundation** for future theoretical investigations in this field. We believe our work represents a **substantial step forward in audiovisual and multimodal research**.


**We believe these clarifications provide valuable insights into our work and would be grateful for your consideration of an improved score.**

Sincerely,

The Authors
I appreciate that the authors plan to release the Monologues dataset. I would like to hear more on how the dataset was properly constructed to avoid any issues related to license or private information.

One quick comment on the additional theoretical background. This theoretical perspective itself sounds really interesting. While I have no doubt on heterogeneity of audio and video, it seems that it is not trivial to show how MoNL contributes to boost connectivity in theory.
Dear Reviewer tkoE,

We sincerely appreciate your time and effort in reviewing our paper. We are glad that you found our work to be a good contribution to the field of audiovisual generation.

We believe that our AVDiT with MoNL effectively models temporal dependencies, handles diverse multimodal tasks, and demonstrates superior performance compared to existing approaches, offering significant advancements beyond prior art. We believe these contributions can be valuable to the community!

We are committed to fostering reproducibility in our research and are happy to make our trained models and code publicly available upon acceptance of the paper.

Thank you again for your valuable insights.

Sincerely,

The Authors
Dear Reviewer bNvz,

We would like to gently remind you that we have **only one day remaining** for further discussion on our manuscript. We have carefully considered your valuable feedback and provided **theoritical backing** for our methodology. 

We greatly appreciate your time and insights thus far. 

Thank you again for your contributions.

Sincerely,

The Authors
Thank you for addressing all my comments. Overall, it is a good paper on audiovisual generation, although I find the proposed method to be somewhat incremental. I suggest the authors make their trained models and code available for reproducibility. I'm willing to increase my rating to BA.
Dear Reviewers,

We kindly request that you review our responses, as we have only **two days** left for further **discussion**. We have addressed your comments thoughtfully and provided a thorough **theoretical background** of our method, which will be incorporated in the final version. We sincerely appreciate the time and effort you have dedicated to reviewing our paper and your constructive and insightful feedback.

Thank you once again.

Best regards,  
The Authors
Dear Reviewer SA7V,

Thank you for your feedback and for raising your score after reviewing our rebuttal. We appreciate your acknowledgment of our efforts to clarify the experimental aspects of our work.

In response to your concern about the theoretical intuition, we want to highlight that we have newly provided the **"Theoretical Background on Mixture of Noise Levels,"** which we hope you will consider as part of our ongoing effort to strengthen our work and contribute to the field. 

As we have only two days left for further discussion, we kindly request that you review our additional responses. We sincerely appreciate the time and effort you have dedicated to reviewing our paper and your constructive and insightful comments.

Thank you once again.

Best regards,  
The Authors
Dear Reviewer tkoE,


We kindly request that you review our responses, as we have only **two days** left for further **discussion**. We have addressed your comments thoughtfully and provided a thorough **theoretical background** of our method. We sincerely appreciate the time and effort you have dedicated to reviewing our paper and your constructive and insightful feedback.

Thank you once again.

Best regards,  
The Authors
Dear Reviewer bNvz,


We kindly request that you review our responses, as we have only **two days** left for further **discussion**. We have addressed your comments thoughtfully and provided a thorough **theoretical background** of our method. We sincerely appreciate the time and effort you have dedicated to reviewing our paper and your constructive and insightful feedback.

Thank you once again.

Best regards,  
The Authors
Dear Reviewer 4xRp,

Thank you for your feedback and for considering our initial response.


We acknowledge the importance of validating our model on publicly available datasets. Unfortunately, training on these datasets with the necessary computational resources (requiring approximately $35k per model) presented a significant challenge at the moment. However, we plan to proceed the experiments as the permit is approved and believe our work with the Monologues dataset, which contains 16M samples with a diverse range of audiovisual cues, provides valuable insights and demonstrates the robustness of our approach. 

In addition, we have newly provided the **"Theoretical Background on Mixture of Noise Levels,"** which we hope you will consider as part of our ongoing effort to strengthen our work and contribute to the field.

We hope you understand our constraints and continue to see the value in the contributions we have made with the resources available. We remain committed to advancing research in **audiovisual generation** and plan to release the Monologues dataset to further support reproducibility in the future and encourage future work.

Thank you for your consideration!
## Theoretical Background on Mixture of Noise Levels
$ $
### 1. Theoretical Background on Multimodal Learning

In *"A Theory of Multimodal Learning"* [1], multimodal learning is shown to offer a superior **generalization bound** compared to unimodal learning, with an improvement factor of **$O(\sqrt{n})$**, where **$n$** denotes the sample size. This benefit relies on **connection** and **heterogeneity** between modalities:

- **Connection**: The bound depends on learned connections between (**$\mathcal{X}$**) and (**$\mathcal{Y}$**).
- **Heterogeneity**: Describes how modalities, **$\mathcal{X}$** and **$\mathcal{Y}$**,  diverge and complement.

If connection and heterogeneity are missing, ill-conditioned scenarios can arise. For instance, if **$x \equiv y$**, perfect connection suggests no need for learning about **$\mathcal{Y}$**. On the other hand, if **$x$** is random noise, there is heterogeneity but no meaningful connection between **$\mathcal{X}$** and **$\mathcal{Y}$**, making non-trivial learning on **$\mathcal{X}$** alone impractical.

The theory also highlights that and learning effective connections between modalities via **generative models** can enhance multimodal learning. This forms the basis for our **Mixture of Noise Levels (MoNL)** approach, which is particularly suited for multimodal learning with **audio** and **video** data.

[1] Zhou Lu. *"A Theory of Multimodal Learning."* NeurIPS 2023.

$ $
### 2. Advantages of Mixture of Noise Level Training (MoNL)

Our **Mixture of Noise Level (MoNL)** training method offers significant benefits for multimodal learning, especially with **audio** and **video** data:

- **Heterogeneity and Connection**: Audio and video are naturally heterogeneous. For example, a video of a person speaking includes **audio** of spoken words and **video** of lip movements and facial expressions. MoNL uses **variable noise levels** to enhance learning by capturing the **generic transition matrix** across the **temporal axis**.
$$p _{\mathbf{\theta}}([\mathbf{z} _{t ^{(1,1)}-1} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}-1} ^{(M, N)}] \mid[\mathbf{z} _{t ^{(1,1)}} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}} ^{(M, N)}]) \qquad \text{(Eq. (4))}$$
where $M$, $N$ are the number of modalities and time-segments, respectively.

- **Enhanced Connectivity**: MoNL improves **connectivity** between **audio** and **video** modalities. Our experiments show that MoNL often surpasses **task-specific learning** approaches by fostering better connections between modalities, adapting its focus more effectively.

$ $
### 3. Enhanced Connectivity - Comparison with Existing Methods

- **MoNL vs. Joint Learning in MMD** [2]: Unlike joint learning methods that focus on the joint distribution $p_{\mathbf{\theta}}(\mathbf{z} _{t-1} \mid\mathbf{z} _{t})$, MoNL trains across **multiple conditioning**, enabling better connections by varying its focus. This is evidenced by MoNL outperforming the Vanilla (see Table 1) and MMD models (see Tables 2 and 3).

- **MoNL vs. Per-Modality Training**: MoNL goes beyond per-modality training in UniDiffuser [3], which uses variable noise between modalities i.e., learning $p_{\mathbf{\theta}}([\mathbf{z} _{t ^{(1)}-1} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}-1} ^{(M)}] \mid[\mathbf{z} _{t ^{(1)}} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}} ^{(M)}])$. MoNL introduces variable noise across different **time segments**, learning connections across **temporal dynamics** as well. This advantage is demonstrated in Table 1.

- **MoNL vs. Masked Training** [4]: Diffusion models often obscure high-frequency details with low noise and low-frequency structures with high noise [5]. MoNL employs variable noise levels to explore diverse **frequency components**, enhancing the model's ability to correlate high and low-frequency elements. This is in contrast to masked self-supervised learning, which limits frequency-specific connections by masking entire elements.

[2] Ruan et al. *"MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation."* CVPR 2023.

[3] Bao et al. *"One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale."* ICML 2023.

[4] Voleti et al. *"MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation."* NeurIPS 2022.

[5] Sander Dieleman. *"Noise Schedules Considered Harmful."* [Link](https://sander.ai/2024/06/14/noise-schedules).

$ $
### Conclusion

In summary, the effectiveness of **MoNL** for **multimodal diffusion models**, particularly with **audio** and **video** data, stems from its strategic use of **connection** and **heterogeneity**. By applying **variable noise levels**, MoNL enhances **connectivity** between modalities and better adapts to diverse **temporal** and **frequency components**, leading to superior performance compared to existing multimodal learning methods.
## Theoretical Background on Mixture of Noise Levels
$ $
### 1. Theoretical Background on Multimodal Learning

In *"A Theory of Multimodal Learning"* [1], multimodal learning is shown to offer a superior **generalization bound** compared to unimodal learning, with an improvement factor of **$O(\sqrt{n})$**, where **$n$** denotes the sample size. This benefit relies on **connection** and **heterogeneity** between modalities:

- **Connection**: The bound depends on learned connections between (**$\mathcal{X}$**) and (**$\mathcal{Y}$**).
- **Heterogeneity**: Describes how modalities, **$\mathcal{X}$** and **$\mathcal{Y}$**,  diverge and complement.

If connection and heterogeneity are missing, ill-conditioned scenarios can arise. For instance, if **$x \equiv y$**, perfect connection suggests no need for learning about **$\mathcal{Y}$**. On the other hand, if **$x$** is random noise, there is heterogeneity but no meaningful connection between **$\mathcal{X}$** and **$\mathcal{Y}$**, making non-trivial learning on **$\mathcal{X}$** alone impractical.

The theory also highlights that and learning effective connections between modalities via **generative models** can enhance multimodal learning. This forms the basis for our **Mixture of Noise Levels (MoNL)** approach, which is particularly suited for multimodal learning with **audio** and **video** data.

[1] Zhou Lu. *"A Theory of Multimodal Learning."* NeurIPS 2023.

$ $
### 2. Advantages of Mixture of Noise Level Training (MoNL)

Our **Mixture of Noise Level (MoNL)** training method offers significant benefits for multimodal learning, especially with **audio** and **video** data:

- **Heterogeneity and Connection**: Audio and video are naturally heterogeneous. For example, a video of a person speaking includes **audio** of spoken words and **video** of lip movements and facial expressions. MoNL uses **variable noise levels** to enhance learning by capturing the **generic transition matrix** across the **temporal axis**.
$$p _{\mathbf{\theta}}([\mathbf{z} _{t ^{(1,1)}-1} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}-1} ^{(M, N)}] \mid[\mathbf{z} _{t ^{(1,1)}} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}} ^{(M, N)}]) \qquad \text{(Eq. (4))}$$
where $M$, $N$ are the number of modalities and time-segments, respectively.

- **Enhanced Connectivity**: MoNL improves **connectivity** between **audio** and **video** modalities. Our experiments show that MoNL often surpasses **task-specific learning** approaches by fostering better connections between modalities, adapting its focus more effectively.

$ $
### 3. Enhanced Connectivity - Comparison with Existing Methods

- **MoNL vs. Joint Learning in MMD** [2]: Unlike joint learning methods that focus on the joint distribution $p_{\mathbf{\theta}}(\mathbf{z} _{t-1} \mid\mathbf{z} _{t})$, MoNL trains across **multiple conditioning**, enabling better connections by varying its focus. This is evidenced by MoNL outperforming the Vanilla (see Table 1) and MMD models (see Tables 2 and 3).

- **MoNL vs. Per-Modality Training**: MoNL goes beyond per-modality training in UniDiffuser [3], which uses variable noise between modalities i.e., learning $p_{\mathbf{\theta}}([\mathbf{z} _{t ^{(1)}-1} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}-1} ^{(M)}] \mid[\mathbf{z} _{t ^{(1)}} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}} ^{(M)}])$. MoNL introduces variable noise across different **time segments**, learning connections across **temporal dynamics** as well. This advantage is demonstrated in Table 1.

- **MoNL vs. Masked Training** [4]: Diffusion models often obscure high-frequency details with low noise and low-frequency structures with high noise [5]. MoNL employs variable noise levels to explore diverse **frequency components**, enhancing the model's ability to correlate high and low-frequency elements. This is in contrast to masked self-supervised learning, which limits frequency-specific connections by masking entire elements.

[2] Ruan et al. *"MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation."* CVPR 2023.

[3] Bao et al. *"One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale."* ICML 2023.

[4] Voleti et al. *"MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation."* NeurIPS 2022.

[5] Sander Dieleman. *"Noise Schedules Considered Harmful."* [Link](https://sander.ai/2024/06/14/noise-schedules).

$ $
### Conclusion

In summary, the effectiveness of **MoNL** for **multimodal diffusion models**, particularly with **audio** and **video** data, stems from its strategic use of **connection** and **heterogeneity**. By applying **variable noise levels**, MoNL enhances **connectivity** between modalities and better adapts to diverse **temporal** and **frequency components**, leading to superior performance compared to existing multimodal learning methods.
## Theoretical Background on Mixture of Noise Levels
$ $
### 1. Theoretical Background on Multimodal Learning

In *"A Theory of Multimodal Learning"* [1], multimodal learning is shown to offer a superior **generalization bound** compared to unimodal learning, with an improvement factor of **$O(\sqrt{n})$**, where **$n$** denotes the sample size. This benefit relies on **connection** and **heterogeneity** between modalities:

- **Connection**: The bound depends on learned connections between (**$\mathcal{X}$**) and (**$\mathcal{Y}$**).
- **Heterogeneity**: Describes how modalities, **$\mathcal{X}$** and **$\mathcal{Y}$**,  diverge and complement.

If connection and heterogeneity are missing, ill-conditioned scenarios can arise. For instance, if **$x \equiv y$**, perfect connection suggests no need for learning about **$\mathcal{Y}$**. On the other hand, if **$x$** is random noise, there is heterogeneity but no meaningful connection between **$\mathcal{X}$** and **$\mathcal{Y}$**, making non-trivial learning on **$\mathcal{X}$** alone impractical.

The theory also highlights that and learning effective connections between modalities via **generative models** can enhance multimodal learning. This forms the basis for our **Mixture of Noise Levels (MoNL)** approach, which is particularly suited for multimodal learning with **audio** and **video** data.

[1] Zhou Lu. *"A Theory of Multimodal Learning."* NeurIPS 2023.

$ $
### 2. Advantages of Mixture of Noise Level Training (MoNL)

Our **Mixture of Noise Level (MoNL)** training method offers significant benefits for multimodal learning, especially with **audio** and **video** data:

- **Heterogeneity and Connection**: Audio and video are naturally heterogeneous. For example, a video of a person speaking includes **audio** of spoken words and **video** of lip movements and facial expressions. MoNL uses **variable noise levels** to enhance learning by capturing the **generic transition matrix** across the **temporal axis**.
$$p _{\mathbf{\theta}}([\mathbf{z} _{t ^{(1,1)}-1} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}-1} ^{(M, N)}] \mid[\mathbf{z} _{t ^{(1,1)}} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}} ^{(M, N)}]) \qquad \text{(Eq. (4))}$$
where $M$, $N$ are the number of modalities and time-segments, respectively.

- **Enhanced Connectivity**: MoNL improves **connectivity** between **audio** and **video** modalities. Our experiments show that MoNL often surpasses **task-specific learning** approaches by fostering better connections between modalities, adapting its focus more effectively.

$ $
### 3. Enhanced Connectivity - Comparison with Existing Methods

- **MoNL vs. Joint Learning in MMD** [2]: Unlike joint learning methods that focus on the joint distribution $p_{\mathbf{\theta}}(\mathbf{z} _{t-1} \mid\mathbf{z} _{t})$, MoNL trains across **multiple conditioning**, enabling better connections by varying its focus. This is evidenced by MoNL outperforming the Vanilla (see Table 1) and MMD models (see Tables 2 and 3).

- **MoNL vs. Per-Modality Training**: MoNL goes beyond per-modality training in UniDiffuser [3], which uses variable noise between modalities i.e., learning $p_{\mathbf{\theta}}([\mathbf{z} _{t ^{(1)}-1} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}-1} ^{(M)}] \mid[\mathbf{z} _{t ^{(1)}} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}} ^{(M)}])$. MoNL introduces variable noise across different **time segments**, learning connections across **temporal dynamics** as well. This advantage is demonstrated in Table 1.

- **MoNL vs. Masked Training** [4]: Diffusion models often obscure high-frequency details with low noise and low-frequency structures with high noise [5]. MoNL employs variable noise levels to explore diverse **frequency components**, enhancing the model's ability to correlate high and low-frequency elements. This is in contrast to masked self-supervised learning, which limits frequency-specific connections by masking entire elements.

[2] Ruan et al. *"MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation."* CVPR 2023.

[3] Bao et al. *"One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale."* ICML 2023.

[4] Voleti et al. *"MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation."* NeurIPS 2022.

[5] Sander Dieleman. *"Noise Schedules Considered Harmful."* [Link](https://sander.ai/2024/06/14/noise-schedules).

$ $
### Conclusion

In summary, the effectiveness of **MoNL** for **multimodal diffusion models**, particularly with **audio** and **video** data, stems from its strategic use of **connection** and **heterogeneity**. By applying **variable noise levels**, MoNL enhances **connectivity** between modalities and better adapts to diverse **temporal** and **frequency components**, leading to superior performance compared to existing multimodal learning methods.
## Theoretical Background on Mixture of Noise Levels
$ $
### 1. Theoretical Background on Multimodal Learning

In *"A Theory of Multimodal Learning"* [1], multimodal learning is shown to offer a superior **generalization bound** compared to unimodal learning, with an improvement factor of **$O(\sqrt{n})$**, where **$n$** denotes the sample size. This benefit relies on **connection** and **heterogeneity** between modalities:

- **Connection**: The bound depends on learned connections between (**$\mathcal{X}$**) and (**$\mathcal{Y}$**).
- **Heterogeneity**: Describes how modalities, **$\mathcal{X}$** and **$\mathcal{Y}$**,  diverge and complement.

If connection and heterogeneity are missing, ill-conditioned scenarios can arise. For instance, if **$x \equiv y$**, perfect connection suggests no need for learning about **$\mathcal{Y}$**. On the other hand, if **$x$** is random noise, there is heterogeneity but no meaningful connection between **$\mathcal{X}$** and **$\mathcal{Y}$**, making non-trivial learning on **$\mathcal{X}$** alone impractical.

The theory also highlights that and learning effective connections between modalities via **generative models** can enhance multimodal learning. This forms the basis for our **Mixture of Noise Levels (MoNL)** approach, which is particularly suited for multimodal learning with **audio** and **video** data.

[1] Zhou Lu. *"A Theory of Multimodal Learning."* NeurIPS 2023.

$ $
### 2. Advantages of Mixture of Noise Level Training (MoNL)

Our **Mixture of Noise Level (MoNL)** training method offers significant benefits for multimodal learning, especially with **audio** and **video** data:

- **Heterogeneity and Connection**: Audio and video are naturally heterogeneous. For example, a video of a person speaking includes **audio** of spoken words and **video** of lip movements and facial expressions. MoNL uses **variable noise levels** to enhance learning by capturing the **generic transition matrix** across the **temporal axis**.
$$p _{\mathbf{\theta}}([\mathbf{z} _{t ^{(1,1)}-1} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}-1} ^{(M, N)}] \mid[\mathbf{z} _{t ^{(1,1)}} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}} ^{(M, N)}]) \qquad \text{(Eq. (4))}$$
where $M$, $N$ are the number of modalities and time-segments, respectively.

- **Enhanced Connectivity**: MoNL improves **connectivity** between **audio** and **video** modalities. Our experiments show that MoNL often surpasses **task-specific learning** approaches by fostering better connections between modalities, adapting its focus more effectively.

$ $
### 3. Enhanced Connectivity - Comparison with Existing Methods

- **MoNL vs. Joint Learning in MMD** [2]: Unlike joint learning methods that focus on the joint distribution $p_{\mathbf{\theta}}(\mathbf{z} _{t-1} \mid\mathbf{z} _{t})$, MoNL trains across **multiple conditioning**, enabling better connections by varying its focus. This is evidenced by MoNL outperforming the Vanilla (see Table 1) and MMD models (see Tables 2 and 3).

- **MoNL vs. Per-Modality Training**: MoNL goes beyond per-modality training in UniDiffuser [3], which uses variable noise between modalities i.e., learning $p_{\mathbf{\theta}}([\mathbf{z} _{t ^{(1)}-1} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}-1} ^{(M)}] \mid[\mathbf{z} _{t ^{(1)}} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}} ^{(M)}])$. MoNL introduces variable noise across different **time segments**, learning connections across **temporal dynamics** as well. This advantage is demonstrated in Table 1.

- **MoNL vs. Masked Training** [4]: Diffusion models often obscure high-frequency details with low noise and low-frequency structures with high noise [5]. MoNL employs variable noise levels to explore diverse **frequency components**, enhancing the model's ability to correlate high and low-frequency elements. This is in contrast to masked self-supervised learning, which limits frequency-specific connections by masking entire elements.

[2] Ruan et al. *"MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation."* CVPR 2023.

[3] Bao et al. *"One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale."* ICML 2023.

[4] Voleti et al. *"MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation."* NeurIPS 2022.

[5] Sander Dieleman. *"Noise Schedules Considered Harmful."* [Link](https://sander.ai/2024/06/14/noise-schedules).

$ $
### Conclusion

In summary, the effectiveness of **MoNL** for **multimodal diffusion models**, particularly with **audio** and **video** data, stems from its strategic use of **connection** and **heterogeneity**. By applying **variable noise levels**, MoNL enhances **connectivity** between modalities and better adapts to diverse **temporal** and **frequency components**, leading to superior performance compared to existing multimodal learning methods.
I appreciate the author's efforts in preparing the rebuttal. 
After reading the rebuttal, I think some of my concerns related to the experiments are clarified. 
On the other hand, this is a rather empirical paper, and I cannot always find the underlying theoretical intuition of this entire line of work. However, maybe the brute-force scaling up can overpass the underlying theoretical justification over the challenge of learning disjoint distributions without much understanding of the data distribution and it seems to work fine.
Anyway, I raise my score to 5 for post-rebuttal.
Thanks for the response. I have read the response as well as the other reviews.

I have considered the author's rebuttal, but it does not provide new or compelling information that would change my evaluation. Therefore, I will maintain my current score.