A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation
Authors: Gwanghyun Kim
Alonso Martinez
Yu-Chuan Su
Brendan Jou
Jose Lezama
Agrim Gupta
Lijun Yu
Lu Jiang
Aren Jansen
Jacob C Walker
Krishna Somandepalli
Venue: NeurIPS 2024 poster (2024)
Keywords: multimodal diffusion
multimodal generation
diffusion timestep
latent diffusion models
Primary area: diffusion_based_models
TL;DR: We present a diffusion framework that can learn a wide range of conditional distributions within audiovisual data using a single model.

Abstract: Training diffusion models for audiovisual sequences allows for a range of generation tasks by learning conditional distributions of various input-output combinations of the two modalities. Nevertheless, this strategy often requires training a separate model for each task which is expensive. Here, we propose a novel training approach to effectively learn arbitrary conditional distributions in the audiovisual space. Our key contribution lies in how we parameterize the diffusion timestep in the forward diffusion process. Instead of the standard fixed diffusion timestep, we propose applying variable diffusion timesteps across the temporal dimension and across modalities of the inputs. This formulation offers flexibility to introduce variable noise levels for various portions of the input, hence the term mixture of noise levels. We propose a transformer-based audiovisual latent diffusion model and show that it can be trained in a task-agnostic fashion using our approach to enable a variety of audiovisual generation tasks at inference time. Experiments demonstrate the versatility of our method in tackling cross-modal and multimodal interpolation tasks in the audiovisual space. Notably, our proposed approach surpasses baselines in generating temporally and perceptually consistent samples conditioned on the input. Project page: neurips13025.github.io

A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation
Gwanghyun Kim 1..*
Alonso Martinez
Yu-Chuan Su²
Brendan Jou
José Lezama 2
Agrim Gupta
Lijun Yur.*
Lu Jiang 1.*
Aren Jansen²
Jacob Walker2
Krishna Somandepalli
Scoul National University
2 Google DeepMind
Stanford University *Carnegie Mellon University
Abstract
Training diffusion models for audiovisual sequences allows for range of genera- tion tasks by learning conditional distributions of various input-output combinations of the two modalities. Nevertheless, this strategy often requires training a separate model for each task which is expensive. Here, we propose a novel training approach effectively learn arbitrary conditional distributions in the audiovisual space. Our key contribution lies in how we parameterize the diffusion timestep in the forward diffusion process. Instead of the standard fixed diffusion timestep, we propose applying variable diffusion timesteps across the temporal dimension and across modalities of the inputs. This formulation offers flexibility to introduce variable noise levels for various portions of the input, hence the term mixture of noise levels. audiovisual latent diffusion model and show that
We propose transformer-basee it can be trained in task-agnostic fashion using our approach to enable variety of audiovisual generation tasks at inference time. Experiments demonstrate the versatility of our method in tackling cross-modal and multimodal interpolation tasks the audiovisual space. Notably, our proposed approach surpasses baselines in generating temporally and perceptually consistent samples conditioned on the input. Project page:
Introduction
Recent years have witnessed a remarkable surge in the development and exploration of multimodal diffusion models. Prominent examples include text-to-image (T2I) [37, 41, 54, 39], text-to-video (T2V) [17, 6, 12]. Despite notable advancements, generating sequences across multiple modalities, like video and audio, remains challenging and an open research area.
Introducing time axis to static data paves the way for diverse multimodal sequential tasks including cross-modal generation (e.g., audio-to- audio-to-video), multimodal interpolation, and audiovisual continuation as shown in Fig 1. Each task can be further divided based on various input-output combinations of the modalities, leading to number of conditional distributions. For example, with video data a N and audio data y of length N, The complexity of configurations grows with tasks like audiovisual (n.+1:N) (
(n.+1:N) Yyu
(1:ma) (1mme)). where n. is the input frame length used for yo -b0 (neNT) (neN)) (n=N.) (neN))), where Ar and yyo
continuation, p(z conditioning, and multimodal interpolation, p(arg p(arfmeN??) yy (
$ Work done while Google. Equal contribution. Corresponding author: Krishna Somandepalli (ksomanngoogle..com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Audio-to-video generation ue)
Video-to-aucloo generation piy, an)
Joint generation p(as,
Coaliia
Genarsted tout
Conditan
Gerersiadesto
Audiovisual continuation with variable input durations
Multimodal interpolation tasks with variable settings
(iw douet (1rs) linelty
Bin
M)
Deditn
Our Audiovisual Diffusion Transformer trained with Mixture of Noise Levels tackles diverse AV generation tasks in a single model: see avdit2024.ithhubio for video demos.
Figure I:
Condition
Cenerated
Condition
Generated output
Figure 2: Comparing conditional inference AV-continuation for MM-Diffusion (left) and Ours (right) on Landscape dataset. Our approach excels at generating temporally consistent sequences.
N2 are input index sets. Training separate models for each variation is expensive and impractical. A more efficient training approach would be to learn these conditional distributions in single model without explicitly enumerating them, i.e., in a task-agnostic manner.
Unconditional diffusion models like MM-Diffusion [40] show potential for learning conditional distributions implicitly, but rely on inference adjustments [18, 40]. This limits performance, as seen in MM-Diffusion's struggle to generate temporally consistent sequences (see Fig. 2). While UniDiffuser [3] and Versatile Diffusion [51] offer methods for joint and conditional text-image distributions, effectively capturing temporal dynamics of audio and video remains open challenge.
Here, we propose a multimodal diffusion framework that empowers single model to learn diverse conditional distributions. This paves the way for versatile framework for multimodal diffusion, tackling various generation tasks. Our core idea is that, applying variable noise levels across modalities and time segments enables a single model to learn arbitrary conditional distributions. This formulation offers flexibility to train diffusion models with a mixture of nosse levels i.e., MoNL, which introduces variable noise levels across various portions of the input. It has number of advantages over previous approaches: requires minimal modifications the original denoising objective simplifying implementation, task-agnostic training, and support for conditional inference of a given task specification without any inference-time modifications.
We apply this approach for audiovisual generation by developing a diffusion transformer, AVDIT. To address the computational complexity of high-dimensional audio and video signals, we implement MoNL in the low-dimensional latent space learned by the MAGVIT-v2 [55] for video and the SoundStream [57] for audio. Importantly, the temporal structure in these latent representations enables to apply variable noise levels. We also introduce a transformer-based network for joint noise prediction. Transformers are natural choice our implemennttion due to their proficiency to model multimodal data [25, 10] capturing complex temporal and cross-modal relationships.
We assess the capability of MoNL model various distributions the audiovisual space by evaluating cross-modal tasks laudio-to-viin tovieeoo and video-to-audio generation), and conditioning on small portions (audiovisual continuation and interpolation tasks), For these tasks, we show that the AVDiT trained
We use the term, "time segment" reference a single unit time timension of inputs (e.g., framee video) during forward diffusion. Whereas, "timestpp" "diffusion timestep' sefers single stop process of adding during forward diffusion process.

---

with MoNL outperforms conventional methods including unconditional and conditional generation models, demonstrating the versatility of our task-agnostic framework as shown in Fig. Notably, qualitative and quantitative evaluations highlight the ability of our framework to generate temporally consistent sequences, as illustrated in Fig.
2 Background
Diffusion Models for Multivariate series data: Consider an example of a video diffusion model where the input sequence of image frames. In general, this task is modeling multivariate series data (i.c., image representations) of d-dimensions with N elements (no. of image frames), henceforth referred as time-segments. Thus, the multivariate series data, 2o z R q(20) be represented sequence of time-segments, where zi R is the n-th time-segment and d-dimensional representation.
During the forward process of diffusion models [44, 16], the original data zo is corrupted by gradually injecting noise in a sequence of timesteps. The noisy data at at time can be written as a; 2; Vā + vI CgEg. Here, Eg e!:N (O I) is Gaussian noise injected to the sequence and Be is the noise schedule, a₁ = 1 Be controls the noise level at each step with 02 I 0g. Each noisy time-segment can be represented as x V v1 aye During the reverse process, the data is sampled through a chain of reversing the transition kernel g( (x)) that is estimated by pe (2t--1/z)) (x)) = [/(xx-i//(( t), of I), where u(x, t) = var( VI-a. Eg(ax, xr, t)).
1-a,
The training objective is to learn a residual denoiser eg at each step as:
min wo [ E
where (((1, T}) is the diffusion timestep.
Unconditional joint generation (generating all modalities simulta- neously) and conditional generation (generating modality conditioned used for multimodal diffusion. Typically, separate models are trained for each task as described below:
Multimodal Diffusion Models:
on the rest) are commonly
Diffusion models for joint generation. For simplicity, let us assume two modalities æg. yo- The objective in joint generation is model the joint data distribution, denoted as (oo yo). To learn this, a joint noise prediction network, denoted as Eg is defined by rewriting Eq. as follows:
min E, Lr po.e.. [ yr. fe2.c,ll2..
(2)
where (Fa, Bo) is random data point, [,] denotes concatenation, Cz. Cy ~ (),, and U([1, 2, T)). Diffusion models trained with this objective can perform conditional sampling g(30))30) using inference-time tricks [18, 40].
Conditional training of diffusion models. learn conditional distributions, expressed as g( noise prediction network Eg conditioned on yo is adopted from Eq. 2: min 0o. l (xc, (er yo. 1) - <x.
0
Separate conditional models need to be trained for every pair of modalities and input configurations.
3 Mixture of Noise Levels (MoNL)
We introduce a novel framework for learning a wide range of conditional distributions within multimodal data by using a mixture of noise levels. The key idea is to formulate the timestep (Eq. 1) that determines a noise level in the forward diffusion as a vector. Then, we present representative strategies for variable noise levels. We then show how conditional inference can be performed without additional training. Finally, assembling all these components, we present our versatile audiovisual diffusion transformer (AVDiT).
3.1 Variable Noise Levels across Modality and Time
Formally, let M represent the number of modalities with sequence representations (latent spaces or raw data). Without loss of generality, assume the representations in each modality have N time-
3

---

Rolil Me.i!
Diffusion timestep
Van Vanilll
Noisy data ee
Predicted noise
modelity
Imput data
0.00 (00.0)
Joint noise prediction network
Forward Diffusion Process
Per -ine-al Pti
Ed
general
Pez cine-slice and pPn)
Noss (0.
pe pell-gtti
(MG-))
(3.1) (aL.W
by minimizing loss:
[ea([--rr (l.2) v. ef
(b) Mixture of noise levels (MoNL)
(a) Training with variable noise levels
Figure 3: Overview of (a) diffusion training with variable noise levels per time-segment and per modalities, and (b) the mixture of noise levels. Intensity of the color is used to indicate variable noise levels applied to multimodal input. The original input data žn consists of M modalities and N time-segments. 2g then perturbed with noise per noise level determined by a diffusion timestep vector & create noisy data 2s, which is input to the noise prediction network.
Insurig
Minosallies
Genwrate wich time-sice
segnents
Theiiee segments
Devoiinn
diop
Pe (i
(1 (1M +MM1
(a) Conditioning acress modalities: cross-modal generation
(b) Conditioning across time segments: multimodal Interpolation
Figure 4: Illustration of the conditional inference our framework for (a) cross-modal generation and (b) multimodal interpolation.
segments*. Let further assume they have the embedding dimension d (which in practice can be achieved by projecting the noisy input from each modality to the desired dimension). The
entire sequence then be simplified 20 Mxdd 20 denotes the n-th time-segment of m-th modality. For reference, Sec. 2 represents two modalities of multivariate series data, w1:N and y using this notation =0 (1,1:N) and =
(I:M,l:N) g(*0). where 2n (m..) Rd
(2,1:N) respectively.
We posit that training single model support learning arbitrary conditional distributions can be realized by using variable noise levels for each modality m and time segment n of the input space z0- We introduce the diffusion timestep vector as = (1:M, MxN to match the dimensionality of the multimodal inputs, where each element f(m.n) e [1, T] determines the timestep. and in turn the level of noise added to the corresponding element =0
(m,n) of the input Za-
Recall (from Sec. that unimodal case, the goal was learn the transition kernel q( (xx)) parameterized by po(a; ( = (xx ((ce-i//ll 1l1/,,,,, ()), of I). Analogously, by introducing a timestep vector E IR MgMxX XN our goal is now to learn a general transition matrix between various modalities and time-segments in at each step:
(1.1) Po pe(1 20 H (M -1 (0.1)) (
(1.1)
(M.N)
(M.N)
La practice, this is rarely video and audio representations, the embedding dimension and temporal compression in the data or latent spaces cun be vasily differentt However, what we propose here can be generalized by keeping track of the frame level correspondences between modalitios

---

space
P
Layer
Latent space
Iaient
Scole
Boole Skitt
Denoising vie AVDIT trained with MoNL
Scale
Meld-
Layer nomm
Genoration
ppantctemponal
pouttional
Terpont postionall
AGUEN
Denoising AVCIT trained
Embectioglayee
Emteeddng
Emberstngleye
Noiey video latent
Noley audio lotent
Diffusion timestep
Latent diffusion with mixture of noise levels (MoNL) and
(b) Audio-video diffusion transformer (AVDIT)
audiovisual diffusion transformer (AVDIT)
Figure 5: Schematic of (a) the proposed approach, and (b) AV-transformer for joint noise prediction.
Then, for diffusion training, we draw a Gaussian noise sequence e c(1:M,1-N), Each noise
element ((m.)) is then added to the corresponding element of the original data 20 (m.n) with noise level determined by ((m,.) as follows:
(m)))
(m,n)
VI dgem (m,n)
(mm,n))
V ag(m.))
Then, the joint and conditional training objectives in Eqs. 2 and 3 can be generalized with single noise prediction objective to learn the joint distribution Eg as follows:
min F4, El (1.11) ([=
( t) E|
where 2u ((zu) is the multimodal input and is the diffusion timestep vector.
3.2 Representative Stratgies for Variable Noise Levels
Using the generalized view of multimodal noise prediction described in Eq. 6, we now examine various strategies for variable noise levels during the forward diffusion. One can imagine an arbitrarily large number of timestep candidates in the vector space of drawn as functions of time-segments of the multivariate series and modalities. Here, we explore four designs create a mixture of noise levels as illustrated in Fig. 3(b). Let us assume we have final diffusion timestep vector for training, (1.5)) ty e R M where each element "d
sampled from ((11, 2.... T}),
Vanilla: Same timestep is assigned to all the time-segments and modalities. This is analogous to performing joint learning as ((m.a) Lgr and would the straightforward way extend the vanilla distillation approach for the multimodal case.
Per Modality (Pm): Variable timesteps are assigned for each modality, but all time-segments in given
modality have the same timestep as (m.n) (on.1) This is expected to promote cross-modal generation tasks. This is generalization of the UniDiffuser' [3] approach for sequences.
Per Time- segment (Pt): Variable timesteps are assigned as g(m,,)) (1.m) by keeping track of the corresponding semeenegments across modalities. Intuitively, this should promote better temporal consistency. Per Lime-segment and Per-modality (Ptm): Variable timesteps assigned for each time-segment
and modality f(m.)) ffm..) This would promote better temporal correspondence between modalities.
To enable learning wide range of conditional distributions, we create training paradigm where a timestep is uniformly randomly selected from the mixture. Specifically, we refer this training paradigm MoNL. A schematic of the overall training process is depicted in Fig. 3(a), with related pseudocodes Algorithms and the Appendix. Also, theoretical background on MoNL. is detailed in Appendix G.
5

---

3.3 Conditional Inference
Once the general transition kernel pe learned in Eq. 4, we investigate the model's ability to handle arbitrary conditional distributions. We achieve this by selectively injecting inputs during inference based on the task specification, i.e., clean (no noise) inputs for conditional portions with g(m.n) = 0, and noisy imputs for generating desired portions of the input with the current diffusion step ((m,m)) = t.
Consider the case of cross-modal generation (Fig. 4(a)), to generate sequence of M mm modalities conditioned on m. E (1, M) modalities, we set timestep elements of M me modalities as and those of me conditioning modalities as O, which achieves:
po (z, (m++ (MM.11MMMNNN) (mo+ +IMMM 200 (1))
Similarly, for multimodal interpolation (Fig. 4(b)), to generate N - ne time-segments of all modalities jointly, conditioned ne N1 time-segments, we set the timestep for the N - The time-segments as t, and for the conditioning n. time-segments as O, which achieves po(2;-1 | (1:M, I:N) =0
(l:M,l:me)). Unconditional joint generation is also possi-
ble by setting each timestep as the same 1, to estimate the transition kernel, po(=;-1 |, Intuitively, our mixture of noise levels is analogous to self-supervised learning which bypasses the need for predefined tasks during training but enables a deeper understanding of multimodal temporal relationships. See also Sec. for the discussion on classifier-free guidance in the Appendix.
4 Audiovisual Latent Diffusion Transformer (AVDiT)
Our model consists of two key components: (1) latent space representations from audio and video autoencoders, and (2) Audiovisual diffusion transformer (AVDiT) for joint noise prediction.
Latent Space Representations: For a video of 1 + L, frames, represented as v R(I+..)xxxXV we use MAGVIT-v2 [55], a causal autoencoder to achieve efficient spatial and temporal compression. MAGVIT-v2 results in a low-dimensional representation, Do IR g(I+t))xxxxxd, by a compression factor of r, H W in space and rt. To in time. Crucially, the use of causal 3D convolutions ensures that the embedding for a given frame is solely influenced by preceding frames, preventing flickering artifacts common in frame-level autoencoders.
Lu
w
For audio with La frames, autoencoder. We use the latents yo E Rlaxa xda prior to quantization as audio latents, a compression Le in time. The time-segments in our formulation refer to the ly and la temporal rate of rt., 1s dimensions in the video and audio latent spaces respectively.
RL we use SoundStream [57]. state-of-the-art neural audio
Audiovisual Transformer for Joint Noise Prediction: Transformers [48] natural fit for multimodal generation as they can: (1) efficiently integrate multiple modalities and their interac- tions [58, 10]. (2) capture intricate spatiotemporal dependencies [8. 51, and (3) have shown impressive video generation capabilities [12, 25]. Inspired by these benefits, we introduce AVDiT, a noise prediction network for latent diffusion as described in Fig. 5. AVDIT utilizes the timestep embedding similar to the condition signal used in W.A.I..T [12]. The Transformer first processes the timestep embeddings and positional encodings to create an embedding of the timestep vector. This embedding serves as a conditioning signal and is utilized to dynamically calculate the scaling and shifting param- eters for AdaLN during the Transformer Layer Normalization step. This enables the normalization to incorporate the conditioning information of variable noise levels. We first consider the 1, and 1 + by dime-dinnssoos for audio and video embeddings respectively. When applying MoNL, we can easily keep track of the corresponding time segments among the la and 1 + Lu dimensions, given the temporal compression factors each modality. The noisy latents are then linearly projected matching the final dimension d by adding appropriate spatiotemporal positional embeddings for video and temporal positional embeddings for audio, resulting in d dimensional embeddings for each modality which are then concatenated.

---

Condition
Geternted
Condition
Denernted
AV continustion for seconds
Condition
Denerated curput
Candition (b) Audio-conditioned video continuation
Cond
Oenorated DUSOU!
Cordinto on
Generated BuSzut
(c) Multimooa interpolation
Figure 6: Full length examples of AVDIT trained with MoNL the Monologue dataset. Samples were generated from unseen conditions at 8fps at 128x 128 and are shown at the same rate.
Viloo
Condition
Generated
Condition
Ganernted
Austo
Figure 7: Unlike MM-Diffusion (left) where clothes and appearance is altered in the continuation (red arrow), our AVDiT with MoNL (right) maintains subject consistency in the + dataset.
Related Work
Video diffusion models. Diffusion models have revolutionized image [45, 35, 16, 39] and video generation with pixel-space [18, 17, 43] and latent-space [14, 56, 6, 9, 12] approaches. Recently, W.A.L.T [12] pushed the boundaries using transformer- baseed latent diffusion with joint image-video training. Tackling diverse audio-video generation tasks remains largely unexplored. With an AVDiT trained with MoNL, our unified approach empowers single model to handle range of tasks. Audio generative models. Audio generation soared with WaveNet [36]'s ss autoregressive approach. Adversarial audio generation [27, 42, 26] emerged. Combining this with differentiable quantiza- tion [38, 47, 1] led to end-to-end neural codecs for efficient audio compression [21, 57]. Recently, diffusion models joined the fray, discrete space [52]. Our AVDIT uses continuous embeddings from SoundStream for audio latents.
using continuous latent spaces [29, 19, 11], others exploring
Multimodal generative modeling, While multimodal diffusion models [37, 41, 54, 39, 17, 23] have achieved impressive results, the field has primarily focused on the visual domain and audiovisual generation remains less explored. Existing approaches for audio-to-video [53, 53, 30] and video-to- audio [20, 33] generation typically learn task-specific conditional models, limiting their flexibility, To address this, recent works [40, 50] propose more versatile audiovisual models. However, they did not examine multimodal interpolation tasks, which we explore in this work. Tasks such as AV-oontinuttion critical understand model's capability to generate temporally consistent multimodal sequence retaining the object consistency from the condition imput.
6 Experiments
6.1 Datasets
Monologues dataset consists of 19.1 million videos for training and 25K videos for evaluation, each with a single person talking. The videos are center-cropped to a 256 256 resolution. This dataset includes a range of person appearances along with rich verbal and non-verbal communication cues. This dataset ideally situated to assess concepts such as audiovisual gestural synchrony and multimodal expressions which key components of human communication and interactions.

---

Table 1: Comparison of AVDiT trained with mixture of noise levels (MoNL) on the Monologues dataset for unconditional joint generation (Joint), cross-modal (A2V, V2A) and multimodal interpolation (AV- -i-inpaint, AV-cont inue) tasks. FAD- = 2.7 and FVD = 3.3 for groundtruth autoencoder reconstructions of the inputs.
Fréchet metries estimated with N=25k.
Setting/ Task
Joint
A2V
V2A
AV-
inpaint
AV
W--continue
Average
FAD |
FVD
FVD L
BAD
FAD
FVD
FAD
FVD L
FAD
FVD
Conditional (task- specific)
7.1
63.6
49.4
11.5
5.3
15.9
7.4
12.1
7.8
35.3
Per modality
7.0
84.4
34,1
4.7
6.2
213.6
4.5
92.1
5.6
106.
Vanilla
7.1
63.6
53.3
8.1
226.8
140.8
7.4
121.1
MoNL (Ours)
6.4
77.6
40.2
5.3
4.6
11.8
3.1
8.8
4.9
34,6
Ablations
time-segment
6.6
96.3
124.5
12.1
5.1
28.2
5.0
72.3
7.2
80.3
Per time-segment Per modality
7.0
84.5
52.5
5,9
5,4
22.9
4.8
61.2
5.7
55.3
Pt.Pnn/Pe PPm
9.0
90.1
43.1
5.1
5.2
13.4
4.1
16.9
5.9
40.9
Table 2: Quantitative comparison between our AVDiT with MoNL and MM-Diffusion (MMD).
Table 3: User study of comparison
between our model and MM-Diffusion
A
Landscape
(MMD) the AIST+ dataset.
Tisk
Method
FAD
FVD | KVD
FAD
FVD.
KVD
AV align
Preference of ours over MMD
Reconstruction
0.90
11.72
0.96
0.76
16.41
-0.25
0.60
AV align
AV quality
Person consiseency
MMD
184.45
33.91
238.33
15.14
0.54
A2V
Ours
38.04
5.27
86.79
4.30
0.57
continue A2V
0.77
0.61
0.75
V28
MMD
Ours
13.30
1.I1
13.60
0.78
0.50
0.51
Joint
V2A
0.74
0.61
0.72
0.49
0.81
0.60
+IST+ is a subset of AIST [46] and contains 1,020 street dance vidcos (5.2 hours). The videos
were segmented into in 8,233 samples for train and 110 for test at 10fps following Ruan al. [40]. Landscape contains natural scenes from 928 [28] videos which were segemented into 5,400 samples for train and 600 samples for test at 10fps. We conduct most of the experiments on the Monologues due to its size and diversity. We use AIST++ and Landscape for comparison with MM-Diffusion.
6.2 Evaluation Settings
Tasks. We study three sets of tasks: (1) Joint audio-video (AV) generation (Joint): (2) Cross-modal generation: Audio-to-video (A2V) and Video-to-audio (V2A), and (3) AV interpolation generative tasks: AV-inpaint where a 1.5s clip is interpolated given one video frame, and 125s audio the beginning and four video frames and 0.5s audio at the end, and AV-continue to fill out 1.5 seconds of AV given the first 5 video frames and corresponding 0.625s of audio
Baselines. On the Monologues dataset, we compare the performance of AVDIT trained with MoNL versus three baselines: Vanilla (Eq. 2), Conditional models separately trained for each task, and the per modality model (Sec 3.2) which may be considered as generalization of the UniDiffuser approach for sequences. We enabled the Vanilla model to generate cross-modal and multimodal interpolation outputs by using the replacement method [18, 40]. We also benchmark MoNL AVDiT against UNct-based MM-Diffusion (MMD) [40], the sole published work with a released model that tackles both audio and video generation within single model. While direct comparison between U-Nets and our transformer architecture is inherently challenging due to their distinct design principles, we show that MoNL AVDIT surpasses this strong U-Net baseline, demonstrating the effectiveness of the transformer architecture in this domain. We restrict our quantitative evaluation to A2V and V2A tasks because MMD fails to generate temporally consistent sequences in case of continuation tasks (see Figs. 2 and 7 for example).
Quantitative evaluation. We use Fréchet video distance (FVD) as our video evaluation metric following Yu et al. [55]. Similarly, we use Fréchet audio distance (FAD) as the audio evaluation metric following Ruan al. [40]. Because we use latent space representations for the video and audio, we also report the FVD and FAD scores between reconstructed signal and the original signal as "ground-truth trutt"" scores as the performance upper bound. While we preferred user studies for assessing mudio-video alignment as existing metrics miss subtle synchrony like dance moves matching music beats or gestures aligning with speech patterns, we computed AV-align score [33], limiting them to the open- open-domant domain Landscape dataset for the comparison with MMD.
User studies. We conducted user studies to evaluate the quality of generated content. We adopt the two axes of measurement introduced by Ruan et al. [40] namely audio/video quality and audio-video alignment, and introduce a third one, "subject consistency" to assess whether the person in the generated content is plausibly consistent with the input. For stimuli, we used total of 360 generated
8

---

samples (not cherry picked) balanced across A2V, V2A, AV-continue and AV-inpaint tasks and for AVDiT trained with three upproaches: MoNL, Vamilla and Per modality on Monologues dataset. The tasks were assessed on a 5-point Likert scale. We also compared rater preference for MoNI. AVDIT vs. MMD with 30 videos and 5 raters per video. Raters were presented with generations from the two methods randomized as two options, A/B and were asked to pick one option for each of the three dimensions instead of using a 5- 5-point scale. See more details on implementation and experimental setup in Secs. B and C in the Appendix.
6.3 Results
Qualitative results. As displayed in Figs. and 6, Our AVDIT model trained with MoNL achieves impressive performance on various tasks within single framework, including audio-to-video, video- to-audio, joint generation, multimodal continuation and interpolation with flexible input settings, generating temporally consistent videos. Notably, ours preserves clothing and appearance attributes during continuation tasks, unlike MMD which can alter these (see Figs. 2 and 7). More qualitative results and comparisons are available Figs. 13, 14 and 15, and at avdit2024. g.thuubbg..bbbo...
Quantitative results. As shown in Table 1, on average across all tasks, AVDiT trained with MoNL outperforms all baselines, demonstrating its versatility learn diverse conditional distributions in task-agnostic manner. MoNL. excelled at generating samples that are temporally and perceptually consistent with the conditioning input, in the case of AV-inpaint and AV-continue tasks, where other baselines generally failed. Per- med-mttyy approach surpassed MoNL for A2V and V2A tasks consistent with the findings in Bao et al. [3] likely because conditional distributions in these cases only need to capture cross-modal associations and not necessarily the underlying temporal dynamics. Unsurprisingly, the vanilla diffusion model trained for joint generation exhibited superior performance in this specific scenario but served as a lower-bound of performance for all other tasks. Finally, MoNL performed better than (if not on-par with) task-specific models for all conditional tasks.
As evident from Table 2, MoNL outperformed MMD in terms of the FAD and FVD metrics across all tasks on the AIST++ and Landscape datasets, as estimated using the code provided by Ruan et al. [40]. The significantly better audio generation in our model, likely due to the combination of MoNL and our choice of the SoundStream audio autoencoder, is also reflected in the ground- -truh FAD scores for audio reconstruction. In case of video reconstruction quality, (ground-trutt FVD) on AIST++, our choice of autoencoder was inferior to MMD, possibly due to the small dataset size. Qualitatively, we observed that the MAGVIT-v2 reconstructions eliminated flickering across frames but the reconstruction of small face regions in AIST+ dance videos was blurry. These findings should be interpreted cautiously due to several factors: the limited size of the AIST+ and Landscape training splits, our use of a transformer backbone versus MMD's coupled U-Nets, and our use of pretrained autoencoders for latent space representations. On the Landscape dataset, AV-align results demonstrate that our model achieves better alignment compared to MMD, which aligns with the findings from the user study below.
Venila
For modamadally (UnDiitsse))
MoNL((ur))
User studies. A comparison of the distribution of Lik- ert scores across all tasks for the three approaches we compared is shown in Fig. 8. Pairwise Mann- Mann-Whitney Whitney
Lo
U tests were conducted with Bonferroni correction
for multiple comparisons ence. Across all axes, raters preferred samples gener- ated from MoNL over that of Vanilla or Per-modality (Pm) approaches. Examining task-specific trends (see Fig. 12 in the Appendix). for the cross-modal tasks, Pm was rated significantly higher than Vanilla, and there was no significant difference between MoNL and Pm (except for the V2A task on AV alignment). For multimodal interpolation tasks, MoNL was rated
assess statistical differ-
AV al aiigmment Figure 8: Comparative analysis across AVDiT models from the user study on AV quality, AV alignment and person consistency. The * significantly higher than Pm. In line with quantitative indicates statistically significant pairwise dif- results, these results suggest that MoNL excelled at ference at p < 0.01 after multiple correction. generating samples, that are perceptually and temporally consistent with the input conditioning.
AV quelity
Person conerstency
As indicated in Table 3, our MoNI. AVDIT outperformed MMD in user studies, especially in consistency where ours showed improved consistency along factors such person's 's appearance

---

or the attire. MMD was preferred slightly more in V2A tasks, possibly because the Soundstream audtoencoder we used for audio was not optimized for music generation like in MMD.
Ablations. Recall that MoNI. training randomly selects one of the four timestep designs described in Sec. 3.2. We compare MoNL with Per time-segment Pt, and Per time-segment and Per modality Ptm and Pt/Pm/Ptm that excludes vanilla from the timestep mixture, approaches separately as shown in Table Overall, Ptm noise excelled inpainting and continuation tasks though it was not on par with per-modality approach for cross-modal tasks. In general, Pt does not perform well by itself. In our experiments, we also observed that the combination of Pm, Pt, Ptm and Pt/Pm/Ptan /PPtm was sufficient for comparable performance on most tasks except for unconditional joint generation. Adding the Vanil approach to the mixture of timesteps improved performance for unconditional joint generation while not substantially compromising the performance on other tasks.
7 Conclusion
We propose a unified approach for multimodal diffusion using a mixture of noise levels (MoNL) for generating and manipulating sequences across modalities and time. This empowers a single model to handle diverse tasks like audio-video continuation, interpolation, and cross-modal generation. We show that an audiovisual latent diffusion transformer (AVDIT) trained with MoNL achieves performance in audiovisual-seque sequence generation, providing new opportunities for expressive and controllable multimedia content creation.
See Sec. A in the Appendix for discussions on limitations and considerations.
Acknowledgements
We thank Kihyuk Sohn, Caroline Pantofaru and Brian Eoff for feedback on early versions of the manuscript. We thank Prof. Se Young Chun for helpful discussions and feedback the theoretical aspects of this work. Special thanks to Alex Siegman and Xuhui Jia for managing compute resources.
References
[1] Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Luc V Gool. Soft-to-hard vector quantization for end-to-end learning compressible representations. Advances in neural information processing systems, 30, 2017.
[2] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dmm: analytic estimate of the optimal reverse variance in diffusion probabilistic models. In ICLR, 2022.
[3] Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint arXiv:2303.06553 2023.
[4] Reza Bayat. study on sample diversity in generative models: Gans vs. diffusion models. 2023.
[5] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.051500 2020.
[6] Andreas Blattmann, Robin Rombach, Huan Ling. Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
[7] Sander Dieleman. Noise schedules considered harmful. https: http://///aadeeaaa//22///////// ai/2024/06/14/ peiisesschedule 2022.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image worth 16x 16 words: Transformers for image recognition at scale. International Conference Learning Representations, 2020.
10

---

[9] Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang. Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 22930-22900 22941, 2023.
[10] Mariana-Iuliann Georgescu, Eduardo Fonseca, Radu Tudor lonescu, Mario Lucic, Cordelia Schmid, and Anurag Arnab. Audiovisual masked autoencoders. In Proceedings the IEEE/CVF International Conference Computer Vision, pages 16144-16154, 2023.
[11] Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using instruction-tuned LLM and latent diffusion model. arXiv preprint:2304.13731, 2023.
[12] Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models, 2023.
[13] Andrey Guzhov, Federico Rauc, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976-980. IEEE, 2022.
[14] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.1322) 2023.
[15] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS, 2021.
[16] Jonathan Ho. Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[17] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv:2210.02303, 2022.
[18] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In ICLR Workshop Deep Generative Models for Highly Structured Data, 2022.
[19] Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-An-Audio: Text-to-audio generation with prompt-enhanced diffusion models. International Conference on Machine Learning, 2023.
[20] Vladimir lashin and Esa Rahtu. Taming visually guided sound generation. arXiv preprint 10.09921 2021.
[21] Srihari Kankanahalli, End-to-end optimized speech coding with deep neural networks. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2521 2525. IEEE, 2018
[22] Kevin Kilgour, Mauricio Zuluaga. Dominik Roblek, and Matthew Sharifi. Fri 'echet audio dis- tance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466 2018.
[23] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426-2435, 2022.
[24] Thomas Koelewijn, Adelbert Bronkhorst, and Jan Theeuwes. Attention and the multiple stages of multisensory integration: A review of audiovisual studies. Acta psychologica, 134(3): 372-384, 2010.
[25] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: large language model for zero-shot video generation. arXiv preprint 2xXiv:2312.14125 2023. 2023.
11

---

[26] Jungil Kong. Jachyeon Kim, and Jackyoung Bac. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems,331 13:12022-17033, 2020.
[27] Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre De Brebisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
[28] Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Chanyoung Kim, Won Jeong Ryoo, Sang Ho Yoon, Hyunjun Cho, Jibyun Bac, Jinkyu Kim, and Sangpil Kim. Sound-guie -uidddd semantic video generation. In ECCV, pages 34-50, 2022.
[29] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang. and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. International Conference on Machine Learning, 2023.
[30] Jiawei Liu, Weining Wang, Sihan Chen, Xinxin Zhu, and Jing Liu. Sounding video generator: A unified framework for text-guided sounding video generation. IEEE Transactions on Multimedia, 2023.
[31] Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[32] Zhou Lu. A theory of multimodal learning. Advances in Neural Information Processing Systems, 0..57244-572555 2023.
[33] Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video- to-audio synthesis with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[34] Kevin G Munhall, P Gribble, L Sacco, and M Ward. Temporal constraints on the megurk effect. Perception & psychophysics, 58:351-362, 1996.
[35] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
[36] Aaron van den Oord, Sander Dicleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint (rXiv:160900349/ 2016.
[37] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. ArXiv preprint, 2022.
[38] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with 2 Advances in neural information processing systems, 32, 2019.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[40] Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10219-10228, 2023.
[41] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022.
[42] Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari. Statistical parametric speech synthesis incorporating generative adversarial networks. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 26(1):84-96, 2017.
12

---

[43] Uriel Singer, Adam Polyak. Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, al. Make-a-video: Text-tovvideo generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
[44] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICLR, 2015.
[45] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[46] Shuhei Tsuchida. Satoru Fukayama, Masahiro Hamasaki, and Masataka Goto. AIST dance video database: Multi-genre, multi-dancer, and multi-camera database for dance information processing. ISMIR, 2019.
[47] Aaron Van Den Oord, Oriol Vinyals, et Neural discrete representation learning. In NeurIPS, 2017.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[49] Vikram Voleti, Alexia Malcoeur-Martineaut and Christopher Pal. Masked conditional video diffusion for prediction, generation, and interpolation. In NeurIPS, 2022.
[50] Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Sceing and hear- ing: Open-domain visual-audio generation with diffusion latent aligners. arXiv preprint :XXi:2442..17233 2024.
[51] Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7754-7765, 2023.
[52] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng. Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions Audio, Speech, and Language Processing, 31:1720-1733, 2023.
[53] Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to-video generation via text-to-video model adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6639-6647, 2024.
[54] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et Scaling autoregressive models for content-rich text-to-image generation. arXiv:2206. 10789, 2022.
[55] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats tiffusion-tokemmizer is key to visual generation. arXiv preprint arX1v:2310.05737, 2023.
[56] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18456-18466, 2023.
[57] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. JEEE/ACM Trans. on Audio, Speech, and Language Processing, 30:495-500 2021.
[58] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13041-13049, 2020.
13

---

Appendix
A Limitations, Impact and Considerations
Our proposed approach, combining mixture of noise levels (MoNL) with the generative capabilities of the Audiovisual diffusion transformer (AVDiT), has certain limitations. As shown with demo videos on avdir2024. githhbb.,, while our models effectively capture subject consistency and intricate nonverbal behaviors such as gestural synchrony with vocal tone, significant improvements are necessary to enhance visual and speech quality. Future research will concentrate on super-resolution systems to address visual quality, while text conditioning could potentially further optimize speech quality.
A key focus in this work was to demonstrate the versatility and use of MoNL across various tasks using simple mixing scheme by randomly choosing between different timestep candidates as representative schemes for applying variable noise levels. This simplicity showcases its broad applicability. We acknowledge that fine-grate fine-grained controlling by weighted mixing of the different schemes could be explored for specific goals or tasks in future work. In fact, one can imagine an arbitrarily large number of timestep candidates in the vector space of the inputs. We specifically chose simple mixture scheme to demonstrate its versatility as proof of concept, rather than optimize for any single task.
Although the method presented in this work is for general multimodal applications, our experi- ments included human-centrie generation tasks. This enabled us to explore unique challenges of that problem setting. For example, consider the case of perceptual expectations for audiovisual aligmment/coherence, where misaligned audio and visual cues can drastically alter perception of speech [34, 24]. Generation of photo-realistic persons, speech, and joint generation of both can perpetuate stereotypes. We recognize the ethical concerns and underscore that our goal here is to explore how understanding aspects such as nonverbal behavior in multimodal communications using generative models can open up new avenues in research.
Specifically, the A2V and V2A tasks in human-centric context, which involve extrapolating visual appearance from speech and vice versa, have the potential perpetuate stereotypes. The generated samples are derived from the model's understanding of cross-modal associations in the training dataset, which can be vastly different from human perception. One possible mitigation to ensure that the model can generate diverse outputs for given input. Diffusion models can achieve this by utilizing different noises at inference time, given a sufficiently large and diverse training dataset. A recent study showed that Diffusion models demonstrate better sample diversity in generations compared to GANs [4], however, addressing potential issues around mode collapse in the generative models, especially with multimodal data is an open research problem.
Our work also introduced "consistency" to qualitatively assess whether the generation remains congruent with the input conditioning, In continuation and interpolation tasks, the disparity between what the model generates and human perception can be generally minimal, as the conditioning provides a perceptual template for the subject's potential appearance or voice. In contrast, A2V and V2A tasks warrant an in-depth analysis of this disparity. As our immediate goal was assess the capability of our proposed approach (and baselines) to generate samples from various conditional distributions, we focused on a broad definition for measuring consistency in user studies. Our future work will focus on extending the consistency measure for a granular understanding of these biases by (1) comparing cross-modal associations in the training data that of the generated samples, and (2) disaggregate model and human evaluations in cross-modal generation tasks by identifying specific dimensions of human appearance attributes like perceived gender expression or human communication aspects such voice and gestural synchrony using diverse rater pools.
B Implementation Details
Autoencoders and AVDIT. Given the domain specific nature of the datasets. we trained dataset- specific MAGVIT-v2 autoencoders following Yu et al. [55]. For the Monologues dataset, we downsampled the data to 8fps and 128 x 128 resolution for video and 16kHz for audio and randomly
14

---

sampled contiguous clip of 2.125 second (17 frames) to match the input requirements of MAGVIT- v2. This resulted in dataset of about 11.8K hours for training. The spatial and temporal video compression ratios were set to = 8 and rt, 4, whereas the temporal audio compression ratio was rt., 320. The embedding dimension of the video and audio latent spaces are d, 8 and d = 1024 respectively, with the target embedding dimension after linear projection, d = 1024. All latents were zero-mean and unit-variance normalized with empirical mean and variance estimates on small subset. AVDIT has 24 transformer layers with 16 heads with MSA with a total of 420M parameters.
Diffusion training and inference. During training, we use linear noise variance schedule and a diffusion step T 1000, and a self- cofdiiiniiion rate 0.9 following Gupta et al. [12]. At the inference time, use 250 DDIM steps. All models were trained for about 400K steps with a batch size of 256. We used the A optimizer [31] with a learning rate of 5e-4, 5K warm-up steps, cosine learning rate scheduler and EMA consistent with the denoising transformer setting in Gupta et al. [12].
Compute resources. Each experiment listed in Table 1 was conducted using 256 v5e TPU chips (with 16x 16 topology) for training (on average, the models were trained for around 350K steps with a batch size of 256 for around five days); inference was conducted using 16 v5e TPU chips with topology of 4 4.4, See https://cloud. www.leoooooooooooooooooooooooooo httpee///o//p//c////eeeee for more details. Benchmarking experiments to compare with MM-Diffusion on the AIST+ and Landscape datasets were conducted using two A100 GPUs for conditional inference and estimating FAD/FVD metrics.
C Experimental Details
C.1 Evaluation metries
Since our primary use case is speech generation with the Monologues dataset, we use VGGish embeddings as feature for FAD estimation [22] for the results reported in Table 1. For AV interpolation generative tasks (AV-continue and AV-inpaint), we carefully excluded the conditioning AV frames while estimating Fréchet metrics.
C.2 Comparison with MM-Diffusion
In order conduct fair comparsion to the results reported in MM-Diffusion publication [40], the data preprocessing and evaluation code provided at FVD, FAD and KVD metrics. Note that the FAD computation here does not use VGGish embeddings. Instead, it uses AudioCLIP [13] which was trained for general sound classification tasks and not suitable for speech generation tasks in the Monologues dataset reported in Table 1.
Mrticnnm/MM-ifuuuiiii for
For the FAD, FVD and KVD results reported in Table 2, we match training conditions for the input image resolution and video FPS with Ruan et al. [40], i... 64 x 64 resolution images at 10fps. We match the duration of audio-video from both models to 2 seconds. For visualization purpose Fig. 2, also train our models with 256 x 256 resolution to match the super-resolution output resolution used by MM-Diffusion.
Ruan et al. [40] introduce method for implementing zero-shot transfer of A2V and V2A tasks, inspired by the reconstruction- guided sampling proposed by Ho et al. [18]. For instance, in V2A tasks, the generated noisy audio à, is computed at each step follows:
az 2t 0av (a.-1. 1441),
(8)
ā = 4: Avi 03 Va, - 110--0el: 0ell2
(9)
where ac+1, 0 N N-length sequence of generated noisy audio and conditioned noisy video at t + 1, 6ar is a parameterized denoising step, and À is a gradient weight. Similarly, the zero-shot transfer of AV-continuation task using the reconstruction-guidee sampling [18] can be described by the following equations:
at Vt lav(Ur+1, 1e+1);
(10)
+INN): a, a,
I:N)
AvI - α, V a.(
1/ - 01
(1mme)
(1mm))
2
+
a
(1:na)
- a. (1:m.)
(11)
fm,+ Mlo+ 1i
(no+1:N) Av1 - a. V
(1:n,) - 0/
(1
(1:me) _ a (1:mm)
+
(12)
15

---

Video to Audio
Audio Video Continuation
audio
audio
video
video
Figure 9: Example stimuli shown to the raters for the user study. We conducted user studies for four tasks, A2V, V2A, audiovisual continuation and multimodal interpolation tasks. One track each for the audio and video modality below the stimulus video were shown to effectively convey the portions that were generated (in green) and condition input (gray).
Table 4: Rater instructions for audio/video quality metric.
Score
Audio Video quality
Pure noise, completely UNRECOGNIZABLI CONTENT
2
The generated content has natural structure SOME places. but The generated content has natural structure MOST places
3
4
The generated content is NATURAL, BUT can be recognized GENERATED content The generated content is so NATURAL that it is indistinguishable from the REAL-WORLD
5
Table Rater instructions for audio-video alignment metric.
Audio-video alignment
The audio-video are total noise and are completely IRRELE EREEEEVANT
2
The generated content has CORRELATION between audio und video in SOME segments, but not most generated content has CORREL The generated content has NATURAL CORRELATION between the audio and video.
hetween
video MOST segments
BUT cun be recogmized as GENERATED content The generated content has CORRELATION between the audio video indistinguishable from BEAL-WORLD
Table 6: Rater instructions for subject consistency.
Score
Subject consistency
The person gencrated INCONSISTENT the input person generated is CONSISTENT content SOME SEGMENTS
The person generated CONSISTENT with generuted The person generated is NATURALISTIC, but be recognized GENERATED content The person generated is INDISTINGUISHABLE from the REAL-WORLD
in MOST SEGMENTS
4
5
where ne is the number conditioned time-segments. We closely follows their V2A codebase to faithfully execute the continuation task. We adopt À = 0.02 to prevent numerical instability, as the results tend diverge for À > 0.02.
C.3 Qualitative Evaluation
Examples of the video stimulus template shown to the raters is presented in Fig. 9. The rater instructions provided for each axis of quality, alignment and consistency shown in Tables 4, and 6 respectively.
16

---

Algorithm 1 Sampling of a diffusion timestep vector
1: function GETTIMESTEPVECCERRe type MoNL then type ((Vaanill,,, Pt, Pm, Ptm)
4: by R MMN 2(
== 0 MMN
6:
for m 1..... M do
7:
for v. 1, N do
if type Vamilla then
9:
f(m,n)
rd
(1.1)
10:
else type Pt then
11:
((,,)) lef
(1.)
12:
else type Pm then
13:
((,,)) (
14:
else type Ptm then
15:
(m..) f(m,n) tnf
16:
end for
17:
end for
18:
return t
19:
end function
Algorithm 2 Training with MoNL
input g(zo), E0. type (timestep sample type)
1: repeat
zo ((0))
E N(O,I)
4: t= BEITIMESTEPVEC(typp))
5: for m =1=1, m,...... M do
for
N do
7: [i..n
26 (m.
1 a) (m.
F,(m.n)
end for 9: end for
10: Take gradient step on
11:
Voleo([=(1..) (1.1) (..))...))...................... (,(M, N) t) - E
(M,NN
12: until converged
A Algorithm 3 Joint generation zu
1: 27 I. Nxd I),
2: for T 1 do
3: 6 RMxNxd N(O, I) if 1, else
4: t MMNN =I
5: 2--1=
2, Vi-a; t))
8.
t)) OrE
var
6: end for
7: return 20
Algorithm 4 Cross-modal generation of &n R(M-m.)xN conditioned on 2o E m Nxd
1: 27 RMM X N),,T)
2: t R RRMXN 0
3: for Ty..., do
else 0 4: 6 ((M-m))xNx0 NO0 I) if
5: (IM,.1.,.1:N) 71
6: = (mn+1:M.N) ([z0, t)
7: 2,-1 va. (2, VI-a,
8-
ê) + 0,E
8: end for
9: return &0
17

---

D Algorithms
The core algorithms for implementing mixture of noise levels (MoNL) are presented in three parts:
sampling of diffusion timestep vector (Algorithm 1), training process (Algorithm 2), and joint/cross- modal generation at inference time (Algorithms 3 and 4). The sampling algorithms are flexible, using DDPM [16] an example, and can be replaced with other efficient learning- frree samplers like DDIM [45] or Analytic-DPM [2]. Notably, conditional generation across time-segments simply change of axis in Algorithm 4.
E
Gratis Classifier-Free Guidance
Classifier-free guidance (CFG) [15], a technique designed to enhance the quality of samples pro- duced by conditional diffusion models using linear combination of the conditional and unconditional outputs as follows:
Eg (e: Yo. t) (1 + 8) S) yo. sco (I).
(13)
= (1 + 8) c) comd
sCo
uncomd
(14)
where is guidance scale and the conditional and unconditional outputs are denoted by e and uncond respectively. Typically, a null token Ø, is used to allow the conditional model to generate unconditional outputs by setting yo = Ø.
60
Gratis CFG. CFG (Eq. 13) is supported in our framework at inference without any additional training similar to UniDiffuser [3]. Instead of using a null token for generating unconditional outputs (emrcodd in Eq. 13), Gaussian noise is injected to the conditional portions input per task specification, and setting ((m.a) = T. Conditional outputs e are obtained as illustrated in Fig. 4. Our vector formulation of the timestep allows us to apply varying levels of noise to different parts of the input. This opens number of possibilities for constructing various CFG forms by emphasizing different time segments or modalities, depending on the task hand.
MoNL supports classificr- free guidance (CFG) without requiring additional design. Unlike the original CFG (see Eq. 13), it does not need a null token either, hence gratis or free. This is achieved by injecting Gaussian noise to the conditional portions of the multimodal space and setting ((mm.) T for the output as illustrated Fig. 10 for the case cross-modal generation of audio-in, video-out. To illustrate mCFG, consider the conditional output of the network in the cross-modal task (see Eq. 7),
denote term used in the gradient step as Z (1:M,::N) [2 2.0,,)] and conditional portions as cond
co (Z, (I:M,I:N) t) where
z (1:m..1:N)
(1:)
Z (mctM. =2: (m.+1:M,..))
20
(IN)
( =t
Then, the output for the cross-modal generation task is: co (Z( (1:M,::N) t).
uncond co
(15)
where
Z (:N)
Z +1M,,NN) (1M,.NN)
20
2
{ 1:N)
(mM+1:M,1:))
where ht (O, I). Then, mCFG operates by blending the conditional e and unconditional
portions ey uncond per task specification as follows:
čo (1+ 5)
seg
uncond
(16)
where is a guidance scale.
E (I.M +:N) (1 s)eg
se
(I:M
(17)
By formulating the timestep as a vector, we can apply varying levels of noise to different input components. This unlocks diverse possibilities for crafting varied CFG structures. Each structure
18

---

Joint nobs prediction network En
Joint notse preciction network Ed
cond Eg
uncond F
(a) Conditional output
(b) Unconditional output
Figure 10: Application of CFG for free our MoNL approach for cross-modal generation tasks. Whereas null token is used in traditional CFG for unconditional output, formulating diffusion timestep as vector enables this by setting the input condition per task-specification to pure noise.
Notvo leved
JUMMNG)
prediction
uncond
0
Zh M Mu
(b) Unconditional output variation
Joint notse
prediction network Eg
cond Eg
(a) Conditional output
Joint noise prediction network E0
uncond
Eg
Unconditional output variation
Figure 11: Application of CFG for free in MoNI. approach for multimodal interpolation tasks. Because our vector formulation of the timestep enables applying variable noise levels to different portions of the input one can construct different CFG with "mix-and-match" of modalities and time- segments for creating unconditional outputs. Here, we show an example for multimodal interpolation task for (a) conditional output with two variations: (b) unconditional output with respect input condition per task specification, (c) partial conditional output, but unconditional output with respect modalities.
can amplify specific time segments modalities based the task demands, demonstrated in Fig. 10 for cross-modal tasks and Fig. 11 for the case of multimodal interpolation generation driven by temporal conditioning.
F Discussion
Limited conditioning information: In Table 7, we compares the results of AV continuation de- pending on the input information: AV-continue- 2s)) to out seconds of AV given the first video frame and corresponding 0. 125s of audio AV-continue-1.. 1.5) to fill out 1.5 seconds of AV given the first video frames and corresponding 0.625s of audio. The model performed better when given more context (5 video frames and corresponding audio) compared to less context (1 frame and corresponding audio), even though task-specific training can be an upper bound for performance. This suggests that limited conditioning information can lead to issues like unnatural motion and inconsistencies, and including more context improves the model's performance.
19

---

Table 7: Comparison of AVDIT trained with mixture of noise levels (MoNL. on the Monologues datasct for AV co-continue-22 2s) fill out 2 seconds of AV given the first video frame and corresponding 0.125s of audio fill 1.5 seconds of AV given the first video frames and corresponding 0.625s of audio. FAD 2.7 and estimated with N=25k.
AV-continue-1.
3.3 ground autoencoder reconstructions the inputs. Fréchet metrics
Setting/Tssk
AV-continue- 2s
AV-continue 11.51......
FAD1
FVDI
FAD1
FVD1
Conditional (task-specific)
8.2
117.3
7.4
12.1
Per modality
5.8
120.5
4.5
92.1
Vanilla
7.5
142.6
6.1
140.8
MoNL (Ours)
3.6
12.9
3.1
8.8
Per semm-nt
6.7
102.5
72.3
Per sene-et Per modality
5.8
82.8
4.8
61.2
Pt/Pm/Ptm
4.1
20.2
4.1
16.9
Vanilla
Per-modallty (UniDiffuser)
MoNL (Ours)
A2V
Rating (t poor & good)
AV qualiy
AV algnment
Person consistency
qualty
AV eligrment
Person consistency
AV-continue
AV-interp
5
4
3
2
AV quality
AV aligrment
Person consistency
quality
AV alignment
Person consistancy
Figure 12: Comparative analysis across AVDIT models from the user study along axes of AV quality, AV alignment and person consistency for two cross cross-modal generation tasks (A2V and V2A), and multimodal interpolation tasks (AV- continnn and AV- WWiinpaint). The indicates statistically significant pairwise difference at p 0.01 after multiple comparison correction. Across the board, MoNL. (Ours) was rated significantly better or on par across all tasks, except for AV-alignment for V2A task (comparison shown in red). For A2V task, there was significant difference between the models compared for the measure of AV quality. For multimodal interpolation tasks (bottom row), Our approach far surpasses other models for quality, alignment and consistency underscoring the ability of our approach to generate temporally consistent samples that are perceptually congruent with the input condition.
Condition
Denee sulput
AV continustion for seconds
Figure 13: Full length examples of AV continuation for 2s from AVDIT trained with MoNL. Samples were generated at fps at 128 x 128 resolution and are shown at the same rate.
20

---

1111111
output
Figure 14: Full length examples of generations from AVDiT trained with mixture of noise levels (MoNL) on Joint generation
Conditiio
Cenerated
Condition
Generated output
AV continuation
putput Multimodal interpolation
Generated
Audio-to-video generation
Condition
Generated output
Video-to-audio generation
the AIST+ datasct. Generated 8fps with 128 128 image resolution.
21

---

Generated output
Joint generation
output
output
AV continuation
Condition
Generated output
Condition
Condition
outpur
Multimodal interpolation
Generated output
Condition
Audio to vivee generation
output
Video-to audio generation
Figure 15: Full length examples of generations from AVDIT trained with mixture of noise levels (MoNI.) on the Landscape dataset. Generated at 8ips with 256 x 256 image resolution.
22

---

G Theoretical Background on Mixture of Noise Levels
G.1 Theoretical Background Multimodal Learning
In "A Theory of Multimodal Learning" " [32], multimodal learning is shown to offer a superior generalization bound compared unimodal learning, with improvement factor of O(vn), where denotes the sample size. This benefit relies on connection and heterogeneity between modalities:
Connection. The bound depends learned connections between (At) and ())).
Heterogeneity. Describes how modalities and y, diverge and complement.
connection and heterogeneity are missing, ill-conditioned scenarios can arise. For instance, y, perfect connection suggests no need for learning about V On the other hand, if is random noise, there is heterogeneity but no meaningful connection between A' and V, making non non-trivial learning X alone impractical.
The theory also highlights that and effective connections between modalities via generative models enhance multimodal learning. This forms the basis for our Mixture of Noise Levels (MoNL) approach, which particularly suited for multimodal learning with audio and video data.
G.2 Advantages of Mixture of Noise Level Training
Our MoNL training method offers significant benefits for multimodal learning, especially with audio and video data:
Heterogeneity and connection. Audio and video are naturally heterogeneous. For example, video of person speaking includes audio of spoken words and video lip movements and facial expressions. MoNL uses variable noise levels to enhance learning by capturing the generic transition matrix across the temporal axis.
(1.1) po ([2 (ZD
(M.N) (LM.M)
(1.1) 20.13
(M.N) AOM
(Eq.
Enhanced connectivity. MoNL improves connectivity between audio and video modalities. Our experiments show that MoNL often surpasses task-specific learning approaches by fostering better connections between modalities, adapting its focus more effectively.
G.3 Enhanced Connectivity - Coomparison with Existing Methods
MoNL Joint learning in MMD [40]. Unlike joint learning methods that focus the joint distribution (0 Zz), MoNL trains across multiple conditioning, enabling better connections by varying its focus. This is evidenced by MoNL outperforming the Vanilla (see Table and MMD
models (see Tables 2 and 3).
MoNL vs. Per-modality training. MoNL goes beyond per-modality training in UniDif-
fuser [3], which uses variable noise between modalities i.e., learning pa pa( po([2 _1_ (0an_
(1)
(M)
D). MoNL introduces variable noise across different time segments, learning connec- tions across temporal dynamics as well. This advantage is demonstrated in Table 1.
(1) (..)...., (M)
MoNL vs. Masked training [49]. Diffusion models often obscure high-frequency details with low noise and low-frequency structures with high noise [7]. MoNL employs variable noise levels to explore diverse frequency components, enhancing the model's ability to correlate high and low- frequency elements. This is in contrast to masked self-supervised learning, which limits frequency- specific connections by masking entire elements.
In summary, the effectiveness of MoNL for multimodal diffusion models, particularly with audio and video data, stems from its strategic use connection and heterogeneity. By applying variable noise levels, MoNI. enhances connectivity between modalities and better adapts to diverse temporal and frequency components, leading to superior performance compared to existing multimodal learning methods.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The two main claims of the paper, i..., task- training to support a range audiovisual generation tasks inference time, and the ability of our model to generate temporally consistent generations have been empirically validated through quantitative and user studies, along with demo videos on the project page.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We have added detailed discussion limitations, potential future work and considerations with respect to human-centere generation experiments and evaluations in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate" "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the
implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: [NA] Guidelines:
24

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yss]
Justification: There are two main components of reproducibility here: (I) Model architecture is described in detail and the pseudocode for the proposed algorithm has been provided. (2) Evaluation: The Fréchet metrics are implemented using publicly available code, and the design of the user interface for we used as stimulus for rater studies shared in the Appendix along with the rubric the measurements used by the raters. Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
25

---

reproducible, they The datasets we have used to benchmark with baselines are already publicly available. The Monologues dataset on which we conduct our core experiments not publicly available.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips cctpublic//acccccccccccccccc.cct /gw.ides////e/eeee/eoo /Codeeooodssssseeeee wwwssssssiiioooeeeeooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: summary is provided in the main text of the paper, and the implementation details are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: For qualitative studies, where examined rater labels of quality or pref- erences, we describe the statistical test along with the p-value at which the statistical significance is estimated after multpple-comparison correction is also reported. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
26

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative
error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The implementation details including compute resources are described in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : nneurpss....//// cc cc/public/ CthissGuiellines" Answer:
[Yes]
Justification: We have carefully considered some of the potential societal impacts of this work and potential harms, and discussed possible mitigations in the section "Limitations, Impact and Considerations" in the Appendix. Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes]
Justification: We have discussed broader impact of the paper in the section "Limitations, Impact and Considerations" in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
27

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes]
Question:
Justification: Potential harms and possible mitigations are discussed in the Appendix. The video demos shared for demonstrating non-verbal audiovisual behaviors are from consented individual.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators of the benchmarking datasets and the code we've ve used in this paper have been approproately cited along with the github URLs where the code was provided by the original authors. Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: Pseudocode relevant reproducing the implementation of the proposed algorithm is available in the appendix. Demo videos showing recognizable persons are from consented individual from whom written consent was obtained.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
28

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Detailed instructions along with the user interface ahown to the raters and the rubric used for rating tasks are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "VxUiQrlwQ1", "reviewer": "Reviewer_bNvz", "summary": "The paper presents a new method for audiovisual generation where the input output condiitions may comprise of 2 modalities, namely video and audio sequence. The authors propose a new training approach to effectively learn conditional distribution in multimodal space. The main novely in the paper is a mixture of noise level formulation for processing the inputs. The method produces temporally consistent samples and outperforms existing arts and the vanilla configuration", "questions": "1. Could the authors provide a proof of why the proposed method would work better\n2. Are there any sampling modifications required to accommodate the proposed training strategy?\n3. Could the authors give an analysis of how the proposed algorithm will work when compared to [1]", "limitations": "Please see weakness.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The utilization of mixture of noise levels is novel and the methods seems to improve robustness in denoising, hence leading to a performance boost\n2. The evaluations seems to be fair and clearly signifies the working of the method.\n3. The paper is well written and easy to follow.", "weaknesses": "1. Although the method seems to work well empirically, the paper lacks theoretical backing for the proposed method. It would be good to see some proofs that the proposed method leads to a better approximation of the variational lower bound and joint distribution.\n2. Are there any sampling modifications required to accommodate the proposed training strategy?\n3. There is a plethora of compositional works derived from the energy based formulation of diffusion models. Could the authors analyze how the proposed method performs in comparison to it.\n[1] https://energy-based-model.github.io/Compositional-Visual-Generation-with-Composable-Diffusion-Models/\n4. I'm rating the paper borderline for now, I will improve my rating if the authors can address my concerns", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **[W1&Q1]**: *\"Although the method seems to work well empirically, the paper lacks theoretical backing for the proposed method. It would be good to see some proofs that the proposed method leads to a better approximation of the variational lower bound and joint distribution.\"*\n\nWe appreciate the reviewer's request for theoretical justification. Our approach leverages variable noise levels to learn a general transition matrix (Eq. 4, Section 3.1) between modalities and time segments. This can be used for the inference of arbitrary conditional distributions with various input-output combinations and enables handling diverse conditional tasks (Section 3.3).\n\n\nTo elucidate the core idea, let's consider a simplified scenario where the multivariate data consists of only two elements, $x$ and $y$. In this case, each transition matrix can be interpreted as learning a conditional distribution of the form $p(x|y=c)$, where $c$ represents a specific condition. By training across multiple variable noise levels, our objective become as follows:\n\n$$L = E_{x,y}[-\\text{log} \\ p(x,y)] + α E_{x,y}[-\\text{log} \\ p(x|y)]$$\n\nwhere $α$ balances the importance of joint and conditional learning depending on the variable noise schedule stratgies. This formulation demonstrates that our method simultaneously learns the joint distribution and conditional distributions, offering potential advantages over methods that solely focus on joint learning $E_{x,y}[-\\text{log} \\ p(x,y)]$, such as MM-Diffusion.\n\nWhile this simplified explanation provides intuition, we acknowledge that this is a simplified explanation and that a more rigorous theoretical analysis is ongoing. We plan to provide a comprehensive proof and generalized formulation in future work.\n\n$ $\n > **[W2&Q2]**: *\"Are there any sampling modifications required to accommodate the proposed training strategy? \"*\n\nWe detailed sampling modifications in Section 3.3 and Figure 4. Inference involves selective noise injection based on the task. Clean inputs are used for conditioned parts ($t^{(m, n)} = 0$), while noisy inputs are injected for desired generation ($t^{(m, n)} = t$). This allows a single model to handle various conditional tasks with diverse input-output combinations.\n\n\n$ $\n > **[W3&Q3]**: *\"There is a plethora of compositional works derived from the energy based formulation of diffusion models. Could the authors analyze how the proposed method performs in comparison to it. [1] \"*\n\nBoth methods involve energy-based diffusion models. However, our work differs significantly:\n- **Target Tasks/Modality**: We focus on Audio-to-Video, Video-to-Audio, and Audiovisual continuation and interpolation tasks with variable durations. Composable Diffusion targets only Text-to-Image generation.\n- **Objective of Method**: Composable Diffusion focuses on composing concepts at inference without further training ($p(\\boldsymbol{x}|\\boldsymbol{y}^{(1)}, ..., \\boldsymbol{y}^{(C)})$ where $c_i \\in [1,C]$ represents each concept). We aim for a single model handling diverse conditional distributions ($p(\\boldsymbol{x}_0^{(n \\in \\mathcal{N}_x^c)}, \\boldsymbol{y}_0^{(n \\in \\mathcal{N}_y^c)} \\mid \\boldsymbol{x}_0^{(n \\in \\mathcal{N}_x)}, \\boldsymbol{y}_0^{(n \\in \\mathcal{N}_y)})$ where \n $\\mathcal{N}_x$ and $\\mathcal{N}_y$ are input index sets).\n- **Training/Inference**: Composable Diffusion uses weighted summation of conditional scores during inference. We learn a general transition matrix during training and perform selective noise injection based on the task for inference.\n \nThese fundamental differences highlight that our methods can be complementary. We envision combining them for complex tasks like predicting background audio from two videos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6vbgHQOtVH", "reviewer": "Reviewer_tkoE", "summary": "This paper introduces the Audiovisual Diffusion Transformer (AVDiT) with Mixture of Noise Levels (MoNL) for audiovisual sequence generation. The key innovation is the use of variable noise levels during the diffusion process, applied across different time segments and modalities. This approach enables the model to effectively learn arbitrary conditional distributions in a task-agnostic manner, making it versatile for various generation tasks such as cross-modal generation, multimodal interpolation, and audiovisual continuation. Experiments on multiple datasets demonstrate that AVDiT with MoNL outperforms existing baselines, generating temporally and perceptually consistent audiovisual sequences.", "questions": "See weakness.", "limitations": "The authors mention some limitations in the appendix, such as the need for further improvements in visual and speech quality and the potential for ethical concerns. However, for me, the lack of extensive evaluation of more diverse and real-world datasets is a noteworthy limitation. Some additional limitations can be found in the Weaknesses section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed variable noise levels across different time segments and modalities is an effective approach to enhance the flexibility of diffusion models.\n2. The model's ability to handle various audiovisual generation tasks within a single framework is impressive.\n3. The paper includes extensive experiments on multiple datasets, with both qualitative and quantitative evaluations. The additional demo page provides an intuitive comparison.", "weaknesses": "1. While the variable noise levels concept is interesting, the overall novelty of the approach may be seen as incremental. Similar techniques in diffusion models and transformers have been explored in papers like MM-Diffusion.\n2. Some technical details are not thoroughly explained, such as the criteria for selecting and varying noise levels across time segments and modalities.\n3. The evaluation is primarily conducted on a few datasets (Monologues, AIST++, Landscape). The model's generalizability to real-world scenarios remains uncertain.\n4. The authors compare the transformer-based AVDiT model to the UNet-based MM-Diffusion model, which might not be a fair comparison due to the different architectures. The authors should consider training a UNet-based model using their proposed approach to provide a more direct comparison and validate the effectiveness of their method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **[W1]**: *\"While the variable noise levels concept is interesting, the overall novelty of the approach may be seen as incremental. \"*\n\nWhile we appreciate the reviewer's acknowledgment of the variable noise levels concept, we believe that our work offers significant advancements beyond prior art.\n\n- **Temporal Dynamics**: Unlike MM-Diffusion [R1], which struggles with generating temporally consistent sequences, our method effectively models temporal dependencies through AVDiT with MoNL. This enables superior performance on tasks requiring temporal coherence, such as multimodal interpolation and audiovisual continuation.\n- **Generalization and Flexibility**: Our approach is designed to handle a wide range of multimodal, sequential tasks, surpassing the limitations of previous methods like UniDiffuser [R2] and Versatile Diffusion [R3], which primarily focus on unimodal or static data. By introducing a generalized framework, we enable the modeling of complex audiovisual interactions and the generation of expressive, controllable multimedia content.\n- **Systematic Evaluation and Performance**: Our extensive evaluation demonstrates that AVDiT with MoNL consistently outperforms state-of-the-art baselines in generating high-quality, temporally coherent audiovisual sequences.\n- **Pioneering Diffusion Transformer for Audio-Video**: To the best of our knowledge, our work is the first to successfully apply a diffusion transformer to the audio-video multimodal domain. This novel architecture, combined with MoNL, has led to significant advancements in audiovisual sequence generation.\n\n[R1] Ruan et al. \"MM-Diffusion: Learning multi-modal diffusion models for joint audio and video generation.\" CVPR 2023\n\n[R2] Bao et al. \"One transformer fits all distributions in multi-modal diffusion at scale.\" ICML 2024\n\n[R3] Xuet al. \"Versatile diffusion: Text, images and variations all in one diffusion model.\" CVPR 2023\n\n$ $\n > **[W2]**: *\"Some technical details are not thoroughly explained, such as the criteria for selecting and varying noise levels across time segments and modalities. \"*\n\nWe appreciate the reviewer's keen interest in the technical details of our work. Section 3.2 and Algorithm 1 provides a detailed explanation of how we selected and varied noise levels across time segments and modalities. For instance,  in MoNL that is a training paradigm where a timestep is uniformly randomly selected from the mixture, we first chosse a stratgy for selecting variable noise levels from $\\mathcal{U}( \\set{ \\texttt {Vanilla}, \\texttt {Pt}, \\texttt {Pm}, \\texttt {Ptm} } )$,  and the noise level is set by setting time step vectors depending on the type of the stratge following Line 123-135.\n\nWe understand the importance of these technical details for reproducibility and have included comprehensive supplementary materials detailing implementation details on autoencoders, AVDiT, diffusion training, inference. Also we provided experimental setup and evaluation details. To enhance readability, we are committed to carefully selecting key details for inclusion in the main paper, ensuring that the core methodology is clear and accessible to a broad audience.\n\n\n$ $\n > **[W3]**: *\"The evaluation is primarily conducted on a few datasets (Monologues, AIST++, Landscape). The model's generalizability to real-world scenarios remains uncertain.\"*\n\nWhile we agree that evaluating on a diverse set of datasets is crucial, we believe that our approach effectively addresses the core challenge of audiovisual alignment. \nIndeed, existing audiovisual datasets often lack the diversity of cues necessary for a comprehensive assessment of audiovisual synchorny.\nThe Monologues dataset (16M), with its rich variety of audiovisual cues provides a strong foundation for assessing this alignment: human appearances(e.g., a range of perceived age, perceived gender expression, head pose, lighting), verbal(e.g., intonation, prosody, emotion), and nonverbal cues(e.g., head nods, body gestures and expressions).\n\nTo further demonstrate the generalizability of our model, we conducted comprehensive experiments on multiple datasets, including AIST++ and Landscape where our baseline evaluated. These datasets represent diverse audiovisual domains, ensuring a wide range of audio (music, speech, natural sounds) and video content (monologues, dance, natural scenes). \nThis diverse evaluation demonstrates the versatility of our model in handling various audiovisual combinations.\n\n\n$ $\n > **[W4]**: *\"The authors compare the transformer-based AVDiT model to the UNet-based MM-Diffusion model, which might not be a fair comparison due to the different architectures. The authors should consider training a UNet-based model using their proposed approach to provide a more direct comparison and validate the effectiveness of their method. \"*\n\n\nWe appreciate the reviewer's insightful comment regarding the architectural differences between our transformer-based MoNL and the UNet-based MMD. While a comparison of MMD with our proposed MoNL and MMD with original joint learning would be ideal, we encountered challenges in effectively integrating our diffusion timestep vector with the sparse multimodal attention module (RS-MMA) – a core component of the MMD architecture.\n\nTo provide a more equitable comparison, we opted to train a transformer-based joint learning that can be regarded as a variant of MMD (Vanilla in Table 1) as a counterpart to our transformer-based MoNL. Our results demonstrate the superiority of MoNL over this transformer-based MMD, highlighting the effectiveness of our proposed approach. Additionally, we directly compare AVDiT, MoNL, and MMD to further underscore the overall strength of our model.\n\nWe will clarify this point and provide additional details in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uHb6DN3cWq", "reviewer": "Reviewer_SA7V", "summary": "The paper tackles the audio-visual cross-modality generation problem and proposes a training approach to learn arbitrary conditional distributions in the audiovisual space. At the methodological level, the authors propose to apply variable diffusion timesteps across the temporal dimension. The experiments are conducted on Monologues, AIST++, and Landscape datasets.", "questions": "In addition to the comments in the Weaknesses section. \nI may have missed several details: how is the scheduler defined in the proposed method Eq. (5)? What is the time cost for training the proposed model? And how does this design yield a difference compared to training on single modality or separate DMs?", "limitations": "The limitations are discussed in the Appendix A.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The high-level motivation to learn arbitrary conditional distributions in the audiovisual space is interesting. The experiments show promising results and the authors conduct user studies as supplementary evaluations.", "weaknesses": "1. The writing and presentation of the paper can be improved. Figure 1 seems to have some issues with the first-row caption for the AIST dataset, which makes it difficult to read. The usage of math symbols is inconsistent, e.g., the $x$ should be $\\bf{x}$ in Line 72. The term “multivariate data” in Line 63 is also not rigorous and confusing, in other words, do the authors imply that the static image data is uni-variante? The use of $\\equiv$ and $=$ is also mixed in the paper.\n\n2. One of the key claims and motivations, “Training separate models for each variation is expensive and impractical”, seems controversial and susceptible to me, while I understand training separate DMs for audio and visual data could be expensive, I don’t think that learning a single mixture is a better option than learning two separate data distributions, because learning a mixture increase the complexity of target data distribution, and thus should not be beneficial for performance especially if the domain gap between separate modality is large. As this is one of the key claims that motivates the methodological design, I was expecting more rigorous theoretical support in the paper, could the authors elaborate on that?\n\n3. For the experiments, while the authors claim that “the sequential representations can be either latent spaces or raw data”, the actual implementation only conducts experiments on low dimensional features of pre-trained models, i.e., MAGVIT-v2 for video representations and SoundStream for audio representations. While this is not a serious flaw, it raises questions about the generalizability of the proposed method, and the fairness in comparison with other methods, as the pre-trained models may (and very likely) directly influence the final performance. And the baseline method MM-Diffusion seems not to operate on the same feature space according to Appendix C?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **[W1]**: *\"The writing and presentation of the paper can be improved. Figure 1 seems to have some issues with the first-row caption for the AIST dataset, which makes it difficult to read. The usage of math symbols is inconsistent, e.g., the $𝑥$ should be  $\\boldsymbol{x}$ in Line 72. The term “multivariate data” in Line 63 is also not rigorous and confusing, in other words, do the authors imply that the static image data is uni-variante? The use of ≡ and =  is also mixed in the paper.\"*\n\n\nWe sincerely appreciate the reviewer’s constructive feedback on the clarity and presentation of our paper. We agree that improving these aspects is crucial for effective communication of our research.\n\nWe will meticulously address the reviewer’s comments by enhancing the readability of Figure 1, ensuring consistency in mathematical notation (e.g., using $\\boldsymbol{x}$ for vectors), and refining ambiguous terminology such as \"multivariate data\" to accurately reflect the sequential nature of our data. We will also carefully distinguish between the use of ≡ and = throughout the paper.\n\n\n$ $\n > **[W2&Q3]**: *\"One of the key claims and motivations, “Training separate models for each variation is expensive and impractical”, seems controversial and susceptible to me, while I understand training separate DMs for audio and visual data could be expensive, I don’t think that learning a single mixture is a better option than learning two separate data distributions, because learning a mixture increase the complexity of target data distribution, and thus should not be beneficial for performance especially if the domain gap between separate modality is large. As this is one of the key claims that motivates the methodological design, I was expecting more rigorous theoretical support in the paper, could the authors elaborate on that?\"*\n\n\nWe appreciate the reviewer's insightful comments. While learning a single mixture model can increase complexity compared to separate models, we argue that the benefits of a unified model outweigh the potential drawbacks, especially in the context of large-scale multimodal tasks.\n\nThe reviewer points out the potential challenges of learning a mixture distribution with large domain gaps. While we acknowledge this, we believe that the shared underlying structure of audio and video data, such as temporal synchronization and physical world correlations, can help mitigate these issues. Our model leverages these shared representations to improve performance across various tasks. This is supported by our experimental results in Table 1 as well as the recent works on audio-visual representation learning (AVMAE [R1]) which demonstrates the benefits of multimodal learning on the audiovisual tasks compared to unimodal learning for audio and video respectively. Also, UniDiffuser [R2] show the success of multimodal generation in the text-image domain.\n\nAs you mentioned, training separate models for each task variation can be prohibitively expensive, especially when considering the vast number of potential combinations in the audio-video domain. Our approach offers a more efficient and scalable solution by learning a single model capable of handling multiple tasks simultaneously. \n\nWe believe that our model offers a promising approach to tackling the challenges of large-scale multimodal tasks. \n\n[R1] Bao et al. \"One transformer fits all distributions in multi-modal diffusion at scale.\" ICML 2024\n\n[R2] Georgescu et al. \"Audiovisual masked autoencoders.\" CVPR 2023\n\n$ $\n > **[W3]**: *\"For the experiments, while the authors claim that “the sequential representations can be either latent spaces or raw data”, the actual implementation only conducts experiments on low dimensional features of pre-trained models, i.e., MAGVIT-v2 for video representations and SoundStream for audio representations. While this is not a serious flaw, it raises questions about the generalizability of the proposed method, and the fairness in comparison with other methods, as the pre-trained models may (and very likely) directly influence the final performance. And the baseline method MM-Diffusion seems not to operate on the same feature space according to Appendix C? \"*\n\nWe appreciate the reviewer's insightful comments. Regarding generalizability, we acknowledge the limitation of our current experiments to latent spaces. While we believe our method is conceptually applicable to raw data, practical constraints such as computational resources from conducting extensive experiments in this setting. We will clarify this point in the modified version. Note that our training data was not seen during training of the audio and video autoencoders. \n\nMMD has their own autoencoders. To address the fairness of comparison, comparison of MMD with our proposed MoNL and MMD with original joint learning would be ideal. We encountered challenges in effectively integrating our diffusion timestep vector with the sparse multimodal attention module (RS-MMA) – a core component of the MMD architecture.\nTo provide a more equitable comparison, we opted to train a transformer-based variant of MMD (Vanilla in Table 1) as a counterpart to our transformer-based MoNL. Our results demonstrate the superiority of MoNL over this transformer-based MMD, highlighting the effectiveness of our proposed approach. Additionally, we directly compare AVDiT, MoNL, and MMD to further underscore the overall strength of our model.\n\n$ $\n > **[Q1]**: *\"how is the scheduler defined in the proposed method Eq. (5)? \"*\n\nThe noise scheduler used in Equation (5) is identical to the one detailed in Section 2 (Lines 68-74).\n\n$ $\n > **[Q2]**: *\"What is the time cost for training the proposed model?\"*\n\nA comprehensive analysis of the training time costs is provided in the Supplementary Material, Section B (Lines 536-541). On average, the models were trained for around 350K steps with a batch size of 256 for around five days.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K69wVTLCne", "reviewer": "Reviewer_4xRp", "summary": "This paper presents a new diffusion model that is capable of generating data following various conditional distributions in multimodal data. Rather than applying a uniform noise level to all data elements, the proposed model permits the use of differing noise levels across modalities and time dimensions. This allows the model to concurrently learn a variety of conditional distributions, including cross-modal conditioning, temporal interpolation, and temporal continuation. The experimental results with audio-video datasets validate the effectiveness of the proposed model.", "questions": "- In the ablations at Table 1, why is Vanilla not used? Without the Vanilla strategy, the model may not encounter cases where all elements are almost entirely noise or close to clean ones during training, which could potentially degrade the performance of the model to some extent. Therefore, a more reasonable approach for ablation might be to gradually add other strategies to Vanilla.\n\n---\n\n<post-rebuttal>\n\nI have updated my rating from 4 to 5, as my concern on the dataset is somewhat resolved through the discussion.", "limitations": "Limitations are properly discussed in the appendix.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- The idea of MoNL is simple and reasonable. It definitely enables the model to effectively learn various conditional distributions simultaneously.\n\n- The proposed method performs well in the experiments. MoNL particularly contributes to boost the performance of the model in AV-inpaint and AV-continue tasks.\n\n- The manuscript is well-written and easy to follow.", "weaknesses": "- The major concern is on the experiments: the majority of the empirical evaluations are conducted with the internal dataset. As there are already several publicly-available datasets (such as Landscape, AIST++, and VGGSound) that are commonly used in the literature, I strongly encourage the authors to do at least the ablation studies with such datasets to make them reproducible. \n\n- The novelty in methoology is somewhat limited. The idea of using various noise levels in multimodal generative modeling has been presented in the prior work [3]. A particular challenge in this paper is to extend this idea also for modeling temporal dynamics, but similar strategies are commonly adopted in several video generation studies:\n    - “Diffusion Models for Video Prediction and Infilling,” TMLR 2022.\n    - “MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation,” NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **[W1]**: *\"The major concern is on the experiments: the majority of the empirical evaluations are conducted with the internal dataset. As there are already several publicly-available datasets (such as Landscape, AIST++, and VGGSound) that are commonly used in the literature, I strongly encourage the authors to do at least the ablation studies with such datasets to make them reproducible. \"*\n\nWhile we agree that evaluating on a diverse set of datasets is crucial, we believe that our approach effectively addresses the core challenge of audiovisual alignment.\nIndeed, existing audiovisual datasets often lack the diversity of cues necessary for a comprehensive assessment of audiovisual synchorny.\n\nOur focus has been on developing a robust audiovisual generative model with the great alignment capable of handling diverse and challenging practical real-world scenarios such as talking avatar. The Monologues dataset (16M samples), with its extensive variety of audiovisual cues such as  human appearances(e.g., a range of perceived age, perceived gender expression, head pose, lighting), verbal(e.g., intonation, prosody, emotion), and nonverbal cues(e.g., head nods, body gestures and expressions), provides a strong foundation for evaluating our model's performance under these conditions. We believe that the insights gained from our experiments on this dataset are valuable contributions to the field.\n\nFurthremore, while we included AIST++(1020 samples) and Landscape(8233 samples) datasets as they are used in our baseline, MM-Diffusion [R1] for the evaluation, we found that AIST++ and Landscape datasets are very small datasets and the results from ablation studies may not be generalizable given the risk of overfitting and memorizing.\n\nWe are committed to advancing the research in audiovisual generation and plan to opensource our Monologue datasets as well as explore the use of publicly available datasets with rich audiovisual cues.\n\n[R1] Ruan et al. \"MM-Diffusion: Learning multi-modal diffusion models for joint audio and video generation.\" CVPR 2023\n\n$ $\n > **[W2]**: *\"The novelty in methoology is somewhat limited. The idea of using various noise levels in multimodal generative modeling has been presented in the prior work [3]. A particular challenge in this paper is to extend this idea also for modeling temporal dynamics, but similar strategies are commonly adopted in several video generation studies\"*\n\nWhile the concept of noise levels in generative modeling has been explored, MoNL offers significant advancements. Unlike UniDiffuser [R2], which struggles with multimodal sequence data, MoNL introduces a generalized formulation for handling diverse modalities with temporal dynamics. Our systematic evaluation demonstrates MoNL's superior performance and versatility across various tasks, including audio-video continuation and interpolation.\n\nUnlike previous video-focused methods such as “Diffusion Models for Video Prediction and Infilling” and \"MCVD\", MoNL excels in handling cross-modal and noisy conditions. Our approach avoids complex masked input mechanisms and hyperparameter tuning, resulting in a simpler and more efficient model. These key distinctions highlight MoNL's novelty and contribution to the field.\n\nWe will incorporate a more detailed comparison to related work, including quantitative performance metrics, in the revised paper.\n\n[R2] Bao et al. \"One transformer fits all distributions in multi-modal diffusion at scale.\" ICML 2024\n\n$ $\n > **[Q1]**: *\"In the ablations at Table 1, why is Vanilla not used? Without the Vanilla strategy, the model may not encounter cases where all elements are almost entirely noise or close to clean ones during training, which could potentially degrade the performance of the model to some extent. Therefore, a more reasonable approach for ablation might be to gradually add other strategies to Vanilla.\"*\n\n\n\nWhile we appreciate the reviewer's suggestion, our ablation design was carefully considered. Our goal was to systematically demonstrate the contributions of different components to our model's overall performance.\n\n- **Vanilla as a Baseline**: Our \"joint generation\" model, $\\texttt{Vanilla}$, serves as a baseline for comparison. This model represents a unconditional training for joint generation without specialized handling of multimodal or cross-modal tasks.\n- **Task-Specific Strategies**: We introduced $\\texttt{Pt}$, $\\texttt{Pm}$, and $\\texttt{Ptm}$ to address specific challenges in multimodal and cross-modal generation. Our ablation compares these strategies against the baseline to quantify their impact.\n- **Combined Approach**: The $\\texttt{Pt/Pm/Ptm}$ model demonstrates the effectiveness of combining task-specific strategies.\n- **MoNL with $\\texttt{Vanilla}$**: Finally, we compare MoNL (which incorporates $\\texttt{Vanilla}$) to $\\texttt{Pt/Pm/Ptm}$ to highlight the additional benefits of the $\\texttt{Vanilla}$ components for joint generation.\n\nFurthermore, we found that even though the model may not encounter cases where all elements are almost entirely noise or close to clean ones during training, each strategy of $\\texttt{Pt}$, $\\texttt{Pm}$, and $\\texttt{Ptm}$ can work overally and excels at each target task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their constructive and insightful feedback.

Their recognition of the paper's strengths has been invaluable. We are particularly grateful for their positive comments on:

- **The effectiveness of our approach**: The reviewers highlighted the innovative use of mixed noise levels, which significantly enhances the model's robustness and overall performance.
- **The comprehensive evaluation**: Our fair and thorough experimental analysis, including qualitative and quantitative assessments, has been acknowledged as crucial in demonstrating the method's efficacy.
- **The clarity and readability of the paper**: The reviewers commended the paper's well-structured presentation and ease of understanding.
- **The flexibility and versatility of our model**: The ability to handle diverse audiovisual generation tasks within a unified framework has been recognized as a key contribution.
-  **The strong empirical results**: The reviewers noted the impressive performance of our method across various benchmarks and tasks, including the notable improvements in AV-inpaint and AV-continue.

We have carefully considered all reviewer comments and have made substantial revisions to the paper accordingly. Detailed responses to each point can be found in the rebuttal.

Meta-review: This paper proposes to use varying noise levels across modalities and time segments towards task agnostic audio-visual synthesis. The paper received four reviewers; before the rebuttal, the reviewers had raised several concerns on multiple aspects, including technical novelty, gaps in experimental studies, datasets used for experiments, and connections to prior works. Authors provided a strong rebuttal, addressing the concerns, resulting in all the reviewers raising their scores inclining towards acceptance. 

AC agrees with the reviewers' assessment of the paper and recommends acceptance. Authors should incorporate all the changes suggested by the reviewers in the camera-ready, including theoretical connections and clarifications from prior methods. Further, as promised, authors should make arrangements for the release of the Monologues dataset.
Decision: Accept (poster)