Title: A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation
Venue: NeurIPS 2024
Accepted: yes
Keywords: multimodal diffusion

multimodal generation

diffusion timestep

latent diffusion models
Abstract: Training diffusion models for audiovisual sequences allows for a range of generation tasks by learning conditional distributions of various input-output combinations of the two modalities. Nevertheless, this strategy often requires training a separate model for each task which is expensive. Here, we propose a novel training approach to effectively learn arbitrary conditional distributions in the audiovisual space. Our key contribution lies in how we parameterize the diffusion timestep in the forward diffusion process. Instead of the standard fixed diffusion timestep, we propose applying variable diffusion timesteps across the temporal dimension and across modalities of the inputs. This formulation offers flexibility to introduce variable noise levels for various portions of the input, hence the term mixture of noise levels. We propose a transformer-based audiovisual latent diffusion model and show that it can be trained in a task-agnostic fashion using our approach to enable a variety of audiovisual generation tasks at inference time. Experiments demonstrate the versatility of our method in tackling cross-modal and multimodal interpolation tasks in the audiovisual space. Notably, our proposed approach surpasses baselines in generating temporally and perceptually consistent samples conditioned on the input. Project page: neurips13025.github.io

A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation

Abstract

Training diffusion models for audiovisual sequences allows for a range of generation tasks by learning conditional distributions of various input-output combinations of the two modalities. Nevertheless, this strategy often requires training a separate model for each task which is expensive. Here, we propose a novel training approach to effectively learn arbitrary conditional distributions in the audiovisual space. Our key contribution lies in how we parameterize the diffusion timestep in the forward diffusion process. Instead of the standard fixed diffusion timestep, we propose applying variable diffusion timesteps across the temporal dimension and across modalities of the inputs. This formulation offers flexibility to introduce variable noise levels for various portions of the input, hence the term mixture of noise levels . We propose a transformer-based audiovisual latent diffusion model and show that it can be trained in a task-agnostic fashion using our approach to enable a variety of audiovisual generation tasks at inference time. Experiments demonstrate the versatility of our method in tackling cross-modal and multimodal interpolation tasks in the audiovisual space. Notably, our proposed approach surpasses baselines in generating temporally and perceptually consistent samples conditioned on the input. Project page: avdit2024.github.io

1 Introduction

Recent years have witnessed a remarkable surge in the development and exploration of multimodal diffusion models. Prominent examples include text-to-image (T2I) [37, 41, 54, 39], text-to-video (T2V) [17, 6, 12]. Despite notable advancements, generating sequences across multiple modalities, like video and audio, remains challenging and is an open research area.

Introducing a time axis to static data paves the way for diverse multimodal sequential tasks including cross-modal generation (e.g., audio-to-video), multimodal interpolation, and audiovisual continuation as shown in Fig 1. Each task can be further divided based on various input-output combinations of the modalities, leading to a number of conditional distributions. For example, with video data x 1: N 0 and audio data y 1: N 0 of length N , The complexity of configurations grows with tasks like audiovisual continuation, p p x p n c GLYPH<0> 1: N q 0 , y p n c GLYPH<0> 1: N q 0 | x p 1: n c q 0 , y p 1: n c q 0 q , where n c is the input frame length used for conditioning, and multimodal interpolation, p p x p n P N c x q 0 , y p n P N c y q 0 | x p n P N x q 0 , y p n P N y q 0 q , where N x and

Audiovisual continuation with variable input durations

Generated output

Condition

Multimodal interpolation tasks with variable settings

Generated output

Condition

Generated output

Condition

Condition

Generated output

Condition

Condition

Generated output

Condition

Generated output

Condition

Generated output

Condition

Condition

Audio-to-video generation

Video-to-audio generation

Generated output

Condition

Condition

Generated output

Generated output

Generated output

Joint generation

Video

Audio

Video

Audio

Video

Audio

Condition

Generated output

Condition

Generated output

p(a0

,yo

,yo

,yo

Condition

Generated output

Condition

Generated output

Video

N y are input index sets. Training separate models for each variation is expensive and impractical. A more efficient training approach would be to learn these conditional distributions in a single model without explicitly enumerating them, i.e., in a task-agnostic manner.

Unconditional diffusion models like MM-Diffusion [40] show potential for learning conditional distributions implicitly, but rely on inference adjustments [18, 40]. This limits performance, as seen in MM-Diffusion's struggle to generate temporally consistent sequences (see Fig. 2). While UniDiffuser [3] and Versatile Diffusion [51] offer methods for joint and conditional text-image distributions, effectively capturing temporal dynamics of audio and video remains an open challenge.

Here, we propose a multimodal diffusion framework that empowers a single model to learn diverse conditional distributions. This paves the way for a versatile framework for multimodal diffusion, tackling various generation tasks. Our core idea is that, applying variable noise levels across modalities and time segments 3 enables a single model to learn arbitrary conditional distributions. This formulation offers flexibility to train diffusion models with a mixture of noise levels i.e., MoNL , which introduces variable noise levels across various portions of the input. It has a number of advantages over previous approaches: it requires minimal modifications to the original denoising objective simplifying implementation, task-agnostic training, and support for conditional inference of a given task specification without any inference-time modifications.

We apply this approach for audiovisual generation by developing a diffusion transformer, AVDiT . To address the computational complexity of high-dimensional audio and video signals, we implement MoNL in the low-dimensional latent space learned by the MAGVIT-v2 [55] for video and the SoundStream [57] for audio. Importantly, the temporal structure in these latent representations enables us to apply variable noise levels. We also introduce a transformer-based network for joint noise prediction. Transformers are a natural choice for our implementation due to their proficiency to model multimodal data [25, 10] capturing complex temporal and cross-modal relationships.

Weassess the capability of MoNL to model various distributions in the audiovisual space by evaluating cross-modal tasks (audio-to-video and video-to-audio generation), and conditioning on small portions (audiovisual continuation and interpolation tasks). For these tasks, we show that the A VDiT trained

with MoNL outperforms conventional methods including unconditional and conditional generation models, demonstrating the versatility of our task-agnostic framework as shown in Fig. 1. Notably, qualitative and quantitative evaluations highlight the ability of our framework to generate temporally consistent sequences, as illustrated in Fig. 2.

2 Background

Diffusion Models for Multivariate series data: Consider an example of a video diffusion model where the input is a sequence of image frames. In general, this task is modeling multivariate series data (i.e., image representations) of d -dimensions with N elements (no. of image frames), henceforth referred to as time-segments . Thus, the multivariate series data, x 0 GLYPH<16> x 1: N 0 P R N GLYPH<2> d GLYPH<18> q p x 0 q can be represented as a sequence of time-segments, where x n 0 P R d is the n -th time-segment and d -dimensional representation.

During the forward process of diffusion models [44, 16], the original data x 0 is corrupted by gradually injecting noise in a sequence of T timesteps. The noisy data x t at time t can be written as x t GLYPH<16> x 1: N t GLYPH<16> ? α t x 0 GLYPH<0> ? 1 GLYPH<1> α t glyph[epsilon1] x . Here, glyph[epsilon1] x GLYPH<16> glyph[epsilon1] 1: N x GLYPH<18> N p 0 , I q is Gaussian noise injected to the sequence and β t is the noise schedule, α t GLYPH<16> 1 GLYPH<1> β t controls the noise level at each step with α t GLYPH<16> -t i GLYPH<16> 1 α i . Each noisy time-segment can be represented as x n t GLYPH<16> ? α t x n 0 GLYPH<0> ? 1 GLYPH<1> α t glyph[epsilon1] n x . During the reverse process, the data is sampled through a chain of reversing the transition kernel q p x t GLYPH<1> 1 | x t q that is estimated by p θ p x t GLYPH<1> 1 | x t q GLYPH<16> N p x t GLYPH<1> 1 | µ p x t , t q , σ 2 t I q , where µ p x t , t q GLYPH<16> ? α t p x t GLYPH<1> 1 GLYPH<1> α t ? 1 GLYPH<1> α t glyph[epsilon1] θ p x t , t qq . The training objective is to learn a residual denoiser glyph[epsilon1] θ at each step as:

where t GLYPH<18> U pt 1 , 2 , . . . , T uq is the diffusion timestep.

Multimodal Diffusion Models: Unconditional joint generation (generating all modalities simultaneously) and conditional generation (generating one modality conditioned on the rest) are commonly used for multimodal diffusion. Typically, separate models are trained for each task as described below:

Diffusion models for joint generation. For simplicity, let us assume two modalities x 0 , y 0 . The objective in joint generation is to model the joint data distribution, denoted as q p x 0 , y 0 q . To learn this, a joint noise prediction network, denoted as glyph[epsilon1] θ is defined by rewriting Eq. 1 as follows:

where p x 0 , y 0 q is a random data point, r , s denotes concatenation, glyph[epsilon1] x , glyph[epsilon1] y GLYPH<18> N p 0 , I q , and t GLYPH<18> U pt 1 , 2 , . . . , T uq . Diffusion models trained with this objective can perform conditional sampling q p x 0 | y 0 q using inference-time tricks [18, 40].

Conditional training of diffusion models. To learn conditional distributions, expressed as q p x 0 | y 0 q , a noise prediction network glyph[epsilon1] θ conditioned on y 0 is adopted from Eq. 2:

Separate conditional models need to be trained for every pair of modalities and input configurations.

3 Mixture of Noise Levels (MoNL)

We introduce a novel framework for learning a wide range of conditional distributions within multimodal data by using a mixture of noise levels. The key idea is to formulate the timestep t (Eq. 1) that determines a noise level in the forward diffusion as a vector. Then, we present representative strategies for variable noise levels. We then show how conditional inference can be performed without additional training. Finally, assembling all these components, we present our versatile audiovisual diffusion transformer (A VDiT).

3.1 Variable Noise Levels across Modality and Time

Formally, let M represent the number of modalities with sequence representations (latent spaces or raw data). Without loss of generality, assume the representations in each modality have N time-

N

Forward Diffusion

Process

N

time-segments

Joint noise

prediction

network

Predicted noise

M

modalities

Diffusion timestep

(a) Training with a variable noise levels

M

Input data

(b) Mixture of noise

levels (MoNL)

Vanilla

Per modality(

Pm

)

Per time-slice(

Pt

)

Per time-slice and

Per modality(

Ptm

)

Noise

Noise level

Noisy data

Learn a general transition:

by minimizing loss:

[t(1,1)

.,t(M,N)]

t=0

t=T

Eo(l(a.1)

t（1,1)·..

之

[2(1,1)

t(1,1)...

Z

t(M,N)

(M,N)

Eθ

(M,N)

(M,N)1)

(1:M,1:nc)

time-segments

Denoising

step

…

Denoising

step

…

(a) Conditioning across modalities: cross-modal generation

(b) Conditioning across time-segments: multimodal interpolation

Generate each time-slice

Generate each modality

Input

N

time-segments

M

modalities

Input

Noise level

20

(1:mc,1:N)

mc

0 = 7

-

t

2t-1

T~N（O,I)

(I0)N~1z

pe

zi-1

(1:M,nc+1:N)

2t

(1:M,nc+1:N)

20

(1:M,1:nc)

t

（mc+1:M,1:N)

po

2i-1

（mc+1:M,1:N）

(m+1:M,1:N)

20

(1:mc,1:N)

segments 4 . Let us further assume they have the same embedding dimension d (which in practice can be achieved by projecting the noisy input from each modality to the desired dimension). The entire sequence can then be simplified as z 0 P R M GLYPH<2> N GLYPH<2> d GLYPH<16> z p 1: M, 1: N q 0 GLYPH<18> q p z 0 q , where z p m,n q 0 P R d denotes the n -th time-segment of m -th modality. For reference, Sec. 2 represents two modalities of multivariate series data, x 1: N 0 and y 1: N 0 , using this notation as z p 1 , 1: N q 0 and z p 2 , 1: N q 0 , respectively.

We posit that training a single model to support learning arbitrary conditional distributions can be realized by using variable noise levels for each modality m and time segment n of the input space z 0 . We introduce the diffusion timestep vector as t GLYPH<16> t p 1: M, 1: N q P R M GLYPH<2> N to match the dimensionality of the multimodal inputs, where each element t p m,n q P r 1 , T s determines the timestep, and in turn the level of noise added to the corresponding element z p m,n q 0 of the input z 0 .

Recall (from Sec. 2) that in a unimodal case, the goal was to learn the transition kernel q p x t GLYPH<1> 1 | x t q parameterized by p θ p x t GLYPH<1> 1 | x t q GLYPH<16> N p x t GLYPH<1> 1 | µ p x t , t qq , σ 2 t I q . Analogously, by introducing a timestep vector t P R M GLYPH<2> N , our goal is now to learn a general transition matrix between the various modalities and time-segments in z 0 at each step:

Raw data space

Video data

Audio data

Latent space

Denoising via AVDiT

trained with MoNL

Diffusion timestep

[  ]

+

+

[  ]

Embedding layer

Embedding layer

Embedding layer

AdaLN

Spatiotemporal

positional

encoding

Temporal

positional

encoding

Layer norm

Multi-head attention

MLP

+

+

MLP

Scale, Shift

Scale

Scale, Shift

Scale

Noisy video latent

Noisy audio latent

Layer norm

Predicted noise

Layer norm

Denoising via AVDiT

trained with MoNL

(T-1)×

Generation

Condition

Condition

Clean video

latent     .

Clean audio

latent     .

Noise

Noise

Noise level

MAGViT-v2

Soundstream

(a) Latent diffusion with mixture of noise levels (MoNL) and audiovisual diffusion transformer (AVDiT)

(b) Audio-video diffusion transformer (AVDiT)

Then, for diffusion training, we draw a Gaussian noise sequence glyph[epsilon1] GLYPH<16> glyph[epsilon1] p 1: M, 1: N q . Each noise element glyph[epsilon1] p m,n q is then added to the corresponding element of the original data z p m,n q 0 with noise level determined by t p m,n q as follows:

Then, the joint and conditional training objectives in Eqs. 2 and 3 can be generalized with a single noise prediction objective to learn the joint distribution glyph[epsilon1] θ as follows:

where z 0 GLYPH<18> q p z 0 q is the multimodal input and t is the diffusion timestep vector.

3.2 Representative Stratgies for Variable Noise Levels

Using the generalized view of multimodal noise prediction described in Eq. 6, we now examine various strategies for variable noise levels during the forward diffusion. One can imagine an arbitrarily large number of timestep candidates in the vector space of t drawn as functions of time-segments of the multivariate series and modalities. Here, we explore four designs to create a mixture of noise levels as illustrated in Fig. 3(b). Let us assume we have final diffusion timestep vector for training, t ref P R M GLYPH<2> N where each element t p i,j q ref is sampled from U pt 1 , 2 , . . . , T uq ,

GLYPH<13> Vanilla : Same timestep is assigned to all the time-segments and modalities. This is analogous to performing joint learning as t p m,n q GLYPH<16> t p 1 , 1 q ref , and would be the straightforward way to extend the vanilla distillation approach for the multimodal case.

GLYPH<13> Per Modality (Pm) : Variable timesteps are assigned for each modality, but all time-segments in a given modality have the same timestep as t p m,n q GLYPH<16> t p m, 1 q ref . This is expected to promote cross-modal generation tasks. This is a generalization of the UniDiffuser [3] approach for sequences.

GLYPH<13> Per Time-segment (Pt) : Variable timesteps are assigned as t p m,n q GLYPH<16> t p 1 ,n q ref by keeping track of the corresponding time-segments across modalities. Intuitively, this should promote better temporal consistency.

GLYPH<13> Per Time-segment and Per-modality (Ptm) : Variable timesteps are assigned for each time-segment and modality as t p m,n q GLYPH<16> t p m,n q ref . This would promote better temporal correspondence between modalities.

To enable learning a wide range of conditional distributions, we create a training paradigm where a timestep is uniformly randomly selected from the mixture. Specifically, we refer to this training paradigm as MoNL . A schematic of the overall training process is depicted in Fig. 3(a), with related pseudocodes in Algorithms 1 and 2 in the Appendix. Also, theoretical background on MoNL is detailed in Appendix G.

3.3 Conditional Inference

Once the general transition kernel p θ is learned in Eq. 4, we investigate the model's ability to handle arbitrary conditional distributions. We achieve this by selectively injecting inputs during inference based on the task specification, i.e., clean (no noise) inputs for conditional portions with t p m,n q GLYPH<16> 0 , and noisy inputs for generating desired portions of the input with the current diffusion step t p m,n q GLYPH<16> t .

Consider the case of cross-modal generation (Fig. 4(a)), to generate a sequence of M GLYPH<1> m c modalities conditioned on m c P p 1 , M q modalities, we set timestep elements of M GLYPH<1> m c modalities as t and those of m c conditioning modalities as 0 , which achieves:

Similarly, for multimodal interpolation (Fig. 4(b)), to generate N GLYPH<1> n c time-segments of all modalities jointly, conditioned on n c P p 1 , N q time-segments, we set the timestep for the N GLYPH<1> n c time-segments as t , and for the conditioning n c time-segments as 0 , which achieves p θ p z p 1: M,n c GLYPH<0> 1: N q t GLYPH<1> 1 | z p 1: M,n c GLYPH<0> 1: N q t , z p 1: M, 1: n c q 0 q . Unconditional joint generation is also possible by setting each timestep as the same t , to estimate the transition kernel, p θ p z p 1: M, 1: N q t GLYPH<1> 1 | z p 1: M, 1: N q t q . Intuitively, our mixture of noise levels is analogous to self-supervised learning which bypasses the need for predefined tasks during training but enables a deeper understanding of multimodal temporal relationships. See also Sec. E for the discussion on classifier-free guidance in the Appendix.

4 Audiovisual Latent Diffusion Transformer (AVDiT)

Our model consists of two key components: (1) latent space representations from audio and video autoencoders, and (2) an Audiovisual diffusion transformer (AVDiT) for joint noise prediction.

Latent Space Representations: For a video of 1 GLYPH<0> L v frames, represented as v P R p 1 GLYPH<0> L v qGLYPH<2> H GLYPH<2> W GLYPH<2> C , we use MAGVIT-v2 [55], a causal autoencoder to achieve efficient spatial and temporal compression. MAGVIT-v2 results in a low-dimensional representation, x 0 P R p 1 GLYPH<0> l v qGLYPH<2> h GLYPH<2> w GLYPH<2> d v , by a compression factor of r s GLYPH<16> H h GLYPH<16> W w in space and r t v GLYPH<16> L v l v in time. Crucially, the use of causal 3D convolutions ensures that the embedding for a given frame is solely influenced by preceding frames, preventing flickering artifacts common in frame-level autoencoders.

For audio with L a frames, a P R L a , we use SoundStream [57], a state-of-the-art neural audio autoencoder. We use the latents y 0 P R l a GLYPH<2> d a prior to quantization as audio latents, a compression rate of r t a GLYPH<16> L a l a in time. The time-segments in our formulation refer to the 1 GLYPH<0> l v and l a temporal dimensions in the video and audio latent spaces respectively.

Audiovisual Transformer for Joint Noise Prediction: Transformers [48] are a natural fit for multimodal generation as they can: (1) efficiently integrate multiple modalities and their interactions [58, 10], (2) capture intricate spatiotemporal dependencies [8, 5], and (3) have shown impressive video generation capabilities [12, 25]. Inspired by these benefits, we introduce AVDiT, a noise prediction network for latent diffusion as described in Fig. 5. A VDiT utilizes the timestep embedding similar to the condition signal used in W.A.L.T [12]. The Transformer first processes the timestep embeddings and positional encodings to create an embedding of the timestep vector. This embedding serves as a conditioning signal and is utilized to dynamically calculate the scaling and shifting parameters for AdaLN during the Transformer Layer Normalization step. This enables the normalization to incorporate the conditioning information of variable noise levels. We first consider the l a and 1 GLYPH<0> l v time-dimensions for audio and video embeddings respectively. When applying MoNL, we can easily keep track of the corresponding time segments among the l a and 1 GLYPH<0> l v dimensions, given the temporal compression factors in each modality. The noisy latents are then linearly projected matching the final dimension d by adding appropriate spatiotemporal positional embeddings for video and temporal positional embeddings for audio, resulting in d dimensional embeddings for each modality which are then concatenated.

Condition

Generated output

Generated output

Condition

(a) AV continuation for 1.5 seconds

Condition

Generated output

Condition

(b) Audio-conditioned video continuation

Condition

Generated output

Condition

Condition

Generated output

Condition

Video

Audio

Video

Audio

Video

Audio

(c) Multimodal interpolation

Condition

Generated output

Condition

Generated output

Condition

Generated output

Condition

Generated output

Video

Audio

5 Related Work

Video diffusion models. Diffusion models have revolutionized image [45, 35, 16, 39] and video generation with pixel-space [18, 17, 43] and latent-space [14, 56, 6, 9, 12] approaches. Recently, W.A.L.T [12] pushed the boundaries using transformer-based latent diffusion with joint image-video training. Tackling diverse audio-video generation tasks remains largely unexplored. With an A VDiT trained with MoNL, our unified approach empowers a single model to handle a range of tasks. Audio generative models. Audio generation soared with WaveNet [36]'s autoregressive approach. Adversarial audio generation [27, 42, 26] emerged. Combining this with differentiable quantization [38, 47, 1] led to end-to-end neural codecs for efficient audio compression [21, 57]. Recently, diffusion models joined the fray, some using continuous latent spaces [29, 19, 11], others exploring discrete space [52]. Our A VDiT uses continuous embeddings from SoundStream for audio latents. Multimodal generative modeling. While multimodal diffusion models [37, 41, 54, 39, 17, 23] have achieved impressive results, the field has primarily focused on the visual domain and audiovisual generation remains less explored. Existing approaches for audio-to-video [53, 53, 30] and video-toaudio [20, 33] generation typically learn task-specific conditional models, limiting their flexibility. To address this, recent works [40, 50] propose more versatile audiovisual models. However, they did not examine multimodal interpolation tasks, which we explore in this work. Tasks such as AV-continuation are critical to understand a model's capability to generate a temporally consistent multimodal sequence retaining the object consistency from the condition input.

6 Experiments

6.1 Datasets

Monologues dataset consists of 19.1 million videos for training and 25K videos for evaluation, each with a single person talking. The videos are center-cropped to a 256 GLYPH<2> 256 resolution. This dataset includes a range of person appearances along with rich verbal and non-verbal communication cues. This dataset is ideally situated to assess concepts such as audiovisual gestural synchrony and multimodal expressions which are key components of human communication and interactions.

AIST++ is a subset of AIST [46] and contains 1,020 street dance videos (5.2 hours). The videos were segmented into in 8,233 samples for train and 110 for test at 10fps following Ruan et al. [40]. Landscape contains natural scenes from 928 [28] videos which were segemented into 5,400 samples for train and 600 samples for test at 10fps. We conduct most of the experiments on the Monologues due to its size and diversity. We use AIST++ and Landscape for comparison with MM-Diffusion.

6.2 Evaluation Settings

Tasks. We study three sets of tasks: (1) Joint audio-video (A V) generation ( Joint ): (2) Cross-modal generation: Audio-to-video ( A2V ) and Video-to-audio ( V2A ), and (3) A V interpolation generative tasks: AV-inpaint where a 1.5s clip is interpolated given one video frame, and 0.125s audio at the beginning and four video frames and 0.5s audio at the end, and AV-continue to fill out 1.5 seconds of AV given the first 5 video frames and corresponding 0.625s of audio.

Baselines. On the Monologues dataset, we compare the performance of AVDiT trained with MoNL versus three baselines: Vanilla (Eq. 2), Conditional models separately trained for each task, and the per modality model (Sec 3.2) which may be considered as a generalization of the UniDiffuser approach for sequences. We enabled the Vanilla model to generate cross-modal and multimodal interpolation outputs by using the replacement method [18, 40]. We also benchmark MoNL AVDiT against UNet-based MM-Diffusion (MMD) [40], the sole published work with a released model that tackles both audio and video generation within a single model. While a direct comparison between U-Nets and our transformer architecture is inherently challenging due to their distinct design principles, we show that MoNL AVDiT surpasses this strong U-Net baseline, demonstrating the effectiveness of the transformer architecture in this domain. We restrict our quantitative evaluation to A2V and V2A tasks because MMD fails to generate temporally consistent sequences in case of continuation tasks (see Figs. 2 and 7 for example).

Quantitative evaluation. We use Fréchet video distance (FVD) as our video evaluation metric following Yu et al. [55]. Similarly, we use Fréchet audio distance (FAD) as the audio evaluation metric following Ruan et al. [40]. Because we use latent space representations for the video and audio, we also report the FVD and FAD scores between reconstructed signal and the original signal as 'ground-truth' scores as the performance upper bound. While we preferred user studies for assessing audio-video alignment as existing metrics miss subtle synchrony like dance moves matching music beats or gestures aligning with speech patterns, we computed AV-align score [33], limiting them to the open-domain Landscape dataset for the comparison with MMD.

User studies. We conducted user studies to evaluate the quality of generated content. We adopt the two axes of measurement introduced by Ruan et al. [40] namely audio/video quality and audio-video alignment, and introduce a third one, 'subject consistency' to assess whether the person in the generated content is plausibly consistent with the input. For stimuli, we used a total of 360 generated

samples (not cherry picked) balanced across A2V, V2A, AV-continue and AV-inpaint tasks and for AVDiT trained with three approaches: MoNL, Vanilla and Per modality on Monologues dataset. The tasks were assessed on a 5-point Likert scale. We also compared rater preference for MoNL AVDiT vs. MMD with 30 videos and 5 raters per video. Raters were presented with generations from the two methods randomized as two options, A/B and were asked to pick one option for each of the three dimensions instead of using a 5-point scale. See more details on implementation and experimental setup in Secs. B and C in the Appendix.

6.3 Results

Qualitative results. As displayed in Figs. 1 and 6, Our A VDiT model trained with MoNL achieves impressive performance on various tasks within a single framework, including audio-to-video, videoto-audio, joint generation, multimodal continuation and interpolation with flexible input settings, generating temporally consistent videos. Notably, ours preserves clothing and appearance attributes during continuation tasks, unlike MMD which can alter these (see Figs. 2 and 7). More qualitative results and comparisons are available in Figs. 13, 14 and 15, and at avdit2024.github.io.

Quantitative results. As shown in Table 1, on average across all tasks, A VDiT trained with MoNL outperforms all baselines, demonstrating its versatility to learn diverse conditional distributions in a task-agnostic manner. MoNL excelled at generating samples that are temporally and perceptually consistent with the conditioning input, in the case of AV-inpaint and AV-continue tasks, where other baselines generally failed. Per-modality approach surpassed MoNL for A2V and V2A tasks consistent with the findings in Bao et al. [3] likely because conditional distributions in these cases only need to capture cross-modal associations and not necessarily the underlying temporal dynamics. Unsurprisingly, the vanilla diffusion model trained for joint generation exhibited superior performance in this specific scenario but served as a lower-bound of performance for all other tasks. Finally, MoNL performed better than (if not on-par with) task-specific models for all conditional tasks.

As evident from Table 2, MoNL outperformed MMD in terms of the FAD and FVD metrics across all tasks on the AIST++ and Landscape datasets, as estimated using the code provided by Ruan et al. [40]. The significantly better audio generation in our model, likely due to the combination of MoNL and our choice of the SoundStream audio autoencoder, is also reflected in the ground-truth FAD scores for audio reconstruction. In case of video reconstruction quality, (ground-truth FVD) on AIST++, our choice of autoencoder was inferior to MMD, possibly due to the small dataset size. Qualitatively, we observed that the MAGVIT-v2 reconstructions eliminated flickering across frames but the reconstruction of small face regions in AIST++ dance videos was blurry. These findings should be interpreted cautiously due to several factors: the limited size of the AIST++ and Landscape training splits, our use of a transformer backbone versus MMD's coupled U-Nets, and our use of pretrained autoencoders for latent space representations. On the Landscape dataset, A V-align results demonstrate that our model achieves better alignment compared to MMD, which aligns with the findings from the user study below.

Person consistency AV alignment AV quality * * * * Rating (1: poor - 5: good) Figure 8: Comparative analysis across A VDiT models from the user study on AV quality, AV alignment and person consistency. The * indicates statistically significant pairwise difference at p GLYPH<160> 0 . 01 after multiple correction. User studies. A comparison of the distribution of Likert scores across all tasks for the three approaches we compared is shown in Fig. 8. Pairwise Mann-Whitney U tests were conducted with Bonferroni correction for multiple comparisons to assess statistical difference. Across all axes, raters preferred samples generated from MoNL over that of Vanilla or Per-modality ( Pm ) approaches. Examining task-specific trends (see Fig. 12 in the Appendix), for the cross-modal tasks, Pm was rated significantly higher than Vanilla, and there was no significant difference between MoNL and Pm (except for the V2A task on AV alignment). For multimodal interpolation tasks, MoNL was rated significantly higher than Pm . In line with quantitative results, these results suggest that MoNL excelled at generating samples, that are perceptually and temporally consistent with the input conditioning. 5 3 2

Vanilla

Per-modality (UniDiffuser)

As indicated in Table 3, our MoNL AVDiT outperformed MMD in user studies, especially in consistency where ours showed improved consistency along factors such as person's appearance

or the attire. MMDwas preferred slightly more in V2A tasks, possibly because the Soundstream audtoencoder we used for audio was not optimized for music generation like in MMD.

Ablations. Recall that MoNL training randomly selects one of the four timestep designs described in Sec. 3.2. We compare MoNL with Per time-segment Pt , and Per time-segment and Per modality Ptm and Pt/Pm/Ptm that excludes vanilla from the timestep mixture, approaches separately as shown in Table 1. Overall, Ptm noise excelled at inpainting and continuation tasks though it was not on par with per-modality approach for cross-modal tasks. In general, Pt does not perform well by itself. In our experiments, we also observed that the combination of Pm , Pt , Ptm and Pt/Pm/Ptm was sufficient for comparable performance on most tasks except for unconditional joint generation. Adding the Vanilla approach to the mixture of timesteps improved performance for unconditional joint generation while not substantially compromising the performance on other tasks.

7 Conclusion

We propose a unified approach for multimodal diffusion using a mixture of noise levels ( MoNL ) for generating and manipulating sequences across modalities and time. This empowers a single model to handle diverse tasks like audio-video continuation, interpolation, and cross-modal generation. We show that an audiovisual latent diffusion transformer ( AVDiT ) trained with MoNL achieves state-of-the-art performance in audiovisual-sequence generation, providing new opportunities for expressive and controllable multimedia content creation.

See Sec. A in the Appendix for discussions on limitations and considerations.

References

Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Luc V Gool. Soft-to-hard vector quantization for end-to-end learning compressible representations. Advances in neural information processing systems , 30, 2017.

Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In ICLR , 2022.

Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint arXiv:2303.06555 , 2023.

Reza Bayat. A study on sample diversity in generative models: Gans vs. diffusion models. 2023.

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020.

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR , 2023.

Sander Dieleman. Noise schedules considered harmful. https://sander.ai/2024/06/14/ noise-schedules , 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020.

Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22930-22941, 2023.

Mariana-Iuliana Georgescu, Eduardo Fonseca, Radu Tudor Ionescu, Mario Lucic, Cordelia Schmid, and Anurag Arnab. Audiovisual masked autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16144-16154, 2023.

Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using instruction-tuned LLM and latent diffusion model. arXiv preprint:2304.13731 , 2023.

Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models, 2023.

Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 976-980. IEEE, 2022.

Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 , 2023.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS , 2021.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020.

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv:2210.02303 , 2022.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In ICLR Workshop on Deep Generative Models for Highly Structured Data , 2022.

Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-An-Audio: Text-to-audio generation with prompt-enhanced diffusion models. International Conference on Machine Learning , 2023.

Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation. arXiv preprint arXiv:2110.08791 , 2021.

Srihari Kankanahalli. End-to-end optimized speech coding with deep neural networks. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 2521-2525. IEEE, 2018.

Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr z 'echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466 , 2018.

Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2426-2435, 2022.

Thomas Koelewijn, Adelbert Bronkhorst, and Jan Theeuwes. Attention and the multiple stages of multisensory integration: A review of audiovisual studies. Acta psychologica , 134(3): 372-384, 2010.

Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125 , 2023.

Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems , 33:17022-17033, 2020.

Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre De Brebisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems , 32, 2019.

Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Chanyoung Kim, Won Jeong Ryoo, Sang Ho Yoon, Hyunjun Cho, Jihyun Bae, Jinkyu Kim, and Sangpil Kim. Sound-guided semantic video generation. In ECCV , pages 34-50, 2022.

Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. International Conference on Machine Learning , 2023.

Jiawei Liu, Weining Wang, Sihan Chen, Xinxin Zhu, and Jing Liu. Sounding video generator: A unified framework for text-guided sounding video generation. IEEE Transactions on Multimedia , 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

Zhou Lu. A theory of multimodal learning. Advances in Neural Information Processing Systems , 36:57244-57255, 2023.

Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized videoto-audio synthesis with latent diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Kevin G Munhall, P Gribble, L Sacco, and M Ward. Temporal constraints on the mcgurk effect. Perception & psychophysics , 58:351-362, 1996.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML , 2021.

Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 , 2016.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. ArXiv preprint , 2022.

Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems , 32, 2019.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10219-10228, 2023.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS , 2022.

Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari. Statistical parametric speech synthesis incorporating generative adversarial networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 26(1):84-96, 2017.

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022.

Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICLR , 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Shuhei Tsuchida, Satoru Fukayama, Masahiro Hamasaki, and Masataka Goto. AIST dance video database: Multi-genre, multi-dancer, and multi-camera database for dance information processing. In ISMIR , 2019.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS , 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Vikram Voleti, Alexia Jolicoeur-Martineau, and Christopher Pal. Masked conditional video diffusion for prediction, generation, and interpolation. In NeurIPS , 2022.

Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. arXiv preprint arXiv:2402.17723 , 2024.

Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7754-7765, 2023.

Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 31:1720-1733, 2023.

Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to-video generation via text-to-video model adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 6639-6647, 2024.

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv:2206.10789 , 2022.

Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737 , 2023.

Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18456-18466, 2023.

Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Trans. on Audio, Speech, and Language Processing , 30:495-507, 2021.

Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 13041-13049, 2020.

Appendix

A Limitations, Impact and Considerations

Our proposed approach, combining mixture of noise levels (MoNL) with the generative capabilities of the Audiovisual diffusion transformer (AVDiT), has certain limitations. As shown with demo videos on avdit2024.github.io, while our models effectively capture subject consistency and intricate nonverbal behaviors such as gestural synchrony with vocal tone, significant improvements are necessary to enhance visual and speech quality. Future research will concentrate on super-resolution systems to address visual quality, while text conditioning could potentially further optimize speech quality.

A key focus in this work was to demonstrate the versatility and use of MoNL across various tasks using a simple mixing scheme by randomly choosing between different timestep candidates as representative schemes for applying variable noise levels. This simplicity showcases its broad applicability. We acknowledge that fine-grained controlling by weighted mixing of the different schemes could be explored for specific goals or tasks in future work. In fact, one can imagine an arbitrarily large number of timestep candidates in the vector space of the inputs. We specifically chose a simple mixture scheme to demonstrate its versatility as a proof of concept, rather than optimize for any single task.

Although the method presented in this work is for general multimodal applications, our experiments included human-centric generation tasks. This enabled us to explore unique challenges of that problem setting. For example, consider the case of perceptual expectations for audiovisual alignment/coherence, where misaligned audio and visual cues can drastically alter perception of speech [34, 24]. Generation of photo-realistic persons, speech, and joint generation of both can perpetuate stereotypes. We recognize the ethical concerns and underscore that our goal here is to explore how understanding aspects such as nonverbal behavior in multimodal communications using generative models can open up new avenues in research.

Specifically, the A2V and V2A tasks in human-centric context, which involve extrapolating visual appearance from speech and vice versa, have the potential to perpetuate stereotypes. The generated samples are derived from the model's understanding of cross-modal associations in the training dataset, which can be vastly different from human perception. One possible mitigation is to ensure that the model can generate diverse outputs for a given input. Diffusion models can achieve this by utilizing different noises at inference time, given a sufficiently large and diverse training dataset. A recent study showed that Diffusion models demonstrate better sample diversity in generations compared to GANs [4], however, addressing potential issues around mode collapse in the generative models, especially with multimodal data is an open research problem.

Our work also introduced 'consistency" to qualitatively assess whether the generation remains congruent with the input conditioning. In continuation and interpolation tasks, the disparity between what the model generates and human perception can be generally minimal, as the conditioning provides a perceptual template for the subject's potential appearance or voice. In contrast, A2V and V2A tasks warrant an in-depth analysis of this disparity. As our immediate goal was to assess the capability of our proposed approach (and baselines) to generate samples from various conditional distributions, we focused on a broad definition for measuring consistency in user studies. Our future work will focus on extending the consistency measure for a granular understanding of these biases by (1) comparing cross-modal associations in the training data to that of the generated samples, and (2) disaggregate model and human evaluations in cross-modal generation tasks by identifying specific dimensions of human appearance attributes like perceived gender expression or human communication aspects such as voice and gestural synchrony using diverse rater pools.

B Implementation Details

Autoencoders and AVDiT. Given the domain specific nature of the datasets. we trained datasetspecific MAGVIT-v2 autoencoders following Yu et al. [55]. For the Monologues dataset, we downsampled the data to 8fps and 128 GLYPH<2> 128 resolution for video and 16kHz for audio and randomly

sampled a contiguous clip of 2.125 second (17 frames) to match the input requirements of MAGVITv2. This resulted in a dataset of about 11.8K hours for training. The spatial and temporal video compression ratios were set to r s GLYPH<16> 8 and r t v GLYPH<16> 4 , whereas the temporal audio compression ratio was r t a GLYPH<16> 320 . The embedding dimension of the video and audio latent spaces are d v GLYPH<16> 8 and d a GLYPH<16> 1024 respectively, with the target embedding dimension after linear projection, d GLYPH<16> 1024 . All latents were zero-mean and unit-variance normalized with empirical mean and variance estimates on a small subset. AVDiT has 24 transformer layers with 16 heads with MSA with a total of 420M parameters.

Diffusion training and inference. During training, we use a linear noise variance schedule and a diffusion step T GLYPH<16> 1000 , and a self-conditioning rate of 0.9 following Gupta et al. [12]. At the inference time, we use 250 DDIM steps. All models were trained for about 400K steps with a batch size of 256. We used the AdamW optimizer [31] with a learning rate of 5e-4, 5K warm-up steps, cosine learning rate scheduler and EMA consistent with the denoising transformer setting in Gupta et al. [12].

Compute resources. Each experiment listed in Table 1 was conducted using 256 v5e TPU chips (with 16 GLYPH<2> 16 topology) for training (on average, the models were trained for around 350K steps with a batch size of 256 for around five days); inference was conducted using 16 v5e TPU chips with a topology of 4 GLYPH<2> 4. See https://cloud.google.com/tpu/docs/v5e for more details. Benchmarking experiments to compare with MM-Diffusion on the AIST++ and Landscape datasets were conducted using two A100 GPUs for conditional inference and estimating FAD/FVD metrics.

C Experimental Details

C.1 Evaluation metrics

Since our primary use case is speech generation with the Monologues dataset, we use VGGish embeddings as feature for FAD estimation [22] for the results reported in Table 1. For A V interpolation generative tasks (AV-continue and AV-inpaint), we carefully excluded the conditioning AV frames while estimating Fréchet metrics.

C.2 Comparison with MM-Diffusion

In order to conduct a fair comparsion to the results reported in MM-Diffusion publication [40], we use the data preprocessing and evaluation code provided at github.com/researchmm/MM-Diffusion for FVD, FAD and KVD metrics. Note that the FAD computation here does not use VGGish embeddings. Instead, it uses AudioCLIP [13] which was trained for general sound classification tasks and not suitable for speech generation tasks as in the Monologues dataset reported in Table 1.

For the FAD, FVD and KVD results reported in Table 2, we match training conditions for the input image resolution and video FPS with Ruan et al. [40], i.e., 64 GLYPH<2> 64 resolution images at 10fps. We match the duration of audio-video from both models to 2 seconds. For visualization purpose in Fig. 2, we also train our models with 256 GLYPH<2> 256 resolution to match the super-resolution output resolution used by MM-Diffusion.

Ruan et al. [40] introduce a method for implementing zero-shot transfer of A2V and V2A tasks, inspired by the reconstruction-guided sampling proposed by Ho et al. [18]. For instance, in V2A tasks, the generated noisy audio ˜ a t is computed at each step as follows:

where a t GLYPH<0> 1 , ˆ v t GLYPH<0> 1 are a N -length sequence of generated noisy audio and conditioned noisy video at t GLYPH<0> 1 , θ av is a parameterized denoising step, and λ is a gradient weight. Similarly, the zero-shot transfer of A V-continuation task using the reconstruction-guided sampling [18] can be described by the following equations:

audio video

audio video

Figure 9: Example stimuli shown to the raters for the user study. We conducted user studies for four tasks, A2V, V2A, audiovisual continuation and multimodal interpolation tasks. One track each for the audio and video modality below the stimulus video were shown to effectively convey the portions that were generated (in green) and condition input (gray).

where n c is the number of conditioned time-segments. We closely follows their V2A codebase to faithfully execute the continuation task. We adopt λ GLYPH<16> 0 . 02 to prevent numerical instability, as the results tend to diverge for λ ¡ 0 . 02 .

C.3 Qualitative Evaluation

Examples of the video stimulus template shown to the raters is presented in Fig. 9. The rater instructions provided for each axis of quality, alignment and consistency are shown in Tables 4, 5 and 6 respectively.

Algorithm 1 Sampling of a diffusion timestep vector

Algorithm 2 Training with MoNL

input q p z 0 q , glyph[epsilon1] θ , type (timestep sample type) 1: repeat 2: z 0 GLYPH<18> q p z 0 q 3: glyph[epsilon1] GLYPH<18> N p 0 , I q 4: t GLYPH<16> GETTIMESTEPVEC p type q 5: for m GLYPH<16> 1 , . . . , M do 6: for n GLYPH<16> 1 , . . . , N do 7: z p m,n q t p m,n q GLYPH<16> a α t p m,n q z p m,n q 0 GLYPH<0> a 1 GLYPH<1> α t p m,n q glyph[epsilon1] p m,n q 8: end for 9: end for 10: Take gradient step on 11: ∇ θ } glyph[epsilon1] θ pr z p 1 , 1 q t p 1 , 1 q , . . . , z p M,N q t p M,N q s , t q GLYPH<1> glyph[epsilon1] } 2 2 12: until converged

Algorithm 3 Joint generation of z 0

2: for τ GLYPH<16> T, . . . , 1 do

4: t P R M GLYPH<2> N GLYPH<16> τ I

6: end for

7: return ˆ z 0

Algorithm 4 Cross-modal generation of ˆ z 0 P R p M GLYPH<1> m c qGLYPH<2> N GLYPH<2> d conditioned on z 0 P R m c GLYPH<2> N GLYPH<2> d

5: t p m c GLYPH<0> 1: M, 1: N q GLYPH<16> τ I

8: end for

9: return ˆ z 0

D Algorithms

The core algorithms for implementing mixture of noise levels ( MoNL ) are presented in three parts: sampling of diffusion timestep vector (Algorithm 1), training process (Algorithm 2), and joint/crossmodal generation at inference time (Algorithms 3 and 4). The sampling algorithms are flexible, using DDPM [16] as an example, and can be replaced with other efficient learning-free samplers like DDIM [45] or Analytic-DPM [2]. Notably, conditional generation across time-segments is simply a change of axis in Algorithm 4.

E Gratis Classifier-Free Guidance

Classifier-free guidance (CFG) [15], a technique designed to enhance the quality of samples produced by conditional diffusion models using a linear combination of the conditional and unconditional outputs as follows:

where s is a guidance scale and the conditional and unconditional outputs are denoted by glyph[epsilon1] cond θ and glyph[epsilon1] uncond θ respectively. Typically, a null token ∅ , is used to allow the conditional model to generate unconditional outputs by setting y 0 GLYPH<16> ∅ .

Gratis CFG. CFG (Eq. 13) is supported in our framework at inference without any additional training similar to UniDiffuser [3]. Instead of using a null token for generating unconditional outputs ( glyph[epsilon1] uncond θ in Eq. 13), Gaussian noise is injected to the conditional portions input per task specification, and setting t p m,n q GLYPH<16> T . Conditional outputs glyph[epsilon1] cond θ are obtained as illustrated in Fig. 4. Our vector formulation of the timestep allows us to apply varying levels of noise to different parts of the input. This opens up a number of possibilities for constructing various CFG forms by emphasizing different time segments or modalities, depending on the task at hand.

MoNL supports classifier-free guidance (CFG) without requiring additional design. Unlike the original CFG (see Eq. 13), it does not need a null token either, hence gratis or free. This is achieved by injecting Gaussian noise to the conditional portions of the multimodal space and setting t p m,n q GLYPH<16> T for the output as illustrated in Fig. 10 for the case of cross-modal generation of audio-in, video-out. To illustrate mCFG, consider the conditional output of the network in the cross-modal task (see Eq. 7), denote term used in the gradient step as Z p 1: M, 1: N q t GLYPH<16> r z p 1 , 1 q t p 1 , 1 q , . . . , z p M,N q t p M,N q s and conditional portions as glyph[epsilon1] cond θ GLYPH<16> glyph[epsilon1] θ p Z p 1: M, 1: N q t , t q where

Then, the output for the cross-modal generation task is:

where

where h T GLYPH<18> N p 0 , I q . Then, mCFG operates by blending the conditional glyph[epsilon1] cond θ and unconditional portions glyph[epsilon1] uncond θ per task specification as follows:

where s is a guidance scale.

By formulating the timestep as a vector, we can apply varying levels of noise to different input components. This unlocks diverse possibilities for crafting varied CFG structures. Each structure

t

t

t

t=0

t=T

20

(1:mc,1:N)

2T

(1:mc,1:N)

Jointnoise

Jointnoise

prediction

.cond

prediction

uncond

network

03

network

Eo

Eθ

(mc+1:M,1:N)

zi

(a) Conditional output

(b) Unconditional output

f

t=0

t=T

2T

(1:M,1:nc)

(1:M,nc+1:N)

Jointnoise

t

prediction

→EO

uncond

network

preaiction

network

Eθ

(1:M,nc:N)

(1:M,nc+1:N)

lointnoise

tvariat

f

(b)Unconditionaloutputvariation1

Jointnoise

prediction

cond

t

network

E

20

(1:M,1:nc)

(1:me,nc:N)

(mc+1:M,nc+1:N)

ut

Jointnoise

prediction

3

uncond

network

E0

preaiciol

network

Eθ

can amplify specific time segments or modalities based on the task demands, as demonstrated in Fig. 10 for cross-modal tasks and Fig. 11 for the case of multimodal interpolation generation driven by temporal conditioning.

F Discussion

Limited conditioning information: In Table 7, we compares the results of AV continuation depending on the input information: AV-continue-2s ) to fill out 2 seconds of AV given the first video frame and corresponding 0.125s of audio AV-continue-1.5s ) to fill out 1.5 seconds of A V given the first 5 video frames and corresponding 0.625s of audio. The model performed better when given more context (5 video frames and corresponding audio) compared to less context (1 frame and corresponding audio), even though task-specific training can be an upper bound for performance. This suggests that limited conditioning information can lead to issues like unnatural motion and inconsistencies, and including more context improves the model's performance.

Setting/Task

AV-continue-2s

AV-continue-1.5s

FAD

GLYPH<211>

FVD

GLYPH<211>

FAD

GLYPH<211>

FVD

GLYPH<211>

Conditional (task-specific)

8.2

117.3

7.4

12.1

Per modality

5.8

120.5

4.5

92.1

Vanilla

7.5

142.6

6.1

140.8

3.6

12.9

3.1

8.8

Per time-segment

6.7

102.5

5

72.3

Per time-segment Per modality

5.8

82.8

4.8

61.2

Pt/Pm/Ptm

4.1

20.2

4.1

16.9

*

*

*

Person consistency

AV alignment

AV quality

Rating (1: poor - 5: good)

AV-continue

A2V

V2A

Person consistency

Vanilla

Per-modality (UniDiffuser)

AV alignment

AV quality

AV-interp

*

*

*

*

*

*

*

*

*

*

*

*

Person consistency

AV alignment

AV quality

Person consistency

AV alignment

AV quality

5

5

4

4

3

3

2

2

1

5

5

4

4

3

3

2

2

Condition

Generated output

Condition

Generated output

Video

Audio

AV continuation for 2 seconds

1

1

1

Generatedoutput

Generatedoutput

Sontgeneration

Condition

Generatedoutput

Condition

Generatedoutput

Aveortniuatior

Condition

Generatedoutput

Condition

Condition

Generatedoutput

Condition

育

Generatedoutput

Condition

Auaio-to-videogeneration

声

市

Condition

Generatedoutput

Generated output

Generated output

Joint generation

Generated output

Condition

Generated output

Condition

AV continuation

Generated output

Condition

Condition

Generated output

Condition

Condition

Multimodal interpolation

Condition

Generated output

Audio-to-video generation

Condition

Generated output

Video-to-audio generation

G Theoretical Background on Mixture of Noise Levels

G.1 Theoretical Background on Multimodal Learning

In "A Theory of Multimodal Learning" [32], multimodal learning is shown to offer a superior generalization bound compared to unimodal learning, with an improvement factor of O p ? n q , where n denotes the sample size. This benefit relies on connection and heterogeneity between modalities:

Connection. The bound depends on learned connections between ( X ) and ( Y ).

Heterogeneity. Describes how modalities X and Y , diverge and complement.

If connection and heterogeneity are missing, ill-conditioned scenarios can arise. For instance, if x GLYPH<16> y , perfect connection suggests no need for learning about Y . On the other hand, if x is random noise, there is heterogeneity but no meaningful connection between X and Y , making non-trivial learning on X alone impractical.

The theory also highlights that and effective connections between modalities via generative models can enhance multimodal learning. This forms the basis for our Mixture of Noise Levels (MoNL) approach, which is particularly suited for multimodal learning with audio and video data.

G.2 Advantages of Mixture of Noise Level Training

Our MoNL training method offers significant benefits for multimodal learning, especially with audio and video data:

Heterogeneity and connection. Audio and video are naturally heterogeneous. For example, a video of a person speaking includes audio of spoken words and video of lip movements and facial expressions. MoNL uses variable noise levels to enhance learning by capturing the generic transition matrix across the temporal axis.

Enhanced connectivity. MoNL improves connectivity between audio and video modalities. Our experiments show that MoNL often surpasses task-specific learning approaches by fostering better connections between modalities, adapting its focus more effectively.

G.3 Enhanced Connectivity - Comparison with Existing Methods

MoNL vs. Joint learning in MMD [40]. Unlike joint learning methods that focus on the joint distribution p θ p z t GLYPH<1> 1 | z t q , MoNL trains across multiple conditioning, enabling better connections by varying its focus. This is evidenced by MoNL outperforming the Vanilla (see Table 1) and MMD models (see Tables 2 and 3).

MoNL vs. Per-modality training. MoNL goes beyond per-modality training in UniDiffuser [3], which uses variable noise between modalities i.e., learning p θ pr z p 1 q t p 1 q GLYPH<1> 1 , . . . , z p M q t p M q GLYPH<1> 1 s | r z p 1 q t p 1 q , . . . , z p M q t p M q sq . MoNL introduces variable noise across different time segments, learning connections across temporal dynamics as well. This advantage is demonstrated in Table 1.

MoNL vs. Masked training [49]. Diffusion models often obscure high-frequency details with low noise and low-frequency structures with high noise [7]. MoNL employs variable noise levels to explore diverse frequency components, enhancing the model's ability to correlate high and lowfrequency elements. This is in contrast to masked self-supervised learning, which limits frequencyspecific connections by masking entire elements.

In summary, the effectiveness of MoNL for multimodal diffusion models, particularly with audio and video data, stems from its strategic use of connection and heterogeneity. By applying variable noise levels, MoNL enhances connectivity between modalities and better adapts to diverse temporal and frequency components, leading to superior performance compared to existing multimodal learning methods.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The two main claims of the paper, i.e., task-agnostic training to support a range of audiovisual generation tasks at inference time, and the ability of our model to generate temporally consistent generations have been empirically validated through quantitative and user studies, along with demo videos on the project page.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have added a detailed discussion of limitations, potential future work and considerations with respect to human-centered generation experiments and evaluations in the Appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: There are two main components of reproducibility here: (1) Model architecture is described in detail and the pseudocode for the proposed algorithm has been provided. (2) Evaluation: The Fréchet metrics are implemented using publicly available code, and the design of the user interface for we used as stimulus for rater studies are shared in the Appendix along with the rubric of the measurements used by the raters.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The datasets we have used to benchmark with baselines are already publicly available. The Monologues dataset on which we conduct our core experiments is not publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: A summary is provided in the main text of the paper, and the implementation details are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For qualitative studies, where we examined rater labels of quality or preferences, we describe the statistical test along with the p-value at which the statistical significance is estimated after multiple-comparison correction is also reported.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The implementation details including compute resources are described in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully considered some of the potential societal impacts of this work and potential harms, and discussed possible mitigations in the section "Limitations, Impact and Considerations" in the Appendix.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed broader impact of the paper in the section "Limitations, Impact and Considerations" in the Appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: Potential harms and possible mitigations are discussed in the Appendix. The video demos shared for demonstrating non-verbal audiovisual behaviors are from a consented individual.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators of the benchmarking datasets and the code we've used in this paper have been approproately cited along with the github URLs where the code was provided by the original authors.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Pseudocode relevant to reproducing the implementation of the proposed algorithm is available in the appendix. Demo videos showing recognizable persons are from a consented individual from whom written consent was obtained.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: Detailed instructions along with the user interface ahown to the raters and the rubric used for rating tasks are provided in the Appendix. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new diffusion model that is capable of generating data following various conditional distributions in multimodal data. Rather than applying a uniform noise level to all data elements, the proposed model permits the use of differing noise levels across modalities and time dimensions. This allows the model to concurrently learn a variety of conditional distributions, including cross-modal conditioning, temporal interpolation, and temporal continuation. The experimental results with audio-video datasets validate the effectiveness of the proposed model.\n\nSTRENGTHS:\n- The idea of MoNL is simple and reasonable. It definitely enables the model to effectively learn various conditional distributions simultaneously.\n\n- The proposed method performs well in the experiments. MoNL particularly contributes to boost the performance of the model in AV-inpaint and AV-continue tasks.\n\n- The manuscript is well-written and easy to follow.\n\nWEAKNESSES:\n- The major concern is on the experiments: the majority of the empirical evaluations are conducted with the internal dataset. As there are already several publicly-available datasets (such as Landscape, AIST++, and VGGSound) that are commonly used in the literature, I strongly encourage the authors to do at least the ablation studies with such datasets to make them reproducible. \n\n- The novelty in methoology is somewhat limited. The idea of using various noise levels in multimodal generative modeling has been presented in the prior work [3]. A particular challenge in this paper is to extend this idea also for modeling temporal dynamics, but similar strategies are commonly adopted in several video generation studies:\n    - “Diffusion Models for Video Prediction and Infilling,” TMLR 2022.\n    - “MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation,” NeurIPS 2022.\n\nQUESTIONS:\n- In the ablations at Table 1, why is Vanilla not used? Without the Vanilla strategy, the model may not encounter cases where all elements are almost entirely noise or close to clean ones during training, which could potentially degrade the performance of the model to some extent. Therefore, a more reasonable approach for ablation might be to gradually add other strategies to Vanilla.\n\n---\n\n<post-rebuttal>\n\nI have updated my rating from 4 to 5, as my concern on the dataset is somewhat resolved through the discussion.\n\nLIMITATIONS:\nLimitations are properly discussed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the audio-visual cross-modality generation problem and proposes a training approach to learn arbitrary conditional distributions in the audiovisual space. At the methodological level, the authors propose to apply variable diffusion timesteps across the temporal dimension. The experiments are conducted on Monologues, AIST++, and Landscape datasets.\n\nSTRENGTHS:\nThe high-level motivation to learn arbitrary conditional distributions in the audiovisual space is interesting. The experiments show promising results and the authors conduct user studies as supplementary evaluations.\n\nWEAKNESSES:\n1. The writing and presentation of the paper can be improved. Figure 1 seems to have some issues with the first-row caption for the AIST dataset, which makes it difficult to read. The usage of math symbols is inconsistent, e.g., the $x$ should be $\\bf{x}$ in Line 72. The term “multivariate data” in Line 63 is also not rigorous and confusing, in other words, do the authors imply that the static image data is uni-variante? The use of $\\equiv$ and $=$ is also mixed in the paper.\n\n2. One of the key claims and motivations, “Training separate models for each variation is expensive and impractical”, seems controversial and susceptible to me, while I understand training separate DMs for audio and visual data could be expensive, I don’t think that learning a single mixture is a better option than learning two separate data distributions, because learning a mixture increase the complexity of target data distribution, and thus should not be beneficial for performance especially if the domain gap between separate modality is large. As this is one of the key claims that motivates the methodological design, I was expecting more rigorous theoretical support in the paper, could the authors elaborate on that?\n\n3. For the experiments, while the authors claim that “the sequential representations can be either latent spaces or raw data”, the actual implementation only conducts experiments on low dimensional features of pre-trained models, i.e., MAGVIT-v2 for video representations and SoundStream for audio representations. While this is not a serious flaw, it raises questions about the generalizability of the proposed method, and the fairness in comparison with other methods, as the pre-trained models may (and very likely) directly influence the final performance. And the baseline method MM-Diffusion seems not to operate on the same feature space according to Appendix C?\n\nQUESTIONS:\nIn addition to the comments in the Weaknesses section. \nI may have missed several details: how is the scheduler defined in the proposed method Eq. (5)? What is the time cost for training the proposed model? And how does this design yield a difference compared to training on single modality or separate DMs?\n\nLIMITATIONS:\nThe limitations are discussed in the Appendix A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Audiovisual Diffusion Transformer (AVDiT) with Mixture of Noise Levels (MoNL) for audiovisual sequence generation. The key innovation is the use of variable noise levels during the diffusion process, applied across different time segments and modalities. This approach enables the model to effectively learn arbitrary conditional distributions in a task-agnostic manner, making it versatile for various generation tasks such as cross-modal generation, multimodal interpolation, and audiovisual continuation. Experiments on multiple datasets demonstrate that AVDiT with MoNL outperforms existing baselines, generating temporally and perceptually consistent audiovisual sequences.\n\nSTRENGTHS:\n1. The proposed variable noise levels across different time segments and modalities is an effective approach to enhance the flexibility of diffusion models.\n2. The model's ability to handle various audiovisual generation tasks within a single framework is impressive.\n3. The paper includes extensive experiments on multiple datasets, with both qualitative and quantitative evaluations. The additional demo page provides an intuitive comparison.\n\nWEAKNESSES:\n1. While the variable noise levels concept is interesting, the overall novelty of the approach may be seen as incremental. Similar techniques in diffusion models and transformers have been explored in papers like MM-Diffusion.\n2. Some technical details are not thoroughly explained, such as the criteria for selecting and varying noise levels across time segments and modalities.\n3. The evaluation is primarily conducted on a few datasets (Monologues, AIST++, Landscape). The model's generalizability to real-world scenarios remains uncertain.\n4. The authors compare the transformer-based AVDiT model to the UNet-based MM-Diffusion model, which might not be a fair comparison due to the different architectures. The authors should consider training a UNet-based model using their proposed approach to provide a more direct comparison and validate the effectiveness of their method.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nThe authors mention some limitations in the appendix, such as the need for further improvements in visual and speech quality and the potential for ethical concerns. However, for me, the lack of extensive evaluation of more diverse and real-world datasets is a noteworthy limitation. Some additional limitations can be found in the Weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new method for audiovisual generation where the input output condiitions may comprise of 2 modalities, namely video and audio sequence. The authors propose a new training approach to effectively learn conditional distribution in multimodal space. The main novely in the paper is a mixture of noise level formulation for processing the inputs. The method produces temporally consistent samples and outperforms existing arts and the vanilla configuration\n\nSTRENGTHS:\n1. The utilization of mixture of noise levels is novel and the methods seems to improve robustness in denoising, hence leading to a performance boost\n2. The evaluations seems to be fair and clearly signifies the working of the method.\n3. The paper is well written and easy to follow.\n\nWEAKNESSES:\n1. Although the method seems to work well empirically, the paper lacks theoretical backing for the proposed method. It would be good to see some proofs that the proposed method leads to a better approximation of the variational lower bound and joint distribution.\n2. Are there any sampling modifications required to accommodate the proposed training strategy?\n3. There is a plethora of compositional works derived from the energy based formulation of diffusion models. Could the authors analyze how the proposed method performs in comparison to it.\n[1] https://energy-based-model.github.io/Compositional-Visual-Generation-with-Composable-Diffusion-Models/\n4. I'm rating the paper borderline for now, I will improve my rating if the authors can address my concerns\n\nQUESTIONS:\n1. Could the authors provide a proof of why the proposed method would work better\n2. Are there any sampling modifications required to accommodate the proposed training strategy?\n3. Could the authors give an analysis of how the proposed algorithm will work when compared to [1]\n\nLIMITATIONS:\nPlease see weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-29"}