Any-to-Any Generation via Composable Diffusion
Authors: Zineng Tang
Ziyi Yang
Chenguang Zhu
Michael Zeng
Mohit Bansal
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative AI
Diffusion Model
Multimodal Generation
Audio-Video Generation
TL;DR: An AI that can generate from any combinations of modalities to any combinations, including text, image, video and audio.

Abstract: We present Composable Diffusion (CoDi), a novel generative model capable of generating any combination of output modalities, such as language, image, video, or audio, from any combination of input modalities. Unlike existing generative AI systems, CoDi can generate multiple modalities in parallel and its input is not limited to a subset of modalities like text or image. Despite the absence of training datasets for many combinations of modalities, we propose to align modalities in both the input and output space. This allows CoDi to freely condition on any input combination and generate any group of modalities, even if they are not present in the training data. CoDi employs a novel composable generation strategy which involves building a shared multimodal space by bridging alignment in the diffusion process, enabling the synchronized generation of intertwined modalities, such as temporally aligned video and audio. Highly customizable and flexible, CoDi achieves strong joint-modality generation quality, and outperforms or is on par with the unimodal state-of-the-art for single-modality synthesis.

Any-to-Any Generation via Composable Diffusion
Zineng Tang'*
Ziyi Yang2t
Chenguang Zhu Michael Zeng²
Mohit Bansal It
"University of North Carolina at Chapel Hill Microsofft Azure Cognitive Services Research https://code gen. github.
Abstract
We present Composable Diffusion (CoDi), a novel generative model capable of generating any combination of output modalities, such language, image, video, or audio, from any combination input modalities. Unlike existing generative AI systems, CoDi can generate multiple modalities in parallel and its input is not limited to a subset modalities like text or image. Despite the absence of training datasets for many combinations of modalities, we propose align modalities in both the input and output space. This allows CoDi to freely condition on any input combination and generate any group of modalities, even if they not present in the training data. CoDi employs novel composable generation strategy which involves building shared multimodal space by bridging alignment in the diffusion process, enabling the synchronized generation intertwined modalities, such temporally aligned video and audio. Highly customizable and flexible, CoDi achieves strong joint-modality generation quality, and outperforms or is on par with the unimodal state-of-the-arr for single-modality synthesis. The project page with demonstrations and code is at https: / //w////ooii///////////// gen, github.io/
"A toy on the street sitting board"
(Raining street ambience)
"Raining, rain, moderate"
"Teddy bear on skateboard,
CoDi
(Raining ambience)
(Rain ambience, street naise, skateboard sound)
Figure 1: CoDi can generate various (joint) combinations of output modalities from diverse (joint) sets inputs: video, image, audio, and text (example combinations depicted by the colored arrows).
WWork done Microsoft internship and UNC.
Conresoonding authors: ziyiyang @ microsoft.com, mbansal wes.sunc.eu.. Work done while Microsoft.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Recent years have seen the rise of powerful cross-modal models that can generate one modality from another, e.g. text-to-text [6, 37], text-to-image [13, 19, 22, 41, 44], or text-to-audio [3]. However, these models are restricted in their real-world applicability where multiple modalities coexist and interact. While one can chain together -ppecift generative models a multi-step generation setting, the generation power of each step remains inherently limited, and a serial, multi- step process can be cumbersome and slow. Moreover, independently generated unimodal streams will not be consistent and aligned when stitched together in processocsssing way (e.g.. synchronized video and audio). The development of a comprehensive and versatile model that can generate any combination of modalities from any set of input conditions has been eagerly anticipated, as it would more accurately capture the multimodal nature of the world and human comprehension, seamlessly consolidate information from wide range of sources, and enable strong immersion in human-Al interactions (for example, by generating coherent video, audio, and text description at the same time).
In pursuit of this goal, we propose Composable Diffusion, or CoDi, the first model capable of simultancously processing and generating arbitrary combinations of modalities as shown in Fig. 1. Training model to take any mixture of input modalities and flexibly generate any mixture of outputs presents significant computational and data requirements, as the number of combinations for the input and output modalities scales exponentially. Also aligned training data for many groups of modalities is scarce or even non-existent, making it infeasible to train with all possible input-output combinations. To address this challenge, propose align multiple modalities in both the input conditioning (Section 3.2) and generation diffusion step (Section 3.4). Furthermore, a proposed "Bridging Alignment" strategy for contrastive learning (Section 3.2) allows us efficiently model the exponential number of input-output combinations with a linear number of training objectives.
Building model with any-to-any generation capacity with exceptional generation quality requires comprehensive model design and training on diverse data resources. Therefore, we build CoDi in an integrative way. First, train latent diffusion model (LDM) for each modality, e.g., text, image, video, and audio. These models can be trained in parallel independently, ensuring exceptional single- modality generation quality using widely available modality-speciff training data (i.e., data with one or more modalities as input and one modality as output). For conditional cross-modalit generation, such generating images using audio+language prompts, the input modalities are projected into shared feature space (Section 3.2), and the output LDM attends to the combination of input features. This multimodal conditioning mechanism prepares the diffusion model to condition on any modality or combination of modalities without directly training for such settings.
The second stage of training enables the model to handle many-to-mane generation strategies that involve simultaneously generating arbitrary combinations of output modalities. To the best of our knowledge, CoDi is the first AI model with this capability. This is achieved by adding cross- attention module to each diffuser, and an environment encoder V to project the latent variable of different LDMs into shared latent space (Section 3.4). Next, we freeze the parameters of the LDM, training only the cross-attention parameters and V. Since the environment encoder of different modalities are aligned, LDM can cross-attend with any group of co-generated modalities by interpolating the representation' 's output by V. This enables CoDi to seamlessly generate any group of modalities, without training on all possible generation combinations. This reduces the number of training objectives from exponential to linear.
We demonstrate the any-to-any generation capability of CoDi, including single-to-single modality generation, multi-condition generation, and the novel capacity of joint generation of multiple modali- ties. For example, generating synchronized video and audio given the text input prompt; or generating video given a prompt image and audio. We also provide a quantitative evaluation of CoDi using eight multimodal datasets. As the latest work from Project i-Code [55] towards Composable AI, CoDi exhibits exceptional generation quality across assorted scenarios, with synthesis quality on par or even better than single single modality SOTA, e.g., audio generation and audio captioning.
2 Related Works
Diffusion models (DMs) learn the data distribution by denoising and recovering the original data. Deep Diffusion Process (DDP) [45] adopts sequence of reversible diffusion steps to model image

---

(1) Conditioning Alignment
(2) Single Diffuser Training
Text Encoder
i
Text
Text UNet
Stage 1 Composable Conditioning (a)
Encoder
Bridging Alignment
Encoder
Vision
C(x)
Vision UNet
"LLSs"" 2
Video Encoder
Image Encoder
Audio Encoder
6 Audio
Audio UNet
Encoder
LOWE
(1) Conditioning
(2) Diffusion Step(s)
(3) Latent Alignment at Step
Text Environment Encoder
Text UNet ||00
Text Encoder
Text UNat
Stage 2 Joint Generation
s Vision Encoder
(xx)
Vision UNeet
"LOSS""
Bridging Aligemant
Visiom Environment Encoder
Vu(43)
Vision UNet
|0o0
(b)
e Audio
Encoder
Audio UNot
Casa
Audio Environment
V.(17)
Audio UNer []000]
diffusion steps
Vision Encoder
Decode
Inference (c)
Vision UNet
C(x)
Text
Audio UNet
Decode
Encoder
Figure 2: CoDi model architecture: (a) We first train individual diffusion model with aligned prompt encoder by "Bridging Alignment"; (b) Diffusion models learn attend with each other via "Latent Alignment"; (c) CoDi achieves any-to-any generation with linear number of training objectives.
probability distribution. It a reversible encoder to map the input image latent space and decoder to map the latent variables to an output image. Denoising diffusion probabilistic model (DDPM) [20] uses a cascade of diffusion processes to gradually increase the complexity of the probability density function model. At each step, the model adds noise to the input image and estimates the corresponding noise level using an autoregressive model. This allows the model to capture the dependencies between adjacent pixels and generate high-quality images. Score- based generative models (SOG) [46] use the score function model the diffusion process. [40] generates high-fidelity images conditioned on CLIP representations of text prompts. Latent diffusion model (LDM) [41] uses a VAE to encode inputs into latent space to reduce modeling dimension and improves efficiency. The motivation is that image compression can be separated into semantic space by a diffusion model and perceptual space by an autoencoder. By incorporating temporal modeling modules and cascading model architectures, video diffusion models have been built upon image diffusers to generate temporally consistent and inherent frames[ 14, 19, 21, 44]. Diffusson models have also been applied to other domains, such generating audio from text and vision prompts[23, 33].
Multimodal modeling has experienced rapid advancement recently, with researchers striving to build uniform representations of multiple modalities using single model to achieve more comprehensive cross-modal understanding. Vision transformers [11], featuring diverse model architectures and training techniques, have been applied to various downstream tasks such as vision Q&A and image captioning. Multimodal encoders have also proven successful in vision-language [1, 8, 57], video- audio [47] and video-speech-languae
[55, 56] domains. Aligning data from different modalities is an active research area [12, 38], with promising applications in cross moddaltyy retrieval and building uniform multimodal representations [33, 35, 41].
3 Methodology
3.1 Preliminary: Latent Diffusion Model
Diffusion models (DM) represent class of generative models that learn data distributions p(æ:) by simulating the diffusion of information over time. During training, random noise is iteratively added

---

to 2, while the model learns to denoise the examples. For inference, the model denoises data points sampled from simple distributions such Gaussian. Latent diffusion models (LDM) [41] learn the distribution of the latent variable z corresponding to significantly reducing computational cost by decreasing the data dimension.
In LDM, an autoencoder is first trained to reconstruct æ, i.c., ã = DEE(z)), where E' and D denote the encoder and decoder, respectively. The latent variable E(x)) is iteratively diffused over time steps t based on a variance schedule 31, Sr, i.e., ((z,) 2t = N (iz;; v1 /1- 1,, [20, 45].
The forward process allows the random sampling of 2, at any timestep in a closed form [20, 45]:
2 = 0,2 + ose, where e ~ NN(, I). a= 1 - 8r and o, == 1 II' Os- The diffuser learns how to demoise from {z,} } to recover z. Following the reparameterization method proposed in [20], the denoising training objective can be expressed as [41]:
LD = ,,,,,, 6o (z 1, (
(1)
In data generation, the denoising process can be realized through reparameterized Gaussian sampling:
3c
pZzz N 21-15 2:
60 B.11
In LD. the diffusion time step ~ U1,, is denoising model with UNet backbone parame- terized by 0; y represents the conditional variable that can be used to control generation; C' is the prompt encoder. The conditioning mechanism is implemented by first featurizing y into C(yy, then the UNet Es conditions on C(y) via cross-attention, as described in [41]. Distinct from previous works, our model can condition on any combinations of modalities of text, image, video and audio. Details are presented in the following section.
3.2 Composable Multimodal Conditioning
To enable our model to condition on any combination of input/prompt modalities, we align the prompt encoder of text, image. video and audio (denoted by C.. C. C.. and C., respectively) to project the input from any modality into the same space. Multimodal conditioning can then be conveniently achieved by interpolating the representations of each modality m: C'zz, Iy, T., Fa) = Em am (m) for m E Iz; 24, Fu, Ta. with Σn, am 1. Through simple weighted interpolation of aligned embeddings, we enable models trained with single-conditioning (i.c., with only one input) to perform zero-shot multi-conditionin (i.e., with multiple inputs). This process illustrated in Fig. 2 (a)(2).
Optimizing all four prompt encoders simultaneously in combinatorial manner is computationally heavy, with Onn?) pairs. Additionally, for certain dual modalities, well-aligned paired datasets limited or unavailable e.g., image-audio pairs. To address this challenge, we propose a simple and effective technique called "Bridging Alignment" efficiently align conditional encoders. As shown in Fig. 2 (a)(1), we choose the text modality as the "bridging" modality due to its ubiquitous presence in paired data, such as text-image, text-video, and text-audio pairs. We begin with a pretrained text-image paired encoder, i.c., CLIP [38]. We then train audio and video prompt encoders on audio-text and video-text paired datasets using contrastive learning, with text and image encoder weights frozen.
In this way, all four modalities aligned in the feature space. As shown in Section 5.2, CoDi can effectively leverage and combine the complementary information present in any combination of modalities to generate more accurate and comprehensive outputs. The high generation quality remains unaffected with respect to the number of prompt modalities. As we will discuss in subsequent sections, continue to apply Bridging Alignment to align the latent space of LDMs with different modalities to achieve joint multimodal generation.
3.3 Composable Diffusion
Training an end-to-end anything-to-anything model requires extensive learning on various data resources. The model also needs to maintain generation quality for all synthesis flows. To address these challenges, CoDi is designed to be composable and integrative, allowing individual modality- specific models to be built independently and then smoothly integrated later. Specifically, start by

---

independently training image, video, audio, and text LDMs. These diffusion models then efficiently learn to attend across modalities for joint multimodal generation (Section 3.4) by a novel mechanism named "latent alignment".
Image Diffusion Model. The image LDM follows the same structure Stable Diffusion 1.5 [41] and is initialized with the same weights. Reusing the weights transfers the knowledge and exceptional generation fidelity of Stable Diffusion trained large-scale high-quality image datasets to CoDi.
Video Diffusion Model. To model the temporal properties of videos and simultaneously maintain vision generation quality, we construct the video diffuser by extending the image diffuser with temporal modules. Specifically, we insert pseudo-temporal attention before the residual block [13]. However, we argue that pscudo-temporal attention only enables video frames to globally attend to each other by flattening the pixels (height, width dimension) to batch dimension, resulting in a lack of cross-frame interaction between local pixels. We argue that this results in the common issue in video generation that locations, shapes, colors, etc. of objects can be inconsistent across generated frames. To address this problem, we propose adapting the latent shift method [2] that performs cemporal-ppttoe shifts on latent features in accordance with temporal attention. We divide the video by the hidden dimension into & 8 chunks, and for each chunk i == 0 to 7, we shift the temporal dimension forward by i positions. Further details will be provided in the appendix.
Audio Diffusion Model. To enable flexible modsltyy attention in joint generation, the audio diffuser is designed to have similar architecture to vision diffusers, where the mel-spectrogram can be naturally viewed as an image with channel. We use a VAE encoder to encode the mel- spectrogram of audio to a compressed latent space. In audio synthesis, VAE decoder maps the latent variable to the mel-spectrogram, and a vocoder generates the audio sample from the mel- -peesppecroor.mm We employ the audio VAE from [33] and the vocoder from [27].
Text Diffusion Model. The VAE of the text LDM is OPTIMUS [29], and its encoder and decoder are [9] and GPT-2 [39]. respectively. For the denoising UNet, unlike the one in image diffusion, the 2D convolution in residual blocks is replaced with ID convolution [53].
3.4 Joint Multimodal Generation by Latent Alignment
The final step to enable cross-attention between diffusion flows in joint generation, i.e., generating two or more modalities simultaneously. This is achieved by adding mross- attention sublayers to the UNet Eg (Fig. 2 (b)(2). Specifically, consider a diffusion model of modality A that cross-attends with another modality B. Let the latent variables of modalities m and m at diffusion step t be denoted as 2 and z respectively. The proposed "Latent Alignment" technique is such that a modality-specifft environment encoder Vy first projects z!" into shared latent space for different modalities. Then, in each layer of the UNet for modality A, a cross-attention sublayer attends to Vz(z,)). For the diffusion model of modality A, the training objective in Eq. (1) now becomes:
Croes R2,c,t/e -,,
where 0g denotes the weights of cross-attention modules in the UNet.
The training objective of + B joint generation is Cârpss + CB Crosss V--) of different modalities trained to be aligned with contrastive learning. Since =1² and = at any time step can be sampled with closed form in the diffusion process Section 3.1, one can conveniently train the contrastive lcarning together with CCross. The purpose of V is to achieve the generation of any combination of modalities (in polynomial) by training on a linear number of joint-generatin trained the joint generation of modalities A, B, and B, C' independently. then we have V (z(2)), VB (z(2,)) ), and Vc(2g) aligned. Therefore, CoDi can seamlessly achieve joint generation of modalities A and C' without any additional training. Moreover, such design automatically effortlessly enables joint generation of modalities A, B, and C concurrently. Specifically, UNet of A can cross-attend with the interpolation of Vn (=(²), and Ve(z{)) although CoDi has not been trained with such task.
tasks. For example, if we have
As shown in Fig. 2(b)(3), we follow similar designs to the "Bridging Alignment" in training joint generation: (1) We first train the cross-an coss-attention weights in the image and text diffusers, well
5

---

Table 1: Training tasks (CT stands for "contrastive learning" to align prompt encoders) and datasets with corresponding statistics. denotes the number of accessible examples in the original datasets.
Categories
Tasks
Datasets
of samples
Domain
Image >Text, Text :Image Texg-Tta
Image Text
Laion400M [42]
400M
Open
AudioSet [16]
900K*
YouTube
Text >udio, Audio yText, Text-Audin+Texx, Audio Text CT
AudioCaps [24]
46K
YouTube
Audio
Text
Freesound 500K
2.5M
Public audio samples
BBC Sound Effect
30K
Authentic natural sound
AudioSet
900K
YouTube
Audiovisual
Image-AAudio, Image- Audio+
SoundNer [3]
1.0M
Flickr, natural sound
Text-Video, Image-+ Video-Text CT
WebvidI0M [4]
10. AM
Short vidcos
Video
HD- ViilV 1004 [54]
100M
YouTube
"mountain view, sunset."
"Concept art by Sylvain
Sarrailh a haunted Japan temple forest"
'A beautifull bailet dancer spinning, view from top.
(Subway sound)
Figure 3: Single-to-single modality generation. Clockwise from top left: text- image-texx image- ivieo- audio mimgee
as their environment encoders on text-image paired data. (2) We freeze the weights the text diffuser and train the environment encoder and cross-attention weights of the audio diffuser on text-audio paired data. (3) Finally we freeze the audio diffuser and its environment encoder, and vidio-vidco paired data. As demonstrated in Section 5.3, although only trained on three paired joint generation tasks (i.e, Text+Audio, Text+Image, and Video+Audio), CoDi is capable of generating assorted combinations of modalities simultaneously that are unseen in training, e.g., joint image-text-audi generation in Fig. 5.
train the joint generation of the video modality
4 Experiments
4.1 Training Objectives and Datasets
We list training tasks of CoDi in Table 1, including single modality synthesis, joint multimodal generation, and contrastive learning to align prompt encoders. Table provides an overview of the datasets, tasks, number of samples, and domain. Datasets are from the following domains: image + text (c.g. image with caption), audio + text (c.g. audio with description), audio + video (c.g. video with sound), and video + text (c.g. vidco with description). As one may have noticed, the language modality appears in most datasets and domains. This echos the idea of using text as the bridge modality to be able to extrapolate and generate new unseen combinations such as audio and image bridged by text, mentioned in Section 3.2 and Section 3.4. Due space limit, more details on training datasets and can found in Appendix C, model architecture details in Appendix Appendix A. and training details in Appendix B.
Image + Text. We use recently developed large-scale image caption dataset, Laion400M [42]. This image-text paired data allows to train with tasks text text=>image, image-ttxt, and the joint

---

Table 2: coCO-caption [32] FID scores for text-to-image generation.
Table 3: MSR- ETTT voxe- Table 4: UCF-101 text-to-video
generation performance.
generation performance.
Method
FID
CogView [10]
27.10
Methed
Zaro-Shot
CLIPSIM
Method
18
FVD (1)
GLIDE
12.24
GODIVA [50]
No
0.24022
CogVideo (Chinese)
23.55
751.34
Make-a-Scene [15]
11.84
NUWA [51]
CogViden (Faglish)
25.27
701.59
LDM [41]
12.63
Make Video [14]
CogVideo [22]
Yes
Yes
0.3049
0.2631
Make Video
33,00
367.23
Stable Diffusion-1,
11.21
Video LDM
Yos
0.2929
Video LDM
33.45
550.61
Stable Diffusion-1.5
11.12
(Ours)
CoDi (Ours)
32.88
596.34
Versatile Diffusion [S3]
11.10
CoDi (Ours)
11.26
Table 5: The comparison between our audio diffuser and baseline TTA generation models. Evaluation conducted on AudioCaps test set. AS, AC, FSD, BBC, and SDN stand for AudioSet, AudioCaps, Freesound, BBC Sound Effect, and Soundnet.
Model
Datasets
FD
1
KL
FAD
OVL
REL
Ground truth DiffSound
83.61
80.11
AS AC
47.68
4.01
2.52
7.75
45.00
43.83
AudioGen
AS + AC + 8 others
2.09
3.13
Audiol .DM-L--Full
AS + AC + FSD BBC
23.31
8.13
1.59
1.96
65.91
65.97
CoDi (Ours)
AS + AC + FSD + +DN
22.90
8.77
1.40
1.80
66.87
67.60
Table COCO image caption-
ing scores comparison.
Table 7: AudioCaps audio cap- tioning scores comparison.
Table 8: MSRVTT video cap- tioning scores comparison.
CIDEi
Antoregresrive Model
Docer [1]]
36.58
304
124.12
Mede
SPICE
Model
B64
METBOR
CIDEr
ClipCap [35]
32.15
27.1
108.35
0.144
ORG-TRL [S8]
43.6
28.8
50.9
OFA [49]
44.9
32.5
154.9
BART
0.753
0.176
MV-GPT (43)
48.9
38.7
60.0
BLIP
43.7
145.8
171
[48]
33.1
Difiviion
MixGen
0,466
0.177
mPLL [52)
57.8
34,9
80.3
DUCapp [59]
CaDi (Ours)
0,680
0.789
0.182
Coll (Ours)
52.1
32.5
SCD-Net [34]
39.4
29.2
131.6
Coll (Oues)
40.2
31.0
149.9
generation of image and text. For the joint generation task, we propose to train with text text- mexa---mmae-texxt where the prompt text is the truncated image caption, and the output text is the original caption. Since the condition information is incomplete, the text and image diffuser will need to learn to attend with each other through the joint generation process.
Audio + Text. We curated new dataset, Freesound 500K, by crawling 500K audio samples together with tags and descriptions from the Freesound website. We also AudioSet [42] with 2 million human-labeled 10-second sound clips from YouTube videos and AudioCaps [24] with 46K audio- text pairs derived from the AudioSet dataset. Audio samples clipped into 10-second segments for training purposes. The paired audio + text data enables us to train text text->audio, audio -txit, text->audi text generation, and audio-text contrastive learning. Similar to image text joint generation, in text->audio text, text prompt is the truncated text, and the output is the original text.
Video. We use the following diverse and high-quality video datasets to train video generation and video prompt encoder. WebVid [4]. a large-scale datasct of web videos together with descriptions; LDD-Vill-1O0M. [54] with high resolution YouTube videos of at least 720P. We perform text->video and video-text contrastive learning task with Web Vid. We use HD- Villa-100M for image-vides generation where the middle frame is the input image.
Audiovisual. Web videos are a natural aligned audio-video data resource. However, many existing datasets, e.g., ACAV 100M [28], feature heavily on videos of human speech rather than natural sounds. Therefore, we leverage sound-oriented datasets AudioSet and SoundNet [3] for joint audio-video generation. For image- + video, we use the middle frame of the target video as the input prompt image. We also use the middle frame as the prompt input to train the model generate the audio, i.c., image-aaudi
7

---

"Oil painting, cosmic horror painting, elegant intricate ArtStation concept art by Craig Mullins detailed"
(Pirate ship sailing ambience)
"Forward moving camera view"
(Sound of flowing water)
(Video of waterfall with moving forward)
"Panda eating bamboo, people laughing."
(A video with sound with people watching panda and laughing)
Figure 4: Generation with multiple input modality conditions. Top to bottom: text+audio-vimage
text+audio-video, video+audio-
5 Evaluation Results
In this section, we will evaluate the model generation quality in different settings including single modality generation, multi-condition generation, and multi-output joint generation. We provide both quantitative benchmarking evaluation datasets well as qualitative visualization demonstrations.
5.1 Single Modality Generation Results
We first show example demo in Fig. 3, where we present various single single modality generation. Then, we evaluate the synthesis quality of the unimodal generation on text, image, video, and audio. CoDi achieves SOTA on audio captions and audio generation, shown in Table and Table 5. Notably for the first time in the field, CoDi, a diffusion- model, exhibits comparable performance on image captioning with autoregressive transformer-base based SOTA (Table 6). CoDi is -iouuiioo-mooel the first diffusion model based for video captioning Table 8. On image and video generation, CoDi performs competitively with state-of-the-ar (Tables to 4). This gives us strong starting points for multi-condition and multi-output generation that will be presented next in Section 5.2 and Section 5.3.
We demonstrate in Section 3.2 that CoDi is capable of integrating representation from different modalities in the generation. Thus, we first show multi-condition generation demo as shown in Fig. 4,
5.2 Multi-Condition Generation Results
For quantitative evaluation, we focus on multiple inputs to image synthesis output since the evaluation metric for this case (FID) does not require specific modality inputs like text. We test with several input combinations including text image, text + audio, image + audio, text video, as well as three inputs text audio image. We test on the validation set of AudioCaps [24] since all four modalities present in this dataset. The prompt image input is the middle frame of the video, As shown in

---

Table 9: CoDi is capable of generating high quality output (image this case) from various combina- tions of prompt modalities.
Table 10: MSR-VTT text-to-video generation per- formance.
Inputs
CLIPSIM
Inputs
FID
Single modality Prompt Text
Single-modee modality Prompt
0.2890
Text
14.2
Dual modalty Prompt
Audio
14.3
TexioAudio
0.2912
Dual-modality Prompt
Text+Image
0.2891
Text Audio
14.9
Auxt++uadmmmmagee
0.2923
Generated Video (Somple fromes)
"Fireworks in the sky."
(Sunnd firework synchromized with video)
"Sea shore sound ambience."
"Wave the shore, seagulls"
(Sound of seoguils wave crashing the shore)
"Teddl skateboard, 4k."
(Raining ombience)
[Skateboarding sound, street noise, roining ombience)
Figure 5: Joint generation of multiple output modalities by CoDi. From top to bottom:
text tuxt-vvidooauddio, -it--mmmatttet+auddi coxtaaddotimaae--vided
Table CoDi achieves high image generation quality given assorted groups of input modalities. We also test with several input combinations with video as output including text, text audio, image + image, well as text + audio image. We also on MSRVTT [24] since all four modalities present in this dataset. Similarly, the prompt image input is the middle frame of the video. As shown Table 10, CoDi achieves high video and ground truth text similarity given assorted groups of input modalities. Again our model does not need to train on multi-condition generation like text + audio or text + image. Through bridging alignment and composable multimodal conditioning as proposed in Section 3.2, our model trained on single condition can zero-shot infer on multiple conditions.
5.3 Multi-Output Joint Generation Results
For joint multimodal generation, we first demonstrate high-quality multimodal output joint generation demo as shown in Fig. 5. For quantitative evaluation, there no existing evaluation metrie since we are the first model that can simultaneously generate across all 4 modalities. Therefore, we propose the following metric SIM that quantifies the coherence and consistency between the two generated modalities by cosine similarity of embeddings:
SIM B) cos C((())
(4)

---

Table 11: Similarity scores between generated modalities. The number on the left of "/" represents the similarity score of independent generation, and the right represents the case of joint generation. Jointly generated outputs consistently show stronger coherence.
Inputs
SIM-IT
SIM-AT
SIM-VT
SIM-VA
Two Joint Outputs
Audio + Image+Texx Text
0.251 / 0.260
0.260
Image + Audio+Text
0.244 0.256
Text + Video+Audio
0.240 0.255
Audio + Video+Text
0.256 / 0.261
Three Joint Outputs
Text + Iideo++Imae+AAudio 0.256 0.270
0.240 0.257
0.250/0.257
0.240 / 0.257
Mulii-lnputs-Outpuus
Text+Image > Video+Audio
0.247 / 0.259
where A, B are the generated modalities, and CA and Cn are aligned encoders that project A and B to the same space. We the prompt encoder as described in Section 3.2. This metric aims to compute the cosine similarity of the embedding of two modalities using contrastive learned prompt encoders. Thus, the higher the metric, the more aligned and similar the generated modalities are.
To demonstrate the effectiveness of joint generation, assume the prompt modality is P. we compare SIM(A, B) of A and B generated separately vs. jointly, i.e., (P A, P + B) (P B). The benchmark is the validation set of AudioCaps [24]. We test on the following settings, audio -immagettxt, image -audio+text, and text vixt=vvidooauudoo, image -video+audio. audio-> video+text, audio-> text+video+image, text -video--image--uudi where the image prompt is the middle frame of the video clip. As shown in Table 11, joint generation (similarity shown the right side of "f") consistently outperforms independent generation (on the left side of "P").
Conclusion
In this paper, we present Composable Diffusion (CoDi), a groundbreaking model in multimodal generation that is capable of processing and simultaneously generating modalities across text, image, video, and audio. Our approach enables the synergistic generation of high-quality and coherent outputs spanning various modalities, from assorted combinations of input modalities. Through extensive experiments, we demonstrate CoDi's remarkable capabilities in flcxibly generating single multiple modalities from wide range of inputs. Our work marks significant step towards more engaging and holistic human-computer interactions, establishing a solid foundation for future investigations in generative artificial intelligence.
Limitations Broader Impacts. See Appendix D for the discussion.
Acknowledgement
We would like to thank Bei Liu for HD-VILA-1O0X data support. We also thank Shi Dong, Mahmoud Khademi, Junheng Hao, Yuwei Fang, Yichong Xu and Azure Cognitive Services Research team members for their feedback.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 01-2776-237766 2022.
[2] Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia- Bin Huang. Jiebo Luo, and Xi Yin. Latent-shii Latent diffusion with temporal shift for efficient text- sot--to-video generation. arXiv preprint 284. 2023. 5, 15, 16
10

---

[3] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances in neural information processing systems, 29, 2016. 6,
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728- 1738, 2021. 6, 7, 16
[5] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Scung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. arXiv preprini arXi: 08818, 2023. 7
[6] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint XXXv:2303.12712, 2023. 2
[7] Sihan Chen, Xingjian Hc, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor; Vision-audio- laniu-la uuni-perception pretraining model and dataset. arXiv preprint arXiv:2304.08345 2023. 7
[8] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying risioo-and-language tasks via text generation. In International Conference on Machine Learning, pages 1931-1942. PMLR, 2021. 3
19] Jacob Devlin, Ming-Wes Bing-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 18X000081000400: 2018. 5
[10] Ming Ding, Zhuoyi Yang. Wenyi Hong. Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: Mastering text-to-image generation via transformers. arXiv preprint arXiv:2105. 13290, 2021. 7
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scalc. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 10687-10696, 2021. 3
[12] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap: Learning audio concepts from natural language supervision. arXiv preprint 2206,4499,,, 2022.
[13] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guidest video synthesis with diffusion models. arXiv preprint arXiv:2302.03011, 2023. 2,5 5
[14] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guides video synthesis with diffusion models. arXiv preprint 1rX1v:2300.03011, 2023. 3
[15] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pages 89-106. Springer, 2022. 7
[16] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-laheled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEFF, 2017. 6
[17] Félix Gontier, Romain Serizel, and Christophe Cerisara. Automated audio captioning by fine-tuning bart with audioset tags. In Detection and Classification of Acoustic Scenes and Events DCAnt 2021, 2021. 7
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference computer vision and pottern recognition, pages 770-778, 2016.
15
[19] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition vidco generation with diffusion models. arXiv preprint arXiv:2210.02300 2022. 2,
[20] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 01166800-85/1 2020. 3,4

---

[21]
Jonathan Ho, Tim Salimans, Alexey Gritsenko. William Chan, Mohammad Norouzi. and David J Fleet. Video diffusion models. arXiv preprint #XXv:2204.03458, 2022. 3, 15
[22] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205. 202.2225......... 15868, 2022. 2,
[23] Rongjie Huang. Jiawei Huang. Dongchao Yang. Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye. Jinglin Liu, Xiang Yin, and Zhou Zhao. Mlkk--n--udiio Text-to-audio generation with rrompt-enhanced diffusion models. Xiviv preprint Xiv:23301. 12661, 2023. 2,
[24] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume 1 (Long and Short Papers), pages 132, 2019. 6, 7, 8, 10
[25] Eungbeom Kim, Jinhee Kim, Yoori Oh, Kyungsu Kim, Minju Park, Jacheon Sim, Jinwoo Lee, and Kyogu Lee. Improving ladio-language learning with mixgen and multi-level test-time augmentation. arXiv preprint arXiv:2210. 171443 2022. 7
[26] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1xXiv::442.666 1et..44126.. 2.1900 2014, 15
[27] Jungil Kong, Jaehyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-1700 2020. 5
[28] Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, and Yale Song. Acav 100m: Automatic curation of large-scale datasets for audio-visual video representation learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10274 10284, 2021." 7
[29] Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, and Jianfeng Gao. Optimus: Organizing sentences via pre-trained modeling of latent space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Prucessing (EMNLP), pages 4678-4699, 2020. 5
[30] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language image pre- wee-trinnng with frozen image encoders and large language models. arXiv preprint arXiv:2301. 125.9 2023.
[31] Xiujun Li, Xi Yin. Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020. 7
[32] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pages 740-755. Springer. 2014.7 7
[33] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301. 125003 2023. 3,
[34] Jianjie Luo, Ychao Li, Yingwei Pan, Ting Yao, Jianlin Feng, Hongyang Chao, and Tao Mei. Semantic- conditional diffusion networks for image captioning. arXiv preprint arXi::2212.03099 2022. 7
[35] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 2021. 7.7
[36] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 1074], 2021.
[37] Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint orXiv:2305.00 303499 2023. 2
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021. 3.4 4
12

---

[39] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9 2019. 5
[40] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2044.00 26220..0055,... 2022. 3
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 10684-10695, 2022. 2, 3, 5,7
[42] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Dataxets and Benchmarks Truck, 2022. 6,7
[43] Paul Hongsuck Seo, Arsha Nagrani, Anurag Amab, and Cordelia Schmid. End-to-end generative pretraining for multimodal video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17959-17968, 2022.
[44] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video. Text-to-vides generation without text-video data. arXiv preprint arXiv:2009. 14792, 2022. 3,7
[45] Jascha Dickssiis,,,,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised lcarning using noncquilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015. 2,
[46] Yang Song, Jascha Sohl- Dick--iDiie.. Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[47] Zineng Tang, Jaemin Cho, Yixin Nie, and Mohit Bansal. TVLT: Textless vision-language transformer. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. 3
[48] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv: 2xXiv:2205.14100 2022. 7
[49] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through simple sequence-to- sequence learning framework. arXiv preprint arXiv:2202.03055. 2022. 7
[50] Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji. Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint 144. 2021. 7
[51] Chenfei Wu, Jian Liang. Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre- training for neural visual world creation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVI, pages 720-736. Springer, 2022.
[52] Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, al. mplug-2: A modularized multi-modal foundation model across text, image and video. arXiv preprint 2rxiv:2302.00400 2023. 7
[53] Xingqian Xu, Zhangyang Wang, Erie Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all one diffusion model. arXiv preprint arXiv:2211.0833,, 2022. 5.7
[54] Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang. Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5036-5045,
2022. 6, 7, 16
[55] Ziyi Yang, Yuwei Fang, Chenguang Zhu, Reid Pryzant, Dongdong Chen, Yu Shi, Yichong Xu, Yao Qian, Mei Gao. Yi-Ling Chen, et al. i-code: An integrative and composable multimodal learning framework. arXiv preprint arXiv:2205.00 01818, 2022. 2.3
13

---

[56] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi. Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[57] Rowan Zellers, Ximing Lu, Jack Hessel. Youngjae Jae Sung Park, Jize Cao. Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34:23634-2300 2021. 3
[58] Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang. Weiming Hu, and Zheng-Jur Jun Zha. Object relational graph with weacher-reoommet conference computer vision and pattern recognition, pages 13278 13288, 2020. 7
learning for video captioning. In Proceedings of the IEEE/CVF
[59] Zixin Zhu, Yixuan Wei, Jianfeng Wang. Zhe Gan, Zheng Zhang, Le Wang, Gang Hua, Lijuan Wang. Zicheng Liu, and Han Hu. Exploring discrete diffusion models for image captioning. arXiv preprint arXiv:2211.1169) 2022.7
14

---

A Model Architecture and Configuration
A.1 Overview
In this section, we provide more details on the model architecture as shown in Table 12, where each modality specifie diffuser is based UNet architecture with different variations detailed in the table. Another notable difference is the video architecture where we add temporal attention and temporal shift discussed in Section 3.3 and will diseuss its detail in the next section.
Table 12: Hyperparameters for our diffusion models. Note the video and image generation uses the same diffuser.
Modality
Video (Image) LDM
Audio LDM
Text LDM
Hyperparameter
Architecture
LDM
LDM
LDM
Z z-shape
x #frames x 64 x 64
8 256 16
768 x x 320 2 1,2,4,4
Channels
320
320 2
Depth
4
Channel multiplier
1,2,4,4
1,2,4,4
Attention resolutions
64,32,
64,32,16
64,32, 16
Head channels
32
32
32
Number of heads
8
8
8
CA embed dim
768
768
768
CA resolutions
64,32,16
64,32,16
64,32,16
Autoencoders
AutoKL
AudioL.DM
Optimus
Weight initialization Parameterization
Stable Diffusion-1.4
Versatile Diffusion
Learning rate Total batch size
2e 5 256
5e
5e 5
1024
1024
Diffusion Setup
Diffusion steps
1000
1000
1000
Noise schedule
Linear
Linear
Linear
3o
0.00085
0.00085
0.00085
By
0,0120
0.0120
0.0120
Sampling Parameters
Sampler
DDIM
DDIM
DDIM
Steps
50
50
50
!)
1.0
1.0
1.0
Guidance scale
7.5
2.0
A.2 Video LDM Architecture
Except for the base image UNet architecture, we also add temporal attention and temporal shift [2] before each residual block. Following VDM [21], the temporal attention is transformer attention module where we flatten the height and width dimension batch size dimension and the self-attention is performed on the time dimension. The temporal shift is illustrated in Fig. 6 where we first split channels into & chunks. Then, we shift the channel dimension numbered 0 to k by temporal dimension from 0 to k times respectively. Eventually. we concatenate the shifted chunks by hidden dimension. Note that we use = 3 in the illustration for simplicity but implementation. We then add a convolution layer before the temporal shift module. Finally, we use residual connection [18] and add the output to the input before the convolution layer. The complete video UNet layer is shown in Fig.
in our
B
Model Training
Prompt Encoders Training. As discussed in Section 3.2, we use bridging alignment to perform contrastive learning between prompt encoders. We use Adam [26] optimizer with learning rate le-4 and weight decay 1e-4.
15

---

H
H
t-1
2
1-1
4
1
3
W
W
W
C
C/3
C/3
C/3
C/3
C/3
C/3
t
H
H
Z;
2
W
W
C/3
C/3
C/3
C/3
C/3
C/3
H
t+1
B++1
Z
Zi
W
W
W
C/3
C/3
C/3
C/3
C/3
C/3
Figure 6: Temporal shift [2] illustration. C, H, W represent channel, height, width, respectively. The vertical line represents time steps from 1, and 1. The grey blocks denote "padding tensors".
Video UNet Layer
Temporal
20 Spatial
Temporal Attantion
Shift
Convolution Block
Figure 7: Video UNet layer architecture details including normalization activation, 2D temporal attention, followed by temporal shift and ID spatial convolution.
Diffusion Model Training. We train diffusion model with training objectives and hyperparameters detailed in Table and Table 12. For video LDM, we adopt a more specific training curriculum. We adopt curriculum learning frame resolution and remee-per-second (FPS). First, the diffuser is trained on the WebVid dataset of 256-frame resolution, with the training objective being text- conditioned video generation. The training clips are sampled from 2-second video chunks with FPS. Second, the model is further trained on HDVILLA and ACAV datasets, with a 512-frame resolution and FPS, and the training objective is image- conditiond video generation (the image is randomly sampled frame of the clip). Each training clip contains 16 frames sampled from 2-second video chunk with FPS.
Joint Generation Training. As discussed in Section 3.2, train joint generation by aligning environment encoders and optimize cross-attention layers only in the diffusion models. We use Adam optimizer with learning rate le-5 and weight decay Ic-4.
C Training Datasets
In this section, we introduce more details about the video and audiovisual training datasets.
Video. WebVid [4] a large-scale dataset of web videos with diverse content, spanning over 40 categories such as sports, cooking, and travel. It contains over 1.2 million video clips (all without sound) that are all at least 30 seconds duration with video descriptions. We perform text->video and video-text contrastive learning task with this dataset. HD- VillVilla-1000 100M [54] is large-scale video dataset with over 100 million video clips sourced from YouTube. The dataset covers wide range of video categories and includes high-quality videos with resolution of at least 720P, Since it lacks
16

---

curated video description and we use the middle frame image input to perform image- video generation.
SoundNet originally contains over two million sounds and spans wide range of categories including music, animal sounds, natural sounds, and environmental sounds. We collected all currently accessible 1M videos.
Audiovisual.
D Limitations & Broader Impacts
While the paper primarily focuses on the technical advancements and potential applications of CoDi, we also consider potential negative social impacts that could arise from the development and deployment of such technology. These impacts can include:
Deepfakes and Misinformation. As part of common issue for generative AI models, the ability of CoDi to generate realistic and synchronized multimodal outputs also raises concerns about the creation and dissemination of deepfakes. Malicious actors could exploit this technology create highly convincing fake content, such as fabricated videos or audio clips, which can be used for misinformation, fraud, other harmful purposes.
Bias and Stereotyping. If the training data used for CoDi is biased or contains stereotypes, the generated multimodal outputs may also reflect these.
E
License
We will publicly release our code and checkpoints. We cite licenses from the individual dataset or package we from the community and provide the following links for references.
LAION-400M: Creative Common CC-BY 4.0
AudioSet: Creative Common CC-BY 4.0
AudioCaps: MIT
Freesound: Creative Commons
BBC Sound Effect: The BBC's Content Licence
SoundNet: MIT
Webvid10M: Webvid
HD-Villa-100M: Research Use Data Agreement v1.0
PyTorch: BSD-style
Huggingface Transformers: Apache
Torchvision: BSD 3-Clause
Torchaudio: BSD 2-Clause
17


Reviews:
[{"review_id": "lM8nvxAwOJ", "reviewer": "Reviewer_7M7p", "summary": "The paper presents a new generative model called Composable Diffusion (CoDi). This model is capable of generating any combination of output modalities from any combination of input modalities, including language, image, video, or audio. Unlike other models that are limited to a subset of modalities like text or image, CoDi can generate multiple modalities in parallel.\n\nThe authors have designed CoDi to align modalities in both the input and output space. This allows the model to condition on any input combination and generate any group of modalities, even if they are not present in the training data.\n\nA key feature of CoDi is its novel composable generation strategy. This involves building a shared multimodal space by bridging alignment in the diffusion process. This feature enables the synchronized generation of intertwined modalities, such as temporally aligned video and audio.\n\nThe paper reports that CoDi achieves strong joint-modality generation quality. It either outperforms or is on par with the unimodal state-of-the-art for single-modality synthesis.", "questions": "1. Evaluation Metrics: Could you provide more details on why you chose FID and CLIPSIM as the primary evaluation metrics? Have you considered incorporating user studies or other qualitative evaluations to assess the perceptual quality and coherence of the generated outputs?\n\n2. Quality of Generated Results: Could you elaborate on the factors that might be contributing to the short and discontinuous text, short video length, and perceptually low-quality images? Are there potential improvements or modifications to the model that could address these issues?\n\n3. Preservation of Input Modality: How does the model ensure the preservation of the identity or characteristics of the input modality in the generated outputs? Are there specific mechanisms in place to ensure this, or is it an area for future work?\n\n4. Cross-Modality Benefits: Could you provide more evidence or a clearer explanation of how the generation results benefit from cross-modality conditions? The results in Tables 8 and 9 suggest that the benefits might be marginal or even negative in some cases. Could you clarify this?\n\n5. Omission of Baselines: Why was the StableDiffusion v1.5 baseline omitted from the comparisons in Table 2? Including this baseline could provide a more comprehensive view of the model's performance relative to existing methods.", "limitations": " The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Originality: The paper introduces Composable Diffusion (CoDi), a new model in multimodal generation. This model is designed to process and generate modalities across text, image, video, and audio simultaneously. This is a novel contribution as it enables the generation of various output modalities from different combinations of input modalities.\n\n2. Quality: The authors have conducted extensive experiments to demonstrate the capabilities of CoDi. The results show that CoDi can generate single or multiple modalities from a wide range of inputs. The model's performance is competitive with state-of-the-art models in tasks such as image and video generation, video captioning, and image synthesis from multiple input modalities.\n\n3. Clarity: The paper is well-structured and provides clear explanations of the model's architecture and its generation strategy. The use of figures and tables helps to understand the model's capabilities and performance.\n\n4. Significance: This work represents a step towards more comprehensive human-computer interactions by enabling the generation of multiple modalities in parallel. CoDi has potential applications in various areas, from content creation to human-computer interaction. The authors also provide a basis for future research in generative artificial intelligence.\n\nIn summary, the paper presents a significant and original contribution to the field of multimodal generation, demonstrating high-quality research and clear presentation.", "weaknesses": "The paper presents a novel approach to multimodal generation, but there are several areas where it could be improved:\n\n1. Evaluation Metrics: The evaluation of the model's performance is primarily based on quantitative metrics such as Frechet Inception Distance (FID) and CLIPSIM. These metrics, while useful, may not fully capture the perceptual quality or coherence of the generated outputs. Incorporating user studies or other qualitative evaluations could provide a more comprehensive understanding of the model's performance.\n\n2. Quality of Generated Results: The quality of the generated results could be improved. The generated videos are relatively short, the quality of the images is perceptually low, and the generated text is often short and discontinuous. These factors could limit the practical utility of the generated outputs.\n\n3. Preservation of Input Modality: The model primarily focuses on understanding between modalities, but it does not always preserve the faithfulness of the input modality. For instance, the output video and images do not consistently preserve the identity of the input image. This could limit the model's ability to generate accurate and coherent outputs across different modalities.\n\n4. Cross-Modality Benefits: The paper does not convincingly demonstrate that the generation results benefit from cross-modality conditions. For example, Table 8 shows that the quality of image generation can even degrade when using conditions from two modalities. Similarly, Table 9 shows only marginal improvements in video quality when using multiple modalities. The authors should establish a benchmark that clearly demonstrates the benefits of using multiple modalities for generation. Without such evidence, the necessity of the proposed architecture could be questioned.\n\n5. Omission of Baselines: In Table 2, the authors omit the StableDiffusion v1.5 baseline, which is the image Latent Diffusion Model (LDM) they used. Including this baseline could provide a more comprehensive comparison of the model's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review.\n\n> **1. \"Evaluation Metrics: Incorporating user studies or other qualitative evaluations could provide a more comprehensive understanding of the model's performance.\"**\n\nWe perform a small scale user study due to time constraints. We will perform more comprehensive studies in the next version of the paper.\n\n**Text to Audio**\n\nWe first conduct text to audio user study by comparing CoDi to previous text-to-audio generation SOTA AudioLDM on 30 text-to-audio generation examples.\n\nFor the user 0: CoDi was favored in 23 instances. AudioLDM was favored in 7 instances. \n\nFor the user 1: CoDi was favored in 18 instances. AudioLDM was favored in 12 instances.\n\nFor the user 1: CoDi was favored in 21 instances. AudioLDM was favored in 9 instances.\n\nOverall, the three users clearly showed a preference for the CoDi method for text-to-audio generation.\n\n**Audio to Image + Text**\n\nWe next conduct audio to image+text user study by comparing CoDi with joint generation to CoDi without joint generation capacity.\n\nFor the user 0: CoDi without joint generation was favored in 10 instances. CoDi with joint generation was favored in 20 instances. \n\nFor the user 1: CoDi without joint generation was favored in 10 instances. CoDi with joint generation was favored in 20 instances. \n\nFor the user 2: CoDi without joint generation was favored in 8 instances. CoDi with joint generation was favored in 22 instances. \n\nOverall, the three users clearly showed a preference for the CoDi method with joint generation.\n\n> **2. \"Quality of Generated Results: The quality of the generated results could be improved.\"**\n\nPlease see General Response 1 for full discussion. In general, CoDi focuses on multi-modal generation and enabling different pretrained DMs to communicate and interact with each other. We only maximally maintain competitive single generation performance without sacrificing individual task performance. Previous SOTA diffusion models are often proprietary and use private data while our model is fully open-source. Moreover, the individual diffusion models are already very competitive and some achieve SOTA performance as shown in tables 2-7.\n\n> **3. \"Preservation of Input Modality: The model primarily focuses on understanding between modalities, but it does not always preserve the faithfulness of the input modality.\"**\n\nThank you for the observation regarding the preservation of input modality. To this end, we have experimented with finetuning CoDi for image animation, i.e., input an image and generate a video that animates it. See **examples in the uploaded pdf**. The **generated videos** are also shared with the AC. Concretely, on top of the original CoDi video diffuser that takes in encoder embeddings, we concatenate the image to the diffuser inputs to improve faithfulness of image animation. This shows that CoDi can be easily modified or finetuned to perform other downstream tasks with higher faithfulness.\n\nIn general, faithfulness, or modality preservation, is an ongoing challenging topic in the diffusion model and AIGC community. Still, it is a meaningful future work to modify the architecture and training on specific tasks where faithfulness to input images for example can be preserved.\n\nThe design choices of CoDi does not focus on preserving the faithfulness of the input modality because of the following reasons:\n\n(1) Focus on Modality Conversion: CoDi's primary goal is to explore modality conversion, enabling seamless transformation between various modalities like text, images, audio, and video. It's designed to handle flexible and innovative tasks, rather than strictly preserving the input modality. This approach can lead to creative and adaptive generation capabilities. \n\n(2) Use Cases: The preservation of input modality may be context-dependent. In some applications or use cases, a higher degree of preservation may be desirable, while in others, more flexible and transformative generation may be preferred. Our model is designed to be versatile, and potential adjustments to emphasize input preservation could be explored depending on specific requirements or user needs. \n\n> **4. \"Cross-Modality Benefits: The paper does not convincingly demonstrate that the generation results benefit from cross-modality conditions.\"**\n\nPlease see General Response 2, where we discuss the difference in FID scores and CLIPSIM between single and multi-modal approaches in a model. The main goal was to show that the model can handle multiple modalities without a significant drop in performance, rather than achieving a lower FID score. Though there is a slight increase in FID scores, it's not statistically significant (p = 0.086) and is outweighed by the model's increased flexibility and potential applications. CLIPSIM, a metric for video generation fidelity, shows that adding audio modalities as input conditions maintains similarity between video output and text input. The data in Table 10 further supports the effectiveness of integrating different input modalities using CoDi for joint video and audio generation (row Text -> Video+Audio (0.240 / 0.255) and row Text + Image -> Video+Audio (0.247 / 0.259),).\n\n> **5. \"Omission of StableDiffusion v1.5 baseline\"**\n\nIn the table below, we compare our work to StableDiffusion v1.5 and still performs comparably. And note that CoDi is composable and modular. Therefore, it will be very efficient to add other diffusion models to the framework without a significant amount of training. Such advantage allows CoDi to take advantage of many SOTA diffusion models.\n\n| **Method**                    | **FID $\\downarrow$** |\n|-------------------------------|-----------|\n| CogView | 27.10     |\n| GLIDE   | 12.24     |\n| Make-a-Scene | 11.84  |\n| LDM    | 12.63     |\n| Stable Diffusion-1.4          | 11.21     |\n| Stable Diffusion-1.5          | 11.12     |\n| Versatile Diffusion | 11.10 |\n| **CoDi (Ours)**               | 11.26     |\n"}, {"review_id": "LPcCXLTbrP", "reviewer": "Reviewer_dG2H", "summary": "The paper introduces Composable Diffusion (CoDi), an innovative generative model capable of producing any combination of output modalities, such as language, image, video, or audio, from any combination of input modalities. Unlike existing generative AI systems, CoDi can generate multiple modalities simultaneously and is not limited to a subset of modalities like text or images. To address the challenge of lacking training datasets for many modalities combinations, the authors propose a modality alignment approach in both the input and output space. This enables CoDi to condition freely on any input combination and generate any group of modalities, even if they are not present in the training data. CoDi employs a unique composable generation strategy that establishes a shared multimodal space through alignment in the diffusion process. This allows for the synchronized generation of intertwined modalities, such as temporally aligned video and audio. Offering high customization and flexibility, CoDi achieves impressive quality in joint-modality generation and either outperforms or matches the state-of-the-art unimodal models for single-modality synthesis.", "questions": "Please refer to weakness.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is addressing an important problem of mapping modalities from any domain to any domain without fully paired data.\n2. The proposed method is novel and reasonable. It is good to see that each different component can be trained separately.\n3. The proposed bridging alignment is interesting.", "weaknesses": "The proposed method shares some similarities with previous works. Nevertheless, this paper still contributes to the community in my opinion. It could be better to have a more specific discussions on the difference with the related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks, we agree that specific comparisons can provide more insights and clarify the unique contributions of CoDi. Here's how we position our work with different areas and community in extension to the the discussion in the related work section: \n\n>**1. \"Comparison with previous diffusion models.\"**\n\nWe first provide a context to the current diffusion models community, in addition to the diffusion background introduced in 66-81.\n\n**Scope:** Most diffusion models focus on specific generative tasks, such as text-to-X or single-to-single generation. Some examples are text-to-video, text-to-image, image/audio/video captioning, etc. In contrast, our work ambitiously aims to build a versatile framework for any-to-any generation within the realms of image, text, audio, and video. \n\n**Multi-modal Integration:** Previous works [1,2,3,4] that involve multi-modal generation often lack a unified encoding space [1] or a method to maintain high correspondence between output modalities effectively [2,3]. CoDi innovatively addresses these limitations by proposing bridging alignment that facilitates the integration of various input modalities and ensures high correspondence in the generated outputs.\n\n[1] Liu, Haotian, et al. \"Visual instruction tuning.\" arXiv preprint arXiv:2304.08485 (2023).\n\n[2] Xu, Xingqian, et al. \"Versatile diffusion: Text, images and variations all in one diffusion model.\" arXiv preprint arXiv:2211.08332 (2022). \n\n[3] Masahiro Suzuki and Yutaka Matsuo. A survey of multimodal deep generative models. Advanced Robotics, 36(5- 6):261–278, 2022.\n\n[4] Wu, Mike, and Noah Goodman. \"Multimodal generative models for compositional representation learning.\" arXiv preprint arXiv:1912.05075 (2019).\n\n>**2. \"Comparison with previous general multimodal frameworks.\"**\n\nWe also provide a broader context to other general multimodal frameworks like BLIP [5], Flamingo [6], Llava [1], etc, in addition to the multimodal background introduced in 82-89.\n\n**Focus on Generation vs. Reasoning:** While these frameworks emphasize multi-modal reasoning tasks such as question answering or dialogue, CoDi's primary goal is to enable flexible generation from any combination of input modalities. This distinct focus sets our work apart and addresses a different set of challenges and opportunities. \n\n**Bridging Alignment Contribution:** A key innovation in CoDi is the design of bridging alignment, which reduces the quadratic training/data cost to linear, enabling efficient handling of unpaired data. This technical advancement further distinguishes our work and has broad implications for efficiency and scalability. \n\nWe recognize the value in a more comprehensive discussion with related works and will indeed include this in the revised manuscript. This discussion will provide readers with a deeper understanding of CoDi's place within the existing landscape and highlight our novel contributions more prominently.\n\n[5] Li, Junnan, et al. \"Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.\" International Conference on Machine Learning. PMLR, 2022.\n\n[6] Alayrac, Jean-Baptiste, et al. \"Flamingo: a visual language model for few-shot learning.\" Advances in Neural Information Processing Systems 35 (2022): 23716-23736.\n"}, {"review_id": "5gLX0SwhI2", "reviewer": "Reviewer_mZPw", "summary": "This paper presents a method that can generate any combination of output modalities, including language, audio, image, or video, from any combination of input modalities. The idea here is to align four modalities in a shared feature space first, and then learn to generate one or more modalities based on the shared feature space. This design enables many combinations of modalities despite the lack of training datasets. Since the feature space is shared, it also flexible to extend to other modalities.", "questions": "* From Table 8, using both text and audio as input achieves higher FID compared to using each single modality as input. Could you explain why model achieves worse performance with more information as input?\n* From table 2 and table 3, CoDi does not outperform previous SOTA results. Do you think a model that can do all tasks need to sacrifice its performance on each specific task?\n* During training, the text encoder weights are frozen after training with images, would it result to a suboptimal problem when training with other modalities?\n* In Sec 3.3, image diffusion model and video diffusion model are introduced separately. However, in Figure 2, only vision UNet and Vision Encoder are shown. Does it mean image diffusion model share parameters with video diffusion model during training?\n* In table 4, why CoDi can outperform other diffusion-based method in image captioning?", "limitations": "The authors adequately address the limitations and potential negative sosietal impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea of any-to-any generation is interesting, and it enables many different tasks in one model.\n* The framework is flexible and customizable to many other potential modalities, such as semantic maps, heat map, depth map and so on.\n* The performance of the proposed method achieves comparable or better results than previous SOTA methods.\n", "weaknesses": "* The method part is not clear. The relation among image diffusion model, video diffusion model, vision encoder and vision unet is confusing. Since 4 diffusion models are introduced and only 3 types of encoders and unet are shown in Figure 2, It s not clear whether image and video models share the parameters or not.\n* The evaluation of Table 3 is not sufficient. Only the text-video faithfulness (CLIPSIM) is evaluated, while the video quality (FVD) is not evaluated.\n* The proposed framework enables many different tasks. However, it does not outperform previous SOTA methods in many tasks, such as text-to-video generation, text-to-image generation, image captioning and video captioning.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review. We are committed to addressing the concerns raised to enhance the quality of this paper.\n\n> **1. \"The method part is not clear. The relation among image diffusion model, video diffusion model, vision encoder and vision unet is confusing. Since 4 diffusion models are introduced and only 3 types of encoders and unet are shown in Figure 2, It’s not clear whether image and video models share the parameters or not.\"**\n\nIn line number 148-149, we stated that we construct the video diffuser by extending the image diffuser with temporal modules. This implies that the image and video generation share the same model. Concretely, both the encoder and UNet use the same architecture for image and video. For vision encoder, we add temporal layers on top of the image encoder for video encoding. Similarly for vision UNet, we also add temporal layers on top of the image UNet for video generation. In the Appendix A, we have also discussed the detailed video architecture. We will make this clearer in the paper.\n\n> **2. \"The evaluation of Table 3 is not sufficient. Only the text-video faithfulness (CLIPSIM) is evaluated, while the video quality (FVD) is not evaluated.\"**\n\nIn the table below, on video generation on UCF-101, CoDi performs competitively with state-of-the-art which matches similar observation in CLIPSIM metric conducted on MSRVTT.\n\n| Method                             | Zero-Shot | IS ($\\uparrow$ )  | FVD ($\\uparrow$ )  |\n|------------------------------------|-----------|---------|----------|\n| CogVideo (Chinese) | Yes       | 23.55   | 751.34   |\n| CogVideo (English) | Yes       | 25.27   | 701.59   |\n| Make-A-Video          | Yes       | 33.00   | 367.23   |\n| Video LDM         | Yes       | 33.45   | 550.61   |\n| **CoDi (Ours)**                    | Yes       | 32.88   | 596.34   |\n\n\n> **3. \"The proposed framework enables many different tasks. However, it does not outperform previous SOTA methods in many tasks, such as text-to-video generation, text-to-image generation, image captioning and video captioning.\"**\n\nPlease see our response in General Response 2 for full discussion. We’d like to reiterate that the main focus of CoDi is not to beat previous text-to-X SOTA methods, because many of them are proprietary and use private training data. In fact, our audio and text diffusion model achieves SOTA performance, and the image diffusion model (DM) uses the best open-sourced one (Stable Diffusion 1.5) at the time of this project. We acknowledge that CoDi’s video DM has a gap between previously-reported SOTAs (such as imagegen-vide, make-a-video), however, those DMs are closed-sourced and trained on private data (such as company-internal videos). In contrast, CoDi’s video DM is trained on public data and open-sourced.\n\nThe main focus of CoDi is enabling different pretrained DMs to communicate and interact with each other, such that we can achieve any-to-any generation even without the existence of paired training data.\n\nMoreover, CoDi is designed with composability and modularity at its core. This means that it can readily incorporate various diffusion models into the framework without requiring a significant amount of retraining. This advantageous quality ensures that CoDi remains agile and adaptable, allowing it to seamlessly leverage many state-of-the-art (SOTA) diffusion models as they become available.\n\n> **4. \"From Table 8, using both text and audio as input achieves higher FID compared to using each single modality as input. Could you explain why the model achieves worse performance with more information as input?\"**\n\nPlease see our response in General Response 2 for full discussion. In general, the primary goal of the model under discussion was to demonstrate the ability to handle multiple modalities without significant performance degradation. The slight increase in FID scores from 14.2 to 14.9 is not seen as a meaningful degradation, especially given the statistical significance level (p=0.086), and this trade-off is justified by the model's increased flexibility and broader application potential. Additionally, CLIPSIM metrics affirm the model's faithfulness in video generation to the input text, even when adding audio modalities. Table 10 further illustrates CoDi's effectiveness in integrating different input modalities, showing clear improvement for video and audio joint generation when adding the image modality.\n\n> **5. \"During training, the text encoder weights are frozen after training with images, would it result to a suboptimal problem when training with other modalities?\"**\n\nThe proposed method bridging alignment leverages the property that the text encoder is frozen. There are several advantages or reasons:\n\n**Efficiency:** The training is much more efficient (without finetuning the text encoder and jointly training all combinations of contrastive learning). Finetuning the text encoder will result in double the cost for all encoder training since text encoder is the bridging encoder that participates in all training.\n\n**Data scale:** The text and image models are trained with a very large scale dataset with 400M which has very strong generalization potential. Such joint image and text embedding learned will be sufficient to be extended to other modalities. On the other hand, the text-audio and text-video dataset has a much smaller scale with only a few million samples. \n\n**Learnable diffusion:** Regardless of the contrastive learned encoders’ performance, the diffusion model cross attention is trainable and will adapt its distribution to the desired or target tasks.\n\n> **6. \"In table 4, why CoDi can outperform other diffusion-based method in image captioning?\"**\n\nWe use LAION-400M as the training data whereas previous works like SCD-Net shown in the table 4 uses a much smaller dataset, MSCOCO, which has below 1M examples. Training on significantly more data with similar architecture will result in better performance."}, {"review_id": "hzqJCmln8a", "reviewer": "Reviewer_hGLR", "summary": "They present Composable Diffusion (CoDi), a novel generative model capable of generating any combination of output modalities, such as language, image, video, or audio, from any combination of input modalities.  Unlike existing generative AI systems, CoDi can generate multiple modalities in parallel and its input is not limited to a subset of modalities like text or image. Despite the absence of training datasets for many combinations of modalities, they also propose to align modalities in both the input and output space.", "questions": "see above", "limitations": "not addressed", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1 One model that takes any combination of modalities as input or output is novel and promising.\n2 As the lack of training data, the alignment of different modalities is very difficult. The proposed method for the alignment is very interesting.", "weaknesses": "1 The simple weighted interpolation of different representations is not so convincing. Why does this method work?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review! Please find our response below:\n\n> **1. \"Missing discussion of limitation and societal impact.\"**\n\nThe discussion of limitation and societal impact can be found in Section D of the appendix. We attach those paragraphs below:\nDeepfakes and Misinformation: As part of a common issue for generative AI models, the ability of CoDi to generate realistic and synchronized multimodal outputs also raises concerns about the creation and dissemination of deepfakes. Malicious actors could exploit this technology to create highly convincing fake content, such as fabricated videos or audio clips, which can be used for misinformation, fraud, or other harmful purposes.\nBias and Stereotyping: If the training data used for CoDi is biased or contains stereotypes, the generated multimodal outputs may also reflect these.\n\n> **2. \"The simple weighted interpolation of different representations is not so convincing. Why does this method work?\"**\n\nThe input modalities are encoded by contrastive learned encoders and thus their features representation is aligned and scaled in the same space and distribution. Assuming that the individual embeddings are in a continuous space where distances and directions have meaningful interpretations, the weighted average should also lie within this space. The final representation would maintain these properties, preserving the relationships between concepts.\n\nThere are also many advantages and reasons to use simple weighted interpolation: \n\n**Simplicity and explainability:** Weighted averaging is a straightforward operation that's easy to understand, implement. The weighted average approach allows for some level of interpretability, as you can analyze the contribution of each modality by manipulating and observing the effect of the weights.\n\n**Robustness:** If one modality is noisy or missing some information, the other modalities can compensate. The weighted average can be seen as a form of ensemble, potentially providing a more stable and robust representation. \n\n**Flexibility in Emphasizing Modalities:** By adjusting the weights, you can emphasize or de-emphasize certain modalities according to their relevance or reliability for the task at hand. \n\n**Efficiency:** This approach can be computationally efficient, as it doesn't require extensive fine-tuning or additional layers to merge the embeddings. \n\nThat said, we agree that it is a meaningful future work direction for creating learned or more complex representations of the contrastive aligned input embeddings to improve the complex interaction of different modalities."}]

Author rebuttal:
We are glad all reviewers appreciated our work and found it well-motivated (7M7p, dG2H, mZPw, hGLR), well-written (7M7p, dG2H, mZPw), and original in introducing Composable Diffusion as a novel model (7M7p, dG2H, mZPw, hGLR). The recognition of CoDi's capability to generate any combination of output modalities and the novel method of alignment (7M7p, dG2H, hGLR), the extensive experiments demonstrating CoDi's competitive performance with state-of-the-art models (7M7p, mZPw), the clear structure and explanation of the model's architecture (7M7p), and the idea of any-to-any generation that provides flexibility and customization (mZPw) were encouraging. Furthermore, the acknowledgment of CoDi's potential applications in various areas and its significance as a step towards more comprehensive human-computer interactions (7M7p) validates the impact of our work. We appreciate the thorough assessment and constructive feedback from all reviewers.

>1. **General Response 1 (the performance of each individual diffusion model)**:

We thank the reviewers for their feedback on the quality of the generated results.

It is essential to clarify the primary objectives of CoDi. Our key contribution lies in enabling different pretrained diffusion models to communicate and interact with each other. The goal is to establish a framework that facilitates this interaction, rather than fine-tuning individual aspects of unimodal diffusion models. 

In building CoDi, we maximally leverage all the open source models including StableDiffusion, CLIP, etc, for better reproduction and open access. We train our own audio, text, video diffusion models. All the training data we used is open sourced. We devote our resources to explore joint generation and generation from multiple modalities input and only aim to maintain a reasonably competitive single generation performance.

**Videos**: It is worth noting that at the time of doing this project, previously reported video diffusion models SOTA are closed-sourced (imagegen-video, gen-2, etc), and trained on internal video data. While the generated videos are acknowledged to be relatively short, our video diffusion model demonstrates competitive performance, aligning with SOTA video diffusion models as shown in Table 3. Generating short videos as demos are common approaches of the video community [1,2,3]. Most video models can be extended to longer length by further finetuning and modifications of the architecture and autoregressively generating the video as also shown in [1,2,3]. Still, generating long and coherent video is an ongoing and challenging topic in the video generation community. 

[1] Hong, Wenyi, et al. "Cogvideo: Large-scale pretraining for text-to-video generation via transformers." arXiv preprint arXiv:2205.15868 (2022).

[2] Khachatryan, Levon, et al. "Text2video-zero: Text-to-image diffusion models are zero-shot video generators." arXiv preprint arXiv:2303.13439 (2023).

[3] Blattmann, Andreas, et al. "Align your latents: High-resolution video synthesis with latent diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.1.5

**Images**: Regarding the perceived low quality of the images, our model is initialized from Stable Diffusion 1.5, the best open-source image diffusion model available at the time of submission. This foundation provides a robust starting point, and we welcome future work to enhance visual quality within the constraints of our novel cross-modal framework. 

**Text**: The generated text primarily serves as a caption, encapsulating the core information in the audio/video/image. This brevity is intentional and in line with common practices in captioning. We have showcased SOTA performance on diffusion-based image captioning, SOTA performance on audio captioning, and competitive performance on video captioning in Tables 4, 5, and 6.

>2. **General Response 2 (multiple inputs performance)**:

We appreciate the observation regarding the difference in FID scores and CLIPSIM between the single modality and multi-modal approaches. However, our primary goal was to demonstrate that our model can successfully handle multiple modalities without significant performance degradation, rather than to achieve a lower FID score per se. The observed difference in FID scores from 14.2 to 14.9, while present, does not constitute a meaningful degradation (statistical significance with only (p=0.086, (0.05<p<0.1)), especially considering the expanded capabilities of handling multiple input types. This trade-off in performance is outweighed by the increased flexibility and potential applications that our model offers. 

CLIPSIM is an evaluation metric that evaluates how faithful the video generation is to the input text. We can see in table 9 that by adding audio modalities as input conditions, the similarity between the video output and text input does not decrease, showing the effectiveness of how CoDi can integrate different input modalities while being faithful to each.

In Table 10, row Text -> Video+Audio (0.240 / 0.255) and Text + Image -> Video+Audio (0.247 / 0.259), we can see a clear improvement for video and audio joint generation by adding the image modality. This further supports the effectiveness of CoDi on integrating different input modalities.

Meta-review: This paper was reviewed by four experts in the field. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to NeurIPS 2023. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)