SlowGuess/NeurIPS_2023_OCR huggingface dataset:
generation of image and text. For the joint generation task, we propose to train with  $\text{text} \rightarrow \text{image} + \text{text}$ , where the prompt text is the truncated image caption, and the output text is the original caption. Since the condition information is incomplete, the text and image diffuser will need to learn to attend with each other through the joint generation process.
Audio + Text. We curated a new dataset, Freesound 500K, by crawling 500K audio samples together with tags and descriptions from the Freesound website. We also use AudioSet [42] with 2 million human-labeled 10-second sound clips from YouTube videos and AudioCaps [24] with 46K audio-text pairs derived from the AudioSet dataset. Audio samples are clipped into 10-second segments for training purposes. The paired audio + text data enables us to train text→audio, audio→text, text→audio + text generation, and audio-text contrastive learning. Similar to image + text joint generation, in text→audio + text, text prompt is the truncated text, and the output is the original text.
Video. We use the following diverse and high-quality video datasets to train video generation and video prompt encoder. WebVid [4], a large-scale dataset of web videos together with descriptions; HD-Villa-100M [54] with high resolution YouTube videos of at least 720P. We perform text→video and video-text contrastive learning task with WebVid. We use HD-Villa-100M for image→video generation where the middle frame is the input image.
Audiovisual. Web videos are a natural aligned audio-video data resource. However, many existing datasets, e.g., ACAV100M [28], feature heavily on videos of human speech rather than natural sounds. Therefore, we leverage sound-oriented datasets AudioSet and SoundNet [3] for joint audio-video generation. For image  $\rightarrow$  audio + video, we use the middle frame of the target video as the input prompt image. We also use the middle frame as the prompt input to train the model to generate the audio, i.e., image  $\rightarrow$  audio.
Figure 4: Generation with multiple input modality conditions. Top to bottom: text+audio→image, text+audio→video, video+audio→text.
5 Evaluation Results
In this section, we will evaluate the model generation quality in different settings including single modality generation, multi-condition generation, and multi-output joint generation. We provide both quantitative benchmarking on evaluation datasets as well as qualitative visualization demonstrations.
5.1 Single Modality Generation Results
We first show example demo in Fig. 3, where we present various single to single modality generation. Then, we evaluate the synthesis quality of the unimodal generation on text, image, video, and audio. CoDi achieves SOTA on audio captions and audio generation, as shown in Table 7 and Table 5. Notably for the first time in the field, CoDi, a diffusion-base model, exhibits comparable performance on image captioning with autoregressive transformer-based SOTA (Table 6). CoDi is the first diffusion-model based for video captioning Table 8. On image and video generation, CoDi performs competitively with state-of-the-art (Tables 2 to 4). This gives us strong starting points for multi-condition and multi-output generation that will be presented next in Section 5.2 and Section 5.3.
We demonstrate in Section 3.2 that CoDi is capable of integrating representation from different modalities in the generation. Thus, we first show multi-condition generation demo as shown in Fig. 4.
5.2 Multi-Condition Generation Results
For quantitative evaluation, we focus on multiple inputs to image synthesis output since the evaluation metric for this case (FID) does not require specific modality inputs like text. We test with several input combinations including text + image, text + audio, image + audio, text + video, as well as three inputs text + audio + image. We test on the validation set of AudioCaps [24] since all four modalities are present in this dataset. The prompt image input is the middle frame of the video. As shown in
Table 9: CoDi is capable of generating high quality output (image in this case) from various combinations of prompt modalities.
Inputs	FID ↓
Single-modality Prompt
Text	14.2
Audio	14.3
Dual-modality Prompt
Text + Audio	14.9
Table 10: MSR-VTT text-to-video generation performance.
Inputs	CLIPSIM ↑
Single-modality Prompt
Text	0.2890
Dual-modality Prompt
Text+Audio	0.2912
Text+Image	0.2891
Text+Audio+Image	0.2923
Figure 5: Joint generation of multiple output modalities by CoDi. From top to bottom: text→video+audio, text→image+text+audio, text+audio+image→video+audio.
Table 9, CoDi achieves high image generation quality given assorted groups of input modalities. We also test with several input combinations with video as output including text, text + audio, image + image, as well as text + audio + image. We also test on MSRVTT [24] since all four modalities are present in this dataset. Similarly, the prompt image input is the middle frame of the video. As shown in Table 10, CoDi achieves high video and ground truth text similarity given assorted groups of input modalities. Again our model does not need to train on multi-condition generation like text + audio or text + image. Through bridging alignment and composable multimodal conditioning as proposed in Section 3.2, our model trained on single condition can zero-shot infer on multiple conditions.
5.3 Multi-Output Joint Generation Results
For joint multimodal generation, we first demonstrate high-quality multimodal output joint generation demo as shown in Fig. 5. For quantitative evaluation, there is no existing evaluation metric since we are the first model that can simultaneously generate across all 4 modalities. Therefore, we propose the following metric SIM that quantifies the coherence and consistency between the two generated modalities by cosine similarity of embeddings:
$$
\operatorname {S I M} (A, B) = \cos \left(C _ {A} (A), C _ {B} (B)\right) \tag {4}
$$
Table 11: Similarity scores between generated modalities. The number on the left of “/” represents the similarity score of independent generation, and the right it represents the case of joint generation. Jointly generated outputs consistently show stronger coherence.
Inputs	SIM-IT	SIM-AT	SIM-VT	SIM-VA
Two Joint Outputs
Audio → Image+Text	0.251 / 0.260	-	-	-
Image → Audio+Text	-	0.244 / 0.256	-	-
Text → Video+Audio	-	-	-	0.240 / 0.255
Audio → Video+Text	-	-	0.256 / 0.261	-
Three Joint Outputs
Text → Video+Image+Audio	0.256 / 0.270	0.240 / 0.257	-	0.240 / 0.257
Multi-Inputs-Outputs
Text+Image → Video+Audio	-	-	-	0.247 / 0.259
where  $A, B$  are the generated modalities, and  $C_A$  and  $C_B$  are aligned encoders that project  $A$  and  $B$  to the same space. We use the prompt encoder as described in Section 3.2. This metric aims to compute the cosine similarity of the embedding of two modalities using contrastive learned prompt encoders. Thus, the higher the metric, the more aligned and similar the generated modalities are.
To demonstrate the effectiveness of joint generation, assume the prompt modality is  $P$ , we compare  $\mathrm{SIM}(A,B)$  of  $A$  and  $B$  generated separately vs. jointly, i.e.,  $\{P\to A,P\to B\}$  vs.  $\{P\to A + B\}$ . The benchmark is the validation set of AudioCaps [24]. We test on the following settings, audio  $\rightarrow$  image  $+$  text, image  $\rightarrow$  audio  $+$  text, and text  $\rightarrow$  video  $+$  audio, image  $\rightarrow$  video  $+$  audio. audio  $\rightarrow$  video  $+$  text, audio  $\rightarrow$  text  $+$  video  $+$  image, text  $\rightarrow$  video  $+$  image  $+$  audio, where the image prompt is the middle frame of the video clip. As shown in Table 11, joint generation (similarity shown on the right side of “/”) consistently outperforms independent generation (on the left side of “/”).
In this paper, we present Composable Diffusion (CoDi), a groundbreaking model in multimodal generation that is capable of processing and simultaneously generating modalities across text, image, video, and audio. Our approach enables the synergistic generation of high-quality and coherent outputs spanning various modalities, from assorted combinations of input modalities. Through extensive experiments, we demonstrate CoDi's remarkable capabilities in flexibly generating single or multiple modalities from a wide range of inputs. Our work marks a significant step towards more engaging and holistic human-computer interactions, establishing a solid foundation for future investigations in generative artificial intelligence.
Limitations & Broader Impacts. See Appendix D for the discussion.
We would like to thank Bei Liu for HD-VILA-100M data support. We also thank Shi Dong, Mahmoud Khademi, Junheng Hao, Yuwei Fang, Yichong Xu and Azure Cognitive Services Research team members for their feedback.
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736, 2022. 3
[2] Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv preprint arXiv:2304.08477, 2023. 5, 15, 16
[56] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16375-16387, 2022. 3
[57] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34:23634-23651, 2021. 3
[58] Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha. Object relational graph with teacher-recommended learning for video captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13278-13288, 2020. 7
[59] Zixin Zhu, Yixuan Wei, Jianfeng Wang, Zhe Gan, Zheng Zhang, Le Wang, Gang Hua, Lijuan Wang, Zicheng Liu, and Han Hu. Exploring discrete diffusion models for image captioning. arXiv preprint arXiv:2211.11694, 2022. 7
A Model Architecture and Configuration
A.1 Overview
In this section, we provide more details on the model architecture as shown in Table 12, where each modality specific diffuser is based on UNet architecture with different variations detailed in the table. Another notable difference is the video architecture where we add temporal attention and temporal shift as discussed in Section 3.3 and we will discuss its detail in the next section.
Table 12: Hyperparameters for our diffusion models. Note the video and image generation uses the same diffuser.
Modality	Video (Image) LDM	Audio LDM	Text LDM
Hyperparameter
Architecture	LDM	LDM	LDM
z-shape	4 × #frames × 64 × 64	8 × 256 × 16	768 × 1 × 1
Channels	320	320	320
Depth	4	2	2
Channel multiplier	1,2,4,4	1,2,4,4	1,2,4,4
Attention resolutions	64,32,16	64,32,16	64,32,16
Head channels	32	32	32
Number of heads	8	8	8
CA embed dim	768	768	768
CA resolutions	64,32,16	64,32,16	64,32,16
Autoencoders	AutoKL	AudioLDM	Optimus
Weight initialization	Stable Diffusion-1.4	-	Versatile Diffusion
Parameterization	€	€	€
Learning rate	2e-5	5e-6	5e-5
Total batch size	256	1024	1024
Diffusion Setup
Diffusion steps	1000	1000	1000
Noise schedule	Linear	Linear	Linear
β0	0.00085	0.00085	0.00085
βT	0.0120	0.0120	0.0120
Sampling Parameters
Sampler	DDIM	DDIM	DDIM
Steps	50	50	50
η	1.0	1.0	1.0
Guidance scale	2.0	7.5	2.0
A.2 Video LDM Architecture
Except for the base image UNet architecture, we also add temporal attention and temporal shift [2] before each residual block. Following VDM [21], the temporal attention is a transformer attention module where we flatten the height and width dimension to batch size dimension and the self-attention is performed on the time dimension. The temporal shift is illustrated in Fig. 6 where we first split channels into  $k$  chunks. Then, we shift the channel dimension numbered 0 to  $k - 1$  by temporal dimension from 0 to  $k - 1$  times respectively. Eventually, we concatenate the shifted chunks by the hidden dimension. Note that we use  $k = 3$  in the illustration for simplicity but  $k = 8$  in our implementation. We then add a convolution layer before the temporal shift module. Finally, we use residual connection [18] and add the output to the input before the convolution layer. The complete video UNet layer is shown in Fig. 7.
B Model Training
Prompt Encoders Training. As discussed in Section 3.2, we use bridging alignment to perform contrastive learning between all prompt encoders. We use Adam [26] optimizer with learning rate 1e-4 and weight decay 1e-4.
Figure 6: Temporal shift [2] illustration.  $C, H, W$  represent channel, height, width, respectively. The vertical line represents time steps from  $t - 1$ ,  $t$ , and  $t + 1$ . The grey blocks denote "padding tensors".
Figure 7: Video UNet layer architecture details including normalization & activation, 2D temporal attention, followed by temporal shift and 1D spatial convolution.
Diffusion Model Training. We train diffusion model with training objectives and hyperparameters detailed in Table 1 and Table 12. For video LDM, we adopt a more specific training curriculum. We adopt curriculum learning on frame resolution and frames-per-second (FPS). First, the diffuser is trained on the WebVid dataset of a 256-frame resolution, with the training objective being text-conditioned video generation. The training clips are sampled from 2-second video chunks with 4 FPS. Second, the model is further trained on HDVILLA and ACAV datasets, with a 512-frame resolution and 8 FPS, and the training objective is image-conditioned video generation (the image is a randomly sampled frame of the clip). Each training clip contains 16 frames sampled from a 2-second video chunk with 8 FPS.
Joint Generation Training. As discussed in Section 3.2, we train joint generation by aligning environment encoders and optimize cross-attention layers only in the diffusion models. We use Adam optimizer with learning rate 1e-5 and weight decay 1e-4.
C Training Datasets
In this section, we introduce more details about the video and audiovisual training datasets.
Video. WebVid [4] is a large-scale dataset of web videos with diverse content, spanning over 40 categories such as sports, cooking, and travel. It contains over 1.2 million video clips (all without sound) that are all at least 30 seconds in duration with video descriptions. We perform text→video and video-text contrastive learning task with this dataset. HD-Villa-100M [54] is a large-scale video dataset with over 100 million video clips sourced from YouTube. The dataset covers a wide range of video categories and includes high-quality videos with a resolution of at least 720P. Since it lacks
curated video description and we use the middle frame as image input to perform image  $\rightarrow$  video generation.
Audiovisual. SoundNet originally contains over two million sounds and spans a wide range of categories including music, animal sounds, natural sounds, and environmental sounds. We collected all currently accessible 1M videos.
D Limitations & Broader Impacts
While the paper primarily focuses on the technical advancements and potential applications of CoDi, we also consider potential negative social impacts that could arise from the development and deployment of such technology. These impacts can include:
Deepfakes and Misinformation. As part of a common issue for generative AI models, the ability of CoDi to generate realistic and synchronized multimodal outputs also raises concerns about the creation and dissemination of deepfakes. Malicious actors could exploit this technology to create highly convincing fake content, such as fabricated videos or audio clips, which can be used for misinformation, fraud, or other harmful purposes.
Bias and Stereotyping. If the training data used for CoDi is biased or contains stereotypes, the generated multimodal outputs may also reflect these.
E License
We will publicly release our code and checkpoints. We cite licenses from the individual dataset or package we use from the community and provide the following links for references.
LAION-400M: Creative Common CC-BY 4.0
AudioSet: Creative Common CC-BY 4.0
AudioCaps: MIT
Freesound: Creative Commons
BBC Sound Effect: The BBC's Content Licence
SoundNet: MIT
Webvid10M: Webvid
HD-Villa-100M: Research Use of Data Agreement v1.0
PyTorch: BSD-style
Huggingface Transformers: Apache
Torchvision: BSD 3-Clause
Torchaudo: BSD 2-Clause