SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zineng Tang $^{1*}$
Ziyi Yang $^{2\dagger}$
Chenguang Zhu $^{2\dagger}$
Michael Zeng2
<sup>1</sup> University of North Carolina at Chapel Hill
$^{2}$ Microsoft Azure Cognitive Services Research
https://codi-gen.github.io
Mohit Bansal
We present Composable Diffusion (CoDi), a novel generative model capable of generating any combination of output modalities, such as language, image, video, or audio, from any combination of input modalities. Unlike existing generative AI systems, CoDi can generate multiple modalities in parallel and its input is not limited to a subset of modalities like text or image. Despite the absence of training datasets for many combinations of modalities, we propose to align modalities in both the input and output space. This allows CoDi to freely condition on any input combination and generate any group of modalities, even if they are not present in the training data. CoDi employs a novel composable generation strategy which involves building a shared multimodal space by bridging alignment in the diffusion process, enabling the synchronized generation of intertwined modalities, such as temporally aligned video and audio. Highly customizable and flexible, CoDi achieves strong joint-modality generation quality, and outperforms or is on par with the unimodal state-of-the-art for single-modality synthesis. The project page with demonstrations and code is at https://codi-gen.github.io/
Figure 1: CoDi can generate various (joint) combinations of output modalities from diverse (joint) sets of inputs: video, image, audio, and text (example combinations depicted by the colored arrows).
*Work done at Microsoft internship and UNC.
†Corresponding authors: ziyiyang@microsoft.com, mbansal@cs.unc.edu
$\ddagger$ Work done while at Microsoft.
Recent years have seen the rise of powerful cross-modal models that can generate one modality from another, e.g. text-to-text [6, 37], text-to-image [13, 19, 22, 41, 44], or text-to-audio [23, 33]. However, these models are restricted in their real-world applicability where multiple modalities coexist and interact. While one can chain together modality-specific generative models in a multi-step generation setting, the generation power of each step remains inherently limited, and a serial, multi-step process can be cumbersome and slow. Moreover, independently generated unimodal streams will not be consistent and aligned when stitched together in a post-processing way (e.g., synchronized video and audio). The development of a comprehensive and versatile model that can generate any combination of modalities from any set of input conditions has been eagerly anticipated, as it would more accurately capture the multimodal nature of the world and human comprehension, seamlessly consolidate information from a wide range of sources, and enable strong immersion in human-AI interactions (for example, by generating coherent video, audio, and text description at the same time).
In pursuit of this goal, we propose Composable Diffusion, or CoDi, the first model capable of simultaneously processing and generating arbitrary combinations of modalities as shown in Fig. 1. Training a model to take any mixture of input modalities and flexibly generate any mixture of outputs presents significant computational and data requirements, as the number of combinations for the input and output modalities scales exponentially. Also aligned training data for many groups of modalities is scarce or even non-existent, making it infeasible to train with all possible input-output combinations. To address this challenge, we propose to align multiple modalities in both the input conditioning (Section 3.2) and generation diffusion step (Section 3.4). Furthermore, a proposed "Bridging Alignment" strategy for contrastive learning (Section 3.2) allows us to efficiently model the exponential number of input-output combinations with a linear number of training objectives.
Building a model with any-to-any generation capacity with exceptional generation quality requires comprehensive model design and training on diverse data resources. Therefore, we build CoDi in an integrative way. First, we train a latent diffusion model (LDM) for each modality, e.g., text, image, video, and audio. These models can be trained in parallel independently, ensuring exceptional single-modality generation quality using widely available modality-specific training data (i.e., data with one or more modalities as input and one modality as output). For conditional cross-modality generation, such as generating images using audio+language prompts, the input modalities are projected into a shared feature space (Section 3.2), and the output LDM attends to the combination of input features. This multimodal conditioning mechanism prepares the diffusion model to condition on any modality or combination of modalities without directly training for such settings.
The second stage of training enables the model to handle many-to-many generation strategies that involve simultaneously generating arbitrary combinations of output modalities. To the best of our knowledge, CoDi is the first AI model with this capability. This is achieved by adding a cross-attention module to each diffuser, and an environment encoder  $V$  to project the latent variable of different LDMs into a shared latent space (Section 3.4). Next, we freeze the parameters of the LDM, training only the cross-attention parameters and  $V$ . Since the environment encoder of different modalities are aligned, an LDM can cross-attend with any group of co-generated modalities by interpolating the representation's output by  $V$ . This enables CoDi to seamlessly generate any group of modalities, without training on all possible generation combinations. This reduces the number of training objectives from exponential to linear.
We demonstrate the any-to-any generation capability of CoDi, including single-to-single modality generation, multi-condition generation, and the novel capacity of joint generation of multiple modalities. For example, generating synchronized video and audio given the text input prompt; or generating video given a prompt image and audio. We also provide a quantitative evaluation of CoDi using eight multimodal datasets. As the latest work from Project i-Code [55] towards Composable AI, CoDi exhibits exceptional generation quality across assorted scenarios, with synthesis quality on par or even better than single to single modality SOTA, e.g., audio generation and audio captioning.
Diffusion models (DMs) learn the data distribution by denoising and recovering the original data. Deep Diffusion Process (DDP) [45] adopts a sequence of reversible diffusion steps to model image
Figure 2: CoDi model architecture: (a) We first train individual diffusion model with aligned prompt encoder by "Bridging Alignment"; (b) Diffusion models learn to attend with each other via "Latent Alignment"; (c) CoDi achieves any-to-any generation with a linear number of training objectives.
probability distribution. It uses a reversible encoder to map the input image to a latent space and a decoder to map the latent variables to an output image. Denoising diffusion probabilistic model (DDPM) [20] uses a cascade of diffusion processes to gradually increase the complexity of the probability density function model. At each step, the model adds noise to the input image and estimates the corresponding noise level using an autoregressive model. This allows the model to capture the dependencies between adjacent pixels and generate high-quality images. Score-based generative models (SOG) [46] use the score function to model the diffusion process. [40] generates high-fidelity images conditioned on CLIP representations of text prompts. Latent diffusion model (LDM) [41] uses a VAE to encode inputs into latent space to reduce modeling dimension and improves efficiency. The motivation is that image compression can be separated into semantic space by a diffusion model and perceptual space by an autoencoder. By incorporating temporal modeling modules and cascading model architectures, video diffusion models have been built upon image diffusers to generate temporally consistent and inherent frames[14, 19, 21, 44]. Diffusion models have also been applied to other domains, such as generating audio from text and vision prompts[23, 33].
Multimodal modeling has experienced rapid advancement recently, with researchers striving to build uniform representations of multiple modalities using a single model to achieve more comprehensive cross-modal understanding. Vision transformers [11], featuring diverse model architectures and training techniques, have been applied to various downstream tasks such as vision Q&A and image captioning. Multimodal encoders have also proven successful in vision-language [1, 8, 57], video-audio [47] and video-speech-language [55, 56] domains. Aligning data from different modalities is an active research area [12, 38], with promising applications in cross-modality retrieval and building uniform multimodal representations [33, 35, 41].
3.1 Preliminary: Latent Diffusion Model
Diffusion models (DM) represent a class of generative models that learn data distributions  $p(\pmb{x})$  by simulating the diffusion of information over time. During training, random noise is iteratively added
to  $x$ , while the model learns to denoise the examples. For inference, the model denoises data points sampled from simple distributions such as Gaussian. Latent diffusion models (LDM) [41] learn the distribution of the latent variable  $z$  corresponding to  $x$ , significantly reducing computational cost by decreasing the data dimension.
In LDM, an autoencoder is first trained to reconstruct  $\pmb{x}$ , i.e.,  $\hat{\pmb{x}} = D(E(\pmb{x}))$ , where  $E$  and  $D$  denote the encoder and decoder, respectively. The latent variable  $\pmb{z} = E(\pmb{x})$  is iteratively diffused over time steps  $t$  based on a variance schedule  $\beta_{1},\ldots ,\beta_{T}$ , i.e.,  $q(\pmb{z}_t|\pmb{z}_{t - 1}) = \mathcal{N}(\pmb{z}_t;\sqrt{1 - \beta_t}\pmb{z}_{t - 1},\beta_t\pmb{I})$  [20, 45].
The forward process allows the random sampling of  $\mathbf{z}_t$  at any timestep in a closed form [20, 45]:  $\mathbf{z}_t = \alpha_t \mathbf{z} + \sigma_t \boldsymbol{\epsilon}$ , where  $\boldsymbol{\epsilon} \sim \mathcal{N}(0, I)$ ,  $\alpha_t := 1 - \beta_t$  and  $\sigma_t := 1 - \prod_{s=1}^t \alpha_s$ . The diffuser learns how to denoise from  $\{ \mathbf{z}_t \}$  to recover  $\mathbf{z}$ . Following the reparameterization method proposed in [20], the denoising training objective can be expressed as [41]:
$$
\mathcal {L} _ {D} = \mathbb {E} _ {\boldsymbol {z}, \epsilon , t} \| \boldsymbol {\epsilon} - \boldsymbol {\epsilon} _ {\theta} (\boldsymbol {z} _ {t}, t, C (\boldsymbol {y})) \| _ {2} ^ {2}. \tag {1}
$$
In data generation, the denoising process can be realized through reparameterized Gaussian sampling:
$$
p \left(\boldsymbol {z} _ {t - 1} \mid \boldsymbol {z} _ {t}\right) = \mathcal {N} \left(\boldsymbol {z} _ {t - 1}; \frac {1}{\sqrt {\alpha_ {t}}} \left(\boldsymbol {z} _ {t} - \frac {\beta_ {t}}{\sqrt {\sigma_ {t}}} \boldsymbol {\epsilon} _ {\theta}\right), \beta_ {t} \boldsymbol {I}\right). \tag {2}
$$
In  $\mathcal{L}_D$ , the diffusion time step  $t \sim \mathcal{U}[1, T]$ ;  $\epsilon_{\theta}$  is a denoising model with UNet backbone parameterized by  $\theta$ ;  $\pmb{y}$  represents the conditional variable that can be used to control generation;  $C$  is the prompt encoder. The conditioning mechanism is implemented by first featurizing  $\pmb{y}$  into  $C(\pmb{y})$ , then the UNet  $\epsilon_{\theta}$  conditions on  $C(\pmb{y})$  via cross-attention, as described in [41]. Distinct from previous works, our model can condition on any combinations of modalities of text, image, video and audio. Details are presented in the following section.
3.2 Composable Multimodal Conditioning
To enable our model to condition on any combination of input/prompt modalities, we align the prompt encoder of text, image, video and audio (denoted by  $C_t$ ,  $C_i$ ,  $C_v$ , and  $C_a$ , respectively) to project the input from any modality into the same space. Multimodal conditioning can then be conveniently achieved by interpolating the representations of each modality  $m$ :  $C(x_t, x_i, x_v, x_a) = \sum_m \alpha_m C(m)$  for  $m \in x_t, x_i, x_v, x_a$ , with  $\sum_m \alpha_m = 1$ . Through simple weighted interpolation of aligned embeddings, we enable models trained with single-conditioning (i.e., with only one input) to perform zero-shot multi-conditioning (i.e., with multiple inputs). This process is illustrated in Fig. 2 (a)(2).
Optimizing all four prompt encoders simultaneously in a combinatorial manner is computationally heavy, with  $\mathcal{O}(n^2)$  pairs. Additionally, for certain dual modalities, well-aligned paired datasets are limited or unavailable e.g., image-audio pairs. To address this challenge, we propose a simple and effective technique called "Bridging Alignment" to efficiently align conditional encoders. As shown in Fig. 2 (a)(1), we choose the text modality as the "bridging" modality due to its ubiquitous presence in paired data, such as text-image, text-video, and text-audio pairs. We begin with a pretrained text-image paired encoder, i.e., CLIP [38]. We then train audio and video prompt encoders on audio-text and video-text paired datasets using contrastive learning, with text and image encoder weights frozen.
In this way, all four modalities are aligned in the feature space. As shown in Section 5.2, CoDi can effectively leverage and combine the complementary information present in any combination of modalities to generate more accurate and comprehensive outputs. The high generation quality remains unaffected with respect to the number of prompt modalities. As we will discuss in subsequent sections, we continue to apply Bridging Alignment to align the latent space of LDMs with different modalities to achieve joint multimodal generation.
3.3 Composable Diffusion
Training an end-to-end anything-to-anything model requires extensive learning on various data resources. The model also needs to maintain generation quality for all synthesis flows. To address these challenges, CoDi is designed to be composable and integrative, allowing individual modality-specific models to be built independently and then smoothly integrated later. Specifically, we start by
independently training image, video, audio, and text LDMs. These diffusion models then efficiently learn to attend across modalities for joint multimodal generation (Section 3.4) by a novel mechanism named "latent alignment".
Image Diffusion Model. The image LDM follows the same structure as Stable Diffusion 1.5 [41] and is initialized with the same weights. Reusing the weights transfers the knowledge and exceptional generation fidelity of Stable Diffusion trained on large-scale high-quality image datasets to CoDi.
Video Diffusion Model. To model the temporal properties of videos and simultaneously maintain vision generation quality, we construct the video diffuser by extending the image diffuser with temporal modules. Specifically, we insert pseudo-temporal attention before the residual block [13]. However, we argue that pseudo-temporal attention only enables video frames to globally attend to each other by flattening the pixels (height, width dimension) to batch dimension, resulting in a lack of cross-frame interaction between local pixels. We argue that this results in the common temporal-inconsistency issue in video generation that locations, shapes, colors, etc. of objects can be inconsistent across generated frames. To address this problem, we propose adapting the latent shift method [2] that performs temporal-spatial shifts on latent features in accordance with temporal attention. We divide the video by the hidden dimension into  $k = 8$  chunks, and for each chunk  $i = 0$  to 7, we shift the temporal dimension forward by  $i$  positions. Further details will be provided in the appendix.
Audio Diffusion Model. To enable flexible cross-modality attention in joint generation, the audio diffuser is designed to have a similar architecture to vision diffusers, where the mel-spectrogram can be naturally viewed as an image with 1 channel. We use a VAE encoder to encode the mel-spectrogram of audio to a compressed latent space. In audio synthesis, a VAE decoder maps the latent variable to the mel-spectrogram, and a vocoder generates the audio sample from the mel-spectrogram. We employ the audio VAE from [33] and the vocoder from [27].
Text Diffusion Model. The VAE of the text LDM is OPTIMUS [29], and its encoder and decoder are [9] and GPT-2 [39], respectively. For the denoising UNet, unlike the one in image diffusion, the 2D convolution in residual blocks is replaced with 1D convolution [53].
3.4 Joint Multimodal Generation by Latent Alignment
The final step is to enable cross-attention between diffusion flows in joint generation, i.e., generating two or more modalities simultaneously. This is achieved by adding cross-modal attention sublayers to the UNet  $\epsilon_{\theta}$  (Fig. 2 (b)(2)). Specifically, consider a diffusion model of modality  $A$  that cross-attends with another modality  $B$ . Let the latent variables of modalities  $m_A$  and  $m_B$  at diffusion step  $t$  be denoted as  $\boldsymbol{z}_t^A$  and  $\boldsymbol{z}_t^B$ , respectively. The proposed "Latent Alignment" technique is such that a modality-specific environment encoder  $V_B$  first projects  $\boldsymbol{z}_t^B$  into a shared latent space for different modalities. Then, in each layer of the UNet for modality  $A$ , a cross-attention sublayer attends to  $V_B(\boldsymbol{z}_t^B)$ . For the diffusion model of modality  $A$ , the training objective in Eq. (1) now becomes:
$$
\mathcal {L} _ {C r o s s} ^ {A} = \mathbb {E} _ {\boldsymbol {z}, \epsilon , t} \| \boldsymbol {\epsilon} - \epsilon_ {\theta_ {c}} \left(\boldsymbol {z} _ {t} ^ {A}, V _ {B} \left(\boldsymbol {z} _ {t} ^ {B}\right), t, C (\boldsymbol {y})\right) \| _ {2} ^ {2}, \tag {3}
$$
where  $\theta_{c}$  denotes the weights of cross-attention modules in the UNet.
The training objective of  $A + B$  joint generation is  $\mathcal{L}_{Cross}^{A} + \mathcal{L}_{Cross}^{B}$ .  $V(\cdot)$  of different modalities are trained to be aligned with contrastive learning. Since  $z_{t}^{A}$  and  $z_{t}^{B}$  at any time step can be sampled with closed form in the diffusion process Section 3.1, one can conveniently train the contrastive learning together with  $\mathcal{L}_{Cross}$ . The purpose of  $V$  is to achieve the generation of any combination of modalities (in polynomial) by training on a linear number of joint-generation tasks. For example, if we have trained the joint generation of modalities  $A$ ,  $B$ , and  $B$ ,  $C$  independently, then we have  $V_{A}(z_{t}^{A})$ ,  $V_{B}(z_{t}^{B})$ , and  $V_{C}(z_{t}^{C})$  aligned. Therefore, CoDi can seamlessly achieve joint generation of modalities  $A$  and  $C$  without any additional training. Moreover, such design automatically effortlessly enables joint generation of modalities  $A$ ,  $B$ , and  $C$  concurrently. Specifically, UNet of  $A$  can cross-attend with the interpolation of  $V_{B}(z_{t}^{B})$ , and  $V_{C}(z_{t}^{C})$ , although CoDi has not been trained with such task.
As shown in Fig. 2(b)(3), we follow similar designs to the "Bridging Alignment" in training joint generation: (1) We first train the cross-attention weights in the image and text diffusers, as well
Table 1: Training tasks (CT stands for "contrastive learning" to align prompt encoders) and datasets with corresponding statistics. * denotes the number of accessible examples in the original datasets.
Categories	Tasks	Datasets	# of samples	Domain
Image + Text	Image→Text, Text→Image	Laion400M [42]	400M	Open
Text→Image+Text
Audio + Text	Text→Audio, Audio→Text,	AudioSet [16]	900K*	YouTube
Text→Audio+Text, Audio-Text CT	AudioCaps [24]	46K	YouTube
Freesound 500K	2.5M	Public audio samples
BBC Sound Effect	30K	Authentic natural sound
Audiovisual	Image→Audio, Image→Video+Audio	AudioSet	900K*	YouTube
SoundNet [3]	1.0M*	Flickr, natural sound
Video	Text→Video, Image→Video,	Webvid10M [4]	10.7M	Short videos
Video-Text CT	HD-Villa-100M [54]	100M	YouTube
Figure 3: Single-to-single modality generation. Clockwise from top left: text→image, image→text, image→video, audio→image.
as their environment encoders  $V$ , on text-image paired data. (2) We freeze the weights of the text diffuser and train the environment encoder and cross-attention weights of the audio diffuser on text-audio paired data. (3) Finally we freeze the audio diffuser and its environment encoder, and train the joint generation of the video modality on audio-video paired data. As demonstrated in Section 5.3, although only trained on three paired joint generation tasks (i.e., Text+Audio, Text+Image, and Video+Audio), CoDi is capable of generating assorted combinations of modalities simultaneously that are unseen in training, e.g., joint image-text-audio generation in Fig. 5.
4.1 Training Objectives and Datasets
We list training tasks of CoDi in Table 1, including single modality synthesis, joint multimodal generation, and contrastive learning to align prompt encoders. Table 1 provides an overview of the datasets, tasks, number of samples, and domain. Datasets are from the following domains: image + text (e.g. image with caption), audio + text (e.g. audio with description), audio + video (e.g. video with sound), and video + text (e.g. video with description). As one may have noticed, the language modality appears in most datasets and domains. This echos the idea of using text as the bridge modality to be able to extrapolate and generate new unseen combinations such as audio and image bridged by text, as mentioned in Section 3.2 and Section 3.4. Due to space limit, more details on training datasets and can be found in Appendix C, model architecture details in Appendix Appendix A.1, and training details in Appendix B.
Image + Text. We use a recently developed large-scale image caption dataset, Laion400M [42]. This image-text paired data allows us to train with tasks text→image, image→text, and the joint
Table 2: COCO-caption [32] FID scores for text-to-image generation.
Method	FID ↓
CogView [10]	27.10
GLIDE [36]	12.24
Make-a-Scene [15]	11.84
LDM [41]	12.63
Stable Diffusion-1.4	11.21
Stable Diffusion-1.5	11.12
Versatile Diffusion [53]	11.10
CoDi (Ours)	11.26
Table 3: MSR-VTT text-to-video Table 4: UCF-101 text-to-video generation performance. generation performance.
Method	Zero-Shot	CLIPSIM↑
GODIVA [50]	No	0.2402
NÜWA [51]	No	0.2439
CogVideo [22]	Yes	0.2631
Make-A-Video [44]	Yes	0.3049
Video LDM [5]	Yes	0.2929
CoDi (Ours)	Yes	0.2890
Method	IS (↑)	FVD (↑)
CogVideo (Chinese)	23.55	751.34
CogVideo (English)	25.27	701.59
Make-A-Video	33.00	367.23
Video LDM	33.45	550.61
CoDi (Ours)	32.88	596.34
Table 5: The comparison between our audio diffuser and baseline TTA generation models. Evaluation is conducted on AudioCaps test set. AS, AC, FSD, BBC, and SDN stand for AudioSet, AudioCaps, Freesound, BBC Sound Effect, and Soundnet.
Model	Datasets	FD ↓	IS ↑	KL ↓	FAD ↓	OVL ↑	REL ↑
Ground truth	-	-	-	-	-	83.61	80.11
DiffSound	AS + AC	47.68	4.01	2.52	7.75	45.00	43.83
AudioGen	AS + AC + 8 others	-	-	2.09	3.13	-	-
AudioLDM-L-Full	AS + AC + FSD + BBC	23.31	8.13	1.59	1.96	65.91	65.97
CoDi (Ours)	AS + AC + FSD + BBC + SDN	22.90	8.77	1.40	1.80	66.87	67.60
Table 6: COCO image captioning scores comparison.
Model	B@4	METEOR	CIDEr
Autoregressive Model
Oscar [31]	36.58	30.4	124.12
ClipCap [35]	32.15	27.1	108.35
OFA [49]	44.9	32.5	154.9
BLIP2 [30]	43.7	-	145.8
Diffusion Model
DDCap [59]	35.0	28.2	117.8
SCD-Net [34]	39.4	29.2	131.6
CoDi (Ours)	40.2	31.0	149.9
Table 7: AudioCaps audio captioning scores comparison.
Model	SPIDEr	CIDEr	SPICE
AudioCaps [24]	0.369	0.593	0.144
BART-Finetune [17]	0.465	0.753	0.176
VALOR [7]	-	0.741	-
AL-MixGen [25]	0.466	0.755	0.177
CoDi (Ours)	0.480	0.789	0.182
Table 8: MSRVTT video captioning scores comparison.
Model	B@4	METEOR	CIDEr
ORG-TRL [58]	43.6	28.8	50.9
MV-GPT [43]	48.9	38.7	60.0
GIT [48]	54.8	33.1	75.9
mPLUG-2 [52]	57.8	34.9	80.3
CoDi (Ours)	52.1	32.5	74.4