SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MultiFusion: Fusing Pre-Trained Models for Multi-Linguial, Multi-Modal Image Generation
Marco Bellagente $^{4*}$ † Manuel Brack $^{2,3\dagger}$  Hannah Teufel $^{1\dagger}$  Felix Friedrich $^{3,6}$  Björn Deiseroth $^{1,3,6}$  Constantin Eichenberg $^{1}$  Andrew Dai $^{1}$  Robert J.N. Baldock $^{1}$  Souradeep Nanda $^{5*}$  Koen Oostermeijer $^{1}$  Andres Felipe Cruz-Salinas $^{1}$  Patrick Schramowski $^{2,3,6,8}$  Kristian Kersting $^{2,3,6,7‡}$  Samuel Weinbach $^{1‡}$ $^{1}$ Aleph Alpha,  $^{2}$ German Research Center for Artificial Intelligence (DFKI),  $^{3}$ Computer Science Department, TU Darmstadt,  $^{4}$ Stability AI,  $^{5}$ University of Texas,  $^{6}$ Hessian.AI,  $^{7}$ Centre for Cognitive Science, TU Darmstadt,  $^{8}$ LAION marco.bellagente@gmail.com brack@cs.tu-darmstadt.de hannah.teufel@aleph-alpha.de
The recent popularity of text-to-image diffusion models (DM) can largely be attributed to the intuitive interface they provide to users. The intended generation can be expressed in natural language, with the model producing faithful interpretations of text prompts. However, expressing complex or nuanced ideas in text alone can be difficult. To ease image generation, we propose MULTIFUSION that allows one to express complex and nuanced concepts with arbitrarily interleaved inputs of multiple modalities and languages. MULTIFUSION leverages pre-trained models and aligns them for integration into a cohesive system, thereby avoiding the need for extensive training from scratch. Our experimental results demonstrate the efficient transfer of capabilities from individual modules to the downstream model. Specifically, the fusion of all independent components allows the image generation module to utilize multilingual, interleaved multimodal inputs despite being trained solely on monomodal data in a single language.
The recent popularity of text-to-image diffusion models (DM) [39, 35, 37] can largely be attributed to the intuitive interface they provide to users. The intended generation can easily be expressed in natural language, with the model producing faithful interpretations of a text prompt. Recent works have demonstrated the output quality to be largely dependent on the input encoders with more powerful variants yielding more expressive DMs [39, 2, 12]. We take these insights one step further, vastly enhancing the capabilities of a pre-trained DM through the sophisticated integration of dedicated modules. We propose MULTIFUSION which effectively supports arbitrarily interleaved inputs of multiple modalities and languages. Further, we transfer these capabilities from an underlying language model (LM), eliminating the need for multilingual or multimodal interleaved training data for the generative model. Our approach can utilize readily available datasets and requires less than  $5\%$  of the training compute needed to build a comparable DM from scratch.
*Work performed while at Aleph Alpha
$\ddagger$  Equal supervision
Figure 1: MULTIFUSION architecture. We augment a Decoder Language Model (1.1.) with adapters, finetuned for multimodality (1.2) as well as biases (2.1), finetuned for semantic search. Next, we condition the diffusion model (1.3) through cross-attention (2.2) on embeddings produced by the LM. During diffusion training, we use either the image or the caption for conditioning, while inference is performed with multimodal input. (Best viewed in color.)
The capabilities of current text-to-image DMs are often restricted by the types of inputs they support. MULTIFUSION addresses two of these limitations, facilitating more expressive prompting. Firstly, most of the widely available models are only designed for one language, whereas MULTIFUSION supports five. Contrary to existing multilingual DMs [12], MULTIFUSION requires no multilingual data for DM training and instead only uses readily available English training data. Secondly, text is inherently limited in its expressiveness—many concepts are difficult to articulate with words alone [30]. Consequently, a number of models have been proposed, allowing visual inputs as reference for image generation. Popular examples include image variation [48], adherence to an exemplary style [38], sketch-guided generation [45], or incorporating personal concepts in the resulting images [19]. All of these methods go beyond established image editing techniques [1, 20, 44, 8] where certain aspects of the input picture are altered. However, these image-to-image methods are limited to specialized tasks and lack the same level of control and generative versatility for image inputs as for natural language. MULTIFUSION provides a powerful and flexible interface for arbitrary combinations of multimodal and multilingual inputs. The extended prompt capabilities result in a more expressive model facilitating a wide range of complex applications and use cases.
Our main contributions are as follows: (1) We present MULTIFUSION, a multilingual, multimodal diffusion model (2) efficiently bootstrapped using a modular encoder based on an auto-regressive language model. (3) We highlight the use of attention manipulation [14] for multimodal prompting, and (4) demonstrate the transfer of an LM's multilingual capabilities to downstream tasks, removing the need for multilingual downstream training data. (5) Lastly, we introduce the MCC-250 benchmark for the evaluation of compositionality from multimodal inputs.
Text-conditioned diffusion. Diffusion models [39, 35, 2, 37] iteratively denoise a Gaussian distributed variable to produce samples of a learned data distribution. Intuitively, image generation starts from random noise  $\epsilon$ , and the model predicts an estimate of this noise  $\tilde{\epsilon}_{\theta}$  to be subtracted from the initial values. The denoising process results in a high-fidelity image  $x_0$  without any noise. Since this is an extremely hard problem, multiple steps are applied, each subtracting a small amount  $(\epsilon_{t})$  of the predictive noise, approximating  $\epsilon$ . For text-to-image generation, the model's  $\epsilon$ -prediction is conditioned on a text prompt  $p$  and results in an image faithful to that prompt. The training objective of a diffusion model  $\hat{x}_{\theta}$  can be written as
$$
\mathbb {E} _ {\mathbf {x}, \mathbf {c} _ {p}, \epsilon , t} \left[ w _ {t} \| \hat {\mathbf {x}} _ {\theta} \left(\alpha_ {t} \mathbf {x} + \omega_ {t} \epsilon , \mathbf {c} _ {p}\right) - \mathbf {x} \| _ {2} ^ {2} \right] \tag {1}
$$
where  $(\mathbf{x},\mathbf{c}_p)$  is conditioned on text prompt  $p$ ,  $t$  is drawn from a uniform distribution  $t\sim \mathcal{U}([0,1])$ ,  $\epsilon$  sampled from a Gaussian  $\epsilon \sim \mathcal{N}(0,\mathbf{I})$ , and  $w_{t},\omega_{t},\alpha_{t}$  influence image fidelity. Consequently, the DM is trained to denoise  $\mathbf{z}_t\coloneqq \mathbf{x} + \epsilon$  to yield  $\mathbf{x}$  with the squared error as loss. At inference, the DM is sampled using the model's prediction of  $\mathbf{x} = (\mathbf{z}_t - \bar{\epsilon_\theta})$ , with  $\bar{\epsilon_{\theta}}$  as described below.
Classifier-free guidance [22] is a conditioning method using a purely generational diffusion model, eliminating the need for an additional pre-trained classifier. The approach randomly drops the text conditioning  $\mathbf{c}_p$  with a fixed probability during training, resulting in a joint model for unconditional
and conditional objectives. During inference score estimates for the x-prediction are adjusted so that:
$$
\tilde {\epsilon} _ {\theta} \left(\mathbf {z} _ {t}, \mathbf {c} _ {p}\right) := \epsilon_ {\theta} \left(\mathbf {z} _ {t}\right) + s _ {g} \left(\epsilon_ {\theta} \left(\mathbf {z} _ {t}, \mathbf {c} _ {p}\right) - \epsilon_ {\theta} \left(\mathbf {z} _ {t}\right)\right) \tag {2}
$$
with guidance scale  $s_g$  which is typically chosen as  $s_g \in (0, 20]$  and  $\epsilon_{\theta}$  defining the noise estimate with parameters  $\theta$ . Intuitively, the unconditioned  $\epsilon$ -prediction  $\epsilon_{\theta}(\mathbf{z}_t)$  is pushed in the direction of the conditioned  $\epsilon_{\theta}(\mathbf{z}_t, \mathbf{c}_p)$  to yield an image faithful to prompt  $p$ . Lastly,  $s_g$  determines the magnitude of the influence of the text  $p$ . Naturally, the prompt  $p$  is not fed directly into the model, but instead a high dimensional representation of  $p$  is obtained through a decoder. In prevalent models, the prompt  $p$  is text in natural language, whereas in the case of MULTIFUSION,  $p$  is a sequence of text and images.
Multimodality. Prevalent encoder models like CLIP [34]—or multilingual variants like AltCLIP[12]—are distinctly unsuited for arbitrarily interleaved sequences of images and text. Unlike MULTIFUSION's multimodal encoder, these architectures rely on two separate encoders for textual and visual inputs, with their respective representations being aligned during training. In contrast to our model, the resulting embeddings only encode a single image or textual description but no interleaved sequences comprised of both modalities. Prior work shows that large language models (LMs) produce meaningful representations for conditioning of generative diffusion models [39, 2]. Additionally, pre-trained capabilities of LMs transfer to downstream tasks even without specific finetuning and beyond the initial modalities [9, 29]. SBERT has demonstrated that pre-trained transformer-based LMs can be used to construct encoders for longer contexts [36], albeit exclusively for natural language sequences. Consequently, MULTIFUSION builds its encoder on a pre-trained LM, achieving context-preserving embeddings for multilingual, multimodal inputs.
Other works have used various forms of image conditioning for diffusion models to enable more expressive prompts. Versatile Diffusion [48] enables the generation of image variations through a unified multimodal framework. Rombach et al. [38] proposed retriever-augmented diffusion models facilitating conditioning on particular visual styles provided via exemplary images. Multiple works have introduced methods to generate high-quality, text-conditioned images from low-resolution inputs such as sketches [45, 51]. Furthermore, textual inversion [19] turns concepts from example images into word embeddings that can subsequently be used during generation. This enables incorporating individual styles or objects into generated images. Lastly, Liu et al. [28] proposed a more general approach for diffusion guidance with image inputs using CLIP. Similarly, Bansal et al. [3] applied arbitrary guidance functions to more capable diffusion models. Such methods facilitate image generation, for example, from segmentation maps, image variations, or style transfer. However, this type of guidance requires a separate model and more complex, hand-crafted pipelines. In contrast, MULTIFUSION introduces a unified, general pipeline for effective direct conditioning through classifier-free guidance, removing the need for separate components. Concurrent with our work, GlueGen also attempt the task of aligning additional input modalities to pre-trained text-to-image models [33]. We encourage the reader to refer to their work for more details.
Multilingualism. Existing LMs pre-trained on multilingual data show impressive multilingual capabilities [40, 26, 13, 49]. Popular text-to-image DM's, however, are usually designed for a single input language [39, 37, 2, 17]. AltDiffusion [12] addressed this issue by proposing a multilingual text encoder (AltCLIP) on which the DM is conditioned instead. AltCLIP embeddings are aligned to the previously used CLIP encoder in a contrastive teacher-student setup using a large multilingual corpus. Subsequently, the cross-attention layers of a pre-trained Stable Diffusion model are finetuned to utilize the AltCLIP encoder instead. The resulting AltDiffusion model can be prompted in 9 different languages. AltDiffusion's image generation is aligned so that the same prompt in different languages results in similar images. With MULTIFUSION we leverage cross-lingual transfer [13, 49] to enable multilingual prompting. More precisely, our generative model obtains multilingual capabilities from the encoder despite being solely trained on English data.
3 MULTIFUSION
By fusing pre-trained model components, MULTIFUSION creates one cohesive system that requires less than  $5\%$  of the computational resources needed to train a comparable model from scratch. Our approach involves replacing the encoder of Stable Diffusion (SD) with a more advanced one built on a pre-trained LM. Fusing these components results in a downstream model with the ability to comprehend multilingual, interleaved multimodal inputs. The image generation component inherits this potential despite being trained solely on mono-modal data in a single language. The architecture
of MULTIFUSION is illustrated in Fig. 1. The vast majority of pre-trained weights remain frozen, resulting in an efficient computational process overall. Further information on the training data sizes, parameter counts, and GPU hours for all components is supplied in Tab. 3, along with details on data splits, languages, and modalities in Tab. 4 of the appendix. Subsequently, we outline how to combine and align the involved modules for image generation effectively.
Input encoders. The CLIP encoder [34] used by SD is unsuited for interleaved multimodal inputs as it disregards context and yields disjoint encodings of text and images. Previous work has demonstrated that text encoders based on context-sensitive LMs improve the expressiveness of downstream image generation models [39, 2]. Accordingly, we model the backbone of MULTIFUSION's encoder as an autoregressive transformer [10] using rotary position embeddings [43] trained on a multilingual corpus of various languages (step 1.1 in Fig. 1). We chose an autoregressive decoder model over a bi-directional architecture since decoder models intuitively outperform bi-directional models on relevant tasks. For example, autoregressive models excel at manipulation with natural language ("Subject X with background Y") (cf. Sec. 4.2) or correct features attributions ("Red X, Blue Y") (cf. Sec. 4). Previous research has identified the natural breaking of permutation equivariance as the source of these capabilities [24], compared to bidirectional models relying entirely on positional embeddings. We acknowledge that bi-directional models may outperform autoregressive ones on other embedding tasks [50], but argue that an autoregressive model is better suited for the tasks studied in MULTIFUSION due to the outlined benefits.
Following the methodology proposed by MAGMA [15], we consider an LM with an added image prefix and dedicated adapters to enable multimodal capabilities (step 1.2 in Fig. 1). Adapters are a suitable architectural choice for multimodal prompts since previous research has already performed extensive ablations on adapter architectures and demonstrated their improved understanding of multimodal inputs over other methods [15]. In this architecture, the image prefix maps the image into sequences of token embeddings in a joint multimodal input embedding space. The adapters are added to each attention and feed forward layer of the transformer and are trained autoregressively on a combination of large-scale image-text datasets (cf. App. A), while the parameters of the language model remain frozen [15, 29, 23]. As a result, the LM enables prompting with arbitrarily interleaved sequences of text and image tokens.
Semantic embeddings. In order to use the LM as an encoder for image generation, we extract the model's hidden representations before the language modeling head. While these representations already capture semantic information from the model's pre-training, they need to be optimized and aligned further for usage in arbitrary downstream tasks (step 2.1 in Fig. 1). Initial experiments without additional alignment have shown low rates of convergence (based on visual inspection of generated outputs). Consequently, we chose to produce semantic embeddings guided by the intuition that a focus on the semantics of a text prompt would best capture the information relevant to image generation. Thus simplifying the learning of mapping from embeddings to image outputs, which was confirmed by our initial experimental observations. In conclusion, we deem semantic fine-tuning an essential condition for successfully fusing an image generation model.
We obtained high-quality semantic embeddings through parameter-efficient bias fine-tuning [4]. The tuning follows the supervised contrastive learning objective outlined in S-GPT [31]. The training data consists of natural language tuples of premise and hypothesis, where entailments serve as positive samples and contradictions as negative ones. Importantly, the tuples are bi-lingual such that the premise and hypothesis are each stated in different languages. We observed that including only two languages in this finetuning task is sufficient to achieve multilingual alignment even beyond bilingualism (cf. Sec. 4). Crucially, the semantic bias weights are tuned independently of the multimodal adapters, allowing for modular extensions of the LM with both components.
Bootstrapping Stable Diffusion. The final MULTIFUSION encoder is the result of combining these separately trained modules (cf. Fig. 1). The DM is conditioned on embeddings extracted from the last hidden layer of the transformer. Subsequently, we denote  $H(x)$  as embedding for input  $x$  after a forward pass through MULTIFUSION's multimodal LM encoder. We now align the pre-trained image generation model with MULTIFUSION's encoder (step 2.2 in Fig. 1). Considering the depicted SD architecture, we only need to alter the conditioning portion of the generative model to use our new encoder instead of CLIP. In line with previous research [12], we keep all weights of the DM frozen and only finetune the cross-attention layers of the U-Net. The training objective remains the same as shown in Eq. 1 and 2 with the conditioning  $\mathbf{c}_p$  being the encoding  $H(x)$ .
Table 1: FID-30k and ClipScores on the MS-COCO validation set for MULTIFUSION (MF) and Stable Diffusion (SD). SD is always prompted with text. All images were generated with image size 256x256 and 100 diffusion steps. Textual prompts consisted of the COCO image caption, multimodal prompts of the caption and COCO reference image, and image prompts of just the image.
FID-30K ↓	CLIPScore (Text-to-Image) ↑
Guidance Scale	SD v1.5	MF (Text)	MF (Multimodal)	MF (Image)	SD v1.5	MF
8.0	14.62	14.19	11.50	8.02	0.31	0.30
6.0	12.73	12.15	10.29	7.18	0.31	0.29
4.0	10.19	9.90	8.53	6.03	0.31	0.29
2.0	9.74	12.21	8.61	6.05	0.30	0.28
1.0	26.09	32.81	24.22	18.93	0.27	0.25
We trained the DM only on monolingual (English) and monomodal inputs with  $x$  being randomly chosen as either an image or a caption. Nonetheless, the final MULTIFUSION model is capable of interpreting multilingual and arbitrarily interleaved text and image prompts. Our method highlights that the capabilities of strong LMs can be transferred to downstream tasks without the need for dedicated downstream training data. We provide empirical evidence in the following section.
Modality alignment. During initial experiments, we observed that further modifications to the inference mechanics are needed to enable stable multimodal prompting. More specifically, for interleaved text and image prompts MULTIFUSION's encoder represents one image as a sequence of 144 tokens in the input embedding space. In most scenarios, the accompanying text prompt contains fewer tokens, resulting in a disproportional influence of the visual inputs on the generated image. To counteract this phenomenon, we utilize attention manipulation [14] to up-weight the impact of textual tokens with respect to the discrepancy in input length. Representative results showcasing the effect of attention manipulation can be found in Appendix D.
Next, we present exhaustive evaluations of the multimodal and multilingual capabilities of MULTIFUSION on various benchmarks. To evaluate compositional robustness on models allowing multi-modal inputs, we introduce the new MCC-250 benchmark. Furthermore, we showcase a variety of applications enabled by MULTIFUSION. We provide a detailed experimental protocol, including information on implementation and training setup in App. A.
4.1 Empirical evaluation
Image fidelity & image-text alignment. We start off with a standard evaluation for image generation models to demonstrate the general functionality of MULTIFUSION. We investigate image fidelity using FID-30k scores on the MS COCO validation set [25]. We report the results using textual, multimodal, and image prompts and a comparison against SD v1.5 in Tab. 1. The results show that the image fidelity of MULTIFUSION with textual prompts is competitive with the underlying SD model. Improved FID scores highlight that the capabilities of the original model are preserved in addition to benefiting from new input modalities and languages.
Using multimodal inputs instead of textual prompts results in a substantial improvement of FID scores. This improvement clearly indicates that visual inputs possess a greater capacity to encode comprehensive information about the underlying distributions, surpassing the effectiveness of textual descriptions alone. We acknowledge that using the MS COCO reference images as prompts provides a strong supervision. However, we argue that the above conclusion of image inputs adding additional and more fine-grained information over text prompts alone still holds. Because MULTIFUSION achieves the improved FID scores by generating meaningful variations of the prompt with more aligned details instead of just trivially reproducing the input image (cf. Sec. 4.2 and App. B). Beyond FID, we evaluated text-image alignment as average CLIPScore [21] between COCO captions and generated images. Again, MULTIFUSION achieves scores on par with SD, confirming the preservation of previous capabilities.
Table 2: Fine-grained human evaluation results on MCC-250. SD and Composable Diffusion were prompted with text, whereas we show results for both text and multimodal prompts for MULTIFUSION. Specifically, multimodal prompts contain one visual reference for each object interleaved with the text input. Recall that each prompt is a complex conjunction of two different objects with different colors. MULTIFUSION with multimodal inputs strongly outperforms both baseline models.
Methods	Zero obj ↓	One obj. ↑	One obj. w/ correct color ↑	Two obj. ↑	Two obj. w/ correct colors ↑
Stable Diffusion [%]	0.92±4.81	99.07±4.89	90.01±13.97	44.89±28.61	29.92±24.76
Composable Diffusion [%]	3.88±7.49	96.01±7.72	88.49±42.83	34.72±22.79	25.59±18.94
MultiFusion (text) [%]	1.08±4.55	98.91±4.57	82.36±18.98	36.03±29.17	21.66±22.23
MultiFusion (multimodal) [%]	0.55±2.81	99.44±2.85	94.88±11.37	65.06±30.64	58.35±30.94
Figure 3: Multilingual alignment of images generated by MULTIFUSION. All images were generated using the same seed and with the respective translation of the prompt 'an image of an astronaut riding a horse'. (Best viewed in color)
Compositional robustness. A challenging task many image synthesis models struggle with is image composition. DMs and, in particular, SD often fail to correctly compose the objects and attributes specified in the text prompt [11, 16].
MULTIFUSION, on the other hand, behaves robustly with respect to challenges in compositional generation as shown in Fig. 2. For evaluation, we propose a new benchmark, Multimodal Concept Conjunction 250 (MCC-250) $^4$ . It builds on a subset of the Concept Conjunction 500 (CC-500) benchmark [16]. CC-500 contains 500 text prompts of the pattern "a red apple and a yellow banana", textually describing two objects with respective attributes. For half of those prompts, we curated a set of images for each object, enabling multimodal prompting, i.e. the textual description is interleaved with a visual reference. For the new MCC-250 benchmark, we present a human evaluation of SD, Composable Diffusion [27] and MULTIFUSION in Tab. 2. Note that all approaches use the same image generation module. For MULTIFUSION we evaluate multimodal prompts (text and image) as well as text-only.