SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 2: The multimodality of MULTIFUSION proves more robust for image compositions. SD is prompted in text with 'A photorealistic image of {Descr}'. MULTIFUSION prompts contain interleaved visual references. (Best viewed in color.)
On this complex task, MULTIFUSION clearly outperforms both baseline models, almost doubling the portion of images containing both objects with correct colors. Successful compositions once again emphasize the capacity of multimodal prompting in MULTIFUSION. Importantly, the observed performance improvement originates from the multimodality of inputs, as the success rate on text-only inputs remains comparable. Surprisingly, Composable Diffusion performs slightly worse than SD for image composition. The resulting images frequently display strange blends of the two objects rather than capturing them as distinct entities. We provide further details on the user study in App C.
Multilingual alignment. Next, we investigate the multilingual capabilities of MULTIFUSION. Therefore, we evaluated the alignment of directly translated text inputs in prompt embedding space and the generated images. We compare MULTIFUSION with AltDiffusion [12], as it also builds on a frozen SD model. AltDiffusion makes for a great comparison, as the key differences between both
4We make the MCC-250 benchmark available at https://huggingface.co/datasets/AIML-TUDA/MCC-250
models lie in the respective encoders and training data. AltDiffusion uses a large multilingual dataset to finetune the DM, whereas MULTIFUSION's finetuning requires only English training data.
The evaluation is based on a multilingual version of DrawBench [39] $^5$ . To that end, we translated all prompts into the secondary languages of MULTIFUSION and AltDiffusion: German and Chinese. Additionally, we include the three languages shared by both models: French, Spanish, and Italian. In Fig. 4a, we plot the alignment of multilingual text embeddings over DrawBench. MULTIFUSION's encoder clearly outperforms AltDiffusion's encoder (AltClip) on embedding alignment, scoring  $97\%$  cosine similarity on average. The alignment of the generated images is similar for both models, although MULTIFUSION was only finetuned using English data (cf. Fig. 4b). These results highlight that good multilingual embedding alignment enables the transfer of multilingualism to downstream tasks without the need for explicitly multilingual training data.
After we empirically demonstrated MULTIFUSION's multimodal, multilingual prompting capabilities, we show how it facilitates diverse interesting use cases. We provide more examples in the Appendix.
Image composition. One of the main strengths of MULTIFUSION's multimodal inputs is image composition. As demonstrated in our empirical evaluation and throughout Figs. 5a and 5b, text and image sequences can be combined arbitrarily and flexibly. Interleaved inputs allow for intuitive combinations of images (cf. Fig. 5b) and serve as a visual reference to enrich text prompts.
Negative prompting. Negative prompting refers to a technique in image generation that aims to suppress certain concepts. To that end, the unconditioned estimate during classifier-free guidance  $\epsilon(z_{t})$  (cf. Eq. 2) is replaced with an estimate conditioned on a negative prompt  $\epsilon(z_{t}, c_{n})$ . Guiding away from  $\epsilon(z_{t}, c_{n})$  results in the concepts described by  $n$  being avoided. Consequently, negative prompting works better with more descriptive prompts  $n$ . Our previous results demonstrated (cf. Tab. 1) a higher expressiveness of images over text that also translates to negative prompts. Fig 5c shows that textual negative prompts are less effective in removing undesired concepts. However, using image prompts, these concepts can be completely suppressed.
Image variation. MULTIFUSION offers a direct interface for generating image variants. Simply providing a single image as input prompt to the default pipeline produces meaningful variations already. Other models, in contrast, rely on inversion or re-noising of the input image. We depict
5We did not include prompts from the categories 'misspellings' and 'rare words'.
(b) Multimodal Image Composition.
(c) Negative Prompting.
Figure 5: Applications of MULTIFUSION highlighting the versatility and expressiveness of multilingual, multimodal prompts. (Best viewed in color)
examples in Fig. 6a. The generated images include sensible deviations from the input but faithfully reproduce the underlying image contents.
Style modification. Furthermore, MULTIFUSION can easily generate images adhering to any artistic style. The style of images is notoriously hard to describe in image generation prompts and has even led to the development of monetized prompt databases for long, obfuscated prompts. This is mostly due to the magnitude of factors that have to be accounted for, such as the color palette, composition, contrast, etc. Fortunately, all of these aspects can easily be expressed through exemplary (reference) images. Fig. 6b depicts examples where arbitrary styles are applied to various scene descriptions. MULTIFUSION delivers high-quality outputs that show the prompted scene in the desired style.
Multilingualism. Lastly, the strong multilingual embedding alignment makes MULTIFUSION largely invariant to the language of input prompts. Our model fully supports input prompts in five languages: English, German, French, Spanish, and Italian. In line with our results on language alignment, Fig. 3 shows that the same prompt in different languages yields largely similar images. This improves accessibility and expands the group of users, regardless of their native language. Moreover, Fig. 5a demonstrates that languages can even differ within the same input. This further emphasizes MULTIFUSION's expressiveness, allowing users to construct prompts in various ways.
The capabilities of MULTIFUSION outlined above emphasize the advantages of the model, which we subsequently discuss alongside the remaining limitations and overall societal impact.
5.1 Advantages of MULTIFUSION
Multimodality and multilingualism of inputs offer various benefits compared to prevalent models. The extension of the interface beyond the usual English-only text-to-image applications makes MULTIFUSION more flexible, expressive, and versatile.
Expressiveness. Natural language and images both have benefits and limitations in the concepts they can convey. When used in combination, however, we are able to draw from the advantages of one and the other, thus eliminating the restrictions of either modality. On the one hand, complex objects that should be included can easily be prompted via exemplary images instead of using long, convoluted, and error-prone textual descriptions. Natural language is often unclear and ambiguous and may lack words to describe certain concepts concisely. For the car depicted in Fig. 5a, for example, MULTIFUSION faithfully infers the vehicle's color, make, type, and era and uses that as a reference for image generation. Achieving similar results through textual descriptions alone requires complicated prompt engineering and may be unsuccessful altogether. Additionally, some necessary
(a) Image Variations. MULTIFUSION produces non-trivial variations when prompted with a single image.
(b) Style transfer from reference images. MULTIFUSION is prompted with a text prompt (left) + 'in the style of <image>'. Generated images are faithful to the targeted scenario and style.
Figure 6: Further applications of MULTIFUSION. (Best viewed in color)
terminology may elude the end user. For example, a non-native speaker may not be aware that the type of building in Fig. 5a can be described as half-timbered. Users may consequently write sub-optimal textual descriptions which can be circumvented using visual prompts. Furthermore, images convey world knowledge that the model does not possess. For example, not all existing artworks or paintings are included in the training data and retained by the model (cf. Fig. 6b). Consequently, using their stylistic characteristics as reference is difficult through text alone. MULTIFUSION easily bridges this knowledge gap by facilitating the inclusion of these images in the prompt. This even enables using individual experiences for reference, such as my car, a sketch I drew, or a building I saw. On the other hand, natural language provides an intuitive interface for abstract concepts and scenarios. For example, the drawing of the car in Fig. 5a is easily turned into a photorealistic version through textual instruction. Consequently, the capabilities of MULTIFUSION result from combining the expressiveness of images and the abstraction of natural language. The inclusion of multiple languages only further increases the model's expressiveness.
Robustness and clarity. Similarly, the availability of multimodal inputs significantly improves the robustness of MULTIFUSION. The combination of visual and natural language inputs allows users to circumvent the limitations of the modalities themselves and those of current DMs. For example, natural languages contain a variety of homographs, i.e., words with the same spelling but different meanings. Inferring the correct one from context is often impossible, whereas images provide clear examples. On the other hand, images alone may contain various sources of noise, such as background objects or stylistic choices that the user might not want to consider for generation. However, when used in combination, one modality is able to make up for the shortcomings of the other.
Furthermore, MULTIFUSION overcomes the limitations of current DMs in compositional generation [11, 16]. As demonstrated on the MCC-250 benchmark, images generated from multimodal prompts with MULTIFUSION are less likely to miss objects specified in the prompt or bind attributes incorrectly. This is due to MULTIFUSION devoting multiple tokens to an object described via an image and using context-aware embeddings. In contrast, text-only generation using models such as Stable Diffusion often leads to regular attribute leakage or interchanged attributes, cf. Fig. 2. In this case, the green color of the apple leaks to the car, or colors are mixed up between objects. We include further examples in App. C showcasing the resilience of MULTIFUSION.
Licensing. Recently, copyright issues and licensing of training images have been heavily discussed topics<sup>6</sup>. Current legal actions<sup>7</sup> may lead to a variety of images—especially creations from artists—being no longer admissible for use in training. Consequently, applications such as instructing the
https://www.copyright.gov/rulings-filings/review-board/docs/a-recent-entrance-to-paradise.pdf  
https://www.govinfo.gov/content/pkg/FR-2023-03-16/pdf/2023-05321.pdf  
https://stablediffusionlitigation.com
model to reference an artist's style will be infeasible even though the end user might have the necessary copyright permissions. With MULTIFUSION, however, a user can still provide reference material not included in the training data simply in the input.
While MULTIFUSION achieves impressive results on various applications, there are some limits to the model's expressiveness. For one, MULTIFUSION always produces meaningful variations when prompted with a single input image. While this is an advantage in some settings, it may be a limitation in others since exactly copying objects from an input is not feasible. This originates from the encoder not being designed to reconstruct images from its representations but to encode relevant characteristics efficiently. Nonetheless, this behavior is generally intended as images are supposed to serve as references to MULTIFUSION instead of performing image editing. Additionally, the quality and composition of an input image have a significant effect on the generated image. We observed that some visual inputs must be carefully selected to achieve the desired outcome. For example, MULTIFUSION may also include unwanted items from the background of an image in the generated output. Furthermore, in some cases, MULTIFUSION has a tendency to copy the input image style even when the prompt's textual portion indicates a different one. Nonetheless, this can be adequately addressed with the proposed attention manipulation method [14], as demonstrated throughout the examples in this work.
5.3 Societal impact
Recent developments in text-to-image models [35, 32, 39, 2] have the potential for a far-reaching impact on society, both positive and negative when deployed in applications such as image generation, image editing, or search engines. Previous research [41, 18] described many potential negative societal implications that may arise due to the careless use of such large-scale generative models. Many of these problems can be attributed to the noisy, large-scale datasets these models rely on. Since recent text-to-image models, such as SD, are trained on web-crawled data containing inappropriate content [42, 6, 5], they are no exception to this issue. Specifically, models relying on the LAION datasets [42] show signs of inappropriate degeneration [41]. Consequently, we assume MULTIFUSION to suffer from similar shortcomings only reinforced by its additional capabilities. Therefore, we will not make the model weights publicly available in their current form.
In this work, we introduced MULTIFUSION, a diffusion model utilizing multilingual, arbitrarily interleaved multimodal inputs. These capabilities provide a versatile interface to users in order to better express themselves. We demonstrated that multilingual alignment in the encoder is sufficient to achieve multilingualism in downstream tasks, eliminating the need for dedicated multilingual datasets. Thus, significantly reducing computational requirements. More generally, MULTIFUSION highlights how separate pre-trained components can be interconnected to realize complex models.
Numerous promising avenues for future research emerge from our current work. One intriguing direction involves expanding the scope of MULTIFUSION to facilitate interactive image generation through integration with a chat interface. This novel extension would offer an enhanced user experience, introducing a dynamic interplay between the system and users. Additionally, we envision the potential for recursive prompting of generated images, enabling a progressive refinement of output through incremental instructions. Moreover, the incorporation of an additional decoder, based on the MULTIFUSION encoder, holds promise for facilitating textual generation resulting in multimodal outputs. Lastly, we propose extending the encoder itself to encompass an even broader range of modalities, including audio, video, time series, and various others.
We gratefully acknowledge support by the German Center for Artificial Intelligence (DFKI) project "SAINT", the Federal Ministry of Education and Research (BMBF) project "AISC" (GA No. 01IS22091), and the Hessian Ministry for Digital Strategy and Development (HMinD) project "AI
Innovationlab" (GA No. S-DIW04/0013/003). This work also benefited from the ICT-48 Network of AI Research Excellence Center "TAILOR" (EU Horizon 2020, GA No 952215), the Hessian Ministry of Higher Education, and the Research and the Arts (HMWK) cluster projects "The Adaptive Mind" and "The Third Wave of AI", and the HMWK and BMBF ATHENE project "AVSV".
[48] Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. arXiv:2211.08332, 2022.
[49] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2021.
[50] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Retrieval-augmented multimodal language modeling. In International Conference on Machine Learning (ICML), 2023.
[51] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv:2302.05543, 2023.
Figure 7: Qualitative examples of images generated from MS COCO images and captions. We compare outputs from MULTIFUSION (MF), prompted with different modalities - text, image, and multimodal - as well as outputs from Stable Diffusion, which was prompted with text only. We use guidance scale 4.0 for both models.
A Experimental protocol
Subsequently, we provide further details on the datasets (4), architecture and training details (3) of MULTIFUSION. We start from Stable Diffusion v1.4 and discard the CLIP text encoder. MULTIFUSION's encoder is built on a 13B decoder-only transformer similar to GPT-3 [10] but using rotary position embeddings [43]. This LM was pre-trained on a multilingual corpus of roughly 400B tokens comprised of English, German, French, Italian, and Spanish data (cf 4).
We turned the model into a visual language model (VLM) following the method of MAGMA [15]. Therefore, we used a ResNet image encoder, initialized with CLIP weights, and extract the feature grid before the final pooling layers. These 12x12 features are then flattened to a sequence of 144 vectors with subsequent drop regularization and layer normalization resulting in the embedding representation of an image. Consequently, an image is represented by 144 token embeddings which are used as inputs to the transformer LM alongside the text embeddings. For parameter-efficient multimodal training, we added a bottleneck adapter with downsampling factor 8 [23] to each attention and feed-forward layer. Only the weights of these adapters and the image prefix module were optimized during multimodal pre-training. The multimodal components (image prefix and adapters) are trained autoregressively for 15.4 million image-text pairs on a proprietary multimodal dataset (cf 4). Following MAGMA [15], the image tokens are prepended to the text tokens, and the language modeling loss for next token prediction is computed over the text only. We based our multimodal dataset on the findings and extensive ablations of MAGMA [15].
In addition to the tuned multimodal adapters, we apply parameter-efficient bias finetuning [4] of the pre-trained language model to improve the semantic alignment of encoded representations. This turns the hidden representations of the transformer, learned during pre-training, into semantic embeddings usable for arbitrary downstream tasks. Our setup is similar to S-GPT [31], pooling the last hidden layer of the decoder into dense representations, which we optimized for semantic search. We used a
Table 3: Training details of MF's individual modules.
Module	Parameters	Trained parameters (MF)	Training data size	GPU hours
LM	13B	-	400B tokens	85K
Semantic embeddings	-	-	50M tokens	6.9K
Multimodal components	2B	-	15.4M image-text pairs	9.5K
SD (v1.4)	1B	152M (15%)	40M images	4.3K
Table 4: Training datasets of MF's individual modules. Only the semantic bias training uses paired multi-lingual data. All other training data is sourced from monolingual corpora.
Module	Dataset	Percentage	Languages	Modalities
LM	web crawl	71%	english, german,italian, spanish,french	text
books	20%
political and legal sources	5%
wikipedia	2%
news	2%
other	1%
Semantic embeddings	MNLI	72%	english, german	text
SNLI	28%
Multimodal components	image captioning tasks	91%	english	text, image
Wikipedia image-text	8.5%
visual question-answering tasks	0.5%
SD (v1.4)	LAION aesthetics	100%	english	text, image
custom version of SNLI [7] and MNLI [47] that extends the original English texts by their machine-translated German versions, which were generated using the DeepL API. Both datasets contain natural language tuples of premise and hypothesis along a judgment label that can either be entailment, contradiction, or neutral. For example, the hypothesis 'The man is sleeping' is a contradiction to the premise 'A man inspects the uniform of a figure in some East Asian country.' We optimize the bias weights of the LM using a contrastive learning objective for 13k steps where entailments serve as positive samples and contradictions as negative ones.
The diffusion model itself is kept frozen with only the cross-attention layers of the U-Net being re-trained to utilize MULTIFUSION's embedding space. As training data, we use LAION aesthetics  $V.25 + ^{8}$ , i.e., the subset of LAION 5B with English captions filtered by a predicted aesthetic score  $> 5$  [42]. Additionally, we discard images with a resolution lower than  $512 \times 512$ , resulting in roughly 40M captioned images. The final model is finetuned for 60k steps with the probability of using an image instead of a caption being 0.2.
We make some modifications to the mechanics at inference to better enable multimodal prompting. For interleaved text and image prompts, MULTIFUSION's encoder represents images as a sequence of 144 token embeddings. This results in a disproportional influence on the generated image compared to significantly shorter text portions. To counteract this phenomenon, we utilize attention manipulation [14] in every attention layer of the transformer encoder. Meaning that every attention score  $s_{ij}$  in an attention head is modified by a per-token factor  $\lambda_i$  so that  $\tilde{s}_{ij} = s_{ij} + \log \lambda_i$ . The authors argue, that  $\lambda_i > 1$  up-weights the  $i$ -th token, whereas  $\lambda_i < 1$  down-weights it. In practice, we use the same  $\lambda$  value to up-weight each text token of a prompt.
B MS COCO examples
In Fig.7, we show exemplary images generated from MS COCO prompts in all modalities (cf. Sec. 4). We show that multi-modal prompts enhance image quality as well as convey more information while still producing diverse and meaningful outputs, which are not mere copies of the original input image.
$^{8}$ https://laion.ai/blog/laion-aesthetics/
C MCC-250 - user study
Subsequently, we present further implementation details of the user study conducted on the MCC-250 benchmark (cf. Sec. 4), along with qualitative examples.
C.1 Study Details
For each model, we generated 10 images per prompt. Let's consider the example 'a green bench and a red car' from the benchmark. The prompt to Stable Diffusion was 'a photorealistic image of a green bench and a red car', whereas we prompted Composable Diffusion with the two prompts 'a photorealistic image of a green bench' and 'a photorealistic image of a red car' which were composed with equal weights. For MULTIFUSION, we interleaved the prompt with image references of each object: 'a photorealistic image of a green bench <image of a green bench> and a red car <image of a red car>.
We evaluated each of the two objects and the respective attribute separately, with questions being posed in the form:
Is there a green bench in the image?
The four answer options were:
There is no bench
- There is a bench, but it is not green
There is a green bench
- Cannot tell
Each user was tasked with labeling a batch of 28 image/attribute pairs; 25 out of those were randomly sampled from our generated images. Each batch contained three hand-selected images from the MCC-250 inputs as a sanity check. If users labeled these three images incorrectly, the batch was discarded and added back to the task pool. Each image/object combination was labeled by three different annotators resulting in annotator consensus if at least 2 selected the same label. The results in the main paper (cf. Tab. 2) only include those images with annotator consensus, which are distributed according to Tab. 5.
Table 5: Annotator consensus on the selected label per model for the MCC-250 user study.
Model	Annotator Consensus in [%]
Stable Diffusion	98.31
Composable Diffusion	93.85
MULTIFUSION (text)	98.71
MULTIFUSION (multimodal)	98.16
To conduct our study, we relied on Amazon Mechanical Turk, where we set the following qualification requirements for our users: HIT Approval Rate over  $95\%$  and at least 1000 HITs approved. Annotators were fairly compensated according to Amazon MTurk guidelines. Users were paid $0.60 for a batch of 28 images.
C.2 Qualitative Examples