Title: MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion

image generation

multimodal
Abstract: The recent popularity of text-to-image diffusion models (DM) can largely be attributed to the intuitive interface they provide to users. The intended generation can be expressed in natural language, with the model producing faithful interpretations of text prompts. However, expressing complex or nuanced ideas in text alone can be difficult. To ease image generation, we propose MultiFusion that allows one to express complex and nuanced concepts with arbitrarily interleaved inputs of multiple modalities and languages. MultiFusion leverages pre-trained models and aligns them for integration into a cohesive system, thereby avoiding the need for extensive training from scratch. Our experimental results demonstrate the efficient transfer of capabilities from individual modules to the downstream model. Specifically, the fusion of all independent components allows the image generation module to utilize multilingual, interleaved multimodal inputs despite being trained solely on monomodal data in a single language.

MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation

Abstract

The recent popularity of text-to-image diffusion models (DM) can largely be attributed to the intuitive interface they provide to users. The intended generation can be expressed in natural language, with the model producing faithful interpretations of text prompts. However, expressing complex or nuanced ideas in text alone can be difficult. To ease image generation, we propose MULTIFUSION that allows one to express complex and nuanced concepts with arbitrarily interleaved inputs of multiple modalities and languages. MULTIFUSION leverages pre-trained models and aligns them for integration into a cohesive system, thereby avoiding the need for extensive training from scratch. Our experimental results demonstrate the efficient transfer of capabilities from individual modules to the downstream model. Specifically, the fusion of all independent components allows the image generation module to utilize multilingual, interleaved multimodal inputs despite being trained solely on monomodal data in a single language.

1 Introduction

The recent popularity of text-to-image diffusion models (DM) [39, 35, 37] can largely be attributed to the intuitive interface they provide to users. The intended generation can easily be expressed in natural language, with the model producing faithful interpretations of a text prompt. Recent works have demonstrated the output quality to be largely dependent on the input encoders with more powerful variants yielding more expressive DMs [39, 2, 12]. We take these insights one step further, vastly enhancing the capabilities of a pre-trained DM through the sophisticated integration of dedicated modules. We propose MULTIFUSION which effectively supports arbitrarily interleaved inputs of multiple modalities and languages. Further, we transfer these capabilities from an underlying language model (LM), eliminating the need for multilingual or multimodal interleaved training data for the generative model. Our approach can utilize readily available datasets and requires less than 5% of the training compute needed to build a comparable DM from scratch.

(e.g., Stable Diffusion Unet)

Multi-Modal

bias

adapter

Q

K V

Q

K V

Q

K V

Q

K V

image

embed

MLP

Norm

Attention

Norm

2.1) Semantic Embed

2.2) Cross-Attention

1.2) Multi-Modality

1.1) LLM

riding on a horse

Multi-Modal Inputs

Transformer Blocks

text

embed

Step 1- Pre-trained Modules:

Step 2 - Fusion Tuning:

1.3) DM

The capabilities of current text-to-image DMs are often restricted by the types of inputs they support. MULTIFUSION addresses two of these limitations, facilitating more expressive prompting. Firstly, most of the widely available models are only designed for one language, whereas MULTIFUSION supports five. Contrary to existing multilingual DMs [12], MULTIFUSION requires no multilingual data for DM training and instead only uses readily available English training data. Secondly, text is inherently limited in its expressivenessÐmany concepts are difficult to articulate with words alone [30]. Consequently, a number of models have been proposed, allowing visual inputs as reference for image generation. Popular examples include image variation [48], adherence to an exemplary style [38], sketch-guided generation [45], or incorporating personal concepts in the resulting images [19]. All of these methods go beyond established image editing techniques [1, 20, 44, 8] where certain aspects of the input picture are altered. However, these image -to-image methods are limited to specialized tasks and lack the same level of control and generative versatility for image inputs as for natural language. MULTIFUSION provides a powerful and flexible interface for arbitrary combinations of multimodal and multilingual inputs. The extended prompt capabilities result in a more expressive model facilitating a wide range of complex applications and use cases.

Our main contributions are as follows: (1) We present MULTIFUSION, a multilingual, multimodal diffusion model (2) efficiently bootstrapped using a modular encoder based on an auto-regressive language model. (3) We highlight the use of attention manipulation [14] for multimodal prompting, and (4) demonstrate the transfer of an LM's multilingual capabilities to downstream tasks, removing the need for multilingual downstream training data. (5) Lastly, we introduce the MCC-250 benchmark for the evaluation of compositionality from multimodal inputs.

2 Background

Text-conditioned diffusion. Diffusion models [39, 35, 2, 37] iteratively denoise a Gaussian distributed variable to produce samples of a learned data distribution. Intuitively, image generation starts from random noise /epsilon1 , and the model predicts an estimate of this noise ˜ /epsilon1 θ to be subtracted from the initial values. The denoising process results in a high-fidelity image x 0 without any noise. Since this is an extremely hard problem, multiple steps are applied, each subtracting a small amount ( /epsilon1 t ) of the predictive noise, approximating /epsilon1 . For text-to-image generation, the model's /epsilon1 -prediction is conditioned on a text prompt p and results in an image faithful to that prompt. The training objective of a diffusion model ˆ x θ can be written as

where ( x , c p ) is conditioned on text prompt p , t is drawn from a uniform distribution t ∼ U ([0 , 1]) , /epsilon1 sampled from a Gaussian /epsilon1 ∼ N (0 , I ) , and w t , ω t , α t influence image fidelity. Consequently, the DMis trained to denoise z t := x + /epsilon1 to yield x with the squared error as loss. At inference, the DM is sampled using the model's prediction of x = ( z t -¯ /epsilon1 θ ) , with ¯ /epsilon1 θ as described below.

Classifier-free guidance [22] is a conditioning method using a purely generational diffusion model, eliminating the need for an additional pre-trained classifier. The approach randomly drops the text conditioning c p with a fixed probability during training, resulting in a joint model for unconditional

and conditional objectives. During inference score estimates for the x -prediction are adjusted so that:

with guidance scale s g which is typically chosen as s g ∈ (0 , 20] and /epsilon1 θ defining the noise estimate with parameters θ . Intuitively, the unconditioned /epsilon1 -prediction /epsilon1 θ ( z t ) is pushed in the direction of the conditioned /epsilon1 θ ( z t , c p ) to yield an image faithful to prompt p . Lastly, s g determines the magnitude of the influence of the text p . Naturally, the prompt p is not fed directly into the model, but instead a high dimensional representation of p is obtained through a decoder. In prevalent models, the prompt p is text in natural language, whereas in the case of MULTIFUSION, p is a sequence of text and images.

Multimodality. Prevalent encoder models like CLIP [34]Ðor multilingual variants like AltCLIP[12]Ðare distinctly unsuited for arbitrarily interleaved sequences of images and text. Unlike MULTIFUSION's multimodal encoder, these architectures rely on two separate encoders for textual and visual inputs, with their respective representations being aligned during training. In contrast to our model, the resulting embeddings only encode a single image or textual description but no interleaved sequences comprised of both modalities. Prior work shows that large language models (LMs) produce meaningful representations for conditioning of generative diffusion models [39, 2]. Additionally, pre-trained capabilities of LMs transfer to downstream tasks even without specific finetuning and beyond the initial modalities [9, 29]. SBERT has demonstrated that pre-trained transformer-based LMs can be used to construct encoders for longer contexts [36], albeit exclusively for natural language sequences. Consequently, MULTIFUSION builts its encoder on a pre-trained LM, achieving context-preserving embeddings for multilingual, multimodal inputs.

Other works have used various forms of image conditioning for diffusion models to enable more expressive prompts. Versatile Diffusion [48] enables the generation of image variations through a unified multimodal framework. Rombach et al. [38] proposed retriever-augmented diffusion models facilitating conditioning on particular visual styles provided via exemplary images. Multiple works have introduced methods to generate high-quality, text-conditioned images from low-resolution inputs such as sketches [45, 51]. Furthermore, textual inversion [19] turns concepts from example images into word embeddings that can subsequently be used during generation. This enables incorporating individual styles or objects into generated images. Lastly, Liu et al. [28] proposed a more general approach for diffusion guidance with image inputs using CLIP. Similarly, Bansal et al. [3] applied arbitrary guidance functions to more capable diffusion models. Such methods facilitate image generation, for example, from segmentation maps, image variations, or style transfer. However, this type of guidance requires a separate model and more complex, hand-crafted pipelines. In contrast, MULTIFUSION introduces a unified, general pipeline for effective direct conditioning through classifier-free guidance, removing the need for separate components. Concurrent with our work, GlueGen also attempt the task of aligning additional input modalities to pre-trained text-to-image models [33]. We encourage the reader to refer to their work for more details.

Multilingualism. Existing LMs pre-trained on multilingual data show impressive multilingual capabilities [40, 26, 13, 49]. Popular text-to-image DM's, however, are usually designed for a single input language [39, 37, 2, 17]. AltDiffusion [12] addressed this issue by proposing a multilingual text encoder (AltCLIP) on which the DM is conditioned instead. AltCLIP embeddings are aligned to the previously used CLIP encoder in a contrastive teacher-student setup using a large multilingual corpus. Subsequently, the cross-attention layers of a pre-trained Stable Diffusion model are finetuned to utilize the AltCLIP encoder instead. The resulting AltDiffusion model can be prompted in 9 different languages. AltDiffusion's image generation is aligned so that the same prompt in different languages results in similar images. With MULTIFUSION we leverage cross-lingual transfer [13, 49] to enable multilingual prompting. More precisely, our generative model obtains multilingual capabilities from the encoder despite being solely trained on English data.

3 MULTIFUSION

By fusing pre-trained model components, MULTIFUSION creates one cohesive system that requires less than 5% of the computational resources needed to train a comparable model from scratch. Our approach involves replacing the encoder of Stable Diffusion (SD) with a more advanced one built on a pre-trained LM. Fusing these components results in a downstream model with the ability to comprehend multilingual, interleaved multimodal inputs. The image generation component inherits this potential despite being trained solely on mono-modal data in a single language. The architecture

of MULTIFUSION is illustrated in Fig. 1. The vast majority of pre-trained weights remain frozen, resulting in an efficient computational process overall. Further information on the training data sizes, parameter counts, and GPU hours for all components is supplied in Tab. 3, along with details on data splits, languages, and modalities in Tab. 4 of the appendix. Subsequently, we outline how to combine and align the involved modules for image generation effectively.

Input encoders. The CLIP encoder [34] used by SD is unsuited for interleaved multimodal inputs as it disregards context and yields disjoint encodings of text and images. Previous work has demonstrated that text encoders based on context-sensitive LMs improve the expressiveness of downstream image generation models [39, 2]. Accordingly, we model the backbone of MULTIFUSION's encoder as an autoregressive transformer [10] using rotary position embeddings [43] trained on a multilingual corpus of various languages (step 1.1 in Fig. 1). We chose an autoregressive decoder model over a bi-directional architecture since decoder models intuitively outperform bi-directional models on relevant tasks. For example, autoregressive models excel at manipulation with natural language (ªSubject X with background Y") (cf. Sec. 4.2) or correct features attributions (ªRed X, Blue Yº) (cf. Sec. 4). Previous research has identified the natural breaking of permutation equivariance as the source of these capabilities [24], compared to bidirectional models relying entirely on positional embeddings. We acknowledge that bi-directional models may outperform autoregressive ones on other embedding tasks [50], but argue that an autoregressive model is better suited for the tasks studied in MULTIFUSION due to the outlined benefits.

Following the methodology proposed by MAGMA [15], we consider an LM with an added image prefix and dedicated adapters to enable multimodal capabilities (step 1.2 in Fig. 1). Adapters are a suitable architectural choice for multimodal prompts since previous research has already performed extensive ablations on adapter architectures and demonstrated their improved understanding of multimodal inputs over other methods [15]. In this architecture, the image prefix maps the image into sequences of token embeddings in a joint multimodal input embedding space. The adapters are added to each attention and feed forward layer of the transformer and are trained autoregressively on a combination of large-scale image-text datasets (cf. App. A), while the parameters of the language model remain frozen [15, 29, 23]. As a result, the LM enables prompting with arbitrarily interleaved sequences of text and image tokens.

Semantic embeddings. In order to use the LM as an encoder for image generation, we extract the model's hidden representations before the language modeling head. While these representations already capture semantic information from the model's pre-training, they need to be optimized and aligned further for usage in arbitrary downstream tasks (step 2.1 in Fig. 1). Initial experiments without additional alignment have shown low rates of convergence (based on visual inspection of generated outputs). Consequently, we chose to produce semantic embeddings guided by the intuition that a focus on the semantics of a text prompt would best capture the information relevant to image generation. Thus simplifying the learning of mapping from embeddings to image outputs, which was confirmed by our initial experimental observations. In conclusion, we deem semantic fine-tuning an essential condition for successfully fusing an image generation model.

We obtained high-quality semantic embeddings through parameter-efficient bias fine-tuning [4]. The tuning follows the supervised contrastive learning objective outlined in S-GPT [31]. The training data consists of natural language tuples of premise and hypothesis, where entailments serve as positive samples and contradictions as negative ones. Importantly, the tuples are bi-lingual such that the premise and hypothesis are each stated in different languages. We observed that including only two languages in this finetuning task is sufficient to achieve multilingual alignment even beyond bilingualism (cf. Sec. 4). Crucially, the semantic bias weights are tuned independently of the multimodal adapters, allowing for modular extensions of the LM with both components.

Bootstrapping Stable Diffusion. The final MULTIFUSION encoder is the result of combining these separately trained modules (cf. Fig. 1). The DM is conditioned on embeddings extracted from the last hidden layer of the transformer. Subsequently, we denote H ( x ) as embedding for input x after a forward pass through MULTIFUSION's multimodal LM encoder. We now align the pre-trained image generation model with MULTIFUSION's encoder (step 2.2 in Fig. 1). Considering the depicted SD architecture, we only need to alter the conditioning portion of the generative model to use our new encoder instead of CLIP. In line with previous research [12], we keep all weights of the DM frozen and only finetune the cross-attention layers of the U-Net. The training objective remains the same as shown in Eq. 1 and 2 with the conditioning c p being the encoding H ( x ) .

We trained the DM only on monolingual (English) and monomodal inputs with x being randomly chosen as either an image or a caption. Nonetheless, the final MULTIFUSION model is capable of interpreting multilingual and arbitrarily interleaved text and image prompts. Our method highlights that the capabilities of strong LMs can be transferred to downstream tasks without the need for dedicated downstream training data. We provide empirical evidence in the following section.

Modality alignment. During initial experiments, we observed that further modifications to the inference mechanics are needed to enable stable multimodal prompting. More specifically, for interleaved text and image prompts MULTIFUSION's encoder represents one image as a sequence of 144 tokens in the input embedding space. In most scenarios, the accompanying text prompt contains fewer tokens, resulting in a disproportional influence of the visual inputs on the generated image. To counteract this phenomenon, we utilize attention manipulation [14] to up-weight the impact of textual tokens with respect to the discrepancy in input length. Representative results showcasing the effect of attention manipulation can be found in Appendix D.

4 Experiments

Next, we present exhaustive evaluations of the multimodal and multilingual capabilities of MULTIFUSION on various benchmarks. To evaluate compositional robustness on models allowing multi-modal inputs, we introduce the new MCC-250 benchmark. Furthermore, we showcase a variety of applications enabled by MULTIFUSION. We provide a detailed experimental protocol, including information on implementation and training setup in App. A.

4.1 Empirical evaluation

Image fidelity & image-text alignment. We start off with a standard evaluation for image generation models to demonstrate the general functionality of MULTIFUSION. We investigate image fidelity using FID-30k scores on the MS COCO validation set [25]. We report the results using textual, multimodal, and image prompts and a comparison against SD v1.5 in Tab. 1. The results show that the image fidelity of MULTIFUSION with textual prompts is competitive with the underlying SD model. Improved FID scores highlight that the capabilities of the original model are preserved in addition to benefiting from new input modalities and languages.

Using multimodal inputs instead of textual prompts results in a substantial improvement of FID scores. This improvement clearly indicates that visual inputs possess a greater capacity to encode comprehensive information about the underlying distributions, surpassing the effectiveness of textual descriptions alone. We acknowledge that using the MS COCO reference images as prompts provides a strong supervision. However, we argue that the above conclusion of image inputs adding additional and more fine-grained information over text prompts alone still holds. Because MULTIFUSION achieves the improved FID scores by generating meaningful variations of the prompt with more aligned details instead of just trivially reproducing the input image (cf. Sec. 4.2 and App. B). Beyond FID, we evaluated text-image alignment as average CLIPScore [21] between COCO captions and generated images. Again, MULTIFUSION achieves scores on par with SD, confirming the preservation of previous capabilities.

English

German

Spanish

French

Italian

品

Compositional robustness. A challenging task many image synthesis models struggle with is image composition. DMs and, in particular, SD often fail to correctly compose the objects and attributes specified in the text prompt [11, 16].

MULTIFUSION, on the other hand, behaves robustly with respect to challenges in compositional generation as shown in Fig. 2. For evaluation, we propose a new benchmark, Multimodal Concept Conjunction 250 (MCC-250) 4 . It builds on a subset of the Concept Conjunction 500 (CC-500) benchmark [16]. CC-500 contains 500 text prompts of the pattern "a red apple and a yellow banana", textually describing two objects with respective attributes. For half of those prompts, we curated a set of images for each object, enabling multimodal prompting, i.e. the textual description is interleaved with a visual reference. For the new MCC-250 benchmark, we present a human evaluation of SD, Composable Diffusion [27] and MULTIFUSION in Tab. 2. Note that all approaches use the same image generation module. For MULTIFUSION we evaluate multimodal prompts (text and image) as well as text-only.

a green apple

and a red car

a blue book

and a red cup

a blue bowl and

a brown horse

.

MultiFusion

Stable Diffusion

Interchanged Attr.

Attr. Leakage

Missing Obj.

On this complex task, MULTIFUSION clearly outperforms both baseline models, almost doubling the portion of images containing both objects with correct colors. Successful compositions once again emphasize the capacity of multimodal prompting in MULTIFUSION. Importantly, the observed performance improvement originates from the multimodality of inputs, as the success rate on text-only inputs remains comparable. Surprisingly, Composable Diffusion performs slightly worse than SD for image composition. The resulting images frequently display strange blends of the two objects rather than capturing them as distinct entities. We provide further details on the user study in App C.

Multilingual alignment. Next, we investigate the multilingual capabilities of MULTIFUSION. Therefore, we evaluated the alignment of directly translated text inputs in prompt embedding space and the generated images. We compare MULTIFUSION with AltDiffusion [12], as it also builds on a frozen SD model. AltDiffusion makes for a great comparison, as the key differences between both

en

de

fr

it

es

Language

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Embedding Similarity

MultiFusion

CLIP

(a) Textual embedding alignment as average cosine similarity between prompt translations. Error bars indicate the standard deviation across paired language comparisons. We provide scores for CLIP as a reference where the horizontal red line indicates the lower bound of average CLIP similarities between uncorrelated english prompts.

en

de

es

fr

it

Language

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Image Similarity

MultiFusion

AltDiff

SD

Figure 4: Comparison of multilingual alignment over DrawBench prompts. MULTIFUSION achieves comparable alignment of the output images although the image generation module was only trained on English data. This can be attributed to the strong alignment of multilingual prompts in MULTIFUSION's embedding space. Similarities are calculated based on paired comparisons between one language and all others. We do not report German results for AltClip/AltDiff nor Chinese for MultiFusion as these languages are not in the respective training data.

models lie in the respective encoders and training data. AltDiffusion uses a large multilingual dataset to finetune the DM, whereas MULTIFUSION's finetuning requires only English training data.

The evaluation is based on a multilingual version of DrawBench [39] 5 . To that end, we translated all prompts into the secondary languages of MULTIFUSION and AltDiffusion: German and Chinese. Additionally, we include the three languages shared by both models: French, Spanish, and Italian. In Fig. 4a, we plot the alignment of multilingual text embeddings over DrawBench. MULTIFUSION's encoder clearly outperforms AltDiffusion's encoder (AltClip) on embedding alignment, scoring 97% cosine similarity on average. The alignment of the generated images is similar for both models, although MULTIFUSION was only finetuned using English data (cf. Fig. 4b). These results highlight that good multilingual embedding alignment enables the transfer of multilingualism to downstream tasks without the need for explicitly multilingual training data.

4.2 Applications

After we empirically demonstrated MULTIFUSION's multimodal, multilingual prompting capabilities, we show how it facilitates diverse interesting use cases. We provide more examples in the Appendix.

Image composition. One of the main strengths of MULTIFUSION's multimodal inputs is image composition. As demonstrated in our empirical evaluation and throughout Figs. 5a and 5b, text and image sequences can be combined arbitrarily and flexibly. Interleaved inputs allow for intuitive combinations of images (cf. Fig. 5b) and serve as a visual reference to enrich text prompts.

Negative prompting. Negative prompting refers to a technique in image generation that aims to suppress certain concepts. To that end, the unconditioned estimate during classifier-free guidance /epsilon1 ( z t ) (cf. Eq. 2) is replaced with an estimate conditioned on a negative prompt /epsilon1 ( z t , c n ) . Guiding away from /epsilon1 ( z t , c n ) results in the concepts described by n being avoided. Consequently, negative prompting works better with more descriptive prompts n . Our previous results demonstrated (cf. Tab. 1) a higher expressiveness of images over text that also translates to negative prompts. Fig 5c shows that textual negative prompts are less effective in removing undesired concepts. However, using image prompts, these concepts can be completely suppressed.

Image variation. MULTIFUSION offers a direct interface for generating image variants. Simply providing a single image as input prompt to the default pipeline produces meaningful variations already. Other models, in contrast, rely on inversion or re-noising of the input image. We depict

Standing in front of a building

+

+

+

Une image photoréaliste

d'une voiture ancienne

Gebäude, aber es ist Winter

und alles ist schneebedeckt

+

+

+

A cartoon of an old car

(a) Multilingual, multimodal prompting. MULTIFUSION prompts seamlessly integrate arbitrary combinations of images and textual prompts in multiple languages.

+

+

This

motorcycle

in front of

a lake

+

This

motorcycle

in a forest

+

(b) Multimodal Image Composition.

Lightbulb

(c) Negative Prompting.

一

examples in Fig. 6a. The generated images include sensible deviations from the input but faithfully reproduce the underlying image contents.

Style modification. Furthermore, MULTIFUSION can easily generate images adhering to any artistic style. The style of images is notoriously hard to describe in image generation prompts and has even led to the development of monetized prompt databases for long, obfuscated prompts. This is mostly due to the magnitude of factors that have to be accounted for, such as the color palette, composition, contrast, etc. Fortunately, all of these aspects can easily be expressed through exemplary (reference) images. Fig. 6b depicts examples where arbitrary styles are applied to various scene descriptions. MULTIFUSION delivers high-quality outputs that show the prompted scene in the desired style.

Multilingualism. Lastly, the strong multilingual embedding alignment makes MULTIFUSION largely invariant to the language of input prompts. Our model fully supports input prompts in five languages: English, German, French, Spanish, and Italian. In line with our results on language alignment, Fig. 3 shows that the same prompt in different languages yields largely similar images. This improves accessibility and expands the group of users, regardless of their native language. Moreover, Fig. 5a demonstrates that languages can even differ within the same input. This further emphasizes MULTIFUSION's expressiveness, allowing users to construct prompts in various ways.

5 Discussion

The capabilities of MULTIFUSION outlined above emphasize the advantages of the model, which we subsequently discuss alongside the remaining limitations and overall societal impact.

5.1 Advantages of MULTIFUSION

Multimodality and multilingualism of inputs offer various benefits compared to prevalent models. The extension of the interface beyond the usual English-only text-to-image applications makes MULTIFUSION more flexible, expressive, and versatile.

Expressiveness. Natural language and images both have benefits and limitations in the concepts they can convey. When used in combination, however, we are able to draw from the advantages of one and the other, thus eliminating the restrictions of either modality. On the one hand, complex objects that should be included can easily be prompted via exemplary images instead of using long, convoluted, and error-prone textual descriptions. Natural language is often unclear and ambiguous and may lack words to describe certain concepts concisely. For the car depicted in Fig. 5a, for example, MULTIFUSION faithfully infers the vehicle's color, make, type, and era and uses that as a reference for image generation. Achieving similar results through textual descriptions alone requires complicated prompt engineering and may be unsuccessful altogether. Additionally, some necessary

国

(a) Image Variations. MULTIFUSION produces non-trivial variations when prompted with a single image.

ancient temple

house at a lake

style

(b) Style transfer from reference images. MULTIFUSION is prompted with a text prompt (left) + ' in the style of <image> '. Generated images are faithful to the targeted scenario and style.

terminology may elude the end user. For example, a non-native speaker may not be aware that the type of building in Fig. 5a can be described as half-timbered . Users may consequently write sub-optimal textual descriptions which can be circumvented using visual prompts. Furthermore, images convey world knowledge that the model does not possess. For example, not all existing artworks or paintings are included in the training data and retained by the model (cf. Fig. 6b). Consequently, using their stylistic characteristics as reference is difficult through text alone. MULTIFUSION easily bridges this knowledge gap by facilitating the inclusion of these images in the prompt. This even enables using individual experiences for reference, such as my car, a sketch I drew, or a building I saw. On the other hand, natural language provides an intuitive interface for abstract concepts and scenarios. For example, the drawing of the car in Fig. 5a is easily turned into a photorealistic version through textual instruction. Consequently, the capabilities of MULTIFUSION result from combining the expressiveness of images and the abstraction of natural language. The inclusion of multiple languages only further increases the model's expressiveness.

Robustness and clarity. Similarly, the availability of multimodal inputs significantly improves the robustness of MULTIFUSION. The combination of visual and natural language inputs allows users to circumvent the limitations of the modalities themselves and those of current DMs. For example, natural languages contain a variety of homographs, i.e., words with the same spelling but different meanings. Inferring the correct one from context is often impossible, whereas images provide clear examples. On the other hand, images alone may contain various sources of noise, such as background objects or stylistic choices that the user might not want to consider for generation. However, when used in combination, one modality is able to make up for the shortcomings of the other.

Furthermore, MULTIFUSION overcomes the limitations of current DMs in compositional generation [11, 16]. As demonstrated on the MCC-250 benchmark, images generated from multimodal prompts with MULTIFUSION are less likely to miss objects specified in the prompt or bind attributes incorrectly. This is due to MULTIFUSION devoting multiple tokens to an object described via an image and using context-aware embeddings. In contrast, text-only generation using models such as Stable Diffusion often leads to regular attribute leakage or interchanged attributes, cf. Fig. 2. In this case, the green color of the apple leaks to the car, or colors are mixed up between objects. We include further examples in App. C showcasing the resilience of MULTIFUSION.

Licensing. Recently, copyright issues and licensing of training images have been heavily discussed topics 6 . Current legal actions 7 may lead to a variety of imagesÐespecially creations from artistsÐ being no longer admissible for use in training. Consequently, applications such as instructing the

model to reference an artist's style will be infeasible even though the end user might have the necessary copyright permissions. With MULTIFUSION, however, a user can still provide reference material not included in the training data simply in the input.

5.2 Limitations

While MULTIFUSION achieves impressive results on various applications, there are some limits to the model's expressiveness. For one, MULTIFUSION always produces meaningful variations when prompted with a single input image. While this is an advantage in some settings, it may be a limitation in others since exactly copying objects from an input is not feasible. This originates from the encoder not being designed to reconstruct images from its representations but to encode relevant characteristics efficiently. Nonetheless, this behavior is generally intended as images are supposed to serve as references to MULTIFUSION instead of performing image editing. Additionally, the quality and composition of an input image have a significant effect on the generated image. We observed that some visual inputs must be carefully selected to achieve the desired outcome. For example, MULTIFUSION may also include unwanted items from the background of an image in the generated output. Furthermore, in some cases, MULTIFUSION has a tendency to copy the input image style even when the prompt's textual portion indicates a different one. Nonetheless, this can be adequately addressed with the proposed attention manipulation method [14], as demonstrated throughout the examples in this work.

5.3 Societal impact

Recent developments in text-to-image models [35, 32, 39, 2] have the potential for a far-reaching impact on society, both positive and negative when deployed in applications such as image generation, image editing, or search engines. Previous research [41, 18] described many potential negative societal implications that may arise due to the careless use of such large-scale generative models. Many of these problems can be attributed to the noisy, large-scale datasets these models rely on. Since recent text-to-image models, such as SD, are trained on web-crawled data containing inappropriate content [42, 6, 5], they are no exception to this issue. Specifically, models relying on the LAION datasets [42] show signs of inappropriate degeneration [41]. Consequently, we assume MULTIFUSION to suffer from similar shortcomings only reinforced by its additional capabilities. Therefore, we will not make the model weights publicly available in their current form.

6 Conclusion

In this work, we introduced MULTIFUSION, a diffusion model utilizing multilingual, arbitrarily interleaved multimodal inputs. These capabilities provide a versatile interface to users in order to better express themselves. We demonstrated that multilingual alignment in the encoder is sufficient to achieve multilingualism in downstream tasks, eliminating the need for dedicated multilingual datasets. Thus, significantly reducing computational requirements. More generally, MULTIFUSION highlights how separate pre-trained components can be interconnected to realize complex models.

Numerous promising avenues for future research emerge from our current work. One intriguing direction involves expanding the scope of MULTIFUSION to facilitate interactive image generation through integration with a chat interface. This novel extension would offer an enhanced user experience, introducing a dynamic interplay between the system and users. Additionally, we envision the potential for recursive prompting of generated images, enabling a progressive refinement of output through incremental instructions. Moreover, the incorporation of an additional decoder, based on the MULTIFUSION encoder, holds promise for facilitating textual generation resulting in multimodal outputs. Lastly, we propose extending the encoder itself to encompass an even broader range of modalities, including audio, video, time series, and various others.

Acknowledgments

We gratefully acknowledge support by the German Center for Artificial Intelligence (DFKI) project ªSAINTº, the Federal Ministry of Education and Research (BMBF) project "AISC ª (GA No. 01IS22091), and the Hessian Ministry for Digital Strategy and Development (HMinD) project ªAI

Innovationlabº (GA No. S-DIW04/0013/003). This work also benefited from the ICT-48 Network of AI Research Excellence Center ªTAILORº (EU Horizon 2020, GA No 952215), the Hessian Ministry of Higher Education, and the Research and the Arts (HMWK) cluster projects ªThe Adaptive Mindº and ªThe Third Wave of AIº, and the HMWK and BMBF ATHENE project ªAVSVº.

References

Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Trans. Graph. , 2023.

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu. eDiff-I: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv:2211.01324 , 2022.

Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , 2022.

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT) , 2021.

Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv:2110.01963 , 2021.

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2015.

Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting. SEGA: instructing diffusion using semantic dimensions. In Proceedings of Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS) , 2023.

Manuel Brack, Patrick Schramowski, Björn Deiseroth, and Kristian Kersting. Illume: Rationalizing vision-language models through human interactions. In Proceedings of the International Conference on Machine Learning (ICML) , 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS) , 2020.

Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attentionbased semantic guidance for text-to-image diffusion models. ACM Trans. Graph. , 2023.

Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, and Ledell Wu. AltCLIP: Altering the language encoder in CLIP for extended language capabilities. In Findings of the Association for Computational Linguistics (ACL) , 2022.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the Annual Meeting of the Association for Computational Linguistics ACL , 2020.

Björn Deiseroth, Mayukh Deb, Samuel Weinbach, Manuel Brack, Patrick Schramowski, and Kristian Kersting. Atman: Understanding transformer predictions through memory efficient attention manipulation. In Proceedings of Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS) , 2023.

Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. MAGMA - multimodal augmentation of generative models through adapter-based finetuning. In Findings of the Association for Computational Linguistics (EMNLP) . Association for Computational Linguistics, 2022.

Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun R. Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In Proceedings of the International Conference on Learning Representations (ICLR) , 2023.

Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE-ViLG 2.0: Improving text-to-image diffusion model with knowledge-enhanced mixture-of-denoising-experts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Felix Friedrich, Patrick Schramowski, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Sasha Luccioni, and Kristian Kersting. Fair diffusion: Instructing text-to-image generation models on fairness. arXiv:2302.10893 , 2023.

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In Proceedings of The Eleventh International Conference on Learning Representations (ICLR) , 2022.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-toprompt image editing with cross attention control. In Proceedings of The Eleventh International Conference on Learning Representations (ICLR) , 2022.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arxiv:2207.12598 , 2022.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the International Conference on Machine Learning (ICML) , 2019.

Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. arXiv:2305.19466 , 2023.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Proceedings of European Conference on Computer Vision (ECCV) , 2014.

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Ves Stoyanov, and Xian Li. Few-shot learning with multilingual generative language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2022.

Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. In Proceedings of European Conference on Computer Vision (ECCV) , 2022.

Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2023, , 2023.

Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Frozen pretrained transformers as universal computation engines. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI Conference on Innovative Applications of Artificial Intelligence (IAAI) ,Symposium on Educational Advances in Artificial Intelligence (EAAI) , 2022.

T. Moore, C. Carling, A. Larsson, and T.C. Merrill. The Limitations of Language . Palgrave Macmillan UK, 1988.

Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search. arXiv:2202.08904 , 2022.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the International Conference on Machine Learning (ICML) , 2021.

Can Qin, Ning Yu, Chen Xing, Shu Zhang, Zeyuan Chen, Stefano Ermon, Yun Fu, Caiming Xiong, and Ran Xu. Gluegen: Plug and play multi-modal encoders for x-to-image generation. In Proceedings of the International Conference of Computer Vision (ICCV) , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In In Proceedings of the International Conference on Machine Learning (ICML) . PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv:2204.06125 , 2022.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , 2019.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Robin Rombach, Andreas Blattmann, and Björn Ommer. Text-guided synthesis of artistic images with retrieval-augmented diffusion models. arXiv:2207.13038 , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-toimage diffusion models with deep language understanding. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. arXiv:2211.05100 , 2022.

Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: an open large-scale dataset for training next generation image-text models. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv:2104.09864 , 2021.

Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan. Unitune: Text-driven image editing by fine tuning an image generation model on a single image. ACM Trans. Graph. , 2022.

Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. In ACM SIGGRAPH 2023 Conference Proceedings , 2022.

Z. Wang, E.P. Simoncelli, and A.C. Bovik. Multiscale structural similarity for image quality assessment. In Proceedings of the Asilomar Conference on Signals, Systems & Computers , 2003.

Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , 2018.

Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. arXiv:2211.08332 , 2022.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , 2021.

Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Retrieval-augmented multimodal language modeling. In International Conference on Machine Learning (ICML) , 2023.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv:2302.05543 , 2023.

standing on top

of an orange.

MF

(multimodal)

MF

(text)

MF

(image)

A dark blue

motorcycle is

parked on the

drive.

Young child

sitting in car

seat with toy

next to door.

An intricately

designed vase

is shown in a

glass case.

Various colored

vases with black

and white

photographs on

wall.

A person image

on a wall with a

skateboard.

There are many

suitcases stacked

on the shelf.

MS COCO

image and caption pair

SD v.

(text)

A Experimental protocol

Subsequently, we provide further details on the datasets (4), architecture and training details (3) of MULTIFUSION. We start from Stable Diffusion v1.4 and discard the CLIP text encoder. MULTIFUSION's encoder is built on a 13B decoder-only transformer similar to GPT-3 [10] but using rotary position embeddings [43]. This LM was pre-trained on a multilingual corpus of roughly 400B tokens comprised of English, German, French, Italian, and Spanish data (cf 4).

We turned the model into a visual language model (VLM) following the method of MAGMA [15]. Therefore, we used a ResNet image encoder, initialized with CLIP weights, and extract the feature grid before the final pooling layers. These 12x12 features are then flattened to a sequence of 144 vectors with subsequent drop regularization and layer normalization resulting in the embedding representation of an image. Consequently, an image is represented by 144 token embeddings which are used as inputs to the transformer LM alongside the text embeddings. For parameter-efficient multimodal training, we added a bottleneck adapter with downsampling factor 8 [23] to each attention and feed-forward layer. Only the weights of these adapters and the image prefix module were optimized during multimodal pre-training. The multimodal components (image prefix and adapters) are trained autoregressively for 15.4 million image-text pairs on a proprietary multimodal dataset (cf 4). Following MAGMA [15], the image tokens are prepended to the text tokens, and the language modeling loss for next token prediction is computed over the text only. We based our multimodal dataset on the findings and extensive ablations of MAGMA [15].

In addition to the tuned multimodal adapters, we apply parameter-efficient bias finetuning [4] of the pre-trained language model to improve the semantic alignment of encoded representations. This turns the hidden representations of the transformer, learned during pre-training, into semantic embeddings usable for arbitrary downstream tasks. Our setup is similar to S-GPT [31], pooling the last hidden layer of the decoder into dense representations, which we optimized for semantic search. We used a

custom version of SNLI [7] and MNLI [47] that extends the original English texts by their machinetranslated German versions, which were generated using the DeepL API. Both datasets contain natural language tuples of premise and hypothesis along a judgment label that can either be entailment, contradiction, or neutral. For example, the hypothesis ' The man is sleeping ' is a contradiction to the premise ' A man inspects the uniform of a figure in some East Asian country. ' We optimize the bias weights of the LM using a contrastive learning objective for 13k steps where entailments serve as positive samples and contradictions as negative ones.

The diffusion model itself is kept frozen with only the cross-attention layers of the U-Net being re-trained to utilize MULTIFUSION's embedding space. As training data, we use LAION aesthetics V.2 5+ 8 , i.e., the subset of LAION 5B with English captions filtered by a predicted aesthetic score > 5 [42]. Additionally, we discard images with a resolution lower than 512 × 512 , resulting in roughly 40 Mcaptioned images. The final model is finetuned for 60k steps with the probability of using an image instead of a caption being 0 . 2 .

We make some modifications to the mechanics at inference to better enable multimodal prompting. For interleaved text and image prompts, MULTIFUSION's encoder represents images as a sequence of 144 token embeddings. This results in a disproportional influence on the generated image compared to significantly shorter text portions. To counteract this phenomenon, we utilize attention manipulation [14] in every attention layer of the transformer encoder. Meaning that every attention score s ij in an attention head is modified by a per-token factor λ i so that ˜ s ij = s ij +log λ i . The authors argue, that λ i > 1 up-weights the i -th token, whereas λ i < 1 down-weights it. In practice, we use the same λ value to up-weight each text token of a prompt.

B MS COCO examples

In Fig.7, we show exemplary images generated from MS COCO prompts in all modalities (cf. Sec. 4). We show that multi-modal prompts enhance image quality as well as convey more information while still producing diverse and meaningful outputs, which are not mere copies of the original input image.

C MCC-250 ± user study

Subsequently, we present further implementation details of the user study conducted on the MCC-250 benchmark (cf. Sec. 4), along with qualitative examples.

C.1 Study Details

For each model, we generated 10 images per prompt. Let's consider the example ' a green bench and a red car ' from the benchmark. The prompt to Stable Diffusion was ' a photorealistic image of a green bench and a red car ', whereas we prompted Composable Diffusion with the two prompts ' a photorealistic image of a green bench ' and ' a photorealistic image of a red car ' which were composed with equal weights. For MULTIFUSION, we interleaved the prompt with image references of each object: ' a photorealistic image of a green bench <image of a green bench> and a red car <image of a red car> '.

We evaluated each of the two objects and the respective attribute separately, with questions being posed in the form:

Is there a green bench in the image?

The four answer options were:

There is no bench

There is a bench, but it is not green

There is a green bench

Cannot tell

Each user was tasked with labeling a batch of 28 image/attribute pairs; 25 out of those were randomly sampled from our generated images. Each batch contained three hand-selected images from the MCC-250 inputs as a sanity check. If users labeled these three images incorrectly, the batch was discarded and added back to the task pool. Each image/object combination was labeled by three different annotators resulting in annotator consensus if at least 2 selected the same label. The results in the main paper (cf. Tab. 2) only include those images with annotator consensus, which are distributed according to Tab. 5.

To conduct our study, we relied on Amazon Mechanical Turk, where we set the following qualification requirements for our users: HIT Approval Rate over 95% and at least 1000 HITs approved. Annotators were fairly compensated according to Amazon MTurk guidelines. Users were paid $0.60 for a batch of 28 images.

C.2 Qualitative Examples

We show further examples of images generated on the MCC-250 benchmark in Fig. 8. The results in Fig. 8a further highlight the robustness of MULTIFUSION for image composition tasks. When prompted with multimodal inputs, the model reliably includes both objects with correct attributes in the generated image. In contrast, Stable Diffusion oftentimes fails to render one of the objects, and Composable Diffusion mostly generates strange mixtures of objects where none of them can be clearly made out.

However, we observed MULTIFUSION to perform consistently poorly on prompts where both objects were animals. As shown in Fig. 8b the generated images contain creatures that exhibit features

Inputs

a photorealistic image

of a green bench

and a

red car

MultiFusion

Stable Diffusion

Composable Diffusion

a photorealistic image

of a green apple

and a

blue boat

a photorealistic

image of a red bird

and a

gold glock

+

+

+

+

+

+

+

(a) Positive examples highlighting the improvement of MULTIFUSION over baseline models.

Inputs a photorealistic image

of a brown bear a photorealistic image

of a yellow dog and a

blue bird and a

brown cat

+

+

+

+

+

+

of both animals. For Stable Diffusion, on the other hand, there is no significant difference in performance on this specific sub-task. In Tab. 6, we show the results of the human evaluation on the animal-only subset. While Stable Diffusion performs comparably to the entire benchmark, both MULTIFUSION and Composable Diffusion portion of correctly generated images drops by over 50%.

D Quantitative and qualitative study of attention manipulation

To evaluate the effect of attention manipulation we compute additional FID scores for the multimodal prompt ablating the attention manipulation weight on the text prompt. In Tab 7 we compare the different FID scores of the multimodal prompts to those of image and text prompts and observe that the higher the weight the closer the FID score is to a text-only prompt. This quantitatively verifies that a higher attention manipulation weight on text prompt increases its influence on the generated image. Further, we qualitativley demonstrate the use of attention manipulation [14] in combination with multi-modal prompts in Fig. 9. We consistently increased the weight of the text prompt and illustrate how this correlates with an increasing influence of the text prompt on the generated images. If the text is weighted equally to the input image, its effect on the generated output remains negligible. Attention manipulation at the input encoder offers an intuitive interface to balance the influence of each input modality. In general, we observed that text prompts should be up-weighted by a factor of 10 -25 to yield the desired results. Additionally, the manipulation behaves quite robustly with no adverse effects on the generated images for higher weights. Further, we show the interpolation

MultiFusion

Stable Diffusion

Composable Diffusion

house at a lake  +

1:1

10:1

20:1

+

This

motorcycle

in a forest

30:1

35:1

+

5:1

15:1

25:1

house at a lake

between two objects - one represented by a text and one by an image prompt - through the use of attention manipulation in Fig. 10. Here we can also observe that text prompts up-weighted by a factor of 10 -25 result either in display of both or a shift of objects.

+

A photorealistic

image of a

green apple

1:1

10:1

20:1

+

A photorealistic

image of a

suitcase

30:1

35:1

+

5:1

15:1

25:1

A photorealistic

image of a

backpack

口

1408

68

8

80

E Qualitative examples of input order

The interleaved inputs are concatenated to one input vector and subsequently fed into the LM, which outputs embeddings for conditioning MultiFusion's image generation U-Net. Changing the order of interleaved inputs will change the embedding produced by the encoder and thus affect the conditioning of the denoising process, leading to a different output. This can be attributed to the autoregressive generation of embeddings with causal attention by the LM. The effect is particularly important for image prompts, where the relationship between multiple concepts is not specified in natural language. Thus, we provide qualitative examples of reversed image prompts in Fig. 11. We can observe the

+

+

+

+

+

+

六

effects of autoregression and causal attention in all three examples of Fig. 11, showcasing that the object or background of the first input image has the highest influence on the output image.

F Further Qualitative Examples

Finally, we provide further qualitative examples, which showcase the multi-modal and multilingual capabilities of MULTIFUSION in Fig. 12. These images further highlight the overall versatility and expressiveness of the model.

+

+

In a photorealistic scene,

high resolution, their face is

+

dieses Bild vom

+

+

Perlenohrgehänge', aber

+

+

Una fotografía

realista de alta

con la siguiente

+

+

+

Cet aliment a

une forme de

cœur

Sur du bois

ayant la

texture suivante

+

+

+

Cartone animato

di un animale

che indossa abiti

A

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MultiFusion, a multilingual multimodal image generation model, which can be effectively trained by fusing existing pre-trained visual models, language models, and stable diffusion models. Using a multilingual autoregressive language model as a bridge, MultiFusion follows MAGMA to enable multimodality by learning adapters. Before connecting the language model with the stable diffusion module, it learns semantic embeddings with a contrastive learning objective in a parameter-efficient setup. Finally, it connects the language model with the diffusion model with monomodal data. i.e., an image or caption.\n\nExperimental results demonstrate that the trained MultiFusion model can generate high-quality images with multimodal interleaved prompts. Besides, with modular design and the fusion of pre-trained models, the training can be quite effective compared to training from scratch. Several analyses such as attention manipulation also provide insights into the multimodal language models.\n\nSTRENGTHS:\n- The paper makes a clever combination of pre-trained models and adapter learning techniques, including 1) MAGMA for cross-modal adaptation/fusing, 2) contrastive learning before fusing LM with the diffusion module, 3) cross-attention learning of SD to align the conditioning with the new embedding space. These operations delicately combine the pre-trained modules into an end-to-end multimodal-text-to-image model.\n\n- Experimental results show that MultiFusion can produce high-quality images conditioned on multimodal and multilingual inputs, with a wide range of applications and use cases. \n\n- The analysis on attention manipulation is quite interesting.\n\nWEAKNESSES:\n- It would be great to improve the presentation of the paper, especially methods and implementation details. Although Figure 1 presents an overview of the architecture, I have to guess some of the implementation details and carefully find clues from a large amount of text. Some suggestions: (1) you could provide some figures to show the details of how the adapters connect the pre-trained models and how you learn them; (2) you could also clarify the training tasks, and data in tables.\n\n- Existing works have explored how to learn adapters to connect pre-trained modules. For example, Flamingo learns gated adapter modules to connect language models with visual models, and generate text conditioned on multimodal inputs. The paper provides a full solution to the problem with careful design but it is kind of an integration of existing adapter methods.\n\nQUESTIONS:\n- Are all the outputs of the language model connected to the diffusion model or only the last token? \n\nE.g., Input: [img_tok1] [img_tok2] [text_tok1] [text_tok2] [text_tok3] -> output vectors: [h1] [h2] [h3] [h4] [h5].\n\nAre all five vectors passed to the diffusion model or only the last one?\n\n- Why is an autoregressive model used as the encoder? I understand the CLIP model has disjoint text and image encodings, but why not use bidirectional models as the multimodal encoder? Any explanations?\n\nLIMITATIONS:\nAs mentioned in the paper, the model always produces variations of input images, which limits its applications in image editing. I think it is worth mentioning this limitation, which provides further understanding of MultiFusion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
In this paper, the authors present a novel approach to expressing complex concepts with arbitrarily interleaved multimodal and multilingual input. Their approach leverages pre-trained models and allows an efficient fusion of different component without training a model from scratch.

STRENGTHS:
1. The paper is well-written and easy to follow
2. The experiments are well-designed and allow one to use existing pre-trained models while reducing the demand to train a system from scratch.
3. The result on various benchmarks are promising and would invite more discussion in this line of work.

WEAKNESSES:
The motivation to attempt such a problem is rather weak. Under what circumstances, would one want to have interleaved multimodal input to generate images? Is it because we want to control the input? If so, why not compare the proposed approach with similar models such as ControlNet and DreamBooth?

QUESTIONS:
See my comments in Weaknesses

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce MultiFusion, a novel approach that enables the expression of complex and nuanced concepts in text-to-image diffusion models (DM) through arbitrarily interleaved inputs of multiple modalities and languages. The “fusion” concept is at the core of the whole work: to fuse modalities together, pre-trained models (a LLM and a stable diffusion backbone) are fused together. Experimental results highlight the efficient transfer of capabilities from individual modules to the downstream image generation module. Notably, MultiFusion empowers the image generation module to effectively utilize multilingual, interleaved multimodal inputs, even when trained solely on monomodal data in a single language. The contributions of this work include the fusion of modalities for image generation, experimental evaluations, and the introduction of a benchmark dataset for further analysis and comparison regarding the multimodal compositionality of the models.\n\nSTRENGTHS:\n1. **Innovative model fusion approach**: The paper introduces an innovative approach by combining a partially frozen multilingual Language Model (LLM) with a stable diffusion backbone. This fusion results in an interesting multilingual and multimodal encoder capable of seamlessly interleaving between input items, treating them as a modality-agnostic sequence.\n2. **Multilingual alignment investigation**: The authors conduct an investigation into the model's multilingual capabilities by translating the prompts from the DrawBench dataset. This exploration demonstrates an understanding of the importance of multilingual alignment. While there is a question regarding the accuracy of the translations, the authors acknowledge the potential benefit of utilizing literal translations in training the multilingual encoder, even though nuances in meaning may not be fully captured. This highlights the authors' attention to addressing the challenges and complexities of multilingual representation.\n3. **Contribution of benchmark dataset**: The authors contribute to advancing research in multimodal compositionality by producing and sharing the MCC-250 dataset. This benchmark dataset, described in detail in the supplementary material, serves as a valuable resource for assessing the compositionality of multimodal inputs, specifically comprising English text and images. The production and release of this dataset demonstrate the authors' dedication to promoting reproducibility, comparison, and further progress in the field of multimodal compositionality.\n\nWEAKNESSES:\n1. **Lack of clear architectural design and novelty**: The paper suffers from a lack of clarity in explaining and justifying its design choices. While references are provided, the underlying motivations and problem-solving aspects of these choices are not adequately explained. While Figure 1 attempts to illustrate the model structure, it is not accompanied by a clear rationale and explanation for the chosen modules and their interactions in the text. Enhancing the clarity of the architectural design would elevate the novelty and originality of the proposed approach. It is suggested to provide a high-level description that guides the reader in understanding the motivations behind specific choices. By focusing on the \"why\" rather than the low-level details, readers can, for example, grasp the purpose of unlocking only the biases in the LLM. Currently (line 130-131), it is unclear if this is a crucial step to obtain good results while keeping a parameter-efficient regime, or if it is marginal in that regard. The supplementary material can be utilized to provide additional low-level details for interested readers.\n2. **Lack of clarity in Figure 4 and semantic search paragraph**: While Figure 4a demonstrates higher similarities of translated prompts in the authors' method compared to competitors, it does not provide insights into the similarities between the reference and other negative samples. This additional information is crucial to establish the range of similarities that can be considered as genuinely low. Furthermore, Figure 4b indicates that the AltDiff competitor generates potentially more consistent images in each language, suggesting that the embedding similarity between references and translations may not be entirely representative. Clarifying these aspects would enhance the understanding of the results and provide a more comprehensive evaluation of the proposed method's performance.\n3. **Missing standard deviation in tables**: Including standard deviation in the results would provide important information about the variability and statistical robustness of the findings. By incorporating this measure, the paper would strengthen the reliability and credibility of the reported results.\n4. **Performance comparison and insights**: In Figure 4b, the AltDiff method demonstrates better performance, raising questions about the potential benefits of adding more languages to the MultiFusion method. While the authors suggest that alignment remains similar despite MultiFusion being fine-tuned using only English data, additional experiments are needed to provide substantial evidence that adding more languages to MultiFusion indeed yields improved results. Further investigation and insights in this area would enhance the value and understanding of the proposed method.\n\nIn offering these critical observations, I would like to emphasize that my intention is not to be harsh, but rather to provide constructive feedback. I acknowledge that explaining such a complex pipeline can be challenging and that significant effort has been invested in this work. However, I strongly believe that there is room for improvement in describing the architectural choices and highlighting the strengths of the paper, and I’ve tried my best to give possible suggestions in this regard. I’m convinced that addressing these aspects would significantly improve the quality and impact of the paper, but I don’t think this is something that could be fixed within the rebuttal period. In any case, I remain open to reconsidering my recommendation if any relevant insights emerge during the discussion.\n\nQUESTIONS:\n**Lack of Chinese column in Figures 4a and 4b**: It is unclear why this column is missing or was included solely for the competitors if the proposed method was not tested on this language.\n\nThere's a typo on line 139: extracted\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed a novel method to build multilingual multimodal generation models that supports prompts composed of interleaved text and image. It combines a strong pre-trained multilingual language model with the image generation model from Stable Diffusion (SD) and achieves alleviated capabilities such as prompting with text and image combined. It also shows that the new model does better in composition generations, as one can provide reference images as part of the prompt. \n\nIn this work, a multilingual language model is trained in the first place (13B encoder-decoder structure model trained on 400B tokens), which itself is a strong multilingual model. It then adds an adapter module to the LLM model to support input in image format, following methods proposed in MAGMA. Finally, it aligns the trained encoder with the diffusion model taken from Stable Diffusion, with 15M text-image pairs. As a result, it can support multilingual, multimodal prompt for image generation, without training on massive text-image pairs dataset.\n\nIn experimentation, it showed that using both text and image as prompts can be beneficial, especially in composition generations. It also shows superior performance to existing multilingual text-image generation model AltDiffusion, possibly due to better alignment in multilingual embeddings. Further, the support of taking image as prompts can enable varies applications such as negative prompting with image, image composition, image variation and style modification.\n\nSTRENGTHS:\n* Novel and efficient method: fusing different pre-trained models works very well which can bootstrapping existing models such as stable diffusion to achieve different input format, avoiding the heavy cost of training model from scratch.\n* Enables prompting using both image and text and generates better images both in terms of metrics such as FID and human evaluation,  comparing to baselines that only takes text prompt.\n* Better results on composition generations from considering reference images in prompts.\n* Well written overall and addressed limitations of the work very well.\n\nWEAKNESSES:\n* Some of the details such as model parameter size, training data source and size are not presented in the main paper (included in appendix), which can be less clear when interpreting results presented in experimental section. It would be better to point those factors out when comparing with baselines in the main paper.\n\nQUESTIONS:\n1. As you mentioned in the limitation section, the model cannot produce images that’s exact or close to the image in prompt. I wonder if there can be simple modifications made that can achieve this?\n2. How is the interleaved data being used? Does it effect the results if you change the order of the interleaved data?\n3. Have you tried other methods in addition to adapter to fuse image and text modality?\n4. Have you done any ablation studies on the semantic embeddings?\n5. On the MSCOCO dataset for generation, where is the image prompt come from (referring results in table 1)?\n6. Have you compared the multilingual LM trained to other ones (mT5 etc.) in multilingual benchmarks?\n\nLIMITATIONS:\nThe authors addressed the limitations relatively well in the paper: \n1. The generated image cannot do copy of exact prompt images;\n2. Sometimes the image prompts need to be carefully chosen and do not always work;\n3. It suffers the same shortcomings (such as inappropriate content) as other generation models trained on very large-scale crawled dataset (LAION).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach for creating a model that can take interleaved sequences of images and multi-lingual text as input, and generate novel images as output, by fusing together pre-trained models: (1) a ResNet image encoder from CLIP (2) an encoder-decoder text Transformer LM and (3) a Stable Diffusion (SD) image decoder. Most of the weights of the models are frozen, with some fine-tuning of adapter layers, the biases of the LM, and the cross-attention layers of the SD U-Net. Multimodal training is done on a combination of large scale image captioning, and VQA datasets, using the standard text-conditioned diffusion objective. The encoder-decoder text model was pre-trained on multi-lingual data, making the resulting image generation model also multi-lingual.\n\nSTRENGTHS:\nS1) The aim of this work, enabling an image generation model to take both images and text (in multiple languages) as input, is exciting and well-motivated by some of the qualitative examples in the paper: multi-modal inputs give complementary info, and multi-lingual text capabilities should broaden model accessibility. \n\nS2) I found the experiments on compositional robustness (MCC-250), with the improved results from the combination of this text encoder and the image inputs, interesting and think it has the potential to be a timely addition to the ongoing conversation about the role of the pre-trained text-encoder in compositional robustness of image generation (but see suggestions on baselines below). Doing a human evaluation user study was also a real strength of these experiments.\n\nS3) The qualitative results were compelling, particularly Figure 5 in the main text and Figure 4 in the appendix.\n\nWEAKNESSES:\nW1) The experimentation was a bit thin. \n- Although there is definitely a shortage of current benchmarks for the new capabilities presented by this model, the contribution of the paper would be stronger if it were able to reappropriate existing benchmarks or create new ones to evaluate some of these capabilities (e.g. negative prompting with images, multimodal image composition). \n- The method has a few steps (e.g. contrastive fine-tuning on a natural language inference dataset; training on a large number of multimodal datasets, both VQA and captions; and using attention manipulation), but I couldn't find any ablations on these components. This, in combination with the lack of details on the [apologies, the rest of this sentence was missing earlier] datasets, makes me worried about whether the overall approach will benefit future work.\n- The quantitative results that are presented here would be more convincing with a few (hopefully) easy-to-run variants of the current settings (another classifier-free-guidance weight; ablating image inputs in the MCC-250 experiment); see questions below. The compositional robustness results are interesting, but giving an image as input is a pretty strong (and potentially unrealistic) source of supervision.\n\n[update after response] : I still feel that point a) above, about capability evaluation, is a weakness, but the author response definitely helped address the other points. Thank you!\n\nW2) The method relies on proprietary datasets and models for the language model (and possibly also for the image datasets, see questions below). I don't think this would be a crucial weakness except that almost no information is given about these datasets and models, even in the appendix. Given that the LM is frozen when doing the multimodal training, and that the capabilities of the fused system (with respect to multi-linguality, and the compositional robustness experiments) seem very likely to me to depend on the properties of this LM, more openness (ideally, using a publicly-released multimodal encoder-decoder transformer, like mT5-XXL, which also has 13B parameters) would really enhance the scientific value of this paper.\n- The encoder is described as a \"13B transformer encoder-decoder similar to GPT-3\", but GPT-3 is a decoder-only model, trained with a language modeling objective.\n- The LM dataset is described only as \"400B tokens of English, German, French, Italian, and Spanish\", and it's unclear whether the multimodal training data includes datasets other than the ones listed in lines 17-18 of the appendix. \n- The German-English versions of SNLI and MNLI used for the semantic embedding objective also seem to be proprietary. \n\nW3) The writing was somewhat unclear. In particular, a lot of details about the model (the pre-trained models used, the training data for the full approach) were unspecified in the main text, although outlined in the appendix (Section A). Some details about the experiments were also unclear, see questions.\n\n[update after response]. The response effectively addressed both W2 and W3 -- thanks!\n\nQUESTIONS:\nQ1) It would be helpful to give any of the following results that are available:\n- Scores for Table 1 with guidance scale 1.0, as SD seems to surpass MF as guidance scale decreases.\n- Results for Table 2 that also remove the image input (i.e. use just the text), to see if there's a benefit in compositional robustness from using the pre-trained encoder-decoder model (this would be very cool if so!).\n- Ablations of any components of the approach, e.g. the semantic embedding fine-tuning, the attention manipulation, or some of the datasets used in multimodal training (e.g., how much value do the VQA datasets add).\n\nQ2) Could more information be given about the encoder-decoder LM, in particular what objective was it pre-trained with (e.g. a denoising objective? prefix-LM)? What is in the training data (e.g. web pages? books? is there paired data across languages, or all monolingual corpora)?\n\nQ3) What data is used to train the multimodal adapters (to input images) and the SD U-net? It wasn't totally clear to me from the appendix. In particular, are other multimodal datasets used beyond the ones listed in lines 17-18 of the appendix? \"such-as\" and \"like\" make it seem like there could be others -- can you say anything about them, if so?\n\nQ4) What image is being provided as input in the Table 1 results? Is it the ground-truth image? If so, could you give some intuition for this? The results are much better with multimodal and image, but ground-truth image would be really strong supervision (and probably also explain why multimodal is worse). \n\nNot crucial for the author response, but it would also really improve the paper to clarify:\n- How are the translations of SNLI/MNLI (line 24 of the appendix) generated?\n- how is the contrastive training on the entailment dataset done?\n- give details of the multimodal training (e.g. learning rates, dataset size, GPU hours) for both the adapters and the SD U-net.\n- I didn't understand Fig 4a, as it seems that similarity needs a translation in two language pairs (e.g. en--de) but the categories here are single language.\n\nLIMITATIONS:\nI felt that the limitations section was pretty solid in qualitatively outlining weaknesses of the approach, although I'd appreciate experiments to quantitatively support the claim that attention manipulation can help prevent image context overriding text context.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}