MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation
Authors: Marco Bellagente
Manuel Brack
Hannah Benita Teufel
Felix Friedrich
Björn Deiseroth
Constantin Eichenberg
Andrew Dai
Robert John Nicholas Baldock
Souradeep Nanda
Koen Oostermeijer
Andres Felipe Cruz-Salinas
Patrick Schramowski
Kristian Kersting
Samuel Weinbach
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion
image generation
multimodal

Abstract: The recent popularity of text-to-image diffusion models (DM) can largely be attributed to the intuitive interface they provide to users. The intended generation can be expressed in natural language, with the model producing faithful interpretations of text prompts. However, expressing complex or nuanced ideas in text alone can be difficult. To ease image generation, we propose MultiFusion that allows one to express complex and nuanced concepts with arbitrarily interleaved inputs of multiple modalities and languages. MultiFusion leverages pre-trained models and aligns them for integration into a cohesive system, thereby avoiding the need for extensive training from scratch. Our experimental results demonstrate the efficient transfer of capabilities from individual modules to the downstream model. Specifically, the fusion of all independent components allows the image generation module to utilize multilingual, interleaved multimodal inputs despite being trained solely on monomodal data in a single language.

MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation
Mareo Bellagente**1 Manuel Brack Hannah Teufel
Felix Friedrich
Björn Deiseroth 13.66 Constantin Eichenberg' Andrew Dai'
Robert J.N. Baldock'
Souradeep * Koen Oostermeijer' Andres Felipe Cruz-Salinas'
Patrick Schramowskf....63 Kristian Kersting2.3.6.7 Samuel Weinbach
Aleehh Alpha, German Research Center for Artificial Intelligence (DFKI), Computer Science Department, TU Darmstadt, "Stability AI, University of Texas, Aessian.AI, 'Centre for Cognitive Science, TU Darmstadt, LLAION
marco. marco. wettrobbellgggeee www.bbeeeeeeeeeemaal...cm brackêes tw-darmstadt. de hannah, www.eeeaeeeeeeeeeaaaeeeeeeeee
Abstract
The recent popularity of text-to-image diffusion models (DM) can largely be at- tributed to the intuitive interface they provide to users. The intended generation can be expressed in natural language, with the model producing faithful interpretations text prompts. However, expressing complex or nuanced ideas in text alone can be difficuul. To case image generation, propose MULTIFUSION that allows to express complex and nuanced concepts with arbitrarily interleaved inputs multiple modalities and languages. MULTIFUSION leverages pre- treitrined models and aligns them for integration into a cohesive system, thereby avoiding the need for extensive training from scratch. Our experimental results demonstrate the efficient transfer of capabilities from individual modules to the downstream model. Specifically, the fusion of all independent components allows the image generation module to utilize multilingual, interleaved multimodal inputs despite being trained solely monomodal data in single language.
Introduction
The recent popularity of text-to-image diffusion models (DM) [39 35 37 can largely be attributed to the intuitive interface they provide users. The intended generation easily be expressed in natural language, with the model producing faithful interpretations of a text prompt. Recent works have demonstrated the output quality be largely dependent the input encoders with more powerful variants yielding more expressive DMs [39 further, vastly enhancing the capabilities of pre-trained DM through the sophisticated integration of dedicated modules. We propose MULTIFUSION which effectively supports arbitrarily interleaved inputs of multiple modalities and languages. Further, transfer these capabilities from underlying language model (LM), eliminating the need for multilingual or multimodal interleaved training data for the generative model. Our approach can utilize readily available datasets and requires less than 5% of the training compute needed to build a comparable DM from scratch.
[2] We take these insights one step
'Work performed while Aleph Alpha Equal contribution
1 Eqqual supervision
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Mutti-Modal Inputs
Embedding Transformation
Image Generator (eg, Stable Diffusion Uneo)
Multi-Modal Generated Images
KV
KV
Image aen beedl
Transformer Blocks
riding on a horse
adapter
biss
Step Pre--rrained Modules:
Step 2- Fusion Tuning:
LLM
Multi Modallty
Semantic Embed 2.2) Cross-Attation
Figure I: MULTIFUSION architecture. We augment a Decoder Language Model (1.1.) with adapters, finetuned for multimodality (1.2) as well biases (2.1), finetuned for semantic search. Next, we condition the diffusion model (1.3) through cross-attention (2.2) on embeddings produced by the LM. During diffusion training. we use either the image or the caption for conditioning, while inference is performed with multimodal input. (Best viewed in color.)
The capabilities of current text-to-in text-to-image DMs are often restricted by the types of inputs they support. MULTIFUSION addresses two of these limitations, facilitating more expressive prompting. Firstly, most of the widely available models are only designed for one language, whereas MULTIFUSION sup- ports five. Contrary to existing multilingual DMs 12], MULTIFUSION requires no multilingual data for DM training and instead only uses readily avai available lable English training data. Secondly, text is inher- ently limited in expressiveness -any concepts are difficult to articulate with words alone [30]. Consequently, number of models have been proposed, allowing visual inputs reference for image generation. Popular examples include image variation [48], adherence to an exemplary style [38),
sketch-guided generation 45]. incorporating personal concepts in the resulting images [19]. All of
these methods go beyond established image editing techniques [I 20 44 8 where certain aspects of the input picture are altered. However, these image-to tonge--toimmgee to-image methods tasks and lack the same level of control and generative versatility for image inputs as for natural language. MULTIFUSION provides powerful and flexible interface for arbitrary combinations of multimodal and multilingual inputs. The extended prompt capabilities result in more expressive model facilitating wide range of complex applications and cases.
limited to specialized
Our main contributions are as follows: (1) We present MULTIFUSION, a multilingual, multimodal diffusion model (2) efficiently bootstrapped using modular encoder based on auto-regressive language model. (3) We highlight the use of attention manipulation 14 for multimodal prompting. and (4) demonstrate the transfer of an LM's multilingual capabilities to downstream tasks, removing the need for multilingual downstream training data. (5) Lastly, we introduce the MCC-250 benchmark for the evaluation of compositionality from multimodal inputs.
2 Background
Text-conditioned diffusion. Diffusion models [39 351 2, 37 iteratively denoise a Gaussian dis- tributed variable to produce samples of a learned data distribution. Intuitively, image generation starts from random noise e, and the model predicts an estimate of this noise ég to be subtracted from the initial values. The denoising process results in high-fidelity image xn without any noise. Since this is an extremely hard problem, multiple steps are applied, each subtracting a small amount (6)) of the predictive noise, approximating e. For text-to-image generation, the model's e-prediction is conditioned on a text prompt and results in an image faithful to that prompt. The training objective of diffusion model de can be written as
Ex,e,.e,t [wy & (axx + wie, cp) - x1
(1)
where (x, c,)) is conditioned on text prompt p. & is drawn from a uniform distribution f. u((0, 1)). e sampled from a Gaussian c <~ (0, I), and we, we, a influence image fidelity. Consequently, the DM is trained to denoise z, + to yield x with the squared error loss. At inference, the DM is sampled using the model's 's prediction of (z: E)). with Eg as described below.
Classifier-free free guidance 22 is conditioning method using a purely generational diffusion model, eliminating the need for an additional pre-trained classifier. The approach randomly drops the text conditioning Cp with fixed probability during training, resulting in a joint model for unconditional

---

and conditional objectives. During inference score estimates for the x x-prediction are adjusted so that:
To(2s, cp) == co(2:) + sg(Co (C,, cp) - co((z))
(2)
with guidance scale sy which is typically chosen as Sy E (0, 20) and cø defining the noise estimate with parameters 6. Intuitively, the unconditioned e-prediction cø (0(z)) is pushed in the direction of the conditioned (z cp) to yield an image faithful to prompt p. Lastly, 8y determines the magnitude of the influence of the text p. Naturally, the prompt p is not fed directly into the model, but instead high dimensional representation of p is obtained through a decoder. In prevalent models, the prompt p is text in natural language, whereas in the case of MULTIFUSION, p is a sequence of text and images.
Multimodality. Prevalent encoder models like CLIP [3 multilingual variants like AICLIPD2--a are distinctly unsuited for arbitrarily interleaved sequences of images and text. Unlike MULTIFUSION's multimodal encoder, these architectures rely on two separate encoders for textual and visual inputs, with their respective representations being aligned during training. In contrast to our model, the resulting embeddings only encode single image or textual description but no interleaved sequences comprised of both modalities. Prior work shows that large language models (L.s)) produce meaningful representations for conditioning of generative diffusion models [39 2 Additionally, pre-trained capabilities of LMs transfer to downstream tasks even without specific finetuning and beyond the initial modalities 9 [9]. SBERT has demonstrated that pre-trained transformer-based LMs can be used to construct encoders for longer contexts [36), albeit exclusively for natural language sequences. Consequently, MULTIFUSION builts its encoder on a pre-trained LM, achieving context-preserving embeddings for multilingual, multimodal inputs.
Other works have used various forms of image conditioning for diffusion models to enable more expressive prompts. Versatile Diffusion [88] enables the generation of image variations through a unified multimodal framework. Rombach et al. 38 proposed retriever-augmented diffusion models facilitating conditioning on particular visual styles provided via exemplary images. Multiple works have introduced methods to generate high-quality, text-conditioned images from low-resolution inputs such as sketches [45] 51.. Furthermore, textual inversion
[9] turns concepts from example images into word embeddings that can subsequently be used during generation. This enables incorporating individual styles objects into generated images. Lastly, Liu et al. (28) proposed more general approach for diffusion guidance with image inputs using CLIP. Similarly, Bansal et al. K applied arbitrary guidance functions to more capable diffusion models. Such methods facilitate image generation, for example, from segmentation maps, image variations, or style transfer. However, this type of guidance requires separate model and more complex, hand-crafie pipelines. In contrast, MULTTIFUSION introduces a unified, general pipeline for effective direct conditioning through classifier-free guidance, removing the need for separate components. Concurrent with our work, GlueGinn also attempt the task of aligning additional input modalities to pre-trained We encourage the reader to refer to their work for more details.
text-to-image models 333
Multilinguallsmm Existing LMS pre-trained on multilingual data show impressive multilingual capabilities [40] 26 13 49]. Popular text-to-image DM's, however, are usually designed for single input language (3 37 [7]. AltDiffusion [12 addressed this issue by proposing a multilingual text encoder (AltCLIP) on which the DM is conditioned instead. AItCIIP embeddings are aligned to the previously used CLIP encoder in a contrastive satheet-tuddnt setup using a large multilingual corpus. Subsequently, the cross-attention layers of pre- pre-trained Stable Diffusion model are finetuned to utilize the A encoder instead. The resulting AltDiffusion model can be prompted in 9 different languages. AltDiffusion's image generation is aligned so that the same prompt in different languages results in similar images. With MULTIFUSION we leverage cross-lingual transfer [13] 49 to enable multilingual prompting. More precisely, our generative model obtains multilingual capabilities from the encoder despite being solely trained on English data.
3 MULTIFUSION
By fusing pre-trained model components, MULTIFUSION creates one cohesive system that requires less than 5% of the computational resources needed to train a comparable model from scratch. Our approach involves replacing the encoder of Stable Diffusion (SD) with a more advanced one built on pre-trained I.M. Fusing these components results in a downstream model with the ability to comprehend multilingual, interleaved multimodal inputs. The image generation component inherits this potential despite being trained solely on mono-modal data in single language. The architecture
3

---

of MULTIFUSION is illustrated in Fig. ! The vast majority of pre- pre-trained weights remain frozen, resulting in an efficient computational process overall. Further information on the training data sizes, parameter counts, and GPU hours for all components is supplied in Tab. 3 along with details on data splits, languages, and modalities in Tab. Do of the appendix. Subsequently, we outline how to combine and align the involved modules for image generation effectively.
Input encoders. The CLIP encoder [34 used by SD is unsuited for interleaved multimodal inputs as it disregards context and yields disjoint encodings of text and images. Previous work has demonstrated that text encoders based on context-sensitive LMs improve the expressiveness of downstream image generation models [39 2). Accordingly, we model the backbone of MULTIFUSION'S encoder as an autoregressive transformer [10] using rotary position embeddings
4S] trained on a multilingual corpus of various languages (step 1.1 in Fig. D. We chose an autoregressive decoder model over a bi-directional architecture since decoder models intuitively outperform bi-directional models on relevant tasks. For example, autoregressive models excel at manipulation with natural language ("Subbect X with background Y"") (cf. S.2. or correct features attributions ("Red X, Blue Y"") (cf. Sec. 4). Previous research has identified the natural breaking of permutation equivariance the source of these capabilities
(24), compared to bidirectional models relying entirely on positional embeddings. We acknowledge that bi-directional models may outperform autoregressive ones on other embedding tasks [50], but argue that an autoregressive model is better suited for the tasks studied in MULTIFUSION due to the outlined benefits.
Following the methodology proposed by MAGMA [T5], we consider an LM with an added image prefix and dedicated adapters to enable multimodal capabilities (step 1.2 in Fig. []. Adapters are suitable architectural choice for multimodal prompts since previous research has already performed extensive ablations on adapter architectures and demonstrated their improved understanding of multimodal inputs over other methods [15. In this architecture, the image prefix maps the image into sequences of token embeddings in joint multimodal input embedding space. The adapters are added to each attention and feed forward layer of the transformer and are trained autoregressively on a combination of large-scale image-text datasets (cf. App. A). while the parameters of the language model remain frozen [15 29 23 As a result, the LM enables prompting with arbitrarily interleaved sequences of text and image tokens.
Semantic embeddings. In order to use the L.M as an encoder for image generation, we extract the model's 's hidden representations before the language modeling head. While these representations already capture semantic information from the model's pre-training. they need to be optimized and aligned further for usage in arbitrary downstream tasks (step 2.1 in Fig. D.. Initial experiments without additional alignment have shown low rates of convergence (based on visual inspection of generated outputs). Consequently, we chose to produce semantic embeddings guided by the intuition that a focus on the semantics of a text prompt would best capture the information relevant to image generation. Thus simplifying the learning of mapping from embeddings image outputs, which was confirmed by our initial experimental observations. In conclusion, we deem semantic fine-tuning an essential condition for successfully fusing an image generation model.
We obtained high-quality semantic embeddings through parameter-efficient bias fine-tuning 4 The tuning follows the supervised contrastive learning objective outlined in S-GPT [11. The training data consists of natural language tuples of premise and hypothesis, where entailments serve as positive samples and contradictions as negative ones. Importantly, the tuples are bi-lingual such that the premise and hypothesis are each stated in different languages. We observed that including only two languages in this finetuning task is sufficient to achieve multilingual alignment even beyond bilingualism (cf. Sec. 4.. Crucially, the semantic bias weights are tuned independently of the multimodal adapters, allowing for modular extensions of the LM with both components.
Bootstrapping Stable Diffusion. The final MULTIFUSION encoder the result of combining these separately trained modules (cf. Fig. TT. The DM is conditioned on embeddings extracted from the last hidden layer of the transformer. Subsequently, we denote (xr) as embedding for input ar after a forward pass through MULTIFUSION 's multimodal LM encoder. We now align the pre-trained image generation model with MULTIFUSION'S 's encoder (step 2.2 in Fig. I.. Considering the depicted SD architecture, we only need to alter the conditioning portion of the generative model to use our new encoder instead of CLIP. In line with previous research 12), we keep all weights of the DM frozen and only finetune the cross-attention layers of the U-Net. The training objective remains the same as shown in Eq. and 2 with the conditioning cp being the encoding (z).

---

Table 1: FID-30k and ClipScores on the MS-COCO validation set for MULTIFUSION (MF) and Stable Diffusion (SD). SD is always prompted with text. All images were generated with image size 256x256 and 100 diffusion steps. Textual prompts consisted of the COCO image caption, multimodal prompts of the caption and COCO reference image, and image prompts of just the image.
FID-30K &
CLIPScore (Text-to-Image) f
Guidance Scale
v1.
MF (Text)
MF (Multimodal)
MF (Image)
SD v1.5
MF
8.0
14.62
14.19
11.50
8,02
0.31
0.30
6.0
12.73
12.15
10.29
7.18
0.31
0.29
4.0
10.19
9.90
8.53
6.03
0.31
0.29
2.0
9.74
12.21
8.61
6.05
0.30
0.28
1.0
26.09
32.81
24.22
18.93
0.27
0.25
We trained the DM only on monolingual (English) and monomodal inputs with x being randomly chosen either image caption. Nonetheless, the final MULTIFUSION model is capable of interpreting multilingual and arbitrarily interleaved text and image prompts. Our method highlights that the capabilities of strong LMs can be transferred downstream tasks without the need for dedicated downstream training data. We provide empirical evidence in the following section.
Modality aligmment. During initial experiments, observed that further modifications to the inference mechanics are needed to enable stable multimodal prompting. More specifically, for interleaved text and image prompts MULTIFUSION' encoder represents one image sequence of 144 tokens in the input embedding space. most scenarios, the accompanying text prompt contains fewer tokens, resulting in disproportional influence of the visual inputs on the generated image. To counteract this phenomenon, we utilize attention manipulation [14] to up-weight the impact of textual tokens with respect the discrepancy in input length. Representative results showcasing the effect of attention manipulation can be found in Appendix
4 Experiments
Next, we present exhaustive evaluations of the multimodal and multilingual capabilities of MULTIFUSION on various benchmarks. To evaluate compositional robustness on models allow- ing multi-modad inputs, we introduce the new MCC-250 benchmark. Furthermore, we showcase variety of applications enabled by MULTIFUSION. We provide detailed experimental protocol, including information on implementation and training setup in App.
4.1 Empirical evaluation
Image fidelity & image-text alignment. We start off with standard evaluation for image generation models to demonstrate the general functionality of MULTIFUSION, We investigate image fidelity using FID-30k scores on the MS COCO validation multimodal, and image prompts and a comparison against SD v1.5 in Tab. [ The results show that the image fidelity of MULTIFUSION with textual prompts is competitive with the underlying SD model. Improved FID scores highlight that the capabilities of the original model are preserved in addition to benefiting from new input modalities and languages.
(25). We report the results using textual,
Using multimodal inputs instead of textual prompts results a substantial improvement of FID scores. This improvement clearly indicates that visual inputs possess a greater capacity to encode comprehen- sive information about the underlying distributions, surpassing the effectiveness of textual descriptions alone. We acknowledge that using the MS COCO reference images as prompts provides strong supervision. However, we argue that the above conclusion of image inputs adding additional and more fine-graet -raiind information over text prompts alone still holds. Because MULTIFUSION achieves the improved FID scores by generating meaningful variations of the prompt with more aligned details instead of just trivially reproducing the input image (cf. Sec. 4.2 and App. B) Beyond FID, we evaluated text-image alignment as average CLIPScore [21] between COCO captions and generated images. Again, MULTIFUSION achieves scores on par with SD, confirming the preservation of previous capabilities.
5

---

Table Fine-grained human evaluation results on MCC-250. SD and Composable Diffusion were prompted with text, whereas we show results for both text and multimodal prompts for MULTIFUSION. Specifically, multimodal prompts contain one visual reference for each object interleaved with the text input. Recall that each prompt is a complex conjunction of two different objects with different colors. MULTIFUSSOO with multimodal inputs strongly outperforms both baseline models.
One obj. w/ correct color
Two obj. w/
Methods
Zero ubj
One obj.
Two obj. 1
correct colors
Stable Diffusion [%]
0.92+ 81
99.07++4.80
90.01 +13.97
44.89 #28.61
29.92++24.76
Composable Diffusion [%]
3,88
96.01 7:772
88.49 12.883
34. 34.72-22.77
25.59118.00
MultiFusion (text) [%]
1.08 4.55
98.91
4:4.57
12.36
36.03
21.66 23
MultiFusion (multimodal) [%]
0.55
99.44 2..85
94.88 11:37
65.06
58.35
English
Gemen
Spanish
French
Italian
Figure 3: Multilingual alignment of images generated by MULTIFUUSON, All images were generated using the same seed and with the respective translation of the prompt 'an image of an astronaut riding horse". (Best viewed in color)
Compositional robustness. A challenging task many image synthesis models struggle with is image composition. DMs and, in particular, SD often fail to correctly compose the objects and attributes specified in the text prompt [1] [6]
Altr. Leakage
Interchanged Attr. Missing Obj.
MULTIFUSION, on the other hand, behaves robustly with respect to challenges in compositional gener- ation as shown in Fig. 2 For evaluation, we pro- pose new benchmark, Multimodal Concept Con- junction 250 (MCC-250) It builds on a subset of the Concept Conjunction 500 (CC-500) benchmark [6]. CC-500 contains 500 text prompts of the pat- tern "a red apple and yellow banana", textually describing two objects with respective attributes. For half of those prompts, for each object, enabling multimodal prompting, i.e. the textual description interleaved with a visual reference. For the new MCC-250 benchmark, we present human evaluation of SD, Composable Dif- fusion [27 and MULTIFUSION in Tab. Note that all approaches use the same image generation module. For MULTIFUSION we evaluate multimodal prompts (text and image) as well as text-only.
Stable Diffusion MuliFussoon
curated
of images
green spple and red car
blue book and red cup
blue bowl and brown horse
multimodality of MULTIFUSION proves more robust for image compositions. SD is prompted in text with 'A photorealistic image of (Descr.'" MULTIFUSION prompts contain interleaved visual references. (Best viewed in color.)
Figure 2:
The
On this complex task, MULTIFUSSON clearly outperforms both baseline models, almost doubling the portion of images containing both objects with correct colors. Successful compositions once again emphasize the capacity of multimodal prompting in MULTIFUSION, Importantly, the observed performance improvement originates from the multimodality of inputs, as the success rate on text-only inputs remains comparable. Surprisingly. Composable Diffusion performs slightly worse than SD for image composition. The resulting images frequently display strange blends of the two objects rather than capturing them as distinct entities. We provide further details on the user study in App
Multilingual alignment. Next, we investigate the multilingual capabilities of MULTIFUSION. Therefore, we evaluated the alignment of directly translated text inputs in prompt embedding space
and the generated images. We compare MULTIFUSION with AltDiffusion [1]. as it also builds frozen SD model. AltDiffusion makes for a great comparison, as the key differences between both
We make the MCC-250 henchmark available httpa:/ //wggg/ ing

---

1.0
MultiFusion
Image Simmaaty
SD
60.6
0.6
Enbeading
0.4
MaltiFusion NNCLIP
002
0.0
0.0
d2
2h
de 2h
Language
Language
(a) Textual embedding alignment average cosine similarity between prompt translations. Error bars in- dicate the standard deviation across paired language comparisons. We provide scores for CLIP as refer- ence where the horizontal red line indicates the lower bound of average CLIP similarities between uncorre- lated english prompts.
(b) Alignment of generated images using average malized multi-scale structural similarity scores [46] over generated images per prompt language. Error bars indicate the standard deviation across lan- guages. SD is included for reference.
Figure 4: Comparison of multilingual alignment over DrawBench prompts. MULTIFUSION achieves comparable alignment of the output images although the image generation module was only trained on English data. This can be attributed to the strong alignment of multilingual prompts in MUSTIIUSSOO'S embedding space. Similarities are calculated based on paired comparisons be- tween one language and all others. We do not report German results for AliClip/AldDiff nor Chinese for MultiFusion as these languages are not in the respective training data.
models lie in the respective encoders and training data. AltDiffusion large multilingual dataset to finetune the DM, whereas MUL.IIFUSION' finetuning requires only English training data.
The evaluation is based on a multilingual version of DrawBench 339 To that end, we translated all prompts into the secondary languages of MULTIFUSION and AltDiffusion: German and Chinese. Additionally, we include the three languages shared by both models: French, Spanish, and Italian. In Fig. 4a we plot the alignment multilingual text embeddings over DrawBBnchh MULTIFUSION'S encoder clearly outperforms AltDiffusion's encoder (AltClip) on embedding alignment, scoring 97% cosine similarity on average. The alignment of the generated images is similar for both models, although MULTIFUSION that good multilingual embedding alignment enables the transfer of multilingualism to downstream tasks without the need for explicitly multilingual training data.
only finetuned using English data (ef. Fig. 4b). These results highlight
4.2 Applications
After we empirically demonstrated MULTIFUSION'S 's multimodal, multilingual prompting capabilities, we show how it facilitates diverse interesting cases. We provide more examples in the Appendix.
Image composition. One of the main strengths of MULTIFUSION's multimodal inputs is image composition. As demonstrated in our empirical evaluation and throughout Figs. (a and[5) text and image sequences can be combined arbitrarily and flexibly. Interleaved inputs allow for intuitive combinations of images (cf. Fig. 5b) and serve visual reference enrich text prompts.
Negative prompting. Negative prompting refers technique in image generation that aims to suppress certain concepts. To that end, the unconditioned estimate during classifier-free guidance e e(z;) (cf. Eq. 2 replaced with an estimate conditioned on a negative prompt E(2,, ca). Guiding away from E(2,, C,) results in the concepts described by n being avoided. Consequently, negative prompting works better with more descriptive prompts n. Our previous results demonstrated (cf. Tab. ( a higher expressiveness of images over text that also translates to negative prompts. Fig shows that textual negative prompts are less effective in removing undesired concepts. However, using image prompts, these concepts can be completely suppressed.
Image variation. MULTIFUSION offers direct interface for generating image variants. Simply providing a single image as input prompt to the default pipeline produces meaningful variations already. Other models, in contrast, rely on inversion or re-noising of the input image. We depict
'e did not include prompts from the categories 'misspellings' and 'rare words'.
7

---

Lie image photoréaliste ancienne
+
+ Standing front a building +
im Histergrund dieses Geblude, aber Winter schaechedeeck
curtoon old
(a) Multilingual, multimodal prompting. MULTIFFSIOO prompts seamlessly integrate arbitrary combinations of images and textual prompts in multiple languages.
This
mutorgycle
+
Lightbulb
front
lake
This
motorcycle +
forest
(b) Multimodal Image Composition.
(c) Negative Prompting.
Figure 5: Applications of MULTIFUSION highlighting the versatility and expressiveness of multilin-
gual, multimodal prompts. (Best viewed in color)
examples in Fig. 6a The generated images include sensible deviations from the input but faithfully reproduce the underlying image contents.
Style modification. Furthermore, MULTIFUSION can easily generate images adhering to any artistic style. The style of images notoriously hard describe in image generation prompts and has even led to the development of monetized prompt databases for long, obfuscated prompts. This is mostly due to the magnitude of factors that have to accounted for, such as the color palette, composition, contrast, etc. Fortunately, all of these aspects can easily be expressed through exemplary (reference) images. Fig. 6b) depicts examples where arbitrary styles are applied to various scene descriptions. MULTIFUSION delivers high-quality outputs that show the prompted scene in the desired style.
Multilingualism. Lastly, the strong multilingual embedding alignment makes MULTIFUSION largely invariant to the language of input prompts. Our model fully supports input prompts in five languages: English, German, French, Spanish, and Italian. In line with our results on language alignment, Fig. shows that the same prompt in different languages yields largely similar images. This improves accessibility and expands the group of users, regardless of their native language. Moreover, Fig. 5a demonstratess that languages can even differ within the same input. This further emphasizes MULTIFUSION'S expressiveness, allowing users construct prompts in various ways.
5
Discussion
The capabilities of MULTIFUSION outlined above emphasize the advantages of the model, which we subsequently discuss alongside the remaining limitations and overall societal impact.
5.1
Advantages of MULTIFUSION
Multimodality and multilingualism of inputs offer various benefits compared to prevalent models. The extension of the interface beyond the usual English-only text-to-image applications makes MULTIFUSION more flcxible, expressive, and versatile.
Expressiveness. Natural language and images both have benefits and limitations in the concepts they can convey. When used combination, however, we are able to draw from the advantages of one and the other, thus eliminating the restrictions of either modality. On the one hand, complex objects that should be included can easily prompted via exemplary images instead of using long, convoluted, and error-prone textual descriptions. Natural language is often unclear and ambiguous and may lack words to describe certain concepts concisely. For the car depicted in Fig. 5a for example, MULTIFUSION faithfully infers the vehicle's color, make, type, and era and uses that as reference for image generation. Achieving similar results through textual descriptions alone requires complicated prompt engineering and may be unsuccessful altogether. Additionally, some necessary

---

(a) Image Variations, MULTIFUSION produces non-trivial variations when prompted with single image.
style
besave at take
dentt temple
(b) Style transfer from reference images. MULTIFUSION prompted with a text prompt (left) + 'in the style of <image>'. Generated images are faithful to the targeted scenario and style.
Figure 6: Further applications of MULTIFUSSIO,, (Best viewed in color)
terminology may elude the end user. For example. non-native speaker may not be aware that the type of building in Fig. [Sacan be described as half-timbered Users may consequently write sub-optimal textual descriptions which can be circumvented using visual prompts. Furthermore, images convey world knowledge that the model does not possess. For example, not all existing artworks or paintings included in the training data and retained by the model (cf. Fig. [bb). Consequently, using their stylistic characteristics as reference is difficult through text alone. MULTIFUSION easily bridges this knowledge gap by facilitating the inclusion of these images in the prompt. This even enables using individual experiences for reference, such as my car, a sketch drew, or a building I saw. On intuitive interface for abstract concepts and scenarios.
the other hand, natural language provides For example, the drawing of the car in Fig. [a is easily turned into a photorealistic version through textual instruction. Consequently, the capabilities of MULTIFUSION result from combining the expressiveness of images and the abstraction of natural language. The inclusion of multiple languages only further increases the model's 's expressiveness.
Robustness and clarity. Similarly, the availability of multimodal inputs significantly improves the robustness of MULTIFUSION. The combination of visual and natural language inputs allows users to circumvent the limitations of the modalities themselves and those of current DMs. For example, natural languages contain variety of homographs, i.e., words with the same spelling but different meanings. Inferring the correct one from context is often impossible, whereas images provide clear examples. On the other hand, images alone may contain various sources of noise, such as background objects or stylistic choices that the user might not want to consider for generation. However, when used in combination, one modality is able to make up for the shortcomings of the other.
Furthermore, MULTIFUSION overcomes the limitations of current DMs in compositional generation [6] As demonstrated the MCC-250 benchmark, images generated from multimodal prompts with MULTIFUSION are less likely to miss objects specified in the prompt or bind attributes incorrectly. This is due MULTIFUSION devoting multiple tokens to object described via image and using context-aware embeddings. In contrast, text-only generation using models such as Stable Diffusion often leads to regular attribute leakage interchanged attributes, cf. Fig. In this case, the green color of the apple leaks to the car, or colors are mixed up between objects. We include further examples App. [showassing the resilience of MULTIFUSION.
Licensing, Recently, copyright issues and licensing of training images have been heavily discussed topic Current legal actions") may lead variety of images especially creations from artists- being no longer admissible for use in training. Consequently, applications such as instructing the
bttpps /www. www..copwrroge..oooowoo copyright, www/ ingingg- e1lsras/ wttp:///wwoooooooooooeeeoee wttpeeseaaeeeeeeeteeeeteeeeeeeee entrance-tteeeeee caricco-tt-pasccee parpaaaaa..pp. pác
Sttps:/ 1/wws////w..../ goviafe.
2-202--03-- 3-03-11
05321.p1.
htttpss//stal ffusionn iggtt

---

model to reference an artist's style will be infeasible even though the end user might have the necessary copyright permissions. With MULTIFUSION, however, user can still provide reference material not included in the training data simply in the input.
5.2 Limitations
While MULTIFUSION achieves impressive results on various applications, there are some limits to the model's 's expressiveness. For one, MULTIFUSION always produces meaningful variations when prompted with a single input image. While this is an advantage in some settings, it may be a limitation in others since exactly copying objects from input not feasible. This originates from the encoder not being designed to reconstruct images from its representations but to encode relevant characteristics efficiently. Nonetheless, this behavior is generally intended images are supposed to serve as references to MULTIFUSION instead of performing image editing. Additionally, the quality and composition of an input image have significant effect the generated image. We observed that some visual inputs must be carefully selected to achieve the desired outcome. For example, MULTIFUSION may also include unwanted items from the background of an image in the generated output. Furthermore, in some cases, MULTIFUSION has tendency to copy the input image style even when the prompt's 's textual portion indicates a different one. Nonetheless, this can be adequately addressed with the proposed attention manipulation method [14], as demonstrated throughout the examples in this work.
5.3 Societal impact
have the potential for far-reaching impact on society, both positive and negative when deployed in applications such as image generation, image editing, search engines. Previous research [41] 18 described many potential negative societal implications that may arise due to the careless use of such large-scale generative models. Many of these problems be attributed to the noisy, large-scale datasets these models rely Since recent text-to-image models, such as SD, are trained on web-crawled data containing inappropriate content they are no exception this issue. Specifically, models relying the LAION datasets show signs of inappropriate degeneration [4T]. Consequently, we assume MULTIFUSION to suffer from similar shortcomings only reinforced by its additional capabilities. Therefore, we will not make the model weights publicly available in their current form.
Recent developments in text-to-image models 32 32 39
42
42
6
Conclusion
In this work, we introduced MULTIFUSION, diffusion model utilizing multilingual, arbitrarily interleaved multimodal inputs. These capabilities provide a versatile interface to users in order to better express themselves. We demonstrated that multilingual alignment in the encoder is sufficient to achieve multilingualism downstream tasks, eliminating the need for dedicated multilingual datasets. Thus, significantly reducing computational requirements. More generally, MULTIFUSION highlights how separate pre-trained components can be interconnected realize complex models.
Numerous promising avenues for future research emerge from our current work. One intriguing direction involves expanding the scope of MULTIFUSION to facilitate interactive image generation through integration with a chat interface. This novel extension would offer an enhanced user experience, introducing a dynamic interplay between the system and users. Additionally, we envision the potential for recursive prompting of generated images, enabling a progressive refinement of output through incremental instructions. Moreover, the incorporation of an additional decoder, based on the MULTIFUSION encoder, holds promise for facilitating textual generation resulting in multimodal outputs. Lastly, we propose extending the encoder itself encompass an even broader range of modalities, including audio, video, time series, and various others.
Acknowledgments
We gratefully acknowledge support by the German Center for Artificial Intelligence (DFKI) project "SAINT", the Federal Ministry of Education and Research (BMBF) project "AISC (GA No. 011S22091), and the Hessian Ministry for Digital Strategy and Development (HMinD) project "AI
10

---

Innovationlab" (GA No. S-DIW04/0013/0033) This work also benefited from the ICTT-88 Network of AI Research Excellence Center "TAILOR" (EU Horizon 2020, GA No 952215), the Hessian Ministry of Higher Education, and the Research and the Arts (HMWK) cluster projects "The Adaptive Mind" and "The Third Wave of AI", and the HMWK and BMBF ATHENE project "AVSV".
References
[1] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Trans. Graph., 2023.
[2] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yo Yu Liu. eDiff-I: Text-lo-image diffusion models with an ensemble of expert denoisers, arXiv:2211.012 2022.
[3] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild. Soumyadip Sengupta, Micah Goldblum, Jonas Geiping. and Tom Goldstein. Universal guidance for diffusion models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[4] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple paramcter-efficient fine-tuning for transformer-based masked segguage-models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2022.
[5] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models too hig? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAceTT), 2021.
[6] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe, Multimodal datasets: misogyny, porno- raphy, and malignant stereotypes. 10.01:2100.0193,, 2021.
[7] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. large annotated corpus for learning natural language inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015.
[8] Manuel Brack, Felix Friedrich. Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting, SEGA: instructing diffusson using semantic dimensions. Proceedings of Advances Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
[9] Manuel Brack, Patrick Schramowski, Bjöm Deiseroth, and Kristian Kersting, Illume: Rationalizing Dision-language models through human interactions. Proceedings of the International Conference on Machine Learning (ICML), 2023.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of Advunces in Neurul Information Processing Systems: Annual Conference Neural Information Processing Systems (NeurIPS), 2020.
[11] Hila Chefer, Yuval Alaluf, Yacl Vinker, Lior Wolf, and Daniel Cohen-Or. Attend- exxextte Attention- based semantic guidance for text-to-image diffusion models. ACM Trans. Graph., 2023.
[12] Zhongzhi Chen, Guang Liu, Bo- Wen Zhang, Fulong Ye, Qinghong Yang, and Ledell Wu. AICCLIP: Altering the language encoder in CLIP for extended language capahilities. In Findings of the Association for Computational Linguistics (ACL), 2022.
[13] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning scale. In Proceedings of the Annual Meeting of the Association for Compudational Linguistics ACL, 2020.
[14] Björn Deiseroth, Mayukh Deb, Samuel Weinbach, Manuel Brack, Patrick Schramowski, and Kristian Kersting. Atman: Understanding transformer predictions through memory efficient attention manipulation. In Proceedings of Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.

---

[15] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. MAGMA multimodal augmentation of generative models through adapter-based finetuning. In Findings of the Association for Computational Linguistics (EMNLP). Association for Computational Linguistics, 2022.
[16] Weixi Feng, Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun R. Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free free structured diffussin guidance for compositional text-to-image synthesis. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[17] Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang, ERNIE-ViLO 2.0: Improving text-to-image diffusion model with knowledge-enhance the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
mixture-of-dente de--ddoinn-expeerrtt In Proceedings of
[18] Felix Friedrich, Patrick Schramowski, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Sasha Luccioni, and Kristian Kersting. Fair diffusion: Instructing text-to-image generation models on fairness. arXiv:2302. 10893, 2023.
[19] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or, An image is worth one word: Personalizing text-to-image generation using textual inversion. In Proceedings of The Eleventh International Conference on Learning Representations (ICLR), 2022.
[20] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. In Proceedings of The Eleventh International Conference on Learning Representations (ICLR), 2022.
[21] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2021.
[22] Jonathan Ho and Tim Salimans. Classifier free diffusion guidance. arxiv:2207.12599 2022.
[23] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Paramette-efficieen transfer learning for NLP. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
[24] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das. and Siva Reddy The impact of positional encoding on length generalization in transformers. arXiv:2305. 19466, 2023.
[25] Tsung-Y Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Proceedings of European Conference Computer Vision (ECCV), 2014.
[26] Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian OoHoo,, Jeff Wang, Luke Zettlemoyer. Zornitsa Kozareva, Mona Diab, Ves Stoyanov, and Xian Li. Few-shot learning with multilingual generative language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
[27] Nan Liu, Shuang Li, Yilun Du Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. In Proceedings of European Conference on Computer Vision (ECCV), 2022.
[28] Xihui Liu, Dong Huk Park, Samanch Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
2023.. 2023.
[29] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Frozen pretrained transformers as universal computation engines. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI Conference Innovative Applications of Artificial Intelligence (IAAI) ,Symposium Educational Advances in Artificial Intelligence (EAAI), 2022.
[30] T. Moore, C. Carling, A. Larsson, and T.C. Merrill, The Limitations of Language. Palgrave Macmillan UK, 1988.
[31] Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search. arXiv:2202. 08904, 2022.
12

---

[32] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the International Conference Machine Learning (ICML), 2021.
[33] Can Qin, Ning Yu, Chen Xing. Shu Zhang. Zeyuan Chen, Stefano Ermon, Yun Fu, Caiming Xiong, and Ran Xu. Gluegen: Plug and play multi-modal encoders for x-to to-immage generation. In Proceedings of the International Conference of Computer Vision (ICCV), 2023.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2021.
[35] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv:2204.06/22, 2022.
[36] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),, 2019.
[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffussin models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[38] Robin Rombach, Andreas Blattmann, and Bjöm Ommer. Text-guided synthesis of artistic images with retrieval-augmeette diffusion models. 2cXiv:2207.13338 2022.
[39] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan. S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to- image diffusion models with deep language understanding. In Proceedings the Conference on Neural Information Processing Systems (NeurIPS), 2022.
[40] Teven Le Seao, Angela Fan, Christopher Akiki, Ellic Pavlick, Suzana Ilce, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang. Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz. Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy. Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue. Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Ade- lani, and al. BLOOM: A 776b-parameter open- multilingual language model. 20..:020...05100 2022.
[41] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffuusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[42] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: an open large-scale dataset for training next generation image-text models. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2022.
[43] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. 2rXiv:2104.09864, 2021.
[44] Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan. Unitune: Text-driven image editing by fine tuning image generation model single image. ACM Trans. Graph., 2022.
[45] Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch guided text-to-imane diffusion models. In ACM SIGGRAPH 2023 Conference Proceedings, 2022.
[46] Z. Wang, E.P. Simoncelli, and A.C. Bovik. Multiscale structural similarity for image quality assessment.
In Proceedings of the Asilomar Conference on Signals, Systems & Computers, 2003.
[47] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HI.7), 2018
13

---

[48] Xingqian Xu. Zhangyang Wang. Eric Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all diffusion model. urXiv:2211.08332, 2022.
[49] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT3: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT),202., 2021.
[50] Michihiro Yasunaga, Armen Aghajanyan. Weijia Shi, Rich James, Jure Leskovee, Perey Liang, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Retrieval -uumeetee multimodal language modeling. In International Conference on Machine Learning (ICML), 2023.
[51] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. mmxiv22302.05543, 2023.
14

---

dark motoreyele is parked the drive.
child sitting car seat orange.
Brown butterfly standing on top
large elephant standing in an on a wall with skateboard.
image
Vartous colored vases with black and white photographs on wall.
intricately designed vase thown glass case.
There many suitenses stacked the shelf.
open field with wild brush.
MS COCO caption
door.
ME
MI (mmage
MF (multinoda))
SD %. (text)
Figure 7: Qualtiative examples of images generated from MS COCO images and captions. We compare outputs from MUL.IFFUSION (MF), prompted with different modalities text, image, and multimodal as well as outputs from Stable Diffusion, which was prompted with text only. We use guidance scale 4.0 for both models.
A Experimental protocol
Subsequently, we provide further details on the datasets (4 architecture and training details (3) of MULTIFUSION. We start from Stable Diffusion v1.4 and discard the CLIP text encoder. MULTIFUSION'S encoder is built on a 13B decoder-only transformer similar to GPT-3 [10] but using rotary position embeddings 43). This LM was pre-trained on multilingual corpus of roughly 400B tokens comprised of English, German, French, Italian, and Spanish data (c
We turned the model into a visual language model (VLM) following the method of MAGMA (15). Therefore, we used a ResNet image encoder, initialized with CLIP weights, and extract the feature grid before the final pooling layers. These 12x 12 features are then flattened vectors with subsequent drop regularization and layer normalization resulting in the embedding representation of an image. Consequently. an image is represented by 144 token embeddings which used as inputs to the transformer LM alongside the text embeddings. For parameter-efficient multimodal training, we added bottleneck adapter with downsampling factor 8 23 to each attention and feed-forward layer. Only the weights of these adapters and the image prefix module were optimized during multimodal pre- troinrniiin.... The multimodal components (image prefix and adapters) trained autoregressively for 15.4 million image-tex pairs proprietary multimodal dataset (cf Following MAGMA (15), the image tokens are prepended to the text tokens, and the language modeling loss for next token prediction is computed over the text only. We based our multimodal dataset on the findings and extensive ablations of MAGMA 15).
sequence of 144
In addition to the tuned multimodal adapters, we apply parameter-efficien bias finetuning [ of the pre-trained language model to improve the semantic alignment encoded representations. This turns the hidden representations of the transformer, learned during pre-training. into semantic embeddings usable for arbitrary downstream tasks. Our setup similar to S-GPT 3I), pooling the last hidden layer of the decoder into dense representations, which we optimized for semantic search. We used
15

---

Table 3: Training details of MF's individual modules.
Module
Parameters
Trained parameters (MF)
Training data size
GPU hours
LM
13B
400B tokens
85K
Semantic embeddings
50M tokens
6.9K
Multimodal components
2B 152M (15%)
15 4M.4M image-texi pairs
9.5K
SD (v1.4)
IB
40M images
4.3K
Table 4: Training datasets of MF's individual modules. Only the semantic bias training uses paired multi lingi-liigual data. All other training data soureed from monolingual corpora.
Module
Dataset
Percentage
Languages
Modalities
web crawl
71%
books
20%
english, german, italian, spanish, french
political and legal sources
5%
LM
wikipedia
2%
2%
other
1%
MNLI
72%
Semantic embeddings
english, german
SNLI
28%
image captioning tasks
91% %
Multimodal components
Wikipedia image-text
8.5%
english
text, image
visual question question-answwente answering tasks
0.5%
SD (v1.4)
LATON aesthetics
100%
english
text, image
custom version of SNLI 7 and MNLI 47 that extends the original English texts by their machine- translated German versions, which were generated using the DeepL API. Both datasets contain natural language tuples of premise and hypothesis along judgment label that can either be entailment, contradiction, or neutral. For example, the hypothesis 'The man is sleeping' a contradiction the premise 'A man inspects the uniform of figure in some East Asian country." We optimize the bias weights of the LM using a contrastive learning objective for 13k steps where entailments serve as positive samples and contradictions negative ones.
The diffusion model itself is kept frozen with only the cross-attention layers of the U-Net being re-trained to utilize MULTIFUSION ' embedding space. As training data, we use LAION aesthetics V.2 5+ i.e., the subset of LAION 5B with English captions filtered by predicted aesthetic score > 5 [2). Additionally, we discard images with resolution lower than 512 512, resulting in roughly 40M captioned images. The final model is finetuned for 60k steps with the probability of using image instead of caption being 0.2.
We make modifications the mechanics inference better enable multimodal prompting. For interleaved text and image prompts, MULTIFUSION'S encoder represents images as a sequence of 144 token embeddings. This results disproportional influence on the generated image compared significantly shorter text portions. To counteract this phenomenon, we utilize attention manipulation 14 in every attention layer of the transformer encoder. Meaning that every attention score Sij in an attention head is modified by per-token factor X; so that 6;y = Siy + log A, The authors argue, that A up- weights the i-th token, whereas A down-weights In practice, we use the same value to up- up-weight each text token of prompt.
B MS COCO examples
In Fig we show exemplary images generated from MS COCO prompts in all modalities (cf. Sec. 4p. We show that multi-modal prompts enhance image quality as well as convey more information while still producing diverse and meaningful outputs, which are not mere copies of the original input image.
16

---

C MCC-250 - user study
Subsequently, we present further implementation details of the user study conducted on the MCC-250 benchmark (cf. Sec. 4) along with qualitative examples.
C.1 Study Details
For each model, we generated 10 images per prompt. Let's consider the example 'a green bench and a red car' from the benchmark. The prompt to Stable Diffusion was 'a photorealistic image of a green bench and red car', whereas we prompted Composable Diffusion with the two prompts a photorealistic image of green bench' and 'a photorealistic image of a red car' which were composed with equal weights. For MULTIFUSION, we interleaved the prompt with image references of each object: 'a photorealistic image of green bench <image of a green bench> and red car <image of a red car>'.
We evaluated each of the two objects and the respective attribute separately, with questions being posed in the form:
Is there a green bench in the image?
The four answer options were:
There is no bench
There is a bench, but is not green
There is a green bench
Cannot tell
Each user was tasked with labeling batch of 28 image/attribute pairs; 25 out of those randomly sampled from our generated images. Each batch contained three hand-selected images from the MCC-250 inputs as a sanity check. If users labeled these three images incorrectly, the batch was discarded and added back to the task pool. Each image/object combination was labeled by three different annotators resulting annotator consensus if least selected the same label. The results in the main paper (cf. Tab. 2 only include those images with annotator consensus, which are distributed according to Tab.
Table 5: Annotator consensus on the selected label per model for the MCC-250 user study.
Model
Annotator Consensus in [%]
Stable Diffusion
98.31
Composable Diffusion
93.85
MULTIFUSION (text)
98.71
MULTIFUSION (multimodal)
98.16
To conduct our study, we relied on Amazon Mechanical Turk, where we set the following qualification requirements for our users: HIT Approval Rate over 95% and at least 1000 HITs approved. Annotators were fairly compensated according to Amazon MTurk guidelines. Users were paid S0.60 for a batch of 28 images.
C.2 Qualitative Examples
We show further examples of images generated on the MCC-250 benchmark Fig. The results in Fig. Sal further highlight the robustness of MULTIFUSION for image composition tasks. When prompted with multimodal inputs, the model reliably includes both objects with correct attributes in the generated image. In contrast, Stable Diffusion oftentimes fails to render one of the objects, and Composable Diffusion mostly generates strange mixtures of objects where none of them can be clearly made out.
However, we observed MULTIFUSSOO to perform consistently poorly on prompts where both objects were animals. As shown in Fig. 8b the generated images contain creatures that exhibit features
17

---

Inputs
MuliiFusion
Stable Diffusion
Composable Diffasion
photorealistic
of green
photorcalistic a green apple
photorealistic
image of red bird
+
gold glock
+
(a) Positive examples highlighting the improvement MULTIFUSION over baseline models.
Inputs
MoltiFusion
Stable Diffucion
Composable Diffesion
photorealistic image brown
and
photorcalistic image yellow deg
and
(b) Failure of MULTIFUSION. Specifically, for animals the model often fails produce two distinet objects instead generates mixtures of both.
Figure 8: Representative examples of generated images the MCC-250 benchmark. MULTIFUSION is prompted with interleaved multimodal inputs, whereas Stable and Composable Diffusion prompted with text only.
Table 6: Fine-grained human evaluation results on animal compositions. Showing results for the subset of MCC-250 where both are animals. SD and Composable Diffusion were prompted with text, whereas we used multimodal prompts for MULTIFUSION, containing one interleaved visual reference for each object. Recall that each prompt is complex conjunction of two different animals with different colors.
One obj. w/
Two obj. w/
Methods
Zero obj +
One obj.
correct color f
Two obj. 1
correct colors
Stable Diffusion [%]
0.00
100.00 =0000000..00
87.24
227.04
31.99_27.90
Composable Diffuusion [%]
4.15
95.67 4.8
81.13
18.41 199.04
10.69
MultiFusion (text) [%]
2.57
97.43
83.60
26.22
21.38
MultiFusion (multimodal) [%]
3.56
96.45 = 83.60
26.22
21.38
of both animals. For Stable Diffusion, on the other hand, there is significant difference in performance on this specific sub-task. In Tab. [6] we show the results of the human evaluation on the animal-only subset. While Stable Diffusion performs comparably to the entire benchmark, both MULTIFUSION and Composable Diffusion portion of correctly generated images drops by over 50%.
D Quantitative and qualitative study of attention manipulation
To evaluate the effect of attention manipulation we compute additional FID scores for the multimodal prompt ablating the attention manipulation weight the text prompt. In Tab we compare the different FID) scores of the multimodal prompts to those of image and text prompis and observe that the higher the weight the closer the FID score is a text-only prompt. This quantitatively verifies that a higher attention manipulation weight on text prompt increases its influence on the generated image. Further, we qualitativley demonstrate the use of attention manipulation with multi-modal prompts in Fig. 99 We consistently increased the weight of the text prompt and illustrate how this correlates with an increasing influence of the text prompt on the generated images. If the text is weighted equally to the input image, its effect on the generated output remains negligible. Attention manipulation the input encoder offers an intuitive interface to balance the influence of each input modality. In general, we observed that text prompts should be up- weighted by factor of 10-25 to yield the desired results. Additionally, the manipulation behaves quite robustly with no adverse effects on the generated images for higher weights. Further, we show the interpolation
14] in combination
18

---

Table 7: Ablation of attention manipulation with FID-30k on the MS-COCO validation set. The weight of the text prompt is increased from to 20.
COCO FID-30K +
Guidance Scale
Image 6.03
Multimodal (1:1) Multimodal (10:1)
Multimodal (20:1)
Text
4.0
6.74
8.53
9.22
9.90
511
10:1
1511
20:3
25:1
30:1
35:1
house at lnke +
motoreycle forest
house lake +
Figure 9: Qualitative examples demonstrating the use of attention manipulation 14] in combination with multimodal prompts text prompt from left to right with the exact weights being depicted in the top line as text_ ext_weight image_wed weight.
and its effect on the generated images. We increase the weight on the
between objects one represented by text and one by an image prompt- through the use of attention manipulation in Fig. [0] Here we can also observe that text prompts up-weighted by factor of 10-25 result either in display of both or a shift of objects.
18:1
photorealistic image af grven apple
photorealistic image suitems
photorealisttic image a backpack
Figure 10: Qualitative ablation of attention manipulation 14 in combination with multimodal prompts of two objects and its effect on the generated images. The text prompt weight is increased from to 35, with the exact weights being depicted the top line as text eext_weight:
E
Qualitative examples of input order
The interleaved inputs are concatenated to one input vector and subsequently fed into the LM, which outputs embeddings for conditioning MultiFusion's image generation U-Net. Changing the order of interleaved inputs will change the embedding produced by the encoder and thus affect the conditioning of the denoising process, leading different output. This can be attributed to the autoregressive generation of embeddings with causal attention by the LM. The effect is particularly important for image prompts, where relationship between multiple concepts not specified in natural language.
Thus, we provide qualitative examples of reversed image prompts in Fig. 11 We can observe the
19

---

Figure 11: Qualitative examples illustrating the effect of reversing the input order of prompt images.
effects of autoregression and causal attention in all three examples of Fig. 11 showcasing that the object background of the first input image has the highest influence on the output image.
Further Qualitative Examples
Finally, we provide further qualitative examples, which showcase the multi-modal and multilingual capabilities MULTIFUSION Fig. These images further highlight the overall versatility and expressiveness of the model.
+
In photorealistic scene, high resolution, their face is
+
berühmte Gemälde 'Das Mädchen mit dem Perlenohrgehānge aber +
im Hintergrund, dieses Bild vom + Fuji Berg
+
die Protagonistin trinkt eine Tasse Tee.
Cartone animato animale che indossa abiti
+
+
Una fotografia realista de alta resolución:
+
+
con siguiente escena en el fondo:
+
Cet aliment a forme de coeur
Sur du bois
+
+
ayant
texture sulvante
Figure 12: Further qualitative examples of interleaved multimodal prompting. We provide one example in each language supported by MULTIFUSION: English, German, Italian, Spanish and French.
20


Reviews:
[{"review_id": "yPb75VV7Ou", "reviewer": "Reviewer_8eHc", "summary": "This paper presents an approach for creating a model that can take interleaved sequences of images and multi-lingual text as input, and generate novel images as output, by fusing together pre-trained models: (1) a ResNet image encoder from CLIP (2) an encoder-decoder text Transformer LM and (3) a Stable Diffusion (SD) image decoder. Most of the weights of the models are frozen, with some fine-tuning of adapter layers, the biases of the LM, and the cross-attention layers of the SD U-Net. Multimodal training is done on a combination of large scale image captioning, and VQA datasets, using the standard text-conditioned diffusion objective. The encoder-decoder text model was pre-trained on multi-lingual data, making the resulting image generation model also multi-lingual. ", "questions": "Q1) It would be helpful to give any of the following results that are available:\n- Scores for Table 1 with guidance scale 1.0, as SD seems to surpass MF as guidance scale decreases.\n- Results for Table 2 that also remove the image input (i.e. use just the text), to see if there's a benefit in compositional robustness from using the pre-trained encoder-decoder model (this would be very cool if so!).\n- Ablations of any components of the approach, e.g. the semantic embedding fine-tuning, the attention manipulation, or some of the datasets used in multimodal training (e.g., how much value do the VQA datasets add).\n\nQ2) Could more information be given about the encoder-decoder LM, in particular what objective was it pre-trained with (e.g. a denoising objective? prefix-LM)? What is in the training data (e.g. web pages? books? is there paired data across languages, or all monolingual corpora)?\n\nQ3) What data is used to train the multimodal adapters (to input images) and the SD U-net? It wasn't totally clear to me from the appendix. In particular, are other multimodal datasets used beyond the ones listed in lines 17-18 of the appendix? \"such-as\" and \"like\" make it seem like there could be others -- can you say anything about them, if so?\n\nQ4) What image is being provided as input in the Table 1 results? Is it the ground-truth image? If so, could you give some intuition for this? The results are much better with multimodal and image, but ground-truth image would be really strong supervision (and probably also explain why multimodal is worse). \n\nNot crucial for the author response, but it would also really improve the paper to clarify:\n- How are the translations of SNLI/MNLI (line 24 of the appendix) generated?\n- how is the contrastive training on the entailment dataset done?\n- give details of the multimodal training (e.g. learning rates, dataset size, GPU hours) for both the adapters and the SD U-net.\n- I didn't understand Fig 4a, as it seems that similarity needs a translation in two language pairs (e.g. en--de) but the categories here are single language.\n", "limitations": "I felt that the limitations section was pretty solid in qualitatively outlining weaknesses of the approach, although I'd appreciate experiments to quantitatively support the claim that attention manipulation can help prevent image context overriding text context.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "S1) The aim of this work, enabling an image generation model to take both images and text (in multiple languages) as input, is exciting and well-motivated by some of the qualitative examples in the paper: multi-modal inputs give complementary info, and multi-lingual text capabilities should broaden model accessibility. \n\nS2) I found the experiments on compositional robustness (MCC-250), with the improved results from the combination of this text encoder and the image inputs, interesting and think it has the potential to be a timely addition to the ongoing conversation about the role of the pre-trained text-encoder in compositional robustness of image generation (but see suggestions on baselines below). Doing a human evaluation user study was also a real strength of these experiments.\n\nS3) The qualitative results were compelling, particularly Figure 5 in the main text and Figure 4 in the appendix. ", "weaknesses": "W1) The experimentation was a bit thin. \n- Although there is definitely a shortage of current benchmarks for the new capabilities presented by this model, the contribution of the paper would be stronger if it were able to reappropriate existing benchmarks or create new ones to evaluate some of these capabilities (e.g. negative prompting with images, multimodal image composition). \n- The method has a few steps (e.g. contrastive fine-tuning on a natural language inference dataset; training on a large number of multimodal datasets, both VQA and captions; and using attention manipulation), but I couldn't find any ablations on these components. This, in combination with the lack of details on the [apologies, the rest of this sentence was missing earlier] datasets, makes me worried about whether the overall approach will benefit future work.\n- The quantitative results that are presented here would be more convincing with a few (hopefully) easy-to-run variants of the current settings (another classifier-free-guidance weight; ablating image inputs in the MCC-250 experiment); see questions below. The compositional robustness results are interesting, but giving an image as input is a pretty strong (and potentially unrealistic) source of supervision.\n\n[update after response] : I still feel that point a) above, about capability evaluation, is a weakness, but the author response definitely helped address the other points. Thank you!\n\nW2) The method relies on proprietary datasets and models for the language model (and possibly also for the image datasets, see questions below). I don't think this would be a crucial weakness except that almost no information is given about these datasets and models, even in the appendix. Given that the LM is frozen when doing the multimodal training, and that the capabilities of the fused system (with respect to multi-linguality, and the compositional robustness experiments) seem very likely to me to depend on the properties of this LM, more openness (ideally, using a publicly-released multimodal encoder-decoder transformer, like mT5-XXL, which also has 13B parameters) would really enhance the scientific value of this paper.\n- The encoder is described as a \"13B transformer encoder-decoder similar to GPT-3\", but GPT-3 is a decoder-only model, trained with a language modeling objective.\n- The LM dataset is described only as \"400B tokens of English, German, French, Italian, and Spanish\", and it's unclear whether the multimodal training data includes datasets other than the ones listed in lines 17-18 of the appendix. \n- The German-English versions of SNLI and MNLI used for the semantic embedding objective also seem to be proprietary. \n\nW3) The writing was somewhat unclear. In particular, a lot of details about the model (the pre-trained models used, the training data for the full approach) were unspecified in the main text, although outlined in the appendix (Section A). Some details about the experiments were also unclear, see questions.\n\n[update after response]. The response effectively addressed both W2 and W3 -- thanks!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review and insightful questions and will address them in the following. We strongly believe that the changes made during the review process have improved the quality of the paper.  [Rebuttal Pdf contains Tab. 5-7 as well as Fig. 11 and 12]\n\n### Q1) Further Results\n1. For guidance scale 1.0 StableDiffusion (SD) achieves a FID score of 26.09 and MultiFusion (MF)  achieves FID scores of 32.81 (Text), 24.22 (Multimodal) as well as 18.93 (Image). These results indeed show that SD does surpass MF for guidance scales <= 2. However, in line with previously reported results for Stable Diffusion, model performance significantly degrades for scales < 2. Further, CLIPScores for guidance scale 1.0 are 0.27 for SD and 0.25 for MF, which is in line with results for guidance scales 8.0 - 2.0. We will extend Tab. 1 with these results for the final version of the paper.\n2. We appreciate the reviewer’s suggestion to further illustrate the benefit of multimodal over text-only input and extend Tab. 2 respectively. Due to the limited time and resources available during rebuttal, we conducted the user study on a smaller scale with 1 sample per prompt. For the final version of the paper, we will provide the complete study at full scale and extend Tab. 2.\n\n|Methods|Zero obj|One obj.|One obj. w/ correct color|Two obj.|Two obj. w/ correct colors|\n|--|--|--|--|--|-|\n| Stable Diffusion [%]| 0.49|99.50| 90.25| 45.63| 28.57\n| Composable Diffusion [%] | 2.93| 97.01| 88.83| 36.65| 25.12\n| MultiFusion [%]| 0.67| 99.32| 93.57| 62.37| 54.24|\n| MultiFusion (text) [%]| 0.8| 99.00| 80.00| 43.00| 24.00|\n\nOne can observe that MultiFusion text-only is roughly on par with the other text-only models.\nConsequently, it is indeed the multimodal inputs and not the change in encoder architecture providing the performance increase.\n\n3. Early experiments have shown higher rates of convergence (based on visual inspection of generated outputs) for experiments using semantic embeddings. Consequently, we do not report ablation results without semantic fine-tuning, as it is an essential condition and not an optional architecture choice for successfully fusing an image generation model. We adjusted section 3 of the paper accordingly to provide more clarity on this behavior.   \nSimilarly, Attention Manipulation is required to counteract the fact that images are encoded by an order of magnitude more tokens than short text prompts. We show representative examples of how attention manipulation strengthens the influence of the text prompt on the generated output in App D Fig. 9 as well as Fig. 11 of the rebuttal. Further, we compute additional FID scores (cf. Tab. 7) for the multimodal prompt ablating the attention manipulation weight on the text prompt, showcasing that the higher the weight the closer the FID score is to a text-only prompt. This quantitatively verifies that a higher attention manipulation weight on text prompts increases the influence on the generated image.\nOur multimodal dataset is based on the findings and extensive ablations of MAGA [16], thus we did not deem it crucial to perform our own dataset ablations.\n\n### Q2) LM Details\n\nWe thank the reviewer for pointing out the inconsistency in the abstract regarding the LMs architecture. As stated in the main body of the paper, MultiFusion uses an auto-regressive transformer, i.e. a decoder-only architecture similar to GPT-3. We adjusted the appendix accordingly. We supply further information on the training data sizes for all components in Tab. 5, along with details on data splits, languages, and modalities in Tab 6. We will add both tables to the final version of the paper. \n\nOnly the semantic bias training uses paired multi-lingual data. All other training data is sourced from monolingual corpora.\n### Q3) Multimodal training\n The multimodal components (image prefix and adapters) are trained autoregressively on a combination of large-scale image-text datasets (c.f. Tab.5 and 6). Following MAGMA [16], the image tokens are prepended to the text tokens, and the language modeling loss for next token prediction is computed over the text only. \nThe cross-attention parameters of SD are finetuned on the LAION aesthetics dataset following the standard diffusion objective. We added further clarification to the paper regarding these training details. \n\n### Q4) Reference Image \nThe reference image used for the experiment in Tab. 1 is indeed the ground truth image from COCO. We agree with the reviewer that this is strong supervision, most likely not available for a real-world use case. However, the key takeaway of the experiment is the additional and more fine-grained information provided by an image input over text alone. We argue this conclusion to be reasonable, given the fact that MultiFusion does indeed not replicate the input image but produces a variation with more aligned details. We modified the discussion of the experiment to reflect the strong supervision signal provided by the input image.\n\n### Additional questions:\nThe translations of SNLI and MNLI were automatically generated using the DeepL API, which we clarified in the appendix of the paper. \nFurther training details are included in Tables 5 and 6\nThe scores in Fig 4a) are indeed based on paired data. In this case, we report the average pair-wise similarity of the displayed language with all other languages. We adjusted the caption of the Figure accordingly to provide further clarification. \n\n[16] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, Anette Frank. MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning. EMNLP, 2022."}, {"review_id": "PWtrefG15C", "reviewer": "Reviewer_sjMA", "summary": "This work proposed a novel method to build multilingual multimodal generation models that supports prompts composed of interleaved text and image. It combines a strong pre-trained multilingual language model with the image generation model from Stable Diffusion (SD) and achieves alleviated capabilities such as prompting with text and image combined. It also shows that the new model does better in composition generations, as one can provide reference images as part of the prompt. \n\nIn this work, a multilingual language model is trained in the first place (13B encoder-decoder structure model trained on 400B tokens), which itself is a strong multilingual model. It then adds an adapter module to the LLM model to support input in image format, following methods proposed in MAGMA. Finally, it aligns the trained encoder with the diffusion model taken from Stable Diffusion, with 15M text-image pairs. As a result, it can support multilingual, multimodal prompt for image generation, without training on massive text-image pairs dataset.\n\nIn experimentation, it showed that using both text and image as prompts can be beneficial, especially in composition generations. It also shows superior performance to existing multilingual text-image generation model AltDiffusion, possibly due to better alignment in multilingual embeddings. Further, the support of taking image as prompts can enable varies applications such as negative prompting with image, image composition, image variation and style modification.  ", "questions": "1. As you mentioned in the limitation section, the model cannot produce images that’s exact or close to the image in prompt. I wonder if there can be simple modifications made that can achieve this?\n2. How is the interleaved data being used? Does it effect the results if you change the order of the interleaved data?\n3. Have you tried other methods in addition to adapter to fuse image and text modality?\n4. Have you done any ablation studies on the semantic embeddings?\n5. On the MSCOCO dataset for generation, where is the image prompt come from (referring results in table 1)?\n6. Have you compared the multilingual LM trained to other ones (mT5 etc.) in multilingual benchmarks?\n", "limitations": "The authors addressed the limitations relatively well in the paper: \n1. The generated image cannot do copy of exact prompt images;\n2. Sometimes the image prompts need to be carefully chosen and do not always work;\n3. It suffers the same shortcomings (such as inappropriate content) as other generation models trained on very large-scale crawled dataset (LAION).\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* Novel and efficient method: fusing different pre-trained models works very well which can bootstrapping existing models such as stable diffusion to achieve different input format, avoiding the heavy cost of training model from scratch.\n* Enables prompting using both image and text and generates better images both in terms of metrics such as FID and human evaluation,  comparing to baselines that only takes text prompt.\n* Better results on composition generations from considering reference images in prompts.\n* Well written overall and addressed limitations of the work very well.", "weaknesses": "* Some of the details such as model parameter size, training data source and size are not presented in the main paper (included in appendix), which can be less clear when interpreting results presented in experimental section. It would be better to point those factors out when comparing with baselines in the main paper.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and subsequently address each limitation and question separately. [Tab. 5-7, as well as Fig. 11 and 12, can be found in the supplementary PDF of this review]\n\n### W1) Training details \nWe supply further information on the training data sizes, parameter counts, and GPU hours for all components in Tab. 5 along with details on data splits, languages and modalities in Tab 6. We will add both tables to the final version of the paper. \n\n### Q1) Image replication from prompt\nDirect replication of an input image would most likely require a different architecture used in encoding embeddings. While this in itself is an interesting research question, our approach aims to enable a more fine-grained conditioning of the image generation process through multimodal prompts that can be arbitrarily interleaved with one or more images. Nonetheless, our diffusion model can easily be combined with existing diffusion-based image editing techniques that can faithfully reconstruct and subsequently alter an image [58, 59]. In this case, MultiFusion would facilitate multimodal image editing, which we believe to be a promising avenue for future research.\n\n\n\n### Q2) Interleaved input data and order influence\nThe interleaved inputs are concatenated to one input vector and subsequently fed into the LM, which outputs embeddings for conditioning MultiFusion’s image generation U-Net. Changing the order of interleaved inputs will change the embedding produced by the encoder and thus affect the conditioning of the denoising process, leading to a different output. This can be attributed to the autoregressive generation of embeddings with causal attention by the LM. The effect is particularly important for image prompts, where the relationship between multiple concepts is not specified in natural language. Thus, we provide qualitative examples of reversed image prompts in Fig. 12, which we add to the appendix. We can observe the effects of autoregression and causal attention in all three examples of Fig. 12, showcasing that the object or background of the first input image has the highest influence on the output image.\n\n### Q3) Ablations on multimodal architecture\nWe limited our investigation to adapters for multimodal fusion. Previous research has already performed extensive ablations on adapter architectures and demonstrated their improved understanding of multimodal inputs over other methods [16]. \nBased on these findings, we argue that adapters are a suitable architectural choice for the task at hand. We adjusted Section 3 accordingly to better reflect these design choices. \n\n### Q4) Ablations on semantic embeddings\nOur choice of semantic embeddings was guided by the intuition that a focus on the semantics of a text prompt would best capture the information relevant for image generation and thus simplify learning the mapping from embeddings to image outputs. We decided to obtain high-quality semantic embeddings through parameter-efficient bias [4] instead of full model finetuning, based on the work of [33]. Early experiments have confirmed higher rates of convergence (based on visual inspection of generated outputs) for experiments using semantic embeddings. Consequently, we do not report ablation results without semantic fine-tuning, as it is an essential condition and not an optional architecture choice for successfully fusing an image generation model. We adjusted section 3 of the paper accordingly to provide more clarity on this behavior.\n\n\n\nQ5) MS-COCO image prompt\nThe reference image used for the experiment in Tab. 1 is the ground truth image from COCO. We realize this is strong supervision, most likely not available for a real-world use case. However, the key takeaway of the experiment is the additional and more fine-grained information provided by an image input over text alone. We argue this conclusion to be reasonable, given the fact that MultiFusion does indeed not replicate the input image but produces a variation with more aligned details. We modified the discussion of the experiment to reflect the strong supervision signal provided by the input image.\n\nQ6) Comparison to multi-lingual LMs \nThis is an interesting comparison that we believe to be relevant for future work! We focused our empirical study on multilingual reasoning with MultiFusion in comparison to current state-of-the-art image generation approaches (as highlighted in section L209, Fig. 5a). At the same time, we believe our method to be robust enough to be reproduced with other multilingual LMs. However, we think that limitations may be encountered when using a bidirectional (e.g. mT5) instead of an autoregressive model, similar to the issue highlighted in response to 2Kwy [54].\n\n[4] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2022.   \n[16] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, Anette Frank. MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning. EMNLP, 2022.   \n[33] Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search.arXiv preprint arXiv:2202.08904, 2022.   \n[54] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The Impact of Positional Encoding on Length Generalization in Transformers. arXiv preprint arXiv:2305.19466, 2023."}, {"review_id": "LHJKvES4nO", "reviewer": "Reviewer_g96C", "summary": "In this paper, the authors introduce MultiFusion, a novel approach that enables the expression of complex and nuanced concepts in text-to-image diffusion models (DM) through arbitrarily interleaved inputs of multiple modalities and languages. The “fusion” concept is at the core of the whole work: to fuse modalities together, pre-trained models (a LLM and a stable diffusion backbone) are fused together. Experimental results highlight the efficient transfer of capabilities from individual modules to the downstream image generation module. Notably, MultiFusion empowers the image generation module to effectively utilize multilingual, interleaved multimodal inputs, even when trained solely on monomodal data in a single language. The contributions of this work include the fusion of modalities for image generation, experimental evaluations, and the introduction of a benchmark dataset for further analysis and comparison regarding the multimodal compositionality of the models.", "questions": "**Lack of Chinese column in Figures 4a and 4b**: It is unclear why this column is missing or was included solely for the competitors if the proposed method was not tested on this language.\n\nThere's a typo on line 139: extracted", "limitations": "The authors adequately addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. **Innovative model fusion approach**: The paper introduces an innovative approach by combining a partially frozen multilingual Language Model (LLM) with a stable diffusion backbone. This fusion results in an interesting multilingual and multimodal encoder capable of seamlessly interleaving between input items, treating them as a modality-agnostic sequence.\n2. **Multilingual alignment investigation**: The authors conduct an investigation into the model's multilingual capabilities by translating the prompts from the DrawBench dataset. This exploration demonstrates an understanding of the importance of multilingual alignment. While there is a question regarding the accuracy of the translations, the authors acknowledge the potential benefit of utilizing literal translations in training the multilingual encoder, even though nuances in meaning may not be fully captured. This highlights the authors' attention to addressing the challenges and complexities of multilingual representation.\n3. **Contribution of benchmark dataset**: The authors contribute to advancing research in multimodal compositionality by producing and sharing the MCC-250 dataset. This benchmark dataset, described in detail in the supplementary material, serves as a valuable resource for assessing the compositionality of multimodal inputs, specifically comprising English text and images. The production and release of this dataset demonstrate the authors' dedication to promoting reproducibility, comparison, and further progress in the field of multimodal compositionality.", "weaknesses": "1. **Lack of clear architectural design and novelty**: The paper suffers from a lack of clarity in explaining and justifying its design choices. While references are provided, the underlying motivations and problem-solving aspects of these choices are not adequately explained. While Figure 1 attempts to illustrate the model structure, it is not accompanied by a clear rationale and explanation for the chosen modules and their interactions in the text. Enhancing the clarity of the architectural design would elevate the novelty and originality of the proposed approach. It is suggested to provide a high-level description that guides the reader in understanding the motivations behind specific choices. By focusing on the \"why\" rather than the low-level details, readers can, for example, grasp the purpose of unlocking only the biases in the LLM. Currently (line 130-131), it is unclear if this is a crucial step to obtain good results while keeping a parameter-efficient regime, or if it is marginal in that regard. The supplementary material can be utilized to provide additional low-level details for interested readers.\n2. **Lack of clarity in Figure 4 and semantic search paragraph**: While Figure 4a demonstrates higher similarities of translated prompts in the authors' method compared to competitors, it does not provide insights into the similarities between the reference and other negative samples. This additional information is crucial to establish the range of similarities that can be considered as genuinely low. Furthermore, Figure 4b indicates that the AltDiff competitor generates potentially more consistent images in each language, suggesting that the embedding similarity between references and translations may not be entirely representative. Clarifying these aspects would enhance the understanding of the results and provide a more comprehensive evaluation of the proposed method's performance.\n3. **Missing standard deviation in tables**: Including standard deviation in the results would provide important information about the variability and statistical robustness of the findings. By incorporating this measure, the paper would strengthen the reliability and credibility of the reported results.\n4. **Performance comparison and insights**: In Figure 4b, the AltDiff method demonstrates better performance, raising questions about the potential benefits of adding more languages to the MultiFusion method. While the authors suggest that alignment remains similar despite MultiFusion being fine-tuned using only English data, additional experiments are needed to provide substantial evidence that adding more languages to MultiFusion indeed yields improved results. Further investigation and insights in this area would enhance the value and understanding of the proposed method.\n\nIn offering these critical observations, I would like to emphasize that my intention is not to be harsh, but rather to provide constructive feedback. I acknowledge that explaining such a complex pipeline can be challenging and that significant effort has been invested in this work. However, I strongly believe that there is room for improvement in describing the architectural choices and highlighting the strengths of the paper, and I’ve tried my best to give possible suggestions in this regard. I’m convinced that addressing these aspects would significantly improve the quality and impact of the paper, but I don’t think this is something that could be fixed within the rebuttal period. In any case, I remain open to reconsidering my recommendation if any relevant insights emerge during the discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. Tab. 5-7 as well as Fig. 11 and 12 can be found in the supplementary PDF of the rebuttal.\nWe agree on the importance of design justification. We reiterate on design choice details for clarity in our global response under the section \"Reiteration on system design\".\n\n### W1) Architectural design\nPrevious work has demonstrated context-sensitive LM text encoders improve the expressiveness of downstream image generation models [40,2]. Accordingly, we model the backbone of MultiFusion’s (Mf) encoder as a 13B autoregressive transformer [10] trained on a multilingual corpus. We justify this decision over a bi-directional architecture by arguing that decoder models outperform bi-directional models on tasks such as manipulation with natural language (“Subject X with background Y) (cf qualitative results) or correct features attributions (“Red X, Blue Y”) (cf MCC-250 results). These capabilities have previously been attributed to the natural breaking of permutation equivariance [54], compared to bidirectional models relying entirely on positional embeddings. \nFollowing MAGMA [16], we add an image prefix and dedicated adapters to enable multimodal capabilities. We argue that adapters are a suitable architectural choice for multimodal prompts, as previous research has already performed extensive ablations on adapter architectures and demonstrated their improved understanding of multimodal inputs over other methods [16].\nOur choice of semantic embeddings was guided by the intuition that a focus on the semantics of a text prompt would best capture the information relevant to image generation and thus simplify learning the mapping from embeddings to image outputs. We decided to obtain high-quality semantic embeddings through parameter-efficient bias [4] instead of full model finetuning, based S-GPT [33]. \nEarly experiments have confirmed higher rates of convergence (based on visual inspection of generated outputs) for experiments using semantic embeddings. Consequently, bias tuning is an essential condition and not an optional architecture choice for successfully fusing an image generation model.  \nIn line with previous research [13], we finetune the cross-attention parameters of SD on LAION aesthetics. \nWe adjusted Fig. 1 & caption and section 3 to clarify details on the architecture and to better reflect the design choices. We would like to clarify that the expected level of implementation details in the main body is highly subjective. We aim to strike a balance between high-level motivation and low level details to satisfy the majority of readers. Indeed, multiple of the other reviewers specifically asked for more low-level information in the main body of the paper.  \n[54] arXiv:2305.19466\n\n### W2) Fig 4a and semantic search + W4) Comparison and insight \nWe agree that the addition of a baseline similarity between uncorrelated sentences is useful for the interpretation of Fig 4a. In fact, the baseline is roughly equivalent to the performance of CLIP reported on zh prompts, suggesting that the regular CLIP model is not aligned for this language and further reinforcing the increased performance of MultiFusion. We explicitly marked the baseline in Fig 4a).\nWe believe there to be a misunderstanding in the assessment of Fig 4b, which we would like to address. We do not believe that the results allow for a strong statement of AltDiffusion (AD) outperforming MF or vice versa due to the significant overlap in error bars. In fact, a good alignment of AD’s images is to be expected as the model’s image generative training is explicitly done on (aligned) multilingual data. The key insight from this experiment lies MF achieving comparable performance despite the image generation being only trained on English data. We attribute this capability to better-aligned embeddings. These alignment results suggest that aligned multilingual data on a downstream task is not necessary to achieve alignment. Rather, good embedding alignment of the backbone model in combination with readily available monolingual task-specific data is sufficient for multilingual alignment on that task.\n\nInvestigating more languages is indeed an interesting avenue for future work. However, the error bars already indicate that the performance of AD is not significantly better than MF’s. Thus demonstrating MF’s potential benefit in low-resource domains with only a few or even no image-text pairs available. This is, however, out of the scope of the current contribution and, therefore, also not a claim we make in the paper.\n\n### W3) Standard deviations  \nWe agree that standard deviations (std) generally improve the interpretability of results, wherefore we included error bars in Fig. 4.\nIn line with the literature do not report std for FID and CLIP in Table 1.\nWe did not initially include std for the empirical analysis in Tab. 2 as we simply reported the binary success rate over the entire benchmark. By considering the per prompt success rate over multiple samples, we now also report standard deviations over differing object compositions and will extend Tab. 2 accordingly. All models exhibit comparatively high stds (ca. 30PP for 2 objects), suggesting a fair amount of outliers for which the models perform significantly better/worse than on average. We believe the investigation of these tasks to be a promising avenue for future work and adjusted the discussion of the results accordingly. \n### Q1) Lack of Chinese columns\nThe models presented in Fig. 4 were evaluated on the languages that they have been trained on. Thus, we do not provide scores for AD on German and MF on Chinese. Further, we argue that it is crucial to include scores for MF on German and AD on Chinese, as these are the respective secondary languages of the models, i.e. the ones with the most aligned training data. We adjusted the caption of Fig. 4 to better reflect this specific aspect of the experiment. "}, {"review_id": "vFcvcnj1s7", "reviewer": "Reviewer_5v9N", "summary": "In this paper, the authors present a novel approach to expressing complex concepts with arbitrarily interleaved multimodal and multilingual input. Their approach leverages pre-trained models and allows an efficient fusion of different component without training a model from scratch.", "questions": "See my comments in Weaknesses", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow\n2. The experiments are well-designed and allow one to use existing pre-trained models while reducing the demand to train a system from scratch.\n3. The result on various benchmarks are promising and would invite more discussion in this line of work.", "weaknesses": "The motivation to attempt such a problem is rather weak. Under what circumstances, would one want to have interleaved multimodal input to generate images? Is it because we want to control the input? If so, why not compare the proposed approach with similar models such as ControlNet and DreamBooth? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and subsequently explain the motivation of MultiFusion in more detail. \n### W1) Motivation and comparison.  \n**Motivation**\nThere exist several motivations for using interleaved multimodal inputs. As demonstrated by the experiments in the paper, reference images often contain more fine-grained and detailed information than textual descriptions can provide (for example, for the style of an image as in Fig. 6b). Furthermore, visual inputs (next to textual) are universally understood by users further broadening model accessibility. On the other hand, text offers more abstract control of the generation and is able to express complex concepts. Consequently, combining the two modalities offers the best of both worlds, substantially increasing model expressibility. \n\n**ControlNet and Dreambooth**\nDreambooth, on the other hand, particularly focuses on generating specific subjects rather than using images as arbitrary reference information. Furthermore, DreamBooth requires computationally expensive fine-tuning for each subject, whereas MultiFusion can use image inputs natively during inference – again increasing accessibility.    \nSimilarly, ControlNet focuses on a completely different problem than Multifusion in providing dedicated control over scene composition using low resolution inputs such as edge maps. Moreover, ControlNet requires training a dedicated model for each control modality.    \nIn fact, we believe DreamBooth and ControlNet to be orthogonal approaches and see use cases where either may be used in combination with MultiFusion."}, {"review_id": "xr6BOUa8Z1", "reviewer": "Reviewer_2Kwy", "summary": "This paper presents MultiFusion, a multilingual multimodal image generation model, which can be effectively trained by fusing existing pre-trained visual models, language models, and stable diffusion models. Using a multilingual autoregressive language model as a bridge, MultiFusion follows MAGMA to enable multimodality by learning adapters. Before connecting the language model with the stable diffusion module, it learns semantic embeddings with a contrastive learning objective in a parameter-efficient setup. Finally, it connects the language model with the diffusion model with monomodal data. i.e., an image or caption.\n\nExperimental results demonstrate that the trained MultiFusion model can generate high-quality images with multimodal interleaved prompts. Besides, with modular design and the fusion of pre-trained models, the training can be quite effective compared to training from scratch. Several analyses such as attention manipulation also provide insights into the multimodal language models.", "questions": "- Are all the outputs of the language model connected to the diffusion model or only the last token? \n\nE.g., Input: [img_tok1] [img_tok2] [text_tok1] [text_tok2] [text_tok3] -> output vectors: [h1] [h2] [h3] [h4] [h5].\n\nAre all five vectors passed to the diffusion model or only the last one?\n\n- Why is an autoregressive model used as the encoder? I understand the CLIP model has disjoint text and image encodings, but why not use bidirectional models as the multimodal encoder? Any explanations?", "limitations": "As mentioned in the paper, the model always produces variations of input images, which limits its applications in image editing. I think it is worth mentioning this limitation, which provides further understanding of MultiFusion.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper makes a clever combination of pre-trained models and adapter learning techniques, including 1) MAGMA for cross-modal adaptation/fusing, 2) contrastive learning before fusing LM with the diffusion module, 3) cross-attention learning of SD to align the conditioning with the new embedding space. These operations delicately combine the pre-trained modules into an end-to-end multimodal-text-to-image model.\n\n- Experimental results show that MultiFusion can produce high-quality images conditioned on multimodal and multilingual inputs, with a wide range of applications and use cases. \n\n- The analysis on attention manipulation is quite interesting.", "weaknesses": "- It would be great to improve the presentation of the paper, especially methods and implementation details. Although Figure 1 presents an overview of the architecture, I have to guess some of the implementation details and carefully find clues from a large amount of text. Some suggestions: (1) you could provide some figures to show the details of how the adapters connect the pre-trained models and how you learn them; (2) you could also clarify the training tasks, and data in tables.\n\n- Existing works have explored how to learn adapters to connect pre-trained modules. For example, Flamingo learns gated adapter modules to connect language models with visual models, and generate text conditioned on multimodal inputs. The paper provides a full solution to the problem with careful design but it is kind of an integration of existing adapter methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 2Kwy for their feedback and suggestions. Below we address each weakness and question separately. Tab. 5-7, as well as Fig. 11 and 12, can be found in the supplementary PDF of this review.\n### W1) Architectural details\nWe thank the reviewer for the suggestions and incorporated this feedback in our global response regarding system design. The adapter layers are added to each attention and feed-forward layer of the transformer. Following the method proposed by MAGMA [16], the adapters are trained autoregressively on a combination of large scale image-text datasets (c.f. Tab.5 and 6). The image tokens are prepended to the text tokens and the language modeling loss for next token prediction is computed over the text only. We adjusted Fig. 1 and its caption to clarify these details. \nAdditionally, we supply further information on the training data sizes, parameter counts, and GPU hours for all components in Tab. 5, along with details on data splits, languages, and modalities in Tab 6. We will add both tables to the final version of the paper. \n\n### W2) Pre-existing work on adapters\nWe agree with the reviewer’s assessment that MultiFusion builds on previous work for multimodal adapter tuning. Nonetheless, we want to highlight that MultiFusion’s contribution lies in the investigation of building a cohesive system for a complex downstream task utilizing these pre-trained components, thus introducing a novel method of expressing prompts and steering image generation for diffusion models, and significantly reducing the computational costs required for this system.\n### Q1) Embedding tokens\nIndeed, the hidden representations of all tokens are passed to the diffusion model. In the example outlined by the reviewer, this would mean 5 embedding vectors are used for image generation conditioning. \n### Q2) Autoregressive encoder\nWe argue that decoder models perform better on tasks such as manipulation with natural language (“Subject X with background Y and style Z”) (cf. qualitative results) or correct features attributions (“Red X, Blue Y”) (cf. MCC-250 results). These capabilities can be attributed to the natural breaking of permutation equivariance [54], compared to bidirectional models relying entirely on positional embeddings. \nFurthermore, as demonstrated in Flamingo [55], multi-modal decoders can reason over multiple images, enabling further flexibility in the model’s capabilities (e.g., elements from multiple images can be trivially composed together, as demonstrated in Fig.2 and Fig.5).\nWe acknowledge that bi-directional models may outperform autoregressive ones on other embedding tasks [56], but argue that an autoregressive model is better suited for the tasks studied in MultiFusion due to the benefits outlined above.\n\n[16] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, Anette Frank.\nMAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning. EMNLP, 2022.  \n[54] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The Impact of Positional Encoding on Length Generalization in Transformers. arXiv preprint arXiv:2305.19466, 2023.   \n[55] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc et al. Flamingo: a visual language model for few-shot learning. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2022.   \n[56] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Retrieval-augmented multimodal language modeling. arXiv preprint arXiv:2211.12561, 2022."}]

Author rebuttal:
We thank all the reviewers for their detailed and helpful feedback.
We are encouraged that they found our solution for expressive image generation to be well-motivated (8eHc), novel (sjMA, g96C), and well-written (sjMA, 5v9N). Reviewers highlighted the efficient fusion of pre-existing models for simple, sample-efficient finetuning of a diffusion model (sjMA, 5v9N, 2Kwy), and improved capacity for flexible expression of complex concepts from combining complementary strengths in multimodal prompt interweaving (8eHc, sjMA, g96C, 2Kwy). We are pleased about the recognition of the importance and value of compositional robustness benchmarking (8eHc, g96c) as well as multilingual alignment evaluation (g96c).


Based on the reviewers’ suggestions, we provide further information on architectural choices, datasets as well as the overall training procedure. In the supplementary PDF, we supply further information on the training data sizes, parameter counts, and GPU hours for all components in Tab. 5, along with details on data splits, languages, and modalities in Tab 6. Further, we share additional qualitative (Fig. 11) and quantitative (Tab. 7) ablations on attention manipulation as well as qualitative examples of reversing the order of image inputs (Fig 12.). We consolidate common concerns and responses here and reply to the remaining comments individually in the hope to address them accordingly.


**@8eHc, sjMA, g96C, 2Kwy**
### Data:
We use proprietary datasets for both multimodal and LM training. However, we acknowledge that downstream capabilities are derived from these models and hope that the information in tables 5 and 6 can provide additional insight.
### Reiteration on architecture design:
Previous work has demonstrated that text encoders based on context-sensitive LMs improve the expressiveness of downstream image generation models [40,2]. Accordingly, we model the backbone of MultiFusion’s encoder as a 13B autoregressive transformer [10] trained on a multilingual corpus (cf Tab. 5 and 6). We justify our choice of an autoregressive decoder model over a bi-directional architecture by arguing that decoder models outperform bi-directional models on tasks such as manipulation with natural language (“Subject X with background Y) (cf qualitative results) or correct features attributions (“Red X, Blue Y”) (cf MCC-250 results). These capabilities have previously been attributed to the natural breaking of permutation equivariance [54], compared to bidirectional models relying entirely on positional embeddings.   
Following the method proposed by MAGMA [16], we add an image prefix and dedicated adapters to enable multimodal capabilities. The adapters are added to each attention and feed forward layer of the transformer and are trained autoregressively on a combination of large scale image-text datasets (cf Tab. 5 and 6), while the parameters of the language model remain frozen. We argue that adapters are a suitable architectural choice for multimodal prompts with arbitrarily interleaved sequences of text and image tokens, as previous research has already performed extensive ablations on adapter architectures and demonstrated their improved understanding of multimodal inputs over other methods [16].   
Our choice of semantic embeddings was guided by the intuition that a focus on the semantics of a text prompt would best capture the information relevant to image generation and thus simplify learning the mapping from embeddings to image outputs. We decided to obtain high-quality semantic embeddings through parameter-efficient bias [4] instead of full model finetuning, based on the work of [33]. The finetuning follows the supervised contrastive learning objective outlined in section 4.1.1 of [33].   
In the final step we finetune the cross-attention parameters of SD on the LAION aesthetics dataset following the standard diffusion objective, which is in line with previous research [13].   
We adjusted Fig. 1 and its caption, as well as section 3, to clarify details on the architecture and to better reflect the design choices. 

**@8eHc, sjMA**
### Ablations:
***Attention Manipulation***: Attention Manipulation is required to counteract the fact that images are encoded by an order of magnitude more tokens than short text prompts. We show representative examples of how attention manipulation strengthens the influence of the text prompt on the generated output in App D Fig. 9 as well as Fig. 11 of the rebuttal. Further, we compute additional FID scores (cf. Tab. 7) for the multimodal prompt ablating the attention manipulation weight on the text prompt, showcasing that with increasing weight the FID scores approach that of text only prompting. The experiment empirically verifies that a higher attention manipulation weight on text prompts increases their influence on the generated image.

***Semantic Finetuning***: Early experiments have shown higher rates of convergence (based on visual inspection of generated outputs) for experiments using semantic embeddings. Consequently, we do not report ablation results without semantic fine-tuning, as it is an essential condition and not an optional architecture choice for successfully fusing an image generation model. We adjusted section 3 of the paper accordingly to provide more clarity on this behavior.

[2] Yogesh Balaji, et al. eDiff-I. arXiv:2211.01324  
[4] Elad Ben Zaken, et al. BitFit. arXiv:2106.10199   
[10] Tom Brown, et al. Language models are few-shot learners. arXiv:2005.14165  
[16] Constantin Eichenberg, et al.. MAGMA. arXiv:2112.05253    
[33] Niklas Muennighoff. Sgpt. arXiv:2202.08904    
[40] Teven Le Scao, et al. BLOOM. arXiv:2211.05100     
[54] Amirhossein Kazemnejad, et al The Impact of Positional Encoding on Length Generalization in Transformers.  arXiv:2305.19466, 2023.

Meta-review: Paper received slightly divergent ratings of: 3 x Accept, 1 Weak Accept and 1 x Borderline Reject. Main concerns from reviewers focused on: (i) limited experimental validation, (ii) reliance on proprietary dataset, (iii) clarity of exposition, (iv) lack of novelty, and (v) lack of motivation for the problem. Provided rebuttal addressed many of these concerns and 4 out of 5 reviewers were left satisfied with responses. [g96C] remained somewhat unconvinced by the paper's contributions and robustness, however, did not strongly oppose acceptance of the paper. AC has looked at the reviews, rebuttal, discussion that followed and the paper itself. AC agrees with the more positive reviewers that paper is interesting, timely and would make a valuable contribution to the community. Therefore the recommendation is to Accept the paper. Authors are encouraged to incorporate their responses to the reviewers, along with additional experimental results provided in the rebuttal, into the main paper for the camera ready.
Decision: Accept (poster)