SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We show further examples of images generated on the MCC-250 benchmark in Fig. 8. The results in Fig. 8a further highlight the robustness of MULTIFUSION for image composition tasks. When prompted with multimodal inputs, the model reliably includes both objects with correct attributes in the generated image. In contrast, Stable Diffusion oftentimes fails to render one of the objects, and Composable Diffusion mostly generates strange mixtures of objects where none of them can be clearly made out.
However, we observed MULTIFUSION to perform consistently poorly on prompts where both objects were animals. As shown in Fig. 8b the generated images contain creatures that exhibit features
(a) Positive examples highlighting the improvement of MULTIFUSION over baseline models.
Figure 8: Representative examples of generated images on the MCC-250 benchmark. MULTIFUSION is prompted with interleaved multimodal inputs, whereas Stable and Composable Diffusion are prompted with text only.
(b) Failure cases of MULTIFUSION. Specifically, for two animals the model often fails to produce two distinct objects and instead generates mixtures of both.
Table 6: Fine-grained human evaluation results on animal compositions. Showing results for the subset of MCC-250 where both are animals. SD and Composable Diffusion were prompted with text, whereas we used multimodal prompts for MULTIFUSION, containing one interleaved visual reference for each object. Recall that each prompt is a complex conjunction of two different animals with different colors.
Methods	Zero obj ↓	One obj. ↑	One obj. w/ correct color ↑	Two obj. ↑	Two obj. w/ correct colors ↑
Stable Diffusion [%]	0.00±0.00	100.00±0.00	87.24±19.00	41.51±27.04	31.99±27.96
Composable Diffusion [%]	4.15±7.90	95.67±8.18	81.13±18.29	18.41±19.04	9.46±10.95
MultiFusion (text) [%]	2.57±6.92	97.43±6.92	83.60±20.31	26.22±22.95	21.38±22.70
MultiFusion (multimodal) [%]	3.56±7.36	96.45±7.36	83.60±20.31	26.22±22.95	21.38±22.70
of both animals. For Stable Diffusion, on the other hand, there is no significant difference in performance on this specific sub-task. In Tab. 6, we show the results of the human evaluation on the animal-only subset. While Stable Diffusion performs comparably to the entire benchmark, both MULTIFUSION and Composable Diffusion portion of correctly generated images drops by over  $50\%$ .
D Quantitative and qualitative study of attention manipulation
To evaluate the effect of attention manipulation we compute additional FID scores for the multimodal prompt ablating the attention manipulation weight on the text prompt. In Tab 7 we compare the different FID scores of the multimodal prompts to those of image and text prompts and observe that the higher the weight the closer the FID score is to a text-only prompt. This quantitatively verifies that a higher attention manipulation weight on text prompt increases its influence on the generated image. Further, we qualitatively demonstrate the use of attention manipulation [14] in combination with multi-modal prompts in Fig. 9. We consistently increased the weight of the text prompt and illustrate how this correlates with an increasing influence of the text prompt on the generated images. If the text is weighted equally to the input image, its effect on the generated output remains negligible. Attention manipulation at the input encoder offers an intuitive interface to balance the influence of each input modality. In general, we observed that text prompts should be up-weighted by a factor of 10-25 to yield the desired results. Additionally, the manipulation behaves quite robustly with no adverse effects on the generated images for higher weights. Further, we show the interpolation
Table 7: Ablation of attention manipulation with FID-30k on the MS-COCO validation set. The weight of the text prompt is increased from 1 to 20.
COCO FID-30K ↓
Guidance Scale	Image	Multimodal (1:1)	Multimodal (10:1)	Multimodal (20:1)	Text
4.0	6.03	6.74	8.53	9.22	9.90
1:1	5:1	10:1	15:1	20:1	25:1
house at a lake +	→
This motorcycle in a forest +	→
house at a lake +	→
between two objects - one represented by a text and one by an image prompt - through the use of attention manipulation in Fig. 10. Here we can also observe that text prompts up-weighted by a factor of 10-25 result either in display of both or a shift of objects.
Figure 9: Qualitative examples demonstrating the use of attention manipulation [14] in combination with multimodal prompts and its effect on the generated images. We increase the weight on the text prompt from left to right with the exact weights being depicted in the top line as text_weight : image_weight.
Figure 10: Qualitative ablation of attention manipulation [14] in combination with multimodal prompts of two objects and its effect on the generated images. The text prompt weight is increased from 1 to 35, with the exact weights being depicted in the top line as text_weight : image_weight.
E Qualitative examples of input order
The interleaved inputs are concatenated to one input vector and subsequently fed into the LM, which outputs embeddings for conditioning MultiFusion's image generation U-Net. Changing the order of interleaved inputs will change the embedding produced by the encoder and thus affect the conditioning of the denoising process, leading to a different output. This can be attributed to the autoregressive generation of embeddings with causal attention by the LM. The effect is particularly important for image prompts, where the relationship between multiple concepts is not specified in natural language. Thus, we provide qualitative examples of reversed image prompts in Fig. 11. We can observe the
Figure 11: Qualitative examples illustrating the effect of reversing the input order of prompt images.
effects of autoregression and causal attention in all three examples of Fig. 11, showcasing that the object or background of the first input image has the highest influence on the output image.
F Further Qualitative Examples
Finally, we provide further qualitative examples, which showcase the multi-modal and multilingual capabilities of MULTIFUSION in Fig. 12. These images further highlight the overall versatility and expressiveness of the model.
Figure 12: Further qualitative examples of interleaved multimodal prompting. We provide one example in each language supported by MULTIFUSION: English, German, Italian, Spanish and French.