Kosmos-G: Generating Images in Context with Multimodal Large Language Models
Authors: Xichen Pan
Li Dong
Shaohan Huang
Zhiliang Peng
Wenhu Chen
Furu Wei
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion Models
Vision-Language
Multimodal Large Language Model
Image Generation
Subject-Driven Generation
Primary area: generative models
TL;DR: Kosmos-G leverages Multimodal Large Language Models for subject-driven image generation

Abstract: Recent advancements in subject-driven image generation have made significant strides. However, current methods still fall short in diverse application scenarios, as they require test-time tuning and cannot accept interleaved multi-image and text input. These limitations keep them far from the ultimate goal of "image as a foreign language in image generation." This paper presents Kosmos-G, a model that leverages the advanced multimodal perception capabilities of Multimodal Large Language Models (MLLMs) to tackle the aforementioned challenge. Our approach aligns the output space of MLLM with CLIP using the textual modality as an anchor and performs compositional instruction tuning on curated data. Kosmos-G demonstrates an impressive capability of zero-shot subject-driven generation with interleaved multi-image and text input. Notably, the score distillation instruction tuning requires no modifications to the image decoder. This allows for a seamless substitution of CLIP and effortless integration with a myriad of U-Net techniques ranging from fine-grained controls to personalized image decoder variants. We posit Kosmos-G as an initial attempt towards the goal of "image as a foreign language in image generation."

Published as a conference paper at ICLR 2024
KOSMOS-G: GENERATING IMAGES IN CONTEXT WITH MULTIMODAL LARGE LANGUAGE MODELS
Xichen Pan 1.
Li Dong'
Shaohan Huang'
Zhiliang Peng' Wenhu Chen?
Furu Wei
Microsoft Research'
New York University?
University of Waterloo
aka.ms/GeneralAt
Cenerated Images
Entity
Mult timodal Prompt
swimming underwater
the Acropolis
batman suit
Generated Images
Entity
Mult Multimodal Prompt
wearing
jacket takes a celfie
Figure 1: Zero-shot subject-driven generation examples with multimodal prompts. Thanks to the advanced multimodal perception capabilities of MILM, KosMos-G can generate high-fidelity subject-driven images by approaching all image inputs as "foreign language".
ABSTRACT
Recent advancements in subject-driven image generation have made significant strides. However, current methods still fall short in diverse application scenarios, they require test-time tuning and cannot accept interleaved multi-image and input. These limitations keep them far from the ultimate goal of "image foreign language in image generation." This paper presents KosMos-G G, a model that leverages the advanced multimodal perception capabilities of Multimodal Large Language Models (MLLMs) to tackle the aforementioned challenge. Our approach aligns the output space of MILM with CL.IP using the textual modality as an anchor and performs compositional instruction tuning on curated data. KosMos-G demonstrates impressive capability of zero-shot subject-driven generation with interleaved multi-image and text input. Notably, the score distillation instruction tuning requires modifications to the image decoder. This allows for seamless substitution of CLIP and effortless integration with a myriad of U-Net techniques ranging from fine-grained controls to personalized image decoder variants. We posit KOSMOS- G as an initial attempt towards the goal of "image foreign language image generation."
Contribution during internship Microsoft Research.

---

Published as a conference paper at ICLR 2024
1 INTRODUCTION
In recent studies, advancements in text-to-image (T2I) generation, particularly with diffusion models, have shown remarkable progress in producing highly photorealistic, accurate, and diverse images from textual descriptions. Building on the unprecedented success of producing highly accurate images from text descriptions, numerous studies have delved into more sophisticated subject-driven generation to integrate images into text prompts to generate new customized images.
A group of approaches (Gal et al., 2022; Ruiz et al., 2022; Kumari et al., 2023; Tewel et al., 2023; Avrahami al., 2023; Hao et al., 2023; Smith et al., 2023) propose to fine-tune the models on each set of reference images, and fail to achieve subject-driven generation through generalized pre-trained model. Xiao et al. (2023); Wei et al. (2023); Chen et al. (2023; 2022) inject image features into the U-Net of diffusion models. However, such injection methods segregate the guidance for text and images, thereby limiting the effectiveness of joint modeling between the two modalities. Additionally, this approach is challenging extend to scenarios involving multiple entities. Recent work BLIP- III--iiffusion (Li et al., 2023a) learns object representations by synthesizing images through the composition of subjects with random backgrounds. This approach effectively endows it with a zero-shot, subject-driven text-to-image generation capability. However, the specific design of its input template and training data restricts its scalability multiple entities.
In contrast to previous methods that work with original CLIP text encoder Radford et al. (2021). I), we propose that through leveraging Multimodal Large Language Models (MLLMs) (Alayrac et al.. 2022; Hao et al., 2022: Aghajanyan et al., 2022; Huang et al., 2023; Li et al., 2023b), most of the challenges in subject-driven generation may be easily resolved. MILMs have expanded the perception capabilities of language models to multimodality, enabling them to perceive diverse modalities such as images. The idea of leveraging MILMS for subject-driven generation presents several advantages: 1) It capitalizes on the inherent vision-language alignment within the MLLM. 2) The MLLM architecture naturally supports interleaved interleaved multi-image and text input. 3) The pre-trained MLLM can effectively model multimodal input in context.
To support zero subject-driven generation with interleaved multi -iati- and text input, we present KosMos-GE G, which leverages the advanced multimodal perception of MLLM following an "align before instruct" manner. Specifically, we start from the multimodal language modeling stage, leading to the 1 (Huang et al., 2023) MILM. It envisions language models universal task layer, perceiving free-form interleaved vision-language inputs and consolidating various task predictions into textual formats. Given the aligned vision-language representation, we then use the language modality as an anchor and align the output space of the MLLM with the CLIP text encoder. Finally, we perform instruction tuning on the curated data, KOSMOS-G G accepts captions as input, where each entity is followed by its segmented image. The model is trained to faithfully reproduce all entities, render the text content, and follow the instructions. In this process, the frozen pre-trained diffusion image decoder serves as score metric. We distill the learned data distribution to pass the differentiable gradient to the MLLM. This enables KOSMOS-G to harness rich features from the image encoder to generate images faithfully reproducing the contents across various contexts (see Figure 1, we also present examples with more diverse interleaving in Figure 9).
Benefiting from general purpose pre-training, KosMOs- G approaches the objective of "image as foreign language in image generation." " This means KosMos-G G can capture novel concepts from input images and guide personalized creations in zero-shot setting. Notably, KosMos-G also stands as the first model to master zero-shot subject-driven generation with interleaved multi-image and text input. Owing to the score distillation instruction tuning, KosMOs-G do not need to modify any parameters of the image decoder, i.c., the diffusion U-Net and VAES. This makes it possible for to seamlessly substitute CLIP with KOsMOs-G in any image generation system. As a result, a plethora of applications can be unlocked in conjunction with U-Net techniques, ranging from fine-grained controls like ControlNet (Zhang & Agrawala, 2023) to personalized or stylized image decoder variants like amazing community contributed LoRA (Hu et al., 2022) checkpoints.
Overall, we propose KOSMOS- G as an initial attempt towards the objective of "image as foreign language in image generation." " We summarize our main contributions as follows:
1. We propose to leverage the advanced multimodal perception of MLLMS for subject-driven generation with interleaved multi-image and text input.
2

---

Published as a conference paper at ICLR 2024
Cross Attn
Cross Attn
3
D
AlignerNet
Multimodal Large Language Model
as an oil painting in the style of
Interleaved Vision-Language Prompt
Figure 2: KOSMOS-G comprises an MLLM for multimodal perception, coupled with an AlignerNet that bridges the MLLM to the diffusion U-Net image decoder. KosMos-G G can pass the fine concept- level guidance from interleaved input to image decoder, and offer a seamless alternative to CLIP. Orange denotes the trainable modules; Blue denotes the frozen ones.
2. We propose a compositional instruction tuning task, leading amazing zero-shot multi- entity subject-driven generation capability.
3. Score distillation instruction tuning allows KosMos-G to seamlessly interface with a spectrum of U-Net techniques, indicating broad applicability and potential for integration into various frameworks.
KOSMOs-CS G: IMAGE AS A FOREIGN LANGUAGE IN IMAGE GENERATION
As shown Figure KOsMOs-G G is model that can perceive interleaved multi-image and text input and generate subject-driven conditions. Specifically, the backbone of KOsMos-G MLLM is Transformer-bis based causal language model, serving as ceneral-purposs interface to multimodal input. We train KOsMOS-G following an "align before instruct" manner, the entire training pipeline can be divided into stages:
1. Multimodal Language Modeling:
We pre-train the MILM from scratch on multimodal corpora, including monomodal data, cross-modal paired data, and interleaved multimodal data with language modeling loss following KosMos-1.
2. Image Decoder Aligning: We use the U-Net (Ronneberger al., 2015) of Stable Diffusion v1.5 (Rombach et al., 2022) as our image decoder. We trained an AlignerNet on only textual data to align the output space of KosMos-G to U-Net's input space through CLIP supervision. Here, the language acts as the anchoring modality, ensuring image input is also compatible with the image decoder.
3. Instruction Tuning: We further fine-tune KOSMOS-G through compositional generation task on curated data, with the differentiable gradient passed from the frozen U-Net.
In Stage 1, only the MLLM is trained. In Stage 2, AlignerNet is trained with MLLM frozen. During Stage both AlignerNet and MI.LM are jointly trained. The image decoder remains frozen throughout all stages.
2.1 MULTIMODAL LANGUAGE MODELING
Following KOsMOS-L 1, KOsMOS-G G perceives general modalities in a unified way. To achieve this, represent the input format single sequence using special tokens. Specifically, the
3

---

Published as a conference paper at ICLR 2024
tokens 
 and 
 to denote start- and end-of-sequence. We also incorporate <image> and < //image> tokens to indicate the start and end of any embedded image representations within the sequence.
Our methodology involves encoding both text tokens and images into vectors, which are then fed into the decoder. For text tokens, we use lookup table to map them into embeddings. To handle the input images, we employ a vision Transformer (Dosovitskiy et al., 2021) as the embedding module. Furthermore, Resampler (Alayrac et al., 2022) is used as an attentive pooling mechanism reduce the number of image embeddings. After obtaining the embeddings of an input sequence, we feed them into the Transformer-basse an auto- regresseeeee manner. A softmax classifier on the Transformer is used to assign probabilities to each token in the vocabulary.
decoder. The left-to-right causal decoder processes the sequence in
KOsMOs-G G is first trained using the next-token prediction task. The training objective is to maximize the log-likelihood of tokens in examples. It's important to note that the training loss only takes into account discrete tokens, specifically text tokens. The MLLM component has 24 layers with 2,048 hidden dimensions, 19192 FFN intermediate size, and 32 attention heads. For faster convergence, the image representation is obtained from a pre-trained CLIP ViT-L/14 model with 1,024 feature dimensions. The images are preprocessed into 224 x 224 resolution during training. We freeze the parameters of the CLIP model except for the last layer during training. The total number of parameters of the MILMM is about 1.6B.
2.2 IMAGE DECODER ALIGNING
After undertaking multimodal language modeling, we have successfully aligned vision and language perception within MLLM. To make KUsMOs-G capable of image generation, we incorporate diffusion models (Sohl-Dickstein et al., 2015) as our image decoder. Specifically, we adopt the widely accepted Stable Diffusion v1.5 (Rombach et al., 2022). It's important to note that we only replace the CLIP text encoder (Radford al., 2021) with multimodal KosMOs-G, G, without making any modifications to the U-Net architecture or weight. This setup allows KOSMOS-G to effectively collaborate with techniques applied to the U-Net, like ControlNet (Zhang & Agrawala, 2023) and various community LoRA (Hu et al., 2022) variants. In this section, we will provide brief preliminaries of latent diffusion models, and then delve into the process of aligning the output space of KosMos-G with the image decoder after the aforementioned replacement.
Preliminaries of Latent Diffusion Models Diffusion models define a Markov chain of forward diffusion process q, adding Gaussian noise samples to the initial real data Zg ~ q(z) over steps. Here, z denotes latent representations rather than pixel values. The efficient, low-dimensional latent space is approximately perceptually equivalent to high-dimensional RGB space, while the redundant semantically meaningless information present in the pixel domain is eliminated. Perceptual compression models (i.e., VQ-VAE) consisting of È and encode the real data into the latent space and reverse, such that D(E(x) x. Latent diffusion models use latent representations z = Ex) instead of working directly with pixel values during the diffusion process. The final output can be decoded back to pixel space via (z)). The separate mild perceptual compression stage only eliminates imperceptible details, leading to competitive generation results with a much lower cost. The forward process qz 12(1) at each time step can be expressed as follows:
q(2: 12-)) -N(z /I 9,21-1,
T a (Zo) = II 2zx((2=11)
(1)
in which 3, (0, 1) denotes the step size. Note B < B2.
Diffusion models learn a U-Net (Ronneberger et al., 2015) denoted as Eg to reverse the forward diffusion process, constructing desired data samples from the noise. Let a, = 1 - Be and ā, I 0g. We can reparameterize the denoising process p(z(-1/z;) also Gaussian distribution. This distribution can be estimated by Eg and takes the following form:
po(2:-1 l2.) = -N(x- :o: (2,, l). Eo(z,, L))
1
9t
(2)
with uo / Va,
(Ze
vI (o t))
4

---

Published as a conference paper at ICLR 2024
Racaraed PARD artwed
KosMOS-G Space
.V-ihear
N-encodar
image
N-decoder
Red
Algist wowken
Pull Close
M
N
Recoonstruct
M-ddacader
MBE L03a
Text
CLIP-T Space
Ippul Copton
(a) Align process. Text serves as an anchor, image embeddings are natu- rally aligned throughout the process.
(b) AlignerNet architecture. Linear layers are used project the output dimension of MLLM to = 768, the purple elements denote the learned latent queries Q. and Qu.
Figure 3: Overview of alignment.
The learning objective of diffusion models is to approximate the mean f (z,, t) in the reverse diffusion process. To achieve this, we can utilize the variational lower bound (ELBO) (Kingma & Welling, 2014) to minimize the negative aogliikelhood po (zo) (Ho et al., 2020). The simplified objective can be expressed as denoising objective:
Laifs F12..=-W0..).
[lle eo (ze, t)]
During inference, (Ho & Salimans, 2022) proposes to use classifier-free guidance to obtain more relevant generation results.
== -0 (g (2,,,,,,t) (w--1 1) (z)..))
(4)
where w is guidance scale, y denotes the condition.
Align Output Space with Diffusion Models Upon replacing the previous CLIP text encoder with KosMos-Gi G, the main focus is to address the misalignment issue between the KosMos-G and the image decoder. We discovered that simply fine-tuning KosMos-C G using the gradient passed from the image decoder results in both trivial alignment and compromised image quality.
Inspired by (Qin al., 2023), we propose the AlignerNet consisting an encoder M and a decoder N to learn the alignment between the KosMos- G source space S and CLIP text encoder target space T. Given single text-only caption C, KOSMOs- source encoder and CLIP target encoder encode the caption into embeddings denoted as R4xx,, xd, and E X4 respectively. Here, and indicate the length of features and embedding dimensions.
As shown in Figure 3a, we employ the encoder M to minimize the distance between the text source embedding and the target embedding, aiming for a close approximation M(s) s t through:
Lmae
F E (
(5)
To mitigate the reduction in feature discrimination, also employ decoder V to reconstruct the source embedding N ((M(s)) ss s through:
Lree Bans (
(6)
Different from (Qin et al., 2023), KosMOs-G is vision- multimodal encoder. The language modality serves as an anchor throughout the process, aligning the entire KosMos-G space with the image decoder input space, thus also achieving semantic alignment for the image embeddings.
To efficiently process lengthy sequences consisting of multiple images and minimize memory usage, KosMOs-G encodes the interleaved vision-language input sequence into variable-length embeddings.
5

---

Published as a conference paper at ICLR 2024
However, the use of variable length embeddings makes the MLP-based GlueNet (Qin et al., 2023) unsuitable for learning alignment. To address this, we employ a Transformer based architecture in AlignerNet, enabling to effectively align the source and target spaces with mismatched sequence lengths and embedding dimensions.
As shown in Figure 3b, both M and N share similar architecture design, consisting of Transformer encoder and Transformer decoder. The Transformer encoder and decoder in both models comprise 12 layers, with an input dimension d 768 and hidden dimension of 3072. This configuration results in approximately 225M parameters in total. In the cross attention module of Transformer decoder, we use variable length learned latent queries Q.M R4xd in M and Qu E R' in W to match sequence length. Note that as discussed in Section 4.3, we can still align MLLM with Kosmos-C G through directly using diffusion loss in Equation 3 with the help of AlignerNet. While is more costly and leads to worse performance under the same GPU days.
2.3 INSTRUCTION TUNING
After achieving a semantic alignment between KosMos-G G and the image decoder. our model successfully generate images following interleaved vision-language guidance. However, the multimodal language modeling and text-only alignment stage only preserve the semantic consisteney between the input and output, KosMos-G still can not leverage rich features extracted from the image encoder to generate images faithfully reproducing the contents in various contexts.
To pursue our objective of "image as a foreign language in image generation," we curate interleaved vision- lasion-language data and use the diffusion loss in Equation 3 to further fine-tune KosMOs-G. Specifically, we propose a compositional generation task which we input captions containing entities, with each of them followed by their corresponding images, like "
 A caf <image> image embedding of the cat </ image> and dog <image> image embedding of the dog </ mmmae>> sleeping in the garden < <image> image embedding of the garden </image> 
". Our model is trained generate images following the input instruction.
To construct the requisite data, we first caption the image, then extract the entities from the caption, and obtain the segmentation results from the image itself. A detailed introduction the entire pipeline can be found in Section 3.1. Additionally, we leverage the data constructed by (Brooks et al., 2023) for InstructPix2Pix to improve KosMos-Gi -Gs image editing capability. This data is structured "<g> caption <image> embedding of the original image < / /image>>>>> edit instruction < 
". We also mix some text-to-image data to preserve the language alignment already achieved.
Our goal is to leverage MLLMs to model image distributions through direct latent space sampling. In this setup, the pre-trained frozen Stable Diffusion U-Net serves as a score metric, distilling the learned data distribution. This strategy is similar to Score Distillation Sampling (SDS) (Poole et al., 2022). From the perspective of score distillation, the KL divergence between KOSMOS-G (denoted which encodes inputs into condition C) and the score function is equivalently minimized for distilling learned probability density in the image decoder:
min Lriss Exo.i. (1(e.(((ze-1ee) (2,, 20) (21-11122;;
This enables KOSMOS-G G to leverage rich features from the image encoder to generate an image faithfully reproducing the contents across various contexts. More details about the score distillation instruction tuning can be found in Appendix C.
3 MODEL TRAINING
3.1 MULTIMODAL TRAINING DATA
The multimodal language modeling stage in Section 2.1 using the same setting of KOsMOS-1 (Huang et al., 2023), where the models are trained on web-scale multimodal corpora, consisted of text corpora, pairs, and interleaved data images and texts. For the image decoder aligning stage in Section 2.2 only use the caption from image-caption pairs. For the instruction tuning stage in Section 2.3, we use constructed data from Open Images V7 dataset (Kuznetsova et al., 2020), the image-caption pairs, as well the image editing data from InstructPix2Pix (Brooks al., 2023).

---

Published as a conference paper at ICLR 2024
Image Captioner
Caption: A bedroom with bed, chair, Iv, and table
Large
Tags: bed,
Language Model
chair, Iv, table
Image Segmentation Model
A bedroom with a bed a chair
a tv
and table
Figure Overview our data construction pipeline for compositional generation instruction tuning.
Captions The image-caption pairs sourced from multiple datasets, including English LAION- 2B (Schuhmann et al., 2022), LAION-400M (Schuhmann et al., 2021), COYO-700M (Byeon et al., 2022), and Conceptual Captions (Sharma et al., 2018; Changpinyo et al., 2021). English LAION-2B, LAION-400M, and COYO-700M are collected from Common Crawl web data by extracting images and the corresponding alt-texts. Conceptual captions are also derived from web pages.
Constructed Data We approximately 9M images from the Open Images V7 dataset (Kuznetsova et al., 2020) to construct our compositional generation instruction tuning data. As illustrated in Figure 4, we begin by generating captions with BLIP-2-OPT-6. we employ an LLM MPT--BBInnttre original image, along with the text of each entity, is then input into the text-prompted segmentation model CLIPSeg (Lüddeeke & Ecker, 2022) to derive the corresponding image of each entity.
(Li et al., 2023b). Subsequently, (Team et al., 2023) to extract entities from the captions. The
3.2 TRAINING SETUP
Our implementation is based on the TorchScale (Ma et al., 2022) library, which is designed for large- scale model training. Following Kosmos- 1 (Huang et al., 2023), also use MAGNETO (Wang et al., 2022), a Transformer variant, as the backbone architecture our MLLM and AlignerNet. The whole training process took around four days with 256 NVIDIA V100 GPUs, i.e., day for image decoder aligning, and three days for instruction tuning. the instruction tuning stage, we use a blend of constructed data, InstructPix2Pix data, and caption data ratio of 2:2:1. For constructed data, to enhance input robustness, we randomly drop the texts of entities with a probability of 0.5 and also maintain the background of the segmented entities with a 0.5 probability. Other training configurations can be found in Appendix A
4 EVALUATION
4.1 MAIN QUALITATIVE RESULTS
As shown in Figure 5, KOSMOS-G delivers impressive zero-shot generation results across diverse settings, yielding meaningful and coherent outputs even for highly customized subjects. The visual samples showcase generative capabilities in re-contextualization, stylization, modification, and accessory incorporation. Notably, multi-entity subject-driven generation is very challenging even for fine-tuning methods like DreamBooth (Ruiz al., 2022; Avrahami al., 2023). While owing from the novel compositional generation instruction tuning, KosMos-G G is the first model that is capable of achieving this in zero-shot setting.
4.2 QUANTITATIVE RESULTS
We do quantitative evaluations of KOSMOS-G G on DreamBench (Ruiz al., 2022) for single-entity subject-driven generation and MS-COCO (Lin et al., 2014) for text-to-image generation.
The DreahBench dataset contains 30 subjects and features 25 prompt templates, resulting in 750 unique prompts covering skills like re-contextualization. modification, accessorization, etc. We follow prior work to generate 4 images for each prompt form the 3000 images for comprehensive evaluation. We follow DreamBooth to adopt DINO, CLIP-I to evaluate the subject fidelity, and CLIP- T to evaluate the text fidelity. We classifier-free guidance scale of 7.5 and 100 DPM-Solver (Lu
7

---

Published as a conference paper at ICLR 2024
as an oif painting by van Gogh
in Unity3D
Minecraft
themed backpack
LA
in suit
wearing
eating
style of
Figure 5: Zero-shot image generation examples with multimodal prompts.
Methods
DINOT CLIP-I CLIP-TY
Methods
FIDJ
Real Images (Oracle)
0.774
0.885
T21 Models
Fine-Taming
GLIDE (Nichol et al., 2022)
12.24
Make-A-Scene (Gafni et al., 2022)
11.84
Textual Inversion (Gal al., 2022)
0.569
0.780
0.255
DreamBooth (Ruiz al., 2022)
0.668
0.803
0.305
DALL-E (Ramesh et al., 2022)
10.39
BLIP- Difffuiiuoion (Li et al., 2023a)
0.670
0.805
0,302
SD v1.5t (Rombach et al., 2022)
9.34
Imagen-3.4B (Saharia et al., 2022)
7.27
Test Time Tuning Free
CLIP-Aligned VL2I Models
Re-Imagen" * (Chen al., 2022)
0.600
0.740
0,270
SuTI (Chen et al., 2023)
0.741
0.819
0,304
GILL-8B (Koh et al., 2023)
12.20
BLIP-Diffusion (Li et al., 2023a)
0.594
0.779
0.300
Emu-14B (Sun et al., 2023)
11.66
KosMos-G" (single image input)
0.694
0.847
0.287
KosMos-G-1.9B
10.99
Table Left: Quantitative comparisons on DreamBench. denotes zero-shot methods. Right: Zero-shot FID comparisons on MS-COCO. indicates results evaluated by under same settings and seed with KosMOs-G.
et al., 2022) inference steps for sampling. As shown in Table 1, zero-shot KOSMOs-G outperforms Textual Inversion and Re-Imagen and exhibits marginally better performance than DreamBooth and BLIP-Diffusion with only a single image input. Furthermore, Our results are also comparable with SuTI, without requiring expensive apprenticeship learning supervision. KosMos-C G accepts only single image as input, we select a clear image from the 4-7 provided images for each subject to avoid occlusion. We slightly modify the prompt template ensure better alignment with the instruction tuning data. The images and prompt used can be found in Appendix B.
For the text-to-image generation, We generate images using 30,000 randomly sampled captions from the MS-COCO (2014) validation set. We use a classifier-free guidance scale of 3.0 and 250 DDIM (Song al., 2021) inference steps for sampling. As shown in Table 1, KosMos-C G surpasses other CLIP-aligned VL2I models, delivering the optimal alignment results.
4.3 ABLATION STUDIES
8

---

Published as a conference paper at ICLR 2024
KOSMOS-G G hefore instruction tuning
Stable Diffusion v1.5 Romhach et al. (2022)
Figure 6: Comparisons with cases presented in the second row of Figure
oil painting in the style of
plays
(a) KOSMOS - with canny control using ControlNet.
(b) KOSMOS- G with LoRA variant.
Figure 7: Various Applications of KosMos- in Conjunction with U-Net Techniques. Figure 7b, the left image is generated using standard U-Net, the right one is produced with LoRA-tuned U-Net.
We conduct ablation studies to find out the importance of the image decoder aligning and instruction tuning. Table demonstrates that direct end-to-end fine-tuning or using decoder-only architecture fail generate meaningful images. Incorporating AlignerNet and CLIP supervision, however, results in outcomes close to the orig- inal SD v1.5. End-to-end training is also feasible with AlignerNet, but it is more costly due to the additional computational of the U-Net. This leads to worse performance within the same GPU days. We also compared the generation results from KOSMOS-G before instruction tuning and the standard SD v1.5 against our final model. As illustrated in Figure without instruction tuning, KOsMOs- G can only generate contents semantically aligned with the vision-language input. SD baseline also remains at the semantic level and fails to faithfully reproduce the entities in the generated images.
Methods
FID
SD v1.5
9.34
E2E w/o AlignerNet
Failed
F2E w/ AlignerNet
11.30
12-Layers Decoder
Failed
12-Layers AlignerNet
9.89
24 Layers AlignerNet 9.55
Table 2: Ablation study results for image decoder aligning on MS-COCO.
4.4 APPLICATIONS
As highlighted in Section 2.3, KOSMOS-G can seamlessly replace CLIP in any image generation system. This remarkable property unlocks myriad of brand-new applications that have never been possible before. We demonstratee variants (Hu et al., 2022) in Figure 7. KosMos-G G works perfectly with these techniques. Building on the CLIP space, we believe our model will push forward the transition from text-conditioned generation toward lasion-language generation, paving the way for numerous novel applications.
integration with ControlNet (Zhang & Agrawala, 2023) and LoRA
CONCLUSION
We propose KosMOs-GG G, model capable of high-fidelity zero-shot subject-driven generation from interleaved multi-image and text input. Our approach hinges on unique "align before instruct" pre- training strategy. KOsMOs-G demonstrates competitive single-entity subject-driven image generation and text-to-image capability, it also stands as the first model to extend zero-shot subject-driven image generation to multi-entity scenarios. Furthermore, KosMos- G allows seamless replacement of CLIP, unlocking various new applications in conjunction with other U-Net techniques such as ControlNet and LoRA. In general, we present KOsMOS-G as preliminary effort aimed achieving the objective of "image as foreign language in image generation."

---

Published as a conference paper at ICLR 2024
REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. CM3: A causal masked multimodal model of the Internet. ArXiv preprint, abs/2201.07520, 2022. URL
httpps:/aaze wtggabs/////ooooooooot...tt..scr 2201. 07520.
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems, 2022. URL https::tp/..o.. /ttp:///epeeeeeeeeeeemeee net/ forum? deetmmiiddbbMuu mAbPbs.
Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scene, Extracting multiple concepts from a single image. arXiv preprint arXiv:2305.16311 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset, 2022.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tal tail visual concepts. In IEEE Conference on Computer Vision and Patten Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 3558-3568 URL
Computer Vision Foundation / IEFF, 2021. doi: 10. 00.1109/CVPR444437220001
https: https: httpss///ppeaaccesssss the.cceeessssteeee..ee com http://otteetttcmeeoe http://cccttetttct2/22///////aa http:/200011tmmh/hhhaaae
Tail oow.aooooaaa...Laa___///o222/22
html.
Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval-augmmnted
text-to-image generator. ArXiv preprint, abs/2209. 14491, 2022. URL abs/2209. 14491.
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming- Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. ArXiv preprint, abs/2304.00186, 2023. URL https: /ttp///aavvvv//aa///a///eee/aaaa wwtp:/iwworg//////o////////b habs///b///an 2304. 00186.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi. Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamlim: Synergistic multimodal comprehension and creation. ArXiv preprint, abs/2309. 1499. 2023. URL https: /ttppp:///axiv....// wttp/////////////////////////// 11499.
abs
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL http::::///peeeveiiiee iew net forum? mimi- Yi CLcF cbNTT
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a- scene: Scene- based soxt-to-mmae generation with human priors ArXiv preprint, abs/2203.13131,, 2022. URL https: / httaaaivvvooo org/ abs 222/222..222 13131..
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. ArXiv preprint, abs/2208.0161A 2022. URL https: : aariv..i www.iivvorr/.... wogg/baaa..........oooo.... /2208. 61....
10

---

Published as a conference paper at ICLR 2024
Shaozhe Hao, Kai Han, Shihao Zhao, and Kwan-Yea K Wong. Vico: Detail-preserving visual condition for personalized -ottte--mmaeeeee generation. ar preprint arXiv:236600977,, 2023.
Yaru Hao, Haoyu Song, Li Dong. Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are canerral-prposs, interfaces. ArXiv preprint, abs/2206.06336, 2022. URL https:t////.. / //axxvivvvv//////aa www.ooggg///////////o...//.ee///
06336.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. ArXiv preprint, abs/2207.1259p www.wiowwaaaaaa..eaaa/oo www.//ggg/s...oooooo 22007 12598.
2022. URL https: http//aavvvv///a///a///ee//..ava
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: httpthttsptpppooeeeeeeeeeeeeeeee http:///ooooceeee..ooeee neurips htcppsscccpppppep hccppapee///2///// html..
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Interna- tional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: : ///p//ppneviiieeee net http://wwwweeeeeeteoooooreeeeaee
Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. ArXiv preprint, 2aw/2302.14045, 2023. URL https://
arxiv. org/ abs/ / 2202222222222 14045.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http:
//axxiv. wrl.vorr//aa www.orgg/bbsbssoee abs//bss1112 6114.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. ArXiv preprint, hbs/2305.17216 2023. URL https: htt//aaivvv.rgg///ee /bss////////// 2305. 17216.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1 2012. URL https: htt////aaaaanltoooaeaeeeea wrg/D18-2000000 -2122..
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Jan Zhu. Multi-conceptt customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1931-1941, 2023.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tused Shahab Kamali, Stefan Popov, Matteo Malloei, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. 1JCV, 2020.
Dongxu Li, Junnan Li, and Steven CH Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. ArXiv preprint, abs/2305. 14720, 2023a. URL orgo alos/ / 22335 14720.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. ArXiv preprint, abs/2301. 2023b. URL https: http///axivvv/. org/ abs 231.2231..000 12597.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
11

---

Published as a conference paper at ICLR 2024
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov, RoBERTa: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692. 2019. URL https: arwiv..iii.vv//.ii http://ttwwww..aaa//aoo//oo 1907. 11692.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu, Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7086-7096, 2022.
Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprinl arXiv:2208. 11970, 2022.
Shuming Ma, Hongyu Wang, Shaohan Huang, Wenhui Wang. Zewen Chi, Li Dong, Alon Benhaim, Barun Patra, Vishrav Chaudhary, Xia Song, and Furu Wei. TorchScale: Transformers at scale. ArXiv preprint, abs/2211.131888 2022. URL https:t////cc arxiv. org/ wwggabs/////oooaooo//////./aaaa/ 2211. .31844.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh. Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 Proceedings of Machine Learning Research, pp. mlr. press v nicholchha 22a. html.
PMLR, 2022. URL https: /tttttps//proceednn pptpe//di/ccceenn.......
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall, Dreamfusion: Text-to-3d using 2d diffusion. ArXiv preprint, abs/2209.14980 2022. URL htttps://arxiv..og////aa 2209.. 14988.
Can Qin, Ning Yu, Chen Xing, Shu Zhang, Zeyuan Chen, Stefano Ermon, Yun Fu, Caiming Xiong, and Ran Xu. Gluegen: Plug and play multi-modal encoders for x-to-image generation. ArXiv preprint, abs/2303.10056. 2023. URL https: //tp://aaaavv//aae/aaeeeeeaaaa.a www.rxwwwww.rra.eooooooo abs /
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021. URL http: http://pocceeeeeeeeeeeee mlr. press/v139/ radford21a html.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. ArXiv preprint, abs/2204.06125,, 2022. URL https: /ettps//arriie org/ alos
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. ArXiv preprint, abs/2208. 12242, 2022. URL https: /ttw/viivvvvv..////////.aa//aaav org abs 2208 12242.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv preprint, abs/2205. 1487, 2022. URL https: wwt.voorg/////o.oeeeooooooosss /2205 11487.
12

---

Published as a conference paper at ICLR 2024
Christoph
Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of hiip-fileered 400 million image-text pairs. ArXiv preprint, abs/2111.021 2021. URL https:
//arxivvoorg abs / 2211 02114.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. ArXiv preprint, abs/2210.08402, 2022. URL htwps:ttp..........tt. : / rx org aas 221 0 08402.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2566 Melbourne, Australia, 2018. Association for Computational I. Linguistics. doi: 10.18653/v1/P18- 1238. URL https::tt///oo.
James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-lora. arXiv preprint wrXiv:2304.06000 2023.
Jascha Sohl-Dickstein Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 2256-2265. JoLR. 2015. URL http: /tt://rrooe pwoceedoceeeeeeeeeoooooooo ngs ml mulr.press//v31 html.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:tpp..... openrevier.r.e..eeomeeee...oe...
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang. Xiaosong Zhang, Yueze Wang. Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. ArXiv preprint,
2023, URL https: arxiv. org/ abs abs/2307 052222.
MN Team al. Introducing MPT-7B: A new standard for open-source, commercially usable 1lms, 2023.
Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to- image personalization. arXiv preprint S.ti::2305.01644 2023.
Hongyu Wang, Shuming Ma. Shaohan Huang. Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, and Furu Wei. Foundation transformers. ArXiv preprint, abs/2210.06423, 2022. URL https: wawpiviivvvv///.v//.aaa/a/aaaava org/ abss 2210.221.2222 2210.06423.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302.13848, 2023.
Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. Fastcom- poser: Tuning-free multi-subject image generation with localized attention. arXiv preprint arXiv:2305. 10431, 2023.
Lymin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
13

---

Published as a conference paper at ICLR 2024
Figure 8: Selected images from DreamBench.
A DETAIL TRAINING SETUP
Multimodal Language Modeling We use a batch size of 1.2 million tokens which is broken down as follows: 0.5 million tokens sourced from text corpora, 0.5 million tokens derived from image-caption pairs, and 0.2 million tokens from interleaved data sets. The MLLM is trained for 300,000 steps, corresponding to about 360 billion tokens total. We adopt the Adam W optimizer with 3 = (0.9, 0.98). Furthermore, we configure the weight decay at 0.01 and the dropout rate at 0.1. The learning rate set to escalate to 2c-4 during the initial 375 warm-up steps and decay linearly 0 for the rest of the training steps. For optimization stability, we initiate using Magneto. We use SentencePiece (Kudo& & Richardson, 2018) to tokenize the text. We preprocess the data in the "full-sentence" format (Liu al., 2019), where each input sequence is populated with complete sentences consecutively sampled from one or multiple documents.
Image Decoder Aligning The AlignerNet undergoes training using a batch size of 3,584 sentences for 300,000 steps, with maximum learning rate of 1e-3. This equates approximately I billion sentences overall. The remaining configurations remain consistent with the previous stage.
Instruction Tuning The MILM and AlignerNet are jointly trained with a batch size of 1,024 images, totaling approximately 200 million images over 200,000 steps. The learning rate peaks at 1e-3, The rest settings are the same as in the previous stage.
B IMAGES AND PROMPTS FOR DREAMBENCH EVALUATION
The input images selected for each entity for DreamBench evaluation are displayed in Figure As depicted in Table 3, We also slightly modified the original prompt make it align better with the
14

---

Published as a conference paper at ICLR 2024
training data. For the remaining prompt, simply remove the prefix "a". We observe the prefix will slightly affect the performance of image editing or customized generation. This might be attributed to the high frequency of captions starting with "a photo of"" produced by BLIP-2 in constructed training data. Given that the compositional instruction tuning data does not contain too much editing data when prompt starts with prefix like "a", the model often refrains from altering the appearance of the input image. This can be further refined by shifting the data paradigm or by fine-tuning the alignment.
Original Prompt
Modified Prompt
red (
J, red
a purple O
(1 purple
shiny ()
(). shiny
a wet ()
[), wet
cube shaped {
(), cube shaped
Table 3: Prompt Modification
C ADDITIONAL DETAILS ABOUT SCORE DISTILLATION INSTRUCTION TUNING
During the instruction tuning stage, still utilize the diffusion loss for model training. However, this loss can also be regarded as score distillation, and is effectively equivalent to optimizing the diffusion loss as the Score Distillation Sampling (SDS) (Poole et al., 2022) loss.
Our objective is to distill the learned score function from the Stable Diffusion U-Net into KosMOs GG. This process enables KOsMOs-G G to encode image features into embeddings that the Stable Diffusion model can understand for subject-driven generation. Essentially, this approach is like pre-training a generalized textual inversion Gal et al. (2022) model, with all conditions learnable by model-seeking. Consider the KOsMos-C G model, denoted as Ø, which takes an input x and produces an output C Φxx). Alongside this, we have the Diffusion U-Net, represented as 0. In our process, optimize the KOsMOs-G model é using the diffusion loss, while keeping the parameters of the Diffusion U-Net 9 frozen. The diffusion loss is expressed follows:
Lasff(ø) -Ezo V(0.))) (() c|13]
(8)
Consider the gradient of Caiss
V (oLasf(o) = -x w(t) (co(zi;C,t) c) dcolzz;
OC
(9)
Noise Residual
U-Net Jacobian
KosMos-G Jacobian
Following the approach in Dreamfusion Poole et al. (2022), we can simplify this equation by omitting certain terms, leading to the SDS loss for KosMos-G:
ac
V VoLSDS(o) = E. er we()) (Eo(2n;C,) -
(10)
06
As established in Dreamfusion Poole et al. (2022), optimizing the SDS loss is effectively equivalent to optimizing the diffusion loss when the U-Net 0 is frozen. From the perspective of score distillation, when using diffusion loss, the KL divergence defined by conditions and the pre-learned score function is equivalently minimized for distilling learned probability density in conditional image synthesis Dong et al. (2023); Luo (2022):
min (oifs(d) = Exo.1. e |DLL (g(2t - 1 20) po (Z.- Iz;; c))
(11)
15

---

Published as a conference paper at ICLR 2024
photo of a dog
beach
wearing sunglasses
An oil painting portrait of in the swit of highly dotailed, masterpiece
dog
in the style of
selfee
wearing sunglasser
standing in front of
highly detaited, best quality
cartoon man
standing in front car
the beach
in the style of
masterpiece
Figure 9: Additional examples under challenging multi-image and text interleaving scenario. These cases show that with KOsMOS-G, G, users prompt Stable Diffusion Rombach et al. (2022) by approaching all image imputs as a "foreign language".
D ADDITIONAL EXAMPLES
In Figure 9, we present cases with more diverse and complex multi-image (3 4) and text inter- leaving scenarios. These examples demonstrate KOsMos-Gi G's robust performance handling these challenging cases.
16


Reviews:
[{"review_id": "NIU5SmWmPy", "reviewer": "Reviewer_hrQd", "summary": "Kosmos-G aligns the outputs of MLLM to the embedding space of CLIP text encoder, which can be fed into Stable Diffusion model for image generation with context of any form.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The KOSMOS-G's ability to achieve zero-shot multi-entity subject-driven generation is notable. The method addresses an underexplored area in image generation by focusing on generalized vision-language inputs and multiple images, the method leverage existing advancements in both multimodal language models and image generation. \n\n2. By the alignment of the output space of MLLMs with CLIP and Score distillation instruction tuning, KOSMOS-G can achieve subject-driven generation and image editing without any training on diffusion models, highlighting its potential for integration into different frameworks.", "weaknesses": "1. The paper repeatedly mentions KOSMOS-G's ability to master zero-shot multi-entity generation and handle interleaved image-text input. However, the practical cases presented in the paper seem to focus primarily on image editing capabilities. Look forward to showing more cases with complex and rich scenarios to further illustrate the capabilities of the model. : 1）the paper only demonstrates cases with a maximum of two images, failing to showcases with more than two images as inputs.  2) the paper predominantly showcases and evaluates image-text-image input scenarios, leaving more diverse multi-image and text interleaving cases unexplored. \n\n2. Section 2.3 discusses the \"Score distillation instruction tuning\" technique, but the description lacks clarity. The paper should provide a more precise definition of the entities involved in calculating the KL divergence, along with any specific mathematical formulas or equations for better understanding.  In addition, is KL divergence loss necessary? Is it feasible to directly apply diffusion model's loss for training?\n\n3. The paper highlights the exceptional subject-driven generation capabilities of KOSMOS-G, particularly when not training the diffusion model.   I would like to ask if the authors have explored the possibility of further enhancing subject-driven generation, like training the diffusion model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8HHYBUN6YG", "reviewer": "Reviewer_VSX4", "summary": "This paper explores image generation from generalized vision-language inputs, especially involving multiple images. Named KOSMOS-G, a model that leverages the advanced perception capabilities of MLLMs, and aligns the output space of MLLM with CLIP using the textual modality as an anchor and performs compositional instruction tuning on curated data.", "questions": "see the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The problem of image generation conditioning on generalized vision-language inputs is an interesting problem, and the proposed approach seems to show some promising results.\nThe idea of aligning KOSMOS-G Space with the CLIP-T Space and then directly leveraging the stable diffusion models seems a valid approach.\nThe qualitative results show some good capabilities of the proposed method.", "weaknesses": "The ablation study seems not very comprehensive, for example, if the goal is to align the two representation spaces, there should be other options to achieve the alignment design, so why is the current AlignerNet design the best, maybe more justification and ablation study are needed here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PNPZEtkquG", "reviewer": "Reviewer_J1rb", "summary": "The paper proposes a framework that combines MLLM and SD to perform image generation/editing with multimodal input. To better bridge the MLLM output space and SD input space, AlignerNet is introduced for feature alignment. Additionally, a large-scale object compositional image generation data is collected and used for training.", "questions": "1. When constructing the compositional generation dataset, what if multiple objects of the same class exist in the same image? Would the corresponding segmentation mask cover multiple instances in the same mask?\n\n-------------- After rebuttal ----------------\nThank the authors for the last-minute efforts. I have raised the rating to 6. Please stick to this new manuscript and the new title in the camera-ready version if accepted, and further improvement in corresponding writing is also encouraged.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea of bridging MLLM and SD for versatile image generation is interesting. MLLM naturally can accept both image and text input, which can provide a more diverse signal to the image generation module and therefore enable new applications.\n2. The newly collected compositional image generation dataset should be useful to the community.", "weaknesses": "1. I don't see much novelty from AlignerNet.  Compared with GlueNet, AlignerNet merely replaces the MLP with encoder-decoder Transformers but they have the same loss and the same domains (both aligning text embedding). AlignerNet is useful from the experiments, but not novel IMO.\n2. Since the training data includes the image editing dataset from InstructPix2Pix. A comparison between previous works on image editing benchmarks should also be conducted. Similarly how is kosmos-g compared with GILL in visual storytelling?\n3. In AlignerNet, both MSE and REC losses are used. However, no ablation is done about those two losses. \n4. In Tab2, it seems the E2E Fine-tuning fails. However, recent works such as BLIP-Diffusion, EMU, and MGIE can successfully connect MLLM with SD via E2E fine-tuning without any specific alignment. Why Kosmos-G's behavior is different from others and relies on additional alignment?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)