Generating Images with Multimodal Language Models
Authors: Jing Yu Koh
Daniel Fried
Ruslan Salakhutdinov
Venue: NeurIPS 2023 poster (2023)
Keywords: multimodal
vision-and-language
language models
TL;DR: We fuse a frozen LLM with pre-trained visual models to demonstrate a wide suite of multimodal abilities, including stronger text-to-image generation on tasks with longer and more complex language.

Abstract: We propose a method to fuse frozen text-only large language models (LLMs) with pre-trained image encoder and decoder models, by mapping between their embedding spaces. Our model demonstrates a wide suite of multimodal capabilities: image retrieval, novel image generation, and multimodal dialogue. Ours is the first approach capable of conditioning on arbitrarily interleaved image and text inputs to generate coherent image (and text) outputs. To achieve strong performance on image generation, we propose an efficient mapping network to ground the LLM to an off-the-shelf text-to-image generation model. This mapping network translates hidden representations of text into the embedding space of the visual models, enabling us to leverage the strong text representations of the LLM for visual outputs. Our approach outperforms baseline generation models on tasks with longer and more complex language. In addition to novel image generation, our model is also capable of image retrieval from a prespecified dataset, and decides whether to retrieve or generate at inference time. This is done with a learnt decision module which conditions on the hidden representations of the LLM. Our model exhibits a wider range of capabilities compared to prior multimodal language models. It can process image-and-text inputs, and produce retrieved images, generated images, and generated text — outperforming non-LLM based generation models across several text-to-image tasks that measure context dependence.

Generating Images with Multimodal Language Models
Jing Yu Koh Carnegie Mellon University jingyukecs. cmu.em edu
Daniel Fried Carnegie Mellon University dfried@cs. cmu. edu
Ruslan Salakhutdinov Carnegie Mellon University rsalakhuucs. .ow.uuuussccuuu...........u .mu
Abstract
We propose method to fuse frozen text-only large language models (LLMs) with pre-trained image encoder and decoder models, by mapping between their embedding spaces. Our model demonstrates wide suite of multimodal capabilities: image retrieval, novel image generation. and multimodal dialogue. Ours is the first approach capable of conditioning arbitrarily interleaved image and text inputs to generate coherent image (and text) outputs. To achieve strong performance propose efficient mapping network to ground the LLM to an off-the-shelf text-to-image generation model. This mapping network translates hidden representations of text into the embedding space of the visual models, enabling us to leverage the strong text representations of the LLM for visual outputs. Our approach outperforms baseline generation models on tasks with longer and more complex language. In addition to novel image generation, our model is also capable of image retrieval from prespecified dataset, and decides whether retrieve or generate at inference time. This is done with learnt decision module which conditions on the hidden representations of the LLM. Our model exhibits a wider range of capabilities compared to prior multimodal language models. It can process image-and-ted inputs, and produce retrieved images, generated images, and generated text outperforming non-LLM based generation models across several text-to-image tasks that measure context dependence.
image generation,
Introduction
Autoregressive language models (LMs) and large language models (LLMs) trained text corpora have shown impressive abilities to efficiently adapt to other modalities. Prior work showcased the effectiveness of grounding text-only LMs to images for vision-and- language tasks [56 4 29 33
31 35), to embodied settings for robotics 3 18), offline reinforcement learning [48]. and more.
These methods typically keep most of the LLM weights frozen. This allows them leverage the capabilities that the LLM learns during large scale text-only pretraining, such as the ability to learn from in-context examples [ more effectively process longer context, and condition inputs more strongly.
In this work, we tackle the task of extending multimodal language models to generate novel images. Our approach, Generating Images with Large Language Models (GILL), is capable of processing arbitrarily interleaved image-and-text inputs to generate text, retrieve images, and generate novel images (Fig. [T. Our findings show that it possible to efficiently map the output embedding space of frozen text-only LLM that of frozen generation model (in this work, Stable Diffusion (49]) despite both models using entirely different text encoders. We achieve this by finetuning a small number of parameters image-caption pairs [52], in contrast to other methods which require interleaved image-text data [ (2. Our approach is computationally efficient and does not require running the image generation model training time. To achieve strong image generation performance, we propose efficient architectural changes to learn the LLM-tt- -oeeeraggnnotioon mapping effectively with the GaLpMrpper module. GIL.Mapper is lightweight Transformer [57] conditioned special
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

User prompts
Retrieved
Generated
thinking animal,
brown am thinking of and furry, and has tall. What is it? aquamrll
thinking brown
and fumy, and has a tail What segumnl.
and furry, and has tall. What is aquiral.
animal thinking
animal thinking river.
animall thinking the
beaver.
Show
beaver
Blsstration beaver
an
beaver.
beswvr
Yie, what about pencil drawing of ona?
Yis, wher about a parci drawing ona?
Yes, what about a penci drewiing of one?
comic
Ours
FROMAGe
Stable Diffusion
Betrieval generation multimodal Decides when to retrieve or generate.
Betrieval only multimodil language model. Image outputs limited to the candidate retrieval
Generation only text-to-lmage model. Less sensitive to longer text inputs (such dialogue).
Figure 1:
Our model is capable of generating text, retrieving images, generating novel images, and interleaving results into coherent multimodal dialogue.
learnt text tokens. We train by minimizing the le distance between its outputs and the outputs of the text encoder of text-to-ine text-to-image generation model. This distillation training allows us to use the image decoder of the text-to-image model at inference time. Despite its simplicity, we show that this allows us to outperform the baseline text-to-image generation model on several tasks that measure language context dependence. Finally, to decide whether to produce retrieved image or generated one at inference time, train a decision model that outputs decision conditioned on the LM hidden representations. This allows us to both generate and retrieve in output sequences, as shown Fig.
Our experimental results demonstrate that GILL is more effective than Stable Diffusion at processing longer-form text, including dialogue and discourse. We show on catllooue-codditione image gener- ation that GILL can outperform non-LLM based generation models, and benefit from multimodal context: generating images that match text better than the backbone generation models that we distill from. In addition, GILL can process arbitrarily interleaved image-text inputs, unlike typical text-to-image models which only process text. GILL is the first model capable of outputting retrieved images, novel images, and text interleaving these for coherent multimodal dialogue generation
Related Work
Multimodal Language Models Several prior works have developed multimodal language models which process image and text inputs to generate text outputs. Frozen (56) showed that it is possible to finetune visual encoder to map images into the hidden space of a text-only LLM, and that this exhibits compelling few-shot, captioning, and question answering abilities. Other methods improve upon this approach by introducing adapters [19] scaling up model and data sizes
333, finetuning on instructions (35 and training unified models on multi-tass objectives [36] 63
improving the visual encoder 4
CM3
2 62 trained multimodal L.Ms on HTML webpages
consisting of interleaved images and text. Many state-of-the-att models also require significant
computational resources to train. For example, Flamingo 4 trained on 1535 TPUs for 15 days, while RA-CM3 [62] use 256 GPUs for days. In contrast, our efficient adaptation method is trained on GPUs for days. The most similar work to our approach FROMAGe (3I), which trains a multimodal language model capable of processing arbitrarily interleaved image and text inputs to
OUr code and pretrained models publicly released at tttps://ggithbl www.hhubb.oommmeeee hoo/koojjai ingyu/g: 1lll

---

Generated Caption (neet tokan prediction)
Frozen Model
GILLNopper
Uieer Layer
Locs
Input Embeddings [ses_ien, 4295)
<ieg>
European
short
cat stating fnn basket
An Furspean thair in soven basket
Oulput Embeddings
(s00, Tor,
European
[3MG(r]
[IN [XNGE+3]
Visual Ercoder
Caption Inpurt
Irque Coption
GILLMapper
Europeain
InfoNCE
5D Test Precsiee
Cross Entropy
in basket
Image-Text Rebrieval
Image Imput
Image
Image and Caption Imputs
Capttor
Image Generation
Leariing
Images
Learning to Produce Images
Figure 2: GILL model architecture overview. It is trained with a captioning loss to learn process images (left), and losses for image retrieval and image generation to learn to produce images (right).
generate text interleaved with retrieved images. While FROMAGe can only retrieve images in their outputs, GILL is capable of both image retrieval and image generation, which allows to outperform retrieval-only models when they are limited by their candidate retrieval set (Fig.
Large Language Models Our work leverages recent advances in Transformer based [57 LLMs. When trained at large enough scale. LLMs exhibit compelling properties, such as the ability to learn from few-shot in-context examples Our approach also builds upon recent efforts on open sourced L.M weights [69 5.
p [ and generate and process long text inputs 61 59 53
Tmxt-to-Image Generation Text-to-image generation is the task of synthesizing a realistic image conditioned on natural language descriptions. 47 was one of the first to tackle this with conditional
GAN [23] Later work improved upon this by introducing multi-stage models [67] attention mecha- nisms 160) and contrastive methods 773 [66.. Several recent papers also formulate the text-to-imant ti-iga generation task as a sequence modeling problem [45] [7 [3], training large Transformer [57] models discretized image tokens [46]. [20 655 improved upon this approach by introducing stronger image
quantizers and scaling model parameters. Several recent methods [38 44 49] apply diffusion
models (26) to improve generated image quality. [50 65] scale up text encoder models achieve
significant gains generating relevant images. contrast with computationally intensive methods that train end-to-end, GILL does not require running the image generation model during training.
3 Method
We efficiently adapt pretrained autoregressive language model of text, process image and text inputs and produce image and text outputs. Most of the model weights (including those of the base LLM and image generator) are kept frozen, and we finetune small number of parameters on image caption data (Fig. 2 to achieve a wide range of capabilities (Fig. 5). There are several challenges that need to be resolved. The model needs to learn to process image-and-ter content (Sec. 3.1 It also needs to learn to produce images (cither retrieved or generated), and determine whether to produce text images at each step (Sec. 3.2). Finally, whenever image is produced, the model needs to decide whether image retrieval (from a candidate set) or generation is more appropriate (Sec. 3.3
3.1 Learning to Process Images
Given image and its text caption y (tokenized ($1....,87) ST)), goal is to adapt frozen LLM to enable it to complete any sequence of arbitrarily interleaved image and text inputs. For example, inputs for the Visual Storytelling dataset (88 consist of 5 images and 5 text descriptions, interleaved
a manner such as (x1, y/, Fs, ys). We follow prior work [56] 19 35 31 in learning translation parameters that map from image features text embedding space.
We first extract visual embeddings (x(x) E IRd with pretrained visual backbone (its weights & and L.LM weights are kept frozen). We learn linear mapping Weap E dd which maps (x))) into sequence of &o e-dimensional vectors that as inputs to the LLM (Fig. 2 left), where is the

---

Model Output Embeddings
Kingo
Hon
atouts
think
[INGEr33
Linear
Decttion Model
Visual Encoder
Tokericer
Predicted Embeddings (hty 7688
GILLMappar
How should display these the farner's market?
Similarity
SD Imaga
Decoder
image
Transformer
Transformer
Image #!
Caption +1
Image-Text Imputs
Retrieved Generated
[ING] Embeddinge
Learnt Queries (k. 512)
think they look best when they with little space
(f) 4096)
Figure 4: GIL.LMapeer model architecture. It conditioned on the hidden [IMG] represen- tations and a sequence learnt query embedding vectors.
Figure 3: Inference time procedure for GILLL The model takes in image and text inputs, and produces text interleaved with image embeddings. After deciding whether retrieve or generate for a particular set of tokens, returns the appropriate image outputs,
LLM input embedding dimension. We train W cap image-caption pairs (details in Sec. 3.4 3.4..b by minimizing the negative log-likelihoon loss the token sequence (s1,..., 8T);
(z, y)
log (s (x( W cap $1
St-1)
(1)
Intuitively. this objective trains mapping W Weap cap that allows to translate images into embedding vectors in the token embedding space of the LLM (illustrated in Fig. left).
3.2 Learning Produce Images
In order to enable the model to produce image outputs, we add special [IMG] tokens to the vocabulary of the LLM,, similar to [71 which introduce special tokens correspond images that should be output by the model. The hidden states that the LLM produces for these tokens will be used to retrieve generate images. While 31 use single token for their image retrieval model, observed in experiments that image generation requires much more finegrained textual information (Sec. (5). In order to improve the expressivity of the frozen LLM for novel image generation, generalize r tokens [IMG1],.... [IMG(r}] for representing visual outputs.
Concretely, add trainable matrix Eing Rexe e to the embedding matrix of the frozen LLM, which represents the r [IMG] token embeddings. We wish to train the model to learn when it should produce [IMG] tokens. This is done by minimizing the negative log- likeelhoooo of producing the first [IMG] token conditioned on previously generated tokens:
(y)) = log ((OuE [IMG1] $1 s1,..., st)
(2)
The LLM weights 0 are kept frozen, and only update Eang- During inference, always generate the [IMG2], [IMG[r}] tokens whenever the first [IMG1] token is produced. During training, the [IMG] tokens appended each caption (Fig. 2 The LLM hidden states of the [IMG] tokens are used for image retrieval and generation, as described in the following sections.
Novel Image Generation In order for LLM to produce image outputs, the [IMG] tokens need to be mapped into a semantically meaningful region of the input space of an image generation model Ge (such as that of the Stable Diffusion [49] image decoder). In initial experiments,
found that training a simple linear mapping such as those used in previous work on retrieval [3I] was insufficient, and that such model was unable to handle more complex prompts (see Sec. 5 for analysis). Hence, we propose GILLMapper (Fig. 4. lightweight 4-lyye encoder-decoder transformer model with trainable weights The GILLMapper module for conditions ( (v, [IMG]) (the [IMG] representations from the last hidden layer of the L.LM) and learnt query embeddings (qI...., qr.) Lxmm (where the maximum input sequence length of the text-to-iman -ottaee generation backbone G.))

---

The purpose of introducing learnable query embeddings is to enable Gappe to extract sequences of features from the LLM [IMG] hidden states. This is similar to the queries introduced in DETR [10] for object detection and BLIP-2 33 for extracting image features. We optimize the GILL trainable weights (?): ........ qT. and w) by minimizing the MSE loss of the Mappee model outputs against the embeddings produced by the text encoder (Tx) of frozen text-to-image generation model:
(,(() = Il5 (Ruq (IMG(131), "u.u....,, (e (v, [IMG(x]]], (1 (t....-..))) q1) (v) (3)
This is essentially distilling from TV to learn a valid mapping from the output representations of our frozen LLM to the input space of Gv. Note that this does not require G during training, so we can precompute (3) ahead of time, making training highly efficient. During inference, when [IMG] tokens are generated, we can synthesize an image by applying GILLMapper and the decoder Gs:
Generated Image- Gau (IQ(OuBm (v, (IMG(1H]),.... hqqu (v, [IMG<x)]] ....... .q.))
where h (mm))m)) (y. [IMG((i]] represents the hidden states from the last hidden layer of the modified LLM corresponding to the 1th [IMG] token. The learnt query embeddings (91....,9L) are part of the GILLMapper model weights, and are hence kept fixed during inference.
Image Retrieval Similar to we learn a linear mapping Wa E Rexp that maps the first token ([IMG1] to a p-dimensional vector. We also learn a linear mapping Wiz E R that map the pooled visual output of the image encoder (a) p-dimensional space. These represent image and text embeddings, and we train the model by minimizing the InfoNCE loss 39
vx) = -olog log explsim(x,, Yu Wox)/T)
exp(sim(x,, ys, y, Wn))//r
Σラ exp(sim(xx, N.. Wa)/T) log
(4)
j== exp(sim(xx,.) Win))/r)
where the similarity is computed as
sim y, W) = (W (z)))(z)) (WT h (v. [IMG1]))
[WTx)())] WT (ou (v, [IMG1]]
During inference, we follow standard procedure 43 in retrieving the image with the highest cosine similarity (between image embeddings and the [IMG] tokens) from candidate set of images.
3.3 Deciding to Generate or Retrieve
While learning to produce [IMG] tokens allows us to decide when to interleave images in text the task of deciding whether retrieve or generate from [IMG] tokens remains. Intuitively, for a given prompt, we would like to retrieve when there is a strong match from our set of candidate images, and generate otherwise. In order to evaluate this, we collect human annotations on PartiPrompts (P2) 655. collection of prompts used to benchmark image generation models. P2 contains some prompts that are well-represented by naturally occurring images, but others that are unlikely to occur in natural image sets, making it a test of generative models. For each of the 1,632 examples in P2, we generate an image with the text-to-image generation model G and use the CLIP ViT-L 43 model to retrieve the top ranked image from CC3M [52] according to the cosine similarity of image embeddings ug-
We have independent human annotators (details in the appendix) select which of the two images for each prompt, retrieved or generated, is better matched to the prompt. We labeled the examples where the generated image was selected as 'gen' (indicating prompts which we should generate an image for) and 'ret' for prompts that should have an image retrieved. We extract the most confident set of these annotations (retaining roughly 900 examples with an inter-annotator agreement of least 4/5), and split them into a 67% train (600) and 33% test (300) split. We use this to train linear classifier on the LLM [IMG] hidden states as a decision model for deciding when to retrieve or generate (more details and baselines are provided in the appendix). Although these annotations of retrieving versus generating are somewhat model dependent, we believe that this data still a valuable metric during model development. We will release annotations to encourage future work in this space.
5

---

3.4 Data and Implementation Details
The final training objective for a batch of image-text pairs y) is the sum of the captioning loss lc (Eg. D. image token prediction loss 1p (Eq. ( generation loss ig (Eq. 3) and retrieval loss 4, (Eq.
1 N
(L(x,,))) + (yy:)) + (xx,).))
(5)
Win 21 W Ei Bing Tii
The decision model is trained separately after convergence of the other components. The multitask loss (Eq. 5) trains GILL to process images (le), produce [IMG] tokens (lp), generate images (ly), and retrieve images (t,). This enables it to generalize a wide range of rision-and-langune tasks.
We train on Conceptual Captions (CC3M) [52], which consists of 3.3M image-text pairs. Following [I], we pack two random examples together during training with probability 0.5 (i.e., 50% of the time, the input is a single image and caption example, while the other 50% of the time the input consists of sequence consisting of two interleaved images and captions). We use the OPT-6.7B (69
model as the LLM backbone (which produce hidden states he with embedding dim = 4096). For
the visual model used to extract features Vo for captioning and retrieval, we use the CLIP [43] ViT-L model. For our text-to-image generation backbone Gs,, we use the Stable Diffusion
49
v1.5 model
(with L = 77 input vectors) We use k = visual tokens, and r 8 learnt [IMG] tokens. We set the GILLMapper query embedding dimension
512. For retrieval, we use an embedding dimension p = 256. All pretrained model weights are kept frozen, and we only train the linear layers Wi2. W 121 W cap- the [IMG] embedding matrix Eing. and the GILLMapper parameters w and query vectors QI In total, there are 50M trainable parameters, significantly fewer than in the frozen LLM and visual models (which total approximately 8B parameters). We use bfloat 16 precision [1] and optimize using Adam 30 (31 = 0.9, 3g 0.95) with a learning rate of 0.001. We train with a batch size of 200 for 20K iterations, which takes days on 2 A6000 GPUs. We follow [II and concatenate captions to encourage the model to attend to relevant images within image-text sequence.
4 Experiments
GILL is the first multimodal language model capable of conditioning on image-and-text inputs to generate meaningful images interleaved with text. Hence, our experiments primarily focus on evaluating its ability to produce novel images (Sec. [.1). Our results show that GILL improves over Stable Diffusion [49] on tasks that require processing long-form text such as dialogue and discourse. We also benchmark the performance of models in deciding whether to retrieve or generate (see appendix). GILL is capable of generating text, retrieving images, and generating images. Despite being more general than prior work [56] 31]. we find that GILL performs comparably to or better than existing multimodal LMs on contextual image retrieval and text generation tasks (see Sec. 5
4.1 Contextual Image Generation
To test the ability of our model against baseline methods for novel image generation, we run experiments on the VIST (28) and VisDial [16] datasets. These are the same datasets used in prior work 31 for benchmarking image retrieval conditioned on multimodal text-and-image context.
The focus of our evaluation is on the ability of generative models to handle complex language descriptions. Hence, we compute metries which measure the relevance of the generated image content. We evaluate models with two metrics:
Evaluation Metrics
1. CLIP Similarity: We use the CLIP 43 ViT-L image encoder to produce pooled repre- sentations of generated images and the corresponding real images, and report their cosine similarity. A higher score indicates that generated image is more similar to the real image.
2. Learned Perceptual Image Patch Similarity (LPIPS): LPIPS [68] evaluates the distance between image patches. We measure LPIPS between real and generated images. A lower value indicates that two images are closer in perceptual space (i.e., more similar), while higher value indicates that two images are more dissimilar.
"These models were selected for their strong performance and open source availability, but in principle our approach can be applied with any other pretrained models with similar function calls.

---

What would pizza with
What would pizza with Daran lust
but in green"
FLET
Image Input
(retrieved)
(generated)
strawberics
but as poper- craff"
Image- Input
(rebrieved)
(genersted]
Add acma dkone
Add mome diCAs
"From on
Lul
ew
Image Text Input
FROMAGe (retrieved)
Ours (generated]
PROMAGe (rstrieved]
(generatied)
Comparison Against FROMAGe
model comporiiee mulimodal produce image outputs. autperform
image
Oncce whrite
2 wake up
can
Trisnte
No had
After ctined
vacation tiis
Tyri te
and antl fresh
pi llyling Espetter
and rest
resel
Image Inputs
Stable Diffusion
Groundtruth
greas this voung
[eate] picture
You can see norningg the takiing Surie the
stattu
wonstigg
Host
clown dver ststuu
before
breskffaat
Tocal- loves his
Image and Teat Inputa
Stable Diftusion
Ours
Groundtruth
Visual Storytelling
Our model can condition on interisnued image. and sext inputs to genarata more relevant images companed to non-LIM based Iext- pontttt--imme tosiiooiime---om genaration models.
I'm looking for some idean fora new What do you
oustamised cako dosign my coug coughter's bithdity
good
I'm rest sume whal you
meen by
How
publicton Theee the mariee??
You can and vegetazles your ramen cootlles, hut you stou cometul not overtio it
but t'm not uu whhrr style uould look
sunfhoowr
would suggeet stmpple gn
with unicom intimaiioal
piction of the cupicaiies
That e beautffL Show one would work complemerary
Wher aew some vegalatiee cann setd
Broczor carrots, green
cholces
Multimodal Dialogue
Our model generate multimodal dialogue, weaving together text, retrivved Images, and generated Images.
User prompts
Retrieved
Generated
Figure Qualitative results over various input and output modalities. GILL is able to process contextual multimodal cues to retrieve and generate appropriate image and text outputs.

---

Table Results on contextual image generation on VIST [28
(averaged over random seeds). Our model can process longer (possibly multimodel) inputs to outperform baseline models.
CLII Similarity (?)
LPIPS (1)
Model
caption
captions
caps, 4 images
caption
5 captions
caps, 4 images
GLIDE 18
0.582
0.591
0.753
0.745
Dillusion
1:0.0007
0.0006
10.0003
0:704 0.0004
GILL (ours)
0,581 +-000005
0.612 +000011
0.641 +0.0011
0,702 +0.0004
0.696 +0.0008
0.693 +0.0008
Table 2: Results contextual image generation VisDial [6 (averaged over 5 random seeds).
Our model can process longer sequences of dialogue-like text to generate more relevant images.
CLIP Similarity (1)
LPTSS (L)
Model
round
5 rounds
10 rounds
round
rounds
10 rounds
GLIDE 18
0.562
0.595
0.587
0,800
0.794
0.799
Stable Diftusion 49
0.552 +0 0.10
0.629 #0.0015
0.622 000001
0.742 +0.0010
0.722 #0.0012
0.723 +0.0008
GILL (ours)
0.528 +0.0014
0.621 +0.0009
0.645 +0.0010
0.742 +0.0022
0.718 =0.0028
0.714 +0,0006
Generating from Visual Stories VIST 28 is a dataset for sequential vision-and-language tasks, with examples of sequences of images and text that constitute a story, shown in Fig. Similar to B1, we test the models on generating the last image in the sequence, conditioned on different inputs:
1. 1 caption: Input consists of the last text description. This is similar to standard text-to- image generation, where model conditions on a single caption to generate an image.
2. 5 captions: Input consists of all text from the entire story sequence. This tests the ability of models to process longer and temporally dependent text descriptions.
3. 5 captions, 4 images: Lastly, we test models with inputs of all images and texts preceding the last image (i.e., sequenced as "<textt><img1>... Singgl...<<txxx>>>>>> <oattt<<<mm>>taaa>> tests the ability of models to effectively process multimodal context in image generation. A novel feature of GILL its ability to process interleaved image-text inputs, which most existing text-to-image generation models are unable to handle.
This
We report results on VIST in Tab. T comparing GILL against text-to-image generation baselines (including Stable Diffusion (SD) 49 which we use as our generation backbone G.)). With single story caption input to both models, the performance comparable, with SD achieving a slightly better CLIP Similarity score. and both models achieving similar LPIPS. However, when all 5 story captions are provided as input, our model outperforms SD, improving CLIP Similarity from 0.598 to 0.612, and LPIPS from 0.704 to 0.696. Interestingly, when further provided with the full multimodal context (the preceding captions and 4 images), our model improves substantially, attaining a CLIP Similarity of 0.641 and LPIPS of 0.693. In contrast, SD is unable to handle interleaved image-text inputs without significant modifications. We also show several qualitative examples in Fig. 5 We find that GILL is generally more sensitive to input context compared to SD, GILL can also condition on image inputs, enabling to use visual context to produce more relevant images.
We highlight that both models use the same image generation backbone, and the primary difference their text encoders. GILL is able better handle long text inputs and multimodal context, which we attribute to the stronger LLM encoder coupled with our GILLMapper model.
Generating from Visual Dialogue We also test our model on the VisDial 16 dataset. VisDial examples contain a sequence of question and answer (Q&A) pairs about a particular image, simulating dialogue between two people who are discussing Q&A dialogue pairs. Similar to VIST, we evaluate the ability of models to accurately synthesize the image being described, provided with increasing amounts of the Q&A dialogue context as input. This experiment tests the ability of our approach to (1) generalize to dialogue-like text (as our approach is only finetuned on image caption data), and (2) process long text sequences.
image. Examples contain up to 10 rounds of
Our results are presented in Tab. 2 Similar to the VIST evaluations, we find that with shorter length inputs, SD outperforms our model. However, when the input context is increased, our model gradually improves, and can synthesize images that are more similar to the groundtruth image. When the full 10 rounds of dialogue are provided, GILL significantly outperforms SD, improving over both CLIP
8

---

Table 3: Image generation performance CC3M
122 and
Table 4: GILL image generation sults on CC3M [52 with different
VIST 28
with different text mapping networks.
number of image tokens (r).
CC3M
VIST
Model
FID (1)
CLIP Sim (t)
CC3M
VIST
Stable Diffusion
49
13.94
0.598
FID (J)
CLIP Sim (1)
Ours + Linear
15.50
0,500
15.93
0,631
Ours + 3-layer MLP
15.33
0,502
15.32
0.629
Ours + Transformer Encuder
16.30
0.605
15.32
0.642
Ours Lapper
15.31
0.641
15.31
0.641
Performance With Increasing Context VIST
CLIP Similarty arty
0.64
Captions only
Table 5: Contextual image retrieval on
0.62
Captions Images
VIST (5 captions, 4 images as input).
'indicates results from [1..
0.58
0.66
VIST Recall @ k: (?)
cap
caP
CAP
cap
5 uap
2 cap. kmg
3 cap. 2 mg
cap, img
ing
Model
R@5
CLIP VIT-L
43
8.8
22.3
29.8
FROMAGe 31
18.2
42.7
51.8
Figure 6: Performance of GILL VIST generation.
GILL (Ours)
20.3
45.0
53.7
Similarity (0.622 to 0.645) and LPIPS (0.723 to 0.714). These results further highlight the efficacy of our model on handling long dialogue-like text inputs.
4.2 Qualitative Results
Finally, one of the more compelling applications GILL perhaps its ability to generalize many different tasks, due to the LLM pretraining and freezing. We showcase several of these capabilities in Fig. many examples,
observed that GILL able to outperform retrieval models such as FROMAGe (3I] on examples where FROMAGe is unable to retrieve relevant images. GILL is also generally more sensitive input context compared Stable Diffusion 49,, and can condition on image inputs, in addition to text, to generate more visually and semantically relevant image outputs.
5 Analysis
Contextual Image Retrieval In addition to generation, GILL is capable of image retrieval condi- tioned on image-text inputs. We run GILL on the VIST retrieval evaluation from [31] We find that GILL performs comparably better compared prior approaches (Tab. This shows that that the image generation objective does not cause image retrieval performance to deteriorate.
The Effect of Context GILL leverages LLM backbone, which allows it to inherit some of the LLM's capabilities, such as improved sensitivity to long imputs. Fig. 6 shows that the performance of GILL generally improves with increasing input contexts on VIST [28]. In particular, when 2 captions and image are provided as context, the model significantly outperforms the model with text-only captions, highlighting the value of multimodal context over unimodal context.
Generation-Only Objective We investigate the effect of removing the retrieval loss (Eq. from the training objective. VIST captions, 4 images), this ablated model achieves CLIP similarity of 0.636 and LPIPS of 0.694, which are comparable to scores of the original model (0.641 and 0.693 respectively). This suggests that the retrieval loss not necessary for strong performance, although such model would only be able to generate images and text and not retrieve images. These results also suggest that GILII
bottlenecked by including the retrieval objective, and that has sufficient capacity to perform both generation and retrieval.

---

GILLMapper Module As described in Sec. 3.2 we propose the GILLMapper module, embeddings and q learnt embedding vectors. The output maps the L.M embeddings into the input space of text-to-image generation model, en- abling image synthesis. We run several baselines to compare effectiiveness, comparing our proposed model against (1) a linear layer, (2) multilayer perceptron (MLP) with LeakyReL.U activations, and (3) a 4-layer bidirectional transformer encoder. All models are conditioned on the r [IMG] token embeddings from the LLM. Our results are presented in Tab, 3 GILLMapper is substantially better than these baseline models at learning the mapping from the frozen LLM to the Stable Diffusion generation model, measured by Fréchet Inception Distance (FID) 225 on the CC3M validation set (which is a measure of image realism), and CLIP Similarity on VIST. On the VIST evaluation (which is out of distribution from CC3M), the other baselines perform significantly worse than GILLMapper, suggesting that they cannot generalize to longer sequences containing multiple images and texts.
lightweight transformer model that conditions on [MG]]
Number of [IMG] Tokens We experiment with varying the number of [IMG] tokens, r (Tab. As increases, generation generally improves, plateauing around r = 4. We observe that lower values of appear to result in worse results, as the inputs to GILLMapper are shorter and less expressive.
Conclusion
We proposed a method of mapping text-only LLMs to strong visual models. This enables them to learn to process arbitrarily interleaved image-and-text inputs, and output generated text, retrieved images, and generated images. We show that possible to efficiently learn mapping between the embeddings of a frozen pretrained LLM and frozen pretrained image generation model, and that doing so effectively boosts image generation for tasks that require stronger language context dependence. Finally, we also showcased several compelling qualitative results on variety of multimodal tasks. Our approach is modular, and can benefit from stronger LLMs or visual models released in the future. Scaling up the LLM backbone, image generation backbone, or visual processing model, are promising directions that will likely induce even stronger iision-and-languee and-language capabilities.
Acknowledgements
This work was partially supported by a gift from Cisco Systems, and by ONR N000142312368 and DARPA/AFRL FA87502321015. We thank Wendy Kua for help with the figures. We thank Jared Fernandez, Yutong He, Saujas Vaduguru, Yonatan Bisk, and our anonymous reviewers for feedback and helpful discussions on previous versions of this paper.
References
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint 1rXiv:1603.000 2016.
[2] Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201.00 2072000 2022.
[3] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do i can, not as say: Grounding language in robotic affordances. CoRI, 2023.
[4] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurlPS, 2022.
[5] Satanjeev Banerjee and Alon Lavie. METEOR:
An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACI. Workshop on Intrinsc and Extrinsic Evaluation Measures for Machine Translation and/or Sunmarization, 2005.
10

---

[6] Emily M Bender, Timnit Gebru, Angelina McMiolan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623, 2021.
[7] Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R Gormley. Unlimiformer: Long- range transformers with unlimited length input. arXiv preprint arXiv:2305.01625, 2023.
[8] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misog- yny, pornography, and malignant stereotypes. arXiv preprint arXiv:2/ 2021.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. NeurlPS, 2020.
[10] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[11] Stephanie CY Chan, Ishita Dasgupta, Junkyung Kim, Dharshan Kumaran, Andrew K Lampinen, and Felix Hill. Transformers generalize differently from information stored in context vs in weights. NeurIPS MemARI Workshop, 2022.
[12] Keshigeyan Chandrasegaran, Ngoe-Trung Tran, Alexander Binder, and Ngai-Man Cheung. Discovering transferable forensic features for cnn-generated images detection. In ECCV, 2022.
[13] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022.
[14] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval- augmented text-to-image generator. ICLR, 2023.
[15] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[16] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In CVPR, 2017.
[17] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. NeurIPS, 2021.
[18] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 2023.
[19] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. augmentation of generative models through adapter-based finetuning.
EMNLP. 2022.
[20] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[21] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. ECCV, 2022.
[22] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah Smith. Realtoxici- typrompts: Evaluating neural toxic degeneration in language models. EMNLP, 2020.
[23] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 2020.
[24] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
11

---

[25] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. NeurIPS, 2017.
[26] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabiliistcc models. NewrIPS, 2020.
[27] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. ICLR, 2020.
[28] Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In NAACL-HILT, 2016.
(29) Gabriel Ilharco, Rowan Zellers, Ali Farhadi, and Hannaneh Hajishirzi. Probing contextual language models for common ground with visual representations. NAACL, 2021.
[30] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2015.
[31] Jing Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal imputs and outputs. ICML, 2023.
[32] Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla. Illustrat- ing reinforcement learning from human feedback (rlhf). Hugging Face Blog, 2022. htpp://huggngghcc..obbiiii/
[33] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML, 2023.
[34] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
[35] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, arXiv preprint arXiv:2304.084855 2023.
[36] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language. and multi-modal tasks. arXiv preprint 2022.
[37] Xiyang Luo, Ruohan Zhan, Huiwen Chang, Feng Yang, and Peyman Milanfar. Distortion agnostic deep watermarking. In CVPR, 2020.
[38] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. ICML, 2022.
[39] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807 037488, 2018
[40] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurlPS, 2022.
[41] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In ACL, 2002.
[42] Can Qin, Ning Yu, Chen Xing, Shu Zhang, Zeyuan Chen, Stefano Ermon, Yun Fu, Caiming Xiong, and Ran Xu. Gluegen: Plug and play multi-modal encoders for x-to-image generation. arXiv preprint arXiv:2303. 10056, 2023.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
12

---

[44] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[45] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Iya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
[46] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. NeurIPS, 2019.
[47] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In ICML, 2016.
[48] Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? arXiv preprint wrXiv:2201.12122 2022.
[49] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[50] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022.
[51] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. L00on-400m: Open dataset of clip-filterdd 400 million image-text pairs. arXiv preprint arXiv:2/11.021/ 2021.
[52] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
[53] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1-28, 2022.
[54] Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint trXiv:2205.05131, 2022.
[55) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302.1397, 2023.
[56] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. NeurlPS, 2021.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
[58] Jason Wei, Maarten Bosma, Vincent Y Zhao. Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. ICLR, 2022.
[59] Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing trans- formers. ICLR, 2022.
[60] Tao Xu, Pengehuan Zhang, Qiuyuan Huang, Han Zhang, The Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text image generation with attentional generative adversarial networks. CVPR, 2018.
[61] Kevin Yang. Dan Klein, Nanyun Peng, and Yuandong Tian. Doc: Improving long story coherence with detailed outline control. arXiv preprint arXiv:2212.10077 2022.
[62] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Retrieval- -ugmentte multimodal language modeling. ICML, 2023.
13

---

[63] Haoxuan You, Mandy Guo, Zhecan Wang, Kai-Wei Chang, Jason Baldridge, and Jiahui Yu. Co- bit: A contrastive bi-directional image-text generation model. arXiv preprint 1rX4v:2303.13455,
2023.
[64] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022.
[65] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. TMLR, 2022.
[66] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. In CVPR, 2021.
[67] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas. Stackgan: Text photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
[68] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In CVPR, 2018.
[69] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint
2022.
[70] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang. Ngai-Man Cheung, and Min Lin. A recipe for watermarking diffusion models. arXiv preprint 1rXiv:2303.10137, 2023.
[71] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022.
[72] Yufan Zhou, Chunyuan Li, Changyou Chen, Jianfeng Gao, and Jinhui Xu. Lafite2: Few-shot -xxttt--mmae generation. arXiv preprint 1xXi:2220..41244 2022.
[73] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxi- ang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards language-free training for text-to-image generation. CVPR, 2022.
14

---

A Limitations
GILL relies on an LLM backbone for many of its capabilities. As such, it also inherits many of the limitations that are typical of LLMs. One limitation is the potential for hallucinations [6] where the model generates content that false or not relevant to the input data. Another limitation of the model in generating text is in repetitions and neural text degeneration [27], where the model generates the same content multiple times. We also observed that the OPT-6.7B model also does not always consistently generate coherent dialogue text.
These limitations may be addressed by techniques that address hallucinations and degenerations in text-only LLMs. or by using improved LLMs that are less prone to these issues. In GILL, we used a 6.7B model. In the future, it will be valuable to scale up the approach with even larger LMs. or those trained with improved objectives [54], instruction finetuning [38] or human feedback [40]. Depending on downstream applications, using models trained explicitly on dialogue data may also be helpful for dialogue capabilities (e.g., deploying multimodal chatbots).
With regards the visual models. another limitation of our approach is in its limited visual processing. At the moment, we use only k = 4 visual vectors to represent each input image (due to computational constraints), which may not capture all the relevant visual information needed for downstream tasks. These vectors are produced by frozen pre-trained visual encoder, and so the visual information in the vectors is heavily constrained by the pre- pre-training task. As a result, the model may not always process images correctly or in enough detail to produce accurate or high-quality results. However, this limitation can potentially be addressed in the future by scaling up the visual model, using models with varied pre-training objectives that encode more visual information while still being mappable to the hidden space of the LLM, or using more sophisticated visual mappings 4 43 that can capture a richer set of visual features. Similarly. we observed during inference that our model sometimes does not generate relevant images for certain types of prompts. We attribute this to our finetuning dataset being CC3M, which is relatively small compared to modern large scale image-text datasets [5I]. It is likely that training GII.I Mapper on an even larger corpus of text data will improve its alignment to the image generation backbonc.
One of the advantages of our model that it is modular, and can benefit from stronger visual and language models released in the future. It is likely that it will also benefit from stronger text-to-image generation backbones, or through finetuning the generation backbone rather than just the GILLMapper module. We leave such scaling explorations for future work.
B Broader Impact
AI Assistants Recent advances in dialogue based chatbots have sparked interest in using LLMs for interactive conversational applications. GILL is multimodal language model capable of processing image and text inputs, and producing image and text outputs. These capabilities may enable a wider range of applications. For example, Al assistants which can produce image and text outputs would be able to answer a wider range of queries, providing visual content when necessary to illustrate certain points. Concrete applications may include creative endeavors (e.g., iteratively refining generated image with instructions), answering questions that benefit from visual outputs (e.g.. describing food items), and more. Scaling GILL and refining it with methods such as reinforcement learning from human feedback (RLHF) 32 are promising directions to improve the capabilities of multimodal AI assistant systems.
Disinformation and Harms Aside from the technical limitations detailed in Sec. A there are broader societal issues that should be considered with the development of generative models of text and images. LLMs have the potential to generate plausible sounding (but false) text [22] 6). propagating disinformation at scale. As GILL uses an LLM backbone, these potential issues. Furthermore, as multimodal generative models which can also produce image content, models such as GILL also introduce potential issues with producing even more convincing disinformation through interleaving text with realistic generated images. As GILL makes use of an image generation backbone, it is also susceptible to the risks that typical text-to-image generation models introduce, such as generating false images of real people. These harms may possibly be mitigated by introducing watermarking into generated images [37 70 or by deploying systems to detect generated images [1].
also susceptible to
15

---

Bias and Safety GILL makes use of pretrained LLMs and multimodal models (such as CLIP 143 and Stable Diffusion 49 ) which are trained on large, noisy, Internet-scraped data (such as LAION- 400M 51 ). Due to their curation process, these datasets often contain undesired biases, malignant stereotypes (see [ for comprehensive discussion on large scaled multimodal datasets). One advantage of GILL is that it is efficient to train and completely modular, allowing its components (i.e., the LLM, visual encoder, or image generator) to be swapped out for other pretrained models (for example, models which have been further calibrated to reduce unintended biases).
Intended Uses GILL is research prototype which showcases possible capabilities of multimodal language models which can both process and produce image and text outputs. Due to the limitations described above, GILL not in its current state intended for deployment in practical applications, especially in high risk or sensitive domains without further analysis. At its current model scale (a 6.7B parameter LLM), GILL also lacks many of the abilities of larger language models [9] and applications would likely benefit from increased scaling of the LLM and visual models.
C Deciding to Generate or Retrieve
As detailed in Sec. 3.3 we evaluate several models on the annotated PartiPrompts [65] dataset. Each prompt is annotated with one of two labels: "ret" or "gen", indicating whether image retrieval or image generation produces a more appropriate image for the corresponding prompt. For example, the prompt "a portrail of statue of the Egyplian god Anubis wearing aviator goggles, white I-shirt and leather jacket, flying over the city of Mars." is labeled as "gen", "ge""." as there are (understandably) no appropriate images the CC3M retrieval set, and generation produces a more relevant output. In contrast, "the geyser Old Faithful" is labeled as "ret," as there are very relevant candidate images available for this prompt. We evaluate several models for making this decision on the validation set (Tab. (6). evaluating using F1 score given the class imbalance of the dataset (201 "gen", 110 "ret" in the validation set labels):
1. Baselines: We measure the Fl score of several baseline methods, which provide lower bound for how well data-driven approaches can do. We find that always retrieving an image, always generating an image, or simply deciding randomly (with a prior proportional to class frequencies) achieve F1 scores of 0.267, 0.389, and 0.451 respectively.
2. Heuristic: We also consider a simple heuristic which considers the maximum cosine similarity of the retrieval embedding against the entire image candidate set (i.e., the training set of CC3M). We run grid search from 0 to for possible threshold values. Whenever the maximum cosine similarity is above threshold, we return "ret" and "gen" otherwise. This achieves an Fl of 0.261 0.559, depending on the threshold used (a threshold of 0.5 gives Fl of 0.261).
3. Linear classifier: Lastly, we train a linear classifier that takes as input the outputs of the LLM for the [IMG] tokens and the maximum cosine similarity. This classifier is trained with the binary cross-entropy loss over the training set of PartiPrompts annotations. This linear classifier achieves an Fl score of between 0.393 - 0.552, depending on the probability threshold used (a threshold of 0.5 gives an F1 score of 0.547).
We use the linear classifier our final model, as it requires less hyperparameter tuning compared to the heuristic baseline, and performs comparably on quantitative metrics. During generation of qualitative samples (Fig. 7 and Fig. 5 in the main paper), we observed that the linear classifier generally performed well for many prompts, and decided correctly whether to retrieve generate.
D Qualitative Results
We present further qualitative samples in Fig. 7 We find that GILL is able to process complex text prompts more effectively than Stable Diffusion for many examples in PartiPrompts (55. On VisDial [16] dialogue inputs, GILL is able to generate more relevant outputs (as measured against groundtruth images). We attribute these improved results to the stronger text representations of the LLM, and the effectiveness of our GILLMapper network.
16

---

Table 6: Results PartiPrompts for classifying retrieval generation.
Method
F1
Always retrieve
0.267
0.389 Always generate
Random
0.451
Heuristic
0.261 0,559
Linear classifier
0.393 0.552
Human performance
0.851
Table 7: Results on image captioning on MS-COCO (2017) 34 and VQA 24
For captioning.
we report BLEU [4] and METEOR [ scores. For VQA, we report the accuracy after applying the normalization described the VQA repo /attp:://githue com/CT www.-Viiiin---a/DA... indicates our reimplementation.
Captioning
VQA
Model
BLEU-4
METEOR
O-shot
Frozen i 56
0.2553
MAGMA 19
0,2835
FROMAGe
0.1023
0.2873 0.2529
0.2851
Ours
0.1059
0.3178
E
Other Evaluations
E.1
Comparison to Prior Multimodal LMs
We ran evaluations on VQAv2 24 and image captioning on MS-COCO
The results are presented in Tab. 7 We found that GILL is comparable to models trained with similar compute and data. On VQAv2, we achieve a zero-shot val accuracy of 0,3 which is slightly better than prior approaches of similar model sizes and compute: FROMAGe [3] achieves zero-shot accuracy of 0.2851, Frozen (56) achieves 0.2553, and MAGMA 199 achieves 0.2835. For image captioning on the MS-COCO (2017) validation set, GILL achieves BLEU@4 @ of 0.1059 and METEOR of 0.2529, which is comparable to FROMAGe (BLEU @4 of 0.1023 and METEOR of 0.2873). GILL is also capable of a wider set of tasks (e.g., generating interleaved image and text outputs) compared to these models.
We note that these scores are lower than SOTA models, as they are usually much larger and trained with significantly more compute and data (e.g., Flamingo uses 23,040 TPU days, BLII--2 [3] uses 144 GPU days, while ours uses 4 GPU days). Scaling up GILL similar data and parameter scales to further push its capabilities is an exciting avenue for future work.
E.2 Increasing Context on VisDial
GILL leverages an LLM backbone, which allows it to inherit some LLM'' capabilities, such as improved sensitivity to long input contexts. In the main paper, we showed that GILLI can better condition on longer image and text inputs to generate more relevant images for VIST (28). We run similar experiment on Visual Dialogue [16], varying the number of dialogue rounds provided as input context to GILL and Stable Diffusion (SD) 49
The results are presented in Fig. g We find that when longer text context is provided to both models, the performance of generating relevant images steadily improves. Interestingly, SD performance plateaus after 6 rounds of dialogue, while GILL continues to improve, outperforming SD when 7 or more rounds of dialogue are provided. These results showease the improved sensitivity of our model to conditioning on long, dialogue-like text. Despite both approaches using the same image generation backbone, GILII able better make use of longer dialogue-text inputs (despite being only finetuned image caption data).
17

---

Stable Diffusion "A digigiiiedd beaaer wear glasses, visil, and colorful neck li. the standd next to tall stack of booka a library."
Ours
Stable Dillusion
Stable Diflusion
Oura
"Brow mountain and tree refeccion lake"
"M panguime in anowatoom"
Stable Diffusion
Ours
Stable Diffusion
Ours
Stable Diffusiin
made vaszing holkdrng ig skyscssper painting style of
"A drop-top top sports car coming around bend in the road"
holking
painting
Vincent Goghh"
Comparison Against Stable Diffusion
Mapper allows our model map eflectively to
image genenttin backbone, outper forming or matching for many examples from PartiPrompts
2: sume
C: shat color
la the man
shut color
ts the
vear tra. cap?
ere the
Rt. can you totallty dark
yea, the Lone
jutsiie
anowboard fa in color
black
color
unjoying
he seens
ViaDial Inputa
Stable Dillusion
Oura
Groundiruth
0:
thes boch wearting hot?
they
they other? they are factng away from each other
white and other deg ta light trows
Ar can't reality tell whet breed they are, perhass garman stephere
only
graxe?
they are standing dirt
other?
Like each
cantt be
Diaaial Imputs
Stable Dillusion
Oura
Groundtruth
Q: to this
strgle
Dr: is there
tottas popert
whar color to the toilet?
Q: la the yout
Bo there eirrer? yeu,
big tt long full tength eirror
e: what calor are walla?? they
purson
or dowe??
bathrucet
there tsn't
12
Imputs
Stable Dillusion
Oura
Groundiruth
Visual Dialogue
model process long, ciaiogue llke inputs generate more relevant images compared non LLM toxt image generation models.
paint l landsoape mountaing and flvor. you some Inapenaiion)
I'm looking for some Rashion inspiration for my apcoming vacal Haroal. Canyyou you
Ined inspiration bridal hol strirllle in garden weddiigg
would
my business?
swetinng, butl trenk should go with
lovaly, can you
Shage
ahso ahhow some summmer cresanes that would be pertuct my vocation
Nove beakts, iPak you should go with
is What plemow of dog inxtwer??
you show mo somo horses?
fann
cuti go
penteer outtroa
should with a white dirsss
ourful attroot austomors?
What about semething more abstroct?
Whit hair shouldd lweer?
outii cog?
Bhikk you shoule wear hoadbund
Multimodal Dialogue
Our modell can decice when to return nitrioued images, genensted imagee, text, allowing to respond affectivvly a wider varkety of dialogue sattings.
User prompts
Retrieved
Generated
Figure 7: Further qualitative samples from GILL. It is more sensitive to text inputs due its LLM backbone, and better at processing complex text prompts.
18

---

Table 8: Zero-shot FID 22 on the LLSCCOCO 34 (2014) validation
Results With Increasing Context on VisDial
0.70
GILL (ours)
set. 30,000 random samples are used to evaluate all models.
CLIP Similarity
Stable Diftusion
0.60
Model
FID (4)
GLIDE
12.24
0.55
Make-A-Scene 21
11.84
DALL-E 2 44
10.39
0.50
LAFITE2 772
8.42
10
Rounds of Dialogue
Imagen [50
7.27
Parti 665
7.23
Performance of our model and Stable Diffu- sion 49 with increasing context for generating VisDial [66 images. Our model is able to better process long dialogue- like text descriptions.
Figure 8:
Re-Imagen 14
6.88
SD 49 v1.5
9.22
GILL (ours)
12.2
E.3 Image Generation
In addition to our evaluations on VIST [28] and VisDial 16 we also run evaluations on our model's ability generate images from MS-COCO 34 captions (Tab. 8). We generate images using 30,000 randomly sampled captions from the MS-COCO (2014) validation set, which is the standard evaluation of text-to-ii soxt-to-mmageeeee generation models. We report zero-shot FID scores [25] of our model, Stable Diffusion [49] v1.5 (which we use as our backbone image generator), and several other approaches in Tab. For our generation results and SD results, we use classifier-free guidance scaling factor of 3.0 and 250 DDIM inference steps. On MS-COCO, our approach achieves worse FID score than SD (9.22 to 12.2). This likely because this task does not benefit as much from the LLM backbone, which has not been trained trains on caption-like data). These numbers will likely improve further by finetuning GILL on even more text data (including image captions), which will allow our model to align more closely to the input space of the SD image generator.
many image captions SD (which exclusively
E.4 Inference Speed
One concern for deploying LLMs the inference throughput and speed. We benchmark the inference performance of GILL on single A6000 GPU. Generating text has the same throughput as a regular LM of the same size (i.e., that of OPT 6.7B). The main increase in inference time occurs when the model produces [IMG] tokens. For batch size of 1, if the model decides to retrieve images, the additional inference overhead is minimal (less than 0.001s on average) as image retrieval fast, requiring just a single matrix multiplication followed by max operation. However, if GILL generates an [IMG] token, takes 3.5s on average for the Stable Diffusion backbone to generate a corresponding image.
Overall, GILL's inference speed bottlenecked by the frequency of image generation, which is dependent on the application domain. In the case of generating dialogue-like text, we observed that images are usually generated or retrieved once or twice natural conversation. Amortized over long conversation, it does not lead to significant increase compared to text-only LLM, though exact numbers would depend on the application.
F Human Annotation on PartiPrompts
In Sec. 3.3 of the main paper, we described the process of annotating PartiPrompts (65] with per-
example labels to retrieve or generate. The interface shown to human annotators is shown Fig. 2 Annotators are tasked to determine which of two anonymized images are (1) more relevant to the provided prompt, and (2) more realistic. We randomize the order of the two images as well (i.e., the output of the retrieval model shows up 50% of the time Image A).
19

---

Messon
Seow bullw are two noge and dnsorttoo the voocs Plcse aroer carkkers
Capdion: "band"
imag mathro
smow hrape my
relovunt above axptiet fanore realaars
Select the image which mort riualy ralticc the mar groce ogenpetate the image content with sepect mage
stert
26 knoge or image more nolletk image? nere the catioon for questiort
Figure 9: User interface shown to human annotators for annotating PartiPrompts
(65 examples.
We show each example to 5 independent human annotators. For determining whether label particular example as "ret" or "gen", we take the majority vote of the 5 annotators on the image relevance question ("Is image the examples with an inter-annotator agreement of at least 4/5. This results in approximately 900 examples remaining (out of the 1,632 examples in PartiPrompts). Our annotations will be publicly released to facilitate future evaluations on this task.
image B more relevant to the above caption?")), and only keep
We conducted evaluations the Amazon Mechanical Turk platform with human annotators located the US and Canada. Annotators were paid at an estimated hourly rate 15 USD per hour. In total, spent approximately 326 USD to collect these annotations.
20


Reviews:
[{"review_id": "JO99kBtg5P", "reviewer": "Reviewer_bnn7", "summary": "This paper proposes to use LLM to do image generation. Their approach consists of two stages of training.\n\nIn the first stage, they try to learn a linear layer to make VIT visual feature space is compatible with LLM space. \n\nIn the second stage, they learn r new tokens representing image. They hided states of these 'image token' are used to do retrieval or image generation. For the image generation, they train a GILLMapper to transfer these hidden states into CLIP text space (which is the input to the unet of SD) so that the transferred feature can be directly used as input the SD. They also found that simple design of the GILLMapper such as linear layer does not work well, thus they train a encoder -decoder like model (like a DETR)", "questions": "na", "limitations": "In my opinion, their comparison is not complete as they can have a simple extension (see weakness) so that it may potentially improve SD performance.\n\nI will raise my score if they could conduct this experiment. If they are going to do the experiment, I hope they carefully prompt a LLM and choose different captioning models to evaluate.    \n\n\n\n------------------------------------------------------------------------------------------------------------\nThe authors address my major concerns in their rebuttal and I recommend them to add these stronger baselines in their final version. \n\nW8 from reviewer QYwk seems a valid concern which I did not notice before. However, I am not very familiar with evaluation on text, thus I can not confidently evaluate their rebuttal in that question.\n\nOverall, I am lean towards the accept, but will not be supervised if got rejected due to other weakness raised by other reviewers  \n\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The writing is clear and I like their idea and motivation in general. ", "weaknesses": "1, I feel maybe it is not appropriate to claim their method can 'generate image'. The generation part is still from SD; what they are doing is to use a multi modal language model to combine both image and text information into CLIP text space. This can be reflected from their training: Neither stage1 nor stage2 actually involve image generation. \n\n2, for table 1, I think current 1 caption case does not make scene, and maybe they can just skip this, as it is not using any previous information at all. For 5 caption case, I don't know how exactly they feed them (maybe just stack them?). I think a better way is to ask a LLM such as GPT4 to combine all captions and generate a better prompt.  \n\n3, for 5 captions and 4 images case in the table 1, the SD can also do this. A simple approach is to use image captioning model such as BLIP2, LLAVA etc to convert images as caption, and prompt GPT4 to generate the final caption by giving 5 captions and 4 images (in the captioning format) \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments. We are pleased that the reviewer found the writing clear, and that they liked the idea and motivation of the paper. We address specific queries below, and will incorporate all feedback.\n\n## 1. Whether GILL Can ‘Generate Images’\n\nWe consider GILL to be the combined visual encoder, LLM, and visual decoder (i.e., the SD generator), not just the LLM. One of our contributions is that this combined model can process interleaved image + text inputs, and generate text, generate images, and retrieve images, and interleave image and text in its outputs. Thus, we think that it is reasonable to claim that GILL can generate images, as this is one of its capabilities. If the reviewer has suggestions for specific lines in the paper that they believe could be re-written to be more accurate, we are also happy to consider rephrasing them.\n\n\n\n## 2. 1 Caption Setting\n\nFor the VIST evaluation, the single caption corresponds to the last image. Since the goal is to generate the last image, it is possible to produce a reasonable image given just this one caption. This allows us to compare what happens when the model has access to a single caption, for the last image, compared to the full sequence of captions, for all images, and it verifies that having access to the full context helps. For these reasons, we think that this is still a useful baseline to include.\n\n## 3. 5 Captions + GPT Baseline\n\nThe results in Tables 1 and 2 concatenate the 5 captions for both SD and our model. We also ran the experiment suggested by the reviewer, where we provided GPT-3.5 (turbo) with the 5 story captions, and asked it to generate a prompt.. We find that our GILL approach outperforms this baseline, see results below. (Below, in 4., we also find that our approach outperforms using GPT-4 as the LLM.) We prompt the LLM as follows:\n\n```\nThere are five images and five story parts that make up a full visual story. The five story parts are provided. Generate a caption that will be used as input to a text-to-image generation model to generate the last image in the sequence. The caption should relate to the full story sequence, but also describe the content that should be in the last image in the sequence.\n\nStory Part for Image 1: \"my sister arrived early to help me with the family bar bq .\"\n**<parts 2 to 4 omitted for brevity>**\nStory Part for Image 5: \"we ended the day shooting off some fireworks .\"\n\nFinal image description for text-to-image generation:\n```\n\nWe find that this generally produces reasonable captions. For the above example, the completion by GPT-3.5 (turbo) is “a group of people gathered around a bonfire, watching fireworks light up the night sky”. We use this completion to generate an image with SD, and run the same VIST evaluation on it. The results are as follows:\n\n| Model      | CLIP Similarity ($\\uparrow$) | LPIPS ($\\downarrow$) | \n| ----------- | ----------- | ----------- |\n| Ours (5 captions + 4 images)                                 | **0.641**       | **0.693**       |\n| Ours (5 captions)                                 |  0.612      |  0.696       |\n| SD (5 captions, concatenated)         |  0.598        | 0.704      |\n| GPT-3.5 + SD  (5 captions)        | 0.605        | 0.707      |\n\n\nWe see that while this improves over simply concatenating the captions, it still underperforms our method even when image information is not provided (despite GPT-3.5 using a much larger LLM). For reasons of cost, we do not evaluate GPT-4 in this setting, but do in 4. below.\n\n\n## 4. Captioning Model + GPT + SD extension\n\nAs suggested, we ran this baseline. We used various captioning model to generate captions for the 4 images, and similar to the above, prompted GPT-4 / GPT-3.5 as:\n\n```\nThere are five images and five story parts that make up a full visual story. The descriptions of the first four images and the five story parts are provided. Generate a caption that will be used as input to a text-to-image generation model to generate the last image in the sequence. The caption should relate to the full story sequence, but also describe the content that should be in the last image in the sequence.\n\nStory Part for Image 1: \"my sister arrived early to help me with the family bar bq .\"\nImage Description 1: \"a woman sitting in the driver's seat of a car\"\n**<parts 2 to 3 omitted for brevity>**\nStory Part for Image 4: \"there was so much food and it was all delicious .\"\nImage Description 4: \"a person is holding a knife and a burger\"\nStory Part for Image 5: \"we ended the day shooting off some fireworks .\"\n\nFinal image description for text-to-image generation:\n```\n\nWhere the “Image Descriptions” are the outputs of an image captioning model. The results of this experiment on VIST are as follows (all models use 5 captions + 4 images):\n\n| Model      | CLIP Similarity ($\\uparrow$) | LPIPS ($\\downarrow$) | \n| ----------- | ----------- | ----------- |\n| Ours                                 | **0.641**       | **0.693**       |\n| BLIP-2 (OPT-6.7B) + GPT-3.5 + SD   | 0.620        | 0.705      |\n| BLIP-2 (T5-XL) + GPT-3.5 + SD        |  0.622        | 0.705      |\n| LLaVA (LLaMA-2-13B) + GPT-3.5 + SD        |  0.620        |  0.704      |\n| BLIP-2 (T5-XL) + GPT-4 + SD        |  0.630        |  0.700      |\n\n\nWe find that this baseline improves results over the SD baseline (which does not use image captioning models), but the results are still worse compared to our model. This is despite our model being significantly smaller and finetuned with less data.\n\nAside from these quantitative results, there are other benefits in using an approach such as ours. GILL generates latent features that are provided to the SD image generator, which bypasses the text bottleneck that models such as the GPT baseline above introduce. With our approach, there is no requirement to autoregressively generate text for the generator.\n\nWe hope that this clarifies the reviewer’s queries on the suggested SD + GPT baselines."}, {"review_id": "zYP0AxQnBs", "reviewer": "Reviewer_5sAf", "summary": "The authors train adapters to map embeddings of pre-trained image encoders and decoders to pre-trained LLMs. This allows them to input interleaved images with text into a pre-trained LLM and also make the LLM generate [IMG] tokens as required, which can be fed into a decoder to generate images or can be used to retrieve from a set of images based on cosine similarity. ", "questions": "Is it necessary to train for both generation and retrieval? How does it work if we just train for generation and not include the retrieval loss? The ablation in the supplemental Table 1 is just during inference, correct?\n\nDuring training, the authors say they use CC 3M. This only has one image and text pair. Do they use any data that has multiple images and text? Are results on Visual Stories without training on it? \n\nWhat are the [IMG_r] tokens learning exactly and how does varying `r` (the number of image tokens) affect results? I am guessing each of the tokens are learning some \n image concepts present frequently among the images in data? An analysis on this would have been great!", "limitations": "Authors discuss limitations in the supplemental. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Clean idea that introduces a fundamental novelty in the capability of text-to-image models - the ability to interleave image and text to generate images.\n- Adapting image to text space to use with pre-trained LLM's is neat since we don't need to train yet another large model. This idea has been shown to work well in contemporary models like LLaVA and miniGPT-4, but they show it works well for generation as well - a capability these other contemporary models lack. \n- The results look very promising. They show longer context helps, further justifying the need of such models that can handle long chains of interleaved image and text. \n- Such a line of work can have multiple interesting follow-up works on evaluating compositionality, etc. ", "weaknesses": "- One of the strengths of interleaving image and text is that one can expect to extract different concepts from different images to compose a new image. For instance: \"a cup that looks like <image_of_come_cup>, but on a table that looks like <image_of_some_table>\" Such an analysis could have been great similar to the goal of this paper: https://arxiv.org/pdf/2212.04488.pdf.\n\n- Some other recent methods aim to solve similar goals as advertised in the qualitative results. For instance, Figure 5 top row - papers like prompt-to-prompt and imagick aim to do this. Although their method cannot handle arbitrary interleaved images and text, a qualitative comparison of these applications that both methods can handle would make the paper very strong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and valuable comments. We are glad that the reviewer appreciated the promising results from our paper, the possibility for interesting follow-up work, and recognized that it is a clean idea which introduces fundamental capabilities (image generation) to multimodal language models. We address specific queries below, and will incorporate all feedback.\n\n\n## 1. Training a generation-only model\nWe ran the suggested experiment, removing the retrieval objective from the training losses. On the VIST evaluation (5 captions, 4 images), this ablated model achieves CLIP similarity of 0.636 and LPIPS of 0.694, which are comparable to that of the original model (CLIP similarity of 0.641 and LPIPS of 0.693). This suggests that it is not necessary to include the retrieval loss, although such a model would only be able to generate images and text and not retrieve images. These results also suggest that the model is not bottlenecked by including the retrieval objective, and that there is sufficient capacity for the model to perform both generation and retrieval.\n\n\n## 2. Training data\n\nOur model is trained on just the CC3M dataset (which contains just single image-text pairs). We follow [1] in randomly packing two distinct examples together during training with probability 0.5. This means that 50% of the time, examples consist of `<image1><text1><image2><text2>`, while the other 50% of the time, it consists of single examples `<image1><text1>`). Although the two examples are distinct (`<image2>` is in general unrelated to `<image1>`), we also find this helps encourage the model to attend to the correct image in the sequence, rather than simply always attending to the first image in the input sequence.\n\n> Are results on Visual Stories without training on it? \n\nThis is correct. Despite not using explicitly interleaved data with multiple related images (such as Flamingo [2] or CM3 [3]), GILL is capable of processing Visual Stories zero-shot (which consists of 5 captions, 4 images, more than even the 2 images + 2 captions we randomly introduce through packing). We attribute this to the ability of the pretrained frozen LLM to generalize to multiple images, as we learn to map the images to embeddings in the LM space.\n\n\n## 3. Analysis on the $r$ image tokens\n\nIn the main paper, we included ablations and discussion on how varying the number of tokens $r$ affects generation results (Table 4, and L275-277 on page 9). We find that lower values of $r$ (e.g., $r=1$ or $r=2$) tends to result in worse results on the VIST task, as the model is less expressive, which motivates our design choice of using $r=8$ tokens for improved generated image quality and image-text match (as measured on VIST).\n\n> I am guessing each of the tokens are learning some image concepts present frequently among the images in data\n\nWe largely agree with this intuition: the $r$ image tokens are used as inputs to GILLMapper (trained by minimizing the L2 loss of its outputs against the CLIP text embeddings), whose outputs are used by the Stable Diffusion image generator. The “concepts” learnt by the image tokens in GILL would likely correspond to the “concepts” used by SD to produce image tokens. Further and more comprehensive analysis would likely be necessary to verify this (e.g., learning linear probes, or finding a way to visualize the embeddings of the individual tokens without significantly altering the SD pipeline), which we believe might be out of the scope of this paper, but would be very interesting to study in future work.\n\n\n## 4. Extracting concepts from images\n\n> One of the strengths of interleaving image and text is that one can expect to extract different concepts from different images to compose a new image. For instance: \"a cup that looks like <image_of_come_cup>, but on a table that looks like <image_of_some_table>\" Such an analysis could have been great similar to the goal of this paper: https://arxiv.org/pdf/2212.04488.pdf.\n\nWe agree that this is a very exciting direction for follow up work! One of the limitations of GILL (discussed further in Appendix A) is that it has somewhat limited visual processing capabilities. Hence, we don’t expect that it will do as well as the referenced paper or prompt-to-prompt on tasks that involve finegrained image editing.\n\nIn GILL, input images are represented as $k = 4$ visual vectors. Although this improves compute efficiency (and allows us to train the model on just 2 A6000 GPUs), it results in the loss of some finegrained visual information. Scaling the model up to adopt pretraining objectives that encode more explicit information (e.g., longer sequences of ViT patches [4]) would be promising directions for future work.\n\n\n### References\n\n[1] Koh, J.Y., Salakhutdinov, R. and Fried, D. “Grounding Language Models to Images for Multimodal Inputs and Outputs”. ICML, 2023.\n\n[2] Alayrac, Jean-Baptiste, et al. \"Flamingo: a visual language model for few-shot learning.\" NeurIPS, 2022.\n\n[3] Aghajanyan, Armen, et al. \"CM3: A causal masked multimodal model of the internet.\" arXiv preprint arXiv:2201.07520, 2022.\n\n[4] Liu, Haotian, et al. \"Visual instruction tuning.\" arXiv preprint arXiv:2304.08485 (2023)."}, {"review_id": "dfJKYb0EWM", "reviewer": "Reviewer_QYwk", "summary": "The paper proposes a novel method to stitch together LLMs and text conditional image diffusion models, to process interleaved image-text and output interleaved image-text. The paper evaluates their methodological contributions on several tasks where outputting images is required.", "questions": "Here are my main questions distilled from the weaknesses above:\n\nQ1. Can some of the clarity related questions be addressed, specifically W5, W6, W7, W10?\n\nQ2. How does your model perform on CoCo captioning and VQAv2 (W8)? These also seem like useful evaluations to make sure that your model interprets the content in images.\n\nQ3. Can authors address the concerns related to the generality of the method for image generation vs. retrieval (W9)?", "limitations": "No limitations or failure analysis is presented in the main paper. I suggest discussing conditions under which the model fails.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1. The ability to output generated image or text OR retrieve images to satisfy a user’s query is not something I have seen in prior work and speaks to the generality of the formulation.\n\nS2. The paper explores how to stitch together large pre-trained models (i.e., LLMs and image diffusion models). It represents a creative exploration of stitching these models together using a combination of learnable modules, captioning losses, contrastive losses, and distillation losses, which is non-trivial exploration.\n\nS3. Generative applications of the model appear impressive in the demo figures.\n\nS4. Ablations related to image generation and retrieval are generally strong, thoughtful, and cover many different key design decisions. \n\nS5. Fine-tuning cost is cheap and reasonably accessible (~96 A6000 GPU hours).\n", "weaknesses": "W1. Missing citation. The following paper conducts probes showing that pre-trained image and text representations can be aligned: [1] Ilharco et al. Probing Contextual Language Models for Common Ground with Visual Representations. 2021.\n\nW2. Missing citations. Since one potential use-case of the model is image captioning (i.e., the model can handle image input and textual output), I also suggest citing recent captioning work: [2] Yu et al. CoCa: Contrastive Captioners are Image-Text Foundation Models. 2022.; [3] Li et al. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. 2022.; [4] Li et al. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. 2023; [5] Dai et al. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. 2023.\n\nW3. Missing citation, unsupported claim. Recently, unified models take image and text inputs and generate image and text outputs for a variety of tasks. e.g., [6] Lu et al. Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks. 2022. Hence, lines 4-6 in the abstract seem a bit overstated: “Ours is the first approach capable of conditioning on arbitrarily interleaved image and text inputs to generate coherent image (and text) outputs.”\n\nW4. Presentation. Consider adding percentage point improvements over baselines in the abstract and intro to give the reader an idea early on of what kinds of gains can be expected by using your model over competitors.\n\nW5. Presentation. Consider giving some more intuition about why you are making certain design decisions, instead of just saying what you are doing. For example, I found the motivation for your introduced IMG1, IMG2, …, IMGr tokens to be a bit confusing. It seems here, the supervision encourages learning when to produce IMG tokens, but not what content should be captured in the representation space, which would be different depending on the image. Why should this design decision will lead to learning good last layer hidden representations for arbitrary images.\n\nW6. Clarity. The Image retrieval section seems incomplete. An objective is provided, but it is not clear to me how the model is ultimately used to retrieve images at evaluation time.\n\nW7. Clarity. The work proposes to use a learned mapping module (GILLMapper) to go from text hidden states (of an LLM) to vision model embedding space (of a text-conditioned image diffusion model). However, it is not clear this module is needed. An alternative strategy might be to feed generated LLM text directly to the text-conditioned diffusion model, thereby bypassing the need to map representations.\n\nW8. Missing evaluation, unsupported claims. The paper claims that the model can generate image or text or retrive images. However, the experiments test only the abilities related to outputting images. How does the model perform on tasks that require outputting text (e.g., CoCo CIDEr and VQAv2 accuracy)? I consider such evals to be critical to verify claims that the model can generate sensible text outputs based on image inputs. Such evaluations also allow for comparison with Flamingo-like models.\n\nW9. Generality of the method. Deciding whether to retrieve or generate images depends on a liner classifier that is dataset specific. This means that a practitioner wanting to use this model out-of-the-box on data of their own may have to train their own classifier (annotate data etc.) to use the model.\n\nW10. Clarity. How is CC3M turned into an interleaved training dataset for model training? Are there any data sampling strategies that are important here? How many images and captions are sampled? Are non-interleaved sequences also trained on (i.e., image or text only)?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments. We are glad that the reviewer recognized the creativity and generality of our approach in combining pretrained LLMs and visual models, and appreciated the impressive qualitative results, strong ablations, and accessible finetuning cost. We address specific queries below, and will incorporate all feedback.\n\n## 1. Clarity Related Queries\n\n### W5\nOur approach includes losses to supervise both learning when to produce `[IMG]` tokens, but also the representation of the `[IMG]` tokens (detailed in L120-140 of the main paper) through minimizing the L2 loss against CLIP text embeddings. The representations depend on context, so different text inputs produce different `[IMG]` token hidden states (due to the LM attending to different things). This allows us to produce the appropriate representations for different text inputs. We will update Sec 3.2 and the caption of Fig. 2 to make this clear. \n\n### W6\nDuring inference, we follow standard procedure [7] and retrieve the image with the highest cosine similarity (between image embeddings and `[IMG]` token embeddings) from a dataset. For VIST (Tab. 5), this is from its val set (consistent with [8]). For other qualitative results, we retrieve from CC3M.\n\n### W7\nOne benefit of our approach is that it treats image outputs as continuous embeddings to be directly fed into an image generator. This allows us to leverage the capabilities of LLMs, such as in-context learning and the ability to handle longer inputs than can be handled by the SD text encoder (i.e., 77 tokens) for generating images. Continuous representations bypass the text bottleneck, and can be optimized completely end-to-end. Feeding generated text would require RL-like approaches, which may not be as straightforward. Please also see our response to reviewer bnn7, which shows that a text-only approach (using generated text from much larger GPT-3.5/4 models) underperforms GILL on VIST, suggesting that using generated text may be insufficient for more complicated tasks.\n\n### W10\nWe follow [8] in packing two random examples during training with probability 0.5. This means that 50% of the time, input data is `<img1><txt1><img2><txt2>`, while the other 50% of the time, it consists of single examples `<img1><txt1>`. Although the examples are distinct (`<img1>` is in general unrelated to `<img2>`), we find this helps encourage the model to attend to the correct image, rather than always attending to the first image.\n\n> Are non-interleaved sequences also trained on (i.e., image or text only)?\n\nWe do not train on image or text-only sequences, though the frozen LLM was originally pretrained on text-only data.\n\n## 2. VQAv2 and COCO\nWe ran evaluations on VQAv2 and MS-COCO captioning, and found that GILL is comparable to models trained with similar compute and data. On VQAv2, we achieve a zero-shot val accuracy of 31.78, which is slightly better than prior approaches of similar model sizes and compute: FROMAGe achieves zero-shot accuracy of 28.51, Frozen achieves 25.53, and MAGMA achieves 28.35. On MS-COCO, GILL achieves a BLEU@4 of 0.1059 and METEOR of 0.2529, which is comparable to FROMAGe (BLEU@4 of 0.1023 and METEOR of 0.2873). GILL is also capable of a wider set of tasks (e.g., generating interleaved image and text outputs) compared to these models.\n\nWe note that these scores are lower than SOTA models, as they are usually much larger and trained with significantly more compute and data (e.g., Flamingo uses 23,040 TPU days, BLIP-2 uses 144 GPU days, while ours takes 4 GPU days). Scaling up GILL to similar data and parameter scales to further push its capabilities is an exciting avenue for future work.\n\n## 3. Generality of the Decision Method (W9)\nOur evaluations were conducted on PartiPrompts (P2), and we find that the decision classifier trained on P2 does reasonably well on a held out subset (Tab. 1 in the appendix).\n\n> This means that a practitioner … may have to train their own classifier\n\nWe agree that this is a fair point, and whether it would generalize would likely depend heavily on the actual application’s data distribution, and the amount of data available. P2 is best suited to generalizing to domains that require separating factual from non-factual captions, which we believe is a key consideration for many downstream tasks. However, there is likely room for improvement in future work, e.g., by taking into account the quality of the generated image before making a decision.\n\n## 4. Missing Citations\n### W1 and W2\nWe will update the intro to include [1]. We will also update the related work section to discuss image captioning approaches as suggested.\n\n### W3\n> Recently, unified models take image and text inputs and generate image and text outputs for a variety of tasks. e.g., [6] …\n\nThanks for the pointer! To the best of our knowledge, ours is still the first approach capable of generating coherent images and text. Unified-IO generates images *or* text, but without modifications, does not seem capable of producing outputs that are interleaved image and text, or decide when to generate text and when to generate images.\n\nIn contrast, GILL generates text interleaved with images, and automatically determines when to produce images instead of text. CM3 and FROMAGe are capable of doing so, but CM3 frequently generates incoherent or low quality images (see appendix of [8]). We are able to do better as we leverage a strong pretrained diffusion model. FROMAGe retrieves images but does not generate novel images.\n\n## 5. Limitations / Failure Analysis\nWe included discussions on limitations, failure modes, potential future directions to alleviate these, and the broader impact of our model in Appendices A and B.\n\n\n### References\n[7] Radford, Alec, et al. \"Learning Transferable Visual Models From Natural Language Supervision.\" ICML, 2021.\n\n[8] Koh, Jing Yu. et al. “Grounding Language Models to Images for Multimodal Inputs and Outputs”. ICML, 2023."}, {"review_id": "GrmYHh3VuX", "reviewer": "Reviewer_Gk8n", "summary": "The paper introduces a new method called GILL that effectively integrates frozen LLMs with pre-trained image encoder and decoder models to create coherent image and text outputs. The authors show GILL's superior performance over baseline generation models in tasks involving longer and more complex text and its capability for image retrieval and generation at inference. GILL extends the capabilities of pre-trained LLMs to multimodal by mapping the embedding spaces of the LLMs to visual models with a mapping network. The proposed framework is inspiring and interesting. It may have a large impact for unified multimodal pre-training.", "questions": "See weakness above", "limitations": "See weakness above", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "* The proposed approach is innovative and inspiring, combining frozen LLMs with visual models to generate coherent image and text outputs. This approach is efficient, as it doesn't require training the image generation model from scratch.\n* The paper is well-written, logically structured, and accessible.\n* Experimental results show that GILL is effective in processing long-form text and generating images that are more closely matched to the text than baselines. It can process arbitrarily interleaved image-text inputs and generate retrieved images, novel images, and text, which expands its capabilities compared to previous models.\n* The paper’s approach to aligning visual tokens with LLMs is promising, and the proposed solutions are insightful.", "weaknesses": "* The paper lacks clear implementation details. Despite Eq. 5 explaining the overall objective to optimize, the parameters to be updated are not clearly stated in the equations. This makes it unclear which parts of the model require training and which parts remain fixed.\n* While the GILL framework is novel, the GILLMapper idea is not new. Aligning a pre-trained/new encoder with the CLIPText encoder of the Stable Diffusion model has been previously explored [1,2], and this related literature is not discussed in the paper.  [1] discussed altering the Language Encoder to extend the multilingual capacities for Stable Diffusion. [2] plug-ins the multimodal encoders to replace the CLIPText of the Stable Diffusion. The Eq. 3 of GILL paper is similar to the Eq. 3 of [2] even though GILL applies visual tokens here. \n\n[1] AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities. arXiv preprint arXiv:2211.06679 (2022).\n\n[2] GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation. arXiv preprint arXiv:2303.10056 (2023) \n\n* The paper lacks comparison with state-of-the-art multimodal methods such as BLIP.\n* The paper doesn’t discuss other relevant works, such as one [3] that unifies both text-to-image and image-to-text in one framework.\n\n[3] CoBIT: A Contrastive Bi-directional Image-Text Generation Model.\" arXiv preprint arXiv:2303.13455 (2023).\n\n* The paper lacks discussion of the inference speed, which is an especially essential factor for diffusion-based image generation task.\n* The paper does not address the possibility of the OPT model not perfectly aligning with CLIPText, which could create cross-model gaps  that are challenging to bridge. It's unclear how the authors would tackle such challenges and whether these difficult-to-address domain gaps would cause a decrease in performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments. We are glad that the reviewer found our proposed approach innovative and inspiring, and recognized that it is efficient. We are pleased that the reviewer found the paper well-written, and appreciated GILL’s improved capabilities over baseline models.\n\n## 1. Implementation Details\n> Despite Eq. 5 explaining the overall objective to optimize, the parameters to be updated are not clearly stated in the equations\n\nWe describe the trainable parameters in L178-180, which are the linear layers $\\\\mathbf{W}\\_{\\\\text{i2t}}$, $\\\\mathbf{W}\\_{\\\\text{t2i}}$, $\\\\mathbf{W}\\_{\\\\text{cap}}$, the IMG embedding matrix $\\\\mathbf{E}\\_{\\\\text{img}}$, and the GILLMapper parameters $\\omega$ and query vectors $q\\_{1:L}$ (while everything else, including the LLM backbone, the visual encoder, and the SD generator, remain frozen), which is also illustrated in Fig. 2. We will also update Eq. 5 to read:\n\n$$\\\\min\\_{\\\\mathbf{W}\\_{\\\\text{i2t}}, \\\\mathbf{W}\\_{\\\\text{t2i}}, \\\\mathbf{W}\\_{\\\\text{cap}}, \\\\mathbf{E}\\_{\\\\text{img}}, \\\\omega, q\\_{1:L}} \n\\\\frac{1}{N} \\\\sum\\_{i=1}^N \\\\big(l\\_c(\\\\mathbf{x}\\_i, \\\\mathbf{y}\\_i) +  l\\_p(\\\\mathbf{y}\\_i) +  l\\_g(\\\\mathbf{y}\\_i) +   l\\_{r}(\\\\mathbf{x}\\_i, \\\\mathbf{y}\\_i) \\\\big) $$\n\nwhich hopefully clarifies. We will also release **pretrained models weights and the full code** for training a model from scratch, to account for any other remaining code and low-level implementation details that are difficult to explain in the space limitations of the paper.\n\n## 2. Discussion of prior work\n\n> While the GILL framework is novel, the GILLMapper idea is not new …\n\n> The paper lacks comparison with state-of-the-art multimodal methods such as BLIP.\n\nThanks for the pointers, and we are glad that the reviewer recognized the novelty of the overall GILL framework. We will update the paper to include these references in our discussion on related work with respect to the GILLMapper module.\n\nIn this paper, we primarily focus on the ability of GILL to generate images conditioned on interleaved image-text inputs, since most existing multimodal models such as BLIP cannot do this. For these reasons, we primarily compared against Stable Diffusion to test their image generation abilities. We also compared retrieval performance against FROMAGe, which is one of the few prior approaches which can process text + image inputs to generate text + image outputs (although FROMAGe retrieves rather than generates). We showed that GILL is better than SD at generating images with longer contexts (Table 1, 2) and as good as FROMAGe in retrieving images (Table 5). We will also include results on VQAv2 and MS-COCO captioning (see response to reviewer QYwk), which show that GILL is competitive with models of similar sizes and amount of training on image-to-text generation.\n\n> The paper doesn’t discuss other relevant works, such as one [3] that unifies both text-to-image and image-to-text in one framework.\n\nCoBIT is concurrent work according to NeurIPS guidelines (< 2 months of the deadline), but we will update the paper to briefly discuss it. To the best of our understanding, CoBIT is capable of generating text *or* image, but does not generate *interleaved* text and image outputs. It does not include methods to determine when to synthesize text vs. images. Our work is capable of generating text and images (by representing images as `[IMG]` tokens). This allows us to generate multimodal dialogue-like outputs consisting of both images and text.\n\nGILL is a proof of concept for multimodal models that can process image + text and produce image + text, trained in an efficient manner (4 GPU days). Scaling it up to the data/compute scale of BLIP-2 (144 GPU days) and CoBIT (6144 TPU days) is a promising direction for future work.\n\n## 3. Inference Speed\n\nGenerating text has the same throughput as a regular LM of the same size (i.e., that of OPT 6.7B). The main increase in inference time occurs when the model produces `[IMG]` tokens. For a batch size of 1, if the model decides to retrieve images, the additional inference overhead is minimal (< 0.001s on average) as image retrieval is fast (requiring a single matrix multiplication followed by a max operation). If the model predicts to generate an image, it takes 3.5s per image on average on a single A6000 GPU, which is the time for SD to generate a single image.\n\nOverall, GILL’s inference speed is bottlenecked by the frequency of image generation, which is dependent on the application domain. In the case of generating dialogue-like text, we observed that images are usually generated or retrieved once or twice in a natural conversation. Amortized over a long conversation, we believe it does not lead to a significant increase compared to a text-only LLM, though exact numbers would depend on the application.\n\n## 4. Alignment\n\nOur experiments on VIST and VisDial (out of domain w.r.t. the CC3M finetuning data) suggest that our current approach does well at aligning the `[IMG]` representations with the CLIP text encoder. With respect to domain shift, our approach does not simply overfit to CC3M: our model surpasses SD on VIST and VisDial. With respect to the modality gap, both SD and our approach use the same image generation backbone, and the only difference is in the text encoder. If the learnt mapping was poor, we would not be able to outperform SD on these benchmarks. These results suggest that the current approach of training the GILLMapper module aligns it well to the CLIP text space.\n\nHowever, we agree that it is possible that on more difficult or more out of distribution tasks, the `[IMG]` representations may not be mapped well. A possible method to alleviate this would be to include the SD generator into the pipeline, and train the whole model end-to-end with the image generation loss rather than the L2 loss on the text encoding. This would be useful to explore in future work, but would likely require significantly more GPU memory."}]

Author rebuttal:
We thank all reviewers for their valuable comments. We are glad that all reviewers appreciated the creativity and novelty of our method, and found it innovative and inspiring (reviewer Gk8n), appreciated its creative exploration (reviewer QYwk), ideas and motivation (reviewer bnn7), and recognized that it introduces a fundamental novelty in the capabilities of multimodal language models that other contemporary models lack (reviewer 5sAf).

We are pleased that the reviewers found the paper well-written (reviewers bnn7, Gk8n), and the results and applications impressive and very promising (reviewers Gk8n 5sAf). We are glad that the reviewers found our proposed modeling approaches insightful (reviewer Gk8n), and recognized its potential for multiple interesting follow-up works (reviewer 5sAf).

We address specific queries and run the suggested baselines in responses to individual reviewers below.

Meta-review: This paper proposes to use LLM to do image generation. Their approach consists of two stages of training. (1) Stage-1: learning a linear layer to align VIT visual feature space with LLM word embedding space. (2) Stage-2: learning new tokens to represent the image. A GILLMapper is trained to transfer these hidden states into CLIP text space as the input to SD. Technique wise, a simple linear projection design of the GILLMapper does not work well, thus a encoder -decoder like model is considered.

All reviewers agree to accept the paper after the rebuttal. This is the first attempt to connect pre-trained image generation with open-source LLM, various technique details are well explored. This can be a valuable baseline for future exploration.
Decision: Accept (poster)