SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Results on contextual image generation on VIST [28] (averaged over 5 random seeds). Our model can process longer (possibly multimodel) inputs to outperform baseline models.
Model	CLIP Similarity (↑)	LPIPS (↓)
1 caption	5 captions	5 caps, 4 images	1 caption	5 captions	5 caps, 4 images
GLIDE [38]	0.582	0.591	-	0.753	0.745	-
Stable Diffusion [49]	0.592 ±0.0007	0.598 ± 0.0006	-	0.703 ±0.0003	0.704 ± 0.0004	-
GILL (ours)	0.581 ±0.0005	0.612 ±0.0011	0.641 ±0.0011	0.702 ±0.0004	0.696 ±0.0008	0.693 ±0.0008
Table 2: Results on contextual image generation on VisDial [16] (averaged over 5 random seeds). Our model can process longer sequences of dialogue-like text to generate more relevant images.
Model	CLIP Similarity (↑)	LPIPS (↓)
1 round	5 rounds	10 rounds	1 round	5 rounds	10 rounds
GLIDE [38]	0.562	0.595	0.587	0.800	0.794	0.799
Stable Diffusion [49]	0.552 ±0.0015	0.629 ±0.0015	0.622 ±0.0012	0.742 ±0.0010	0.722 ±0.0012	0.723 ±0.0008
GILL (ours)	0.528 ±0.0014	0.621 ±0.0009	0.645 ±0.0010	0.742 ±0.0022	0.718 ±0.0028	0.714 ±0.0006
Generating from Visual Stories VIST [28] is a dataset for sequential vision-and-language tasks, with examples of sequences of 5 images and text that constitute a story, as shown in Fig. 5. Similar to [31], we test the models on generating the last image in the sequence, conditioned on different inputs:
1. 1 caption: Input consists of the last text description. This is similar to standard text-to-image generation, where a model conditions on a single caption to generate an image.
2. 5 captions: Input consists of all text from the entire story sequence. This tests the ability of models to process longer and temporally dependent text descriptions.
3. 5 captions, 4 images: Lastly, we test models with inputs of all images and texts preceding the last image (i.e., sequenced as “<text1><img1>...<text4><img4><text5>”). This tests the ability of models to effectively process multimodal context in image generation. A novel feature of GILL is its ability to process interleaved image-text inputs, which most existing text-to-image generation models are unable to handle.
We report results on VIST in Tab. 1, comparing GILL against text-to-image generation baselines (including Stable Diffusion (SD) [49], which we use as our generation backbone  $G_{\psi}$ ). With a single story caption input to both models, the performance is comparable, with SD achieving a slightly better CLIP Similarity score, and both models achieving similar LPIPS. However, when all 5 story captions are provided as input, our model outperforms SD, improving CLIP Similarity from 0.598 to 0.612, and LPIPS from 0.704 to 0.696. Interestingly, when further provided with the full multimodal context (the preceding 5 captions and 4 images), our model improves substantially, attaining a CLIP Similarity of 0.641 and LPIPS of 0.693. In contrast, SD is unable to handle interleaved image-text inputs without significant modifications. We also show several qualitative examples in Fig. 5. We find that GILL is generally more sensitive to input context compared to SD. GILL can also condition on image inputs, enabling it to use visual context to produce more relevant images.
We highlight that both models use the same image generation backbone, and the primary difference is in their text encoders. GILL is able to better handle long text inputs and multimodal context, which we attribute to the stronger LLM encoder coupled with our GILL Mapper model.
Generating from Visual Dialogue We also test our model on the VisDial [16] dataset. VisDial examples contain a sequence of question and answer (Q&A) pairs about a particular image, simulating dialogue between two people who are discussing an image. Examples contain up to 10 rounds of Q&A dialogue pairs. Similar to VIST, we evaluate the ability of models to accurately synthesize the image being described, provided with increasing amounts of the Q&A dialogue context as input. This experiment tests the ability of our approach to (1) generalize to dialogue-like text (as our approach is only finetuned on image caption data), and (2) process long text sequences.
Our results are presented in Tab. 2. Similar to the VIST evaluations, we find that with shorter length inputs, SD outperforms our model. However, when the input context is increased, our model gradually improves, and can synthesize images that are more similar to the groundtruth image. When the full 10 rounds of dialogue are provided, GILL significantly outperforms SD, improving over both CLIP
Table 3: Image generation performance on CC3M [52] and VIST [28] with different text mapping networks.
Model	CC3M	VIST
FID (↓)	CLIP Sim (↑)
Stable Diffusion [49]	13.94	0.598
Ours + Linear	15.50	0.500
Ours + 3-layer MLP	15.33	0.502
Ours + Transformer Encoder	16.30	0.605
Ours + GILLMapper	15.31	0.641
Table 4: GILL image generation results on CC3M [52] with different number of image tokens  $(r)$
r	CC3M	VIST
FID (↓)	CLIP Sim (↑)
1	15.93	0.631
2	15.32	0.629
4	15.32	0.642
8	15.31	0.641
Figure 6: Performance of GILL on VIST generation.
Table 5: Contextual image retrieval on VIST (5 captions, 4 images as input).  $\dagger$  indicates results from [31].
Model	VIST Recall@k(↑)
R@1	R@5	R@10
CLIP ViT-L [43]†	8.8	22.3	29.8
FROMaGe [31]†	18.2	42.7	51.8
GILL (Ours)	20.3	45.0	53.7
Similarity (0.622 to 0.645) and LPIPS (0.723 to 0.714). These results further highlight the efficacy of our model on handling long dialogue-like text inputs.
4.2 Qualitative Results
Finally, one of the more compelling applications of GILL is perhaps its ability to generalize to many different tasks, due to the LLM pretraining and freezing. We showcase several of these capabilities in Fig. 5. In many examples, we observed that GILL is able to outperform retrieval models such as FROMAGe [31] on examples where FROMAGe is unable to retrieve relevant images. GILL is also generally more sensitive to input context compared to Stable Diffusion [49], and can condition on image inputs, in addition to text, to generate more visually and semantically relevant image outputs.
Contextual Image Retrieval In addition to generation, GILL is capable of image retrieval conditioned on image-text inputs. We run GILL on the VIST retrieval evaluation from [31]. We find that GILL performs comparably or better compared to prior approaches (Tab. 5). This shows that the image generation objective does not cause image retrieval performance to deteriorate.
The Effect of Context GILL leverages an LLM backbone, which allows it to inherit some of the LLM's capabilities, such as improved sensitivity to long inputs. Fig. 6 shows that the performance of GILL generally improves with increasing input contexts on VIST [28]. In particular, when 2 captions and 1 image are provided as context, the model significantly outperforms the model with 5 text-only captions, highlighting the value of multimodal context over unimodal context.
Generation-Only Objective We investigate the effect of removing the retrieval loss (Eq. 4) from the training objective. On VIST (5 captions, 4 images), this ablated model achieves CLIP similarity of 0.636 and LPIPS of 0.694, which are comparable to scores of the original model (0.641 and 0.693 respectively). This suggests that the retrieval loss is not necessary for strong performance, although such a model would only be able to generate images and text and not retrieve images. These results also suggest that GILL is not bottlenecked by including the retrieval objective, and that it has sufficient capacity to perform both generation and retrieval.
GILLMapper Module As described in Sec. 3.2, we propose the GILLMapper module, a lightweight transformer model that conditions on [IMG] embeddings and  $q$  learnt embedding vectors. The output maps the LM embeddings into the input space of a text-to-image generation model, enabling image synthesis. We run several baselines to compare effectiveness, comparing our proposed model against (1) a linear layer, (2) a multilayer perceptron (MLP) with LeakyReLU activations, and (3) a 4-layer bidirectional transformer encoder. All models are conditioned on the  $r$  [IMG] token embeddings from the LLM. Our results are presented in Tab. 3. GILLMapper is substantially better than these baseline models at learning the mapping from the frozen LLM to the Stable Diffusion generation model, as measured by Fréchet Inception Distance (FID) [25] on the CC3M validation set (which is a measure of image realism), and CLIP Similarity on VIST. On the VIST evaluation (which is out of distribution from CC3M), the other baselines perform significantly worse than GILLMapper, suggesting that they cannot generalize to longer sequences containing multiple images and texts.
Number of [IMG] Tokens We experiment with varying the number of [IMG] tokens,  $r$  (Tab. 4). As  $r$  increases, generation generally improves, plateauing around  $r = 4$ . We observe that lower values of  $r$  appear to result in worse results, as the inputs to GILLMapper are shorter and less expressive.
We proposed a method of mapping text-only LLMs to strong visual models. This enables them to learn to process arbitrarily interleaved image-and-text inputs, and output generated text, retrieved images, and generated images. We show that it is possible to efficiently learn a mapping between the embeddings of a frozen pretrained LLM and a frozen pretrained image generation model, and that doing so effectively boosts image generation for tasks that require stronger language context dependence. Finally, we also showcased several compelling qualitative results on a variety of multimodal tasks. Our approach is modular, and can benefit from stronger LLMs or visual models released in the future. Scaling up the LLM backbone, image generation backbone, or visual processing model, are promising directions that will likely induce even stronger vision-and-language capabilities.
This work was partially supported by a gift from Cisco Systems, and by ONR N000142312368 and DARPA/AFRL FA87502321015. We thank Wendy Kua for help with the figures. We thank Jared Fernandez, Yutong He, Saujas Vaduguru, Yonatan Bisk, and our anonymous reviewers for feedback and helpful discussions on previous versions of this paper.
[1] Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.
[2] Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201.07520, 2022.
[3] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. CoRL, 2023.
[4] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022.
[5] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005.
[63] Haoxuan You, Mandy Guo, Zhecan Wang, Kai-Wei Chang, Jason Baldridge, and Jiahui Yu. Co-bit: A contrastive bi-directional image-text generation model. arXiv preprint arXiv:2303.13455, 2023.
[64] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022.
[65] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. TMLR, 2022.
[66] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. In CVPR, 2021.
[67] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
[68] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[69] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[70] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Ngai-Man Cheung, and Min Lin. A recipe for watermarking diffusion models. arXiv preprint arXiv:2303.10137, 2023.
[71] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022.
[72] Yufan Zhou, Chunyuan Li, Changyou Chen, Jianfeng Gao, and Jinhui Xu. Lafite2: Few-shot text-to-image generation. arXiv preprint arXiv:2210.14124, 2022.
[73] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards language-free training for text-to-image generation. CVPR, 2022.
GILL relies on an LLM backbone for many of its capabilities. As such, it also inherits many of the limitations that are typical of LLMs. One limitation is the potential for hallucinations [6], where the model generates content that is false or not relevant to the input data. Another limitation of the model in generating text is in repetitions and neural text degeneration [27], where the model generates the same content multiple times. We also observed that the OPT-6.7B model also does not always consistently generate coherent dialogue text.
These limitations may be addressed by techniques that address hallucinations and degenerations in text-only LLMs, or by using improved LLMs that are less prone to these issues. In GILL, we used a 6.7B model. In the future, it will be valuable to scale up the approach with even larger LMs, or those trained with improved objectives [54], instruction finetuning [58] or human feedback [40]. Depending on downstream applications, using models trained explicitly on dialogue data [15] may also be helpful for dialogue capabilities (e.g., deploying multimodal chatbots).
With regards to the visual models, another limitation of our approach is in its limited visual processing. At the moment, we use only  $k = 4$  visual vectors to represent each input image (due to computational constraints), which may not capture all the relevant visual information needed for downstream tasks. These vectors are produced by a frozen pre-trained visual encoder, and so the visual information in the vectors is heavily constrained by the pre-training task. As a result, the model may not always process images correctly or in enough detail to produce accurate or high-quality results. However, this limitation can potentially be addressed in the future by scaling up the visual model, using models with varied pre-training objectives that encode more visual information while still being mappable to the hidden space of the LLM, or using more sophisticated visual mappings [4, 33] that can capture a richer set of visual features. Similarly, we observed during inference that our model sometimes does not generate relevant images for certain types of prompts. We attribute this to our finetuning dataset being CC3M, which is relatively small compared to modern large scale image-text datasets [51]. It is likely that training GILLMapper on an even larger corpus of text data will improve its alignment to the image generation backbone.
One of the advantages of our model is that it is modular, and can benefit from stronger visual and language models released in the future. It is likely that it will also benefit from stronger text-to-image generation backbones, or through finetuning the generation backbone rather than just the GILLMapper module. We leave such scaling explorations for future work.
AI Assistants Recent advances in dialogue based chatbots have sparked interest in using LLMs for interactive conversational applications. GILL is a multimodal language model capable of processing image and text inputs, and producing image and text outputs. These capabilities may enable a wider range of applications. For example, AI assistants which can produce image and text outputs would be able to answer a wider range of queries, providing visual content when necessary to illustrate certain points. Concrete applications may include creative endeavors (e.g., iteratively refining a generated image with instructions), answering questions that benefit from visual outputs (e.g., describing food items), and more. Scaling GILL and refining it with methods such as reinforcement learning from human feedback (RLHF) [32] are promising directions to improve the capabilities of multimodal AI assistant systems.
Disinformation and Harms Aside from the technical limitations detailed in Sec. A, there are broader societal issues that should be considered with the development of generative models of text and images. LLMs have the potential to generate plausible sounding (but false) text [22, 6], propagating disinformation at scale. As GILL uses an LLM backbone, it is also susceptible to these potential issues. Furthermore, as multimodal generative models which can also produce image content, models such as GILL also introduce potential issues with producing even more convincing disinformation through interleaving text with realistic generated images. As GILL makes use of an image generation backbone, it is also susceptible to the risks that typical text-to-image generation models introduce, such as generating false images of real people. These harms may possibly be mitigated by introducing watermarking into generated images [37, 70], or by deploying systems to detect generated images [12].
Bias and Safety GILL makes use of pretrained LLMs and multimodal models (such as CLIP [43] and Stable Diffusion [49]), which are trained on large, noisy, Internet-scraped data (such as LAION-400M [51]). Due to their curation process, these datasets often contain undesired biases, malignant stereotypes (see [8] for a comprehensive discussion on large scaled multimodal datasets). One advantage of GILL is that it is efficient to train and completely modular, allowing its components (i.e., the LLM, visual encoder, or image generator) to be swapped out for other pretrained models (for example, models which have been further calibrated to reduce unintended biases).
Intended Uses GILL is a research prototype which showcases possible capabilities of multimodal language models which can both process and produce image and text outputs. Due to the limitations described above, GILL is not in its current state intended for deployment in practical applications, especially in high risk or sensitive domains without further analysis. At its current model scale (a 6.7B parameter LLM), GILL also lacks many of the abilities of larger language models [9], and applications would likely benefit from increased scaling of the LLM and visual models.
C Deciding to Generate or Retrieve
As detailed in Sec. 3.3, we evaluate several models on the annotated PartiPrompts [65] dataset. Each prompt is annotated with one of two labels: "ret" or "gen", indicating whether image retrieval or image generation produces a more appropriate image for the corresponding prompt. For example, the prompt "a portrait of a statue of the Egyptian god Anubis wearing aviator goggles, white t-shirt and leather jacket, flying over the city of Mars." is labeled as "gen", as there are (understandably) no appropriate images in the CC3M retrieval set, and generation produces a more relevant output. In contrast, "the geyser Old Faithful" is labeled as "ret," as there are very relevant candidate images available for this prompt. We evaluate several models for making this decision on the validation set (Tab. 6), evaluating using F1 score given the class imbalance of the dataset (201 "gen", 110 "ret" in the validation set labels):
1. Baselines: We measure the F1 score of several baseline methods, which provide a lower bound for how well data-driven approaches can do. We find that always retrieving an image, always generating an image, or simply deciding randomly (with a prior proportional to class frequencies) achieve F1 scores of 0.267, 0.389, and 0.451 respectively.
2. Heuristic: We also consider a simple heuristic which considers the maximum cosine similarity of the retrieval embedding against the entire image candidate set (i.e., the training set of CC3M). We run a grid search from 0 to 1 for possible threshold values. Whenever the maximum cosine similarity is above a threshold, we return "ret" and "gen" otherwise. This achieves an F1 of 0.261 - 0.559, depending on the threshold used (a threshold of 0.5 gives F1 of 0.261).
3. Linear classifier: Lastly, we train a linear classifier that takes as input the outputs of the LLM for the [IMG] tokens and the maximum cosine similarity. This classifier is trained with the binary cross-entropy loss over the training set of PartiPrompts annotations. This linear classifier achieves an F1 score of between 0.393 - 0.552, depending on the probability threshold used (a threshold of 0.5 gives an F1 score of 0.547).
We use the linear classifier in our final model, as it requires less hyperparameter tuning compared to the heuristic baseline, and performs comparably on quantitative metrics. During generation of qualitative samples (Fig. 7 and Fig. 5 in the main paper), we observed that the linear classifier generally performed well for many prompts, and decided correctly whether to retrieve or generate.
We present further qualitative samples in Fig. 7. We find that GILL is able to process complex text prompts more effectively than Stable Diffusion for many examples in PartiPrompts [65]. On VisDial [16] dialogue inputs, GILL is able to generate more relevant outputs (as measured against groundtruth images). We attribute these improved results to the stronger text representations of the LLM, and the effectiveness of our GILL Mapper network.
Table 6: Results on PartiPrompts for classifying retrieval or generation.
Method	F1
Always retrieve	0.267
Always generate	0.389
Random	0.451
Heuristic	0.261 – 0.559
Linear classifier	0.393 – 0.552
Human performance	0.851
Table 7: Results on image captioning on MS-COCO (2017) [34] and VQA [24]. For captioning, we report BLEU [41] and METEOR [5] scores. For VQA, we report the accuracy after applying the normalization described in the VQA repo (https://github.com/GT-Vision-Lab/VQA).† indicates our reimplementation.
Model	Captioning	VQA
BLEU-4	METEOR	0-shot
Frozen† [56]	-	-	0.2553
MAGMA [19]	-	-	0.2835
FROMAGe [31]	0.1023	0.2873	0.2851
Ours	0.1059	0.2529	0.3178
E Other Evaluations
E.1 Comparison to Prior Multimodal LMs
We ran evaluations on VQAv2 [24] and image captioning on MS-COCO [34]. The results are presented in Tab. 7. We found that GILL is comparable to models trained with similar compute and data. On VQAv2, we achieve a zero-shot val accuracy of 0.3178, which is slightly better than prior approaches of similar model sizes and compute: FROMAGe [31] achieves a zero-shot accuracy of 0.2851, Frozen [56] achieves 0.2553, and MAGMA [19] achieves 0.2835. For image captioning on the MS-COCO (2017) validation set, GILL achieves a BLEU@4 of 0.1059 and METEOR of 0.2529, which is comparable to FROMAGe (BLEU@4 of 0.1023 and METEOR of 0.2873). GILL is also capable of a wider set of tasks (e.g., generating interleaved image and text outputs) compared to these models.
We note that these scores are lower than SOTA models, as they are usually much larger and trained with significantly more compute and data (e.g., Flamingo [4] uses 23,040 TPU days, BLIP-2 [33] uses 144 GPU days, while ours uses 4 GPU days). Scaling up GILL to similar data and parameter scales to further push its capabilities is an exciting avenue for future work.
E.2 Increasing Context on VisDial