SlowGuess/NeurIPS_2023_OCR huggingface dataset:
GILL leverages an LLM backbone, which allows it to inherit some of the LLM's capabilities, such as improved sensitivity to long input contexts. In the main paper, we showed that GILL can better condition on longer image and text inputs to generate more relevant images for VIST [28]. We run a similar experiment on Visual Dialogue [16], varying the number of dialogue rounds provided as input context to GILL and Stable Diffusion (SD) [49].
The results are presented in Fig. 8. We find that when longer text context is provided to both models, the performance of generating relevant images steadily improves. Interestingly, SD performance plateaus after 6 rounds of dialogue, while GILL continues to improve, outperforming SD when 7 or more rounds of dialogue are provided. These results showcase the improved sensitivity of our model to conditioning on long, dialogue-like text. Despite both approaches using the same image generation backbone, GILL is able to better make use of longer dialogue-text inputs (despite being only finetuned on image caption data).
"A dignified beaver wearing glasses, a vest, and colorful neck tie. He stands next to a tall stack of books in a library.
"A drop-top sports car coming around a bend in the road"
Snow mountain and tree reflection in the lake"
A raccoon wearing formal clothes, wearing a top hat and holding a cane. The raccoon is holding a garbage bag. Oil painting in the style of Vincent Van Gogh."
"a group of penguins in a snowstorm"
"A tornado made of sharks crashing into a skyscraper, painting in the style of watercolor."
Comparison Against Stable Diffusion GILLMapper allows our model to map effectively to the SD image generation backbone, outperforming or matching SD for many examples from PartiPrompts.
Visual Dialogue
Our model can process long, dialogue-like text inputs to generate more relevant images compared to non-LLM based text-to-image generation models.
Figure 7: Further qualitative samples from GILL. It is more sensitive to text inputs due to its LLM backbone, and better at processing complex text prompts.
Figure 8: Performance of our model and Stable Diffusion [49] with increasing context for generating VisDial [16] images. Our model is able to better process long dialogue-like text descriptions.
Table 8: Zero-shot FID [25] on the MS-COCO [34] (2014) validation set. 30,000 random samples are used to evaluate all models.
Model	FID (↓)
GLIDE [38]	12.24
Make-A-Scene [21]	11.84
DALL-E 2 [44]	10.39
LAFITE2 [72]	8.42
Imagen [50]	7.27
Parti [65]	7.23
Re-Imagen [14]	6.88
SD [49] v1.5	9.22
GILL (ours)	12.2
E.3 Image Generation
In addition to our evaluations on VIST [28] and VisDial [16], we also run evaluations on our model's ability to generate images from MS-COCO [34] captions (Tab. 8). We generate images using 30,000 randomly sampled captions from the MS-COCO (2014) validation set, which is the standard evaluation of text-to-image generation models. We report zero-shot FID scores [25] of our model, Stable Diffusion [49] v1.5 (which we use as our backbone image generator), and several other approaches in Tab. 8. For our generation results and SD results, we use a classifier-free guidance scaling factor of 3.0 and 250 DDIM inference steps. On MS-COCO, our approach achieves a worse FID score than SD (9.22 to 12.2). This is likely because this task does not benefit as much from the LLM backbone, which has not been trained on as many image captions as SD (which exclusively trains on caption-like data). These numbers will likely improve further by finetuning GILL on even more text data (including image captions), which will allow our model to align more closely to the input space of the SD image generator.
E.4 Inference Speed
One concern for deploying LLMs is the inference throughput and speed. We benchmark the inference performance of GILL on a single A6000 GPU. Generating text has the same throughput as a regular LM of the same size (i.e., that of OPT 6.7B). The main increase in inference time occurs when the model produces [IMG] tokens. For a batch size of 1, if the model decides to retrieve images, the additional inference overhead is minimal (less than 0.001s on average) as image retrieval is fast, requiring just a single matrix multiplication followed by a max operation. However, if GILL generates an [IMG] token, it takes 3.5s on average for the Stable Diffusion backbone to generate a corresponding image.
Overall, GILL's inference speed is bottlenecked by the frequency of image generation, which is dependent on the application domain. In the case of generating dialogue-like text, we observed that images are usually generated or retrieved once or twice in a natural conversation. Amortized over a long conversation, it does not lead to a significant increase compared to a text-only LLM, though exact numbers would depend on the application.
F Human Annotation on PartiPrompts
In Sec. 3.3 of the main paper, we described the process of annotating PartiPrompts [65] with perexample labels to retrieve or generate. The interface shown to human annotators is shown in Fig. 9. Annotators are tasked to determine which of two anonymized images are (1) more relevant to the provided prompt, and (2) more realistic. We randomize the order of the two images as well (i.e., the output of the retrieval model shows up  $50\%$  of the time as Image A).
Shown below are two images and a description for the images. Please answer two questions:
1. Select the image which matches best with the text. For this task, ignore how realistic the images may look. Please only focus on how appropriate the image content is with respect to the caption.
2. Select the image which is more visually realistic. For this task, ignore how appropriate the image content is with respect to the caption. Please only focus on how visually realistic the image is.
Caption:"bond"
1) Is image A or image B more relevant to the above caption? (ignore image realism for this question)
B is more relevant
Image A
Image B
2) Is image A or image B a more realistic image? (ignore the caption for this question)
A is more realistic