Can MLLMs Perform Text-to-Image In-Context Learning?
Authors: Yuchen Zeng
Wonjun Kang
Yicong Chen
Hyung Il Koo
Kangwook Lee
Venue: COLM (2024)
Keywords: MLLMs, In-Context Learning, Multimodal In-Context Learning, Text-to-Image Generation, multimodal large language models, large language models
TL;DR: We first identify text-to-image in-context learning problem, and introduce a benchmark to assess this capability of multimodal large language models.

Abstract: The evolution from Large Language Models (LLMs) to Multimodal Large Language Models (MLLMs) has spurred research into extending In-Context Learning (ICL) to its multimodal counterpart. Existing such studies have primarily concentrated on image-to-text ICL. However, the Text-to-Image ICL (T2I-ICL), with its unique characteristics and potential applications, remains underexplored. To address this gap, we formally define the task of T2I-ICL and present **CoBSAT**, the first T2I-ICL benchmark dataset, encompassing ten tasks. Utilizing our dataset to benchmark six state-of-the-art MLLMs, we uncover considerable difficulties MLLMs encounter in solving T2I-ICL. We identify the primary challenges as the inherent complexity of multimodality and image generation, and show that strategies such as fine-tuning and Chain-of-Thought prompting help to mitigate these difficulties, leading to notable improvements in performance. Our code and dataset are available at https://github.com/UW-Madison-Lee-Lab/CoBSAT.

Published as a conference paper at COLM 2024
Can MLLMs Perform Text-to-lmage In-Context Learning?
Yuchen Zeng**
Wonjun Kang* 2,3
Yicong Chen'
Hyung Il Koo and Kangwook Lee!
University of Wisconsin-Madiion
FuriosaAl
Seoul National University
"Ajou University
Abstract
The evolution from Large Language Models (LLMs) Multimodal Large Language Models (MLLMs) has spurred research into extending In-Context Learning (ICL) to multimodal counterpart. Existing such studies have primarily concentrated on image-to-text ICL. However, the Text-to-lmage ICL (I2Í-ICL), with its unique characteristics and potential applications, remains underexplored. To address this gap, we formally define the task of T21-ICL and present CoBSAT, the first 121-ICL benchmark dataset, encom- passing ten tasks. Utilizing our dataset to benchmark six tatte-of-theaart MLLMs, we uncover considerable difficulties MLLMs encounter in solving T2I-ICL. We identify the primary challenges as the inherent complexity of multimodality and image generation, and show that strategies such as fine- tuning and Chain-of-Thouuht prompting help to mitigate these difficulties, leading to notable improvements in performance. Our code and dataset are available at https::///t thub. -on-Lee
1
Introduction
(d) Text-to-Image ICL (Our Focus)
(a) Textual ICL (Brown et al., 2020)
Example Application Interior Decor Design
Tree
Outpo
bypol
Cappu
thankoc
anerci
hellox
boejour
mint
menthh
Living
MILM
(b) Visual ICL (Bar et al., 2022)
Example Application Product Conceptualization
Inewr
Jopor
Chrgst
Phone
(c) Image-to-Toe -ene- ICL (Alayrac et al., 2022)
Example Application 3: Cartoon Character Design
Airihila The crcousst
Tukkk
Optimistic
Overweight
coral-pink stanfish:
Amegrant
MILM
pistern
sponge:
octopus:
Figure 1: Comparison of various In-Context Learning (ICL) settings. (a) Textual ICL, where both the input and output in each example are textual. (b) Visual ICL, where both input and output in each demonstration are presented images. (c) Image-to- Text ICL (I2T-ICL), featuring images as input and texts as output in each demonstration. (T2I-ICL, our focus), which involves text input and image output in each demonstration. T21-ICL introduces greater complexities and presents different potential applications. The examples in (d) provide three potential applications of T2I-ICL, with the output generated using ChatGPT-4 (OpenAI, 2023) with DALL-E3 3 (Betker et al., 2023) capabilities.
(d) Text-to-Image ICL
"Fqual contribution. Emails: yzeng58wwisc.co edu, kangwj 995@furiosa.
1

---

Published as a conference paper at COLM 2024
Object-nnference Tasks
Altribute-Inference Tasks
Jatent Yar
Prompt
Outgot
atrot Var
Prompi
Output
White:
Bluse
Red:
Black
Chaäc
Cupt
Box
Colle-
Caluell
Cup
Green:
Purple:
Orange:
Pink
Leaf:
Can
Book:
Beacle
Clacier:
Volcano
Zebra:
Tiges:
Barkgprande
Back Backgrundd
Zebra
Cave:
Space:
Waterfall:
Scalloor
Bind:
Fig:
Monkey:
Apple
Lagu:
Origami:
Style-
Stykei
Apple:
Singg
Sleep
Adiun--
Arlon-IE
Sleep:
Flys
Sherp:
Ball
Metali
Leathen
Wood:
Denim
Busc
Cup:
Apple
Testure
Teshur-11
Wicken
Plastic
Papen
Wood
Chalin
Hall:
Figure 2: Overview of example prompts in the CoBSAT benchmark. CoBSAT covers five themes: color, background, style, action, and texture, each with different emphases: object-inference and attribute-inferenee In object-inference tasks, the attribute (e-g,, color) is directly provided in the textual input, and the model is required to infer the object (e gegg, car) from the images. In other words, the latent variable (denoted as "Latent Var." in the figure) of object-inference tasks is the object. Conversely, in attribute-inference tasks, the object is specified in the text. The model is tasked with inferring the attribute from the images in the demonstrations, i.e., the attribute serves as the latent variable serribute-inferenne tasks.
In the rapidly evolving landscape of artificial intelligence, Multimodal Large Language Models (MLLMs) (Ge al., 2023b; Koh et al., 2023; Sun et al., 2023c; OpenAI, 2023; Liu et al., 2023a; Bai et al., 2023b; Gemini Team Google: Anil et al., 2023; Li et al., 2023; Anthropic, 2024) extend the frontier of Large Language Models (LLMs) (Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020; OpenAl, 2023; Touvron et al., 2023) by handling not only text but also images, videos, and audio. This multimodal capability enables MLLMs to undertake complex tasks, integrating visual, auditory, and textual cues. The versatility of MLLMs makes them powerful tools in AT, offering context-rich interpretations across various domains.
In-Context Learning (ICL) (see Figure 1(a)) is a prevalent technique that enables predictions based on context through a sequence of input-output pairs, termed demonstrations, without requiring any model parameter updates. This capability was initially identified and applied by Brown et al. (2020) and has since become widely used standard prompt engineering method to enhance LLM inference performance for various downstream tasks. This method has been applied in computer vision to produce output images contextually aligned with
2

---

Published as a conference paper at COLM 2024
provided image-image pair examples, termed Visual ICL (V-ICL) (see Figure 1(b)) (Bar et al., 2022; Wang et al., 2023a). In another development, Tsimpoukelli et al. (2021) in- troduced Multimodal ICL (M-ICL) for the first time for image-to-text generation tasks, including applications such as visual question answering and image captioning. Unlike ICI, which is exclusively text-focused, and V-ICL, which is solely image-oriented, M-ICL uses demonstrations that incorporate samples from two modalities.
The majority of existing M-ICL work (Isimpoukelli et al., 2021; Alayrac et al., 2022; Mon- ajatipoor et al., 2023; Chen et al., 2023b; Zhao et al., 2023) has mainly centered on the performance of image-to-text tasks, the goal of which is transforming high-dliensional, image-based input into low- lowddimesioona,, text-based output. However, when the roles are re- versed, models can exhibit significantly different performance characteristics. To distinguish between these two tasks, we refer to M-ICL for image- smage-to-texx to-text generation as Image-to- Text ICI. (I2T-ICI.) (see Figure (()), and M-ICL for text-to-image generation as Text-to- Text-to-Image ICL. (I2I-ICL) (see Figure 1(d)), with the latter being the focus of our work. It is important to note that potential applications of T21-ICL are completely different from 12T-ICL, which include areas like product design and personalized content creation.
Our Contributions. We summarize our main contributions as follows.
T2I-ICL. Our work first identifies the important yet underexplored ICL setting on text-to-image generation, termed T2I-ICL.
Identifying an Important Problem:
Introducing the CoBSAT Benchmark. To systematically assess the T21-CL capability of MLLMs, we introduce comprehensive benchmark featuring ten tasks across five different themes Color, Background, Style, Action, and Texture, which named as CoBSAT (see Figure 2).
Benchmarking MLLMs in T2I-ICL. We utilize our dataset to evaluate the T2I-ICL capa- bilities of fen state-of-the-art MLLMs. This includes Emu (Sun et al., 2023c), GILL (Koh et al., 2023), SEED-LLaMA (Ge et al., 2023b), Qwen-VL. (Bai et al., 2023b), Gemini (Gemini Team Google: Anil et al., 2023), Claude (Anthropic, 2024), and GPT-4V (OpenAI, 2023), which are elaborated upon in the main paper, alongside Emu2 (Sun et al., 2023a), LLaVA- 1.5 (Liu et al., 2023a), and LLaVA-NeXT (Liu et al., 2024), detailed in the appendix. We peerformaneeee
observe that the T2I-ICL performance of these models is significantly influenced by their
respective training paradigms. Among them, SEED-LLaMA, Qwen-VL, Gemini, Claude, and GPT-4V demonstrate the capability to perform T2I-ICL. Yet, except for Gemini, their accuracy rates hover around or fall below 60% in most scenarios.
Understanding Challenges in T2I-ICL. We then investigate the key factors contributing to the underperformance of MILMSs in T2I-ICL. Our findings point to two principal challenges: (i) the intrinsic complexity involved in processing multimodal data, and (ii) the inherent difficulties associated with the task of image generation.
Enhancing MLLMs' T21-ICL Capabilities. To augment MLLMs' T21-ICL capabilities, we delve into various potential techniques. Our study demonstrates that fine-tuning and Chain-of-Thought (CoT) (Wei et al., 2022) significantly boost T2I-ICL performance.
2 Related Works
Unimodal ICL. Ever since Brown et al. (2020) demonstrated that language models are in-context learners (see Figure 1(a)), there has been substantial interest comprehending this capability, both empirically (Liu et al., 2022; Min et al., 2022b; Chen et al., 2022; Mishra et al., 2022; Lampinen et al., 2022; Garg al., 2022; Hendel al., 2023) and theoretically (Xie et al., 2022; Wies et al., 2023; Akyürek et al., 2023; Von Oswald et al., 2023; Bai et al., 2023c; Ahn et al., 2023; Zhang et al., 2023b). Textual ICL (T-ICL) enables the adaptation downstream tasks simply by providing few illustrative examples, bypassing any need for updating model parameters. The concept of V-ICL is then employed in computer vision, starting with the introduction of visual prompts (see Figure 1(b)). The pioneering works by Bar et al. (2022); Wang et al. (2023a) propose to automatically generate output images that are contextually aligned with provided examples. Specifically, Bar et al. (2022) developed
LLMs to
3

---

Published as a conference paper at COLM 2024
method that combines three images example input, its corresponding output, and a query- - into a single composite image. In this layout, the example input is placed in the upper left, the example output in the upper right, the query image in the bottom left, and the bottom right patch is left blank for output construction via an image inpainting model. Bar et al. (2022) demonstrated the effectiveness of V-ICL in tasks like edge detection, colorization, inpainting, etc. Unlike T-ICL and V-ICL which are limited to handling unimodal inputs, M-ICL integrates demonstrations encompassing both text and images.
Image-to-Text ICL. Most existing work on M-ICL focuses image-to-ter text generation, i.e., I2T-iCL (Tsimpoukell et al., 2021; Alayrac et al., 2022; Monajatipoor et al., 2023; Chen et al., 2023b; Zhao et al., 2023). In particular, Tsimpoukelli et al. (2021) were the first to extend ICL from the text domain to the multimodal domain, focusing on image-to-text generation such as visual question-answering (see Figure I()). Alayrac et al. (2022) introduced Flamingo, an MLLM that achieves good performance in variety of image and video understanding tasks using I2T-ICL with 32 demonstrations, implying the efficacy of I2T-ICL in performance enhancement in their model. Concurrently, efforts have been made to develop datasets specifically designed for evaluating the I2T-ICI, capability of MLLMs (Zhao et al., 2023).
Text-to-Image ICL, There are limited attempts to evaluate MLLMs based on their T2I-ICL capabilities. A notable exception is concurrent research by Sun et al. (2023a). They evaluated the performance of their model on T2I-ICL. with DreamBooth dataset (Ruiz et al., 2023). However, it is important to note that the DreamBooth dataset, primarily developed for fine-tuning models to modify image contexts, was not specifically designed for T2I-ICL applications, making it more challenging and mostly focusing on background altering, The complexity, as seen in style transfer examples that emulate artists like Vincent van Gogh or Michelangelo, can pose challenges even for human interpretation.
MLLMs. Recently, there has been a surge in the release of MILMs, which are designed to address more challenging multimodal tasks, thereby enabling the perception of images, videos, and audios (Li et al., 2022; Alayrac et al., 2022; Hao et al., 2022; Laurençon et al., 2023; Huang et al., 2023b; Peng et al., 2023b; Li et al., 2023; Ge et al., 2023b; Koh et al., 2023; Zhu et al., 2023a; Sun et al., 2023c; Zheng et al., 2023a; OpenAI, 2023; Liu et al., 2023b;a; Bai et al,, 2023b; Sun et al., 2023a; Driess et al., 2023; Gemini Team Google: Anil et al., 2023; Borsos et al., 2023; Huang et al., 2023a; Chen et al., 2023a; Zhang et al., 2023a; Anthropic, 2024).
Since our main focus is T2I-ICL, we only consider models capable of processing both text and multiple images. We consider two types of MLLMs: (i) proficient in generating both text and images, including Emu (Sun et al., 2023c), Emu2 (Sun et al., 2023a), GILL (Koh et al., 2023), and SEED-LLaMA (Ge et al., 2023b), and (ii) those limited to text generation, including GPT-4V (OpenAI, 2023), LLaVA-1.5 (Liu et al., 2023b), LLaVA-NeXT (Liu et al., 2024), Gemini (Gemini Team Google: Anil et al., 2023), Claude (Anthropic, 2024) and Qwen- VL. (Bai et al., 2023b). For text-only MLLMs, we evaluate their capacity to infer visual outputs by prompting them to describe the anticipated image. Conversely, for MILMMs capable of image generation, we not only elicit image outputs but also ask for descriptive text, ensuring an apple-to-apple comparison with text-only models.
Owing to page constraints, we provide more detailed overview of related works in Sec. B.
3 Dataset: CoBSAT
We start by describing the definition of in-context learning. Consider a task with data (x, y), where input E X' output y ~ fo(x), where distribution fo is parameterized by latent variable 0 E @ We denote the model by M. For in-context demonstrations, we are given N input-output pairs (x)m and one test query IN In-context learning make the prediction by incorporating these demonstrations {(x,,yn)) and the test query XN+1 in the prompt. The prediction made by model M is formulated as i = M(2,, 1/1 y2........ NN, yN, : In this work, we mainly focus on scenarios where the input x is textual data and output y corresponds to image. We use notation
4

---

Published as a conference paper at COLM 2024
[Image: description] to denote an image corresponding to the text description. For instance, [Image: red car] refers to an image depicting a red car.
Dataset Structure. We begin by outlining the structure of our dataset, which evaluates whether models are capable of learning the mapping from textual input to visual output, based on the given in-context demonstrations. For instance, task Color-I in our experiment involves generating an image of an object of particular color, where the object to be drawn is not explicitly stated in the text query IN+1- The information of the object is instead implicitly contained in & (and hence in y/'s since y/: ~ fx( for all i = I,..., N), which can be learned from the demonstrations. An example prompt when 0 = "car" is
example
example
query
red: [Image: red car]
blue;
[Image: blue car]
pink:
y1
X2
Ideally, MLLMs can learn the object @ from the context, and generate an image of a pink car.
CoBSAT comprises ten tasks, divided into two categories: (i) object-inference tasks, which give the attributes (e.g., color, texture) in the text input and require identifying objects (e-g., car, cup) from images, and (ii) attribuleinnference lasks, which provide the object to be drawn in the text input but require identifying the common attribute from the images (see Figure 2). Each task has predefined lists for text inputs and latent variables, denoted as X and O each containing ten distinct items. For instance, in the Color-I task, the predefined list for the latent variable (i.e., the object) is @ fleaf, hat, cup, chair, car, box, book, ball, bag, apple), and the predefined list for the text input (i.e., the attribute) is X = [yellow, white, red, purple, pink, orange, green, brown, blue, black). The predefined lists for all tasks are p
provided in Sec. C. In our experiment, for each specified number of shots (i.e., 2, 4, 6, our experiments), we create 1,000 prompts per task. This is accomplished by randomly selecting latent variable from the predefined list O and sequence of textual inputs (xn) from xN+11 Then, we pair each textual input Xn with the corresponding image yn ~ (xx) to instruct in-context demonstrations.
Data Collection. For each task, we gather one image for every possible pairing of the textual input x E X' and latent variable 0 E O, resulting in | x |O| = 10 x 10 = 100 images for each task. For instance, for task Color-I, we collect an image of a red car to correspond to the case where x ="red" and 0 = =carr, and likewise for other images. It is noteworthy that the tasks with the same theme, such as Color-I (object-inference task) and Color-II nntrrbbut-iinfernne task), share the same images. In addition, all object lists and attribute lists, along with the images, are carefully selected so that LLaVA can correctly identify the specified objects and the corresponding attributes (i.e., color, background, texture, action, and style) of the images. This ensures an appropriate level of difficulty for T2I-ICL tasks and allows LLaVA In total, we collect 500 images from the web and DALL-E 3 (Betker et al., 2023). We then construct in-context prompts for 2, 4, 6, and shots as previously described, with each shot resulting in 10,000 prompts.
perform reliable evaluations on generated images.
4 Methodology
MLLMs. In our study, we assess the performance of models in T2I-ICL, specifically Emu (Sun et al., 2023c), Emu2 (Sun et al., 2023a), SEED-LLaMA (Ge et al., 2023b), and GILL (Koh et al., 2023), which can generate images. In addition to image generation sce- narios, we instruct the text-only generation models - Qwen-VL (Bai et al., 2023b), LLa 1.5 (Liu et al., 2023a), LLaVA-NeXT (Liu et al., 2024), Gemini (Gemini Team Google: Anil et al., 2023), Claude (Anthropic, 2024), and GPT-4V (OpenAI, 2023)), together with aforemen- tioned models capable of generating images, to generate textual descriptions for expected images. This assesses if they learn the mapping from low-dimensional textual input to high-dimensional visual output based the demonstrations. An extensive review of these
5

---

Published as a conference paper at COLM 2024
Example
Inference Stage
Evaluation Stage
1 Eon
Eval Model (VLM/MLLM)
Promupt
MLLM
Prompe
chair
Desciip Cen
sasi propede you arthi fert consples nette wespes Compleve comppie descriitinn of lmsge
Wiack
Hack duir os wiite Aoogemend
Iioor
Eval Model (VLM/MLLA)
Macc
MLLM
orjert?
chair
Clustr
Procmpt
Figure 3: Benchmarking pipline for MLLMs in T2I-ICL with CoBSAT. (i) For MLLMs with image generation capabilities, we feed prompts from our dataset into the MLLM under evaluation to prompt image generation. If the MLLM accurately interprets the text-image relationship in the provided demonstrations, it should produce an image of a "black chair." " To verify this alignment, we employ one evaluation model, could be either Vision- Language Model (VLM, e.g,, CLIP) or an MLLM adept at visual question answering (e.g,, LLaVA). This allows us to determine whether the generated image accurately corresponds to the target label. (ii) For MLLMs that do not generate images, we modify the process by instructing the MLLMs to describe the image textually, following the same evaluation criteria as in the image generation scenario.
MLLMs, and detailed information about the prompts used for each model, are provided in Sec. A and Sec. D.1, respectively.
In particular, since LLaVA models are primarily designed for visual question answering (Liu et al., 2023a; 2024) and are tailored to work with single-image inputs accompanied by questions, they do not perform well on T2I-ICL tasks as expected. Furthermore, Emu2 requires a significant amount of memory, especially for cases with a large number of demonstrations, which limits our ability to obtain comprehensive results due to resource constraints. Therefore, we defer the results of LLaVA models, as well as the partial results obtained for Emu2 in two-shot and four-shot cases, to Sec. F. In the main body of the paper, we primarily focus on discussing the other seven models.
Evaluation. Our evaluation pipeline is depicted in Figure 3, where we leverage both VLM and MLLM to assess whether the generated images or descriptions accurately represent the intended objects (e.g., "car" " in the first example in Figure 2) and attributes (e.g., "red" in the same example). Specifically, we employ CLIP for its proficiency in vision-and-languaag tasks (Hessel et al., 2021; Ruiz et al., 2023), and MLLMs including LLaVA, Qwen-VL, and Gemini to determine the accuracy of the generated content. For CLIP's evaluation, we identify the main object and attribute in the generated content by calculating the similarity between the embeddings of the generated content and the embeddings of all entries within our object and attribute lists. The items with the highest similarity are deemed the predicted labels. In the case of MILMs, the generated content is embedded into the input, prompting MLLMs to identify the main object and attribute in the generated content, which are then assigned as the predicted labels. We then measure the accuracy of these predictions against the true labels to determine the correctness of the generated content.
In Sec. E, we compare these evaluation models terms of alignment with human evaluation, and find Gemini > LLaVA-1.5 CLIP > Qwen-VL in terms of alignment. Since Gemini is not open-sourced and there is a high correlation between the accuracies of VLa15- and Gemini, we use free and open- open-sourced LA-15 for all accuracy evaluations in our paper, unless otherwise stated. Additionally, we find that LLaVA-1.5 accurately identifies the correct object and attribute for all images in dataset, ensuring the reliability of our evaluations. We provide more details such as prompts utilized for evaluation in Sec. D.2.
Benchmarking MILMs in T2I-ICL
We visualize the T2I-ICL performance of the considered MILMss in Figure 4.
6

---

Published as a conference paper at COLM 2024
Subir
Eodail
Godont
S-lait
Dikes
Backgomeel
Stybe]
Artionel
Tisture
Da
(a) Accuracy of generated images on object-inference tasks in CoBSAT.
E-bot
O-fine
S-lott
OokonT
Jackgrand--l
Style
Actom-ll
Testane1I
CIZZ
CLL
(b) Accuracy of generated images on attribute-inference tasks in CoBSAT.
2-hot
6-hoot
B-boct
Cluee
Harkwwandd
Skylel
Artond
Naaurol
ou
Owe
oa
caro
Ba
CPPT
(c) Accuracy of generated image descriptions on object-inference tasks in CoBSAT
2
6-sbot
Sccbot
Glell
Stya-ll
Actiael
GBL
OFT d
Clock
Ove
(d) Accuracy of generated image descriptions on attribute-inference tasks in CoBSAT
Figure 4: T2I-ICL performance of MLLMs on CoBSAT with 2,4,6,8 demonstrations.
Assessing Generated Images. In terms of image generation, we focus on the three MLLMs that have this capability: Emu, GILL, and SEED-LLaMA. Among these, SEED-LLaMA significantly outperforms the others, evidenced by Figure 4(a) and (b), where it attains accuracies exceeding or nearing 20% across various tasks. Notably, on the Color-I task, SEED-LLaMA reaches an impressive 68% accuracy. In contrast, Emu and GILL exhibit low performance, achieving accuracies around or even below 10%.
GILL's limited performance can be attributed to its training paradigm, which is not opti- mized for tasks requiring a unified understanding and generation of multimodal content (Ge et al., 2023b). Specifically, this limitation stems from its training that omits interleaved image- text data and the absence of an image generation model during its training process (Koh et al., 2023). In contrast, SEED-LLaMM benefits from instruction fine-tuning across broad range of datasets, including both multimodal and text-to-image generation datasets such as Instructpix2pix (Brooks et al., 2023), MagicBrush (Zhang et al., 2024), Journey DB (Sun et al., 2024), DiffusionDB (Wang et al., 2023c), LAION-Aesthetin VIST (Huang et al., 2016). Emu, on the other hand, has been fine-tuned exclusively on the LLaVA dataset (Liu et al., 2023b) in the context of image-text tasks. This expansive and varied instruction fine-tuning likely accounts for SEED-LLaMA's enhanced performance in T2I-ICI. tasks when compared to Emu.
(LAION, 2022), and
7

---

Published as a conference paper at COLM 2024
Model
Shot
Method
Object-inference
Attribte-Inference
Colord
Background-1
Style1|
Action-
Texture4
Background-Il Style-II
Action-l
Testure-l
204
360
.725
Gemini
T-IC1.
.979
.077
.692
.895
764
-100
.645
468
T2H-CL
908
340
-737
861
709
484
818
T-IC1.
.988
965
A88
965
922
2222
.780
.35
783
.812
Table 1: Comparison of T2I-ICL v.s. T-ICL accuracy (see Table 7 for the full version). To perform THICL on our dataset, we replace all images in the prompts with their corresponding descriptions. Underlined numbers indicate the highest accuracy achieved for each model and task across various shot numbers, while bold numbers indicate the highest accuracy for each specific combination of model, task, and shot count.
Assessing Generated Image Descriptions. Figures 4(c) and (d) reveal that Gemini, Qwen- VL, Claude, and GPT-4V stand out by significantly surpassing other MLLMs in most tasks. It is observed that MLLMs with image-generation capabilities often struggle with generating image descriptions. Among these leading models, Claude, Qwen-VL and GPT-4V show comparable results, whereas Gemini outperforms all of them. Given the lack of detailed information on the training datasets and paradigms for Gemini, Claude, and GPT-4V, our analysis can only extend to Qwen-VI. Notably, Qwen-VI. benefits from pretraining on a broader dataset than Emu, GILL,, and SEED-LLaMA, contributing its enhanced performance (Bai et al, 2023b).
Impact of Number of Demonstrations. An interesting observation from Figure 4 is the lack of consistent pattern in how performance is influenced by an increase in the num- ber of demonstrations. For example, the accuracy in generating image descriptions for models such as Emu and Qwen-VL. first increases and then decreases with an increasing number of demonstrations generally. Conversely, SEED-LLaMA's accuracy first decreases and then increases. This non-monotonic performance trend with growing number of demonstrations can potentially be attributed to two factors. Firstly, with a higher number of demonstrations, there may be insufficient number of pertaining samples featuring the corresponding number image inputs. Secondly, existing evidence indicates that an increase in demonstrations does not necessarily correlate with enhanced performance (Xie et al., 2022; Brown et al., 2020; Lin & Lee, 2024). Brown et al. (2020) demonstrate that for some datasets (e.g., LAMBADA, HellaSwag, PhysicalQA, RACE-m, CoQA/SAT analogies for smaller models), GPT-3's zero-shot performance may surpass one-shot performance. Similarly, Xie et al. (2022) found that zero-shot scenarios can sometimes outperform few-shot ones, although performance tends to recover with the addition of more examples. Lin & Lee (2024) provided a theoretical explanation for this phenomenon by considering in-context learning process that involves both task retrieval and task learning.
We offer a more in-depth analysis in Sec. F.1, which delves further into the discussion above, and additionally (i) explores the impact of textual and visual information on predictions, (ii) investigates the performance MLLMs in accurately generating the objects and at- tributes, respectively, and (iii) presents results for more challenging variant of the CoBSAT benchmark.
Understanding Challenges in T2I-ICL
In Sec. 5, we observe that most MLLMs still face challenges in performing T2I-ICL effectiveel Notably, SEED-LLaMA, Gemini, and Qwen-VI, are notable free models, each capable of performing T2I-ICL tasks; SEED-LLaMA performs well for image generation scenarios, whereas Gemini and Qwen VL specialize in image description generation scenarios. There- fore, unless otherwise stated, our subsequent investigations concentrate these three models, specifically utilizing SEED-LLaMA for image generation scenarios and Gemini and Qwen-VL for image description generation.
In this section, our goal is to understand the main difficulties leading to this suboptimal performance in T2I-ICI. We hypothesize that the primary difficulties lie in (i) the complexity
8

---

Published as a conference paper at COLM 2024
Model
Shot
Precise Testual Imputs
Object
Task
Altibuue- Inference Task
Color-I
Bickgroundd))
Style-
Action-]
Testure-
Color-ll
Background-0
Style-1
Action-1l
Testure-11
0
.230
.366
.256
.254
.275
552
.316
.255
210
.88
341
163
877
200
305
.69
318
.281
J61
.866
.88
.411
.299
278
.268
Table 2: Accuracy comparison: with or without providing precise textual inputs (see Table for the full version). Bold numbers represent the highest accuracy for each task and shot count, comparing scenarios with and without descriptive textual inputs. Underlined numbers indicate the highest accuracy for each task across various shots.
Model
Shot
Fine-tuned
Object-inference Task
Attribute-nnferenc Task
Color-l
Beckground11 Style-1
Action-1
Texture-1
Color-If
Backgread-11 Style-f
Action-If
Texture-f
540
236
248
A12
332
276
244
112
233
224
Qwen-VI.
.892
244
212
.866
.532
.816
344
.48
.520
.284
680
.92
.88
228
536
.12
448
240
320
420
.866
.04
216
.812
.88
.696
308
088
.866
.400
Table 3: T2I-ICL accuracy comparison of pretrained-onyyy versus fine-tuned (FT) MLLM (see Table for the full version). Underlined numbers denote the highest performance achieved across different methods and shots for each task, while bold numbers indicate the top performance for each shot across various methods within their tasks.
inherent to multimodality, and (ii) the intrinsic challenges of the image generation task itself, which might be independent of the T2I-ICL process. We test these hypotheses as below.
Is Multimodality Primary Challenge in T2I-ICL? The low performance of MLLMs in T2I-ICL is in contrast to the impressive results their underlying LLM demonstrated in T-ICL (Touvron et al., 2023; Bai et al., 2023a). To study whether multimodality is one primary challenge for T2I-ICL, we consider textual version of our tasks by replacing every image in the prompts with corresponding detailed descriptions, which are initially created by LLaVA and ChatGPT and reviewed and updated by humans. Results in Table show that T-ICL significantly improves the accuracy, especially in the 4-shot scenario. This improvement is also observed in the performance of Qwen-VL. and SEED-LLaMA. For an in-depth exploration of the performance of Qwen-VL. and SEED-LLaMA, detailed experimental settings, and comprehensive discussion, refer to Sec. F.2.1. These findings validate our hypothesis that multimodality a principal challenge in T2I-ICL.
Is the Image Generation Primary Challenge in T2I-ICL? We conduct experiment with 0, 2, and 4-shot image generation tasks, with textual inputs updated as precise labels. For example, in the initial scenario from Figure 2, the terms "White," "Blue," and "Red" are updated to "White car," " "Blue car," and "Red car," respectively. The results, as shown in Table 2, reveal that even when precise textual inputs are provided, the accuracies of SEED- LLaMA remain below 50% in most scenarios, maintaining a similar relative performance across different tasks to scenarios without these inputs. This indicates that the task of image generation itself poses a significant challenge for current MLLMs, contributing to their underperformance on the CoBSAT dataset. Similar investigations with Emu and GILL yield consistent conclusions (see Sec. F.2.2).
7 Enhancing MLLMs' T2I-ICL Capabilities
In the previous sections, we observed the suboptimal performance of MLLMs in executing T2I-CCL and investigated the primary challenges involved. This section delves into exploring techniques that could potentially enhance the performance of MLLMs in T2I-ICL. Additional details on our experiments, including choices of hyperparameters, prompt templates, results of other MI.MMs, and other interesting technique explorations, are provided in Sec. F.3.

---

Published as a conference paper at COLM 2024
Model
Shot
Objec-Infferncee
Attibute-Inferenc
Colon-I Background-1
Style-1
Action-l
Texture-
Background-II
Style-If
SBID-LLaMA
389
105
242
.77
Table 4: Accuracy comparison between T2I-ICL with v.s. without CoT (see Table 10 for the full version). Numbers in bold highlight the highest accuracy achieved for each model, number of shots, and task, and underlined numbers indicate the highest accuracy achieved for each model and task across different numbers of shots.
Fine-tuning MLLMs CoBSAT. Building on the work of Min et al. (2022a), which demonstrates that tuning models on a collection of ICL tasks enables them to learn new tasks in context at test time, we fine-tune two instances of Qwen-VL, one 2-shot dataset and the other on a 4-shot dataset, and then compare their performances with their the T2I-ICL test set. Note that all objects and attributes in the test set are not present in the training set. The results are summarized in Table 3. The results indicate a significant improvement in Qwen-VL's T2I-ICL performance post fine-tuning. A similar trend is observed with SEED-LLaMA, as discussed in Sec. F.3.1. This suggests that fine-tuning MLLMS on T2I-ICL dataset enhances T2I-ICL capability of MLLMs. Furthermore, more challenging training test dataset split is considered in Sec. E331 to study the generalizability of the fine-tuned models in terms of T2I-ICL.
non-fine-tuned counterparts
Intergrating Chain-of-Thought with T2I-ICL. Another widely utilized method in prompt engineering is Chain-of-Thoughh (CoT) (Wei et al., 2022). This approach involves incor- porating a simple instruction, such as "let's think step by step, prompting the model to sequentially generate concise sentences that outline the reasoning process, commonly referred to as reasoning chains or rationales. The chains are subsequently embedded into the subsequent prompt to obtain the final answer. In this experiment, we investigate the impact of integrating CoT on the T2I-ICL performance MLLMs. The results are reported in Table 4. With the integration of CoT, SEED-LLaMA shows significant improvement in T21-ICL performance across all ten tasks in the 4-shot scenario. Similar improvement is observed for Gemini, Sec. F.3.2.
Conclusion and Future Works
In this work, we identify an important yet underexplored problem - T2I-ICL, and explore the capability of MILMS to solve it. To facilitate this investigation, we introduce CoBSAT, comprehensive benchmark dataset. Our experimental evaluation of MILMSs on this dataset reveals that many MLLMs have difficulty in effectively performing T2I-ICL. We identify two key challenges in T2I-ICL: (i) the integration and understanding of multimodal information; and (ii, particularly for image generation models) the actual process of image creation. To improve MLLMs' performance in T2I-ICL, we carry out additional experimental studies, which suggest that fine-tuning and CoT can substantially enhance T2I-ICL capabilities.
As we identify T2I-CCL as important problem for the first time, many interesting questions remain open. First, the impact of demonstration selection on T2I-ICL performance yet to be fully understood. Furthermore, the application of other prevalent prompt engineering techniques to T2I-ICL remains open. While our dataset only covers basic themes, we identify expanding the themes of our dataset and extending for image editing tasks as two interesting future directions. For more in-depth discussion, please refer to Sec. G.
Acknowledgement
The work of Kangwook Lee is supported in part by NSF CAREER Award CCF-2339978, Amazon Research Award, and a grant from FuriosaAL. We would like to express our
10

---

Published as a conference paper at COLM 2024
appreciation to Prof. Dimitris Papailiopoulos, Hanrong Ye, Changho Shin, Mu Cai, and anonymous reviewers for their insightful comments.
References
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. CM3: causal masked multimodal model of the internet. arXiv preprint arXio:2201.07520, 2022.
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. arv preprint arXiv:230600029: 2023.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou, What learn- ing algorithm is in-context learning? Investigations with linear models. In International Conference on Learning Representations, 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In Advances in Neural Information Processing Systems, volume 35, pp. 23736, 2022.
Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku. 2024. URL hatps: semanticscholar. httpo/ooaarrrrrrr//oo///e httg/CoopusD:22622222000
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. OpenFlamingo: An open- source framework for training large autoregressive vision-language models. arviv preprint arXiv:2308.01330 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint ar arXio:2309. 16609, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VI: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint ar arXi:2238..12940 3(1), 2023b.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisti- cians: Provable in-context learning with in-context algorithm selection. arviv preprint arXiv:2306.04637 2023c.
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. ar Xiv preprint ar arXit:2312.007888 2023d.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1728-1738, 2021.
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. In Advances in Neural Information Processing Systems, volume 35, pp. 25005-250177 2022.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cddn. openai. com/papers/iall-e-.. pdf, 2023.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. AudioLM: a language modeling approach to audio generation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
11

---

Published as a conference paper at COLM 2024
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Sae- hoon Kim. COYO-700M: Image-text pair dataset. https://gi thub. hww.iitt...c.oo.eeeeeeoemeo..e.e homm/akkabbainn/ coyo-dataset, 2022.
Mu Cai, Zeyi Huang, Yuheng Li, Haohan Wang, and Yong Jae Lee. Leveraging large language models for scalable vector graphics-driven image understanding. ar arxiv preprint arXiv:2306.06094, 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, Wen Wang, Siqi Zheng, et al. LauraGPT: Listen, attend, understand, and regenerate audio with GPT. ariiv preprint ar arXio:2310.04673, 2023a.
Shuo Chen, Zhen Han, Bailan He, Mark Buckley, Philip Torr, Volker Tresp, and Jindong Gu. Understanding and improving in-context learning on vision-language models. arXiv preprint Xiv::311.118021 2023b.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. Xrv preprint ar XrXiv:1504.0032) 2015.
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via lan- guage model in-context tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 719-730, 2022.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4171 2019.
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy- yong Sohn, Dimitris I'apailiopoulos, and Kangwook Lee. LIFT: iinuuag-iinteffaced fine-tuning for non-language machine learning tasks. In Advances in Neural Information Processing Systems, volume 35, pp. 11763-11784, 2022.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. DreamLLM: Synergistic multimodal comprehension and creation. ar Xiv preprint ar Xiw::3099.1111 1149,, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence, PaLM-E: An embodied multimodal language model. In International Conference Machine Learning, volume 202, pp. 8469-8488, 2023.
12

---

Published as a conference paper at COLM 2024
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA: exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19358-19369, 2023.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. DataComp: In search of the next generation of multimodal datasets. ar preprint ar Aix::2300 14108, 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arviv preprint ar Xrx::22001.0000 2020.
Shivam Garg, Dimitris Isipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? A case study of simple function classes. In Advances Neural Information Processing Systems, volume 35, pp. 30583-30596 2022.
Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting SEED of vision in large language model. arv preprint arXio:2307.0e 000411, 2023a.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making LLaMA SEE and draw with SEED tokenizer. Xrv preprint arX::33310 2023b.
Rohan Gemini Team Google: Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Bapiiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. ar preprint ar arXi::2312.11810 2023.
Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stre Dhruv Batra, and Devi Parikh. Making the in VQA matter: Flevating the role of image understanding in visual question answering. Int. Comput. Vis., 127:398-411 414, 2019.
Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purpose interfaces. arXiv preprint ar arXiv:2206.06336, 2022.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP, pp. 9318-9333, 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. arXiv preprint ar XrXt::2004.08718, 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. TIFA: Accurate and interpretable ext-to--mage faithfulness evaluation with question answering. Xiv preprint ar arXir:2303.1139) 2023.
Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et AudioGPT: Understanding and generating speech, music, sound, and talking head. arXiv preprint ar axXto:2304.12995 2023a.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. Xrx preprint XrX0o:2302.14045, 2023b.
13

---

Published as a conference paper at COLM 2024
Ting-Hao K. Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Aishwarya Agrawal, Ross Cirshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2016.
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEF/CVE Conference Computer Vision and Pattern Recognition, pp.
2019.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. URL https://doi. org/10 5281/zenodo. 5143773.
Kushal Kafle, Brian L.. Price, Scott Cohen, and Christopher Kanan. DVQA: understanding data visualizations via question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5648-5656, 2018.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. ReferItGame: Referring to objects in photographs of natural scenes. In Findings of the Association for Computational Linguistics: EMNLP, pp. 787-798, 2014.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal
language models. ar Xie preprint ar Xrv::2305..1711 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis., 123:32-73, 2017.
LAION. LAION-Aesthetics. http::///aaaooaaabb//////baaaats
2022.
Andrew Lampinen, Ishita Dasgupta,Stephaniee Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. Can language models learn from explanations in context? In Findings of the Association for Computational Linguistics (EMNLP), pp. 537-563, 2022.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, et al. OBELISC: An open web-scale filtered dataset of interleaved image-text documents. arxiv preprint ar Xiv::330 16527, 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, volume 162, pp. 12888-129999 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, volume 202, pp. 19730-19742, 2023.
Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning. arXiv preprin! ar arXiv:2402.18819 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. ar Xio preprint ar xrXi::2310.03744, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLa VA-NeTT: Improved reasoning, OCR, and world knowledge. https://llavar git. httbbbboo//logg/244-0------ 1eae-lllaaaaeee////eee 2024.
14

---

Published as a conference paper at COLM 2024
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100-114, 2022.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
11-20, 2016.
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A dataset for VQA on document images. In JEEE Winter Conference on Applications of Computer Vision, WACV, pp. 2199-2208, 2021.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetalCL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2791-2809, 2022a.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048-11064, 2022b.
Suvir Mirchandani, Fei Xia, Pete Florence, brian ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. In 7th Annual Conference on Robot Learning, 2023.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR-
VQA: visual question answering by reading text in images. In 2019 International Conference on Document Analysis and Recognition, ICDAR, pp. 947-952, 2019.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. Reframing instructional prompts to GPTk's language. In Findings of the Association for Compulational Linguistics, pp. 589-612, 2022.
Masoud Monajatipoor, Liunian Harold Li, Mozhdeh Rouhsedaghat, Lin Yang, and Kai-Wei Chang. MetaVL: Transferring in-context learning ability from language models to vision- language models. In Anna Rogers, Jordan I.. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics, pp. 495-508, 2023.
Thao Nguyen, Yuheng Li, Utkarsh Ojha, and Yong Jae Lee. Visual instruction inversion: Image editing via visual prompting. arviv preprint ar Xto:2307.14331, 2023.
OpenAI. Can i sell images i create with DALL-E? htttps://hely// http:://hhepppppai.coo//o/////// ppepai.ccm/////tticll... 1-22277-ca--i-sel1 -mages-i-ccrt www.--eewtthtaaaaeeeleeee 2023.
OpenAI. GPT-4 technical report. arXie preprint arXiv:2303.08774, 2023.
Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. Im2Text: Describing images using million captioned photographs. In Advances in Neural Information Processing Systems, volume 24, pp. 1143-1151, 2011.
Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos- G: Generating images in context with multimodal large language models. arxio preprint axXit:2310.02992, 2023.
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11410-11420 2022.
15

---

Published as a conference paper at COLM 2024
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXio preprint ar Xix::3306..45888 2023a.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. ar Xiv preprint (iXi::2306.188224, 2023b.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high- resolution image synthesis. arXiv preprint arXiv:2307.00
2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Πly Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, volume 139, pp. 8748-8763, 8763, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer- Assiste Intervention, pp. 234 241, 2015.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655-2671, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proczedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wight- man, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: an open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, vol- ume 35, pp. 25278-25294, 2022a.
Christoph Schuhmann, Andreas Köpf, Richard Vencu, Theo Coombes, and Romain Beau- mont. LAION COCO: 600M synthetic captions from laion2b-en. nttps://laionn ai/blog/ Maion-coco/,20220 2022b.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics, pp. 2556-2565, 2018a.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguiitso (Volume 1: Long Papers), pp. 2556-2565, 2018b.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Selective annotation makes language models better fhw-shot learners. In International Conference on Learning Representations, 2023.
16

---

Published as a conference paper at COLM 2024
Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. JourneyD:: A benchmark for generative image understanding. Advances in Neural Information Processing Systems, 36, 2024.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multi- modal models are in-context learners, 2023a.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: Improved training techniques for CLIP at scale. arXiv preprint arXio:2303.15339 2023b.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multi- modality. arXiv preprint ar Xiv:2307.05222, 2023c.
Yanpeng Sun, Qiang Chen, Jian Wang, Jingdong Wang, and Zechao Li. Exploring effective factors for improving visual in-context learning. ar arXiv preprint ar Xi1::2304.04448, 2023d.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal fho-shot learning with frozen language models. In Advances in Neural Information Processing Systems, volume 34, pp. 200-212, 2021.
Unsplash Team.
Unsplash dataset. https: comp://aaa.accccaaaaaaaaaaaaaaae 2023. Accessed: 2024-01-30.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoghu. Neural discrete representation learning. In Advances in Neural Information Processing Systemss, volume 30, pp. 6306-6315, 2017.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, volume 202, pp. 35151-35174, 2023.
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6830-6839, 2023a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assess- ment: from error visibility structural similarity. IEEE transactions on image processing, 13: 600-612, 2004.
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. DiffusionDB: A large-scale prompt gallery dataset for text-to- toxttt--imag generative models. In Proceedings of the 61st Annual Meefing of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 993-911, 2023c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-Thouugh prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pp. 24824-24833 2022.
Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. ar Xiv preprint XrX1:22303.07895, 2023.
17

---

Published as a conference paper at COLM 2024
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of Thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, 2023.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi- modal models: Pretraining and instruction tuning. arXiv preprint arXtr:239.005591 2023a.
Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Xinlong Wang, and Jingjin Liu. CapsFusion: Rethinking image-text data at scale. arXiv preprint ar arXii:2310.20550, 2023b.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. MERL RESERVE: Neural script knowledge through vision and language and sound. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16375-16387, 2022.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. SpeechGPT: Empowering large language models with intrinsic cross-modal conver- sational abilities. In Findings of the Association for Computational Linguistics: EMNLP, pp. 15757-15773, 2023a.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. MagicBrush: A manually annotated dataset for instruction-guider image editing, Advances in Neural Information Processing Systems, 36, 2024.
Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. Xiv preprint ar XxXr:22066009927 2023b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. ar Xiv preprint ar axXto:225.00166, 2022a.
Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. GPT-4V (ision) as a generalist evaluator for vision-language tasks. arXio preprint ar arXio:2311.0136) 2023c.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. LlaVAR: Enhanced visual instruction tuning for text-rich image understanding. Xiv preprint #rXiv:2306.17100 2023d.
Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9134-9148, 2022b.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? arXiv preprint ar 0tt::23311.3370,, 2023e.
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. MMICL: Empowering vision-language model with multi-modal in-context learning. Xiv preprint ar X30902300007915 2023.
Kaizhi Zheng, Xuehai He, and Xin Eric Wang. MiniGPT-5: Interleaved sisio--and-annguagee generation via generative vokens. arv preprint ar arXio:2310.02239, 2023a.
Lianmin Zheng, Wii-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. Xiv preprint ar arXio:2306.0566 2023b.
18

---

Published as a conference paper at COLM 2024
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint ar Xiv::2304.1055 2023a.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal C4: An open, billion-scale corpus of images interleaved with text. arxiv preprint arXiv:2304.06939 2023b.
19

---

Published as a conference paper at COLM 2024
Appendix
A In-Depth Overview of MLLMs
21
Extended Related Works
23
C More Details of CoBSAT Dataset
25
D Detailed Experiment Setup
25
D.1 Prompt Templates for Model Inference
25
D.1.1 Instructing MLLMs for Generating Image Descriptions
26
D.1.2 Articulating the Text-to-Image Relationship in Prompts
27
D.2 Prompt Templates for Model Evaluation
28
Comparison of T21-ICL Evaluation Metrics
29
Detailed and Extended Results of Experiments
29
F1
Benchmarking MLLMs in T2I-ICL (Detailed Version of Sec. 5)
30
F111
Assessing Generated Images
30
F1.2
Assessing Generated Image Descriptions
32
E113
Impact of Number of Demonstrations
32
E11.
Textual Information v.s. Visual Information
33
E.1.5
Object Generation v.s. Attribute Generation
33
F1.6
A Challenging Version of CoBSAT
34
F22
Understanding Challenges in T2I-ICL (Detailed Version of Sec. 6)
35
F2.1
Is Multimodality a Primary Challenge in T2I-ICL?
35
F2.2
Is the Image Generation Primary Challenge in T2I-CCL?
35
F3
Enhancing MLLMs T2I-ICL Capabilities (Detailed Version of Sec. 7)
36
E3.1
Fine-tuning MLLMs CoBSAT
36
F3.2
Intergrating Chain-of-Thought with T21-ICL
37
E3.3
Articulating the Text-to-lmage Relationship in Prompts
39
G Extended Discussion
40
G.1 Conclusion
40
G.2 Limitations and Future Works
40
H Sample Outputs Generated by MLLMs
41
H.l Sample Prompts and Corresponding Outputs
41
H.2 Sample Outputs from Fine-tuning SEED-LLaMA on CoBSAT
42
H.3 Sample Outputs from Integrating CoT with T2I-ICL,
42
20

---

(Yu et al., 2023b), and language-focused data from Published as a conference paper at COLM 2024
A In-Depth Overview of MLLMs
In this section, we provide a detailed overview of the MLLMs used in our experiments, including (i) four MILMs with image generation capabilities: Emu (Sun et al., 2023c), Emu2 (Sun et al., 2023a), SEED-LLaMA (Ge et al., 2023;bb,, and GILL (Koh et al., 2023), and (ii) five state-of-the-art MLLMs that can only generate text: Qwen-VL (Bai et al., 2023b), LLaVA models (LLaVA-1.5 (Liu et al., 2023a) and LLaVA-NeXI (Liu et al., 2024)), Gem- ini (Gemini Team Google: Anil et al., 2023), and GPT-4V (OpenAI, 2023).
Emu (Sun et al., 2023c). Emu integrates EVA-CLIP (Fang et al., 2023) as the Visual Encoder, the Causal Transformer, LLaMA-13B (Touvron et al., 2023), and Stable Diffusion v1.5 as the Visual Decoder. Given any sequence including images and texts, the images are en- coded into dense visual features via EVA-CLIP (Fang et al., 2023). These features are then transformed into visual causal embeddings via a Causal Transformer, which converts 2D spatial visual signals into 1D causal sequences. Two special image tokens, [IMG] and [/IMG], are prepended and appended to the visual causal embeddings of each image. The visual causal embeddings are then combined with the text tokens and fed into the LLaMA. In the output generated by LLaMA, the visual embeddings in-between image tokens [IMG] and [/IMG] are decoded using the fine-tuned Stable Diffusion 1.5. All components of Emu are further trained from their initial state using image-text pairs from LATON-2B (Schuh- mann et al., 2022a) and LAION-COCO (Schuhmann et al., 2022b), video-text pairs from WebVid-10M (Bain et al., 2021), interleaved image and text from MMC4 (Zhu et al., 2023b), an expanded version of the text-only C4 (Raffel et al., 2020), and interleaved video and text from YT-Storyboard-ID (Zellers et al., 2022; Sun et al., 2023c). Furthermore, Emu also process videos by treating various frames as a sequence interspersed with text and images.
Emu2 (Sun et al., 2023a). Emu2 represents upscaled version of its predecessor, Emu, featuring significant upgrades in its component architecture. Unlike Emu, which utilized EVA-CLIP, LLaMA-13B, and Stable Diffusion v1.5 for its Visual Encoder, Multimodal Mod- eling, and Visual Decoder, respectively, Emu2 employs larger versions: EVA-02-CLIP-B plus (Sun et al., 2023b) for the Visual Encoder, LLaMA-33B for Multimodal Modeling, and SDXL (Podell et al., 2023) as the Visual Decoder. Moreover, Emu2 replaced Emu's C-Former with mean pooling followed by linear projection for connecting Visual Encoder and Multimodal modeling. Its pretraining regime also differs, utilizing datasets that includes image-text pairs from LAION-2B (Schuhmann et al., 2022a) and CapsFusion-120M (Yu et al., 2023b), video-text pairs from WebVid-10M (Bain et al., 2021), interleaved image-text data from MMC4 (Zhu et al., 2023b), interleaved video-text data from YT-Storyboard-1B (Zellers et al., 2022; Sun et al., 2023c), grounded image-text pairs from GRIT-20M (Peng et al., 2023b) and peossuuioongruuneee--eee Pile (Gao et al., 2020).
SEED-LLaMA (Ge et al., 2023b). SEED-LLaMA introduces tokenizer named SEED, which consists of a ViT encoder (Dosovitskiy et al., 2021) derived from the pretrained BLIP-2 (Li et al., 2023), Causal Q-Former, VQ Codebook (van den Oord et al., 2017), multi-layer perception, and a UNet decoder (Ronneberger et al., 2015) derived from the Stable Diffusion model. When given an input that includes both text and images, the images are first transformed into 2D raster-ordered features by the ViT encoder. These features are then converted into a sequence of causal semantic embeddings via the Causal Q-Former, discretized by the VQ Codebook, and projected by a multi-layer perceptron. The resulting embeddings are integrated with the text embeddings and fed into the LLaMA. The generated image embeddings are subsequently inputted into the Stable Diffusion model to generate realistic images. All components, except for the embedding layer, have been further trained on datasets including COCO Caption (Chen et al., 2015), CC3M (Sharma et al., 2018b), Unsplash (Unsplash Team, 2023), LAION-COCO (Schuhmann et al., 2022b), MMC4 (Zhu et al., 2023b), OBELISC (Laurençon et al., 2023), and Web (Bain et al., 2021). Additionally, 26 datasets are employed for supervised instruction tuning of SEED-LLaMA to align it with human instructions.
21

---

Published as a conference paper at COLM 2024
GILL (Koh et al., 2023). GILL employs a pretrained visual backbone and linear projection mapping to process image input, while a tokenizer is used for text input. These inputs are concatenated and fed into OPT-6.7B (Zhang et al., 2022a). The output image embeddings are then processed by a decision model to determine whether to retrieve real images or generate realistic fake ones. For generating realistic images, GILL proposes G.I.Mapperr which encompasses a Transformer Encoder that receives image embeddings, and a Transformer Decoder that processes the Encoder's 's outputs along with certain learned queries. The sequences produced by the Decoder are transformed through linear layer to generate the predicted embeddings, which are then provided the Stable Diffusion v1.5 model to create realistic images. For image retrieaal, GILL projects the image embeddings via a linear layer and then measures the similarity between these embeddings and those of potential image candidates obtained through the CLIP ViT-L. model (Radford et al., 2021). The image exhibiting the highest similarity score is then selected for output. GILL is pretrained on the CC3M dataset (Sharma et al., 2018b).
The three models previously mentioned are MLLMs capable of generating images. Next, will describe MLLMs that can only generate text.
Qwen-Vi (Bai et al., 2023b). Qwen-VL is an extension of the Qwen-7B language model (Bai et al., 2023a), equipped with visual capabilities. To achieve this, Qwen-VL incorporates a Vision Transformer (ViT) (Dosovitskiy et al., 2021) with weights initialized from OpenCLIP's ViT-bigG (Ilharco et al., 2021), and a single-layer cross-attention module to convert images into feature sequence that can be directly fed into Qwen-7B. Qw is pre-trained using (i) variety of web-crawled image-text datasets, including LAION-5B, LAION-COCO (Schuhmann et al., 2022a), DataComp (Gadre et al., 2023), Coyo (Byeon et al., 2022), CC12M (Changpinyo et al., 2021), CC3M (Sharma et al., 2018a), SBU (Ordonez et al., 2011), COCO Caption (Chen al., 2015), and in-house data (Bai et al., 2023b); and (ii) other visual question-answering
datasets and visual reasoning datasets, including GQA (Hudson & Manning, 2019), VGQÅ (Krishna et al., 2017), VQAv2 (Goyal et al., 2019), DVQA (Kafle et al., 2018), OCR-VQA (Mishra et al., 2019), DocVQA (Mathew et al., 2021), GRIT (Peng et al., 2023a), Visual Genome (Krishna et al., 2017), RefCOCO (Kazemzadeh et al., 2014), RefCOCO+, and RefCOCOg (Mao et al., 2016).
LLaVA (Liu et al., 2023a). LLaVA is built upon the Vicuna-v1.5-13B LLM (Zheng et al., 2023b). To enable the visual perceiving capability, it incorporates vision encoder, specifi- cally the CLIP-VTT-L-336p (Radford et al., 2021), along with an MLP projection to encode visual features into image embeddings. These image embeddings, along with text embed- dings encoded by tokenization, the textual output. Its training follows a two-stage protocol. First, during the vision- language alignment pretraining stage, the model leverages the image-text pairs dataset CC3M (Sharma et al., 2018a) to align the visual features with the language model's 's word embedding space. Second, the visual instruction tuning stage involves tuning the model on visual instructions to enable it to follow users' diverse requests involving visual content. For this stage, LLaVA utilizes GPT-4V (OpenAI, 2023) to expand the existing COCO (Chen et al., 2015) bounding box and caption dataset into a multimodal instruction-followee dataset, which includes three types of -ostuuction-follooiigg data: conversational-style QA, detailed description, and complex reasoning. I.L (Liu et al., 2024) is an improved version of LLLVV, particularly in reasoning, OCR, and world knowledge. It achieves this by increasing the input image resolution to capture more visual details and utilizing Mistral-7B and Nous-Hermes-22 2--rmme-2-YY-34BB as the additional backbones. Moreover, LLaVA-NeXT utilizes a better mixture of visual instruction tuning data, comprising high-quality user instructions and multimodal document/ chart data.
then concatenated and fed into the LLM to generate
Claude (Anthropic, 2024). Claude series is one of the leading LLMs developed by An- thropic Anthropic recently introduced Claude 3, family of MLLMs: Claude 3 Öpus, Claude 3 Sonnet, and Claude 3 Haiku. Claude 3 can understand multimodal inputs such as photos, tables, and graphs. Besides multimodality, Claude 3 shows better fluency, especially for non-English languages. We chose Claude Haiku for our experiment due to its speed and cost-effectiveness.
22

---

Published as a conference paper at COLM 2024
Gemini (Gemini Team Google: Anil et al., 2023). Gemini, family of MLLMs developed by Google, is built on Transformer decoders and trained on extensive images, audio, video, and text datasets (including natural images, charts, screenshots, and PDFs). With 32k context length support, it provides three variants: Ultra, Pro, and Nano, with Ultra offering the highest capabilities and Nano excelling in efficiency. We employ Gemini-pro in our paper.
GPT-4V (OpenAI, 2023). GPT-4V has emerged as one of the most proficient MLLMs, demonstrating exceptional performance and achieving human-level results on majority of professional and academic examinations. Despite being a closed-source MLLM, with undisclosed details about its architecture and dataset construction, GPT-4V is included in our evaluation due to its superior performance compared to other MLLMs (Bai et al., 2023b).
B Extended Related Works
This section provides detailed related works.
Textual ICL. Ever since Brown et al. (2020) demonstrated that language models are in- context learners (see Figure 1(a)), there has been substantial interest in comprehending this capability, both empirically (Liu et al., 2022; Min et al., 2022b; Chen et al., 2022; Mishra et al., 2022; Lampinen et al., 2022; Garg al., 2022; Hendel et al., 2023) and theoretically (Xie et al., 2022; Wies et al., 2023; Akyürek et al., 2023; Von Oswald et al., 2023; Bai et al., 2023c; Ahn et al., 2023; Zhang et al., 2023b). Textual ICL (T-ICI.) enables the adaptation of L.MS to downstream tasks simply by providing few illustrative examples, bypassing any need for updating model parameters. The existing works indicate that LLMs possess the capability to comprehend context and perform reasoning through T-ICL (Brown et al., 2020).
Visual ICL. The concept of V-ICL is then employed in computer vision, starting with the introduction of visual prompts (see Figure (b)). The pioneering works by Bar et al. (2022); Wang et al. (2023a) propose to automatically generate output images that are contextually aligned with provided examples. Specifically, Bar et al. (2022) developed a method that combines three images- an example input, its corresponding output, and query into single composite image. In this layout, the example input is placed in the upper left, the example output in the upper right, the query image in the bottom left, and the bottom right patch is left blank for output construction via an image inpainting model. Bar et al. (2022) demonstrated the effectiveness of V-ICL in tasks like edge detection, colorization, inpainting, segmentation, and style transfer. Wang et al. (2023a) introduced a similar approach and trained a generalist model named "Painter," which exclusively uses visual prompts without any textual data for V-ICL. Experiments on standard computer vision benchmarks revealed competitive performance against task-specific models. Nguyen al. (2023) further applied visual prompts to image editing by inverting visual prompts into text-based editing directions, leveraging the pre-trained capabilities of diffusion models.
A subsequent empirical study by Zhang et al. (2023e) highlighted that the success of V-ICL significantly depends on the choice of in-context demonstrations. The aspect of demonstra- tion selection was further explored by Sun et al. (2023d), who also examined the impact of prompt fusion on performance. Their findings indicate a high sensitivity of performance to the arrangement of sub-images in in-context learning. Moreover, innovative approaches to structuring V-ICL, such as the concept of "visual sentences," have been introduced in recent studies, notably by Bai et al. (2023d). Unlike V-ICL which only handles images, M-ICL integrates demonstrations encompassing both text and images.
MLLMs. In light of the significant success of LLMs, there has been an increase in the release of MLLMs. These models are designed to address more challenging multimodal tasks, thereby enabling the perception of images (Li et al., 2022; Alayrac et al., 2022; Hao et al., 2022; Laurençon et al., 2023; Huang et al., 2023b; Peng et al., 2023b; Li et al., 2023; Ge et al., 2023b; Koh et al., 2023; Zhu et al., 2023a; Sun et al., 2023c; Zheng et al., 2023a; OpenAI, 2023; Liu et al., 2023b;a; Bai et al., 2023b; Sun et al., 2023a; Gemini Team Google: Anil et al.,
23

---

Published as a conference paper at COLM 2024
2023; Driess et al., 2023; Anthropic, 2024), videos eLi et al., 2022; Alayrac et al., 2022; Li et al.,
2023; Sun et al., 2023c; Gemini Team Google: Anil et al., 2023), and audio (Hao et al., 2022; Borsos et al., 2023; Huang et al., 2023a; Chen et al., 2023a; Zhang et al., 2023a; Gemini Team Google: Anil et al., 2023). Existing models capable of handling images can be categorized as follows: (i) those that use language as a general interface and directly employ LLMs without altering the model architectures (Dinh et al., 2022; Cai et al., 2023; Aghajanyan et al,, 2022; Yu et al., 2023a; Huang et al., 2023b; Mirchandani et al., 2023; Cai al., 2023); (ii) those that add one or more modules before feeding the input sequence into the LLM to perceive multimodal inputs (Isimpoukelli
al., 2021; Alayrac et al., 2022; Awadalla et al., 2023; Laurençon et al., 2023; Li et al., 2023; Hao et al., 2022; Liu et al., 2023b;a; Zhu et al., 2023a; Gemini Team Google: Anil et al., 2023; Liu et al., 2024); (iii) those that add one or more modules after the LLM processing for generating multimodal outputs (Pan et al., 2023); (iv) those that add modules to both inputs and outputs of the LLMs to process the multimodal input and generate multimodal outputs (Dong et al., 2023; Sun et al., 2023c; Koh et al., 2023; Ge et al., 2023b; Zheng et al., 2023a; Sun et al., 2023a).
In this paper, our main focus is T2I-ICL. We aim to investigate whether MLLMs can learn to transform -iw-deensional textual input into high-dimensional visual output based on demonstrations, and to accurately generate images from new textual queries. Consequently, we focus on models capable of processing both text and multiple images. We consider two types of MLLMs: (i) proficient in generating both text and images, including Emu (Sun et al., 2023c), Emu2 (Sun et al., 2023a), GILL (Koh et al., 2023), and SEED-LLaMA (Ge et al., 2023b), and (ii) those limited to text generation, including GPT-4V (OpenAI, 2023), Gemini (Gemini Team Google: Anil et al., 2023), Claude (Anthropic, 2024), LLaVA-1.5 (Liu et al., 2023a), LLaVA-NeXT (Liu al., 2024), and Qwen-VL (Bai et al., 2023b). For text-only MLLMs, we evaluate their capacity to infer visual outputs by prompting them to describe the anticipated image. Conversely, for MLLMs capable of image generation, we not only elicit image outputs but also ask for descriptive text, ensuring apple-to-aplle comparison with text-only models.
Image-to-Text ICL in MLLMs. Most existing work on M-ICL focuses on the image-to-text generation, i.e., I2T-ICI, which involves mapping from high-dimensional input (i.e., images) to low-dimensional output (i.e., text). In particular, Tsimpoukelli et al. (2021) were the first to extend ICL from the text domain to the multimodal domain, focusing on image-to- text generation such as visual question-answerrinn introduced Flamingo, an MLLM that achieves state-of-the-art performance in a variety of image and video understanding tasks using I2T-ICL with 32 demonstrations, implying the efficacy of I2T-ICL in performance enhancement in their model. In contrast, Monajatipoor et al. (2023) explores whether the in-context capabilities of LLMs can be seamlessly extended to I2T-ICL by incorporating a visual encoder. Chen et al. (2023b) conducted a systematic study on the importance of visual and textual information in 12T-ICL. Concurrently, efforts have been made to develop datasets specifically designed for evaluating 12T-ICL capability of MLLMs (Zhao et al., 2023). In contrast, there have been only a few attempts (Sun al., 2023a) to evaluate the T2I-ICL capability of MLLMs, a domain that remains relatively unexplored compared to its image-to-text counterpart.
(see Figure 1(c).. Alayrac et al. (2022)
Zero-Shot Image Generation in MLLMs. A relatively small number of MLLMs are capable of image generation (Yu et al., 2023a; Dong et al., 2023; Zheng et al., 2023a; Sun et al., 2023c; Ge et al., 2023b; Koh et al., 2023; Pan et al., 2023; Sun et al., 2023a). Zero-shot text-to- image generation typically generates images directly from textual descriptions without relying on any examples. This does not require the model to integrate a combination of textual and visual inputs. Another common task for MILMss in image generation is context modifications. In this more complex scenario, the model receives visual inputs (e-g, an image of dog) along with associated textual instructions (e.g., "swimming underwater"). This task requires a nuanced understanding and manipulation of the image, guided by the textual instructions, thereby blending image comprehension with contextual transformation based on text. Unlike zero-shot image generation, our focus is on studying whether MLLMs can learn the implicit relationship between the input and output from multiple in-context demonstrations.
24

---

Published as a conference paper at COLM 2024
Text-to-Image ICL in MLLMs. There are limited attempts to evaluate MLLMs based their T2I-ICL capabilities. A notable exception is concurrent research by Sun et al. (2023a). They evaluated the performance of their model on T2I-ICL with DreamBooth dataset (Ruiz et al., 2023). However, it is important to note that the DreamBooth dataset, primarily developed for fine-tuning models to modify image contexts, was not specifically designed for T21-ICL applications. This leads to certain constraints, such as its concentrated emphasis on altering backgrounds only and level of complexity that may not align well with T2I-ICL. In contrast, our dataset spans five themes and provides well-designed prompts to assess whether models can understand both visual and textual information, learn mappings from demonstrations, and make inferences.
Image Evaluation Metrics. A variety of metrics exist for assessing the quality of generated images. Classical ones like Peak Signal-to-Noise Ratio (PSNR) (Wang et al., 2004) evaluate the quality of reconstructed images or videos by measuring pixel-level errors compared to the target images. Fréchet Inception Distance (FID) (Parmar et al., 2022) gauges the quality of images produced by generative models, such as Generative Adversarial Networks, by calculating the similarity between the distributions of generated and real images. However, these metrics are not entirely suitable for our purpose, where single definitive ground- truth target image exists but rather a textual label (e.g,, "red car" in the first example of Figure 2).
In the realm of text-to-image generation, the CLIP similarity (Radford et al., 2021) metric has gained popularity (Ruiz et al., 2023). It measures the cosine similarity between the CLIP embeddings of the textual ground truth and the visual output. Meanwhile, there is a growing trend of utilizing MLLMs for evaluation (Zhang et al., 2023c; Hu al., 2023), showing promising results in text-to-image tasks. Our study both approaches, utilizing CLIP (Radford et al., 2021) and MILMs including LLaVA-1.5 (Liu et al., 2023a), Gemini (Gemini Team Google: Anil et al., 2023), and Qwen-VL (Bai et al., 2023b) to assess the accuracy of generated images. To be more specific, we utilize CLIP and MLLMs to identify the object (e.g., "car") and attribute (e.g,, "red") in the image generated by MLLMs and then compare these identifications with the actual label (e-g., "red car" for the first example in Figure 2). The details are provided in Sec. 4. Unless specified otherwise, the accuracy reported in our studies is primarily estimated using LLaVA-1.5, whose effectiveness has been validated by by its ability to accurately recognize objects and attributes, achieving 100% accuracy rate within our dataset, and closely aligning with human evaluation, as detailed in our analysis in Sec. E.
C More Details of CoBSAT Dataset
Detailed Structure. The detailed structure of all tasks in our dataset is provided in Table 5.
Copyright Considerations. It is important to note that the images generated using DALL- E for our dataset are not subject to copyright restrictions. As per the content policy and terms of the DALL-E 3 service, users retain ownership rights over the images they create, including the rights to reprint, sell, and merchandise, irrespective of whether the images were generated using free or paid credits (OpenAT, 2023).
D Detailed Experiment Setup
In this section, we provide the details of our experiment setup, including prompt template design for model inference (Sec. D.1) and prompt design for model evaluation (Sec. D.2).
D.1 Prompt Templates for Model Inference
For generating images based on in-context input-output pairs, we employ the prompt template depicted in Figure 3 for SEED-LLLMM and Emu. This template simply includes
25

---

Published as a conference paper at COLM 2024
Calegory
Task
Text Inpot C
Latent Variahle P
Image Outputy ~ fo(x)
[Text: color E (yellow, white. red, purple, pink, orange, green, brown. blue, black)]
object (leaf, cup, chair, car, box, book, ball, bag. apple)
Color-1
[[mage object of color.x]
[Text: background e [heach, desert, glacier, volcano, park. gyms, waterfall, space, cave. seafloor]]
animal e (zebra, tiger, sheep, monkey, lion. dog, sow, cat, bind)
Background-1
[Image: animal in background
Object- Inference
[Text: style F (watercolor, sketch, pisel, lego, icen, graffiti futuristic, wireframe, old]]
object (leaf, hat, cup, chair, car, box, book, ball, bag. apple)
Style-1
[Image: object style
[Text: action {swim, sleep. sing. run, read, fly, cat, drink, angry]]
animal E (zebra, tiger, sheep, pig. monkey, lion, dog, saw, cat, bird)
Action-1
[Image: animal doing
[Text: texture E (wood, wicker, sequined, plastic, paper, metal, leather lace, denim, ceramic]]
objecct (leaf, hat, cup, chair, car, box, book, ball, bag. apple)
Texture-I
[Imaag: object texture x]
[Text: object e (leaf, hat, cup. chair, box, book, ball, bag. apple]]
color e (yellow, white, red, pur- ple, pink, orange, green, brown, blue, black)
Color-I1
[[mage object of color 8]
[Text: animal = (zebra, tiger. sheep, pig, monkey, lion, cow, cat. bird]]
background E (beach, desert, glacier, volcano, park, gym, wa- terfall, space, cave, seafloor)
Background-II
[Image animal in background 6]
Attribute- Inference
[Text: object E (leaf, hat, cup. chair; car, box, book, ball, bag. apple}]
style E (watercolor, sketch, pixel, origami, lego, icon, graf- futurislic, witeframe, old)
Style II
[Image: object in style l]]
[Text: animal E (zebra, tiger. sheep, pig, monkey, lion, dog. cow, cat, bird)]
action C (swim, sleep, sing, run, read, fly, eat, drink, cry, angry)
Action-1l
[Imager animal xx doing e|]
[Text: object C [leaf, hat, cup. chairs car, box, book, ball, bag. apple]]
texture [wood, wicker, se- quined, plastic, paper, metal, leather, lace, denim, ceramic)
Texture-II
[[mage: object in texture R]
cermmic
Table 5: Task summary of CoBSAT. We use [Text: description] to denote the text providing the corresponding description. For instance, [Text: color] could refer to terms such as "red" and "black." " Each task is characterized by the input space X' and the latent variable space O. For N-shot inference, we generate 1,000 prompts. Each prompt is obtained by randomly sampling 0 E O and (xn) E XN+1, followed by collecting the corresponding images (ym))=rr where yn ~ fx(nn)
the in-context samples and the text query, without any additional instructions. For GILL, we add an additional system message: "You are a professional assistant who can generate new image based on the sequence."
In the subsequent subsections, we present our prompts for instructing MLLMs to generate image descriptions, continuing from the discussion in Sec. 4, and prompts for articulating the text-to-image relationship, continuing from the discussion in Sec. 7.
D.1.1 Instructing MLLMs for Generating Image Descriptions
In this part, we provide the prompt templates used for instructing all considered models to generate image descriptions:
Emu: We add the instruction as a system message: "Based on the sequence, describe the next image clearly, including attributes such as the main object, color, texture, background, action, style, if applicable."
We append "Based on the sequence, describe the next image clearly, including details such as the main object, color, texture, background,, action, style, if applicable." to the end of the input.
Emu2:
GILL: We insert "You are professional assistant and always answer my question directly and perfectly without any excuses." at the beginning of the prompt and append "Based on the sequence, describe what the next image should be clearly, including attributes such as the main object, color, texture, background, action, style, if applicable. Your response should only contain description of the image, and any additional information can cause significant loss." at the end of the input.
26

---

Published as a conference paper at COLM 2024
SEED-LLaMA: We insert "I will provide you few examples with text and image. Complete the example with the description of next image. Tell me only the text prompt and I'll use your entire answer direct input to A Dalle-3. Never say other explanations. at the beginning of the prompt.
LLaVA-1.5 & LLaVA-NeXT: We add "Based on the sequence, describe the next image to be generated clearly, including attributes such as the main object, color, texture, background, action, style, if applicable.. at the end of the prompt.
Qwen- We insert "You professional assistant and always answer my question directly and perfectly without any excuses." " to the start of the prompt and append "Based on the sequence, describe what the next image should be clearly, including altributes such the main object, color, texture, background, action, style, if applicable. Your response should only contain description of the image, and all other information can cause huge loss." " to the end of the input.
We append "Based on the sequence, describe the next image clearly, inchuding details such as the main object, color, texture, background, action, style, if applicable. " at the end of the prompt.
Gemini:
We prepend "I will provide you a few examples with text and image. Complete the example with the description of next image. Never say other explanations. to the beginning of the prompt, and append "Give me the description of the your predicted next image." at the end of the prompt.
Claude:
We add "I will provide you with a few examples with text and images. Complete the example with the description of the next image. The description should be clear with main object, and include attributes such as color, texture, background, style, and action, if applicable. Tell me only the text prompt and I'll use your entire answer as direct input to A Dalle-3. Never say other explanations." at the start of the input.
GPT-4V:
D.1.2 Articulating the Text-to-lmage Relationship in Prompts
We now present the instructions for articulating the text-to-image relationship for the experiment presented in Sec. 7.
For image generation, we add the following sentences to the start of the prompts for each task.
"Please identify the common main object in the images, and generate another image of this object of the requested color."
Color-I:
Color-II: "Please identify the common color in the images, and generate another image of the requested object in the same color."
Background-I: "Please identify the common animal in the images, and generate another image of this animal walking in the requested background."
Background-II: "Please identify the common background in the images, and generate another image of the requested animal walking in the same background." "
Style-I: "Please identify the common object in the images, and generate another image of this object the requested style." "
Style-II: "Please identify the common style in the images, and generale another image of the requested object in the same style."
Action-I: "Please identify the common animal in the images, and generate another image of this animal doing the requested action."
Action-ll: "Please identify the common action/nood the animal is doing in the images, and generate another image of the requested animal doing the same action/mood." "
Texture-I: "Please identify the common main object in the images, and generate another image of this object of the requested texture."
Texture-II: "Please identify the common texture of the objects in the images, and generate another image of the requested object in the same texture."
27

---

Published as a conference paper at COLM 2024
For image description, we add the following sentences to the start of the prompts for each task.
"Please identify the common main object in the images, and describe the next image to be generated based the sequence below. Your description of the image should contain the description of the common main object and the requested color."
Color-I:
"Please identify the common main color in the images, and describe the next image to be generated based on the sequence below. Your description of the image should contain the description of the requested object and the common color."
Color-II:
"Please identify the common animal in the images, and describe the next image to be generated based on the sequence below. Your description of the image should contain the description of the common animal and the requested background." "
Background-I:
"Please identify the common background in the images, and describe the next image to be generated based on the sequence below. Your description of the image should contain the description of the requested animal and the common background."
Background-II:
Style-I: "Please identify the commnn object in the images, and describe the next image to be
generated based on the sequence below. Your description of the image should contain the description of the common object and the requested style."
Style-II: "Please identify the common style in the images, and describe the next image to be generated based on the sequence below. Your description of the image should contain the description of the requested object and the common style."
"Please identify the common animal in the images, and describe the next image to be generated based on the sequence below. Your description of the image should contain the description of the common animal and the requested action."
Action-I:
Action-lI: "Please identify the common action/nood the animal is doing in the images, and describe the next image to be generated based on the sequence below. Your description of the image should contain the description of the requested animal and the common action/mood."
"Please identify the common main object in the images, and describe the next image to be generated based the sequence below. Your description of the image should contain the description of the common main object and the requested texture."
Texture-I:
Texture-II: "Please identify the common texture of the objects in the images, and describe the next image to be generated based on the sequence below. Your description of the image should contain the description of the requested object and the common texture."
D.2 Prompt Templates for Model Evaluation
In this section, we present our prompt templates for model evaluation. The evaluation encompasses two scenarios: (i) assessing the generated images, and (ii) assessing the generated image descriptions.
Assessing Generated Images. Unless otherwise stated, we employ LLaVA-1.5 evaluate the generated images in terms of whether they generated the right object (e.g., "car" in the first example Figure 2) and attribute (e-g,, "red" in the first example Figure 2). To facilitate this evaluation, we design specific prompts for LLaVA. Here are the prompts designed for tasks Color-1 and II:
Object Identification: "[Image: generated image] What is the main object in this image? Answer from the following options: (1)f (2)hat (3)cup (4)chair (5)car (6)box (7)book (8)blll (9)bag (10)apple. Answer the number only and do not include any other texts (e.g., 1)."
Attribute Identification: "[Image: generated imagel What is the color (of the main object) in this image? Answer from the following options: (Dyellow (2)white (3)red (4)purple (5)pink (6)orange (7)green (8)brown (9)blue (10)black. Answer the number only and do not include any other texts (e.g., 1)."
For other tasks involving different themes, the options and the attribute category (e.g., replace "color" " in the attribute inference prompt with "style" for tasks Style-I and II) are updated correspondingly.
28

---

Published as a conference paper at COLM 2024
Assessing Generated Image Descriptions. We also use LLaVA-1.5 to evaluate the gen- erated image descriptions. However, in this case, we modify the prompts used for as- sessing generated images by replacing "[Image: generated imagel" with "Image caption: [Text: generated description]."
E Comparison of T2I-ICL Evaluation Metrics
In our experiments, we leverage LLaVA-1.5 for estimating the accuracy of the output of T21- ICL. However, there are also many other alternatives such as CLIP,, Gemini, and Qwen-VL. In this experiment, we study and compare the effectiveness of different models in terms of evaluating the performance of T2I-ICL.
Evaluation Metrics. This comparison focuses the accuracy metrics derived from CLIP and MLLMs including Gemini, LLaVA-1.5, and Qwen-VL, with results gathered from SEED-LLaMA's 2-shot T2I-ICL on CoBSAT. MLLM accuracy is determined by using MLLM to identify the main object and specific attribute (e.g,, color) in the generated images or descriptions leveraging prompts provided in Sec. D.2, which then matched against the true labels. CLIP асситису is computed based on CLIP similarity. CLIP similarity measures the cosine similarity between the true label's CLIP embedding and that of the generated content. CLIP accuracy involves selecting the most similar object and attribute from the predefined list based on their CLIP embedding's cosine similarity with the generated image or description. These selections are then compared with the true labels to determine accuracy.
Alignment of T2I-CCL Evaluation Metrics with Human Evaluation. We first investigate their alignments with human evaluation. We manually labeled 100 images generated by SEED-LLaMA through T2I-CL,, selecting ten random images from each task to serve baseline. It is important to note that some images were of suboptimal quality, presenting ambiguities that could be interpreted both correct or incorrect. Despite these difficulties, our evaluations using the LLaVA-1.5 show strong alignment with human assessments, achieving consistency rate of 89% (computed by the ratio of agreement between the two methods). Notably, other MLLMs, especially Gemini, also exhibited commendable performance, as shown Table
Model
CLIP
LLaVA-1.5
Qwen-VL
Gemini
Consistency Rate to Human Evaluation
.85
.89
.78
.92
Table 6: Alignment between human evaluations and automatic evaluations performed by CLIP, LLaVA-1.5, Qwen-VL, and Gemini.
Comparison among Evaluation Metrics. We further conducted a scaled statistical study with 20,000 images compare the performance of these automatic metrics, particularly focusing on how other metrics relate to Gemini's results, given its closest alignment with human evaluations.
Figure 6, 7, and 8 depict the alignment between the accuracy estimates of Gemini and those provided by CLIP, Qwen-VI. and L.L.VA-1.5, respectively. The analyses demonstrate robust correlation between the accuracy estimates of LLaVA-1.5 and Gemini, highlighted by the narrow confidence interval represented by the purple shadow the figures. This correlation strengthens our confidence in LLaVA-1.5 as reliable and accessible open-source evaluation alternative to closed-source models in evaluating MLLMs' T21-ICL performance.
Detailed and Extended Results of Experiments
In this section, we supplement the experimental details, extended experiments, and discus- sions that could not be addressed in the main body due to space limitations. Specifically,
29

---

Published as a conference paper at COLM 2024
Sec F.1, F2, and E3 provide additional experiment results and discussions for Sec 5, 6, and 7, respectively.
E1 Benchmarking MLLMs in T2I-ICL (Detailed Version of Sec. 5)
This an extended discussion of Section 5.
In this section, present and analyze our experimental results the evaluation of all the considered MLLMs' performance on T2I-ICL, including the MLLMs that are not discussed in the main paper, i.e., LLaVA-1.5, LaLaV--eXT, and Emu2. The full evaluation results are visualized in Figure 9. In addition to supplementing more detailed information on top of the main body, we also present comparison of textual and visual information in Sec. F.1.4, and a comparison of object and attribute generation in Sec. F115. Furthermore, we explore more complex variation of our dataset, with detailed descriptions of the experiments and results presented in Section F.1.6.
E1.1 Assessing Generated Images
In terms of image generation, we focus the four MLLMs that have this capability: Emu, Emu2, GILL, and SEED-LLaMA. Among these, SEED-LLaMA significantly outperforms the others, evidenced by Figure (()) and (b), achieving score of 68% on Color-I tasks. In contrast, Emu, Emu2, and GILL exhibit low performance, achieving accuracies around or even below 10%. For the images generated using Emu, Emu2, GILL, and SEED-LIaMA in Figure 14, 15, 16, 17, and 18. We observe that while Emu, Emu2, and GILL exhibit low performance, GILL does manage to generate images that either align with the textual query (e.g, "pink" in the fourth example of Figure 14(a)) or adhere to common visual patterns (e.g, "monkey" in the fourth example of Figure 15(a)). Conversely, Emu occasionally generates random images, as seen in the fourth example of Figure 14(a). On the other hand, Emu2's generated images more closely resemble a blend of the input images in the prompt, such as the fifth example of Figure 16(b).
tangible understanding, we present specific prompts alongside
(a) Image evaluation
(a) Image evaluation
(a) Image evaluation
Quea-VL Accuracy
LLAVA-1.5 Accurac
0.6
CLI Acouraey
0.4
0.4
0.2-
0.2
0.2
0.4
0.6
0.2
0.4
0.6
0.2
0.4
0.6
Gomini Acuumay
Gomini Aortray
Gomini Acuuaa
(b) Image description evaluation
(b) Image description evaluation
(b) Image description evaluation
0.08
0.15
Qwea-ViL Acmrury
ML.A-1155 Aceney
CLI Acouray
0.15
0.06
0.10
0.10
0.04
0.05
0.02
0.05
0.00
0.05
0.10
0.15
0.05
0.10
0.15
0.05
0.10
0.15
Gemini Accuracy
Acttraay
Gemini Aoturay
Figure 6: Accuracy estimated by CLIP versus accuracy esti- mated by Gemini.
Figure 7: Accuracy estimated by Qwen-VL versus accuracy estimated by Gemini.
Figure 8: Accuracy estimated by LLaVA-1.5 versus accu- racy estimated by Gemini.
30

---

Published as a conference paper at COLM 2024
4obot
Gudoo
Saladt
Colas]
Backayounbl
Skykol
Attinn
auu
cnu
Exs
(a) Accuracy of generated images on object-inference tasks in COBSAT.
fodail
Godzz
Solaii
Cakeell
Beiegombill
Slylell
Antionll
Tuttrrll
ORL
GEL
Bas
(b) Accuracy of generated images on attribute-inference tasks in COBSAT.
Sodac
Caoc
Dacagrund
State
Artoun
Tostze
SEED
ELan it
Da
Haltaa spe
air n
Dx
culf
Matthus p
Goea
Quea
ars
Cladi
Cled
Da
1al% WATT
GIPSA
(c) Accuracy of generated image descriptions on object-inference tasks CoBSAT
Belect
Oakel
Badarond
Solall
Adton
Taturell
Averag
Elaok
Da
GRED Eab
Geea
are
Clad
Gu
Qun
To Clei
Gut
Qon arro
SEED
(d) Accuracy of generated image descriptions on attribute-inference tasks CoBSAT
Figure 9: T21-ICL performance of all evaluated MLLMs on the CoBSAT benchmark with 2,4,6,8 in-context demonstrations.
GILL's limited performance can be attributed to its training paradigm, which is not opti- mized for tasks requiring unified understanding and generation of multimodal content (Ge al., 2023b). Specifically, this limitation stems from its training that omits interleaved image- text data and the absence of image generation model during its training process (Koh et al., 2023). Meanwhile, both Emu and Emu2 update all components in their model, and there is empirical evidence showing that they can better understand multimodal prompts. During instruction fine-tuning, Emu has been fine-tuned exclusively on the LLaVA dataset (Liu et al., 2023b) in the context of image-text tasks, while Emu2 is fine-tuned on more image-text pair data, including LLaVA and LLaVAR (Zhang et al., 2023d). In contrast, SEED-LLaMA benefits from instruction fine-tuning across a broad range of datasets, including both multi- modal and text-to-image generation datasets such as Instructpix2pix (Brooks et al., 2023), MagicBrush (Zhang et al., 2024), JourneyDB (Sun et al., 2024), DiffusionDB (Wang et al., (LAION, 2022), and VIST (Huang et al., 2016). This specific text- to-image generation dataset for instruction ane-tuning likely accounts for SEED-LLaMA's enhanced performance in T2I-ICL tasks when compared to Emu and Emu2.
2023c),
Warning: it should be noted that Fmu2 has results for and 4-shot scenarios, but results for 6 and 8-shot scenarios unavailable due to resource constraints, as Emu2 demands excessive memory.
31

---

Published as a conference paper at COLM 2024
E1.2 Assessing Generated Image Descriptions
For image description generation, Figure 9() and (d) illustrate the performance of MILLMs in performing T2I-ICL for object-inference and attribute-iffrrnnce tasks, respectively. We observe that Gemini, Qwen-VL, Claude, and GPT-4V stand out by significantly surpassing other MLLMs in most tasks. Among these leading models, Qwen-VL, Claude, and GPTT-VV show comparable results, whereas Gemini outperforms them all.
To further investigate the performance of each model, we offer examples of prompts and their corresponding image descriptions, generated by MLLMs in Figure 19, 20, 21, 22, and 23. We observe that SEED-LLaMa, and GILL often struggle to produce relevant textual output. GILL, tends to produce disjointed sentences like "person - bird on the beach- watercolor painting watercolor," exemplified in Figure 20(a). SEED-LLaMA, on the other hand, predominantly generates images, defaulting to the text "I have generated an image," " regardless of varying instructions. Emu, Emu2, LLa,A-115, and LLaVA-NeXT all tend to describe the images contained in the prompt instead of making predictions. This is expected for LLaVA models since they are mostly trained for single image inputs with related questions and answers. Their primary function is to describe and answer questions related to the single image inputs rather than making predictions. In terms of image-text datasets, Emu and Emu2 are also instruction fine-tuned on LLaVA and LLaVAR datasets, thus sharing the same property as LLaVA models. However, they perform slightly better than LLaVA models. For instance, Emu makes the correct prediction in the second example in Figure 19(.. This improvement can be attributed to their pretraining on many other datasets, including interleaved image and text datasets such Multimodal-C4 (Zhu et al., 2023b). For the leading models, which include Gemini, Claude, Qwen-VL, and GPT- 4V, Qwen-VL is the only one that includes detailed information on the training datasets and paradigms. Notably, Qwen-VL benefits from pretraining on a broader dataset than Emu, GILL, SEED-LLaMA, LLaV1-,,5, and LLaVA-NeXT, contributing to its enhanced performance (Bai et al., 2023b).
E1.3 Impact of Number of Demonstrations
In this part, we analyze how the number of demonstrations affects the performance of T2I-ICL. An interesting observation from Figure 4 is the lack of a consistent pattern in how performance is influenced by an increase in the number of demonstrations. For example, the accuracy in generating image descriptions for models such as Emu, Qwen-VI, and LLaVA initially increases and then decreases with an increasing number of demonstrations generally. Conversely, SEED-LLaMA's accuracy first decreases and then increases.
This non-monotonic performance trend with a growing number of demonstrations can potentially be attributed to two factors. Firstly, with a higher number of demonstrations, there may be an insufficient number of pertaining samples featuring the corresponding number of image inputs. For example, LLaVA encounters a context length limitation when presented with eight image inputs, resulting in the model generating only empty strings in 8- shot cases. Secondly, existing evidence indicates that an increase in demonstrations does not necessarily correlate with enhanced performance (Xie et al., 2022; Brown et al., 2020). Brown et al. (2020) demonstrate that for some datasets (e.g., LAMBADA, HellaSwag, PhysicalQA, RACE-m, CoQA/ /SAT analogies for smaller models), GPT-3's zero-shot performance may surpass one-shot performance. Similarly, Xie et al. (2022) found that zero-shot scenarios can sometimes outperform few-shot ones, although performance tends to recover with the addition of more examples. Xie et al. (2022) posit that an initial decrease in accuracy may be due to the distracting structure of prompts in such settings. Theoretical insights from Lin & Lee (2024) shed light on this phenomenon, suggesting that models initially rely on task retrieval and prior knowledge for predictions with a low number of demonstrations, shifting towards task learning the number of demonstrations increases. The presence of a limited number of initial demonstrations might result in the retrieval of an incorrect task, potentially causing decline in ICL performance. As more demonstrations are added, performance is anticipated to improve, as the model increasingly depends on task learning, which is improved by a greater number of demonstrations. However, in the MLLM scenario,
32

---

Published as a conference paper at COLM 2024
due to the scarcity of prompts with multiple images in the pretrained dataset, we do not
anticipate observing this phenomenon.
E1.4 Textual Information v.s. Visual Information
(b) Image description
(b) Image description gen- (a) Image generation generation eration
(a) Image generation
Algereet Testral Netoraiio Algurent Viraal Inderstiie
Algreeet Tistral hommacion Abgreeet old Venal Brlonationn
CloestCanaarrica ArtrbetsGeroeeca Avamy
Garee-GGmiiin Ausizacy
Avrraay
.504
0.5
0.5-
0.5
0.0
Rana
GILL
SERD SERD-LLMM
LLaMis
0.0
Elara
Emnz
GILL
SURD-1BaMt
1Laxoe11 1.5
LLAV--NoNT
Genini
Quer VI.
GPT--N
Clande
.00
0.0
Das
tt ELA
Gann
Crers Vi
OPEW
Cludk
Figure 11:
Comparison of alignment with tex- Figure 12: Comparison of object genera- tual information versus visual information when tion accuracy and attribute generation MLLMs perform two-shot T2I-ICL on the CoBSAT accuracy when MLLMs perform two- shot T2I-ICL on the CoBSAT tasks.
tasks.
In this part, we investigate whether textual or visual information contributes more to the prediction of MLLMs. We assess this by evaluating how well the output of MLLMs aligns with both types of information. For textual alignment, we concentrate on how accurately the models generate images image descriptions that match the given textual query. As example, consider the scenario in Figure 2, where the text instructs "red." In this context, we consider the output textually aligned if it features a red object. We employ similar approach to measure the visual alignmmett of the outputs. Similarly, for visual alignment, we examine whether the generated ted images or their descriptions accurately incorporate elements from the images presented in the prompts. Taking the same example, an output is visually aligned if it correctly represents aspects like "car," which the common feature in the demonstration images. Employing these criteria allows us to determine which models are more influenced by textual queries and which lean toward visual cues.
Figure 11 reveals distinct patterns in how MILMs respond to these inputs. Models such as Emu, Emu2, GILL, LLaVA-1.5, and LLAVA-NeXT demonstrate a marked reliance on visual information in their inputs. This aligns with our findings discussed in Sec. F.1.1 and F.1.2. As we discussed in Sec. F.1.1, GILL's training exclusively on the CC3M dataset (Sharma et al., 2018b), an image-caption corpus, limits its predictive capabilities. For Emu, Emu2, LLaVA-1.5, and LLaVA-Next, they consistently generate descriptions of the images present in the prompt rather than predicting the next image based on the sequence, thus ignor- ing the textual query in the prompt. In contrast, the models that perform well, such as SEED-LLaMA for image generation and Qwen-VL, GPT-4V, Claude, and Gemini for im- age description generation, demonstrate a more balanced use of both textual and visual information compared to the other models.
E1.5 Object Generation v.s. Attribute Generation
We are also interested in evaluating the proficiency of different MLLMs in inferring objects (e.g., car, chair) and attributes (e.g., color, style). As such, we report the accuracy of MLLMs in generating the correct objects and attributes. These results are depicted in Figure 12. Our observations reveal that all MLLMs perform better in generating correct objects, indicating that the task of generating accurate attributes presents a greater challenge compared to generating correct objects.
33

---

Published as a conference paper at COLM 2024
(a) Accuracy of generated images on object-inference tasks.
(b) Accuracy of generated images on attribute-inferene tasks.
Talp
Toral
(c) Accuracy of generated image descriptions on object-inference tasks.
Beliewe
(d) Accuracy of generated image descriptions on attribute-inference tasks.
Figure 13: Performance of considered MLLMs the challenging version of the CoBSAT dataset with misleading information in the textual inputs. We evaluate the T2I-ICL performance of various MLLMs with 2,4,6,8 in-context demonstrations. Low performance is observed across almost all evaluated MLLMs on this variant of our dataset, indicating limited capacity of existing MLLMs in filtering out misleading information from prompts. 2
E1.6 A Challenging Version of CoBSAT
We also investigate a more challenging task type that introduces misleading information into the textual input. This aims to evaluate whether MLLMs can accurately identify and ignore irrelevant information. Note that Claude is not considered in this experiment.
Prompt Design. To this end, we consider inputs of the form (x, instead of just x, where E represents the misleading information that does not affect the output y. Here is an example with 4-shot input when f "car."
comple
examgle
commle
esomple
quary
red
box:
Image: red
blue
[Image:
yellow
leaf: [Image: yellow
black
book: [Image: black car)
green
hag.
For prompt generation, we base it on the original prompt design, but with an added twist: for each prompt created with a sampled latent variable 0, we introduce a misleading instruction for each example within the prompt. This is done by sampling the misleading information ê E 0 (0} without replacement, thereby adding an extra layer of complexity to the task. In the case of prompts with misleading textual inputs, for each prompt with sampled 0, we further sample e 0/ (0) without replacement for each example.
Warning: it should be noted that Fmu2 has results for 2 and 4-shot scenarios, but results for 6 and 8-shot scenarios unavailable due to resource constraints, as Emu2 demands excessive memory.
34

---

Published as a conference paper at COLM 2024
Results. Figure 13 illustrates the performance of MLLMs this challenging version of the CoBSAT dataset. We note a significantly poor performance across all MLLMs, with the exception of SEED-LLaMA in image generation. However, even SEED-LLaMA's per- formance shows a decline in most tasks compared to those in the CoBSAT dataset, which is visualized in Figure 4. These results suggest that current MLLMs also heavily rely textual instructions and struggle to filter out misleading information. We anticipate this to be challenging task for future MLLMs to overcome.
F22 Understanding Challenges in T21-ICL (Detailed Version of Sec.
In this section, we add more comprehensive details related to the experiments in Sec 6, to better understand the challenges T21-ICL. Our further experiments will mainly explore SEED-LLaMA, Gemini, and Qwen-VL. Previously, we identified SEED-LLaMA as the leading free model for image generation, whereas Gemini and Qwen-VL excel as the top free models for image description generation scenarios.
E2.1 Is Multimodality a Primary Challenge in T2I-ICL?
In Sec. 5, we find that SEED-LLaMA and Qwen-VL. achieve only around or less than 50% accuracy on most tasks. This is in contrast to the impressive results their underlying LLM demonstrates in Textual ICL (T-ICL) (Touvron et al., 2023; Bai et al., 2023a). In this experiment, our objective is to determine whether multimodality the primary cause of this reduced performance, whether MLLMs intrinsically struggle with these tasks even for T-ICL.
Prompt Design. In this part, we evaluate SEA's capability in image generation and Qwen-VL and Gemini's proficiency in image description generation by modifying the prompts be entirely textual. We achieve this by replacing every image in the prompts with corresponding detailed descriptions, which are initially created by LLaVA and ChatGPT. These descriptions are then reviewed and corrected by humans ensure their accuracy. For example, in the first example depicted in Figure 2, the image [Image: red car) is replaced with a descriptive text: "The image portrays red Volkswagee Golf R, a compact sports car, stationed on wel road under a dark sky, with its vivid red color prominently contrasting the background." " Furthermore, to guide SEED-LLaMA in generating images rather than their descriptions, append the following instruction at the beginning of the prompt: "We provide a few examples, each with an input, and an output of the image description. Based on the examples, predict the next image description and visualize it." Similarly, for Qwen-VL and Gemini, we include an instruction: "We provide a few examples, each with an input, and an output of the image description. Based on the examples, predict the next image description," " with the focus on predicting the next image description without the visualization component. This distinction aims to direct SEED-LLaMA towards image generation, whereas Qwen-VL and Gemini are instructed to generate image descriptions.
Results. The results are reported in Table 7. For 2-shot cases, SEED-LLaMA exhibits similar accuracies in both T21-ICL and T-ICL, but in 4-shot instances, T-ICL surpasses T2I-ICL in eight out ten tasks. The disparity is even more evident for Qwen- VL and Gemini; under THICL, it significantly outperforms T21-ICL, especially in 4-shot situations. These findings confirm our first hypothesis, indicating that multimodality is the primary challenge in T2I-ICI.
F2.2 Is the Image Generation a Primary Challenge in T2I-ICL?
To verify the second hypothesis, that image generation itself presents primary challenge in T2I-ICL, we conduct an experiment with 0, 2, and 4-shot image generation tasks, with textual inputs updated precise labels. For example, in the initial scenario from Figure 2, the terms "White," " "Blue," and "Red" are updated to "White car," "Blue car," and "Red car," respectively. For this experiment, exclude MLLMs that do not generate images. Instead, we focus on MLLMs that are capable generating images, including Emu, GILL, and SEED-LLaMA.
35

---

Published as a conference paper at COLM 2024
Model
Shot
Melhod
Object-Infenecce Task
AttrbuuteInferene
Colord
Bockground
Style1
Actiond
lestured
Color-11
Background-1
Style11
Actior-11
Testure-11
SUED-LLaMA
2
T23-CC
T-CCL
614
.680
.15
.266
.200
.279
.182
.96
267
.315
467
206
.27
.184
.261
.192
163
T2I-CL.
482
211
114
.153
.22
252
4076
.68
.307
118
T-CCL
.381
.299
.317
.33
266
.15
.228
Qwen-VI
2
T21-CL
T-CCL
.844
475
313
692
.78
464
338
457
272
469
579
.899
238
A28
388
.16
4
T21-CL
T4CL
473
560
439
851
.97
571
.679
.972
454
.90
740
568
.885
364
288
341
719
546
.27
434
Gemini
2
T23-CL
T-CL
.979
865
.077
791
.692
.315
.995
5177
.744
201
.150
.355
.100
.583
.685
.60
.725
.668
.361
390
T23-CCL
931
900
-300
.237
.851
709
773
.81
.318
553
T-CCL
9988
965
.888
.965
1927
.77
7700
.835
.763
.812
Table 7: Comparison of Text-to-lmage ICL (T21-ICL) versus Textual ICL (T-ICL) accuracy on our dataset. To perform T-ICL on our dataset, we replace all images in the prompts with their corresponding detailed descriptions. Underlined numbers indicate the highest accuracy achieved for each model and task across various shot numbers, while bold numbers indicate the highest accuracy for each specific combination of model, task, and shot count. In this experiment, we focus on three MLLMs: SEED-LLaMA, which is used for image generation; Qwen-VL and Gemini, utilized for generating image descriptions. MLLMs demonstrate notably superior performance in T-ICL compared to T21-ICL, particularly in the 4-shot scenario.
Model
Shot
Protise Textual Inputs
Object-Infermece Task
Attribute-nffnwnce Task
Color-I
Background-]
Style1
Action-1
Texture
Coloe
Background II
Style II
Action
Texture 11
.356
.356
.264
.286
.182
.667
.241
.635
.265
4
.609
.82
318
211
.284
3303
.161
305
.286
.12
.688
232
.441
1776
.299
.355
.278
207
.245
.005
.064
055
075
1.00
4
.663
.001
018
.045
.366
.056
.045
.091
.097
336
.05
083
.122
107b
.009
.077
1772
.065
X00
0
251
256
.211
J33
.32
252
223
J28
1226
.22
106
.041
.073
.87
022
.159
.044
452
166
.178
.054
.125
.064
.133
.092
.055
.037
.065
Table 8: Accuracy comparison SEED-LLaMA, Emu, and GILL: with without provid- ing precise textual inputs. Bold numbers represent the highest accuracy for each task and shot count, comparing scenarios with and without descriptive textual inputs. Underlined numbers indicate the highest accuracy for each task across various shots.
Results. Table 8 presents a comparative analysis of three considered MLLMs's perfor- mance in T2I-ICL, both with and without the inclusion of precise textual inputs. We observe that in scenarios with both 2 and shots, the presence of precise textual inputs leads to significantly higher accuracy in image generation compared to when these inputs are absent for SEED-LLaMA and GILL, whereas Emu's performance does not follow discernible trend. Crucially, the analysis shows that, even with precise inputs, all models sustain comparable level of performance across different tasks, with accuracies remaining under 50% in most cases. This suggests that the task of image generation remains a considerable challenge for contemporary MLLMs, affecting their efficacy on the CoBSAT dataset.
E3 Enhancing MLLMs T21-ICL Capabilities (Detailed Version of Sec.
In this section, we supplement Sec 7 with additional experimental details, discussion, and expanded experiment results, exploring techniques that could potentially enhance the performance of MLLMs in T21-ICL.
E3.1 Fine-tuning MLLMs CoBSAT
In this experiment, we investigate the impact fine-tuning MLLMs on our dataset in improving its T2I-ICL capabilitis We focus SEED-LLaMA Qwen- VL for this investiga-
36

---

Published as a conference paper at COLM 2024
tion. Consequently, we compare the T2I-ICL performance of the pretrained-only version of Qwen-VL nad SEED-LLaMA with their corresponding variant that is fine-tuned on our dataset.
Training Setup. We fine-tune two instances of both SEED-LLaMA and Qwen-VL, one 2-shot dataset and the other on 4-shot dataset, and then compare their performances with their non-fine-tuned counterparts on the T2I-ICL test set. For both models, we fine-tune their LLM backbone only using LoRA (Hu et al., 2022) with a rank of 64, a weight decay of 0.1, and a warm-up ratio of 0.01 for 5 epochs.
Training and Test Sets. We employ two distinct strategies for splitting the training and test datasets. In the first strategy, the training set comprises prompts from all ten themes, ensuring that attributee and objects in the test set are not previously exposed in any training prompts. In the second strategy, the training set excludes the themes that are included in the test set, enabling us to assess whether a model fine-tuned on specific tasks can generalize to other tasks. Note that the tasks configured by the second approach are inherently more challenging.
(Data Split A) The training and test sets are constructed by splitting the predefined lists of text inputs and latent variables (from Table 5, denoted as X' and +) into training (Auain train) and testing (Nest, Orest) subsets for each task, in a 1:1 ratio. Therefore, all the in- context demonstrations and textual queries in the test sets are unseen from the training set. For the training set, we create the dataset by considering all possible combinations and sequences of text inputs from Xtain and latent variables @train across all tasks. We fine-tune Qwen-VL on this unified training set containing the prompts from all tasks. For the testing set, we generate 250 prompts for each shot across various tasks. Each prompt is obtained by randomly sampling 0 E Otest and (xx) N+1 E test which are then paired with the corresponding collected images (yn (yn) where yn ~ fo(x)) This process results in in-context demonstrations (xn, )N and a single textual query IN
N++
(Data Split B) In this data split, we intensify the challenge by increasing the disparity be- tween the training and testing distributions. Instead of merely including unseen objects and attributes from the same themes in the testing dataset, this split introduces unseen themes. For example, the results shown in Table 9 for split B color-themed tasks (i.e., Color-I and Color-II) are derived from a model fine-tuned on the other four themes (i.e., eight tasks). Thus, the model is not fine-tuned on color-themed tasks but is evaluated on them. This method is uniformly applied across all themes: each theme is evaluated using model fine-tuned on tasks from the other four themes. Thus, the training set includes four themes, while the test set comprises a different, fifth theme. Consequently, the results in Table for split B reflect different models, each fine-tuned on distinct training set.
Results. The results are summarized in Table For data split A, both models demonstrate significant improvements in T21-ICL performance following fine-tuning. In the more chal- lenging tasks defined by data split B, SEED-LLaMA demonstrates strong generalization to unseen tasks after fine-tuning, while Qwen-VL exhibits more difficulty in generalizing. Overall, these results suggest that fine-tuning MLLMs on a T2I-ICL dataset generally en- hances their overall T2I-ICL capabilities. Example output images from the pre-trained and fine-tuned versions of SEED-L.LaMA on split A are provided in Sec. H.2.
E3.2 Intergrating Chain-of-Thoughe with T21-ICL
Another widely utilized method in prompt engineering is Chain-of-Thought (CoT) (Wei et al., 2022). This approach involves incorporating a simple instruction, such as "let's think step by step," " prompting the model to sequentially generate concise sentences that outline the reasoning process, commonly referred to reasoning chains or rationales. The chains are subsequently embedded into the subsequent prompt to obtain the final answer. CoT has been particularly effective in enhancing performance, especially for complex reasoning tasks,
37

---

Published as a conference paper at COLM 2024
Model
Shot
Fine-tuned
Split
Obfern-nnerrence Task
Altribue-Inferene Task
Colorel
Backgroand--
Secle-
Action-
Tecture-
Color-ll Backgroeand-11
Style-ll
Actionall
Tecture-ll
.108
.196
.76
540
166
.344
208
465
.58
.105
.192
.140
SEEDALLaMA
5
752
.855
330
272
2000
196
376
.280
.180
300
x
.012
.350
192
.944
.088
.580
532
.140
.196
.365
7
.245
.132
8
.540
236
246
.12
.32
.26
244
.112
.222
224
A
.852
.244
212
.86
.32
.566
314
J18
.520
Qwnn-V
2008
2022
2726
.08
.328
192
272
224
.212
172
7
216
.58
.996
306
.085
.656
.80
.922
728
.30
.661
.740
350
.240
.212
.065
Table 9: T2I-ICL accuracy comparison of pretrained-only versus fine-tuned (FT) MLLMs.
Underlined numbers signify instances where the fine-tuned model surpasses the pretrained model in the same scenario, while bold numbers indicate the top performance for each shot across various methods within their tasks.
when applied to large-scale models (Wei et al., 2022). In our experiment, we investigate the impact of integrating CoT on the T2I-ICL performance of MLL.Ms.
Prompt Design. We employ two-step inference process utilizing two distinct prompts. The initial prompt builds upon the default examples showcased in Figure 2. To this, we prepend the statement, "We provide few examples, each of which is an imput-output pair where the output is a description of the image associated with the input. Based on the examples, the task is to predict the next image description. nn...nnnnnnra nn)nn" This placed at the beginning of the prompt. Additionally, we append, "\n') n n Before predicting the next image, let's first think step by step and analyze the relationship between the text input and image output each example. nn an n1n""" n to the end of the prompt.
Following the MLLMs' responses, the second prompt comes into play. includes the first prompt and the MLLM's response as part of the new prompt and extends it with, "An'n'y on the analysis, plense generate the next image for the request 'red: for the image generation scenario, and "nn'snn Based on the analysis, please describe what the next image should look like for the request 'red:" for the image description generation scenario when the textual query is 'red.' In each case, 'red' is replaced with the respective textual query according to different prompts.
Model
Shot
Method
Object-Inference Task
Attribute-laferencc Task
Color
Background-1
Styl-I
Action
Toxture
Color
Backgrounc
Style
Actionn
Toxture
T21-CL Col T2I-ICL
.600
.388
210
.152
.196
.297
.163
SCID-LLaMA
.179
206
.222
388
.195
T21-KCL
482
211
141
1153
.22
.222
.76
268
207
106
CoT T2I-ICL
.600
.33
244
.42
.208
.303
.300
.355
2411
.171
Qwen-VL
2
Col T2I-ICL
T21-X1
.45
.211
.444
313
387
378
.664
217
.388
.363
.477
.150
349
.379
.260
.288
176
.388
111
.16
T21-X1
.579
.41
CoT T2I-ICI
379
274
404
340
T2H-3I
794
315
5177
Gemini
CoT T2I-ICI
.445
T2I-K1
.944
938
540
737
.61
2009
773
.484
818
553
Col T21-ICL
.966
.957
798
.16
.45
.917
.97
.20
.97
258
Table 10: Assessing the impact of Chain-of-Thought (CoT) prompting on T2I-ICL. The evaluation metric is accuracy, with the numbers in bold highlighting the highest accuracy achieved for each model, number of shots, and task, and underlined numbers indicate the highest accuracy achieved for each model and task across different numbers of shots. In this experiment, we evaluate three MLLMs: SEED-LLaMA for image generation, and Qwen-VL. VL and Gemini for image description generation. Our findings reveal that CoT significantly improves Gemini's performance. For SEED-LLaMA and Qwen-VL, the enhancement offered by CoT is ambiguous in 2-shot scenarios. However, in 4-shot instances, CoT markedly enhances the performance of SEED-LLaMA, while it still shows no benefit for Qwen-VI."
The results are reported in Table 10. With the integration of CoT, Gemini shows better performance across the most of tasks in both 2-shot and 4-shot scenarios. Similarly, SEED-L.LaMA shows significant improvement in T2I-ICL performance across all ten tasks
Results.
38

---

Published as a conference paper at COLM 2024
in the 4-shot scenario. Conversely, for Qwen-VL, no concrete evidence suggests that CoT enhances its T2I-ICL performance, In fact, we find that Qwen-VL often avoids providing definitive answers in the second step of making predictions, and responds with general statements like "Given the request 'black'', , we can infer that the image outpul should be related to a black object or theme. However, without more specific information, it's difficult to determine the exact relationship between the text imput and image output. Without additional context, it's impossible to accurately predict the next image." Therefore, in many instances, standard T2I-ICL without CoT appears to outperform the version integrated with CoT for Qwen-VL. Exploring additional prompt engineering methods such as self-consistency sampling (Wang et al., 2023b) or Tree-of-Thought (Yao et al., 2023) to elicit more concrete responses from Qwen-VL is possibility. Specifically, self-consistency sampling involves generating multiple outputs at non-zero temperature setting and selecting the most appropriate one from these options. On the other hand, Srre-offThouge expands upon CoT by considering multiple lines of reasoning at each step. However, such investigations fall outside the scope of this paper, and we identify it as one of the interesting future directions. We provide example conversations of integrating CoT and T2I-ICL in Sec. H.3.
E3.3 Articulating the Text-to-lmage Relationship in Prompts
In our dataset, the goal is to check if the MLLMs are able to learn the mapping from the textual input and the visual output based on the in-context demonstrations. experiment, we investigate the performance of MLLMs T21-ICL if we explicitly write down this relationship in the text prompt. For instance, for the Color-I task, we directly add the following sentence to the beginning of the prompt: "Please identify the common main object in the images, and generate another image of this object in the requested color." " The detailed prompts for all tasks are provided in Sec. D.1.2.
this
Model
Shot
Explict Instruction
Object-Inference Task
Attribute- Infereecee Task
Color
Background
Style
Action-l
Texture-1
Colze-1l II
Background II
Style
Action- 11
Texture LI
SERIALLaMA
2
.779
680
.391
368
.231
.005
.301
.182
.270
.19
357
.287
.467
446
.350
.297
.249
.261
.185
.163
X
211
.141
053
122
.268
105
.281
318
322
.433
.426
.268
238
Qwen-VI.
196
240
.516
.317
.373
7
315
291
341
.755
.73
277
.91
.317
.11
527
.04
Gemini
2
.119
.865
.74
431
.535
315
.60
.177
700
.76
128
.55
735
363
.155
.360
.755
.37
.118
310
3
.908
.908
.500
.737
.61
.709
273
.486
.818
.53
.755
-125
.91
.19
.125
Table 11: Effect of explicit instruction on T2I-ICL performance of MLLMs: articulating the text-to-image relationship in prompts. The evaluation metric is accuracy, where underlined numbers denote the highest accuracy achieved by each model and task across varying shot numbers, and bold numbers represent the top accuracy for each specific combination of model, task, and shot count. This evaluation focuses SEED-LLaMA for image generation, and Qwen and Gemini for image description generation. We find that explicit instructions significantly enhance the T2I-CC capability of SEED-LLaMA, especially in the 4-shot scenario. However, for Qwen-VI. and Gemini, explicit instructions do not show similar performance gains.
Results. We present the experiment results in Table 11. Results show that SEED-LLaMA's
performance significantly improves with explicit instructions, surpassing its performance in T21-ICL without instructions for seven out of ten tasks in the 2-shot cases and all tasks in the 4-shot cases. Notably, in the 4-shot case for the Color-T task, SEED-LLaMA achieves high accuracy of 83.2% with explicit instructions, compared to only 48.2% without them. Furthermore, the performance of T21-ICL with explicit instructions improves when moving from 2-shot to 4-shot scenarios, in contrast to the situation without explicit instructions where SEED-LLaMA's T2I-ICL performance declines as the number of demonstrations increases from 2 to 4. In contrast, Qwen-VL does not show comparable improvements, owing to reasons similar to those discussed in Sec. F.3.2, including the generation of irrelevant responses like "Received." ". Similarly, Gemini also fails to demonstrate improvements. To be
39

---

Published as a conference paper at COLM 2024
more specific, we find that Gemini consistently ignores the textual query after articulating the text-to-image relationship in prompts. To handle these issues, more careful prompt engineering could be applied, although it is beyond the scope of this paper.
G Extended Discussion
This section is an expanded version of Sec 8, discussing the conclusion, limitations, and future works in greater detail.
G.1 Conclusion
In this work, we identify an important yet underexplored problem - T2I-ICL, and explore the capability of MLLMs to solve it. To facilitate this investigation, introduce CoBSAT, comprehensive benchmark dataset encompassing ten tasks. Our experimental evaluation of MLLMs on this dataset reveals that while many MLLMs have difficulty in effectively learn- ing from in-context demonstrations during text-to-image generation, a few MLLMs, such as GPT-4V, Qwen-VL, Gemini, Claude, and SEED-LaMA, show comparatively reasonable performance. T'rrough further studies on free top-performing models SEED-LLaMA, Gem- ini, and Qwen-VL for both image and image description generation, we identify two key challenges in T2I-ICL: (i) the integration and understanding of multimodal information, evi- denced by superior results achieved with textual ICL for the same tasks; and (ii, particularly for image generation models) the actual process of image creation, as even straightforward image requests with clear descriptions often yield suboptimal performances.
To improve MLLMs' performance in T21-ICL, we carry out additional experimental studies. These studies suggest that fine-tuning and CoT can substantially enhance T2I-ICL capabili- ties. Meanwhile, it is worth noting that in our dataset, intentionally exclude explicit task descriptions to assess whether MLLMs can autonomously adapt to tasks based solely on in-context demonstrations alone. In the ablation studies, we find that providing clearer task instructions might be promising strategy for enhancing T2I-ICL performance. However, these prompting engineering strategies might need to be combined with others to achieve consistent improvements.
G.2 Limitations and Future Works
While our study is early attempt to explore the T21-ICL benchmark dataset, many interesting questions remain open.
Impact of Demonstration Selection on T2I-ICL Performance. Existing research in textual ICI. has consistently demonstrated that the choice of demonstrations significantly influences ICL performances (Liu et al., 2022; Su et al., 2023; Rubin et al., 2022; Zhang et al., 2022b). In our study, we only employ random sampling select in-context demonstrations. This opens an interesting question: to what extent does the selection of demonstrations affect T2I-ICL performance? Moreover, our evaluation primarily assesses whether MLLMs can accurately generate images with the current content, without a specific focus on the quality of these images. Another natural question arises: how significantly does the quality of images used in demonstrations influence the overall quality of the generated image output?
Prompt Engineering Techniques for MLLMs. As discussed in Sec. 7, CoT demonstrates significant improvements in T2I-ICL performance for SEED-LL.MM and Gemini. How- ever, the prompt sensitivity of models like Qwen-VL poses challenge, as they tend to provide non-committal responses such as, "Without additional context, if's impossible to accu- rately predict the next image." This issue underscores the necessity of implementing more advanced prompt engineering techniques, including methods like Tree-of-Thought and self-consistency sampling, to address these limitations.
Once such issues are resolved, another interesting question arises: feasible to enhance existing prompt engineering techniques with multimodal capabilities? For instance, while
40

---

Published as a conference paper at COLM 2024
Sec. 7 focuses on prompting MLLM to perform CoT through textual analysis (as further exemplified in Sec. F.3.2), expanding this approach to a multimodal CoT that integrates both textual analysis and image grounding could potentially yield better performance. These open questions are identified as interesting future directions.
T2I-ICL for Image Editing. One notable absence in our dataset is tasks related to image editing. For instance, in the Color-I task, the goal is to generate an image of car in a color specified by the text query. In our evaluation, the type and background in both the example images from the prompt and the newly generated image may differ. However, there is a growing need for image editing applications where the task is to alter specific attributes (e.g., the color of a car) in an otherwise unchanged image. For such tasks, selecting images that strictly adhere to the given criteria (identical images with only specific attributes or objects altered), coupled with the development of sophisticated metrics, is critical to assess these more complex challenges effectively.
Exploring Wider Range of Themes. Our dataset primarily assesses MLLMs on elemen- tary themes, incorporating a specific range of objects and attributes within narrowly defined categories. For instance, in the style task, we consider styles such as watercolor, sketch, pixel art, origami, and others. Nonetheless, the realm of styles in real-world applications is far more intricate and varied, extending to include oil painting, rococo, steampunk, and beyond. Additionally, our dataset encompasses a limited set of themes. There are also many other interesting themes such counting. While it serves to test basic capabilities in T2I-ÍCI, a more comprehensive dataset, covering a broader spectrum of themes and a finer list of objects and attributes, will be crucial for evaluating more advanced model capabilities. In this scenario, the evaluation methodology may require refinement to more accurately identify the more fine-grained attributes and objects.
H Sample Outputs Generated by MLLMs
In this section, we provide sample outputs generated by the models under different scenar- ios. Specifically, Sec. H.1 contains a selection of sample prompts along with the correspond- ing responses generated by MLLMs across all ten tasks. Sec. H.2 displays selected sample prompts and the images produced by both the pretrained and fine-tuned SEED-LLaMA for all ten tasks. Furthermore, in Sec. H.3, we illustrate sample dialogues between users and MLLMs (including SEED-LLaMA and Qwen-VL) from experiments that combine the CoT with T2I-ICL.
H.1 Sample Prompts and Corresponding Outputs
Here, we showcase examples of outputs generated by various MLLMs, accompanied by their respective prompts.
Image Generation For image generation, five examples are provided for each themed task, as depicted in Figures 14, 15, 16, 17, and 18 for color, background, style, action, and texture themes, respectively. Observing these figures, it is evident that SĒED-LLaMA excels in image quality among the three MLLMs capable of image generation. Notably, SEED-LLaMA produces images that not only align with the true labels but also closely resemble the images in the prompts. For instance, in Figure 14, images with plain backgrounds in the prompts lead to similarly styled outputs.
Image Description Generation In Figures 19, 20, 21, 22, and 23, we showcase two exam- ples for each task, covering color, background, style, action, and texture themes. For brevity, some lengthy responses from MLLMs have been truncated these figures, retaining only the key parts of the responses.
41

---

Published as a conference paper at COLM 2024
Imput
Fmmz
Browen:
Purple
Brown:
Purple:
Orange:
Het
Grren:
Yellow:
White
Book
Purple:
While:
Pinks
White:
Yellows
(a) Color-I
Latent Var.
Input
Ema
Emu2
GILL
SEED-LLaMA
Purple
Copp:
Box:
Bag:
Orange
Book:
Books
Hal:
Fink
Box:
Flat:
Book:
Grenn
Ball:
Bage
Bos:
(b) Color-I1
Figure 14: Examples of prompts and corresponding images generated by MLLMs for tasks within the Color theme.
H.2 Sample Outputs from Fine-tuning SEED-LLaMA CoBSAT
In Figure 24, we provide sample outputs generated by pretrained-only and fine-tuned SEED-LLaMA from the experiments described in Sec. and F.3.1. We observe that the images generated by fine-tuned SEED-LLaMA generally align better with the expected output.
H.3 Sample Outputs from Integrating CoT with T2I-ICL
In this section, we provide sample outputs generated by Qwen-VL and SEED-LLaMA from the experiment of integrating CoT with T2I-ICL, as detailed in Sec. 7. Specifically, Figures 25, 26, and 27 illustrate the example prompts and corresponding outputs generated by Qwen-VL for the tasks Color-I, Action-ll, and Texture-II. Similarly, Figures 28, 29, and 30 display the example prompts and outputs produced by SEED-LLaMA for the tasks Color-I, Background-II, and Style-I.
42

---

Published as a conference paper at COLM 2024
Labent Van
Expul
Finu
Emu2
GILL
SEED-LLaMA
Bind
Seallooe:
Beacht
Watcrtall:
Lion
Gym:
Seafloor:
Beach:
Tiger
Glacier
Monkey
Waterfall:
Caver
Desett:
Zébra
Beach:
Wateriall:
Care:
(a) Background-I
Latent Var.
Exput
Eina
Emu2
GILL
LTED-LLAMA
Woterfall
Tigur:
Monkeys
Burd
Beach
Monkeys
Dog:
Lion:
Glacker
Monkeys
Lion:
Tigen
Monkey:
Cows:
Zebra:
(b) Background-II
Figure 15: Examples of prompts and corresponding images generated by MLLMs for tasks within the Background theme.
43

---

Published as a conference paper at COLM 2024
Lalend Van
Expul
Finui
Emu2
GILL
SEED-LLaMA
Cup
Lego:
Origami:
Sketchs
Chair
Jcon:
Pooel:
Waterooion:
Skeech:
Lego:
Pival:
Swetch
Graffitt
Apple
Futuristic
Origami:
Icon:
(a) Style-I
Latent Var.
Emput
Ems
Emu2
GILL
STED-LLaMA
Sketch
Car:
Apple:
Cup:
Watenoloe
Cap:
Apple:
Chain
Clup:
Ball:
Book:
Bags
Leaf:
Apple:
(b) Style-1l
Figure 16: Examples of prompts and corresponding images generated by MLLMs for tasks within the Style theme.
44

---

Published as a conference paper at COLM 2024
Lalend Van
Expul
Finui
Emu2
GILL
SEED-LLaMA
Dog
Rum:
Drinie
Swim:
Zeora
Elys
Switm::
Drinic
Fig
Eal:
Drinic
Angry:
Sheep
Eat
Angry:
Sleep:
(a) Action-I
Latent Var.
Emput
Ems
Emu2
GILL
STED-LLaMA
Swim
Monkey:
Sheep:
Dogi
Drink
Cat:
Starps:
Zabra
Eat
Bird:
Fige
Cat
Bird:
Sleep
Pige
Dog:
Sheaps
(b) Action-ll
Figure 17: Examples of prompts and corresponding images generated by MLLMs for tasks within the Action theme.
45

---

Published as a conference paper at COLM 2024
Labent Van
Expul
Finui
Emu2
GILL
SEED-LLaMA
Ball
Leather:
Wicker
Metal:
Box
Metal:
Soquined:
Wood:
Paper
Bag
Ceramic
Soquined:
Denim:
Cup
Plastic:
Denim:
Ceramic
(a) Texture-I
Latent Var.
Emput
Emna
Emu2
GTLL
STED-LLaMA
Metal
Blag:
Ball:
Wood
Lewl:
Book:
Box:
Wicker
Ball:
Cups
Hal:
Ceramic
Chaic:
Cup:
(b) Texture-Il
Figure 18: Examples of prompts and corresponding images generated by MLLMs for tasks within the Texture theme.
46

---

Published as a conference paper at COLM 2024
Input
Van
Imput
Purple:
Purple:
Frm
Fmmu2
GIFl
Fimn
Fmu2
Gi
feateer basipock sippes.
A buckyack the stgpers
imagy baclpacii
sentoy sparts oor driiee
A pioyle cur drising dence
sutonsbill
wagg
sparts
BLAVA-15
HLAVA-NAXT
LLAVA-NeXT
loange
image
Joor penmotted
The actt image sloulf bv a rlberap of Miv backpazk strgs.
Iveou fnatker Anctyrrk
jane grrvnated in image
car, slenscasing is grar
the purple spvets
heatlights The
grttle the yarple quart
geneaated should
thweeap the jrie!
car
etwet.
Qwen-VL
GPT-V
Qwen VL
GFT-4V
The nexr maae slould be clacopp
purple bacipack prrssus tugge, style stbran
arst sagge should venege dackgoontt
A alok race orr is bright anange, uith detaingg glowy fusik, raning aulin Mar the busggrannd
a Abraussing chanty
Gemint
Claude
Cemini
Claude
The ueul inage apears ae purviz baclrcct The betrouk lieyy vaar festuna rusttple comparaantt paciets for stnnge and arymitationn
purplr Aarkyoct liac niype seatl! lag zu
inghqpality
A brijht arange Lamborghini Avents
The lvagee apenns depic! skatt and specty sports orr to vibnert enange
(a) Color-I
Intuet Var
Imput
Latent Var
Imput
Purple
Cap:
Box:
Bag:
Orange
Ball:
Book:
Car:
GILL
purye cofle mag siting Mip a Dou.
A papple tex skilng akile bechgraased.
prodict
tox the poslict.
As entree parked en a adile suylies.
Al evange bast sitting on e inlite beckgound.
pessinailty and Iis you asse your Bull.
SEED LLMMA
LLaVA-15
LONT
SEED LLaMA
LLAVA-L5
LLAV
nest imsaee should expooit with
The next lvagee should be perio'' lend lxeding the fuol.
Mur
stapt
fieetigg planed nkite suyfer.
ciffte wag..
iwage.
Qwwn-V11
CUT-4V
Qwwn-VL
CPT-IV
The aust sharl be clanop a puryle gi boy en whide lack- grannd.
parple box unth steple clem doige on ulite teckgromnd
poesidty soors our or a dbost cer The color of the car stoud de bright edle, ounge riz.
anst vhould
A hask uith liing for with pogo unsble fives thr sile iwiand with backywordd
Gemini
Claude
Gemini
Claude
pwple wilh mAll dieer
squrtedippee ber contaises It Aas seis, unfere purple ovir und steple attuesaie desge.
of the Jont 1800s car mall chnore bumner and wkite sail tims.
The lseage skitse evange oolive best
with Hiik cvter and payes
(b) Color-II
Figure 19: Examples of prompts and corresponding images description generated by MLLMs for tasks within the Color theme. For brevity, some lengthy responses from MLLMs have been truncated, retaining only the key parts of the responses.
47

---

Published as a conference paper at COLM 2024
Input
Van
Imput
Seafloor:
Frm
Fmmu2
GIET
Fin
Fmu2
Gl
The west haage in clow- up of covnloskng antrraal
A Mar Nivi walking de waz in
perree (Ne like ausierostor paint
In lie vest owsge, Hate aisiting shrough apajyvune.
lsuagr of makking
putsott parser perwn
persss jersis
SENALLMMA
BLVA--15
HLAVA-NOTT
SEHD-LLMA
LLAVA-NeXT
vagg passily the jy. The beckgoontt skould a Mrssi of aiomants
2 jate genvnoted
the beech and the deroere restoreennt
300 fore byker the surthor
tslle the diclntor
image
sand, water, and posality scmoed
posably aeez
Qwen-VL
GPT-V
Qwen VL
The sexr mage slould de of wsterki! chear long
Murber stooding frost of collig naterfall somurded grown
The acst image should a clocopp if linn staiaing andyy Mar He
rusesti walling strdy bauch
billig
into belis.
ieclgrandd
Gemint
Clande
Cemini
Claude
Man renes monding in skuttee part
The ucul inage qrears dayki sturring wateffll some The imay liady divenior passrndd mavadie foingg and natinal
a nieer The nater ergenal cleer. ieckgresed grist
A malking on Anach
The wit innge appars At depiet bvach
(a) Background-I
Intuett Var
Imput
Latvent Var
Imput
Waterfall
Tiger:
Monkey:
Bind:
Beach
Monkey:
Dog:
Lion:
professioeas
shittggapp ligee stanting on a 1902 in foor of unterfall
A mankey Haa G stens ing we 1962.
diigia!
auildny the buach sitk ily Jun cetting in banggmand.
persoe Are
Jack
enter pove the Limilld States
belcke wost the acaas.
day
SEED LLaMA
LLaVA-15
LLAVA-NeNT
SEED LLMMA
LLAVA-15
LeAV
The ncd image should weetay's and Resal fiaturn, Thr auskgaand unterSai.
The next isagee skoud up doy's shostd be brech, watn nw swaa to the didtenor
gessvatal
walling
inage
Book
Qwvn-V11
GUT-4V
Qwwn-VL
CPT-IV
The shuuld festurs bind pundhcl se kranch is fruet of vadee-
aidd with astnont percked soney brouch is the Aeegrandd, pith sersee asterfill axssaling backgrursl
A Junn stmaing moby in Ahr seremaah, with the are writng Insigrond.
A lass eiting smames srde clar bir diy, vith dry gnow anund saan thr haciyaand
Gemini
Claude
Gemini
Claude
Toe owage ayparr
natestie Jollge The coeediigg sakee too ster rody cifs, crewting Amatitsking sanind ipectadi
The lsage appers deric fou ot with stosting
from a swiejaat. The and tij dul the antryfal dhe backgrand.
bird
A valliny the Avark wnthk agtniwour for Avdigrond
ts langrrondd ant lintheuer ofter
counts) atrecture stoble is the dutense.
(b) Background-II
Figure 20: Examples of prompts and corresponding image descriptions generated by MLLMs for tasks within the Background theme. For brevity, some lengthy responses from MLLMs have been truncated, retaining only the key parts of the responses.
48

---

Published as a conference paper at COLM 2024
Input
Var
Input
Origami:
Sketdo
Cluir
Frmu
Fmmu2
GILl
Fmn
Fmme
Cir
The sen megge a nod ou
A paperr vang satligg
The anst regge naterolor patsting of asalen chair.
A ptudined isage of Buskgraund
The aent Image is
SEMALLMMA
HLYAA-NeXT
SEHISTLaA
tmage of onfic, vain the plocal on a lall. The stglz maliti
lvage be
should poxt slyse allstro-
Evjay lWss picture
sysnatte whould le platagnpsk of Logo arructur.
jane grnvnated in image
date of waader
clair aitti purper
rin of sceeden chals.
buskgroudd image
boosn
style
He nsal festors
Qwen-VV
GPT-V
Qween
sex raag
disc-ap
complex pennl of Anacyp dotilial skantig
acst image slould satimester
sastmmair painting of with ghing juii appsnooe
lago
pointing
Gemini
Clande
Cemini
Claude
ner intel, retn-stelr medering of avesde osti: oadr single, gossetto doign
The and imnge aprwars au orgesti contion. it shawe aeilis Silke paper sirictur that resetiio slaps.
A satinallrr pointing of chein The cleir made of sood imnass
pmntl drowiny of cffec mmg
rectongule paltori.
(a) Style-I
Inteet Var
Input
Latent Var
Imput
Skeech
Car:
Apple:
Cap:
Wasercolor
Cap:
Apple:
Chair:
A not apple saan grewn fates os shiile buct grasssi
poaetl druniag of ax spple.
A Jivarieg of with a lof" on
persosaitt
A paisting of an apple uilh grenn lasees
sn apple nikite beiggrosss
LLMMA
LLaVA-15
LLaVA-NeXT
SEED LLMMA
LLaVA-L5
LLAV
The acd inage Wisch
The nnst lmage to be I'm yanting ing Care you bel av?
de coffee anttl spas a. The stylh of the paintly inprrsiziitt
this
Ancrpptiee provoded jor the rup osage.
of can. The divavig should &e aiene in minteesiee style
iwage
Qwon-V11
GUT--V
Qwwn-VV
CPT-IV
The stouls matt bendlo, slle vith cofer lws and suronnkel by cffer betes loi lates,
drvming
nuterolee
kandie on dhe right stk,, didded in gite a dire- discciinn (feet, on plain badgroand any diysts.
A vetionator peintngg a md civir with grawn laff the back.
fertuning gres leevs end paiieity spyle witl stulte enststtike on budggoundd
style miiseis
Gemini
Claude
Gemini
Clande
aveder
paiil dvaiiigg of cgfe oy
The abstsl or drviiigg eup atgg
pount!
cluir mith centio The char patited is light braw color, and the roskie painted rake
Plaver
(b) Style-ll
Figure 21: Examples of prompts and corresponding images description generated by MLLMS for tasks within the Style theme. For brevity, some lengthy responses from MLLMs have been truncated, retaining only the key parts of the responses.
49

---

Published as a conference paper at COLM 2024
Input
Var
Imput
Zeéora
Frm
Fmm2
GILl
Finn
Fmm2
Gl
The sest mager is certoon fiy drriiiig
A curtans dey licking
restrr (thotrotien of Ang drrstiig mater froe
In ibe noit isagee nodre seth vaiyss wess sutreneiig somn
artre nittl sunglasee rattig au Alter
persoe on skblle bask grownd
HLAAA-NeXT
nest image
gravatee shoula be catinen zehra sitliig nainheasstrigal buyy
lougue niching the vator tooque malling
and awile alg.
cleuts in the sty.
Qwen-VL
GPT-V
Qwen VL
The seat mage sloulf de of dag suisennng trngue egjyyeest
cartonn degune and white day unti augging sall paliiing
The artt sagge should of oehre drokiigg froe a sadee bottle wnth is planed
A can adva cartass chanater driek- lng tis nylecting surms siniight
exprosiin fict
Gemint
Clande
Cemini
Claude
The ucal inage apears dlear ca hase des Aepally playiny pusdaie of soter minntyyy piagfi! araana, the nate:
The nou inage appears ts dkyict extorful and nkimssedd wime of sebro foutiny as odaefil tonrr
cartaon dag auterting
mrtre arisking five
by ingltt
(a) Action-I
Inteet Var
Input
Latent Var
Imput
Swith
Monkey:
Shoap:
Dog:
Drink
Cat
Sheep:
Zebra:
certons dag newiing We prosentor.
A carlass slace on ge peserterr in silier.
of
A achis B drisiig aster How poist
shopp dnisking with strax.
You ber beker drtuls.
on
LLMMA
LLaVA-155
LLAVA-NeNT
SEED SCED-LLaMA LLaMA
LLAVA-L5
LeAV
The nest image do gesovitee slovil carioon
The nest isage do by de
pranide & icteaanl ahe image
avsnd wankeys
jicher and millng surt- board vn anve.
rulke.
Qwwn-V11
CUT-IV
Qwwn-VI
CPT-IV
steuld veinminn with server arc
nethen whory with flffy abite jucpinn reto el not una saitee iplury agsise' piste light badkgrurd
anst ehouls Modi whire strpod aetro druiding mattr froe a roud. The octin aculd be of the Libing ap of auter
sitn chur whse sky thr Aangeound Tis adiaa's paw should augest mare asni grising.
Gemini
Claude
Gemini
Clande
dog weaning HP
Wastestin appars cortste dipicied us sastooingg foatiig yow. surmanide by vader and hultiee, suggoning a plagil or qquatic
pink arrishig frose wardin glaws The rohre standing on the kini logs and glass
Daage shetse 2eb0 siesding backyaiid The adva aysswrs driskrey from the ustes adink foemai imaye.
widee mui aniing. The Nic and elor..
(b) Action-Il
Figure 22: Examples of prompts and corresponding images description generated by MLLMS for tasks within the Action theme. For brevity, some lengthy responses from MLLMs have been truncated, retaining only the key parts of the responses.
50

---

Published as a conference paper at COLM 2024
Input
Van
Input
Leather:
Wicker
Metal:
Sequined:
Wind:
Frmu
Frmu2
GIET
Finn
Fmue
Gl
broas loakerr secor sliiing
A
Provon ves a Asl
The acst wagge at
A atsitinaleed piitiog redaal uiith
lmage of isage image taage
of
SEMALLMMA
BLVAA-15
HLAVA-NeXT
LLAVA-NeXT
should clsee o' saeden rail soin muree in-- tuns placed
The mntel le!.
chhite Aackgmonnd being
Viosken
cleesap cstortal sliry modia for unti at
The suiel Air
iuskgrasad.
Qwen-VL
GPT-V
Queen
sxcer
mstal bui siih a shitr,, chrons-iile light againe aaithll coplasising
aest vaage should clonap a
hinget
sucolan destare utth boon graie
aplanical surfict
Gemini
Claude
Cemini
Claude
A seeden fer wath a kinget lMt Yier is made et light-islennl seedd, and deoostal
ued contatter or dox. Il Hill vade of stariios annn on stedar awtillic muteridl metmguler slape
sotol
Inoring
The imagy aprears secmm wicker oplan.
sige.
(a) Texture-I
Inteet Var
Input
Latent Var
Imput
Metal
Bag:
Dat
Ball:
Wood
Leaf:
Book:
Box:
GILL
sibor Aeteet kniik Mar strgs.
A alteer Anlass sitting en a nÀdee bestgrosed
persssaitt
aeodes 1S8 cress ooee.
creas de cieer.
beck
The
sist Ans senlal clagg me Aoy
SEED- LLMDA
LLAVA-15
LLAVA-NeNT
SEED LLaMA
LLAVA-L5
LLAVA-NeNT
The ncd image should The intiel should wade awatl and hase a slivr or grny wdke.
The by getsoatee shruld &e platogrpp rostly and vetsral, suiree tnem
The nnxt lvagee skould
2015
Qwvn-V1
CUT-IV
Qwwn-VI
CPT-IV
The stould be clecoap sparts ball, pousity recorr bell or a bolleyhal,, soth day and deunled of instue daigs.
natund mutel iylrnd meniniest sediool arove, with must uni a selk islind skine lackgmund.
antt ehould cleseap mooden her with introatee coriing as ldt The bas mafe of dark sood glosy fiust.
A snaades bank uith tetricat Stirst dedggn and Cyrilic äetering in bemin, granyy kott
Gemini
Claude
Gemini
Clande
apeears
inggr appears
sphoncsl
diid slanng mill rM No1
sate baice a shalkr wateral that is awal for plating verions spurte guns.
A aesios bax ssll siding lid.
pretsins euappes could pelnuiilly &e imay of ber padags, pe-- kxps cadl of inaa
(b) Texture-II
Figure 23: Examples of prompts and corresponding images description generated by MLLMS for tasks within the Texture theme. For brevity, some lengthy responses from MLLMs have been truncated, retaining only the key parts of the responses.
51

---

Published as a conference paper at COLM 2024
Latent Var
Input
Pretrained
Fine-tuned
Color-1
Brown:
Blue:
Green:
Color-11
Bag:
Box:
Ball:
BG-1
Cave:
Seafloor:
Gym:
BC-11
Gym
Cat:
Dogi
Bird:
Style-1
Box
Old:
3d:
Graffiti:
Sky 1-11-11
Graffiti
Bag:
Book:
Box:
Action-
Crys
Eat:
Fly:
Action-11 11
Cow:
Doge
Cal:
Texture-]
Apple
Denim:
Lace:
Metal:
Texture-11
Metal
Bag:
Book:
Apple:
Figure 24: Comparison between images generated by the pretrained-only SEED-LLaMA and its fine-tuned counterpart. The shorthand 'BG' represents 'background'.
52

---

Published as a conference paper at COLM 2024
User
We provide fes cnampies, each of aitick is inputroutput pair nviere the cutput is description of the umage associated with the ixpur. Doeni the exampies, tie taok profict the wext image description.
Blue:
BEONTE:
Orange
Black
Red:
Refore prodicting image, think step by step and onalyee relationship betwvee ieput and image output in each exampls.
Owen-VL
Sure, let's go thruugh anch crample and auyzee the relationship Ietwoen the toxt imput uni image output. libur Pichure I: This image is Blue picture of a laaf.
text imput is "blue" and the imagr output the text imput matcies
bine insf The relationshpp betsoenn tir test imput and image ontput the image output. 2. brown Pictzre This hmage is a broace picture of a lesf. the text input is "broaun" and the imagee autput is bresnn laf. The restionshp between the taxt tnpat and imuge oulput is that the tast input matches the color the smage output orange. Picture 3: This image an orangr picture of losf lxpul is "oraage" and the image oulput nu arauge ieaf. The vobiianchhp betsoon the text lapal and image oulput is that
color of
fest
lext tepat maiche the color of the image output. black insf The wistionship hemwer the test
Baack Picture 4. This image is back
pixuse of leff The uett buput u "Wark" and the image outgui the text input matches the coler of the lanng culput. exumples,
input and image ouljut
Thenfore, bose
predict
image description.
examples, we can make a prefiction for the wext inage description constdering the color of the lasf in eoch with background"
exsmple. etampie, with the ubiie background as the wex imoge deverittin.
mmgg descrppton
image
User
Rased on the unslysii, please deverile whhtt the nest image shouid by fook like give the request 'red"
Qwen-VT.
Givse rejuest sne have sen four ensmples, rhosse the image with the red color os the next image descriptins Threffrr, the next image description should be "s red loof with a white background".
prediction by considoring color of logf example Since
Figure 25: four-shot example of outputs generated by Qwen-VL using CoT prompting for task Color-l.
53

---

Published as a conference paper at COLM 2024
User
We prouide for cxampies each of alkich is au onput-ouiput jar usiere the cuiput is descripiioe of the umage the leput. Bown the examples, the prelict the wext inage description.
Monkeyy
P
Dog
Sheep:
Before predicting the nost maage, le's fiest Dunk step by step and anilyee the relationsipp betasoou the text ispul image cuiqut each extmple.
Qwen-VI.
Sure, I'll do my best to snalgze the reletionship betaoen text input and image cuiput in each compple Plaase the examples.
User
Based unstysii, desville whaa the image should lost givw riquest 'sheep.
Qwan-VL.
serry, any examples my analyois on, you planse provile inputs and image oulputs for each exnopie so I can anatyze the relatiossip betssonn the tost tuput and image outqui?
Figure 26: A four-shot example of outputs generated by Qwen-VL using CoT prompting for task Action-ll.
User
We providee fen esampios, of input--oulput yair auipul description of inuge assocusted the lsput. Bosed the examples, the tosk protict the wext image description.
Bools
Chair:
Ball
Leaf:
Cup
Before predicting the nest image, let's first think step by step and snalyez the relationship beraaen the text imput image culpul each example.
Qwen-VL
Sure, let's through anch craspple and musyee the relationsiip bettoon Bie lest imput and image outgut.
I. book: Pictire 1: This example about boolt noth a lnce cover The cover is described as being emblssso
with Roosers and larces The image cutput baok auth late cscar, which is consistont with the text input.
2. chair: Paclurt This exmmle is about chair made a lice coner. The case is deseriied leing ahile and made of face. The
oulpu! 4 chair walk lace oany, aolish is consetent with the lest lipul.
hull: Picture 3: This example is about ball with iace pottern. The ball described bring covere fanal lace patkern. The image oulpu a hall rnverd in firall lav pattern whinh comsident with the test inpul.
lest: Picture This cnampie about leff wtth lace pathrm, The leaf descrbbe
embossed with
deleate lace pattern. The imugr impot.
leaf with delcate ince pattern, sahich consistent with the
such eample, the imnge output is visual represonation of the text imput. The text teputs deserite objects or pathenss and image outuuts images watch the deseiled dijects patienss.
User
Bused on the ansiyots, please deserite what the naxt image should be look like gives the request 'cup.
Qwen- VL
Rased an the onalysis, the nost image should be cup with iace pattern. The cup could be any type of cup, offfee cup, tes cup, decorative cup, pattero could mny type of geometric lace. or abstenct lace. The color of the could be any color, such as white, olue, pink, green The lmage should clar
flonal
repesoiatiin cup laze patioon
Figure 27: A four-shot example of outputs generated by Qwen-VL using CoT prompting for task Texture-ll.
54

---

Published as a conference paper at COLM 2024
User
We prouide fro cnampie.. anch of athick is on inpottoutptt pair wiere the output is description of the imsge assccstll the ixput. Bosed the examples, the tosk the wext inage descriptio.
Orange
Black
Red:
Before preficting the nost image, let's first Hhink step by step and snalerr the relationship berween the toxt imput and imuge cuipust each example.
SHED-LLaMA
anilyzz
lest teputt smage output comppic. image output image autput is picture log. The exsmple of
crampie, has ixput
input euample "broon" image output pichure of a glli cumppi fourch cromple "erange" picture of logf The sixth topust example lopal "unange"
"onange" image enigusl pictare of
image outgut magge cutput pictune a iesf.
tapat
image outyut pietone a loof rightt exumple extoppe
image oulput pictoee of leaf. and image output is pichure leof. The tonth example las the test imput "broon" and image output picture of leff
From these exmppees, ixput color used to descrile the color of the laff in the cutput mmage.
that the input related to color the lesf in the image The
User
Rased anslysia, please deseriiv image gives request
SHED-LLaMA
bave generated tie nest imagge.
Figure 28: A four-shot example of outputs generated by SEED-LLaMA using Co'T prompting for task Color-1.
55

---

Published as a conference paper at COLM 2024
User
Wr pronide for coampies, sach of nibikk is inprttonuppit pair whrre the cueput description the associated with the ispur. Desel the exmppees, the took profict the next image description.
Cat:
Cow:
Lion:
Band:
Shoep:
Before prelicting imagee, step by step onalyte relationship imput and image cutput in each cxmmpls.
SCED-LLaMA
Sume. let's toke closer look the weiationsip betoown the impost and the in mich example. In first exmuple, the text shoses walking the banch The
the the waiking thr beach the
saiking the beach. Tie
the
"cow"
waiee, In thind wolking
the
"lann the umage
llon welling bid saniking the beach. shary uaiking
The inage
wailing catput bid beach
"bard"
beach. The
znauppe, lopul "chorp" and the image sheep From these examples, the can we that the text imput and the image ontpuu are welated in that they both deverite walting bring the boach.
the
bnach.
usage
the image different, they share the
User
Based anslyais, deserile salat the image should te loot: request 'sheep.
STED-LLaMA
lane generated inage.
Figure 29; A four-shot example of outputs generated by SEED-LLaMA using CoT prompting for task Background-II.
56

---

Published as a conference paper at COLM 2024
User
Wr posvide frn coampies of iryottouuppu yair wire curput description of the image associated with the ispur. Besei the examples the tesk prefict the wext inaaee description.
Wireframe:
Peturistic
Icon:
Oi:
Pivel:
Before prodicting the nest image, Ie's first Ahink step by step and onalyte the relationship benooee the text ieput and imege eutput ensmpls.
SEED-LLaMA
Yes,
topet image output compii In exuuple, impat "sitoffrme" tmage output imuge
umage example. input image output futuristic This image cutput
represotation image rp-- image cutput filh exmuple, image piselated representation image sutput a futurritic leaf. input "pixe!" legf.
exampie. resentation of loaf inputt
comple
age.
image represeniation
aged.
imput "plzst" of iesf
image outpat pisslated laaf. Ths
ispust "futuristic" sanus tmage futuristic represmnaaton of image pixelaind rxpessutation and image oulpul pruzlnied leg.. mains rightii cummle, the text imput "uireffome"" and image output is winfrrme of loaf This meens image soirefamne represenation insage sntyui
essmple,
example
lesf image repesentatine of fraf input
ninth exemple,
and
monns tenth exmmple, the text impaf is "pize!" and the image output is pinelated laff. This monns that the image pizelated representation of lost
with age.
agod.
summary, relationship between the input and the image used describe the type of repessuation of the leff in the imuge outpui.
each example the text
User
Rased on the andlysis, please deserlle what the next image should be look like glves the request "pizal:
STED-ILaMA
Here the next image hesed on requcall 'pintt
Figure 30: A four-shot example of outputs generated by SEED-LLaMA using CoT prompting for task Style-I.
57


Reviews:
[{"review_id": "ibqlWn9TSv", "reviewer": "Reviewer_6kcQ", "summary": "Quality:\nThe work presented is of high quality, demonstrating rigorous experimental design and thoughtful analysis of results. The introduction of the CoBSAT benchmark to systematically assess T2I-ICL capabilities across a range of MLLMs is a commendable effort. The methodology section is detailed and includes a clear explanation of the evaluation pipeline, which adds to the reliability of the findings.\n\nClarity:\nThe paper is well-structured and written with a clear narrative that guides the reader through the introduction of the problem, the development of the CoBSAT benchmark, experimentation, and conclusions. Each section builds logically on the last, with sufficient detail provided for replication of the experiments. Figures and examples are used effectively to illustrate key points, enhancing the overall readability and understanding.\n\nOriginality:\nThis work addresses a gap in the research on multimodal in-context learning by focusing on text-to-image tasks, which is less explored compared to image-to-text configurations. The creation of a dedicated benchmark for T2I-ICL is original and provides valuable resources to the research community. The exploration of fine-tuning and Chain-of-Thought prompting to enhance MLLM performance in T2I-ICL also contributes new insights into effective strategies for multimodal learning.\n\nSignificance:\nThe significance of this work lies in its potential to advance the field of multimodal learning. By identifying challenges and proposing methods to enhance T2I-ICL, the paper sets the groundwork for further developments in this area. The findings could influence future research directions and have practical implications for applications in product design, personalized content creation, and more. The availability of the CoBSAT dataset and the detailed analysis of MLLM performance also make significant contributions to ongoing discussions and research into multimodal AI systems.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are encouraged that you found our paper to be of high quality, well-structured, well-motivated, significant, and well-written. \n\n--- \n\n> Q: Generalizability and Practical Relevance.\n\nWe intentionally designed our dataset for a simplified setting because:\n\n1. It was unclear if existing models could perform T2I-ICL.\n2. Practical applications are hard to evaluate; for example, no method exists to automatically assess generated anime character designs.\n3. Current MLLMs generate unsatisfactory results even in simplified settings (Fig4a,b).\n\n> Q: Diversity, representativeness and stereotypical biases of dataset.\n\nTo ensure the CoBSAT dataset's diversity and representativeness, we selected key image attributes: color, background, style, action, and texture, as supported by related works ([[1]](https://arxiv.org/abs/2208.12242), Figure 5 in [[2]](https://arxiv.org/abs/2310.01506)). \n\nWhile our dataset currently avoids stereotypical biases by not involving demographic groups, future expansions to include more attributes could introduce such challenges, which we acknowledge but are beyond our present scope.\n\n> Q: Other metrics.\n\nEvaluating images is inherently challenging. Initially, we only used CLIP and LLaVA for validating the generated images, and we validated the performance of CLIP and LLaVA using two methods:\n\n1. LLaVA correctly identified all attributes and objects in our dataset with 100% accuracy.\n2. In Sec. E, we showed a strong correlation between CLIP and LLaVA evaluations, validating LLaVA's consistency with CLIP and its systematic, rather than arbitrary, assessments.\n\nFollowing your suggestion, we included Qwen-VL and Gemini in our evaluation. We manually labeled 100 SEED-LLaMA-generated images. LLaVA's evaluations showed strong alignment with human assessments, and Gemini performed the best.\n\n| | CLIP | LLaVA | Qwen-VL | Gemini | \n|---|---|---|---|---|\n| Concordance Rate to Human Evaluation | .85 | .89 | .78 | **.92** |\n\nWe conducted a statistical study with 20,000 images to compare automatic metrics, focusing on their relation to Gemini's results, which align closely with human evaluations. The [figure](https://hackmd.io/_uploads/BJJCTxw4C.jpg) shows a strong correlation between LLaVA's and Gemini's accuracy estimates, reinforcing our confidence in LLaVA as a reliable, open-source evaluation model.\n\n> Q: Long appendices.\n\nWill update.\n\n---\n\n**Final Note**: Thanks for the suggestions. We will incorporate all discussions above in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Significant Contribution to Multimodal Learning: The paper addresses a notably underexplored area in multimodal learning by focusing on Text-to-Image In-Context Learning (T2I-ICL). This study not only extends the current understanding of In-Context Learning (ICL) but also adapts it to a more challenging and less studied multimodal context, which is significant for the field.\n\n2. Introduction of CoBSAT Benchmark: The creation of the CoBSAT benchmark is a major contribution of this work. It systematically assesses the T2I-ICL capabilities of Multimodal Large Language Models (MLLMs) across a variety of tasks. This benchmark will likely become a valuable resource for future research in the area.", "reasons_to_reject": "1. Generalizability and Practical Relevance: The scenarios tested appear highly specific and might not generalize well to other text-to-image contexts. The practical applications described (e.g., product design, personalized content creation) are mentioned only briefly without exploring the operational challenges or effectiveness in real-world scenarios.\n\n2. In summary, while the paper addresses an innovative and relevant topic, these concerns suggest it may not yet meet the high standards required for publication at COLM. Addressing these issues could potentially make the paper suitable for publication.\n\n3. Too much content has been placed in the appendices.", "questions_to_authors": "1. How do you ensure the diversity and representativeness of the CoBSAT dataset? Given the challenges of bias in AI, what measures were taken to ensure that the dataset does not reinforce stereotypical biases, particularly in the context of generating images from text?\n\n2. Beyond the use of CLIP and LLaVA for validating the generated images, what other quantitative metrics were considered or used to assess the performance of the MLLMs on the CoBSAT benchmark? Please provide details on the statistical significance of the results, if applicable.", "extra_scores": {}}, {"review_id": "Pr9UwnhWTc", "reviewer": "Reviewer_u6Uf", "summary": "This paper proposes a new benchmark to evaluate MLLM's capability to perform in-context learning for text-to-image generation tasks，specifically it supports two types of visual inference tasks: object-inference and attribute-inference. The authors collect the images of the dataset from both internet resource and Generative model with a carefully designed object list and attribute lists to examine the MLLM's capability to properly generate the images or image descriptions that align with these proposed lists;\nFinally, it benchmark both text-to-image generation model and closed source MLLM that can only generate text (e.g GPT4V) and identify some interesting insights: (1) The visual prompt post a significant challenge for in-context learning of the current MLLM; (2) The Image generation task is a much harder task than text-generation even when the required attributes are relatively constrained; \nThe author than proposed two simple solutions to address these two challenges including: chaini-of-thoughts & finetuning, both lead to obvious improvements.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging our contribution in i) a new benchmark for T2I-ICL that was underexplored, ii) identifying the challenges, and iii) suggesting effective methods. Your concerns are addressed below.\n\n---\n\n> Q: The qualities of benchmark images are questionable.\n\nWe avoided this issue through meticulous data collection. We manually scrutinized and selected each image in our dataset, regardless of whether they were sourced from the web or produced by generative models. We ensured that every image aligned with the intended object and attribute specifications.  Throughout the image collection process, we ensured all images in our dataset were of sufficient clarity to allow state-of-the-art MLLMs to identify their objects and attributes with 100% accuracy.\n\n> Q: The paper mainly leverages LLAVA to evaluate the performance. However, given that LLAVA is a relatively worse model compared to some of the closed models, the evaluation results of LLAVA over these strong MLLMs are less reliable.\n\nWe appreciate the reviewer's insightful question. We believe in LLaVA's reliability for several reasons: \n\n1. Performing T2I-ICL and visual question answering (VQA) requires different capabilities of MLLMs—T2I-ICL necessitates a deep understanding of the interplay between images, whereas VQA does not. Our evaluation primarily requires VQA capability, and LLaVA is well-known for excelling in visual question answering.\n2. LLaVA achieves 100% accuracy in identifying the attributes and objects within our dataset.\n\nWe also incorporated Gemini as an additional evaluation model. Our comparative analysis visualized in this [figure](https://hackmd.io/_uploads/r1RD30LV0.jpg) shows that the accuracy estimates from LLaVA align closely with those from Gemini. This correlation reinforces our confidence in LLaVA as a reliable and accessible open-source evaluation alternative to these closed-source models.\n\nFurthermore, we manually labeled 100 images generated by SEED-LLaMA through T2I-ICL, selecting 10 random images from each task to serve as a baseline. Our evaluations using the LLaVA show strong alignment with human assessments, achieving a concordance rate of 89%.\n\n---\n\n**Final Note**: Thank you again for your encouraging feedback and great questions. If you have any remaining questions, please do not hesitate to let us know. If our responses have resolved your concerns, we kindly request you consider increasing your score and championing our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A new benchmark that aims to evaluate MLLM's capability to perform in-context learning for text-to-image tasks, which was previously under-explored. \n2. Through this benchmark, the author identifies two major challenge of the current MLLM: (1) Lack of capability to understand the visual prompt in the in-context learning examples (2)limited capability to generate image compared to the text generation capability. \n3. Comprehensively compare multiple models on this new benchmark and identify the challenges the current methods encounter. It also proposes two effective method: fine-tuning and chain-of-thoughts to improve the model's capability of in-context learning.", "reasons_to_reject": "1. Some of the benchmark images come from generative models (e.g., Dalle-E). The qualities of these generated images are questionable. \n2. The paper mainly leverage LAVVA to evaluate the performance of the generated image or the generated caption of the image. However, given that LAVVA is a relatively worse model compared to some of the closed model (GPT-4V and Gemini), the evaluation results of LAVVA over these strong MLLM is less reliable.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A9ILquqoBv", "reviewer": "Reviewer_Nirr", "summary": "The paper proposes a new task, T2I-ICL where the goal is to generate an image given a demonstration. More precisely, the authors propose two tasks. In the first, the images define the object (e.g. a car, *object inference*) to generate, and in the second, they define the attribute (e.g. red, *attribute inference*) – and vice-versa for the text.\n\nA dataset (CoBSAT) is proposed to evaluate such models, and the authors evaluate different types of models, namely (1) MLLMs than can generate images like Emu/Emu2, SEED-LLaMA, and GILL; (2) LLMs using only the textual description of images. The overall setting is clearly described, and the motivation – to analyze the limits of currents models in this simple setting, is interesting.\n\nThe evaluation of models is done automatically by using VQA systems, and show that object-inference tasks are more difficult for current state-of-the-art models.\n\nThe authors analyze why current MLLMs fail, i.e. infering attributes/objects from images and generating images are two difficult tasks and propose ways to improve the current models (by finetuning or using chain-of-thought).\n\nOverall, the paper is interesting and well-constructed and demonstrates the limits of current MLLMs in a simple experimental setting.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s encouraging feedback, particularly for recognizing our contributions in identifying a new task for MLLMs and providing benchmarks, as well as our analysis of the current limitations of these models. Our response is detailed below.\n\n---\n\n> Q: The evaluation protocol is questionable (relying on a VQA system)\n\nTo address the reviewer's concern, we manually labeled 100 images generated by SEED-LLaMA through T2I-ICL, selecting 10 random images from each task. It is important to note that some images were of suboptimal quality, presenting ambiguities that could be interpreted both as correct or incorrect. Despite these difficulties, our evaluations using the VQA system show strong alignment with human assessments, achieving a concordance rate of 89% (computed by the ratio of agreement between the two methods).\n\n> Q: The dataset might be too easy in the near future (as demonstrated when finetuning models on the data).\n\nIn Sec. F.1.6, we have included a challenging version of our dataset, which incorporates misleading information in the prompts. This requires MLLMs to identify the correct information while disregarding the misleading content. Notably, almost all existing MLLMs struggle to perform well on this dataset. Based on your comment, we plan to highlight this further in the main body of our final version.\n\nAnother way to increase the difficulty of our dataset is to incorporate more fine-grained attributes (e.g., dark red and light red in color-themed tasks) and more challenging themes (e.g., counting). Currently, these elements are excluded as our preliminary experiments indicate that existing MLLMs do not perform well on these more detailed tasks. We also identified this as a promising future research direction, which we have detailed in Sec. 8.\n\n--- \n\n**Final Notes**: We are delighted to hear that you appreciate our paper. We will incorporate these discussions into the revised version. Thank you once again for your insightful and encouraging feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- a new task with a new dataset\n- the analysis of why current MLLMs fail", "reasons_to_reject": "- the evaluation protocol is questionable (relying on a VQA system)\n- the dataset might be too easy in the near future (as demonstrated when finetuning models on the data)", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ueSYQPQFY", "reviewer": "Reviewer_8Cxt", "summary": "This paper identifies an important yet underexplored problem - T2I ICL. It introduces a benchmark for evaluating T2I-ICL ability of MLLMs. The authors introduce two approaches to measure the performance of MLLMs. For models with image generation ability, they evaluate the generated image under an I2T retrieval setting. For models that are only able to generate texts, they replace the image content with text descriptions.Finally, the authors identify the challenges of MLLMs performing T2I-ICL and provide two approaches to improve the performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q: Insufficient MLLMs.\n\nWe included nine MLLMs, detailing three (Emu2, LLaVA-1.5, and LLaVA-NeXT) in the appendix due to resource limitations. We then added Claude, bringing our total to ten MLLMs. The updated results are shown in [Figure 4(c,d)](https://hackmd.io/_uploads/H1wJmGvNC.png), though 8-shot results for Claude are not yet available.\n\n> Q: Experiments on fine-tuning.\n\nIn response, we implemented a new leave-one-out setting for our experiments. We fine-tuned the model on all themed tasks except one (e.g., Texture-I and Texture-II) and tested on the excluded task to assess generalization.\n\nBelow are the preliminary 2-shot results for SEED-LLaMA. Initial results indicate that fine-tuning various T2I-ICL tasks enhances the models' ability to perform T2I-ICL on unseen tasks, thus ICL remains necessary on unseen tasks for efficiency.\n\n| Fine-tuned SEED-LLaMA | Style-I  | Style-II | Texture-I | Texture-II | \n| -------- | -------- | -------- | -------- | -------- |\n| ✗ | .088 | .164 | .108 | .080 | \n| ✓ | **.208** | **.240** | **.200** | **.104** | \n\n> Q: Evaluation metric.\n\nTo investigate, we manually reviewed 100 SEED-LLaMA-generated images, noting some ambiguities due to suboptimal quality. Despite this, our VQA system evaluations align strongly with human assessments, achieving an 89% concordance rate.\n\n> Q: Underperformance of image-generation MLLMs. \n\nThe underperformance of MLLMs with image generation capabilities compared to those without is due to:\n\n* Image-generating MLLMs are newer and less mature, with smaller training data and model sizes compared to established models like GPT-4V.\n\n* They are primarily trained to generate images, which can hinder tasks like image description generation. For instance, SEED-LLaMA fails to generate text in our experiments.\n\nThe performance gap is due to training disparities. With extensive training, image-generating MLLMs should match their text-only counterparts.\n\n> Q: Solution to two challenges. \n\nThe main limitation is inadequate training data. We need large datasets that emphasize image interplay, which current VQA and image captioning datasets (used by Emu [[1]](https://arxiv.org/abs/2307.05222)) lack. Suitable sources include anime characters and product designs, where image interaction is crucial for context.\n\n> Q: Other.\n\nFixed. \n\n--- \n\n**Final Note**: We sincerely thank you for your encouraging feedback and constructive suggestions which will significantly enhance our final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well motivated. It identifies an important yet underexplored problem T2I-ICL.\n\n- The settings of all experiments are well controlled and introduced, providing much soundness for this paper.\n\n- The analysis in Section 5 seems reasonable and insightful.\n\n- The paper is overall well written and easy to follow.", "reasons_to_reject": "- As a benchmark, it would be better to test more models instead of just 6.\n\n- The evaluation process is formulated as an Image-to-Text retrieval task via prompts (LLaVA) or embedding similarity (CLIP). But what if the generated image does not belong to any class in the predefined set? In this case, LLaVA may fail to follow instructions and thus fail to provide a valid evaluation result. Hence, it is a major concern that the authors do not discuss such cases. \n\n- Texts in Figure 4 are almost unreadable.\n\n- The first technique for enhancing T2I-ICL abilities is not surprising. ICL was an emergent ability of (M)LLMs to perform task learning without finetuning them. If we can finetune a model, then ICL would be needless.", "questions_to_authors": "- In Figure 4(c) and 4(d), what is the potential reason of models without image generation \ncapability outperforming the models with such capability? On one hand, “capability tax” would be an interesting point, which means that a model capable of doing both may underperform a model that has a single capability. On the other hand, is it because the label itself is in textual modality, so performing a T2T evaluation would be an advantage for text-output-only models?\n\n- To support your argument that “finetuning can enhance T2I-ICL ability”, it’s better to construct a test set which has distinct task types compared to the training set. However, the argument is still not strong and this finding still lacks excitement.\n- The two challenges are not well solved by the two approaches the authors propose. While solving them is beyond the scope of this paper, it would be more insightful to discuss the potential solutions to the two challenges.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The work examines text-to-image in-context learning and introduces a benchmark to assess this capability of MLLM. All reviewers agree to accept the submission. It's a nice contribution to have a new benchmark dataset for the problem. As the release of gpt-4o, there will be more papers that are interested in this capability.
Decision: Accept