Exploring Diverse In-Context Configurations for Image Captioning
Authors: Xu Yang
Yongliang Wu
Mingzhuo Yang
Haokun Chen
Xin Geng
Venue: NeurIPS 2023 poster (2023)
Keywords: Image Caption; Few-shot Prompt; Vision Language Model;
TL;DR: Using Open-Flamingo to explore the effects of diverse in-context configurations for image captioning.

Abstract: After discovering that Language Models (LMs) can be good in-context few-shot learners, numerous strategies have been proposed to optimize in-context sequence configurations. Recently, researchers in Vision-Language (VL) domains also develop their few-shot learners, while they only use the simplest way, \ie, randomly sampling, to configure in-context image-text pairs. In order to explore the effects of varying configurations on VL in-context learning, we devised four strategies for image selection and four for caption assignment to configure in-context image-text pairs for image captioning. Here Image Captioning is used as the case study since it can be seen as the visually-conditioned LM. Our comprehensive experiments yield two counter-intuitive but valuable insights, highlighting the distinct characteristics of VL in-context learning due to multi-modal synergy, as compared to the NLP case. Furthermore, in our exploration of optimal combination strategies, we observed an average performance enhancement of 20.9 in CIDEr scores compared to the baseline. The code is given in https://github.com/yongliang-wu/ExploreCfg.

Exploring Diverse In-Context Configurations for Image Captioning
Xu Yang'; Yongliang Wu', Mingzhuo Yang", Haokun Chen', Xin Geng'
School of Computer Science & Engineering, Key Lab of New Generation Artificial Intelligence
Technology & Its Interdisciplinary Applications (Ministry of Education), Southeast University "The Chinese University of HongKong, Shenzhen
wwwyang-palmmen edu. cn, yongli angwu@seu. eeeeouueeeeeeu...eeeee edu. cn, 118020068c1inz cuhk, edu, en, chenhaokun@seu. edu.cn, cn, xgeng@seu, edu, cn
Abstract
After discovering that Language Models (LMs) can good in-context few-shot learners, numerous strategies have been proposed to optimize in-context sequence configurations. Recently, researchers in Vision-L.nguage (VI.) domains also de- velop their few-shot learners, while they only use the simplest way, i.e. rrandolly sampling. to configure in in-context image-text pairs. In order to explore the effects of varying configurations image selection and four for caption assignment to configure in-context image-text pains for image captioning. Here Image Captioning is used the case study since it can be seen as the conally-coniiiooned LM. Our comprehensive experiments yield counter-intuitive but valuable insights, highlighting the distinct characteristics of VL in-context learning due to multi-modal synergy, as compared to the NLP case. Furthermore, in exploration of optimal combination strategies, we observed average performance enhancement of 20.9 in CIDEr scores compared to the base- line. The code given in httttttp:/gggiaaie wow.:uuwww.ooooo/giaanngggo. mu/ExploreCig
VL in-context learning, we devised four strategies for
Introduction
In contemporary times, the Language Model (LM) has emerged a pivotal player in the field of Natural Language Processing (NLP). It accomplishes this by unifying range of diverse NLP tasks into a shared prompt paradigm conditional language modeling on a substantial volume of con-annttteed data collected from the web, can reformulate various downstream tasks into fitting textual prompts. These prompts contain slots, the word probabilities of which are calculated using the pre-trained L.M. This process obviates the necessity for gradient updates to the parameters of L.M, thus mitigating the challenges associated with additional data collection and fine-tuning. To further enhance the effectiveness of this paradigm, the few-shot prompt (or in-context learning) is proposed where a few examples are provided as additional contexts to guide the LM generate the desired result.
To elaborate, LM, unsupervised trained via
Witnessing the success of the prompt paradigm in the NLP field, there has been a concerted effort by researchers to replicate its function in the Vision-Language Model (VLM). To facilitate this, Flamingo (6] is proposed to align well-trained large-scale vision and language models through some trainable cross-modal adapters. Consequently, the resultant model is capable of addressing Vision-Language (VI.) tasks by processing a prompt sequence, which includes several interleaved image and text examples for in-context learning. Since the primary objective of Flamingo is to build a VLM for the few-shot prompt, they only apply a straightforward strategy to configure the in-context
CCorresponiing author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Which country Barack Obama from? America.
Q: Which country Mohandas Gandhi from?
a: Which country Yao Ming from?
"China"
(a)
LM
Q What do dogs like to eal? A Bone
Q What do cats like to eet?
Q Which country is Yan Ming from?
"Banana" X
(b)
A: Fish.
A:
Foe de pat.
The ri hise thema matte
A cyclist riding in the bike
of city street. [CIDE: 164.6]
VLM
Neople ng street
strest
Poozle walking down and strest Caption)
A man riding a bike in street streat.
[CIDEr 61.4)
(d)
Rad Eaploo)
Brough Reed BSood nantion
desphars
frige and
some
A bicycler riding the bike lane next traffic.
|Sood Captoot
[CIDEr 126.8
VLM
Some Aephants are [Bad Capt on)
Michen Sire [ol Captoo))
A g of people riding on cit street.
[CIDér 72.9)
(f)
Figure 1: The distinction between LM and VLMS as few-shot learners. LM generally excel with examples akin to the test case (blue blocks in (a)). In contrast, for VLMs, the performance is not strictly correlated with image similarity but heavily relies on the caption quality. For instance, when low-quality captions are used, similar images (d) lead to worse performance than dissimilar ones (f) since VLMs may build short-cut by reusing in-context captions without seeing the given images.
sequence by randomly sampling a few image-text pairs. Nevertheless, a plethora of studies within the NLP field 77] have demonstrated that diverse in-context configurations lead to dramatic effects on few-shot performance, e.g.. the selection or ordering of in-context samples [8] 9 [0], while only a limited number of studies systematically explore such effects in the VL case.
To narrow this gap, we explore the effects of various in-context configurations on the performance of few-shot VL tasks. Among various VL. tasks, Image Captioning (IC) aims at generating a text conditioned on the source image, and thus can be considered as the risually-coddition multitude of NLP tasks can be recast as LM tasks, IC performs similar function [II 12 13 which motivates our decision to select IC as the subject our case study. However, unlike in NLP, where only single-moda texts are considered in the in-context configuration, in IC, the synergy between multiple modalities significantly influences the performance. For instance, our experiments revealed that selecting images similar to the test image for the in-context sequence does not always lead to good performance, a result closely tied to the quality of the associated captions. Figure shows the comparison between LM and VLM as the few-shot learners.
LM. Just as a
Consequently, we design diverse ways select images as the in-context images. After selection, the captions of different qualities are assigned to these images for constructing the multi-modal in-context sequence. By combining diverse image selection and caption assignment techniques, we undertake a comprehensive exploration of the effects of multi-modal mutual synergy on VL in-context captioning. We implement all experiments using the prevalent captioning dataset, MSCOCO 14,, employing its training set as the database for image selection.
To select images, we use 4 different ways, which are Random Sampling, Similarity- based Image- Image-Caption Retrieval, and Diversity-based Image- Image Re-
Image Retrieval, Similarity-basee trieval. Following image selection, various types of captions produced by 4 different strategies are assigned, which are Ground- -ruth Captions, Model-Generater Captions, Iteratively Prompting, and GeneraGee Captions as Anchors.
Through extensive evaluation of various image selection and caption assignment strategies, we un- cover two counter-intuitive yet valuable insights. (1) Caption quality is determined by descriptiveness and language patterns, but their influence on in-context captioning performance is unequal. When captions adequately describe salient image objects, simpler language patterns may yield better results. (2) The efficacy of similar images depends on the quality of the paired captions. Excessive similarity might cause VLM to create a short-cut inference [T5] from in-context captions, potentially misleading the model with low-quality captions. Beyond these findings, we introduce a practical in-context captioning strategy, Iterative Prompting, for cases with limited or no Ground- -Truhh Captions. Further- more, when Ground-Trus Captions are available, we recommend using anchors to identify which Ground-Truth Caption is a more suitable in-context caption. Experimental results indicate that even when utilizing low-quality model-generated captions, there is an average CIDEr improvement of 7.3. Moreover, optimal conditions, the average enhancement reaches up to 20.9 points compared to the random sampling baseline.
Captions as

---

Since the Flamingo code is not publicly available, our experiments mainly utilize the unofficial Bhen-Flamingo [6 It's worth noting that the performance of Open-Flamings not on par with the official Flamingo due to its training on less data.
Related Work
Prompting Language Model (LM). The research paradigms of NLP have encountered two sea changes in the past few years [3] The first one the LM that is pre-trained by predicting the next word conditioned on observed words and can be fine-tuned for solving various downstream tasks, including GPT [77, BERT [ and BART [8]. The second sea change the emergence of the prompt paradigm, which was introduced with GPT-3 [2]. Within this paradigm, pre-trained LM does not require fine-tuning to solve downstream tasks; instead, tasks are reformulated into appropriate prompts with empty slots to be filled. Subsequently, more advanced prompt- based techniques have
been proposed, including prompt-tuning 19 20 21 and Shain-of-Thhuuhhe [22
25
Prompting Language Model (VLM). In contrast to NLP, the Vision-Language (VL) domain has made significant strides in the first sea change, as evident by the development of various VL- BERT models. These models leverage large volumes of web-collected image-caption pairs to learn
However, the prompt paradigm, despite revolutionizing NLP studies, only appears when a certain scale of the model is reached [34). This scale prerequisite poses further challenges to the development of a VLM with prompt and in-context learning ability.
VL-generalizable embeddings [26] 27
29 30;
32: 33
To mitigate training burdens, instead of updating all the parameters of VLM 335 36), some VLMs (35 36
37 freeze well-trained Language Models and only train a smaller network, referred as adapters [38] 39 40), align the pre-trained vision and language models. Inspired by these models, both Frozen [41] and Flamingo [6] evolve into multi-modal few-shot learners by training
vision and cross-modal adapters, respectively. Given its superior in-context learning ability, use Flamingo to explore the effects of various in-context configurations [16].
There are also models that address VL tasks through in-context learning. For instance, PICa (II utilizes captions as mediators to construct an in-context text for solving Visual Question Answering
(VQA) tasks, while this may lose the mutual synergy in the representation space of different modalities. The models proposed in [42] and (36] both fine-tune VLMs for specific VQA tasks, but lack the generalized few-shot prompt ability for other VL tasks. UNIFIED-IO 4 demonsttatss unified approach to a myriad of tasks, from classical computer vision to natural language processing, without task-specific fine-tuning. And ProGrad 44 introduces technique to prevent prompt tuning from forgetting pre-trained vision-language models' general knowledge by only updating aligned prompts, outperforming other methods in various few-shot learning scenarios.
Exploring In-Context Configurations in NLP. Upon observing that pre-trained LMs are good few- shot learners, researchers also discover that diverse in-context configurations have dramatic effeets on performance 3 This observation sparks numerous studies aimed at determining the optimal in-context configurations such the format of the in-context examples 45 21 5) the selection of
these examples g 9 46 47). and even the order in which these examples are presented [48] 49 [0]
However, these studies predominantly conducted within the NLP field and fail consider the unique characteristics and complexity of multi-modal data. In order to address this gap, we propose a series of strategies to explore the effects of various multi-modal in-context sequence configurations.
Image Caption. Image Captioning (IC) [50] aims at correctly verbalizing one image using descriptive languages, which can be solved through both retrieve (51) and generation [50). the former retrieves complete sentences from a corpus, while the latter generates words sequentially. Researchers have recently combined these approaches by first retrieving image-caption pairs and inputting them into generation models [52] (3). This process resembles in-context captioning, which also retrieves image-caption pairs to help captioning. However, in contrast to them [52) 53). work introduces novel image and caption selection strategies to study in-context captioning and these methods can also enhance traditional Strrival-geeneattion methods.
Prrojcct: http::://aaionnai//lgggg
3

---

LAT
Ensuder
Gryan Gopp
Farmutte
Yokan trowke
fop Samples
5G
yoar
Top
Ensoder
Samw Grash
Extartor
a niling
@ SIRR-CLIP
(b) SUR TAG
strect
object
[object, relation, attribution, dlass)
Relation
SIR-TAG
attrbuttion
Disccer
Top 4 Sampko
A new the shoss.
[object, relation, attribution, class)
Somne
DUR
SICRCLIP
Figure 2: Image selection strategies: (a) SURR-CLI,, (b) SIRR-TAG, (c) DIRR-T,, (d) SICR-CLIP.
3
Configuring In-Context Sequences
The in-context captioning can be treated as a vision-language conditioned text generation task. Given the multi- multi-modal in in-context sequence S = {(11, C1); (12, C2):.... (In, C.); } that contains n-shot image-caption pairs (I, C) and one test image f, we hope to generate the corresponding image caption C ( úit } in auto-regressive manner. Here the f-th word in is sampled from the following word distribution:
P( 01=-1),
where the probability P(-) calculated by pre-trained Vision-Language Model (VLM) (e.g..
Flamingo [6] Otter 54
field have shown that the performance of in-context learning varies significantly with different in-context configurations. We explore these effects the case of tasks, Image Captioning (IC). Unlike pure task, IC is dual-modal task, which makes it more complex than NI.P tasks. Specifically, the mutual synergy of image-caption examples must be considered in in-context learning, opposed considering the images the captions independently. We next respectively introduce the image selection (cf. 3.1) and caption assignment (cf. 3.2) strategies used to configure in-context image-caption pairs.
Various studies in NLP
8
45
48
46
3.1 Selecting Images
Random Selection (RS). Given a set D = {(I,, CI), Cm)} with M image-caption pairs, we randomly sample n images as {I,,..., IN) in S.
Similarity-based Image-Image Retrieval (SIRR) Certain NLP studies suggest that performance be enhanced by retrieving examples similar the test case g 46 47). Following this approach, retrieve images with the highest similarity scores to the test image I from D. We employ two methods to compute these similarities: 1) SHR--CIPP (Figure (a)): Using the vision encoder of
CLIP (35). we extract image embeddings to determine similarities. 2) SHR-TAG (Figure (b)): We utilize some scene graph extractors to derive tags for each image. Specifically, we employ Vinvl (55) extract objects and their attributes, and IETrans (56) to determine the relations present within the image. Following this extraction, we conduct an AND operation to compute the similarity between tags.
Similarity- -based Image-Caption Retrieval (SICR-CLIP (Figure 2 (d)). Taking advantage of the cross-modal retrieval capability of CLIP 35 we use its vision and language encoders embed images and captions into a shared space for computing similarities. Given İ, calculate its cross-modal embedding similarities with (CI,..., Cm} E D, and select the images whose captions have top-n similarities with í. Note that we use different kinds of captions mediators; the methods to generate these captions will be detailed in Section 3.2 These captions are also used as C1,... ...... CN E S.

---

Diversity-based Image-Image Retrieval (DIIR). For single test sample, beside using similar in- context examples, some NLP studies find that diversity also crucial [57]. Consequently, we retrieve diverse set of images from 'D to configure S. Specifically, we employ the following strategies: 1) We extract discrete tags from the aforementioned VI. and IETrans. Then we randomly divide the tags into N clusters and apply SHIR-TAG to retrieve the most similar image from each cluster. 2) DUR-TT (Figure[(c)): To conveniently control the number of shots, we incorporate class tags generated by the IETrans model, extending the basis of SIIR-Tage to four categories: object, class, attribute, and relation. We then employ SIIRTTAG to identify the top-k similar images within each category, allowing us to create shoshot images in S. Note that both DIIR methods take into account a certain level of similarity during retrieval, rather than selecting entirely distinct images.
DIR-TR:
3.2 Assigning Captions
After selecting images, we need to assign one caption to each image to construct in-context image- caption pairs. To explore the effects of multi-modal mutual synergy, we use captions with diverse qualities as introduced in the following.
Ground- Truuh Captions (GTC). In the MSCOCO dataset each image has 5 GTCs and we simply use the first one in S.
ModeleGenerated Captions (MGC), Compared with GTC, MGC has lower quality due to two disadvantages. Firstly, MGC uses poorer language, e.g., simple words or rigid sentence patterns. Secondly, MGC has less descriptiveness that it may mis-verbalize or miss certain salient vision patterns of an image. However, performance compared with GTC in in-context captioning and surprisingly, we find that sometimes simple words or rigid sentence patterns even help generate good captions.
will see these two disadvantages do not equally cause worse
Here we apply two different models to generate the captions with diverse qualities. MGC-TF X: a Transformer is trained from scratch by using VL features and X denotes the CIDEr score [58] on the test set. To get captions of different qualities, we use the checkpoints from different training epochs and totally generate three kinds of MGCs. 1) MGC- AGCC-FFG @66 contains grammar mistakes while can describe the most salient objects. 2) MGC-TF@ 88 can use relatively correct grammar to describe the salient objects. 3) MGC-TF@ 135 is generated by well-trained Transformer, i.e., the loss converges. MGC-VLM( @ X: Another way to get MGCs is to use VLM in a N-shot manner. To achieve this, for each image I D, treat I as the test image and then use Eq. (1) generate corresponding caption C conditioned on S constructed by only N image-caption pairs. As a result, two kinds of captions are got which are MGC- (LM() @63 and MGC- (GC-VLLM32))881
Both two ways can construct the set D with M image-caption pairs. Then for novel test image i, we can select some image-caption pairs from D by some above mentioned approaches, e.g., using RS SIIR-CLIP to get images and assigning the MGCs to the image, to configure S for generating a new caption. Compared with MGC-TE MGC- VLM is more practical as it addresses scenarios where only a handful even no human- captions are available, which means we do not have enough data to train a Transformer from scratch.
Iteratively Prompting (IP). For MGC-VLM introduced before, one natural extension is IP. To achieve this, in the first iteration, we generate a caption for each image D by MGC-VLM. In the subsequent iteration, these generated captions are paired with the selected images to prompt VLM to generate enhanced captions. This process can be repeated across multiple iterations, thereby iteratively prompting the VLM.
Gener-Gen Captions as Anchors (MGCA), A MGC can serve not only as an in-context caption but also as an anchor for selecting a suitable caption from the five GTCs. As MGCs typically verbalize salient visual patterns in an image but may miss finer details, using them as anchors can lead to the selection of GTCs that highlight these salient patterns. Furthermore, the selected GTC may supplement interesting details about these patterns, potentially assisting VLM to generate superior captions during in-context learning. In the MGCA implementation, for each selected image, we measure the similarity between the MGC and five GTCs using CIDEr and select the GTC with the highest CIDEr score.
5

---

0001
(a3R5
[b1SIR--CIPP
[CISIIN-AG
8
(d]SICR-CLIP
(o)oRR-TR
(JDURTTT
Figure 3: The line charts of various in-context captions with diverse image-selection strategies.
CLD D41-7T
CID
[0)GTC
(bIMGC-T7-988
(cJMGC-TT 0966
100
BIMGCTFF135
(+)MGCNUM/222001
mMMc VLM101863
Figure 4: The line charts of various in-context images with diverse caption-assignmen strategies.
4 Experiments
4.1 Dataset and Implementation Details
MSCOCO. We evaluate the proposed strategies on MSCOCO dataset [14] which is the most widely
used benchmark in image captioning. used the Karpathy split [59 the experiments, which
contains 113,887/5000/500 raining/validation/ees images and each image is associated with
human-annotated captions.
Implementation Details We employ the Open-Flamingo model [16] to test our strategies, setting the length penalty to -2.0 and maximum generation length 20. We follow Flamingo [6] to use 4, 16, and 32 shots. We respectively use ViT-L/14 and as the vision and language encoders to extract image and sentence embeddings that used in standard Transformer decoderr architecture the MSCOCO dataset and use the checkpoints underwent 1000, 3000, and 170,000 iterations respectively. These checkpoints generate the captions with CIDEr scores of 66, 88, and 135 on the test set. We implement all experiments on single RTX 3090 using FP16.
and SICR-CLIP. For MGC-TF, we train the
4.2 Results and Analyses
Given our varied strategies for image selection and caption assignment, displaying results for each configuration in table format, especially at 4, 8, 16, and 32-shot levels, could become overwhelming For clarity, we' we've chosen present results using line charts and histograms in the main paper, while detailed numerical outcomes are in the supplementary material. The line charts in Figures] 3 and illustrate trends as the shot number grows. Each subplot within these figures corresponds histograms of
unique
strategy for image selection or caption assignment. Furthermore, Figures average CIDEr scores for the various shot results. To facilitate comprehension, we first analyze the effects of caption qualities 4.2. and then of image selection strategies 4.2.2
4.2.1 Effects of Caption Qualities
Figure 4 (e) reveals that performance typically improves with increase in shot numbers. How- ever, the rate of improvement varies, or even declines, depending on the quality of the captions used. For instance, in Figure A a), using Ground-Trie -ruth Captions (GTC), the increase rates of the

---

MOCA
(a3RS
(bJSHR-CLIP
(G)SII-TAG
1MA
10:0
RECUMEPROANID (8)SICR-CLIP
(eJOUIR-FR
Figure 5: The histograms of various in-context captions with diverse selection strategies.
1649
RARE
C00C
GRGP CLP
NE BED CUP
SHRTE
SER
SROP 8870 NER-OUP
ORE OILT
R33
(GNGGTTFF135
(6IMGC VLM132)861
(TIMG VLMI01
Figure 6: The histograms of various in-context images with diverse caption- assignment strategies.
six image selection strategies surpass those in Figure where MGC-TF@88 is used. Further, low-quality captions, such as MGC-TF@66 66, may become "toxic examples" " which can misguide the Vision-Language Model (VLM) the shot number increases. Next we compare different kinds of captions to figure out what characteristics of the captions affect the performance of in-context captioning.
Ground-Truth Captions (GTC) vs. Model-Generated Captions (MGC), As discussed See 3.2 MGC exhibits primary shortcomings when compared to GTC: poorer language and less de- scriptiveness. However, we will see that these two shortcomings do not always make MGC achieve poorer performance when compared GTC. Specifically, find that up certain level of descriptiveness, simpler sentence patterns are more easily recognized by the VLM, thereby improving caption generation.
Evidence for this can observed in Figure comparing GTC with MGC-TF@ 135. Compared to the caption generated by MGC-TF@ 135, grounddtruthh caption has better language patterns, e.g., rich vocabulary and complex sentence pattern, and better descriptiveness. Then when the selected images cannot provide enough vision cues, i.e., when Random Selection (RS) is applied in Figure (a), GTC outperforms MGC-TF@ 135. However, once the similarity- based retrieval methods like Similarity- based Image- Image Retrieval (SIR) Similarity- based Image-Cart Caption Retrieval (SICR-CE (ICR-CLIP) is used, the selected similar images offer useful visual patterns that help address the descriptiveness issue. Consequently, the VLM is more likely to recognize the consistent, simple patterns in MGC than the rich, diverse patterns in GTC and then generate better captions. Figure (a) visualizes examples about the above comparisons.
This effect is more pronounced in 4-shot cases, where VLM lacks sufficient in-context captions to discern sentence patterns, making simpler patterns advantagcous, As long the descriptivenes: issue addressed, MGC often outperforms GTC, e.g., MGC-MFFFFFF 88>GTC in Figure ((-d). Especially, when SICR-CLIP is employed to select captions with high cross-modal similarity to the test image, significantly mitigates the descriptiveness problem. Then as demonstrated Figure 3 d), even
MGC-TF @ surpasses GTC.
MGC-TF vs. MGC-VLM. Before we see that using more low-quality captions like MGC-TF@66 will misguide VLM to generate worse captions. Yet, Figure 4 (f) indicates that using MGC- VLM(() 63 improves performance with increased shot numbers, contrasting MGC-TF@66 in
Figure (c). This raises the question: why do weaker captions (MGC-VLM(0) surpass those
7

---

of rebisydds parted
parted frons
parked frons
A place cako on-a (i
A piece of cake-on plane
pidte with fork and
Secoal mnto soodies makket struet
row bicydle
Rows micdoy scooteer
This dice looks balf voriill
taken of
Thes dice of cako looks chetsooake
from
half vere
(6) MGC-TF@ CeTTTF@L666bal 135(hlu)) GTC(red)
A dose view trofic fiin: thet' red
diase red taffic light.
picture of trafic ig that's red.
Two saling boats mocred In hartour
of with bose and dont in the feroground.
bost and dock fonoground.
ssstosiign that ona
A stop sign strect
Popp Sign on the strent street
A group of bouss in
A bost on boot
Ai boot with bost
water
(b) MECC--LMMGGGGNUNE MGC-TF@RRrre
Figure 7: (a) Two examples show that GTC uses more diverse words than MGC-TF@ 135, making VLM hard to recognize the major pattern, e.g., it mis-generates "a store" in left or neglects "a fork" in right. (b) Two examples demonstrating how VLM is misguided by syntactic errors in MGC- VLM@66, where certain phrases are repeated such as "a street" or "a boat in the water".
with higher CIDEr (MGC-TF@66)2 This discrepancy aligns with our assumption: descriptiveness and language pattern influence in-context captioning differently.
NGC-TF@65 excels in object detection but struggles with language decoding, causing salient objects to be identified correctly but with syntactical errors in captions. As such examples increase, VLM produces worse captions due to these errors. Conversely, MGC-VLM(() @3,, though limited in object recognition, maintains better grammar. When more vision cues are provided, VLM leverages these along with the better-formed captions from MGC-VLM(0)6 63, resulting in improved captions. Figure 7 (b) offers two comparison examples.
Captions as Anchors (MGCA). From Figure 5 we see that using MGC as the in-context captions usually underperforms GTC. However, by the MGCA strategy, we observe consistent improvements over GTCs, as demonstrated by the higher blue histograms of different MGCs compared to the grey dashed line. For example, despite MGC-TF@66 only achieves 64. 24 CIDEr score Figure Ja), we still observe a 3.2 CIDEr improvement when using MGC-TF@66 as the anchor, compared to simply selecting the first GTC. Specifically, when using NGC-TF@66/MGC: 66/MGC- TF@ 88/M8C//MTTTTT TF @ 135/MGCC (LM @ 63/MG3/ VLM @81 as the anchors, the average im- provements over six image selection strategies compared to GTC are 7.3/8.0/8. 8/3.6/4.8 respectively. In contrast to solely leveraging the RS+GTC method, the combination MGC-TF@ @ 135 anchor chalked up average boost of 20.9.
SIRRCLIP and using
The primary reason for such improvement likely that MGC, to some extent, verbalize the major patterns, such as the salient objects of an image. This helps identify which GTC provides more detailed information about these patterns. Such detailed information of the salient objects provide help VLM generate better captions. This assumption is further supported by comparisons between MGC-TF and MGC-VLM. Given that MGC-TF prioritizes verbalizing the salient objects of an image, using MGCCTTF@6/MGC-TT 88 anchors tends to select superior GTC than MGC- thus yiclding higher improvements.
Iteratively Prompting (IP). Table | showcases the IP CIDEr scores. "Iter represents either the 0 or 32-shot performance, with subsequent columns reflecting average CIDEr scores for 4, 16, and 32-shot scenarios. In the inaugural iteration, we employ consistent set of 0 or 32-shot image-caption pairs. For the following iterations, images are selected via the Random Sampling (RS) ap- proach, and captions generated in the prior iteration serve as the in-context captions. Analyzing the data, it's evident that MGC-VLM(O) stabilizes by the third iteration and MGC-VLM(32) by
79.9
MGC VLM(32)
$5.3
30.5
79.4
78.9
77.1
Table The CIDEr scores of in dif- ferent iterations.

---

with
adomed with wooden chairs with skue
1206 wooden ChARTS with nec
AT with weoden chain
on top of nummunded dhars.
on sop or summunded
accoden cheam and
(a) Experiment (1)
SPOP
Mlong plant
addorned prreette
plant surmended with woodon chairs
A to a20 mad with wetdden (cconnss
with venrddn rhuiis
banaries on
(b) Experiment
on top of surmundee avoddnn chairs.
woodenn challs with blue accenit.
car with signn that lays etep
on of summunded chairs
wooden chalrs with accents
Benaram
(c) Experiment (3)
Figure 8: Four examples illustrate the phenomenon of short-cut inference. (a) When the in-context images are identical to the test image, the generated caption mirrors the in-context captions. (b) When using SIRR-LIP to select similar examples, the generated caption tends amalgamate features from both the in-context and test images, sometimes leading to ambiguous or partially accurate descriptions. (c) In contrast, when the in-context images are distinct from the test image, the generated caption more aptly describes the image. including specific words such as "car" or "bananas".
the second, indicating that extended VLM iterations might be redundant. Remarkably, even when confined to just 32-shot GTCs, merely two iterations of -P- for instance, where MGC-VLM(32) attains an average CIDEr of 80.5 can rival performances seen when all GTCs are utilized, as exemplified by RS-GTC's average CIDEr of 80.04 Figure
4.2.2 Effects of Image Qualities
We evaluated the outcomes of several image selection techniques. SIRRCCIP, leveraging vision embeddings to compute retrieval similarities, generally identifies images that are more analogous than those found by SIRRTAGG This is likely attributed to the intrinsic noise present semantic tags. SICR-CLIP, emphasizing captions that spotlight prominent objects, naturally gravitates towards images showcasing similar objects. In contrast, both DIIR-E TR and DIIR-TT produce more varied selections. Nevertheless, when benchmarked against RS, every retrieval- basel model consistently fetches images that exhibit greater similarity.
At first glance, one could reasonably infer that using more similar images would invariably lead superior performance. Yet, as demonstrated in Figure 6 this assumption doesn't universally hold. When engaging high-quality captions, namely GTC in (a) and MGC-TF@ 135 in (d), the correlation stands, with more analogous images indeed translating enhanced results. For instance, all retrieval-based techniques surpass RS in this context. However, with medium-quality captions, in MGC-TF@SS (b) and VLM(32) SIR or SICR-CLI manage to outdo RS, and this is specifically observed in (e). Most intriguingly, when low-quality captions like MGC-TF@66 (c) and TLM(32)@63 in (f) are in play, analogous images inversely impact performance, as illustrated by RS outperforming SIR CLIP in (f). This critical insight underscores pivotal revelation: the efficacy of utilizing similar images is intricately tied to the caliber of the corresponding captions.
in (e), only the similarity-bass -based retrieval methods like
Similar Images Lead to Short-Cut Inference. A relatively bold hypothesis to elucidate this phenomenon is: when in-context images are similar to the test image, VLM may take short-cut by leveraging in-context captions generate new one, rather than learning how to caption from the in-context pairs. Consequently, the greater the similarity between in-context and test images, the more the VLM influenced by the in-context captions.

---

To robustly test our underlying assumption, we designed three distinct experimental setups. In each setup, the in- context captions remain consistent, comprising 5 ground- truth captions soureed randomly from an image unrelated to the test image. However, the in-context images are cho- sen differently in each experiment: (I) they are identical to the test image; (2) they are picked using SHR-CLIPP (3)
Tn-Comeet Images
GTC CIDEs
ICC CIDE
Test Imagee 6.7
192.
SHRR
12.8
180.7
Table 2: The results for verifying short- cut inference.
they are chosen via RS. This progression results in decreasing similarity between the in-context and test images from experiments (1) through (3). Table T showcases two sets of CIDEr scores. One set compares the generated captions to the five ground -round-truth captions (GTC) of the test image, and the other set contrasts them with the five in-context captions (ICC). Our findings suggest clear pattern: the closer the in-context images are to the test image, the more the VLM tends to mirror the ICC in its generated caption. For illustration, method (1) registers the highest CIDEr score when compared to the ICC. However, the caption it produces doesn't tt accurately depict the image, as reflected by its notably lower CIDEr score in relation the GTC. These observations solidify our hypothesis that images with high similarity can inadvertently prompt short-cut inference. Additionally, Figure g provides visual representations further elucidate the phenomenon of short-cut inference.
DIR, As depicted in Figure 6 the performance of the two DIIR methods noticeably lags behind SIIR-TAG. This observation underscores the contention that the strength of diversity may not always translate to superior results in every context. One plausible explanation we propose is the inherent nature of captioning be instrumental in offering a multifaceted understanding of a problem [57 captioning is relatively straightforward and may not benefit much from diverse in-context examples. Delving deeper into the DIR methods, DIR--T consistently outshines DIRR-R. This leads us to infer that the clustering based on semantic tag types might be a more optimal strategy. Such strategy, we believe, not only ensures diversity but also completeness in the selection of images. For instance, given a caption like "a brown dog is running". DIIR-TT could potentially source images that highlight elements like "brown objects", "dogs", and the "action of running".
task. Contrary certain complex NLP challenges, where diversity can
5 Conclusion and Limitations
In this study, we utilize image captioning as a case study to examine the impacts of varied multi- modal in-context configurations few-shot vision-language prompting. Specifically, we design 4 different ways to select images and 4 different strategies to assign captions to the selected images for constructing multi-modal in-context configurations. Exhaustive experiments reveal that, contrary to single-modal NLP cases, multi-modal mutual synergy significantly influences performance. Notably, observe that the descriptivenesss and language patterns of the captions differently affect perfor- mance: better performance may be achieved with simpler and consistent sentence patterns when selected images compensate for descriptiveness issues. And also discover that when the in-context images are similar to the test one, VLM may build a short-cut by directly using the in-context captions instead of really learning captioning. Moreover, our optimal strategy lead a 20.9 average increase in CIDEr scores compared to a random sampling baseline.
This study's 's primary limitation is that the inception of our exploration, the only open-source multi-modal few-shot learner available is Open- Open-Flaminno [16). However, Open-Flaminge when compared with the official Flamingo (6), underperforms due to its training on significantly less data. Consequently, some findings in this paper might shift if a more robust multi-modal few-shot learner is employed. Nevertheless, even in such scenario, the diverse configuration strategies proposed in this paper maintain their utility, aiding researchers in swiftly exploring the characteristics of the employed VLMs. Additionally, we have provided experimental results on Otter (54) and smaller version of Open-Flamingo, with detailed findings available in Appendix.
Acknowledgments
This work is supported by National Science Foundation of China (62206048), Natural Science Foundation of Jiangsu Province (BK20220819) 19), Young Elite Scientists Sponsorship Program of Jiangsu Association for Science and Technology Tj-2022-027 and the Big Data Computing Center of Southeast University.
10

---

References
[1] Jacob Devlin Ming-Wo Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2, 2019.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[3] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-33) 2023.
[4] Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. Augmented language models: survey, arXiv preprint orXiv:2302.078844 2023.
[5] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP, 2022.
[6] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716-23736 2022.
[7] Eric Zelikman, Jesse Mu, Noah D Goodman, and Yuhuai Tony Wu. Star: Self-taught reasoner bootstrapping reasoning with reasoning. 2022.
[8] Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? DeeLIO 2022, page 100, 2022.
[9] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816-3830, 2021.
[10] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086-8098, 2022.
[11] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3081-3089 2022.
[12] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-languaae tasks via text generation. In International Conference on Machine Learning, pages 1931 PMLR, 2021.
[13] Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. Multi-modal answer validation for knowledge- saaseedge-based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 27 1712-21 222122 2022.
[14] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Visi 2014: 13th Europeun Conference, Zurich, Switzerland, September 6-12 12, 2014, Proceedings, Part 13, pages 740-755. Springer, 2014.
[15] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shorteut learning deep neural networks. Nature Machine Intelligence, ((11):665-111 -76,6666666666 2020.

---

[16] Anas Awadalla, Irena Gao, Josh Gardner. Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Korblithh Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.013900 2023.
[17] Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018
[18] Mike Lewis Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettleoyer. Bart: Denoising sequence-to-sequeee pre-training for natural language generation, translation, and comprehension. arXiv preprint W:Xiv:/900.13400 2019.
[19] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4921-4933 2021.
[20] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11 International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, 2021.
[21] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, 2021.
[22] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
[23] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou Self- consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177 2022.
[24] Hangfeng He, Hongming Zhang, and Dan Roth. Rethinking with retrieval: Faithful large language model inference. arXiv preprini arXi::3001.00303 2022.
[25] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with thain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:22 10509, 2022.
[26] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for ussion-and-laguue tasks. Advances in neural information processing systems, 32, 2019.
[27] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1100-5111, 2019.
[28] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng. and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, pages 104-120. Springer, 2020.
[29] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for lasion-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020.
12

---

[30] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi- task vision and language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10437-10446, 2020.
[31] Ziwang Fu, Feng Liu, Hanyang Wang. Siyuan Shen, Jiahao Zhang, Jiayin Qi, Xiangling Fu, and Aimin Zhou. Lmr-cbt: Learning modality-fused representations with cb-transformer for multimodal emotion recognition from unaligned multimodal sequences. arXiv preprins 21.1v22122.11999 2021.
[32] Zhong Ji, Jingwei Ni. Xiyao Liu, and Yanwei Pang. Teachers cooperation: team-knowledge distillation for multiple cross- dossid few-shot learning. Frontiers of Computer Science, 17(2): 172312, 2023.
[33] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint 2rX.v:2306. 2023.
[34] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https Survey Certification.
//openreview nett/forum//--yzkSutzza
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 9748-8763. PMLR, 2021.
[36] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Mul- timodal chain-of-thought reasoning in language models. arXiv preprint rrXiv:2302.0092: 2023.
[37] Jing Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. 2023.
[38] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[39] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. /2597, 2023.
[40] Jiahui Yu, Zirui Wang. Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. 2022.
[41] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212, 2021.
[42] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507-2521,22 2022.
[43] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations, 2022.
[44] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 15659-15669, 2023.
13

---

[45] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-453 2020.
[46] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in- context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655-2671, 2022.
[47] SU Hongjin, Jungo Kasai, Chen Henry Wu, Weijia Shi. Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zeullemoer. Noah A Smith, al. Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Representations.
[48] Sawan Kumar and Partha Talukdar. Reordering examples helps during priming-based few-shot learning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4507-4518, 2021.
[49] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 2022.
[50] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156-3164, 2015.
[51] Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis. and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8415-8424, 2021.
[52] Rita Ramos, Bruno Martins, Desmond Elliott, and Yova Kementchedjhieva. Smallcap: lightweight image captioning prompted with retrieval augmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2840-2849, 2023.
[53] Sara Sarto, Marcella Comia,, Lorenzo Baraldi, and Rita Cucchiara. Retrieval -agmmettdd transformer for image captioning. In Proceedings of the 19th International Conference Content- based Multimedia Indexing, pages 1-7, 2022.
[54] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprini arXiv:2305.03776 2023.
[55] Pengehuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-languaa models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588, 2021.
[56] Ao Zhang, Yuan Yao, Qianyu Chen, Wei Ji, Zhiyuan Liu, Maosong Sun, and Tat-Seng Chua. Fine- graineg scene graph generation with data transfer. In ECCV, 2022.
[57] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:22/0.03493, 2022.
[58] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image
description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[59] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 3128-3137, 2015.
14

---

Supplementary Material
A
Experimental results Open-Flamingo v1 9B
Here we present all the numerical results from the experiment, divided into four tables (Table Table Table S and T based on different Image Selection strategies. Additionally, we have calculated the average values for each method in the "mean" column.
Image Selection
Caption Assignment
4-shot
8-shot
16-shot
32 -shot
mean
RS
GTC
75.80
78.93
81.97
83.47
80.04
RS
MGC-TF@66
69.18
64.59
62.73
60.46
64.24
RS
MGCA-TF@66
78.52
82.44
85.48
86.53
83.24
RS
MGC-TF@88
74.60
74.47
77.80
78.71
76.39
RS
MGCA-TF@88
78.40
81.28
84.93
87.62
83.06
RS
MGC-TF@ 135
72.35
70.10
72.73
77.76
73.23
RS
MGCA-TF@ 135
78.81
80.64
83.86
86.74
82.51
RS
MGC-VLMM()@63
70.45
73.92
74.83
77.00
74.05
RS
MGCA-VLM(0) @63
76.13
79.61
82.14
83.63
80.38
RS
MGC-VLM(32)MM @
78.33
80.94
82.62
82.75
81.16
RS
MGCA-VLM(32) @85
77.22
80.16
82.97
85.01
81.34
Table Using Random Selection (RS) image selection strategy results.
Image Selection
Caption Assignment
4-shot
8-shot
16-shot
32 -shot
mean
SIIR-CLA
GTC
80.32
88.76
95.18
98.24
90.62
SIIR-CLIP
MGC-TF@66
69.08
66.78
65.79
65.51
66.79
SIRRCLIP
MGCA-TF@66
89.69
97.53
102.31
104.63
98.54
SUR-CLIP
MGC-TF@$8
81.86
82.43
84.77
84.89
83.49
SHRR-CIP
MGCA-TF@88
91.11
98.23
102.74
106.24
99.58
SIRR-LIP
MGC-TF@ 135
95.64
96.62
97.66
98.32
97.06
SIRR-CIPP
MGCA-TF@ 135
92.73
99.47
104.09
107.52
100.95
SIRR-LIP
MGC-VLM(0))63
65.98
69.52
71.88
73.49
70.22
SHRR-CIP
MGCA-VLM(0) @333
86.72
93.30
98,42
100.85
94.82
SIRR -CLIP
MGCCVLLM32)@88
79.46
81.34
84.14
84.86
82.45
SHIR-CLIP
MGCA-VLM(32) @85
88.09
95.70
98.98
102.12
96.22
GTC
78.37
86.40
92.08
94.94
87.95
-AIS
MGC-TF@66
71.23
68.27
66.77
65.12
67.85
SUR-TAGS
MGCA-TF@66
87.40
93.31
97.85
102.16
95.18
SHIR
MGC-TF@88
82.70
81.80
82.51
83.33
82.58
SIRRTAGGS
MGCA-TF@88
87.86
94.26
98.69
102.64
95.86
SIRS
MGC-TF@ 135
89.35
91.14
92.73
94.86
92.02
-
MGCA-TF@ 135
88.89
94.45
100.34
104.44
97.03
SIIR-TAGS
MGC-VLMM0))63
67.11
70.21
72.21
74.45
71.00
BIR-TAGS
MGCA-VLM(()&G @63
83.23
89.35
94.51
98.79
91.47
MGC-VLM(32)GM
79.69
81.96
82.82
84.25
82.18
SIRR TAGS
MGCA-VLM(32)
@85
85.04
91.23
96.10
100.10
93.12
Table 4: Using Similarity-based Image-Caption Retrieval (SIR--LIPP) image selection strategy
results.
15

---

Image Selection
Caption Assignment
4-shot
8-shot
16-shot
32 -shot
mean
SICR-CLIP
GTC
72.91
81.76
89.40
94.51
84.64
SICR-CLIP
MGC-TF@66
76.03
74.66
74.73
73.75
74.79
SICR-CLIP
MGCA-TF@66
87.50
95.67
101.56
104.42
97.29
SICR-CLIP
MGC-TF@88
84.30
84.09
85.34
84.83
84.64
SICR-CLIP
MGCA-TF@88
88.41
96.37
101.89
105.07
97.94
SICR-CLIP
MGC-TF@ 135
92.60
93.69
94.64
95.30
94.06
SICR-CLIP
MGCA-TF@ 135
91.19
98.41
103.24
107.24
100.02
SICR-CLIP
MGC-VLM(00)63
68.19
70.34
73.97
76.91
72.35
SICR-CLIP
MGCA-VLMM() @3
82.49
91.08
96.64
101.55
92.94
SICR-CLIP
MGC-VLM(32)888
82.39
85.02
87.22
87.08
85,43
SICR-CLIP
MGCA-VLM332)@SS
84.18
91.52
98.02
102.89
94.15
Table 5: Using Similarity-based Image-Capten Caption Retrieval (SICR) image selection strategy results.
Image Selection
Caption Assignment
4-shot
8-shot
16-shot
32 -shot
mean
DIRR-R
GTC
76.01
82.88
87.13
93.25
84.82
DIRR-T -R
MGC-TF@66
71.45
66.82
66.22
65.80
67.57
DIR-TR
MGCA-TF@66
82.27
87.45
93.56
97.49
90.19
DIR-TR
MGC-TF@882
81.03
79.50
81.88
83.06
81.37
DIR-TR
MGCA-TF@88
82.85
88.84
93.93
99.18
91.20
DUR-TR
MGC-TF@ 135
83.59
82.63
84.33
89.03
84.89
DIR-TR
MGCA-TF@ 135
83.44
88.83
94.26
98.23
91.19
DIRR-R
MGC-VLM(()@63
67.64
70.91
73.21
74.86
71.66
DIRRTR
MGCA-VLM(0)66
78.68
84.41
89.96
93.59
86.66
DIR--R
MGC-VLM(32)@88 85
77.35
80.72
81.97
83.72
80.94
DIR-TR
MGCA-VLM((2)
79.88
85.40
89.59
94.62
87.37
DIRR-T
GTC
75.65
83.21
89.70
95.42
86.00
DIRR-T
MGC-TF@66
72.59
68.51
66.66
65.53
68.32
DIR--T
MGCA-TF@66
84.88
91.36
96.81
100.73
93.44
DIR-TT
MGC-TF@88
81.94
81.76
82.19
83.80
82,42
DIR--T
MGCA-TF@88
85.13
92.61
97.35
101.61
94.17
DIR-TT
MGC-TF@ 135
88.01
88.65
90.58
94.04
90.32
DIR--T
MGCA-TF@ 135
86.77
93.25
97.92
102.97
95.23
DUR-TT
MGCCVLM(0))63
66.42
69.64
72.71
74.56
70.83
DIR-TT
MGCA-VLM(()G) @63
79.32
87.36
92.76
97.69
89.28
DIIR-TT
MGC-VLM(32)@85
76.91
80.21
83.24
84.43
81.20
DIR-TT
MGCA-VLM(32)@85
81.39
87.83
93.64
98.61
90.37
Table Using Diversity-based Image-Image Retrieval (DIIR) image selection strategy results.
B Experimental results on Open-Flamingo v2 3B
Here we present the numerical results on Open-Pleniinnn Flamingo 3B model | Table based on two different Image Selection strategies. Additionally, we have calculated the average values for each method in the "mean" column.
From the values in the table, can be seen that the trend is basically consistent with the v1 model. can be considered that our strategy and analysis can be transferred different models.
C Experimental results on Otter
Here present the numerical results Otter model in Table X based on two different Image Selection strategies. Additionally, we have calculated the average values for each method in the "mean" column.
Projecct
16

---

Image Selection
Caption Assigmment
4-shot
8-shot
16-shot
32 -shot
mean
RS
GT
77.90
86.14
90.80
93.17
88.47
RS
MGC-TF@66
82.16
86.49
88.16
86.61
86.55
RS
MGC-TF@55
82.47
88.43
91.81
93.83
90.12
RS
MGC-TF@ 135
84.64
89.93
90.70
92.03
90.31
RS
MGCA-TF@66
77.95
87.64
92.54
95.14
90.09
RS
MGCA-TF@88
78.95
88.21
92.17
95.28
90.19
RS
MGCA-TF@) 135
79.61
87.99
91.92
95.22
89.95
SIR
GT
84.53
92.36
96.36
98.66
94.36
SIR
MGC-TF@66
84.72
77.69
75.29
76.49
77.09
SIIR
MGC-TF@88
93.38
91.29
90.50
89.25
90.90
SIR
MGC-TF@ 135
104.10
103.80
103.31
102.39
103.56
SIIR
MGCA-TF@66
90.00
97.51
102.04
103.50
99.78
SIRR
MGCA-TFGG
91.15
98.88
102.17
104.55
100.53
SIIR
MGCA-TFA TF@ 135
91.91
99.01
103.26
104.93
101.14
Table Open-Flamingo v2 3B results with various strategies.
From the values in the table, it can be seen that the trend is basically consistent with the vl model. be considered that strategy and analysis can be transferred to different models.
Image Selection
Caption Assignment
4-shot
8-shot
16-shot
32 -shot
mean
GT
83,43
88.36
91.86
93.09
90.11
RS
MGC-TF 135
75,49
80.77
85.46
89.53
83.115
RS
MGCA-TF@ 135
84.67
89.6
92.45
93.97
91.025
SIR
GT
87.4
90.72
94.42
95,94
92.57
SIR
MGC-TF@ 135
95.02 90.93
97.37
98.8
99.88
98.085
SIR
MGCA-TF@ 135
96.59
97.08
101.3
96.835
Table Otter results with various strategies.
D More Results of MGC-TF@135 vs. GTC
We further elucidate the performance of MGC-TF @ 135(blue) and GTC(red), by offering additional examples shown in Figure 9 can be easily observed that GTC has a more diverse range of sentence structures in comparison to MGC-TF@ 135. In the initial two examples, the words "cat" and "dachshunds" were inaccurately recognized by GTC, demonstrating its limitation some specific instances. A shift can be noticed in the subsequent three examples, where the GTC generates captions with complex sentence structures, with a notable proportion of incorrect information. This finding serves reinforce the notion that simpler sentence patterns, up certain degree of descriptiveness, are more readily deciphered by the VLM, thereby enhancing the quality of generated captions.
E
More Results of MGC-TF@66 vs. MGC-VLM(0663
Similarly, we supplement with more examples, as depicted in Figure 10 to facilitate the compar- ison between MGC-TF@66 and MGC- (LM()) (3llue)). striking observation from this comparative study reveals that MGC-TF BRCC-FF@6 66 demonstrates a significant number of syntactical errors. This flaw becomes problematic as it misdirects VLM, resulting substantial volume of syntactical mistakes in the produced sentences. This correlation implies that the syntax errors in the initial input by MGC-TF@ 66 tend to propagate into the VLMM's output. Therefore, becomes clear that the accuracy of grammar in the prompt is crucial for achieving better results.
17

---

dogs leying top
leying couch.
leying bed
leying
two dogs laying couch.
floor
A couple dogs steeping top ol couch.
twn white bleck and brown lying
dags
Cats sleeping on top of striped white
Dog and cnt lying on flcor side slouping.
Dog and siwpping on couch.
threw clgss leying top af couch.
bwo cats laying top of
two dogs laying on tap of bed
Iwo dogs laying on top alla bed.
two dogs loying top of couch.
A couple of steeping top of couch.
bed with blla and whilee striped blankets.
DOGS BED ON TOP OF THE COVERS.
Seep bed.
dachohurels and
Sloop on couch.
dachshuund
with
hobel room with
hotel room with
table.
tefersion.
A bed a bwdroam real to glans
A pink bachpread is featured badrooem..
Twos beds in hatel room with window looking into
A very clean bedraamn, with two bedss lights
bed hotel room with window looking into the bethromm
person with pizza bok.
pizza siting box table
pizza in bas on table
paza sitting box table.
person with pizaa sitting in box on table.
has different tog.
car doddaardddd box on table.
pizza Inside the box.
of wooden cutting board
vesdden cuiling board with different toppings.
woman playing
young boy flying sky.
playing with
person holding
youung girl Rling a
A Rid is trying catch sometring while standing in park
yourgg buy Rlyng kit Blue Sky with clouds.
Amen that is chawing frebee the grass.
person in park holding krte.
kid trying to catch fribbee while stancing in park.
Figure 9: MGC- TF@ 135(blue) vs. GTC(red). Five examples demonstrate that more diverse words will be used GTC than in MGC-TF@ 135 which making VLM hard catch the major pattern. e.g., the first line and the second line incorrectly state "cat" and "dachshunds."
18

---

tome cup coffee
Acup of oofiee sitting onaa
uase table
daffodils and cup
wooden table
coffee
Etchen tabre.
soble
table
table
fowers
bable
bacle coffee
dose up a dock on side dock
cock on the of buiding
A dock the side af buildig
clock on the of suilding
close up of dack the odde of bullcinng
dock
with dlokk
with clock
clock dock on
lppipp on table
sitt with Bytop front fhim
laptop deak hatel room
lastep computer tabl with cup coffee.
lastop front of her.
bptop siting on top
with
Aaptop string on top a
Eaptop siting on desk with
battop diting on top of
motorbik
motorille halcnn on cell phone
riding motorblle with woman and child the back.
motorbike
fox daughts:
at mm of motorryctee tie.
police ratlo
group of people wolking
man riding motorcycle on motorrytle
à man riling motercycle motosryele
man ricing motorgyle on motarnyele
man rkling motorcycl en matorryele
purple
vas
with purple
glas wase crange
orange
Rowers
in
with vine and saDe
VARE 19n0 anrd UiGe
VEDe with vo0e and 1oee
table vase and fowers on
vess
Figure 10: MGC- (LGC-V @33bbuc) vs. MGC-TF@ misguided by the syntactic errors in IGC-VLM@@6 66. Lots of outputs have repeated phrases, such as "a table", "a clock", "a desk", motorcycle" and "a vase".
Five examples show that VLM will be
19

---

More Results of short-cut inference
In this section, we provide additional examples demonstrate how similar images can lead VLM to exhibit a tendency for short-cut inference, where it might relies heavily on in-context captions to generate the captions, disregarding the image information in the prompt and the inherent relationships within the in-context pairs. In Figure [1] we present the results for the same test image under two different scenarios of choosing in-context images: "identical to the test image" (top row) and "via random sampling" (bottom row).
In the first scenario, for the majority of cases, results largely unrelated the image content but similar to the in-context captions. with only a few instances capturing some elements from the image, such as "pink" in (a) and "on the beach" in (b). However, in the second scenario, our results heavily influenced by the in-context captions, as evident in (c) and (d) with the mention of "wooden chairs".
A rete rant has modem chars.
lang restammnt mnt table back shais.
A lang tabl with plant surounded ecoden
A long fade with prongementin the moddle meebing
taple adormed with accents
table acer ned with pivi
accents
(a)
daughoe
woodon tables and
with sattan rounded
on of itsurcundio
flower anungementin middle meetings
reoddnn chills with blue acowets
are carring their byggage
ectddin
resta mann has mideem woodon
(A long retts met table bork
A ling takie with plan secrden caain.
A ling bakle with anongementin
tatle adormed with proests
rable acter aed with weode
braach
(b)
woodien tables and chairs
long resiuutant sattan rounded back chibss
oong plom on stsurcundeed with wooden halkss
Rower amangement the middle for meedings
wooden cheirs blue
running twort
accents
restaurant modam woodin tables and
lang restacend tible with sattnn rounded
lang labie with on top suuouunddd
Alng table with flower inancementio
laple adoreed woodan chairs with blue
tabll woedes chais with red
meetings
rostaurant uan has modomm Chains
Aloong rosta rom table back diills.
A longg tible with plant thth eecddon chills.
A long table will the middle for meedings.
Baole adommed pcconts
Gable is 2oer red with bovf of dog food
modem woodinn call and
long table with Vitan rounded
lakie with on 1og of sunoundo
bable Rowe zhongementin
tazle adommed wolddon chills blue
table conted with wooden dhalts with rod
resia mmn moddmm
long resta min table back cheiss.
long tabie with plan with eccddnn chairs.
long tañe the meddle for meedings.
Sardee adoomed eccents.
siss woodee chai
chars.
Figure 11: Five examples of verifying short-cut inference. we provide two different ways of choosing in-context images: "identical to the test image" (top row) and "via random sampling" (bottom row).
20


Reviews:
[{"review_id": "LkQvgT1LyP", "reviewer": "Reviewer_zeNe", "summary": "This paper explores the in-context configurations for few-shot ability in Vision-Language Models (VLMs). Therefore, they design four strategies for image selection and four for caption assignment to explore the influences of in-context pairs in image captioning. As a result, extensive experiments uncover two valuable insights (1) The captions adequately describe salient image objects and simpler language patterns may yield better results. (2) Excessive similarity might cause VLMs to create a short-cut inference from in-context captions. Finally, they introduce the iterative prompting for images with limited or no ground-truth captions, which boosts the performance of models with an average CIDEr improvement of 7.3.", "questions": "I tend to increase the review score once the following questions are answered well. The following questions are shown in weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.\t**The considered problem is very relevant and timely for the AI community.**. It is urgent to explore the in-context configurations for vision-language pre-trained models. \n2.\t**The observations are valuable.**. They observe that better performance may be achieved with simpler sentence patterns when selected images compensate for descriptiveness issues. Moreover, they observe that the VLM may build a shortcut rather than learn to caption when the in-context images are similar to the test one.\n3.    **The experiments are comprehensive.**. They devised four strategies for image selection and four for caption assignment to explore the influences of in-context pairs in image captioning. \n", "weaknesses": "1.\t**The selected metric is not convinced.** They use CIDEr to evaluate caption models' performances. However, CIDEr is a statistical indicator related to description length, which is hard to completely reflect the quality of the generated captions. Therefore, it is necessary to explore generative evaluation indicators such as CLIPScore[1] and GPTScore[2] that use pre-trained models.\n2.\t**The choices of VLMs are limited, which may lead to unobjective conclusions.** They just use the OpenFlamingo as the multi-modal learner.  The authors should add more VLMs to explore the in-context configurations for image captioning, e.g. MiniGPT4[3], LLaVA[2].\n\n3.\t**The observation is a little overclaiming.** The in-context configurations observation from the image caption is not equal to the VL in-context learning. There are many visually-conditioned VL tasks that need to be explored such as VQA[4], and dense image captioning.\n\n4.    **The method section of iteratively prompting (IP) is confusing.** It is hard to understand how the algorithm of IP works, and the lack of analysis for IP.\n\n\n[1] Hessel, Jack, et al. \"Clipscore: A reference-free evaluation metric for image captioning.\" arXiv preprint arXiv:2104.08718 (2021).\n\n[2] Liu, Haotian, et al. \"Visual instruction tuning.\" arXiv preprint arXiv:2304.08485 (2023).\n\n[3] Zhu, Deyao, et al. \"Minigpt-4: Enhancing vision-language understanding with advanced large language models.\" arXiv preprint arXiv:2304.10592 (2023).\n\n[4] Tsimpoukelli, Maria, et al. \"Multimodal few-shot learning with frozen language models.\" Advances in Neural Information Processing Systems 34 (2021): 200-212.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. CLIPScore Assessment.**\nUpon your recommendation, we employ the CLIPScore to re-assess the quality of our generated captions. Due to space constraints, we present only the key findings here which validate two key conclusions shown in Lines 227-229 and 302-306. We first show the results about the first conclusion:\n| Image Selection | Caption Assignment | Mean CLIPScore |\n|-----------------|---------|------|\n| RS | GTC | 79.14 |\n| RS | MGC-TF@135 | 78.05 |\n| SIIR-CLIP | GTC | 80.46 |\n| SIIR-CLIP | MGC-TF@135 | 79.79 |\n\nHere, RS denotes Randomly Sampling, SIIR-CLIP denotes CLIP embedding Similarity-based Image-Image Retrieval, GTC denotes using the first ground-truth caption among 5 human-labelled ones, and MGC-TF@135 denotes using the model generated captions whose CIDEr score is 135.\nFrom the table, it's evident that under RS, GTC (79.14) outperforms MGC-TF@135 (78.05), while utilizing SIIR-CLIP causes MGC-TF@135 (80.46) surpassing GTC (79.79). This observation is consistent with the findings given in Lines 230-238, which also supports our claim in Lines 227-229. \n\nThen we show the results about the second one conclusion that similar images lead to short-cut inference, where we measure the captions whose CIDEr scores are reported in the following table:\n|   | In-Context Images | GTC CLIPScore | ICC  CLIPScore |\n|---|-------------------|------------------|------------------|\n|(1)  | Test Image        | 50.70            | 77.78            |\n|(2)  | SIIR-CLIP         | 53.50            | 77.44            |\n|(3)  | RS                | 73.90            | 59.03            |\n\nwhere we still observe that as in-context images more similar to the test image, VLM tends to mimic in-context captions (ICC), which is consistent with the findings in Lines 315-319. For instance, as the similarity from method (1) to (3) declines, the CLIPScore diverges more from ICC but converges towards ground-truth captions (GTC). More results using CLIPScore and GPTScore will be incorporated in revision.\n\n**2. LLaVa and MiniGPT.**\nLLaVa and MiniGPT are primarily designed for instruction tuning where it works in a multi-round chat manner instead of an in-context learning manner. The network architectures of both models are not feasible to handle extensive, interleaved vision and language data as Flamingo. Moreover, they lack the appropriate training losses to effectively handle few-shot prompts inputs in the manner that Flamingo does. As such, they may be capable of processing 4-shot inputs in the manner of multi-turn chat, while will fail to deal with much more shot data like 8 or 16-shots. However, to investigate the impact of diverse configurations, a model should be able to handle much more shot data. Considering these disadvantages, we do not use them to explore in-context configurations. \n\nHowever, we still try some other VLMs to explore in-context configurations. Please refer to **\"7. Experiments on Smaller Open-flamingo and Otter.\"** in the response to **Reviewer tg5h** to see the results test on Otter and a smaller version of Open-Flamingo, which are two models with in-context learning ability published after the NeurIPS submission deadline.\n\n**3. VQA Results.**\nWe follow your suggestion to explore another VL task to test the major conclusion of our study. Given our computational limitations and the tight rebuttal timeframe, we choose to explore VQAv2 since it is more different from image captioning compared to dense captioning, which is also suggested by Reviewer vuiF. To explore different configurations in VQA. We use two different image selection strategies which are Random Selection (RS) and  CLIP Similarity-based Image-Image Retrieval (SIIR-CLIP). Moreover, to ensure varying quality levels of in-context text—best, middle, and worst, we use matched (MAT), half-matched (HMAT), and mismatched (MMAT) question-answer pairs. Here, MAT, HMAT, and MMAT signify that all, half, or none of the answers are correct, respectively.\n| Image Selection | QA pairs     | Mean Accuracy |\n|-----------------|--------------|--------------|\n| RS              | MAT| 47.97|\n| RS              | HMAT| 47.34|\n| RS              | MMAT| 47.13|\n| SIIR-CLIP | MAT| 48.48|\n| SIIR-CLIP | HMAT| 47.55|\n| SIIR-CLIP | MMAT| 46.95|\n\n\nWe can find that when the text quality is high, i.e. using matched question-answer pairs (MAT), we have SIIR-CLIP > RS, suggesting that using similar images achieves higher accuracy than random sampling. However, when using low text quality, e.g., HMAT or MMAT, similar images do not always lead to better results, e.g., with MMAT, SIIR-CLIP < RS. Such observation is consistent with the finding in Lines 230-238 and supports the major conclusion that image and text selection strategies influence each other. \n\n**4. Iterative Prompting (IP).**\nBriefly, IP employs VLM for iterative in-context caption generation. It is designed for scenarios with abundant images but limited human-labelled captions. Consider $\\mathcal{D}=\\{(I_1,C_1);...;(I_n,C_n), \\tilde{I}_1;...;\\tilde{I}_N\\}$ where $n<<N$ (e.g., $n=32$ and $N=10000$) and only $n$ images contain human-labelled captions. Given a test image $\\hat{I}$, traditional image selection becomes impractical due to the scarcity of available captions. Therefore, we suggest generating a caption $\\tilde{C}$ for each $\\tilde{I} \\in \\mathcal{D}$ using Eq.(1) where $\\mathcal{S}=\\{(I_1,C_1);...;(I_n,C_n)\\}$. This provides each image in $\\mathcal{D}$ with a corresponding caption, allowing the use of earlier image selection methods like SIIR or SICR (Similarity-based Image-Caption Retrieval). This process can be iteratively executed, hence the name Iterative Prompting.\n\nIn essence, MGC-VLM(N) introduced in Line 169 is equal to employing this method once. Given the in-depth analysis of the captions of MGC-VLM(N) in section 4.2.1, we bypass their re-examination in lines 274-286 but try to know how many iterations IP needs to make the performance saturate. We will clarify this in the revision.\n"}, {"review_id": "j57YsQoSiD", "reviewer": "Reviewer_FMu1", "summary": "This work investigates the configuration of in-context image-caption pairs for few-shot image captioning in a VL model: OpenFlamingo. Specifically, authors compare four different image selection methods: (1) random selection, (2) image-image similarity based selection (SIIR), (3) image-caption similarity based selection (SICR), and (4) image-image diversity based selection (DIIR). \n\nOn top of the selected images, the authors also test out four different caption assignment methods: (1) ground-truth caption (GTC), (2) model-generated caption (MGC), (3) Iterative prompting (IP), and (4) Model-generated Captions as Anchors (MGCA). \n\nThe findings from this work suggest that (1) selecting similar images with the test image is better than random sampling; (2) simpler language patterns in captions may be better if they adequately explain the salient info of the image; and (3)similar in-context images can induce short-cut inference from their captions.", "questions": "(1) I find it interesting that GTC performs worse most of the time compared to MGC. Why do you think this the case? I am very suspicious that it is due to the consistent format of captions across in-context samples instead of the simple pattern of the MGCs. However, this is hard to analyze with the current setup. MGCs will have very similar sentence structures for similar images compared to GTCs, which is also shown in Figure 7-(a). I suggest controlling the format consistency across GTCs within the in-context samples. For example, you can set captions to have the same complex structure and see how that results in performance. If simplicity is the true cause of improved performance, setting consistent but complex format should not give improve performance. Also, you can try very diverse format across captions but make them simple in structure.\n\n(2) Don’t almost all GTCs include the most salient object in their captions? I find it hard to believe that MGCA improves performance because it finds the GTC that describes the salient pattern with more info. The selected GTCs from MGCA will probably have similar structure with the MGCs, hence it reduces the format variance of GTCs across the in-context samples. What happens if you select the GTC that is least similar with the MGC?\n\n(3) When giving the in-context samples, do the images really help in performance? What if you drop all in-context images and give only the captions instead? \n\n(4) I think it would be better to swap the location of Figure 3 & 4 with Figure 5 & 6. Is there a reason why you put Figure 3 & 4 first? I found it difficult to go back and forth because the main draft explains Figure 5 & 6 first.", "limitations": "Yes, the authors also acknowledge the limitation that these analyses were conducted on a single model.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Analyses on in-context sampling method for VL models can expand the literature. Especially, the fact that MGCA results lead to better performance than random ground-truth captions is interesting.", "weaknesses": "All experiments are conducted on a single open-source VL model; this leads to questions whether these results hold generalizability.\n\nFurther analyses should have been conducted to backup the authors’ claims: “simpler caption pattern improves caption generation” & “verbalizing the major patterns help identify which GTC provides more detailed info about these patterns”. Please see the question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Consistent Format.**\nWhen we say that the model-generated captions (MGC) have \"simple patterns\", part of the meaning we wish to convey is that they often have a consistent format for captions, e.g., in Lines 236-237, we mention that ground-truth captions (GTC) contain more diverse patterns than MGC, which challenges the VLM to recognize one uniform format. We agree with you that the statement of \"consistent format of captions\" is more accurate than \"simple pattern\" and we will use this statement in the revision.\n\nTo further validate this claim and follow your suggestion, we conduct an additional experiment by manually selecting 1000 captions which have a consistent format and building a candidate set $\\mathcal{D}$ by these selected captions and their corresponding images. Then we apply random sampling (RS) to select images and use these captions with a more consistent format as in-context captions to build a baseline \"CF\" shown in the following table.\n| Image Selection | Caption Assignment | Mean CIDEr|\n|-----------------|--------------------|--------|\n| RS | GTC | 80.04 |\n| RS | CF |82.25 |\n| RS | MGCA-TF@88 | 83.06 |\n\nNote that \"GTC\" used in this table denotes to use the first ground-truth caption among 5 human-labelled ones and MGCA-TF@88 denotes to select one ground-truth caption by using the caption of MGC-TF@88 as the anchor. It can be seen that using the captions with a consistent format outperforms GTC, suggesting that consistent format plays a significant role in in-context captioning. However, MGCA still outperforms CF, suggesting that consistent format is not the only factor that improves the performance.\n\n**2. Salient Objects.**\nOne image usually contains lots of objects where some are salient and some are not. However, different human annotators have diverse preferences in that they may describe different details of an image, including some non-salient objects. For example, [one image](https://cocodataset.org/#explore?id=280810) from the COCO dataset has the following ground-truth captions:\n- a small plate holds a pot pie and broccoli.\n- pot pie and broccoli on a plate on a table with a laptop in the background\n- a plate holds a meat pie, carrots and broccoli.\n- lunch is in front of the laptop computer.\n- a red and white plate topped with a pot pie and broccoli.\n\nWe can find that only the 2nd one contains both the objects \"plate\" and \"laptop\". Then an MGC \"a plate near a laptop\" helps select the 2nd caption, which also remedies some details about the plate that may help generate better captions as discussed in Lines 268-270. \n\nTo quantitatively verify the conclusion, we use VinVL [A] to extract the salient objects from each image based on the predicted probabilities and then calculate how many salient objects are overlapped with the nouns in the selected caption. We use three kinds of captions, (1) the caption which has the highest CIDEr score with the MGC, which gets **0.954** overlapping ratio, (2) the first caption among five ground-truth captions, which gets **0.845** overlapping ratio, and (3) the caption which has the lowest CIDEr score with the MGC, which gets **0.737** overlapping ratio. These findings confirm that MGCA-selected captions are more likely to contain salient objects, leading to improved performance. \n\nWe also conduct an experiment using the caption with the lowest score to MGCA as our in-context captions. Some results are given here:\n| Image Selection | Anchor | MGCA Score | Mean CIDEr|\n|-----------------|----------------|---------|------|\n| RS              | GTC            | --      | 80.45 |\n| RS              | MGC-TF@88      | highest | 83.11 |\n| RS              | MGC-TF@88      | lowest  | 73.84 |\n| SIIR-CLIP       | GTC            | --      | 91.97 |\n| SIIR-CLIP       | MGC-TF@88      | highest | 99.58 |\n| SIIR-CLIP       | MGC-TF@88      | lowest  | 74.39 |\n\nWe see that selecting the caption with the lowest score to MGCA greatly reduces performance, suggesting that the captions with the lowest scores contain fewer salient objects.\n\n\n**3. Dropping In-Context Images.**\nWe follow your suggestion to drop all in-context images and give only the captions in in-context samples. The results are given in the following, where the RS and GTC strategies are used to select the images and captions:\n|      | Mean CIDEr|\n|-----------|-------|\n| $w$ image   | 80.04 |\n| $w/o$ image | 2.99  | \n\nWe can find that without the images, the model cannot generate meaningful captions. We check the generated captions and find that they usually trend to repeat some of the provided in-context captions, suggesting the importance of the visual cues of the in-context images.\n\n\n**4. Figure 3-6.**\nWe will follow your suggestion to swap Figure 3 & 4 with Figure 5 & 6.\n\n**5. Generalizability of the Key Conclusions.**\nPlease refer to **\"7.Experiments on Smaller Open-flamingo and Otter.\"** in the response to the **Reviewer tg5h**.\n\n[A] VinVL: Revisiting Visual Representations in Vision-Language Models"}, {"review_id": "xypNB9w32b", "reviewer": "Reviewer_Qg6d", "summary": "This paper examines the effects of varying configurations on Vision-Language (VL) in-context learning, specifically in the field of image captioning. The authors developed four strategies each for image selection and caption assignment to determine how different methods of configuring image-text pairs impact the learning process. The study revealed two insights: 1) Caption quality, influenced by descriptiveness and language patterns, impacts captioning performance, with simpler language yielding better results when paired with descriptive captions. 2) The effectiveness of similar images relies on caption quality, with excessive similarity potentially misleading the model when paired with low-quality captions.", "questions": "My major concerns are about the current setting of in-context learning, and I would like to hear more about how to define the quality of the captions, which is important in choosing samples.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The general goal is reasonable and interesting. The field of in-context learning in multimodality is well worth exploring.\n\n+ The experiments were comprehensive. The authors compared numerous in-context learning strategies, including some new proposals of their own.\n\n+ The paper provides some useful insights about how to choose in-context samples.", "weaknesses": "- The authors said that “the performance ... heavily relies on the caption quality” in Fig 1 Caption. I'm curious about how the authors define the quality of a caption. The example given in Figure 1 suggests that the caption in the blue box is better than the one in the red box. It is easy to understand because the red box caption contains clear errors, such as repeated phrases. When selecting in-context samples, such samples obviously should not be selected. But, in practice, such samples should not even be available as candidates, right? Could the authors display what captions the model considers as high-quality or low-quality? Or intuitively illustrate what captions are expected to consider as a high quality among several ground truths?\n\n- One insight of the paper is that \"When captions adequately describe salient image objects, simpler language patterns may yield better\nresults.\" But aren't the ground truth captions in the training set in the language style expected by the benchmark? If we use captions generated by the model, then how does the model know what style is required by the benchmark? \n\n- I'm not certain whether the experimental setting in this paper can effectively evaluate the capabilities of in-context learning. In-context learning aims to use a small number of samples as demonstrations to learn some new things, such as a new task, new input-output format,  new object naming rules, etc. This work conducts experiments on the MSCOCO caption dataset. I'm unsure what the purpose of in-context learning is in this case, as these samples seem quite common for large models. So, what patterns or things are expected to learn by model from these in-context samples?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Quality of Image Captions.**\nImage captioning quality hinges on the ability of the caption  to accurately and grammatically describe the main events of an image. Inadequate grammar or descriptions that either neglect or misinterpret key events indicate poor quality. Human-labeled ground truth captions (GTC) typically adhere to these standards and are thus deemed high-quality. This paper assesses various model-generated captions (MGC) with adjustable quality to explore their influence on in-context captioning. Directly quantifying MGC quality based on the aforementioned standards is challenging. Hence, researchers suggest computing the similarity between MGC and several human-labeled captions using CIDEr[A] as the primary metric. As noted in lines 167-172, we allocate a CIDEr score to each MGC to measure its quality, e.g., MGC-TF@66 implies the model-generated captions with a CIDEr score of 66. Higher CIDEr scores generally suggest superior quality. Our experiments reveal that varying caption qualities, denoted by different CIDEr scores, impact the final outcomes.\n\nRegarding Figure 1, it underscores that even when the used in-context images are similar to the test one, if the in-context captions have low-quality, the generated captions will still have low quality. To emphasize this, we present examples of particularly low-quality captions.\n\n**2. Diversity in Ground-Truth Caption Patterns.**\nSince ground-truth captions (GTC) are labelled by different annotators, they usually have much more diverse sentence formats compared to model-generated captions (MGC). Consider the diverse styles of these four GTC from the COCO dataset which are used as in-context captions: \n- A homemade pizza with sauce and cheese on foil. \n- Man leaning his mouth down to a plate that has a sandwich on it and a blue water bottle behind it. \n- Bunches of bananas hanging from fire on a line. \n- There are several birds that are standing at the beach.\n\nThen as demonstrated in Lines 236-237, we show that such diverse patterns can be more challenging for a VLM to recognize compared to simple, uniform patterns, hence MGC sometimes outperforms GTC. \n\nFor the Transformer models shown in Lines 202-205 that are used to generate MGC, they are trained following [B] where the CIDEr scores between the generated captions with ground-truth captions are used as the objectives. In this way, these Transformer models will still learn the sentence style of the GTC captions.\n\nFor more discussions about caption quality and patterns, you can refer to **\"1. Consistent Format.\"** and **\"2. Salient Objects.\"** in the response to the **Reviewer FMu1**.\n\n**3. Significance of Our Study.**\nWe agree with you that the primary objective of In-context learning (ICL) is to use a few samples to learn new knowledge. However, this is not the major motivation for our research. Instead, we aim to explore how different image and caption selection strategies affect the performance of multi-modal ICL. The significance of our research is that it provides a roadmap for researchers in multi-modal ICL. For instance, our research shows that, unlike the NLP domain where using similar samples as in-context examples usually leads to good performance, in the multi-modal domain, the strategies for selecting images and texts influence each other, e.g., one major finding of our study is that it is not guaranteed that choosing similar images and high-quality texts will always yield the best results.\n\n[A] Cider: Consensus-based image description evaluation.\n\n[B] Self-critical Sequence Training for Image Captioning"}, {"review_id": "oDcS0DGoVe", "reviewer": "Reviewer_vuiF", "summary": "The paper studies how to choose in-context examples (images and text) for open flamingo models on the CoCo captioning task. Extensive experiments are run with many baselines. Based on these experiments, the authors conclude (1) given sufficient descriptiveness, simpler captions are better, (2) similarity of in-context images to the test image is less important when the corresponding captions to said images are poor.", "questions": "The following questions are distilled from my most significant concerns:\n\nCan the authors address the concerns surrounding presentation (W1, W2, W3, W5, W7, W8, W10, W11)? While the baselines seem thorough, they are not always clear for me to follow. Additionally, while the authors run many experiments, the presentation of these experiments is not always interpretable and it is not clear to me that the experiments actually support the conclusions that the authors draw. Some more clarity surrounding this would be greatly appreciated.\n\nIs it possible to do some experiments for the best, middle, and worst baselines on VQAv2 to see if trends are similar for another image to text task (W4)?", "limitations": "The authors acknowledge that findings for the open flamingo model tested may not translate to the more powerful, but also closed-source flamingo model.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "S1. Strong empirical investigation of how different image and text sampling strategies affect in-context image captioning performance. The paper is correctly framed and emphasizes experimentation over methodology.\n\nS2. Extensive baselines implemented for the investigation.\n\nS3. Seemingly strong performance relative to random sampling on the train set, which is traditionally used to generate in-context examples.", "weaknesses": "W1. Presentation. Consider adding some of the key findings to the abstract. 7.3 CIDEr points is significant! It may be nice to mention this in the abstract as well.\n\nW2. Presentation. Some of the images in Figure 1 are hard to follow, consider making the example more demonstrative, or making the images bigger. Also grouping the good and bad captions within the same bubble was initially confusing and it took me some time to parse what was going on.\n\nW3. Presentation. L51-59 potentially have too much detail for the introduction. Consider distilling the content. The key takeaway from this paragraphs seems to be that the authors experimented with many sampling strategies.\n\nW4. Experiments. It would be nice to see similar analysis for the VQAv2 dataset, which is also supported in the OpenFlamingo evaluation suite. One natural question is do some of the findings for image captioning translate to VQA-like tasks, which are potentially more similar to the QA tasks shown in Figure 1.\n\nW5. Presentation. Consider removing the abbreviations SIIR, DIIR, etc. from the intro. They were a bit tough for me to follow as they are not standard to the best of my knowledge and made reading the intro more difficult. In general, the many abbreviations in the paper make the writing difficult to follow.\n\nW6. Related work. Consider adding a related work section on image captioning, since this is a focus of the paper.\n\nW7. Clarity. A bit of a nit, but please specify how how the samples are randomly sampled. I am assuming uniformly at random?\n\nW8. Clarity. Can you please expand on the SIIR-TAG baseline? In DIIR, how can SIIR-TAG be applied to a collection of images (a cluster)?\n\nW9. Missing baseline. What about using models like BLIP2 for synthetic captioning (MGC)?\n\nW10. Clairity. It is hard to tell what the takeaways from Figures 3 and 4 are from glancing at them, especially because all of the abbreviations can be confusing. Consider adding they key takeaways to the figure captions. It is not currently clear to me how to read these figures. Is the takeaway that MGC is best? In general, it is not always clear how the conclusions or claims in the paper are supported numerically by the figures or tables (e.g., claims in L231-233).\n\nW11. Presentation. In some cases numbers are averaged over multiple shots and in other cases this is not the case. Why did the authors choose to do this? And why is there not consistency in this choice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Presentation Issues.**\nWe will add key findings in \"Abstract\" (W1), enlarge Figure 1 (W2), divide the good and bad captions in Figure 1 (W2), distill the contents in Lines 51-59 (W3),  and use the full names of different sampling strategies in \"Introduction\" and at the beginning of each sections to avoid confusions (W5). \n\n**2. VQA Results (W4).**\nPlease refer to **\"3. VQA Results.\"** in response to **Reviewer zeNE**.\n\n**3. Related Work about Captioning (W6).**\nDue to the space limitation, we show a short version of the related work about captioning and will expand it in the revision.\n\nImage Captioning (IC)[A] aims at correctly verbalizing one image by descriptive languages, which can be solved through both retrieve[B] and generation[A], where the former one seeks a whole sentence from a given corpus, while the latter one generates the words one by one. Researchers have recently combined these approaches by first retrieving image-caption pairs and inputting them into generation models[C,D]. This process resembles in-context captioning, which also retrieves image-caption pairs to help captioning. However, in contrast to them[C,D], our work introduces novel image and caption selection strategies to study in-context captioning and these methods can also enhance traditional retrieval-generation methods.\n\n[A] Show and tell: A neural image caption generator.\n\n[B] Probabilistic Embeddings for Cross-Modal Retrieval.\n\n[C] SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation.\n\n[D] Retrieval-augmented transformer for image captioning.\n\n**4. Randomly Sampling (W7).**\n\"Randomly Sampling\" means uniformly sampling and we will revise it.\n\n**5. SIIR-TAG (W8).**\nSIIR-TAG first calculates how many objects, attributes, and relations tags are overlapped between the images with the test image and then returns the ones which have more overlapping tags. \nFor example, given one test image containing three tags \"dog\", \"white\", \"drink\", the image A contains three tags \"cat\", \"white\", \"sit\", the image B contains three tags \"dog\", \"brown\", \"drink\". Then SIIR-TAG will return image B since it has two overlapping tags with the test one. To efficiently count how many tags are overlapping, we list all the discrete tags in a one-hot manner and then apply the \"AND\" operation. \n\nPlease refer to **\"1. DIIR in Lines141-150.\"** in the response to **Reviewer tg5h** to see details of Diversity-based Image-Image Retrieval (DIIR). Briefly, DIIR divides all the tags into different clusters. Then when calculating the ratio of the overlapped tags, only the tags in that cluster are used. This is how we use SIIR-TAG to retrieve images in each cluster. \n\n**6. BLIP2 Captions (W9).**\nWe use the official BLIP2 code from [link](https://github.com/salesforce/LAVIS/tree/main/projects/blip2) to produce model-generated captions (MGC) with a CIDEr score of 124. The subsequent table presents results using Random Sampling (RS) and CLIP Similarity-based Image-Image Retrieval (SIIR-CLIP) for image selection.\n\n| Image Selection | Caption Assignment | Mean CIDEr|\n|-----------------|--------------------|--------|\n| RS              | MGC-BLIP2@124      | 71.76 |\n| RS              | MGC-TF@88          | 76.40 |\n| SIIR-CLIP       | MGC-BLIP2@124      | 89.67 |\n| SIIR-CLIP       | MGC-TF@88          | 83.49 |\n\nFrom the table, despite BLIP2 captions achieving a CIDEr score of 124, outperforming MGC-TF@88's score of 88, it only surpasses MGC-TF@88 in the SIIR-CLIP case. This might be because BLIP2, trained on various captioning datasets, introduces styles different from COCO. When randomly sampling some images, this diversity might affect VLM's caption quality. However, BLIP2's detailed descriptions, due to its extensive training, enhance VLM's outputs when in-context images are similar to the test image. This highlights the interdependency of image and caption selection strategies.\n\n**7. Takeaways (W10).**\nWe will highlight the key takeaways in figure titles. For instance, Figure 3 suggests that regardless of the image-selection strategy employed, the use of MGCA (using model-generated captions to select a ground-truth caption as the in-context caption) consistently outperforms GTC (simply selecting the first ground-truth caption as the in-context caption). This is evident as the blue blocks surpass the grey dash lines.\n\nIn Figure 4, each sub-figure shows captions of varying quality. Across these sub-figures, six image selection strategies yield diverse rankings, e.g. SIIR-CLIP tops in (a) and (d) but performs less optimally in other sub-figures. A primary insight here is that the image selection strategy must adapt when using captions of diverse quality to ensure optimal performance.\n\n**8. Claim in Lines 231-233 (W10).**\nThe claim in Lines 231-233 (ground-truth caption has better language patterns than the model-generated captions) is not a claim observed from our experiments, but is a consensus in image captioning. We employ this consensus as a foundation for analyzing our experimental results to support our claim in Lines 227-229. We will add one footnote in the revision to avoid confusion.\n\n**9. Average or Single-Shot Scores (W11).**\nDuring analysis, in most cases, as the number of shots changes, we observe consistent conclusions, leading us to utilize average scores. For example, in Lines 259-273, MGCA consistently improves performance irrespective of the shot number.  However, in some cases, the shot count becomes pivotal, revealing interesting details. For example, as highlighted in Lines 239-240, when only scarce 4-shot in-context examples are available, simpler patterns can be better recognized by VLM for better captions, as shown in Figure 5 (b-d) where MGC-TF@88 > GTC in the 4-shot scenario. But as shot numbers increase to 16-shots, VLM can better recognize the complex patterns of GTCs, producing superior results as illustrated in Figure 5 (b-d) where GTC > MGC-TF@88 in the 16-shot scenario."}, {"review_id": "mBJGjFHnu3", "reviewer": "Reviewer_tg5h", "summary": "This paper presents a case study that examines the design of prompts and its effectiveness in generative visual language models for image captioning tasks. The authors delve into four methods for image selection and four methods for assigning captions to create contextually relevant learning samples. To evaluate different prompt design formats, the authors employed OpenFlamingo, an open-source version of the Flamingo paper. Several ablations and experiments were conducted to investigate the optimal inclusion of images and captions in the prompt of the Visual Language Model (VLM) to enhance performance. The authors put forth several findings:\n\n1. Unlike single-modal NLP cases, the performance of the model is significantly influenced by the multi-modal mutual synergy.\n\n2. The authors observe that the descriptiveness and language patterns of the captions have varying impacts on performance. In cases where selected images compensate for descriptiveness issues, better performance can be achieved by using simpler sentence patterns.\n\n3. Additionally, the authors discover that when the in-context images resemble the test image, the VLM may take a shortcut by directly utilizing the in-context captions instead of genuinely learning to generate captions.", "questions": "1. The paper could benefit from considering alternative visual language models to enhance the quality of the results. One such model is FROMAGe [1], which also utilizes a sequence of interleaved images and texts, similar to OpenFlamingo. It would be valuable for the authors to explore the use of FROMAGe in addition to OpenFlamingo and examine if the findings remain consistent across different models.\n\n2. Additionally, it is worth investigating the impact of using smaller language models within OpenFlamingo. As per the OpenFlamingo GitHub repository, there are newer versions pre-trained on smaller LLMs. The authors could conduct ablations with these models to determine how the findings vary when using models with fewer parameters. This would provide insights into the robustness of the results across different model sizes.\n\n[1]. Grounding Language Models to Images for Multimodal Inputs and Outputs, ICML 2023", "limitations": "There are not any limitations. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Generally, this paper is not novel, as there are similar papers in the NLP literature (see [1, 2]). While the authors mention [1] as one of the prior works, they appear to have overlooked [2]. \n\nIn contrast to the aforementioned prior works, I believe this paper seeks to delve into prior studies in more complex tasks, specifically open-ended generation tasks such as image captioning, and employing visual language models. In terms of quality, the paper explores a distinct aspect of the research questions and presents compelling findings that align with some prior works. Experimentally, the paper demonstrates rigor by thoroughly exploring different prompt constructions.\n\n[1]. \"Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?\" EMNLP 2022.\n\n[2]. \"What Makes Good Examples for Visual In-Context Learning?\" presented at ICML 2023.", "weaknesses": "The paper has a few notable weaknesses that can be addressed:\n\n- The writing, particularly in lines 141-150, is challenging to comprehend. It would be beneficial for the authors to rephrase or provide visual aids to clarify this section, as it contains one of the paper's ablation studies.\n- Sections 4.2.1 and 4.2.2 should be swapped, considering that in the pipeline, images are typically selected first, followed by the assignment of captions. The current ordering of the text makes it difficult to parse.\n- It would be helpful to confirm the accuracy of the statement, \"Then, as demonstrated in Figure 5(d), even MGC-TF@66 surpasses GTC.\"\n- The author's choice of abbreviations throughout the manuscript makes the reading experience challenging. It requires readers to frequently refer back and forth between different parts of the text to fully understand a paragraph. It would be advisable to use less abbreviated phrasing.\n- Including more elaborate captions for the figures would enhance their comprehensibility as standalone visual aids.\n- The paper exclusively explores one visual language model, specifically OpenFlamingo, with 9B parameters. It is important to acknowledge that the findings may not be generalizable to other models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. DIIR in Lines 141-150.**\nIn Diversity-based Image-Image Retrieval (DIIR), we first extract the discrete semantic tags including object categories, attributes, and relations from an image. Suppose there are total $M$ kinds of tags, we divide them into $N$ clusters and there are $M/N$ kinds of tags in each cluster. Then in each tag cluster, we only use the tags in that cluster to calculate the similarity score between two images, i.e., the number of overlapped tags of two images. For DIIR-TR, it randomly divides all the tags into $N$ clusters, while DIIR-TT divides the tags according to their types, e.g., one cluster only contains attribute tags and another one only contains relation tags. \n\n**2. Section 4.2.1 and 4.2.2.**\nWe will swap Section 4.2.1 and 4.2.2.\n\n**3. Accuracy Statement.**\nWe will revise this statement into \"Then, as demonstrated in Figure 5(d), in the 4-shot case, even MGC-TF@66 (the blue point) surpasses GTC (the grey point).\"\n\n**4. Abbreviations.**\nWe will use the full name at the beginning of each section and in the title of the tables and figures to avoid confusions.\n\n**5. Figure Titles.**\nWe will add some key takeaways in some figures. Please refer to **\"7. Takeaways.\"** in the response to the **Reviewer vuiF** to see two examples about Figure 3 and 4.\n\n**6. Experiments on FROMAGe.**\nSince FROMAGe does not provide codes for in-context learning (ICL) and lacks quantitative analyses in the paper, such as computing CIDEr scores, we modify parts of its code originally designed for multi-turn chat. We adopt a prompt structure like \"<img1>caption1<img2>caption2...<test img>\" for in-context learning. The test results are presented below. \n| Image Selection | Caption Assignment | 4-shot| 8-shot|\n|-----------------|--------------------|--------|--------|\n| RS              | GTC     | 2.50 | 3.11 |\n\nWe can find that it achieves a very low CIDEr score.\nThese observations are consistent with the analysis in the FROMAGe paper whose Figure.7 shows that when provided with 5-shot inputs, FROMAGe's output is more story-like and less relevant to the target image. Thus FROMAGe does not support the study about in-context configurations. \n\n**7. Experiments on Smaller Open-flamingo and Otter.**\nWe test our methods in another two VLMs: [Open-Flamingo 3B model](https://github.com/mlfoundations/open_flamingo) and [Otter](https://github.com/Luodian/Otter), which are proposed **after the NeurIPS submission deadline**. However, due to the computational limitations and the tight rebuttal timeframe, we employ parts of experiments to validate the robustness of our two key conclusions shown in Lines 227-229 and 302-306. \n\nWe first show the results about the first conclusion, where the top and bottom tables are from Open-Flamingo 3B model and Otter, respectively.\n| Image Selection | Caption Assignment | Mean CIDEr|\n|-----------------|--------------------|------|\n| RS              | GTC                | 90.31 |\n| RS              | MGC-TF@135         | 88.47|\n| SIIR-CLIP       | GTC                | 94.36 |\n| SIIR-CLIP       | MGC-TF@135         | 103.56|\n\n| Image Selection | Caption Assignment | Mean CIDEr|\n|-----------------|--------------------|------|\n| RS              | GTC                | 88.36 |\n| RS              | MGC-TF@135         | 80.77 |\n| SIIR-CLIP       | GTC                | 90.72 |\n| SIIR-CLIP       | MGC-TF@135         | 97.37 |\n\nwhere RS and SIIR-CLIP denote randomly sampling and clip Similarity-based Image-Image retrieval, GTC uses the first ground-truth caption among 5 human-labelled ones, and MGC-TF@135 uses the model-generated captions. We can find that in both two Tables, when RS is used to select images, GTC outperforms MGC-TF@135. For example, in the smaller open-flamingo model (the top one), RS-GTC achieves 90.31, which is higher than RS-MGC-TF@135's 88.47. However, when selecting similar images as in-context images, i.e., using SIIR-CLIP, MGC-TF@135 achieves higher performances, e.g., in the last two rows of the bottom table, MGC-TF@135 achieves 97.37, which is larger than GTC's 90.72. These comparisons in both two tables are consistent with the findings in Lines 230-238, which confirm the robustness of our conclusion given in Lines 227-229 that when the visual cues are abundant, the consistent and simple patterns of MGC has more advantages. \n\nThen we show the results about the second conclusion that similar images lead to short-cut inference, where the top and bottom ones show the results from the Open-Flamingo 3B model and Otter, respectively.\n|   | In-Context Images | GTC CIDEr | ICC  CIDEr |\n|---|-------------------|------------------|------------------|\n|(1)| Test Image        | 38.50             | 150.70            |\n|(2)| SIIR-CLIP         | 55.98             | 102.70            |\n|(3)| RS                | 70.72             | 43.50             |\n\n|   | In-Context Images | GTC CIDEr | ICC  CIDEr |\n|---|-------------------|------------------|------------------|\n|(1)| Test Image        | 66.40             | 37.90          |\n|(2)| SIIR-CLIP         | 73.04             | 27.90         |\n|(3)| RS                | 84.32             | 4.80           |\n\nFrom (1) to (3), we respectively use (1) the test image, (2) the retrieved similar images as the test image, (3) randomly sampled images as the in-context images, where the similarity to the test image from method (1) to (3) declines. Then we observe that in both tables, from (1) to (3), the CIDEr scores diverge more from in-context captions (ICC) but converge towards ground-truth captions (GTC). These results are consistent with the findings in Lines 315-319, and thus the conclusion \"similar images lead to short-cut inference\" still validates in smaller open-flamingo 3B model and Otter.\n\n**8. Limitation.**\nWe acknowledge that the findings may not be generalizable to other models in Lines 340-346."}]

Author rebuttal:

Meta-review: This work received unanimously positive rates. The final version may need to involve the rebuttal.
Decision: Accept (poster)