SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This effect is more pronounced in 4-shot cases, where VLM lacks sufficient in-context captions to discern sentence patterns, making simpler patterns advantageous. As long as the descriptiveness issue is addressed, MGC often outperforms GTC, e.g., MGC-TF@88>GTC in Figure 3(b-d). Especially, when SICR-CLIP is employed to select captions with high cross-modal similarity to the test image, it significantly mitigates the descriptiveness problem. Then as demonstrated in Figure 3(d), even MGC-TF@66 surpasses GTC.
MGC-TF vs. MGC-VLM. Before we see that using more low-quality captions like MGC-TF@66 will misguide VLM to generate worse captions. Yet, Figure 4 (f) indicates that using MGC-VLM(0) $@63$  improves performance with increased shot numbers, contrasting MGC-TF@66 in Figure 4 (c). This raises the question: why do weaker captions (MGC-VLM(0) $@63$ ) surpass those
(b) MGC-VLM(0) $@$ 63(blue) v.s. MGC-TF@66(red)
Figure 7: (a) Two examples show that GTC uses more diverse words than MGC-TF@135, making VLM hard to recognize the major pattern, e.g., it mis-generates "a store" in left or neglects "a fork" in right. (b) Two examples demonstrating how VLM is misguided by syntactic errors in MGC-VLM@66, where certain phrases are repeated such as "a street" or "a boat in the water".
with higher CIDEr (MGC-TF@66)? This discrepancy aligns with our assumption: descriptiveness and language pattern influence in-context captioning differently.
MGC-TF@66 excels in object detection but struggles with language decoding, causing salient objects to be identified correctly but with syntactical errors in captions. As such examples increase, VLM produces worse captions due to these errors. Conversely, MGC-VLM(0)@63, though limited in object recognition, maintains better grammar. When more vision cues are provided, VLM leverages these along with the better-formed captions from MGC-VLM(0)@63, resulting in improved captions. Figure 7 (b) offers two comparison examples.
Model-Generated Captions as Anchors (MGCA). From Figure 5, we see that using MGC as the in-context captions usually underperforms GTC. However, by the MGCA strategy, we observe consistent improvements over GTCs, as demonstrated by the higher blue histograms of different MGCs compared to the grey dashed line. For example, despite MGC-TF@66 only achieves 64.24 CIDEr score in Figure 5(a), we still observe a 3.2 CIDEr improvement when using MGC-TF@66 as the anchor, compared to simply selecting the first GTC. Specifically, when using MGC-TF@66/MGC-TF@88/MGC-TF@135/MGC-VLM(0)@63/MGC-VLM(32)@81 as the anchors, the average improvements over six image selection strategies compared to GTC are 7.3/8.0/8.8/3.6/4.8 respectively. In contrast to solely leveraging the RS+GTC method, the combination of SIIR-CLIP and using MGC-TF@135 as anchor chalked up an average boost of 20.9.
The primary reason for such improvement is likely that MGC, to some extent, verbalize the major patterns, such as the salient objects of an image. This helps identify which GTC provides more detailed information about these patterns. Such detailed information of the salient objects provide help VLM generate better captions. This assumption is further supported by comparisons between MGC-TF and MGC-VLM. Given that MGC-TF prioritizes verbalizing the salient objects of an image, using MGC-TF@66/MGC-TF@88 as anchors tends to select superior GTC than MGC-VLM(0)@63/MGC-VLM(32)@81, thus yielding higher improvements.
Iteratively Prompting (IP). Table 1 showcases the IP CIDEr scores. "Iter 1" represents either the 0 or 32-shot performance, with subsequent columns reflecting average CIDEr scores for 4, 8, 16, and 32-shot scenarios. In the inaugural iteration, we employ a consistent set of 0 or 32-shot image-caption pairs. For the following iterations, images are selected via the Random Sampling (RS) ap
proach, and captions generated in the prior iteration serve as the in-context captions. Analyzing the data, it's evident that MGC-VLM(0) stabilizes by the third iteration and MGC-VLM(32) by
Iter	1	2	3	4	5
MGC-VLM(0)	63.0	74.1	79.9	79.3	77.3
MGC-VLM(32)	85.3	80.5	79.4	78.9	77.1
Table 1: The CIDEr scores of IP in different iterations.
A long table with a plant on top of it surrounded with wooden chairs.
(a) Experiment (1)
A stop sign on a road.
A long table with. . . . . bananas on it.
A long table with a plant on top of it surrounded with wooden chairs.
Figure 8: Four examples illustrate the phenomenon of short-cut inference. (a) When the in-context images are identical to the test image, the generated caption mirrors the in-context captions. (b) When using SIIR-CLIP to select similar examples, the generated caption tends to amalgamate features from both the in-context and test images, sometimes leading to ambiguous or partially accurate descriptions. (c) In contrast, when the in-context images are distinct from the test image, the generated caption more aptly describes the image, including specific words such as "car" or "bananas".
(b)Experiment (2)
A car with a sign that says stop.
(c) Experiment (3)
Bananas on a wooden table.
the second, indicating that extended VLM iterations might be redundant. Remarkably, even when confined to just 32-shot GTCs, merely two iterations of IP — for instance, where MGC-VLM(32) attains an average CIDEr of 80.5 — can rival performances seen when all GTCs are utilized, as exemplified by RS-GTC's average CIDEr of 80.04 in Figure 5(a).
4.2.2 Effects of Image Qualities
We evaluated the outcomes of several image selection techniques. SIIR-CLIP, leveraging vision embeddings to compute retrieval similarities, generally identifies images that are more analogous than those found by SIIR-TAG. This is likely attributed to the intrinsic noise present in semantic tags. SICR-CLIP, emphasizing captions that spotlight prominent objects, naturally gravitates towards images showcasing similar objects. In contrast, both DIIR-TR and DIIR-TT produce more varied selections. Nevertheless, when benchmarked against RS, every retrieval-based model consistently fetches images that exhibit greater similarity.
At first glance, one could reasonably infer that using more similar images would invariably lead to superior performance. Yet, as demonstrated in Figure 6, this assumption doesn't universally hold. When engaging high-quality captions, namely GTC in (a) and MGC-TF@135 in (d), the correlation stands, with more analogous images indeed translating to enhanced results. For instance, all retrieval-based techniques surpass RS in this context. However, with medium-quality captions, as in MGC-TF@88 in (b) and VLM(32) $@81$  in (e), only the similarity-based retrieval methods like SIIR or SICR-CLIP manage to outdo RS, and this is specifically observed in (e). Most intriguingly, when low-quality captions like MGC-TF@66 in (c) and VLM(32) $@63$  in (f) are in play, analogous images inversely impact performance, as illustrated by RS outperforming SIIR-CLIP in (f). This critical insight underscores a pivotal revelation: the efficacy of utilizing similar images is intricately tied to the caliber of the corresponding captions.
Similar Images Lead to Short-Cut Inference. A relatively bold hypothesis to elucidate this phenomenon is: when in-context images are similar to the test image, VLM may take a short-cut by leveraging in-context captions to generate a new one, rather than learning how to caption from the in-context pairs. Consequently, the greater the similarity between in-context and test images, the more the VLM is influenced by the in-context captions.
To robustly test our underlying assumption, we designed three distinct experimental setups. In each setup, the in-context captions remain consistent, comprising 5 ground-truth captions sourced randomly from an image unrelated to the test image. However, the in-context images are chosen differently in each experiment: (1) they are identical to the test image; (2) they are picked using SIIR-CLIP; (3)
In-Context Images	GTC CIDEr	ICC CIDEr
(1)	Test Image	6.7	192.2
(2)	SIIR-CLIP	12.8	180.7
(3)	RS	58.5	54.8
Table 2: The results for verifying shortcut inference.
they are chosen via RS. This progression results in decreasing similarity between the in-context and test images from experiments (1) through (3). Table 2 showcases two sets of CIDEr scores. One set compares the generated captions to the five ground-truth captions (GTC) of the test image, and the other set contrasts them with the five in-context captions (ICC). Our findings suggest a clear pattern: the closer the in-context images are to the test image, the more the VLM tends to mirror the ICC in its generated caption. For illustration, method (1) registers the highest CIDEr score when compared to the ICC. However, the caption it produces doesn't accurately depict the image, as reflected by its notably lower CIDEr score in relation to the GTC. These observations solidify our hypothesis that images with high similarity can inadvertently prompt short-cut inference. Additionally, Figure 8 provides visual representations to further elucidate the phenomenon of short-cut inference.
DIIR. As depicted in Figure 6, the performance of the two DIIR methods noticeably lags behind SIIR-TAG. This observation underscores the contention that the strength of diversity may not always translate to superior results in every context. One plausible explanation we propose is the inherent nature of captioning as a task. Contrary to certain complex NLP challenges, where diversity can be instrumental in offering a multifaceted understanding of a problem [57], captioning is relatively straightforward and may not benefit as much from diverse in-context examples. Delving deeper into the DIIR methods, DIIR-TT consistently outshines DIIR-TR. This leads us to infer that the clustering based on semantic tag types might be a more optimal strategy. Such a strategy, we believe, not only ensures diversity but also completeness in the selection of images. For instance, given a caption like "a brown dog is running", DIIR-TT could potentially source images that highlight elements like "brown objects", "dogs", and the "action of running".
In this study, we utilize image captioning as a case study to examine the impacts of varied multi-modal in-context configurations on few-shot vision-language prompting. Specifically, we design 4 different ways to select images and 4 different strategies to assign captions to the selected images for constructing multi-modal in-context configurations. Exhaustive experiments reveal that, contrary to single-modal NLP cases, multi-modal mutual synergy significantly influences performance. Notably, we observe that the descriptiveness and language patterns of the captions differently affect performance: better performance may be achieved with simpler and consistent sentence patterns when selected images compensate for descriptiveness issues. And we also discover that when the in-context images are similar to the test one, VLM may build a short-cut by directly using the in-context captions instead of really learning to captioning. Moreover, our optimal strategy lead to a 20.9 average increase in CIDEr scores compared to a random sampling baseline.
This study's primary limitation is that at the inception of our exploration, the only open-source multi-modal few-shot learner available is Open-Flamingo [16]. However, Open-Flamingo, when compared with the official Flamingo [6], underperforms due to its training on significantly less data. Consequently, some findings in this paper might shift if a more robust multi-modal few-shot learner is employed. Nevertheless, even in such a scenario, the diverse configuration strategies proposed in this paper maintain their utility, aiding researchers in swiftly exploring the characteristics of the employed VLMs. Additionally, we have provided experimental results on Otter [54] and smaller version of Open-Flamingo, with detailed findings available in Appendix.
This work is supported by National Science Foundation of China (62206048), Natural Science Foundation of Jiangsu Province (BK20220819), Young Elite Scientists Sponsorship Program of Jiangsu Association for Science and Technology Tj-2022-027 and the Big Data Computing Center of Southeast University.
A Experimental results on Open-Flamingo v1 9B
Here we present all the numerical results from the experiment, divided into four tables (Table 3, Table 4, Table 5, and Table 6) based on different Image Selection strategies. Additionally, we have calculated the average values for each method in the "mean" column.
Image Selection	Caption Assignment	4-shot	8-shot	16-shot	32 -shot	mean
RS	GTC	75.80	78.93	81.97	83.47	80.04
RS	MGC-TF@66	69.18	64.59	62.73	60.46	64.24
RS	MGCA-TF@66	78.52	82.44	85.48	86.53	83.24
RS	MGC-TF@88	74.60	74.47	77.80	78.71	76.39
RS	MGCA-TF@88	78.40	81.28	84.93	87.62	83.06
RS	MGC-TF@135	72.35	70.10	72.73	77.76	73.23
RS	MGCA-TF@135	78.81	80.64	83.86	86.74	82.51
RS	MGC-VLM(0)@63	70.45	73.92	74.83	77.00	74.05
RS	MGCA-VLM(0)@63	76.13	79.61	82.14	83.63	80.38
RS	MGC-VLM(32)@85	78.33	80.94	82.62	82.75	81.16
RS	MGCA-VLM(32)@85	77.22	80.16	82.97	85.01	81.34
Table 3: Using Random Selection (RS) image selection strategy results.
Image Selection	Caption Assignment	4-shot	8-shot	16-shot	32 -shot	mean
SIIR-CLIP	GTC	80.32	88.76	95.18	98.24	90.62
SIIR-CLIP	MGC-TF@66	69.08	66.78	65.79	65.51	66.79
SIIR-CLIP	MGCA-TF@66	89.69	97.53	102.31	104.63	98.54
SIIR-CLIP	MGC-TF@88	81.86	82.43	84.77	84.89	83.49
SIIR-CLIP	MGCA-TF@88	91.11	98.23	102.74	106.24	99.58
SIIR-CLIP	MGC-TF@135	95.64	96.62	97.66	98.32	97.06
SIIR-CLIP	MGCA-TF@135	92.73	99.47	104.09	107.52	100.95
SIIR-CLIP	MGC-VLM(0)@63	65.98	69.52	71.88	73.49	70.22
SIIR-CLIP	MGCA-VLM(0)@63	86.72	93.30	98.42	100.85	94.82
SIIR-CLIP	MGC-VLM(32)@85	79.46	81.34	84.14	84.86	82.45
SIIR-CLIP	MGCA-VLM(32)@85	88.09	95.70	98.98	102.12	96.22
SIIR-TAGS	GTC	78.37	86.40	92.08	94.94	87.95
SIIR-TAGS	MGC-TF@66	71.23	68.27	66.77	65.12	67.85
SIIR-TAGS	MGCA-TF@66	87.40	93.31	97.85	102.16	95.18
SIIR-TAGS	MGC-TF@88	82.70	81.80	82.51	83.33	82.58
SIIR-TAGS	MGCA-TF@88	87.86	94.26	98.69	102.64	95.86
SIIR-TAGS	MGC-TF@135	89.35	91.14	92.73	94.86	92.02
SIIR-TAGS	MGCA-TF@135	88.89	94.45	100.34	104.44	97.03
SIIR-TAGS	MGC-VLM(0)@63	67.11	70.21	72.21	74.45	71.00
SIIR-TAGS	MGCA-VLM(0)@63	83.23	89.35	94.51	98.79	91.47
SIIR-TAGS	MGCA-VLM(32)@85	79.69	81.96	82.82	84.25	82.18
SIIR-TAGS	MGCA-VLM(32)@85	85.04	91.23	96.10	100.10	93.12
Table 4: Using Similarity-based Image-Caption Retrieval (SIIR-CLIP) image selection strategy results.
Image Selection	Caption Assignment	4-shot	8-shot	16-shot	32 -shot	mean
SICR-CLIP	GTC	72.91	81.76	89.40	94.51	84.64
SICR-CLIP	MGC-TF@66	76.03	74.66	74.73	73.75	74.79
SICR-CLIP	MGCA-TF@66	87.50	95.67	101.56	104.42	97.29
SICR-CLIP	MGC-TF@88	84.30	84.09	85.34	84.83	84.64
SICR-CLIP	MGCA-TF@88	88.41	96.37	101.89	105.07	97.94
SICR-CLIP	MGC-TF@135	92.60	93.69	94.64	95.30	94.06
SICR-CLIP	MGCA-TF@135	91.19	98.41	103.24	107.24	100.02
SICR-CLIP	MGC-VLM(0)@63	68.19	70.34	73.97	76.91	72.35
SICR-CLIP	MGCA-VLM(0)@63	82.49	91.08	96.64	101.55	92.94
SICR-CLIP	MGC-VLM(32)@85	82.39	85.02	87.22	87.08	85.43
SICR-CLIP	MGCA-VLM(32)@85	84.18	91.52	98.02	102.89	94.15
Table 5: Using Similarity-based Image-Caption Retrieval (SICR) image selection strategy results.
Image Selection	Caption Assignment	4-shot	8-shot	16-shot	32 -shot	mean
DIIR-TR	GTC	76.01	82.88	87.13	93.25	84.82
DIIR-TR	MGC-TF@66	71.45	66.82	66.22	65.80	67.57
DIIR-TR	MGCA-TF@66	82.27	87.45	93.56	97.49	90.19
DIIR-TR	MGC-TF@88	81.03	79.50	81.88	83.06	81.37
DIIR-TR	MGCA-TF@88	82.85	88.84	93.93	99.18	91.20
DIIR-TR	MGCA-TF@135	83.59	82.63	84.33	89.03	84.89
DIIR-TR	MGCA-TF@135	83.44	88.83	94.26	98.23	91.19
DIIR-TR	MGC-VLM(0)@63	67.64	70.91	73.21	74.86	71.66
DIIR-TR	MGCA-VLM(0)@63	78.68	84.41	89.96	93.59	86.66
DIIR-TR	MGC-VLM(32)@85	77.35	80.72	81.97	83.72	80.94
DIIR-TR	MGCA-VLM(32)@85	79.88	85.40	89.59	94.62	87.37
DIIR-TT	GTC	75.65	83.21	89.70	95.42	86.00
DIIR-TT	MGC-TF@66	72.59	68.51	66.66	65.53	68.32
DIIR-TT	MGCA-TF@66	84.88	91.36	96.81	100.73	93.44
DIIR-TT	MGC-TF@88	81.94	81.76	82.19	83.80	82.42
DIIR-TT	MGCA-TF@88	85.13	92.61	97.35	101.61	94.17
DIIR-TT	MGC-TF@135	88.01	88.65	90.58	94.04	90.32
DIIR-TT	MGCA-TF@135	86.77	93.25	97.92	102.97	95.23
DIIR-TT	MGC-VLM(0)@63	66.42	69.64	72.71	74.56	70.83
DIIR-TT	MGCA-VLM(0)@63	79.32	87.36	92.76	97.69	89.28
DIIR-TT	MGCA-VLM(32)@85	76.91	80.21	83.24	84.43	81.20
DIIR-TT	MGCA-VLM(32)@85	81.39	87.83	93.64	98.61	90.37
Table 6: Using Diversity-based Image-Image Retrieval (DIIR) image selection strategy results.
B Experimental results on Open-Flamingo v2 3B
Here we present the numerical results on Open-Flamingo v2 3B model in Table 7 based on two different Image Selection strategies. Additionally, we have calculated the average values for each method in the "mean" column.
From the values in the table, it can be seen that the trend is basically consistent with the v1 model. It can be considered that our strategy and analysis can be transferred to different models.
C Experimental results on Otter
Here we present the numerical results on Otter model in Table 8 based on two different Image Selection strategies. Additionally, we have calculated the average values for each method in the "mean" column.
3Project: https://laion.ai/blog/open-flamingo-v2/
Image Selection	Caption Assignment	4-shot	8-shot	16-shot	32 -shot	mean
RS	GT	77.90	86.14	90.80	93.17	88.47
RS	MGC-TF@66	82.16	86.49	88.16	86.61	86.55
RS	MGC-TF@88	82.47	88.43	91.81	93.83	90.12
RS	MGC-TF@135	84.64	89.93	90.70	92.03	90.31
RS	MGCA-TF@66	77.95	87.64	92.54	95.14	90.09
RS	MGCA-TF@88	78.95	88.21	92.17	95.28	90.19
RS	MGCA-TF@135	79.61	87.99	91.92	95.22	89.95
SIIR	GT	84.53	92.36	96.36	98.66	94.36
SIIR	MGC-TF@66	84.72	77.69	75.29	76.49	77.09
SIIR	MGC-TF@88	93.38	91.29	90.50	89.25	90.90
SIIR	MGC-TF@135	104.10	103.80	103.31	102.39	103.56
SIIR	MGCA-TF@66	90.00	97.51	102.04	103.50	99.78
SIIR	MGCA-TF@88	91.15	98.88	102.17	104.55	100.53
SIIR	MGCA-TF@135	91.91	99.01	103.26	104.93	101.14
Table 7: Open-Flamingo v2 3B results with various strategies.
Image Selection	Caption Assignment	4-shot	8-shot	16-shot	32 -shot	mean
RS	GT	83.43	88.36	91.86	93.09	90.11
RS	MGC-TF@135	75.49	80.77	85.46	89.53	83.115
RS	MGCA-TF@135	84.67	89.6	92.45	93.97	91.025
SIIR	GT	87.4	90.72	94.42	95.94	92.57
SIIR	MGC-TF@135	95.02	97.37	98.8	99.88	98.085
SIIR	MGCA-TF@135	90.93	96.59	97.08	101.3	96.835
Table 8: Otter results with various strategies.
D More Results of MGC-TF@135 vs. GTC
We further elucidate the performance of MGC-TF@135(blue) and GTC(red), by offering additional examples shown in Figure 9. It can be easily observed that GTC has a more diverse range of sentence structures in comparison to MGC-TF@135. In the initial two examples, the words "cat" and "dachshunds" were inaccurately recognized by GTC, demonstrating its limitation in some specific instances. A shift can be noticed in the subsequent three examples, where the GTC generates captions with complex sentence structures, with a notable proportion of incorrect information. This finding serves to reinforce the notion that simpler sentence patterns, up to a certain degree of descriptiveness, are more readily deciphered by the VLM, thereby enhancing the quality of generated captions.
E More Results of MGC-TF@66 vs. MGC-VLM(0)@63
Similarly, we supplement with more examples, as depicted in Figure 10, to facilitate the comparison between MGC-TF@66(red) and MGC-VLM(0) $@63$  (blue). A striking observation from this comparative study reveals that MGC-TF@66 demonstrates a significant number of syntactical errors. This flaw becomes problematic as it misdirects VLM, resulting in a substantial volume of syntactical mistakes in the produced sentences. This correlation implies that the syntax errors in the initial input by MGC-TF@66 tend to propagate into the VLM's output. Therefore, it becomes clear that the accuracy of grammar in the prompt is crucial for achieving better results.
Figure 9: MGC-TF@135(blue) vs. GTC(red). Five examples demonstrate that more diverse words will be used in GTC than in MGC-TF@135 which making VLM hard to catch the major pattern. e.g., the first line and the second line incorrectly state "cat" and "dachshunds."
a vase filled with purple flowers.	A vase made out of a newspaper.	A blue vase with a purple flower in it.	a glass vase with orange flowers.	A vase filled with orange flowers.
a vase with a vase and a vase.	a vase with a vase and a vase.	a vase with a vase and a vase.	a vase with a vase and flowers on it.	a vase with a vase in a vase and a vase.
Figure 10: MGC-VLM(0)@63(blue) vs. MGC-TF@66(red). Five examples show that VLM will be misguided by the syntactic errors in MGC-VLM@66. Lots of outputs have repeated phrases, such as "a table", "a clock", "a desk", "a motorcycle" and "a vase".
F More Results of short-cut inference
In this section, we provide additional examples to demonstrate how similar images can lead our VLM to exhibit a tendency for short-cut inference, where it might relies heavily on in-context captions to generate the captions, disregarding the image information in the prompt and the inherent relationships within the in-context pairs. In Figure 11, we present the results for the same test image under two different scenarios of choosing in-context images: "identical to the test image" (top row) and "via random sampling" (bottom row).
In the first scenario, for the majority of cases, our results are largely unrelated to the image content but similar to the in-context captions, with only a few instances capturing some elements from the image, such as "pink" in (a) and "on the beach" in (b). However, in the second scenario, our results are heavily influenced by the in-context captions, as evident in (c) and (d) with the mention of "wooden chairs".
Figure 11: Five examples of verifying short-cut inference. we provide two different ways of choosing in-context images: "identical to the test image" (top row) and "via random sampling" (bottom row).