Title: Exploring Diverse In-Context Configurations for Image Captioning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Image Caption; Few-shot Prompt; Vision Language Model;
Abstract: After discovering that Language Models (LMs) can be good in-context few-shot learners, numerous strategies have been proposed to optimize in-context sequence configurations. Recently, researchers in Vision-Language (VL) domains also develop their few-shot learners, while they only use the simplest way, \ie, randomly sampling, to configure in-context image-text pairs. In order to explore the effects of varying configurations on VL in-context learning, we devised four strategies for image selection and four for caption assignment to configure in-context image-text pairs for image captioning. Here Image Captioning is used as the case study since it can be seen as the visually-conditioned LM. Our comprehensive experiments yield two counter-intuitive but valuable insights, highlighting the distinct characteristics of VL in-context learning due to multi-modal synergy, as compared to the NLP case. Furthermore, in our exploration of optimal combination strategies, we observed an average performance enhancement of 20.9 in CIDEr scores compared to the baseline. The code is given in https://github.com/yongliang-wu/ExploreCfg.

Exploring Diverse In-Context Configurations for Image Captioning

Abstract

After discovering that Language Models (LMs) can be good in-context few-shot learners, numerous strategies have been proposed to optimize in-context sequence configurations. Recently, researchers in Vision-Language (VL) domains also develop their few-shot learners, while they only use the simplest way, i . e . , randomly sampling, to configure in-context image-text pairs. In order to explore the effects of varying configurations on VL in-context learning, we devised four strategies for image selection and four for caption assignment to configure in-context image-text pairs for image captioning. Here Image Captioning is used as the case study since it can be seen as the visually-conditioned LM. Our comprehensive experiments yield two counter-intuitive but valuable insights, highlighting the distinct characteristics of VL in-context learning due to multi-modal synergy, as compared to the NLP case. Furthermore, in our exploration of optimal combination strategies, we observed an average performance enhancement of 20.9 in CIDEr scores compared to the baseline. The code is given in https://github.com/yongliang-wu/ExploreCfg .

1 Introduction

In contemporary times, the Language Model (LM) [1; 2] has emerged as a pivotal player in the field of Natural Language Processing (NLP). It accomplishes this by unifying a range of diverse NLP tasks into a shared prompt paradigm [3; 4]. To elaborate, a LM, unsupervised trained via conditional language modeling on a substantial volume of non-annotated data collected from the web, can reformulate various downstream tasks into fitting textual prompts. These prompts contain slots, the word probabilities of which are calculated using the pre-trained LM. This process obviates the necessity for gradient updates to the parameters of LM, thus mitigating the challenges associated with additional data collection and fine-tuning. To further enhance the effectiveness of this paradigm, the few-shot prompt (or in-context learning) [5] is proposed where a few examples are provided as additional contexts to guide the LM to generate the desired result.

Witnessing the success of the prompt paradigm in the NLP field, there has been a concerted effort by researchers to replicate its function in the Vision-Language Model (VLM). To facilitate this, Flamingo [6] is proposed to align well-trained large-scale vision and language models through some trainable cross-modal adapters. Consequently, the resultant model is capable of addressing Vision-Language (VL) tasks by processing a prompt sequence, which includes several interleaved image and text examples for in-context learning. Since the primary objective of Flamingo is to build a VLM for the few-shot prompt, they only apply a straightforward strategy to configure the in-context

VLM

Q

: Which country is

A

: India.

Q

: Which country is Yao

Ming from?

A

:

Q

: What do dogs like to

eat?

A

: Bone.

Q

: What do cats like to

eat?

A

: Fish.

Q

: Which country is Yao

Ming from?

A

:

Q

: Which country is

A

: America.

Q

: Which country is

A

: India.

Q

: Which country is Yao

Ming from?

A

:

Q

: What do dogs like to

eat?

A

: Bone.

Q

: What do cats like to

eat?

A

: Fish.

Q

: Which country is Yao

Ming from?

A

:

Q

: Which country is

A

: America.

'

'

×

LM

'

China

'

√

(a)

(b)

'

'

×

LM

'

China

'

√

(a)

(b)

A women riding a bike on

the bike path.

[Good Caption]

People riding on a street

on a street.

[Bad Caption]

People walking down a

street and a street.

[Bad Caption]

The man is riding a pink

bike threw traffic.

[Good Caption]

A women riding a bike on

the bike path.

[Good Caption]

People riding on a street

on a street.

[Bad Caption]

People walking down a

street and a street.

[Bad Caption]

The man is riding a pink

bike threw traffic.

[Good Caption]

A man riding a bike in a street a

street.

[CIDEr 61.4]

A cyclist riding in the bike lane

of a city street.       [CIDEr 164.6]

(c)

(d)

A man riding a bike in a street a

street.

[CIDEr 61.4]

A cyclist riding in the bike lane

of a city street.       [CIDEr 164.6]

(c)

(d)

VLM

People ride on elephants

through a field.

[Good Caption]

Some elephants are

walking and walking.

[Bad Caption]

A kitchen with a stove

and a stove.

[Bad Caption]

A kitchen scene with a

fridge and an oven.

[Good Caption]

People ride on elephants

through a field.

[Good Caption]

Some elephants are

walking and walking.

[Bad Caption]

A kitchen with a stove

and a stove.

[Bad Caption]

A kitchen scene with a

fridge and an oven.

[Good Caption]

A group of people riding on a

city street.              [CIDEr 72.9]

A bicycler riding in the bike lane

next to traffic.        [CIDEr 126.8]

(e)

(f)

A group of people riding on a

city street.              [CIDEr 72.9]

A bicycler riding in the bike lane

next to traffic.        [CIDEr 126.8]

(e)

(f)

sequence by randomly sampling a few image-text pairs. Nevertheless, a plethora of studies within the NLP field [7] have demonstrated that diverse in-context configurations lead to dramatic effects on few-shot performance, e . g ., the selection or ordering of in-context samples [8; 9; 10], while only a limited number of studies systematically explore such effects in the VL case.

To narrow this gap, we explore the effects of various in-context configurations on the performance of few-shot VL tasks. Among various VL tasks, Image Captioning (IC) aims at generating a text conditioned on the source image, and thus can be considered as the visually-conditioned LM. Just as a multitude of NLP tasks can be recast as LM tasks, IC performs a similar function [11; 12; 13], which motivates our decision to select IC as the subject of our case study. However, unlike in NLP, where only single-modal texts are considered in the in-context configuration, in IC, the synergy between multiple modalities significantly influences the performance. For instance, our experiments revealed that selecting images similar to the test image for the in-context sequence does not always lead to good performance, a result closely tied to the quality of the associated captions. Figure 1 shows the comparison between LM and VLM as the few-shot learners.

Consequently, we design diverse ways to select images as the in-context images. After selection, the captions of different qualities are assigned to these images for constructing the multi-modal in-context sequence. By combining diverse image selection and caption assignment techniques, we undertake a comprehensive exploration of the effects of multi-modal mutual synergy on VL in-context captioning. We implement all experiments using the prevalent captioning dataset, MSCOCO [14], employing its training set as the database for image selection.

To select images, we use 4 different ways, which are Random Sampling, Similarity-based ImageImage Retrieval, Similarity-based Image-Caption Retrieval, and Diversity-based Image-Image Retrieval. Following image selection, various types of captions produced by 4 different strategies are assigned, which are Ground-Truth Captions, Model-Generated Captions, Iteratively Prompting, and Model-Generated Captions as Anchors.

Through extensive evaluation of various image selection and caption assignment strategies, we uncover two counter-intuitive yet valuable insights. (1) Caption quality is determined by descriptiveness and language patterns, but their influence on in-context captioning performance is unequal. When captions adequately describe salient image objects, simpler language patterns may yield better results. (2) The efficacy of similar images depends on the quality of the paired captions. Excessive similarity might cause VLM to create a short-cut inference [15] from in-context captions, potentially misleading the model with low-quality captions. Beyond these findings, we introduce a practical in-context captioning strategy, Iterative Prompting, for cases with limited or no Ground-Truth Captions. Furthermore, when Ground-Truth Captions are available, we recommend using Model-Generated Captions as anchors to identify which Ground-Truth Caption is a more suitable in-context caption. Experimental results indicate that even when utilizing low-quality model-generated captions, there is an average CIDEr improvement of 7.3. Moreover, in optimal conditions, the average enhancement reaches up to 20.9 points compared to the random sampling baseline.

Since the Flamingo code is not publicly available, our experiments mainly utilize the unofficial Open-Flamingo [16] 2 . It's worth noting that the performance of Open-Flamingo is not on par with the official Flamingo due to its training on less data.

2 Related Work

Prompting Language Model (LM). The research paradigms of NLP have encountered two sea changes in the past few years [3]. The first one is the LM that is pre-trained by predicting the next word conditioned on observed words and can be fine-tuned for solving various downstream tasks, including GPT [17], BERT [1], and BART [18]. The second sea change is the emergence of the prompt paradigm, which was introduced with GPT-3 [2]. Within this paradigm, a pre-trained LM does not require fine-tuning to solve downstream tasks; instead, tasks are reformulated into appropriate prompts with empty slots to be filled. Subsequently, more advanced prompt-based techniques have been proposed, including prompt-tuning [19; 20; 21] and Chain-of-Thought [22; 23; 24; 25].

Prompting Vision-Language Model (VLM). In contrast to NLP, the Vision-Language (VL) domain has made significant strides in the first sea change, as evident by the development of various VLBERT models. These models leverage large volumes of web-collected image-caption pairs to learn VL-generalizable embeddings [26; 27; 28; 29; 30; 31; 32; 33]. However, the prompt paradigm, despite revolutionizing NLP studies, only appears when a certain scale of the model is reached [34]. This scale prerequisite poses further challenges to the development of a VLM with prompt and in-context learning ability.

To mitigate training burdens, instead of updating all the parameters of a VLM [35; 36], some VLMs [35; 36; 37] freeze well-trained Language Models and only train a smaller network, referred to as adapters [38; 39; 40], to align the pre-trained vision and language models. Inspired by these models, both Frozen [41] and Flamingo [6] evolve into multi-modal few-shot learners by training vision and cross-modal adapters, respectively. Given its superior in-context learning ability, we use Flamingo to explore the effects of various in-context configurations [16].

There are also models that address VL tasks through in-context learning. For instance, PICa [11] utilizes captions as mediators to construct an in-context text for solving Visual Question Answering (VQA) tasks, while this may lose the mutual synergy in the representation space of different modalities. The models proposed in [42] and [36] both fine-tune VLMs for specific VQA tasks, but lack the generalized few-shot prompt ability for other VL tasks. UNIFIED-IO [43]demonstrates a unified approach to a myriad of tasks, from classical computer vision to natural language processing, without task-specific fine-tuning. And ProGrad [44] introduces a technique to prevent prompt tuning from forgetting pre-trained vision-language models' general knowledge by only updating aligned prompts, outperforming other methods in various few-shot learning scenarios.

Exploring In-Context Configurations in NLP. Upon observing that pre-trained LMs are good fewshot learners, researchers also discover that diverse in-context configurations have dramatic effects on performance [3]. This observation sparks numerous studies aimed at determining the optimal in-context configurations, such as the format of the in-context examples [45; 21; 5], the selection of these examples [8; 9; 46; 47], and even the order in which these examples are presented [48; 49; 10]. However, these studies are predominantly conducted within the NLP field and fail to consider the unique characteristics and complexity of multi-modal data. In order to address this gap, we propose a series of strategies to explore the effects of various multi-modal in-context sequence configurations.

Image Caption. Image Captioning (IC) [50] aims at correctly verbalizing one image using descriptive languages, which can be solved through both retrieve [51] and generation [50], the former retrieves complete sentences from a corpus, while the latter generates words sequentially. Researchers have recently combined these approaches by first retrieving image-caption pairs and inputting them into generation models [52; 53]. This process resembles in-context captioning, which also retrieves image-caption pairs to help captioning. However, in contrast to them [52; 53], our work introduces novel image and caption selection strategies to study in-context captioning and these methods can also enhance traditional retrieval-generation methods.

Top-N Samples

CLIP

man

bike

on

car

man

near

…

man

bike

on

car

car

near

…

Top-N Samples

A man is riding

a bike on the

street.

A dog is lying

near the shoes.

…

CLIP

Text Encoder

CLIP

Top-N Samples

Vision Encoder

&

{object, relation,

attribution, class}

Dot Product

AND

SG

SG

object

Relation

attribution

class

{object, relation,

attribution, class}

SIIR-TAG

Top-N Samples

(a) SIIR-CLIP

(b) SIIR-TAG

CLIP

Vision Encoder

Vision Encoder

SG

SG

&

Extractor

Extractor

Extractor

Extractor

(c) DIIR-TT

(d) SICR-CLIP

3 Configuring In-Context Sequences

The in-context captioning can be treated as a vision-language conditioned text generation task. Given the multi-modal in-context sequence S = { ( I 1 , C 1 ); ( I 2 , C 2 ); ... ; ( I n , C n ); ˆ I } that contains n -shot image-caption pairs ( I , C ) and one test image ˆ I , we hope to generate the corresponding image caption ˆ C = { ˆ w 1 , ..., ˆ w T } in an auto-regressive manner. Here the t -th word ˆ w t is sampled from the following word distribution:

where the probability P ( · ) is calculated by a pre-trained Vision-Language Model (VLM) ( e . g ., Flamingo [6] or Otter [54]).

Various studies in NLP [8; 45; 48; 46] field have shown that the performance of in-context learning varies significantly with different in-context configurations. We explore these effects in the case of Image Captioning (IC). Unlike pure NLP tasks, IC is a dual-modal task, which makes it more complex than NLP tasks. Specifically, the mutual synergy of image-caption examples must be considered in in-context learning, as opposed to considering the images or the captions independently. We next respectively introduce the image selection (cf. 3.1) and caption assignment (cf. 3.2) strategies used to configure in-context image-caption pairs.

3.1 Selecting Images

Random Selection (RS). Given a set D = { ( I 1 , C 1 ) , ..., ( I M , C M ) } with M image-caption pairs, we randomly sample n images as { I 1 , ..., I N } in S .

Similarity-based Image-Image Retrieval (SIIR). Certain NLP studies suggest that performance can be enhanced by retrieving examples similar to the test case [8; 46; 47]. Following this approach, we retrieve n images with the highest similarity scores to the test image ˆ I from D . We employ two methods to compute these similarities: 1) SIIR-CLIP (Figure 2 (a)): Using the vision encoder of CLIP [35], we extract image embeddings to determine similarities. 2) SIIR-TAG (Figure 2 (b)): We utilize some scene graph extractors to derive tags for each image. Specifically, we employ Vinvl [55] to extract objects and their attributes, and IETrans [56] to determine the relations present within the image. Following this extraction, we conduct an AND operation to compute the similarity between tags.

Similarity-based Image-Caption Retrieval (SICR-CLIP) (Figure 2 (d)). Taking advantage of the cross-modal retrieval capability of CLIP [35], we use its vision and language encoders to embed images and captions into a shared space for computing similarities. Given ˆ I , we calculate its cross-modal embedding similarities with { C 1 , ..., C M } ∈ D , and select the images whose captions have topn similarities with ˆ I . Note that we use different kinds of captions as mediators; the methods to generate these captions will be detailed in Section 3.2. These captions are also used as C 1 , ..., C N ∈ S .

Diversity-based Image-Image Retrieval (DIIR) . For a single test sample, beside using similar incontext examples, some NLP studies find that diversity is also crucial [57]. Consequently, we retrieve a diverse set of images from D to configure S . Specifically, we employ the following strategies: 1) DIIR-TR : We extract discrete tags from the aforementioned VinVL and IETrans. Then we randomly divide the tags into N clusters and apply SIIR-TAG to retrieve the most similar image from each cluster. 2) DIIR-TT (Figure 2 (c)): To conveniently control the number of shots, we incorporate class tags generated by the IETrans model, extending the basis of SIIR-Tags to four categories: object, class, attribute, and relation. We then employ SIIR-TAG to identify the topk similar images within each category, allowing us to create 4 k -shot images in S . Note that both DIIR methods take into account a certain level of similarity during retrieval, rather than selecting entirely distinct images.

3.2 Assigning Captions

After selecting images, we need to assign one caption to each image to construct in-context imagecaption pairs. To explore the effects of multi-modal mutual synergy, we use captions with diverse qualities as introduced in the following.

Ground-Truth Captions (GTC). In the MSCOCO dataset [14], each image has 5 GTCs and we simply use the first one in S .

Model-Generated Captions (MGC). Compared with GTC, MGC has lower quality due to two disadvantages. Firstly, MGC uses poorer language, e . g ., simple words or rigid sentence patterns. Secondly, MGC has less descriptiveness that it may mis-verbalize or miss certain salient vision patterns of an image. However, we will see these two disadvantages do not equally cause worse performance compared with GTC in in-context captioning and surprisingly, we find that sometimes simple words or rigid sentence patterns even help generate good captions.

Here we apply two different models to generate the captions with diverse qualities. MGC-TF@ X : a Transformer is trained from scratch by using VinVL features and X denotes the CIDEr score [58] on the test set. To get captions of different qualities, we use the checkpoints from different training epochs and totally generate three kinds of MGCs. 1) MGC-TF@ 66 contains grammar mistakes while can describe the most salient objects. 2) MGC-TF@ 88 can use relatively correct grammar to describe the salient objects. 3) MGC-TF@ 135 is generated by a well-trained Transformer, i . e . , the loss converges. MGC-VLM( N )@ X : Another way to get MGCs is to use VLM in a N -shot manner. To achieve this, for each image I ∈ D , we treat I as the test image and then use Eq. (1) to generate a corresponding caption C conditioned on S constructed by only N image-caption pairs. As a result, two kinds of captions are got which are MGC-VLM( 0 )@ 63 and MGC-VLM( 32 )@ 81 .

Both two ways can construct the set D with M image-caption pairs. Then for a novel test image ˆ I , we can select some image-caption pairs from D by some above-mentioned approaches, e . g ., using RS or SIIR-CLIP to get images and assigning the MGCs to the image, to configure S for generating a new caption. Compared with MGC-TF, MGC-VLM is more practical as it addresses scenarios where only a handful or even no human-labelled captions are available, which means we do not have enough data to train a Transformer from scratch.

Iteratively Prompting (IP). For MGC-VLM introduced before, one natural extension is IP. To achieve this, in the first iteration, we generate a caption for each image I ∈ D by MGC-VLM. In the subsequent iteration, these generated captions are paired with the selected images to prompt VLM to generate enhanced captions. This process can be repeated across multiple iterations, thereby iteratively prompting the VLM.

Model-Generated Captions as Anchors (MGCA). A MGC can serve not only as an in-context caption but also as an anchor for selecting a suitable caption from the five GTCs. As MGCs typically verbalize salient visual patterns in an image but may miss finer details, using them as anchors can lead to the selection of GTCs that highlight these salient patterns. Furthermore, the selected GTC may supplement interesting details about these patterns, potentially assisting VLM to generate superior captions during in-context learning. In the MGCA implementation, for each selected image, we measure the similarity between the MGC and five GTCs using CIDEr and select the GTC with the highest CIDEr score.

85

MGC-TF@66

GTC

90

95

80

MGC-TF@88

MGC-TF@135

85

MGC-VLM(0)@63

70

MGC-VLM(32)@81

75

65

70

70

60

4

(a)RS

16

32

65

8

(b)SIIR-CLIP

16

32

65

8

(c)SIIR-TAG

16

32

S6

95

90

90

90

75

5

75

70

70

70

65

65

8

16

32

16

32

8

16

32

(d)SICR-CLIP

(e)DIIR-TR

(f)DIIR-TT

S6

Φ

RS

SIIR-CLIP

84

76

SIIR-TAG

90

SICR-CLIP

DIIR-TR

DIIR-TT

80

78

64

75

76

62

(a)GTC

16

32

74

4

8

(b)MGC-TF@88

16

32

60

4

8

(c)MGC-TF@66

16

32

95

86

76

90

84

74

80

80

70

75

78

68

70

66

4

(d)MGC-TF@135

8

16

32

(e)MGC-VLM(32)@81

8

16

32

(f)MGC-VLM(0)@63

8

16

32

4 Experiments

4.1 Dataset and Implementation Details

MSCOCO. We evaluate the proposed strategies on MSCOCO dataset [14], which is the most widely used benchmark in image captioning. We used the Karpathy split [59] in the experiments, which contains 113,287/5000/5000 training/validation/test images and each image is associated with 5 human-annotated captions.

Implementation Details We employ the Open-Flamingo model [16] to test our strategies, setting the length penalty to -2.0 and a maximum generation length of 20. We follow Flamingo [6] to use 4, 8, 16, and 32 shots. We respectively use ViT-L/14 and as the vision and language encoders to extract image and sentence embeddings that used in SIIR-CLIP and SICR-CLIP. For MGC-TF, we train the standard Transformer encoder-decoder architecture on the MSCOCO dataset and use the checkpoints underwent 1000, 3000, and 170,000 iterations respectively. These checkpoints generate the captions with CIDEr scores of 66, 88, and 135 on the test set. We implement all experiments on a single RTX 3090 using FP16.

4.2 Results and Analyses

Given our varied strategies for image selection and caption assignment, displaying results for each configuration in table format, especially at 4, 8, 16, and 32-shot levels, could become overwhelming. For clarity, we've chosen to present results using line charts and histograms in the main paper, while detailed numerical outcomes are in the supplementary material. The line charts in Figures 3 and 4 illustrate trends as the shot number grows. Each subplot within these figures corresponds to a unique strategy for image selection or caption assignment. Furthermore, Figures 5 and 6 show histograms of average CIDEr scores for the various shot results. To facilitate comprehension, we first analyze the effects of caption qualities 4.2.1 and then of image selection strategies 4.2.2.

4.2.1 Effects of Caption Qualities

Figure 4(a)-(e) reveals that performance typically improves with an increase in shot numbers. However, the rate of improvement varies, or even declines, depending on the quality of the captions used. For instance, in Figure 4(a), using Ground-Truth Captions (GTC), the increase rates of the

110

110

110

100

90

MGCA

CIDEr

100

90

90.62

98.54

99.58

97.06

100.95

94.82

96.22

100

90

87.95

95.18

95.86

92.02

97.03

91.47

93.12

80

70

80.04

64.24

83.24

76.4

83.06

73.24

82.51

74.05

80.38

81.1681.34

80

70

66.79

83.49

70.22

82.45

80

70

67.85

82.58

70.99

82.18

60

MGC-TF

@66

MGC-TF

88

(a)RS

MGC-TF

MGC-VLM(0)MGC-VLM(32)

63

@81

60

MGC-TF

@66

MGC-TF

88

(b)SIIR-CLIP

MGC-TF

@135

MGC-VLM(0)MGC-VLM(32)

@63

@81

60

GTC

MGC-TF

66

MGC-TF

@88

(c)SIIR-TAG

MGC-TF

@135

MGC-VLM(0)MGC-VLM(32)

9

@81

110

110

110

CIDEr

100

90

80

70

84.65

74.79

97.29

84.64

97.94

94.06

100.02

72.35

92.94

85.43

94.15

CIDEr

100

90

80

70

84.82

67.57

90.19

81.37

91.2

84.9

91.19

71.66

86.66

80.94

87.37

100

90

80

70

86.0

68.32

93.44

82.42

94.18

Z06

95.23

70.83

89.28

81.2

90.37

60

GTC

MGC-TF

66

MGC-TF

88

MGC-TF

135

MGC-VLM(0)MGC-VLM(32)

@63

@81

60

GTC

MGC-TF

@66

MGC-TF

88

MGC-TF

@135

MGC-VLM(0)MGC-VLM(32)

@63

@81

60

GTC

MGC-TF

66

MGC-TF

@88

MGC-TF

@135

MGC-VLM(0)MGC-VLM(32)

@63

@81

(d)SICR-CLIP

(e)DIIR-TR

(f)DIIR-TT

100

100

100

90.62

CIDEr

80

90

80.04

87.95

84.65

84.82

86.0

80

90

76.4

83.49

82.58

84.64

81.37

82.42

CIDEr

80

90

74.79

70

0

70-

64.24

66.79

67.85

67.57

68.32

60

RS

SIIR-CLIP SIIR-TAG SICR-CLIP DIIR-TR

DIIR-TT

60

RS

SIIR-CLIP SIIR-TAG SICR-CLIP DIIR-TR

DIIR-TT

60

RS

SIIR-CLIP SIIR-TAG SICR-CLIP DIIR-TR

DIIR-TT

(a)GTC

(b)MGC-TF@88

(c)MGC-TF@66

100

97.06

100

100

06

92.02

94.06

90.32

90

90

CIDEr

84.9

CIDEr

81.16

82.45

82.18

85.43

80.94

81.2

80

80

80

70

73.24

70

70

74.05

70.22

70.99

72.35

71.66

70.83

60

RS

SIIR-CLIP SIIR-TAG SICR-CLIP DIIR-TR

DIIR-TT

60

RS

SIIR-CLIP SIIR-TAG SICR-CLIP DIIR-TR

DIIR-TT

60

RS

SIIR-CLIP SIIR-TAG SICR-CLIP DIIR-TR

DIIR-TT

(d)MGC-TF@135

(e)MGC-VLM(32)@81

(f)MGC-VLM(0)@63

six image selection strategies surpass those in Figure 4(b) where MGC-TF@88 is used. Further, low-quality captions, such as MGC-TF@66, may become "toxic examples" which can misguide the Vision-Language Model (VLM) as the shot number increases. Next we compare different kinds of captions to figure out what characteristics of the captions affect the performance of in-context captioning.

Ground-Truth Captions (GTC) vs. Model-Generated Captions (MGC). As discussed in Sec 3.2, MGC exhibits two primary shortcomings when compared to GTC: poorer language and less descriptiveness. However, we will see that these two shortcomings do not always make MGC achieve poorer performance when compared to GTC. Specifically, we find that up to a certain level of descriptiveness, simpler sentence patterns are more easily recognized by the VLM, thereby improving caption generation .

Evidence for this can be observed in Figure 5 by comparing GTC with MGC-TF@135. Compared to the caption generated by MGC-TF@135, ground-truth caption has better language patterns, e . g ., rich vocabulary and complex sentence pattern, and better descriptiveness. Then when the selected images cannot provide enough vision cues, i . e . , when Random Selection (RS) is applied in Figure 5 (a), GTC outperforms MGC-TF@135. However, once the similarity-based retrieval methods like Similaritybased Image-Image Retrieval (SIIR) or Similarity-based Image-Caption Retrieval (SICR-CLIP) is used, the selected similar images offer useful visual patterns that help address the descriptiveness issue. Consequently, the VLM is more likely to recognize the consistent, simple patterns in MGC than the rich, diverse patterns in GTC and then generate better captions. Figure 7 (a) visualizes 2 examples about the above comparisons.

This effect is more pronounced in 4-shot cases, where VLM lacks sufficient in-context captions to discern sentence patterns, making simpler patterns advantageous. As long as the descriptiveness issue is addressed, MGC often outperforms GTC, e . g ., MGC-TF@88>GTC in Figure 3(b-d). Especially, when SICR-CLIP is employed to select captions with high cross-modal similarity to the test image, it significantly mitigates the descriptiveness problem. Then as demonstrated in Figure 3(d), even MGC-TF@66 surpasses GTC.

MGC-TF vs. MGC-VLM . Before we see that using more low-quality captions like MGC-TF@66 will misguide VLM to generate worse captions. Yet, Figure 4 (f) indicates that using MGCVLM(0)@63 improves performance with increased shot numbers, contrasting MGC-TF@66 in Figure 4 (c). This raises the question: why do weaker captions (MGC-VLM(0)@63) surpass those

...

...

Several motor scooters

are jammed into a small

market street.

A row of parked bicycles

sitting in front of a store.

Rows of motor scooters

are parked in front of a

store.

This slice of cake looks

like half cheesecake and

half vanilla.

A bite is taken out of a

piece of cake.

This slice of cake looks

like half cheesecake and

half vanilla cake.

A piece of cake on a

plate with a fork.

A piece of cake on a

plate with a fork.

A piece of cake on a

plate with a fork and a

spoon.

A row of motorcycles

parked in front of a street.

A group of motorcycles

parked in front of a

street.

A group of motorcycles

parked in front of a

street.

A piece of cake on a

plate with a fork.

A piece of cake on a

plate with a fork.

A piece of cake on a

plate with a fork and a

spoon.

A row of motorcycles

parked in front of a street.

A group of motorcycles

parked in front of a

street.

A group of motorcycles

parked in front of a

street.

(a) MGC-TF@135(blue) v.s. GTC(red)

...

...

A stop sign that is on a

street.

A stop sign on a street

with a street.

A stop sign on the side

of a street with a street

light.

A group of boats in the

water and a boat.

A boat on a boat in the

water.

A boat with a boat in the

water and a boat in the

water .

Two sailing boats are

moored in a harbour.

A view of a lake with a

boat and a dock in the

foreground.

A view of a lake with a

boat and a dock in the

foreground.

A close up view of a

traffic light that's red.

A close up picture of a

red traffic light.

A picture of a traffic

light that's red.

Two sailing boats are

moored in a harbour.

A view of a lake with a

boat and a dock in the

foreground.

A view of a lake with a

boat and a dock in the

foreground.

A close up view of a

traffic light that's red.

A close up picture of a

red traffic light.

A picture of a traffic

light that's red.

DA

S

lerode

CHAYTORC

(b) MGC-VLM(0)@63(blue) v.s. MGC-TF@66(red)

with higher CIDEr (MGC-TF@66)? This discrepancy aligns with our assumption: descriptiveness and language pattern influence in-context captioning differently.

MGC-TF@66 excels in object detection but struggles with language decoding, causing salient objects to be identified correctly but with syntactical errors in captions. As such examples increase, VLM produces worse captions due to these errors. Conversely, MGC-VLM(0)@63, though limited in object recognition, maintains better grammar. When more vision cues are provided, VLM leverages these along with the better-formed captions from MGC-VLM(0)@63, resulting in improved captions. Figure 7 (b) offers two comparison examples.

Model-Generated Captions as Anchors (MGCA). From Figure 5, we see that using MGC as the in-context captions usually underperforms GTC. However, by the MGCA strategy, we observe consistent improvements over GTCs, as demonstrated by the higher blue histograms of different MGCs compared to the grey dashed line. For example, despite MGC-TF@66 only achieves 64.24 CIDEr score in Figure 5(a), we still observe a 3.2 CIDEr improvement when using MGC-TF@66 as the anchor, compared to simply selecting the first GTC. Specifically, when using MGC-TF@66/MGCTF@88/MGC-TF@135/MGC-VLM(0)@63/MGC-VLM(32)@81 as the anchors, the average improvements over six image selection strategies compared to GTC are 7.3/8.0/8.8/3.6/4.8 respectively. In contrast to solely leveraging the RS+GTC method, the combination of SIIR-CLIP and using MGC-TF@135 as anchor chalked up an average boost of 20.9.

The primary reason for such improvement is likely that MGC, to some extent, verbalize the major patterns, such as the salient objects of an image. This helps identify which GTC provides more detailed information about these patterns. Such detailed information of the salient objects provide help VLM generate better captions. This assumption is further supported by comparisons between MGC-TF and MGC-VLM. Given that MGC-TF prioritizes verbalizing the salient objects of an image, using MGC-TF@66/MGC-TF@88 as anchors tends to select superior GTC than MGCVLM(0)@63/MGC-VLM(32)@81, thus yielding higher improvements.

Iteratively Prompting (IP). Table 1 showcases the IP CIDEr scores. "Iter 1" represents either the 0 or 32-shot performance, with subsequent columns reflecting average CIDEr scores for 4, 8, 16, and 32-shot scenarios. In the inaugural iteration, we employ a consistent set of 0 or 32-shot image-caption pairs. For the following iterations, images are selected via the Random Sampling (RS) ap-

proach, and captions generated in the prior iteration serve as the in-context captions. Analyzing the data, it's evident that MGC-VLM(0) stabilizes by the third iteration and MGC-VLM(32) by

A long table with a plant

on top of it surrounded

with wooden chairs.

A table is adorned with

wooden chairs with blue

accents.

A table is adorned with

wooden chairs and

bananas.

...

A long table with a plant

on top of it surrounded

with wooden chairs.

A table is adorned with

wooden chairs with blue

accents.

A table is adorned with

wooden chairs with red

accents.

...

(a) Experiment (1)

A long table with a plant

on top of it surrounded

with wooden chairs.

A table is adorned with

wooden chairs with blue

accents.

Bananas on a wooden

table.

...

A long table with a plant

on top of it surrounded

with wooden chairs.

A table is adorned with

wooden chairs with blue

accents.

A car with a sign that

says stop.

...

A long table with a plant

on top of it surrounded

with wooden chairs.

A table is adorned with

wooden chairs with blue

accents.

A long table with

bananas on it.

...

A long table with a plant

on top of it surrounded

with wooden chairs.

A table is adorned with

wooden chairs with blue

accents.

A stop sign on a road.

...

(b) Experiment (2)

STOP

STOP

STOP

STOP

STOP

STOP

(c) Experiment (3)

the second, indicating that extended VLM iterations might be redundant. Remarkably, even when confined to just 32-shot GTCs, merely two iterations of IP - for instance, where MGC-VLM(32) attains an average CIDEr of 80.5 - can rival performances seen when all GTCs are utilized, as exemplified by RS-GTC's average CIDEr of 80.04 in Figure 5(a).

4.2.2 Effects of Image Qualities

We evaluated the outcomes of several image selection techniques. SIIR-CLIP, leveraging vision embeddings to compute retrieval similarities, generally identifies images that are more analogous than those found by SIIR-TAG. This is likely attributed to the intrinsic noise present in semantic tags. SICR-CLIP, emphasizing captions that spotlight prominent objects, naturally gravitates towards images showcasing similar objects. In contrast, both DIIR-TR and DIIR-TT produce more varied selections. Nevertheless, when benchmarked against RS, every retrieval-based model consistently fetches images that exhibit greater similarity.

At first glance, one could reasonably infer that using more similar images would invariably lead to superior performance. Yet, as demonstrated in Figure 6, this assumption doesn't universally hold. When engaging high-quality captions, namely GTC in (a) and MGC-TF@135 in (d), the correlation stands, with more analogous images indeed translating to enhanced results. For instance, all retrieval-based techniques surpass RS in this context. However, with medium-quality captions, as in MGC-TF@88 in (b) and VLM(32)@81 in (e), only the similarity-based retrieval methods like SIIR or SICR-CLIP manage to outdo RS, and this is specifically observed in (e). Most intriguingly, when low-quality captions like MGC-TF@66 in (c) and VLM(32)@63 in (f) are in play, analogous images inversely impact performance, as illustrated by RS outperforming SIIR-CLIP in (f). This critical insight underscores a pivotal revelation: the efficacy of utilizing similar images is intricately tied to the caliber of the corresponding captions .

Similar Images Lead to Short-Cut Inference. A relatively bold hypothesis to elucidate this phenomenon is: when in-context images are similar to the test image, VLM may take a short-cut by leveraging in-context captions to generate a new one, rather than learning how to caption from the in-context pairs. Consequently, the greater the similarity between in-context and test images, the more the VLM is influenced by the in-context captions.

To robustly test our underlying assumption, we designed three distinct experimental setups. In each setup, the incontext captions remain consistent, comprising 5 groundtruth captions sourced randomly from an image unrelated to the test image. However, the in-context images are chosen differently in each experiment: (1) they are identical to the test image; (2) they are picked using SIIR-CLIP; (3)

they are chosen via RS. This progression results in decreasing similarity between the in-context and test images from experiments (1) through (3). Table 2 showcases two sets of CIDEr scores. One set compares the generated captions to the five ground-truth captions (GTC) of the test image, and the other set contrasts them with the five in-context captions (ICC). Our findings suggest a clear pattern: the closer the in-context images are to the test image, the more the VLM tends to mirror the ICC in its generated caption. For illustration, method (1) registers the highest CIDEr score when compared to the ICC. However, the caption it produces doesn't accurately depict the image, as reflected by its notably lower CIDEr score in relation to the GTC. These observations solidify our hypothesis that images with high similarity can inadvertently prompt short-cut inference. Additionally, Figure 8 provides visual representations to further elucidate the phenomenon of short-cut inference.

DIIR. As depicted in Figure 6, the performance of the two DIIR methods noticeably lags behind SIIR-TAG. This observation underscores the contention that the strength of diversity may not always translate to superior results in every context. One plausible explanation we propose is the inherent nature of captioning as a task. Contrary to certain complex NLP challenges, where diversity can be instrumental in offering a multifaceted understanding of a problem [57], captioning is relatively straightforward and may not benefit as much from diverse in-context examples. Delving deeper into the DIIR methods, DIIR-TT consistently outshines DIIR-TR. This leads us to infer that the clustering based on semantic tag types might be a more optimal strategy. Such a strategy, we believe, not only ensures diversity but also completeness in the selection of images. For instance, given a caption like "a brown dog is running", DIIR-TT could potentially source images that highlight elements like "brown objects", "dogs", and the "action of running".

5 Conclusion and Limitations

In this study, we utilize image captioning as a case study to examine the impacts of varied multimodal in-context configurations on few-shot vision-language prompting. Specifically, we design 4 different ways to select images and 4 different strategies to assign captions to the selected images for constructing multi-modal in-context configurations. Exhaustive experiments reveal that, contrary to single-modal NLP cases, multi-modal mutual synergy significantly influences performance. Notably, we observe that the descriptiveness and language patterns of the captions differently affect performance: better performance may be achieved with simpler and consistent sentence patterns when selected images compensate for descriptiveness issues. And we also discover that when the in-context images are similar to the test one, VLM may build a short-cut by directly using the in-context captions instead of really learning to captioning. Moreover, our optimal strategy lead to a 20.9 average increase in CIDEr scores compared to a random sampling baseline.

This study's primary limitation is that at the inception of our exploration, the only open-source multi-modal few-shot learner available is Open-Flamingo [16]. However, Open-Flamingo, when compared with the official Flamingo [6], underperforms due to its training on significantly less data. Consequently, some findings in this paper might shift if a more robust multi-modal few-shot learner is employed. Nevertheless, even in such a scenario, the diverse configuration strategies proposed in this paper maintain their utility, aiding researchers in swiftly exploring the characteristics of the employed VLMs. Additionally, we have provided experimental results on Otter [54] and smaller version of Open-Flamingo, with detailed findings available in Appendix.

Acknowledgments

This work is supported by National Science Foundation of China (62206048), Natural Science Foundation of Jiangsu Province (BK20220819), Young Elite Scientists Sponsorship Program of Jiangsu Association for Science and Technology Tj-2022-027 and the Big Data Computing Center of Southeast University.

References

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT , volume 1, page 2, 2019.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1-35, 2023.

Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842 , 2023.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP , 2022.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35: 23716-23736, 2022.

Eric Zelikman, Jesse Mu, Noah D Goodman, and Yuhuai Tony Wu. Star: Self-taught reasoner bootstrapping reasoning with reasoning. 2022.

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? DeeLIO 2022 , page 100, 2022.

Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 3816-3830, 2021.

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8086-8098, 2022.

Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 3081-3089, 2022.

Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning , pages 1931-1942. PMLR, 2021.

Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. Multi-modal answer validation for knowledge-based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 2712-2721, 2022.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673, 2020.

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An opensource framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 , 2023.

Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019.

Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4921-4933, 2021.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4582-4597, 2021.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045-3059, 2021.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems .

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Selfconsistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 , 2022.

Hangfeng He, Hongming Zhang, and Dan Roth. Rethinking with retrieval: Faithful large language model inference. arXiv preprint arXiv:2301.00303 , 2022.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509 , 2022.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems , 32, 2019.

Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 5100-5111, 2019.

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX , pages 104-120. Springer, 2020.

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16 , pages 121-137. Springer, 2020.

Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multitask vision and language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10437-10446, 2020.

Ziwang Fu, Feng Liu, Hanyang Wang, Siyuan Shen, Jiahao Zhang, Jiayin Qi, Xiangling Fu, and Aimin Zhou. Lmr-cbt: Learning modality-fused representations with cb-transformer for multimodal emotion recognition from unaligned multimodal sequences. arXiv preprint arXiv:2112.01697 , 2021.

Zhong Ji, Jingwei Ni, Xiyao Liu, and Yanwei Pang. Teachers cooperation: team-knowledge distillation for multiple cross-domain few-shot learning. Frontiers of Computer Science , 17(2): 172312, 2023.

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 , 2023.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research , 2022. ISSN 2835-8856. URL https: //openreview.net/forum?id=yzkSU5zdwD . Survey Certification.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923 , 2023.

Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. 2023.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. 2022.

Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems , 34:200-212, 2021.

Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35: 2507-2521, 2022.

Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations , 2022.

Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15659-15669, 2023.

Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics , 8:423-438, 2020.

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for incontext learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2655-2671, 2022.

SU Hongjin, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Representations .

Sawan Kumar and Partha Talukdar. Reordering examples helps during priming-based few-shot learning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 4507-4518, 2021.

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625 , 2022.

Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3156-3164, 2015.

Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8415-8424, 2021.

Rita Ramos, Bruno Martins, Desmond Elliott, and Yova Kementchedjhieva. Smallcap: lightweight image captioning prompted with retrieval augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2840-2849, 2023.

Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Retrieval-augmented transformer for image captioning. In Proceedings of the 19th International Conference on Content-based Multimedia Indexing , pages 1-7, 2022.

Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726 , 2023.

Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5579-5588, 2021.

Ao Zhang, Yuan Yao, Qianyu Chen, Wei Ji, Zhiyuan Liu, Maosong Sun, and Tat-Seng Chua. Fine-grained scene graph generation with data transfer. In ECCV , 2022.

Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 , 2022.

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4566-4575, 2015.

Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3128-3137, 2015.

Supplementary Material

A Experimental results on Open-Flamingo v1 9B

Here we present all the numerical results from the experiment, divided into four tables (Table 3, Table 4, Table 5, and Table 6) based on different Image Selection strategies. Additionally, we have calculated the average values for each method in the "mean" column.

B Experimental results on Open-Flamingo v2 3B

Here we present the numerical results on Open-Flamingo v2 3B model 3 in Table 7 based on two different Image Selection strategies. Additionally, we have calculated the average values for each method in the "mean" column.

From the values in the table, it can be seen that the trend is basically consistent with the v1 model. It can be considered that our strategy and analysis can be transferred to different models.

C Experimental results on Otter

Here we present the numerical results on Otter model in Table 8 based on two different Image Selection strategies. Additionally, we have calculated the average values for each method in the "mean" column.

From the values in the table, it can be seen that the trend is basically consistent with the v1 model. It can be considered that our strategy and analysis can be transferred to different models.

D More Results of MGC-TF@135 vs. GTC

We further elucidate the performance of MGC-TF@135(blue) and GTC(red), by offering additional examples shown in Figure 9. It can be easily observed that GTC has a more diverse range of sentence structures in comparison to MGC-TF@135. In the initial two examples, the words "cat" and "dachshunds" were inaccurately recognized by GTC, demonstrating its limitation in some specific instances. A shift can be noticed in the subsequent three examples, where the GTC generates captions with complex sentence structures, with a notable proportion of incorrect information. This finding serves to reinforce the notion that simpler sentence patterns, up to a certain degree of descriptiveness, are more readily deciphered by the VLM, thereby enhancing the quality of generated captions.

E More Results of MGC-TF@66 vs. MGC-VLM(0)@63

Similarly, we supplement with more examples, as depicted in Figure 10, to facilitate the comparison between MGC-TF@66(red) and MGC-VLM(0)@63(blue). A striking observation from this comparative study reveals that MGC-TF@66 demonstrates a significant number of syntactical errors. This flaw becomes problematic as it misdirects VLM, resulting in a substantial volume of syntactical mistakes in the produced sentences.This correlation implies that the syntax errors in the initial input by MGC-TF@66 tend to propagate into the VLM's output. Therefore, it becomes clear that the accuracy of grammar in the prompt is crucial for achieving better results.

Dog and cat sleeping on a

couch.

two dogs laying on top of a

brown couch.

Dog  and  cat  lying  on  floor

side by side sleeping.

a dog and a cat laying on

the floor.

Cats sleeping on top of a

bed with blue and white

striped blankets.

two cats laying on top of a

bed.

two  white  black  and  brown

dogs

are

lying

on

a

red

couch.

two dogs laying on top of a

red couch.

A  couple  of  dogs  sleeping

on top of a couch.

three dogs laying on top of a

couch.

two dachshunds and a cat

sleep on a couch.

two dogs laying on top of a

couch.

two dachshunds and a cat

sleep on a couch.

two dogs laying on top of a

couch.

two  dachshunds  and  a  cat

sleep on a bed.

two dogs laying on top of a

bed.

TWO DOGS LYING ON A

BED ON TOP OF THE

COVERS.

two dogs laying on top of a

bed.

A  couple  of  dogs  sleeping

on top of a couch.

three dogs laying on top of a

couch.

Cats sleeping on top of a

bed with blue and white

striped blankets.

two cats laying on top of a

bed.

A bed in a hotel room with a

window looking into the

bathroom.

a hotel room with two beds

and a window.

A bed in a hotel room with a

window looking into the

bathroom.

a hotel room with two beds

and a window.

A  very  clean  bedroom  with

two beds, lights and a TV.

a hotel room with two beds

and a television.

Two beds in a hotel room

with a window looking into

the bathroom.

a hotel room with two beds

and a mirror.

A bed in a bedroom next to

a slide glass door.

a hotel room with two beds

and a table.

A pink bedspread is featured

in this bedroom.

a bedroom with a bed with

two lamps.

A  pizza  sitting  on  top  of  a

wooden  cutting  board  with

different toppings.

a person with a pizza sitting

in a box on a table.

A  pizza  sitting  on  top  of  a

wooden  cutting  board  with

different toppings.

a person with a pizza sitting

in a box on a table.

A  pizza  sitting  on  top  of  a

wooden cutting board.

a pizza sitting in a box on a

table.

There is an open box with a

pizza inside the box.

a pizza in a box on a table.

A  box  with  pizza  in  it  that

has different toppings.

a person with a pizza in a

box.

A pizza sitting on top of a

cardboard box on a table.

a pizza sitting in a box on a

table.

A kid is trying to catch a

frisbee while standing in a

park.

a young girl flying a kite in

the sky.

A kid is trying to catch a

frisbee while standing in a

park.

a young girl flying a kite in

the sky.

A person in a park holding a

kite.

a person holding a kite in a

field.

A man that is chasing a

frisbee in the grass.

a man playing with a frisbee

in a park.

A

kid

is

trying

to

catch

something while standing in

a park.

a woman playing with a

frisbee in a park.

A kid is trying to catch a

frisbee while standing in a

park.

a young girl flying a kite in

the sky.

A person in a park holding a

kite.

a person holding a kite in a

field.

A man that is chasing a

frisbee in the grass.

a man playing with a frisbee

in a park.

A

kid

is

trying

to

catch

something while standing in

a park.

a woman playing with a

frisbee in a park.

A young boy flying a kite in

a blue sky with clouds.

a  young boy flying a kite  in

the sky.

D

D

AEDDUES

a vase of flowers sitting on a

table and a cup of coffee on a

table.

A vase of daffodils and a cup of

coffee on a table.

a  vase  of  flowers  sitting  on  a

table.

A vase of pink flowers sitting on

a wooden table.

a group of people sitting on a

table.

A vase of daffodils on a table

with a cup of coffee.

a  close  up  of  a  laptop  on  a

table.

A cup of coffee sitting on a

desk.

a  kitchen  with  a  table  and  a

table.

a rome with a cup of coffee and

a kettle.

a clock tower with a clock on it.

a close up of a clock on the side

of a building reads 12:00.

a clock tower with a clock on it.

a close up of a clock on the side

of a building reads 12:00.

a clock tower with a clock on it.

A clock on the side of a

building.

a clock tower with a clock on it.

A clock on the side of a

building reads 12:00.

a clock tower with a clock on it.

a close up of a clock on the side

of a building.

a clock tower with a clock on it.

a clock on the side of a

building.

a laptop sitting on top of a desk

with a desk and a desk.

A woman sitting at a desk with

a laptop in front of her.

a laptop sitting on top of a desk

with a desk and a desk.

A woman sitting at a desk with

a laptop in front of her.

a laptop sitting on a desk with a

desk.

A laptop computer on a table

with a cup of coffee.

a laptop sitting on top of a

desk.

A laptop on a desk in a hotel

room.

a  laptop  sitting  on  top  of  a

desk.

A laptop on a table.

a desk with a desk and a desk.

A man sitting at a desk with a

laptop in front of him.

a man riding a motorcycle on a

motorcycle.

A man riding a motorbike with

a police radio.

a man riding a motorcycle on a

motorcycle.

A man riding a motorbike with

a police radio.

a man riding a motorcycle on a

motorcycle.

a man riding a motorbike with a

woman and a child on the back.

a man riding a motorcycle on a

motorcycle.

A man riding a bicycle in front

of a row of motorcycles.

a

group

of

people

walking

down a street.

A man riding a motorbike with

his daughter.

a man riding a motorcycle on a

motorcycle.

A man rides a motorbike while

talking on a cell phone.

a vase with a vase in a vase and

a vase.

A vase filled with orange

flowers.

a vase with a vase in a vase and

a vase.

A vase filled with orange

flowers.

a table with a vase and flowers

on it.

a glass vase with orange

flowers.

a vase with a vase and a vase.

A blue vase with a purple flower

in it.

a vase with a vase and a vase.

A vase filled with purple

flowers.

a vase with a vase in a vase and

a vase.

A vase filled with orange

flowers.

a table with a vase and flowers

on it.

a glass vase with orange

flowers.

a vase with a vase and a vase.

A blue vase with a purple flower

in it.

a vase with a vase and a vase.

A vase filled with purple

flowers.

a vase with a vase and a vase.

A

vase

made

out

of

a

newspaper.

9

18

F More Results of short-cut inference

In this section, we provide additional examples to demonstrate how similar images can lead our VLM to exhibit a tendency for short-cut inference, where it might relies heavily on in-context captions to generate the captions, disregarding the image information in the prompt and the inherent relationships within the in-context pairs. In Figure 11, we present the results for the same test image under two different scenarios of choosing in-context images: "identical to the test image" (top row) and "via random sampling" (bottom row).

In the first scenario, for the majority of cases, our results are largely unrelated to the image content but similar to the in-context captions, with only a few instances capturing some elements from the image, such as "pink" in (a) and "on the beach" in (b). However, in the second scenario, our results are heavily influenced by the in-context captions, as evident in (c) and (d) with the mention of "wooden chairs".

A long table with a

flower arrangement in

the middle for meetings.

A table is adorned with

wooden chairs with blue

accents.

A table is adorned with

wooden chairs with pink

accents.

(a)

A long restaurant table

with rattan rounded

back chairs.

A long table with a plant

on top of it surrounded

with wooden chairs.

A restaurant has modern

wooden tables and

chairs.

A long table with a

flower arrangement in

the middle for meetings.

A table is adorned with

wooden chairs with blue

accents.

A mother and daughter

are carrying their

luggage.

A long restaurant table

with rattan rounded

back chairs.

A long table with a plant

on top of it surrounded

with wooden chairs.

A restaurant has modern

wooden tables and

chairs.

A long table with a

flower arrangement in

the middle for meetings.

A table is adorned with

wooden chairs with blue

accents.

A table is adorned with

wooden chairs with red

accents.

(d)

A long restaurant table

with rattan rounded

back chairs.

A long table with a plant

on top of it surrounded

with wooden chairs.

A restaurant has modern

wooden tables and

chairs.

A long table with a

flower arrangement in

the middle for meetings.

A table is adorned with

wooden chairs with blue

accents.

A small boy sits on a

wooden chair.

A long restaurant table

with rattan rounded

back chairs.

A long table with a plant

on top of it surrounded

with wooden chairs.

A restaurant has modern

wooden tables and

chairs.

A long table with a

flower arrangement in

the middle for meetings.

A table is adorned with

wooden chairs with blue

accents.

A table is adorned with

wooden chairs with red

accents.

(c)

A long restaurant table

with rattan rounded

back chairs.

A long table with a plant

on top of it surrounded

with wooden chairs.

A restaurant has modern

wooden tables and

chairs.

A long table with a

flower arrangement in

the middle for meetings.

A table is adorned with

wooden chairs with blue

accents.

A table is adorned with

bowl of dog food.

A long restaurant table

with rattan rounded

back chairs.

A long table with a plant

on top of it surrounded

with wooden chairs.

A restaurant has modern

wooden tables and

chairs.

A long table with a

flower arrangement in

the middle for meetings.

A table is adorned with

wooden chairs with blue

accents.

A table is adorned with

wooden chairs with blue

accents on the beach.

(b)

A long restaurant table

with rattan rounded

back chairs.

A long table with a plant

on top of it surrounded

with wooden chairs.

A restaurant has modern

wooden tables and

chairs.

A long table with a

flower arrangement in

the middle for meetings.

A table is adorned with

wooden chairs with blue

accents.

A man is running on the

beach.

A long restaurant table

with rattan rounded

back chairs.

A long table with a plant

on top of it surrounded

with wooden chairs.

A restaurant has modern

wooden tables and

chairs.

vadakkus

vadakkus

vodakkus

vadakkus

vadakkus

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a case study that examines the design of prompts and its effectiveness in generative visual language models for image captioning tasks. The authors delve into four methods for image selection and four methods for assigning captions to create contextually relevant learning samples. To evaluate different prompt design formats, the authors employed OpenFlamingo, an open-source version of the Flamingo paper. Several ablations and experiments were conducted to investigate the optimal inclusion of images and captions in the prompt of the Visual Language Model (VLM) to enhance performance. The authors put forth several findings:\n\n1. Unlike single-modal NLP cases, the performance of the model is significantly influenced by the multi-modal mutual synergy.\n\n2. The authors observe that the descriptiveness and language patterns of the captions have varying impacts on performance. In cases where selected images compensate for descriptiveness issues, better performance can be achieved by using simpler sentence patterns.\n\n3. Additionally, the authors discover that when the in-context images resemble the test image, the VLM may take a shortcut by directly utilizing the in-context captions instead of genuinely learning to generate captions.\n\nSTRENGTHS:\nGenerally, this paper is not novel, as there are similar papers in the NLP literature (see [1, 2]). While the authors mention [1] as one of the prior works, they appear to have overlooked [2]. \n\nIn contrast to the aforementioned prior works, I believe this paper seeks to delve into prior studies in more complex tasks, specifically open-ended generation tasks such as image captioning, and employing visual language models. In terms of quality, the paper explores a distinct aspect of the research questions and presents compelling findings that align with some prior works. Experimentally, the paper demonstrates rigor by thoroughly exploring different prompt constructions.\n\n[1]. \"Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?\" EMNLP 2022.\n\n[2]. \"What Makes Good Examples for Visual In-Context Learning?\" presented at ICML 2023.\n\nWEAKNESSES:\nThe paper has a few notable weaknesses that can be addressed:\n\n- The writing, particularly in lines 141-150, is challenging to comprehend. It would be beneficial for the authors to rephrase or provide visual aids to clarify this section, as it contains one of the paper's ablation studies.\n- Sections 4.2.1 and 4.2.2 should be swapped, considering that in the pipeline, images are typically selected first, followed by the assignment of captions. The current ordering of the text makes it difficult to parse.\n- It would be helpful to confirm the accuracy of the statement, \"Then, as demonstrated in Figure 5(d), even MGC-TF@66 surpasses GTC.\"\n- The author's choice of abbreviations throughout the manuscript makes the reading experience challenging. It requires readers to frequently refer back and forth between different parts of the text to fully understand a paragraph. It would be advisable to use less abbreviated phrasing.\n- Including more elaborate captions for the figures would enhance their comprehensibility as standalone visual aids.\n- The paper exclusively explores one visual language model, specifically OpenFlamingo, with 9B parameters. It is important to acknowledge that the findings may not be generalizable to other models.\n\nQUESTIONS:\n1. The paper could benefit from considering alternative visual language models to enhance the quality of the results. One such model is FROMAGe [1], which also utilizes a sequence of interleaved images and texts, similar to OpenFlamingo. It would be valuable for the authors to explore the use of FROMAGe in addition to OpenFlamingo and examine if the findings remain consistent across different models.\n\n2. Additionally, it is worth investigating the impact of using smaller language models within OpenFlamingo. As per the OpenFlamingo GitHub repository, there are newer versions pre-trained on smaller LLMs. The authors could conduct ablations with these models to determine how the findings vary when using models with fewer parameters. This would provide insights into the robustness of the results across different model sizes.\n\n[1]. Grounding Language Models to Images for Multimodal Inputs and Outputs, ICML 2023\n\nLIMITATIONS:\nThere are not any limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies how to choose in-context examples (images and text) for open flamingo models on the CoCo captioning task. Extensive experiments are run with many baselines. Based on these experiments, the authors conclude (1) given sufficient descriptiveness, simpler captions are better, (2) similarity of in-context images to the test image is less important when the corresponding captions to said images are poor.\n\nSTRENGTHS:\nS1. Strong empirical investigation of how different image and text sampling strategies affect in-context image captioning performance. The paper is correctly framed and emphasizes experimentation over methodology.\n\nS2. Extensive baselines implemented for the investigation.\n\nS3. Seemingly strong performance relative to random sampling on the train set, which is traditionally used to generate in-context examples.\n\nWEAKNESSES:\nW1. Presentation. Consider adding some of the key findings to the abstract. 7.3 CIDEr points is significant! It may be nice to mention this in the abstract as well.\n\nW2. Presentation. Some of the images in Figure 1 are hard to follow, consider making the example more demonstrative, or making the images bigger. Also grouping the good and bad captions within the same bubble was initially confusing and it took me some time to parse what was going on.\n\nW3. Presentation. L51-59 potentially have too much detail for the introduction. Consider distilling the content. The key takeaway from this paragraphs seems to be that the authors experimented with many sampling strategies.\n\nW4. Experiments. It would be nice to see similar analysis for the VQAv2 dataset, which is also supported in the OpenFlamingo evaluation suite. One natural question is do some of the findings for image captioning translate to VQA-like tasks, which are potentially more similar to the QA tasks shown in Figure 1.\n\nW5. Presentation. Consider removing the abbreviations SIIR, DIIR, etc. from the intro. They were a bit tough for me to follow as they are not standard to the best of my knowledge and made reading the intro more difficult. In general, the many abbreviations in the paper make the writing difficult to follow.\n\nW6. Related work. Consider adding a related work section on image captioning, since this is a focus of the paper.\n\nW7. Clarity. A bit of a nit, but please specify how how the samples are randomly sampled. I am assuming uniformly at random?\n\nW8. Clarity. Can you please expand on the SIIR-TAG baseline? In DIIR, how can SIIR-TAG be applied to a collection of images (a cluster)?\n\nW9. Missing baseline. What about using models like BLIP2 for synthetic captioning (MGC)?\n\nW10. Clairity. It is hard to tell what the takeaways from Figures 3 and 4 are from glancing at them, especially because all of the abbreviations can be confusing. Consider adding they key takeaways to the figure captions. It is not currently clear to me how to read these figures. Is the takeaway that MGC is best? In general, it is not always clear how the conclusions or claims in the paper are supported numerically by the figures or tables (e.g., claims in L231-233).\n\nW11. Presentation. In some cases numbers are averaged over multiple shots and in other cases this is not the case. Why did the authors choose to do this? And why is there not consistency in this choice?\n\nQUESTIONS:\nThe following questions are distilled from my most significant concerns:\n\nCan the authors address the concerns surrounding presentation (W1, W2, W3, W5, W7, W8, W10, W11)? While the baselines seem thorough, they are not always clear for me to follow. Additionally, while the authors run many experiments, the presentation of these experiments is not always interpretable and it is not clear to me that the experiments actually support the conclusions that the authors draw. Some more clarity surrounding this would be greatly appreciated.\n\nIs it possible to do some experiments for the best, middle, and worst baselines on VQAv2 to see if trends are similar for another image to text task (W4)?\n\nLIMITATIONS:\nThe authors acknowledge that findings for the open flamingo model tested may not translate to the more powerful, but also closed-source flamingo model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the effects of varying configurations on Vision-Language (VL) in-context learning, specifically in the field of image captioning. The authors developed four strategies each for image selection and caption assignment to determine how different methods of configuring image-text pairs impact the learning process. The study revealed two insights: 1) Caption quality, influenced by descriptiveness and language patterns, impacts captioning performance, with simpler language yielding better results when paired with descriptive captions. 2) The effectiveness of similar images relies on caption quality, with excessive similarity potentially misleading the model when paired with low-quality captions.\n\nSTRENGTHS:\n+ The general goal is reasonable and interesting. The field of in-context learning in multimodality is well worth exploring.\n\n+ The experiments were comprehensive. The authors compared numerous in-context learning strategies, including some new proposals of their own.\n\n+ The paper provides some useful insights about how to choose in-context samples.\n\nWEAKNESSES:\n- The authors said that “the performance ... heavily relies on the caption quality” in Fig 1 Caption. I'm curious about how the authors define the quality of a caption. The example given in Figure 1 suggests that the caption in the blue box is better than the one in the red box. It is easy to understand because the red box caption contains clear errors, such as repeated phrases. When selecting in-context samples, such samples obviously should not be selected. But, in practice, such samples should not even be available as candidates, right? Could the authors display what captions the model considers as high-quality or low-quality? Or intuitively illustrate what captions are expected to consider as a high quality among several ground truths?\n\n- One insight of the paper is that \"When captions adequately describe salient image objects, simpler language patterns may yield better\nresults.\" But aren't the ground truth captions in the training set in the language style expected by the benchmark? If we use captions generated by the model, then how does the model know what style is required by the benchmark? \n\n- I'm not certain whether the experimental setting in this paper can effectively evaluate the capabilities of in-context learning. In-context learning aims to use a small number of samples as demonstrations to learn some new things, such as a new task, new input-output format,  new object naming rules, etc. This work conducts experiments on the MSCOCO caption dataset. I'm unsure what the purpose of in-context learning is in this case, as these samples seem quite common for large models. So, what patterns or things are expected to learn by model from these in-context samples?\n\nQUESTIONS:\nMy major concerns are about the current setting of in-context learning, and I would like to hear more about how to define the quality of the captions, which is important in choosing samples.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates the configuration of in-context image-caption pairs for few-shot image captioning in a VL model: OpenFlamingo. Specifically, authors compare four different image selection methods: (1) random selection, (2) image-image similarity based selection (SIIR), (3) image-caption similarity based selection (SICR), and (4) image-image diversity based selection (DIIR). \n\nOn top of the selected images, the authors also test out four different caption assignment methods: (1) ground-truth caption (GTC), (2) model-generated caption (MGC), (3) Iterative prompting (IP), and (4) Model-generated Captions as Anchors (MGCA). \n\nThe findings from this work suggest that (1) selecting similar images with the test image is better than random sampling; (2) simpler language patterns in captions may be better if they adequately explain the salient info of the image; and (3)similar in-context images can induce short-cut inference from their captions.\n\nSTRENGTHS:\nAnalyses on in-context sampling method for VL models can expand the literature. Especially, the fact that MGCA results lead to better performance than random ground-truth captions is interesting.\n\nWEAKNESSES:\nAll experiments are conducted on a single open-source VL model; this leads to questions whether these results hold generalizability.\n\nFurther analyses should have been conducted to backup the authors’ claims: “simpler caption pattern improves caption generation” & “verbalizing the major patterns help identify which GTC provides more detailed info about these patterns”. Please see the question section.\n\nQUESTIONS:\n(1) I find it interesting that GTC performs worse most of the time compared to MGC. Why do you think this the case? I am very suspicious that it is due to the consistent format of captions across in-context samples instead of the simple pattern of the MGCs. However, this is hard to analyze with the current setup. MGCs will have very similar sentence structures for similar images compared to GTCs, which is also shown in Figure 7-(a). I suggest controlling the format consistency across GTCs within the in-context samples. For example, you can set captions to have the same complex structure and see how that results in performance. If simplicity is the true cause of improved performance, setting consistent but complex format should not give improve performance. Also, you can try very diverse format across captions but make them simple in structure.\n\n(2) Don’t almost all GTCs include the most salient object in their captions? I find it hard to believe that MGCA improves performance because it finds the GTC that describes the salient pattern with more info. The selected GTCs from MGCA will probably have similar structure with the MGCs, hence it reduces the format variance of GTCs across the in-context samples. What happens if you select the GTC that is least similar with the MGC?\n\n(3) When giving the in-context samples, do the images really help in performance? What if you drop all in-context images and give only the captions instead? \n\n(4) I think it would be better to swap the location of Figure 3 & 4 with Figure 5 & 6. Is there a reason why you put Figure 3 & 4 first? I found it difficult to go back and forth because the main draft explains Figure 5 & 6 first.\n\nLIMITATIONS:\nYes, the authors also acknowledge the limitation that these analyses were conducted on a single model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the in-context configurations for few-shot ability in Vision-Language Models (VLMs). Therefore, they design four strategies for image selection and four for caption assignment to explore the influences of in-context pairs in image captioning. As a result, extensive experiments uncover two valuable insights (1) The captions adequately describe salient image objects and simpler language patterns may yield better results. (2) Excessive similarity might cause VLMs to create a short-cut inference from in-context captions. Finally, they introduce the iterative prompting for images with limited or no ground-truth captions, which boosts the performance of models with an average CIDEr improvement of 7.3.\n\nSTRENGTHS:\n1.\t**The considered problem is very relevant and timely for the AI community.**. It is urgent to explore the in-context configurations for vision-language pre-trained models. \n2.\t**The observations are valuable.**. They observe that better performance may be achieved with simpler sentence patterns when selected images compensate for descriptiveness issues. Moreover, they observe that the VLM may build a shortcut rather than learn to caption when the in-context images are similar to the test one.\n3.    **The experiments are comprehensive.**. They devised four strategies for image selection and four for caption assignment to explore the influences of in-context pairs in image captioning.\n\nWEAKNESSES:\n1.\t**The selected metric is not convinced.** They use CIDEr to evaluate caption models' performances. However, CIDEr is a statistical indicator related to description length, which is hard to completely reflect the quality of the generated captions. Therefore, it is necessary to explore generative evaluation indicators such as CLIPScore[1] and GPTScore[2] that use pre-trained models.\n2.\t**The choices of VLMs are limited, which may lead to unobjective conclusions.** They just use the OpenFlamingo as the multi-modal learner.  The authors should add more VLMs to explore the in-context configurations for image captioning, e.g. MiniGPT4[3], LLaVA[2].\n\n3.\t**The observation is a little overclaiming.** The in-context configurations observation from the image caption is not equal to the VL in-context learning. There are many visually-conditioned VL tasks that need to be explored such as VQA[4], and dense image captioning.\n\n4.    **The method section of iteratively prompting (IP) is confusing.** It is hard to understand how the algorithm of IP works, and the lack of analysis for IP.\n\n\n[1] Hessel, Jack, et al. \"Clipscore: A reference-free evaluation metric for image captioning.\" arXiv preprint arXiv:2104.08718 (2021).\n\n[2] Liu, Haotian, et al. \"Visual instruction tuning.\" arXiv preprint arXiv:2304.08485 (2023).\n\n[3] Zhu, Deyao, et al. \"Minigpt-4: Enhancing vision-language understanding with advanced large language models.\" arXiv preprint arXiv:2304.10592 (2023).\n\n[4] Tsimpoukelli, Maria, et al. \"Multimodal few-shot learning with frozen language models.\" Advances in Neural Information Processing Systems 34 (2021): 200-212.\n\nQUESTIONS:\nI tend to increase the review score once the following questions are answered well. The following questions are shown in weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-31"}