SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xu Yang $^{1*}$ , Yongliang Wu $^{1}$ , Mingzhuo Yang $^{2}$ , Haokun Chen $^{1}$ , Xin Geng $^{1}$
$^{1}$ School of Computer Science & Engineering, Key Lab of New Generation Artificial Intelligence Technology & Its Interdisciplinary Applications (Ministry of Education), Southeast University
2The Chinese University of HongKong, Shenzhen
xuyang_palm@seu.edu.cn, yongliangwu@seu.edu.cn,
118020068@link.cuhk.edu.cn, chenhaokun@seu.edu.cn, xgeng@seu.edu.cn
After discovering that Language Models (LMs) can be good in-context few-shot learners, numerous strategies have been proposed to optimize in-context sequence configurations. Recently, researchers in Vision-Language (VL) domains also develop their few-shot learners, while they only use the simplest way, i.e., randomly sampling, to configure in-context image-text pairs. In order to explore the effects of varying configurations on VL in-context learning, we devised four strategies for image selection and four for caption assignment to configure in-context image-text pairs for image captioning. Here Image Captioning is used as the case study since it can be seen as the visually-conditioned LM. Our comprehensive experiments yield two counter-intuitive but valuable insights, highlighting the distinct characteristics of VL in-context learning due to multi-modal synergy, as compared to the NLP case. Furthermore, in our exploration of optimal combination strategies, we observed an average performance enhancement of 20.9 in CIDEr scores compared to the baseline. The code is given in https://github.com/yongliang-wu/ExploreCfg.
In contemporary times, the Language Model (LM) [1; 2] has emerged as a pivotal player in the field of Natural Language Processing (NLP). It accomplishes this by unifying a range of diverse NLP tasks into a shared prompt paradigm [3; 4]. To elaborate, a LM, unsupervised trained via conditional language modeling on a substantial volume of non-annotated data collected from the web, can reformulate various downstream tasks into fitting textual prompts. These prompts contain slots, the word probabilities of which are calculated using the pre-trained LM. This process obviates the necessity for gradient updates to the parameters of LM, thus mitigating the challenges associated with additional data collection and fine-tuning. To further enhance the effectiveness of this paradigm, the few-shot prompt (or in-context learning) [5] is proposed where a few examples are provided as additional contexts to guide the LM to generate the desired result.
Witnessing the success of the prompt paradigm in the NLP field, there has been a concerted effort by researchers to replicate its function in the Vision-Language Model (VLM). To facilitate this, Flamingo [6] is proposed to align well-trained large-scale vision and language models through some trainable cross-modal adapters. Consequently, the resultant model is capable of addressing Vision-Language (VL) tasks by processing a prompt sequence, which includes several interleaved image and text examples for in-context learning. Since the primary objective of Flamingo is to build a VLM for the few-shot prompt, they only apply a straightforward strategy to configure the in-context
Figure 1: The distinction between LM and VLMs as few-shot learners. LM generally excel with examples akin to the test case (blue blocks in (a)). In contrast, for VLMs, the performance is not strictly correlated with image similarity but heavily relies on the caption quality. For instance, when low-quality captions are used, similar images (d) lead to worse performance than dissimilar ones (f) since VLMs may build a short-cut by reusing in-context captions without seeing the given images.
sequence by randomly sampling a few image-text pairs. Nevertheless, a plethora of studies within the NLP field [7] have demonstrated that diverse in-context configurations lead to dramatic effects on few-shot performance, e.g., the selection or ordering of in-context samples [8; 9; 10], while only a limited number of studies systematically explore such effects in the VL case.
To narrow this gap, we explore the effects of various in-context configurations on the performance of few-shot VL tasks. Among various VL tasks, Image Captioning (IC) aims at generating a text conditioned on the source image, and thus can be considered as the visually-conditioned LM. Just as a multitude of NLP tasks can be recast as LM tasks, IC performs a similar function [11; 12; 13], which motivates our decision to select IC as the subject of our case study. However, unlike in NLP, where only single-modal texts are considered in the in-context configuration, in IC, the synergy between multiple modalities significantly influences the performance. For instance, our experiments revealed that selecting images similar to the test image for the in-context sequence does not always lead to good performance, a result closely tied to the quality of the associated captions. Figure 1 shows the comparison between LM and VLM as the few-shot learners.
Consequently, we design diverse ways to select images as the in-context images. After selection, the captions of different qualities are assigned to these images for constructing the multi-modal in-context sequence. By combining diverse image selection and caption assignment techniques, we undertake a comprehensive exploration of the effects of multi-modal mutual synergy on VL in-context captioning. We implement all experiments using the prevalent captioning dataset, MSCOCO [14], employing its training set as the database for image selection.
To select images, we use 4 different ways, which are Random Sampling, Similarity-based Image-Image Retrieval, Similarity-based Image-Caption Retrieval, and Diversity-based Image-Image Retrieval. Following image selection, various types of captions produced by 4 different strategies are assigned, which are Ground-Truth Captions, Model-Generated Captions, Iteratively Prompting, and Model-Generated Captions as Anchors.
Through extensive evaluation of various image selection and caption assignment strategies, we uncover two counter-intuitive yet valuable insights. (1) Caption quality is determined by descriptiveness and language patterns, but their influence on in-context captioning performance is unequal. When captions adequately describe salient image objects, simpler language patterns may yield better results. (2) The efficacy of similar images depends on the quality of the paired captions. Excessive similarity might cause VLM to create a short-cut inference [15] from in-context captions, potentially misleading the model with low-quality captions. Beyond these findings, we introduce a practical in-context captioning strategy, Iterative Prompting, for cases with limited or no Ground-Truth Captions. Furthermore, when Ground-Truth Captions are available, we recommend using Model-Generated Captions as anchors to identify which Ground-Truth Caption is a more suitable in-context caption. Experimental results indicate that even when utilizing low-quality model-generated captions, there is an average CIDEr improvement of 7.3. Moreover, in optimal conditions, the average enhancement reaches up to 20.9 points compared to the random sampling baseline.
Since the Flamingo code is not publicly available, our experiments mainly utilize the unofficial Open-Flamingo  $[16]^2$ . It's worth noting that the performance of Open-Flamingo is not on par with the official Flamingo due to its training on less data.
Prompting Language Model (LM). The research paradigms of NLP have encountered two sea changes in the past few years [3]. The first one is the LM that is pre-trained by predicting the next word conditioned on observed words and can be fine-tuned for solving various downstream tasks, including GPT [17], BERT [1], and BART [18]. The second sea change is the emergence of the prompt paradigm, which was introduced with GPT-3 [2]. Within this paradigm, a pre-trained LM does not require fine-tuning to solve downstream tasks; instead, tasks are reformulated into appropriate prompts with empty slots to be filled. Subsequently, more advanced prompt-based techniques have been proposed, including prompt-tuning [19; 20; 21] and Chain-of-Thought [22; 23; 24; 25].
Prompting Vision-Language Model (VLM). In contrast to NLP, the Vision-Language (VL) domain has made significant strides in the first sea change, as evident by the development of various VL-BERT models. These models leverage large volumes of web-collected image-caption pairs to learn VL-generalizable embeddings [26; 27; 28; 29; 30; 31; 32; 33]. However, the prompt paradigm, despite revolutionizing NLP studies, only appears when a certain scale of the model is reached [34]. This scale prerequisite poses further challenges to the development of a VLM with prompt and in-context learning ability.
To mitigate training burdens, instead of updating all the parameters of a VLM [35; 36], some VLMs [35; 36; 37] freeze well-trained Language Models and only train a smaller network, referred to as adapters [38; 39; 40], to align the pre-trained vision and language models. Inspired by these models, both Frozen [41] and Flamingo [6] evolve into multi-modal few-shot learners by training vision and cross-modal adapters, respectively. Given its superior in-context learning ability, we use Flamingo to explore the effects of various in-context configurations [16].
There are also models that address VL tasks through in-context learning. For instance, PICa [11] utilizes captions as mediators to construct an in-context text for solving Visual Question Answering (VQA) tasks, while this may lose the mutual synergy in the representation space of different modalities. The models proposed in [42] and [36] both fine-tune VLMs for specific VQA tasks, but lack the generalized few-shot prompt ability for other VL tasks. UNIFIED-IO [43] demonstrates a unified approach to a myriad of tasks, from classical computer vision to natural language processing, without task-specific fine-tuning. And ProGrad [44] introduces a technique to prevent prompt tuning from forgetting pre-trained vision-language models' general knowledge by only updating aligned prompts, outperforming other methods in various few-shot learning scenarios.
Exploring In-Context Configurations in NLP. Upon observing that pre-trained LMs are good few-shot learners, researchers also discover that diverse in-context configurations have dramatic effects on performance [3]. This observation sparks numerous studies aimed at determining the optimal in-context configurations, such as the format of the in-context examples [45; 21; 5], the selection of these examples [8; 9; 46; 47], and even the order in which these examples are presented [48; 49; 10]. However, these studies are predominantly conducted within the NLP field and fail to consider the unique characteristics and complexity of multi-modal data. In order to address this gap, we propose a series of strategies to explore the effects of various multi-modal in-context sequence configurations.
Image Caption. Image Captioning (IC) [50] aims at correctly verbalizing one image using descriptive languages, which can be solved through both retrieve [51] and generation [50], the former retrieves complete sentences from a corpus, while the latter generates words sequentially. Researchers have recently combined these approaches by first retrieving image-caption pairs and inputting them into generation models [52; 53]. This process resembles in-context captioning, which also retrieves image-caption pairs to help captioning. However, in contrast to them [52; 53], our work introduces novel image and caption selection strategies to study in-context captioning and these methods can also enhance traditional retrieval-generation methods.
$^{2}$ Project: https://laion.ai/blog/open-flamingo/
(c) DIIR-TT
(d) SICR-CLIP
Figure 2: Image selection strategies: (a) SIIR-CLIP, (b) SIIR-TAG, (c) DIIR-TT, (d) SICR-CLIP.
3 Configuring In-Context Sequences
The in-context captioning can be treated as a vision-language conditioned text generation task. Given the multi-modal in-context sequence  $S = \{(I_1, C_1); (I_2, C_2); \ldots; (I_n, C_n); \hat{I}\}$  that contains  $n$ -shot image-caption pairs  $(I, C)$  and one test image  $\hat{I}$ , we hope to generate the corresponding image caption  $\hat{C} = \{\hat{w}_1, \dots, \hat{w}_T\}$  in an auto-regressive manner. Here the  $t$ -th word  $\hat{w}_t$  is sampled from the following word distribution:
$$
P \left(\hat {w} _ {t} \mid \mathcal {S}, \hat {w} _ {1: t - 1}\right), \tag {1}
$$
where the probability  $P(\cdot)$  is calculated by a pre-trained Vision-Language Model (VLM) (e.g., Flamingo [6] or Otter [54]).
Various studies in NLP [8; 45; 48; 46] field have shown that the performance of in-context learning varies significantly with different in-context configurations. We explore these effects in the case of Image Captioning (IC). Unlike pure NLP tasks, IC is a dual-modal task, which makes it more complex than NLP tasks. Specifically, the mutual synergy of image-caption examples must be considered in in-context learning, as opposed to considering the images or the captions independently. We next respectively introduce the image selection (cf. 3.1) and caption assignment (cf. 3.2) strategies used to configure in-context image-caption pairs.
3.1 Selecting Images
Random Selection (RS). Given a set  $\mathcal{D} = \{(I_1, C_1), \dots, (I_M, C_M)\}$  with  $M$  image-caption pairs, we randomly sample  $n$  images as  $\{I_1, \dots, I_N\}$  in  $\mathcal{S}$ .
Similarity-based Image-Image Retrieval (SIIR). Certain NLP studies suggest that performance can be enhanced by retrieving examples similar to the test case [8; 46; 47]. Following this approach, we retrieve  $n$  images with the highest similarity scores to the test image  $\hat{I}$  from  $\mathcal{D}$ . We employ two methods to compute these similarities: 1) SIIR-CLIP (Figure 2 (a)): Using the vision encoder of CLIP [35], we extract image embeddings to determine similarities. 2) SIIR-TAG (Figure 2 (b)): We utilize some scene graph extractors to derive tags for each image. Specifically, we employ Vinvl [55] to extract objects and their attributes, and IETrans [56] to determine the relations present within the image. Following this extraction, we conduct an AND operation to compute the similarity between tags.
Similarity-based Image-Caption Retrieval (SICR-CLIP) (Figure 2 (d)). Taking advantage of the cross-modal retrieval capability of CLIP [35], we use its vision and language encoders to embed images and captions into a shared space for computing similarities. Given  $\hat{I}$ , we calculate its cross-modal embedding similarities with  $\{C_1, \dots, C_M\} \in \mathcal{D}$ , and select the images whose captions have top- $n$  similarities with  $\hat{I}$ . Note that we use different kinds of captions as mediators; the methods to generate these captions will be detailed in Section 3.2. These captions are also used as  $C_1, \dots, C_N \in S$ .
Diversity-based Image-Image Retrieval (DIIR). For a single test sample, beside using similar in-context examples, some NLP studies find that diversity is also crucial [57]. Consequently, we retrieve a diverse set of images from  $\mathcal{D}$  to configure  $S$ . Specifically, we employ the following strategies: 1) DIIR-TR: We extract discrete tags from the aforementioned VinVL and IETrans. Then we randomly divide the tags into  $N$  clusters and apply SIIR-TAG to retrieve the most similar image from each cluster. 2) DIIR-TT (Figure 2 (c)): To conveniently control the number of shots, we incorporate class tags generated by the IETrans model, extending the basis of SIIR-Tags to four categories: object, class, attribute, and relation. We then employ SIIR-TAG to identify the top- $k$  similar images within each category, allowing us to create  $4k$ -shot images in  $S$ . Note that both DIIR methods take into account a certain level of similarity during retrieval, rather than selecting entirely distinct images.
3.2 Assigning Captions
After selecting images, we need to assign one caption to each image to construct in-context image_caption pairs. To explore the effects of multi-modal mutual synergy, we use captions with diverse qualities as introduced in the following.
Ground-Truth Captions (GTC). In the MSCOCO dataset [14], each image has 5 GTCs and we simply use the first one in  $S$ .
Model-Generated Captions (MGC). Compared with GTC, MGC has lower quality due to two disadvantages. Firstly, MGC uses poorer language, e.g., simple words or rigid sentence patterns. Secondly, MGC has less descriptiveness that it may mis-verbalize or miss certain salient vision patterns of an image. However, we will see these two disadvantages do not equally cause worse performance compared with GTC in in-context captioning and surprisingly, we find that sometimes simple words or rigid sentence patterns even help generate good captions.
Here we apply two different models to generate the captions with diverse qualities. MGC-TF@X: a Transformer is trained from scratch by using VinVL features and  $X$  denotes the CIDEr score [58] on the test set. To get captions of different qualities, we use the checkpoints from different training epochs and totally generate three kinds of MGCs. 1) MGC-TF@66 contains grammar mistakes while can describe the most salient objects. 2) MGC-TF@88 can use relatively correct grammar to describe the salient objects. 3) MGC-TF@135 is generated by a well-trained Transformer, i.e., the loss converges. MGC-VLM(N) $@X$ : Another way to get MGCs is to use VLM in a  $N$ -shot manner. To achieve this, for each image  $I \in \mathcal{D}$ , we treat  $I$  as the test image and then use Eq. (1) to generate a corresponding caption  $C$  conditioned on  $S$  constructed by only  $N$  image-caption pairs. As a result, two kinds of captions are got which are MGC-VLM(0) $@63$  and MGC-VLM(32) $@81$ .
Both two ways can construct the set  $\mathcal{D}$  with  $M$  image-caption pairs. Then for a novel test image  $\hat{I}$ , we can select some image-caption pairs from  $\mathcal{D}$  by some above-mentioned approaches, e.g., using RS or SIIR-CLIP to get images and assigning the MGCs to the image, to configure  $S$  for generating a new caption. Compared with MGC-TF, MGC-VLM is more practical as it addresses scenarios where only a handful or even no human-labelled captions are available, which means we do not have enough data to train a Transformer from scratch.
Iteratively Prompting (IP). For MGC-VLM introduced before, one natural extension is IP. To achieve this, in the first iteration, we generate a caption for each image  $I \in \mathcal{D}$  by MGC-VLM. In the subsequent iteration, these generated captions are paired with the selected images to prompt VLM to generate enhanced captions. This process can be repeated across multiple iterations, thereby iteratively prompting the VLM.
Model-Generated Captions as Anchors (MGCA). A MGC can serve not only as an in-context caption but also as an anchor for selecting a suitable caption from the five GTCs. As MGCs typically verbalize salient visual patterns in an image but may miss finer details, using them as anchors can lead to the selection of GTCs that highlight these salient patterns. Furthermore, the selected GTC may supplement interesting details about these patterns, potentially assisting VLM to generate superior captions during in-context learning. In the MGCA implementation, for each selected image, we measure the similarity between the MGC and five GTCs using CIDEr and select the GTC with the highest CIDEr score.
Figure 3: The line charts of various in-context captions with diverse image-selection strategies.
Figure 4: The line charts of various in-context images with diverse caption-assignment strategies.
4.1 Dataset and Implementation Details
MSCOCO. We evaluate the proposed strategies on MSCOCO dataset [14], which is the most widely used benchmark in image captioning. We used the Karpathy split [59] in the experiments, which contains 113,287/5000/5000 training/validation/test images and each image is associated with 5 human-annotated captions.
Implementation Details We employ the Open-Flamingo model [16] to test our strategies, setting the length penalty to -2.0 and a maximum generation length of 20. We follow Flamingo [6] to use 4, 8, 16, and 32 shots. We respectively use ViT-L/14 and as the vision and language encoders to extract image and sentence embeddings that used in SIIR-CLIP and SICR-CLIP. For MGC-TF, we train the standard Transformer encoder-decoder architecture on the MSCOCO dataset and use the checkpoints underwent 1000, 3000, and 170,000 iterations respectively. These checkpoints generate the captions with CIDEr scores of 66, 88, and 135 on the test set. We implement all experiments on a single RTX 3090 using FP16.
4.2 Results and Analyses
Given our varied strategies for image selection and caption assignment, displaying results for each configuration in table format, especially at 4, 8, 16, and 32-shot levels, could become overwhelming. For clarity, we've chosen to present results using line charts and histograms in the main paper, while detailed numerical outcomes are in the supplementary material. The line charts in Figures 3 and 4 illustrate trends as the shot number grows. Each subplot within these figures corresponds to a unique strategy for image selection or caption assignment. Furthermore, Figures 5 and 6 show histograms of average CIDEr scores for the various shot results. To facilitate comprehension, we first analyze the effects of caption qualities 4.2.1 and then of image selection strategies 4.2.2.
4.2.1 Effects of Caption Qualities
Figure 4(a)-(e) reveals that performance typically improves with an increase in shot numbers. However, the rate of improvement varies, or even declines, depending on the quality of the captions used. For instance, in Figure 4(a), using Ground-Truth Captions (GTC), the increase rates of the
Figure 5: The histograms of various in-context captions with diverse image-selection strategies.
Figure 6: The histograms of various in-context images with diverse caption-assignment strategies.
six image selection strategies surpass those in Figure 4(b) where MGC-TF@88 is used. Further, low-quality captions, such as MGC-TF@66, may become "toxic examples" which can misguide the Vision-Language Model (VLM) as the shot number increases. Next we compare different kinds of captions to figure out what characteristics of the captions affect the performance of in-context captioning.
Ground-Truth Captions (GTC) vs. Model-Generated Captions (MGC). As discussed in Sec 3.2, MGC exhibits two primary shortcomings when compared to GTC: poorer language and less descriptiveness. However, we will see that these two shortcomings do not always make MGC achieve poorer performance when compared to GTC. Specifically, we find that up to a certain level of descriptiveness, simpler sentence patterns are more easily recognized by the VLM, thereby improving caption generation.
Evidence for this can be observed in Figure 5 by comparing GTC with MGC-TF@135. Compared to the caption generated by MGC-TF@135, ground-truth caption has better language patterns, e.g., rich vocabulary and complex sentence pattern, and better descriptiveness. Then when the selected images cannot provide enough vision cues, i.e., when Random Selection (RS) is applied in Figure 5 (a), GTC outperforms MGC-TF@135. However, once the similarity-based retrieval methods like Similarity-based Image-Image Retrieval (SIIR) or Similarity-based Image-Caption Retrieval (SICR-CLIP) is used, the selected similar images offer useful visual patterns that help address the descriptiveness issue. Consequently, the VLM is more likely to recognize the consistent, simple patterns in MGC than the rich, diverse patterns in GTC and then generate better captions. Figure 7 (a) visualizes 2 examples about the above comparisons.