Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Models
Authors: Xu Yang
Yingzhe Peng
Haoxuan Ma
Shuo Xu
Chi Zhang
Yucheng Han
Hanwang Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: In-Context Learning
MultiModal
Primary area: natural_language_processing

Abstract: As Archimedes famously said, ``Give me a lever long enough and a fulcrum on which to place it, and I shall move the world'', in this study, we propose to use a tiny Language Model (LM), \eg, a Transformer with 67M parameters, to lever much larger Vision-Language Models (LVLMs) with 9B parameters. Specifically, we use this tiny \textbf{Lever-LM} to configure effective in-context demonstration (ICD) sequences to improve the In-Context Learinng (ICL) performance of LVLMs. Previous studies show that diverse ICD configurations like the selection and ordering of the demonstrations heavily affect the ICL performance, highlighting the significance of configuring effective ICD sequences. Motivated by this and by re-considering the the process of configuring ICD sequence, we find this is a mirror process of human sentence composition and further assume that effective ICD configurations may contain internal statistical patterns that can be captured by Lever-LM. Then a dataset with effective ICD sequences is constructed to train Lever-LM. After training, given novel queries, new ICD sequences are configured by the trained Lever-LM to solve vision-language tasks through ICL. Experiments show that these ICD sequences can improve the ICL performance of two LVLMs compared with some strong baselines in Visual Question Answering and Image Captioning, validating that Lever-LM can really capture the statistical patterns for levering LVLMs. The code is available at \url{https://anonymous.4open.science/r/Lever-LM-604A/}.

Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Models
Xu Yang 4:2g 12- Yingzhe Peng 12, Haoxuan Ma 122 Shuo Xu
Chi Zhang', Yucheng Han Hanwang Zhang
Southeast University
Key Laboratory of New Generation Artificial Intelligence Technology & Its Interdisciplinary Applications, (Southeast (ilvverstt),Miinseee of Education
3 Westlake University
Nanyang Technological University
(xuyang_palm, yingzhe. peng, haoxuan-ma, xushuo)&seu edu. cn
chizhang@restlake wawaakeeeedu.ccomm edu.cn, yucheng0020e. edg.s, hanvangzhangdntu.ee edw.ettuuedd.....
Abstract
As Archimedes famously said, "Give me lever long enough and fulerum on which to place it, and shall move the world", in this study, we propose to use tiny Language Model (LM), e.g., a Transformer with 67M parameters, lever much larger Vision- LanguaLe Models (LVLMs) with 9B parameters. Specifically, we use this tiny Lever-LM to configure effective mn-conext demonstration (ICD) sequences to improve the In-Context Learinng (ICL) performance of LVLMs. Previous studies show that diverse ICD configurations like the selection and or- dering of the demonstrations heavily affect the ICL performance, highlighting the significance of configuring effective ICD sequences. Motivated by this and by re-considering the the process of configuring ICD sequence, find this is mirror process of human sentence composition and further assume that effective ICD con- figurations may contain internal statistical patterns that can be captured by Lever- LM. Then a dataset with effective ICD sequences constructed to train Lever-LM After training, given novel queries, new ICD sequences configured by the trained Lever-LM M to solve vision-language tasks through ICL. Experiments show that these improve the ICL performance of two LVLMS compared with some strong baselines Visual Question Answering and Image Captioning, vali- dating that Lever-LM can really capture the statistical patterns for levering L.V.Ms. The code is available at
ICD sequences
Introduction
With the escalation in model size and training data [1H6], Large Language Models (LLMs) emerge the ability of In-Context Learning (ICL) (7-2). ICL, akin to few-shot learning [10H22) utilizes a few exemplary In-Context Demonstrations (ICDs) adapt LLMs new tasks without gradient updates. This achievement in NLP has inspired researchers to similarly enhance Large Vision-Language Models (LVLMs) with ICL capabilities [ 14. However, just in NLP, the effectiveness of ICL in LVLMs is significantly influenced by the configurations of ICDs, such as their selection and ordering [3]. Recent studies 26) have shown that this sensitivity in LVLMs further exacerbated by the multimodal combinatorial complexity of vision and language data.
In NLP, researchers employ various strategies to optimize in-context sequences to improve ICL performance, including retrieving representative examples as the ICDs [15 27 28 and re-ordering these ICDs based on specific principles [29] 21]. While these methods have shown improvements,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Infermnce
Lower Lif Cenvreann
Amprrinn
X31
Satroct Eye
Kii
Seaador
X
Thaabere Decade
Eibedding
Sample
(+) Traditionaa Two Steps method
(b) Our Lever-LM
Figure I: (a) The traditional ICD configuration methods separately select and order the ICDs, leading to sub-optimal ICL performance. (b) Our Lever-LM enables the step-by-ste generation of ICD configurations and simultaneously considers the selection of ICDs and the ordering of ICD sequences.
their application remains largely confined to NLP and is less explored in the lision-language domain. Moreover, as shown in Fig-Taa the independent operations of retrieval and reordering often result in sub-optimal outcomes. A critical reconsideration of the ICD sequence generation reveals that configuring an optimal ICD sequence should be coherent process. Instead of independently selecting and re-ordering, each ICD should be chosen conditionally based on the previous ICDs. This mirrors the sequential nature of human sentence composition, where each word is sequentially selected to ensure overall fluency. Such fluency can be characterized by temporal statistical patterns, which allows the design of statistical learning methods model and learn from data, where Language Model is one typical technique that demonstrates the effectiveness. This analogy supports the hypothesis that optimal ICD sequences may contain inherent temporal statistical patterns.
Although not explicitly stated, some previous studies NL.P work toward this direction by calculating statistical metrics like the perplexity [30] or the entropy BT discover what statistical character- istics a good prompt should have. However, the requirement to know the probability of each token when calculating these metrics limits their applicability in VL. This is because recently proposed LVLMs 113 [4 33 use continuous image patches rather than the tokenized discrete elements as the vision input, rendering these techniques used in NLP inapplicable in VL and the following two questions remain unaddressed: (1) whether effective VL ICDs exhibit certain statistical patterns and (2) whether such patterns can be leveraged to compose new ICD sequences for given query. This study aims to address these questions. Specifically. we employ tiny Language Model, e.g., a Transformer, to capture the inherent statistic patterns. This tiny LM is named as "Lever-LM" since it can lever/control much larger VLM by composing suitable ICD sequence. Compared with the classic LM,, the only difference is that the vocabulary of Lever-LM consists not of standard words, but of examples from the supporting set that will be used as ICDs.
Fig. (b) shows the training pipeline for Lever-LM. Initially, a "ground-truth" dataset is constructed to indicate which examples or their orders can form good ICD sequences. Specifically, we employ frozen LVLM evaluate if an ICD sequence facilitates accurate predictions for a given query, e.g., answering questions correctly or generating appropriate captions. Lever-LM is then trained to concurrently learn the selection and ordering of ICDs, streamlining the process by eliminating the need for two separate stages typical of previous methods. Our experiments, conducted with two LVLMSs- L.L.MsOOpnnFFiie Open-Flamingo 34 and IDEFICS [4 on classic VL lasks-mage Captioning and Visual Question Answering demonstrate that Lever-LM surpasses several strong baselines, including those that retrieve ICDs based on image similarity. These results confirm that effective ICD sequences contain inherent temporal statistical patterns and such patterns can be learned for composing new ICD sequences for test queries.
Besides the above-mentioned advantages, Lever LM emerges two interesting abilities. First, it has strong length extrapolation ability, e.g., when trained on a dataset with only 2-shot ICDs, Lever LM can generate 4 or more-shot ICDs that outperform several strong basclines. Second, Lever LM can construct a "golden" ICD sequence of 8 predetermined ICDs in a fixed order. This sequence can be uniformly applied across different test queries to assist LVLM in label generation, thereby reducing the computational overhead for configuring new ICD sequences for each query. Experiments in IC/VQA tasks show that "golden" ICD sequence achieves 6.91/1 .24 improvements compared to
Note: This method requires ground-trui truth labels and thus not be used the test stage.

---

strong baseline. In addition, we use exhaustive ablations, including applying different ways to construct training set and changing the architecture of Lever-LM, discover which factors and analyze why they will affect the ICI. performace.
2 Related Work
Models with In-Context Learning Ability. Prompt engineering enables Large Language Models (LLMs) to address downstream tasks without the need for fine-tuning [35] I 36]. A variant, ICL, enhances this ability by constructing prompts with few examples. This has been demonstrated in LLMs such as GPT-3 M. LLaMA [51, and MPT [37] Recently, witnessing such success in NLP, the
domain has also developed numerous LVLMs with prompt engineering abilities [X 33
39
B 40
44 and ICL ability like [4). Flamingo [3]. and IDEFICS 14] Among them, we use Flamingo and
IDEFICS the LVLMs explore the effectiveness of Lever-LM since they have stronger and more robust ICL ability by using better language encoders and more training date
Configuring In-Context Demonstrations. Although ICL assists LLMs in better adapting to down- stream tasks, its performance highly sensitive to the selection [T5] 19 20) and ordering (21 of ICDs. Numerous studies have explored diverse methods to select ICDs in the NLP field
(29 48).
For example, [15) selects ICDs based on the embedding similarity between ICDs and test samples where the embeddings are extracted from existing language encoder. Such a method is further developed by training an encoder specifically for selection
50 53
Regarding the ordering of ICDs, researchers calculate diverse statistical-basee metrics to measure the quality ICD configurations, e.g., the Minimal Description Length (29] and Global and Local Entropy [21]. Besides them, researchers focus more on discovering the statistical patterns of good prompts. For example, [30 uses perplexity to measure which prompts can better help LLMs perform task. Furthermore, (31] unifies diverse statistics-base prompt selection methods [54] 55 from the
perspective of mutual information and discover that mutual information or its variants can uncover certain statistical patterns of effective prompts. However, these statistical basdd methods require to calculate the token probabilities, making them infcasible to address continuous image patches, thus not be used in VL.
Besides these NLP studies, in VL, 24 and 255 explore diverse ICD configurations IC and VQA, while only the -auset methods are used for selecting ICDs and do not consider the ordering. In contrast, our Lever-LM can simultaneously learn how to select and reorder the samples and moreover, our Lever-LM is model-specific.
3 Lever Language Model
In this section, introduce how to build Lever Language Model (Lever-LM) for configuring the ICD sequence to lever a given LVLM. First, we briefly introduce the formulations of ICL for Vision- Language (VL) tasks. Then we introduce the construction of the dataset used train Lever-LM. After that, we show the architecture of Lever-LM and briefly discuss how to train Lever-LM and use it to configure ICD sequences.
The Formulation In-Context Learning (ICL). Given a query input x', ICL, predicts the corre- sponding output y' using well-trained foundation model M, conditioned on the concatenation of an in-context sequence S and this query. We denote in-context sequence with K-shot ICDs d as sk (di, .... dx), Then ICL can be formulated as:
y' + (M |
(1)
where PM denotes the predicted probability of M and repreessssntts the decoding strategy, e.g., beam search. For each d, it is selected from supporting set DDs (di,.., dv) ). where each sample d; = yt): x; and y; respectively denote the input and the corresponding label. is noteworthy that in diverse VL tasks, and y have different forms. For instance, in Image Captioning (IC), is the image and y is the caption; and in Vision Question Answering (VQA), x contains the image and the question, while y the answer.
Since Flamingo does not open-source the model, use an unofficial implementation. OpenFlamingo 34
3

---

(4.3))
[.22)
KBB
FXA
Train
Transformer Decoder Blocks
Select
Val
Dbp
(a) Data Construction Pipeline
Structure Lever-LM
Figure 2: (a): The pipeline of constructing DM- Darker color of S indicates a higher score given by 2 2 (b): Top: Lever-LM is two-layer Transformer. Bottom: Each input embeddings the sum of the random initialized learnable embeddings, the image and text embeddings extracted by CLIP. The dotted block means that some tasks do not exist the text input, e.g., IC.
Constructing the Training Dataset. To train Lever-LM for generating effective ICD sequences for given LVLM M, should first construct dataset DM containing high-quality ICD sequences for different query inputs. Simply, we use one VI. Paaaase-CCCC [56] from IC- to show how construct DM, which is shown in Fig. 2 Formally, given dataset 'D which is already split into the training part DR and the test part DE. we build DM only from DR- As Fig. 2 (a. shows, initially, we randomly select n samples from DR to form an anchor set A. Then for each
sample am {Im, Dm } E A, we construct a K-shot in-context sequence S for it. Then DM {(a), S (a2, S ), .... (am, SK)) where each training sample contains an query am and the corresponding K K-shot in-context sequence S.K.
To avoid confusion, we remove the subscript m in following texts. To construct SK {di, ..., dx). we need to select K-shot samples from the supporting set Ds, which is set to the complement set of AA DR: DR A. Meantime, we also need to decide which samples should be selected in turn. To achieve this, given the anchor sample sequence, e.g. h 1-shot S we need to know that after adding which sample d E Ds, the ICL performance improvement can be maximized by applying the given LVLM M:
{xx y and the partially constructed in-context
d arg max L (td, '.. IM(S .a) deDs
(2)
where IM is one kind of ICL performance measurement related to M. Note that Eq. 2 actually uses the greedy sampling method to select the samples every time, while we can use beam search here to further achieve a better solution. Additionally, to improve the diversity of the dataset, we will keep the top-b highest-scoring ICD sequences {S| .S .... SK } at the last iteration for each where is equal to the beam size. For example, when setting beam size to as shown in Fig. (a.3), can get 5 diverse high-quality ICD sequences for an anchor sample.
Intuitively, for diverse tasks, we can use the corresponding "golden measurement" as TM, e.g., setting it to CIDEr /57//rccrayy for IC/VQA. However, this strategy encounters two limitations. The first one is that for diverse VL tasks, we need diverse corresponding measurements, which is inconvenient. Second, some "golden measurements" may be impractical to deploy. For example, for IC, calculating CIDEr requires the LVLM to forward multiple times to sample an integral sentence, and then costs expensive time burdens to construct the dataset. While for VQA, accuracy is binary value (accuracy=1 when correct and when wrong), then maybe lots of candidates in Ds will make the accuracy change from 0 to I and then it is hard to judge which one of them is the most suitable one.
To overcome these two limitations, we use relatively general measurement as Ly. Formally, since we have the ground-truth results y of the anchor sample, we can use the given LVLM M to measure the prediction confidence of y given the input x and the in-context sequence SK;
LM(S" a) Panly SK .x) II, (' xx y (L:-11)).
In VI. tasks, the ground-truth label y (,),,...,, (T) is sequence, thus we can decompose the probability distribution into a series of productions. Then Eq. (2) selects a sample that can further maximize the prediction confidence given the query input and the current in-context sequence.

---

In implementation, Ds usually contains huge amounts of samples, e.g., Ds in COCO [56 contains about 10th samples. However, we need to calculate Eq. 2 for each d E Ds when selecting dg for each a E A. which means the whole process of building DM is quite time-consuming. To alleviate the cost, as shown in Fig. 2 (a.2), for each specific a, we narrow the set size by sampling much smaller subset Dg. e.g., containing 64 samples D {d' og .... d from Ds for selecting dx- We use diverse sampling strategies to construct this subset, e.g., retrieving some samples similar to a, and implement exhaustive ablation studies to explore which strategies are useful in Section 4.3
Training Lever-LM. After getting DM: we use to train Lever-LM, as Fig. b)) shows, it is a
two-layer tiny Transformer (58). The primary difference between Lever-LM and the traditional LM lies in the tokens of the vocabulary, whose tokens are the samples from the supporting set Ds, e.g., the first token corresponds to the first sample in Ds. Then given the query sample, the ICDs can be selected one by one based on the token distribution produced by the trained Lever-LM, just as when composing sentence, the words are selected one by one from the word vocabulary.
Besides the tokens from Ds, three special tokens are added into the vocabulary to help con- figure the ICD sequence, which are [BOS], [EOS], and [QUERY], respectively representing the beginning of sequence, the end of a sequence, and the query sample, Given data sample (S% = [d..... .... dx) ). x²) from DM where S is the ICD sequence and x' is the query input, we reformulate it into { [BOS], [QUERY] + + di,... dx, [EOS]] } where [QUERY] + x' denotes to add two embeddings. This reformulated sequence is input into Lever-LM for training.
To train Lev we should embed the tokens of the vocabulary to get dense embeddings. Since each token contains both image and text, we use the vision encoder (-)-) and the language encoder (-) of CLIP [59] to embed the image and text, respectively. Meanwhile, we add each of these embeddings with learnable part that is randomly initialized. Then for the i-th token d; = (4;, Ti) in the vocabulary where IT,, are the corresponding image/ text, its token embedding is e;;
e; = (I,) + (T,) + T.
(4)
Note that T. varies between IC and VQA tasks where it denotes caption in IC and question in VQA.
For test query z', we use the same vision and language encoders to embed it. For VQA, the image and question are embedded and summed, while for IC, only the image is embedded. Lastly, we use the cross-entropy loss for training maximize the probability of the k-th ground-trum token.
standard LM that given the previously h 1 tokens,
Configuring the ICD Sequence to Lever LVLM. After training Lever-LM, we use it to configure the ICD sequence. Given a query sample x', we initialize the input sequence as { [[BOS], [QUERY] + ezy) and then generate the ICDs one by one, where ex/ is the embedding of x' computed by Eq. (4. After iteratively sampling K-shot ICDs, we can compose the corresponding in-context sequence S for x! and then use Eq. (I) to implement the ICL.
4 Experiments
4.1 Datasets and implementation details
Our approach is evaluated on MS-COCO [56] for Image Captioning (IC) and VQAV2 60 for Visual Question Answering (VQA). For each corresponding dataset, we use the train split to construct the DM and use the validation split to evaluate the performance of ICD configurations generated by Lever-LM: More details are given in Appendix A
To get D.. we select 5000 samples to get the anchor set A. For each anchor sample, we randomly choose 64 samples to build the sub-supporting set Dg. The beam size for these processes is 5. To train Lever-LM, different strategies are employed for IC and VQA. In IC, the weight of CLIP model will be frozen, and an MLP adapter is introduced to its output. While, for VQA, the CLIP encoder remains trainable, and no adapter is appended. The training phase leverages the AdamW optimizer [61] and a cosine learning rate scheduler. We set the learning rate to 1 x 1 and the batch size to 128. We train our Lever-LM for 20 epochs. To implement ICI., we use OpennlamingoVV2999 K4 and IDEFICS-9B [4] as our L.V.Ms. We use beam search during inference where the beam size is set to 3. Besides, we set the maximum number of generated tokens as 20 in IC and 5 in VQA.
5

---

4.2 Results and Analyses
4.2.1
Comparison Methods
We compare Lever-LM with ICD selection strategies:
Random Sample (RS): RS constructs S by randomly selecting and ordering & ICDs from Ds.
Similarity- based Retrieval methods: To date, only few studies foeus on configuring ICD sequence for solving VL tasks [24] 25 where both studies show that, despite their simplicity, similarity- retrieval methods are effective for selecting ICDs. We therefore consider these strategies as current SOTA benchmarks to assess our effectiveness. | They form S by computing the cosine similarity between the query input x' and ICDs in Ds where CLIP is used to extract features. We follow (25) to sort examples in ascending order by their similarity to the query input, so the rightmost demonstration methods contain three variants: (1). Similarity-
the closest example. Similarity-ban Image-Image Retrieval (SIR)) We select & ICDs from Ds with highest image similarity to the query image. (2). Similarity-based Text-Text Retrieval (STTR): We select & ICDs from Dg with highest text similarity to the query text. This method is only applicable to VQA where question is used as text and not infeasible for IC. (3). Similarity- -based Image-Text Retrieval (SITR): We compute the similarity between query image and all text of d; E DD and select ICDs whose texts have the top-k similarities with the query image. For IC/VQA, we use caption/question for IC/VQA.
4.2.2 Main Result
The results for various ICD selection strategies are shown in Table IC and VQA. For Lever-LM, it is trained by DM whose ICD length is set to 2. Due to increased inference time with more shots, we do not test the inference results for 5- and 7-shots. The table shows the length interpolation and extrapolation ability of Lever-LM. Interpolation refers to performance with ICDs shorter than those in the training set DMM which contains only 2-shot ICDs, and is denoted as "Avg:1~~".. Extrapolation pertains to performance with ICDs longer than those in DM, represented as "Avg:3~8". The notation "Avg:I~8" indicates overall performance across 1 to 8 shots. Future analysis will focus on comparing these averages to minimize potential bias across methods.
Overall, Lever-LM achieves the best performance on most cases compared to other methods on both LVLMs. Notably, Lever-LM excels in Avg:1~2. Specifically, in VQA, Lever-LM surpasses the best performing SIIR method by 3.07 (48.75 vs. 45.68) and 0.57 (53.65 vs. 53.08) in accuracy on the IDEFICS and OpenFlamingo models, respectively. In IC, Lever-LM outperforms the best baseline, SHR,, by 6.03 (84.32 vs. 78. 78.29) CIDEr on OpenFlamingo. Similarly, for IDEFICS, Lever-LM achieves a higher CIDEr of 3.2 (89.57 vs. 86.37) compared to the best baseline, RS.
Moreover, Lever-LM has remarkable extrapolation abilities. Regarding Avg:3~8, 8, Lever-LM main- tains the top performance in both IC and VQA. Specially, on OpenFlamingo. Lever-LM outperforms SIIR with 0.8 higher CIDEr in IC (96.52 vs. 95.72) and 0.75 greater accuracy in VQA (52.59 vs. 51.84 14).. Meanwhile, on IDEFICS, compared with RS, Lever-LM achieves a 2.93 higher CIDEr in IC (105.79 vs. (6) and 0.49 higher accuracy in VQA (54.84 vs. 54.35). These results indicate that Lever-LM can effectively identify and utilize internal statistical patterns to compose longer, high-quality ICD sequences, even from a dataset comprising only two shots.
When we delve deeper into the results in Table D we find that the relative performance of similarity- based methods and RS varies by LVLM and task. For example, for IC, SIIR outperforms RS ( Avg:1~8: 89.91 vs. 88.48) when OpenFlamingo is used to implement ICL while SIR significantly lags behind RS (Avg: 1~~8: 88.19 vs. 97.36) when IDEFICS is used. Also, for VQA, STTR is comparable to RS (Avg:I~-8: 47.98 vs. 47.94) on OpenFlamingo while STTR is defeated by RS (Avg:1~8: 49.75 vs. 53.54) when IDEFICS used. These performance fluctuations demonstrates the instability of these heuristic- based methods. However, Lever-LM does not have such serious fluctuations where it outperforms both RS and similarity- -ased retrieval methods across various LVLMs and tasks average. Such observations also suggest that Lever-LM may capture the stable statistic patterns between ICDs.
Besides the above-mentioned advantages, Fig. 3 shows that Lever-IM is more robust to the Short-cut Inference brought by using similarity-based retrieval methods 24 25 For example, in (a) and
Note that we use different experiment settings from 24 and more details are given in Appendix A.4

---

Table 1: Results of diverse ICL methods on IC and VQA, where "OF" and "IDE" denote Open- Flamingo and IDEFICS, respectively. Lever-LM is trained by DM whose ICD length is set to 2.
Interpolation
Extrapolation
Avg:1~8
Shot
Shot 2
Avg:1~22
Shot 3
Shot
Shot 6
Shot 8
Avg:3-8
RS
73.32
82.95
78.14
87.72
93.65
95.81
97.42
93.65
88.48
SITR
66.05
77.69
71.87
83.46
85.05
89.84
93.57
87.98
82.61
OFIC IC
SIR
71.71
84.87
78.29
90.83
93.22
97.80
101.01
95.72
89.91
Lever-LM
80.02
88.63
84.32
93.41
96.06
97.26
99.35
96.52
92.45
RS
41.97
45.92
43.95
48.17
48.95
51.18
51.44
49.94
47.94
SITR
40.17
43.58
41.88
46.03
47.5
49.72
50.75
48.50
46.29
OF VQA
STIR
43.31
47.46
45.39
49.85
50.68
53.23
53.58
51.84
49.69
STTR
44.6
46.75
45.68
47.92
49.05
50.06
49.47
49.13
47.98
Lever-LM
46.66
50.83
48.75
51.91
52.15
53.29
53.01
52.59
51.31
RS
76.44
96.31
86.37
100.80
101.82
103.64
105.18
102.86
97.36
SITR
62.08
75.50
68.79
82.57
86.64
90.34
92.88
88.11
81.67
IDE IC
SIR
66.61
83.31
74.96
89.43
93.02
97.07
99.70
94.81
88.19
Lever-LM
78.70
100.45
89.57
104.52
104.86
106.44
107.33
105.79
100.88
RS
$2.40
53.21
52.81
53.47
53.70
$4.00
54.48
$3.91
$3.54
SITR
51.52
51.72
51.62
52.55
52.59
52.83
50.49
$2.12
$1.95
IDE VQA
STRR
52.87
53.28
53.08
53.77
53.92
54.81
54.88
54.35
53.92
STTR 48.13
49.32
48.73
49.43
50.11
50.58
50.93
50.26
49.75
Lever-LM
53.31
53.98
53.65
54.39
54.58
55.09
55.3
54.84
54.44
(b), all the ICD questions are yes-or-no type, which causes LVLMs to output "no" even when faced
with a "What" type question of the query sample. For IC in (c) and (d), SITR incorrectly outputs "London" even though the location is not explicitly indicated in the query image, while SIIR even leads LVLMs to directly copy the text from the ICD. Conversely, Lever-LM generates more diverse ICD configurations, thereby preventing misleading inferences.
4.3 Ablation Studies
We use ablation studies to explore the effects of diverse settings on our approach, including (1) diverse DM configurations; (2) diverse scorers TM in Eq. (3); (3) diverse LM structures; (4) DM with 4-shot ICDs; and (5) randomly ordering the ICD sequences generated by Lever-LM.
Diverse DM Configurations. We generate 2-shot DM by different settings to investigate the corresponding effects. Three factors are ablated: beam size by the number n samples in A: and the sampling method of Dg. including 3 methods: Random: Selecting randomly from Dy; Similar Text (Sim-T): Selecting the highest textual similarity sample with anchor sample a from Ds; Similar Image (Sim-l): Selecting the highest visual similarity sample with anchor sample from Ds.
Table 2 (3) (11) shows the results for different DM configurations on IC and VQA. We find that Lever-I.M can consistently improve the performance compared with the baseline RS As for length extrapolation capability, only Sim-T' gets lower score than RS. These comparisons confirm the robustness of our method.
27 RS in 1
Table (3) ~ (5) show that appropriately increasing the beam size can improve performance. Specif- ically, as increases from to 5, in Avg:1~8, CIDEr/accuracy increases by 2.79/1.73 for IC/VQA. This suggests that diverse DM encompasses a broader range of high-quality ICD configurations, which can help train better Lever-LM. However, an excessively large b can negatively impact performance. For instance, in VQA, the accuracy of b 10 decays 0.12 than b hypothesize this drop in performance is due to the introduction of lower-scoring ICD sequences with a large beam size, potentially misleading Lever-I.M during training.
Avg: We
Table (6) ~ (8) show that using more anchor samples can improve the interpolation performance in both IC and VQA, e.g., when n increases from 1000 to 5000, the Avg:l~2 CIDEr/accuracy of IC/VQA increases from 83.94/45.39 to 84.32/ 48.75. However, we find that on IC, although the interpolation performance increases when n changes from 3000 5000, the extrapolation performance decays, e.g., Avg:3~~8 decreases from 97.60 to 96.52. One possible reason is that using
'The RS means Random Sample ICD retrieval method which is mentioned in Section 4.2.1

---

Table 2: Results of diverse ablation studies on IC and VQA.
VQA
Avg:1~2
Avg:3~8
Avg:1~8
Avg:1~2
Avg:3~8
Avg:1~8
RS Lever-LM
78.14
93.65
88.48
43.95
49.94
47.94
84.32
96.52
92.45
48.75
52.59
51.31
1
79.91
94.53
89.66
46.47
51.13
49.58
5
84.32
96.52
92.45
48.75
52.59
51.31
b 10
84.96
97.12
93.06
48.58
52.49
51.19
1000
83.94
96.74
92,48
45.39
50.44
48.76
n 3000
84.13
97.60
93.11
47.56
51.11
49.93
5000
84.32
96.52
92,45
48.75
52.59
51.31
(
Sim-I
81.96
96.11
91.40
47.10
51.79
50.23
(10)
Sim-T
81.22
87.66
85.52
45.38
49.55
48.16
(11)
Random
84.32
96.52
92.45
48.75
52.59
51.31
(12)
CIDEr Scorer
87.93
93.52
91.65
(13)
Lever-LM LSTM
83.93
96.21
92.12
46.60
50.68
49.32
(14)
Golden-1
81.78
97.44
92.22
47.78
52.51
50.93
(15)
Golden-2
91.20
99.63
96.82
45.32
49.05
47.80
Table 3: Results of Lever-LM with 4-shot DM on IC and VQA.
IC
VQA
Model
Avg:1~4
Avg:6~8
Avg:1~8
Avg:1~4
Avg:6~8
Avg:1~8
RS
84.41
96.62
88.48
46.25
51.31
47.94
SITR
78.06
91.71
82.61
44.32
50.24
46.29
SIR
85.16
99.40
89.91
47.83
53.41
49.69
STTR
47.08
49.77
47.98
Lever-LM(4-shot DM)
87.35
97.96
90.88
48.56
52.68
49.93
Lever-LM((-shhot DM)
89.53
98.30
92.45
50.39
53.15
51.31
Eq. (3 to build D. may introduce certain in-domain bias which is beneficial for interpolation while detrimental for extrapolation on IC.
For different sample methods of constructing the D3 in table (9) (11), we find Random the best in both IC and VQA. We suppose this is because selecting similar ICDs with the anchor sample from Ds will damage the diversity. Previous study [62] in NLP validates that the diversity of the ICD sequences will also help improve the performance of L.LMs.
Diverse scorers TM for evaluating ICD sequences. To evaluate the quality of ICD configurations, we can use task-specific metrics as TM to build DM such as CIDEr in IC. Table 2 (2) and (12) compare the results between using prediction confidence Eq. (2) and CIDEr (12) L.. We find that using CIDEr achieves 3.61 higher than Confidence in Avg:I~2 suggesting that CIDEr can assign more accurate and reasonable score for ICD configurations. However, the length extrapolation capability decreases obviously, which is 3.0 lower than Confidence in Avg:3~-8, validating the robustness of Confidence scorer. Moreover, will cost more time to construct D. by task sssc-fpecifiic metric is used, e.g., CIDEr costs approximately 10 times of Confidence when constructing D-
Lever-LM, find that this still achieves excellent performance. For example, in IC, the overall performance improves by 3.64 (92.12 12 88.48) compared to the RS baseline, while in VQA, it is improved by 1.38 (49.32 vs. 47.94). However, due to the weak representation learning capability of LSTM, its performance lower than Transformer.e.g. the scores decrease by 0.33/1.99 in IC/VQA, respectively. Overall, these results suggest that effective ICD configurations contain internal statistic patterns which can be captured by different temporal learner.
Diverse LM Structures. Table (13) shows the results of using LSTM 6%

---

100 00
Aetteu glaas
Are proshersos tyggthough teriy?
brocoa thifod??
pirtored
conditone
you
Acsmn
SITR VOR
SEB WDD-BB----D--ee- KDD-LMan
soro strett and athens beniael
sranoo seara
eiematt of
texan unatin
Prelctee gouo pepple raed
Areaboocroonn
inllngg lis.
ona With phlks
Predecioon e double dicker
water
Predition Au partad
14) BITRvv.IGD
Figure 3: Visualizations of diverse ICDs configurations, where the first and the last ICDs are given due space limitation. We can find that Lever-LM use more diverse ICDs and thus not lead to short-cut inference.
Golden ICD Sequence. In ex-
periments, find that using high learning rate and not freez- ing the CLIP model may make Lever-LM converge to a specific solution that for any query input, the ICD configuration is fixed while still return good ICL per- formance. For example, in IC, best version, Golden-2, can outperform the non-Fixed case (Table 2 (2)) by 4.37 points in Avg:1~8. Such improvement suggest that if we do not have enough computation burdens configure diverse ICD sequence for each query, we can preserve one Golden ICD Sequence for the latter usage. However, also find that the performance of Golden ICD Sequence fluctuates significantly, e.g., Golden-2 poorer than RS in VQA in Avg: This also points out a new direction to study how to get more stable Golden ICD Sequence.
Table 4: Results of Random Order of Lever-LM generated ICDs.
2-Shot DM
4-Shot DM
VQA
IC
VQA
IC
Original
50.83
88.63
51.12
85.97
Random Order
50.42
88.56
50.63
85.77
Longer Few-shot DM- We further explore the performance of Lever-LM using 4-shot D.M- as presented in Table It is evident that Lever-LM continues to outperform other retrieval- bassd methods in Avg:1~8 metric. However, we observe notable performance reduction when Lever-LMy trained with the 4-shot DM compared to the 2-shot DM Specifically, for IC. there is a performance decrease approximately 1.57 in the Avg:1~88 metric when using the 4-shot DM to train Lever-LM, One possible reason is that when constructing DM some approximation operations are applied where sub-optimal ICD sequences are got. Then parts of statistic patterns may be salient is M that more easily captured by Lever-LM where using longer ICD sequences may further encourage Lever-LM to capture such patterns, and thus causing less effective ICD sequences than the shorter direction to study how to build D with longer and more robust ICD
DMM- This points out sequences for better training Lever-LM.
Random Order ICD sequence. To validate that whether Lever-LM captures effective ICD orders, we randomly rearrange the ICD sequences generated by Lever-LM trained with 2-shot and 4-shot Dy and then evaluate the performance the 2-shot and 4-shot ICD configurations, respectively, in Table 4 evident that the original order of ICDs generated by Lever-LM attains the highest score in both VQA and IC, validating that Lever-LM, can learn how order the ICDs.

---

Conclusion
After observing that configuring ICD sequence is mirror process of composing sentence, assume effective ICDs may contain statistic patterns that can be captured by temporal learner. Then we use tiny LM named as Lever-LM to capture such patterns for configuring ICDs lever LVLMs. To achieve this, we construct a dataset containing effective ICD sequences to train this Lever-LM. After training, we validate the effectiveness of Lever-LM by comparing with similarity-based retrieval methods and find that Lever-LM can capture the statistic patterns between ICDs. Extensive ablations are deployed discover which factors and why they will affect the results, which also pointing out few future research directions.
Acknowledgement
This work is supported by the National Science Foundation of China (62206048), the Natural Science Foundation of liangsu Province (BK202208) 19), the Young Elite Scientists Sponsorship Program of Jiangsu Association for Science and Technology (Tj-2022-027), and the Fundamental Research Funds for the Central Universities (2242024330035), This research work is also supported by the Big Data Computing Center of Southeast University.
10

---

References
[1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Souan-Tiien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[2] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. ArXiv preprint, (lbs/2205.01066 2022.
[3] Hyung Won Chung, Le Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Shayne
Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetueed language models. ArXiv preprint, abs/2210. 1416, 2022.
[4] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ArXiv preprint, abs/2204.00311 2022.
[5] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, htts/2302.1977// 2023.
[6] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai. Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. ArXiv preprint, abs/2210.02414 2022.
[7] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. ArXiv preprint, abs/2206.07682, 2022.
[8] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. ArXiv preprint, abs/2303.000 038466, 2023.
[9] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning, ArXiv preprint, abs/2301 2l022301,,0234, 2023.
[10] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In 2020 IEEE/CVF Conference Computer Vision and Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 8805-8814. IEEE, 2020.
[11] Da-Wei Zhou, Yu- Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan. Forward compatible few-shot class-incremental learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 9036- 9046. IEEE, 2022.
[12] Qiufeng Wang, Xu Yang, Shuxia Lin, and Xin Geng. Learngene: Inheriting condensed knowledge from the ancestry model to descendant models. ArXiv, abs/2305.0223 2023.
[13] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23788 2022.
[14] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
11

---

[15] Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeelJO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100-114, Dublin, Ireland and Online, 2022. Association for Computational Linguistics.
[16] Jincen Jiang. Qianyu Zhou, Yuhang Li, Xuequan Lu, Meili Wang, Lizhuang Ma, Jian Chang, and Jian Jun Zhang. Dg-pic: Domain generalized point-in-context learning for point cloud understanding. In European Conference on Computer Vision (ECCV). Springer, 2024.
[17] Chengwei Qin, Aston Zhang, Anirudh Dagar, and Wenming Ye. In-context learning with iterative demonstration selection. arXiv preprint urXiv:2310.098811 2023.
[18] Chengwei Qin, Wenhan Xia. Fangkai Jiao, and Shafiq Joty. Improving in-context learning via bidirectional alignment. arXiv preprint arXiv:2312.1705: 2023.
[19] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816-3830, Online, 2021. Association for Computational Linguistics.
[20] Feng Nie, Meixi Chen, Zhirui Zhang, and Xu Cheng. Improving few-shot performance of language models via nearest neighbor calibration. ArXiv preprint, abs/2212.02216,, 2022.
[21] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- une 1: Long Papers), pages 8086-8098, Dublin, Ireland, 2022. Association for Computational Linguistics.
[22] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. ArXiv preprint, 10ss)2205.10655, 2022.
[23] Sawan Kumar and Partha Talukdar. Reordering examples helps during priming- based few-shot learning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4507-4518, Online, 2021. Association for Computational Linguistics.
[24] Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Geng Xin. Exploring diverse in-context configurations for image captioning. ArXiv preprint, abs/2305.14800 14800, 2023.
[25] Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How to configure good in-context sequence for visual question answering. ArXiv preprint, abs/2312.015711, 2023.
[26] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint 1.Xi::2306.13499 2023.
[27] Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang. Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. Selective annotation makes language models better few-shot learners. ArXiv preprint, abs/2209.01973 2022.
[28] Eshaan Tanwar, Manish Borthakur, Subhabrata Dutta, and Tanmoy Chakraborty. Multilingual Ilms are better cross- cross-lingual in in-context learners with aligmment. ArXiv preprint, 005500000000 2023.
[29] Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. Self-adaptive in-context learning. ArXiv preprint, abs/2212. 10375, 2022.
[30] Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. ArXiv preprint, abs/2212.04037, 2022.
[31] Sohee Yang, Jonghyeon Kim, Joel Jang, Seonghyeon Ye, Hyunji Lee, and Minjoon Seo. Improving probability- based prompt selection through unified evaluation and analysis. ArXiv preprint, abs/2305. 14877, 2023.
12

---

[32] Dan Iter, Reid Pryzant, Ruochen Xu, Shuohang Wang, Yang Liu, Yichong Xu, and Chen- guang Zhu. In-context demonstration selection with cross entropy difference. ArXiv preprint. abs/2305.1477 14726, 2023.
[33] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[34] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre. Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. ArXiv preprint, abs/2308.01330, 2023.
[35] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. 2019.
[36] Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. ArXiv preprint, abs/2107.02137. 2021.
[37] MosaicMI. NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. Accessed: 2023-05-05.
[38] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. ArXiv preprint, abs/2304.10592, 2023.
[39] Bo Li, Yuanhan Zhang, Liangyu Chen, Jínghao Wang, Jingkang Yang. and Ziwei Liu. Otter: multi-modal model with in-context instruction tuning. ArXiv preprint, abs/2305.03726, 2023.
[40] Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 200-212, 2021.
[41] Congzhi Zhang, Linhai Zhang, and Deyu Zhou. Causal walk: Debiasing multi-hop fact verification with front-door adjustment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19533-19541, 2024.
[42] Congzhi Zhang, Linhai Zhang, Jialong Wu, Deyu Zhou, and Yulan He. Causal prompting: Debiasing large language model prompting based on front-door adjustment, 2024.
[43] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality, 2023.
[44] Yixiao Yuan, Yawen Huang, and Yi Zhou. Rethinking unified generative adversarial model for mri modality completion. In Anirban Mukhopadhyay, Ilkay Oksuz, Sandy Engelhardr, Dajiang Zhu, and Yixuan Yuan, editors, Deep Generative Models, pages 143-153, Cham, 2024. Springer Nature Switzerland.
[45] Tai Nguyen and Eric Wong. In-context example selection with influences. ArXiv preprint, abs/2302. 10042, 2023.
[46] Xiaonan Li and Xipeng Qiu. Finding supporting examples for in-context learning. ArXiv preprini, abs/2302. 13539, 2023.
[47] Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134-9148, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.
13

---

[48] Xinyi Wang. Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. In Thirtty-eventh Conference on Neural Information Processing Systems, 2023.
[49] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in- context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655-2671, Seattle, United States, 2022. Association for Computational Linguisties.
[50] Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. Unified demonstration retriever for in-context learning. ArXiv preprint, abs/2305.04322 043200, 2023.
[51] Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Compositional exem- plars for in-context learning. ArXiv preprint, abs/2302.05698, 2023.
[52] Liang Wang, Nan Yang, and Furu Wei. Learning to retrieve in-context examples for large language models. ArXiv preprint, abs/2307 .06644 2023.
[53] Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, and Zornitsa Kozareva. Improving in-context few-shot learning via self-supervised training. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 3558-3573, Seattle, United States, 2022. Association for Computational Linguistics.
[54] Chonghua Liao, Yanan Zheng, and Zhilin Yang. Zero-label prompt selection. ArXiv preprint, abs/2211. 1.04688, 2022.
[55] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 8086-8098, Dublin, Ireland, 2022. Association for Computational Linguistics.
[56] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCC 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[57] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 4566-4575. IEEE Computer Society, 2015.
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
[59] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021. 24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR, 2021.
[60] Yash Goyal, Tejas Khot, Douglas Summers-Star Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pages 6325-6334, IEEE Computer Society, 2017.
14

---

[61] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[62] Itay Levy, Ben Bogin, and Jonathan Berant. Diverse demonstrations improve in-context compositional generalization. ArXiv preprint, abs/2212.06800, 2022.
[63] Haşim Sak, Andrew Senior, and Françoise Beaufays. Long short-term memory based recur- rent neural network architectures for large vocabulary speech recognition. arXiv preprint vrXiv:1402.11222 1228, 2014.
[64] Andrej Karpathy and Fei-Fei Li. Deep visual-semantic alignments for generating image descriptions. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages
IEEE Computer Society, 2015.
[65] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[66] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642. 2013.
[67] Yongshuo Zong, Ondrej Bohdal, and Timothy Hospedales. Vi-iel bench: The devil in the details of benchmarking multimodal in-context learning. arXiv preprint arXiv:2403. 13164, 2024.
15

---

A Implementation Details.
A.1 Lever-IM Hyperparameters
We provide additional implementation details for each experiment in Table For all experiments, the batch size is set to 64, the warmup steps to 5% of total training steps, the scheduler as cosine scheduler, and the optimizer AdamW [61]. All experiments are deployed RTX 3090. All training processes are carried out with mixed precision and 2 RTX3090 GPUs. While for LVLM ICL experiments,
BF16 mode.
Table 5: Different settings of Lever-LM experiments, where the n is the number of anchor samples in A, is the beam size, and I is the length of ICD configurations.
Training Purameners
Dar Purameters
Avgal-88
Ir
weight docay
spuch
Freeze
Aclopter
Zohot (QpanFlmingg))
10-4
92.45
Zohoe Das (DEFICC)
10x
10x
20
5000
100.38
10
10x 10
20
10000
90.85
Lever LMLLSTM)
10x 10-3
10x 10-1
20
5000
92.12
CIDE: Scorer
10x 11-
10x 10-3
20
5000
91.63
X
Fixed Set-2
10x 10-3
10x 10-3
20
X
5000
96.82
2-hhot (QpenFlamingo)
10x11"
10x
X
X
5000
51.31
Anhot Dai (IDFFICS)
10x
X
5000
$4.44
4abux
10x
10x 10
20
X
16000
49.61
X
49.32
Fixed See-1
10x 10x
10x 10-3
20
X
5000
50,93
Fixed Set
10x 18-3
10x10-1
X
5000
A.2 Datasets
MS-COCO (56) is widely used in IC, which is divided into 118, 287 training, 5, 000 validation, and 5, 000 test image-caption pairs. Notably, each training image associated with five distinct human- human-annotated captions.
VQAV2 [60] emphasizes open-ended VQA tasks, which encompasses 437, 570 question auester answer pairs in its training split, supplemented by an additional 2, 143, 540 pairs in the validation split.
A.3 Prompt Template
For different LVLMs and tasks, the input format also varies. We followed the prompt templates provided by OpenFlamingo and IDEFICS for our experiments. We show the prompt templates in 6 In the VQA tests of IDEFICS, of the input. Therefore, our final input format of k-shot is: [instruction] + [ICDI prompt] + (ICD2 prompt] ... [ICDk prompt] [Query prompt].
additional instruction needs to be added at the beginning
A.4 Similarity-based retrieval method
In our study, we adopt the methodology proposed by 24 as a baseline, yet we incorporate several distinct experimental settings. Firstly, unlike their use of OpenFlamingoV/ (34] which employs LLaMA [5] as the underlying LLM, we utilize OpenFlamingoV2 based on the MPT LLM [37). This version of OpenFlamingo leverages a more extensive dataset and more advanced LLM, resulting in enhanced robustness and improved ICL capabilities.
Secondly, while they focus exclusively ICL strategies in IC, which lacks textual input, our research extends to VQA. VQA involves textual queries; hence, we have adapted the STTR retrieval method for sourcing samples with similar textual prompts ICDs, drawing inspiration from their approach.
Lastly, in the context of Image Captioning, our experiment utilizes the MSCOCO 2017 dataset, in contrast to their employment of the Karpathy split [64] of the MSCOCO 2014 dataset.
16

---

Table 6: Prompt Formats for IC and VQA tasks with placeholders.
INLM
Prompt Template
Instruction
IDERICS
Ongtton::C
VQA
Queatiou:<ip <p Saezt andver:<k>
proviee an uiswer question. image auswer.
OpenPlamingo
Oatgut:<D
Oseettoos<qp> Store anceer
Table 7: Results of diverse DM configurations IC and VQA.
Interpolation
Extrapolation
Avg:T--8
Avg:1=22
Shot
Shot
Avg:3~8
R8
73.32
82.95
78.14
87.72
93.65
955.5
42
93.65
88.48
8 =
75.67
81.15
79.91
90.10
92.93
95.92
99.16
94.53
89.66
52
92.45
IC
10
80.64
89.27
84.96
94.47
96.26
98.58
99.15
97.12
93.06
1000
79.55
$8.34
83.94
91.16
95.25
99.70
99.87
96.74
92.48
3000
79.30
88.96
84.13
93.28
97.66
99.34
100.15
97.60
93,11
5000
80.02
88.63
84.32
93.41
96.06
97.26
99.35
96.52
92.45
Sim-1
75.32
88.61
81.96
93.28
95.
97.71
98. 19
96. 11
91.40
Sim-T
75.78
86.66
81.22
84.39
14.227
88.47
93.53
87 66
85.52
80.02
88.63
84.32
93.41
96.52
R8
41.97
45.92
43.95
48.17
48.95
51.18
51.44
49.94
47.94
44.56
48.38
46.47
49.76
49.95
52.27
52.54
51.13
49.58
48.75
VQA
46.89
50.27
51.54
52.17
53.26
52.49
51.19
1000
44.66
46.12
45.39
47.96
50.15
52.09
51.56
50.44
48.76
3000
16.04
19.08
47.56
50.42
50.88
51.54
51.59
51.11
49.93
5000
46.66
50.83
48.75
51.91
52.15
52.59
Sim-1
45.00
49.19
47.10
50.49
51.38
52.64
52.66
51.79
50.23
Sim-T
44.30
46.45
45.38
48,
48.95
50.46
50.44
49.55
48 16
Random
46.66
50.83
48.75
52.18
53.29
53.01
$2.59
$1.31
Table CIDEr of diverse scorers on IC.
Interpolation
Extrapolation
Avg:1-88
Shot
Sbot
Avg:1~2
Sbot 3
Sbot
Sbot
Shot 8
Avg:3-88
RS
73.32
82.95
78.14
87.72
93.65
95.81
97.42
93.65
88.48
Confidence CIDE:
80.02
88.63
84.32
93.41
96.06
97.26
99.38
96.52
92.45
84.86
90.99
87.93
92.53
94.25
94.98
92.31
93.52
91.65
Table 9: Results of diverse LM structures on IC and VQA.
Inserpolation
Extrapolation
Axg:
Avg:1-22 1-22
Shot
Shot
Shor
Avg:3-.8
95.81
(ove--1MLLSMM)) Lever LMTTrnssoommrr
Lever LM/L) Lever LM(TansSocmer)
51.31
B Detail Experiments Result
In this section, we present detailed experimental results and ablation studies on OpenFlamingo and IDEFICS models. The results include CIDEr scores, CHAIR metrics, fine- fine-grained analyses, model size comparisons, inference time comparisons, and CLIPScore evaluations.
B.1 Detail Ablation Studies Results On OpenFlamingo
In Table 7 Table 8 Table results on OpenFlamingo.
9 Table 8 Table Table10aan and
and Table 11
we present all the detail ablation studies
B.2 CIDEr Results of Fixed Random ICD Sequence on Image Captioning
To evaluate the effectiveness of Golden-Set, we compare the fixed set learned by Lever-LM with the one constructed by random selection. Specifically, we use OpenFlamingo to randomly select 3 sets of k-shot ICD sequences with different seeds for the image captioning task and then conduct ICL
17

---

Table 10: Results of Lever-LM with 4-shot DM on IC and VQA.
Interplaatios
Extrapeiation
Arg:1-8
Avg:1-4
Aug:6-88
71:71
Lever-LM(6- shot Par)
75.73
85.97
91.28
96.41
$7.35
97.56
98.35
97.96
90.88
Lovos-LMM(2shot Par)
80.53
90:35
99.35
43.31
47.46
50.68
47.83
53.23
53.58
VQA
44.6
46.75
47.90
49.05
47.0 08
49.47
47.55
Leves LM(4 sbot Pa4)
44.55
48.05
50.65
50.98
48.56
52.43
52.92
$2.68
49.93
Leve--M(2-shot Par)
60
50.83
51.91
52.15
50.39
53.29
53.01
53.15
51.31
Table 11: Results of Fixed Set Lever-LM on IC and VQA.
Inkopelaass
Eutreitaies
Stor
Anp21-2
Skor
Sho:
SUK
11.71
14.37
18.20
141.3
93.22
135.11
98.32
$2.90
Foodd Suk-
15.41
38.14
01.13
02.48
14.20
05.23
189.26
97:44
72.23
N6.77
96.1
41.28
09.88
94.83
105.5
96.00
RS
41.90
65.82
13.05
03.3
18.90
51.02
51.41
09.34
47.98
SURR
43.31
47.49
45.39
29.33
$5.31
53.99
51.94
49.06
Wit Flonti fe
16:99
00.03
16.75
51.11
53.15
$3.34
$1.00
$2.54
$1.31
44.24
56.99
50.92
Table 12: CIDEr results of fixed random ICD sequences and Golden-Set on Image Captioning with OpenFlamingo.
Method
Avg:1~22
Avg:3~88
Avg:1~88
Random Fix-1
68.95
84.62
79.40
Random Fix-2
64.80
82.15
76.36
Random Fix-3
68.84
86.61
80.68
Golden-Set 91.20
99.63
96.82
Table 13: Fine- tine-grained analysis of diverse ICL methods on VQAv2 with IDEFICSV1.
Type
Method
Avg:1~22
Avg:4-8
Avg:1~88
Lever-LM
0.5623
0,5939
0.5833
RS
0.5402
0.5795
0.5664
Yes/No
SHR
0.5593
0.5853
0.5799
SITR
0.5229
0,5652
0,5511
Lever-LM
0.3192
0.3574
0,3446
RS
0.2852
0,3267
0,3129
Other
SIRR
0.2917
0,3349
0.3205
SITR
0.2643
0,3049
0.2914
Lever-LM
0.2782
0.3079
0.2980
RS
0.1323
0.1719
0.1587
Counting
SUR
0.1381
0.1917
0.1738
SITR
0.1051
0.1325
0.1234
tests. As shown in Table 12 can be observed that randomly selecting a fixed of ICD sequences results in relatively poor performance. The Golden-Set outperforms the best Fix Random set (seed 1) in Avg:1~8 by 16.14. This also demonstrates the importance of high-quality ICD sequences.
B.3 Fine-grained Analysis of Diverse ICL Methods for VQAv2
Table 13 provides a detailed comparison of different ICL methods on the VQAv2 dataset using IDEFICS 1 We compare the accuracy different question types in VQAv2 and find Lever-LM consistently outperforms other methods, highlighting its effectiveness in VQA tasks.
18

---

Table 14: CIDEr results of different Lever-LM sizes in Image Captioning with IDEFICSvI.
Model Size
Avg:1~2
Avg:4~8
Avg:1~8
1-layer Transformer (64.2M)
89.48
107.49
101.15
2-layer Transformer (67.4M)
89.58
105.79
100.05
layyer Transformer (73.7M)
89.12
106.10
100.44
Table 15: Accuracy results of diverse ICL methods on SST2 with 1we.1..--1.BB.
Method
Avg:1~2
Avg:4~8
Avg:1~8
RS
0.6227
0,6579
0,6438
STTR
0.6807
0.7312
0.7110
Lever-LM
0.7087
0.7450
0.7305
Table 16: The inference time of different ICL methods with IDEFICSv1.
Method
Retrieval Time (s)
SIR
0.317
Lever-IM
0.328
B.4 CIDEr Results of Different Lever-LM Sizes in Image Captioning
Table 14 shows the impact of different Lever-LM model sizes on CIDEr scores for image captioning on IDEFICSv1. As shown in Table 14 we evaluate 1-layer/4-layer Transformer decoder layers. We find that the size of Lever-LM has minimal impact on performance. We believe that capturing the ICD sequence distribution is simple task that can be learned by only a few Transformer Decoder layers. Moreover, our motivation is to use small model to enhance the ICL performance of large model, it is not appropriate to design Lever-LM to be too large.
B.5 Lever-LM in NLP domain.
To show Lever-LM general method, we train a Lever-LM in a NLP task. Specifically, we use Qwen1.5 [65] 1.8B and generate 2-shot ICD datasets for SST-2 (66]. which is a sentiment classification task. The accuracy results displayed in the Table 13 be observed that our Lever-LM outperforms the Random method and STTR in 1v and Avg: 4~~, demonstrating the potential of Lever-LM in NLP.
B.6 Inference Time Comparison between Lever-LM and SITR.
Table 16 compares the inference time between SIR and Lever-IM methods on IDEFICSv1. Both methods have similar retrieval times, indicating that Lever-LMM's performance gains do not come the cost of efficiency.
B.7 Performance on VL-ICL Benchmark Tasks
We test Lever-LM on two tasks of VL-bench 67 due to computation limitation and show the results in Table [7 can be found Lever-LM achieves higher performance than other retrieval-basee methods, validating the generalizability of Lever-LM.
B.8 Performance on IDEFICS+2-8B
We also validate Lever-LM's generalization ability using IDEFICSV2. IDEFICSv2 is an open-source model specifically designed for ICL. Besides, modern LVLMs with robust ICL ability belong to two mainstream architectures (1) Flamingo-based (using cross-attention to fuse vision and language, like Open-Flamingo or IDEFICSvI that are tested in our paper) and (2) LLaVA- (directly concatenating image and language tokens like IDEFICSv2). Thus, testing with IDEFICSv2 assesses Lever-LM's generalization across both architectures. Since IDEFICS directly concatenate vision
19

---

Table 17: Results of diverse methods on two tasks from VL-ICL benchmark with IDEFICSVI.
Task
Method
Avg:1~2
Avg:4~8
Avg:1~8
RS
0.145
0.209
0.188
VL-ICL CLEVR
SIR
0.170
0.260
0.230
Lever-LM
0.300
0.270
0.280
RS
0.1923
0.230
0.218
VI.-ICI OCRText
SIR
0.155
0.164
0.161
Lever-LM
0.262
0.241
0.248
Table 18: CIDEr score diverse ICL methods on Image Captioning with IDEFICSv2-8B.
Method
Avg:1~2
Avg:3~44
SUR
81.93
99.72
RS
87.34
106.07
Lever-LM
100.72
121.68
and text tokens, its input sequence will be longer than IDEFICSvI and 4-shot inference reaches our GPU limitation, thus we report the results of 1-4 shots. The results in Table[I]sshow that Lever-LM achieves better results than RS and SIIR. Note that RS outperforms SIRR here, one possible reason that IDEFICSv2 is more likely be damaged by short-cut inference due similar ICDs are used, while Lever-LM is more robust.
C Fixed Set ICD Configurations.
We present four ICD configurations of the Fixed Set. Figure F displays two ICD configurations for IC, and Figure displays the other two ICD configurations for VQA.
20

---

Caption: propeller
Caption: riding wave top surfboard.
airplane is in an airplane hanger:
Cuption: A mun skute bearding with another man louking
Cuption: a white sitting bore station.
Caption: A Large white polar bear swimming in a water tank
Caption: camousel horse has mane.
Caption: pizza with suuce cheese sausage ttting on lable.
Caption: A woman sitting couch her hands to her head.
Caption: A black and white image man his phone and toddler.
Caption: couple of ships sitting in middle of a body of water.
Caption: dinner plate has of different food on
Caption: Boats floating close tagether in calm body of water.
Caption: This a toilet with black top.
Caption: band curved vase sitting on blue cloth.
Caption: The bowl has half a grapefruit the plate has donut and the other hulf.
Caption: A black tray holding a bog cheese pizza.
Fixed Set- 1: IC
Fixed Set-2: IC
Figure 4: 8-shot ICD configurations visualizations of IC Fixed Set.
21

---

Q: How many different types vehicles ure shown? A:
Q: W What is the girl holding on to kite
Q: What the man weating over shirt? A: vest
Q: What hand is tennis player holding the racket? A: right
Q: What orange item on the floor? squid
is person's arm?
A: shärt
Q: Is there ramp in the picture? A: no
Q: Is the elophant being both?
Q: What is the person's back? A: bockpack
Q: ne they in the woods? A: yes
Q: Does the word "balance" in the refer scale? A: no
What coloris the room's walls? A: white
GROCERY
Q: coloris
Q: W hat color is bird standing on?
the thing the silting on? A: blue
A: red
What is in the bockground?
Q: W hat color are trees? green
A: sun
Fixed Set-1: VQA
Fixed Set-2: 2: VQA
Figure 5: 8-shot ICD configurations visualizations VQA Fixed Set.
D Limitation
One major limitation of our study is the strategy used to build DM is not optimal, which requires further improvement. This limitation is revealed by observing that the 4-shot DM performs worse than the 2-shot one, highlighting the need for a more effective approach in searching for longer ICD sequences. To address this, we plan to design function IM to evaluate the effectiveness of ICD sequences; and use better sampling strategies in Eq. 2 to avoid the ICD sequence deviating from the global optimum.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have elaborated on the contributions of our approach in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
we do not have theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We show all hyperparameters and detailed experimental procedures for all experiments in Appendix.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: We submit all code for reproducing all experiments of our paper in supplemen- tary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
hetpooliiigguddde/ooooooo wwweeeeeee/oeSeubmmimmo ssicnpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The number of our experiments is quite large. and the cost of repeating experiments relatively high.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide all computer resources information in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our research conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our approach is primarily used to enhance the ICL capabilities of the model without modifying the parameters of the large model. Therefore, it does not involve social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our approach is not related to this.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We respect all licenses of the model and datasets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not create assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research is not related to the crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "zjS9Zj9UkH", "reviewer": "Reviewer_sise", "summary": "The authors focus on configuring effective in-context demonstration (ICD) sequences to improve the In-Context Learinng (ICL) performance of LVLMs. The proposed Lever-LM enables the step-by-step generation of ICD configurations and simultaneously considers the selection of ICDs and the ordering of ICD sequences. Experimental results validate that Lever-LM can capture the statistical patterns for levering LVLMs compared with similarity-based retrieval methods.", "questions": "More comparisons with recent MLLMs are required.", "limitations": "See weakness", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The tokens of the vocabulary in Lever-LM are the samples from the supporting set. Given the query sample, the ICDs can be selected one by one based on the token distribution produced by the trained Lever-LM.", "weaknesses": "1. In lines 196-197, what is the reason for “randomly” choosing samples to build the sub-supporting set? I think this method is suboptimal, personally.\n2. Table 2 (15) uses fixed ICD configuration for any query input, and still returns significantly improved performance compared with other methods in IC. I think this phenomenon shows that the ICL method fails to work on IC. Or I am curious about the performance of “randomly” selected fixed ICD configuration compared with model selected.\n3. The experiments part is not convincing in the field of MLLMs. I hope to see results with more benchmarks and models (e.g., M-ICL[1]). To assess the generalization capability of their approach, it would be advantageous for the authors to evaluate their methodology using benchmarks that emphasize fine-grained analysis, such as MM-Vet and SEED-Bench, particularly focusing on segments that require in-depth evaluation.\n4. I think the performance between two order strategies are comparative in Table 4, personally. \n\n[1] What Makes Multimodal In-Context Learning Work?, CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Random Sampling.**\n\nThe goal of randomly selecting samples to form the sub-supporting set $\\mathcal{D_{\\mathcal{M}}}$ is to enhance training data diversity, promoting Lever-LM to capture complementary knowledge among ICDs. We initially deemed this to be a strategy sub-optimal, exploring alternatives like selecting similar texts and images (Lines 265-269). Surprisingly, Table 2 (9)-(11) shows random sampling as most effective. This finding (Lines 288-291) suggests that using similar ICDs to form $\\mathcal{D_{\\mathcal{M}}}$ may make Lever-LM generated ICDs contain redundant knowledge, hindering ICL performance. This also aligns with multiple studies underscoring the benefits of diversity for ICL performance [43,54,A,B].\n\n[A] Diversity of Thought Improves Reasoning Abilities of LLMs\n\n[B] In-Context Learning with Iterative Demonstration Selection\n\n**2.Fixed ICD Configuration.**\n\nActually, the existence of a good fixed set for IC proves that ICL works on IC. This is because ICL's main goal is to facilitate efficient learning of new tasks with minimal examples. To achieve this, one ongoing research direction in ICL is to find a minimal supporting set for ICD selection [C,D], and our discovery of a \"golden fixed set\" requiring only 8 ICDs is an extreme instance of this direction. Hence, our findings robustly suggest ICL's potential for IC.\n\nWe also follow your suggestion to compare the fixed set learned by Lever-LM with the one constructed by random selection.\nSpecifically, we use OpenFlamingo to randomly select 3 sets of k-shot ICD sequences with different seeds for the image captioning task and then conduct ICL tests. As shown in Table A in the rebuttal PDF, it can be observed that randomly selecting a fixed set of ICD sequences results in relatively poor performance. The Golden-Set outperforms the best Fix Random set (seed 1) in Avg:1~8 by 16.14 points. This also demonstrates the importance of high-quality ICD sequences.\n\n[C] Finding Support Examples for In-Context Learning\n\n[D] Compositional Exemplars for In-context Learning\n\n\n**3.1.Why VQA and IC.**\n\nIt should be stressed that our experimental settings are different from M-ICL [E] that only needs to forward LVLM for ICL inference, while we need to train Lever-LM by modifying various settings (Lines 262-292) to identify which factors influence the learning. Given limited resources, we focus on essential vision-language tasks, IC and VQA, to evaluate Lever-LM. IC and VQA also play important roles in LVLM ICL. First, they are commonly used for evaluating LVLM's ICL performance [12,13], even are specifically studied to help understand the LVLM ICL abilities [20,21]. Also, IC and VQA incorporate diverse vision-language skills: IC tests object, attribute, and relation recognition, while VQA covers diverse question types, integrating various capabilities like classification ('what is the object?') or counting ('how many of one object?'). Modern benchmarks for evaluating LVLM often derive from the IC (COCO dataset) and VQA (VQAv2 dataset). For example, SEED-Bench [F], MME [G] and MMBench [H] incorporate core characteristics from IC and VQA tasks, assessing various aspects such as scene understanding, instance identity, spatial relations, and diverse question types.\n\n[E] What Makes Multimodal In-Context Learning Work?\n\n[F] SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension\n\n[G] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models\n\n[H] MMBench: Is Your Multi-modal Model an All-around Player?\n\n**3.2.Fine-grained Analyses.**\n\nWe value your fine-grained analysis suggestions but find MM-Vet [I] and SEED-Bench not fitting our study. MM-Vet's limited scale (200 images, 218 questions) is not sufficient for effective Lever-LM training. SEED-Bench, designed for multiple-choice questions, does not match well with LVLMs that focus on word generation in ICL, requiring specific adaptations that are challenging to us given the rebuttal's time constraints. However, as previously mentioned, VQA and IC are appropriate for fine-grained analysis. For IC, we assess object hallucination using CHAIR [J], providing object-level results, where larger CH$_s$ and CH$_i$ values indicate more object hallucinations. As Table B\\&C shows, Lever-LM ICDs, compared to random sampling-based and similarity-based retrieval methods, achieve smaller hallucination scores because they contain fewer similar images to the query, preventing LVLM from short-cut learning. For VQA, we calculate the accuracy of three question types for fine-grained analyses: `Yes/No`, `Counting`, `Others` (Table D), which shows that Lever-LM significantly outperforms other methods in `Counting`, suggesting Lever-LM ICDs helps LVLM get stronger fine-grained object recognition ability.\n\n[I] MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities\n\n[J] Object Hallucination in Image Captioning\n\n**3.3.More Models and Benchmarks.**\n\nWe follow your suggestion to test Lever-LM on more models and benchmarks. Specifically, we examine the generalizability in VL-ICL-Bench (**A.1.2 of R.BDyQ**),  Qwen1.5 model in NLP (**A.2.1 of R.RjFC**), and by another LVLM architecture, IDEFICSv2 (**A.1.2 of R.BDyQ**).\n\n\n**4.Order strategies.**\n\nThe comparative results in Table 4 primarily stem from Lever-LM unifying the generation and ordering of ICDs. Lever-LM generates ICDs with complementary knowledge, enhancing the overall performance and diminishing the importance of ICD ordering. Thus, it is hard to mirror the significant improvements as the studies focusing solely on ICD ordering. Yet, Table 4 shows that Lever-LM's generated order outperforms random order given high-quality generated ICDs, demonstrating its effectiveness in ordering ICDs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iR5ka6otVA", "reviewer": "Reviewer_RjFC", "summary": "This paper proposes using a Tiny Lever-LM to assist in ICD selection for LVLM's ICL scenarios, thereby enhancing ICL performance without significantly increasing computational costs. Lever-LM unifies the modeling of multiple scenarios (VQA, IC) in complex multimodal ICL, eliminating the need for manually designed heuristic ICD selection strategies. Additionally, Lever-LM jointly learns ICD selection and ordering, achieving end-to-end learning of ICD sequences. Lever-LM achieves excellent performance across multiple LVLM models, significantly outperforming existing multimodal ICD heuristic selection methods.", "questions": "1. How would a smaller or larger Lever-LM affect the results?\n\n2. Can you show the input format of LVLM when doing ICL? Do Idefics will use the same format of OF?\n\n3. Can you evaluate Lever-LM in LLM? I believe it can better demonstrate the generalizability of the method.\n\n4. Compared with RS/SIIR methods, how about the inference time of Lever-LM", "limitations": "Please see the Weaknesses.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Lever-LM's structure uses only two layers of Transformer Decoder, making it a highly efficient model for generating ICD sequences.\n\n2. Lever-LM unifies the modeling of complex multimodal ICL scenarios, eliminating the need for manually designed heuristic ICD selection strategies. Furthermore, Lever-LM simultaneously models both ICD selection and ordering steps, achieving end-to-end optimization of ICD sequence modeling. Besides, It is the first attempt to establish the ICD selection and sorting task as a generation task. \n\n3. Lever-LM demonstrates robustness on multiple levels. First, the authors test it across various tasks and models, consistently outperforming the best ICD selection methods in the current multimodal context. Meanwhile, manually designed heuristic ICD selection strategies show significant performance fluctuations across different models. Additionally, they evaluate the use of different metrics for constructing ICD sequence datasets and experiment with different language model architectures, all of which show excellent performance.\n\n4. The authors validate that the sequence order generated by Lever-LM performs better than randomly generated sequence orders, proving that Lever-LM indeed learns the order information of ICD sequences.\n\n5. Lever-LM demonstrates interesting length extrapolation ability after being trained on a 2-shot ICD sequence dataset. It performs strongly even when generating 4-shot, 6-shot, and 8-shot sequences. This proves the efficiency of the Lever LM method: it requires only short-shot ICD sequences to exhibit excellent performance on long-shot ICL.", "weaknesses": "1. It has not been explored whether the performance of Lever-LM strongly depends on its size. \n\n2. It seems like LLMs can also use this method for ICL tasks. It may be better to evaluate Lever-LM in LLMs to demonstrate its versatility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Lever-LM with Different Sizes.**\n\nWe follow your suggestion to obtain different sizes of Lever-LM by controlling the number of Transformer layers and test these Lever-LMs on the IC. As shown in Table E, we evaluate 1-layer/4-layer Transformer decoder layers. We find that the size of Lever-LM has minimal impact on performance. We believe that capturing the ICD sequence distribution is a simple task that can be learned by only a few Transformer Decoder layers. Moreover, our motivation is to use a small model to enhance the ICL performance of a large model, so it is not appropriate to design Lever-LM to be too large.\n\n\n**2.1 Lever-LM in NLP.**\n\nWe follow your suggestion to train a Lever-LM in NLP. Specifically, we use Qwen1.5 [A] 1.8B and generate 2-shot ICD datasets for SST-2 [B], which is a sentiment classification task. The accuracy results are displayed in the Table F. It can be observed that our Lever-LM outperforms the Random method and STTR in Avg:1\\~2 and Avg: 4\\~8, demonstrating the potential of Lever-LM in NLP.\n\n[A] Qwen Technical Report\n\n[B] Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank\n\n**2.2. Generality in VL.**\n\nOur approach is actually a generalizable method in VL. Different VL tasks require diverse configuration strategies. For example, in IC, [20] proposes that the image and caption need to be matched. In VQA, [21] proposes the best ICD selection strategy involves sorting based on image similarity first and then question similarity. In other words, LVLM requires different heuristic ICD selection strategies for different tasks, and it is difficult to find a universal and effective configuration strategy. Our method, on the other hand, provides a generalizable solution that can directly adapt to different models and tasks.\n\n**3. The format of IC and VQA.**\n\nWe have detailed the prompt template in A.3.Prompt Template of Appendix. Our prompt template is entirely based on the settings described in the Open-Flamingo and IDEFICS.\n\n**4. The Inference Time of Lever-LM.**\n\nPlease refer to **A.2 of R.Djwv** where we show the inference time of Lever-LM and SIIR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZsAtTsqcnr", "reviewer": "Reviewer_Djwv", "summary": "This paper presents a novel approach called Lever LM, which uses a tiny language model to configure effective ICD sequences for LVLMs. The key innovation is leveraging the small Lever LM to select and order ICDs, improving the ICL performance of LVLMs on tasks like VQA and image captioning. The paper demonstrates that Lever LM captures statistical patterns in ICD configurations, leading to significant performance gains over existing methods.", "questions": "1.\tWill CLIP model as data encoder affect inference efficiency?\n\n2.\tTypo: In Figure 3, what ICD-LM means Lever LM?", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper proposes an innovative approach: training Lever LM to select ICD for in-context learning. \n\nTreating ICD selection as a sequence generation task is novel. The experimental results also demonstrate the effectiveness of this modeling approach. Compared to heuristic ICD selection methods, Lever LM demonstrates greater robustness, significantly outperforming other selection methods in ICL across multiple models and tasks, achieving optimal performance. \n\nIn addition, the method shows the surprising ability to perform k-shot learning despite being trained in a 2-shot setting. The golden fixed set is an interesting phenomenon, which points that finding golden fixed set may become an important research direction in the future.", "weaknesses": "1.\tThe model needs to be encoded with CLIP first. It's seemed CLIP is also a big model compare than Lever LM. \n\n2.\tThere is no comparison with traditional ICDs ranking methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. The role of CLIP**\n\nThe role of the CLIP model is to encode the data in the supporting set. Therefore, in practical applications, the dataset can be pre-encoded and stored locally. \n\n**2. Lever-LM Inference Time.**\n\nDuring inference, only two layers of Transformer decoders are needed, which results in low computational overhead. We show the inference time of diverse methods in Table G. It can be found that Lever-LM does not have a significant gap in inference time compared to the SIIR method.\n\n\n**2. Compared with other Ranking Methods.**\n\nIn Line 102, we mention that the ordering of ICD sequences has been primarily studied within NLP [17,24]. Their methods are mostly tested on datasets with limited labels (e.g., binary sentiment classification) or require calculating the probability of each input token. Transferring these methods to VL tasks presents two challenges: 1. VL tasks often include continuous image features, and modeling the probability distribution of continuous features is difficult. 2. VL tasks like IC or VQA are often open-ended word generation tasks, which do not have a limited label space. Facing these challenges, there is limited study explores how to order VL ICDs for improving ICL performance. In fact, we are the first to model the ordering of VL ICD sequences up until the submission deadline, thus it is hard to find suitable comparable methods. Table 4 demonstrates that when the quality of the selected ICDs is the same, the order produced by Lever-LM is optimal, indicating that Lever-LM can generate the proper order.\n\n**3. Typo in Figure 3.**\nWe apologize for the typo, and we will revise this part in revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CfOWsP20k8", "reviewer": "Reviewer_g8yL", "summary": "This paper proposes Lever-LM, a small language model designed to configure effective in-context demonstration (ICD) for improving the in-context learning performance of large vision-language models. The authors construct a dataset of effective ICD sequences to train Lever-LM, which then generates new ICD configurations for novel queries to solve vision-language tasks through in-context learning. Experiments on image captioning and VQA tasks demonstrate that Lever-LM outperforms strong baselines and can capture statistical patterns in ICD sequences to effectively leverage LVLMs.", "questions": "There are many studies arguing about the poor metrics (Rouge, CIDEr, BLEU, etc) of image captioning because the n-gram based metrics give higher scores for captions that have a similar format as the GT. This is especially true for the ICL setting because the model will follow the given ICD, and therefore the predicted caption will just be more similar to the given caption but not necessarily better in terms of accuracy, factuality, etc. Have you tried to use LLM-based evaluation for image captioning?", "limitations": "See above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces an interesting method using a small language model (Lever-LM) to configure in-context demonstrations for LVLMs.  Empirical results on image Captioning and VQA demonstrate the effectiveness of Lever-LM across these settings.\n- The paper includes a wide range of ablation studies exploring various aspects of the proposed method, such as different configurations for constructing the training dataset, different model architectures, and the impact of ICD sequence length. These studies provide valuable insights into the factors affecting the performance of Lever-LM.\n- The paper is easy to understand.", "weaknesses": "- In practice, we use ICL because we do not want to update the model parameters or use more computing, while this approach requires training a small model for ICD selection. Thus, the authors need to show that this method can generalize to more models (e.g., Qwen-VL, InternLM-XComposer2, IDEFICS2, GPT4, etc) and new tasks beyond VQA and captioning (e.g., [1]), and therefore show whether the additional computations worth it.\n- Zero-shot performance should also be shown as a reference to few-shot performance in Table 1, 2, 3, and more.\n- Lever-LM is trained on 2-shot and the authors show the extrapolation to up to 8 shots, and the performance is almost constant wrt. to the number of shots for extrapolation except for OF IC. This may indicate the small number of shots during training makes this strategy hard to generalize to more shots beyond, which limits more performance gain and applications such as many-shot ICL.\n\n[1] VL-ICL Bench: The Devil in the Details of Benchmarking Multimodal In-Context Learning. arXiv, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.1.Why training auxiliary model.**\n\nJust as you commented, ICL does not need to update the model parameters. However, this refers to LLM or LVLM, not to the auxiliary models used for selecting good ICDs. In fact, NLP researchers have developed various auxiliary methods to retrieve and order ICDs [41,42,44,45], while few studies have done so for VL ICL as of the submission deadline. Our work is a pioneering effort in this area. Furthermore, we are the first to apply the concept of treating ICD generation as a Language Modeling task, a strategy yet unexplored in NLP, to the more challenging VL field. \n\n**1.2. More benchmarks and models.**\nWe test Lever-LM on two tasks of VL-bench [A] due to computation limitation and show the results in Table I. It can be found Lever-LM achieves higher performance than other retrieval-based methods, validating the generalizability of Lever-LM. We also follow your suggestion to validate Lever-LM's generalization ability using IDEFICSv2 from the diverse LVLMs you mentioned, due to computational constraints. Our choice is based on two factors. First, IDEFICSv2 is an open-source model specifically designed for ICL. Second, modern LVLMs with robust ICL ability belong to two mainstream architectures : (1) Flamingo-based (using cross-attention to fuse vision and language, like Open-Flamingo or IDEFICSv1 that are tested in our paper) and (2) LLaVA-based (directly concatenating image and language tokens like IDEFICSv2). Thus, testing with IDEFICSv2 assesses Lever-LM's generalization across both architectures. Since IDEFICSv2 directly concatenate vision and text tokens, its input sequence will be longer than IDEFICSv1 and 4-shot inference reaches our GPU limitation, thus we report the results of 1-4 shots. The results in Table J show that Lever-LM achieves better results than RS and SIIR. Note that RS outperforms SIIR here, one possible reason is that IDEFICSv2 is more likely be damaged by short-cut inference due to similar ICDs are used, while Lever-LM is more robust. \n\nTo validate our hypothesis, we review examples of text generated by SIIR, such as the following:\n```\nSIIR Model Output [The first is the most similar ICD, and the last is the query]:\n   Caption:Many children are posing together outside of the building window. [ICD1]\n   Caption:A group of children are sitting together wearing dresses and suits and ties. [ICD2]\n   Caption:Many children are posing together outside of the building window. [Query Prediction]\nLever-LM Model Ouput:\n   Caption:School children cheer a tennis match with a pirate and giant tennis racket. [ICD1]\n   Caption:A boy in a baseball cap holding baseball mitt.  [ICD2]\n   Caption:A group of school children posing for a photo. [Query Prediction]\n\nGround Truth\n1. Many small children are posing together in the black and white photo. \n2. A vintage school picture of grade school aged children.\n3. A black and white photo of a group of kids.\n4. A group of children standing next to each other.\n5. A group of children standing and sitting beside each other.\n```\nIt is evident that the model directly copied the caption from the first ICD, and such errors are not uncommon. We speculate that the LVLM's coarse visual models cannot distinguish excessively similar images, leading the LVLM to treat the ICD and the query as the same image, resulting in the model copying the ICD's caption.\n\nAdditional, we test Lever-LM in NLP ( **A.2.1 of R.RjFC**) and show more fine-grained analyses based on IC and VQA (**A.3.2 of R.sise**). **A.2.2 of R.RjFC** also discusses why Lever-LM is a generalizable ICD configuration strategy in VL.\n\n[A] VL-ICL Bench: The Devil in the Details of Benchmarking Multimodal In-Context Learning\n\n**2.Zero-shot Performance.**\n\nWe evaluate the zero-shot performance in Table H, showing that when compared with zero-shot performance, all the ICL methods achieve significantly improvements. We will incorporate zero-shot results in all Tables in revision.\n\n**3.Constant performance of more shots.**\n\nMany-shot ICL performance is influenced by two factors: LVLM's capacity for addressing long input sequences and the in-context sequence quality.\nStudies on Open-Flamingo v2 and IDEFICSv1, Which are two LVLMs used in our experiments, indicate that these LVLMs exhibit limited ability with many-shot ICDs [12][13][29]. Specifically, these LVLMs are trained by 5-shot ICEs, meaning limited capacity for more shot inputs. Our findings, such as Table 1, also show limited performance gains from various retrieval-based methods using more-shot ICDs. Yet, Lever-LM outperforms these methods on average. This does not imply Lever-LM's generalization to more-shot cases is compromised. We appreciate your feedback and plan to test Lever-LM with LVLMs better suited for longer inputs.\n\n\n**4.ClipScore.**\n\nWe appreciate your suggestion about calculating the CLIPScore. High-quality captions generated from a model should hinge on two aspects: linguistic correctness and visual congruity. CIDEr and CLIPScore respectively values the former and the latter.\nThe CLIPScores of diverse methods are given in Table K. We can find that RS has higher CLIPScore than SIIR, while SIIR has higher CIDEr than RS (Table 1). One possible reason is that when using SIIR, LVLM might copy the captions of ICDs with similar images as the query and overlook the vision content of query image. However, since RS returns less similar in-context images as the query, the short-cut inference is alleviated and thus has higher CLIPScore. Furthermore, we find Lever-LM has the highest CLIPScore, suggesting Lever-LM generates dissimilar ICDs to the query while meantime help LVLM generate the captions which are more grounded to the images. These comparisons validate that Lever-LM can capture useful statistic patterns between high-quality ICDs instead of simply generating similar ICDs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We gratefully thank all the reviewers for their valuable and constructive feedback. We are pleased to see that the reviewers recognize our motivation: to use a tiny model to enhance the performance of in-context learning (ICL) for LVLMs. We are encouraged to see that they find our method is novel, interesting (Reviewer Djwv, g8yL, RjFC), our method is a unified method for ICL in NLP and Visual Language domain (Reviewer RjFC), our experiment is very detailed and effective (Reviewer g8yL, Djwv, sise, RjFC). 


We address the concerns and questions in detail below and have appended a PDF file with tables. **Note that to distinguish from the Tables and References in the submitted manuscript, the Table and Reference numbers in the rebuttal are marked with A, B, C, etc.**
Based on these comments, we have summarized the common questions and our responses as follows:

1. We conduct supplementary experiments on other tasks and LVLMs to demonstrate the general applicability of Lever-LM (Table F/I/J, to Reviewers sise, RjFC, g8yL).


2. We provide more analysis of Lever-LM, including a fine-grain analysis of VQA and IC and another metric to evaluate different ICL methods' performance on IC (Table B/C/D/K, Reviewer sise, BDyQ).


3. We show the inference time of Lever-LM and one retrieval-based method to show that the Lever-LM will not influence the LVLMs inference time (Table G, Reviewer RjFC, Djwv).

We also address other specific concerns in separate responses.

Meta-review: This work proposes training a small language model (couple of transformer layers) with "golden" in-context demonstration (ICDs) sequences, in order to (1) select, and (2) re-order in-context demonstration sequences for new queries. The ICDs so-produced are then input into vision language models (VLMs) for in-context learning.

This work generally received positive reviews, with reviewers appreciating:
(1) using the small-LM as an innovative/interesting approach for improving in-context learning.
(2) detailed experimentations, and ablations, demonstrating the proposed method to be robust to: (1) various tasks, (2) models, (3) number of examples, and (4) also applicable to LLMs / NLP tasks.

There are concerns regarding: extra-compute, which in-context learning in principle seeks to avoid, e.g., adaptation without fine-tuning, and requiring a sizeable dataset of in-context +/ golden examples. Further, no comparisons to other (learnt) ICD selection/re-ordering methods is provided. 

I urge the authors to include the experiments presented in the review discussions in their paper. Further, experiments addressing the weaknesses highlighted above would help strengthen the paper: e.g., comparison against directly fine-tuning the VLM with the same compute budget and data, and, comparisons with other ICD selection/re-ordering methods from NLP -- as the method has been demonstrated to be applicable to LLMs as well (in Table F).
Decision: Accept (poster)