MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
Authors: Zihan Chen
Song Wang
Zhen Tan
Jundong Li
Cong Shen
Venue: ICML 2025 poster (2025)
Keywords: In-context learning
Many-shot In-context learning
Large language models
Primary area: deep_learning->large_language_models

Abstract: In-Context Learning (ICL) empowers Large Language Models (LLMs) to tackle diverse tasks by incorporating multiple input-output examples, known as demonstrations, into the input of LLMs. More recently, advancements in the expanded context windows of LLMs have led to many-shot ICL, which uses hundreds of demonstrations and outperforms few-shot ICL, which relies on fewer examples. However, this approach is often hindered by the high cost of obtaining large amounts of labeled data. To address this challenge, we propose **M**any-Shot **A**daptive **P**seudo-**L**ab**E**ling, namely **MAPLE**, a novel influence-based many-shot ICL framework that utilizes pseudo-labeled samples to compensate for the lack of label information. 
We first identify a subset of impactful unlabeled samples and perform pseudo-labeling on them by querying LLMs. These pseudo-labeled samples are then adaptively selected and tailored to each test query as input to improve the performance of many-shot ICL, without significant labeling costs.
Extensive experiments on real-world datasets demonstrate the effectiveness of our framework, showcasing its ability to enhance LLM adaptability and performance with limited labeled data. Our code is provided at https://github.com/Chen-1031/MAPLE_ICL.

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
Zihan Chen Song Wang
Zhen Tan
Jundong Li
Cong Shen
Abstract
Unlabeled
Labeled
In-Context Learning (ICL) empowers Large Lan- guage Models (LLMs) to tackle diverse tasks by incorporating multiple input-output examples, known as demonstrations, into the input of LLMs. More recently, advancements the expanded con- text windows of LLMs have led many-shot ICL, which uses hundreds of demonstrations and out- performs few-shot ICL, which relies fewer examples. However, this approach is often hin- dered by the high cost of obtaining large amounts of labeled data. To address this challenge, we propose Many-Shot Adaptive namely MAPLE, novel influence-basee many- shot ICL framework that utilizes pseudo-labeled samples to compensate for the lack of label infor- mation. We first identify subset of impactful un- labeled samples and perform labulllaabelig on them by querying LLMs. These pseudo-labelee labeled samples are then adaptively selected and tailored to each test query as input to improve the per- formance of many-shot ICL, without significant labeling costs. Extensive experiments world datasets demonstrate the effectiveness of our framework, showcasing its ability to enhance I.LM adaptability and performance with limited labeled data. Our code is provided at https: //github. com/ 1031 /MAPLE_IPLLLLI
0.56
Date
0.37
GPQA
0.36
Accuracy
0.54
Accuracy
0.35
0.34
0.52
0.33
0.32
20
80 Number samples
40
60
100
20
40
60
80
100
Number of samples
Figure 1: Accuracies Date and GPQA datasets with different amount demonstrations. The LLM is Gemini 1.5 Flash.
Labuing
enhancing the performance of LLMs (Rubin et al., 2022; Lu et al., 2021). More recently, advancements in expanding the context windows of LLMs to accommodate a large number of input tokens have enabled many-shor ICL (Agarwal et al., 2024), where hundreds of demonstrations are incorporated into the input. Many-shot ICL has been shown to signif- icantly improve performance, particularly for complex or nuanced tasks, by providing richer task-specific informa- tion (Baek al., 2024).
real-
However, many-shot ICL faces critical limitation: the high cost associated with acquiring a large volume of labeled data as demonstrations (Li et al., 2024). This challenge is particularly pronounced in resource- cosbuure-conttrainee settings where manual labeling is expensive or infeasible (e.g., com- plex reasoming tasks), thereby limiting the applicability of many-shot ICL. An alternative strategy to use unlabeled samples as demonstrations, However, as shown in Fig. 1, the performance improvement becomes marginal and unstable in comparison to using labeled demonstrations.
1. Introduction
In-Context Learning (ICL) has emerged as fundamental ca- pability of large language models (LLMs) (Zhao et al., 2023; Chang et al., 2024), enabling them to perform diverse tasks with set of input-output examples, i.c., demonstrations, as input (Brown al., 2020; Wang et al., 2023; 2024b). In contrast fine-tuning strategies, ICL does not update model parameters, making it an efficient and flexible approach for
In order to effectively leverage unlabeled samples, a practi- cal solution is to perform pseudo-labeling on them, allowing these samples to serve as demonstrations for many-shot ICL without incurring significant human labeling costs. How- ever, this approach presents two key challenges: (1) Se- lection of unlabeled samples for pseudo-labelingg Given potentially large pool of unlabeled samples, identifying the most informative samples for pseudo-labeling is a crit- ical challenge. This is particularly relevant in tasks where unlabeled samples may contain limited information (e.g., question answering). Selecting the most beneficial unla-
'qqual contribution 'University of Virginia, Charlottesville, VA, USA AAiizoa State University, Tempe, AZ, USA. Correspon- dence to: Cong Shen <cong @ virginia.edus.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
beled samples is essential to ensure that pseudo-labeling provides meaningful additional information to enhance task performance. (2) Impact of unrelated demonstrations: With pseudo-labeled samples, the challenge remains in selecting the most suitable ones as demonstrations, tailored to each test query. Since not all pseudo-labeled and labeled samples are beneficial for inference, using unrelated demonstrations for a specific input query may involve unnecessary noise. Therefore, it is essential to adaptively select demonstrations to minimize the impact of irrelevant pseudo-labeles and labeled samples, ensuring that only the most informative and reliable ones are used as demonstrations.
for labeled data in many-shot ICL, thereby improving the feasibility of LLMs in real-world scenarios where labels are scarce. Through extensive experiments diverse datasets, we demonstrate the superior performance of our framework over other baselines.
2. Related Works
In-Context Learning. In-context learning (ICL) (Brown et al., 2020) equips large language models (LLMs) with the ability to leverage a handful of input-output demonstrations for reasoning. ICL has proven remarkably successful in han- dling complex tasks, including summarization (Jain et al., 2023; Baek al., 2024) and reasoning (Wang al., 2024a; Lee et al., 2024; Chen et al., 2024b).
In this work, we propose Many-Shot Adaptive Pseudo- LabEling, namely MAPLE, a novel influence-based frame- work to effectively utilize unlabeled samples for many-shot ICL. We consider the scenario where a smaller set of labeled samples and potentially large set of unlabeled samples are available, as it is affordable to mamually label several sam- ples in practice. Our approach tackles the two primary chal- lenges with the following designs: (1) Influence-Basee Sam- ple Selection for Beeud-LLabelingg We leverage the concept of node influence on graphs to identify the most impactful unlabeled samples relative labeled samples. By con- structing graph encompassing both labeled and unlabeled samples, we exploit their relationships to inform selection. This ensures the pseudo-labeled samples provide the neces- sary information for inference. (2) Adaptive Demonstration Selection: For each input query, we adaptively select labeled and pseudo-labeled demonstrations tailored specifically to the query. By identifying and incorporating samples with the most significant influence the test query, our approach avoids involving unrelated demonstrations and ensures the effective utilization of demonstrations. These designs col- lectively maximize the effectiveness of leveraging unlabeled samples, while minimizing reliance on costly labeled data and extending the applicability of LLMs various real- world tasks with only limited labeled samples. We conduct extensive experiments on various the results validate the effectiveness of our framework. Our main contributions are summarized as follows:
To effectively harness ICL, researchers have devised various adaptive strategies for selecting the most suitable demon- strations (Su et al., 2022; Chen et al., 2024a). Broadly, these methods can be grouped into learning-free and learning- based categories. The former uses heuristic strategies, in- cluding semantic similarity (Liu al., 2021) entropy (Lu et al., 2021), without actively querying the model (Zhao et al., 2021; Agrawal et al., 2022). Learning based meth- ods, in contrast, incorporate feedback from the LLM into a training loop. As a classic example, EPR (Rubin et al., 2022) applics contrastive learning and learns a score based on the probabilities of language model outputs. Moreover, CEIL (Ye et al., 2023) selects multiple demonstrations while considering their correlations, and IDS (Qin et al., 2023) iteratively select demonstrations based on zero-shot chain- of of-thhuuhh reasoning (Wei et al., 2022).
Many-Shot ICL. With advancements in expanding the con- text windows of LLMs (Li et al., 2024; Bertsch et al., 2024; Team et al., 2024), many-shot ICL has emerged as promis- ing approach, leveraging larger sets of demonstrations to significantly boost performance at the cost of longer input lengths (Li et al., 2023; Jiang et al., 2024; Huang et al., 2024; Chen et al., 2025). As a pioneering effort, Agarwal et al. (2024) explored the capability of scaling many ICL to thousands of demonstrations, showcasing its superior performance across a wide variety of tasks. While this work reduces human labeling costs by utilizing model-genee generated (i.c., pscudo-labeling) answers as demonstrations, it does not address the selection of unlabeled samples (for pseudo- labeling) or the adaptive selection of demonstrations for individual test queries. Back et al. (2024) further highlight the importance of using extensive demonstrations of LLMs in many-shot settings, in contrast to the careful selection of demonstrations. However, the limited effectiveness of existing selection methods stems from their specific design for few-shot ICL (Zhang et al., 2025). This underscores the need for novel strategies tailored to the unique challenges of many-shot ICI.
datasets, and
Novelty. We are the first to explore the capability of many- shot ICL under the pseudo-labeled setting. This novel perspective highlights the potential of leveraging abun- dant unlabeled samples for pseudo-labeling to alleviate the data bottleneck, broadening the applicability of ICL beyond reliance on labeled data.
Algorithm. We propose an influence-base based mechanism to select and pseudo-label only the most impactful unlabeled samples and adaptively select demonstrations for each test query, ensuring strong performance without extensive pseudo- labeling.
Practicality. Our approach significantly reduces the need
2

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
3. Methodology
samples. Here V is the set of nodes, and & is the set of edges. We utilize the entire demonstration pool (i.e., dataset) D = DD to build G, and each sample z E D is treated as a node v E V. For each node 24 E V, we identify the k closest nodes in terms of the relevance score r(u;, U2), as defined in Contriver (Izacard et al., 2021), and connect them as its neighbors. The relevance score is calculated from the dot product of the encoded representations of x; and zy:
3.1. Preliminaries
We begin with formulating the setup of ICL in this work. Consider dataset D = Dr U Du, where Dr= = ((r,,,e)] consists of a small set of labeled samples, and Du { contains large set of unlabeled samples. Here represents the textual input (i.e., query), and y is the corresponding label. Generally. L < U holds true, due to the substantial cost associated with human annotations.
r(u,, 0ÿ) = (fa(zz), fox;))
(5)
The goal of (few-shot) ICL is to select a set of demonstra- tions S _ Dr. acting as the additional input for a pre-trained language model M. The model M then utilizes 8 along with test query Tust to make predictions:
where 2; and 2 are the corresponding queries of u; and ej. respectively. fo(-) is the pre-trained encoder. Notably, as the unlabeled samples in Du do not contain labels (i.e., y), we only use the queries (i.e., a:) to calculate the relevance score. Based on the obtained relevance scores r(u,, vz), the adjacency matrix A of the habeled-unlabeled graph is obtained by connecting the k nodes with the largest rele- vance scores to each node ey. Formally, the entries of A are calculated as follows:
i M (S( (Test): Freat)
(1)
where (Tiss) denotes the set of selected demonstrations, conditioned on the test query Visst.
3.2. Overview
Aij = r(o u;), if ifr( 0,
E Top-k((r((s, 0x) +x))
otherwise,
As the labeled set Dr may not be sufficient for many-shot ICL, we propose to identify a subset of unlabeled samples from Dy and perform pseudo-labeling on these samples. The overall process is described in Fig. 2. We use Di: C Du to denote the set of unlabeled samples selected for pseudo- labeling. The pseudo-labeling process is executed by language model Mp. which assigns a pseudo-label ūن to each selected sample Ty:
(6)
where Top-k(-) selects the largest values in {r(us, 1;)) according to the relevance scores. This ensures that only the most relevant connections are retained in the constructed labeled-unlabeled graph G. In concrete, G' enables to capture the relationships among both labeled and unlabeled samples, which is critical for selecting impactful unlabeled samples for pseudo-labeling.
VI
ú = Mp(xy)) Vxz E Dij-
3.4. Selecting Unlabeled Samples for Pseudo-Labeling
Notably, [Dy] = P. where the size P iisa hyper-parameter and serves as the pseudo-labeling budget. The pseudo- labeled samples, along with the labeled samples in Dr, are then aggregated into Dr, referred to as the candidate demonstration set:
Notably, due to the prohibitive cost of labeling, the num- ber of labeled samples is significantly smaller than that of unlabeled samples, i.e., [Dz] < |Durl. To complement the limited labeled information in Dz. we propose leverage the concept of node influence, which describes the extent to which the representation of a node can be impacted by another node (Xu et al., 2018; Huang & Zitnik, 2020; Wang & Leskovec, 2020). We first provide the formal definition as follows:
Dr ((z),0;) xJ E DijUUD.
With demonstrations in DF, adaptively select a set of demonstrations ((Tea) C 'Dr for each input query Ftest E Dies. which serves as the many-shot demonstrations for inference. The model M then uses these demonstrations to make predictions for i
Definition 3.1. [Node Influence] The node influence from node v; to node v, is defined as [(,,, v,) = =/0v./vvlll where v, and V, are the node representations of v; and vj learned by the widely adopted neighborhood aggregation mechanism, respectively. Öv./öv, is a Jacobian matrix, and the norm can be any specific subordinate norm.
g M(S(ris)) F.aa).
3.3. Relationship Construction
In our framework, we aim to maximally utilize the in- formation from the labeled set Dr to decide which un- labeled samples to select for pseudo-labeling. Therefore, we first construct a sabeled-unlabeled graph G = EV, that captures the relationships among labeled and unlabeled
According to Definition 3.1, larger node influence indicates that the representation of a node can more easily affect an- other node, i.e., having a higher influence. We propose to consider unlabeled samples that have a higher influence on the entire set of labeled samples because these high-
3

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
Sample Selection for Pseudo-Labellnn Dataset D
Relationship Construction
Influence- Sample Selection
Labeled P-labeled
Candidate Dr
Pseudo- Labeling
Labeled
Unlabeled
Low
High
Adaptive Demonstration Selection Candidate Pool DD,
Relationship Construction
Influence-Bass Based Sample Selection
M Many-shot ICI. Inference
Selected Samples
Q Query
Query
Labeled O P-labeled
O QQuery
Low
High
Prediction
Figure 2: Overview of MAPLE framework. Given a dataset with small fraction labeled samples, we select unlaheled samples for pseudo-labeling using the proposed influence score. During inference, we adopt similar approach to select relevant samples from the candidate pool for each query, filtering out unrelated ones. The remaining samples are then used for many-shot ICL.
influence unlabeled samples are inherently more representa- tive of the underlying data distribution. By pseudo-labeling these impactful samples, enrich the demonstration pool with samples that reflect critical patterns and relationships within the specific dataset. We denote the node set as == VL U VU, where V and Vo correspond to samples in Dr and Du, respectively.
where Vy and V/, denote the node sets of unlabeled samples and labeled samples, respectively.
We rank all unlabeled nodes and select the top P nodes with the highest values of s(V,, v) for pscudo-labeling, denoted as Vū:
V
argmax NEVo.IVTI=P
E s(VL, v). VeVo
(9)
To effectively estimate the node influence of an unlabeled sample on the entire set of labeled samples, we propose the following theorem that provides lower bound for the node influence any set of nodes V in G:
The pseudo-labels for selected samples in Vo generated using language model Mp:
Ú = Mp(x), Vay E Di
(10)
Theorem 3.2. Consider the node influence from node u to node set V. Denote the geometric mean of the node influence all nodes in V as Tv(u) = TV T(u, vz), where vi is the i-th node V. Assume the node degrees are randomly distributed with the mean value d. Then,
where Di, denotes the samples that corresponding to nodes in Vū. The pseudo-labeled samples, along with the assigned pseudo-labels, are then aggregated with the labeled set Dr to constitute the final candidate demonstration pool Dy:
Dr (1,,
Vū UDL
(11)
E(log [v()) 2 log Ps(u, V) log d Is(u, V), (7)
This process ensures that the most influential unlabeled sam- ples are identified and heuuoo-abelld, resulting high- quality demonstration pool from which the demonstrations for all input queries can be selected.
where Ls(u, V) is the average shortest path distance be- tween and nodes in V. Ps(u, V) is the geometric of the numbers of shortest paths between and nodes in V.
We provide the proof in Appendix A. From Theorem 3.2, conclude that in order to select nodes that are most influential to set of nodes, we need to consider both the shortest path distance and the number of shortest paths.
3.5. Adaptive Demonstration Selection
Now Dr contains only labeled and pseudo-labeled samples, which exhibit high influence and can serve as demonstra- tions for many-shot ICL. However, it still remains unsolved which samples will ultimately contribute meaningfully to the prediction of a specific test query Trat by the LLM. To adaptively select the final demonstration set from Dr, we again utilize the concept of node influence. We first estab- lish pseudo-labeled graph G' (Tieea) (V",E') tailored
Specifically, for each unlabeled sample z E Du. which corresponds to e Vo. we define its influence score on the labeled set D,,
s( sVV,, v) log Ps(U, VL) log d. (v, VL),
(8)

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
for each test query Tesst G' (eas consists of all samples in Dr, along with Trest, thereby |V|| [Dyl + 1. The edges El in G" are constructed in the same pattern as the labeled- unlabeled graph, leveraging the relevance scores between any pair of nodes. Notably, as G' involves labeled sam- ples and pseudo-labeled samples, we calculate the relevance scores based on both the queries (i.e., x) and labels (i.e., y/:
and does not scale with the number of test-time queries. With more queries involved during the test, the computa- tional cost of the graph becomes negligible. As for adap- tive demonstration selection, we further note that adaptive demonstration selection is an optional component that offers a trade-off between efficiency and performance, which will be discussed in Section 4.5.
{ (Jo(2z, fo(2), 0,)), if if * Tiest (fo(zz)) fo(2))),
F(er, 03)
4. Experiments
otherwise,
(12) With the calculated relevance scores, we construct the pseudo-labeled graph G" (Iies)) The adjacency matrix is obtained as follows:
4.1. Experimental Settings
Baselines. In our experiments, we consider baselines that focus on how to select unlabeled demonstrations for pseudo- labeling to further improve many-shot ICI. performance, given a fixed set of labeled instances. We compare our approach against the following baseline methods: (1) Zero- shot: This method provides only the task instruction the LLM, without any demonstrations. (2) Few-shot: This ap- proach incorporates only the labeled demonstrations for the LLM. (3) Random: This method randomly selects demon- strations for pseudo-labeling. (4) RAG: This approach uses Contriever (Izacard et al., 2021) to compute embeddings of both the query and the unlabeled demonstrations, selecting the most similar demonstrations for pscudo-labeling. (5) RAG-Adapt: This approach adds an adaptive demonstra- tion selection process based on RAG based embeddings of the query and the candidate demonstration pool Dr.
F(ur. Uy): if 0/) Top-k (fF(0, (F())) (0,
A/3
0,
otherwise,
(13)
With the pseudo-labeled graph, we evaluate the influence of the labeled and lseude-labeled samples on the test query Trea to select demonstrations that are highly relevant to the query. We aim to select nodes in V" whose influence on the test node exceeds the average influence of labeled nodes. Formally, we estimate the influence of each node U E V" { on Biet, based on Theorem 3.2. Notably, the calculation becomes a specific case in Theorem 3.2, where the node set only involves one node. The score is denoted as s(1, Uess). In this manner, the demonstration set S (3ret) is then constructed as follows:
Datasets. We evaluate the effectiveness of our approach on eight datasets across four tasks. (I) Summarization: This task assesses the ability of models to generate con- cise and coherent summaries from articles. We use the widely adopted XSum dataset (Narayan et al., 2018) for evaluation, with the ROUGE-L score as our evaluation met- ric. (2) Reasoning: This task tests the models' capacity for complex reasoning. We evaluate performance on three challenging datasets (Date, Salient, and Tracking7) from the Big Bench Hard (BBH) (Suzgun et al., 2023), follow- ing the experimental setup of the Long-Context Frontiers (LOFT) benchmark (Lee et al., 2024). (3) Classification: In this task, we focus on Financial PhraseBank (FP) sentiment analysis (Malo et al., 2014; Wei & Liu, 2025) and a sub- set of challenging benchmark datasets (Li et al., 2024) that are specifically designed for ICL tasks with diverse classes and long inputs, including Banking77 and GoEmotion. (4) Question Answering: We evaluate performance on the Google-Prodd Proof QA (GPQA) dataset (Rein et al., 2023). a multiple-choice QA benchmark. The questions are designed to challenge graduate-level reasoning in subjects such as biology and chemistry. The detailed descriptions of the used datasets are provided in Appendix C.
S(Iwea) = argmax (Mer) vEVs
Σ
ste, Urs));
(14)
where Vvs = a [V%
E R is hyperparameter to control the percentage of samples we would like to include in S(Ties). The final demonstration set S(Zre) is then used as input to the pre- trained language model M for predicting the output of mest
Ueest = ((S(arr))) T.es).
(15)
3.6. Computation Cost Analysis
Our proposed method, MAPLE, performs graph construc- tion in both selecting relevant unlabeled samples and adap- tive demonstration selection. In this section, we analyze the computational cost associated with graph construction. Given a graph g = (V7, E), the graph construction requires the computation of the relevance score r among any pair of nodes, which will be ((V))). To compute shortest paths, we use breadth-first search for each node, and the cost is O(D) + [E]) = O(D)) as m = (kl Therefore, the whole shortest path computation cost is O(PL)DV)) No- tably, the above cost is only required once before inference,
Implementation. In our main experiment, we set k = 20 and a = 0.75, and we sample 1,000 demonstrations for
5

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
Zero-shot
Few-shot
Random
RAG
-
RAG-Adapt
MAPLE
GoEmotion
Banking77
Date
0.20
0.38
0.80
ROUGE-L
0.19
Accuracy
0.36
Accuracy
Accuracy
0.56
0.18
0.78
0.54
0.34
0.52
0.17
0.76
0.50
0.16
20
40
60
80
100
20
40
60
80
100
40
60
80
100
20
60
80
100
# of pseudo-labeled
# of pseudo-labeled
of pseudo-labeled
of pseudo-labeled
Salient
Tracking7
GPQA
FP
0.90
0.72
0.37
0.85
Accuracy
0.70
Accuracy
0.34
Accuracy
0.36
Accuracy
0.68
0.32
0.35
0.80
0.30
0.34
0.66
0.75
0.28
0.21
0.64
20
60 # of labuledlabeled
40
80
100
20
40
60
80
100
20
40
60
80
1000
20
40
60
80
100
# of pseudo-labeled
pseudo pseedo-labelle labeled
of
of pseudo-labeled
Figure 3: Performance comparison various sample selection strategies many-shot ICL setting using Gemini 1.5 Flash across multiple datasets. 'Zcro-shot' refers to the query LLM being provided only with the task instruction. We randomly selected 20 labeled samples to construct Dr, and the results obtained using just Dr, are presented as 'Few-shot', Based on Dz, we compare MAPLE with other pseudo-labeling baselines with different Di.
labeling and 300 for testing. For datasets with fewer than 1, 1,000 training samples or fewer than 300 test samples, we use the entire dataset. We randomly select 20 demonstra- tions to form Dz. Unless specified otherwise, we eval- uate the many-shot ICL performance of the Gemini 1.5 Flash (Team et al., 2024) model with 1M token context length. We apply the Contriver (Izacard et al., 2021) as fo(.). We conduct experiments with pscudo-labeled sizes ranging from 20 to 100. For most tasks, while increasing the number of demonstrations further improves performance, many-shot ICI. reaches sufficiently good performance with around 100 to 2 demonstrations (Agarwal et al., 2024). The prompts used to elicit responses from ICL are provided in the Appendix B. Each experiment is run five times, and the average performance is reported.
on Complex Tasks: MAPLE demonstrates particularly strong performance on complex datasets such as Banking77, Data, and GPQA. This suggests that in tasks requiring nu- anced understanding, our framework's 's ability select ap- propriate unlabeled samples provides clear advantage over baseline methods. 2 Limited Gains in Certain Datasets: The inclusion of additional pseudo-labeled samples does not always lead to performance improvements. For datasets like Tracking7 and XSum, the limited benefits can be at- tributed to low-quality pseudo-labeles samples that cannot effectively assist in the inference process. ? Effectiveness of ICL Settings: Across all datasets, the few-shot setting consistently outperforms the zero-shot setting, demonstrat- ing the value of in-context learning. Furthermore, involving more pseudo-laheled samples selected by our framework leads to notable performance enhancements, underscoring the effectiveness of leveraging lebeled data in many- shot ICI. 5 Performance Degradation with More Pseudo- labeled Demonstrations: For certain tasks, such Track- ing77 and Salient, we observe a performance drop as the number of pseudo-labeled demonstration increases. This can be attributed to the fact that as more demonstrations are included, the influence of noisy labels becomes more pronounced. Additionally, each query does not necessar- ily require large number of samples, and the inclusion of irrelevant examples can degrade performance.
4.2. Main Result
As shown in Fig. 3, we evaluate our method, MAPLE, in comparison to all baselines across eight datasets. From the results, we observe the following: Superior Perfor- mance Across Datasets: Our method consistently outper- forms all other baselines across all eight datasets, showing significant improvements with various numbers of pseudo- labeled demonstrations. This highlights the effectiveness of our framework in selecting suitable unlabeled samples for in shoy ICL. @ Exceptional Results
6

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
Few-shot
RAG
MAPLE
Zero-shot
Few-shot
RAG
MAPLE
Random
RAG-Adapt
GPQA
Date
0.58
Banking?7
0.64
Date
0.37
0.82
0.56
0.62
Accuracy
Accuracy
0.81
0.36
0.54
Accuracy
Accuracy
0.60
0.80
0.52
0.35
0.79
0.58
0.50
0.78
0.56
1:5
1:2
1:1
2:1
5:
13
1:2
1:1
2:1
5.1
of p-labeled: # labeled
of p-labeled # of labeeee
50
100
150
200
50
100
150
200
# of pseudo-labeled
# of pseudo-labeled
Figure 5: The results of varying the fraction of pseudo-labeled demonstrations. We fix the size of the candidate demonstration pool Dr as 120 and adjust the proportion of pseudo-labeles sam- ples, while randomly selecting the labeled samples.
Figure 4: Results of MAPLE compared to baselines two datasets with a larger number of demonstrations. We increase the size of Dr to 100 and compare performance for different sizes of Di (50, 100, 150, and 200).
4.3. Impact of More Demonstrations
4.5. KV Cache Ablation
Given that long-context LLMs, such as Gemini 1.5 Flash, can process over 100 million tokens, we extend our experi- ments by increasing the number of labeled demonstrations 100, while varying the number of pseudo-labeled demon- strations from 50 to 250. According to results presented in Fig. 4, we first observe that incorporating a larger number of demonstrations leads to improved performance, which validates the effectiveness of many-shot ICI. Furthermore, MAPLE consistently outperforms all baselines as the num- ber of demonstrations increases, indicating the scalability of MAPLE in scenarios with large number of demonstrations.
In MAPLE, we provide personalized demonstrations for each query as the many-shot ICL prompt. which can be time-consuming during inference. In this section, we exam- ine variant of MAPLE that omits adaptive demonstration selection. In this setup, the labeled and pseudo-labeled demonstrations are fixed across all queries, allowing us to apply the KV Cache (Pope et al., 2023) to cache the demonstrations in the LLM before inference for improved efficiency. To illustrate the effect of KG Cache in our frame- work, we provide a detailed analysis regarding the FLOPs with KV Cache in Appendix D.
4.4. Results of Different LLMs
Furthermore, we investigate the trade-of off between efficiency and performance of our framework MAPLE when using KV Cache. Particularly, we conduct experiments on datasets XSum and Date. We additionally consider using various numbers of pseudo-labeled demonstrations, and report the performance and inference time. From the results presented in Table 2, we observe that applying KV Cache can re- duce inference time as expected. Moreover, the efficient improvement is more significant with larger number of pseudo-labeled demonstrations. However, including more demonstrations as input for each query introduces more noise and irrelevant information, which negatively impacts performance. Moreover, we also note that the efficiency gains from KV Cache are not significant. This is potentially due to the time required for the internal loading of the KV cache in API-based models, such as Gemini 1.5 Flash.
In our main experiments, we use Gemini 1.5 Flash. In this section, we adopt a stronger L.M,, Gemini 1.5 Pro, to show- case the performance of MAPLE under different LLMs. The experimental results in Table 1 highlight several key observations. First, the proposed MAPLE method consis- tently outperforms the baseline across all tasks and LLM variants, demonstrating its ability to effectively leverage pseudo-labeled demonstrations. For instance, in the Bank- ing77 task, MAPLE achieves an average accuracy improve- ment of 1.7% with Gemini Pro and 1.5% with Gemini Flash compared to the baseline. Second, MAPLE exhibits no- table advantage in scalability, showing steady improvements in performance as the number of pseudo-labeled demonstra- tions increases. This trend is particularly evident in the Date and GPQA tasks, where MAPLE achieves its highest accuracy at 100 demonstrations, outperforming the baseline by significant margin. Finally, the stronger Gemini Pro model consistently amplifies the performance of MAPLE across all tasks, indicating that the method's benefits are further enhanced when paired with more advanced LLMss These observations validate the robustness and adaptability of MAPLE in varying contexts.
4.6. Fraction of Pseudo-labeled Demonstrations
In this subsection, we conduct experiments to evaluate the impact of noisy pseudo-labeled data on many-shot ICL per- formance. In this manner, we further assess the quality of demonstrations selected by MAPLE in comparison to other baselines. In particular, we fix the total number of
7

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
Table Results of many-sho across three tasks (extreme classification, reasoning, and question answering) different
LLMs (Gemini 1.5 Pro and Gemini 1,5 Flash), We report the performance in % under different numbers of [ MAPLE consistently
outperforms the baseline across all tasks and LLM variants, demonstrating its ability to effectively leverage pseudo-labeled demonstrations.
of Pscudo-labeled Demonstrations
Task
Model
Method
Avg.
20
40
60
80
100
Random
76.7 + 3.9
76.0 + 4.4
75.6 + 3.1
75.4 + 3.7
77.8 + 3.2
76.3
Gemini Flash
RAG
77.0 2.3
76.8 2.3
76.4 2.4
76.3 3.4
78.3 4.1
77.0
MAPLE
77.0
78.3 2.6
2.6
78.1 2.7
80.8 3.4
78.5
Banking77
Random
74.7 2.3
76.7 + 1.9
79.3 + 2.2
78.5 2.0
78.3 2.1
77.5
Gemini Pro
RAG
76.3 1 2.5
78.3
78.7
79.3 2.8
78.8
2.5
78.3
MAPLE
79.3 = 2.2
79.7 + 1.5
80.3 + 2.3
81.3 1 1.5
80.7 = 1.9
80.3
Random
52.1
52.7 +
53.3 0.6
54.4 1.6
56.4 1.0
53.8
Gemini Flash
RAG
52.7 1.0
52.2 + 1.5
1..0
54.4 1.4
55.6 1.8
53.6
MAPLE
54.3 1.2
54.8 1 1.3
55.7 1.7:100 0.9
55.6 1 1.8
58.4 2.4
55.8
Date
Random
64.4 + 1.0
66.2 1 1.2
66.8 + 1.8
67.2 1.1
68.0 1 0.8
66.5
Gemini Pro
RAG
66.4 -:4:1:0 1.6
66.8 + 1.5
66.0 + 1.2
67.2 0.5
68.4 1 1.4
67.0
MAPLE
67.6 0.8
67.6 1.7
67.2 0.6
68.8 1.0
3.0 68.8
68.0
Random
36.2 + 0.6
35.1 + 1.9
34.6 3.1
2.3
33.1 + 2.0
34.7
Gemini Flash
RAG
35.7 1.2
34.7 + 1.3
34.8 0.5
32.7
33.8
1.0
34.3
MAPLE
37.7
37.0 0.4
36.9
3.9
37.4
37.1
GPQA
Random
41.4 0.8
41.4
43.3 + 2.0
42.4 = 2.0
42.9 1.1
42.3
Gemini Pro
RAG
43.9
2.5
41.9
41.9
43.3
42.9
MAPLE
44.4 1 2.3
44.4 1.0
44.9 1 2.0
43.8 1 0.6
43.9
1.6
44.3
Table 2: Comparison of MAPLE with its variant, which removes the adaptive demonstration selection component. We report the average performance and the inference time (s) per query two datasets XSum and Date.
Random Random-R Salient
RAG RAG-R
MAPLE
MAPLE-R
GPQA
100
100
Task
KV
# of Pscudo-labeled
20
60
100
# pseudo- lall--lalleull
# of labedo-labeled
80
XSum
0.200/1.89
0.191/3.57
0.198/4.08
0. 199/1.999
0.189/3.74
0.195/4.50
60-
250
300
40
Date
0.584/0.94
0.596/1.92
0.636/1.96
0.578/0.91
0.589/1.99
0.622/2.04
20
20
0.65
0.70
0.75
0.34
0.36
0.38
Accuracy
Accuracy
demonstrations at 120 and adjust the fraction of labeled pseudo-labeled demonstrations (rang- ing from 20 to 100 labeled). The results are presented in Fig. 5. We can obtain the following observations: (1) The experimental results demonstrate that MAPLE consistently outperforms baseline methods across all fractions of labeled and pseudo-labeled demonstrations, highlighting effec- tiveness in selecting high-quality pseudo-labeled data. (2) Increasing the fraction of labeled demonstrations generally improves performance, as higher-quality labeled data pro- vides stronger guidance. (3) When the number of labeled demonstrations decreases, appropriately selected pseudo- labeled data compensates for the loss, maintaining or even improving performance. This is particularly evident cases where pseudo-labeled data provides additional information
Figure 6: Impact of the order of the two sources of samples. Green dots represent methods with the default order, where labeled demonstrations appear first, and pseudo-lahcled demonstrations are positioncd closer to query. Red dots represent methods with the order reversed, where pseudo-labeled demonstrations appear first and labeled demonstrations are positioned at the back.
demonstrations
to offset the reduction in labeled examples, showcasing the robustness and adaptability of MAPLE in leveraging both labeled and pseudo-labeled demonstrations effectively.
4.7. Study of the Order of Demonstrations
In our main experiments, we put labeled demos the front of the input and pseudo-labeled demos the back of the input and closer to the query. We examine how the or-
8

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
der of labeled and pseudo-labeled demonstrations impacts many-shot ICL performance. The baseline setup places la- beled demonstrations at the front and pseudo-labeled ones closer to the query, while the alternative swaps their posi- tions. Results Fig. reveal that placing labeled demon- strations near the query generally improves performance, highlighting the importance of proximity to high-quality data. However, this improvement diminishes the number of pseudo-labeled demonstrations increases, indicating that the influence of noisy pseudo-labeled data becomes more dominant with higher proportions. These findings empha- size the significance of demonstration order in enhancing ICL outcomes.
Agrawal, S., Zhou, C., Lewis, M., Zettlemoyer, L.,, and Ghazvininejad, M. In-context examples selection for machine translation. arXiv preprint arXiv:2212.02433, 2022.
Baek, J., Lee, J., Gupta, P., Dalmia, S., Kolhar, P., et al. Revisiting in-context learning with long context language models. arXiv preprint arXiv:2412.1692 2024.
Bertsch, A., Ivgi, M., Alon, U., Berant, J., Gormley, M. R., and Neubig. G. In-context learning with long- context models: An in-depth exploration. arXiv preprint vrXiv:2405.00200 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
5. Conclusion
In this work, focus enhancing many-shot ICL perfor- mance in resource-constrained tasks. we propose novel adaptive pseudo-labeling framework for many-shot ICL, which selects the most impactful unlabeled samples for pscudo-labeling. Additionally. we introduce an adaptive method to select both pseudo-labeled and labeled samples as demonstrations for LLM input. Extensive experiments across various datasets validate the effectiveness of our framework. In future work, aim to further explore the interaction between labeled and pseudo-labeled samples and its impact on ICL performance. We believe this will help identify the most useful samples for pseudo-labeling.
Casanueva, I., Temčinas, T., Gerz, D., Henderson, M., and Vulié, I. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.00 048007 2020.
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., al. A survey on evaluation of large language models. ACM Transactions Intelligent Systems and Technology, 15(3):1-45, 2024.
Chen, Z., Wang, S., Shen, C., and Li, J. Fastgas: Fast graph- based annotation selection for in-context learning. arXiv preprint arXiv:2406.0333
2024a.
Acknowledgements
Chen, Z., Weiss, G.. Mitchell, E., Celikyilmaz, A., and Bosselut, A. Reckoning: reasoning through dynamic knowledge encoding. Advances in Neural Information Processing Systems, 36, 2024b.
This work is supported in part by the National Science Foun- dation (NSF) under grants IIS- 100-244, IIS-2144209 IIS-
2223769, CNS-2154962, BCS-2228534, CMMI-2411248 BCCS-2029978 CPS-2313110, 2CC5-2143559, AST- 2132700, and ECCS-2033671: the Office Naval Research (ONR) under grant 000014241260 and the Common- wealth Cyber Initiative (CCI) under grant VV-1Q24-011, and the research gift funding from Kneron.
Chen, Z., Wang, S., Tan, Z., Fu, X., Lei, Z., Wang, P., Liu, H., Shen, C., and Li, J. A survey of scaling in large lan- guage model model reasoning. arXiv preprint arXiv:2504.02180 2025.
Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., and Ravi, S. Goemotions: A dataset of fine-grained emotions. arXiv preprint 07X5v:2005.00547, 2020.
Impact Statement
This paper presents work whose goal is to improve the per- formance many-shot ICL resource-constrained settings where manual labeling is expensive or infeasible. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Hamilton, W. L., Ying, R., and Leskovec, J. Inductive rep- resentation learning on large graphs. In NeuriPS, 2017.
He, P., Liu, X., Gao. J., and Chen, W. Deberta: Decoding- enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0306 03650000.366,.... 2020.
References
Agarwal, R., Singh, A., Zhang. L. M., Bohnet, B., Rosias, L., Chan, S. C., Zhang, B., Faust, A., and Larochelle, H. Many-shot in-context learning. In ICML 2024 Workshop on In- Co-tett Learning, 2024.
Huang, B., Mitra, C.,, Arbelle, A., Karlinsky, L., Darrell, T., and Herzig, R. Multimodal task vectors enable many- shot multimodal in-context learning. arXiv preprint arXiv:2406. /5334, 2024.

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
Huang, K. and Zitnik, M. Graph meta learning via local subgraphs. In NewrfPS, 2020.
Qin, C., Zhang, A., Chen, C., Dagar, A., and Ye, W. In- context learning with iterative demonstration selection. arXiv preprint 23.x::3300008880 2023.
Izacard, G., Caron, M., Hosseini, L.., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense infor- mation retrieval with contrastive learning arXiv preprint arXiv:2112.091 2021.
Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:I908 10084, 2019.
Jain, S., Keshava, V.. Sathyendra, S. M., Fernandes, P., Liu, P., Neubig, G., and Zhou, C. Multi-dimensional evaluation of text summarization with in-context learning. arXiv preprint urXiv:2306.0/200 2023.
Rein, D., Hou, B. L.. Stickland, A. C., Petty, J., Pang, Y., Dirani, J.. Michael, J., and Bowman, R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311. 2023.
Jiang. Y., Irvin, J. A., Wang, J. H., Chaudhry, M. A., Chen, J. H., and Ng, A. Y. Many-shot in-context learning in multimodal foundation models. In ICML 2024 Workshop on In-Conin Context Learning. 2024.
Rubin, O., Herzig, J., and Berant, J. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pp. 2655 2022.
Kipf, T. N. and Welling, M. Sumi-supe classification with graph convolutional networks. In ICLR, 2017.
Su, H., Kasai, J., Wu, C. H., Shi, W., Wang, T., Xin, J., Zhang, R., Ostendorf, M., Zettlemoyer, L.,, Smith, N. A., et al. Selective annotation makes language models bet- ter few-shot learners. arXiv preprint XXXi:2229.009755 2022.
Lec, J.. Chen, A., Dai, Z., Dua, D., Sachan, D. S., Boratko, M., Luan, Y., Arnold, S. M., Perot, V., Dalmia, S., al. Can long-context language models subsume retrieval, rag, sql. and more? arXiv preprint arXiv:2406.13122 2024.
Li, M.. Gong, S., Feng, J., Xu, Y., Zhang, J., Wu, Z., and Kong, L. In-context learning with many demonstration examples. arXiv preprint arXiv:2302 009314 2023.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay. Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., al. Challenging big-bench tasks and whether chain-of- thought can solve them. In Findings of the Association for Computational Linguisties: ACL 2023, pp. 13003-13051, 2023.
Li, T., Zhang, G., Do, Q. D., Yue, X., and Chen, W. Long- context ilms struggle with long in-context learning, arXiv preprint 2rXiv:2404.000 02060, 2024.
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.068/0 2021.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXhv:2403.05530, 2024.
Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. Fantastically ordered prompts and where find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2 104.08866 2021.
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. Graph attention networks. In ICLR, 2018.
Malo, P., Sinha, A., Korhonen, P., Wallenius, J., and Takala, P. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Infor- mation Science and Technology, 65(4):782-796. 2014.
Wang, H. and Leskovec, J. Unifying graph con- volutional neural networks and label propagation. arXiv:2002.06755, 2020.
Narayan, S., Cohen, S. B., and Lapata, M. Don't ton'ten give me the details, just the summary! awpee convolutional neural networks for extreme summarization. In Pruceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, 2018.
Wang, L., Yang, N., and Wei, F. Learning to retrieve in-context examples for large language models. arXiv preprint arXiv:2307.07166 2023.
Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606-62 624, 2023.
Wang, S., Chen, Z., Shi, C., Shen, C., and Li, J. Mixture of demonstrations for in-context learning. Advances in Neu- ral Information Processing Systems, 17:88091-8811 16, 2024a.
10

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
Wang, X., Zhu, W., Saxon, M., Steyvers, M., and Wang, W. Y. Large language models are latent variable mod- els: Explaining and finding good demonstrations for in- context learning. Advances in Neural Information Pro- cessing Systems, 36, 2024b.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F,, Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thouge prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Wei, X. and Liu, L. Are large language models good in- context learners for financial sentiment analysis? arXiv preprint arXiv:2503.0455 04473, 2025.
Xu, K., Li, C., Tian, Y., Sonobe, T., Kawarabayashi, K.-i,, and Jegelka, S. Representation learning graphs with jumping knowledge networks. In ICMI,, 2018.
Ye, J., Wu, Z., Feng. Yu, T., and Kong, L. Compositional exemplars for in-context learning. In International Con- ference Machine Learning, pp. 19818-9833.. PMLR, 2023.
Zhang, X., Lv, A., Liu, Y., Sung. F., Liu, W., Shang, S., Chen, X., and Yan, R. More is not always better? enhanc- ing many-shot in-context learning with differentiated and reweighting objectives. arXiv preprint arXiv:2501.04070 2025.
Zhao, W. X., Zhou, K., J., Tang. T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, Dong, Z., al. survey of large language models. arXiv preprint arXiv: 18223, 2023.
Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate before Improving few-shot performance of language models. In International Conference Machine Learning, pp. 12697-12706. PMLR, 2021.

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
A. Theorem 3.2 and Proof
Prior to proving Theorem 3.2, we present a lemma that establishes the lower bound node influence between two nodes. In the subsequent proof, we adopt the approach outlined in (Huang Zitmik, 2020) and (Xu et al., 2018), utilizing GCNs (Kipf Welling, 2017) as the exemplar GNN for simplicity. It is worth noting that our proof can be readily extended different types of models (such GAT (Veličković et al., 2018) and GraphSAGE (Hamilton al., 2017)) by assigning different values to edge weights. Specifically, the propagation process in the I-th layer can be represented as H = o (AAH W(), where H and W denote the node representation and weight parameter matrices, respectively. A D-1 A represents the adjacency matrix after row normalization, ensuring that each row of A sums up to 1. Following the convention of Huang & Zitmik (2020), Wang & Leskovec (2020), and Xu et al. (2018), we set as the identity function and W as the identity matrix. Additionally, we assume that the propagation process is performed over sufficient number of iterations. Consequently, the output representation of a node can be expressed as a function of the representations of its neighboring nodes.
Lemma A.1. Consider the log-ex eggeeppctationn of node inftuence between node vi and node Uj, i.e.. E (log (I(),, U,))). Assume that the node degrees are distributed uniformly for each node with the mean value d. Then, E (log (1(1, w,))) 2 log (ss 0z)) Lg(Us, 12)) loog d, where Ls(ve, U3) is the shortest path distance between v and Vy. and Ps (U, 03) is the number of paths with length of Ls(us, 1;)) from Us to Vz-
Proof. According to the propagation strategy used in GCNs, we acknowledge that the representation of node can be represented as
Σ ag
h
DE REN())
where (i)) denotes the of neighboring nodes of node u. After that, we can expand the equation via incorporating more neighbors of node U/:
h; Da kEN())
Σ
aik Dikk (EN(K)
Σ
aghh;
(16)
= Da kENT))
Σ ask Dak LEN(K)
Σ Ogd Dew
Omo hg-
(EN(mm
In this manner, the node influence Lig |10h:/Oh|| can be represented as:
0h;
Σ
aik Dik LEN(K)
Σ agi
Σ
amobo
0h
0h
Dis
D Drmm oEN(m)
kEN(()
aike
Oxd, shi
(17)
Th
Di
DEFA
Dus, kb.
1
+
+
D
UikT
DETET
DEn k.
Okn
In the above derivation, we begin by replacing the term h, with the iterative expansion of its neighboring nodes. This expansion involves selecting only n paths from vú and to Ug where n; represents the number of intermediate nodes along the i-th path. This selection is motivated by the fact that, when considering the gradient between 2 and By, the derivatives along paths that do not include vy would be 0 and thus can be disregarded. Subsequently, we proceed to extrast the shared
12

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
term (Ohy/Ohy)
((
0h;
Th v
1 Dis
dikt
|h,]
Das, kai
+
+
asky Duy
a
Dst
D kmn ko
(18)
1
Dis
aiki
DEAA
DEL, kbs
akh
1 Lain Dint
Oke
Dii
Deper
Day kn
In this derivation, we first employ the identity 1/0h//Oh
1. This identity holds because ||h, /0h = |||
bill [h]] 1. The resulting term represents an expectation that involves summing the products of node degrees along all paths connecting v; and Vj. As result, surpasses the values obtained by summing the products of node degrees along (potentially multiple) paths with the minimum node degree product:
1
auft D DAP
0h,
Dii
D DRPIKP Pr Kin
+ .... +
Dis
muef Dffmefm
D Kamm kmm
ag Pas Kin Pm
(19)
Pm. max (Da100) DajA1
Dai, kh,
1 Dis
Clem 5 momm
DEPAT
Am D Piym
Assuming that the node degrees are uniformly distributed, the expectation of node degree products on path P. is pfne,t+) where n, + 1 is the length, and d is the expectation of node degrees. Furthermore, it is noteworthy that these paths are exactly the shortest paths between 2 and By. Therefore,
(|
Oh. Th,
E
2 Pm (1/a)(urt1) Pm (L.((v)9)))
(20)
where Ls(u;, uz) denotes the shortest path distance between node vs and node uy, and Pm is the number of these paths. Then we can achieve the final result:
(1
Th. Th
)) 2 log Ps(u, 02) Ls(v,, log
(log (T(2, v,))) = log E
(21)
where Ps(u,, U2) is the number of paths with length of Ls(u,, 0z) from e uy.
Lemma A.1 demonstrates that the expectation of logarithmic node influence between two nodes is related to two perspectives: the shortest path distance and the number of shortest path distances between them. Now with Lemma A.1, we can prove Theorem 3.2.
Theorem 3.2. Consider the node influence from node u to a node set V. Denote the geometric mean of the node influence to all nodes in V as Iv(u)- "VII"" [(u, vi), where Vi is the i-th node in V. Assume the node degrees are randomly distributed with the mean value as d. Then,
B(log Iv(w)) 2 log Ps(u, V) - log Ls(u, V),
(22)
where Ls(u, V) is the average shortest path distance between u and nodes in V. Ps(u, V) is the geometric mean of the numbers of shortest paths between u and nodes V.
13

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
Proof. We know log (vg) be represented follows:
M
1
log Iv(u) V
log [u,, v)).
(23)
=11
Based on Lemma A.I, we know:
E (log [v(wx)
log E (1(,, 0²))
M
(24)
(log Ps(u, u.) - Lu, U/) log dogd),
where (s, 2;)) denotes the shortest path distance between node and node and Ps(u, U;) is the number of the shortest paths between u and uú Note that Ps(u, v.) 2 1, as any connected node pair should have at least a path. By rearranging the term, we can obtain the final inequality:
V
M Σ t=1
Eloo Iv(u))
2
Σ
log
log d us(
Ls(u, v.)
V
1=1
" (s(u, D)
|
(25)
log
II
logg d Ls(u, V)
= log Ps(ui, v) logd Is(v,, V).
B. Prompts
We provide the prompts used for many-shot ICL the summarization, reasoning, and question answering tasks Table 3, and for classification tasks Table
14

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
Table 3: A list of prompts that we for many-shot ICI. on summarization, reasoning, and question answering tasks.
Types
Prompts
You are an expert in article summarization. am going to give you some examples of article and its summary in fluent English. Here several examples.
(provide examples here with the following format.)
Article: <article>
Summarization
Summary: <summary>
am going to provide another article and want you to summarize it. Give only the summary, and no extra commentary, formatting, chattiness.
Article: (TARGET_QUERY
You expert in multiple-chee choice question answering tasks. I am going give you examples in a multiple-choice question answering format. Here are several examples.
Reasoning
(provide examples here with the following format.)
Question: <question>
Answer: <answer>
Quesition Answering
am going to provide another question and want you to predict answer. Give only the choice correct answer by selecting one of the options (c.g., "(A)", '(B)").
Question: SHARGET_QUERT
Table 4: A prompts that we use for many-shot ICT. on different extreme classification tasks.
Types
Prompts
You expert in financial sentiment analysis. Here several examples.
(provide examples here with the following format.) Sentence: <sentence>
Answer: <sentiment>
Financial PhraseBank
am going to provide another sentence and I want analyze the sentiment of and respond with only word: "positive'. 'negative', or 'neutral", No extra commentary, formatting, or chattiness.
Sentence: TARGGE_QQERRY
Given customer service query, please predict the intent of the query. Here several examples.
(provide examples here with the following format.) service query: <query> intent category: <category>
Banking77
am going to provide another customer service query and want you to predict the intent of the query. Give only the intent of the query, and no extra commentary, formatting, or chattiness. You can only make prediction from the following categories: (77 classes of the Banking?7 task]
service query: (TARGET (TARGET_QUERY)
Given comment, please predict the emotion category of this comment. Here are several examples.
(provide examples here with the following format.)
comment: <comment>
emotion category: <category>
GoEmotion
am going to provide another comment want you to predict the emotion category of the comment. Give only the emotion category, and no extra commentary. formatting, or chattiness, You can only make prediction from the following categories: {28 classes the GoEmotion task)
comment: TTARGET _QURRY]
15

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
C. Datasets
XSum (Narayan et al., 2018): Extreme Summarization (XSum) is dataset designed for evaluating abstractive single- document summarization systems. It contains 226,711 news articles from BBC (2010-2017), spanning various domains such as news, politics, sports, weather, business, technology, science, health, family, education, and entertainment. The goal is generate concise one-sentence summary answering the question, "What is the article about?"
Date (Suzgun et al., 2023): This dataset is designed for Date Understanding, where the task is to answer a provided question based on a small set of sentences related to a particular date.An example is: "Today is Christmas Eve of 1937. What is the date tomorrow in MM/DD/YYYY? Options: (A) 12/1 1/1937: (B) 12/25/1937; (C) 01/04/1938; (D) 12/04/1937; (E) 12/25/2006; (F) 07/25/1937"
Salient (Suzgun et al., 2023): This dataset is designed for Salient Translation Error Detection, where, given source sentence written in German and its English translation, the task is to determine the type of translation error present in the translated sentence. An example is: "Source: Karl Borrom00e4us Joseph F00ferst von Liechtenstein war ein kaiserlicher Feldmarschall. Translation: Charles Borromeo Joseph Prince Liechtenstein was an judicial field marshal. The translation contains an error pertaining to Options: (A) Modifiers or Adjectives; (B) Numerical Values; (C) Negation Antonyms; (D) Named Entities; (E) Dropped Content; (F) Facts"
Tracking7 (Suzgun et al., 2023): This dataset is designed for Tracking Shuffled Objects, where, given the initial positions of a set of seven objects and a series of transformations (specifically, pairwise swaps) applied to them, the goal is to determine the final positions of the objects. An example is: "Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a soccer match. At the start of the match, they are each assigned to a position: Alice playing striker, Bob is playing right winger, Claire playing left winger, Dave playing benehwarmer, Eve is playing goalkeeper, Fred is playing center midfielder, and Gertrude is playing cheerleader. As the game progresses, pairs of players occasionally swap positions. First, Eve and Claire trade positions. Then, Gertrude and Alice trade positions. Then, Fred and Bob trade positions. Then, Dave and Fred trade positions. Then, Fred and Bob trade positions. Then, Bob and Eve trade positions. Finally, Claire and Alice trade positions. At the end of the match, Gertrude is playing: (A) striker; (B) right winger; (C) left winger; (D) benchwarmer; (E) goalkeeper; (F) center midfielder; (G) cheerleader"
Financial PhraseBank (FP) (Malo et al., 2014): is sentiment analysis dataset consisting of 4,840 sentences from English-language financial news, categorized by sentiment. The annotators were instructed assess the sentences from an investor's perspective, determining whether the news would likely have positive, negative, neutral impact on stock prices. An example is: "Sentence: Pharmaceuticals group Orion Corp reported a fall in its third third-quarter earnings, which were impacted by larger expenditures on R&D and marketing. Label: negative."
Banking77 (Casanueva al., 2020): BANKING77 is a dataset for banking-domain domain intent detection. comprising 13,083 annotated examples across 77 distinct intents. It offers a complex and realistic representation of commereial systems. An example is: "Text: I found my lost card. Am I still able to use it? Label: Link to Existing Card."
GoEmotion (Demszky et al., 2020): GoEmotion is the largest human--nnotatee annotated dataset, consisting of 58k carefully selected Reddit comments, labeled with 27 emotion categories or "Neutral." " The comments are extracted from popular English subreddits. An example is: "Text: I'm not even sure what it is, why do people hate it. Label: confusion."
GPQA (Rein et al., 2023): GPQA is a multiple-choice question answering benchmark that features challenging graduate-level questions in biology, physics, and chemistry, designed to assess advanced reasoning skills. An example question is: "If sperm from species A is injected into an egg from species B, and both species have the same number of chromosomes, what would be the main cause of the resulting zygote's 's mortality?"
D. KV Cache Analysis
We begin by providing a simple analysis of the FLOPs between the vanilla transformer and the transformer with KV Cache. We denote the length of the prefix context, model hidden size, the number of model layers, and the number of new tokens to generate as n, dmostet, I., and respectively.
16

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
Table Basic information statisties of datasets adopted in our experiments.
Dataset
Task
#(Training samples)
#(Test samples)
XSum
Summarization
204,045
11,334
Date
Reasoning
369
250
Salient
Reasoning
998
250
Tracking7
Reasoning
1,750
250
Financial PhraseBank
Classification
2,952
500
Banking77
Classification
10,003
3,080
GoEmotion
Classification
43,410
5,427
GPQA
Question Answering
250
198
Vanilla decoding without KV Cache.
The per-step cost for length-m forward pass in one layer can be approximated as:
(m dmmodel + m² dlmoade).
At each step t, re-run forward pass
(n + 1) tokens, thus the total cost across all I decoding steps is:
n t 1)² damodes)
O(T (n+ TT2 dmmoade))
With KV Cache. To build the cache, it is the same performing full forward pass over the prefix (length ni) once:
On dmodel clmeeee) L.
This stores the keys and values for all tokens in each layer. The per-step cost for each new token is:
(n damddel + denodee)) x L.
Thus, the total cost with cache over new tokens is:
(n dmodal)
o (n dmodel + emodal)
initial prefix
decoding T tokens
Hence, once we have built the cache for the prefix, each new token only requires ((n) operations for attention plus Od (osode)) for projections and feed-forward, rather than re-encoding the entire sequence each time.
E. Additional Experiments
E.1. Effect of Encoder Models
In our main results, we use the Contriever (Izacard et al., 2021) model, a widely adopted encoder for information retrieval tasks, as the embedding model fa(.). To assess the impact of different embeddings, we conduct ablations with Sentence- BERT (SBert) (Reimers & Gurevych, 2019) and DeBERTa (He et al., 2020) as alternative encoders, evaluating RAG and MAPLE with 20, 60, and 100 pseudo-labeled examples. The results, presented in Table 6, show that while performance varies across encoders, MAPLE consistently outperforms the RAG baseline, demonstrating its robustness and effectiveness across different embedding choices.
E.2. A blation Study of Inffuence Score Components
We conduct an ablation study to analyze the components the influence score defined in Equation 8, namely the geometric mean of the numbers of shortest paths (:( -) and the average shortest path distance .t.).. Experiments are performed under the setting with 20 labeled and 100 pseudo-labeled examples. Results are presented in Table 7.
17

---

MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
Table Impact different embedding model fa(-).
Date
GoEmotion
Encoder
Method
|22-1=20
|-6
[2:|=100
[Dt] 1=20=20
[Pt |60
12//=1100
RAG
51.4
52.4
54.4
31.3
32.7
33.3
SBert
MAPLE
52.7
54.0
55.2
34.7
36.7
37.3
RAG
52.0
53.6
55.2
32.7
33.7
34.4
DeBERTa
MAPLE
54.4
55.2
57.6
37.3
37.2
39.3
While the length of the shortest path captures how quickly information travel, it overlooks robustness-relyinn single path can be fragile to noise minor data variations. On the other hand, using only the number of shortest paths captures redundancy but disregards distance; many long paths may not imply strong influence. Our influence score is designed to capture both efficiency (via short paths) and robustness (via multiple paths), resulting in more reliable and informative demonstration selection for many-shot ICL.
Table 7: Ablation study of components the influence score
Dataset
Banking77
GoEmotion
GPQA
Ls(s-)
75.3
37.6
36.4
Ps(..))
78.6
37.2
36.9
Influence score
80.8
38.1
37.4
18


Reviews:
[{"review_id": "NjrHv7wpgL", "reviewer": "Reviewer_K6CP", "summary": "This work develops a semi-supervised in-context learning framework by exploiting small amount of labeled and large unlabelled dataset. A Ken graph is built upon the labeled and unlabelled dataset. The unlabelled samples (nodes) that are similar to the labeled ones are selected \nFor pseudo labelling. Finally, demonstrations that are highly relevant to the test query are selected from the combined dataset for prediction.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**W1.** Building a graph for selecting relevant unlabelled samples itself induces additional computation overhead. There is no discussions on the computation cost for graph construction.\n>\n**Response**: Thank you for bringing up this point. The graph construction requires the computation of the relevance score $r$ among any pair of nodes, which will be $\\mathcal{O}(|\\mathcal{V}|^2)$. To compute shortest paths, we use breadth-first search for each node, and the cost is $\\mathcal{O}(|\\mathcal{V}|+|\\mathcal{E}|) = \\mathcal{O}(|\\mathcal{V}|)$ as $\\mathcal{E}=\\mathcal{O}(k|\\mathcal{V}|)$. Therefore, the whole shortest path computation cost is $\\mathcal{O}(|\\mathcal{D}_L||\\mathcal{V}|)$. Notably, the above cost is only required **once** before inference and thus is independent of the number of queries. With more queries involved during the test, the computational cost of the graph becomes more negligible. Moreover, as shown in Table 2 in our paper, the adaptive demonstration selection component does not incur much computational cost.\n\nThank you so much for your suggestion. We will include this discussion in the appendix.\n\n>**W2.** The design for the influence score seems arbitrary. It would be good to see if keeping only the shortest path or number of shortest paths is worse.\n>\n**Response**: We appreciate your suggestion and have added results (#labeled=20, #p-labeled=100) using only the shortest path and the number of shortest paths. While the shortest path captures how quickly information can travel, it overlooks robustness—relying on a single path can be fragile to noise or minor data variations. On the other hand, using only the number of shortest path captures redundancy but disregards distance; many long paths may not imply a strong influence. Our influence score is designed to capture both efficiency (via short paths) and robustness (via multiple paths), resulting in a more reliable and informative demonstration selection for many-shot ICL.\n\n\n|Dataset|Banking77|GoEmotion|GPQA|\n|-|-|-|-|\n|len(shortest path)|75.3|37.6|36.4|\n|# \\|shortest path\\||78.6|37.2|36.9|\n|Influence score|80.8|38.1 |37.4|\n\n>**W3.** It is worth noting that the impact of increasing the number of pseudo labeled samples is indeterministic. For GoEmotion, Banking77 and Date, the performance may still go up with more pseudo-labels. While, Tracking 7 seems does not benefit any pseudo labeled samples. A deeper analysis is necessary.\n**Q1.** Please further explain why increasing the number of pseudo labels may harm certain datasets.\n>\n**Response**: \nThank you for highlighting this point. The inclusion of additional pseudo-labeled samples can sometimes harm performance because pseudo-labels generated by LLMs are not always accurate. Incorrect pseudo-labels may introduce misleading information when used as demonstrations, negatively influencing LLM predictions for specific datasets [1]. However, our method addresses this issue by adaptively selecting pseudo-labeled samples based on their relevance to each test query. This approach mitigates the negative impact of inaccurate pseudo-labeling, as demonstrated by consistent improvements across most tasks. We note that the performance decline is limited to certain datasets, likely due to the higher difficulty or inherent ambiguity of the samples. Nevertheless, MAPLE consistently outperforms baselines even though the number of pseudo-labeled samples is not optimal.\n\n[1] Agarwal et al. Many-Shot In-Context Learning. NeurIPS 2024.\n\n>**Comm1.** It is unclear why the x axis starts with 20 in figure 3. With 0 pseudo-labeled samples, would this be equivalent to few-shot?\n>\n**Response**: Thank you for the observation. Yes, when the number of pseudo-labeled samples is 0, Random, RAG, and MAPLE all reduce to the few-shot setting. To avoid redundancy, we omit x=0 from the x-axis and instead include the few-shot performance as a green horizontal dashed line in Figure 3 for comparison. We will clarify this explicitly in the revised version.", "claims_and_evidence": "The claims made in this paper are mainly supported with empirical results.", "methods_and_evaluation_criteria": "The methodology and benchmarking datasets are mostly appropriate.", "theoretical_claims": "Not thoroughly checked.", "experimental_designs_or_analyses": "The experiment designs are mostly appropriate.  However, evaluation on higher number of pseudo labeled samples are missing which may demonstrate the potential upperbound of the proposed method.", "relation_to_broader_scientific_literature": "Semi-supervised learning has been thoroughly investigated. The proposed method exploits a graph and the influence score shares similarity with graph based SSL and label propgation. However, integrating SSL to ICL is novel.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n1. Exploiting graph for relevance calculation could exploit the data manifold and potentially leads to better results.\n\n2. Semi-supervised in-context learning can alleviate the reliance on excessive human annotation.\n\nWeakness:\n1. Building a graph for selecting relevant unlabelled samples itself induces additional computation overhead. There is no discussions on the computation cost for graph construction.\n\n2. The design for influence score seems arbitrary. It would be good to see if keeping only the shortest path or number of shortest paths is worse.\n\n3. It is worth noting that the impact of increasing the number of pseudo labeled samples is indeterministic. For GoEmotion, Banking77 and Date, the performance may still go up with more pseudo-labels. While, Tracking 7 seems does not benefit any pseudo labeled samples. A deeper analysis is necessary.7", "other_comments_or_suggestions": "It is unclear why the x axis starts with 20 in figure 3. With 0 pseudo-labeled samples, would this be equivalent to few-shot?", "questions_for_authors": "- Please further explain why increasing the number of pseudo labels may harm certain datasets.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WfEXvdpM8d", "reviewer": "Reviewer_qNQK", "summary": "The paper considers semi-supervised many-shot in-context learning setting, i.e., having small labeled and large unlabeled support sets to perform in-context learning with long-context LLMs. Authors argue that within this problem setting it would be beneficial to (i) identify the most impactful unlabeled samples to pseudo-label them, and, subsequently (ii) use adaptive example selection mechanism to select examples for each test query from the union of the pseudo-labeled examples along with labeled ones. To reach both, authors leverage the concept of node influences in graphs. In particular, first, the graph is built with nodes representing examples from both labeled and unlabeled sets, and edges are assigned according to the similarity of the examples in some embedding space. Subsequently, top-p nodes from the set of unlabeled examples are selected according to the score that is lower bound to the node influence. Adaptive demonstration selection is built in the similar fashion, relying on node influence of the labeled and pseudo-labeled samples on the test query. Authors evaluate their approach of the diverse set of problems and show that it outperforms the considered baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1.** Do you really need to cut Top-K edges for each node? Is it correct that Top-K only controls speed and does not affect the performance, or there is some interplay between both Top-K and Top-P that affect the performance? \n>\n**Response**: Thank you for the question. We want to clarify that Top-K pruning is essential and does influence selection. Our influence score in Eq(8) depends on the shortest path and its count. In a fully connected graph, all node pairs are directly connected with only one shortest path of length 1, which removes meaningful structural differences and reduces influence estimation to near-random. Moreover, computing the shortest path on a fully connected graph incurs complexity in $\\mathcal{O}(|\\mathcal{V}|^2)$. Thus, Top-K is important for both efficiency and effective selection.\n\n>**Q2.** Given Figure 6 shows that, mostly, it is more beneficial to put pseudo-labeled examples at back and examples with ground truth labels closer to the query, could you also run the baseline when you just put questions for unlabeled samples (without pseudolabels). It is basically Unsupervised ICL of [1], which was shown to improve upon few-shot baseline that employs only labeled set. \n>\n**Response**: Thank you for the suggestion, and we have added results using only unlabeled questions without pseudo-labels, following the unsupervised ICL setup (#labeled=20, #unlabled=100). Consistent with Figure 6, we observe that placing labeled examples closer to the query still leads to better performance. However, without pseudo-labels, the benefit from unlabeled examples is limited (compared to Fig 6.), highlighting the importance of label information—even if approximated—for effective many-shot ICL.\n|Dataset|Salient|GPQA|\n|-|-|-|\nRand w/w.o. R|65.6/66.0|33.3/34.3|\nRag w/w.o. R|68.8/68.2|33.8/34.8|\nMAPLE w/w.o. R|70.4/70.8|35.3/36.1|\n\n>**Q3.** What part of the proposed approach is actually the most important?\n>\n**Response**: We appreciate the reviewer’s question. The RAG-Adapt baseline in our paper can be seen as a variant of MAPLE without the graph structure, as both rely on Contriever for relevance score. Without pseudo-labeling, as in our response to Q2, performance drops due to the lack of label information. Further, in our response to Reviewer K6CP W2, we detail how removing individual components of the influence score degrades performance. Together, these results demonstrate that each part of MAPLE is crucial to its overall effectiveness.\n\n>**Q4.** There are some inconsistencies in set of baselines for different tables and Figures. For example, Table 3 misses zero-shot, few-shot and RAG-Adapt (compared to Figure 1). Similarly, Figure 4 misses RAG-Adapt.\n>\n**Response**: Thank you for pointing this out. We believe the reference is to Table 1 (rather than Table 3) since Table 3 is a list of prompts. Due to rebuttal time constraints and API budget limits, we have now additionally run Gemini 1.5 Flash for zero-shot, few-shot, and RAG-Adapt on Table 1, as well as RAG-Adapt for Figure 4. We will include these updated results in the revised version of the paper.\n\n|Datset|0-shot|few-shot|20|40|60|80|100|\n|-|-|-|-|-|-|-|-|\n|Banking77|75.1|76.9|77.0|76.9|76.5|76.7|78.5|\n|Date|49.1|52.9|53.6|53.5|53.1|55.4|56.1|\n|GPQA|34.3|35.8|37.1|36.2|35.7|32.5|34.0|\n\n|Datset|50|100|150|200|\n|-|-|-|-|-|\n|Banking77|79.3|81.3|81.3|81.7|\n|Date|56.4|59.2|59.4|63.4|\n\n>**Q5.**What was used as the embedding model to construct graphs?\n>\n**Response**: Thank you for the question. We use Contriever as the embedding model. This is mentioned in the right column of line 115: “...the relevance score r(vi, vj), as defined in Contriever...” For clarity, we will also explicitly restate this in the Implementation section in the revised version.\n\n>**Q6.** Can we compute node influence directly and not rely on the lower-bound? Does lower bound have some benefits? How the performance will be different if the node influence would be computed directly?\n>\n**Response**: We claim that it is computationally impractical to directly calculate the node influence. As stated in Eq. (17) in Appendix A, the node influence between any pair of nodes, $v_i$ and $v_j$, is calculated from the iterative expansion of the neighboring nodes of $v_i$. This involves all the nodes that exist in any path between $v_i$ and $v_j$, and their corresponding derivatives regarding the embedding of $v_j$. This can be computationally prohibitive when the number of such nodes is massive due to the long distance between $v_i$ and $v_j$. \n\nTherefore, we propose to rely on the lower bound to compute the influence score instead of directly computing the node influence. With our proposed Theorem 3.2, the influence score is computed based on the shortest path distance and the number of shortest paths. Thus, it is much easier to compute, compared to the massive computation of derivatives in the original node influence.", "claims_and_evidence": "See Questions for Authors.", "methods_and_evaluation_criteria": "The proposed method is evaluated using challenging datasets and with the recent models.", "theoretical_claims": "I checked Theorem 3.2 and briefly checked the proof of Lemma A.1.", "experimental_designs_or_analyses": "Overall, seems valid.", "relation_to_broader_scientific_literature": "Many-shot in-context learning is very recent and promising approach to perform adaptation of large-context LLMs. Given the high data labeling cost, it is important to consider the semi-supervised setting, thus I believe that the paper studies an important topic.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See Questions for Authors.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Do you really need to cut Top-K edges for each node? Since, you still will be selecting top-P nodes only after all, and it’s single cost thing, you can just run Top-P selection over fully connected graph? In other words, is it correct that Top-K only controls speed and does not affect the performance, or there is some interplay between both Top-K and Top-P that affect the performance?\n\n2. Given Figure 6 shows that, mostly, it is more beneficial to put pseudo-labeled examples at back and examples with ground truth labels closer to the query, could you also run the baseline when you just put questions for unlabeled samples (without pseudolabels). It is basically Unsupervised ICL of [1], which was shown to improve upon few-shot baseline that employs only labeled set.\n\n3. What I am currently missing is the ablation of what part of the proposed approach is actually the most important or the evidence that both are important. \n\n4. There are some inconsistencies in set of baselines for different tables and Figures. For example, Table 3 misses zero-shot, few-shot and RAG-Adapt (compared to Figure 1). Similarly, Figure 4 misses RAG-Adapt.\n\n5. Maybe I missed it somewhere, but what was used as the embedding model $f$ to construct graphs?\n\n6. Can we compute node influence directly and not rely on the lower-bound? Does lower bound have some benefits? How the performance will be different if the node influence would be computed directly?\n\n[1] Agarwal et al. Many-Shot In-Context Learning. NeurIPS 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BN5nuxZJQC", "reviewer": "Reviewer_y2DN", "summary": "This paper presents MAPLE, a method for pseudo-labeling in many-shot ICL settings. Key innovation includes similarity-based selection for pseudo-labeling and demonstration example selection.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Claim** The author claims “strong performance” for the MAPLE, but didn’t specify the baselines nor quantitative results for comparison.\n>\n**Response**: In our experiments, MAPLE is primarily compared quantitatively against 5 baseline methods on 8 datasets with 5 settings (i.e., the number of pseudo-labeled samples). The results are shown in Figure 3. Among the 8 datasets, MAPLE performs the best on 5 datasets on all settings. On the other 3 datasets, MAPLE performs the best on 4 out of 5 settings. Therefore, these results can validate MAPLE's strong performance.\n\n\n>**Exp** It seems the embedding module is important for MAPLE, and an ablation on that is important.\n>\n**Response**: Thank you for the suggestion. We have conducted ablations using Sentence-BERT (SBert) [1] and DeBERTa [2] as alternative embedding models, evaluating MAPLE with 20, 60, and 100 pseudo-labeled examples. While performance varies across models, MAPLE consistently outperforms baseline, demonstrating its robustness and effectiveness regardless of the specific embedding choice.\n\n|Embed|Date|GoEmotion|\n|-|-|-|\n|RAG+SBert|51.4/52.4/54.4|31.3/32.7/33.3|\n|MAPLE+SBert|52.7/54.0/55.2|34.7/36.7/37.3|\n|RAG+DeBERTa|52.0/53.6/55.2|32.7/33.7/34.4|\n|MAPLE+DeBERTa|54.4/55.2/57.6|37.3/37.2/39.3|\n\n[1] Reimers N, Gurevych I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.\n\n[2] He P, Liu X, et al. DeBERTa: Decoding-enhanced BERT with Disentangled Attention.\n\n>**Comm1.** IMPORTANT Most figures are not rendered correctly. Labels for legend and axes are missing.\n>\n**Response**: We sincerely appreciate your feedback on figure clarity. In the revision, we will ensure all figures are correctly formatted and rendered. \n\n>**Q1.** Do you have any explanation why MAPLE works well for GPQA, given the diversity of topics in GPQA?\n>\n**Response**: Thank you for pointing this out. MAPLE's strong performance on GPQA can be attributed to its adaptive demonstration selection, which tailors pseudo-labeled demonstrations specifically for each test query. This adaptability allows MAPLE to effectively handle the topic diversity in GPQA by selecting demonstrations that are contextually relevant to each individual query. Consequently, MAPLE can leverage pseudo-labeled samples to improve performance even when the topics are diverse.", "claims_and_evidence": "1.\tIt’s interesting to study many-shot ICL under pseudo-label settings, which has practical value. \n2.\tThe author claims “strong performance” for the MAPLE, but didn’t specify the baselines nor quantitative results for comparison.", "methods_and_evaluation_criteria": "1.\tThe baselines are reasonable, and the datasets are up-to-date and commonly used. \n2.\tIt seems the embedding module is important for MAPLE, and an ablation on that is important.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "See methods", "relation_to_broader_scientific_literature": "It's an interesting extension towards both many-shot ICL and pseudo-labeling.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "1.\t**IMPORTANT** Most figures are not rendered correctly. Labels for legend and axes are missing.", "questions_for_authors": "1.\tDo you have any explanation why MAPLE works well for GPQA, given the diversity of topics in GPQA?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9oW9yZReKK", "reviewer": "Reviewer_fpdn", "summary": "The paper proposes is an effective framework for enhancing many-shot ICL performance in scenarios with limited labeled data. B  selecting unlabeled samples for pseudo-labeling based on their influence on labeled data and by adaptively selecting demonstrations tailored to each query, the method significantly reduces the reliance on costly labeled data while improving the in-context learning performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**W1.** Practicality Concerns.\n>\n**Response**: Thank you for raising concerns regarding computational complexity. To address practicality, our framework incorporates strategies to improve efficiency significantly:\n\nBy employing a KV cache, we reduce computational costs by fixing labeled and pseudo-labeled demonstrations across queries, allowing caching of demonstrations within the LLM prior to inference. As demonstrated in Table 2, this approach effectively enhances efficiency without loss of accuracy, making MAPLE more practical for real-world scenarios. A detailed analysis is provided in Appendix D.\n\nAdditionally, the influence graph construction process is precomputed. In this way, we ensure that its computational cost does not scale with the number of queries, further enhancing efficiency and practical feasibility.\n\n\n\n>**W2.** Unstable Gains from Pseudo-Labeling.\n>\n**Response**: We would like to clarify that in our experiments, MAPLE **consistently outperforms other baselines**. MAPLE is primarily compared quantitatively against 5 baseline methods on 8 datasets with 5 settings (i.e., the number of pseudo-labeled samples). The results are shown in Figure 1. Among the 8 datasets, MAPLE performs the best on 5 datasets in all settings. On the other 3 datasets, MAPLE performs the best in 4 out of 5 settings. Therefore, these results can validate MAPLE's strong performance.\n\n\n>**W3.** Limited Experimental Scope.\n>\n**Response**: We appreciate your suggestion regarding the experiments. Our current experiments include both Gemini 1.5 Pro and Gemini 1.5 Flash, which are widely adopted and representative models for many-shot ICL [1,2]. Due to limited rebuttal time, we were unable to include additional models like LLaMA and Qwen, but we plan to explore them in future work. To broaden task diversity, we have added results on a math benchmark GSM8K, and the results demonstrate the superiority of MAPLE on math tasks. We will expand to more datasets in the next version.\n\n[1] Agarwal et al. Many-Shot In-Context Learning. NeurIPS 2024.\n\n[2] Baek et al. Revisiting In-Context Learning with Long Context Language Models. arXiv 2024.\n\n|Method|20|60|100|\n|-|-|-|-|\n|Random|90.0|91.0|91.5|\n|RAG|90.5|92.0|93.0|\n|MAPLE|92.5|94.0|95.0|\n\n\n>**W4.** Lack of Discussion on Theoretical Upper Bound: The paper does not discuss the upper-bound performance of the proposed method. For instance, if many-shot data were fully annotated with golden labels, how would the proposed approach compare against a retrieval-augmented generation (RAG) baseline? A discussion on this aspect would provide a clearer perspective on the fundamental limitations of the method.\n>\n**Response**: \n\nWe appreciate your point. While deriving a theoretical upper bound for many-shot ICL is impractical due to the complexity of LLMs like Gemini 1.5 Flash, we provide an empirical upper bound by comparing MAPLE to RAG using 40, 80, and 120 fully labeled examples. MAPLE consistently outperforms RAG even under full annotation, highlighting its effectiveness and robustness beyond limited-label settings.\n\n|Embed|GPQA|Banking77|\n|-|-|-|\n|RAG+Golden|36.8/37.8/40.4|78.0/81.7/83.3|\n|MAPLE+Golden|38.3/42.4/44.9|79.3/81.7/86.2|\n\n>**W5.** Comparison to More Direct Labeling Approaches: With the cost of large model inference decreasing, a straightforward alternative would be to use a sota model (like GPT4) to label all data directly. The paper does not clearly articulate the advantages of the proposed method over this simpler and more practical alternative. A thorough comparison is necessary to justify the additional complexity introduced by the framework.\n>\n**Response**:  Thank you for raising this important point. We agree that directly using a state-of-the-art model like GPT-4 (or Gemini) to label all data is a natural and increasingly viable alternative. In fact, our zero-shot baseline involves using Gemini to label the entire dataset directly. As shown in Figure 3, MAPLE significantly outperforms this zero-shot approach across all tasks.\n\nMoreover, MAPLE only requires labeling a very small portion of the data. For instance, XSum contains over 2 million training examples, yet MAPLE achieves strong performance with at most 100 pseudo-labeled samples—representing a reduction of over 99.99% in labeling cost, even with decreasing inference costs.\n\nTo further highlight MAPLE’s advantage, we include results on GPQA using Gemini 1.5 Pro. In Table 1, MAPLE achieves 44.9% accuracy, while—as reported in Figure 8 in [1]—even fully using all labeled data yields less than 44% accuracy. This clearly demonstrates the effectiveness and efficiency of our method.\n\n[1] Agarwal et al. Many-Shot In-Context Learning. NeurIPS 2024.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I skimmed through it, but I didn’t carefully examine the proof.", "experimental_designs_or_analyses": "Yes. I checked the experimental designs including baselines, implementations and ablations.", "relation_to_broader_scientific_literature": "The paper proposes selecting the most valuable demonstrations for many-shot pseudo-label learning.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and well-structured, making it easy to follow.\n\n2. The ablation study is comprehensive, providing useful insights into the contributions of different components.\n\nWeaknesses:\n1. Practicality Concerns: The proposed method is overly complex and impractical for real-world applications. The framework requires constructing two influence graphs and selecting different demonstrations for each test query, resulting in significant computational costs. However, the performance improvements do not appear substantial enough to justify the high cost of this approach.\n\n2. Unstable Gains from Pseudo-Labeling: As shown in Figure 3, increasing the number of pseudo-labeled samples does not consistently lead to performance improvements across most tasks. This introduces an additional challenge of tuning the pseudo-labeling budget. In practical scenarios, there is no reliable way to verify the correctness of pseudo-labels, which may lead to worse performance than a simple few-shot approach.\n\n3. Limited Experimental Scope: The experiments are somewhat narrow in scope. The authors only evaluate the framework using the Gemini model and on relatively simple tasks. To strengthen the empirical validation, the paper should include experiments with additional models, such as LLaMA and Qwen. Moreover, the selected tasks should be more representative and general, such as those from the SuperGLUE and MATH benchmarks.\n\n4. Lack of Discussion on Theoretical Upper Bound: The paper does not discuss the upper-bound performance of the proposed method. For instance, if many-shot data were fully annotated with golden labels, how would the proposed approach compare against a retrieval-augmented generation (RAG) baseline? A discussion on this aspect would provide a clearer perspective on the fundamental limitations of the method.\n\n5. Comparison to More Direct Labeling Approaches: With the cost of large model inference decreasing, a straightforward alternative would be to use a sota model (like GPT4) to label all data directly. The paper does not clearly articulate the advantages of the proposed method over this simpler and more practical alternative. A thorough comparison is necessary to justify the additional complexity introduced by the framework.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a semi-supervised many-shot in-context learning approach - having small labeled and large unlabeled support sets to perform in-context learning with long-context LLMs.

Strengths:

- Very important line of work and has potential to be used across various academic and industrial applications.

- Solid experiments and results.
 
I enjoyed reading this work, and authors provide a detailed rebuttal addressing most of the concerns raised by reviewers. Specifically, major concerns about computation overhead for graph construction and ablation of different components of MAPLE are addressed by authors in the rebuttal. Overall, I recommend accept.
Decision: Accept (poster)