Demonstration Selection for In-Context Learning via Reinforcement Learning
Authors: Xubin Wang
Jianfei Wu
Yuan Yichen
Deyu Cai
Mingzhe Li
Weijia Jia
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
In-Context Learning
Reinforcement Learning
Chain-of-Thought
Primary area: deep_learning->large_language_models

Abstract: Diversity in demonstration selection is critical for enhancing model generalization by enabling broader coverage of structures and concepts. Constructing appropriate demonstration sets remains a key research challenge. This paper introduces the Relevance-Diversity Enhanced Selection (RDES), an innovative approach that leverages reinforcement learning (RL) frameworks to optimize the selection of diverse reference demonstrations for tasks amenable to in-context learning (ICL), particularly text classification and reasoning, in few-shot prompting scenarios. RDES employs frameworks like Q-learning and a PPO-based variant to dynamically identify demonstrations that maximize both diversity (quantified by label distribution) and relevance to the task objective. This strategy ensures a balanced representation of reference data, leading to improved accuracy and generalization. Through extensive experiments on multiple benchmark datasets, including diverse reasoning tasks, and involving 14 closed-source and open-source LLMs, we demonstrate that RDES significantly enhances performance compared to ten established baselines. Our evaluation includes analysis of performance across varying numbers of demonstrations on selected datasets. Furthermore, we investigate incorporating Chain-of-Thought (CoT) reasoning, which further boosts predictive performance. The results highlight the potential of RL for adaptive demonstration selection and addressing challenges in ICL.

Demonstration Selection for In-Context Learning via Reinforcement Learning
Xubin Wang 123 Jianfei Wu Yichen Yuan
Deyu Cai
Mingzhe Li"
Weijia Jia
Abstract
Input Text:
'The product was okay, but expacted better quality"
Diversity in demonstration selection is critical for enhancing model generalization by enabling broader coverage of structures and concepts. Con- structing appropriate demonstration sets remains a key research challenge. This paper introduces the Gelevance-Diversil Enhanced Selection (RDES), an innovative approach that leverages reinforce- ment learning (RL) frameworks to optimize the selection of diverse reference demonstrations for tasks amenable to in-context learning (ICL), par- ticularly text classification and reasoning. in few- shot prompting scenarios. RDES employs frame- works like Q-learning and a PPO-based variant to dynamically identify demonstrations that maxi- mize both diversity (quantified by label distribu- tion) and relevance to the task objective. This strategy ensures a balanced representation of ref- erence data, leading to improved accuracy and generalization. Through extensive experiments on multiple benchmark datasets, including di- verse reasoning tasks, and involving 14 closed- source and open-source L.Mss, we demonstrate that RDES significantly enhances performance compared to ten established baselines. Our eval- uation includes analysis of performance across varying numbers of demonstrations on selected datasets. Furthermore, we investigate incorpo- rating Thought (CoT) reasoning, which further boosts predictive performance. The results highlight the potential of RL for adaptive demon- stration selection and addressing challenges in ICL.
"labsluuttll loved this product!
perfectly" (Positive
best purchase ever made!" (Posttive)
Postive
3. "Fantastic quality, I'm very satistied" (Positive) Demonstrations
Decision
Diversity-Basdd Method
Input Text: "The product wee okey. but I expected beller quality"
"labsolutely loved this product!
pertectly" (Posttiv)
worst purchase I've made! (Negative)
Makes Neutral
3. "it's okay, not what expected. (Noutral)
Demonstrations
Dactaion
Diversity-Bazen Method
Figure I. An example shows how diversity- demonstration method works. In this example, the diversity-based method helps the model recognize that the input text expresses a sentiment that neither strongly positive nor negative, while the no diversity- based method may lead to an inaccurate positive classification due to its lack of varied demonstrations.
1. Introduction
LLMs have demonstrated exceptional capabilities across a wide array NLP tasks, including text annotation (Wu et al., 2025), question answering (Shao al., 2023). and di- alogue generation (Hu al., 2023). These models leverage extensive corpora of textual data learn rich representa- tions, which empower them to perform reasoning with high accuracy (Devlin et al., 2018; Radford et al., 2019; Brown et al., 2020). However, as the size and complexity of these models continue to expand, enhancing their reasoning capa- bilities becomes increasingly crucial. Effective reasoning essential for tasks that demand logical reasoning, common- understanding. and contextual awareness (Marcus, 2020; Bender et al., 2021). The ability to reason effectively not only improves the performance of LLMs in existing applications but also expands their potential for novel use cases that demand a deeper understanding of language and context.
'BNU-BNBU Institute of Artificial Intelligence and Future Networks, Beijing Normal-Hong Kong Baptist University, Zhuhai, China HHnn Kong Baptist University, Hong Kong, China Beijing Normal University at Zhuhai, Zhuhai, China. Correspondence to: Weijia Jia <jiawj @bw..cdu.cn.. Xubin Wang <wangxu- bin@icce. org..
In the realm few-shot learning, in-context learning (ICL) has emerged as promising approach to enhance reasoning in LLMs (Dong et al., 2024). ICL utilizes LLMs, such as those based on the GPT architecture, to perform reasoning by providing a carefully curated set demonstrations as context, rather than relying solely on extensive model re- training (Brown al., 2020; Gao et al., 2021; Zhang et al.,
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Demonstration Selection for In-Contest Learning via Reinforcement Learning
2022). This methodology allows LLMs to leverage their inherent capabilities for understanding and processing text, making them particularly suitable for tasks with limited la- beled data. However, the effectiveness of ICL is contingent upon the selection of appropriate and representative demon- strations from the knowledge base to serve as contextual references during reasoning on test data. This critical aspect of few-shot learning often overlooked in existing literature (Wang et al., 2020; Song et al., 2023). The careful selec- tion of demonstrations is essential, as it directly influences the model's ability generalize and perform accurately in novel situations.
Take action a: Choose demonstrations from knowledge basc
Environment
+
Input x)
Seotect
Agent
Knowledge Base
State ->
For Reasoning
Demonstrations
RL Algorithe
LLM
Adjust
Diversity
Reward
Reward
Prediction
Diversity Calculatiin
Figure 2. The RDES framework an adaptive RI. approach for few-shot ICI. demonstration selection in LLMss. It employs RI- based agent to dynamically balance the relevance and diversity of selected examples, guided by reward function that incorpo- rates a label distribution diversity score. This strategy enhances classification accuracy and generalization by mitigating overfitting associated with pure similarity- -baedd methods. The framework
Despite the promise of ICL, a significant challenge persists in selecting the most relevant and diverse demonstrations from the knowledge base optimize reasoning performance. Traditional methods of demonstration selection often pri- oritize similarity, which can inadvertently overlook the im- portance of diversity in capturing the full spectrum of the data distribution (Song et al., 2023). This oversight may lead to biased representations that do not generalize well to unseen data, ultimately hindering the predictive accuracy of LLMs (Wang et al., 2020; Liu & Lapata, 2019). Further- more, conventional selection techniques typically employ fixed strategies that fail to dynamically adapt to the spe- cific requirements of the reasoning task at hand (Wang et al., 2020; Song et al., 2023). This rigidity can limit the effective- ness of ICL, as the selected demonstrations may not align optimally with the context or nuances of the task, further exacerbating the challenges in achieving robust reasoning performance.
involves an Agent interacting with an Environment (including a Knowledge Base and the L.MM to learn an optimal selection pol- icy.
Our RL approach (including Q-learning and PPO variant) optimizes selection by balancing relevance and diversity, mitigating overfitting and improving gen- eralization.
RDES integrates seamlessly with advanced reasoning techniques, such as Chain-of-Thought (CoT) prompt- ing. further enhancing LLM reasoning capabilities.
We perform comprehensive evaluations against ten baseline methods across multiple benchmark datasets, showcasing significant improvements in diverse tasks with 14 closed-source and open-source LLMs.
The core motivation behind Relevance-Diversin Enhanced Selection (RDES) is to enhance performance in tasks amenable to ICI., particularly text classification and rea- soning. This is achieved by selecting demonstrations that maximize relevance while ensuring diversity. While similarity- selection improves accuracy, it risks over- fitting, whereas diversity promotes generalization, particu- larly in few-shot learning. By framing demonstration selec- tion sequential decision-making problem, RDES effec- tively leverages reinforcement learning (RL) frameworks, including Q-learning and a Proximal Policy Optimization (Pso)-based variant, to balance exploration and exploita- tion, improving model robustness and adaptability. RDES is evaluated and compared against two baseline categories: prompt engineering and demonstration selection. Prompt engineering focuses on crafting effective prompts, while demonstration selection aims to identify the most informa- tive references. Our main contributions are as follows:
In summary, RDES advances few-shot learning in NLP by addressing ICL's key challenges through adaptive demon- stration selection. By jointly optimizing relevance and diver- sity, it enhances LLMs' performance tasks amenable to ICL, particularly classification and reasoning. The schematic framework of RDES is illustrated in Figure 2.
2. Related Work
2.1. In-Context Learning
ICL has emerged as a transformative paradigm in NLP, par- ticularly with the advent of LLMs, enabling models adapt to new tasks by conditioning tions within the input context, thus eliminating the need for extensive retraining. The seminal work by Brown et al. introduced ICL with the GPT-3 model, demonstrating that LLMs can effectively perform a wide range of tasks by being exposed to a few exemplars in the prompt (Brown
small set of demonstra-
We introduce RDES, a RL-based framework that dy- namically selects demonstrations to enhance perfor- mance and model robustness for tasks amenable to ICI.
2

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
et al., 2020). Subsequent research has further explored ICL's capabilities, highlighting its effectiveness in few-shot and zero-shot learning scenarios (Wei et al., 2021; Liu et al., 2023). However, the effectiveness of ICL is critically con- tingent upon the selection of appropriate and representative demonstrations from the knowledge base to serve as con- textual references, an aspect often overlooked in existing literature. The careful selection of demonstrations is essen- tial, as it directly influences the model's ability to generalize and perform accurately in novel situations. Recent studies emphasize the need for effective demonstration selection strategies to enhance ICL outcomes, especially in contexts with limited labeled data (Zhang et al., 2022; Min et al., 2022; Ye et al., 2023; Bai et al., 2024). Approaches such as Determinantal Point Process (DPP) methods and Iterative Demonstration Selection (IDS) have highlighted the impor- tance of selecting a diverse and relevant subset of demon- strations to optimize model performance ( (Wang et al., 2024; Qin et al., 2024).
timize the selection and sequencing of examples for ICL (Scarlatos & Lan, 2023). Similarly, (Zhang al., 2022) for- mulated the demonstration selection problem sequential secision-making task, utilizing Q-learning framework to enhance example selection. The proposed RDES method builds upon these foundational studies, sitting at the in- tersection of ICL, demonstration selection, and RL-based post-training for LLMs. However, while prior RL meth- ods like (Scarlatos & Lan, 2023) and (Zhang et al., 2022) primarily focused on objectives such as relevance or uncer- tainty, RDES uniquely focuses on the dual objectives of diversity and relevance, explicitly aiming to optimize both by incorporating a diversity score into the reward function. RDES employs Q-learning framework and also explored PPO-based variant (RDES/PPO) which showed competi- tive results. This combination of a RL framework with an explicit label distribution- based diversity metric to achieve dynamic demonstration selection that balances relevance and diversity is highlighted as novel. RDES integrates seam- lessly with advanced reasoning techniques, such as CoT prompting, with the RDES/C variant incorporating CoT rea- soning to further enhance predictive performance. While RL from Human Feedback (RLHF) fine-tunes model weights for alignment, RDES optimizes input selection while keep- ing the model fixed, presenting lighter-weigh alternative for post-training improvements. By learning an adaptive policy per query, RDES can better optimize selection com- pared to methods like (Yang et al., 2023) and (Mavromatis et al., 2023).
2.2. Demonstration Selection Techniques
Effective demonstration selection is crucial for ICL success. Traditional methods often relied on heuristics or statisti- cal measures to identify representative examples, includ- ing selecting informative examples to minimize uncertainty, a concept central to active learning (Settles, 2009). Re- cently, the importance of diversity in the selection process has gained prominence, with research showing that a di- verse set of examples can improve generalization capabil- ities (Sener& & Savarese, 2018). Techniques such as clus- tering, coverage-based selection (e.g., SERRTScore-Reall)) and representative sampling are employed to ensure chosen examples cover a broad range of the input space (Zhang et al., 2023; Gupta et al., 2023). These approaches aim to enhance model performance, particularly in compositional tasks where diverse demonstrations offer better coverage. Skill-based few-shot selection methods, like SkilI-KNN, op- timize example selection by eliminating irrelevant features (An et al., 2023). Methods such as (Yang et al., 2023) pri- oritize diversity statically, while (Mavromatis et al., 2023) selects based on uncertainty and diversity without training policy. The work (Levy al., 2023) also aligns with the core idea of using diversity to enhance generalization. Cali- bration techniques (Zhao et al., 2021) focus on correcting biases in demonstration selection, which overlaps with the effort to ensure diverse label coverage.
3. Methodology
The RDES framework tackles the demonstration selection challenge using a principled RL approach that jointly opti- mizes for relevance and diversity. This section systemati- cally presents our methodology in four progressive compo- nents: (1) formal problem formulation as Markov Decision Process, (2) dual optimization strategies using Q-learning and PPO, and (3) implementation details.
3.1. Theoretical Foundations
1. RI. FORMULATION
RL provides a natural framework for sequential decision making in demonstration selection. We model the interac- tion between the selection policy and language model as an iterative process where the policy learns to construct optimal demonstration sets through trial-and-error interactions.
2.3. RL in Demonstration Selection
RL has emerged as a promising framework for optimizing demonstration selection across diverse machine learning tasks. Unlike static approaches, RL allows selection poli- cies to adapt iteratively based on feedback from the model's performance. Prior work like RetICL. leveraged RL to op-
3.1.2. MARKOV DECISION PROCESS CONSTRUCTION
The demonstration selection process is formalized as a finite- horizon MDP M (S,,,PRR,,)) with the following components:
3

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
State Space (S): Captures the complete decision con- text through four components:
3.2. Optimization Framework
The RDES framework employs two primary RL algorithms to handle different complexities of the state space and com- putational resources.
Textual features: dx (x+) Rd (TF-IDF vector of imput text)
Demonstration memory: (E(E)) Rd (Aggre- gated embeddings of selected examples)
3.2.1. Q-LEARNING APPROACH
Prediction history: (Dt) E RD (One-hot en- coded previous predictions)
Q-learning provides a model-free solution for learning demonstration selection strategies through temporal differ- ence updates. This approach is particularly effective for environments with relatively small or discretizable state spaces. The action-value function S A - IR esti- mates the expected cumulative rewards starting from state 8, taking action a, and subsequently following the optimal policy:
Diversity tracking: D; = [CE E [1] (Normal- ized label diversity)
The state embedding is constructed by concatenating these four distinct components:
(s,)) (2()) (E(E)) (y(1)) e (D,) (1)
0r d
where + denotes vector concatenation, and each ø. represents an embedding for the respective component.
Q(s,a) = E
(5)
Action Space (A):: Discrete selection over candidate demonstrations K., with action a, E {1,... KK] indi- cating the chosen example index from the knowledge base.
The Q-values are updated via the standard Q-learning rule:
Q(s, a) + Q(s,)) + + max max Q(s', Q(s ((ssa))
a)
(6)
where a is the learning rate, r is the immediate reward, s' the next state, and the max,' Q(s', , a') term represents the estimated value of the next state under the greedy policy, Key implementation aspects for applying tabular Q-learning include state discretization, for example, through TF-IDF feature binning, and using an e-greedy exploration strategy with exponential decay (e-annealing) to balance exploration and exploitation. Tabular Q-value storage is used, with sparse updates focusing on visited state-action pairs.
Transition Dynamies (P): Deterministic state up- dates through demonstration set modification. When action 0₃ (selecting candidate ka,) is taken state 8t = (Zz, Er, Ut. Dr), the next state 8 becomes:
Si+1 = f(s,, ax) = (z,, Eg U ( (ka, 1c+1, Do+1) (2)
where ic+1 is the new prediction based on the updated example set and D1+1 is the new diversity score.
Reward Function (R): A multi-objective reward bal- ancing prediction accuracy and diversity gain:
3.2.2. PROXIMAL POLICY OPTIMIZATION VARIANT
For high-dimensional state spaces where tabular methods are infeasible, we implement actor-critic architecture utilizing PPO. This approach uses neural networks to ap- proximate the policy and value functions.
R(s a+) I[ Vtnue
Uz) + (De+1 D.)
(3)
Accuracy
Diversity improvement
where I[(-) is the indicator function, Voree is the true label, dr is the prediction at step f, D, is the diversity at step f, D441 is the diversity after adding the selected example, and controls the exploration-exploiaaiie tradeoff. The diversity coefficient adapts during train- ing via an annealing schedule:
Policy Network (To): A neural network with parame- ters that produces demonstration selection probabili- ties for each action given a state:
(als) softmax Sacimmxx(Woorf (Wi + b1) + bz) (7)
where (ss) is the state embedding, non-linear activation function, and W,, learned weights and biases.
X(t) Amia + (Amax Amin)
(4)
This schedule prioritizes early exploration of diverse examples before focusing on accuracy.
Value Network (V₂)): A neural network with param- eters 1/ that estimates the value function of state, representing the expected cumulative reward from that state:
Discount Factor (7): 7 E [0, 1) emphasizes immedi- ate rewards, which is suitable for finite-horizon few- shot learning scenarios where a fixed number exam- ples are selected.
(s(s) (La(W36(s) bz) + bs
(8)

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
where (()). o, W;, b are similarly defined.
Algorithm 1 RDES Training Framework
Require: Knowledge base K, Test inputs Diest LLM M,
Optimization Objective: PPO optimizes a clipped surrogate objective function that encourages moderate
RL algorithm AA
1: Initialize selection policy (Q-table or neural networks)
policy updates, ensuring stability. The overall objec- tive function for training the policy (0) and value (t)
2: Precompute TF-IDF vectors for each sample 24 E Drest and for knowledge base K.
networks combines the clipped surrogate loss, value function loss, and an entropy bonus:
for i == to N do
Sample input 2; E Diest Select demonstrations E with initial candidates based on relevance (e.g., top-k TF-IDF matches from XC) and apply diversity adjustment. Generate prompt p = Format( E)
(0) c L (1) e2fl CE (0) [(0, v) E, [
5:
(9)
where E, denotes the empirical average over batch of trajectories, and C2 are coefficients controlling the weights of the value and entropy terms, respectively. The components are defined as:
6:
7:
Ohtain prediction ý = M(p)
Compute diversity score D = (C(E))
8:
(."'(() min (r,( clip (r, C) (v+ = (+,) - R²)² CENT (0) Σ Te Te(a|st) log
9:
Encode state d(x;, F, ú. D)
10: Select action a ~ (()) (example index from AC)
11: Calculate reward r = I( = () + A X(Daew - Dow)
12: Update policy parameters 0 using A with (8, (,
13: end for
14: Return: Optimized policy nt
Here, (0) [o(a)(ss) the probability ratio be- T0
tween the new and old policies, A, is the advantage estimate (c.g., using Generalized Advantage Estima- tion- GAE), clip(-, 1= = e) clips the ratio to be within [1 - c, 1 + c). RG is the estimated return, and C is the entropy the policy distribution state 8t, which encourages exploration.
3.3.3. PROMPTING STRATEGIES
To enhance the performance of the L.M in few-shot set- tings, employ two distinct prompting strategies using the selected examples:
3.3. Algorithmic Implementation
Standard Prompting: This strategy constructs a prompt by concatenating the input text, the selected demonstrations (input-output pairs), and the set of pos- sible labels. The LLM is then asked to predict the probability of label y given this prompt structure:
3.3.1. 1. UNIFIED TRAINING PARADIGM
The core training procedure shared by both Q-learning and PPO approaches is formalized in Algorithm 1. The algo- rithm iteratively samples test inputs, selects set of demon- strations (initially based on relevance, then potentially ad- justed), uses these to prompt the LLM for a prediction, calculates the diversity of the selected set, encodes the cur- rent decision context into a state, selects an action (which corresponds choosing an example, although the algorithm abstractly shows (s, a, for policy update, implying the action leads to state transition and reward), calculates the reward based on prediction accuracy and diversity change, and updates the policy parameters using the chosen RL algorithm.
p v, E) PLM (y Prompt: x, (0r 9:) ))
(10)
where x is the input text, {(i,, F.) are the k se- lected demonstrations, and y is the set of possible labels.
This strategy incorporates CoT rea- soning into the prompt, allowing the LLM generate intermediate reasoning steps before producing the fi- nal label. This is formulated as marginalizing over possible reasoning chains R:
CoT Prompting:
3.3.2. STATE REPRESENTATION DETAILS
As detailed in the MDP construction, the state embedding ((s:) is concatenation of four components: the TF-IDF vector of the input text, aggregated embeddings of selected examples, a representation of the prediction history, and the normalized label diversity. This comprehensive representa- tion provides the RL agent with sufficient context to make informed selection decisions.
plylz, = PLLM(rIE PLM(V)e,E,) (11) FER
The model first computes the probability of a reasoning chain r given the input and demonstrations, then the probability of the label y conditioned on the input, demonstrations, and the generated reasoning chain.
5

---

Demonstration Selection for In-Context Learning via Reinforeement Learning
4. Experiments
(RDS) (Yang et al., 2023), which identifies diverse subsets of demonstrations for better generalization; and Adaptive Demonstration Selection for In-Context Learning (ADA) (Mavromatis et al., 2023), which focuses on uncertain cases to enhance robustness and adaptability. Together, these methods leverage different aspects of prompting and demon- stration selection to improve classification accuracy and response quality.
4.1. Datasets
In our framework evaluation, we utilize four widely recog- nized datasets that encompass diverse range of domains and intents. The BANKING77 dataset (Casanueva et al., 2020) provides comprehensive set of intents specifically relevant to the banking sector. Additionally, the HWU64 (Liu et al., 2021) and LIU54 (Liu et al., 2021) datasets offer extensive multi-domain coverage, making them partic- ularly valuable for comparative analysis. We also include the CLINC150 dataset (Larson et al., 2019), which further enriches our evaluation framework. To better align our eval- uation with real-world application scenarios, we employed a challenge set sampling strategy, drawing the principles outlined in (Lu et al., 2024). This approach allowed us to select a demanding subset from the original test splits based on the precision margin, ensuring a rigorous assessment of our model's performance. To further assess the generaliz- ability and effectiveness of RDES on tasks requiring more complex reasoning, we also conducted additional experi- ments challenging benchmarks. These include subsets of BigBenchHard (Suzgun et al., 2023) (specifically, boolean expressions and web of lies), GSM-8K (Cobbe et al., 2021), and SST5 (Socher et al., 2013). These datasets require ad- vanced reasoning capabilities from the LLMs and were used to evaluate how RDES performs in more complex scenarios compared to baselines.
4.3. LLMS Used in Experiments
To evaluate our method, we employ a diverse set of LLMs, including both closed-source and open-source mod- els. Closed-source models such GPT-335-turbo, Doubao- lite-4k, Doubao-pro-4k, and Hunyuan-lite, developed by industry leaders, offer strong NLP capabilities for tasks like content generation and long-context understanding (Zhang et al., 2024; Team, 2024a; Cloud, 2024). In contrast, open-source models-innluuing Gemma-2-2BB Gemma-2- 9B. LLaMA-3.2- LLaMA-3.2-3BB LLaMA-3-8B, Qwen- 2B5-7B, Qwen-2.5-148 and Qwen-1.5-72B- customization and adaptability. The Gemma series empha- sizes efficiency, the Qwen series excels in scalability, and the LLaMA series is known for strong benchmark perfor- mance (Team et al., 2024; Bai et al., 2023; Yang et al., 2024; Dubey et al., 2024). The open-source nature of these mod- els fosters innovation and wider experimentation compared to proprietary alternatives. Our primary experiments on the four classification benchmarks (BANKING77, HWU64, CLINC150, LIU54) were conducted using RDES/B (our base version) and RDES/C (RDES with CoT reasoning), both of which are based on the Q-learning framework de- tailed in Section 3.2.1. Furthermore, we conducted addi- tional experiments on challenging benchmarks such as sub- sets of BigBenchHard (boolean expressions and web of lies), GSM-8K, and SST5. For these supplementary experiments, we specifically utilized models known for their strong per- formance in complex tasks: Qwen-25-728 25-BBB (Team, 2024b) and DeepSeek-R 1-22B (DeepSeek-Al. AII, 2025). Crucially, in these supplementary experiments, we evaluated not only the Q-learning based variants (RDES/B and RDES/C) but also a PPO-based variant, denoted as RDES/PPO. The results and analysis using these models and RDES variants on the challenging reasoning benchmarks are presented in Section 4.5.
greater
4.2. Compared Methods
In this study, we evaluate ten baseline approaches for their ef- fectiveness in various classification tasks, categorizing them into two main groups: Prompt Engineering Methods and Demonstration Selection Methods. The Prompt Engineering Methods manipulate prompt structures to enhance model understanding and decision-making, including Zero-Shot Prompting (ZS) (Radford et al., 2019), which tests general- ization without prior demonstrations; Knowledge Prompting (KP) (Liu et al., 2022), which provides contextual informa- tion improve accuracy; Least-to-Most Prompting (1.2M) (Zhou et al., 2023), which breaks tasks into manageable steps; Chain of Thought (CoT) prompting (Wei et al., 2022), which encourages step-by-step reasoning; and Self-Refine (SF) (Madaan et al., 2024), which allows the model to itera- tively critique and refine its solutions. The Demonstration Selection Methods utilize selected demonstrations inform predictions, including Few-Shot Prompting (FS) (Lu al., 2024), which enhances predictions through a limited num- ber of text-label pairs; Few-Shot with CoT (FSC) (Lu et al., 2024), which combines demonstrations with explanations; Active Demonstration Selection (AES) (Zhang et al., 2022), which iteratively selects relevant demonstrations to improve learning efficiency; Representative Demonstration Selection
4.4. Reasoning Performance Analysis
This section presents a comprehensive evaluation of the reasoning accuracy of both closed-source and spen-source LLMs using a range of prompt engineering and demonstra- tion selection techniques across four benchmark datasets: BANKING77, CLINC150, HWU64, and LIU54. The evalu- ation focuses on popular closed- source models such as GPT- 3.5-turbo, Doubao-lite-4k, Doubao-pro- -ro044,, and Hunyuan-
6

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
Table 1. Performance comparison of methods designed to boost L.M reasoning across various datasets closed-source M.M.s,, with
Tocus on accuracy. The RDES/B denotes the base version, while RDES/C indicates the version enhanced with CoT reasoning.
Prompt Engineering Methods
Demonstration Selection Methods
Ours
Datasets
Models
ZS
KP
L2M
CoT
FS
FSC
AES
RDS
ADA
RDES/B
RDES/C
G-rrbb
0.340
0.240
0.260
0.200
0.380
0,520
0.320
0.260
0.240
0.360
0.767
0.858
Doubao-lite-4k
0.300
0,300
0.300
0.320
0.300
0,500
0.360
0.300
0.280
0.400
0.750
0.830
BANKING77
Doubao-pro-4
0.500
0.400
0.500
0.480
0.600
0.540
0.540
0.700
0.680
0.900
0.838
0.888
Hunyuan-lite
0,300
0.233
0.433
0.200
0,300
0.233
0.133
0.320
0.320
0.600
0.593
0.775
Average
0.360
0.293
0.373
0.300
0.395
0.448
0.338
0.395
0.380
0.565
0.737
0.838
GPT-3.5-turbo
0.460
0.420
0.400
0.480
0.460
0.600
0,380
0,300
0,380
0.720
0.845
0,949
Doubac-lite44k
0.700
0,600
0.600
0.700
0,500
0.680
0,440
0.680
0.660
0.700
0.825
0,927
CLINC150
Doubao-proo
-ro---
0.660
0,680
0.620
0.700
0.700
0,800
0.640
0.680
0,640
0,900
0.938
0.961
Hunyuan-lite
0.633
0.800
0.767
0.700
0.633
0.467
0.500
0.480
0.620
0.800
0,730
0.772
Average
0.613
0.625
0.597
0.645
0.573
0.637
0.490
0.535
0.575
0.780
0.835
0,902
GPT-3.5-turbo
0.260
0.360
0.280
0.540
0.280
0,560
0,360
0.100
0.260
0.520
0.850
0.914
Douhao-lite-4k
0.500
0,500
0.500
0,480
0.500
0.520
0.340
0.360
0.420
0.700
0.765
0.873
HWU64
Doubao-pro-ke
0.640
0.760
0.620
0,800
0.640
0.680
0.600
0.620
0,640
1.000
0.862
0.918
Hunyuan-lite
0.533
0.367
0.333
0.433
0.233
0.600
0.433
0.540
0.320
0,700
0.514
0.784
Average
0.483
0.497
0.433
0.513
0.413
0.590
0.433
0.405
0.410
0.730
0.748
0.872
GPT-3.5-turbo
0,380
0,260
0.360
0,460
0.240
0.480
0,480
0.140
0.180
0,300
0.743
0.868
Doubao-lite-4k
0.500
0.400
0.500
0.540
0.660
0.600
0.440
0.520
0.520
0.600
0.690
0.841
LIU54
Doubao-pro-44
0.400
0.420
0.400
0.520
0.520
0.800
0.760
0.500
0.520
0.900
0,829
0.884
Hunyuan-lite
0.533
0.500
0.567
0.700
0.633
0.367
0,500
0.460
0.620
0.560
0.565
0,704
Average
0.453
0.395
0.457
0.555
0.513
0.562
0.545
0.405
0.460
0.590
0.707
0.824
lite, as well as open-source alternatives including Gemma, LLaMA, and Qwen models. Each dataset is used to test the models' accuracy in understanding and classifying various domain-specifie tasks. The results are presented in tables, with the lop-performing techniques highlighted in bold and the second-best results underlined for clarity.
formance across various datasets, reinforcing its versatility when paired with advanced techniques like RDES/C. The comparison between RDES/B and RDES/C reveals that the latter's incorporation of CoT reasoning consistently leads to superior performance, as seen in the BANKING77 dataset with an average accuracy of 0.838, significantly surpassing traditional methods like SF and ADA. This trend is echoed across other datasets, with RDES/C achieving an average ac- curacy of 0.902 in CLINC150, further emphasizing its effec- tiveness. Overall, the evaluation underscores the substantial impact of advanced prompt engineering and demonstration selection techniques on the performance of closed- source LLMs, suggesting that adaptive, context-aware prompting strategies, particularly those integrating CoT reasoning, are essential for optimizing LLMs for domain-spedne specific tasks and guiding future research in this area.
4.4.1. ANALYSIS OF REASONING PERFORMANCE ON CLOSED-SOURGE MODELS
As illustrated in Table 1, RDES/B and RDES/C consistently outperform alternative methodologies across the evaluated datasets, with RDESIC, which incorporates CoT reason- ing, achieving the highest accuracy scores in nearly all instances. While no single prompt engineering method dominated across all datasets, both KP and CoT reasoning yielded strong results, particularly on the HWU64 dataset, where CoT achieved the highest average accuracy of 0.5 Task-speeiff prompt engineering appears crucial, though its advantages are often dataset-dependent. In terms of demon- stration selection, ADA and FSC produced competitive re- sults, especially on the CLINC150 and HWU66 datasets, with ADA showing flexibility in curating demonstrations based on task-specific factors. However, it was generally outperformed by RDES/B and RDES/C, indicating that the integration of CoT reasoning in RDES/C provides signifi- cant benefits. Among the evaluated models, Doubao-pro- 4k excelled, achieving peak performance score of 0.961 on the CLINC150 dataset with RDES/C, while Doubao- lite-4k struggled, particularly on challenging datasets like HWU64, highlighting the importance of model capacity and architecture. GPT-3.5-turbb also demonstrated stable per-
4.4.2. ANALYSIS OF REASONING PERFORMANCE ON OPEN-SOURCE MODELS
Table 2 reveals significant performance variations across dif- ferent datasets, highlighting the unique challenges faced by open-source LLMs. Our RDES/C approach consistently out- performs other methods in the BANKING77 dataset, demon- strating its effectiveness in understanding fine-grained cus- tomer service intents, and shows similar robustness in the HWU64 dataset across diverse user queries. The CLINC150 dataset, characterized by its technical nature, benefits no- tably from our methods, particularly when utilizing larger models like Dwwn--1.5-725 underscoring the importance of scale in managing domain-specifis content. In the LIUS4 dataset, which features specialized queries, both RDES/B
7

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
Table 2. Performance comparison of methods designed to boost I.LM reasoning across various datasets on open-souree L.Mss, with
Tocus on accuracy. The RDES/B denotes the base version, while RDES/C indicates the version enhanced with CoT reasoning.
Prompt Engineering Methods
Demonstration Selection Methods
Ours
Datasets
Models
7.
KP
1.2M
CoT
SF
FS
FSC
AFS
RDS
ADA
RDES/B
RDES/C
G
0.200
0.280
0.200
0.200
0.260
0.300
0.340
0.280
0.220
0.900
0.831
0.861
Gemma-2-9B
0.560
0.400
0.500
0.500
0.400
0,440
0.380
0.400
0.400
0.700
0.831
0.886
LLaMA-3.2-IB IB
0.120
0.100
0.000
0.000
0.100
0.000
0.000
0.020
0.040
0.680
0.024
0.744
LLaMA-332-3B
0.200
0.200
0,400
0.500
0,300
0.320
0.060
0,360
0.440
0.700
0.770
0.805
BANKING77
LLaMA-3-88
0.578
0.560
0.563
0.552
0.458
0.090
0.182
0.531
0.536
0.758
0.784
0.847
Qwen-2.5-7B
0.700
0.480
0,600
0,420
0.480
0,440
0.180
0.440
0.420
0.700
0. 8.80
0.859
Qwen-2.5-14B
0.400
0.400
0.420
0.420
0.420
0.480
0.460
0.500
0.520
0.800
0.839
0.868
Qwen-1.5-72B
0.529
0.480
0,524
0.528
0.551
0.653
0.612
0.509
0,542
0.775
0.785
0.892
Average
0.411
0.363
0.401
0.390
0,371
0.340
0.277
0.380
0.390
0.752
0.708
0.845
Gemma-2-2B
0.400
0,600
0,420
0.460
0.560
0,560
0.540
0,500
0.380
0.800
0.875
0.929
Gemma-2-9R
0.700
0.700
0.800
0.800
0.700
0.800
0.680
0.800
0,800
0.780
0.864
0.819
L.L.M-3,2-11B IB
0.400
0.520
0.060
0.380
0.600
0.000
0.020
0.080
0.080
0.400
0.256
0.122
LLaMA-3.2-3B
0.800
0.700
0.800
0,400
0,700
0,580
0.260
0.600
0,680
0.800
0.845
0.703
CLINC150
LLaMA3-8B
0.523
0.439
0.594
0.504
0.569
0.007
0.285
0.571
0.543
0.767
0,840
0.783
Qwen-2.5-7B
0.740
0.800
0,800
0.780
0.700
0.740
0.460
0,780
0.780
0.800
0.879
0,741
Qwen-2.5-14B
0,900
0.900
0,900
0.800
0.840
0.700
0.700
0.740
0,740
0.900
0.944
0.792
Qwen-1.5-72B
0.726
0.517
0.683
0.641
0.660
0.850
0.652
0.696
0.656
0.861
0.897
0,963
Average
0.649
0.647
0.632
0.596
0,666
0,530
0,450
0.596
0,582
0.763
0.800
0.731
Gemma2-2B
0.300
0.320
0.300
0.300
0.400
0.460
0,440
0.420
0.360
0.600
0. 0.832
0,851
Gemma2-9B
0.600
0,600
0,600
0.600
0.600
0.700
0.700
0,700
0.700
0.800
0.910
LLAMA-3.2-1BB
0.200
0.100
0.080
0.000
0.100
0.020
0.000
0.020
0.060
0.360
0.381
0.687
LLaMA-3.2-3B
0.300
0.100
0.300
0.200
0,300
0.220
0.180
0.300
0.300
0.700
0.747
0,817
HWU64
LLaMA-3-8B
0.478
0.407
0.493
0.479
0.563
0,632
0.498
0.651
0.645
0.837
0,816
0.859
Qwen-2.5-7B
0.780
0.700
0.800
0.600
0.800
0.640
0.540
0.760
0.740
0.800
0.805
0.880
Qw5n-2.5-14B
0.780
0.800
0.800
0.440
0.740
0.800
0.800
0.720
0.680
0.900
8888
0.895
Qwen-1.5-72B
0.698
0.615
0.676
0.661
0.668
0,825
0.817
0.749
0.774
0.877
0.867
0.924
Average
0.317
0.455
0.306
0,410
0.521
0.537
0.497
0.540
0.532
0.734
0.776
0.853
Gemmu-2-2B
0.400
0.400
0,500
0.400
0.400
0.620
0.480
0.500
0.440
0.600
0.733
0.854
Gemma-2-99
0.500
0,500
0.600
0.600
0.600
0,580
0,580
0.500
0.500
1.000
0.722
0,837
LLaMA-332-1B
0.200
0.160
0.300
0.400
0.080
0.040
0.040
0.360
0.320
0.700
0.058
0.651
LLLMMA332-3B
0.400
0.400
0,400
0.300
0.400
0.360
0.320
0.500
0,400
0.600
0.772
0.749
LIU54
LLaMA-3-8B
0.358
0.409
0.428
0.360
0,392
0.396
0,320
0.347
0.312
0.763
0.779
0,811
Qwen-2.5-7B
0.800
0.700
0.700
0.640
0.520
0.620
0.500
0.500
0.660
0.804)
0.794
0.765
Owen-2.55114
0.700
0,860
0.700
0.660
0.700
0.660
0.600
0.740
0.780
1.000
0.849
0,743
Qwen-1.5-72B
0.496
0.445
0.487
0.491
0.550
0.609
0.647
0.514
0.492
0.769
0.781
0,880
Average
0.482
0.484
0,514
0.481
0.455
0.486
0.436
0.495
0.488
0.779
0.686
0.786
and RDES/C exhibit considerable advantages, showcasing their capacity for nuanced reasoning. Methodologically, the comparison highlights the strengths and weaknesses of various prompt engineering and demonstration selection techniques, with ZS and KP showing limitations compared to ADA and our RDES methods. While demonstration selection strategies like FS and FSC provide moderate im- provements, they are often outpaced by more sophisticated approaches like AES and ADA, validating the efficacy of CoT reasoning in enhancing model understanding, particu- larly for complex tasks.
when combined with advanced methods like RDES/C, high- lighting the synergistic effect of scale and sophisticated reasoning techniques. Overall, our findings emphasize that RDES methods, particularly when integrated with CoT rea- soning, provide a clear advantage across all datasets, rein- forcing the necessity of computational resources for achiev- ing higher accuracy. The dataset-specific trends further underscore the importance of tailored approaches, as differ- ent datasets require distinct handling strategies for optimal results. This study suggests that future research should focus on refining adaptive and CoT-based methods to en- hance model generalization across domains, positioning adaptive reasoning as a core component for advancing LLM architectures and increasing their versatility in real-world applications.
The analysis reveals distinct performance patterns tied to model architecture and scale. Smaller models, such as Gemma-2-2B and LLaMA-3.2-IB, generally exhibit lower accuracy, particularly with simpler prompting strategies, in- dicating limited capacity for nuanced comprehension. In contrast, larger models like Qwen-2.5-14D and 1.en-1.5- 72B show marked performance improvements, especially
8

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
4.4.3. AVERAGE PERFORMANCE CLOSED-SOURCE AND OPEN-SOURCE MODELS
and web of lies subsets), SST5 and GSM-8K. These datasets require advanced reasoning capabilities from the LLMs, al- lowing us to evaluate how different demonstration selection methods perform in complex scenarios. We evaluated the performance of our proposed RDES variants (RDES/B and RDES/C based on Q-learning, and RDES/PPO based on PPO) against several baseline methods (FS, FSC, AES, RDS, ADA) on these benchmarks, utilizing both the DeepSeek- R1-32B and Qwen-2.5-72B models.
The data presented in Figure 3 summarizes the average performance results of various LLMs, encompassing both closed-source and open-source variants, across different datasets. This figure highlights the effectiveness of prompt engincering and demonstration selection methods in compar- ison to our proposed approaches, RDES/B and RDES/C. In the BANKING77 dataset, RDES/C achieved a remarkable accuracy of 0.843, significantly surpassing other methodolo- gies, including RDES/B at 0.718 and ADA at 0.689, while the baseline method, ZS, recorded the lowest accuracy of 0.394. In the CLINC150 dataset, RDES/B demonstrated strong performance with an accuracy of 0.812, followed closely by RDES/C at 0.788, with ADA achieving 0.769 and the KP method at 0.640. The HWU64 dataset further highlighted RDES/C's superiority, it led with accuracy of 0.859, while RDES/B achieved 0.767 and ADA recorded 0.733, with ZS lagging at 0.506. Finally, in the LIU54 dataset, RDES/C attained an accuracy of 0.799, outperform- ing RDES/B at 0.693 and ADA at 0.716, while the CoT method exhibited an accuracy of 0.506. Overall, these re- sults illustrate the effectiveness of our proposed approaches in enhancing model performance through advanced prompt engineering and demonstration selection strategies, under- scoring their potential to improve classification accuracy across various applications.
As shown in Table 5 in Appendix A.6, the results on these reasoning benchmarks further highlight the competitive per- formance of RDES, especially RDES/C which incorporates CoT reasoning, and RDES/PPO. For instance, RDES/C achieves the highest accuracy on both BigBenchHard and GSM-8K with the DeepSeck-RI-325 model. With the Qwen-2.5-72B model, ADA shows the highest accuracy these specific reasoning tasks, while RDES/PPO also achieves competitive results. These findings demonstrate that RDES, by effectively balancing relevance and diver- sity in demonstration selection through a RL framework, can maintain strong performance even on tasks requiring more complex reasoning, supporting its broader applicabil- ity beyond straightforward classification. The exploration of different RL algorithms like PPO also shows promise for these tasks.
5. Conclusion
BANKING77
CLINC160
In this study, we introduced RDES, a novel framework uti- lizing RI. (specifically Q-learning, with exploration into PPO-based variant) to optimize demonstration selection for ICL in LLMs by balancing relevance and diversity, thereby enhancing generalization and mitigating overfitting. Our extensive evaluation against ten baselines on four bench- mark classification datasets demonstrated that RDES sig- nificantly outperforms existing methods. We also showed that integrating RDES with CoT reasoning (RDES/C) gen- erally enhances performance, though its benefit vary depending on model and dataset characteristics. We con- ducted additional experiments on more challenging reason- ing benchmarks and with a variable number of demonstra- tions, which further validated RDES's effectiveness and highlighted diversity-driven generalization, especially with the RDES/PPO variant, even in complex tasks or varying settings. These results underscore the potential of RL to facilitate adaptive demonstration selection and its promise for addressing complexities in NLP tasks. Future work in- cludes refining diversity metrics, extending RDES to tasks beyond classification like generation and question answer- ing, making CoT usage adaptive within the RL framework, analyzing computational cost/sample efficiency, exploring different retrieval methods, and assessing the generalization capabilities of strategies across datasets.
0.689
0.718
0.640
0,789
0,708
0.394
Z8
ADA
RDES/B
ADES/C
KP
ADA
RDES/B
RDES/C
0.733
HWU66
0,767
0.85A
0.718
LIUS4
0.683
0,798
ADA
RDES/B
RDESIC
COT
ADA
RDES/C
Dest Results in PE
Best Results in DG
RDDS/B
RDES/C
Figure 3. These figures illustrate the average results across closed- source/open-soutee source models on different datasets, comparing the best results from the prompt engineering (PE) and demonstration selection (DS) methods with our proposed approach.
4.5. Evaluation on More Challenging Reasoning Tasks
To further assess the generalizability and effectiveness of RDES beyond simple text classification, we conducted ad- ditional experiments on more challenging reasoning bench- marks: BigBenchHard (specifically, boolean expressions

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
Acknowledgements
Y., Ge, W., Han, Y., Huang. F., al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
This work was supported in part by the Chinese National Research Fund (NSFC) under Grant 62272050 and Grant 62302048; in part by the Guangdong Key Lab of Al and Multi-modal Data Processing, United International Col- lege (UIC), Zhuhai under 2023-2024 Grants sponsored by Guangdong Provincial Department of Education; in part by Institute of Artificial Intelligence and Future Networks (BNU-Zhubai) and Engineering Center of Al and Future Education, Guangdong Provincial Department of Science and Technology, China; Zhuhai Science-Tech Innovation Bureau under Grant No. 2320004002772, and in part by the Interdisciplinaryy Intelligence SuperComputer Center of Beijing Normal University (Zhuhai).
Bai, Y., Chen, E., Wang, H., Xiong, C., and Mei, S. Trans- formers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Bender, E. M., Gebru, T., McMillan-Maan Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on Fairness, Accountability, and Trans- parency, pp. 610-623, 2021.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 1877-1901, 2020.
Impact Statement
This paper introduces the RDES framework, leveraging RL to optimize demonstration selection for LLMs in few- shot text classification and reasoning tasks by balancing relevance and diversity. The primary positive impact is significant enhancement in LLM accuracy and robustness in data-limited scenarios, making them more effective for practical applications like intent detection and sentiment analysis. This approach helps mitigate the overfitting bi- ases often associated with purely similarity- -based selec- tion. However, potential negative impacts include the risk that enhanced classification capabilities could be misused for applications like surveillance or censorship. While not the focus, extending the core method to generative tasks could, in the future, potentially contribute to the spread of misinforration. Furthermore, training RDES involves significant computational cost due to numerous LLM calls, potentially limiting its accessibility. The current work lacks user studies, meaning real-world human-centric impacts like user over-reliance are not yet evaluated. To maximize positive impacts and mitigate risks, future work should ex- plore computational efficiency improvements, necessitate safeguards against misuse, partieularly if extended to more complex tasks like generation, and conduct user studies to comprehensively assess real-world performance and user interaction. Adherence to strong ethical principles regarding transparency, fairness, and accountability is encouraged in the application and dissemination of this technology.
Casanueva, I., Temčinas, T., Gerz, D., Henderson, M., and Vulić, I. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pp. 38-45, 2020.
Cloud, T. Hunyuan, 2024. URL https: //cloud. tencent. coomproodut/hunyuue
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv 2021.
DeepSeck-AL. Deepseek-rl: Incentivizing reasoning ca- pability in llms via reinforcement learning, 2025. URL
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint 180.:408,, 2018
Dong, Q., Li, L.., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z.,, Chang, B., et al. A survey on in- context learning, In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1107-1128, 2024.
References
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llaaa 3 herd of models. arXiv preprint 2407:2200722783 2024.
An, S., Zhou, B., Lin, Z., Fu, Q., Chen, B., Zheng, N.. Chen, W., and Lou, J.-G. Skill-based few-shot selection for in- context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13472-13492, 2023.
Gao, T., Yao, X., and Chen, D. Simese: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- 2021.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan,
guage Processing, pp.
10

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
Gupta, S., Gardner, M., and Singh, S. Coverage-based example selection for in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 13924-139993 2023.
Lu, Z., Tian, J., Wei, W., Qu, X., Cheng, Y., Xie, W., and Chen, D. Mitigating boundary ambiguity and inherent bias for text classification in the era of large language models. In Findings of the Association for Computational Linguistics ACI. 2024, pp- 7841-7864, 2024.
Hu, Z.. Feng, Y., Luu, A. T., Hooi, B., and Lipani, A. Un- locking the potential of user feedback: Leveraging large language model as user simulators to enhance dialogue system. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 3953-3957, 2023.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, I.., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang. Y., et al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems, 36, 2024.
Langley, P. Crafting papers machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Marcus, G. The next decade in ai: four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.06177, 2020.
Mavromatis, C., Srinivasan, B., Shen, Z.,, Zhang, J., Rang- wala, H., Faloutsos, C.,, and Karypis, G. Which examples to annotate for in-context learning? towards effective and efficient selection. arXiv preprini 1001v23310.20046, 2023.
Larson, S., Mahendran, A., Peper, J. J., Clarke, C., Lee, A., Hill, P., Kummerfeld, J. K., Leach, K., Laurenzano, M. A., Tang, L., et al. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 1311-1316, 2019.
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H.,, and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1 11048- 11064, 2022.
Levy, I., Bogin, B., and Berant, J. Diverse demonstrations improve in-context compositional generalization. In Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1401-1422, 2023.
Qin, C.,, Zhang, A., Chen, C., Dagar, A., and Ye, W. In- context learning with iterative demonstration selection. In Findings of the Association for Computational Linguis- tics: EMNLP 2024, pp. 7441-7455, 2024.
Liu, J.. Liu, A., Lu, X., Welleck, S., West, P., Le Bras, R., Choi, Y., and Hajishirzi, H. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pp. 3155- 3169, 3169,0022.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Scarlatos, A. and Lan, A. Reticl: Sequential retrieval of in-context examples with reinforcement learning. arXiv preprint arXiv:2305. 2023.
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
Sener, O. and Savarese, Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
Liu, X., Eshghi, A., Swietojanski. P., and Rieser, V. Bench- marking natural language understanding services for building conversational agents. In Increasing natural- ness and flexibility in spoken dialogue interaction: 10th international workshop on spoken dialogue systems, pp. 165-183. Springer, 2021.
Settles, B. Active learning literature survey. 2009.
Shao, Z.,, Yu, Z., Wang, M., and Yu, J. Prompting large language models with answer heuristics for knowledge- based visual question answering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14974-14983 2023.
Liu, Y. and Lapata, M. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing LEMNLP-IJCNLP pp. 3721. Association for Computational Linguisties, 2019.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods natural language processing, pp. 1631-1642, 2013.

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
Song, Y., Wang. T., Cai, P., Mondal, S. K., and Sahoo, J. A comprehensive survey of few-shot learning: Evolu- tion, applications, challenges, and opportunities. ACM Computing Surveys, 55(13s):1-40, 2023.
Yang, Z., Zhang, Y., Sui, D., Liu, C., Zhao, J., and Liu, K. Representative demonstration selection for in-context learning with two-stage determinantal point process. In Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 5443-5456, 2023.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le. Q., Chi, E., Zhou, D., et al. Challenging big-bench tasks and whether chain-of- thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13003-13051, 2023.
Ye, J., Wu, Z., Feng, J., Yu, T., and Kong. L. Compositional exemplars for in-context learning. In International Con- ference on Machine Learning, pp. 39818-39833. PMLR, 2023.
Zhang, G., Jin, Q.. Zhou, Y., Wang, S., Idnay, B., Luo, Y., Park, E.,, Nestor, J. G., Spotnitz, M. E., Soroush, A., et al. Closing the gap between open source and commercial large language models for medical evidence summariza- tion. npj Digital Medicine, 7(1):239, 2024.
Team, D. Doubao pro models, 2024a. URL https:// team. doubao.......o com/ /en/
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., al. Gemma 2: Improving open language models at practical size. arXiv e-prints, pp. arXiv-2408, 2024.
Zhang, Y., Feng, S., and Tan, C. Active example selec- tion for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9134-9148 2022.
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024b. URL https:://// / /wwenlww 1m. wtm.m.ittttt thub. io/ /log//wen2.5/.
Zhang. Z., Zhang. A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. The Eleventh International Conference on Learning Represenaatioss, 2023.
Wang, P., Wang, X., Lou, C., Mao, S., Xie, P., and Jiang. Y. Effective demonstration annotation for in in-context learn- ing via language model-based determinantal point pro- cess. In Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing, pp. 1266-1280, 2024.
Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pp. 12697-12706. PMLR, 2021.
Wang, Y., Yao, Q., Kwok, J. T., and Ni, L. M. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys, 53(3):1-34, 2020.
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q. V., et al. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023.
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned Ian- guage models are zero-shot learners. In International Conference on Learning Representations, 2021.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., al. Chain-of-thougs prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824-24822 2022.
Wu, J., Wang, X., and Jia, W. Enhancing text anno- tation through rationale-driven collaborative few-shot prompting. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2025.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F, et al. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
12

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
A. Appendix
A.I. Algorithm Comparison between Q-learning and PPO
In this section, we provide a comparative analysis of two prominent RL algorithms: Q-learning and Proximal Policy Optimization (PPO). The table below highlights key aspects of each algorithm, including their policy types, exploration strategies, value estimation methods, and update rules. This comparison aims to elucidate the strengths and weaknesses of each approach in the context of RL. applications.
Table 3. Algorithm Comparison
Aspect Policy Type
Q-learning
PPO
Deterministic
Stochastic
Exploration
e-gredy
Entropy bonus
Value Est.
Tabular
Neural network
Update Rule
Temporal Difference
Surrogate objective
A.2. Convergence Analysis
Theoretical analysis supports the convergence properties of the adopted RL algorithms under standard conditions.
A.2.1. Q-LEARNING CONVERGENCE
Under standard stochastic approximation conditions for the learning rates and bounded rewards, the Q-learning updates are guaranteed converge.
Theorem A.1 (Q-Learning Convergence). Given learning rates og satisfying the Robbins-Monro conditions (Ei= 0g x and Ei= of <. oc), and bounded rewards [re] S RRmax the Q-learning updates converge almost surely to the optimal Q-function Q*.
A.2.2. PPO POLICY IMPROVEMENT
The PPO algorithm's clipped objective provides theoretical guarantees on policy improvement in each update step. Theorem A.2 (Policy Improvement). For any policy Nh a policy Te.... updated via PPO's clipped objective with advantage estimates Å, satisfiex:
B [Ar] 2 ETo
(12)
This means the expected advantage of the new policy is greater than or equal to that of the old policy, guaranteeing monotonic policy improvement unless the policy has alreudy converged.
A.3. Datasets
In this section, we present a summary of the datasets used in our experiments. Each dataset is characterized by its unique intents, the size of the knowledge base (KB), the number of test samples, and the distinct domains it covers. Table 4 provides comprehensive overview of these datasets:
Dataset
Intents
Size of KB
Test
Domains
BANKING77
77
9,003
3,080
1
CLINC150
150
18,000
2,250
10
HWU64
64
8,828
1,104
21
LIUS4
54
20,382
2,548
21
Table 4. Summary of Experimental Datasets: This table presents the number of unique intents represented in each dataset (Intents), the total number of knowledge base entries available for retrieval (Size of KB), the number of test samples used for evaluation (Test), and the number of distinct domains covered by each dataset (Domains).
In addition to the datasets summarized in Table 4, we performed supplementary experiments on several reasoning benchmarks
13

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
evaluate RDES performance on tasks requiring more complex capabilities. These datasets, including BigBenchHard (Suzgun al., 2023) (boolean expressions and web of lies subsets), GSM-8K (Cobbe et al., 2021), and SST5 (Socher et al., 2013), were utilized to address reviewer feedback and demonstrate the framework's ' applicability beyond standard text classification. Due to time constraints, we randomly sampled 1,000 examples from the test sets for evaluation. The experimental results and analysis on these benchmarks are presented and discussed in the main body.
A.4. Baselines
In this study, we conduct a comprehensive evaluation of ten baseline approaches to assess their effectiveness in various classification tasks. These approaches are categorized into two main groups: Prompt Engineering Methods and Demonstra- tion Selection Methods. Each method employs distinct strategies to enhance the model's performance, leveraging different aspects of prompting and demonstration selection to improve classification accuracy and response quality.
A.4.1. PROMPT ENGINEERING METHODS
This category focuses on techniques that manipulate the structure of prompts to guide the model's understanding and decision-making process. The methods included in this category are:
Zero-Shot Prompting (ZS) (Radford al., 2019): This method prompts the model classify text without providing prior demonstrations, directly asking it to select the most appropriate label from predefined set of options. This approach tests the model's 's ability to generalize from its training data.
Knowledge Prompting (KP) (Liu et al., 2022): This technique prompts the model to generate relevant contextual information about the input text before selecting label. By providing additional context, this method may enhance classification accuracy and improve the model's understanding of the task.
Least-to-Most Prompting (L2M) (Zhou et al., 2023): This approach structures the classification task into smaller, manageable steps. guiding the model through the selection process. By breaking down the task, the model can focus on each component, potentially leading to more accurate predictions.
Chain of Thought (CoT) (Wei et al., 2022): This variant encourages the model to articulate its reasoning step-by-step before arriving at a classification. By making the reasoning process explicit, CoT prompting can enhance response quality and provide insights into the model's decision-making.
Self-Refine (SF) (Madaan et al., 2024): This approach prompts the model to solve problem, critique its own solution, and then refine its answer based on the critique. This iterative process continues until a stopping condition is met, allowing the model to improve its responses through asfessssessmnt.
A.4.2. DEMONSTRATION SELECTION METHODS
This category encompasses methods that utilize a selection of demonstrations to inform the model's predictions. The approaches included in this category are:
Few-Shot Prompting (FS) (Lu et al., 2024): This approach provides the model with a limited number of text-label pairs selected randomly. By leveraging contextual information from these demonstrations, the model can enhance its predictions based on the provided demonstrations.
Few-Shot with CoT (FSC) (Lu et al., 2024): This method combines few-shot learning with reasoning, presenting demonstrations alongside explanations to guide the model's classification process. This integration aims to improve the model's understanding and accuracy in making predictions.
Active Demonstration Selection (AES) (Zhang et al., 2022): This approach involves iteratively selecting and annotating unlabeled demonstrations to enhance ICL using RL methods. By actively choosing relevant demonstrations, the model can improve its learning efficiency.
Representative Demonstration Selection (RDS) (Yang et al., 2023): This method aims to identify a high-quality and diverse subset of in-context demonstrations that can effectively prompt various test instances for a specific task. By ensuring diversity, this approach enhances the model's ability to generalize across different scenarios.
14

---

Demonstration Selection for In-Context Learning via Reinforeement Learning
Adaptive Demonstration Selection for In-Context Learning (ADAICL, ADA) (Mavromatis et al., 2023): This approach employs model-adaptive, optimization-free algorithm identify uncertain demonstrations and perform semantic diversity-based selection. By focusing on uncertain cases, the model can improve its robustness and adaptability.
A.5. LLMs Used in Experiments
To evaluate the effectivvnness of the proposed method, we employ diverse range of LLMs, encompassing both closed-source and open-source options. Closed-source models, such as GPT-3.5-turbo, Doubao-lite-4k, Doubao-pro-4k, and Hunyuan-lite, are proprietary systems developed by leading technology companies. For instance, renowned for its advanced NLP capabilities, making it a popular choice for various applications, including conversational agents and content generation (Zhang et al., 2024). Similarly, Doubao-pro released by ByteDance in May 2024, excels in multiple benchmarks, demonstrating strong performance in natural language understanding and generation tasks, positioning as a versatile tool for applications ranging from question answering to complex text creation (Team, 2024a). Hunyuan lite, developed by Tencent, is distinguished by its extensive parameter count and advanced capabilities in handling long-context inputs, thereby enhancing its performance across diverse tasks (Cloud, 2024).
created by OpenAI, is
In contrast, open-source models such as 6-2mm-2-22B Gemma-2-99. LLaMA-3.2-1 3.2--B, LLaMA-3.2-3B, LLaMA-3-8BB
Qwen-2.5-77, Qwen-2.5-14B, and Qwen-1.5-72B offer researchers and developers the flexibility to modify and adapt the models for specific use cases. The Gemma series emphasizes efficient training techniques while maintaining high performance across various NLP tasks, encouraging customization and experimentation within the community (Team et al., 2024). The Qwen series, particularly noted for its scalability and adaptability, allows users fine-tune models according to their needs, fostering collaboration and innovation in AI research (Bai et al., 2023; Yang et al., 2024). The LLaMA series has garnrred attention for its performance across various benchmarks while enabling users to tailor the models to their specific requirements (Dubey et al., 2024). The open-source nature of these models promotes collaboration and innovation within the Al community, facilitating a broader range of experiments and applications compared to their closed-source counterparts.
In addition to these models, for supplementary experiments conducted on challenging reasoning benchmarks (such as BigBenchHard, GSM-8K, and SST5) as detailed in Section 4.5, we also utilized the Qwen-2.5-72B (Team, 2024b) and DeepSeck-R1-322 (DeepSeek-Al, 2025) models. These additional experiments were performed to demonstrate RDES's performance in more complex scenarios and were specifically included in response to reviewer feedback. The detailed results and analysis using these models are presented in the main body of the paper.
A.6. Supplementary Results
Table 5 presents performance comparison of the Qwen-2.5-72B and DeepSeek-R1 -1-beekk--132BBBB-- models across several supplemen- tary datasets, including SST5, BigBenchHard (specifically focusing boolean expressions and the web of lies tasks), and GSM-8K. It lists various methods, encompassing traditional Few-Shot (FS) and Few-Shot with Chain of Thought (FSC) approaches, as well as demonstration selection methods such as Active Demonstration Selection (AES), Representative Demonstration Selection (RDS), and Adaptive ICL (ADA). Notably, the table also includes different variants of the pro- posed RL-based RDES method: RDES/B, RDES/C (the base version and the one combined with Chain of Thought), and RDES/PPO (the PPO-based variant). These supplementary results further support the efffettivnness of the RDES method across diverse tasks and models, including text classification, complex reasoning, and mathematical reasoning.
Table 5. Supplementary Performance Comparison on SSTS, BigBenchHHard, and GSM-8K
Methods
SST5
BigBenchHard- boolean expressions
BigBenchHard- web of
GSM-8K
Qwen-2.5.72B
DeepSeek-R1-32
Qwen-2.5-72R
DeepSeck-R1 SeoSeckkR1-325
Qwen-2.5.72R
DeepSeck R--32B
Qwen-2.5-722
DeepSeek-RP R1-33e
ES
0.56
0.70
0.98
0.38
0.58
0.98
0.50
0.28
FSC
0.54
0.66
0.60
0.46
1.00
1.00
0.56
0.64
AES
0.84
0.84
0.53
0.60
0.85
0.72
0.92
0.08
RDS
0.76
0.84
0.53
0.60
0.89
0.68
0.90
0.48
ADA
0.90
0.53
0.60
0.72
0.36
RDES/B
0.44
0.57
0,76
1.00
0.50
0.93
0.87
0.37
RDES/C
0.51
0.52
0.90
0.99
0.98
1.00
0.92
0.73
RDES/PPO
0.84
1.00
1.00
0.90
0.94
0.48
Table 6 presents experimental results on the GSM-8K and SST5 datasets, specifically investigating the impact of varying the number of demonstrations (k) used for ICL. The results are reported for the Qwen-72B model. The table evaluates
15

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
the performance (Aceuracy) of various demonstration selection methods, including FS, FSC, AES, RDS, ADA, RDES/B, RDES/C, and RDES/PPO, as the number of demonstrations is set 3, 5, 7, and 10. These results, included as part of the authors' experimental revisions, highlight how the performance of different methods can change depending on the size of the demonstration set.
Table 6. Performance of Methods Across Varying Numbers of Demonstrations (k) Using Qwen-2.5-728 Model
GSM-8K (Accuracy)
SST5 (Aceuracy)
Methods
k=3
k=5
k=7 k=10
k=3
k=5
k=7
k=10
FS
0.50
0.28
0.50
0.28
0.54
0.56
0.54
0.56
FSC
0.56
0.64
0.56
0.64
0.52
0.54
0.52
0.54
AES
0.92
0.08
0.92
0.08
0.82
0.84
0.82
0.84
RDS
0.90
0.48
0.90
0.48
0.74
0.76
0.74
0.76
ADA
0.98
0.36
0.98
0.36
0.88
0.90
0.88
0.90
RDES/B
0.87
0.37
0.87
0.37
0.42
0.44
0.42
0.44
RDES/C
0.92
0.73
0.92
0.73
0.49
0.51
0.49
0.51
RDES/PPO
0.94
0.48
0.94
0.48
0.82
0.84
0.82
0.84
A.7. Ablation Study
This ablation study investigates the impact of diversity pechanisms-nanee No-Diversity, RDES/B, and ADES/C-across both closed-source and open-source models using four datasets. The results, illustrated in Figures 4 and 5, consistently demonstrate that incorporating diversity generally enhances model performance, although the degree of improvement varies depending on the dataset and model type.
In the closed-source context (Figure 4), the RDESIC mechanism consistently yields superior results across all datasets. the BANKING77 dataset, RDES/C achieves an average accuracy of 0.838, significantly higher than the No-Diversity baseline of 0.600, indicating its effectiveness in handling diverse banking-relaitn related queries. Similarly, in the CLINC150 dataset, RDES/C reaches 0.902 in accuracy, outperforming the No-Diversity baseline of 0.770, demonstrating improved capacity to interpret a broad range of user intents. The HWU64 dataset also shows clear benefit from diversity mechanisms, with RDES/C achieving 0.872 accuracy, up from 0.690 in the No-Diversity setting, highlighting the mechanism's 's advantage in technical and domain-specific scenarios. In the LIU54 dataset, RDES/C's accuracy of 0.824 surpasses the 0.640 baseline of No-Diversity, further emphasizing the benefits of diversity for nuanced classification tasks. Overall, the RDES/C mechanism stands the most effective strategy for closed-source models, particularly in complex and varied environments.
For open-source models (Figure 5), the impact of diversity mechanisms shows more variation. In the BANKING77 dataset, RDES/C provides modest improvement, achieving average accuracy of 0.845 compared to the No-Diversity baseline of 0.747, suggesting moderate gains from diversity, especially in larger models such as Qwen-1.5-72B. However, in the CLINC150 dataset, RDES/B outperforms both No-Diversity (0.768) and RDES/C (0.731), reaching an average accuracy of 0.800. This suggests that the choice of diversity mechanism should be informed by dataset characteristics, as RDES/B appears more effective in certain contexts. A clearer trend is observed in the HWU64 dataset, where RDES/C significantly boosts accuracy to 0.853 from the No-Diversity baseline of 0.732, underscoring the value of diversity in handling complex queries. In the LIU54 dataset, performance differences subtler, with No- Diversitty and RDES/B yielding similar results (6.769 vs. 0.686), while RDES/C achieves slightly higher accuracy of 0.786. These findings indicate that while diversity mechanisms often enhance performance, the specific choice and impact may vary across different datasets and open-source models, requiring a nuanced approach to model and dataset pairing.
16

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
BANKING77
1.000
0.800
0.600
0.200
0.000
ROESA
9pt-3.5-turto
Doubao-Ita-4k
Doubac-pro-4k
Hunyuan-Ita
Average
ROESIC
Diversity
0.420
0.900
0.680
RDES/S
0.767
0,750
0.838
0.593
0.737
CLINC150
0.800
0.600
0.400
0.200
No chvenity
0.000
gpt 3.5-1ubbo
Doubao-Ite- lite-4kk Doubao pro-4x
Hunyuan-ite
Average
ROEAAC
NNo-Divrsity
0.680
0.900
0.770
#RDES/B
0.845
0.825
0.938
0.730
0.835
#DDES/C
0.949
0.927
0.951
0.772
0.902
HWU64
1.000
0.800
0.400
0.200
upt-3.6-turbo
Doubac-Ite-44 -fte-kkeekk Doubao-pro-4k
Hunyuan-iRe
Average
No Diversity
0.400
0.700
0.900
0.680
0.690
RDES/B
ROES/C
0.914
0.873
0.918
0.784
0.872
LIU54
1,000
0.800
0.400
0.200
got-3.5-turbo
Doubeo-ite-4t
Doubeo-pro-de
Hunyuan-fic fil
Average
#No Diversity
0.520
0.600
0.900
0.540
0.840
#DES/C
0.868
0.841
0.884
0.704
0.824
Figure These figures illustrate the performance of various closed-source models across different datasets, highlighting the impact of diversity mechanisms.
17

---

Demonstration Selection for In-Context Learning via Reinforcement Learning
1.000
No Diversty
RDES/B RDES/C
0.800
0.600
0.200
Gamma-2
Gamma-2
LLaMA-3
LLAMA-3
LLaMA3-B
Qwwnn-2.
QwAn-2.5. P.5-
Qwan-1 1.5-
73B
Average
# No- Mo--Diersitt
0.800
0.700
0.820
0.700
0.757
0.700
0,820
0.775
0.747
RDE8/B
0.831
0.831
0.024
0.770
0.784
0.803
0.839
0.785
0.708
RDES/C
I No Divanshy
RDES/B ROES/C
0,800
0.400
0.000
Gomma-32
LLaMA-3.
LLaMA3-8
Owen-2.5
Owen-2.5 148
Ow
2-18
2:3B
7B
728
Average
#No-Diveraity
RDES/B
0.875
0.884
0.256
0.845
0.840
0.879
0.944
0.897
0.800
RDES/C
0.929
0.819
0.122
0.703
0.783
0.741
0.752
0.963
0.731
1.000
No-Diversil RDES/B RDES/C
0.600
0.400
0.200
0.000
Gemma-2
LLAMA-3,
LLaMA-3,
LLaMAS-8
7B
148
72B
Average
No-Dhveralty
0.732
m RDES/B
0.832
0.877
0.881
0.747
0.816
0.805
0.886
0.867
0.776
RDES/C
0.851
0.910
0.687
0.817
0.859
0.880
0.895
0.624
0.853
1.000
No-Diversty RDES/B RDES/C
0.800
0.800
0.200
Gemmaa-2 2 Gemmaa-2 2
LLaMA-3,
LLaMA-3,
LLaMA3-8
Qwan-2.5
Owen-2.5
Qwen-115
Average
NNo-Divrrsity
0,600
1,000
0.700
0.600
0.891
0.800
0.980
0.783
0.769
RDES/BB
0.733
0.722
0.058
0.772
0.779
0.794
0.849
0.781
0.688
0.854
Figure 5. These figures illustrate the performance of various open-source models across different datasets, highlighting the impact of diversity mechanisms.
18


Reviews:
[{"review_id": "McUHsxyk28", "reviewer": "Reviewer_3ANe", "summary": "The paper presents RDES, a reinforcement learning framework for selecting diverse and relevant demonstrations to support in-context learning in text classification tasks. It formulates demonstration selection as a Markov Decision Process and employs Q-learning with a composite reward function to balance relevance and diversity. The authors report experimental evaluations on four benchmark datasets, noting that a variant of RDES incorporating Chain-of-Thought reasoning is associated with improved classification performance relative to ten baseline methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3ANe,\n\nThank you for your positive feedback on our work. Regarding your suggestions:\n\n- **Lack of units in tables**: We appreciate your suggestion and will explicitly label the units (e.g., accuracy will be presented as a percentage) in all tables in the revised manuscript.\n\n- **No user studies**: We understand the importance of user studies for evaluating LLM quality. However, the focus of this research is to introduce a novel reinforcement learning-based demonstration selection method and evaluate its effectiveness through quantitative metrics (classification accuracy). Future work can consider conducting user studies to more comprehensively assess the performance of RDES in real-world applications. Meanwhile, we have launched a RAG system on our school's platform, and we will actively integrate our methods into this system.\n\nTo further support our claims, we have included additional experimental results on more challenging reasoning tasks. Here are the results on SST5, BigBenchHard (boolean expressions, web of lie), and GSM-8K using Qwen25-72b and Deepseek-r1-32b models; The RDES series is our proposed method, where RDES/B and RDES/C are based on Q-learning, and RDES/PPO is based on the PPO method:\n### **SST5**\n| Methods | Qwen25-72b | Deepseek-r1-32b |\n| ------ | ------ | ------ |\n| FS | 0.56 | 0.70 |\n| FSC | 0.54 | 0.66 |\n| AES | 0.84 | 0.84 |\n| RDS | 0.76 | 0.84 |\n| ADA | 0.90 | 0.90 |\n| RDES/B | 0.44 | 0.57 |\n| RDES/C | 0.51 | 0.52 |\n| RDES/PPO | 0.84 | 0.84 |\n\n### **BigBenchHard - boolean expressions**\n| Methods | Qwen25-72b | Deepseek-r1-32b |\n| ------ | ------ | ------ |\n| FS | 0.98 | 0.38 |\n| FSC | 0.60 | 0.46 |\n| AES | 0.53 | 0.60 |\n| RDS | 0.53 | 0.60 |\n| ADA | 0.53 | 0.60 |\n| RDES/B | 0.76 | 1.00 |\n| RDES/C | 0.90 | 0.99 |\n| RDES/PPO | 1.00 | 1.00 |\n\n### **BigBenchHard - web of lie**\n| Methods | Qwen25-72b | Deepseek-r1-32b |\n| ------ | ------ | ------ |\n| FS | 0.58 | 0.98 |\n| FSC | 1.00 | 1.00 |\n| AES | 0.85 | 0.72 |\n| RDS | 0.89 | 0.68 |\n| ADA | 0.83 | 0.72 |\n| RDES/B | 0.50 | 0.93 |\n| RDES/C | 0.98 | 1.00 |\n| RDES/PPO | 1.00 | 0.90 |\n\n### **GSM-8K**\n| Methods | Qwen25-72b | Deepseek-r1-32b |\n| ------ | ------ | ------ |\n| FS | 0.50 | 0.28 |\n| FSC | 0.56 | 0.64 |\n| AES | 0.92 | 0.08 |\n| RDS | 0.90 | 0.48 |\n| ADA | 0.98 | 0.36 |\n| RDES/B | 0.87 | 0.37 |\n| RDES/C | 0.92 | 0.73 |\n| RDES/PPO | 0.94 | 0.48 |\n\nThank you once again for your insightful comments.\n\nBest regards,\n\nAuthors of Submission 1061", "claims_and_evidence": "The claims made in the submission are supported by extensive comparison to other prompt engineering and demonstration selection methods, across a variety of language models and datasets.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria, including the reinforcement learning framework and diverse benchmark datasets, are appropriate for addressing the challenges in in-context learning for text classification. These choices provide a relevant and practical framework for assessing model performance and robustness.", "theoretical_claims": "Did not check correctness.", "experimental_designs_or_analyses": "Experimental design is sound and straightforward. The author compared against a wide variety of methods and datasets.", "relation_to_broader_scientific_literature": "The paper extends prior work on in-context learning and demonstration selection by using reinforcement learning to balance relevance and diversity, building on ideas from chain-of-thought prompting, clustering, and DPP-based methods. Its contributions complement recent efforts like RetICL and adaptive demonstration selection with theoretical guarantees and extensive empirical evaluation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n- Extensive comparison to other methods across a variety of datasets.\n- Clear explanation of method.\n\nWeaknesses:\n\n- Lack of units in tables.\n- No user studies, which often serve as better metrics of LLM quality.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "None for now", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QaZ3CkrcXG", "reviewer": "Reviewer_5sFY", "summary": "This paper claims that applying a Q-learning method to demonstration selection can be beneficial for classification tasks in the context of in-context learning.\n\nThe authors attempt to frame demonstration selection as a Markov decision process, though the details of this formulation remain unclear in the draft.\n\nAlgorithm 1 in the paper presents the final strategy: first, the top-k samples are selected based on the cosine similarity of candidate demonstrations. Then, the remaining samples are chosen to enhance the diversity of the selected demonstration set until the diversity of the final set exceeds a predefined threshold.\n\nWhile the paper asserts that demonstration selection is important for reasoning tasks, the evaluation is conducted solely on simple classification tasks, where the proposed method is claimed to outperform other baselines.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 5sFY,\n\nThank you for your thorough review and valuable feedback. We appreciate your concerns and have provided detailed responses below, referencing other reviewers' comments to support our points.\n\n### Claims and Evidence\nYou noted the lack of evidence for the usefulness of reinforcement learning in demonstration selection. We highlight the following points, supported by other reviewers:\n\n- **Adaptive Selection Strategy**: Unlike traditional methods, RDES uses reinforcement learning to dynamically adjust selection policies based on current inputs and previously selected demonstrations. Reviewer ZBB1 noted that RDES \"facilitates adaptive demonstration selection.\"\n  \n- **Balancing Relevance and Diversity**: RDES employs Q-learning to balance the selection of relevant and diverse demonstrations. The reward function considers both classification accuracy and diversity. Reviewer 3ANe mentioned that our approach \"balances relevance and diversity.\"\n\n- **Sequential Decision Process**: We model demonstration selection as a Markov Decision Process (MDP), where each selection is an action. Q-learning helps maximize future rewards. Reviewer NHQh stated that our method optimally picks examples based on similarity and diversity metrics.\n\n- **Improved Experimental Results**: RDES significantly outperforms traditional methods across multiple datasets, demonstrating the effectiveness of our approach. Reviewer ZBB1 concluded that \"RDES significantly boosts classification accuracy.\"\n\n### Methods and Evaluation Criteria\nYou expressed uncertainty about Algorithm 1's relationship with Q-learning. We clarify:\n\n- **Algorithm 1** is used during inference to quickly select demonstration examples based on TF-IDF similarity and a diversity threshold.\n  \n- **Q-learning** is applied during training, where the agent interacts with the environment to receive rewards based on accuracy and diversity changes. Reviewer ZBB1 described the RL agent's iterative selection process.\n\n- The diversity threshold can be a fixed hyperparameter or learned from the Q-learning model. Algorithm 1 improves efficiency while ensuring diversity.\n\nRegarding the simplicity of evaluation tasks:\n\n- Our initial focus was on demonstrating RDES's effectiveness in text classification. We have now supplemented our results with reasoning-oriented benchmarks like BigBenchHard and GSM-8K. **Due to character limitations, please refer to the response to Reviewer 3ANe for the corresponding experimental results.**\n\n### Theoretical Claims\nWe introduced theorems to support RDES's effectiveness:\n\n- **Lemma 3.1** establishes bounds for our diversity metric. Reviewer ZBB1 noted its importance.\n  \n- **Theorem 3.2** shows Q-learning's convergence under certain conditions, ensuring policy reliability. Reviewer ZBB1 confirmed this.\n\n- **Theorem 3.3** explains that increased diversity can enhance classification accuracy, justifying our emphasis on diversity.\n\n### Experimental Designs or Analyses\nWe appreciate your suggestion for more analysis:\n\n- An **ablation study** in Appendix A.6 analyzes the impact of different diversity mechanisms, confirming that diversity improves performance. Reviewer ZBB1 highlighted this.\n\n- We have added results on reasoning benchmarks and will further analyze demonstration characteristics and compare RL algorithms in future work.\n\n### Supplementary Material\nThe supplementary materials are closely related to the main paper:\n\n- **Appendix A.1** includes theoretical proofs, supporting key claims. Reviewer ZBB1 stated this inclusion.\n\n- **Appendix A.3** details baseline methods, crucial for understanding our distinctions. Reviewer ZBB1 noted this clarity.\n  \n- **Appendix A.4** lists LLMs used, ensuring transparency. Reviewer ZBB1 acknowledged this.\n\n- **Appendix A.5** provides implementation details for reproducibility. Reviewer ZBB1 confirmed this.\n\n- **Appendix A.6** contains the ablation study, verifying diversity effectiveness.\n \n### Essential References Not Discussed\nWe will cite and discuss the work \"Diverse Demonstrations Improve In-context Compositional Generalization (ACL 2023)\" in future revisions, as it aligns with our core idea of using diversity to enhance generalization.\n\n### Other Strengths and Weaknesses\nSections 3.2 and 3.3 serve different purposes: 3.2 introduces theoretical foundations, while 3.3 describes the implementation process. The former supports the latter.\n\n### Other Comments or Suggestions\nWe will ensure acronyms are defined before use in future revisions.\n\n### Questions for Authors\nWe chose Q-learning for its maturity, suitability for discrete action spaces, and clear convergence theory. We have also experimented with combining RDES with PPO, achieving competitive results. \n\nWe hope these explanations address your concerns. We will consider your suggestions for improvements in future revisions. Thank you again for your feedback.\n\nBest regards,\n\nAuthors of Submission 1061", "claims_and_evidence": "- Claim: Reinforcement learning, especially Q-learning, is useful for demonstration selection.\n- Evidence: To the best of my knowledge, I cannot find evidence in the draft on why RL is helpful for demonstration selection. Sections 3.1 and 3.2 state very general notions of Q-learning and related theorems and readers can hardly obtain enough information on why RL is specifically helpful for demonstration selection, compared to previous existing methods.", "methods_and_evaluation_criteria": "- Algorithm 1 is very simple and easy to understand; however, it is a well-known technique in the literature. More importantly, I am unsure of the relationship between Algorithm 1 and Q-learning. The algorithm merely involves computing cosine similarities using TF-IDF vectors, which is a well-established and basic approach. To the best of my knowledge, I cannot confirm that Algorithm 1 is directly related to Q-learning or any RL-based methods, especially based on the explanations provided in the paper.\n- While the authors claim to evaluate the reasoning performance of LLMs, the actual tasks used for evaluation are merely simple classification tasks rather than reasoning benchmarks such as GSM8K.\nTherefore, I am uncertain whether the evaluation results presented in the paper adequately support the authors’ claims.", "theoretical_claims": "The draft includes several theorems, but it is unclear how they relate to the effectiveness of the proposed method. There is not enough clear explanation of why the authors introduced these theorems.", "experimental_designs_or_analyses": "- While the authors claim to evaluate the reasoning performance of LLMs, the actual tasks used for evaluation are merely simple classification tasks rather than reasoning benchmarks such as GSM8K.\nTherefore, I am uncertain whether the evaluation results presented in the paper adequately support the authors’ claims.\n- More analysis or ablation studies are encouraged to probe what is the strengths and weaknesses of the proposed method.", "relation_to_broader_scientific_literature": "This paper explores a new approach to demonstration selection for in-context learning, a topic that has been widely studied in recent years.\nIf the proposed method indeed offers a meaningful improvement in selecting better demonstrations, it could have a significant impact. However, I am uncertain whether the method is correctly specified and whether it introduces genuine novelty.", "essential_references_not_discussed": "Not mandatory but related: Diverse Demonstrations Improve In-context Compositional Generalization (ACL 2023)", "other_strengths_and_weaknesses": "Weaknesses\n- Basically, I'm uncertain about the current writing status of the paper. I'm sorry to say that, but personally, I feel like this draft is not yet in the stage of publication or being reviewed. \n- Each component and section in the paper provides independent concepts which are not well-aligned. For instance, what is the exact relationship between Sections 3.2 and 3.3?", "other_comments_or_suggestions": "Before using acronyms, please provide their full names first. For instance, before introducing RDES in the Introduction, define it explicitly (although it is mentioned in the Abstract, that alone is not sufficient).", "questions_for_authors": "Why must we particularly rely on Q-learning-like methods instead of other reinforcement learning approaches, such as PPO and others?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J0AAUidZim", "reviewer": "Reviewer_NHQh", "summary": "The paper proposes a smarter way to select examples/demonstrations for In-Context Learning in LLMs. Picking the right example set for the task at hand is a serious challenge and authors propose an RL-based technique (with Q-learning) to optimally pick examples from a golden set based on a similarity metric + a diversity metric. “RDES”.\n\nThey’ve attempted to establish that the model should converge and that higher diversity in the examples should lead to higher accuracy via statistical theorems. To demonstrate the effectiveness of the method in classification tasks, they test it with 4 different LLMs on 4 different datasets. They’ve shown superior performance against a large number of different selection & prompt engineering methods - with & without COT.\n\nAs for the algorithm, they start with similar examples from TF-IDF representations & then augment the set with more diverse data points.\n\n## update after rebuttal\n1. The authors did, address this to some degree by sharing metrics for more tasks in their rebuttal. Inclusion of SST ask from GLUE helps make their case.\n2. The authors did, address this to some degree by sharing metrics for more tasks in their rebuttal. Inclusion of SST ask from GLUE helps make their case", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer NHQh,\n\nThank you for your recognition of our work and the valuable suggestions.\n- **Work as an iteration of existing work**: We acknowledge that RDES builds upon existing work on RL for ICL and diverse demonstration selection. However, we emphasize that the novelty of RDES lies in **the combination of a reinforcement learning framework (both Q-learning and PPO) with an explicit label distribution-based diversity metric to achieve dynamic demonstration selection that balances relevance and diversity.** We believe this combination is significant for improving few-shot learning performance.\n- **Suggestion to test on GLUE tasks**: Thank you for the suggestion. Our current work focuses on evaluating the performance of RDES on intent classification tasks, which are common benchmarks for assessing the effectiveness of demonstration selection strategies. We have included results on **SST5 (sentiment analysis, part of GLUE) in our additional results. RDES/PPO achieved competitive results on SST5 (e.g., for both Qwen25-72b and Deepseek-r1-32b, AES and RDES/PPO achieved an accuracy of 0.84).** As also noted by Reviewer 5sFY regarding the simplicity of evaluation tasks, we have now included results on more complex reasoning benchmarks like GSM-8K and BigBenchHard, further demonstrating the applicability of RDES beyond basic classification. Future work can further explore the applicability of RDES to a broader range of GLUE tasks. \n\nMoreover, here are the results on SST5, BigBenchHard (boolean expressions, web of lie), and GSM-8K using Qwen25-72b and Deepseek-r1-32b models; The RDES series is our proposed method, where RDES/B and RDES/C are based on Q-learning, and RDES/PPO is based on the PPO method:\n### **SST5**\n| Methods | Qwen25-72b | Deepseek-r1-32b |\n| ------ | ------ | ------ |\n| FS | 0.56 | 0.70 |\n| FSC | 0.54 | 0.66 |\n| AES | 0.84 | 0.84 |\n| RDS | 0.76 | 0.84 |\n| ADA | 0.90 | 0.90 |\n| RDES/B | 0.44 | 0.57 |\n| RDES/C | 0.51 | 0.52 |\n| RDES/PPO | 0.84 | 0.84 |\n\n### **BigBenchHard - boolean expressions**\n| Methods | Qwen25-72b | Deepseek-r1-32b |\n| ------ | ------ | ------ |\n| FS | 0.98 | 0.38 |\n| FSC | 0.60 | 0.46 |\n| AES | 0.53 | 0.60 |\n| RDS | 0.53 | 0.60 |\n| ADA | 0.53 | 0.60 |\n| RDES/B | 0.76 | 1.00 |\n| RDES/C | 0.90 | 0.99 |\n| RDES/PPO | 1.00 | 1.00 |\n\n### **BigBenchHard - web of lie**\n| Methods | Qwen25-72b | Deepseek-r1-32b |\n| ------ | ------ | ------ |\n| FS | 0.58 | 0.98 |\n| FSC | 1.00 | 1.00 |\n| AES | 0.85 | 0.72 |\n| RDS | 0.89 | 0.68 |\n| ADA | 0.83 | 0.72 |\n| RDES/B | 0.50 | 0.93 |\n| RDES/C | 0.98 | 1.00 |\n| RDES/PPO | 1.00 | 0.90 |\n\n### **GSM-8K**\n| Methods | Qwen25-72b | Deepseek-r1-32b |\n| ------ | ------ | ------ |\n| FS | 0.50 | 0.28 |\n| FSC | 0.56 | 0.64 |\n| AES | 0.92 | 0.08 |\n| RDS | 0.90 | 0.48 |\n| ADA | 0.98 | 0.36 |\n| RDES/B | 0.87 | 0.37 |\n| RDES/C | 0.92 | 0.73 |\n| RDES/PPO | 0.94 | 0.48 |\n\nThank you again for your insightful comments.\n\nBest regards,\n\nAuthors of Submission 1061", "claims_and_evidence": "The paper claims that for classification tasks using LLMs, their technique (adding a diversity metric to an RL selection process ) yields better results than a lot of existing methods. They have backed their claims with trials on a large number of techniques in ICL & prompt-tuning  on a variety of LLMs.\n\nI believe that the claims are well supported by their evidence in numbers.", "methods_and_evaluation_criteria": "Yes - their evaluation strategy for text classification tasks looks good. They could have, however, also shown some results form GLUE-based tasks like SST2 for easier comparison to models like BERT, etc\n\nEDIT (post rebuttal): The authors did, however address this to some degree by sharing metrics for more tasks in their rebuttal. Inclusion of SST ask from GLUE helps make their case.", "theoretical_claims": "Their theoretical proofs around (1) convergence and (2) usefulness of diversity in accuracy looks ok.\nHowever, I did not do a very rigorous analysis", "experimental_designs_or_analyses": "Checked the experimental setup for method comparison to baseline. Looks sound\nShould have also run tests on more widely used benchmarks like SST, etc\n\nEDIT (post rebuttal): The authors did, however address this to some degree by sharing metrics for more tasks in their rebuttal. Inclusion of SST ask from GLUE helps make their case.", "relation_to_broader_scientific_literature": "Other very similar papers:\nRetICL: https://arxiv.org/pdf/2305.14502\nActive Example Selection for In-Context Learning (Zhang et al): https://arxiv.org/pdf/2211.04486 \n\nOther unacknowledged, but very similar publications: \n1. ICL-D3IE (https://arxiv.org/pdf/2303.05063)\n2. Diverse Demonstrations Improve In-context Compositional Generalization (https://aclanthology.org/2023.acl-long.78/)\n\n\nThe work is a small iteration on top of some existing works. Eg: “Active Example Selection for In-Context Learning (Zhang et al) and “RetICL (Scarlatos et al.)”\nThe idea of using diverse examples for ICL has been explored by lot of recent works. Similarly, using RL techniques to perfect ICL has also been well-researched and published. I’m however not aware of any work on using Q-learning along with diversity metric in example selection for ICL.", "essential_references_not_discussed": "1. ICL-D3IE (https://arxiv.org/pdf/2303.05063)\n2. Diverse Demonstrations Improve In-context Compositional Generalization (https://aclanthology.org/2023.acl-long.78/)", "other_strengths_and_weaknesses": "Strengths:\n\nPaper has clear presentation and context on the problem being solved, along with good details of existing work on the topic. Their idea of adding diversity metric to the prompt-generation process is novel. Their proposal is also simple to add on top of existing infrastructure for RAG since it only changes the demonstration selection process in the prompt generation pipeline.\n\nThey have made good effort to exhaustively test the methodology on a wide array of tasks and with a lot of different LLMs. I believe the paper should be accepted as it is an iterative upgrade to existing RL-based techniques to improve context creation for ICL.\n\n\nWeaknesses:\n\nThe work is a small iteration on top of some existing works. Eg: “Active Example Selection for In-Context Learning (Zhang et al) and “RetICL (Scarlatos et al.)”\nThe idea of using diverse examples for ICL has been explored by lot of recent works. Similarly, using RL techniques to perfect ICL has also been well-researched and published. I’m however not aware of any work on using Q-learning along with diversity metric in example selection for ICL.\nThe overall study will also benefit a lot from more research on generation tasks, instead of just classification. Most other recent works do try to show impact on generation tasks.\nOn that note, it would be useful to see how much encoder-type models woud benefit from the added context (since the authors are sticking to classification-only tasks, in the real world, decoder-only models are not always the most efficient ways to do classification)\nThe authors also seem to have missed some notable references in the paper (more details in questions to author)", "other_comments_or_suggestions": "1, Figure 1 needs to be clearer in the intention. The \"Demonstrations\" boxes in the figure look like they're being LLM-generated. IIUC, the intent is to show how different example types can be based on selection strategy. The \"LLM\" logo is confusing in this scenario.\n\n2. Equation 1: needs an explanation of the \"r_t\" variable used in the expression\n\n3.  (Nitpick) Equation 2 can be labelled as Bellman's equation in the preceding text", "questions_for_authors": "1. Can you please shed light on why you chose only classification tasks for the analysis? If generative datasets & baselines were studied, do you know this compares to RetICL & other techniques on MWP & other problems?\nEDIT (post rebuttal): The authors did, however addressed this to some degree by sharing metrics for more tasks in their rebuttal. Inclusion of SST ask from GLUE helps make their case.\n\n2. Points from \"Other Comments\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "wDoMGNDFA6", "reviewer": "Reviewer_ZBB1", "summary": "The paper introduces Relevance-Diversity Enhanced Selection (RDES), a reinforcement learning framework for selecting in-context demonstrations that balance relevance and diversity for few-shot text classification tasks. The core idea is to use a Q-learning agent to iteratively pick examples from a knowledge base such that the selected set covers a broad range of labels (high diversity) while remaining relevant to the input query  . By defining a diversity score (proportion of unique labels in the chosen demos) and including it in the reward function, RDES biases the selection toward a label-balanced set of demonstrations, aiming to improve the model’s generalization. The approach is evaluated on four intent classification benchmarks (BANKING77, CLINC150, HWU64, and a multi-domain “Liu54” dataset ) using 12 different Large Language Models (LLMs) (both closed-source like GPT-3.5-turbo and open-source models from LLaMA, Qwen, etc.  ). Experimental results show that RDES significantly boosts classification accuracy, outperforming ten baseline methods across all datasets. The authors also propose a variant, RDES/C, which incorporates Chain-of-Thought (CoT) reasoning during prompting, further enhancing predictive performance in most cases. In summary, RDES demonstrates the potential of using reinforcement learning as a post-training strategy to adapt an LLM’s in-context demonstration selection policy, achieving higher accuracy than existing prompt-engineering and demo selection techniques.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ZBB1,\n\nThank you for your insightful feedback.\n\n**On the analysis of RDES's computational cost (LLM invocation frequency)**: We appreciate your attention to this important aspect. While our current submission primarily focuses on the effectiveness of RDES, we recognize that computational cost is a critical factor in practical applications. In the revised version, we will include a discussion on the number of LLM invocations, specifically the inference counts on test samples during the training process of RDES, and explore potential optimization strategies.\n\n**Regarding the potential misleading nature of average results across different model scales**: Thank you for your reminder. We understand that performance differences among models of varying scales can be significant. Therefore, in our paper, we not only provide average results across models for macro comparison but also include detailed performance metrics for each model under different methods in the tables. This allows readers to conduct a more nuanced analysis of the performance of models at different scales.\n\n**On the suggestion of using semantic embedding retrieval to improve demonstration selection**: We appreciate your valuable suggestion. We agree that retrieval methods based on semantic embeddings (e.g., Sentence-BERT or embeddings generated by the LLM itself) may better capture semantic similarity, thereby enhancing the quality of demonstration example selection. In our current work, we utilized TF-IDF for initial retrieval, which is a simple and widely used method. In future work, we will actively explore and evaluate the potential of semantic embedding retrieval methods within the RDES framework.\n\n**Regarding the generalizability of RDES beyond classification tasks**: Thank you for raising this profound question. Extending the ideas of RDES to tasks beyond classification (such as generation or structured prediction) is an exciting direction for future research. As you suggested, we need to rethink and redefine the concept of \"diversity\" in these new tasks. For instance, in open-ended question answering, diversity may refer to covering different aspects or types of answers. We will mention this potential extension in the conclusion and future work section.\n\n**On the manual switching between RDES/B and RDES/C and the potential for RL strategies to learn when to invoke CoT**: This is an insightful suggestion! We fully agree that incorporating the use of CoT into the learning process of the RL strategy (e.g., treating \"whether to use CoT\" as an action) is a promising direction. This would make RDES more adaptive and could potentially address the slight performance drop observed in certain cases with RDES/C. We will actively explore this idea in future work.\n\n**Regarding attempts at semantic retrieval**: As mentioned earlier, our current work primarily employs TF-IDF for initial retrieval. In the future, we will actively experiment with and evaluate semantic embedding retrieval methods.\n\n**On whether the learned strategies are specific to each dataset or can generalize**: This is indeed a fascinating and important research question. Currently, our experiments involve training the RDES strategy independently on each dataset. In future work, we will explore the generalization capabilities of strategies across datasets, such as whether a strategy trained on one dataset can be fine-tuned or directly applied to another. This will help assess the robustness and generalizability of RDES.\n\n**Regarding the specific use of CoT in RDES/C**: In RDES/C, we employ the standard CoT prompting method, which involves adding a fixed prompt before the input text and selected demonstration examples, such as \"Let's think step by step and give your explanation to verify the answer.\" This prompts the LLM to generate a reasoning chain and the final answer.\n\nThank you once again for your valuable comments.\n\nBest regards,\n\nAuthors of Submission 1061", "claims_and_evidence": "Claim 1: Diversifying the demonstrations improves model generalization. The paper argues that “diversity in demonstration selection is crucial for enhancing model generalization” . This claim is supported by both theoretical and empirical evidence. The authors formalize a diversity metric (fraction of unique labels in the demo set) and prove that maximizing this diversity can improve accuracy (Theorem 3.3) . Empirically, an ablation study compares RDES to a “No-Diversity” variant and finds that adding the diversity mechanism consistently boosts accuracy on all evaluated datasets. For example, in the BANKING77 intent dataset, using diverse demos yields 0.838 accuracy vs. 0.600 without diversity, a large improvement. Figure 1 from the paper illustrates this effect with a sentiment classification example: when all demonstrations express positive sentiment, the model misclassifies a nuanced input as “Positive,” whereas a diverse set of positive, negative, and neutral examples leads to the correct “Neutral” label. This concrete example and the across-the-board gains in the ablation study provide convincing evidence for the claim that diverse demonstrations lead to better generalization.\n\nClaim 2: The RDES method outperforms existing prompt engineering and demo selection baselines. RDES is claimed to “significantly enhance classification accuracy compared to ten established baselines”. The paper backs this by evaluating on four benchmarks against two categories of baselines: (a) prompt-engineering methods (Zero-Shot, Knowledge Prompting, Least-to-Most, Chain-of-Thought, Self-Refine) and (b) demonstration selection methods (Few-Shot with/without CoT, Active Demo Selection, Representative Demo Selection, Adaptive ICL). In the results tables, RDES (denoted RDES/B for base and RDES/C with CoT) achieves the highest accuracy on nearly all dataset-model combinations. For instance, on CLINC150, RDES/C reaches 90.2% average accuracy versus the best baseline (ADA) at 78.0%   – an absolute gain of ~12%, which is substantial. Similar margins are seen in other datasets and with both closed models (GPT-3.5, Doubao, etc.) and open models (LLaMA, Qwen variants)  . The paper’s summary of Table 1 notes that “RDES/B and RDES/C consistently outperform alternative methodologies across the evaluated datasets”. This broad performance lead, along with statistical averages over 5 runs to ensure reliability, provide evidence for the claim that RDES is state-of-the-art in in-context demonstration selection.\nClaim 3: Integrating Chain-of-Thought reasoning (RDES/C) further improves performance. The authors claim that adding CoT reasoning to RDES “further enhances the model’s predictive performance”. Evidence is shown by comparing RDES/C vs the base RDES/B. In most cases, RDES/C achieves the top accuracy (often a few points higher than RDES/B) on the benchmarks . For example, on the BANKING77 dataset with GPT-3.5-turbo, RDES/C achieves 85.8% vs 76.7% for RDES/B . The paper explains that CoT prompting allows the model to reason step-by-step, which, when combined with optimal demo selection, leads to more accurate answers . However, the evidence also reveals an interesting nuance: on certain settings with smaller open-source models, RDES/B slightly outperformed RDES/C (e.g., CLINC150 with open models saw RDES/B 0.800 vs RDES/C 0.731)  . The authors acknowledge this variation, suggesting that CoT’s benefit may depend on model capacity and dataset characteristics . Overall, the claim is mostly supported – CoT generally provides a boost, especially for powerful models and complex tasks, though it’s not uniformly beneficial in every scenario.\nClaim 4: A reinforcement learning policy can adaptively select demonstrations, offering an advantage over static methods. By framing demo selection as an MDP, the authors claim RDES “facilitates adaptive demonstration selection” and can adjust to classification challenges dynamically. This is implicitly evidenced by RDES outperforming baselines like Representative Demo Selection (RDS)  and Adaptive ICL (ADA) , which themselves focus on diversity or uncertainty but without a learning agent. The RL agent’s adaptivity is further highlighted by the ability to handle different model types: RDES worked well across both closed and open LLMs, adjusting its selections even when the underlying model’s behavior differed. While the claim of “deepening the understanding of classification challenges” is somewhat broad, the paper does show that analyzing the learned policy and its outcomes yields insights. In summary, the evidence supports the adaptive advantage of the RL approach: RDES learns a policy that generalizes across tasks and models better than static selection methods. Any lofty suggestion that it “deepens understanding” of all classification challenges is not directly proven, but the work does encourage thinking of demo selection as a learning problem, which is a valuable perspective.", "methods_and_evaluation_criteria": "Methods & Evaluation Summary (Shortened)\nThe proposed Relevance-Diversity Enhanced Selection (RDES) method frames demonstration selection as a Markov Decision Process (MDP), where an agent uses Q-learning to iteratively select examples from a knowledge base (KB), balancing relevance and diversity. The reward function encourages both correct classification (+1) and diversity gains. To manage the large state space, the model focuses on summary statistics (e.g., diversity score) rather than raw text. An ε-annealing strategy ensures a balance between exploration and exploitation.\n\nAt inference time, RDES first retrieves top-k most similar candidates using TF-IDF cosine similarity, then applies the learned RL policy (or a heuristic threshold) to ensure diversity. If the selected examples are too homogeneous, additional demos with new labels are added until the diversity requirement is met. The RDES/C variant incorporates Chain-of-Thought (CoT) prompting, prompting the LLM to generate a step-by-step reasoning chain before predicting the final answer.\n\nEvaluation Criteria & Benchmarks\nRDES is tested on four benchmark datasets spanning different domains: BANKING77, HWU64, CLINC150, and LIU54, ensuring comparability with prior few-shot intent classification research. A “challenge set” of hard-to-classify queries is used to stress-test model performance. The evaluation includes 12 LLMs (4 closed-source like GPT-3.5-turbo, Doubao, Hunyuan and 8 open-source LLaMA, Qwen variants from 1B to 72B parameters), demonstrating robustness across different model architectures.\n\nThe method is compared against 10 baselines:\n\nPrompt engineering methods (ZS, KP, L2M, CoT, SF) focus on structuring the prompt.\nDemonstration selection methods (FS, FSC, AES, RDS, ADA) optimize example selection.\nNotably, AES and ADA also involve learning-based selection, making them strong baselines. RDES consistently outperforms all alternatives. The authors average results over 5 runs, ensuring statistical reliability. One limitation is that the cost of RDES (LLM calls during training) is not explicitly analyzed, leaving sample efficiency unaddressed. However, the extensive experiments (multiple models, datasets, baselines) make the evaluation thorough, well-aligned, and convincing.", "theoretical_claims": "The paper presents several theoretical contributions supporting RDES, all of which appear logically sound.  \n\nFirst, Lemma 3.1 (Diversity Bounds) establishes fundamental properties of the diversity metric. It states that for a given demonstration set, diversity is measured as the fraction of unique labels within the set, and this value is always constrained between a minimum (when all examples share the same label) and a maximum (when every example belongs to a different label). This is a straightforward yet useful observation, reinforcing the idea that increasing label diversity leads to broader contextual coverage. Since the claim follows directly from how diversity is defined, the proof is simple and based on basic counting arguments, making it undisputed.  \n\nA more significant result is Theorem 3.2 (Q-Learning Convergence), which states that the tabular Q-learning algorithm used in RDES is guaranteed to converge to an optimal policy, provided that standard reinforcement learning conditions are met. Specifically, if the learning rate follows a diminishing schedule and the reward values remain within a bounded range, the Q-learning process will stabilize over time. This claim aligns with well-established reinforcement learning theory, particularly the classic Watkins & Dayan (1992) result on Q-learning convergence. Given that the state and action spaces in RDES are finite—since the knowledge base (KB) has a limited set of candidate demonstrations and the selection process involves discrete choices—the theoretical conditions for convergence should hold. While practical implementations might not strictly follow all assumptions (e.g., if a fixed learning rate is used instead of a decaying one), the theoretical foundation is robust and aligns with established reinforcement learning principles.  \n\nThe final theoretical result, Theorem 3.3 (Diversity-Accuracy Dominance), provides a justification for why diversity enhances model accuracy. It states that if one demonstration set is more diverse than another, there exists a threshold beyond which increasing diversity leads to a measurable improvement in classification accuracy. The extent of this improvement depends on the number of unique labels present, with a diminishing returns effect—the more diverse the set, the smaller the additional accuracy gains per new label. The proof, detailed in the appendix, likely relies on probability bounds or coverage arguments to demonstrate how additional diversity reduces classification errors. While the exact mathematical formulation is omitted in the main text, the result logically aligns with empirical observations: ablation studies in the paper confirm that increasing diversity consistently improves accuracy.  \n\nThese theoretical claims collectively reinforce the validity of RDES. None of them appear to be incorrect or exaggerated:  \nLemma 3.1 is straightforward and follows from definition.  \nTheorem 3.2 is a direct application of established Q-learning convergence results.  \nTheorem 3.3 introduces a novel insight connecting diversity to classification accuracy, which aligns well with the experimental findings.  \n\nOne potential critique is that Theorem 3.3 presents its conclusion in an asymptotic form, meaning it assumes a sufficiently large number of labels before its effects become prominent. However, the core idea—that increasing diversity generally enhances accuracy—remains intuitively and empirically valid.  \n\nOverall, the theoretical analysis is well-grounded and adds credibility to the paper. The authors correctly apply reinforcement learning principles and provide a meaningful contribution by formally linking diversity to model performance, strengthening the case for using RL-based adaptive demonstration selection in LLMs.", "experimental_designs_or_analyses": "The experimental design is well-executed, ensuring fairness and statistical reliability. The authors evaluate RDES across multiple datasets and models, comparing it against ten baselines (five prompt engineering, five demo selection methods). Key questions addressed include whether RDES outperforms others, the impact of Chain-of-Thought (CoT), and the importance of diversity.\n\nControls & Comparisons\nBy comparing RDES with Active Demonstration Selection (AES) and Representative Demo Selection (RDS), the study isolates the benefits of reinforcement learning and diversity selection. Results consistently favor RDES/RDES-C, reinforcing its advantage. Including CoT prompting as both a baseline and a variant (RDES/C) helps analyze its added value—while CoT alone isn’t always best, RDES + CoT is usually the top performer, validating the combined approach.\n\nStatistical Rigor & Experimental Setup\nThe authors average results over five runs, mitigating variance from random demo selection. Although statistical significance tests aren’t provided, large accuracy gains (5-15 percentage points) strongly support RDES’s superiority. The setup is well-documented, detailing hyperparameters like learning rate (0.1), discount factor (0.9), and an epsilon-greedy policy. All methods use the same number of demonstrations (5 per prompt), ensuring fair comparisons. The dataset’s challenge set selection effectively isolates hard-to-classify cases, making performance gains more meaningful.\n\nAblation Study & Analysis\nAn ablation study (Appendix A.6) compares No-Diversity, RDES/B, and RDES/C, confirming that adding diversity improves accuracy, and CoT further enhances it. However, in one case (CLINC150, open-source models), RDES/B slightly outperforms RDES/C, suggesting that dataset/model characteristics influence CoT effectiveness. This nuanced analysis demonstrates a rigorous experimental approach.\n\nPotential Improvements\nWhile the experiments are robust, a few limitations remain:\n\nComputational cost: Training RDES via repeated LLM queries (especially with GPT-3.5) is potentially expensive, yet the paper does not discuss efficiency or required queries.\nModel averaging: Aggregating accuracy across models of vastly different sizes (8B vs. 70B parameters) might be misleading, though per-model breakdowns are provided.\nTF-IDF retrieval: While effective, semantic embedding-based retrieval (e.g., Sentence-BERT) could further improve demo selection.", "relation_to_broader_scientific_literature": "This work sits at the intersection of in-context learning, demonstration selection, and RL-based post-training for LLMs, offering a notable improvement over prior methods.\n\nRL-Based Demonstration Selection: RDES builds on previous RL approaches like RetICL (Scarlatos & Lan, 2023) and Zhang et al., 2022, which optimized demo selection via reinforcement learning. However, these methods mainly focused on relevance or uncertainty, while RDES introduces a diversity-driven reward, ensuring varied label coverage. This avoids diminishing returns from selecting similar examples and leads to significantly better results, as shown by RDES outperforming AES (Active Demonstration Selection).\n\nNon-RL Demonstration Selection: Many prior works use embedding retrieval, clustering (DPP), or heuristics (BERTScore-Recall, skill coverage) to select examples. Representative Demo Selection (RDS, Yang et al., 2023) also prioritizes diversity but does so statically, while RDES learns an adaptive policy per query. Similarly, Adaptive ICL (ADA, Mavromatis et al., 2023) selects based on uncertainty and diversity without training a policy—RDES outperforms it, showing that an RL-driven approach can better optimize selection.\n\nAdvancement Over Prior Work: While diversity-based selection, RL for demo selection, and CoT prompting have been studied individually, RDES uniquely combines all three. Unlike prior work that focused on smaller models (e.g., GPT-2), RDES successfully operates on large-scale LLMs like GPT-3.5, proving its practical viability for real-world applications. It demonstrates that RL-based optimization of LLM inputs (without fine-tuning) is an effective alternative to full model retraining.\n\nComparison to RLHF: While Reinforcement Learning from Human Feedback (RLHF) fine-tunes model weights for alignment (Ouyang et al., 2022), RDES optimizes input selection while keeping the model fixed. This makes it a lighter-weight alternative for post-training improvements, aligning with a growing trend of treating LLM prompting as an RL-optimized decision process.\n\nRDES represents a meaningful advancement in RL-driven post-training for LLMs, outperforming previous methods while deepening the theoretical understanding of why diversity matters in in-context learning. Future research could explore new reward functions or adapt RL frameworks to reasoning tasks, further expanding on RDES’s contributions.", "essential_references_not_discussed": "The paper does a solid job covering the key references on demonstration selection and prompting, making it clear that the authors are well-versed in the field. That said, there are a few things they could have mentioned to round out the discussion.\n\nOne noticeable omission is Reinforcement Learning from Human Feedback (RLHF). Since RLHF is such a big deal in RL + LLM research, even a quick comparison between RDES and RLHF would have been helpful—especially for readers coming from the RL community. While RDES optimizes input selection rather than modifying model weights, making this distinction explicit would clarify where it fits within RL-based LLM tuning.\n\nAnother area that could have been included is Self-Consistency and Calibration in few-shot learning. Self-Consistency (Wang et al., 2022) has been a major improvement for Chain-of-Thought (CoT) reasoning, and given that RDES/C incorporates CoT, it would have been interesting to see how it relates. Meanwhile, Calibration (Zhao et al., 2021) focuses on correcting biases in demo selection, which overlaps with RDES’s effort to ensure diverse label coverage. Neither of these are core to RDES’s main idea, but acknowledging them would have made the paper’s positioning even stronger.\n\nThat said, these are minor gaps rather than major flaws. The authors clearly did their homework, and there’s no glaring omission that weakens the paper’s claims. Adding these references wouldn’t change the conclusions but would give a more complete picture of the broader landscape. Overall, the coverage is strong, and these missing pieces are more about fine-tuning the context than fixing any major oversight.", "other_strengths_and_weaknesses": "Strengths\nNovel Approach: RDES uniquely combines reinforcement learning, diversity-aware selection, and Chain-of-Thought prompting, creating a new way to optimize few-shot classification. The explicit accuracy-diversity tradeoff in its reward function addresses a key limitation in prior RL-based demo selection.\n\nStrong Evaluation: The paper thoroughly tests RDES across 4 datasets, 12 models, and 10 baselines, with theoretical guarantees and ablation studies confirming its effectiveness. The inclusion of both API-based and open-source models demonstrates its practical applicability.\n\nWeaknesses\nLimited to Classification: RDES relies on label diversity, making it hard to apply to open-ended tasks like generation or structured prediction without significant modifications.\n\nHigh Computational Cost: Q-learning requires repeated LLM queries, which may be expensive and slow, especially for API-based models. The paper doesn’t discuss training efficiency or scalability, which could limit real-world adoption.\n\nInconsistent Performance on Open Models: While RDES performs well overall, its gains on smaller open-source models are less consistent, and CoT sometimes reduces accuracy. The method doesn’t adaptively decide when to use CoT, requiring manual tuning based on the model.", "other_comments_or_suggestions": "Generality to Other Tasks: It would be interesting to see how RDES could be applied beyond classification. A suggestion for future work is to generalize the notion of \"diversity\" to other settings – for example, in open-ended question answering, diversity could mean covering different topical aspects or answer types in the demonstrations. While it’s understandable the paper focused on classification (where diversity is well-defined), exploring a generalized RDES could broaden its impact. Perhaps the authors could mention this as a potential extension.\n\nAdaptive CoT Usage: The results indicated that chain-of-thought helps with some models/datasets but not all. A suggestion is to make the CoT aspect part of the learning as well – e.g., an agent could decide whether or not to prompt the model with \"Let’s think step by step\" based on the state. Right now, RDES/B vs RDES/C was a manual switch. If the RL policy could also learn when to invoke CoT (maybe treat it as an action: \"ask model to think\" vs \"ask model directly\"), that might yield an even more adaptive system. This would prevent the slight performance drop seen for RDES/C on certain tasks by not using CoT when it’s unhelpful.", "questions_for_authors": "Have you considered how RDES might apply to tasks beyond classification?\n\nYou used TF-IDF for initial retrieval of candidate demos. Did you try any semantic retrieval (e.g., using embeddings from Sentence-BERT or from the LLM itself)? If so, did it make any difference? If not, do you anticipate any benefit from it? \n\nDid you notice if the learned policy for demonstration selection is specific to each dataset, or could a policy trained on one set of intents work (maybe with slight fine-tuning) on another?\n\nFor RDES/C, how exactly is CoT used – do you simply prepend a fixed prompt like “Let’s think step by step” and then have the model generate a reasoning chain and an answer (like the standard CoT approach )?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xzs2FRXIbw", "reviewer": "Reviewer_Sqdn", "summary": "This work studies the problem of demonstration selection for in-context learning in LLM. The problem is formulated as a reinforcement learning process, i.e., the effect of a combination of demonstration is similar to that of taking a sequence of actions. During the RL process, heuristic reward based on both accuracy and diversity is leveraged. Experiments are performed on both closed-source and open-source LLMs, comparing to multiple baselines on different datasets.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Sqdn,\n\nThank you for your insightful review and valuable feedback. We appreciate your concerns and have provided detailed responses below：\n\n### Theoretical Claims\n\n- We acknowledge your point that **Lemma 3.1 (Diversity Bounds) and Theorem 3.2 (Q-Learning Convergence) are relatively straightforward**. Their inclusion was intended to provide a complete theoretical foundation for our framework. Lemma 3.1 establishes the basic properties of our diversity metric, while Theorem 3.2 confirms the theoretical guarantee of convergence for the Q-learning algorithm under standard conditions.\n- Regarding **Theorem 3.3 (Diversity-Accuracy Dominance)**, we understand your concern about the vagueness of the proof in the main text and the use of Hoeffding's inequality. **A more detailed proof is provided in Appendix A.1.3**, which attempts to justify how increased diversity can lead to improved accuracy by reducing prediction bias and variance. Reviewer ZBB1 also found empirical evidence (ablation study results and the sentiment classification example in Figure 1) convincing in supporting the claim that diverse demonstrations lead to better generalization. We hope the detailed proof in the appendix clarifies our reasoning.\n\n### Experimental Designs and Analyses\n\n- We appreciate your acknowledgment of the **extensive nature of our experiments**, covering multiple datasets, models, and baselines. While you feel limited observations/findings are provided, we aimed to demonstrate the consistent improvements of RDES across a wide range of settings.\n- You suggested analyses on the number of demonstrations selected and their characteristics. In our implementation (Appendix A.5), **we used a fixed number of five demonstrations as references in the baseline to ensure fair comparisons across all methods**. RDES dynamically selects demonstrations during training based on learned Q-values and the diversity reward. Algorithm 1 shows how, during inference, if the diversity of the top-k similar examples is below a threshold, we iteratively add more diverse examples. The key characteristic of RDES is its ability to **balance relevance and diversity**. Reviewer ZBB1 highlighted this balance as a core contribution of RDES. The ablation study in Appendix A.6 further provides insights by comparing RDES with a \"No-Diversity\" baseline, demonstrating the impact of incorporating diversity.\n- Regarding whether Q-learning or other RL choices matter, our current work focuses on the effectiveness of a Q-learning framework for this problem. However, we have also designed a PPO-based version (RDES/PPO), which has shown competitive results (**due to character limitations, please refer to the response to Reviewer 3ANe**). Exploring the performance of different RL algorithms, including PPO, is a potential direction for future research, as suggested by Reviewer 5sFY.\n\n### Algorithmic Implementation\n\n- We understand your concern about the vagueness of the RL training part in Section 3.3 and Algorithm 1. **Algorithm 1 describes the demonstration selection process during inference**, where we retrieve top-k similar examples and enhance diversity as needed. The **RL training formulates demonstration selection as an MDP (Section 3.1.1) and uses Q-learning (Section 3.1.2)** to learn a policy maximizing expected rewards, including accuracy and diversity.\n\n### Purpose of the Diversity Term\n\n- You questioned the purpose of the diversity term in the reward, as the ultimate goal is accuracy. We included diversity in the reward because **we hypothesize and empirically demonstrate that diversity is crucial for enhancing model generalization, especially in few-shot learning scenarios**. By rewarding the selection of demonstrations with a more balanced label distribution, we encourage the RL agent to explore a broader coverage of the input space, mitigating the risk of overfitting to similar examples. Reviewer ZBB1 supports the claim that diversifying demonstrations improves model generalization.\n\n### Sample Complexity\n\n- We acknowledge that **the paper does not provide a detailed analysis of sample complexity**. Evaluating RDES's sample efficiency is important, and we plan to investigate this further.\n\n### Comparison with Zhang et al., 2022\n\n- Both our work and Zhang et al., 2022, take an RL perspective for demonstration selection. However, **a key difference lies in the reward function and the objective**. While Zhang et al., 2022, primarily focus on improving accuracy, **RDES explicitly aims to optimize both relevance and diversity by incorporating a diversity score into the reward function**. We believe this dual objective is crucial for robust performance in few-shot learning. Reviewer NHQh noted that our idea of adding a diversity metric to the prompt generation process is novel.\n \nThank you again for your valuable feedback. We hope this response clarifies your concerns.\n\nBest regards,\n\nAuthors of Submission 1061", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The method make sense, viewing the selection problem from a sequential perspective. The evaluation also follows the standard ICL one.", "theoretical_claims": "The theoretical results in this work are mostly trivial (which is acceptable given the experimental nature of this work). However, I have the following suggestions/concerns:\n\n- There seems to be no need to state Lemma 3.1, which is very straightforward (i.e., just one-line proof) and adds no value to the intuition.\n\n- Theorem 3.2 also provides minimal value to the paper, as it is a default convergence result from Q-learning.\n\n- Theorem 3.3 is a bit problematic in my mind. The proof is very vague and causes me doubting its correctness. In particular, I am confused by how the Hoeffding's inequality is used here (which typically bound the gap between sample mean and true mean): what's the sample here and what's the estimation to be made. I hope the authors can clarify this during the response.", "experimental_designs_or_analyses": "The experiments are extensive from the perspective of datasets, models, and baselines. Improvements are indeed observed, while I feel limited observations/findings are provided. \n\nFrom example, some analyses can be made on how many demonstrations are selected by each method, what key characteristics are the demonstration selected by RDES but no other baselines, whether Q-learning or other RL choices matters for performance, etc.", "relation_to_broader_scientific_literature": "I feel this work has limited contributions to the broader community: (1) viewing demonstration selection as a RL problem is already reported as mentioned in the literature review section; (2) standard RL algorithm is adopted; (3) the importance of diversity has also been reported before in ICL.", "essential_references_not_discussed": "Related works are properly cited, but some key relationships need some further discussions, e.g., what's the difference between this work and Zhang et al., 2022, both of which take an RL perspective for demonstration selection.", "other_strengths_and_weaknesses": "Other concerns that I have and would like to have the authors clarify:\n\n- The RL training part is vague in the algorithmic implementation part (section 3.3 and algorithm 1). I would love to hear about a more complete design of RDES.\n\n- I am wondering about the purpose of adding a diversity term in the reward -- The final goal of the optimization is to get better accuracy (if I understand correctly), and encouraging diversity is a method to achieve this goal (instead of being a part of the goal). I woud like to hear from the authors more on the justification of this reward design.\n\n- During the optimization process, if I understand correctly, a lot of interaction with LLM need to happen (in particular, to get the accuracy score). Can additional results and comparison be provided on the impact from the number of training samples (e.g., how many samples are needed in this work and other baselines)?", "other_comments_or_suggestions": "NA", "questions_for_authors": "My questions are provided in other sections of the review, which are summarized in the following:\n\n1. The necessity and correctness of the theoretical results (in 'Theoretical Claims')\n\n2. Additional details of the experiments (in 'Experimental Designs Or Analyses')\n\n3. Comparison with previous work on RL for ICL demonstration selection (in 'Essential References Not Discussed')\n\n4. A few other aspects regarding reward, the overall design, and sample complexity (in 'Other Strengths And Weaknesses')", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The work introduces a reinforcement learning framework for selecting in-context demonstrations that balance relevance and diversity for few-shot text classification tasks.

This paper received mixed reviews. I took a deep dive in understanding the reviews and the authors rebuttal. Authors did an excellent job in answering most of the key concerns raised by reviewers. 

Strengths of paper:

- Interesting approach

- Extensive experiments to validate the proposed approach

Weaknesses:

- No user studies

- Hard to apply to open-ended tasks like generation or structured prediction without significant modifications

Overall, I recommend acceptance of this work. Please add missing citations as suggested by reviewers.
Decision: Accept (poster)