In-context Learning with Retrieved Demonstrations for Language Models: A Survey
Authors: Man Luo
Xin Xu
Yue Liu
Panupong Pasupat
Mehran Kazemi
Venue: Accepted by TMLR ()

Abstract: Large language models have demonstrated remarkable few-shot in-context learning (ICL) capabilities, adapting to new tasks with few-shots demonstrations. However, the efficacy of ICL is highly dependent on the selection of these demonstrations.
Recent developments have introduced retrieval-based in-context learning (RetICL), which dynamically retrieves demonstrations tailored to each input query. This approach leverages existing databases and retrieval systems, enhancing efficiency and scalability while mitigating biases inherent in manual example selection.
Given the promising results and growing interest in RetICL, we present a comprehensive survey of this field. Our review encompasses:
design choices for ICL demonstration retrieval models, retrieval training procedures, inference strategies and current applications of RetICL. In the end, we explore future directions for this emerging technology.

Published in Transactions on Machine Learning Research (10/2024)
In-context Learning with Retrieved Demonstrations for Lan- guage Models: A Survey
Man Luo Intel Lab.
man.luo
Xin Xu Google Research
amjjssminee@ggogle..ccm
Yue Liu Google Research
glinjuune@hooglee.com
Panupong Pasupat
ppasupat@@gongee.com
Google Research
Mehran Kazemi
mehrenkoosemi@@googee.com
Google Research
Reviewed on OpenReview: https: opwp:eviee. net/ forum? id= HQPo8ZhQPa
Abstract
Large language models have demonstrated remarkable few-shot in-context learning (ICL) capabilities, adapting to new tasks with few-shots demonstrations. However, the efficacy of ICL is highly dependent on the selection of these demonstrations. Recent developments have introduced retrieval-based in-context learning (RetICI.). which dynamically retrieves demonstrations tailored to each input query. This approach leverages existing databases and retrieval systems, enhancing efficiency and scalability while mitigating biases inherent in manual example selection. Given the promising results and growing interest in RetICL, we present comprchensive survey of this field. Our review encompasses: design choices for ICL demonstration retrieval models, retrieval training procedures, inference strategies and current applications of RetICL. In the end, we explore future directions for this emerging technology.
1 Introduction
Few-shot in-context learning (ICL) is the ability of large language models (LLMs) to perform few input-output examples, demonstrations, for the new task are given alongside the actual task input. Importantly, the model parameters do not have to be fine-tuned towards the new task. ICL is popularized by the work on pre-trained large language models, which can perform ICL without being trained to do so (Brown al. 2020), though smaller language models can also be explicitly trained to perform ICL. (Min ct al. 2022a).
task when
ICL presents several advantages over the conventional methodology for adapting language models to downstream task, which typically involves initial pre-training followed by subsequent fine-tuning. One significant mcrit of ICL is the circumvention of fine-tuning, which might not always be possible due to limited
access Lo the model parameters or constraints on computational resources (Brown el. al. 2020). Furthermore,
ICL avoids common issues associated with fine-tuning, such as overfitting (Ying 2019 Kazemi et al. 2023a).
Compared parameter-officient fine-tuning mothods (PEFT) (Hu et al., 2021
Dettmers
2023 Lester
et al. 2021), , ICL is computationally cheaper and remain the model parameters unchanged thus preserving the generality of the L.MSs.

---

Published in Transactions on Machine Learning Research (10/2024)
Early ICL implementations use fixed set of demonstrations for each target task. These demonstrations
could be hand-crafted by human (Hendrycks et al. 2021 Wei et al. 2022 Kazemi et al. 2023b randomly
chosen from training data (Brown et al. 2020 Lewkowycz et al. 2022]. Beyond random selection, there are
2022 diversity (Li & more advanced selection processes based on metrics such as complexity (Fu et al.,
Qiu 2023a), difficulty (Drozdov et al., 2023 ; concept learning (Wang al.) 2023b) and perplexity (Gonen
ct al. 2023). Importantly, the demonstrations remain content-insensiifie (i.c. the same demonstrations
are used regardless of the query) which could hinder unlocking the true potential of the LLMs. The effectiveness of such demonstrations is influenced by factors such as the quality, quantity, and ordering of the
demonstrations (Brown et al. 2020 Naik et al. 2023 Kojima et al. Lu et al. 2022b).
Retrieval-based ICL (RetCL presents & paradigm shift in the optimization of language model performance, moving beyond static, pre-defined demonstration sets to a dynamic, context-sensitive approach. At the heart of this innovation is the concept of adaptive demonstration selection, where a specialized retriever intelligently curates tailored demonstrations for each spocific task input. This method has not only consistently outshined
approaches relying on random static hand-crafted demonstrations (Liu et al. 2022 Rubin et. al. 2022 Luo
et al.
2023) but has also demonstrated remarkable resilience
variety of influencing factors (Li et al.
2023b).
RetICL shares similaritics with the broader concept of Retricval Augmented Generation (RAG) Guu
et al. 2020 Lewis et al., 2020 Izacard et al. 2023 Both approaches aim to retrieve external information to augment the prompt and improve model inference performance. However, RAG encompasses a wider spectrum of retrieval methods and use cases. While RetICL specifically focuses on retrieving demonstrations for in-context learning, RAG can involve retrieving various types of relevant information in response to a query, not necessarily limited to demonstrations.
The efficacy of RetICL pivots on the "relevance" and "usefulness" of the demonstrations it selects, process intricately influenced by multiple elements. These include the nature of the retriever retriever-ranging rranging from general off-the-shelf models to finely-tuned, domain-specific variants the source and diversity of the retrieval corpus, the retricver's objectives (focusing on cither similarity or diversity), and the strategics for integrating multiple remonstrations. Over the past two years, numerous and sometimes concurrent works have st udied RetICL each with different terminology and with variations in problem definition and subsequent methodologies, making it difficult to comprehend the current state of rescarch and practice in RetICL, espocially for newcomers to the field. In this comprehensive survey, we meticulously analyze 22 seminal papers in the field of RetICL, detailed in Table ! and provide a categorization of their main building blocks (See Figure 1 Our work not only provides a thorough synthesis of existing research but also underscores the areas where RetICL significantly surpasses previons ICL methods, and illuminates many paths forward for future innovations in this area, thus serving as a critical resource for ICL.
2 Few-shot In-context Learning for Language Models
Language models (LMs)
Zhao et al. 2023 Rosenfeld 2000 Jurafsky 2021 Radford et al., 2019 Raffel et al.,
2020: Lewis et al., 2019 Touvron et al. 2023) are probabilistic models that assign probabilitics to sequences of words and are essential components in many lasks. Let s represent a sequence of words (e.g., sentence) and W1, 122...., W/, represent the tokens in the sequence. Based on the chain rule, the probability p(s) can be decomposed into the following product of probabilities:
72
p(s) = p( )p(pz w1) ... p(u, Wi..... wn-1) II p(ws Wi...., wk wk-1)
where each element in the product corresponds to the probability of a token given the previous tokens. Based the above decomposition, an LM can be constructed by learning the probability of the next token given the previous ones.
Earlier LMs were mostly based on N-gram models
(Jrraskky
2021 | and has been used in computing
approximations to English word sequences (Shannon
1948) and speech recognition system (Baker 1990
Jelinek et al. 1975 Baker 1975
Bahl et 1983
Jelinck
1990).
N-gram models are based on the
The list of papers in this table can found here httppr///gttbb.cce////aaa//////e reneereterrccttt_e_reeeee////
2

---

Published in Transactions on Machine Learning Research (10/2024)
Definition ICL
Number of Demonstrations
Sec2: Few-shot for LLM
Demonstration Format
Factors to Effect Performance
Order of Demonstrations
Diversity of Demonstrations
Chain-of-Thoughts
Similarity
Retrieval Objectives
Diversity
One-hoc
Sec3: ICL with Demonstration Retrieval
Few shots Retrieval Inference Strategy
Clustering
Iterative
Retrieval-base context learning for LLM (Ret-ICL)
Training Data
Retrieval corpus
Mix Domain Data
BM25
Cross Domain Training Data
Sec4: Off-the-
Shelf Retriever
Sentence Embedding
Unlabeled Queries
Dual Encoder based Retriever
Free Form Corpus
LLM Based Signals
Collect Training data
Fine-tuned Demonstration Retriever
Model Free
List-wise Ranking
Training Objectives
InfoNCE (Contrastive Learning)
Natural Language Understanding
KL-Divergency
Reasoning
Iterative Training
Sec 6: Tasks and Application
Knowledge- Question Answering
Diversity Training
Re-ranker Training
Text Generation
Figure 1: Structure of the Survey.
3

---

Published in Transactions on Machine Learning Research (10/2024)
Paper
LLMs
Retrieval Method
Retrieval Corpus
Evaluation Tasks
Retriever Training
Retrieval
Strategy
SYNCHRO
GPT-3
SBERT Target Sim-
In Domain
CodeGen
Target Similarity Tun- Top-k
MESH 2021
ilarity
ing
KATE 2022
GPT-3
RoBERTS
In Domain
SA, Table2TTxx, QA
Sentence Similority Embeddding
Top-k
EPR 2022
GPT-J,
SBERT, BM25, Fine- In Domain
SP
TufoNCE
Top-k
CODEX,
tuned Retriever
Loss (contrastive
GTP-3
learning)
GPT-J,
SimCSE (Sentence Free Form Corpus SA
Sentence Embeddings Similarity
Top-
ICL 2022
GPT-
Embeddings)
NeoX,
GPT-3
Mem
GPT-3
Query tion SBERT
Transforma- Tn Domain with Hu- Word Reasoning, Eth-
Sentence: Similarity Embeddings
Top-k
Prompt 2022
man Feedback
ical Reasoning
Teach
T5-11B
BM25
In Domain with Hu- QA
Term-based Similarity
Top-k
Me 2022
Feedback
1C-
GPT-3
Fine-tuned SBERT
In Domain
DSI-to-SQL
Contrastive Learning
Top-k
DST 2022
Retriever
Vote-
GPT-Neo,
SBERT
Selected subsets of In Domain, labeled
SA, NLI, QA, Summ, Sentence Embeddings
Top-k
k 2022
GPT-
CSR, RC
Similarity
J,GPT-3,
CODEX
Auto-
CPT-3
Clustering SBERT
with
LLM Generated CoT for In Domain
Mathlt, QA Text-to-SQL
Sentence Emboddings Clustering Similarity
Col 2022b
XRICL 2022
2022
CODEX
Fine-tuned Retriever Combined with Ro- ranker
Cross Domain
Distillation by KL Di- Top-k vergence
DSP 2022
GPT-3
ColBERIV2
In Domain
QA
Token Embeddings Iterative Similarity
PARC 2022
mBERT
Multilingual SBERT
Cross Domain
SA, TopicC, NLI
Senteuce Embeddings Similarity
Top-k
Cover-
CODEX,
BM25 and Diversity
In Domain
SP
Diversity Training
Iterative
2022
T5-large
Selection
Prompt
CPT-3
Fino-tuned Retriever
In Domain
Mathit
Policy Gradicnt
Top-k
PG 2022a
Dynamic
DODEX
Tree Structure Simi- In Domain
SP
Diversity and Lexical Overalp
Iterative
Least-to-
larity
Most 2022
Self-
Instruct
SBERT with Cluster- Free Form Corpus
QA
Sentence Embeddings Clustering Similarity
Prompt-
GPT,
ing
ing 2122a
CODEX
CEIL 2023a
GPT-
BM25, BERT, Fine- In Domain
SA, PD, NLI, CSR, Diversity Training QA, CodeGen, and SP
Iterative
NEO.
tundd Retriever
GPT12XL,
CODEX
XGLM-
BM25, BM25 and
In Domain
Term- based Similarity
Top-k
BM252023
7.5G
Term Recall hased
rerank In Domnin
ICL-
GPTNeo-
BM25
In Domsin Cross Do- MT main
Term- based Similarity Top-k
MC 2023
2.7B,
XGLM-
2.9G.
BLOOM-
ICL-
OPT-
SBERT
In Dounin
Inteut Classification
Senteuce Emboddings Top-k Similarity
ML 2023
13/175B,
I.LMMA-
7/70B
UP
CPT-Nco,
Fino-tuned Retriever
Cross Domaius Hu- RC, QA, NLI, SA, Contrastive Learning Top-k Feedback
RISE
BLOOM,
CSR, CR, PD
OPT,
GPT-3
4

---

Published in Transactions on Machine Learning Research (10/2024)
Paper
LLMs
Retrieval Method
Retrieval Corpus
Evaluation Tasks
Retriever Training Retrieval
Strategy
Dr. 2023
PaLM,
BM25, GTR. Fine- tuned Retriever
In Domain
QA, NLI, Mathi
Contrastive Learning Top-
Flan-
PaLM
L.M--
LLaMA-
Fine- Fine-tuned tuned Retriever Mix Domain and Reward Model
QA. CSR, CR, PD, Contrastive learning +
Top-k
R
2023a
7B
RC, SA, D2T, Summ, KT
NLI
CDR 2023b
GPT-J.
Fine-tuned Unified Mix Domain
SA. TC, CSR, NLI, Contrastive Learning Top-k
GPT-Neo,
Retriever with Itera- tive Data Mining
SP. StoryGen, Summ,
CODEX,
D2T
GPT-3
MoT 20236 2023b
ChatGPT
SBERT with Clus- Unlabelled Queries
MathR, NLI, CSR, Sentence Embeddings Clustering Similarity
tering and Filtering, with LLMs CoT
QA
LLM as Hetriever
RetICL 2123
CODEX
Iterative LSTM with In Domain
MathR PPO and GAE
Iterative
Reinforcement Learn-
ing
Ambig-
Flan-
Fine-tuned multilin- Domain gual T5, and Filter- ing Algorithu Based ou LLM foodback
TC,,
Contrastive Learning
TTp-k
ICL 2023
PaLM
Table 1: Comparison with Related Work. Abbreviation for Evaluation Tasks: CodeGen (code generation), SA (sentiment analysis), Table2Text (Table to Text generation). QA (question answering), SP (semantic parsing), DST (Dialogue State Tracking), D2T (Data-to-Text), Summ Summarizationn) CSR ( commonsense reasoning), RC (reading comprehension), NLI (natural language inference), CR (Coreference Resolution), MathR (mathematical reasoning). PD (paraphrase detection), TQA (Table Question Answering), TC (Topic Classification), StoryGen (Story Generation). MT (Machine Translation)
Based this assumption, p(w, wi,...: Ws-1) is approximated, e.g., by p(ws, Wk-2, Wk-1) in the case of a bi-gram model; p(wk Wk-2, wk-1) is then approximated statistically based on the number of times wy appeared after Wk-2 in a large corpora of text, Wx-1 divided by the total mumber of times W2-2, Wk-1 appeared in the corpora.
Markovian assumption that the next token only depends the recent context (Markov 1913
With the advent word embeddings (Bengio et al. 2000 Mikolov el al. 2013). neural approaches to language modeling gained more popularity, in which a neural network used to predict the next token probability. The use of powerful neural networks such feedforward network (Bengio al.] (2000,, recurrent neural model (Mikolov et al. 2012), specifically, long-short term memory (LSTM) models (Hochreiter & Schmidhuber 1997) and Transformer models (Vaswani et al. 2017) allowed for predicting the next token probability based on a much longer and a variable length context, thus enabling better estimation of p(wk Wi,..., Wk-1). 1).
The increased power of neural LMs led to a new learning paradigm for NLP problems. Historically, the dominant learning paradigm for NLPP problems was train models on task-specific data from scratch. Consequently, for each new task, the model had learn everything from scratch. This often resulted in poor generalization, especially in the cases where previously unobserved vocabulary was observed at the test time. In the subsequent paradigm, how language works and gain vast amount of knowledge about the world (Petroni et al. 2019 Lin et al. 2020 Sung et al.] (2021] Yuan el al. 2023); ; the pre-trained LM (PLM) was then further finetuned on data from the new tasks (Sarzynska- Wawer et al.) 2021 Devlin et al., 2018 thus teaching the general PLM the specifics of the new task. This paradigm often resulted in faster learning and higher predictive performance. It was later shown that further finetuning a PLM on multiple tasks leads to better transfer of knowledge across tasks and may lead to better performance on new tasks (Raffel et al. 2020
LM was first pre-trained on large corpora of text making it learn about
5

---

Published in Transactions on Machine Learning Research (10/2024)
2.1 In-Context Learning
As the scale of the PLMs and the scale of the datasets on which these models were pre-trained increased leading to pre-trained Large Language Models (LLMs), it was discovered that pre-trained LLMs (hereafter, referred to LLMs for brevity) have remarkable capability of learning in-context from a few demonstrations (Brown et al. 2020). That is, LLMs were shown to be able lo adapt to new tasks by only seeing a few examples of the new task in their imput, as opposed to needing additional training data or fine-tuning, This is typically referred to as few-shot in-conteat learning. In the following, we will illustrate this formalll.
Let be a task and q- T represent sample query from this task for which we would like to find an answer using an LLM. In the of few-shot learning, find construct multiple demonstrations {di,-.., ,dk) where each demonstration d; = (gi, a;) contains query qi ~ T and the answer a; to that query, and feed an input of the form
Prompt 1: Few-shot Prompt
a1
to the LLM. The input is typically referred to as prompl. It is common to add some separator tokens to the prompt so the boundaries of the rationss and the questions and answers within those demonstrations are clear. An example prompt will then be as follows:
Prompt 2: Prompt 1 + Separator Tokens
Demonstration 1: Query: Answer:
Demonstration Query: 9&, Answer: Demonstration + 1: Query: ge, Answer:
The demonstrations serve as few examples of the task helping the LLMs learn both the input and label space as well as the mapping from the inputs to the labels (Wei et al, 2023 Pan 2023 both in context (without any weight updates), so a similar mapping can be applied to g.. Few-shot learning a remarkable capability of LLMs, showcasing their generalization ability to rapidly adapt to a wide range of tasks with explicit instruction. While LLMs show strong few-shot learning capabilities off-the-shelve, it has been shown that warming them up by finctuning them few-shot data from multiple tasks will further boost their Radford et al. 2019).
few-shot learning capability (Min et al. 2022a Chen et al. 2022
Another remarkable ICL capability of LLMs is to learn from in in-context instructions: finetuning LLMs on instructions from multiple tasks makes them learn to follow instructions for new tasks (Ouyang et al. 2022 Longpre et al., 2023 Zhang el al.] 2023). In this case, commonly known as instruction luninng, the LLM is finetuned on data of the type I' .q" where I" represents the instructions for task T describing how the task should be performed, 9º represents query from task T and represents the answer. The finetuning is performed on data from multiple tasks and multiple queries from each task. It is also possible to combine instructions with few-shot demonstratoons, in which case example prompt is as follows:
Prompt 3: Prompt 2 Task Instructions
(Task Instructions] [Prompt 2]
Benefits of ICL: Compared to the aforementioned approach of utilizing LLMs which involves pre-training followed by line-tuning, ICL offers several key advantages. Firstly, fine-tuning may not always be feasible due to restricted access to the LLM, inadequate computational resources, or inadequately labeled data (Brown) ct al. (020)), whereas ICL requires fewer resources, less data, and casier to serve through API calls. Additionally, ICL avoids the issues commonly associated with fine-tuning, such as overfitting or shocks (Ying)
2019 Kazemi et al. 2023a), it does not modify the model's parameters, allowing it to remain general.
6

---

Published in Transactions on Machine Learning Research (10/2024)
2.2 What Makes for Good Demonstrations?
Several works try to provide thel justifications and insights into how LLMs learn from few in-context demonstrations (Xie et al.) 2021 Garg et al.) 2022 Von Oswald et al.) 2023). However, the exact reasons behind this capability are still largely unclear making it difficult to select optimal few-shot demonstrations. Fortunately, various empirical results show the effect of the few-shot demonstrations on the predictive accuracy of the I.LMs and provide suggestions on the best practices for preparing them. They also show the brittleness of the LLMs in the choice, format, and order of the few-shot demonstrations. Here, we describe some of the more prominent ones.
Number of Demonstrations: LLMs generally benefit from more demonstrations, but as the number of demonstrations increases the rate of improvement typically decreases (Brown et al. 2020: Ye et al. 2023b Min et al., 2022b). Generation tasks have been shown to benefit from an increased number of demonstrations more than classification tasks (Li al. 2023b). Toward increasing the number of demonstrations, one barrier is the maximun context size of the LLM. While the size of the context has been increasing over time with newer LLMs (Team et al.] 2024 Chen et al.) Reid et. al., 2024 Peng et al.) 2023a Gu & Dao 2023 Li el. al., 2024), it may still be problematic for datasets with long input texts or classification datasets with many classes.
Demonstration Formatting: Varions works have shown that the formatting and wording of the prompts can play a crucial role in the performance of the LLM (Jiang et al.) 2020 Shin et al. 2020 Kojima et al.j Yang al., 2023). For example, Kojima et al. shows that simply adding Let's think step by step to the prompt makes LLMs reason step by step and solve substantially more problems, and Weller et al. (2023) show that adding According to Wikipediaa to the prompt makes them more factual. Moreover, Min et al. (202bb) shows that besides the text formatting, the label space and the distribution of the input text in the demonstrations are also of immense importance.
Order of Demonstrations: The order of demonstrations has been shown to substantially affect the model performance. For example, Lu et al.] (2022b) show that on some tasks, the model performance can range from near-random state-of-the-arr depending on the order of the prompts, and Zhao et al. (2021) show that answers appearing toward the end of the prompt are more likely to be predicted by the model.
Diversity of Demonstrations: Another important factor in the success of few-shot learning is the diversity of the demonstrations (Li et al.) 2022b). Zhang et al.] (2022b) propose to select a diverse set of questions as few-shot examples. Ma et al. (2023) propose a fairness metric for selecting demonstrations which encourages selecting diverse few-shot demonstrations that produce a near uniform predictive distribution for semantic-free input. These works have shown that the diversity of demonstrations crucial for achieving better in-context learning (ICL) performance.
Chain of Thought (CoT): It has been shown that including a rationale for the answer significantly improves model performance, especially for models that are larger than certain size (Suzgun et al. 2022 The rationale is commonly known as choin of thought (CoT) (Wei et al. 2022 In the case of CoT prompting, the demonstrations are typically formatted as:
Query: 4i
Rationale ri Answer :
with the rationale appearing before the final answer. Naik et al. (2023) found that the diversity of reasoning path is important as well, therefore they propose DiversePrompting where an LLM is prompted to generate
diverse reasoning path to solve a problem. Several works have investigated the reason behind the efficacy of
CoT prompting and how to improve the prompts and rationales (Wang et al. 2022a Lanham et al. 2023
3 In-context Learning with Demonstration Retrieval
Traditionally, the same set of few-shot demonstrations is used on all queries, which can be suboptimal especially when there are high variations among the queries. An alternative is retrieve few-shot demonstrations that.
7

---

Published in Transactions on Machine Learning Research (10/2024)
are tailored to the current query. Previous work has shown that demonstration retrieval leads to substantial improvements in the task metrics, compared to manually curated randomly selected demonstrations (Luo et al. 2023 Ye et al. 2023a). Furthermore, LLMs have been shown to become less sensitive to the factors such as demonstration ordering (Section 2.2 when retrieved demonstrations are used (Li et al. 2023b).
This section gives an overview of the retrieval-based ICL, (RetICL). We start by defining TCL with refrieved demonstrations. Formally, given query q. and a retrieval corpus C, a demonstration retriever DR selects set of demonstrations {dl,..., d ~ C, where each demonstration is d, (g. a.). The LLM imput sequence becomes (d1...., dg q-). The goal of the retriever is to select demonstrations that maximize the probability of the correct answer 0..
The success of RetCC depends on several factors. This section explores design choices, including the retrieval objectives, retrieval inference strategy, and retrieval corpus. Then in Sections and 3 we explore the retriever models and how to train them to tailor to downstrcam tasks.
3.1 Retrieval Objectives: Similarity and Diversity
Various retrieval objectives for selecting and tailoring in-context examples for LLMs have been explored (Luo
et al. 2023 Rubin et al.] 2022 Ye et al. 2023a Dalvi et al. 2022 Cheng et al. 2023: Li et al. 2023b
There are two primary retrieval objectives for selecting demonstrations: similarity and diversity. Similarity
involves selecting demonstrations most akin to the query and can be based on language similarity (term
matching (Luo et al. 2023 Rubin et al.
2022 Agrawal et al. 2022 Ye et al.
2023a Dalvi et al. 2022 or
semantic matching (Rubin et al., 2022: Li & Qiu 2023b Wang al.] 2023a. Liu ct al. 2022) ) structural
aspects (sentence structure (Poesia al.
2022), reasoning structure (Fu 2021 Levy al. 2022 Drozdov al.
et al., 2022)). ). Most studies focus on language similarity, with fewer addressing structural similarity, often due to the challenges in extracting query's structure in many tasks (Levy et al. 2022) Beyond similarity, some work has found that the diversity of demonstrations is important. The motivations for diversity include avoiding repetitive demonstrations (Zhang et al., 2022b), bringing different perspectives (Yu el al.) 2023), and maximizing the demonstrations' coverage of the test query, in terms of covering either its words or syntactic structures (Levy al. 2022). Mcasuring the diversity of multiple demonstrations is major technical challenge. Ye et al. (2023a) applied determinantal point processes (DPP) a probabilistic model to measure the negative interaction (Kulesza et al., 2012), to measure the diversity. Levy et al. (2022) found that diversity and coverage are important when the model is unfamiliar with the output symbols space. It is noteworthy that researchers have found that ICL benefits more from demonstrations with higher complexity in some scenarios (Fu et al.
2022). where they define the complexity in terms of the query length or reasoning steps. However, Fu ct al. [2022) employed houristic rules to define complexity and pre-selected demonstrations accordingly. Their research revealed that using a similarity- relriever led to improved performance in a specific mathematical reasoning task. This might indicate that combining similarity and complexity considerations could be a promising strategy for enhancing the approach to reasoning tasks.
3.2 Inference Strategy to Retrieve Few-shots Demonstrations
This section explores varions strategies for employing a retriever to gather k demonstrations, We divide these into three distinct methooologges.
Top-k Retrieval This is the simplest and most. popular retrieval strategy (Liu et al. 2022 Rubin et al. 2022: Li et al.) 2023b Luo et al., [023 Gao et al., (002).. To obtain & demonstrations, given a query, the retriever ranks the demonstrations and then selects the top-k demonstrations. Thus, each demonstration is chosen independently of the others. This method is straight and fast, however, it might not yield the best combination of demonstrations as these demonstrations might be homogeneous.
Clustering Retrieval To mitigate the issue of homogeneity in one-hot retrieval, clustering retrieval approaches (Li et al., 2022a Zhang et al. 2022bb Li & Qiul 2023b) categorize all demonstrations into k sub-groups aiming to group similar demonstrations together. Then given a query, the retriever picks the most similar demonstration from each sub-group resulting in final set of k demonstrations. The core principle of
8

---

Published in Transactions on Machine Learning Research (10/2024)
clustering is to select diverse range of demonstrations. Most of the work use SBERT (Reimers & Gurevych 2019) to encode the demonstrations (only the question or the entire demonstrations) and then apply k-means for clustering.
Iterative Retrieval The earlier retrieval strategies acquire each demonstration independently. However, in iterative retrieval, a retriever selects demonstrations based on both the query and previously retrieved demonstrations (Khattab et al. 2022 Levy et al., 2022 Drozdov el al. 2022 Ye el al., 2023a Scarlatos & Tan 2023). This process starts with a single query, for which the retriever finds one best demonstration. The query is then augmented (c.g. combined with the demonstration) to retrieve the next demonstration. This step is iteralively executed k times to gather k demonstrations. The general idea is to select the demonstrations that can complement each other. An an example of a work from this categorym, Scarlatos & Lan (2023) train an LSTM retriever using a reinforcement learning framework. During the inference phase, the retriever processes the imput query to select the best initial demonstration. It then generates a new query representation by integrating the query with prior demonstrations, specifically utilizing the hidden state representation from the LSTM model. This process of updating the query representation and obtaining subsequent demonstrations continues iteratively until & demonstrations retrieved.
3.3 Retrieval Corpus
The retrieval corpus forms a pool of demonstrations that the retriever can access. Using annotated data is one of the most straightforward ways to construct the retrieval corpus. This setting assumes that training data related to a task is available, and thus can be used as the retrieval corpus. Under this setting, there are three main ways to construct the corpus that we will discuss individually below.
In-Domain In this setting, an in-domain training set, independently and identically distribution (IID) with the test queries, is available and serves as the retrieval corpus. Most cxisting work take the full training set as the corpus. However, to be more annotation efficient., Hongjjn et al. (2022) uses only a subset. M of the training set N which includes the most representative and diverse ones, where |M] << [V]. One question that remains unanswered from the work of Hongjin et al. (2022) is how the predictive performance is affected function of retrieving from subset M instead of the entire training set N. While there is no follow-up work to answer this question, the closest comparison we find is the results in Ye et al. (0023a) where a similar setup as Hongjin et al. (2022) is used except that they use the entire training set as the retrieval corpus, and report lower performance on the SST-5 dataset (compare the Figure 3 in Hongjin et al. (022) and Table 3 in (Ye et al.] 2023a)): ) While there might be other differences (e.g. the number of the demonstrations and the templates being used to do the inference) between the two sctups that may affect the final performance, this comparison implies that retrieving from carefully selected subset might have comparable results to retrieving from the entire training set.
Mix-Domain The previous scenario has one individual retrieval corpus for different tasks. Assuming that we want to test model performance on two tasks, then in the in-domain setting, there will be two retrieval corpora separately. Furthermore, the in-domain setting assumes that the model has knowledge about which task the test question belongs to such that when it comes to the retrieval phase, it knows which corpus to select the demonstrations from. However, this assumption does not hold several real-world applications of LLMs. In the mix-domain setting (Wang et al., 2023a),, the retrieval corpus is constructed from the combination of all tasks. At the inference time, given question, the retriever will retrieve demonstrations from this mixed corpus; the demonstrations can come from the same domain the test question or from other tasks. However, the authors have not discussed whether the mix-domain approach is more beneficial than the single-domain approach. Li et al. (2023b) propose a unified retriever trained on a mixture of 40 tasks. This unified retriever can be used for different tasks during inference. The authors have compared the unified retriever with the single-domain retriever (Rubin al.) 2022) and demonstrated improved performance. It's worth noting that while the unified retriever is trained on mix-domain data, the actual selection of demonstrations during inference depends on the specific implementation.
9

---

Published in Transactions on Machine Learning Research (10/2024)
Cross-Domain In this setting, IID human-annotann demonstrations are not available for the test queries, S0 one uses annotated demonstrations from other similar tasks but different domain (Cheng et al.) 2023 Shi et al. 2022). The distinguish between the mix-domain and the cross-domain settings is that in the former setting, part of the corpus is IID. Shi et al. (2022) describes a scenario where the goal is to parse a Chinese query into SQL. However, the remonstrations are sourced from an English Text-lo-SQI. corpus, a domain with significantly more resources than the target domain. [hhi et al. (2022) employs this high-resource data as the retrieval corpus. To adapt to the target domain during inference with a LLM, the target query is translated into the same language as the demonstrations, Nie et al. (202)) presents similar approach but the retrieval corpus consists of multiple cross-domain datasets. The main benefit of this setting is to address low-resource training data issues or situations where no training data are available.
Unlabelled Queries with Automatically Generated Answers The previous three corpora all presup- pose the availability of human-annotater data. However, this assumption may not hold in real-life scenarios, particularly in streaming set settings where users can pose questions without any pre-annolated answers. Several studies (Zhang al. 2022b Li & Qiu 2023b) have suggested using LLMs to generate answers for unlabeled data. They apply filtering techniques to determine the quality of these generated answers, adding only those examples with high-quality answers to the retrieval corpus. The most widely used filtering technique is based on self self-consistency (Wang et al., 2022c). This approach involves prompting the language model to generate multiple chains of thought and answers, then selecting the most common answer as the final response.
Free Form Corpus Another approach to deal with the lack of human-annotater data for similar tasks is to create pseudo-demonstration from unstructured text. Toward this goal, [yu et al. (2022) utilized the Demix dataset (Gurrragaa et al. 2022), which is not tailored for any specific task. To generate pseuuloddemonsttaaiionn. retriever selects the top-k most relevant sentences from the datasel. Subsequently, arbitrary labels are attached to each sentence to form the examples. Such pseudo-demmonstrations are beneficial compared to the zero-shot setting. The motivation behind using arbitrary labels comes from (Min) et al. 2022by, who found that replacing gold labels with random labels only marginally hurts performance, suggesting that the input-output format is key. Li et al. (2022a) propose a synthetic question answering generation method to create QA pairs using the synthetic generated passages by an LLM. The authors also compared the synthetic generated data with the training data as demonstrations and found that the performance is comparable to using the training set, without requiring extensive human labor to collect the training data.
4 Off-the-shelf Demonstration Retrievers
To achieve the retrieval objectives outlined above, rescarchers have explored various types of demonstration retrievers Robertson et al. (200 ; Reimers & Gurevych (2019); Liu el al. (2019) Yang et al. (2019); Ni el al.
[2021]: Santhanam et al. (2021). typical demonstration retriever encodes examples from the retrieval corpus
and the query into some vector representations, and then a similarity measure (e.g. cosine similarity) is calculated between candidate demonstration embeddings and the query embedding to locate the most related demonstratiooss Rubin et al. (022) Li & Qiu (023b); Wang et al. (023a)). Given the limited understanding of the underlying mechanism through which retrieved demonstrations enhance the performance of LLMs, initial research efforts focused on a houristic ovaluation of readily available retrievers for this task (Liu lt al. 2022: Zhang et al. 2022b). Subsequent research endeavors explored the design and development of learning- based retrievers specifically customized for retrieving demonstrations (Luo et al.] 2023 Cheng et al. 2023 Li et al., 2023b). This section reviews representative off-the-shelf models and we will discuss the learning based models in Section 5]
Term-based Similarity BM25 (Robertson et al. 2009) is one of the most popular term- scoring methods due to its simplicity and effectiveness in producing relevant results. It takes into account both term frequencies and document lengths. It has been empirically demonstrated in various works (Luo et al.) 2023: Rubin et al. 2022, Agrawal et al. 2022 Ye et al. 20223 Dalvi et al., 2222) that using BM25 to select similar examples as few-shots in ICL can help improve the performance of many LLM inference tasks. While BM25 has become standard baseline model in the field, it is not without its limitations. Due to its heavy
10

---

Published in Transactions on Machine Learning Research (10/2024)
reliance on term frequency and document length, this approach may overlook crucial aspects such semantic meaning and sentence structure. Consequently, BM25 might fail to retrieve instances that are semantically similar to the input query but lack exact term matches (Guo et al. 2016 Xiong et al. 2017 Mitra et al.) 2018p. Another drawback is that BM25 lacks the capability for finc-tuning in downstream tasks, making it less competttive compared to neural models which can be fine-tuned and customized for specific downstream tasks.
Sentence Embedding Similarity In this approach, queries and documents are encoded to the same dense embedding space using an off-the-shelf sentence embedding model, and then similarity scores (e.g. cosine similarity) are calculated to rank the most relevant documents for each query. A rich collection of sentence embedding methodologies exists in the literature (Reimers & Gurevych 2019, Yang et al. 2019: Xue et al., 2020 Wang et al. 2022b). Sentence-BERT (SBERT) (Reimers & Gurevych 2019 is modification of the pretrained BERT network that uses siamesc and triplet network structures to derive scmantically meaningful sentence embeddingg. The effectiveness of SBERT embeckinng for demonstration ret rieval has been investigated in several works (Rubin et al. 2022 Li & Qiu 2023b Wang et al.) 2023a), and the results show that retrieving demonstrations based on SBERT emboddings often provides a boost in performance compared to zero-shot or random few-shot selection. In the KATE method (Liu et al.) 2022), the authors studied using vanilla RoBERTa (Liu et al. 2019) and finetuned RoBERTa on NI.I (Bowman et al., 2015b) and STS-B (Cer et al. 2017) datasets for selecting good demonstrations, and found that the finetuned version on task-related datasets offered further empirical gains. Note that here, the demonstration retriever is not trained for ICL demonstration retrieval based on lask-specific data (a topic which we will discuss in Section 5 instead, the retriever is finetuned related tasks to provide a better notion of similarity for the task at hand. So we still categorize it as an off-the-shelf retriever. (Shi et al. (2022) extends the use case to cross-lingual few-shot retrieval in the Text to-SQL semantic parsing task, and they use mSBERT (Reimers & Gurevych] 2019),
mUSE (Yang et al., 2019) and mT55 (Xue et al.) 2020) the baseline models for comparison. Other widely used baseline models for demonstration retrieval include E5tase (Wang et al. 2022b), SimCSE (Gao et al., 2021b). Instead of relying on "word matches" as in BM25, these sentence embedding similarity approaches can better capture semantic similarity (for example, synonyms, and related topics), however computationally they might be more expensive.
Pretrained Dual Encoder Dual Encoder that are pretrained on information retrieval or question-answering tasks can grasp the intricate relationships between complex logical concepts and reasoning processes by employing different semantic embeddings for queries and candidates (Li & Qiu 2023b). Most of the Dual Encoders use one global semantic representation for the query and candidates, such as GTR (Ni et al., 2021) ) while few others such as CoLBERT (Santhanam et al.) 2021) use token semantic representations. GTR (Ni et al. 2021) is T5-based dual encoder model that is pretrained
the CommunityQA (Abujabal al.) 2019
and finetuned on the MS Marco dataset (gguyen et al.] 2016). ColBERTv2 is a state-of-art retrieval model that adopts the late interaction architecture (Khattab & Zaharia 2020) and is trained on the MS Marco dataset. While publicly available pretrained dual-encoder retrievers are not specifically optimized for few-shot demonstrations retrieval tasks, they have been demonstrated to be successful in helping LLMs to learn from the selected examples. For instance, [uo et al. (2023) studied applying GTR (Ni et al. 2021 to select semantically similar examples as demonstrations, and empirically proved that this approach brought in better performance gain than random fewshots for both PaLM (Chowdhery et al.) 2023) and FLAN (Chung et al.) 2022) models. Another example is Khattab et al. (022), who reported results for employing ColBERIv2 (Santhanam et all AAAOONSTRRAEE-SEARRHH-REEAE
(DSP) framework for ICL. In the proposed framework, it is used to retrieve both (i) related knowledge during the search slage and (2) top k similar examples as demonstrations. The pretrained models serve as strong retrievers for demonstrations even without fine-tuning on downstream tasks. Additionally, the pretrained weights can be utilized for fine-tuning the demonstration retriever, significantly reducing the amount of training data required compared to the pre-training phase, as we will show in
2021) as the retrieval module in their
11

---

Published in Transactions on Machine Learning Research (10/2024)
5 Fine-tuned Demonstrations Retrievers
Although off-the-shelf retricvers have shown some promise in retrieving demonstrations for LLMs, the retrieved demonstrations given by the off-the-shelf retrievers might not represent the nature of the lask and how the task should be solved in general. Therefore, it might lead to sub-optimal performance. Researchers thus have started to explore lcarning-based methods to further push the boundarics. A typical objective when designing a good demonstration retriever is: if an LLM finds a demonstration useful when being used as an illustrative example, the retriever should be encouraged to rank the demonstration higher. This allows us to train models directly relying on signals from query and output pairs in the task of interest, without human annotations. To develop a demonstration retriever, the majority of approaches utilize current dual encoder models (Krrpukhin et al. 2020 Ni et al. 2021). The key variations lie in the methods of gathering training data and formulating training objectives. We will explore these aspects in more detail in the subsequent sections.
5.1 Collecting Training Data for Demonstration Retriever
Based on LLMs Signals A popular approach to collecting training examples is to use the supervisory signals from LLMs. In this case, typical paradigm is to first employ some filtering mechanisms (Cheng et al. 2023) unsupervised retrievers (e.g. BM25 and SBERT) (Luo et al. 2023 as the initial retriever, this step can help limit the pool size for mining the right training data. Then a scoring LLM, which serves as a proxy for the inference LLM, is used to score each candidate demonstration d. Here the score is defined as s(e) = p(ald, 9) which is the conditional probability of output answer a given the input query and demonstration
Auother approach is to train a smaller reward model that can provide more fine-grained supervision for dense retrievers. For example, Wang et al. (023a) proposed finetune cross-encoder model serving teacher model for training the retriever.
Once score is obtained, a retriever can be trained that predicts these scores directly (Ye et al. 2023a Alternatively, the candidate demonstrations can be ranked for each query based on their scores, considering the top-ranked demonstrations as positive examples that help the LLM get to the right answer and the bottom-ranked ones as negative examples that mislead the LLM towards the wrong answers; then a retriever
can be trained which separates positive examples from negative examples (Rubin et al. 2022: Cheng et al. 2023
Luo et al. 2023).
There different strategies for choosing the scoring LLM. Ideally, uses the inference LLM itself the scorer in order to perfectly reflect its preferences (Li et al. 2023b Shi et al. 2022). However, training retrievers requires large amounts of labeled data, and it may be expensive use very large models for labeling.
Consequently, for scoring one may gravitate towards utilizing smaller models, especially those within the
same model family as the inference LLM (uuo et al. 2023 Cheng et al. 2023 Rubin et al. 2022
Model-Free One approach to collecting training data for demonstration retriever is to directly measure the similarity between the labels of the candidate demonstrations and the label of the query, and use this similarity as proxy of the importance of a demonstration (Hu al., 2022 Pocsia et al.] 2021). For instance, Hu et al. 2002)) explored a dialogue context where labels are structured as a sequence of stages. The similarity between query's label and demonstration's label is determined by calculating the average F1 scores of these two labels. This method adopts heuristic approach (i.e. stage changes), presuming that the similarity metric can closely resemble the preference for good demonstrations from an LLM, and it often necessitates domain-specific expertise for design.
5.2 Training Objectives
Thus far, we have explored the creation of training data for demonstration retrievers in the context of ICL. We now proceed to examine the commonly used loss functions for training retrievers.
List-wise Ranking Loss The list-wise ranking approach looks a list of candidate documents for a given query and tries to capture the correct ordering for it. Li et al. (002bb proposed to inject the ranking signals
12

---

Published in Transactions on Machine Learning Research (10/2024)
into the retriever using an approach inspired by LambdaRank (Burges 2010). More formally, given each query 9, they first rank all candidate documents according to their relevant scores S = {s(d:) i=1 according to which the associated ranking R = {r(d)));= is computed. Then the loss function is defined as follows:
Σ max (0 r(d;) r(d,) log(1 + (xim(g:,))) simmsggd.))))
1
1 )
Liistwise
d.,d,
where sim(g, d) is the relavance between candidate demonstration d and the imput q. In the list-wise ranking objective, retriever can benefit. from the full ranking of the candidate set to make accurate predictions for the most relevant demonstrations. However, obtaining the full ranking list and calculating the loss function on top of it might be very expensive and time-consuming. Additionally, the model trained to discern the relative preferences between examples without explicitly determining whether an example can serve as an absolute good demonstration.
InfoNCE Loss
Another widely adopted training procedure is contrastive learning using the InfoNCE
loss (Rubin et al. 2022 Cheng et al.) 2023 Luo et al. 2023 When positive and negative examples can be correctly identified, InfoNCE loss an effective loss function because it can take advantage of the supervisory labels produce a representation that sets apart the useful examples for demonstration retrieval. In this approach, each training instance is given in the form of qi, df d >. Here df is selected positive example concerning the input qt, and the negative examples consist one hard nogative example d,,1 and k random examples from the other instances in the same mini-batch. Then the typical contrastive loss can be defined as
esim(aesd))
Lcont L(g,, d.,1...d7. llog
csim(g.d)) Σ sim((
The random negative examples from the same mini-batch are called in-batch negatives. They are typically selected from both the positive examples and hard negative examples of other instances.
Distillation by KL Divergence Ye et al. (023a claims that although the InfoNCE loss has been found effective in training demonstration retrievers and can learn which examples might be superior to others, has the same treatment for all negative examples and the predicted scores from LLM are not fully utilized. As alternative to train demonstration retriever using positive and negative examples, Shi et al. 2002 proposed to train the retriever by directly distilling the LLM's scoring function. More specifically, the retriever model is designed to produce ranking scores that match the usefulness of a demonstration to help with the LLM inference; this is done by minimizing the KL-divergence between the top K examples score distribution from scoring LLM and the ranking score distribution produced by the retriever
(d)) Pretriever (dk))
Laistinl KT [ Pretriever)
(LeM(de)log
Multiple Objectives The training signals, especially the LLM scores, are typically rich enough to define multiple training objectives that can be ensembled. For instance, Wang et al. (2023a) proposed to train the demonstration retriever model with combined objectives: (1) knowledge distillation from the trained reward model which can capture the proferences of LLMs over the retrieved candidates (2) InfoNCE-based contrastive loss to incorporate the in-batch negatives. More specifically, the resulting loss function is as follows:
Lcombined = &houut + Ldistill
Here a is a constant that controls the relative importance of the two losses. They claimed that with the multi-objective function, both the absolute scores and supervised signals are taken into consideration. Another example is Li al. (2023b), who trains a universal retriever with both the wist-wise ranking loss and InfoCNE loss. Both losses show additive effects the model.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Iterative Training Regarding training strategies, most research efforts have centered on fine-tuning single retriever. Wang et al. (2023a) and Li et al. 2023b) instead proposed to iterate the retriever model multiple times. More specifically, the retriever trained in iteration will be employed to retrieve new set of candidates for the subsequent itcration i + 1. Such an iterative training approach allows progressively improving retriever quality by mining better positive and hard negative examples at each iteration.
Diversity Training The Determinantal Point Process model (Alex Kulesz 2012 defines a probability distribution over all the combinations of candidate demonstrations, giving high probability to subsets that contain relevant and diverse items (Levy et al. 2022). It models diversity by incorporating cross-candidate similarity scores, and models similarity via per-candidate relevance score, ie.. a similarity score between a candidate and the test query. In addition to using DPP directly (Levy et al. 2022.. Ye et al. (2023a) also fine-tuned DPP model and demonstrated mcaningful improvements over pure similarity-based methods.
Re-ranker Training It is not uncommon that people adopt a two-stage retriever-reranker architecture for ICL retrieval in the literature further improve the exemplar selection process (Shi et al. 2022) Generally, dual-encoder- based retriever can encode query and candidate documents for fast indexing and searching, but neglect the finer-grained token-level interactions. Cross-encoder-basst reranker, on the other hand, can capture the subtle relationship but is time-consuming. We can benefit from both of these methods by chaining two methods together. In the first stage, a retriever model is used to quickly select the top N examplers to limit the candidate pool of interest, then a reranker reranks the retrieved N examplars and uses the top K exemplars to construct a prompt. Sigmoid cross-cntropy loss is typically used for training the reranker. Lu et al. (2022a) also utilizes a similar structure as the reranker to select the demonstrations from random candidates. The reranker is trained using reinforcement learning.
5.3 Summary
Here, we summarize the advantages and disadvantages of various retriever models. The off- offthh-sshef retrievers are easy to use without any downstream task finetuning and typically demonstrate stronger performance than random demonstrations. One exception is in commonsense reasoning tasks where Zhang et al. (202bb) and Yo et al. (2023a) found that for these tasks, random demonstrations are consistently better than retrieval-based method. Cheng et al. (2023) also show that retrieved demonstrations harm commonsense reasoning and coreference resolution tasks. Among the three categories of off-the-shelf retrievers, sparse retrievers such BM25 more index-effciiett. This feature becomes particularly valuable when dealing with large volumes of demonstrations and limited hardware memory, making BM25 a preferable choice under such circunstances. In contrast, centence-embeddding similarity- based methods and dual-encoder- based retrieval systems, which are trained on language tasks, excel in capturing more semantically focused retricval. Regarding performance, Luo et al. (2023) compared BM25 with dual encoder (GTR) across 5 lasks, and they found that the average performance these two is very similar (within 0.5% difference), and BM25 outperfrreed the dual encoder in some tasks and vice versa. In another study, Ye al. (2023a) observed a similar trend highlighting that single retriever consistently outperforms others across different tasks. Both Rubin et al. (2022) and Li
et al. 2023bb found that BM25 is better than SBERT on semantic parsing tasks, while Li et al. 2023b found that SBERT is better than BM25 on sentiment analysis tasks.
Nevertheless, fine-tuned retrievers demonstrate superior performance compared to their off-thh-shelf counter- parts. The main drawback of fine-tuned retrievers lies in the high cost of obtaining training data. Additionally, the common practice of employing task-specific retricvers complicates the system and limits its generalizability, though there were recent works allempting to address this. For instance, Li et al. (2023b) trained a universal retriever that shows stronger performance than task-specific demonstration retriever most of the tasks.
The choice of training objectives for fine-tuning retricver depends on the specific goals, datasct charactcristics, availability of labeled positive and negative examples, and computational constraints. For general retrieval tasks, when absolute relevance score is important, KL divergence is the ideal solution since is optimized for distribution alignment. If there is a way to clearly differentiate positive examples from negative examples, InfoNCE is computationally cheap and works well, but as it does not emphasis absolute relevance, the ordering of the retrieved documents may be not accurate. List-wise ranking loss optimizes the entire ranking order
14

---

Published in Transactions on Machine Learning Research (10/2024)
directly, but is the most computationally expensive. With these trade-offs, there is no clear winner, we often see in literature that empirical results serve as the primary guide for selecting which training objective to use, and ensembling training objectives proves to be effective (Wang et al. 2023a Li et al. 2023b).
6 Applications
The effectiveness of retrieval-based ICL has been showed in four categories of tasks: 1) natural language understanding, 2) reasoning, 3) knowledge-basee QA, and 4) Text generation. We discuss each category below.
Natural language understanding tasks that benefit from RetICL include sentiment analysis (SA) (Socher et al,
2013
Zhang et al.]
2015
Go al. 2009); paraphrase delection (PD) (Dolan al. 2004 Zhang al. 2019] 9
reading comprehension
RCC Rajpurkar et al.
2016
Khashabi et al. 2018 Clark et al. 2019 Khashabi
et al. 2018 Clark et al.
2019
Mihaylov et al.
2018), and natural language inference (NLI) (Williams et al.
2018
Wang et al. 2018
Bowman et al. 2015a
De Marneffe et al. 2008 Specially, RetICL shows noticeable
improvements on SA and NLT tasks (Liu et al.
2022 Ye et al.
2023a
Reasoning tasks that benefit from RetICL include hathamatiaal reasoning (Cobee et al. 2021 Lu et al.
2022a
Ling al.] 2017). commonsense reasoning (CSR) (Talmor et al., 2019 Zellers al.) 2019 Bisk et al. 2020
Roemmele et al.
2011 2011),and and ethical Reasoning (Jiang al. 2021 Such tasks are usually accompanied by
CoT. Luo et al.
2023 demonstrate that retrieved demonstrations can be combined with the CoT technique
to further enhance performance in mathematical reasoning tasks, showing that RetICL improves on top of COT. Zhang et al. (202bb) emphasize the importance of diversity for these tasks. The iterative retrieval strategy, as noted by (Scarlatos & Lan 2023). shows the most significant improvement in mathematical reasoning. Conversely, some studies have found that retrieval-based demonstrations perform worse than random demonstrations commonsense reasoning tasks, such as CMSQA.
In Knowledge-based QA, external knowledge is required to answer the question (Berant et al. 2013 Kwiatkowski et al. 2019 Joshi et al. 2017 Clark et al.) 2018 To tackle such tasks, the state-of- art systems usually retrieve relevant passages that might contain the answer to the question, and then feed such passages and questions together to a language model to generate the answer. Liu et al. (2022) shows that using sasieval-based ICL (sentence semantic similarity-basse retriever with GPT-3) is almost comparable to a fine-tuned method. Ye al. 2023a) shows that BM25 achieve 10+% improvement on open-domain QA.
Text generation tasks such as code generation (CodeGen) (Zelle & Mooney
1996 Lin al. 2018 semantic
parsing (SP) (Wolfson et al. 2020 Li et al.) 2021
Andreas et al.) 2020)
text-to-SQL (Shi et al. 2022
fable-to-text (Table2Text) generation (Parikh et al.
2020), Data-to-Teess
(D2T) (Nan et al.) 2021 Dušek
ct al. 2019) benefit from RetICL (Pocsia et al.; 2021
Rubin et al. 2022
Hu et al.) 2022 Hongjin et al.
2022 Shi et al. 2022 Ye et al. 2023a Agrawal et al.
2023). Rubin et al.
(2022) shows that the retrieved
demonstrations significantly outperform random demonstrations (e.g. BM25 is 25+% better than random,
and EPR is 30% better than random).
A Apart from different types of tasks, Hongjin et al. (002 shows that in scenarios with limited training data, RetICL outperforms fine-tuning a model such sparse data. Furthermore, leveraging data from high-resoure domain can enhance performance in low-resource domain, seen in cross-lingual contexts Shi
et al.
2022 Nie et al. 2022
Cheng et al. 2023).
7 Discussion
7.1
Guidelines for Choosing the Methods
Based on the surveyed works, we summarize the following guidelines for choosing a RetICL recipe.
Similarity and diversity: While simple similarity-based retrieval is often sufficient, if the retricval corpus contains many demonstrations that are similar to each other, the retrieved demonstrations may become homogeneous. If the demonstrations be binned into sub-groups, could clustering
15

---

Published in Transactions on Machine Learning Research (10/2024)
retrieval
Li et al. 2022a
Zhang al. 2022b Li & Qiu 2023b).
Alternatively, detrimental point
processes (DPP) can be used if the distance between two demonstrations can be easily computed Ye
et al. 2023a Levy et al. 2022).
Number of demonstrations: As mentioned in Scction 2.2 LLMs benefits from having more demonstrations, but has a diminishing return. To get the most gain, one can simply add as many demonstrations as the LLM context size allows. All retrieval strategies in Section 3.2 can select a dynamic number of examples (e.g. by over-retrieving and removing ones that overflow the context. size).
The retrieval corpus should contain demonstrations that are close to the target task. While some works found success at using mix-domain or cross-domain demonstrations (Section 3.3 the retricval corpus still often contains demonstrations that are relatively similar to the larget lask; e.g. examples from different datasets of a similar task (Wang et al. 2023a or different. languages of the same task (Shi al. 2022).
Retrieval corpus content:
Correctness of the demonstrations: LLM of different. sizes learn from the demonstrations in different ways: small LLMs use the demonstrations for task recognition and rely more its pretraining prior for deriving the answer, while large LLMs can learn the label mappings from the demonstrations (Kossen et al. 2024 Wei et. al. 2023). Tt is thus more important to ensure the correctness of the demonstrations when using large LLMs.
Demonstration retriever: As mentioned in Section 5.3 there no clear consensus on the best off-the-shelf retriever. One recommendation would be to try BM25, which is simple but highly generalizable, as well the most convenient model-based retriever such as SBERT GTR. For retriever trained specifically for demonstrations, one could first try the task-generic demonstration retriever Li et al. 2023b
7.2 Future Directions
Retrieve Demonstrations From Raw Text Much research assumes the availability of annotated samples that can be utilized as a retricval corpus. Yet, when faced with a novel task, it often the case that no such training dataset exists. While there are preliminary efforts lo create pseudo demonstrations from open-ended corpora like Wikipedia (Lyu al. 2022), the proposed method is restricted classification tasks and the label to the demonstrations are randomly assigned. A potential approach to obtain pseudo demonstrations for generation tasks is the approach of Wan et al. (2023). where they assume a set of unlabelled queries available (without ground truth labels), and use LLMs to generate chain-of-thoughts and answers and then apply self- celf-consistency (Wang et al. 2022c to select high-quality demonstrations to form pseudo demonstrations pool. Employing this mothod of generating answers with sentences retrieved from a free-form corpus could potentially create high-quality pseudo demonstrations.
Demonstration Retriever Design Current demonstration retriever architectures are not significantly different from those used for raw text, with most efforts focusing on constructing the downstream training data. An open question remains: "How can retrievers be designed specifically for demonstrations rather than raw text?" A potential direction could involve training task-aware retriever with instructions Asai et al. (023)), such that the fine-tuned retriever learns to tailor itself to instructional content. The instruction retriever should not only understand the semantic instructions but also retrieve the most relevant demonstrations from the database that follow these instructions.
Retriever Training Methods In Section we explore various methods for training a retriever to search demonstrations. These methods largely depend on using an LLM to identify positive and negative demonstrations for a given question. This approach, while innovative, comes with significant computational demands. Morcover, the ambiguity in choosing what constitutes a positive or negative demonstration raises concerns about the quality of the training data for the retriever (Hashimoto al.) 2023).. Addressing these challenges is crucial for the development of more efficient and reliable retriever training methods.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Active Demonstration Retrieval Much of the current research is based on static framework where the retrieval corpus remains constant. In practical situations, input distributions may change over time and models may come across new instances. In these cases, one may like to keep updating the retrieval corpus based on the new incoming queries, but since the labels for the new samples may not be available, a selection strategy is needed to select. a representative subset of the incoming examples for annolation so they can be added to the retrieval corpus. This problem is reminiscent of the well-studied active learning problem. Zhang et al. (2022a) have studied how to actively select examples with unlabelled data, however, this study is not based on the retrieval setting. The combination of active learning and RetICL can be an interesting future direction.
Retrieved Demonstrations for Small LM Most existing work focuses on large language models (LLMs) with more than 100 billion parameters, which are often used as off-the-shelf models without adaptation to downstream tasks. In contrast, small LMs not only offer the advantage of inference efficiency but also are easier to fine-tune compared to LLMs. This presents an opportunily to optimize both the retriever and the LM simultancously, similar to end-to-end retrieval-angmented generation training methods (Guu al. 2020) Lewis et al., 2020). By tailoring retrievers to produce demonstrations that align closely with the capabilities of small LMs, and fine-tuning these small LMs to adapt to in-context learning tasks, can potentially enhance their performance on specific tasks.
Theoretical Understanding of Why Are Similar Demonstrations Better Demonstrations? While extensive research has demonstrated that similar demonstrations are more effective than random ones in in- context lcarning (ICL), the underlying reasons for this effectiveness remain unclear. A parallel line of research focuses on understanding the mechanisms behind ICL's success. Several hypotheses have been proposed, including: Bayesian inference (e.g. locate previous learned concepts) (Xie et al.). inductive heads (OIsson et al. 2022), implicit gradient descent (Von Oswald et al., 2023 Dai et al.). These research directions may provide the foundation for explaining why similar demonstrations outperform random ones in ICL scenarios.
Many-Shot Prompting With the rapid increase in the context length of the frontier LLMs (Reid et al. 2024| Anthropic 2024), a new ICL paradigm has emerged where instead of providing only few demonstrations to the models, one can provide many shots, It has been observed that this approach can provide substantial boost in performance (Agarwal et al. 2024 Jiang et al. 2024 Anil ct al., 2024), and also alleviate some of the issues with few-shot ICL such as sensitivity to the order (Bertsch et al. 2024).. However, the increased context size in many-shot ICL introduces an extra inference cost. An interesting area for future research is to find ways of combining many-shot ICL and RetICL to get similar predictive performance with a lower inference cost.
Fine-tuning For RetICL Few-shot Learning Most of the existing research utilizes frozen LLM as the few-shot learner in the RetICL framework. As demonstrated by Gao et al. (2021a) and Izacard et al. (2023). fine-tuning LLMs on a few-shot learning task can be a promising approach to enhancing LLM performance during inference. An intriguing avenue could involve adapting the finc-tuning stratogy from open-domain question answering models to RetICL (Lewis et al., 2020 Guu et al. 2020
RetICL for Vision and Language Models Vision and language models (VL) have demonstrated
proficiency as few-shot learners (Alayrac et al.] 2022. Awadalla al., 2023 Li et al. 2023ab. and rescarchers
have integrated retrieval augmented generation methods with VL. models (Luo et al. 2021 Gao al. 2022
Yasunaga et al. 2023). In lieu of employing random demonstrations, Yang et al. (0024) leverages retriever
to select demonstrations for image captioning tasks. Additionally, |Peng et al. [2023b] propose an ICD-LM
to generate demonstrations for given query, with each demonstration being represented by token. The effectiveness of this mothod is evidenced by its performance on image captioning and vision question answering
tasks. Despite the increasing significance of vision and language generation models in real-world applications, research on RetICL for VL models remains relatively underexplored.
Many-Shot ICL: With the rapid increase in the context length of the frontier LLMs (Reid et al. 2024
Anthropic 2024), new ICL, paradigm has emerged where instead of providing only few demonstrations to
17

---

Published in Transactions on Machine Learning Research (10/2024)
the models, one can provide many shots. It has been observed that this approach can provide substantial boost in performance (Agarwal et al. 2024 Jiang et al. 2024 Anil et al., 2024 and also alleviate some of the issues with few-shot ICL such as sensitivity to the order (Bertsch et al., 2024). However, the increased context size in many-shot ICL introduces an extra inference cost. An interesting area for future research is to find ways of combining many-shot ICL and Ret[CL to get similar predictive performance with a lower inference cost.
8 Conclusion
This survey concentrates on few-shots In-Context Learning (ICL) using retrieved examples for large language models, key aspect of Aetrieval-Angmenter Generation (RAG). We outline various retrieval strategies, diverse retrieval models, retrieval pools, techniques for training demonstration relrievers, and applications Based on the comprchensive understanding of current trends, we suggest several promising future paths for enhancing the efficacy and functionality of this approach.
References
Abdalghaní Abujabal, Rishiraj Saha Roy, Mohamed Yahya, and Gerhard Weikum. ComQA: community- sourced dataset for complex factoid question answering with paraphrase clusters. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 307 317, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/NDB 1027. 1002
1027. URL htttp://acclarthooogg)
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Ab- bas, Azade Nova, John D Co-Reyes, Eric Chu, et al. Many-shot in-context learning. arXiv preprint arXiu:2404.11018 2024.
Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. In-context examples selection for machine translation. arXiv preprint ar Xrxi::2212.02: 02437,, 2022.
Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. In-context examples selection for machine translation. In Findings of the Association for Computational Linguisties: ACL 2023, pp. 8857 8873, 2023.
Jean-Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 26-23716-23736, 2022.
Ben Taskar Alex Kulesv. Determinantal point processes for machine learning. Foundations and Trends3 in Machine Learning, 5:123 286, 2012.
Jacob Andreas, John Bufe, David Burkett, Charles Chen Jr, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, et al. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics, 8:556 571, 2020.
Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking. Anthropic, April, 2024.
Al Authropic. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 1, 2024.
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. Task-aware retricval with instructions. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 3650 3675, 2023.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, al. Openftamingo: An open-source framework for training large autoregressive vision-language models. Xio preprint arXiv:2308.0133 2023.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Lalit R. Bahl, Frederick Jelinek, and Robert L Mercer. A maximum likelihood approach to continuous speech recognition. IEEE transactions on pallern analysis and machine intelligence, (2):179-105 190, 1983.
James Baker, The dragon system-an an overview. IEEE Transactions on Acoustics, speech, and signal Processing, 23(1):24-29, 1975.
James K Baker. Stochastic modeling for aulomatic speech understanding. Readings in Speech Recognition. San Francisco: Morgon Kaufmann Publishers, pp. 297-307, 1990.
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Tiang. Semantic parsing on freebase from question- answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1533 1544, 2013.
Amanda Bertsch, Maor lvgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig, In-context learning with long-context models: An in-depth exploration. arXiv preprinl ar arxiv:2405 200:0400000000 2024.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourtds ATAT Conference Artificial Intelligence, 2020.
Samuel Bowman, Gabor Angeli, Christopher Polts, and Christopher Manning. large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632 642, 2015a.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Longuage Processing (EMNLP), pp. 582-642. 2015b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
Christopher J.C. Burges. From ranknet to lambdarank to lambdamart: An overview. Technical report, Microsoft Research Technical Report MSR-TR-2010--2 2010.
Daniel M. Cer, Mona T. Diab, Eneko Agirre, Iñigo Lopez-Gazpia Gazpio, and Lucia Specia. Semeval-2017 task 1: Scmantic textual similarity multilingual and cross-lingus focused evaluation. CoRR, /////1708.008888 2017. URL http:://axixti http:://axxv.org//// httgg/bb/1708..000
Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, and Zor- nitsa Kozareva. Improving in-context few-shot learning via self-supervised training. arXio preprinl WrXx:22205.01003 2022.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient. fine-tuning of long-context large language models. In The Twelft International Conference on Learning Representations.
Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Denvy Deng, and Qi Zhang. Uprise: Universal prompt retrieval for improving zero-shot evaluation. arXiv preprint Stfe:2303.08111 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction fenercttoo-finetumeed language models. arXiv preprint ar X10:2210. 1 11410 1416, 2022.
19

---

Published in Transactions on Machine Learning Research (10/2024)
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no no questions. arXiv preprinl or Xix:1905 1004,, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. ArXie, abs/ 10550180.00447,, 2018. URL
http:/ppp..emaniiccchhhlaalt (ttp:c/coo..a.oooo//ropaaaiiaaa- 3922816
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. 2021.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furn Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models.
Bhavana Dalvi, Oyvind Tafjord, and Peter Clark. Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement. arXiv preprint arXiv:2204.1307/ 2022.
Marie-Catherine De Marneffe, Anna N Rafferty, and Christopher D Manning. Finding contradictions in text. In Proceedings of acl-08: Hlll, pp. 1039 1047, 2008.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 14314, 2023.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 0ato:181004400. 2018.
Bill Dolan, Chris Quirk, and Chris Brockett. Unsupervised construction large paraphrase corpora: exploiting massively parallel news sources. In Proceedings of the 20th international conference on Compulational Linguistics, pp. 350-es, 2004.
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language models. In The Fleventh International Conference on Learning Representations, 2022.
Andrew Drozdov, Honglei Zhuang, Zhuyun Dai, Zhen Qin, Razieh Rahimi, Xuanhui Wang, Dana Alon, Mohit lyyer, Andrew McCallum, Donald Metzler, et al. Parade: Passage ranking using demonstrations with large language models. arXiv preprint arXiv:2310. 14408, 2023.
Ondrej Dušck, David M Howcroft, and Verena Ricser. Somantic noise matters for neural natural language generation. In Proceedings of the 12th International Conference on Natural Language Generalion, pp- 421 021-426,140 2019.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity prompting for multi-step reasoning. arXie preprint 2000:2210000720. 2022.
Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. Transform- retrieve-gonerate: Natural language-centric ontside-knowleddg visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5067 5077, June 2022.
Lingyu Gao, Aditi Chandhary, Krishna Srinivasan, Kazuma Hashimoto, Karthik Raman, and Michael Ben- dersky. Ambiguity-aware in-context learning with large language models. ar preprint arXi:22009.020 079000 2023.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 3816-3830, 2021a.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings.
In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 6894 -9100,
November 2021b.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context?
case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583- 2022.
Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009, 2009.
Hila Gonen, Srini Tyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 10136 10148, 2023.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.000 20.7552,, 2023.
Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. A deep relevance matching model for ad-hoc retrieval. In Proceadings of the 25th ACM international on conference on informalion and knowledge management, pp. 55 64, 2016.
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A Smith, and Luke Zettlemoyer. Demix layers: Disentangling domains for modular language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5557-5576, 2022.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938 PMLR, 2020.
Kazuma Hashimoto, Karthik Raman, and Michael Bendersky. Take one step at a time to know incremental utility of demonstration: An analysis on reranking for few-shot in-context learning. ar preprint arXhw:2311.09619 2023.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty- Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Sepp Hochreiter and Jürgen Schmidluber. Long short-term memory. Neural computation, 988):1735-1780, 1997.
SU Hongjin, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zeutlemoyer, Noah A Smith, et al. Selective annotation makes language models better few-shot. learners. In The Eleventh International Conference on Learning Representations, 2022.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Yushi Hu, Hhua-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A Smith, and Mari Ostendorf. In-context learning for few-shot dialogue state tracking. In Findings of the Association for Computational Länguisties: FMNLP 2022, pp. 2627-2643, 2022.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yie Armand Joulin, Scbastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1-43, 2023.
Fred Jelinek. Self-organized language modeling for speech recognition. Readings in speech recognition, pp. 550-506, 1990.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Frederick Jelinek, Lalit Bahl, and Robert Mercer. Design of a linguistic statistical decoder for the recognition of continuous speech. IEFE Transactions on Information Theory, (3):2500- 1975.
Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, et al. Can machines learn morality? the delphi experiment. Xiviv preprint ar XxX::2110007754, 2021.
Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H Chen, and Andrew Y Ng. Many-shot in-context learning in multimodal foundation models. ar preprint NNv:2405.00999 2024.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423 438, 2020.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeling of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1601-1611, 2017.
James H. Jurafsky, Dan; Martin. N-gram language models. In Speech and Language Processing (3rd ed.). 2021.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Melhods in Natural Longuage Processing (EMNLP), pp. 6769 6781, 2020.
Mehran Kazemi, Sid Mittal, and Deepak Ramachandran. Understanding finetuning for factual knowledge extraction from langnage models. Xii preprint arXiv:2301.1 110390011111100 11293, 2023a.
Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, and Deepak Ramachandran, Lambada: Backward chaining for automated reasoning in natural language. In ACI, 2023b.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface:a challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapler of the Association for Compulational Linguislies (NAACT.) 2018.
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. CoRR, abs/2004.12882,, 2020. URL http:://arrv...oraaa///00.... 12832
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstraes-cardpprriit Composing retricval and language models for knowledge-nntensiee nlp. Xiviv preprint Xiv:22221 14024, 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2022. URL htlps://arzin. org/abs/ 1 11916.
Jannik Kossen, Yarin Gal, and Tom Rainforth. In-context learning learns label relationships but is not conventional learning. In The Twelfth International Conference on Learning Representations, 2024.
Alex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning. Foundations and TrendsS in Machine Learning, 5(2 (2-3)))2332200 286, 2012.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Lönguistics, 7:453-466, 2019.
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Mcasuring faithfulness in chain-of th-tughhttt reasoning. XiXio preprint or 2rX::2307..3370 2023.
22

---

Published in Transactions on Machine Learning Research (10/2024)
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-effcient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3054 2021.
Itay Levy, Ben Bogin, and Jonathan Berant. Diverse demonstrations improve in-context compositional generalization. arXiv preprint orXie:2212.06800 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeene pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13/000 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. aetrieval-angmeeneed generation for knowledge- intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-944 9944,, 2020.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843 3857, 2022.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprinl Xixvv 05425, 2023a.
Haoran Li, Abhinav Arora, Shnohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mchdad. Mtop: A comprehensive multilingual lask-oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2950 2962, 2021.
Junlong Li, Zhuosheng Zhang, and Hai Zhao. Self-prompting large language models for zero-shot open-domain qa. arXic preprint 2rxiv:2212. 2022a.
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context Ilms struggle with long in-context learning. CoRR, 2024.
Xiaonan Li and Xipeng Qiu. Finding supporting examples for in-context learning. ar preprinl arXiv:2302.1355 2023a.
Xiaonan Li and Xipeng Qiu. Mot: Pre-thinking and recalling enable chatgpt to self-improve with memory-of- thoughts. arXie preprint arXiv:2305. 051111 2023b.
Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. Unified demonstration retriever for in-context learning. arXie preprint arXiv:2305. 04320,, 2023b.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making large language models better reasoners with step-aware verifier. arXie preprint arXiv:2206. 02336, 2022b.
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. Birds have four legs?! numersense: Probing numerical commonsense knowledge of pre-trained language models. arXiv preprint ar xrxx:2205500068 2020.
Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D Ernst. N12bash: corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Longuage Resources and Evaluation (LREC 2018), 2018.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 158 167, 2017.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100-114, 2022.
23

---

Published in Transactions on Machine Learning Research (10/2024)
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint Ar 11692, 2019.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint ArXiv:2301.198 13688, 2023.
Pan Lu, Liang Qiu, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In The Fleventh International Conference on Learning Representations, 2022a.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 8086 8098, 2022b.
Man Luo, Yankai Zeng, Pratyay Banerjoe, and Chitta Baral. Weakly-supervised visual-rctriovarrredeer for knowledge-based question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6417-6431, 2021.
Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y Zhao. Dr. icl: Demonstration-rrtriivvd in-context learning. arXiv preprint arXiv:2305. 14.9005..1411 14128, 2023.
Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajishirzi. Z-icl: Zero-shot in-context learning with pseuub-demmonstrationn.. arXiv preprint 2rX::2212.00565,, 2022.
Huan Ma, Changqing Zhang, Yatao Bian, Lemao Liu, Zhirui Zhang, Peilin Zhao, Shu Zhang, Huazhu Fu, Qinghua Hu, and Bingzhe Wu. Fairness-guided few-shot prompting for large language models. arXiv preprint Xrx::22303. 13217, 2023.
Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXie:2201.0000 2022.
AA Markov. Essai d'une recherche statistique sur le texte du roman. Eugène Onëgin", illustrant la liaison des épreuves en chaine", Bulletin de l'Académie Impériale des Sciences de St.-Pétersborrg VI série, 7(3): 153-162, 1913.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.978. 2013.
Tomáš Mikolov et al. Statistical language models based on neural networks. 2012.
Aristides Milios, Siva Reddy, and Dzmitry Bahdanau. In-context learning for text classification with many labels. In Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP, pp. 2023.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannanch Hajishirzi. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2791-2809, 2022a.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint ar0i 0002002.18377. 12837, 2022b.
Bhaskar Mitra, Nick Craswell, et al. An introduction to neural information retrieval. Foundations and Trendso in Information Retrieval, 13(1):1-126, 2018.
24

---

Published in Transactions on Machine Learning Research (10/2024)
Ranjita Naik, Varun Chandrosekaran, Mert Yuksekgonul, Hamid Palangi, and Besmira Nushi. Diversity of thought improves reasoning abilities of large language models. arXiv preprint. arXiv:2310.0700 070008 2023.
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsich, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al. Dart: Open-domain structured data record to text generation. In Proceedings of the 2021 Conference of the North American Chapler of the Association for Computational Linguistics: Human Longuage Technologies, pp. 432 447, 2021.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A humman generated machine reading comprehension dataset. choice, 2640:660, 2016.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Zhao, Yi Luan, Keith B Hall, Ming-W Wei Chang, al. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.07880 2021.
Ercong Nie, Sheng Liang, Helmut Schmid, and Hinrich Schütze. Cross-lingual retrieval augmented prompt for low-resource languages. ArXio, Abs/2221.0096 2022.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. ar areie preprint arXiv:2209. 1 1199,, 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730 27744, 2022.
Jane Pan. What in-context learning "learns" in-context: Disentangling task recognition and task learning. Master's thesis, Princeton University, 2023.
Ankur Parikh, Xuczhi Wang, Scbastian Gchrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. Totto: controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1173 1186, 2020.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derezynski, et al. Rwkv: Reinventing rnns for the transformer era. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.
2023a.
Yingzhe Peng, Xu Yang, Haoxuan Ma, Shuo Xu, Chi Zhang, Yucheng Han, and Hanwang Zhang. Tod-Im: Con- figuring vision-language in-context demonstrations by language modeling. arXiv preprint ar XrXiv:2312.100 10104, 2023b.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wa, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909, 000000 2019.
Gabriel Pocsia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations, 2021.
Aloc Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl / blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485 5551, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 1rXiv:1606.05220 2016.
25

---

Published in Transactions on Machine Learning Research (10/2024)
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint (XX::20030055300 2024.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. Empirical Methods in Natural Language Processing and International Joint Conference on Nutural Language Processing (EMNLP-IJCNLP) pp. 3982 3992, 2019.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trendss in Information Retrieval, 3(4):333-389, 2009.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
Ronald Rosenfeld. Two decades of statistical language modeling: Where do we go from here? Proceedings of the IEEE, 88(8):1270 1270, 2000.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655 2671, 2022.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon. Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. CoRR, abs/2112.01 488, 2021. URL
bttps://arxii..rrg/bbsss /b12//////11118
Justyna Sarzynska- Wawer, Aleksander Wawer, Aleksandra Pawlak, Julia Szymanowska, Izabela Stefaniak, Michal Jarkiewicz, and Lukass Okruszek. Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research, 304:114135, 2021.
Alexander Scarlatos and Andrew Lan. Reticl: Sequential retrieval of in-context examples with reinforcement learning. ar preprint arXiv:2305.14500 2023.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technicol journal, 27 (3):379-422 1948.
Peng Shi, Rui Zhang, He Bai, and Jimmy Lin. XricI: Cross laass-ingual vetrieval-auggenner in-context learning for cross-lingual text-to-sql semantic parsing. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5248 2259, 2022.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. ar preprint ar Xio:2210..500 2020.
Suzanna Sia and Kevin Duh. In-context lcarning as maintaining coherency: A study of on-the-fly machine translation using large language models. In Proceedings of Machine Translation Summil XIX, Vol. 1: Research Track; pp. 173-185, 2023.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionaliit over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pp. 1631-1642, 2013.
Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sungdong Kim, and Jaewoo Kang. Can language models be biomedical knowledge bases? arXiv preprint arXiv:2100.07154 2021.
Mirac Suzgun, Nathan Scales, Nathanacl Schärli, Sebastian Gchrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, al. Challenging big- bench tasks and whether chain-of- thought solve them. ar Xio preprint 2rXim:2210.099 2022.
26

---

Published in Transactions on Machine Learning Research (10/2024)
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge largeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4149-4158, 2019.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini rescarch and technology. arXiv preprint arXiv:2403.088 08229, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint XXxu:3307009258 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintser, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp. 1511443 PMLR, 2023.
Xingchen Wan, Ruoxi Sun, Hanjun Dai, Sercan Arik, and Tomas Pfister. Better zero-shot reasoning with sclf-adaptive prompting. arXi preprint arXiv:2305. 14 14106, 2023.
Alex Wang, Amanprect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-Lask benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackbonNT.P: Analyzing and Interpreting Neural Networks for NLP, pp. 353-35- 353-,55, 2018.
Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. To- wards understanding chain-of of- prompting: An empirical study of what matters. arXiv preprint 211: 2.1001,,,.000,,,, 2022a.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, 2022b.
Liang Wang, Nan Yang, and Furu Wei. Learning to retrieve in-context examples for large language models. Xixiv preprint Niv:2907.07164, 2023a.
Xinyi Wang, Wanrong Zhu, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. arXiv preprint 11X11:2301.11116, 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint ar0i 11903, 2022.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint Xix:2303.008844 2023.
Orion Weller, Marc Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. according to...' prompting language models improves quoting from pre-training data. arXiv preprint ar 13952,, 2023.
27

---

Published in Transactions on Machine Learning Research (10/2024)
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chopler of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, 2018
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8:183-190 2020.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2021.
Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. End-to-end neural ad-hoe ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval, pp. 55 64, 2017.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. ColR,, abs/2010.11934, 2020. URL https:///arxa org/ abs/ 2010. 11934
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models optimizers. ar preprint arXiv:2309.083 03409, 2023.
Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. Exploring diverse in-context. configurations for image captioning. Advances in Newral Information Processing Systems, 36, 2024.
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernández Ábrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Multilingual universal sentence encoder for semantic retrieval. CoRR, abs/ 1.907.04307, 2019. URL http:://axxxii wrgg/b////00.....///////
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-Tau Yih. Retrieevl--uugmeneed multimodal language modeling. In International Conference on Machine Learning, pp. 39769. PMLR, 2023.
Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong, Compositional exemplars for in-context learning. arXiv preprint 2rXiv:2302.05698, 2023a.
Seonghyeon Ye, Hyeonbin Hwang, Sohee Yang, Hyeongu Yun, Yireun Kim, and Minjoon Seo. In-context instruction learning, arXiv preprint arXiv: 14691, 2023b.
Xue Ying. An overview of overfitting and its solutions. In Journal of physics: Conference series, volume 1168, pp. 022022. 1OP Publishing, 2019.
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. In The Fleventh International Conference on Learning Representations, 2023.
Quan Yuan, Mehran Kazemi, Xin Xu, Isaac Noble, Vaiva Imbrasaite, and Deepak Ramachandran. Tasklama: probing the complex task understanding of language models. arXiv preprint arXiv:2308. 15299, 2023.
John M Zelle and Raymond J Mooney. Tearning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pp. 1050-1055, 1996.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4806 2019.
28

---

Published in Transactions on Machine Learning Research (10/2024)
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofci Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. ar preprinl 2rXiv:2308.1077 2023.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neurol information processing systems, 28, 2015.
Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9134-914S, 2022a.
Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase Adversaries from Word Scrambling. In Proc. of NAACL, 2019.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The Fleventh International Conference Learning Represenlations, 2022b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint (rXiv:2303.189) 2023.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pp. 12697 12706. PMLR, 2021.
29


Reviews:
[{"review_id": "IJDCij8jUZ", "reviewer": "Reviewer_sZKR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I found the paper well-structured and well-written. Basically, I do not see any major weaknesses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a survey of in-context learning with retrieved demonstration. This field is concerned with performing in-context learning, in which, for a given query, a tailored set of examples is loaded. \n\nThe paper conceptualizes in-context learning (Sec 2), describing what drives the performance (Sec 2.2). Further, it introduces the setting of in-context learning with retrieved demonstration (RetICL) in Sec 3, describing its basic blocks: the retrieval objectives and strategies (Sec 3.1/Sec 3.2) and possible settings of retrieval corpus (Sec 3.3). The most important part of retrievers to be used with the discussion of pros and cons is contained in Section 4 and Section 5. In the subsequent Section 6, the paper discusses applications separately for NLP, reasoning, knowledge-based QA, and text generation. Finally, it is concluded with a discussion of open questions and future research directions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Perhaps it is worth explaining the relation of RetICL to RAG. \n2. It is highly subjective, but I'd consider adding even more questions, perhaps even speculations, in the Section 7", "broader_impact_concerns": "non", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yWjZkpvLsh", "reviewer": "Reviewer_m6ZC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- ICL is a fast-growing topic, so this survey is quite timely.\n- The categorization is clear, with works cited falling along the lines drawn by the paper.\n\n## Weaknesses:\n\nI will go into detail in the points below. At a high level I believe that this survey makes a number of claims throughout the text that are not well-substantiated by citations, which makes it a less than ideal entry point to the field. In addition I'm not totally certain why it makes sense to focus on such a specific topic as RetICL, though this isn't major to my review as the TMLR guidelines suggest against judging significance.\n\n**Missing citations:** often, I feel that many claims are stated either without justification, or justified much later in the text far from the original claim. Examples:\n  - \"Furthermore, ICL avoids common issues with fine-tuning, such as overfitting\" as far as I understand the works cited after this claim just say that fine-tuning has a tendency to overfit, not that ICL fixes the issue\n  - \"The effectiveness of such demonstrations is influenced by factors such as the quality, quantity, and ordering of the demonstrations\" this is definitely discussed later, but I believe something should be cited at this point\n  - \"This method has not only consistently outshined approaches relying on... influencing factors\" same as above this should probably have citations given it is such a strong claim and we the readers do not yet know much about the RetICL subfield\n  - In the beginning of Section 2, \"Language models (LMs)\" could probably have more references, I think the goal is to be comprehensive, but I am willing to accept another perspective here\n  - \"Early LMs, which were mostly based on N-gram models... recent context\" to cite only one object here, which is a survey chapter itself, doesn't seem right to me. I know that this topic is somewhat tangential to the main topic of the survey, but it seems like the history of the development is missing.\n  - Similar to the above, after \"neural approaches to language modeling gained more popularity\" there are again only two citations for architectures separated by 20 years. There was a lot of development during that time and it seems a bit wrong for the text to ignore that.\n  - \"Few-shot learning is a remarkable capability... not trained on such data\" I believe this really is a claim, and as such should be substantiated, though it is difficult to do so without looking at the entire dataset. The reason I disagree with this point is that it's possible that there are significant portions of the internet devoted to these repetitive demonstrations (for example math word problems), so in particular if an LLM is tested on math word problems then the capability is in-domain. I would think rather the power of few-shot learning is not that it is out of distribution per se, but rather that it applies quite generally.\n  - \"While the size of the context has been increasing over time with newer LLMS\" it would be nice to have a citation here.\n  - \"Similarity involves selecting demonstrations... language similarity... structural aspects... or other criteria\" these are described in detail later, but to be comprehensive it would be good to include the citations here.\n  - the \"Iterative Retrieval\" heading relies entirely on Scarlatos and Lan, 2023, are there no other citations that the text might reference?\n  - \"To achieve... demonstration retrievers.\" cite here\n  - \"between candidate demonstration embeddings... demonstrations\" cite here\n  - \"initial research efforts focused on a heuristic... for this task\" cite here\n  - \"Due to its sole reliance on term frequency... certain instances\" citations needed\n  - \"A rich collection... in the literature.\" citations needed here\n  - \"Text generation tasks that benefit... Data-to-Text.\" citations needed here to say that RetICL actually helps, beyond just citations for the tasks. The entire point of this survey is to promote and explain RetICL, so there should be ample evidence for that.\n  - \"There are some hypotheses... implicit gradient descent\" this requires citations for the hypotheses and specific papers.\n\n**Missing analysis:** There also seems to be a missed opportunity to make a more specific analysis in many cases:\n- \"While there might be other differences... training set\" I think this is a little vague, it would be nice if the text exposed the actual differences between the two works and then attempted to make a more specific suggestion\n- Under \"Mix-Domain\": \"In the mix domain setting... from other tasks\" the text does not present any analysis as to whether this mix-domain retrieval helps or hurts. It would be useful if such discussion were included.\n- \"Cross-domain\": similar to above, the end of the paragraph does not discuss whether the referenced methods are helpful or hurtful, or what specific drawbacks/benefits are, it just mentions their existence.\n- \"Free Form Corpus\": similar to the above two points, it would be nice to have some discussion on the tradeoffs between settings that are seen in the actual cited works. It's unclear what the effect of these different settings are, and the survey should expose some specific points.\n- I believe the discussions in Section 7 make suggestions that are much too general. In particular \"Is there a potential... tasks?\" seems very straightforward to answer no for the simple reason that one might write rules for very closed domains. I would hope that there would be more insight on how the retriever might need to be designed specifically around *demonstrations* rather than raw text.\n- Similar to above, \"Examining and improving the ICL capabilities... direction to explore\" does not have any suggestions or opinions on what is missing, which I might hope that such a survey would convey.\n- \"Such tasks are usually accompanied by CoT... reasoning task.\" This makes it seem like it is hard to disambiguate the effect of retrieval, if so what conclusion can we really draw from the importance of RetICL?\n- At the end of the paragraph labeled \"Pretrained Dual Encoder\" it would be nice again to include discussion on the practical benefits/drawbacks of these specific encoders for RetICL, or something closer to the eventual setting.\n- In the \"Diversity of Demonstrations\" heading, the methods are presented without discussion as to the benefit or drawback of the approaches, so I don't understand if this is a line worth pursuing or not.\n\n**Other points:**\n- Table 1 takes up more than a page, but I don't really understand how it is helpful for distilling the information. It presents a categorization, but because the organization is chronological it's not really clear what should be compared between methods. One option would be to carve up this table into the sections that are later described in the text, another might be to have separate tables for separate subsections. It would be nice if the caption were to highlight any specific trends we the readers are supposed to take.\n- \"One-hoc Retrieval\" this is a strange way to phrase? Perhaps \"One-step\" or \"Independent\" might describe what the paper is aiming for? In addition, this heading contains no references, it might just be nice to have a few for comparison with later headings\n- \"Naik et al. 2023 propose... solving the problem\" I do not understand what this particular line means. It would be helpful to rewrite somehow, though as I don't understand the intention I'm not sure what I can say constructively.\n- In Section 2.1 it would be nice to number the centered demonstrations of prompt techniques. Then the 3rd set can reference the 2nd directly. It would also be helpful to bold \"[Task Instructions].\" I missed it initially on first glance and did not understand why both were included.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey focuses on retrieval-augmented in-context learning (ICL) in large language models, where demonstrations are retrieved from some existing dataset in order to improve the ICL process. The survey categorizes and describes a number of related work along axes such as choice of retrieval dataset or choice of retriever. It then makes some suggestions for future work in the RetICL area.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see Weaknesses above for specific comments. The most critical points for acceptance are the additional citations to fill in many parts of the writing, as well as additional analysis, especially in the final section.", "broader_impact_concerns": "I don't have any particular concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eqv9bOY2ln", "reviewer": "Reviewer_8UR5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is quite comprehensive, covering a lot of papers within this space\n- The paper is well organized, allowing for a reader to look at what aspects are important to them\n\nWeaknesses\n- I find survey papers most helpful when they can synthesize viewpoints about the field and consider the connections between different works. This paper gave great summaries of all the contributions, but I do not feel like I was left with a greater sense of understanding of how all the works interact with each other. This is partially addressed in places like Section 5.3, but in my opinion, this could be a bigger emphasis.\n- I believe this paper would be useful to a practitioner who would like to know the methods used for in-context demonstration retrieval. Its harder for me to see which communities outside of this group would benefit from the aggregation of summaries of methods in the current manuscript.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper summarizes how to retrieve examples for in-context learning. It starts by reviewing in-context learning and demonstration retrieval, summarizing the main approaches in retrieval algorithms, and what tasks one would apply this framework to in practice.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. [Major] I hope for a better guide on which method to pick for practical use cases and how different losses on fine-tuning affect the downstream model. In my opinion, this would be the main practical relevance of this paper.\n2. The following works might help contextualize the effect of demonstrations on in-context learning: https://arxiv.org/abs/2303.03846, https://arxiv.org/abs/2305.09731\n3. Though more recent than your submission, these works discuss using larger context windows to fit more in-context learning examples which would remove the need for selection of examples: https://cdn.sanity.io/files/4zrzovbb/website/af5633c94ed2beb282f6a53c595eb437e8e7b630.pdf, https://arxiv.org/abs/2405.00200\n4. General formatting errors throughout the document. For example, SBERT citation in Section 3.2, numbering of application domains at the start of Section 6...", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are in agreement that the paper is comprehensive, cleanly organized, and well-written. I therefore recommend survey certification for this work.
Decision: Accept (poster)