Venue: NAACL 2024

# Take One Step at a Time to Know Incremental Utility of Demonstration:

An Analysis on Reranking for Few-Shot In-Context Learning

 Kazuma Hashimoto  Karthik Raman  Michael Bendersky

Google Research, Mountain View

{kazumah, karthikraman, bemike}@google.com

###### Abstract

In-Context Learning (ICL) is an emergent capability of Large Language Models (LLMs). Only a few demonstrations enable LLMs to be used as blackbox for new tasks. Previous studies have shown that using LLMs' outputs as labels is effective in training models to select demonstrations. Such a label is expected to estimate utility of a demonstration in ICL; however, it has not been well understood how different labeling strategies affect results on target tasks. This paper presents an analysis on different utility functions by focusing on LLMs' output probability given ground-truth output, and task-specific reward given LLMs' prediction. Unlike the previous work, we introduce a novel labeling method, incremental utility, which estimates how much incremental knowledge is brought into the LLMs by a demonstration. We conduct experiments with instruction-tuned LLMs on binary/multi-class classification, segmentation, and translation across Arabic, English, Finnish, Japanese, and Spanish. Our results show that (1) the probability is effective when the probability values are distributed across the whole value range (on the classification tasks), and (2) the downstream metric is more robust when nuanced reward values are provided with long outputs (on the segmentation and translation tasks). We then show that the proposed incremental utility further helps ICL by contrasting how the LLMs perform with and without the demonstrations.

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3973–3990

June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Recent advances of Large Language Models (LLMs) [5, 8, 1] have been pushing the field of Natural Language Processing (NLP) to the next level in many different aspects. A notable capability of LLMs is few-shot _In-Context Learning (ICL)_, which uses only a few _demonstrations_ (i.e., input-output pairs) to perform new tasks without finentuning [2, 10].

A crucial research topic is demonstration selection for ICL. Liu et al. (2022) proposed retrieval-based ICL, and Rubin et al. (2022) have shown that further finentuing retrieval models is effective [11, 12, 13]. However, it is not conclusive how to train the selection models in several aspects: training labels, objectives, and overall pipelines.

This paper focuses on the training labels of the selection models. First, we use two types of values to estimate utility of a demonstration for an input: (1) an LLM's output probability of generating the ground-truth output (as in the previous work), and (2) a task-specific reward function given the LLM's prediction (as in reward optimization [1]). Next, we introduce a novel method to estimate _incremental utility_ by contrasting the 1-shot ICL performance and the 0-shot performance (Figure 1). We expect the incremental utility to estimate how much incremental knowledge is brought by the demonstration, assuming that the LLMs have a capability of 0-shot prediction [2].

We conduct experiments in a standard retriveral-reranking framework with instruction-tuned LLMs. The evaluation tasks are binary/multi-class classification, segmentation, and translation across five

Figure 1: Incremental utility of a demonstration.



languages: Arabic, English, Finnish, Japanese, and Spanish. Our results and analysis show the following insights:

* the output probability of generating the ground-truth output is indeed good proxy for the utility of a demonstration, especially when the probability values are distributed across the whole value range with short outputs (e.g., classification),
* the downstream metric reward with nuanced values, on the other hand, is more robust for longer outputs (e.g., text generation tasks),
* our proposed incremental utility further improves ICL given contrastive training examples to effectively train the reranking model, and
* constrained retrieval is helpful when the retrieved candidates are imbalanced for training.

## 2 Inference and Training Pipelines

This section describes our experimental design for the comparative study. For a target task, we have a training set \({\cal T}=[e_{1},\ldots,e_{i},\ldots]\), where \(e_{i}=(x^{\prime}_{i},y^{\prime}_{i})\) is an input-output pair; for example, \(y^{\prime}_{i}\) is a class label on a text classification task.

### Inference

Demonstration retrieval and rerankingGiven a new input \(x\), we use a demonstration-selection model \(M(e_{i},x)\) to predict utility of a training example \(e_{i}\) as a demonstration. \(M\) is a cross-attention encoder that takes a concatenation of \(e_{i}\) and \(x\) as an input and then outputs a utility score. We select demonstrations with the \(k\) highest scores. To reduce the search space for the cross-attention encoder, we use an off-the-shelf text retriever to retrieve top-\(n\) training examples, according to similarity between \(x\) and \(x^{\prime}_{i}\). This is motivated by the success of the retrieval-based demonstration Liu et al. (2022) and the standard retrieve-then-rerank framework Zhuang et al. (2023). Hence we call \(M\) a _reranking_ model.

ICL: prompting LLMsWe create a prompt: \(\mathrm{prompt}(x,\mathcal{E}_{x})\), where \(\mathcal{E}_{x}\) is the set of the top-\(k\) demonstrations selected from the \(n\) retrieved examples. The prompt is then fed into an LLM to generate a prediction as a string:

\[y=\operatorname{LLM}(\mathrm{prompt}(x,\mathcal{E}_{x})), \tag{1}\]

where the prediction \(y\) is used for task-specific evaluation.

### Training

We use a utility function \(u(e_{i},x)\) to train the reranking model \(M\); the larger the value is, the more \(e_{i}\) is expected to contribute to ICL for \(x\). To create the training examples for \(M\), we simulate the inference process within the training set; for each \(x^{\prime}_{j}\in{\cal T}\), we retrieve the top-\(n\) set and compute \(u(e,x^{\prime}_{j})\) with \(e\in\mathcal{E}_{x^{\prime}_{j}}\). We ensure that \(e_{j}\) is not in \(\mathcal{E}_{x^{\prime}_{j}}\), and consequently we have \(n|{\cal T}|\) values of \(u(e,x)\) in total. Assuming that the value range of the utility and the reranking score is \([0.0,1.0]\), we use logistic regression for pointwise training Nie et al. (2019); Nogueira and Cho (2019):

\[- u(e_{i},x)\log M(e_{i},x) \tag{2}\] \[- (1-u(e_{i},x))\log(1-M(e_{i},x)).\]

## 3 Utility functions

We define multiple utility functions to investigate the effects on target tasks. Note that the utility functions are used only in the training phase.

### Direct Utility

To compute \(u(e,x^{\prime}_{j})\), it is considered to be the most straightforward to inspect how the LLM behaves given \(\mathrm{prompt}(x^{\prime}_{j},\{e\})\). We describe two utility functions that directly use the _1-shot_ result.

Output Probability (OP)We follow a widely-used approach of using the likelihood of predicting the ground-truth output Li et al. (2023):

\[u_{\rm OP}(e,x^{\prime}_{j})=p(y^{\prime}_{j}|\mathrm{prompt}(x^{\prime}_{j}, \{e\})), \tag{3}\]

assuming that we have access to the probability by feeding the prompt and \(y^{\prime}_{j}\) together into the LLM.1

Footnote 1: Li et al. (2023) normalized the output probability values across possible output candidates for classification and multichoice tasks. We do not apply the normalization, because the computational cost is not negligible when handling a large number of classes.

Downstream Metric (DM)The primary goal is to improve downstream metrics of the target tasks. As in reward optimization Ranzato et al. (2016), we define the following function:

\[u_{\rm DM}(e,x^{\prime}_{j})=R(y^{*},y^{\prime}_{j}), \tag{4}\]

where \(y^{*}=\operatorname{LLM}(\mathrm{prompt}(x^{\prime}_{j},\{e\}))\) is the LLM's prediction, and \(R\) is a pre-defined reward function that is correlated with a task-specific metric; 

Section 4.2 describes a reward function for each dataset.

### Incremental Utility

Another way of interpreting the value of \(u(e,x^{\prime}_{j})\) is how much _incremental_ knowledge \(e\) brings into the LLM. For this, we would like to inspect the LLM's capability of handling \(x^{\prime}_{j}\) in the _0-shot_ inference:

\[\mathrm{LLM}(\mathrm{prompt}(x^{\prime}_{j},\{\})), \tag{5}\]

where \(\{\}\) represents an empty demonstration set. We can use the above-defined utility functions to evaluate the inference result; we use \(u_{0}(x^{\prime}_{j})\) to denote a utility function without \(e\).

For example, let us think about two cases where the use of \(e\) improves the value by \(0.1\): \((u_{0}(x^{\prime}_{j}),u(e,x^{\prime}_{j}))=(0.9,1.0)\) and \((0.0,0.1)\). If we use \(u(e,x^{\prime}_{j})\) alone, the former case has much larger value; however, this could underestimate the latter case's value. We interpret those cases from different angles:

* both have the same value in the absolute difference (\(u(e,x^{\prime}_{j})-u_{0}(x^{\prime}_{j})=0.1\)), or
* the absolute difference is only 10% of \(u(e,x^{\prime}_{j})\) in the former case and 100% in the latter case.

The idea A has been explored in previous work Li and Qiu (2023), while the idea B is expected to be more reasonable with the example cases. One potential caveat is that the ratio-based approach overestimates the value, for example, with \((u_{0}(x^{\prime}_{j}),u(e,x^{\prime}_{j}))=(0.0,0.0001)\).

We propose incremental utility \(r\) to take into account all the aspects discussed above:

\[r(e,x^{\prime}_{j})=\frac{u(e,x^{\prime}_{j})-u_{0}(x^{\prime}_{j})}{\max(u( e,x^{\prime}_{j}),u_{0}(x^{\prime}_{j}))^{\ell}}, \tag{6}\]

where \(\ell\in[0.0,1.0]\) is a hyper-parameter. Here are the following key features we can read:

* \(\ell=0.0\) corresponds to the idea A, \(\ell=1.0\) to B, and the others take a balance between A and B; examples (a), (b), and (c) in Table 1 describe the above-mentioned cases.
* Examples (d) and (e) in Table 1 show the _symmetricity_ of the function.
* The value range of \(r\) is \([-1.0,1.0]\), and \(r<0\) means that \(e\)_negatively_ affects ICL.

Finally, we linearly transform \(r\) into \([0.0,1.0]\) to define the following incremental utility function:

\[u^{+}(e,x^{\prime}_{j})=\frac{r(e,x^{\prime}_{j})+1}{2}. \tag{7}\]

This can be applicable to both the direct utility functions: \(u^{+}_{\rm OP}\) and \(u^{+}_{\rm DM}\).

## 4 Experimental Settings

This section describes our experimental settings; more comprehensive descriptions are in Appendix.

### LLM and Prompt

We use Flan-PaLM 2 (L) Google et al. (2023) as our LLM. This is an instruction-tuned model, and we follow Gao et al. (2023) to design the prompt \(\mathrm{prompt}(x,\mathcal{E}_{x})\) that concatenates a task instruction, demonstrations \(\mathcal{E}_{x}\), and an input \(x\).

### Tasks and Reward Functions

We focus on NLP datasets that are _not_ used in the Flat instruction tuning Chung et al. (2022), across different tasks and languages; Arabic (ar), English (en), Finnish (fi), Japanese (ja), and Spanish (es) are covered. Table 2 shows the dataset statistics, and Table 12 in Appendix shows some examples.

\begin{table}
\begin{tabular}{l|c c|c c||c c c c} \hline  & \multicolumn{1}{c|}{\multirow{2}{*}{\(u_{0}(x^{\prime}_{j})\)}} & \multicolumn{1}{c|}{\multirow{2}{*}{\(u(e,x^{\prime}_{j})\)}} & \multicolumn{1}{c|}{\multirow{2}{*}{\(u(e,x^{\prime}_{j})-u_{0}(x^{\prime}_{j})\)}} & \multicolumn{1}{c|}{\multirow{2}{*}{\(\max(u(e,x^{\prime}_{j}),u_{0}(x^{\prime}_{j}))\)}} & \multicolumn{1}{c}{\multirow{2}{*}{\(\ell=0.0\)}} & \multicolumn{1}{c}{\multirow{2}{*}{\(\ell=0.5\)}} & \multicolumn{1}{c}{\multirow{2}{*}{\(\ell=0.8\)}} & \multicolumn{1}{c}{\multirow{2}{*}{\(\ell=1.0\)}} \\ \hline (a) & 0.0 & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 & 0.316 & 0.631 & 1.0 \\ (b) & 0.9 & 1.0 & 0.1 & 1.0 & 0.1 & 0.1 & 0.1 & 0.1 \\ (c) & 0.0 & 0.0001 & 0.0001 & 0.0001 & 0.0001 & 0.01 & 0.158 & 1.0 \\ (d) & 0.3 & 0.5 & 0.2 & 0.5 & 0.2 & 0.283 & 0.348 & 0.4 \\ (e) & 0.5 & 0.3 & –0.2 & 0.5 & –0.2 & –0.283 & –0.348 & –0.4 \\ \hline \end{tabular}
\end{table}
Table 1: Synthetic examples of \(r(e,x^{\prime}_{j})\) in Equation (6) to illustrate how the incremental utility works.

\begin{table}
\begin{tabular}{l|c|c|c} \hline  & Train & Validation & Test \\ \hline \begin{tabular}{l} ISD (en) \\ \end{tabular} & \begin{tabular}{l} 3,122 \\ 2,792 \\ \end{tabular} & \begin{tabular}{l} 346 \\ 310 \\ \end{tabular} & \begin{tabular}{l} 1,400 \\ 4,000 \\ 3,398 \\ \end{tabular} & \begin{tabular}{l} 1,400 \\ 486 \\ \end{tabular} & \begin{tabular}{l} 1,400 \\ 970 \\ \end{tabular} \\ \hline \begin{tabular}{l} SLENT (en) \\ \end{tabular} & \begin{tabular}{l} 1,744 \\ 1,438 \\ \end{tabular} & \begin{tabular}{l} 249 \\ 206 \\ \end{tabular} & \begin{tabular}{l} 499 \\ 410 \\ \end{tabular} \\ \hline XML-MT (ja) & \begin{tabular}{l} 100,033 \\ 97,893 \\ \end{tabular} & \begin{tabular}{l} 500 \\ 500 \\ \end{tabular} & 
\begin{tabular}{l} 1,500 \\ 1,500 \\ \end{tabular} \\ \hline \end{tabular}
\end{table}
Table 2: Dataset statistics.

 

Binary classificationThe task is to output a single class label for binary detection, and the value of the reward function \(R(y^{*},y_{j}^{\prime})\) is \(1.0\) if \(y^{*}=y_{j}^{\prime}\), and \(0.0\) otherwise. The evaluation metric is corpus-level detection F1.

* **ISD** is a dataset for detection of "sarcasm" text in English and Arabic Abu Farha et al. (2022).
* **EDOS-A** is a dataset for detection of "sexist" text in Egnlish Kirk et al. (2023).

Multi-class classificationThe task is to output a single class label for fine-grained classification, and \(R(y^{*},y_{j}^{\prime})\) is the same as that of binary classification.

* **EDOS-B** is a dataset for 4-way fine-grained classification about the sexist text in English Kirk et al. (2023); the evaluation metric is the macro F1.
* **CLINC** is a dataset for 150-way intent classification of user input text in English, which also tests out-of-domain detection Larson et al. (2019); the evaluation metric is the joint accuracy proposed in Zhang et al. (2020).

SegmentationThe task is to output a tagged version of an input text. \(R(y^{*},y_{j}^{\prime})\) is a word-level F1, and the evaluation metric is the same.

* **SSENT** is a decomposed subtask of structured sentiment analysis Barnes et al. (2022), and we use the OpENER portion Agerri et al. (2013) in English and Spanish.

TranslationThe task is to translate text from a language to another, and \(R(y^{*},y_{j}^{\prime})\) is example-level GLEU Wu et al. (2016). The evaluation metric is BLEU Papineni et al. (2002).

* **XML-MT** is a dataset for translation of XML-tagged Web documentation Hashimoto et al. (2019), and we take the English-to-Japanese/Finnish subsets.

### Demonstration Retrieval and Reranking

As the off-the-shelf text retriever, we use a generic t5x retriever Ni et al. (2022) used in previous work Chaudhary et al. (2023); Gao et al. (2023), which has a multilingual capability based on mT5 Xue et al. (2021). We also use mT5 following RankT5 Zhuang et al. (2023)) to train the reranking model, which can be seamlessly applied to the different languages' data.

We use \(n=10\) to train and validate the reranking model, and use \(n=50\) for the final test evaluation to investigate the generalization ability. We have run training of the reranking model in about 100 configurations in total, for different datasets, hyperparameter search of \(\ell\) in Equation (6), and checkpoint selection with evaluation on the validation sets, before touching the test sets.

### Methods to be Compared

For the evaluation, we report results by the following methods including the standard baselines and our reranking methods:

* **"0-shot"** is a baseline to know how the LLM performs only with the task instruction.
* **"RETR**" is another baseline to know how the LLM performs by simply selecting the top-\(k\) retrieved demonstrations.
* "\(u_{\mathrm{OP}}\)" and "\(u_{\mathrm{DM}}\)" are our main baselines by the reranking models trained with the direct utility functions (Section 3.1).
* "\(u_{\mathrm{OP}}^{+}\)" and "\(u_{\mathrm{DM}}^{+}\)" are the main methods for our analysis, by the reranking models trained with the incremental utility functions (Section 3.2).

When constructing the prompts, we order the demonstrations according to the retrieval scores for RETR, and according to the reranking scores for the reranking-based methods.

## 5 Results and Discussions

Table 3 shows the \(k\)-shot ICL results on the test sets, where we use \(k=1,3,5\) for all the datasets, except for XML-MT with \(k=1,2,3\) due to its prompt being too long. All the evaluation scores range in [0, 100]. We also conduct a cross-lingual experiment with SSENT; we use the English training examples for retrieval, and directly apply the English reranking models to the Spanish test set.

We first summarize general observations, and then show our analysis to provide insights.

* RETR ICL outperforms the 0-shot baseline on all the datasets, except for ISD (en). This indicates that the general input-text similarity would not always find useful demonstrations 

* Reranking (with at least one utility function) improves upon RETR. Only one exception is ISD (ar) with \(k=1\), but \(k=3\) and \(k=5\) still show the benefit.
* The OP utility tends to be more effective than the DM utility, especially on the classification tasks, while we can see potential advantages of DM on SSENT (es) and XML-MT.
* Our proposed incremental utility tends to improve upon the direct utility, especially with the OP utility.
* English-based reranking is well transferred to Spanish on SSENT. This is encouraging, because it is often the case that we only have English resources for training.
* As motivated and expected in Section 3.2, the incremental utility functions work the best with \(0.0<\ell<1.0\), especially \(\ell=0.8\) (more details in Appendix E).

### Analysis on Classification Tasks

A clear advantage of OP on the classification task is to provide denser training signals than DM. Figure 2 (best viewed in color) shows distributions of \(u_{\mathrm{DM}}\) and \(u_{\mathrm{OP}}\) on the training sets of ISD (en), EDOS-B, and CLINC; the OP-based utility values are well distributed across the value range. Figure 3 then shows distributions of \(u^{+}_{\mathrm{DM}}\) and \(u^{+}_{\mathrm{OP}}\), and we can see the consistent trend.

Existence of negative incremental valuesA notable observation is that not all the demonstrations bring positive incremental values, as shown in Figure 3. Based on the definition of the incremental utility function (Equation (7)), a demonstration has a _negative_ effect when its incremental utility value is less than 0.5. It is interesting to see that even the top-10 retrieved candidates can have negative impacts in ICL.

When is incremental utility effective?Now that we have seen the difference of the distribu 

tions of the OP and DM values, a remaining question is why the incremental utility is not always effective. Specifically, Table 3 shows that \(u_{\mathrm{OP}}^{+}\) performs worse than \(u_{\mathrm{OP}}\) on ISD (ar) and CLINC. To explain the results, we count the number of _contrastive_ training examples that meet the following criterion:

a training example \(e\) is considered to be contrastive, if its corresponding \(n\) (\(=10\)) demonstration candidates cover both the \(u(e,x_{j}^{\prime})>u_{0}(x_{j}^{\prime})\) and \(u(e,x_{j}^{\prime})<u_{0}(x_{j}^{\prime})\) cases.

We expect such a training example to contribute to effective contrast of the demonstrations. Table 4 shows the counting results, along with the improvement (averaged across \(k=1,3,5\)) over \(u_{\mathrm{OP}}\) by \(u_{\mathrm{OP}}^{+}\) in Table 3, and we can see a positive correlation as expected.

Can we increase contrastive examples?We then discuss how to increase the number of the contrastive examples. Inspired by constrained retrieval Gao et al. (2023), we modify our simple retrieval process with the following steps:

1. retrieving top-\(N\) (\(N>n\)) demonstration candidates for a training example \(e\),
2. selecting \(n^{\prime}\) candidates according to the retrieval scores, by ensuring that all the class labels' examples are equally selected,
3. computing the direct utility scores of all the \(n^{\prime}\) candidates, and
4. selecting \(n\) (=10) candidates to make \(e\) a contrastive example if possible.

The step 4. is optional to control the number of candidates to be used for training the reranking model. For ISD (ar), we set \(N=300\)2 and \(n^{\prime}=50\). Table 5 shows how the number of the contrastive examples increases as expected, and Table 6 shows the ISD (ar) test evaluation results. It is notable that the constrained retrieval lets \(u_{\mathrm{OP}}^{+}\) perform the best with \(k=1\).

Footnote 2: \(N\) is a dataset-dependent (or retriever-dependent) hyperparameter, and we suggest to set the value to maximize the class label coverage of the retrieved candidates.

Necessity of compositional utilityWe see in Table 6 that \(u_{\mathrm{OP}}\) still performs better than \(u_{\mathrm{OP}}^{+}\) with \(k=3,5\); however, this is less controllable in our work because the utility values are defined for each demonstration independently. Therefore, it is a crucial next step to investigate compositional effects of adding multiple demonstrations Gupta et al. (2023); Ye et al. (2023).

General instructionIn summary, we provide the following instruction for classification tasks:

* using \(u_{\mathrm{OP}}\) if the number of the contrastive training examples is low (e.g., less than 50%), and otherwise \(u_{\mathrm{OP}}^{+}\), and
* using the constrained retrieval if there are much less contrastive training examples (e.g., less than 40%).

### Analysis on Non-Classification Tasks

Figure 4 and Figure 5 (best viewed in color) show the distributions of the utility values on SSENT (es) and XML-MT (ja) as in Section 5.1. Unlike the classification tasks, the direct utility values are well distributed by the task-oriented reward.

Is incremental utility effective with OP?One observation is that 60-80% of the \(u_{\mathrm{OP}}\) values fall into the [0.0, 0.05] bucket; this is presumably because the LLM is not finetuned with the task formats of SSENT and XML-MT, and generating long texts leads to lower probability values in general Wu et al. (2016).3 This would hinder the effectiveness of \(u_{\mathrm{OP}}\) on the non-classification tasks,

\begin{table}
\begin{tabular}{l|c|c} \hline  & Original & Constrained \\ \hline \(u_{\mathrm{OP}}\) & 52.92, 57.14, 57.53 & 53.31, **58.25.****59.17** \\ \(u_{\mathrm{OP}}^{+}\) & 52.44, 55.87, 56.82 & **54.00**, 57.29, 57.10 \\ \hline \end{tabular}
\end{table}
Table 6: Comparison between original (Table 3) and constrained retrieval strategies on ISD (ar).

\begin{table}
\begin{tabular}{l|c|c} \hline  & Contrastive examples & Improvement \\  & & by \(u_{\mathrm{OP}}^{+}\) \\ \hline EDOS-B & 88.6\% & +8.79 \\ \hline ISD (en) & 62.8\% & +0.18 \\ \hline EDOS-A & 56.2\% & +0.71 \\ \hline CLINC & 49.1\% & -0.71 \\ \hline ISD (ar) & 37.9\% & -0.82 \\ \hline \end{tabular}
\end{table}
Table 4: Importance of the contrastive examples (with OP) for the classification tasks.

\begin{table}
\begin{tabular}{l|c|c} \hline  & Original & Constrained \\ \hline \(u_{\mathrm{OP}}\) & 52.92, 57.14, 57.53 & 53.31, **58.25.****59.17** \\ \(u_{\mathrm{OP}}^{+}\) & 52.44, 55.87, 56.82 & **54.00**, 57.29, 57.10 \\ \hline \end{tabular}
\end{table}
Table 5: The effects of \(n^{\prime}\) on ISD (ar).

 

and actually Table 3 shows that \(u_{\mathrm{DM}}\) consistently outperforms \(u_{\mathrm{OP}}\) (with \(k=1,2,3\)) on XML-MT. However, as in the classification tasks, we can see that \(u^{+}_{\mathrm{OP}}\) helps improve the scores, and Table 7 shows the effects of the contrastive examples.

Limitation of DMWe have seen the potential advantage of DM by the comparison between \(u_{\mathrm{DM}}\) and \(u_{\mathrm{OP}}\) on the SSENT and XML-MT datasets. This is an expected observation that \(u_{\mathrm{DM}}\) works well on the non-classification tasks based on the nuanced values as the reward, only by having access to the output strings. However, \(u^{+}_{\mathrm{DM}}\) does not improve the scores in most of the cases (shown in Table 3). Table 8 shows the effects of the contrastive training examples from a viewpoint of DM. Surprisingly, there are much less contrastive examples than those with OP.

Necessity of compositional utilityOne reason why we could not effectively create the contrastive training examples is that, the LLM's actual prediction is not easily affected by only one demonstration (i.e., \(k=1\)). For example, if we inspect the LLM's predictions on the validation set of SSENT (en), only 27.4% of the independent demonstrations lead to different predictions than those with the 0-shot setting. If we include demonstrations that can modify the LLM's predictions with \(k=2\), the ratio increases to 40.1%. We have then missed potentially useful signals for the utility estimation. This indicates that it is necessary for future work to estimate the utility of the demonstrations with \(k>1\) as also discussed in Section 5.1.

General instructionIn summary, we provide the following instruction for non-classification tasks:

* using \(u_{\mathrm{DM}}\) if well-defined nuanced metrics for the reward can be implemented, and
* considering the use of \(u^{+}_{\mathrm{OP}}\) otherwise, as in the classification tasks.

### Demonstrations with High Utility

We have investigated the effectiveness of incremental utility, and in this section, we focus on how the utility values are related to the demonstrations' ground-truth outputs. For the sake of simplicity, we take the classification datasets in this analysis.

For each dataset, we list all the \(n|\mathcal{T}|\) input-demonstration pairs from the training set as in Section 2.2. We then sort the pairs to select the top-\(m\)% of them either by \(u_{\mathrm{OP}}\) or \(u^{+}_{\mathrm{OP}}\), and calculate the ratio of the pairs whose demonstration labels match with their paired input's labels.

Table 9 shows the results, and overall, we can see that \(u_{\mathrm{OP}}\) and \(u^{+}_{\mathrm{OP}}\) rank the pairs differently as expected in Section 3.2. Judging from Table 3

Figure 4: Distributions of \(u_{\mathrm{DM}}\) and \(u_{\mathrm{OP}}\) on the training sets of the two non-classification datasets.

\begin{table}
\begin{tabular}{l|c|c} \hline \hline  & Contrastive examples & Improvement \\  & & \(u^{+}_{\mathrm{DM}}\) \\ \hline XXIL-MT (ja) & 28.4\% & -0.11 \\ \hline XML-MT (fi) & 27.1\% & +0.14 \\ \hline SSENT (es) & 0.06\% & -1.01 \\ \hline SSENT (en) & 0.05\% & -0.13 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Importance of the contrastive examples (with DM) for the non-classification tasks.

 

and Table 4, \(u_{\rm OP}^{+}\) performed the best on EDOS-B, and we can see much more matched labels when sorted by \(u_{\rm OP}^{+}\) than \(u_{\rm OP}\). This is considered to be a factor of the improvement, related to previous studies about the importance of the demonstrations' labels Min et al. (2022); Gao et al. (2023), while there are other factors like coverage Gupta et al. (2023) or compositionality Ye et al. (2023).

### Generalization Ability

We have used Flan-PaLM 2 (L) as an LLM, and the t5x retriever as a baseline retriever; these are the core components in our evaluation pipeline. The last question we would like to answer here is whether our empirical findings are generalized by replacing the LLM or the retriever.

#### 5.4.1 Llms

We conduct experiments by simply replacing Flan-PaLM 2 (L) with another instruction-tuned model, Flan-T5 XXL Chung et al. (2022), or a _non_-instruction-tuned model, GPT-J Wang and Komatsuzaki (2021). For Flan-T5, we use exactly the same prompt designs, and for GPT-J, we use different prompt designs as detailed in Appendix F.

Table 10 reports the results on EDOS-B, and we can see the consistent advantage of \(u_{\rm OP}^{+}\) on the dataset. One interesting observation is that GPT-J performs much worse than the instruction-tuned models in the baseline results, but the scores are dramatically improved by the reranked demonstrations. The results show that different LLMs make use of the demonstrations differently, which would be dependent on the target tasks, inherent biases of the LLMs, and the pre-training strategies. For example, the "Animosity" class on the dataset is much better predicted by GPT-J than the others. As a result, from a viewpoint of the macro F1 metric, GPT-J has an advantage, because performing poorly on a class significantly hurts the metric.

#### 5.4.2 Retrievers

We then conduct another set of the experiments by replacing the t5x retriever with a TF-IDF retriever4 at the test time, to see how the reranking model (trained with the t5x retriever) works with different candidate sets. Table 11 shows the results on EDOS-B and SSENT (en). We can see that the reranking model works even with the TF-IDF-based candidates, showing the robustness of the reranking with the proposed utility functions.

Footnote 4: [https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)

## 6 Related Work

We discuss relationships between our work and previous work from several viewpoints. Interested readers may refer to a survey by Luo et al. (2024).

Utility functionRecent work Rubin et al. (2022); Li et al. (2023); Luo et al. (2023); Wang et al. (2023b) has proposed finetuning text retrieval models by estimating direct utility of a demonstration by \(u_{\rm OP}\)-like utility functions. While they showed the effectiveness, we have made a deeper dive into understanding the effects of the different utility functions, especially with the enhance of incremental utility.

Loss functionTo finetune the retrieval models, some used contrastive learning to contrast a high-utility demonstration against a number of low-utility demonstrations Rubin et al. (2022);

\begin{table}
\begin{tabular}{c|c|c} \hline \hline  & Flan-T5 XXL & GPT-J \\ \hline
0-shot & 35.28 & 19.78 \\ \hline RETR & 36.59, 38.67, 38.37 & 37.12, 35.16, 29.68 \\ \hline \(u_{\rm OP}\) & 36.02, 38.81, 39.71 & 55.78, 57.23, 56.40 \\ \cline{2-3} \(u_{\rm OP}^{+}\) & **41.30**, **45.42**, **46.89** & **58.79**, **59.15**, **58.54** \\ \cline{2-3} \(u_{\rm OP}^{-}\) & **38.10**, **40.13**, 41.98 & 51.28, 53.08, 53.60 \\ \cline{2-3} \(u_{\rm OM}^{-}\) & **36.85**, 39.89, 40.70 & 52.33, 54.25, 54.55 \\ \hline \end{tabular}
\end{table}
Table 10: Results on EDOS-B by using Flan-T5 XXL or GPT-J as the LLM component.

\begin{table}
\begin{tabular}{c c|c|c|c} \hline \hline  & & Top-10\% & Top-30\% & Top-50\% \\ \hline ISD (en) & \(u_{\rm OP}^{+}\) & 68.7\% & 67.1\% & 64.8\% \\ \cline{2-3}  & \(u_{\rm OP}^{+}\) & 68.7\% & 67.2\% & 65.1\% \\ \hline ISD (ar) & \(u_{\rm OP}^{+}\) & 93.6\% & 89.4\% & 86.3\% \\ \cline{2-3}  & \(u_{\rm OP}^{+}\) & 81.1\% & 81.2\% & 81.7\% \\ \hline EDOS-A & \(u_{\rm OP}^{+}\) & 69.3\% & 66.7\% & 66.3\% \\ \cline{2-3}  & \(u_{\rm OP}^{+}\) & 72.5\% & 74.1\% & 73.7\% \\ \hline EDOS-B & \(u_{\rm OP}^{+}\) & 57.4\% & 53.8\% & 51.1\% \\ \cline{2-3}  & \(u_{\rm OP}^{+}\) & 93.3\% & 72.3\% & 62.8\% \\ \hline CLINC & \(u_{\rm OP}^{+}\) & 86.2\% & 86.9\% & 86.1\% \\ \cline{2-3}  & \(u_{\rm OP}^{+}\) & 85.3\% & 78.4\% & 78.7\% \\ \hline \hline \end{tabular}
\end{table}
Table 9: Label matching ratio with high utility.

 

Luo et al., 2013; Wang et al., 2023), and others used ranking losses Li et al. (2023). Contrastive learning Karpukhin et al. (2020) and ranking losses Zhuang et al. (2023) are widely used for document retrieval and reranking, but simple pointwise regression has shown to be effective as well Nie et al. (2019); Nogueira and Cho (2019); Asai et al. (2020). We employed the pointwise regression Equation (2), based on our observations that it significantly improves ICL upon the RETR baseline. Instead, we focused on the contrast between the 0-shot and 1-shot performances to derive the incremental utility functions.

Overall pipelineLin et al. (2023) have taken one step further to finetune LLMs with a retriever, while our work has focused on using an LLM as it is. Within this framework, it is a common practice to directly finetune a dense text retrieval model Karpukhin et al. (2020) for the demonstration retrieval Rubin et al. (2022); Li et al. (2023); Luo et al. (2023), while Wang et al. (2023) employed a two-step approach by knowledge distillation with a cross-attention reward model (similar to our reranking model). By contrast, we followed another major framework with retrieval and reranking Zhuang et al. (2023), where we use a generic text retriever for all the different settings. This is useful in making our experiments controllable for analysis, in that the reranking models are always applied to the consistent candidate sets. One potential drawback of the cascaded pipeline is that the search space is limited; still, we have shown that there is large room for improvement even within the limited search space.

Constrained retrieval and selectionThis paper has focused on using feedback from LLMs to train the reranking models. Another line of related work is to improve the demonstration retrieval/selection process without finetuning additional models. Gao et al. (2023) take into account difficulty of each demonstration and entropy of the LLMs' prediction, Mavromatis et al. (2023) select demonstrations based on uncertainty and diversity, Margatina et al. (2023) adopt active learning, and Gupta et al. (2023) propose to increase coverage of information about inputs. In Section 5.1, we have shown that a simple constrained retrieval has the potential to improve the reranking models, and it is an interesting direction to incorporate those kinds of techniques into the training of the reranking models.

Unified modelingWe conducted our analysis on the different tasks in several languages, and all the reranking models were trained independently. Li et al. (2023) made an interesting attempt to finetune a retriever across different tasks, and showed promising results. Then another interesting future direction is to integrate our findings into such a unified framework; for example, instruction-based demonstration retriever/reranker is worth investigating by following Asai et al. (2023).

## 7 Conclusion

This paper has investigated how the output probability and downstream metric affect the utility estimation of demonstrations for ICL. Our in-depth analysis has shown that the output probability is robust on the classification tasks, and the downstream metric is robust especially when the output probability is not well distributed across the whole value range. Furthermore, we provided discussions about why and when our proposed incremental utility helps improve the ICL results.

### Acknowledgements

First of all, we thank Aditi Chaudhary and Krishna Srinivasan for their contributions to the basics of the ICLR work in our team. We also appreciate discussions with Lingyu Gao about her internship project and related work. Feedbacks from Zhuyun Dai and Tania Bedrax-Weiss were also valuable in polishing the draft. Lastly, we would like to thank anonymous ARR reviewers for their constructive feedbacks.

### Limitations

Inference-time costWe adopted the retrieval-reranking framework, and the reranking model is a large cross-attention encoder model based on mT5 XXL. This is known to be much slower than another standard framework with dense retrievers Karpukhin et al. (2020). The main goal in this paper is to investigate how to transfer the feedback signals from the LLMs to the demonstration selection, and this is still an open research question. Therefore, we first prioritized having an enough model capacity instead of restricting the inference-time cost, as in previous work on document retrieval Asai et al. (2020). Once it gets practically useful, we can consider applying knowledge distillation Wang et al. (2023), or directly training dense retriever frameworks Rubin et al. (2022) 



## Ethics Statement

Inherent biasesIt is known that LLMs have inherent biases that affect ICL results (Zhao et al., 2021; Min et al., 2022). We also observed that specific labels are more generated than others on the classification tasks, and the trends are different among different LLMs like PaLM 2 and GPT-J. It is then possible that the biases are transferred to the reranking models, and potentially we can further improve the results by debiasing the feedback signals. This has not been addressed in this paper, and is left for future work.

Contents of the datasetsWe used publicly available datasets that have been properly processed for privacy concerns. The EDOS dataset contains offensive contents as described in Kirk et al. (2023), and we intentionally avoided showing examples from the dataset. We do not intend to enhance the offensive contents in this paper, and instead solely focus on how the LLMs work on the challenging fine-grained classification.

## References

* J. Abu Farha, S. Vlad Oprea, S. Wilson, and W. Magdy (2022)SemEval-2022 task 6: iSACasmEval, Intended sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp. 802-814. Cited by: SS5.
* R. Agerri, M. Cuadros, S. Gaines, and G. Rigau (2013)OpenNER: open polarity enhanced named entity recognition. In Sociedad Espaniola para el Procesamiento del Lenguaje Natural, Vol. 51, pp. 215-218. External Links: Link Cited by: SS1, SS2.
* A. Asai, K. Hashimoto, H. Hajishirzi, R. Socher, and C. Xiong (2020)Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering. In International Conference on Learning Representations, Cited by: SS1, SS2.
* A. Asai, T. Schick, P. Lewis, X. Chen, G. Izacard, S. Riedel, H. Hajishirzi, and W. Yih (2023)Task-aware Retrieval with instructions. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 3650-3675. Cited by: SS1, SS2.
* J. Barnes, L. A. Maria Oberlander, E. Troiano, A. Kutuzov, J. Buchmann, R. Agerri, L. Ovrelid, and E. Velldal (2022)SemEval-2022 Task 10: Structured Sentiment Analysis. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), Cited by: SS1, SS2.
* D. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandli, A. Radford, I. Sutskever, and D. Amodei (2020)Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Cited by: SS1, SS2.
* A. Chaudhary, K. Raman, K. Srinivasan, K. Hashimoto, M. Bendersky, and M. Najork (2023)Exploring the Viability of Synthetic Query Generation for Relevance Prediction. arXiv preprint cs.IR 2305.11944. Cited by: SS1, SS2.
* H. Won Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. Shane Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei (2022)Scaling Instruction-Finetuned Language Models. arXiv prepring cs.LG 2210.11416. Cited by: SS1, SS2.
* A. de Brebisson and P. Vincent (2016)The Z-loss: a shift and scale invariant classification loss belonging to the Spherical Family. arXiv preprint arXiv:1604.08859. Cited by: SS1, SS2.
* L. Gao, A. Chaudhary, K. Srinivasan, K. Hashimoto, K. Raman, and M. Bendersky (2023)Ambiguity-Aware InContext Learning with Large Language Models. arXiv preprint cs.CL 2309.07900. Cited by: SS1, SS2.
* G. Anil, A. M. Dai, O. Firrat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, L. El Shafey, Y. Huang, K. Meier-Hellstern, G. Mishra, E. Moreira, M. Ozemrick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G. Hernandez Abrego, J. Ahn, J. Austin, P. Barham, J. Botha, J. Bradbury, S. Brahma, K. Brooks, M. Catasta, Y. Cheng, C.Cherry, C. Choquette-Choo, A. Chowdberg, C. Crepp, S. Dave, M. Debshian, S. Dev, J. Devlin, M. Diaz, N. Du, E. Dyer, V. Feinberg, F. Fengxiaoyu Feng, V. Finberre, X. Garcia, S. Gehrmann, L. Gonzalez, G. Gur-Ari, S. Hand, H.



Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenelay, Maxim Krikun, Sneha Kudupanta, Chang Lan, Kathereine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023. PaLM 2 Technical Report.
* Gupta et al. (2023) Shivanshu Gupta, Matt Gardner, and Sameer Singh. 2023. Coverage-based Example Selection for In-Context Learning. _arXiv preprint cs.CL 2305.14907_.
* Hashimoto et al. (2019) Kazuma Hashimoto, Raffaella Buschiazzo, James Bradbury, Teresa Marshall, Richard Socher, and Caiming Xiong. 2019. A High-Quality Multilingual Dataset for Structured Documentation Translation. In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_, pages 116-127.
* Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. _ACM Computing Surveys_, 55(12).
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781.
* Kirk et al. (2023) Hannah Kirk, Wennjie Yin, Bertie Vidgen, and Paul Rottger. 2023. SemEval-2023 Task 10: Explainable Detection of Online Sexism. In _Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)_, pages 2193-2210.
* Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1311-1316.
* Li et al. (2023) Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023. Unified Demonstration Retriever for Context Learning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 4644-4668.
* Li and Qiu (2023) Xiaonan Li and Xipeng Qiu. 2023. Finding Support Examples for In-Context Learning. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 6219-6235, Singapore. Association for Computational Linguistics.
* Lin et al. (2023) Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Scott Yih. 2023. RA-DIT: Retrieval-Augmented Dual Instruction Tuning. _arXiv preprint cs.CL 2310.01352_.
* Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What Makes Good In-Context Examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_, pages 100-114.
* Luo et al. (2023) Man Luo, Xin Xu, Zhuytun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y Zhao. 2023. Dr.ICL: Demonstration-Retrieved In-context Learning. _arXiv preprint cs.CL 2305.14128_.
* Luo et al. (2024) Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi. 2024. In-context Learning with Retrieved Demonstrations for Language Models: A Survey. _arXiv preprint arXiv:2401.11624_.
* Margatina et al. (2023)Katerina Margatina, Timo Schick, Nikolaos Aletras, and Jane Dwivedi-Yu. 2023. Active Learning Principles for In-Context Learning with Large Language Models. _arXiv preprint cs.CI 2305.14264_.
* Mavromatis et al. (2023) Costas Mavromatis, Balasubramaniam Srinivasan, Zhengyuan Shen, Jiani Zhang, Huzefaa Rangwala, Christos Faloutsos, and George Karypis. 2023. Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selection. _arXiv preprint cs.CL 2310.20046_.
* Min et al. (2022) Sewoon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 11048-11064.
* Ni et al. (2021) Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei 

Yang. 2022. Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 1864-1874. Cited by: SS3.
* Y. Nie, S. Wang, and M. Bansal (2019)Revealing the Importance of Semantic Retrieval for Machine Reading at Scale. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Vol. , pp. 2553-2566. External Links: Document Cited by: SS1, SS2.
* R. Nogueira and K. Cho (2019)Passage Re-ranking with BERT. arXiv preprint cs.IR1901.04085. Cited by: SS1.
* OpenAI (2023)GPT-4 Technical Report. arXiv preprint arXiv:2303.08774. Cited by: SS1.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318. Cited by: SS1.
* K. Raman, I. Naim, J. Chen, K. Hashimoto, K. Yalasangi, and K. Srinivasan (2022)Transforming Sequence Tagging Into A Seq2Seq Task. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11856-11874. Cited by: SS1.
* M. Ranzato, S. Chopra, M. Auli, and W. Zaremba (2016)Sequence Level Training with Recurrent Neural Networks. In International Conference on Learning Representations, Cited by: SS1.
* A. Roberts, H. Won Chung, A. Levskaya, G. Mishra, J. Bradbury, D. Andor, S. Narang, B. Lester, C. Gaffney, A. Mohiuddin, C. Hawthorne, A. Lewkowycz, A. Salcianu, M. van Zee, J. Austin, S. Goodman, L. Balini Soares, H. Hu, S. Tsvyashchenko, A. Chowdhery, J. Bastings, J. Bulian, X. Garcia, J. Ni, A. Chen, K. Kenealy, J. H. Clark, S. Lee, D. Garrette, J. Lee-Thorp, C. Raffel, N. Shazeer, M. Ritter, M. Bosma, A. Passos, J. Maitin-Sheard, N. Fiedel, M. Omernick, B. Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo (2022)Scaling Up Models and Data with t5x and seqio. arXiv preprint arXiv:2203.17189. Cited by: SS1.
* O. Rubin, J. Herzig, and J. Berant (2022)Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655-2671. Cited by: SS1.
* N. Shazeer and M. Stern (2018)Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. In Proceedings of the 35th International Conference on Machine Learning, pp. 4596-4604. Cited by: SS1.
* H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosalse, D. Bilkel, L. Blecher, C. Canton Ferrer, M. Chen, G. Cucurull, D. Eisibot, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseinii, R. Hou, H. Ina, M. Kardas, V. Kerkez, M. Khabsa, I. Klochumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Lakovich, Y. Lu, Y. Mao, X. Martinet, T. Mihalyov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Rezenstein, R. Runga, K. Saladi, A. Schelten, R. Silva, E. Michael Smith, R. Subramanian, X. Ellen Tan, B. Tang, R. Taylor, A. Williams, J. Xiang, K. Xu, Z. Yan, I. Zarvo, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Sciolom (2023)LIMA 2: Open Foundation and Fine-tuned Chat Models. Cited by: SS1.
* B. Wang and A. Komatsuzaki (2021)GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. Note: [https://github.com/kingofoliz/mesh-transformer-jax](https://github.com/kingofoliz/mesh-transformer-jax) Cited by: SS1.
* L. Wang, L. Li, D. Dai, D. Chen, H. Zhou, F. Meng, J. Zhou, and X. Sun (2023)Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning. arXiv preprint arXiv:2305.14160. Cited by: SS1.
* L. Wang, N. Yang, and F. Wei (2023)Learning to Retrieve In-Context Examples for Large Language Models. arXiv preprint cs.CL 2307.07164. Cited by: SS1.
* Yonghui Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, Xiaobine Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kondo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean (2016)Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint cs.CL 1609.08144. Cited by: SS1.
* L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel (2021)MT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498.

 

Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023 Compositional Exemplars for In-Context Learning. In _Proceedings of the 40th International Conference on Machine Learning_, ICML'23. JMLR.org.
* Zhang et al. (2020) Jianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, Philip Yu, Richard Socher, and Caiming Xiong. 2020. Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5064-5082.
* Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In _International Conference on Machine Learning_, pages 12697-12706.
* Zhuang et al. (2023) Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Berdersky. 2023. RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses. In _SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 2308-2313.

 



## Appendix A Prompt Design for Flan Models

We use the prompt template defined in Gao et al. (2023) for the instruction-tuned LLMs in our experiments. We then need to have a task definition for each dataset with an input-output format.

## B Datasets

Table 12 shows examples of input-output pairs from the datasets used in our experiments.

### Isd

The task definition of ISD is as follows:

The goal of this task is to identify if an input text is sarcastic or non-sarcastic.

Our initial analysis showed that the LLMs in our experiments have inherent bias towards generating "saracusic" and "non-sarcastic" instead of "Sarcastic" and "Non-sarcastic," and then we decided to use the all-lowercased label strings. The same task definition is shared in Arabic and English.

### Edos-a

The task definition of EDS-A is as follows:

The goal of this task is to identify if an input text is Sexist or Non-sexist.

In contrast to ISD, the LLMs in our experiments are biased towards generating "Sexist" and "Non-sexist" with the first letters capitalized.

### EDOS-B

The task definition of EDOS-B is as follows:

The goal of this task is to identify a category of asexual text from Threat, Derogation, Animosity, or Prejudice.

The first letters are capitalized in the class labels with the same reason as in EDOS-A. Gao et al. (2023) incorporated more detailed descriptions of the fine-grained class labels into their task definition, while we simply listed the class labels to make all the tasks be tested under the consistent setting.

### Clinc

The task definition of CLINC is as follows:

The goal of this task is to identify a service domain and an intent given a user input. There are 10 domains: "auto_and_commute" banking" credit_cards" "home" kitchen_and_dining"meta" small_talk"travel" "utility" work". For each domain, there are 15 intents: "auto_and_commute"=[ current_location"oil_change_when" oil_change_how"uber"traffic" tire_pressure"schedule_maintenance" gas"mpg"distance"directions" last_maintenance"gas_type" tire_change"jump_start"], "banking"=[ "freeze_account"routing" pin_change"bill_doe"pay_bill" account_blocked"interest_rate" min_payment"bill_balance" transfer"order_checks"balance" spendinghistory"transactions" report_fraud"], "credit_cards"={ "replacement_card_duration" expiration_date"damaged_card" improve_credit_score" report_lost_card"card_declined" credit_limit_change"apr" redeem_rewards"credit_limit" rewards_balance"application_status" credit_score"new_card" international_fees"}, "home"=[ what_song"play_music" todo_list_update"reminder" reminder_update" calendar_update" order_status"update_playlist"shopping_list"calendar" next_song"order"todo_list" shopping_list_update"smart_home" ], "kitchen_and_dining"=[ food_last"conform_reservation" how_busy"ingredients_list" calories"nutrition_info" recipe"restaurant_reviews"restaurant_reservation" meal_suggestion" restaurant_suggestion" cancel_reservation" ingredient_substitution

*cook_time" "accept_reservations"
1),"meta"={ "change_speed"
2"user_name" whisper_mode" "yes"
3"change_volume" "no" "change_language"
4"repeat" "change_accent" "cancel"
5"sync_device" "change_user_name"
6"change_ai_name" "reset_settings"
7"maybe" ],"small_talk"={
8"who_made_you" "meaning_of_life"
9"who_do_you_work_for" "do_you_have_pets"
10"what_are_your_hobbies" "fun_fact"
11"what_is_your_name" "where_are_you_from"
12"goodbye" "thank_you" "greeting"
13"tell_joke" "are_you_a_bot"
14"how_old_are_you" "what_can_i_ask_you"
15"travel"=[ "plug_type"
16"travel_notification" "translate"
17"flight_status" "international_visas"
18"timezone" "exchange_rate"
19"travel_suggestion" "travel_alert"
20"vaccines" "lost_luggage" "book_flight" "book_hotel" "carry_on" "car_rental"
21"utility"=[ "weather" "alarm" "date" "find_phone" "share_location" "timer"
22"make_call" "calculator" "definition"
23measurement_conversion" "flip_coin"
24"spelling" "time" "roll_dice" "text" ],"work"={ "pto_request_status" next_holiday" "insurance_change"
25"insurance" "meeting_schedule" "payday"
26"taxes" "income" "rollover_40lk"
27"pto_balance" "pto_request" "wz"
28 schedule_meeting" "direct_deposit"
29"pto_used" ]. Then the output is like
30 domain: intent". If the input does not belong to any of the domains, the Answer is "oos".

The class labels are grouped according to the domains.5 The output label is with the domain name, but in the evaluation, we only refer to the intent class. The "oos" class is used to evaluate the detection of out-of-domain inputs Larson et al. (2019).

Footnote 5: [https://github.com/clinc/ooss-eval/blob/master/data/domains.json](https://github.com/clinc/ooss-eval/blob/master/data/domains.json)

In our preliminary experiments, we have observed that simply using the LLMs' output strings leads very low recall of the "oos" class. This is mainly because the LLMs are biased towards generating in-domain class labels affected by in-domain demonstrations, which is consistent with Min et al. (2022) about LLMs' bias in ICL. We then follow a threshold-based strategy in Zhang et al. (2020); we replace an output of an in-domain class label with "oos" if its prediction probability is below a pre-defined threshold. The threshold values are 0.6 for \(k=1\), 0.7 for \(k=2\), and 0.8 for \(k=3,4,5\).

### Ssent

The task definition of SSENT is as follows:

The goal of thistask isto copy the given hotel review by tagging sentiment-expressing phrases with the markup tags: <Positive></Positive> or <Negative></Negative>. Then the output is like "word1 <Positive> word2 word3</Positive> word4 <Negative>word5</Negative>". If there are not such tagged phrases, the Answer is Neutral.

For the output format, we followed Raman et al. (2022). The same task definition is shared in English and Spanish.

The structured sentiment analysis is a complex task in the original dataset Barnes et al. (2022), where the task requires extracting tuples of (polar expression, subject, object) from an input text. In our preliminary experiments, we observed that, without finetuning, it is not trivial how to handle this complicated task in few-shot ICL.

We then decided to only focus on the polar ex

\begin{table}
\begin{tabular}{|c|l|l|} \hline Dataset & Input & Output \\ \hline CLINC & what expression would i use so say & travel: translate \\  & i love you if i were an aalian & \\ \hline SSENT & The food was very basic, but edible with & The food was \(<\)Negative-very basic</Negative> , but \(<\)Positive>edible \\  & good bread , somp and deserserts , & ¡Positive with \(<\)Positive>ead , positive> bread , soup and deserts . \\ \hline XML-MT & If you have permission to edit public templates, from & Jos insulla on oikeus moukata julikisa malleja, kirjiota Mainitykset-valikson \\  & Setup, enter “stevening-Email Templates/“userinput> & ¿parammance?Pikhakuo-parmmame>-kentinian cuserinput¿Sällöpostiamalli \\  & in the de-parmmaneyoück Finde/parmmames> box, then & ¿desertipu¿ ja vailite cuicontrolcb-Classie-säklopositionallí/ciocontrolpb. \\  & select ciocontrol-Classic Email Templates/ciocontrolpb. & \\ \hline \end{tabular}
\end{table}
Table 12: Examples of input-output pairs in the training sets.

 

pression extraction task as a first step, and the sub-task is considered to be a combination of sentiment classification and sequence labeling. It is an interesting future direction to investigate end-to-end evaluation of such a complex task in ICL.

### Xml-mt

The task definition of XML-MT is as follows:

The goal of this task is to translate an XML-tagged text from English to LANGUAGE by preservin the XML structure. Boththe input and output are like "word1 <tag-A>word2 word3</tag-A> word4 <tag-B>word5</tag-B>". The placeholder LANGUAGE is replaced with a language name (Finnish or Japanese in our experiments).

The test set has not been publicly released in the original dataset.6 We sampled 500 examples from the original validation set for our validation set, and used the rest (1,500 examples) for our test set.

Footnote 6: [https://github.com/salesforce/localization-xml-mt/tree/master/data](https://github.com/salesforce/localization-xml-mt/tree/master/data)

## Appendix C Hallucination in LLMs' Prediction

A concern about using the text generation models is hallucination Ji et al. (2023). In general, we cannot perfectly prevent the models from generating texts with hallucinations. We do not discuss fact checking or semantics of the outputs, but instead solely focus on the output formats.

### Isd

We did not observe any hallucinations on ISD in our experiments. This is because we carefully selected the label strings as discussed in Section B.1.

### Edos-a

We did not observe any hallucinations on EDOS-A in our experiments. This is also because we carefully selected the label strings as discussed in Section B.2.

### Edso-b

We observed a very small number of hallucinations on EDOS-B:

1. Sexual threat
2. Sexual Objectification
3. Objectification

The case 1. is simply mapped to Threat, and the cases 2. and 3. are mapped to Derogation according to the description of the label in Kirk et al. (2023). However, the frequency is low; for example, the cases 2. and 3. are observed with 2 out of 486 examples on the validation set.

Note that we never observed such hallucinations when we used Flan-T5 XXL in Section 5.4. Therefore, this can be different across different LLMs.

### Clinc

We observed only one hallucination case on CLINC, out of the 3,100 validation examples. That case missed its domain label, only generating its intent label; in our evaluation pipeline, we automatically penalized such outputs that do not meet the defined output format. However, as mentioned above, there was only one such case in our experiments.

### Ssent

For SSENT, we have tried several different input-output formats, before we decided to use the format described in the task definition. Specifically, the use of "Neutral" significantly reduced the chance to generate outputs with hallucinations. We could not perfectly mitigate hallucinations on SSET, though.

However, we observed that most of the hallucinations can be trivially fixed with simple rules, by referring to the original inputs. For example, the followings are typical cases:

* original: don 't \(\rightarrow\) model output: do n't
* original: weren 't \(\rightarrow\) model output: were n't
* original: wasn 't \(\rightarrow\) model output: was n't

We can see that the model outputs look more natural than the original tokens, where the LLM unexpectedly _fixed_ the unnatural tokenization. This is then more like a dataset issue.

Another type of hallucination was actually output formatting errors, such as missing a closing tag </Positive>. For such hallucinations that are not trivially fixed, our evaluation pipeline automatically replaces the outputs with "Neutral." 



### Xml-Mt

For XML-MT, only the potential hallucination is XML tag formatting issue as in SSENT. The formatting issue is rarely observed, but as in the original evaluation script,7 we penalize such predictions.

Footnote 7: [https://github.com/salesforce/localization-xml-mt/blob/master/scripts/evaluate.py](https://github.com/salesforce/localization-xml-mt/blob/master/scripts/evaluate.py)

## Appendix D Training of Reranking Model

We use the T5X code base Roberts et al. (2022)8 for the reranking model. Among the available checkpoints, we use mT5 XXL9 for all the experiments. We finetune the original mT5 checkpoint to be used as a regression model Zhuang et al. (2023), once we create a set of training examples described in Section 2.2.

Footnote 8: [https://github.com/google-research/t5x/blob/main/docs/models.md#](https://github.com/google-research/t5x/blob/main/docs/models.md#) mt5-checkpoints

We use the Adafactor optimizer Shazeer and Stern (2018), along with Z-loss regularization de Brebisson and Vincent (2016), a constant learning rate of 0.001, and a batch size of 256. We evaluate checkpoints after every 100 updates of the model parameters; for the checkpoint selection, we use the reranking model to select demonstrations on the validation set of each dataset, and then select a checkpoint that leads to the best downstream metric with the LLMs' prediction.

To save the inference time of the LLMs on the validation sets, we pre-compute and cache all the possible \(k\)-shot predictions beforehand. For each validation example, the number of the demonstration combinations is \({}_{n}C_{k}\), where demonstrations are ordered according to the original retrieval scores. This caching strategy allows us to quickly check the LLMs' performance without repeating the inference multiple times.

## Appendix E Effects of \(\ell\) in Incremental Utility

In the beginning of Section 5, we have mentioned that \(\ell=0.8\) in Equation (6) is recommended to implement the incremental utility function. We report results of tuning the value of \(\ell\) on a classification dataset, EDOS-B, and a non-classification dataset, SSENT (es).

Table 13 and Table 14 show the results on the validation sets. Note that changing the values of \(\ell\) does not affect the results with \(u^{+}_{\mathrm{DM}}\) for the classification tasks, because the original reward values are binary (i.e., 0.0 or 1.0). We can see that \(\ell=0.8\) leads to the best scores, presumably because it takes a good balance between \(\ell=0.0\) and \(\ell=1.0\) as motivated in Section 3.2.

## Appendix F Prompt Design for GPT-J

GPT-J is a publicly available variant of the GPT models,10 and it has been used in previous work Wang et al. (2023a). GPT-J is not an instruction-tuned model, and we design our prompt in a question-answering format:

**Text: INPUTTEXT\n Question: which sexist category does the text express, Animosity, Derogation, Prejudice, or Threat?\n Answer: thesexist category is**

The placeholder INPUTTEXT is replaced with an actual input text, and this corresponds to the 0-shot inference. Then, a prompt in 2-shot ICL is as follows:

**Text: DEMONSTRATIONINPUT 1\n Question: which sexist category does the text express, Animosity, Derogation, Prejudice, or Threat?\n Answer: thesexist category is**

**Text: DEMONSTRATIONINPUT 1\n Question: which sexist category does the text express, Animosity, Derogation, Prejudice, or**

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline  & \(\ell=0.0\) & \(\ell=0.5\) & \(\ell=0.8\) & \(\ell=1.0\) \\ \hline \(u^{+}_{\mathrm{OP}}\) & 46.91 & 46.80 & **48.80** & 46.98 \\ \hline \end{tabular}
\end{table}
Table 13: The effects of changing the value of \(\ell\) in Equation (6), on the EDOS-B validation set.



Threat?\n
**Answer:** thesexist category is DEMONSTRATION OUTPUT 2\n
**Text:** INPUT TEXT\n
**Question:** which sexist category doesthetext express, Animosity, Derogation, Prejudice, or Threat?\n
**Answer:** thesexist category is

Once the prompt is designed, we can seamlessly run the experiments with GPT-J.

## Appendix G Compute Resources

Flan-PaLM 2 (L)Readers may refer to Google et al. (2023).

Flan-T5 XXLThe size of Flan-T5 XXL is 11B Chung et al. (2022), and we used 64 v3 TPU chips to run inference with it.

Gpt-JThe size of GPT-J is 6B Wang and Komatsuzaki (2021), and we used one NVIDIA A100 GPU to run inference with it.

mT5 XXL for rerankingThe size of mT5 XXL is 13B Xue et al. (2021), and we used 64 v3 TPU chips to run both training and inference with it for reranking.