Data Curation Alone Can Stabilize In-context Learning
Year: 2023

Abstract:
In-context learning (ICL) enables large language models (LLMs) to perform new tasks by prompting them with a sequence of training examples. However, it is known that ICL is very sensitive to the choice of training examples: randomly sampling examples from a training set leads to high variance in performance. In this paper, we show that carefully curating a subset of training data greatly stabilizes ICL performance without any other changes to the ICL algorithm (e.g., prompt retrieval or calibration). We introduce two methods to choose training subsets—both score training examples individually, then select the highest-scoring ones. CONDACC scores a training example by its average dev-set ICL accuracy when combined with random training examples, while DATAMODELS learns linear regressors that estimate how the presence of each training example influences LLM outputs. Across five tasks and two LLMs, sampling from stable subsets selected by CONDACC and DATAMODELS improves average accuracy over sampling from the entire training set by 7.7% and 6.3%, respectively. Surprisingly, the stable subset examples are not especially diverse in content or low in perplexity, in contrast with other work suggesting that diversity and perplexity are important when prompting LLMs.

heading: 1 Introduction
text: In-context learning (ICL) is a new paradigm for few-shot learning with pretrained large language models (LLMs) without any parameter updates. In ICL, an LLM can perform a new task simply by conditioning on a prompt1 consisting of a sequence of labeled training examples. First introduced by GPT-3 (Brown et al., 2020), ICL with LLMs has reached state-of-the-art few-shot performance across many tasks (Rae et al., 2021; Smith et al.,
1Prior work has different definitions of prompt; in this paper, we fix the task templates and follow Rubin et al. (2021) to denote prompt as a sequence of training examples for ICL.
2022; Thoppilan et al., 2022; Chowdhery et al., 2022). Compared with alternatives that use finetuning (Devlin et al., 2018; Schick and Schütze, 2020; Gao et al., 2020), ICL does not require taskspecific training, which enables its use with very large language models, and it uses a unified model for all tasks, enabling easier deployment.
Despite its impressive few-shot performance, ICL often exhibits unintuitive behavior (Min et al., 2022). The standard ICL approach is to randomly sample a few examples from a training set to construct a prompt (Brown et al., 2020); however, prior work (Liu et al., 2021; Zhao et al., 2021; Lu et al., 2021) has found that ICL is very sensitive to the choice of training examples and their order in the prompt. ICL is also sensitive to small changes in prompt format (Chen et al., 2022).
In this paper, we show that carefully curating a smaller training dataset from a larger pool can make ICL much more stable. We define a training subset E to be stable if randomly sampling a sequence of examples as a prompt from E yields much higher average and worst-case accuracy than randomly sampling from the original training set. We propose two methods to identify such a sta-
8123
ble subset. Our CONDACC method scores a training example by its average dev-set ICL accuracy when combined with random training examples; these scores are closely related to Data Shapley values (Ghorbani and Zou, 2019). Our DATAMODELS method fits a linear regressor that predicts the LLM’s output based on which example is present at each index in the prompt; we score a training example highly if the associated weights from the linear model indicate that its presence improves accuracy. For both methods, we then select training examples with the highest scores to form the stable subset. While some prior work improves ICL accuracy by retrieving a suitable prompt for each test example (Liu et al., 2021; Rubin et al., 2021; Su et al., 2022), we show that it is possible to achieve stably good accuracy with a randomly sampled prompt for all test examples, when given the “right” training (sub)set.
Our subset selection methods greatly improve performance across 5 classification datasets and 4 LLMs, with main experiments on GPTJ-6B (Wang and Komatsuzaki, 2021) and OPT-13B (Zhang et al., 2022a). On average, CONDACC and DATAMODELS outperform the baseline that uses the entire training set without selection (named ALL) by 7.7% and 6.3%, respectively, when comparing the average accuracy over multiple sampled prompts. In contrast, baselines that choose examples found in high-performing prompts (TOPPROMPTS) or examples that lead to high one-shot ICL accuracy (ONESHOT) to form the subsets do not perform as well (see Figure 1). Our stable subset examples generalize to out-of-distribution test data, and we can even find stable subsets for binary classification tasks that only contain examples of one label; these findings suggest that the stable subset examples help LLMs understand the overall task definition.
Finally, we study what makes stable subset examples special by analyzing sequence length, perplexity, and diversity in both raw text and embedding spaces. We find that these examples do not have abnormally long sequence lengths or high perplexities. In contrast with prior work optimizing diversity for prompt selection (Su et al., 2022; Ye et al., 2022), we find our stable subsets no more diverse than random subsets of the training data. In summary, we show that curating training data appropriately leads to more stable and accurate ICL performance; we hope future work can develop new strategies for writing such helpful ex-
amples. Code and data are publicly available at https://github.com/terarachang/DataICL.

heading: 2 Problem Setups
text: We use the original ICL formulation proposed by GPT-3, also known as the direct method, for all our experiments. Specifically, an LLM performs in-context learning on a new task based on a taskspecific prompt Z formed by concatenating K labeled training examples, i.e., Z = [z1, ..., zK ], where each zj is a training example (x, y) consisting of an input x and label y. The LLM then makes predictions on a test input xtest conditioned on the prompt Z followed by xtest, denoted by argmaxy∈CP (y|Z, xtest), where C is the set of possible labels.
Given a training set Dtr, a dev set Ddev, a heldout test set Dtest, and a predefined number of shots K, our goal is to select a stable training subset E ⊂ Dtr, |E| > K, such that randomly sampling a sequence of K examples from E to form a prompt generally yields good performance on Dtest.
We propose two setups, Labeled and Unlabeled. In Labeled, our goal is to study which training examples consistently lead to high ICL accuracy. We assume access to a large labeled DLtr = {(xi, yi)}Ntri=1 of input-label pair, and a small labeled Ddev. Unlabeled is closer to the true fewshot learning setup (Perez et al., 2021), where we only have access to Ddev and a large unlabeled training set {xi}Ntri=1. We pair each input xi with every possible label y ∈ C to create an unlabeled training set DUtr = {{(xi, ỹ)|ỹ ∈ C}}Ntri=1. In both setups, our goal is to select a subset of E training examples from either DLtr or D U tr . Note that we may select examples with incorrect labels under Unlabeled. We only use the large labeled Dtest for evaluating our selection methods.

heading: 3 Methods
text: We propose the following steps to identify a stable subset of E training examples:
1. Construct DICL, a large set of M prompts, each consisting of K examples randomly sampled from Dtr. We then run ICL on the dev set M times given different prompts.
2. Estimate the value of each training example based on the results of Step 1. We propose two methods to do this: CONDACC (§3.1) and DATAMODELS (§3.2).
3. Select training examples with the highest importance scores per class to make up E (§3.3).

heading: 3.1 CondAcc
text: We hypothesize that a good training example leads to higher accuracy, on average, when occurs in a prompt. Given a prompt Z , we denote its dev set ICL accuracy as Acc(Z). Thus, a simple way to score the i-th training example (xi, yi) ∈ Dtr is to calculate the expected accuracy conditioned on this example appearing in a prompt Z from DICL:
sca(i) = EZ∼DICL [Acc(Z) | (xi, yi) ∈ Z] (1)
We ensure that each training example occurs in DICL multiple times in different orders and with different examples.
In Appendix A.1, we show that Eq. 1 is similar to Data Shapley value (Ghorbani and Zou, 2019), where we define the valuation function of Data Shapley on subsets of K training examples since we focus on K-shot ICL.

heading: 3.2 Datamodels
text: The CONDACC method does not consider the order of training examples, which has a great impact on ICL performance. Also, it takes a simple average over the dev set, ignoring the LLM’s confidence in individual dev examples. Thus, we propose another data valuation method that leverages Datamodels (Ilyas et al., 2022) for ICL.
Ilyas et al. (2022) study a complex target model’s behavior in terms of the training data by replacing it with a linear, easy-to-analyze, proxy model called a datamodel. Specifically, given a subset of training data S, a datamodel predicts the outcome of a
test input x̄ if training the target model on S. The outcome f(x̄;S) is defined as the margin of the correct class, i.e., the logit for the correct class minus the highest incorrect logit. To train datamodels, Ilyas et al. (2022) first create a dataset consisting of subset-outcome pairs (S, f(x̄;S)), requiring training the target model from scratch multiple times on different subsets to obtain the outcomes. In our work, the target model is a pretrained LLM, and is inference-only during ICL. Thus, our dataset collection only requires inference of the LLM multiple times with different prompts in DICL.
In particular, given a prompt Z = [z1, ..., zK ], we run ICL on a dev example (x̄, ȳ) and define an LLM’s outcome as f(x̄;Z) = o(ȳ|Z, x̄) − maxy′∈C, y′ ̸=ȳ o(y′|Z, x̄), where o(y|Z, x̄) is the output logit of the LLM on label y before softmax. We hypothesize that we can approximate an LLM’s outcome with a linear regressor taking two simple features: the existence of a training example and its index in Z , which we consider to be the most important factors in ICL.Our linear datamodel is parameterized by weights w ∈ RNtr×K and bias b ∈ R; we use w(i, j) ∈ R to denote the weight for the i-th training example appearing at index j. For each dev example (x̄, ȳ) ∈ Ddev, we train a datamodel gw,b to predict the LLM’s outcome of x̄, f(x̄;Z) ∈ R, with mean-squared error:
gw,b(x̄;Z) = K∑
j=1
w(id(zj), j) + b, (2)
min w,b
1
M
M∑
n=1
(gw,b(x̄;Zn)− f(x̄;Zn))2, (3)
where zj is the training example at index j in the
prompt Z , and id(·) maps zj back to its example ID i in the training set, i.e., zj = (xi, yi).
By definition, f(x̄;Z) > 0 if the LLM is correct on x̄. Hence, a positive w(i, j) indicates that having the i-th training example at index j in the prompt encourages answering correctly. We hypothesize that a good training example should have a beneficial effect for many dev examples, regardless of its index in the prompts. Thus, we can aggregate the datamodels of all dev examples and marginalize over all possible orders to calculate the score of the i-th training example:
sdm(i) = ∑
(x̄,ȳ)∈Ddev
K∑
j=1
1{wx̄(i, j) > 0}, (4)
where wx̄(i, j) is the weight value w(i, j) of the datamodel for x̄. Calculating the total number of positive weights empirically works better than averaging the weights of all the datamodels.
In Appendix A.3, we validate our hypothesis that we can linearize an LLM’s outcomes with two simple features. Table 6 shows that our datamodels can accurately approximate the outcomes of unseen prompts outside of DICL across different tasks.

heading: 3.3 Select Training Examples
text: Now that we assign a score for each training example (Eq. 1, 4), let C be the number of classes and E′ = E/C, we select the top-E′ training examples of each class with the highest scores to form the stable subset E ⊂ Dtr, |E| = E.

heading: 4 Experiment

heading: 4.1 Setups
text: Tasks. We experiment on 5 classification tasks: SST-2 (Socher et al., 2013), BoolQ (Clark et al., 2019), Subj (Pang and Lee, 2004), Scicite (Cohan et al., 2019), and AGNews (Zhang et al., 2015). We set the stable training subset size E = 20 for all the tasks. For binary classification tasks, we set K = 4 and do not balance the classes in the prompts. Thus, the collected DICL for a binary task covers all 24 label patterns, including prompts with all positive ([1, 1, 1, 1]) and all negative ([0, 0, 0, 0]) labels, allowing us to better understand the impact of label patterns on ICL. For multiclass tasks (Scicite and AGNews), we balance the classes, sampling a training example per class to form the prompts. Table 8 in the appendix summarizes our setups.
Data Splits. For all the tasks, we use classbalanced Dtr, Ddev, and Dtest sampled from the original training set, as we do not have the gold labels of the original test set. We choose |Dtr| = 1000 to ensure a diverse range of training examples for subset selection, and |Dtest| = 1000 for reliable evaluation. Ddev consists of 50 examples per class. All three sets are balanced, randomly sampled from the original training set, and do not overlap.
Models. We run our main experiments with two LLMs: GPTJ-6B (Wang and Komatsuzaki, 2021) and OPT-13B (Zhang et al., 2022a). More experiments on GPT-Neo-2.7B (Black et al., 2021) and OPT-6.7B can be found in Table 9 in the appendix, where our methods also work well.

heading: 4.2 Evaluation and Baselines
text: Recall that our goal is to select a training subset E ⊂ Dtr that is more stable than Dtr. To evaluate a selection method, we randomly sample 50 prompts from the selected subset E , run ICL on the test set Dtest, and report the average accuracy, standard deviation, and worst accuracy.
As shown in Zhao et al. (2021), when a prompt only contains examples of a single label, LLMs are prone to always predict that label on every test example. Thus, in our main experiments (§5.1), we ensure that every sampled prompt contains at least one example from every class. In §5.2, we separately investigate performance when the prompt contains only one label of binary tasks. We split the selected subset E into two subsets, E0 and E1, where E0 only contains negative training examples and E1 only contains positive examples. We then sample 50 prompts from E0 and E1, respectively.
Besides the two proposed selection methods, CONDACC and DATAMODELS, we design 5 baseline methods: ALL, CALIB, RANDOM, ONESHOT, and TOPPROMPTS. ALL uses the entire training set as E . CALIB uses the same prompts as ALL, but with calibration to prevent LLMs biased toward certain labels, where we closely follow the implementation of Zhao et al. (2021). RANDOM randomly selects a balanced training subset of E = 20 examples. ONESHOT first runs ICL with K = 1, using each training example alone as the prompt, and then scores the example by the corresponding ICL accuracy on Ddev; these scores are used in the same way as our main methods to select examples (§3.3). ONESHOT tests if we can extrapolate ICL performance from K = 1 to K > 1. TOP-
SST-2 BoolQ Subj Scicite AGNews
Avg std Min Avg std Min Avg std Min Avg std Min Avg std Min Avg. Tasks
PROMPTS-5 and TOPPROMPTS-10 select the union of the examples from the top-{5, 10} prompts in DICL with the highest dev set accuracy, where the subsets contain at most K × 5 and K × 10 examples, respectively. Finally, we apply baselines and our CONDACC method to the unlabeled setup (§2), named with the UN- prefix.

heading: 5 Results

heading: 5.1 Main Results
text: The proposed methods outperform all baselines. Table 1 shows the test set accuracy with different training subset selection methods. Among methods without calibration, our CONDACC and DATAMODELS methods are the most stable, achieving substantially higher average and worst-case accuracy across all tasks, and lower variances on most tasks. Compared with CALIB, our methods perform better in 8/10 setups. Overall, CONDACC and DATAMODELS outperform the no-selection baseline ALL by 7.7% and 6.3% on average, respectively.
TOPPROMPTS is the strongest baseline. Within the baselines, ALL and RANDOM have similar performance. Applying calibration improves the worst accuracy on most tasks and the average accuracy on some tasks, but is not always beneficial, especially on Scicite. ONESHOT outperforms ALL and RANDOM on SST-2 and BoolQ, but performs similarly on other tasks, indicating that we cannot extrapolate ICL behavior from K = 1 to K = 3 or K = 4. TOPPROMPTS-5 and TOPPROMPTS-10 are the strongest baselines, performing especially well on SST-2, Subj, and Scicite, showing that the training examples that compose the highestaccuracy prompts are more stable than others.
Our method works without training set labels. Randomly sampling prompts from the unlabeled training set (UN-ALL) underperforms sampling from the original labeled training set (ALL), especially on SST-2 and AGNews. This shows that gold labels do matter in ICL in general, in contrast with the findings of Min et al. (2022). However, apply-
ing our selection method to the unlabeled training set (UN-CONDACC) surprisingly outperforms not only UN-ALL but ALL (which uses correctly labeled examples), although some selected training examples actually have the wrong labels, implying that having gold-labeled prompts is not necessary for ICL. Other baselines, UN-ONESHOT and UN-TOPPROMPTS, outperform UN-ALL but substantially underperform our method. Overall, UNCONDACC outperforms baselines UN-ALL and ALL by 10.2% and 5.7% on average, respectively.
Does UN-CONDACC benfit from gold labels? We study the number of the stable training examples selected by UN-CONDACC that indeed have gold labels. In most of the tasks, the numbers are much higher than the expected numbers by majority guess, where BoolQ is the exception with half of the selected examples having wrong labels. We thus study if we can achieve even better results by correcting those selected examples that have wrong labels with their gold labels; the other correct examples in the subset remain unchanged. After the label correction on BoolQ, the average and worst accuracy drops by 1.9% and 4.5% respectively on GPTJ, 0.4% and 5.7% respectively on OPT. These results again suggest that on the one hand, ICL benefits from gold-labeled examples in most cases; on the other hand, some training examples with wrong labels can surprisingly achieve better performance. The full results are in Table 10 in the appendix.
Finally, we study the alternative that uses more shots in A.8. Table 12 shows that using 4 curated examples (CONDACC) can outperform K = 24, 16 randomly sampled ones in SST-2 and AGNews.

heading: 5.2 Single-Label Prompts
text: We now evaluate whether it is possible to achieve good accuracy while only using training examples of a single class in a prompt (See §4.2 for more details). Table 2 compares the results of different methods. First, the baselines ALL and TOPPROMPTS perform near chance in most cases, as the LLMs are biased by the prompts to predict the same label on every test example. In contrast, single-label prompts sampled from the subsets of CONDACC and DATAMODELS substantially outperform majority guessing across all setups. We conclude that the selected training examples are beneficial because they help LLMs understand the overall definition of the task. Thus, even when used in single-label prompts, they can still give LLMs
useful signal to perform the desired task.

heading: 5.3 Out-of-Distribution Tasks
text: We further evaluate on out-of-distribution (OOD) tasks, where there is a distribution shift between prompts and test data. Specifically, we apply our selection methods on a source task, sampling 50 prompts from the selected subsets of the source task as done in the main experiments, and then evaluate on test data of a target task. We use SST-2 and BoolQ as the source tasks, and IMDb (Maas et al., 2011) and BoolQ Contrast Set (Gardner et al., 2020) as our target tasks, respectively. Table 3 shows that our CONDACC and DATAMODELS methods still outperform baselines on OOD tasks, especially on IMDb, implying that instead of simply overfitting the source tasks, the selected stable examples are indeed task-level examples that can generalize to OOD test data.

heading: 6 Analysis
text: We further analyze what makes the selected training examples special along different dimensions: sequence length, perplexity, and diversity. We compare good with bad training examples, where we use our CONDACC method to identify a bad (resp., good) subset by selecting E′ training examples with the lowest (resp., highest) importance scores in each class (§3.3). Please refer to Table 11 in the appendix for the full results of the bad subset.

heading: 6.1 Sequence Length and Perplexity
text: In Figure 3, we plot the accuracy against either sequence length or perplexity, where each dot corresponds to a training example. Here, the accuracy (Y-axis) is the importance score sca assigned to each training example by CONDACC in Eq. 1, i.e., the average dev-set accuracy when that example is combined with random other training examples in a random order.
Sequence Length. The first two subfigures show that while the bad examples (red dots) span across different sequence lengths, the good examples (blue dots) do not cover the tail distribution of long sequences. We observe little correlation between accuracy and sequence length across different tasks and LLMs (see more in Figure 6), except for a slightly negative correlation when the sequence length is very long, suggesting that abnormally long training examples can hurt ICL performance.
Perplexity. We calculate the perplexity of the inputs of training examples with respect to the same LLMs we run ICL on. Figure 3 shows that good examples are not outliers that have high perplexity. This could suggest future work filter out examples that have extraordinarily high perplexity in the training set before running ICL, and could be combined with active learning for ICL (Zhang et al., 2022b; Su et al., 2022), as we only need the
unlabeled inputs to calculate perplexity. However, we observe no correlation between accuracy and perplexity across all the tasks and LLMs (Figure 7), implying that using perplexity alone is not enough for identifying good training examples. Our findings are inconsistent with concurrent work (Gonen et al., 2022), which shows that lower prompt perplexity strongly correlates with better performance, probably because Gonen et al. (2022) focus on perplexities under different instructions, while we focus on the differences between training inputs.

heading: 6.2 Diversity
text: DIV-I and DIV-F. We measure the diversity of a training subset with DIV-I (Yuan et al., 2020) and DIV-F (Zhdanov, 2019) metrics, following prior work in active learning. DIV-I measures diversity in raw text, while DIV-F measures diversity in a feature space. For DIV-F, we use SentBERT (Reimers and Gurevych, 2019) to encode the inputs of training examples into sentence embeddings, following Su et al. (2022). We compare the selected good subset and bad subset with 5000 randomly sampled subsets ⊂ Dtr, each containing E′ training examples per class. Figure 4 shows that good subsets (blue dots) sometimes have low diversity scores in both metrics, especially on BoolQ and AGNews. Overall, they do not seem to be more diverse than randomly sampled subsets. Our findings are different from Su et al. (2022), which finds that diverse training subsets are better for prompt retrieval. We hypothesize that diversity matters more when retrieving similar training examples for each test input, but is less important when using a single fixed prompt. On the other hand, the bad subsets have much higher DIV-I scores than random subsets across different tasks, because they often include examples with long sequence lengths (see Figure 3), covering more distinct unigrams. However, in the SentBERT feature space, the bad subsets are often less diverse than random subsets.
Datamodels Embeddings. In §3.2, we learn a datamodel for each dev example in Ddev. Here, we concatenate the weights assigned on a training example learned by all the datamodels, creating an embedding ∈ R|Ddev|×K for each training example. We then project the embeddings of the entire training set to a two-dimensional space with UMAP (McInnes et al., 2018) for visualization. Figure 4 and Figure 8 in the appendix show that both CONDACC and DATAMODELS choose tightly clustered sets of examples in the embedding space, instead of diverse ones scattering over the training set. Moreover, the two methods actually select similar examples in the embedding space, although having very different scoring methods.

heading: 6.3 Do LLMs find the same stable examples?
text: We further study if the identified stable subset examples are transferable across different LLMs. Given two LLMs, we calculate the Pearson correlation coefficient between their example scores assigned by CONDACC (§3.1) and the actual number of overlapped examples in the stable subsets of the two LLMs. Table 4 shows the mixed results: some pairs have moderate correlation, especially when both LLMs are from the OPT family; however, we find little correlation between many pairs and there are only a few overlapped examples between the stable subsets identified by different LLMs.
Interestingly, when using the four stable SST2 examples shared by GPTJ-6B and OPT-13B as the prompt, evaluating all 4! permutations, we achieve very high test accuracy: 88.6 ± 3.7 on GPTJ, 89.8± 3.0 on OPT-6.7B, and 87.3± 5.2 on OPT-13B. This may indicate that there exist successful factors of training examples shared among LLMs. We include the four stable examples in
Table 5 and hope future work can discover what distinguishes them from other examples.

heading: 7 Discussion and Related Work

heading: 7.1 Prompt Retrieval
text: The performance of ICL greatly depends on the choice of the prompt. A common way to do prompt selection is to retrieve the top-K similar training examples for each test input (Liu et al., 2021; Rubin et al., 2021; Su et al., 2022), where the similarity is captured by sentence embeddings (Reimers and Gurevych, 2019; Robertson et al., 2009). Such instance-dependent prompt retrieval is critical for semantic parsing tasks (Rubin et al., 2021), as LLMs need to see relevant logical forms in the context to generate the appropriate predicates for the test example. In this paper, we focus on classification tasks and identify task-level training examples that work for all test examples, avoiding the retrieval process.

heading: 7.2 The influence of in-context labels
text: The proportions of labels appearing in context can greatly bias LLMs’ predictions (Zhao et al., 2021). However, with careful prompt selection, Zhang et al. (2022b) finds that LLMs can perform well without observing the entire label space of a classification task in the prompt. In §5.2, we identify a subset, instead of just one prompt, of single-label examples that perform well. On the other hand, the correctness of in-context labels may not matter as much, as Min et al. (2022) find that randomly flipping them barely hurts performance. Kim et al. (2022) re-examine the importance of gold labels, showing it varies largely across tasks and experimental settings; our unlabeled experiments also show varying importance of gold labels across different tasks. Our method also identifies some training examples with wrong labels that can yield surprisingly good performance.

heading: 7.3 Data Valuation
text: Given a learning algorithm trained on a training set to produce a predictor, data valuation quantifies the value of each training example to the predictor performance. Prior work includes influence functions (Koh and Liang, 2017), Data Shapley (Ghorbani and Zou, 2019), DVRL (Yoon et al., 2020), TracIn (Pruthi et al., 2020), and Datamodels (Ilyas et al., 2022). Our setup also aims to attribute the performance of a predictor (in our case, an LLM) to each training example (in our case, in-context examples). Our CONDACC method closely resembles Data Shapley, and we adapt Ilyas et al. (2022)
as our DATAMODELS method. However, the main difference is that we are doing K-shot ICL, where training examples are used as prompts, and there are no parameter updates to LLMs.
In concurrent and independent work, Nguyen and Wong (2023) also propose methods based on Data Shapley and Datamodels to study the influence of training examples. The main differences are: (1) we adapt Datamodels to consider the positions of training examples, while Nguyen and Wong (2023) follow the original implementation and study example ordering with influence scores. (2) They propose a Perplexity baseline that selects examples according to individual perplexity, while we use perplexity in analysis to study the correlation between example perplexity and their average performance. (3) Nguyen and Wong (2023) explore a larger number of shots in ICL (K ∈ [10, 52]), while we assume a few-shot setting and fix K ∈ {3, 4}. Our findings on good training examples are consistent with each other: both papers find little correlation between performance and potential factors such as example length and perplexity. In general, our work demonstrates the importance of data curation on in-context examples, even in the unlabeled and OOD scenarios, while Nguyen and Wong (2023) focus more on developing influence-based example selection frameworks. Taken together, two papers present a comprehensive view of data valuation for in-context learning.

heading: 8 Conclusion
text: We propose two methods to identify stable training subsets for in-context learning, achieving substantially higher average and worst-case accuracy across different setups. Our CONDACC method is intuitive and easy-to-implement, while our DATAMODELS method provides informative weights that enable further analyses. The success of our methods implies that when provided with the “right” training set (in our case, a subset of 20 examples to randomly sample prompts from), ICL could be far less sensitive to the choice of training examples and their orders in a prompt. Our analyses on stable subsets find that they do not contain outliers with especially long sequence lengths or high perplexities, and are also no more diverse than random subsets of the training data. We hope our work is a step towards developing guidelines for finding or writing better training examples.
Limitations
The main limitation of our work is the high memory and computation cost. As both our methods estimate the importance of training examples based on the prompt-performance statistics, we first need to run in-context learning on the dev set multiple times with different prompts in DICL. Although ICL does not require any parameter updates, LLMs still require a large amount of memory footprint during inference, especially when the model size is large and the average sequence length is long. For each setup, our DICL contains around 50,000 prompts (see Table 8) and 50 dev examples per class, so the most expensive setup (running OPT13B on BoolQ) costs more than 500 GPU hours on an RTXA6000 GPU. Our preliminary study shows that the proposed methods need the statistics of at least 10,000 randomly sampled prompts to perform well. Future work may use search algorithms instead of random sampling during data collection to reduce the number of prompts in DICL. We also release our collected data of every setup in https://github.com/terarachang/DataICL to support future studies on ICL.
In this paper, we only study classification tasks, for the sake of easy evaluation. Future work may study the influence of in-context examples in generation tasks under different evaluation metrics. Due to hardware constraints, we do not study LLMs of sizes over 13B, and we fix the number of shots and prompt templates for simplicity. In independent work, Nguyen and Wong (2023) complement these limitations of our paper, showing that similar approaches work well on larger models and a diverse number of shots for in-context example selection. Still, the influence of in-context examples for gigantic LLMs larger than 100B parameters has not been studied. Due to emergent abilities of LLMs (Wei et al., 2022), it is unclear whether our methods and findings would still apply when prompting these gigantic LLMs.

heading: Acknowledgements
text: We thank Ameya Godbole, Johnny Wei, Wang Zhu, Albert Xu, Deqing Fu, Qinyuan Ye, the members of USC NLP, and our anonymous reviewers for valuable feedback on the paper. We thank Sameer Singh and Matt Gardner for helpful discussions. We thank Jesse Thomason for his support. This work was funded in part by gifts from Open Philanthropy and Google.

heading: A Appendix
text: A.1 Connection with Data Shapley Data Shapley has a valuation function V (S) for any subset of training examples. We define V (S) as the expected dev set ICL accuracy across all permutations of S if |S| = K, and V (S) = 0 otherwise since we focus on K-shot learning. Then, the Data Shapley value sshap(i) for the i-th training example ei = (xi, yi) is:
E{z1,...,zK−1}∼(Dtr\eiK−1 ) [ V ( {z1, ..., zK−1, ei} )]
−E{z1,...,zK}∼(Dtr\eiK ) [ V ( {z1, ..., zK} )] .
We claim that sshap(i) is a monotonically increasing (in fact, affine) function of sca(i), thus establishing a very tight connection between CONDACC and Data Shapley.
First, note that the first term is exactly equal to sca(i), the expected conditional accuracy when ei occurs in a prompt, since V returns the expected accuracy over all orders of {z1, . . . , zK−1, ei}.
Similarly, the second term is the expected conditional accuracy when ei does not occur in a prompt. Denote this quantity as t(i), and let A denote the overall expected accuracy when randomly sampling a prompt. Since the probability of choosing a given example to be in a prompt is exactly KNtr , we have
A = K
Ntr sca(i) + Ntr −K Ntr t(i)
t(i) = Ntr
Ntr −K
( A− K
Ntr sca(i)
)
= Ntr Ntr −K A− K Ntr −K sca(i).
Now we can rewrite sshap(i) as follows:
sshap(i) = sca(i)− t(i)
= sca(i)− (
Ntr Ntr −K A− K Ntr −K sca(i)
)
= Ntr
Ntr −K A+ Ntr Ntr −K sca(i).
Since Ntr, K, and A are all constants that do not depend on i, and Ntr,K > 0, this establishes that sshap(i) is a monotonically increasing affine function of sca(i).
A.2 Relation to Prompt Tuning At test time, our setup is similar to Prompt Tuning (Lester et al., 2021), where a fixed prompt for
a task is prepended to the test inputs. In our case, a prompt is a sequence of training examples randomly drawn from the selected subset (fixed for all test examples). In Prompt Tuning, it is a set of continuous embeddings, called a soft prompt, learned through backpropagation.
At training time, however, Prompt Tuning needs to backpropagate through the LLM and thus is more memory-expensive and tends to suffer from training instability (Asai et al., 2022). In comparison, our method finds good prompts without accessing the LLM’s parameters, which is a realistic setup as many LLMs only provide API access.
A.3 Evaluating Datamodels
Recall that given a prompt Z and a dev example (x̄, ȳ), a datamodel learns to approximate an LLM’s outcome of x̄ (§3.2). To evaluate how accurate our datamodels can approximate an LLM, we create a test set for datamodels DDM, consisting of 5000 pairs of newly sampled Z and the LLM’s groundtruth outcomes of every dev example, where our sampling assures that every Z is made up of an unseen combination of training examples. We evaluate the learned datamodels on DDM, calculating the correlation and L1 distance between the predicted outcomes and the ground-truth outcomes. More specifically, each datamodel yields 5000 outcomes, which we calculate the Pearson correlation coefficient with the ground-truth outcomes of the associated dev example. We report the average correlation and L1 distance over all datamodels in Table 6. We also randomly sample 10,000 outcomes across all datamodels to visualize the ground truths against predictions in Figure 5.
Overall, our datamodels can accurately approximate LLMs’ outcomes across different tasks and
LLMs. As our datamodels only consider two simple features, the existence of a training example and its index in Z , accurate test predictions may imply that these two features have a dominating effect on ICL.
A.4 Training Details of Datamodels
As the pattern of in-context labels (e.g., [0,0,0,0], [0,0,0,1], [1,0,0,1]) has a great impact on LLMs’ predictions, for each label pattern, we train a set of |Ddev| datamodels. Specifically, we apply twophase training: in the first phase, we train on all data with shared weights. In the second phase, we first bucket prompts in DICL by their label patterns. Initializing from the weights learned in the first phase, we then separately fine-tune a set of datamodels for each label pattern. For example, for a binary task with 4-shot learning, there are 24 = 16 label patterns; thus, we have 16 sets of datamodels after the second-phase training, namely, 16×|Ddev| datamodels in total. We find that having two-phase training leads to more accurate predictions of LLMs’ outcomes in appendix A.3. Thus, when assigning scores for training examples, we aggregate all sets of datamodels to obtain sdm. When creating datamodel embeddings (§6), we use the unified weights learned by the first phase.
A.5 Implementation Details
We use PyTorch and Huggingface transformers to implement in-context learning on GPT-Neo2.7B (Black et al., 2021), GPTJ-6B (Wang and Komatsuzaki, 2021), OPT-6.7B (Zhang et al., 2022a),
and OPT-13B. We run all our evaluations on a single RTXA6000 GPU (48GB). Most of our experiments can also be run on an RTX3090 GPU (24GB), except that OPT-13B model requires a GPU with larger memory.
Our data collection on DICL costs hundreds of GPU hours on an RTXA6000. Once we finish the collection, our DATAMODELS method only takes about 5 seconds to train a datamodel on an i7-10700 CPU, and our CONDACC method does not involve any training, but simply calculates accuracy.
Table 7 shows our task templates and label mappings, where we closely follow Bach et al. (2022); Lu et al. (2021). Table 8 summarizes our experimental setups on different tasks.
A.6 More Experiments Table 9 shows experiments on more LLMs and the MNLI task, where we evaluate on test data using 50 sampled prompts, as done in the main experiments (§4.2). Since ICL performs poorly on MNLI (majority: 33.3%) in both prior work (Su et al., 2022) and our results in Table 9, we do not experiment more on this task. Overall, our methods CONDACC and DATAMODELS substantially outperform other 4 baselines on all setups.
A.7 Why not Instruction-Finetuned LLMs? The massive multitask learning in instructiontuning leads to leakage in the datasets we evaluate on. For example, T0 (Sanh et al., 2021), FLANT5 (Chung et al., 2022), and OPT-IML (Iyer et al., 2022) are all trained on several tasks in our paper.
A.8 Larger Number of Shots We further compare our methods with an alternative that takes as many labeled, balanced, incontext examples as the context window can fit, named MAXSHOT. Similar to the ALL baseline, MAXSHOT samples 50 prompts from the entire training set, each containing K training examples. The differences are that MAXSHOT uses a much larger K and balances the classes in the prompt for binary tasks as well. Here, our LLM is GPTJ-6B, which has a context window of 2048 tokens.
Table 12 shows the number of shots K for each task and compares the average test set accuracy of MAXSHOT with the ones of ALL and CONDACC in Table 1. The ∆All column shows that using K ∈ [8, 24] examples in the prompt substantially improves over only 3 or 4 examples in most tasks, except for AGNews. However, MAXSHOT only outperforms CONDACC on two out of five tasks (∆Our, blue). This shows the advantages of curated training examples over randomly sampled ones.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations section
7 A2. Did you discuss any potential risks of your work? We use public available language models and commonly used datasets
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Sec1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Sec 3,4
3 B1. Did you cite the creators of artifacts you used? Sec 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We use public available artifacts
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Sec 4 and Limitations
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Sec4, Table6
C 3 Did you run computational experiments? Sec5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix A.5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Sec 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sec 4
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future work may use search algorithms instead of random sampling during data collection to reduce the number of prompts in DICL. We also release our collected data of every setup in https://github. com/terarachang/DataICL to support future studies on ICL. In this paper, we only study classification tasks, for the sake of easy evaluation. Future work may study the influence of in-context examples in generation tasks under different evaluation metrics. Due to hardware constraints, we do not study LLMs of sizes over 13B, and we fix the number of shots and prompt templates for simplicity. In independent work, Nguyen and Wong (2023) complement these limitations of our paper, showing that similar approaches work well on larger models and a diverse number of shots for in-context example selection. Still, the influence of in-context examples for gigantic LLMs larger than 100B parameters has not been studied. Due to emergent abilities of LLMs (Wei et al. , 2022), it is unclear whether our methods and findings would still apply when prompting these gigantic LLMs