On the Noise Robustness of In-Context Learning for Text Generation

Authors: hongfu  gao
feipeng  zhang
wenyu  jiang
jun  shu
feng  zheng
hongxin  wei
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:08:36

Abstract: Large language models (LLMs) have shown impressive performance on downstream tasks by in-context learning (ICL), which heavily relies on the quality of demonstrations selected from a large set of annotated examples. Recent works claim that in-context learning is robust to noisy demonstrations in text classification. In this work, we show that, on text generation tasks, noisy annotations significantly hurt the performance of in-context learning. To circumvent the issue, we propose a simple and effective approach called Local Perplexity Ranking (LPR), which replaces the "noisy" candidates with their nearest neighbors that are more likely to be clean. Our method is motivated by analyzing the perplexity deviation caused by noisy labels and decomposing perplexity into inherent perplexity and matching perplexity. Our key idea behind LPR is thus to decouple the matching perplexity by performing the ranking among the neighbors in semantic space. Our approach can prevent the selected demonstrations from including mismatched input-label pairs while preserving the effectiveness of the original selection methods. Extensive experiments demonstrate the effectiveness of LPR, improving the EM score by up to 18.75 on common benchmarks with noisy annotations.

Main paper: 
 introduction: 
 Large language models (LLMs) have shown remarkable performance on downstream tasks by incontext learning (ICL) with only a few task demonstrations [7, 10] . Without requiring explicit parameter updates, in-context learning consistently outperforms zero-shot inference on various tasks (e.g., classification and generation), making it a compelling alternative to supervised fine-tuning [13, 16] . In particular, the success of ICL heavily relies on the quality of demonstrations selected from a large set of annotated examples [21, 29, 51, 60] . For those candidates, input-label mappings solicited from humans [61, 73] or LLMs [58] can often be noisy, especially in complex tasks. This gives rise to the importance of noise-robust ICL, which aims to construct effective demonstrations in the presence of noisy and erroneous labels.Previous works show that in-context learning on classification tasks is fairly robust to label noise in the in-context demonstrations [9, 12, 32, 37, 54, 55] . However, it is still mysterious how noisy labels affect the performance of ICL on text generation tasks. In this work, we present the first study on in-context learning with a noisy annotated dataset for generation. Surprisingly, we empirically find that label noise in the demonstrations significantly degrades ICL's performance on generation tasks, which is different from previous results on classification. Moreover, increasing the number of selected demonstrations with a fixed noise rate or utilizing more effective selection methods (e.g., TopK [28] and DPP [62] ) will intensify the negative effect of noisy labels. This motivates our method, which can universally improve the noise robustness of existing selection methods for in-context learning.In this paper, we show that the issue of noisy annotations can be mitigated through the perplexity ranking of noisy candidates (i.e., input-label pairs) during selection. Our method, Local Perplexity Ranking (dubbed LPR), is motivated by our analysis of the perplexity deviation caused by noisy labels (i.e., incorrect answers). We find that wrong answers generally result in a higher perplexity of large language models compared to correct ones, in response to the same question. To explain this phenomenon, we decompose the perplexity into two components: inherent perplexity, which measures the task complexity of the question and the correct answer, and matching perplexity, which assesses the perplexity deviation caused by noisy outputs. Therefore, our key idea behind Local Perplexity Ranking is to decouple the matching perplexity by performing the ranking among the neighbors in semantic space. This can be achieved by ranking candidates' perplexity alongside their nearest neighborhoods, which usually have similar levels of inherent perplexity. In particular, we replace each low-rank candidate selected by existing methods (e.g., random, TopK, and DPP) with its nearest neighbor that is highly ranked. In effect, our LPR strategy can prevent the selected demonstrations from containing mismatched input-label pairs while preserving the effectiveness of the original selection methods. In this way, we ensure the correctness and relevancy of demonstrations, thereby improving the noise-tolerant ability of in-context learning.To verify the effectiveness of our method, we conduct extensive evaluations on six text generation datasets, including NQ [22] , WebQ [5] , SQuAD [46] , SCIQ [56] , GeoQuery [39] and NL2Bash [27] datasets. The results demonstrate that local perplexity ranking can largely improve the noiserobustness of all existing selection methods under irrelevant and relevant noises. For example, on SCIQ with 60% irrelevant label noise, LPR improves the exact match score of the TopK method from 29.31 to 48.06 -a significant direct improvement of 18.75. Moreover, our method can be easily adopted in practice. The performance of LPR is insensitive to the hyperparameters, including the threshold γ and the number of local neighbors k. This approach can effectively generalize to various LLMs to improve their noise-robustness with in-context learning.Our contributions are summarized as follows:• We present the first study to show that annotation quality is crucial for in-context learning in text generation, where noisy annotations significantly hurt the performance. Increasing the set size of demonstrations cannot bridge the gap, as well as picking other selection methods. • We propose Local Perplexity Ranking (LPR), a simple and effective method to enhance the noise robustness of in-context learning. The key idea is to decouple the matching perplexity by performing the ranking among the neighbors of each candidate in semantic space. • We empirically show that LPR can improve the noise robustness of existing demonstration selection methods in ICL across various types of label noise. In addition to text generation, we also validate the effectiveness of our method in text classification tasks.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The abstract and introduction clearly state our scope, motivation, method, experimental results and contribution. See section 1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
preliminary 2.1 in-context learning for generation: 
 We consider in-context learning (ICL) of large language models (LLMs) in generation tasks, where we aim to generate text outputs y = (y 1 , ..., y |y| ) (i.e., token sequences) conditioned on the inputs x = (x 1 , ..., x |x| ) and the context C K . In particular, the context C K = {(x i , y i )} K i=1 contains K task demonstrations (e.g., input-output pairs), selected from a large annotated dataset with N examples D = {(x j , y j )} N j=1 . Given a new test input text x test , we make the generation of output y test via large language models asEQUATIONwhere ∼ refers to decoding strategies(e.g. greedy decoding and nuclear sampling [17, 62] ). Generation with the ICL procedure is especially attractive as it does not require the parameter updating of large language models, which is often expensive and impractical. Existing studies show that the selection strategy of demonstration plays a crucial role in the ICL performance [25, 31, 43, 44, 47] . A naive method is to randomly sample the demonstrations from annotated examples without repetition [36] . To introduce the relevancy, TopK [28] proposes to select the closest examples to the test input in the embedding spaceC K = R K (x test ) = TopK x (s(x test , x)),where R is a retriever, s(x test , x) denotes the cosine similarity score between x test and examples x from the annotated dataset. We use TopK to denote the top K examples ranked by the score.These selection strategies focus on the inputs of demonstrations, assuming that all examples are labeled correctly in the large dataset [28, 36, 62] . However, collecting a large-scale dataset with perfectly correct labels is challenging and expensive, especially for generation tasks [2, 64] . In practice, researchers often use crowdsourcing [61, 73] or large language models (LLMs) [58] such as GPT-4 [38] to create input-output pairs for new tasks, which inevitably leads to some mistakes in the annotations. This motivates us to analyze the issue of label quality in ICL for generation tasks.Given a large-scale dataset with noisy annotations D = {(x j , ỹj )} N j=1 , the selected demonstration might contain mismatched input-output pairs (x, ỹ), i.e., the output ỹ might be not a correct answer to the input x. Conditioned on the noisy demonstrations, the generation of output via ICL is made asEQUATIONIn the real world, noisy annotations may arise from unintentional mistakes or limited knowledge, resulting in various types of noise in the demonstrations. In this work, we define two categories of noisy annotations based on the input-output relevance, as follows:Irrelevant noise assumes that the generation of noisy annotations is conditionally independent of inputs. For example, crowdsource workers may make mistakes accidentally, introducing random words or sentences in annotations. This can be simulated by reconstructing the output with random words from a subset that does not contain tokens presented in the original input-output pairs.Relevant noise is a more realistic setting where the corrupted output is relevant to the inputs despite its incorrectness. This type of corruption may occur due to the limited knowledge of annotators and LLMs. We simulate the relevant noise by generating related yet incorrect outputs using ChatGPT-4.In Table 1 , we present an ICL example of question answering (QA) tasks to illustrate the difference between the two noisy settings. In this example, the clean annotation for the test input is "Cells". For noisy annotations, the irrelevant noise is randomly sampled as "Earth", while the relevant noise "tissues" exists in the support of in-context demonstration. We proceed by analyzing the empirical effects of noisy annotations in generation tasks. Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Our approach is suboptimal in cases of high noise rates due to the assumption that clean annotations are the majority in the dataset. In addition, we do not provide a theoretical analysis to show how noisy annotations affect ICL, which will be an interesting direction for future research. See section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
empirical study of noisy icl in text generation: 
 In this section, we investigate the impact of noisy annotations on the performance of in-context learning for text generation. In particular, we conduct experiments on three types of generation tasks, including: question answering (NQ [22] , WebQ [5] ), reading comprehension (SQuAD [46] , SCIQ [56] ), code generation (GeoQuery [39] , NL2Bash [27] ). To simulate the noise, we generate noisy annotations with a pre-defined probability (e.g., 20%, 40%, 60%) in the annotated datasets. We use the output of an input from a different generation task as irrelevant noise, and adopt ChatGPT-4 to generate relevant yet false outputs as relevant noise. Furthermore, we compare the performance of noisy ICL with demonstrations across various set sizes (e.g., 2, 4, 8) and selection methods, including Random [36] , TopK [28] and DPP [62] . Following previous work [16, 28, 62] , we report the average Exact Match (EM) score with Llama2-7B [49] .ICL is not robust to noisy annotations in text generation. Figure 1 presents the empirical results of ICL methods with noisy annotations. The results show that both the two types of noises significantly deteriorate the performance of in-context learning on text generation tasks, which is different from the observations of ICL on classification tasks [9, 12, 32, 37, 54, 55] . In particular, a higher noise rate in annotated datasets leads to poorer performance of in-context learning. Moreover, irrelevant noises have a more negative influence than relevant noises, which may benefit the inference in the way of task recognition [40] .The impact of demonstration selection. To provide a deep understanding of noisy annotations, we analyze the performance of noisy ICL across different demonstration settings, including the set size (i.e., K) and selection methods. Results in Figure 1 show that, under the noisy settings, selecting a larger set of demonstrations does not enhance -and may even worsen -the performance of text generation. For example, the ICL performances with K = 8 are basically lower than those with K = 2, which is inconsistent with the clean setting. In addition, the advantages of those powerful selection methods (i.e., TopK and DPP) are neutralized in the presence of noisy annotations.Through the empirical analysis, we find that noisy annotations significantly hurt the performance of ICL in text generation tasks. More importantly, increasing the set size of demonstrations cannot bridge the gap, as well as picking an existing selection method, like DPP. This motivates us to design noise-robust methods, which can universally improve the noise robustness of in-context learning.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: Our approach is built on two natural assumptions that are naturally satisfied in the real world. In the literature, the assumptions are also supported by previous findings that paragraphs whose representations are close to each other share the same intrinsic task. See section 1.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
methodology: 
 In this section, we first analyze the perplexity deviation caused by noisy annotations and introduce the disentanglement of perplexity to explain the phenomenon. In light of this, we propose a novel method -local perplexity ranking -to improve the noise robustness of in-context learning for text generation. Our method can be easily incorporated into existing methods of demonstration selection.For language models, perplexity measures the degree of uncertainty in generating new tokens. In particular, a low perplexity indicates that the model makes the prediction with high confidence. Therefore, perplexity is commonly used to evaluate the language quality of generated content, e.g., detecting attack prompts [3] , out-of-distribution instances [4, 57] , hard-to-learn instances [13] , and corrupted instances [64] . In light of this, we conjecture that mismatched input-output pairs may result in higher perplexity of LLMs due to their low co-occurrence rate. For instance, in the example presented in Table 1 , the term "earth" rarely co-occurs with "cells" and "organ", so LLMs are more likely to exhibit high perplexity in the input-output pair.Empirical study To validate this assumption, we compare the perplexity of clean and noisy annotations in text generation tasks. Specifically, we concatenate each tokenized input-output pair (x, y), and obtain the corresponding tokenized sequence z = (z 1 , ..., z |z| ) = (x 1 , ..., x |x| , y 1 , ..., y |y| ), where |z| = |x| + |y|. Now, the perplexity of z is calculated as:EQUATIONwhere log p θ (z i |z <i ) is the log-likelihood of the i-th token conditioned on the preceding tokens z <i , from the given language model parameterized by θ.In Figure 2 , we present the perplexity distribution of Llama2-7B [49] on clean and noisy annotations of four datasets. The results illustrate that examples with noisy annotations indeed obtain higher perplexity than those with clean annotations, which confirms our assumption. In particular, relevant noises achieve slightly lower perplexity than irrelevant noises since relevant outputs are close to the inputs despite their erroneous information. However, the deviation of the perplexity distribution caused by noisy annotations is marginal, making it suboptimal to differentiate noisy annotations from clean ones. In the following, we explain this phenomenon with the disentanglement of perplexity.Disentanglement of perplexity Given an input-output pair, the perplexity of large language models (LLMs) stems not only from how well the output matches the input, but also from the inherent complexity of the task. For example, a mathematical question with a correct answer can exhibit a higher perplexity than a question of daily life with an incorrect answer. Informally, we decompose the overall Perplexity into two components3 , as shown below:Perplexity = Inherent Perplexity + Matching PerplexityHere, the inherent perplexity measures how the model is familiar with the task (i.e., the input and the correct output). The matching perplexity quantifies the perplexity deviation caused by noisy outputs, so it can be zero with correct outputs. A higher matching perplexity indicates that the output is more likely to be incorrect for the input. However, directly computing the matching perplexity is non-trivial as clean outputs are unknown. To circumvent the issue, we aim to design an effective method to decouple the matching perplexity from the overall perplexity.Intuition Motivated by the previous analysis, we propose local perplexity ranking (LPR), a general strategy that can improve the noise robustness of in-context learning. Our key idea is to decouple the matching perplexity by performing the ranking among the neighbors in semantic space. Here, our approach is built on two natural assumptions that are naturally satisfied in the real world:1. The clean annotations are the majority in the annotated dataset.2. Examples that are semantically similar share the same level of inherent perplexity.In the literature, Assumption 2 is also supported by previous findings that paragraphs whose representations are close to each other share the same intrinsic task [14, 28, 73] . With the two assumptions, we can approximate the inherent perplexity of a candidate through its neighbors, where most examples are correctly annotated. In other words, the candidate is more likely to be wrongly annotated if its perplexity is relatively higher than its neighbors, and vice versa. With this in mind, we present the details of our approach in the following.Finding the local neighbors Given a test input, we first sample a candidate set C with a pre-defined selection strategy, such as Random [36] , TopK [28] or DPP [62] . For each candidate z * , we adopt k-Nearest-Neighbors (k-NN) to find its local neighbors that are close to the candidate in token space. Formally, the k local neighbors are obtained as:N k (z * ) = {z π(1) , z π(2) , ..., z π(k) }, where π(i) is the index of the example with the i-th smallest distance to the candidate. In particular, we use the cosine similarity score to measure the distance between the candidate z * and other examples z:cos(z i , z * ) = z ⊤ i z * ||z i || 2 ||z * || 2 .Ranking the perplexity As discussed above, the local neighbors share the same level of inherent perplexity, which enables the comparison of their matching perplexity. For each candidate z * , we propose to rank the perplexity of examples in the cluster of local neighbors z * ∪ N k (z * ). Formally, we first sort all examples in the cluster in increasing order by the perplexity and obtain the original indices for the sorted scores as:EQUATIONwhere Perplexity(•) is the overall perplexity defined in Equation 3. In this way, the high-ranking examples are more likely to be correctly annotated than the low-ranking example in the sorted list I.Substituting the noisy candidates To build the final demonstration set, we propose to replace the noisy candidates with their nearest neighbors that are more likely to be clean. In particular, we can determine whether a candidate should be replaced by:EQUATION)where γ is the pre-defined threshold (e.g., 50%), 1(•) is the indicator function and Loc(z n , I) return the index of z n in the sorted list I. It is worth noting that the proposed method is not sensitive to the value of the hyperparameter γ, as shown in Subsection 5.1. Then, for those candidates with g(z n ), we pick the substitutes from their neighbors by:min{i ∈ N k |g(z π(i) ) = 0},where π(i) is the index of the example with the i-th smallest distance to the candidate. After the replacement, we establish the final demonstration set for in-context learning. Noticeably, our method offers several compelling advantages:• Algorithm-agnostic: LPR can be easily incorporated into existing demonstration selection methods, consistently improving the robustness against noisy annotations. • Easy to use: LPR does not require heavy hyperparameter tuning, as it is insensitive to the threshold value (see Figure 3 ). LPR does not introduce much computational cost due to the efficient computation of perplexity (see Table 4 ). Models and ICL methods. For the main results, we use Llama-2-7B-Chat [49] as the LLM throughout our experiments. We also provide experiments on other models including Llama2-13B-Chat [49] , Mistral-7B [19] and OPT-6.7B [66] . We use bert-base-uncased sentence encoder as the similarity tokenizer [11, 62] . We conduct experiments with existing demonstration selection methods, including Random [36] , TopK [28] and DPP [62] . For hyperparameters, we set the number of neighbors k = 4 and the threshold γ = 50% by default. The details of our implementation is presented in Appendix A.2.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We detailedly introduce to our method and provided code and data to make our experiment reproducible. See section 4. 
main results: 
 Can LPR improve the noise-robustness of in-context learning? Table 2 presents the average in-context learning performance of the baselines and our method on six generation tasks, under various types of noisy annotations. A salient observation is that our method drastically improves the the noise-robustness performance of the existing demonstration selection methods by employing LPR. For example, on the NQ with 60% irrelevant noise, our approach improves the EM score of the naive random selection method from 28.21 to 42.00 -a 13.79 of direct improvement. Moreover, we show that the LPR can boost performance for a wide range of existing demonstration selection methods such as TopK [28] and DPP [62] . For example, we observe that, on SCIQ with 60% irrelevant label noise, LPR improves the exact match score of the TopK method from 29.31 to 48.06 -a significant direct improvement of 18.75. Our method also establish strong robustness against all types of noisy annotations. Appendix A.3 reports the results with various demonstration sizes.How does the threshold γ affect the noise-robustness of LPR? In Figure 3 (a) and (b), we ablate how the parameter γ in our method (cf. Eq. 5) affects the noise-robust performance. The base indicates all candidate demonstrations are selected without our method. It's noteworthy that LPR shows robustness to the choice of threshold γ, even if we set γ = 75% also yield significant EM score improvements. We can also observe that as the threshold γ decrease, the noise-robust performance also improve, especially under 60% noise conditions. Due to space constraints, we only report the average results of multiple baselines on various generation tasks.Does LPR work with the different number of k nearest neighbors? We evaluate how the number of nearest neighbors k in our method affects the LPR performance. Specifically, We vary the number of neighbors k = {2, 4, 6}. As is shown in Figure 3 (c) and (d), an increase in the number of nearest neighbors beyond 0 leads to an evident improvement in EM score, and the performance starts to reach a point of saturation with the further addition of neighbors. Concernedly, more perplexity of nearest neighbors needs to be calculated as k value increase, but the improvement is limited. For simplicity, we employ a moderate range of neighbors and use k=4 throughout our experiments.Is LPR effective with different LLMs? To show our proposed method is model-agnostic, we conduct experiments on a diverse collection of model architectures and present the results in Table 3 . From the results, we observe that our method consistently improves the ICL performance when using Llama2-13B [49] , Mistral-7B [19] and OPT-6.7B [66] . For instance, with Mistral-7B, using our method boosts the ICL performance using the random selection method from 19.24 to 27.07, an average 7.83 of direct improvement on 6 datasets with irrelevant-60% noisy annotations.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We submit our code and data as supplemental materials. We provide instructions that contain the exact command and environment needed to run to reproduce the results.Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We specify all demonstration selection and test details in the section A.2. The full details can be found in our code which be provided as supplemental material. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
discussion: 
 Global Perplexity Ranking vs. Local Perplexity Ranking. While our method has demonstrated strong promise in in-context learning, one may also ask: can a similar effect be achieved by selecting 1RLVH5DWLR $FFXUDF\ (a) SST2 1RLVH5DWLR $FFXUDF\ (b) AGNews -Symmetric 1RLVH5DWLR $FFXUDF\ (c) AGNews -AsymmetricFigure 4 : Average test accuracy on SST2 [48] and AGNews [67] . Different colors indicate the selection methods. The solid lines denote existing selection methods, and the dotted lines represent the method integrated by our method. We omit the noisy type on the binary classification -SST2.demonstrations with the lowest perplexity in the whole dataset? In this ablation, we compare our method with a global perplexity ranking method that selects demonstrations with the lowest perplexity values of input-label pairs from a large candidate set (e.g., {(x i , y i )} 100 i=1 ). Table 4 presents the performance comparison between our method and the global perplexity ranking method. While both the two perplexity ranking methods improve the robustness of ICL against noisy annotations, the global approach obtains inferior performance compared to our proposed method in most cases, especially in the cases of clean and low noise rates. In efficiency, Table 4 also show that the local ranking approach requires only 20% of the time required by the global ranking. This is because our method only calculates the perplexity of the local neighbors for each candidate, instead of using a large candidate pool. Overall, we show that the global ranking method cannot outperform the local ranking while introducing much more computational loads.Transfer to text classification tasks. Text classification is a common task of in-context learning, which may also suffer from a noisy annotation issue. To this end, we verify the effectiveness of the proposed method in text classification. Here, we consider two classification tasks (SST2 [48] and AGNews [67] ) with popular label noise types: the symmetric noise and the asymmetric noise [8, 33] . We report the average accuracy with GPT-Neo-2.7B [6] on datasets with the two noise types. More detailed experimental settings are presented in Appendix A.2.Figure 4 demonstrates that noise annotations barely hurt the performance of ICL when employing the random demonstration selection method [36] . However, the performance of ICL is significantly compromised when utilizing more effective selection methods like TopK [28] and DPP [62] . After integrating our method, both TopK and DPP methods are significantly improved in the inference performance, which indicates the noise robustness of our method in text classification.Potential failure cases. Our approach is built on two assumptions that are naturally satisfied in the real world (See Section 4.2). In this section, we conduct experiments on four generation tasks, including NQ, WebQ, SCIQ, and SQuAD, to determine whether our proposed method remains effective when one of these two assumptions is dissatisfied. The detailed analysis is presented below. Assumption 1 (Data): clean annotations are the majority in the annotated dataset. Given a dataset with extremely high noise ratios (e.g., 60%, 70%, 80%, 90%), the perplexity ranking of local neighbors may not reflect the correctness of the annotations, as most (even all) neighbors can be wrongly annotated. To explicitly show that, we conduct an experiment to validate the performance of LPR under extremely high noise ratios. The Table 5 below presents the average EM score of the baselines and our method. We use Llama2-7B [49] as the LLM throughout our experiments. The results show that the improvements of our approach decrease as the noise ratios increase. For example, when the irrelevant label noise ratio increases from 60% to 90%, the improvement of our method for the TopK method decreases from 10.26 to 0.92. Assumption 2 (Model): examples that are semantically similar share the same level of inherent perplexity. The model affects the the performance of LPR through the concept of inherent perplexity. This assumption cannot hold if the model is not capable of precisely measuring the semantic distance between examples. In this case, the local neighbors may not share the same level of inherent perplexity so that we cannot compare the Matching Perplexity. To validate this, we conduct experiments with language models with various sizes, including OPT-1.3B, OPT-2.7B and OPT-6.7B [66] . The results in Table 6 reveal that the performance of LPR decreases as the parameter size of language models decreases. For instance, for 60% irrelevant noise, the improvement of our method decreases from 5.23 to 0.46 when the parameter size of the language model decreases from 6.7B to 1.3B. 
conclusion: 
 In this paper, we introduce Local Perplexity Ranking (LPR), a general strategy that can universally enhance the noise robustness of in-context learning on generation tasks. To the best of our knowledge, this work is the first to analyze the noisy annotations in ICL for text generation. Our key idea is to decouple the matching perplexity by performing the ranking among the neighbors in semantic space. In particular, we replace each low-ranked candidate with its nearest neighbor that is highly ranked. Extensive experiments demonstrate that LPR can improve the noise robustness of existing demonstration selection methods in ICL across various noise types. Our approach is easy to use in practice, as it is insensitive to the hyperparameters and does not introduce heavy computational cost. Limitations. LPR is suboptimal in cases of high noise rates due to the assumption that clean annotations are the majority in the dataset. In addition, we do not provide a theoretical analysis to show how noisy annotations affect ICL, which will be an interesting direction for future research.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We report average results and standard deviation to illustrate the statistical significance of our method over 3 runs. We also conduct ablation studies to confirm the superiority of our method. See Table 2 and Figure 3 .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. See section 1. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: All pretrained model and dataset in this paper can be collected from Huggingface. See sections 5.1 and A.2. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the original paper that produced the code package or dataset. See sections A.2 and 5.1. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The dataset used in this paper has been submitted as supplemental materials. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In-context learning In-context learning (ICL) has become a new paradigm for natural language processing (NLP), where LLMs make predictions only based on contexts augmented with a few demonstrations [7, [34] [35] [36] . The popularity of ICL also raises growing concerns regarding its instability: given different selected demonstrations, ICL's performance can vary from near state-of-the-art to random [9, 12, 15, 26, 32, 37, 55, 54] . Existing studies show that ICL's performance is highly sensitive to order [30, 60] , template [37] and labels [50] of selected demonstrations. For example, on the one hand, some previous studies show that flip classification of demonstration can significantly hurt ICL performance on classification tasks [51, 60] . On the other hand, many researches show that ICL is fairly robust to noisy demonstrations [32, 37, 55, 54] . However, the existing studies only focus on classification tasks and the research of generation tasks is limited. We expand the previous finding from text classification tasks to generation tasks and find that demonstrations selected from noisy annotations significantly hurt the ICL performance of generation tasks.In practice, researchers often use crowdsourcing [61, 73] or large language models (LLMs) [58] such as GPT-4 [38] to create input-output pairs for new tasks, which inevitably leads to some mistakes in the annotations. However, the existing demonstration selection methods for generation tasks such as TopK [28] or DPP [62] only consider the input of demonstrations and assume the demonstrations are selected from a completely clean dataset such as [16, 28, 62, 65] . In comparison, we aim to propose a training-free demonstration selection method for generation tasks that can consistently and significantly improve the robustness of the existing methods under noisy annotations.Learning with noisy labels Label noise is common in many real-world datasets, especially generation tasks [2, 64] . The existing approaches to learning with noisy labels can be classified into two types: (1) training noise-robust models with noisy training datasets: designing noise-robust loss function [1, 63, 52, 72] or designing noise-robust model architectures [2, 18, 64] to mitigate label noise. However, this method is not suitable for ICL, which usually hypothesizes that users are unable to apply fine-tuning techniques [68] . (2) detecting noisy labels and reducing their impacts: comparing model predictions with noisy labels [42, 71] or checking the noisy label consensuses of nearby features [73] . Different from the above literature that focuses on classification tasks, we mainly consider a training-free solution to improve noise-robust ICL for generation tasks.Datasets We conduct experiments on 6 generation tasks, and examples of each dataset are shown in Tables 12 and 13 . For open-domain question-answering tasks, we choose the Natural Questions (NQ) dataset [22] and WebQuestions (WebQ) [5] . For reading comprehension tasks, we choose two reading comprehension datasets: Stanford Question Answering (SQuAD) Dataset [46] and Science Questions (SCIQ) dataset [56] . For code generation tasks, we choose Generating Tabular Answers for Multi-Table Question Answering (GeoQuery) Dataset [39] and Natural Language Interface to the Linux Operating System (NL2Bash) dataset [27] . Following previous studies [16, 25, 62] , we report Exact Match (EM) for NQ, WebQ, SQuAD and SCIQ, BLEU for NL2Bash and GeoQuery. We collect these dataset from Huggingface. The train sets of these datasets are regarded as examples datasets and the test sets are used to evaluate the performance of ICL. We randomly subsample 20,000 examples from the train set to generate noisy annotations and select demonstrations. We provide a few examples of noisy annotations of each dataset in Tables 14, 15 and 16.Baselines Our model LPR is essentially a data-centric retriever for in-context demonstration selection. We consider both learning-free and other learning-based retrievers as baselines:1. Random randomly selects demonstrations from a example set without repetition [36] . 2. TopK retrieve demonstration that are semantically-similar to a test query sample [28] . 3. DPP uses the original BERT embedding as above without fine-tuning, and adopts MAP inference for subset retrieval [62] .Experiment details We run our experiments on NVIDIA L40 GPU. We adopt a large portion of the code from the OpenICL repository [59, 60] . The whole experiment around one week on 8 GPUs and each experiment around one hour on a single GPU. Figure 6 : The distribution of perplexity of Llama2-7B [49] on clean and noisy annotations. Examples with noisy annotations indeed obtain higher perplexity than those with clean annotations.Transfer to classification tasks Inspired by the idea implemented in above assumption, we assume that examples that are semantically similar share the similar task, indicating they should belong to same classification. We don't need to calculate the perplexity of input-output pair and only identify whether the classification of candidate demonstration is same with its local neighbors or not. Similar to generation tasks, we replace the noisy candidates with their nearest neighbors that are more likely to be clean. We investigate whether our local-based method can transfer across to classification tasks.Empirical study of noisy ICL in text generation In this section, we provide the detailed results of GeoQuery and NL2Bash. Following existing studies [25, 62] , we adopt BLEU score [41] to evaluate ICL performance on code generation tasks. Figure 5 shows that both the two types of noises significantly deteriorate the performance of in-context learning on code generation tasks. This phenomenon motivates us to further investigate the noise-robustness of in-context learning.In Figure 6 , we present the perplexity distribution of Llama2-7B [49] on clean and noisy annotations of GeoQuery and NL2Bash datasets. As a complement, we observe that examples selected from noisy annotations set indeed obtain higher perplexity than those collected from clean annotations, which confirms the deviation can also transfer to code generation tasks.Analysis of small noise ratios. In this section, we conduct experiments on datasets with smaller noise ratios (e.g. 5%, 10%, 15%). Figure 7 (a) and (b) present the average EM score on four generation tasks, including NQ, WebQ, SCIQ, and SQuAD. Figure 7 (a) and (b) show that our method can benefit the ICL performance from a small noise rate (e.g. 5%)Open Benchmark Evaluation. Long-form and open-domain QA tasks such as MT-bench [70] and Arena-Hard [24] serve as valuable additions to the current standardized LLM benchmarks. In this section, we conduct experiments on these complex and open tasks to confirm the effectiveness of our method. The results on MT-bench [70] and Arena-Hard [24] are shown in the Figure 7 (c) and (d), which presents the average answer grading (0-10) [70] of baselines and our method. Figure 7 shows that our approach significantly improves the efficacy of existing selection methods on long-form question-answering tasks.Evaluation on the not-demonstration-selection-based baselines. Here, we add Zero-Shot baseline, as well as some CoT-related baselines, including Zero-Shot-CoT [20] and Manual-CoT [53] , Auto-CoT [69] . Specifically, Manual-CoT [53] and Auto-CoT [69] require to select demonstrations from an annotated examples set. The table below presents the BLEU score of the baselines and our method on the two code generation tasks: Geoquery [39] and NL2Bash [27] . We use Llama2-7B [49] as the LLM throughout our experiments. The results in Table 7 show that our method can outperform Zero-Shot and Zero-Shot-CoT [20] , and improve the noise robustness of Manual-CoT [53] and Auto-CoT [69] .Transfer to similarity score In LPR, we select reference demonstrations for candidate examples using cosine similarity. While cosine similarity captures some aspects of semantic similarity, it is limited to a single embedding [13] . Another measure of similarity necessitates an accurate characterization of the word levels. One way might be to use larger syntactic substructures of the input as terms with BM25, which is a sparse information retrieval algorithm belonging to a class of TF-IDF measures that view the test input and the candidates as bags of terms and measures relevance as a weighted recall of these terms:BM25(x i , x * ) = n i W i R(q i , x * )where q i is each token of x i , R(q i , x * ) and W i are the term frequency and inverse document frequency statistics that measure the coverage of a particular term and the relative importance of terms. In this section, we replace the cosine similarity score with the BM25 similarity score to verify the effectiveness of our proposed method.Our results in Table 8 show that the improvement still holds when BM25 is used as the cluster, confirming the superiority and robustness of our method compared with the naive demonstration selection methods. The above result also demonstrates that examples that are semantically similar in both token space and word space share the same level of inherent perplexity.Reordering Some studies demonstrate that in-context learning is highly sensitive for demonstrations' ordering when using random demonstrations [25, 30, 60] . Specifically, the same randomly sampled demonstrations with different orders can lead to the performance between random guesses and near state-of-the-art. In LPR, we reorder exemplars based on their similarities to the test input in ascending order after the process of our method, in accordance with common practices [23, 28, 45, 47, 62] . Here we compare our method with reordering and without reordering to explore the effect of reordering on example-specific demonstrations retrieved by our method.Across our experiments, Table 9 shows our method without the reordering process still improves the existing demonstration selection methods across various types of noise. The above results indicate that local perplexity ranking rather than the reordering process is crucial for the success of noiserobust ICL. Additionally, we believe high-quality demonstrations are less sensitive to the ordering and stabilize in-context learning, which is consistent with the previous work [16] .Various demonstration sizes To verify the effectiveness of our proposed method, we also present the ICL performance with our method across various set sizes K. Concretely, Tables 10 and 11 report the number of demonstrations to be 2 and 8 and show our method effectively mitigates the issue of noisy annotation in various demonstration selection methods across various demonstration sizes. NeurIPS Paper Checklist• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.This disentanglement is conceptual rather than mathematical. 


Decision: This paper introduces a method to improve the robustness of in-context learning for text generation tasks. They identified the importance of using clean examples as demonstrations for text generation tasks. Authors also addressed reviewers' suggestions well by adding experiments on real-world open domain QA results and comparisons with chain-of-thought methods. Overall, this paper presents solid experimental results and analysis.

Reviews: review: paper_summary: paper_summary: This paper proposes Local Perplexity Ranking (LPR), a method to improve the robustness of in-context learning for text generation tasks when dealing with noisy annotations. The key contributions are:

- Empirically demonstrating that noisy annotations hurts performance of in-context learning for text generation, unlike for classification tasks.

- Proposing LPR, which ranks candidate demonstrations based on perplexity within local semantic neighborhoods to identify and replace likely noisy examples.

- Experiments showing LPR improves noise robustness in most scenarios across multiple text generation datasets, 2 noise types, and 3 baseline selection methods.

The method is motivated by analyzing perplexity distributions of clean vs noisy examples and decomposing perplexity into inherent and matching components. Overall, LPR provides a simple but effective approach to mitigate issues with noisy demonstrations in in-context learning for text generation.
strength_weakness: strength_weakness: 1. The author states that the paper's motivation stems from the occurrence of noisy annotations in in-context demonstrations:
> For those candidates, input-label mappings solicited from humans [58, 68] or LLMs [55] can often be noisy, especially in **complex tasks**. This gives rise to the importance of noise-robust ICL, which aims to construct effective demonstrations in the presence of noisy and erroneous labels.

However, the current evaluation of the proposed LPR approach is limited to short-form, closed-domain question-answering tasks using traditional NLP datasets. These datasets typically don't suffer from noisy annotations, as the tasks are relatively simple, and ensuring the correctness of a few (fewer than 10) in-context samples shouldn't be challenging. Including experiments on long-form, open-domain question-answering tasks would better justify the paper's motivation and demonstrate the broader applicability of LPR to more novel tasks.

2. The evaluation experiments on justifying the benefits of LPR are conducted using noisy annotations with noise ratios exceeding 20%, which is likely unrealistic for short-form QA tasks in real-world scenarios. This experimental setting appears overly synthetic.

3. It would be helpful if the author could provide a clear illustration demonstrating how LPM is conducted.
questions: questions: 1. Have the authors conducted experiments on long-form, open-domain question-answering tasks (e.g., MT-Bench)?

2. I observe that the benefits of LPR decrease as the noise level diminishes. In fact, in Table 2, when the labels are clean, the effect of LPR appears negligible, as it doesn't improve upon the baseline 50% of the time. While I understand that LPR is designed to handle noisy labels, have the authors conducted experiments with noise ratios between 0% and 20%? At what threshold does LPR start to show a noticeable effect compared to the baseline?
limitations: limitations: Yes. Would be great to further discuss some possible failure mode of LPR.
paper_summary: paper_summary: The paper "On the Noise Robustness of In-Context Learning for Text Generation" investigates how LLMs handle noisy annotations during in-context learning (ICL). The authors propose a method called Local Perplexity Ranking (LPR) that replaces noisy candidates with nearby neighbors that are less noisy. They also explore the impact of noisy labels on ICL performance and compare different selection methods for these examples, such as TopK and DPP. Their proposed method can effectively mitigate the negative effects of noise, offering insights into enhancing the noise robustness of ICL in practical applications.
strength_weakness: strength_weakness: - The absence of code makes it difficult for the broader research community to reproduce the results claimed in the paper or verify the method's effectiveness on the tasks (the authors claim that the code is released in section 4, but I'm not sure where it is).
- The paper did not provide an analysis of the types of errors that LPR makes versus the baseline methods. Such an analysis could provide insights into the strengths and weaknesses of the approach.
- While LPR is compared to baseline selection methods like TopK and DPP, it is not compared to other techniques specifically designed to handle noisy annotations during in-context learning.
questions: questions: - What are the specific cases where LPR may tend to fail? Does each model's inherent capabilities affect these results?
- What is the specific model version for GPT-4 and the approximate cost for generating relevant noise for each task? What was the specific method (or prompt) used for generating relevant/irrelevant noise?
limitations: limitations: The authors have addressed the potential limitations of this paper. It would be useful if the authors also present examples where their proposed method have failed.
paper_summary: paper_summary: The paper introduces a new method to deal with noisy annotations for in-context learning. The authors suppose that the examples that cause higher perplexity than their neighbors are more likely corrupted than their neighbors. So, the authors suggest replacing the examples, causing suspiciously high perplexity by their neighbors with lower perplexity (the process is formalized by the "Local Perplexity Ranking"  formula). They show experimentally that this method significantly improves the quality of in-context learning for several tasks for several models of size 13B or less.
strength_weakness: strength_weakness: - In Table 2, the authors compare their method with several demonstration selection methods. However, it is still unclear whether **any** demonstration selection method, including LPR, is really the best way to improve the model quality in this situation. What about simpler methods to improve the answer quality, such as a chain of thought? What if it could work better even when coupled with the demonstration of noisy examples? Adding more types of baselines, such as some simple variations of a chain of thoughts and chain of thoughts + noisy examples would make the paper's claims much more solid, but there is no such comparison.
- See the "Questions" section and "Limitations" section.
questions: questions: Essential questions:
- Why Perplexity = Inherent Perplexity + Matching Perplexity? I didn't find any experimental or theoretical confirmation.
- In Figure 2, there is the perplexity distribution of Llama2-7B on clean and noisy annotations. Does this perplexity distribution change for bigger models, such as Llama-13B? Does it change for smaller models, such as OPT?
- How to choose a good gamma? You wrote in lines 233-234 that "LPR shows robustness to the choice of threshold γ", but I didn't find any experimental confirmation of this point.

Questions about the presentation:
- What model is used for Table 2 results? Is it LLAMA-2 7B?
- What metric is shown in Table 3?
- You wrote that figures 3 (a)-(d) use different gamma values, but which exactly? There is no information in the figures caption.
limitations: limitations: I would suggest adding the lack of any not-demonstration-selection-based baselines (such as chain of thought) and a narrow scope of models to the limitations section. **Or** I would suggest performing corresponding experiments and adding them to the paper if it is possible to do so in a limited time.

Comments: title: Thank you for increasing the score
Response to authors
Thank you for raising the score
Response to the authors
comment: We appreciate the valuable suggestions and feedback from the reviewer. We are also glad that most of your concerns have been addressed. Thanks again for increasing the rating!
The authors' response resolves most of my issues. Thus, I have changed my score accordingly.
Thank you for checking our rebuttal and raising your score. We will incorporate the new results and explanations into the final version appropriately.  Sincerely thanks for your valuable time on this paper!
I appreciate the author's responses and the experiments conducted on MT-Bench and Arena-Hard, as well as the additional experiments with small noise ratios. I have raised my score accordingly.