$k$NN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inference
Authors: Benfeng Xu
Quan Wang
Zhendong Mao
Yajuan Lyu
Qiaoqiao She
Yongdong Zhang
Venue: ICLR 2023 poster (2023)
Keywords: Large Language Models
In-Context Learning
K Nearest Neighbors

Abstract: In-Context Learning (ICL), which formulates target tasks as prompt completion conditioned on in-context demonstrations, has become the prevailing utilization of LLMs. In this paper, we first disclose an actual predicament for this typical usage that it can not scale up with training data due to context length restriction. Besides, existing works have shown that ICL also suffers from various biases and requires delicate calibration treatment. To address both challenges, we advocate a simple and effective solution, $k$NN Prompting, which first queries LLM with training data for distributed representations, then predicts test instances by simply referring to nearest neighbors. We conduct comprehensive experiments to demonstrate its two-fold superiority: 1) Calibration-Free: $k$NN Prompting does not directly align LLM output distribution with task-specific label space, instead leverages such distribution to align test and training instances. It significantly outperforms state-of-the-art calibration-based methods under comparable few-shot scenario. 2) Beyond-Context: $k$NN Prompting can further scale up effectively with as many training data as are available, continually bringing substantial improvements. The scaling trend holds across 10 orders of magnitude ranging from 2 shots to 1024 shots as well as different LLMs scales ranging from 0.8B to 30B. It successfully bridges data scaling into model scaling, and brings new potentials for the gradient-free paradigm of LLM deployment. Code is publicly available at https://github.com/BenfengXu/KNNPrompting

Published as a conference paper at ICLR 2023
KNN PROMPTING: BEYOND-CONTEXT LEARNING WITH CALIBRATION-FREE NEAREST NEIGHBOR IN- FERENCE
Benfeng Xu', Quan Wang", Zhendong Mao 14, Yajuan Lyu", Qiaoqiao She Yongdong Zhang"
1 Uniieesiity of Science and Technology of China, Hefei, China
"Beijing University of Posts and Telecommunications, Beijing, China Baidu Inc., Beijing, China *Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China benfeng@mail 11. ustc. ustc.edu. cn, zdmaol uutoo.usst..... www.ee......... cn
ABSTRACT
In-Context Learning (ICL), which formulates target tasks as prompt completion conditioned on in-context demonstrations, has become the prevailing utilization of LLMs. In this paper, we first disclose an actual predicament for this typical usage that it can not scale up with training data due to context length restriction. Besides, existing works have shown that ICI, also suffers from various biases and requires delicate calibration treatment. To address both challenges, we advocate simple and effective solution, ANN Prompting, which first queries LLM with training data for distributed representations, then predicts test instances by sim- ply referring to nearest neighbors. We conduct comprehensive experiments demonstrate its two-fold superiority: 1) Calibration-Free &NN Prompting does not directly align LLM output distribution with task-specific label space, instead leverages such distribution to align test and training instances. It significantly out- performs state-of-thee
calibration- methods under comparable few-shot scenario. 2) Beyond-Context ANN Prompting can further scale up effectively with as many training data as are available, continually bringing substantial im- provements. The scaling trend holds across 10 orders of magnitude ranging from 2 shots to 1024 shots as well as different LLMs scales ranging from 0.8B 30B. It successfully bridges data scaling into model scaling, and brings new potentials for the gradient-free paradigm of LLM deployment. Code publicly available
INTRODUCTION
0.62
Maximum Context Length
Avg Perf. on 10 NLP P Tasks
0.80
Large language models (LLMs), when scale up
to billions of parameters, have demonstrated re-
0.78
markable capabilities in wide range of NLP
0.76
tasks (Radford et al., 2019 Brown et al., 2020).
However, such models are prohibitively expen-
0.74
sive to train with most of the research- or consumer-level devices, though some of them are already publicly available (Zhang et al.) 2022). As a result, is now an emerging paradigm that
0.72
kNN Prompting In-Context Learning
0.70
0.68
LLMs are hosted in a remote data center while accessed by end users or applications via simple API request(?) 2 The typical usage of LLM un- der such paradigm is In- Context Learning, where LLM reads and completes prompt sequence how it is pretrained on massive text corpora. The
18
32
64
128
256
512
Num. of Training Shots
Figure 1: KNN Prompting brings substantial im- provements over standard ICL, and can contin- ually scale up beyond the context with as many data as are available. Conducted with GPT XL.
"Correponning author.
com https://openai.coo /

---

Published as a conference paper at ICLR 2023
prompt is constructed by concatenation of several training examples and test instance, and the prediction is obtained by mapping the LLM word continuations back to label space.
It is widely investigated and acknowledged that modern neural networks generally perform better w.r.t. increased training data. Specifically, there cxists a power law between expected model per- formance and available data scale (Hestness et al. 2017 Rosenfeld et al. 2020). For ICI., it is also empirically observed that the performance continually improves when more training examples are prepended into the prompt (Brown al. 2020). However, such improvements are quickly prevented by the predicament of context length restriction, as language models are designed and trained to only process sequences within a fixed length, which in fact 1024 or 2048 tokens. In order to utilize more training data, several works try to select the most relevant examples to compose the prompt before querying I.I. (Liu et al., 2022b; Rubin et al. 2022), but still only in-context examples can actually participate the LLM inference while most training data are discarded beforehand, thus pro- viding marginal data scaling benefits. Besides, their reliance on external retriever also incurs further complications. As a consequence, such a situation poses serious challenge for many practical scenarios where more than a few training data are available.
Another vulnerability of ICL is the severe bias existed in the output distribution of LLMs, which results in considerable performance degradation (Holtzman et al. 202I) and instability (Lu et al. 2022) as shown in existing works. Accordingly. many have proposed various ways calibrate the output distribution (Zhao et al., 2021: Jiang et al.) 2021 Min et al. 2022a). For example, ZZao et al. (2021) measure such bias by probing LLM with "NA" example and record the according prior. However, as LLMs are pretrained on general-domat
natural language, its capability to com- plete fabricated prompt is essentially not aligned with downstream task-specific label space. As consequence, such calibration-based methods can only alleviate the bias to a limited extent.
In this paper, we advocate simple and effective solution, ANN Prompting, to address both chal- lenges. Specifically, we assign training data into demonstration set and an anchor set. We append each anchor example into the prompt and query LLM, then instead of aligning word continuations with labels, we collect the language modeling probability as distributed representation and cache it into a local datastore. At inference time, for each test instance, we similarly obtain its representation and match it against the maintained datastore to make predictions. In general, the proposed frame- work enables both calibration-Fi optimization because avoids forced input-label alignment, and beyond- cooondt learning because the anchor set allows utilization of unlimited training data.
We conduct comprehensive experiments using 10 established text classification tasks to demonstrate the significant superiority of KNN Prompting across various scenarios and against competitive op- ponents: 1) Under few shot scenario where training data is very limited and fits the context, KNN Prompting outperforms state-of-the-ar calibration-based methods by considerable margin (up to +7.07). 2) Under low resource or fully supervised scenario where training data can not fit in the context, ANN Prompting further exhibits its major advantage. It can effectively scale up with as many training data as are available across 10 orders of magnitude (2 shots~ shots, see Figure [] for illustration) as well as different LLMs scales (0.8B~30B). Specifically, with only 32 shots train- ing data, it dramatically improves ICL by +13.58 in average score at its most, and achieves absolute improvements up to 188.84 under fully supervised setting. We also provide formal explanation on the intrinsic mechanism of effectiveness, as well as detailed analyses regarding its robustness and choices of design. Accompanied with these appealing aspects, kNN Prompting is in general a promising solution that bridges the benefits of data scaling into model scaling to take the gradient- free paradigm of LLM deployment one step further.
2 BACKGROUND: IN- IN-CONTEXT LEARNING
In this section, formulate the task and recap the ICL baseline. Assuming a target task with training data set T = ((I,, 34)}, and Y as its categorical label space. At inference time, the model is asked to predict Atest given test instance Trest- We then denote an LLM 0 that is pretrained with standard language modeling objective. At employment, it samely predicts a probability distribution p(uy/wer 0) for the next token t-th position conditioned on previous context Wet-
In-context learning first formulates training examples (Zz,,)) in the format of input-label map- ping via intuitive templates (see Appendix F for illustration), and concatenates them into natural
2

---

Published as a conference paper at ICLR 2023
SST2
SUBJ
MPQA
AGNews
CB
CR
DBPedia
MR
RTE
TREC
Numn of Shots (TP)
20 (2%)
12 (1%)
(0%)
(0%)
2 (0%)
14 (4%)
(77%)
(4%)
(0%)
8 (1%)
Mf
16
32
2
8
Table I: Maximum number of training shots (per class) allowed by 1024 tokens of context. Calcu- lated using GPT2 tokenizer. Inside the parentheses are Truncation Probability (TP, i.e. whether or not truncated, restricted below 5%). We set Mr for each task T in experiments for simplicity.
language sequence along with the test instance construct a prompt:
P = T (21, /1) ( x (x2, 122) @ ... @ x (F[TT- M[T]) @ π (Fiest: +)
(1)
where т denotes template-hased transformation (see Appendix for illustration) and @ denotes con- catenation operation. Note that implies verbalization process that maps label space Y to responding tokens V picked from the LM vocabulary. When queried by the prompt P. LLM will try mimic the prepended training examples in the context and predict probability distribution P, 0) for the next token v. We then map it back to label space Y as prediction:
Yrest
arg (ax(u) yEY
y - v
Figure 2 provides pilot study showing that when prompt P includes more demonstrations, the performance consistently improves, which is in accord with the power law of data scal- ing investigated many existing studies (Hest- ness et al. 2017, Rosenfeld et al. 2020). How- ever, this trend then prevented by the context length restriction. We provide more compre- hensive statistics in aad Appendix(A) [ In conclusion, this situation poses actual chal- lenge in many scenarios where one would fur- ther collect training examples from few-shots to dozens and expect improved performance, but the power law fails.
0.9
0.8
0.7
0.8
MPQA
0.5
SST2
SUBJ
2
4
8
16
32
of Training Shots
Figure 2: ICL improves with num. of training ex- amples but is limited by context length restriction.
ANN PROMPTING
In this section, we introduce the ANN Prompt- ing framework. For a training data set T, we split and exploit it in two respective usage: T= i.e., demonstration set D (x1 ){ and an anchor set A = ((*f.,f)) ). ANN Prompting consists of two stages namely meta test and formal test, the overall framework is illustrated in Figure 3
Meta Test We build a datastore that caches all anchor examples in A for later inference time usage. For each 27 we respectively concatenate it into prompt P, where the prompt prefix is constructed using the demonstration set the same as Equation!] []
P. = π (0f vft) @ π (x² vd) o @ क ( BiDi) e (x]
(3)
By querying LLM using P., we obtain the distribution p(u) P: 0). Instead of mapping it back to label space V, we cache the entire language modeling distribution as the key representation:
l; p(v P., 0)
(4)
Accordingly, label yf is the value. The entire datastore thus consists of paired {ks, vf J. we denote the set of keys as K.
Formal Test At inference time, for each test instance Prest: we construct the same prompt as Equation [ and obtain distribution Prest. Ptestt 0). We then match the distribution against cached K. in the datastore, where standard K L divergence is used to measure the distance:
DKL (ks) = Σ p(ppPrrets, 0) log PtolPtests !)
(5)
P(,
3

---

Published as a conference paper at ICLR 2023
Demonstration
Oxf.yf)
Anchor Set
Distribution
Nearest Neighbors
k
(+f+)
2.227
LLM
N2
Datastore
kp 0.293
Interface
Meta-Test
(xigit)
Sorod
KL Divergence
Formal-Test Test
Figure 3: The overall framework of kNN Prompting
The predictions are calculated by aggregating its k nearest neighbors:
Sreed = arg max
Σ
1 = y)
(6)
PEY
NENN (Peest
where (*, KC) denotes the set of nearest neighbors in K.
4 EXPERIMENTS
4.1 SETUP
We use 10 established text classification datasets, respectively SST2 (oocher et al. 2013
SUBJ (Pang & Lee, 20044, MPQA (Wiebe al. 2005). AGNews Zhang et al. 2015),
CB (De Marneffe et al. 2019), CR (Hu & Liu 2004). DBPedia (Zhang et al., 2015). MR (Pang & Leel 2005), RTE (Dagan al. 2005) and TREC (Voorhees & Tice 2000]. For each dataset, we devise intuitive prompt template (Appendix F). and other regarding statistics are listed in Ap- pendix A We investigate a wide range of LLM scales, including GPT2 (0.8B and 1.5B) (Radford et al. 2019) and the OPT (Zhang et al. 2022) series (3B-30B). GPT2 XIL is used for most analyses unless explicitly indicated. We invariantly hyper-parameters as the entire framework is wriniin-frree We run with 5 different random seeds for Figure and Figure 10 seeds for all other results. Mean and standard deviation are reported.
the number of neighbors i to There are no other
4.2
DATA UTILITY
In this paper, refer to data utility whether and how performance scales up with increased training data. It can be formally depicted by the power law (Hestness et al. 2017):
<(m) am
(7)
where m the training data size, a a constant scaling factor, 3 describes the exponential steepness, and s(*) calculates the generalization error, i.e., test time performance. We refer m to training shots, i.e. number examples per class throughout the paper. In the following sections, we investigate various settings of m. respectively few shot (m .M),, low resource (m 128), and the overall scaling law (m < 1024). We refer My to the maximum training shots allowed in context for each task T, the specific statistics can be found in Appendix A
4.2. DATA UTILITY UNDER FEW SHOT SCENARIO
We first investigate NNN Prompting under few shot setting, i.e., Σ My. We simply set [D] and use all other examples for A. We leave further exploration of split strategies to Appendix C.5 In order to avoid context length restriction and maintain comparability to baselines
much as possible, 6 out of 10 tasks where My 2 8 are selected. The baselines are ICL and state-of tf-th- art calibration based augmentations. ContextualCalibrattio (Zhao et al. 2021) probs the prior bias w.r.t. each category and accordingly calibrate the outputs, and NoisyChannel (Min et al. (022a) formulates ICI. computing input likelihood conditioned on labels. Results in Table 2 show that
4

---

Published as a conference paper at ICLR 2023
Setting&MMehhod
SST2
SUBJ
MPQA
CR
MR
TREC
AVG
ICL
59.8
51.4g16
60.2g142
57.3
64.7gu1
50.0
57.24
Contextual Calibration
76.277
69.8
63.4+m
60.1-17
75 75.6484
45.5+71
65.12
77
Noisy Channel
82.6+31
64.6+50
60.2+91
83,3-23
79.4+1
35,2+98
67.54
ANN Prompting
77.5 1m:
13.4
16.61m
69.71mm
412
ANN Prompting (Partial)
27.8
68. 68.9_m
53.3gm0
66.7
81.6161
40.2
64.73
ICL
67.2+126
56.3-115
70.8+114
60.8
62.8+103
50.9+43
61.50
Contextual Calibration
70.8+112
60.0+s.1
70.5
59.6-47
70.0+10
43.6+45
62.43
T 4
Noisy Channel
80.9 10.9
60.5
66.4
83.9 83.9_23
79.013a
40.9
68.58
ANN Prompting
260
73.5
66.4g117
71,22
82.9
51 61.6gma
72.14
ANN Prompting (Partial)
85.9
70.5-1033
67.4
68.1
82.0=35
52.2+79
71.00
ICL
57.8
66.2_113
11.2
66.0gns
61.5
50.9
63.27
Contextual Calibration
68.5+75
64.5+99
72.7+1111
64.9-as
68.2+88
44.0+52
63,80
Noisy Channel
82.0+2:
62.5esa
70.1
85.0-21
79.2+2.
41.6+95
70.06
ENN Prompting
88.9
77.7
72.51124
75.4
84.6117
63.7
77.13
ANN Prompting (Partial)
88.9ga1
69.2_mm11
67.8g154
72.82m
84.3ga
54.6217
72.92
67.7 6.07 1:ow
75.5
77.6177
73.3
61.81
$2.01
67.97
Contextual Calibration
75.7
59.71
75.2
73.0
73.1
46.5
171 Noisy Channel
84.4114
62.4]n2
70.4
83.7
79.6
54.2
72.44
ANN Prompting
88.8
80.914
68.2.77
80.1
84.8129
70.0
78.80
ENN Prompting (Partial)
89.7
71.4148
60.51127
79.8_st
84.8130
55.8
73.67
ICL
66.5
70.1
77.42a3
67.7
63.6
$2.0
66.22
Contextual Calibration
76.9
58.61 2022
76.5279
78.5*
71.21
44.3
67.66
11 32
Noisy Channel
84.8 Lus
61.1 13M
70.8
82.5
80.0
47.6
71.14
ANN Prompting
89.0=1a
83.2+19
69.3429
77.8-56
85.0+20
73.5+19
79.64
ANN Prompting (Partial)
86.7
65.9_129
64.41121
74.2_n4
83.9
59.414m
72.41
Table 2: Results under few-shot scenario. Calibration- baselines are reproduced using their released code i denotes necessary truncation. The overall scaling trend is accordingly visualized in Figure 4 and Appendix C.I Partial means only label woodd distribution is utilized.
ANN Prompting significantly outperforms competitive baselines under strictly comparable settings (m < 8), specifically, +3.56 for shot, and +7.07 for 8 shot.
Superiority of Whole LM Distribution Calibration-baset methods (Zhao et al., 2021 well as standard ICL only access label words instead of whole LM distribution, which inferior in two aspects: 1) loss of information. LLM always generate distribution over all words in the entire vocabulary, non-label words probabilities also reflect its understanding in certain perspectives; 2) multiple label words competing with each other. There exist various choices for label words but oracle rules to select one, and alternative choices potentially compete with the selected label words, distorting the label space distribution. This is also referred surface form competition (Holtzman we very this benefit by masking out non-label words (referred to as Partial).
et al. 202 In Table 2
4.2.2 DATA UTILITY BEYOND THE CONTEXT
We then investigate a major advantage of ANN Prompting, which is scaling up to more training examples that otherwise would not fit in the context. We increase to 128 and compare with: 1) ICL Ensemble which is an intuitive alternative to scale ICL up, and has been adopted in previous works (Jiang et al. 2020); 2) finetuning of standard PLM such BERT or GPT Large, which could produce meaningful uu results with such amount of data. For all methods, we append maximum Mr examples into prompt P. For ICL Ensemble, split T into multiple non-overlap demonstration sets T = D, U D2 U... DN to construct different prompts, and ensemble their predictions.
Results in Table 3 show that ANN Prompting continues to improve and outperform ICL and its ensemble respectively by +16.96 and +16.08 (0.8B model, average score). Besides, the ensemble baseline is also very inefficient, assuming Mr = 8, need query 128/8 16 times for every test instance, and this keeps growing linearly if we use more training data, eventually becomes prohibitively inefficient and can not scale up either. ANN Prompting also outperforms FT when the adopted LLM scales above 6B (82.73). By contrast, it would require LLM to scale above 30B (82.45) using ICL Ensemble and even larger using standard ICL.
https: http:s////ggthuu...o.////o////// httpr///h///////a/o////////////r https: thub. com wwwsw25 of prompt templates, we report their hest template out of four.
(comm//hmss25/Chaneel Channel MMM-rrortpiinn there exists slightly difference
5

---

Published as a conference paper at ICLR 2023
& Methods
SST2
SUBJ
MPQA
AGNews
DBPedla
MR
RTE
TREC
AVG
BERT Large FT
114
10.7186
74.5_
88.0110
78.6
88.0
95.1..1
13.
58. 11.
78.8
82.31
GPT Large
90.7
86.1
87.5.0
88.3g15
70.0ga0
86.7
96.5
86.2
55.4g98
71.2
88
ICL
63.4233
58.9
:0.5.12
61,7.154
45.0a9:
59.9
77.0
53.6211
24.4
62.77
0.8
ICL Ensembbee
16.
69.1
67.4
41.
83.8
67.8
72.7 122
13.
59.0
63.65
KNN Prompting
84.5
85.8
83.1
84.5
62.1
89.7206
95.8
84.0
53.631
74.2
79.73
ICL
a1.3
a4.1am
60.7
a6.
83.5
22.2
53.0
54.2
68.31 31
1.B
ICL Enseemble
83.4
63.4g1
15.0
52.9
63.4gm
83.7
13
55.9218
59.6
69:21
ENN Prompting
86.3429
83.822
82.9
87.2
64.6279
88.9
96.5
86.4
51.1
74.0229
80.12
ICL
145
84.5
7848
92.6106
88.8 11
11
52.4144
61.
77.29
2.7B
ICL Eassemble
90.2433
77.7
86.5-19
78.4
54.6110
93.4g11
89.3_.0
92.4
54.3
78.8
79.57
ENN Prompting
93.4
87.5170
90.2g30
98 98.6_06
91.4
53.4166
80.8g50
82.25
ICL
92.7+11
81,6246
86.2-19
68.8es1
52.5
92.0=26
89.8=09
91.6+11
55.0
62.0
77.19
6B
ICL Ensembble
93.2
85.6138
87.7
11.
45.7,20
93.4
909_.1
12.
54.811
70.3
78.47
ANN Prompting
92.2417
87.4gi
85.2.15
67.4g16
63.6441
90.6g33
98.5-44
91.0415
55.620
75.8g1
82.73
ICL
89.0
91.3+24
78.4-12
18.1
53.2a44
93.4g11
92.2-14
$9, 30.9
55.8.30
s8.5
77.68
13B
ICL Ensemble
18.2
90.7111
78.4
82.6115
62.
94.0
94.5
12.
56.2
59.8
79.56
KNN Prompting
94.8,ns
86.2..3
87.4g1
78.9
89.6g11
96.9.6
92.0gas
58.2
23.
84.93
ICL
90.8
23.5
80.7.:1
74.8
64.6
87.7
93.3-00
93.421a
61.6227
71.7
80.21
30B
ICL Ensembbe
92.6
84.1
79.9
78.
67.
88. 88.1g39
93.8
93.4
65.1
82.0ga
82.45
KNN Prompting
94.3
92.7
84.5
87.1
70.7
91.0g12
98.8203
93.1
61.7248
79.8
85.38
Table 3: Results under low resource scenario (m 128). Compared with FT and ICL Ensemble.
0.80
0.86
Avg Perf. on 6 NLP Tasks
Avg Perf. 10 NLP Tasks
0.75
0.84
0.70
0.82
0.80
0.65
0.78
0.88
66
0.60
Contextual Calbration Nokiy Channel
1.58
13B
KNN Prompting
0.76
2.7B
30B
2
4
8
16
32
32
64
128
256
512
1024
Num. of Training Shots
Num. of Training Shots
Figure 4: Data scaling under few shot scenario. Figure 5: Data scaling under fully supervised Compared with calibration methods.
scenario. Conducted across various LLM scales.
4.2.3 CONTINUALLY SCALING UP TO THOUSANDS OF TRAINING DATA
We now fully scale data up to thousands of training examples and provide extensive results across model scales to observe their overall scaling performance. Figure shows kNN prompting can con- tinually generalize across the tested range to provide effective data utility, re-enabling the power law under gradient-free paradigm of LLM deployment. The full results can be found in Appendix C.2 With only 32 shots training data, ANN prompting dramatically improves ICL by +13.58 in average score at its most (0.8 B), and achieves absolute improvements up to +18.84 under fully supervised setting. With the largest model OPT 30B, it achieves a best performance of 86.02.
Comparison to Demonstration Selection A line of related works try to utilize available train- ing data by firstly retrieving the most relevant ones from the entire training set, then selectively composing the prompt before querying LLM (Liu et al. 2022b Rubin et al. 2022). We since re- produce such methods according to Liu et al. 2022b). We employ sfate-of-the-ar art seneral-purpose sentence encoder: to represent test and training instance, and compute their cosine similarity, the most similar Mj examples selected to construct prompt P. These retrieving models include BM25 (Trotman et al., 2014), Centence-BERe (Reimers & Gurevych 2019), SimCSE (Gao et al.
2021 and Trans-Encoder (Liu al., 2022a).
Figure 6 shows that although such methods indeed exhibits marginal scaling benefits, they are nowhere near competitive against &NN Prompting. Full results are listed in Appendix C.3 To fur-
Note that Liu et al. 2022h) also employ RoBERTa model finetuned on SST2, but this overlaps with selected henchmark and does not generalize other tasks heyond sentiment classification.

---

Published as a conference paper at ICLR 2023
0.825
Method
SST2
SUBJ
MPQA
Avg Perl. on Tasks
0.800
ICT. Baseline
81,3
64.1 2113
75.2_
DemonSelcction (upper)
92.6 (1%e)
(3%)
87.5 (1%cc
0.775
JNN Prompting
88.2
88.4_1
84.1
0.750
MR
CR
TREC
0.725
ICL Baseline
72.2+139
66.2+162
54.2-49
DemonSelection (upper)
88.7 (1%e)
88.7 (1%e)
73.0 (1%e
0.700
ANN Prompting
84.4
86.7-13
83.0
RUSE
Toans Fooader
Table Comparison upper-bound of Demon- stration Selection. 1% inside the parentheses means the result can be achieved runs out 1,000
202
Num. of Training Shots
searches.
Figure 6: cf. Demonstration Selection.
ther solidify this conclusion, push PrompiCompose to extreme situation trying approximate its upper-bound. As such methods ultimately resort to compose the prompt, it should be bounded by the best composition scheme from finite compositions. We thus search for 1,000 prompts with different examples and report their best run. In Table with surpasses such upper-hound approximation. In conclusion, kNN Prompting essentially makes better use of training examples than PromptComposs as the latter still only refer to in-context examples during LLM inference while most training data are discarded beforehand.
4 we find kNN Prompting performs on par
4.3
ANALYSES AND EXPLANATION
4.3.1
ROBUSTNESS W.R.T. DIFFERENT TRAINING EXAMPLES
previously found that vanilla ICL suffers from severe instability (Zhao al., 2021). Figure is produced with 10 different seeds, which results in different choices and permutations of training data. We show that &NN Prompting significantly improves the robustness. Besides, perfor- mance becomes more robust with increasing anchor set size. On 10 investigated tasks, the standard deviation ANN Prompting (3.83) is significantly smaller than ICL (9.14
SST2
0.9
Demostration Set Size [D]
0.875
0.8
0.850
Acc
0.7
0.6
0.750
MPQA SUBJ
0.725
0.5
15
0.700
ICL
64
128
256
512
11
13
15
Anchor Set Size [A]
Anchor Set Size [A]
Figure 7: Robustness.
Figure 8: Split strategy.
4.3.2 SPLIT STRATEGY BETWEEN DEMONSTRATIONS AND ANCHORS
For fully supervised scenario where m > Mr. we can simply set [D] to its maximum My. Oth- erwise, might need to deliberate the trade-off between the demonstration set and anchor set. In Figure 13 we search through all possible combinations given that [A] + [D] & My. We see the left part of the heatmap generally performs inferior, i.c., [A] {1,2). This means larger [A] contributes more to the performance while the choice for [D] is relatively more robust. The con- clusion also corresponds to few shot results that anchor set yields better data utility than context concatenation. More datasets are also provided in Appendix
'Calculated using Table statistics with 32 shot setting, 0.8B model

---

Published as a conference paper at ICLR 2023
KNN Prompting
ICL Pred
20
Acc
83.5
97.8
B
film (X)
album
book
Bin
builddigg (X)
-40
building (X)
nature
atu
book (X)
artist
-20
Landy (born October 23 1974) is an trish and screamseiter best known for the Skulduggery Pleasnne series of children's books.
ICL Pred
Golden Label
distancel
(Test)
Britihh authar thrillr fiction currendly best known for 2010 Codex ..... non-fiction rather than liction for such subject matter
26 Januery
book (X)
artist
0.1355
ANN (Anchor)
Michelle Harrison (born 1979 Grays Essex England) British author 13 Prize ... Young
the Waserstone's Children's Book
book
artist
0.2267
was released
Barry Kinglam Oukley (born 24 Fubruary (981) is Australian writer
artist (V)
artist
0.2932
Figure 9: t-SNE (van der Maaten & Hinton 2008 for anchors and test cases. Cases are randomly selected given that ANN Prompting outperforms ICL. DBPedia an ontology classification task.
4.3.3 QUALITATIVE ANALYSES AND REASONS OF EFFECTIVENESS
We first formally organize the explanation as follows according to Figure
The output language modeling (LMM) distribution of LLM is essentially not well aligned with task- specific label space, resulting in inferior performance (83.5 test accuracy) of default ICL.
If we similarly perform inference anchor set, we would expect to get approximately 83.5 anchor accuracy by assuming i.i.d. data distribution. However, we are already aware of each of their golden labels, which actually gives 100 anchor accaracy.
LM distribution is inferior for making direct predictions, but superior for matching examples because entails distributional, delicate and comprehensive representations generated by LLM. &NN Prompting leverages such representations (83.5 accuracy) only for matching, and refer to their golden labels (100 accuracy) for predicting, thus successfully transfer part of the knowledge originating from anchor labels to test instances.
In the visualization, the representations generally exhibit partially clustered pattern, we can identify proportional examples that get entangled with different categories and crowded together (Case A, C, D), these confusing cases are likely to cause erroneous predictions in ICL. and corresponds to under- performed 83.5 accuracy as mentioned above. Specifically, case A is an abstract about novelist and should belong to category artist, but it is easily confused with category book using ICL because the context did mention books. By contrast, &NN Prompting can correctly predict by referring similar anchors that are also about novelists and their books (as listed in the table). Some of the anchors are also incorrectly predicted as book, but it no longer matters because INNN Prompting only use the distribution for nearest neighbor search but refer to golden labels for prediction. Besides, we can clearly know how the prediction made, i.e., which anchor examples are referred, the proposed method also exhibits explainability further advantage.
5 RELATED WORKS
Large language models, since firstly scaled up to hundreds of billions parameters by Brown et al. and followed by several others (Rac et al. 2021 Zhang et al. 2022 Chowdhery et al.
2020
2022
Cohen et al. 2022 Smith et al. 2022), have become the most prominent direction of NLP area. Although these models exhibit surprisingly powerful and even emergent capabilities in a wide
range of NLP tasks (Wei et al. 2022b Hendrycks al.) 2020 Srivastava et al. 2022 they
prohibitively expensive for most researchers or users to train or even hold. In- Context Learning, which suits L.M to various tasks while requires training, therefore becomes the typical usage as
8

---

Published as a conference paper at ICLR 2023
is popularized by Brown al. (2020). Similar ideas of formulating target tasks into natural language sequences can also be found in earlier works (Trinh & Le 2018: Raffel et al. (020).
To better exploit LLM for various scenarios, it becomes crucial problem to develop augmented methods for ICL DDog et al. 2022). Xie et al. (002)) provide theoretical explanations that formalize ICL as Bayesian inference. Dai al. (2022) reveal that ICL can be seen as implicit finetuning where LLM produces meta- vetaa-gradeens from in-context demonstrations to adapt the model behavior. [Wei] al. (2022a) and (Sanh et al. (2022) propose instruction tuning, which further pretrains LLM with a collection of downstream tasks in a shared prompting format. Min et al. (2022b) and Chen et al. (2022b) introduce meta-learning to better adapt LMs to ICL. Wei al. (2022c) and Kojima et al. (2022) propose to augment the demonstrations with human-aided reasoning steps or hints, which surprisingly improved the performance for arithmetics and other reasoning tasks. Closely related to this work, [Liu et al. (2022b) and Rubin et al. (2022) propose to compose prompt P by selecting most similar training examples. (Zhao et al. (2021) and Min et al.) (2022a) propose to calibrate ICL. prediction via either probing the bias or reversing the conditional prediction formulation.
ANN is classical machine learning algorithm (Fix & Hodges 1989) well known for its simplicity
and inspired a wide range of application (Papernot & McDaniel [2088] (rrhan|2018, In the field of
NLP, Kaiser et al. (2017) construct a differentiable memory module for nearest neighbor searching which improves generalization to rare events. Similar idea has also been explored for generation
tasks (Guu et al. 2018), such as dialog generation (Weston et al. 2018), machine translation (Khan- delwal al. 2021), etc. Wang al. (2022) and Chen et al. (222aa) propose to retrieve similar training examples and incorporate them into the input to jointly train the model. While Khandelwal al. (2020) and (Shi et al. (2022) consider unsupervised corpus as datastore, retrieve and interpolate them with the current step language modeling probability. The retrieved corpus can also serve as references for knowledge intensive tasks, but the retriever would need explicitly training for such purpose (Lewis et al. 2020; Borgeaud et al. [2022] Izacard et al., (022). Different from these works, KNN Prompting is suitably situated in the gradient-free paradigm L.LM deployment, which avoids calibration treatment and effectively bridges data scaling into model scaling.
6 DISCUSSION
Under the existing ICL paradigm, it is often impossible to take advantage of both the capability of LLM and the data utility of finetuning, i.c., model scaling and data scaling. &NN Prompting finds an effective solution to promise them both. Nevertheless, be inferior to the specialized finetuning of LLMs, if given sufficient computation resources in an ideal setting. We believe that it is very important and promising direction to further approach this upper-bound and expect to raise more interests in future works.
assume its data utility should still
A potential concern for eetrieval-basee models is their efficiency, especially when corpus level data- store is utilized. ANN Prompting is free of such concerns as it considers training data, which is in manageable scale. Under few shot scenario, ANN Prompting even reduces computational costs deployment time compared to standard ICL. It works well with one shot demonstrations as experi- mented in Section 4.2.1] while the anchor examples are queried for only once and cached locally. By contrast, existing methods only perform better when all examples are prepended in single prompt. This advantage rather important as we need to repeatedly query the prompt in practical usage of LLM service, and longer prompt results in linearly more monetary costs if charged by token numbers or super-linearly more computational costs if measured by FL.OPS.
CONCLUSION
In this paper, we propose NNN Prompting as simple and effective solution to advance gradient-fred free deployment of LLM inference. Motivated as calibration-free optimization, kNN Prompting signifi- cantly outperforms state-of-the-art calibration-based methods under comparable few shot scenario. While its major advantage is further revealed when training data increases and can not fit in the context. KNN Prompting can effectively scale up with as many training data as are available, suc- cessfully bridging the utility of data scaling into model scaling. The proposed framework endeavors to realize more effective, efficient and applicable utilization of large language models in realistic scenarios, and hopefully could inspire further research interests toward the same goal.

---

Published as a conference paper at ICLR 2023
ETHICAL CONSIDERATIONS
This work built upon the ICL paradigm and involves querying LLM for responses. These models might generate contents with potential ethical risks regarding faimess and bias (Bommasani et al.) 2021 Blodgett et al. 2020), depending on specific downstream tasks. Although the scope this paper remains on how to better exploit LLM for task performance, it is worth further discussion to combine the proposed framework in conjunction with well-established methods that can mea- sure (Nadeem et al. 2021 and mitigate (Nadeem et al. 2021 Gupta al. 2022) such ethical risks.
ACKNOWLEDGMENTS
This work is supported in part by the National Natural Science Foundation of China under Grant 62222212, 62232006 and 61876223, Science Fund for Creative Research Groups under Grant 62121002.
REFERENCES
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5454-5476, Online, July 2020. Asso-
ciation for Computational Linguistics. doi: 10.18653/v 22000002mm22m..mmia..aamm //aclanthology wwg/2020. acl-maiin..
URL https:
main. 485
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Dur- mus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsumori Hashimoto, Peter Henderson, John Hewitt Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keel- ing, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Ku- ditipudi, and et al. On the opportunities and risks of foundation models. CoRR, abs/2108 08.0228, 2021. URL https: //tp:v/aavvvvvv..v org abs/ / 22008 07258
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George Bm Van Den Driessche, -aptisse Lespiau, Bogdan Damoe, Aidan Clark, Diego De Las Casas, Aurelia Guy. Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irv- ing, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaud- huri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Pro- ceedings of the 39th International Conference on Machine Leurning, volume 162 of Proceed- ings of Machine Learning Research, pp. 2206-2240. PMLR, 17-23 Jul 2022. URL https: html
//proceedings.. iongs.mml mlr. wwrras//wi2/booggaaeao
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL https: ftttps://proceediing neur 1457c0dbb cb49674 9674 1ow.1aaabbaaccooo 42 42f64a 42aeaa-aaaa------------ea .dd
neur i pss wttppppppp//////1111111
Xiang Chen, Lei Li, Ningyu Zhang, Xiaozhuan Liang, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. Decoupling knowledge from memorization: Retrieval- -uggeeende prompt learning. arXiv preprint arXiv: 14704, 2022a.
10

---

Published as a conference paper at ICLR 2023
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via lan- guage model in-context tuning. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 719-730, Dublin, Ireland, May 2022b. Association for Computational I Linguistics. doi: 12.1666/11/202222cccoon..... URI. wrw.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0 02311, 2022.
Aaron Daniel Cohen, Adam Roberts, Alejandra Molina, Alena Butryna, Alicia Jin, Apoorv Kul- shreshtha, Ben Hutchinson, Ben Zevenbergen, Blaise Hilary Aguera-Arcas, Chung-ching Chang, et al. Lamda: Language models for dialog applications. 2022.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment chal- lenge. In Proceedings the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entail- ment, MLCCW'5, pp. 177-190, Berlin, Heidelberg, 2005. Springer- Verlag. ISBN 3540334270. org/ 0. 1 /07//173669900
doi: 10.7007/ 1736790 9. URL http:httt////tttm
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Why can gpt learn in- context? language models secretly perform gradient descent meta optimizers. arXiv preprint arXiv:2212. 2022.
Marie-Catherin De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. proceedings of Sinn und Bedeutung. volume 23, pp. 107-124, 2019.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. survey for in-context learning. arXiv preprint wwXiv:2301.00234 2022.
Evelyn Fix and Joseph Lawson Hodges. Discriminatory analysis. nonparametrie discrimination: Consistency properties. International Statistical Review/Revue Internationale de Statistique, 57 -37,2,8-247, 1989.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods Natural Lan- guage Processing. pp. 6894-6910, Online and Punta Cana. Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. https::/acllantho..l
http:////221...mmmmmaaee org. wwwp://01..mnnmmmm...e in. 552
URL
Umang Gupta, Jwala Dhamala, Varun Kumar, Apury Verma, Yada Pruksachatkun, Satyapriya Kr- ishna, Rahul Gupta, Wei Chang, Greg Ver Steeg, and Aram Galstyan. Mitigating gender bias in distilled language models via counterfactual role reversal. In Findings of the Associa- tion for Computational Linguistics: ACL 2022, pp. 658-678, Dublin, Ireland, May 2022. Asso- ciation for Computational Linguistics. doi: 10. 10.18653/11/2022. 1/aclanthology.. wrt.oy22.rr.0000.... wtw.:/2....nnnnssac....m
http:////222...nnns-aaaaa findings-acl.55. URL https:
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450, 2018. doi: 1
a 000000 URL
org/
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint vrxiiv2009.033300, 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint 1.Xi:1111..000000 2017.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form com- petition: Why the highest probability answer isn't always right. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 7038-7051, Online
11

---

Published as a conference paper at ICLR 2023
and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 1/206l//1222...mmmlllmo m6in.564. URL URLhhttps: htttps://acaaattoloooyooo conip-main... 564
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD *04, pp. 168-177, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 138811 doi: 01144//10144022.11407733 URL 1014052 1014073
Gautier Izacard, Patrick Lewis. Maria Lomeli, Lucas Hosseini. Fabio Petroni, Timo Schick, Jane Dwivedi- Nu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with re- trieval augmented language models. arXiv preprint
2tfi::220000399999 2022.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How models know? Transactions of the Association for Computational Linguistics, 8:423-473 2020. doi: 11.1162/ a 000224. URL //ac antho loy. org/ 2020 tacl 28
know what language
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions the doi: 10.1162/tacl a 00407. URL
Association for Computational Linguistics, 9:962-977 977, 2021.
https::/acclanee htwp://llaaa...oooooooooooo/////
ogg. www.orgg/22200 tawl-1......l1....
Lukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning remember rare events. In
International Conference on Learning Representations, 2017. httpst/:::/ /ttp:///ppeeeeeeeeeeeeeeee
nat/forum?id-ee
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learn- ing Representations, 2020. URL
net forum ?id HK
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer. and Mike Lewis. Nearest neigh- bor machine translation. In International Conference on Learning Representations, 2021. URL
htttss://pperrevieee net. Forum? (ttruummdddwwccooffo30
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models, 2022. URL fttpps://ppezevieer wttp:iieeeeeteeeeeoooeeeeeeeeeee www//eeetumnii.mmmmm 6р3uuнНAii
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih. Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmeened generation for knowledge-intensiee nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 9459-9474. Curran Asso- ciates, Inc., 2020. URL 6b493230205f 780 200 / Paper. pdf
neurip wttpaaaiippp.cppppeeepeee wctpppappr//22///1eeeee..
Fangyu Liu, Yunlong Jiao, Jordan Massiah, Emine Yilmaz, and Serhii Havrylov. Trans-encoder: Un- supervised sentence-pair modelling through self- and mutual-distillations. In International Con- ference on Learning Representations, 2022a. URL https: http:///eeereeiieeeee /tpp:t/eeeeeeeeeeeeeee... net forum?
id=AmUhwwOH..
Jaahhang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (Dee- LIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100-114, Dublin, Ireland and Online, May 2022b. Association for Compu- 1.10. URL httts:://clanthhoet
tational Linguistics. doi: 10. N0.18653/v1/2022.d6e org/ 202/ deelio-1.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically dered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I:
12

---

Published as a conference paper at ICLR 2023
Long Papers), pp. 8086 8098, Dublin, Ireland, May 2022. Association for Computational Lin- guistics. doi: 10.865551//202..c4logg.... acl acl-long.555 556
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 5316-5330, Dublin, Ireland, May 2022a. Association for Computational Linguistics. doi: 10. 10.18655//1/20222.ac 365. URL peaetps://aclanne
/ httpppss//aaaaaaaooloooo//o///// wrtpl/ogggrrg///22/////// wtt./2222...-oooooo.e 365
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Asso- Seattle, United States, July 2022b. Association for Computational Linguistics. doi: 10.18653/1 1/2022. naacl-main.201. URL U http:///aaaaanntolo.ooor//////// wtw.:o2.naaaclccoo mwacc-maate -maimmiin.. 201
ciation for Computational Linguistics: Human Language Technologies, pp.
Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5356-5371, Online, August 2021. Association for Computational Linguistics. doi: 1121..cl-loom 4166
22tp:////2221...1oog4111111 URL https://ach antehonnthhhoooo egy org/
Emin Orhan. A simple cache model for image recognition. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.). Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL httpss////..occeee.n....eeeee 000317449214411110557766662
http::/iw....aaaoooo//o///////// cc/ppaper/201//illl
pdf
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Associa- fion for Computational Linguistics (ACL-04), pp. 271-278, Barcelona, Spain, July 2004. doi:
10.3115/1218955.1222 12199999999 URL
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categoriza- tion with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Associa- tion for Computational Linguistics (ACL'05) pp. 115-124, Ann Arbor, Michigan, June 2005.
Association for Computational Linguistics. doi: 10.3115/1119860.2219855 URL https:
//aclantho 0rGGyyyoggP000-000
Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv: 1803 04765, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David I Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann,, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11
2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140): 2020. URL http:
7/jmlr. wttpoo/ppppp/p/////////
074 html
Nils Reimers and Iryna Gurevych. Sentence-BERY Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 3982-3992 Hong Kong, China. November 2019. Association for Com- putational Linguistics. doi: 10.18653/v 1/D19 1410. URL https: http://holaaaannccc..oooooooool. .org D19-1410
13

---

Published as a conference paper at ICLR 2023
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference on Learning Representa- tions, 2020. URL
net/ www.:/eeee/ommmmiiemememm www.mmiidryeeeoeai....
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. Proceedings of the 2022 Conference of the North American Chapter of the Asso- ciation for Computational Linguisties: Human Language Technologies, pp. 2655-2671, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/1/2022. naacl-main. URL httpt:tttt/aaeee http///clannntooot
antho loacchoooyy. www.gyy.rrg/222.//////// naacl maa n. 191
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Repre- sentations, 2022. URL /tttps://epenreviiee wwt.eewwwwwwweeeeeeeeeeeeeeeeeee
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255-269, Online, April 2021. Association for Computational Linguistics. doi: 10. 10.11863/v1/02...comm waccl-maii.20 URL //aclanthology. wttp www.:eeeealmmma.....me
https:
Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. Nearest neighbor zero-shot inference. arXiv preprint arXiv:2205.13792, 2022.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari,, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. Using deep- speed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprini arXiv: 11900 2022.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL bttpes//aclaattoooooo.or//////1
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
Trieu H Trinh and Quoc V Le. A simple method for commonsense reasoning. arXiv preprint 1806v:8806.02850 2018
Andrew Trotman, Antti Puurula, and Blake Burgess. Improvements to bm25 and language models examined. In Proceedings of the 2014 Australasian Document Computing Symposium, ADCS '14 pp. 58-65, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 9781 9781450330008. doi: 10.1145/268286..222 2.622268862..288866.....
URL https: /ttps:://di.... http//do..or// www.ooooo.....oooooooo//oooooo.. 1145/
2682862.2 682863
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 9(6))2557-226) 2008. URL http::/ jmir wttp://ppppaaappssss/
vande rmmatte0a.a html
Ellen M. Voorhees and Dawn M. Tice. Building question answering test collection. In Pro-
ceedings of the 23rd Annual International ACM SIGIR Conference on Research and Develop-
ment in Information Retrieval, SIGIR '00, pp. 200-207, New York, NY, USA, 2000. Asso-
ciation for Computing Machinery. ISBN 1581132263. doi: 00.1145/345508.4455777 URL
https: ttops://do: www.do..org//..110 wtt.g/....11/355511 345577]
14

---

Published as a conference paper at ICLR 2023
Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. Training data is more valuable than you think: A simple and effective method by retrieving from training data. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3170-3179, Dublin, Ireland, May URL
2022. Association for Computational Linguistics. doi: 10.18663/11/0022.aac
long. 226
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In Interna- lional Conference on Learning Representations, 2022a. URL
net/
Eorum?idnqQarGCeean
Jason Wei, Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language mod- els. Transactions on Machine Learning Research, 2022b. URL [tttps://openres: net net/forrum1ddyyz33555o Survey Certification.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Lc, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 2022c.
Jason Weston, Emily Dinan, and Alexander Miller. Retrieve and refine: Improved sequence gen- cration models for dialogue. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI, pp. 87-92, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: N0.1865//1/W18-5177 URL http:///www..ccweee....////aaaaa
Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2):165-2)) 2005.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022. URL httpp:://peeeeev.ee..ttttt/e/e
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et Opt: Open pre-trained transformer language models. arXiv preprint arXhv:2205.01063 2022.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama. and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015. URL ht 23ccff
pdf
Zihao Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. Calibrate before use: Im- proving few-shot performance of language models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Pro- ceedings of Machine Learning Research, pp. 12697-12706. PMLR, 18 18-24 Jul 2021. URL
15

---

Published as a conference paper at ICLR 2023
A DATASET STATISTICS: MAX SHOT IN CONTEXT
We provide detailed statistics about the number of maximum shots in Table 5 i.e., My for each task, corresponding to Table D in the main manuscript.
SUBJ
MPQA
AGNews
DBPedia
MR
Num, of Classes
2
2
3
2
14
2
2
Average Length Templates
10.4
90.8
32.7
79.8
1024 Toks
Max Shots (TP)
Mr
(2%)
(1%)
39 (0%)
32
3 (0%)
2 (0%)
14 (49)
1 (77%)
14 (46)
8
4 105)
8 (15))
2048 Toks
Max Shats (TP)
Mj
(5%) 25 (15)
81 (3%)
32
(1%)
(2%)
28 (4%)
1 (0%)
27 (25)
16
10 (4%)
17 (59)
Table 5: Dataset statistics and the maximum shots (per class) that context of 1024 tokens or 2048
tokens can allow. We provide maximum shots under 5% Truncation Probability (TP) restriction as well as the actual Mr taken in this paper, which is set from {1, 2, 4, 8, 16, 32} for simplicity.
B MORE ANALYSES
B. DISTANCE MEASUREMENT
We investigate euclidean distance as an alternative distance measurement, which has also been ex- plored in Khandelwal et al. (2020). We take the contextual representation h of LLM, and denotes their distance as DL2 (/bueet h4). Table [ shows that both methods are effective but DKL (Equa- tion 5) based on LM distribution p superior measurement. Actually, p a projection of h through the word embedding, we think this procedure exploits the well-structured word embeddings of L.LM to provide more disentangled representations, thus can better serves as the anchor space.
Mensurements
SURI
MPQA
AGNews
CR
TREC
In-Contest Learning
81 8133
61.1a
75.2
22.7
60.7 424
66.2
83.5g1a
72.2
53.0
54.2
68.31
Contesttual Repr + Dea
84.
73.0
75.2
82.0
60.4 1811
11.0
78.0
95.1
82.6
53.3
70.3 144
75.40 (-7 ...09)
LM Distribation
87.7
77.03s
75.2
86.2
58.9
94.1gn
83.9
536
64.8
76.97 (-8 66)
Table Comparison for alternative distance measurement.
B.2 RELIANCE ON PRIOR KNOWLEDGE
Conclusion We further explore the robustness of kNN Prompting regarding the reliance of prior knowledge on target distribution. We show that among the investigated baselines, ICI. and Con- textualCalibration (Zhao et al. 202I) are greatly impacted by prior knowledge of test distribution, while ANN Prompting and NoisyChann (Min et al. 2022a) much more robust.
Experimental Setting We investigate various combinations of prior distribution by controlling the imbalance ratio Atvain and Atest- For every setting, we include binary classification tasks (SST2, MPQA, SUBJ, MR, CR) and run with 10 random seeds, we report the average score of these results. Specifically, Atrain 0.125 means one category (positive) accounts for 12.5% of the entire train set, and Atrain = 1.5 corresponds to the balanced setting. We investigate both Atrain < 0.5 and Atest < 0.5, which results in three different settings.
ContextualCalibratton explicitly include prior distribution, at such imbalanced scenario, one can either use the default assumption (pos neg 1 1, designed in the original paper) or use the observed ussumption from train set (pos neg Atrain/ (1 Atrain)). )). We respectively refer to them as Balanced Prior and Trainset Prior. Note that test distribution is unaccessible so we can not use it. Other methods (ICL., NoisyChannel and INN Prompting) do not technically incorporate any prior knowledge. So they are not concerned with this investigation dimension.
Analyses The results reported in Table For ICL, LLM naturally suffers from the bias learned in pretraining stage, thus is vulnerable to any different prior distributions. The performance greatly degrades in Setting B (-22.06) and Selling C (-22.17).
16

---

Published as a conference paper at ICLR 2023
For ContextualCalibratie technically, it necessarily requires prior distribution to rectify the LLM predicted label word probabilities. the prior knowledge does not match the (unaccessible) test dis- tribution, its performance will be greatly degraded. Specifically, if Balanced Prior is consistent with test distribution, the method performs well (Setting A, -1.77), otherwise, the performance degrades (Setting B, -20.24 and Setting C, -12.65.. Similarly, if Trainset Prior Assumption is consistent with test distribution, the method performs well (Setting C, +8.65), otherwise, the performance degrades (Setting B, -20.24 and Setting -18.9.22..
For CoisyChannel, its performance is rather robust (-2.60/-0.63/-..87 respectively in Setting A, B, C.. By re-formulating ICL into computing conditional probability of the input given the output, it is indeed an effective way to calibrate the task prediction.
For the proposed KNN Prompting, technically, it does not incorporate any prior knowledge of train or test distribution. Both the construction of datastore and the retrieving then predicting procedure do not vary w.r.t. different prior knowledge of distribution. The proposed method can robustly adapt to all imbalanced settings, including imbalanced trainset, testset and both. There is basically no performance degradation (-1.3/-2.6/+0.. respectively in Setting A, B, C, where -1.3/-2.6 can be considered within ordinary fluctuation).
Methods
0.125
0.25
0.375
0.5 (Balanced)
AVG
MaxDrop
Setting A.
Atrain
0.5, Ateat
0.5
ICI.
62.4
64.2
65.7
68.2
65.15
-5.85
ContextualCalibratt w/ Balanced Prior
74.0
69.7
68.5
70.3
70.64
-1.77
Contextual Callbbrtaaai w/ Trainset Prior
51.4
71.5
80,2
68.36
-18.92
NoisyChannel
70.0
70.9
71.9
71.33
-2.60
ANN Prompting
79.0
80.1
80.5
80.3
79.98
-1.29
Selting B.
Atrain
0.5, Aiest < 0.5
ICL
45.9
53.4
60.9
67.9
57.04
-22.06
ContextualCalibratn w/ Balanced Prior
50.0
$7.2
64.2
70.2
60.42
-20.24
Calibatin w/ Trainset Prior
50.0
57.2
64.2
70.2
60.42
-20.24
NoisyChannel
72.1
72.2
71.8
72.4
72.14
-0.63
ANN Prompting
77.1
78.0
78.7
79.7
78.38
-2.62
Setting C.
Atrain
Atst < 0.5
ICL
45.8
50.0
58.6
67.9
$5.57
-22. 17
CaatexttaaCalibrattin w/ Balanced Prior
63.0
57.6
61.7
70.2
63.13
-12.65
CantextualCalibration w/ Trainset Prior NoisyChanncl
87.8
84.5
78.8
70.2
80.33
+8.56
64.6
68.6
70.8
72.4
69.10
-7.87
ANN Prompting
79.8
80.7
80.3
79.7
80.12
+0.05
Table 7: Reliance on prior knowledge. All reported results are averaged across 5 datasets, and we further report average performance across all imbalance ratios. Xtrain/tes, denotes the subsampled ratio. MaxDrop measures the performance degradation compared to ordinary balanced setting (A, 0.5), where the best bolded and no drop highlighted in pink.
B.3 EMPIRICAL CHOICE OF k
In Figure 10 search for different choices of & MPQA, and found that it is generally rather robust choice within the wide range [3, [A]/2 1].
B.4 ROBUSTNESS UNDER IMBALANCED SCENARIO
We further test the robustness of KNN Prompting under imbalanced label scenario. Take binary classification like SST2 as example, we simulate imbalance ratio by controlling one the category proportionally to (0.5, 0.25, 0.125, 0.0625, 0.03 0.015625] of the entire training set, where 0.5 corresponds to the ordinary balanced scenario. We keep the test set intact, which results in a challenging out-of-distribution (OOD) setting. Results in Figure [I reveal the vulnerability of the proposed method. Under imbalanced setting, kNN Prompting is overwhelmed by the large quantity of anchors from the majority class where it simply far more easier find closer neighbors.
17

---

Published as a conference paper at ICLR 2023
0.82
0.000
0.953
3.86
0.0
0.235
A09
0:00
0.2
0.900
0.775
12
220
0.790
441
AI 256
14)-51
[4- 512
322
3776
14]-60
141-512
[4]
3224
14]
3
15
58
127
255
511
127 611
Figure 10: Empirical choices of k. Conducted SST2, SUBJ and MPQA respectively (left to right).
SST2 (FewShot)
SST2 (FullySuperrised)
0.9
0.9
0.8
Acc
0.8
Acc
0.7
0.7
0.6
InC Leuning
ANN Prompling
ANN Prompiing
Promping Nommstzwion)
ANN Prompptng (w Normaizzaioo))
0.015825
0.03126
0.0625
0.128
0.28
0.125
26
0:376
0.5
Imbalance Ratio
Imbalance Ratio
MPQA (FewShot)
MPQA (FullySupervssed))
0.8
Acc
Acc
0.6
Prompting
ANN Prompiing
JAN Promping Normstaniio)
ANN Prompting (Wl Normalizzior)
0.015625
0.05126
0.0625
0.128
0.25
0.5
0.125
0.375
0.5
Imbalance Ratio
Imbalance Ratio
Figure 11: Robustness under imbalanced scenario. Left: few-shot scenario (32). Right: fully-
supervised scenario (1024).
To address such performance degradation under challenging imbalanced scenario, we propose simple normalization trick: we average anchor representations to produce one centered anchor for each class, the resulting anchor is thus more representative and also avoids quantity distrac- tion. Such an adaptation works surprisingly well with no loss of performance even under ordinary balanced setting.
C COMPLETE RESULTS
C. DATA SCALING CURVE UNDER FEW-SHOT SCENARIO
We provide scaling curve w.r.t. each specific dataset Figure 12 corresponding to Table and Figure in the main manuseript.
18

---

Published as a conference paper at ICLR 2023
SST2
SUBJ
MPQA
1.0
0.9
0.9
0.8
0.8
0.7
0.7
0.8
0.7
0.5
Collout
0.5
Compona at Callretton
Contonall Calleccion
0.4
J9H8
Prometne
0.5
2
8
18
32
8
16
32
2
16
32
Num. of Training Shots
Num. of Training Shots
Num. of Training Shots
CR
MR
TREC
0.9
0.7
0.8
0.8
Ago
0.7
Acc
0.7
0.5
0.6
0.8
0.4
Contectaad
Comtonall
Chanral
0.3
Char uid Promeiie
2
2
32
Num. of Training Shots
Num. of Training Shots
Num. of Training Shots
Figure 12: Scaling curve under few shot setting for each specific dataset. The baselines are strictly comparable under m S 8, some baselines might be truncated when m 2 16.
Models Methods
SST2
SURI
MPQA
AGNews
CB
CR
DHPPddia
MR
KTE
TREC
AYG
In- In-Contest Learning
63. 63.427a
58.9
70.5
61.7=114
45.0gs51
85.3
59 59.92115
77.0g157
53.6211
$4.4
62.77
32
82,0
82.8a11
70.5=11
82.2en0
56.8-13
87.5+50
93.7+11
83.0a25
53.6240
71.5=s1
76.35
84.2,54
84.6,a7
81.2.10
83.3g1a
59.3
90.3a14
95.2
82.2
23.8
-24-44
78.66
0.5B
m 125
84.5a53
85.8
83.1
84.5
62.1
89.7
95.8
84.0a1s
53.6.11
74.2=44
79.73
256
14.8
86.0
53.2
86.5
62.1
89.5
96.1
53.5
$1.0
90.66
512
85.5
86 86.7
83.0.14
86 86.3
12.
89.
96.4au
83.5
52.8a14
824413
80.80
me 1024
85.5
87.6
84.8
87.6gus
62.1
87.5.10
96.7
83.6223
54.1ase
S6.7
$1.61
In-Contest Learning
81.3454
75.2
72.7=153
60.7=22
66.2a:81
83.5
12.2
53.0
54.2
68.31
199 32
87.7
77.0
75.2
86.2
58,9
88.2
91.1
13.9
53.6 110
61.8
76.97
64
87.2
82.5
83.0.40
86.211
62.9.61
95.5
84.3
19.1
68.9-38
78.86
1.5
m 128
86.3
g3
62.3g1
87.2
64.6.79
88.9
96.5
86.4
51.1
74.0
$0.12
m= = 256
19.
L17
61.6
97.3
85.6
52.7 im
78.1
51.27
612
89.0
87.6
83.7
88 88.4
64.6:79
90.1
97. 97.6aus
85.1
$2.9
80.5
$1.94
1024
88.2
88.4gi
1
64.6_19
86.7
97.8
$4.4
53.3
$3.0
$1.96
Context Learning
89.9
84.5
78,8
51.1.61
92.6in4
g8.$
92.5g14
52.4g12
64.6g1
77.29
Tr $2
89.9145
82 144
84
84.4113
50.0 54
89.9
97.0
91.2124
$2.8
75.1
79.72
64
91.4231
84.9
80.8
86 862211
27.1
91.0217
97.7
91.5218
51.6g24
29.
$1.14
2.7B
128
93.4g1
g7.5
$3.3
16.7
$7.1
90.2
98.6
914411
53.4
50.8
$2.25
m 250
92.933
k177
83.3
16.4113
57.1
91.6115
99.0
90.5
53.5
81.2
$2.32
m m 612
91.921n
89.4
$3.8
88, 11n1
57.1g18
91.3
99. land
9122
$4.3
83.1
$2.93
mo 1024
91.7
9155
83.7
113
57.1
89.8
19.1
90.9
57.7
86.7
$3.64
In-Contest Learning
92.7114
81
16.2
$2.5
92.0gsa
6200g3
Yo 82
92.7+18
85.5
86.2219
83.0
65.0=11
91.0425
97.4
91
55.2
71:2a52
$1.90
mi 64
92.7ans
87.2
$5.2210
15.
66.4
90.5
98. lans
91.121
51.6
72.3
$2.02
6B
128
92.211a
K7
85.2gg4
874411
63.6211
9p.6230
98 1104
91 91.0
55.6g30
75 85.8
82.79
256
92.940s
90.5
84.929
86.422
63.6-41
90.7am7
91
56.2a14
83.22
mo 612
93.5g1a
90.8
88.6 Gans
63.6_41
90.4g14
99.0ga2
90.3
59,4
$16
$4.27
m 1024
93.0111
900611
19.5
63.6g44
88. 38.3
98 Vans
90.415
57.0g33
81.51a8
83.82
In- Co-nntet I Learning
89 89.0443
91.
78 1455
$3.2.44
93. 93.4
y2 2424
89-9
55.8210
55.5
77.68
82
89.0g45
91.1
78.4
84.5 5422
77.9-41
89.5
97.4
91.1
56.8213
11444
$2.69
m 64
94.8
80
86.3
860
77.5
91.8211
57.7
70.5
84.47
13B
m. = 128
94X40s
1
26.223
87 87.4
78.9-44
89.62111
98 Vanc
92.0
58.2244
75.1
84.93
256
94.5200
90.5
85.3g53
87.2411
78.9-48
91. 91.2g11
99.5 99.5
92.0
59.3g11
84.8
$5.31
512
94.5
86.3
86 9619
78. 18
90.5g13
86.13
m 1024
94.5g04
91 4.24
87.0g2)
88. 58.5
78.924
88.1 111
99.3
63.0g1
16.1
85.87
In-Cantest I Learning
90.84411
83. K3.5
80.72.9
74.8445
64.6223
87.7250
93. 3404
93.4g11
61.6223
71.7a21
81.21
32
90.8
91. 91.3ams
80.72..
84.1
70.0.so
89.0as5
98. 3402
91.5
59.5
77.6223
83.56
43.6214
91. 91.9
85.5
85 85.3
69.3_7A
91.6414
3
94.1
661.225.
81.0g25
30B
m 128
94.3400
92.7
84.5
87. 1+14
70.7=x2
91.0g12
93.1
61.7245
79.8220
85.30
m= 256
94.4a64
93.8a11
$4.1
87.6
70.7=42
90.3210
99. lan3
93.7ans
62.1
82.1
85,80
512
14.3
94. 14.4
85.6213
87.9 87.9
70.7-12
89 89.811
99.1
93.2
61.821
83.8
86.01
93.9
85.4g33
87.9423
70.7
88.4
2222
93.4
61.0g40
86.2g13
86.02
Table 8: Full results for data scaling, corresponds to Figure 5 Some ICL results are reused
19

---

Published as a conference paper at ICLR 2023
C.2 DATA SCALING RESULTS UNDER FULLY SUPERVISED SCENARIO
We provide comprehensive results of NNN Prompting across data scales and L.MM scales in Table g corresponding to Figure$3jjn the main manuseript.
C.3 FULL RESULTS OF COMPARISON TO PROMPTCOMPOSE
We provide the full results of comparison between ANN Prompting and PrompiCompose in Table 9 corresponding to Figure in the main manuscript. The employed sentence encoder can be found at
cw/mode 1s
Setting Methods
S8T2
SURJ
MPQA
AGNews
CB
CR
DBPella
MR
RTE
TREC
AVG
My
In-Coutest Leamiigg
$1.3a54
64.3a113
75.2
72.7atus
60.7328
66221167
K3.5
72.2
55.011
54,2
68.31
68.451
63.62m0
75.2
69.0.3>
65.0asa
55.2-11
80.7a1
59/4114
53.0as
65.5
65.50
SBERT
71.0gas
67.5
75.2
8233
62.939
57
K3.
52.7g53
51.2
58.4.77
66.78
172 32
SimCSE
68.1245
69.8uai
75:211a
80.7420
66.4223
55.9-25
82.3a15
57.3
52.8117
54.5
66:31
Tram- Encoder
67.931
70.9
75.2
T7.7
61.4a0
56.721
82.5a10
57.3
$2.7
59. 59.3g50
66:21
JNN Prompling
87.7a35
77.0
75.2
86.2
58.9133
38.2
94.1
23.9
53.6150
64.8
76.97
BM25
68.7a:>
67.7
79.4s0
71.7a19
68.6241
-1.7
83das0
58.7ata
52.0a1a
65.9
67.18
SBERT
7188311
71.62xx
80.2
19.8
57.6216
52,8
63.7g57
09.37
(72
SimCSE
68.4347
69.7
81.6su
$3.1a12
714444
57,9-23
84S411
57.3-20
52,3
58.1
Trans Encuder
69:3
78.0g12
82.0a13
29.3
60.3
$7.9227
57.9
52.1
122
INN Prompling
87.2
82.5g12
K3.0g40
8622
62.9k.)
89.1
95.5
84.3
49.1g32
68.9
78,86
BM25
69.1as>
66.8
75 25.2
77.5a14
71A4s0
$6.1212
K5.5
59.8
$4.5a15
72.3
68.87
SBERT
71.2+
71.9
796656
85.3g10
69.6250
28.8
8722.9
10.1
53.3
24.9
70.24
128
SImCSE
709933
71.6
816636
54.5
73.2450
58.5_23
87.0,11
28.7
53.435
59,6
69.89
Trans Encoder
68.0.:3
25.5
82,6119
$2.9
73.2210
56.1 -ss
87.0.14
27.1
52.9
63.9-25
70.02
Prompting
36.3g19
83.8
8233
S7.2
64.6g1
38.9224
96.5gm1
86.4
51.1g17
74.0gz2
$0.12
BM25
72.0aa
37.3
71.4as
57.1 uns
83:2
58 584
53.8
76.6
70.60
SBERT
60.9g
72.3
K2.2
86 S211
6910
58
88 5.00
59 89.8
52.3
69. 1ans
70.89
02 256
SimCSE
7LA431
23.7
82.9
85.5
732230
19.7
89.1g1
57.8
$1.1
64.3
70.57
Trans Encoder
70.0..
76.6.14
84.
73.2a50
58.0.11
89.
58. 18.1
52.0aza
71 70.9:22
Prompting
89.1
85.0ga0
83.411
87.5g17
64.6
89.3
97.3200
85.62n0
52/7
78.1
$1.27
74.61
71.8
79.0
81.3
71Aaco
58.4.11
88.7
54.7
53.7
62.9.1.
72.14
SBRRT
71.6
24.3
R3, 13
1ate
6350
59.2-3n
83.6
58 88.8
81.8
73
72.
m 512
SimCEE
73.6220
75.5
83.9
$6.6116
73.1a50
29.1
904.1
59.2g21
5022,3
69.1ga1
72.07
Trans Enceder
73.1as
78.9217
81.2
$7.1
73.2210
$7.7
89 89.6.13
58.4
52.1
74.1
72.54
INN Prompling
89.04
87.6g1a
83.7g15
88.4.15
64.6,1
90.121
97.0ges
85.1g1
$2.911
80.5
$1.94
BM25
76
24.
79.3
83.611
71.4as
56.9.1a
90.2.1
60.3
52.7
73.12
76.623
83.7218
6450
91.1a11
59.6.n9
52.8a11
75.7gm
73.12
SimCNE
73.5
24
81.6111
$8.711
73:2500
57.5
91.0.0s
59.8
53.1g10
69.4.13
72.59
Trans-Encode Encoder
73. 73.4
78.421a
82.9
87.8
73.2200
$5.9
91.7ama
59.5
54.8
76.6
73.
INN Prompling
88.24
88.4gi
84.1
89.1
64.6g1s
86.721
97.8
84.4
53.3
83.0g14
$1.96
Table 9: Full results for comparison to PromptCompose, corresponding to Figure 6 results are reused (MPQA, 32 shot), see footnote in caption of Table for explanation.
Some ICL
C.4 BENEFITS OF WHOLE LM DISTRIBUTION UNDER FULLY SUPERVISED SCENARIO
We provide comparison between whole and partial LM distribution also on fully supervised scenario The conclusion remains invariant with few shot scenario demonstrated in Table 2
in Table T0
Methods ICL
S8T2
SUBJ
MPQA
AGNews
CB
CR
DBPedia
MR
RTE
TREC
AYG
81.3
61.
75.2
72.7
60.7
66.2
83.5
12.2
53.0
54,2
68.31
ANN Prompting
$8.2
$8.4a18
84.1a12
89.1
64.627a
86.7413
97.8400
84.4
53.3-42
83.0+14
81.96
JNN Prompting (Purtial)
87.7
69.3
8344
84.5
63.2
34.1
96.8
84.7 84.7g
49.0_a9
65.7
76.84
Table 10: Comparison between whole and partial LM distribution on fully supervised scenario.
In Table there are few cases where ANN Prompting gets identical results with ICL baseline. This happens when Mr= which leaves |A] = 0 as have invariably set [D] = My. To avoid exploiting exceptional split strategies for such specific case, we simply re-use the Underperformed ICL bascline results. In general, this only occurs on MPQA (when m = 32) and SST2 (when m = 32 and LLM >> 2.7B, 2048 tokens context), and should have few impact on the overall results and conclusion. Similar situation happens in Table L6-v2
htt
base
20

---

Published as a conference paper at ICLR 2023
SUBJ
MPQA
0.85
Demostration Set Size [D]
Demostration Set Size [D]
0.75
0.80
21 252321 17 15 13 9
0.70
0.85
0.70
0.80
0.85
0.55
5
6
7
3 111315171921 23252722831
Anchor Set Size |.A]
Anchor Set Size [.4]
CR
MR
0.88
Demostration Set Size [D]
Demostration Sat Size [D]
0.82
0.80
0.84
0.82
0.78
0.72
0.78
0.70
Anchor Set Size |.||
Anchor Set Size LA]
Figure 13: Split of Demonstration and Anchor Set. | [D] S MT.
C.5 SPLIT STRATEGY OF DEMONSTRATION AND ANCHOR SET
We provide results on more datasets regarding the investigation of split strategy in Figure corre- sponding to Section 4.3.2
D COMPARISON TO FINETUNING UNDER FULLY SUPERVISED SCENARIO
We compare ANN Prompting to standard PLM finetuning in more extensive data scale. For finetuning baselines in both Table and Figure [4 we set aaper-parameters following previous works (Schick & Schütze 2021). We set learning rate to le-5, batch size to 16, and training steps to 125, 250 or 500, respectively for m E {32, 64), ; {128, 256). 5122, 1024]. For CB, AGNews and RTE, batch size is adjusted to 8, for DBPedia, batch size is adjusted to 4 to avoid OOM. We observe that with the same model scale, &NN Prompting is superior than finetuning under the low resource setting, but inferior under fully supervised setting. This indicates its data utility factor is still smaller than finetuning. However, the main advantage KNN Prompting comes with LLM, which significantly outperforms the finetuning baseline without any gradient-baae based optimization. We have also discussed this in Section
E MORE CASE STUDY
We provide more case study, respectively on SST2, TREC, AGNews and MR (Figure and 18.
[5
17
16
21

---

Published as a conference paper at ICLR 2023
Avg Performance on 10 Tasks
0.88
0.84
0.82
0.80
0.78
Finetuning 0.88
0.78
kNN Prompting 0.8B
KNN Prompting 30B
32
64
128
256
512
1024
Num. of Training Shots
Figure 14: Comparison to finetuning baseline.
20
kNN Prompting
ICL Pred
10
Acc
81.3
positive (X)
negative
0
C
positive (X)
negative
-10
D
positive (X)
negative
negalive
E
positive (X)
negative
-20
positive
positive (X)
negative
(Test)
looks and feels like project better suited the small screen.
ICL Pred
Golden Label
distance
see simone and consider dvd rencal Instead
posicive (X)
negative
0.000348
kNN (Anchor)
positive (X)
nogative
bast-iinue
posicive (X)
negative
0.000430
Figure 15: Case study on SST2, all test instances are randomly selected given that ANN Prompting outperforms ICI.
kNN
20
ICL Pred
Prompting
Acc
54,2
83.0
B
entity (X)
human
-10
C
number (X)
entity
D
number (X)
expression
-30
E
entity (X)
human
cndty exp meston
desor
-40
human
entity (X)
description
-30
-20
-10
10
20
30
40
(Test)
What amitriptyline?
ICL Pred
Golden Label
discance
|
What digicalis? 2
human (X)
description
0.0697
KNN (Anchor)
What BPH
number (X)
expression
0.0748
agent orange?
(X)
description
Figure 16: Case study on TREC, all 5 test instances are randomly selected given that ANN Prompting outperforms ICL.
22

---

Published as a conference paper at ICLR 2023
30
B
ICL Pred
kNN Prompting
20
Acc
72.7
89.1
10
B
technology (X)
business
0
D
technology (X)
business
-10
technology (X)
business
-20
technology
world sports
E
world (X)
sports
-30
business
technology (X)
business
-30
-20 -10
10
30
Ofl Falls From Record on Concern High Prices May Slow Growth. Crude oil fell from yesterdby #3Pss record of 11$54 barrel in New concern
A
Golden Label
ICL Pred
(Test)
high prices may slow
reduce demond for energy
"Crude record-high settlement overnight for the fitt stright aession, closing below li$US54 a barrel, reflecting real and perceived threats globil all spply."
technology (X)
business
0.0146
Update 13: Oll Prices Little Changad on Norway News, Crude all changed Tuesday will and their four-morth strike after the government ordered them back the
KNN (Anchor)
Norway they
world (X)
business
0.0186
Ofl Rebounds After Iraq Pipeline Arack. Oil prices rose on Friday sabotage attack on lraqi infrastructure regnited about the country despite peace
reliabiiity exports end an uprising Najaf.
world (X)
business
0.0432
Case study on AGNews, all 5 test instances are randomly selected given that ANN Prompting outperforms ICI.
Figure 17:
positive
negative
kNN Prompting
ICL Pred
2
Acc
72.2
84.4
positive (X)
negative
positive (X)
negative
-1
positive (X)
negative
positive (X)
negative
-3
positive (X)
negative
-00
-10
0
10
20
Golden
A (Test)
"it's not bad premise jjust bad movie.
ICL Pred
distance
Label
among five friends, fails lend the characters' individual stories enough dramatic resonance make about them.
"as abruptly
positive (X)
negative
0.0004465
"the three leads produce adoquate performances, but what''s missing this material any depth feeling.
positive (X)
negative
0.00052309
(Anchor)
"morrissatte performed difficult task indeed hel's takan one of the world's most fascinating stories and made dull. lifeless, invitating."
positive (X)
negative
0.00061
Figure 18: Case study on MR, all test instances are randomly selected given that ANN Prompting outperforms ICL.
F PROMPT TEMPLATE
See Table II for the used templates (Adopted from Lu et al. (022)). They are intuitively designed and the proposed method should be robust with choices of templates.
23

---

Published as a conference paper at ICLR 2023
Task
Template
Label Space
SST2
Review: contains wit, only labored gogs
negative, positive
Sentiment: negative
Review; film is powerful accessibll funny.
Sontiment:
SUBJ
Toput: the script isn't very good
someone as gifted hoffiman the actor) make
subjective, objective
work
Type: subjective
Input: be must do this in secret so that the parents and school personnel know nothing of his plan.
Type:
MPQA
Review; would not lind it at all strunge
negative, positive
Sentiment: negative
Review; as small yet acceptable as possible
Sentiment:
AGNews
Input: Carlyle Looks Toward Commercial Aerospace (Reuters). "Reuters- Private investment firm Carlyle Group, which has reputation for making well timed and oecasionally conirovesiial plays in the defense industry, has quietly placed its hets an another part of the market.
world, sports, business, technology
Type: tochnology
Input: Superstar Kewell remains centre of attention. Secceroo forward Harry Kewell loosens up by tossing around
Bondi beach yesterday. Photo: Craig Golding, There were dozen Socceroos standing on raised platform Sydney #39;s
Type:
CB
Premise: was complex language. Not written down but handed down. One might say it was peeled down.
False, True, Neither
Hypothesis: the language was peeled down
Prediction: False
Premise: A: so I don't 't know if wasn't drug tested based on that or because the man who hired me didn't 't requestt the drig test, because know that my company does dmig testing on occasion. B: before have the right or do they have
Right, Well, for instance, does the company you worked the ability to say, hey, we've already drug tested her and she came up negative. A: Well, no, don't think they can another compuny not drug me just by saying that didn't mean. Hypothesis: they can force another company not drug test her Prodiction:
CR
Review: 's not as stylized as a sony or samsunn.
negative, positive
Sentiment: negative
Review:
i went out and got the canon today.
Sentiment:
DBPedia
Input: Geoffrey D. Falksen (horn July 31 1982) is an American steampunk writer.
company, school, athlete, politics. transportation.
Type: artist
Inpur: Monster Night is 2006 film directed by Leslic Allen and Lorenzo Doumani. Type:
building, nature, village, animal, plani. album, lilm, book
MR
Review: "you might say tykwer has done all that heaven allows if you wanted to make as anti- kiesloskii pun possible. suffice say total promise left slightly unfulfilled
negative, pusitive
Sentiment: negative
Review;
an alternately raucous and sappy ethnic sitcom... ... you 'd be wise to send your regrets.
Sentiment:
RTE
Premise: man duc in court later charged with the murder 26 years ago teenager whose case was the first to he featured on BBC One's Crimewatch. Colette Aram, 16, was walking to her boyfriend's house Keyworth. Nottinghamshire, on October 1983 when she disappeared. Her body was later found
lield close to ber home. Paul Stewart Hutchinson. 50, has been charged with murder and is due before Nottingham magistrates later."
Hypothesis: Paul Stewart Hutchinson is accused of having stabbed girl.
Prodiction: false
pay accumulated after six maternity leave would be just 5,300 in the UK and 5,850 in Irelund. Emtilementt in Germany would also be relatively low, at 5, 5,900, along with those in France. Spain and the Notherlands, all at 6.750. At the other end of the scale, pay received after six months leave Italy would be 9,150 while Denmark and Norway it would be as much as 1,000.
Premise: For
earning 22,000 year,
Hypothesis: Maternity leave varies in Europe.
Prediction:
TREC
Question: How did serfdom develop in and then leave Russia ? Typc: description
description, entity. expression, homan, location, number
Question:
What Shukespeare nickname
Type:
Table 11: Templates for ICL. These are minimum cases with only one demonstration example for illustration.
24


Reviews:
[{"review_id": "iooYfnqXjZ", "reviewer": "Reviewer_SRJJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes kNN prompting, a novel retrieval-based model for more accurate inference accuracy. kNN prompting queries LLM with the training data and get the representation as anchors. In the prediction phase, kNN prompting only performs nearest neighbor for a better accruacy.", "summary_of_the_review": "This paper proposes a novel idea for deploying large-scale LLM in practice. This paper is also well-written with clear illustration of kNN prompting.  The empirical evaluation also suggest the strength of kNN prompting in long sequences.", "strength_and_weaknesses": "The kNN prompting idea is novel. It aligns with the retrieval philosophy that we can make full use of the trained large language models for a better inference. The strength can be summarized as follows:\n\n1. The kNN prompting is a gradient-free paradigm for deploying LLM.\n\n2. The kNN prompting integrates the idea of the nearest neighbor in the inference of LLM, improving the LLM performance without further tuning.\n\n3. The empirical results are promising, especially for long sequences. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L8rc7db9kIm", "reviewer": "Reviewer_vijR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In-context Learning (ICL) is one of the most prevailing paradigms for inference with large language models (LLM) without parameter update. However, as the sizes of context windows are limited for LLMs, only a small number of training examples can be used for prompting, preventing ICL from scaling to larger training set. To overcome this bottleneck, this paper proposed a new method for LLM inference with no parameter update called kNN prompting: unlike ICL, where labels are directly predicted based on next-token distribution, kNN prompting make predictions by leveraging/retrieving training examples (when available) that share similar next-token distributions with the test example. Empirical results show that kNN prompting not only scales effectively as the number of training examples increase but also outperforms vanilla ICL on some datasets in the few-shot setting. Moreover, when a relatively large training set is available, kNN prompting even outperforms the (relatively smaller) fine-tuned models as the number of parameters of LLMs exceed a certain threshold (6B). To summarize, this paper proposed a simple yet effective way of scaling LLMs inference to large training sets with no parameter update.\n", "summary_of_the_review": "Overall this is a good paper that proposes a simple and elegant approach with strong empirical results. The only weakness is the limited novelty of the methodology itself; to provide more unique insights, the authors may want to provide a more detailed analysis about the question discussed in 4.3.5.", "strength_and_weaknesses": "Strengths:\nkNN prompting is a simple yet very effective approach for scaling LLMs inference to large training sets; comprehensive ablation study demonstrates the robustness of the approach across different choices of hyper parameters (k) and random seeds. \n\nWeaknesses:\nThe discussion (Sec. 4.3.5) on why kNN prompting outperforms vanilla ICL in a few-shot setting is very interesting: it seems to suggest that kNN prompting is implicitly solving the issue of “surface form competition” where some answers have higher probabilities in the prior distribution of LLM. I wonder how kNN prompting compares against to other calibration techniques [3] that solves the same problem.\n\nIn terms of novelty, the method itself is more of a variation of the methods proposed in prior works that retrieve extra data via kNN to help model inference [1, 2].\n\n[1] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learn- ing Representations, 2020. URL https://openreview.net/forum?id=HklBjCEKvH.\n[2] Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. Nearest neighbor zero-shot inference. arXiv preprint arXiv:2205.13792, 2022.\n[3] Min, Sewon, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. \"Noisy channel language model prompting for few-shot text classification.\" arXiv preprint arXiv:2108.04106 (2021).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6Gye9EFUISq", "reviewer": "Reviewer_wqbJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes kNN Prompting, which addresses the problem of in-context learning (ICL) that the context length is limited, so that the performance cannot scale with the number of available training examples. Similar to ICL, kNN Prompting also formulates the downstream tasks as prompt completion tasks. Differently, at inference time, kNN Prompting not only relies on the demonstration examples in the context of the input, but also will retrieve anchor examples from a datastore built by taking all the training examples. \n\nThe paper conducts experiments on ten downstream tasks, with different model sizes and different scales of training set. They find kNN Prompting outperform ICL under few shot settings and can scale up with as many training examples as are available. Ablation studies and analyses are conducted to study the model design choices. ", "summary_of_the_review": "In summary, I think this paper is well-motivated and is written clearly. It presents very interesting and insightful results. I hold a positive attitude towards accepting this paper.\n", "strength_and_weaknesses": "Strengths:\n- In general, the paper proposes a simple and effective solution to enable the prompting methods to leverage more training examples, instead of merely relying on the demonstrations in the context. The approach is very interesting and the results are strong. \n- The motivation of the paper is clear and makes sense to me; and the paper is positioned well. I appreciate the authors' efforts on addressing the sequence-length issue when using ICL. I believe that this is a crucial problem of using LLMs on downstream tasks in a full supervision setting and the community should facilitate addressing this problem.\n- The paper is presented clearly. \n- I also like the discussion section where the authors raise potential concerns for the existing retrieval-based models. I believe addressing these problems can further reveal the capacity of retrieval-based models.\n- The paper has conducted ablation studies for understanding the model designs, e.g., KL divergence vs L2 distance; the sensitivity of the number of demonstrations/anchors in kNN Prompting. These studies are informative.\n\n\nWeaknesses:\n- What confuses me is that during inference of kNN Prompting, the method will retrieve anchor examples that have similar predicted probability distributions (measured by KL divergence). Intuitively, this means that the predictions are likely to be the same (or at least very similar) to the current testing example. That means it is not likely that the prediction will be different. I expect detailed discussion about how retrieving anchor examples can actually help the predictions.\n- The predictions are made based on majority voting of retrieved examples. This method may suffer when the labels are imbalanced. For example, when one label has much fewer examples compared to other labels, this label will not be favored by the majority voting system.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5WL63ncjFn", "reviewer": "Reviewer_P4ZY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work considers application scenarios that utilize large language models in a in-context learning paradigm with training data that cannot fit the context due to the input-length limitation.\nTo alleviate the challenge posed by the maximum length limit of language models, this work proposes the knn prompting method.\nThe key idea of knn prompting is to store the information of a large amount of training data in an additional memory cache.\nDuring the inference process, the output is generated by the nearest neighbor voting for a given input in the memory cache, rather than by the model directly.\nThis work claims that knn prompting can greatly improve in-context learning, especially in low-resource or fully\nsupervised scenarios.\n\n", "summary_of_the_review": "There is a potential misleading on the novelty of this work, since neither the conceptual relevance to previous work nor the experimental comparison with previous work is clearly mentioned in this work.", "strength_and_weaknesses": "# Strength\n* This work proposes knn prompting that can help the large language models to better suit the application scenarios with enough training data rather than a few shots.\n* For the proposed knn prompting, this work conducts further analysis and shows the robustness of this method. The qualitative analysis illustrates the explanations behind the performance of this method.\n\n# Weakness\n* **Lack of comparisons with previous retrieving-based in-context learning methods**. For the scenario considered in this work, a line of closely related work is about prompt retrieving, which also considers to store the extra training data in an additional memory and utilizes similarity-based choosing. I suppose the main difference between retrieving methods and knn prompting is that the former requires to store the whole data while the latter only need to store a probability distribution. But I suppose both of them do not go beyond your considered scenarios. For instance, [1] considers to retrieve prompting examples from the example bank with knn selection. Since you have mentioned this work in your related work, why not experimentally compare with it?\n**In a word, a line of previous work can also deal with the challenge considered in this work, but this line is neither clearly mentioned nor experimentally compared**.\n\n* **The basic idea behind knn prompting (i.e., calibration) is not clearly introduced and compared**. It seems that the the basic idea behind knn prompting is to calibrate the output probability of the model. Such an idea (along with in-context learning) has also been explored in many previous works, such as [2]. Actually, [2] is already mentioned in the paper but you didn't clearly show the relevance. **In a word, the basic idea of calibration should be clearly introduced to readers and further compared with previous works**.\n\n* **Potential misleading on the novelty of this work**. It seems that this work wants to highlight its novelty on the considered application scenario (i.e., length limitation for in-context learning) and proposed methods (i.e., probability calibration for in-context learning).\nBut in my view, both two points are not novel enough (according to the two weakness mentioned above).\n**For several highly related works, this work simply lists them but avoid to show a clear relevance and comparison, causing a potential misleading on its novelty**.\n\nAll in all, both the considered application scenario and the basic idea of proposed method in this work have been explored in many previous works, but neither the conceptual relevance nor the experimental comparison is mentioned.\n\n[1] What makes good in-context examples for GPT-3?\n\n[2] Calibrate before use: Improving few-shot performance of language models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster