Generative Calibration for In-context Learning
Authors: Zhongtao Jiang
Yuanzhe Zhang
Cao Liu
Jun Zhao
Kang Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: Calibration
In-context Learning

Abstract: As one of the most exciting features of large language models (LLMs), in-context learning is a mixed blessing. While it allows users to fast-prototype a task solver with only a few training examples, the performance is generally sensitive to various configurations of the prompt such as the choice or order of the training examples. In this paper, we for the first time theoretically and empirically identify that such a paradox is mainly due to the label shift of the in-context model to the data distribution, in which LLMs shift the label marginal $p(y)$ while having a good label conditional $p(x|y)$. With this understanding, we can simply calibrate the in-context predictive distribution by adjusting the label marginal, which is estimated via Monte-Carlo sampling over the in-context model, i.e., generation of LLMs. We call our approach as generative calibration. We conduct exhaustive experiments with 12 text classification tasks and 12 LLMs scaling from 774M to 33B, generally find that the proposed method greatly and consistently outperforms the ICL as well as state-of-the-art calibration methods, by up to 27% absolute in macro-F1. Meanwhile, the proposed method is also stable under different prompt configurations.

Generative Calibration for In-context Learning
Zhongtao Jiang'?, Yuanzhe Zhang Cao Liu", Jun Zhao? Kang Liu 12,4 Liu 'The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences MMetuan, and "Shanghai Artificial Intelligence Laboratory {zhongtao. jiang, yzzhang, jzhao, kliuj@nlpr. ia. cn, liucaoêmei tuan. com
Abstract
et al., 2022; Kim et al., 2022: Wei et al., 2023). Notably, ICL found be pathogenically sensi- tive to the prompt configuration, c.g., the template, choice, and even the order permutation the train- ing examples can cause the performance to vary from nearly chance to state-of-the-an (Gao et al., 2020; Zhao et al., 2021; Lu et al., 2021). This in- stability motivates lots of efforts on searching for better prompt in terms of better few-shot training examples (Rubin al., 2021; Liu et al., 2021; Su et al., 2022: Wu et al., 2022: Wang et al., 2023), better order permutation of those examples (Lu et al., 2021; Wu et al., 2022), and slot position in the template (Holtzman et al., 2021; Min et al., 2021). Another promising direction is calibration (Zhao et al., 2021; Han et al., 2022; Fei et al., 2023) that adjusts the ICL predictive distribution via an estimated bias. Compared with prompt optimiza- tion methods, such a stream usually much more lightweight and gains substantial and consistency improvement, while also reducing the instability. However, many of those ideas are based on heuris- tic intuitions, and there are few principled investi- gations characterizing how the prompt affects the predictive distribution.
As one of the most exciting features of large language models (LLMs), in-context learning is a mixed blessing. While it allows users to fast-prototype task solver with only a few training examples, the performance is gener- ally sensitive various configurations of the prompt such as the choice order of the train- ing examples. In this paper, we for the first time theoretically and empirically identify that such a paradox is mainly due to the label shift of the in-context model to the data distribution, in which LLMs shift the label marginal p(v) while having a good label conditional p(zly). With this understanding, we can simply cali- brate the in io-tentext predictive distribution by adjusting the label marginal, which is estimated via Monte-Carlo sampling over the in-context model, i.c., generation of LLMs. We call our approach as generutive calibration. We con- duct exhaustive experiments with 12 text classi- fication tasks and 12 LLMs scaling from 774M 33B, generally find that the proposed method greatly and consistently outperforms the ICI. as well as state-of-the-ee art calibration methods, by up to 27% absolute in macro-F1. Mean- while, the proposed method is also stable under different prompt configurations.
This paper fills this gap. In specific, we focus on the text classification task. We study the in-context model p(z, v), i.e., the joint distribution of the in- put x and label y given a prompt consisting of a few training examples, while the true data distribu- tion denoted as ((x, y). Specifically, we identify that the in-context model p(r. y) poses label shift (Saerens et al., 2002) to the data distribution g( y) both theoretically and empirically. Our theoretical analysis (Section 3.1) 1) is based on the Bayesian in- terpretation of ICL (Xic et al., 2021; Wang et al., 2023), showing that the in-context model p(æ,, y) is not ensured be valid estimate of the true data distribution g y), due to the prior preference of LLMs and the limited amount of training examples. Next, we empirically find that the in-context label
1 Introduction
The learning paradigm has revolutionized in the era of large language models (LLMs), in which one of the most exciting is in-context learning (ICL) (Brown et al., 2020). Compared to regular su- pervised learning, LLMs can learn implicitly by prompting a few training examples as demonstra- tions, i.c., context. This paradigm allows users including amateurs to fast-prototype task solver with much less annotated data, while sometimes also gaining remarkable performances.
There are plenty of studies empirically analyz- ing ICL (Zhao al., 2021; Lu al., 2021; Min
(oode implementation is available at con/ com/ comm/changgenseng//eeee http://geerrrreeeeeeeeeeeeeeeeee calibration.

---

conditional p(z ) is good approximation of the data label conditional g( while the in-context label marginal p(y) generally deviates from the true one q(y), which provides persuasive evidence of label shift in the in-context model (Section 3.2). Then, with this understanding, we can calibrate the in-context classifier p(y(zz) by simply adjusting the label marginal p(y), which is naturally obtained by marginalizing out the input and can be effectively estimated via Monte-Carlo sampling, i.e., generat- ing new instances from the in-context model. We call our approach generative calibration (GC). Actually, most previous saute-of-the-are calibra- tion approaches (Zhao et al., 2021; Han et al., 2022: Fei et al., 2023) implicitly assume that the ICL. pre- dictive distribution has a shift on label marginal. However, none of them validate the soundness of that assumption, while we for the first time verify this systematically. What's more, their estimates of the label marginal are too heuristic to be solid, while our MinteeCCrll estimate an unbiased one to the in-context label marginal p(y).
template to format the example to be demonstra- tion. We call this distribution the in-context model and use p(e) as its shorthand: p(e) = p (())
In this work, ICL is utilized for classification tasks. Then, each training example e, is a tuple:
e (Ii, Vi), where E & is the input sequence and y E D is its corresponding label. In this case. the generative model factorizes as:
p(z, y) p(z)p(ulz)) PIM
PLM
@ (x)) (2)
In most cases, we only use the classifier p x)) to predict the label of the given input sequence x. By this intuitive construction, the classifier performs surprisingly well on many NLP tasks, sometimes even achieving state-of-the-art.
Distribution Shift of In-context Model
In this section, we provide both theoretical and em- pirical evidence showing that the in-context model has distribution shift the label.
We conduct exhaustive experiments on 12 text classification tasks and 12 LLMs scaling from 774M to 33B. The results show that the proposed GC greatly and consistently improves the perfor- mance of ICL (by up to 27% absolute in macro-IFl) and outperforms previous seate-of-the-ae calibra- tion methods (by up to 9% absolute in macro-F] for all LLM scales. Meanwhile, GC is also stable towards changes of the choice and order of training examples in the prompt, and exceeds or be compet- itive to prompt optimization methods, even though some of which are not in the true few-shot learn- ing setting (Perez al., 2021). Overall, GC is lightweight and effective approach making LLMs better few-shot learners and saves the effort of cum- bersome prompt engineering.
3.1 A Bayesian View
It is shown that the in-context model would im- itate the prompting examples to generate similar sequences (Meyerson et al., 2023). One principled and popular explanation is that such a model is approximation of the posterior predictive distribu- tion in Bayesian statisties (Xie al., 2021; Wang et al., 2023):
(l ~ p(0)D.))px, v/ 0)d0 p(0)D.) (()) 1T, p(a,, Vil0)
p(D.)
where E θ is latent parameter controlling the "topic" of the sequence like LDA (Pritchard et al., 2000) topic model?. p(((D.) is the posterior given the training examples, which softly seleets the topic. Also note that this identity approximately holds (R), since Bayesian approaches consider all the ex- amples to be exchangeable", while clearly, LLMs have order preference. We assume that LLMs have
2 Background
In the ICL paradigm, we have few training ex- amples D. (ct) where each one is inde- pendently and identically sampled from the data distribution, which we denote as q(e). Prompting a language model LM with those examples, the continuation distribution a generative model:
"The topic could viewed as task. 'Exchangeability means that the joint distribution a se- quence c1.k is invariant to any order permutations p(rr.)) (())())=<(R)). According to de Finetti's 's theorem (Aldous et al., 1985). if random sequence is exchangeable, it is then equivalent to a mixture model that each sample is conditional identically independent given latent parameter, which forms the null hypothesis of Bayesian inference.
p(e(D?) (IM( 2 (DT) - ((
where π represents a specific order permutation, + denotes the text concatenation operation, (-) uses

---

est, i.c. there exists 0* O such that p(0*) > 0 and p(z,y/0") = g(z, v). Ideally, the posterior recognizes the true topic 6* of the training exam- ples without uncertainty, i.e., p(() = 8(0 - 8*), then the in-context model p(z,3) y) approaches the true data distribution ((2,, v) exactly. According to Schwartz's theorem for posterior consistency (Schwartz, 1965), this ideal case can be asymptoti- cally realized as the number of training examples K goes to infinity, which is impossible for LLMs* a topic supporting the data distribution of inter-
of label shift: p(rly) ((>ly)) and p(y) A q(y). While the second point is straightforward by simply comparing the empirical label marginals, the first one poses a challenge given that we are agnostic to the true model ((z, v) but only some samples from it (i.e., the validation set). As a surrogate, we don't 't seek to verify p(ally) approaches g(rly)) exactly. We shall show that p(zly) performs very well in ranking the examples, illustrating that it is good approximation of q(æly). Then, our goal is to verify: 1) in-context label conditional p(arly) is good to g(rly) and 2) in-contexi label marginal p(y) is different from the data label marginal (v)- Our investigation is on SST2 dataset (Socher et al., 2013) using GPT2-XL (1.5B) (Radford et al., 2019), GPT-NEO (2.7B) (Black et al., 2021) and GPT-J (6B) (Wang, 2021; Wang and Komatsuzaki, 2021). For simplicity, we only display results of
When only a few training examples are accept- able to LLMs, there are at least two negative con- sequences. On one hand, as shown in Equation (3), the prior preference of LLMs, i.e., topic prior p(0), would dominate the posterior P()) Empirical evidence include common token bias (Zhao et al.. 2021; Razeghi et al., 2022) and prediction insensi- tivity when the training labels are corrupted when K is small (Min et al., 2022; Wei et al., 2023; Pan et al., 2023). On the other hand, it is not enough to express the data distribution with just a few train- ing examples. To see this, consider movie review sentiment classification task. Suppose there exists a topic 6' such that p(z, y|0") only generates posi- tive reviews". When we happen to have only few positive samples hand, the population of the pos- terior (0[ is easy to fall into regions close to 0' instcad of the true topic 0*, which would shift the label marginal p(x(y) so that the probability of pos- itive label is risen to a level far beyond the true one. While being -oddere- in previous works in terms of majority bias (Zhao al., 2021) and under- specification (Si et al., 2023), this phenomenon is formally termed as label shift (Schölkopf et al., 2012), which proposes that 1) the model and data share the same label conditional: p(xly) = (x)))). and 2) differ in the label marginal: p(y) + q(v). Besides this theoretical analysis, in what follows, we empirically validate that label shift exists in in- context models.
GPT2-XL (1.5B) and left others in Appendix E.
3.2.1 In-context Label Conditional Good
Without the label marginal p(y), the label condi- tional p(æly) alone can't tell if the input æ belongs to the class y. However, it can tell which of the two samples æ and x' is more likely to belong to y in- stead of other classes. In specific, let's first consider binary classification task in which Y = {N, P}, where we denote N and P as negative and positive, respectively. We score each input æ as the ratio of label conditionals:
s sxx) = p(x)P) p(N))((PIz)
p(Plz)
(4) What does this score mean? Since the score is pro- portional to the odds: the ratio of the probability that the in-context recognition model p(y|æ) pre- diets positive to the probability that it doesn't, t, it measures how much more confidence of the posi- tive prediction than the negative prediction. Given this score, we can obtain a rank r, of the entire val- idation set, where examples of the higher ranking are more likely to be positive. The rank quality can be justified by the receiver operating characteristic curve (ROC) and the area under it AUROC AU- ROC formally stands for the empirical probability that random positive sample is ranked above a negative sample (Zhou, 2021). Concretely, denote the validation set D,, = D.' U D., where the su- perscript represents the subset that contains all the hard to analyze the model if we don't 't have validation set for grounding results.
& p(N(z)
P(
P ((P))(N[Z)
3.2 Label Shift Empirical Validation
By assuming to be accessible to a labeled valida- tion ser we now empirically verify two points
'Though there may also exists other conditions for a good posterior estimation, c.g., high-quality training examples, we just can't 'tatteee ensure them to happen in the few-shor scenario.
'It is reasonable to believe that 0' has support in the topic prior such that p(0') ) > O. since there are contiguous positive reviews in the pretraining corpus.
"Validation set violates the true few-shot learning setting
"In multiple-class case, we use macro one-to-one AUROC, which is abbreviated as macro-AUROS for simplicity.
only when it's used for model development or selection. It is

---

P domminated
Balanced
N dominated
dominated
Balanced
N dominated
Macro F MiMarro AUROC
.75
Macro-F1 MacrovNUROC
Fly
.25
Pppp
Nppp
Purp
Ppip
PPPN
Nhep
NPPp
PAMP
NPPN
PAPP
PPHN
NNNN NPW PWEN
10490
PPPp
hppp
PHPP
PPPp
PPPN
NNPp
NeW
PhMP NPPN PHPP PPiN nevi NeNn NPPN PVIN ANNIN
Prompt configuraaiio
Prompt configuration
(a) Macro-FI/AUROC performances.
(b) Estimated in-context label marginals.
Figure 1: Macro-FI/AURRO performances (a) and estimated in-context label marginals (b) of GPT2-XL (1.5B) across different prompt configurations on SST2.
examples in the corresponding label, AUROC is:
3.2.2 In-context Label Marginal is Different
The second point, i..., p(y) A g(v) is straightfor- ward verify by comparing the empirical label marginals of in-context model and data. First, we need to estimate the in-context label marginal, which is the marginalization of the joint distribu- tion:
ExpEDy ExNEDY ((rs(xP) > ))
(5)
|
where (2) is the rank of x. Now, if a model is perfect, i.e., p(zly) = (<>ly), it will place all the positive examples above the negative examples, where AUROC reaches its maximum 1. Therefore AUROC somehow measures the closeness between p(xly) and g(<)) y)-
p(v) Σ p(z)p(ula) ZEX
(6)
The exact marginalization is intractable since we can't enumerate all the sequences in the input space X. Therefore, we apply Monte-Carks sampling to obtain an unbiased estimate:
Since the rank invariant to the z-independent term, e.g.. p( P))p(N) in Equation (4), we can use the odds of the classification distribution p(y(x) to compute AUROC, while this metric is actually eval- uating p(lly). We evaluate the 4-shot ICL perfor- mance with prompts having different class balances and orders on SST2, where each prompt configura- tion is evaluated in three random runs. The results in Fl and AUROC of GPT2-XL. (1.5B) are shown in Figure (a)), where the x-axis represents differ- ent prompt configurations. For example, "PNPP" indicates three positive examples and one negative example ordered in the second place in the prompt.
p(v) p(v)
1 = - PLM (3W12(7))
(7)
where xt is sample from PLM (D))) i.e., a generated sequence of LLMs prompted with training examples in Dr, The whole process is shown in Figure 2. In this paper, set /. 100, which is enough for a stable estimation (Details are shown in Appendix J). As for the data label marginal q(y). simply counting the number dif- ferent labels in the validation set and then normal- izing them forms a maximized likelihood estimate. We plot the estimated in-context model and data marginal probability of the positive label p(y = P) and q(y = P) in Figure 1(b). We
We can obtain two findings: 1) On average, ICL achieves very high AUROC values that don't 't match the F1 across prompt configurations: the average AUROC performance typically exceeds 0.95, while the average Fl performance hardly reaches 0.8. 2) In contrast the sensitivity of F1, AUROC is stable the prompt changes with low variances. These findings provide strong evidence that p(*ly) is a stable good approximation q(<ly)) no mat- ter of prompt configurations, roughly establishing p(zly) z q(xly).
that the in-context label marginal deviates from the data label marginal in most cases. Concretely, the devi- ation extent positively correlated with the majority label, which is in expectation as discussed in Sec- tion 3.1. Also, GPT2-XL (1.5B) has much prior

---

InReview: are more deeply thought through than most right-thinking films InSentiment: positiveln
InReview: delivers roughly equal amounts of beautiful movement and inside information .nSentiment: positivein
generations of LLMs
(PP)=)
pioneering analysis the natural religion movement.in
0.17
0.83
Average
pretty dam good.An An
0.02
0.98
InReview:
InSentiment
unexpected and absolutely amazing.In In
0.03
0.97
a surprisingly short and clear book. In
0.12
0.88
((N) = 0.04 ((P) 0.96
Figure 2: Illustration of the marginal distribution estimation on SST2. We first sample 1. sequences (the generation ends when it meets line break "ln") conditioned on the prompt concatenated with few-shot training examples and the input prompt phrase ("nReview:" in this case), and next concatenate the output prompt phrase ("InSentiment"" in this case) to obtain their in-context predictive distributions. The in-context label marginal is then estimated by averaging those in-context predictive distributions.
preferences on the positive label, where except the case when all the training examples are negative, i.e., "NNNN", the marginal probability of the label positive exceeds 0.5 in other cases.
Since our method involves multiple generations (for estimating p(y)), we denote it generative calibration (GC). GC could be also explained via cost-sensitive learning theory (Elkan, 2001; Ling and Sheng, 2008), where the label marginal is one of the most widely-used costs (Buda et al., 2018). Although previous state-of-the-art calibration methods (Zhao et al., 2021; Han et al., 2022; Fei et al., 2023) share the same form as ours in Equa- tion (9), in contrast to our principled unbiased es- timate, their estimates too heuristic to be solid. For example, contextual calibration (Zhao et al., 2021) estimates the label marginal via heuristically constructed seemingly context-free texts such as "N/A": p(y) = p(yl "N/A"). However, these texts are never verified to be context-free as claimed, be- cause LLMs might have preference bias on them. Also, LLMs barely see these texts in the pretraining corpus, which would pose -istr--distt (OOD) (Kim et al., 2020) problem.
In short summary, in-context models are mainly biased by shifting the label marginal, which is gen- erally implicitly assumed in previous works (Zhao et al., 2021; Han et al., 2022; Fei et al., 2023). How- ever, to the best of our knowledge, we are the first to verify this systematically.
4 Generative Calibration
With the understanding in the previous section, if we accept that p(xly) = q(=ly), the solution is quite straightforward. that is, adjusting the label marginal the desired one yields the true label predictive distribution:
( .r = ((v))) q(v)
&
plul=)
(8)
g(x)
p(u)
where the model label marginal p(y) is estimated via Equation (7). The data label marginal q(v) is hard to estimate since we only have a few training examples in our setting. Previous works (Zhao et al., 2021; Min et al., 2021; Han et al., 2022; Fei et al., 2023) typically assume ((y) to be uniform, yielding the following classifier:
Experiments
5.1 Setups
Datasets
We use 12 text classification tasks in our experi-
ments: SST2 and SST5 (Socher et al., 2013), CR
p(y
(Hu and Liu, 2004), MR (Pang and Lee, 2005), SUBJ (Pang and Lee, 2004), AGNews (Zhang
(9)
p(y)
We follow this convention and leave accurate esti- mation of data label marginal q(y) in future 8
EM algorithm in Sacrens et al. (2002). Both methods involve estimating expected statistics of p(a, y). which is done by generations similar to estimating p(y). However, we find that their estimates are extremely unstable in our case, causing severe performance drops even compared with vanilla ICI..
*oome methods to estimate the data lahel marginal include BBSE (Lipton et al., 2018; Azizzadeneshelii et al., 2019) or

---

al., 2015), DBPedia (Zhang et al., 2015), TREC (Voorhees and Tice, 2000), CB (De Marneffe et al., 2019), RTE (Dagan et al., 2006), QQP (DataCa- nary et al., 2017) and SNLI (Bowman et al., 2015). For datasets whose testing set size is greater than 2000, we sub-sample 2000 examples for evaluation. The prompt template and example generations of each dataset are shown in Appendix A and B, re- spectively.
domly sample the original training of the dataset to construct the training examples. LLMs scaling less than 30B are evaluated in runs, while those larger than 30B are evaluated in 2 runs using dif- ferent random seeds. This finally yields 1944 runs for each method, which the results should be solid, The performance is measured by macro-F1. Im- plementation details are shown in Appendix C. We also present time complexity analysis in Appendix D.
Language Models
We investigate 12 LLMs in wide range of scales,
5.2
Main Results
including wransformer-base models Garg-Large (774M), GPT2-XL (1.5B) (Radford et al., 2019), GPT-NEO (2.7B) (Black et al., 2021), GPT-J (6B)
We plot the average 4-shot performance across dif- ferent datasets in different runs in Figure 3. For the and 8-shot results please refer to Appendix F. We find that our proposed GC effective in the following aspects:
(Wang, 2021; Wang and Komatsuzaki, 2021), GPT-
NEOX (20B) (Andonian et al., 2021; Black et al., 2022), OPT (13B and 30B) (Zhang et al., 2022). LLaMA (13B and 33B) (Touvron et al.. 2023) and recently proposed RNN-based models RWKV (3B, 7B, and 14B) (Peng al., 2023).
1) The proposed GC significantly improves the ICL performance, by up to 27% absolute (2-shot case of GPT2-Large (774M)). The improvement is con- sistency for LLMs in all parameter scales. Notably, our approach enables small models to outperform larger models' vanilla ICL performances. For ex- ample, GC lifts GPT2-XL (1.5B) to 0.64 in macro- F1, while OPT's (30B) ICL performance is only 0.57, despite being over 20 times larger. Also, note that the improvement is more obvious for smaller models, which is useful in the limited resources scenario.
Compared Methods
Besides vanilla ICL, we include the following state- of-the-art ICL calibration methods for comparison: 1) Noisy channel (NC) (Min et al., 2021) changes the slot position of the input and output in the template, and then uses the label likelihood p(zly) = PLM(.9 ( (x)) (v)) for pre- diction, where Φ (D,) denotes the concatena- tion of flipped demonstrations. Since p(zly) x p(zlu)/p(t /p()), this method actually has the same form ours in Equation (9), so we categorize it as calibration method.
2) The proposed GC outperforms all the calibration baselines, by up to 7% absolute (4-shot case of LLaMA (33B)), suggesting that our method is not only theoretically guaranteed to be superior (being an unbiased estimate of p(y)), but also empirically verified to be better.
Contextual calibration (CC) (Zhao et al., 2021)
estimates the label marginal via context-free texts. 3) Domain-context calibration (DC) (Fei et al., 2023) proposes a further requirement for the context-free texts: they must be also context-free in the task domain. They construct such domain context-free texts by randomly sampling and con- catenating words from the task dataset.
Analysis
Following the finding that GC generally outper- forms ICL and other calibration methods, we con- duct further analysis in this section.
6.1 Robustness
4) Prototypical calibration (PC) (Han et al., 2022) learns a Gaussian mixture model (GMM) from the output probability vectors. They then consider each cluster corresponds uniquely to a label, where the learned cluster weights are the estimated la- bel marginal. For fair comparison, we learn the GMM on the set of generative sequences the same as GC.
Since we've 've shown in Section 3 that p(xly) is a good and stable approximation of g(rly) no mat- ter of the prompt configurations, if the estimated p(y) is solid, then GC is expected to be also robust towards changes of the prompt. This is verified empirically, as shown in Figure 4 that depicts the 4-shot performance distribution of 10 randomly sampled training sets and 10 random order arrange- ments of one particular training set for GPT2-XI.
We consider 2, 4, and 8-shot true sew-shot learn- ing settings. For evaluating each method, we ran-

---

53
.75
75
(1401
GPI NEOK 22440
OFF 11:80
OFT 3340
LLAMA (1360
LLaMA (306)
Figure 3: Average 4-shot performances on 12 datasets, where the best result of each LLM is colored green. We also conduct Hotelling's s I-square test (Hotelling, 1992) to show that the improvement of GC is statistically significant in the level significance 0.01, as shown in Appendix
Mazro- F1 of GPT2XL (1.9B) wrt differntt domonstration choloes
8-
MecroFy
.2
method
ICL
SS12
8875
CR
MR
TREC
AGNews
DBPedia
RTE
GB
SNLI
QCP
CC
DC
MacnnF
8-
S812
SSTS
MR
Subtj
TREC
ACNews
DBPedie
RTE
SNLI
QGP
Macroel11 GPT2XL (1.38) wit diferent demonstration orders
Figure 4: Sensitivity results for GPT2-XL (1.5B).
(L.5B) (Results of GPT-NEO (2.7B). GPT-J (6B), and LLaMA (13B) be found in Appendix H). It is obvious that the proposed GC greatly and consis- tently reduces the performance variance for almost all datasets. For instance, GC reduces the choice standard deviation from 0.25 to 0.03 for GPT2-XL (1.5B) on AGNews, while CC only achieves 0.10.
dominated
Balanced
dominated
Macro-F1
To further show that GC does address the major- ity and recency bias (Zhao et al., 2021) introduced by the choice and order of training examples, we conduct 4-shot experiments with prompts that have different class balances and orders in SST2 dataset,
mPpp
Nagg
Phup
PPap
M007
Norgon
configuration
Figure 5: 4-shot results of GPTT-XL (1.5B) on SST-2 using prompts with different configurations.
where each prompt configuration is evaluated in three random runs. The results of GPT-XL (1.5B) are shown in Figure 5. According Zhao al. (2021) and our analysis in Section 3.2, LLMs tend to predict labels that appear more frequently in the prompt and are closer to the testing input, which
is damageable the performance. For example, if all the training examples are positive, i.e., "PPPP" in Figure 5, L.LMs would pathogenically predict all the testing inputs as positive, leading to an ex- tremely low Fl performance. As shown, our pro-

---

posed GC greatly and consistently alleviates this issue across all the prompt configurations, where the increase is up to 54%.
7 Related Works
In-context learning, or few-shot prompting (Brown et al., 2020) has become one of the most curious emergent abilities (Wei et al., 2022a) of LLMs, which benefits in fast-prototyping and much less requirement of annotated data. ICL also forms the basis of its further extensions, including chain of thoughts (Co') (Wei et al.. 2022b; Wang et al., 2022; Lyu et al., 2023), in-context instruction learn- ing (ICIL) (Ye et al., 2023), meta-ICL (Coda-Forno et al., 2023) and so on.
6.2 Comparison with Prompt Optimization Methods
In addition to calibration methods, we also compare our approach against prompt optimization methods for order and choice of training examples, where 4-shot results for all LLMs are shown in Figure (2 and 8-shot results are shown in Appendix I).
There are many works empirically analyzing in-context learning, which generally find that in- context learning is pathogenically unstable towards the template, choice, and order of the training ex- amples (Gao al., 2020; Zhao al., 2021; Lu et al., 2021). This motivates numerous works trying to find a better prompt in terms of template (Min et al., 2021), choice (Rubin et al., 2021; Su et al., 2022: Wang et al., 2023; Iter et al., 2023; An et al..
6.2.1 Order
We compare GC with LocalE and GlobalE (Lu et al., 2021), which aim to find the best order per- mutation of the training examples in the true few- shot setting based on heuristic scores. We only reproduce the 2 and 4-shot results because these methods would rate every order permutation, where the complexity is OKK) and larger shot is compu- tational prohibitive. As shown in Figure 6, our pro- posed GC exceeds both order optimization methods by considerable margins, while being much more efficient with complexity (L)).
2023), and order of the training examples (Lu et al.,
2021; Wu et al., 2022). However, most of them ei- ther violate the true few-shot learning (Perez et al., 2021), or complicated learning- based (Wang ct al., 2023), or both (Rubin al., 2021), which deviates from the original intention of ICL. On the other hand, except for a few works that fine-tune LLMs to learn in-context (Chen et al., 2021; Coda- Forno et al., 2023), most LLMs are pretrained on a large amount of raw text. So it is not reasonable to believe that LLMs faithfully express the label distri- bution of given task. To address this, other lines of work try calibrate the in-context classifier by estimating and then countering this bias introduced from the prompt (Zhao et al., 2021; Yang et al., 2023; Han et al., 2022). These methods are much more lightweight compared to prompt optimization methods but fail in their heuristic bias estimations.
6.2.2 Choice
We compare GC with KATE (Liu et al., 2021). which for each testing example, it retrieves the K-th most similar training examples in the whole training set to construct the prompt. Note that this method violates the true-few shot learning setting by accessing a large training set containing much more examples than that of ours?. We include two implementations that use BM25 and Roberta- Large fine-tuned text similarity and natural language inference tasks (Reimers and Gurevych, 2019) for retrieval, denoted as KATE-BM25 and KATE-ROBERTa. As shown Figure 6, though with only a few training examples, the proposed GC is usually competitive with choice optimiza- tion methods. For example, 4-shot performance gaps between GC and KATE-BM25 don't 't exceed 0.03 for GPT2-XL (1.5B), OPT (13B), OPT (30B), LLaMA (13B) and LLaMA (33B).
Theoretical understanding is also attractive researchers. For example, Dai al. (2022); von Oswald et al. (2022) find surprising similarities between ICL and gradient descent formally and empirically, establishing explanations via implicit gradient descent. Han et al. (2023) provide an explanation of ICL under the view of kernel regres- sion. Xic et al. (2021) demonstrate ICL implicit Bayesian inference, where it occurs when the LLM infers shared latent topic of demonstrations in the prompt. While their theory is appealing, they assume the pre- trained data distribution as a mix- ture of hidden Markov models (HMMs), and con-
In general, GC outperforms is competitive with strong prompt optimization methods, thus sav- ing the effort of cumbersome prompt engineering.
"For example, SNLI has 56k training examples in the training set, which is over 10k times more than our case.

---

Pareoge moor F1
GPTZ4erge (77
OPTpas. (1.00)
OPT-65 (L70)
G974 (89)
RAKY (35)
RNKY (15)
RENSN (14))
GPT HETH (200)
OPT (138)
DPT (S00)
U.AMA (138)
UAMMA (350)
Figure 6: Average 4 shot performance comparison results with prompt optimization methods.
duct experiments on a small-scale synthetic dataset, which is far different from the real case.
doesn't affect main claims of this paper. We leave this point for future works.
This work is also limited by using AUROC to measure the closeness of p(al y) and g(rly). Note that AUROC is high is just necessary condition to say p(r|y) y) and ((<l)) are close, but not the suf- ficient condition. Therefore, when AURCO is high, can't 't rigorously say the p(æly) and ((>r) v) are close. To the matter of fact, for our method to be ef- fective, we don't 't need the model label conditional p(zly) to be close to the true one, we only need the in-context model to perform well in inter-label ranking: it can put most of the positive examples in the front, and most of the negative samples in the back (in binary classification setting). If so, all we need is to set proper decision threshold (an 1D decision boundary). which is actually what our calibration method do: if the model has shift on the positive label, i.e., p(P) > p(N), the pro- posed GC actually increases the decision threshold from the default 1 to p(P)
8 Conclusion
In this work, we for the first time theoretically and empirically identify that the in-context model mainly poses a label shift to the data distribu- tion. We then propose generative calibration, lightweight solution to mitigate this distribution shift. Exhaustive experimental results on 12 tasks and 12 LLMs demonstrate the effectiveness of our approach, in its improvement and stability.
Acknowledgement
This work was supported by the National Key R&D Program of China (2022ZD0100503) and the National Natural Science Foundation of China (No.62276264), This research was also supported by Meituan.
P(N)* i.e., predict P when P p(N) and predict otherwise. Choosing this decision boundary can ensure that the model
Limitations
p(Pr) > p(P).
The main limitation of this work is to assume the data label marginal qvy) to be uniform. Though to the best of our knowledge there is no work re- searching this problem and previous works also fol- low this assumption, this assumption is not true in general realistic cases containing much label imbal- ance scenarios. However, estimating (v)) in true few-shot learning setting is challenging since we only have few training samples and an unlabeled testing set in hand. We' ve actually experimented with BBSE (Lipton et al., 2018) and EM algorithm in (Saerens et al., 2002), which naturally fit our idea because they require samples from the model distribution. However, the estimate is just plausible in few runs accidentally, in general we fail to obtain consistent solid estimates, especially for the dataset which the number of labels large such as DBPedia. Although this limitation is important,
label marginal is uniform but not biased to any spe- cific labels. So a high AUROC is actually enough for our method to work.
References
David Aldous, Illar A Tbragimov, Jean Jacod, and David Aldous. 1985. Exchangeability and related topics. Springer.
Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, and Jian-Guang Lou. 2023. Skill-based few-shot selection for in-context learning. arXiv preprint arXiv:2305. 23..22305.....00
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler. Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. 2021. GPT-NeoX:

---

Large Scale Autoregressive Language Modeling in PyTorch.
Marie- CarieeCatherine De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124.
Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. 2019. Regularized learn- ing for domain adaptation under label shifts, arXiv preprint arXiv: 1903.09734.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm. int8 (): 8-bit matrix mul- tiplication for transformers at scale. arXiv preprint 2rXiv:2208.077 07339.
Sid Black, Stella Biderman, Eric Hallahan, Quentin An- thony, Leo Gao, Laurence Golding, Horace He, Con- nor Leahy. Kyle McDonell, Jason Phang. Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT--NeXX-00B source autoregressive language model. In Proceed- ings of the ACI. Workshop on Challenges & Perspec- rives in Creating Large Language Models.
Charles Elkan. 2001. The foundations of cost-sensitive learning. In International joint conference on ar- rificial intelligence, volume 17, pages 973-978. Lawrence Erlbaum Associates Ltd.
open-
Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. Mitigating label biases for in-context learning. arXiv preprint arXiv:2305. 1914488..........
Sid Black, Leo Gao, Phil Wang. Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow. you use this software, please cite using these metadata.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre- pre-trained language models better few-shot learners. arXiv preprint arXiv20012.15723.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.0533
Chi Han, Ziqi Wang, Han Zhao, and Heng Ji. 2023. In-context learning of large language mod- els explained as kernel regression. arXiv preprint arXiv:2305.1270
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Zhixiong Han, Yaru Hao, Li Dong. Yutao Sun, and Furu Wei. 2022. Prototypical calibration for few- shot learning of language models. arXiv preprint 22Xiv2205..0183.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competi- tion: Why the highest probability answer isn't 'tt always right. arXiv preprint arXiv:2104.08315
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. 2018. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106:249-25
Harold Hotelling, 1992. The generalization of Student's ss ratio. Springer.
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis. and He He. 2021. Meta-learning via language model in-context tuning. arXiv preprint arXiv: 210.0781,
Minqing Hu and Bing Liu. 2004. Mining and summa- rizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 168-177
Julian Coda-Forno, Marcel Binz, Zeynep Akata, Matthew Botvinick, Jane X Wang, and Eric Schulz. 2023. Meta-in-context learning in large language models. arXiv preprint 233iv:2305.J 12907.
Dan Iter, Reid Pryzant, Ruochen Xu, Shuohang Wang. Yang Liu, Yichong Xu, and Chenguang Zhu. 2023. In-context demonstration selection with cross entropy difference. arXiv preprint arXiv:2305. 1405.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment chal- lenge. In Machine Learning Challenges. Evaluat- ing Predictive Uncertainry, Visual Object Classifi- cation, and Recognising Tectual Entailment: First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers, pages 177-190. Springer.
Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, Kang Min Yoo, and Taeuk Kim. 2022. Ground-truth labels matter: A deeper look into input-label demon- strations. arXiv preprint 220.
Damai Dai, Yutao Sun. Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10530
Siwon Kim, Jihun Yi, Eunji Kim, and Sungroh Yoon. 2020. Interpretation of nlp models through input marginalization. arXiv preprint arXiv:2010. 13984.
Charles X Ling and Victor S Sheng. 2008. Cost- sensitive learning and the class imbalance problem. Encyclopedia of machine learning, 201 2011:2311 1:221-233.
DataCanary, hilfialkaff, Lili Jiang, Meg Risdal, Nikhil Dandekar, and tomtung. 2017. Quora question pairs.

---

Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. 2018. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pages 3122-3130. PMLR.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, 1lya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):
Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.07206.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint Skii:2104.087550
Nils Reimers and Iryna Gurevych. 2019. Sentence- bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Qing Lyu, Shreya Havaldar, Adam Stein, Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of- thought reasoning. arXiv preprimt arXiv:2301. 13379.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning, arXiv preprint arXiv:2112.08633
Marco Saerens, Patrice Latinne, and Christine De- caestecker. 2002. Adjusting the outputs of a classi- fier to priori probabilities: simple procedure. Neural computation, 14(1):21-41.
Elliot Meyerson, Mark Nelson, Herbie Bradley, Arash Moradi, Amy K Hoover, and Joel Lehman. 2023. Language model crossover: Variation through few- shot prompting, arXiv preprint ar32v:
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. 2012. On causal and anticausal learning. arXiv preprint arXiv:1206.6477
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2021. Noisy channel language model prompting for few-shot text classification. arXiv preprini arXiv:2108.041000
Lorraine Schwartz. 1965. On bayes procedures. Zeitschrifi für Wahrscheinlichkeissheerr und ver- wandte Gebiete, 4(1):10-26.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi. and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in-context learning work? arXiv preprint arXhv:2202. 22.222222..2877 12837.
Chenglei Si, Dan Friedman, Nitish Joshi. Shi Feng. Danqi Chen, and He He. 2023, Measuring induc- tive biases of in-context learning with underspecified demonstrations. arXiv preprint arXiv:2305.1332 132999
Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. 2023. What in-context learning" learns" in-context: Disentangling task recognition and task learning. arXiv preprint 205
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631-1642.
Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. arXiv preprint cs/0409058.
Bo Pang and Lillian Lec. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv preprint cs/0506075,
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang. Jiayi Xin, Rui Zhang, Mari Ostendorf. Luke Zettlemoyer, Noah A Smith, et al. 2022. Selec- tive annotation makes language models better few- shot learners. arXiv preprint arXiv:2209.01975.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Al- balak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, al. 2023, Rwkv: Reinventing mns for the trans- former era. arXiv preprint arXiv:2305. /3048.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 11377.2302.13971.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. Ad- vances in neural information processing systems, 34:1 11000
Johannes von Oswald, Eyvind Niklasson, Ettore Ran- dazzo, João Sacramento, Alexander Mordvintsev, An- drey Zhmoginov, and Max Vladymyrov. 2022. Trans- formers learn in-context by gradient descent. arXiv preprint 22X
Jonathan K Pritchard, Matthew Stephens, and Peter Don- nelly. 2000. Inference population structure using multilocus genotype data. Genetics, 155(2):945-959.

---

Ellen Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR confer- ence Research and development in information retrieval, pages 200-207.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066
Ben Wang. 2021. Mesh- Transformer-Ja Model- Parallel Implementation of Transformer Lan- guage Model with JAX. https://github..ce kingoflolz/mesh-trrae transmmrrsttttasfffrrriiiia
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances neural information processing systems, 28.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In ternational Conference on Machine Learning, pages 12697-12706. PMLR.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A 6 Billion Parameter Autoregressive Lan- guage Model. https: /ttg///ttttttt/eee//////eteeee llll mesh-transformer jax.
Xinyi Wang, Wanrong Zhu, and William Yang Wang. 2023. Large language models are implicitly topic models: Explaining and finding good demon- strations for in-context learning. arXiv preprint urXiv:2311.11916
Zhi-Hua Zhou. 2021. Machine learning. Springer Na- ture.
A Prompt Template
Table shows prompt templates used for different datasets.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Fdd Chi, and Denny Zhou. 2022. Self-consistency im- proves chain of thought reasoning in language mod- els. arXiv preprint 2rX1v.2203.11171.
B Example Generations
Table 2 shows example generations using GPT2- XL (1,5B) of different datasets. Note that the se- quences are generated by vanilla sampling, there- fore they might not be coherent sentence. But they are reasonable to estimate the in-context label marginal.
Jason Wei. Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. 2022a. Emergent abilities of large language models. arXiv preprint 223.v:220060768882
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022b. Chain thought prompting elicits reasoning large language models. arXiv preprint arXiv:2201.1 1903.
C Implementation Details
We use LLM.int80) (Dettmers et al., 2022) quanti- zation for all LLMs to reduce the memory usage" This techniques allows to use LLMs in few consumer GPUs. Specifically, use two servers with 8 NVIDIA GeForce RTX 2080Ti (1 (IIB mem- ory) and 10 NVIDIA GeForce RTX 3090 (24GB memory) in the experiments, where device usages of different LLMs are shown Tabel 3. We cache the prompt hidden states to improve the inference speed since the ICL prompt remains unchanged for every testing example. Note that RNN-based mod- els RWKV requires much less memory compared to Transformer-basee models in the same scale. This is because the cached hidden states of RWKV are only of length 1, while that of Transformer- based models equal to the prompt length.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang. Denny Zhou, et al. 2023. Larger language models do in-context learning differently. arXiv preprint wwX.ixx2333038...
Zhiyong WU, Yaoxiang Wang, Jiacheng Ye, and Ling- peng Kong, 2022. Self-adaptive in-context learning. arXiv preprint arXiv:2212.10375.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learn- ing implicit bayesian inference. arXiv preprint arXiv:2111.02080.
Sohee Yang, Jonghyeon Kim, Joel Jang, Seonghyeon Ye, Hyunji Lec, and Minjoon Sco. 2023. Im- proving probability- prompt selection through unified evaluation and analysis. arXiv preprint arXiv:2305.1487.
In the generation process, the prompt is the ICL demonstration prompt 2 plus the input slot name, e.g., "Review: <.>lnnentiment: <y>innnRevieww."
Seonghycon Ye, Hyeonbin Hwang, Sohee Yang, Hyeongu Yun, Yireun Kim, and Minjoon Sco. 2023. In-context instruction learning. arXiv preprint arXiv:2302. 14691.
"uggingFace's transformers library supports this fea- ture, see https:/ htt.//hhugggaaacccc wttaaacccc/ccooooeeeeeeeeeeeeooe co/docs/ fornsrs v0.30. d//n/perte serf_infer_ggu_one spu_one.

---

Dataset
Prompt Template
Label Words
SST2
Review: <e=nnmmimmm <y>
negative, positive
SSTS
Review: <a> lSSentimnn: <y>
terrible, bad, neutral. good, great
CR
Review: <o> InSetimmen: <y>
negative, positive
MR
Review: <a>>nSentiment: <y>
negative, positive
Subj
Input: <x>inType:: <y>
objective, subjective
TREC
Question: <u>inAnswer Type: <y>
abbreviation, entity, description, person, loca- tion, number
AGNews
Input: <x>/alyype: <y>
world, sports, business, technology
DBPedia
Input: <r>lnType: <yp
company, school, artist, athlcte, polities, trans- portation, building, nature, village, animal, plant, alhum, film, book
RTE
Premise: <x siHooothhsssss <x srddinProiiction: <y>
true, false
Premise:
<r' sinHypothesis: <x PlnPrrecccioo::
<t>
true, false, neither
SNLI
Premise:
< >llllyoohheii: <x >>pPrediction:
<g>
entailment, neutral, contradiction
QQP
Questionl: <x
>laQuestion2:: < PluPrediccion:
<y>
false, true
Table Prompt templates and label words of different datasets, where x and denote the input sequence and output label, respectively, Each demonstration split by double line breaks "Inkn".
for SST2, Given that each slot is split by line break as shown in Table 1, when the generation en- counters line break, we force the next contiguous tokens to be the next slot phrase of the template. The maximum generated length is set to 384.
performances on 12 datasets. We also show 2, 4, and 8-shot full results (averaged by random runs) of different calibration methods on different datasets in Table 5, 6, and 7.
D Time Complexity
G Hotelling's T-square test
In deployment, if we have N testing examples, the method includes generations and ICL infer- ences, which the time complexity is O(L + N). When N > L, the additional time cost of the gen- eration can be neglected. Time complexities of different methods are shown in Table 4. As seen, our method is superior on par to previous meth- ods.
We show that the improvement of GC to the other methods statistically significant. Concretely, the complete method performance is represented by a matrix with shape M x 12, where each row con- tains results of 12 datasets in random run (each run has different training examples randomly sam- ple from the dataset training set). We then use Hotelling's t-square test (Hotelling, 1992) to test whether the 12-dimensional performance vectors of two methods have the same mean, where the null hypothesis is that their means are the same. To compute the test static, the covariance matrix samples should be invertible, requiring M > 12, while in our main experiments, we only run each method in or random runs. Since running each method of each LLM more than 12 times is pro- hibitively time-consuming, here we only consider the 4-shot of GPT2-XL (1.5B), where set M 50. As shown in Table 8, GC signifi- cant different from other methods given that the p-values are lower than 0.01.
E Additional Results of Label Shift Empirical Validation
As a supplement to Section 3.2, Figure and display the macro-F1/AUROC performances and estimated in-context label marginals across differ- ent prompt configurations SST2 of GPT-NEO (2.7B) and GPT-J (6B), respectively. The results show similar trends with that of GPT2-XL (1.5B), detailed in 3.2.
F Full Results of Main Experiments
We provide full results of main experiments sec- tion 5. Figure shows the average 2 and 8-shot

---

Dataset
Generated data
SST2
Review: everyone else should watch this; don't be surprised stingy, it might be your last one.
Review: any playthrough starts with of 2-3up, best games worth the coin!
SST5
Review: a grim. disgusting, thoroughly pleasant reflection on mankind.
Review: on first viewing it feels like the actors are auditioning to star in a gag reel.
CR
Review: It a wonderful tool and i would use without warry. one moment to the next.
Review: good photo put to all took about 4 hrs
MR
Review: 's hard to freeze an action sequence.
Review: he has thus managed to reconcile his proven wavelike power with the sheer inevitable tre future friction frustration of this endless writing process.
Subj
Input: can you imagine ever audition meeting anyone else the world who can bear your burden, miss kumin
surely.
Input: for the weeks, when my 100 BOP brother and have been US, we went to trade mart everyday every day... This is kind of effect it has!.
TREC
Question: What the official reason given for Ben Franklin's death?
Question: What city is named for San Francisco's Golden Gate Bridge?
AGNews
Input: 446 billion times faster Than Neptune 1 June 2018 26:53m The counter-clockwise nucleus of Europa hinted at at experiences better than those quenched by hyperplaces. or ancient enigmasmas Input: BBC hrowsers golden age loses its shine The golden age of BBC browsers looks set to date back to 1996, with both IF Microsoft and FI team Lotus falling prey penalty points..
DBPedia
Input: Rush Limbaugh is an American radio host, radio talk show host, and raconteur who hosts, produces and produces and hosts. He moved to SiriusXM in 2006. He currently hosts the nationally syndicated "Rush I. Limbaugh Show" on SiriusXM which programs airs on weekday aftermoons at 2:05 p.m. castern time. Input: Mushy Plant is a photo type of forming pur fastaccous or lichen, with many, many major species by the name of Petrosphaeria. It is the most common compound fungus known, with about 35,000 species believed to exist, all either spore forming, or forming mushrooms.
RTE
The most successful dam-builders are those that find out how to mitigate accidents quickly and avoid frivolous Hoothhsisss It is expensive to know when you are going into an accident; not knowing where it happens is just as costly.
Premise:
Premise: Partner-long-tee
international agreements should be recogmized as the best way to reduce trade barriers between developed, emerging, and underdeveloped nations..nHyyothessi. International trade can be reduced through tariff's there.
CB
Premise: In the work room, he sent the monitor onto the floor. "Safe", lettering in green ink on the monitor rectangle. It scemed empty, aside from possihly one of Tara's remains or..heyottessis: in yellow ink the monitor was flooded covered in bloody print
Premise: She scrunched up her face up to her shoulder, buried the note in her grief, and then found walk up the stairs. hnyphhhssss:: Tara couldn't walk up the stairs. She lay down in the sunshine on a sunny day, antiseptic mustache hair once again surrounding her blushing checks, and then made another toothy, hom dismissive sniff and.
SNLI
Premise: A woman sitting on her couch. HnHyothhesi::: a real family from all around the country. Premise: A beautiful lady is waiting for customers in an area of a grocery store. Hyyoothsii:: The customers don't exist exist.
QQP
How can calculate next company carnings by splitting carnings by homonyminquusstion: How do use computer algebra a second language? Question1:
Questionl:
How should learn how to code i don't 't know Java, C++, PHP. Java Script?unQuestion2: How should I make mobile app? an Android app from without write single line of Java?
Table 2: Example generated sequences of GPT2-XL (1.5B) when prompting with 4 demonstrations.

---

P dominnated i
Balanced
N dominated
dominated
Balanced
N dominated
Macro MiMacro AUROC
.75
Ply
.25
M1cro-F1 Mecro-AUROC
Pppp
Nppp
Pupp
PPPp
PPPN
Nhep
NPPp
PAMP
NPPN
PAPN
PPHN
NAND
NNPN
NPWN
PWEN
PPPp
hppp PHPP
PPRp
PPPN
Namp
NPW
PhMP NPPN
PAPP PPIN nevi NNNN NPPn PVNN ANRRN
Prompt configuration
Prompt configuration
(a) Macro-FI/AUROC performances.
(b) Estimated in-context label marginals.
Figure 7: Macro Macro-FF/AUROO performances (a) and estimated in in-context label marginals (b) of GPT-NEO (2.7B)
across different prompt configurations on SST2.
d.omiintted
Balanced
dominated
dominated
Balanced
N dominated
Macro MMaoro AUROC
.75
aly P)
ply
Macro-F11 Mecro-AURO
pppp
NPPPp
PhPp
PPNP
PPPN
Attop
NPPp
NPPN
PAPW
PPHN
MAno
MAPH
NPW
PMW
PPPP
NPPP
PhPP
PPPP
PPPN
NPPo
NPPW
PNPW
PPHN
nonoo
NAP
NPW
PAW
NAW
Prompt configuration
Prompt configuration
(a) Macro-F1/AURRO performances.
(b) Estimated in-context label marginals.
Figure 8: Macro-FI/AUROC performances (a) and estimated in-context label marginals (b) GPT-J (6B) across
different prompt configurations SST2.
H Additional Results of Robustness Analysis
Figure 10 and 11 show sensitivity results of GPT- NEO (2.7B) and GPT-J (6B) to further support Section 6.1.
Model
Device
GPrg-Large (774M)
2080Ti
GPT2-XL. (1.5B)
2080Ti
I Additional Results of Comparison with Prompt Optimization Methods
GPT-NEO (2.7B)
2080Ti
GPT-J (6B)
2080Ti
RWKV (3B)
2080Ti
RWKY (7B)
X 2080Ti
Figure 12 shows 2 and 8-shot comparison results of the proposed GC and prompt optimization methods further support section 6.2.
RWKV (14B)
2080Ti
GPT-NEOX (20B)
3090
OPT (13B)
x 3090
OPT (30B)
3090
LLLMA (138)
x 3090
Effeec of the Number of Generations
LLaMA (33B)
3 x 3090
We study the effect of the number of generations. i.e., L in Equation (7). As shown in Figure 13, the number of generations increases, the model per- formance rapidly increases and converges. Also, the performance has been improved considerably when the number of generations is small, which
Table 3: Device usages of LLMs in our experiments.

---

Method
Complexity
Remark
ICL
O(N)
NC
O(
| size of label space
PC
O(L + N)
L: number of generations
CC
O(L + N)
L: number of context-free inputs
DC
O(L (IL+N) N)
L: number of domain context-free inputs
LocalE/GlobalE
O(KL++N)
L: number of generations per-order permutation
KATE
O(M + +N + +M)
M: size of the whole training set
GC (Ours)
OLL +N)
L: number of generations
Table Time complexities different methods.
benefits in the limited computational resource sce-
nario.

---

Model
Method
SST5
MR
TREC
AGINew
DBPedia
CB
SNLI
QQP
Avg
GPT2-Larg arge
41.74
46.05
20.96
20.09
774M
NC
K3.8
36.83
79.00
80.77
64.23
22.92
54.79
64.14
52.42
24.93
32.84
35.67
52.7
80.07
40.39
CC
56.41
19.7
62.95
66.94
59.49
46.15
56,01
71.81
44.58
19.17
20.53
45.29
47.42
DC
75.1
30.46
75.79
77.36
57.9
42,84
52.88
70.11
47.51
26.55
20.89
41.69
51.59
GC
85.64
33,98
85.87
82.67
60.5
45.5
60.39
75.77
47.22
28.65
29.52
45.78
56.79
GPT2-XL
KCL
53.75
9.3
49.7 49.79
55.91
44.78
25.75
32,97
45.26
46.7
20.75
24.39
40.8
$7,51
1.5
NC
81.24
36.66
76.48
77.17
65.62
18.88
52.87
62.93
52.76
24.87
33,44
42.09
52.08
86.5
15.36
877.29
81.02
55.75
20.88
40.22
1.66
48.48
36.33
24.71
39.0
44.77
CC
73.73
31.22
66.2
74.14
64.34
40.51
66.3
77.91
47.28
32.32
23.44
48.27
53.81
85.82
29:97
87.1
83.1
46.18
72.23
77.19
29.76
19.67
45.2
57.42
GC
84,88
34,22
84,58
83.18
66.97
48.2
74.22
80.02
49.65
43.45
34.04
47.92
60.94
OPT-NEO
KCL
61.81
14.12
78.02
70.02
39.56
36.14
59.94
63.15
40.25
24.48
19.52
42.44
45,79
80.15
78.02
28.76
58.65
38.33
PC
87.97
19.6
84.3
83.57
58.85
31.29
55.9
6.19
47.26
32.33
21.88.
44.74
47.82
31
48.29
53.4
56.83
DC
89.42
34.69
89.02
86.67
61.87
52.84
80.45
78.73
47.76
34,09
20.77
98
60.27
GC
88.5
36.12
85,76
86.7
68.03
53,46
80.09
78.18
48.91
38.5
31.5
48.74
62,04
GPT-J
81.12
18:97
81,45
75.33
61.74
37.55
55,06
67.68
50.23
33.04
17.66
36.42
$1,35
68
NC
87.2M
38 64
83.22
85 (9)
67.33
31.69
61.97
69.65
55.93
37.22
42.48
51.55
59.33
80,06
30.46
86.9
89.94
54.35
30.2
52,16
4.52
52.9
28.19
18.07
46.12
47,82
CC
89.19
41.76
89.37
89.07
67.6
55.73
73,02
88.83
39.43
27.96
24.63
46.41
61.08
DC
93.47
43.95
89.4
90.22
68.82
61,25
74.97
88.73
51.1
26.59
27.0
49.03
63,71
90.52
61.37
34,95
65.23
RWKY
ICL
64.6
16.82
71.78
37.22
23.54
50.27
45.09
89
20.8
20.77
35.37
42.57
3B
NC
40.16
32.38
44.42
74.74
$6.94
30.74
42,99
$5.08
36.54
30.41
32.87
44.04
43.44
44.45
CC
83.78
30,96
89.07
83.07
59.94
38.27
70.42
77.69
50.51
24.55
30.96
43.1
56.86
76.596
GC
92.07
33.73
89.91
88. 49
67.35
41.7
76.51
77.69
52.78
24.46
31.97
45.14
60.15
RWKV
ICL
64.68
17.54
69.1
79.96
39,22
32,59
22.47
63.98
43.81
31.81
18.85
39.77
43.65
32.47
74.32
61.21
30
30.78
27.11
45,29
89.01
13.76
84,25
73.0
32.91
15.28
30.4
5.85
36.7
31.98
19.36
40.1
39.38
CC
86.64
34.38
84.13
87.1
46.55
48.79
53,12
86,18
45.67
28.91
34.9
45.09
56.79
84,66
35,
53.62
29.45
GC
88.56
37.2
83.27
86.13
66.77
52.26
77.6
83.39
50.97
34.62
31.45
49.89
61.84
RWKY
ICL
77.51
14.47
187.02
85.26
35,46
39,06
28.34
51,01
47.77
24.01
21.04
35.4
45.53
14B
45.94
35.93
54,93
81.4 .4.49
60.07
32.33
49.47
62.63
38.44
27.77
35.67
49.8
47.87
PC
87.91
10.22
90.05
86.45
34.96
26.09
31.27
5.13
56.85
22.95
18.72
43.69
42,86
35.31
85.
DC
91 42
34.23
91.54
87 88
40.22
57.57
65.77
77.44
50.74
36.48
31 11.44
48.72
59.04
GC
88.17
36.08
90.79
85.05
41.62
5
73.8
81.62
60.01
34.55
39.9
42.39
60.76
GPT-NEOX 208
NC
87.62
37.
$2.09
84.93
73.15
34.7
66.84
71.077
56.76
40.21
44.93
49
60.75
92.95
26.72
91.01
90.74
64.97
32.42
70.11
7.08
58.6
33.98
18.5
49.68
53.06
CC
95.0
42.08
91.94
90.89
61.72
56.25
81.68
90.2
41.06
41.77
35.22
49.61
64,79
DC
94.87
41.04
91.7
91.59
62,43
58.06
83,95
88.58
49.6
35,22
32.09
$2.4
65.13
99.96
91.27
58.24
84.49
52.08
67.67
OPT
ICL
85.49
18.69
84.06
S0.35
42.02
38.5
54.85
73.62
39.53
32.82
16.97
44.32
50.93
13B
NC
87.75
40.79
83,71
85.74
67.81
40.1
65.46
76.26
54.52
37.07
45.73
51.2
61.34
93.24
91.72
41.53
49.72
CC
94.34
35.21
90.83
91.2
64,28
52.95
$1.23
88.17
$3.69
44.06
26.19
44.29
63.87
89.4
34.99
62,51
GC 93.27
43.21
90.94
92.34
66.97
57.98
84.78
84.32
55.2
53,39
37.17
47.07
67.22
OPT
ICL
84.16
14.0
85.62
90.7
51.49
46.7
37.72
$1.46
55.43
24.18
16.9
39.02
$2.28
PC
92.81
23.98
91.1
87.97
48.72
38.05
71.66
8:26
45.5
22.35
29.06
36.42
49.4
CC
91.8
42.97
93.06
91.31
71.31
53.65
69.95
93.6
48.88
18.97
29.72
46.07
62.61
DC
94.81
44.93
93.21
91 65
67.25
62.05
74.17
92.55
56.68
27.37
18.05
49.36
64.34
GC
93.53
44.19
90.77
89.11
72.31
60.44
77.76
83.01
61.66
31.61
44.09
48.96
66.46
LLSMA 138
ICL
95.85
18.91
91.0
91.43
49.37
58.86
77.12
64.5
64.18
33.71
24.91
36.52
58.86
86.72
38. 71
83,39
85.60
66.81
47.91
72.13
73.32
56.62
34.61
49.28
62.44
PC
93.13
29.85
88.95
91.57
$7.02
50.31
71.61
3.76
74.21
50.53
28.0
40.54
56.62
96.25
70.05
DC
95.79
45.15
88.32
92.27
63.32
72.05
$1.79
82.33
73.15
48.94
26.84
40.73
67.22
44.73
77.45
76.33
86.83
60.65
73.55
L.LMMA
95.35
15.
86.59
77.12
30.85
63.97
33B
NC
86.02
36.73
83.89
86.76
69.72
52.72
73.72
74.19
62.83
25.54
55.58
61.73
64.12
87.01
30.12
90.97
55.97
48.31
76.15
2.54
52.8
18.96
$3.81
$6.71
CC
96.05
34.05
91.0
44
80.69
74.6
88.11
92.91
66.66
52.9
29.88
52.37
70.64
94.33
37.
89.97
91.92 92
55.46
82.07
88.48
87.73
64.84
59.29
32.7
66.4
70.89
90.25
73.72
91.46
66.92
59.39
74.06
Table 2-shot full results.

---

Model
Method
S5T5
MR
TREC
AGiNew
DBPedia
CB
SNLI
Avg
GPT2-Larg arge
43.53
30.83
22.12
36.35
774M
NC
83.86
36,
79.83
81.8
68.51
27.48
53.97
70.17
52.81
28.23
31.69
42.73
54,82
72.
29.95
41.16
CC
55.75
20.33
65.22
49.55 55
45.9
45.
60.29
73.02
39.35
31.59
17.99
52.48
46.23
DC
79.54
32.41
85,17
78.71
60.84
49.58
65.3
70.84
51.43
35.99
17.67
50.14
56.47
GC
85.84
37.13
86.22
80.96
70.34
51.16
72.2
77.13
50.21
38.44
26.29
$1.8
60.64
GPT2-XL
KCL
48.2
11:75
48.71
41.88
47.55
38, 36
$2.42
66.55
48,9
35.52
19.71
41.85
41.76
1.5
NC
80.94
33.8
77.98
79.69
66.99
22.1
52.13
61,88
53.48
28.64
33.51
46.0
53.1
81.89
13.71
82.448
81.97
59.34
31.28
55.52
2.77
43.59
39.32
23.4
45.95
46.77
CC
62.4
28.94
64.45
60.36
59,96
45.47
75.52
83.61
41.56
43.49
26.94
54,94
53.97
85,2
$3.08
71.96
75.72
23.09
54.91
GC
87.16
36.31
82,96
$1.86
74.32
56.92
77.69
83.18
48.35
50.47
34,28
55.08
64,05
OPT-NEO
KCL
71.23
11.28
81,83
75.93
42,47
42,08
66.44
79.09
47.55
30.69
19.75
50.09
51,54
80,89
77.6
78.26
70.37
31.41
37.61
55.74
PC
67,62
21.54
87.53
87.19
41,99
35,92
66.53
4.46
45.23
30.93
24.23
50.41
46.97
33.
14.27
DC
90.72
36.03
88.61
85.59
66.12
57.7
80.98
84.12
50.18
39.27
25.24
52.97
63.13
GC
91.22
39.15
87.41
87.2
68.26
58.22
81.81
86,02
51.23
39.82
34.32
54,01
64,89
GPT-J
87.6
25.32
83.2
67.39
46
58.18
85.01
47.41
32.2
24.15
46.22
57.67
68
NC
87
38. 12
85.85
86 26
67.46
38.64
64.35
72.8
53.66
43.91
44.06
55.98
61.38
92.75
33.18
88,79
89.22
54.36
39.91
68.37
5.93
50.09
34,39
16.99
38.62
51.05
CC
84.61
43.44
86.63
86.16
73.36
59.78
73.32
92.43
39.49
25. 86
26.54
50.47
61.84
DC
93:73
44.23
89.41
90.04
73.39
63.27
78.66
91.56
46.05
37.07
27.81
50.67
65.49
37.81
67.23
RWKY
ICL
73.17
21.06
86.1
61.11
48.05
33.31
65.26
43.87
23,26
19.67
32.51
46.19
3B
NC
40.41
31,73
42.7
75.14
$3.17
31.17
43.68
$7.89
36.77
32.14
32,36
46.
43.61
90.98
17.29
31.53
47.83
CC
79.01
24.05
85.84
81.54
56.43
45.62
66.52
82.06
46.53
27.57
33.88
39:45
55.71
27.57
GC
92.09
39.41
89.9
87.93
65.12
48.98
79.25
82.18
54.79
32.17
35.14
48.65
62.97
RWKV
ICL
78.95
16.9
79.92
80.43
38.05
32,84
33.27
77.76
45.77
35,82
19.23
41.61
48.38
29,
73
33.14
54.55
45.41
$7.16
13.69
84.88
83.33
32.61
20.02
48.33
6.28
34.59
36.84
19.29
38.86
42.16
CC
90.37
34.85
84.73
86.81
50.82
51.3
59.9
87.49
48.5
35.07
31.08
41.79
58.56
86.63
28.
80.95
86.07
31,99
47.2
GC
89.72
39. 13
84.27
86.83
64.57
53.78
79.31
$4.03
50.36
37.73
34.42
48.92
62.76
RWKY
ICL
85,33
20.37
900.88
79.75
43,24
39.46
33.87
69,96
48.61
29.02
21,11
32.22
49.42
14B
45,9
34.75
52.67
77.97
58.75
42.27
54.31
68.93
37.86
27.48
37.21
47.78
48.82
PC
87.91
19.26
91.54
85.04
33.34
21,86
34,27
3.1
59.05
32.74
19.11
39.13
43,86
91.34
37.41
89.1
DC
91.0
38.9
91.61
83 28
44.49
58.00
65:53
80.81
56.02
47.36
27,41
51
61.22
GC
89.66
41.56
90.68
88,
47.5
55.5
76.9
84.44
60.41
46.75
35.88
42.83
63.39
GPT-NEOX
208
NC
88.09
38.
82.36
84.30
78.62
39.05
68.55
72.86
55.24
35.31
44.77
52.54
61.67
95.06
32.66
89,42
90.52
$7.75
37.52
71.6
5.96
60.24
34.52
24.42
44.72
53.7
CC
95.9
37.89
91.89
91.18
63.82
62.03
85.41
92.25
46.94
42.12
38.31
49.38
66.43
DC
95.58
36.6
92.18
91.39
65.77
61,26
84.03
87.62
61.82
41.87
35.15
54.12
67.28
95.52
45.
OPT
ICL
85.21
24.55
88.57
88.76
76.43
42.09
57.79
80.41
46.01
38.7
18.09
45.36
57.66
13B
NC
87.86
41.15
82,003
85.5
69.45
43.73
65.98
77.84
39.47
39.8
45.68
55.47
62,83
93.85
28.07
90:97
29.76
49.82
CC 95.56
42.85
91.29
91.82
66.85
60.36
$1.9
91.36
$3.61
$1.89
22.79
48.69
66.58
64.57
35.38
26.32
53.12
65,16
GC 95.37
44.38
90.69
90.5
78.91
58.8
85.07
90.04
57.34
55.7
40.06
49.06
69.66
OPT
ICL
82.84
26.
91.94
89.76 76
55.97
55.03
47.85
77.88
$1.47
37.3
17:47
49.79
$6.97
62.74
PC
92.61
96. 12
91.93
87 55
66.
40.42
73.63
0.92
48.11
38.91
17:74
47.78
$3.49
CC
93.87
93.87
37.11
90.52
92.26 26
77.68
62.89
78.99
93.1
34.94
36.41
24.87
41.84
63.71
DC
95.16
40.44
92.33
92.55
83.54
62.23
79.24
91.7
56.41
43.51
21.33
41.54
66.65
GC
94.77
44.09
91.98
90.69
87.77
71.21
86.44
94.17
61.98
42.28
41.48
51.64
71.54
LLSMA 138
ICL 95.56
29.36
91.62
89.95
72.86
62.82
80.2
80,92
73.19
51.7
36.48
52.89
68.13
89.03
38,
45
73.5
50.76
74,01
73.32
61.87
39.82
52.9
60.35
65.24
PC
95.27
43.5
92.08
89.95
85.04
54.91
$0.58
10.22
63.76
49.35
40.34
39.46
62.04
96.69
72.5
DC
96.32
40.46
90.38
91.83
82.73
73.66
82.45
87.16
75.95
$2.15
44.76
$3.5
72.61
90.91
76.39
87.81
77.08
LLLMMA
92.72
70.77
72.21
33B
NC
87.43
40.29
85..48
87.41
74.9
$7.47
73.449
73.19
61.89
27.96
51.61
65.98
65.59
95.38
32.22
92.44
$4.69
85.94
1.89
74.43
$4.91
35.32
65.0
64.29
CC
95.88
39.05
91.1
88.
64.5
76.61
87.62
95.43
73.47
58.99
33.09
58.79
71.9
DC
95.46
35.7
91.03
92.1
66.25
80.18
87.73
87.62
62.18
55.5
50.37
33.14
69.77
95.08
91.63
80.34
85.91
69.48
72.86
78.7
Table 6: 4-shot full results.

---

Model
Method
SST5
MR
TREC
AGiNww
DBPedia
CB
SNLI
QQP
Avg
GPT2-Larg arge
17.65
55.94
61.34
57.21
24.47
42.06
774M
NC
84.41
38.63
80.7
81.38
67.75
26.04
52.38
72.04
52.43
27.91
31.81
40. 18
54,64
76.21
25.54
37.45
39.8
CC
67. 38
30.91
69.13
65.35
53.12
48.09
63.87
77.59
39.5
36.57
17.71
50.18
51.77
DC
85.55
30.62
84,55
79.73
70.1
53.08
70.26
76.25
50.63
38,04
18.31
45.76
58.57
QC
88.17
38.02
87.96
82:23
78.92
50.5
74.33
79.28
53.04
39.95
26.88
49:43
62.39
GPT2-XL
KCL
43.01
13:45
45.35
48 88
52.1
42.02
58.16
73.28
44.13
34,022
18.29
41.72
42.87
1.5
NC
82.35
35.48
75.19
79.97
68.16
20.91
51.34
65.39
53.5
33.48
33.78
43.72
53.61
82.34
18.05
84.15
69.5
45.51
30.42
61.51
2.4
44.81
35,28
23.74
45.38
45.26
CC
64.15
24.09
59.95
64.16
60.04
45.77
73,06
86,62
38.62
28.42
29.18
46.4
51,7
86.5
22.55
76.52
50.05
77.18
85.9
35.37
60.29
GC
86.13
33.14
84.54
84.42
78.83
52.84
79,53
89.57
51.43
43.2
36.85
51.48
64.33
OPT-NEO 2.78
KCL
81.98
17.53
188.43
75.32
44.95
47.24
74.25
83.89
53.58
32.26
21.86
46.03
55.61
80.31
35.81
77.23
71.87
29.34
70.68
28.38
39.66
55.81
PC
79.03
22,
87.59
73.27
48.54
29.28
67.96
6.45
53.48
27.93
21.73
46.37
46.98
83.6
DC
92.58
31.69
88.92
85.94
70.16
57.62
81.91
85.43
50.48
42.11
29.36
45.64
63.49
GC
92.57
38.68
$6.48
86.73
71.77
60.24
81.15
87.23
53.2
44.2
35.53
51.19
65,58
GPT-J
94.73
37.21
89.66
90.61
70.03
$4.93
66,09
85.8
40.83
43.6
61.06
68
NC
87.66 66
39. 48
144
86 89
71.05
38 18.96
64.64
75.73
57.93
43.52
43,46
50.24
61.99
93.96
38.75
90.19
90.55
70.17
46.12
77,46
13.49
49.07
46.1
16.76
42.34
56.25
CC
89.77
42.05
39.81
88.97
77.61
62.08
71.63
92.42
37.5
40.68
28.25
52.32
64,42
DC
95.12
43.23
90.4
90.89
79.08
66.09
77.19
91.09
42.53
46,13
27.74
50.43
66.71
51.62
RWKY
ICL
65.27
29:03
77.75
66.02
38.22
39.36
47.53
67.71
43.55
25.78
21.51
35.91
46.47
3B
NC
39.07
32.38
42.04
71.35
54.28
30.96
43.18
56,76
36.07
36.46
32,33
45.95
43.42
26.79
24.36
45.41
CC
70.52
27.0
87.24
79.11
53.63
50.23
68.59
78.25
38.87
31.23
30.3
40.64
$4.63
30.82
GC
91.11
38.91
89.09
87.81
60.56
50.44
80.62
81.46
54.32
34.7
34.15
45.44
62.38
RWKV
ICL
80.85
19.71
$1.83
82.7
42.76
38.36
47.54
76.51
49.21
27.0
17.54
41.48
50.46
42.52
74.41
25,1
$4.45
37.62
27.91
54.9
45.34
PC
80.48
12.05
72.77
89.89
$2.59
16.89
48.91
1.19
48.14
38.13
17.81
38.84
39.81
CC
88.82
36.41
81,23
84.04
48.97
50.23
77.25
85,62
48.43
30.96
34.31
46.92
58.85
84.77
22.97
GC
89.77
41.59
84.52
85.88
63.46
55.74
81.83
85.26
52:25
35.
31,16
48.43
62,92
RWKY
ICL
91.21
23.13
89.8
89.34
42,12
42.85
43.75
73,67
48.83
27.65
26.57
40..13
53.25
14B
44.54
36.
52.43
80.95
59.16
41.25
50.59
69.18
38.02
31.8
34.58
45.25
48.67
PC
$1.0
20.78
89.38
90.52
38.39
33,51
36.69
9.02
49.9
30.36
17:54
41.79
44,66
92.82
91.39
DC
89.03
40.25 24
91 49
84.47
44,82
60.13
64.7
84.5%
54.27
39.
25.42
42.71
59.93
GC
91.09
43.41
90.15
90.08
46.0
61.42
76.75
86.59
60.76
47.6
30.01
44.88
64.06
GPT-NEOX
95.89
61.16
201
NC
89.07
28.2
82.77
86-21
79.04
41.87
68.43
75.05
$4.79
39.0
46.89
50:59
62,66
94.87
38.71
90.62
90.41
75.57
42,28
55,77
21.52
56.5
47.56
17.44
39.9
55.93
CC
95.8
39.33
92.07
91.08
59.16
68.95
84.1
90.2
43.57
41.51
34.3
$2.32
66.03
DC
95.82
36.05
92.27
91.21
62.51
65,73
81.9
88.14
63.86
49.36
34.96
55.8
68.13
91.34
67.9
57.07
72.33
OPT
ICL
91.05
36.31
91.14
89.02
77.0
47.33
65.97
81.71
45.79
40.31
17.55
42.36
60.46
13B
NC
88.78
40.08
82.31
86 69
72,58
42.91
66.57
79.51
58.57
41.25
47.37
52.12
63.23
95.87
37.9
90.97
34.9
37.99
53.19
CC
95.84
43.98
91.69
92.61
67.1
56.33
$1.62
91.05
$6.86
47.28
24.43
48.05
66.4
80,75
40,
65.67
GC 94.77
45.34
90.03
91.31
85.28
57.4
84.82
90.1
60.06
62.84
38.02
49.05
70.75
OPT
ICL
92.46
38,
90.29
89.74
81.52
49.25
72.26
82.05
$6.15
40.34
19.44
46.78
63.26
41.11
64.38
PC
95.22
37. 77.74
87.86
91. 68
$1.18
44.4
59.33
12.86
48.69
39.43
23.94
48.77
$5.92
CC
95.55
95.58
43.08
91.91
91.87
78.36
63.85
80.82
90.23
35.98
41.24
27.07
35.05
64,59
DC
96.54
45.0
92.77
92.07 07
79.38
63.92
76.69
91.36
60.96
30.29
29.68
66.4
GC
96.32
48.65
91.88
91.99 '8
90.01
66.05
84.14
91.46
64.76
$5.62
39.53
51.63
72.67
LLSMA 138
ICL
96.72
39.47
92.14
92.41
66.6
71.38
79.98
83.92
77.84
52.51
48.22
45.33
70.54
80.93
41.89
84.45
78.47
52.9
75.07
74.42
39.47
54.34
53.58
65.95
PC
95.39
41. 64
91.3
92.72
86.32
64,22
81.21
27.3
72.6
51.63
39.07
38.84
65.19
96.55
92.90
77.78
72.55
DC
95.98
39.35
90.76
92.93
67.95
78.54
83.63
83.3
76.63
$6,33
43.29
$5.39
72.01
86.98
65.5
58.73
77.29
LLLMAA
93.62
66.54
53.44
72.6
33B
NC
89.37
41.04
84.65
88.45
79.49
58.45
75.55
75.39
$7.88
34.09
56.93
64.69
67.17
96.79
50.11
92.78
93.2
52.52
27.03
76.54
45.87
66.62
56.04
68.89
CC
96.54 -
36.72
92.68
92.34
68.07
81.57
85.38
93.01
72.54
60.43
49.1
73.36
75.14
DC
96-92
39.45 47
92.28
93.22 29
49.73
83.74
84.44
83.51
66.18
69.53
59.33
55.66 66
72.76
96.68
50.55
95.92
73.39
$0.56
Table 7: 8-shot full results.

---

GPT228L
12.781
(PT1
(38)
1C
MC
53
52
55
PC
45
4B
63
./5
-75
.75
.75
46
51
59
64.
4n
41
61
5C
DC
.88
47
24
74
23
25
25
75
(145)
(205)
(208)
ULINA (338)
(a) 2-shot results.
GPT2- (7744))
GITIZOLL (2.38)
OITIEO 22/81
C2T5 198)
FWWY 1380
BAPSI (28)
1CL
42
.41
.66
NC
BC
57
60
62
66
25
-75
-75
PC
0C
14
45
55
DC
68
66
23
25
25
775
75
75
RNC/ (148)
KBOX (208)
OFT (138)
LLAMA (1381
LLAMA (538)
(b) 8-shot results.
Figure 9: Average and 8-shot performances 12 datasets.
Method
ICL
CC
DC
P-value
4.32 10
7,47 10
3.80
10
1.02 10 23
Table 8: P-values of Hotelling's t-square test by comparing GC and other methods.

---

(2.7B) wit different domonstration
Macro-F1
method
ICL
SST2
SSTS
CR
MR
Sukj
TREC
AGNews
DBPedia
RTE
C8
SNLI
QGP
PC
CC
DC
Macro-11
SST2
SSTS
CR
MR
Subj
TREC
AGNews
DBPedia
RTE
CB
SNLI
QGP
Micro-FF1 GPT-NEO demonstration
(a) Sensitivity results for GPT-NEO (2.7B).
Macro different demonatration
Eqpa
2
method
ICL
SST2
SST5
CR
MR
Subj
TREC
AGNews
DBPedia
RTE
CB
SNLI
QGP
PC
CC
Mac0oF1
2
SST2
SST5
CR
MR
Subj
TREC
AGNews
DBPedia
RTE
CB
SNLI
QQP
Macro-F1 of GPT- (8B) different demonsttraton orders
(b) Sensitivity results for GPT-J (6B).
Macro-F1 of LLAMA (13B) wit different damonsiration choices
22
method
ICL
sub
CB
PC
CC
DC
GC
22
DBPcdia
CB
Macro-F1 LLaMA (138) wit different demonstration orders
(c) Sensitivity results for LLaMA (13B).
Figure 10: Sensitivity results of GPT-NEO (2.7B). GPT- (6B), and LLaMA (13B).

---

Pdommnated
Balanced
dominatedd
P dominaaedd
Balanced
N domiinaed
Hbcro F1
3
PP9p
Plop
PPip
PPW
Woop
Plage
PPPP
Nrpp
Phapo
P9PU
Mape
configuration
configuration
(a) 4-shot performances GPT-NEO (2.7B) using prompts (b) 4-shot performances different configurations.
GPT-J (6B) using prompts with
with different configurations.
Figure 11: s-ohot performances of GPT-NEO (2.7B) and GPT-J (6B) using prompts with different configurations.
Locat
Global
KATE-BM25
KATE-RUBERTA
cPrizaanga (7716)) GP12-LL
OPTAMER
GP7-4
RUMA
GPT-MEOX
OFT
(LOAA
LLaMA
(a) Average 2-shot performance comparison results with prompt optimization methods.
RuBERTl
Gorceange (T76M) OPTA (1-58)
GPTMBS 3.788
FOWNY (145)
GPY-NEOX (208)
(138)
(308)
LLOAE (138)
(LAMAN
(b) Average 8-shot performance comparison results with prompt optimization methods.
Figure 12: Average 2 and 8-shot performance comparison results with prompt optimization methods.
.8
.8
5ST2 macro-F)
SUB) macro-F1
66
6
GPT2-XL (1.58)
(2.78)
(68)
.2
100
100
Number of generations (L)
Number of generations (L)
Figure 13: Effeect of the number of generations to GC on SST2 and SUBJ, where I. = 0 indicates ICI. performance.


Reviews:
[{"review_id": "RDWPdoYfTo", "reviewer": "Reviewer_HMv7", "summary": "", "questions": "Q1) What is the relationship between the topic prior p(θ) in LDA and that used in LLM? It is still confusing. Q2) What is the definition of the topic? (the Multinomial distribution on words?)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer HMv7 for the valuable feedback. Now we will reply one by one according to the points mentioned by the reviewers.\n\n**Reasons to Reject#1:** The key finding is only justified by empirical evidences without theoretical proofs.\n\n- **We strongly disagree to take this as a reason to reject** for the following reasons:\n  - We actually have theoretical analysis in Section 3.1, which we think is clear and enough for illustration. Our empirical evidence is also sufficient in Section 3.2, which Reviewer HMv7 has also pointed out as a reason to accept.\n  - \"The in-context label marginal $p(y)$ generally deviates from the true one $q(y)$\" is NOT an assumption, but a key finding in our paper. For empirical findings, we believe there is no mandatory requirement for theoretical proofs, especially given that recent popular works like CoT hardly have theoretical support but also greatly promote the development of NLP. Thus, we feel this criticism is a bit unfair.\n  - We think this criticism falls into the quick heuristics \"The paper doesn't use [my preferred methodology], e.g., deep learning\" listed in the review policy of EMNLP2023. NLP is an an interdisciplinary field and relies on many kinds of contributions including data/model analysis and theory. There should be no distinction between different types of contributions, as long as it is enough for illustration.\n  - The goal of the paper is to calibrate the ICL prediction, not theoretical proofs of why the shift happens.\n\n**Reasons to Reject#2:** Equations (1) and (2) are not very readable.\n- **We strongly disagree to take this as the reason to reject.** Equation (1) and (2) require those notations for mathematical rigor and self-contained, and it's also readable enough given the background. Perhaps there is a more compact way for expression and we will revise this, this can't be so strong to be a reason to reject.\n\n**Reasons to Reject#3:** The topic prior $p(\\theta)$ is not well discussed.\n- Topic refers to the task as shown in footnote 2, and we just follow the convention to call it \"topic\". The topic prior $p(\\theta)$ is the marginal distribution of the topic in the pretrained corpus. We can simply take it as the topic's occurrence frequency in the pretrained corpus since the latter is it's MLE estimation. **While topic prior $p(\\theta)$ is detailedly discussed in the previous work in ICL's Bayesian interpretation like [1], we don't talk about it too much but only discuss its effect to ICL in line 175-182. We think this discussion is proper and enough. We apologize for the non-clearness and will add more details about the Bayesian interpretation in the appendix.**\n\n**Question#1:** Relationship between the topic prior p(θ) in LDA and that used in LLM\n- To answer this question, we first clarify the the relation and difference between LDA and LLMs:\n  - Relation: Both LDA and LLMs are to model the text distribution $p(x)$.\n  - Difference: With different goals, LDA and LLMs have completely different assumptions and practices.\n    - LDA aims to cluster the words in the document, then it has a latent topic assumption: a document is generated by first sampling topic $\\theta$ from topic prior $p(\\theta)$, and then sampling a document from $p(x|\\theta)$. Then, the text distribution $p(x)$ is obtained via marginalization: $p(x)=\\int p(x|\\theta)p(\\theta)d\\theta$. In LDA, $\\theta$ is a Dirichlet variable lies in a simplex $\\Delta^{P-1}$, where $P$ is the number of the predefined word clusters (There are other latent variables in LDA, but we only talk about the topmost one for compactness). Hence, each element in $\\theta$ represents the probability that the word in the corresponding cluster appears in the document. Note that this assumption greatly simplifies dependencies in text (for example, it considers the document as a bag-of-word). However, the goal is not generation but to use the posterior $p(\\theta|x)$ for clustering. Thus, this assumption is reasonable.\n    - LLMs aim to generate fluent and coherent sentences, then they model the text distribution $p(x)$ directly. In specific, LLMs consider $p(x)$ to be a cascade of Categorical distributions, which can capture any dependencies in text in theory.\n\n  So how do we connect LDA and LLMs? The logic is that though LLMs directly model the text distribution $p(x)$, the community generally consider the text, i.e., the pretrained corpus, is structured and arranged by some latent variables controling the generation of text like the assumption in LDA. Therefore, if a LLM $p(x)$ approaches the true text distribution exactly (which is a widely used assumption in theoretical analysis and first proposed in [1]), $p(x)$ can be decomposed into the marginalization of these latent variables. \n\n  Here, the key problem is that we don't know what exactly the latent variables as well as the generative structure look like. Previous works have different but also self-contained assumptions such as mixture of HMMs [1], constituency tree [2], or even embedding [3]. Basically, we can call all of these latent variables as \"topic\", since they abstract the content of text and then play a role of clustering.\n\n  **In our work, we just assume the topic existence but not the form: we presume there is a topic $\\theta$ existing in the pretrained corpus such that $p(x,y|\\theta)$ can model the distribution of examples in the target task, and we doesn't assume the specific mathematical form of the topic. This assumption is widely used in previous works and also enough for our analysis.** So we apologize for the  confusion and we will add more clarifications to make the paper be more clear.\n\n**Question#2:** What is the definition of the topic?\n\n- According to reply to Question#1, topic is only a latent variable abstracting the task and we don't care about the specific form. However, the assumption of multinomial distribution on words as Reviewer HMv7 mentioned is non-proper, since it's a bag-of-word assumption but clearly the task example has dependencies between input sequences and output label tokens.\n\n**Clarifications**\n\n- We note that Reviewer HMv7 has much concerns about the relation between topic models and LLMs. While the connection is detailedly discussed in the previous works [1][2][3], our paper only takes it as a tool for theoretical analysis, which is for scientific rigor and not the main contribution of our works. For deeper understanding on this topic, Reviewer HMv7 can refer to those previous works. Because of this and considering the contribution of our work, we sincerely think the paper's soundness is a bit underrated and hope Reviewer HMv7 can raise the score.\n\n**References**\n\n1. Xie S M, Raghunathan A, Liang P, et al. An explanation of in-context learning as implicit bayesian inference[J]. arXiv preprint arXiv:2111.02080, 2021.\n\n2. Hahn M, Goyal N. A theory of emergent in-context learning as implicit structure induction[J]. arXiv preprint arXiv:2303.07971, 2023.\n\n3. Wang X, Zhu W, Wang W Y. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning[J]. arXiv preprint arXiv:2301.11916, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is a calibration method which is lightweight for reducing the the instability. \n2. It provides a new viewpoint from the Bayesian interpretation perspective of ICL. \n3. The paper explicitly validated the soundness of the previous assumption, i.e., the ICL predictive distribution has a shift on label marginal. \n4. Exhaustive experiments on 12 text classification tasks and 12 LLMs scaling from 774M to 33B are provided.", "reasons_to_reject": "1. The key assumption that \"the in-context label marginal p(y) generally deviates from the true one q(y)\" is only justified by empirical evidences in Section 3.2, without theoretical proofs.\n2. The Equations (1) and (2) are not very readble. \n3. The topic prior p(θ) is not well discussed for interpreting the meaning for LLM.", "questions_to_authors": "Q1) What is the relationship between the topic prior p(θ) in LDA and that used in LLM? It is still confusing. Q2) What is the definition of the topic? (the Multinomial distribution on words?)", "paper_topic_and_main_contributions": "The authors proposed a novel calibration based approach for the in-context classifier p(y|x) by simply adjusting the label marginal p(y), which can be obtained by marginalizing out the input x via Monte-Carlo sampling. The results show that the proposed approach greatly and consistently improves the baselines for different LLM scales with stability.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mu5i0X0yue", "reviewer": "Reviewer_Vz7F", "summary": "", "questions": "1. More details about the implementation of “multiple generation”.\n2. What is the time cost compared to ICL, with different sequence lengths and the number of generations (as shown in Figure 12, ICL is L=0) \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer Vz7F for the valuable feedback. Now we will reply one by one according to the points mentioned by the reviewers.\n\n**Reasons to Reject#1:** The proposed calibration method is not well explained.\n\n- All we have to do compared to regular ICL is to estimate an additional label marginal distribution $p(y)$, which is nartually estimated by the Monte-Carlo sampling:\n$$\np(y)=\\sum_{x}p(x)p(y|x)\\simeq\\frac{1}{L}\\sum_{i=1}^lp(y|x^l)\n$$\nwhere $l$ is the $l$-th sample from $p(x)$ and $L$ is the number of samples. Note that according to Equation (2), $p(x)$ is the continuation distribution given a prompt containing a few training examples. Therefore, the sample of $p(x)$ is naturally obtained by generating (sampling) sentences conditioned on that prompt. \n\n  **We sincerely thank for Reviewer Vz7F for pointing out that it is better to elaborate our method more detailedly, like presenting a figure, but we don't consider this issue to be strong enough as a reason to reject, because:** \n\n  - In the level of algorithm, the motivation and process of \"multiple generation\" should be VERY clear: Monte-Carlo sampling and LLM generation are common and familiar to the community. Thus, we omit some details for compactness.\n  - In the level of implementation, we actually introduce the generation details in line 937-944 in appendix C, including the prompt and how we maintain the template of the generation. We also present some examples in Table 2. Reviewer Vz7F can check this for details.\n\n**Reasons to Reject#2:** Time complexity concerns.\n\n- **We believe that Reviewer Vz7F has a misunderstanding about the paper**. $L$ is NOT the generated sequence length, it's the number of generated sequences for estimating $p(y)$. According to line 941-944, the generation ends when the line break \"\\n\" is sampled or the generation length exceeds to 384 (The actual length is generally far smaller than 384). According to appendix I, the calibration results converge rapidly as $L$ increases. So in our experiments, we set $L$ to just 100, which we believe is small. In deployment, if we have $N$ query examples, the method includes $L$ generations and $N$ ICL inferences, which the time complexity is $O(L+N)$. When $N>>L$, the additional time cost of the generation can be neglected. Other calibration methods except NC have the same time complexity as ours. Time complexities of the different methods are as follows:\n\n  | Method         | Complexity  | Remark                                           |\n  | -------------- | ----------- | ------------------------------------------------ |\n  | ICL            | $O(N)$      |                                                  |\n  | NC             | $O(\\|Y\\|N)$   | $\\|Y\\|$: number of labels                          |\n  | PC             | $O(L+N)$    | $L$: number of generations                       |\n  | CC             | $O(L+N)$    | $L$: number of context-free inputs               |\n  | DC             | $O(L+N)$    | $L$: number of domain context-free inputs        |\n  | LocalE/GlobalE | $O(K!L+N)$  | $L$: number of generations per-order permutation |\n  | KATE           | $O(M+N+MN)$ | $M$: size of the whole training set            |\n  | GC (Ours)      | $O(L+N)$    | $L$: number of generations                       |\n\n  **Therefore, our approach is lightweight and has no complexity concern.** We think analyzing the time complexity is straightforward and easy, so we do not elaborate it in details in the paper. We will add this table in the appendix for reference.\n\n**Question#1 & Question#2:**\n\n- For question#1, please refer to appendix C and Reply to Reasons to Reject#1. For question#2, please refer to Reply to Reasons to Reject#2.\n\n**Clarification**\n\n- We believe appendix of the paper and our response should address the concerns of Reviewer Vz7F. **We also find that those concerns have nothing to do with the soundness of the paper. Given that Reviewer Vz7F thinks that our analysis is clear and also supported by experiments well and experimental results are inspiring, we sincerely think the paper's soundness is a bit underrated and hope Reviewer Vz7F can raise the score.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and the Bayesian-based analysis is clear and well-supported by the empirical results.\n2. The research topic is important to the area and experimental results are inspiring as existing models are quite sensitive to the selected samples and their orders.\n", "reasons_to_reject": "1. The proposed calibration method is not well explained. It would be better if more details are added about the “multiple-generation” solution.\n2. Concerns about the efficiency of the solution as the model need to generate multiple times (if I understand it correctly), especially when the L(sequence length) is larger as its complexity is to be O(L), as well as a large number of generation.\n", "questions_to_authors": "1. More details about the implementation of “multiple generation”.\n2. What is the time cost compared to ICL, with different sequence lengths and the number of generations (as shown in Figure 12, ICL is L=0) \n", "paper_topic_and_main_contributions": "\nThis paper proposes a label-shift calibration method for in-context learning after analysing the distribution shift theoretically and empirically. Especially, it first decomposes the posterior distribution of the generated sequence based on the assumption of the Bayesian process. Then, it derives the ratio of label condition distribution (positive/negative) as the odds of predicting as positive and further uses the AUROC to empirically show that the label distribution is almost unbiased. Next, it adopts the Monte-Carlo sampling methods to derive the label marginal distribution and shows the discrepancy between generated labels and that in the validation samples. Based on the observations, it proposes to use the multiple generation method to approximate the true label marginal distribution as the true posterior distribution is approximated. \n\nContributions:\nIt analysis the posterior distribution by decomposing it into label conditional distribution and label marginal distribution, which can be empirically calculated and used to verify the motivation.\nThe proposed method demonstrates improvement over the listed baselines under the paper settings.\n", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Eq(4), p(P) should be the denominator, and p(N) should be the numerator. "}, "extra_text": {}}, {"review_id": "RMZhjos2hO", "reviewer": "Reviewer_eRKA", "summary": "", "questions": "In equation (4), the second equation should be $\\frac{p(N) p(P \\mid x)}{p(P) p(N \\mid x)}$?\n\nOn certain datasets, such as AGNews and SNLI, I believe the original dataset contains sufficient data to train $q(y \\mid x)$ (say, finetune a BERT), $q(y)$ (simple statistical estimate would work), $q(x \\mid y)$ (finetune LM on the data). Authors could use these to establish how well $p(x \\mid y)$ estimates $q(x \\mid y)$.\n\nIf we believe $p(x \\mid y)$ is a good estimate, why can't we use $p(x \\mid y) q(y) \\approx p(x \\mid y)$ for classification?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer eRKA for the valuable feedback. Now we will reply one by one according to the points mentioned by the reviewers.\n\n**Reasons to Reject#1:** Non-rigor description of Schwartz's theorem.\n- **We don't deny the reviewer's view on the Schwartz's theorem, but we also don't claim that posterior estimation would be poor when the number of samples K is small, which Reviewer eRKA may have some misunderstandings on this point.** In line 171, we just say that LLMs can't accept much training examples due to the limited context length, so according to Schwartz's theorem we can't ENSURE that the posterior is good. But we don't say that posterior would or must be poor when K is small, where neither the literal nor the extended meaning of this part can be understood in this way. So if there are some writing issues causing the misunderstanding, we apologize for this and we would modify the description to make it more clear and easier to follow. \n\n**Reasons to Reject#2:** \n- First of all, we need to clarify that the goal of in-context learning is estimate the true data distribution $q(x,y)$ (which is agnostic to us and we only have a few samples from it, i.e., the training dataset) using in-context generative distribution $p(x,y)$. (which is already in hand by prompting LLMs with a few training examples). So what do you mean to estimate $p(x,y)$? We feel like it is a clerical error of Reviewer eRKA.\n\n  **We believe Reviewer eRKA doesn't follow or understand line 174-193. Basically, this part is to show how the few-shot learning setting could affect the posterior $p(\\theta|D_t)$, and then further affect the quality of $p(x,y|D_t)$ according to the Bayesian interpretation**. In specific, we propose two reasons of label-shift when the number of training examples $K$ is small, namely shifted prior and bad samples.\n\n  - Shifted prior (described in line 175-182): When $K$ is small, the prior $p(\\theta)$ (the prior actually could be viewed as the topic's occurrence frequency in the pretrained corpus, since the latter is its MLE estimation) would dominate the posterior. This is because according to Equation (3), few product terms of $p(x_i,y_i|\\theta)$ makes $p(\\theta)$ dominate the posterior, and when there are no product terms (zero-shot) the posterior equals to the prior. **In this case, once the prior has a bias on a specific label token, it would cause a shift on this label.** For example, we have tested that GPT2-XL predicts positive on 74.32% of SST2 evaluation examples in the zero-shot setting, suggesting it has a label-shift on prior. This is reasonable because the token \"positive\" seems to appear more that token \"negative\" in the pretrained corpus. Other evidences include common token bias [1] and prediction insensitivity [2] when the training labels are corrupted studied in the previous works.\n\n  - Bad samples (described in line 182-193): To see the second reason, let's consider an example of review sentiment classification (labels are positive (P) and negative (N)). Since pretrained corpus contains diverse data from different websites, we can imagine that there not only exists the true topic $\\theta^\\star$ (for example, IMDB reviews page) such that the corresponding label marginal $p(y|\\theta^\\star)$ is uniform, i.e., $p(y=P|\\theta^\\star)=p(y=N|\\theta^\\star)=0.5$, but also exists a completely positive label-imbalanced topic $\\theta'$ in the corpus (for example, IMDB reviews page filtered by ratings 5) such that $p(y=P|\\theta')=1$ and $p(y=N|\\theta')=0$. For any positive sample $x,P$, clearly $p(x,P|\\theta')>0$ and $p(x,P|\\theta^\\star)>0$, since both topics support positive reviews. **Once our few-shot training dataset happens to ONLY contains positive samples, the posterior mass on $\\theta'$, i.e., $p(\\theta'|D_t)$ can be non-zero or even large, which would amplify the marginal distribution of the positive label and cause a label shift.** This is also called majority bias [1] in previous studies.\n\n  **Therefore, line 174-193 is actually sound and coherent in logic and not \"suddenly jump to the conclusion\"**. We think the reason why reviewer eRKA has confusion on this part is that we omit some details and leave deep understanding in the cited works. We apologize for this and will make it more clear in the next version.\n\n**Reasons to Reject#3**: $p(x|y)$ showing high AUROC is not a sufficient condition for it to be close to $q(x|y)$.\n\n- We admit that high AUROC is a necessary but not sufficient condition to take $p(x|y)$ be close to $q(x|y)$. Actually, we don't say so and we've used very careful wordings in the paper: we say in-context label conditional is good but NOT close to $q(x|y)$, where good means the model performs well in inter-label ranking, i.e., has a high AUROC.\n\n  **To the matter of fact, for our method to be effective, we don't need the model label conditional $p(x|y)$ to be close to the true one, we only need the model to perform well in inter-label ranking:** it can put most of the positive examples in the front, and most of the negative samples in the back (in a binary classification setting). If so, all we need is to set a proper decision threshold (an 1D decision boundary), which is actually what our calibration method do: if the model has shift on the positive label ($p(P)>p(N)$), our proposed generative calibration actually increases the decision threshold from the default 1 to $\\frac{p(P)}{p(N)}$, i.e., predict $P$ when $\\frac{p(P|x)}{p(N|x)}>\\frac{p(P)}{p(N)}$ and predict $N$ otherwise. The merit of choosing this decision boundary is to ensure that the model label marginal is uniform but not biased to any specific labels. \n\n  Therefore, we use AUROC to measure the model's inter-label ranking performance to illustrate that $p(x|y)$ is good, which is enough for our proposed generative calibration to work. We apologize for the misleading description in the paper. Especially, we don't carefully explain what the word \"good\" means. We will revise this point in the next version.\n\n**Question#1:** Equation (4) is wrong?\n\n- No, Equation (4) is right. We can see it by simply using Bayesian formula:\n$$\n\\frac{p(x|P)}{p(x|N)}=\\frac{p(P)p(x|P)/p(x)}{p(N)p(x|N)/p(x)}=\\frac{p(P)p(x|P)}{p(N)p(x|N)}\n$$\n\n**Question#2:** How well $p(x|y)$ estimates $q(x|y)$?\n\n- As discussed in the reply to Reasons to Reject#3, we don't actually need the in-context label conditional $p(x|y)$ to be close to the true one, so in the paper we don't estimate an oracle $q(x|y)$ and compare two distributions.\n\n**Question#3:** Why don't use $p(x|y)q(y)$?\n- We actually have mentioned this point in line 332-340. The problem is that we don't know the label marginal distribution $q(y)$ of the true scenario, i.e., the testing data. therefore we follow previous works to take $q(y)$ to be uniform. This predictor is equivalent to $p(x|y)=p(y|x)/p(y)$, where $p(y|x)$ is the original in-context predictor and $p(y)$ is the estimated marginal (via generation).\n\n**Clarification**\n- We find that much of concerns of Reviewer eRKA come from misunderstandings, some of which might cased by our writining issues. **We will revise the paper to make it more clear. We also sincerely hope that Reviwer eRKA can revisit to related parts and increase the soundness score, especially given the richness of our experiments and the effectiveness of our method (outperforms SOTA method DC in large margins) in different settings.**\n\n**References**\n1. Zhao Z, Wallace E, Feng S, et al. Calibrate before use: Improving few-shot performance of language models[C]//International Conference on Machine Learning. PMLR, 2021: 12697-12706.\n2. Min S, Lyu X, Holtzman A, et al. Rethinking the role of demonstrations: What makes in-context learning work?[J]. arXiv preprint arXiv:2202.12837, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is simple and straightforward, therefore it is likely to be a robust finding, and also provides good opportunities for follow-up studies to build upon.\n\nNonetheless, this simple idea (that in-context learning suffers from domain shift; i.e. marginals are poorly estimated, while conditionals are well-estimated) does not seem to have been clearly stated and empirically tested. This strengthens our understanding of in-context learning.\n\nExperiments are fairly extensive; authors experiment with 12 text classification tasks, 12 LLMs, and three different few-shot settings. The empirical benefit of the proposed method seems to be quite consistent across all these settings.", "reasons_to_reject": "Some of the statements lack scientific rigor.\n\nIn line 171, authors claim that according to Schwartz's theorem, posterior estimation would be poor when the number of samples K is small; but I believe the theorem is only a sufficient condition for the posterior to be good, and doesn't preclude small-sample estimation to be poor?\n\nIn line 174-193 authors argue that the estimation of $p(x, y)$ is hard because $p(\\theta \\mid \\mathcal{D}_t)$ is hard to estimate (which is an assumption rather than a fact), but then suddenly jump to a conclusion that it will lead to label shift $p(y) \\neq q(y)$. I don't see the logic connecting the assumption to the conclusion.\n\n$p(x \\mid y)$ showing high AUROC is not a sufficient condition for it to be close to $q(x \\mid y)$. It can still be terribly ill-calibrated. There are potentially other ways of measuring the goodness of estimation (see Questions for Authors). At the least, authors should clearly state the limitation.", "questions_to_authors": "In equation (4), the second equation should be $\\frac{p(N) p(P \\mid x)}{p(P) p(N \\mid x)}$?\n\nOn certain datasets, such as AGNews and SNLI, I believe the original dataset contains sufficient data to train $q(y \\mid x)$ (say, finetune a BERT), $q(y)$ (simple statistical estimate would work), $q(x \\mid y)$ (finetune LM on the data). Authors could use these to establish how well $p(x \\mid y)$ estimates $q(x \\mid y)$.\n\nIf we believe $p(x \\mid y)$ is a good estimate, why can't we use $p(x \\mid y) q(y) \\approx p(x \\mid y)$ for classification?", "paper_topic_and_main_contributions": "Authors propose a method of calibrating in-context learning (ICL) classifications by estimating generation probabilities. Authors empirically show that $p(x|y)$ is effective at ranking examples in terms of AUROC, and thus its only limitation is the lack of calibration. Authors propose to estimate the marginal probability $p(y)$ by generating samples and using Monte-Carlo estimation. Authors compare against a good range of existing methods for calibration, and demonstrate that the proposed method outperforms prior calibration methods consistently across various settings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are agreed on the excitement, but are finding small errors and issues that need further clarification in the text. These should be cleared up in further drafts of this work.
Decision: Accept-Findings