Prototypical Calibration for Few-shot Learning of Language Models
Authors: Zhixiong Han
Yaru Hao
Li Dong
Yutao Sun
Furu Wei
Venue: ICLR 2023 poster (2023)

Abstract: In-context learning of GPT-like models has been recognized as fragile across different hand-crafted templates, and demonstration permutations. In this work, we propose prototypical calibration to adaptively learn a more robust decision boundary for zero- and few-shot classification, instead of greedy decoding. Concretely, our method first adopts Gaussian mixture distribution to estimate the prototypical clusters for all categories. Then we assign each cluster to the corresponding label by solving a weighted bipartite matching problem. Given an example, its prediction is calibrated by the likelihood of prototypical clusters. Experimental results show that prototypical calibration yields a substantial improvement on a diverse set of tasks. Extensive analysis across different scales also indicates that our method calibrates the decision boundary as expected, greatly improving the robustness of GPT to templates, permutations, and class imbalance.

Published as a conference paper at ICLR 2023
PROTOTYPICAL CALIBRATION FOR FEW-SHOT LEARNING OF LANGUAGE MODELS
Zhixiong Han; Yaru Hao, Li Dong, Yutao Sun; Furu Wei
Microsoft Research
(zh i ixan8,,, aunyuta.20001 (yaruhao, 11 dongl, fuwei @mi croso com
@mma i 1 com,
ABSTRACT
In-context learning of GPT-like models has been recognized as fragile across dif- ferent hand-crafted templates, and demonstration permutations. In this work, we propose prototypical calibration to adaptively learn more robust decision bound- ary for zero- and few-shot classification, instead of greedy decoding. Concretely, our method first adopts Gaussian mixture distribution to estimate the prototypical clusters for all categories. Then we assign each cluster to the corresponding label by solving weighted bipartite matching problem. Given an example, its prediction is calibrated by the likelihood of prototypical clusters. Experimental results show that prototypical calibration yields substantial improvement on a diverse tasks. Extensive analysis across different scales also indicates that our method calibrates the decision boundary as expected, greatly improving the robustness of GPT to templates, permutations, and class imbalance. The code will be released at https: /i thub. com/ htt.//ixxaaaaaaaa Proca.
INTRODUCTION
Large-scale language models (LMs) have shown strong generalization ability on a wide range of downstream tasks (Devlin et al.. 2018; Radford et al., 2019; Yang et al., 2019; Lewis et al., 2019; Brown et al., 2020; Dong et al., 2019; Bao et al., 2020). Fine-tuning has been the common strategy to transfer the extensive knowledge to downstream tasks for long time.
However, fine-tuning such large LMs suffers from the over- parameterization issue under few-shot settings. Brown et al. (2020) propose the concept of in-context learning with GPT, which enables LMs to quickly adapt new task by conditioning on hand-crafted prompts as shown in Figure 1. The prompts consist of task-specific templates and several input-label pairs (demonstrations), In-context learning is surprising as GPT can perform various tasks without any parameter updating.
has been noticed that the predictions of GPT conditioned on prompts tend to bias toward some specific answers and can be highly volatile across different templates, demonstrations, and their permutations (Lu et al., 2021; Jiang et al., 2020). Zhao et al. (2021) propose to calibrate the model prediction by the content-free output to mitigate this problem. Rubin et al. (2021) and Lu al. (2021) focus on the training examples retrieval and optimal ordering selection respectively to produce more performant prompts than random sampling. However, they did not explain why the in-context learning performance is fragile across different scenarios.
In this paper, we analyze the intrinsic reason for the instability of few-shot learning with GPT. We observe sigmificant distinctions among the prediction distributions of GPT under different prompts. As shown in Figure the conventional decision boundary of GPT (i.e., naively uses the output with the largest probability as the predicted label) often fails to discriminate the predictions. We argue that the predictions can be more discriminative when provided with a calibrated decision boundary.
Specifically, we term the model outputs of examples whose ground- are the same category as prototypical clusters and adopt Gaussian Mixture Model (GMM) to estimate the distributions of them for all categories. The decision boundaries of the prototypical clusters are adaptively learned,
Contrihution during intersship Microsoft Rescarch.

---

Published as a conference paper at ICLR 2023
P( Positive context)
Output
GPT
P( Negatiive comtext)
Inout
Negative compellingly
Demonstration
Demonstration
Teat Example
Figure 1 Example of few-shot learning with GPT.
Negativn label Postive
14
Negative Label
100
Postive Lobel
PsodA
Prompt1 Density
РHоCА
GPT
Prompt2 Demsity
10
GPT
PnnCa
SST-2 Accuracy(%))
90.8
91.0
70-
3-
6-
60
63.5
2-
4-
50-
1-
0
0-
2.0
-1.5
-
1.0
0.5
0.0
-0.6
0.4
0.2
0.0
30
log P Poottiic
log, P Poositive
Prompt1
Prompt2
Prediction distribution of GPT-2-XL XL under two different prompts for SST- Two distributions colored by blue and red represent model predictions for negative and positive ground-truth examples respectively. pusiiivee denotes the prediction probability of positive label. The orange dashed line represents the decision boundary commonly used by GPT (i.e., P = 0.5 for binary classification). The green dashed line represents the decision boundary of prototypical calibration (PROCA). Right: Performance comparison of GPT-2-XL and PROCA under the two prompts, which indicates that PROCA is effective because the calibrated decision boundary is more discriminative for classification.
Figure 2: Left and Middle:
which is called prototypical calibration (PROCA). Then the prototypical clusters are assigned to the corresponding labels through weighted bipartite matching. We also propose to improve estimations according to cluster-label assignment. Finally, the predictions of test examples are more precise owing the calibrated decision boundary (as shown in Figure 2).
Experimental results show that we achieve on average 13% absolute improvement for different sizes of GPT models across nine text classification datasets. We demonstrate that PROCA is effective across various templates and different demonstration permutations.
To summarize, our key contributions are follows:
We find that the decision boundary plays a critical role few-shot evaluation. Moreover, performant decision boundaries inconsistent across language models and prompts.
We propose prototypical calibration to adaptively learn better decision boundary for the few-shot classification of language models.
Experiments show that PRoCA achieves 13% absolute improvement over the conventional approach wide range of text classification tasks.
DECISION BOUNDARY OF FEW-SHOT LEARNING WITH GPT
A decision boundary refers an explicit prediction criterion in the output space for given classifica- tion problem. As shown in Figure 2, two dashed lines represent two different decision boundaries, which classify examples into negative and positive categories. In this section, explore the effect of the decision boundary on few-shot learning. We demonstrate that optimal decision boundaries are inconsistent under different LMs and prompts.
2

---

Published as a conference paper at ICLR 2023
GPT-J(6B) GPT-e-Large( LBB
57
68
81
87
85
79
71
64
60
55
53
51
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
52
56
73
85
50
P3
88
92
92
90
87
82
78
74
68
64
60
56
54
52
51
50
50
50
50
50
SST-2 Accuracy(
50
50
50
50
51
54
65
73
86
89
86
77
54
50
70
61
62
64
66
70
72
77
78
83
95
88
91
96
94
P2
54
55
56
58
59
61
63
65
68
71
74
78
80
83
87
90
94
95
92
76
60
76
78
80
81
83
84
85
86
88
89
90
91
93
94
95
95
96
95
93
83
53
56
61
62
67
70
88
-50
.0,693
Decision Boundary(log (lg P position)
Figure 3: Few-shot performance of GPT-2-Large (0.8B) and GPT-J (6B) using different decision boundaries. P1, P2, P3, and P4 represent different prompts. The red rectangle indicates the performance under the conventional decision boundary (P positive = 0.5 for the example task), i.e., naively using the outputs with larger probabilities as the predicted labels. It is observed that the decision boundary plays a critical role few-shot evaluation.
Decision boundary greatly influences the few-shot performance. We evaluate the performance of different models and prompts using different decision boundaries. Results are shown in Figure 3. The red rectangle indicates the conventional decision boundary used by GPT, which naively decodes the label with a larger prediction probability. We observe that shifting the decision boundary can cause wild fluctuations of few-shot accuracy, from near state-of-the-art to random guessing. For each prompt, there is an exclusive region where the decision boundary is relatively robust. The model exhibits poor performance when the decision boundary is far from the robust region.
Performant decision boundaries are not transferable across LMs or prompts. Figure demon- strates that all prompts exhibit strong performance if the decision boundary locates in the robust region. However, different prompts and models lead to different degrees of deviation between the optimal decision boundary and the conventional one. It suggests that performant decision boundaries are inconsistent across models or prompts. Based on the above analysis, we argue that all prompts can achieve better performance when the decision boundary is calibrated into the robust region.
3 PROTOTYPICAL CALIBRATION
We have illustrated that the conventional decision boundary generally deviates from the robust region, which renders in-context learning fragile. this section, we present prototypical calibration (PRoCA) to adaptively learn a better decision boundary.
3.1 PROTOTYPICAL CLUSTER ESTIMATION
Considering an N- W-way few-shot classification task, let X denote the N-dimensional model outputs. For examples whose ground truth is the n-th category, the model outputs compose a prototypical cluster. For instance, the red and blue areas Figure refer to two prototypical clusters respectively.
We assume that each prototypical cluster follows Gaussian distribution:
Pa (X (n: Σn)
n2 N//2 Σ, 1/2 exp( (X - pn) -x H4))
where un and Mn are the mean vector, and covariance matrix of the distribution, respectively. Next, we estimate N prototypical clusters for N categories with Gaussian mixture model (GMM):
N
(a = Σ an Pa(XPuu En),
(2)
n=l
3

---

Published as a conference paper at ICLR 2023
where on is the mixing coefficient for the n-th distribution. In work, we formulate the model prediction x = [21, 22,, NN as follows:
In = log exp(on)
exp(o;)
i==1
where on and o are the logits predicted by GPT, corresponding to label token and label token i respectively. Intuitively, 2n represents the log probability of the n-th category.
After clarifying the GMM definition under few-shot learning, utilize small-scale unlabeled in-domain dataset, named as estimate set (Desi). to estimate the parameters {an, Un 2n 1. by the Expectation-Me Mnaxiiion-Maxiiizzationn (EM) algorithm (Moon, 1996). Notice that the estimate set does not contain any human annotation. Specifically, EM is an iterative method to find the optimal estimation of GMM's parameters by maximizing the likelihood
11 (x)
DEDeo.
3.2 CLUSTER-LABEL ASSIGNMENT
Then we assign the estimated prototypical clusters to the target labels. Concretely, for an estimation
= {Onn: Mn, Σn ) Mn,i is the I-th element of fnn and indicates how much the n-th cluster of belongs label Therefore, we propose cluster-label assignment score (-).()), which represents the overall belongingness of cluster-label assignment. Let the tuple k (k1, kg. AN) denote cluster-label assignment, where k is a permutation of { 2, .... Nj. It means that the n-th cluster is assigned the label k,, The assignment score CLA(() is defined as:
CLA(E, Σ Mn.k..
Then it is transformed into a weighted bipartite matching problem between N clusters and N labels. The optimal assignment is obtained by maximizing CL,((e,)) k):
kn (e) arg max CLA(e, A:),
(5)
kEK
where IC indicates the set of assignment permutations. In the worst case, this process requires N! attempts to find the optimal assignment, which is time-consuming when N is large, thus we adopt Kuhn-Munkre algorithm (Kuhn, 1955) to accelerate
3.3 ESTIMATION SELECTION BASED ON CLUSTER- ASSIGNMENT
The EM algorithm is empirically sensitive to the different initializations of GMM parameters. So we repeat the estimation multiple times with different random seeds. Then we define metric to evaluate how good these estimations are and select the best estimation. As CLA(e, k" reflects the overall label belongingness of the optimal assignment of an estimation e, thus it can be used to evaluate estimations. Formally, we select the estimation e* according to the assignment score of ke follows:
arg max CLA(e, K* (c)),
eEE
where & is the set of estimations obtained by different initializations of GMM parameters.
3.4 INFERENCE
After selecting the desired estimation e", we use GMM to make predictions instead of the conventional approach used in GPT (Brown al., 2020). Due to the class-distributtin discrepancy between the estimate set and the test set, we discard the mixing coefficient on, of each sub-distribution during inference. For a test example, the LM prediction is æ. It will be assigned to the most likely cluster:
arg max n=
Pa( Eg)
Finally, the predicted label is kg (e"), where the cluster-label assignment hy" (e*) is obtained according to Equation (5).
4

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We evaluate five models from GPT-family including GPT-2-large (Radford et al., 2019) with 0.8B parameters, GPT-2-XL (Radford et al., 2019) with 1.5B parameters, GPT-neo (Black al., 2021) with 2.7B parameters, GPT-J (Wang & Komatsuzaki, 2021) with 6B parameters, and Bloom (BigScience, 2022) with 176B parameters.
As for the estimate set, it can be constructed by generating from LMs (Lu et al., 2021; Wang et al., 2021; Meng et al., 2022; Ye et al., 2022) or sampling a light subset of training examples but without golden labels. For simplicity, we choose the latter way construct the estimate set and we further compare their differences in Section 4.5. Moreover, the estimate set size is proportional to the number of classes of the task. For more details, please refer Table 7 in Appendix.
We use the k-means algorithm to initialize GMM parameters to accelcrate the convergence. The maximum iterations and the convergence threshold for each EM process are set to 100 and 1e-3 respectively. Moreover, we repeat the estimation multiple times with different random initializations to avoid getting stuck in local optima. It is worth noting that multiple repetitions bring little additional time consumption compared to the inference of GPT, we thus simply set it to 100 for all tasks.
4.2 EVALUATION PROTOCOL
We evaluate the proposed method nine widely-used text-classification datasets including SST- 2 (Socher et al., 2013), SST-5 (Socher et al., 2013), Subj (Pang Lee, 2004), MR (Pang Lee, 2005), AP (Zhang et al., 2015), DBPedia (Zhang et al., 2015), AGNews (Zhang et al., 2015), RTE (Dagan et al., 2005), and TREC (Voorhees & Tice, 2000). SST-2, SST-5, MR, and AP are sentiment classification tasks. RTE is a textual entailment recognition task and TREC is text retrieval question classification task. Subj and AGNews are subjectivity and topic classification tasks respectively, and DBPeida is ontology classification task. We use the full validation set for evaluation except for AGNews, DBPedia, and AP, for which we randomly sample 2000 test examples.
We compare PROCA with the conventional approach used by GPT (Brown al., 2020) and contextual calibration (Zhao et al.. 2021). Experiments are conducted under O-shot, 1-shot, 4-shot, and 8-shot scenarios. We fix the template format for each dataset (details of templates are shown Table 6) and use the randomly sampled training examples as demonstrations. We compute the average accuracy on the validation set over five random seeds for each setting except for Bloom using 2 seeds. We conduct the evaluation on 8 Tesla A 110 GPUs for Bloom and Tesla V100 GPUs for other models.
4.3 MAIN RESULTS
We report the mean and standard deviation of accuracy across five different random seeds for GPT-2- XL. GPT-J, and Bloom in Table 1. The results of GPi-22-Lagg and GPTT-ne are shown in Table 9 of Appendix. From Table I and Table 9, we observe that PROCA achieves, on average, 13% absolute improvement compared to the conventional approach and a 6% absolute improvement over contextual calibration. In some cases, the absolute improvement can be up 40% and 20% respectively, like GPT-J 0-shot on DBpedia and GPT-2-XL 8-shot on AGNews.
Results show that PROCA maintains high effectiveness across different model sizes and few-shot scenarios, indicating its strong generalization ability. Moreover, compared to the conventional approach, PROCA achieves considerable improvements with lower variance across different prompts in most cases, which suggests that PROCA can effectively calibrate the decision boundary for various prompts (as illustrated in Figure 2). It also reflects that our estimation strategy is reliable and insensitive to different estimate sets, because of the low variance of PROCA's 's zero-shot performance. We observe that the performance gain on Bloom is smaller than that on relatively small models. suggests that huge LMs have less suffering the decision boundary deviation problem. In addition, PROCA seems invalid for GPT-2-XL on RTE. We identify the reason is that the entailment recognition task is too challenging for relatively small models like GPT-2-XI. and the output of LM on such
5

---

Published as a conference paper at ICLR 2023
Shot Method
SST-2
SST-5
MR
Subj
AP
AGNews DBpedia RTE
TREC
Avg
GPT-2-XI.
1.5B
GPT 58.7p.0
28.40.0
58.9u.0
57.6u.e
51.8p.0
41.60.0
60.3u.0
50.0u.0
28.6u.u
48.4
0-shot ConCa 69.30.0
22.60.0
66.90.0
72.90.0
49.8p.0
67.70.0
54.30.0
50.40.0
42.80.0
55.2
PROCA
84.8p.2
45.01.3
82.0p.2
73.3v.1
49.8p.3
64.61.4
73.63.0
49.2a.7
42.02.7
62.7
GPT 59.814.0
26.2s.5
51.30.6
54.58.6
51.0p.1
37.46.7
51.312.7
53.81.0
29.16.5
46.0
1-shot ConCa
76.42.2
30.2s.7
69.45.0
62.07.0
60.34.0
65.0s.a
70.97.
53.10.9
40.53.3
58.6
PROCA 89.42.4
42.52.9
84.31.0
71.85.7
69.8s.2
69.84.3
79.93.8
49.51.9
43.65.0
66.7
GPT 66.313.7
31.37.4
56.56.9
53.44.9
50.9p.1
40.913.0
61.37.6
52.03.6
23.85.7
48.5
4-shot ConCa 79.910.2
33.53.5
67.76.9
68.0s.T
75.65.9
59.96.3
74.9..0
52.90.7
41.14.3
61.5
PROCA 90.40.6
39.64.5
78.1
74.8
80.17.1
67.413.5
87.24.9
52.21.5
46.02.5
68.4
GPT 57.0g.0
30.57.9
65.212.7
57.911.2
50.90.0
42.94.2
67.97.1
53.02.1
37.24.9
51.4
8-shot
ConCa 13.9
28.73.4
7..18.4
68.38.3
71.17.4 80.23.3
55.914.0
75.04.2
53.10.2
45.81.7
60.7
PROCA 88.01.a
36.54.4
80.8..4
79.37.8
75.53.2
89.4u.7
51.32.0
46.02.6
69.7
GPT-J 6B
GPT 66.60.0
26.60.0
65.90.0
67.90.0
54.20.0
33.70.0
21.80.0
55.2o.o
23.40.0
46.1
O-shot ConCa 57.70.0
35.40.0
57.1p.0
59.90.0
63.1p.0
60.1p.0
49.90.0
55.60.0
42.2g.0
53.4
PROCA 74.2p.2
42.1o.
73.10.4
69.5o.2
63.30.2
55.10.4
66.1
57.01.0
53.4
61.5
GPT 67.77.3
31.74.9
68.14.1
65.010.9
92.92.7
65.614.6
65.614.8
52.64.6
41.89.0
61.2
1-shot
ConCa 89.32.2
46.53.4
88.5
58.83.0
93.51.3
75.55.7
79,93.3
53.1a.s
64.75.3
72.2
PROCA 90.81.7
47.62.3
87.91.a
77.9..
95.1u.a
79.8s.4
90.02.2
56.7a.1
55.3a.4
75.7
GPT 88.61.3
44.7a.3
84.4s.2
58.26.3
89.410.0
72.16.5
80.513.2
55.66.7
38.13.4
68.0
4-shot
ConCa 92.93.7
47.74.4
87.81.8
66.5
93.41.0
76.44.0
88.63.0
54.71.5
48.54.9
72.9
PROCA 95.0p.4
46.24.6
89.4..
79.4s.8
95.8p.s
79.9..8
91.92.6
61.22.7
57.1
77.3
GPT 91. 91.1u.1
44.92.9
89.52.3
82.1a.9
95.21.7
76.9u.7
87.7..1
61.0a.9
44.4s.6
74.8
8-shot ConCa 93.41.8
46.61.4
90.10.5
80.5s.8
96.20.3
79.96.4
90.82.0
59.64.8
53.57.9
76.7
PROCA 94.41.0
47.44.4
90.70.7
83.64.2
96.1p.5
84.21.s
95.10.6
61.77.2
61.07.a
79.4
Bloom 176B
Bloom 73.40.0
26.0p.a
71.0p.o
53.30.0
60.1p.o
27.1p.0
48.5o.0
62.5o.0
59.0o.n
53.4
O-shot ConCa 73.9p.0
25.30.0
71.8p.o
49.0n.0
51.1p.0
38.2p.0
61.0n.0
$3.80.0
41.0o.0
51.7
PROCA 76.4v.1
31.8u.a
73.4..4
61.3u.3
80.4u.s
60.1s.s
75.8u.1
62.6u.2
52.9u.s
63.9
Bloom 91.72.6
31.17.5
84.62.3
60.4s.s
96.1o.1
67.6n.9
81.82.0
61.23.4
55.17.1 70.0
1-shot ConCa 91.81.a
38.94.a
86.81.6
51.22.a
96.1u.4
78.4p.s
80.41.9
54.05.6
69.31.a
71.9
PRoCa 93.60.8
47.52.8
88.0o.s
72.01.8
95.70.4
81.60.7
83.71.6
65.70.4 67.52.5
77.3
Bloom 96.3..1
46.70.8
87.36.3
72.26.4
94.22.5
68.8s.2
86.2
64.12.4
29.10.a 71.7
4-shot
ConCa 96.0b.1
46.92.9
89.71.1
70.47.7
94.21.9
78.0p.1
86.62.4
56.3u.7
64.87.6
75.9
PRoCa 95.70.2
50.22.8
91.20.1
78.50.5
95.8p.5
82.71.2
87.01.3
68.6p.4
56.8
78.5
Bloom 94.62.0
43.24.a
90.9u.s
78.62.2
96.0u.9
75.41.9
88.42.1
65.92.4
48 48.9
75.8
8-shot
ConCa 96.1
42.2s.5
91.00.9
75.8
95.90.4
81.92.0
89.52.6
59.00.5
73.9
78.4
PROCA 95.31.a 53.1
92.0n.s
80.61.9
95.6p.s
82.12.0
85.1
69.52.7
68.67.8
80.2
Table 1: Performance comparisons among the conventional approach (GPT; Brown et al. 2020),
contextual calibration (ConCa; Zhao et al. 2021) and prototypical calibration (PROCA; Ours). We report the mean and the standard deviation of accuracy across 5 different prompts the validation set except for Bloom, for which we only use 2 random seeds to reduce the computational cost. We also show the average performance across nine datasets. The results of ConCa replicated based on the released code'. The standard deviation of O-shot accuracy for PROCA is caused by the difference of estimate sets over 5 random seeds. It shows that PROCA generally outperforms GPT and ConCa.
challenging tasks is no more discriminative (same for GPT-2-Large, shown in Table 9 in Appendix).
4.4 EFFECTIVENESS ANALYSIS
We conduct more experiments verify the effectiveness of PROCA. The experimental results are the average accuracy of GPT-2-XL conditioned on 5 different 4-shot prompts unless otherwise specified.
PROCA is consistently effective across different templates. We conduct the experiments across nine different prompts templates and label spaces (details of templates are shown in Table 8 of
hhttps://// ://p:////wwwwww//.... gi thub. www www.ttoyyoaazhhi.o/oooooo few- shot- learning

---

Published as a conference paper at ICLR 2023
Phocis (55T-2)
Phota (Seb))
ConTa (S5T-2)
ConCa (Song)
76
SST-2 Accuary(
SST-2 Accuracyt
00-
Sabj Accuracy %%
69
86
84
80-
0.1
0.2
0.7 08
0.9
Proportion of positive examples in estimate set
GPT
ConCa
PROCA
Figure The impact of class- imbalanced estimate set on PROCA's performance.
Figure 4:
Performance comparison
across nine different templates.
SST-2 Accuary(%)
80-
70-
60-
GPT ConCa
PRoCA
0.0
0.125
0.25
0.375
0.5
0.625
0.75
Proportion of Positive Examples Prompt
Figure 6: Performance comparison under different label proportions and permutations of demonstra- tions. Each box indicates the accuracy of twelve randomly sampled permutations.
Appendix). The performance comparison among three approaches on SST-2 is shown in Figure 4. We observe that contextual calibration remains high variance although it improves the average accuracy. However, our proposed prototypical calibration can bring a large improvement with low variance, which indicates that PROCA is effective various prompt templates.
PROCA is robust under demonstration perturbations. Previous works (Zhao al., 2021; 1; Lu et al., 2021) have noticed that the order of training examples has significant effects on the performance of few-shot demonstrations. In this part, we evaluate our prototypical calibration conditioned on nine 8-shot prompts with different class proportions for SST-2, and show the accuracy of twelve randomly sampled orderings for each proportion Figure 6. We find that contextual calibration can improve the performance in most cases but is still sensitive to the orderings. However, PROCA is significantly superior to the others and keeps an extremely low variance across different permutations, indicating the sonssensitsseness to the class proportion and permutation. It is also shown that although the class- balanced prompts tend to have higher performance, there are some exceptions(e.g g., the prompt with all negative samples the most performant for both the conventional approach and contextual calibration). We think that it GPT-2-XL33 intrinsic bias to the positive class that leads toward counter-intuitive results.
PROCA robust class imbalance. Due to the unavailability of the labels of the estimate examples, PROCA may suffer the problem of class imbalance. We construct nine estimate sets with different imbalance levels for SST-2 and Subj by controlling the proportion of positive examples in the sampled set. Then we evaluate PROCA and contextual calibration them. The experimental results in Figure 5 show that the estimate set's class imbalance level affects the performance of PRoCA some extent and the class balance estimate set can lead to higher accuracy. As described in Section 3.1. standard GMM estimates the weight of each cluster, which reflects the proportion "weights- operation, the problem
of different classes in the estimate set. Owing
7

---

Published as a conference paper at ICLR 2023
of class imbalance has much less negative impact PROCA, which even with extremely Ulass-imbalances estimate set surpasses contextual calibration on both SST-2 and Subj. Besides, the absolute advancement for the class-balances estimate set can reach 20% and 15% respectively.
Shot
Method
SST-2
SST-5
MR
Subj
AP
AGNews
DBpedia
RTE
TREC
GPT-nco
84.5s.7
33.2s.0
68.713.
61.914.8
85.810.3
68.2s.7
71.81a.9
47 47.9u.a
37.2a.4
4-shot
ConCa
91.71.1
41.3p.o
81.07.2
62.611.5
93.4u.7
60.4s.9
86. 86.73.b
$2.9s.s
57.38.0
PROCA-g
91.91.4
43.53.4
83.94.3
73.46.9
91.60.8
70.75.4
80.12.9
50.41.6
57.32.a
PROCA-t
91.61.8
38.75.6
85.61.2
79.72.9
93.30.5
75.65.6
90.4s.7
55.01.0
59.22.9
GPT-neo
68.01a.2
31.3n.
70.2
57.58.1
90.42.7
66.5g.0
78.86.3
49.22.6
50.85.6
8-shot
ConCa 81.2p.1
33,94.7
77.8
71.05.7
93.60.9
73.43.5
90.31.0
51.36.6
56.08.0
PROCA-g 90.12.4
39.6..1
81.14.5
75.53.8
92.01.4
77.34.5
81.82.4
52.42.7
63.45.2
PROCA-t
91.91.2
39.44.0
77.813.9
81.3
93.9u.7
78.92.5
92.01.5
56.81.s
56.03.G
Table 2: 4- and 8-shot performance comparison of different estimate set construction methods for GPT-neo across nine text classification tasks. PROCA-g and PROCA-t represent PROCA based on the unlabeled estimate set generated by LM and randomly sampled from the training set, respectively.
4.5 ABLATION STUDIES
Comparison between different estimate set construction methods. There two ways to construct the estimate set. One is using light unlabeled examples from the training set, which is simple and convenient. The other is utilizing the generation ability of LMs construct unlabeled dataset (Lu et al., 2021; ; Wang et al., 2021; Meng et al., 2022; Ye et al., 2022). For the generation method, we follow Lu al. (2021) to generate diverse estimate examples based on the various permutations of demonstrations. Specifically, we only use two labeled examples per category as demonstrations for generation except for DBPedia(one labeled example per category) and all these labeled examples are not involved in the evaluation. The 4-shot and 8-shot experimental results are shown in Table and the 0-shot and 1-shot results are shown in Table 5 of Appendix. We observe that PROCA greatly outperforms the original LM whether using unlabeled data generated by LM or randomly sampled from the training set. also shows that PROCA-I performs slightly better than PROCA-g and we speculate that it is due to the lower quality of the unlabeled data generated by LM.
relatively small-scale estimate set is sufficient for PROCA. In Figure 7, evaluate PROCA with ten different estimate set sizes across five datasets. We report the average accuracy over five randomly sam- pled estimate sets, conditioning on the same 4-shot
Accuracy(
60-
prompt in each setting. We observe that increasing the scale of the estimate set within a certain small range can greatly improve the classification accuracy and reduce the variance. However, a larger estimate set can hardly bring further improvement, which indi- cates that small estimate set can support PROCA to be optimal. It also shows that PROCA has acceptable performances with just several estimate examples on SST-2, MR, and Subj, which even surpasses both the conventional approach and contextual calibration.
50
40-
4GNum Sat
35T-3
100
300
380
500
1090
2000
Estimate set size
Figure 7: Performance of PROCA across different estimate set sizes.
Estimation selection according to assignment score useful to PROCA. The standard estimation of GMM aims to maximize the likelihood of all observations and select the optimal estimated parameters among multiple repetitions. We argue that the estimation with maximum likelihood is not consistently beneficial to PROCA especially in multi-classes tasks, because there is no supervision to force the predictions to be assigned to their inclined classes during the estimation procedure. We propose to determine estimation according to the assignment score (as described in Section 3.3). We compare PROCA with the strategies for GPT-2-XL and GPT-J respectively in Table 3. It indicates that our determination strategy can achieve more stable improvements on AGNnews and DBPedia regardless of model size.
8

---

Published as a conference paper at ICLR 2023
0-shot
1-shot
4-shot
8-shot
Dataset
Strategy
GPT-2
GPT-J
GPT-2
GPT-J
GPT-2
GPT-J
GPT-2
GPT-J
max-likelihood
64.01.1
55.2
65.85.2
79.23.s
60. 11.1
79.76.7
71.78.7 78. 11.7
AGNews
assignment score 64.61.4
55.1u.4
69.8
79.83.4
17.4
79.96.6
75.5 84.21.s
max-likelihood
59.24.6
71.87.3
77.73.8
76. lsss
82.23.9
79.3s.0
83.82.4
DBPeida
assignment score 73.63.0
66.11.5
1.5 79.93.8
90.0
87.2
91.92.6
89.40.7 95.10.5
Table 3:
Performance of PROCA with different strategies of estimation selection (maximum likelihood. and assignment score as Equation (4)) for GPT-2-XL and GPT-J on AGNews and DBPedia.
5 RELATED WORK
Instability of Few-shot Learning with Language Models. has been recognized that the few-shot performance of language models is unstable under different in-context scenarios. Language models are prone to predict some specific labels due to the intrinsic bias or demonstration permutations (Zhao et al., 2021; Lu et al., 2021). Lu et al. (2021) demonstrate LM's sensitivity to the order of few-shot demonstrations, and introduced an Entropy-based metric to select the most performant prompts. Zhao al. (2021) attribute the instability to three biases of prompts, including majority bias, recency bias, and common token bias, and proposed a contextual calibration approach. However, the selected content-free test inputs can not precisely reflect the bias of models and lead to the problem of over-correction or
On the contrary, we adaptively provide the classification criterion according the text inputs' overall prediction distribution, and completely calibrate the bias introduced by models and prompts.
Improving and Understanding In-context Learning with Language Models. Due to the instabil- ity, prior efforts propose various methods to improve the in-context learning performance. Holtzman et al. (2021) explores the surface form competition problem in zero-shot models and proposes Domain Conditional Pointwise Mutual Information to reweigh the answer scores. Min (2021) introduces noisy channel approach that computes the conditional probability of the input given the label, which boosts improvements with lower variance and higher worst-case accuracy. Moreover, Liu al. (2021) focus prompt engineering to construct more semantically-similar demonstrations. To the best of our knowledge, we are the first to study the intrinsic reason for the instability of in-context learning from the perspective of the decision boundary and propose prototypical calibration improve Another line work aims to understand how in-context learning works through casting it as implicit Bayesian inference (Xie al., 2022), analyzing corpora sources and statistics (Shin et al., 2022; Razeghi et al., 2022) and learning which aspects of demonstrations contribute most to the downstream performance (Min et al., 2022).
CONCLUSION AND LIMITATION
To our analysis, the decision boundary of critical importance to the performance of few-shot demonstrations and the traditional decision boundary leads to the fragility of prompting LMs. We propose prototypical calibration to adaptively lear more robust decision boundary. Experiments show that the calibrated decision boundary is effective across various prompt templates, class proportions, and permutations. We achieve on average 13% absolute improvement across different sizes of pretrained language models on nine popular text classification tasks.
limitation of our method that not applicable for tasks whose label space open-ended since fixed label space is necessary for estimating prototypical clusters. Furthermore, our method is designed for in-context learning on individual downstream tasks, fails to calibrate the inherent bias of language models like gender and occupation bias. For future work. we would like extend our method to the tasks with open-ended answer space, such as generative question- answering and text summarization tasks.

---

Published as a conference paper at ICLR 2023
REFERENCES
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang. Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. My: Pseudo-masked language models for unified language model pre-training. In ICML 2020, volume 119 of Proceedings of Machine Learning Research, pp. 642-652. PMLR, 2020.
BigScience. Bigscience language open-science open-access multilingual (bloom) language model. Technical report, May 2022.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Languagg Modeling with Mess-Tenno,,,ow, March 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS 2020, 2020.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pp. 177-190. Springer, 2005.
Jacob Devlin, Ming-Woi Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805 2018.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In NeurlPS 2019, pp. 13042-13054, 2019.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoer. Surface form compe- tition: Why the highest probability answer isn't always right. arXiv preprint arXiv:2104.08375 2021.
Zhengbao Jiang, Frank Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to- to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint 11Xiv:1910.13611, 2019.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in in-context examples for gpt-3? arXiv preprint arXiv:2101.0668 2021.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CORR, abs/1907.1 1692, 2019.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint 0887:22004.08766. 2021.
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. ArXiv, 2022.
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. arXiv preprint arXiv:2108. 100:::000.04166,, 2021.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. 12837, 2022.
10

---

Published as a conference paper at ICLR 2023
T.K. Moon. The expectation-maximil maximization algorithm. IEEE Signal Processing Magazine, 13(6): 47-60, 1996. doi: 10.1109779.549975
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summariza- tion based on minimum cuts. arXiv preprint cx/0409058, 2004.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv preprint cs/0506075, 2005.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, nly Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies few-shot reasoning. CoRR, abs/2202.07206, 2022.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633, 2021.
Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woo-Myoung Park, Jung-Woo Ha, and Nako Sung. On the effect of pretraining corpora in-context learning by large-scale language model. CoRR, abs/2204. 13509, 2022. doi: 10.48550/arxiv.22044 13500
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Ellen M Voorhees and Dawn Tice. Building question answering test collection. Proceedings of the 23rd annual international ACM SIGIR conference Research and development in information retrieval, pp. 200-207, 2000.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https: gttbbttthbugg/ggg/// com www.ooio/o///o.oooo..o.o.eo..w.e flolz http:///esstttaaassooeeaa jax, May 2021.
Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. Towards zero-label language learning. ArXiv, abs/2109.00119. 2021.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xinet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong, Zerogen: Efficient zero-shot learning via dataset generation. ArXiv, 2022.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving
few-shot performance of language models. In International Conference Machine Learning, pp. 12697-12706. PMLR, 2021.
11

---

Published as conference paper at ICLR 2023
A EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS
Table 4: Average performance of the conventional approach, contextual calibration (ConCa) and
PROCA for GPre--Large and GPT-neo across nine text classification tasks.
Shot Method SST-2
SST-5
MR
Subj
AP
AGNews DBpedia RTE
TREC
Avg
GPT-2-Large
0.88
GPT 7..
26.20.o
70.2p.o
51.70.0
53.2n.0
33.5p.o
56.70.o
52.70.0
34.60.0
50.1
0-shot ConCa
80.5u.o
41.7u.u
66.9u.9
54.5u.0
60.6u.a
59.3u.9
57.4u.0
53.4a.u
32.0c.u
56.3
PROCA
85.6u.3
43.5u.s
82.Au.3
69.5u.a
61.2u.s
59.62.2
79.81.s
53.6u.4
48.31.4
64.8
GPT
16.1
28.510.4
53.34.
50.51.5
50.70.s
28.85.3
38.314.3
52.80.1
32.73.4
1 1-shot ConCa
75.312.7
40.13.6
68.712.0
61.42.2
56.16.4
65.0s.8
69.8g
51.33.4
39.68.1
58.6
PRoCa
83.12.1
37.92.5
79.5
65.2s.5
84.12.0
53.7p.8
18.8
51.32.6
47.27.5
64.5
GPT 52.72.0
31.5a.6
59.4s.6
60.310.7
52.82.7
33.76.3
38.616.8
51.02.7
32.46.9
45.8
4-shot ConCa 71.811.9
42.32.3
70.1
58.510.3
72.07.2
56.6p.3
77.64.1
52.9..1
42.27.1
60.4
PROCA 86.93.0
35.54.6
81.5,.2
75.13.2
87.63.9
60.311.0
82.7G.1
52.22.6
49.46.2
67.9
GPT 72.017.4
31.27.0
61.89.9
57.9s.7
51.81.9
40.96.7
60.412.7
53.91.4
38.0a.a
52.0
8-shot ConCa
83.812.0
40.0a.s
72.77.6
63.4
62.66.9
52.610.8
77.43.6
49.62.3
47.07.7
61.0
PROCA 88.0s.3
37.64.3
84.5o.6
79.76.7
90.52.4
72.52.6
87.90.7
52.33.2
47.5s.0
71.2
GPT-neo 2.7B
GPT 59.6n.0
27.60.o
58.0n.o
77.40.0
48.6n.0
45.0p.o
35.1o.0
54.5a.0
22.6g.0
47.6
0-shot ConCa 77.0n.o
29.00.0
72.5p.0
52.4n.0
48.4n.0
61.3p.0
56.60.0
52.0n.0
29.6g.a
53.2
PROCA 86.1o.7
45.5
80.8n.1
77.20.4
46.1o.1
61.51.0
71.3
54.3g.9
41.8..3
62.7
GPT 73.1a.3
33.49.3
72.36.2
53.2u.b
80.34.s
45.210.2
54.312.3
47.0u.s
29.613.1
54.3
1-shot ConCa 84.011.5
40.12.7
83.74.4
56.56.4
82.0s.1
58.73.7
73.64.0
48.92.4
55.27.7
ProCa 90.02.0
41.24.4
86.2p.7
68.23.3
86.52.1
69.8s.4
85.5
52.92.0
57.6
70.9
GPT 84.5s.7
33.2g.0
68.713.4
61.914.8
85.810.3
68.2s.7
71.810.9
47.90.8
37.26.4
62.1
4-shot ConCa 91.71.1
41.3a.0
81.07.2
62.611.
93.4n.7
60.4s.9
86.73.6
$2.93.6
$7.38.a
69.7
PRoCa 91.61.8
38.75.6
85.61.2
79.72.9
93.3u.s
75.6s.s
90.4a.7
55.01.0
59.24.9
74.3
GPT 68.019.2
31.36.9
70.214.4
57.56.1
90.42.7
66.58.0
78.86.3
49.22.6
50.8s.a
8-shot ConCa 81.2g.1
33.94.7
77.89.8
71.0s.7
93.60,9
73.45.5
90.31.0
51.38.6
56.0s.o
PRoCa 91.91.2
39.44.0
77.813.9
81.33.6
93.90.7
78.92.5
92.01.
56.81.
56.03.6
74.2
Table 5: 0-shot and 1-shot performance comparison of different estimate construction methods for GPTT-no across nine text classification tasks. PROCA-g and PROCA-t represent PROCA based on the unlabeled estimate set generated by LM and randomly sampled from the training set, respectively.
Shot
Method
SST-2
SST-5
MR
Subj
AP
AGNews
DBpedia
RTE
TREC.
GPT-neo
59.6n.o
27.60.
58.0p.o
77.4p.0
48.6..0
45.0p.o
35.lo.0
54.5
22.60.0
0-shot ConCa
77.00.o
29.00.n
72.50.0
52.4p.o
48.4o.0
61.3a.o
56.6p.o
52.0n.p
29.60.n
PROCA-g
85,90.o
43.90.0
80.7e.a
71.90.9
46.60.0
62.7a.v
42.90.a
53.4u.b
27.40.u
PROCA-t
86.1u.7
45.5
80.8
77.2u.4
16.
61.51.u
71.31.9
54.3u.
41.8
GPT-neo
73.16.3
33.49.3
72.36.2
53.26.6
80.3
45.210.2
54.312.3
47.0o.s
29.613.1
1-shot
ConCa
84.011.5
40.12.7
83.74.4
56.56.4
82.0s.1
58.7a.7
73.64.0
48.92.4
55.27.7
PROCA-g
89.02.1
37.35.2
82.02.8
66.92.9
85.61.9
67.3a.
74.16.2
49.82.3
48.65.a
PROCA-t
90.02.0
41.2..4
86.2o.7
68.2a.3
86.5..1
69.8..4
85.5
52.92.0
57.64.5
12

---

Published as a conference paper at ICLR 2023
Table 6: Prompt templates for nine datasets our experiments.
Dataset
Template
Label Space
SST-2
Review: (Sentence)
Positive / Negative
Sentiment: (Label)
SST-5
Review: [Sentence] Sentiment: (Label)
terrible / bad /okay good great
MR
Review: [Sentence]
Positive / Negative
Sentiment: (Label)
Subj
Input: (Sentence)
objective subjective
Type: (Label)
Title: [Title)
Positive / Negative
Review: [Content)
Is the review positive negative? (Label)
AGNews
Classify the news articles into the categories of World, Sports, Business, and Technology.
World / Sports Business / Technology
Article: (Sentence) Answer: (Label)
DBPedia
Classify the documents based on whether they are about a Company, School, Artist, Athlete, Politician, Transportation, Building. Nature, Village, Animal, Plant, Album, Film, or Book.
Company School Artist / Athlete / Politician / Transportation / Building Nature Village/ / Animal Plant Album / Film Book
Article: [Sentence)
Answer: (Label)
RTE
(Premise)
False / True
question: (Hypothesis) True False?
answer: [Label]
TREC
Classify the questions based on whether their answer type is a Number, Location, Person, Description, Entity, or Abbreviation.
Number / Location Person Entity Abbreviation
Description
Question: (Sentence)
Answer Type: (Label)
Table 7: Estimate set size for nine datasets in our experiments. GPTs include GPT-2-Large, GPT-2- XL. GPT-neo, and GPT-J.
LM
SST-2
SST-5
MR
Subj
AP
AGNews
DBpedia
RTE
TREC
GPT
500
2000
1000
1000
1000
2000
3000
1000
2000
Bloom
300
1000
500
500
500
1000
2000
500
1000
B COMPARISONS AMONG USING LOGITS, PROBABILITY, AND LOG-PROBABILITY FOR PROCA.
We evaluate the performance of PROCA based on three commonly used model output formalizations including logits, probability, and log- probbabiiyyy on SST-2, AGNews, and AP for GPT-2-XL. The experimental results are plotted in Figure 8. We find that PROCA with log- bog-probabilityy consistently outperforms that with probability, especially for AGNews. We identify that the log operation makes the predictions more fitted to Gaussian distribution and more discrepant. Although PROCA with logits seems to have superior performance when few-shot demonstrations are provided, it degrades severely at zero-shot setting. It suggests that output embedding normalization over label space is necessary to PROCA, because the output is more likely biased to the tokens outside the label space when prompts accessed, which is consistent with the conclusion of Min et al. (2022).
13

---

Published as a conference paper at ICLR 2023
Table Different templates used Section 4.4 for SST-2.
ID
Template
Label Space
Review: (Sentence)
Positive / Negative
Sentiment: (Label)
2
Input: (Sentence)
Positive / Negative
Prediction: [Label]
3
Review: (Sentence)
good / bad
Sentiment: (Label)
4
(Sentence) It was (Label)
good / bad
5
Review: (Sentence)
Yes / No
Positive Review: [Label]
6
Review: (Sentence)
5/0
Stars: [Label]
7
(Sentence) My overall feeling was that the movie was (Label)
good / bad
8 Review: (Sentence)
Positive / Negative
Question: the sentiment of the above review Positive or Negative?
Answer: (Label}
9 My review for last night's film: (Sentence) The critics agreed that this movie was [Label]
good / bad
SST-2
AGNews
90
80-
70-
Accuary (%)
Accuary (%)
Accuary
70-
60+
log prob
50-
log prob logits
log prob
50-
logits
50-
logits
4
0
Number of Examples in Context
Number of Examples in Context
Number of Examples Context
Figure 8: Different model output formats for estimation can influence the performance of PROCA,
including logits, probability (prob), and log log-probability (log-prob).
14

---

Published as a conference paper at ICLR 2023
Table 9: Average performance of the conventional approach and PROCA for RoBERTa-Large across nine text classification tasks.
Shot
Method
SST-2
SST-5
MR
Subj
AP
AGNews
DBpedia
RTE
TREC
arBeRTa-Large
73.90.0
30.7a.0
73.30.0
60.90.0
79.4p.0
65.7p.0
43.60.0
51.30.0
37.2p.0
O-shot
PROCA
90.40.2
45.41.9
85.8p.2
68.80.2
79.71.4
69.0p.s
79.9u.s
58.7
42.82.2
RoBeRTa-Large
94.10.4
40.82.0
80.27.7
51.93.4
67.55.8
63.716.1
54.917.6
57.80.8
35.42.7
8-shot
PROCA
93.8p.2
47.92.0
88.0
63.1s.3
90.52.6
78.8
90.52.6
65.22.0
39.82.9
Table 10: Templates used Appendix
Task
Template
Label Space
SST-2
<S> Sentiment: [MASK].
terrible great
SST-5
<S> Sentiment: [MASK].
terrible / bad / okay good / great
MR
<S> Sentiment: [MASK].
terrible great
Subj
Type: [MASK]
objective / subjective
TREC
<S> Type: [MASK].
Number Locattio Person Description
/Entity Abbreviation
AGNews
Topic: [MASK]
World Sports / Business / Technology
DBPedia
Topic: [MASK].
Company / School Artist Athlete Politician / Transportation / Building Na- ture Village Animal Plant Album / Film Book
C PROCA FOR BIDIRECTIONAL LANGUAGE MODELS.
To validate the effectiveness of PROCA for bidirectional language models, we conduct the evaluation on RoBERTa-Large (Liu et al., 2019) with 355M parameters across the nine text classification tasks. The templates for RoBERTa are constructed by replacing the label symbols with [MASK] symbols. We compare the 0- and 8-shot average performance of LaBerta-Large and PRoCA over random seeds and the detailed results are shown in Table 9. We can see that PROCA greatly outperforms RoBERTa-Large
templates in Table
all tasks except for 8-shot on SST-2. Therefore, we suggest that PROCA is also effective for Bidirectional Language Models.
15

---

Published as a conference paper at ICLR 2023
Table 11: Performance comparison among domain conditional PMI, noisy channel method and PROCA when using GPPg-2-Large on text classification tasks.
Shot
Method SST-2 SST-5
MR
Subj
AGNews DBpedia
TREC
PMIpC 85.6p.0
22.0g.o
64.1p.0
44.0p.0
O-shot Channel 76.90.0
29.8p.0
72.80.o
64.20.0
59.8p.0
58.1o.0
46.40.0
PROCA
84.6p.1
40.72.9
81.10.2
72.
69.70.7
68.71.9
42.8p.9
8-shot Channel 86.41.2 35.32.5
80.52.5
49.9 1233
71.53.6
72.52.
30.8p.3
PROCA 86.43.8 37.75.2 85.5
82.46.0
75.72.1
87.43.7
51.8g.4
D PERFORMANCE COMPARISON WITH DOMAIN CONDITIONAL PMI SCORING FUNCTION AND NOISY CHANNEL METHOD.
Domain Conditional PMI Scoring Function (Holtzman et al., 2021) and Noisy Channel models (Min et al., 2021) also have strong performance on language model prompting in zero- and few-shot text classification. To compare against these approaches, we conduct the 0-shot and 8-shot evaluation on G-r---Large for channel models' and PROCA. We use simpler prompt templates for fair comparison and the details are shown in Table 10. The 0-shot accuracies of PMIpe reported by Holtzman et al. (2021) are also presented for further comparison. As shown in Table 11, clear that PROCA outperforms channel models in most cases and has competitive performance with PM on the 0-shot scenario. PMIDC also calibrates the output distribution based on the domain premises and different domain premises may lead different calibration results. our opinion, channel models transform the decision boundary from label space to input space and it is more robust than the conventional language model in-context learning. There may still exist the decision boundary deviation problem in channel models. In comparison, our method directly calibrates the decision boundary by estimating the prototypical clusters and therefore boosts higher performance.
"The results of noisy channel models are replicated based on the released code /shnsw25 Satpssws//Chhaeee Channel I.M- Prompting
16


Reviews:
[{"review_id": "lqprhzJ2Yn", "reviewer": "Reviewer_f9YD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a highly empirically effective approach for in-context learning using so-called _prototype calibration_. The simple, yet dramatically effective approach is well motivated by the authors and demonstrated to be empirically effective under a wide variety of settings, testing sensitivity to number of shots, prompt template, class imbalance and more. ", "summary_of_the_review": "This is a strong paper which presents a simple and effective approach for in context learning. The approach uses a GMM cluster-based approach to make predictions. The effectiveness of the approach is demonstrated through extensive experiments that evaluate the performance along many dimensions of number of shots, prompts, estimate set, etc.", "strength_and_weaknesses": "Overall I believe this to be a very strong empirical investigation into in-context learning. The proposed approach is clearly motivated, simple and seems to be highly effective. \n\nStrengths:\n* Dramatic performance benefits from the proposed approach. 10+ pt improvement in most cases on average, 5+ in a few others (Table 1). ]\n* Detailed empirical analysis that includes: effectiveness across templates, demonstration perturbation, class imbalance, estimate set construction, estimate set size, \n* The results and analysis are clearly presented as are the methodological approach.\n* Provides basis for future work on using using prototypes/clusters in the midst of in-context learning\n\nWeaknesses: \n* It might be interesting to consider how the fit of the GMM distribution effects the downstream task performance. Also since the number of points in the estimate set is not so many, it could be interesting to consider a wider array of clustering inference methods including more exhaustive/expensive methods. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "iXtjpUjNZL", "reviewer": "Reviewer_9WSu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes prototypical calibration for zero-shot and few-shot classification tasks when evaluating large language models. The approach first uses Gaussian mixture distribution to estimate the prototypical clusters for all categories of the classification task and then assigns each cluster to the corresponding label by solving a weighted bipartite matching problem. Each example’s prediction is calibrated by the likelihood of the prototypical clusters. They demonstrate significant improvements over contextual calibration (Zhao et al., 2021) on a wide range of tasks.\n", "summary_of_the_review": "The paper has a novel idea for calibrating language models and shows strong empirical improvements over baselines. One concern is that the paper does not discuss a few critical related works or show empirical comparisons to these works. ", "strength_and_weaknesses": "Strength\n- Very strong and consistent empirical results from calibration. The performance is exceptionally strong for relatively smaller-sized models with fewer in-context learning examples. \n\n\nWeaknesses\n- There exist related works, though they do not necessarily claim to develop calibration approaches, they share the same spirit as this method to modify the output distributions for zero-shot or few-shot approaches [1][2]. Discussing these previous works and comparing results with them would help better position this paper. \n- Related to the previous point, I think the related work section does not present the best relevant paper, especially the first section. I suggest the authors conduct a more thorough literature review, along with the direction of calibrating language models. \n- I would also love to see if the calibration method works for masked language models given that MLMs have much stronger zero-shot performance on these evaluation datasets. \n\n[1] Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right \n[2] Noisy Channel Language Model Prompting for Few-Shot Text Classification\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4DOw5EI5Wr", "reviewer": "Reviewer_uD7y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores the importance of the decision boundary in few-shot learning in language models with GPT. Then, the authors propose an improvement over the existing few shot methods by adaptively learning this decision boundary. The decision boundary is learned by modeling the output distributions of GPT few-shot model as GMM and fitting to the class labels. The results shown are better than the baselines up to 13%. The ablation study is done by measuring the effect of template choices, class imbalance and class permutations. The robustness is measured by running with several random initializations.", "summary_of_the_review": "The authors propose a method where they learn decision boundary for an in context GPT few shot learning method. The motivation, story and the experiments parts are the strongest side of the paper. The ideas and results are plenty and well supported, which surpass the disadvantage coming from the computational complexity. ", "strength_and_weaknesses": "Strengths:\n\n* The experimental results and applicability are well supported by using multiple datasets and their ablation study. Substantial improvements are obtained in terms of the accuracies. \n\n* The idea of focusing on the decision boundary is well motivated and demonstrated. \n\nWeaknesses:\n\n* The label assignment part and multiple initializations and runs in the proposed algorithm can be computationally heavy as the authors mentioned. However, this might not be concerning since it is performed at the output space after the training.\n\n*  Why the authors chose 4 and 8 shot in their experiments? It looks limiting and not reasoned well. Is it a standard procedure for these datasets?\n\n* Although the authors mentioned and used averaging and careful initializations, the stability of the method is still a bit concerning to me. Too much unstable/sensitive methods are combined and dataset and task itself also poses another sensitivity issue by nature and since they are scarce. However, the results suggest otherwise unless it is 0 or 1 shot.\n\n* It would be good to explore the boundaries of the method in terms of the original dataset/labels and the desired dataset/task, and a notion of closeness between these tasks. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "3zFXI2me3C", "reviewer": "Reviewer_iRwh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents the analysis that decision boundary is of critical importance to the performance of few-shot demonstrations and the traditional decision boundary leads to the fragility of prompting LMs. The authors propose prototypical calibration to adaptively learn a more robust decision boundary for few-shot learning of language models.", "summary_of_the_review": "Although calibrating prototypes for few-shot learning sounds reasonable, I have difficulty in understanding the specific approach of this submission, especially 3.2 and 3.3.\n\n", "strength_and_weaknesses": "Strength: The analysis that decision boundary is of critical importance to the performance and Instability of few-shot demonstrations is interesting and suggestive. The authors propose prototypical calibration to adaptively learn a decision boundary for\nfew-shot classification of language models. They present provide rich experiments to show the effectiveness of their proposed PROCA on a wide range of text classification tasks.\n\n\nWeaknesses: The paper is somewhat vague for me and I am still confused about some notions or symbols. For example, μn and Σn should be placed with vector and matrix symbols. I am confused the relationship between μn and μn,l. Does the {μn,l} is the l-th element of μn or they are different variables? Since tuple k is a cluster-label assignment, does this mean that k represents a matrix of N * N?  The authors transforme the assignment score to a weighted bipartite matching problem between N clusters and N labels. It would be better if the authors can provide more ditails about the  bipartite matching problem. In my opinion, the key problem in this work is learn cluster-label assignment, which however omits useful details and also overall algorithm. Since we are dealing a N way few shot problem, whether the few labeled samples in N way few shot task are utilized for the learning of assignment score. Besides, the authors utilize a small-scale unlabeled in-domain dataset, named as estimate set (Desti), to estimate the parameters. Does this mean the estimate set is sampled from \"N way\" without labeling?  In that cases, is the PROCA still fair for a typical N way few shot setting? Maybe the authors can clarify the estimate set  from this view.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster