Measuring Pointwise $\mathcal{V}$-Usable Information In-Context-ly
Authors: Sheng Lu
Shan Chen
Yingya Li
Danielle Bitterman
Guergana K Savova
Iryna Gurevych
Venue: EMNLP 2023 Findings (2023)
Keywords: Hardness
Pointwise V-Usable Information
In-Context Learning

Abstract: In-context learning (ICL) is a new learning paradigm that has gained popularity along with the development of large language models. In this work, we adapt a recently proposed hardness metric, pointwise $\mathcal{V}$-usable information (PVI), to an in-context version (in-context PVI). Compared to the original PVI, in-context PVI is more efficient in that it requires only a few exemplars and does not require fine-tuning. We conducted a comprehensive empirical analysis to evaluate the reliability of in-context PVI. Our findings indicate that in-context PVI estimates exhibit similar characteristics to the original PVI. Specific to the in-context setting, we show that in-context PVI estimates remain consistent across different exemplar selections and numbers of shots. The variance of in-context PVI estimates across different exemplar selections is insignificant, which suggests that in-context PVI estimates are stable. Furthermore, we demonstrate how in-context PVI can be employed to identify challenging instances. Our work highlights the potential of in-context PVI and provides new insights into the capabilities of ICL.

Measuring Pointwise V-Usable Information In-Context-ly
Sheng Lu', Shan Chen Yingya Lit, Danielle Bitterman Guergana Savova', and Iryna Gurevych'
Ubiquitous Knowledge Processing Lab (UKP Lab). Technical University of Darmstadt
Artificial Intelligence Medicine (AIM) Program, Mass General Brigham, Harvard Medical School
Department of Radiation Oncology. Brigham and Women's Hospita//Dana-Farbee Cancer Institute
4 Computational Health Informatics Program, Boston Children's 's Hospital, Harvard Medical School
www.ukp tu-darmstadt.de
Abstract
model that is fine-tuned on input-targel pairs, and another model fine-tuned on only the target labels. PVI measures the amount of usable information in an input for a given model, which reflects the ease with which a model can predict certain label given an input. Though it is a recently proposed method, the effectiveness of PVI has been demon- strated in various NLP tasks (Chen et al., 2022a; Kulmizev and Nivre, 2023; Lin et al., 2023; Prasad et al., 2023).
In-context learning (ICL) is a new learning paradigm that has gained popularity along with the development of large language models. In this work, we adapt a recently proposed hard- ness metric, pointwise V-usable information (PVI), to an in-context version (in-contex: PVI). Compared to the original PVI, in-context PVI is more efficient in that it requires only few exemplars and does not require fine-tuning. We conducted a comprehensive empirical analy- sis to evaluate the reliability of in-context PVI. Our findings indicate that in-context PVI esti- mates exhibit similar characteristics the orig- inal PVI. Specific to the in-context setting, we show that in-context PVI estimates remain con- sistent across different exemplar selections and numbers of shots. The variance of in-context PVI estimates across different exemplar selec- tions is insignificant, which suggests that in- context estimates PVI are stable. Furthermore, we demonstrate how in-context PVI can be em- ployed to identify challenging instances. Our work highlights the potential of in-context PVI and provides new insights into the capabilities of ICL.'
Recent years have seen remarkable progress in the development of large language models (LLMs), and it is expected that they will continue to be an essential topic in NLP (Brown et al., 2020; Chung et al., 2022: Chowdhery et al., 2022; Touvron et al., 2023). In the era of LLMs, PVT is useful in many aspects measure of hardness, such as the de- velopment of high-quality new benchmarks that can better evaluate the capabilities of LLMs and the selection of in-context exemplars that enhance model performance. However, given the scales of the statc-of-the- LLMs, the calculation of pv1 can be challenging due to the need for fine-tuning. Motivated by the need to leverage PVI for LLMs and the recent discoveries that in-context learn- ing (ICL) is similar to fine-tuning (Akyürek et al., 2022; von Oswald et al., 2022; Dai et al., 2022), we implement PVI an in-context manner (i.e., in- context PVT). Rather than fine-tuning, we prompt a model using two sew-shot prompts, an imput-targe prompt and a target-only prompt (see A.2 for an example). Figure shows the difference between the calculation of PVT and in-context PVT.
1 Introduction
Understanding the hardness of a dataset or an in- stance is crucial for understanding the progress in machine learning, since dataset is designed as proxy for real-world tasks (Torralba and Efros, 2011). The significance of hardness has been ac- knowledged in the field of natural language process- ing (NLP) (Hahn et al., 2021; 1; Perez et al., 2021; Zhao et al., 2022; Gadre et al., 2023). Extended from the predictive V-information framework (Xu et al., 2020), pointwise V-usable information (PVI) is recently proposed metric for measuring the hardness of individual instances (Ethayarajh et al., 2022). PVT is estimated through supervised learn- ing. which involves fine-tuning two models: one
This work aims to investigate the feasibility of obtaining reliable PVT estimates in an in-context setting. We conducted experiments with various datasets and range of LLMs. We showed that in-context PVI behaves similarly the original PVI: in-context PVT estimates are consistent across different models, and the threshold at which pre- dictions become incorrect is similar across datasets.
'Our code is available at https: //g/ wtt.gghh..c.m.......... cow/UKPLab/ in cn-cotexxt-p/

---

language, which provides more interpretable in- terface and greadly reduces the computation costs compared to supervised training (Dong et al., 2022). ICL has shown strong performance on a series of natural language tasks (Kojima et al., 2022; Saparov and He, 2022; Srivastava et al., 2022; Wei et al., 2022a.b).
targets only
fine-tuninng-
training
targets
fine-tuning-
g
(a) PVI
Recent studies have enhanced our understanding of ICL and its mechanism. Akyürek et al. (2022) and von Oswald et al. (2022) investigate ICL in regression problems, and show that transformer- based in-context learners implement gradient de- scent in implicit way, which be related gradient-based meta-learning formulations. Simi- larly, Dai et al. (2022) argue that ICL performs im- plicit fine-tuning, and understand ICL, process of meta-optimization. transformer- model a meta-optimizer which produces meta-gradients according to the exemplars through forward pass. These meta eeta-ggaadeets are applied to the original model through attention. Dai et al. (2022) provide empirical evidence that ICL behaves similarly fine-tuning at the prediction, representation, and attention levels.
prompting
only
training data
select exemplars
inputs and targets
prompting
(b) In-context PVI
Figure I: The main difference between the calculation PVI and in-context PVL. g and g' are two fine-tuned models and G is an LLM.
Specific to the in-context setting, we found that in- context PV1 estimates are consistent across different selections of exemplars and numbers of exemplars in the prompt. We also found that the variance of in-context PVI estimates across different exemplar selections is insignificant, which suggests that in- context PVI estimates are quite stable. We assessed estimates
One of the most serious concerns in ICL is that in-context learners reported to be very sensitive changes in the input (Liu et al., 2022: Lu et al., 2022; Zhao al., 2021: Chang and Jia, 2023; Chen et al., 2022b; Wang et al., 2023). The reliability of an ICL related method significantly reduced the output changes drastically with the use another set of exemplars. Therefore, it is critical to ensure that an ICL related method maintains consistency across varying exemplar selections.
the correlation between in-context and inter-annotator agreement, and our findings in- dicate that in-context PVI estimates made by larger models are more reliable. We conducted a qualita- tive analysis to show how in-context PVI assists in identifying challenging instances. Additionally, we presented preliminary analysis on the potential of in-context PVT improve model performance through exemplar selection. Our contributions are summarized as follows:
2.2 Hardness
We propose in-context PVI, new approach to calculating PV1 which more efficient than the original method.
Hardness refers to the difficulty of an instance given distribution or the difficulty of a dataset for a given model (Ethayarajh et al., 2022). It is an important concept to understand the quality of dataset (Torralba and Efros, 2011; Zhao et al., 2022; Cui et al., 2023). In addition, the concept of hardness is crucial the study of human-Al inter- action, where hardness estimates are essential evaluate each Al agent's capabilities and facilitate more effective collaboration (Spitzer et al., 2023).
We present an extensive empirical analysis to demonstrate the reliability of in-context PVI.
Our work provides new insights into the capa- bilities of ICL.
2 Related work
2.1
In-context learning
Pointwise V-usable information (PVI) cently proposed metric that measures the hardness of an instance (Fthayarajh al., 2022). PVT is based on the predictive V-information framework, which incorporates mutual information and other
In-context learning (ICL) is popular learning ap- proach that emerges alongside the advent of LLMss (Brown et al., 2020; Liu et al., 2023). It typically involves using several demonstrations natural
2

---

types of informativenness such as the coefficient of determination (Xu al., 2020). Ethayarajh al. (2022) extended this framework by framing dataset or instance difficulty as the lack of V-usable in- formation. A high PVT estimate indicates that the input is well represented in the model, and thus the instance is considered to be easier for the model. A low PVI estimate indicates that the input contains little information, so the instance is considered to be harder. PVI allows us to compare the hardness of different instances, or the hardness of subsets by computing the average PV1 over the instances. Although it is recently proposed metric, PVT has received considerable attention and has proven be effective in various tasks. used as qual- ity estimate of Universal Dependencies treebanks in Kulmizev and Nivre (2023). PVI is used to sc- lect synthetic data as an augmentation to an intent detection classifier, which achieves state-of-the-ar performance (Lin et al., 2023). Chen et al. (2022a) and Prasad et al. (2023) incorporate PVI into informativeness metric to evaluate rationales, and find that it captures the expected flow of informa- tion in high-quality reasoning chains.
is calculated as Equation (1):
C(z, v) = loga (v) + log2 (v) (I)
Given the observation that ICL resembles fine- tuning (Dai et al., 2022), log2 (lp and log2 9 (v) are the ICL approximations of log2 g g (y) and log, g' [x][() in Algorithm 1.
The calculation of the original PVI is based on sequence classification, i.e., the dimension the output space of g' and is the number of unique labels in a task. Different to the original PVI, the calculation of in-context PVI is based on generation, where the output is sequence of tokens such ["un", "acceptable")) Instead of asking the model to produce label prediction such as "unacceptable", we matched a numerical index to label as the tar- get in p' and p (see A.2 for an example), so that the expected output is a single token, which makes the calculation of log2 (v) and log2 G (p) eas- ier.
4 Experiment settings
Dataset We conducted experiments on 7 datasets, including BoolQ (Clark et al., 2019), CoLA (Warstadt et al., 2019), MultiNLI (Williams et al., 2018), SNLI (Bowman et al., 2015), RTE (Wang et al., 2019), and two domain specific datasets- Health Advice (Li et al., 2021), and Causal Lan- guage (Yu et al., 2019). The selected datasets cover question answering. natural language inference and reasoning in both general and domain specific set- tings. We included the three tasks that are used in Ethayarajh et al. (2022), i.c., CoLA, MultiNLI, and SNLI BoolQ and RTE are two commonly used datasets from SuperGlue (Wang et al., 2019). Health Advice and Causal Language involve fun- damental tasks in the biomedical domains, which require domain specific knowledge. The details of the datasets are shown in Tables 10, and 11 in A.
3 Method
Algorithm I shows the calculation of PVI, which involves fine-tuning a model G on two different datasets.
Algorithm The calculation of PV1
I: Input: a dataset D, a model G, a test instance (x, v)
2: g' < fine-tune G on DD
3: < fine-tune G on {(0, ( v.) E D}
4: PVI(a, v) <- log, g (v) + loga g (x)
In Algorithm 1, g' is fine-tuned on D, i.e., the input- target pairs [(r;; (z)) v:) E D). and is fine- tuned null-target pairs {Ø, 9:)l(zz 31) E D} (2 is empty string). PVI is interpreted as the information gain when an input is provided to fine- tune G.
Model We tested models with sizes varying from 125M to 175B: GPT2-125M (Radford et al., 2019), GPT-Neo-1.3B. GPT-Neo-2.7B (Gao et al., 2020),
GPT-J-6B, GPT-JT-6B (Wang and Komatsuzaki, 2021), Alpaca-7B, Alpaca- 13B (Taori et al., 2023), and OpenAI text-davinci-003 (GPT3-175B). We limited the use of GPT3-175B to experiments on CoLA, RTE, and the first 500 test instances in MultiNLI and SNLI due to cost considerations.
In-context pointwise V-usable information (in-context PVI) is adapted from the original PVI. Instead of fine-tuning model G, two few-shot prompts, i.e., an input-target prompt // = (I 42, 1/2 In: Un, ..) and null-target prompt = (0,y1, Ø, y2, ..., Ø, yn, Ø), are used to prompt G. In-context PVI, denoted as C(x,U).
Exemplar selection We used sets of exemplars which were randomly selected from the training
3

---

dataset
accuracy
low
high
for TRUE
PVI for FALSE
CoLA 0.7242
0.0000
1,0000
1.1640
-7.6833
MultiNLI
0.7322
0.0000
0.9738
2.5702
-3.5963
RTE
0.7401
0.0417
0.9345
1.7554
-3.9994
SNLI 0.6704
0.0000
0.9967
2.5118
-4.7711
Table I: prediction accuracy, the average prediction accuracy instances with lowest 20% in-context estimates (acc. low PVT), the average prediction accuracy for instances with the highest 20% in-context Pvr estimates (acc. high PVT), average in-context PVI estimates for correct predictions (PVI for TRUE), and average in-context PVI estimates for incorrect predictions (PVI FALSE) for runs using GPT3-175D Statistics are averaged over the results obtained using sets of exemplars. Please refer to Table 14 in A.3 for more results.
set. See A.2 for examples the prompts we used. Number of shots We tried different numbers of shots. The minimum number of shots is equal to the number of unique labels in dataset. Apart from that, we also tried number of shots that is twice the minimum number of shots.? For instance, there
and bottem 50th percentile PVI Top and bottom 20th percentile PVI
0.4
Accuracy difference
0.2
are three unique labels in MultiNL., so we tried and 6-shot on it. We did not consider methods that scale up in-context exemplars such as structured prompting (Hao et al., 2022), since we want to test in-context PV1 in typical in-context setting, i.c., few-shot setting (Dong et al., 2022).
GPT2-125N
GPTitee 38
GPT-Re0-2.75 7B
GPT 4-88
OFT ds
Alpaca 7B
Alpaca 138
Figure 2: The difference prediction accuracy between instances with the top 20th/50th percentile and bottom 20th/50th percentile in- in-context PVI estimates. Statistics averaged the results obtained using datasets, 3 sets of exemplars, and 2 numbers of shots.
5 Results
Table provides overview of in-context PVI esti- mates across datasets, along with their relationship with prediction accuracy. The prediction accuracy estimates is
variable
avg.
med.
0.6
0.3
Exemplars
0.55
0.71
39.4%
22.9%
for instances with low in-context lower than that for instances with high in nn-oonextt PV1 estimates, and the average in-context PV1 esti- mate for correct predictions is higher than that for incorrect predictions. Note that the interpretation of in-context PVI is relative, which means that single value on its own is not as important as com- paring range of in-context PVI estimates. Please refer Table 14 in A.3 for more results.
Shots
0.44
0.74
57.6%
29.9%
Table 2:
The average (avg.) and median (med.) Pear- correlation coefficients of in-context estimates obtained using different sets of exemplars (Exemplars) and numbers of shots (Shots). We also report the per- centages of cases where there is strong correlation (r >> 0.6) and weak correlation (r < 0.3).
The results show that there is a moderate to strong correlation between in-context PVI estimates obtained using different sets of exemplars, with an average Pearson correlation of 0.55 and me- dian of 0.71. In fact, 59.4% of the cases showed strong correlation with Pearson coefficient above 0.6. We observed stronger correlation between in-context PYT estimates across different numbers of shots, with an average Pearson coefficient of 0.44 and a median of 0.74. Nearly 60% of the cases showed Pearson coefficient over 0.6, which suggests substantial level of consistency."
Figure 2 shows the difference in prediction ac- curacy between instances with the top 20th/50th percentile and bottom 20th/50th percentile of in- context PVI estimates. We observed an increase in accuracy gain when model scales with one excep- tion (GPT2-125M).
5.1 The consistency of in-context PVI
Table shows the consistency in-context PV1 es- timates across different selections of exemplars and numbers of shots, which are two essential features for in-context learning.
*Please see full results in https://github. http:/gtttubccoo/oo/ cww. LKLLb/ in- context-pv pvi for the corresponding p-values of the statis- tics in Table
"Since CoL A contains only 2 labels and the input sentences are short, we tried 4- and 8-shot for it.
4

---

Geritte- 133
Grr-teo 78
Overall our results point towards considerable degree of consistency in in-context PV1 estimates across different exemplar selections and numbers of shots. There are no sigmificant changes in in- context PVT estimates given different sets of exem- plars, which is critical for the practical application of in-context PVI. We also observed that in-context PVI estimates are much more consistent among closer models, i.e., those having similar architec- tures, training data, etc.
GPT2 PPT2-125M 325M
DPT-J488
GM1-1T-68
Alpaca- 78
Apaca-1 138
GITT 1758
GP12-125M
10
0:98
0:97
0.02
0.05 0.13
G.T-NoD-1.38
0.98
1.0
0.97
0.01
0.11
0..8
GPT-Neo-2 78
0.97
0:97
1.0
0.96
0.89
0.04
0.04
-0.1
CFT-)-68
0.96
0:95
0.96
1.0
0.80
0.01
0.02
0.09
GPT-JT-68
0.9
0.89
0:89
0.88
1.0
0.21
0.19
-0.0
0.4
coofficiont
Alpeca-22 70 10.02
0.01
0.04
0.01
0.21
1.0
0.74
0.28
Alpaca 138
.0.5
-0.0
0.04
0.02
0.19
0.74
1.0
0.34
5.2
The variance of in-context PVI estimates across different sets of exemplars
GPT3-1758
0.11-00.11 -0.1
-0.09
-0.0
0.28
0.34
1.0
0.0
We used one-way analysis variance (ANOVA) examine the variance of in-context PVI estimates across different exemplar selections. Our hypothe- sis is that in-context PVT estimates obtained using different sets of exemplars have similar means. Ta- ble 3 shows part of the results (please refer to A.5
CoLA
GPT-Nin 3B
OPT Noo-No--2a 2.18
GPTZ 123M
GIT Kia
GFF-F-eiB
Alpaca- 7B
Alsnca- 13B
1000
GPT2-125M
1.0
0.75
0.05
0.57
0.8
0.82
0.09
0.75
GPT-Neo-1.38
0.75
1.0
0.54
0.93
0.32
Pearson correlation coefiiiinn
for the rest of the results):
0.50
GPT 2.78
0.05
0.54
3.0
0.78
0.53
0.01
0..4
0.25
dataset
model
F-statistic
p-value
J 68
0.08
GPPT2-125M
0.2958
0.7634
GPT-neo-1.33 1.3B
0.5543
0.6239
GPT-JT-08 0.8
0.93
0.51
0.91
3.0
0.7
0.36
GPT-neo-2.7B
0.0383
0.9629
0.25
MultiNLI
GPT-J-6B
0.2604
0.7866
Alpaca-115 -4.82
0.67
GPT-JT-6B
0.3099
0.7545
-0.50
Alpaca-7B
0.0363
0.9647
Alaaca1138 -4.99
0.32
-0.11
0.08
0.36
0.67
1.0
Alpaca-13B
0.4304
0.6850
-0.75
Table 3:
The results of one-way ANOVA for the variance
Figure 3: The Pearson correlation coefficients among in-context PVI estimates made by different models for SNLI and CoLA instances. In-context PVI estimates obtained using 4-shot prompt for CoLA and a 6-shot prompt for SNLI.
of in-context estimates of MultiNLI instances across different sets of exemplars.
The results reveal that there are no statistically sig- nificant differences (p-value > 0.05) in in-context PVI estimates obtained using different sets of exem- plars. F-statistics also show that the between-growe group difference of in-context PVI estimates among the three sets of exemplars is smaller than the differ- ence within each set. other words, in-context PVI estimates are quite stable.
Figure 3 shows the Pearson correlation coeffi- cients between in-context PVI estimates for CoLAA and SNLI instances across different models. We noted that in-context PVT estimates made by GPT- Nco-1.3B, GPT-Neo-2.7B. GPT-J-6B, and GPT-JT- 6B, and those made by and Alpaca-7B and Alpaca- 13B are generally much more consistent than, for example, those made by GPM and Alpaca- 13B (see A.4 for more examples). This is probably due to certain models being closer to one another in terms of their architecture, training data, etc. Also, instances that challenging for smaller models may not pose similar challenges to larger models (Srivastava et al., 2022; Wei et al., 2022a), which explains the inconsistency between in-context PVT estimates made by larger models, such as GPT3- 175B, and those made by smaller models.
5.3 In-context PVI threshold for incorrect predictions
Figure displays the density distribution of in- context PVI estimates made by the smallest and largest model we tested, i.e., GPT2-125M and GPT3-175B, for correctly and incorrectly predicted instances. Statisties in Figure 4 are based on in- context PVI estimates obtained using one randomly selected set of exemplars, and in-context mates made by GPT2- 125M were obtained using more exemplars than those made by GPT3-175B. Similar to what Ethayarajh et al. (2022) report,
esti-
5

---

high in-context instances are more likely to be predicted correctly while low in-context pv1 instances are not, and the threshold which in- stances start being predicted incorrectly is similar across datasets. However, in-context PVT estimates made by GPT2-125M (see Figure 4a) are much noisier than those made by GPT3-175B (see Figure 4b). This suggests that in-context PVI estimates made by larger models better capture information that is useful to produce the correct predictions.
5.4 Correlation with inter-annotator agreement
MultiNLI (Williams et al., 2018) and SNLT (Bow- man et al., 2015) both contain annotations made by five annotators. The difference in human anno- tations can be viewed hardness: the less the annotators agree, the harder an instance is. We adopted straightforward mea- sure of agreement, variation-ratio (Freeman, 1965), and measured inter-annotator agreement as the fre- quency of the most frequent annotation among the five annotations.
indicator of instance
correct
incorrect
dataset
p-value
BoolQ
MultiNLI 0.3240
0.01
SNLI 0.3350
0.01
CoLA
Table 4: The Pearson correlation coefficients (r) he- tween the original PVI estimates and inter-ane ann-ttont agreement, and the corresponding p-values.
NulsiNLI
Table shows the Pearson correlation coeffi- cients between the original PVT estimates and inter- annotator agreement, which reveals positive cor- relation between the original PVT estimates and inter-annotator agreement. Although the hardness of instance for humans is not necessarily equiv- alent that for given model, there should be a positive correlation between them. Thus, the results shown in Table 4 are considered reasonable, which show weak but positive correlations (r = 0.3) MultiNLI and SNLI. A weak positive correlation between PVI estimates and annot-ann agree- ment can be viewed as sign of reliability.
RTE
Density
SNLI
Health Advice
Causal Language
(a) GPT2-125M
CoLA
Figure 5 shows the Pearson correlation coeffi- cients between in-context PVT estimates and inter- annotator agreement. Most of the estimates made by smaller models (with less than 7B parameters) have a negative correlation with inter-annotator agreement. However, even for the smaller models, the correlation becomes more positive as the num- ber of shots increases. A positive correlation be- tween in-context PV1 estimates and inter- inter-annotator agreement seems to be associated with model scale, as positive correlations cases for models larger than 7B.
Mu tiNL
Density
SNU
6
1
2
observed in most of the
In-context PVI
(b) GPT3-175B
Table shows the Pearson correlation coeffi cients between the original PVI estimates made by BERT, and between in-context PV1 made by GPT3- 175B and inter-annotator agreement. It shows that in-context PVI estimates made by this 175B model
Figure 4: The density distribution of in-context PVI
estimates made by GPT2-125M and GPT3-175B for correctly and incorrectly predicted instances in terms of datasets. GPT3-175B results for MultiNLI and SNLI based the first 500 test instances.
"The original PVI estimates are taken from https:// github. con/ kwwike/wiieeeee datastt_diiffiullt.
6

---

Pearson correlation coefficient
created by some linguistic cues. Certain sentences may contain noticeable advice indicators (such as "should be further explored," as exemplified in the first example of Table 6) which aim at suggesting the need for subsequent investigations. However, based on the annotation schema, sole suggestions pertaining to health-related practices or policies are considered as advice, whereas suggestions for further studies do not fall within the category of it. addition, according to the annotation schema, claim that suggests the advantages or effective- ness of an intervention is categorized as "weak advice" or "strong advice". However, terms like "effective" also appear when reporting experimental results, and experimental results are not considered to be advice according to the annotation schema of Health Advice, which makes the instance chal- lenging for the model, such as the second example in Table 6. In Causal Language, some instances contain causal markers such as "association" subordinate clauses, which do not suggest any re- lationship according to the annotation schema and thus cause confusion. This is evident from the third and fourth examples in Table 6. These patterns identified through in-context PV1 in Health Advice and Causal Language align with the typical predic- tion error patterns observed in previous studies (Yu et al., 2019; Li al., 2021). See also Tables 17, 18, and 19 A.7 for the most challenging instances in each category in MultiNLI and SNLI identified by in-context PVI estimates made by GPT3-175B.
MuitiNLI
SNU
0.2
3-shot
6-shot
-0.1
0.2
0.1
13
0.1
13
Model scale (billions parameters)
Figure 5: The average Pearson correlation coefficients between in-context PVI estimates and inter annotator agreement for runs on MultiNLI and SNLI using dif- ferent models and the number of shots. Statistics averaged over the results obtained using sets of exem- plars, Corresponding p-values are shown in Table 16 in A.6.
for MultiNLI are very close to the original PVI es- timates, despite that they are obtained using only 3-shot prompts. However, in-context PVT estimates made by GPT3-175B for SNLI are less correlated with annottarnttator agreement compared to the original PVI.
dataset
model
p-value
BERT
0.2316
< 0.01
MulGNLI
GFT3-175B
0.2217
< 0.01
BERT
0.3678
< 0.01
SNLI
GPT3- 175B
0.1732
< 0.01
Table 5: The Pearson correlation coefficients (r) be- tween the original PVI estimates (made by BERT) and inter-annotator agreement, and between in-context PVI (made by GPT3-175B) and inter-annotator agreement. Results are based on the first 500 test instances in MultiNLI and SNLI Statistics regarding GPT3-175B are calculated using in-context PVI estimates obtained using 3-shot prompts and are averaged over the results obtained using sets exemplars.
5.6 Exemplar selection using in-context PVI
We conducted preliminary experiments to evaluate the effectiveness of in-context PVI in selecting ex- emplars that enhance performance. Our approach simple and straightforward: based on in-context PVI estimates obtained using one set of randomly selected exemplars, we selected the most challeng- ing training instance from each category in CoLA, MultiNLI, and SNLI as the exemplars. Intuitively, hard instances are better exemplars, since a model is already proficient in dealing with easier exam- ples, which makes easier examples less valuable for a demonstration. The exemplars we used are shown in Tables 17, 18, and 19 in A.7.
In a nutshell, in-context PVI estimates made by larger models and larger numbers shots are more reliable in the sense that they tend to have a more positive correlation with inter-annotator agreement.
5.5 In-context PVI for challenging instances
To investigate the potential of in-context PVI in identifying challenging instances, we performed a qualitative analysis of Hcalth Advice and Causal Language. A.1 shows more details of the tasks. The results indicate that the most challenging annotations often fall into the class of "no advice" (Health Advice) and "no relationship" (Causal Lan- guage). This is primarily due to the confusion
Table 7 indicates that models utilizing the hard- instances as exemplars perform slightly bet- ter than those using randomly selected ones for MultiLI and SNLL. However, this approach leads to a decrease in performance for CoLA. We specu-

---

example
target
PVI
Supplementation with I., reuteri 6475 should be further explored novel approach prevent age age-associated bone loss and ostcoporosis.
no advice
-3.6240
In patients with active ophthalmopathy, reprotumumab placebo in reducing proptosis and the Clinical Activity Score.
effective than
no advice
-3,4560
This work has important policy implications for public health, given the continuous nature the BMI-IHD association and the modifiable nature of BMI.
no relationship
-5,1582
The observed associations between pre dee-iggnostic serum GGT and different breast cancer subtypes may indicate distinct underlying pathways and require further investigations to tease out their clinical implications.
no relationship
-4.9277
Table 6: Examples of challenging instances in Health Advice and Causal Language identified by in-context PVI. Expressions that may lead to confusion are bolded and marked in red.
dataset
random
hardest
Compared to the original PVI, our proposed method significantly reduces computational cost while be- having similarly to it. We showed that in-context PV1 estimates are consistent across different mod- els, selections of exemplars, and numbers of shots. We discovered that larger models tend to produce more reliable in-context PVT estimates, suggesting that in-context PV1 is not strict replacement for the original PVI, and that for smaller models (espe- cially for those that are readily "fine-tunable", the original PVT is a better choice. We demonstrated that in-context PVI helps identify challenging stances. We also presented a preliminary analysis on the potential of in-context PVI to enhance model performance through exemplar selection.
CoLA
0.7571
0.7230
MultiNLI
0.7180
0.7380
SNLJ
0.6700
0.6980
Table 7: The accuracy GPT3-175B using randomly selected exemplars (random) and the hardest training instances (hardest) as the exemplars. MultiNLI and SNLI results are based on the first 500 testing instances.
late that this is due to some exemplars being misla- beled, which can be misleading to the model. Table 8 shows two of the most challenging instances in CoLA, which used as the exemplars. At least for the authors of this paper, the first example is not linguistically acceptable, while the second acceptable.
The utility of in-context PVI for discerning in- stance difficulty may be effectively harnessed for dataset selection in joint learning scenarios, facili- tating the construction of balanced and optimized training sets. Concurrently, future work may utilize in-context PV1 as a tool to design an efficient cur- riculum learning protocol for LLMs (Bengio et al., 2009; Hacohen and Weinshall, 2019). Developing an adaptive curriculum for ICL., progressing from easier to more difficult instances, as gauged by their in-context PVI estimates, represents significant direction. A further avenue of exploration lies in developing an efficient training method that strate- gically prioritizes challenging instances, as iden- tified by in-context PVI, thereby creating focused mini-batches for training LLMs. This approach could potentially provide both computational sav- ings and enhance learning efficacy. Achieving these objectives, however, necessitates addressing series of research questions, including optimiz- ing joint learning dataset composition and creating algorithms that effectively prioritize difficult in- stances in the training process. These questions constitute the main focus of our future research.
sentence
label
The harder it has rained, how much flster a flow appears in the river?
acceptable -13.37
John wrote books.
unacceptahle
-11.26
Table 8: The hardest instance in each category in Col.A, determined by in-context PVI estimates obtained using GPT3-175B. Examples in red considered to be mis- labeled.
Our findings demonstrate that hard instances can indeed enhance model performance in some cases. However, to fully leverage the power of in-context PV1 for selecting exemplars that enhance model performance, it is imperative adopt more so- phisticated approach. Moreover, we identified chal- lenging instances based on in-context PVT obtained using only one set of randomly selected examplars. Future work should take into account prompt varia- tions.
6 Conclusion
This paper introduces in-context PVL, an alterna- tive approach to access the hardness an instance.
8

---

Limitations
Hanjic Chen, Faeze Brahman, Xiang Ren, Yangfeng Ji, Yejin Choi, and Swabha Swayamdipta. 2022a. Rev: Information-theoreicc evaluation of free-text rationales. arXiv preprint wrXiv:2200.0495.
A theoretical analysis is still needed to enhance our understanding and validation of in-context PVI. order manage costs, we limited our experi- ments with OpenAT text-davinci-003 (GPT3-175B)
Shan Chen, Yingya Li, Sheng Lu, Hoang Van, Hugo JWL Aerts, Guergana K Savova, and Danielle S Bitterman. 2023. Evaluation of chatgpt family of models for biomedical reasoning and clas- sification. arXiv preprint 230.04630.
to CoLA, RTE, and the initial 500 test instances in MultiNLI and SNLI. Due to the closed source nature of GPT3-175B, the reproducibility of the results related to it may be concern as well.
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKe- own, and He He. 2022b. On the relation between sensitivity and accuracy in in-context learning. arXiv preprint arXiv:2209.0766)
Acknowledgements
The authors would like to thank the anonymous reviewers for feedback that improved the pa- per, the funding provided by the LOEWE Distin- guished Chair "Ubiquitous Knowledge Processing" (LOEWE initiative, Hesse, Germany), US National Institutes of Health (NTH) grant 5R01 GMI 1435, and the Woods Foundation. Any opinions, find- ings, conclusions, or recommendations expressed this material are those of the authors and do not necessarily reflect the views of NIH or the Woods Foundation.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint 02X3v1224..02311.
Hyung Won Chung, Le Hou, Shayne Longpre. Bar- ret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling instruction- finntuned language models. arXiv preprint 22Xh::2110.14111
Christopher Clark, Kenton Lee, Ming-W Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT, pages 2924-2936.
References
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma. and Denny Zhou. 2022. What learning algorithm is in-context learning? investigations with linear madels. arXiv preprint arXiv:221 2211 15661.
Peng Cui, Dan Zhang, Zhijie Deng, Yinpeng Dong, and Jun Zhu. 2023. Learning sample difficulty from pre- trained models for reliable prediction. arXiv preprint arXiv:2344.1012
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international confer- ence machine learning, pages 41-48.
Damai Dai. Yutao Sun, Li Dong. Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:22/2.10559,
Samuel R. Bowman, Gabor Angeli, Christopher Potts. and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv prepriat arXiv:2301.0023:
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022. Understanding dataset difficulty with mathcule-usable information. International Conference on Machine Learning, pages 5988 6008. PMLR.
Tom Brown Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Linton Freeman. 1965. Elementary applied stafistics: for students in behavioral science. New York: Wiley.
Ting- Yun Chang and Robin Jia. 2023. Data curation alone can stabilize in-context learning. Proceed- ings of the st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8123-8144 1144-144.. Toronto, Canada. Association for Computational Linguistics.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smymis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. 2023. Datacomp: In search of the next generation multimodal datasets. arXiv preprint arXiv:2304 14108.

---

Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling, arXiv preprint arXiv:2101.0022
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Guy Hacohen and Daphna Weinshall. 2019. On the power of curriculum learning in training deep net- works. In International Conference on Machine Learning, pages 2535-2544. PMLR.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. Rissanen data analysis: Examining dataset charac- teristics via description length. In International Conference on Machine Learning, pages 8500-8513. PMLR.
Michael Hahn, Dan Jurafsky, and Richard Futrell. 2021. Sensitivity as complexity measure for sequence classification tasks. Transactions of the Association for Computational Linguistics, 9:891-908.
Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal. 2023. Receval: Evaluating reasoning chains via correctness and informativeness. arXiv preprimt arXiv:2304. 10703.
Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yux- ian Gu, and Furu Wei. 2022. Structured prompting: Scaling in-context learning 1,000 examples. arXiv prepriat 2rXiv:2212.06713.
Alee Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in neural information processing systems, 35:22199- 22213.
Abulhair Saparov and He He. 2022. Language models are greedy reasoners: A systematic formal analysis of of-in-of-thought In The Eleventh International Conference on Learning Representations.
Artur Kulmizev and Joakim Nivre. 2023. Investigat- ing ud treebanks via dataset difficulty measures. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 1076-1089.
Philipp Spitzer, Joshua Holstein, Michael Vössing, and Niklas Kühl. 2023, On the perception of difficulty: Differences between humans and ai. arXiv preprint arXiv:230.009803
Yingya Li, Jun Wang. and Bei Yu. 2021. Detecting health advice in medical research literature. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6018- 6029, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao. Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXivv2206.0460
Yen-Ting Lin, Alexandros Papangelis, Seokhwan Kim, Sungjin Lee, Devamanyu Hazarika, Mahdi Namazi- far, Di Jin, Yang Liu, and Dilek Hakkani-Tuu Tur. 2023. Selective in-context data augmentation for intent de- tection using pointwise V-information. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 1463-1476, Dubrovnik, Croatia. Association for Computational Linguistics.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-followin model. Stan- ford Center for Research on Foundation Models. https://orfm. stanford. edu/2023/03/1 3/alpaca. himl, 3(6)77.
Antonio Torralba and Alexei A. Efros. 2011. Unbiased look dataset bias. In CVPR 2011, pages 1521- 1528.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-1 14, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint ar0iv:2302. 13971.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods natural language processing. ACM Computing Surveys, 15(5):1335.
Johannes von Oswald, Eyvind Niklasson, Ettore Ran- dazzo, Sacramento, Alexander Mordvintsev, An- drey Zhmoginov, and Max Vladymyrov. 2022. Trans- formers learn in-context by gradient descent. arXiv preprint 222
10

---

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. Advances in neural information processing systems, 32.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706. PMLR.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A 6 Billion Parameter Autoregressive Lan- guage Model. https: /gi www. com/kingoflolz/ mesh-trrnnffomer--J
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are fair evaluators. arXiv preprint 23X.::3305.1726.
Alex Warstadt, Amanpreet Singh, and Samuel Bow- man. 2019. Neural network acceptability judgments. Transactions the Association for Computational Linguistics, 7:625 641.
Jason Wei, Tay. Rishi Bommasani, Colin Raf- fel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Met- zler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Perey Liang, Jeff Dean, and William Fedus. 2022a. Emergent abilities of large language models. arXiv preprint Xix:2200600766
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. 2022b. Chain-of-thougg soning in large language models. Advances Neural Information Processing Systems, 35:24824-244 24837.
prompting elicits rea-
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long Pupers). pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stew- art, and Stefano Ermon. 2020. A theory usable information under computational constraints. arXiv preprint 2002 10689.
Bei Yu, Yingya Li, and Jun Wang. 2019. Detecting causal language
science findings. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Languuge Processing and the 9th Inter- national Joint Conference Natural Language Pro- cessing (EMNLP-IJCNI.P) pages 4664-4674, Hong Kong, China. Association for Computational Linguis- tics.
Xinran Zhao, Shikhar Murty, and Christopher Manning. 2022. On measuring the intrinsic few-shot hardness of datasets. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing. pages 3955-3963, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.

---

A Appendix
A.1
Experiment details
All experiments were run using the same configuration on either a NVIDIA A100 or GTX3090,
through OpenAl API. Tables 9, 10, and 11 give more details of the datasets we used.
dataset
domain
testing set size
type of task
Ethayarajh et al. (2022)
BoolQ
General
3270
Yes/no question and answering
No
CoLA
General
Acceptability
Yes
MulGiNI
General
10000
Natural language inference
Yes
RTE
General
277
Entailment
No
SNL.I
General
10000
Natural language inference
Yes
Health Advice
Biomedical 4784
Suggestion mining
No
Causal Language
Biomedical
2446
Causal relation
No
Table 9: Overview of datasets used in this study. These datasets cover variety of tasks and domains, providing comprehensive base for our analyses.
dataset
instunce
label
"Windows Movie Maker Windows Movie Maker (formerly known as Win- dows Live Movie Maker in Windows discontinued video editing software by Microsoft. is part Windows Essentials software suite and offers the ability create and edit videos well to publish them OneDrive, Facebook, Vimeo, YouTube, and Flickr."
PASSAGE:
true
BoolQ
QUESTION:"is windows movie maker part of windows essentials"
PASSAGE: "The Golden Compass (film) - 2011, Philip Pullman remarked the British Humanist Association annual conference that due the first film's disappointing in the United States, there would not any sequels made." "
false
QUESTION:"iS there sequel the movie the golden compass"
The sailors rode breeze clear of the rocks.
acceptable unacceptable
CoL.A
more does Bill smoke, the more Susan hates him.
PREMISE: Your gift is approciated by each and every student who will benefit from your generosity.
neutral
HYPOTHESIS: Hundreds of students will henefit from your generosity.
PREMISE: yes now you know if everybody like in August when everybody's on vacation MultiLII or something we can dress a little more casual.
contradiction
HYPOTHESIS: August is a black out month for vacations in the company.
PREMISE: Al the other end of Teansylvamia Avenue, people began to line for a White
House tour.
entailment
!lYPOTHESI:: People formed line the end of Pennsylvania Avenue.
PREMISE: This church choir sings to the masses they sing joyous songs from the book neutral church.
HYPOTHESIS: The church has cracks in the ceiling.
SNLI
PREMISE: A statue museum that
be looking
contradiction
HYPOTHESIS: Tons people are gathered around the statue.
PREMISE: A
with green headscari, blue shirt and very big grin. very happy.
entailment
HYPOTHESIS: The
PREMISE: Valero Energy Corp., on Monday, said found "extensive" additional damage
250,000 Maro-bbrrel-pppe perrel-per-day Port Arthur refinery.
entailment
RTE
HYPOTHESIS: Valero Energy Corp. produces 250,000 barrels per day.
PREMISE: prices fall back Yukos oil threat lifted. Oil prices rise.
no entailment
HYPOTHESIS:
Table 10: Instance examples of BoolQ, CoLA, MultiNLI RTE, SNLL Health Advice, and Causal Language.
12

---

dataset instance
label
Nurses should assess patient decision decision-making styles to ensure maximum
patient involvement in the decision-men makinn process based on personal desires
strong advice
regardless of age.
Health Advice
Adolescents with high risk factors, especially those with menstrual disorders and peraaddrogenn,m may need careful clinical screening
weak advice
Former smokers prevalence of overweight and obese subjects in this group.
risk for hypertension, probably because of the higher
no advice
The findings from this large prospective study show that men are taller
and who have greater adiposity have an elevated risk of high-grade prostate cancer and prostate cancer death.
correlational
MIHIF A1298C polymorphism might contribute an increased risk breast cancer and/or ovarian cancer susceptibility.
Causal Langauge
conditional causal
Participatory community-
nutrition education for caregivers improved
direct causal
child dietary diversity even a food insecure area.
This approach may, however, be difficult to implement on large scale.
no relation
Table 11: Continuation of Table 10.
A.2 Example of prompts
Table 12 and Table 13 are examples of the prompts we used. Our prompt design can be cross-validater with Chen et al. (2023).
CONTEXT: How much harder has it rained, the faster flow you see in the river?
QUESTION: this (0) unacceptable, or (1) acceptable?
ANSWER: 1
CONTEXT: The more obnoxious Fred, the less attention you should pay to him. QUESTION: Is this (0) unacceptable, or (1) acceptable?
ANSWER: 0
CONTEXT: I'm glad I saw anybody.
QUESTION: this (0) unacceptable, or (1) acceptable?
ANSWER: 0
CONTEXT: Julie and Jenny arrived first
QUESTION: Is this (0) unacceptable. or (1) acceptable?
ANSWER: 1
Table 12: An example of 4-shot input-target prompt for CoLA.
ANSWER:
ANSWER: O
ANSWER: O
ANSWER:
Table 13: example a 4-shot null-target prompt for CoLA.
13

---

A.3 More general results
Table 14 shows a broad overview of in-context PVI estimates across datasets and models, along with their relationship with prediction accuracy.
dataset
accuracy
low
high
for TRUE
PVI for FALSE
BoolQ
0,4543
0.3173
0.6241
0.2009
-0.1075
CoLA
0.5175
0.3374
0.6586
0,2416
-0.1486
MultiNLI
0.3483
0.2126
0.4691
0,6609
0.1094
RTE
0.4482
0.3330
0.5287
0.8452
0.7243
SNLI
0.3426
0.2715
0.3929
0,5872
0.2482
Health Advice
0,3027
0.1874
0.4336
0,4162
0.0101
Causal Language
0,2277
0.1321
0.3499
0,5136
-0.1918
(a) dataset-wise
model
accuracy
acc. low PVI
scc. high PV1
PVI TRUE
PVI for FALSE
GPT2-125M
0.3753
0.2077
0.5523
0.7691
0,2888
GPT-Neo-I.3B
0.3610
0.3077
0,3930
0.1842
0.0808
GPT-Neo-2.7B
0.3672
0,3141
0.4137
0,3880
0,2858
GPT-J-6B
0.3775
0,3553
0.4199
0.1292
GPT-JT-6B
0.3811
0,2792
0.4555
0,2156
GPT3-175B*
0.7167
0.0104
0.9763
2.0003
-5.0125
Alpaca-7B
0.3791
0.1414
0.6018
0.6852
40,2365
Alpaca-153 13B
0.3975
0.1847
0.6077
1.4344
0.4778
(b) model-wise
Table 14: The average prediction accuracy, the average prediction accuracy for instances with the lowest 20% in-context PVI estimates (acc. low PVI), the average prediction accuracy for instances with the highest 20% context PVI estimates (acc. high PVI), average in-context PVI estimates for correct predictions (PVI for TRUE), and average in-context PVI estimates for incorrect predictions (PVI for FALSE) for runs using different datasets or models. Statisties are averaged over the results obtained using 3 sets of exemplars, and numbers of shots. GPT3-175B was only tested on CoLA, RTE, and the first 500 test instances in MultiNI. and SNLI.
14

---

A.4 More the consistency of in-context Figure shows the Pearson correlation coefficients between in-context PVI estimates across different models.
GPP2 125M
CPPT GIPT-Nec
C2 OPT-Nee- 78
CPP 448
OPT/I-BB
Alpaca 7B
Alpacaa 23B
GPT2 112--22M
Gny-ee 3B
COPT WoriiNn 2.78
GP
GFF-1-88
Albaca-
Alpaca- 13B
1.0
GPT2-125M
1.0
0.63
0.55
0.61
0.62
0.28
-0.53
0.75
GPT2-125M
1.0
0.96
0.03
0.49
0.04
-0.36
-0.59
0.8
C0rroon comee Cometation codfficier
GFT 1.33
0.61
1.0
0.97
0.97
8.96
0.41
0.87
0.6
0.50
0.55
0.97
0.92
GFI-Nec-2.70
0.83
0.86
1.0
0.87
0.98
0.13
0.11
6.2
68
G8
0:40
0.51
0.67
1.0
0.83
0.55
0.29
0.00
GPT-/T688
0.02
0.90
0.99
0.90
1.0
0.41
0.85
GPT,
-0.25
A6peca-78
0.28
0.41
0.38
0.39
0.41
1.0
0.18
Alpaca-78
76
-0.38
-0.36
0.13
0.35
0.96
1.0
0.83
Alpaco-138
138
-0.50
.0.58
.0.17
0.29
-0.2
0.83
3.0
BoolQ
MuitiNLi
GPT2 125M
GF GPt/tee-- 3B
GIPT APF-lee-22
GPT 0PT-3-80
GPr (T44
Alpacca 7B
Ajaca1115
GPT3
OP72 1251
OPT MiM 30
GPF Neo
GPT j-B
CPP 4B
Alosca 7B
Alpect 138
1.0
GPT2-125M
1.0
0.09
89
0.9
0.3
0.23
0.32
0.33
GPF2-125M
1.0
0.53
0.93
00.90
40.10
0.37
0.93
Ncc 1.38 1.38-0.69
-0.71
0.23
0.6
GFTNNo-1..3 1.36
0.63
1.0
0.86
0.44
0.61
+0.
0.37
PeRSSO COT oorelation coeffic
GPT-NeD-2.70
0.24
0.29
0.1
6B
0.97
0.71
0.98
1.0
0.29
0.24
0.331
0.36
0.2
GPT
-0.96
40.44
+0.82
0.38
-0.7
-0.97
0.00
IT 68
0.0
GPT JT
5.160.16
0.61
0.19
0.38
1.0
.0.67
.0.43
-0.25
Appcce70
Alpaca- 138
-0.32
0:23
00.29
-0.31
0.01
0.35
1:0
0.16
-0.4
0.57
.0.1
0.31
40.7
-0.67
1.0
0.74
-0.50
GFT3 175B
4.33
-0.36
.0.33 0.16
Alpaca
0.37
.0.43
0.74
-0.75
RTE
SNLI
CPT Netto 38
GFT Nelo 76
Grr-Nee- 38
GIT Noc 2.78
0PT2-123M
GPT CP7--78-88
Afaaca 7B
Appacc 33
GPY2 223M
Alsaca-T 7B
Alpaca 39
Carn 48
GFT:448
1.0
1.0
GPT2-125M
0.15
GPT2-125M
0.8
0.8
CPT-Nec-1.30
0.98
2.0
1.0
0.92
0.98
0.14
0.48
Conn UDX/PAJ
GPT-Neo-1.30
0.06
1.0
0.99
0.99
0.85
0.22
0.58
PO0TOn corro 2ton coeffice
0.6
GFT 2.78
0.96
1.0
1.0
0.92
0.98
0.14
00.48
0.4
Get
0.15
0.99
1.0
0.92
0.82
-0.14
00.57
0.4
GPT:)68
0.2
GPT--68
0.14
0.2
GPT.JT-8
0.93
0.90
0.90
0.9
1.0
-0.11
<0.42
0.0
GPT.TT68
0.17
0.05
0.02
0.8
1.0
0.06
-0.3
0.0
Alpaca-70
0.15
.0.14
.0.14
.0.13
4.21
3.0
0.34
0.7
Alpaca
0.1
.0.11
.0.34
00.06
0.06
1.0
0.43
-02
Alpsca-33B
Apaca-138
0.25
1.0
-0.4
Health Advice
Causal Language
Figure 6: The Pearson correlation coefficients among in-context PVI estimates made by different models. In-context PVI estimates are obtained using prompts that contain the minimal number exemplars, i.e., the number of unique labels in each dataset. CoLA is exception. The minimal number of exemplars for CoL.A is 4, since it contains only 2 labels and the input sentences are short.
15

---

A.5 More the variance of in-context Table 13 shows the full results of one-way ANOVA for the variance of in-context PVI estimates across different sets exemplars.
dataset
model
F-statistic
p-value
GPT2-125M
L0405
0,4537
GPT-Neo-1.3B
0.2433
0.7981
GPT-Neo-2.7B
0.5357
0.6325
BonIQ
GPT-J-6B
0.1484
0.8680
GPT-JT-6B
0.1978
0.8305
Alpaca-7B
0.9711
0.4729
Alpaca-
0.3944
0.7046
GPT2-125M
0.3024
0.7592
GPT-Neo-1.3B
0.8949
0.4957
GPT-Neo-2.78
0.1543
0.8634
CoLA
GPT-J-6B
2.3202
GPT-JT-6B
0.1690
0.8520
Alpaca-7B
1.8533
0.2986
Alpaca-133
1,0781
0.4438
GPT2-125M
0,2958
0.7634
GPT-Nen-1 1,38
0.5543
0.6239
GPT-Neo-2.7B
0.0383
0.9629
MultiNLI
GPT-J-6B
0.2604
0.7866
GPT-JT-6B
0.3099
0.7545
Alpaca-7B
0.0363
0,9647
Alpaca- 13B
0.4304
0,6850
GPT2- 125M
0.4876
0.6556
GPT-Neo-1.33 1.3B
0.0803
0.9247
GPT-Neo-2.7B
0.7250
0.5535
RTB
GPT-J-6B
1,2598
0.4007
GPT-JT-6B
1.5383
0.3468
Alpaca-7B
0.4289
0.6857
Alpaca-13B
0.2346
0.8042
GPT2-125M
0.3518
0.7290
GPT-Neo-1.3B
0.3369
GPT-Neo-2.7B
3,0431
0.1897
SNLI
GPT-J-6B
0.7946
GPT-JT-6B
62,8332
0.0036
Alpaca-7B
0.1426
Alpaca- 13B
0.0905
0.9158
GPT2-125M
0,3786
0.7135
GFT-Neo-1.3D
5.7712
0.0937
GPT-Nco-2.7B
0,0386
0.9626
Health Advice
GPT-J-6B
1,3833
0.3752
GPT-JT-6B
0,4463
0.6766
Alpaca-7B
0,1852
0.8398
Alpaca- 1 13B
0.6809
0.5704
GPT2-125M
0.1922
0,8345
GPT-Neo-1.3R 1.38
36.8063
0.0077
GPT-Neo-2.7B
12.0928
0.0367
Causal Language
GPT-J-6B
0.4149
0.6933
GPT-JT-6B
1.6823
0.3236
Alpaca-7B
0.0084
0,9917
Alpaca-13B
1.3805
0.3758
Table 15: The results one-way ANOVA for the variance of in-context PVI estimates across different sets exemplars. p-values that smaller than 0.05 bolded, which suggest that there significant differences in in-context PVI estimates obtained using different sets of exemplars.
16

---

A.6 More the correlation between in-context PVI and inter-annotator agreement
dataset
model
number of shots
p-value
3 -0,1412
4.61E-34
GPT2-125M
6
-0.0416
2.83E-01
-0.1590
2.44E-46
GPT-Neo-1.3B
6
-0.0096
1.22E-01
-0.1630
7.18F-46
GPT-Neo-2.7B
-0.0780
1.08E-18
-0.1782
9.74F-68
MultiNLI
GPT-J-6B
-0.1725
2.26E-59
-0.1577
7.62F-41
GPT-JT-6B
0.0798
3.56E-13
-0.0271
5,28E-04
Alpaca-7B
0.0817
7.20E-02
0.1183
6,87E-08
Alpaca-13B
0.1656
6,27E-23
-0.0656
2.68F-48
GPT2-125M
0.0230
2.36E-02
-0.1376
2.19E-03
GPT-Nco-1.3B
-0,0423
L82E-42
-0.1097
1.49E-17
GPT-Nco-2.7B
-0.1039
2.80F-11
-0,2042
7.14E-77
SNLI
GPT-J-6B
-0.1116
4.34F-18
-0.1961
1.29E-73
GPT-JT-6B
-0.0508
3.90F-49
0.0578
8.66E-05
Alpaca-7B
0.0155
6.06E-02
0.1787
L07E-21
Alpaca-13B
-0.0019
4.67E-05
Table 16: The average Pearson correlation coefficients (r) and corresponding p-values between in-context PVI estimates and inter-annotator agreement for runs on MultiNLI and SNLI using different models and the number of shots. Statistics are averaged over the results obtained using sets exemplars.
A.7 More challenging instances
sentence
target
PVI
The harder it has rained, how much faster flow appears in the river?
acceptable
13.3700
As John eats more, keep your mouth shut tighter. OK?
acceptable
-13.0345
The people you say that will buy tickets, the happier I'll be.
unacceptable
-11.4478
John wrote books.
unacceptable
-11.2615
Table 17: The hardest instances in each category in CoLA, determined by in-context PVI estimates obtained using GPT3-175B.
sentences
target
PVI
SENTENCE:: It's not that the questions they asked weren't it interesting or legitimate (though most did fall under the category of already asked and answered).
neutral
-12.0748
SENTENCE2: All of the questions were interesting according to a focus group consulted the subject.
SEXTENCE1: know i
don't know anybody in their right mind that says that that
i'm doing it because want to
contradiction
-8.1535
SENTENCE2: know there people who think I'm doing it because desire
SENTENCE]: And in another shift the economy,
found that lamb could be raised richer, more nutritious
more cost-efficttively on lowland farms part because grazing land available there and result Lakeland farms became less profitable. SENTENCE2: Another shift in economy was found more nutritious.
entailment
6.
6.7892
Table 18: The hardest instance in each category in the first 500 training instances in MultiNLI, determined by in-context PVI estimates obtained using GPT3-175B.
17

---

sentences
target
SENTENCE:: A white horse is pulling cart while man stands and watches.
neutral
998986
SENTENCE2: A horse hauling goods.
SENTENCE:; couple holding hands walks down
contradiction
-8.3236
SENTENCE2: There are people sitting the side the road.
SENTENCE1: foreign family walking along dirt path next to water.
entailment
-1.4902
SENTENCE2: foreign family walks by trail along body of water.
Table 19: The hardest instance in each category in the first 500 training instances in SNLI, determined by in-context PVI estimates obtained using GPT3-175B.
18


Reviews:
[{"review_id": "Zv3ZCd1RY3", "reviewer": "Reviewer_jyth", "summary": "", "questions": "A: Are there other natural language descriptions for Prompt? Would it be helpful to include a more detailed task description?\n\nB: The current strategy is to select exemplar randomly, is there any plan to improve?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review.\n\n**Reason-to-Reject #1: It is better to put the specific in-context PVI results of each model in the appendix for easy comparison.**\n\nWe will add the detailed results in the appendix that compare in-context PVI estimates across models. The table below gives an example of the results we will add in the appendix. This table shows the overall accuracy and in-context PVI related results for CoLA from different models.\n\n|model|overall accuracy|accuracy for instances with the lowest 20% in-context PVI|accuracy for instances with the highest 20% in-context PVI|\n|--|--|--|--|\n|GPT-neo-1.3B|46.21|46.53|53.38|\n|GPT-neo-2.7B|48.29|46.86|51.97|\n|GPT-J-6B|50.03|45.13|54.25|\n|Alpaca-7B|52.69|15.57|85.49|\n|Alpaca-13B|56.26|13.68|86.45|\n|GPT3-175B|72.42|0|100|\n\n**Reason-to-Reject #2: Although the current experiment is relatively comprehensive, it is still limited to the comparison between in-context PVI, and it is recommended to compare them with traditional PVI.**\n\nWe would like to clarify that in Section 5.4, we reported the comparison between in-context PVI and the original PVI in an indirect way: via the correlation between in-context PVI estimates and inter-annotator agreement, and between the original PVI estimates and inter-annotator agreement (see Table 5). Although we very much agree that it is more intuitive to directly compare in-context PVI with the original PVI (such as correlating in-context PVI estimates with the original PVI estimates), we did not do that for several reasons:\n \n* It would be prohibitively costly to obtain original PVI estimates for larger models. The calculation of the original PVI requires fine-tuning. Though it is feasible to fine-tune a model such as GPT2-125M, fine-tuning larger models such as GPT3-175B requires much more computational resources.\n* Our experimental results have shown that in-context PVI estimates from smaller models have lower qualities than those from larger models.\n  * We show in Section 5.3 that in-context PVI estimates from GPT2-125M are much noisier than those from GPT3-175B. This suggests that in-context PVI estimates from larger models better capture information useful to make correct predictions.\n  * We show in Section 5.4 that in-context PVI estimates from smaller models are less reliable because they are negatively correlated with the inter-annotator agreement score. Results also show that in-context PVI estimates from larger models such as GPT3-175B are positively correlated with the inter-annotator agreement score, which indicates high reliability.\n \nTo summarize, in-context PVI estimates made by larger models are reliable and they better capture information useful for making correct predictions. However, it is prohibitively costly to obtain their original PVI estimates. While it is feasible to obtain original PVI estimates by fine-tuning smaller models, our evidence has shown that in-context PVI estimates made by smaller models are less reliable. Thus, we do not include a section in our paper that directly compares in-context PVI estimates with the original PVI; however we do provide an indirect comparison in Section 5.4. We will make this reasoning clear in the revised version of the paper.\n \nIn Section 5.3, we reported the results comparing the noisiness of in-context PVI with GPT2-125M and GPT3-175B. In the revised version of the paper, we will include more results with other models we experimented with (i.e., GPT-neo-1.3B, GPT-neo-2.7B, GPT-J-6B, GPT-JT-6B, Alpaca-7B, and Alpaca-13B) in the appendix for a more comprehensive comparison.\n\n**Reviewer1-Question-A: Are there other natural language descriptions for Prompt? Would it be helpful to include a more detailed task description?**\n\nPrompt description: Our paper includes prompt examples in Table 12 and 13 in Appendix A.2. We used two kinds of prompts in the calculation of in-context PVI: an *input-target* prompt and a *null-target* prompt. An *input-target* prompt has the following form:\n```\nCONTEXT: I’m glad I saw anybody.\nQUESTION: Is this (0) unacceptable, or (1) acceptable?\nANSWER: 0\n```\nand a *null-target* prompt looks like the following:\n```\nANSWER: 0\n```\nDetailed task description: We agree that it would be helpful to include a more detailed task description. We currently have a short paragraph in Section 4 that briefly describes the seven tasks with citations to the primary sources. Further descriptions and sample instances are provided in Tables 9-11 in Appendix A.1. We will add a more detailed description of each task in the appendix.\n \n**Reviewer1-Question-B: The current strategy is to select exemplar randomly, is there any plan to improve?**\n\nThere are some demonstration selection methods in the current literature (such as in [1], [2], and [3]), and we agree that the quality of in-context PVI estimates may be improved if a more sophisticated method of selecting exemplars is applied. Our aim in the current study is to demonstrate that in-context PVI estimates show good qualities even with randomly selected exemplars. Furthermore, we set to develop in-context PVI and compare it with the original PVI. Experimenting with randomly selected exemplars is an unbiased approach. Exemplar selection is an active area of research with many variables to consider (e.g., avoiding bias, over-representing patterns). Thus, we leave the research on the combination of in-context PVI with other exemplar selection methodologies as the next step.\n \n[1] Li, X., & Qiu, X. (2023). Finding supporting examples for in-context learning. arXiv preprint arXiv:2302.13539.  \n[2] Chang, T. Y., & Jia, R. (2022). Careful Data Curation Stabilizes In-context Learning. arXiv preprint arXiv:2212.10378.  \n[3] ​​Zhang, Y., Feng, S., & Tan, C. (2022). Active example selection for in-context learning. arXiv preprint arXiv:2211.04486.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper introduces in-context PVI, a new approach for calculating PVI that exhibits higher efficiency compared to the original PVI method.\n- The paper offers a comprehensive empirical analysis, which serves to establish the reliability of in-context PVI. The findings provide new understanding of the capabilities of In-Context Learning (ICL).\n", "reasons_to_reject": "- It is better to put the specific in-context PVI results of each model in the appendix for easy comparison.\n- Although the current experiment is relatively comprehensive, it is still limited to the comparison between in-context PVI, and it is recommended to compare them with traditional PVI.\n", "questions_to_authors": "A: Are there other natural language descriptions for Prompt? Would it be helpful to include a more detailed task description?\n\nB: The current strategy is to select exemplar randomly, is there any plan to improve?\n", "paper_topic_and_main_contributions": "This paper introduces in-context pointwise V-usable information (in-context PVI), a modified version of the recently proposed hardness metric PVI, tailored for the ICL context. In-context PVI requires only a few exemplars and eliminates the need for fine-tuning. To validate the reliability of in-context PVI, a comprehensive empirical analysis was conducted. The results demonstrate that in-context PVI estimates exhibit similar characteristics to the original PVI. In the in-context setting, the estimates remain consistent across different exemplar selections and numbers of shots, indicating their stability. The work emphasizes the potential of in-context PVI as a tool for assessing model performance and provides valuable contributions to the understanding and application of ICL in the context of large language models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u2zyrw9AHq", "reviewer": "Reviewer_Ydp3", "summary": "", "questions": "- Table 1: How demonstrations and the number of demonstrations are chosen for these results? Is it a random choice of demonstrations (and number of demonstrations) or are they chosen based on how well they correlated with accuracy?\n- The caption and explanations for Table 5 are confusing. I think you are only measuring the correlation between in-context PVI and inter-annotator agreement. But the caption suggests that you are also measuring the correlation with the original-PVI, which one is that? \n- I can not fully understand section 5.5. What confuses me is that the task is not explained upfront. So it is hard to understand why some samples are \"challenging\". \n- In lines 381-384 you explain why the hardness of an instance for humans is not necessarily equivalent to that for a given model. I very much agree with this point. What would be particularly interesting is to look into the samples that are hard for the model and not hard for humans (or visa versa), Do you have any insights on this? Do you think in-context PVI can help uncover datasets artifacts through such analysis?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review.\n \n**Reason-to-Reject #1: I believe whether or not to use in-context PVI or PVI would depend on how we want to use the model. If the model is going to be prompted to solve the downstream task, it makes more sense to use in-context PVI. However, especially in scenarios where in-context performance is far from the state-of-the-art fine-tuned model, it would be best to use PVI to measure dataset difficulty. In that scenario, I think the paper lacks a more thorough and theoretical discussion on how PVI and in-context PVI relate to each other (e.g., Is one a lower bound for another?)**\n\nWe very much agree that whether or not to use in-context PVI depends on how we want to use the model. We agree that if fine-tuning leads to a better performance, and it is not prohibitively costly to fine-tune the model, we should use the original PVI. We do not propose in-context PVI as a replacement for the original PVI in this case. In fact, we show in Section 5.3 and 5.4 that in-context PVI estimates made by smaller models (which are more readily “fine-tunable”) have lower qualities, which implies that the original PVI is a better choice for those models. We believe that the primary value of in-context PVI is for cases where fine-tuning is challenging, such as for very large models like GPT3-175B. In the revised version of the paper, we will make it clearer that in-context PVI is not meant to be a strict replacement for the original PVI in cases where fine-tuning is possible (i.e., for smaller models, the original PVI is better than in-context PVI).\n \n**Reason-to-Reject #2: It seems necessary to compare and contrast in-context PVI with PVI directly. I am not convinced why PVI and in-context PVI are only measured in comparison (w.t.r. correlation) to inter-annotator but the correlation of the two measures is not assessed directly.**\n\nWe agree that it is more intuitive to directly compare in-context PVI with the original PVI (such as correlating in-context PVI estimates with the original PVI estimates). We did not do that for the following reasons:\n \n* It would be prohibitively costly to obtain original PVI estimates for larger models. The calculation of the original PVI requires fine-tuning. Though it is feasible to fine-tune a model such as GPT2-125M, fine-tuning larger models such as GPT3-175B requires much more computational resources.\n* Our experimental results have shown that in-context PVI estimates from smaller models have lower quality than those from larger models.\n  * We show in Section 5.3 that in-context PVI estimates from GPT2-125M are much noisier than those from GPT3-175B. This suggests that in-context PVI estimates from larger models better capture information useful to make correct predictions.\n  * We show in Section 5.4 that in-context PVI estimates from smaller models are less reliable because they are negatively correlated with the inter-annotator agreement score. Results also show that in-context PVI estimates from larger models such as GPT3-175B are positively correlated with the inter-annotator agreement score, which indicates high reliability.\n \nTo summarize, in-context PVI estimates made by larger models are more reliable and they better capture information useful for making correct predictions. However, it is prohibitively costly to obtain their original PVI estimates. While it is feasible to obtain original PVI estimates by fine-tuning smaller models, our evidence has shown that in-context PVI estimates made by smaller models are less reliable. Thus, we do not include a section in our paper that directly compares in-context PVI estimates with the original PVI. Instead, we provide an indirect comparison in section 5.4.\n\n**Reason-to-Reject #3: Significance of the results for Table 1., Figure 5., and Table 7 seems to be necessary and is not reported. For instance, in line 268, the authors claim that low in-context PVI estimates are significantly lower than that for instance with high in-context PVI estimates. The corresponding results in Table 1, however, do not report any statistical significance level, and thus do not support this claim. My suggestion would be to directly measure the correlation between in-context PVI and accuracy and report statistical significance.**\n\nFor Table 1, we did not base our claim that “the prediction accuracy for instances with low in-context PVI estimates is significantly lower than that for instances with high in-context PVI estimates” on statistical analysis such as Pearson correlation. Our claim is simply based on the observation that, for example, on BoolQ, the accuracy for instances with low in-context PVI is 0.3173, and the accuracy for instances with high in-context PVI is 0.6241, between which we think there is a big gap (we used the word \"significant\" to describe it). We have revised the texts that describe Table 1 and 7 to disambiguate.\n\nWe confirm that 23 out of 28 results are statistically significant in Figure 5. We will report the significance values in the revised paper.\n\n**Reason-to-Reject #4: After reading the last experiment, I am not convinced that in-context PVI should be used for selecting demonstrations. The reason is that the performance gain in Table 7 is not quite large and again the significance is not measured. Particularly, due to the large body of research showing the variance of performance with prompt variations, it seems necessary to control for this in such experiments. My suggestion is to vary the demonstrations and report the average and standard dev. of accuracy, maybe also doing statistical tests to prove the significance of the result.**\n\nIndeed, the performance gain shown in Table 7 in Section 5.6 is not large. As we pointed out in Section 5.6, we took a straightforward approach, simply selecting the hardest training instances as exemplars. Our approach is based on the intuition that hard instances are better exemplars. We also pointed out that this straightforward approach only helps improve model performance in MultiNLI and SNLI, and we advocated for a more sophisticated approach. This is a potential research question, and we agree that future work should take into account prompt variations in the experiments.\n \n**Reviewer2-Question-A: Table 1: How demonstrations and the number of demonstrations are chosen for these results? Is it a random choice of demonstrations (and number of demonstrations) or are they chosen based on how well they correlated with accuracy?**\n\nDemonstrations are chosen randomly from the training set of each task, and the number of demonstrations is chosen according to the number of unique labels in a dataset. For instance, MultiNLI has 3 unique labels, so we tried 3- and 6-shot (please refer to Exemplar selection in Section 4 and Table 12 for examples of the demonstrations). We did not choose them based on the performance. Please also see our response to Reviewer1, Question-B regarding exemplar selection.\n \n**Reviewer2-Question-B: The caption and explanations for Table 5 are confusing. I think you are only measuring the correlation between in-context PVI and inter-annotator agreement. But the caption suggests that you are also measuring the correlation with the original-PVI, which one is that?**\n\nThank you for catching that error. In Table 5, apart from the correlation between in-context PVI and inter-annotator agreement, we also report the correlation between the original PVI estimates and inter-annotator agreement, which is taken from [1]. We have updated the caption and explanations for Table 5.\n\n[1] Ethayarajh, K., Choi, Y., & Swayamdipta, S. (2022, June). Understanding Dataset Difficulty with V-Usable Information. In International Conference on Machine Learning (pp. 5988-6008). PMLR.\n \n**Reviewer2-Question-C: I can not fully understand section 5.5. What confuses me is that the task is not explained upfront. So it is hard to understand why some samples are \"challenging\".**\n\nWe will add the task descriptions for Health Advice and Causal Language to the manuscript, and we will carefully edit Section 5.5 to enhance its clarity.\n\nIn general, the challenging examples are caused by the potential discrepancy between common sense and annotation schema. For example, a sentence might contain noticeable advice indicators (such as “should be further explored,” as exemplified in the first example of Table 6) aimed at delineating the implications for subsequent investigations. However, based on the annotation schema, solely suggestions pertaining to health-related behavior or policy changes are labeled as health advice, whereas recommendations for forthcoming studies do not fall within the category of health advice. By utilizing the in-context pvi estimates, we are able to identify these examples.\n\n**Reviewer2-Question-D: In lines 381-384 you explain why the hardness of an instance for humans is not necessarily equivalent to that for a given model. I very much agree with this point. What would be particularly interesting is to look into the samples that are hard for the model and not hard for humans (or visa versa), Do you have any insights on this? Do you think in-context PVI can help uncover datasets artifacts through such analysis?**\n \nIn Table 8, we present two of the hardest instances discovered by in-context PVI in CoLA. Although the grammatical acceptability of the sentence “John wrote books” is obvious for English speakers, it is considered to be hard here by the model because it was mislabeled (the gold label for that sentence is UNACCEPTABLE, which is incorrect in our opinion). Table 6 shows challenging instances identified by in-context PVI, such as the sentence \"Supplementation with L. reuteri 6475 should be further explored as a novel approach to prevent age-associated bone loss and osteoporosis.\" According to the annotation schema of Health Advice, this sentence should be labeled as \"no advice\", as it just calls for the need for further scientific study, rather than gives recommendations for health-related behavior or policy changes. However, a general domain model may find the expression \"should be further explored\" as indicative of advice, making this instance challenging for it. The annotation errors (Table 8) and potential discrepancy between common sense and annotation schema (Table 6) are examples of dataset artifacts that can be uncovered by in-context PVI.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Computing PVI for large language models can be expensive. Particularly, for models with API access (like GPT-3) it is infeasible to fine-tune models to compute PVI. The method introduced in this paper does not require fine-tuning models and aims to measure PVI only through prompting.\n- Based on the empirical results, in-context PVI seems to correlate positively with accuracy, correlate with inter-annotator agreement, which suggests that it can serve as a replacement for PVI. ", "reasons_to_reject": "- I believe whether or not to use in-context PVI or PVI would depend on how we want to use the model. If the model is going to be prompted to solve the downstream task, it makes more sense to use in-context PVI. However, especially in scenarios where in-context performance is far from the state-of-the-art fine-tuned model, it would be best to use PVI to measure dataset difficulty. In that scenario, I think the paper lacks a more thorough and theoretical discussion on how PVI and in-context PVI relate to each other (e.g., Is one a lower bound for another?)\n- It seems necessary to compare and contrast in-context PVI with PVI directly. I am not convinced why PVI and in-context PVI are only measured in comparison (w.t.r. correlation) to inter-annotator but the correlation of the two measures is not assessed directly. \n- Significance of the results for Table 1., Figure 5., and Table 7 seems to be necessary and is not reported. For instance, in line 268, the authors claim that low in-context PVI estimates are **significantly** lower than that for instance with high in-context PVI estimates. The corresponding results in Table 1, however, do not report any statistical significance level, and thus do not support this claim. My suggestion would be to directly measure the correlation between in-context PVI and accuracy and report statistical significance. \n- After reading the last experiment, I am not convinced that in-context PVI should be used for selecting demonstrations. The reason is that the performance gain in Table 7 is not quite large and again the significance is not measured. Particularly, due to the large body of research showing the variance of performance with prompt variations, it seems necessary to control for this in such experiments. My suggestion is to vary the demonstrations and report the average and standard dev. of accuracy, maybe also doing statistical tests to prove the significance of the result. ", "questions_to_authors": "- Table 1: How demonstrations and the number of demonstrations are chosen for these results? Is it a random choice of demonstrations (and number of demonstrations) or are they chosen based on how well they correlated with accuracy?\n- The caption and explanations for Table 5 are confusing. I think you are only measuring the correlation between in-context PVI and inter-annotator agreement. But the caption suggests that you are also measuring the correlation with the original-PVI, which one is that? \n- I can not fully understand section 5.5. What confuses me is that the task is not explained upfront. So it is hard to understand why some samples are \"challenging\". \n- In lines 381-384 you explain why the hardness of an instance for humans is not necessarily equivalent to that for a given model. I very much agree with this point. What would be particularly interesting is to look into the samples that are hard for the model and not hard for humans (or visa versa), Do you have any insights on this? Do you think in-context PVI can help uncover datasets artifacts through such analysis?", "paper_topic_and_main_contributions": "This paper proposes a new way to measure pointwise $\\mathcal{V}$-usable information using in-context learning. Instead of fine-tuning models on input-output and output data, their approach prompts the models with demonstrations. The authors empirically assess the robustness of in-context PVI measure to the variations in the prompt, and find that the in-context PVI is quite robust. Furthermore, they measure the correlation between inter annotator agreement and in-context PCI and find that larger models correlate with inter-annotator agreement better. Finally, they employ in-context PVI as a way to select demonstrations for in-context learning. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z9kqjZjPoG", "reviewer": "Reviewer_pfMK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review.\n\n**Reason-to-Reject #1: My major concern lies in the accuracy difference between fine-tuning and in-context learning. In [1], fine-tuning achieves 80%~90% accuracy on the tested datasets, while in this paper, Table 1 (the first column) shows that in-context learning's accuracy on all the seven datasets are below or similar to random guess, i.e., in-context learning cannot make any meaningful predictions on these datasets. I think this may make the analysis in this paper invalid, and the authors could test on other datasets where in-context learning has a relatively competent performance.**\n \nWe would like to clarify that the results in Table 1 are the averaged results over 7 models, 3 sets of exemplars, and 2 numbers of shots. There is a large difference between the performance of smaller models and larger models. The following table shows the accuracy and in-context PVI related results of GPT2-125M and GPT3-175B on RTE using a 2-shot prompt:\n \n|model|accuracy|accuracy for instances with the lowest 20% PVI|accuracy for instances with the highest 20% PVI|\n|--|--|--|--|\n|GPT2-125M|42.4|20.8|59.3|\n|GPT3-175B|74.0|4.2|93.5|\n\n As a reference, the accuracy of fine-tuned BERT-large on RTE is 70.1%, and the accuracy of fine-tuned RoBERTa is 86.6% [1, 2]. GPT3-175B has a solid performance compared to the two fine-tuned models.\n \nTable 1 gives a broad overview of in-context PVI estimates across datasets. What we are trying to convey in Table 1 is that the prediction accuracy differs for instances with low and high in-context PVI estimates, and that in-context PVI estimates for correctly predicted instances are higher than those for incorrectly predicted instances, which align with what we would logically expect for a hardness metric.\n \nWe will replace Table 1 with the results of the best performing model (i.e., GPT3-175B), and we will also report specific in-context PVI results of each model in the appendix (please also see our reply to Reviewer1, Reason-to-Reject #1).\n \n[1] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.  \n[2] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L. & Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. \n \n**Reason-to-Reject #2: It is known that the performance of in-context learning depends on the quality of the in-context examples (and the number of shots), thus many methods of selecting good examples has been proposed. But in section 5.1 the authors conclude that pointwise v-usable information is stable w.r.t. different exemplar sets. This seems contradictory.**\n\nPrevious work has shown that the performance of ICL depends on the selection of exemplars, and we examine this in our paper, where we perform a comprehensive analysis regarding the variance of in-context PVI estimates across different sets and different numbers of randomly selected exemplars. As reported in section 5.2, our results show that in-context PVI estimates are quite stable. Our results show that the stability of in-context PVI estimates is not as sensitive to the selection of exemplars as the accuracy of in-context learners. \n\nFor more information regarding our method to select exemplars and numbers of shots, please refer to our response to Reviewer1 Question-B and Reviewer2 Question-A.\n \n**Typos Grammar Style And Presentation Improvements: line 257: should it be \"three unique labels\"?**\n\nThank you for noticing this. We have modified it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is novel to generalize pointwise v-usable information from fine-tuning to in-context learning.", "reasons_to_reject": "(1) My major concern lies in the accuracy difference between fine-tuning and in-context learning. In [1], fine-tuning achieves 80%~90% accuracy on the tested datasets, while in this paper, Table 1 (the first column) shows that in-context learning's accuracy on all the seven datasets are below or similar to random guess, i.e., in-context learning cannot make any meaningful predictions on these datasets. I think this may make the analysis in this paper invalid, and the authors could test on other datasets where in-context learning has a relatively competent performance.\n\nSpecifically, in Eq(1), the first term only depends on the input label and is identical for instances with the same label; the second term reflects the model's prediction. But if this prediction is merely a random number, the hardness measured by Eq(1) is also a random number without any meaning. Because the model's prediction is defined in a similar way as the second term, it is straightforward that the value of Eq(1) will correlate well with the model's accuracy (which the rest of Table 1 shows), but this cannot be taken as a justification of the metric. \n\n(2) It is known that the performance of in-context learning depends on the quality of the in-context exemplars (and the number of shots), thus many methods of selecting good exemplars has been proposed. But in section 5.1 the authors conclude that pointwise v-usable information is stable w.r.t. different exemplar sets. This seems contradictory. \n\n\n\n[1] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with v-usable information. ICML 2022.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper generalize the method of using pointwise v-usable information to analysis instance hardness w.r.t. fine-tuning to that w.r.t. in-context learning. The authors first showed strong correlations between the proposed metric and the accuracy, justifying that it is a reasonable hardness metric. They next demonstrate the reliability of the newly proposed measure, including relatively consistent results w.r.t. different exemplar sets and shot numbers, and low variance among different exemplar sets. They also use the metric to identify hard instances and use these instances to improve in-context learning.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 257: should it be \"three unique labels\"?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces in-context PVI as an efficient alternative to traditional PVI calculations, particularly beneficial for LLMs without the need for fine-tuning, and measures PVI only through prompting. The empirical analysis provides insights into the reliability and potential applications of in-context PVI. 

Reviewers appreciate the efficiency and novelty of the in-context PVI method and the novel generalization of v-usable information from fine-tuning to in-context learning, which is seen as a valuable contribution. The comprehensive empirical analysis conducted in the study adds substantial credibility to the concept and establishes its reliability. The reviewers also express that the empirical results indicate a positive correlation between in-context PVI and accuracy, as well as inter-annotator agreement, suggesting its potential as a replacement for PVI.  

However, the reviewers express concerns about the lack of a direct comparison between in-context PVI and traditional PVI. I agree with the suggestions by reviewers to directly measure the correlation between these two methods and report statistical significance, which would provide a more robust evaluation. I also suggest the authors to have a more in-depth discussion of the relationship between the two measures. 

Overall, the paper offers an innovative and efficient approach to PVI calculation and presents valuable empirical insights. I suggest authors addressing the concerns regarding method comparison and adding statistical significance for experimental results to strengthen the paper's quality.
Decision: Accept-Findings