Estimating Large Language Model Capabilities without Labeled Test Data
Authors: Harvey Yiyun Fu
Qinyuan Ye
Albert Xu
Xiang Ren
Robin Jia
Venue: EMNLP 2023 Findings (2023)
Keywords: large language model
accuracy prediction
confidence
calibration
in-context learning

Abstract: Large Language Models (LLMs) have exhibited an impressive ability to perform in-context learning (ICL) from only a few examples, but the success of ICL varies widely from task to task. Thus, it is important to quickly determine whether ICL is applicable to a new task, but directly evaluating ICL accuracy can be expensive in situations where test data is expensive to annotate---the exact situations where ICL is most appealing. In this paper, we propose the task of ICL accuracy estimation, in which we predict the accuracy of an LLM when doing in-context learning on a new task given only unlabeled test data for that task. To perform ICL accuracy estimation, we propose a method that trains a meta-model using LLM confidence scores as features. We compare our method to several strong accuracy estimation baselines on a new benchmark that covers 4 LLMs and 3 task collections. The meta-model improves over all baselines across 7 out of 12 settings and achieves the same estimation performance as directly evaluating on 40 collected labeled test examples per task. At the same time, no existing approach provides an accurate and reliable ICL accuracy estimation in every setting, highlighting the need for better ways to measure the uncertainty of LLM predictions.

Estimating Large Language Model Capabilities without Labeled Test Data
Harvey Yiyun Fu
Qinyuan Ye Albert Xu Xiang Ren Robin Jía
University of Southern California, Los Angeles, CA, USA (harveyfu, qinyuany, albertxu, xiangren, robinjiaJ@usc. edu
Abstract
Train Dataset
Large Language Models (LLMs) have the im- pressive ability to perform in-context Iearn- ing (ICL) from only a few examples, but the success of ICL varies widely from task to task. Thus, it important quickly determine whether ICL is applicable to a new task, but directly evaluating ICL accuracy can expen- sive in situations where test data is expensive to annotate-the exact situations where TCL is most appealing. In this paper. we propose the task of ICI. accuracy estimation, in which predict the accuracy of an LLM when doing in-context learning unlabeled test data for that task. To perform ICL accuracy estimation, we propose a method that trains meta model using LLM confidence scores as features. We compare our method to several strong accuracy estimation baselines new benchmark that covers 4 LLMs and 3 task collections. The meta-model improves over all baselines across 8 out of 12 settings and achieves the same estimation performance as directly evaluating on 40 collected labeled test examples per task. At the same time, exist- ing approach provides an accurate and reliable ICL accuracy estimation every setting, high- lighting the need for better ways to measure the uncertainty of LLM predictions.
Inpui
Confidence
Pred
True
[Promp]] name sleeping beauly
0.8
(Promet] highest dam india
Nurck Dam
Tehri
Train
Meta Model
0.8
Accuracy
Confidence Proflle
Test
Meta Model
?
task given only
Accuracy
Confidence Profile
Test Dataset
Unlabeled
Input
Confidence
Pred
True
[Prompt] Who is
Remeey Reneldo?
Ramsey
[Prompu] What sport did Jack Sock and Raffeella Reggi
0.5
valleybal
Accuracy Estimation without Labeled Test Data
demonstration of task setting: given ICL accuracy observations of labeled training datasets, we want to estimate the dataset level ICL accuracy on the unseen test datasets without labeled test data. We propose the method of training meta-model based on the confidence score distributions (we denote them dataset confidence profiles)
Figure 1:
1 Introduction
In-context learning (ICL) with large language mod- els (LLMs) has shown great potential in perform- ing a wide range of language tasks (Brown al., 2020). ICL has the unique advantages of being data-efficient (Lee, only a few labeled training ex- amples are needed) and accessible (i.e., expertise in training models is no longer required). With these advantages, non-expert user can create system perform new task within minutes by writing few examples. This gives rise to the popularity of
ICL- it is being adopted and tested for a varicty of use cases that stretch the boundary of what is considered possible to do with language models.
Despite the advantages of ICI., its performance is highly task-dependent (sce Figure 6). It sur- passes expectations on some tasks that diffi- cult for humans, such answering trivia questions or riddles, but achieves near-zero performance on scemingly trivial tasks such as some text editing or spelling tasks (Srivastava et al., 2022). While evalu-
'Code publicly released at titb. cow//arrvy--Pa/// icl-estimate

---

ating ICL with a labeled test set a direct solution know whether ICL will be effective, it greatly reduces the appeal of ICL, as one of ICL's key sell- ing points is that it does not require a large labeled dataset. In addition, many tasks do not come with a labeled test set due to high annotation costs (e.g., medical/law-related questions that require profes- sional knowledge to answer). In such cases, it is highly desirable to estimate the ICL performance without a labeled test set. This would help system developers determine whether ICL is likely to be useful for their problems of interest.
2 Related Work
2.1 Model Confidence and Calibration
Calibration of LLMs has been studied on a diverse range of tasks such classification (Desai and Durrett, 2020) and question answering (QA) (Jiang et al., 2021; Kadavath et al., 2022). It aims to study whether LLMs assign meaningful correctness likelihood also known as model confidence the oulputs (Guo et al., 2017). Most prior work evaluates calibration at the example level (Desai and Durrett, 2020; Kamath et al., 2020); in this paper, focus using overall model confidence distributions to estimate dataset-level accuracies. We propose method to learn model calibration patterns based on observations of LLMs' perfor- mance at the dataset level.
Guided by this motivation, we formalize the problem of few-shot ICL accuracy estimation: given handful of labeled in-context examples and a set of unlabeled test examples, our goal is to es- timate the overall accuracy of ICL on these test examples. Our contributions are twofold:
2.2 In-context Learning
We propose to address the accuracy estimation problem by training "meta-model," which takes in LLM confidence features imput and outputs the task accuracy. The meta-model is trained with observed ICL accuracies on seen datasets, and then used to estimate ICL accuracy on unseen datasets (see Figure 1).
LLMs pre-trained with auto-regressive language modeling objectives have been shown to be capa- ble of "learning" in context when given prompt composed of prompt template and a few labeled demonstrations (Brown et al., 2020; Chowdhery et al., 2022). While LLMs can learn a new task only through model inference, the accuracy is sen- sitive to the choices of prompt templates and in- context examples (Lu et al., 2021; Zhao et al., 2021; Perez et al., 2021). Therefore, we aim to develop method to accurately estimate ICL performance for dataset prompted with any prompt template and combination of in-context examples.
We obtain 42,360 observations of LLM ICL performance, by conducting extensive ICL experiments spanning two tasks (multiple- choice QA and closed-book QA), 91 datasets, and 4 LLMs. We then benchmark the meta- model method and multiple baselines on total of 12 evaluation settings derived from these observations.
2.3 Out-of-distribution (OOD) Prediction
Machine learning models in the real world com- monly encounter distribution shifts between train- ing and test time. Prior work (Guillory et al., 2021; Garg et al., 2022; Yu et al.. 2022: Singhal et al.. 2022; Li et al., 2022) aims to predict models' OOD performance under different setups. Garg et al. (2022) predict target domain accuracy for image classification tasks with distribution by fitting a threshold on model confidence using only labeled source data and unlabeled target data. Singhal et al. (2022) use a few additional target-domain exam- ples to predict the accuracy, focusing on known source-target dataset pairs on which models often have low OOD accuracy due overfitting spu- rious correlations (e.g., MNLI-HANS and QQP- PAWS). They find that accuracy on the given small set of target examples is a strong baseline to ap- proximate accuracy on the full-test set. We include
Our meta-model can estimate ICL accuracies without the need for labeled test examples. In 10 out of 12 settings, the meta-model estimates are at least accurate directly evaluating on 16 la- beled examples. In 2 out of 12 settings, they match with evaluating on 128 labeled examples. On av- erage, we are able to save the annotation cost of 40 test labels per task by using the meta-model. Further, the meta-model outperforms all baseline methods in 8 out of 12 settings, improving the rela- tive estimation error by 23.6% However, we also find that there exists substantial room for improve- ment across all settings. We envision estimating ICL accuracy without labeled test data as an open challenge and encourage the community to develop new techniques that can more accurately predict when ICL will be effective.

---

the accuracy for a small set of labeled test exam- ples as an oracle baseline (see Scction 3.3). These papers all try to predict the OOD accuracy of a model trained on in-distribution training data; in contrast, in our setting we have access to some la- beled datasets but the language models we study were never finetuned on those datasets. In order to avoid confusion, we instead use the terms "scen/un- seen tasks" to describe the datasets available to us, rather than "in-distributioonouutof -diii--bbbitiiitiioo.
3.3 Comparing with Labeled Test Data
To put our results context, we compare all meth- ods to the Oracle approach of sampling labeled examples from the test dataset Drest and measur- ing accuracy on those I examples, which we call oracle'. This approach is used by Singhal et al. (2022) and it represents how well we can evaluate ICL performance for Drest by collecting labeled examples. With large value of I, we get better evaluation of the test dataset at the cost of collect- ing expensive annotations. In proposing the task of accuracy prediction, we hope to develop methods that outperform the l-labeled oracle for values of that represent non-trivial annotation costs.
3 Accuracy Prediction
3.1 Problem Definition
We formalize the task of ICL accuracy estima- tion for unseen datasets given observations of the same model's performance on other datasets. A method for the ICL accuracy estimation task takes in four inputs: a language model MM; a set of labeled seen datasets {Di where each Dj consists of set of labeled examples ((x(") uf ),..... (I.(")) ./, (("))) and n, - Dil: a prompt e for the test task; and an unlabeled test (m)) dataset Drest {= Fiest) of size m. In a typical setting, each seen task should consist of a sufficient amount of labeled examples, i.e., n; 2 100. The method should output the esti- mated accuracy of M on Drest when prompted with prompt C; we denote the actual accuracy M.c of the model as acciest and aco,est M,e as the pre- dicted accuracy. Note that with the labeled datasets D; and a corresponding prompt we can com- pute the corresponding dataset-level ICL accuracy ace. M, for i I.....r.
Confidence Profile Meta-Model
We propose new method that trains a meta-mode based on the confidence profiles of seen datasets {D:M== to estimate ICL performance. We use the term confidence profile to denote the distribution of model confidence scores on each example in the dataset. We extract the confidence profiles (see Figure 1) from each seen dataset and convert them to feature vector. We then train a meta-model to map confidence feature vectors to the dataset- level ICL accuracies. The benefits of using the confidence feature vector are twofold. First, we do not need any labeled test data, which saves annota- tion costs. Second, this approach is applicable to any pre-trained language model like GPT3 (Brown et al., 2020) and OPT (Zhang al., 2022).
M.c
4.1 Confidence Profile
3.2 Prompt Formulation and Data Splits
In general, given a (not-necessarily labeled) dataset D, LM M, and a prompt c, we obtain the con- fidence profile by first computing the confidence score s MM. for each E D). The score for each input can be computed by one forward pass of M; the exact value of the score differs based on the task, as described below. Next, we sort the scores to obtain list [si, .DD] where each 8; S 81+1. Then we create diimensiial feature vec- tor M.c whose i-th component is linear in- terpolation between ([DD/s//j and (T[D)xi/d]- In- tuitively, the i-th feature represents the D/th percentile confidence score. We refer to the fea- ture vectors derived from confidence profiles as confidence vectors.
We construct prompts by sampling k in-context ex- amples uniformly at random from available labeled data and formatting them with prompt templates to form a prompt (see Section B and Table 6 in the Appendix). For each dataset D;, we sample K prompts, each consisting of a prompt template followed by a list of in-context examples to form a prompt Cij for j = 1,.... K. Note that each sample
dataset has training/test data split: in-context examples only from the training set, and measure accuracy only on the test set. For simplic- ity, we use e to denote a prompt in general, C; to denote the set of training prompts for dataset D;, and Clest to denote test prompts for dataset Drest-

---

4.2 Confidence scores
4.4 Meta-Model Architectures
The confidence score s2 is calculated differ- ently for closed-set generation and open-ended gen- eration.
We choose meta-models that are easy to train and contain far fewer parameters than LLMs for compu- tational efficiency. In this paper, consider three meta-modes architectures. First, use k Near- est Neighbors regression (k-NN), which measures feature similarity. In the context of this paper, k- NN retrieves the most similar confidence profile from the seen datasets to the test dataset confidence profile and predicts based on the observed ICL accuracy on the retrieved in-distribution datascts. We use the implementation in scikit-learn library. Second, we use two-layer Multilayer Percep- tron (MLP) that takes confidence feature vectors as input. Third, we use the tree-based method XG- Boost (Chen and Guestrin, 2016) with the same confidence features. We use XGBoostRegressor implemented in the XGBoost library* and tune the hyperparameters as described in Appendix C.2.5
Closed-set generation. Closed-set generation tasks have a pre-defined label space y. We take outputs from LLMs and identify the answers only by labels (Kadavath et al., 2022). For each exam- ple. we take model confidence as the normalized probability across the label space:
8 (x) = EdEy Pg
(1)
where Py is the model-assigned probability for label y input Pg is the probability for the output label ÿ from model M, and ý arg maxgey Pý-
Open-ended generation. We refer to tasks that require sequence generation (e.g., closed-book QA, summarization, machine reading comprehension, etc.) as open-ended generation tasks. We use neg- ative log-likelihood (NLL)² to obtain confidence scores from each generated sequence. Let ý be the model-generated sequence. We compute the confidence score as:
4.5 Evaluation
For task performance evaluation, we use Exact Match (EM) accuracy to measure accuracy for closed-set QA, and Fi-score to measure accuracy for open-ended QA.
We evaluate accuracy prediction models based
absolute error, defined |acctest M,Cteat
16 Mx = Σ
M.Croot)) where both are computed using the test acC
log (éee)
dataset Dteet with test prompt Cteat- We then av- erage the absolute error over all prompts Ctest and compute the dataset-specific mean average error:
Pt is the model-assigned probability distribution at output token / and is the 1-th output token
M.c M,o |acoues
CTT Dheat
acC i
[Cread
4.3
Meta-Model Training Data
oECtest
For each seen dataset (D:) sample K prompts {ct)) Then for each prompt sampled we compute the confidence vector conf. and accuracy acc, M,e;j create one meta-training ex- ample (conf; Mayy acc, acc;
Finally, to evaluate the overall success of accuracy prediction across collection of test datasets T, we measure mean absolute error (MAE), defined as:
M.ay). ). This creates total of K meta-training examples. The meta-model is trained on the meta-training examples and pre- Mfett based on
Σ DIeNET
errT
erT
(3)
4.6 Baselines
diets the estimated accuracy aCC the test dataset feature vector conl M, for each test prompt Ctest Ctest- Note that since closed- set/open-ended generations have different confi- dence scores and accuracy evaluation metrics, the meta-model does not train on datasets that have a different task formulation than the test datasets.
We consider four baselines for accuracy estimation.
Diest
Average training accuracy (AYGTrAIN). We simply take the average dataset-level accuracy of
Phttps://scikkt--eemm.a
https:///ggoost.reeaaeeeeeeeee io/en/stablee 'We ulso considered linear models but did not include them here as they perform much worse in estimating ICI. accuracies.
Wo also tried perplexity but found NLJ. to yield better results.

---

the seen datasets as our accuracy estimation:
dataset be the average of the ATC estimates from each seen dataset:
K
M,ey acc;
ACGATGTRIIN TXK
aCCATC = Σ
atc ittet:
Average Calibration Error (AvGConF). We take the average confidence across the test dataset as the accuracy estimation:
where abciteest is the Dj Dtest ATC estimate.
M.c.
4.7 Alternative Featurizations
In addition to the confidence profiles, we experi- ment with another featurization method that uses model embeddings from the LM M. Given a dataset D, LM M, and prompt c, obtain the model embedding by first taking the last-layer, last- token embedding eMx((x)) for each x D, and then averaging across the dataset:
OCCAVGCONE
Crest X m cECrest Dt
Note that this baseline is only applicable to closed- set generation and not open-ended generation tasks since the accuracy metric for open-ended genera- tion (FI score) and confidence metric (NLL) do not share the same range ((0, 1] vs. (ooo, 0)). The intuition behind AvGCONF is that if the model con- fidence scores are well-calibrated (at the example level), then the expected value of the model's con- fidence scores should be equal to the accuracy. In fact, note that the MAE of AvGCONF is similar Expected Calibration Error (ECE), which mea- sures the example-level calibration error (Naeini al., 2015; Guo et al., 2017; Kumar et al., 2019; Desai and Durrett, 2020).
M.c 1 |D]
embedD = Σ e (x)).
FED
M,C
Since embed is very high-dimensional (e.g., 5120 dimensional for 13B models), we use Prin- ciple Component Analysis (PCA) to reduce its mensionality. We fit the PCA model on all dataset M,c embedding vectors embed and transform them into de di-messioonnl vectors embed M.c which we can use feature vector. As an additional exper- iment, we can concatenate the confidence vector and embedding vector to form a combined feature vector:
Temperature Scaling (TS) Temperature scaling is widely used calibration method (Hinton et al., 2015; Guo al., 2017; Si et al., 2022). By fitting single scaler parameter called temperature r, produces softer model-assigned probabilities
ce ce = cooff + embed
M,r
exp (Zg/T)
Pg Eg exp (ZHEN/T))
Reducing the dimensionality makes the comparison with confidence features more fair, and does not dilute the influence of confidence features when concatenating them with embedding features.
We then obtain scaled confidence scores with Equa- tion and evaluate AvGCONF on the test dataset, Note that we optimize temperature 7 based on the AVG of the training datasets instead of the common approach of using NLL as an objective function.
5 Experiments
Accuracy Estimation Benchmark
We benchmark both meta-model method for ICL accuracy estimation and the baseline methods mentioned in Section 4.6 on total of 12 LLM- dataset collection pairs (3 dataset collections x 4 LLMs). For each evaluation setting, we evaluate 3 different featurization methods mentioned in Sec- tion 4.7. This adds up to 36 experiment settings.
Average Threshold Confidence (ATC) We ATC (Garg et al., 2022) as one of our OOD ac- curacy estimation baselines. ATC takes accuracy estimation by fitting confidence threshold on single source dataset and generalizes to the target dataset. We take the estimated accuracy for the test
"ECE computes a weighted average of the difference be- tween confidence and accuracy for each confidence interval bin. The main difference is that ECE is commonly computed by binning confidence scores into buckets, whereas in our setting each "bucket" is a different OOD dataset.
Datasets. We use three different collections of datasets in total: multiple-choice QA (MCQA) from MMLU (Hendrycks et al., 2020) and both

---

MMLU
230
MAD
73-
3.-
2.5
LLaMA-7B
LLaMA-133 13B
OPT-6.7B
OPT-138
MCQA
23.
MAE
+
73
30
255
LLaMA-7B
LLaMA-13B
OPT-6.7B
OPT-13B
CBQA
MAE
13.0
3.0
LLaMA-78 7B
LLaMA 13B
OPT-6.7B
OPT-136
MLP
XGBoost
1000
AvgConf
oracle 4
oracle 16
oracle_ 64
3-NN
AvgTrain
oracle_
oracle_ 32
Figure 2: Bar graph of evaluation results (MAE) for all meta- meta-models, baseline methods, and Oracle baselines of all 3 dataset collections with all 4 LLMss. We use the confidence vector the meta-feature. Red/blue hars represent the meta-model/baasline evaluation results and the horizontal lines show the Oracle baselines.
MCQA and closed-book QA (CBQA) from Cross- Fit (Ye et al., 2021) (see Table 5 for the full list of tasks). We heneeforth use MCQA and CBQA refer the CrossFit dataset collections respectively. We use the implementations and training/test partitions from HuggingFace Datasets (Lhoest et al., 2021). We split each collection of datasets into meta-training/tee splits using 5-fold cross-vall malas-valiiaiion-www we partition each dataset collec- tion into equal-sized subsets and run five versions of each experiment, one with each subset being used as the mela-test set and the remaining subsets used as meta-training data. We take the average of the meta-tes test results as our final result.
LLMs. We run our experiments on four LLMs: OPT-6.7B. OPT-13B (Zhang et al., 2022), LLaMA-
7B. and LLaMA-13B (Touvron al., 2023). We use the OPT models from HuggingFace Models" and the LLaMA model from Meta AI. More de- tails are included in Appendix C.I.
Experimental Details.
We generate prompts for each dataset using the method noted in Section 3.2. For each dataset in MMLU, we combine the "val- idation" set and 11 set to be the training set
"https::/huggiingfaae http:///wag.iaaaaccccco/ocoooese co/models
thttps://githuu thub. htmm/aacbbookreeearee. laama phttps://huggingfaae co/datasets/caa
19The "dev" set contains 5 examples for each dataset is meant for few-shot development purposes.

---

Methods
LLaMA-7B
LLaMA-13B
MMLU
MCQA
CBQA
MMLU
MCQA
CBQA
Meta Models
MI.P
7.06 + 1.03
8.82 + 2.64
6.32 2.13
5.80 + 0.63
11.04 3.30
8.08 + 2.99
3-NN
5.98 +: 0.97
5.62 2.35
7.10 + 2.15
5.50 + 0.66
11.26 + 4.56
8.56 + 2.85
XGBoost
5.42 1: 4.52
5.22 2.17
7.00 2.44
5.00 + 0.62
11.52 5.20
9.00 1 4.85
Baselines
AVGTRAIN
5,26 + 1.40
10.26 2.50
11.62 5.88
9.90 + 1.70
11.40 3.36
13.14 7.25
AVGCONF
5.54 + 0.54
6,88 3.14
n/a
5.10 + 0.86
8.58 = 1.75
n/a
5.60 + 1.63
6.32 3.58
n/a
14.06 1.53
14.00 6.95
TV/a
ATC
20.34 1 4.10
34.66 1 9.36
31.80 1. 13.44
20.50 1 4.92
24.14 1 7.72
31.80 1 12.49
Oracle
5.82 (32)
6.44 (32)
7.06 (16)
6.18 (32)
13.44 (8)
10.82 (8)
ACC
31.08 + 6.32
39.00 10.52
23.7 + 10.44
45.50 + 11.74
50.34 + 12.84
29.30 + 12.38
Methods
OPT-6.7B MCQA
OPT-13B
MMLU
CBQA
MMLU
MCQA
CBQA
Meta Models
MLP
5.70 0.77
6,66 1.25
7.28 1: 2.75
6.30 + 1.14
7.18 + 1.69
5.90 1 1.18
3-NN
4.06 + 0.33
2.98 0.57
6,46 2.05
5.16 + 0.24
2.78 : 1.06
5.84 + 2.19
XGBoost
3.76 + 0.32
2.60 0.73
8.06 3.07
4.54 + 0.35
2.54 + 1.16
6.00 + 2.43
Baselines
AVGTRAIN
3,48 + 0.37
10.66 1.75
8.38 3.86
4,28 + 0.35
11.46 2.15
18.00 3.77
AVGCONF
8.42 1 0,60
13,42 1 1.70
n/a
6,20 0.77
7.14 1.77
n/a
3.92 0.17
4.60 0.80
n/a
4.36 1 0.31
2.72 1 1.03
ATC
24.54 + 2.12
37.64 + 7.93
32.16 + 14.60
24.72 3.32
29.40 + 10.28
30.34 + 12.48
Oracle
5.58 (32)
2.76 (128)
6.50 (16)
5.58 (32)
2.76 (128)
6.70 (16)
ACC
26.68 1 4.42
33.16 1 9.80
17.54 5.92
26.82 5.28
33.68 11200
19.34 1 6.30
Table I: Evalutaion results (MAE) and variations (SD) for all 4 LLMs, 3 dataset collections settings, using confidence vector as the meta-feature, For Oracle baselines, we include the closest lower-bound. e.g. the error is between 0 and @r put oracle (32). OPT-13B settings have the lowest average MAE (5.13). compared to OPT-667B (5.28), LLaMA-7B (6.50). 1..MAA113((..4) We report overall accuracy (ACC) by Exact Match MMLU and MCQA, and Fl-score for CBQA
that we sample in-context examples from. We sam- ple 10 3-shot, 10 4-shot, and 10 5-shot prompts!! and decorate each of them with prompt tem- plates chosen for MMLU (see Table 3). We choose deom 20 here since many of the datasets contain only 100 text examples. For MCQA and CBQA, we sample in-context examples from pool of 100 examples the training set, and obtain test of 1000 examples (see Section in the appendix for implementation details). For each dataset, we sample 30 3-shot and 30 4-shot prompts and deco- rate them with only the null template. We choose dromf 100 for both MCQA and CBQA settings. Section 5.3 contains more details about the abla- tion studies for doonf- Due to computational rea- sons, sample only 30 prompts (compared to 60
prompts for MCQA/CBQA datasets) for MMLU because contains a very large number datasets.
5.2 Main Results
The meta-model outperforms all baselines un- der certain evaluation settings. Table 1 shows the meta-model estimation error for each evaluation setting. For 8 out of 12 settings (all CBQA settings. LLaMA-7B on MCQA, LLaMA-13B on MMLU, both OPT models on MCQA), the best meta-model architecture has 23.67% lower relative MAE than the best baseline method on average. In the best (OPT-6.7B MCQA), the meta-model can achieve 43.5% lower relative MAE than all base- lines. However, for the other 4 settings (both OPT models on MMLU, LLaMA-7B on MMLU, and LLaMA-13B on MCQA), baseline methods pro- vide more accurate estimates ICL accuracy. Fig-
We choose up to 5-shot setting because it is studied in previous studies (Touvron et al., 2023; Rac et al., 2021).

---

OPT-136
100
2.6
accurscy
0.4
0.4-
conf dimensionss
80
3.1
2.7
3.6
0.2
0.2
3.4
0.2
0.8
0.3
0.4
8
4.1
3.1
2.7
2.7
2.6
predicted accuracy
predicted
3.2
0.8
40
3.9
3
2.8
2.6
2.6
0.5-
0.4
actudl
20
3
2.8
2.7
2.5
2.6
0.2
6.2
200
400
600
800
1000
0.2
predicted
0.4
0.5
number of unlabeled examples m
Figure 3: We plot the meta-model predicted accuracy versus the actual accuracy across 4 settings. We use the XGBoost meta-model and the confidence vector meta-feature. Fach entity represents observation for one dataset. Red/blue represents higher/lower absolute error
Figure Estimation results for ablating the number of unlabeled examples m (X-axis) and confidence vector dimensions de (y-axis), evaluated on the OPT-13B on MCQA using the XGBoost meta-model.
for 5 settings (LLaMA-13B on MMLU, both OPT models on MMLU, and OPT-6. 7B on MCQA).
ure 2 shows the evaluation results graphically. On average across all 12 settings, the best estimation errors from the meta-models are 32.5% less than the actual accuracy standard deviations. In 11 out settings, the estimation errors are within one standard deviation of the actual accuracy.
Ablation Model Architecture Across threc meta-model structures, the XGBoost meta-model overall provides the most accurate estimation as has the lowest MAE for out of 12 evalua- tion settings. The average MAE is 5.88 for XG- Boost meta-models, 5.94 for 3-NN meta-models, and 7.18 for MLP meta-models. Surprisingly, 3- NN meta-models have a lower average MAE than MLP meta meta-models despite having simpler model structure. In Figure 3, show that the XGBoost meta-model provides well-correlated accuracy esti- mation across different evaluation settings.
Oracle baselines indicate useful accuracy esti- mations. In comparison to the Oracle baselines, the meta-model outperforms the oracle base- line in all MMLU and MCQA settings except for LLaMA-13B on MCQA (achieves oracle and outperforms the oracle baseline in all CBQA set- tings except for LLaMA-13B (achieves oracle"). In the two best-case settings (using XGBoost the meta-model on MCQA with either OPT model), the meta-model achieves the oracle 128 bascline, i.e., is equivalent to estimating the accuracy using 128 annotations.
A Ablation on Featurization Methods We con- sider three featurization methods as described in Section 4.7. Table 2 in the appendix shows that the best overall accuracy estimation for all settings is attained by using the confidence vectors meta- features (achieves the lowest MAE for 26 out of 36 evaluation settings). The average MAE is 6.27 for con f. 8.34 for embed, and 7.36 for ce. Further, using con as features demonstrates a more domi- nant advantage for all CBQA tasks, achieving the lowest MAE for 11 out of 12 evaluation settings.
Baseline methods are effective in some settings. While ATC is a weak baseline for ICL accuracy estimation, AVGTRAIN, AvGCONF, and TS are strong baselines for MMLU and MCQA. AvG- TRAIN is able to achieve 3racle32 for 3 out of 12 settings (LLaMA-7B, OPT-6.7B 7B, and OPT-13B on MMLUU;; AVGCONF is able to achieve 3racle for 2 settings (LLaMA-7B and LLaMA-13B MMLU), and 16 for 3 settings (both LLaMA models on MCQA and OPT-13B on MMLU). Note that temperature scaling improves calibration in certain settings: TS is able to achieve oracle 128 for one setting (3PT-13B on MCQA), and 3r
5.3 Efffet of Unlabeled Data and Confidence Vector Dimensions
We now study confidence feature vector ablations by varying the number of unlabeled test examples m each unseen dataset and the dimension of the
As noted in Section 4.6, the AvGCOnF and TS baseline is not applicable to CBQA datuscts.

---

confidence vector deonf. We with OPT-13B on MCQA datasets using the XGBoost meta-model since we achieve the lowest MAE in this setting. Figure 4 shows that increasing m enables better accuracy estimation, reducing the average MAE (across all d from 3.92 for m = 200 to 2.56 for m 1000. Note that increasing requires per- forming additional LLM inferences unlabeled examples, leveraging unlabeled test data is con- strained by computational cost considerations. The quality of our accuracy estimates does not vary much as we change the confidence vector dimen- sion d as shown in Figure 4.
0.07
0.06
Anerage ACC difference
0.05
0.04
0.03
0.01
0.00
LLMMA7B
L3MMA-138
LLMs
MMLU
CBQA
MCQA
MMLU
5.4 Effect of Number of Shots
Figure 5: Prompt selection results for evaluation settings, measured by the absolute difference between ICL accuracy when prompted with and the average accuracy. Blue bars show the actual best prompt (i.e., c*), and red bars show the selected best prompt (i.e. c=e,==e)
We compare ICL accuracy estimation performance given different k-shot ICL accuracy observations for LLLMAA13B
MMLU datasets. Table 4 in the Appendix shows that the meta-model produces slightly better ICL accuracy estimation for the 3-shot setting. Overall, the meta-model gives con- sistent accuracy estimates across different k-shot settings as they all achieve oracle
observed accuracies seen datascts. We show that without using any labeled test data, the meta-model is often able attain accurate estimates of ICL accuracy, which practically useful for predicting LLMs' accuracy on datasets that have high annota- tion costs. We also construct a large-scale bench- mark for dataset-level ICL accuracy estimation by evaluating the meta-model and multiple baseline methods across 12 evaluation settings and 3 meta- feature options. We observe that while some base- line methods provide good accuracy estimates, our meta-model demonstrates non- non-trivial improve- ment in estimation abilities over baseline methods in 8 out of 12 evalutation settings.
5.5 Prompt Selection
Previous works demonstrated that ICL perfor- mance is highly sensitive to the prompt templates as well as in-context examples (Zhao et al., 2021; Perez et al., 2021; Chen et al., 2022); we are thus interested in whether our ICL accuracy estimation method can be upplied to select the best ICL prompt e CE Crest for the test dataset. For each dataset, use the XGBoost meta-model to select the best prompt c² as opposed to the actual best prompt c*. We then compute the corresponding ICL accu- racies and compare them to the average accuracy across all test prompts. Figure 5 shows that there is significant difference in ICL accuracy given dif- ferent prompts for all 12 settings, and the selected prompts lead better ICL accuracies than the av- erage accuracy for out of 12 settings. On average, the selected prompt 15.6% as effective as the actual best prompt. The limited improvement from the random baseline indicates there's 's large room for improvement and we encourage future work to derive better prompt selection standard.
We encourage future work to develop better meta-model architectures as well as better meta- features and study potential implications for the meta-model, such as acting as prompt tem- plate/ICL example selection method. We believe that benchmark can serve as an open challenge for improving datasct-level ICL accuracy estima- tions, leading to an improved understanding of when ICL is likely to be effective.
Limitations
While we conducted extensive experiments to study ICL accuracy estimations, there are many more LLMs that have exhibited impressive capabilities on a variety of tasks. Due to computational con- straints, we do not benchmark accuracy estima- tions based on LLMs with limited access (e.g.,
6 Discussion and Conclusion
this paper, we study the problem of few-shot ICL accuracy estimation. We propose training a meta-model based on LLM confidence features and

---

GPT-4 (OpenAI, 2023)) it difficult to extract model embedding features, or those larger than 13B. We also don't 't consider instruction-tunee mod- els to avoid possible overlaps between their training datasets and our evaluation datasets. Meanwhile instruction tuning sometimes hurts model perfor- mance on canonical datasets such as MMLU, as shown in Gudibande et al. (2023). It might also significantly hurt calibration as reported in OpenAl (2023). For the same reasons, we include only a limited number of prompt templates and in-context example variations for ICL prompting. While we choose only few-shot settings for MMLU and 2 for MCQA and CBQA, it is possible to achieve better accuracy estimations with more observations in the training data.
the 22nd aem sigkdd international conference on knowledge discovery and data mining, pages 785- 794.
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. 2022. On the relation between sensitivity and accuracy in in-context learning. arXiv preprint arXiv:2209.0766
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint 02X3v:220..02311.
Shrey Desai and Greg Durrett. 2020. tion pre-trained transformers. arXiv preprint arXiv:2003.0782
Calibra-
Saurabh Garg, Sivaraman Balakrishnan, Zachary C Lipton, Behnam Neyshabur, and Hanie Sedghi. 2022. Leveraging unlabeled data to predict out-of-distribution performance. arXiv preprint arXiv:2201.0425.
In terms of dataset selection, we use 13 closed- book QA tasks for the open-ended generation set- ting. Our findings might not generalize to other open-ended generation tasks such as summariza- tion or long-form question answering. Overall, the meta-model provides effective accuracy estima- tions, but there's 's still substantial room for improve- ment.
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprictary llms.
Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi. Trevor Darrell, and Ludwig Schmidt. 2021. Pre- dicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 1134-1144 114.
Acknowledgements
We would like thank Ting-Yu Yun Chang for her valuable contributions. We thank Ameya Godbole, Johnny Wei, and Wang Zhu for their valuable dis- cussions. We also thank all members of the Al- legro lab at USC for their support and valuable feedback. RJ was supported by an Open Philan- thropy research grant, Cisco Research Award, and HI was supported by the USC Provost Fellowship Award.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International conference machine learn- ing, pages 1321-1330. PMLR
Dan Hendrycks, Collin Burn,, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under- standing. arXiv preprint XiXx:2009.03300
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, 2015. Distilling the knowledge in neural network. arXiv preprint arXiv: 150300233...
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Zhenghao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? the calibration of language models for question answering. Transactions of the Associa- tion for Computational Linguistics, -962-9777.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom
Henighan, Dawn Drain, Ethann Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.052211
Amita Kamath, Robin Jia, and Percy Liang, 2020. Se- lective question answering under domain shift, arXiv preprint 200.022006.0966..
Tianqi Chen and Carlos Guestrin. 2016. Xgboost: scalable tree boosting system. In Proceedings of

---

Ananya Kumar, Percy S Liang, and Tengyu Ma. 2019. Verified uncertainty calibration. Advances in Neural Information Processing Systems, 32.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1 239.71
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. 2021. Datasets: commu- nity library for natural language processing. arXiv preprint arXiv: 200.04609
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. Crossfit: few-shot learning challenge for cross-task generalization in nlp. arXiv preprint arXiv:2104.08835.
Zeju Li, Konstantinos Kamnitsas, Mobarakol Islam, Chen Chen, and Ben Glocker. 2022. Estimating model performance under domain shifts with class- specific confidence scores. In Medical Image Com- puting and Computer Assisted InterventionMMICCAI 2022: 25th International Conference, Singapore, September 18-22, 2022, Proceedings, Part VII, pages 693-703. Springer.
Yaodong Yu, Zitong Yang. Alexander Wei. Yi Ma, and Jacob Steinhardt. 2022. Predicting out-of- distribution error with the projection norm. In In- ternational Conference on Machine Learning, pages 25721-25746. PMLR.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- Mona Diab, Xian Li, Xi Victoria Lin, al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint 2rX5v:2205.001
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08788
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance language models. In ternational Conference on Machine Learning, pages 12697-12706. PMLR.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated prob- abilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial In- telligence, AAAI' 15, page 2901-2907. AAAI Press.
OpenAL. 2023. Gpt-4 technical report.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. Ad- vances in neural information processing systems,
34:1 34:11054-11070.
Jack W Rae, Sehastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 12.
Chenglei Si, Chen Zhao, Sewon Min, and Jordan Boyd- Graber. 2022. Re-examining calibration: The case of question answering. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 28 2814-2829.
Prasann Singhal, Jarad Forristal, Xi Ye, and Greg Dur- rett. 2022. Assessing out-of-domain language model performance from few examples. arXiv preprint arXiv:2210.06725.
Aarohi Srivastava, Abhinav Rastogi, Abhishck Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro. Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0665..

---

MMLU
Methods
LLaMA-7B
LLaMA-13B
OPT-6.7B
OPT-13B
Meta Models
conf
embed
cc
conf
embed
ce
conf
embed
cc
conf
embed
ce
MLP
7,06
9.76
8.00
5,80
9.92
8.84
5.70
10.66
8.02
6.30
11.24
9.48
3-NN
5.98
5.50
6.80
6,80
4.60
4.60
5.16
5.20
5.20
XGBoost
5.42
4.52
4.56
5.00
7.30
5.16
3.76
3.94
3.76
4.54
4.88
4.72
Baselines
AVGTRAIN
5.26 1 1.40
9,90 +: 1.70
3.48 0.37
4.28 0.35
AVGCONF
5.54 +: 0.54
5.10 + 0.86
8.42 0.60
6.20 0.77
TS
5.60 + 1.63
14.06 1.53
3.92 0.17
4.36 + 0.31
ATC
20.34 + 4.10
20.50 4.92
24.54 + 2.12
24.72 3.32
Oracle
5.82 (32)
6.18 (32)
5.58 (32)
5.58 (32)
ACC
31.08 + 6.32
45.50 + 11.74
26.68 +: 4.42
26.82 +: 5.28
MCQA
Methods
LLaMA-7B
LLaMA-13B
OPT-6.7B
OPT-13B
Meta Models
conf
embed
ce
conf
embed
Cc
conf
embed
ce
conf
embed
ce
MLP
8.82
12.50
10.62
11.04
16.86
14.64
6.66
13.00
10.30
7.18
14.54
13.72
3-NN
5.62
6.22
6.22
11.26
11.24
11.24
2.98
3.84
3.84
2.78
5.60
5.58
XGBoost
5.22
6.34
6.12
11.52
10.90
11.34
2.60
5.34
3.00
2.54
6.74
2.94
Baselines
AVGTRAIN
10.26 2.50
11.40 3.36
10.66 + 1.75
11.46 + 2.15
AVGCONF
6.88 1 3.14
8.58 1: 1.75
13.42 1: 1.70
7.14 1: 1.77
TS
6.32 + 3.58
14.00 6.95
4.60 0.80
2.72 1.03
ATC
34.66 9.36
24.14 7.72
37.64 1 7.93
29.40 10.28
Oracle
6.44 (32)
9.30 (16)
2.76 (128)
2.76 (128)
ACC
39.00 + 10.52
50.54 1: 13.86
33.42 11.58
33.68 12.06
CBQA
Methods
..L.MA-BB
I.L.MA-13B
OPT--67B
OPT-13B
Meta Models
conf
embed
cc
conf
embed
ce
conf
embed
ce
conf
embed
ce
MLP
6.32
14.26
7.14
8.08
12.18
10.96
7.28
8.44
8.46
5.90
7.42
9.12
3-NN
7.10
10.74
8.56
10.74
10.72
10.44
10.18
10.62
10.64
XGBoost
7.00
13.10
8.56
9.00
11.60
10.64
8.06
7.58
7.74
6.00
8.54
7.74
Baselines
AVGTRAIN
11.62 1: 5.88
13.14 7.25
8.38
3.86
18.00 +. 3.77
ATC
31.80 + 13.44
31.80 + 12.49
32.16
14.60
30.34 +: 12.48
Oracle
(16)
10.82 (8)
6.50 (16)
6.70 (16)
ACC
23.72 + 10.44
29.30 12.38
17.54 + 5.92
19.34 6.30
Table 2: Full estimation results for LLMs, 3 dataset collections, and feature choices. XGBoost the best overall meta-model structure with average MAE of 6.60. The confidence vector is the best overall feature with an average MAE of 6.38 across all evaluation settings. The best meta-model outperforms all baseline methods in
12 evaluation settings. We report overall accuracy (ACC) by Exact Match for MMLU and MCQA, and F1-score for CBQA

---

A Dataset Implementation Details
some LLMs (except for OPT-13B) can be run on GPUs with smaller memory. We evaluate 42,360 ICL observations in total, where each observation is dataset with 100 to 1000 examples. The total inference process takes around 2000 GPU hours.
Following Section 5.1's discussion of specific dataset implementations for MCQA and CBQA, our approach for each setting is: for tasks that have a defined train/test split in HuggingFace Datasets, we sample in-context examples from the first 100 examples in the training set and then choose the first 1000 examples from the test questions; while for other datasets that only have the training set defined, we first sample a subset of 100 examples from the first 80% of the dataset and then sample in-context examples from the subset. The test questions are obtained by sampling 1000 examples from the rest 20% of the dataset.
C.2 Meta-model Implementations
All meta-model architectures can be trained on an i7-10700 CPU. The total training time of the meta- model on one experiment setting varies from 1.5 hours to 24 hours depending on training data dimen- sions. We include the implementation details for each of the meta-model architectures. We use the random seed for all processes involving random- ness. For K Nearest Neighbors regression, we use the implementation of KNeighborsRReressoo from sklearn library. We use euclidean distance as the weight metric, and fit the model on the meta- training data.
be our test
B Prompt Templates
We collect general prompt templates for MMLU datasets: 1 Null template, 1 self-constructed prompt template, 2 from previous work (Rae et al., 2021; Hendrycks et al., 2020), and I generated by ChatGPT. For MCQA and CBQA, we only use the Null template duc to resource considerations.
For MLP, we implement with Pytorch. We use 2-layer MLP with the size of the hidden state 1536, learning_ faagg_rrteeaa le 5, and dropouttrott = 0.2. We use Adam Opti- mizer and MSELoss, and we perform early stop- ping with the validation data. The validation data is a 20% random partition of the meta-training data. For early stopping, the max epoch is 50 and the patience is 7.
Prompt Template
Source
Null
m/a
You are an expert in (sabject], here are some multiple-choice questions
Self cansttannte
The following are multipic choice questions (with answers) about (subject)
For XGBoost, we implement with the XGBRegressor from the XGBoost library(Chen and Guestrin, 2016). We use a 5-fold random search cross-validation for 300 iterations to choose the hyperparameter. The candidates are:
Hendrycka et al. (2020)
A highly knowledgeeable and intelligent Al multiple-choice questions about (subject)
Rae al. (202.))
Test your knowledge of (subject) with maltiple -choicc questions discover the
ChaiGPT
Table 3: Prompt templates for MMLU datasets. {sub-
"Ir": uniforn(8.81, 0.5).
depxtt"phh:: randint(3.10),
ject] is replaced by the dataset subsets (e.g., ab- stract stract_algebra) algebra)
"n_estimators": randint 1000),
"colsample_bytree" [0.5, 8.6, 8.7, 8.8. 8.9,
95, 1J.
"subsanple". [a.5, 0.6, a.7, 0.a, 0.9, 8.95, 11, "gamna": uniforn(e,1) "reg_alpha" unifarm(o.1). "regalenbda": uniforn(0,1)
C Implementation Details
10
We will release code to reproduce our results upon publication.
C.3 Other Implementation Details
C.1 LLMs Implementations
Output Collection For multiple-choice QA tasks (MMLU and MCQA). we collect generated choice labels (c.g., "(A)") from the first tokens generated. For closed-book QA tasks (CBQA), we collect the first 16 newly generated tokens as the model output and truncate the outputs by the newline token.
We use LLaMA-7B and LLaMA-13B (Touvron et al., 2023) from Meta AI, and OPT-6.7B and OPT- 13B (Zhang et al., 2022) from HuggingFace trans- formers. For LLaMA-7B, OPT-6.7B, and OPT- 13B, we run evaluations using a single RTXA6000 LLaMA-13B, we run evaluations GPU (48GB). For For
Temperature Scaling We search for the optimal temperature 7 based on the meta meeattraining set. The search grid is:
by parallel inference on two RTXA6000 GPUs. We use half-precision for both OPT models. Note that

---

np. (1.iisspaee(.0,, 0, 3.0, 100)
D Few-shot setting ablation results
Methods
3-shot
4-shot
5-shot
mixed
Meta Models
conf
conf
conf
MLP
5.54
5.92
5,90
5.80
3-NN
5.34
5.80
5.92
5.50
XGBoost
4.86
5.34
5,40
5,00
Baselines
9.78
9.92
10.16
9,90
ACB
5.00
5.10
5,18
5.10
ATC.
20.10
20.44
20.68
20.50
Oracle
6.14 (32)
6.14 (32)
6.12 (32)
6.18 (32)
ACC
45.50
45.36g11.26
45.52_112.0
45. 211177
Table 4: Estimation results for separate and mixed few-shot settings, measured by MAE and tested on the LLaMA-133 the MMLU setting. The accuracy esti- mation is consistent across different shot settings. We report the accuracy (ACC) as Exact Match accuracy

---

Dataset Collection
Datasets
MMLU
"abstract "abstracttalgebaa. "anatomy", "astronomy". "college biology" "college_biology"
colllge_chemisstr" satngge_commtte_ssieneee
coollege_mathemmaee
Solllee__physies
computer_securjc ponceptual_physiee
(lletrical_engineeniii
jiementary_mathhmaatee
"high_school school biology". "high school chemistry". "high school computer scienne_""eee "high_school school_ mathematics", "high_school sch_s_choo__pg _ph_sschh_p_sccss.....____ "high_schaee sirgh_shhol____ "ct_soootssttttttice""
statistics"
"machine Jearning". "chho gotebroonggeeemmeett..oo and_ politics"" "cconometries". "high school macroeconomicc". "high school microeconomicss. "sociology"
"high_school hathhoolssychllogy"
[human_sexualii
irofessional_psycholle
puuliicrelations"
"ecuritt_studies" "us_ vus_foreign_police "formal world history", "intemational_lawy "jrisppuddee" "logical_ fallacies", "moral_ connalscceaaros", "philosophy". "prehistory", "professional_la ___a""""" "world_ religionn,,. "business ethics". "clinical knowledge', "'ollege_medicine"
"hihh_ school lal_uus__ss__ history", "high school_world
hhighsschool___e
medicine". "global facts" "human aging". "management", , "marketing", "medical genetics", "miscellancous". "moral disputes", "professional accounting". "professional medicine", "viroloyy"
"esinnsssethhiss..
"nuttimtt "nutrition"
MCQA
"wiqa", siind__gande" "swag", "superglue-copa" "socialli_ga"
"race-middle,, woiuko-wit__koowlddee
quaatz-no_knowledee "quarel", "qase"
"openbookqa". "hellaswag". "dream". "cosmos qa". "commonsense qu" "ai2 are", "codah", "aqua_aqraaa rat", "race-high", "quail", "definite_pronnuus resolution" rrastitttooooneesottttoo
CBQA
"squad-nu caclioo_ooooo context", "numer_sense", "kilt_trex", "kilt_zsre", "lama-trex" 'lama-squad" "lana-google_re, "lama- 'lama-conceptnet,, "kilt_hotpotqa" "kilLnq", "freebase_ web__questions" "jeopardy"
Table list dataset collections: MMLU, MCQA, and CBQA. There are datasets in MMLU, 21 datasets MCQA, and 13 datasets in CBQA.

---

Task
Examples
Labels
MMLU
What is the second most common element in the solar system?
(A) Iron
(B) Hydrogen
(C) Methane
(D) Helium
Answer: (D) Helium
On which planet in our solar system can you find the Great Red Spot?
(A) Venus
(B) Mars
(C) Jupiter
(D) Saturn
Answer:
(C)
CBQA
Who are the members of Aespa?
Answer: Karina, Giselle, Winter, and Ningning
Who is the first overall pick in the 2011 NBA draft?
Answer:
Kyrie Irving
Table 6: Two 1-shot examples decorated with default null prompt template for MMLU datasets and CBQA dataset.
We take the same prompt template for MCQA as MMLU.

---

0.4
OPT-13B
0.3
F1
0.2
0.1
+
0.0
0.7
LLaMA-13B
0.6
0.5
0.4
F1
0.3
0.2
0.1
0.0
squad-no context
numer sense
kilt trex
kilt zsre
lama- lama-trex trex
lama squad
lama lama-google google re
lama conceptnet
kilt hotpotqa
kilt nq
freebase qa
web questions
jeopardy
Figure 6: 4-shot ICL accuracy for OPT-13B and LLaMA-13B on CBQA tasks, where each boxplot summarizes the Fl scores over ICL prompt variations one dataset. Both OPT-13B and L.aMA-I3B have large variances across different tasks, showing the challenge of ICL accuracy estimation.


Reviews:
[{"review_id": "MllIt5odha", "reviewer": "Reviewer_zZme", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for helpful comments and insightful questions! Here are our responses to the questions and concerns:\n\n> **Q1:** The assessment criteria for the performance of large language models is limited to accuracy metrics, which is not a comprehensive representation of the performance of large language models in real-world applications.\n\nWe acknowledge that there is more than one dimension to evaluate large language model capabilities as outlined in [1], but in this work, our main focus is to estimate in-context learning accuracies for QA tasks, for which the accuracy metrics are particularly important. Meanwhile, we would like to point out that the method we use to estimate accuracy also evaluates model calibration and robustness, as we incorporated model confidence (Sec. 4, L209) as the meta feature, and estimate performance on datasets that are out-of-domain (L404).\n\n> **Q2:** The method exhibits dependence on similar examples from the training dataset. This raises potential concerns regarding the distribution consistency between the training and test datasets adopted in the study. An in-depth visualization and analysis of the data distributions might be beneficial to address such concerns.\n\nWe hope to clarify that the test datasets are out-of-distribution from the training datasets. \n\n* There exists **domain shifts** between datasets within the same collection (Table 5 in the appendix). For example, the *abstract algebra* dataset would be an out-of-distribution dataset from the *anatomy* dataset in MMLU. \n* We perform 5-fold cross-validation for each dataset collection, which prevents overfitting to possible training/test consistency patterns. Except that the meta-training/test data share the same task format (e.g., multiple-choice QA), we would not consider them as any more similar.\n\nFor further clarification, we would like to point out that the reason we choose not to test the cross-task generalizability (e.g., from multiple-choice QA datasets to closed-book QA datasets) is that they have different accuracy metrics (Exact match v.s. F1-score). Although it is possible to solve multiple-choice QA tasks as a generation task and report F1-score instead, it is not the common practice.\n\nWe promise to include a visualization of domain distributions for the dataset collections upon acceptance. \n\n> **Q3:** The evaluative framework appears somewhat limited in scope. With considerations restricted to merely three Question-answer tasks and two language models, there are reservations about the method's broader applicability. Its potential to generalize to other reasoning or generation tasks or more advanced models, such as vicuna or alpaca, remains a subject of inquiry.\n\nWe conduct experiments on three collections of Question-Answering datasets containing **91** datasets in total, and there are **4** LLMs that we experimented with. Considering the number of permutations of prompts and in-context examples, we actually evaluate a total of **42360** in-context learning performance observations. While we would be interested in running experiments on additional datasets, the number of datasets we can evaluate is subject to computational resource limitations and we consider exploring such generalizability as future work.\n\nWe acknowledge that instruction-tuned models are of great interest to the community recently, but in our task setting, some concerns need to be addressed.\n \n* We aim to estimate in-context learning performance and the LLM of interest should therefore be capable of performing in-context learning, but many instruction-tuned models such as Vicuna and Alpaca are not instruction-tuned to do few-shot in-context learning. \n* Instruction-tuning sometimes hurts model performance on canonical datasets such as MMLU, as shown in [2], and might also significantly hurt calibration as reported in [3]. \n* To evaluate an additional language model we would need to evaluate around 10,000 in-context learning settings. We therefore did not include instruction-tuned models in the study, but consider it future work to investigate whether the same framework applies to such models. \n* We do not consider LLMs that are directly trained on our evaluation datasets such as T0 to prevent data leakage.\n\n*We hope our responses have addressed the concerns of the reviewer and we are happy to answer any further questions.*\n\nReferences:\n\n[1] Liang, Percy, et al. \"Holistic evaluation of language models.\" arXiv preprint arXiv:2211.09110 (2022)\n\n[2] Gudibande, Arnav, et al. \"The false promise of imitating proprietary llms.\" arXiv preprint arXiv:2305.15717 (2023).\n\n[3] OpenAI. “GPT 4 Technical Report”. arXiv preprint arXiv:2303.08774 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method offers the capability to predict the in-context learning performance of expansive language models, circumventing the need for human annotations.\n2. It has been empirically demonstrated that the proposed model consistently outperforms a range of baseline approaches,  based on two distinct large language models.", "reasons_to_reject": "1. The assessment criteria for the performance of large language models is limited to accuracy metrics. Such a limited view does not necessarily provide a comprehensive representation of the performance of large language models in real-world applications.\n2. The method exhibits dependence on similar examples from the training dataset. This raises potential concerns regarding the distribution consistency between the training and test datasets adopted in the study. An in-depth visualization and analysis of the data distributions might be beneficial to address such concerns.\n3. The evaluative framework appears somewhat limited in scope. With considerations restricted to merely three Question-Answering tasks and two language models, there are reservations about the method's broader applicability. Its potential to generalize to other reasoning or generation tasks or more advanced models, such as vicunna or alpaca, remains a subject of inquiry.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigated  how to estimate the in-context learning capability of large language models in the absence of labeled test data. To address this  challenge, the authors developed a meta-model designed to predict the performance of these large language models, complete with an associated confidence features. Specifically, the authors designed a training dataset wherein the input encapsulates the model's confidence features, and the output represents the performance metrics of the large language models. Concurrently, the authors calibrated the model confidence features in alignment with the disparities observed between closed-set generation and open-ended generation. Empirical studies across three question answering tasks, employing two  large language models,  illustrate that the introduced methodology consistently outperforms  baselines.\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vKvh3OxzjL", "reviewer": "Reviewer_8wF1", "summary": "", "questions": "An intriguing avenue for future research would be extending this estimation technique to encompass other performance metrics like sensitivity, calibration, and selective prediction. Such extensions could significantly enhance the practical utility of the approach, especially concerning scenarios where the model abstains from answering due to low confidence.\n\nDo authors have any opinions on this?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful questions and comments! We are glad that the reviewer found our paper to be practically interesting and useful. Here are our responses to the individual questions:\n\n> **Q1:** The observed improvements appear modest, particularly when juxtaposed with the fact that the Oracle method only requires 32 examples for accurate estimation. Given the relatively manageable nature of manually evaluating 32 examples, the incremental benefits of the proposed approach could be perceived as limited.\n\nWe believe that matching the estimation accuracy with collecting 32 labeled examples could be a substantial improvement. \n\n* The annotation cost varies by the domain. For some domains that require professional knowledge to annotate, collecting 32 examples might already be costly. \n* One main feature of our method is the ability to generalize to new test datasets. If a practitioner wanted to estimate accuracy on many out-of-distribution datasets, our method would become more advantageous, since it can save 32 labeled examples per dataset. \n\nLastly, part of our contribution is to establish the evaluation setting as a benchmark, and we would be happy to see future work improve on our baselines. \n\n> **Q2:** The evaluations are weak. Expanding the evaluation to include a more diverse set of datasets, such as the Natural-Instruction V1/2, would enhance the generalizability of the findings and bolster the argument for the approach's robustness.\n\nWe agree with the reviewer that including more datasets would help prove the robustness of the proposed method, but we would like to clarify that we conduct experiments on three collections of Question-Answering datasets containing 91 datasets in total, and there are 4 LLMs that we experimented with. Considering the number of permutations of prompts and in-context examples, we actually evaluate a total of **42360** in-context learning performance observations. While we would be interested in running experiments on additional datasets, the number of datasets we can evaluate is subject to computational resource limitations and we consider exploring such generalizability as future work.\n\n> **Q3:** The high variance in accuracy, as evident from Table 1, might diminish the perceived significance of the improvements. Integrating a statistical test for a more rigorous comparison could provide a more compelling case\n\nWe are sorry about the possible confusion. To clarify, the variances reported in the table refer to the variance in in-context learning performance across all datasets and prompt permutations (see Figure 5 in the appendix for an illustration). The high variances indicate that accuracy estimation is nontrivial, as a constant estimator cannot achieve low error, so this actually demonstrates that our calibrator is performing a nontrivial task. For example, in our best case, the variance of actual accuracy is 12.06 whereas the estimation error is just 2.54. Across all 12 settings, the best estimations from the meta-models are 67.5% of the standard deviations, and in 11 out of 12 settings, the estimation error is less than one standard deviation.\n\nWe show the accuracy estimation variance across all test prompts for all settings below. It is evident that the meta-model estimation standard deviations are **significantly smaller** than the standard deviation of the actual accuracy, which indicates non-trivial improvements in accuracy estimations. \n\n| Methods | | LLaMA-7B |  |  | LLaMA-13B | |\n| :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |  :-----------: |  :-----------: |\n|  | MMLU  | MCQA | CBQA | MMLU | MCQA | CBQA|\n| MLP | 7.06 &pm; 1.03 | 8.82 &pm; 2.64 | 6.32 &pm; 2.13 |5.80 &pm; 0.63|11.04 &pm; 3.30|6.32 &pm; 2.13|\n|3-NN | 5.98 &pm; 0.97 | 5.62 &pm; 2.35 | 7.10 &pm; 2.15 |5.50 &pm; 0.66| 11.26 &pm; 4.56 | 7.10 &pm; 2.15|\n|XGBoost | 5.42 &pm; 1.09 | 5.22 &pm; 2.17 | 7.00 &pm; 2.44 |5.00 &pm; 0.62| 11.52 &pm; 5.20 | 7.00 &pm; 2.44|\n|AvgTrain| 5.26 &pm; 1.40| 10.26 &pm; 2.50 | 11.62 &pm; 5.88 |9.90 &pm; 1.70| 11.40 &pm; 3.36| 11.62 &pm; 5.88|\n|AvgConf| 5.54 &pm; 0.54| 6.88 &pm; 3.14 |n/a |5.10 &pm; 0.86|8.58 &pm; 1.75|n/a|\n|TS|5.6 &pm; 1.63| 6.32 &pm; 3.58 |n/a|14.06 &pm; 1.53|14.00 &pm; 6.95|n/a|\n|ATC|20.34 &pm; 4.1| 34.66 &pm; 9.36 |31.8 &pm; 13.44|20.50 &pm; 4.92|24.14 &pm; 7.72|31.80 &pm; 13.44|\n|ACC| 31.08 &pm; 6.32 | 39.00 &pm; 10.52 | 23.70 &pm;10.44 | 45.50 &pm; 11.74 | 50.34 &pm; 12.84 | 29.30 &pm; 12.38|\n\n| Methods | | OPT-6.7B |  |  | OPT-13B | |\n| :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |  :-----------: |  :-----------: |\n|  | MMLU  | MCQA | CBQA | MMLU | MCQA | CBQA|\n| MLP | 5.70 &pm; 0.77 | 6.66 &pm; 1.25 | 7.28 &pm; 2.75 | 6.30 &pm; 1.14 | 7.18 &pm; 1.69 | 5.90 &pm; 1.18 |\n|3-NN | 4.06 &pm; 0.33 | 2.98 &pm; 0.57 | 6.46 &pm; 2.05 | 5.16 &pm; 0.24 | 2.78 &pm; 1.06 | 5.84 &pm; 2.19 |\n|XGBoost | 3.76 &pm; 0.32 | 2.60 &pm; 0.73 | 8.06 &pm; 3.07 | 4.54 &pm; 0.35 | 2.54 &pm; 1.16 | 6.00 &pm; 2.43 |\n|AvgTrain| 3.48 &pm; 0.37 | 10.66 &pm; 1.75 | 8.38 &pm; 3.86 | 4.28 &pm; 0.35 | 11.46 &pm; 2.15 | 18.00 &pm; 3.77|\n|AvgConf| 8.42 &pm; 0.60 | 13.42 &pm; 1.70 | n/a | 6.20 &pm; 0.77 | 7.14 &pm; 1.77 | n/a|\n|TS| 3.92 &pm; 0.17 | 4.60 &pm; 0.80 | n/a | 4.36 &pm; 0.31 | 2.72 &pm; 1.03 | n/a|\n|ATC| 24.54 &pm; 2.12 | 37.64 &pm; 7.93 | 32.16 &pm; 14.60 | 24.72 &pm; 3.32 | 29.4 &pm; 10.28 | 30.34 &pm; 12.48|\n|ACC| 26.68 &pm; 4.42 | 33.16 &pm; 9.80 | 17.54 &pm; 5.92 | 26.82 &pm; 5.28 | 33.68 &pm; 12.06 | 19.34 &pm; 6.30|\n\n> **Q4:** The reporting of the lower-bound $oracle^{32}$ might be revised. Instead, a graphical representation comparing Oracle's accuracy against other methods with an increasing number of examples would offer a more intuitive and informative visualization.\n\nWe thank the reviewer for the great suggestion. We prepared the graphical representations of the oracle baselines from $oracle^{4}$ to $oracle^{128}$ of all settings but did not include them in the final draft. Since the oracle baselines vary across different settings, it would be messy to concatenate the graphs together. We agree that visualizing the oracle baselines would provide more interpretability, and we will include them upon acceptance.\n\n> **Q5:** An intriguing avenue for future research would be extending this estimation technique to encompass other performance metrics like sensitivity, calibration, and selective prediction. Such extensions could significantly enhance the practical utility of the approach, especially concerning scenarios where the model abstains from answering due to low confidence.\n\nWe appreciate the reviewer’s idea of how future research can extend the utility of this work. We would like to point out that besides the accuracy metrics that we directly evaluate, the method is built on model calibration as we incorporated model confidence (Sec. 4, L209) as the meta feature. We also take average calibration error as one of the baseline methods to evaluate the model calibration on a dataset level. We consider it as future work to compare the estimation from the meta-model to this baseline to analyze when the LLMs become overconfident. \n\nWe also agree with envisioning selective prediction and abstaining from answering as exciting future directions. Meanwhile, it is also worth investigating whether the same method can be applied to estimate the example-level accuracy as opposed to the dataset-level estimation scenario in this work. \n\n*We hope our responses have addressed the concerns of the reviewer and we are happy to answer any further questions.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper addresses a significant and practical concern in NLP, namely, the estimation of ICL performance using a meta-model framework. This approach has the potential to considerably save time and resources.\n\n* The comparative evaluation against baselines showcases the paper's effectiveness. The results highlight its state-of-the-art performance, which is a strong indicator of the approach's viability.\n\n* The ablative analysis enhances the clarity of the proposed system. By discerning the impact of different components, the authors provide insights into the workings of the model and its underlying mechanisms.", "reasons_to_reject": "* The observed improvements appear modest, particularly when juxtaposed with the fact that the Oracle method only requires 32 examples for accurate estimation. Given the relatively manageable nature of manually evaluating 32 examples, the incremental benefits of the proposed approach could be perceived as limited.\n\n* The evaluations are weak. Expanding the evaluation to include a more diverse set of datasets, such as the Natural-Instruction V1/2, would enhance the generalizability of the findings and bolster the argument for the approach's robustness.\n\n* The high variance in accuracy, as evident from Table 1, might diminish the perceived significance of the improvements. Integrating a statistical test for a more rigorous comparison could provide a more compelling case.\n\n* The reporting of the lower-bound $oracle^{32}$ might be revised. Instead, a graphical representation comparing Oracle's accuracy against other methods with an increasing number of examples would offer a more intuitive and informative visualization.", "questions_to_authors": "An intriguing avenue for future research would be extending this estimation technique to encompass other performance metrics like sensitivity, calibration, and selective prediction. Such extensions could significantly enhance the practical utility of the approach, especially concerning scenarios where the model abstains from answering due to low confidence.\n\nDo authors have any opinions on this?", "paper_topic_and_main_contributions": "The paper under review presents a study on estimating in-context learning (ICL) accuracies for both seen and unseen tasks, utilizing minimal ground truth labels. The authors introduce a meta-model approach and evaluate it across three datasets and two language model architectures (OPT and LLaMA), demonstrating its superior performance over baselines. The inclusion of confidence-based evaluation strategies adds further value to the paper, given their relevance to empirical practices in the NLP community.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4UQH2SGyTK", "reviewer": "Reviewer_rur3", "summary": "", "questions": "- In Section 3.1, how the prompt c^~ (L177)  is different than prompt c?\n- What does j represent in L188 and consequently what does c_ij refer to? What does K prompt mean? Isn't there a single prompt per task?\n- Why do the authors think that the meta models perform better on the CBQA datasets compared to the others? What could be the explanation that there are not consistent results/patterns?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful questions and comments and we are glad that the reviewer appreciates the paper writing and acknowledges the practicality of our proposed methods. Here are our responses to the individual questions:\n\n> **Q1:** I am not sure how a meta-model can be useful to solve the problem (of requiring labeled test data) in practice, since labeled data will always be needed to train a meta-model\n\nWe are sorry for the confusion. The meta-model would be useful for estimating in-context learning accuracy for test datasets without acquiring labeled test data, which is particularly beneficial for the low-resource tasks that annotations are expensive to collect. On the other hand, we are able to generate meta-training data from observations of LLMs performance on datasets with plenty of labeled examples readily available (Sec 4.3, L266). We then train the meta-model using the meta-training data to estimate the performance on the test datasets, therefore saving the cost of collecting annotations for the test datasets. Figure 1 provides a visual demonstration of our task setting. \n\n> **Q2:** Even though the problem formulation (i.e., ICL accuracy estimation) is interesting and novel, the proposed methods to address it are not.\n\nWe proposed the confidence metric for free-form generation and proved that the ICL accuracy estimation can be done with a simple meta-model. We are the first to apply such confidence metrics to estimate in-context learning accuracies on a dataset level.\n\n> **Q3:** The results are not convincing that the proposed approach can be actually beneficial in a real low (labaled) data resource setting (e.g., 'In 2 out of 12 settings, they match with evaluating on 128 labeled examples.').\n\nWe would like to clarify: \n* The oracle baselines indicate how many labeled test data we could otherwise save from annotating to achieve the same estimation accuracy. A high oracle baseline (e.g.,  $oracle^{128}$) would represent a high estimation accuracy.\n* As mentioned by the reviewer, the meta-model is not able to achieve $oracle^{128}$ in every setting. However, it is able to achieve  $oracle^{16}$ in 10 out of 12 settings, which still represent the **non-trivial** cost of annotation 16 examples for the real low resource setting. Therefore, in the practical case, it is highly likely that the meta-model could be still beneficial to save a substantial amount of expensive annotations.\n\n> **Q4:** In Section 3.1, how the prompt $c^{~}$ (L177) is different than prompt $c$?\n\nWe are sorry for the confusion. $c^{~}$ is used to denote the prompts for the labeled datasets (seen datasets) as opposed to the prompts for the unlabeled datasets (unseen test datasets) $c$. A subscript could work better here but might look messy as we incorporate c in the accuracy notations. (e.g., L179)\n\n> **Q5:** What does $j$ represent in L188 and consequently what does $c_{ij}$ refer to? What does $K$ prompt mean? Isn't there a single prompt per task?\n\nWe are sorry for the confusion. For each task, we have several prompt templates (sec B in appendix, L765), and each prompt template is followed by in-context example permutations (L418). $c_{ij}$ refers to one single prompt, and $K$ is the total number of prompt variations. For example, there are 5 prompt templates for MMLU datasets. For each dataset, we generate 30 in-context example permutations, which results in $K=5\\times30=150$ prompts. Each $c_{ij}$ refers to one prompt in these $K$ prompts. \n\n> **Q6:** Why do the authors think that the meta-models perform better on the CBQA datasets compared to the others? What could be the explanation that there are not consistent results/patterns?\n\nWe would like to draw attention to Table 2 in the appendix where we include the full results. To clarify, we did not claim that meta-models perform better on the CBQA datasets. However, the meta-models appear to more frequently outperform the baselines because the baseline methods do not estimate as accurately for free-form generation tasks as closed-set generation tasks. One possible explanation for the inconsistency is that free-form generation tasks have different confidence metrics from closed-set generation tasks (Sec 4.2, L241).\n\nWe thank the reviewer for the meticulous review and will fix all the typos.\n\n*We hope our responses have addressed the concerns of the reviewer and we are happy to answer any further questions.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and easy to follow. The problem studied is well motivated and of practical importance. \n- The experimental setup is solid and the experiments thorough (multiple baselines, models, datasets).\n- The authors proposed several (existing) methods to constitute the model architecture for training the meta-model, which shows to be performing well in some settings. They also provide an ablation study to explain design choices.", "reasons_to_reject": "- I am not sure how a meta-model can be useful to solve the problem (of requiring labeled test data) in practice, since labelled data will always be needed to train a meta-model\n- Even though the problem formulation (i.e., ICL accuracy estimation) is interesting and novel, the proposed methods to address it are not.\n- The results are not convincing that the proposed approach can be actually beneficial in a real low (labaled) data resource setting (e.g., 'In 2 out of 12 settings, they match with evaluating on 128 labeled examples.').", "questions_to_authors": "- In Section 3.1, how the prompt c^~ (L177)  is different than prompt c?\n- What does j represent in L188 and consequently what does c_ij refer to? What does K prompt mean? Isn't there a single prompt per task?\n- Why do the authors think that the meta models perform better on the CBQA datasets compared to the others? What could be the explanation that there are not consistent results/patterns?", "paper_topic_and_main_contributions": "The paper is focusing on the practical value of in-context learning (ICL) with large language models (LLMs). Specifically, the authors argue that, naturally, when evaluating ICL with a labeled test set is a direct solution to know whether ICL will be effective, it greatly reduces the appeal of ICL, as one of ICL’s key selling points is that it does not require a large labeled dataset. In addition, many tasks do not come with a labeled test set due to high annotation costs (e.g., medical/law-related questions that require professional knowledge to answer). The authors thus introduce the problem of few-shot ICL accuracy estimation: given a handful of labeled in-context examples and a set of unlabeled test examples, the goal is to estimate the overall accuracy of ICL on these test examples. They test a series of models (e.g., MLP and XGBoost) to train a meta-model for accuracy estimation. They evaluate the models against baselines and oracle in 3 datasets and 4 models, showing the potential value of the proposed approach.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- In 5.2 there are several 'orcale' instead of 'oracle'."}, "extra_text": {}}, {"review_id": "KKkW0WBTrr", "reviewer": "Reviewer_cyhj", "summary": "", "questions": "1. Line 78 'conducting extensive ICL experiments spanning 91 datasets', I didn't see the results on 91 datasets in the subsequent sections.\n2. It seems that the problem is highly influenced by label distribution of a task?\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful questions and comments! Here are our responses to the individual questions:\n\n> **Q1:** As a general problem for ICL, the benchmark proposed in this paper should include more tasks and organize them better.\n\nWe agree with the reviewer that including more tasks would add to the comprehensiveness. Considering the number of permutations of prompts and in-context examples, we actually evaluate a total of **42,360** in-context learning performance observations based on **91** datasets and **4** LLMs. While we would be interested in running experiments on additional datasets, the number of datasets we can evaluate is subject to computational resource limitations and we motivate future work to include evaluations on more tasks.\n\n> **Q2:** Table 1 is hard to read, maybe a better metric for this problem is required.\n\nWe are sorry for the inconvenience. The oracle baselines and the estimation errors might not be intuitive. We propose to include graphical representations of the oracle baselines from $oracle^{4}$ to $oracle^{128}$ of all settings in the final draft upon acceptance. \n\n> **Q3:** The estimation variance is also critical for this problem, how is the variance of each method (including baselines) under different prompt and settings, does the meta-model shows a small variance across various scenarios? If the authors can elaborate on this point, the method's effectiveness will be further confirmed.\n\nWe thank the reviewer for the suggestion. In response, we show the standard deviation of each method across all test prompt permutations below. Notably, in most settings, the meta-models achieve a lower standard deviation than the baseline methods. When compared to the actual accuracy distributions, the standard deviation of the estimation is **significantly smaller**, indicating a non-trivial improvement. Overall the meta-model makes stable accuracy estimations across different settings. We promise to update the variance to the results table upon acceptance.\n\n| Methods | | LLaMA-7B |  |  | LLaMA-13B | |\n| :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |  :-----------: |  :-----------: |\n|  | MMLU  | MCQA | CBQA | MMLU | MCQA | CBQA|\n| MLP | 7.06 &pm; 1.03 | 8.82 &pm; 2.64 | 6.32 &pm; 2.13 |5.80 &pm; 0.63|11.04 &pm; 3.30|6.32 &pm; 2.13|\n|3-NN | 5.98 &pm; 0.97 | 5.62 &pm; 2.35 | 7.10 &pm; 2.15 |5.50 &pm; 0.66| 11.26 &pm; 4.56 | 7.10 &pm; 2.15|\n|XGBoost | 5.42 &pm; 1.09 | 5.22 &pm; 2.17 | 7.00 &pm; 2.44 |5.00 &pm; 0.62| 11.52 &pm; 5.20 | 7.00 &pm; 2.44|\n|AvgTrain| 5.26 &pm; 1.40| 10.26 &pm; 2.50 | 11.62 &pm; 5.88 |9.90 &pm; 1.70| 11.40 &pm; 3.36| 11.62 &pm; 5.88|\n|AvgConf| 5.54 &pm; 0.54| 6.88 &pm; 3.14 |n/a |5.10 &pm; 0.86|8.58 &pm; 1.75|n/a|\n|TS|5.6 &pm; 1.63| 6.32 &pm; 3.58 |n/a|14.06 &pm; 1.53|14.00 &pm; 6.95|n/a|\n|ATC|20.34 &pm; 4.1| 34.66 &pm; 9.36 |31.8 &pm; 13.44|20.50 &pm; 4.92|24.14 &pm; 7.72|31.80 &pm; 13.44|\n|ACC| 31.08 &pm; 6.32 | 39.00 &pm; 10.52 | 23.70 &pm;10.44 | 45.50 &pm; 11.74 | 50.34 &pm; 12.84 | 29.30 &pm; 12.38|\n\n| Methods | | OPT-6.7B |  |  | OPT-13B | |\n| :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |  :-----------: |  :-----------: |\n|  | MMLU  | MCQA | CBQA | MMLU | MCQA | CBQA|\n| MLP | 5.70 &pm; 0.77 | 6.66 &pm; 1.25 | 7.28 &pm; 2.75 | 6.30 &pm; 1.14 | 7.18 &pm; 1.69 | 5.90 &pm; 1.18 |\n|3-NN | 4.06 &pm; 0.33 | 2.98 &pm; 0.57 | 6.46 &pm; 2.05 | 5.16 &pm; 0.24 | 2.78 &pm; 1.06 | 5.84 &pm; 2.19 |\n|XGBoost | 3.76 &pm; 0.32 | 2.60 &pm; 0.73 | 8.06 &pm; 3.07 | 4.54 &pm; 0.35 | 2.54 &pm; 1.16 | 6.00 &pm; 2.43 |\n|AvgTrain| 3.48 &pm; 0.37 | 10.66 &pm; 1.75 | 8.38 &pm; 3.86 | 4.28 &pm; 0.35 | 11.46 &pm; 2.15 | 18.00 &pm; 3.77|\n|AvgConf| 8.42 &pm; 0.60 | 13.42 &pm; 1.70 | n/a | 6.20 &pm; 0.77 | 7.14 &pm; 1.77 | n/a|\n|TS| 3.92 &pm; 0.17 | 4.60 &pm; 0.80 | n/a | 4.36 &pm; 0.31 | 2.72 &pm; 1.03 | n/a|\n|ATC| 24.54 &pm; 2.12 | 37.64 &pm; 7.93 | 32.16 &pm; 14.60 | 24.72 &pm; 3.32 | 29.4 &pm; 10.28 | 30.34 &pm; 12.48|\n|ACC| 26.68 &pm; 4.42 | 33.16 &pm; 9.80 | 17.54 &pm; 5.92 | 26.82 &pm; 5.28 | 33.68 &pm; 12.06 | 19.34 &pm; 6.30|\n\n> **Q4:** Line 78 'conducting extensive ICL experiments spanning 91 datasets', I didn't see the results on 91 datasets in the subsequent sections.\n\nWe are sorry for the confusion. 91 is the total number of datasets across all 3 dataset collections (MMLU, MCQA, and CBQA). The full dataset collection is in Table 5 in the appendix. For evaluation results, we show the average results across all the datasets in each dataset collection. \n\n> **Q5:** It seems that the problem is highly influenced by label distribution of a task?\n\nWe assume that the reviewer refers to the label space of the multiple-choice QA tasks as the closed-book QA tasks would not have a fixed label space. We would like to clarify that all implementations of the dataset collections are from Huggingface Datasets and there is no sign of label distribution skewness in our experiment data. Meanwhile, there is no evidence that the estimation results would be affected by the label distributions of tasks. \n\n*We hope our responses have addressed the concerns of the reviewer and we are happy to answer any further questions.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem proposed in this paper is interesting, and the authors formalize the problem well, accompanied by several strong baselines.\n2. The confidence profile meta-model is effective across different datasets on the dataset-level ICL estimation and shows potential in prompt selection.", "reasons_to_reject": "1. As a general problem for ICL, the benchmark proposed in this paper should include more tasks and organize them better.\n2. Table 1 is hard to read, maybe a better metric for this problem is required.\n3. The estimation variance is also critical for this problem, how is the variance of each method (including baselines) under different prompt and settings, does the meta-model shows a small variance across various scenarios? If the authors can elaborate on this point, the method's effectiveness will be further confirmed.", "questions_to_authors": "1. Line 78 'conducting extensive ICL experiments spanning 91 datasets', I didn't see the results on 91 datasets in the subsequent sections.\n2. It seems that the problem is highly influenced by label distribution of a task?\n", "paper_topic_and_main_contributions": "This paper focuses on estimating the in-context learning (ICL) ability of LLMs on different tasks. Two main contributions are: 1. first formalizing the problem of few-shot ICL accuracy estimation: previous works focus on example-level calibration, but this work proposes to work on dataset-level estimation. (new problem, new benchmark) 2. proposing the method of training meta-model based on LLM confidence and accuracy for the dataset-level estimation problem. (new method)", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. May be the '91' (Line 78) is a typo."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a meta-model to estimate the accuracy and confidence of other language models with in-context learning in the absence of labeled test data. This approach has the potential to considerably save time and resources, and the experiments are solid and well-motivated. Multiple reviewers expressed concerns about the applicability of this method, but this issue seems to have been answered in the rebuttals.
Decision: Accept-Findings