How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench
Authors: Qinyuan Ye
Harvey Yiyun Fu
Xiang Ren
Robin Jia
Venue: EMNLP 2023 Findings (2023)
Keywords: large language models
performance prediction
benchmarking

Abstract: We investigate the predictability of large language model (LLM) capabilities: given records of past experiments using different model families, numbers of parameters, tasks, and numbers of in-context examples, can we accurately predict LLM performance on new experiment configurations? Answering this question has practical implications for LLM users (e.g., deciding which models to try), developers (e.g., prioritizing evaluation on representative tasks), and the research community (e.g., identifying hard-to-predict capabilities that warrant further investigation).

We study the performance prediction problem on experiment records from BIG-bench. On a random train-test split, an MLP-based predictor achieves an $R^2$ score greater than 95%, indicating the presence of learnable patterns within the experiment records. We then formulate the problem of searching for "small-bench," an informative subset of BIG-bench tasks from which the performance on the full set can be maximally recovered. We find a subset as informative as BIG-bench Hard for evaluating new model families, while being $3\times$ smaller. Additionally, we find competitive subsets by clustering task representations learned by our MLP-based predictor and selecting tasks close to cluster centroids, highlighting the importance of task diversity in constructing "small-bench."

How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench
Qinyuan Ye Harvey Yiyun Fu Xiang Ren Robin Jia
University of Southern California, Los Angeles, CA, USA {qinyuany, harveyfu, xiangren, robinjia)@usc.oo edu
Abstract
Model Family
param
Task
shot
Perf.
GPT-3
3B
strategy qa
0
0.48
We investigate the predictability of large lan- guage model (LLM) capabilities: given records of past experiments using different model fam- ilies, numbers of parameters. tasks, and num- bers of in-context examples, can accurately predict LLM performance on new experiment configurations? Answering this question has practical implications for LLM users (e.g., de- ciding which models to try), developers (e.g., prioritizing evaluation representative tasks), and the research community (e.g., identifying hard-to-predict capabilities that warrant further investigation).
BIG-G T=1
elementary math
PaLM
64B
code_ line desc 2
0.23
GPT-3
elementary math
How predictable are LLM capabilities?
Task 87
Task 6
Task 214
Task 127
Task 32
BIG-G
PaLM
GPT-3
New Model
Test
BIG-bench
"small-bench"
We study the performance prediction problem on experiment records from BIG-bench. On a random train-test split, an MLP-based pre- dictor achieves an R2 score greater than 95%, indicating the presence of learnable patterns within the experiment records. We then formu- late the problem of searching for bennhh. an informative subset of BIG-bench tasks from which the performance on the full can be maximally recovered. We find subset as infor- mative as BIG-bench Hard for evaluating new model families, while being 3x smaller. Addi- tionally, we find competitive subsets by cluster- ing task representations learned by our MI.P- based predictor and selecting tasks close to clus- ter centroids, highlighting the importance of task diversity in constructing "small-bench."
How to evaluate new models within budget constraints?
Figure 1: Overview. We study the problem of (1) pre- dicting LLM performance on new experiment configu- rations; (2) searching for subset of tasks which most informative for predicting performance remaining tasks when evaluating new model family.
et al., 2022; Touvron et al., 2023). Given the com- binatorially large space of possible experimental configurations, running all possible experiments for a new set of LLMs impractical. This begets a critical question: to what extent can we predict the capabilities an LLM given experimental setting?
Studying this problem helps address various practical issues. For LLM users, a performance prediction model could offer guidance for exper- iment design and decision-making by answering questions such as, "What model scale and how many shots are necessary to attain satisfactory per- formance for my task?" For LLM developers and the research community, performance prediction model could lead to insights into LLM capabili- ties by identifying which capabilities are hard-to- predict and require further investigation, and which capabilities are highly correlated and may be depri- oritized during evaluation to save budget.
1 Introduction
Large language models (LLMs) have revolution- ized natural language processing (NLP) research. Typically, when researchers introduce new set of LLMs, they release them in various sizes and con- duct extensive evaluation on different tasks, while also considering different experiment configura- tions, such as prompting strategies and the number in-context examples (Black al., 2021; Zhang
"Code can he found at https: //gp::///iihhnn wttb. www.lcoom/iiu///eee INKKUS// predicting-bie bigh bench. bincc...h.......

---

We investigate the predictability of LLM capa- bilitics on the BIG-bench (Srivastava et al., 2023) evaluation suite, as includes a vast collection of experiment records. BIG- BJG-bench is a collaborative initiative aimed at "probfing] large language mod- els and extrapola([ing] their future capubilities." It has extensively evaluated various state-of-the-art LLMs on diverse set of tasks contributed by the community. We gather and carefully filter these records, yielding a total of 56k+ records which we use as the "dataset" for our analysis.
specific scenarios (e.g., when related emergent task is present in the training set) our model can accurately predict emergent abilities.
4. BIG-bench Lite and BIG-bench Hard (Suzgun et al., 2023), two subsets of BIG-bench com- monly used for evaluating new models, are sub- optimal if the goal is to recover the performance remaining tasks. We are able to find a subset that is informative as BIG-bench Hard while being 3 smaller by using randomized search. 5. Task diversity and task value are critical factors in constructing "small-benchh task representations learned by the MLP-basee predictor and selecting tasks close to cluster cen- troids, we obtain competitive "small-hench" can- didates. This strategy is further improved by incorporating task value information.
We first formulate the problem of performance prediction given experiment configurations such as model family, model scale, task, and the number of in-context examples used. We compare various matrix completion, tree-based, and neural network methods in random train-test split scenario ($3). Further, we design and experiment with various data splits, representing different types of general- ization challenges, to simulate practical situations researchers may face ($4).
By clustering
Related Work
Scaling Laws and Emergent Abilities. Pre- training scale is critical to language model capa- bilities. Research on scaling laws (Kaplan et al.. 2020; Rae et al., 2021; Hoffmann et al., 2022) aims to categorize the relationship between pre-training compute, corpus size, model size and the test log- likelihood loss. Our work can be loosely consid- ered as an extension to scaling laws, with three notable distinctions: (1) we focus on predicting downstream task performance; (2) we use model scale along with other experiment configuration in- formation; (3) mainly experiment with machine learning methods instead of explicit power laws. In this same vein, recent work has studied the effect of scale in a "pre-train then fine-tune" paradigm (Tay et al., 2022) and has explored non-monotonic scaling laws for complex scaling behaviors (Ca- ballero et al., 2023). Another important observa- tion about scale is that very large language models exhibit emergent abilities (Wei et al., 2022a), which are described as "unpredictable." In this work we empirically examine this claim and quantify the prediction errors under various assumptions.
We then consider the problem searching for "small-bench," a compact and informative subset of BIG-bench. This subset should allow for maximum recovery of performance on the complete BIG- bench, enabling efficient evaluation of new LLMs while maintaining evaluation generality. We formu- late this subset search problem ($5) and empir- ically compare various search methods, clustering- based subset construction methods, along with widely-adopted subsets such as BIG-bench Lite and BIG-bench Hard (Suzgun et al., 2023).
Our key findings are summarized as follow:
1. LLMs' performance on BIG-bench follows pre- dictable patterns. In the default random train-test split scenario, our best predictor, an MLP model, achieves an RMSE lower than 0.05 (i.e., on av- erage mis- mii-ppreicct by 0.05 when the range is [0, 1)) and an R2 greater than 95% (i.e., explains more than 95% variance in the target variable).
2. The predictor's performance is dependent on the assumptions of the train-test distribution.
a more challenging setting where we hold out the Cartesian product of complete model families (all model scales) and complete tasks (all num- bers of shots), the predictor's performance de- creases (R²2 95% 86%).
Benchmarking for LLMs. Along with the de- velopment and scaling of LLMs, there are contin- uing efforts to create benchmarks that assess the capabilities of these models. One general trend for these benchmarks is transitioning from single-task (Bowman et al., 2015; Rajpurkar al., 2016), multi-task (Wang et al., 2018, 2019), and finally to massively multi-task (Hendrycks et al., 2021; Srivastava et al., 2023). However, due to budget or
3. Performance of emergent tasks (Wei et al.. 2022a) is not entirely unpredictable. In gen- cral, performance of emergent tasks harder to predict than that of non-emergent tasks. In

---

API constraints, models are typically evaluated on only subset of the full range of available bench- marks. The selection often made arbitrarily by the models' developers, making it challenging to compare models in a fair and holistic way (see Liang et al. 2023, Fig. 4). In response to this issue, we study the "small-bench" problem and hope it offers insights on efficient benchmarking of LLMs.
Normalized performance y [0,1] Our goal is learn a regression model f that pre- dicts ý based on (I, Pporams L, Nahot).
Data Splits. We obtain a large set of experiment records D = {(1, [[l,aaa L, Rehot, v)} and split them into three non-overlapping subsets, Dtrain. random splitting
Dew Drest- By default, and adopt 10-fold cross validation." In subsequent sections of this paper, we also use other splitting strategies for controlled analysis ($3.4 and $4).
Performance Prediction. NLPERF (Xia et al., 2020) is a pilot work on performance prediction in NLP. focusing bilingual and cross-lingual tasks. It demonstrates the potential of selecting an informative subset of tasks for evaluation, which inspired our work on searching for "small-benchh Ye et al. (2021) extend NLPERF to account for fine-grained performance measures, confidence in- tervals and calibration. Zhu et al. (2022) study pre- dicting model's downstream performance (GLUB, Wang et al. 2018) 8) using probing tasks performance (SentEval, Conneau and Kiela 2018), and advocate for incorporating probing during pre-training. Our work aims to add to the discussion by focusing on the performance prediction of LLMs. Given the ongoing advancements and substantial influence of LLMs in the field of NLP, we believe this topic is both timely and relevant, potentially holding impli- cations for future development of LLMs.
Evaluation. We report root mean square error (RMSE) and coefficient of determination score (R2) the test set Dtest- RMSE is defined V E -y3² R² score characterizes the proportion of variance explained by the model, i.e.,
R² Explained Variance
> (g y) 2i=1
Total Variance
(g y) 2i=1
7 71
where
y
i=1
In the main paper, we focus on RMSE and R² scores as they are widely accepted evaluation met- rics for regression problems. In Appendix C.I we introduce two alternative metrics based on rank correlation and discuss our findings.
3 Performance Prediction on BIG-bench
3.2 Data
3.1 Problem Definition
We construct our dataset D from the BIG-bench repository: We design series of filtering crite- ria (e.g., excluding tasks where all models have accuracy, excluding tasks with <100 examples), which are detailed in Appendix A. After filtering, our dataset has 56, 14,143 experiment records. We list high-level statistics about this dataset in Table 1. We would like to highlight that this dataset cov- ers diverse tasks and models. According to Sri- vastava et al. (2023), tasks in BIG-bench cover "problems from linguisties,, childhood development, math, common- reasoning, biology, physics, social bias, software development, and beyond." We refer the readers Fig. and Table App. 3 in Srivastava et al. (2023) for an overview. We also made our best effort to incorporate all avail- able model families in BIG-bench. The six model
In this section, we focus on the problem of learning from experiment records of large language models, and predict the performance of an unseen combina- tion of experiment settings.
Notations. We use L to denote the model fami- lies in consideration (e.g., PaLM, GPT-3). We use T to denote the diverse collection of tasks in con- sideration (e.g., 2-digit subtraction, amoji_movie). Formally, an experiment record is defined by the following values:
Model family e C
Number of model parameters Apooamm N
Task being evaluated on t T
Number of in-context examples Nishot N
'Here we use Pparam as the general representation of model scale. It is important to acknowledge limitation: Pperam does not provide a comprehensive description of model scale. Pre-training compute and corpus size should he included should such information he available.
'For tasks metrics different range, e.g., [0, 100,, normalize the score be in [0, 1] for consistency. "More specifically, we first split into disjoint subsets, then rotate on which ones are Daeo Dreat hhttps: //g thub. com/ google/BIG-bene

---

Experiment Records
56,143
Matrix Completion
Trees
Neural Networks
Model Families
Model+Task Baseline
0.7068
BIG-G T=0, BIG-G T=1,
BIG-G Sparse, PaI.M
Adapted SVD
0,8082
GPT-3, Gopher
Model-madel
0.8999
Models
51
Task-task KNN
0.9048
BIG-hench Tasks
134
Random Forest
0.9397
BIG-bench Subtasks
313
Gradient Boosted Trees
0.9510
{nshoe}
{0, 3,
MLP
0.9508
0.6
0.7
0.9
Table 1: Statisties of BIG-bench experiment records after filtering. 'MMode"" is defined by model family and Nporam, e.g., PaLM 535B. 'The 313 subtasks fall into the 134 tasks. For simplicity we disregard the task-subtask hierarchy this study. the remaining of this paper, "tasks" refers BIG-bench subtasks.
Test
Figure 2: Model Comparison for Performance Predic- tion using Default Data Split. Gradient boosted trees and MLP achieve strong performance (RMSE 0.05 and R² > 0.95).
familics included are representative, and offer con- siderable diversity. We provide a brief summary of these model families (release date, company, model architecture choices, etc.) in Table 2.
Neural Networks. We train simple (g) multi- layer perceptron (MLP) models to predict ý from the input (1, Pparam L, Nshot)) Hyperparameters such as number of layers and hidden dimensions are determined based on Daco performance.
3.3 Compared Models
Featurization and Hyperparameters. Featur- ization for trees and MLPs are explained in Ap- pendix B. Hyperparameters and training details of these methods are discussed in Appendix D.1.
Matrix Completion. Our problem is analogous to recommender systems or 2D matrix completion, if "models" (described by I and ppprra) are con- sidered as "users," and "n-shot tasks" (described by t and nighot) are considered "items." Each n-shot task (item "rated" by each model (user w). With such adaptation, we consider the follow- ing matrix completion methods: (a) Model + Task Baseline: ÿ = + by +b;, where z is the global av- erage performance, b, represents the effect brought by user u, and by represents the effect brought by item i. These parameters are learned by minimiz- ing mean square eгror on Diroin- (b) Adapted SVD: /4 bu b, + Zn Pi- Compared to (a), additional vector Qu is learned for each user and p& for each item i. The term q Pt expected to model the interaction between user u and item i. (c) Model-model KNN: First find the top models most similar to u. then aggregate the performance of these k models on item i by using weighted averaging. (d) Task-task KNN: similar to model- model KNN, but finds similar tasks and aggregate performance these tasks instead.
3.4 Results and Analysis
Trees and MLPs achieve strong performance. We experiment with the prediction models men- tioned above and present their performance Fig. 2. (1) Tree- bascd methods and MLP outper- forms matrix completion methods by a large mar- gin. We hypothesize that the 2D user-item simplifi- cation may cause loss of information the input space. For example, the value of Rparam is merely used to distinguish different "users," and does not contribute to the computation of ğ directly. (2) Gradient boosted trees and MLP are the strongest among all compared models; both achieving RMSE < 0.05 (i.e., on average mis- mis-predict by < 0.05) and R² > 0.95 (i.e., more than 95% of variance in y is explained). This suggests that learnable patterns exist in LLM experiment records- LLM perfor- mance unseen configurations can be predicted considerable extent in the current setting.
Trees. We two common tree-based methods that can directly learn make predictions ý from the input (1, Pprrams t ): (e) Random Forest (Breiman, 2001) and (f) Gradient Boosted Trees (Friedman, 2001: Chen and Guestrin, 2016).
Performance varies on different test groups (Fig. 3 To have more fine-grained under- standing of the predictions, we group D ex- amples according to the features such as Nshots Nparams and model family and then compute

---

R2 each of these test groups. We use the MLP model predictions and present the results Fig. using dark blue bars find that it is harder to predict zero-shot perfor- mance than 2- or 3-shot performance. In terms of the model family 1, we believe the three BIG-G models (T=0, T=1, sparse) are easier to predict be- cause their pre-training pipelines are similar. For pparam, we group all models into four buckets. For example, bucket contains the smallest 25% mod- els. We observe trend that performance of larger models are harder to predict.
Trained on...
Single-group (1000 Exampies)
Singll-group (All Examples)
terms of Rshots we
Multi-group (All Examples)
(a) Group by nanat
1
2
3-
0.6
0.7
0.8
0.9
1.0
(b) Group by Model Family
BIG-G T-0
We also group Diest according to whether the task ( is an emergent task (see Appendix E in Wei al. 2022a). Our predictor achieves an R score of 0.94 on the emergent group and 0.95 on the non- emergent group. This suggests in general emergent abilities are indeed harder to predict.
BIG-G T-1
BIG-G sparse
GPT-3
Gapher
Multi-group training is helpful (Fig. 3, We further conduct a set of controlled experiments by only training on examples from the test group of interest (e.g., examples with ahoot = 0). We name them as single-group experiments opposed group experiments
PaLM
0.6
0.7
0.8
0.9
1.0
(c) Group by Mparam (4 Buckets)
Bucket 1
15mallest)
as
Bucket 2
done in previous sections where the predictor is trained on all groups. Notably, in all settings, multi-gre group R² is always larger than single-group R2, There are limited observations of Gopher models in the train- ing set, and they benefit from multi-group learning significantly (R² increases from 0.74 to 0.87). This reaffirms the claim that LLM performance exhibits shared patterns across various settings.
Bucket 3
Bucket 4
(Largest
0.6
0.7
0.8
0.9
1.0
Test Each Group
Figure 3: R Score when Grouped with Rihhtt 1, and Πporam- Example: Multi-group Vishor = means training the complete Deraio (containing all n,oot values) and evaluate on Alghot = examples in Drest. Single-group (1000 Examples) rishot means using 1000 naooo examples in Derai as the training data.
Some groups benefit more from multi-group training, some are intrinsically harder to predict. Our controlled experi- ments also allow us to distinguish between two fac- tors: the group is intrinsically harder to predict or the group benefits more from multi-group learning. In Fig. 3(a), results suggest that ahoot 0 group is not necessarily harder to predict than ahooo = 2 and Nihhtt 3 on its own (indicated by bars), but the latter two benefit more from multi-group learning. Typically, when evaluating LLMs on task, there is a huge performance boost when going from zero-shot to one-shot, and the performance improves more stably when more shots become available. It is easier to predict 3-shot performance when given 0, 1,22-ohot performance, than to predict
(Fig. 3,
O-shot performance when given 1,2,3-shot perfor- mance. This partly explains why the O-shot group does not benefit much from multi-group training.
In Fig. 3(b). BIG-G T=0 and BIG-G T=1 benefit from multi-group learning more than the GPT-3 model family, resulting in higher R² scores in the multi-group setting. In Fig. 3(C), we observe that larger models tend be intrinsically more challeng- ing to predict. This observation is more significant when the single-group training set size is controlled to be 1000 (represented by bars), where we ob- serve a clear trend that groups consisting of larger models achieve lower R² score.
Identifying most/least "predictable" tasks. We further experiment with grouping test examples ac-
"Noe that the denominator in R², total variance, will he different for each group.

---

Highest R²
Model + Task Baseline
Gradient Boosted Trees MLP
ma:three_dig lelpddiiiiiiiiiioe aadddiididnooniii control
0.9885
Task-task KNN
ma:two_digg
ler_ddiiii_muiiipliii multiplicationnconiii control
0.9818
Im:sentence
tonssentene__eeaationn_ negation_ison
0,9805
qa wikidata
0,9802
L1
ma:three_diglt http:/iggggtsubrrrtttoooooooo control
0.9788
(1. Porram Nshhc)
0.7
0.8
0.9
Tasks
R
linguistics_pure puzzles
4110 7146
L2.1 (l. Oparam
checkmate In_ one
-38.2871
gender_ inclusive_senttenees. german
-12.1233
paragraph_segaeeee segmentationn
-5,9370
authorship verphhiipeeefficctoosswpppeep
-1.8346
L2.2
Mahce)
400
-00
Figure 4: Most and Least "Predictable" Tasks Based on R² Score. ma stands for "modi- "ritttttteeti"", Im stands for "linguisii_mmppinng", Fig. and 10 illustrate their scaling behaviors.
L3 ())
Composition
Crsst x Trsst
cording to the task & they belong to, to identify the most and least predictable tasks. The five most predictable tasks (Fig. 4 Top) include qa_ wikidata and pegguittic_mapiingss which were tasks marked as having high linearity" in their scaling behavior (Srivastava et al. 2023, Sec 3.4). This observation is reasonable because high linearity typically im- plies predictability. However, modified arithmetic, a task marked as having high breakthroughness (Srivastava al., 2023) and emergence (Wei et al., 2022a), is considered highly predictable in ting. Our hypothesis is that the predictor is able to infer this by learning from experiment records with similar configurations: breakthroughness is observed with other models or other number of shots, the trained predictor is able to infer the trend for new experiment configuration. We believe it is still challenging to predict breakthroughnnessc- mergence in more restricted setting, e.g., based on task meta-data input text alone.
0.7
0.8
0.9
1.0
Test R
Figure Performance Different Prediction Mod- els on Challenging Train-Test Splits. As the setting becomes more challenging (L1 - L2 + L3 + L3 Com- position), performance gradually drops and variance creases. MLP is least sensitive to these changes.
(1, Pporam: f, Nishot) combinations. This is a rela- tively easy setting; for example, when the model family I, number of parameters Nparam: and task t are kept the same, it may be easy for model to pre- dict performance of nihoo
when the records of Nahho 1 and nahho 3 appear in Dtrein- A challenging data split would ensure that the com- binations (I, Rppram, t) the test set are com- pletely unseen in Dirains and the model is required to predict for all possible ahot values. Taking it a step further, one may want make predictions on an unseen configuration of (1, t) for all possible values of rpparom and nihhtt
For the five tasks with lowest R² scores (Fig. Bottom), we manually examined their scaling be- havior (Fig. 10) and indeed find these curves to be surprising. For future work, it will be interesting to investigate the underlying reasons, and to identify common characteristics shared among these tasks.
4.1 Train- Test Split Settings
To simulate these cases, we design and com- pare model performance on three additional set- tings 1.2.2, I.3).
4 Creating Challenging Train-Test Splits
L1: Random (1, Appramm t, Dishot), used in $3
Previously, we randomly split D into Dtrain. Dees, and Drest- ie., we randomly sampled
L2.1: Group by (I, Nparams t)
L2.2: Group by (1, Nshhtt
'Linearity measures "the extent to which performance improves reliably with scale:" Breakthroughness measures "the extent to which a model is able to learn task only once the model grows beyond a critical scale." See Srivastava et al. (2023) Appendix B for the formal definition.
L3: Group by (I,t)
For example, in L.3, we first group all experiment records in D according to (L, t), then create Dtrain. D Daens and D Drest by random splitting the groups.

---

Additionally, we make L3 setting even more challenging by holding out one entire subset of ftest x Trest- Specifically, we first select Ltests a subset of model families C L. and Treat, a sub- set of tasks S TT After this, Diost is defined as {(l, Rparam, t, Thshot: f)(l E Ciest, t E T This corresponds to a practical scenario where a new model family is developed, and researchers want take "sneak peek" the full picture of its capabilities- evaluate on a subset of tasks (i.e., Ttroin T Test)) and predict the model's perfor- mance on the remaining tasks (i.e., Trest). We refer to this as the "L3 Composition" setting.*
new model family is evaluated on it, the per- formance of the remaining tasks can be maximally
recovered? In the following we give a formal defini- tion of this problem ($5.1), construct "small-bench" candidates using different search algorithms and strategies ($5.2), and present our findings ($5.3).
5.1 Problem Definition
Our goal is to find Ttrain. a subset of all tasks T, that are selected and used for evaluating new model families Ctest. We use b to represent the evaluation budget, i.e., IT.rann = h. We use Trest = T Tirain to denote the tasks whose performance we wish to recover. The problem of finding the optimal T can be formulated as the following:
4.2 Results and Analysis
Main Results. Results on four representative models in these settings are visualized in Fig. 5. We observe that as the settings becomes more chal- lenging (LI + L2 - L3 + L3 Composition), per- formance gradually decreases and the standard de-
(Trost X Lrest)
arg max Terain
s.t. Traain ST, |Tirainl = b
R2 (Trest X Lrest) represents the R2 score on Tiest X Crest when a predictor trained on the remaining experiment records, as previously done in $4.
viation increases. Another important observation is that though MLP and gradient boosted trees are comparable in the LI setting. MLPs are less sen- sitive to the increased difficulty (performance de- crease is smaller and standard deviation is smaller).
Evaluation. Ideally the optimal Tarain should al- low to predict the performance of any new model family, without overfitting to specific held-out model family. To evaluate this, we adopt nested cross-vaie validation on C during evaluation of se- lected Ttrain. Specifically, given that |C| = 6, we create 6 x 5 = 30 different ways to hold out one model family as Ldev and one model family Ctsst- We then train 30 prediction models and port the average of 30 R² (Trest x Lisst) scores.
Sample Prediction Results in L.3. In Fig. 7 we visualize predictions four sample (1, combi- nations by the MIP model two achieving high R2 scores and two achieving low R2 scores. We have two high-level observations: (1) Predictions are more accurate on (1, () combinations which has observations on similar tasks t' or similar models families I' in Draain (2) Over-estimation is com- mon type of mistake made by our trained prediction models. We observe several cases of "false posi- tives" of emergent abilities. Due space limit, we defer more discussion in SC.2.
5.2 Compared Methods
We consider different evaluation budget h E {4, 8, 16, 24, 32, 42}. We compare the following methods for selecting Tirain-
5 Searching for "small-bench"
Baselines. (a) BIG-bench Lite (Srivastava et al., 2023): A subset of BIG-bench for cheaper evalu- ation, proposed in the original BIG-bench paper. |Traann| 42 for BIG-bench Lite. 9 (b) BIG-bench Hard (Suzgun et al., 2023): A subset of BIG- bench containing challenging tasks that cannot be solved with direct in-context learning but can be improved with chain-of -houhht prompting (Wei et al., 2022b). |Tirain| 24 for BIG-bench Hard. (c) Random: For each b, randomly sample TTrain
There has been recent emphasis on assessing the generality of large language models, which in- volves evaluating these models numerous tasks and scenarios. However, it will be extremely ex- pensive to conduct all these experiments every time a new model is developed in the future. Extend- ing from the holding out Liest x Trest setting in $4.1. in this section, we formulate and study the problem of searching for "small-bench." Can we find a subset of BIG-bench tasks, such that when
"Tasks in BIG-hench Lite/Hard are listed in Appendix G. To the best of our knowledge, the selection process for BIG- hench Lite is not disclosed.
*In SD22 we describe our efforts to ensure that the perfor- mance is as comparable as possible across settings.

---

such that [Ttrain| b. We report the mean and standard deviation of these runs.
0.85
0.80
Search Algorithms. (d) Greedy Search: Based on the search result train at budget b 1, enumer- ate all tasks not present in ((b-1) and select the train
30-Fold R (Tond x Lnwe)
0.75
BB-Hard
one task that achieves the highest R22 test x Ltstt) to form the Ttrasn at budget b. (e) Best of 5000:
Re?(Tee))
DB-Lite
Baselines
Random
For each b, randomly select 5000 Tirain and select the one achieving the highest (Cdev x Tiest).
0.65
Search Algarithms
Clustering-
Gready Search
Kimeans
Best of 5000
Kmeeas Task Value
Note that these search algorithms optimize R2 (Teest x devu) during search, to ensure that Trest Liest is held-out for evaluation. Addition- ally, to make the search computationally tractable, we only use fixed fold from the 30 folds during search. We discuss the impact of these experimen- tal decisions in Appendix C.5.
0.60
4
16
24
42
Size of Teaa
Figure "Small-bench" Search Results. X-axis: size of "small-bened" bench" (Tirasn), Le., number of tasks selected for evaluating new model family. Y-axis: 112 score on recovering performance of remaining tasks. The complete BIG-bench will be at (313, 1.0) in this fig- ure. Takeaways: (1) BIG-bench Lite and Hard are sub-optimal for recovering performance on remaining tasks; (2) Task diversity and task value are important for constructing effective "smaal-bench" candidates.
Clustering-baeed. We hypothesize that a good "enall-bench" should be diverse (covering the task space comprehensively while avoiding redundancy by excluding similar tasks) and representative (each selected task providing informative insights for recovering the performance of other tasks). To validate this, we use the following methods to con- struct Ttrain. (f) k-means: We extract the task representations learned by our MLP models in $3. We apply k-meaas clustering to these represen- tations, group them into clusters, and then select the task closest to the centroid of the each cluster. (g) k-means Task Value: We first calculate the task value for each task in T by aggregating their contributions from the Best of 5000 search history. For example, if a task is present in 20 trials out of the 5000, its task value will be the average of the R2 scores from those 20 trials. We then incorporate this information into k-means clustering, by select- ing the task closest the centroid among tasks that are top 25% valuable globally.
sults and find several cases where BIG-bench Hard fails to represent the complete BIG-bench. For ex- ample, according to full BIG-bench performance recovered from BIG-bench Hard, BIG-G T=1 2B better than GPT-3 Large; however, according to the ground-truit BIG-bench performance, GPT-3 Large is better than BIG-G T=1 2B, which is cap- tured more accurately when a 24-task small-bench candidate is used. See Table 3 for the details.
It is important to note that BIG-bench Hard was not specifically designed for our goal, and thus is not expected to be competitive in our problem. Yet it is surprising that it underperforms randomly- sampled subsets. As general recommendation for evaluating newly-developed models, suggest using the Tirain subsets found by solving the op- timization problem in $5.1. If there is a specific evaluation goal in mind (e.g., focus on frontier, as in the case of BIG-bench Hard), Ttrain should still be manually selected.
5.3 Results and Discussion
We visualize the results of all compared methods in Fig. 6. We have the following key observations.
BIG-bench Hard and Lite are sub-optimal for recovering performance on remaining tasks. A 8-task subset found by Best of 5000 and randomly- sampled 16-task subsets can match the 24- task BIG- bench Hard for this goal. We further examine the re-
Greedy search is unstable and finds sub-optimal solutions. Search algorithms consistently out- performs randomly sampled Tirain sets; however, greedy search appears to be unstable, with occa- sional performance drops as the budget increases. Furthermore, at b = 42, it underperforms the Best of 5000 approach. We include additional results on other search algorithms, including beam search
1"The task / is represented as an one-hot feature in the input space. Thus, for each task, there is a vector corresponding to each task in the first layer of the MI.P. We refer to these as "task representations."

---

and simulated annealing, in $C.4, where ob- serve similar instablitiy. One possible explanation is the complexity of the search space, where the greedy search algorithm cannot guarantee finding the optimal solution. The gaps between the search objective (Ldev X Tiest in one fold) and the evalua- tion objective (Crest x TTest in 30 folds) could also contribute to this issue ($C.5).
tasks, making it hard compare different models and quantify the progress in LLM development. Moreover, task selection is often heuristic, follow- ing past practices, or chosen arbitrarily without principled justifications. We anticipate more active discussion on establishing evaluation practices that assess LLM capabilities efficiently, reliably and rigorously, and we hope work provides use- ful insights towards this. Related to our efforts searching for "small-benchh Perlitz investigate the impact of benchmarking options on the trade-off between computational efficiency and reliability, and develop HE,M, an efficient alternative to HELM (Liang et al., 2023). Vivek et al. (2023) propose Anchor Point Selection to select representative examples in the test set and reduce evaluation cost at the instance-level.
Task diversity and task value are important fac- tors for constructing "small-bench. Firstly, k- means is comparable to surpasses Best of 5000, despite that it is not explicitly optimized for the R² objective. This supports the notion that diversity is an important factor in constructing "small-bench." This finding also suggests that the MLP models for performance prediction produce meaningful task representations as side product. Secondly, k-means + Task Value is comparable to or out- performs k-means, confirming that task value is another important factor for constructing "small- bench," complementing the diversity aspect.
(2023)
Broadening observations on LLM capability landscape. Complementary to BIG-bench, sev- eral ongoing initiatives, such as HELM (Liang et al., 2023), Open LLM Leaderboard!! and Eleuther AI LM Harness are dedicated sys- tematically evaluating existing LLMs. Integrating insights from these great initiatives into future work has the potential to enhance the accuracy of L.LM performance prediction and deepen our understand- ing of LLM capabilities. Additionally, it would be intriguing to take into account recent advances such as chain-of of-thought prompting (Wei et al., 2022b) and instruction tuning (Sanh et al., 2022; Ouyang et al., 2022), and systematically measure their fects on L.M capabilities.
6 Conclusion and Future Work
In this work, we began with the question, "How predictable are large language model capabilities?" and conducted detailed case study on BIG-bench. We first formulated the machine learning prob- lem of predicting performance given configurations such as model family, the number of parameters, task, and the number of in-context learning exam- plcs. Our strongest prediction model achieves an R2 score greater than 95%, which suggests past LLM experiment observations can be used to pre- dict the performance of new experiment configura- tions. To address the problem of increasing evalua- tion cost on massively multi-task benchmarks, introduced the problem of searching for an infor- mative semall-bench." Results suggest that popular subsets such as BIG-bench Lite and BIG-bench Hard are not optimal for this purpose. Instead, sub- sets characterized by diversity and high task values offer competitive "small- -encch" candidates, high- lighting the importance of these two factors.
Limitations
Limited to BIG-hench results. We choose BIG- bench for our study due to its extensive collection of experiment records. Though it offers consider- able diversity in terms of tasks and models, several limitations exist. (1) Tasks: It's important to note that BIG-bench tasks are sourced from the research community and may not accurately reflect the ac- tual distribution of tasks encountered by LLMs in real-world scenarios. Therefore, our study has lim- itations in terms of generalizing our conclusions to the real-world task distribution.
In closing, while our study primarily focused on the predictability of LLM capabilities, we hope to initiate discussions on the following broader topics.
(2) Models: Though we have made every effort to incorporate as many model families as possible, there only
Rethinking LLM Evaluation. Currently, there is a lack of consensus regarding evaluation prac- tices for newly developed LLMs. Often times new LLMs are evaluated on different set of selected
httpss///heegiiiggeerrrssssseeep hot.s/pacee//tuggiiigaaaa.t. open_ Le___em__eadeebban
'nttpss:// //gi thub. com/E com/FleutherAT/ 1m- wwweeeeaiinnn-tee harness

---

6 model families in our experiment record dataset derived from BIG-bench. Such scarcity introduces instability and increases the difficulty in investigat- ing the "small--bench problem.
the DARPA MCS program under Contract No. N660011924033 the Defense Advanced Research Projects Agency with award W911NF-19-20211 NSF IIS 2048211, 1, and gift awards from Google and Amazon. HF was supported by a USC Provost Fellowship Award. RJ was supported by an Open Philanthropy research grant and Cisco Research Award.
Limited to publicly-vaillale LLM meta-data. LLMs capabilities are dependent many factors, beyond the model family and number of param- eters Pperra used in this study. Factors such as pre-training stability, convergence, pre-train corpus composition, etc., all play important roles. How- ever, we often don't 't have access to this information. In this work, we assume that the input features (1, Rparam) can capture such information during training implicitly. In the future, we believe our method can be expanded to include additional pre- training meta-data when they become available.
References
Sid Black, Leo Gao, Phil Wang. Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow. In GPT-Neo.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empiri- cal Methods Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Limited to interpolation settings. Our experi- ments mainly concentrate on interpolation settings, where the combinations of (1, Pparom: I, Mahot) ) are new in the test set, but each of the input element is seen at least once in the training set. As LLMs continue to grow in size, very important aspect is predicting performance of larger models in an extrapolation setting. We present some preliminary findings in this setting SC.3.
Leo Breiman. 2001. Random forests. Machine learning. 45:5-32,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tlya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Limitations in evaluation metrics. We use RMSE and R² as they are widely-used metrics for regression tasks. However, both metrics have their limitations for our problem, especially in the con- text of conducting group-wise comparison ($3.4). RMSE does not account for the variance in the tar- get variable. A low RMSE value for a task may be solely due to the fact that the task performance is relatively insensitve to different experiment set- tings. On the other hand, while R² score accounts for variance, it creates discrepancies when conduct- ing group-wise comparison since the denominator used to compute R2 differs for each group. To get a more comprehensive picture of our prediction model, we introduce task-average Pearson Correla- tion and Kendall Rank Correlation for evaluation and discuss our findings in Appendix C.I.
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. 2023. Broken neural scaling laws. In The Eleventh International Conference on Learning Rep- resentations.
Vladimír Cerný. 1985. Thermodynamical approach to the traveling salesman problem: An efficient simula- tion algorithm. Journal of optimization theory and applications, 45:41-51.
Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16. page 785-794, New York, NY, USA. Association for Computing Machinery.
Acknowledgements
We thank anonymous reviewers and members of USC NLP for their valuable feedback. QY and XR were supported in part by the Office of the Director of National Intelligence (ODNT), Intelligence Ad- vanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-200 2207220006,
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton,

---

Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Ko- reeda. 2023. Holistic evaluation of language models. Transactions on Machine Learning Research. Fea- tured Certification, Expert Certification.
Alexis Conneau and Douwe Kiela, 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Nelson F. Liu, Tony Lee, Robin Jia, and Perey Liang- 2023. Do question answering modeling improve- ments hold across benchmarks? In Proceedings of the ó1st Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 13186-1322 1886-13218, Toronto, Canada, Association for Computational Linguistics.
Jerome H Friedman. 2001. Greedy function approx- imation: a gradient boosting machine. Annals of statistics, pages 1189-1232.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin. Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder. Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-221 27744. Curran Associates, Inc.
Deep Ganguli, Danny Hemandezz, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Con- crly, Nova Dassarma, Dawn Drain, Nelson Elhage, Sheer El Showk. Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kernian, Shauna Kravec, Ben Mann, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei. Tom Brown, Jared Kaplan, Sam McCandlish, Christopher Olah, Dario Amodei, and Jack Clark. 2022. Predictability and surprise in large generative models. In 2022 ACM Conference on Fair- ness, Accountability, and Transparency, FAceT '22, page 1747-1764, New York, NY, USA. Association for Computing Machinery.
Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, Scheuer, and Leshem Choshen. 2023. Ef- ficient benchmarking (of language models). arXiv preprint arXiv:2308 1696.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring. Susan- nah Young. al. 2021. Scaling language models: Methods, analysis insights from training gopher. arXiv preprint arXiv: 11446.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In International Conference on Learning Representations.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.7555
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas, Association for Computational Linguisties.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scout Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint ar axXiv:2001.0836/
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Percy Liang. Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayaman, Yuhuai Wu, Anamya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan. Ce Zhang, Christian Alexander Cosgrove, Christo- pher D Manning, Christopher Re, Diana Acosta- Navas, Drew Arad Hudson, Eric Zelikman. Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao. Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksckgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji. Omar Khattab, Peter Henderson, Qian Huang, Ryan An- drew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Ieard, Tianyi Zhang, Vishrav Chaudhary, William Wang,
A Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor

---

Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazary, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anan- tharaman S. Iyer, Anders Johan Andreassen, An- drea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La. Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tahassum, Arul Menezes, Arun Kirubara- jan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat. Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe. Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatmia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argucta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison- Burch, Christopher Waites, Christian Voigt. Christo- pher D Manning, Christopher Potts, Cindy Ramirez. Clara E. Rivera, Clemencia Siro, Colin Raffel, Court- ney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Ju- rgens, Dehajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko. Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez- Plumed, Francesca Happé, Fran- cois Chollet, Frieda Rong. Gaurav Mishra, Genta In- dra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Glo- ria Xinyue Wang, Gonzalo Jaimovitch-Lopper Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hin- rich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jae- hoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng. James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein Jason Phang. Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song. Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-
Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tennnbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishman, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chia- fullo, Ksenia Shkaruta, Kumar Shridhar, Kyle Me- Donell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras- Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros-Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi. Manaal Faruqui. Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátýás Schu- bert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee. Michael Co- hen, Michael Gu, Michael Ivanitskiy, Michael Star- ritt, Michael Struhe, Michal Swędrowski, Michele Bevilaequa, Michihiro Yasunaga, Mihir Kale, Mike Cain. Mimee Xu, Mirac Suzgun, Mitc Walker. Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng. Nathan Andrew Chi, Nayeon Lee. Neta Gur- Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Fil- baghdadi. Omer Levy, Owain Evans, Pablo Anto- nio Moreno Casares, Parth Doshi, Pascale Fung. Paul Pu Liang, Paul Vicol, Pegah Alipoormolabushi, Peiyuan Liao, Percy Liang, Peter W Chang. Pe- Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu. Qinlang Chen, Ra- bin Banjade, Rachel Etta Rudolph. Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Ro- han Sikand, Roman Novak, Roman Sitelew, Ro- nan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov. Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh. Saif M. Mohammad. Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sehastian Bischoff, Sehastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan. Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixi- ang Shane Gu, Shubh Pachchigar, Shubham Tosh- niwal, Shyam Upadhyay, Shyamolima Shammie Debuath, Siamak Shakeri, Simon Thormeyer, Si- mone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene. Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad,

---

Steven Piantadosi, Stuart Shieber, Summer Mish- erghi, Svetlana Kiritchenko. Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbor- des, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Ko- mev, Titus Tunduny, Tobias Gerstenherg, Trenton Chang. Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victo- ria Nyamai, Vikas Raunak, Vinay Venkatesh Ra- masesh. vinay uday prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadol- lah Yaghoobzadeh, Yair Lakretz, Yangqiu Song. Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yu- fang Hou, Yuntao Bai. Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu, 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.
353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emer- gent abilities of large language models. Transactions on Machine Learning Research. Survey Certifica- tion.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma. brian ichter, Fei Xia, Ed H. Chi, Quoe V Le. and Denny Zhou. 2022b. Chain of thought prompt- ing clicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. 2020. Predicting performance for natural language processing tasks, In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 8625- 8646, Online. Association for Computational Lin- guistics.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery. Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. 2023, Challenging BIG-bench tasks and whether In Findings of the Association for Computational Lin- guistics: ACL 2023, pages 13003-13051, Toronto, Canada, Association for Computational Linguistics.
Zihuiwen Ye, Pengfei Liu, Jinlan Fu, and Graham Neu- big. 2021. Towards more fine-grained and reliable NLP performance prediction. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguisticx: Main Volume. pages 3703-3714, Online, Association for Computa- tional Linguistics.
ofain--o--hought can solve them.
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Met- zler. 2022. Scale efficiently: Insights from pretrain- ing and finetuning transformers. In International Conference on Learning Representations.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv prepriat arXiv:2205.0106
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Zining Zhu, Soroosh Shahtalebi, and Frank Rudzicz. 2022. Predicting fine-tuning performance with prob- ing. In Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing, pages 11534-11547, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. 2023. Anchor points: Benchmarking mod- els with much fewer examples. arXiv preprint arXiv:2309.08638:
A Filtering BIG-bench Records
We use the following criteria when filtering the records. After filtering we obtain dataset with 56k+ records which is described in Table 1. For now we limit the scope to predicting performance on the preferred evaluation metric.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understand- ing systems. In Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc.
1. Keep json tasks and remove programmatic tasks.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE:
A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages
2. Remove TO/T5 models because there are only 14 records for each of these two. Keep BIG-G, PaLM, GPT-3, Gopher models.

---

3. Remove BIG-bench subtasks whose perfor- mance on the preferred metric zero for all models.
be in the range of [0, exact_str_mato and mul siiple_choice_graeee already satisfy this con- straint. The reported rougel values are the range of [0, 100] and are multiplied by 0.01 to form our dataset.
4. Keep
experiments whose preferred metric
in exact_str_matoe multiple_choice ge_rade, sougeLsum.
C Extended Experiment Details and Results
This keeps 93% of all records before this step.
5. Remove entries of aggregating results from multiple subtasks the performance of task.
C.1 Additional Evaluation Metries for Performance Prediction
6. Remove subtasks with less than 100 examples because small sample size may lead to large variance during evaluation
In addition to RMSE and R² score, two common metrics for evaluating regression models, we in- troduce two new metrics, Task-average Pearson Correlation and Task-average Kendall Rank Cor- relation. The usage of Pearson Correlation and Kendall Rank Correlation inspired by Liu al. (2023) and we further adapt them be averaging across tasks.
We present a summary of the 6 model families in these records in Table 2.
B Featurization
In the main body of the paper we use the abstrac- tion of (1, Aparom: Nlshot) describe experiment configurations. In the actual training of tree-based models and MLP,, we modify how these features are represented. In particular, the input features contain the following:
Concretely, we first group the test set Dtest into IT| groups, based on the associated task t of each example. Within each group, we compute the Pear- son Correlation or Kendall Rank Correlation be- tween the predicted performance and the actual performance. Finally, the average of these rank cor- relation values across all tasks T then reported the task-average rank correlation. We report these numbers along with RMSE and R2 in Table Generally, prediction models with higher R² scores exhibit higher rank correlation. Exceptions emerge when closely comparing tree-based models and MLP models. While MLP models compara- ble outperform brsed models terms of R² score, tree-based models tend to outperform MLP models in terms task-avg (rank) correlation. Our further investigation reveals that tree-based models make more errors with large absolute errors, which are penalized heavily by RMSE and R22 (involv- ing taking square of these errors), whereas rank correlation is less sensitive to such errors.
1. I is converted as binary features for each model family, e.g., is_PaLM
2. (1, Mparam) is converted as binary features for each model, e.g., is Pa__M_M5553B..
3. There are numerical features for the num- ber of parameters: number of total parame- ters, number of non-embedding parameters, number of FLOP-matched non-embedding pa- rameters; and the natural log of these three values.
4. t converted as binary features for each task, e.g. is_code_ wecoodeeeeeeeeeeeeeeeeeeeeeeeeee denereeirppptio..
5. m is a binary feature for the preferred metric associated with the task I, e.g., is_exact_str_match BIG-bench defines preferred metric for each task, so in the ab- straction it is covered by t.
Throughout our paper, we primarily focus on ex- perimentation with MLP models due their faster runtime and the values of the learned representa- tions in MLP. practice, we recommend selecting methods based on the final goal: ML.P models for more accurate prediction in terms of exact values; free-based methods for more accurate ranking of different experiment settings.
6. Nlkhot is directly used as an input feature.
For numerical features (6 features for the num- ber of parameters and feature for the num- ber of shots), we use StandardScaler in the sklear libarary to normalize them. Addition-
C.2 Sample Predictions in the L3 setting
In Fig. 7 we visualize predictions on four sample (1,t) combinations by the MLP model. The left
ally, we normalize the performance value to

---

Model Family
Company
Release Date
Pre-train Corpus
# Parameters
Architecture Choices
Infrastructure
(T-d, T-1)
Google
Jun 2022
2.8T tokens
so 137B
relative attention
TPCv3, GSPMD
BIG-G Spane
Google
Jun 2022
2.8T tokens
51M 468
BIG-G Mixture-of ox-eets
TPCx3, GSPMD
PaLM
Google
Apr 2022
780B tokens
8B, 62B. S40B
SwiGLU activation, paralke) layers, dtc.
TPUs4, Pathway
GPT-3
OpenAl
May 2020
300B tokens
125M 179B
Alteratinn dense and spurse aitention
GPUss
Gopber
Desphind
Nov2021 2021
300B
tokens
44M 2803
RMSNomm, relative position cocodingg
vPC+3, model parallelism, etc.
Table 2: Summary of model famillies included in this study. These model families offer considerable diversity. Information in this table is aggregated from Srivastava et al. (2023); Chowdhery et al. (2022); Brown et al. (2020); Rae al. (2021)
BIG-G T=1 28 Wins
Tie
GPT-3 Large Wins
Conclusion
Performance Recovered from BBII (24 subtusks) Performance Recovered from small bench (24 subuskss)
20.1%
70.1%
9.8%
DIG G T-I 2B better
19.1%
56.3%
24.6%
GPT-3 Large better
Ground truth full BIG bench Performance
14.3%
55.3%
30.4%
GPT33 Large is better
Table 3: Using BIG-bench Hard and "small-bench" to recover performance and compare models. In this example, BBH is less informative in recovering performance remaining tasks, and thus is the comparison is less accurate.
two are cases achieving high R² scores. The right
(1, pprram) ) (GPT-3, 200B)
are cases achieving low R² scores.
E2.2:
Dtest contains examples with
For (1, 1) combinations achieving high n2 scores, our observation is that either a combination (1, E') exists in Diroin such that / and 1 are similar (e.g.. t and t' are two sub-tasks from the same BIG-bench task), or (t',t) exists in Dtrain such that I' and I are similar (c.g., both I and l' from the three BIG- G model families). Our interpretation is that the learned predictor is capturing model family simi- larities and task similarities and therefore predicts more accurately when I or has a similar counter- part in the training set.
(1, [pparam) (PaLM, 535B)
Relaxing Constraints by leaking 10% Dtest- Pure extrapolation may be extremely challenging. To better contextualize the results and understand limitations,
compare model performance in three slightly different settings. The first setting (S1) is holding 10% Dtrain as dev set for hyperpa- rameter selection and early stopping. This corre- sponds to the pure extrapolation setting, as no in- formation about Drest is available at training time. The second setting (S2) is holding 10% Dtest as dev set. Information about Dreat is indirectly leaked to model training. The third setting is leaking 10% Dtest during training. This third setting (S3) is mainly for reference.
For tl, combinations achieving low R² scores, we observe several cases of overestimating perfor- mance or predicting "false positives" of emergent abilities. The selection of these combinations largely due to using R²2 selection criteria-thh they have small total variance as the denominator for R22 score, so any overestimation will results in an extremely negative R² score. Nevertheless, these qualitative results suggest that overestimation is a common type of mistake made by our trained prediction model.
More specifically, we split the original Dtrain into 90% D D'rain and 10% D Also we split the original Drest into 90% D'est and 10% Decu2- In S1, model training, selection and evaluation is done with (D'rain: Dáevi; Drest). In S2, we al- low model selection with Diev2, the experi- ment is done with (Dirain Dec22; Dlest). In S3, leak D (which from the test distribution) to training time, and the experiment done with (D'rain U Ddev2, D D(est).
C.3 Performance Prediction in Extrapolation Settings
In the input space of our problem, Alshot and Rpparam are numerical features. Thus it is possible to test the extrapolation capabilities on these two inputs. We create three settings for testing the model's extrapolation capabilities:
Results and Findings. We present the results in Table 4. (1) Extrapolation in terms of nihhot is promising, achieving R2 which is greater than 0.9. However, extrapolation to increased model size remains challenging. This closely related the observation that emergent abilities is difficult to predict (Ganguli et al., 2022; Wei et al., 2022a).
El: Diest contains examples with ahoot = 3,
Dtrain contains examples with nghhot = 0, 1,2 E2.11
Drest contains examples with

---

medal:
taxe: 16c lhe Joon:
taxt: modiled antihmmete:
model PALM
task: medited anttmmatic
Bba_itte Celdonallyy.mmmg
dgltt conboo
tirce dlgp. subrrcctioon
12-
24
1-fat pred
7
p-doopoed
0.
2.h0.
pres
5-0h. and]
30"
10
17"
107
10'
30
Figure 7: Sample Prediction Results L.3 setting ($C.2). Left 2: (4, combinations achieving high /t2 score;
Right 2: (l,t) combinations achieving low R score.
(2) Performance in S2 is consistently better than S1. Note that the only difference between these two setting is the Dee used to do model selection. This suggests that the model overfits and fails to extrapolate well in the strict SI setting. Leaking some information about the test distribution (as in S2 and S3) can greatly help improve prediction accuracy.
Baselines
Beam Searth (#beeam-4)
Random
Simulated Annealing
30-Fold (Tiie x Coar)
0.80
0.75
BB-Ilard
C.4 Additional Results on "small-bench" search
We experiment with two additional search algo- rithms for "small-bench" search: (1) Randomized Beam Search: Similar regular beam search, ex- and we enumerate 1/q randomly selected task candidates at each timestamp. This ensures the search run- time is equivalent to greedy search. (2) Simulated Annealing (Černý, 1985): Initialize with seed Tirain: at time I, iteratively search in the neigh- bourhood of the Trrain at time - 1 and occasion- ally allowing uphill moves (i.e., moving towards worse solution). Results are visualized in Fig. Similar greedy search, these two search methods face optimization challenges discussed in $5.3 and may lead to sub-optimal solutions.
0.65
8
16
42
Size of Tsmin
cept that we maintain beam size
Figure Additional Semall-bench" Search Results. See $C.4 for in-depth analysis.
Gap between fold and 30 folds. Due to run- time concerns, we only launch search algorithms on fold, while at evaluation time we use all 30 folds. The 30 folds, derived from 6 distinct model families, may exhibit significant variations. Con- sequently, the search result fold may overfit that specific fold and become less optimal on all 30 folds.
C.5
Gaps between the "small-bench" search objective and evaluation objective
D Reproducibility
In $5.3, observe that the performance of greedy search is unstable and hypothesize that this is partly due to the gaps between the search and evaluation objective.
D.1 Hyperparameters and Training Details For all the following methods and for each data split setting (L1/22.///2.2//33133 Composition), we select hyperparameters based on the dev perfor- mance on the first fold, and select the best set of hyperparameters from 100 random combinations from pre-defined list.
Gap between Taco and Trest- To simulate the scenario that the prediction model is expected to make predictions an unseen model family, make sure that the search algorithm optimizes on Teev Liasst and holds out Trast X Liest for evalua- tion. This creates dev-test shift which may affect search algorithm results.
For Random Forest, we use the implementa- tion of RandomForestRegrest _ecuured_eroorrre error as optimization crite- rion. The hyperparameter candidates are sampled
from sklearn
library. We use squared

---

S1: Dac 10% Diroin
10% Desa RMSE R (t)
Dáeo
S3: Leak 10% Drest
RMSE (U)
RC (t)
RMSE ())
R (7)
More shots
E1: Hold out Rahut
0.0618
0.9297
0.0587
0.9367
0.0545
0.9454
Scaling up model
E2.1: Hold out GPT-3 200B
0.1314
0.6968 0.1535
0.1017 0.1923
0.8187
0.0830 0.1668
0.8791
F2.2: Hold out Pal .M 535B
0.2708
0.5731
0.6789
Table 4: Results of Extrapolation Experiments ($C.3).
from the following:
Model
Runtime (sccond)
Matrix+llask Baseline
"nLestinators": [30. 188, 1601.
Adapted SVD
2
3
"nax_ (None, 16, 32, 64, 1281,
Model-model KNN
2
4
"nin_samples_sple [2, 81.
Task-task KNN
5
"nin_samples__ea" [l, 2. 41.
6 "nax_foaturas"; [1.0, 8.8, 0.8, 8.7, 0.6, sqrt"].
Random Forest
98
Gradient Boosted Trees
242
7
"nax_samples"; (1.0, 8.9, 8.8, 8.7, 0.61
MLP
86
MLP (optimized for scarch)
28
For Gradient Boosted Trees, use the im- plementation of XGBRegressor from XGBoost li- brary (Chen and Guestrin, 2016). We use the reg: squarederro as objective. The hyperparam- eter candidates are sampled from the following:
Table 5: Runtime (Training-Evaluation) of Performance Prediction Models.
10-fold cross validation for all these settings, sim- ilar the practice in L.I. This ensures for every fold, the sizes of Dtroin/ Daev/ Drest are consistent across settings, and each example D appears Drest exactly once. In this case, the only changing variable is the data splitting strategy.
2
"n_astinators": [3a, 188, 300, 1888],
3
"learning_raae"" [0.1, 0.3, 8.5, 8.8, 1.61.
"nax_depth"; (None, 16, 32, 64, 1281, None
indicates no limit
"genna": [0. 8.1, 8.2].
6
"subsample"; [0.6, 0.7, 8.a, 0.9, 1.01,
L33 Composition: Holding out Liest X Trest- To align with the 10-fold setting in L1-L3 much possible, split L into 3 non-overlapping subsets, and split T into 3 non-overlapping subsets. This gives 3 3 9 different Drest for cross-validation.
For MLP, we implement with Pytorch. We use Adam optimzier and use MSELoss. The hyperpa- rameter candidates are sampled from the following:
"Ir": Cle-3, Je-4, le-4]
3
"batch_ [32, 64, 128].
D.3 Hardware and Runtime
"dropout": L0.0,
8.1, 8.21,
5
Chidden_dins"
Matrix completion experiments and tree-based ex- periments are done on server with 56 Intel Xeon CPU E5-2690v4 (@ 2.6 GHz). MLP experiments are done on one NVIDIA GeForce RTX 2080 Ti. The runtime for training model for performance prediction is listed in Table 5. For small-bench search, optimized MLP training by not saving copy of the best checkpoint in memory, and moving Dtrain and D to GPU before training to reduce GPU I/O. For Greedy Search, the performance pre- diction algorithm repeated for 12326 times, using approximately 5 days. For Best of 5000, the per- formance prediction algorithm is repeated for 5000 times, using approximately 2 days.
6
(128, 64, (256, (2,.0222.666...33..3 64.332. (28,64,37) (256, 128, (64, (24,32), (1), (64.)
7
8
10
1.
bniight_dccay [o.0, 8.001, 8.8881, 0.0a1 0.01].
D.2 Details Cross Validation
LI: Random Splitting. We first split /) into 10 disjoint subsets, and then rotate which ones are Daar and Diest To save computation budget, hyperparameter selection done on the D of the first fold.
T.2.1/1.../1.3:: Random Splitting Different Granularity. To ensure that the results are com- parable across settings as much as possible, we use
(Continued on next page)

---

E Full Results of Performance Prediction
RMSE
R² Score
Task Avg. Pearson
Task Avg. Kendall
1. Setting: Random
Nparamm L, Mahot): ). 10 folds
Model+Task Baseline
0.1220 (= 0,0016)
0.7068 (# 0.0067)
0.5207 (# 0,0078)
0.3617 (= 0.0077)
Adapted SVD
0.0986 0.0014)
0.8082 (+ 0.0053)
0.5149 (+ 0.0089)
0.3443 (= 0.0053)
Model-model KNN
0.0712 (+ 0.0016)
0.8999 (+ 0.0041)
0.7046 (+ 0.0085)
0.5133 (+ 0.0070)
Task-task KNN
0.0695 (+: 0.0012)
0,9048 (+: 0.0035)
0.7006 (+: 0.0091)
0.5109 (# 0.0078)
Random Forest
0.0553 0.0014)
0.9397 (1: 0.0030)
0.8378 (#: 0.0062)
0.6590 (= 0.0058)
Gradient Bousted Trees
0.0499 + 0.0013)
0.9510 (1: 0.0025)
0.8566 (# 0.0085)
0.6690 = 0.0060)
MLP
0.0500 (= 0.0009)
0.9508 (1: 0.0014)
0.8203 (# 0.0097)
0.6128 == 0.0065)
L2.1 Setting: Random (l, Bpereo t). 10 folds
Model+Task Baseline
0.1221 (#: 0.0042)
0.7057 (+ 0.0158)
0.4823 0.012)
0.3629 (=: 0.0152)
Adapted SVD
0.1000 0.0039)
0.8026 (1: 0.0135)
0.4575 (#: 001155)
0.3229 0.0121)
Model-model KNN
0.0721 (=: 0.0034)
0.8974 (1: 0.0085)
0.6669 (.018)
0,5043 0,0110)
Task-task KNN
0.0763 (= 0.0031)
0,8852 (#: 0.0078)
0.6102 (# 0.0190)
0.4548 (= 0.0136)
Random Forest
0.0724 0.0034)
0.8962 (#: 0.0109)
0.7468 (# 0.0152)
0.5830 (= 0.0101)
Gradient Boosted Trees
0.0676 += 0.0029)
0.9095 (#: 0.0084)
0.7458 (# 0.0155)
0.5759 (= 0.0094)
MI.P
0.0616 (+ 0.0022)
0,9251 (+ 0.0040)
0.7008 (+ 0.0163)
0.5244 (+ 0.0126)
L2.2 Setting: Random (1, Pahot) 10 folds
Model+Task Baseline
0.1315 0.0033)
0.6577 (1: 0.0288)
0,4556 (# 0.0327)
0.3373 (= 0.0229)
Adapted SVD
0.1208 (=: 0.0042)
0.7107 (#: 0.0324)
0,4338 (# 0.0229)
0.3095 (= 0.0170)
Model-model kNN
0.0983 (=: 0.0062)
0.8072 (+: 0.0339)
0.5885 0.020)
0.4287 (= (=00016)
Task-task KNN
0.0985 (+ 0.0060)
0.8079 (+ 0.0231)
0.6059 0+081)
0.4449 (- 0.0137)
Random Forest
0.0675 0.0031)
0.9098 (+ 0.0096)
0.7571 (+ 0.0144)
0.5862 (- 0.0107)
Gradient Boosted Trees
0.0624 (+: 0.0033)
0.9229 (1: 0.0092)
0.7819 (+ 1601
0.6057 = 0.0104)
MLP
0.0555 ( 0.0019)
0.9391 (# 0.0050)
0.7424 (# 0.0210)
0.5520 (= 0.0143)
1.3 Setting: Random (), 10 folds
Model-Task Baseline Adapted SVD
0.1321 0.0075)
0.6542 (+ 0.0302)
0-474 0.0288)
0-3575 0.0229
0.1 (+ 0.0088)
0.7146 (+ 0.0353)
0.4211 0.0099))
0-2979
0.0211) (+0001))
Model-model KNN
0.1028 (#: 0.0130)
0,7880 (#: 0,0531)
0,5954 0.022)))
0.4345 (= 0.0106)
Task-task KNN
0.1032 0.0076)
0.7878 (1: 0.0308)
0.5667 (1 0.0242)
0.4102 0.0156)
Random Forest
0.1015 + 0.0103)
0.7940 (+: 0.0414)
0.6617 (+ 0.0234)
0.4961 (= 0.0174)
Gradient Boosted Trees
0.0947 (+ 0.0084)
0.8212 (+ 0.0315)
0.6589 (+ 0.0302)
0.4827 (+ 0.0246)
MLP
0.0736 (=: 0.0064)
0.8922 (+: 0.0164)
0.6573 (+ 0.0156)
0. 0.4680 (= 0.0148)
1.3 Composition Setting: Holding out Crest
Trxst 9 folds
Model+Task Baseline
0.1383 0.0115)
0.6231 (1: 0.0265)
0.5099 (#: 0.0763)
0.3709 (= 0.0457)
Adapted SVD
0.1318 (1 0.0142)
0.6575 (1: 0.0460)
(4
0.3365 (= 0.0541)
Model-model KNN
0.1138 (+ 0.0173)
0.7398 (+ 0.0731)
0.5342 0.1.1198)
0.3918 (- 0.0956)
Task-task KNN
0.1152 (= 0.0139)
0.7362 (+: 0.0530)
0.5692 0.073)
0.4086 (= 0.0657)
Random Forest
0.1118 (=: 0.0154)
0.7475 (1: 0.0782)
0.6136 (1: 0.0781)
0.4347 (= 0.0494)
Grudient Boosted Trees
0.1072 (= 0.0104)
0,7706 (1: 0.0451)
0,5970 (1: 0,0941)
0.4034 (= 0.0588)
MLP
0.0843 0.0072)
0,8597 (1 0.0129)
0.6228 (# 0.0991)
0.4236 0.0675)
Table 6: Full Results of Performance Prediction.
Matrix Completion
Trees
Neural Network

---

F Scaling Behavior of Most/Least "Predictable" Tasks ($3.4)
BC-5 CT1 Diiin
Fot 377
Prancs
8000
R.C.
mway
9000
(a) modified_ariiee arithmeeti:: modiifed_artthmeeii:::
(b) modiffed_arithmettie two_ two_digit__o mwltiplication_ccommm control
(c) minguistie_mappingg
three thre__digit_ae seigiiiadditionocon control
sentence sentenee_negation json
B0
(e) coodified_arttmmetic
(d) gp__ikidata
three_ sese_diiii__aatee caaoil_uuutcctonooooaae control
Figure 9: Scaling Behavior of Tasks with Highest R Score. Figures are obtained from https: com/ google/BIG-bent

---

WC20 Dreas
490 0020 Domary
20 2aee
8P (AiNd)
(c) gender_ inclusive sentences_german
(a) linguistics_puzae puuzles
(b) checkmate in_one
800%
2020 DANG
authorstipp.vrrificttioo verification:
(d) paragraph segmentation
swapped
Figure 10: Scaling Behavior of Tasks with Lowest R Score. Figures are obtained from https://github.coom google/BIG-bench

---

"Small-bench" Search Results
G.1 BIG-bench Hard and BIG-bench Lite
The two subsets listed below slightly differ from the original subsets, because we filtered out several subtasks as described in SA.
2.
"bblite": ['bbq_ httpoooo.t_..onnabiai. htt...o....o...l._.aa.....eeeee
bbq_ htt.._ie..ooooossiilll.....eeaee statess__sttts
"bbq_iite_jsoo.
bbq__1tte_jsen:
"bbq_iite__jon:
"bbq lite "oo__iit__joon:
aog_ltte__50_:es__ma11
conbination:::
emerzent_propertiess. "om.aooaalloooois....lllooo..
"hinu knondo.nowwlegee
languaga_idantificaat "Iingutsticcsspprzlll "logic.grid_uuzzle" "logtcal_deduction
five_objects', degical_deductionnnsee sontrooosstt....s..ooo..eeeeeet obddeo..eeeeciinnooooooo.eo
decscll_eduutiiottiie three_objects"
novel_concepts". "operators""
"strange_stories boollen strange_storiess.nltiii..... htt.
'strategyqa
symbal_interrpreattion...o adverssraai...... sytbol_iiterpreetaiion..mm aog.aaiiooonooooo.gntgttie
ognooootit cymbol_interpretatiin..... tricky "witaninc
"sybbll imbbol__nterrretttionn
'winowhy'!
"b
"bbhard": ['coussi_judgment" "fatteettttttttttuuudneeee 'dstt_,nderstandinng
'disantiguatien__p" "dyck_languoges"
fornsl fuleelellllcicsssilloiissss.... five_objects logtcal__educctoon oet. soven..ccctssss "logical_ "logical_dducctton the.eddcttttttereeeeeeooooeo threecoeettt
"pennetric.shapes" "hyperbaton 'logicel_dee deductontdduuctio:::
movie_ rrvie_rrcoomendationn 'navigate'. "object_counting"
ionguins_in_a_tabl
wtwpen/nngooottoooorododoo 'ruin_names', 'ruinnnnnss,,.,,
translation.erre_.deee
'snarks"
imperrelsequennce
"tracking. trackknn_shhuffled shuffled seuuffeeddbjectte objects: five_aeeive
tracking_shuff shutigg_dufflledoofeettt seven_objests tracking_shufffldd_
three_objects
word_sorting"
G.2 Best of 5000
[' t'arithmaticcs.. s.g.sss.sttttttttracittnnn "goal_step_wrkihon goal_inference'. "linguistic_nappi
past cesst_tone__ie cense_irrogular_jass natural_instructionn
aywpppp_sserrttpoooooooo '],
3
8: [' ("arithretic: hit.::cs__.iiittniipiiii....
"chess_state_traae
real_neiunn 'elementary
question_oni
jiiguistic_neppinng
res__teese.regular..5s "nstural..nstruction 1
aeggerattion_freuunnn _winoarande_ouestionn wwt.:uuwwwwtt.o.ooiiiiioooo.oeo otg.:aameooccciaaaaaaaaaaaaaatto
natural_instruccione subeaske34
"natural_
cttptasckkoo_uscccooooiiiiii wwsc_combined
"physics"],
4
16: L' "iaheeii:._.diitt.cditin.. "arithnetie:3_dig http:::s__diittmmmliipliiiaaio
'arithnetic:4
bbq bog__iite.ionnphhssic...
disanbig cww.e..nd..feectttee..eeeeeee
"genn:s""essteeeeeee reettcc_s
durtii_atvvetttdrtttiio duration intrn subtask033
_wwttirc.wiite.nnooreee....e sociaal_eea']
24:
('anachronisms" w
'arithnetic:
"arithmetio:4
_digit__subractionn
ldyck_languagss'
gge::asset "implicatures"
imiss_asci1"
natural_ins
natural_instrucct
subtakkket
.rops__nsser_eeneratiirrr 'navigatt"""..
" "rensensc_.orcc__rramm=""" 'persianiiiinnn
"social_iss'
"strange_storrreeo
'winswhy
('aattrattparratiivvuueeerrr
4_distrastorss "arithaetaaiaiiiiiii
"arithmatic.22diiit...di
'arithnatic.2
i_digit_divission :
digiit_subtrection wrritheeticc__diiitt.ddee
"arrithtttic....lllll....iiiass 'bbe_lite_json=
"arithnetie:
bag_llte.je..edde.dddattttisasad "common_marphene" "elementary.nath.c.
aupetiinn.iitttttttttaaaaaaaaaa. "imliccture "Interset_.gonetryy shappes_ 'kannada"
question_kith__0
"spistemic
"gem:cs_restsurants"
identify..nath.hherrr
'lagical_deduction:
-at.uanaaaannttteerrrr........a "
three_objectie
"natural
cossesaa.uuestion..ee...
"natural,instroctiooss
"natural_instructtan subtakskes
Multirc__ritt_core
'natural_ "natural_instretneer htw.aaiiistttitossssbtataaa.
.nultirc_clessify
(llassify_ccrreet_nnseeee "unit_ pnat_interrretatton.1oe 'unnatural. 'unnatural.in_conter
dates_unnatural. content" unnatural__nn inntuurallinncoteeee www.cottttteeteecoccec www.ttt_eeaannneeeee reverse_ natural content

---

7
42: narrstive.underrsne s 9_distractors"" smeloricel..sinlaarr. 'arithnetic:2
wetss/ouleipplcatteee "arithmatic: (ertameecc.2..iggs..tteeeeeeee sutt_ssutractton". brithnetic:3__dg d.igtte:..dgggt.ddvsstooo prithnetie:::.c
jbgg.___j. discanbbig 'cause_and_ seuse_and_effectt:
two. seosssetenees" "color:hs"" " "ss_eigorrthnssees
wallo
mlmenntryymmath.oo htt..:::llagggg_hhit..oooo./ img.laah...sssssnnooreessss : "fantasy_ reasoninn..
webnig_en
se "eem:webnig_ru shapes_3', pannguusicc.appinn--pir........
luman_orgens_sensee
"intent
Entersect_geometee
"scist_oscii"
"oodf/ied_artthmetic wtw.e_.dgg...o11tto..ooorrr
"novieerrcooeenattion
antural_instructen subssssske http:///cccccccccercreccmm
subtaské744
'natural
inatural.instructions
subbaske?4_cosmosppe
subtask822
prro_puestion_.eerrttionn
ress_aemerate.rclleee.eeee 'nstursnnnnllliii
"nauurd
quesstions
"natural_:nstrructtnn otw.aas.....s..r..sssss.s..oeeea
paragraph_segneen segnentatinn"
simple_ethical. htt.ciiclll.usstionss "social
'social_iqu"
scrange.storiee beooreeebbollan weeeish.to.pernnn..rroo
"aymol_.nterpretation
tracking_shufffe
five_abjects" "unit_interpresaation
amuturalliin_com Jearning unnninngunnatuua.. 2.iit
G.3
Greedy Search
"ssenccs.restturentt comaturallin.coneee inernnn
natural
nasurall_nstrrutttossssssueese wtt.:oossssaaaaeeessllttteet.eee
"rresupossttoosssss... "],
3
[''em:cs_
"presupesitions.es.nii
arithnetic:5_dii
"sim__turing_concopp.
addittiona_vve_22.,
4
16:
[ ('gen: rsssttcsrrstussaatss htg.::/w..................oaaaae com.aaaaaa..aaa..mmmmmmc..ooo...
arit http://www......ooaaaaaaaaaaaaee
"presuposittions.ss..n
-smp__uring_connee
weditiomsl_set_2.. "
wen::eekiinguuaeer 'gen:
1yz', atg.:aaaaaattttss....stt...eeee.
comnon_gen's
"gen:asset', "unit_interpretacionn "rttnnaaas"
otr.apass...............eaaaaaaa "physics's arttmmetie:.5ddigg...dvvss
24: "'en:cc_resttuurants
natural (tt.rll.iirrrcccioosssssssss....
"presuppssttios.es..l
additiomal_set_2",,
"gen:wsiliinguuerr gen:
"gen:asset" uniit_interprecacion
'physics', feal_or__akee_ter
"nonsense__oree wowdss_rramar code__iine_desriiption "louical_sequence
ceceptuallconinattiosssenerr
hhh anh_alignrent Helpuunnstill
cs_restaurants
answering
ard stg.eeeeeeieeeeee..ssss.tteeeeee "rai_nammes"
simp_turrtggcoocepp
additional_set_""
weii:iininnnguueen gen: iinn
common_gen" wwwaaaaetieiise... lv?', Innertuul.cominnatiinssseerree
"gen:asset" intt__ineerrettttion
'physics', 'real_or__ake_textt:ae
conceptusl_combe
"logical_sequene
hhh Helllmeeentttllllueesssssss ttapaaieess..eesttttttaaaearrrrr
"gen:xsun"
sg_algorithhs, thrrithhsseess htap::/eea_aaass..aaaaaaraaaaaa. in cot.rllitnnotttttlaroonno dates'
entailed_polarityy
421
L'gen: cs_restaurants o
"presupesiitonns.e
otg.aaa..e.........ooo.o....aaaa ggen ggn:wwkklinguu_een 'gen:
httptpaieeetee...oooooccccaaaanr conmon_gen" arr
'gen:asset" "unit_interpretatinn
'physics' "rekl_or_feek_ feke_text:eessy
"nonsanse_ "narsnss_vorss_gaaaar
"logical_sequeiee"
knh hon.aaligmenttHHelllunneee
"gen:xsun"
antaileet_ollarttty.hnnid! htg_aaaaaaaaaaaaaaeaaaaeeaeeeeea uumatural_.n.conexx.
es.algorithns:
natural_instnn cnttrralinstuucttinn www.iiinutttonsssueteee.tt htw.:ucccoosssuttsseeee..eoooooo 2
natural_instruction subtaskee
ott.:uourrtttnn..ruuuooooooseeee "questionn.eeeettion "simp_curing_conce""
additional_set_11 'mstrixshapes" "mtt_rac
rovie_dialog_sanm wwalllggsane_o______ driforrrrdfffftt 'natural_ iisuralinsttuction
subtask821
Btaassk@2__nttto_rrr___a_ cramnttical_ogiee 'arithneliic d..iigi.tiivviiion...
divisien"
subtaske61_roops
'nodified_arc
tirea_digtt__ubb cuuttraction_conraall
late_understandding
G.4 Beam Search (q=4, p=1/4)
2
4: C'implicatures' haturallinss sas.rol.iissrrttiissssttttsss Sebttaskkks_nnltirc... wultirc__uut questiiossessisseeooeeeooo.o needed, to.
Linguistic_nnn mappings pest__tense__sse json ' "natural..isttrection subtaske35

---

8: "iipliccturee',
questtons seeeded__o_anne to.snseer
1ingutstic__appp peapiing-pastt.eeeee...
nntuura_-nstruuctinn subtasskee subtaske57
www.ssify_incorrreeeeece lite_.ite.sson bbt. wiiite_esoo_naeee jbo_iitte_son..ntiinnlll.. sttoooonaatttiiiiiii
"sen:ete_nis". 'arithnetic: b.igii.sttiitttttttacttte
sew. ('imliicatures" "natural_instructtue suttaassssttskksssniiirre
nsturat_instruecion:::: subtaske57
"gem:tde_nis"
eryebhologgyspaniss""
qa_wikidata", abstract_narrative. wtt.riiiwuundrrttdddnnnggg htt.:a/www..eetttttiiiaaaaaaaaro
"Ingusstic_maping::
mustioo..iifh.mattmmmtaiaa..... iwneeoiiteerrrreiiiii snoii question_only
plurel_.ison'.
"eiementary.nssh..e.
"elementary_nach.pp"
24: ('inolicatures"
mmnuustic..mpppngsppe...
_ cryobiology_spanizg.
"gen:a7e_nlg"
"vitamine_ fanarinn_.aat..rificcttt
qa_wikidate". _ wtt.eeueenerteeiinnddiitttteee
plural_json', "yytoll aroji_agnosttic" "clomentar__math_pp.
http._uurinn.ccocpt:::ddtttooo question_only" "naturalliinsrrutionns
guesiion.iith.mmthemtiiee
clementary_nathe
empirical__jugnents"" "gre.rrearrrrreiiini con.reaeinnccomrrehenttttto llogical_deduction
art.mstee:..ddgg..ubraaaaa ahhhaaligneent:Hare "bbq_lite__jon.
ht..Litt_ioonnnttimnliii.
ccssosp..icorreet..aeeee.c
32:
l'inolicatures "nstural.insstructiinnn
iinuiistic.mappings.
ieturel_instructionn subtaske57
'gem:aze_nlg",
logital__edductionssew "ewpeettoonseen_oeectttss cryabiologg.5$antta
"bstroct. hngp::siweeuueerrttoinnnaiddeeee qa_wikidata
'Linguistic_napping:
plural.jjon "yttblllitterprettttinn enojin
siementary_natth.ee.
Elenentary_nath...e
sestsacce_sssee_eeeee genrratioenttt derteeertttddttion duration io
empiricall_jugnents:
otg.a oegicalldetiiiiiiiititi oer three_obectt
elamentary.math.ar
mathematical_iiducttinn
'unnstural_in_coneen ce..ontxxt__earning...
'logicel sequence
('imolicatures http::annsnciiiiiiiccmmmmeessss.
pas__eoss_reguaar_isen...
"parsinlu_readin_.com
42:
#nguusstic_napping-
" subtassskee subtask85?
cryobioloog..spsiss" higp::tii....iiit..uubrrrtoi.
'sen:ele_nls".
qa
"bbstrett.errrativeeum wtt.:uneeertttiinngeiiitttteee
"linguisticcneeppin::
plural_json' "sythol aat.aaaioooooooooaagottttt ttl.etttcr.....ootaartr..... clomentary_math_q..
htt.://.wit.ttttemmcccl....ii "sinp_ "sinp_turing_com
secceentary_nath...
question_only", "ntturaal..nstrruiion:s
empirical_judgeen htr.aaasseeddddne_ccarrreeeeeee
"logicol
'ciferie_
nath natheratical__nnucctioo
"unaaturll innatural__n_contott.. leaanntn
identity' atrrr "riiimetssss..dissstaaaapppppppe haa.:aattttttttttttaaaaattttttti subtaskk5t
"logical_sequence"
multirc.write_iiee o -.horrpnee_.caattooooooffffaaaa. aia.
instural_instructionn
"Ianuuge_ieetiiicctttio
'crass_ai',
"perssaa_iddions"
"disembiquation.ee"
ggen assetsset' 'dark_ htt.aaail_ppooettcccaaaaeeeeee
arithmatic:44 4_ti:::::ddiiiidiiiiiidieeeeeeed
internatioanil
"natural iatural_instructions.
subtaskees
_nctaco_ perreettteetttttttrttiettttttt htwp:iottrrnsienssstttiioo
8
G.5 k-means
The numbers reported in Fig. is the average of runs. We ran the k-means algorithms with 5 different random initialization. the following we list the "small-bench" candidates from run. The same applies to "k-means+ + Task Value" results.
2
4: aaanchronnssss 'natural, instructions inatural Linguistic_nappieen seauuiistcc.apppnngsseeeeeeee httprii__nnappnnnsdeeeeeeessee
stterrrctionnssstssseee.e subtuske12 www.asseeeiiiiiieeaaaeeeeaaoaeaa
ott.r/asner.ooooo..u.aaaaannnttn subtasssa subtask833
natural_instructions
wwinogrande innpgrratt_aaeee_.geerrttttooo
3
8: "nanchronnssss
'arithnetic
_ddigit,eed sadditttdddiion
"Lingusttic.nnnpinnnee
hor.s/eeeeeesi...ettttooooooo... 'natural_
jo__inograade_ffll..pee
bbq.
subtaskids subtaskkll__sss__fidddd htt.://ssea.aa..oooooooooaaaeeeo
16:
('smschronisms" itmaa
Iinguistic__spppie sasa "tr.aaiin....iittttiiiiii. "ai.ihhettiec.dliiii.....e
"natural_instructtonn aubbaakkaaau
netural_instruction
'nndified_arithnes
wwo.diigt.mullpplcccttto....
"ibo_iite_jaon htt.:mmeb_.lttejoooneiiiiiaa..
lasss_statn_trackee real__miunn
inguusstic_mappnnee htg.::.dd__ssttttsrrrrlllla
subtask832 wubtsskes__nooorrnne choorande__qustiinn senranneennatttnoeersrsn i natursl_instruetti
subtask841
turk""rk". unnatural... wttttuallinncoooo. www.iintet..eeee....ec
dates_unnatural. content "natural souurll.ie instructions: i subtask848 multire question. gerrrannnnnnnnnon '3",

---

24: ('snschronisms "reei_ frei__or_aake.... text_eexteesyy "aritmeeii::.digit..ee. natural natural_instren : bbo.liee.......a...neeee........
ween:webnlg.ee en
singutstic__aapingg
"gen:xsun
shaiingg_ssuflleedoooeeettt seven iaturral_instruttiinsssstaa tracking.shuffic
_mccaso_temporal__ seraagsttmrrrsrrosoonnn.ttteenn wtt.aaaaio.iattttoooooooaaaaaaa. http://ww........aaaaaaaaaaaaaaa
_cosmosqa_ answer senerraiin....eeeertiii....o.
cnverse_t__naurrl_ccontttt "authoor "attorsaip.vmiiictttoooooo
"naturral_ sus subtaskes
int.rpeeeiiiiiii.....aaaaaaaeeei on
bbd
_dissrractors' 'calor:hsl', "fact_checker:fever
'tense' "intersect_ srrrsecct.eomtrry.nnee "''"
("snschronisms" 32:
"arithnetic:
"ge: webnlg.eeg.en
"iinguistic.
past__tens___er
'gen:xsun'
tracking_
baacking_shufleed.o shuffled_objettee seveooosseeeen.ojeect
_.ecaco_.pen_
wiigggaanee_uuestionn acs.cc..ssstooo..ggaaatttaeeeee. :
auuthership..erfficciiooo swapped saal__.te_.wkkhooogooo.....ddd
two
www.suul.riiettttion........ disanbig
aestratt.oraaiiv..aarrtttinnnns
wfact_chcckerrrreeecc. 'tense
intersacc__ccomntttyyssseee uunatuural inraaliiinn_ooieeee
"hyperbaton"
synbol_itterprrttationpppppl warthhett "artthhettccs_.eiitt.aee
"chess_stato_tracking real_nedtun
pphysics""".
42:
d'anachronisms"
"arithmetic::_ddiitt.e
"gen:webrlg_en""
na htt.inssruuttoossssutskkk....... htr.:a/ccsc..iinooooapiiii..nnoa past__
"[inuusttic.napsnngs
insturaal..sttrrriiinns 'gen:xaun".
mectaco.tenprrllreeassee
"attrrrlannnntttrlnnnnntttt subtask117
wwnogrande_full_prreee
natural_instructinn subtaskee
"natursl_
outherssip.yerrffiaiiinsssspppp "bq__lite_jsem
radified_arithmetil
two
satrrrruannnntttdnnnddaaraaaerrr distracctoss 'color:hex', "fat_""faccceehee :avckrrr:aaar ' "tense"
regularssggul "unnstural_in_centtt___ dates_umsstral..ontett" "na.ucnt insturrl
hyperbaton's ipp.osiittttinneeeetttiiooon aatttteeeee:::::i1i1..oo.ddd
simp_turing_concme sedocopptadiiiiionn_.......
'physics', 'cifarte
classification:tasee.. "intersect_geone seterrect_gonetrry::hhh
"elementary_nath _atttrrrr__theeee
"goal_.tep_ wwwhhwwwwwkiwwwweeee: step.oxeeee..feeeeneee unit_:dentificacion mmatuui "naturál__nsttrune instrucctiinns subtaskess cubtoskesi_muitiree.
'unft_conversion:
strange srrange_storiee
sttrreesttrrriesulle choicee' "discourse, parker_prediction isoo:
"natural_ instructions :ubbtat
subtaska24 caattakka44cconaaee
G.6
k-means + Task Value
2
4: winogrande_full.otj.e 'gen:xsun'
natural_instruation subtask82g
kinogrande-
3
81
('hygerbaton" "natural__nstruuciins wuustion.foraationn..o.. 'naturál
'arithnetic:5
_digit_addition "1inguistic_napplee "9_winagrande__fll_
insturalinstructtonsssooo
natural incturr=iiittructtonsssseee
'gem:xsun "gr
hhhyeebbaton' 'real_or_fake_tee rrailoor_feeecoxtttgtt.... artpptat......tttttttesss... past
"Linguistic_mapoongs
inference'. ioaiiirreeeeiieeeeeeeeee.e
periadic_.lamnts::nn
chess_state_trackee
quoree..nnsee...ener.. "ge::ttuiinnne::t:rrr
'gen:turk
ltt.mmingg:ddess.nnnttrrr conteent
http://ttetttonnnsniaaaaaaaaaaaa "nnatrral_instructionn
24: L'snarks', oor
additionsl hrt.:pai.........nn.o...oooooooo
'simp_ Saap_ttuinngcom
"bbg_lite_jsan:
bog.lttt..o.ggpystaaaaapaaaaeaed 'gen:
"gen:
" "analogical_.imnlaritty
"natural_instructione
subtask
"ntuural subtask822
_
.xinogrante_falllohhjettt"" aht.aaaetttttttcsscstttttttteeer real_shott "natural_instructionn
sts.rruuttionssssttseess sabtask845
Lavarso_to_natural.ccom
'hyperbaton". "taal_stto..ikthoorrr ooal inffarnnnee.eee
redifiee__ritthmetieeee
www..diii..mltiiicciii...oom nonsenss_words.srmmnr
"bb__lit_jsoon.
disanbiss" " discourse.narkerpprreiittto. 'implicatures' escii_.ord_.ecoonnitior shapes_4']*,
discourse.narker.
'Intersect_geonettry:
32: ('anachronisms'. 'real_or_fase fea__or_akeetextssee x1't2x1',, "aritheetie: disii_
alay_ddalog_.5 or different "natural. "natural_Instructuu Instructions subtaske58. www.irrouueeteeeteeeoooo question_answering
linguistic sesssic_mmne nappings past_ jense_ "bbq_ lite json teo.eoon psy_ssss_nphpsicall
physical appparaneep....

---

sen:
"nstorel_isttructtonn: subtask928
westhurship.errfiieate swapped",
diccorss..nrer.pprcictttoo iimplcaturess
aascii "accii__ard.eeconntton oomastrrl_iinnooosss...errnn wttpaaaeeeeeeettmtoroooooooooo
cntersect__gonettee
shapes_d',
"natural instruction:
'typerbaton 'symbol_ interpratation plaainn
"sinp_turing__oncept.
additiomel_set_ fuumen.orgams_smmees:
42:
s'anchrrnnssss" ieo...or.ake..extttppoee
"srithmetici1_diii "itimerci::..diiii.ddvvssoo.""
alay.dtaleg_.mmmoor.ffffeee " subtaskkek
pttp:iisssppttttts___saasse "bbq_lite_jjso
"gen:
"tracking.shuufflee. "natural_ inturallinttucttions
obvct_
subtaske17_nctars
-winogranddfuul_ojeettt
reverse_ ionrrss___.nsttrra_coottmmm
"toei_step_
soal_inference
'bbq_lite_json:
'calor:hex"
implicatures"
instural subtaskk22
_.roo_quusttton.gnnrattioo "hyparbattoo
ssppoollnnsrprrettttooo..pp 'arithnetic:5
"physics"
ccfarri_.caassificaiiin basess.
intersect searectt_eeeeetrrsssssee__-
multiirc_classlf.ccoreee...
strange_stories:
discourse
aiscourse_nareer.preddeee markker..reedicionn
natural_instructee


Reviews:
[{"review_id": "KyZTVwOc5a", "reviewer": "Reviewer_Ku3n", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! It appears that there are some misunderstandings that we wish to clarify. We hope the following response helps address your concerns.\n\n### Response to reasons to reject\n\n> This article relies solely on superficial information to assess the predictability of LLM capabilities, which weakens its persuasiveness.\n\n* We are not entirely sure what this statement is referring to. The experiment records used in this study are real evaluation of LLMs on diverse language tasks. We would appreciate it if reviewer Ku3n can provide more information regarding the “superficialness”.\n* If “superficial” refers to sourcing the experiment records from a public repository, we consider this as a strength rather than a limitation, as these records are open to the public and it is easy to reproduce our study. \n\n> The experiments in this article are quite limited, as all tests were conducted on just one dataset. This might introduce potential biases and uncertainties regarding its generalization capabilities.\n\n* Our work is a meta-analysis on 313 (sub-)tasks and 51 language models presented in BIG-bench. BIG-bench is one of the most comprehensive evaluation benchmarks available for large language models. We believe this represents our most rigorous effort to address \"generalization\" across various models and tasks.\n* In addition to the default experiment (Sec. 3), we have experiments on controlling training set composition (Sec. 3.4); constructing more challenging train-test splits (Sec. 4); designing “small-bench” using the performance prediction model (Sec. 5). We hope this addresses your concern that the experiments are limited.\n\n### Reproducibility\n\n> Would be hard pressed to reproduce the results. The contribution depends on data that are simply not available outside the author's institution or consortium; not enough details are provided.\n\n* The experiment records we use in this study come from the BIG-bench repository (https://github.com/google/BIG-bench) which is open to the public.\n* We have included our code in the supplementary material. Additionally, we provide dataset creation and filtering details in Appendix A and model training details such as hyperparameters, hardware information in Appendix D.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experiments in this article demonstrate that some capabilities of LLMs are predictable. This method might serve as a reference metric for future LLM development. \n2. Furthermore, the article emphasizes the significant impact of data on LLMs and suggests a vast space for further research.", "reasons_to_reject": "1. This article relies solely on superficial information to assess the predictability of LLM capabilities, which weakens its persuasiveness.\n2. The experiments in this article are quite limited, as all tests were conducted on just one dataset. This might introduce potential biases and uncertainties regarding its generalization capabilities.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The theme of this paper is the predictability of the capabilities of Large Language Models (LLM). Specifically, it focuses on using five types of data as a dataset: the model family, the number of parameters, the type of small tasks, the number of context examples, and the performance results on sub-tasks. An additional model was employed to learn from this dataset. Based on this approach, experiments were conducted, high-quality data subsets were sought and organized, and findings and analyses from the experiments were summarized.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q3AvcVvi5X", "reviewer": "Reviewer_WS63", "summary": "", "questions": "1. In L473-479, I do not see how the first-layer representation of a MLP could capture diversity and representativeness.\n\n2. There are some issues during the authors' data processing and model building:\n   1. In L970-972, the authors remove all performance metrics equal to 0. Despite sparsity in available information to train the model, a useful model should also be able to predict 0 performance metrics. Could authors explain this choice?\n   2. The authors do not provide a rationale for choosing specific features when building their most performant model - an MLP-based regressor. For example, why do authors use 6 numerical features for the number of parameters rather than 1?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your thorough review and insightful feedback! It is encouraging to hear that you believe our research questions are “real and very relevant to the needs of LLM evaluation” and our work is backed by “comprehensive experimental results”. We will incorporate your feedback and updated discussions in our next version.\n\n### Response to reasons to reject\n\n> Choosing explanatory variables in your design matrix does not have a sufficient explanation. For example, two models with the same architecture could perform widely differently if training one converges while the other fails.\n\n* Thanks for raising this point! We understand that LLMs capabilities are dependent on many factors, including pre-training stability and convergence, pre-train corpus composition, etc. In an ideal case, we wish to incorporate all these meta-data as the input feature to our performance prediction model. However, we often don't have access to this information.\n* In this work, we assume that the input features (model_family, n_param) can implicitly learn to capture such information. In the future, we believe our method can be readily expanded to include additional pre-training meta-data when they become available.\n\n> The authors do not discuss how the emergent abilities of LLMs could influence the regression models' performance. For example, how do the LLMs' emergent abilities contribute to the RMSE of your regression models?\n\n* __In general__, performance prediction accuracy on emergent tasks is lower than that on non-emergent tasks. We divide the test set into records associated with emergent and non-emergent tasks (characterized in Appendix E in https://arxiv.org/abs/2206.07682). We take the MLP model trained in the L1 setting, and report RMSE and $R^2$ within each group in the following table. \n\n|  | RMSE ($\\downarrow$) | $R^2$ ($\\uparrow$) |\n| ----------- | ----------- | ----------- |\n| Emergent | 0.0541 | 93.86% |\n | Non-emergent | 0.0496 | 95.16% |\n | All | 0.0499 | 95.07% |\n\n* __In certain cases__, emergence can be predicted by our model accurately. In Figure 7 Subfigure 2, we visualize the predicted and actual performance of BIG-G T=0 models on the three_digit_addition, an emergent task. We hypothesize that emergence on a task may be accurately predicted when a similar emergent task (e.g., similar arithmetic tasks) appears in the training set. See Line 322-334 and Line 1065-1075 for more discussion.\n\n> The authors need to discuss what conditions should be met to apply their proposed approach. For example, the authors do not analyze T0/T5 models (L967-969) because of the unavailability of sufficient data to train the model. However, the authors do not discuss the minimal data requirement to develop a valid analysis with the proposed approach.\n\n* T0/T5 models have 14 experiment records in BIG-bench. In contrast, BIG-G/PaLM/GPT-3 models have 5000+ records. Due to the magnitude of difference, we decided to discard T0/T5 models in our experiments.\n* Regarding the minimal data requirement, we think this is highly related with the “small-bench” problem we are trying to solve in Sec. 5.\n\n> The authors claim that there are limitations of BIG-Bench Hard (BBH) and BIG-Bench Lite (BBL) based on the proposed metrics. However, to establish the discovered BIG-Bench subset's advantage over BBH and BBL, the authors should use an independent source of information. For example, a previously ignored LLM due to low ranking on BBH and BBL is found to rank high on the discovered benchmark, and this model is able to do something that high-ranking models on BBH and BBL could not do well.\n\n* Thank you for your suggestions! We agree it would be more convincing if we can validate the small-bench candidates on more model families (i.e., using independent sources of information). However this requires running new LLMs on the hundreds of tasks in BIG-bench, which is prohibitive for us at this time.\n* During the rebuttal period, we examined the existing model families in our study and indeed found several cases where BBH fails to represent the full BIG-bench. For example, according to BBH, BIG-G T=1 2B is more capable than GPT-3 Large on the full BIG-bench; however, in reality, GPT-3 Large is better than BIG-G T=1 2B, which is captured more accurately when a small-bench candidate is used.\n\n|  | BIG-G T=1 2B Wins | Tie | GPT-3 Large Wins |\n| ----------- | ----------- | ----------- | ----------- |\n| Performance Recovered from BBH (24 subtasks) | 20.1% | 70.1% | 9.8% |\n| Performance Recovered from small-bench (24 subtasks) | 19.1% | 56.3% | 24.6% |\n| Ground-truth full BIG-bench Performance | 14.3% | 55.3% | 30.4% |\n\n### Response to questions\n\n> In L473-479, I do not see how the first-layer representation of a MLP could capture diversity and representativeness.\n\n* The first-layer representation of MLP can be considered as representation for the task. We manually verified this by visualizing the representations in 2D space (e.g., similar tasks are close to each other in the first-layer representation space).\n* Diversity is captured by clustering these representations. We select one task from each cluster, so that similar tasks will be deduplicated, and this promotes task diversity.\n\n> In L970-972, the authors remove all performance metrics equal to 0. Despite sparsity in available information to train the model, a useful model should also be able to predict 0 performance metrics. Could authors explain this choice?\n\n* To clarify, we remove subtasks that are _too hard_ so that _no_ model achieves performance better than 0. We believe these are outlier tasks and provide little information in training the performance prediction model.\n\n> The authors do not provide a rationale for choosing specific features when building their most performant model - an MLP-based regressor. For example, why do authors use 6 numerical features for the number of parameters rather than 1?\n\n* We use 6 numerical features mainly because the BIG-bench repository directly provides 3 features (# total parameters, # non-embedding parameters, # FLOP-matched non-embedding parameters). Further, since scaling laws typically use the log of the pre-train scale as a feature, we take the log of these 3 values.\n* It is possible that alternative featurizations may lead to better performance. However due to the complexity of our study we did not further investigate this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The problem the authors are trying to solve is real and very relevant to the needs of LLM evaluation: (1) how to predict an LLM's performance without actually running the experiment, and (2) how to quantitatively select the best subset of comprehensive benchmark for quick evaluation. \n- The proposed approach is effective even for challenging scenarios. \n- The presentation of the main idea is easy to follow, with comprehensive experimental results to back up.", "reasons_to_reject": "- Choosing explanatory variables in your design matrix does not have a sufficient explanation. For example, two models with the same architecture could perform widely differently if training one converges while the other fails. \n- The authors do not discuss how the emergent abilities of LLMs could influence the regression models' performance. For example, how do the LLMs' emergent abilities contribute to the RMSE of your regression models?\n- The authors need to discuss what conditions should be met to apply their proposed approach. For example, the authors do not analyze T0/T5 models (L967-969) because of the unavailability of sufficient data to train the model. However, the authors do not discuss the minimal data requirement to develop a valid analysis with the proposed approach.\n- The authors claim that there are limitations of BIG-Bench Hard (BBH) and BIG-Bench Lite (BBL) based on the proposed metrics. However, to establish the discovered BIG-Bench subset's advantage over BBH and BBL, the authors should use an independent source of information. For example, a previously ignored LLM due to low ranking on BBH and HHL is found to rank high on the discovered benchmark, and this model is able to do something that high-ranking models on BBH and BBL could not do well. ", "questions_to_authors": "1. In L473-479, I do not see how the first-layer representation of a MLP could capture diversity and representativeness.\n\n2. There are some issues during the authors' data processing and model building:\n   1. In L970-972, the authors remove all performance metrics equal to 0. Despite sparsity in available information to train the model, a useful model should also be able to predict 0 performance metrics. Could authors explain this choice?\n   2. The authors do not provide a rationale for choosing specific features when building their most performant model - an MLP-based regressor. For example, why do authors use 6 numerical features for the number of parameters rather than 1?", "paper_topic_and_main_contributions": "This paper tries to interpolate and extrapolate an LLM's normalized performance metric based on a quadruple of model type, number of parameters, task, and number of shots by training a regression model with existing BIG-Bench statistics. The authors show they could obtain decent RMSE even for a challenging setting, where a tuple of model and task combination is not seen in the training set. \n\nBased on the regression model, the authors further show a method to select a representative subset of BIG-Bench. The subset selected by the proposed approach is more representative than the widely-used BIG-Bench Hard and BIG-Bench Lite.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wXsywxorgC", "reviewer": "Reviewer_6piE", "summary": "", "questions": "Is there a reason why certain settings are able to leverage multi-group training more than others? It’s unclear to me why the 0-shot group may not benefit from multi-group training.\n\nIn table 1, are the total # of tasks examined the sum of the number of subtasks and the number of tasks? Or are there 313 subtasks that are evaluated which fall into 134 task categories? \n\nHow do the small-bench experiments compare to that of BIG-bench? Both in number of tasks and performance? While the subgroups for BIG-bench do seem like more reasonable baselines, is there a specific reason this direct comparison is never made?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and feedback! We are glad that you think our work offers “significant practical implications”, presents “notable results”, and “could reduce computational barriers [for LLM evaluation]”. In response to your comments, we offer further information and clarification below.\n\n### Response to reasons to reject\n\n> Only RMSE and R2 scores are reported. While this limitation is addressed in the paper, they do not capture the complete picture of a model’s accuracy.\n\n* In the main paper, we focus on RMSE and R2 scores as they are widely accepted evaluation metrics for regression problems. \n* Additionally we introduced task-average Pearson Correlation and Kendall Rank Correlation in Appendix C.1, in an attempt to get a more comprehensive picture of our prediction model. We reported the full results in Table 4.\n* We will revise our paper to highlight these additional metrics and our considerations. \n\n> There are a relatively small number of model families in BIG-bench. […] It’s unclear whether the model families present in BIG-bench are significantly different from each other.\n\n* We would like to note that the 6 model families in this study are representative, and offer considerable diversity. We provide a brief summary in the following:\n\n| Model Family | Company | Release Date | Pretrain Corpus | #Parameters | Architecture Choices | Infrastructure\n| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| BIG-G (T=0, T=1) | Google  | Jun 2022 | 2.8T tokens | 2M to 137B | Gated-GELU, relative attention | TPUv3, GSPMD |\n| BIG-G Sparse | Google | Jun 2022 | 2.8T tokens | 51M to 46B  | BIG-G + Mixture-of-Experts | TPUv3, GSPMD |\n| PaLM     | Google  | Apr 2022 | 780B tokens | 8B,62B,540B | SwiGLU activation, parallel layers, etc. | TPUv4, Pathway |\n| GPT-3    | OpenAI | May 2020 | 300B tokens | 125M to 175B | Alternating dense and sparse attention | GPUs |\n| Gopher  | DeepMind | Nov 2021 | 300B tokens | 44M to 280B | RMSNorm, relative position encoding | TPUv3, model parallelism, etc. | \n\n* As we chose BIG-bench as our testbed for the study, we have made every effort to incorporate all the model families evaluated in it. Recognizing this limitation, we encourage further community efforts on “broadening observations on LLM capability landscape” (see our discussion on page 8). We hope our methodology presented in this work will remain useful when more observations become available.\n\n> There is little discussion of the variety of models and tasks in BIG-bench.\n\n* Models: Please see the discussion above.\n* Tasks: Quoting from the original BIG-bench paper (https://arxiv.org/abs/2206.04615), “task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond.” The paper also provides a summary on the task diversity in Fig. 3 and Table App.3. We will include this information and refer our readers to the original paper.\n\n### Response to questions\n\n> Is there a reason why certain settings are able to leverage multi-group training more than others? It’s unclear to me why the 0-shot group may not benefit from multi-group training.\n\n* For most tasks, there is a huge performance boost when going from zero-shot to one-shot, and the performance improves more stably when more shots become available (see Figure 1 in https://arxiv.org/abs/2102.09690). It is easier to predict 3-shot performance when given {0,1,2}-shot performance, than to predict 0-shot performance when given {1,2,3}-shot performance. This partly explains why the 0-shot group does not benefit much from multi-group training.\n\n> In table 1, are the total # of tasks examined the sum of the number of subtasks and the number of tasks? Or are there 313 subtasks that are evaluated which fall into 134 task categories?\n\n* Sorry for the confusion. There are 313 subtasks that fall into 134 tasks. For example, “arithmetic” is a task, while “2-digit addition” is a subtask under the task of “arithmetic”.\n\n> How do the small-bench experiments compare to that of BIG-bench? Both in number of tasks and performance? While the subgroups for BIG-bench do seem like more reasonable baselines, is there a specific reason this direct comparison is never made?\n\n* To clarify, the “small-bench” problem is a subset search problem. When a new LLM is evaluated on a subset of BIG-bench (i.e., “small-bench”), we aim at maximally recovering the performance on the remaining tasks. Therefore, we compare with different BIG-bench subsets as baselines.\n* To put BIG-bench into comparison directly, BIG-bench will appear at the coordinates (313, 1.0) in Figure 6 in the paper, as it has 313 subtasks in total.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a novel method for predicting the performance of Large Language Models (LLMs), which has significant practical implications. This method could aid developers and researchers in conserving resources when training models, given the vast array of possible combinations of models, hyperparameters, evaluation tasks, and in-context examples.\n\n\nThe results of the paper are notable. The high R2  scores indicate that the BIG-bench dataset contains highly predictable patterns, thus validating the authors' approach to performance prediction.\n\n\nIn addition, the authors present the idea of “small-bench,” which could reduce computational barriers for models with limited training computation at their disposal. They provide an understanding of how to construct “small-bench” effectively, suggesting potential applications of these methods to optimize other benchmarks in different contexts", "reasons_to_reject": "Only RMSE and R2 scores are reported. While this limitation is addressed in the paper, they do not capture the complete picture of a model’s accuracy. \n\nThere are a relatively small number of model families in BIG-bench. This could lead to results, even in the challenging Train-Test split, being due to intrinsic correlation between these model families that might not translate to other models.\n\nThere is little discussion of the variety of models and tasks in BIG-bench. It’s unclear whether the model families present in BIG-bench are significantly different from each other (at least as much as another model with a different pre-training pipeline might be).\n", "questions_to_authors": "Is there a reason why certain settings are able to leverage multi-group training more than others? It’s unclear to me why the 0-shot group may not benefit from multi-group training.\n\nIn table 1, are the total # of tasks examined the sum of the number of subtasks and the number of tasks? Or are there 313 subtasks that are evaluated which fall into 134 task categories? \n\nHow do the small-bench experiments compare to that of BIG-bench? Both in number of tasks and performance? While the subgroups for BIG-bench do seem like more reasonable baselines, is there a specific reason this direct comparison is never made?\n", "paper_topic_and_main_contributions": "This paper investigates the extent to which the performance of Large Language Models (LLMs) can be predicted prior to any training. If performance was found to be cheaply predictable, then LLM development could be made more accurate, faster, and cheaper computationally. The authors introduce a novel method for predicting LLM performance based on the experiment records from BIG-bench, which include data from various model families, tasks, parameters, and in-context examples.\n\n\nThe main contribution of this paper lies in the authors’ demonstration of high predictability of LLM performance using an MLP-based predictor, with an R2 score of over 95%, indicating that strong learnable patterns exist within the data. They define the prediction problem using 4 inputs: Model family, number of model parameters (as a proxy for model scale), evaluated tasks, and number of in-context examples. The model’s normalized performance is then predicted. They evaluate many different prediction models, ultimately finding strong performance in Tree-based methods and MLPs. They also provide insight into what test groups (such as zero-shot vs one-shot models) are more difficult to predict, and why. They find certain difficult groups cannot easily make use of multi-group training (such as the zero-shot group) and others are just intrinsically hard to predict (such as models that are larger). They additionally experiment with the intrinsic predictability of certain tasks, and provide hypotheses as to why certain tasks may be more or less predictable than others.\n\n\nAdditionally, the paper introduces the concept of “small-bench,” a condensed subset of BIG-bench tasks, which aims to maintain equivalent informativeness while significantly reducing the computational resources and time required for evaluating new model families. The authors underline the importance of task diversity and value in constructing an informative \"small-bench\" that can maintain similar predictive power to the larger BIG-bench. They showed that certain methods of creating \"small-bench\" subsets outperform baseline subsets of BIG-bench that are not explicitly designed for informative prediction.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main conclusions of the reviews and the post-rebuttal discussions:
- 2/ 3 reviewers consider the paper sound (scores 2, 5, 3)
- 3/ 3 reviewers find the paper exciting (scores 4, 3, 3)

From reading the rebuttal and seeing the scores above, I find that the reviewers consider strong points for soundness the following:
- tackle an important problem: (1) how to predict an LLM's performance without actually running the experiment, and (2) how to quantitatively select the best subset of comprehensive benchmark for quick evaluation.
- the authors present the idea of “small-bench,” which could reduce computational barriers for models with limited training computation 
- solution is effective even for challenging scenarios.
- comprehensive experimental results
- easy to read
Decision: Accept-Findings