Zero-Shot Performance Prediction for Probabilistic Scaling Laws
Authors: Viktoria Schram
Markus Hiller
Daniel Beck
Trevor Cohn
Venue: NeurIPS 2025 poster ()
Keywords: Gaussian Processes
Active Learning
Scaling Laws
Primary area: general_machine_learning
TL;DR: Leveraging correlations among hierarchies present in natural language processing tasks is used for zero-shot peformance prediction of learning curves for scaling law research.

Abstract: The prediction of learning curves for Natural Language Processing (NLP) models enables informed decision-making to meet specific performance objectives, while reducing computational overhead and lowering the costs associated with dataset acquisition and curation. In this work, we formulate the prediction task as a multitask learning problem, where each task’s data is modelled as being organized within a two-layer hierarchy. To model the shared information and dependencies across tasks and hierarchical levels, we employ latent variable multi-output Gaussian Processes, enabling to account for task correlations and supporting zero-shot prediction of learning curves (LCs). We demonstrate that this approach facilitates the development of probabilistic scaling laws at lower costs. Applying an active learning strategy, LCs can be queried to reduce predictive uncertainty and provide predictions close to ground truth scaling laws. We validate our framework on three small-scale NLP datasets with up to $30$ LCs. These are obtained from nanoGPT models, from bilingual translation using mBART and Transformer models, and from multilingual translation using M2M100 models of varying sizes.

Zero-Shot Performance Prediction for Probabilistic Scaling Laws
Viktoria Schram' Markus Hiller! Daniel Beck² Trevor Cohn"
School of Computing and Information Systems, The University of Melbourne School of Computing Technologies, Royal Melbourne Institute of Technology {v. wchhrram, m. m.hiller, trevor.
cohn @unimebb edu.au au daniel. beck@rmit aetermit.euu..aa. au
Abstract
The prediction of learning curves for Natural Language Processing (NLP) models enables informed decision-making to meet specific performance objectives, while reducing computational overhead and lowering the costs associated with dataset acquisition and curation. this work, we formulate the prediction task as mul- titask learning problem, where each task's data is modelled being organized within a two-layer hierarchy. To model the shared information and dependencies across tasks and hierarchical levels, employ latent variable multi-output Gaus- sian Processes, enabling to account for task correlations and supporting zero-shot prediction of learning curves (LCs). We demonstrate that this approach facilitates the development of probabilistic scaling laws at lower costs. Applying an active learning strategy, LCs can be queried reduce predictive uncertainty and provide predictions close to ground truth scaling laws. We validate our framework on three small-scale NLP datasets with up to 30 LCs. These are obtained from nanoGPT models, from bilingual translation using mBART and Transformer models, and from multilingual translation using M2M100 models of varying sizes.
Introduction
Large language models are increasingly being employed across various research domains and industrial applications, requiring substantial computational resources not only for the training, fine- tuning. and testing of these models but also for their deployment, prompting, and generation processes. As model complexity and the number of processed tokens increase, energy demands and processing times grow extensively, resulting high costs and significant environmental impact 22 5]. In this context, performance prediction has emerged as promising research direction, aiming to alleviate these costs by providing informed guidance for model selection, data acquisition strategies, and computational requirements [6
One important task in this field of research is learning and training curve prediction. The former evaluates the generalization ability of machine learning models function of resource of interest, such as dataset size model complexity, while the latter models the evolution of the loss over training iterations, epochs, steps [9]
[0 II] Early approaches to learning and training curve prediction relied on fitting parametric functional forms observed performance metrics and extrapolating future outcomes [12 3 14 15 [6]. Since then, particularly within the context of Bayesian optimization, various surrogate models, such Gaussian processes [7,, Bayesian neural networks [18] 19, and ensemble methods (20), have been employed for curve extrapolation tasks, providing uncertainty estimates alongside point predictions.
Also at Google Australia
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Training curves also play key role in formulating and validating scaling laws, which are empirical relationships that describe how model performance varies as function of factors such as model size, dataset size, and compute budget [3 essource-intensive, as requires training numerous models and conducting extensive experiments across different configurations to obtain representative set of training curves.
21 22 However, deriving these scaling laws is highly
The core hypothesis that we explore in this paper is that our Natural Language Processing (NLP) learning and training curve datasets exhibit hierarchical, specifically bi-level, structures. A key advantage of hierarchical models is their ability to incorporate prior knowledge through the explicit definition of the hierarchy's 's structure. This not only enhances interpretability but also facilitates the transfer of knowledge across different tasks and domains [23]2]] 25 26] 27]. Hierarchical modelling approaches have found applications in areas such as hierarchical text classification, hierarchical
clustering of words within a vocabulary, and hierarchical Bayesian methods that incorporate multiple layers of data or prior information, for example, in word segmentation tasks [28 (9 30 31, among
others. However, hierarchical modelling in the context of learning and training curve prediction tasks has surprisingly not yet been explored.
We aim to bridge this gap by modelling learning curve (LC) prediction tasks in NLP as multitask problems, where each task comprises data organized in two- layer hierarchical structure, as illustrated in Figure [ An additional hypothesis is that these hierarchies are exchangeable. We demon- strate that by leveraging cor- relations among tasks, zero- shot prediction of training curves for scaling law re- search be achieved significantly reduced com-
Number of Embedding Parameters
Source
Source Language
Languages
Model Architecture Size
1024
English
English 175M
8
10
48
Indonesian
Malay Javanese
Indonesion
Malay Javanese
Numbor of Layers
Target
Target
Languages
Langruggss
Figure A task from each dataset illustrating the modelled two-layer hierarchy, (left) Learning curves assuming common prior information when the same number of embedding parameters used. (middle) Learning curves for bilingual translation, assuming common prior information when translating from the same source language. (right) Learning curves for multilingual translation, assuming common prior information when the source language and model size are the same.
putational costs. Our framework employs latent variable multi-output Gaussian process models 24] which account for the hierarchical structure within each task as well as correlations across tasks, Our approach outperforms competitive baselines and leads to our final hypothesis that this model can be effectively used for scaling law prediction due to its strong zero-shot performance. This approach not only provides a principled mechanism for uncertainty quantification but also enables the derivation of probabilistic scaling laws via Monte Carlo simulation. Furthermore, by incorporating an active learning strategy, the uncertainty of the predictions can be substantially reduced. To demonstrate the generality of our data modelling approach, we extend this framework beyond scaling law prediction to performance prediction tasks in bilingual and multilingual translation across models of varying capacities. We show that, compared to baseline methods, leveraging hierarchical structure and task correlations yields significant improvements in zero-shot performance prediction, particularly for small-scale datasets. In the following, we will refer to training and learning curves as learning curves.
2 Related Work
Learning Curves.
Early studies on learning curve (L.C) extrapolation analyzed functional forms, including power laws and exponential functions, among others 12
34 35 36). LC trends 38 or used to define early stopping criteria [39]. In the context of Bayesian optimization, surrogate models such as Gaussian processes with exponentially decaying kernels [17]. Bayesian neural networks [18), or product kernels 19 have been employed for LC extrapolation. More recently. Transformer based prior-data fitted networks have demonstrated competitive performance in LC extrapolation tasks [40]. Our work differs in that we explore and exploit existing hierarchical structures in small-scale NLP LC datasets.
16
32
33
have been extrapolated for larger training dataset sizes 9 12 32
37
Zero-Shot Prediction. Zero-shot learning refers to model's ability to make confident predictions for test data that were not encountered during training 41.. In NL.,, zero-shot prediction has been applied to transferring models to new tasks 42 43
performing sentiment analysis in zero-shot

---

multilingual settings [5], and addressing cross poos--omain adaptation tasks [46,, among others. In contrast, Bayesian optimization methods in hyperparameter optimization learn LC trends across various hyperparameter configurations and can predict outcomes for unseen configurations [8] 20 Our work most related to Klein al. [88, where Bayesian neural network with a basis-function layer performs zero-shot prediction for unseen configurations. Their smallest dataset contained 256 configurations. In contrast, our study operates on much smaller datasets, the largest containing only 30 learning curves, and explicitly models the underlying hierarchical structure. Orthogonal to these are methods transferring optimal hy hyperaarameters from smaller to larger models (20 47 In contrast, we fix hyperparameters and zero-shot predict learning curves for larger models.
Gaussian Processes and Hierarchical Datasets. Multi-task learning with Gaussian Processes (GPs) was introduced by Bonilla et al. 48) assuming flat structures within the dataset. Hierarchical structures, however, are commonly observed in biological datasets, for example, in modelling gene expression time series [23] organism taxonomies [49], protein families [5).. or enzyme classifications
51
Such datasets often contain multiple layers of hierarchies and a large number of examples
52
Park and Choi [53] employed hierarchical GPs to model medical data, demonstrating bi-level
hierarchy corresponding to clusters genes and their respective prototypes. Lawrence and Moore
[54] introduced hierarchical GP latent variable models for dimensionality reduction, and Hensman et al. 233 proposed a hierarchical kernel designed to capture inherent hierarchies in gene expression datasets. In contrast, we explore hierarchical structures in small datasets with at most 30 examples and a bi-level hierarchy. Building on Ma et al. [24), who introduced latent variable multi-output GPs, capturing hierarchies and task correlations, and probabilistic scaling law estimation via Monte Carlo simulation.
this model for zero-shot learning curve prediction
Scaling laws are functional forms that extrapolate the behavior of cost-intensive large language models based on a set of learning curves [4. These laws provide improved interpretability of neural networks [55] 56 57] enable effective planning of training sample sizes, and help reduce the associated carbon footprint 36
Scaling Laws.
58 Most successful methods in this area are based on empirical,
computationally intensive studies 3 21
59
60
61 62] Hägele et al. [59] proposed alternative
model training techniques to reduce the computational requirements for obtaining scaling laws. Our work is orthogonal to this approach,
perform scaling law prediction via Monte Carlo simulation.
More related work is provided in Appendix G
3 Zero-Shot Prediction for Probabilistic Scaling Laws
3.1 A Brief Introduction to Scaling Laws
10
Compute Efficient Frontier
In this work, we focus on scaling laws that describe how the validation loss scales with the compute budget c. Em- pirically, this involves generating loss-compute learning curves during training and estimating the parameters of the resulting functional form.
Sealing Law
(e(() -0.058e 4. 4.62
Loss
As model and dataset sizes increase, the compute required for a single training iteration also grows, making training progressively more expensive. Let (c1, C22 ........00 CM) denote an increasing sequence of compute values, with e < C2 <... < CM- We denote the sequence of loss values achieved by a model of size n, at these compute values as 1 Figure 2 illustrates learning curves 1 for nanoGPT models [63] of varying sizes in green and the full set of Q learning curves is given by D [cc, 1 3 Model size is implicitly reflected by the position at which each curve begins the compute axis: smaller models, requiring less compute per iteration, appear further left.
1046
10
Compute
Figure 2: A set of learning curves from nanoGPT models, each showing loss sus compute for different model sizes. Larger models achieve lower test loss.
While various forms of scaling laws exist, we focus on the I(c) scaling law. Following the approach outlined by Hoffmann et al. [3], this scaling law is obtained by fitting the following functional form to the compute-efficient frontier within a defined compute region of interest: I(c)- = (c/ca) -? with
co and y being the parameters to be optimized 4. 22 Figure 2 illustrates the obtained scaling
3

---

law (shown in blue) for the nanoGPT dataset in log-log scale. In this representation, the scaling law assumes a linear form. In our study, we aim to estimate the scaling law within a compute- efficient frontier defined for the compute range of 10Ls to 1020 FLOPs (shown in black). As noted by Hoffmann et al. A Figure A5] the specific form of the I(c) scaling law depends on the compute range of interest and the subset of learning curves considered, and can therefore vary accordingly.
3.2 Gaussian Process Model for Scaling Law Prediction
In our analysis, we employ the latent variable multi-output Gaussian process (GP) model proposed by Ma et al. (24], hereafter referred to as Ma In our modelling setup, indexes tasks and d indexes data instances within each task, forming bi-level hierarchy. As shown in Figure T I may represent embedding parameters, while d corresponds to the number of layers. Thus, models share embedding parameters but differ in layer count, defining distinct model sizes n, computed from both quantities. The generative model of GP is specified as
g(x) ~ GP((, Kig (x,x')),, (x)) (xx,( xx(x (x, h.) + Et: 6, ~ N((, h
Each multitask output y/ models noisy learning curves with independent and identically distributed Gaussian noise e,. Each curve 1f( is drawn from GP with mean g(x) and covariance ky xx). The shared mean function g(x) encodes prior information common to all tasks, with covariance kg(x, x'). To model correlations between tasks, latent variable h, is introduced, distributed as he ~ I). The predictive distribution for new values I' at inputs x* in vector-matrix form can be approximated by (X'' f q(1 X H) H)H, with |X* .H) (V jm., K.). Hence, the posterior predictive distribution over the zero-shot predicted learning curves is given by a multivariate Gaussian distribution with mean m, and covariance K,, This distribution represents the predicted learning curves in our zero-shot setting. As a detailed derivation and discussion of this model is beyond the scope of this paper, the reader referred to Ma et [24]
3.3 A Probabilistic Approach to Scaling Laws
The generative scaling law in the log-log domain given by ((0 N( + 3Ic, with coefficients 30 and $1. As detailed in Subsection [.1 its form depends the available learning curves. Assuming Gaussian prediction error, we estimate 3o and B3 via Monte Carlo simulation over R runs. Starting from N observed learning curves D= {(c, 1" the Ma model is trained on D in each run E R to zero-shot predict M missing curves, yielding D = {(c, Im==)) Jim for unseen model sizes m,, In each run r, a scaling law is then fit to the compute-efficiet frontier from DUD* and final estimates of 30 and 81 are obtained by averaging over R runs: Bn 1 1/R ΣR 1= p(6 (D))
and B1 = R S P(3T D*) 64]
4 Defining Hierarchies in Natural Language Processing Datasets
In the following, we describe the hi- erarchical structures of each dataset. Hierarchies are exchangeable, as we will show in the experimental section.
Number Layers
+
Nuraber Embbedlnng Paranneeess eats
8
10
12
24
32
48
512
3.31
3.36
3.22
3.09
3.02
2.96
More details on the models are pro-
vided in Appendix B
708
3.12
3.08
3.04
2.92
2.80
2.84
Learning Curves for Scaling Laws.
983
3.06
2.98
2.96
2.87
2.81
2.73
The nanoGPT dataset used for scaling law experiments is shown in Figure 3 with learning curves defined
1024
3,03
2.98
2.94
2.84
2.80
2.71
as loss over step size. For the final experiments, these will be converted to learning curves depicting the loss as a function of compute. Each task I corresponds to a row, with data in- stances d across columns, forming the
1000
2. 2,86
2,83
2.79
2.70
2.67
2.60
Figure 3: The Ian dataset. LCs for different combi- nations of embedding parameters and layers are shown, with the minimum achieved loss indicated above each curve.

---

Target
Target -
Source
id
jv
111
ta
11
< M2M100
jv
11
175M
id
iv
615M
1.2B
Dataset size
Dstaset size
Figure 4: (left) Bilingual learning curve dataset. No data is available on the main diagonal. Light blue squares indicate test-set performance curves. Dashed lines: learning curves from the fine-tuned mBART50 model. Solid lines: learning curves from the Transformer model trained from scratch. (right) Multilingual learning curve dataset. We perform few-shot extrapolation for the M2M100-1.2 model, translating from Indonesian (id) into multiple target languages. The 615M and 175M models correspond to two separate tasks under this assumption. Light blue sections of the curves mark the extrapolated data points.
hierarchical structure illustrated in Figure Figure([eeft)) Here, indexes embedding parameters and d the number of layers, defining model size ni, Except for combinations 512-8 and 512-10, loss decreases when the number of layers or embedding parameters is increased. (Additional evidence that loss decreases with increasing model size can be found in Kaplan et al. 4 and Henighan et al. (1T). Consequently, this supports the assumption that including correlations among tasks into the modelling process is beneficial for zero-shot prediction. Additionally, we experiment with a subset, referred to as nanoGPTamall containing learning curves for embedding parameter sizes 512, 960, and 1600, and for layer counts 8, 12, 32, and 48.
Learning Curves for Bilingual Translation. The bilingual dataset is created by fine-tuning mBART50 and training a Transformer on the EMNLP2021 dataset [65] 66 67 Learning curves (Figure left) show translation performance in ChrF (68 over increasing dataset sizes. Each language involved in the dataset introduces distinct linguistic properties, such as word order, morphological complexity, and syntactic structure. The translation performance of a model is not only influenced by the size of the dataset but also by these linguistic characteristics [69] [70 [7].. Prior work shows that factors like typological similarity, dataset size, and source-language pretraining influence performance
15 72 73.. Additionally, in fine-tuning scenarios, model performance has been shown to depend on whether the source language was part of the model's pretraining corpus (74). Consequently, we structure the dataset hierarchically to improve prediction accuracy. Each task t (row) corresponds
outcomes (6 7
translation from specific source language. As a translation into target language is equally influenced by its linguistic properties, correlations among tasks d correspond to correlations among target languages (Figure (middle)).
Learning Curves for Multilingual Translation using Architectures of Various Sizes. Specifically, we investigate extrapolating the learning curve of the largest, most expensive model, assuming learning curves of smaller models are available. Curves represent BLEU performance over increasing dataset sizes, which improves with model size 4 Each task & corresponds translation from source language to target languages d using model size (Figure [ (right)) Figure 4 right, illustrates translation learning curves from Indonesian into multiple target languages across different model sizes. Leveraging correlations among curves of different model sizes for the same source- source-target pairs expected to improve prediction. Zero-shot prediction not applicable for learning curves obtained after multilingual fine-tuning, since all curves are already available. Therefore, we focus on few-shot prediction for the 1.2B model estimate expected performance over increasing dataset sizes, based on the performance of smaller models and the available data points of the largest model.
5

---

5 Experiments
The code for all experiments will be made publicly available. Evaluation metrics are reported as averages over ten independent runs for each model. The Ma model trained on r-score-normalizer learning curves and logarithmic step dataset size values, but evaluated in the original domain.
Common Baseline for all Tasks Deep Hierarchical Gaussian Processes. Hensman et al. [23 introduced the Deep Hierarchical Gaussian Process (DHGP) model capable of capturing patterns in data by modelling a two-layer hierarchical relationship, specifically between tasks and data samples d. Extending this framework with an additional hierarchy layer allows the model to identify clusters, i... groups of similar tasks t 23 24). More information about this model is given in Appendix [ DHGP is particularly suited for discovering clusters in the data, where each cluster can be thought of as a group of tasks sharing similar characteristics (e.g., number of embedding parameters). This is in contrast to Ma which assumes that inter-task correlations exist and can be captured through a latent variable. such correlations are weak or non-existent, we expect the hierarchical structure of DHGP to provide more appropriate inductive bias than Ma allowing it to learn within-cluster trends effectively.
Additional Baselines for Each Task. In addition to evaluating DHGP, we include distinct baselines for each task. This differentiation is necessary because the datasets vary in nature, and the correspond- ing learning curves are structured differently in each setting, A brief introduction to each baseline is given Appendix D with more details provided in the following.
5.1 Zero-Shot Prediction for Scaling Law Research
The nanoGPT dataset comprises learning curves of loss evaluated over 19,072 validation steps, corresponding to one complete pass through the Feb-WedEEdu (10B) dataset (75). Due to computational constraints, the nanoGPT datasets are subsampled to 11 data points per learning curve, resulting in discretised
234
compute learning curves. We employ the train-test splits Quad and Tri, as illustrated in F5gure[5] Additionally, we consider train-test split containing only the learning curves of the largest five models in the test set, referred to as TI. The largest models in the dataset correspond to the most computationally expensive ones; by predicting their performance, we aim derive accurate scaling laws at significantly reduced computational cost.
200
Ax
AB
1/6
300
186
256
380
Figure 5: Schematic show- ing the train-test splits of the nanoGPTlage dataset: Quad (top) and Tri (bottom). Blue occluded learning curves the test data.
Bayesian Neural Networks. Bayesian Neural Networks (BNNs) have shown promising results in zero-shot learning curve prediction [8]. We compare two variants: BNN (orig), which uses the originally proposed basis functions (vapor- pressure, Hill3, and logarithmic) adapted to model decaying learning curves; and BNN (I.C), which employs basis functions better suited to capture decaying learning curves trends, namely the power law, exponential decay, and logarithmic decay functions. The functional forms and illustrations of the used fasis-functions are given in Ap- pendix E Instead of a (conventional) hyperparameter configuration space, we model a configuration space defined by the number of layers and embedding parameters, normalized to the range [0, 1] between the smallest and largest model sizes. Based on previously observed training data, zero-shot predictions are made for novel layer-embedding configurations in the test set.
Alternative Baseline-
Analysis of Zero-Shot Prediction of Learning Curves. The zero-shot prediction performance of GP and its baseline methods is shown in Table [ Across all metrics, Ma consistently outperforms the baselines. This result supports our core hypothesis that this NI.P dataset exhibits hierarchical structure that can be effectively leveraged. As expected, reducing the dataset to nanoGPTsmall degrades Ga performance, reflecting the increased difficulty due to fewer available learning curves for training. Among the BNN variants, BNN (LC) performs slightly better than BNN (orig), though both show the weakest overall results, likely due to the limited training data. For consistency, all models were evaluated using fixed sample size of 11 points per learning curve. In comparison, the smallest dataset used in Klein et al. (18) contained 256 configurations, whereas the largest dataset in our study comprises only 29 learning curves, each with 11 data points. This

---

limitation also explains why we do not report BNN results for T1 11smalll on this dataset. Never- theless, model performance can be substantially improved with more data points. As shown in Appendix E Figure E.3 all error metrics improve once the sam- ple size exceeds approximately 100 data points. It should also noted that the relatively high pre- diction uncertainty is primarily due the inherent noise in our learning curve dataset.
Table 1: Zero-shot prediction performance of Ma DHGP, BNN (orig), and BNN (LC) on the nanoGPT dataset (Train-Tess (TT) splits: Quad, Tri, TI) and zero-shot performance on the nanoGPT dataset (TT split: TI T1small)
Method
TT
JMSE
JMAE
JMNLPD
Ma
Quad
0.03 = 0.01
0.12 0.02
2.58 + 2.66
DHGP
Quad
0.07 = 0.00
0.20 1 0.01
3.25 1: 0.24
BNN (LC)
Quad
10.69 = 0.43
2.82 + 0.03
596,05 +: 23.79
BNN (orig)
Quad
13.96 + 0.75
3.06 + 0.05
778.95 + 41.90
Ma
Tri
0.02 0.01
0.10
0.87 0.29
DHGP
Tr
0.07 0.00
0.00
1.85 + 0.63
BNN (LC)
Tr
11.02 = 0.47
0.04
612.30 26.22
BNN (orig)
Tn
13.67 0.82
3.06
759.96 45.42
Ma
T1
0.04 = 0.05
0.12
0.80 + 1.54
A detailed overview of the perfor- mance on the last predicted data point, as well as the last three data points in the zero-shot pre- diction setting, is provided in Ap- pendix H An error analysis of these points highlights whether the trend of each learning curve was captured accurately. In both cases, Ma significantly outperforms the other methods the nanoGPTlarg dataset and exhibits comparable performance on the nanoGP7ssmal dataset.
DHGP
T1
0.08 0.00
0.18 0.000
0.87 0.09
BNN (LC)
TI
10.39 0.86
2.00
590.54 48.97
BNN (orig)
T1
14.05 1.39
2.10
798.85 79.19
Ma
T small
0.08 0.00
0.21 0.00
0.50 + 0.06
DHGP
Tismal
0.11 1 0.00
0.21 1 0.00
0.99 1: 0.42
In our initial experiments, we set the number of embedding parameters as the first hierarchy and the number of layers as the second. But are these hierarchies interchungeable? To val- idate our additional hypothesis that hierarchies indeed exchangeable, we present results for the Ma and DHGP models in Table 2 Here, each number of layers defines a task while models with the same layer count but different numbers of embed- ding parameters form the second hierarchy level d. The results show that Ma still outperforms the baseline when the hierarchy is exchanged, supporting our hypothesis. Subsequent experiments and analyses continue with the original hicrarchical assumption illustrated Figure
Table 2: Zero-shot prediction performance of Ma and DHGP with exchanged (exch) hierarchies on nanoGPT for train-test (TT) splits Quad, Tri, and T1.
Method
JMSE
IMAE
1MNLPD
Ma exch
Quad
0.04 0.04
0.11 = 0.07
1.82 : 2.23 0.34 = 0.02
DHGP exch
Quad
0.11 = 0.00
0.26 = 0.01
Ma exch
Tri
0.02 0.01
0.09 0.03
0.37 1 0.79
DHGP exch
Tri
0.13 0.05
0.29 0.06
0.48 + 0.16
Ma exch
TI
0.01 0.02
0.08
0.04
0.88 = 0.58
DHGP exch
TI
0.16
0.34 0.02
1.18 = 0.43
10
10
Ground Truth Scaling Law. (l(c)) -0.050c 3.51
Ground Trath Scaling Low: [=a(e] -0,05% + 3.51
Triining Data Predictoo
Troining Data Prodictio
Compute ellccie, frunliet
Compute efficien froatin:
Predicned Sealing Low: (l(()) -0.02
Provicted Staling Law:
(9[(c -U.0-Se
Lose
1003
10
10
10
1015
10
20
Figure 6: Predicted learning curves (red) using training curves (green) for Quad (left) and Tri (right) training-test splits in a single run. Dashed blue lines show the predicted scaling laws; solid blue lines show the ground truth scaling law.
7

---

Scaling Law Analysis. Figure [ shows the Quad and Tri train-test splits and the predicted scaling law derived from the predicted learning curves D and the available training data D for a single run. We consider compute range between 1 and 10e0 FLOPs. Note
Table 3: Predicted scaling laws and Area between Curves (AbC) to the ground truth law (-0.056c 3.51), averaged over 10 runs for different Train-Test (TT) splits. Reported (compute) costs for obtaining the training data are given in PetaFLOPs (full nanoGPTTuge, 6.081 x 1
PetaFLOPS).
TT
Predicted Scaling Law
JAAC
[Cost
Quad
(-0.043 t 0. 0.002)c + (2.957 + 0.074)
0.521 t 0.069
1.28 10
Tri
(-0.052 + 0.005 + (3.352 + 0.202)
0.160 + 0.172
2.06
10
TI
(-0.059 + 0.006)e+ + (3.636 + 0.245)
0.111 0.222
5.15 10
that due to using subset of the full dataset, the ground-truth scaling law for nanoGPT differs from that in Figure 2 which was obtained from a larger dataset of 90 learning curves. Recent scaling law studies typically estimate the blue scaling law using the full dataset, D DD' by exhaustively training all models rather than leveraging zero-shot prediction to reduce computational cost. These results serve the "ground truth' baseline. The compute-efficient frontier appears discretised rather than smooth because each learning curve contains only 11 data points. Using Monte Carlo averaging over 10 runs, the resulting predicted scaling laws are summarized in Table Additionally, to quantify the deviation between the predicted and ground-truth scaling laws, we compute the Area between Curves (AbC)) metric, where AbC E (0, oo). The concept is illustrated in Appendix F For consistency, AbC is evaluated over the compute range 10 10023 FLOPs, with smaller values indicating closer alignment between predicted and ground-truth scaling laws. For each train-test split (TT), the slope coefficient is estimated with higher certainty than the intercept, which shows greater variability. As expected, increasing the amount of training data reduces the mean AbC but also increases its uncertainty. While the Quad split is the most compute-efficient in terms of training data acquisition, it yields the highest AbC. In contrast, the Tri and T1 splits achieve lower AbC values but the cost of higher predictive uncertainty. This motivates the question of whether a more informative train-test split strategy can be designed. To explore this, and inspired by active learning, we investigate the effect of different query strategies AbC performance (76] 77 78).
Analysis of Various Query Strategies. We begin with an initial training set of learning curves D from the nanoGPT dataset, consisting of one curve per task t corresponding to the smallest layer size (i.e., the first column in Figure At each iteration, one additional learning curve is queried according to predefined query strategy. After each query, the Ma model is retrained on the existing training set DD together with the newly queried curve. Subsequently, the test set curves D are predicted (these are the remaining curves in the nanoGPTlage dataset). Predictions are averaged over 10 runs to capture variability. Following each prediction step, a scaling law is fitted, and its performance is assessed using the AbC metric.
We evaluate four query strategies. The first, Largest First, selects the learning curve corresponding to the largest model from the test set; the second, Smallest First, selects the learning curve corresponding to the smallest model; the third, Active Learning, queries the most uncertain learning curve based on predictive variance; and finally, we use Random Order for random choice of the next learning curve to be queried. For the Active Learning strategy, the predictive mean I of each learning curve is obtained by averaging over 10 runs. The variance at each sample point is then computed as
var= E/e (Im4+ - |m Finally, the mean variance per learning curve, denoted as mvar, is calculated by averaging over its 11 sampled points: mvar= 1/11 Σjn vary.
The results of these query strategies, along with cost analysis over the number of queries, are presented in Figure 7 Querying according to Active Learning consistently results in more certain AbC values across runs. Moreover, these values are either below the other query strategies (e.g. Smallest First) or comparably high (e.g. Largest First). In contrast, random querying introduces substantial variability and uncertainty into the predictions made by MP Although the Largest First strategy achieves AbC values similar to Active Learning, it incurs higher computational cost and larger standard deviations. Overall, averaged over 10 runs, Active Learning produces the most confident learning curve predictions and the most reliable predicted scaling laws. For all query counts, the AbC from Active Learning remains consistently below (or comparable to) that of the initial training set (query number 0), a property not shared by the other strategies. The order of queried LCs for each scenario is illustrated in Appendix | Additionally, Appendix [] Figure I.I presents AbC values as function of computational cost.
8

---

1.00
1.00
Active Learning Random Order
Sanillest Pirst
Area between Curves
Area between Curves
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
01225
-1.25
15
20
25
15
20
25
Queries
Querios
1.00
Compute (PetFFLPPs)
Largest First
Area betwoen Curves
0.75
0.50
0.25
Active Learning
109
Ramdom Order
0.00
Targest First.
Smallest First
-0.25
5
10
20
25
0
10
15
20
25
Queries
Queries
Figure 7: AbC over queries using various query strategies and a cost analysis.
5.2
Zero-Shot Performance Prediction for Bilingual Translation
To further support our core hypothesis that learning curve datasets exhibit hierarchical structures, we conduct additional zero-shot prediction experiments using bilingual translation data. The correspond- ing train-test split is illustrated in Figure (left). While the figure shows learning curves measured in ChrF function of dataset size, also evaluate zero-shot prediction performance using BLEU scores, Unlike the nanoGPT dataset, this dataset is considerably noisier and more variable, with each learning curve containing between 11 and 17 points, depending on the available data for each source-target language pair.
Assuming correlations among learning curves sharing the same source s' or target t' language, we define a naive baseline (NBL) as the average curve over these respective sets for language pair s't' of interest, given by: 1/2 1/ let' 1/r E 1/1) for s' f and t' tt where S and T denote the number of available source and target languages, respectively.
Alternative Baseline.
The results are shown in Figure S BLEU and ChrF performances are not directly comparable due to differing scales. Ma exhibits higher predictive uncertainty when forecasting BLEU learning curves but consistently outperforms all baselines terms of mean RMSE. NBL shows substantial variability, indicating that simple averaging correlations. While DHGP improves over NBL, it still displays heightened predictive uncertainty. These findings confirm our core hypothesis that modeling bilingual translation tasks with two-layer hierarchical structures allows the model to leverage correlations among tasks, enhancing prediction accuracy. Further analysis is provided in Appendix and Appendix L and results for exchanged hierarchies supporting our additional hypothesis are in Appendix M
source and target language curves fails to capture meaningful
15
MaGP
20
RMSE
MNN.PD
DHGP
NBL
10
ChrF maat
Chri Transformer
BLEU mBart.
BLEU Transformer
ChrF mBat
ChrF Transformer BLEU mBart
BLEU Transformer
Figure 8: shro-shot performance prediction on the bilingual dataset.

---

15
100
Mr
DBGP
10
MNLPD
75
NTBL
RMSE
50
25
5
Figure 9: Few-shot performance prediction on the multilingual dataset, extrapolating the last nine data points of each learning curve for the M2M100 1.2B model. The extrapolated learning curves show performance over dataset size for translations into a common target language.
5.3 Few-Shot Performance Prediction for Architecture Design
hypothesis, conduct few-shot performance prediction learning curves across various model architectures in multilingual transla- tion. We assume that learning curves from already fine-tuned M2M100 models of sizes 175M and 615M are available. Using these, we extrapolate the final nine missing points of the 1.2B model's learning curves, corresponding the largest dataset sizes. This allows us assess whether collecting additional data for translation into specific target languages or from particular source languages is justified, based on the predicted performance. The extrapolation leverages both the learning curves of the smaller models and the initial performance values of the largest model.
To provide final experimental setting for validating
Alternative Baseline. We define naive regression baseline (NRBL) as a non-linear regression model trained on the available initial training data points of the 1.2B model, together with the complete learning curves of the 175M and 615M models. This model fits several functional forms to the training data, including vapor-pressure function, the MMF function, a power law function, and logarithmic function. These functions are given in Appendix O The reported performance the average across these fitted model.
Figure 9 shows the extrapolation performance when predicting into common target language. Using Ma GP we consistently outperform the baseline models in terms of RMSE, despite higher predictive uncertainty. The highest uncertainty is observed for learning curves predicting performance over dataset size for Tagalog (1/) as a target language. Nevertheless, mean predictions across all languages and translation scenarios achieve RMSE values superior the baselines. These results confirm that incorporating hierarchical assumptions and leveraging correlations among tasks enhances prediction performance. Further analysis is provided in Appendix N The hicrarchical structures in this setting are exchangeable as demonstrated in Appendix N This aligns with the previous results and confirms our additional hypothesis.
Conclusion
We demonstrate that framing learning curve prediction tasks in natural language processing as multi- task problems, in which each task is organized according to a two-layer hierarchical structure and correlations among tasks are leveraged, leads to substantial improvements in zero-shot prediction performance. Through experiments across three small-scale datasets, we provide empirical evidence supporting our core hypothesis that learning curve datasets in NLP inherently exhibit bi-level hierar- chical structure. Furthermone, our results show that these hierarchies are exchangeable, highlighting the robustness and flexibility of the hierarchical modeling approach in capturing underlying task relationships and enabling accurate prediction even when the ordering of hierarchical levels is altered.
Building the Ga framework, we further demonstrate that probabilistic scaling laws can be derived through Monte Carlo simulation, supporting our final hypothesis. By employing an active learning strategy, observe notable reduction in model uncertainty average, and the resulting predicted scaling laws closely align with the ground truth, highlighting the effectiveness of combining hierarchical modeling with principled uncertainty quantification.
Further discussion and additional details on computational complexity are provided Appendix Q and Appendix P respectively. An impact statement and limitations are given in Appendix A
10

---

Acknowledgment
We would like thank the anonymous reviewers for their constructive and insightful feedback on the initial version of this paper. Their thoughtful comments and engagement with our work have led to valuable improvements, which we believe have further strengthened the quality and contribution of this study.
This research was supported by The University of Melbourne's Research Computing Services and the Petascale Campus Initiative. MH was supported by the Australian Research Council (ARC) through grant DP230102775.
References
[1] Emily Bender, Timnit Gebru, Angelina McMillan Major, and Margaret Mitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAccT), pages 610 6000622,2220 2021.
[2] Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, et al. From words to watts: Benchmarking the energy costs of large language model inference. In High Performance Extreme Computing Conference (HPEC), pages 1-9, IEEE, 2023.
[3] Jordan Hoffimann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, et al. Training compute- optimal large language models. In Proceedings of the 36th International Conference on Neural Informa- tion Processing Systems (NeurIPS), pages 30016-30030, 2022.
[4] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, al. Scaling laws for neural language models. arXiv preprint arXiv:2001, 08361, 2020.
[5] Ahmad Faiz. Sotaro Kaneda, Ruhan Wang. Rita Osi, et al. LImcarbon: Modeling the end-to-end carbon footprint of large language models. Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024.
[6] Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, al. Predicting performance natural language processing tasks. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 8625-8646, 2020.
[7) Zihuiwen Ye, Pengfei Liu, Jinlan Fu, and Graham Neubig. Towards more fine- ine-grained and reliable NLP performance prediction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 3703-3714, 2021.
[8] Viktoria Schram, Daniel Beck, and Trevor Cohn. Performance prediction via baycsian matrix factorisation for multilingual natural language processing tasks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 1782-1795 2023.
[9] Felix Mohr and Jan N van Rijn. Learning curves for decision making in supervised machine learning: survey. Machine Learning, pages 8371 8371-8425, 2024.
[10] Tom Viering and Marco Loog. The shape of learning curves: review, Transactions Pattern Analysis and Machine Intelligence (TPAMI), pages 7799-7819, 2022.
[11] Taylan Turan, David Tax. Tom Viering, and Marco Loog. Learning learning curves. Patiern Analysis and Applications (PAA), 28:1-13, 2025.
[12] Prasanth Kolachina, Nicola Cancedda. Mare Dymetman, and Sriram Venkatapathy. Prediction of learning curves machine translation. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL), pages 22-30, 2012.
[13] Philipp Kochn. Europarl: A parallel corpus for statistical machine translation. Proceedings of the 10th Machine Translation Summit, pages 79 86. International Association for Machine Translation, 2005.
[14] Marco Turchi. Tijl De Bie, and Nello Cristianini. Learning performance of machine translation system: a statistical and computational analysis. In Proceedings of the 3rd Workshop on Statistical Machine Translation, pages 35 25-43, 2008
[15] Alexandra Birch, Miles Osbome, and Philipp Koehn. Predicting success in machine translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 745 745-754, 2008

---

[16] Baohua Gu, Feifang Hu, and Huan Liu. Modelling classification performance for large data sets: An empirical study. In Proceedings of the 2nd International Conference Advances in Web-Age Information Management (WAIM), pages 317-328, 2001.
[17] Kevin Swersky, Jasper Snoek, and Ryan P. Adams. Freeze-thaw bayesian optimization. arXiv preprint 106.9614 2014.
[18] Aaron Klein, Stefan Falkner, Jost Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. In International Conference on Learning Representations (ICLR), 2016.
[19] Jihao Lin, Sebastian Ament, Maximilian Balandat, and Eytan Bakshy. Scaling gaussian processes for learning curve prediction via latent kronecker structure. Proceedings of the Workshop on Bayesian Decision-making and Uncertainty of the 38th Internatinal Conference on Neural Information Processing Systems (NeurIPS), 2024.
[20] Arlind Kadra, Maciej Janowski, Martin Wistuba, and Josif Grabocka. Scaling laws for hyperparameter optimization. In Proceedingy of the 37th International Conference Neural Information Processing Systems (NeurlPS), pages 47527 47553, 2023.
[21] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv: 2010. 2020.
[22] Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws. Transactions on Machine Learning Research (TMLR), 2024.
[23] James Hensman, Neil Lawrence, and Magnus Rattray, Hicrarchical bayesian modelling of gene expression time series across irregularly sampled replicates and clusters. BMC bioinformatics, pages 1- 12, 2013.
[24] Chunchao Ma. Arthur Leroy, and Mauricio Alvarez. Latent variable multi-output gaussian processes for hierarchical datasets. arXiv preprint arXiv:2308. 16822, 2023.
[25] Myrthe Veenman, Angelika Stefan, and Julia Haaf. Bayesian hierarchical modeling: An introduction and reassessment. Behavior Research Methods, pages 4600-4631, 2024.
[26] Scott Lynch. Introduction to applied Bayesian statistics and estimation for social scientists. Springer, 2007.
[27] Greg Allenby and Peter Rossi. Hierarchical bayes models. The handbook of marketing research: Uses, misuses, and future advances, pages 418 118-440,220 2006.
[28] Daphne Koller and Mehran Sahami, Hierarchically classifying documents using very few words. In Proceedings of the 14th International Conference on Machine Learning (ICML), pages 170-178, 1997.
[29] Frederic Morin and Yoshua Bengio. Hierarchical probabilistic neural network language model. In Proceedings of the International Workshop on Artificial Intelligence and Statistics (AISTATS), pages 246-252, 2005.
[30] Henrik Stryhn and Jette Christensen. The analysis hicrarchical models: past, present and future. Preventive veterinary medicine, pages 304-312, 2014.
[31] Narges Saazrf-Razaviinn and Andreas Zollmann. An overview of nonparametrie bayesian models and applications to natural language processing. Science, pages 71-93, 2008
[32] Lewis Frey and Douglas Fisher. Modeling decision tree performance with the power law. In Proceedings of the 7th International Workshop on Artificial Intelligence and Statistics (AISTATS). 1999.
[33] Sameer Singh. Modeling performance of different classification methods: deviation from the power law. Project Report, Department of Computer Science, Vanderbilt University, 2005.
[34] Mark Last. Predicting and optimizing classifier utility with the power law. In Proceedings of the 7th International Conference on Data Mining Workshops (ICDMW), pages 219-224, IEEE, 2007.
[35] Aleš Černezel. Best-fit learning curve model for the C4.5 algorithm. Informatica, pages 385-399, 2014.
[36] Tobias Domhan, Jost Springenberg, and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Proceedings of the 24th International Joint Conference on Artificial Intelligence (ICCAI), 2015.
12

---

[37] Corinna Cortes, LD Jackel, Sara A Solla, Vladimir Vapnik, and John S Denker. Learning curves: asymptotic values and rate of convergence. In Proceedings of the 7th International Conference Neural Information Processing Systems (NeurIPS), pages 327-334, 1993.
[38] Rui Leite and Pavel Brazdil. Improving progressive sampling via meta-learning on learning curves. In In Proceedings of the 15th European conference on machine learning (ECML.), pages 250-261, 2004.
[39] Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping gradient descent learning. Constructive approximation, pages 289-315, 2007.
[40] Steven Adriaensen, Herilalaina Rakotoarison, Samuel Müller, and Frank Hutter. Efficient bayesian learning curve extrapolation using prior-data fitted networks. Advances in Neural Information Processing Systems, 36:
2023.
[41] Christoph Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In Proceedings of the Conference on Computer Vision and Puttern Recognition (CVPR), pages 951-958. IEEE, 2009.
[42] Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In Proceedings of the 23rd Conference on Artificial Intelligence (AAAI), pages 646-651, 2008.
[43] Alec Radford, Jong Kim, Chris Hallacy, Aditya Ramesh, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), pages 8748-8763, 2021.
[44] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. Language models few-shot learners. In In Proceedings the 34th International Conference on Neural Information Processing Systems (NeurlPS), pages 1877-1901, 2020.
[45] Andraž Pelicon, Marko Pranjié, Dragana Miljković, al. Zero-shot learning for cross-lingual news sentiment classification. Applied Sciences, page 5993, 2020.
[46] Richard Socher, Milind Ganjoo, Christopher Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In In Proceedings of the 25th InternationL Conference on Neural Information Processing Systems (NeurIPS), volume 26, 2013.
[47] Greg Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, et al. Tensor programs v: Tuning large neural networks via shro-shot hyperparameter transfer. arXiv preprint arXi::2203.0344 2022.
[48] Edwin V. Bonilla, Kian Chai, and Christopher Williams. Multi-task gaussian process prediction. In In Proceedings the 19th International Conference Neural Information Processing Systems (NeurlPS), volume 20, 2007.
[49] Elmar Pruesse, Christian Quast, Katrin Knittel, Bernhard Fuchs, al. Silva: a comprehensive online resource for quality checked and aligned ribosomal RNA sequence data compatible with ARB. Nucleic acids research, pages 71 7188-7196, 2007.
[50] Pahalage Sandaruwan and Champi Wannige. An improved deep learning model for hierarchical classifi- cation of protein families. Plos one, page e0258625, 2021.
[51] Nils Strodthoff,, Patrick Wagner, Markus Wenzel, and Wojciech Samek. UDSMProt: universal deep sequence models for protein classification. Bioinformatics, pages 2401-2408 2020.
[52] Pámela Rezende, Joicymara Xavier, David Ascher, et al. Evaluating hierarchical machine learning approaches to classify biological datahases. Briefings in bioinformatics, page bbac216, 2022.
[53] Sunho Park and Seungjin Choi. Hierarchical gaussian process regression. In In P Proceedings of the 2nd Asian Conference on Machine Learning (ACML), pages 95-110. 2010.
[54] Neil Lawrence and Andrew Moore. Hierarchical Gaussian process latent variable models. Proceedings of the International Conference in Machine Learning (ICMI., pages 481 481-488, 2007.
[55] Yasaman Bahri, Ethan Dyer, Jared Kaplan, et al. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, page e231 1878121, 2024.
[56] Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre- pre-training. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
13

---

[57] Yamini Bansal. Behrooz Ghorbani, Ankush Garg, et al. Data scaling laws in NMT: The effect of noise and architecture. In Proceedings the International Conference Machine Learning (ICML), pages 1466-1482, 2022.
[58] Mark Johnson, Peter Anderson, Mark Dras, and Mark Steedman. Predicting accuracy on large datasets from smaller pilot data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACT.), pages 450-455, 2018.
[59] Alex Hägele, Elie Bakouch, Atli Kosson, Leandro Von Werra, et al. Scaling laws and compute-optimal training beyond fixed training durations. In Proceedings of the 38th International Converence on Neural Information Processing Systems (NeurlPS), pages 76232-76264, 2024.
[60] Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, et al. Scaling laws for neural machine translation. Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[61] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, ct al. Deep learning scaling prodictable, empirically, arXiv preprint arXiv: 2017.
[62] Jonathan Rosenfeld. Scaling laws for deep learning. arXiv preprint 108.0:2 2021.
[63] Andrej Karpathy. NanoGPT.
2022.
[64] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[65] Yuqing Tang. Chau Tran, Xian Li, et Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint ar Xiv::0000 00400, 2020.
[66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, et al. Attention is all you need. In Proceedings of the 21th International Conference on Newral Information Procesing Systems (NeurlPS), 2017.
[67] Conference on Machine Translation (WMT). Wmt 2021 large-scale multilingual translation task. https //www.ste. statmt. http://mt2ilaaae-eeeallllemmmala wwa.saasttttaaaaannnioooaaaoeoee task. html 2021.
Accessed: 2023.
[68] Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the 10th Workshop on Statistical Machine Translation, pages 392-395, 2015.
[69] Kahif Shah and Lucia Specia. Quality estimation for translation selection. In Proceedings of the 17th Annual Conference of the European Association for Machine Translation, pages 109-116, 2014.
[70] Zhanihek Kozhirbayev. Enhancing neural machine tramslation with fine-tuned mBART50 pre-trained model: An examination with low-resource translation pairs. Ingenierie des Systemex d' I'Informatin, page 831, 2024.
[71] Eric Khiu, Hasti Toossi, David Anugraha, Jinyu Liu, et al. Predicting machine translation performance on new-resource languages: The role of domain similarity. Findings of the Association for Computational Linguisties (EACL). pages 1474-1486, 2024.
[72] Anirudh Srinivasan, Sunayana Sitaram, Tanuja Ganu, et al. Predicting the performance of multilingual NLP models. arXiv preprint arXiv: 10.00875. 2021.
[73] Surangika Ranathunga, En-Shiun Lee, Marjana Prifti Skenduli, et al. Neural machine translation for low-resource languages: A survey. ACM Computing Surveys, pages 1-37, 2023.
[74] En-Shiun Lee. Sarubi Thillainathan, Shravan Nayak, et al. Pre-trained multilingual sequence-to-sequene models: A hope for low-resource language translation? In Findings of the Association for Computational Linguistics (ACL), pages 58-67, 2022.
[75] Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, al. The fineweh datasets: Decanting the web for the finest text data at scale. In Proceedings of the 38th International Conference on Neural Information Processing Systems (NeurIPS), pages 3081 1-30849, 2024.
[76] David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning, In Proceedings of Machine Learning, pages 201-221, 1994.
[77] Burr Settles. Active learning literature survey. University of Wisconsin-Madison Madison Department of Computer Sciences, 2009.
14

---

[78] David Lewis and William Gale. A sequential algorithm for training text classifiers. In Proceedings of the 17th annual International Conference on Research and Development in Information Retrieval (ACM SIGIR), pages 3-12, 1994.
[79] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understand- ing by generative pre- pre-training, 2018
[80] Ilya Loshchilov, Cheng--Pnng Hsieh, Simeng Sun, and Boris Ginsburg. ngpt: Normalized transformer with representation learning on the hypersphere. In Proceedings of the 13th International Conference on Learning Representations (ICLR), 2025.
[81] Mayukh Deb. Mainak Deb, and N Murty. Toponets: High performing vision and language models with brain-like topography. arXiv preprint arXiv:2501.163%. 2025.
[82] Eli Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky, and Assaf Arbelle. Nu- merologic: Number encoding for enhanced Ilms' numerical reasoning. arXiv preprint arXiv:2404.0000 004549 2024.
[83] Bali Ranaivo-Malançon. Automatic identification of close languages-case study: Malay and indonesian. Transactions on Computer and Information Technology (ECTI-CIT), pages 126-134, 2006.
[84] Thomas Conners and Claudia Brugman. Javanese Influenced Indonesian: Features from two conversations. PhD thesis, Tokyo University of Foreign Studies, 2020.
[85] Thomas Wolf, Lysandre Debut, Victor Sanh, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the conference on Empirical Methods in Natural Language Processing (EMNLP), pages 38 45, 2020.
[86] Aran Komatsuzaki. One epoch is all you need. arXiv preprint arXiv:1906.06669 2019.
[87] Myle Out, Sergey Edunov, Alexci Baevski, et al. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint 1XXv:0994400038 2019.
[88] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:
arXiv:/808.00 (0w::8888000220 06226, 2018.
[89] Naman Goyal, Cynthia Gao, Vishrav Chaudhary, et al. The flores-101 101 evaluation benchmark for low- resource and multilingual machine translation. Transactions of the Association for Computational Linguistics (ACL), pages 522-538, 2022.
[90] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (ACI.) pages 311-318, 2002.
[91] Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. Neural machine translation for low-resource languages: survey, arXiv preprint arXiv:2106. 15115, 2021.
[92] Michael A Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow, survey on recent approaches for natural language processing in low-resource scenarios. arXiv preprint 2010 12309, 2020.
[93] Alexandre Magueresse, Vincent Carles, and Evan Heetderks. Low-resource languages: A review of past work and future challenges. arXiv preprint 20Xx::2006.07644 2020.
[94] Angela Fan, Shruti Bhosale, Holger Sehwenk, et al. Beyond anglish-centri multilingual machine translation. In P Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 718-728, 2021.
[95] Roce Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. arXiv preprint arXiv: /tt:://903000000 2019.
[96] Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. Transfer learning for low-resource neural machine translation. arXiv preprint 160iv:1604.0220) 2016.
1971 Toan Q Nguyen and David Chiang. Transfer learning across low-resource, related languages for neural machine translation. arXiv preprint 1708 09803, 2017.
[98] Marcus Hutter. Learning curve theory. arXiv preprint arXiv:1102.04074. 2021.
15

---

[99] Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension. Journal of Machine Learning Research (JMLR), pages 1-34, 2022.
[100] Ibrahim Alahdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. In Proceedings of the 26th International Conference Neural Information Processing Systems (NeurIPS), pages 22300-22311 2022.
[101] Niklas Muennighoff, Alexander Rush, Boaz Barak, al. Scaling data-constrained language models. In Proceedings of the 28th International Conference Neural Information Processing Systems (NeurIPS), 2024.
[102] Pablo Villalobos, Anson Ho. Jaime Sevilla, et Position: Will we run out data? limits of scaling based on human- -umanngeeerattdd data. In Proceedings of the 41th International Conference on Machine Learning (ICML), 2024.
[103] Dan Hendrycks. Introduction to Al Safery, Ethics and Society. Dan Hendrycks, 2024.
[104] Gediminas Adomavicius and Alexander Tuzhilin. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. Transactions on knowledge and data engineering, pages 734-749 2005.
[105] Aditya Siddhant and Zachary Lipton. Deep bayesian active learning for natural language processing: Results of a large-scale empirical study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2904-2900 2018.
[106] Steven Hoi, Rong Jin, Jianke Zhu, and Michael Lyu. Batch mode active learning and its application to medical image classification. In Proceedings of the 23rd International Conference on Machine Learning (ICML), pages 417 4244424,22 2006.
[107] Ashish Kapoor, Kristen Grauman, Raquel Urtasun, and Trevor Darrell. Active learning with gaussian processes for object categorization. In Proceedings of the 11th International Conference on Computer Vision (ICCV), pages 1 8., IEEE, 2007.
[108] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint 1rxiv:/112.555 2011.
[109] Andreas Kirsch, Joost Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In Proceedings of the 23th International Conference on Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[110] Christoffer Riis, Francisco Antuncs, Frederik Hüttel, et al. Bayesian active learning with fully bayesian gaussian processes. Proceedings of the 26th International Conference on Neural Information Processing Systems (NeurIPS), pages 12141-12111 2022.
[111] Zhenwen Dai, Mauricio Álvarez, and Neil Lawrence. Efficient modeling of latent information in supervised learning using gaussian processes. In Proceedings of the International Conference on Neural Information Processing Systems 2017
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims substantiated throughout the experiment descriptions in Section 5, backed up by corresponding figures and tables. Scope of the paper's sections is aligned with abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the separate section, which is placed the appendix/kupplementary material given the current page limits.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
17

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The experimental details are briefly outlined in the paper at the beginning of Section 5; each section then refers additional sections placed in the supplementary material which provide additional details for reproducibility. Code will also made publicly available upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
All code as well as created data will be made publicly available upon accep- tance, accompanied with detailed instructions of how to code and data. All models that have been used as well as publicly available datasets have been referenced accordingly in the paper.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nips.. cc/pu
blic/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
htwblii/gg.ddes/ddeuubiiiii //nips.cc//// wwwpccccccppccpu..eceec wwwpc//ggs/e//eeee Code//omeeeeeeeeeeeee SasSSuubiissss iobbbbmsssiinii ssiooioliiiic for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: A brief overview is provided in the experimental section of this paper (Section 5), with references to additional sections placed the supplementary material (given page limitations) and also introduced and described in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Justification: we provide the mean and standard deviation over a set of runs, typically 10 experiments to indicate the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Information provided in the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ltttps://neurippt http://ss....u.u.ii//cccccc..eee
Answer:
[Yes]
Justification:
No violations of the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Individual section, currently located in the supplementary material given the page limits.
20

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets (datasets and models) are properly credited via references to either the assets themselves or the corresponding publications.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use the package should be provided. For popular datasets, paperswithcode com datastts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Datasets consisting of a set of learning curves that have been introduced will be made publicly available under appropriate license upon acceptance, and correspondingly discussed.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (cttep/s//aarrrrrcecc////o/eeees for what should or should not be described.
23

---

A Impact Statement and Limitations
Impact Statement. Considering the performance achieved with limited computational resources and a small number of samples per learning curve, these results demonstrate the superiority and strong potential of Ma for zero-shot prediction.
In the context of scaling laws, the proposed approach offers cost-efficient strategy for determining scaling behaviors while providing valuable uncertainty estimates. When combined with active learning, enables an effective trade-off between uncertainty reduction and computational cost.
For bilingual translation tasks, assuming performance curves for certain datasets are already available, the proposed framework can save expensive training and testing times via zero-shot performance prediction of bilingual translation performances for other language- gaggussge-arsssssee of interest.
The implications for architecture design are substantial. In particular, Ma within our experimental setting, can support informed decision-making regarding dataset sizing to meet specific performance objectives when deploying larger model architectures.
Limitations. Due to computational constraints, we were limited to sample points per learning curves for the scaling law experiments, and a five times six grid of learning curves. While our results already demonstrate the advantages of Ma we suspect that more sampling points and larger grid could show additional interesting insights using this framework.
Another current limitation of Ma is the interpolation across the architectural grid. Specifically, our dataset is structured on a fixed grid defined by the number of layers and embedding parameters. As result, the model its current form cannot interpolate unseen values of embedding parameters. Interpolation across unseen numbers of layers is technically feasible, but it would yield an averaged estimate between the nearest known configurations, which may not reflect true model behavior. We see this as a promising direction for future work and plan to investigate more flexible representations address this limitation (e.g. meta parameters).
The variance- based active learning strategy explored in our work focuses on reducing the uncertainty, independent of associated compute costs. Note that there exist many different ways how the cost- function in the active learning process could be structured, but this outside the scope of this paper. We see the design of cost-terms that consider trade-offss like compute-s--unccttiitt--dddddd interesting direction of future work.
as an
B
Newly Created Datasets and their Specifications
B.1 Scaling Laws
nan.GPT. We train all nanoGPT models 663 on the 10B token split of the FineWeb-Edl
dataset [75] using four NVIDIA A100 (80GB) GPUs. A learning rate of 6 10-4 is applied with
linear warm-up and cosine decay scheduling, a minimum learning rate of 5 10 and total batch size of 524,288.
NanoGPT introduced in 2022 and is minimal, educational implementation of GPT (Generative Pre-trained Transformer) (79). The model implements the standard GPT architecture with, multi-head self-attention mechanisms, ferd-forward networks, layer normalization, positional embeddings and Transformer decoder blocks. This architecture is representative of the Transformer-based models widely used in state-of-the-ard natural language processing. Since its introduction, it has been used to conduct research on novel model architectures [80]. on topographic models [8] and on numerical reasoning (82) (among others).
B.2 Bilingual Translation
mBART. mBART50 is a multilingual sequence-to -etuuueee----seeencee model trained with denoising ob- jective over 50 languages, using special language tokens to indicate the desired target language for generation. At decoding time, a token ID must be specified to direct the model to translate into
appropriate target language. Following the approach of Lee et al. (74), we select related languages for tokenisation cases involving unseen languages. Based on syntactic and phylogenetic proximity, for example, the id tokenizer is applied for both jv and ms [83 84).
24

---

To generate new dataset of learning curves for bilingual translation. we fine-tune the mBART50 model using the HuggingFace implementation [85] Fine-tuning performed on NVIDIA A GPUs for one epoch [86], with learning rate of 5 10-5, a dropout rate of 0.1, maximum sequence lengths of 200 for both souree and target texts, and batch size of 10. For decoding, beam search employed with beam size of 5. For each language direction, the encoder-decoder parameters are initialized from the pretrained mBART50 model's corresponding encoder and decoder components.
Transformer. To create new dataset of learning curves for bilingual translation using Transformer models, we use the implementation provided by fairseq [87] and train on NVIDIA A100 GPUs. Depending on the size of the training dataset, two model configurations containing fewer than 10k parallel sentences, the model architecture comprises 3 encoder and 3 decoder layers, each with embedding dimensions of 512 and 2 attention heads. For dataset sizes equal to or exceeding 10k parallel sentences, the architecture consists of 6 encoder and decoder layers, with an embedding dimension of 256 and 2 attention heads 774.
employed. For datasets
Training is initiated with a learning rate of 1- 10-*, a weight decay of 1 10 a dropout rate of 0.4, and a batch size of 32. Early stopping is applied based on the validation loss. Input sequences are tokenised into subword units using SentencePiece [88] For decoding, beam search with a beam size of 5 is employed.
Train- Test Dataset. We use the small track dataset from the EMNLP2021 shared task on large-scale multilingual machine translation for fine-tuning mBART50 (74 and training the Transformer model [66]. This dataset consists of sentences extracted from the OPUS corpu which encompasses data of varying quality across multiple domains. The languages contained in this dataset are Tamil (ta), Tagalog (if), Malay (ms), Javanese (jv), Indonesian (id) and English (en). The number of sentences available for each translation pair varies, reflecting realistic scenario with learning curves of variable lengths for model evaluation. Consistent with the original shared task, we employ the Flores dataset 89 the test set and report BLEU 90 and ChrF scores.
The total number of sentences available for each language pair are given in Table B.1 Our learning curves exclusively represent low-resource scenarios for the language pairs id-jv / jv-id, id-ta / ta-id, jv-ms ms-jv. jv-ta / ta-jv, jv-tl / tl-jv, ms-ta ta-ms, and ta-tl tl-ta (denoted as LR in Table [BII) Among these, the language pairs jv-ms/ ms-jv. jv-ta / ta-jv, and ms-ta / ta-ms will represent extremely low-resource scenarios (eLR). Consequently, the dataset provides 14 out of 30 language pairs that reflect real-world conditions of limited bilingual corpora availability, meaning no sufficient additional data is available to create a learning curve data point indicative a high-resource setting for these pairs. It is important note that all learning curves data points correspond extremely low-resource conditions for dataset sizes below 0.5 million sentences, and to low-resource conditions for sizes below 1 million sentences. Only 16 of the 30 language pairs will have performance data points
org/ wtgg/mm////aagge-aaea scall scale- mul tettiiinguall translation task, html https://opus.. nnpp..ul
Table B.1: Total number of available sentences per language pair, sorted from largest smallest.
Language Pair
No. Sentences
en-id / id-en
34,075,906 13,612,416
en-tl / fl-en
en-ms/ /ms-en
13.437, 738
en-jy jjveen
3,044,926
id-t/ /l-id
2,743,314
id-ms/ / ms-id
2,000,000
en-ta / ta-en
2,115
ms-t tt-mm
1,358,493
jv-tl/ tl-jy
817,149
id-jy jv-id (LR)
780,125
ta-t//l-ta t-tta (LR)
563,342
ms-ta ta-ms (LR)
372,631
jv-ms / ms-jv (eLR)
434,714
id-ta/ ta-id (cL.R)
500,000
jv-ta/ ta-jy (cl.R)
66,000
25

---

available for high- nigh-resource bilingual machine translation settings, i.e., dataset sizes of above 1 million sentences.
low resturce-setting can occur if either the language or the domain of interest are low-resourced LRL could be threatened endangered languages for which little spoken/written (linguistic) resources or datasets, or number of native speakers/experts are available [9 92 93
[9]_
B.3 Multilingual Translation
In multilingual neural machine translation, the objective is to train single model capable of translat- ing between multiple language pairs 194]. Such multilingual models offer practical advantages, as they reduce the total number of models and overall model parameters required, thereby simplifying deployment (compared to bilingual machine translation). Additionally, multilingual NMT models pro- mote transfer learning: when low-resource language pairs trained alongside high-resource pairs, the shared representations and parameter sharing can lead to notable improvements in translation quality for the low-resource languages [95 96 97
Traditionally, neural machine translation systems have been predominantly English-centric, with most training data involving English either source target language. However, this does not reflect the broader translation needs of global users [94]. To address this limitation, the M2M100 model was developed.
M2M100. This model is capable of translating directly between any pair 100 languages without relying English
intermediate [94), supporting 9,900 translation directions. To pre-train M2M100, a large-scale dataset was constructed through extensive mining of parallel corpora, en- compassing wide range of language pairs beyond those involving English. This dataset enables the model to learn direct translations between numerous language combinations.
We construct a new dataset of learning curves derived from the 175M, 615M, and 1.2B parameter M2M-100 models, fine-tuned on the EMNLP2021 dataset. M2M100 models are built using the Transformer architecture (66] and implemented via fairseq [87]. We employ the Adam optimizer with parameters 31 0.90, 32 0.98, and a weight decay of 0.0001. The training objective the label-smoothed cross-entropy criterion with label smoothing of 0. The initial learning rate is set to 0.0003, scheduled by the inverse square root learning rate scheduler with 2,500 warm-up steps. The batch size (number of tokens) is 4096 32 for the 175M model, and 2048 64 for both the 615M and 1.2B models. Additional architectural details are provided in Table B.2
Train- Test Dataset.
We use the same train-test dataset in Appendix B.2
Table B.2: M2M architecture configurations.
Model Size
175M
615M
1.2B
Vocabulary Size
256K
256K
128K
Word Representation Size
512
1,024
1,024
Feed-forward Layer Dimension
2,048
4.096
8,192
Encoder/Decoder Layers
6
12
24
Attention Heads
16
16
Dropout Rate
0.1
0.1
0.1
Layer Dropout Rate
0.05
0.05
0.05
26

---

C The Generative Model of DHGP
Ma introduces an additional latent variable h to model correlations among tasks t, allowing it to capture cross-task dependencies, In contrast, DHGP does not rely on this shared latent representation. Instead, leverages hierarchical modeling approach defined as
f(x) GP(O, (x(,
9(x) ky
1(x) (x
v(x) +(x)+ + ((0,0)
Following and inspired by the approach of Hensman et al. [23] we interpret f as general function capturing global learning curve trends across tasks. The next level, g. captures variations due to coarser factor, i.e., the number of embedding dimensions. At the lowest level, models individual learning curves, which vary with the number of layers.
D
The Reasoning Behind Additional Baselines for Each Experiment
For the nanoGPTlage dataset, we have access to a sufficient number learning curves and data points, making a Bayesian Neural Network (BNN) valid and effective baseline. This choice is further motivated by prior work, such as Klein al. [88] where BNNs were successfully applied for learning curve extrapolation and zero-shot prediction.
In contrast, applying a BNN in the bilingual translation setting is more challenging. While explored modeling the learning curves as a function of source and target language pairs, the BNN struggled generalize and capture the upward trend in learning curves. Also note, we only have 25 learning curves for training, and each source and target language is only 5 times represented, which given the noisy nature of this dataset (see main paper, Figure left) is not enough training data to perform well.
Given the assumption that source and target languages may share structural similarities, instead opted for a simpler and more robust baseline: averaging available learning curves across source and target languages. This approach offers language-agnostic approximation of performance trends.
The final experiment involves an extrapolation task, where we predict learning curves from a single source (or target) language into five different target (or source) languages. Unlike the bilingual translation setup, the training grid in this scenario is defined by three distinct model sizes (see main paper, Figure 4 right). The dataset for this experiment contains only 15 learning curves, which is (again) insufficient for effectively training a BNN to generalize. Hence, regression- baseline, such polynomial power-law fitting, is more suitable and reliable choice.
E
More Information on Bayesian Neural Networks
BNN LC uses the following basis functions
fesp(x)) = a.e + fp(() =a + 10-5)-b+ +c, figg()) = a b -llo(( + 10-5) c,
Expooenntaal Deay
Power Law Denae
Logaritlanie Decay
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Figure E.1: Samples of basis functions used for BNN LC.
27

---

Vapor Preseure
Hill
Log Function
0.8
0.6
0.75
0.58
0.4
0.50
0.6
0.00
0.25
0.00
0.70
00
0.00
0.25
0.50
0.75
1.00
1.00
0.00
0.25
0.50
0.75
1.00
Figure E.2: Samples of basis functions used for BNN orig.
with samples given in Figure E.
We adjusted the originally used basis functions, designed for the extrapolation of increasing learning curves to
fv))or(x)) exp (-exp)
b
log(x 10 10=))
x 10-5
fiog(r)) = =1 c a log(bx
10010)
10
10
(a+10-
+1
with samples given in Figure H.2
For the experiments in the main paper, due to computational constraints, each nanoGPT learning curve is subsampled to 11 data points. These 11 data points are equally distributed in log-step size space to be fit using Gaussian Processes. For a fair comparison, we keep this transformation for BNN model. However, changing the sample size to be equally spaced in normal domain leads to increased performance in terms of RMSE of to below 2.0 as demonstrated in Figure
BNN orip.
1.5
1.5
RMSSE
MAB
0.5
Numberr Sumples
Number of Sumples
800
3INLPD
800
400
200
10
10
10a
Number Samples
Nursker Secples
Figure E.3: Zero-Shot BNN performance for increasing sample sizes per learning curves.
28

---

F More Information on the Introduced Metric: Area between Curves
The Area between Curves (AbC) is illustrated in Figure F. In order to apply this measure, one needs to define an AbC range of interest. In this case, it is 1013 to 1023 FLOPS (shaded area).
3.00
Ground
[*) = -0,05%c + 5.32
Proticted
2.50
[*(c) -0.02% 2.96
2.25
Aren Netween Curves
G Extended Literature
1.75
G.1 Scaling Laws
1.30
The recent surge in scaling law research offers several benefits, including improved interpretability of neural net- works [55, 56 [5], more effective training dataset size ac- quisition, and a reduced carbon footprint [36 58). While theoretical approaches have been developed facilitate the prediction of scaling laws for only a limited num- ber of architectures [55 98 991, the majority of suc- cessful methods for predicting scaling behavior across wide range of networks based empirical studies
1.25
1.00
10'
1676 10"
C
Figure F.I: Area between Curves (AbC) for the compute range of interest being 1013 to 1025 FLOPS (shaded area). The ground truth scaling law is compared to an example of a predicted LC.
3 4
21
57
58,
59
60 61 62].
Scaling laws depict a functional form f3) that characterizes the development of performance measure, such as validation loss L, relative to a scale, which may refer to compute C, the number of samples in the dataset D, or the number of parameters in the model N. In empirical approach, LCs are generated, and the parameters of the functional form f(z) = 3x + c are then estimated, for being the scale of interest, denoting the irreducible loss 21 59 and f(-) being the functional form of its performance measure [00]. As shown by Hoffmann et al.
3 Figure A5), depending on the compute range of interest, the slope of T.(C') in the log-log domain will vary.
Scaling laws operate under the assumption of an ideal scenario where increasing dataset size cor- responds to having consistently same-quality data available. However, as noted in Muennighoff et al. [OII, this is not the reality for most languages, where available data is limited, leading to data-constrained large language models (LLMs). Even for English, high-quality data is predicted to be exhausted by 2024 according to the Chinchilla scaling laws, given the current trend of training ever-larger models [102]. In data-limited regimes, strategies such repeating data alter scaling law behavior. Moreover, as mentioned in Kaplan et al. [4]. scaling law trends must eventually level off, phenomenon that straightforward power-law trend cannot predict. In certain cases, such as discriminative models like image classifiers, clear scaling laws do not emerge, and performance may plateau dataset model size increases [03
G.2 Active Learning
Active learning (AL) enables machine learning model to achieve better accuracy with fewer training instances, when the model chooses the training data learn from. It also leads to significant cost- reductions scenarios, where labeled data is expensive to obtain [77], i... perfectly suited for scaling law research expensive bilingual multilingual translation performances. AL is widely used method areas such as recommender systems [04], natural language processing [105] medical
imaging [106] or image processing [107].
AL strategies query either single data points 108 or batch data points [09
When acquisition
functions are based on a model's uncertainty, they are called model-based 110
One goal of these
functions is to minimize the expected predictive loss of the model [10]. our scenario we assume 10 runs, i.c., a committee of 10 Ma models, whose predictive uncertainty aim reduce. We define the maximum disagrecment of the models by the largest average (per sample point) variance per learning curve.
29

---

H Zero-Shot Performance Prediction for Probabilistic Scaling Laws
We are particularly interested in the prediction performance the final data points of each learning curve, these indicate the values to which the zero-shot predicted learning curves are expected to converge. error analysis of these points highlights whether the overall trend of each learning curve has been correctly captured. The results of this analysis are reported in Table H.1 and Table H.2 the nanoGPT dataset, Ma achieves substantial improvement over competing methods, while on the nanoGPTsmal dataset it maintains performance comparable to the best baselines.
Table H.I: sero prediction performance the last extrapolated point using Ma DHGP, BNN (orig), and BNN (LC). Results are shown for the nanoGPTTurg train-test (TT) splits Quad, Tri, and T1, and for T1 small on the nanoGPT<mall dataset.
(=: Exploding MNLPD values caused excessive uncertainty and limited trining data.)
Method
MSE
IMAE
(MNLPD
Ma
Quad
0.01 = 0.01
0.07 + 0.03
0.93 + 1.43
DHGP
Quad
0.03 = 0.00
0.17 0.00
0.51 + 0.08
BNN (LC)
Quad
14.44 = 2.21
3.79 + 0.29
BNN (orig)
Quad
23.79 = 3.58
4.86 + 0.38
Ma
Tri
0.00 0.00
0.06 + 0.02
-1.27 + 0.21
DHGP
Tri
0.02 + 0.00
0.15 + 0.00
-0.22 + 0.13
BNN (LC)
Tri
15.26 = 3.22
3.88 + 0.41
BNN (orig)
22.32 3.18
4.71 + 0.34
Ma GP
TI
0.00 + 0.01
0.05 + 0.05
-1.08 + 0.92
DHGP
T1
0.01 = 0.00
0.00
-0.67 10.77=0.00 0.02
BNN (L.C)
TI
15.52 3.42
3.92 0.40
434.26 32.64
BNN (orig)
T1
24.22 = 5.22
4.90 + 0.52
532.18
86.57
Ma
TI small
0.05 0.00
0.22 + 0.00
-0.03 0.01
DHGP
small
0.04 0.01
0.19 + 0.00
0.23 0.04
Table H.2: Zero-shot prediction performance on the last three extrapolated points using Ma DHGP, BNN (orig), and BNN (LC). Results are shown for the nanoGP large train-test (TT) splits Quad, Tri, and T1, and for T1 small on the nanoGPTsmal dataset.
(x: Exploding MNLPD values caused excessive uncertainty and limited training data.)
Method
TT
MSE
JMAE
JMNLPD
Ma GP
Quad
0.00 + 0.01
0.06 + 0.03
11.11 1.12
DHGP
Quad
0.03 = 0.01
4.15
0.14
BNN (LC)
Quad
14.26 2.16
3.76
480.69 9.49
BNN (orig)
Quad
23.85 = 3.36
4.86 + 0.35
582.48 + 49.97
Ma
Tri
0.00 = 0.00
0.06 + 0.02
+.10
DHGP
Tri
0.02 0.00
0.14
0.38 1 0.10
BNN (LC)
Tri
15.06 = 3.09
3.85 + 0.40
472.28
+ 20.15
BNN (orig)
Tri
22.44 3.04
4.72 + 0.33
549.79 44.15
Ma
TI
0.01 0.02
0.07 + 0.08
-1.08 0.95
DHGP
T1
0.01 = 0.00
0.09 + 0.00
-0.77 0.02
BNN (L.)
TI
15.10 + 3.27
0.39 3.86 + 0.39
442.54 37.73
BNN (orig)
T1
24.06 = 4.84
4.88 + 0.49
564.10 + 90.25
Ma
TI small
0.02 = 0.00
0.12 + 0.01
-0.54
DHGP
Tl small
0.02 + 0.00
0.12 + 0.00
-0.54
30

---

I AbC Values as a Function of Computational Costs
Figure I.J presents AbC values function of computational cost. From cost perspective, Smallest First remains competitive with Active Learning when higher uncertainty is acceptable. Beyond 2- 10s PetaFLOPs, its mean AbC value drops below that of Active Learning. In contrast, Largest First yields mean AbC values exceeding those of Active Learning after the third query, while Random Order exhibits high instability. Considering queries relative to compute costs, Smallest First enables more queries for the same PetaFLOP budget at the expense of higher uncertainty, whereas Active Learning achieves greater certainty at a higher cost.
1.0
1.0
Area between Curees
Active Learning
Area between Curves
Smallost First
Random Order
0.5
01.5
0.0
0.0
200000
400000
6000X0
206000
600DX0
Compate Cust (Less
Compute Cost (PouaFLOP))
1.0
25
Largest First
Active Leatning
Area between Curves
20-
Randoma Onier
Largest First
0.5
Queries
15
Smullestt First
10
0.0
2000000
400000
6000000
2000000
A00000
600000
Compute Cost (PeuFLOPPs)
Compute Cost. (PetaFLOPs)
Figure L.1: AbC over cost in PetaFLOPs using various query strategies and a cost analysis.
J More Information about Query Order of used Query Strategies
Figure ( shows the order various query strategies were using when querying the next learning curve. () are the learning curves in the initial training dataset. Using the active learning strategy results in more certain learning curve predictions and translates into more certain scaling law predictions (compare main paper).
31

---

Number Layers
+ Number Embeeling Paraneters
g
10
12
24
32
48
512
3
10
768
0
21
19
17
14
966
20
15
25
24
1024
23
16
1000
12
(a) Active Learning.
Number Layers
Number Eabelling Paraenelers
512
768
9
960
15
11
1024
23
17
3
22
1600
8
24
10
(b) Random Order.
Number Layers +
Number Embeeding Parametens
g
10
12
24
32
48
512
25
24
23
20
16
708
0
22
21
14
13
980
17
12
1021
15
1600
10
(c) Largest First.
Number Layers
Number Enheding Parsmetess
10
12
24
32
512
10
768
960
14
18
21
1024
15
20
1600
25
(d) Smallest First.
Figure J.1: Query order.
32

---

K
Zero-Shot Performance Prediction for Bilingual Translation (Random Train-Test Split)
The train-test split in the main paper follows structured pattern to ensure that each row and column of the dataset is missing exactly one language pair. In addition, we conducted experiments using random train-test split. The zero-shot prediction performance under this setting is reported in Table K.T further analysis of the prediction performance for the last value and the last three values is provided in Table K.3 and Table K.2 respectively. All experimental settings were otherwise kept
as in the main paper.
All results show that Ma outperforms the baseline method DHGP
Table K.1: Performance for zero-shot prediction on the bilingual dataset assuming random train-test
split. All metries are computed on all samples of the entirely predicted learning curves.
Method
Metric
Model
JRMSE
JMSE
JMAE
JMNLPD
MP
ChrF
mBart
8.29 1.71
83.94 + 35.16
6.51 + 1.48
6.26 3.31
DHGP
ChrF
mBart
12.20 + 2.16
179.34 + 58.02
10.93 + 2.05
8.61 3.05
Ma
ChrF
Transformer
4.79
1.54
30.24 1: 23.21
3.72 1 1.03
0.72
DHGP
ChrFF
Transformer
5.97
1.30
44.07 + 19.67
4.46 :: 0.81
4.75 1.45
BLEU
mBart
3.90
1.48
22. 16 +:: 16
3.52 4: 1.49
7.01 4. 14
DHGP
BLEU
mBart
6.43 = 1.17
61.07 + 22.01
5.62 + 1.00
18.03 6.61
GP
BLEU
Transformer
1.45
0.63
4.92 + 4.16
0.95 + 0.39
4.55 5.68
DHGP
BLEU
Transformer
1.80 1.80::00 0.30
5.21 + 1.59
1.12
22.29 28.13
Table K.2: Performance for zero-shot prediction on the bilingual dataset assuming random train-test split. All metrics are computed for the last three predicted values.
Method
Metric
Model
JRMSE
JMSE
MMAE
JMNLPD
Ma
ChrF
mBart
4.88 1. 2.27
51.01 1. 48.57
4.40 1 1.89
4.67 1.88
DHGP
ChrF
mBart
9.78 +
149.89 + 88.37
9.42 = 2.46
6.03 + 2.84
Ga
Chrl
Transformer
6.23
2.98
65.01 + 72.79
6.00 2.93
5.12 2.48
DHGP
ChrF
Transformer
8.88 2.80
113.57 69.29
8.52 = 2.76
8.42 5.03
Ma
BLEU
mBart mBart
3.65
23.43 +: 17.11
3.60 1.67
5.79
DHGP
BLEU
7.10
100.70 + 42.13
7.86 = 2.15
21.09 10.79
Ma
BLEU
Transformer
2.48 + 1.17
14.22 + 11.16
2.40 1.16
11.98
14.99
DHGP
BLEU
Transormer
2.99 0.71
14.47 + 5.59
2.74 = 0.65
53.24 = 83.49
Table K.3: Performance for zero-shot prediction on the bilingual dataset assuming random train-test split. All metrics are computed for the last predicted value.
Method
Metric
Model
JRMSE
JMSE
JMAE
[MNLPD
Ma
ChrF
mBart
4.01 + 1.50
23.65 + 17.76
4.01 + 1.50
4.14 1.69
DHGP
ChrF
mBart
8.84 + 2.06
116.56 + 52.88
8.84 + 2.06
4.71 1.16
Ca
ChrF
Transformer
6.60 + 3.55
82.83 + 98.82
6.60 = 3.55
6.00 = 3.64
DHGG
ChrF
Transformer
9.37 1. 3.65
144.29 1 103.84
9.37 1 3.65
9.51 1 5.95
CP
BLEU
mBart
3.67 .L 1.75
24.32 .. 18.25
3.67 1 1.75
5.81 3.42
DHGP
BLEU
mBart
8.05 + 2.38
107.86 + 48.53
8.05 + 2.38
22.03 + 13.63
aa
BLEU
Transformer
2.55 1.12
14.06 + 10.21
2.55 1.12
12.23
14.38
DIGGP
BLEU
Transformer
3.12.11.10
15.40 1 9.46
3.12 1 1.10
43.26 69.56
33

---

LMSE L Zero-Shot Performance Prediction for Bilingual Translation (Train-Test Split as in the Main Paper)
We provide further analysis of the zero-shot performance prediction results for the bilingual dataset this appendix.
Table presents the performance metrics for the full zero-shot predicted learning curve. The predicted performance the last three data points and the last data point is reported in Table L.22 and Table L.3 respectively. Table L.] demonstrates that Ma outperforms the baselines in terms of RMSE and MSE. DHGP achieves comparable performance in MAE when using learning curves obtained from the Transformer model with the ChrF metric. Although NBL exhibits the lowest all metrics. The error metrics the last
uncertainty, its predictions yield the largest errors three and the last data point show that Ma outperforms all baselines in all error metrics, but showing increased MNLPD values compared the NBL baseline. visualization of the RMSE and MNLPD values for the zero-shot prediction of the last three data points and the last data point is additionally provided in Figure [..1 and Figure L.2 respectively. This demonstrates that, on average, when averaged over the learning curves in the test set, modeling hierarchical structure and accounting for correlations among tasks beneficial for zero-shot performance prediction in bilingual translation.
Table L.1: Performance for zero-shot prediction on the bilingual dataset for the train-test split used in the main paper. All metrics are computed on all samples of the entirely predicted learning curves.
Method
Metric
Model
JRMSE
LMSE
JMAE
JMNLPD
Ga
ChrF
mBart
7.02 + 1.21
56.21 + 21.24 24
5.84 + 1.05
3.96 + 0.55
DHGP
ChrF
mBart
9.54 +: 0.08
143.74
1: 3.06
8.75 0.06
7.01 t: 0.76
ChrF
3.99
123.76
8.72
3.77 0.23
GP
ChrF
Transformer
4.24 + 0.63
19.98 + 6.32
3.42 + 0.54
3.27 + 0.52
DHGP
ChrF Transformer
4.38 1: 0.00
22.35 0.01
3.37 0.00
3.24 +: 0.00
NBL
ChrF
Transformer
6.16 1: 3.49
50.14 48.39
4.30 2.05
2.75
Ca
BLEU
4.47 1: 1.09
33. 16.80
4.06 1.07
6.53
DHGP
BLEU
mBart
5.53 =: 0.00
49.13 0.18
4.77 0.01
13:28 + 1.59
NBL
BLEU
mBart
5.94 +: 4.04
56.89 51.57
5.13 3.39
3.13 + 0.45
Ca
BLEU
Transformer
1.29 1: 0.30
3.12 1 2.47
0.25
2.96 1.74
DHGP
BLEU
Transformer
1.63 -: 0.00
3.51 + 0.01
1.09 0.00
5.84 0.07
NBL
BLEU
Transformer
2.26 +: 1.23
6.63 +: 5.54
1.16 0.67
-2.75 + 0.65
Table L.2: Performance for zero-shot prediction the bilingual dataset for the train-test split used in the main paper. All metrics are computed for the last three predicted values.
Method
Metric
Model
JRMSE
JMAE
LMNLPD
ChrF
mBart
3.73 + 0.16
20.50 0.70
3.66 0.14
3.29 0.18
DHGP
ChrF
mBart
7.85 3 0.099
97.79 +: 1.85
7.77 + 0.09
5.19 0.56
NBL
ChrF
mBart
11.1 10.78
239.5 + 319.75
10.73 10.63
3.98 0.51
Ga
ChrF
Transformer
5.58 + 0.81
39.40 +: 11.64
5.39 -:99-00 0.83
4.21 +: 0.93
DHGP
ChrF Transformer
5.84
53.14 0.09
5.44
4.21 0.01
NBL
ChrF
Transtormer
11.59 1.157
201.05 225.75
11.24
3.96 0.44
BLEU
mBart
3.99 1.55
30.18 19.14
3.95 1.57
5.88 7.21
DHGP
BLEU
mBart
7.75 0.01
108.46 0.110
7.70 4: 0.00
27.75 3.73
NBL
BLEU
mBart
8.52 7.47
128.33 156.92
8.41 7.39
3.69 0.59
MaG
BLEU
Transformer
1.93 +- 0.60
9.36 9.39
1.87 + 0.61
7.19 6.27
DHGP
BLEU
Transformer
2.822-
11.24 0.06
2.63 0.001
16.63 0.22
NBL
BLEU
Transformer
2.79
30.43
+28.3
34

---

Table L.3: Performance for zero-shot prediction on the bilingual dataset for the train-test split used in the main paper. All metrics are computed for the last predicted value.
Method
Metric
Model
[RMSE
LMSE
1MAE
1MNLPD
Chri
mBart
3.64
19.88
3.19 0.19
DHGP
Chri
mBart
7.53 0.10
93.63 1.72
7.53 0.10
5.14
NBL.
Chri
mBart
13:23 + 13.85
367 490.53
13.23 13.85
4.10 0.65
Ma CP
Chri
Transformer
6.29 1: 0.84
51.33 15.49
6.29 0.84
4.72 1.22
DHGP
Chrl
Transformer
6,82 1: 0.01
91.06 0.09
6.82 0.001
5.72 0.02
NBL
ChrF
Transformer
13.19 = 8.99
254.96 283.48
13.19 + 8.99
4.11 = 0.42
M
BLEU
mBart
3.91 + 1.56
30.37 19.66
3.91 1.56
5.63 + 7.04
DHGP
BLEU
mBart
0.01
120.50
0.210
8.04
33.29 1:
NBL
BLEU
mBart
9.56 4: 9.26
177.08 226.57
9.56
3.87 1 0.77
Ma GP
BLEU
Transformer
1.87 0.49
8.79 8.20
1.87
- 5.10
DHGP
BLEU
Transformer
3.33 0.01
17.66 0.07
3.33 0.001
20.59 1: 0.25
NBL
BLEU
Transformer
5.62 3.62:33.224
42,00 32.91
6.62 4.223
3.24 + 0.43
25
35
Ma
RMSE
20
MNLPD
30
25
DHGP
15
20
NBL
10
15
10
ChrF miBart
CheF Trausformet BLEU mBBart
BLEU Transformer
Chri mBart
ChrF Transformet BLEU mBart
BLEU Transformer
Figure L.I: Performance for zero-shot prediction on the bilingual dataset for the train-test split used the main paper. All metries are computed for the last three predicted values.
25
35
MaGP
RMSE
20
MNLPP PD
30
25
DHGP
15
20
NBL
10
Chri mBart
ChrF Transformer BLEU mBart
BLEU Transformer
ChrF mBat
ChrF Transformer BLEU mBart
BLEU Transformer
Figure L.2: Performance for zero-shot prediction the bilingual dataset for the train-test split used in the main paper. All metrics are computed for the last predicted value.
Additionally, Figure and Figure L.4 show the individual performance predictions for each learning curve, which corresponds to the translation performance from source into target language, using BLEU and ChrF as evaluation metries. respectively. Figure L.3 shows, that while Ma yields RMSE values higher than at least one baseline using the mBART50 (mB) dataset and the language pairs id-jv, en-id, and jv-ms, performs comparably to the baselines using the Transformer dataset, underperforming only for en-id and jv-ms. Similarly, Figure[L.4]eer [e that Ma underperforms the baselines for the language pairs id-jv, en-id, jv-ms using the mBART50 (mB) dataset, and for ju-ms and ta-tl with the Transformer (T) dataset.
35

---

50
RASS
NBL
HDGP
MNLPD
taf:: tn
mil:. then
mitt griid
mntt mrans mitt
milt kd-jv sutl:
mit cnrid
Heen
mi.
150
MAE
MSE
mil: 1d-iv
mil: cot
them
mit: Jram
Inf: kd-iv
mit d-a
mb: Hd
nilf mil
Figure L.3: Performance for zero-shot prediction on the bilingual dataset for the train-test split used in the main paper. All metries are computed on all samples of the entirely predicted learning curves using either mBART50 (mB) or the Transformer (T) dataset. Performance prediction was done for the metric BLEU.
MP
600
HDGP
RMSE
NBL
MSR
10
200
maB: lobje mill ma ti
mitt. ca-ld
mBl kijs
mit: mh de
mitt on-ld
mill
mill
mill
20
MAE
miB: idbje
nuft salid
23
mft je-jv
mmn
Jp-01e
milk
mid:
mil
Figure L.4: Performance for zero-shot prediction on the bilingual dataset for the train-test split used in the main paper. All metrics computed on all samples of the entirely predicted learning curves using either mBART50 (mB) or the Transformer (T) dataset. Performance prediction was done for the metric ChrF.
M Exchanged Hierarchies on the Bilingual Dataset
To validate the exchangeability of hierarchies, we present additional results for the Ma and DHGP models in Figure M.I where target languages are assigned to the upper hierarchy level and source languages to the lower level. As shown, Ma significantly outperforms DHGP in terms of RMSE under this assumption. Furthermore, comparison of both hierarchical assumptions in Figure M.2 indicates that the exchanged hierarchy yields a slight RMSE improvement. This suggests that translations into common target language share more information suitable for the upper hierarchical level with significantly reduced uncertainty.
36

---

10
Ma exch
20
DRGP exch
RMSE
MNLPD
Chri mBart
CheF Trausformer BLEU mBart
BLEU Transformr
ChrF mBart
ChrF Transformer BLEU mBart
BLEU Transformer
Figure M.1: Zero-shot learning curve prediction results on the bilingual dataset assuming the exchanged (exch) hierarchy: target languages over sources languages.
7.5
20
M
RMSE
MNLPD
MP
exch
5.0
10
2.5
ChrP mart
ChrP Transformet
BLEU mBart
BLEU Transformer
ChrF mBat
ChrF Transformer BLEU mBart
BLEU Transformer
Figure M.2: Zero-shot learning curve prediction results on bilingual dataset. Comparing the RMSE values of both hierarchical assumptions: source languages over target languages and the exchanged version (exch).
N Few-Shot Performance Prediction for Architecture Design
To offer another experimental setting to confirm our core hypothesis that learning curve datasets exhibit hierarchical structures, in this section provide experiments for few-shot performance pre- diction of learning curves across various model architectures obtained from multilingual translation.
We assume that learning curves of already fine-tuned M2M100 models of sizes 175M and 615M are available. Based on these, we predict (i.c., extrapolate) the final nine missing data points of the learning curves for the 1.2B model, which correspond to performance values at the largest dataset sizes. Accordingly, we assess whether acquiring additional data for translation into a specific target language from particular source language is justified based the predicted performance. To this end, we utilize the learning curves of the smaller models alongside the initial performance values of the largest model extrapolate the final data points.
Exchanged Hierarchies on the Multilingual Dataset
Figure N.1 shows the extrapolation performance when predicting the performance for a translation from common source language. Similar to predicting the performance into common target language (main paper), using G. we consistently outperform the baseline models in terms of RMSE, despite higher predictive uncertainty. The highest uncertainty observed for learning curves predicting performance over dataset size for Tagalog (r/) and Tamil (ta) as source languages. Nevertheless, mean predictions across all languages and translation scenarios achieve RMSE values superior to the baselines. These results confirm that incorporating hierarchical assumptions and leveraging correlations among tasks enhances prediction performance. Furthermore, this confirms our additional hypothesis, that hierarchies are exchangeable. The numerical values for these Figures are given in T.ble
37

---

10
RMSE
Figure N.1: Few-shot performance prediction on the multilingual dataset, extrapolating the last nine data points of each learning curve for the M2M100 1.2B model. The extrapolated learning curves show performance over dataset size for translations from a common source language.
Extended Analysis
The RMSE extrapolation performance for the last 3 and values are demonstrated in Figure N.2 and Figure N.3
Additionally, all performance metrics are provided in Table N.2 and Table N.3 respectively. In these scenarios, Ma does not consistently outperform the baselines for every individual source target language, but its performance remains broadly comparable. These experiments underline the particular strength of the proposed framework in situations where multiple data points require extrapolation, demonstrated in Figure N.1 and Figure 9 in the main paper. In those settings, Ma substantially outperforms all baseline methods.
10
10
MP
DHGP
NRBL
RMSE
RMIS
111 6d
115
La
381
jv
1121
La
Figure N.2: Few-shot performance prediction on the multilingual dataset, extrapolating learning curves for the M2M100 1.2B model. The extrapolated learning curves show performance over dataset size for translations into a common target language. Left: Extrapolation
last three data points. Right: Extrapolation final data point.
10
10
RMSE
211
DB
1v
112
1.
Figure N.3: Few-shot performance prediction on the multilingual dataset, extrapolating learning curves for the M2M100 1.2B model. The extrapolated learning curves show performance over dataset size for translations from a common source language shown on the x-axis. Left: Extrapolation of the last three data points. Right: Extrapolation of the last data point.
38

---

Table N.1: Few-shot performance prediction the multilingual dataset assuming interested in the extrapolation of the last missing values of learning curves for the 1.2B model. The results show the extrapolation performances for learning curves of the multilingual translation from common source or into a common target language.
Method
Language
JRMSE
JMSE
.MAEE
JMNLPD
Ga
Source: en
3.29 + 0.63
12.17 + 4.41 4.41
2.89 + 0.61
6.91 + 2.81
DHGP
Source:
5.79 0.56
45.72 7.55
4.81 0.47
3.10 = 0.10
NRBL
Source: en
= 2.20
7.70 2.20
7,00 3.50
Ma GP
Source: id
1.86 0.32
4.00
1.60 0.32
6.43 3.33
DHIG
Source: id
3.33 0.20
15.92 1.60
3.02 0.20
2.82 = 0.04
NRBL
Source: id
6,10 3.80
5.80 3.90
5,30 = 5.80
3.60 = 0.39
Ma
Source: jv
0.98 = 0.01
1.09 + 0.01
0.74 0.01
6,20 = 0.32
DHGP
Source: jv
5.21 + 0.02
31.01 0.15
1.38 + 0.01
3.20 = 0.02
NRBL
Source: jv
5.70 = 1.10
4.90 + 1.00
3.40 = 1.20
5.90 = 1.50
Ma
Source: ms
1.68 + 0.10
3.01 0.33
1.46 + 0.10
8.66 + 2.80
DHGP
Source:
4.61 + 0.00
30.10 + 0.02
4.15 +.00
2.98 - 0.00
NRBL
Source:
4.50 2.20
4.10 + 2.00
2.50 = 2.00
1.80 2.90
Ma
Source: ta
1.48 + 0.03
2.31 0.07
1.16 + 0.02
15.01 = 0.55
DHGP
Source: ta
3.73 0.47
17.32 3.64
3.20 0.40
2.65 0.12
NRBL
Source: ta
6.50 1.90
5.80 1.70
4.60 2.20
=
Ma
Source: tl
1.87 0.01
3.52 0.04
1.58 0.01
15.46 0.55
DHGP
Source: tl
8.01 0.02
72.42 0.48
7.26 0.02
4.03 = 0.19
NRBL
Source: tl
7.60 + 1.70
7.00 1.70
6.10 + 2.40
6.00 = 1.40
Ma
Target: en
2.11 0.19
4.81 0.74
1.82 0.19
11.09 7.94
DHGP
Target: en
2.90 0.15
12.24 0.40
2.52 0.19
2.57 = 0.02
NRBL
Target: en
7.20 4.10
6.70 + 4.10
6.80 + 5.80
5.20 = 1.50
MaGi
Target: id
1.66 0.15
3.21 0.79
1.43 = 0.15
20.50 = 17.62
DHGP
Target: id
3.42 0.12
13.03 + 0.83
2.83 + 0.10
2.61 = 0.06
NRBL
Target: id
7.30 2.50
6.90 2.60
6.00 + 3.80
2.00 = 2.90
Ma
Target: jv
1.43 1 0.11
2.06 0.32 0.32
1.19 + 0.05
9.96 = 3.77
DHGP
Target:
2.12 + 0.41
5.42 2.32
1.81 + 0.39
2.16 = 0.29
NRBL
Target:
3.60 + 1.10
3.40 1.10
1.40 8.10
8.20 + 8.10
Ma
Target:
0.92 0.01
0.90 0.01
0.78 0.01
6.50 0.29
DHGP
Target:
3.00 0.64
11.02 3.31
2.47 0.48
2.30 0.12
NRBL
Target: ms
5.60 + 1.60
4.90 1.40
3.40 1.60
4.80 =
Ma GP
Target:
2.50 0.01
6.51 0.03
1.78
19.21 = 0.37
DHGP
Target: ta
11.21 = 0.17
128.25 3.99
10.02 = 0.15
4.62 = 0.12
NRBL
Target: ta
7.70 + 1.20
7.10 1.20
6.10 + 2.00
3.80 - 0.32
Ma
Target:
3.36 = 1.09
13.10
8.25
2.93 = 0.93
67.88 = 38.98
DHGP
Target: tl
7.84 1 0.28
65.56 4.32
7.16 = 0.26
3.84 = 0.18
NRBL
Target: tl
7.10 + 0.60
6.30 0.62
5.10 + 8.70
4.00 = 0.72
39

---

Table N.2: Few-shot performance prediction the multilingual dataset assuming interested in the extrapolation of the last missing values of learning curves for the 1.2B model. The results show the extrapolation performances for learning curves of the multilingual translation from common source (Source) or into a common target (Target) language.
Method
Language
JRMSE
JMSE
J.MEE
JMNLPD
Ma
Source: en
2.31 0.82
7.30 3.96
2.16 + 0.87
2.73 0.15
DHGP
Source: en
3.26 0.08
19.67 + 1.97
3.02 + 0.09
3.15 0.12
NRBL
Source: en
1.40 + 2.20
1.40 + 2.30
0.07 + 0.00
7.00 + 0.01
Ma
Source: id
1.45 = 0.51
2.81 1.81
1.38 + 0.52
1.99 + 0.24
DHGP
Source:
1.15 0.04
2.43 0.06
1.02 + 0.05
1.79 0.01
NRBL
Source: id
3.20 2.00
3.20 + 2.00
1.40 + 1.90
2.70 0.49
Ga
Source: jv
0.62 + 0.03
0.50 0.08
0.53 + 0.02
1.12 + 0.10
DHGP
Source: jv
0.67 0.01
0.59 0.02
0.62 1 0.00
1.31 +: 0.01
NRBL
Source: jv
2.00 1.70
1.90 1.70
6.70 + 8.20
2.90
Ga
Source: ms
1.48 0.28
2.880
1.41 + 0.26
3.26 1.08
DHGP
Source: ms
1.09 + 0.07
1.56 +: 0.12
1.02 + 0.06
1.76 1 0.11
NRBL
Source:
7.70 1.00
7.70 1.00
0.02 1 0.03
4.50 1: 2.60
GP
Source: ta
1.56 0.07
2.85 + 0.32
1.42 0.07
9.49 8.14
DHGP
Source: ta
1.72 0.02
3.81 0.05
1.62 0.01
2.51 0.01
NRBL
Source: ta
2.90 +: 1.20
2.80 + 1.30
1.00 + 6.20
2.80 + 0.22
Ma
Source: t
1.80 0.02
3.74+
1.76 0.00
5.48 + 2.63
DHGP
Source:
1.56 0.00
3.08 0.03
1.42 0.01
2.00 0.01
NRBL
Source: tl
2.60 2.20
2.60 1 2.30
1.20 1 1.50
2.90 0.44
Ma
Target: en
2.26 0.58
5.74 2.29
2.15 + 0.62
11.67 11.42
DHGP
Target: en
2.00 + 0.02
4.95 0.06
1.80 + 0.01
2.05 + 0.00
NRBL
Target:
1.30 =: 0.71
1.20
2.30 2.10
3.80 + 2.70
Ga
Target: id
1.74 + 0.24
3.77 0.54
1.65 0.24
5.63 4.31
DHGP
Target: id
1.48 0.12
3 0.18
1.39 0.11
2.40 0.11
NRBL
Target: id
7.80 + 1.00
7.70
0.02 + 0.03
7.00 + 0.01
Ga
Target: jv
1.68 + 0.08
3.04
1.53 0.06
4.60 + 1.25
DHGP
Target: jv
1.93 +.03
5.26
1.84
3.29 0.02
NRBL
Target: jv
2,90 1.40
2.80 1.40
1.00 + 8.40
2.50 0.49
GP
Target: ms
0.93 0.03
1.12 0.07
0.89 + 0.02
3.73 1.03
DHGP
Targel: ms
0.95 + 0.01
1.44 0.06
0.89 + 0.01
1.73 + 0.02
NRBL
Target: ms
1.70 1.00
1.70 1.00
4.00 + 4.70
2.80 0.05
Ma
Target: ta
1.60 1.60 0.21
3.62
1.41 + 0.26
4.12 1.79
DHGP
Target: ta
0.00 1.59
2.75 0.02
1.36 + 0.02
2.19 0.01
NRBL
Target: ta
1.60
1.10
0.07 + 0.01
3.90 + 1.00
Ga
Target: 1l
2.31 0.99
6.94 4.34
2.18 + 0.88
28.23 28.66
DHGP
Target:
0.75
2.25 2.24
1.05 + 0.67
1.62 0.37
NRBL
Target: tl
2.20 + 1.00
2.20 + 1.00
6.00 + 3.80
2,80 + 0.13
40

---

Table N.3: Few-shot performance prediction the multilingual dataset assuming interested in the extrapolation of the last missing values of learning curves for the 1.2B model. The results show the extrapolation performances for learning curves of the multilingual translation from common source (Source) or into common target (Target) language.
Method
Language
JRMSE
LMSE
JMAE
JMNLPD
Ma
Source: en
1.44 = 1.10
3.88 = 4.44
1.44 + 1.10
1.85 + 0.54
DHGP
Source:
1.16 = 0.02
2.06 = 0.05
1.16 1 0.02
1.79 0.01
NRBL
Source: en
1.30 = 0.88
1.30 = 0.88
2.40 + 2.70
2.90 + 1.60
Ma
Source: id
0.80 = 0.72
1.28 = 2.01
0.80 + 0.72
1.29 + 0.51
DHGP
Source: id
0.94 0.07
1.13 0.13
0.94 + 0.07
1.49 + 0.05
NRBL
Source: id
1.70 = 1.10
1.10 1.70
4.10 3.90
4.90 + 5.20
Ma
Source: jv
0.34 + 0.02
0.14 +14+000 0.01
0.34 + 0.02
0.54 + 0.03
DHGP
Source: jv
0.89 0.16
0.28 1.09
0.89 0.16
0.18
NRBL
Source: jv
0.59 = 0.51
0.59 0.51
0.61 = 0.83
1.70 = 0.33
Ma
Source: ms
0.54 = 0.21
0.40 0.26
0.54 0.21
1.22 - 0.70
DHGP
Source: ms
0.54 0.10
0.45 0.23
0.54 0.10
1.25 0.19
NRBL
Source: ms
1.20 = 1.00
1.20 = 1.00
2.60 = 3.70
1.70 0.70
Ma
Source: ta
0.77 + 0.28
0.75 = 0.51
0.77 0.28
2.46 + 1.74
DHGP
Source: ta
1.10 = 0.12
1.62 = 0.32
+
1.70 0.13
NRBL
Source: ta
1.30 = 0.81
1.30 = 0.81
2.40 + 2.20
1.90 + 0.49
Ma
Source: tl
0.95 0.19
1.09 0.34
0.95 + 0.19
1.66 0.36
DHGP
Source: tl
1.12 = 0.01
1.66 0.00
1.12 = 0.01
1.67 + 0.00
NRBL
Source: tl
0.86 = 1.20
0.86 = 1.20
2.10 = 3.90
1.80 0.53
Ma
Target:
0.97 = 0.99
2.03 3.54
0.97 = 0.99
1.67 + 0.85
DHGP
Target: en
1.23 0.03
1.73 = 0.08
1.23 0.03
1.74 0.04
NRBL
Target:
1.00 = 0.70
1.00 = 0.70
1.60 + 1.80
4.40 + 5.50
Ma
Target:
id
1.22 = 0.50
1.93 = 1.09
1.22 = 0.50
1.66 0.36
DHGP
Target:
1.11 = 0.08
2.09 = 0.22
1.11 0.08
2.16 0.17
NRBL
Target: id
0.79 = 0.45
0.70 0.45
0.83 = 0.78
2.40 1.80
Ma
Target:
0.35 0.01
0.21 0.01
0.35 + 0.01
0.75 0.05
DHGP
Target: jv
0.85 = 0.05
0.89 0.09
0.85 - 0.05
1.36 + 0.05
NRBL
Target:
0.41 = 0.35
0.41 = 0.35
0.29 0.35
1.50 = 0.76
MaGP
Target: ms
0.72 0.08
0.60 = 0.08
0.72 = 0.08
1.40 0.12
DHGP
Target: ms
0.53 + 0.03
0.45 0.04
0.53 + 0.03
1.03 +.3++0.00
NRBL
Target:
0.48 0.59
0.48 + 0.59
0.59 + 1.00
0.25
MPGP
Target:
1.64 0.52
3.32 1.67
1.64 + 0.52
8.72 5.24
DHGP
Target: ta
1.09 = 0.60
1.72 = 1.33
1.09 + 0.60
1.67 + 0.31
NRBL
Target:
2.80 0.66
2.80 = 0.66
8.10 + 3.10
2.70 0.26
Ga
Target:
1.20 = 0.24
2.17 0.75
1.20 - 0.24
3.89 + 2.63
DHGP
Target: tl
0.61 = 0.00
0.44 - 0.00
0.61 + 0.00
1.39 + 0.00
NRBL
Target:
1.50 0.82
1.50 0.82
2.90 + 2.30
2.20 0.21
41

---

Naive Regression Baseline for Few-Shot Prediction
We define a naive regression baseline (NRBL) as a non-linear regression model trained on the available initial training data points of the 1.2B model, together with the complete learning curves of the 175M and 615M models. This model fits several functional forms to the training data, including vapor-pressure function, the MMF function, power law function, and logarithmic function. These functions are given by
[v) - exp a+ + +c logoog
JMMP ab + cr + x
fpowee (z) ax" + C, flog(2) = + a log(bz))
The reported performance is the average across these fitted models.
P On Compute Resources and Complexity
The Ma framework and DHGP model were trained and evaluated on an Intel TM i7-8565U
CPU. Baseline methods were trained and evaluated on Intel Xcon Gold 6448H CPU.
implemented using the inducing points method. Ma is originally referred as HMOGP- LV Ma et al. (24). HMOGP-LV is derived from LVMOGP Dai et al. [III] The computational complexity is
Ma
O (max(NNR, MH) max D, Mx) max(MH, Mx)),
with N being the available data points per learning curve, R being the number of learning curves per task (d), D being the number of tasks f, MM M, x R for M, being the number of inducing input points in the Th learning curve of a task and Myr being the number of inducing output points.
The wall-clock time for Ma between 17 min and 31 min, and DHGP around min (including logging. file-saving, etc.). Note that while DHGP is used as originally implemented by Ma et al. [24).
optimized libraries, Ma in its current form
Importantly: While additional compute would certainly enable larger-scale experiments, current setup demonstrates that even with a limited number of data points and resources, corresponding to reduced evaluation frequency and faster training, the Ma model performs effectively.
Additional Discussion
Data Relationships. The data relationships in the paper mimic more closely CP because they not only have a hierarchical structure, but also exhibit correlations among tasks. As discussed in the main paper, we expect this to be particularly well-suited for Ga hypothesis that is strongly supported by the obtained experimental results.
Key Advantage of Scaling Law Prediction. In Section 5.1 Table 3 we present results illustrating how closely the scaling laws predicted via our Monte Carlo-based approach approximate the original scaling laws obtained through conventional methods, namely, by fitting the scaling function to the full dataset. For our evaluation, we use the AbC measure. The key advantage of our approach lies in significantly reduced computational cost, while still achieving close alignment with the original scaling law. (Note that our test sets contain the most expensive learning curves.) This outperforms simple fitting training data it provides more support to fit the scaling law in the compute region of interest.
Range of Models Considered for Scaling Law Experiments. Due to computational constraints, we focused our setup on training models for one epoch using the Fine Web-Eeuu dataset. Our model sizes range from 51M to 1.5B parameters, aligning with the scale used in Kaplan et al. 4 who explored models from 768M to 1.5B parameters. We therefore consider this range representative for investigation.
42

---

Hypothesis on Hierarchical Structures. To support our hypothesis that hierarchical structures exist and can be exploited in learning curve datasets, we provide extensive experiments across multiple domains, including bilingual and multilingual translation tasks. To the best of our knowledge, our work is the first investigate bi-level hierarchies for zero-shot learning curve prediction and extrapolation.
43


Reviews:
[{"review_id": "sN0exIAuey", "reviewer": "Reviewer_QuFj", "summary": "The paper applies hierarchical Gaussian processes, often used in other types of hierarchical datasets, to the task of predicting learning curves (LCs) for NLP models and estimating frontier scaling laws for these LCs.\n\nHierarchical Gaussian processes encode outside knowledge/belief about relationships/correlations between tasks which permit stronger few-shot modeling or zero-shot modeling in LCs from tasks without much data.\n\nThe study is predominantly empirical, examining LC prediction on different small models while tweaking two model or dataset parameters (e.g. size of embedding AND number of hidden layers, or target language AND source language of translation). Hierarchical GPs permit structuring the modification of one parameter as a subtask of the supertask of modification of the other parameter.\n\nUsing a GP from Ma et al. [1], the authors explore LC prediction across a number of tasks and zero-shot few-shot settings. They also explore the error in scaling laws (linear frontier functions in log-log space) estimated from LCs. The authors compare to baselines including other GPs, Bayesian Neural Networks, and naive baselines such as averaging across tasks that are known to exist in the same subgroup in the hierarchy.\n\nResults indicate that good performance prediction and scaling law estimation can be made with Hierarchical GPs, and furthermore, that using uncertainty in probabilistic prediction can be used to select candidates for querying LCs in active learning.", "questions": "Most of my questions are actually covered in my W1 and W2. I am very curious about a deeper analysis of many of the experiments and arguments made.\n\nI have only one more question to write here that has not yet been covered.\n\n**Q1**\n\nIn Figure 7, can you include a curve with cumulative compute cost as the x axis? I wonder if Smallest First is still competitive if you consider the fact that Smallest First is much cheaper to get to a critical mass of queries as compared to any other method.", "limitations": "Limitations section is a bit short. It mostly discusses limitations on the scope and breadth of analysis. I am curious about the limitations of this methodology. For example, which relationships will Gaussian processes struggle to handle? What are the failure modes of MaGP that DHGP or other models don't suffer from?\n\nAs far as societal impacts go, I do not think there are major adversary societal impacts to be concerned with. Many techniques have been proposed or already are in use for scaling law estimation. Generally, being aware of scaling laws helps us plan to improve efficiency and reduce waste while training models.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n**S1**\n\nNovel application of this model to my knowledge.\n\n**S2**\n\nWide applicability, and selection of hierarchical modeling for LC prediction is an insightful way to apply belief about relationships in LCs.\n\nWeaknesses\n\n**W1**\n\nThe paper suffers a lack of clarity in communicating its assumptions and how they map to modeling decisions. A one-off statement in Section 5 that \"If correlations between data samples of different tasks are not relevant, or even a hindrance, this baseline is supposed to outperform the MaGP model\" but provides no further explanation of this distinction or the theoretical basis for the claim. I think extra attention should be paid to explaining a) the hypothesis about the specific types of hierarchical relationships that DHGP and MaGP can each express and b) why all of the data relationships in the paper more closely mimic MaGP and not DHGP.\n\nAs another example, another one-off statement in 5.3 mentions that \"hierarchies are indeed exchangeable\", which I interpret to mean that modeling one parameter axis as the higher level and the other as the lower, or vice versa, leads to no difference in effect. Yet what results indicate this? How does this interact with our chief assumptions? Are hierarchies exchangeable for all examined problems?\n\nIn essence I am requesting additional focus on the core hypothesis, a categorization of what evidence would contribute toward or against it, a clear set of research questions that are being examined, and clearly formatted evidence to address each research question.\n\n**W2**\n\nThe scope of experiments is relatively wide, but the depth in analysis, explanation, and experimentation granted each experimental setting leaves some to be desired. For example, why pursue a different set of baselines in each approach? Is there a justifiable reason, is it convenience, etc.? Each experimental setting is introduced technically but not in enough detail to describe these sorts of decisions.\n\nThe haphazard use of baselines can harm my assessment of the significance. I have no reason to believe right now that the paper captures a sharp enough view of the SoTA in performance prediction for each task type that the integration of the MaGP method actually means something substantial for those practitioners interested in predicting LCs.\n\nAs another example of some missing depth of analysis, the process of using Monte Carlo estimation and then fitting a scaling law to the estimates in addition to the real data is interesting but worth giving a full treatment or exploration. Why does bootstrapping a GP and then fitting a linear law tend to outperform merely fitting the linear law to the existing data? By how much does it do so? Is there any justification as to why this would be better?\n\n**W3**\n\nSeveral figures suffer from clarity issues. For example:\n\nFigure 4 Left -- how is the main diagonal modeled as test data if the test set is actually the next diagonal over to the right?\n\nFigure 4 Left -- No legend, no axes labels, no sense of what conclusions to take away.\n\nFigure 4 Right -- Font is too small to read, unclear what red vs blue numbers are, legends poorly formatted.\n\nFigure 8 -- no legend.\n\nFigure B.5 -- hard to read.\n\n**W4**\n\nTables suffer from clarity issues. For example:\n\nConvention states that a bold value is used to indicate the winner of a head-to-head competition in tables. Yet sometimes the bold values are inappropriately applied. For example:\n\nTable E.2 -- no bolding when MaGP does not win?\n\nTable H.3 -- bolding of non-winning values?\n\nUse of arrows on column headers to indicate direction of optimal metric values inconsistent. Present, for example, in Table 2 but not Table 1.\n\n**Grammatical and textual nitpicks**\n\n82 -- no space before “Recently”\n\n178 -- (512-8 two times?)\n\n263 -- Figure ??\n\n267 -- \"weather\" -> whether\n\n281 -- Flops exponents formatting off\n\n294 --\"while the compute costs\" -- incomplete phrase\n\n296 -- \"weather\" -> whether\n\n336 -- \"hense\"\n\n364 -- sentence has no following punctuation\n\n372 -- proof -> prove\n\n373 -- \"baselines\"\n\nAppendix A Line 33 -- grammar needs a revision\n\nAppendix B Line 72 -- \"increaseing\"\n\nAppendix D Line 122 -- \"commitee\" & 123 \"leanring\"", "was_revised": true, "final_justification": "Note: August 6, 2025 update: Score increased from 2->4 after author rebuttal. Final decision pending discussion with other reviewers. Quality 2->3, Clarity 1->3.", "rebuttal": "We thank you for your constructive review, and address your points individually in the following:  \n\n**[W1] - Assumptions and Modeling decisions:**  \nWe thank you for bringing this to our attention! \n\n> *I think extra attention should be paid to explaining a) the hypothesis about the specific types of hierarchical relationships that DHGP and MaGP can each express*\n\n$\\rightarrow$ MaGP introduces an additional latent variable $\\mathbf{h}$ to model correlations among tasks $t$, allowing it to capture cross-task dependencies.\n\n$\\rightarrow$ In contrast, DHGP does not rely on this shared latent representation. Instead, it leverages a hierarchical modeling approach defined as:\n$$\n\\begin{aligned}\nf(x) &\\sim \\mathcal{GP}(0, k_f(x, x')) \\\\\\\\\ng(x) &\\sim \\mathcal{GP}(f(x), k_g(x, x')) \\\\\\\\\nl(x) &\\sim \\mathcal{GP}(g(x), k_l(x, x')) \\\\\\\\\ny(x) &= l(x) + \\varepsilon, \\quad \\varepsilon \\sim \\mathcal{N}(0, \\sigma^2)\n\\end{aligned}\n$$\nFollowing and inspired by the approach of Hensman et al. [27], we interpret $f$ as a general function capturing global learning curve trends across tasks. The next level, $g$, captures variations due to a coarser factor, i.e., the number of embedding dimensions. At the lowest level, $l$ models individual learning curves, which vary with the number of layers.\n\n$\\Rightarrow$ DHGP is therefore particularly suited for discovering clusters in the data, where each cluster can be thought of as a group of tasks sharing similar characteristics (e.g., number of embedding parameters).   \n$\\Rightarrow$ This is in contrast to MaGP, which assumes that inter-task correlations exist and can be captured through a latent variable $\\mathbf{h}$.   \n$\\Rightarrow$ If such correlations are weak or non-existent, we expect the hierarchical structure of DHGP to provide a more appropriate inductive bias than MaGP, allowing it to learn within-cluster trends effectively.\n\n> *[...] b) why all of the data relationships in the paper more closely mimic MaGP and not DHGP.*  \n\nThe data relationships in the paper mimic more closely MaGP, because they not only have a hierarchical structure, but also exhibit correlations among tasks. As discussed above, we expect this to be particularly well-suited for MaGP -- a hypothesis that is strongly supported by the obtained experimental results. \n\n---\n> *[...] \"hierarchies are indeed exchangeable\", which I interpret to mean that modeling one parameter axis as the higher level and the other as the lower, or vice versa, leads to no difference in effect. [...] Are hierarchies exchangeable for all examined problems?*\n\nTo provide experimental evidence towards the hypothesis of exchangeable hierarchies (everything else kept identical), we provide the following additional results: \n\n1. Scaling law experiments (layer-over-embedding):\n| Method | TT | MSE | MAE | MNLPD |\n|-|-|-|-|-|\n| MaGP | Quad | 0.04 ± 0.04 | 0.11 ± 0.07 | 1.82 ± 2.23 |\n| DHGP  | Quad | 0.11 ± 0.003 | 0.26 ± 0.01| 0.34 ± 0.02 |\n|-|-|-|-|-|\n| MaGP | Tri | 0.02 ± 0.01 | 0.09 ± 0.03 | 0.37 ± 0.79 |\n| DHGP | Tri | 0.13 ± 0.05 | 0.29 ± 0.06 | 0.48 ± 0.16 |\n|-|-|-|-|-|\n| MaGP | T1 | 0.01 ± 0.02 | 0.08 ± 0.04 | -0.88 ± 0.58 |\n| DHGP | T1 | 0.16 ± 0.02 | 0.34 ± 0.02 | 1.18 ± 0.43 |\n\n2. Bilingual experiments (target-to-source)\n | Method | TT | MSE  | MAE  | MNLPD |\n|-|-|-|-|-|\n| MaGP | mBart chrF | 43.24 ± 13.72 | 5.297 ± 0.87 | 3.698 ± 0.43 |\n| DHGP | mBart chrF | 127.53 ± 0.10 | 8.3 ± 0.00 | 7.44 ± 0.07 |\n| MaGP | mBart BLEU | 13.287 ± 4.41 | 2.625 ± 0.45 | 5.15 ± 2.43 |\n| DHGP | mBart BLEU | 33.17 ± 0.22 | 3.835 ± 0.01 | 4.26 ± 0.14 |\n|-|-|-|-|-|\n| MaGP | Transf chrF | 18.801 ± 3.32 | 3.08 ± 0.22 | 2.866 ± 0.14 |\n| DHGP | Transf chrF | 35.45 ± 0.02 | 4.15 ± 0.00 | 4.34 ± 0.04 |\n| MaGP | Transf BLEU | 1.018 ± 0.23 | 0.53 ± 0.86 | 2.06 ± 0.48 |\n| DHGP | Transf BLEU | 4.577 ± 0.00 | 1.08 ± 0.0 | 4.62 ± 0.16 |\n\n3. Architecture design experiments:  \nAlready provided in Section 5.3, Figure 9.\n\n$\\Rightarrow$ All results provide strong evidence that the hierarchies across all tasks examined in our work are indeed exchangeable without sacrificing performance.\n\n---\n> *In essence I am requesting additional focus on the core hypothesis, a categorization of what evidence would contribute toward or against it [...].*\n\n*Core Hypothesis*:  \nWe hypothesize that learning curve (LC) datasets exhibit hierarchical structure, specifically bi-level hierarchies.   \n$\\rightarrow$ We perform a range of experiments to provide insights and evidence towards this core hypothesis by analyzing the respective performance of MaGP vs DHGP. \n\nAdditional background info: As previously discussed, if we assume that clusters existed in the dataset, DHGP would be the preferred model, as it is capable of discovering and modeling such latent structure. On the other hand, if the dataset's underlying structure exhibits strong inter-task correlations and a hierarchical structure within each task, then MaGP are expected to outperform.\n\n*Additional hypothesis*: The exact ordering of bilevel hierarchies present in this data is exchangeable.  \n$\\rightarrow$ As stated above, our newly-included experimental results provide evidence that supports this hypothesis.\n\n*Final hypothesis*: The model which best captures the structure of learning curve datasets can be successfully used for scaling law extrapolation, due its good zero-shot performance.  \n$\\rightarrow$ This hypothesis is empirically supported in Subsection 5.1.\n\n---\n**[W2.1&2] - Experiments and different baselines:**  \nWe have reworked the experiment section to provide more explanations for our choice of baselines across all experiments. In brief: \n\n$\\rightarrow$ The necessity for different baselines arises primarily from the different nature of the datasets and the way learning curves are structured in each setting.\n\n- For the nanoGPT dataset, we have access to a sufficient number of learning curves and data points, making a Bayesian Neural Network (BNN) a valid and effective baseline. This choice is further motivated by prior work, such as Klein et al., where BNNs were successfully applied for learning curve extrapolation and zero-shot prediction. \n\n- In contrast, applying a BNN in the bilingual translation setting is more challenging. While we explored modeling the learning curves as a function of source and target language pairs, the BNN struggled to generalize and capture the upward trend in learning curves. Also note, we only have $25$ learning curves for training, and each source and target language is only $5$ times represented, which given the noisy nature of this dataset (see Figure 4 left) is not enough training data to perform well.  \n $\\rightarrow$ Given the assumption that source and target languages may share structural similarities, we instead opted for a simpler and more robust baseline: averaging available learning curves across source and target languages. This approach offers a language-agnostic approximation of performance trends.\n\n- The final experiment involves an extrapolation task, where we predict learning curves from a single source (or target) language into five different target (or source) languages. Unlike the bilingual translation setup, the training grid in this scenario is defined by three distinct model sizes (see Figure 4, right). The dataset for this experiment contains only 15 LCs, which is (again) insufficient for effectively training a BNN to generalize. Hence, a regression-based baseline, such as polynomial or power-law fitting, is a more suitable and reliable choice. \n\n--- \n**[W2.3] - Monte Carlo:**  \nIn Section 5.1 and Table 2, we present results illustrating how closely the scaling laws predicted via our Monte Carlo-based approach approximate the original scaling laws obtained through conventional methods -- namely, by fitting the $L(C)$ scaling function to the full dataset. For our evaluation, we use the AbC measure.   \n$\\rightarrow$ The key advantage of our approach lies in significantly reduced computational cost, while still achieving close alignment with the original scaling law. (Note that our test sets contain the most expensive learning curves.)  \n$\\rightarrow$ This outperforms simple fitting to training data as it provides more support to fit the scaling law in the compute region of interest.\n\n---\n**[W3-4] - Textual quality:**  \nWe have rectified all formatting and textual issues -- thank you for bringing these to our attention!\n\n--- \n**[Q1]**  \nAddressed in [W1&2]\n\n---\n**[Q2] - Compute on x-axis:**  \nThank you for this great suggestion! We have added the curves -- and this indeed provides very interesting insights!  \nIn brief: E.g. 17 queries for 'Smallest First' are roughly equivalent to only 3 for 'Largest', 7 'Active' & 10 Random;  \n$\\rightarrow$ This changes the ranking based on AbC metric to prefer the 'Smallest First', followed by 'Active' and 'Random', while 'Largest' stays least competitive.  \n\n---\n**Limitations:**  \nWe acknowledge a current limitation of MaGP regarding interpolation across the architectural grid. Specifically, our dataset is structured on a fixed grid defined by the number of layers and embedding parameters. As a result, the model in its current form cannot interpolate to unseen values of embedding parameters. Interpolation across unseen numbers of layers is technically feasible, but it would yield an averaged estimate between the nearest known configurations, which may not reflect true model behavior. We see this as a promising direction for future work and plan to investigate more flexible representations to address this limitation (e.g. w/ meta parameters).\n\n---\n---\nWe thank you again for your detailed feedback, and acknowledge that framing and textual quality *did* require some polishing. We'd like to highlight that a large part of your concerns were related to these issues -- all of which have been taken care of in the revised version of our manuscript thanks to your input!  \n\nWe hope we have addressed your questions; if any remain, please let us know."}, {"review_id": "KTn5MiLWKa", "reviewer": "Reviewer_FUMJ", "summary": "This paper presents a framework for predicting learning and training curves of NLP models that leverages latent variable multi-output Gaussian process models. The key idea behind the approach is to model correlations between tasks and hierarchies to facilitate predictions for new configurations of models (ie with different numbers of parameters or different languages). Experiments validate the framework in three settings: scaling law prediction and predicting zero-shot performance/few-shot performance of translation models.", "questions": "- Line 232: Why do authors subsample to 11 datapoints? How much more computational demand does it require to use the full dataset?\n- Why is there no direct comparison to prior scaling law estimation methods (e.g., those used by Kaplan et al., Hoffmann et al., Henighan et al.)?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The idea of using active learning for model prediction is creative, and results show that this method performs comparably to using the largest first strategy while reducing computational costs.\n- Experiments show the broad applicability of the proposed method for developing scaling laws, selecting model architectures, and predicting performance for bilingual translation models.\n\nWeaknesses:\n- There are no comparisons to more recent scaling law estimation methods, despite them being cited in the introduction.\n- The experiments are with limited models and datasets, and it is therefore unclear how robust the results are. For example, for the scaling laws experiments, the paper subsamples from the NanoGPT dataset and only evaluates language models in this previous data, which were not trained for many steps (and not larger model families). \n- The datasets used in the paper are never properly introduced; authors should describe upfront what the datasets they use are and where they come from. \n\nMinor Notes:\n- Line 263: Figure has \"??\"\n- Line 372: \"proof that\" --> \"prove that\"", "was_revised": true, "final_justification": "My main concerns have been addressed with the authors' promise to more thoroughly introduce datasets, along with their explanations of how the experiments in this work compare to prior works. In particular, they have justified their choices of model sizes/datasets and clarified that they compare to prior methods.", "rebuttal": "We thank you for your feedback, and address your points in the following.\n\n**[W1] - Comparison to recent scaling law estimation methods:**  \nRecent scaling law experiments typically predict the scaling law displayed in 'blue' in Figure 2 by utilizing the full available dataset (i.e. expensive training of all models), without employing zero-shot prediction to reduce computational complexity/cost.   \n$\\rightarrow$ In this sense, these experiments are effectively represented by our baseline 'ground-truth' scaling law, to which we compare our results.\n\n---\n**[W2] - Limited Models and Datasets:**  \nThank you for raising this concern. Due to computational constraints, we focused our setup on training models for one epoch using the FineWeb-Edu dataset. Our model sizes range from 51M to 1.5B parameters, aligning with the scale used in Kaplan et al., who explored models from 768M to 1.5B parameters. We therefore consider this range representative for our investigation.  \n\nTo support our hypothesis that hierarchical structures exist and can be exploited in learning curve datasets, we provide extensive *experiments across multiple domains*, including *bilingual* and *multilingual translation tasks*. To the best of our knowledge, our work is the first to investigate bi-level hierarchies for zero-shot learning curve prediction and extrapolation. \n\n---\n**[W3] - Introduction of Datasets:**  \nWhile the datasets we are using are introduced in Section 4, we agree that the detailed composition and origin is somewhat unclear;  \n$\\rightarrow$ We have added an extended Section describing their creation as well as inherent properties to the appendix of the paper (including references in the main text). Note that all these datasets were created by us.\n\n--- \n**Re: Minor Notes** -- We thank you for bringing these to our attention; textual and grammatical errors have been corrected.\n\n--- \n**[Q1] - Subsampling & computational demand:**  \n> Line 232: Why do authors subsample to 11 datapoints? How much more computational demand does it require to use the full dataset?\n\nDue to computational constraints, we were limited to using 11 datapoints across 36 learning curves, which was the maximum we could accommodate on our available *CPU* resources (RAM-limited). \n\nRegarding model complexity:\n- MaGP is implemented using the inducing points method. \n- MaGP is originally referred to as HMOGP-LV (Ma et. al., 2023). HMOGP-LV  is derived from LVMOGP (Dai et al., 2017).   \n\nThe computational complexity is \n$$\n\\mathcal{O}\\left(\\max(N R, M_H) \\cdot \\max(D, M_X) \\cdot \\max(M_H, M_X)\\right),\n$$\nwith $N$ being the available data points per learning curve, $R$ being the number of learning curves per task ($d$), $D$ being the number of tasks $t$, $M_X=M_r\\times R$ for $M_r$ being the number of inducing input points in the r-th learning curve of a task and $M_H$ being the number of inducing output points. \n\n$\\Rightarrow$ Importantly: While additional compute would certainly enable larger-scale experiments, our current setup demonstrates that *even with a limited number of datapoints and resources* -- corresponding to reduced evaluation frequency and faster training -- the MaGP model *performs effectively*!\n\n---\n**[Q2] - Comparison to prior work:**   \n> *Why is there no direct comparison to prior scaling law estimation methods (e.g., those used by Kaplan et al., Hoffmann et al., Henighan et al.)?*\n\nAs stated above in [W1], recent scaling law experiments typically predict the scaling law displayed in 'blue' in Figure 2 by utilizing the full available dataset (i.e. expensive training of all models), without employing zero-shot prediction to reduce computational complexity/cost.   \n$\\rightarrow$ In this sense, these experiments are effectively represented by our baseline 'ground-truth' scaling law, to which we compare our results.\n\n---\n---\nWe hope our answers addressed all your questions.  \nIf you have any further queries, please do not hesitate to reach out."}, {"review_id": "FldM0QZDPu", "reviewer": "Reviewer_bTJp", "summary": "The paper frames learning-curve and training-curve extrapolation in NLP as multi-task, two-level hierarchical regression problem and instantiates it with the latent-variable multi-output Gaussian process of Ma et al. (2023) (“MaGP”). Main claims are as follows :\n\n1. Zero-shot prediction of unseen learning curves is feasible by leveraging task-level correlations and hierarchical priors .\n2. The GP posterior can be Monte-Carlo–sampled to yield probabilistic scaling laws, giving uncertainty bands on fitted γ-laws .\n3. By employing an active learning approach, the uncertainty of the predictions can be significantly reduced   \n4. The approach generalises beyond scaling-law data (nanoGPT) to bilingual and multilingual translation curves\n\nThe approach outperforms DHGP, two BNN variants, and a naïve regression baseline across MSE/MNLPD metrics. The empirical evidence relies on three small datasets (≤ 30 curves per domain) with only few points per curve, yet MaGP consistently beats baselines.", "questions": "- GP cubic complexity is O(N³); the paper omits runtime or inducing-point strategies. Please report wall-clock time and memory for nanoGPT-large fits.\n- Would cost-aware fits (e.g., loss/FLOP) further improve results?\n- Authors should try to incorporate the irreducible loss in these predictions\n- For BLEU/ChrF tasks only MSE/MNLPD are shown; confidence intervals overlap heavily. Maybe break it down further and improve the plotting?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- Casting curve extrapolation as hierarchical multitask GP clearly articulated.\n- Folding model uncertainty into γ-law fits via MC integration is principled and valuable for budgeting decisions in LLM training.\n- Active-learning cost analysis a realistic compute/accuracy trade-off, absent from most scaling-law studies .  \n- Comprehensive baselines, Includes DHGP, two BNN variants, and task-averaging heuristics, isolating the gain from hierarchy + latent correlation.\n\n### Weaknesses\n\n- All experiments use very few curves; scalability to realistic hyper-parameter spaces (≥ 10²−10³ curves) is unclear. LLM training would often depend on multiple hyperparameters and this is a combinatorially complex space\n- Does not discuss vertical offset and irreducible loss, how would that impact these predictions?\n- Variance-only ignores compute-cost heterogeneity across model sizes. \n- Query budget is ~25; realistic large-model experiments may allow far fewer queries.\n- Hierarchical grouping is hand-crafted. How sensitive is the approach to mis-specified hierarchies?", "was_revised": true, "final_justification": "After authors response some of the concerns have been addressed. However I am still not strongly recommending an accept since there remains concerns if the framework is scalable and applicable to actual LLM model scaling ladder approaches.", "rebuttal": "We thank you for your helpful suggestions, and address your points in the following.\n\n**[W1] - Scaling to many hyperparameters:**  \nAt present, our model considers only the number of embedding parameters and the number of layers as structural inputs/hyperparameters.  \nDue to computational constraints, we were limited to using 11 datapoints across 36 learning curves, which was the maximum we could accommodate on our available CPU resources (RAM-limited).  (see [Q1] below for model complexity) \n\n$\\Rightarrow$ Importantly: While additional compute would certainly enable larger-scale experiments, our current setup demonstrates that even with a limited number of datapoints and resources -- corresponding to reduced evaluation frequency and faster training -- the MaGP model performs effectively! \n\n---\n**[W2] - Vertical offset and irreducible loss:**  \nVertical offset is not explicitly modeled, as all learning curves considered have the same number of layers vertically and an identical number of datapoints. Additionally, we do not incorporate the irreducible loss term in our scaling law analysis, following the frameworks established by Kaplan *et al.* and Hoffmann *et al.*. While the irreducible loss is addressed in Hennighan’s work, incorporating it represents a natural and compelling extension for future research. \n\n$\\rightarrow$ Please feel free to reach out if you require further clarification or additional details.\n\n---\n**[W3] - Variance-based Active Learning:**  \nThank you for raising this point. You are correct that the variance-based active learning strategy explored in our work focuses on reducing the uncertainty, independent of associated compute costs. In the revised version of the paper, we therefore additionally provide AbC over cumulative compute cost to reflect the strategies' performances when computational resources are constrained.  \n\nIn brief: E.g. 17 queries for 'Smallest First' are roughly equivalent to only 3 for 'Largest', 7 'Active' \\& 10 Random;  \n$\\rightarrow$ This changes the ranking based on AbC metric to prefer the 'Smallest First', followed by 'Active' and 'Random', while 'Largest' stays least competitive. \n\nNote that there exist many different ways how the cost-function in the active learning process could be structured, but this is outside the scope of this paper. We see the design of cost-terms that consider trade-offs like compute-vs-uncertainty-reduction as an interesting direction of future work. \n\n---\n**[W4] - Query budget:**  \nWe thank you for raising this concern. In total, we have 25 learning curves available for querying -- however, this is the 'upper limit' that is considered in our experiments.  \nNote that Figure 7 illustrates the performance when using fewer queries, demonstrating that after querying for example just 14 curves, the AbC value already falls below 0.25.  \n\n---\n**[W5] - Hierarchical Grouping:**  \nWe are slightly unsure what you mean with 'mis-specified' in the context of model hyperparameters.  \nHowever, we have further validated our hypothesis that hierarchies are indeed exchangeable (and therefore indicating robustness against potential mis-specification).  \n$\\Rightarrow$ For the new experimental results with exchanged (swapped) hierarchy, please see the tables provided in the response to reviewer *QuFj - [W1]*.\n\n---\n**[Q1] - Wallclock times and memory:**  \n- MaGP and DHGP were trained and evaluated on an Intel(R) Core(TM) i7-8565U CPU. \n- MaGP follows the same inducing point strategy as suggested in the original work in Ma et al., by inducing every second point. (Using every point increased the runtime but did not improve on the negative log likelihood optimization criterion.) \n- The wallclock time for MaGP is between 17 min and 31 min, and DHGP around 4 min (including logging, file-saving, etc.); Note that DHGP uses optimized libraries, while MaGP in its current form is not yet optimized.\n\n---\n**[Q2] - Cost-aware fits:**  \nWe are slightly unsure if we correctly understand your question. If your question refers to an active learning approach that jointly considers e.g. variance-reduction and compute-cost, this is discussed in [W3] -- and we consider this a natural and promising extension of our work for future consideration.\n\n$\\rightarrow$ Please feel free to reach out if you require further clarification or additional details, or if your question has not yet been fully addressed.\n\n--- \n**[Q3] - Irreducible loss:**  \nAs the analyses in our paper are based on Kaplan et al.'s and Hoffmann et al.'s works, we have not incorporated the irreducible loss term in our scaling law analysis. While the irreducible loss is addressed in Hennighan’s work, we see this as a natural and compelling extension for future research -- but outside the scope of this current work.\n\n--- \n**[Q4] - Improved plot quality - BLUE/ChrF:**  \nThank you for pointing this out. We included appropriate legends and present separate performance figures for each metric in the revised version of the paper.\n\n---\n---\nWe hope our answers have clarified all your questions.  \nIf you have any further queries, please let us know and we are happy to answer them."}, {"review_id": "cybfqKBgeh", "reviewer": "Reviewer_Mugk", "summary": "The paper proposes using latent variable multi-output Gaussian processes (specifically the MaGP model) to predict training and learning curves in a zero-shot manner. The key insight is modeling these prediction tasks as hierarchical multitask learning problems, where correlations across tasks and within hierarchies can be exploited. They demonstrate this on three NLP datasets: nanoGPT training curves, bilingual translation learning curves, and multilingual translation performance. An active learning component helps reduce uncertainty in predictions.", "questions": "NA", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The motivation is solid - developing scaling laws is indeed computationally expensive, and any principled method to reduce these costs would be valuable.  I think the probabilistic nature of the approach is a clear advantage over deterministic scaling law methods. Getting uncertainty estimates on scaling law parameters could be quite useful for decision-making about model development. The two-layer hierarchy matches intuitive structure in all three tasks and lets the GP share signal efficiently.\n- The experimental evaluation is reasonably comprehensive given the scope. Testing on three different types of NLP tasks (language modeling, bilingual, and multilingual translation) shows some generality. The comparison against multiple baselines, including BNNs with different basis functions, helps establish the method's effectiveness. I also liked the active learning analysis showing how uncertainty can be reduced through strategic querying.\n- Table 2 and Figure 7 don’t just report accuracy; they tag each train-test split or query strategy with the PetaFLOPs burned to obtain it. The active-learning policy slices AbC by roughly one-half relative to a Largest-First schedule while staying a good 2× cheaper than the Random schedule .\n\n\n\nWeaknesses\n\n- I'm curious about the computational costs of the MaGP method itself. While the goal is to reduce the cost of obtaining scaling laws, how expensive is the GP inference, especially with the hierarchical structure and active learning?\n- The active learning results are promising, I'd also like to see more analysis of which queries are actually selected. Maybe the patterns might provide insights into what information is most valuable for scaling law prediction.\n- The writing is quite dense in places, particularly sections 3 and 4. I found myself re-reading several passages to understand the model setup. Given that this is targeting a NeurIPS audience who might not all be familiar with hierarchical GPs, clearer exposition would help.", "was_revised": true, "final_justification": "The authors' response has addressed my remaining concerns. I will keep the score.", "rebuttal": "Thank your for your supportive feedback! We address your questions individually in the following:\n\n**[W1] - Model complexity and runtime**:  \n\n*Algorithmic Complexity:*  \nMaGP is implemented using the inducing points method. MaGP is originally referred to as HMOGP-LV (Ma et. al., 2023). HMOGP-LV  is derived from LVMOGP (Dai et al., 2017). The computational complexity is \n$$\n\\mathcal{O}\\left(\\max(NR, M_H) \\cdot \\max(D, M_X) \\cdot \\max(M_H, M_X)\\right),\n$$\nwith $N$ being the available data points per learning curve, $R$ being the number of learning curves per task, $D$ being the number of tasks, $M_X=M_r\\times R$ for $M_r$ being the number of inducing input points in the r-th learning curve of a task and $M_H$ being the number of inducing output points.\n\n*Runtimes:*  \nMaGP was trained and evaluated on an Intel(R) Core(TM) i7-8565U CPU. \n\n$\\rightarrow$ nanoGPT-Large experiments: The wallclock time for MaGP in this scenario (w/o active learning) is between 17 min and 31 min, which includes logging and saving files. It is worth noting that MaGP is not (yet) optimized in terms of speed.  \n\n$\\rightarrow$ The 'active learning' component only adds negligible overhead (in comparison to the other query strategies), as the only additional operation is the computation of the variance across the already-predicted curves (which uses an optimized library).  \n\n---\n**[W2] - Patterns of Query Selection in Active Learning:**  \nWe thank you for raising this question. This was indeed of great interest to us as well.   \n$\\rightarrow$ The paper has already a corresponding figure included in Appendix F, and we will ensure that a proper reference to this is added in the main paper. (Or we might move this to the main paper to provide further insights if space permits.)\n\n---\n**[W3] - Partially dense writing, esp. around hier. GPs:**   \nThank you for drawing our attention to this! We have already expanded the relevant sections and now provide more detailed explanations to improve clarity and completeness, as well as some underlying intuitions to cater for a wider audience. \n\n---\n--- \n\nWe would like to thank you again for the support of our work, and hope our answers have clarified your questions.  \nIf any further queries remain, please let us know and we will do our best to promptly address them."}]

Author rebuttal:

Meta-review: The paper was reviewed by four experts who provided detailed reviews including strengths and weaknesses of the paper. The authors engaged with the reviewers in providing clear responses to the reviewers' questions. All four reviewers were satisfied with the response and raised their scores (or maintained accept recommendations). In the end the reviewers are unanimous in recommending accepting the paper and the Area Chair sees no reason to override their recommendation.
Decision: Accept (poster)