Scaling Laws for Predicting Downstream Performance in LLMs
Authors: Yangyi Chen
Binxuan Huang
Yifan Gao
Zhengyang Wang
Jingfeng Yang
Heng Ji
Venue: Accepted by TMLR ()

Abstract: Precise estimation of downstream performance in large language models (LLMs) prior to training is essential for guiding their development process. Scaling laws analysis utilizes the statistics of a series of significantly smaller sampling language models (LMs) to predict the performance of the target LLM. For downstream performance prediction, the critical challenge lies in the emergent abilities in LLMs that occur beyond task-specific computational thresholds. In this work, we focus on the pre-training loss as a more computation-efficient metric for performance estimation. Our two-stage approach FLP consists of first estimating a function that maps computational resources (e.g., FLOPs) to the pre-training Loss using a series of sampling models, followed by mapping the pre-training loss to downstream task Performance after the critical "emergent phase". In our experiments, this FLP solution accurately predicts the performance of LLMs with 7B and 13B parameters using a series of sampling LMs up to 3B, achieving error margins of 5% and 10%, respectively, and significantly outperforming the FLOPs-to-Performance approach. Further, we present FLP-M, a fundamental approach for performance prediction that addresses the practical need to integrate datasets from multiple sources during pre-training, specifically blending general corpus with code data to accurately represent the common necessity. FLP-M extends the power law analytical function to predict domain-specific pre-training loss based on FLOPs across data sources, and employs a two-layer neural network to model the non-linear relationship between multiple domain-specific loss and downstream performance. By utilizing a 3B LLM trained on a specific ratio and a series of smaller sampling LMs, FLP-M can effectively forecast the performance of 3B and 7B LLMs across various data mixtures for most benchmarks within 10% error margins.

Published in Transactions on Machine Learning Research (04/2025)
Scaling Laws for Predicting Downstream Performance in LLMs
Yangyi Chen Binxuan Huang', Yifan Gao?, Zhengyang Wang', Jingfeng Yang", Heng Ji University of Illinois Urbana- Urbana-Champatay Champaign', Amazon yangyic3@illinois edu, Sabboaamzon.ccom com
Reviewed OpenReview: httppp///ppaaaae.eee..eee//e/eep
Abstract
Precise estimation of downstream performance in large language models (LLMs) prior to Iraining is essential for guiding their development process. Scaling laws analysis utilizes the statistics of series of significantly smaller sampling language models (LMs) to predict the performance of the target LLM. For downstream performance prediction, the critical challenge lies in the emergent abilities in LLMs that occur beyond task-specific computational thresholds. Tn this work, we focus on the pre-training loss as more computation-efficeeu metric for performance estimation. Our two-stage approach FLP consists of first estimating a function that maps computational resources (e.g., FLOPs) to the pro- pre-training Loss using series of fully-converged sampling models, followed by mapping the pre-training loss to downstream task Performance using the intermediate models with emerged performance. In our experiments, this FLP solution accurately predicts the performance of LLMs with 7B and 13B parameters using series of sampling LMs up to 3B, achieving error margins of 5% approach.
and 10%, respectively, and significantly outperforming the M.OPs-to-Performant Further, we present FLP-M, a fundamental approach for performance prediction that addresses the practical need to integrale datasets from multiple sources during pre-training, specifically blending general corpus with code data accurately represent the common necessity. FLP-M extends the power law analytical function to predict domain-specific pre-training loss based on FLOPs across data sources, and employs two-layer neural network to model the non-linear relationship between multiple domain-specific loss and downstream performance. By utilizing a 3B LLM trained on a specific ratio and series of smaller sampling LMs, FLP-M can effectively forecast the performance 3B and 7B LLMs across various data mixtures for most benchmarks within 10% error margins.
1 Introduction
Large language models (LLMs) form the basis for numerous real-world applications (Brown et al., 2020; Jiang et al., 2023; Xu et al., 2024; Hadi et al., 2023) and scaling laws analysis serves as the foundation for L.MSs development (Kaplan et al., 2020; Bahri et al., 2024). The key idea of scaling laws involves training a sequence of language models (LMs) to gather data (e.g., expended compute and corresponding model performance). This data then used to build predictive model that estimates the performance of a substantially larger target LLM (Su al., 2024; Hoffmann et al., 2022).
Previous efforts focus predicting the target LLM's pre-training loss and establish power-law relation between the computational resource expended (e.g., floating-point point operations per second (FLOPs)) and the final loss achieved (Kaplan et al., 2020; Mucnnighoff et al., 2024; Henighan et al., 2020). Further, we aim to predict the downstream performance in pre-trained LLMs (i.e., zero- or few-shot evaluation) to more accurately reflect the primary concerns regarding their capabilities. The critical challenge the emergent abilitics in LLMs, which states that LLMs only exceod random performance when the FLOPs expended during training surpass task-specific thresholds (Wei al., 2022a).

---

Published in Transactions on Machine Learning Research (04/2025)
Supposing task threshold of Fe, typical methods require training N LMs, expending total FLOPs F Fi=E==1 FLOPs; > N x Fe, to obtain N effective data points, thereby necessitating significant computational resources. Fig. demonstrates that the sampling LMs require more than 5 x 1020
Performance of Sampling LMs
101
Performance
x
FLOPs to perform better than random on most benchmarks, with only three data points available to fit the predictive curve across these benchmarks. Hu et al. (2023) address this challenge by significantly increasing the sampling times to compute the PassUntil of a task, basically increasing the "metric resolution" enable the abilitics to emerge earlier (i.e., reducing F.). However, this approach faces challenges in translating the PassUntil back to the original task metric of concerns and requires huge amounts of FLOPs spent on sampling.
10a
RACE
TriviaDA
10-1
DDII
ARC
Hellasway
Human
10
10
1020
10at
10
FLOPS
Figure 1: The performance of sampling LMs with increasing compute. x represents non-emerged data points, and indicates emerged data points that sur- pass a randomness threshold of 5.
In this work, we target the actual task performance prediction based on two intuitions: (1) Predicting the target pre-training loss is easier and achievable since there no "emergent phase" in the pre-training loss, as extensively verified in Kaplan et al. (2020); Hoffmann et al. (2022); (2) There is an observed correlation between the pre-training loss and the downstrcam task performance after the "emergent point" (f.e., the pro- training loss goes below critical threshold) (Du et al., 2024; Huang el al., 2024). Specifically, LMs that achieve comparable pre-training loss demonstrate consistent performance levels on downstream tasks, regardless of variations in syper-parameter configurations (e.g., model size, the number of training tokens, and learning rate schedules). Thus, different from estimating the FLOPs-to-Loss function that requires training LMs to convergence, we can collect. (pre-training loss, performance) data points at intermediate checkpoints that have function, thereby enhancing sample efficiency.
emerged performance estimate the Lues-to-Perffrminn
Formally, our approach, named FLP, consists of two sequential stages: (1) FLOPs - Loss: Predict the target pre-training loss based on the expended FLOPPs Following previous work, we train series of sampling LMs within the same model family and collect data points from their final fully-converged checkpoints to develop a power-law predictive model. For this stage, the expended FLOPs are not required to reach above the task- specific emergent threshold. (2) Loss + Performance: Predict the rownst performance based on the pre- training loss. We collect data points from intermediate checkpoints of various sampling LMs that exhibit above- random performance, and develop a regression model for prediction. In our experiments with sampling LMs up to 3B, this FLP solution prediets the performance of 7B and 13B LLMs across various benchmarks with error margins of 5% and 10% respectively, significantly outperforming direct PT.OPs-to-Performan
predictions.
Motivated by these findings, we present FLP-M, a fundamental solution for performance prediction that addresses the growing demand for integrating diverse datasets during LLMs pre-training, focusing on integrating the general corpus with code data in this work. FLP-M targets fine-grad gen-igr dopeif-specific pre-training loss to capture the performance changes. Specificallyy, we extend the power law analytical function to predict the domain-specific loss based on FLOOPs across multiple data sources. Then we employ two-layer neural network to model the non-linear relationship between multiple domain-specifee loss and the downstream performance. Through evaluation, we demonstrate that FLP-M effectively predicts the performance of 3B and 7B LLMs trained on various data mixtures (within 10% error margins for most benchmarks). This is achieved by utilizing a 3B LLM trained on a specific data mixing ratio along with a series of smaller sampling LMs. Furthermore, our extensive experiments demonstrate that FLP-M effectively optimizes data mixture compositions to improve downstream task performance. Through comprehensive ablation studies, we validate the design choices in our analytical functions.
Overall, this work presents three major contributions to LLMs development. First, we introduce systematic and principled framework for forecasting downstream performance in LLMs. Second, we innovatively employ pre- training loss as an intermediate variable, enabling both the handling of training dynamics and quantitative
2

---

Published in Transactions on Machine Learning Research (04/2025)
monitoring of downstream performance. Third, present method extension to manage data mixing scenarios, which is crucial for LLMs pre-training.
2 FLP: Downstream Performance Prediction
We introduce FLP, a two-stage approach to predicting downstream performance in LLMs based on two estab- lished findings: (1) Predicting the target pre-training loss and establishing the power-law relation is feasible as it does not involve an emergent phase (Kaplan et al., 2020; Hoffmann et al., 2022). (2) When pre-training loss goes below a task-specific threshold, there is an observed correlation between pre-training loss and downstream task performance (Du et al., 2024; Huang et al., 2024) regardless of the arper-paramcter configurations.
2.1 FLOPs + Loss
We follow the previous practice to use the analytical power law function to characterize the relation between expended FLOPs C' and the pre-training loss L:
ON
L(C)
(1)
CN
where ON and an are constant terms to be estimated. In FLP, we train a series of LMs within the same model family in the same pre-training distribution, progressively increasing model size and training tokens to achieve even sampling. Then we measure their pre-training loss in our curated validation dataset obtain N pairs of (C,, Ta) to estimate the constants in Eq. 1. Note that the data points are only collected from the final fully-convergee sampling LMs, which ensures the one-to-one mapping from FLOPs to pre-training loss (Kaplan et al., 2020).
2.2 Loss - Performance
Based on our empirical observation of the scatter plots showing (pre-training loss, performance) data points (see SA) and the strong statistical evidence that the average coefficient of determination R² is 93% across all benchmarks for the linear fitting curves, we select the analytical linear function' to characterize the relation between the pro-training loss L on general validation data and the task performance P:
P(L) = wip + wi L,
(2)
where wo and wi are constant terms to be estimated. In FLP, we fetch the intermediate checkpoints of each sampling LM, and measure its task performance and pre-training loss. If the performance P. of LM, exceeds the random performance, we can obtain one effective data point (L,, P.) to estimate the constants in Eq. 2, where Lj is the pre-training loss of LM,.
2.3 The Necessity of Two-Stage Modeling
A natural question is why cannot simply combine Eq. 1 and Eq. 2 to directly model the relationship between FLOPs and performance. The key issue is that the relationship between FLOPs and task performance is not a simple direct mapping. For a given FLOPs budget, the resultant task performance can vary significantly depending on the training dynamics (e.g., learning rate schedule, model size, and the number of training tokens). This many-to-one relationship between FLOPPs and performance makes fitting a direct analytical function between them unreliable. Our two-stage approach addresses this by using pre-training loss as intermediate variable. Critically, we reliably fit the Coss-to-Performnn mapping using (pre-training loss, performance) data points, since models with similar pre-training loss tend to demonstrate comparable task performance, regardless of their specific training trajectories. However, we cannot fit the direct PLOPP-tooperffrmmncc mapping using (FLOPs, performance) data points due to the variation in training dynamics. Overall, the two-stage formulation allows us to more accurately model
In preliminary investigations, we also explore nonlinear alternatives, including the sigmold function, though these approaches demonstrate suboptimal predictive performance.
3

---

Published in Transactions on Machine Learning Research (04/2025)
Table 1: The configurations of the sampling and target LMs with various sizes. HD denotes the hidden dimension, BS denotes the batch size, and LR denotes the learning rate.
Model Size
Layyer
# Head
FFN
# Tokens Non-embedding FLOPs
BS
43M
384
3
1032
8.021.606.400
+1
448
0.0052
64M
512
4
1376
11,714.691.072
1.18417E 18
544
0.0042
89M
640
5
1720
16.184.770.560
3.03607E+1
576
0.0038
0.12B
768
6
2064
21,799.895.00 895.040
6.81931E 18
640
0.0040
0.15B
896
2408
28.8446.225.700 760
1.39581E
672
0.0042
0,2B
1024
8
2752
37.213.962.244
2.63435E 119
736
0.0036
0.25B
1152
9
3096
47.563.407.4 360
4.71817E+19
768
0.0034
0.32B
10
1280
10
3440
39 666,674,600100
8.0157 119
800
0.0028
0.5B
12
1536
12
4128
30,502.5945560
2.5963E++20
960
0.0023
0.72B
14
1792
14
4816
132.026,204,
4.70331E 20
1024
0.0019
1B
16
2048
16
5504
185.535,037,44
9.75926E
1152
0.0016
3B
24
3072
24
8256
556, 736,793.856 000
9.63212E 21
1536
0.0004
7B
32
4096
32
11008
1.258 291 200,0000
5.09208E +-22
2048
0.0003
13B
40
5120
40
13824
1,258 201 200,000
9 9,89592E 22
2048
0.0003
and optimize the relationship between computational resources and model performance. We implement a baseline in $3 for comparison to demonstrate that directly composing Eq. 1 and Eq. 2 cannot fully capture the complex training dynamics accurately.
3 Validation of FLP Framework
3.1 Sampling and Target LMs
We train a series of 12 sampling LMs up 3B parameters to predict the performance of target LLMs with 7B and 13B parameters. The configurations of LMs are shown in Tab. 1. We adopt the fixed data-model ratio scaling strategy (Kaplan et al., 2020; Hoffmann el al., 2022) that maintains a fixed ratio between training tokens and model size while varying compute (FLOPs). In this scaling strategy, for any given compute budget, there exists a pre-determined allocation between training tokens and model size. We first determine the number of training tokens required for the 7B LLM (approximately 180 times the model size), considering practical needs and inference-time costs. In real-world applications, prioritizing inference efficiency often involves training smaller LMs with a higher toaarntt- ratio beyond the optimal factor of 20x (Hoffmann et al., 2022). Our preliminary experiments indicate that scaling laws remain applicable even in this over-training regime (within 2.8% error margins). We then proportionally scale down this number to determine the required training tokens for the sampling LMs.
3.2 Data: Pre- Trainiinn Validation, Evaluation
Pre- Pre-Training We use the RedPajama v1 (Computer, 2023), which consists of 1.2T tokens in total, and the data is sourced from Arxiv, C4, Common Crawl, GitHub, Stack Exchange, and Wikipedia.
Validation We curate validation dataset to measure the final pre-training loss, which includes 5 distinct domains: math, code, scientific paper, Wikipedia, and general language corpus. Specifically, we utilize subsets from GitHub, ArXiv, Wikipedia, and the English portion of C4, all from the RedPajama validation sets, along with Proof Pile (Touvron al., 2023) for the math domain.
Evaluation We select the following tasks for eval- uation, covering fundamental capabilities in LLMs (e.g., knowledge, reasoning, coding): RACE (Lai et al., 2017), TriviaQA (Joshi et al., 2017), BygBench-Challeneg (BBH) (Suzgun et al., 2022), ARC-Challenge (ARC) (Clark et al., 2018), Hel- laswag (Zellers et al., 2019), and HumanEval (Chen et al., 2021). The evaluation settings for these bench- marks listed in Tab. 2. We adopt In-evaluation- (Gao et al., 2023b) for unified evaluation.
Table 2: The evaluation settings of the benchmarks.
Dastaet
Fiodustion Type
Fesiuation Method
Mitric
Randov Porfernaees
ABC
Siultinke Chokee
10-hho
Accunacy
25
BRH
Generation
GoT-bbbut
FasetMatc
D
Hellaenog
Multiple Chokee
shos
Aceuacy
25
HomaDDal
Generation
O-shot
RACE
Multiale Choiee
Oslutt
Accunacy
Generation
sbott
EasetMotch
4

---

Published in Transactions on Machine Learning Research (04/2025)
ARC
BBH
Hellaswag
10P
Par MOsSEae
Performance
101-
30-1
EPhRt Kampline LLMG
Ouro
Sonceiis 1.6
Scrcliis
Iarpet 11.4
Targe
TriviaQA
Nefurmance
Purformance
FLP' Sampline
227 Dane
Sumpping
Sucplag
1322
1019
10a6
10
101
1077
1027
1013
100
10
FLOPS
PLOPS
PLOPS
Figure 2: The downstream performance prediction using FP and FLP fit curves. FLP can better predict the downstream performance of target 7B and 13B LLMs across all evaluation benchmarks.
3.3 Experimental Setting
Baseline We consider directly using the expended FLOPs C to predict the downstream performance P, and experiment with the following analytical form for comparison:
P(C) = (
C
(3)
where Om and aM are constant terms to be estimated. We denote this approach as FP, For fair comparison, we collect data points using identical sampling LMs across both FLP and FP.
Implementation of FLP To the FL.OP>too-Toss curve, we utilize the final checkpoints from each sampling LM. In addition, during LMs training, checkpoint saved every 1/30th increment of the total training progress. We monitor and record the pre-training loss on the training dataset, rounded to decimal places. Only those checkpoints demonstrating checkpoints, we evaluate the downstream performance and pre-training loss on the validation set. We then discard those that do not surpass the random benchmark performance by at least 5, and use the remaining data points to fit the Loss-to-Performance curve.
improvement in pre- pre-training loss are retained. For these selected
Evaluation Metrics In addition presenting the fitting curves for intuitive visualization. we quantify the prediction accuracy by measuring the relative prediction error:
Predictive Metric Actual Metriel
Relative Prediction Error
(4)
Actual Metric
3.4 Results
The downstream performance prediction results are visualized in Fig. Across all evaluation lasks, FLP fit curve can better predict the performance of target LLMs with 7B and 13B parameters using the sampling LMs up to 3B. In contrast, while FP more effectively fits the data points of sampling LMs, has difficulty accounting for the "emergent phase" characterized by rapid performance shifts, due to the scarcity of data points from this period. As solution, FLP utilizes pre-training loss more fine-grained indicator to
5

---

Published in Transactions on Machine Learning Research (04/2025)
Relative Prediction Error of 7B LLM
Relative Prediction Error of 13B LLM
1.2
Relative Prediction Error
Relative Prediction Error
PP
1.0
0.6
0.0
0.4
0.6
0.3
0.4
0.2
0.2
0.1
0.0
0.0
JAAZ
AMC
Halacvag
RACZ
TaiisOA
thmaniivil
Dataset
Dataset
Figure 3: The relative prediction error of 7B and 13B LLMs. FLP achieves a more accurate prediction with error margins of 5% and 10% across all benchmarks for two LLMs respectively.
monitor performance changes and effectively incorporates data from intermediate chockpoints, cnhancing sample efficiency. The evaluation results of relative prediction error are shown in Fig. 3. Unlike the suboptimal predictions of FP, FLP delivers precise forecasts, maintaining relative error margins of 5% and 10% across all benchmarks for 7B and 13B LLMs, respectively.
Compared to FP, FLP is less effective at fitting the data points of sampling LMs, especially in HumanEval and TriviaQA. The reason is that we do not align with the "non-emergent" phase of the Loss-to-Performance curve, where LMs exhibit random performance when pre-training loss is beyond the task-specific threshold. Thus, FLP predicts higher pre-training loss for LMs with fewer FLOPs, resulting in below-random performance. This issue is not within the scope of FLP, as is specifically designed to predict the performance of LLMs trained with significantly larger FLOPs in practice.
In addition, we discuss additional results in Appendix for the presentation purpose since adding these data points may distort the vertical axis scaling in Fig. 2. We compare FLP further with the analytical forms and approaches proposed in GPT-4 (Achiam et al., 2023) and Llama-3 (Dubey et al., 2024) technical reports. The results are shown in $C and $D respectively. We also evaluate the feasibility of employing FLP to predict the performance of 13B LLM on MMLU (Hendrycks al., 2020), using intermediate checkpoints from a 7B LLM ($B). Overall, the results demonstrate the general effectiveness and applicability of FLP.
4 FLP-M: Data Mixing for Downstream Performance Prediction
Motivated by the encouraging results of FLP ($3), propose FLP-M, fundamental approach to meet the practical needs of integrating data from various sources (Groeneveld et al., 2024; Penedo et al., 2024). In our work, we focus on mixing general corpus with code data, considering two distinct yet overlapping data sources. This intersection offers a more realistic perspective than treating them as distinct domains (Ye et al., 2024), as real-world corpus often spans multiple domains, necessitating an analysis of the interdependence between data sources when formulating our analytical functions.
Compared to the straightforward implementation of FLP ($2), FLP-M operates fine-grained, domain-specific pre-training loss, due to the observation that the average loss on the entire validation set fails to effectively reflect performance variations in downstream tasks in the data mixing context ($6.2). This may be due to the fact that changes in pre-training data mixtures simultaneously impact multiple capabilities of the LMs. For instance, an increase in code data loss coupled with a decrease in general data loss may leave the average validation loss unchangod, yet result in LMs with distinct capabilities and downstream performance. Note that unlike the pre-training data mixture, the validation set is deliberately curated by domain, as creating smaller, domain-specific validation sets is manageable.

---

Published in Transactions on Machine Learning Research (04/2025)
4.1 FLOPs Domain Loss
Given the FLOPs C spent on the general corpus and C spent the code data, naturally extend the power law function to the following analytical form to predict the domain-specifie pre-training loss L on domain D:
C IC ()))0.0)) = - C + CC joc x (co jaes x CT
C
Cc )ae2
(5)
CG
where Cr, Ca. Cc. ac, ac, and ac, are constants to be estimated. In FLP-M, we first select a sequence of total compute {Ci) spent on pre-training. For each selected Ci we experiment with various ratios to mix two data sources, and decompose C; into C and CF. We measure the domain-specific pre-training loss LP on a domain-specific subset of validation data to obtain (Cf, CT LP)) data pairs. Then we can estimate the constants in Eq. We also experiment with other potential analytical forms in $6.2.
4.2 Domain Loss + Performance
Given the pre-training loss {T.P)=- on K domains, we train a two-layer neural network with a hidden layer size of 3 and the ReLU activation function (Agarap, 2018) to predict the downstream performance. The network is optimized using the regression loss with L2 regularization and the Adam optimizer (Diederik, 2014), employing a learning rate of 0.05 that linearly decays to 0 within 2,000 steps and weight decay of 0.01. In FLP-M, we adopt the same strategy in FLP to fetch the intermediate checkpoints and only retain the results that the LMs achieve above-random performance (see $2). Thus, for LM,, we can obtain a sequence of effective data points ([L")D_ Pt), where LP the pre-training loss on domain D and P. is the LM's performance. Then we can use these data points to train the neural network. We also explore other functions for fitting in $6.2.
Note that while neural networks are well-suited for the stage-2 Loss-to-Performant mapping due to abundant data points collected from intermediate checkpoints, they are less appropriate for stage-1 ELOPs-to-Loo mapping because it requires data points from fully-converged models, yielding limited data points from our configurations (Tab. 1).
5
Experiment for FLP-M
5.1 Sampling and Target LMs
We train a series of sampling LMs with sizes {0.12B, 0.2B, 0.32B, 0,5B, 0.72B, IB), and the corresponding training token numbers are shown in Tab. We train the LMs on the general and code data mixture with {0, 0.1, 0.2, 0.3, 0.4, 0.5] as the mixing ratios of code data reflect real-world usage. We also add one sampling LM of 3B size and 0.3 mixing ratio. For evaluation, we train 3B LLMs with the other mixing ratios and 7B LLM with 0.3 the mixing ratio due to the limited compute budget.
5.2 Data: Pre- Training, Validation, Evaluation
Pre-Training For general corpus, we use DCLM (Li et al., 2024b), curated high-quality pre-training corpus including heuristic cleaning, filtering, deduplication, and masel-based filtering. For code data, we use The Stack v2 (Lozhkov et al., 2024), which initially contains over 3B files in 600+ programming and markup languages, created as part of the BigCode project. We mix these two data sources to create the pre-training data mixture using the ratios specified in $5.1.
Validation We use the same validation data mixture specified $3.2 that includes 5 distinct domains.
Evaluation The evaluation benchmarks and settings are the same those in $3.2.
5.3 Experimental Setting
Baseline We implement FLP within this data mixing context as a baseline, which first predicts the average pre-training loss the validation set and uses this estimate downstream performance via linear regression.
7

---

Published in Transactions on Machine Learning Research (04/2025)
ARC
BBH
Hellaswag
Target Performnn refformance
XL
ILP
TriviaQA
Tazget Perfermaace
Darget Pertarmmnn
Turget
PLP Soretes
Sumpting USb
Predictive Performance
Predictive Performence
Prodictive Performancee
Figure 4: The downstream performance prediction using FLP and FLP-M fit curves. FLP-M can better predict the downstrcam performance of target LLMs across various data mixing ratios.
Relative Prodiction Error 3B-0. 1-Mising
Relative Prediction Error of 3B- 3B-0.2-Mixing
Relative Prodiction Error of 3B-0.4-Mixing
05
JUH
Relative Prodiction Ereor
Belative Prodtction
6.
0.3
0.5
0.1
0.0
-1C
0.0
3023
TWh00
leng
0.0
6904
604
TexeO4
Datawet
Relative Prediction Error of 38-0.5-Mixdng
Relative Prediction Error of 38-0-Mixing
Relative Prediction Error of 78-0.3-Mixing
0.00
Belative Preditction Exror
0.30
Relative Predlction
0.53
Retaatvve Drowiiee 1imm
0.17
0.01
To
Miluow-
795004
9:6 m0g
25/004
Figure 5: The relative prediction error of downstream performance prediction using FLP and FLP-M. FLP-M can better predict the performance of target. LLMs across various data mixing ratios.
Implementation of FLP-M We adopt the same implementation as in FLP (details in $3.3). The distinction is that we individually measure the pro-training loss on cach domain of the validation mixture.
5.4 Results
The downstream performance prediction results are visualized in Fig. 4. We update the x-axis to "predicted performance" improve clarity, as the presence of two variables (C C²) complicated 3D visualization. Overall, we find that FLP-M demonstrates better performance compared to FLP when considering the data
8

---

Published in Transactions on Machine Learning Research (04/2025)
Performance on RACE
Performance on ARC
38
FLP Curve Actunl
37.0
Actrall
Optimization
Optmiizatin
36.5
37
Pexformance
Porformance
36.0
36*
35.5
35
35.0
34.5
34
34.0
33
0.3
0.6
0.0
02
0.4
Mixing Ratio
Mixing Ratio
Figure 6: We use the scaling laws function derived via FLP-M to find the optimal data mixing ratio that yields the estimated best performance on the corresponding benchmarks.
mixing as an extra factor in scaling laws analysis. Using average validation loss indicator for assessing the performance of LMs pre-trained on mixed data sources, such general text and code, is limited. Thus, the average loss fails to trace performance variations in downstream tasks because changes in data mixtures can affect different capabilities of the LMs. In contrast, FLP-M effectively leverages the domain-specific validation loss to capture the capabilities improvement in LMs, and thus can better predict the downstream performance. our experiments, FLP-M accurately predicts the performance of 3B LLMs across various data mixtures and the 7B LLM with 0.3 data mixing ratio with error margins within 10% for most benchmarks.
However, on TriviaQA, despite significantly outperforming FLP, FLP-M shows higher relative prediction error, ranging from 20% to 30%. This discrepancy can be explained by the substantial performance improvement when scaling LLMs from under 1B to 3B parameters (increasing from below 12 to over 28). In our sampling LMs configurations (sce Tab. 1), we lack sufficient data points to adcquately characterize the phase of accelerated performance improvement. To better model this trend, a practical solution is to add several sampling L.Ss between 1B and 3B parameters.
6
Further Analysis
6.1
Optimizing Data Mixture Using FLP-M Scaling Laws
We demonstrate how the derived scaling laws using FLP-M can be effectively applied to optimize data mixtures, enhancing downstream performance. We focus 1B LMs in this analysis due to compute constraints. For each dataset, we use the FLP-M to estimate the function that maps expended FLOPs in each data source to the downstream performance. Then we use this function to predict performance across mixing ratios from 0 to 0.5, in intervals of 0.01.
Among all evaluation datasets, the estimated scaling laws function exhibits non-monotonic behavior on the RACE and ARC datasets, reaching its peak at mixing ratios of 0.01 and 0.22, respectively. To verify, we train 1B LMs with these two mixing ratios and measure their performance on the corresponding benchmarks. The results are shown in Fig. 6. We find that the selected optimal mixing ratio can reliably yield better performance compared to the six mixing ratios adopted for the sampling LMs, highlighting FLP-M a practical approach for optimizing data mixtures to enhance performance on specific target tasks.
6.2 Ablation Study
We conduct further analysis to better understand the two stages in FLP-M. Specifically, we compare various approaches to estimate the FL.OPs-to-Los and Loss-to-Performance curves in FLP-M.

---

Published in Transactions on Machine Learning Research (04/2025)
Relathor Prodiction Average Prodtaininng lows
Prodic tien General Predimining
Prediction Predrratiig Lowa
60
Relathve Prediction
Predicttoo
1.5
20:04
1M43
304.
3642
38-22
30-24
Target
Prediction
Prediction
00:
Relative Prediction Erowr
90.2
Relative Predlction
2hd
2.8
20.0
231
10:0-
30-5.4
Target LLMS
Target 1.Ms
Target E.1Ms
Figure 7: The relative prediction error of average and domain-spocific pre-training loss. M, provides more stable and overall more accurate predictions for domain-specific loss (within 2.5% relative prediction error across most domains).
FLP-M: FLOPs Loss We experiment with several candidate analytical forms listed in Tab. 3. We assess their performance in estimating the average pre-training loss across the entire validation set, well as the domain-specifie pre-training losses on corresponding subsets. We present the fit curves in Fig. 14 (SE), and the relative prediction errors for pre-training loss estimation are shown in Fig. 7.
For average pre-training loss prediction, using more complex analytical models that account for the in- dividual impact of each data source can lead per- formance degradation. However, relying solely the total compute for prediction (M1) can cause high prediction errors in certain domains (e.g., code) and are not stable for various mixing ratios. More complex analytical models generally perform bet- ter in predicting domain-specifie loss. Among them, M4, the adopted model in FLP-M, provides stable (within 2.5% relative prediction error across most domains) and overall more accurate predictions (achieving the lowest average error shown in Tab. 3).
Table 3: Candidate analytical forms for fitting the FLOPs-to-Loss curve. Except for CC and CC represent- ing the compute used for general and code data sources, other constants need to be estimated. The average er- ror is computed across all domains and model types.
LP(CE, (C)==
Analytical Form
Average Error
MI
(0940 you Cl yeur x (C youn
0.029
M2
0.026
M3
Bur 10 jac
0.017
M4 (Ouss
Jac
( Jues
0.014
FLP-M: Loss + Performance We experiment with various approaches to estimate the function that maps the pre-training loss the downstream performance. In this study, we utilize the actual pre-training loss of target LLMs, rather than the predictive loss used in $4. We consider the following candidates with different. inputs:
(1) FLOPs:
We adopt the analytical form used to predict the pre-training loss based on training compute (see Eq. 5), only changing the target metric to the downstream performance.
22 Average Loss (Average): We implement a linear regression model to map the average pro-training loss on the whole validation set to the downstream performance.
3 Domain Loss via Linear Combination (Domain-Lineall We apply lincar regression model to correlate pre-training loss across domains with downstream performance.
10

---

Published in Transactions on Machine Learning Research (04/2025)
Relative Prediction Error of 38-0.1-Mixing
Relative Prediction Error 2--iingMixing
Relative Prediction Error of 38-0.4-Mixing
0.2
0.40
Retative Predictionn trodicttoon Broor
0.175
0.20
The Naw
0:30
Seeaiin Preitee
0.100
0.20
0.0
Hetattive
0.00
0.00
8.10
TrYko
200
Tr1004
TrYDAl
0.40
RUN
Relative Prediction Error
Drewh
Erross
gelati Prediction
0.32
0.23
0.30
1222
0.10
0.02
Datavot
Dataset
Dataset
Figure 8: The relative prediction error of various approaches to estimate the
curve.
Neural network estimation with domain-specifie loss as input achieves the best prediction.
Relative Prediction Error of 7B LLM
Relative Prediction Error of 13B LLM
RLP
0.8
FLP
Relative Prediction Error
0.5
Relative Prediction Error
0.7
0.6
0.5
0.3
0.4
0.3
0.2
0.1
0.1
0.0
KBC
TEsNOA
TamisAA
Bellwwing
HomantToll
Dataset
Dataset
Figure 9: The relative prediction error 7B and 13B LLMs using FLP and FLP-M. FLP achieves significantly belter performance.
(4) Domain Loss via Neural Network (Ours) (Domain-Neural); We implement a two-layer neural network map the pre-training loss across domains to the downstream performance. The network configuration and optimization process introduced in $4.
The fit curves shown in Fig. 15 ($E) and the results of relative prediction error shown in Fig. 8. Consistent with the findings in $3, directly estimating the performance based on expended compute (FLOPs) leads to highly inaccurate iondictions (FLOPs us. Loss). Pre-training loss serves as more reliable metric for performance estimation, and decomposing it into domain-specific loss can further enhance prediction accuracy (Average us. Domain Loss). For the predictive models, using neural network estimation can better leverage the abundant data points produced by FLP-M, resulting in better performance compared the linear regression model (Linear Neural Network).
11

---

Published in Transactions on Machine Learning Research (04/2025)
6.3 Using Domain Loss in FLP
We explore the application of FLP-M when pre-training on a consistent distribution (the experimental setting described in $3), and compare it with FLP. The fitting curves are shown in Fig. 16 (Appendix) and the results of relative prediction error are shown in Fig. 9. We show that FLP-M fails to effectively predict the performance of target LLMs when sampling LMs are pre-trained on a fixed distribution. This ineffectiveness is attributed the closely related domain-specific validation losses among the sampling LMs within the same training distribution, which suggests that decomposing the pre-training validation loss yields no additional information in this pre-training setting. Thus, estimating five domain-specific loss, rather than a single average validation loss, can further increase the risk of error propagation. Moreover, using highly correlated features as neural network inputs may lead to overfitting.
7
Related Work
7.1 Scaling Laws
Estimating the performance of the target LLM prior to training is essential due to the significant resources required for pre-training (Minaee et al., 2024; Wan el al., 2023; Touvron el al., 2023; Chen et al., 2024). The scaling laws of LLMs guide the systematic exploration in scaling up computational resources, data, and model sizes (Kaplan et al., 2020; Hestness et al., 2017). Previous efforts in this field demonstrate that LLMs' final pre-training loss on a held-out validation set decreases with an increase in expended FLOPs during pre-training (Kaplan et al., 2020; Hoffmann et al., 2022; Yao et al., 2023). The following work subsequently establishes the scaling laws for computer vision models (Zhai et al., 2022), vision-language models (Henighan et al., 2020; Alabdulmohsin et al., 2022; Li et al., 2024a), mixed quantization (Cao et al., 2024), graph self-supervised learning (Ma et al., 2024), reward modeling (Gao et al., 2023a; Rafailov et al., 2024), data filtering (Goyal et al., 2024), knowledge capabilities of LLMs (Allen-Zhu & Li, 2024), data-constrained LMs (Muennighoff et al., 2024), data poisoning (Bowen al., 2024), LLMs vocabulary size (Tao al., 2024), retrieval-angmeened LLMs (Shao et al., 2024), continued pre-training of LLMs (Que et al., 2024), LLMs training steps (Tissue et al., 2024), fine-tuning LLMs (Tay et al., 2021; Lin et al., 2024; Hernandez et al., 2021), learning from repeated data (Hernandez et al., 2022), the sparse auto-encoders (Gao et al., 2024),
in LLMs pre-training (Yang et al., 2022; Lingle, 2024), and the mixture-of-expert LLMs (Clark et al., 2022; Frantar et al., 2023; Yun et al., 2024; Krajewski et al., 2024).
Despite the efforts, directly estimating the downstream performance of LLMs more accurately reflects the models' capabilities pertinent to our concerns, yet it confronts challenges associated with emergent abilities in LLMs (Wei et al., 2022a), such as thain-of-thoughn reasoning (Wei et al., 2022b; Chen et al., 2023; Suzgun et al., 2022). In general, the compute required for pro-training must surpass a task-specific threshold to enable pre-trained LMs to perform better Ihan random chance. Previous work addresses this challenge by using the answer loss as an alternative metric (Schaeffer et al., 2024) or increasing the metric resolution, such as measuring the average number of attempts to solve the task (Hu et al., 2023). However, they encounter difficulties in aligning the proposed metric with the original task metric, which is of paramount interest to us. Our research directly predicts the task performance metries of the target L.MSs by utilizing readily available intermediate LMs. This approach operates independently from and complements existing approaches.
7.2 Data Mixture
Creating the pre-training dataset necessities collecting data from different sources (Liu et al., 2023; Shen et al., 2023; Bi et al., 2024; Wei et al., 2023), making the data mixture a critical factor in the study of scaling laws. Ye et al. (2024) propose the data mixing laws lo predict the pre-training loss of the target LLM given the mixing ratios. Liu et al. (2024) build the regression model to predict the optimal data mixture regarding the pre-training loss optimization, and Kang et al. (2024) further show that the optimal data composition depends on the scale of compute. In this work, we focus on integrating the data mixture factor to better predict the downstream performance.
12

---

Published in Transactions on Machine Learning Research (04/2025)
8 Conclusion
This paper introduces a two-stage FLP solution to predict downstream performance in LLMs by leveraging the pre-training loss to address the challenges of LLMs' emergent abilities. In addition, we propose FLP-M, a core solution for performance prediction that addresses the practical challenges of integrating pre-training data from diverse sources. The effectiveness of FLP and FLP-M is validated through extensive experiments. Furthermore, we demonstrate both the effectiveness of our selected analytical forms and the utility of FLP-M in optimizing data mixtures to enhance downstream lask performance.
Limitations
Our approach FLP-M is generally applicable across various data sources, yet currently, is demonstrated only in binary cases involving code and text data due to computational constraints. Our specific emphasis the mix- ing ratio of code is deliberate, reflecting its practical significance in real-world applications and the dominance of these two data sources in LLMS pre-training compared to others (e.g., academic articles, books, encyclopedias).
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.03375 2018.
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohna Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300 22312, 2022.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint arXiv:2404.05405, 2024.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling 2024.
laws. Proceedings of the National Academy of Sciences,
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepsock Ilm Scaling open-source language models with longtermism. arXiv preprint arXiv22401.02954, 2024.
Dillon Bowen, Brendan Murphy, Will Cai, David Khachaturov, Adam Gleave, and Kollin Pelrine. Scaling laws for data poisoning in Ilms, 2024. URL htttps///aaxxv.orr/aaa///// 02946.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Zeyu Cao, Cheng Zhang, Pedro Gimenes, Jianqiao Lu, Jianyi Cheng, and Yiren Zhao. Scaling laws for mixed quantization in large language models, 2024. URL https://arxiv. hrrg/aba/24e http//21100672..
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Measuring and improving chain-of-thought reasoning in vision-language models. arXiv preprint arXiv:2309.0446) 2023.
Yangyi Chen, Xingyao Wang, Hao Peng, and Heng Ji. A single transformer for scalable vision- langunle modeling. arXiv preprint (XXi:2407.064488 2024.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Scbastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057-4058 PMLR, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the al2 reasoning challenge. arXiv preprint arXiv:1803.05455 055477 2018.
Together Computer. Redpajama: An open source recipe to reproduce llama training datasct, 2023. URL https://giiti
P Kingma Dicderik. Adam: A method for stochastic optimization. 2014.
Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15796 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evci. Scaling laws for sparsely-connected foundation models. arXiv preprint arXiv:2309.08522 2023.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp.
PMLR, 2023a.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023b. URL http::://anoooooorerrr//////////
org/records/1025683
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, 1lya Sutakever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint #=Xiv:2406.04093 2024.
Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and Zico Kolter. Scaling laws for data litering-date curation cannot be compute agnostic. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 22702-22711, 2024.
Dirk Groeneveld, Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Tvison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838. 2024.
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023.
Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Hcewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.1477 2020.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01 293, 2021.
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-oddds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205.04487 2022.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling predictable, empiri- cally. arXiv preprint arXiv:1712.004009, 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 2022.
Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, et al. Predicting emergent abilities with infinile resolution evaluation. In The Twelfth International Conference on Learning Representations, 2023.
Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelligence linearly. arXiv preprint M.Xi::2444.099311 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Tample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint acXiv:2310.06825, 2023.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.08351. 2017.
Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rañdd Mahmood, and Ruoxi Jia. Au- toscale: Automatic prediction of compute-optimal data composition for training 1lms. arXiv preprint 2..iv2240.22117,, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0833 2020.
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczowski, Maciej Pióro, Michal Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasu Odrzygódde, Piotr Sankowski, et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07877 2024.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehen- sion dataset from examinations. arXiv preprint 4xXiv11004.04685. 2017.
Bozhou Li, Hao Liang, Zimo Meng, and Wentao Zhang. Are bigger encoders always better in vision large models?, 2024a. URL https: /ttp///axvvv//////////ee/.aaa org/ httgg/bb/2440.0000 00620.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Tvgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jicyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanic Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-Im: In search of the next generation of training sets for language models, 2024b.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Michael Y Li, Emily B Fox, and Noah D Goodman. Automated statistical model discovery with language models. arXiv preprint arXiv:2402.1788 2024c.
Haowei Lin, Baizhou Huang, Haotian Ye, Qinyu Chen, Zihao Wang, Sujian Li, Jianzhu Ma, Xiaojun Wan, James Zou, and Yitao Liang. Selecting large language model to fine-tune via rectified scaling law. arXiv preprint arXiv:2402.02314, 2024.
Lucas Lingle. A largo-scale exploration of p-transfer. arXiv preprint arXiv:2404.05728 2024.
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint. arXiv:2407.01492, 2024.
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. Llm360: Towards fully transparent open-source llms. arXiv preprint arXiv:2312.0652. 2023.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhno, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas KrauB, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generalion, 2024.
Nouamane Tazi,
Qian Ma, Haitao Mao, Jingzhe Liu, Zhehua Zhang, Chunlin Feng, Yu Song, Yihan Shao, Tianfan Fu, and Yao Ma. Do neural scaling laws exist on graph self-supervised learning?, 2024. URL aba/2408. 11243.
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, arXiv preprint arXiv:2402.06196, 2024.
Niklas Muennigholf, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
Guilherme Penedo, Hynek Kydliček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024.
Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, et al. D-ept law: Domain-specific continual pre-training scaling law for large language models. arXiv preprint arXiv:2406.01375, 2024.
Rafael Rafailov, Yaswanth Chittopu, Ryan Park, Harshit Sikehi, Joey Hejna, Bradley Knox, Chelsca Finn, and Scott Niekum. Scaling laws for reward model overoplimization direct aligument algorithms. arXiv preprint arXiv:2406.02940 02900, 2024.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2024.
Rulin Shao, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke Zettlemoyer, and Pang Wei Koh. Scaling retrieval- based language models with a trillion-token datastore, arXiv preprint arXiv:2407. 11.224477.185.4 2024.
16

---

Published in Transactions on Machine Learning Research (04/2025)
Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing. Slimpajama-de: Understanding data combinations for I1 training. arXiv preprini. arXiv:2309. 2023.
Hui Su, Zhi Tian, Xiaoyu Shen, and Xunliang Cai. Unraveling the mystery of scaling laws: Part i. arXiv preprint arXiv:2403.06563, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of- thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. arXiv preprint arXiv:2407.1362. 2024.
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109.10686 2021.
Howe Tissue, Venus Wang, and Lu Wang. Scaling law with learning rate annealing, 2024. URL https:
//arxiv.org/ab//2260/. 11029
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Sounya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092:
2023.
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, et al. Efficient large language models: A survey. arXiv preprint arXiv:2312.0386) 1, 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Scbastian Borgcaud, Dani Yogatama, Maarten Bosma, Demny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07622 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Cf prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24889 2022b.
Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijic Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, et al. Skywork: A more open bilingual foundation model. arXiv preprint arXiv:2310.19341, 2023.
Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao. Sayself: Teaching iims to express confidence with reff-reftective rationales. arXiv preprint mxXiv:2405.20974, 2024.
Greg Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466 2022.
Yiqun Yao, Xiusheng Huang, Xuezhi Fang, Xiang Li, Ziyi Ni, Xin Jiang, Xuying Meng, Peng Han, Shuo Shang, Kang Liu, et al. nanolm: an affordable 11m pre-training benchmark via accurate loss prediction across scales. arXiv preprint arXiv:2304.06877. 2023.
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2408.16952, 2024.
Longfei Yun, Yonghao Zhuang, Yao Fu, Eric P Xing, and Hao Zhang. Toward inference-optimal mixture-of- expert large language models. arXiv preprint arXiv:24D4.02852. 2024.
17

---

Published in Transactions on Machine Learning Research (04/2025)
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905078330 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer Scaling vision transformers. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 12104-12113, 2022.
18

---

Published in Transactions on Machine Learning Research (04/2025)
30
25
Poeformaar
Peeformanr
85
2.50
2.35
2.30
2.15
2.10
3,05
3.00
3.0
26
20
HumanEEvl
DIVGOA
Performance
Perfumannce
-0
12
10-
2.6
2.3
2.5
28
2.0
97
7.0
7.2
Loss
Loss
Loss
Figure 10: We visualize the relation between pre-training loss and task performance for all LMs that surpass random baseline performance on the target benchmark, observing generally linear trend.
ARC
BBHI
Hellowway
Pertormauce
Performance
FRouve Sandectih
Sureding
1044
101a
101t
10ta
1013
11L0PS
FLOPS
PLOPS
HummanEvll
RACE
TriviaQA
10'
1.0
Performance
Ieffrrmancc
30F
317
TB Care
Fl: Toree LTM.
107
Terpe 12.0
Torret 12.06
1075
1279
1077
1022
1003
10
10
1003
1077
1022
102
30/1
10.
10:00
1077
10%:
10
FLOPS
FLOPS
PLOPS
Figure 11: The downstream performance prediction using FP (Achiam al., 2023) and FLP fit curves.
can better predict the downstream performance of target 7B and 13B LLMs across all evaluation benchmarks, while FP's predictions very unstable (e.g., HumanEval, TriviaQA).
Appendix
A Linear Relation Between Loss and Performance
We gather data points from intermediate checkpoints of all sampling LMs and visualize the relationship between pre-training loss and corresponding task performance in Fig. 10. We observe generally lincar trend
19

---

Published in Transactions on Machine Learning Research (04/2025)
Relative Prediction Error of 7B LLM
Relative Prediction Error of 13B LLM
Relative Prediction Error
0.5
FLP
Relative Prediction Error
F1.2
0.5
0.4
0.3
0.2
0.1
0.1
RAGU
TehisOA
AdC
italacunng
MACX
TrvisO
.b11
ABC
Dataset
Dataset
Figure 12: The comparison to the downstream task prediction approach in Llama-3 development (Dubey et al., 2024). We find that initially estimating the negative log-likeli of the target answer does not effectively predict performance based on our data points.
across all benchmarks, which motivates our selection of linear analytical form to characterize the mapping from pre-training loss to downstream performance. In addition, the decision is further supported by the following evidence:
Huang et al. (2024) demonstrate through extensive experiments across 12 benchmarks and 31 LLMs that language modeling ability (compression) correlates linearly with benchmark performance (intelligence).
Du et al. (2024) provide empirical validation and visualization of this linear relationship specifically in the post-emergence regime.
Overall, the above results and supporting evidence also justify that monitoring pre-training loss alone is sufficient. lo measure downstream benchmark performance in LLMs. Changes in pre-training loss correspond proportional changes in downstream benchmark performance, following a consistent linear trend without significant variance.
MMLU
B MMLU Experiment
Target Performance
45
Our sampling LMs, up to 3B, exhibit random performance (i.e., 25%) on the MMLU benchmark (Hendrycks et al., 2020). Consequently, these models do not provide effec- tive data points for estimation. Accordingly, we utilize intermediale checkpoints from 7B LLMs to estimate the performance of 13B LLMs on MMLU using FLP. The re- sults are shown in Fig. 13, and the relative prediction crror is 3.54%. FLP can also effectively predict the performance on MMLU by leveraging intermediate LMs checkpoints that emerge on this task.
40
33
30
Samplng Zaopei LLNS
x-x
Predictive Performance
Figure 13:
The performance prediction on MMLU
using FLP.
C Analytical Form to Fit Perforrmaneee Curve
We also experiment with the analytical form proposed in Achiam et al. (2023) to estimate the FLOPs-to- Performance curve:
C
log P(C)
( yoar,
(6)
20

---

Published in Transactions on Machine Learning Research (04/2025)
Average Pre-Training Loss Prediction
General Pre-Training Loss Prediction
Arxiv Pre-Training Loss Prediction
20
28
23
Targee
te yrt L1.0
Mo 1..9
Mo Teage
Ho Turpe
Hy Dagal
33
No Torpet
2.0
My Tergit
Tagel
NJ Torp
Tege
34
30-
3.8
3.5
2.5
Pre-Training
Pre-Truining
Pre-ninnng
1.00
2.00
1.35
20
2:75
1.50
Target Loos
1.73
3:25
2.50
2.55
Zeget
Targut
Tarqet
Samping LMs
2.
Pendting
Senctingie 13fe
2.5
4.25
2,10
1.75
1.50
1.25
1.0
2.2
2,1
2,2
1.
4.00
2,30
3.25
1.00
2550
225
Predictive Loss
Predictive Loss
Predictive Loss
Figure The pre-training loss prediction using various analytical forms. M4 provides more stable and overall more accurate predictions for domain-specific loss.
ARC
BBR
Hellaswag
Torget Per Performnncc
80
MLNON Ansrae
mires Awrnge
Jmsage
Snewty Sears
Tenan-NWar
Domar derrr
Seapling
Sempling
70-
90
Predictive Performance
Predictive Performance
Preslictive
Humaneval
RACE
TriviaQA
Targeet Port fornancce
20-
JUSh Denats
IUNN
Euspper
10
Hichs
Dreab-
10
Demate
Figure 15: The downstream performance prediction using various approaches. The domain loss coupled with neural network estimation demonstrates the best prediction performance.
where CM and aM are constant terms to be estimated, The fit curves are shown Fig. 11. We observe that FLP still consistently outperforms FP across all evaluation benchmarks. In addition, FP can yield very unstable prodictions on certain datasets, like HumanEval and TriviaQA, due a lack of sufficient data for accurate modeling.
21

---

Published in Transactions on Machine Learning Research (04/2025)
ARC
BBH
Hellaswag
101
Purfommanc
Perfrrmncc
1.0
101
FLP Care
Came barppin (146
RIP Dame Semeiie
kampling LMN
Darger
Sarger
Raper
HvmanEval
TriviaQA
1n1
10'
Performaacc
Performance
2x3p
107
FLPM FA
PLPME FRCN
RUFMFR
2014
101h
1171
2011
10
10
2040
10
10:3
1051
1077
10
1012
12:4
1020
1014
1071
1017
FLOPS
FLOPS
FLOPS
Figure 16: The downstream performance prediction using FLP and FLP-M fit curves. FLP can better predict the downstream performance of target LLMs with 7B and 13B parameters.
D Compare with Llama-3 Approach
We compare with the Llama-3 approach for downstream task prediction (Dubey et al., 2024). Our implemen- tation strictly adheres to the two-stage performance prediction framework outlined in the Llama-3 paper: (1) Stage 1: We estimate the negative log-likelihood (NLL) based on FLOPs using the prescribed analytical forms; (2) Stage 2: We map the estimated NLL to task performance through sigmoid function. The comparison results are shown in Fig. 12. We find that the two-stage approach proposed in Dubey et al. (2024) fails to effectively estimate the performance based on our data points, compared to FLP. The divergence between our results and those reported in the Llama-3 paper can be attributed to the key implementation difference: while the Llama-3 paper leverages comprehensive datasel including a huge amount of Llama-2 model checkpoints fit their NLL-too-erformance curve, our analysis relies on more limited set of data points. In addition, the compute used to train the sampling LMs also matters for the prodiction accuracy. Larger training FLOPs is easier to get emerged performance, which is more useful for fitting the analytical form. The difference in training data density and the FLOPSs used to train sampling LMS significantly impacts the sensitivity of the fitted analytical forms, particularly in regions where data points are sparse.
E FLP-M: Fit Curve for Ablation Study
The FLOPs-to-Lee fit curves are in Fig. 14 and the Loss-to-Per Performance fit curves are in Fig. 15. M, in Tab. 3 offers more stable and accurate predictions for domain-specifie loss, with the combined approach of domain loss and neural network estimation delivering the best overall downstream performance prediction.
F Automated Statistical Model for FLP-M Loss-to-Performaan Mapping
We also conduct experiments with automated statistical model discovery (Li et al., 2024c) to evaluate inter- petable approaches (us. neural networks) for the Loss-to-Performance mapping FLP-M. Our implementation involves using GPT-4o with multinodal capabilities and maximum iterations. We also apply human post-processing to extract key insights from model-generated natural language feedback. We evaluate the
22

---

Published in Transactions on Machine Learning Research (04/2025)
Table 4:
The relative prediction error of automated statistical model (Li et al., 2024c), linear function, and neural network in FLP-M Loss-to to to-terrrroaaace mapping.
Method
BBH
ARC
TriviaQA
Hellaswag
RACE
HumanEval
et (2024c)
14.5
13.2
5.9
18.6
Linear Function
4.3
11.8
7.8
2.5
4.1
12.0
Neural Network
9.1
8.6
0.4
1.3
3.4
14.3
relative performance prediction error across six benchmark tasks (%). Our analysis reveals several key findings:
Performance Comparison: The automatically discovered interpretable method shows higher error rates across all benchmarks compared to our neural network approach.
Model Complexity Trade-off: While the linear function offers the highest interpretability, it achieves intermediate performance between our neural network and the automated discovery method. This suggests a clear trade-off between model simplicity and prediction accuracy.
Neural Network Architecture: Our approach uses a minimalist architecture (two-layer network with hidden size 3 and ReLU activation) that balances interpretability with performance. This simple structure allows to (a) Express the model in analytical form, (b) Perform detailed numerical analysis, and (c) Maintain transparency in the prediction process
These results demonstrate that while fully interpretable hothods are valuable, they currently face challenges in matching the performance of even simple neural networks for this specific task. However, we maintain interpretability in our approach through architectural simplicity and comprehensive analysis capabilitics.
23


Reviews:
[{"review_id": "xqzL1xJXOK", "reviewer": "Reviewer_f9Zj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n1. Predicting downstream performance accurately is very useful in practice.  \n2. This paper conducts comprehensive experiments and considers the domain mixing setting, demonstrating effectiveness of the method.\n\n\n### Weaknesses\n\n1. I am a bit confused about the method novelty and the baseline FP here. I wonder how it is necessary to explicitly state the method as “two-stage”, for example, isn’t it straightforward to directly compose Eq (1) and (2) into one single equation? And that single equation, compared to the baseline FP in Eq. (3), only differs by a bias term I think. Also, proposing Eq (1) and Eq (2) is not novel but pretty straightforward – the FLOPs -> loss predictive law has been studied widely, and the linear relationship between loss and downstream performance is already reported in the cited paper Huang et al. 2024.  \n2. It is obvious that the linear equation in Eq (2) cannot capture well the points before the “emergent phase”, which are reflected in Figure 2 where FLP cannot fit well many sampled points. In Section 2.2, the authors noted that the linear equation is only adopted to fit the data points where the performance exceeds the random performance, does this mean the authors only consider the data points after the emergent phase? Then, do the authors adopt the same filtering strategy when evaluating the baseline FP? If not, I don’t think this is a fair comparison with the baseline.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a two-stage approach, FLP, to predict downstream performance of target LLMs directly from FLOPs. Specifically, the authors first fit a function to predict pretraining loss given FLOPs, which is a standard operation. Then the authors fit a linear function to predict the downstream performance of larger LLMs through data points from smaller LLMs given losses. The prediction is relatively accurate compared to directly fitting a power law function between downstream performance and FLOPs. Later on, this paper extends the  framework to a text+code mixed training setup to predict downstream performance given the FLOPs on each pretraining domain respectively. The authors also demonstrate interesting analysis to derive the optimal pretraining corpus mixing ratio given the predictive law.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned in the weaknesses above, I think more clarifications are needed to address the novelty of the method, and comparison with the baseline FP. Theoretically it looks like the composed equation FLP should be similar to the baseline FP, and I don’t understand why they produced drastically different curves in Figure 2 – my current guess, as mentioned in Weakness 2, is the authors filter the randomly-performing data points when fitting FLP but didn’t filter them when fitting the baseline, please correct me if I am wrong.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pDiHxpp3id", "reviewer": "Reviewer_XjyB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. **Conceptual Novelty**: The paper presents a unique two-stage approach to performance prediction, separating FLOPs-to-loss estimation from loss-to-performance mapping. This provides a more fine-grained and computationally efficient alternative to direct FLOPs-to-performance methods.\n2. **Empirical Rigor**: The authors conduct extensive experiments, using models up to 3B parameters to predict 7B and 13B LLMs, showing strong predictive accuracy and outperforming baseline methods.\n3. **Scalability**: FLP-M addresses the practical challenge of integrating multiple data sources, extending the approach to real-world scenarios where training data is a mixture of different domains.\n4. **Practical Impact**: By optimizing data mixture compositions, FLP-M provides insights into how pre-training data should be allocated, offering practical utility for LLM training pipelines.\n5. **Comprehensive Ablations**: The authors conduct ablation studies to validate design choices, demonstrating robustness across different data mixtures and analytical formulations.\n\n### Weaknesses\n\n\n1. **Choice of Function for Loss-to-Performance Mapping**: The authors assume a linear relationship between pre-training loss and downstream performance. While this is empirically supported, it may not generalize across all task types, especially in domains where performance gains exhibit non-linear trends. More investigation into the modelling assumptions would make this paper stronger. \n2. **Neural Network Usage in FLP-M**: The paper employs a two-layer neural network to map domain-specific loss to downstream performance. Given that interpretability is already sacrificed in this stage, why not use a high-capacity neural network for the first stage as well? Or directly combine both stages with a high-capacity neural net? If accuracy is the goal, why limit the complexity of the first-stage model?\n3. **Alternative Interpretable Methods**: The authors could explore interpretable regression-based alternatives, such as [1] and [2], which may offer greater insight into the learned relationships while maintaining competitive accuracy.\n\n\nLi, Michael Y., Emily B. Fox, and Noah D. Goodman. \"Automated statistical model discovery with language models.\" *arXiv preprint arXiv:2402.17879* (2024). https://arxiv.org/abs/2402.17879\n\nGandhi, Kanishk, et al. \"BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery.\" *arXiv preprint arXiv:2501.01540* (2025). https://arxiv.org/abs/2501.01540", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a two-stage approach, FLP, to predict downstream performance in LLMs based on pre-training loss, addressing the challenge of emergent abilities. The FLP framework first models the relationship between computational resources (FLOPs) and pre-training loss using a power-law function, followed by modeling the relationship between pre-training loss and task performance using regression techniques. The authors further introduce FLP-M, an extension designed to accommodate domain-specific data mixtures by incorporating a neural network to model non-linear dependencies between loss across multiple domains and downstream performance.\n\n### Key Contributions\n- A novel two-stage FLP approach for predicting downstream performance with improved sample efficiency.\n- FLP-M, an extension for data mixing scenarios, effectively optimizing pre-training corpus compositions for better task performance.\n- Experimental validation demonstrating superior prediction accuracy compared to FLOPs-to-performance methods, with error margins of 5%-10%.\n- Empirical analysis showing that domain-specific loss provides better signals for performance estimation in mixed datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Clarify the Justification for Using a Linear Model in Loss-to-Performance Mapping**: The authors should discuss why they opted for a linear function when emergent abilities might suggest more complex relationships.\n2. **Consider Using a Neural Network in FLOPs-to-Loss Stage**: Since the final stage already uses a neural network, it would be interesting to compare the performance of a neural network in the first stage as well.\n3. **Experiment with Alternative Interpretable Approaches**: The authors should explore methods such as automated statistical model discovery to evaluate whether interpretable methods can achieve similar performance. (Though I'm not familiar with this subject, I would guess methods like [1] and [2] could potentially be more interpretable, or you could also explore other interpretable methods)\n \nLi, Michael Y., Emily B. Fox, and Noah D. Goodman. \"Automated statistical model discovery with language models.\" *arXiv preprint arXiv:2402.17879* (2024). https://arxiv.org/abs/2402.17879\n\nGandhi, Kanishk, et al. \"BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery.\" *arXiv preprint arXiv:2501.01540* (2025). https://arxiv.org/abs/2501.01540", "broader_impact_concerns": "I don't have any concerns here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ccAPRlQmYC", "reviewer": "Reviewer_pXiU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The paper highlights the challenge of emergent phenomena in direct performance prediction and proposes a two-stage framework to address this issue. \n2. The consideration of a data training mixture for predicting the scaling law is interesting, and the findings are insightful. \n3. The paper is clearly written, with extensive experiments conducted across multiple model sizes and diverse benchmarks.\n\n## Weaknesses\n\n1. The largest model used in this paper is only 13B, whereas many LLMs are much larger. From this perspective, the conclusions of this paper are limited.\n\n2. Extending the work to parameter-efficient fine-tuning (PEFT) methods, such as LoRA or any preference-optimization approaches, could have a broader impact. Additionally, related to the above scalability issue, using LoRA, for example, would make it easier to train larger models.\n\n3. The authors fine-tune a series of intermediate checkpoints during training for downstream prediction, which is relatively unconventional. This is because the complete training schedule significantly impacts downstream performance. Sometimes, a small improvement in loss during the final training phase can lead to a non-negligible improvement in downstream tasks. Hence, given the same loss, the training phase of the checkpoints can also play a major role in the prediction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents two practical contributions to scaling laws research for LLMs.  First, it introduces FLP, a two-stage framework (FLOPs -> Loss, Loss -> Performance) designed to predict the downstream task performance of larger LLMs based on data from smaller models. Second, it proposes FLP-M, an extension of FLP that considers data training mixtures (specifically general text and code) during predicting scaling laws for downstream task performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses, especially point 2 and 3.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviews of this paper were mixed, with aforementioned concerns being raised about more convincing experiments and the notability of the results. After a read over the paper, I do think the claims are reasonably well-supported and I do think the paper could be of interest to the TMLR community. However, if feasible, the authors should include the additional experiments in their camera-ready.
Decision: Accept (poster)