Scaling Laws for Predicting Downstream Performance in LLMs

## Abstract

Abstract Precise estimation of downstream performance in large language models (LLMs) prior to training is essential for guiding their development process. Scaling laws analysis utilizes the statistics of a series of significantly smaller sampling language models (LMs) to predict the performance of the target LLM. For downstream performance prediction, the critical challenge lies in the emergent abilities in LLMs that occur beyond task-specific computational thresholds. In this work, we focus on the pre-training loss as a more computation-efficient metric for performance estimation. Our two-stage approach consists of first estimating a function that maps computational resources ( e.g., F LOPs) to the pre-training L oss using a series of sampling models, followed by mapping the pre-training loss to downstream task P erformance after the critical “emergent phase”. In preliminary experiments, this FLP solution accurately predicts the performance of LLMs with 7B and 13B parameters using a series of sampling LMs up to 3B, achieving error margins of 5% and 10%, respectively, and significantly outperforming the FLOPs-to-Performance approach. This motivates FLP-M , a fundamental approach for performance prediction that addresses the practical need to integrate datasets from multiple sources during pre-training, specifically blending general corpora with code data to accurately represent the common necessity. FLP-M extends the power law analytical function to predict domain-specific pre-training loss based on FLOPs across data sources, and employs a two-layer neural network to model the non-linear relationship between multiple domain-specific loss and downstream performance. By utilizing a 3B LLM trained on a specific ratio and a series of smaller sampling LMs, FLP-M can effectively forecast the performance of 3B and 7B LLMs across various data mixtures for most benchmarks within 10% error margins.

## 1 Introduction

Figure: Figure 1: The performance of sampling LMs with increasing compute. x represents non-emerged data points, and $\bullet$ indicates emerged data points that surpass a randomness threshold of 5.
Refer to caption: extracted/5918555/figs/motivating.png

Large language models (LLMs) form the basis for numerous real-world applications (Brown et al., ; Jiang et al., ; Xu et al., ; Hadi et al., ) and scaling laws analysis serves as the foundation for LLMs development (Kaplan et al., ; Bahri et al., ).
The key idea of scaling laws involves training a sequence of language models (LMs) to gather data (e.g., expended compute and corresponding model performance).
This data is then used to build a predictive model that estimates the performance of a substantially larger target LLM (Su et al., ; Hoffmann et al., ).

Previous efforts focus on predicting the target LLM’s pre-training loss and establish a power-law relation between the computational resource expended (e.g., floating-point operations per second (FLOPs)) and the final loss achieved (Kaplan et al., ; Muennighoff et al., ; Henighan et al., ).
Further, we aim to predict the downstream performance in LLMs to more accurately reflect the primary concerns regarding their capabilities.
The critical challenge is the emergent abilities in LLMs, which states that LLMs only exceed random performance when the FLOPs expended during training surpass task-specific thresholds (Wei et al., ).
Supposing a task threshold of $F_{c}$, typical methods require training $N$ LMs, expending total FLOPs $F_{t}=\sum_{i=1}^{N}\text{FLOPs}_{i}>N\times F_{c}$, to obtain $N$ effective data points, thereby necessitating significant computational resources.
Fig. 1 demonstrates that the sampling LMs require more than $5\times 10^{20}$ FLOPs to perform better than random on most benchmarks, with only three data points available to fit the predictive curve across these benchmarks.
Hu et al. () address this challenge by significantly increasing the sampling times to compute the PassUntil of a task, basically increasing the “metric resolution” to enable the abilities to emerge earlier (i.e., reducing $F_{c}$).
However, this approach faces challenges in translating the PassUntil back to the original task metric of concerns and requires huge amounts of FLOPs spent on sampling.

In this work, we target the actual task performance prediction based on two intuitions:
(1) Predicting the target pre-training loss is easier and achievable since there is no “emergent phase” in the pre-training loss, as extensively verified in Kaplan et al. (); Hoffmann et al. ();
(2) There is an observed correlation between the pre-training loss and the downstream task performance after the “emergent point” (i.e., the pre-training loss goes below a critical threshold) (Du et al., ; Huang et al., ).
Rethinking previous practice, training LM $i$ to convergence requires expending FLOPsi for obtaining a single data point.
In contrast, our approach stems from a crucial insight:
Collecting (pre-training loss, performance) data points at intermediate checkpoints prevents the need for fully training LMs to convergence, thereby enhancing sample efficiency.
Essentially, we can collect a huge amount of useful data between the initial point of above-random performance in LMs and their convergence point for performance prediction.

Thus, our approach consists of two sequential stages:
(1) $\text{{F}LOPs}\rightarrow\text{{L}oss}$: Predict the target pre-training loss based on the expended FLOPs.
Following previous work, we train a series of sampling LMs within the same model family to develop a power-law predictive model.
For this stage, the expended FLOPs are not required to reach above the emergent threshold.
(2) $\text{{L}oss}\rightarrow\text{{P}erformance}$:
Predict the downstream performance based on the pre-training loss.
We collect data points from intermediate checkpoints of various sampling LMs that exhibit above-random performance, and develop a regression model for prediction.
In preliminary experiments with sampling LMs up to 3B, this FLP solution predicts the performance of 7B and 13B LLMs across various benchmarks with error margins of 5% and 10% respectively, significantly outperforming direct FLOPs-to-Performance predictions.

Motivated by these findings, we present FLP-M, a fundamental solution for performance prediction that addresses the growing demand for integrating diverse datasets during LLMs pre-training, focusing on integrating the general corpus with code data in this work.
FLP-M targets fine-grained domain-specific pre-training loss to capture the performance changes.
Specifically, we extend the power law analytical function to predict the domain-specific loss based on FLOPs across multiple data sources.
Then we employ a two-layer neural network to model the non-linear relationship between multiple domain-specific loss and the downstream performance.
Through evaluation, we demonstrate that FLP-M effectively predicts the performance of 3B and 7B LLMs trained on various data mixtures (within 10% error margins for most benchmarks).
This is achieved by utilizing a 3B LLM trained on a specific data mixing ratio along with a series of smaller sampling LMs.

## 2 Related Work

### 2.1 Scaling Laws

Estimating the performance of the target LLM prior to training is essential due to the significant resources required for pre-training (Minaee et al., ; Wan et al., ; Touvron et al., ; Chen et al., ).
The scaling laws of LLMs guide the systematic exploration in scaling up computational resources, data, and model sizes (Kaplan et al., ; Hestness et al., ).
Previous efforts in this field demonstrate that LLMs’ final pre-training loss on a held-out validation set decreases with an increase in expended FLOPs during pre-training (Kaplan et al., ; Hoffmann et al., ; Yao et al., ).
The following work subsequently establishes the scaling laws for computer vision models (Zhai et al., ),
vision-language models (Henighan et al., ; Alabdulmohsin et al., ; Li et al., ), mixed quantization (Cao et al., ), graph self-supervised learning (Ma et al., ), reward modeling (Gao et al., ; Rafailov et al., ), data filtering (Goyal et al., ), knowledge capabilities of LLMs (Allen-Zhu & Li, ), data-constrained LMs (Muennighoff et al., ), data poisoning (Bowen et al., ), LLMs vocabulary size (Tao et al., ), retrieval-augmented LLMs (Shao et al., ), continued pre-training of LLMs (Que et al., ), LLMs training steps (Tissue et al., ), fine-tuning LLMs (Tay et al., ; Lin et al., ; Hernandez et al., ), learning from repeated data (Hernandez et al., ), the sparse auto-encoders (Gao et al., ), hyper-parameters in LLMs pre-training (Yang et al., ; Lingle, ), and the mixture-of-expert LLMs (Clark et al., ; Frantar et al., ; Yun et al., ; Krajewski et al., ).

Despite the efforts, directly estimating the downstream performance of LLMs more accurately reflects the models’ capabilities pertinent to our concerns, yet it confronts challenges associated with emergent abilities in LLMs (Wei et al., ), such as chain-of-thought reasoning (Wei et al., ; Chen et al., ; Suzgun et al., ).
In general, the compute required for pre-training must surpass a task-specific threshold to enable pre-trained LMs to perform better than random chance.
Previous work addresses this challenge
by using the answer loss as an alternative metric (Schaeffer et al., ) or increasing the metric resolution, such as measuring the average number of attempts to solve the task (Hu et al., ).
However, they encounter difficulties in aligning the proposed metric with the original task metric, which is of paramount interest to us.
Our research directly predicts the task performance metrics of the target LLMs by utilizing readily available intermediate LMs.
This approach operates independently from and complements existing approaches.

### 2.2 Data Mixture

Creating the pre-training dataset necessities collecting data from different sources (Liu et al., ; Shen et al., ; Bi et al., ; Wei et al., ), making the data mixture a critical factor in the study of scaling laws.
Ye et al. () propose the data mixing laws to predict the pre-training loss of the target LLM given the mixing ratios.
Liu et al. () build the regression model to predict the optimal data mixture regarding the pre-training loss optimization, and Kang et al. () further show that the optimal data composition depends on the scale of compute.
In this work, we focus on integrating the data mixture factor to better predict the downstream performance.

## 3 FLP : Downstream Performance Prediction

We introduce a two-stage approach to predicting downstream performance in LLMs based on two established findings:
(1) Predicting the target pre-training loss and establishing the power-law relation is feasible as it does not involve an emergent phase (Kaplan et al., ; Hoffmann et al., ).
(2) When pre-training loss goes below a task-specific threshold, there is an observed correlation between pre-training loss and downstream task performance (Du et al., ; Huang et al., ).
In this section, we present FLP as a proof-of-concept for this framework with a straightforward implementation.

### 3.1 FLOPs → → \rightarrow → Loss

We follow the previous practice to use the analytical power law function to characterize the relation between expended FLOPs $C$ and the pre-training loss $L$:

$$ $L(C)=\left(\frac{C}{C_{N}}\right)^{\alpha_{N}},$ (1) $$

where $C_{N}$ and $\alpha_{N}$ are constant terms to be estimated.
In FLP, we train a series of $N$ LMs within the same model family in the same pre-training distribution, progressively increasing model size and training tokens to achieve even sampling.
Then we measure their pre-training loss in our curated validation dataset to obtain $N$ pairs of ($C_{i}$, $L_{i}$) to estimate the constants in Eq. 1.

### 3.2 Loss → → \rightarrow → Performance

Based on our empirical observation of the scatter plots showing (pre-training loss, performance) data points (see §A),
we select the analytical linear function to characterize the relation between the pre-training loss $L$ on general validation data and the task performance $P$:

$$ $P(L)=w_{0}+w_{1}*L,$ (2) $$

where $w_{0}$ and $w_{1}$ are constant terms to be estimated.
In FLP, we fetch the intermediate checkpoints of each sampling LM, and measure its task performance and pre-training loss.
If the performance $P_{i}$ of LMi exceeds the random performance,
we can obtain one effective data point ($L_{i}$, $P_{i}$) to estimate the constants in Eq. 2, where $L_{i}$ is the pre-training loss of LMi.

## 4 Validation of FLP Framework

### 4.1 Sampling and Target LMs

**Table 1: The configurations of the sampling and target LMs with various sizes. HD denotes the hidden dimension, BS denotes the batch size, and LR denotes the learning rate.**
| Model Size | #Layer | HD | #Head | FFN | #Tokens | Non-embedding FLOPs | BS | LR |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 43M | 3 | 384 | 3 | 1032 | 8,021,606,400 | 3.70504E+17 | 448 | 0.0052 |
| 64M | 4 | 512 | 4 | 1376 | 11,714,691,072 | 1.18417E+18 | 544 | 0.0042 |
| 89M | 5 | 640 | 5 | 1720 | 16,184,770,560 | 3.03607E+18 | 576 | 0.0038 |
| 0.12B | 6 | 768 | 6 | 2064 | 21,799,895,040 | 6.81931E+18 | 640 | 0.0040 |
| 0.15B | 7 | 896 | 7 | 2408 | 28,846,325,760 | 1.39581E+19 | 672 | 0.0042 |
| 0.2B | 8 | 1024 | 8 | 2752 | 37,213,962,240 | 2.63435E+19 | 736 | 0.0036 |
| 0.25B | 9 | 1152 | 9 | 3096 | 47,563,407,360 | 4.71817E+19 | 768 | 0.0034 |
| 0.32B | 10 | 1280 | 10 | 3440 | 59,674,460,160 | 8.01571E+19 | 800 | 0.0028 |
| 0.5B | 12 | 1536 | 12 | 4128 | 90,502,594,560 | 2.05963E+20 | 960 | 0.0023 |
| 0.72B | 14 | 1792 | 14 | 4816 | 132,026,204,160 | 4.70331E+20 | 1024 | 0.0019 |
| 1B | 16 | 2048 | 16 | 5504 | 185,535,037,440 | 9.75926E+20 | 1152 | 0.0016 |
| 3B | 24 | 3072 | 24 | 8256 | 556,793,856,000 | 9.63212E+21 | 1536 | 0.0004 |
| 7B | 32 | 4096 | 32 | 11008 | 1,258,291,200,000 | 5.09208E+22 | 2048 | 0.0003 |
| 13B | 40 | 5120 | 40 | 13824 | 1,258,291,200,000 | 9.89592E+22 | 2048 | 0.0003 |

We train a series of 12 sampling LMs up to 3B parameters to predict the performance of target LLMs with 7B and 13B parameters.
The configurations of LMs are shown in Tab. 1.
We first determine the number of training tokens required for the 7B LLM (approximately 180 times the model size), considering practical needs and inference-time costs.
In real-world applications, prioritizing inference efficiency often involves training smaller LMs with a higher token-to-parameter ratio beyond the optimal factor of 20x (Hoffmann et al., ).
Our preliminary experiments indicate that scaling laws remain applicable even in this over-training regime (within 2.8% error margins).
We then proportionally scale down this number to determine the required training tokens for the sampling LMs.

### 4.2 Data: Pre-Training, Validation, Evaluation

Pre-Training
We use the RedPajama v1 (Computer, ), which consists of 1.2T tokens in total, and the data is sourced from Arxiv, C4, Common Crawl, GitHub, Stack Exchange, and Wikipedia.

Validation
We curate a validation dataset to measure the final pre-training loss, which includes 5 distinct domains:
math, code, scientific paper, Wikipedia, and general language corpus.
Specifically, we utilize subsets from GitHub, ArXiv, Wikipedia, and the English portion of C4, all from the RedPajama validation sets, along with Proof Pile (Touvron et al., ) for the math domain.

**Table 2: The evaluation settings of the benchmarks.**
| Dataset | Evaluation Type | Evaluation Method | Metric | Random Performance |
| --- | --- | --- | --- | --- |
| ARC | Multiple Choice | 10-shot | Accuracy | 25 |
| BBH | Generation | CoT-3-shot | ExactMatch | 0 |
| Hellaswag | Multiple Choice | 10-shot | Accuracy | 25 |
| HumanEval | Generation | 0-shot | Pass@100 | 0 |
| RACE | Multiple Choice | 0-shot | Accuracy | 25 |
| TriviaQA | Generation | 0-shot | ExactMatch | 0 |

Evaluation
We select the following tasks for evaluation, covering fundamental capabilities in LLMs (e.g., knowledge, reasoning, coding):
RACE (Lai et al., ),
TriviaQA (Joshi et al., ),
BigBench-Challenge (BBH) (Suzgun et al., ),
ARC-Challenge (ARC) (Clark et al., ),
Hellaswag (Zellers et al., ),
and HumanEval (Chen et al., ).
The evaluation settings for these benchmarks are listed in Tab. 2.
We adopt lm-evaluation-harness (Gao et al., ) for unified evaluation.

### 4.3 Experimental Setting

Baseline
We consider directly using the expended FLOPs $C$ to predict the downstream performance $P$, and experiment with the following analytical form for comparison:

$$ $P(C)=(\frac{C}{C_{M}})^{\alpha_{M}},$ (3) $$

where $C_{M}$ and $\alpha_{M}$ are constant terms to be estimated.
We denote this approach as FP.

Implementation of FLP
To fit the FLOPs-to-Loss curve, we utilize the final checkpoints from each sampling LM.
In addition, during LMs training, a checkpoint is saved at every 1/30th increment of the total training progress.
We monitor and record the pre-training loss on the training dataset, rounded to two decimal places.
Only those checkpoints demonstrating an improvement in pre-training loss are retained.
For these selected checkpoints, we evaluate the downstream performance and pre-training loss on the validation set.
We then discard those that do not surpass the random benchmark performance by at least 5, and use the remaining data points to fit the Loss-to-Performance curve.

Evaluation Metrics
In addition to presenting the fitting curves for intuitive visualization.
we quantify the prediction accuracy by measuring the relative prediction error:

$$ $\text{Relative Prediction Error}=\frac{|\text{Predictive Metric}-\text{Actual Metric}|}{\text{Actual Metric}}$ (4) $$

Figure: Figure 2: The downstream performance prediction using FP and FLP fit curves. FLP can better predict the downstream performance of target 7B and 13B LLMs across all evaluation benchmarks.
Refer to caption: extracted/5918555/figs/FLP/performance_prediction_ARC_challenge.png

Figure: Figure 3: The relative prediction error of 7B and 13B LLMs. FLP achieves a more accurate prediction with error margins of 5% and 10% across all benchmarks for two LLMs respectively.
Refer to caption: extracted/5918555/figs/FLP/performance_gap_dense_31.png

### 4.4 Results

The downstream performance prediction results are visualized in Fig. 2.
Across all evaluation tasks, FLP fit curve can better predict the performance of target LLMs with 7B and 13B parameters using the sampling LMs up to 3B.
In contrast,
while FP more effectively fits the data points of sampling LMs, it has difficulty accounting for the “emergent phase” characterized by rapid performance shifts, due to the scarcity of data points from this period.
As a solution, FLP utilizes pre-training loss as a more fine-grained indicator to monitor performance changes and effectively incorporates data from intermediate checkpoints, enhancing sample efficiency.
The evaluation results of relative prediction error are shown in Fig. 3.
Unlike the suboptimal predictions of FP, FLP delivers precise forecasts, maintaining relative error margins of 5% and 10% across all benchmarks for 7B and 13B LLMs, respectively.

Compared to FP, FLP is less effective at fitting the data points of sampling LMs, especially in HumanEval and TriviaQA.
The reason is that we do not align with the “non-emergent” phase of the Loss-to-Performance curve, where LMs exhibit random performance when pre-training loss is beyond the task-specific threshold.
Thus, FLP predicts higher pre-training loss for LMs with fewer FLOPs, resulting in below-random performance.
This issue is not within the scope of FLP, as it is specifically designed to predict the performance of LLMs trained with significantly larger FLOPs in practice.

In addition, we discuss additional results in Appendix for the presentation purpose since adding these data points may distort the vertical axis scaling in Fig. 2.
We compare FLP further with the analytical forms and approaches proposed in GPT-4 (Achiam et al., ) and Llama-3 (Dubey et al., ) technical reports.
The results are shown in §B and §C respectively.
We also evaluate the feasibility of employing FLP to predict the performance of a 13B LLM on MMLU (Hendrycks et al., ), using intermediate checkpoints from a 7B LLM (§D).
Overall, the results demonstrate the general effectiveness and applicability of FLP.

## 5 FLP-M : Data Mixing for Downstream Performance Prediction

Motivated by the encouraging results of FLP (§4), we propose FLP-M, a fundamental approach to meet the practical needs of integrating data from various sources (Groeneveld et al., ; Penedo et al., ).
In our work, we focus on mixing general corpus with code data, considering two distinct yet overlapping data sources.
This intersection offers a more realistic perspective than treating them as distinct domains (Ye et al., ), as real-world corpus often spans multiple domains, necessitating an analysis of the interdependence between data sources when formulating our analytical functions.

Compared to the straightforward implementation of FLP (§3), FLP-M operates on fine-grained, domain-specific pre-training loss, due to the observation that the average loss on the entire validation set fails to effectively reflect performance variations in downstream tasks in the data mixing context (§7.2).
This may be due to the fact that changes in pre-training data mixtures simultaneously impact multiple capabilities of the LMs.
For instance, an increase in code data loss coupled with a decrease in general data loss may leave the average validation loss unchanged, yet result in LMs with distinct capabilities and downstream performance.
Note that unlike the pre-training data mixture, the validation set is deliberately curated by domain, as creating smaller, domain-specific validation sets is manageable.

### 5.1 FLOPs → → \rightarrow → Domain Loss

Given the FLOPs $C^{G}$ spent on the general corpus and $C^{C}$ spent on the code data, we naturally extend the power law function to the following analytical form to predict the domain-specific pre-training loss $L^{D}$ on domain $D$:

$$ $L^{D}(C^{G},C^{C})=(\frac{C^{G}+C^{C}}{C_{T}})^{\alpha_{C}}\times(\frac{C^{G}} {C_{G}})^{\alpha_{C_{1}}}\times(\frac{C^{C}}{C_{C}})^{\alpha_{C_{2}}}$ (5) $$

where $C_{T}$, $C_{G}$, $C_{C}$, $\alpha_{C}$, $\alpha_{C_{1}}$, and $\alpha_{C_{2}}$ are constants to be estimated.
In FLP-M, we first select a sequence of total compute $\{C_{i}\}_{i=1}^{N}$ spent on pre-training.
For each selected $C_{i}$, we experiment with various ratios to mix two data sources, and decompose $C_{i}$ into $C_{i}^{G}$ and $C_{i}^{C}$.
We measure the domain-specific pre-training loss $L_{i}^{D}$ on a domain-specific subset $D$ of validation data to obtain ($C_{i}^{G}$, $C_{i}^{C}$, $L_{i}^{D}$) data pairs.
Then we can estimate the constants in Eq. 5.
We also experiment with other potential analytical forms in §7.2.

### 5.2 Domain Loss → → \rightarrow → Performance

Given the pre-training loss $\{L^{D}\}_{D=1}^{K}$ on $K$ domains,
we train a two-layer neural network with a hidden layer size of 3 and the ReLU activation function (Agarap, ) to predict the downstream performance.
The network is optimized using the regression loss with L2 regularization and the Adam optimizer (Diederik, ), employing a learning rate of 0.05 that linearly decays to 0 within 2,000 steps and a weight decay of 0.01.
In FLP-M, we adopt the same strategy as in FLP to fetch the intermediate checkpoints and only retain the results that the LMs achieve above-random performance (see §3).
Thus, for LMi, we can obtain a sequence of effective data points ($\{L^{D}_{i}\}_{D=1}^{K}$, $P_{i}$), where $L_{i}^{D}$ is the pre-training loss on domain $D$ and $P_{i}$ is the LM’s performance.
Then we can use these data points to train the neural network.
We also explore other functions for fitting in §7.2.

## 6 Experiment for FLP-M

### 6.1 Sampling and Target LMs

We train a series of sampling LMs with sizes of $\{$0.12B, 0.2B, 0.32B, 0.5B, 0.72B, 1B$\}$, and the corresponding training token numbers are shown in Tab. 1.
We train the LMs on the general and code data mixture with $\{$0, 0.1, 0.2, 0.3, 0.4, 0.5$\}$ as the mixing ratios of code data to reflect real-world usage.
We also add one sampling LM of 3B size and 0.3 mixing ratio.
For evaluation,
we train 3B LLMs with the other mixing ratios and a 7B LLM with 0.3 as the mixing ratio due to the limited compute budget.

### 6.2 Data: Pre-Training, Validation, Evaluation

Pre-Training
For general corpus, we use DCLM (Li et al., ), a curated high-quality pre-training corpus including heuristic cleaning, filtering, deduplication, and model-based filtering.
For code data, we use The Stack v2 (Lozhkov et al., ), which initially contains over 3B files in 600+ programming and markup languages, created as part of the BigCode project.
We mix these two data sources to create the pre-training data mixture using the ratios specified in §6.1.

Validation
We use the same validation data mixture specified in §4.2 that includes 5 distinct
domains.

Evaluation
The evaluation benchmarks and settings are the same as those in §4.2.

### 6.3 Experimental Setting

Baseline
We implement FLP within this data mixing context as a baseline, which first predicts the average pre-training loss on the validation set and uses this to estimate downstream performance via linear regression.

Figure: Figure 4: The downstream performance prediction using FLP and FLP-M fit curves. FLP-M can better predict the downstream performance of target LLMs across various data mixing ratios.
Refer to caption: extracted/5918555/figs/FLPMix_FLP/arc_challenge_performance_prediction.png

Figure: Figure 5: The relative prediction error of downstream performance prediction using FLP and FLP-M. FLP-M can better predict the performance of target LLMs across various data mixing ratios.
Refer to caption: extracted/5918555/figs/FLPMix_FLP/error/0.png

Implementation of FLP-M
We adopt the same implementation as in FLP (details in §4.3). The distinction is that we individually measure the pre-training loss on each domain of the validation mixture.

### 6.4 Results

The downstream performance prediction results are visualized in Fig. 4.
We update the x-axis to “predicted performance” to improve clarity, as the presence of two variables ($C^{G}$, $C^{C}$) complicated 3D visualization.
Overall, we find that FLP-M demonstrates better performance compared to FLP when considering the data mixing as an extra factor in scaling laws analysis.
Using average validation loss as an indicator for assessing the performance of LMs pre-trained on mixed data sources, such as general text and code, is limited.
Thus, the average loss fails to trace performance variations in downstream tasks because changes in data mixtures can affect different capabilities of the LMs.
In contrast, FLP-M effectively leverages the domain-specific validation loss to capture the capabilities improvement in LMs, and thus can better predict the downstream performance.
In our experiments, FLP-M accurately predicts the performance of 3B LLMs across various data mixtures and the 7B LLM with 0.3 data mixing ratio with error margins within 10% for most benchmarks.

However, on TriviaQA, despite significantly outperforming FLP, FLP-M shows higher relative prediction error, ranging from 20% to 30%.
This discrepancy can be explained by the substantial performance improvement when scaling LLMs from under 1B to 3B parameters (increasing from below 12 to over 28).
In our sampling LMs configurations (see Tab. 1), we lack sufficient data points to adequately characterize the phase of accelerated performance improvement.
To better model this trend, a practical solution is to add several sampling LMs between 1B and 3B parameters.

Figure: Figure 6: We use the scaling laws function derived via FLP-M to find the optimal data mixing ratio that yields the estimated best performance on the corresponding benchmarks.
Refer to caption: extracted/5918555/figs/application/performance_race.png

## 7 Further Analysis

### 7.1 Optimizing Data Mixture Using FLP-M Scaling Laws

We demonstrate how the derived scaling laws using FLP-M can be effectively applied to optimize data mixtures, enhancing downstream performance.
We focus on 1B LMs in this analysis due to compute constraints.
For each dataset, we use the FLP-M to estimate the function that maps expended FLOPs in each data source to the downstream performance.
Then we use this function to predict performance across mixing ratios from 0 to 0.5, in intervals of 0.01.

Among all evaluation datasets, the estimated scaling laws function exhibits non-monotonic behavior on the RACE and ARC datasets, reaching its peak at mixing ratios of 0.01 and 0.22, respectively.
To verify,
we train 1B LMs with these two mixing ratios and measure their performance on the corresponding benchmarks.
The results are shown in Fig. 6.
We find that the selected optimal mixing ratio can reliably yield better performance compared to the six mixing ratios adopted for the sampling LMs, highlighting FLP-M as a practical approach for optimizing data mixtures to enhance performance on specific target tasks.

### 7.2 Ablation Study

We conduct further analysis to better understand the two stages in FLP-M.
Specifically, we compare various approaches to estimate the FLOPs-to-Loss and Loss-to-Performance curves in FLP-M.

**Table 3: Candidate analytical forms for fitting the FLOPs-to-Loss curve. Except for $C^{G}$ and $C^{C}$ representing the compute used for general and code data sources, other constants need to be estimated. The average error is computed across all domains and model types.**
| $L^{D}(C^{G},C^{C})=$ | Analytical Form | Average Error |
| --- | --- | --- |
| M1 | $(\frac{C^{G}+C^{C}}{C_{T}})^{\alpha_{C}}$ | 0.029 |
| M2 | $(\frac{C^{G}}{C_{G}})^{\alpha_{C1}}\times(\frac{C^{C}}{C_{C}})^{\alpha_{C2}}$ | 0.026 |
| M3 | $(\frac{w_{0}*C^{G}+w_{1}*C^{C}}{C_{T}})^{\alpha_{C}}$ | 0.017 |
| M4 (Ours) | $(\frac{C^{G}+C^{C}}{C_{T}})^{\alpha_{C}}\times(\frac{C^{G}}{C_{G}})^{\alpha_{C<br>_{1}}}\times(\frac{C^{C}}{C_{C}})^{\alpha_{C_{2}}}$ | 0.014 |

FLP-M: FLOPs $\rightarrow$ Loss
We experiment with several candidate analytical forms listed in Tab. 3.
We assess their performance in estimating the average pre-training loss across the entire validation set, as well as the domain-specific pre-training losses on corresponding subsets.
We present the fit curves in Fig. 13 (§E),
and the relative prediction errors for pre-training loss estimation are shown in Fig. 7.
For average pre-training loss prediction, using more complex analytical models that account for the individual impact of each data source can lead to performance degradation.
However, relying solely on the total compute for prediction (M1) can cause high prediction errors in certain domains (e.g., code) and are not stable for various mixing ratios.
More complex analytical models generally perform better in predicting domain-specific loss.
Among them, M4, the adopted model in FLP-M, provides more stable (within 2.5% relative prediction error across most domains) and overall more accurate predictions (achieving the lowest average error shown in Tab. 3).

Figure: Figure 7: The relative prediction error of average and domain-specific pre-training loss. M4 provides more stable and overall more accurate predictions for domain-specific loss (within 2.5% relative prediction error across most domains).
Refer to caption: extracted/5918555/figs/FLPMix_Loss/error/loss_prediction_error.png

Figure: Figure 8: The relative prediction error of various approaches to estimate the Loss-to-Performance curve. Neural network estimation with domain-specific loss as input achieves the best prediction.
Refer to caption: extracted/5918555/figs/FLPMix_Loss2Per/performance_gap/0.png

FLP-M: Loss $\rightarrow$ Performance
We experiment with various approaches to estimate the function that maps the pre-training loss to the downstream performance. In this study, we utilize the actual pre-training loss of target LLMs, rather than the predictive loss used in §5.
We consider the following candidates with different inputs:

- (1)
FLOPs: We adopt the analytical form used to predict the pre-training loss based on training compute (see Eq. 5), only changing the target metric to the downstream performance.
- (2)
Average Loss (Average):
We implement a linear regression model to map the average pre-training loss on the whole validation set to the downstream performance.
- (3)
Domain Loss via Linear Combination (Domain-Linear):
We apply a linear regression model to correlate pre-training loss across domains with downstream performance.
- (4)
Domain Loss via Neural Network (Ours) (Domain-Neural):
We implement a two-layer neural network to map the pre-training loss across domains to the downstream performance. The network configuration and optimization process are introduced in §5.

The fit curves are shown in Fig. 14 (§E) and the results of relative prediction error are shown in Fig. 8.
Consistent with the findings in §4, directly estimating the performance based on expended compute (FLOPs) leads to highly inaccurate predictions (FLOPs vs. Loss).
Pre-training loss serves as a more reliable metric for performance estimation, and decomposing it into domain-specific loss can further enhance prediction accuracy (Average vs. Domain Loss).
For the predictive models, using neural network estimation can better leverage the abundant data points produced by FLP-M, resulting in better performance compared to the linear regression model (Linear vs. Neural Network).

## 8 Conclusion

This paper introduces a two-stage FLP solution to predict downstream performance in LLMs by leveraging pre-training loss.
Encouraged by promising preliminary results, we propose FLP-M, a core solution for performance prediction that addresses the practical challenges of integrating pre-training data from diverse sources.
The effectiveness of FLP-M is validated through extensive experiments.

## Limitations

Our approach FLP-M is generally applicable across various data sources, yet currently, it is demonstrated only in binary cases involving code and text data due to computational constraints.
Our specific emphasis on the mixing ratio of code is deliberate, reflecting its practical significance in real-world applications.
This limitation marks a key area for future expansion.

## Appendix

Figure: Figure 9: We visualize the relation between pre-training loss and task performance for all LMs that surpass random baseline performance on the target benchmark, observing a generally linear trend.
Refer to caption: extracted/5918555/figs/scatter/ARC_challenge.png

## Appendix A Linear Relation Between Loss and Performance

We gather data points from intermediate checkpoints of all sampling LMs and visualize the relationship between pre-training loss and corresponding task performance in Fig. 9.
We observe a generally linear trend across all benchmarks, which motivates our selection of linear analytical form to characterize the mapping from pre-training loss to downstream performance.

Figure: Figure 10: The downstream performance prediction using FP (Achiam et al., ) and FLP fit curves. FLP can better predict the downstream performance of target 7B and 13B LLMs across all evaluation benchmarks, while FP’s predictions are very unstable (e.g., HumanEval, TriviaQA).
Refer to caption: extracted/5918555/figs/FLP_app/performance_prediction_ARC_challenge.png

Figure: Figure 11: The comparison to the downstream task prediction approach in Llama-3 development (Dubey et al., ). We find that initially estimating the negative log-likelihood of the target answer does not effectively predict performance based on our data points.
Refer to caption: extracted/5918555/figs/llama_compare/performance_gap_dense_31.png

## Appendix B Analytical Form to Fit FLOPs-to-Performance Curve

We also experiment with the analytical form proposed in Achiam et al. () to estimate the FLOPs-to-Performance curve:

$$ $\log P(C)=(\frac{C}{C_{M}})^{\alpha_{M}},$ (6) $$

where $C_{M}$ and $\alpha_{M}$ are constant terms to be estimated.
The fit curves are shown in Fig. 10.
We observe that FLP still consistently outperforms FP across all evaluation benchmarks.
In addition, FP can yield very unstable predictions on certain datasets, like HumanEval and TriviaQA, due to a lack of sufficient data for accurate modeling.

## Appendix C Compare with Llama-3 Approach

We compare with the Llama-3 approach for downstream task prediction (Dubey et al., ).
They suggest initially estimating the negative log-likelihood (NLL) of the target answer based on the computational cost in FLOPs, followed by using this NLL to model the task performance through a sigmoid function.
The comparison results are shown in Fig. 11.
We find that the two-stage approach proposed in Dubey et al. () fails to effectively estimate the performance based on our data points, compared to FLP.

## Appendix D MMLU Experiment

Figure: Figure 12: The performance prediction on MMLU using FLP.
Refer to caption: extracted/5918555/figs/FLP/mmlu_performance_prediction.png

Our sampling LMs, up to 3B, exhibit random performance (i.e., 25%) on the MMLU benchmark (Hendrycks et al., ).
Consequently, these models do not provide effective data points for estimation.
Accordingly, we utilize intermediate checkpoints from 7B LLMs to estimate the performance of 13B LLMs on MMLU using FLP.
The results are shown in Fig. 12, and the relative prediction error is 3.54%.
FLP can also effectively predict the performance on MMLU by leveraging intermediate LMs checkpoints that emerge on this task.

## Appendix E FLP-M : Fit Curve for Ablation Study

Figure: Figure 13: The pre-training loss prediction using various analytical forms. M4 provides more stable and overall more accurate predictions for domain-specific loss.
Refer to caption: extracted/5918555/figs/FLPMix_Loss/loss_prediction_avg.png

Figure: Figure 14: The downstream performance prediction using various approaches. The domain loss coupled with neural network estimation demonstrates the best prediction performance.
Refer to caption: extracted/5918555/figs/FLPMix_Loss2Per/arc_challenge_performance_prediction.png

The FLOPs-to-Loss fit curves are in Fig. 13 and the Loss-to-Performance fit curves are in Fig. 14.
We observe that M4 in Tab. 3 offers more stable and accurate predictions for domain-specific loss, with the combined approach of domain loss and neural network estimation delivering the best overall downstream performance prediction.

Figure: Figure 15: The downstream performance prediction using FLP and FLP-M fit curves. FLP can better predict the downstream performance of target LLMs with 7B and 13B parameters.
Refer to caption: extracted/5918555/figs/FLPdomain/arc_challenge_performance_prediction.png

Figure: Figure 16: The relative prediction error of 7B and 13B LLMs using FLP and FLP-M. FLP achieves significantly better performance.
Refer to caption: extracted/5918555/figs/FLPdomain/error/performance_gap_dense_31.png

## Appendix F Using Domain Loss in FLP

We explore the application of FLP-M during pre-training on a consistent distribution (the experimental setting described in §4),
and compare it with FLP.
The fitting curves are shown in Fig. 15 and the results of relative prediction error are shown in Fig. 16.
We show that FLP-M fails to effectively predict the performance of target LLMs when sampling LMs are pre-trained on a fixed distribution.
This ineffectiveness is attributed to the closely related domain-specific validation losses among the sampling LMs within the same training distribution, which suggests that decomposing the pre-training validation loss yields no additional information in this pre-training setting.
Thus, estimating five domain-specific loss, rather than a single average validation loss, can further increase the risk of error propagation.
Moreover, using highly correlated features as neural network inputs may lead to overfitting.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Here are a few additional questions for the authors in addition to the weaknesses above:

1. Were the pre-training datamixes used for FLP-M experiments deduped against the validation set used in FLP and FLP-M experiments? Because it might affect the scaling behaviour if there's any overlap.
2. For the comparions with Llama 3 in Section C / Figure 11, what specific sigmoidal function was used? And did the authors ensure to choose the one that results in the best fit on the sampling LMs?
3. Can the authors please correct the typos in Equation 5 and Table 3 corresponding to $C_G$ and $C_C$?
4. In Figure 2, the sampling LMs corresponding to $\leq 10^{18}$ flop scale seem to be missing. Is there a specific reason for this?
soundness: 2
strengths: - The paper tackles an important problem of building scaling laws to measure the downstream task performance, especially when we know that task-specific behaviour emerges at different scales and smaller scale LMs might not be able to accurately capture the predictive behaviour of larger models on certain tasks. The paper's two-stage approach of separating the FLOPs $\rightarrow$ Loss and Loss $\rightarrow$ Performance predictive models circumvents the emergent behaviour issue with the FLOPs $\rightarrow$ Performance power law.
- The paper provides good insights on the mixing behaviour during pre-training on general text vs code by extending the FLP approach to FLP-M, with good empirical results on deriving the optimal mixing ratios (in a controlled setting).
- The experiments and results are exhaustive and involve a range of tasks including ARC-C, BBH, Hellaswag, HumanEval, RACE, and TriviaQA.
summary: This paper introduces FLP (Flops $\rightarrow$ Loss $\rightarrow$ Performance), a two stage framework incorporating scaling laws to accurately predict the downstream performance of language models (LMs) on specific tasks by leveraging the pre-training loss. The first stage uses a power law equation to estimate the relation between flops and loss, $L(C) = \big(\frac{C}{C_N}\big)^{\alpha_N}$ by training 12 sampling LMs ranging from 43M to 3B parameters. The second stage involves using a linear function to estimate the relationship between the loss, $L$, and the task performance, $P$ [$P(L) = w_0 + w_1 * L$]. The second stage is applied carefully on those checkpoints that surpass the threshold of random performance + 5 additional performance points. The authors demonstrate better scalign law fits compared to the baseline of just using a power law function.

The paper then further extends the FLP approach to data mixing during pre-training (FLP-M) and presents an analysis on data-mixing ratios across general text and code and how mixing affects the downstream task performance, and extend the same two-stage framework of FLP, to predict downstream performance under different mixing ratios.
weaknesses: - The sharp transition in performance of TriviaQA from 1B to 3B models highlights the brittleness of the approach, where the error margins can be huge for downstream task performance prediction. And it's very hard to characterize this behaviour for a whole range of tasks that are usually used to compare various LMs.
- I don't agree with the authors' point on enhancing sample efficiency by collecting losses corresponding to intermediate checkpoints and actually creates a biased estimator for the power law operands. Moreover intermediate checkpoints exhibit transient behaviours especially corresponding to learning rate adjustments (different intermediate checkpoints exhibit different learning rate schedules).
- I think there's a major typo in Equation 5, where the denominators of the second and the third terms are identical to the numerators. It hinders the understanding of the readers and it persists in the later sections too. [Although it's not a huge weakness and I am not basing my score on this point, assuming the authors will correct it in the rebuttal phase].
- The experimental setting corresponding to the comparison with Llama-3 is not explained properly, and it's hard to believe the results from Figure 11, provided that the estimated Llama-3 405B performance was quite close to the actual performance on ARC-C, whereas in this paper it's shown to be above 25%.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Have ongoing experiments been conducted on larger-scale models？
2. How do you justify the usage of intermediate checkpoints for acquiring scaling law datapoints?
soundness: 2
strengths: A notable strength of the paper is the quality of the writing: the narrative is clear, and the experiments are thorough. Besides, the FLP-M method accurately predicting performance based on data loss from different domains, thus enhancing prediction accuracy in mixed data scenarios. Additionally, Figure 6 demonstrates that FLP-M can be used to derive the optimal data mixing ratio for training.
summary: This paper proposes two methods, FLP and FLP-M, for efficiently predicting the downstream performance of large language models. These methods achieve high-precision performance prediction.
weaknesses: 1. The authors utilize intermediate checkpoints to gather data points; however, for the same amount of FLOPs, models with different N (parameters) and D (data)  would yield distinct loss. This raises a critical question: why is it valid to use checkpoints that have not converged and are not optimized configurations to obtain data points?

2. The second drawback is a lack of novelty. Both using FLOPs to predict loss and using loss to predict downstream performance have been explored in prior work.

3. The third drawback is that the authors use a 1B model to validate the effectiveness of FLP-M scaling law for achieving an improved data mixture. However, this claim may be overstated, as 1B models often rely on guesswork for many tasks, undermining the reliability of these results.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1. **Practical Application Value** This paper introduces FLP-M, linking computational resources with LLM downstream performance. This research holds significant importance for real-world applications.
summary: In this paper, the authors manage to predict the downstream performance of LLMs according to the computaional resouces (e.g., FLOPs). Experimental results shows that, by utilizing a 3B LLM trained on a specific ratio and a series of smaller sampling LMs, FLP-M can effectively forecast the performance of 3B and 7B LLMs across various data mixtures for most benchmarks within 10% error margins.
weaknesses: 1. **Limited Scale of LM** The largest model used in this paper is only 7B, yet there are many LLMs much larger than 7B (e.g., Llama-3 70B, Llama-3 405B). From this perspective, the conclusions of this paper are limited.

2. **Limited Domains in Data Mixing** As stated in the limitations, this paper only considers the domains of text and code under Data Mixing settings. Including more domains would enhance the explanatory power of the conclusions.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: My main concerns are twofold.

1. Does the scaling law proposed in the paper have sufficient innovations compared to the classical scaling law? What are their essential differences? Please explain how the two-stage approach in FLP fundamentally differs from classical scaling laws in terms of methodology and theoretical underpinnings.

2. If the paper focuses on the model performance around the emergent point, is a linear description really suitable? Should other nonlinear descriptions be considered, such as the sigmoid function, when considering scaling near the point of near-perfect accuracy? Is there any comparation?
soundness: 2
strengths: 1. The paper identifies the issue of discontinuous performance when models approach the emergent edge, which is difficult to address with classical scaling laws, and proposes a method to resolve with the continuous variant ------ loss.
2. FLP creates more data points for fitting the scaling la, potentially making the fitted curve more generalizable.
3. FLP-M is introduced for data mixtures, providing a more accurate prediction by considering the different impacts of code and general text on downstream tasks.
4. The paper conducts extensive experiments to support its claims.
summary: The paper introduces FLP to address the limitations of classical scaling laws, which fail to accurately predict performance when small models perform poorly on evaluation tasks, approaching random sampling. FLP leverages the scaling law of FLOPs to predict pre-training loss and uses this loss to predict downstream performance. FLP successfully predicted the performance of 7B and 13B models across six tasks using a series of language models up to 3B.

Based on FLP, the paper further introduces FLP-M, which aims to predict downstream performance trained with various mixtures of general text and code.
weaknesses: 1. In section 3.2 Loss->Performance, there is a strong assumption that loss and accuracy have a linear relationship. Firstly, in all generative tasks shown in Figure 9, the linear relationship between loss and metric is not evident. The authors should provide more explicit statistical indicators to prove this linear correlation. Additionally, in the classification tasks shown in Figure 9, the relationship between loss and accuracy also encounters deviations near the emergent point, indicating that FLP does not completely bypass this issue but only circumvents it in the Flops -> Loss process.
2. A simple w_1*L+w_0 is not fundamentally different from classical scaling laws.
3. FLP-M only considers code and general text, while data mixtures typically need to consider at least five domains, including common crawl (cc), academic, books, encyclopedias, and code.
4. If the paper considers the situation around the emergent point in benchmarks, it lacks a discussion on the scenario when the model approaches near-perfect scores on a particular benchmark.