Establishing Task Scaling Laws via Compute-Efficient Model Ladders
Authors: Akshita Bhagia
Jiacheng Liu
Alexander Wettig
David Heineman
Oyvind Tafjord
Ananya Harsh Jha
Luca Soldaini
Noah A. Smith
Dirk Groeneveld
Pang Wei Koh
Jesse Dodge
Hannaneh Hajishirzi
Venue: COLM 2025 (2025)
Keywords: scaling law
model ladder
downstream tasks
TL;DR: We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting.

Abstract: We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting. Standard power laws for language modeling loss cannot accurately model task performance. Therefore, we leverage a two-step prediction approach: (1) use model and data size to predict an intermediate loss, then (2) use it to predict task performance. We train a set of small-scale "ladder" models, collect data points to fit the parameterized functions of the two prediction steps, and make predictions for two target models: a 7B model trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder models only costs 1\% of the compute used for the target models. On four multiple-choice tasks formatted as ranked classification, we can predict the accuracy of both target models within 2 points of absolute error. We find that tasks with higher prediction error also have higher variance in the metrics over model checkpoints. We also contrast multiple design choices for predicting accuracy, and present recommendations for extending our method to new models and tasks.

Published as a conference paper at COLM 2025
Establishing Task Scaling Laws via Compute-Efficien Model Ladders
Akshita Bhagia*t
Jiacheng Liu
Alexander Wettig*
David Heineman'
Oyvind Tafjord*
Ananya Harsh Jha
Luca Soldaini*
Noah A. Smith t
Dirk Groeneveld*
Pang Wei Koh
Jesse Dodge
Hannaneh Hajishirzi
+ Allen Institute for Al "qqual contribution.
University of Washington
Princeton University
|akshhtab,,jaacheng])
org
Abstract
We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting. Standard power laws for language modeling loss cannot accurately model task performance. Therefore, we leverage two-step prediction approach: (1) use model and data size to predict an intermediate loss, then (2) use it to predict task performance. We train set of small-scale "ladder" models, collect data points to the parameterized functions of the two prediction steps, and make predictions for two target models: 7B model trained to 4T tokens and 13B model trained to 5T tokens. Training the ladder models only costs 1% of the compute used for the target models. On four multiple-choice tasks formatted as ranked classification, we can predict the accuracy of both target models within points of absolute error. We find that tasks with higher prediction error also have higher variance in the metrics over model checkpoints. We also contrast multiple design choices for predicting accuracy, and present recommendations for extending our method to new models and tasks.
190M (fitted)
750M (fitted)
7B-4T (target)
13B-ST (target)
370M (fitted)
1.3B (fitted)
7B-4T (predicted)
13B-5T (predicted)
Step
Step
Chained
1.50
Task RC accuracy
Task RC accuracy
0.6
1.25
0.5%
0.3%
0.4
1.00
0.4
0.3
0.75
1.2%
10th
10a1
10
10
103h
10tH
10
10:5
Takans (D)
Task loss
Tokons (D)
Predicting MMLU accuracy with our method. We use model size N and data size D to predict "task loss" on MMLU (step 1), and then use this task loss to predict task accuracy in ranked classification format (step 2). The chained plot shows end-to-end prediction from (N, D) to task accuracy. The functions in step and are fitted on data points collected from ladder models (markers colored in red, orange, green and cyan); 7B- 4T and 13B-5T are the target models for which make predictions. We report relative prediction error in the plot next to each target model point.
Figure 1:
1

---

Published as a conference paper at COLM 2025
1 Introduction
Language models (LMs) are expensive to train. Building a good LM requires a wide range of experimentation over data and architecture choices. Scaling laws, which use the performance of smaller models to predict the performance of a larger model without actually training it, enable more efficient resource allocation for such experimentation. Further, modern LMs are compared based on their performance on downstream tasks, rather than perplexity. Certain downstream tasks (e.g,, HellaSwag) are also often used as development benchmarks for creating training recipes and good data mixtures for training large LMs (OLMo al., 2024; Li et al., 2024). Scaling laws for downstream tasks are, therefore, important for LM pretraining.
Prior work has shown results on predicting the average top-1 error over many tasks (as opposed to predicting task accuracy directly) (Gadre et al., 2024), or the accuracy for only one specific task - (Dubey et al., 2024), using at least 5% of the compute required to train the target models. Additionally, these require developing a testbed of many small models to first find compute-optimal models, which adds significant compute costs. Predicting LM performance efficient way (e.g,, without finding compute-optimal models) remains an open problem.
range of individual downstream tasks in a computationally
Here, we tackle the challenge of predicting individual task performance of LMs as function of model size and training data size. In particular, we predict the performance of 7B and 13B models. We focus on range of multiple-choice tasks and predict the task accuracy for problems written in the ranked classification (RC) format. We use a set of small models (190M to 1.3B non-embedding parameters), which we train for varying durations (Ix to 10x Chinchilla optimal data size) to develop scaling laws. Training these fixed-size ladder models costs only 1% of the compute of the two target models combined.
We employ a two-step approach to (1) use the number of model parameters N and training tokens D (the input features) to predict a task-specific loss (an intermediate feature), and then (2) use this task loss to predict accuracy. We experiment with different design choices for the intermediate feature, as well as input features, and apply our method on 8 selected tasks from OLMES (Gu et al., 2024). We quantify the predictability of the tasks based on the variance of target metrics for a representative small model over the last few training checkpoints. Finally, we provide recommendations for developing downstream scaling laws for new models, and selecting a design choice for a given task. From our recommendations, using a task-specific loss as the intermediate feature, points for our target models on four tasks (MMLU, HellaSwag, PIQA, SocialIQA), and an average absolute error of 4 points across both target models and all tasks.
achieve an absolute error of < 2
2 Setup
We aim to predict the task performance of LMs with an arbitrary training scale the combi- nation of model size (N) and number of training tokens (D). As recent LMs are overtrained (Dubey et al., 2024; Li et al., 2024; Bai et al., 2023), we do not constrain N and D to stay close to the compute-optimal regime (Hoffmann et al., 2022). We validate our predictions on the OLMo 2 models after stage pretraining and before stage annealing (OLMo et al., 2024); 7B model trained to 4T tokens ("7B-4T") and 13B model trained to 5T tokens ("13B-5T"), both trained from scratch with the same data mixture. These are target models.
2.1 Ladder models
To predict the performance of large models, we extrapolate from data points collected from training many small models ("ladder models"). These models have the same architecture and are trained with the same data mix as the target models. This is useful for guiding pretraining development, as data mixtures and modeling decisions can be tested in controlled setting. The models span a relatively wide range of model size and training data size, while collectively costing a small fraction of compute used to train the target models.
2

---

Published as a conference paper at COLM 2025
Table 1: Model setup. Up to 1.3B are ladder models; 7B-4T and 13B-5T are target models.
190M
370M
760M
1.3B
7B-4T
13B-5T
Model (N)
190,354,176
371.262,464
758,220,288
1,279,395,844
6,887 5787,575,552
13,202,396,160
1xC data size (D)
3,8,7,083,520
7,425,249,25 ,80
15,164,405,760
25,587,916,800
Batch (sequences)
192
320
384
1,024
2,048
Batch size (tokens)
524,288
786,432
1,310,720
1,572,864
4,194,304
8,388,608
Training steps (1xC)
7,272
9,452
11,580
16,279
Peak LR
9.7 10
10
10
5.2
x 10
10
Warmup steps
363
472
378
813
2000
1000
Dimension
768
1,024
1,536
2,048
4,096
5,120
Num heads
12
16
16
16
32
40
Num layers
16
16
32
40
MLP ratio
8
8
5.375
Table 2: Example for RC format from HellaSwag (Zellers et al., 2019). Tokens on which task loss is computed are marked in green.
Original problem
A is outside with bucket and The dog is running around trying to avoid bath. She
rinses the bucket off with soap and blow dry the dog's head. uses a hase to keep it fron getting seapy- gety the dog wet, then it away again.
gets inta a bath tub with the dog.
Task loss calculation
Question: woman is outside with bucket and dog. The dog is running around trying avoid bath. She then it runs away again.
Answer: gets the
We define four model sizes N E { 190M, 370M, 760M, 1.3B } (considering only non- embedding parameters) by varying the width and depth of the transformer. We use D = 20 N as the Chinchilla optimal setting (Hoffmann et al., 2022) (denoted "IxC"), and train each model with number of tokens E { 1xC, 2xC, 5xC, 10xC }. In total, we train 4 x 4 = 16 models. We save intermediate checkpoints every 200 steps for 1xC and 2xC runs, every 500 steps for 5xC runs, and every 1000 steps for 10xC runs. Table 1 lists the configurations of each model size.
The target models are trained with cosine LR schedule with linear warmup, where the LR decays to 10% of the peak LR over horizon of 5T tokens. We match this in the ladder models, where the decay horizon adjusted to match the training data size of each model. Unfortunately, the OLMo 2 7B model was only trained to 3.9T tokens in stage pretraining, whereas the cosine scheduler was set to 5T tokens. To account for this incomplete training, we train this model with an additional 50B tokens where we linearly decay the LR down to 10%. Our target 7B-4T model is thus trained on a total of 3.95T tokens, where the linear decay instead of the slower cosine decay allows
use fewer tokens.
For the ladder models, we set the peak LR, batch size, and warmup steps by extrapolating from the configuration of 7B-4T using the method introduced in Porian al. (2024). All other configurations follow from the 7B-4T model.
Compute cost. Using C R 6ND to estimate the compute (Kaplan et al., 2020), the total amount of compute used for training the ladder models is 5.2 x 10 FLOPs. This is only 3.2% of that used for training the 7B-4T model (1.6 x 20 FLOPs), 1.3% of the 13B-5T model (3.9 x 1023 FLOPs), and less than 1.0% of both target models combined.
2.2
Intermediate features and accuracy
Task loss. We define task loss the negative log-likelihood of the correct answer sequence, divided by its length in bytes, also known as the bits-per-byte (bpb) metric. Table 2 shows the problem formatting for computing task loss an example. We use bpb instead of normalizing by number of tokens reduce the impact of the tokenizer.'
TaskCE. Task cross-entropy loss which accounts for incorrect answers (SC.2).
similar "normalized NI.I per char" (Dubey al., 2024).
3

---

Published as a conference paper at COLM 2025
370M (fitte)
760M i1tbed)
30 (Rited)
70-4T
(precicted)
130 ST (gredicted)
2.5%
ARC-Challenge
0.5
0.8
0.6
17.5%...
4%
0.4
0.6
0.5
0.7
0.4
0.6
0.3
0.5
1011
10
10
103
1010
10t1
1012
1013
10
10
103
1037
10
10
1012
1013
PIDA
Commonsense@A
Social I0a
OpentBookgA
1.1%
0.8
0.5
0.80
0.7
1.2%
0.80
+2.0%
10.6%
7.8%
0.75
0.6
0.55
0.70
0.50
0.5
0.65
0.45
0.4
103t
10tt
1022
103
1010
10tt
1012
1000
1010
10
10ah
10
1030
10t1
1012
2013
Takers
Tokons
(D)
7B-4T
13B-5T
Task
Split Examples
Pred
Actual
Error
%Error
Pred
Actual
Frror %Error
MMLU
test
14,042
0.6
51.6
0.7%
HellaSwag
val
10,042
82.5
81.3
1.2
1.4%
85.3
83.2
2.1
2.5%
ARC-Challenge
test
1,172
51.5
61.9
16.9%
52.7
63.8
17.5%
ARC-Easy
test
2,376
76.6
84.6
8.0
9.4%
77.2
87.2
10.0
11.4%
PIQA
val
1,838
81.2
82.0
0.8
1.0%
82.1
R3.0
0.9
1.1%
CommunserseeA
val
1,221
75.7
72.6
3.1
4.2%
77.6
74.1
3.5
4.7%
Social IQa
val
1,958
58.7
59.9
12
2.0%
54.9
61.6
1.7
2.7%
OpenBookQA
test
500
44.2
49.4
5.2
10.6%
44.9
48.6
3.7
7.8%
Average
3.8
5.9%
4.2
6.1%
Figure Task accuracy prediction for the target models using task loss as the intermediate feature.
1xC; + = 2xC; 5xC; * 10xC. Prediction error is next to the target.
LM loss.
Standard language modeling loss on the C4-en validation set (Raffel et al., 2019).
Task accuracy. Ranked classification (RC) and multiple-choice (MC) are two different formats to pose multiple-choice problems.) In RC, the predicted answer is the one with the minimum task loss. In MC, all the answer choices are included in the prompt, and the predicted answer is the answer code (e.g., A, B, C, etc.) with the smallest loss (Gu et al., 2024). Here, we focus on the RC format.
Following Gadre et al. (2024), we compute the relative error for the task accuracy to evaluate the goodness of prediction, defined as
Evaluation.
prediction actuall actual
Relative Error
100%.
2.3 Task selection
We include the following 8 tasks from the OLMES evaluation suite (Gu et al., 2024): MMLU (Hendrycks et al., 2021), HellaSwag (Zellers et al., 2019), ARC-Challenge (Clark al., 2018), ARC-Easy (Clark et al., 2018), PIQA (Bisk et al., 2020), CommonsenseQA (Talmor et al., 2019), Social IQa (Sap et al., 2019), and OpenBookQA (Mihaylov et al., 2018). We exclude BoolQ (Clark et al., 2019) and Winogrande (Sakaguchi et al., 2020), as the task loss and accuracy are noisier for these tasks. $5 discusses task predictability further. We use the test set where possible, and fall back to the validation set otherwise. For all tasks, we use the 5-shot setting provided in OLMES. Figure shows the task statistics.
²RC reliably measures progress for models across wide range of scales, whereas MC capability does not emerge until the model has several billion parameters.
4

---

Published as a conference paper at COLM 2025
190M (nttee)
370M (1
763M (fitte)
(ftted)
70-4T
70 (predicted)
138-5T
130-37 30-3T (predicted)
HellaSwag
ARC-Challenge
(1.18%)
1.2
0.9
1.2
1.0
1.0
0.8
1.0
0.8
0.8
0.7
0.8
5.0%
0.6
16.0%
0.2%
10t0
10
10
10a2
103c
1011
10a2
10
1020
10
10a2
10th
1022
1011
10
10
PIDA (0.315%
Commonsense@A (1.05%)
Social IQa (0.54%)
OpenBookOA (0.59%)
1.3
1.50
1.4
1.1
1.25
1.2
1.6
1.0
1.00
1.4
0.9
1.7%%
1.0
10.55
1024
10
10-3
10
1022
10
104
10
1013
10t
1022
10
Doars (9)
Toeens (D)
Tikans IDI
Bienns (D)
Task loss vs training scale (N, DN, with fitting on the power function in Equation 1. 2xC; 5xC; * 10xC. We report the average relative fitting error in parentheses after the task name, and prediction error next to the target model point.
Figure 3:
1xC; +
3 Method
We break down task accuracy prediction into two steps: 1) predicting the intermediate feature (we use task loss to illustrate this), and 2) using it to predict the task accuracy.
3.1 Step 1: Use (N,D) to predict the intermediate feature
We consider three intermediate features: task loss, taskCE loss, and LM loss (defined in $2).
As proposed by Hoffmann et al. (2022) and followed by others (Muennighoff et al., 2023; Gadre et al., 2024; Zhang et al., 2024), the LM loss of a model on a held-out eval set can be modeled power function with respect to N and D:
L(N, = A/N* + B/D +E,
(1)
where A, B, a, B, E are parameters to fit.
We postulate that the same functional form applies to these losses. We validate this assumption by looking at the goodness of function fitting.
Function fitting. We take the loss value of the final checkpoint of each ladder model {(N,, L,) (where n = 16), and use this data to fit the parameters of Equa- tion 1. We fit separate set of parameters for each task. Following Hoffmann et al. (2022), we minimize the Huber loss between the logarithm of predicted and actual loss: 1 E Huber, log L(N,, D;) - log Li), where d = 10-3, We optimize A and B in log space we apply transformation
log A, log B and optimize b, B, E). We use the L-BFGS-B optimizer as implemented in scipy. menimize O with constraints A, B, a, B, E 2 0; with these constraints, the function convex, and thus L-BFGS-B is guaranteed to converge.
We take the average loss over the last 5 checkpoints of each ladder model to reduce noise.
Results preview. Figure 3 shows the function fitting of step 1 on the ladder models, and the prediction of task loss as the intermediate feature for the target models. The power function gives an average relative fitting error ranging from 0.2% to 1.2%, which suggests that Equation is good functional form to describe task losses. We have relative prediction error within 3% on MMLU, HellaSwag, PIQA, and OpenBookQA. The method underestimates the loss for CSQA, and overestimates on ARC-Challenge, ARC-Easy, and Social IQa. On
3To test if this function is convex, confirm that the Hessian matrix of second derivatives is positive semi-definite everywhere.
5

---

Published as a conference paper at COLM 2025
(ntbed)
370M (fited)
TE0M
1:38
70-47
(predicted) 138-5T
136 (predicted)
MMLU (Pitting error: 0.59%)
HollaSwag (Fitting errer: 1.10%)
ARC Challiegee (Fitting errer: 2.19%)
Eac-Easy (Fitting error: 1.10%)
BC socusec
0.6
122%
1.0
0.5
0.6
0.6
0.4
0.4
0.3
1.5
0.6
1.2
0.5
PIDA (Fitting error: 0.49%)
CommonsensQA (Fitting error: 1.74%)
Social IQa (Fitting error: 0.87%)
OpenBookgA (Fitting error: 2.50%)
0.6
Toak 3C actuecy
0.8
2.3%
00%
0.7
0.5
8.%
0.6
0.7
2.5%
79
0.4
0.3
0.0
0.4
0.2
0.8
1.0
1.2
1.4
0.5
1.0
1.5
2.0
0.8
1.0
1.2
1.4
1.0
1.25
1.50
1.75
2.00
Taak 1088
Tave loss
Taak lo6s
Task Inos
Figure 4: Task RC accuracy vs task loss, with fitting on the sigmoid function in Equation 2. The prediction error next to the target model point. The shaded area represents the prediction intervals for the fitted function.
target models, across all tasks get average relative error of 5.2% for 7B-4T, and 6.6% for 13B-5T. Results for the other intermediate features are discussed SC.2 and $C.3.
In SC.1, we compute-FLOP's input variable instead of (N, D), and observe higher fitting errors for all tasks. We also note that FLOPs cannot distinguish between compute- optimal and overtrained models.
3.2 Step Use intermediate feature to predict accuracy
Following Dubey et al. (2024), we model the mapping from the intermediate loss to accuracy with sigmoidal function:
a
Acc(L) = 1 K(L La)
+
(2)
where a, b, k, Lo are parameters to fit.
The choice of sigmoidal functional form is motivated as follows: weak model has high task loss and random task accuracy, and a strong model has low task loss and high task accuracy that saturates at 100%. We observe (as shown in Figure 4) that the (L, Acc;) points collected from different ladder models tend to fall on a shared sigmoidal curve; this applies to both intermediate and final model checkpoints.
Function fitting. We use the intermediate loss and accuracy values from both final and intermediate checkpoints of the ladder models {L, Acc;) (where m ~ 1400) to fit the parameters of Equation 2. We fit a separate set of parameters for each downstream task. We minimize the I.2 loss between the predicted and actual accuracy: m Em (Acc(L,) - Acc,) wwwpmiiee.cure.....oo
We non-linear least squares implemented by equation, as sigmoid functions are not convex.
to fit this
The variation from checkpoint checkpoint a moving average on the intermediate loss and task accuracy over all checkpoints of each training run, with window size of 5. We also discard the checkpoints from the first 10% of each training run as these are quite noisy, and add an extra data point (L = 0.0, Acc = 1.0) to the training pairs. This helps avoid the cases when the fitting function degenerates for very noisy data, and the ladder models are too small to do well on certain tasks.
be high. To smoothen the noise, we apply
Results preview. Figure 4 shows the function fitting of step 2 on the ladder models, and the prediction of task accuracy for the target models using their actual task loss as the intermediate feature. For all tasks, all data points fit well with shared sigmoidal function,

---

Published as a conference paper at COLM 2025
Table 3: Comparison of design choices. Upper: Average step prediction error. Lower: Average chained prediction error. For MMLU, using task loss yields the lowest prediction error. We observe higher errors for ARC-C, ARC-E, and OBQA with task loss, which aligns with our variance analysis in $5, and that C4 as the intermediate loss works better for them.
7B-4T
13B-5T
Design choice
MMLU HS ARC-C ARC-E PIQA CSQA SIQu OBQA|MMLL HS ARC-C ARC-E PIQA CSQA SIQa OBQA
[N, D) (83)
13.3%
11.7%
1.3%
FLOPs task loss (g.1))
4.3%
2.1%
24%
5.3%
2,0%
12.5%
5.%% 0.5%
5.2%
2.6%
3.5%
7.0%
2.6%
18.8%
5.7%
1.4%
78-4T
138-5T
Design choice
MMLU
HS
ARC-C ARC-E PIQA
CSQA SIQa OBQA/MMEUU
HS
ARC-C ARC-E PIQA
CSQA
SIQa
OBQA
(N, D) ($3)
13%
944%
0.7%
11.4%
4.7%
2.%%
TaskCI (SC.2)
18.3%
7.2%
21.3%
5.4%
3.1%
2.6%
0.7%
7.1%
20.2%
10.5%
19.5%
6.2%
2.9%
2.7%
1.2%
0.2%
C4 loss (8C.3)
2.2%
4.5%
1.4%
1.2%
0.7%
5.8%
6.2%
2.0%
5.0%
5.7%
3.9%
1.9%
1.5%
7.0%
7.6%
10.4%
Single (CC.)
5.8%
0.5%
21.7%
6.1%
0.7%
5.1%
5.4%
9.0%
7.1%
0.8%
22.6%
7.9%
0.6%
6.6%
6.8%
4.%%
with the average relative filling ervor ranging from 0.4% to 2.6%. We get within 3% relative prediction error on MMLU, HellaSwag, PIQA, and CommonsenseQA. Overall, we get an average relative error of 3.7% for 7B-4T, and 3.5% for 13B-5T.
3.3 Chaining the two steps
We chain the two steps by first predicting the intermediate loss with the fitted function in step 1, and then inserting it into the fitted function in step 2 to predict the task accuracy.
Results preview. Using task loss the intermediate feature, we get average absolute error of 3.8 points for 7B-4T, and 4.2 points for 13B-5T. In SC.5, we also show the results of predicting the task accuracy directly from (N, D) in single step, and observe higher average prediction errors for both target models (> 30 points).
4 Results
Table 3 compares prediction enrors for all design choices.
Task loss. Figure 2 shows the chained prediction results the target models using task loss as the intermediate feature. On four tasks - MMLU, HellaSwag, PIQA, and Social IQa predict the accuracy within overestimate the task loss in step 1, and underestimate the task accuracy in step 2. In $5, we analyze the ability of the ladder to predict task performance by considering variation between checkpoints. We find that our results here track with the variation analysis (ARC-E and ARC-C display higher variance).
absolute error of 2 points. For ARC-C and ARC-E, we
Using TaskCE as the intermediate feature results overall higher prediction errors for several tasks (including MMLU and HellaSwag; see 56 on why these tasks are especially important). Full results using TaskCE are shown in SC.2.
TaskCE.
LM loss. Interestingly, the loss on C4-en validation set is a good predictor of task accuracy for several tasks. One possible explanation for this is potential domain overlap between the task and the validation set. We discuss pros and cons of using LM loss in S6. Full results using LM Loss are show in SC.3.
Figure 16 compares absolute and relative errors for all three intermediate features.
5 Analysis: Task predictability with the ladder
Some tasks are inherently more challenging to predict reliably with the ladder models; for example, a test set with an inadequate sample size, low-quality test instances or questions that are too difficult for small models. We anticipate three sources of prediction failure:
High variance in intermediate loss; less reliable data points for function fitting.
7

---

Published as a conference paper at COLM 2025
1.36 (Itermedialo chackpoints)
1.38 (fhea checkpoints)
MMLU (Loss relative SD 0.26%)
MMLU (Accuracy relative y0 0.28%)
OpenBookOA
OpenBook
(Loss relative SD 0.34%)
(Accuracy relative SD 2.51%)
Took RC acsurnry
0.36
RC acsurccy
0.40
Tosk loss
1.05
0.35
1.45
0.38
Tock
0.34
0.35
1.00
1.40
0.33
1.00
1.05
1.40
1.45
Figure 5: Relative SD the final 10 checkpoints (SD10) for the 1B-10xC ladder model on MMLU and OpenBookQA. OBQA metrics appear noisier than MMLU, resulting in higher SD10- Tasks with high intermediate checkpoint noise indicate higher prediction error (Table 4). Results across all tasks are in Figure 6.
Table 4: Absolute and relative standard deviation of last 10 training checkpoints (SD10) for the 1B-10xC model on task loss and accuracy, and relative error for predicting the 7B-4T model on task loss (step 1 only), accuracy (step 2 only) and chained accuracy (step 1 and step 2). We observe that tasks with above-average SD for 1B-10xC, highlighted in red, tend to have high prediction errors for the 7B-4T model. In particular, we observe a strong
Pearson correlation between loss SD and accuracy prediction error (r 0.821, = 0.004).
Std. dev. of final 1B checkpoints
Predictions for 7B-4T
Loss
Loss
Accuracy
Accuracy % SD10
Loss Error
Accuracy % Error
Chained
Task
SD10
% SD10
SD10
% Error
Winogrande
0.0115
0.75
0.0048
0.77
4.5
23.7
BoolQ
0.0068
1.76
0.0186
2.86
11.4 %
4.3 %
1.8
CommonsenseQA
0.0056
0.56 %
0.0035
0.55 %
11.7 %
1.0 %
4.2 %
OpenBookQA
0.0047
0.34 %
0.0095
2.51 %
1.3 %
8.2 %
10.6 %
ARC-Easy
0.0045
0.66
0.0043
0.61 %
13.3 %
4.5 %
9.4 %
ERC-Challenge
0.0037
0.40
0.0040
1.00
7.6
16.9 %
MMLU
0.0026
0.26 %
0.0010
0.28 %
1.3 %
0.3 %
1.3 %
Social IQa
0.0024
0,23 %
0.0U32
0.61 %
4.3 %
4.7 %
2,0 %
PIQA
0.0019
0.19 %
0.0024
0.31 %
2.0 %
2.3 %
1.0 %
HellaSwag
0.0007
0.09
0.0016
0.25
1.1 %
1.4
High variance in task accuracy; higher spread of prediction targets.
Random-chance task accuracy in smaller ladder models due to task difficulty.
We try to determine which tasks will exhibit high prediction errors, using task loss the intermediate feature. We use the intermediate checkpoints of the largest ladder model (IB-1xC) measure the noise for task loss and task accuracy. Failure due random-chance accuracy is discussed when predicting multiple-choice tasks in SB.2.
Variance analysis. We compute the standard deviation of the last training checkpoints of 1B-10xC (SD,). We also compute relative SD (also called the coefficient of variation) to compare between tasks:
SD (final n checkpoints)
Relative SD, Mean (final n checkpoints) X 100%
(3)
A task where ladder models exhibit higher standard deviation across adjacent evaluated checkpoints potentially indicates higher prediction error for the target models. To illustrate SD,, we show the intermediate checkpoints for the largest ladder model on OpenBookQA and MMLU in Figure 5, where we find that SD10 captures the apparent noise between adjacent training checkpoints.
We present 1D for the 1B-10xC model Table 4, alongside prediction errors for the 7B-4T model. Benchmarks with low SD10 (e.g. MMLLU and HellaSwag), also have low prediction
8

---

Published as a conference paper at COLM 2025
errors for the target, indicating these tasks are easier for the ladder to predict. We also find that Loss SD10 is correlated with step 2 accuracy error (Pearson r = 0.821, p 0.004 for 7B-4T and r 0.855, p 0.002 for 13B-5T). Thus, SD can be reported alongside predictions to explain which benchmarks may have high error before running the target model.
6 Discussion and Recommendations
In $4, we observed that different design choices work well for different tasks. Here, we discuss factors that affect final predictions, and present some guidelines develop task scaling laws for a new overtrained model or new task (Figure 16 illustrates these findings):
Target variance. Variance analysis is useful for determining if it is feasible to predict the target metrics (both intermediate loss and accuracy). For tasks with high variance, especially for task accuracy, we generally expect to see lower predictability. High variance in a particular intermediate loss can potentially be mitigated with alternative intermediate features. For new tasks, variance analysis should be conducted on the largest ladder model that can be trained with available compute (we used 1B-10xC).
Choice of intermediate features. is important for the intermediate feature to be both 1) predictable, and 2) a good predictor for task accuracy. Tasks with a noisy mapping between their intermediate feature and accuracy (e.g., ARC-C, ARC-E and OpenBookQA) may have compounded errors in the two-step prediction. For ARC-C and OpenBookQA in particular, their high variance for both the intermediate task loss and accuracy make them challenging to predict with the ladder. HellaSwag and PIQA, the tasks where we observe the least variance, are the easiest to predict across all design choices, including the combined single step approach. We also note that while LM loss on C4 is a reasonable predictor of task accuracy for several tasks in our setup, this may not hold for other novel downstream tasks on different domains, As a general rule, using a task-specific loss may work across a wider range of downstream tasks.
Number of task instances. Tasks with fewer instances show inconsistent results. E.g. OpenBookQA has only 500 instances and has high prediction error for most design choices. Even using LM loss as the intermediate feature, while the prediction error is low, the fitting error is actually high (2.35%) as shown in SC.3, indicating randomness, On the other hand, MMLU and HellaSwag have large sample sizes 5x larger than the other tasks. Given their low sample variance and noise around the prediction target, we expect that their errors are most representative of the true difficulty of predicting downstream performance, so even though taskCE and task loss perform comparably for several tasks, task loss is better considering task size. If the goal is to select a single design choice that works for all tasks, the design choice should be validated on tasks with large number of instances.
Robustness of the ladder models. SD we predict the task accuracies for larger OLMo 2 target model: 32B-6T, using the same ladder models (0.45% of the target model compute). While the absolute errors are higher, they still follow the same trend as the smaller target models (lower variance tasks exhibit relatively lower prediction errors), indicating that the same ladder can reliably be used to estimate the performance of much larger models.
Related Work
Scaling laws for language modeling. Kaplan et al. (2020) and Hoffmann et al. (2022) were among the first to postulate the functional form of LM losses power function of model parameters and data size. The Chinchilla equation, in particular, has become the basis of many subsequent scaling law works (Muennighoff et al., 2023; Gadre et al., 2024; Zhang et al., 2024). This line of work focuses on predicting the LM loss (rather than downstream performance) on held-out set with similar distribution as the pretraining data, but not on downstream tasks which is more important and challenging.
Scaling laws for downstream tasks. Scaling laws for downstream tasks have been ex- plored in Gadre et al. (2024). Instead of directly predicting accuracy for individual tasks,

---

Published as a conference paper at COLM 2025
they compute the average top-1 error over 17 LLM-foundry (MosaicML, 2024) evaluation tasks as a function of cross entropy loss on C4 (Raffel et al., 2019). Dubey et al. (2024) uses two-step prediction to first map the the training compute to the negative log-likelihood of the correct answer for a single task in an evaluation benchmark, and then relate the log-likelihood to the task accuracy. Unlike our work, which uses fixed ladder of small models, they rely on first finding compute-optimal models. Chen et al. (2024) also employs two-stage approach for predicting downstream performance, but uses the pre-training loss instead of a task-specific loss as the intermediate step. Isik et al. (2024) studies scaling laws in the finetuning setup, for machine translation tasks. Polo al. (2025) leverages models of different families to predict downstream performance, but this leads to less accurate predictions for guiding pretraining development for a specific family of models.
Conclusion and Future Work
We develop model ladders and task scaling laws to predict downstream task performance of overtrained LMs (7B-4T and 13B-5T). Specifically, our contributions include predicting individual task performance for a range of tasks with multiple design choices; using a fixed set of ladder models (rather than using large set of models to find compute-optimal models for each size); and using a small compute budget (1% of target compute). We also conduct variance analysis determine task predictability, and provide recommendations for picking good design choices.
In future work, we hope to explore ways to improve task predictability, such as increasing the size of evaluation sets, alternative evaluation formats, reducing the impact of randomness, etc. We also hope to extend our prediction method tasks the multiple-choice (MC) format to more accurately reflect the capabilities of larger LMs. MC accuracy is harder to extrapolate from our smaller ladder models; we show some preliminary results $B.2. Finally, we also hope to validate our method on larger models (70B parameters and beyond).
References
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wen- hang Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, K. Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Yu Bowen, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xing Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiao- huan Zhou, and Tianhang Zhu. Qwen technical report. ArXiv, Abs/2209.16609 2023. URL https://api. semanti cschoolar org/CorpusID :0:0oppuuID::2114555.... 263134555.
Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05);7432-7439, Apr. 2020. doi: 10.1609 10.1609/aaai.v34410 aaai.v34105.6299 URL
https://ojs. aaai httpe..ppp/AAA//aaatieeeee.e
Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji. Scaling laws for predicting downstream performance in Ilms. 2024. URL https: //manti htt.://wnn..cc....oooooooooot//a orgg/Corpus1D::23333377
Christopher Clark, Kenton Lee, Ming-We Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes no questions. pp. 2924-2936, Minneapolis, Minnesota, June 2019. doi: 10.18653/v1/N19-1000 1300. URL N19-1300.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, 2018. URL http://arxx: http:///xxxvvorrgg..o// www.:oorgg/////////e 05457.
10

---

Published as a conference paper at COLM 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Cantón Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Carcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab A. AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriele Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guanglong Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan/ Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Eytimov, Jade Copet, Jaewon Lee, Jan Laurens Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Ju-Qing Jia, Kalyan Va- suden Alwala, K. Upasani, Kate Plawiak, Keqian Li, Ken-591 neth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Made- line C. Muzzi, Mahesh Babu Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melissa Hall Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri S. Chatterji, Olivier Duchenne, Onur cCelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasić, Peter Weng, Prajjwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Chandra Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yiqian Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zhengxu Yan, Zhengxing Chen, Zoe Papakipos, Aa- ditya K. Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adi Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ranchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Fisenman, Azadeh Yazdan, Beau James, Ben Maurer, Ben Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Shang- Wen Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Flaine Montgomery, Fleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzm'an an, Frank J. Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory G. Sizov, Guangyi Zhang, Guna
11

---

Published as a conference paper at COLM 2025
Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Han Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Moly- bog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kaixing(Kai) Wu, U KamHou, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, A Lavender, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Timpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bon- trager, Pierre Roux, Piotr Dollár, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govin- daprasad, Sumit Gupta, Sung-Bae Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Andrei Poenaru, Vlad T. Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xia Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Ilama herd of models. ArXiv, 0///2440.227788 2024. URL https: api. semanticscholar wwwphhholarrrgg/oorrrooee org/Corpusi[D 271571434.
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean-Pierre Mercat, Alex Fang, Jeffrey Li, Sedrick Scott Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt. Language models scale reliably with over-training and on downstream tasks. ArXiv, abs/2403.0880 08850,, 2024. URL https://api. semanticscholarr wwwchoaar...rg//oorssll.ooooee wrg/CorppsII: BBB/CCCrPSIDD:223799144 268379614.
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A standard for language model evaluations, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre. Training compute-optimal large language models. Arxiv, abs/2203.155566, 2022. URL https: //api. semanticscholar org wwwp:/aarrrrg/oooooeeeee http//oppuID:2277700 247778764.
12

---

Published as a conference paper at COLM 2025
Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, Zhiyuan Liu, and Maosong Sun. Predicting emergent abilities with infinite resolution evaluation. In International Conference on Learning Representations, 2023. URL http:://app.semanticccchot
semanticscholart .wweeroggggoooee org/CorpusID: 263672005.
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. ArXiv, abs/2442..0417, 2024. URL https://app. http://pp..mmaaiiiiiicsecmea semanticscholar. www.arroooooooooeeeee org/CorpusID: 267499829.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, /abs//000..86// 2020. URL https://appi http::::/ppi..mmmtiicchl...... semanticscoolar org/
CorpusID:21088
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Kumar Guha, Sedrick Scott Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean-P'ierre Mercat, Mayee Chen, Suchin Gu- rurangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham M. Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke S. Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groen- eveld, Luca Soldani, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-Im: In search of the next generation of training sets for language models. ArXiv, abs 1446,19994, 2024. URL https: / /api. www.nticschollaaaa.. CoggCorps htte//oopuuiD:228666338 270560330.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. pp. 2381-2391 Brussels, Belgium, October-November 2018. doi: 10.18653/v1/D18--1266 URL 1260.
MosaicML. Llm foundry, 2024. 2024-12-03.
Accessed:
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. ArXio, abs/ 2005/2300.1666,, 2023, URL https: //ppi. semanticscholar org/ C
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James Validad Miranda, Jacob Daniel Morri- son, Tyler C. Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Chris Wilhelm, Michael Wilson, Luke S. Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hanna Hajishirzi. 2 olmo furious. 2024. URL https://api. www.sti .eeellarrrrrgg/ge org/CCrppsSDD) 27521122722333 3098.
Felipe Maia Polo, Seamus Somerstep, Leshem Choshen, Yuekai Sun, and Mikhail Yurochkin. Sloth: scaling laws for 11m skills predict multi-benchmark performance across families, 2025. URL https: / /ttp////xvvvvva////v///e///aaa wwtrxxi..or/aabee orgg/bb//412.....
Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models. ArXiv, abs/2406.19146, 2024. URL https: / /api. wwwantiischooleeeeooee
org/CorpusiD. 2707648387
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-110:67, 2019. URL https://api semanticscholar. wwtpcc/aaa.....ooooooooo.ooa.aa. (ttg/CoopuSID:22438800
13

---

Published as a conference paper at COLM 2025
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, (4(05))8732-8740) Apr. 2020. doi: 10.1609/ aaai.v34400.6633 URL
https://ojs... aaai. org/index.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. pp. 4473-4473, Hong Kong, China, November 2019, doi: (0.1869/v11/119-4453
D19-1444.
Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why has pre- dicting downsstrea capabilities of frontier AI models with scale remained elusive? In Trustworthy Multi-modal Foundation Models and Al Agents (TiFA), 2024. URL https:
//openreview.net wwweeeewwwwweeeeeeeeeeee -ew..//orrrrmiiddrbrrrrrrr
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. pp. 4149-4158, Minneapolis, Minnesota, June 2019. doi:
URL N19-1421.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? pp. 4791-4800, Florence, Italy, July 2019. doi: URL
10.18653/v1 /1/
Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Yi Ma, Yizhi Li, Ziyang Ma, Bill Yuchen Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quehry Que, Ruibo Liu, Si yang Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang, Yizhi Zhou, Yubo Wang, Yuelin Bai, Yuhan Zhang, Yuxiang Zhang, Zenith Wang, Zhen Yang, Zi-Kai Zhao, Jiajun Zhang, Wanli Ouyang, Wenhao Huang, and Wenhu Chen. Map-neo: Highly capable and transparent bilingual large language model series. ArXiv, abs/2405.19327, 2024. URL https:tt/////. //api.appppp..p wwwanticscholar.rmmmmmm C 270094960.
A Fitted Parameters
In Table 5, we list the parameters of the fitted functions in Figure 3 and Figure 4.
B Additional Analyses
B.1 Additional variance analysis results
Figure 6 shows the intermediate checkpoints for the largest ladder model across all tasks, along with the relative standard deviation over the final 10 checkpoints (% SD1o). In Table 4, we show that this intermediate checkpoint noise corresponds to the difficulty of predicting the task performance for the target model.
B.2 Predicting task accuracy under the MC format
In this paper we have been focusing on the ranked classification (RC) format of tasks. Here, we explore if we can make predictions when problems are written in the multiple- choice (MC) format. This a challenging problem, as small models often exhibit random performance on standard downstream tasks (e.g., MMLU) until certain size threshold (Hu et al., 2023). Thus, it is not practical to fit the mapping from task loss to MC accuracy in step 2 with data points from the ladder models. Instead, we use data points from early intermediate checkpoints of the target models.
Observing the MC accuracy curves during training of the 7B-4T and 13B-5T models (Figure 7 upper), we find that they have three phases: (1) very early in training, MC accuracy is random; (2) at some point, MC accuracy increases rapidly; (3) finally, it starts growing
14

---

Published as a conference paper at COLM 2025
Task
Step Fitted Function
MMLU
L(N, D) 38.07 100.00000090000
0.45
HellaSwag
L(N,D) 11.23/ 60.37 0.50
ARC-Challenge
L(N, D) = 702974.93/ /00/3///0.99./
4 0.65
ARC-Easy
L(N, 79412.07 /
19957.51/D04 0.56 0.56
PIQA
L(N, D) = 405.66 20..6//0..40 + +
CommonsenseQA
L(N, D) 56.86/N /
+.000 0.00
Social IQa
L(N,D) 1200.94 / 7897. 0.95
OpenBookQA
L(N,D) D) = 86346.32 No.99 + 137.33 D0.6 1.20
Task
Step 2 Fitted Function
MMLU
Acc(L) -0.74/ +
HellaSwag
Acc(L) -0.73/ + exp
ARC-Challenge
Acc(L) -0.78/ + expl -5.91
(.91(L-0.71))
-0.71))
ARC-Easy
Acc(L) 0.65/ 1111eeepp exp( 4.13(L
0.74))
1.00
PIQA
Acc(L) = 0.46/ {xp( 5.03(L
0.96)
1.00
CommonsenseQA
Acc(L) 0.86/(1 exp( 2.21(L
1.13)
1.00
Social IQa
Acc(L) 0.60/ [1 exp( 7.16(L
0.89)
1.00
OpenBookQA
Acc(L) 0.79/ exp( 4.31
1.08))
1.00
Table 5: Parameters for the fitted functions for Figure 3 and Figure 4.
steadily with more training steps. The phase of rapid growth for a model is strikingly identical across all tasks: around 70k steps for the 7B-4T model, and around 20k steps for the 13B-5T model. As result, the data points {[L,, Acc;)} fall on a peculiar shape that cannot be described by sigmoidal function (Figure lower left).
Therefore, we fit the sigmoidal curve on data points collected from intermediate checkpoints during the third phase of the MC accuracy curve. For 7B-4T, we use steps between 170k and 450k; for 13B-5T, we use steps between 50k and 150k. Noticing that the curves for the two models do not coincide, single joint fitting in RC). We conduct a case study with MMLU and show the fitting and prediction in Figure 7 (lower right). Using data points from the third phase, extrapolate the mapping from task loss to MC accuracy. We predicted the MC accuracy of 7B-4T with 3.0% relative error, and that of 13B-5T with 1.0% relative error. When chaining this with the step 1 fitted function, our end-to-end prediction of MMLU accuracy has an absolute error of 0.3 points on the 7B-4T model, and 0.4 points on the 13B-5T model.
fit a function for each model separately (as opposed to
reliably
Compute overhead. To fit the step 2 function for MC accuracy, we do need to collect data points from an early part of the target model training. 7B-4T is trained for 928k steps, and use data points up to 450k, which is about 50% of the full training compute. 13B-5T is trained for 596k steps, and we use data points up to 150k, which is about 25% of the full training compute. We note this is significantly more compute than used in the model ladders for predicting RC performance. However, as noted above, smaller models (which are less compute intensive) don't provide useful signal here. This is an opportunity for future work.
B.3 Compute requirements for predicting each task
We consider the impact of the scale of the model ladder on the prediction for the target model. In general, the prediction is harder if the difference of scale between the ladder and the target models is larger. We explore this trade-off using the 7B-4T model the target.
In Figure 8, we progressively increase the compute-FLOPs used for function fitting. The prediction error is significantly worse with fewer FLOPs. On MMLU, our full ladder (3.2% compute of the target model) gets 1.3% error. Reducing the number of FLOPs to 0.1% of the target model compute increases the error to 12% which is order of magnitude higher.
15

---

Published as a conference paper at COLM 2025
1.38 (intcmediito checkpoinnsi
1.38 thaal 10 chockpointsi
HellaSwag
(Loss relative 5Dyt 0.26%)
(Accuracy relative 5Dyt 0.28%
(Loss relative 5Dyt 0.09%
(Accuracy relative 50gt 0.25%)
1.08
Tank RC accurccy
0.36
0.80
Tok sccurscy
0.64
0.79
1.05
0.35
0.62
0.78
1.03
0.60
0.34
0.77
1.00
0.58
0.98
0.33
0.76
0.56
10a4
1.00
1.05
10-h
0.76
0.78
0.80
ARC-Easy
Easy
CMC-CChaleegge
MiC-Chalonnge
relative 5Dys 0.66%)
(Accuracy relative 5Dyt 0.61%)
(Loss relative 5Dp 0.40%)
(Accuracy relative 5Dyci 1.00%)
1.00
Tosk 3C accunccy
Tosk SC accursry
0.75
0.70
0.97
0.73
0.95
0.88
0.70
0.93
0.08
0.86
0.90
1021
0.70
0.75
102:
0.90
0.95
1.00
Boolo
BootO
CommonsenseOA
Commonsense@A
(Accuracy SDp,
SDnt
(Accuracy SDi:
0.45
0.64
0.44
0.80
0.85
bok
1:10
scourto
0.42
0.55
1.05
0.40
0.38
0.45
1.00
10a1
0.38
0.40
0.43
0.45
10a:
1:00
1:05
1.10
OpenBBookOA (Loss relative 50 0.34%)
OpenBooKOA
PIAA
PIQA
(Accuracy relative 5Dpt 2.51%)
(Loss relative SDyt 0.19%)
(Accuracy relative 50., 0.31%)
1.48
Sast RC sccuricy
0.40
1.05
BC sccurscy
0.76
1.45
0.38
1.04
0.75
2.43
0.36
1.02
0.74
1.40
0.34
0.73
1.38
1.00
0.72
10
1.40
45
10ah
1.00
1:02
1004
1.06
Social IQa (Loss relative SDyt 0.23%)
Social 10a
Winogrande (Loss relative 5Dyt 0.75%
Winogrande
(Accuracy relative SDut 0.61%)
(Accuracy relative SD 0.77%)
1.37
1.60
Task RE accurncy
0.52
1.57
SC scturssy
0.62
2.10
1.55
0.60
0.50
1.08
1.52
0.58
10a1
1.08
1.10
1.12
10a
Tokens
Task
Task
Figure 6: Intermediate checkpoints and standard deviation over the final 10 checkpoints (SD10) for the 1B-10XC ladder model. We find some tasks exhibit high noise between adjacent training checkpoints, which is indicative of the inherent difficulty in predicting performance for such tasks using the model ladder.
This is more observable in certain tasks. Interestingly, we see a slight improvement in errors with less compute in ARC-C and ARC-E, potentially due to the variance as seen in $5.
We also consider the impact of each axis of the ladder models (model size N and Chinchilla multiplier xC) on the prediction error for each task.
Figure 9 shows the prediction errors for each step as we progressively increase the model size included in the ladder (i.e., ladder upto 760M will include 190M, 370M, 760M). We observe downward trend in the prediction error for most tasks as the ladder size gets closer to the target model.
Figure 10 similarly shows the impact of including models with longer training regimes. Interestingly, we do not see significant reduction in the prediction error as we include the ladder models trained to higher Chinchilla multipliers*. There is slight downward trend (noticeable in MMLU), but this dimension has more flexibility.
'The target model 7B-4T is trained to approximately 28xC.
16

---

Published as a conference paper at COLM 2025
MMLU (3.82%)
MMLU (Fitting error: 0.39%)
0.6
13B-ST-full (fitted)
78-4T.full (fitted)
Task MC accuraty
0.65
1.0%
3.0%
78-4T-170k-400k (fitted)
7B 4T (target)
(predicted)
lack MC
0.4
0.55
138-5T-50k-1 10k (fitted)
138-5T (target)
(predicted)
0.50
0.75
0.90
0.85
Task loss
Task
Figure 7: Upper: MC accuracy curves during training of 7B-TT and 13B-5T. Lower left: Task MC accuracy vs task loss, with data points from all intermediate checkpoints of 7B-4T and 13B-5T. A sigmoidal function cannot fit the data points. Lower right: Task MC accuracy vs task loss, where the sigmoidal function (Equation 2) is fitted on data points from the intermediate checkpoints between 170k-450k steps of 7B-4T, and between 50k-150k steps of 13B-5T.
---chated
04
12"
13
10"
12"
Commonsenssia
OpenRaccGG
06
16"
12.
used sndiiiii
Bepi prodiatoo
Tital sogi proditin
Prediction error the 7B-4T target model function of the total compute- FLOPs used in the ladder models for prediction. The left-most point uses only the smallest model (190M-1xC) for prediction. The right-most point uses the full ladder (all 16 models). The prediction error generally reduces as the ladder FLOPs increase. ARC-C, ARC-E, and OBQA display higher variation (which can be attributed to the variation analysis in $5), but still have a downward trend.
Figure 8:
For users of our approach, and future work, we encourage exploration in reducing the overall cost by considering both these dimensions. Concretely, given fixed budget for the ladder models, increasing the model size N more likely to improve the prediction error, compared to training for longer.
17

---

Published as a conference paper at COLM 2025
step1
---choled
HollaSwig
ARC-Chullenge
ARC-Ean
1.0
1.0
1.0
1.0
0.5
0.5
12
02
0.2
0.2
2.0
0.0
0.0
00
100
1u0
PIDA
CommorsereeQA
Social K2n
OpentrokGAA
2.8
Prstitio
2.8
0.6
0.5
0.5
1.4
04
0.4
0.25
0.50 0.75
1.00 1.25
0.25
0.59
0.75
1.00
1.25
0:55
0.00
0.75
1.00
1.25
0:25
4.50
0.75
1.00
1.25
(UpRo) Model Size IN)
(B/30) Mocol Sizv (1)
500
1e0
[0000) Model Sloe (N0
Figure 9: Prediction error on the 7B-4T target model when including up to model size (N) in the ladder for prediction. Eg. N up to 760M will include 190M, 370M, 760M models trained to 1xC, 2xC, 5xC, 10xC. For most tasks, we observe downward trend in the prediction error as N increases.
-lep!
HellsSwag
ARC-Challeng
1.0
1.0
1.0
2.
0.6
0.6
0.8
Prodee1
00
0.
2.0
0.0
0.
0.0
PIDA
CommonosnseOA
Social I0n
OpentlookQA
Proddee Ero
2.0
0.6
1.4
0.4
0.4
0.4
0.2
10
10
Mullppier (KC)
Muliiperr (63
Mall phe
Figure 10: Prediction error on the 7B-4T target model when including models trained up to chinchilla multiplier (xC). Eg. xC up to 2xC will include 190M, 370M, 760M, 1B models trained to 1xC, 2xC. The downward trend of the prediction error is less significant than with varying N.
C Additional Details on Design Choices
C.1 Compute-FLOPP C instead of (N, for task loss prediction
We test if compute-FLOPs C can be used directly for task loss prediction instead of (N, D) in the over-trained regime with the ladder models. We fit similar power function
L(C) = A/C +E
(4)
where A,a, a, E are parameters to fit.
Figure 11 shows the function fitting and prediction. Compared with Figure 3, the relative fitting error is higher than using (N, on all tasks, likely because Equation 4 has fewer free parameters than Equation 1. For the target models, the task loss prediction errors are generally worse than using (N,D), including MMLU; ARC-Challenge and ARC-Easy are the outliers (Table 3).
Overall, compute-FLOPs are not expressive enough to distinguish between compute-optimal and overtrained models with equal FLOPs but different losses (Hoffmann et al., 2022). Thus, do not use it in our main method.
18

---

Published as a conference paper at COLM 2025
(targtt)
Itarget) 138-5T (predicted)
HellaSwag (0.64%)
ARC-Challenge (0.95%)
1.4
1.2
1.4
1.0
0.8
1.0
1.3%
2.1%
2/18
5.3%
55.2%
2..0%
3.5%%
5 7.00%
1020
102a
10
1090
1042
10a4
10aa
10a2
10
1020
1022
10a
PIQA (0.72%)
CommonsenseOA (1.36%)
Social IQa (1.21%)
OpenBooKQA (0.80%)
1.3
1.4
1.8
1.50
Tock lozs
1.1
1.25
1.0
1.00
0%
0.75
0.5%
2.6%
11..8 18.8%
1..4%
1036
10
1024
10
10
1034
10M
10
1014
10t0
10
1014
Flaps (C)
Flops
Flops
Flops
Figure 11: Step using compute-flops C instead of (N,D). = 1xC; 2xC; 5xC; 10xC. We report the average relative fitting error in parentheses following the task name, and prediction error in the plot next to the target model point.
C.2 Task cross-entropy as the intermediate feature
Task loss only considers the correct choice of each problem. However, task RC accuracy is determined by the losses of both correct and incorrect choices, which is major challenge for predicting downstream performance (Schaeffer et al., 2024):
Acc = N Σ 1
arg max
(-4 R00
5
where 1x (i) is the loss over the k'th answer option of the i'th example and ((0) is the label of
the correct answer
To account for incorrect answers, we define an alternative intermediate feature. We com- monly maximize the accuracy by minimizing the task cross-entropy surrogate loss, in which the Lg terms are used as logits:
TaskCE
1 N
Σ
L (8)
Σ log exp (-I.7)
(6)
20
k
We refer to this as TaskCE to distinguish it from the language modeling cross-entropy over tokens.
Step 2 scaling with Task Cross-Entropy. When inspecting the ladder models, we notice that the task cross-entropy correlates with accuracy in more linear fashion than is the case for BPB. We therefore seek for a function Acc( TaskCE) that is linear for large values of TaskCE, but for small values of TaskCE should approach an asymptote of perfect accuracy (where all the probability mass concentrated on the correct answer). A good candidate for this transition is the log-sigmoid function, leading us to conjecture the following formula:
a logo (-K(TaskCE Acc(TaskCE) (
TaskCEo)),
where or(.) is the sigmoid function, and a, k, TaskCE, are constants to be fit. This expression is not valid in the regime of large task cross-entropies, where models perform random guessing. Thus, we only use data points from the last 50% of training for curve fitting.
*Loss terms may include some normalization, e.g., by the number of characters in each answer or the unconditional answer probability (Gu al., 2024).
19

---

Published as a conference paper at COLM 2025
7B-4T
13B-5T
BPB
TaskCE
BPB
TaskCE
Error
%Error
Error
%Error
Error
%Error
Error
%Error
MMLU
0.6
1.3%
9.0
18.3%
0.3
0.6%
10.4
20.2%
HellaSwag
1.2
1.4%
5.9
7.2%
2.1
2.5%
8.7
10.5%
ARC-Challenge
10.4
16.9%
13.1
21.3%
11.1
17.5%
12.3
19.5%
ARC-Easy
8.0
9,4%
4.5
5.4%
10.0
11.4%
5.4
6.2%
PIQA
0.8
1.0%
2.5
3.1%
0.9
1.1%
2.4
2.9%
CommonsenseQQA
3.1
4.2%
1.9
2.6%
3.5
4.7%
2.0
2.7%
Social IQa
1.2
2.0%
0.5
0.7%
1.7
2.7%
0.8
1.2%
OpenBookQA
5.2
10.6%
3.5
7.1%
3.7
7.8%
0.1
0.2%
Table 6: Comparison of original and new prediction errors for 7B-4T and 13B-5T models across tasks. Absolute (Error) and relative (%Error) differences are shown.
-- (1 --- 373M (TRtec) --- 780M ((Esed) --- 1.38 (fited]
7B-4T (larget)
75-4T (precicted)
135-5T [target)
135-5T 5T-55T (predicted)
ARC-Challenge
1.40
1.40
1.3
1.35
1.7%
1.30
1.35
1.2
6.8%
1.25
1.30
1.30
1.25
1.1
1.20
1.4%
10
10
10
10
10th
10
10aH
1014
10ah
104
10
10
10
10th
10
10a
PIQA (0.12%)
Commonsense@A (0.51%)
Social IQ (0.26%)
OpenBookQA (0.22%)
0.68
1.4
1.05
1.40
0.67
0.66
1.2
1.30
3.6%
0.0%0.95
0.4%
0.0%
10
10
10
10
10t6
10
10
10
10ah
10
10a2
10
10
10a
(2)
[D)
Tolens
Tokens
Figure 12: Predicting final task cross-entropy (Equation 6) from model parameters and token budget in step 1.
Results. Figure 12 and Figure 13 show the results of fitting step 1 and step 2 with the task cross-entropy as output and input variable, respectively. Figure 14 visualizes the predictions of combining these two steps, and Table compares the predictions errors between using task cross-entropy or the correct answer loss as the intermediate feature. We make the following observations:
In step 2, the task cross-entropy is overall more accurate predictor of RC accuracy: The average prediction errors across 7B and 13B models is 2.75% vs. 3.6% for the correct answer loss in Figure 4. We note that the step 2 fit in Figure 13 is particularly good ARC-Challenge and OpenBookQA but substantially worse on HellaSwag, We note all our observations for HellaSwag are in the "linear" regime of the log-sigmoid, making it difficult to estimate the curvature of the transition to the horizontal asymptote.
The average fitting and extrapolation errors in step of Figure 12 are moderate and comparable to Figure 3. However, we notice that the fitting errors for TaskCE tend to be more skewed, i.e., the fitted Chinchilla curves tend to underestimate the performance of the 190M-1xC model and overstimate the 190M-10xC, while underestimating the 1B-1xC and overestimating the 1B-1OXC. As these errors appear more systematic, it raises questions whether this parametric form is good fit over much larger scales. Notably, it results in substantial errors in the task cross-entropy of MMLU and ARC-Challenge
Finally, we note that the values for TaskCE fall into narrow range. This means that the predictions in step 2 are highly sensitive to small changes in the task cross-entropy, e.g., difference in task cross-entropy of less than 0.05 nats in PIQA can make the difference
20

---

Published as a conference paper at COLM 2025
190M (fitee)
370M 760M (fitte)
HellaSwag (Fitting error: 0.S8%aRC-Challene (Pitting erren: 2.10 (Fittinn (predicted)
1.38
7B-4T
78-47 (predicted)
(Fitting eroor: 0.90%
1.0
0.8
1.0
Task PC accurscy
0.5
0.5%
856
0.5
0.8
0.4
0.4
0.3
0.4
1.20
1.25
1.30
1.0
(Fitting enrori 0.70% CommonsenseQA (Fitting erron 1.10 IQa (Fitting errori 0.93% (Fitting error: 2.69%)
Toak eccurscy
0.80
1,00
0.7
1.%%
0.65
0.5
0.2%
3.7%
0.6
0.4
0.70
0.50
0.60
0.64
0.68
0.95
1:00
1.05
TAsiCE
TaskCE
Figure 13: Predicting the task metric from the task cross-entropy (Equation 6) in step 2.
1900 (fitee)) --- 370M (Tited) --. 760M (fited) --- 1.38 (fited)
76-47 (target)
x
75-4T (predicted)
135-ST Barget)
x 155-51 (predicted)
MMLUU
HellaSwag
10.4%
ARC- MRc-Chaleenge
ARC-Easy
02%
Takk occuady
0.5
2%
19.5%
20.2%
0.5
0.4
10.3% 0.6
0.4
0.3
0.2
2010
10th
10
10
10t0
10a
10th
10
10aa
10:4
10
10
10tH
10ah
PIQA
2.0%
CommonsenceQA
Social IQu
1.2%
OpenBookOA
0.2%
TS sccraxy
0.8
1.1%
0.6
0.6
0.7
0.5
0.6
0.4
10t0
10th
10sh
10
10
10a
10sh
10
10ah
10-1
10th
10
10
1041
101H
1022
Figure 14: Chaining predictions from step 1 (Figure 12) and step 2 (Figure 13) with task cross-entropy as the intermediate feature.
between random task accuracy (at TaskCE = 0.69) and 83% accuracy (at TaskCE = 0.65). As consequence, small relative errors in step 1 (0.6% for the 13B model PIQA) are amplified in Step 2 and result in larger overall errors in accuracy (2.9% for PIQA), despite a good fit in step 2 (0.9% prediction error for the 13B accuracy). Similarly, step-1 prediction error of 4.7% for the 13B model on ARC-Challenge results in an overall error of 19.5%.
C.3 General language modeling loss as the intermediate feature
Language modeling loss (LM loss) on held-out sets has been shown to follow the power law (Hoffmann et al., 2022). Here, we consider if we can map it to task performance.
We experiment with using the LM loss on the C4-en validation set (Raffel et al., 2019) the intermediate feature.
Figure 15 shows the function fitting and predictions.
In step 1, the fitted function underestimates the C4 loss by 2.0% for 7B-4T and 3.8% for 13B-5T. This error is higher than that on task loss prediction for 4 out of 8 tasks. The step
21

---

Published as a conference paper at COLM 2025
C4 (0.25%)
190M (fitted)
3.5
370M (fitted)
760M (fitted)
11.38 (fitte]
70-4T (Larget)
(predicted)
2.5
2.0%
13B-5T (target)
3.8%
130-5T (predicted)
10
10th
Tokens (D)
190M (fiited)
370M (fitted)
700M (fiite)
1:30 (fitee)
70-4T (target)
70-ST (predicted)
138-5T (aagett
130-51 (predicted)
MMLU (Fitting error: 0.58%)
HellaSwag (Fitting error: 1.12%aRc-Challene (Fitting arro:: 1.91%/RC-Easy (Fitting arror: 0.87%)
0.8
Task RC accuracy
0.6
0.8
2.2%
0.8
6.0%
0.5
0.6
0.4
0.3
0.4
2.0
2.5
3.0
3.5
2.0
3.0
3.5
0.0
2.5
3.5
4.0
2.0
2.5
3.5
PIOA (Fitting error: 0.49%) Commonsense@A (Fitting error: 1.70'Secial IQa (Fitting error: 0.99%)ppenlBookk. (Fitting error: 2.35%)
0.9
0.8
3.5%
9.8%
0.7
0.6
0.5
11
10%
3.3% N. 1.0%
Task RC
0.7
0.6
2.0
2.5
3.0
3.5
4.0
2.0
3.0
3.5
4.0
2.0
2.5
3.0
3.5
4.0
2.0
2.5
3.0
3.5
C4 loss
C4 loss
CA
loss
C4 loss
(target)
(predicted)
(targe))
(predicte)
MMLU
5.0%
HellaSwag
ARC-Challenge
ARC Eany
1.0%
4.5%
ToCk SEcurcE
0.5
2.%
.4%
0.7
0.4
0.4
0.6
0.3
0.4
10:0
10t1
1012
10
10
10ah
10t2
1013
1010
10th
1012
1013
1010
1021
1012
1013
PIQA
1.5%
Commoraense@A
7.0%
Social
7.8%
OpenBookKA
10.4%
Tisk IC accursoy
0.8
017%
0.7
0.6
0.5
0.4
0.7
0.5
0.5
0.3
0.4
10.0
1011
1012
103s
1039
1011
10
1013
2010
1013
1012
20
1010
1031
1012
1013
Tokens (D)
Tokens (D)
Tokens (D)
Takens (D)
Figure 15: Using language modeling loss on C4-en validation as the intermediate feature. From top to bottom: step step 2, and chaining the two steps.
error also higher than using task loss on out of tasks. When chaining the two steps to predict task accuracy, using C4 loss intermediate feature resulted in higher error on 5 tasks MMLU, HellaSwag, CommonsenseQA, Social IQA, and OpenBookQA. Prediction error is lower ARC- ARC-Challenge and ARC-Easy. We conclude that using C4 loss can benefit certain tasks where task loss does not work well.
Comparison of three intermediate features C.5
C.4
Combining step and into single step
Here, we try to directly predict task accuracy from the training scale (N, D) in one step, by combining Equation and Equation 2 into single parameterized function, and merging
22

---

Published as a conference paper at COLM 2025
18-4 Mebe Abeuur Eices
139-6 Mode AEsuda Erers
Tace
Dampler
Taxk Lowr
CALOR
Caae
Roy Aweraee
Taxk Name
Compler
Teak Love
D4 Lows
Taece
127
H042
93
131
AR0-Caeeagg
1122
11
AROGay
277
63
10
45
44
ARD Sag
200
98
17
34
541
37
02
ComdinnE
63
19
32
3.0
11
TB 4T Mode
sisa Braw.
Model Rebatio fron (9)
Dumplen
04tom
Lake
Raw Anerage
Taxk
Comple
TerCC
Anenge
MUU
16A6
135
12%
18.%
7.2%
MMU
MM4
07%
57%
2..5
64%
45%
72%
43%
raledway
1042
25%
57%
KTOGay
12%
ARD-Day
200
11,4
125
FON
100
135
47%
31%
145
PIAA
100
13%
12%
186
18%
186
20%
21%
6.5%
Opatkcatin
78h
Avarae
137%
110
837%
1.8%
Aveage
58%
6.85
6.4%
Figure 16: Comparison of absolute and relative prediction errors for all three intermediate features. Using C4 as the intermediate works well for certain tasks, but results in overall higher errors. Task loss and TaskCE loss perform comparably. However, MMLU and HellaSwag have higher number of instances, and task loss performs better for them.
parameters k and La into A, B, E, so that we reduce free parameters:
a
Acc(N,D) D) 1 exp (A/Na + B/D B)D +
(8)
With this, we remove the a priori definition of a specific intermediate feature (i.e., the A/N" + D/D + E expression no longer carries specific meaning), while preserving the representation power of the function. This function can be harder to fit, since it has more free parameters, is not convex, and we cannot use any data points collected from intermediate checkpoints as we did in step 2.
We fit this function with data points from the final checkpoints of the ladder models, using the same optimization method as step (Figure 17). The prediction error is higher on out of 8 tasks - MMLU, ARC-Challenge, Commonsense@A., and Social IQA. In particular, CommonsenseQA the fitted functions does not vary with respect to model size, indicating degenerated function fitting. We conclude that the single-step approach is not robust as the two-step approach.
D Scaling law predictions for 32B-6T
We demonstrated our method on 7B-4T and 13B-5T models, using only 1% the compute required for the target models. Here, test the robustness of the ladder, by trying to
23

---

Published as a conference paper at COLM 2025
190M (tRted)
370M (1eted)
760M (1itee)
(Rited)
70-4T
70-4T
130 37--5 (predicted)
HellaSwagg (0.66%)
5.8%
ARC-Challenge
ARC-Easy (0.42%)
bakk bsk acCuSAC
0.5
0.8
0.6
22.6%%
0.4
0.6
0.5
21.7% 0.7
0.4
0.3
0.5
1030
1021
1012
1023
10th
10tt
10
10
1022
10
10
10
10:0
1032
10
1022
Tolk acuaary
PIDA (0.515%)
7.%
0.6%
0.7
Commonsense@A (5.58%)
6.6%
Social I0a (0.53%)
5.5%%
0.5
OpenBookkQA (1.73%)
5.4%
0.75
0.6
0.4
0.5
0.3
0.65
0.45
0.4
0.60
102h
10
Takers (3)
Tier's (D)
Rokens (D)
Takens (D)
Figure 17: Task RC accuracy vs training scale (N, (N, with fitting on the single-step function in Equation 8.
--- 190M 370M (fitte)
760M (itted)
3. (fited)
70-4T
70-4T (precicted)
130 ST
130 (gredicted)
2.5%
ARC-Challenge
0.5
0.8
0.6
175%
4%
0.4
0.6
0.5
0.7
0.4
0.6
0.3
0.3
1011
10
10
10
10th
10
10
10t
10th
10ah
10
10
10
10
10
201h
PIQA
1.1%
0.8
CommonssesaaAA
IQ
OpenBookQA
0.80
0.7
2%
0.60
2.00%
10.0%
7.0%
0.75
0.6
0.35
0.70
0.5
0.50
0.65
0.45
0.4
10
10
10
10
10tH
10
10
1014
10th
10
10"
10
10a
10th
10
1014
Tolars (2)
Theens (D)
Talans
Theers (D)
32B-6T
Task
Pred
Actual Error
%Error
HellaSwag
88.0
85.3
2.7
3.07%
ARC-Challenge
53.6
66.8
13.2
24.63%
ARC-Easy
77.8
88.8
11.0
14.14%
PIQA
82.9
83.6
0.7
0.84%
CommonsenseQA
79.5
75.4
4.1
5.16%
Social IQa
61.1
62.3
1.2
1.96%
OpenBookQA
45.4
53.2
7.8
17.18%
Average
5.81
9.57%
Figure 18: Task accuracy prediction for the 32B model using task loss the intermediate 2xC: 5xC; * 10xC. Prediction error next to the target.
feature. =1xC;
predict the accuracies for a much larger model of the same family: OLMo 2 32B, trained to 6T tokens (32B-6T). This model was trained with 1.15 x 10 FLOPs, and the ladder models account for only 0..5% of the compute of this model.
24

---

Published as a conference paper at COLM 2025
Figure 18 shows the predictions and errors for 32B-6T model. As expected, and following from B.3, the overall prediction error is higher We also observe a similar trend of higher errors for high variance tasks like ARC-C and ARC-E as with the smaller target models. However, for lower-variance tasks such as HellaSwag, PiQA, and SocialIQA, we can still predict the accuracy within an absolute error of 3 points. This indicates that the ladder models are fairly robust in their prediction abilities even as the target model is scaled to be much larger.
25


Reviews:
[{"review_id": "86QCGj243w", "reviewer": "Reviewer_Jq1a", "summary": "The paper proposes the use of smaller capacity ladder models (ranging between 190M to 1.3B parameters) to predict the downstream task performance of larger (7B and 13B parameter) OLMo target models over a selection of tasks from the OLMES evaluation suite.\nThe investigation takes into consideration the number of model parameters (N) and the amount of training data (D) used for the creation of the models.  \n\nThe proposed approach involves two stages.  \nThe first one tries to predict the task-specific loss based on the parameters N and D, producing so-called intermediate features.  \nDuring the second stage, the task loss is used for predicting the downstream task performance.  \n\nThe approach is easy to follow, but the raises several question mostly due to insufficient comparison to alternative approaches that are elaborated among the weaknesses in more detail.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Predicting task performance of LLMs is a timely chosen topic and the writing is generally easy to follow.", "reasons_to_reject": "The paper lists most of the relevant related work, but does not include a meaningful comparison to the related approaches.  \nFor instance, the related work section claims that ``Chen et al. (2024) also employs a two-stage approach for predicting downstream performance, but uses the pre-training loss instead of a task-specific loss as the intermediate step.'' (line 261--263), but there is no empirical evidence presented that the latter approach is more performant and there is no discussion on the pros and cons of the different approaches.\n\nEven though the amount of compute required to train the ladder models is marginal compared to the costs of training the target model(s), it is not negligible in absolute terms.  \nThe amount of compute used for pre-training the additional models, and extracting the intermediate features from them could just be used in a variety of alternative ways for giving a task performance estimate (e.g. evaluating the target models on a sample of the downstream task from time to time).  \nThe paper does not convincingly demonstrate why the proposed approach is superior to such alternatives.\n\nAs LLMs are known to be highly sensitive to prompt wording it is debatable to what extent task performance prediction is feasible as using slightly different prompts could result in radically different task performance.  \n\nExperiments are only performed on a single model family.", "questions_to_authors": "The paper claims that recent LMs are overtrained (l. 54), but it also mentions that one of the target OLMo models was __only__ trained over 3.9T tokens (l. 77).\nThe latter statement feels contradictory to the former one.\nAlso, the fact that the small amount of training data was meant to be mitigated by adding 50B  (0.05T) extra training tokens, increasing the 3.9T tokens to 3.95T feels insufficient.", "extra_scores": {}}, {"review_id": "KNVF4ajjGh", "reviewer": "Reviewer_ExxQ", "summary": "The authors develop a means of estimating task-specific LLM performance as a function of model and dataset size. In theory, this enables estimating the performance of large models trained on large dataset on specific downstream tasks, without actually training them, by extrapolating the scaling laws. The evaluation is focused on predicting performance on tasks in the ranked classification format, which simplifies evaluation because candidate answers can be scored directly, instead of requiring models to select a specific answer from a list of candidates, which smaller models cannot do effectively. To predict the task-specific loss, two functions are fit -- one to approximate the task specific loss as a function of model and dataset size, and the second to predict the accuracy as a function of the predicted loss,  thus the predicted task-loss is treated as an \"intermediate feature\". The evaluation is very thorough and the presented results are compelling, but because there is no benchmark or baseline for this setting, it's difficult to know how compelling the results actually are.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- clearly written paper with very thorough exposition and evaluation of task-specific accuracy prediction using ladder models to estimate performance of larger models.     \n\n- clear applications to foundation and larger model training, especially in the context of planning.     \n\n- simple implementation that should be easy to replicate or extend", "reasons_to_reject": "- lack of baselines makes empirical results difficult to interpret -- no discussion of alternative approaches.     \n\n- limited to ranking-style tasks in its current formulation, does not work for many tasks because the approach relies upon the log-loss of the model, which may not be enough to connect to task accuracy for many settings.    \n\n- limited discussion of real-world applications and limitations", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RLSMV7jIYt", "reviewer": "Reviewer_Dg36", "summary": "This paper presents an effective method to estimate the benchmark performance of LLMs, moving beyond previous work that only focuses on the pre-training loss as a target metric. The authors conduct comprehensive experiments to demonstrate the effectiveness of the proposed approach and use extended analysis to verify several design choices.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper focuses on a very important research topic. The paper writing is easy to follow, and the presentation is wonderful. \n\n2. For the method part, the proposed algorithm is sound and effectively addresses the challenges in estimating the benchmark performance. \n\n3. For the experiments part, the setting is reasonable, and the results are impressive. The further analysis is extensive and interesting.", "reasons_to_reject": "Major concern: \n\nThis paper is overall a very good paper, regarding the research problem, the method effectiveness, experiments, writing, and the figures/tables/results presentation. The only major concern is that the proposed method is almost the same as the one mentioned in the Llama-3 technical report [1]. And according to the arxiv date, there is a roughly 5-month gap, so they should not be considered as concurrent work. The authors address this by saying, \"Unlike our work, which uses a fixed ladder of small 261 models, they rely on first finding compute-optimal models.\" and \"Prior work has shown accuracy for only one specific task (ARC-Challenge).\". These two statements, at least for my understanding, cannot distinguish the unique difference, since (1) Llama-3-series is also over-trained, so the compute-optimality may not be an important design choice; (2) You cannot imagine Meta only tries on a single benchmark for the scaling laws, alghouthg they are only presenting one in the paper. \n\nGenerally, this paper lacks a unique contribution and novelty as a good research paper. But this paper provides new, comprehensive details, including the concrete implementation, design choices, and large-scale verification of an existing, well-established \"closed-source\" method. So it's a very good technical report. I think the insights/details provided in this paper are valuable and useful, so I vote for acceptance. But the AC/SAC may consider the raised concern if COLM prefers papers with unique contributions and novel ideas.\n\n\nMinor: \nIt would be interesting to see the comparison with the two-stage approach proposed in [2] to decide which is the best intermediate metric to keep track of (pre-training loss vs. task loss). But I totally understand if such a comparison cannot be achieved due to resource limitations. So this is a minor point. \n\n\n[1] The Llama 3 Herd of Models. Aaron Grattafiori et al.\n\n[2] Scaling laws for predicting downstream performance in llms. Yangyi Chen et al.", "questions_to_authors": "N/A", "extra_scores": {}}, {"review_id": "9WiaJjP0ZY", "reviewer": "Reviewer_s6ea", "summary": "The paper introduces *model ladders*, a two-step framework that fits power-law surfaces and then maps them to downstream task accuracy for larger language models. By training a fixed suite of small “ladder” models, the authors can forecast the accuracy of much larger targets (7 B and 13 B parameters) while spending only a fraction of the compute. Their approach predicts downstream scores to within ~4 % relative error and is validated across eight diverse language-understanding benchmarks. Overall, the work offers a fresh perspective on scaling laws by directly linking small-model experiments to practical accuracy estimates for significantly larger systems.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper seems to offer a novel perspective towards scaling laws in general, where instead of just predicting the LM loss, there was an attempt to fit the accuracy function. The study spans 8 diverse tasks, ablation over three intermediates, two moderately large LMs, FLOPs-only baselines, a single-step variant, and detailed checkpoint variance analysis. It is without doubt an empirically rigorous work.", "reasons_to_reject": "There were some assumptions made and some result reported in the paper which might require a bit more justification -\n1. For four tasks, the test sets were used, and for the other four, the validation set. I am assuming one or the other wasn't available for all which prompted the authors to make this choice, but it should be explained in the paper.\n2. The training converged after 4T in 7B and went up to 5T in 13B. Why?\n3. The two-step model plus per-task fits introduces many degrees of freedom; although L-BFGS is used, the risk of local minima or over-parameterization is not analyzed.", "questions_to_authors": "1. Have you attempted to predict larger-parameter checkpoints? If not, what evidence suggests the fitted exponents (α, β) remain stable?\n2. Could the same ladder (trained on OLMo architecture) predict performance of a different LM family model if both share token budgets?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper describes a method for estimating LM performance on downstream tasks by first (1) estimating a standard scaling law for a task-specific loss function, then (2) learning a translation from task losses into classification accuracies. Reviewers agree that this is an important topic and that the method is simple and convincingly evaluated. There were some concerns about related work: (1) methods that use alternative intermediate features, which the authors point out they already compare to (though Tab 3 could make this clearer); and (2) similar ideas in the most recent Llama system description paper, which I think make an open study like the present paper more valuable.
Decision: Accept