LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
Authors: Prasanna Mayilvahanan
Thaddäus Wiedemer
Sayak Mallick
Matthias Bethge
Wieland Brendel
Venue: ICML 2025 poster (2025)
Keywords: LLMs
scaling laws
data-centric ML
generalization
Primary area: deep_learning->large_language_models
TL;DR: Pretraining data distribution, rather than model architecture or other factors, determines the loss-to-loss scaling behavior of large language models⁠.

Abstract: Scaling laws guide the development of large language models (LLMs) by offering estimates for the optimal balance of model size, tokens, and compute.
More recently, loss-to-loss scaling laws that relate losses across pretraining datasets and downstream tasks have emerged as a powerful tool for understanding and improving LLM performance and generalization.
In this work, we investigate which factors most strongly influence loss-to-loss scaling.
Our experiments reveal that the pretraining data determines the scaling trend.
In contrast, model size, optimization hyperparameters, tokenizer and even significant architectural differences, such as between transformer-based models like Llama and state-space models like Mamba, generally have limited impact.
Consequently, practitioners should carefully curate pretraining datasets for optimal downstream performance, while architectures and other settings can be freely optimized for training efficiency.

LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
Prasanna Mayilvahanan 1234
Thaddäus Wiedemer' * 1234 Sayak Mallick
Matthias Bethge 2.34
Wieland Brendel 123
Abstract
Loss-to-Loss Scaling
Scaling laws guide the development of large lan- guage models (LLMs) by offering estimates for the optimal balance of model size, tokens, and compute. More recently, loss-to-loss scaling laws that relate losses across pretraining datasets and downstream tasks have emerged a powerful tool for understanding and improving LLM per- formance and generalization. In this work, we investigate which factors most strongly influence loss-to-loss scaling. Our experiments reveal that the pretraining data determines the scaling trend. In contrast, model size, optimization hyperparam- eters, tokenizer and even significant architectural differences, such between transformer-base based models like Llama and state-space models like Mamba, generally have limited impact. Conse- quently, practitioners should carefully curate pre- training datasets for optimal downstream perfor- mance, while architectures and other settings can be freely optimized for training efficiency.
2.50
-0.9.9
=
HellaSwag Test BPB
2.5
2.00
1.75
1.50
1.25
1.00
1.0
1.2
1.8
2.0
FineWeb-Edu Validation BPB
Base Model
Different Architecture
Different Tokenizer
Different Context Length
Different Pretraining Data
Different Optimizer
Different Size
Figure 1. LLMs' loss-to-loss scaling follows power laws pri- marily shaped by the choice of pretraining data. Using Ilama trained Fine Web-Edu baseline, intervene various factors to assess their impact on train-to-ter loss scaling. Chang- ing the prefraining data has largest effect. Changing the tokenizer, the architecture (e.g., from Llama to Mamba), model size, context length, and optimizer settings have little-to-no influ- ence. For fair comparison across tokenizers, (negative-lo--lkeell)
1. Introduction
normalize loss nll) by bytes (bits-per-byte, BPB). report goodness of fit for the fitted power laws as R Power laws are fitted using all checkpoints, but only a random subset shown to avoid visual clutter. Llama-7B checkpoints run for few steps are highlighted with black border.
Scaling laws have long guided Large Language Model (LLM) pretraining, determining model and data size der a fixed compute budget (Kaplan et al., 2020; Hoffmann et al., 2022; Grattafiori et al., 2024). Typically, scaling laws relate model performance, usually measured as training or validation loss, to total compute measured in floating point operations (FLOPs). FLOPs account for both parameter count and the number of training tokens. While useful for pretraining, scaling laws do not capture how well model ul- timately performs downstream tasks (Gadre et al., 2024; Schaeffer et al., 2024; Du et al.. 2025). Consequently, mul- tiple works have begun to investigate downstream scaling
laws: Scaling laws that directly predict downstream loss from FLOPs (Schaeffer et al., 2024; Gadre et al., 2024).
Brandfonbrener et al. (2024) show that downstream scaling laws can be decomposed into compute-to- compute-to-trainee to--rain--ossss scaling laws and (rrainn-los-tto-(tss)-oooo scaling laws. The com- bination of compute-to-loe and loss-to-loss scaling laws enables efficient and accurate prediction of model's down- stream performance. Moreover, holistic downstream scaling laws often optimize for single task or average performance across tasks (Gadre et al., 2024; Schaeffer et al., 2024), whereas loss-to-loss (especially test-to-test) scaling laws can help tune a model's performance across broader range of downstream tasks, e.g., to ensure broad or robust gen-
'Equal contribution 'Max Planck Institute for Intelligent Sys- tems 'ELLIS Institute Tübingen "Tübingen AI Center University of Tübingen. Correspondence to: Prasanna Mayilvahanan aaaasannammailvaahaaaa @ com,, Thaddäus Wiedemer tthadaacus wiedemer @ goail.com>. Proceedings of the 42nd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LLMs on the Line
eralization. Furthermore, train-to-test (or val-to-test) loss scaling laws characterize model generalization behavior. Specifically, these scaling laws are curves depicting the rela- tionship between training or validation loss and downstream test performance, onto which we fit power-law functions.
2. From Scaling Laws to Interventions
Compute-to- Teaaate-to-Train Scaling Laws Scaling laws aim to optimize model size and token allocation within a fixed compute budget (expressed in FLOPs) by modeling the re- lationship between parameters, training tokens, and training loss (Hestness et al., 2017; Kaplan et al., 2020; Hoffmann et al., 2022). However, these laws are inherently shaped by the data distribution, architecture, and optimization set- tings (Tay et al., 2022; Wang et al., 2024; Brandfonbrener et al., 2024; Porian et al., 2025), making their application across setups non-trivial.
While the impact of design choices like pretraining distri- bution, architecture, tokenizer, optimizer settings, etc. on compute-to-los scaling laws is fairly well understood (Ka- plan et al., 2020; Hoffmann et al., 2022; Tay et al., 2022; Wang et al., 2024; Porian et al., 2025; Du et al., 2025), sim- ilar understanding is missing for loss-to-loss scaling laws. To close this gap, we extend the work of Brandfonbrener et al. (2024); Du et al. (2025), which analyze loss-to-loss relationships within a single architectural and training setup. Adding to that, our study systematically explores how mul- tiple factors influence scaling laws across a diverse range of architectures and training configurations. Such an analy- sis on train-to-test (or val-to-test) scaling laws can help to understand factors that influence model generalization.
Compute-to-Downsttee Scaling Laws Recent works extend scaling laws to directly predict downstream task per- formance from compute (Gadre et al., 2024; Isik al., 2024; Du et al., 2025). While some initial works attempt to map compute budgets to accuracy on individual tasks, multiple tasks, or aggregate benchmarks, this mapping is usually noisy due to several transformations in the accuracy computation that degrade the statistical relationship (Scha- effer et al., 2024). More recent efforts instead use the model's average loss the correct answers of the task as proxy (Madaan et al., 2024; Brandfonbrener et al., 2024). Such compute-tor topputttto-oowssrraaam scaling laws provide more practical perspective on scaling but are still specific to a given training setup.
Our study draws inspiration from a body of work in robust- ness evaluation of vision (and later language) models (Taori Miller al., 2021; Fang al., 2022; Awadalla et al.. 2022). These works show that model performance on different distributions is frequently strongly correlated, and most model and training settings have little-to-no im- pact on the task-to-task scaling trend of model performance. We treat loss-to-loss curves similarly and perform a series of interventions using over 6000 model checkpoints to un- derstand what design choices causally affect loss-to-loss scaling laws.
al., 2020;
Loss-to-Loss Scaling Laws Loss-to-loss scaling laws aim to improve the transferability of scaling insights between training setups by examining the relationship between train- ing (or validation) and test losses, between different valida- tion losses, or between different test losses (Brandfonbreneer et al., 2024). Typically, "training" or "validation" refers to general-purpose, open-text corpora used for pretraining, whereas "test" corresponds specifically to downstream tasks. This perspective is crucial for several reasons. First, train- to-train (or validation- toroonn-ttovvvitiionnnnno scaling implies how scaling laws transfer across datasets (Brandfoobreneer et al., 2024). Second, incorporating train-to-test (or validation-to- test) scaling laws alongside compute-to tonpute-to-train scaling laws provides more precise insight into how compute budgets translate to downstream performance and can help study emergent abilities of models (Du et al., 2025). Third, while compute-to- -t-psss-t--los scaling laws often target a single down- stream task or average task performance, train-to-test and test-to-test scaling laws can help tune model's perfor- mance across diverse tasks, e.g., to foster the development of generalist LLMs with balanced task performance. Fi- nally- and importantly- cennttattl-triin--te train-to-test (or validation to-tttst)) scaling laws serve as useful tools for understanding general- ization.
We make three main observations, illustrated in Fig. 1:
1. LLMs' loss-to-loss scaling consistently follows shifted power laws.
2. Pretraining data is the most salient factor for these scaling laws.
3. In contrast, architecture and tokenizer generally play a minor role, while. model size, context length, and optimizer settings have little-to-no impact on loss-to-loss scaling.
These observations imply that if two models with different training data-achieve similar training losses, they will exhibit closely matched downstream test performance.
but trained on the same
Our results indicate that common LLM architectures and training setups might encode very similar inductive biases, freeing practitioners to optimize them for training efficiency without adversely affecting downstream generalization.
and
2

---

LLMs on the Line
Accuracy on the Line Our work is inspired by robustness research in image classification. Prior studies (Taori al., 2020; Miller et al., 2021; Fang et al., 2022) demonstrate a strong and consistent correlation between in-distribution and out-of-distribution (OOD) accuracy across various im- age classification models and settings. We are not the first to observe the similarity to LLMs, where recent works (Gadre et al.. 2024; Brandfonbrener et al., 2024; Du et al., 2025) highlight strong scaling trends (linear or power-law-like between losses. However, these studies are typically con- strained to a single architecture or training setup. In contrast, we examine trends across wide range of architectures and training conditions (see $4), showing for the first time that loss-to-loss scaling follows consistent laws across settings.
Train-to-Traile
Train-to-Tel to-st
6.5
6.0
6.0
5.5
Validation Loss
5.5
Test
5.0
5.0
4.5
4.5
4.0
4.0
3.5
3
3
FineWeb-Edu Validation Loss
FineWeb-EEu Validation Loss
Validation Set
Test Set
The Pile
ARC-Challenge
RefineWeb
ARC-Easy
Slimpajama
COPA
Robustness Interventions Accuracy-to-aan to--ccurrcccyy relation- ships in the vision, vision-language, and language domains have also been used study how scaling laws shift under robustness interventions like dataset size, adversarial train- ing, architectural details, loss functions, supervision type, or OOD shifts (Taori et al., 2020; Fang et al., 2022; Awadalla et al., 2022; Mayilvahanan et al., 2024a;b; Wiedemer et al., 2024). For vision- Saiiunl models, Taori et al. (2020); Fang et al. (2022) find that most interventions do not impact OOD performance; only increasing data diversity has significant positive impact. Their findings suggest that curating better datasets is crucial for training vision and vision- languae models that generalize broadly.
PIQA
Winogrande
HellaSwag
Figure 2. Loss-to-loss scaling consistently obeys power laws. We extend results from Brandfonbrener et al. (2024) to many architectures, training settings, and validation/test sets. We show illustrative shifted power laws for Mamba trained on Fine here; more configurations and test sets can be found in App. E. For clarity, scatter plots display a random sample of all data points; all points used to fit the scaling laws.
correct answer given the question as context. This is some- times called the cloze formulation of a task since the model is essentially evaluated on its ability to fill in blanks.
Motivated by these insights, we aim to uncover the fac- tors determining loss-to-loss scaling to better understand what matters for generalization. Our insights complement the findings from Awadalla et al. (2022), who show that accuracy-accurace scaling trends in comprehension tasks are agnostic to architecture type (e.g., encoder-only, encoder- decoder, decoder-only) after fine-tuning. In contrast to their study, we focus on zero-shot generalization across a di- verse set of tasks, specifically investigating state-of-the-art decoder-only architectures such as GPT (Radford et al., 2019), Llama (Grattafiori et al., 2024), and Mamba (Gu & Dao, 2024; Dao & Gu, 2024).
Brandfonbrener et al. (2024) predict train-to-train and train- to-test scaling laws to follow a shifted power law!
Ly (F RJ K (Lo - -Ealp) - -Eglp- (1)
where Lay Ly are the losses on datasets Dz., Dy shown on the x- and y-axis. f fND is a model trained with N parame- ters on D tokens on the pretraining set Dp. and K and k are parameters to be fit. Empp: Egip are the irreducible errors (i.e., minimum loss) that Sp trained on Dp can achieve on the datasets Dz, Dy,
Given a model configuration, we collect 200 to 800 check- points throughout training, across model sizes, and for var- ious seeds. All points are used to first estimate Eglps Egjo from individual compute-to-loss scaling laws and then fit K, K. Refer to App. A and App. B for more details.
3. Fitting Loss-to-Loss Scaling Laws
We focus our analysis on train-to-train and train-to-test scal- ing. Combined with known compute-to-tree -o-tain scaling laws, these loss-to-loss scaling laws paint a complete picture of a model's downstream performance given a compute bud- get and characterize model's downstream performance distribution across tasks (Brandfonbreneer et al., 2024).
Note that to study the impact of various training settings, the x- and y-axis show losses on different datasets of the same model. This should not be confused with some figures in Brandfonbrener al. (2024) where the and y-axis show losses of different compute-matched matched models.
As is standard in the recent literature, we report test loss as a proxy for downstream performance. Following Brandfon- brener et al. (2024); Madaan et al. (2024); Schaeffer et al. (2024), we track the test loss as a model's 's loss on only the
'Fq. (1) here follows from Brandfonbrener al. (2024) Eq. (4) when assuming an irreducible error as in Eqs. (6, 7); see App-
3

---

LLMs on the Line
4. A Causal Analysis of Loss-to-Loss Scaling
We now perform interventions on the model and training configurations to find what factors cause the exact shape of loss-to-loss scaling laws.
Ineffective Intervention
Base Model
Loss
Our basic procedure is outlined in Fig. 3. As mentioned in $2, our approach is motivated by similar studies in the robustness literature. In contrast to that setting, here we lack paired in-distribution and out- -ut-of-distribution datasets. In- stead, we simply consider all combinations of validation and test sets. Typically, "validation" refers to general-purpose open-text corpora used during pretraining, whereas "test" corresponds specifically to downstream tasks. For ease of visualization when intervening on the pretraining data, we always show Web-Edu validation loss on the x-axis, even for models trained on different pretraining distribu- tions. This choice is arbitrary and does not affect our results; see App. F. Similarly, we here report results for scaling laws of average validation and test loss; results for individual losses can be found in App. G.
Effective Intervention
Loss 1
Figure 3. Schematic our causal analysis. Checkpoints of base model trained on different numbers of tokens and with dif- ferent seeds lie on the same loss-to-loss line. Better-performing models (typically with higher compute) achieve lower loss (to- wards the bottom left). We intervene on training settings (c.g., pretraining data, architecture, etc.) and retrain from scratch, yield- ing new models that again constitute loss-to-loss lines. An effec- tive intervention produces models new line; an ineffective intervention yields models that lie on the line of the base model.
For our analysis, we consider the impact of pretraining data, tokenizer, architecture, model size, context length, and optimizer settings.
With this setup, we can now analyze the loss-to-loss scaling laws of models trained with different configurations. Brand- fonbrener et al. (2024) only showed loss-to-loss scaling a sin- gle architecture with fixed training recipe: Olmo (Groen- eveld et al., 2024). We extend their analysis by multiple architectures, pretraining sets, tokenizers, and training set- tings. all listed in $4. As an illustrative example, we show loss-to-loss scaling for Mamba trained on FineWeb-EEd in Fig. 2; more examples with additional test sets are listed in App. and throughout $4.
Pretraining Sets Our models are trained on Febe Edu (Penedo et al., 2024), C4 (Dodge et al., 2021), and an uncopyrighted version of The Pile dubbed The Pile UC. Some models from Hugging Face are trained on the orig- inal version of The Pile (Gao et al., 2020) and The Pile Deduped (Biderman et al., 2023), a deduplicated version.
Validation Sets Models are evaluated on 5000 sequences sampled from the validation sets of FineWeb-Fdu,, C4, The Pile, RefinedWeb (Penedo et al., 2023), and SlimPa- jama (Shen et al., 2024).
Overall, across models, datasets, tokenizers, and optimiza- tion hyperparameters shifted power laws describe loss-to- loss scaling well (Eq. (1)). On some datasets, the perfor- mance of models with high loss (towards the top right of each curve) is not captured perfectly by the power law for- mulation proposed by Brandfonbrener et al. (2024). This is unsurprising, given that these data points typically represent models in early training stages but might hint refined formulation of Eq. (1) for the high-loss regime.
Test Sets We use LM Evaluation Harness framework (Gao et al., 2024) to assess model performance on Hel- laSwag (Zellers et al., 2019), COPA (Gordon et al., 201 1), WinoGrande (Sakaguchi et al., 2019), PIQA (Bisk et al., 2019). SocialIQA (Sap et al., 2019), CommonsenseQA (Tal- mor et al., 2019), MMLU (Hendrycks et al., 2021), as well as ARC-Easy and ARC-Challenge (Clark et al., 2018).
Note also that loss-to-loss scaling follows power law even for datasets on which the model never reaches high accu- racy. E.g., Mamba in Fig. 2 never surpasses chance perfor- mance on ARC-Challenger yet Eq. (1) describes the test loss equally well. This underlines the usefulness of loss-to-loss scaling laws to study model behavior.
Models We train Llama-3 (Grattafiori et al., 2024) with 417 M parameters and Mamba (Gu & Dao, 2024) with 420 M parameters using the Lingua framework (Videau et 2024), following Chinchilla scaling laws (Hoffmann al., 2022). We supplement our analysis with pretrained GPT (Black et al., 2021; 2022; Biderman et al., 2023), Llama (Penedo et al., 2024), and Mamba (Gu & Dao, 2024; Dao & Gu, 2024) variants from Hugging Face (Wolf et al., 2020). We present more details on the models in App. C.
Takeaway 1 Across architectures and training settings,
loss-to-loss scaling generally follows shifted power law as described in Eq. (1) and illustrated Fig. 2.
4

---

LLMs the Line
Architecture Llama Tokenizer tiktoken
Architecture Llama gpt2
Architecture Llama gpi2-HF
Architecture Mamba Tokenizer tiktaken
Architecture Mamba Tokenizer gp12
Architecture GPT Tokenizer gpt-neox
Average Val. Loss
Train-to-
Train-to-Ter
Average
4.5
3.5
FW Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
Ed-Eddu Val. Loss
Pretraining
FineWeb-Edu
Deduped
Figure 4. Pretraining data has a substantial impact on loss-to-loss scaling laws. Models are matched on architecture and tokenizer.
Tokenizers We train Llama and Mamba with either tiktoken tokenizer (128 k vocabulary size) the gpt tokenizer (50 257 vocabulary size). Pretrained models from Hugging Face use an almost identical GPT-2 tokenizer, dubbed 2pt2-HF. This version does not explicitly pad text with beginning and -nd-encee tokens. A few Hugging Face GPT models instead use the gpt-neon tok- enizer with slightly different vocab size of 50 254, which results in different internal mapping compared to gpt2,
Takeaway With fixed architecture and tokenizer, changing the pretraining data leads substantial shifts in loss-to-loss scaling laws; see Fig. 4.
Effect of Tokenizer Fig. illustrates the impact of to- kenizer choice on loss-to-loss scaling laws. To enable fair comparisons between models trained with different tokenizers, we measure performance using bits-per bits-per-byte (BPB) Gao al. (2020). In our experiments, only minor lokenizer-induced variations in scaling curves, as they remain closely aligned. However, the pretrained 'ablation models' from Penedo et al. (2024) (first and sec- ond columns) notably deviate from our curves. One key distinction is that the models from Penedo et al. (2024) uti- lize 'weight tying' between the token embeddings in the first layer and the output token predictions in the final layer, whereas our models do not. This difference in training setup could explain the observed discrepancy in the scaling curves. We leave a detailed investigation of this hypothesis for future work.
observe
4.1. Pretraining Data, Tokenizer, and Architecture
First, we jointly examine the effect of pretraining data, ar- chitecture, and tokenizer. Since we face limited compute to train models from scratch, we do not have checkpoints for all possible combinations of these factors. Instead, we analyze the effect of an intervention on each factor when matching models in the two other factors. Note that we do not have sufficient checkpoints for some Hugging Face models to fit power law. Nevertheless, in all these cases, the available data points follow a clearly discernible trend.
Takeaway 3 With fixed architecture and pretraining data, changing the tokenizer generally leads to minor changes in loss-to-loss scaling laws; see Fig. 5.
Effect of Pretraining Data Fig. illustrates the sub- stantial impact pretraining data has on loss-to-loss scaling. Across architectures and compute (in different columns), changing the pretraining data leads loss-to-loss curve. Notably, in the last column, compare Hugging Face models trained on The Pile versus its dedupli- cated variant. Models trained on these two datasets fall onto slightly different scaling curves, indicating that deduplica- tion has meaningfully altered the underlying distribution and thereby also affecting the loss-to-loss scaling.
large shift in the
Efffec of Architecture Lastly, Fig. 6 illustrates that chang- ing the architecture results in only very slight changes in the loss-to-loss curves across pretraining data and tokenizer settings. Unlike pretraining data and tokenizer, architecture has little influence on train-to- train and train-to-test scaling. This is particularly surprising given the significant architec- tural differences between Llama or GPT (transformer-basee
5

---

LLMs the Line
Architecture Llama Pretraining
Architecture Llama Pretraining
Architecture Liama Pretraining Pile
Architecture Mamba Pretraining
Architecture Mamba Pretraining Pile UC
Architecture GPT Pretraining Pile
Train-to to Train
Average Val. BPS
1.4
1.2
1.0
0.8
Average Test BPB
Train- Test
2.0
1.5
0.8
1.0
0.6
0.8
1.0
0.8
1.0
0.8
1,0
1.0 FW-Edu Val. BPB
0.8
1.0
1.2
FW-Edu Val. BPB
FW-Edu Val. BPB
FW-Edu Val. BPB
FW-Edu Val. BPB
FW-Edu Val. BPB
Tokenizer
gpt-neax
gpt2
gF12-HF
tiktaken
Figure The tokenizer has a minor impact on loss-to-loss scaling laws. Models are matched on pretraining data and architecture.
models) and Mamba (a state-space model). These results raise an important question: Do current architectures encode distinct inductive biases or converge to similar solutions given the same training data? Further research is needed to understand the implications of this finding.
These findings align well with Du et al. (2025), who ob- served that model size has little effect on loss-to-loss scaling for GPT models. We extend this conclusion to Llama and Mamba and across multiple pretraining distributions.
Effect of Context Length We next investigate the effect of varying the context length between 1024, 2048, and 3076 tokens. As shown in Fig. 8, this change does not meaning- fully affect the loss-to-loss scaling curves.
Takeaway 4 With fixed pretraining data and tokenizer, changing the architecture has limited impact loss-to- loss scaling laws raising questions about the distinc- tiveness of their inductive biases; see Fig. 6.
Effect of Optimization Settings Finally, we evaluate a range of common optimization settings: We consider the Adam (Kingma & Ba, 2017) and AdamW (Loshchilov & Hutter, 2019) optimizers, cosine (Loshchilov & Hutter, 2017) and WSD (Hu et al., 2024) schedules, learning rates of 3 and 3e-3, and a weight decay of 0.1 or 3.3e-2. In our training setup, models using the Adam optimizer generally did not converge, and we exclude them from the analysis. Variations of the other settings do not affect loss- to-loss scaling coefficients, as shown in Fig.
4.2. Model Size, Context Length, and Optimization
We now examine the effect of other common design de- cisions, such the number width of layers, the text length, optimizer, learning schedule, learning rate, and weight decay. In contrast to $4.1, we can perform these interventions separately since we can compare among our own Llama and Mamba models whose training settings are matched by default.
To provide more succinct overview, we only show train-to- train scaling laws in this section; additional train-to-test scal- ing laws for the same intervention can be found in App. H. We also do not show fitted power laws here since we display many more models per plot than in 1, and the scaling trends are clearly discernible.
Takeaway 5 Model size (Fig. 7), context length (Fig. 8), and optimization settings (Fig. 9) have negligible impact loss-to-loss scaling laws.
Effect of Model Size We first examine the influence of model size by training Llama and Mamba models with vary- ing depths and widths (see App. C for details). Fig. shows the results: Despite significant differences parameter count, the loss-to-loss scaling trends remain unchanged.
Given the limited impact of the factors studied in this section, the conclusions from $4.1 should generalize well across variations in model size, context length, and optimization settings. For example, the substantial impact the pretrain- ing distribution can also be observed in Figs. 7 and 8.
6

---

LLMs the Line
Pretraining C4
Pretraining C4 Tokenizer tiktoken
Pretrsining FW-Edu Tokenizer gpt2
Pretreining FW-Edu Tokenizer tiktoken
Pretraining Pile UC Tokenizer tiktoken
Pretraining Pile Tokenizer gpt-neox
Tokenizer gpt2
5.0
Loss
4.5
Train-to to
Average
4.0
Average Test Loss
Train- -ost Test
6
5
3.0
3.5
4.0
3.5
3.0
3.5
4.0
3.0
3.5
4.0
3.5
4.0
4.5
3.25
3.50
3.75
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
Architecture
GPT
Llama
Mamba
Mamba2
Figure 6. Architecture has limited impact on loss-to-loss scaling laws. Models are matched on pretraining data and tokenizer.
Architecture
Liama Mamba
Architecture Llama Mamba
Tokenizer tiktoken
Tokenizer
tiktoken
Size
Context Length
1e8
3000
Average Val. Loss
7
2500
6
Average
5
2000
1500
FineWeb-Edu Validation Loss
FineWeb-Edu Validation Loss
Pretraining Data
Pretraining Data
C4
Ein
The Pile UC
FineWeb-Edu
The Pile
Figure 7. Model size does not affect loss-to-loss scaling. The distinet lines correspond different pretraining distributions (see Fig. 4), reinforcing that their influence is consistent across scales.
Figure 8, Context length does not affect loss-to-loss scaling, Again, distinct lines correspond different pretraining distribu- tions (compare Fig. 4), validating their consistent impact.
6. Discussion and Future Work
5. Point-wise Comparisons
Our findings add to the understanding of loss-to-loss seal- ing laws and reinforce prior results from vision and vision- language research (Taori et al., 2020; Fang et al., 2022) on the importance of choosing the pretraining data.
In our experiments so far, we have demonstrated that nearly all factors we investigated-exeet the choice of pretraining data-have minimal to no effect on loss-to-loss scaling. This implies useful generalization: if two distinct training setups achieve similar training or validation losses on the same pretraining data, they will exhibit similar test losses across many downstream tasks. To explicitly illustrate this point, we present detailed point-wise comparisons between Mamba and Llama models trained on identical datasets in Table 1. highlighting their closely matched downstream performance.
Implications for Optimizing Downstream Performance Our results emphasize that the data distribution is the key for achieving desireable loss-to-loss scaling and in turn achieve a great downstream performance. Conversely, since architecture has little impact on the train-to-test conversion, it can be freely optimized for better compute scaling without affecting downstream scaling or performance.

---

LLMs on the Line
Table Comparison of Mamba and Llama models. Models are compared after seeing approximately 8 Billion Tokens of their pretraining data, at two different model sizes. We observe similar downstream validation and test losses. PT: Pretraining, SPJ: SlimPajama. RW: RetinedWeb, FW-E: Web-eeebb-dee ARC-C: ARC- CRC-Challenge, and ARC-E: ARC-Easy.
Model Type
Size
PT. Data
Validation Losses FW-E
Test Task Losses HellaSwag
Avg
Avg Test
RW
ARC-C
ARC-E
MMLU
PIQA
Task
Mamba
421M
FW-Edu
3.66
4,02
2,90
3.77
3.78
5.06
5.27
3.01
2.14
3,94
3.63
3.88
Llama
417M
FW WWdEdd
3.66
3.89
2,90
3.74
3.72
5.26
5.73
3.02
2.24
3.96
3.58
4.04
Mamba
421M
3.70
2.75
3.32
3.76
3.50
5.58
6.07
3.23
4.51
4.14
3.41
4.70
Llama
417M
Pile UC
3.71
2.75
3.32
3.77
3.49
5.55
5.96
3,26
4.46
4.18
3.41
4.68
Mamba
421M
C4
3.17
4.13
3.31
3.85
3.61
5.69
6.34
2.94
5.04
3.73
3.61
4.75
Llama
417M
C4
3.17
4.05
3.29
3.82
3.57
5.63
6.25
2.95
5.33
3,76
3.58
4.78
Mamna
368M
FW.Fdu FW-Edu
3.74
4.15
2.98
3.85
3.86
5.13
5.42
3,10
3.26
4.01
3.72
4.18
Llama
365M
3.74
3.98
2.98
3.84
5.39
5.78
3.10
3.52
4.04
3.67
4.36
Mamba
368M
Pile UC
3.78
2.83
3.40
3.85
3.59
5.75
6.35
3.31
5.08
4.17
3.49
4.93
Llama
365M
Pile UC
3.79
2.82
3.41
3.86
3.58
5.97
6.56
3.32
4,96
4.28
3.49
5.02
Mamba
368M
3.25
4.23
3.39
3.92
5.92
6.66
3.03
5.45
4.97
Llama
365M
C4
3,27
4.13
3,38
3.91
3.65
5.77
6.33
3.05
5,56
3.90
3.67
4.92
Architecture
Liama| | Mamba Tokenizer tiktaken Pretraining
to converge toward comparable solutions when the train- ing loss approaches zero (Roeder et al., 2020). However, even checkpoints of our smaller models, when trained on fewer tokens, follow the same scaling across architectures. Understanding whether this implies representational and behavioral similarity remains an intriguing open question. Beyond this, it remains to be seen whether it is possible to formulate architectures that fit the data well but exhibit different scaling trends
FineWeb-Edu
12
Optinder [R] Malgin Sohedeer
Aden 2e
Cove
Adam
WDD
Adon 1- 110m-1 Wab
Adon 130ed 133ed Wab
Average
8
Adm 13633 11bx1 WSD
Adamt 120e3
[23b22
6
Adomm 12003 |1.b= Codds
Akmmh 130e4 133:21 1W0
Adamt 330e 1:3e
1950
On Other Training Paradigms Our study intentionally focuses models trained with standard loss functions and conventional training settings to guide practitioners. The limited impact of existing paradigms does not preclude in- novative training approaches from improving loss-to-loss scaling. fact, a recent work by Saunshi et al. (2024) demonstrates that gradually increasing model depth and ini- tializing based on layers from a smaller model produces markedly different scaling behavior, particularly in how perplexity translates downstream accuracy. Similar struc- tured growth approaches could offer new pathways for im- proving scaling efficiency and generalization for decoder- only LLMs trained with next-token prediction. Additionally, in $4.1 we observed that weight tying could play a potential role in shifting these generalization curves. We leave these analyses for future work.
4
10
12
FineWeb-Edu Validation Loss
Figure 9. Optimization settings not affect loss-to-loss scaling.
Implications for Balancing Performance If the aim is not only optimal average downstream performance but also a specific weighting between different tasks, e.g., to ensure a balanced downstream performance, individual train-to-test scaling laws can be used to tune a model's performance. Here, too, the pretraining data has the largest impact and practitioners should thus consider the final application of their model already during the data curation stage. Ulti- mately, our findings underscore that pretraining data cura- tion, rather than architectural innovation, can be the primary driver in developing robust, generalist models.
On the Exhaustiveness of Interventions in $4.1 Our study clearly distinguishes between factors with substantial and limited impact on loss-to-loss scaling. While our con- clusions inherently shaped by the specific settings explored, the observed trends provide strong empirical evi- dence for these distinctions. Given the strong and consistent impact of pretraining data and tokenizer, we can confidently conclude that these interventions affect loss-to-loss scaling. While we observed only a limited impact of the architecture,
The limited impact of even dras- tically different architectures on loss-to-loss scaling behay- ior illustrated in $4.1 and Fig. 6 suggest that architectures trained on the same data may implicitly learn highly similar representations. This might seem intuitive, as all models minimize the same loss function. One might expect them
On Architectural Biases
8

---

LLMs on the Line
this effect was also consistent across major state-of-the-art architectures including Llama, GPT, and Mamba- which collectively represent the dominant paradigms in large-scale language modeling. Given this exhaustive set, it is hard to argue that other architectures would meaningfully alter loss-to-loss scaling.
7. Conclusion
In this work, we systematically investigate loss-to-loss scal- ing in LLMs, identifying key factors that shape its behavior. Our large-scale interventional analysis - spanning over 6000 model checkpoints across architectures, tokenizers, and training setups reveals that loss-to-loss scaling con- sistently follows shifted power-law trends, enabling predict- ing test performance from training loss.
On the Exhaustiveness of Interventions in $4.2 Across the wide range of size configurations (App. C) we test, all models exhibit very consistent loss-to-loss scaling. Simi- larly, the effect we observed for different context lengths very consistent within our test range (1024, 2048, 3076), which aligns with commonly used configurations (Black et al., 2021; Wang & Komatsuzaki, 2021; Biderman al., 2023; Penedo et al., 2024; Black et al., 2022). While we acknowledge the possibility that larger models or longer context lengths could influence loss-to-loss scaling, such an effect- - if present - is unlikely. For optimization set- tings, we again consider configurations widely used in LLM training (Shoeybi et al., 2020; Karpathy, 2022; Videau et al., 2024), including variations in optimizer type, learning rate, weight decay, and scheduling. While our results indicate that these choices do not meaningfully alter loss-to-loss scal- ing within the explored settings, we acknowledge that the space of optimization techniques is vast, and our list is not exhaustive. It remains possible that a principled optimiza- tion strategy, different from current best practices, could induce new scaling behaviors. However, our findings sug- gest that optimization settings are not primary driver of loss-to-loss loss scaling trends within the bounds of conventional language model training.
We identify pretraining data and tokenizer as the dominant factors shaping these scaling laws, highlighting the impor- tance of data curation. Architecture has limited impact, with models as different as LLaMA (transformer- -aanerd and Mamba (a state-space model) exhibiting nearly identical scaling when trained on the same data and tokenizer. Model size, context length, and optimization settings have negligi- ble influence, such that loss-to-loss scaling remains stable across different configurations.
Our findings underline the importance of pretraining data for downstream performance and robustness and suggest that different LLM might share similar architectural biases. Given our observations, practitioners should prioritize cu- rating high-quality pretraining data to optimize downstream performance, while architectures and training settings can be adjusted freely for efficiency.
On the Goodness of Fit of Scaling Laws For given model size, we train on number of tokens up to the Chinchilla-optimal amount (e.g., 8.4 tokens for a 420 M parameter model) and maintain a constant warmup of 5000 steps, a learning rate of 3,, and one-cycle cosine decay schedule. We use intermediate checkpoints as a proxy for models trained on fewer tokens. While (Hoffmann et al., 2022) suggest adapting the scheduler in this case align with the number of tokens, we constrained by compute and cannot train thousands of models from scratch. While using intermediate checkpoints alongside models of dif- ferent sizes may influence the specific shape of the fitted compute-to-loss scaling laws, we find that the overall quality of fit benefits greatly from the additional data points. Since we only use compute-to-loss scaling laws to estimate the entropy terms in Eq. (1) and are interested primarily in the impact of interventions on loss-to-loss scaling, we do not expect this choice to significantly impact our conclusions. We also not that using intermediate checkpoints for fitting scaling laws is not unprecedented in the literature (Schaeffer et al., 2024; Brandfonbrener et al., 2024).
9

---

LLMs on the Line
Impact Statement
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language, 2019. URL https: / 1911.11641.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Black, S., Gao. L., Wang, P., Leahy, C., and
Biderman, S. Gpt-neo: Large scale autore- gressive language modeling with mesh-tensorflow. 2021. URL https://api ttttts:://api..mmattt. semant semanticscholles cscholar. oxg//oorpusD:224755777
Author Contributions
This project was co-led and coordinated by PM and TW. TW and PM jointly developed the method, incorporating insights from WB and MB. PM was responsible for training and evaluating the models. SM assisted with certain evaluations and provided some details for Hugging Face models. TW analyzed the data and conducted the scaling law and loss- to-loss fits. Both TW and PM contributed to writing the manuscript, with feedback from WB. TW created all the figures with feedback from PM.
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., Hc, H., Leahy, C., McDonell, K., Phang, J., Pieler, M., Prashanth, U. S., Purohit, S., Reynolds, L., Tow, J., Wang, B., and Weinbach, S. Gpt-neox-20b- An open-source autoregressive language model, 2022. URL https://arxie: 2/0p://aw.....r//aa///2//02424.. 2204...455.
Brandfonbrener, D., Anand, N., Vyas, N., Malach, E., and Kakade, S. Loss-to-loss prediction: Scaling laws for all datasets, 2024. URL https://aet https:///xxxv.vrvr oab/
Acknowledgements
We would like to thank (in alphabetical order) Ameya Prabhu, Attila Juhos, Evgenia Rusak, Fanfei Li, Jack Brady, Thomas Klein, and Vishaal Udandarao for helpful discus- sions and feedback.
2411.12925.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https: wwwaaiiiooggaaaaaaaeea.a. 1803. 05457.
This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: O1IS180394 WB acknowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1 and via the Open Philantropy Foundation funded by the Good Ventures Foundation. WB is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project num- ber 390727645, This research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG.
Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. URL abs Abs/2405.0 21060.
Dodge, J. Sap. M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M.. and Gardner, M. Doc- umenting large webtext corpora: case study on the colossal clean crawled corpus, 2021. URL https: / aaaariivvvo.mmm //aaaiiioggaaabbb........... /ttzgg/ab////...... 08758.
We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting PM and TW.
Du, 7., Zeng, A., Dong, Y., and Tang, J. Understanding emergent abilities of language models from the loss per- spective, 2025. URL https: /atpaaaaivv.r../..a.....//... abs/ 2403.
References
Awadalla, A., Wortsman, M., Ilharco, G., Min, S., Mag- nusson, I., Hajishirzi, H., and Schmidt, L. Exploring the landscape of distributional robustness for question an- swering models, 2022. URL httpttttt//// https://arxiv.ort abs/2210.122
Fang, A., Ilharco, G., Wortsman, M., Wan, Y., Shankar, V., Dave, A., and Schmidt, L. Data determines distri- butional robustness in contrastive language image pre- training (clip), 2022. URL https: https://arxiv.. /ttp//xxxvv..r///// abs/2205. 013997
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https: //ar:i///xvvi org/ abs///////////// 4 01373.
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., Xin, R., Nezhurina, M., Vasiljevic, I., Jitsev, J., Sol- daini, L.,, Dimakis, A. G., Ilharco, G., Koh, P. W. Song,
10

---

LLMs on the Line
S., Kollar, T., Carmon, Y., Dave, A., Heckel, R., Muen- nighoff, N., and Schmidt, L.. Language models scale reli- ably with over-training and on downstream tasks, 2024. URL https: / arwivv.ivvvv/////ivi//...aaaaaav org/ abs/2403 08540.
multitask language understanding, 2021. URL https: llaaaxiv.orggaa/////.s.essao 03300.
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically, 2017. URL www.h:tt////et btttss://axxiv.oott /ab/////////// 22. 00409.
Gao, L., Biderman, S., Black, S., Golding, L.,, Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https: // //arxivvom org/ abs/ 21 01 00027.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E.,, Rae, J. W., Vinyals, O., and Sifre, L. Training compute-optinnl large language models, 2022. URL https: peettps:/aaxxii.orrt arriiv.rgg////// a abs abs/20 2203. 5556.
Gao, I.., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Noac"h, ha A. L., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang. B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL www.thttp........... wtt.d/eeeddd.oo/ooo org/ records/ 12608602.
Hu, S., Tu, Y., Han, X., He. C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., Zhang, X., Thai, Z. L.,, Zhang. K., Wang, C., Yao, Y.. Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng, G., Li, D., Liu, Z., and Sun, M. Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024. URL https: / arxiv. .rgg /abs///////// 24044 06395.
Gordon, A. S., Kozareva, Z., and Roemmele, M. Choice of plausible alternatives: An evaluation of common- sense causal reasoning. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, 2011. URL https://aat http:s//pp....mmmtiiccc....//// wwwanticsccool............. org/CorpusID: 434646.
Isik, B., Ponomareva, N., Hazimeh, H., Paparas, D., Vassil- vitskii, S., and Koyejo, S. Scaling laws for downstream task performance of large language models, 2024. URL https /tte:///aaaiiii org/ /2tp://.s//22...... 04177.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., et al. The Ilama herd of models, 2024. URL wwthtttps//// arxiv. http://wwwv..aa.......////a 2tt////7.227.2..
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, Scaling laws for neural language mod- els, 2020. URL pattts::/arree 08361.
Groeneveld, D., Beltagy, I., Walsh, E., Bhagia, A., Kin- ney, R., Tafjord, O., Jha, A., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M., Pyatkin, V., Ravichan- der, A., Schwenk, D., Shah, S., Smith, W., Strubell, E., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N., Richardson, K., Zettlemoyer, L.., Dodge, J., Lo, K., Sol- daini, L., Smith, N., and Hajishirzi, H. OLMo: Acceler- ating the science of language models. In Ku, L.-W., Mar- tins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties VVllume 1: Long Pupers), pp. 15789-15809, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10.18653//1/00224ac-oooo tattt 841. URL https: http::ss///ccaaatoooo.....////// wwg.:///oorrr//2/2////// acl-long 841/.
Karpathy, A. nanogpt, 2022. URL https://githhi:
fonn/karpathy/nanoon
Kingma, D. P. and Ba, J. Adam: A method for stochastic op- timization, 2017. URL https: 1412.6980.
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradi- ent descent with warm restarts, 2017. URL https: //arxiv. org. wwtp/oogggaabaaoaeeeeeeee 011.//6688.008.....
Loshchilov, and Hutter, Decoupled weight decay regu- larization, 2019. https:/:::// /abs//////
Madaan, I.,, Singh, A. K., Schaeffer, R., Poulton, A., Koyejo, S., Stenetorp. P., Narang, S., and Hupkes, D. Quantifying variance in evaluation benchmarks, 2024. URL https: wwwii.ovooo...oooo.o wwwp/aab/240..122222
Gu, A. and Dao, T. Mamba: Linear-time sequence mod- eling with selective state spaces, 2024. URL https: / //arxiv.or. orgo abs/2312.... 00752.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring massive
Mayilvahanan, P., Wiedemer, T., Rusak, E., Bethge, M., and Brendel, W. Does clip's generalization performance

---

LLMs on the Line
mainly stem from high train-test similariit??, 2024a. URL https: / aariv. www.riiw.............o.. abs 231 09562.
Schaeffer, R., Schoelkopf, H., Miranda, B., Mukobi, G., Madan, V., Ihrahim, A., Bradley, H., Biderman, S., and Koyejo, S. Why has predicting downstream capabilities of frontier ai models with scale remained elusive?, 2024. URL pttpp:::.
Mayilvahanan, P., Zimmermann, R. S., Wiedemer, T., Rusak, E., Juhos, A., Bethge, M., and Brendel, W. In search of forgotten domain generalization, 2024b. URL https:/ /taa://axxiv www//org/e....ooooooo.oe 2410 08258.
.wi.orrrgooooo..ooooooo...oo.ooe abss////// / /200. 043
Shen, Z.. Tao, T., Ma, L.. Neiswanger, W., Liu, Z., Wang. H., Tan, B., Hestness, J., Vassilieva, N., Soboleva, D., and Xing, E. Slimpajama-dc: Understanding data combi- nations for I1m training, 2024. htt.t::t//// //axxvv.
Miller, J., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., and Schmidt, L. Accuracy on the line: On the strong correlation be- tween out-of-distribution and in-distribution generaliza- tion, 2021. URL https:://///. //tp///aavvvv/aa////a/eeee.awava org/ abs/ abc//77. 2107. 04 64 9.
org/ abs /0009 10818.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi- billion parameter language models using model par- allelism, 2020. URL https: https////xxvvoorr///.
Penedo, G., Malartic, Q., Hesslow, D.,, Cojocaru, R., Cap- pelli, A., Alobeidli, H., Pannier, B., Almazrouci, E., and Launay, J. The refinewweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only, 2023. URL https: /atpps://arae /arxiv...vv./////i///a.////avtvv org/ abs/ 2306. 01116.
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Com- monsenseqa: question answering challenge target- ing commonsense knowledge, 2019. URL https:
/ //arxivv.vv.//////// wttp:iiv.rg//b////e/ee/ http:///bbb111.0993.7.
Penedo, G., Kydlīček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL /tap:///axxvva wwtp:/org/aaaa//./ abs/ 2406.17557
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring robustness to natural distribution shifts image classification, 2020. URL https:// a arxiv xiv. org/ abs 2007 00644.
Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Car- mon, Y. Resolving discrepancies in compute-optimal scaling of language models, 2025. URL https:// arxiv. org/ abs //.//240..190 19146.
Tay, Y., Dehghani, M., Abnar, S., Chung, H. W., Fedus, W., Rao, J., Narang, S., Tran, V. Q., Yogatama, D., and Metzler, D. Scaling laws vs model architectures: How does inductive bias influence scaling?, 2022. URL https: /ttp//aaxvvvv/vvv/////////a///aa www.oorgga/o.ooo..ooooo.o abs 2207 10551.
Radford, A., Wu, J. Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsu- pervised multitask learners. 2019. URL https: //api. www.stiisccoolleeeeeeee 160025533.
Videau, M., Idrissi, B. Y., Haziza, D., Wehrstedt, L., Copet, J., Teytaud, O., and Lopez-Paz, D. Meta Lingua: A mini- mal Torch LLM training library, 2024. URL https: //github. com/ www.faacoceoeearrcco.ooo /innuu..
org/CorpusID: htw.://ccccooaa.oooooorrpppsaaaa
Roeder, G., Metz, L., and Kingma, D. P. On linear identifiability of learned representations, 2020. URL. btttps://arxtt
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C. Polat, i., Feng, Y., Moore, E. W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, F. A., Harris, C. Archibald, A. M., Ribeiro, A. H., Pedregosa, F. van Mulbregt, P., and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing Python. Nature Methods, 17:261-272, 2020. doi: 10. (0033//41592-01--0000
httg//bs//2000/ 00810.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https: // /wwxxvivvvv...////////. orgoorgggao abs/ 1907.10642
Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y. Socialiqa: Commonsense reasoning about social in- teractions, 2019. http::/s://aat 1904.09728.
abs/
Wang, B. and Komatsuzaki, A. Gpt-j-6b: A billion param- eter autoregressive language model, 2021.
Saunshi, N., Karp, S., Krishnan, S., Miryoosefi, S., Reddi, S. J., and Kumar, S. On the inductive bias of stacking towards improving reasoning, 2024. URL https:// arxiv. w.v.vorg/aaee http//bb//244...
Wang. S., Chen, Z., Li, B., He, K., Zhang, M., and Wang. J. Scaling laws across model architectures: comparative
12

---

LLMs on the Line
analysis of dense and moe models in large language mod- els, 2024. / abs 2410.. 05661.
Wiedemer, T., Sharma, Y., Prabhu, A., Bethge, M., and Brendel, W. Pretraining frequency predicts compositional generalization of CLIP on real-world tasks. In NeurIPS 2024 Workshop Compositional Learning: Perspectives, Methods, and Paths Forward, 2024. URL https:// /// openreview ew www. forum? dddNdDo wYgl. wYgl.
Wolf, T., Debut, L.., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C.. Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Huggingface's ' transformers: State-of-the-art natural language processing, 2020. URL https: http///axvvv/.e///ee org/ abs 910. 03771.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and
Choi, Y. Hellaswag: Can machine really finish your sentence?, 2019. URL https: //a/ //axxiv....///////////////////v/ org/ /bs/ 1905.07830.
13

---

LLMs on the Line
A. Scaling Law Details
We adopt the compute-to-loss scaling law formulation from Brandfonbrener et al. (2024) Eq. (4):
L (f(N.D)) =E+ E+ =E+
(((
A
$ +
B D
9
(2)
where f(N.D) is a model with parameters trained on tokens and E, A, B, a. 3 are parameters to be fit. Notably, the irreducible error E captures the minimum loss possible for model f in the limit of infinite model and data size.
By default, Eq. (2) is fit using the training validation loss. However, as demonstrated by Brandfonbrener et al. (2024) and our experiments, we can alternatively predict the loss Lg on dataset Dr achieved by model fp ((N.D) trained on the pretraining set D,:
0
La (H.DDD) EElp
A N
B +
(3)
D
As in Brandfonbrener et al. (2024) Eq. (7), the irreducible error
Ezpp = (z(f))
(4)
then captures the minimum possible loss on D₂ of a model trained on Dp-
With that, we can formulate the loss-to-loss scaling law for arbitrary combinations of pretraining data and two test or validation sets, stated in Eq. (1).
B. Fitting Details
As explained in $3, we fit loss-to-loss scaling law in Eq. (1) by collecting 200 to 800 model checkpoints throughout training and across model sizes and seeds.
For each line in a plot corresponding to loss-to-loss scaling law from Eq. (1), we first fit the two compute-to scaling laws L. (. ()) and L, given by Eq. (3). This yields estimates for the irreducible errors Exlps Eylp- which correspond to the minimum x- and y-value of the loss-to-loss line. We use SciPy's default curve_ ff optimizer for fitting (Virtanen et al., 2020). In rare cases when all checkpoints have the same number of parameters N or same number of tokens D (this is the case only for small subset of the Hugging Facemodels) and a compute-tor to-loss scaling law cannot be fitted, we instead estimate the irreducible error the minimum loss achieved:
min To (1/ (F(N.D)).
Ezlp
(5)
N.D
We show example compute-to-loss fits for some of the loss-to-loss scaling laws from Fig. 2 in Fig. 10. With Exlps Egip from the compute-to-loss fits, we again SciPy's curve.fit fit K, x for the loss-to-loss scaling law from Eq. (1).
C. Model Details
Table 2 supplements the discussion of the models used for our study from $4.1 with additional details.
D. Quantitative Analysis of Interventions
We quantify the impact of different interventions as the area between fitted curves in Table 4. Pretraining data clearly has the biggest impact on the scaling laws.
E. Loss-to-Loss Scaling Across Settings
We supplement Fig. from $3 with additional architecture- urchitectur--petrrriinng pairings in Figs. 11 16.
14

---

LLMs on the Line
FineWeb-Edu Validation Loss
The Pile UC Validation Loss
1e9 HellaSwag Loss
1e9
1e9
8
8
6.5
4.5
6.0
6
4.5
6
4.0
5.5
4.0
4
4
5.0
2
2
4.5
2
3.5
3.0
4.0
2
4
N
1e8
N
1e8
N
1e8
Figure 10. Example compute-to-loss scaling law fits. Each loss-to-low scaling law requires fitting two compute-to-lon to-los scaling laws estimate Exlp, Hylp. The three fits here used for the The Pile UC and HellaSwag curves in Fig. which showed curves for FeneWeb-Edu-traie Mamba, i.e, p is FineWeb-Edu. All curves in Fig. use FineWeh-Edu as the x-axis; the corresponding E, p is given by the fit depicted in the left-most plot. Eglp for y The Pile UC and HellaSwag given by the fits in the center and right plot, respectively.
Table Architecture of the models trained from scratch. All models generally the tiktoken tokenizer. In addition these models, for the tokenizer ablation $4.1, we train the largest Llama and Mamba models with the gpt2 tokenizer with beginning and end of sequence tokens.
Architecture
Width
Depth
Parameters
Llama
1024
12
416 M
Llama
1024
8
365 M
Llama
1024
4
314 M
Llama
512
12
172 M
Llama
512
8
158
Llama
512
145
Llama
256
12
76 M
Llama
256
8
72 M
Llama
256
4
59 M
Mamba
1024
24
420
Mamba
1024
16
367
Mamba
1024
8
315 M
Mamba
512
24
172 M
Mamba
512
16
158 M
Mamba
512
8
145 M
Mamba
256
24
76 M
Mamba
256
16
73
Mamba
256
69
15

---

LLMs on the Line
Table Details the pretrained models used. All models do not use beginning sequence tokens.
Name
Pretraining data
Tokenizer Class
Parameters
GPT-J
The Pile
GPT2
6B
GPT-Neo
The Pile
GPT2
{ [125M, 1.3B, 2.7B)
Web Ablation Models
The Pile
GPT2
1.7B
FineWeb Ablation Models
GPT2
1.7B
FineWeb Ablation Models
Web-Web-Edu
GPT2
1.7B
GPT-NeoX
The Pile
GPT-NeoX
20B
Pythia (small)
The Pile
GPT-NeoX
(70M, 160M, 410M)
Pythia (large)
The Pile
GPT-NeoX
{IB, 1.4B, 2.8B, 6.9B, 12B)
Mamba
The Pile
GPT-NeoX
{130M, 370M, 790M, 1.4B, 2.8B)
Mamba2
The Pile
GPT-NeoX
[130M, 370M, 790M, 1.4B, 2.8B)
Train-to- Train-to-Train
Train-to- Wein-to-Teest
Validation Loss
5.5
Test Loss
5.0
5
4
2
3
4
5
FideWeb-Edu Validation Loss
FineWeb Edu Validation Loss
Validation Set
Test Set
The Pile UC
ARC-Challenge
RefineWeb
ARC-Easy
Slimpajama
COPA
C4
PIQA
Winogrande
HellaSwag
CommonSenseOA
Social IQa
MMLU
Figure 11. Loss-to-Loss Scaling for PineWeb-Edu--iee Llama.
16

---

LLMs on the Line
Train-to-Train
Train-to-Tend
Validation Loss
5.5
TestLoss
9
4.5
6
4.0
4
3
4.0
3.5
4.5
C4 Validation Loss
C4 Validation Loss
Validation Set
Test Set
The Pile UC RefineWeb
ARC-Challenge
ARC-Easy
Slimpajama
COPA
FineWeb-Edu
PIQA
Winogrande
HellaSwag
CommonSense@A
Social IQa
MMLU
Figure 12. Loss-to-Loss Scaling for C4-trained Llama.
Train-to- Trainn
Train-to-Teed
5.5
8
Validation Loss
Test
4.0
6
5
3.5
3.5
4.0
4.5
5.0
FineWeb-Edu Validation Loss
FineWeb-Edu Validation Loss
Validation Set
Test Set
The Pile UC
ARC-Challenge
RefineWeb
ARC-Easy
Slimpajama
COPA
PIQA
Winogrande
HellaSwag
CommonSense@A
Social IQa
MMLU
Figure 13. Loss-to-Loss Scaling for The Pile-trained Llama.
17

---

LLMs on the Line
Train-to-Tran
Train-to- Test
Validation Loss
5.5
Test Loss
5.0
5
4
3.5
FineWeb-Edu Validation Loss
FineWeb-Edu Validation Loss
Validation Set
Test Set
The Pile
ARC-Challenge
RefineWeb
ARC-Easy
Slimpajama
COPA
PIQA
Winogrande
HellaSwag
CommonSenseQA
Social IQa
MMLU
Figure 14. Loss-to-Loss Scaling for trndtaee-Ed Mamba.
Train-to-Train
Train-to-
5.5
Validation Loss
5.0
Loss
4.5
5
4
3.5
4.0 Validation Loss
4.5
C4 Validation Loss
Validation Set
Test Set
The Pile UC
ARC-Challenge
RefineWeb
ARC-Easy
Slimpajama
COPA
FineWeb-Edd
PIQA
Winogrande
HellaSwag
CommonSense@A
Social IQa
MMLU
Figure 15. Loss-to-Loss Scaling for C4-trained Mamba.
18

---

LLMs on the Line
Table Area between fitted BPB for different interventions, evaluated the interval [0, 2].
Base
Tokenizer
Data
Architecture
Optimizer
Size
Context
Base
0.01
0.22
0.02
0.02
0.02
0.02
Tokenizer
0.01
0.19
0.03
0.03
0.03
0.03
Data
0.22
0.19
0.17
0.19
0.19
0.19
Architecture
0.02
0.03
0.17
0.02
0.02
0.02
Optimizer
0.02
0.03
0.19
0.02
0.00
0.01
Size
0.02
0.03
0.19
0.02
0.00
0.00
Context
0.02
0.03
0.19
0.02
0.01
0.00
F. Intervention Results for Different Choices of x-Axis
We show variations of Figs. 4 to 6 from $4.1 with C4 validation loss as the x-axis in Figs. 17 to 19. Variations for Figs. 7 to from $4.2 are shown in Figs. 20 to 22.
G. Intervention Results without Averaging
The causal analysis in $4 was performed scaling laws for average validation or test loss. Figs. 23 to 28 show illustrative results on scaling laws for individual datasets.
H. Additional Train-to-Test Scaling Laws
We provide train-to-ted scaling laws for the interventions performed in $4.2 and Figs. in Figs. 29 34.
19

---

LLMs on the Line
Train-to-Tran Train
Train-to- Test
5.5
Validation Loss
4.5
Test Loss
6
4.0
3.5
3.5
4.0
4.5
5.0
Eid Validation Loss
FineWeb Edu Validation Loss
Validation Set
Test Set
The Pile UC RefineWeb
ARC-Challenge
ARC-Easy
Slimpajama
COPA
C4
PIQA
Winogrande
HellaSwag
CommonSenseQA
Social IQa
MMLU
Figure 16. Loss-to-Loss Scaling for The Pile-trained Mamba.
Architecture Llama Tokenizer tiktoken
Architecture Llama Tokenizer
Architecture Liama Tokenizer gpi2-HF
Architecture Mamba Tokenizer tiktoken
Architecture Mamba Tokenizer gpl2
Architecture GPT Tokenizer gpt-neox
Average Val. Loss
Train-to-Train
Average Test Loss
Train-to-Test Test
6
Anahe 5
4.0
5
3.5
4.0
4.0
4.5
5
3.0
3.5
Val. Loss
C4 Val. Loss
C4 Val. Loss
C4 Val. Loss
C4 Val. Loss
C4 Val. Loss
Pretraining Data
C4
FineWeb Edu
The Pile
The Pile Deduped
The Pile
Figure Pretraining has substantial impact on loss-to-loss scaling laws.
20

---

LLMs on the Line
Architecture Llama
Architecture Liama Pretraining FW-Edu
Architecture Llama Pretraining Pile UC
Architecture Mamba Pretraining C4
Architecture Mamba Pretraining Pile UC
Architecture GPT Pretraining Pile
Pretraining C4
Average Val. BPB
1.4
Train-to- Trainnto-Trrinn Train
1.2
1.0
0.8
0.6
Average Test BPB
Train-to to-n-t Test
2.0
1.5
0.8
1.0
1.0
1.2
1.0
1.2
1.0
1.0
1.2
1.2
1.4
04 Val. BPB
04 Val. BPB
04 Val. BPB
C4 Val.
C4 Val. BPB
Val. BPB
Tokenizer
gpt-neox
gpt2
gp12
tiktoken
Figure 18.
The tokenizer a minor impact on loss-to-loss scaling laws.
Pretraining C4 Tokenizer gpt2
Pretraining C4 Tokenizer tiktoken
Pretraining FW-Edu Tokenizer gpt2
Pretraining FW-Edu Tokenizer tiktoken
Pretraining Pile UC Tokenizer tiktoken
Pretraining Pile Tokenizer gpt-nex
5.0
Average Val. Loss
Train-ta to Train
4.5
4.0
3.5
Average Test
6
Trainto
4.5
C4 Val. Loss
C4 Val. Loss
C4 Val. Loss
C4 Val. Loss
C4 Val. Loss
04 Val. Loss
Architecture
GPT
Llama
Mamba
Mamba2
Figure 19. Architecture has limited impact loss-to-loss scaling laws.
21

---

LLMs on the Line
Architecture Tokenizer
Llama Mamba tiktoken
Size
1e8
Average Val. Loss
7
3
6
2
4
4
7
C4 Validation Loss
Pretraining Data
C4
FineWeb-Edu
The Pile UC
Figure 20. Model size does not affect train-to-test scaling.
Architecture Tokenizer
Llama Mamba tiktoken
Context Length
3000
Average Val. Loss
6
2500
5
2000
1500
3
6
C4 Validation Loss
Pretraining Data
C4
FineWeb-Edu
The Pile UC
Figure 21. Context length does not affect train-to-test scaling.
22

---

LLMs on the Line
Architecture
Llama Mamba Tokenizer tiktoken Pretraining
FineWeb-Edu
12
Optibierr LR Weight
Schedaler
3004 3
Average Val. Loss
10
Adom 3.c3 3.c2
Cosine
Adam 3.we3 I 1.0w1
Oosine
1.0e1
Adam
3.0e3 33t-2 WSD
Cosine
AdamW 330ee4 1.0e:1 Cosina
Adamt)
33.w33 3.3e=2 Cosine
Adamtl 3.0e-4 3.3e-2 MSB
AdamW 3.0e-4 1.3e-1 WSD
330wa
AdamW
4
4
6
8
10
12
C4 Validation Loss
Figure 22, Optimizer settings do not affect train-to-test scaling.
Architecture Llama Tokenizer tiktoken
Architecture Llama Tokenizer gpt2
Architecture Llama Tokenizer gpt2-HF
Architecture Mamba Tokenizer tiktaken
Architecture Mamba Tokenizer gp12
Architecture GPT Tokenizer gpt-neax
Train-to to-irain Train
6
The Pile UC Val.
4
HellsSwag Loss
Train-to-Test
4
3.5
4.0
4.5
4.5 FW-Edu Val. Loss
3.5
4.0
3.5
4.0
4.5
3.5
4.0
4.5
3.5
4.0
4.5
3.0
3.5
Val. Loss
FW-Edu Val. Loss
Loss
EdVEduui Val. Loss
FW-Edu Val. Loss
Pretraining Data
C4
FineWebbEdd
The Pile
The Pile Deduped
The Pile UC
Figure 23.
Pretraining data has a substantial impact on loss-to-loss scaling laws.
23

---

LLMs on the Line
Architecture Liama
Architecture Liama Tokenizer gpt2
Architecture Llama Tokenizer gp12-HF
Architecture Mamba Tokentzer tiktoken
Architecture Mamba Tokenizer gpt2
Architecture GPT Tokenizer gpt-neox
Tokenizer tiktoken
Train-to Train
C4
3
Winagrande Test Loss
5.0
Train-o-
4.5
4.0
3.5
4.0
4.5
4.5
3.5
4.0
4.5
4.5
3.5
4.0
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
Pretraining Data
FineWebEEdu
The File
The Pile Deduped
The Pile UC
Figure 24.
Pretraining has substantial impact on loss-to-loss scaling laws.
Architecture Llama Pretraining C4
Architecture Liama Pretraining FW-Edu
Architecture Llama Pretraining Pile UC
Architecture Mamba Pretraining C4
Architecture Mamba Pretraining Pile UC
Architecture GPT Pretraining Pile
Val. BPB
1.5
Train-to Train
1.0
The Pile
0.5
HellaSwag Test BPB
1.4
Train-t- to-
1.2
1.0
0.8
1.0
FW-Edu Val. BPB
FW-Edu Val. BPB
FW-Edu Val. BPB
FW-Edu Val. BPB
FW-Edu Val. BPB
FW-Edu Val. BPB
Tokenizer
gpt-neox
gpt2
9P12-HF
Skttkken
Figure 25, The tokenizer has a minor impact loss-to-loss scaling laws.
24

---

LLMs on the Line
Architecture Llama
Architecture Liama Pretraining FW-Edu
Architecture Llama Pretraining Pile UC
Architecture Mamba Pretraining C4
Architecture Mamba Pretraining Pile UC
Architecture GPT Pretraining Pile
Pretraining C4
1.4
Train-to-Train Train
Train-to
1.2
1.0
0.8
Winogrands mhograndde Test
1.6
Train-o
1.4
1.2
1.0
0.8
1.0
0.8
1.0
0.8
0.8
1.0
FW-Edu Val. BPB
FW-Edu Val. BPB
FW-Edu Val. BPB
FW-Edu Val. BPB
FW-Edu Val. BPB
FW-Edu Val. BPB
Tokenizer
gpt-neox
gpt2
9p12+HF
Sktokknn
Figure 26.
The tokenizer a minor impact loss-to-loss scaling laws.
Pretraining Tokenizer gpt2
Pretraining Tokenizer tiktoken
Pretraining FW-Edu Tokenizer gpt2
Pretraining FW-Edu Tokenizer tiktoken
Pretraining Pile UC Tokenizer tiktoken
Pretraining Tokenizer gpt-neox
The UC Loss
Train-to Train
5
4
Hel Test Loss
1.0
Train
3.0
2.5
3.0
3.5
3.5 FW-Edu Val. Loss
3.50
3.75
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
Architecture
GPT
Llama
Mamba
Mamba2
Figure 27, Architecture has limited impact on loss-to-loss scaling laws.
25

---

LLMs on the Line
Pretraining C4
Pretraining C4 Tokenizer tiktoken
Pretraining FW-Edu gp12
Pretraining FW-Edu Tokenizer tiktoken
Pretraining Pile UC Tokenizer tiktoken
Pretraining Pile Tokenizer gpt-neox
Tokenizer gpl2
5.0
Train-to Train
Val. Logs
4.5
4.0
3.0
Winogrande Test Loss
5.0
Train-to- Test
4.5
4.0
20
3.5
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
FW-Ea WW-Edu Val. Loss
FW-Edu Val. Loss
FW-Edu Val. Loss
Architecture
GPT
Llama
Mamba
Mamba2
Figure 28.
Architecture has limited impact on loss-to-loss scaling laws.
Architecture Tokenizer
Llama Mamba tiktoken
Size
1e8
Average Test Loss
8
7
3
6
2
5
4
7
C4 Validation Loss
Pretraining Data FineWeb-Edu
C4
The Pile UC
Figure 29, Model size does not affect train-to-test scaling.
26

---

LLMs on the Line
Architecture Tokenizer
Llama Mamba tiktoken
Size
1e8
Average Test Loss
8
7
3
6
2
3
4
5
FineWeb-Edu Validation Loss
Pretraining Data
C4
FineWeb-Edu
The Pile UC
Figure 30. Context length does not affect train-to-fest scaling.
Architecture Llama Tokenizer
Mamba
tiktoken
Context Length
7.5
3000
Average Test Loss
7.0
2500
6.5
6.0
2000
5.5
1500
5.0
7
C4 Validation Loss
Pretraining Data
C4
FineWeb-Edu
The Pile UC
Figure 31. Optimizer settings do not affect train-to-test scaling.
27

---

LLMs on the Line
Architecture
Llama tiktoken
Mamba
Tokenizer
Context Length
7.5
3000
Average Test Loss
7.0
6.5
2500
6.0
2000
5.5
1500
5.0
3
5 FineWeb-Edu Validation Loss
4
Pretraining Data FineWeb-Edu
C4
The Pile UC
Figure 32. Model size does not affect train-to-test scaling,
Architecture
Llama Mamba Tokenizer tiktoken Pretraining
FineWeb-Edu
Optiizer
Scheduler
Adam
33+2
Adam
3.0e4
1.0e-
Costne
Average Loss
33e2
10
1.0c-1
Adam
3.u-4 93w WSO
Adom
3.004 | 1.0e1 WBD
Adam
3 1.0e1 WS0
AdamW
3.30e-4 3.3e-2 Cosine
3.0e-4
Dosine
Cosine
AdanWW /333w3 1.0w Coniie
Adanli 333e4 3.3w WSD
AdamW 3.0e- 3.3e:2 WED
Adami 3.30e-3 1.0e11 WSD
C4 Validation Loss
Figure Context length does affect train-to-test scaling.
28

---

LLMs on the Line
Architecture
Llama Mamba tiktoken Pretraining
Tokenizer
FineWeb-Edu
12
Optimber LR Weight 33
Schedider
11
Adam 3.0c4 | 1:0c1
Costne
Average Test I Loss
Adan 3 33e
Oosine
10
33+2
Adam 3 1.0e11 WSO
AdamW 330e-4 3.3e Cosino
Adanti
333e44 1.0w-1 Cosine
7
AdamtW 3.0w3
Cantne
AdamWi
3.0e-4 3.3e22 WED
AdanWW
3.3w 1.0e WSD
FineWeb-Edu Validation Loss
Figure Optimizer settings affect train-to-test scaling.
29


Reviews:
[{"review_id": "O7xPHbIqjq", "reviewer": "Reviewer_fuqS", "summary": "The paper investigates how loss-to-loss scaling (i.e. scaling laws between losses on different datasets) for LLMs is influenced by model architecture, tokenizer, and training datasets. The authors experimentally find that:\n\n1. loss-to-loss scaling consistently follows shifted power laws.\n2. The effects of pretraining data are more pronounced than the effects of model architecture and HPs.\n\n## update after rebuttal\"\nI will keep the current score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the helpful feedback. We address your concerns as follows:\n\n**[W1 & Q2a - On comparing raw numbers]**: Comparing individual performance metrics like MMLU is indeed possible and can illustrate the effectiveness of an intervention for a specific model scale and setting. For example, we added [Table 3](https://ibb.co/tpgMmP1Z) to the appendix, which shows the raw numbers for MMLU and several other tasks. Note that Figs. 22-27 in App. F already shows versions of Figs. 4-6 for specific validation and evaluation datasets (C4, The Pile UC, ARC-Easy, and HellaSwag).\nThat said, comparisons on a single model scale and setting or on a single test set fall short in understanding whether the effectiveness of an intervention is dependent on model size, dataset size, downstream task or other factors. Additionally, evaluations on a single test set like MMLU are noisy, as evident from [Fig. 10](https://ibb.co/Rpn1xJRX), [Fig. 11](https://ibb.co/WNkR7SV3), [Fig. 12](https://ibb.co/jvPSBB2N), [Fig. 13](https://ibb.co/gZmwBTz0), [Fig. 14](https://ibb.co/sdJ3nLWz), and [Fig. 15](https://ibb.co/Zpn5c6k8), which we have added to the appendix. Our comprehensive study addresses this by systematically evaluating performance across multiple scales, factors, and downstream tasks while rigorously controlling all training parameters (including learning rate, optimizer, context length, tokenizer, model and dataset sizes).\n\n**[W1 & Q2b On the utility of loss-to-loss scaling curves]**: Loss-to-loss scaling laws across datasets provide critical insights beyond single-setting performance comparisons, as detailed in our related work and discussion sections. We reiterate the most important points here and have updated the corresponding sections in the paper to state this more clearly:\n- **Generalization**: Loss-to-loss scaling laws (train-to-train, train-to-test, or test-to-test) provide insight into how performance transfers across datasets (Taori et al., 2020, Fang et al., 2022, Brandfonbrener et al., 2024). Specifically, these scaling laws help answer the generalization question: if a model achieves a certain performance on a given dataset (generally training dataset), how well does it do on another task/dataset? As **Reviewer EE76** rightly points out, this can be of “broader interest to the community, in terms of advancing our understanding of the role of architecture, optimization, etc. in downstream properties.”\n\n- **Compute Budget Translation**: By combining train-to-test scaling laws with compute-to-train scaling laws, we can more precisely understand how compute budget translates into downstream task performance (Brandfonbrener et al., 2024), and also help uncover emergent model abilities (Du et al., 2025).\n\n- **Factor Decomposition**: Separately analyzing compute-to-train and train-to-downstream scaling laws helps isolate factors influencing each step. For instance, in our work, we identify that **architecture and optimizer settings do not influence loss-to-loss scaling laws — but they do affect compute-to-train scaling laws** (Kaplan et al., 2020; Hoffmann et al., 2022; Brandfonbrener et al., 2024; Porian et al., 2024; Li et al., 2025). As a result, architectures and optimizers can be independently optimized to enhance compute scaling without negatively impacting downstream task performance.\n\n- **Limitations of Single-Dataset Validation**: As **Reviewer EE76** highlights, our findings caution against relying solely on one validation dataset's loss (as typical in data curation methods, e.g., (Liu et al., 2024)) — achieving a particular loss on one validation set can correspond to varying downstream task performances. Thus, a single validation dataset may not comprehensively indicate downstream efficacy.\nBy illuminating the factors that consistently impact loss-to-loss scaling laws, we enable practitioners and researchers to use them as a tool for analyzing and optimizing model training.\n\n**[Q1 - On the irreducible errors]**: We have added two Appendix sections that detail the scaling law formulation and explain how parameters are estimated; please refer to reply to **Reviewer EE76** for more details.\n\n**[Regarding the referenced paper]**: We are happy to include the suggested reference in our camera-ready version. As you rightly point out, the impact of pretraining data on loss-to-loss scaling laws has been shown before (Taori et al., 2020, Fang et al., 2022, Brandfonbrener et al., 2024), and our results further confirm this finding. However, to the best of our knowledge, our study is the first to show this comprehensively for a large number of datasets and model settings and is the **first to analyze the impact of other factors like architecture, optimizer settings, and context length**.\n\nWe hope this addresses your concerns fully and encourages you to raise your score.\n\n(Li et al, 2025) (Mis)Fitting: A Survey of Scaling Laws", "claims_and_evidence": "The claims are supported by evidence.", "methods_and_evaluation_criteria": "See the strengths and Weaknesses section.", "theoretical_claims": "na", "experimental_designs_or_analyses": "Experiments are sound", "relation_to_broader_scientific_literature": "na", "essential_references_not_discussed": "The paper discusses how the final loss depends upon pretraining data among other things. The recent paper \"Scaling Optimal LR Across Token Horizons\" also discusses how pretraining data will influence scaling properties.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is very clearly written, and the claims it makes are well supported.\n2. Scaling laws are impactful.\n3. Identifying what interventions are effective for LLM training can save lots of compute. \n\nWeaknesses.\n1. The practical utility of the paper is limited. The authors demonstrated that training data has a larger effect on the scaling laws than model architecture. An alternative way to conclude that pretraining data is more important than architecture is to just use e.g. MMLU numbers. It is not clear how the methodology the authors propose would be better than this baseline approach to determine what interventions are effective and which are not.", "other_comments_or_suggestions": "na", "questions_for_authors": "1. Could you discuss more in detail how the irreducible errors are estimated?\n2. Could you somehow quantify how your approach is better at identifying what interventions are effective than a baseline approach which would just use e.g. MMLU numbers as a quality metric?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5BRy3Lzsjt", "reviewer": "Reviewer_EE76", "summary": "This paper explores how loss-to-loss scaling laws depend on various factors in the training setting. While compute-to-loss scaling laws are often studied (i.e., training on X tokens, Y parameters will give you Z loss), there is recent interest in loss-to-loss scaling laws, which show how evaluation/training on one dataset can translate into evaluation/training on another. This paper finds that by varying the pre-training dataset, the relationship between validation loss and downstream test loss can change significantly. Varying the tokenizer can also change the scaling trend, while things like the model architecture (like Llama vs Mamba), optimization hyperparameters, context length, and model size have little impact.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments and feedback. We address your concerns below: \n\n**[Q1, Q3, OSW Clarity 1 - On the construction and fitting of scaling laws]**: We have added two Appendix sections that (1) detail the scaling law formulation and (2) explain how parameters are estimated. Here’s a slightly abridged version:\n\n> **Scaling Law Details**\nWe adopt the compute-to-loss scaling law formulation from (Brandfonbrener et al.) Eq. (4):\n> $$L\\left(f^{(N, D)}\\right) = E + \\left( \\left( \\frac{A}{N} \\right)^{\\frac{\\alpha}{\\beta}} + \\frac{B}{D} \\right)^\\beta,$$\n> where $f^{(N, D)}$ is a model with $N$ parameters trained on $D$ tokens and $E, A, B, \\alpha, \\beta$ are parameters to be fit.\n> Notably, the irreducible error $E$ captures the minimum loss possible for model $f$ in the limit of infinite model and data size.\n> \n> By default, this is fit using the training or validation loss.\n> However, as demonstrated by (Brandfonbrener et al.) and our experiments, we can alternatively predict the loss $L_x$ on dataset $\\mathcal D_x$ achieved by model $f_p^{(N, D)}$ trained on the pretraining set $\\mathcal D_p$:\n> $$L_x\\left(f_p^{(N, D)}\\right) = E_{x|p} + \\left( \\left( \\frac{A}{N} \\right)^{\\frac{\\alpha}{\\beta}} + \\frac{B}{D} \\right)^\\beta.$$\n> As in (Brandfonbrener et al.) Eq. (7), the irreducible error $E_{x|p} = L_x(f_p^*)$ then captures the minimum possible loss on $\\mathcal D_x$ of a model trained on $\\mathcal D_p$.\n>\n> With that, we can formulate the loss-to-loss scaling law for arbitrary combinations of pretraining data and two test or validation sets, as stated in Eq. (1).\n\n> **Fitting Details**\n> For each line in a plot corresponding to a loss-to-loss scaling law from Eq. (1), we first fit the two compute-to-loss scaling laws $L_x(f_p^{(N, D)})$ and $L_y(f_p^{(N, D)})$.\n> This yields estimates for the irreducible errors $E_{x|p}, E_{y|p}$, which correspond to the minimum x- and y-value of the loss-to-loss line.\n> We use SciPy's default `curve_fit` optimizer for fitting.\n> In rare cases when all checkpoints have the same number of parameters $N$ or same number of tokens $D$ (this is the case only for a small subset of the HuggingFace models) and a compute-to-loss scaling law cannot be fitted, we instead estimate the irreducible error as the minimum loss achieved: $E_{x|p} = \\min_{N,D} L_x\\left(f_p^{(N, D)}\\right)$.\n> With $E_{x|p}, E_{y|p}$ from the compute-to-loss fits, we again use SciPy's `curve_fit` to fit $K, \\kappa$ for the loss-to-loss scaling law  from Eq. (1).\n\nWe add examples of compute-to-loss fits for some of the loss-to-loss scaling laws from Fig. 2 in this section; see new [Fig. 10](https://ibb.co/SqkwfwF). Note also that while Figs. 4-6 show averaged eval/validation performance, Figs. 22-27 in App. F includes results for specific validation and evaluation datasets (C4, The Pile UC, ARC-Easy, and HellaSwag).\n\n**[OSW Clarity 1a - More details on experimental setups]**: We have updated all the figure captions, Tab. 1, and [Tab. 2](https://ibb.co/xq3KgQ4N) in the appendix to include more details for the models used.\n\n**[OSW Clarity 1b - Clearer axes labels]**: Thank you for pointing this out. In our case, plot colors correspond to different dimensions in different plots. In Fig. 1, the x- and y-axis correspond to two specific datasets, and colors denote different interventions. In Fig. 2, the x-axis is fixed, and colors denote the dataset of the y-axis (most similar to Brandfonbrener). In Figs. 4-6, axes are again fixed (and y-axis reports an average over multiple datasets), and colors denote different values for the specific intervention. In all cases, the training set is specified in the caption. We have now updated the figure captions and legend layout to clarify this.\n\n**[W1 and Q2 - Modeling the influence of pretraining data]**: Quantitatively modeling the influence of data distributions is an intriguing open problem. The central issue is the difficulty in mapping differences in data distributions onto a simple feature space (e.g., scalar or vector values). In Data Mixing Laws [2], the mixing ratio serves this purpose effectively. However, we face the additional complexity of quantitatively comparing disjoint pretraining distributions with unknown compositions, a challenge beyond the scope of this work. We also do not believe multi-loss-to-loss scaling laws offer a satisfying solution. Without a reliable way to quantify diverse large-scale distributions and their relationships, we would need separate multivariate power laws for each combination of losses, again without a direct means for meaningful comparison.\n\n**[OSW2 and Q4 - Practical utility of the paper]**: We refer you to our response to Reviewer fuqS (section: **[W1 & Q2b - On the utility of loss-to-loss scaling curves]**) for an abridged version. \n\nWe hope this addresses your concerns and encourages you to raise your score.", "claims_and_evidence": "Evidence from numerous empirical runs supports the claim that data primarily determines loss-to-loss scaling, while other factors (optimization, architecture, etc.) impact this scaling less.", "methods_and_evaluation_criteria": "The evaluation criteria makes sense for the problem at hand.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "Experimental design and analyses appear sound, although there could be more clarity on how the shifted power law is fit (i.e., how to compute $E_{x|p}, E_{y|p}$ and parameters $K$ and $\\kappa$).", "relation_to_broader_scientific_literature": "Strengths: \n+ This paper discovers that loss-to-loss scaling is not significantly impacted by changes in model architecture, model size, context length, and optimization settings. It is especially interesting that the scaling is not impacted by model architecture (Llama vs Mamba), which suggests that different architectures can converge to similar representations. This is of broader interest to the community, in terms of advancing our understanding of the role of architecture, optimization, etc. in downstream properties. \n+ The implications of pre-training data determining the loss-to-loss scaling laws are interesting, because many data curation strategies primarily focus on optimizing one validation dataset's loss (for example, [1]). This paper suggests that we should exercise caution in this practice---achieving X loss on just one validation dataset could mean various different performances on a downstream task. That is, one validation dataset is not comprehensive enough to determine downstream performance. \n\nWeaknesses:\n- While this paper has the above interesting implication, it feels like it falls short in understanding how pre-training data alters the loss-to-loss scaling law. That is, if there is this extra confounding factor---the pretraining dataset---in these loss-to-loss scaling laws, how do we model this factor and eliminate the confounding? My hunch is that a multi-loss to loss scaling law could result in things lying on the same line (for example, see Data Mixing Laws [2]). The paper would have broader impact if it could provide a path forward for consistent loss-to-loss scaling laws, or how to make sense of pre-training data impacting these laws. \n\n[1] https://arxiv.org/abs/2407.01492\n[2] https://arxiv.org/abs/2403.16952", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Weaknesses:\n\n\nClarity:\n- It is slightly unclear how a train-to-train scaling law is defined. In Brandfonbrener (2024), there are *two* models being trained on different data with the same number of params and tokens, and that makes up the x-axis vs y-axis. However, in Figure 4 each point is derived from one model trained on a particular dataset, and evaluated on FineWeb-Edu as well as other validation domains (and averaged). I see that lines 156-160 briefly mention that each point in the paper's plots shows losses of one model; however, train-to-train and train-to-test should still be formally defined. \n     - The clarity of the paper could be improved by explaining what each point on the various loss-to-loss plots represents, and concretely describing the experimental setup for each type of scaling law in the Appendix. \n     - The clarity could also be improved by adding more description to the x and y axes in the plots. For instance, in Brandfonbrener et. al., they often write \"Loss on Hellaswag (Trained on data 1)\" and \"data 1\" is depicted by the colors and the legend. This would make it easier for the reader to understand what the different colored points mean in relation to the axes.\n\nSignificance: \n- The paper needs to go one level deeper in discussing the practical applications of their findings. For instance, how can a practitioner, who is using one training dataset/one validation dataset, use the insights from this paper to better predict performance on another dataset/task? Moreover, in section 5, the paper says that the data distribution is the key for achieving a desirable loss-to-loss scaling and in turn achieve a great downstream performance---what defines \"desirable\" here?", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. Can you clarify how the shifted power laws are fit? i.e., how to compute $E_{x|p}, E_{y|p}$ and parameters $K$ and $\\kappa$.\n2. How can we better understand the role of pre-training data in the loss-to-loss scaling law?\n3. Improve clarity: can you precisely explain how train-to-train and train-to-test scaling laws were constructed?\n4. Add more discussion of the practical applications of their findings - how can a practitioner use the insights from this paper? What defines a desirable loss-to-loss scaling that can be used to achieve better downstream performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kWpsrDql6e", "reviewer": "Reviewer_he3E", "summary": "The paper studies loss-to-loss scaling laws in language models, covering both predicting the language modeling log loss across different data distributions (“train-to-train”) and predicting log loss proxies for downstream tasks performance (“train-to-test’). The main finding in this paper is that loss-to-loss scaling is insensitive to model architecture and optimization hyperparameters, mildly sensitive to the choice of tokenizer, and very sensitive to the training data distribution.\n\n## Update after the rebuttal\n\nThe authors have adequately addressed my concerns. I therefore increase my score to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your helpful comments and feedback. We address each of your questions and concerns below.\n\n**[Q1 - Models trained and evaluated and complete results]**: Thank you for noticing this; we have made amendments in multiple places: First, we now mention in Sec. 4 the size of not only our models, but also the models sourced from HuggingFace. Second, we have expanded on App. A and Tab. 1 to clarify the models we trained and created a new table for models we evaluated from HuggingFace (see [Tab. 2](https://ibb.co/xq3KgQ4N)). Third, we will release the entire data frame with all model settings and evaluation results for the roughly 6000 checkpoints and our complete code with the camera-ready version.\n\nHere’s an abridged overview of the evaluated models: \n\n- We trained Llama and Mamba models ranging from approximately 60M to 420M parameters from scratch. We have also added some 7B models now, see below.\n - We evaluated pretrained models from HuggingFace: Pythia (70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, 12B), GPT-Neo models (125M, 1.3B, 2.7B), GPT-NeoX (20B), and GPT-J (6B), all trained on variants of The Pile. These models are included in the right-most columns of Figs. 4–6, 16–18, and 22–27.\n- We also evaluated FineWeb ablation models (1.7B) from HuggingFace trained on The Pile, C4, and FineWeb-Edu. These models are also included in Figs. 4-6 (e.g. Fig. 4 column 3).\n\nNote that although we fit scaling laws using all available checkpoints, we display a randomly selected subset in figures for readability. We have made this more evident in the text and figure captions.\n\n**[CE1 - Training/Evaluating large models]**: As explained above, our analysis already includes publicly available pretrained models with up to 1.7B parameters for Llama, up to 20B parameters for GPT, and up to 2.7B parameters for Mamba. We apologize for this not being evident in the original manuscript. As illustrated in Figs. 4-6 (especially in the right-most column showing the largest GPT models), these models generally achieve lower loss on validation and test sets compared to our models trained from scratch, but still adhere to the power-law trends and confirm our key insights. For example, the right-most column in Fig. 6 contains all of the largest HuggingFace models and clearly demonstrates that architecture does not affect loss-to-loss scaling laws even at these scales. \n\nFollowing your suggestion, we revisited available pretrained models. However, most other publicly available checkpoints of large scale models (≥7B) are unsuitable for our analysis as they are trained on (often undisclosed) data mixtures that prevent us from directly comparing them to other models (.e.g, the Falcon series).\n\nAll that said, we agree that especially the impact of the pretraining data in Fig. 4 is not conclusively shown for larger models, since the largest HuggingFace models included in our analysis were all trained on variants of The Pile (see Fig. 4, right-most column). To remedy this, we have now trained additional 7B models for 1B tokens ourselves. Due to compute and time constraints we have limited this to three settings: \n\n - Llama-7B (tiktoken tokenizer) on FineWeb-Edu / on C4\n - Llama-7B (GPT-2 tokenizer) on FineWeb-Edu\n\nWe’ve included all these models in our revised [Fig. 1](https://ibb.co/kgpRZNZP). All models follow the established power-law scaling curves and confirm our main conclusions.\n\n**[OCS1 - Clarifying taxonomy used in paper]**: Thanks for the suggestion. We will state and clarify the taxonomy used in the paper early in the manuscript. We have also added an appendix section that explains the scaling laws in more detail, which we describe in our reply to reviewer EE76.\n\n**[OCS2 - Normalizing tokenizer comparisons]**: Thank you for this excellent suggestion! We performed the experiment and have updated [Fig. 1](https://ibb.co/kgpRZNZP) for BPB. Indeed, after normalizing negative log-probability by bytes, the previously distinct tokenizer lines collapse onto each other. This confirms even more strongly that architecture, tokenizer, and optimization hyperparameters minimally affect loss-to-loss scaling—only pretraining data distribution truly matters. We will update other figures and sections of the manuscript accordingly.\n\nWe hope this addresses your concerns fully and encourages you to raise your score.", "claims_and_evidence": "The claims of this paper are generally well supported by evidence, except for one notable issue: it is not clear if the language models studied in this paper can be called “large.” The authors train models of up to 420M parameters and supplement the analysis with additional models of undisclosed sizes. Based on the FW-Edu validation loss, none of the supplementary models are large: using their own models the authors reach a loss of about 2.9 (according to Fig. 30) while the lowest loss shown in the paper is about 2.7 (in Fig. 1), which makes me unsure if even the 1.8B parameter FineWeb-Edu ablation model by HuggingFace was included.  \n\nClearly, truly large models (say with 7B parameters or more) were not included in this study, and I am not sure why: the web is full of scaled families of open-weights LM’s, some including intermediate checkpoints and trained on fixed datasets. These models could be downloaded, evaluated, and used to test the claims of the paper on actually large language models. \n\nThe lack of evaluation on larger models is my primary concern regarding this paper - if it is addressed I will be happy to increase my score.", "methods_and_evaluation_criteria": "See “Claims And Evidence” above.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "See “Claims And Evidence” above.", "relation_to_broader_scientific_literature": "This paper contributes to the body of evidence that the performance of neural networks under distribution shift tends to be predictable with very few “effective robustness” interventions beyond changing the training data", "essential_references_not_discussed": "None to my knowledge.", "other_strengths_and_weaknesses": "N/A.", "other_comments_or_suggestions": "1. I had a hard time parsing what is meant by “train-train” and “train-to-test” - I suggest including a more detailed and self-contained explanation in the beginning of the revised paper.\n\n2. When comparing different tokenizers, comparing the average negative log probability per token is not entirely appropriate, since different tokenizers have different compression efficiencies and hence represent the same data using a different number of tokens. Instead, you should normalize the negative log-probability by a tokenizer-independent quantity, such as the number of bytes of uncompressed text in the sequence, or the number of tokens used by a fixed tokenizer. It would be interesting to see if such improved reckoning moves the “lines” corresponding to different tokenizers closer.", "questions_for_authors": "Precisely which models were evaluated and what were the evaluation results? You should answer this by tabulating the results of your entire testbed and include it as supplementary material. See “Claims And Evidence” above for context.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QD3ui638VJ", "reviewer": "Reviewer_6jji", "summary": "The paper focuses on loss-to-loss scaling laws in large language models (LLMs), which relate losses across pretraining datasets and downstream tasks. The author finds that 1. loss-to-loss scaling consistently follows shifted power-law trends, enabling prediction of test performance from training loss, as detailed in the conclusion section. 2. Pretraining data and tokenizer are identified as the dominant factors shaping these scaling laws, with experiments showing significant impact when these are varied. 3. Architecture has limited influence and model size, context length, and optimization settings (e.g., Adam, cosine schedules) have negligible effects. 4. The authors recommend that practitioners should prioritize curating suitable pretraining datasets for optimal downstream performance, while architectures and other settings can be freely optimized for training efficiency.\n\n## update after rebuttal\nI am keeping my score since my concerns are not fully addressed.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful comments and feedback.\n\n**[EDA1 - Conclusions are subjective]**: We now quantify our findings in two ways:\n1. We quantify the goodness of fit of the loss-to-loss power laws as $R^2$. We show this in our revised [Fig. 1](https://ibb.co/kgpRZNZP) (note that Fig. 1 has also received other updates; please refer to our reply to Reviewer he3E). Generally, the goodness of fit is very close to $1$. We will similarly update all figures for the camera-ready.\n2. We quantify the impact of different interventions as the area between fitted curves in a newly added [Tab. 2](https://ibb.co/7tjs1xL5). Pretraining data clearly has the biggest impact on the scaling laws. Please also refer to our reply to Reviewer he3E for our updated conclusion on the impact of the tokenizer.\n\n**[Conventional vs. loss-to-loss scaling laws]**: The distinction between compute-to-loss and loss-to-loss scaling laws may not have been sufficiently clear in the text and caused some confusion; we have updated our introduction and related work sections to explain this better.\nTo reiterate, we focus on _loss-to-loss scaling laws_ (i.e., train-to-train, train-to-test, test-to-test), not _compute-to-loss scaling laws_ (e.g., Kaplan et al., 2020; Hoffmann et al., 2022). While compute-to-loss scaling laws are primarily used to find optimal compute budgets, loss-to-loss scaling laws can help study generalization, i.e., how performance transfers from training distributions to downstream tasks (Brandfonbrener et al., 2024; Du et al., 2025). Although compute-to-test scaling can be informative, they do not allow for connecting training loss to downstream performance, as you pointed out. This is where loss-to-loss scaling laws become a fascinating object of study, as they explicitly model how training/validation loss is converted to test loss, a proxy for test performance.\n Please also see our response to reviewer fuqS on the utility of loss-to-loss scaling laws.\n\n**[W1, W2 - Findings are implied by existing scaling law formulations]**: We respectfully disagree. As the reviewer points out correctly, conventional scaling laws (i.e., compute-to-loss scaling laws) are a function of model and data size. That formulation alone does not automatically imply that changing data _quality_ without affecting data _size_ should impact the scaling law.\n\nHaving said that, for _compute-to-loss scaling laws_, it is of course well known that data quality _does_ play a role. However, other factors like architecture, optimizer, and learning rate scheduler also matter for compute-to-loss scaling laws (Li et al., 2025). It is not at all “inherently implied in existing scaling law formulations” which factors impact scaling laws and which don’t – after all, none of these factors are quantifiable and explicitly enter the scaling law.\n\nNow, _loss-to-loss scaling laws_ are a relatively recent object of study. Prior to our study (Takeaway 1), it was unclear whether loss-to-loss scaling laws could generally be described by shifted power laws at all, since previous studies like Brandfonbrener et al. are very limited in their settings. On top of that, it is unclear which factors, if any, play a role. Again, it is not at all “inherently implied in existing scaling law formulations” whether pretraining data (quality, not size), architecture, tokenizer, or other hyper parameters might impact the scaling law. If we were to go by the experience of compute-to-loss scaling laws, we might assume that most of these factors matter. In contrast, we find that loss-to-loss scaling laws are notably insensitive to many factors, while strongly depending on the pretraining data. As reviewer EE76 points out, “this is of broader interest to the community, in terms of advancing our understanding of the role of architecture, optimization, etc. in downstream properties”.\n\n**[W3b - Mamba & Llama perform similarly]**: As you note, determining whether drastically different architectures like transformer-based Llama and state-space-based Mamba achieve similar downstream performance given identical training setups is a significant open question. \n\nWe have to emphasize the distinction between compute-to-loss and loss-to-loss training laws. While compute-to-loss training laws can answer whether two models converge to the same training loss with increasing compute, loss-to-loss training laws answer whether models reach the same downstream performance given a training loss.\n\nIn this sense, yes, it is precisely the case that Mamba and Llama models “converge to similar downstream performance across tasks” given they reach the same training loss. To further illustrate this, we present in [Tab. 3](https://ibb.co/tpgMmP1Z) multiple Llama and Mamba models trained under identical conditions, which show nearly indistinguishable downstream performances.\n\nWe hope this addresses your concerns and encourages you to raise your score.\n\n(Li et al, 2025) (Mis)Fitting: A Survey of Scaling Laws", "claims_and_evidence": "Yes, each claim is accompanied by empirical results.", "methods_and_evaluation_criteria": "Yes, the authors train a series of models to validate their scaling laws.", "theoretical_claims": "The paper is empirical.", "experimental_designs_or_analyses": "The experiment design is sound and the ablation study is thorough. However, the conclusions drawn are not quantitative, drawing on subjective judgments when deciding which factors affect scaling laws more.", "relation_to_broader_scientific_literature": "The paper extends recent scaling law studies by systematically exploring factors influencing loss-to-loss scaling, contributing to understanding LLM performance optimization and generalization across tasks.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The authors conduct a comprehensive analysis of factors affecting loss-to-loss scaling.\n2. The experiments are well organized and the results are clear. \n\nWeakness:\n1. The experiments do not bring to the table findings not implied by existing scaling law formulation. For instance, scaling law is a function of model size and data size. Changing the dataset would shift the scaling law by affecting the fitted parameters. Hence, there is a plethora of studies and open source efforts on producing higher quality datasets, whose scaling curve is different from the lower quality datasets (DCLM, redpajama, etc.)\n\n2. Similarly, what the authors identified as less important is also inherently implied in existing scaling law formulation. \n\n3. The author did not improve upon the existing scaling law's limitation, which is to connect training/validation loss to individual downstream task performance. For instance, if the authors believe that changing the architecture has a limited impact on loss-to-loss scaling laws, is it possible to show two models (one decoder-only transformer-based, one Mamba-based) with fixed recipes converge to similar downstream performance across tasks? This remains one of the most popular open questions in the field to this day.", "other_comments_or_suggestions": "The paper is well written.", "questions_for_authors": "My questions are listed above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper investigates factors that determine loss-to-loss scaling of LLMs. The paper find that the pretraining data and tokenizer determine the scaling trend, and model size, optimization hyperparameters, and model differences have less of an impact. 

The paper received very mixed reviews (1,2,3,4):
- Reviewer fuqS (2) findes the practical utility limited. The authors emphasize that loss-to-loss scaling laws has several benefits: Understanding performance of one dataset as the function of performance on another; the finding that while model parameters (size, hyperparameters, architecture) impact compute scaling laws, they do not impact loss-to-loss scaling laws; and cautining against reyling on one validation sets loss `as typical in data curation methods’. I agree with the benefits of loss-to-loss scaling laws, but like to mention that it is not typical to rely on one validation set's loss in data curation methods. 
- Reviewer EE76 (3) find the key finding that loss-to-loss scaling is not impacted by model parameters interesting. The reviewer notes, however, that the paper falls short in understanding how pre-traing data alters the loss-to-loss scaling. 
- Reviewer he3E (4) find the claims generally well supported except that the models considered are relatively small, which is the reviewer's main concern. The authors clarified that they also based their results on larger models, and revised Fig 1 amongst other parts of the manuscript. 
- Reviewer 6jji (1) raised concerns that were addressed by the reviewers.

I agree with reviewers EE76 and he3D that the paper's results are interesting and the claims are well supported. I therefore recommend acceptance. Reviewers EE76 and he3D also made several constructive comments that can help to further improve the data. An toy example or theoretical derivation on how loss-to-loss scaling laws can look like would also be interesting.
Decision: Accept (poster)