ICML 2025

LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
Prasanna Mayilvahanan * 1 2 3 4 Thadd¨aus Wiedemer * 1 2 3 4 Sayak Mallick 1 4 Matthias Bethge 2 3 4
Wieland Brendel 1 2 3
Abstract
Scaling laws guide the development of large lan-
guage models (LLMs) by offering estimates for
the optimal balance of model size, tokens, and
compute. More recently, loss-to-loss scaling laws
that relate losses across pretraining datasets and
downstream tasks have emerged as a powerful
tool for understanding and improving LLM per-
formance and generalization. In this work, we
investigate which factors most strongly influence
loss-to-loss scaling. Our experiments reveal that
the pretraining data determines the scaling trend.
In contrast, model size, optimization hyperparam-
eters, tokenizer and even significant architectural
differences, such as between transformer-based
models like Llama and state-space models like
Mamba, generally have limited impact. Conse-
quently, practitioners should carefully curate pre-
training datasets for optimal downstream perfor-
mance, while architectures and other settings can
be freely optimized for training efficiency.
1. Introduction
Scaling laws have long guided Large Language Model
(LLM) pretraining, determining model and data size un-
der a fixed compute budget (Kaplan et al., 2020; Hoffmann
et al., 2022; Grattafiori et al., 2024). Typically, scaling laws
relate model performance, usually measured as training or
validation loss, to total compute measured in floating point
operations (FLOPs). FLOPs account for both parameter
count and the number of training tokens. While useful for
pretraining, scaling laws do not capture how well a model ul-
timately performs on downstream tasks (Gadre et al., 2024;
Schaeffer et al., 2024; Du et al., 2025). Consequently, mul-
tiple works have begun to investigate downstream scaling
*Equal contribution 1Max Planck Institute for Intelligent Sys-
tems 2ELLIS Institute T¨ubingen 3T¨ubingen AI Center 4University
...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000076. Conference bucket: ICML.