What Scales in Cross-Entropy Scaling Law?
Authors: Junxi Yan
Zixi Wei
Qingyao Ai
Yiqun LIU
Jingtao Zhan
Venue: ICLR 2026 Poster (2026)
Keywords: Cross-Entropy Loss; Error-Entropy; Neural Scaling Laws; Loss Decomposition; Large Language Models;
Primary area: foundation or frontier models, including LLMs

Abstract: The cross-entropy scaling law has long served as a key tool for guiding the development of large language models. It shows that cross-entropy loss decreases in a predictable power-law rate as the model size increases. However, recent evidence indicates that this law breaks down at very large scales: the loss decreases more slowly than expected, which causes significant trouble for developing large language models. In this paper, we hypothesize that the root cause lies in the fact that cross-entropy itself does not truly scale; instead, only one of its hidden components does. To investigate this, we introduce a novel decomposition of cross-entropy into three parts: Error-Entropy, Self-Alignment, and Confidence. We show both theoretically and empirically that this decomposition precisely captures the training dynamics and optimization objectives. Through extensive experiments on multiple datasets and 32 models spanning five orders of magnitude in size, we find that only error-entropy follows a robust power-law scaling, while the other two terms remain largely invariant. Moreover, error-entropy constitutes the dominant share of cross-entropy in small models but diminishes in proportion as models grow larger. This explains why the cross-entropy scaling law appears accurate at small scales but fails at very large ones. Our findings establish the error-entropy scaling law as a more accurate description of model behavior. We believe it will have wide applications in the training, understanding, and future development of large language models.

Published as a conference paper at ICLR 2026
WHAT SCALES IN CROSS-ENTROPY SCALING LAW?
Junxi Yan, Zixi Wei, Qingyao Ai, Yiqun Liu, Jingtao Zhan Tsinghua University
sanjxx2egmaai.. com, zixiwei 30mmei1@m......mmmmm com, aiqy@tsinghus edw.. cn
yigunLinettinghua.cc edu. en, singtaczharrtsinnhua. edu. en
ABSTRACT
The cross-entropy scaling law has long served as key tool for guiding the de- velopment of large language models. It shows that cross-entropy loss decreases in a predictable power-law rate as the model size increases. However, recent evi- dence indicates that this law breaks down at very large scales: the loss decreases more slowly than expected, which causes significant trouble for developing large language models. this paper, we hypothesize that the root cause lies in the fact that cross-entropy itself does not truly scale; instead, only one of its hid- den components does. To investigate this, we introduce a novel decomposition cross-entropy into three parts: Error-Entrope We show both theoretically and empirically that this decomposition precisely cap- tures the training dynamics and optimization objectives. Through extensive ex- periments on multiple datasets and 32 models spanning five orders of magnitude size, we find that only error-entropy follows a robust power- scaling, while the other two terms remain largely invariant. Moreover, error-entropy constitutes the dominant share of cross-entropy in small models but diminishes in proportion as models grow larger. This explains why the cross-entropy scaling law appears accurate at small scales but fails very large ones. Our findings establish the error-entropy scaling law as more accurate description of model behavior. We believe it will have wide applications the training, understanding, and future development of large language models.
Self-Aligmment, and Confidence.
INTRODUCTION
The cross-entropy scaling law has played vital role in the development of large language models. This empirical law states that as model size and dataset size increase, the cross-entropy loss de- creases in predictable power-law manner (Kaplan et al., 2020). Its influence is both practical and theoretical. On the practical side, it has become an indispensable tool for training large language models, such as balancing model parameters and data scale (Hoffmann et al. (022). extrapolating performance from smaller models to larger ones (Wei et al.) 2022a), and tuning training hyperpa- rameters (Kadra al. 2023; Li et al. 2025). On the theoretical side, it has opened gateway for understanding the principles of intelligence itself. Researchers have proposed various theories to explain why such a law emerges (Bahri al., [024] 2024 Michaud et al. 2024), hoping that these explanations will shed light on the nature of artificial intelligence.
However, the cross-entropy scaling law has recently faced growing skepticism, both in practice and in theory. Practitioners have raised concerns about whether the law can continue to provide accurate predictions on larger scales. Empirical studies show that while cross-entropy decreases with an accurate power-law trend for small models, this trend becomes noticeably slower for very large models (Kaplan et al.) 2020 Chen et al.] 2025). On the theoretical side, the situation is equally unsettled. Although most existing theoretical frameworks error-based metrics, such as mean squared error (Lyu et al. 2025), they cannot directly generalize to cross-entropy loss. Thus, the theoretical foundation of this law still unclear. These problems have fueled skepticism about whether cross-entropy truly scales, and in turn, they undermine confidence in scaling up models as a reliable path forward.
somehow prove the scaling law for
"Corresoonding author

---

Published as a conference paper at ICLR 2026
Cress Entrayy Less's Nature?
-Pplog9
Error- Groor-Entopy
Confidence
Self
Aligmmett
Figure 1: Decomposition of the Cross-Entree Loss. The upper panel illustrates the cross-entropy loss. The lower panel presents the three components decomposed from cross-entropy. The blue curve denotes the Rank-based Error (RBE) distribution, and the orange curve represents the dis- tribution of the ground-truth scores. Error-Entropy pushes the ground-truth tokens towards higher ranks. Seff-Alignment aligns the probability score distribution with RBE distribution. And Confi- dence term increases the norm of the probability score.
In this work, we hypothesize that it is not cross-entropy loss itself that truly scales, but rather dominant component hidden within it. This component gives the illusion that the cross-entropy itself follows scaling law. Identifying this scaling component would therefore be of significant value. On the practical side, it would provide a more reliable law to guide the development of large language models. On the theoretical side, it could offer better objective for investigating the principles of artificial intelligence. Motivated by this, the research question of this paper is:
What scales in cross-entropy scaling law?
To address this question, we begin by introducing novel decomposition of cross-entropy. Specifi- cally, we propose an urror-based metric rooted in ranking. termed Rank-based Error (RBE). Unlike cross-entropy, which measures the probability score of the correct token, RBE equals the rank of the correct token. For example, if four other tokens are scored higher than the ground truth, then the RBE Building on this notion, we decompose cross-entropy exactly into the sum of three terms: Error-Entrope
Seff-Alignment and Confidence, which is illustrated in Fig. Among them, Entor-Entropy measures the entropy of the RBE distribution. Minimizing this quantity effectively pushes the correct token toward higher ranks and thus makes the model's 's predictions more accurate. The other two terms characterize how the model aligns probability scores with the RBE distribu- tion. Through both qualitative and quantitative analysis, we demonstrate that this decomposition accurately reflects the training dynamics of language models.
Building on this decomposition, we investigate the scaling behavior of these components and find that only Error-Entropy truly scales. We refer to it as the Error-Entropy Scaling Law. Concretely, we conduct experiments on multiple real-world language datasets using 32 models spanning five orders of magnitude in size. We then evaluate the scaling behavior of cross cross-entropy alongside its three components. The results are clear. (1) Error-Entropy decreases according to power law with model size, whereas the other two terms remain random for different model sizes. (2) The power-law fit of EError-EEtropy Entropy is even better than that of cross-entropy. It indicates that the Error-Entropy scaling law is potentially the reason why cross-entropy approximately exhibits scaling behavior.
We believe that our decomposition of cross-entropy and the discovery of the Error-Entropy scaling law provide better description of how language models actually scale, and open the door to broad applications. As one example, this framework helps resolve a oong-standigg puzzle (Kaplan et al. 2020 Chen al. 2025):: why does the cross cross-entropy scaling law appear accurate for small models but break down for larger ones? The answer lies the proportion of Error- Entrr-Etropy. It dominates in small models, making cross-entropy follow clean power law, but its declining share in larger models allows non-scaling terms to dominate, breaking the law. This specific example highlights how Error-Entropp can sharpen our understanding of model behavior. More broadly, we expect that
The code is available at http:::s///hthhh....///////aaaaa http://annnaaooo////eeeeenn. /Rethinkkk
2

---

Published as a conference paper at ICLR 2026
Gound Truth's Feature
A5
Categoried
1
.90
Output
by RBE
25
65
RBE
Prob. Distribution
Decompose Crees-Entropy Loes
Entropy
Erao-Entropy
Self-Alignment
23
Log Soors,
Σ loane
& Σ
log
Figure 2: Overview of the decomposition. Rank-based Error (RBE) is the ranking of the ground- truth token. Model's predictions are grouped based on RBE values. For each group, we compute its proportion (termed RBE distribution), the normalized prediction score le. and the norm of scores C. Based on these definitions, cross-entropy can be mathematically decomposed into three components: Error-Entrope Self- Alignmeetttt and Confidence.
will have wide-ranging implications, such as guiding the design and training of large language models, probing model mechanisms, and finding fundamental theories of artificial intelligence.
RELATED WORK
Neural Scaling Laws. Early empirical studies had already suggested that deep learning perfor- mance often follows predictable scaling behavior across tasks and modalities (Hestness al., 2017) Kaplan et al. (2020) first systematically observed power-law relationship between model perfor- mance, parameter count, dataset size, and compute language modeling, and proposed compute allocation rules; Hoffmann et al. (2022) later refined this into compute-optimal scaling strategy. Henighan et al. (020) extended the cross-entropy scaling law to images, videos, multimodal tasks, and mathematics. while Zhai et al. (2022) and Ghorbani et al. (2021) validated similar behaviors in vision and machine translation. Beyond these, Hernandez et al. (221): ;; Barnett (2024) studied scaling in transfer learning. Although numerous works have documented empirical scaling laws, the fundamental question of why cross-entropy decreases according to a power-law remains un- clear. Some studies have attempted explain scaling behavior for error-based metrics such mean squared error (Lyu et al.) (025), but these results are difficult to generalize to cross-entroe cross-entropy, the standard loss for training classifiers and language models. In this work, we address this gap by identifying the components of cross-entropy that truly drive its scaling behavior.
Cross-Entropy Loss. In the neural network domain, prior studies about cross-entropy have mainly examined its properties regarding consistency, robustness, calibration, and regularization (Guo et al.) 2017 Fort & Scherlis 22018 Wei et al., 2022b] Soudry et al., 2024 Vysogorets et al., 2024). For instance, Mao al. (2023) conducted systematic theoretical analysis of cross-entropy and its variants, deriving H-consistency bounds. However, these analyses do not investigate whether-and how-such intrinsic properties of cross-entropy interact with macroscopic scaling behavior. Our work complements these directions by decomposing cross-entropy into components with clearer operational meaning and studying how these components scale with model size.
FROM CROSS-ENTROP! TO ERROR-ENTROPE
In this section, we decompose cross-entropy into three components and show how these components relate to model performance. First, Sec. Sec.3.1 introduues new metric. Second, Sec. 3.2 uses this met- ric to mathematically decompose cross-entropy loss and then empirically examines the correctness of this decomposition. Finally, See. 3.3 compares the three components and shows that one of the components, termed Error- EntrorEEttroy,, is closely tied model performance.
3

---

Published as a conference paper at ICLR 2026
3.1 RANK-BASED BASED ERROR
Which better reflects the performance of a language model, the probability assigned to the correct token, or its rank among all tokens? While cross-entropy defined terms of probabilities, we argue that rank provides a more robust indicator. Probabilities are easily manipulated: during infer- ence, techniques such as temperature scaling, top-k sampling, and nucleus (top-p) sampling directly alter probability values, and such changes also affect the cross-entropy loss. By contrast, the relative ordering of tokens is much harder to distort. None of the common sampling strategies changes the ordering. For this reason, we view the rank position of the correct token more fundamental measure of model performance than its raw probability scores.
Therefore, propose new metric, Rank-based Error (RBE), which equals the rank of the ground- truth token predicted by the language model. It serves as a measure how far the prediction deviates from the right answer. Formally, let be the index of the input data, and e; be the correct token for this data. V is the vocabulary. We use Sv, to denote the model's 's output probability score for e;- Thus, RBE for v; is defined as:
RBE(U:)
E1ss > s.,
VEV
where 1 denotes the indicator function that equals the condition is true and 0 otherwise. Intuitively, smaller RBE indicates that the model ranks the correct token near the top and thus corresponds to better model performance.
Based RBE, define two distributions. First, treat RBE random variable and analyze its probability distribution over the corpus. Second, we group model predictions by RBE and compute the mean score for each group. An illustration is shown in Fig. The details follows:
RBE Distribution. We define the RBE distribution Pe as the probability mass function of RBE:
= Pr [BE((,)) = e E D).
(2)
where 'D denotes the collection of all output ground-truth tokens in the test corpus. Intuitively, Pe characterizes how often the ground-truth token appears at rank model predictions. For a well- trained model, we expect Pe to concentrate more heavily on smaller ranks e, indicating that the model more likely to place the ground-truth token near the top of its prediction list.
Score Distribution. Complementary to Pe, define the Score distribution qe by grouping the predictions with the same RBE values and computing their average scores. Formally, qe is the geometric mean of {s.,) for those V, with RBE(U.) =
Q. GeoMean RBF(U;) = =))),
(3)
where the geometric mean is GeoMean( ((a,) ((,))= (II an) 1/m. We further normalize {Q.) as:
Qe Le = C
C= -
(4)
de is the normalized score distribution, and C' is the norm of the prediction scores. A larger C indicates that the model is generally more confident in its predictions.
Next, we will use Pe. ge, and C' to decompose cross-entropy.
3.2 MATHEMATICAL DECOMPOSITION
This section mathematically decomposes cross-entropy. We start with its definition: cross-entropy measures the divergence between the predicted distribution and the ground-trutt distribution. In language modeling, the ground-truth distribution is usually implemented takes the value 1 at the ground-truth token and 0 elsewhere. Let sv, denote the score of ground-truth token e.. Then cross-entropy loss is log S Given a corpus, it is the average of all - log S
one-hot vector that
The core of our decomposition is to group cross-entropy terms by their associated RBE values. Consider corpus with N inputs. For each input, the model produces cross-entropy item. We
4

---

Published as a conference paper at ICLR 2026
Cross-Entrapy (Wikipedia)
frror Error-Entropy (Wiipadia)
Self-Alinmmet (WIkipedia)
Confidence (WiMppddia)
02
Staps [e) Cross Entropy (C4)
Siaza
Stape
Error Entropy (C4)
Salf Alignment (C4)
(h) Confidincce (C4)
0.4
0.2
(I) Cress Entropy (Ghhub)
@ Errar-Entrapy (Githuh]
(3] Self-Alignment (Githb)
ID Confidence (Github)
0.30
0.1
0:25
62
5.12
5.00
10
32"
101
100
100
10
2yth 8- 500m
1yttlb-413m
-Mh PyETS 3b
Figure Evolution of cross-entropy and its decomposed components during training on three datasets. All components are indeed optimized during training, suggested by the mathematical de- duction: Error- Entropy steadily decreases, Self-Alignment declines in the end, and Confidence term increases. The shaded regions highlight the training progress with the most rapid metric changes.
group these cross-entropy items by putting those with the same RBE values in the same group. Formally, the cross-entropy loss can be rewritten as:
LCE
N Σ log Sv,
Σ
log Suz
N
Let ne be the number of items in the group whose RBE value equals By converting the summation of logarithms into a logarithm of products, the above expression can be further rewritten as:
LCE
log
II
8u;
Σ Pe log
II
Soz
N
nc
:RBB():)=C
(RBE(())=
Q.
Note that above equation uses Eq, 2 and Eq, 3 to replace the variables with pe and Qe. Next, according to Eq. 4 each Qe can be further expressed Qe C. ge, yielding:
LCE = - log Qe - Pe log qe + log C) Σ Pe log qe log C. Epe.
pe.
Since Pe is a probability distribution, we can simplify E. Pc = 1. Moreover, the cross-entropy between pe and qe can be rewritten as the sum of the Shannon entropy of pe and the KL divergence between Pe and qe, yielding:
LCE = Σ - Pe log
Pe 9e Pe
- log C== Σ log +
Σ Pe log
Pe
log C
(8)
qe
Contidence
Error-Entropy
Self-Aligmment
Based the above mathematical derivation, decompose the cross-entropy loss into three com- ponents: (i) Error-Entropy: the Shannon entropy of pe, (ii) Self-Alignment the KI, divergence between Pe and qe. and (iii) Confidence: the logarithm of the normalization constant C.
We analyze the training process to empirically verify that these components constitute the cross- entropy loss and optimized throughout training Fig. 3 shows their values during the entire
EEperimental details for this section are provided in App. A.1

---

Published as a conference paper at ICLR 2026
(hip 124
18-1
Probubiitty
z0-1
200
400 933
3000
Error
Bank
Figure 4: Evolution of RBE distribution Pe during training. It shows that RBE distribu- tion shifts from nearly uniform to concen- trated, thereby minimizing Error-Entropy.
Figure 5: Output score distribution when ground- truth tokens are ranked at 2 and 4, respectively. Scores drop for tokens ranked lower than ground- truth tokens, thereby increasing Confidence term.
training process. We can see that all three components are optimized during training. Error-Entropy and Self-Aligmment decreases, and Confidence term increases. Moreover, the different magnitudes of the three components lead to difference in the order of optimization during training. Since Error-Entropy has the largest initial value, the model focuses on reducing it first in the early stages of training. In contrast, Self-Alignment and Confidence terms the model begins to optimize them only after Error-Erte Entropy has been largely minimized. Therefore, our decomposition captures the dynamics of training in a clear and interpretable way.
much smaller in magnitude, and
3.3 COMPARISON OF THE THREE COMPONENTS
Now, we compare the three components based their optimization objectives.
Error-Entropy: Error- Error-Entropy uses the form of Shannon entropy to quantify how severely the model makes mistakes. Mathematically, minimizing this term requires the RBE distribution to be- come as concentrated as possible. Achieving this means the model shall learn to identify which token is the correct one. Fig. 4 illlstrrtes the evolution of the RBE distribution during training. At step O, when the model is untrained, the RBE distribution is nearly uniform. This indicates that the model has no understanding of which token correct and is essentially ranking tokens random. At this stage, Error- Entropy is maximal. As training progresses, optimization drives the RBE distri- bution to concentrate toward the head. This means that the model increasingly able to recognize the correct token and place it at the top of the ranking. Therefore, optimizing error-entropy directly encourages the model to develop a clear distinction between right and wrong.
Self- Aegmmeligmmnt: Self elignAligmment uses KL divergence to measure the difference between the RBE distribution Pa and the normalized score distribution ge- Mathematically, minimizing self-alignment requires qe to exactly match Pe- This provides a new perspective how to interpret model's s output scores. Unlike cross-entropy, which assumes that the predicted probabilities approximate the true distribution of language, self-alignment suggests that the model instead assigns probabilities based on its own likelihood of making errors. To illustrate this, Fig. 6]shows Pe and qe distributions different stages of training. Early in training, the two distributions diverge significantly. However, after sufficient training, they become almost indistinguishable. This indicates that the model learns to align its output probabilities with its internal error distribution. Since different models have dif- ferent error distributions, they also produce different probability scores. If, by contrast, probabilities truly represented the single real language distribution as suggested by cross-entropy, we would not expect variation in probability scores across models, which deviates from empirical observations (Guo et al. 2017 Liang et al. 2023).
Confidence: Confidence term appears with a negative sign in our decomposition, meaning that it increased during optimization. It is the magnitude of the probability scores of correct tokens, formally expressed as log Σ. Qe. Note that Q. is the probability score for all tokens ranked at e + position. Thus, the maximum value of Q. is 1/(e + 1). which corresponds to the case where the top + tokens share equal probability scores while the rest tokens receive probability zero. If this maximum is achieved for every e, the Confidence term becomes log(E, 1/1). Since the harmonic series diverges, the confidence term could be very large. Intuitively, this extreme case represents highly confident model: it is confident that the tokens ranked lower than e + 1 can never be correct answers and thus are assigned zero probability. This motivates the name Confidence term. Fig.

---

Published as a conference paper at ICLR 2026
16
10000
Diz1 beiis
10
10
10
160
331
101
127
101
105
10
Emor
Ernor
Emor
Error
Figure 6: Evolution of RBE distribution Pe and normalized score distribution q. during training. The two distributions gradually become identical, thereby minimizing the Self-Alignment term.
Table 1: Comparison of power-law fit quality for cross-entropy and its three components. Best values within each family are bolded. Error- Entropy achieves the highest R2 in nearly all cases and even outperforms cross-entropy.
A
Wikipedia
C4
GliHuuh
CB
SA
CONF
CE
EE
SA
CONF
CE
EB
SA
CONF
Qwen
0.9731
0.9753
0.9441
0.2977
0.9493
0,9529
0.9896
0.9455
0.1371
Pythio
8.9448
0.0190
0.812
0.9930
0.9225
0.3082
0.8366
0.7058
0.7328
0.0760
0.6965
GPT2
0.9577
0.9734
0.7212
0.6518
0.9892
0.9872
0.3357
0.9444
0.0717
0.9262
0.05%6
0.1167
All
0.8421
0.5626
0.3553
0.0202
0.8699
0.9012
0.2188
0.0492
0.6743
0.7229
0,3233
0.0203
illustrates how models behave in practice. It shows the output score distribution for two cases. In the first case, the correct token is ranked at 2 (RBE = 1). We can see that the probability mass drops sharply after the third token. In the second case, the correct token is ranked at 4 (RBE = 1). We can see that the drop occurs after the fifth token. Both cases echo our above analysis that models try to be confident in their predictions and assign small scores to lower-ranked tokens.
We believe that error- best captures the performance of a language model. Unlike the other terms, error-entropy depends only on the ranking of tokens rather than their raw probability scores. Minimizing it requires the model to place the correct token ahead of incorrect ones, which directly reflects the ability distinguish right from wrong. Moreover, because it is independent of probabil- ity values, error-entropy is immune to post-processing techniques such as temperature rescaling or top-p sampling, making a robust metric. By contrast, both the self-alignment and confidence terms are tied to probability scores. While they describe how the model assigns scores to tokens, they do not fully capture model accuracy and are more vulnerable to distortions introduced by sampling strategies. In summary, through this decomposition, we obtain fine-grained view of model behav- ior and identify error-entropy as the component that most faithfully reflects model performance.
4 ERROR-ENTRON SCALING LAW
In this section, we systematically examine the scaling behavior of the decomposed components of the cross-entropy loss. Sec. 4.1 visualizes how different components change when model size is scaled up. Then Sec. 4.2 quantitativey examines the scaling law.
4.1 QUALITATIVE ANALYSIS
In this section, we study how the three decomposed Enror-rr (EE), Self- Aligmment (SA), and Confidence (Conf), scale with model size The results are shown in Fig. 7
We can see that, among the decomposed components, only Error-Entropy exhibits clear scaling property. Specifically, in a log-log plot, we observe that Error-Entroppy decreases approximately linearly with increasing model size, closely resembling that of the cross-entropy. In contrast, Self- Alignment does not improve with larger models but instead shows an overall upward trend, while Confidence term displays larger variance and lacks consistent pattern.
Moreover, we find that Error- Ertor-Entropy generally scales better than cross- cross-entropy. In particular, most points for Error-Entropy lie closer to the fitted line. For example, on GitHub dataset, some data points of cross-entropy are far from the fitted line, while the points of error-entropy are generally closer. This suggests that Error-Entropy shall be the truly scaled part of cross-entropy.
Experimental details for this section are provided in App. A.2

---

Published as a conference paper at ICLR 2026
bai Cross Entropy pelccpeia)
EntonSntropy Kinelial
gerent (eldpedda)
Cerl dence kizedial
4x
003
604
3x22
603
2x00
2+277
30
300
139
100
D011
107
101
Jet Cross Entrosy (C4)
10 Brioe Entropy (C4)
1g) Seaiggmment (C4)
Ohi Corfieecee (C4)
0003
6.5-
2x227
3x
0.33
2x00
0.12
324
301
Cross (Gktub)
Seffidigmment (Shhub)
Conffdrncee (Githu))
4x
2x19"
237
30"
25'
25"
10
20th
1011
36a
10" 1011
14'
22'
10"
10
10"
1.7
35'
107
10
Figure 7: Illustration of cross-entropy and its decomposed components for different model sizes. Different rows correspond to different datasets. Only cross-entropy and Error-Entropy exhibit clear scaling behavior, with Error-Entropy lying closer to the fitted power-law line.
Table 2: Comparison of exponent difference (|A) for cross-entropy components. | measures the difference scaling exponent between each component and cross-entropy. Best values are bolded. Results show that the scaling exponent of Error-Entrop: is closest to cross-entropy.
Wikipedia
C4
GitHub
EE
SA
CONF
EE
SA
CONF
EE
SA
CONF
Qwen
0.0104
0.2347
0.0786
0.0069
0.1599
0.0652
0.0214
0,2138
0.1234
Pythia
0.0057
0.0494
0.0059
0.0038
0.0969
0.0354
0.0178
0.1171
0.0344
GPT2
0.0069
0.1585
0.1076
0.0059
0.0904
0.0883
0.0352
0.2126
0.0866
All
0.0147
0.2678
0.1002
0.0058
0.1672
0.0625
0.0406
0.3307
0.2116
4.2 QUANTITATIVE ANALYSTS
We further examine the scaling properties of the components with quantitative fitting experiments.
We use two metrics, namely R22 to examine how well the results follow power-law rate, and A to examine how close the scaling behavior is to that of cross-entropy. They are defined as follows. Let N be the number model eoneeddddingg parameters. For each metric M E {CE, EE, SA, Conf), we perform a log-log linear regression across models and report the goodness of fit R²:
log M CM + a. log N
where cM is the intercept and 0.M is the scaling exponent (slope). We further define the scaling difference between each decomposed component and cross-entropy to quantify their scaling consis- tency: | jaM- acE|. where smaller A indicates closer alignment with cross-entropy. The results across datasets are summarized in Table (for R² values) and Table (for A values).
The quantitative experiments further support our hypothesis that only Error-Entropy consistently exhibits scaling behavior. Specifically: (i) Error-Entropy scales robustly across both single-model families (Qwen, Pythia, GPT-2) and the mixed-model setting, achieving strong fits with R² values close to 0.9 across Wikipedia, C4, and GitHub. (ii) Self-Aligmment lacks clear power-law pattern. Apart from the Qwen series showing relatively strong fitting, most other cases yield lower I22 values. Moreover, displays the largest | among the components, indicating the greatest deviation from cross-entropy. (iii) Confidence term is effectively signal-poor -poor. Its R² values are highly unstable across model families and datasets. the mixed-mode case, they drop to extremely low levels (0.1002, 0.0625, 0.2116), suggesting the absence of consistent scaling. Although its | is not
8

---

Published as a conference paper at ICLR 2026
3.85
0.43
Slgerert
Micethan har s000
Mcde soidd
(a) Wikipedia
(b) C4
(c) Pilc-Github
Figure 8: Relative ratio of decomposed components at different model sizes. For small models,
ErtorEntropp dominates (about 80-90%). Its share gradually decreases as model size grows.
always large, the overall evidence implies that Confidence term primarily reflects noise-dominated curves rather than a stable power-law relationship.
Taken together, these results demonstrate that Error-Entropy is the true driver of cross-entropy scal- ing. Its 1?² values exceed those of cross-entropy in nearly all settings, which suggests that it is the truly scaled part within cross-entropy. The observation that Error-Entropy consistently achieves the smallest | across all cases shall be the result of Error-Entrop being the true scaling driver of cross-entropy. Therefore, these results indicate that cross-entropy scaling originates in the behavior of Error-Entropy Scaling Law.
5 APPLICATION OF ERROR-ENTROPY SCALING LAW
Our decomposed result and the discovered Error-Entropy scaling law better characterize the model scaling behavior. We believe they will have wide applications for LLMs. In this section, we provide an example in using them to understand the puzzle of slowing cross-entropy scaling law.
The puzzle that cross-entropy scales precisely a power-law rate for small models, and yet slows down for very large models. Concretely, for small models, Kaplan et al. (2020) described the relationship between cross-entropy Lce and model scale M as LCE O M which is a pure
power-law form. Yet, when model size grows, OpenAl et al. (2024) found cross-entropy loss starts slow down. To account for this problem, the expression rewritten
Lee M- bias.
Nowadays, researchers even start to find out cross-entropy slows down even more (Qin et al. 2025
Lourie et al. 2025). This phenomenon has been widely observed but poorly understood.
Our decomposition and Error-Entropy scaling law offer natural explanation. Since only Error- Entropy scales reliably, the validity of cross-entropy scaling depends on how much of the total loss is contributed by Error-Entropy. Fig. & shows the relative ratio of the three components" For small models, Error- Entropy dominates, accounting for nearly 90% of cross-entropy. and the overall loss therefore appears to follow clean power law. For larger models, however, the proportion of Error-Entropy declines, while the non-scaling components- -Sel--Seffaeee -eef-- Self-Aligmment and Confidence take up larger share, causing cross-entropy deviate from the expected power-law. This explains the observed breakdown of cross-entropy scaling.
DISSCUSION AND FUTURE WORK
In this section, we further discuss the contributions and theoretical implications our work. We analyze the deeper principles behind Error-Entrop in Sec. 66.1 We also outline possible training objectives and future directions motivated by this perspective in Sec. 6.2
6.1 DEEPER PRINCIPLES BEHIND
Although definition of Error-Entropy (EE) arises from cross-entropy decomposition, believe reflects deeper underlying principles. In information--heoretii learning (TTL. and signal process- ing field, error entropy has been studied extensively (Erdogmus & Principe) 2002 Liu al.) 2006 Li et al., (022). Our work establishes a link between these fields and large language models, offering new perspective on model behavior and training dynamics.
*Eperrmmnnta details elaborated in App. A.2.2

---

Published as a conference paper at ICLR 2026
While our formulation of Error-Entrop differs from the classic ITL version, the two are closely connected. ITL studies focus on nonparametric regression. where both imput and output are real- valued. The error is defined objective. A common -syle error entropy is: (e) modeling is classification task where each token in the vocabulary assigned probability, and the goal is to raise the rank of the correct token. We define a rank-based error that restores an error distribution in this setting, and use decompose cross-entropy into error-related and probability- related components. Our error-entropy shares similar form with the ITL definition, which links the probabiiityy-aas based formulation used in LLM training to the error- grror-dstribbutinn perspective from ITL.
y Ypred, which makes optimizing the error distribution natural p(e) log p(e) de. In contrast, language
This connection paves the way for new line of research. From an optimization perspective, tech- niques developed in ITI. kernel-based methods for minimizing error entropy (Chen al., 2013] 2019) may inspire new training objectives for language models. These approaches often use stochas- tie gradient methods in kernel spaces, and have been shown to handle heavy-tailed and -aa-Gaas noise effectively. From theoretical perspective, existing insights from ITL can be revisited in the LLMs setting to explain the observed scaling behavior of EE. In particular, the robustness of error- entropy under noisy mismatched supervision, one of its original motivations in ITL (Tang & Li, 2011). may also help explain why EE scales more consistently than CE in large-scale models.
6.2 TRAINING OBJECTIVES INSPIRED BY ERROR-ENTROPY
In this subsection, we show that is feasible to design practical training objectives inspired by Error- Entropy. EE itself is non-differentiable with respect to the logits, so we instead consider surrogate training objectives. Specifically, we define loss that adds a penalty term on Confidence:
Lx CE CONF, 0 <<<1.
(10)
We penalize the Confidence term because it is not directly related ranking ability. As discussed in Section 3.3 increasing Confidence alone does not bring significant improvement in performance. Moreover, our empirical results (Fig. 8 shows that its share in cross-entropy grows with model size, which suggests that current training over-emphasizes this component. By reducing its weight in the loss, we aim shift optimization toward EE, which is the component with more desired scaling.
Mathematically, the compensated loss is differentiable with respect to the logits, so it can be op- timized with standard gradient-based training. In our notation, cross-entropy can be written as N Σ, log 8,. where s; is the score of the correct token for example i. The Confidence term takes the form CONF log E. Qe, where Qe is the mean score over examples with error level e. Differentiating Ls with respect to 8; yields (See D the full derivation.)
CE 1
OL*
(1-x Qs
1
(11)
0s: Ns;
Pe
where Pe. is the error distribution and ge is the score distribution. With original cross-entropy loss, once token attains the correct rank, further increasing its probability only raises qe without im- proving Pe The factor (1 - X p. encourages the model to keep qe closer to Pe, so that optimization focuses more on improving the error distribution captured by error-entropy.
Beyond such differentiable surrogates, error-entropy can also be used directly as non-differentiable objective in peafforcement-learning style fine-tuning, where serves as a reward signal that depends only ranks. This avoids differentiating error-entropy directly and may provide a complementary route to align training with error-entropy.
7 CONCLUSION
In this paper, we propose a mathematical decomposition of the cross-entropy loss and conduct wide- ranged experiments to find out the truly scaled component. Results show that our decomposition clearly characterizes model behavior during training. Based on our decomposition, we find that Error-Entropy is the truly scaled part hidden within cross-entropy, while the other two components do not scale. The proposed decomposition and new scaling law provide a novel perspective to understand model behavior, such as how models assign probability scores and why the rate of cross- entropy scaling slows down. We believe they will facilitate a more fundamental understanding of model mechanics and may enable training paradigms in the future.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27). June 2024. ISSN 1091-6490. doi: 231p://pa....111811111 UtLhttp:/ /xx wwtpdxx.ooo...oooo 7311878121
paas
Matthew Barnett. An empirical study of scaling laws for transfer, 2024. URL https://arxix:
org/aba/ 2408 16947
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https: : wtt.aaivvvvvo........ org/ abs 2304 01373
Badong Chen, Zejian Yuan, Nanning Zheng, and José C. Principe. Kernel minimum entropy algorithm. Neurocomput. 121:160-169, December 2013. ISSN 0925-2312. doi: 10.10
neucom.2013.04.033 wtR.httts::... htttttt/t/pooccooto otre 101 wew.o...ccmmmmmmmmm
Badong Chen, Lei Xing, Nanning Zheng, and José C. Príncipe. Quantized minimum emvor entropy criterion. IEEE Transactions on Neural Networks and Learning Systems, 3065):177-1138) 2019. doi: 10.1109/TNNLS..... 2018.28688122
Zhengyu Chen, Siqi Wang. Teng Xiao, Yudong Wang. Shiqi Chen, Xunliang Cai, Junxian He, and Jingang Wang. Revisiting scaling laws for language models: The role of data quality and training strategies. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers). pp. 23881-23899, Vienna, Austria, July 2025. Association for Com- putational Linguistics. ISBN 979-8-89176-2511
doi: 18.18653/v 1/2025 long.1163,, URL www.///.oooooooooooooooooooooooo 2025.
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus, 2021.
0004 08758
D. Erdogmus and J.C. Principe. An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems. IEEE Transactions on Signal Processing, 50(7):1880-1786, 2002. doi: 10.1099TSSP220.1111111
Stanislav Fort and Adam Scherlis. The goldilocks zone: Towards better understanding of neural network loss landscapes, 2018. htttts://////ttt// oww.or///aao.... httg:/bb//10000 .25811
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL.ttts:: htt.ttts///axxiv.rrrg abs/2101.0000
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation, 2021. URL.
fttts:://ariis orgg abss 0tt.//119.00777
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks, 2017. URL huttps://aaree www.arwwww....oaa.oooooooooooore wwwp/rggg/aa/ooooee/..ooeeeeesee //bs/106.. 04599
Mandy Guo, Zihang Dai, Denny Vrandecic, and Rami Al-Rfou. Wiki-40b: Multilingual lan- guage model dataset. In LREC 2020, 2020. URL httt:://ww..recccoo org/ 1rec- 22111.96...1 296.pd. pdf
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Hee- woo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling, 2020. URL hetps:
7/arxiva wtt.v/.rggabbb///s/ 14700 14701
11

---

Published as a conference paper at ICLR 2026
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021. /tt.://////xx....//////.aaaa
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017. URL hattts://arxiv.... http://axvvvv.o/ org /ttp///bba... aba /1t.. 1712.00409
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL https: http://axxvv.r.r/oo 2203. 15556
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng. Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xu- ancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report, 2024. URL fttps://arxiv: org/ abs 2409. 12186
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b, 2023. URL https:
2630000625
Arlind Kadra, Maciej Janowski, Martin Wistuba, and Josif Grabocka. Scaling laws for hyperparam- eter optimization, 2023. URL http:///aaxiivvoooa///////////// wtt../r///bb//2....4..../ 00441
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffiey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. https:/// http://aaxvv.vv.. org/ /201 00366666600000000
Houyi Li, Wenzhen Zheng, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan, Zhenyu Ding, Haoying Wang, Ning Ding, Shuigeng Zhou, Xiangyu Zhang, and Daxin Jiang. Predictable scale: Part i, step law optimal hyperparameter scaling law large language model pretraining, 2025.
04715
Yuanhao Li, Badong Chen, Natsue Yoshimura, and Yasuharu Koike. Restricted minimum error en- tropy criterion for robust classification. IEEE Transactions Neural Networks and Learning Sys- tems, 33(11):6599 November 2022. ISSN 2162-2388. doi: 10. 1099/00ls.2002..000202 3308551.. URL httt::://...oooiitottttt
1099 TNNLSS 2021 3082571
Percy Liang. Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuk- sekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattah, Peter Hen- derson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yi- fan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models, 2023. URL https: arxiv. org/ abs abwpg//b////
2211 09110
Weifeng Liu, P. Pokharel, and J. C. Principe. Error entropy, correntropy and m-estimation. In 2006 16th IEEE Signal Processing Society Workshop on Machine Learning for Signal Processing, pp. 179-184, 2006. doi: 10. 00.1099MMLSP:200227555..
Nicholas Lourie, Michael Y. Hu, and Kyunghyun Cho. Scaling laws are unreliable for downstream tasks: A reality check, 2025. URL hhttps::/aatt wttpx//vvvvv.//////e//eee/eaaev /ttp////aw...////////5/........ 00885
12

---

Published as a conference paper at ICLR 2026
Bochen Lyu, Di Wang, and Zhanxing Zhu. A solvable attention for neural scaling laws. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /oopenzvieww.o wttp:eeeeoooommmmmmddtaaaaaaaaaa
Anqi Mao, Mehryar Mohri, and Yutao Zhong. Cross-entropy loss functions: Theoretical analysis and applications. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 23803-23828. URI. httpp:://procttt
PMLR, 23-29 Jul 2023.
ngs ml www..press/v2222. mmao23b.
html
Eric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling, 2024. URL https: /ttp:://axxxvvvooo aa///b///22000 13506
OpenAI, Josh Achiam. Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Sermaadet-Shaping Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang. Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte. Isabella Fulford. Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes. Jonathan Gordon, Morgan Grafstein, Scott Gray. Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim. Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, tukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichir Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder. Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry. Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov. Jessica Shich. Sarah Shoker, Pranav Shyam. Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich. Hannah Wong, Lauren Work- man, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao
13

---

Published as a conference paper at ICLR 2026
Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL
https:// hatpp::////xvv.../// wttp://////33.00/..77
Zeyu Qin, Qingxiu Dong. Xingxing Zhang. Li Dong, Xiaolong Huang, Ziyi Yang. Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, and Furu Wei. Scaling laws of synthetic data for language models, 2025. URL https:
//aasii.orr/abs//5533.. 1555.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. URL https://api.
Language models are unsupervised multitask learners.
semanticscholedt ott.oorrrrr/Coorr.D.I 60025533
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias of gradient descent on separable data, 2024. URL attps://arxiv.or///a
1710. 10345
Huajin Tang and Haizhou Li. Information theoretic learning: Reny's 's entropy and kernel perspectives (principe. 2010) [book review]. JEEE Computational Intelligence Magazine, 6(3):60-62, 2011. doi: 10.1100./111 10.000///L22011.000 1.941592.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation
language models, 2023. URL http::/ /eattps://ariv... www.iii..oogaabaa..o 2202. 13971
Artem Vysogorets, Anna Dawid, and Julia Kempe. Deconstructing the goldilocks zone of neural network initialization, 2024. URL http:/tttee//aaab...o/////////// httpb////2222000059...
Jason Wei, Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Perey Liang. Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022a.
/awxiv org/ abs/ 220 6 07682
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels, 2022b. URL httpttt/////xii.orrrg
2106.04144
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12104-12113, June 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. Opt: Open pre-trained transformer language models, 2022. URL https://arxiv.
org/aba/2205 2205 01068
Zhengkang Zhang. Neural scaling laws from large-n field theory: Solvable model beyond the ridge- less limit, 2024. URL https:://///mm //atxps://rxxv...o/ http://axxxi..../a////////////// 19398
A MORE DETAILS OF EXPERIMENTAL SETUP
A.1 EXPERIMENTAL SETUP FOR TRAINING DYNAMICS
This section summarizes the general experiment settings used in our training-dynamics analyses. The detailed settings for each individual experiment are placed in Table 4
Models. We adopt the Pythia family (Biderman et al. 2023 at 160 M, 410M, and B parame-
ters. Pythia is widely used scaling-law and training- dynamics research because it (i) provides a consistent architecture across sizes, and (ii) publicly releases dense training checkpoints, enabling fine-grained temporal analysis without re- re-training. Unless otherwise noted, we evaluate checkpoints (0, 16, 128, 1 0000, 10 000, 100 000), chosen to span orders of magnitude in optimization steps.
14

---

Published as a conference paper at ICLR 2026
Table 3: Total number of evaluation tokens used for the main experiments.
Wikipedia
C4
Pile-GitHub
# Tokens
704,185
417,140
1,091,481
Datasets. We use three English corpora: Wikipedia (20231 101 snapshot) (Guo et al. 2020 which offers clean encyclopedic prose and commonly included in pretraining pipelines: C4 (Dodge et al. 2021), a large-scale web web-derived corpus covering diverse styles and domains; and the GitHub subset of The Pile (Gao et al., 2020), which introduces code-like sequences and thus serves deliberately harder distribution stress-test our decomposition.
Sampling protocol. Unless otherwise noted, evaluate the first 7 100 instances from each dataset (after shuffling with a fixed seed 42). Note that these instances are full sequences con- taining many tokens; the total number of evaluation tokens is summarized in Table B We further verified robustness by repeating selected experiments with substantially larger sample sizes (e.g., 10k instances), and observed identical trends, indicating that conclusions insensitive the particular sample count or ordering.
Metries. For every evaluation token, we record the softmax score of the ground-truth token and its Rank-based Error e (the rank position of the ground-truth token). From these, each checkpoint we estimate the empirical Error distribution Pe, the Score distribution qe, and the global Confidence terms statistic C. Then the corpus level we report CE and its three components- Ereiis----rr----ree Error-Entropy (EE), Self-Alignment (SA), and Confidence (Conf)-as defined in Sec. 3.2
Table 4: Experimental setups for Fig.
Fig.
Data
Model & Checkpoints
3
Wikipedia (1k), C4 (Ik), Pile-GitHub (1k)
Pythia- 160M/-410M/-IB @ (0.16,128, 10K,
4
Wikipedia (Ik)
Pythia Pythia-160M @ [0.128,10k]
Wikipedia (Ik)
Pythia-70M @ (32k)
Wikipedia (1k)
Pythia-410M @ (0,16,128,10k)
A.2 EXPERIMENTAL SETUP FOR SCALING
Datasets. We use three English corpora: Wikipedia (20231 101 snapshot) (Guo et al. 2020 which offers clean encyclopedic prose and commonly included in pretraining pipelines; C4 (Dogge et al.] [2021), a large-scale web-derived corpus covering diverse styles and domains; and the Hub subset of The Pile (Gao et al., (020), which introduces code-like sequences and thus serves deliberately harder distribution to stress-test our decomposition.
Sampling protocol. Unless otherwise noted, we evaluate the first / 100 instances from each dataset (after shuffling with fixed seed of 42). Note that these instances are full sequences con- taining many tokens; the total number of evaluation tokens is summarized in Table 3 We further verified robustness by repeating selected experiments with substantially larger sample sizes (e.g., 10k instances), and observed identical trends, indicating that conclusions insensitive to the particular sample count or ordering.
A.2.1 MODELS FOR QUALITATIVE SCALING ANALYSIS
Model. For the qualitative study of scaling trends, we selected 16 pretrained models covering wide range non-embedding parameter counts, from millions to tens billions. The models, listed in order of increasing non-embedding parameter counts, are: pythia setthia-11 -14, distil pettilgltt2t 2, gpt pythia-160m. 600m pythia-41 0m, qwen2. 5-0. 5B, opt-1. 3b, gpt2-x1, 11ax1,1aann 1lama-3,am.m 2-1B, swen2.5-3B, 11ama-3.2-10
opr-6. 7b, -lama-2-133 2-13b-hf. qwen2. 5-14B, 11ama-2- 1aama-2-70ba -h- This selection spans 14M to 70B models, enabling qual- itative analysis of scaling behaviors across different parameter regimes.
mistral-7B-vae
15

---

Published as a conference paper at ICLR 2026
A.2.2 MODELS FOR QUANTITATIVE SCALING ANALYSIS
Model. For the quantitative scaling experiments, we evaluate over 30 pretrained models, ranging
from 14M to 70B parameters. All parameter counts are reported as non-embedding parameters, i.e., excluding the token embedding layers. These models cover diverse families including GPT-2 [Radford et al. 2019), OPT (Zhang et al. 2022), Pythia (Biderman et al. 2023), LLaMA (Touvron
ct al., 2023), Mistral (Jiang et al.) (023), and Qwen(Hui et al. 2024). Table 5 summarizes the full
set of models and their parameter scales.
family
model
params
family
model
params
gpt-1/2
openai-gpt
85,054,464
pythia
pythia-14m
1,189,888
gp12
85,056,000
pythia-31m
4,739.072
gpt2-medium
1,2,311,424
pythia-70m
18,915,328
gp12-large
708,390,400
0ythia-160m
85,056,000
gpt2-xl
1,475,561,600
pythia-410m
302,311,422
opt
opt- 12-m
85,056,000
pythia- 1.thia-1.4bb
1,208,602,624
opt-350m
303,357,952
pythia-
2,517,652.480
opt-
1,206,00,6224
pythia-6. 6.9b
6,444.163,072
opt-2.7b
2.517,652,480
Ilama
Ilama-3 1lama-322-bb
973,146,
opt-6.7b
6,444,163,072
llama-3.2-1
2,818,747,392
qwen
qwen2.5-0.5b
357,898,
Ilama-7b
6,476,271,611
qwen2.5-1.5b
1,310,340,608
meta- 1ata--llama-3-8b
6,979,588,00
qwen2.5-3b
2,774,773,760
llama-2-13b--
12,688, 184,320
qwen2.5-7b
6,525,621,760
llama-2-70b-hf 68,452,360,
qwen2.5-14b
13,212,898,304
mistral mistral-7b-vO.1
6,979,588,096
qwen2.5-72b
70,214,787,072
others
distilgp12
42,528,768
Table 5: Full list of evaluated models and their non-embedding parameter counts.
B EXTENSIVE TRAINING DYNAMICS EXPERIMENTS
In Figure we illustrate how cross-entropy and its decomposed components evolve throughout training. The plots cover three datasets and track training progress on Pythia-122 one of the largest open-source models with available checkpoints. This confirms our main results by showing that all three components are indeed optimized during training.
C EXTENSIVE QUALITATIVE ANALYSIS
In Figure 10 we report extended qualitative results for cross-entropy and its decomposed compo- nents across model scales. This figure uses the same of 32 models as in our main quantitative analysis and further includes Meta-L.ama-3.1 1-0--3.--00B one of the largest open-source language models currently available.
The overall trends are fully consistent with our main results. Both cross-entropy and Error-Entropy exhibit clear scaling behavior, with Error- Ervor-Entropy lying systematically closer to the fitted power-law line. In contrast, the Self-Alignment and Confidence terms do not show stable decreasing pattern with model size and display much larger variance. Importantly, adding the 405B-parameter model does not change these qualitative conclusions, reinforeing claim that Error-Entroe Entropy is the robust component follows a scaling law in the open-model regime we can study.
D DERIVATION OF THE COMPENSATED LOSS GRADIENT
We collect here the notation and give detailed derivation of the gradient for the compensated loss
Lx CF X CONF, 0<
(12)
16

---

Published as a conference paper at ICLR 2026
Cross-Entrapy (Wikipedial
Error Ertor-Entrapy (Wilipedia)
Self-Alinmmet (WIlipedia)
[40 Confidence (Wikppddia)
12
32-
1.5
62
00-
Staps
Staza
Suea
Stapa
[e] Cross
(C4)
Confilincce (C4)
1.-
0.2
-0.-
Stope
Strpe
Craes-Entropy (Github)
Errar-Entrapy [Ghthuh)
(1] Self-Alignment (Ghhub)
Confidence (Github)
1.2-
0.5
0.2
a:22b
Figure 9: Evolution cross-entropy and its components during training across three datasets with Pythia-12B.
Notation. We consider corpus with N evaluation tokens indexed by i 1, ,,..., N. For each token i: denotes the softmax score of the ground-truth token, denotes its rank-based error level.
For each error level e, we group tokens via Ie {i e, c).n, ng = /Lel. and define the error distribution Pe Re/N and the geometric mean score Qe (II.e1. s.) 1/me We then aggregate over error levels by C E. Qe and qe Qe/C,
In this notation, the corpus-level cross-entropy and Confidence terms are
CE
Σ
log Si.
(13)
N 1=1
CONF = log C = log E Qe.
(14)
Derivation. We treat the scores {si} as independent variables.
Cross-entropy term. The derivative of cross-entropy with respect to 8; is
àCE
1
(15)
Os;
N
No;
Confidence term. For the geometric mean Qe we have
log Qe The gel.
Σ 1jg
(16)
Thus, for token i with error level Cis
Olog Q.
1 1 The 5;
aQc
Qe
(17)
Os:
Øs:
ne
Since C = Ee Qc, only the term
6; depends 8i,
ac
aQe,
Qe
(18)
dsc Os:
ne, 8i
Using CONF = log C, we obtain
aCONf 1 ac
Qe.
ge.
0s;
C 0s;
=
Cne, 8i
Ne,
(19)
17

---

Published as a conference paper at ICLR 2026
Cross (wkinedia)
Ernoe
Sef mmmert (ifeppdda)
Canfdence
2:10
IXIE
0.2
(6) Cross Entroyy (045
Erroe ony (C4)
190 Sefiiligment (C4)
Confidence
4x30
1x104
suge
62
136"
10
16
104
104"
(Ghrub)
(Gthub)
Sefffgrmee Nithus)
Confidence (Gitust
227
10
30
127
30
Figure 10: Extended illustration of enoss-entropy and its decomposed components over model size.
where we used d. Qe/C. Recalling that Pe ne/N N so that ne Nee. we can rewrite this as
acONf
ge:
ges
(20)
,s;
NP., St
Ns;
Pe.
Compensated loss. Combining equation 15 and equation 20 the derivative of LA CE-A-CONF with respect to 8;
OL*
OCF
aCONF
X
(21)
Os:
0s;
Øs;
qe:
(22)
Ns;
Ns;
Pe.
(1
les
X
(23)
Ns;
Pe:
Finally, since each score 8; is a smooth function of the logits via the softmax transformation, the compensated loss C, is differentiable with respect to the logits, and can be optimized with standard gradient-based training.
E STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
We used large language models (LLMs) solely for polishing the writing of this manuscript, including grammar correction, style harmonization, and minor sentence-level rewrites. All technical content- definitions, theorems, methods, experiments, and conclusions-wer authored and verified by the authors. Any LLM-suggested phrasing that was adopted was manually reviewed and edited. No unpublished data, proprictary code, or sensitive information was provided to any LLM. This usage does not affect the reproducibility of our methods or results. This disclosure follows the ICLR 2026 policy on LLM usage.
18


Reviews:
[{"review_id": "JcZPID7GTd", "reviewer": "Reviewer_6E3g", "summary": "This paper proposes an alternative rank-based metric, error-entropy, and argues that it has a more robust scaling law than CE. It shows that CE can be decomposed into three components: error-entropy, self-alignment, and confidence, and through experiments, authors demonstrate that only error-entropy scales consistently with model size, while the other components remain random. Since error-entropy becomes a smaller part of CE as model size increases, this can potentially explain why scaling laws fail for larger models.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novelty: the rank-based metric, error-entropy, as well as the CE decomposition are novel as far as I know. The error entropy has nice properties, in particular robustness to post-processing such as temperature scaling. \n\n2. Extensive experiments: this paper includes many experiments, covering the training dynamics of smaller models and the scaling of the three components in larger models. There is a variety of models and datasets, including text and coding tasks. \n\n3. Clarity: the paper is well-written and the exposition is clear.", "weaknesses": "My main concern is the significance of the findings:\n- Is there evidence that using EE, one can predict the EE of a larger model better than using CE to predict CE?\n- Is EE as indicative of downstream performance as CE? \n- Regarding self-alignment and confidence: from section 3, self-alignment decreases during training and confidence increases, so I would expect self-alignment to be smaller and confidence to be larger as model size increases. However, self-alignment has a clear upward trend as models get larger, and confidence doesn't have a clear trend at all. Are these metrics meaningful in practice? Why would self-alignment increase for larger models that have more capacity for learning?\n- In section 5, the paper argues that self-alignment and confidence might be the reason for the CE scaling law to break down at larger model sizes. However, in the scaling plot, it's not clear that there's a CE scaling breakdown and EE has more robust scaling. Can the authors provide evidence that EE doesn't suffer from the same issue?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SUNT9mPa09", "reviewer": "Reviewer_VUMH", "summary": "The paper decomposes cross-entropy into three components—Error Entropy (EE), Self-Alignment, and Confidence. It experimentally shows that the part that actually scales with model size is Error Entropy. Based on this, it proposes an explanation for why the cross-entropy scaling law fails at very large model sizes: as the model gets larger, the proportion of cross-entropy contributed by Error Entropy becomes smaller.", "questions": "1. In Table 1, GPT-2’s CE \\(R^2\\) on the GitHub dataset is only 0.0717, which is far from the other results. Is this a typo?\n2. Can you provide a possible explanation for why the share of EE in CE decreases as model size increases?\n3. Why are the models used in the qualitative analysis different from those in the quantitative analysis? In other words, since the quantitative analysis already covers 30 models, why does the qualitative analysis only show 16?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The overall structure is clear, the figures are appropriate, and readers can easily grasp the core ideas.\n2. The mathematical derivations are clear and rigorous.\n3. It offers a new perspective on the cross-entropy scaling law, which is thought-provoking.", "weaknesses": "1. Experimental results are not strong enough.\n   The authors aim to use Error Entropy to explain the failure of the cross-entropy scaling law at large parameter sizes. However, possibly due to the experimental setup or model size, the experiments in the paper do not clearly demonstrate the failure of the scaling law. In this situation, relying only on EE having a slightly higher \\(R^2\\) than CE (and not even for all model x dataset combinations) to argue that EE is the part that actually scales is not very convincing.  \n   If the authors could change the experimental setting (e.g. try different datasets, context lengths, or model sizes) to actually reproduce the failure of the CE scaling law, and at the same time show that EE still fits well, then the results would be much more persuasive.\n\n2. Confusing phrasing. \n  While the overall argument is clear and the main findings are well laid out, a few parts are awkwardly expressed. For instance, in Section 3.3 on Self-Alignment, lines 296–303 already state clearly that Self-Alignment is about aligning \\(p_e\\) and \\(q_e\\). However, the two sentences that follow (lines 304–308) are hard to follow. - “Since different models have different error distributions, they also produce different probability scores.” — this seems to claim a causal link between error distributions and probability scores, but the paper does not substantiate that claim (correlation does not imply causation). - “If, by contrast, probabilities truly represented the single real language distribution as suggested by cross-entropy, we would not expect variation in probability scores across models, which deviates from empirical observations.” — this is also unclear, because a model’s probability distribution can vary for many ordinary reasons (initialization, training dynamics, local optima, etc.), so variation across models is not in itself evidence against cross-entropy. If I am misunderstanding the authors’ intent here, clarification would help — the current wording leaves too much room for interpretation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ea5qdffZBD", "reviewer": "Reviewer_Vend", "summary": "This paper investigates the widely observed phenomenon that the cross-entropy scaling law, which predicts that model loss decreases as a power-law of model size, begins to break down at larger scales. The authors hypothesize that the root cause is that cross-entropy loss itself is not the quantity that truly scales. To test this, they introduce a novel and exact mathematical decomposition of the cross-entropy loss into three interpretable components: Error-Entropy, Self-Alignment, and Confidence. Through extensive experiments on 32 models spanning five orders of magnitude (up to 70B parameters), they find that only Error-Entropy, which measures the entropy of the rank of the correct token, exhibits a robust and predictable power-law scaling. The other two components remain largely invariant or noisy. The paper argues that the apparent scaling of cross-entropy in smaller models is an illusion caused by Error-Entropy being the dominant component. As models grow larger, its relative contribution diminishes, allowing the non-scaling components to disrupt the overall power-law trend. The authors propose the \"Error-Entropy Scaling Law\" as a more fundamental principle for understanding and predicting model performance.", "questions": "How many tokens are used to train the models in appendix A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The core idea of decomposing the cross-entropy loss is highly original. The three proposed components are not only mathematically sound but also intuitively interpretable. They map to distinct aspects of a model's learning process: getting the answer right (Error-Entropy), calibrating its output probabilities to its own error patterns (Self-Alignment), and expressing certainty by assigning low scores to incorrect tokens (Confidence). This provides a powerful new analytical tool.\n\n2. The paper offers a simple and elegant explanation for a complex and important problem—the breakdown of cross-entropy scaling laws. The finding that only one component truly scales, and that its proportion changes with model size (Figure 8), compellingly explains why the law holds for smaller models but falters for larger ones.\n\n3. The discovery of the Error-Entropy Scaling Law could have broad implications. It suggests a more reliable metric for extrapolating model performance and could inspire new training objectives focused directly on minimizing Error-Entropy, potentially leading to more efficient training or better-calibrated models.", "weaknesses": "1. The paper is motivated by the failure of scaling laws in the very large model regime, yet the analysis stops at 70B parameters. The most dramatic and puzzling scaling behaviors are often observed in models exceeding 100B parameters. Without data from these frontier models, the claim that the Error-Entropy law holds true at the absolute largest scales remains an extrapolation.\n\n2. The analysis of how the components evolve during training (Figures 3-6) is based on the Pythia model family, with the largest model being 1B parameters. While insightful, it's unclear if these same dynamics (e.g., Error-Entropy being optimized first) hold true when training much larger models (e.g., 70B), where training dynamics can be qualitatively different.\n\n3. The study exclusively analyzes existing pre-trained checkpoints. It does not explore how different training configurations (e.g., optimizers, data quality, learning rate schedules) might influence the behavior and proportions of the three components. It's possible that certain training strategies could alter the dynamics of Self-Alignment or Confidence, affecting the overall cross-entropy behavior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mOZhjcmGwo", "reviewer": "Reviewer_cnon", "summary": "This paper investigates why the widely used cross-entropy scaling law—which predicts that cross-entropy loss decreases as a power law with model size—begins to fail for very large language models. The authors hypothesize that cross-entropy itself does not truly scale, but rather only a hidden component within it does. To uncover this, they introduce a novel mathematical decomposition of cross-entropy into three distinct parts: Error-Entropy, Self-Alignment, and Confidence. Using a new metric called Rank-Based Error (RBE)—the rank position of the correct token—the paper expresses cross-entropy as the sum of the entropy of the RBE distribution (Error-Entropy), the KL divergence between that distribution and the model’s score distribution (Self-Alignment), and a confidence term related to score normalization. This decomposition offers a more interpretable view of what cross-entropy measures in language model training.\n\nThrough extensive experiments across multiple datasets (Wikipedia, C4, and GitHub) and 32 models spanning five orders of magnitude in size, the authors find that only Error-Entropy consistently follows a clean power-law scaling with model size, while the other two components remain mostly invariant. They therefore propose an Error-Entropy Scaling Law as a more fundamental description of model behavior. The paper argues that this new law explains why the classical cross-entropy scaling law appears valid for small models but breaks down for large ones—because Error-Entropy dominates at small scales but shrinks in relative importance as models grow. The authors conclude that Error-Entropy may provide a more accurate and theoretically meaningful measure for understanding and predicting large model performance.", "questions": "1. Error-Entropy is defined using rank statistics, which are inherently\nnon-differentiable. This raises an essential question:\nCan Error-Entropy (or a differentiable surrogate) be used as a\npractical training objective or evaluation metric?  If not, what is\nits functional utility beyond post-hoc analysis? The paper hints (p.9) that Error-Entropy could inspire new loss\nfunctions, but provides no method for computing gradients or\napproximations.  Bridging this gap between theoretical definition and\npractical optimization would determine whether this concept can\nactually influence model development.\n\n\n\n2. The paper’s key claim is that Error-Entropy—the Shannon entropy of\nthe rank-based error (RBE) distribution—is the true quantity that\nscales.  However, is Error-Entropy revealing a genuinely new\ntheoretical property of learning dynamics, or is it simply a\nre-expression of cross-entropy under a rank-based transformation? If Error-Entropy’s scaling behavior arises purely from the algebraic\ndecomposition of cross-entropy, it may not have independent\nexplanatory power.  Clarifying whether this metric corresponds to a\ndeeper principle (e.g., a conserved information quantity or\nenergy-like invariant) is crucial for establishing its theoretical\nlegitimacy.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper’s most significant strength is its conceptual novelty.  By\ndecomposing cross-entropy into Error-Entropy, Self-Alignment, and\nConfidence, it provides a new interpretive framework for understanding\nwhat cross-entropy actually measures in language model training. This\nmove reframes the long-standing “scaling law” discussion away from\nraw loss values toward a more structured, information-theoretic\nunderstanding of model behavior.\n\n\n\n2. The work offers a coherent explanation for an empirical puzzle that\nhas troubled practitioners: why the cross-entropy scaling law fits\nwell for small models but fails at very large scales.  By showing that\nonly Error-Entropy continues to follow a power law, the paper proposes\na simple and intuitive reason — as models grow, non-scaling\ncomponents (Self-Alignment and Confidence) dominate.  This insight\ngives practical and theoretical clarity to the field’s observed\nirregularities.\n\n\n\n3. The decomposition is mathematically clean and transparent.  Starting\nfrom the standard definition of cross-entropy, the authors derive the\ndecomposition step by step (Eq. 5–8, p.5), connecting it to\nmeasurable quantities like rank distributions and score norms.  This\nclear derivation enhances reproducibility and provides a framework\nthat can be extended or critiqued rigorously by others.\n\n\n\n4. The paper demonstrates solid empirical design: it tests the hypothesis\nacross 32 language models (Pythia, GPT-2, Qwen, LLaMA, OPT, Mistral,\netc.) and three distinct corpora (Wikipedia, C4, GitHub) over five\norders of magnitude in scale.  Although limited in token sampling,\nthis breadth of architectures strengthens the empirical robustness and\nhelps generalize the scaling claim within the text domain.\n\n\n5. Finally, the findings have broad implications.  Practically,\nrecognizing that only Error-Entropy scales could inform training\nobjectives and compute allocation for large models.  Theoretically,\nthe decomposition opens a potential bridge between rank-based\nevaluation and information theory, offering a new direction for\nresearch on learning dynamics and interpretability in large-scale\nneural systems.", "weaknesses": "1. Although the decomposition is mathematically valid (Eq. 8, p. 5), it\nis fundamentally algebraic rather than theoretical.  The decomposition\nintroduces new quantities (RBE), but this does not derive them from a\nprincipled model of language or probability theory. In other words, “Error-Entropy” is justified empirically, not from first principles\n— there is no information-theoretic or Bayesian derivation explaining\nwhy the entropy of rank distribution should obey a power law. Consequently, the argument is more descriptive than explanatory: it simply\nre-expresses cross-entropy in new coordinates rather than uncovering\ncausal structure. In short: the authors show that “error-entropy scales,” but not why\nit should do so.\n\n\n2. The central variable, Rank-Based Error (RBE), depends on sorting model\noutputs by rank (p. 4 – 5).  While this makes intuitive sense (rank as semantic accuracy), it has drawbacks.  Primarily,  ranking operations are non-differentiable. Hence, RBE and its entropy cannot directly serve as a training objective; the authors\nmention this only implicitly. Rank distributions depend on vocabulary size.  This can distort entropy values when models differ in\ntokenization or vocabulary. Thus, “error-entropy” may not generalize well across architectures\nor token sets.\n\n\n\n3. The paper shows strong empirical correlations (R^2 being 0.97 ),\nbut: it does not test alternative decompositions — e.g., whether\nother monotonic functions of cross-entropy could produce similar\nscaling. It provides no ablation or causal test: does directly minimizing\nerror-entropy improve model scaling?  (They hint at this as “future\nwork.”)\n\n\n4. Despite spanning “five orders of magnitude,” the experiments rely\nmostly on open-weight transformer families (Pythia, Qwen, GPT-2,\nLLaMA, etc.) trained on text-only datasets (Wikipedia, C4, GitHub;\np. 7 – 8). All datasets share similar linguistic distributions.  The sample size\nper dataset is small (≈ 1 000 tokens per model; App. A.2 p. 12),\nmeaning statistical noise could bias scaling exponents. Hence, the “Error-Entropy Scaling Law” may not generalize beyond\ntheir curated setup.\n\n\n\n5. In Table 1 and 2 (p. 7 – 8), the differences in R^2 and |\\delta| between\ncross-entropy and error-entropy are numerically modest (e.g., 0.973 →\n0.975).  Yet the paper interprets these small gains as evidence of a\nfundamentally new law.  Given log–log regression’s sensitivity to\noutliers and limited sample size, the improvement could easily arise\nfrom noise or fitting artefacts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)