Hidden Breakthroughs in Language Model Training
Authors: Sara Kangaslahti
Elan Rosenfeld
Naomi Saphra
Venue: ICLR 2026 Poster (2026)
Keywords: interpretability techniques
loss disaggregation
phase transitions
Primary area: interpretability and explainable AI
TL;DR: We decompose changes in loss along an arbitrary basis of the low rank training subspace to find breakthroughs obscured in the aggregate loss

Abstract: Loss curves are smooth during most of model training, so visible discontinuities stand out as possible conceptual breakthroughs. Studying these breakthroughs enables a deeper understanding of learning dynamics, but only when they are properly identified. This paper argues that similar breakthroughs occur frequently throughout training but they are obscured by a loss metric that collapses all variation into a single scalar. To find these hidden transitions, we introduce POLCA, a method for decomposing changes in loss along arbitrary bases of the low-rank training subspace. We use our method to identify clusters of samples that share similar changes in loss during training, disaggregating the overall loss into that of smaller groups of conceptually similar data. We validate our method on synthetic arithmetic and natural language tasks, showing that POLCA recovers clusters that represent interpretable breakthroughs in the model's capabilities. We demonstrate the promise of these hidden phase transitions as a tool for unsupervised interpretability.

Published as a conference paper at ICLR 2026
HIDDEN BREAKTHROUGHS IN LANGUAGE MODEL TRAINING
Sara Kangaslahti Harvard University
Elan Rosenfeld Google Research elanr@google.. com
Naomi Saphra Harvard University
sarakangaslahtiiee haw.aiid...aaaa..eeeee..
nsaphraēfas harvard.
ABSTRACT
Loss curves are smooth during most of model training, so visible discontinuities stand out possible conceptual breakthroughs. These breakthroughs enable a deeper understanding of the model's concept structure, but only when they are properly identified. This paper argues that similar breakthroughs occur frequently throughout training, but they are obscured by loss metric that collapses all vari- ation into a single scalar. To find these hidden transitions, we introduce POLCA, a method for decomposing changes in loss along arbitrary bases of the low-rank train- ing subspace. We use our method to identify clusters of samples that share similar changes in loss during training, disaggregating the overall loss into that of smaller groups of conceptually similar data. We validate our method on synthetic arithmetic and English language modeling, showing that POLCA recovers clusters that rep- resent interpretable breakthroughs in the model's 's capabilities. We demonstrate the promise of these hidden breakthroughs tool for unsupervised interpretability.
INTRODUCTION
As large language models train, various internal structures develop during abrupt breakthroughs. These sudden drops in loss reveal the formation of mechanisms for in-context learning (Olsson et al., 2022b), natural language grammar (Chen et al., 2024a). hierarchical generalization (Murty et al., 2023), and many other concepts (McGrath et al., 2022; Lovering et al., 2022: Power et al., 2022; Abbe et al., 2021). However, the loss curve as whole remains stubbornly smooth. As result, these momentary conceptual breakthroughs are treated as isolated curiosities, while the majority of training behavior is considered predictable.
These breakthroughs often colloquially termed phase transitions (Olsson et al., 2022a; Chen et al., 2024a; Murty et al., 2023)-are extremely consequential for our understanding of neural networks. Phase transitions represent critical periods of learning, so they offer key insights for training and optimization. For instance, introducing noisy data changing the optimizer during a phase transition can significantly reduce the downstream performance of a model (Achille et al., 2017: Chen et al., 2024a). Their timing used as evidence the role of specific mechanisms in model capabilitiess (Zhong et al., 2023; Olsson et al., 2022b; Chen et al., 2024a). If we could find more of these momentary training events, it could expand our understanding even further.
Prior work identifies breakthroughs through a top-down approach by measuring the training dynamics of predefined concept or skill and searching for sudden changes. We instead propose bottom-up unsupervised method for finding breakthroughs by grouping data points that have similar training behavior. This data-centric approach can be used inform optimization choices such as data selection or learning rate scheduling. Like other bottom-up interpretability methods such as SAFs, PCA, and transcoders, our method seeks concepts that are used naturally by the model, rather than imposing an assumed structure onto learning and representation.
This work shows that in fact, a model undergoes many breakthroughs during training, but most are concealed when averaging all data into single loss curve. Instead of averaging, we divide up the loss curve in two different ways to find hidden breakthroughs. First, we disaggregate the aggregate loss into losses on individual examples. By clustering the individual loss curves, identify subsets data that experience synchronized changes loss, implying that they rely on the same conceptual breakthrough. However, any individual example might benefit from multiple

---

Published as a conference paper at ICLR 2026
Avcrago
0.1 b2
by
Figure 1: A smooth loss function may change sharply for particular direction or data subset. POLCA works by decomposing and disaggregating the loss to discover these sharp changes. Left: Loss [a;;; 0) changes as the parameter setting ( moves in low-rank training subspace. The loss is sigmoidal on each axis, with differently timed inflections along basis vectors b and b2. These breakthroughs disappear in the smooth sum of the sigmoids which represents the exact loss. Right: The average of loss along basis vectors b and by-elides individual breakthroughs.
sigmoidal functions-includee The more differently-times breakthroughs underlie the loss, the more hidden each breakthrough is.
breakthroughs; such an example may undergo changes synchronized with different data subsets at different times. Furthermore, distinct concepts might appear simultaneously. erroncously merging their data clusters, To recover the underlying learned concepts, we may have to identify multiple separate breakthroughs which affect the loss eurve of single example.
To disentangle these effects for single sample, we separate the optimization space into specific gradient directions. When the loss changes during training. it is the result of movement across all parameters in high-dimensional space. We decompose this loss change from exact trajectory in the full-rank parameter space into a collection of movements along each dimension. By analyzing these loss curves along specific basis vectors, we identify conceptual breakthroughs that rely particular directions movement. The latter analysis permits further granularity in clustering data, as final performance an individual example may rely on multiple conceptual breakthroughs, each corresponding to a particular linear direction in training. In summary:
We introduce modified form of Loss Change Allocation (Lan al., 2020) called Projection Oriented Loss Change Allocation (POLCA) to measure changes in loss due to parameter adjustments in arbitrary directions during training (Section 3.2).
We show that some learned concepts can be identified by clustering exact loss, while others cannot (Section 4.2). Using POLCA, we extend our cluster analysis to identify these hidden conceptual breakthroughs obscured in the exact loss curves. We automatically identify specific concepts learned during breakthroughs in both synthetic (Section 4) and natural language settings (Section 5).
BACKGROUND: HOW MUCH CAN WE LEARN FROM LEARNING DYNAMICS?
Various loss breakthroughs have been interpreted as learning specific concepts. But why expect additional interpretable breakthroughs underlie periods of undifferentiated, gradual model improve- ment? Our approach is justified by the nature of the loss surface's 's complexity, illustrated by Figure 1, in which smooth curve emerges by eliding breakthroughs from each dimension.
2

---

Published as a conference paper at ICLR 2026
Why expect multiple breakthroughs? A very early phase transition is to be expected early in training after a brief memorization stage (Shwartz-Ziv & Tishby, 2017). In this sense, the most celebrated breakthroughs- those signaling the formation of induction heads arithmetic grokking are not the first drops in their training curves. Some breakthroughs even depend on earlier breakthroughs, as observed in synthetic tasks (Abbe et al., 2021) and in grammar acquisition (Chen et al., 2024a). If one concept depends on another, each must appear at a different timestep, requiring multiple breakthroughs. Furthermore, as shown by Saxe et al. (2019), summing many phase transitions can result in smooth curve, supporting our hypothesis that these breakthroughs can appear in stable regions of the loss curve.
Multiple breakthroughs can also come from differences in gradient scale along different directions. Ma et al. (2022) even attributed the early adge-of-stabbilit phase transition (Jastrzehski et al., 2020; Cohen et al., 2022) to multiscale structure of the loss surface and, furthermore, noted that this multiscale structure emerges at the range where models become singular: their loss lacks a quadratic approximation in terms of model parameters, creating conditions for breakthroughs under Singular Learning Theory (Watanabe, 2010; Wei et al., 2020; Wang et al., 2024). They argued that this structure is the product of both nonuniform data and nonconvex objectives, respectively justifying the disaggregation and decomposition which we apply to interpret training dynamics.
Why disaggregate the aggregate loss? We track learning on training datapoints and subpopulations, rather than the whole training set, because relevant skills can be acquired at different rates (Arora & Goyal, 2023; Chen et al., 2024b). Individual samples thus exhibit changes in loss out of line with the monotonic average trend (Xia et al., 2023; Rosenfeld & Risteski, 2024). In full-batch gradient descent, Cohen et al. (2022) identified non-monotonicity arising from oscillation about the maximum Hessian cigenvector. Rosenfeld & Risteski (2024) demonstrated that these oscillations occur across different axes for different samples and identified the primary cause: surprisingly human-interpretable semantic features. Even when the loss seems stable, performance can oscillate on edge cases until the model develops relevant capabilities (Qin et al., 2024; Bhaskar et al., 2024). We hypothesize that oscillation represents competing skills that are relevant to different subsets of data. To test this hypothesis, and to interpret the meaning of these directions, we disaggregate the loss into clusters with similar dynamics.
Why decompose the exact loss? Michaud et al. (2024) analyzed the scaling behavior of models with respect to individual tokens and identified a limitation of token-wise analysis of breakthroughs, which they called polygenic scaling exhibit breakthroughs at multiple scales. Our POLCA decomposition directly addresses this limitation by decomposing the loss of each token along multiple basis vectors. If we assume that a specific skill is enabled by movement along that skill's basis vector, then the loss change attributed to movement along that vector will accelerate at the moment the skill is acquired, for every sample that requires that skill. In this manner, the sample transitions from early to late dynamics through basis-specific loss breakthrough. In other words, by monitoring changes in directions corresponding to specific skills, we support the speculation of Nanda et al. (2023) that "phase transitions are everywhere."
samples which combine multiple skills and therefore
Why is linear decomposition sufficient? In practice, a conceptual breakthrough might not occur in single direction that persists throughout training. However, there abundant evidence that linear bases of the low-rank (Gur-Ari et al., 2018) training subspace are conceptually meaningful. In the late stages of training, loss is convex on the line connecting pair of checkpoints (Frankle et al., 2020) if those checkpoints express similar capabilities (Juneja et al., 2023) and mechanisms (Lubana et al., 2023). If a pair of high-dimensional models lack this linear connection, they still connect nonlinearly (Draxler et al., 2019); however, while parameter settings sampled from their linear connections improve broadly on the capabilities of the original models, those sampled from their nonlinear connections are less robust than the originals (Juneja et al., 2023, ref Appendix G). These observations suggest that linear decomposition should preserve meaningful conceptual features on the loss surface, and experiments show that the resulting directions are interpretable in practice.
METHODS
The key to our approach is the separate consideration of each example's datapoint loss changes throughout training. We contrast this individualized metric with aggregated loss across entire
3

---

Published as a conference paper at ICLR 2026
dataset. Using the datapoint loss, cluster individual example the basis of its loss L(x; fe), change in loss [zx; 0,) L(x; 01-1), or magnitude of change [L 01) - [ax; ft-1) during training.
We next decompose the loss itself into specific directions in the weight space, motivated by several considerations: First, while we have moved from an aggregated loss metric to a more granular datapoint loss metric, we are still only considering breakthroughs that are general enough to be perceived in loss curves. Second, an individual datapoint may benefit from a variety of conceptual breakthroughs, but will not be clustered on the breakthroughs individually. Finally, once we have identified a subset of the data as benefiting from particular conceptual breakthrough, decomposing into individual weight directions allows to locate where in the weights the breakthrough occurs and to thereby identify the mechanism involved.
When we break the exact loss curve down by directional movement during training, the resulting decomposed loss will reveal breakthroughs that are specific to given direction. Our procedure follows three steps: (1) select a basis, (2) decompose the loss along that basis to highlight particular learning events, (3) cluster datapoints according to their shared learning events.
3.1 FINDING THE BASIS
Algorithm Finding the decomposed optimization hasis
input: Training set X, Model checkpoints (0,)T=1- Bu < E Rdx0 for t = 1=....7 T do
I1 - 1- (BL
H <- V22(X, 0).
Define B+ E RRdx k
as the top & eigenvectors of II H (e.g., via the Lanczos method).
B, T B+].
[B,-1,
end for return BT
To decompose the loss, first require interpretable orthogonal basis. We efficiently compute the cigenvectors of the Hessian matrix using CoLA (Potapczynski et al., 2023) to construct a restricted training subspace. We expect this basis to be interpretable because each basis vector captures large gradient covariance and therefore represents a potential decision boundary. We select this basis for interpretability, but approach can use an arbitrary choice of basis for example, one which targets a particular use case or efficiently leverages optimizer preconditioner values.
The basis is constructed as shown in Algorithm 1. Given T intermediate training checkpoints and a number l of eigenvectors to compute at each checkpoint, we seek a low rank Tk-dimensional subspace which captures most of the movement during optimization (Gur-Ari al.. 2018). We construct this basis iteratively, starting with Bo = 0:: at each checkpoint f, we take checkpoint weights O, E Id and project their loss Hessian onto the nullspace of B E (t-)(( From the resulting projection, we append the top ls eigenvectors B. We compute the eigenvectors using Hessian-vector products Golmant et al. (2018) to avoid explicitly constructing the full Hessian matrix. The resulting basis is designed include directions of highest curvature at each checkpoint so that will capture synchronized loss behavior throughout training. Note that the very top cigenvectors are likely to reflect local oscillation, rather than conceptually meaningful long-term movement (Song et al., 2024), but as we continue to add to the low rank basis, we include more directions of long-term stable movement. We discard the oscillatory directions which do not provide an overall decrease in loss over the course of training according to POLCA by removing the directions with an increase in the mean projected loss from checkpoint to T. In this manner, we first construct a basis based on local information, then filter out directions that do not represent long-tew term movement. This construction finds local high-curvature directions that may be important for breakthroughs in the intermediate stages of training while ensuring that the basis does not overfit to local oscillations.
4

---

Published as a conference paper at ICLR 2026
3.2 DECOMPOSING THE LOSS WITH POLCA
To decompose the loss along our basis, we propose a modified version of Loss Change Allocation (LCA) (Lan al., 2020) which we call Projection-Oriented Loss Change Allocation (POLCA). LCA is a tool for analyzing changes in aggregated loss on dataset X between two checkpoints. The output of LCA the empirical loss change between pair of checkpoints which can be attributed to the motion of each individual weight unit. Given two consecutive checkpoints with parameters 6, and 0.41, LCA reformulates the change in loss as its first-order Taylor approximation, sum of the loss changes attributed to the movement of each individual model parameter (();
L(X;8 (( = Σ (V)L(X;0)) ( (0/2) - 0 Σ LCA(X;00) (I)
3=0
The POLCA decomposition differs from LCA in three key ways. First, do not restrict each direction to correspond to single unit (55), instead permitting an arbitrary orthonormal basis vector b E BT to replace the axis-aligned basis vectors in LCA; we project onto this basis vector using the dot product (b, -). Second, we are interested in changes in the loss each individual example x E X, not the entire dataset X. These first two modifications provide the first-order POLCA decomposition:
L(X;0(+1) 0(:;8.)
>' (2; (x;
FEX
Σ (b, VoL(2; 0141 Oi)
DE Br
The third key difference is that we use a second-order approximation because this basis is constructed explicitly from the Hessian eigenvectors. To understand why this choice of basis warrants second- order approximation, recall that each basis vector b is an eigenvector of the Hessian matrix Hrr (X) at some timestep t', where b is chosen because has the largest eigenvalue Ay (X, b) over the whole dataset. If we assume that the top eigenvectors of the aggregate Hessian maintain high curvature at other points in training and on individual datapoints, then the scaling factor in the second-order Taylor term will be very large even at the datapoint level. Limiting the approximation to only the first order term gives poor guarantees on error, as the second-order term could be expected to dominate. Although empirically the difference between the first and second-order values is small (see Appendix I), we nonetheless guarantee a better estimate due to lower Lagrange error bounds by computing the second-order approximation below.
Exact computation of the second- order term would be intractable, requiring computation of the top cigenvalues/vectors for each individual datapoint æ. Instead,
approximate it by substituting the true eigenvalue, denoted >x X, b) == bT (X)b,, with the curvature of the individual loss in the direction b, i.e. {x( = b HH (ar) )b. If the aggregate Hessian eigenvector b is close to the span of the top cigenvectors of the datapoint-specinf
Hessian for x, this provides a reasonable estimate while reducing calculation to a single Hessian-vest vector product per eigenvector. We therefore approximate the basis projection of the datapoint Hessian h(æ, b, 0e) as derived in Appendix C.
h(x, b, b,),) = de(x,, b)
(fe41 6),
(3)
2
Xx
(L(x; - L(x; 02), (0c+1 b), x (I(X;00+1) -xX
(4)
2
L(X;0,),))
7,(x,,0))
(5)
Equipped with this second-order approximation of the datapoint Hessian's projection onto our basis, we account for the high curvature and possible domination by the higher order term by modifying Equation 2 into the second-order Taylor expansion using the approximation from Equation 5. We can compute this second-ord -rder term with limited additional computational complexity by keeping track of the eigenvalues for each Hessian eigenvector and the aggregate gradient at each checkpoint.
L(X;0(+1) 0( R Σ
Σ
(b, VoL(x; 0.)) (b, 0 02) + h((x b, 0c)
(6)
TEX be By
Σ
POLCA(I,b:0.)
(7)
X By
5

---

Published as a conference paper at ICLR 2026
3.3 CLUSTERING THE LOSS
POLCA, above, provides curves that show how a decomposed loss changes with respect to each training example. We assume that if several examples show similarly timed loss changes in the same direction, they likely rely on the same conceptual breakthroughs or learning events; therefore, they are likely share a required skill, or specific capability needed for given task. We cluster POLCA training histories to recover these skill groups. For each datapoint a, we compute the total cumulative change in loss along each basis vector by summing over the previous POLCA values. We denote this sum the projected loss [1(2, 0t).
L, 01) = Σ POLCA(2,,;
(8)
We obtain Id projected loss trajectories for breakthrough clustering by computing (o 8t) at every time t. We cluster trajectories using Hierarchical Density-Band -Baeed Spatial Clustering of Applications with Noise (HDBSCAN) Campello et al. (2013) because it distinguishes cluster outliers and discovers clusters with variable density (i.e., similarly shaped curves that lay far apart in their metries). We cluster the trajectories for each basis vector separately to ensure that the clustering capture multiple skills per token.
3.4 DEFINING AND IDENTIFYING HIDDEN BREAKTHROUGHS
We POLCA to recover hidden breakthroughs in training. so must quantify whether clustered trajectories are distinguished by breakthroughs. We use the formulation defined by Chen et al. (2024b) to compute the start breakthrough in a given function for given datapoint x and basis vector b:
break (f, x, A) = arg max [f t A) f(x, t)] - [f 1) f(x, A)
Here, break x, A) approximates the maximum point of acceleration of x in f. f can be either the projected loss L, for given basis vector b or the exact loss L. We define hidden breakthrough as breakthrough that occurs in the flat region of the exact loss curve. That is, if we set a threshold r beyond which the exact loss curve is flat, then given set X' C X has a hidden breakthrough in metric f if the expected value of the start of breakthroughs in X' is greater than T:
hidden(f, (f,', = ExEX' arg (xx A) f(z t)] (f(x,t) f(x,t -(x,tt 2) >5}
(10)
ARITHMETIC LANGUAGE MODELING
We validate our POLCA clustering method in synthetic setting using an arithmetic addition task. Our clusters reflect categorical concepts within the data, even when those concepts are not discoverable by clustering directly on loss curves. Specifically, if we cluster exact loss curves, we recover digit dositions-bus if we instead cluster on POLCA curves, we also recover the skill of "carrying" a digit.
4.1 EXPERIMENTS
Data Our synthetic experiments use data from the arithmetic addition setting in Chen et al. (2024b), where the model is trained to compute the sum of two 3-digit numbers. This setting has 4 skills corresponding to each of the digits the output sum. Note that the digit in the 1000s place is always <0> or <1> token since the two input summants are 3 digits long. As shown in Appendix Figure 6 and Chen et al. (2024b), the skills corresponding to the digits have different loss curves, so we will casily recover the digit skill categories by clustering exact loss curves.
Unlike our source material, we also consider an additional skill: arithmetic carries to the output token (Figure 2). This skill corresponds to the case where instead of simply adding the two tokens at the corresponding digit in the input, the model must also carry a from the previous digit. Digit-specific addition skills lead clearly distinguishable exact loss curves, whereas carrying skills do not (Appendix Figure 7) but carries will become clear on our decomposed gradient basis. We provide additional skill and labeling details in Appendix E.

---

Published as a conference paper at ICLR 2026
Digit diil
9
0
7
Full-rank loss
1000
1004
+
5
2
6
Output
1
4
3
3
Carrsing skill
token
Carr
No
Digit
skill
1000s
100s
10s
36
Iteration
2000
4000
Cluster
Carry skill
(a) Mcan exact loss trajectories per (b) Arithmetic skill composition of cluster.
the clusters.
Figure 2: Diagram of arith- metic addition task. An exam- ple of 3-digit addition, labeled with the skills required for each of the output tokens.
Figure 3: Exact loss trajectory clustering on the arithmetic task. We use HDBSCAN to cluster the exact loss trajectories. This ap- proach, unlike our POLCA clustering method, fails to recover clus- ters associated with the carrying skill (the maximum fraction of carries is 0.51).
Setup details We train 3-layer (9 million parameter) Transformer model with embedding dimension 512, attention heads per layer, and an MLP dimension of 2048, following prior work (Olsson et al., 2022a). We study validation with 1250 data points and 5000 output tokens throughout training. We compute the loss and POLCA values for each token at each interval of 20 training steps. The POLCA basis uses the eigenvectors the Hessian, estimated using 1250 data point sample of the training set as detailed in Algorithm 1. We compute one new basis vector every 200 training steps, for total of 50 basis vectors. We provide further ablations on the decomposition strategy and choice of POLCA basis in Appendices G and H. We train the model for 10000 steps, but trim the x-axis of the plots 4000 to better display the breakthroughs.
Clustering As described in Section 3.1, some of the top Hessian directions may represent directions of oscillation (and not learning) during training. To ensure that we investigating directions where the model learning on average, we only consider the basis vectors for which the mean projected loss decreases. Then for each remaining basis vector, remove all of the tokens for which the decomposed loss increases, therefore retaining only tokens which rely on the vector during training. This removes 2360.8 out of 5000 tokens to learning many examples. We use HDBSCAN to cluster the remaining tokens, discarding the tokens marks as outliers. Through this process, we find subpopulations of the data that have similar projected loss trajectories.
that any given direction is irrelevant
4.2 RESULTS
Clusters with hidden breakthroughs
Decomposition strategy
Maximum carry homogeneity
Comparison to the exact loss In our cluster- ing experiments on arithmetic addition skills, we first consider whether decomposition is neces- sary for identifying conceptual skills. As base- line, we therefore cluster tokens solely their exact loss curves, rather than their decomposed loss curves. According to the exact loss clus- tering results in Figure 3, we can recover- to substantial degree- the digit skill by clustering only on the exact loss, likely because the digits have very different loss trajectories. However, as shown in Figure 3 and Table 1, we cannot
Exact loss
0.514
0.0
Chunge in exact loss LCA (Lan et al., 2020)
0.524
0.0
0.792
0.019
POLCA
0.973
0.355
Table 1: Cluster quality comparison. We com- pute the maximum fraction of points within all clus- ters that contain a carry for the specified digit and the fraction of clusters with hidden breakthroughs past the plateau in the exact loss at = 1000. For details and other metries, see Appendix G.
recover clusters that are homogenous with respect to the carrying skill from the exact loss alone.
Recovering concepts with POLCA clustering Because exact loss clustering failed to recover the carrying skill, will recover it with different clustering method. Instead of exact loss, will cluster on each basis vector's 's projected loss using the POLCA decomposition. The projected loss value T. (2, 0.) (Equation 8) represents the cumulative loss change of x attributed to movement
7

---

Published as a conference paper at ICLR 2026
Classir
Digit skill
11848
Clestor
Projected
Full 1
200
skill
Carry
cauy
2000
4000
2000 Iteration
1000
Iteration
Cluster
(a) Median projected loss of basis vector #1'S 's POLCA clusters.
(b) Median exact loss basis vec- tor #1 's POLCA clusters.
(c) Arithmetic skill composition of basis vector #1 POLCA clusters.
Digitakill skill
0-
TE0M
Projected
Full-rand rank
Chuaze
Carsing skill
casy
1500 Iteration
Iteration
Cluster
(d) Median projected loss basis vector #2's POLCA clusters.
(e) Median exact loss basis vec- tor #2's POLCA clusters.
(f) Arithmetic skill composition of basis vector #2's POLCA clusters.
Figure 4: Arithmetic data clusters with POLCA. We perform POLCA clustering the top basis vectors, and report the cluster medoid and quartiles (leff). median exact loss (center), and cluster skill composition (right) for each basis vector in order. Vertical lines mark the timestep when the relevant basis vector was sampled; note that vector's breakthroughs are not directly associated with this timestep. We find that the first basis vector recovers the digit skill whereas the second basis vector recovers the carrying skill (cluster #1 has homogeneity 0.90). The clusters computed from the POLCA trajectories show changes in the decomposed loss that are obscured in the exact loss curves.
along basis vector By clustering the projected loss trajectories, we find that the top basis vectors produce two easily-described and hea--homogeneous clusters: one which contains examples of the 1000s place digit and one which contains examples of arithmetic carrying for all digits (Figure Appendix Figure 8). We therefore determine that POLCA clustering recovers subtler skills- like carrying-thir that are challenging to reconstruct from the exact loss or parameter-aligned LCAA curves alone (Table 1). In Appendix Table 9, we also use Equation 10 to compare the fraction of clusters with hidden breakthroughs past step 1000 (with = 100), where the mean exact loss platcaus (Appendix Figure 6), and find in Table 1 that POLCA discovers the highest fraction of clusters with hidden breakthroughs.
Figure shows POLCA clustering for the first two basis vectors. Along these basis vectors, certain data subpopulations show changes in the projected loss (Figures 4a and 4d) that do not occur as visibly in their smoother exact loss curves (Figures 4b and 4e). We conclude that arithmetic carries rely on breakthroughs along specific dimensions during training, but these breakthroughs may be elided in the exact loss curve.
ENGLISH LANGUAGE MODELING
We apply our approach to a real-world causal language modeling task and show that POLCA breakthrough clustering recovers interpretable conceptual skills in the natural language setting.
5.1 EXPERIMENTS
For the natural language modeling setting, we use the English Wikipedia dataset (Wikimedia Founda- tion, 2022) from March 2022 to train 3-layer (40m parameter) model. We use the same POLCA setup in the arithmetic addition setting (see Appendix E.2 for details). As in the arithmetic setting, we only consider directions where the projected aggregate loss overall decreases to filter out directions of oscillation. We also discard the datapoint-speccfie POLCA trajectories along the directions where
8

---

Published as a conference paper at ICLR 2026
Claster
Chaster Label
Contests closest to modold
intermediate cup twice, in 1942-43 and 1944-45 seasors, AntnAfter the war the
Projected loss
00-
seasors.
frome the ofter
the 19th century.
happened to live through the tense period of the struggle for indeperdence of Peru Froe
the first
0.1
clause
Voight), his rentor. While attempting to prevent files containing information all IMF field agents Fron
5000
10000
2885, and 2010. In 2000. wes naved National Blue Ribbon School of Excellence, Anin duplscation. progras was in 1998 and was ciscontirued on March 31, 2009. Anin
Treration
(i) Projected loss
2 Repeated newliie
Claster
Mission: Irpossible film series. He is portrayed by Ton Cruise. Antn
protagonist
10-
Full-rank All-rank loss
nost countries. conferences are those of India and thina, followed by the Philigpines,
largeat oplacopal
Comna ffer paranthatioa phrase
3
1888 by the Swidiah physiclat Johames Bydberg, theoretically Ntels Bohr in 1913,
klas Athletic F.C. a rootball club based Leagrave, in Luton,
10000
Iteration
(iii) Cluster data examples
(ii) Exact loss
(a) POLCA clusters from basis vector 13.
Claster
Contexts closest to medoid
Projected loss
R appear in chapter of Air Force Instruction 36-2406: Officer and Fallitte Fsaluation
0.0-
this was changed to the current NVEN on July Tin Noodran of Oz: Faithful
Apposi tive noun phrase
-0.1-
relative pathin
derivative Newton's notation'n diseresis, type discritic
5000
10000
year fantasy, an essay on the year's best fantasy books, and introductory notes to the
Iteration
Non-appositive phrase
Projected loss
Saiunkoku Monogat, llyper Agent,
preceded by comma
calculate wavelangths of hydrogon spectral serdes. Anvatustory 1858,
Chasser
Full-rank loss
10
1t actually gathers together places originally published during tearyear pariod, 1973 and 1374,
Repeated
Township of the village In Schooleraft Brady
Townshhi..Vni Aniin
(1964) and its sequel Breakin' 2: Electric Rongaloo (1984).An3s
5000
10000
Iteration
(iii) Cluster data examples
(ii) Exact loss
(b) POLCA clusters from basis vector 23.
Figure 5: Examples of English LM data clusters with POLCA. After clustering on POLCA trajectories for two illustrative basis vectors, we report their average decomposed POLCA trajectories (5a(i) and 5b(i). Figures 5a(ī)) and 5b(ii) show the average of the exact loss trajectories for each of the POLCA trajectory clusters. For each cluster, we provide a label based on the top POS tags and tokens in the cluster and the top 10 contexts closest its medoid. We report the 3 contexts closest to the cluster medoid and color the corresponding token. Clustering on the decomposed POLCA trajectories reveals low-rank breakthroughs at times when the full-rank exact loss remains smooth. See other examples in Appendix K.

---

Published as a conference paper at ICLR 2026
the token'ss 's decomposed loss increases, which removes average 6655.5 out of 12600 tokens per direction. After clustering the remaining token trajectories, we discard HIDBSCAN-marked outliers.
Automatic labeling. To analyze the concepts represented by each cluster, we look for syntactic and lexical patterns shared by the cluster data.' ! Our automatic labels are based on the target token and its preceding trigram. To obtain automatic labels for a cluster, we compute the frequency of each POS tag (tagged by spacy) in it. We then automatically label each cluster with the smallest set of POS tags required to compose 70% of the cluster's 4-gram samples. For example, if over 70% of the token instances in a cluster are preceded by a comma s,>, the cluster would be automatically labeled as 1 token after PUNCT. We consider basis vectors with at least one cluster with simple oneel-one including most two POS tags (not counting <PAD> tokens). Starting with 30 basis vectors, we remove 4 because the average decomposed loss increases. We find 22 of the remaining basis vectors have at least one cluster with a simple label as defined. We refine these automatically assigned labels by examining the most frequent tokens in the cluster and the ten examples closest to the cluster medoid and manually selecting a label that is consistent with the automatic POS label and the ten closest examples to the medoid. See Appendix E.2 for further labeling details.
5.2 RESULTS
On language models, POLCA clustering again reveals hidden breakthroughs along each basis vector. We show a selection of clusters from specific basis vectors in Figure 5 and others in Appendix K (Figures 10 and 11). Clustering projected loss trajectories along each basis vector, we find groups that correspond to various grammatical constructions. Some show apparent breakthroughs in their projected loss, like the cluster corresponding to predicting < to> and from> after the first clause in sentence (Figure 5a(i)). We also observe clusters whose projected loss trajectories move in opposing directions along certain basis vectors. For instance, in Figure 5b, cluster contains appositive noun phrases and cluster 2 has syntactically similar- non-appositive-nouu phrases (such list items). These clusters visibly mirror each other's movement-thougee the clustered decreases in loss are generally larger than their opposing cluster's increases loss.
Figure 5 shows that despite their smooth exact loss curves, POLCA clusters have sudden changes in their decomposed loss curves at different points during training. Clusters from the exact loss curves, by contrast, do not reveal breakthroughs except very early in training (Appendix Figure 9). We conclude that POLCC reveals breakthroughs in the decomposed loss that are obscured in the exact loss. Through clustering, POLCA can explain how different skills are learned during training.
CONCLUSIONS
This work introduces POLCA clustering, a method identify learned skills from decomposed loss trajectories. POLCA decomposes the loss on two levels: individual datapoints and specific directions in the weight space. We use this decomposition to discover clusters that share breakthroughs obscured by loss metrics. In language modeling and synthetic settings, these clusters recover interpretable skills which appear to emerge at particular moments during training.
These are promising findings for meaningfully interpreting large models. By recovering breakthroughs in identifiable skills, we support the hypothesis that high-dimensional learning typically entails a series of breakthroughs at various scales. When breakthrough appears in training, it suggests naturally discrete category; the model either knows the concept or doesn't 't know it, with little middle ground. Humans think in categorical concepts, these breakthroughs are far more interpretable than the continuous data interpolations often assumed in learning theory.
II principle, our method can uncover skills that are not describable through these templates. but templating allows automatic labeling. Our templating approach is similar to the automated explainer tool N2G (Fonte al., 2023), popular ngram-based evaluation metric for unsupervised interpretability methods (Gao et al., 2024).
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We implement our models and experiments using open-source libraries and datasets. We provide detailed hyperparameters in Appendix D and thorough explanation of the experimental setup in Section E. Our code is available at https: h hots:::b/tttbb.oo/ com/ cwtp:l..coo/aaaaa//aa/.. wtt.///aaaaa///O.........
ETHICS STATEMENT
This work provides method for better understanding the training dynamics of language models, The trained models may contain biases from the training datasets.
ACKNOWLEDGEMENTS
This work was informed by helpful conversations with Nikhil Vyas, Nicholas Lourie, Mike Lepori, and Ekdeep Singh Lubana. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE 2140743, Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors(s) and do not necessarily reflect the views of the National Science Foundation. This work was enabled in part by gift from the Chan Zuckerberg Initiative Foundation establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
REFERENCES
Emmanuel Abbe, Enric Boix-Adsera, Matthew Brennan, Guy Bresler, and Dheeraj Nagaraj. The staircase property: How hierarchical structure guide deep learning, 2021.
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep neural networks. CoRR, abs/1771.08856, 2017. URL http: //axxiv org/
08856.
Sanjeev Arora and Anirudh Goyal. theory for emergence of complex skills in language models. arXiv preprint urXiv:2307.13956 2023.
Adithya Bhaskar, Dan Friedman, and Danqi Chen. The Heuristic Core: Understanding Subnetwork Generalization Pretrained Language Models, June 2024. URL http://arki...rr/bbb///// 03942. 240.22403 [es].
Ricardo J. G. B. Campello, Davoud Moulavi, and Joerg Sander. Density-based clustering based on hierarchical density estimates. In Jian Pei, Vincent S. Tseng. Longbing Cao, Hiroshi Motoda, and Guandong Xu (eds.), Advances in Knowledge Discovery and Data Mining, pp. 160-172, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. ISBN 978-3-642-37456-2.
Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L. Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms, 2024a.
Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36, 2024b.
Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability, 2022.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht. Essentially No Barriers in Neural Network Energy Landscape. 17X3vv1803.000 0088508855 [es, stat], February 2019. URL 00885. arXiv: 1803.00885.
http:///aaxxv..rr/aabba
Alex Foote, Neel Nanda, Fshen Kran, Ioannis Konstas, Shay Cohen, and Fazl Barez. Neuron to graph: Interpreting language model neurons at scale, 2023. URL https: / http//axivvvo//// www.////////aeeeee 19911.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3269. PMLR, 2020.
11

---

Published as a conference paper at ICLR 2026
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, llya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024. URL https:
wwwxiv.orggabss///. 040933 04093.
Noah Golmant, Zhewei Yao, Amir Gholami Gholami, Michael Mahoney, and Joseph Gonzalez. pytordh-hessian-iigentinngs efficient pytorch hessian eigendecompooition, October 2018. URL www./////////e http://hww...cooooooooooonn////t hatpr//ooonannttttttoooo-eeeeeee h
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace, 2018.
Stanistaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho and Krzysztof Geras*. The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations, 2020.
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connec- tivity reveals generalization strategies. In The Eleventh International Conference on Learning Representations, 2023, URL
met/forummid==NGMMM Jul
Janice Lan, Rosanne Liu, Hattie Zhou, and Jason Yosinski. Lea: Loss change allocation for neural network training, 2020.
Charles Lovering, Jessica Forde, George Konidaris, Ellie Pavlick, and Michael Littman. Evaluation beyond task performance: Analyzing concepts in alphazero in hex. Advances in Neural Information Processing Systems, 35:25992-26006, 2022.
Ekdeep Singh Lubana, Eric J. Bigelow, Robert P. Dick, David Krueger, and Hidenori Tanaka. Mechanistic mode connectivity, 2023. URL http:://rrxv.orr//bb///1111 /21.00222211..
Chao Ma, Daniel Kumin, Lei Wu, and Lexing Ying. Beyond the quadratic approximation: the multiscale structure of neural network loss landscapes. arXiv preprint 1331::224..13326, 2022.
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47):e2266655 19, 2022.
Eric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling, 2024.
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D Manning. Grokking of hierarchical structure in vanilla transformers. arXiv preprint arXiv:2305.18741, 2023.
Neel Nanda and Joseph Bloom. Transformerlens.
Lransf 2022.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023. URL https::/arxxv..rrg/bbb//
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac HatfieldDDodds Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022a. URL httpss///axx: .or.o/oog.......... org/abs/ abs/ 2209. 11895.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 11895, 2022b.
Andres Potapezynski, Mare Finzi, Geoff Pleiss, and Andrew Gordon Wilson. CoL.A: Exploiting Compositional Structure for Automatic and Efficient Numerical Linear Algebra. arXiv preprins arXiv:2309. 03060, 2023.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- cralization beyond overfitting on small algorithmic datasets. arXiv preprint 2201..220..02.17, 2022.
12

---

Published as a conference paper at ICLR 2026
Tian Qin, Naomi Saphra, and David Alvarez-Melis. Sometimes i am a tree: Data drives unstable hierarchical generalization, 2024. URL URLLnttp://aavvv..rr///////////
04619.
Elan Rosenfeld and Andrej Risteski. Outliers with opposing signals have an outsized effect on neural network optimization. In The Twelfth International Conference on Learning Representations, 2024.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(23): 11537-11546, 2019, doi: 10.
URL https: /wwp///wwwww/oooo/ pnas. wwwp:paaa..o//////aat/ooooo htttps:/www..nas.orrrr
10. 10..p///psss. 1820226116.
Ravid Shwartz-ZZiv and Naftali Tishby. Opening the black box deep neural networks via information, 2017. URL /trv:://rxxv htt.:/orgg/bb/////// 00810.
Minhak Song. Kwangjun Ahn, and Chulhee Yun. Does SGD really happen in tiny subspaces? In High-dimensionall Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024.
URL httptttts////eeeeiiiiiii wttp://wwww....ooooooooooottt//a
George Wang, Jesse Hoogland, Stan van Wingerden, Zach Furman, and Daniel Murfet. Differ- entiation and specialization of attention heads via the refined local learning coefficient. ArXiv, abs/2410.02984, 2024. URL https://apil semanticscholar semmanttccccoola...ro//orrosssss wwgp:ooppws..:: 273162605.
Sumio Watanabe. Asymptotic equivalence of bayes cross validation and widely applicable in- formation criterion in singular learning theory. ArXiv, abs/1004.2316, 2010. URL https:
a/manntsmmmtt ienraanticcchenn cscholar. wwwpcchlarr.rg/ooouuussssse. org/CorpusID:11 15093314.
Susan Wei, Daniel Murfet, Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas Quella. Deep learning is singular, and that's good. IEEE Transactions on Neural Networks and Learning Systems, 34:10433-10486, 2020. URL https: semanticscholar wwwphhhlar.rgg/oorrreoeee org/CorpusID: 225041126.
Wikimedia Foundation. Wikimedia downloads, 2022. URL
org.
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. Training trajectories of language models across scales, 2023.
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks, 2023. URL https: //arxiv org/ ab////bss/3....../// 17844.
13

---

Published as a conference paper at ICLR 2026
A LIMITATIONS AND FUTURE WORK
Our method of constructing basis is inspired by the existing literature training in restricted subspaces, but represents an obvious site of improvement. The top eigenvectors of the Hessian, like the axis-aligned basis, could represent many concepts in superposition. Therefore, some non- orthogonal basis might represent interpretable concepts more cleanly than our orthogonal basis, though it would no longer provide a low-rank decomposition. Furthermore, by local curvature and then filtered to favor directions of long-term movement; other bases may favor long-teri term movement by construction. In general, we consider the ideal basis be an open question.
basis constructed
Our experiments are limited to small models. The two main challenges with scaling POLCA are the Hessian basis computation and the frequency of checkpoints used to sample the POLCA trajectories. Scaling this work to larger models may require using basis that is less computationally expensive to compute than Hessian eigenvectors, but our results from Appendix indicate that this is likely possible with limited impact on the cluster quality. The small scale of models that use in our experiments allows for very high granularity of checkpoints used to compute both the basis and the POLCA trajectories. For larger models, this may be computationally infeasible and lower checkpoint frequency may be needed, resulting in less signal for clustering the POLCA trajectories.
Our current experiments are limited to language models. However, in principle our approach is model-agnostic and can be applied to any deep neural network. Applying POLCA to other modalities is an exciting direction for future work.
The labeling approach that we use in the natural language setting relies on POS tagging. This labeling strategy allows for unsupervised, automatic identification of these syntactic skills and ensures striet interpretable labels. However, it fails to capture many human-interpretable language modeling skills. The discarded vectors may (and likely do) contain other interpretable skill clusters that are not found by automated labeling.
B STATEMENT ON USE OF LARGE LANGUAGE MODELS
We used generative AI for debugging and minor grammar edits when writing. The authors made all significant contributions the research, analyses, and writing.
C DERIVATION OF APPROXIMATE SECOND-ORDER TERM
We can approximate the difference between the gradient at time f. and f ++ as
9x+1(X) (x) Hc(X)(00-1 0z) (X)+ (x ~ b HX()bbbbb 0,+1 0t)
(11)
(12)
(b) 01+1 0₂)
(13)
If assume also be an eigenvector of the datapoint Hessians (x), can apply similar argument for the gradient on the datapoint level.
(g)+1 (...)) (TH((2)6((bb 0,11 0.)
(14)
Note that the assumption above (of matching Hessian eigenvectors between data points and their aggregate) is unlikely to be correct. If this assumption violated, then the scaling factor in the following second-order Taylor term will be minuscule the datapoint level. In practice, have found that the second-order term has limited impact at the datapoint level (see Appendix D, but we nonetheless it to improve approximation. Then may approximate it as:
14

---

Published as a conference paper at ICLR 2026
(0i+1 (1) ()
6 Hz B1t1 01)
(15)
(0+41 (X) - ()),
Xx(X, 2c+1 - 01)
(x)) (((2).
(M,())) 0.+1 - 01)
(16)
(9++1(X) ()))
>e(X, 6) 00+1 - -
(0(1(()) ())).
XxXX, b)
(k((z).)) b)
(17)
(x))( - ()),
Empirical Validation We run a small-scale empirical study test the accuracy of this estimation. For sample of 400 tokens and the first 10 POLCA checkpoints and the top 5 basis vectors in the arithmetic setting, we compute the root mean squared error (RMSE) between the approximated second-order term (x (9/=1
( (xt (X),5) and the true second-order term (xx()); b) averaged across tokens, checkpoints, and basis vectors. We find that our approximation has an RMSE of 0.145
when compared to the ground truth second-order term, indicating that this approximation is close to real value.
D HYPERPARAMETERS
the tables below, we provide the hyperparameters used during training of the models the synthetic arithmetic and language modeling settings. We use NVIDIA H100 80GB HBM3 GPUs for experiments and run each training run on a single GPU.
We selected the clustering hyperparameters maximize empirical performance in the synthetic setting and used similar values relative to the number of tokens the natural language experiments. We chose the POLCA hyperparameters maximize the frequency of POLCA checkpoints and basis checkpoints within computational constraints.
Table 2: Hyperparameters for training the synthetic arithmetic model
Hyperparameter
Value
Number of Parameters
9475594
Steps
10000
Epochs
Batch size
64
Number of training tokens
2560000
Optimizer
AdamW
Learning rate
le-5
Weight decay
0.1
Betas
(0.9,0.95)
LR Schedule
min(i/100, 1.0)
Table 3: POLCA hyperparameters for the synthetic setting
Hyperparameter
Value
Basis checkpoint interval (steps)
200
T
50
k
1
POLCA checkpoint interval (steps)
5
15

---

Published as a conference paper at ICLR 2026
Table 4: Arithmetic clustering hyperparameters
Hyperparameter
Value
Clustering algorithm
HDBSCAN
Minimum cluster size
150
Minimum samples
Number of tokens / 15
Table 5: Hyperparameters for training the natural language model
Hyperparameter
Value
Number of Parameters
40274737
Steps
14000
Epochs
1
Batch size
Number of training tokens
114688000
Optimizer
AdamW
Learning rate
le-5
Weight decay
0.1
Betas LR Schedule
(0.9,
min(i/100 1.0)
Table 6: POLCA hyperparameters for the natural language setting
Hyperparameter
Value
Basis checkpoint interval (steps)
750
T
30
k
1
POLCA checkpoint interval (steps)
200
Table 7: Natural language clustering hyperparameters
Hyperparameter
Value
Clustering algorithm
HDBSCAN
Minimum cluster size
300
Minimum samples
Number of tokens 15
E EXPERIMENTAL DETAILS
E. ARITHMETIC SETTING
Setup. In the arithmetic experiments in Section 4, we train a 3-layer (9m parameter) Transformer model with embedding dimension 512, 4 attention heads per layer, and an MLP dimension of 2048 (Nanda & Bloom, 2022). For validation set with 1250 data points and 5000 output tokens, we compute the loss and POLCA values for each token at intervals of 20 steps throughout training. We compute POLCA basis using the eigenvectors of the Hessian estimated using 1250 data point sample of the training set. We compute vectors.
basis vector every 200 steps for total of 50 basis
Labeling details. We automatically label each token with the ground truth value of the digit and earry skills using the definition of these two skills. We label the digit skill based on the position of the
16

---

Published as a conference paper at ICLR 2026
token in the output: the first token is 1000s, the second token is 100s, the third is 10s, and the fourth Is. We label the carry skill by computing the sum up to the next lowest digit place and determining whether it resulted in carry to the current token. For instance, for output in the 10s place, if the two inputs in the Is place sum to 10 or higher, then will be labeled with "carry", otherwise will be labeled with "no carry".
E.2 NATURAL LANGUAGE SETTING
Setup. For the natural language experiments in Section 5, we train on the English Wikipedia dataset (Wikimedia Foundation, 2022) from March 2022. We use the same POLCA setup as in the arithmetic addition setting. We train 3-layer (40m parameter) transformer model with embedding dimension 512, attention heads, and an MLP dimension of 2048 (Nanda Bloom, 2022). We compute the loss and POLCA values for each token validation set of 12600 output tokens. We analyze intermediate checkpoints at intervals of 200 steps throughout training. We apply POLCA to the basis derived from the eigenvectors of the Hessian estimated using 1000 data point sample of the training set as detailed in Algorithm with k = 1. We compute a new basis vector every 750 steps.
Labeling details. We label each token and the three tokens before it using spacy for part-of-speech (POS) tagging. This produces sequence of four POS tags as the automatic POS label for each token. To label given cluster, compute the frequency (across the tokens in the cluster) of each POS tag each index. For each index in the sequence, we then automatically label the cluster with the smallest set of POS tags that index required to make up 70% of the cluster. We then filter out any labels that require more than POS tags to describe the cluster. To generate the labels reported in Section 5, manually refine the automatically generated label by looking the top 10 contexts closest to the medoid the cluster. Although these manually refined labels are challenging to verify automatically, we ensure that the contexts closest the medoid follow the manual label and that the manual label follows the automatic label generated using the POS tags.
F EXACT LOSS TRAJECTORIES FOR THE DIGIT AND CARRY SKILLS
Cluster
1000s
100s
Full-rank loss
10s
1s
0-
5000
10000
Iteration
Figure 6: Median and quartiles of the loss trajectories for each digit.
17

---

Published as a conference paper at ICLR 2026
Cluster
Cluster carry 100s
No carry 1000s
Carry 1000s
Carry 100s
Full-rank loss
Full-rank loss
2-
5000
10000
Iteration
Iteration
(a) 1000s place
(b) 100s place
Cluster No carry 10s
Cluster No carry
Cary
Full-rank loss
Full-rank loss
5000
10000
Iteration
Iteration
(c) place
place
Figure 7: Median and quartiles of the loss trajectories for each digit and carry combination.
G DECOMPOSITION STRATEGY COMPARISON
We investigate whether POLCA is required for decomposing the loss. We expand the results from Table by showing the top three homogeneity scores for the carrying skill and adding empirical Fisher information and first order POLCA results. To compute the homogeneity score for given basis vector, we take the maximum fraction of carries in any given cluster for that basis vector (or across all of the clusters for exact loss or change in exact loss). We then report the top three homogeneities across the full basis, as well as the recall and Fl score for the corresponding clusters. note that we exclude HDBSCAN outliers from recall and Fl computations. Importantly, we only consider clusters for which over 85% of the tokens with the carry skill correspond to the 10s or 100s place, since carrying to the 1000s place corresponds to simply predicting a or in the first position (see Figure for reference) and is recovered with high homogeneity for all trajectory types except exact loss.
We compare carry skill homogeneity across the following trajectory types:
Loss:
Exact loss trajectories.
Change in exact loss: We compute the change in exact loss by subtracting the loss checkpoint / from the loss at checkpoint / for each timestep / > 0 in the exact loss trajectory.
Fisher information: We approximate the empirical Fisher Information V 81 as Achille et al. (2017). For each basis vector b, the Fisher Information projected onto is (b, VoL(2, 2.)
18

---

Published as a conference paper at ICLR 2026
Loss Change Allocation (LCA): We compute the datapoint wise LCA trajectories (Equation 1) projected onto the parameters that have the top 50 highest magnitudes at the end of training.
First order POLCA: We calculate the POLCA trajectory without the second order term (Equation 2)
POOCC:: We compute the projected loss (Equation 7).
The results from Table 8 demonstrate that POLCA finds the most homogeneous clusters with respect to the carrying skill. Moreover, POLCA and first order POLCA have comparable scores. Note that many low homogeneity clusters can have high recall because they are large and thus contain most of the carry tokens while having low homogeneity.
Table 8: Carry skill homogeneity comparison. For each type of trajectory, we compute the fraction of points within each cluster that contain carry to the output token and report the homogeneity, recall, and FI score for the three clusters with highest homogeneity across all 50 vectors. POLCA recovers carry clusters with the highest homogeneity.
Cluster
Decomposition strategy
Number
Homogeneity
Recall
F1
Loss
0.514
0.771
0.617
Change in exact loss
0.524
0.958
0.678
0.664
0.947
0.781
Fisher information
0.643
0.740
0.688
3
0.637
0.874
0.737
0.792
0.772
0.782
Loss change allocation (LCA) (Lan al., 2020)
2
0,614
0.873
0.721
0.592
0.626
0.609
0.948
0.767
0.848
First order POLCA
0,928
0.769
0.841
0.887
0.751
0.813
0,973
0.736
0.838
POLCA
0.946
0.773
0.850
3
0.903
0.762
0.827
We also compare the fraction of clusters with hidden breakthroughs for each type of trajectory. To compute whether or not given cluster has a hidden breakthrough, we use Equation 10 with A = 100 identify breakthroughs past step r 1000 where the exact loss plateaus. We find that POLCA produces the highest fraction of clusters with hidden breakthroughs. We hypothesize that this is mostly due to the basis construction, since the Fisher information and first order POLCA (both of which are computed using the same basis as POLCA) produce the next highest fraction of clusters with hidden breakthroughs.
Table 9: Hidden breakthroughs comparison. For each type of trajectory, we use Equation to compute the fraction of clusters with hidden breakthrough past the plateau in the exact loss step 1000.
Decomposition strategy
Fraction of clusters with hidden breakthroughs
Loss
0.0
Change exact loss
0.0
Fisher information
0,284
Loss change allocation (LCA) (Lan et al., 2020)
0.019
First order POLCA
0.307
POLCA
0.355
19

---

Published as a conference paper at ICLR 2026
H POLCA BASIS COMPARISON
We test ablated bases to analyze the effect basis choice on the POLCA breakthrough clustering. To do so, we compute the maximum carry skill homogeneities over all of the clusters when performing POLCA breakthrough clustering. We use the following bases:
Random orthonormal: randomly sampled orthonormal vectors
Random shuffled Hessian: basis computed using Algorithm but randomly shuffling the model checkpoints
Top Hessian eigenvectors: basis computed using Algorithm
We find in Table 10 that these ablations result in only slightly lower quality clusters with respect to homogeneity (although random orthonormal vectors have lower recall than the other two bases on average), indicating that different bases can used for larger experiments to trade off between compute and interpretabilltty We also compute the fraction of clusters with hidden breakthroughs for each basis in Table 11 and find that the random orthonormal basis has a significantly lower fraction of hidden breakthroughs recovered than the other two approaches, indicating that this random orthonormal basis is not sufficient to find hidden breakthroughs late in training.
In addition to these bases, have tested variety of additional basis constructions, such stacked Jacobian, Hessian computed using a sliding window, and Hessian computed at the end of training, and chose the top Hessian eigenvectors since they had the best performance in the arithmetic setting.
Table 10: Carry skill homogeneity comparison. For each basis, we compute the fraction of points within each cluster that contains carry to the output token and report the homogeneity, recall, and F1 for the clusters with maximum homogeneity. Using the top Hessian eigenvectors recovers slightly more homogeneous carry clusters than the other basis selection strategies. The random orthonormal vectors have high homogeneity but lower recall than the other two bases.
Cluster
POLCA basis
Number
Homogeneity
Recall
0.902
0.655
0.759
Random orthonormal
2
0,858
0.533
0.658
3
0.838
0.586
0.689
0.856
0.699
0.769
Random shuffled Hessian
2
0,852
0.734
0.789
3
0.834
0.730
0.779
0.973
0.736
0.838
POLCA
0,946
0.773
0,850
0.903
0.762
0.827
Table 11: Hidden breakthroughs basis comparison. For each type of basis, we use Equation 10 compute the fraction of clusters with a hidden breakthrough past the plateau in the exact loss at step 1000.
POLCA basis
Fraction of clusters with hidden breakthroughs
Random orthonormal
0.031
Random shuffled Hessian
0.304
POLCA
0.355
20

---

Published as a conference paper at ICLR 2026
I SECOND VERSUS FIRST ORDER POLCA APPROXIMATION
Table 12: Empirical comparison of second and first order POLCA values. For the arithmetic setting, we compute the average cosine similarity and L.2 distance between the second (Eq 7) and first (Eq 2) order POLCA trajectory vectors. The first and second-orde approximations of the POLCA trajectories are very similar on average.
Cosine similarity
1.2 norm
5.4891 F-4
0.99987
J ADDITIONAL ARITHMETIC LANGUAGE MODELING CLUSTERS
Claster
skill
Full Josss io
Projected
200
Caryting deill
Z
Cary
Nocurry
2009
2000
0003
Iteration
Iteration
Cluster
(a) Median projected loss of basis vector #3's POLCA clusters.
(b) Median exact loss of basis vcc- tor #1 POLCA clusters.
(c) Arithmetic skill composition of basis vector #3's 23's POL.CA clusters.
Figure 8: Arithmetic data clusters with POLCA. We perform POLCA clustering on the third basis vector, and report the cluster medoid and quartiles (lefi), median exact loss (cenfer), and cluster skill composition (right). Vertical lines mark the timestep when the relevant basis vector was sampled; note that vector's breakthroughs are directly associated with this timestep. We find that the third basis vector recovers the carrying skill in the 1000s place.
K ADDITIONAL NATURAL LANGUAGE CLUSTERS
Chaster Label
Contexts elosest to ueddold
Canterbury, New ZoalandvnBiveers of Now
Zealand<PAD-er BD-.--ADDDPPPPPD------- (PP---//---PP---------------00 <PAD--NNDS-PAD
10-
Repested <PAD>
Full-ran rank loss
<P ADD PAD---AA
Tokens
<PARt-PPADD-
<PAD 400- www-e--A---PP----a--- Mtw.---AA--------PAP---
<PAD-<PES--PA
Sythet Conference
BruneetahrFclleessscccaa Province Bruneg and Garsan people togather for dialogue Anvedistoryinnn
0-
5000
Iteration
100000
Contiruinng
and educational settlement in Silesian
India and China. folloved by the Philippines, and Indonesia. AninList of Ronan Catholii dioceses
Exact loss
(ii) Cluster data examples
Figure 9: English language modeling data clusters with the exact loss. We cluster the exact loss trajectories and report the average loss by cluster (9i). For each cluster, provide label based the top POS tags of tokens in the cluster and the top 10 contexts closest to the cluster medoid. We report the contexts closest the cluster medoid. Clustering on the loss trajectories only discovers a relatively simple skill, continuing nouns composed of multiple tokens. POLCA breakthrough clustering recovers similar skill in Figures 10i and 10ii as well as discovering other skills.
21

---

Published as a conference paper at ICLR 2026
C Clater
Chaster Label
Contesis closest to medoed
Projected loss
Fao Neaspaper Company Linited O under Ho Man-fat. It was initially publishee every three
Punctustion after noun
the 6th and 5th centuries BC, to denote suse type of sibant sound.
phrase
president of the United States commander Forces, Vany presidents,
regular officers, eiliesa officers, valunteers). vannto Table United States presidents
5000
Iteration
tournesent to have a neber of teams power tno, gives extra advantage
Projected loss
noun phrase
down from 1,989 at the 2800 census. The Albert Gallatin School District
Chaster
Full Full-rank mall-rank loss
official from the Flactorate of the Palatinate son and civil parish in Hilttnn Keynes, cereronial Buckingharshiire, England. For local ering
Brandenburg-
of>
coast.
Cape Lookout State Park located on the north side of
Iteration
(iii) Cluster data examples
(ii) Exact loss
(a) POLCA clusters from basis vector 10.
Claster
Projected loss
Claster Label
Contexts elosest medoodd
Starting or continuting foreigh
arsteveld (In Hungarlan), also (povina (in Slavak), Frunza de time and effort ne devoted to nedicine. He died Lina 1861. AninCayetane
language
proper
Beauharnois, Châtesugusy lluntingion and Saint
noun
English churchmon who known his combative presching his Latin poetry. Anin (1964) and its sequel Breakin' 2: Electric hongatoo (1984). Anin acon offers apparturities to participate in organizations, versity athletics, comunity ty service, and international travel, Anin
Iteration
Projected loss
2
Repeated nexline
10
Full loss
Staphyloesccus 53p.. Escherichia col, Salnonella.
ede Rukawa in Slan Dunk, Ayato Sakanaki in Disbolik Lovers,
Token after proper
piace South African
Joharnesburg. AnknnThy foreed in 1998 when Martin, Rade and
(iii) Cluster data examples
Iteration
(ii) Exact loss
(b) POLCA clusters from basis vector
Figure 10: English language modeling data clusters with POLCA. We compute breakthrough clustering on POLCA trajectories for each vector and report the average decomposed POLCA trajectories (10ai and 10bi). Figures 10aii and 10bii show the average the per-token loss trajectories for each of the clusters found using the POLCA trajectories. For each cluster, we provide a label based on the top tokens in the cluster and the top 10 contexts closest to its medoid. We then report the 3 contexts closest to the cluster medoid. Clustering on the decomposed POLCA trajectories reveals breakthroughs at points training where the per-token loss curve remains smooth.
22

---

Published as a conference paper at ICLR 2026
00-
Claster
Projected loss
0.14
Choster Label
Contests closest to medoid
publishes the panphiet Les Peintres
Ver h/ro//noun
ispressionistes.M klie perfects
painter
-022
sentence- initial
Foremost nast called jibs, headsails, forasalls.
2000
10000
Iteration
won the Minnesota pageant compoted at Niss USA, Vntnnnniia Nlese of On
Projected loss
AnSee alsn/vRoyal wea.lattneeFADDAAPPAtAP <PAD>-PPDo -A------------------- PDD
(disarbi
Chaser
Repeated
<PAD>
op o
Full-rank loss
<PAto wttpo//o---ttttt-------------
cp <PAtt--PAto CPAtt chot-- ott.
<PAD--PADD<PPA
(iii) Cluster data examples
5000
10000
Iteration
(ii) Exact loss
(a) POLCA clusters from basis vector 28.
0.0-
Claster
Projected loss
-0.14
Chuster Label
Contexts closest to medoid
Rooan Caresti (cnater), crater planet Cores, known
Tokan after S,>
his first novel, The Watsens Go to Birminghar.
Bud Caldwell, the main
5000
10000
character, travels built, bypassing Fingal. by the Pare Marquette walyy,, boost
Iteration
Projected loss
Chasser
Repeated
<PA oP oft wa shoo PA
<FAD>
Full-rank loss
<PAT wtt.--/AAt---A---PP---------
cP sPA PPDo--PP--------- PADD-PPP PPAto PAw.PAAAAA-A---------A-- wtt.aan------a--------------
5-
<PAD-PPED-PPAD
(iii) Cluster data examples
5000
10000
Iteration
(ii) Exact loss
(b) POLCA clusters from basis vector 29,
Figure 11: English language modeling data clusters with POLCA. We compute breakthrough clustering on POLCA trajectories for each vector and report the average decomposed POLCA trajectories (1 lai and 11bi). Figures 1lai and l show the average of the per-token loss trajectories for each of the clusters found using the POLCA trajectories. For each cluster, we provide label based on the top tokens in the cluster and the top 10 contexts closest to its medoid. We then report the 3 contexts closest to the cluster medoid. Clustering on the decomposed POLCA trajectories reveals breakthroughs at points in training where the por-token loss curve remains smooth.
23


Reviews:
[{"review_id": "ZsnRvtPOV1", "reviewer": "Reviewer_f3uN", "summary": "The authors propose to decompose the learning process of neural networks into contributions based on subsets of the data and directions in parameter space, so as to more clearly exhibit “hidden” phase transitions or breakthroughs corresponding to individual concepts. This kind of study of neural network development is an emerging area of interest and they propose a new simple and (perhaps) scalable methodology POLCA for discovering these breakthroughs and attributing them to patterns in the data distribution. This methodology is tested in two settings: multi-digit addition where interpretable breakthroughs are found by the method, and language model training (40m parameters) where the results are interpretable but more mixed (in my opinion).", "questions": "* What is the reasoning for excluding basis vectors where the loss increases? It does seem like even when projected onto the remaining basis vectors loss increases are typical for many clusters at some point in training.  \n* The only limit to scalability I see here is approximation of Hessians, which is quite memory intensive at scale. The experiments done here are in relatively small models but I do not doubt this could be done at larger scale. Could the authors comment on any significant limitations here?  \n* The aggregated loss decreases, but it has been widely observed that some individual token losses have non-monotonic behaviour. Even if we see a clear “breakthrough” trend where the projected loss plateaus, decreases and then plateaus, I’m unsure whether I would think of that as a genuine breakthrough. It would seem to depend on how semantically coherent the cluster is. Can the authors confirm that this concern is what is motivating the details in the “automatic labelling” section e.g. the 70% threshold? I may be incorrect, but this “unity” of the cluster seems to be quite a crux for the methodology and I’d appreciate seeing more detail on it.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Well motivated problem, and a clean derivation of a simple methodology for addressing it  \n* The paper is very clearly written, with appropriate and well-captioned figures  \n* Some quite compelling results in a toy setting with addition and carrying  \n* The automatic labelling for clusters in the language model setting seems to be well-done, I found this interesting in its own right.", "weaknesses": "* Major: I am not completely convinced by the framing of “breakthroughs in the loss” being discovered in the smooth learning curve via POLCA for the larger language models. If I can summarise (8) we decompose the loss into a sum of components, which may be positive or negative. By their nature (since they depend on narrower subsets of the data distribution and directions in parameter space) these components will tend to have much more variety in their behaviour over training, and by construction them sum to something we know is generically fairly featureless. It is therefore no surprise to see empirically that there is such variation. This seems not sufficient in my mind to justify language like “breakthroughs in the loss projected onto that basis” in e.g. Fig 5\\. In both cases the projected losses either seem fairly uninteresting (cluster 2 in 5(a)(i) and cluster 3 in 5(b)(i)) or non-monotonic. Could the authors elaborate further on why I should look at these plots and see a strong analogy to Fig 4 (where I do agree with the interpretation of hidden breakthroughs).  \n* Minor: in both spirit and methodology the study here seems quite close to that used by Michaud et al (2024) and I think it is worth addressing the novelty of POLCA relative to that work directly.  \n* Minor: I think many of the figures with training steps on the x-axis would benefit from being plotted in log-scale. For instance all the interesting information in Fig 4 is compressed into the very beginning of the plot, making it hard to distinguish the clusters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jFRkZ4sMvY", "reviewer": "Reviewer_rAUf", "summary": "This work investigates how training breakthroughs, sharp decreases in training loss attributable to a meaningful change in the model’s predictions, can be automatically discovered. Analyzing the loss aggregated over all examples can allow for the discovery of some breakthroughs. However, the authors argue that the interplay of how the model learns new skills, as well as how different examples may require different skills for correct prediction, leaves some breakthroughs obscured. To discover new breakthroughs, the authors propose a new approach, POLCA, that determines using a Taylor approximation the change in loss attributable to a specific direction in parameter space. Applying clustering methods to the projected loss of specific examples helps elucidate human interpretable features that the model has learned in order to decrease the loss. The authors experiment with their approach on a synthetic addition task with pre-specified skills, as well as a general task of language modelling over a natural language dataset. They demonstrate that POLCA can discover breakthroughs that are obscured in the aggregated, unprojected loss.", "questions": "- [Q1] Notably only two of the 22 basis vectors collected for the experiment in Section 5.1 are presented in the main text, and 5 in the Appendix. This is after discarding several basis vectors as well. How should one think of these vectors, meant to represent a “breakthrough” since they present a mean decrease in the loss, when they also don’t provide a meaningful interpretation? Is POLCA meant to provide a set of results that have to be further interpreted and classified, rather than presenting a set of vectors that are inherently important?\n- [Q2] On line 315 and 316, it is mentioned that tokens are discarded if their loss increases along a given basis vector. How many tokens are typically discarded this way? If some tokens have their loss increase along this direction, how can we know it still represents a training breakthrough?\n- [Q3] Line 198 of section 3.1 states that by repeatedly adding top eigenvectors from successive checkpoint Hessians, directions of long-term stable movement will be added. However, since each successive Hessian is projected onto the nullspace of the current set of vectors, is this not more likely to collect directions reflective of local oscillations, as the stable directions are likely already represented in the collected set of vectors?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The paper is well presented, with clear writing, high quality and intuitive figures, and a strong argumentative flow.\n- The selected problem of discovering training breakthroughs is interesting and pertinent to many areas, such as interpretability and the study of model training dynamics.\n- The new method, POLCA, is intuitive, well justified theoretically, and provides a satisfying means for attributing changes in loss to different directions in the parameter space. While it bears a strong similarity to its predecessor LCA, the change to an arbitrary basis over the parameter space is important, as these training breakthroughs should not be expected to be attributable to individual weights in the model.\n- The chosen experimental settings (synthetic counting task and natural language modelling) are good choices for both explanatory and argumentative purposes.\n- Some presented results are very interesting and convincing. For instance, demonstrating that there is a clear clustering on digit but not on carry skill (Figure 3) is interesting, and justifies the need for additional methods for discovering training breakthroughs.", "weaknesses": "- [W1] The results presented are not entirely convincing in terms of showing that POLCA discovers training breakthroughs based on human interpretable features. In figure 4c and section 4.2, for instance, the authors claim that the first basis vector “recovers the digit skill”. However, clusters 1 and 3 are composites of multiple different digit positions, so it is difficult to say that all of those examples are having their loss improve because of the same skill, especially since one digit is excluded.\n- [W2] It is not always clear that the projected loss trajectories imply a training breakthrough. For figure 4a, does the slight decline of cluster 1 imply a training breakthrough? What is the implication that cluster 3 has a sharper projected loss drop, but has very few examples prescribed to it? Figure 5 shows directions where projected loss for some examples increases while for others it decreases. Why would the understanding of one human interpretable concept reduce understanding of a related concept?\n- [W3] It’s unclear how the method should be applied given certain methodological choices and the results presented. See [Q1] and [Q2].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wqe1OGVCgf", "reviewer": "Reviewer_VeXn", "summary": "The manuscript \"Hidden breakthroughs in Language Model Training\" introduces POLCA, an unsupervised learning analysis of the loss-function trajectory in training. The overall loss is projected onto a suitably defined orthogonal basis and decomposed in contributions for each data point. Subsequently, the decomposed trajectories are analyzed with the HDBSCAN (a density-based and hierarchical) clustering algorithm, chosen mainly for its ability to single out outliers and deal with variable density in the representation space.", "questions": "I'd like the weaknesses points to be addressed", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors compare their approach to the previously introduced Loss Change Allocation (LCA) and convincingly debate that the introduced POLCA differs significantly from LCA. The mathematical justification and description is sound and the authors take great care to describe computationally feasible implementation solutions.\nAs for any unsupervised learning analysis, it is very difficult to define objective performance indicators as well as design demonstrative datasets where the algorithm finds what it is expected to find, without being too trivial. Keeping in mind these inherent difficulties, it is my opinion that the authors do a remarkable job in both aspects.\nAs seminal idea, the paper is polished enough to be publishable.", "weaknesses": "However, I see a few weaknesses:\n- The authors use frequently terms like \"(conceptual) breakthrough\", \"phase transition\", \"skill\", without providing a proper scientific definition of them. As much as these terms are used in the related (and cited) literature, it would be better to attempt a self-contained definition to avoid talking to a very specific audience only.\n- it looks to me that nothing in the introduced methodology is specific to language models, but rather it could be applied to any (deep) network, with obviously different narrative related to the found clusters. It would be good to comment on this observation.\n- It is unclear if there is a somewhat optimal choice for k (for the top k ranked eigenvectors) and the frequency of checkpoints. There is a tension between computational burden and explicative power, but it would be nice to know if there is any suggested \"light\" analysis one may do preliminarily on the model and training data to assess good values of those parameters.\n- I did not find a description on how to, in practice, \"discard the oscillatory directions which do not provide an overall decrease in  loss over the course of training according to POLCA.\" (lines 200-201)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GWD2PGWyKy", "reviewer": "Reviewer_sGxR", "summary": "The authors introduce a new method (POLCA) for decomposing per-sample loss trajectories over training and validate this clustering as an interpretability technique in a synthetic arithmetic setting and a natural language setting. \n\nThe methodology involves approximating the change in per-sample loss between successive checkpoints using a second-order Taylor approximation in the weight update restricted to a subspace constructed from the top-k eigenvalues of the Hessian at each of a set of $T$ intermediate checkpoints. For efficiency, the authors additionally approximate the aggregate Hessian using a per-sample Hessian. This decomposes the loss into a set of \"projected losses,\" one for each basis element. \n\nTo use POLCA for interpretability, the authors cluster the projected loss trajectories. They demonstrate that clustering the projected losses splits samples into interpretable clusters and provide evidence that this beats baselines like clustering (non-decomposed) per-sample losses.", "questions": "Questions:\n- Is $B$ in (2) the same as $B_T$ from Section 3? Or is it $B_t$ for each individual timestep? \n- Some of these cluster error bars are super wide in figures 10–11. That suggests to me that the clustering technique is running into problems. Why cluster just the projected losses and not the full $Tk$ decomposed loss vector? What other ways have you thought about for constructing the basis? Why this one? \n- Do you really need this evolving Hessian? Why not just project against (the top Tk eigenvectors of) the final Hessian?\n- How accurate is this aggregate-to-per-sample-Hessian approximation in Appendix C really? Do you have an empirical comparison? \n\nSuggestions:\n\n**Improve the description of the method.** I found difficult to parse lines 224–240 the first time around. I didn't realize that \"we use a second-order approximation\" here referred to \"[adding on an additional quadratic term in (1)]\" rather than \"we use a second-order approximation [in order to choose the basis B].\" \n\nMy recommendation:\n- In (1), show the full second-order expansion. \n- Subsequently, define the LCA as the first-order term in this expansion. This would be a new equation (2)\n- On line 224, rephrase to \"we use the full second-order approximation in (1)\"\n- On line 235, rephrase to \"Exact computation of the second-order term in (1) would be intractable.\"\n\n**Put the clusters online!** It would be super valuable to be able to browse through the clusters on an interactive website. LLMs make this super easy nowadays.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Problem**: The problem of finding ways to decompose the training trajectory is important and of growing interest, given the rise of the communities working on learning dynamics and developmental interpretability.  \n\n**Technique**: The general approach of modeling updates to losses using a Taylor approximation (plus various additional approximations) and clustering trajectories seems sensible and clever. Some of the particular choices around basis construction seem rather ad hoc. Still, this technique seems likely to become a go-to method and baseline for decomposing loss trajectories.", "weaknesses": "**Interpretation**: Section 4 and the Figure 3–4 captions are missing a discussion of what \"recover the skill of X\" means. My concern here would be largely addressed if there were a description of how the ground-truth is established in the arithmetic setting (how do you know what the \"skills\" are here?). Section 5 is in a better place because it includes a discussion of how clusters are automatically labeled, but lines 450–451 are still vague. Does this mean that you manually come up with names for the labels in Figures 5 and 9–11? If this involves a manual labeling stage, then the tables in these figures should also report how accurate these hypotheses are (i.e., what fraction of in-cluster samples are explained by the given hypothesis?) It should be relatively simple to write automated tests once the tests have been formulated.  \n\nIf you address all of the above concerns, then I will change my review from a \"weak accept\" to an \"accept.\" Note: I am open to being convinced to change this even if you do not manage to run substantive new experiments and focus on making the writing around this clearer. \n\n**Benchmarks**: The baselines and ablations in appendices G and H are solid, but the benchmarking is still quite weak. In particular, I don't understand why I should trust the \"carry skill homogeneity\" metric. I would like to see some benchmarking on additional metrics/tests in the natural language setting. I'm sure it's possible to derive suitable benchmarking metrics from the automatic labeler.\n\nIf you address my previous concerns and significantly buttress the benchmarking, then I will change my review to a \"strong accept.\"\n\n**Compute**: The paper is missing a discussion of computational complexity. Hessians are expensive, even when you restrict to the top-k eigenvalues. This could be in the appendices. (The rest of this paragraph is optional.) It would also be valuable to include a (brief) comparison of the computational complexity against the baselines. In particular, I'm interested in understanding how much more compute-intensive second-order POLCA is versus first-order POLCA. If you get second-order POLCA essentially for free, then that would be worth mentioning as a strength in the main body. \n\n**Hyperparameters**: It is not clear what hyperparameters are being used for POLCA. This would be addressed by an additional set of tables in the appendices detailing the hyperparameters used for the Hessian estimation and the hyperparameters used for HDBSCAN. The paper would also benefit from some additional ablations to POLCA hyperparameters. I'd also be interested in a comparison to other clustering algorithms (though this is a marginal addition).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)