Formation of Representations in Neural Networks
Authors: Liu Ziyin
Isaac L. Chuang
Tomer Galanti
Tomaso A Poggio
Venue: ICLR 2025 Spotlight (2025)
Keywords: representation learning
neural collapse
neural feature ansatz
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Understanding neural representations will help open the black box of neural networks and advance our scientific understanding of modern AI systems. However, how complex, structured, and transferable representations emerge in modern neural networks has remained a mystery. Building on previous results, we propose the Canonical Representation Hypothesis (CRH), which posits a set of six alignment relations to universally govern the formation of representations in most hidden layers of a neural network. Under the CRH, the latent representations (R), weights (W), and neuron gradients (G) become mutually aligned during training. This alignment implies that neural networks naturally learn compact representations, where neurons and weights are invariant to task-irrelevant transformations. We then show that the breaking of CRH leads to the emergence of reciprocal power-law relations between R, W, and G, which we refer to as the Polynomial Alignment Hypothesis (PAH). We present a minimal-assumption theory proving that the balance between gradient noise and regularization is crucial for the emergence of the canonical representation. The CRH and PAH lead to an exciting possibility of unifying major key deep learning phenomena, including neural collapse and the neural feature ansatz, in a single framework.

Published as a conference paper at ICLR 2025
FORMATION OF REPRESENTATIONS IN NEURAL NETWORKS
Liu Ziyin'.3, Isaac Chuang', Tomer Galanti", Tomaso Poggio
Massachusetts Institute of Technology
2 Texas A&M University
3 NTT Research
ABSTRACT
Understanding neural representations will help open the black box of neural net- works and advance our scientific understanding of modern Al systems. However, how complex, structured, and transferable representations emerge in modern neu- ral networks has remained a mystery. Building on previous results, we propose the Canonical Representation Hypothesis (CRH), which posits set of six alignment relations to universally govern the formation of representations in most hidden lay- ers of neural network. Under the CRH, the latent representations (R), weights (W), and neuron gradients (G) become mutually aligned during training. This alignment implies that neural networks naturally learn compact representations, where neurons and weights are invariant to task-irrelevant transformations. We then show that the breaking of CRH leads to the emergence of reciprocal power- law relations between R. W, and G, which refer to as the Polynomial Alignment Hypothesis (PAH). We present a minimal-assumppion theory proving that the bal- ance between gradient noise and regularization is crucial for the emergence of the canonical representation. The CRH and PAH lead to an exciting possibility of unifying major key deep learning phenomena, including neural collapse and the neural feature ansatz, in a single framework.
INTRODUCTION
The success of deep learning is often attributed its ability learn meaningful latent represen- tations from data (Bengio et al. 2013). These latent representations, progressively formed as data passes through the network's 's layers, are found to encode increasingly abstract features of the input:
x =²
h 0,
(1)
where is the input, the output, D the network depth, and h the activation of the i-th layer. For neural networks to perform well, the transformations between layers must capture meaningful structures in the data. Understanding how these latent representations are formed and structured is a foundational problem in deep learning, with implications for both theoretical understanding and practical applications. Despite significant advances, how neural networks organize and transform these internal representations remains an open question. This gap in understanding hinders our ability design more efficient, interpretable, and generalizable models.
In this work, we seek to bridge this gap by introducing the Canonical Representation Hypothesis (CRH). At its core, the CRH posits that neural networks, during training, inherently align their representations with the gradients and weights. The satisfaction and breaking of subset of CRH equations are found to delineate the universal phases, which are empirically observable scaling re- lationships between the weights, activations, and gradients. The CRH reveals a striking aspect of representation learning: there may exist set of universal equations that govern the formation of representations and universal phases which distinguish the layers in modern neural networks, inde- pendent of the task, architecture, or loss function. Thus, the CRH provides an useful perspective on how neural networks evolve toward compact and interpretable solutions.
Our main contributions are the following:
1. Proposal and justification of the CRH which states that within layer, the neuron gradients, latent representation, and parameters are driven into mutual alignment after training, due to moise-regularization balance (Section 3 and

---

Published as a conference paper at ICLR 2025
2. Identification of mechanisms that break alignments, quantified via Polynomial Alignment Hypothesis, which predicts power law scaling behaviors that characterize distinct phases of neural networks arising when the CRH is broken (Section
The experiments are presented in Section 6 Section 7 discusses the implications of CRH to the formation of neural representations and the connection of the CRH prior observations. Related works are discussed in Section 2 The proofs are left to the Appendix Section B
2 RELATED WORKS
Empirical results show that the representations of well-trained neural networks share universal char- acteristics (Maheswaranathan et al., 2019 Huh al.) 2024 Ziyin et al., 2025). For us, closely related phenomenon is the neural col collapse (NC) (Papyan et al. 2020), which studies how structured and noise-robust low-rank representations emerge in a classification model. The CRH can be seen as generalization of NC because one prove that when restricted to certain settings, the CRH is equivalent to the NC (Section 7).. Another related phenomenon is the neural feature ansatz (NFA) [Radhakrishnan et al. 2023). which shows that the weight matrices of fully connected layers evolve according to the gradient outer product during training. However, the NEA studies the weight evo- lution, not the representations. Empirical power-laws
known to exist in large neural networks Kaplan et al., 2020; Bahri et al. (024), which relates the model performance to their sizes. The power laws discovered in our work are different because they reciprocal relations that relate dual objects (R, G. W) to each other rather than to the performance. Other related works are discussed in the context where they become relevant. More related works are discussed in Appendix
CANONICAL REPRESENTATION HYPOTHESIS
Let us consider an arbitrary hidden layer hú of any model after linear transformation:
h Wh, (x).
(2)
For convention, hy is called the "preactivation" of the next layer, and ha is the "postactivation" of the previous layer. The gradients of the representations are also of interest: 9a -V l and gb = -, where l' is the sample-wise loss function. We note the generality of this setting, as ha can be arbitrarily nonlinear function of x. and (x)) (x)) can be another arbitrary transformation. Also, letting h. (N,(x), 1) accounts for when there is trainable bias |
Much recent literature has suggested that the quantities has ha, W and their gradients are correlated with each other after throughout training. The neural collapse phenomenon suggests that in a deep overparameterized classifier, EE h oc WTW for the penultimate fully connected layer (Papyan
al. 2020 Xu al., 2023b Ji et al. 2021 Kothapalli| 2022, Rangamani al., 2023). In the study of kernel and feature learning, a primary mechanism of how the neural tangent kernel changes is
that after few steps of update, the representations become correlated with the weights (Everett
itt al. (024)) and so the quantity W ha will be significantly away from zero, which also implies strong relationship between WTW and [B] The recent work on neural feature ansatz shows that WTW oc FV, h. for fully connected networks (Radhakrishnan et al. 2023 Gan & Poggio 2024). The idea that the latent variables will become correlated with the weight updates is also central notion in the feature learning literature (Yang Hul 2020 Everett al. 2024).
Taken together, these results suggest a simple and unifying set of equations that can describe a fully connected layer after (and perhaps during) training. Let e e {a, and define H. h. G = [[ and Z, = M. where Ma = WT M,. One can imagine six alignment relations between all the quantities within the same layer:
ppresentation-gradien alignment (RGA); H. x Gos representation-weighh alignment (RWA): ; He Ze:
(3)
(4)
weigitnt alignment (GWA): G. x 7c,
(5)
where E denotes the averaging over the training set. Because hú, comes after h2 during computation, will refer to the alignment between any of the b-subscript matrices a forward relation and all a-subscript matrices as a backward relation.
As an example, consider two-layer network. f(x) = W2o(Wiz), with fully connected layers. For the first layer, hi x, and hig Wy.. For the second layer, h (WW and h Woh
2

---

Published as a conference paper at ICLR 2025
For the formation of representations in neural networks, the most important relations in Eq. ( (5) are perhaps the forward and backward RGA, as they directly relate the representations E hh to their gradients with respect to the loss function. Because there is scientific reason for to believe that the forward representation is more important than the backward representation or vice versa, one should study both directions carefully. The backward RWA in its general form has not been discussed in the literature but has been implicitly studied in the particular setting of neural collapse, which happens in the penultimate layer of an image classification task (Section 7 while the backward GWA seems to be unknown to the best of our knowledge. The backward GWA is not identical the neural feature ansatz (NFA) but can be seen as an equivariant correction to the NEA (Section 7). and the forward GWA is also unknown to the community. Adding together the forward and backward versions, there are six alignment relations. That these relations simultaneously hold for any fully connected layer will be referred to as the canonical representation hypothesis (CRH).
Three scientifically fundamental questions are thus (1) does there exists a rigorous set of assumptions under which the CRH can be proved; (2) what mechanisms can cause the CRH to break; and (3) can predictions of the CRH and its breaking be empirically observed in realistic deep neural network settings? We devote the rest of the paper to answering these three questions in the given order, and then we collect insights from all these answers.
4 NOISE-REGULARIZATUR BALANCE LEADS TO ALIGNMENT
Notation AA denotes the difference in the quantity A()) after one step of training algorihtm iteration time step t: AA A(0++1)~ A(O,). 77 denotes the learning rate and y denotes the weight decay. E denotes the empirical average over the training set. ((x, f) denotes the per- sar-se loss function, where x the data point and is the parameters. Its empirical average is the empirical loss L: L(O)= B[((x,))) 0)].
In this section, we present a formal and rigorous framework under which the CRH can be proved. As will become clear in the next section, this proof also explains how and when the CRH may fail. The problem setting is the same as in Eq. 2). The training proceeds in an online learning setting in which the training proceeds with weight decay of strength y. We make the following assumption.
Assumption 1 (Mean-fiele norms). The norms of and h approximate their empirical averages: (A1) 1 = = hh ][ (A2) |1.9% 1 [²]
This assumption holds, for example, for a high-dimensional Gaussian random vector, whose norm is of order Odd) with vdd standard deviation. A1 also holds automatically if the representations are normalized. Note that only a subset of all the assumptions is needed to prove each equation we derive below. For example, Eq. (6) below only requires Al to prove. The minimal set of assumptions required to prove each equation in this section are stated in Section B.3 We discuss the main intuition for the proof in the main text, and present the formal theorem at the end of the section.
Forward alignment.
Consider the time evolution of hghg during SGD training:
A(h(()) (x)) l
hool 2yhigh)) [/all 9u96 001 Ia(ha
where 7/ is the learning rate. At the end of training, the learned representations should reach station- Taking the expectation of both sides, we obtain
arity and so A 0F
0 = zn [[ + + E[
(bahh) + [
(6)
learning
regularization
noise
where 1]. The noise term is due to the discretization error of SGD and can be significant either when the step size is large, or the gradient is noisy. The mechanism behind this alignment is that while the gradient noise expands the representation, weight decay contracts it. When the
dynamics reaches approximate stationarity, the dynamics due to learning plus these two effects must balance at the end of training.
Now, if additionally either (a) [[AW]= = 0 (namely, at a local minimum) or (b) B[A(WWT)]] holds (Section B), the weight will also align with the cross terms between Do and hs: WWT oc
SSee Figure 21 for the evolution of AH.
3

---

Published as a conference paper at ICLR 2025
B[goh]]] + E. which leads to the effect that the learning term above must also balance with the regularization term. Thus, eventually, the regularization effect will have to be balanced with the gradient noise. Lastly, if both (a) and (b) hold, the alignment between all three matrices emerges: G2 oc H3 < Zg.
Backward Alignment. One can similarly derive condition for B[AG,] 0: za (B(hag0] B[gahg]) + Bhahh 2y1[[9.9h1 where we have defined 2a = B[ 22. The forward CRH can then be derived if WTW W and W reaches stationarity.
The following theorem formalizes these results.
Theorem 1. Under Assumption ! when B[AH,] = O, E[AG,] = O. B[A(WW")] = 0, and B[A(WTW)] = O, there exist real- constants C1 c22 c3, Cs 0 such that
WW + E[ e2B[hgh W + +g c.B[aao.]
(7)
Additionally, if at local minimum,
WW oc [[ggo]]] < [[ggh]], WTW x [ oc [[gagh]
(8)
Remark. A strength of this derivation is that is oblivious to the loss function, the model architec- ture, or the type of activation used (as long as the second moments exist). This may explain the wide applicability of the RGA observed in Section 6 The above aligument relations can be seen as type of fffuctuation-dissippae
theorem in theoretical physics (Kubo) (966). which states that in driven stochastic dynamics, the fluctuation of the force must balance with the rate of energy loss - a sun- damental law first discovered by Einstein (1905). Prior applications of the fluctuution-dissipall theorem to deep learning have focused on the covariance of the model parameters (Yaida) 2018 Lie et al. 2021) and are not directly relevant to the representations.
CRH BREAKING AND POLYNOMIAL ALIGNMENT HYPOTHESIS
While the CRH can be found to hold for many scenarios, is highly unlikely that it always holds perfectly and for every layer (e.g., see Section 66. In this section, we study what happens if the CRH is broken; we then suggest two mechanisms which cause the CRH to break. The following theorem shows that all six relations
intimately connected, even if only subset of the CRH holds. For square matrix A, we use A := (A+)" to denote the n-th power of the pseudo inverse of A, and A AA* is orthogonal projection matrix to the column space of A.
Theorem 2 (CRH Master Theorem). Let A, B, C be a permutation of B[hh"], E[gg"), }, and Z, and let D == PDP be projected version of D for projection matrix P. Then,
1. (Directional Redundancy) if any two forward (backward) alignments hold, all forward (back- ward) aligrometss hold;
2. (Reciprocal Polynomial Alignments) if one of any forward alignments and one of any backward alignments hold, there exists scalars an. Sc, and de satisfying -1 Ccey Bo, dg S 3 such that
An B. os C%:
(as detailed in Table where e {a, b) denotes the backward and forward relations respectively, and the corresponding projection Pa E (Ze, EThhh [[ ]0), e.g. such that À = P.APc.
3. (Canonical Alignment 1) If (any) one more relation holds in addition part 2, then all six align- ments hold in the 7 subspace; in addition, at local minimum, all six alignments hold;
4. (Canonical Alignment II) If all six alignments hold, E[hh") E[gg") 2 xr P, where P is an orthogonal projection matrix.
The idea behind the proof is that there some redundancy in the six matrices: every forward relation implies backward relation and vice versa. As an example, if Z, Ha. then we also have Z3 oc Ho. which can be obtained by multiplying W on the left and WT on the right.
Part (4) of the theorem clarifies what it means to satisfy the CRH: the latent representation is fully compact, where weight and representation are only nonvanishing in the subspaces where the gradient is nonvanishing. Moreover, the weight matrix does nothing but rotates the representation, implying that the information processing is invertible once the CRH is fully satisfied. This is consistent with the observation that once a layer has almost perfect alignment, all the layers after it also
4

---

Published as a conference paper at ICLR 2025
have perfect alignment, sign that the representation cannot be further compressed (Section 6 Therefore, the CRH is consistent with the observation that last layers neural networks are low- rank and invariant to irrelevant features.
Part (2) is especially relevant when the CRH is broken. Depending on which subset of the hypothe- holds, the learning process may be classifiable into many as 2 64 phases. In different phases, the learning dynamics and the found solution will likely be different due to different scaling relations. For example, positive exponents between Za and H. will imply that the layer is enhancing the principle components of Ha. while suppressing the lesser features; a negative exponent would imply the converse. Even if we remove the redundancy implied by the theorem, there are still least 22 phases. One can also define additional phases according to the ordering of the degree of breaking for each relation, which gives 6! 720 phases, although these ordering phases may not have major influence. In many experiments, we performed, at least one of the forward relations and one of the backward relations are observed to hold very well (e.g., see Figure R). This means that one is quite likely to observe the power-law relations predicted in Table We also find it common for different layers to be in different phases, even within the same network. We discuss more potential meanings and examples in Section D
Broadly interpreted, part (2) predicts a power law relation between the speetrum of all six matrices, which is also what observe in almost all experiments (Section
6). What is quite surprising is that
almost all positive exponents we observed are within the range [7],,3] which is exactly the range of exponents that the theorem predicts (e.g., see Section C.8 Formally, that the H, Z, and G' are polynomially related can be called the "Polynomial A Alignment Hypothesis (PAH)" and is a natural extension of the CRH. That scaling relations can be used characterize different phases of matter is an old idea in science. In physics, phases can be classified according to their scaling exponents, and having different set of exponents implies that the underlying dynamics and mechanism are entirely different (Pelissetto & Vicari 2002). This connection corroborates our physics-inspired proof.
Breaking of CRH. A major remaining question is whether we can find mechanisms such that the CRH breaks. The theory in the previous section implies one primary mechanism that the CRH breaks. For all six alignments to hold, it needs to be the case that both [E] [[]. [[ are zero, but these two conditions may not be easily compatible with each other, they together imply that AW has zero variance. While this may be possible for some subspaces (e.g., see Ziyin et al. (002a)), it does not hold for every subspace. In fact,
easy show that unless the minibatch size is large enough the SGD updates will never reach zero variance that is AW # O even for / - (see Lemma in Xu et al. (202bb); see also Xu et al. (. Thus, the competition between reaching a training loss of zero and the need to reach a stationary fluctuation is primary cause of the CRH breakage.
Phase
Back. Alignment
Forw. Alignment Back. Power Law Forw. Power Law
NC NEA CU
1lm
CRH
He 0c Za x Go H. x Z. < G.
H. << Zo
x Gr
back. CRH
H 0 28 x Gr (H, 233
forw. CRH
Ho x Zo xc Go
H. x 2 x Go.
(3--)
(2, x G.)
H. Gn
Hd Gh.
H. x Z. x G
H. x 2, G H. 7, Go
3(3-6)
2
Hg x Za
Hg x 7
H. x 7, x Go
Ga Zu
Gb x Zv
H. Z. G.
H. 7, G.
Ha Ga
Ho x Zo
H. 2% G.
H G
Ha x Za
Hg x Go
H. x 2 x G.
H1
Cr
((3-6)
Ha G.
Go x Zo
H. 72 G.
H 7 G
/(1)
7
Ga Z..
H₁ x Go
H. Z, G.
Hi 0 2 G.
1
Ha x Za
Go 0 Za
H² x 22 x G.
H. 0 23 06 G
/(2)
Ga Z.
H, Zo
H, 7 GA.
17. 7g G
Table 1: The reciprocal polynomial relations of the CRH Master Theorem. When one forward relation and one backward relation hold simultancously, all six matrices polynomially aligned in a subspace (Thcorem 2 Each scaling relationship can be regarded panel shows how existing observations about neural networks fit into the phase diagram. A / denotes that this phenomenon is compatible with the specified phase. NC refers to the neural collapse. NFA refers to the neural feature ansatz. CU (correlated update) refers to the (idcalization of the) common observation that ha is correlated with W few steps after training (Everett al. 2024). The Ilm column shows the compatibility of the scaling relation for transformer observed in Figure
possible phase for the layer during actual training. The right
5

---

Published as a conference paper at ICLR 2025
penultimate layer
last layer
1.0
penultimate layer
1.0
A/// layer
0.8
0.5
0.6
0.6
0.4
0.4
MRGA
6.4
HGNA
022
HDNA
MINA
1-RMA
6.0
150
50
150
100
Figure 1: Six alignment relations the penultimate layer and output layer a ResNet18 trained on CIFAR- (res1). Left: forward CRH. Right: backward CRH. We see that all six relations hold significantly across two fully connected layers. Also, we show that the matrix cov(g, /t) is well aligned with WWT in the appendix Section[..7 which is strong piece of evidence supporting the key theoretical step that the cross terms will be aligned with the weights (and G, H).
2b
Hb
Go
80
80
60
60
60
40
40
40
20
25
50
75
0
25
50
75
0
25
50
75
neuron index
neuron index
neuron index
Figure 2:
Penultimate layer of the conjugate matrices (H, G, 2) after training (fc2). This is an example of CRH being well satistied, where all three matrices are well aligned after training.
Because the rank tends to decrease for later layers in networks performing classification, we expect that CRH holds better for later layers than for earlier layers.
This problem is especially troublesome in the layers where E [W] has high rank, which holds true for the earlier layers of the network not the later layers (uu et al. (023a). This consistent with the observation that the CRH holds much better in the latter layers than in the beginning layers.
This analysis is consistent with the fact that a stronger alignment is strongly correlated with a more compact representation (Figure
Tinite-Time Breaking of CRH.
To leading order in y, the proof of the CRH implies that
GB + 0(+) 72 x WWWT, H. + +(7) G2G o WTW.
(10)
This means that when y is small, the forward alignment between H; and WWT are strong (because the prefactor is independent of 7), while the other two are weak because the huge disparity be- tween the two matrices, it might take gradient descent longer than practical to reach such solution. Similarly, the backward alignment between G, and W W is strong for a small 7y This prediction will be verified in Section 7 when discuss the neural feature ansatz.
EXPERIMENTS
In this section, we present experimental evidence that supports predictions resulting from the CRH. We also perform experiments to test mechanisms that break the CRH. When the CRH is broken, we put special emphasis on verifying the RGA. as directly links the formation of representations and may arguably be the most important relation of the three subsets.
Metric for alignment. We would like to measure how similar and well-aligned the six matrices in the CRH are. Let A and B be two square matrices, each with d² real-valued elements. We use the Pearson correlation to measure the alignment between two matrices (Herdin et al. 2005):
1 ( K T
1 EA 1 Ar
( B)
(11)
15 1j
where K is a normalization factor that ensures a [-1,1]. alignment between the two matrices and B. Note that a #1 if and only if A B for some
B) will be referred to as the

---

Published as a conference paper at ICLR 2025
2
2
2
2
2
l0g
0
0
-2
-2
logAp
log Ag
1og2g
log
Figure 3: The power-law alignment hetween the cigenvalues An and X of H3 and Go in a six-hidden layer transformer (Ilm). Left to Right: first the penultimate layers. The grey dashed lines show the power-law relations A oc 2 for a 1, 2,3 respectively. We see that the first layer has an exponent of 3, the second has an exponent of 2. and all the layers after it are observed to have an exponent of 1. Different colors show different heads within the same layer. The range of the power exponents is in almost perfect agreement with the predicted range in Table [ Referring to the table, this implics that these layers in phases 5, 8, and 6, respectively. The setting the same as the LLM experiment. Also, see Section C.8 for fully connected nets.
constant co. Therefore, a can be seen as a quantitative metric for the aligmment and if |a| 1, the alignment is perfect. Our initial pilot experiments suggest that the alignment effects are the strongest when the h and g are normalized and if the mean of each is subtracted Thus, we always normalize h and I and subtract the mean to be consistent in the experiments. This is equivalent to measuring cov oo, h) and cov(û. (û.9), where cov denotes covariance and à a/la|]. As notational shorthand, we use Clabedd to denote (<cov(a, b), cov (c, d)) for the rest of the paper.
Settings. We experiment with the following settings and name each setting with unique identifier. fel: Fully connected neural networks trained on synthetic dataset that we generated using a two- layer teacher network. This experiment is used for controlled study of the effect of different hyperparameters. fc2: the same as fcl, except that the output dimension is extended to 100 and the input distribution interpolates between an isotropic and nonisotropic distribution. resl: ResNet- 18 (11M parameters) for the image classification; res2: ResNet-18 self-supervised learning tasks with the CIFAR-10/100 datasets. Ilm: a six-layer eight-head transformer (100M parameters) trained on the OpenWeeTeer (OWT) dataset. The details of training methods are described in Section C
CRH. We start with the supervised learning setting with ResNet-18 trained on CIFAR-10. We measure the covariances matrices with data points from the test set. Figure shows that very good alignment a > 0.7 is achieved quite early in the training, and continues to improve during the later stage of training. This case might remind some readers of neural collapse (NC) because NC is also most significant in the penultimate layer of large image classifiers. As we will show in the next section, in the interpolation regime of classification tasks, the NC is equivalent the CRH.
Another example of CRH is provided in Figure&bbeow, in case of a fully connected network in regression task. The result shows
10a
wank-n deficiency
that when the weight decay is not too small, the CRH is quite close to being perfectly satisfied for intermediate layers. Exam- ples of the representation and the dual matrices are presented in Figure 2 for a layer that (almost) satisfies the CRH.
10-1
Breaking of CRH. A clear evidence for the correctness of the theory is that at a small weight decay, the strongest alignments the forward- RWA and the backward-RGA. which will be pre- sented in the experiment in Figure g after we discuss the relation- ship of the CRH the NFA.
10
10-1
Figure 4: The rank deficiency and the backward 0gg. in fully connected nets (fc2). The rank of representation strongly nega- tively correlated with a. Here, ev- ery color is a different weight de- cay (from 10-6 to 10 "), and every point is different layer in the net. The setting is the same the fully connected net experiment.
Two indirect evidences are that (1) the compactness of the repre- sentation is found to be negatively correlated with the alignment level (Figure and (2) the observed positive exponents between the eigenvalues of the matrices are almost aways within the range [1/3, 3.. which is the predicted range by Theorem 2 (Figure 3)
RGA. A relation of particular interest to the formation of representations is the RGA, which pre- diets that the represenations are aligned with the gradients across them. Now, we show that the
"These may be due to the fact that the gradient and activation have rare outliers that tend to disturb the balance. See Section C for the RGA on transformers without subtracting the mean.
7

---

Published as a conference paper at ICLR 2025
1.0
1.0
02-
0.8
0.8
0.8
0.5
0.6
0.6
0.6
04
0.4
random
02
0.
0.0
50000
100020
50000
100000
50000
100000
200000
Rocation
Acration
tcration
iboration
Figure 5: Alignment between cov(h, h) and cov(g. g) in a six-layer transformer trained on the OWT dataset (Ilm). From left to right: layer 1, 2, 4, 5. Also, see layer 3 in Figure [5] The shaded region shows the variation (min and max) across eight different heads in the same layer. The ROA is significnatly stronger than the alignment between initial and final representation. and the alignment between different heads.
RGA holds well across a broad range of tasks in common training settings. When the CRH holds, the RGA holds as well, and so the experiments in the CRH section already shows that the RGA holds for the last layers of ResNet.
Large Language Models (Im). We measure the covariances of the output of each attention head in every layer. See Figure for the evolution of the alignment and Figure [4] for examples of the representation. Three baselines of comparisons are (1) the alignment between the covariances of a rank-40 (roughly equivalent to the actual ranks of cov(h, h) and cov(g. g)) random projection of two 200 dimensional isotropic Gaussian, which stays close to 0.14, (2) the alignment between the feature covariance of different heads, which starts high but drops to value significantly lower than hgg.hh. (3) the alignment between the initial feature and current feature for the same head, which starts from 1 and also drops quickly. The RGA is found to hold stronger than the other baselines.
Self-Supervised Learning (res2). Self-supervised learning fo- cuses learning good and versatile representation without knowing the labels for the problem. See Figure 66 for the time evolution of a. We see that the RGA holds well for the fully connected layers, but not so strongly for the conv layers. Exper- iments do show increased values if we decrease the batch size and increase y. Also, see Figure 13 for examples of the represen- tations in the convolutional layers. We see very good qualitative alignment between the two matrices.
1.0
byer 1
0.6
ayer 2
aaer 3
byer 4
syer 5
0.2
0.0
250
300
1000
opoch
Figure 6: The alignment for dif- ferent layers during 1000 training epochs (res2). Layers 1-3 are con- volutional layers, and layers 4-5 fully connected ones.
Fully Connected Nets (fel). We also perform systematic ex- ploration of how hyperparameters of training and model archi- tectures affect the formation of representations. Results pre- sented in Section Section[C.6 show the following phenomena: (1) Grad-
ual Alignment: deeper layers tend to have better alignment; if a layer has an almost perfect alignment (a 1), then any layer after it will also have almost perfect alignment (note the similarity of this phenomenon with neural collapse); this is consistent with the tunnel effect discovered in (Masarczyk) et al.) 2024); (b) Critical Depth and Alignment Separation: batch size B affects the alignment sig- nificantly; earlier layers have worse alignment as B increases; later layers have better alignment as B increases; wider networks have similar level of alignment as thinner ones for SGD; for Adam, it is more subtle: early layers have worse alignment for large width, later layers have better alignment for large width; (c) weight decay y affects the alignment significantly and systematically; larger y always leads to better alignment.
diag.
INSIGHTS
off-diag.
This section first studies the implication of the CRH (mainly the
cov(h.)
RGA) for the representation of neural networks 7 1-2). We then
clarify the relation of the CRH to NC and NFA 3-4).
7.1 PLASTICITY- EXPTRIIIVYYEXPESSSIVVVV COUPLING
0
2
direct interpretation of the RGA is that the plasticity of neurons in neural networks is strongly coupled to their expressivity after training:
covlg.g)
Figure 7: The response diversity (the diagonal terms of the covari- ance) and correlation is coupled to the plasticity of neurons (fel).
E [[hhh]]]
(12)
expressivity
plasticity
8

---

Published as a conference paper at ICLR 2025
The first term directly measures the variability of the neuron response different inputs, thus mea- suring how expressive or capable this neuron is. The second term measures the degree of plasticity because the gradient for the weight matrix prior to h2 is proportional to ga and if 9b is zero with zero variance, the input weights to this neuron will never he updated. It is, therefore, a measure of plasticity. This coupling is quite unexpected because the expressivity should be independent of plas- ticity at initialization. See Figure 7 for scattering plot of how the elements of B[hh') J are strongly aligned with that of E[gg"]. J. While this experiment shows a strong alignment between // and G, also shows a limitation of our theory. If the activation norm h is really independent of its direction h/l, one would only see one branch in the figure, whereas there are three branches. This can be another reason for CRH breaking.
7.2 TEATURE-IMPORTANE
ALIGNMENT AND INVARIANT LEARNING
Another direct interpretation of E[gg"] ] is the saliency of a latent feature. The quantity E[gg" measures how the loss function changes neuron activation is changed by a small degree and is often used on the input layer to measure the importance of a feature (Selvaraju et al.) 2017 Adadi & Berrada 2018). This interpretation justifies many heuristics for understanding neural networks: take the principal components of a hidden layer and study its largest eigenvalues. If there is no direct link between the magnitude of the eigenvalues in these latent representations and their importance in affecting the learning (measured by the training loss), these analyses will be unreasonable.
Let be any unit vector that encodes a feature irrelevant to the task. The following theorem shows that the invariance of the representation to such features is equivalent to the invariance of the model.
Proposition
Let f(h(x)) be a model whose hidden states h obey RGA. Let i be a vector and a scalar. The following statements are equivalent: (1) E(f(h + en)) = E(f(h) + O(E); (2) E[hh Jn = 0; the CRH also holds) Wf = 0, and Gh 0.
In some sense, Proposition be seen as a generalization of NC (next section) because it applies to both regression and classification tasks. Note that there are two ways for the condition E( ((](h+e)) e(((h) to be satisfied: (1) ʃ(h + en) f(h), which means that when the model output itself invariant to such variation, the latent representation in this direction will vanish; (2) {(fhh) en' (h)) = {f)),). which means that any variation that does not change the loss function value will vanish. This fact also means that the model will learn a compact representation: any irrelevant latent space will have zero variation. This is consistent with the often observed matching between the rank of the representation and the inherent dimension of the problem (Papyan) 2018 Ziyin 2024). A major phenomenon of well-trained CNNs is that the latent representations learn to become essentially invariant to mask-irrelevan features (Zeiler & Fergus| 2014 Selvaraju et al.) (017)) an observation that matches the high-level features of the human visual cortex (Booth Rolls 1998). The CRH thus suggests reason for these observed invariances.
This result also means that the invariances of the models achieved through the CRH are robust: the objective function is invariant to small noises in the irrelevant activations. While artificial neural networks rarely contain such noises in the activations, it is plausibly the case that biological neurons do suffer from environmental noises, and our result suggests mechanism to achieve robustness against irrelevant noises or perturbation.
7.3 REDUCTION TO NEURAL COLLAPSE IN PERFECT CLASSIFICATION
We now prove that NC is equivalent the CRH in an interpolating classifier. We focus on the first two properties here because NC3-4 are essentially consequences of NC1-2 (Rangamani & Banburski-Fahey 2022). NC1 states that the inner-class variations of the penultimate representa- tion vanishes. NC2 states that the average representation fe of the class is orthogonal to each other: H., Spct The ground truth model must be invariant to inner class variations in classi- fication task. Let e denote the index of class, it should be the case that the ground truth model I' satisfies: fx)) = (I., where 1, is a one hot vector on the e-th dimension and Ç > 0 an arbitrary scalar. Proposition [ thus suggests that any model that recovers the ground truth must have such invariance in the output and in the latent representation h, which is exactly NCL.
Now, we show that when the CRH holds, perfectly trained model must have neural collapse. Let Ie denote any data point belonging to the c-class among a set of C' labels.

---

Published as a conference paper at ICLR 2025
Theorem 3. Consider a classification task and the penultimate layer postactivation ha- the model is innaiiiiterpolating f(x.) = h2 = Wh (x)( = Ç1c, and the loss covariance is proportional to identity: ,VV [e] oc I, then h satisfies all four properties of neural collapse (NC1-NC4) if and only if h satisfies the CRH.
The assumption B[V [VVee oc I is empirically found to hold very well in standard classification tasks at the end of training. See Section C.3 for an experiment with Resnet on CIFAR-10 where the phenomenon of neural collapse is primarily observed.
7.4 EQUIVARIANCE OF CRH AND NEURAL FEATURE ANSATZ
A closely related hypothesis is the NEA, which is related to the feature learning in fully connected layers (Radhakrishnan et al., 2023 Beaglehole al.) 2023). Using our notation, the NFA states W W oc E Vh, /V.. J']] To see its connection to the CRH, note that it can be written as
WTW oc WTE[sg, JW [[g.9]] = E B)V [")]
(13)
where B= V ) T Therefore, the NEA is identical to the GWA (Eq. (5) when B I.
This is an instance of a consistency problem of the NFA: the NFA is not invariant trivial redefinitions of the loss function and model. Let f(x) be model trained on loss
0.3
0.9
function E(f) Let us assume that NEA applies to this model. Now, let construct a trivially redefined model and loss: fx(x) == fx) where Z is an invertible matrix, and ('(f) == f( Now, for NEA must have WTW oc Blv f). but for NEA to hold for f', we need WTW B[ V f]. These cannot hold simultaneously.
0.7-
0.7
hold for f. we
Opos A/
0.5
eNFA
0.5
In contrast, the GWA is invariant to such redefinitions:
Figure 8: After the training of a six- layer fully connected network, all six rela- tions (Eq. (3)(55 hold strongly at a large weight decay (fc2). For a small weight de- cay. at least one forward and backward re- lation holds strongly. The shaded region shows the variation across five hidden lay- ers, and the solid lines show the median of these alignments. At small y, the best alignment is between H, and W W for the forward relation, and G. and WW the backward, in agreement with the theoretical prediction.
v(([)v(()) = VE'(''
(14)
Therefore, the CRH is more likely to be fundamental law of learning as invariant to subjective choice of basis. For this reason, one may also refer to the GWA as an "equivariant NEA" " (eNFA). Furthermore, if we treat NFA as a special case of the forward alignment, the back- ward alignment relations imply a novel variant of the NFA, which can be referred to the forward NFA: WWT oc In this picture, the original NEA should
E [Vn,VVVn, thus be called the backward NFA. See Figure 8 which val- idates both forward and backward eNFA.
8 CONCLUSION
In this work, we propose the Canonical Representation Hypothesis (CRH), a new perspective for studying the formation of representations in neural networks. The CRH suggests that representa- tions align with the weights and gradients after training. It is a generalization of the neural collapse phenomenon for any fully connected layer in a neural network. In this view, representations formed based on the degree and modes of deviation from the CRH. This deviation leads to the Poly- nomial Alignment Hypothesis (PAH), which posits that when the CRH is broken, distinct phases emerge in which the representations, gradients, and weights become polynomial functions of each other. A key future direction is to understand the conditions that lead each phase and how these phases affect the behavior and performance of models. The CRH may also have biological impli- cations as it implies that neural networks tend to learn an orthogonalized representation, which has been observed in the biological brain recently (Sun et al., 2025). The CRH may also have algorith- mie implications. If representations align with the gradients (as in RGA), it might be possible to manually inject noise into neuron gradients to engineer specific structures in the model's represen- tations. However, the CRH and PAH have several limitations. They apply only to fully connected layers, and a future step is extending them to other types of layers. Additionally, we have primarily focused on characterizing the final stage of representation formation. A more comprehensive theory of representation dynamics could lead better training algorithms.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
ILC acknowledges support in part from the Institute for Artificial Intelligence and Fundamental Interactions (IAIFI) through NSF Grant No. PHY-2019786. This work was also supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF 1231216.
REFERENCES
Amina Adadi and Mohammed Berrada. Peeking inside the black-box: survey on explainable artificial intelligence (xai). IEEE access, 0021108-2160,, 2018.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27))e23118788 2024.
Daniel Beaglehole, Adityanarayanan Radhakrishnan, Parthe Pandit, and Mikhail Belkin. Mech- anism of feature learning in convolutional neural networks, 2023. URL https://arxiv:
erg/abs/ 2309. 00570
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 7588):1788-1888, 2013.
MC Booth and Edmund T' Rolls. View-invariant representations of familiar objects by neurons in the inferior temporal visual cortex. Cerebral cortex (New York, NY: 1991), 8(6):510-523 1998.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Albert Einstein. On the theory of the brownian movement. Ann. Phys., 17(549), 1905.
Patrick Esser, Robin Rombach, and Bjorn Ommer. A disentangling invertible interpretation network for explaining latent representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9223-9232, 2020.
Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jachoon Lee, and Jeffrey Penning- ton. Scaling exponents across parameterizations and optimizers, 2024.
Yulu Gan and Tomaso Poggio. For hyperbfs agop is greedy approximation to gradient descent. Technical report, Center for Brains, Minds and Machines (CBMM), 2024.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http:///8kyliim007.gihhhr io/ CoreWeeeeetttorpuus 2019.
Markus Herdin, Nicolai Czink, Hüseyin Ozcelik, and Ernst Bonek. Correlation matrix distance, meaningful measure for evaluation of non- non-stationary mimo channels. In 2005 IEEE 6l Vehicular Technology Conference, volume 1, pp. 136-140 IEEE, 2005.
Sepp Hochreiter.
The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge- Bused Systems, 6(02): 107-116, 1998.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.0799: 2024.
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie Su. An unconstrained sayer-peeled perspective neural collapse. arXiv preprint arXiv:2110.02796, 2021.
Sekitoshi Kanai, Yasuhiro Fujiwara, and Sotetsu Iwamura. Preventing gradient explosions in gated recurrent units. Advances in neural information processing systems, 30, 2017.

---

Published as a conference paper at ICLR 2025
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arX1v:2001.08361, 2020.
Seijin Kobayashi, Yassir Akram, and Johannes Von Oswald. Weight decay induces low-rank atten- tion layers. arXiv preprint urXiv:2410.238119 2024.
Vignesh Kothapalli. Neural collapse: review on modelling principles and generalization. arXiv preprint arXiv:2206.0440 2022.
R. Kubo. The fluctuation dissipptonnnssiiiiiinn theorem. Reports on progress in physics, 29(1)) 1966.
Kangqiao Liu, Liu Ziyin, and Masahito Ueda. Noise and fluctuation of finite learning rate stochastic gradient descent, 2021.
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Uni- versality and individuality in neural dynamics across large populations of recurrent networks. Advances in neural information processing systems, 32, 2019.
Wojeiech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Miloš, and Tomasz Trzcinski. The tunnel effect: Building data representations in deep neural networks. Advances in Neural Information Processing Systems, 36, 2024.
Vardan Papyan. The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size. arXiv preprint 1XXi::8811.07066 2018.
Vardan Papyan, XY Han, and David I. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Andrea Pelissetto and Ettore Vicari. Critical phenomena and renormalizationn-roup theory. Physics Reports, (6867)5599-777,200 2002.
Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Mecha- nism of feature learning in deep fully connected networks and kernel machines that recursively learn features, 2023. URL
/2tprg//aa////2.....a/// 2212. 13881
Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J DiCarlo. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of of-the-art deep artificial neural networks. Journal of Neuroscience, 38(33): 7255-7269, 2018
Akshay Rangamani and Andrzej Banburski-Faiie Fahey. Neural collapse in deep homogeneous classifiers and the role of weight decay. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4243 4247. IEEE, 2022.
Akshay Rangamani, Mengjia Xu, Andrzej Banburski, Qianli Liao, and Tomaso Poggio. Dynamics and neural collapse in deep classifiers trained with the square loss. Center for Brains, Minds and Machines (CBMM) Memo, (117), 2021.
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In International Conference on Machine Learning. pp. 28729 -27455. PMLR, 2023.
Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E Vogt, Randall Balestriero, Wieland Brendel, and David Klindt. Cross-entropy is all you need to invert the data generating process. arXiv preprint 2rXiv:2410.21800 2024.
Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned representations. In International Conference on Machine Learning, pp. 9030-9039. PMLR, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
12

---

Published as a conference paper at ICLR 2025
Weinan Sun, Johan Winnubst, Maanasa Natrajan, Chongxi Lai, Koichiro Kajikawa, Arco Bast, Michalis Michaelos, Rachel Gattoni, Carsen Stringer, Daniel Flickinger, et al. Learning produces orthogonalized state machine in the hippocampus. Nature, pp. 1-11, 2025.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 9(11), 2008.
Loek van Rossem and Andrew M Saxe. When representations align: Universality in representation learning dynamics. arXiv preprint arXiv:2402.09142, 2024.
Liwei Wang. Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. Advances in neural information processing systems, 31, 2018.
Mengjia Xu, Tomer Galanti, Akshay Rangamani, Lorenzo Rosasco, and Tomaso Poggio. The janus effects of sgd vs gd: high noise and low rank. 2023a.
Mengjia Xu, Akshay Rangamani, Qianli Liao, Tomer Galanti, and Tomaso Poggio. Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and gener- alization bounds. Research, 6:0024, 2023b.
Sho Yaida. Fluctuation Fuuctuationdiisiiaaiinn relations for stochastic gradient descent. arXiv preprint arXiv:1810.0000% 2018.
Greg Yang and Edward J Hu. Feature learning in inftnite-width neural networks. arXiv preprint arXiv:2011.14522 2020.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part 1 13, pp. 818-833. Springer, 2014.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pp. 12979-12990. PMLR, 2021.
Liu Ziyin. Symmetry induces structure and constraint of learning. In Forty-first International Con- ference on Machine Learning, 2024.
Liu Ziyin, Mingze Wang, Hongchao Li, and Lei Wu. Loss symmetry and noise equilibrium of stochastic gradient descent, 2024a. URL hatttpss/arxxvv.or///b/////////
07193,
Liu Ziyin, Mingze Wang, Hongchao Li, and Lei Wu. Parameter symmetry and noise equilibrium of stochastic gradient descent. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024b.
Liu Ziyin, Yizhou Xu, Tomaso Poggio, and Isaac Chuang. Parameter symmetry breaking and restoration determines the hierarchical learning in ai systems. arXiv preprint arXiv:2502.05300, 2025.
13

---

Published as a conference paper at ICLR 2025
A MORE RELATED WORKS
The neural representation is central object of study in both Al and neuroscience (Esser et al. 2020 Wang et al. 2018). The widespread use of techniques like t-SNE (Van der Maaten & Hinton 2008 for analyzing the penultimate layer representation in deep learning highlights the importance of understanding representation space. However, the theory of how the representations are formed in neural networks is scarce. A recent work approached this problem from the angle of symmetries Ziyin 2024), showing that permutation symmetries in the latent layers lead to neuron merging and low rankness in the representation. Ziyin et al. (2024b) showed that after training, the latent repre- sentation is aligned with a linear transformation of the prediction residual. A recent work showed that the evolution of the representation during the initial stage of training have universal proper- ties shared by different types of models (van Rossem & Saxe 2024). Empirically, |Rodeer et al.] (202T) and (Huh al.] (2024) showed that the neural representations learned by different networks are essentially similar, which may offer a partial explanation for why the CRH seems to hold across different architectures. Also, an emergent field in neuroscience studies how the representations learned by neural networks closely resemble those of biological neurons (Rajalingham et al. 2018)) These results suggest the existence of some fundamentally shared mechanisms of the formation of representations.
The finding that the neural networks find invariant structures is interesting and should be contrasted with the finding that sometimes the neural networks learns an invertible function (Zimmermann ett al. 2021, Reizinger et al.) 2024). This may have to do with the fact that both neural collapses have been found to emerge when there is nonnegligible strength of regularization (such as weight decay (Rangamani et al.] 202T)). This suggests that some form of simplicity bias is required to make neural networks prefer invariant structures, which may be another interesting topic for future research.
B THEORY AND PROOFS
B. GRADIENT MOMENT IS DOMINATED BY GRADIENT COVARIANCE
We show that when there is a bias term in the layer, the expected neuron gradient must be negligible close to a local minimum:
hbs = W',' = Who + 3,
(15)
At the local minimum, we have
0= [[E[A]] = M,, 7bb)
e] = o [[gb] ] bb),
(16)
which implies that This implies that which is negligible. This agrees with the experimental observation in Section C.2
>b=E[gb] E.g..
(17)
E =
(18)
B.2 PROOF OF THEOREM
We need few lemmas. The following lemmas are quite general and apply arbitrary symmetric matrices A and B, which is clear from the lemma statement. Eventually, when we apply these lemmas, the specific A and will be the H, G, Z matrices we defined in the CRH statement.
Lemma 1. Let A, B be symmetric matrices such that B ABA, then,
ker( () S ker (B) T B²
(19)
and BD ABO is a projection matrix.
Proof. Let n be in the null space of A. We have
Bn ABAn 0.
(20)
14

---

Published as a conference paper at ICLR 2025
Thus, is also in the null space of B. This means that the kernel of A is in the kemel of B. Now,
Let P Bd and D PDP, we have that
B ÄBA,
(21)
where the rank of A is the same as the rank of B. This implies that there is orthonormal matrix O such that
B' OBO", A' OAO
(22)
are full rank, A' is diagonal, and
B'= A'B''
(23)
This implies that
B = a,a
(24)
where is the i-th diagonal term of A'. Because B' is full-rank. We have that for all i,
0;
(25)
This implies that A'A'O is orthogonal projection matrix. Because it also has the same kernel as B, we have
Ā = B
(26)
The following two lemmas are straightforward to prove by multiplying A from the left and right
Lemma 2. Let A, B be symmetric matrices such that A ABA, then,
A BA
(27)
Lemma 3. Let A, B symmetric matrices such that ABA, then,
A A'=A'BAA
(28)
Lemma 4. ke) = lekkB)),)..... (B), then kor(CACT)) ker (CBC any symmetric and and arbitrary matrix C.
Proof. For n to be in the null space of CACT,, it must satisfy one of the following two conditions:
C=n=00
(29)
which implies that ker(CBB)), Or
VAC.==
(30)
which implies that CT n ker (VA) ker(A) ker( ker(B), which again implies that
ker(CBBC)
(31)
This finishes the proof.
Now, we are ready prove Theorem
Proof. (Part 1) Fix the direction to be forward or backward. Let A, B, C. be permutation of three moment matrices. Then, that two alignments hold implies that
A x B.
(32)
and oc C. By transitivity, A xc B oc C. This proves part 1.
(Part 2) There are many combinations, which be divided into few types. We thus prove prototype for each type of relation, and the rest can be derived in similar but tedious manner. The key mechanism that every forward relation implies backward relation and vice versa. For example, if we know E [[hgh]] xc [[ we also have
WW oc [[
(33)
*Recall that A is the pseudo inverse, and Ag AA- is orthogonal projector.
15

---

Published as a conference paper at ICLR 2025
which implies that
W W W W' E[
(34)
Now, let A be either W W WWT, B, C be some permutation of the gradient and neuron covari- ance.
Type O: this is the simplest and most straightforward type of relations. First, consider relation 5 in the table. We have, by assumption,
Hg Za WH,W -b oc Gb.
(35)
(36)
Together, we have
W WHAW W..W Z2g.
(37)
So,
za x GGW = G..
(38)
This means that
Ha Ze oc Ga.
(39)
Also, this implies that
H, x Z3 <c Gb-
(40)
similar proof derives relation is thus not shown.
Type 1: ABA ACA B C. Using the above lemmas and defining D == PDP, where
P= B" = = we obtain
-= B CO.
(41)
Type 2: ABA x A x ACA. This type of equation simply solves to
B x A x Č
(42)
where the tilde defined with respect to A
Type 3: B x A x ACA. By the above lemmas, we have
A- x Č.
(43)
We thus have
Ax
(44)
Type 4: ABA x A2 x C². This implies that
B x A CO.
(45)
where tilde is defined with respect to P = A Type 5: This type is little strange, and so explicitly solve them. There are two cases:
E[gagh] oc 7m oc 72 E[9og]] oc Z, oc Z3
(46)
(47)
This directly implies that Z. and Zo are projection matrices. For E[hh"], we have that by definition
E [h oc WE[hah.]WT.
(48)
This means that the kernel of [[ must contain the kernel of WWT Za. Therefore, letting P [o
Bhigh]] P7, PR[g6g]]P.
(49)
Likewise, have that
WE[hah.JWT [.
(50)
and so
ker WB[hhhh ker ker(B((hhh))) ker
(51)
This implies that
ker Z, (.ahhhhh kerz.-
(52)
16

---

Published as a conference paper at ICLR 2025
This means that have proved
E| h1 Z% ][
(53)
similar derivation applies to the when
El Za Zz
(54)
B[ Zb Z2,
(55)
(Part 3) By the pigeonhole principle, two of either the forward or backward relations must be sat- isfied. This means that by part 2 of the theorem, there two cases: (a) three forward relations satisfied, (b) three backward relations are satisfied. Since the argument is symmetric in forward and backward directions, we focus case (a).
Case (a). We have
B[ Z, E[g.9h] W =[
(56)
There are now three subcases depending on which of the backward relations are satisfied.
Case (al): Zn ]. which implies that
Za [ W WTZ,W Zz
(57)
and so all the three forward matrices are (scalar multiples of) projections.
But W W and WWT share eigenvalues and Zo [[goo]]] also projection matrices. Now,
E[ JW W WZ, WT 2
(58)
which is also a projection matrix. This, in turn, implies that all three backward relations hold.
Case (a2): [[ [[ This implies that
z W WT
WB(hhhhJJT B[hah]] E[
(59)
In turn, this implies that
Zu E [[ga9.]
W E[gb9] W z2W Z
(60)
This implies that Zo only contains zero and as eigenvalues, which implies that Z. Z2 Z is a projection. Similarly, Zo is a projection. Together, this implies that all six relations hold.
Case (a3): 7, [[hah]]] This case is subtly different. Here,
B[hgh]] WE[",h.JW* WT.WT 23 Zn.
(61)
and so all three forward matrices are projections.
For the backward relations,
Z, x E[hgh WET x 2z.
(62)
which is also a projection matrix. Now, we have essentially no relation for [[gng]]] as it cannot be directly derived from any other quantity. But noting that
W W'E["o = E[gug]] oc Zn.
(63)
one obtains that
PE[
(64)
where Zg. Therefore, when restricted to the subspace of W, B[gbg]] is also a projection matrix.
For case (b), it is the same, except for the case when Z. E[gayh]] When this is the case, the alignment happens with
PE[
(65)
for P = Z
At local minimum. At any local minimum first-order stationary point, Lemma Saammaaappiln applies, which implies that
ker( 7 s ker (ee [h))))))
(66)
17

---

Published as a conference paper at ICLR 2025
ker [c) ker [ [[ D)
(67)
ce {a, 6). This because and for any n ker(E[(hhh.)) holds with probability one that and
Ze B[gcht],
(68)
n====0 0,
(69)
Zen B[hoge]] O.
(70)
The same argument applies to 9c- Now, it suffices to prove the two subtle cases. For case (a3), we have that PE[nog] P.
(71)
However, the local minimum condition implies that
[9 ker(P).
(72)
This can only happen if P [[gog]]] P. Therefore, all six relations hold. The same applies to the other subtle case.
(Part We have
E h x Ze oc E[ga9.] W B[gg,, E[hgh]]] = W 7, E-
(73)
(74)
Plugging the forward relation into the backward relation, we obtain that
W 7 Zb;
(75)
which implies that Z& is proportional to a projection matrix. This implies that [[hgh]]] and E also projection matrices.
9b9,
Likewise, one can plug the backward relation into the forward, which implies that
7 Za:
(76)
and is thus proportional to a projection matrix. This completes the proof.
B.3 PROOF THEOREM
The proof will be divided into several theorems, each of which proves a claimed relation in Section The following theorem proves Eq, 6
Theorem 4. Assume assumption
7 [[E 0 and H[] = 0. then,
0 2, [gth]]
(77)
where 23
Proof. Let Bg = h- and B, hghh. Let H, B[B;] and H, R B.] We consider an online learning setting. Let æ denote the input at time step t. Consider the time evolution of the feature covariance during the SGD training:
A(ho(z)h,, (=)) A(Wh, WWT
(78)
A.W.WW" W WT AWB,AWT + +((Ba)
(79)
=-n(Vag Th + +W) 7 (a Vhg (TT+ +77 th +7W)+
>WT)
(80)
= =-n(-1ha 96hg + -BB) "((hhal?99hh, + hh, (hag] 2 B.) 11² I hhlll 1999696-
- 77(-11/aal) 22000 + BB) + 2721 haal 9697 + 0(12))
(81)
(82)
Taking expectation of both sides, obtain that = 2 + E[hhog n2? E[966] Ho.
(83)
where -E|ba
18

---

Published as a conference paper at ICLR 2025
Similarly, one show that the pre-activations follow similar relationship.
Theorem 5. Assume assumption 7
[G and E AGb] then,
Za [a [[gga)]) + B1 ()=
Ey
(84)
where we have defined co = [=[bbb]11]]
Proof. We have
Aggaga) = A(W W)
(85)
= -W Mos W W gb96 AW + AW 9696 AW + -(1Agng)
(86)
= =Z(hha - (9a9 + 7 haa (9ag.) + +1 hahh O()²r),
(87)
where have used the relations Or W gb and AW nngshi 7W). We have also defined Za = 199 2 ]]
Taking expectation, this leads
2a [ hh [ [[ggh + B =2yB[9.9.]
(88)
The following lemma proves the balance condition at local minimum.
Lemma 5. (First-order stationary point condition / local minimum balance.) At any stationary point of the loss function, we have
E [ [[ght] WWW
(89)
E3 [[ggh = W
(90)
Proof. Close stationary point, have
B[Aw] [[[ -
(91)
Multiplying W from the left and the right, we obtain, respectively:
[[ggh W wWW = E[ggh[]] - WWW
(92)
0 = W W' W W = [[ghh]] > W, where we have used the definition h4 Wha and the chain rule 9a WT gb. This condition implies that
(93)
E [g -[ WWWT
(94)
E : W
(95)
L
Now, we derive the stationarity condition for AWWT and AWTW
Lemma 6. (Stationary alignment of parameter-outer productt ) If at a local minimum and assuming Assumption 7 then,
of [[(AWW"))]=
E. WWT
(96)
2. (E[A(WTWW]]= O,
1.hhhh)) W
(97)
19

---

Published as a conference paper at ICLR 2025
Proof. We have
0= "/WW:/))JE/WWWWWWWLWWVWV.
(98)
=0+E]AWAW"]
(99)
=E[(ghh -
(100)
- - -BB[ghh]]" + w
(101)
2.15[9991 > WW
(102)
where we have used Eq. (89 in the last line.
Likewise, when E AWWT W] 0,
0 FEE = (9 - 7 [W)
(103)
z. b -B -1B(haaa1 +
(104)
=z. 2
(105)
where we have defined 2a 19 13].
Now, we are ready prove Theorem !
Proof. The above two lemmas imply that
2zh
E [gog.]] [[ E[goh]]'
(106)
This relation can be substituted into Eq. 6 to obtain
2 + 2) 2y Hb 20 2WT
(107)
Likewise,
2= E hhahh B[hag]]+ 1
(108)
This relation can be plugged into Eq.
84
to obtain that
(m + h 2y E[ga9]]] W..
(109)
This completes the proof.
B.4 PROOF OF PROPOSITION
Proof. We first prove that part (1) implies part (2). By assumption, we have that B[hh"] or E[gg J.
and sc
E[hh" Jú E[gg"]; Já
(110)
[[V
(111)
= 0,
(112)
where the last line follows from the fact that
l(f(h + - = =
v =(])
(113)
which is possible only if â [V definition of CRH: B[hh"] E[gg" x Z.
One can derive the other two relations simply using the
For the backward direction, we have that
Eihh Jn [[[g" [[
] 0.
(114)
NNte that the proof still works if we replace the second moments with the covariances.
20

---

Published as a conference paper at ICLR 2025
This implies that n" 0 with probability 1. Now,
{(f(h+ +n)) ((f()) = g ( =((2).
(115)
(116)
The proof is complete.
B.5 PROOF THEOREM
As NC4 is trivial consequence of NC1-3, we focus NC1-3 here. For notational simplicity, state neural collapse in the case when there is no bias in the last layer. The first three properties are defined at the end of training
1. NCI: h(x.)) He. where Te is any data point in class c;
2. NC2: plpe = dee'
3. NC3: WTW Ee Matle
Proof. We first prove that NC1-4 implies the CRH. When NCI holds,
he (xee = He.
(117)
This means that
Ha x EuolE.
(118)
By NC2, we have
Mole Sed. This means that Hg is proportional to an orthogonal projection. By NC3, have that
(119)
WWW x ΣnoHE.
(120)
Additionally,
G. G.W WWW,
(121)
where we have used the assumption that G B[V [[
Together, this implies the backward CRH
Ha oc Ga x Za-
(122)
For the forward CRH, because WWT CxC has rank C and they have equal eigenvalues, must be proportional to identity:
2n I.
(123)
By the interpolation assumption, also have
E[gghT]] x 1 I.
(124)
Therefore, we have proved the forward CRH. This proves one direction of the theorem.
Now, we prove that the CRH implies NC1-NC3, We first prove NC1. By the (backward) alignment hypothesis for the last layer, we have
H. G. W G&W WTW,
(125)
where W is the weight matrix for the last layer. This means that there exists an orthonormal matrix U such that
W x H and that rank of H2 must be C. By the interpolation hypothesis, must the case that
(126)
Wh 1
(127)
21

---

Published as a conference paper at ICLR 2025
which implies that for fixed c. This implies that
he Zc
(128)
where 2e a constant vector and We = However, by Proposition
[ v must vanish, and so
he Ze Pe- This proves NC1.
This means that
Za Ha
(129)
is essentially projection matrix. So.
Wha Wha = UHaha( = (T) -c.
(130)
This implics that in turn, ha (x.) = Me = Up. This proves NC3. Due to the orthogonality of U, we have that
po = Ue ded.
(131)
This proves NC2. The proof is complete.
22

---

Published as a conference paper at ICLR 2025
C EXPERIMENTS
C.I EXPERIMENTAL DETAILS
fel: Fully connected neural networks trained on a synthetic dataset that we generated using two-layer teacher network. This experiment is used for controlled study of the effect of different hyperparameters. To control the variables, we consider a synthetic task where the input x IR R0000 is sampled from an isotropic Gaussian distribution, and the label generated by nonlinear function of the form: y(x) = Σi=1 uj sin((W;)) z + b() é R, where u", wi and b are fixed variables drawn from Gaussian distribution. In this form, the target function can be seen as a two-layer network with sin activation. Our model is fully connected neural network trained with SGD in an online fashion, and the representations are computed with unseen data points. Unless specified to be the independent variable, the controlled variables of the experiment are: depth of the network (D 4), the width of the network (d- 100), weight decay strength (7 = x 10-5), minibatch size (B = 100).
fc2: Here, choose a high-dimensional setting where the teacher net is given by y(x) E u, sin((W)) + b)) R. where un R2000 w IR 10000 and bg I are fixed variables drawn from a Gaussian distribution. In this form, the target function can be seen as a two-layer network with sin activation. The distribution of controlled by an independent variable &r such that x' ~ (100,, and æ = (1 0z) + oz 1100. where Z is a zero-one matrix generated by Bernoulli distribution with probability 0.8. When Φ. is small, the input features are thus highly cor- related to each other, and the covariance matrix deviates far from J. The training proceeds with SGD with a learning rate of 0.1 with momentum 0.9 and 7 = 10-4 for 10 steps when the loss function value has stopped decreasing. The training proceeds with batch size of 100. All the expectation and covariance matrices are estimated using 3000 independently sampled data points. The trained model is 5-hidden layer fully connected network with the ReLU activation.
resl: ResNet-18 18 (11M parameters) for CIFAR-101 10; we apply the standard data augmentation tech- niques and train with SGD with learning rate 0.01, momentum 0.9, cosine annealing for 200 epochs, and batch size 128. The model has four convolutional blocks followed by two fully con- nected layers with ReLU activations. The model has 11M parameters and achieves 94% test accu- racy after training. in agreement with the standard off-the-shelf ResNet- 18 for the dataset.
ResNet- 18 for self-supervised learning tasks with the CIFAR-L0/100 datasets. The model is the same resl, except that the last fc layer output becomes 128-dimensional, which is known as the projection dimension in SSL. The training follows the default procedure in the original paper (Chn et al. 2020 proceeding with batch size of 512 and 7 5 10 for 1000 epochs.
res2:
llm: a six-layer transformer (LOOM parameters) trained on the OpenWebText (OWT) dataset (Gokaslan & Cohen 2019); the number of parameters of this model matches the smallest version of GPT2. The model has six layers with eight heads per layer, having 100M trainable parameters in total. For each experiment, we train with Adam with a weight decay strength of x 10-4 for 10 iterations, when the training loss stops changes significantly. Since every representation has three dimensions: data N, token T, and feature F, we treat each token as they are a separate sample in computing the covariances. Namely, contract the representation tensor along the data and token dimension, resulting in x F covariance matrix.
23

---

Published as a conference paper at ICLR 2025
1.0
1.0
1.0
0.3
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
50000
100000
50000
100000
50000
100000
Iteration
iteration
Iteration
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
crass head
init end
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
50000
100000
50000
100000
100000
iteration
iteration
iteration
Figure 9: Alignment of the matrices B[hh"] and B[gg"] The experimental setting is the same as the L.M experiment.
1.0
0.8
0.4
0.2
50000
100000
itcration
Figure 10: The ratios of the traces of the second moment matrices for transformer: Tg We see that rg essentially con- verges to zero, which means that E[gg"] ] cov cov(g.g g) at the end of training. ra is generally non-zero but essentially negligible. The experiment setting is the same as the LLM experiments.
[[TT and rg [E
C.2 SECOND MOMENT ALIGNMENTS
This section shows the results for the alignment of the matrices E[hh"" and E[gg"]. See Figure The results are qualitatively similar the result for the alignment between cov(h, h) and cov but with larger variation. The reason for the similarity is that it is often the case that the covariance term dominates the second moments the end of training. See Figure
24

---

Published as a conference paper at ICLR 2025
covig, g)
covig. h)
covth,
neuron index
neuron index
neuron index
covíg. g)
covíg. h)
cov(h, h)
6-
4-
2-
0
6
index
Figure 11: Representation of the output layer of Resnet18. Essentially, these are the covariances of the output at the end of training. First Row: Initialization. Second Row: End of training.
cov(g.g)
covig.h)
cov(h,
40
40
40
30
30
30
20
40
neuron Index
neuron Index
neuron Index
covlg.g)
covíg.h)
cov(h.h)
40
40
40
30
30
30
20
20
20
10
10
10
0
20
40
20
40
20
Index
Figure 12: Representation covariance of the last convolution block at initialization (upper) and end of training (lower).
C.3 REPRESENTATIONS RESNET18
See Figure 12 for the representations of the last convolutional layer of Resnet18 before and after training on CIFAR-IO. See Figure for for the representations of the output layer. Interestingly, for the classification task, both cov(g, (g. (g.g) and cov(h, h) become proportional to the identity.
25

---

Published as a conference paper at ICLR 2025
covíg. g)
cov(h, h)
covlg. g)
cov(h,
40
40
40
40
30
30
30
20
0
20
40
40
20
40
20
40
Index
index
neuron index
Figure 13: Examples of representation for Resnet- 18 after self-supervised contrastive training. Left: second convolution block representation, Right: penultimate convolution block representa- tion.
covlg.g)
covth,
covig.g)
10a
10a
20
10
10
20
40
40
25 50
reuron index
neuron index
eigenvalue index
Figure 14: Examples of the representation learned by the transformer the third layer. Left-Mid: examples. Right: The spectra of the two matrices exactly the same for the leading eigenvalues. The difference is mainly in the smaller eigenvalues, and this difference gets smaller as the training proceeds.
1.0
4ga
0.4
Inf end
random
100000
Figure 15: The alignment of feature and gradient covariance (hkg remains high during most of the training (Ilm). The shaded region shows the variation across 8 different heads in the same layer.
C.4 REPRESENTATIONS SELF-SUPERVISED LEARNING
See Figure 13 for the representations of the last and the penultimate convolutional layers. They have significant alignments, but the agreement is perfect. For fully connected layers, the alignment is much better (see the main text; examples not shown).
C.5
LARGE LANGUAGE MODEL
See Figure 14 and 15
26

---

Published as a conference paper at ICLR 2025
C.6 FULLY CONNECTED NETS
See Figure 16 and 17 We see that the alignment effect is significant for both SGD and Adam. Also,
Figure 18 for the effect of having different depths.
1.-
1.
1.0
0.8
0.8
oyer
oyer
layer
layer
Wyye
layer 2
0ys1
02
0.2
lyyer
0.2
layer
0.0
0.0
Figure 16: The effect of the width, weight decay 7, and batch size on the alignment for fully connected network. The training proceeds for 10a iterations, when the training stops decreasing significantly. The training proceeds with SGD.
1.0
0.8
0.8
0.8
$
0.6
0.6
Byye
byye 1
layer
layer
layar
layar
layer
layer 5
layer
0.0
0.0
0.0
withh
batch size
Figure 17: Same as the previous figure, except that the training proceeds with Adam.
1.0
0.8
Spaih
0.6
0.6
0.4
0.4
mp.m
0.0
0.0
2
101112
layer
leyer
Figure 18: Alignment of of different layers of fully connected ReLU network at different layers (layer is the input layer). Left: 6-layer network. Right: 12-layer network.
27

---

Published as a conference paper at ICLR 2025
0.8
0.6
0g.t
0.5
0.6
0.6
8.
0.4
0.4
0.4
Rap
0.
50
100
150
200
50
100
150
200
50
100
150
200
50
100
150
200
aooth
apech
opoch
opoch
Figure 19: Evolution of during training for different layers of Resnet-18 on CIFAR-IO. For refer- ence, the training accuracy (grey) and testing accuracy (black) are shown in the dashed line. Left to Right: (1) penultimate convolution block representation, (2) last convolution block representation, (3) penultimate fully connected layer, (4) output layer.
C.7 CRH IN RESNET-18
See Figure I
C.8
CRH AND PAH IN FULLY CONNECTED NETS
See Figure 20 The task is the same as other fully connected net experiments. The model a 4-hidden-layy layer tanh net with the same width.
Ge
My
Go
H
100
Figure 20: The alignment scalings in fully connected nets. The dashed lines show power laws with exponents 1/3, 1/2, 1, 2, respectively.
C.9 STATIONARITY OF THE COVARIANCE MATRIX
See Figure 21 for the evolution of Acov ft, h) to zero for three convolutional layers (0-2) and the two fully connected layers (3-4) in Resnet-18 during training.
layer
300
layer
layer
JAcov(h,
200
layer
layer
100
100
200
epoch
Figure 21: The change in the representation covariance converges to (near) zero at the end of training. The model is Resnet-18 18 trained CIFAR-10 10 with standard SGD.
C.10
FULL FIGURE TO FIGURE 4
See Figure 22
28

---

Published as a conference paper at ICLR 2025
10a
rank deficiency
0.000001
0.000003
10-1
0.000008
0,000022
0,000060
0.000167
10"2
10
1
Figure 22: This is the same figure as Figure The legend shows the weight decay value used for each experiment. Dots with the same color come from the five different hidden layers for the model trained at this specific weight decay value.
D MEANING OF THE PAH
D.1 LINEAR REGRESSION AND PHASES THE PAH
In some sense, the phases of the PAH already appear implicitly in many standard models. Consider a linear regression problem: W min E| y1
(132)
where E denotes averaging over the training set. Its solution is well known:
(W*)*= = =[xx''] -[=="'><<<<
(133)
From the perspective of the PAH, this solution can be seen as composition of two functional layers: 4 V1 V2, where the first layer is B[xx"]-1//² and the second layer is V2 E[äy], and ã = E[ [xxx 1-1/2
The first layer normalizes the input distribution and is apparently related to the PAH because one can identify
H. [xxx'']
(134)
as the input representation to the layer, and so
B(rx" (W*)TW"* Za.
(135)
which implies that
H = Za.
(136)
which can be identified as the 7th phase of the PAH in Table
D.2 POTENTIAL IMPLICATIONS OF THE PAH
Here, discuss some possible and interesting meanings of the phases of the PAH. Validating these intuitions could be of great interest.
One way the phase could imply, as suggested in the manuscript, is that positive relation be- tween Ha and Z, could imply a low-rank mechanism, where larger eigenvalues of H., are expanded while smaller eigenvalues are suppressed. The fact that neural networks learn these low-rank rep- resentations is consistent with common observation. For example, see Kobayashi et al. (024). Another implied mechanism is representation normalization. This happens when H Z which means that H₁ will be normalized after this weight. See the discussion about linear regression in Section D.I for how this phase exists implicitly, even in linear regression.
Another kind of interesting phase is where the alignment condition implies gradient vanishing or ex- plosion problems, which have been well-known for decades (Kanai et al. 20177] Hochreiter, 1998). Noticing that 9a W gb. it is naturally the case that if Zi, and Go are aligned with positive expo- nent, the gradient in the previous layer will face an explosion and vanishing problem simultaneously (as larger eigenvalues become larger). In contrast, the phase Z, oc G seems to be the ideal training phase as implies that the gradient the previous will be normalized. Identifying the causes of these phases is also of great future interest.
29


Reviews:
[{"review_id": "o0KNFJerzf", "reviewer": "Reviewer_LWzQ", "summary": "First of all, I apologize for the late review with the Authors, the AC, Senior AC and the PC.\n\nThis paper introduce the Canonical Representation Hypothesis (CRH), positing that in neural networks, as an effect of training, the representations, weights, and gradients (with respect to the representations) within a layer align with each other. \n\nThis alignment is seen as an essential part of how neural networks form structured, invariant representations. \nThe Authors introduce the CRH and support it with a theory that shows how this alignment emerge as a consequence of stationarity towards the end of training, through a balance of gradient noise and regularization.   \n\nAdditionally, they propose the Polynomial Alignment Hypothesis (PAH), which describes patterns in cases where CRH is broken, leading to specific scaling relationships in neural networks, of a different nature of scaling laws involving network performances.\n\nThis framework offers a new perspective on understanding important aspects of neural representations, unifying phenomena such as neural collapse and feature learning.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The CRH provides a new framework to interpret basic statistical aspects of how neural networks develop representations. The system of CRH equations generalize and organize into a coherent view previous, partial attempts to reveal alignment phenomena between representations, gradients and weights.  By relating CRH to neural collapse, the Authors offer a broader view that positions this phenomenon as a specific case of a more universal behavior in neural networks. \n\nThe framework is very general, focusing on alignment rather than on specific network designs or loss functions. This  increases the potential impact of CRH on future studies, as it provides a general model for understanding basilar statistical aspects of representation formation.\n\nThe Authors present a detailed theoretical derivation supporting the CRH. The theoretical derivations adds credibility to their claims, encouraging further studies, in particular related to dynamics.", "weaknesses": "While this work aims at universality, the Authors are the first to recognize that their focus is on specific types of networks and tasks, commenting that further empirical tests across more diverse datasets and architectures are necessary to confirm the CRH and PAH in practice.\n\nThe weakness I found is that certain assumptions - which are critical for the theoretical foundation of CRH - may not hold in general, like the self-averaging conditions (Hypothesis A.1, A.2). I think the work woud benefit for an empirical assessment on how severe is the departure from self-averaging in a concrete, relevant case.\n\nFurthermore, I found particularly difficult to follow Section B (THEORY AND PROOFS) of the Supplementary Materials. For example, I could not figure out, Lemma 1 in Section B, mainly because some of the \"actors\" in the scene (like the A tilde in eq. 21) were not presented first. But it may be also a lack of understanding from my side.\n\nHowever, adding a few more explanations in this section would improve readibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uJWIcbQXCn", "reviewer": "Reviewer_84cQ", "summary": "The authors propose an overarching theory for neural network learning which encompasses the alignment of representations, gradients, and weights throughout training. This framework uniquely combines a diverse set past work under a unified 'canonical representation hypothesis' (CRH) umbrella, and prove theoretically that the CRH does hold. Succinctly, they show that gradient nose 'expands the representation' and weight decay 'contracts it', and they must reach stationarity by balancing these terms. They then use this framework to make predictions about neural network training when each of the six elements of the CRH is broken, provided a set of 64 potential phases for neural network training.  \n\nIn conclusion,the paper is well written, organized, and situated in the literature. While I am not an expert in this domain, I therefore defer to other more informed reviewers to make statements about the novelty and accuracy of the authors' claims. Otherwise, the results appear impressive and impactful for the theoretical neural network community, and thus carry significant value.", "questions": "- Can you explain the limitation of the theory exposed by Figure 7? Why do you believe that you see the three branches as opposed to one?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written, and carefully presented to ensure correctness and clarity.\n- The framework appears general, encompassing significant past work. \n- The theorems are general, avoiding reference to any specific loss function or activation functions, helping to explain their ubiquity. \n- The experimental results appear to strongly support the developed theory and appear both rigorous and extensive.\n- The insights section is very much welcomed, and provides significant value to readers coming from outside the primary domain. \n- The limitations are clearly stated.", "weaknesses": "- The authors state that the CRH holds better for later layers than earlier layers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8rlMBfIXLD", "reviewer": "Reviewer_vsHn", "summary": "The paper introduces six alignment relations to quantify and analyze model representations, collectively termed the Canonical Representation Hypothesis (CRH). This hypothesis proposes that neural networks naturally develop compact representations, with neurons and weights remaining unchanged by transformations that do not affect the task. The study theoretically examines conditions under which CRH holds and demonstrates that when CRH is disrupted, reciprocal power-law relationships arise among representations, weights, and gradients—a phenomenon referred to as the Polynomial Alignment Hypothesis (PAH). Together, CRH and PAH provide a potential unified theory that encompasses several significant phenomena in representation learning, such as Neural Collapse and the Neural Feature Ansatz.", "questions": "- How might different initialization strategies impact the different observed alignment phenomena?\n- Could the authors elaborate on how the phases correspond to observable behavioural or performance changes in the model? \n- Beyond the two mechanisms identified in the paper, could other factors contribute to the disruption of alignment? \n- How might the different alignments discussed in this paper relate to concepts from the rich and lazy learning literature? My understanding is that the neural feature ansatz describes and quantifies the rich regime by analyzing feature evolution. Could these alignments and phases potentially provide a way to further describe and quantify distinct learning regimes ( Specifically further define the different types of rich regimes)?\n\nI hope this review will be helpful in the further development of the paper. I encourage the author to continue this research and incorporate the feedback provided. With improvements in structure, clarification of key concepts, and development of the practical significance of these hypotheses, this paper has the potential to earn a higher score.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "First of all, congratulations on your ICLR submission! I’d like to commend the author for this great piece of research. Below is my review of the work:\n\n**Originality**  \nThis paper introduces a unifying framework for studying neural network representations, with a notable emphasis on backward representation—a relatively unexplored area. \n\n**Quality**  \nThe paper is of generally high quality:\n- It acknowledges its own limitations effectively.\n- Experimental results are detailed and well-aligned with theoretical insights.\n- Findings show that the theory holds across different types of networks.\n- Two Mechanisms that disrupt alignment are identified. Quantification is provided via the Polynomial Alignment Hypothesis.\n\n**Clarity**  \nThe paper is clearly written in good English and is well-structured. Figures are clear, informative, and effectively support the content.\n\n**Significance**  \nThis paper offers a potentially unifying framework for studying neural network representations and identifying key alignment metrics that inform network alignment. It contributes to an active research area and presents predictions relevant to widely used models like ResNet and LLMs, enhancing its impact in the field. Furthermore, this research could have potential interdisciplinary applications such as neurosciences.", "weaknesses": "**Originality**\nFurther clarifying how this work uniquely contributes to and advances the field would enhance its perceived originality.\n\n**Quality**\nNo clear weakness in the quality.\n\n**Clarity**\n-  The term \"hypothesis\" may be somewhat misleading, as the proposed concept functions more as a measurement framework rather than a traditional hypothesis. Reframing it as such would emphasize its role in quantifying neural network representations without introducing potentially confusing terminology.\n- The organization of the paper could benefit from refinement. Early references to later sections (such as mentioning Figure 9 before the relevant context is provided) disrupt the flow, and frequent shifts in reference to the later text and figures make the argument more challenging to follow.\n- Providing more detailed explanations for terms such as \"neural collapse\" and \"alignment\" would enhance the paper's accessibility.\n- Figure 4 lacks a colour bar for the weight decay parameter, which complicates the interpretation of results.\n\n**Significance**\nTo improve the paper’s significance, it would be helpful to expand on the implications of the different alignment phases. For instance, as mentioned in the text, positive exponents between Za and Ha suggest that the layer enhances the principal components of Ha while diminishing lesser features, whereas a negative exponent indicates the opposite. Greater detail on these implications would improve interpretability, especially regarding how these phases contribute to understanding network behaviour. Additionally, exploring factors that lead to alignment breakdown would deepen the paper's contribution to interpretability in this area.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "keAX5eYihH", "reviewer": "Reviewer_FkCt", "summary": "The paper proposes and studies the 'canonical representation hypothesis'. They study the evolution of weights, representations and gradients in neural networks and establish that these quantities become aligned. The paper includes theoretical derivations and experimental measurements that study how the CRH is fulfilled or broken.", "questions": "23: emergence + of\n\n44: Furthermore, ... - this sentence makes no sense without more context\n\n49: How does the CRH have anything to do with 'interpretable' solutions?\n\nRelated work, you might want to include: https://arxiv.org/abs/2007.00810\n\n68: weight matrices evolve during training? This seems rather tautological, what is the point?\n\n117: missing verb?\n\n149: In practice, it is often not the case that stationarity is reached. Comment?\n\n159-160: Can you explain?\n\n217: Neural collapse / invariance to irrelevant features seems to be at odds with these works (https://proceedings.mlr.press/v139/zimmermann21a.html; https://arxiv.org/abs/2410.21869). Comment?\n\n232: Why is this 'surprising' if it is exactly what the theory predicts?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper makes an effort to be thorough in examining all possible combinations of the forward and backward alignment relations. Moreover, they include a significant number of theoretical results which distinguishes this paper from most work in machine learning.", "weaknesses": "Why should we be interested in the alignment of weights, gradients and representations? How does this contribute to the bigger picture of understanding why deep neural network work. If the authors provided a clear motivation there would be much more incentive to dig through the notation and derivations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)