Title: Neural (Tangent Kernel) Collapse
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural Collapse

Neural Tangent Kernel

NTK alignment

Local Elasticity

Gradient Flow
Abstract: This work bridges two important concepts: the Neural Tangent Kernel (NTK), which captures the evolution of deep neural networks (DNNs) during training, and the Neural Collapse (NC) phenomenon, which refers to the emergence of symmetry and structure in the last-layer features of well-trained classification DNNs. We adopt the natural assumption that the empirical NTK develops a block structure aligned with the class labels, i.e., samples within the same class have stronger correlations than samples from different classes. Under this assumption, we derive the dynamics of DNNs trained with mean squared (MSE) loss and break them into interpretable phases. Moreover, we identify an invariant that captures the essence of the dynamics, and use it to prove the emergence of NC in DNNs with block-structured NTK. We provide large-scale numerical experiments on three common DNN architectures and three benchmark datasets to support our theory.

Neural (Tangent Kernel) Collapse

Abstract

This work bridges two important concepts: the Neural Tangent Kernel (NTK), which captures the evolution of deep neural networks (DNNs) during training, and the Neural Collapse (NC) phenomenon, which refers to the emergence of symmetry and structure in the last-layer features of well-trained classification DNNs. We adopt the natural assumption that the empirical NTK develops a block structure aligned with the class labels, i.e., samples within the same class have stronger correlations than samples from different classes. Under this assumption, we derive the dynamics of DNNs trained with mean squared (MSE) loss and break them into interpretable phases. Moreover, we identify an invariant that captures the essence of the dynamics, and use it to prove the emergence of NC in DNNs with block-structured NTK. We provide large-scale numerical experiments on three common DNN architectures and three benchmark datasets to support our theory.

1 Introduction

Deep Neural Networks (DNNs) are advancing the state of the art in many real-life applications, ranging from image classification to machine translation. Yet, there is no comprehensive theory that can explain a multitude of empirical phenomena observed in DNNs. In this work, we provide a theoretical connection between two such empirical phenomena, prominent in modern DNNs: Neural Collapse (NC) and Neural Tangent Kernel (NTK) alignment .

Neural Collapse. NC [39] emerges while training modern classification DNNs past zero error to further minimize the loss. During NC, the class means of the DNN's last-layer features form a symmetric structure with maximal separation angle, while the features of each individual sample collapse to their class means. This simple structure of the feature vectors appears favourable for generalization and robustness in the literature [12, 31, 40, 47]. Though NC is common in modern DNNs, explaining the mechanisms behind its emergence is challenging, since the complex non-linear training dynamics of DNNs evade analytical treatment.

Neural Tangent Kernel. The NTK [30] describes the gradient descent dynamics of DNNs in the function space, which provides a dual perspective to DNNs' evolution in the parameters space. This perspective allows to study the dynamics of DNNs analytically in the infinite-width limit, where the NTK is constant during training [30]. Hence, theoretical works often rely on the infinite-width NTK to analyze generalization of DNNs [1, 20, 28, 49]. However, multiple authors have argued that the infinite-width limit does not fully reflect the behaviour of realistic DNNs [2, 10, 22, 27, 36, 43], since constant NTK implies that no feature learning occurs during DNNs training.

NTK Alignment. While the infinite-width NTK is label-agnostic and does not change during training, the empirical NTK rapidly aligns with the target function in the early stages of training [5, 7, 44, 45]. In the context of classification, this manifests itself as the emergence of a block structure

in the kernel matrix, where the correlations between samples from the same class are stronger than between samples from different classes. The NTK alignment implies the so-called local elasticity of DNNs' training dynamics, i.e., samples from one class have little impact on samples from other classes in Stochastic Gradient Descent (SGD) updates [23]. Several recent works have also linked the local elasticity of training dynamics to the emergence of NC [33, 53]. This brings us to the main question of this paper: Is there a connection between NTK alignment and neural collapse?

Contribution. In this work, we consider a model of NTK alignment, where the kernel has a block structure , i.e., it takes only three distinct values: an inter-class value, an intra-class value and a diagonal value. We describe this model in Section 3. Within the model, we establish the connection between NTK alignment and NC, and identify the conditions under which NC occurs. Our main contributions are as follows:

We derive and analyze the training dynamics of DNNs with MSE loss and block-structured NTK in Section 4. We identify three distinct convergence rates in the dynamics, which correspond to three components of the training error: error of the global mean, of the class means, and of each individual sample. These components play a key role in the dynamics.

Weshow that NC emerges in DNNs with block-structured NTK under additional assumptions in Section 5.3. To the best of our knowledge, this is the first work to connect NTK alignment and NC. While previous contributions rely on the unconstrained features models [21, 38, 48] or other imitations of DNNs' training dynamics [53] to derive NC (see Appendix A for a detailed discussion of related works), we consider standard gradient flow dynamics of DNNs simplified by our assumption on the NTK structure.

We analyze when NC does or does not occur in DNNs with NTK alignment, both theoretically and empirically. In particular, we identify an invariant of the training dynamics that provides a necessary condition for the emergence of NC in Section 5.2. Since DNNs with block-structured NTK do not always converge to NC, we conclude that NTK alignment is a more widespread phenomenon than NC.

We support our theory with large-scale numerical experiments in Section 6. Source code to reproduce the results is available in the project's GitHub repository.

2 Preliminaries

We consider the classification problem with C ∈ N classes, where the goal is to build a classifier that returns a class label for any input x ∈ X . In this work, the classifier is a DNN trained on a dataset { ( x i , y i ) } N i =1 , where x i ∈ X are the inputs and y i ∈ R C are the one-hot encodings of the class labels. We view the output function of the DNN f : X → R C as a composition of parametrized last-layer features h : X → R n and a linear classification layer parametrized by weights W ∈ R C × n and biases b ∈ R C . Then the logits of the training data X = { x i } N i =1 can be expressed as follows:

where H ∈ R n × N are the features of the entire dataset stacked as columns and 1 N ∈ R N is a vector of ones. Though we omit the notion of the data dependence in the text to follow, i.e. we write H without the explicit dependence on X , we emphasize that the features H are a function of the data and the DNN's parameters, unlike in the previously studied unconstrained feature models [21, 38, 48].

We assume that the dataset is balanced , i.e. there are m := N/C training samples for each class. Without loss of generality, we further assume that the inputs are reordered so that x ( c -1) m +1 , . . . , x cm belong to class c for all c ∈ [ C ] . This will make the notation much easier later on. Since the dimension of features n is typically much larger than the number of classes, we also assume n > C in this work.

2.1 Neural Collapse

Neural Collapse (NC) is an empirical behaviour of classifier DNNs trained past zero error [39]. Let ⟨ h ⟩ := N -1 ∑ N i =1 h ( x i ) denote the global features mean and ⟨ h ⟩ c := m -1 ∑ x i ∈ class c h ( x i ) , c ∈ [ C ] be the class means. Furthermore, define the matrix of normalized centered class means as M := [ ⟨ h ⟩ 1 / ∥⟨ h ⟩ 1 ∥ 2 , . . . , ⟨ h ⟩ C / ∥⟨ h ⟩ C ∥ 2 ] ⊤ ∈ R n × C , where ⟨ h ⟩ c = ⟨ h ⟩ c -⟨ h ⟩ , c ∈ [ C ] . We say that a DNN exhibits NC if the following four behaviours emerge as the training time t increases:

(NC1) Variability collapse: for all samples x c i from class c ∈ [ C ] , where i ∈ [ m ] , the penultimate layer features converge to their class means, i.e. ∥ h ( x c i ) -⟨ h ⟩ c ∥ 2 → 0 .

(NC2) Convergence to Simplex Equiangular Tight Frame (ETF): for all c, c ′ ∈ [ C ] , the class means converge to the following configuration:

(NC3) Convergence to self-duality: the class means M and the final weights W ⊤ converge to each other:

∥ ∥ M / ∥ M ∥ F -W ⊤ / ∥ W ⊤ ∥ F ∥ ∥ F → 0 . (NC4) Simplification to Nearest Class Center (NCC): the classifier converges to the NCC decision rule behaviour:

Though NC is observed in practice, there is currently no conclusive theory on the mechanisms of its emergence during DNN training. Most theoretical works on NC adopt the unconstrained features model, where features H are free variables that can be directly optimized [21, 38, 48]. Training dynamics of such models do not accurately reflect the dynamics of real DNNs, since they ignore the dependence of the features on the input data and the DNN's trainable parameters. In this work, we make a step towards realistic DNN dynamics by means of the Neural Tangent Kernel (NTK).

2.2 Neural Tangent Kernel

The NTK Θ of a DNN with the output function f : X → R C and trainable parameters w ∈ R P (stretched into a single vector) is given by

We also define the last-layer features kernel Θ h , which is a component of the NTK corresponding to the parameters up to the penultimate layer, as follows:

Intuitively, the NTK captures the correlations between the training samples in the DNN dynamics. While most theoretical works consider the infinite-width limit of DNNs [30, 52], where the NTK can be computed theoretically, empirical studies have also extensively explored the NTK of finite-width networks [19, 36, 45, 49]. Unlike the label-agnostic infinite-width NTK, the empirical NTK aligns with the labels during training. We use this observation in our main assumption (Section 3).

2.3 Classification with MSE Loss

We study NC for DNNs with the mean squared error (MSE) loss given by

where Y ∈ R C × N is a matrix of stacked labels y i . While NC was originally introduced for the cross-entropy (CE) loss [39], which is more common in classification problems, the MSE loss is much easier to analyze theoretically. Moreover, empirical observations suggest that DNNs with MSE loss achieve comparable performance to using CE [14, 29, 41], which motivates the recent line of research on MSE-NC [21, 38, 48].

3 Block Structure of the NTK

Numerous empirical studies have demonstrated that the NTK becomes aligned with the labels Y ⊤ Y during the training process [7, 32, 45]. This alignment constitutes feature learning and is associated with better performance of DNNs [9, 13]. For classification problems, this means that the empirical

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

a)

∑

k

Θ

k,k

(

X

)

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

b)

∑

k

Θ

h

k,k

(

X

)

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

c)

‖

Θ

k,s

‖

F

h

1

h

11

h

21

h

31

h

41

h

51

h

61

h

1

h

11

h

21

h

31

h

41

h

51

h

61

d)

‖

Θ

h

k,s

‖

F

0

100

200

300

400

0

.

4

0

.

6

0

.

8

e)

Alignment

〈

Θ

,Y Y

T

〉

‖

Θ

‖

F

‖

Y Y T

‖

F

〈

Θ

h

,Y Y

T

〉

‖

Θ

h

‖

F

‖

Y Y T

‖

F

0.1

0.2

0.5

0

0.5

1

NTK develops an approximate block structure with larger kernel values corresponding to pairs of samples ( x c i , x c j ) from the same class [44]. Figure 1 shows an example of such a structure emergent in the empirical NTK of ResNet20 trained on MNIST. 2 Motivated by these observations, we assume that the NTK and the last-layer features kernel exhibit a block structure, defined as follows:

Definition 3.1 (Block structure of a kernel) . We say a kernel Θ : X × X → R K × K has a block structure associated with ( λ 1 , λ 2 , λ 3 ) , if λ 1 > λ 2 > λ 3 ≥ 0 and

where x c i and x c j are two distinct inputs from the same class, and x c ′ j is an input from class c ′ = c .

Assumption 3.2. The NTK Θ : X × X → R C × C has a block structure associated with ( γ d , γ c , γ n ) , and the penultimate kernel Θ h : X × X → R n × n has a block structure associated with ( κ d , κ c , κ n ) .

̸

̸

This assumption means that every kernel Θ k,k ( X ) := [Θ k,k ( x i , x j )] i,j ∈ [ N ] corresponding to an output neuron f k , k ∈ [ C ] and every kernel Θ h p,p ( X ) corresponding to a last-layer neuron h p , p ∈ [ n ] is aligned with Y ⊤ Y (see Figure 1, panes a-b). Additionally, the "non-diagonal" kernels Θ k,s ( X ) and Θ h k,s ( X ) , k = s are equal to zero (see Figure 1, panes c-d). 3 Moreover, if γ c ≫ γ n and κ c ≫ κ n , Assumption 3.2 can be interpreted as local elasticity of DNNs, defined below.

Definition 3.3 (Local elasticity [23]) . A classifier is said to be locally elastic (LE) if its prediction or feature representation on point x c i from class c ∈ [ C ] is not significantly affected by performing SGD updates on data points from classes c ′ = c .

̸

To see the relation between Assumption 3.2 and this definition, consider a Gradient Descent (GD) step of the output neuron f k , k ∈ [ C ] with step size η performed on a single input x c ′ j from class c ′ = c . By the chain rule, block-structured Θ implies locally-elastic predictions since

̸

i.e., the magnitude of the GD step of f ( x c i ) is determined by the value of Θ( x c i , x c ′ j ) . Similarly, block-structured kernel Θ h implies locally-elastic penultimate layer features because

This observation provides a connection between our work and recent contributions suggesting a connection between NC and local elasticity [33, 53].

4 Dynamics of DNNs with NTK Alignment

4.1 Convergence

As a warm up for our main results, we analyze the effects of the NTK block structure on the convergence of DNNs. Consider a GD update of an output neuron f k , k ∈ [ C ] with the step size η :

̸

Note that we have taken into account that Θ k,s is zero for k = s by our assumption. Denote the residuals corresponding to f k as r ⊤ k := f ⊤ k ( X ) -Y k ∈ R N . Then we have the following dynamics for the residuals vector:

The eigendecomposition of the block-structured kernel Θ k,k ( X ) provides important insights into this dynamics and is summarized in Table 1. We notice that the NTK has three distinct eigenvalues λ global ≥ λ class ≥ λ single, which imply different convergence rates for certain components of the error. Moreover, the eigenvectors associated with each of these eigenvalues reveal the meaning of the error components corresponding to each convergence rate. Indeed, consider the projected dynamics with respect to eigenvector v 0 and eigenvalue λ global from Table 1:

where we omitted O ( η 2 ) for clarity. Now notice that the projection of r t k onto the vector v 0 is in fact proportional to the average residual over the training set:

where ⟨·⟩ denotes the average over all the training samples x i ∈ X . By a similar calculation, for all c ∈ [ C ] and i ∈ [ m ] we get interpretations of the remaining projections of the residual:

We where ⟨·⟩ c denotes the average over samples x c i from class c , and r ⊤ k ( x c i ) is the k th component of f ⊤ ( x c i ) -y c i . Combining (10), (11) and (12), we have the following convergence rates:

Overall, this means that the global mean ⟨ r ⟩ of the residual converges first, then the class means, and finally the residual of each sample r ( x c i ) . To simplify the notation, we define the following quantities:

where R ∈ R C × N is the matrix of residuals, R class ∈ R C × N are the residuals averaged over each class and stacked m times, and R global ∈ R C × N are the residuals averaged over the whole training set stacked N times. According to the previous discussion, R global converges to zero at the fastest rate, while R converges at the slowest rate. The last phase, which we call the end of training , is when R class and R global have nearly vanished and can be treated as zero for the remaining training time. We will use this notion in several remarks, as well as in the proof of Theorem 5.2.

4.2 Gradient Flow Dynamics with Block-Structured NTK

We derive the dynamics of H , W , b under Assumption 3.2 in Theorem 4.1. One can see that the block-structured kernel greatly simplifies the complicated dynamics of DNNs and highlights the role of each of the residual components identified in Section 4.1. We consider gradient flow, which is close to gradient descent for sufficiently small step size [16], to reduce the complications caused by higher order terms. The proof is given in Appendix B.1.

Theorem 4.1. Suppose Assumption 3.2 holds. Then the gradient flow dynamics of a DNN can be written as

We note that at the end of training, where R class and R global are zero, the system (19) reduces to

and ˙ b = 0 . This system differs from the unconstrained features dynamics only by a factor of κ d -κ c before H . Moreover, such a form of the loss function also appears in the literature of implicit regularization [4, 6, 11], where the authors show that WH converges to a low rank matrix.

5 NTK Alignment Drives Neural Collapse

The main goal of this work is to demonstrate how NC results from the NTK block structure. To this end, in Section 5.1 we further analyze the dynamics presented in Theorem 4.1, in Section 5.2 we derive the invariant of this training dynamics, and in Section 5.3 we finally derive NC.

5.1 Features Decomposition

We first decompose the features dynamics presented in Theorem 4.1 into two parts: H 1 , which lies in the subspace of the labels Y , and H 2 , which is orthogonal to the labels and eventually vanishes. To achieve this, note that the SVD of Y has the following form:

where O ∈ R C × ( N -C ) is a matrix of zeros, and P ∈ R C × C and Q ∈ R N × N are orthogonal matrices. Moreover, we can choose P and Q such that P = I C and

where ⊗ is the Kronecker product. Note that by orthogonality, ˜ Q 2 ∈ R m × ( m -1) has full rank and 1 ⊤ m ˜ Q 2 = O . We can now decompose HQ into two components as follows:

The following equations reveal the meaning of these two components:

where ⟨ h ⟩ c ∈ R n is the mean of h over inputs x c i from class c ∈ [ C ] , and H ( c ) ∈ R n × m is the submatrix of H corresponding to samples of class c , i.e., H = [ H (1) , . . . , H ( C ) ] . We see that H 1

is simply the matrix of the last-layer features' class means, which is prominent in the NC literature. We also see that the columns of H ( c ) ˜ Q 2 are m -1 different linear combinations of m vectors h ( x c i ) , i ∈ [ m ] . Moreover, the coefficients of each of these linear combinations sum to zero by the choice of ˜ Q 2 . Therefore, H 2 must reduce to zero in case of variability collapse (NC1), when all the feature vectors within the same class become equal. We prove that H 2 indeed vanishes in DNNs with block-structured NTK as part of our main result (Theorem 5.2).

5.2 Invariant

We now use the former decomposition of the last-layer features to further simplify the dynamics and deduce a training invariant in Theorem 5.1. The proof is given in Appendix B.2.

Theorem 5.1. Suppose Assumption 3.2 holds. Define H 1 and H 2 as in (23) . Then the class-means of the residuals (defined in (17) ) are given by R 1 = WH 1 + b1 ⊤ C -I C , and the training dynamics of the DNN can be written as

where µ single := κ d -κ c and µ class := µ single + m ( κ c -κ n ) are the two smallest eigenvalues of the kernel Θ h k,k ( X ) for any k ∈ [ n ] . Moreover, the quantity

is invariant in time. Here α := κ n m µ class + Cκ n m .

We note that the invariant E derived here resembles the conservation laws of hyperbolic dynamics that take the form E hyp := a 2 -b 2 = const for time-dependent quantities a and b . Such dynamics arise when gradient flow is applied to a loss function of the form L ( a, b ) := ( ab -q ) 2 for some q . Since the solutions of such minimization problems, given by ab = q , exhibit symmetry under scaling a → γa, b → b/γ , the value of the invariant E hyp uniquely specifies the hyperbola followed by the solution. In machine learning theory, hyperbolic dynamics arise as the gradient flow dynamics of linear DNNs [42], or in matrix factorization problems [3, 15]. Moreover, the end of training dynamics defined in (20) has a hyperbolic invariant given by

Therefore, the final phase of training exhibits a typical behavior for the hyperbolic dynamics, which is also characteristic for the unconstrained features models [21, 38]. Namely, "scaling" W and H by an invertible matrix does not affect the loss value but changes the dynamic's invariant. On the other hand, minimizing the invariant E eot has the same effect as joint regularization of W and H [48].

However, we also note that our invariant E provides a new, more comprehensive look at the DNNs' dynamics. While unconstrained features models effectively make assumptions on the end-of-training invariant E eot to derive NC [21, 38, 48], our dynamics control the value of E eot through the more general invariant E . This way we connect the properties of end-of-training hyperbolic dynamics with the previous stages of training.

5.3 Neural Collapse

We are finally ready to state and prove our main result in Theorem 5.2 about the emergence of NC in DNNs with NTK alignment. We include the proof in Appendix B.3.

Theorem 5.2. Assume that the NTK has a block structure as defined in Assumption 3.2. Then the DNN's training dynamics are given by the system of equations in (25) . Assume further that the last-layer features are centralized, i.e ⟨ h ⟩ = 0 , and the dynamics invariant (26) is zero, i.e., E = O . Then the DNN's dynamics exhibit neural collapse as defined in (NC1)-(NC4).

Below we provide several important remarks and discuss the implications of this result:

(1) Zero invariant assumption: We assume that the invariant (26) is zero in Theorem 5.2 for simplicity and consistency with the literature. Indeed, similar assumptions arise in matrix decomposition papers, where zero invariant guarantees "balance" of the problem [3, 15]. However, our proofs in fact only require a weaker assumption that the invariant terms containing features H are aligned with the weights W ⊤ W , i.e.

where we have taken into account our assumption on the zero global mean ⟨ h ⟩ = 0 .

(2) Necessity of the invariant assumption: The relaxed assumption on the invariant (28) is necessary for the emergence of NC in DNNs with block-structured NTK. Indeed, NC1 implies H 2 = O , and NC3 implies H 1 H ⊤ 1 ∝ W ⊤ W . Therefore, DNNs that do not satisfy this assumption do not display NC. Our numerical experiments described in Section 6 strongly support this insight (see Figure 2, panes a-e). Thus, we believe that the invariant derived in this work characterizes the difference between models that do and do not exhibit NC.

(3) Zero global mean assumption: We note that the zero global mean assumption ⟨ h ⟩ = 0 in Theorem 5.2 ensures that the biases are equal to b = 1 C 1 C at the end of training. This assumption is common in the NC literature [21, 38] and is well-supported by our numerical experiments (see figures in Appendix C, pane i). Indeed, modern DNNs typically include certain normalization (e.g. through batch normalization layers) to improve numerical stability, and closeness of the global mean to zero is a by-product of such normalization.

(4) General biases case: Discarding the zero global mean assumption allows the biases b to take an arbitrary form. In this general case, the following holds for the matrix of weights:

For optimal biases b = 1 C 1 C , this reduces to the ETF structure that emerges in NC. Moreover, if biases are all equal, i.e. b = β 1 C for some β ∈ R , the centralized class means still form an ETF (i.e., NC2 holds), and the weights exhibit a certain symmetric structure given by

where γ := 1 C (1 -| 1 -βC | √ 1 -αC ) < 1 C . The proof and a discussion of this result are given in Appendix B.4. In general, the angles of these two frames are different, and thus NC3 does not hold. This insight leads us to believe that normalization is an important factor in the emergence of NC.

(5) Partial NC: Our proofs and the discussion suggest that all the four phenomena that form NC do not have to always coincide. In particular, our proof of NC1 only requires the block-structured NTK and the invariant to be P.S.D, which is much weaker than the total set of assumptions in Theorem 5.2. Therefore, variability collapse can occur in models that do not exhibit the ETF structure of the class-means or the duality of the weights and the class means. Moreover, as shown above, NC2 can occur when NC3 does not, i.e., the ETF structure of the class means does not imply duality.

6 Experiments

We conducted large-scale numerical experiments to support our theory. While we only showcase our results on a single dataset-architecture pair in the main text (see Figure 2) and refer the rest to the appendix, the following discussion covers all our experiments.

Datasets and models. Following the seminal NC paper [39], we use three canonical DNN architectures: VGG [46], ResNet [24] and DenseNet [26]. Our datasets are MNIST [35], FashionMNIST [51] and CIFAR10 [34]. We choose VGG11 for MNIST and FashionMNIST, and VGG16 for CIFAR10. We add batch normalization after every layer in the VGG architecture, set dropout to zero and choose the dimensions of the two fully-connected layers on the top of the network as 512 and 256 . We use ResNet20 architecture described in the original ResNet paper [24], and DenseNet40 with bottleneck layers, growth k = 12 , and zero dropout for all the datasets.

f)

LeCun normal init.

10

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.003

Acc.:

99

.

49

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

45

LeCun normal init., LR=0.049

Acc.:

99

.

69

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

93

Uniform init., LR=0.003

Acc.:

99

.

60

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

51

Uniform init., LR=0.049

Acc.:

99

.

69

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

88

He normal init., LR=0.005

Acc.:

99

.

51

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

38

He normal init., LR=0.049

Acc.:

99

.

64

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

92

0

100

200

300

400

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

d)

NC2

0

100

200

300

400

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

e)

NC3

0

.

98

0

.

99

Test accuracy

NC1

0

.

98

0

.

99

NC2

0

.

98

0

.

99

NC3

0

.

00

0

.

25

0

.

50

0

.

00

0

.

25

0

.

50

0

.

75

g)

Uniform init.

0

.

00

0

.

25

0

.

50

0

.

75

h)

He normal init.

0

.

98

0

.

99

1

.

00

0

.

98

0

.

99

1

.

00

10

-

3

10

-

1

Learning rate

0

.

98

0

.

99

1

.

00

0

.

0

0

.

5

1

.

0

1

.

5

f)

LeCun normal init.

0

.

0

0

.

5

1

.

0

1

.

5

g)

Uniform init.

0

.

0

0

.

5

1

.

0

1

.

5

h)

He normal init.

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.003

Acc.:

99

.

49

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

45

LeCun normal init., LR=0.049

Acc.:

99

.

69

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

93

Uniform init., LR=0.003

Acc.:

99

.

60

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

51

Uniform init., LR=0.049

Acc.:

99

.

69

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

88

He normal init., LR=0.005

Acc.:

99

.

51

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

38

He normal init., LR=0.049

Acc.:

99

.

64

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

92

0

100

200

300

400

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

d)

NC2

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

e)

NC3

3

10

-

Learning rate

-

1

Optimization and initialization. We use SGD with Nesterov momentum 0 . 9 and weight decay 5 × 10 -4 . Every model is trained for 400 epochs with batches of size 120. To be consistent with the theory, we balance the batches exactly. We train every model with a set of initial learning rates spaced logarithmically in the range η ∈ [10 -4 , 10 0 . 25 ] . The learning rate is divided by 10 every 120 epochs. On top of the varying learning rates, we try three different initialization settings for every model: (a) LeCun normal initialization (default in Flax), (b) uniform initialization on [ -√ k, √ k ] , where k = 1 /n ℓ -1 for a linear layer, and k = 1 / ( Kn ℓ -1 ) for a convolutional layer, where K is the convolutional kernel size (default in PyTorch), (c) He normal initialization in fan_out mode.

Results. Our experiments confirm the validity of our assumptions and the emergence of NC as their result. Specifically, we make the following observations:

While most of the DNNs that achieve high test performance exhibit NC, we are able to identify DNNs with comparable performance that do not exhibit NC (see Figure 2, panes f-h). We note that such models still achieve near-zero error on the training set in our setup.

Comparing DNNs that do and do not exhibit NC, we find that our assumption on the invariant (see Theorem 5.2 and (28)) holds only for the models with NC (see Figure 2, panes a-e). This confirms our reasoning about the necessity of the invariant assumption for NC emergence.

The kernels Θ and Θ h are strongly aligned with the labels Y ⊤ Y in the models with the best performance, which is in agreement with the NTK alignment literature and justifies our assumption on the NTK block structure.

We include the full range of experiments along with the implementation details and the discussion of required computational resources in Appendix C. Specifically, we present a figure analogous to Figure 2 for every considered dataset-architecture pair. Additionally, we report the norms of matrices H 1 H ⊤ 1 , H 2 H ⊤ 2 , and ⟨ h ⟩⟨ h ⟩ ⊤ , as well as the alignment of both the NTK Θ and the last-layer features kernel Θ h in the end of training, to further justify our assumptions.

0

.

75

7 Conclusions and Broad Impact

This work establishes the connection between NTK alignment and NC, and thus provides a mechanistic explanation for the emergence of NC within realistic DNNs' training dynamics. It also contributes to the underexplored line of research connecting NC and local elasticity of DNNs' training dynamics.

The primary implication of this research is that it exposes the potential to study NC through the lens of NTK alignment. Indeed, previous works on NC focus on the top-down approach (layer-peeled models) [18, 21, 38, 48], and fundamentally cannot explain how NC develops through earlier layers of a DNN and what are the effects of depth. On the other hand, NTK alignment literature focuses on the alignment of individual layers [7], and recent theoretical results even quantify the role of each hidden layer in the final alignment [37]. Therefore, we believe that the connection between NTK alignment and NC established in this work provides a conceptually new method to study NC.

Moreover, this work introduces a novel approach to facilitate theoretical analysis of DNNs' training dynamics. While most theoretical works consider the NTK in the infinite-width limit to simplify the dynamics [1, 20, 28, 49], our analysis shows that making reasonable assumptions on the empirical NTK can also lead to tractable dynamics equations and new theoretical results. Thus, we believe that the analysis of DNNs' training dynamics based on the properties of the empirical NTK is a promising approach also beyond NC research.

8 Limitations and Future Work

The main limitation of this work is the simplifying Assumption 3.2 on the kernel structure. While the NTK of well-trained DNNs indeed has an approximate block structure (as we discuss in detail in Section 3), the NTK values also tend to display high variance in real DNNs [22, 44]. Thus, we believe that adding stochasticity to the dynamics considered in this paper is a promising direction for the future work. Moreover, the empirical NTK exhibits so-called specialization, i.e., the kernel matrix corresponding to a certain output neurons aligns more with the labels of the corresponding class [45]. In block-structured kernels, specialization implies different values in blocks corresponding to different classes. Thus, generalizing our theory to block-structured kernels with specialization is another promising short-term research goal. In addition, our theory relies on the assumption that the dataset (or the training batch) is balanced, i.e., all the classes have the same number of samples. Accounting for the effects of non-balanced datasets within the dynamics of DNNs with block-structured NTK is another possible future work direction.

More generally, we believe that empirical observations are essential to demistify the DNNs' training dynamics, and there are still many unknown and interesting connections between seemingly unrelated empirical phenomena. Establishing new theoretical connections between such phenomena is an important objective, since it provides a more coherent picture of the deep learning theory as a whole.

Acknowledgments and Disclosure of Funding

R. Giryes and G. Kutyniok acknowledge support from the LMU-TAU - International Key Cooperation Tel Aviv University 2023. R. Giryes is also grateful for partial support by ERC-StG SPADE grant no. 757497. G. Kutyniok is grateful for partial support by the Konrad Zuse School of Excellence in Reliable AI (DAAD), the Munich Center for Machine Learning (BMBF) as well as the German Research Foundation under Grants DFG-SPP-2298, KU 1446/31-1 and KU 1446/32-1 and under Grant DFG-SFB/TR 109, Project C09 and the Federal Ministry of Education and Research under Grant MaGriDo.

References

Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 74-84. PMLR, 2020.

Laurence Aitchison. Why bigger is not always better: on finite and infinite neural networks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 156-164. PMLR, 2020.

Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 244-253. PMLR, 2018.

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 7411-7422, 2019.

Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Bubacarr Bah, Holger Rauhut, Ulrich Terstiege, and Michael Westdickenberg. Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers. Information and Inference: A Journal of the IMA , 11(1):307-353, 02 2021.

Aristide Baratin, Thomas George, César Laurent, R. Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event , volume 130 of Proceedings of Machine Learning Research , pages 2269-2277. PMLR, 2021.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.

Shuxiao Chen, Hangfeng He, and Weijie J. Su. Label-aware neural tangent kernel: Toward better generalization and local elasticity. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Lénaïc Chizat, Edouard Oyallon, and Francis R. Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 2933-2943, 2019.

Hung-Hsu Chou, Carsten Gieshoff, Johannes Maly, and Holger Rauhut. Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank. arXiv preprint: 2011.13772 , 2020.

Moustapha Cissé, Piotr Bojanowski, Edouard Grave, Yann N. Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. In Proceedings of the 34th

International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 854-863. PMLR, 2017.

Nello Cristianini, John Shawe-Taylor, André Elisseeff, and Jaz S. Kandola. On kernel-target alignment. In Advances in Neural Information Processing Systems 14: Natural and Synthetic, NIPS 2001, December 3-8, 2001, Vancouver, British Columbia, Canada , pages 367-373. MIT Press, 2001.

Ahmet Demirkaya, Jiasi Chen, and Samet Oymak. Exploring the role of loss functions in multiclass classification. In 54th Annual Conference on Information Sciences and Systems, CISS 2020, Princeton, NJ, USA, March 18-20, 2020 , pages 1-5. IEEE, 2020.

Simon S. Du, Wei Hu, and Jason D. Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 382-393, 2018.

Omer Elkabetz and Nadav Cohen. Continuous vs. discrete optimization of deep neural networks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 4947-4960, 2021.

Tolga Ergen and Mert Pilanci. Revealing the structure of deep neural networks via convex duality. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 3004-3014. PMLR, 2021.

C Fang, H He, Q Long, and WJ Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences of the United States of America , 118(43), 2021.

Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d'Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning. Journal of Statistical Mechanics: Theory and Experiment , 2020(2):023401, 2020.

X. Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamics on the central path. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Hangfeng He and Weijie J. Su. The local elasticity of neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770-778. IEEE Computer Society, 2016.

Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2020.

Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages 2261-2269. IEEE Computer Society, 2017.

Jiaoyang Huang and Horng-Tzer Yau. Dynamics of deep neural networks and neural tangent hierarchy. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 4542-4551. PMLR, 2020.

Kaixuan Huang, Yuqing Wang, Molei Tao, and Tuo Zhao. Why do deep residual networks generalize better than deep feedforward networks? - A neural tangent kernel perspective. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs crossentropy in classification tasks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 8580-8589, 2018.

Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Dmitry Kopitkov and Vadim Indelman. Neural spectrum alignment: Empirical study. In Artificial Neural Networks and Machine Learning - ICANN 2020 - 29th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 15-18, 2020, Proceedings, Part II , volume 12397 of Lecture Notes in Computer Science , pages 168-179. Springer, 2020.

Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.04041 , 2022.

Alex Krizhevsky et al. Learning multiple layers of features from tiny images, 2009.

Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2, 2010.

Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. Advances in Neural Information Processing Systems , 33:15156-15172, 2020.

Yizhang Lou, Chris E Mingard, and Soufiane Hayou. Feature learning and signal propagation in deep neural networks. In International Conference on Machine Learning , pages 14248-14282. PMLR, 2022.

Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11619 , 2020.

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):24652-24663, 2020.

Federico Pernici, Matteo Bruni, Claudio Baecchi, and Alberto Del Bimbo. Fix your features: Stationary and maximally discriminative embeddings using regular polytope (fixed classifier) networks. arXiv preprint arXiv:1902.10441 , 2019.

Tomaso A. Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint arXiv:2101.00072 , 2021.

Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 , 2013.

Mariia Seleznova and Gitta Kutyniok. Analyzing finite neural networks: Can we trust neural tangent kernel theory? In Mathematical and Scientific Machine Learning, 16-19 August 2021, Virtual Conference / Lausanne, Switzerland , volume 145 of Proceedings of Machine Learning Research , pages 868-895. PMLR, 2021.

Mariia Seleznova and Gitta Kutyniok. Neural tangent kernel beyond the infinite-width limit: Effects of depth and initialization. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 19522-19560. PMLR, 2022.

Haozhe Shan and Blake Bordelon. A theory of neural tangent kernel alignment and its influence on training. arXiv preprint arXiv:2105.14301 , 2021.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6398-6407, 2020.

Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 21478-21505, 2022.

Tom Tirer, Joan Bruna, and Raja Giryes. Kernel-based smoothness analysis of residual networks. In Mathematical and Scientific Machine Learning , volume 145, pages 921-954, 2021.

Tom Tirer, Haoxiang Huang, and Jonathan Niles-Weed. Perturbation analysis of neural collapse. arXiv preprint arXiv:2210.16658 , 2022.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Greg Yang. Tensor programs II: neural tangent kernel for any architecture. arXiv preprint arXiv:2006.14548 , 2020.

Jiayao Zhang, Hua Wang, and Weijie J. Su. Imitating deep learning dynamics via locally elastic stochastic differential equations. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 6392-6403, 2021.

A Related works

NC with MSE loss. NC was first introduced for DNNs with cross-entropy (CE) loss, which is commonly used in classification problems [39]. Since then, numerous papers discussed NC with MSE loss, which provides more opportunities for theoretical analysis, especially after the MSE loss was shown to perform on par with CE loss for classification tasks [14, 29].

Most previous works on MSE-NC adopt the so-called unconstrained features model [21, 38, 48]. In this model, the last-layer features H are free variables that are directly optimized during training, i.e., the features do not depend on the input data or the DNN's trainable parameters. Fang et al. [18] also introduced a generalization of this approach called N -layer-peeled model, where features of the N -th-to-last layer are free variables, and studied the 1-layer-peeled model (equivalent to the unconstrained features model) with CE loss as a special case.

One line of research on MSE-NC in unconstrained/layer-peeled models aims to derive global minimizers of optimization problems associated with DNNs [17, 18, 48]. In particular, Tirer et al. [48] showed that global minimizers of the MSE loss with regularization of both H and W exhibit NC. Moreover, Ergen & Pilanci [17] showed that NC emerges in global minimizers of optimization problems with general convex loss in the context of the 2-layer-peeled model. In comparison to our work, these contributions do not consider the training dynamics of DNNs, i.e., they do not discuss whether and how the model converges to the optimal solution.

Another line of research on MSE-NC explicitly considers the dynamics of the unconstrained features models [21, 38]. In particular, Han et al. [21] considered the gradient flow of the unconstrained renormalized features along the "central path", where the classifier is assumed to take the form of the optimal least squares (OLS) solution for given features H . Under this assumption, they derive a closedform dynamics that implies NC. While they empirically show that DNNs are close to the central path in certain scenarios, they do not provide a theoretical justification for this assumption. The dynamics considered in their work is also distinct from the standard gradient flow dynamics of DNNs considered in our work. On the other hand, an earlier work by Mixon et al. [38] considered the gradient flow dynamics of the unconstrained features model, which is equivalent (up to rescaling) to the end-oftraining dynamics (20) that we discuss in Sections 4.2 and 5.2. Their work relies on the linearization of these dynamics to derive a certain subspace, which appears to be an invariant subspace of the non-linearized unconstrained features model dynamics. Then they show that minimizers of the loss from this subspace exhibit NC. We note that, in terms of our paper, assuming that the unconstrained features model dynamics follow a certain invariant subspace means making assumptions on the end-of-training invariant (27). In comparison to these works, we make a step towards realistic DNNs dynamics by considering the standard gradient flow of DNNs simplified by Assumption 3.2 on the NTK structure, which is supported by the extensive research on NTK alignment [7, 9, 44, 45]. In our setting, the NTK captures the dependence of the features on the training data, which is missing in the unconstrained features model. Moreover, while other works focus only on the dynamics that converge to NC, we show that DNNs with MSE loss may not exhibit NC in certain settings, and the invariant of the dynamics (26) characterizes the difference between models that do and do not converge to NC.

Notably, works by Poggio & Liao [41] adopt a model different from the unconstrained features model to analyze gradient flow of DNNs. They consider the dynamics of homogeneous DNNs, in particular ReLU networks without biases, with normalization of the weights matrices and weights regularization. The goal of weights normalization in their model is to imitate the effects of batch normalization in DNNs training. In this model, certain fixed points of the gradient flow exhibit NC. While the approach taken in their work captures the dependence of the features on the data and the DNN's parameters, it fundamentally relies on the homogeneity of the DNN's output function. However, most DNNs that exhibit NC in practice are not homogeneous due to biases and skip-connections.

NC and local elasticity. A recent extensive survey of NC literature [33] discussed local elasticity as a possible mechanism behind the emergence of NC, which has not been sufficiently explored up until now. One of the few works in this research direction is by Zhang et al. [53], who analyzed the so-called locally-elastic stochastic differential equations (SDEs) and showed the emergence of NC in their solutions. They model local elasticity of the dynamics through an effect matrix, which has only two distinct values: a larger intra-class value and a smaller inter-class value. These values characterize how much influence samples from one class have on samples from other classes in the SDEs. While the aim of their work is to imitate DNNs' training dynamics through SDEs, the authors

do not provide any explicit connection between their dynamics and real gradient flow dynamics of DNNs. On the other hand, we derive our dynamics directly from the gradient flow equations and connect local elasticity to the NTK, which is a well-studied object in the deep learning theory.

Another work by Tirer et al. [50] provided a perturbation analysis of NC to study "inexact collapse". They considered a minimization problem with MSE loss, regularization of H and W , and additional regularization of the distance between H and a given matrix of initial features. In the "near-collapse" setting, i.e., when the initial features are already close to collapse, they showed that the optimal features can be obtained from the initial features by a certain linear transformation with a block structure, where the intra-class effects are stronger than the inter-class ones. While this transformation matrix resembles the block-structured effect matrices in locally-elastic training dynamics, it does not originate from the gradient flow dynamics of DNNs and is not related to the NTK.

B Proofs

B.1 Proof of Theorem 4.1

Proof of Theorem 4.1. We will first derive the dynamics of h s ( x c i ) , which is the s -th component of the last-layer features vector on sample x c i ∈ X from class c ∈ [ C ] . Let w ∈ R P be the trainable parameters of the network stretched into a single vector. Then its gradient flow dynamics is given by

where ∇ w f ( X ) ki ′ ∈ R P is the component of the DNN's Jacobian corresponding to output neuron k and the input sample x c ′ i ′ . Since entries of f ( X ) can be written as

we obtain

By chain rule, we have ˙ h s ( x c i ) = ⟨∇ w h s ( x c i ) , ˙ w ⟩ . Then, taking into account that

and that ⟨∇ w h s ( x c i ) , ∇ w h s ′ ( x c ′ i ′ ) ⟩ = Θ h s,s ′ ( x c i , x c ′ i ′ ) by definition of Θ h , we have

Now by Assumption 3.2 we have Θ h s,s ′ = 0 if s = s ′ . Therefore, the above expression simplifies to

̸

̸

̸

To express ˙ H = [ ˙ h s ( x c i ) ] s,i ∈ R n × N in matrix form, it remains to express Θ h s,s ( x c i , x c ′ i ′ ) as the ( i ′ , i ) -th entry of some matrix. We will separate the sum into three cases: 1) i = i ′ , 2) i = i ′ and c = c ′ , and 3) c = c ′ . According to Assumption 3.2, the first case corresponds to the multiple of identity κ d I N . The second corresponds to the block matrix of size m with zeros on the diagonal,

which can be written as κ c ( Y ⊤ Y -I N ) . The third matrix equals to κ n ( 1 N 1 ⊤ N -Y ⊤ Y ) . Therefore we can express the dynamics of H as follows:

Now we notice that HY ⊤ Y /m is the matrix of stacked class means repeated m times each and H1 N 1 ⊤ N /N is a matrix of the global mean repeated N times. Therefore, we have

according to the definitions of global and class-mean residuals in (18) and (17).

The expressions for the gradient flow dynamics of W and b follow directly from the derivatives of f ( X ) w.r.t. W and b . This completes the proof.

B.2 Proof of Theorem 5.1

Proof of Theorem 5.1. Recall from (23) in Section 5.1 that we have the following decomposition

with orthogonal Q = [ Q 1 , Q 2 ] ∈ R N × N . We now artificially add QQ ⊤ (= I N ) to the dynamics (19) in Theorem 4.1 and obtain

 Let us simplify the expression. Since Q 1 = 1 √ m I C ⊗ 1 m and Q 2 = I C ⊗ ˜ Q 2 , we have

Plugging (37) into (36), we see the dynamics can be decomposed into

To further simplify (38), we define the following quantities

Notice that µ single and µ class are the two largest eigenvalues of the block-structured kernel Θ h s,s ( X ) (see Table 1 for the eigndecomposition of a block-structured matrix), and R 1 is a matrix of the stacked class-mean residuals, which is also defined in (17). The the dynamics (38) simplifies to

It remains to simplify the expression for ˙ H 1 . By using the relation

we can deduce that the dynamics for ˙ H 1 in (40) can be expressed as (recalling that N = mC )

We notice that ( I C + κ n m µ class 1 C 1 ⊤ C ) -1 = I C -α 1 C 1 ⊤ C , where α := κ n m µ class + Cκ n m . Then we can derive the invariant of the training dynamics by direct computation of the time-derivative ˙ E , where

Since ˙ E = O , we get that the quantity E remains constant in time. This completes the proof.

B.3 Proof of Theorem 5.2

We divide the proof into two main parts: the first one shows the emergence of NC1, and the second one shows NC2-4.

(NC1) . Following the analysis in Section 3, the dynamics eventually enters the end of training phase (see Section 4.1). Then the dynamics in Theorem 5.1 simplifies to the following form:

As we note in Section 4, this dynamics is similar to the gradient flow of the unconstrained features models and is an instance of the class of hyperbolic dynamics, which is discussed in Section 5.2. During this phase the quantity

does not change in time. Hence we can decouple the dynamic using the invariant as follows:

Since E is p.s.d (or zero, as a special case), ˜ E is p.s.d as well, and the eigendecomposition of the invariant is given by ˜ E = ∑ k c k v k v ⊤ k for some coefficients c k ≥ 0 and a set of orthonormal vectors v k ∈ R n . Then we also have H 2 H ⊤ 2 = ∑ k,l α kl v k v ⊤ l , where α kl are symmetric (i.e. α kl = α lk ) and α kk ≥ 0 for all k = 1 , . . . n (since H 2 H ⊤ 2 is symmetric and p.s.d.). Note that coefficients c k here are constant while coefficients α kl are time-dependent. Let us then write the dynamics for α kl using the dynamics of H 2 H ⊤ 2 :

Then for the elements of α we have:

For the diagonal elements α kk , this gives:

Since c k ≥ 0 , α kk ≥ 0 and α 2 kj ≥ 0 , we get that

And, therefore, all the non-diagonal elements also tend to zero. Thus, we get that

Now we notice that from the expression for H 2 in (24) it follows that H 2 = O implies variability collapse, since it means that all the feature vectors within the same class are equal. Indeed, H ( c ) ˜ Q 2 = O ∈ R n × ( m -1) means that there is a set of m -1 orthogonal vectors, which are all also orthogonal to [ h i ( x c 1 ) , . . . , h i ( x c m )] for any i = 1 , . . . , n , where x c i are inputs from class c . However, there is only one vector (up to a constant) orthogonal to all the columns of ˜ Q 2 in R m and this vector is 1 m . Therefore, [ h i ( x c 1 ) , . . . h i ( x c m )] = γ 1 m for some constant γ for any i = 1 , . . . , n . Thus, we indeed have h ( x c 1 ) = · · · = h ( x c m ) , which constitutes variability collapse within classes.

(NC2-4) . Set β = 1 C . We first show that zero global feature mean implies b = β 1 C . At the end of training, since R 1 = O , we have

On the other hand, zero global mean implies H 1 1 C = C ⟨ h ⟩ = O . Then multiplying (53) by 1 C on the right, we get the desired expression for the biases. Given the zero global mean, we have

By the proof of NC1, H 2 → O . Together with the assumption that E is proportional to the limit of W ⊤ W (or zero, as a special case), we obtain

for some γ ≥ 0 . Note that since H 1 H ⊤ 1 is p.s.d. this implies ˜ λ c := µ class -γ ≥ 0 . By multiplying the left and right with appropriate factors, we have

Consequently (according to (53))

Since both WW ⊤ and H ⊤ 1 H 1 are p.s.d., we have

To establish NC2, recall that H 1 = [ ⟨ h ⟩ 1 , . . . , ⟨ h ⟩ C ] and that M , as a normalized version of H 1 , satisfies

To establish NC3, note that from (55) and (58) together, it follows that the limits of M and W ⊤ only differ by a constant multiplier.

To establish NC4, note that using NC3 we can write

This completes the proof.

and thus

B.4 General biases case

Proof. As in the proof of Theorem 5.2, at the end of training we have WH 1 + b1 ⊤ C = I C . Moreover, since E = O and H 2 → O , we have

Multyplying the above expression to the left by W and to the right by W ⊤ , we obtain the general expression (29) for the matrix ( WW ⊤ ) 2 mentioned in the main text:

This expression implies that the rows of the weights matrix may have varying separation angles in the general biases case, i.e., there is no symmetric structure is general. However, for constant biases b = β 1 C , the above expression simplifies to

Since α < 1 /C and (1 -βC ) 2 ≥ 0 , we have that (1 -(1 -αC )(1 -βC ) 2 ) /C ≤ 1 /C . Therefore, the RHS of (61) is always p.s.d. and has a unique p.s.d square root proportional to I C -γ 1 C 1 ⊤ C for some constant γ < 1 /C . Denote ρ := (1 -(1 -αC )(1 -βC ) 2 ) /C , then we have γ = (1 - √ 1 -Cρ ) /C . Note that ρ < 1 /C ensures that γ is well defined. Then the configuration of the final weights is given by

This means that the norms of all the weights rows are still equal, as in NC2. However, since γ < 1 /C if β = 1 /C , the angle between these rows is smaller than in the ETF structure.

We can derive the configuration of the class means similarly by multyplying (59) to the left by H ⊤ 1 and to the right by H 1 . In the general biases case, we get

̸

As with the weights, we see that this is not a symmetric structure in general. Thus, NC2 does not hold in the general biases case. However, for the constant biases b = β 1 C , the above expression simplifies to

̸

Analogously to the previous derivations, we get that the unique p.s.d. square root of the RHS is given by I C -˜ ρ 1 C 1 ⊤ C , where ˜ ρ := (1 -| 1 -βC | ) /C < 1 /C for β = 1 /C . On the other hand, the unique p.s.d root of I -α 1 C 1 ⊤ C is given by I C -ϕ 1 C 1 ⊤ C , where ϕ := (1 -√ 1 -αC ) /C . Thus, we have the following

Therefore, the structure of the last-layer features class means is given by

̸

where θ := ˜ ρ + ϕ/ (1 + ϕC ) -C ˜ ρϕ/ (1 + ϕC ) < 1 /C for β = 1 /C . Thus, similarly to the classifier weights W , the last-layer features class means form a symmetric structure with equal lengths and a separation angle smaller than in the ETF. However, the centralized class means given by M = H 1 ( I C -1 C 1 T C /C ) still form the ETF structure:

This holds since the component proportional to 1 C 1 ⊤ C on the RHS of equation (66) lies in the kernel of the ETF matrix ( I C -1 C 1 ⊤ C /C ) . Thus, we conclude that NC2 holds in case of equal biases, while NC3 does not.

Remark on α → 0 case: Simplifying the expressions for constants γ and θ , which define the angles in the configurations of the weights and the class means above, we get the following:

̸

Analyzing these expressions, we find that they are equal only if 1 -αC = 1 , i.e. α = 0 . However, this can only hold if κ n = 0 by definition of α , i.e., when the kernel Θ h is zero on pairs of samples from different classes. While α = 0 in general, there are certain settings where α approaches zero. Simplifying the expression for α , we can get the following

One can see that α → 0 if C →∞ or when κ c /κ n →∞ . Since the kernel Θ h is strongly aligned with the labels in our numerical experiments, the value of κ c /κ n is large in practice. Thus, α is not zero but indeed significantly smaller than 1 /C . Thus, in our numerical experiments the angles θ and γ are close to each other. However, we note that the equality of these two angles does not imply NC3, since the value of θ characterizes the angles between the non-centralized class means.

Remark on α → 1 /C case: If α = 1 /C , the equation (63) for the structure of the features class means with general (not equal) biases simplifies to

i.e., in this case the class means always exhibit the ETF structure, even without the assumption that all the biases are equal. Moreover, in this case γ = 1 /C as well. Thus, both NC2 and NC3 hold. While by definition α < 1 /C , we can analyze the cases when it approaches 1 /C using the expression (69) again. One can see that when m →∞ and κ c /κ n → 1 , we have α → 1 /C . However, the requirement κ c /κ n → 1 implies that the kernel Θ h does not distinguish between pairs of samples from the same class and from different classes. Such a property of the kernel is associated with poor generalization performance and does not occur in our numerical experiments.

C Numerical experiments

Implementation details We use JAX [8] and Flax (neural network library for JAX) [25] to implement all the DNN architectures and the training routines. This choice of the software allows to compute the empirical NTK of any DNN architecture effortlestly and efficiently. We compute the values of kernels Θ and Θ h on the whole training batch ( m = 12 samples per class, 120 samples in total) in case of ResNet20 and DenseNet40 to approximate the values ( γ d , γ c , γ n ) and ( κ d , κ c , κ n ) , as well as the NTK alignment metrics, and compute the invariant E using these values. Since VGG11 and VGG16 architectures are much larger (over 10 million parameters) and computing their Jacobians is very memory-intensive, we use m = 4 samples per class (i.e., 40 samples in total) to approximate the kernels of these models. We compute all the other training metrics displayed in panes a-e of Figures 3, 4, 5, 6, 7, 8, 9, 10, 11 on the whole last batch of every second training epoch for all the architectures. The test accuracy is computed on the whole test set. To produce panes f-h of the same figures, we only compute the NC metrics and the test accuracy one time after 400 epochs of training for every learning rate. We use 30 logarithmically spaced learning rates in the range η ∈ [10 -4 , 10 0 . 25 ] for ResNet20 trained on MNIST and VGG11 trained on MNIST. For all the other architecture-dataset pairs we only compute the last 20 of these learning rates to reduce the computational costs, since the smallest learning rates do not yield models with acceptable performance.

Compute We executed the numerical experiments mainly on NVIDIA GeForce RTX 3090 Ti GPUs, each model was trained on a single GPU. In this setup, a single training run displayed in panes a-e of Figures 3, 4, 5, 6, 7, 8, 9, 10, 11 took approximately 3 hours for ResNet20, 6 hours for DenseNet40, 7 hours for VGG11, and 11 hours for VGG16. This adds up to a total of 312 hours to compute panes a-e of the figures. The computation time is mostly dedicated not to the training routine itself but to the large number of computationally-heavy metrics, which are computed every second epoch of a training run. Indeed, to approximate the values of Θ and Θ h , one needs to compute C ( C +1) + n ( n +1) kernels on a sample of size mC from the dataset, and each of the kernels requires computing a

gradient with respect to numerous parameters of a DNN. Additionally, the graphs in panes f-h of the same figures take around 1.5 hours for each learning rate value for ResNet20, 3 hours for DenseNet40, and 4 hours for VGG11 and VGG16, which adds up to approximately 1350 computational hours.

Results We include experiments on the following architecture-dataset pairs:

Figure 3: VGG11 trained on MNIST

Figure 4: VGG11 trained on FashionMNIST

Figure 5: VGG16 trained on CIFAR10

Figure 6: ResNet20 trained on MNIST

Figure 7: ResNet20 trained on FashionMNIST

Figure 8: ResNet20 trained on CIFAR10

Figure 9: DenseNet40 trained on MNIST

Figure 10: DenseNet40 trained on FashionMNIST

Figure 11: DenseNet40 trained on CIFAR10

The experiments setup is described in Section 6. Panes a-h of Figures 3, 4, 5, 6, 7, 8, 9, 10, 11 are analogous to the same panes of Figure 2. We include additional pane i here, which displays the norms of the invariant terms corresponding to the feature matrix components H 1 and H 2 , and the global features mean ⟨ h ⟩ at the end of training. One can see that the global features mean is relatively small in comparison with the class-means in every setup, and the "variance" term H 2 is small for models that exhibit NC. We also add pane j, which displays the alignment of kernels Θ and Θ h for every model at the end of training. One can see that the kernel alignments is typically stronger in models that exhibit NC.

C.1 Additional examples of the NTK block structure

We include the following additional illustrative figures (analogous to Figure 1 in the main text) that show the NTK block structure in dataset-architecture pairs covered in our experiments:

Figure 13: VGG11 trained on MNIST

Figure 14: VGG11 trained on FashionMNIST

Figure 15: VGG16 trained on CIFAR10

Figure 16: ResNet20 trained on FashionMNIST

Figure 17: ResNet20 trained on CIFAR10

Figure 18: DenseNet40 trained on MNIST

Figure 19: DenseNet40 trained on FashionMNIST

Figure 11: DenseNet40 trained on CIFAR10

Overall, the block structure pattern is visible in the traced kernels in all the figures. As expected, the block structure is more pronounced in the kernels where the final alignment values are higher. While the norms of the "non-diagonal" components of the kernels are generally smaller than the "diagonal" components in panes c) and d), we notice that there is a large variability in the norms of the "diagonal" components in some settings. This means that different neurons of the penultimate layer and different classification heads may contribute to the kernel unequally in some settings. Moreover, certain "non-diagonal" components of the last-layer kernel may have non-negligible effect in some settings. We discuss how one could generalize our analysis to account for these properties of the NTK in Appendix D.

C.2 Preliminary experiments with CE loss

While CE loss is a common choice for training DNN classifiers, our theoretical analysis and the experimental results only cover DNNs trained with MSE loss. For completeness, we provide experimental results for ResNet20 trained on MNIST with CE loss in Figure 12. One can see that

smaller invariant norm and higher invariant alignment correlate with NC in the figure. However, DNNs trained with CE loss overall reach better NC metrics but have much larger norm of the invariant in comparison with DNNs trained with MSE loss.

D Relaxation of the NTK Block-Structure Assumption

In this section, we first derive the dynamics equations of DNNs with a general block structure assumption on the last-layer kernel Θ h (analogous to the equations presented in Theorem 4.1 and Theorem 5.1). Then we discuss a possible relaxation of Assumption 3.2, under which our main result regarding NC in Theorem 5.2 still holds.

D.1 Dynamics under General Block Structure Assumption

We first formulate the most general form of the block structure assumption on Θ h as follows:

Assumption D.1. Assume that Θ h : X × X → R n × n has the following block structure

where A d,c,n ∈ R n × n are arbitrary p.s.d. matrices. Here x c i and x c j are two distinct inputs from the same class, and x c ′ j is an input from class c ′ = c .

̸

This assumption means that every kernel matrix Θ h k,s ( X ) , k, s ∈ [1 , n ] still has at most three distinct values, corresponding to the inter-class, intra-class, and the diagonal values of the kernel. However, these values are arbitrary and may depend on the choice of k, s ∈ [1 , n ] .

Under the general block structure assumption, the gradient flow dynamics of DNNs with MSE loss takes the following form:

This is the generalized version of the dynamics presented in Theorem 4.1. Consequently, the decomposed dynamics presented in Theorem 5.1 takes the following form under the general block structure assumption:

The derivation of the above dynamics equations are identical to the proofs of Theorem 4.1 and Theorem 5.1 presented in Appendix B.

Rotation invariance We notice that the dynamics of ( W , H ) in (72) has to be rotation invariant, i.e., the equations should not be affected by a change of variables W → WQ , H → Q ⊤ H for any orthogonal matrix Q . This holds since the loss function only depends on the product WH , which does not change under rotation. This requirement puts conditions on the behavior of A d,c,n under rotation. Indeed, assume that the rotation W → WQ , H → Q ⊤ H for some Q corresponds to the following change of the kernel:

then the rotation invariance of the dynamics implies the following equality for any Q :

These equations are satisfied trivially with our initial assumption, where A d,c,n = ˜ A d,c,n ( Q ) ∝ I n . However, as we can see, any generalized assumption should specify the behavior of the kernel under rotation, and satisfy the above equation.

For general A d,c,n , the following behavior under rotation trivially satisfies the above condition: ˜ A d,c,n ( Q ) = Q ⊤ A d,c,n Q . This behaviour of the kernel under rotation is intuitive, since it implies that the gradients of the last-layer features h are rotated in the same way as the features. However, we note that gradients of parametrized functions do not in general behave this way, since the rotation of the function has to be realized by a certain change of parameters. Consider, for instance, a one-hidden-layer linear network with weights V in the first layer. Then we have H = V X , and a rotation H → Q ⊤ H corresponds to the change of parameters V → Q ⊤ V . In this case, the kernel does not change under rotation, i.e., ˜ A d,c,n ( Q ) = A d,c,n .

Dynamics invariant We note that the dynamics in 73 does not in general have an invariant analogous to the one we identified in Theorem 5.1. Indeed, if we define a quantity E := W ⊤ W -c 1 H 1 H ⊤ 1 -c 2 H 2 H ⊤ 2 for some constants c 1 , 2 ∈ R , and additionally assume centered global means H 1 1 C = 0 , we get the following expression for the derivative of E :

which is not equal to zero with arbitrary matrices A d,c .

D.2 Neural Collapse under Relaxed Block Structure Assumption

We now propose a relaxation of our main assumption, under which our main result regarding NC in Theorem 5.2 still holds. In terms of Assumption D.1 on the general block structure of Θ h , our initial Assumption 3.2 in the main text is the special case with A n = κ n I n , A c = ( κ c -κ n ) I n , A d = ( κ d -κ c ) I n . The relaxed assumption can be formulated as follows in terms of matrices A d,c,n :

Assumption D.2. Assume that A n is an arbitrary p.s.d. matrix and ( A c , A d ) satisfy the following conditions:

where N ⊤ c,d ∈ ker( R ⊤ W ) , i.e., N c,d W ⊤ R = O . Further, assume that the kernel changes under rotation with an orthogonal matrix Q as follows:

Since A n is arbitrary, this relaxation allows arbitrary non-zero values of non-diagonal kernels Θ h k,s with k = s . The following observations justify the consistency of the above assumption:

̸

Since W ∈ R C × n , R ∈ R C × N and N > n > C , R ⊤ W has a non-empty kernel (possibly time-dependent).

The dynamics is rotation invariant under the assumption, i.e., the equation (75) holds.

The expression of the assumption is rotation invariant, in a sense that ˜ A d,c ( Q ) = κ d,c I n + ˜ N d,c ( Q ) , where ˜ N ⊤ d,c ∈ ker( R ⊤ WQ ) for any orthogonal Q .

Under the above assumption, the derivative in 77 becomes zero, so the dynamics has an invariant of the form E := W ⊤ W -c 1 H 1 H ⊤ 1 -c 2 H 2 H ⊤ 2 . Moreover, the statement and the proof of our main Theorem 5.2 remains unchanged. Thus, DNNs satisfying the conditions of Theorem 5.2 display NC under Assumption D.2.

D.3 Discussion

The analysis of the DNNs dynamics is simplified significantly by assuming that Θ h has a block structure. However, formulating a reasonable and consistent assumption on the NTK and its components is non-trivial. The Assumption 3.2 that we used in the main text is justified by the empirical results but may not capture all the relevant properties of the NTK. We believe that studying DNNs' dynamics under a more general or a more reasonable assumption on the NTK is a promising future work direction. The relaxed block structure assumption proposed in this section is the first step into this direction.

f)

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

200

400

600

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.007

Acc.:

99

.

36

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

88

LeCun normal init., LR=0.348

Acc.:

99

.

55

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

88

Uniform init., LR=0.007

Acc.:

99

.

58

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

76

Uniform init., LR=0.131

Acc.:

99

.

58

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

98

He normal init., LR=0.013

Acc.:

99

.

55

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

74

He normal init., LR=0.131

Acc.:

99

.

57

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

95

0

100

200

300

400

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

c)

NC1

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

d)

NC2

0

100

200

300

400

0

.

0

0

.

5

1

.

0

1

.

5

e)

NC3

0

.

98

0

.

99

Test accuracy

NC1

0

.

98

0

.

99

NC2

10

-

3

10

-

1

Learning rate

0

.

98

0

.

99

NC3

0

.

00

0

.

25

0

.

50

0

.

00

0

.

25

0

.

50

0

.

75

g)

Uniform init.

0

.

00

0

.

25

0

.

50

0

.

75

h)

He normal init.

0

.

98

0

.

99

1

.

00

0

.

98

0

.

99

1

.

00

10

-

3

10

-

1

Learning rate

0

.

98

0

.

99

1

.

00

0

.

0

0

.

5

1

.

0

1

.

5

f)

LeCun normal init.

0

.

0

0

.

5

1

.

0

1

.

5

g)

Uniform init.

0

.

0

0

.

5

1

.

0

1

.

5

h)

He normal init.

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.007

Acc.:

99

.

36

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

88

LeCun normal init., LR=0.348

Acc.:

99

.

55

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

88

Uniform init., LR=0.007

Acc.:

99

.

58

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

76

Uniform init., LR=0.131

Acc.:

99

.

58

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

98

He normal init., LR=0.013

Acc.:

99

.

55

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

74

He normal init., LR=0.131

Acc.:

99

.

57

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

95

0

100

200

300

400

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

d)

NC2

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

e)

NC3

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

200

400

600

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

-

1

10

1

10

3

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

0

.

75

LeCun normal init.

f)

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

1

10

3

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.007

Acc.:

93

.

12

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

66

LeCun normal init., LR=0.348

Acc.:

93

.

24

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

87

Uniform init., LR=0.007

Acc.:

93

.

39

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

54

Uniform init., LR=0.131

Acc.:

93

.

07

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

91

He normal init., LR=0.013

Acc.:

92

.

66

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

76

He normal init., LR=0.131

Acc.:

93

.

35

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

81

0

100

200

300

400

0

.

0

0

.

1

0

.

2

0

.

3

c)

NC1

0

100

200

300

400

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

d)

NC2

0

100

200

300

400

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

e)

NC3

0

.

98

0

.

99

Test accuracy

NC1

0

.

98

0

.

99

NC2

10

-

3

10

-

1

Learning rate

0

.

98

0

.

99

NC3

0

.

00

0

.

25

0

.

50

0

.

00

0

.

25

0

.

50

0

.

75

g)

Uniform init.

0

.

00

0

.

25

0

.

50

0

.

75

h)

He normal init.

0

.

90

0

.

91

0

.

92

0

.

93

0

.

94

0

.

90

0

.

91

0

.

92

0

.

93

0

.

94

10

-

2

10

-

1

Learning rate

0

.

90

0

.

91

0

.

92

0

.

93

0

.

94

0

.

0

0

.

5

1

.

0

1

.

5

f)

LeCun normal init.

0

.

0

0

.

5

1

.

0

1

.

5

g)

Uniform init.

0

.

0

0

.

5

1

.

0

1

.

5

h)

He normal init.

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.007

Acc.:

93

.

12

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

66

LeCun normal init., LR=0.348

Acc.:

93

.

24

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

87

Uniform init., LR=0.007

Acc.:

93

.

39

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

54

Uniform init., LR=0.131

Acc.:

93

.

07

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

91

He normal init., LR=0.013

Acc.:

92

.

66

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

76

He normal init., LR=0.131

Acc.:

93

.

35

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

81

0

100

200

300

400

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

d)

NC2

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

e)

NC3

0

.

75

LeCun normal init.

f)

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

1000

2000

3000

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.007

Acc.:

78

.

72

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

50

LeCun normal init., LR=0.131

Acc.:

87

.

25

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

88

Uniform init., LR=0.007

Acc.:

82

.

91

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

59

Acc.:

89

.

28

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

87

He normal init., LR=0.018

Acc.:

86

.

34

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

86

He normal init., LR=0.068

Acc.:

88

.

78

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

90

0

100

200

300

400

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

c)

NC1

0

100

200

300

400

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

d)

NC2

0

100

200

300

400

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

e)

NC3

0

.

98

0

.

99

Test accuracy

NC1

0

.

98

0

.

99

NC2

10

-

3

10

-

1

Learning rate

0

.

98

0

.

99

NC3

0

.

00

0

.

25

0

.

50

0

.

00

0

.

25

0

.

50

0

.

75

g)

Uniform init.

0

.

00

0

.

25

0

.

50

0

.

75

h)

He normal init.

0

.

6

0

.

7

0

.

8

0

.

9

0

.

6

0

.

7

0

.

8

0

.

9

10

-

2

10

-

1

Learning rate

0

.

6

0

.

7

0

.

8

0

.

9

0

.

0

0

.

5

1

.

0

1

.

5

f)

LeCun normal init.

0

.

0

0

.

5

1

.

0

1

.

5

g)

Uniform init.

0

.

0

0

.

5

1

.

0

1

.

5

h)

He normal init.

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.007

Acc.:

78

.

72

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

50

LeCun normal init., LR=0.131

Acc.:

87

.

25

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

88

Uniform init., LR=0.007

Acc.:

82

.

91

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

59

Acc.:

89

.

28

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

87

He normal init., LR=0.018

Acc.:

86

.

34

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

86

He normal init., LR=0.068

Acc.:

88

.

78

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

90

0

100

200

300

400

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

d)

NC2

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

e)

NC3

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

1

10

3

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

0

.

75

LeCun normal init.

f)

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

200

400

600

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.003

Acc.:

99

.

49

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

45

LeCun normal init., LR=0.049

Acc.:

99

.

69

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

93

Uniform init., LR=0.003

Acc.:

99

.

60

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

51

Uniform init., LR=0.049

Acc.:

99

.

69

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

88

He normal init., LR=0.005

Acc.:

99

.

51

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

38

He normal init., LR=0.049

Acc.:

99

.

64

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

92

0

100

200

300

400

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

d)

NC2

0

100

200

300

400

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

e)

NC3

0

.

98

0

.

99

Test accuracy

NC1

0

.

98

0

.

99

NC2

10

-

3

10

-

1

Learning rate

0

.

98

0

.

99

NC3

0

.

00

0

.

25

0

.

50

0

.

00

0

.

25

0

.

50

0

.

75

g)

Uniform init.

0

.

00

0

.

25

0

.

50

0

.

75

h)

He normal init.

0

.

98

0

.

99

1

.

00

0

.

98

0

.

99

1

.

00

10

-

3

10

-

1

Learning rate

0

.

98

0

.

99

1

.

00

0

.

0

0

.

5

1

.

0

1

.

5

f)

LeCun normal init.

0

.

0

0

.

5

1

.

0

1

.

5

g)

Uniform init.

0

.

0

0

.

5

1

.

0

1

.

5

h)

He normal init.

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.003

Acc.:

99

.

49

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

45

LeCun normal init., LR=0.049

Acc.:

99

.

69

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

93

Uniform init., LR=0.003

Acc.:

99

.

60

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

51

Uniform init., LR=0.049

Acc.:

99

.

69

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

88

He normal init., LR=0.005

Acc.:

99

.

51

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

38

He normal init., LR=0.049

Acc.:

99

.

64

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

92

0

100

200

300

400

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

d)

NC2

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

e)

NC3

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

200

400

600

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

1

10

2

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

0

.

75

LeCun normal init.

f)

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

200

400

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.005

Acc.:

91

.

82

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

63

LeCun normal init., LR=0.094

Acc.:

93

.

84

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

74

Uniform init., LR=0.005

Acc.:

93

.

00

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

56

Acc.:

93

.

71

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

70

He normal init., LR=0.007

Acc.:

92

.

53

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

57

He normal init., LR=0.068

Acc.:

93

.

64

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

88

0

100

200

300

400

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

d)

NC2

0

100

200

300

400

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

e)

NC3

0

.

98

0

.

99

Test accuracy

NC1

0

.

98

0

.

99

NC2

10

-

3

10

-

1

Learning rate

0

.

98

0

.

99

NC3

0

.

00

0

.

25

0

.

50

0

.

00

0

.

25

0

.

50

0

.

75

g)

Uniform init.

0

.

00

0

.

25

0

.

50

0

.

75

h)

He normal init.

0

.

90

0

.

92

0

.

94

0

.

90

0

.

92

0

.

94

10

-

2

10

-

1

Learning rate

0

.

90

0

.

92

0

.

94

0

.

0

0

.

5

1

.

0

1

.

5

f)

LeCun normal init.

0

.

0

0

.

5

1

.

0

1

.

5

g)

Uniform init.

0

.

0

0

.

5

1

.

0

1

.

5

h)

He normal init.

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.005

Acc.:

91

.

82

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

63

LeCun normal init., LR=0.094

Acc.:

93

.

84

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

74

Uniform init., LR=0.005

Acc.:

93

.

00

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

56

Acc.:

93

.

71

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

70

He normal init., LR=0.007

Acc.:

92

.

53

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

57

He normal init., LR=0.068

Acc.:

93

.

64

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

88

0

100

200

300

400

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

d)

NC2

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

e)

NC3

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

1

10

2

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

1

10

2

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

0

.

75

LeCun normal init.

f)

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

200

400

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.007

Acc.:

69

.

19

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

66

LeCun normal init., LR=0.068

Acc.:

87

.

64

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

45

Uniform init., LR=0.004

Acc.:

69

.

40

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

48

Uniform init., LR=0.068

Acc.:

87

.

03

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

47

He normal init., LR=0.018

Acc.:

81

.

89

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

40

He normal init., LR=0.068

Acc.:

86

.

53

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

77

0

100

200

300

400

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

c)

NC1

0

100

200

300

400

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

d)

NC2

0

100

200

300

400

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

e)

NC3

0

.

98

0

.

99

Test accuracy

NC1

0

.

98

0

.

99

NC2

10

-

3

10

-

1

Learning rate

0

.

98

0

.

99

NC3

0

.

00

0

.

25

0

.

50

0

.

00

0

.

25

0

.

50

0

.

75

g)

Uniform init.

0

.

00

0

.

25

0

.

50

0

.

75

h)

He normal init.

0

.

6

0

.

7

0

.

8

0

.

9

0

.

6

0

.

7

0

.

8

0

.

9

10

-

2

10

-

1

Learning rate

0

.

6

0

.

7

0

.

8

0

.

9

0

.

0

0

.

5

1

.

0

1

.

5

f)

LeCun normal init.

0

.

0

0

.

5

1

.

0

1

.

5

g)

Uniform init.

0

.

0

0

.

5

1

.

0

1

.

5

h)

He normal init.

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.007

Acc.:

69

.

19

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

66

LeCun normal init., LR=0.068

Acc.:

87

.

64

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

45

Uniform init., LR=0.004

Acc.:

69

.

40

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

48

Uniform init., LR=0.068

Acc.:

87

.

03

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

47

He normal init., LR=0.018

Acc.:

81

.

89

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

40

He normal init., LR=0.068

Acc.:

86

.

53

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

77

0

100

200

300

400

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

d)

NC2

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

e)

NC3

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

1

10

2

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

1

10

2

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

1

10

2

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

0

.

75

LeCun normal init.

f)

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

1

10

2

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

1

10

2

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

1

10

2

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

10

-

1

10

0

10

1

10

2

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.004

Acc.:

99

.

21

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

53

LeCun normal init., LR=0.049

Acc.:

99

.

64

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

75

Uniform init., LR=0.004

Acc.:

99

.

36

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

61

Uniform init., LR=0.049

Acc.:

99

.

67

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

79

He normal init., LR=0.007

Acc.:

99

.

48

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

47

He normal init., LR=0.049

Acc.:

99

.

65

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

72

0

100

200

300

400

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

d)

NC2

0

100

200

300

400

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

e)

NC3

0

.

98

0

.

99

Test accuracy

NC1

0

.

98

0

.

99

NC2

10

-

3

10

-

1

Learning rate

0

.

98

0

.

99

NC3

0

.

00

0

.

25

0

.

50

0

.

00

0

.

25

0

.

50

0

.

75

g)

Uniform init.

0

.

00

0

.

25

0

.

50

0

.

75

h)

He normal init.

0

.

98

0

.

99

1

.

00

0

.

98

0

.

99

1

.

00

10

-

2

10

-

1

10

0

Learning rate

0

.

98

0

.

99

1

.

00

0

.

0

0

.

5

1

.

0

1

.

5

f)

LeCun normal init.

0

.

0

0

.

5

1

.

0

1

.

5

g)

Uniform init.

0

.

0

0

.

5

1

.

0

1

.

5

h)

He normal init.

10

-

1

10

0

10

1

10

2

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.004

Acc.:

99

.

21

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

53

LeCun normal init., LR=0.049

Acc.:

99

.

64

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

75

Uniform init., LR=0.004

Acc.:

99

.

36

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

61

Uniform init., LR=0.049

Acc.:

99

.

67

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

79

He normal init., LR=0.007

Acc.:

99

.

48

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

47

He normal init., LR=0.049

Acc.:

99

.

65

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

72

0

100

200

300

400

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

d)

NC2

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

e)

NC3

0

.

75

LeCun normal init.

f)

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

1

10

2

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.007

Acc.:

93

.

00

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

62

LeCun normal init., LR=0.094

Acc.:

94

.

01

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

75

Uniform init., LR=0.007

Acc.:

92

.

95

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

69

Acc.:

94

.

08

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

75

He normal init., LR=0.007

Acc.:

91

.

71

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

56

Acc.:

94

.

16

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

77

0

100

200

300

400

0

.

06

0

.

08

0

.

10

c)

NC1

0

100

200

300

400

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

d)

NC2

0

100

200

300

400

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

e)

NC3

0

.

98

0

.

99

Test accuracy

NC1

0

.

98

0

.

99

NC2

10

-

3

10

-

1

Learning rate

0

.

98

0

.

99

NC3

0

.

00

0

.

25

0

.

50

0

.

00

0

.

25

0

.

50

0

.

75

g)

Uniform init.

0

.

00

0

.

25

0

.

50

0

.

75

h)

He normal init.

0

.

90

0

.

92

0

.

94

0

.

90

0

.

92

0

.

94

10

-

2

10

-

1

10

0

Learning rate

0

.

90

0

.

92

0

.

94

0

.

0

0

.

5

1

.

0

1

.

5

f)

LeCun normal init.

0

.

0

0

.

5

1

.

0

1

.

5

g)

Uniform init.

0

.

0

0

.

5

1

.

0

1

.

5

h)

He normal init.

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.007

Acc.:

93

.

00

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

62

LeCun normal init., LR=0.094

Acc.:

94

.

01

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

75

Uniform init., LR=0.007

Acc.:

92

.

95

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

69

Acc.:

94

.

08

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

75

He normal init., LR=0.007

Acc.:

91

.

71

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

56

Acc.:

94

.

16

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

77

0

100

200

300

400

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

d)

NC2

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

e)

NC3

0

.

75

LeCun normal init.

f)

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

1

10

2

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.007

Acc.:

76

.

99

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

57

LeCun normal init., LR=0.094

Acc.:

88

.

31

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

70

Uniform init., LR=0.007

Acc.:

78

.

71

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

64

Acc.:

88

.

33

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

68

He normal init., LR=0.007

Acc.:

70

.

19

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

49

Acc.:

88

.

34

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

72

0

100

200

300

400

0

.

08

0

.

10

0

.

12

0

.

14

0

.

16

c)

NC1

0

100

200

300

400

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

d)

NC2

0

100

200

300

400

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

e)

NC3

0

.

98

0

.

99

Test accuracy

NC1

0

.

98

0

.

99

NC2

10

-

3

10

-

1

Learning rate

0

.

98

0

.

99

NC3

0

.

00

0

.

25

0

.

50

0

.

00

0

.

25

0

.

50

0

.

75

g)

Uniform init.

0

.

00

0

.

25

0

.

50

0

.

75

h)

He normal init.

0

.

6

0

.

7

0

.

8

0

.

9

0

.

6

0

.

7

0

.

8

0

.

9

10

-

2

10

-

1

10

0

Learning rate

0

.

6

0

.

7

0

.

8

0

.

9

0

.

0

0

.

5

1

.

0

1

.

5

f)

LeCun normal init.

0

.

0

0

.

5

1

.

0

1

.

5

g)

Uniform init.

0

.

0

0

.

5

1

.

0

1

.

5

h)

He normal init.

10

-

2

10

-

1

10

0

10

1

a)

Invariant norm

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.007

Acc.:

76

.

99

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

57

LeCun normal init., LR=0.094

Acc.:

88

.

31

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

70

Uniform init., LR=0.007

Acc.:

78

.

71

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

64

Acc.:

88

.

33

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

68

He normal init., LR=0.007

Acc.:

70

.

19

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

49

Acc.:

88

.

34

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

72

0

100

200

300

400

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

c)

NC1

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

d)

NC2

0

100

200

300

400

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

e)

NC3

0

.

75

LeCun normal init.

0

.

98

0

.

99

1

.

00

0

.

98

0

.

99

1

.

00

10

-

3

10

-

1

Learning rate

0

.

98

0

.

99

1

.

00

0

.

0

0

.

5

1

.

0

1

.

5

f)

LeCun normal init.

0

.

0

0

.

5

1

.

0

1

.

5

g)

Uniform init.

0

.

0

0

.

5

1

.

0

1

.

5

h)

He normal init.

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

10

2

10

3

i)

Features norms

no NC

NC

init.,

no NC

init.,

NC

He. init.,

no NC

He. init.,

NC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

j)

Kernels alignment

‖

H

1

H T

1

‖

F

‖

H

2

H T

2

‖

F

‖〈

h

h

〉

T

‖

F

〈

Θ

/

‖

Θ

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

〈

Θ

h /

‖

Θ

h

‖

F, Y Y

T /

‖

Y Y

T

‖

F

〉

10

0

10

1

a)

Invariant norm

0

.

4

0

.

6

0

.

8

1

.

0

b)

Inv. alignment to

W T W

LeCun normal init., LR=0.003

Acc.:

99

.

41

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

56

LeCun normal init., LR=0.049

Acc.:

99

.

69

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

86

Uniform init., LR=0.003

Acc.:

99

.

56

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

70

Uniform init., LR=0.049

Acc.:

99

.

61

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

82

He normal init., LR=0.003

Acc.:

99

.

23

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

48

He normal init., LR=0.049

Acc.:

99

.

61

%,

〈

Θ

h

, Y Y

T

〉

:

0

.

87

0

100

200

300

400

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

c)

NC1

0

100

200

300

400

0

.

2

0

.

4

0

.

6

0

.

8

d)

NC2

0

100

200

300

400

0

.

2

0

.

4

0

.

6

0

.

8

e)

NC3

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

a)

∑

k

Θ

k,k

(

X

)

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

b)

∑

k

Θ

h

k,k

(

X

)

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

c)

‖

Θ

k,s

‖

F

h

1

h

43

h

85

h

127

h

169

h

211

h

253

h

1

h

43

h

85

h

127

h

169

h

211

h

253

d)

‖

Θ

h

k,s

‖

F

0

100

200

300

400

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

e)

Alignment

〈

Θ

,Y Y

T

〉

‖

Θ

‖

F

‖

Y Y T

‖

F

〈

Θ

h

,Y Y

T

〉

‖

Θ

h

‖

F

‖

Y Y T

‖

F

0.01

0.1

0.2

0.5

0

0.5

1

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

a)

∑

k

Θ

k,k

(

X

)

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

b)

∑

k

Θ

h

k,k

(

X

)

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

c)

‖

Θ

k,s

‖

F

h

1

h

43

h

85

h

127

h

169

h

211

h

253

h

1

h

43

h

85

h

127

h

169

h

211

h

253

d)

‖

Θ

h

k,s

‖

F

0

100

200

300

400

0

.

6

0

.

7

0

.

8

0

.

9

e)

Alignment

〈

Θ

,Y Y

T

〉

‖

Θ

‖

F

‖

Y Y T

‖

F

〈

Θ

h

,Y Y

T

〉

‖

Θ

h

‖

F

‖

Y Y T

‖

F

0.01

0.1

0.2

0.5

0

0.5

1

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

a)

∑

k

Θ

k,k

(

X

)

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

b)

∑

k

Θ

h

k,k

(

X

)

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

c)

‖

Θ

k,s

‖

F

h

1

h

43

h

85

h

127

h

169

h

211

h

253

h

1

h

43

h

85

h

127

h

169

h

211

h

253

d)

‖

Θ

h

k,s

‖

F

0

100

200

300

400

0

.

4

0

.

6

0

.

8

e)

Alignment

〈

Θ

,Y Y

T

〉

‖

Θ

‖

F

‖

Y Y T

‖

F

〈

Θ

h

,Y Y

T

〉

‖

Θ

h

‖

F

‖

Y Y T

‖

F

0.01

0.1

0.2

0.5

0

0.5

1

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

a)

∑

k

Θ

k,k

(

X

)

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

b)

∑

k

Θ

h

k,k

(

X

)

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

c)

‖

Θ

k,s

‖

F

h

1

h

11

h

21

h

31

h

41

h

51

h

61

h

1

h

11

h

21

h

31

h

41

h

51

h

61

d)

‖

Θ

h

k,s

‖

F

0

100

200

300

400

0

.

4

0

.

5

0

.

6

0

.

7

e)

Alignment

〈

Θ

,Y Y

T

〉

‖

Θ

‖

F

‖

Y Y T

‖

F

〈

Θ

h

,Y Y

T

〉

‖

Θ

h

‖

F

‖

Y Y T

‖

F

0.01

0.1

0.2

0.5

0

0.5

1

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

a)

∑

k

Θ

k,k

(

X

)

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

b)

∑

k

Θ

h

k,k

(

X

)

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

c)

‖

Θ

k,s

‖

F

h

1

h

11

h

21

h

31

h

41

h

51

h

61

h

1

h

11

h

21

h

31

h

41

h

51

h

61

d)

‖

Θ

h

k,s

‖

F

0

100

200

300

400

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

e)

Alignment

〈

Θ

,Y Y

T

〉

‖

Θ

‖

F

‖

Y Y T

‖

F

〈

Θ

h

,Y Y

T

〉

‖

Θ

h

‖

F

‖

Y Y T

‖

F

0.01

0.1

0.2

0.5

0

0.5

1

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

a)

∑

k

Θ

k,k

(

X

)

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

b)

∑

k

Θ

h

k,k

(

X

)

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

c)

‖

Θ

k,s

‖

F

h

1

h

23

h

45

h

67

h

89

h

111

h

1

h

23

h

45

h

67

h

89

h

111

d)

‖

Θ

h

k,s

‖

F

0

100

200

300

400

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

e)

Alignment

〈

Θ

,Y Y

T

〉

‖

Θ

‖

F

‖

Y Y T

‖

F

〈

Θ

h

,Y Y

T

〉

‖

Θ

h

‖

F

‖

Y Y T

‖

F

0.01

0.1

0.2

0.5

0

0.5

1

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

a)

∑

k

Θ

k,k

(

X

)

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

b)

∑

k

Θ

h

k,k

(

X

)

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

c)

‖

Θ

k,s

‖

F

h

1

h

23

h

45

h

67

h

89

h

111

h

1

h

23

h

45

h

67

h

89

h

111

d)

‖

Θ

h

k,s

‖

F

0

100

200

300

400

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

e)

Alignment

〈

Θ

,Y Y

T

〉

‖

Θ

‖

F

‖

Y Y T

‖

F

〈

Θ

h

,Y Y

T

〉

‖

Θ

h

‖

F

‖

Y Y T

‖

F

0.01

0.1

0.2

0.5

0

0.5

1

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

a)

∑

k

Θ

k,k

(

X

)

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

c

1

c

2

c

3

c

4

c

5

c

6

c

7

c

8

c

9

c

10

b)

∑

k

Θ

h

k,k

(

X

)

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8

f

9

f

10

c)

‖

Θ

k,s

‖

F

h

1

h

31

h

61

h

91

h

121

h

151

h

181

h

1

h

31

h

61

h

91

h

121

h

151

h

181

d)

‖

Θ

h

k,s

‖

F

0

100

200

300

400

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

e)

Alignment

〈

Θ

,Y Y

T

〉

‖

Θ

‖

F

‖

Y Y T

‖

F

〈

Θ

h

,Y Y

T

〉

‖

Θ

h

‖

F

‖

Y Y T

‖

F

0.01

0.1

0.2

0.5

0

0.5

1

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrevious work has observed an increased alignment and emergence of an approximate block structure in a trained network's Neural Tangent Kernel (NTK) as well as the Neural Collapse (NC) phenomenon in the last hidden layer. The paper attempts to connect the two by showing that in an extreme case of perfect block structure in NTK, NC follows from the dynamics of squared error minimization on the kernel.\n\nSTRENGTHS:\nThe paper is extremely well-written and well organized. I did not check the details of the proofs in the appendix but main paper is correct as far I can verify. The relevant literature is well covered in the paper and even more so in the appendix. The invariance derived to establish the connection can be an interesting result on its own and its cooccurrence with NC is demonstrated in realistic experiments.\n\nWEAKNESSES:\nWhile I enjoyed reading the paper, I cannot come to agree with the main premise of the work and so I'm voting for rejection. The results rely on a perfect block structure which is unlikely to be achievable in a realistic setting. Even in a restricted setting where such structure will emerge, it will likely emerge along with NC at the end of training and it is hard to argue that it is a cause or driving force behind NC. I will elaborate on these comments (W1, W2) as well as other weaknesses. Overall, a block NTK would have been interesting as a theoretical model (like unconstrained features) and the derivation of NC from this model is insightful, but the current paper is pushing this model far beyond its domain of applicability by assuming that the NTK of the whole network has this structure.\n\nW1: Perfect block structure in NTK assumes the gradients (of an output) are perfectly orthogonal across data points from different classes. I think a necessary requirement for this would be that the data is already linearly separable at the input or the first hidden layer. Otherwise there is some inevitable interference between the classes in the first few layers where data is not linearly separable, making gradients correlated in those layers and preventing the block structure, no matter how long the network is trained. The paper demonstrates a nearly perfect block structure on the extremely simple task of MNIST and the extremely overparametrized network of ResNet20. The perfect block structure is likely far from realistic in larger tasks and smaller networks.\n\nW2: If the block structure was a property of the NTK at initialization then one could argue for a causal relationship from the block structure to NC. The problem is, both the structure and NC are emergent properties of training dynamics and it is not clear to me why the former would be causing the latter. Previous work arguing that NTK alignment facilitates optimization focuses on alignment already present at initialization or on a partial increase in alignment within an early phase of training. It may be reasonable then to argue that the rest of the training will be faster compared to training with a misaligned NTK. The perfect block structure in this paper is not established as an early phase phenomenon.\n\nW3: I'm surprised by the block structure in the individual hidden layer neurons in Fig 1 (b,d). While the output layer neurons are individually fit to the labels, the hidden layer neurons are updated as a whole through backpropagation. Emergent properties in this layer are then likely to be rotation invariant in general. The fact that each individual neuron is showing a pattern here could be a byproduct of the activation function (ReLU). Does the same pattern emerge with leakyrelu activations (with default parameters)? If not, the current results are limited to ReLU. (I may be mistaken here and perhaps the property is rotation invariant.)\n\nW4: The main result assumes centralized activations and the argument is that Batch Normalization (BN) to some extent enforces this property. Are the representations in the experiments extracted right after a BN layer or right after a ReLU layer? The former would partially satisfy this assumption but the latter would in fact ensure that this is not the case. The text refers to Fig 3 in the appendix but some of the numbers in that figure are ~10. Even if they were smaller, I'm not sure how they would support this assumption since it seems to me that it is not the absolute magnitude of the mean but its magnitude relative to the overall norm of the features that matters.\n\nW5: Previous work on NTK alignment (Baratin et al, Atanasov et al) generally shows a final alignment in the approximate range of 0.1 and 0.5 in different networks and tasks. The kernel alignments in the submission are consistently higher. I understand that different experiments in these works would simply result in different alignment values. Nevertheless the large gap suggests to me that there is a key factor (in training setup or centralizing the kernels or another aspect of the experiment) that differs between the submission and previous work which should be spelled out to prevent the reader from drawing premature conclusions across these papers.\n\n---------\n\n**Post-rebuttal:** I read the other reviews and responses. The rebuttals answered my comment about causality (W2) and so I raised the score. In particular, the response clarified that NTK alignment as measured in the submission is similar to previous work and that it is established as an early-time phenomenon (while NC is a late-time phenomenon). The intro should mention this before claiming a causal relationship. Also showing an example of early increase in alignment instead of the current example in Fig. 1 (e) helps with motivating the causal relationship.\n\nRegarding W1: I agree that the theory doesn't assume a perfect block pattern. My issue is what the title of this paper implies. Unless NTK shows a near perfect block pattern like Fig. 1 (a-d) in more general cases, calling its behavior \"collapse\" is misleading. The revision should either provide results like Fig. 1 (a-d) in more general cases or clarify that this figure is a rare and extreme case.\n\nOther reviewers argued that the new result has little significance beyond what what is known from unconstrained features model. I do not have the expertise to comment on this issue and leave this decision to other reviewers.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
The paper proposes a mechanism behind the empirical phenomenon of Neural Collapse in deep neural networks. The paper derives and analyzes the training dynamics of DNNs with MSE loss and block-structured NTK, identifying three distinct convergence rates in the dynamics.

STRENGTHS:
The strengths of the paper include its theoretical rigor, the clarity of its presentation, and the large-scale numerical experiments that support the theory. The paper also provides a new perspective on the empirical phenomenon of Neural Collapse and identifies the conditions under which it occurs.

WEAKNESSES:
The weaknesses of the paper include the assumption of balanced datasets and the lack of exploration of the effects of non-balanced datasets on the dynamics of DNNs with block-structured NTK. The paper also does not explore the effects of adding stochasticity to the dynamics considered in the paper.

QUESTIONS:
Please dicuss the relationship of your work with this phenomenon [cite1]

[cite 1] Liu D, Wang S, Ren J, et al. Trap of feature diversity in the learning of mlps[J]. arXiv preprint arXiv:2112.00980, 2021.

LIMITATIONS:
The limitations of the paper include the focus on MSE loss and block-structured NTK, which may not be applicable to other loss functions and NTK structures. The paper also does not provide practical solutions to prevent or mitigate Neural Collapse in deep neural networks.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main contribution of the paper \"Neural (Tangent Kernel) Collapse\" is the connection of the Neural Tangent Kernel (NTK) alignment and Neural Collapse (NC) phenomenon in deep neural networks (DNNs). The authors assume that the empirical NTK develops a block structure aligned with the class labels. They derive the dynamics of DNNs trained with mean squared (MSE) loss and identify three different convergence rates for certain components of the error. They also identify a hyperbolic invariant that captures the essence of the dynamics and use it to prove the emergence of NC in DNNs with block-structured NTK. Also, it provides large-scale numerical experiments to support their theory. Overall, the paper provides valuable insights into the dynamics of DNNs with block-structured NTK and the emergence of NC.\n\nSTRENGTHS:\n- Originality: Yes, this is the first work to connect NTK alignment and NC. Another new contribution is exploring the dynamics of DNNs with block-structured Neural Tangent Kernel (NTK).\n\n- Quality: High quality, the paper presents a thorough analysis of the dynamics of DNNs with block-structured Neural Tangent Kernel (NTK) and the emergence of Neural Collapse (NC) phenomenon. Also, they provide large-scale numerical experiments on three common DNN architectures and three benchmark datasets to support their theory.\n\n- Clarity: Yes, the paper is well-structured and clearly explains technical terms and concepts, making it easy to understand for readers.\n\n- Significance: Yes, for explaining the emergence of NC, most theoretical works adopt the unconstrained feature models. However, this paper provides a new point of view of NTK to explain it. It makes a step towards realistic DNN dynamics by means of the NTK.\n\nWEAKNESSES:\nThe main weakness of the paper is that its fundamental assumption 3.2 is not justified enough. It assumes the block structure of the NTK kernel but it’s often not the case in real DNNs which is not well-trained.\n\nSome minor remarks:\n1. I think the subtitles for a) and b) in Figure 1 should be the **sum** over the classes or feature dimensions.\n2. Equation 2: missing part of the parentheses.\n\nQUESTIONS:\n1. This paper only focuses on DNNs trained with mean squared error (MSE) loss. Since Cross Entropy(CE) loss is a common choice for training classification networks, does this dynamics of DNNs and conclusions in this paper also hold for CE loss? \n\n2. Can we get some theoretical insights from this work to improve the design and training of deep neural networks?\n\nLIMITATIONS:\nThe authors have addressed their limitations in chapter 7. There is no negative societal impact to be expected from this work", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work provides a theoretical connection between two related phenomena in deep learning dynamics: the structural change to the empirical NTK during training, specifically its alignment with class labels; and the neural collapse, which refers to a set of behaviors exhibited by NNs trained on multiway classification. The authors asked whether the first phenomenon causes the second, and studied NN dynamics under a toy-model NTK that has a block structure.\n\nSTRENGTHS:\nBoth neural collapse and NTK alignment are prominent empirical features of DNN learning (at least in some cases), and the authors provided a careful treatment of a relevant toy model that connects the two. The writing is clear despite the convoluted nature of the subject (speaking as someone who has written on this specific topic before). Assumptions, claims and proven results are easy to find and understand.\n\nWEAKNESSES:\nMy primary concern is with the amount of insight that this work brings.\n\n(1) The central question posed by the authors, in line 37, is stated as \"does NTK alignment lead to neural collapse?\". But I do not understand the cause-and-effect relation implied here. It has been long understood that NTK provides a dual perspective to NN training dynamics. So much as the NTK does not \"lead to\" the reduction of loss in NN training, NTK alignment does not \"lead to\" neural collapse. \n\n(2) My second point is closely related to the first one. The assumptions about the NTK kernel and feature kernel in Sec. 2.2 are motivated by empirical observations of trained NNs and make ensuing analysis simpler. But I'm concerned that they are sufficiently strong to \"guarantee\" neural collapse -- these are very strong assumptions and already imply that the NN representation/dynamics are in a stage where things are \"collapse-y\". In other words, on a conceptual level, this work risks falling into a circular logic -- if we assume the NN is in neural collapse, we can derive that it is in neural collapse. This work does not touch on the topic of how the kind of block-like structure arises, which in my view is what \"causes\" neural collapse (Conceptually, I think the emergence of block-like kernels and NC are two sides of the same coin).\n\n(3) As the authors pointed out, an important implication of NC is the generalization benefits that it brings. The current analysis, however, appears to entirely focus on the training set. This is a limitation shared by previous work on empirical NTK dynamics, though.\n\n(4) A minor point, but I find Sec. 4.1 to be a little excessive. It is well known that if you do regression with a kernel, eigenstructure of the kernel determines how quickly different components of the target vector gets learned. There are plenty of prior work on the subject. Perhaps it is good to cite them and shorten this section.\n\nQUESTIONS:\nI wonder how much the analysis can be generalized to weakened assumptions about the kernels? For example, instead of assuming that the kernel has a block-like structure, can we assume that it has the form K=K0 + Kb, where Kb is the block-like structure proposed by the authors, and K0 is the eNTK at initialization? I think this is a much more realistic assumption about the dynamics of NN learning.\n\nI also welcome additional comments from the authors about the insights (of course, what counts as \"insights\" is different for different people!) that this work brings about the emergence of block structures / NC.\n\nLIMITATIONS:\nPlease see weaknesses. There are no negative societal impact. I appreciate the author discussing the weakness of making the assumptions in 3.2", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper connects NTK (neural tangent kernel) and NC (neural collapse) by assuming NTK has a block structure in the late training stage, meaning the kernel for samples within the same class is much larger than samples from different classes. The technical assumption additionally assumes that the gradients of classification heads are independent of each other, and the gradients of neurons of the penultimate layer are also independent of each other. The authors then claim that the kernel gradient descent of this NTK leads to NC. Empirical results provide the correlation between the block structure and NC.\n\nSTRENGTHS:\nThe paper is the first paper I have seen that builds the connection between NTK and NC. It is inspiring in concept with good novelty.\n\nThe presentation of the paper is overall good and easy to read.\n\nWEAKNESSES:\nI believe the assumption of the paper is too strong because the authors not only assume the block structure of NTK but also assume the independence between gradients of neurons/classification heads for any input. Taking a closer look at Assumption 3.2, one can already derive strong results. For example, one can wrote $\\nabla f_k(x)$ as a function of $\\nabla h_s(x)$, $\\mathbf W$ and $h_s(x)$ by chain rule, and plug into $\\Theta$. Then it is not hard to find that \n\n-  $\\langle \\mathbf W_k, \\mathbf W_s\\rangle$ is identical for any $k\\neq s$;\n- $\\langle \\mathbf W_k, \\mathbf W_k-\\mathbf W_s\\rangle=\\gamma_d/\\kappa_d=\\gamma_c/\\kappa_c=\\gamma_n/\\kappa_n$;\n- $\\langle h(x), h(x')\\rangle =-\\kappa \\langle \\mathbf W_k, \\mathbf W_s\\rangle$ ($\\kappa$ is $\\kappa_d, \\kappa_c$ or $\\kappa_n$ depends on the relationship between $x$ and $x'$).\n\nThose are some implications I derived within an hour. There might be some flaws but the message is clear --- the independence between gradients is a very very strong assumption.\n\nAnother thing I want to point out in the proof is that the training may not be in the kernel regime. The authors assume after hundreds of training epochs, the empirical NTK satisfied the property, but it is not clear whether it changes afterward (meaning it is not in the kernel regime but feature learning regime, which does not support linearization of the model).\n\nQUESTIONS:\nCan you verify the calculation I did above?\n\nLIMITATIONS:\nI do not see any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}