Can Kernel Methods Explain How the Data Affects Neural Collapse?
Authors: Vignesh Kothapalli
Tom Tirer
Venue: Accepted by TMLR ()

Abstract: A vast amount of literature has recently focused on the “Neural Collapse” (NC) phenomenon, which emerges when training neural network (NN) classifiers beyond the zero training error point. The core component of NC is the decrease in the within-class variability of the network’s deepest features, dubbed as NC1. The theoretical works that study NC are typically based on simplified unconstrained features models (UFMs) that mask any effect of the data on the extent of collapse. To address this limitation of UFMs, this paper explores the possibility of analyzing NC1 using kernels associated with shallow NNs. We begin by formulating an NC1 metric as a function of the kernel. Then, we specialize it to the NN Gaussian Process kernel (NNGP) and the Neural Tangent Kernel (NTK), associated with wide networks at initialization and during gradient-based training with a small learning rate, respectively. As a key result, we show that the NTK does not represent more collapsed features than the NNGP for Gaussian data of arbitrary dimensions. This showcases the limitations of data-independent kernels such as NTK in approximating the NC behavior of NNs. As an alternative to NTK, we then empirically explore a recently proposed data-aware Gaussian Process kernel, which generalizes NNGP to model feature learning. We show that this kernel yields lower NC1 than NNGP but may not follow the trends of the shallow NN. Our study demonstrates that adaptivity to data may allow kernel-based analysis of NC, though further advancements in this area are still needed. A nice byproduct of our study is showing both theoretically and empirically that the choice of nonlinear activation function affects NC1 (with ERF yielding lower values than ReLU).

Published in Transactions on Machine Learning Research (04/2025)
Can Kernel Methods Explain How the Data Affects Neural Collapse?
Vignesh Kothapalli
1k2115
Courant Institute of Mathematical Sciences New York University
Tom Tirer
tirer. firer.tom Lown@@gmmiiaa com
Faculty of Engincering
Bar-Hlan University
Reviewed on OpenReview: https: openreview. net/ formm? id= NbF1gyf11
Abstract
vast amount of literature has recently focused on the "Neural Collapse" (NC) phenomenon, which emerges when training neural network (NN) classifiess beyond the zero training error point. The core component. of NC is the decrease in the within-class variability of the network's deepest features, dubbed as NC1. The theoretical works that study NC are typically based on simplified unconstrained features models (UFMs) that mask any effect of the data on the extent of collapse. To address this limitation of UFMs, this paper explores the possibility of analyzing NC1 using kernels associated with shallow NNs. We begin by formulating an NC1 metric as a function of the kernel. Then, we specialize it to the NN Gaussian Process kernel (NNGP) and the Neural Tangent Kernel (NTK), associated with wide networks at initialization and during gradient-based training with a small learning rate, respectively. As key result, we show that the NTK does not represent more collapsed features than the NNGP for Gaussian data of arbitrary dimensions. This showcases the limitations of lata-independent kernels such NTK in approximating the NC behavior of NNs. As an alternative to NTK, we then empirically explore a recently proposed dato-aware Gaussian Process kernel, which generalizes NNGP to model feature learning. We show that this kernel yields lower NC1 than NNGP but may not follow the trends the shallow NN. Our study demonstrates that adaptivity to data may allow kernel-based analysis of NC, though, further advancements in this area are still needed. A nice byproduct of our study showing both theoretically and empirically that the choice of nonlinear activation function affects NC1 (with ERF yielding lower values than ReLU), The code is available at: https://githuu www. httpa//haalllaaaaaaaae
1 Introduction
Deep Neural Network classifiers are often trained beyond the zero training error point (Hoffer et al. 2017 Ma et al., 2018). In this regime, a phenomenon dubbod "Neural Collapse" (NC) emerges (Papyan al, 2020). NC typically described by the following components: (NC1) the networks' deepest features exhibit significant decrease in the variability of within-class samples, (NC2) the mean features of different classes approach certain symmetric structure, and (NC3) the last layer's weights become more aligned with the penultinate layer features' means. This behavior has been observed both when using the cross-entropy (CE) loss (Papyan et al. 2020) and the mean squared error (MSE) loss (Han et al. 2022
Recently, a vast amount of literature has been dedicated exploring NC (as surveyed in (Kohappall] 2023 studying the effect of imbalanced data (Fang et al.) 2021 Thrampoulidis al. (002)) depthwise evolution (Tirer & Bruna 2022
Rangamani et al. 2023 Sukenik al. 2023 He & Su 2023), fine-grained structures

---

Published in Transactions on Machine Learning Research (04/2025)
Tirer et al. 2023 Yang et al.
2023 Kothapalli et al. 2023 and implications (huu et al. 2021 Galanti
et al. 2022: Yang et al. 2022). Note that without a sufficient decrease in the features' within-class variability around the class means, measured by NC1 metrics, one may not gain valuable insights from the structure of the means of different classes. Therefore, oftentimes, NC papers focus specifically on NC1 rather than
on other components of NC (Tirer et al. 2023 Yang el al. 2023: Kothapalli et al., 2023 Galanti et al.
2022: He & Su 2023 Xu & Liu (023). Notably, most of the works that attempt to theoretically analyze
the NC behavior Miixon et al.; 2020; Fang et al. 2021 Zhu et al.
2021; Han et al., 2022 Ji et al.
2022.
Tirer & Bruna 2022 Zhou al. 2022; Thramponlidis et al.; 2022 Yaras et al. 2022 Tirer et al.
2023
Dang et al.] 2023 Sukenik et al. 2023 Wojtowytsch et al. 2020) are based on variants of the unconstrained
features model (UFM) (Mixon et al. 2020), which treats the deepest features of the training samples as free optimization variables. Therefore, such UFM analyses rely solely on the labels and cannot predict the effect. of training data on the extent of collapse.
Since theorctically analyzing the behavior of (doep) NNs is challenging, simplifying approaches that are basod kernel meth (Spchölkopf et al. 2002)) have gained massive popularity (Neal 1995 Lee el. al. 2018 Jacot
iet al. 2018 Chizat et al. 2019 Arora et al. 2019 Lee et al. 2019). Prominent examples include the NN Lee et al. 2018 Matthews ct al. 2018). associated with the
Gaussian Process kernel (NNGP) (Ncal 1995
infinitely wide NNs at initialization, and the Neural Tangent Kernel (NTK) (Jacot et al., 2018). associated with their training in the "lazy regime", where the learning rate is sufficiently small. These approaches, and in particular NTK, were used to provide mathematical reasoning for deep learning phenomena such as achieving zero training loss (Jacot et al., 2018: Arora et al., 2019: Lee al.] 2019), faster learning of lower frequencies
Ronen et al., 2019/ benefits of ResNets over fully connected networks (Huang et al. 2020 Tirer et al. 2022
Barzilai et al.
2022),
usefulness of positional cncoding in coordinated-basedd NNs (Tancik ct al. 2020):
, and
more. More recently, finite width variants of the aforementioned kernels have been studied (Hanin & Nica
2019; Lee et al.)
2020
Seroussi et al.] 2202)). aiming to mitigate the gap between practical NNs behavior and
infinite width analyses [Woodworth et al. 2020 Ghorbani et al. 2020:
Wei et al. 2019 Li et al. 2020
Why Kernels on Gaussian data for studying NC? In this paper, we provide a kernel-based analysis for NC1 (the core component of NC) to bypass the limitations of UFM-based analysis. More importantly, our work aims to highlight the role of data in the study of NC and thus leverages the Gaussian datasets for systematic thcoretical and empirical analysis of NC1 with kernels. This setup allows us to explore how different aspects of the data such as dimension, sample size, class imbalance, and linear separability affect the collapse in a controlled fashion. To this end, as also mentioned above, we focus on (1) data-independent /NGT/NTKKT.
kernels such as NNGP/ /NTK whose functional form does not change with data, and (2) data-aware GP kernels (Seroussi et al. 2023) which model the feature mappings learned by finito-width shallow NNs (especially shallow Fully Connected Networks (FCNs)). Our main contributions can be summarized follows:
Given an arbitrary kernel function, we establish expressions for the traces of the within and between- class covariance matrices of the features - and consequently, an NC1 metric that depends on the features only through the kernel function.
We specialize our kernel-based NC1 to kernels associated with shallow NNs. We analyze it for NNGP and NTK and show that, perhaps surprisingly, the NTK does not represent more collapsed features than the NNGP for Gaussian data.
We analyze the data-aware GP kernel (errussi et al. 2023 which generalizes NNGP to model the feature mapping learned from the training data by shallow NNs. We present empirical results on Gaussian data to show that the adaptivity of such kernels yields lower NC1 and allows them to reasonably approximate the NC1 behavior of shallow NNs for lincarly separable datascts. However, they may fail in regimes with imbalanced and non-lincarly separable data in higher dimensions.
byproduct of our study is showing both thcoretically and empirically that the choice of the noulinear activation function affects NC1. Specifically, we show that using ERF yields lower NC1 (more collapsed features) than using ReI.U.
2

---

Published in Transactions on Machine Learning Research (04/2025)
Paper Organization. We review previous works on the UFM and kernels in the context of NC in Section 2 Section B introduces the setup based on 2 layer FCNs and kernels used throughout the paper. Section
introduces & generic NC1 metric for any arbitrary kernel function. This formulation is then used in Section to theoretically and empirically analyze the NC1 of NNGP and NTK kernels. Note that the NTK feature map considers the FCN architecture and the GD optimization process (although not data-aware) for the NC1 analysis, and allows us to gain insights beyond NNGP (Seleznova et al.) (023)). Owing to the limitations of the NTK, we wish to contrast NNGP with kernel that takes into account both optimization and data. To
this end, we transition to the recently introduced dato-awore GP kernels by (Seroussi et al., 2023) for FCNs
in Section G Finally, we discuss the limitations of such kernels in Section 7 and the conclusion in Section g
2
Related Work
The majority of the works that attempt to analyze NC behavior theoretically Mixon et al.
2020
Fang et al
2021
Zhu et al. 2021
Han et al.) 2022 Ji et al.; 2022 Tirer & Bruna, 2022 Zhou et al.,
2022
Yaras et al.
2022
Dang et al. 2023 Sukenik et al. 2023) are based on variants of the UFM (Mixon et al.
2020). The work
in Tirer et al.) 2023) attempts to generalize the UFM by adding a penalty term to the loss that ensures that
the features matrix is in the vicinity of a predefined matrix. Yet, the model still lacks an explicit connection between the features and the data. In (Wang et al., 2023), the authors avoid optimizing the fcatures directly but assume that the model is linear and that the data nearly orthogonal, which are restrictive assumptions. In (Seleznova et al., 2023), the authors claim that having exact class-wis block structure in the Gram matrix of the empirical NTK on training samples implies NC. Yet, they do not provide reasoning for reaching this collapsed Gram matrix and the analysis is still disconnected from the data.
Here, however, we fully depart from the UFM approach and provide analysis that explicitly depends on the data. Furthermore, unlike most works, our analysis applies to the less studied case where the data is class imbalanced (Fang et al., 2021; Yang et al., 2022, Dang et al., 2023 Thrampoulidis et al.] (022). Our
based analysis utilizes results on NNGP and NTK from (Williams) 1996 Cho & Saul, 2009 Lee et al.
2018 Jacot et al., 2018 Lee et al. 2019 To simplify the analysis, we theoretically analyze kernels associated with shallow fully connected NNs. Focusing on shallow networks is justified by recent works demonstrating
monotonic depthwise evolution of NC1 (Tirer & Bruna 2022; Rangamani et al.) 2023 Sukenik et al. 2023
He & Su 2023: Tirer et al. 2023). Specifically, a data structure that promotes a larger reduction in NC1 for shallow NNs is expected to be more collapsed when using deep NNs. Since NC is related to training, and we show the limitation of NTK to capture it when compared to NNGP, we also utilize the generalization of NNGP that has been proposed in (Seroussi et al.) 2023). In this data-aware GP kernel approach, there is explicit kernel function that depends on the training data. Recently, this richer model has been used to study phase transition behaviors, such as grokking (Rubin et al. 2024), that cannol be captured with the iata-independent kernels.
3 Problem Setup
Data: We consider a dataset X E _ comprising N data points of dimension do belonging to C' classes. Each class has size ce, E [C], where [C] == 21. ,C} and Σ. ne N. The dataset is represented in an "organized" matrix form as X = [x1,1
xlm x211.. x xCne] E RR where x E do represents the 4th data point of the th class. Specific assumptions on the data distribution will be presented during the paper logether with the related theory or experiments.
Neural Network: Unless stated otherwise, we consider a 2-layer fully connected neural network (2L-FCN)
bb : de + dd with 1th layer width d,, (1,2), and point-wise activation function o(-) : R + R. Let w(O E Rdxx 1, ())(0 Te denote the weight and bias parameters of the 1sh layer. At initialization, the
entries (!!))
G ((") are drawn i.id from Gaussian distributions of mean and variance /2/d4-1, of. respectively. For an input de to the network (-),, we denote the 7th component of the output vector (x) R as:
di
da
0x(x)= = ((2)
w (2) i5
d (x)(x),
25(x) = 6/50 +
Σ k=1
w
(1) jk
Ikk
(1)
Ø=1
3

---

Published in Transactions on Machine Learning Research (04/2025)
Task: We train the network *(()) to classify the data points x , [C], [nd] to their respective classes. Let Y, Y E BdxxN denote the prediction and ground truth label matrices respectively:
[
Y [yl
(2)
We aim to minimize the Mean Squared Error (MSE) between Y, Y using the following objective:
1 N Y Y
Σ (1wor]2+ + +|b()(²)
R(v,X.Y) X. Y)
(3)
Note that training classifiers with MSE loss has been shown to be a useful strategy Hui & Belkin 2021
Han
et al. 2022 and is commonly considered in NC analyses (Han et al.
2022
Tirer Bruna 2022
Pre- and Post-activation Kernels: For any two imputs x*. x x o E Rd we denote their corresponding pre- and post-activation features as z "2 E Rd and ((2²)), pz 2) E post-activation kernels corresponding to layer = 1 are given by:
Rd, respectively. The pro and
K (x",, x" (xxxxxx1))) af 2 (b()) + w(ix T (b()+ w(l))e)).
(4)
( x( ((z (of+t)((z''>) (5)
4 Within-Class Variability Metric (NC1) for Kernels
In this section, we derive an NC1 metric that depends on the features only through the kernel function. Let He Nxd be a matrix encapsulating arbitrary feature vectors associated with samples of the C classes. We define the within-class covariance Ew (H) and between-class covariance Eg(H) matrices of the features as:
1 (H)H = N
cF h²) (n=--h) (H) = C Σ h ( (n (n-
(5)
Σ - Σ ci represent the class mean vectors and the where h - Σ c. Vc E [C] and h N
global mean vector, respectively. Additionally, we consider the lotal covariance (H and non-centered between-class covariance EH(H) matrices as follows:
ai hee
ET H
ΣB(H) 3 KE
(6)
Based on these formulations, we define the variability metric NC (H), introduced in Tirer et al. 2023 and
used also in (Kothapalli el. al. 2023 Wang el al. 2023 Yaras el al. 2023
as:
NC, = tr tr (H)
(7)
tr ({
In the following theorem, we formulate the traces tr( tr(Ew(f) (H)) and tr tr(En()) using an arbitrary kernel function Q Rd x Rd R that expresses inner product of data samples in feature space. Theorem 4.1. For any two data points x x x²t,''', let the inner producproduct of their associated features h he's be given by a kernel Q dddo do + as Q(xat, x(d) has has The traces of covoriance matrices tr Ew (H)) and tr(E (H)) can now be formulated as:
C Re
The
(r()w(H))
Qx
Q x()),
(8)
N
c=
tr(E
1 =
Σ
Qx
(
ΣΣ
Th
Q(x x())).
((x)(xx)))
(9)
4

---

Published in Transactions on Machine Learning Research (04/2025)
disn-! sia-112 dleard azandi2
duani Era-Ted dbowr22 42m-000
201
Q
22
O
(a) Balanced classes
(b) Imbalanced classes
Figure 1: Visualizing the kernel matrix Q for the limiting NNGP post-activation kernel function Qar-Erf
R² x R² + R. The data is sampled from two Gaussian distributions in (a) balanced and (b) imbalanced
fashion to illustrate the structure of the sub-matrices Qc,', E {1,2}-
The proof (in Appendix A leverages matrix trace properties and direct expansions of the covariance matrices (H), (H), Ew HH) terms of vector outer-products arrive the results.
Observe that Theorem 4.1 allows to replace Q(:-) with any suitable kernel formulation corresponding to the features. In the following sections, leverage this flexibility to analyze and compare NC1 for kernels that model the behavior of neural networks.
5 Activation Variability of NNGP and NTK
In the UFM-based analysis NC, the assumption is that the deepest features H associated with the training samples are free optimization variables, thus losing any ability to analyze the effect of the training data, apart from the balancedness of its labels Y. this section, we address these shortcomings by analyzing the role of data distributions NC, in the infinite width regime, where the NN behavior be well modeled by NNGP and NTK. In particular, we focus on the case where data is sampled from mixture of Gaussians and understand the limils of NC, (H) reduction.
5.1 Limiting NNGP Kernel
Under the NN model and initialization stated in Section as the hidden layer width d oc, we can characterize the pre-activation kernel K(I) (x** x" d) in terms of the GP limit (Neal 1995) (commonly
referred to as the NNGP limit (Lee et al. 2018 as follows:
(1) (x x 5)
of + -e x x
(10)
do
In this limit, the post-activation kernel activation function ((-) [ee et al.
can have a closed form representation depending on the is:
CPP
2018
2019
The expression for Erf activation (Williams 1996)
QCr-EEr Ex x² x
2 arcsin
2K GP
(x x
(11)
π
2k Cp
(xe:) xxx)
( GP (x xe'
The formulation for ReLU-based kernel Q
Rew (s)) Cho & Saul
2009)
presented in the Appendix
Given a kernel function Q(.-)) and samples X, we can formulate the kernel Gram matrix Q E IR NxN
as:
Q1.1
Qı.
Q(xa. (xc.l
(x xdlme)
Qu,e'
(12)
Qc.
Qc.cJ
Q(x
Qixcito x ax
NxN
n.XRd
Considering the NNGP kernel in [1]. we illustrate an example Q matrix in Figure to visualize the sub-matrices based the imbalance/balance of class sizes.
5

---

Published in Transactions on Machine Learning Research (04/2025)
5.2 Limiting NTK
In the infinite width limit, we also analyze the NTK (Jacot el al. 2018) to understand the effect of optimization on the NN's features in the "lazy regime". Specifically, a well-known result is that in the infinite width limits (with initialization as per Section 3), the doepest feature mapping of the NN is fixed during gradient descent optimization with small enough learning rate, and is characterized by the NTK. Formally, the recursive relationship between the NTK and NNGP (Jacot et al. 2018 Lee et al.) 2019) can be given as follows:
@ NTK
WNTK(X* (x** x J) (2)
d K (x xx J) ()QGr( (x x
(13)
Here, (x x) can be defined using the recursive formulation (eee et al. 2018
Jacot et al.
2018
KGp () or + o ,x"
(14)
Similar to the activation function specific formulations of Q6P (..)) in
11
define the Erf based derivative
kernel Q82 (()-)) as follows:
(( 11+2 x 2K (1 2 x (,S))
2K 2k (('s)) 1) 1 2x (x
-1/2
Car CGP-Ers(x' (x x) 4 det
(15)
The formulation for ReLU-based kernel Q Retu- x (),-) is presented in the Appendix B
Remark on the limiting NTK. Denote by w, o the parameters and the activation function of an L-layer NN ()).. The limiting NNGP is defined directly on the product of neurons, (x,,(((( (xx()))) and provides kernel expressions for the inner product of features at different layers of the network (via recursion similar to ). On the other hand, the limiting NTK is defined as the inner product of the output's gradients W w bx()(x)) (x)). The NTK thcory shows that this can model the inner product only of the deepest features (i.e., the oulput of the penultimate layer Yet, NC particularly considers the deepest. features.
5.3 On the Limitations of NTK to Exhibit 'More' Collapsed Features than NNGP
Notice that even for shallow NNs, is challenging to theoretically analyze NNGP and NTK for general data. Therefore, we consider simplified setting to analyze the NC1 properties of these kernels. Formally, consider a 1-dimensional Gaussian dataset (i.e., do 1) with C = 2 classes. The data points {x² Vi E [mi] belonging to class Similarly, the data points (1.2.), Vj E [#2] belonging to class c = 2 are independently sampled from (N og) and have the labels y 1, Vj E [mz].
are independently sampled from .V (( of) and have the labels y -1, E [m]].
Assumption 1: For μl < 02 0, let 01.02 be small enough such that |ua| >> T1, [p2l > 02 and
vi E [m]]. [ne], 20 < almost surely.
Assumption 2: The dataset X E Nx consists of large enough samples n1,72 > 1.
Assumption 3: The 2L-FCN (+)-) has output layer dimension d2 = 1 and σg + 0.
These assumptions present a scenario where the samples the two classes are sufficiently far from the origin in opposite directions. Thus, the simplest prediction rule pertains to the sign of a sample.
Theorem 5.1 (ReLU Activation). Under Assumptions 1-3, let Φ(-) be the ReLU activation. Denole by
(1) HGP, HNTK the features associated with NNGP QGP-ReLI RELV and NTK O NTK RaLLs respectively. Then:
neH
2
E [HCP) (H(TK))
+ A..o.t
(16)
2
N2
N2
T Bellee
where Ah.o is term that vanishes as }nc} increase.
Note that we denote NNGP (x أو) and NTK (e NTK (x x with different superscripts be consistent. with the literature. Yet. both are associated with the output of the single hidden layer.
(x
6

---

Published in Transactions on Machine Learning Research (04/2025)
-1.0
-1.0
Log (NC1(H))
-1.2
log (NC1(H))
login (NC100))
-1.4
-14
-1.6
N-128
-16
N-128
-18
-1.8
N:756
N=256
N=b12
-18
N=612
N-256
N-1024
N-1024
-20
1010
25
50
75
100
125
25
50
75
100
125
da
(a) QCP-EyS
(b) ÓNTK-RI Rrf
(c) 2L-FCN d: = 500
Figure 2: NCr (H) of (a) the post-activation NNGP kernel (Q
(PGPErT Bey), (b) NTK (O
(1)
-NTK-eff
NTK Beth (c) 2L-FCN
with d1 500 for Erf activation on dataset D (N, do) (as per 117
-0.850
-0.65
00860
N-125
N-258
-0.70
-0.75
N=512
log (NC)(H))
N=512
log (NCI(H))
-0.500
0.75
lognn 10hOTAC1OR)
-0,900
-0.925
-0.80
K=128
-0.900
-0.85
-0.960
-0.9/5
-0.90
N-512
K=1024
-1 0.000
-0.95
25
50
75
100
125
25
50
75
100
125
-1 1.005
(a) QQP RELU
(b) ONTK RELU
(c) 2L-FCN di = 500
Figure 3: WC2 (H) of (a) the post-activation NNGP kernel (Qar- Rotu), (b) NTK (0 -TKKRREUU RELU), (0) 2L-FCN
with d1 500 for ReLU activation dataset D (i(N,d)) (as per
Appendix D presents the proof by calculating the expected values of Q ReLU (x* x dd and employing Theorem For a better understanding of the result, consider the balanced class scenario with n1 n2 = N/2. This gives us: (HG(//TT) = 2 + +02)/ (H1 -2) An.o. which captures the sum of the within-class variance of in the numerator and the between-slass variance the denominator of the first. term.
The Erf activation case. We present similar analysis with the NNGP and NTK with Erf activation in Appendix D (as the terms involved in the formulation are relatively complex than the ReLU case). In the case of ReLU with balanced data, obscrve that the numerator (corresponding to Ew (HGP//TKK)) solely depends on de however, for Erf activation, our analysis shows dependence on terms с of i.e, it depends on inverse of higher powers of class mcans Me as well (see (22 in Appendix Eb. Similar analysis for E (HGP/NTK) with Erf in 1000 shows dlependence on terms x o. Importantly, under Assumptions 1-3, we show similar values of the expected NC1 metric for NNGP and NTK even for the Erf activation.
Key Takeaway. Our results show that with 1-D Gaussian data, the expected NC, (H) of the NTK closely approximates the NNGP counterparts. Perhaps surprisingly, this shows that NTK docs not represent more collapsed leatures than NNGP, despite being associated with NN gradient-based optimization. Namely, we have established another result that shows that training in the lazy regime provably deviates from the
practical fcature learning of NNs (Woodworth et al. 2020 al.) 2020).
Ghorbani et al. 2020 Wei et al. 2019 Yohudai &
Shamir 2019
5.4
Experiments with High Dimensional Data
Sctup. We conduct experiments on datascts with varying sample sizes and input dimensions to verify our thcoretical results and show that insights generalize (e.g., beyond d, = =1). For C = 2, dataset size chosen from {128, 256, 512, 1024), and input dimension do chosen from {1, 8, 32, 128), create the data vector
7

---

Published in Transactions on Machine Learning Research (04/2025)
and label pairs as follows:
Di = 1(xl1) ~ -N(-2* 1 0.25 Ido) gl = -1), vi E [N/2])] u Ui(22 ~ N222 * 1d, 0.25 1.)).' 1/ = vj Vj E [N/2])]
(17)
The vectors and labels from the datasel can then be arranged into the matrix form (as described in the setup) for analysis. The sampling procedure is repeated 10 times for each (N, do).
Erf activation leads to more collapsed features than ReLU. For the low-dimensional case of do and Erf activation, observe from Figure 2a that .VC, (H) has small value of R 10 10-2.1 On the contrary, Figure 3a illustrates that for ReLU, NN, (H) is more than an order of magnitude larger (+s 10-0.96) than the former. Furthermore, Figures 2b and 3b corresponding to NTK (with Erf and ReLU respectively) do not exhibit siguificantly different values from the NNGP counterparis. These observations empirically verify our theoretical results above for d= 1. Furthermore, notice that the NC (H) values are higher in the case of ReLU than Erf across all the dimensions. We also observe such behavior when training the 2L-FCN networks with hidden layer width d1 500 (see Appendix G for training details). When d1 500, the finite width effects of training come into play (see following sections) and we observe a shift in the trend of NC1 (H) for do > 32. Nonetheless, the Erf case in Figure still exhibits more collapse than ReLU case in Figure 3c
NTK does not result in more collapsed features than NNGP in higher dimensions. As the data dimension do increases, (H) increases at similar rates for Qar-Erf and ONTK-Erf- With ReLU activation, (C,(H) remains almost constant for QGP-ReIR ReLU and exhibits an increasing trend for θ ReEU implying less collapse for NTK. All these observations corroborate our theory on the limitation of analyzing NC with NTK. We present additional experimental results for imbalanced datasets in Appendix G and show that for given N, the trends of NC (H) for incrcasing dg can vary based on the imbalance ratio of classes (i.e., ni n1/n)). These observations provide zero-order reasoning for NN behavior where the feature mapping is learned based on data properties such dimension.
6 Activation Variability of data-aware GP Kernels
The explicit kernel formulations in the infinite width limit (d1 + oo) have allowed us to go beyond the unconstrained features assumption and preserve a link between the features and the data. Yet, the NC phenomenon relates to NN training, while NNGP relates to NN at initialization and NTK has been found unsuitable for NC analysis. Thus, we wish to contrast NNGP with kernel that is an alternative to NTK and takes into account both optimization and data. To this end, we transition to a large but finite width (d1 >> 1) and large sample (N >> 1) setting and analyze the data-aware GP kernels by (Seroussi et al. 2023) for FCNs.
6.1 Equations of State (EoS) for the data-aware GP Kernel
A transition from the infinite to finite width regime can introduce various "corrections" to the pre-and post-activations of T-llaye FCN, which account for the different behavior. In this context, (Seroussi et al., 2023) have observed the following dominant corrections: (1) The mean and covariance of the pre-activations deviate from that of a random FCN in a nearly Gaussian manner and, (2) the collective effect of activations from the 11th and (1 - 11tt)th layers determine the covariance of activations the 1th layer. Therefore, while for infinite-width DNNs, the kernels representations are inert, for finite DNNs they adapt to the data and yield tractable data-aware Gaussian Process. Specifically, based on the aforementioned observations, (Seroussi et al. 2023) employ Variational Gaussian Approximation (VGA) approach and derived the following system of equations for the pre and post-activation kernels (-()-,)). -), Q()())))))))))))) 1, [Z. respectively. The solution of this system models the state of the DNN at convergence (after the training phase), and has been empirically shown to yield accurate predictions in various settings. We formally define the EoS for a 2-layer FCN as
follows (based on specializing the generic L-layer formulation in equation in (Seroussi et al. as done in equation 95 in their arxiv extended version):
2023 to L = 2,
8

---

Published in Transactions on Machine Learning Research (04/2025)
Definition 6.1. The "Equations of State" (EoS) for pre and post-activation kernels of 2-lager FCN with Erf aclivation, no bias, and d2 = are given by:
f [0 Q
2 2 da aresin π
2k! (11+22
QQ Ja
(18)
CC [c = do bij
1 tr { A 0c, )n)
+
t
di
A
T) I1T (Q +
Here, C E Rdoxdo models the statistical cocariance of row of w(), inilialized with (ow//do) K >>0 0 is the regularization parameter, and f E R corresponds to the prediction of the 2-lager FCN (governed by the EoS). Additionally, K(),, Q(1) E NNN are the kernel matrices associated with kernel functions (,)(,,), (,Q())))))
CX XC E RNN, >>>>>
Relationship with NNGP. At initialization, we set C= (o which implies K (do) X.
This resulting (()) exactly matches the kernel matrix for the pre-activation GP kernel 10 (x()),
(x xx
(o /d) x x" σb - 0. Similarly, the matrix Q() corresponds to the QGP-Fey( kernel function defined in [11.. The EoS provides mechanism for transitioning from NNGP kernels to finite-width-bae based kernels that adapt to the data. Intuitively, observe that the predictions are formulated based on kernel ridge regression with Q(1) and y. The Q(1) matrix along with and the initial predictions f are then used to update the weight covariance matrix C. Here every entry [C- involves a trace operation on a matrix product, resulting in weighted sum across entries of dc., Q() (i.e the N pairs of data samples).
6.2 Numerical Solutions of EoS
We solve the EoS using the Newton-Krylor method with an annealing schedule (as originally proposed by (Seroussi et al.] 2023)) using the Ecipy.optimin newton _rylov python APL. We iniiialize C' with the GP limit value of (o////)) /d) and choose large annealing factor (ex: 10*) the value for d1. The result of optimizing with newton newton_krylor krylov is new C, which in addition to lower annealing fac- tor is used as an input for the next newton_ krylov newton_krylov function call. This loop is repeated until the end of an annealing schedule. For instance, to analyze the EoS corresponding to d, = 500, we choose the following step-wise annealing factors:
-12
+1.
-0.6
Ne
N=126
N-355
N-250
N:512
N-512
N11024
(a) EoS di = 2000
(b) EoS di 500
Figure 4: NC, (H) of the Q(1) kernel obtained by solv-
ing the EoS (a) d1 = 2000 (b) d1 = 500 on (N,,
factors [10", 9 * 10 ... 2 + 10 10a. 9* 10a
.2.
2 10a 10
500
(19)
stap=-10* 104
step
10a
step 102
Similarly, for choice of d1 = 2000, select the slice of the above list up to 2000. Selecting the schedule is manual operation and can be treated as hyper-parameter: In our experiments, we obscrved that this schedule is sufficient to obtain insights on the NC1 metrics of Q(1) Thus, we leave the exploration of various annealing strategies as future work.
6.3
Experiments
Setup: We train a 2L-FCN with d, = 500, ow = 1, 0 and Erf activation using (vanilla) Gradient
Descent with a learning rate of 10-3 and weight-decay 10-6 for 1000 steps. To numerically solve the EoS, we employ the approach described in Section 6.2 with the final annealing factor of d, 500 and of 1/128 (as per the critical scaling value suggested in (Seroussi et al. 2023

---

Published in Transactions on Machine Learning Research (04/2025)
0.1
Be320
0.16
0.16
0.0
No512
0.14
0.14
0.03
R- 1004
0.12
M-128
0.12
-4.3
6.00
3.0
4:256
N-512
k-512
N-235
R=1
R-256
0.08
N-1024
2.08
N-1024
R-232
01
25
50
75
100
125
25
50
75
100
125
%
66
(a) QGP-Ees Erf
(b) ONTK-Er
(c) EoS
(d) 2L-FCN
Figure 5: NCI of the limiting kernels, adaptive kernel (EoS) with final annealing factor di 500 and
2L-FCN with d1 500 and Erf activation. The dimension do on the x-axis is chosen from {8, 16, 32, 64, 128). For particular N, we sample the vectors x ~ N Idg 4 * Id,), y = =11, E [N/2] for class 1 and 2 N(22 + do 4 I 1, E [N/2] for class 2.
NC1 of EoS on linearly separable datasets. We solve the EoS (initialized with C (d and obtain the stable state using the Newton-Krylov method with an annealing schedule, as described above. At initialization, Q()) is exactly described by the limiting NNGP kernel matrix, which has boen analyzed in the previous section. Now, by solving the EoS with the final annealing factors as 2000 and 500 (which correspond to 2L.-FCN with hidden layer widths d1 = 2000, 500 respectively), we illustrate the NC1 metrics of Q()) in Figure 4 for the rumning example of parameterized balanced 2 class datasets Di (NV, do) (7). Notice that for di = 2000, the NC1 values for Q in Figure 4a closely resemble the plots for the limiting NNGP kernel QGP-Ers Figure 2a and the NTK Figure 2b However, we observe noticeable changes the metries when d1 = 500. Especially, for do 2 8 and N 2 512, the (H) values for the EoS in Figure 4b exhibit noticeable reduction compared to QGP- CCP-Erf and ONTK-Fef- Furthermore, notice that such reduction aligns with the behavior of 2L-FCN in Figure 2c and reflects the departure of Q() in EoS from the initial NNGP state to a feature learning state.
NC1 of EoS on non-lincarly separable datasets. Unlike the separable Gaussian datasets in the above sections, the real-world datasets are relatively more complex. To simulate such scenario, sample x ~ NV-2 # 1d. 4 * I -1, E [N/2] for class and x2 ~ N222 # Idy 4 # It.). 8²vd = 1, E [N/2] for class 2 of our dataset. Essentially, these are scenarios where there a significant. overlap between samples of the two classes. First, we note that we had to increase the learning rate of our 2L-FCN from 10 to 5 10-3 and run GD for 2000 epochs for convergence. For dimensions dg = {8, 16, 32), the EoS reasonably approximates the 2L-FCN but for do {64, 128), the (C,(H) values for 2L-FCN turned out to be almost twice large as the EoS (see Figure 5p. We also present additional experiments with more number of classes, imbalanced classes and deeper FCN networks in Appendix G
7 Limitations
We highlight the difficulties in the theoretical/ /empiriiall analysis of NC1 with FoS. The primary bottleneck is a lack of rigorous study on the existence and uniqueness of solutions (As also highlighted by (Seroussi et al.) 2023)). ) Since we deviate from the lazy regime and deal with kernels in the feature learning sctup, we cannot expect simpler closed-form solutions like the limiting NNGP/NTI /NNK for the EoS. However, numerical solutions to the EoS can sometimes be time-consuming and require a manual selection of the annealing schedule. This is a tradeoff that can be improved with future research. Furthermore, the role of scaling N, do, di on NC1 is yet to be fully understood and we hope that our analysis lays the groundwork for such efforts.
Finally, we point the reader to Appendix F for a discussion on relative NC1 metric that explicitly incorporates the variability collapse of the data vectors into the NC1 metric. In particular, we aim to differentiate between settings where the neural network learned meaningful features and learned to classify complex datasots or was simply able to leverage the already collapsed data vectors. Our results showcase that in higher dimensions, the data vectors are 'more' collapsed than the activations themselves. Thus showcasing the limitations of the current. NC1 metrics and encouraging the reader to explore much richer variants.
Remark.
10

---

Published in Transactions on Machine Learning Research (04/2025)
8 Conclusion
In this paper, explored whether kernel-based approaches can help understand the role of data in the emergence of the NC phenomenon. By considering a general kernel function, we first formulated the trace expressions for the variability collapse (NC1) of the features of the data samples. By leveraging these results, we provided thcoretical and empirical results to showcase that the NTK does not represent more collapsed leatures than the NNGP for various Gaussian datasets. During this investigation we showed how the choice of the nonlinear activation function affects NC1. Next, to capture the feature-learning aspects of finite-witth neural networks, we switched a data-aware GP kernel approach whose state equations (EoS) facilitate the transition of the post-activation kernel beyond the NNGP limit. We empirically showed that it yiclds lower NC1 than NNGP but may not be aligned with trends of FCNs. The key message of the paper is that advancements in modeling NNs by adaptive kernels can benefit NC analysis and lead to a considerable leap compared to current UFM based theory.
Acknowledgment
TT was supported by ISF grant No. 1940/23 and MOST grant No. 0007091. The authors also thank the anonymous reviewers for their valuable feedback.
References
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, pp. 8141 8150, 2019.
Daniel Barzilai, Amnon Geifman, Meirav Galun, and Ronen Basri. A kernel perspective of skip connections in convolutional networks. arXiv preprint or wrXic:2211.1/810 2022.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in neural information processing systems, 22, 2009.
Hien Dang, Tan Minh Nguyen, Tho Tran, Hung The Tran, Hung Tran, and Nhat Ho. Neural collapse in deep linear networks: From balanced to imbalanced data. In International Conference on Machine Learning, 2023.
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-pceled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118 (110::110,,, 2021.
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. Tn International Conference on Learning Representations, 2022.
Bchrooz Ghorbani, Song Mci, Thcodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? Advances in Neural Information Processing Systems, 33:14820 14830, 2020.
XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. In International Conference on Learning Representations, 2022.
Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. arXiv preprint Ar 00..999..099,,. 2019.
Hangfeng He and Weijie Su. A law of data separation in deep learning. Proceedings of the National Academy of Sciences, 120(36)::222110411) 2023.
11

---

Published in Transactions on Machine Learning Research (04/2025)
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. Aduances in neural informalion processing systems, 30, 2017.
Kaixuan Huang, Yuqing Wang, Molei Tao, and Tuo Zhao. Why do deep residual networks generalize better than deep feedforward networks?- neural tangent kernel perspective. Aduances in neural information processing systems, 33:2698- 2709 2020.
Like Hui and Mikhail Belkin. Evaluation neural architectures trained with square loss vs cross-entropy in classification tasks. In The Ninth International Conference on Learning Representations (ICLR), 2021.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer-peeled perspective on neural collapse. In International Conference on Learning Representalions, 2022.
Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. Transactions on Machine Learning Research, 2023.
Vignesh Kothapalli, Tom Tirer, and Joan Bruna. A neural collapse perspective on feature evolution in graph neural networks. In Thirtg-seventh Conference on Neural Information Processing Systems, 2023.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl- Dickstein. Deep neural networks as gaussian processes. In International Conference on Learning Represen- tations, 2018.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Jachoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Diekstein. Finite versus infinite neural networks: an empirical study. Advances in Neural Information Processing Systems, 33:15156-15172. 2020.
Yuanzhi Li, Tengyu Ma, and Hongyang R Zhang. Learning over- over- wer-parametrized two-layer neural networks beyond ntk. In Conference on learning theory, pp. 2613 2682. PMLR, 2020.
Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern parameeriid learning. In International Conference on Machine Learning, pp. 3325-3334 PMLR, 2018.
Alexander G de G Matthews, Jiri Hron, Mark Rowland, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations, 2018.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. aie preprinl arXiw:2011.11619, 2020.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 1995.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40)224652222 24663, 2020.
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In International Conference on Machine Learning, pp-
PMLR, 2023.
Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. Advances in Neural Information Processing Systems, 32, 2019.
12

---

Published in Transactions on Machine Learning Research (04/2025)
Noa Rubin, Inbar Scroussi, and Zohar Ringel. Grokking as first order phase transition in two layer networks. In The Twelfth Internaiional Conference on Learning Representations, 2024.
Bernhard Schölkopf, Alexander J Smola, Francis Bach, et al. Learning with kernels: support vector machines, regularization, optimization, and beyond. 2002.
Mariia Scleznova, Dana Weitzner, Raja Giryes, Gitta Kutymiok, and Hung- Hiung-Hsu Chou. Neural (tangent kernel)
collapse. In seirbh-seventh Conference on Neural Information Processing Systems, 2023.
Howard Seltman. Approximations for mean and variance of a ratio. unpublished note, 2012.
Inbar Seroussi, Gadi Naveh, and Zohar Ringel. Separation of scales and thermodynamic description of feature learning in some cnns. Nature Communicatiins, 14(1):908, 2023.
Peter Sukenik, Marco Mondelli, and Christoph H Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model. In Thirty-seuenth Conference on Neural Information Processing Systems, 2023.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7544 2020.
Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. Advances in Neural Information Processing Systems, 35:27225 27238, 2022.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. Tn International Conference on Machine Learning, pp. 21478-21505 PMLR, 2022.
Tom Tirer, Joan Bruna, and Raja Giryes. Kernel-based smoothness analysis of residual networks. Mathematical and Scientific Machine Learning, pp. 921 1944944. PMLR, 2022.
Tom Tirer, Haoxiang Huang, and Jonathan Niles Weed. Perturbation analysis of neural collapse. In International Conference on Machine Learning, pp. 34301-34329. PMLR, 2023.
Roman Vershynin. How close is the sample covariance matrix to the actual covariance matrix? Journal of Theoretical Probability, 25(3):655-680) 2012.
Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, and Qing Qu. Understanding deep rep- resentation learning via layerwise feature compression and discrimination. arXiv preprint Stt::2311002000 2023.
Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Gencralization and optimization of neural nets vs their induced kernel. Advances in Neural Information Processing Systems, 32, 2019.
Christopher Williams. Computing with infinite networks. Advonces in neural information processing systems, 9, 1996.
Stephan Wojtowytsch et al. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. arXiv preprint ar arXiv:2012.05: 05420, 2020.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. Conference on Learning Theory, pp. 3635-3673. PMLR, 2020.
Jing Xu and Haoxiong Liu. Quantifying the variability collapse of neural networks, In International Conference Machine Learning, pp. 38535-38550. PMLR, 2023.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Yibo Yang. Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network? Advances in Neural Information Processing Systems, 35:37991 15537991-38002, 2022.
Yongyi Yang, Jacob Steinhardt, and Wei Hu. Are neurons actually collapsed? the fine-grained structure in neural representations. In International Conference on Machine Learning, pp. 14453-39487, PMLR, 2023.
Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu Neural collapse with normalized features: A geometric analysis over the riemannian manifold. Advances in neural information processing systems, 35: 11547-11560, 2022.
Can Yaras, Peng Wang, Wei Hu, Zhihui Zhu, Laura Balzano, and Qing Qu. The law of parsimony in gradient descent for learning deep linear networks. arXiv preprint Sti:23306.0115/ 2023.
Gilad Yehudai and Ohad Shamir. On the power and limitations of random features for understanding neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In International Conference Machine Learning, pp. 27179 27202. PMLR, 2022.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-29888 2021.
14

---

Published in Transactions on Machine Learning Research (04/2025)
A Proof of Theorem 4.1
To obtain the NC1 formulation corresponding to an arbitrary feature matrix H, we start with a simple relationship between (H) (H) Ew (H) as follows:
(H) = Ew (H) + (H)
(20)
= tr WE (H) = ur (H)(H)) - tr [En( (H)
G GT Similarly, by considering (H) h h
we get:
Eg(H) Eg(H) (H)
(21)
tr (EB(HH) (bB(H) (Ec(H)).
Formulating tr (E)(H): Expanding (H) into individual outer-products of vectors and leveraging the trace properties leads to the following:
tr (T)(H) tr
a h hee
1
no tr
h
N
n
Lr (h o hat)
N
Qx x²)
c=1
Formulating tr (H)):): Similar to the above analysis, we can reformulate the trace of non-centered
between-class covariance matrix Σg as:
1
C
tr(En) = tr
h
tr
=
tr
h
C
C
c=1
C
tr
h
h
11 i=1
1=1
Σ
C
ΣD
The mo
haiTha haj
1
C
n
(harT
n
1=1 1=
1=1
C
ΣΣ
Qix xx0)
15

---

Published in Transactions on Machine Learning Research (04/2025)
Formulating tr (Ec (H)): Reformulation of tr (Ec(H)) can be approached along the same lines: tr (E)(H)
GT
GT
= tr
h h
= =t
h
h
C
C
1
tr
h
h
N
N
c=1 1=1
C
C Bay
C
C
n2
ΣΣ
h
Σ tr
N tr
N2
c==j=1
C
0x
(x
N
By using these intermediate results, we formulate tr (Ew (Ew(H)) tr (H)) as:
tr(Ew(H) (H)) tr(Er(H)) tr (B(H)
C
Q(x
Qxx x
= N
C
c=1
tr(E = tr (5 (H)) tr Eg (H)))
C
C
nl
0(x*
x
2(x x"
c e=1
N2
Hence, proving the theorem.
B Limiting NNGP and NTK for ReLU
Consider the GP limit characterization of the pre-activation kernel K (x x
as follows:
o² K()) ( x of + x x d
(22)
do
Observe that (x (x,'" () is independent of the activation function. Now, the closed form representation of the post-activation NNGP kernel (.)(s))) for the ReLU activation is given by:
v KCp (X
2T
( (e()) sin 0 +(TT OC's)) cos 0'
(23)
K KGP (x (
0C's arccos
(1) (xS)) x(x) K ((1) C (xe' x)
Next, we define the ReLU based derivative kernel 6() QCP-ReLU () as follows:
QCP- (x x xd))
( 0)
(24)
2T
Finally, the NTK can be formulated as follows:
NTK (x x² w) = (x (dd) + (x x Rx xc'.))
(25)
Here, k/ Re (x xe d) can be defined using the recursive formulation:
k(2) FeL (x (x = of + o Q - (x x(')).
(26)
16

---

Published in Transactions on Machine Learning Research (04/2025)
C General Results for NC1 with Kernels
In this section, we present some general results to calculate the expected value of E [ (H)) for any given kernel function -) that is associated with the features H. To begin with, we consider a generic formulation
of the three cascs for E Q((*) o od))
(C) (c) if ic=
Rc=di==)
(27)
(((()
cf
Lemma C.1. Given the coses for the expected values of a kernel function (++-)) as per 27 the [ir((2 is given by:
[[)))
E [rr[Ew H))
( (c)
(ne(ne ()) + ne (()
(28)
N
2n
Proof. By leveraging Theorem
4.1
we can compute the expected value of (r(Ew (H) as follows:
E (()
E
Qwa 2000
N
C c=1
2
2
E
[Q(
E Q [991]
2 c=1
jm
j=l
(29)
2
Be
(c)
1
Σ
2
1
(nenne-1)ee 1) (c) + n. n (c)
2
n
2
n
(c)
(no(ne- (c) + no (()(c))
2n
Lemma C.2. Given the cases for the expeeced values of kernel function Q(:;)) as per is given by:
27 the E [tr EH(H))
2
E =
Σ
2n
N2
1
2m1
N2
(33 (1,2)
(30)
Proof.
The expected value tr( (r(BB( (H)) can be computed using Theorem 4.1 as:
C
Ro
Rs
C
no
ngy
E (trr(EB(H)) = E
Q 205)
Q
N2
1=1
2
ne - ()) n.w
2
+
(31)
2
noo - ()) + n. VUN(o)
(33
(c = 1,d' = =
N
i=
j=1
2
1
1) n. va)()
2n1n2
(33)
2n²
N²
N2
17

---

Published in Transactions on Machine Learning Research (04/2025)
Lemma C.3. Given the cases for the expected values of kernel function Q(:,-) as per is given by:
27
the E [ (H)
(no(ne- ( (c) ()
(c))
2ng
E [ (H)]
+ A
(32)
(ne(ne - Nz
(1) (c) (cVW(()) ()
2n, 12 (1.2)
2m
N2
Proof. Note that the expectation of the ratios can be given as:
E tr HH)
E [ (H)j
+ Ah.o.t
(33)
E [tr(2 (H))
n. (c)
(ne(ne- (o)
2
+ Ah.ott
(34)
(Me(ne - ()) ((e(nc-))v(((e)) + (c) N
2ming (1.)
2m
N
Here, Ab.o.t corresponds higher order terms given by (Seltman 2012
Var(tr(Epp (( [tr(Iw (H)) Cov Ew (H)), tr (En(H)))
A h.o.t
(35)
E [(r(E((H))
E [rr((B (H)))
where, based on the well-studied concentration of sample covariance matrices around the statistical covariance Vershynin 2012), Ah.a. tend to for large ne values.
Lemma C.4. For random variable at N(ees o.) which represents the tth sample of class (as per
notation in Section 5.3), the expected value
is given by:
20 + 20
T(c)
(36)
(ug + og)
(Ne + o?)3
Proof. Based on the standard result on the expectation ratios Scltman 2012), we get:
V Var((z
E
(37)
E [[xad)]]
E (2p
E((x)1)))] (n. + og)²
+
(38)
(4 + o??)
(42 + o2)3
Based on the results from the moment-generating function, we know that:
B[(xSt)4] = 30 + 60 ns.
(39)
which gives us:
30 + 60 2 + 11. ( + o2
E
+
(40)
(ng o?)
(ug 02)3
20 + 40
(41)
(n3 0z)
(n3 + 0233
Hence proving the lemma.
18

---

Published in Transactions on Machine Learning Research (04/2025)
D Proof of Theorem 5.1
D.1 NC1 of limiting NNGP with ReLU activation
In the limit d1 - 00, we leverage the kornels in the GP limit as per
10
23
Observe that for any two
data points ad xe' E IR, the value of 0 can be given as:
K(1) Gp (ast,,tt,
g
arccos
6,
(1) K Kpp (ati xc.1) K (xt' 200'
of + L 2
arccos
oB
ot
(oz
o
do
do
Since ob + 0, the value of g simplifies to:
ifc
0 02
(42)
if
2 x v Volggdd / we Sge'
which follows from
sign(x sign(x* 3) and x2d < 0, > 0 almost surcly. This leads
to:
dh QGP ReLu (x ,0001
1
V og (20t))
sin oin (1= (5- 05 cos 0
(43)
2π
-h b
ifc=d
Qar ReIu (00))
200))
2
(44)
if
For the c e' case, the value of the kernel boils down to the product of norms of independent random variables drawn from the same distribution. Since we assume xd a Of d, the equation 44 can rewritten as:
ReLu (288 (00)) Lyoetdd if==
Cer-ReLU
(45)
if cf
Additionally, since sd are random variables, the expected value of the kernel can be formulated as:
(o: + v2)
if dii=j
1) E OoP (2 200
fc=
(46)
if
Thus, based generic formulation of cases in 17 in Appendix
C we get:
02 (o: + +
(c) = a2
(c)
2 ((0)c, =.
2.
(47)
2
As N > 1 and n. > 1, Vc E {1,2}, Lemma C.3
gives us:
ne v.W (c)
((2)
[[
c=
N
+ Ah.ot
(n2V(2(ci)
2nyna N
(1,2)
207
N
(48)
2
E[[C,(HF)]]
n2
2
N
19

---

Published in Transactions on Machine Learning Research (04/2025)
D.2 NC1 of limiting NTK with ReLU activation
The recursive relationship between the NTK and NNGP (Tee et al. 2019 follows
Tirer et al. 2022 can be given as
13
o NTK- ReLU (x x) Kar- RaLL (x x (x 3 S
- (x x(05)
(49)
Here, KGP- ReLU (x x) can be defined using the following recursive formulation:
K KGP (x x)') of + (x*) x x)
(50)
Based on 24
the derivative Q QP-RRLD can be given follows:
OCP ReLUR (x ,x' )
1
0e's"
2m
K-ll Gx x
(51)
arccos
K (xcx, x K (xe's x(()))
We build on the results from the NNGP analysis (with Q R (x for computing the variability collapse with the limiting NTK. First, note that:
8
cf
(52)
if cf
When σb + 0, we get:
( NTK Real (x (x x
= o20 G
(x x 3)+ + k-11) (x xx (2) (x
(x
(53)
From 45
know that:
d Recu 2005
2 r
(54)
0
ifc
2 NIK ReLU (0
d geggg oegegee
22
ifc=d
(55)
0
if
o4 +
o2
x gnn cd
(56)
0
iccdd
Notice that @27 Ret (x S دو is scaled version of Q(1) up with the same result as 48
Ro (x 2d' (as per 45
Thus, we end
Rep nep tm
E (HNKK)
+ Aholl
(57)
N
20

---

Published in Transactions on Machine Learning Research (04/2025)
E Results for NC1 with Erf activation
E.1 NC1 of Limiting NNGP with Erf activation
Under the Assumptions described in Section 5.3 with do = 1,d1 + 00, observe that
11
gives us:
2K 2K
GP (x 2.'5)
(1)
Qar-Ers ,00/)) aresin
(58)
+ 2
(a( 2
a0tit V 2K GP (1
20 + 20 20
aresin
(59)
/ + 20% + 20 (2 12 /1 + 20 + 20 (xo RJ22
Considering Øb + 0:
d Bry (x ,00/)) (2)))
2
20 ": 2
aresin
(60)
1 + 20 (2
/1 + 20 (e(
2
sign(x sign (2
aresin
(61)
20 (aurryy
20 1232
where the last equality comes from:
sign (2 sign(x
(62)
[ 2rd (zeaa)
lx 21 1
//11
20 (ae.s)z V
20 (z
22
For notational simplicity, consider:
p(x 22/))) ()
+ 20 (x V
(63)
2o (x 13)2
w
and represent [1] QC- Erf ( , as:
(x sign 5)
QGr-Er 1) arcsin
(64)
plzci xt
Based on Assumption 1, we know that 1i < 0, 0, 2. > 0 almost surely. This leads to:
2 aresin (plaei
ifc
En (x ,.
(65)
arcsin (
icc
[ Eay
E.1.1 Calculating
For |u] < 1, we consider the expansion of arcsin(u) = +
to obtain:
6
arcsin (aann xel 25) )
(66)
6 x
To this end, based on Assumption 1 of large enough [ur |22l we approximate the expectation with only the first term and denote En.tt to capture the effects of the higher order terms. Notice that since p(x x
21

---

Published in Transactions on Machine Learning Research (04/2025)
for finite (x x0 2)dd), the effects of En.o. finite but decay rapidly compared to the first term. To this end, we get:
( 1 plartit 2orwr) )
1
E aresin
E
[aloris ae
+ Sh.o.t
(67)
Calculating the expectation
E
now be split based on c, d.
plani, x015)
Case d; j:
pla ,00))
(68)
20
(Inn;)22
E [p(oss 2201)]] =
E
(69)
20
(20.1)2
T(c)
1
(70)
20
The last equality is based on Lemma C.4 which gives the expanded version of T(c).
Finally, the value of E
be given
plass wc.a)
Var(p(x xrii)
E
(71)
ploedt zoit)
EE po west]]
E Ip(xse, x0)))s
)nor(p(a 200))
(72)
1
T(e 20
Notice that even in this simple case, the expressions are non-trivial to fully expand. Nonetheless, along with Assumption 1, we consider large enough |r1| |p2l such that:
T(c) 1
1
20 + 2
<1
(73)
20
20
[43 + 02))
(1. o3)3
Thus, based on the expansion of (1+w) u-u+u² + we obtain the following cleaner approximation:
T(c)
(1) (1)
(c).
(74)
plrait, xoii)
2o
((()) 2
(T(c) 3
Here (12 captures all the higher order terms corresponding
+
and
20
2o 10
8n.o.(p(20 x)+)) ((())) as denoted above.
Case == iiii j:
In the case of c= i' (iiA) j, the expectations the square roots do not have particular closed form. To this, end we leverage Assumption to obtain the following approximation:
p(a (00)
1 +
(75)
20 (2
20 (xod)2
1
+hoot) (
(1+
40 (2au)z + hoo.t)
(76)
40 (2a.zz
E [p
E
hot
B.
1
40 (o hot
(77)
40 (anipa
22

---

Published in Transactions on Machine Learning Research (04/2025)
Observe that the inner terms in the expectations are scaled versions of the above case. To this end, we approximate p(zas 2002)
as:
EB
2 + 6 ,x))))
(78)
plreit wai))
1
T(e)
+ 0 " ,20d))
(79)
T(c)
T(c)
20
I60
Similar to the assumption that led to
74
we get:
T(c)
T(c)
E
place 2255)
~
20
16
+ A
(80)
Case c +
A similar analysis as above applies in this case:
p( ,00
1 1+
(81)
20 (1rci)z
20
20
1 +
402 (zora)2 + thott)
1
(1+
40 (2r2 + hot))
1
(82)
E plast a
E
1
Ao (zad)z + hool
E 1
40 + how
(83)
Observe that the inner terms in the expectations are similar to the above case. To this end, we approximate
E
p(rre 2) as:
E
23
+ On x x(')))
(84)
pl ad
1
T(c) 1
(1 [
T(c')
40
+ @naelppass (c(1))
(85)
+(c)+
((c)T(e))
16og
Similar to the assumption that led to
74
we get:
T(c) T(c)
T(c)((d)
A
(3)
(86)
plr x('r)
4
160
Finally, based on 74
80
86
we obtain the following result for (1)
1P (x 20'0), as:
Fr
E
Qu1 Ery si
, .
T(c)
+ 4 (c)
if di==j
(87)
1 T(e)
T(o)²
+ A
h.a. (c)
if d
2nd
T(++T(())
T(c]iie)
+ +
(
(3)
ifeed
102
166
Here 1 (c), (c) (c) ( (e, are the collective higher order terms that tend 0 |pe| increases
relative to smaller values of de- These cases be plugged into generic formulation of expected
23

---

Published in Transactions on Machine Learning Research (04/2025)
values of kernel function (i.e (c), (c), (c,)), as per 27 in Appendix C C.3 for sufficiently large (ne) we get:
Thus, based on Lemma
no
(e)
((a(e)
E (H))
cm
+ Ah.ot
(88)
Ex=1
(n2V(2)(0))
2m N= (13)
2n
Numerator in the balanced class sctting.
To better understand the result, let's consider the balanced class scenario with nn n2 N/2, for which the numerator simplifies to:
nhe ((2) (c)
(c)
2
(c) (c)
(89)
2
2
2
T(e
+ A (1)
(o)((c)
A (c)
160
(90)
If we were to ignore the effects of the higher order terms, then observe that the numerator primarily depends on T(c)², which can be given based on Lemma C.4 as:
T(c) ( + 20 + 40
(91)
(u2 03)
(u? + o2)3
02)3
Thus, showcasing the dependence on Me, in determining the extent of collapse. For sufficiently large |Mel >> de, we can approximate this value to:
1
2022
2
1
80
16o
T(c)² ~
1
+
=
1 +
+
(92)
Pre
11
mrz
Denominator in the balanced class setting.
Similar to the numerator analysis, observe that when
n2 = N/2, the denominator can be given as:
1
n (12))
2n1n2
(3) (1,)),,
(93)
2n
No
N2
(12 (2)
(1,))
(94)
4
T(1)
T()²
(2)
(1)
T(2)
T(2)
+ A
h.a.e (2)
20
166
h.o.t
20
165
(95)
4
2
+()+
+2
T()T((2)
24
(3)
40
16st.
+
(96)
T())
1667
T(2)
1664
+ 2
T())((2)
1604
+ ho.
(2) (1) + + 1 (2)
(2) - -2.
(97)
(T(1)-TT 2
+
hoo.t (1) + (2)
(2)
(2) (2) 24 (3)
h.o. (1,2)
46
h.o.t
(98)
Observe that the term T(1) - T(2) represents:
1
201 4ofu²
201 + 402
T(1) T(2)
(99)
(μ² + oî)
(Mf o7
(#2 + +
(n² + o3)3
24

---

Published in Transactions on Machine Learning Research (04/2025)
and for sufficiently large |Pel
des essentially represents:
4
402
T(1) - T(2
=
+
(100)
11
11
P1
H2
E.2
NC1 of Limiting NTK with Erf activation
Recall from
13 that the recursive relationship between the NTK and NNGP can be given as follows:
NTK - Ery (x 5) = K(P Ery Bry S. + K (op x (
(x (o
(101)
where:
k-2) GGP-Er
GP-Erf (1 (x )= = o. + -
GP- Eegi-EEfhee Erf( .2055)
(102)
QCP- QGP-Rry ,. (1)
2 aresin
2K 2K (2 (2''))
(108)
V1 2k (1)
Gp (10), x0.1) V
2x GP (2 werl))
QCP- CoP-Ery(a"t x6
4 [ + 2k aa x44)))))))))) π
(1 2k (k (x 2,
(104)
1/2
22 @oplla ())
Considering σb 00 do = -1 (as per the setting and assumptions), we get:
K() (x x) = a2
(105)
20 2
,00 = π
aresin
(106)
/1 + 20 (z / 20 (2r2
- (20 (20200)1001)) (1+22 (1 20 2
-1/2
@cp-Er ( =
(1+20 20 (1+
(107)
/ 1 + 20 (x(Ct)2 + 20 (zz
This gives us:
K(1)
(1) (x ( G Ery (2
x + 20 (20.))2 + 20 (x2 (2613)
(108)
40 pit
(109)
To m 1. |
2o (1 (1
(20 +
(2uic)s
2
40 n. sign (z sign(x"'')
(110)
oz (ood)))) + (0l + (z
For notational simplicity, consider:
2 (x
V
(2 df +
(111)
25

---

Published in Transactions on Machine Learning Research (04/2025)
which simplifies the kernel formulation to:
(2)) NTK-ErJ NT at = Po Erg P ( + 40 sign(x"..) sign(x* J)
(112)
Tu(r00 v.
[ (x 200
E.2.1 Calculating
Similar to the NNGP analysis, we break down the calculation of IR [<(2*t, (a af into three cases.
Case = j
V2 (1
+
(113)
o 02 (2
(xr,))2
oh (reii)s (IZ.T)2
1
+ Eh.at
(114)
2
o
(xord)z
1
1
2
+
+
(war))z + En.a.
(115)
2
20
This gives us:
2
( cc.
- +E
+ E
+ B[En.at]
20 (2ra))
(2ss)) (ar((z
1
1
V Var((z5.)32
(116)
2
+
2x
E [I(
E (2
+
TE [[xad)]]]
E (()7))273
+ EE
Based on the results from the moment-genorating function, we know that:
[[(x"*)] = 30 + 60. pe:
(117)
which can be used along with Lemma C.4 obtaïn:
E [m( m0)) (0))
2T(c) B[Cn.o.]
(118)
For notational simplicity, we define a helper function as follows:
1 S(Me, (c) + E[Sn...)
(119)
which gives us:
E [ [ =1 ((pc.od)
(120)
Finally, the value of E
can be given as:
Var(k(oS* x01))
E
(121)
k( xc1)
[x(ast xai)
E E 201))3
+ Sn.aee1s (001)))))))))))))
(122)
S(),,
26

---

Published in Transactions on Machine Learning Research (04/2025)
Notice that even in this simple case, the expressions are non-trivial to fully expand. Nonetheless, along with Assumption 1, we consider large enough | such that:
S(He: σe) < 1.
(123)
Thus, based on the expansion of (1+u)-1 = =111111-- - u uu² -u +..., we obtain the following cleaner approximation:
E
1 S(He, + ( ,x0)))
(124)
16 k( 20))))
Case = j= j:
2
2
( 20
+
(125)
2 (
2
2
(126)
((ree)
(x
2
+ Shoa
(127)
(
(x )2 (xori))2
1
1
+
+
(fmee) + (mrroez Shou
(128)
2
20 un
(I (z(r)))z
Thus, based on Lemma C.4
we get:
1
E [n(a
]
+
E
E
+ E
E 1ELS.o.cl
(129)
2
2x w
(w (
(2
(x
1
1
1
+
E
E
+ E
Ekhh...
(130)
2
20
(22,))2
(x
1
T(c)2
(T) + Eich.nal
(131)
2
20 kc
This leads to:
=E E
(132)
k( x..5)
1+
T(c)
(T) + Blgh.
20
=1
1
+
T(c)2
(T) + Blsh.o)) ) + 5% C ,209))
(133)
2
202
T(c)
2T
5
(, x(,)))
(134)
20
Case c + d'
Ki xdds
( +
(xrt + (w0.8)2
(135)
+
(20)) Ja + (arld)a (xos) + cho
(136)
2
20 (x
27

---

Published in Transactions on Machine Learning Research (04/2025)
Thus, based on Lemma
C.4
get:
E (o 200 5)
E
E
+ E
Elch.o.d
(137)
20 (reit) ()² (
(xx v2
(x
1
E
E
+E
E
+ Blch.
(138)
20 8
(202;)²
(x011))2
(aci
T(c)T(d))
T(C) TTc) 10
(139)
20
This gives us:
Ri Ki x))is)
= 1+ 2 + T(oTT() + +(d))+ BiSk
(140)
20
+ T(c)T(c)
+ + + 1 (x(0))))))))) xz0d))
(141)
20
3 T(c)T(C) T(c) -T(') + o x² st))
(142)
2
20
Finally, the cases for the expected value of the kernel can be given as:
E o 2)
(r -
E
o Q
GP-Er Erg
+
E
TTA rara (eri))
1a
d'
(143)
NTK Ers
E
0 d
-FF (x xx w2)
E
4o
c# d
From 87
we know that:
c Ery (x ( (2)
E
T(c)
+ A.oo(())
if dii=j
20
(144)
T(e)
T(c)
+ + (2)
(c)
iic=citi
2n
16
T(c)+((d')
+ A (3)
(c,e)
ifee
16
To simplify the presentation, we can ignore the higher-order terms and obtain:
E
(2)
NTK Ery (x 2201v))
(2)
(145)
-²
1
T(c)
45
(3
(c)(
c= c=di-=)
20
x
0²
1
T(c)
T(c)2
Ar
3
T(o)
2TT(c)
c=dii
(146)
20
16a
20
d 1
T(c)+T(cc)
T(c)((())
T(c)T(c')
T(c) T(c) T(d)).
ctd
462
16.7
20
Observe that the order of the T(c) terms involved here resemble that of the NNGP scenario in 87 can make similar conclusions regarding the role of the order of pes in determining the value of E [NC,
Thus, we
F Activation Variability Relative to Data
In this section, we introduce a relative measure of activation variability collapse with respect to the data. First, we begin by defining the within-class and between-class data covariance matrices Ew SX),, Ex(X) E x for the data samples as:
Ew (X)
1 N
(x x²) (x - T²);; Ex(X)
1 Σ (x - (x C
(x -x- (147)
28

---

Published in Transactions on Machine Learning Research (04/2025)
where x -
d x Voe [C] and x The i=1 the data global mean vector respectively.
Σ
Σ
Σ
i= o represent the data class mean vectors and
Definition F.1. Set small > 0. The variability collapse relative to the data is given by:
(H) (X) + where (X,
tr(2w(X)) (x())(
(H
(148)
The constant T prevents numerical instabilities. Through this approach, we capture the extent of variability collapse of activation features relative to the variability collapse of the data samples itself.
Corollary F.2. Under Assumptions 1-9 (as per Section 5.5 let Φ(-) be the ReLU activation, and the limiting NNGP kernel be O QGP-ReLU (x x² d) hecThe' then:
E WC (H)
IT c= noPe
(149)
E [WC (X)]
Σ mr m2u2 N2
Proof. To keep the derivation similar to those for the kernel formulation in equation 45 we consider a simplified kernel on X (identity feature map):
Kduta (1 .00")) 20
(150)
Additionally, since x are 1-d random variables, the expected value of the kernel is given by:
ot + u
c -;i-- ii if ff
E
Kdota(I [()
(151)
Molle
We use Lemma
C.3 with cases (c) 0². + H² (c) μ² and (())
Mofe to obtain:
non 2 Ex=1
2
E [X)
E [tr(Ew (X))
E (tr(E(())) (X))]
n2 +n
N
n2u?tnod?
2ng
I Botte
+ A
hoo.
(152)
cm
2nd
N²
N
E [C (H)
Finally, the ratio
for ReLU (Theorem 5.1 with large enough ne >> 1 is given by:
E [CC (X)]
E WWC (H)
==
nall
c=
No
2
I nottee
Sho
(153)
E [WC (X))
n²y²
2
Teft2 +n
m²
c=
2
N
c=
2
N2
N
nefte
+ A Ahoo
(154)
Σ
E
2
R2
nan? N
2
N
I
c=1
Bele
A Sh.o.
(155)
22
na 2
n²u² N
m=
To better understand the result, let us consider the balanced class scenario where nı n2 N/2. This results in ratio of s 1 (2p1)22/(11 + ng). Furthermore, if |m| |uza (so f4) -1²2), then the ratio x 2.
29

---

Published in Transactions on Machine Learning Research (04/2025)
Thus, it emphasizes the interplay between class babbalance/balanee and the values of expected class means the relative variability collapse.
Addressing misleading , values: Consider the case where 01, - 0. Then Theorem 5.1 for QGP-ReLL RELL indicates that (H)) + 0 (considering smaller fluctuations from Ap.o.) in the balanced class setting. Such an observation can be misleading one were to ignore (X). For instance, such an empirical result while training deep neural networks fails differentiate between settings where the network learned meaningful features and learned to classify complex datasets was simply able to leverage the already collapsed data vectors. This applies to Erf activation as well. We justify this argument with the following experiment. For a sample size N chosen from {128, 256, 512, 1024). and input dimension do chosen from {1, 2, 8, 32, 128}, we sample the vectors x 1 ~ (- * 1dg I,,,i E [N/2] for class and x2 ~~:J~~(11 N10 * 1d, I..). E [N/2] for class 2 as our dataset. From Figure 6a 6b observe that (H values for QCP-Ers can be orders of magnitude larger than NC, (H), and for higi-dimensions. NC. (H > 1. Esscntially, the raw data is 'more' collapsed than the activations in these scttings. Similar observations can
be made for the NTK ONTK-FA Fef in Figure 6c 6d
D0 ANC (NNICCC
-1.
0.5
Vo (N
-2.
-20
-25
05
-0.
-30
-10
N-1004
-15
-30
N-312
-10
N-1024
-35
20
10
100
125
0
25
50
75
100
125
25
50
100
125
100 125
(a) Qar- Brff, (C,,
(b) Ref- NC, (HIX)
(c) @yTK-Ref, NC, (H)
(d) @yTK- NC, (H(X)
Figure 6: NC (H), of Q (1,2, 8, 32, 128) For particular N, we sample the vectors x1 ~ ~ do 1d -1, 1 E [N/2] for class 1 and x2 ~ N101 # 1d, I,.Itd)), y. = 1,j E [N/2] for class 2.
QGP-Ers and ( NTK-Er The dimension do on the x-axis is chosen from
0.
-10
-12
-12
02
0.0
(2555))-225..12 2011
(0u-n)i(2355 (258
(236) 268)
(256, (255, 268)
-20
(h).0)) P0) (171..94)
(P% Pg) (126,04)
(hh (b) (178,
(hi.ng) [ ((12
40
100
120
23
100 120
60 100
120
53 100 120
(a) EoS (C
(b) EoS NC, (H
(c) 2L-FCN (H)
(d) 2L-FCN (H
Figure 7: NC, NC of the adaptive kernel (EoS) with final amnealing factor dj = 500 and 2L-FCN
with d1 = 500 and ErF activation. The dimension do on the x-axis is chosen from {1, 2, 8, 32, 128}. For a tuple (n,, n2)) such that n, + n2 = N = 1024, we sample the vectors x ~ WN(2 * 1 0.25 * II,, I,,,, = 1, 1 E [n,]
for class 1 and 2. ~ (22 * 1d, 0.25 * Ia,), y2 = 1,j E [mz] for class 2.
G Training Details and Additional Experiments
Setup: We train a 2L-FCN with d, 500, ow L, 0 and Erf activation using (vanilla) Gradient Descent with learning rate of 10-3 and weight-decay 10 for 1000 steps to reach the terminal phase of
training (see Figure Sh for plots with N = 1024 and vectors x 1, ~ A/(22 * 1ag 0.25 * I 12 -1, E [N/2] for class 1 and 2 ~2(2 * 1d, 0.25 * La,). 2 = j. E [N/2] for class 2). The ReLU activation experiments use a learning rate of 10 4 To numerically solve the EoS, we employ the approach described in Section 6.2
All the experiments in this paper were executed on a machine with 16 GB of host memory and 8 CPU cores. Experiments with the EoS on datasets of varying dimensions and sample sizes took the longest time hour.
30

---

Published in Transactions on Machine Learning Research (04/2025)
0.30
1.00
1.00
D.40
0:35
0.30
0.30
8 3210
0.00
8.05
0.00
6073
0200
B20
200
400
1097
+00
602
1000
490
800
803
100
002
500
spoch
torpll Focex
epoch
comple Index
(a) Loss (du
(b) Sign prediction (du 1)
(c) Loss (du 8)
(d) Sign prediction (du 8)
Figure 8: Training loss and predictions (sign) of the 2L-FCN model with d1 500 and Erf activation.
-16
-20
-13
1.25
1.4
-1.6
-14
-1.
-2.0
20
M-513
-22
No-11
24-
-20
N1-216
No-117
-2.00
9-1031
-24
Ne1124
-24
Nw-1244
M-1024
20
40
80
50
100
12c
20
40
60
80
100
120
20
40
60
100
320
22
43
00
00
302
120
(a) L ==
(b) L ==
(c) L =
(d) L
Figure 9: NC, (H) of deeper FCN networks with Erf activation and hidden later width 500. The dimension do on the x-axis is chosen from (1, 8, 82,2, 128}. For particular N, we sample the vectors x ~ ~N(2 1d, 0.25 I -1, [N/2] for class and x25 ~ NN(2 * 1d: 0.25 Ia,), 1,j E [N/2] for class 2.
On increasing the number of classes with separable data. for C' =, 2, we formulate D2 do) for C 4 as follows:
Similar the formulation of D1 (N,do) do)
D2(N, do) {xx1i N 1d2 0.25 Lto y -3), Vi [N/4)))
Uf(22 ~ N 1dg 0.25 Ido) 1122 2. = -1), Vÿ E [N/4D)
(156)
U ((x0i ~ W/(2 * 1d, 0.25 * Id,), y = 1), E [N/44)]
U ((x4)) ~ ~((6 Id 0.25 1dn), n 3), VI E [N/4])]
The trends in NC, (H) for EoS and 2L-FCN hold even for datasets D2 dg) with C' 2. Observe from Figure 10 that the .C, (H) values for the kernels QCP -.P-Erf. ONTK-ErF- EoS and for 2L-FCN are consistently even when comparing them at do = 1. We
lower than the C = 2 case as shown in Figures 2a 2b 2c believe this because D2,N, da) is constructed by adding new classes to (N,,, (N, whose distance between the means is much larger than the within class co-variances (see Figure I1. Nonetheless, similar to the C case, the EoS deviates from the NNGP behavior and tends approximate the trends of 2L-FCN for high dimensional data with a sufficiently large sample size.
4b
-12
-1.25
10gg (NCIAAI)
-1.50
50
-16
-1.5
-1.75
-1
-2000
N: 122
3.00
N-525
:20
-2.8
20250
I-512
-226
N-512
N4255
No 1604
N-3024
25
80
75
107
125
50
75
150
125
d
130
(a) QGP- Erf
(b) ONTK-EPY Brf
(c) EoS
(d) 2L-FCN
Figure 10: (C, of the limiting kernels, adaptive kernel (EoS) with final annealing factor 500 and 2L-FCN with d 500 and Erf activation dataset D2,,, do) (as per 156
31

---

Published in Transactions on Machine Learning Research (04/2025)
-1.5
-15
-16
-2.0
-20
-00
-20
-2.5
-25
-25
-3.0
1-0
10 125
202
-30
10-252
2-122
he12
-0.5
N-512
1004
Ne512
1024
35
1x224
-35
h/224
-95
25
75
102
125
25
50
75
100
125
4:
4
(a) -es
(b) ΦNTK- Erf
(c) EoS
(d) 2L-FCN
Figure 11: NC, (H) of the limiting kernels, adaptive kernel (EoS) with final annealing factor di = 500 and 2L-FCN with d1 = 500 and Erf activation. The dimension do on the x-axis is chosen from 2,, 32, 128). For particular N, we sample the vectors x ~ N 1dg > 0.25 Ido), =1,, [N/2] for class 1, and x ~ N/6 + do, 0.25 Ido), 2d 1,j E [N/2] for class 2.
Class imbalance hurts EoS in approximating 2L-FCN. One of the key conditions for the EoS to approximate the NC, (H) behavior of a 2L-FCN is the presence of sufficient data samples (Seroussi et al.) 2023) (depending on the scale of the input dimension dy as shown above). Thus, extreme class imbalances can lead to biased NV2, (H) trends in EoS. To this end, we consider a collection of imbalanced datasets based on Di (NN, do) where N 2048 is split into two classes as follows: Case 1: (768, 1280), Case 2: (512, 1536), Case 3: (256, 1792). We observe that as the imbalance ratio increases, the magnitude of (H)(H) in 2L-FCN and kernel methods show opposing trends (see Figure 12 We make similar observation by considering D2(N, do) with C = 4 and N = 1024 with 3 different imbalance ratios as in Figure 13
-0.75
-0.75
100 (0cupn
-1.00
-1.00
-1.0
-1.25
-102104 1024)
T. (1024, 1024)
-1.76
[768, 1250)
(512.
-175
- 1768, 1280)
0. (1044, 1021
-2.00
(2566216700 1792)
-2.0
-20
(292 17931
(292)
25
80
75
100
125
160 125
100 122
100
(a) Qar-Erf
(b) ONTK-EFF
(c) EOS
(d) 2T-FCN
Figure 12: NC1 (H) of the limiting kernels, adaptive kernel (EoS) with final anncaling factor d1 500 and 2L-FCN with di = 500 and Erf activation. The dimension do on the x-axis is chosen from {1, 2, 8, 32, 128]. For
tuple n. (n,, 122) such that ni + n2 N = 2048, we sample the vectors x ~ ~N(22 0.25 Id,) y² =
- i E [n1] for class 1 and x2. ~ (2 x 1. 0.25 * Id,), 2 = 1 E [12]] for class 2.
-1.00
1100
1ogu (WC100)
2555
-179
-1.00
-1.50
-1755
FL (238, 256 236 236)
-175
(236, 256, 255, 2550
-2.00
-2000
N- 12b6, 236, 235, 206)
Fu- 1236, 236 256 2541
-225
06 (400)
Pv (312, 256, 128, 1281
312,
128, 354)
-235
16-0000 112, 125, 294)
16 m (532, 256, 125, 125)
A. 1354, 128, 114)
0. (426 112, 128, 384)
-23
FL 1354, 304)
112. 1/01 2041
-2.0
-24
126, 126)
50
75
100
125
123
66
(a) QCP-ER Ees
(b) θNTK- Enf
(c) EOS
(d) 2L-FCN
Figure 13: NC, (H) of the limiting kernels, adaptive kernel (EoS) with final annealing factor di = 500 and
2L-FCN with d1 = 500 and Erf activation. The dimension do on the x-axis is chosen from {1, 32, 128). For tuple ne = (n1, n22 n3g n4) such that n, + n2 + -ing + n4 N = 1024, we sample the vectors x
N 1d, 0.25 Ito), y li -3,1 E [ni]; 2a - + 1d, 0.25 L2) 25 --1 E [naal. 3.*
(222 Li, 0.25. * It ), y 3,6 = 1. E [ma] and 4 N6(6 * 1.g. 0.25 * 16). y = 3, E [n4].
32


Reviews:
[{"review_id": "dy7d41bOEd", "reviewer": "Reviewer_TjHC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-organized and easy to follow.\n\nIt is also well-motivated: The authors seek to address UFM's shortcomings by leveraging insights from kernel literature to assess whether different kernels offer additional insights or demonstrate other limitations. This is a meaningful step toward developing a more accurate theoretical model. \n\nHowever, some discussions on the empirical results raise confusion, and further clarification would be helpful. I will expand on this below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores an alternative to the unconstrained features model (UFM), a widely used theoretical model for studying neural collapse. The main limitation of UFM is that it assumes unconstrained features, ignoring the role of the input data distribution. This work investigates whether kernels associated with neural networks can offer a more realistic alternative. The authors focus on NC1 (the collapse of features within a class) as a first step, leaving the geometric properties described by NC2–NC4 for future work. \n\nThey study three types of kernels: 1) NNGP kernel corresponding to infinitely wide neural networks (NNs) at initialization, 2)NTK corresponding to infinitely wide NNs in the lazy training regime, and 3) Data-aware GP kernel, an alternative kernel that reflects the effect of data on a finite-width neural network, capturing aspects of feature learning.\n\nTheoretically, they show that for binary scalar Gaussian data and a shallow 2-layer network, the expected NC1 values for NNGP and NTK are equal—highlighting a gap with practical training where features learned during training exhibit more collapse than those at initialization, further reinforcing NTK's limitations in capturing true training dynamics.\n\nEmpirically, they compare NC1 values from the kernels with those from features learned by training a shallow 2-layer network. The results demonstrate the potential and limitations of kernel-based models in explaining neural collapse and offer comparisons across different setups, e.g., different activation functions or data distributions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.  Since NC1 requires reasonable overparameterization and occurs in the interpolation regime, it would be helpful to confirm and report whether, at the end of training for the shallow networks trained by GD, the model has entered the interpolation/low training-loss regime for the different configurations reported in the paper.\n\n2. The points below might need more careful discussion in order to make the message of the experiments clearer:\n\n    (i) *On NTK limitation*: The paper argues (theoretically and empirically) that NTK limitation is due to the fact that it matches NNGP in terms of the NC1 metric. One confusion can arise from comparing figs 2 and 3 (a) vs (c), where even the NC1 of the features learned by gradient descent and at initialization (characterized by NNGP) are very close as well (In fact, in figure 2, all the three subplots (a), (b), and (c) show the same NC1 trend and value). \n    \n    (ii) *On the benefit of data-aware GP*: When discussing the EoS results, it's stated that for larger network dimensions, the data-aware kernel matches both NNGP and NTK closely. Does this suggest that this kernel has the same limitation as NTK in explaining neural collapse? \n\n    In the lower-dimensional case, the behavior changes. However, 1) it also doesn’t fully align with the fully connected case. For different sample sizes $N$, the curves show varying trends, unlike the shallow network experiments in fig. 2, and 2) The higher-dimensional case is more relevant to the neural collapse regime since NC typically emerges under overparameterization.\n\n    (iii) The message of fig. 6 is not very clear: When comparing the collapse of input and kernel features, is this another reason for the limitation of NTK in explaining neural collapse? Or is it a limitation of the shallow NN setup? Specifically, does the same observation ($NC(H|X)>1$) hold for the features of trained shallow networks as well?\n\n3. *Typo:* The caption of Figure 4 refers to subfigures (c) and (d), which do not exist.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1gp51ZRFLz", "reviewer": "Reviewer_DFdc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Studying neural collapse with data dependent analyses indeed seems a necessary step to get a better understanding of neural collapse, in which situations does it happen and how it does so.\n\nThe paper strongly lacks in clarity in my opinion: in particular, I have difficulties navigating through the different results and interpreting them. In particular, the original intent of the authors is to describe neural collapse in a data-aware fashion. However, the 8 first pages of the paper are dedicated to NNGP and NTK kernels analysis, which are not \"data-aware\". More importantly, such analyses seem to describe the Neural Collapse at initialization of the training, while neural collapse seems to clearly be a phenomenon appearing during the training of networks (being there at convergence, but not at initialization). Therefore, I do not really understand how all the results before Section 6 serve the original author's intent.\n\nAdditionally, I feel that the main sections are a sequence of technical results, but they lack, in my opinion, some higher level interpretation by the authors. More detailed, high level, interpretations of these results would help the reader in understanding this work, and also understanding where we are going while reading the paper. \nIn the same spirit, I would like an extended introduction to the EoS framework, as it is not yet very common in the literature. In particular, I did not understand whether the EoS equations are supposed to describe the state of the network at convergence, initialization or some point of the training dynamics. \n\nOverall, this notion of training dynamics seems missing in the paper, while it should be key in understanding neural collapse. Even the experiments do not reflect any evolution of the neural collapse throughout training (but only at convergence?).\n\nOn the technical aspect of the claimed results, all the maths seem sound, despite some notations/assumptions/definitions that need to be clarified (see below). Overall, my opinion is that this paper writing needs to be largely polished to help the reader navigate through the different results and how they relate to the original intent of the authors.\n\nAlso, the data-aware kernel part seems to be the most significant one of the paper (at least when reading the abstract), and I thus think it would be nice to enrich this part (notably by giving a more detailed introduction to the EoS framework and give more detailed interpretations of the results derived here).\n\n-------------\n# Minor remarks\n\n- Equations (1) and (2): $\\psi$ and $\\hat{y}$ are exchangeably used for the same thing. I would recommend to stick with a single notation\n- beginning of Section 4: that would be helpful to precise the dimension of the matrix $H$ here\n- in Section 5, there sometimes are dot appearing above the kernel functions $Q$. What does it mean? \n- Assumptions 1 and 2: what does the $\\gg$ notation mean? It is not clear how it intervenes in Theorem 5.1\n- Theorem 5.1: what is $Q_{GP}$? Is it $Q_{GP-Erf}$?\n- Theorem 5.1: \"that vanishes as $\\{n_c\\}$ increases\" -> is it for $\\min_c n_c$?\n- Key takeaway (section 5.3): I do not see where you \"have established another result that shows that training in the lazy regime provably deviates from the practical feature learning of NNs\" -> this is clearly the kind of thing we need more discussion about\n- same about \"these observations empirically verify our theoretical results\": I think the comparison is only given in the appendix. It would be nice to have some comparison in the main text then", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work aims at theoretically studying neural collapse for neural networks. While previous theoretical works mostly use the Unconstrained Feature Model (UFM), this work studies it through kernel methods, being notably relevant in the NTK regime when training neural networks.\n\nFor Gaussian Process and NTK kernels, it then proposes an expression for neural collapse (being the ratio between inner class variability and interclass variability), and also proposes an analysis through Equation of State (EoS) to get a data-aware kernel.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like the paper to be largely improved in clarity, especially by adding many high level paragraphs explaining the implications of the different theoretical results (see above)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ckLv3jzqzj", "reviewer": "Reviewer_NTBv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. This paper studies the effect of data distribution on NC, which I think it's interesting\n2. I think the study of NC of kernel learning methods, especially with data-dependent kernels is an interesting topic \n3.  In section 6, the authors empirically find that by adding data-dependent correction terms to the NNGP kernels improve the NC1 metric, which is an interesting finding in my opinion.\n4. I think this paper is well-written and easy to follow \n\n**Weaknesses**\n\n1. My major concern is that I don't see the point of directly considering the NTK or NNGP kernels as the feature kernel, and studying the NC1 of NTK or NNGP kernels. If one consider the feature kernel is the  NNGP or NTK, it implies that the features **are fixed** at initialization, meaning that the features are just \"random features\" and there's no feature learning in this case. It is kind of obvious that NC1 will not occur in these cases. Concretely, in the NNGP case for example,  the feature $h(x)$ of a data point $x$ is just $h(x) = \\phi( W^{(1)} x),$ where $W^{(1)}$ is the i.i.d Gaussian matrix at initialization. The NTK case is similar with a more complicated expression of the feature.  \n \n2. I don't see the utility of Theorem 5.1 in general.  First of all, the assumptions that there's only two classes of $1$-dim Gaussian data, and $|\\mu| \\gg \\sigma$ is kind of restrictive, since this means the dataset is already well-separated. In particular, the expected NC1 is of the same order of the variance as discussed in the part below Theorem 5.1, meaning that the  kernels does not improve the NC1 of the original dataset. Besides, under this setting, the main information from Theorem that NC1 for the NTK and NNGP kernel is the same up to higher-order terms are again not surprising in my opinion, since in both NTK and NNGP case, the feature is again a \"random feature\", and it is not hard to imagine that the NC1 of feature is equivalent to the NC1 of the datasets.\n\n3. Another main issue of this paper is that the authors do not consider any training on the features (since the feature kernel is the NNGP kernel or the NTK kernel). However, as mentioned in the original NC paper [1], NC is considered to be a phenomenon occuring at the ending phase of the training. Thus, I think the usefulness of this paper's results in understanding NC might be limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the neural collapse for kernel methods. The authors first derive the NC1 matric for general kernel methods, applying the results to NTK and NNGP kernel, and then doing experiments to support the theoretical findings. Then the authors discuss the data-dependent kernels and also conduct experiments on data-dependent kernels.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Could you address the **Weaknesses** part? I don't have further request changes.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the core component of neural collapse, namely the within-class variability collapse. The UFM, which constitutes the established framework to study neural collapse, is data-agnostic and, to address this issue, the paper proposes to focus kernel methods, considering the NNGP, the NTK and a data-aware GP kernel. The results show the limitation of data-agnostic kernels (NNGP, NTK) and that the data-aware GP kernel exhibits superior collapse. 

The rebuttal period has helped clarify some misunderstandings and the most pressing issues raised by the reviewers have been resolved. The three reviewers agree on the overall assessment: although the results are not particularly surprising, they are technically correct and prove an important point concerning the importance of data adaptivity for neural collapse. I concur and, considering the TMLR guidelines, I recommend accepting the paper.
Decision: Accept (poster)