How Classifier Features Transfer to Downstream: An Asymptotic Analysis in a Two-Layer Model
Authors: HEE BIN YOO
Sungyoon Lee
Cheongjae Jang
Dong-Sig Han
Jaein Kim
Seunghyeon Lim
Byoung-Tak Zhang
Venue: NeurIPS 2025 poster ()
Keywords: open-set clustering
feature transfer
two-layer neural network
feature learning
metric learning
retrieval
Primary area: theory
TL;DR: This paper investigates feature transfer in classifier-trained networks, analyzing the impact of similarity between training and unseen data on unseen data clustering performance and feature extraction.

Abstract: Neural networks learn effective feature representations, which can be transferred to new tasks without additional training.
While larger datasets are known to improve feature transfer, the theoretical conditions for the success of such transfer remain unclear.
This work investigates feature transfer in networks trained for classification to identify the conditions that enable effective clustering in unseen classes.
We first reveal that higher similarity between training and unseen distributions leads to improved Cohesion and Separability.
We then show that feature expressiveness is enhanced when inputs are similar to the training classes, while the features of irrelevant inputs remain indistinguishable.
We validate our analysis on synthetic and benchmark datasets, including CAR, CUB, SOP, ISC, and ImageNet.
Our analysis highlights the importance of the similarity between training classes and the input distribution for successful feature transfer.

How Classifier Features Transfer to Downstream: An Asymptotic Analysis in a Two-Layer Model
HEE BIN YOO Interdisciplinary Program in Artificial Intelligence
Sungyoon Lee Department of Computer Science
Hanyang University
Seoul National University yooheebin@snu. ac.kr
sungyoonlee@hanyang.soo ac. kr
Cheongjae Jang
Dong-Sig Han Department of Computing Imperial College London d. han@imperial ac uk
Jaein Kim
Department of Computer Science Hanyang University chastro@gmail. com
Interdisciplinary Program in Neuroscience Scoul National University
qpwodlsqp@snu. ac kr
Seunghyeon Lim
Byoung- Takung-Takk Zhang' Artificial Intelligence Institute Scoul National University btzhang@snu. ac.
Artificial Intelligence Institute Scoul National University shl chlimiii.. snu. ac.kr
Abstract
Neural networks learn effective feature representations, which can be transferred new tasks without additional training. While larger datasets are known to improve feature transfer, the theoretical conditions for the success of such transfer remain unclear. This work investigates feature transfer in networks trained for classification identify the conditions that enable effective clustering in unseen classes. We first reveal that higher similarity between training and unseen distributions leads improved Cohesion and Separability. We then show that feature expressiveness is enhanced when inputs are similar to the training classes, while the features of irrelevant inputs remain indistinguishable. We validate our analysis on synthetic and benchmark datasets, including CAR, CUB, SOP, ISC, and ImageNet. Our analysis highlights the importance of the similarity between training classes and the input distribution for successful feature transfer.
Introduction
Neural networks have demonstrated remarkable success through their ability learn hidden represen- tations, which are crucial for generalization [Damian et al.) [2022]. These learned features, especially those extracted from the penultimate layers, are semantically meaningful and transferable across tasks Yosinski et al. 2014 Kornblith et al. 2019). A wide range of techniques, from open-set clustering Roth et al. 2020 vision- languge models [Li et al. 2023 and language models [Kojima al. 2023]. leverage feature transfer for downstream tasks. Recent successes in feature transfer have been largely attributed the increasing volume of available data [Brown et al.) 22200] Radford et al.)
2021 An et al., 2023). However, the underlying conditions under which features can be effectively
transferred remain underexplored.
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Among various applications, classification based visual open-set clustering [usggrve et al. 2020 is a fundamental benchmark for evaluating feature transferability on unseen data. The task typically involves training a classifier on one set of classes and then testing it on a disjoint set of unseen classes to assess whether the extracted features form cohesive and separable class-wise clusters in unseen data [Wang et al. [0188] Seidenschwarz et al. 2021] Deng al., [022]. Given this context, we aim to investigate feature transfer through open-set clustering with the following research questions:
Can we capture the presence of feature learning in classification and identify the conditions under which features cluster effectively in new distributions?
To address this, we consider a two-layer nonlinear network trained with single gradient descent step a mean-squared classification loss in the proportional regime (in Section 2 Then, we show that the dominant part of the trained feature function consists of random initialization and a spike component (Def. (3.4) associated with the training data (in Section 3). Finally, leveraging the dominant features, we identify conditions for effective clustering in unseen distributions (in Section First, we assess the intra-class Cohesion and inter-class Separability of trained features through numerical and analytical analysis, standard measures of clustering performance [Clémençcon] 2011 Papa et al. 2015 Liu et al. 2017 Li and Liu [202]] As result, find that Cohesion increases as the train-unseen similarity grows.
Meanwhile, for Separability, if classes assigned to different training classes, it also increases with the train- unseen similarity; otherwise, it decreases, shown in Figure
F(x)
P1
Mapping input data (left) to learned feature space (right); Training classes are solid balls, unseen classes a.b.p.n are dashed lines. Cohesion: Strong Cohesion occurs for a.p.n, which have high similarity to the training data compared to b. Separability of o,n: as- signed to different training class, they show high Separability. Separability of a.pr assigned the same training class, they exhibit low Separability.
Figure I:
Second, we analyze the "spike component," the predomi- nant element of the feature function multi-class settings. We show that the inner product between the spike compo- nent and a new input governs the feature's expressiveness. Specifically, the spike component maps new inputs using linear combination of randomly initialized classifier-head weights. Therefore, tributes to feature extraction only when it aligns with the input data, as illustrated in Figure
find that the spike direction con-
Empirically, we evaluate train-unseen similarity, Cohe- sion, Separability under our theoretical assumptions using synthetic datasets. As predicted by our analysis, Cohesion and Separability follow the expected trends with respect train- uniten similarity. We further study open-set clustering in realistic set- tings and demonstrate the practical implications our theoretical results. In most cases, clustering performance is higher when the unseen classes belong to the same semantic domain the training classes (Expr. IV). Moreover, adding dissimilar classes does not enhance performance (Expr. V), whereas incorporating semantically related training classes does (Expr. VI) consistent with theoretical claims.
This work provides new insights showing that effective feature transfer depends on the train-unseen similarity, further implying that features can be learned in data-efficient manner without relying on large-scale data. Our contributions are summarized follows:
We analyze the classifier's 's feature representation, providing insights into how feature extractors operate:
Higher train-unseed similarity increases Cohesion.
Higher train-unseen similarity increases Separability between data assigned to different classes but reduces it otherwise.
Feature expressiveness improves as more spike directions become non-orthogonal to the input.
We validate the theoretical results through experiments and show that they hold in practical settings.
We generalize the distribution assumption of prior works to non-centered sub-Gaussian distributions and present novel proof techniques for classifier analysis (refer to Assumption 2.2).

---

1.1 Related Works
Deep Metric Learning and Open-Set Clustering Metric learning has been proposed cluster visually similar unseen classes using classification or triplet losses [Movshovitz- Attias et al. [017 Zhai and Wu 2019 Boudiaf et al. (02T). Several recent approaches have focused on increasing the number of training classes to improve clustering performance. One line of work introduces virtual classes [Chen al. 2018 Qian et al. 2020, Gu et al., [02T]. Another approach leverages a large number of classes induced from the dataset of [Schuhmann et al. [202T] to achieve state-of-the-art performance [An et al., 2023). This empirical trend aligns with our analysis, which suggests that performance improves the number of relevant classes for clustering increases. However, to the best of our knowledge, theoretical analyses of feature frameworks without additional training- unexplored.
within metrie-learning
Neural Collapse (NC) and the Unconstrained Layer-Peeled Model (ULPM) Recent studies have introduced the concept of Neural Collapse (Papyan et al. 2020) to explain the emergence of intra-class feature compactness and feature-weight alignment trained neural networks. To analyze these phenomena, several studies propose the ULPM which treats understand the training dynamics of NC treating features and classifier weights as unconstrained free variables [Fang et al., 2021 Zhu et al. 2021 Ji et al. 2022 Tirer and Bruna 2022]. J. Unlike the two-layer network model adopted in our study, ULPM assumes free variable features, which limits the analyzability of input distribution and, consequently, prevents the study of feature transferability.
Feature Learning in Two-Layer Networks The Conjugate Kernel (CK) has been widely used to analyze feature learning in two-layer networks (Louart et al. 2017, Goldt et al. [020 Hu and Lu 2022]. More recently, Ba et al. [2022). Moniri et al. [2024] have shown that feature learning can reduce population risk in teacher-student regression settings. In contrast, our work investigates feature transfer in classifier-traine
networks, particularly in scenarios resembling metric learning, which remain theoretically underexplored. To this end, we extend CK-based regression analyses to accommodate non-centered sub- Gab-sassiinn inputs with Hermite-expandeee HD. enabling classifier-base training with arbitrary labels and distributions. Moreover, Table I in compares our framework with contemporary studies on feature learning in shallow networks.
activations (Sections]] [[
Appendix C
Additional related work on shallow-network feature learning, feature transferability, and classifier- based feature analysis is provided in Appendix C
2 Problem Statement
Notations
We denote by |I-II either the I,2 or operator norm. Let @ be the Hadamard product and A be the Hadamard power. Constants C, e > and x E R may vary from line to line. Define [d] & 21,2.... .d. For o, O,0 notations regarding complexity, we follow Moniri et al. (2024).
Moniri
Training Data We consider a one-vs-one classification problem with K classes that constructs a K) problems. Let 01... OK denote the conditonnti distributions of the training data, and the training dataset be D (X, Y), where X E Raxd and Y [K]]. Here, X = UK (fx 2x x m); where m is the number of instances per class and Km
Network Structure in the Proportional Regime We consider two-layer networks in the propor- tional regime (Ba al., 2022], which represents a scenario where the network width and the size of the dataset are of similar scales. Let n, d and N be sample size, data, and feature dimension, respectively. We perform our analysis under d/ n, N/n + as n, d, N + co. Concretely, the initial weights of the first layer and the second layer Wo RdxN and anj RN for i, [K] s.t. i j. respectively. These are initialized as Woli] Uni U for i E [[] and ai follows a Gaussian distribution n0, N I). Denote W as the first-layer weights after a single gradient step. Accord- ingly, we define (x)x) A (W, x) and F(x) & o( x) as the initialized and trained features, respectively.
"Our theorem also holds in imbalanced settings, but we balanced settings here for simplicity.
3

---

Optimization Problem Denote the set of all network parameters as 0 = {W, 012. 1 Let Xij be a matrix in 2 where the first m rows contain samples æ ~ o; and the last m rows contain samples oj. Let y - [1,1...., l........ ..]" E R2mh be vector consisting of m ones followed by m negative ones. To classify the given data, we use the Mean Squared Error,
L(2, U)
Eieelly o Wayyll
(1)
2n
Sw The weight update for the first layer given by W Wo G, where G a - = Eies Gij. s.t.
Gi 72 X_]lo(X, W,, wa o o' (X.,
We then introduce the assumptions for the theoretical analysis. We omit the learning rate for simplicity, but it may be explicitly included as 1) = (1)) without affecting the proof.
Assumption 2.1 (Activation Function). Let or (xx) be element-wist activation s.t. a', o',a", o" is bounded by Do almost surely. It admits a Hermite decomposition i.c. [(2) = -i= Ct (2),, where
C kd (z) for constants C, w > 0 e.g., the Shifted ReLU, defined as max (z. 0) V22
for a standard Gaussian 2. We assume oo = 0, C1 > 0 and cf & Ck 3/2-w
Assumption 2.2 (Training Data). We assume the class-conditional distributions e; are non-centered Sub-Gaussians [eershynin 2018 Cao et al. 2021 Cole and Lu, 2024), necessary generalization for classification tasks involving non-identical distributions with bounded support. This extends common
assumptions such as Gaussians and Gaussian mixtures (see Appendix Table [ for comparison).
To accommodate non-centered Sub-Gaussian distributions, we generalize Hermite expectation tech- niques from centered Gaussians non-centered Lub-Gaussians (Lemmaa [.5] Appendix (H), extending prior work on centered Gaussian cases O'DDonell,, (002 Moniri et al. [024].. We also adapt key
results for centered Sub-Gaassians [Vershynin] 2010 2018 to the non-centered setting (Appendix [].
Please refer to Appendix Difor preliminary information on Hermite polynomials and for additional notations used in the proofs.
3 Feature Decomposition
This section analyzes the one-step trained feature extractor in the proportional regime. As first step, we linearize the gradient G in the proportional regime to approximate the trained feature.
Theorem 3.1 (Gradient Approximation). Let Aug - e X.yya.. a rank-one structure for each ij-th and for sufficiently large n.
classification problem. Under Assumptions 2.7
anddd 2.2
log"
||G
Σ
Auj S
w.p. 1 o(1)
(3)
n
For the proof, please refer Appendix Theorem 3.1 shows that the gradient G with respect to Wo exhibits an almost Rank-P structure and can be expressed as Σies Aij. Next, leveraging Eiey Auj and the Hermite decomposition of the activation a, we deterministically decompose the one-step trained feature F(x)= = ((W, + G) x) into its dominant and residual components, in order to analyze the feature structure for new inputs based on its deterministic and dominant terms.
Definition 3.2 (Spike Direction). The spike direction is defined as Big tX.JU Rd. If the distinction between i and j is not essential, we omit the subscripts.
Theorem 3.3 (Feature Decomposition). Under Assumptions 2.1 and 2.2 let X E Ruxd follow Sub-Gaussian, L A log n, F & Σi Hy (XXW). and, 8L _ Ei ci (X) Eies Bija,)
F = F + 8L + A, w.p. 1 - o1),
where ||>b|| = D(VD)) FL O(vn) and ||||| o(Vn).
Please refer to Appendix (oror the proof. Theorem 3.3 represents that the learned features are mainly composed of the spike component-which maps new inputs via inner products with the data-label covariance Bnd-and Fo,r., inherited from the random initialization. Under Theorems 3. and 3.3 follows that for sufficiently large the one-step trained feature is primarily described by the term Fo. + 81,- Therefore, we introduce the deterministic Dominant Feature (z( below, which serves as the foundation for our subsequent analysis.

---

Definition 3,4 (Dominant Feature FL (x)).. For sufficiently large n, w.h.p and retaining only compo- nents of F larger than o( ovnn,, we define the dominant feature Ft. follows.
I. c
FL(x) A
cu Hn (W
x)
(B)Ja)a
(5)
(x)
(1.(2)
where Fo,L is the approximated randomly initialized feature, and 8g the trained Spike Component.
(x) defined in Definition 3.4 represents the dominant part of the feature decomposition using the Hermite expansion for sufficiently large n. This approximation is expressed through linear operations and polynomials, facilitating explicit closed-form representation of the clustering criteria.
4 Feature Analysis
4.1 Clustering Criteria Analysis
We analyze two key criteria- Cohesion and Separability- -which capture intra-class compactness and inter-class distinction of features to identify the conditions under which features from unseen classes form coherent and separable clusters. We suggest that these quantities are governed by the train (3)-unseen (μ) similarity under Condition 4.2 in binary classification. Specifically, we express both Cohesion and Separability (Definition 4.3 as closed-form polynomials of B (4.
Definition 4.1 (Train-Unseen Similarity). Given the spike direction 3 in Definition 3.2 and the mean of the unseen distribution μr, the Train-Unser Similarity is defined as 3
Condition 4.2. Suppose n, d, N are fixed and sufficiently large. Under Assumptions 2.1 2.2 let c; (ja, Ia) for i E [2] be the class conditional distributions. Define Pa.k',r,r 0 to depend only on N and d. and to be independent of and B, as in Definition G.]
Definition 4.3 (Cohesion and Separability). We define two clustering performance criteria as func- tionals based the inner product similarity between features, for given any feature G:
e(G) Eo
S(G) Ry [Enwey G(z) ( TEx G(r)))
Cohesion (GG) quantifies the expected inner product similarity between i.i.d. features belonging to the same class c, with the expectation taken over network parameters 0. Separability 8 (G)) quantifies the expected dissimilarity between independent features from different classes 0r and 0g again with the expectation taken over 6.
Since x. l/I|=ul| + as n + do by Theorem 3.3 we focus on presenting C(sT.) and 8
Proposition 4.4 (Cohesion of sL). Following Condition
4.2 the Cohesion ((sL) for o, is given by:
K
((sL)
C&CK
ΣΣ
Pk se 18 kKK ||
(6)
r=0r==
Proposition 4.5 (Separability of sL). Under Condition 4.2 the Separability S(sL) 01, 02 yields:
K' ΣV
r (8 ph ( 1 111))rtt
S(st)
ChCck
r=0y"
The proofs of Propositionns444aandde are Appendix the above Propositions, C(sr.) and (ST.) are computed in closed-form expressions and expressed polynomials of
Similarity. Also, ((Fz) and S(Fz). which include the Fo, term, are polynomials of the train-unseen similarity, shown in Propositions G.2 and G.3 With the polynomial from, to identify conditions that improve the clustering criteria, we numerically evaluate ((FF) and (FF)) by varying μ B and p.j 3.
Empirical support for the dominance of in ((Fz) and 8(F2) presented in Appendix K
5

---

As illustrated in Figure 2
we find that Cohesion increases with |u 8), i.e., as the "unseen" data becomes more aligned with the Spike Direction B. Also. Separability rises when the magnitudes of "4 [3 and /22 3 grow with opposite signs and decreases when they grow with the same sign i.e., it increases when the two "unseen" classes are assigned to different classes as explained below:
Note 4.6 (Explanation of assignment and B μt). 8 represents the normal vector of the linear decision boundary in binary classification within the data space before training, i.e. the direction that determines class assignment based on the sign of its inner product with the data. Therefore, the sign of) B y indicates the class assignment of unseen data with mean μ
These two phenomena can be interpreted based on the property of the leading term SL in Fz. The phenomena follow the ex-
125
1000
100
44
66
pressions inside the brackets of Eqs. (6)
Coherion
75
22
and (7). Furthermore, if the Shifted ReLU is used as the activation function, the anal- ysis in Appendix shows that the first Hermite coefficient c₁ takes significantly positive value, while the subsequent coef- ficients oscillate and gradually diminish in magnitude. This suggests that the positive contribution may dominate the summation Decers which is consistent with our em-
50
-500
66
111
-1000
1000
000
500
0
500
1000
PT
(a) Cohesion C(FL)
(b) Separability ((F))
Figure 2: F by adjusting 3 d1 and B U2. We set n 320000, μ4n = - Pl2 E R and || ll /l/2lll : || =. pirical observations. However, further investigation is required validate this hypothesis rigorously. Additionally, in Appendix K we show that our theory tends to hold over a wider range as n increases, and in Note G.6 we discuss the ||||| term.
4.2
Spike Component Analysis
B3
B2
Pa
Ps
T22
81
8
8
#combinaton
Az
A 84
B combination
P1
A
0
01
8:
8:
F1
02
Ba 84
8a 8c
9:
8:
03
9. 0.5 Expressiblity
8c
04
0.0
Expressibiiit
(a) Dominant Feature Fz (b) Two-layer Network F
(c) Illustration of 31. B2, By, and 31
Figure Expressibility of Fz and F trained with combinations of [3,, 83. and Ba (a, b): While including 32 and 33 allow Fy, and F to extract distinct features as and vary, 34 yields less informative representations. (c): Including 3n and 34 does not contribute to distinguish xı and T2-
This section examines the impact of the spike component sy. on Fy. Fo. + sr, specifically how the learned component 87, contributes to feature extraction for unseen inputs.
Corollary 4.7 (Corollary of Theorem 3.3). Let the imputs 21, E Rd and remark Dominant Feature
x2 + (e.g., when æ 22 33. Then, FL(x) = (x) (z) in Definition 3.4 1f xι T
(x1) - #1. (x22)l - 0 and [L()))) - (x2) (x1) (x
Corollary 4.7 shows how the dominant feature Fi behaves under different conditions. Specifically, if the direction of x is not orthogonal to 3.y, then spike of B.g involve feature extraction. Conversely, when æ orthogonal to S.y, the impact of spike Big is eliminated, indicating that the training becomes irrelevant and the feature extractor relies solely on the random features.
To demonstrate this, we define four directions based on two test inputs 11, 22 E gd (vdd) as Midpoint 3 = 2
tr2 Interpolation 3₂ = -1z Extrapolation 33 = 5522 and Orthogonal
34. a random unit vector orthogonal to both x; and x2 s.t. ||33]|| Vi, vi E [4]. By construction, 31 and 89 cannot contribute distinguishing the two inputs via inner product, while 3n and f can distinguish the two inputs. See Figure (c) for an illustration.

---

We evaluate both the approximated features FL and the trained two-layer neural network F on four binary classification tasks, where each task involves Gaussian distributions with means 3, and -3; to form given B,, and covariance 0.17 with n = d = N = 2 For all non-empty combinations of fr, the L2 distances between F(x1) and F(22), and between Fr. (x1) and (x2))), are measured by varying the angle between 1 and 22 from 0 to A radians. The results are summarized using Expressibility, defined as the maximum Lg distance between two feature vectors for a given class combination, normalized by the global maximum distance observed across all combinations, This metric quantifies how well a representation distinguishes structural variations in the input space.
As shown in Figure 3 (a,b), features derived from B1 exhibit limited Expressibiliry. In contrast, features from B2 and B3 are more responsive to imput variations, capturing meaningful structural differences. When multiple expressive fis (e.g.. 52 and (3) are combined, the resulting features are even more sensitive to input changes, reflecting improved representational capacity. These results suggest that training with structurally relevant training classes enhances feature Expressibility, while incorporating unrelated classes as B4 has minimal impact.
Additional experimental results are presented in Appendix L including analyses of B1 and experi- ments isolating the contributions of the Fo, and s7, components. We also provide the original data used in the Expressibility computation of Figure 3 In summary, Consistent with 3q. 31 also shows limited expressiveness. In addition, the sL component from 31 and 3g maps 21 and 22 to nearly identical feature vectors. In contrast, the Fo.L component introduces some separability, suggesting that Fy. and F' calculated from less expressive can still extract weakly distinguishable features.
Experiments
We conduct seven experimental setups to validate our analysis and explore practical implications of the above analysis for open-set clustering, using feasible clustering criteria Recall@) (Remark(5.1)
Remark 5.1. Recall@ / A Err,,s - INN denotes the class of the nearest feature to vi. This serves practical metric for evaluating whether new unseen test classes form coherent clusters.
The choice of recall @1 I is motivated by its ability to reflect clustering quality: when both Cohesion and Separability are high, the nearest neighbor is more likely to belong to the same class, leading to a higher recall @1 1. First, in Experiments I, II and III, we examine the relationship between train- unseen similarity (i.e. ,3 p.) and Cohesion, Separability- as discussed in Section 14. and Recall / within theoretical assumptions. Second, to demonstrate how our theoretical explanations can provide intuition practical settings, we conduct the Experiments IV, V, VI, and VII. For this purpose, we employ the open-set clustering problem using fine-grained real image datasets. For all experiments, we repeat each experiment with three repetitions, except for VII, where we perform five runs.
5.1 Setup for Theory Vaildation: Expr. II, III
Synthetic Data Design Based on Assumptions 2.1 and 2.2 and Condition 4.2 we construct three types of non-centered sub-Gaussian training datasets uniform ball, truncated Gaussian, and uniform sphere (Data 1, 2, and 3). All distributions are origin-symmetrrc. For evaluation, we introduce two Gaussian test distributions- -Eval 1 and Eval 2 parameterized by a translation vector and a rotation matrix A E (O(n) which control the train-unseen similarity. Specifically, as e increases from to I, 3T p increases due to mean shift; similarly, as R approaches the identity matrix I, B8 " also increases due to rotational alignment. Refer to Figure 24 and Appendix for illustrations and details.
Experiment Design For each experiment, we utilize Data 1, 2, and 3, with each dataset paired with a distinct evaluation dataset. Expr. I uses two Eval 1 distributions with translation parameter e1 0.9. 0.9] and e2 -e1, so they are assigned to opposite training classes. Experiments II and III are based on two Eval 2 distributions, each parameterized by a small-angle random rotation matrix R generated as described in Appendix PP] In Expr. II, considering the case where the datasets are assigned to opposite classes, the first distribution uses R, and the second distribution uses R. In Expr. III considering the situation where the datasets are assigned to the same class, the first distribution uses R and the second uses R.
7

---

1250
600
1.0
1000
Siparaciity
400
750
200
500
0.6
250
-00
-00
-50
-00
-20
(a) Cohesion: Expr II
(b) Separability: Expr II
(c) Recall@ I(P): Expr II
600
500
Separability
Recelli3
800
400
500
-60
-20
-60
-40
-20
-60
-40
PT
PT
(d) Cohesion: Expr III
(e) Separability: Expr III
(l) Recall@ I(IP): Expr III
In Expr II, all metrics increase |8 μ4] increases. Lower In Expr III, where two test classes are assigned to single train class, Recall@ and Separability tend to decrease as 133 μa| increases. This aligns with our predictions. In all cases, results using Data 1 reported. The red line represents the values after one step of training, i.e., F. The blue line - represents the values from initialization, i.e., Fo.
Figure 4: Upper
hi
10a.
10
10
10 1.
10-1.
10-5
10-7
10
DjC
Dy-C
Di-5 DEC
D3-5
D.C
Expr.
Expr. II
Expr. III
Figure 5: Summary of the synthetic data experiments: The large and dark circles represent low train-unseen similarity, while the small and light circles indicate high train-unseen similarity. The datasets Dj. D2, and D3 correspond to synthetic Data 1, 2, and 3, respectively. C denotes Cohesion, and denotes Separability. We scale all measurements using the absolute value at the 85th percentile.
5.2 Results of Theory Vaildation: Expr. II,
In Figure 4 we present the results of Expr. II and III using Data uniform ball. These results show that the train-unseen similarity increases, the Cohesion consistently increases. On the case of Separability, Expr. II, the two unseen classes are assigned to different training classes. the train-unseen similarity increases, the Separability also increases accordingly. In contrast, in Expr. III, where the two unseen classes are assigned to the same training class, we observe that Separability decreases as the train-unseed similarity increases. A similar trend is observed for Recall @ which follows the behavior of Separability.
Figure summarizes the results across all cases. For individual plots, refer to Appendix M.2 Except for the Separability in Expr. III, both Cohesion and Separability generally increase with train-unseen similarity (from big dark points to small light ones), with the drop in Expr. III attributed to duplicated training class assignment. These observations align with our analysis. On top that, Appendix M.3 provides empirical evidence that the relationship between the clustering criteria and train-unseed similarity holds in both multi-step and multi-layer settings. These findings suggest that our theoretical framework remains valid, at least to some extent, beyond the two-layer, one-step setting.

---

5.3 Setup for Practical Vaildation: Expr. IV, V, VI, VII
Building on our analysis, which identified conditions for effective feature clustering under unseen distributions, we conduct experiments to examine whether the theoretical insights hold in practical settings. First, we examine if the positive relationship between train-unseen similarity and clustering criteria also holds for semantic similarity and Recall@ (Expr. IV). Second, we evaluate whether the theoretical
feature expressiveness improves only with larger number of non-orthogonal spike directions holds in practice, by examining whether increasing the number of semantically similar training classes enhances clustering performance (Expr. V. VI). Finally, we test if removing redundantly assigned unseen classes improves performance more than random removal (Expr. VII).
We use the datasets CAR (Vehicle, Krause et al. (013)) CUB (Bird, Wah et al. (20TT)) SOP (Product, [Song et al. 2015)), and ISC (Clothing, Liu et al. (2066), referred as Domain. We define ImageNet [Dngg et al. 2009] subsets for the Vehicle, Bird, Product, and Clothing domains, denoted as I(V), I(B), I(P), , and I(C), and refer to them as sub Inl. The classes are manually selected and listed in Appendix O We also define a dataset called subsampled whole Inlk, which includes all classes. To balance the number of samples per class with those in Domain data, we subsample 82, 58, 5, and 6 samples per class in ImageNet variants, respectively. Most experimental configurations follow the approach of Zhai and Wu [019]. a seminal baseline. We utilize ResNet- 18 and 50 (He) et al. 2015) with random initialization and ImageNet pre-trained weights.
5.4 Results of Practical Vaildation: Expr. IV, V, VI, VII
Test
CAR
CUB
SOP
ISC
For Expr. IV, we train with each Domain dataset (CAR, CUB, SOP, and ISC train) and Domain+sub In/k dataset (CV CUB+I(B), SOP+I(P), and IC)+ and then measured how well each model clusters across all test datasets (CAR, CUB, SOP, and ISC test). Tigure@@demmonstraee that clustering test data semantically related to training classes more effective than using classes from unrelated domains. This observation aligns with our theoretical prediction that similarity increases,
CAR
26.97
8.92
27.65
19.75
CUB
14.13
12.54
29.93
18.68
Train
SOP
17.51
9.81
36.63
30.84
clustering performance increases as provided under limited redundant assignments. The trend held in 29 out of 32 experiments. Refer to Appendix [.. for comprchensive results, which also include additional experiments demonstrating the correspondence between train- train-unseen similarity and semantic similarity, well between clustering criteria and Recall@ 1.
ISC
14.17
8.34
31.40
47.88
ResNet1 (init)
Test
CAR
CUB
SOP
ISC
In Expr. V, after learning the Domain, Domain+sub Inlk, and Do- subsumlle whole Inlk, we measure Recull @1 I for corre- sponding testsets. Including all ImageNet classes to Domain offers no clear advantage over including only related ImageNet classes (Figure 7al Mcanwhile, clustering on CAR and CUB with random initialization improved with sum Inlk compared to Domain. This supports our results that feature extraction is driven by sim- ilar training classes with the input forming non-orthogonal spike directions. See Figure [igure33Jand 33d Table 9 for additional results.
CAR
20.96
6.75
25.95
14.86
CUB
7.33
6.59
19.21
Train
SOP
18.17
9.80
37.55
29.77
ISC
11.77
6.76
28.63
47.36
ResNet50 (init)
In Expr. VI, we investigate how gradually increasing the number of semantically related training classes affects clustering performance on each Domain test dataset. Each Domain training datasets is divided into four stages, introducing 25%. 50%, 75%, and 100% of its classes at Steps 0 through 3, respectively. The classes are randomly sampled. As shown in Figure increasing the number semantically related training classes improves performance. Refer to Figure 36 for more results.
Figure Expr. IV, Recall@ I measurements. In most cases, the highest performance is ob- served when the domains of the training and test datasets correspond.
For Expr. VII. during the evaluation phase, removing redundantly assigned unseen classes resulted in a 2 percentage point improvement in Recall@ compared to random removal of the same number of unseen classes, with maximum improvement of 12.71 percentage points, amaximum drop of -6.04 percentage points, and a success rate of 82.81%. This suggests that duplicate asvignments hinder clustering, corroborating our analysis of Separability. We refer more details in Appendix N.2

---

40
40
Recall
Recall
20
20
CAR CUB
SOP
ISC
0
1
2
3
(a) Expr in ResNec18(init. The pink and blue represent Domain, Domain+sub Inlk, Domain-subsampled whole Inlk, respectively.
(b) Expr VI, Recall@1 values for CAR, CUB, SOP, and ISC datasets are shown with dashed lines for ResNet18 and solid lines - for ResNet50.
Figure 7: Key results (a) Expr. V and (b) Expr. VI. (a) Adding all ImageNet classes (Domain + subsampled whole Inlk) yields similar performance gains to adding only domain-relevant classes (Domain sub Inlk). (b) Using more classes within the Domain improves performance.
6 Conclusion
In this study, we investigate the feature learning dynamics of a two-layer classifier in the proportional regime by extending the conventional regression setting and introducing novel technical lemmas, thereby providing a deeper understanding of the mechanisms underlying feature transfcrability. To our knowledge, this is the first work to quantitatively connect train-unsecn alignment with open-set clustering performance, rather than relying solely on hypothesis-level arguments. Our analyses reveal that Cohesion increases with higher similarity between training and unseen data, while Separability depends not only on this similarity but also on avoiding redundant class assignments. Furthermore, our framework characterizes how feature expressiveness for unseen classes depends on the number of semantically related training classes, showing that only spike directions non-orthogonal to the input contribute meaningfully to feature extraction. Beyond validation on synthetic datasets, our theoretical insights also hold for real-world datasets, where we observe improved clustering performance when training classes are semantically aligned with the target domain.
6.1 Limitations and Future Works
While our study provides valuable insights into feature learning and transferability, several important directions remain for future research. First, while the Hermite approximation was helpful for our feature analysis, it introduced challenges due to discrepancies between nonlinear neural networks and polynomial functions used to approximate them. Specifically, we observed that our theoretical predictions tend to align better as the dimensionality increases (Figure 2}. In finite dimensions, while similar trend could be identified the unnormalized results shown in Figure 3 precise scaling alignment was lacking, Moreover, in Section 4.1 we hypothesized that the influence of Hermite approximation coefficients would be positive and confirmed this experimentally, although theoretical verification remains open avenue for future work. These observations suggest promising directions for future research exploring alternative approximation techniques or more refined analytical methods. Second, important direction for future research is to extend the concepts of Cohesion and Separability to multi-step, multi-layes and multi-class softmax, integrating closely
techniques like normalization and temperature scaling. This could help align our analysis with practical settings, particularly in the context of Neural Collapse research under arbitrary input distributions. Finally, in contrast to prior work that emphasizes large-scale datasets broad domains [rown et al. 2020 An et al. 2023). our arguments highlight the importance of domain-relevan datasets for effective feature transfer. This perspective aligns with recent research on constructing efficient training coresets MMirzasoleiman et al. 2020 Paul et al., 2023 Xia et al. 2023 Zhang
al. 2023 Jain al. 2022 and suggests potential new direction for representation aware coreset selection. For instance, reducing the number of training classes or excluding training data that do not contribute relevant directions can lower computational complexity while still enabling successful clustering and feature transfer.
10

---

Acknowledgements
We are deeply grateful Inwon Lee for her invaluable support and encouragement. We are also grateful to Won-Seok Choi for insightful scholarly discussions.
This work was partly supported by the IITP 225-2021-11212000 110--1122220--AAHU//00 RS-2021-11211343 GSAI/10%, 25--222-1220551-BBB/A0
RS- RS-2022- 11122-1220953-ICAA/10
RS-2020-11201377
Artificial Intelligence Graduate School Program (Hanyang \55, RS-2023-002206284 Artificial intelligence for prediction of structure-based protein interaction reflecting physicochem- ical principles/5%) NRF ( 0023-00024896-1mpiccc bias of optimization algorithms for robust generalization of deep learn- ing/4%, the BK21 FOUR (Fostering Outstanding Universities for Research) project/3% NRF- 2024S A45533000444555550000 -oi-----ooo---ccoloogiiiii Technologica Solutions for Bridging the AI Divide; A Blockchain and Federated Learning-Basee AI Training Data Platform/3% RS-2023-002497 14/3%, RS-2025- 25427337/3%) KEIT 200222222234400/11 and Gwangju Metropolitan City (Artificial intelli- gence industrial convergence cluster development project/8%) grant funded by the Korean govern- ment.
RS-20023-02272280-HHII
RS-
References
Xiang An, Jiankang Deng, Kaicheng Yang, Jaiwei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Unicom: Universal and compact representation learning for image retrieval, 2023. URL
https://arxiv.. wwwpii.oorg/bb////0 2330 05884
Jimmy Ba, Murat A. Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High- dimensional asymptotics of feature learning: How one gradient step improves the representation, 2022. /ttps://aaxx org/ abs 211.2225504444
Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, and Denny Wu. Learning in the presence of low-dimensional structure: A spiked random matrix perspective. In Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 17420- 77420-17449, Curran Associates, Inc., 2023.
Zhidong Bai and Jack W. Silverstein. Spectral Analysis of Large Dimensional Random Matrices. URL
Springer New York, 2010. ISBN 9781441906618. doi:
h/dt::/dx.... doi. org/10.
Aurélien Bellet and Amaury Habrard. Robustness and generalization for metric learning. Neurocom- puting, 151:229-2677 March 2015. ISSN 0925-2312. doi: 10..0166jiecccm.2144..00000 URL
http://dx htt.:/xxdoooooor/////// ttw./j.eeucom.....o.mm. www.ee...om.oomm 2014. 09. 044
Peter Bienstman. Mathematics for photonics. Course Syllabus, September 2023, URL https //studiekiezer ugent. be/ wttee/suieeeaaee i hete/en//EEOee0 E002640/curren current Course size: 4.0 credits, Study time: 120 hours. Offered in English and Dutch.
Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses, 2021. URL http::t/p//// :// arxiv. arg//bs/ /000 2tt///003000000 08983
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter. Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https //arxiv. abs/ 2005. 14165
Yuan Cao, Quanquan Gu, and Mikhail Belkin. Risk bounds for over per-parameterized maximum margin classification on sub-gause sub-gaussian mixtures. M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8407-8415 Curran Associates, Inc., 2021. URL http:tttt///rroceddnnnsssss neurips cc/ htapop//...oooooooo...ofoooooole
-Paeeee---PPPPeP-- .ddf

---

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021. URL https
7/arxiv. org/ abs//104........ 14294
Y. Chang, C. Hu, and M. Turk. Manifold of facial expression. In 2003 IEEE International SOI Conference. Proceedings (Cat. No.03CH37443), pages 28-35, 2003. doi: 10.1 A000 1240820.
Niladri S. Chatterji, Philip M. Long, and Peter L. Bartlett. When does gradient descent with logistic loss find interpolating two-layer networks?, 2021. URL battps:///arxt. wwwpriiv...gg.ooooo org/abs/ abs/2012 2012. 02409
Binghui Chen, Weihong Deng, and Haifeng Shen. Virtual class enhanced discriminative embedding learning, 2018. /ttps:////////xxi.v.. wttp://....rr///.1/1..... 1b/11111..1111
S. Chopra, R. Hadsell, and Y. LeCun. Learning& similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'055 005), volume 1. pages 539-546 vol. 1, 2005. doi: 10.1109/CVRR22005.22
Stéphan Clémençcon. On u-processes and clustering performance. In J. Shawe- Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL http:///p/tcpcceeaaiiiiiii neureeddddinnnnn..s.... newrips cc/
.ddf
Frank Cole and Yulong Lu. Score-based generative models break the curse of dimensionality in learning a family of sub-gaussian distributions. In The Twelfth International Conference on Learning Representations, 2024. http:tts::/ /tttpse///peerrevier net forum? wewrrrmmmmiii02x0SSeeee
Hugo Cui, Luca Pesce, Yatin Dandi, Florent Krzakala, Yue M. Lu, Lenka Zdeborová, and Bruno Loureiro. Asymptotics of feature learning in two-layer networks after one gradient-step. 2024. URL https://arxii...oo org/ /bs/
Alex Damian, Jason D. Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent, 2022. URL htlphtttp://arxiv.org /206 15144
Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. How two-layer neural networks learn, one (giant) step at a time, 2023. URL http::///x/itiioorrrg org www.oogggg/bbbbeeeeeeoo /bb/23055 18270
Yatin Dandi, Luca Pesce, Hugo Cui. Florent Krzakala, Yue M. Lu, and Bruno Loureiro. A random matrix theory perspective on the spectrum of learned features and asymptotic generalization capabilities, 2024.
abs/
/tt.b//////..... 22100 18938
Rudrajit Das and Subhasis Chaudhuri. On the separability of classes with the cross-entropy loss function, 2019. URL http:ttss//xxxx.orr///aaaaa// wwtrr//ab//999.....// 190.9090990000 06930
Samet Demir and Zafer Dogan. Asymptotic analysis of two-layer neural networks after one gradient step under gaussian mixtures data with structure. In The Thirteenth International Conference on Learning Representations, 2025. httptttsss/ppereeeeeeeee
net
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. doi: 10. 1ww.1///VV.........222
Jiankang Deng, Jia Guo, Jing Yang, Niannan Xue, Irene Kotsia, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):5962-5979 October 2022. ISSN 1939-3539. doi: 10.1 oog. org/10 1109/TPAMI 2021 3087709
tpami.2021.3087709. /ttp: doi
Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev, and Hervé Jégou. Training vision transformers for image retrieval, 2021. Utl.ittpp://axxii.orr/// wttp://wrwwbbb/....eeeeeeee 05644
Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, and Ivan Oseledets. Hy- perbolic vision transformers: Combining improvements in metric learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7409-7419, 2022.
12

---

Zhou Fan and Zhichao Wang. Spectra of the conjugate kemel and neural tangent kernel for linear- width neural networks, 2020. https://///axi www.aaawwi......aaaaoo..oo abs/2005. 11879
Cong Fang, Hangfeng He, Qi Long, and Weijie J. Su. Exploring deep neural networks via layer- peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43), October 2021. ISSN 1091-6490. doi: 1.0073/pnaa.210309111 URL
http://dx.co doi. org 10. 1073 / pnas .ataa.21100 2103091118
Spencer Frei, Niladri S. Chatterji, and Peter L. Bartlett. Random feature amplification: Feature learning and generalization in neural networks, 2023. URL https://arxiv.orr wttpp///.rrgaa///2222/2/ 07626
Hengyu Fu, Zihao Wang. Eshaan Nichani, and Jason D. Lee. Learning hierarchical polynomials of multiple nonlinear features. In The Thirteenth International Conference on Learning Representa- tions, 2025. URL http:ttt/////eerreeeee
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning, 2022.
org/ abs /abb//2111 /212..22....2. 15121
Erin George, Michael Murray, William Swartworth, and Deanna Needell. Training shallow relu networks on noisy data using hinge loss: when do we overfit and is it benign?, 2023. URL
https://arxiv org 2306. 09955
Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Mare Mézard, and Lenka Zde- borová. The gaussian equivalence of generative models for learning with shallow neural networks. Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, PMLR 145:426- 471 (2021), 06 2020. URL /tttttsss//rrttii org/ org/pdf/ /006 14709. pdf
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016
Geonmo Gu, Byungsoo Ko, and Han-Gyu Kim. Proxy synthesis: Learning with synthetic classes for deep metric learning, 2021. URL httppppa/aaa.i...rrraaaa///////a www.///////////..a//////
15454
X. Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under mse loss: Proximity and dynamics on the central path, 2022.
02073
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL
Hong Hu and Yue M. Lu. Universality laws for digh-dimensional learning with random features, 2022. URL / arxiv. org/ abs 20t.//09990000699999
Mengdi Huai, Hongfei Xue, Chenglin Miao, Liuyi Yao, Lu Su, Changyou Chen, and Aidong Zhang. Deep metric learning: The generalization analysis and an adaptive algorithm. In Proceedings of the Twenty-Eighuh International Joins Conference on Artificial Intelligence, IJCAI-19, pages 3535-2541. International Joint Conferences on Artificial Intelligence Organization, 7 2019. doi: 10.24966/icci.221197332 URL https:://di.oorg/ 21wp://1112999/1jaaaaa 2019/352
Like Hui. Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. 2022. URI. htt.:tttss://rrri.orrrr//// 2200: 08384
L. Isserlis. On formula for the product-moment coefficient of any order of normal frequency distribution in any number of variables. Biometrika, 12(1/2): 1918. ISSN 00063444, 14643510. http: h/tp:///ww..rrrroo.oo jjstor sisstor.rrr//taale//3333333
Saachi Jain, Hadi Salman, Alaa Khaddaj, Eric Wong. Sung Min Park, and Aleksander Madry. A data-based perspective on transfer learning, 2022.
http:/227...5550
Wenlong Ji. Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J. Su. An unconstrained layer-peeled perspective on neural collapse, 2022. URL
02796
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zcoo reasoners, 2023. URL https: //aariv...iii// wwg.abs./////eeetoooo...o.....at abs/ /aee/2205..r 11916
13

---

Simon Kormblith, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better?, 2019. URL httpss//aaxxvvorrr abs/ 1800108974008.88881188888
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fin-grrined categorization. In Proceedings of the IEFE International Conference on Computer Vision (ICCV)
Workshops, June 2013.
Arun Kumar Kuchibhotla and Abhishek Chakrabortty. Moving beyond sub-gaussianity high- dimensional statistics: applications in covariance estimation and linear regression. Information and Inference: Journal of the IMA, 11(4): June 2022. ISSN 2049-8772. doi:
(0.1093/imaia//axc0012. URL http: / /oi. org. wtt.:/.0000//mmmaaaaaaaoooo
Andrew K. Lampinen and Surya Ganguli. An analytic theory of generalization dynamics and transfer
learning in deep linear networks, 2019. URL https://arxiv: wwt.://////aaeaa.ee http:/abb/18809. Uttppptss///arrr...//////////// 10374
Kuang-Chii Lee, J. Ho, Ming- Hsuan Yang, and D. Kriegman. Video-based face recognition using probabilistic appearance manifolds. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings, volume 1, pages I-I, 2003. doi: 10.1109/ CVPR.2003.122 121 1369.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2 Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. URL https: 12597
Shaojie Li and Yong Liu. Sharper generalization bounds for clustering. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6392-6441 PMLR, 18 24 Jul 2021.
URL httpps//pproceedings.
.ow.ediiininmmm.mmi.mmmmm..merem mlr wttp:///nwnn.....eee/ss////31//
Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda. Zhihui Zhu, and Qing Qu Understanding and improving transfer learning of deep models via neural collapse, 2024. URL
https://arxiv.org org abs/ 2212. 12206
Christopher Liaw, Abbas Mehrabian, Yaniv Plan, and Roman Vershynin. A simple tool for bounding the deviation of random matrices on geometric sets, 2016. URL https::/aazii...tt aba/ 1603.00897
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolu- tional neural networks, 2017. URL http://aaiii...rr/////1...... 02295
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hy- persphere embedding for face recognition, 2018. URL https: httptp////a.......ogaaa///////// 08063
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Cosme Louart. Zhenyu Liao, and Romain Couillet. A random matrix approach to neural networks, 2017. URI httpppttss///axxiv.. org/ abs/ 05419
Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent on two-layer nets: Margin maximization and simplicity bias, 2021. URL http::::://axxvv..rr/aa//////// 13905
Hancheng Min, Enrique Mallada, and René Vidal. Early neuron alwgnmentt in two-layer relu networks with small initialization, 2024. URL httpss/ts:::/// wwtpa/wwwww...rar/////////////// .2851
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models, 2020. http:/ttpp://axxiv.orrr//b//////
01827
Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks, 2024. hett/foruumidd-WWesbbpeeee
14

---

Alireza Hossvii,,,,,, Denny Wu, Taiji Suzuki, and Murat A Erdogdu, Gradient-basee feature learning under structured data. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. MR. https:://openrevieu http://pwww..eeeeeeeeeeooooraaa/
Yair Movshovitz-Attias. Alexander Toshev, Thomas K. Leung, Sergey loffe, and Saurabh Singh. No fuss distance metric learning using proxies, 2017. URL
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check, 2020. URL https://arxiv.ort abs attpaaabb////.///// 201./.0000.0000 08505
Eshaan Nichani, Alex Damian, and Jason D. Lee. Provable guarantees for nonlinear feature learning in three-layer neural networks, 2023. Jttpss/////aaxi wwwatiii..../iieee. abs///bb///3330// 06986
Ryan O'Donnell. Analysis of boolean functions, 2021. URL 10386
Guillaume Papa, Stéphan Clémençon, and Aurélien Bellet. Sgd algorithms based incomplete u-statistics: Large-scale minimization of empirical risk. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL UtL/attps://prrceeddang neurips. ccppsscccppprrrrpp
Vardan Papyan, X. Han, and David L.. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40); 24652-24663, September 2020. ISSN 1091 166990. doi: 10. 10011073ppaa.20155591177 / //dx.doi.co ww..oo.g//11... 1073///..////pa. 2015509117
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding 07007
important examples early in training, 2023. URL
Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, and Rong Jin. Softtriple loss: Deep metric learning without triplet sampling. 2020. URL
wwtpa///9.rr//b///99///.// 05235
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https
00020
Maria Refinetti, Sebastian Goldt, Florent Krzakala, and Lenka Zdeborová. Classifying high- dimensional gaussian mixtures: Where kernel methods fail and neural networks succeed, 2021.
URL https: /ttp://///axvvv////////// http//abb2222111111
Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjorn Ommer, and Joseph Paul Co- hen. Revisiting training strategies and generalization performance in deep metric learning. In International Conference on Machine Learning, pages 8242-8252. PMLR, 2020.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filterdd 400 million image-text pairs, 2021. URL https:/tt::///tr
org/ http:/abb/212111/// 02114
Jenny Seidenschwarz, Ismail Elezi, and Laura Leal- Taixé. Learning batra-batch connections for deep metric learning, 2021. URL http::///aaaaaioo//////////..000
Zhenmei Shi, Junyi Wei, and Yingyu Liang. A theoretical analysis feature learning in neural networks: Emergence from inputs and advantage over fixed features, 2022. URL https://arxiv
org/abs/ abs/abbs/2222/ 01717
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No sub- class left behind: Fine-grained robustness in coarse-grained classification problems. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neurul Information Processing Systems, volume 33, pages 19339-19352. Curran Associates, wwwp:ooeeeddinnnnnnnniierppep wwtpssseweerrrp.riaeppeeeeeeeeee ccwpaper.pppppppcep files/ www.s//pppeeeee/e///// /aperr/2000/file
Inc., 2020. URL e01288LL005a100000e1111111256663
.dd
15

---

Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding, 2015. htttpp://axxi..og//aaaaaa 1641.
G. Szegö. Orthogonal Polynomials. American Math. Soc: Colloquium publ. American Mathemati- cal Society, 1975. ISBN 9780821810231. URL https://books..oooott///// google co kr /bo Z0hmnsX1cYOC
Ameet Talwalkar, Sanjiv Kumar, and Henry Rowley. Large-scale manifold learning. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8, 2008. doi: 10. 10.1109/CVPR.
Tom Tirer and Joan Bruna. Extended unconssttned features model for exploring deep neural collapse, 2022. URL
08087
Nilesh Tripuraneni, Michael I. Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity, 2020. URL
hwtgg//bs/200.... 11650.
Roman Vershynin. Introduction the non-asymptotic analysis of random matrices. Chapter 5 of: Compressed Sensing, Theory and Applications. Edited by Y. Eldar and G. Kutyniok. Cambridge University Press, 2012, 11 2010. URL URL httphttts://artt wtt/ ppff/11111111 3027 .pf
Roman Vershynin. High- Sigh-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018.
C. Vignat. A generalized isserlis theorem for location mixtures of gaussian random vectors, 07 2011. URL https: org/ /ptp//df///// 11007 2309. pdf
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Hao Wang, Yitong Wang. Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition, 2018. URL https
org/ abs/aa01..111...
Zihao Wang, Eshaan Nichani, and Jason D. Lee. Learning hicrarchical polynomials with three-layer httpgg/bb/2231111 neural networks, 2023. httt//ttt////xitiit httppss//arrx..oorr//////1111111 13774
Jingfeng Wu, Vladimir Braverman, and Jason D. Lee. Implicit bias of gradient descent for logistic regression at the edge of stability, 2023. URL https:///aar org/ abs/ /a0ss/2305...2 11788
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations, 2023. URL http:pppp///pprrooee.e/////oooep 7D5EECb0af9
Yongyi Yang, Jacob Steinhardt, and Wei Hu. Are neurons actually collapsed? on the fine-grained structure in neural representations, 2023. URL http:/::://aaiv.oor/aaaaa///////
17105
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks?, 2014. bttctt:://aaxii.orr//bbbsse org
John Zarka, Florentin Guth, and Stéphane Mallat. Separation and concentration in deep networks, 2021. httptttss://////m
10424
Andrew Zhai and Hao-Yu Wu. Classification is strong baseline for deep metric learning, 2019.
URL /tttp:://rxxas wttp/ba////.....o/ 12649
Yihua Zhang, Yimeng Zhang, Aochuan Chen, Jinghan Jia, Jiancheng Liu, Gaowen Liu, Mingyi Hong, Shiyu Chang, and Sijia Liu. Selectivity drives productivity: Efficient dataset pruning for enhanced transfer learning, 2023. URL AttLhhtt:///axii
htrggabs//230 08782
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features, 2022.
URL https: / /arxivivv.. org/ hwtgg//b//200... 01238
16

---

Junyu Zhou, Puyu Wang, and Ding- Xuan Zhou. Generalization analysis with deep relu networks for metric and similarity learning, 2024.
www..///g......../////////// 06415
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features, 2021. URL https: //arxiv org/abs/2105.. 02375
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and/or introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in ??
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
The paper provide the full set of assumptions in Assumption 2.1
1 and Assump-
tion 2.2and a complete (and correct) proof in Appendix [
Appendix F and Appendix G
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed reproduce the main experimental results in Section 5 Appendix M. Appendix O and Appendix P
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The paper focuses theoretical analysis and validation based publicly available datasets and existing code. Therefore, no additional code is provided. However, sufficient information is included to allow for faithful reproduction of the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips..om cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
y /nips. /nips.cc//ppp http:/ccccccccguuddeeeseeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper specify all the training and test details in Section 5 Appendix M.
and Appendix O
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper report error bars suitably. We repeat experiment least three times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources is noted in Appendix
A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neuriis.... http:///uurrppsccppplpbleeeee httpp/ubbl..bhiccccieeeeeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work performed in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We answered [NA] because our work does not involve generative models, lan- guage models, any components with high risk of misuse. We only benchmark datasets that are widely adopted in metric learning research and commonly used in demonstrations.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We used four retrieval datasets proposed in previous computer vision research well as the ImageNet dataset for research purposes, and we have included proper citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Since our work focuses on analyzing existing models, we do not introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not conduct crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We answered [NA] because LLMs are not used as important, original, or non-standard components in the core methods of this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neuri)) /tter:///eerrrpp.cc///oooerrerrr ccC///ofeeeeeeeee/222///... for what should or should not be described.
24

---

A Computational Resources
For all experiments, we use one NVIDIA GeForce RTX 3090, Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz and 128GB CPU memory.
B Broader impacts
This paper presents work aimed advancing the field of Machine Learning. In this research, we analyze the potential for clustering performance improvement through the classification training of many classes. Such an approach may reduce the level of personal data masking required for fine-grained data differentiation, which could trigger new ethical discussions regarding privacy protection. Additionally, effectively implement this approach, there may be tendency collect more data, which can have sigmificant implications for the scale and scope of data collection and data management practices.
C Additional Related Works
Table 1: Comparison of Data and Target Settings Between Our Work and Other Feature Learning Studies: We use non-centered Sub-Gaussian distribution the data distribution, encompassing all existing studies below. Moreover, the classification problem requires multiple non-identical distributions, so we adopt non-centered assumption. Unlike these works that assume teacher structure and study whether the feature extractor can learn it, we analyze what structure the feature extractor learns when trained on arbitrarily labeled targets. This enables us to characterize the criteria when transferring features to clustering task, making our setting more aligned with real-world applications.
Data Distribution
Key Target Function
Miscellaneous 7(0,0²)
Ba et al.
2022
70, I)
f(w x:) +
Dandi et al.
2023
10,
f(w] x, w. x)
Cui et al.
2024
x 1),,
f( Vi
Dandi et al.
2024]
x I)0,I)
f(w x)
Moniri et al.
2024
x ~ 1(0,T) T)
[(w x:)
Demir and Dogan
2025
Σ pl (M. Σ)
f(w T xlcc)
is GMM component assign 3 E 10,
Ba et al.
2023
I 11, I + dd pu
f( v Vi4d8
W x))
Mousavi- Hosseini et al.
2023
10,
I+A00 *+1
f( wTa x
+
||0| = L1,
| wil
Damin et al.
2022
x ~ 10, centered SG
f(Ax)
Nichani 2023
fxx Ax)
SG denotes Sub-Gaussian
Wang et al.
2033
1)
fx Ax)
Fu et al.
2025
Uni
f(x A1x, xx. A,x))
ours
non-centered SG
The target is an arbitrary label
Additional Works on Feature Learning in Shallow Networks Dandi et al. [2023 studied how
neural networks can learn feature structure step-by-step during gradient descent. To demonstrate that neural networks can effectively learn low-dimensional structures, Ba et al. [2023] and Mousavi
Hosseini et al. [0233 analyzed the dynamics of learning when both the teacher and the data contain
low-dimensional structures such as spikes.
Demir and Dogan 2025 assumed a Gaussian mixture model for the data, proposed Conditional Gaussian Equivalence, and stochastically approximated a 2-layer network. While this approach might appear similar to our Theorem 3.3 our proof does not require conditional approximations or constructing a stochastically equivalent model, as we rely solely on the Sub-Gaussian property. This provides a more straightforward approach to analyzing the features of new data.
25

---

Building on the work of Damian et al. 2022], several studies analyze the sample complexity of neural networks in learning internal representations while performing regression tasks. These studies consider progressively more complex forms of teacher, e.g., g( Axx), where the data comes from a centered distribution, as discussed in Nichani et al. [2033], Wang et al. [023]. and Fu et al. [2025]. These works primarily focus on regression problems with a teacher, whereas we assume arbitrarily assigned classification labels without using a teacher function. Furthermore, the transfer learning aspect in these studies examines sample complexity when the function head, e.g., 9, is changed, but the internal structure, e.g., x 2;, is preserved. In contrast, our work investigates feature transfer in classification settings where new input distributions are introduced without additional learning.
Extending the heuristic results Cui et al. (2024). Dandi et al.) [2024] analyzed the learned feature extractor using an equivalent model to characterize the test emror in regression problems and analyze the spectral tail behavior of the feature extractor's covariance matrix. Their study explored the phenomenon of spikes in the spectrum, resulting in heavy-tailed distributions. Similarly, we derive an equivalent model suited our classification setup and examine the characteristics of clustering errors for unseen distributions (see Section 4.1). We then investigate the behavior of the spike term in the feature extractor (see Section 4.2).
Feature Transferability in Deep Metric Learning The explanation for how Deep Metric Learning learns transferable features towards unseen data remains insufficient. [hoprr et al. (2005) suggested that CNNs' robustness to geometric distortions enables the creation of generalizable features. This explanation has been replaced in transformer- based research by the idea that, without the inductive biases of CNNs. transformers are less constrained and thus capable of extracting generalizable features EEl-Nobby et al. 2021 Caron et al. 2021].
Lee et al. 2003 Talwalkar et al., 2008 Goodfellow et al.] [0166] [iu et al. [2018). Ermolov et al. 2022 explained that normalized sofumax for metric learning works well because the waarsspherica/hyprrooooo feature space and the data lie on manifold. However, these studies do not adequately analyze how features are learned and transferred through classification.
Following the manifold hypothesis [Chang et al. 2003
There are empirical studies on classifier transfer learning (not feature transfer like ours, which doesn't ' require learning) (Li et al. 2024) and theoretical studies (Lampinen and Ganguli 2019). Trpurrnnni
ct al. 22200))
Neural Collapse (NC) and Features Learned by Classifiers Studies exist exploring Neural Collapse (NC) and features learned by classifiers under the free variable assumption. Hui et al. [2022] argue that NC does not manifest on test data. (Shoni al. [2020), Yang et al. 2023 claim that even on training data, NC is not fully realized, with critical tne-ggained structures concealed. Notably,
Yang et al. [2023] utilized a two-layer network to analyze training data features. Regarding NC on novel data, Galanti et al. [2022] statistically analyzes NC in transfer learning, suggesting that NC generalizes to new samples within training classes and unseen classes with empirical observations.
However, their analysis is constrained by focusing on general function spaces rather than specific
neural network architectures.
Also, there are papers addressing the alignment phenomenon between the network and the training data in neural network classification tasks (Min al. (2024)) ). studies on the increased separability and cohesion of training data features in classification settings (Zarka et al. [2021]. Das and Chaudhuri (019)). ) and research showing that classifier networks outperform linear classifiers through feature learning (Shi et al. 20022) Frei et al. [2023). Refintti et al. (222T).
Furthermore, there are theoretical papers studying optimization properties like implicit margin maxinization (Lyu et al. 2021 Wu et al. 2023]) and interpolation (Chatterji et al. 2021] George
al. 2023)).
MSE for Classification Utilizing MSE in classification is as well-established as using softmax- cross entropy, especially in theoretical analyses of classification problems (Han et al. 2022 Zhou
et al. 2022
26

---

Generalization Bound for Metric Learning Research on the generalization bounds of metric learning related the U-process we use is also ongoing (Bellet and Habrard 2013 Huai et al.) 2019 Zhou et al. 2024 However, these studies do not analyze the exact feature learning structure.
D Additional Notations
The operator diag(-) creates matrix with the input vector elements placed along the diagonal. Let 1 condition be if the condition true and otherwise. Let m!! be factorials of m. Let n!! be double factorial. We define (-1)!! O!! 1. For or, Op, Oy notations,
follow Moniri et al. 2024 |
is the Frobenius norm. ||- is the infinity norm. ||-Ily, is orlicz-2 norm e Standard basis vector with at position [n/2] denotes the floor of n/2. [(2) is the Gamma function.
Additional information on Hermite Polynomials We employ the probabilist's Hermite poly-
nomials (Szegō 1975 Bienstman 2023 Moniri et al. 2024 We denote Ha Hx(x) as k-th Hermite
polynomial.
The n-th Hermite polynomials, H (.), are defined by the recurrence relation: (x)1(x) = = 1 for 2 with the initial conditions Ho(x) 1, (z) r. Using this recurrence, we have (2 = I² 1, Ha(x) = r 3x,
Hermite polynomials can be represented in the following explicit form:
22 d" 22 Hx(x) (-1)"e
+
dx
for n E Ng. Lastly, there is another expression:
(-1)
x nm2m 2m
(x) = n!
(8)
m== m!(n
2m)!
2
The probabilist's Hermite polynomials form orthogonal set with respect to the standard normal weight function <(x)
V2x" e & on the interval (-00,o)) Their orthogonality condition is given
by:
1 02 dr = _ e 22T
Hm (x) Hn(a)
D.1 Hermite Coefficients of shifted ReLU
One of the activation functions that satisfy our condition 2.1 is shifted ReLU,
(Ir) = max(0, x) /2n
(9)
This enables a Hermite decomposition with coefficients given by
Cn (x))())H,,>)).
(10)
Then the zeroth coefficient calculated as
1
co = [x(o(z) x 1 = E, |max (0,, x)]
/2
(11)
/000 nΦ(x)dr
1
/2T
By the way, if + 0, El x Hn]
1 E[ x Ha]
E.Ho x Hn] = 0 by orthogonality.
22n
Thus, the shift only affects 0.
27

---

adapting and extending The coefficient Cn of the expansion of is defined as:
0,
if n - 0,
On
[n/2]
2m
(12)
m=
otherwise.
mll(n 2m / 2m
We directly calculate Equation
12)
and obtain the following result in Figure g
0.5
0.4
0.3
5
0.2
0.1
0.0
0
10
20
30
40
Figure 8: Hermite coefficient of shifted ReLU
E Proof of Theorem 3.1
This section builds upon the proof structure introduced by Ba al. 2022 their technique to our classification learning framework. Unlike their assumption of centered Gaussian training data, we consider non-centered novel approach involving the concentration of the operator norm on random matrix. Also, since our framework not a teacher-student setting, we use class labels instead of teacher function.
data distributions. In this process, we apply
Gradient Decomposition We decompose the gradient (Equation using Hermite decomposition, which allows us extract the essential rank-one matrix structure for each ij-th classification problem. Denote of C1 o' for the Hermite decomposition result.
C1 XTyyaT X.Jyal, @ Gy = n
n XIJo(X, oX, (13)
n
Denote
Aij ya,
1
B3
x ya @ 0' (X))W))
(14)
C₁ L x Wo) o (x).)))))
We derive the norm bound for the terms A.y. B.y, and Ciy in Lemma E.1 We establish the following Thcorem 3.T using these bounds.
We will omit the subscript ij since it does not cause any confusion in Lemma E.1 Remark E.2 and Proof of Lemma E. The following statements hold for Vij. For the aforementioned A, and C. we obtain bounds for each operator norm as follows.
28

---

Lemma E.1 (Probabilistic Bounds on Gradient Decomposition Components).
C c_vá_
) - 2( oN
A
C
VnN
+e
VN
[B|
z
(Vn+ (d+vv)(vV=- vN) S sC(e + te-od_ Ne-olog" + -
nv
C
[|]||
2
(2v vn) log Nog 2 + ne
clog' + Ne -looo n).
VIN
(15)
Remark E.2 (Reformulated Lemma E. In the proportional regime, these quantities can inter- changed to constant n, d, N + Thus, Lemma E.1 is reformulated as follows
A[ I k/Vn) s Cn)
Cloog N
P |BB||
S C(e en + ne -ee-looog n)
(16)
C' log" N
P
||]|
2
S C(ne - en + ne cllglllg n)
Also, for the gradient, we have
1
log log" Opl
||||| || B CII
||
|B]]
+ |C Op(
(17)
n
Proof of Theorem 3.1 Using ||Gi - Aijll [B + Ciyll S
|B + Cisil and Remark E.2
[Gig Anll 2 C log" n
log n
log n
- P | - Aijll 2 C(
< C'ne cloog n (18)
n
n
n
Therefore, almost surely, in the proportional limit,
||Gr Asjll log"1
C Clog"n n Vn R
C log R vn
log" n
||
n¹
(1G + ||G
Aij l)-
vn
vn
(19)
We get (1 - xx' loo'n
|[G
All < n
log'n vn
Gtyll For large enough n for 1 - k¹ loog'n 2
vn
vn
2
log² 2
log" n
|| ا S
GGyll <
(20)
n
Sum up for Vij.
log²
||G - Σ Aull EGs Giz A.sll S
Σ GG
Aul S C
(21)
ic.
with probability C'ne-eneee + -
E.1 Proof of Lemma E.1
Use the norm bound of sub-Gaussian vectors and the spectral bound of Sab-Gaussinn matrices as the main idea of the proof.
Proof of Lemma E.1 (A). We obtain
A
X
ya"
(22)
n/N
Then, we can find an explicit notation of the norm as
|
C1
XX
ya?
X
yllallal12
(v XX
1/2
(23)
nvN
N
nyN
the following two paragraphs, We probabiliitically bound the two components of Equation respectively.
29

---

aa 2 study 2018
By definition,
~ N(0, ) so vNali) is Sub--aussian. Use Thm 3.1.1 in Vershynin
[IvTall- NN 2t) S 2e
lett= = VN
(24)
P([|a]22 S 1) S 2e -NN
(VTXTT) U2 study Note that the U, V matrices resulting from the SVD belong to the O-group. so there is no length transformation.
u112 = | T y Xx y= 1 u1 = IUEV
3/1
Σ V.
P Σ
V y/ 2 o wvll no 2
(25)
We get Tx y) Gaussian matrix. With the result of Remark 1.2 in Liaw et al. (2016)
1/2
2
min. min is the singular value of X which an anistropic Sub-
Pomin I (vn evd)) Se
(26)
Therefore, P(IA S C(
C vdd VnN )) 2( + m
n
VN
Proof of Lemma E.1 (B).
Remark on the definition of IB.
1
B
X ya C (XW.)...
(27)
nvN
BB|
XI| Iya
(XWol
(by Property of Operator Norm)
nvN
| ly | (XW))l |allo (by Fact 1.2
(28)
n/N
1
XX| ||o
(XWa)llaall
(lyl 1)
nvN
We probabilistically bound the three components in the last line of Equation paragraphs, respectively.
28 in the following
||o' (XW)))l study Use the result of D.4 in Fan and Wang 2020 which is held for orthogonal columns. X is sampled from continuous support distribution C1 C2- The first vector is linearly independent with probability vector, which is drawn independently, the probability that it lies in the span of the first vector is 0, it also has a continuous density. This reasoning extends to vectors, implying that, with high probability, they are orthogonal or nearly orthogonal because no vector falls into the span of the others. Thus, AB* 0, the following holds.
due the continuous support of its distribution. For the second
P( 2 C(vá + BJ. S 2c oN
(29)
A = (1ll B)
Woo - 11) < B²}}.
Therefore,
P(I C(vV + VN N)A, B) < 2c -N
<N + PP(T))
(30)
P(AB) study
We choose t t= C N. B = C N
case of [|oo < B
By Lemmal I.3
d
P(IvNNW. 2/N vd) 26 oN
P([ 2 C
< 2e eN
(31)
N
Therefore, |Wol| B at least w.p. 1 - -eN
30

---

case of E ( - -1) B² We know p) S 2e 2c
By definition, [Wo,il 1, so 0 s B², trivially.
P C(vT+VVV
S 2e cN
(32)
Studies of the remaining components
Use Lemma
J.6 and L.3
nN
N d
10' s
w.p. Cle
-N te
(33)
|aa|o0 S
t
w.p. 1 2N -c²
(34)
VN ||]]| S vn vd+E
w.p. - 22 2e
(35)
In summary, we get
BB < C Vv (v=+va+t)
(V
Nu
N2
+
nvN
d
let t = log n, 1' = v
[[[[
(Vi+ va)(vá+ No)log s C(ee-N +e e + Ne Ne-clog" + e -
(36)
Proof of Lemma E.1 (C). Remark the definition of C,
C nN X (xWo)la")ood
(37)
We can bound the norm as follows,
||||| <
X X|llooaa @o'll (by Property of Operator Norm)
nN
< nN
X |X||||0|||]||||||000|
(by Fact J.2)
(38)
| |||||a|||<)||||>>>>>>
(a' is bounded, so ||o'll do VnN).
VnN
We probabilistically bound the ||aa|| to finish the proof.
at consider one Control of ||aa||oo Let Vd. Given X PP[ vd vd) < 2e ct element o (X, wola- = Σ a. (xJ Woli).
We know a. vnWo,s is an independent centered Sub-Gaussian, and use Fact then a( VNWo)a is sub-exponential and mean is since [ Wo.)llv, ||a|] (x Wo,))lee, < co. Apply the Bernstein inequality for the sub-exponential.
J.3
P(lo(X) a)l 2 log given { {X, vdl > vdj) S 2e clog'
(39)
For every element (Xo do < log w.p. 1 [nne" clog nt 2ne
By Lemma 1.6
P([[aa]ao < 11/vN 2 - -et and Lemma I.3 with = vd
C (2vd vn) log nogglogN N) (ne ne -loooloo + Ne- -elogg? u).
|||
(40)
Vn
31

---

Proof of Theorem 3.3
Corollary F.1 (Corollary of Theorem 3.1 By Lemma J.4
we have wip. 1 - o11),
log" n
vn) = 00 log n
|XG
c1 B:jaj = 0
Σ
(41)
n
vn
Remark F.2 (Updata Rule). W1 Wo so XW1 XWo XG. X R is sub-Gaussian random matrix
We generalize Corollary F.I which provides monomial approximation of the data titta-grrdient product polynomial form, into Lemma F3 This result approximates the Hadamard power of the data- gradient product, enabling the decomposition of the dominant feature.
Lemma F.3 (Polynomial Approximation of Data-Gradient Product). For any & E N, sufficiently large n, and w.p. o(1),
- cx Σ B1joT)oki = O(n -i-n log n) [1X
(42)
isj
Lemma F.4 below provides norm bounds for single Hermite polynomial, enabling the Hermite decomposition of the activation function to be bounded in the proof of Theorem 3.3
Lemma F.4 (Operator Norm Bound of ((W)))) Following condition in Section Assume event
Ω = sup (WT
) olop C occurs, the following statement holds.
[H;((WW))lor Op(vnlog3 ny j!)
(43)
We introduce the following lemma to prove Lemma
Lemma F.5 (Bound ElIH((X.)) II)- Following condition of Lemma F:4
E (X llop S Cj!
(44)
The proofs of the lemmas above can be found in Section F.1
Finally, we prove Theorem 3.3
Proof of Theorem 3.3 Let = O(logn)
Denote (2(2) = Et=r at Hx(2), Fr. = (I((W) and FL,0= = oz then,
F= Fi (a
(45)
Bound (a (o-oo) OL)(WW) term Using Lemma
F.5 w in assumption 2.1 w.p. 11-00 o(1)
|E|(o (xW) (o))
C
Met < C
w- --s-w=c/ sc
- dk < CL-2-u
(46)
k=L+1 k=L+1
Therefore, following same proof technique as Lemma F.4 F.5
and J.7 one can obtain follows:
(a -
= or nlog" n --2 = op(vn)
(47)
Also, because 1opp = O(1)
||oo (z)x\()\ = nlog n -- op
(48)
32

---

Bound Ft term
We write FL + Fu,L = =F + Fl. then
FL FL,0 Σ (Ha((X)) - HxW)).
(49)
Thus, We have to study H&(XW) HA (XTo) term.
Hx(XW)- (XWo)
= WH + +GGT)= - (Ho(
(50)
(XG)) + +Σ (1)
k-1
(k)
HR- Hn-)(X o (XG) (by Lemma J.1
j=1
Substituting Equation (50p into Equation 49 expand the resulting expression and complete the proof by bounding A. 2 and Ag, making use of the facts that Po, @(v nn) (as shown by ok Moniri et al. (2024)) and that Eie otck Eies Biga.,)
= D(vD).
p4 = Fo +
c
(XG)
ΣΣ on
((h))))))) (XG)))
k=1
k=1s=1
re + Σ
cf (X
3
k=1
Σ
Big Biaa5)y0 To
k=
i<)
A
(X
(51)
(
k
HR (
Ck
4
Gock
о 18 Σ
Biaa
i<j
( Hk- 2 [
A
ceck
31yayy1 10
i<j
For A,, A2, Ag. is derived as follows:
t
|A <
Σ
ck (x
-c((8 Σ
BijaJ)att
(by Triangle Inequality)
k=1
i<j
(52)
log² 2
C
log 2h
n n # = 0
=(1)
(by Lemma
n
K=1
33

---

k-1
| - Σ2
Σ
Ci
H (XW,)) o (( -c - Σ
(by Triangle Inequality)
k= k= j=1
icj
Hx- H )I - D B(ya
(by Fact J.5
0
vn log 2 / n log (by Lemma
F44 and Lemma F3
n f log +2j n Olog n)
0
(53)
L k-1
A
S
H H (xW) o [
(by Triangle Inequality)
k=1Ø=1
k-1
< C
diag X |||
(by Fact (.2
k=1
(54)
Σ
(by Lemma J.4
k=1==1
k-1
< Σ>
do log' n O(log $ n) (by Lemma F.4
k=1==
Therefore, we conclude the proof.
Proofs of Lemma
and
Proof of Lemma F.3 k: is trivial Corollary F.1 For k 2 2, need to show 3C' > 0, w.p. 1-0(1),
1(&G)k - ct
Bija7))a ook| S Cn log n
(55)
icj
We use Binomial Expansion.
(XG) (&G - xx) Σ
Bijar + +2g8 Σ
i<j
iej
Σ JXG -2x Σ
o5 @ (ax Σ
o0k + +x((8 Σ
Biya
i<j
(56)
Thus,
(X - -xt(x' Σ
BiJJTook
i<]
(57)
(KG xxx) Σ
BijaT To o k-j ' (x oo(k-)
1=1
i<.
Now will show
[[X ciX Σ
BBaTT) 3 o k-1 [
(8
(X
Or (log k+j
$4).
(58)
34

---

(XG--.. Σ
) oc A-1 Σ
icj
ijj
s C ax
Bsya 1 @ (X Jo(k--))))))) (by triangle inequality)
2 C'Iddag( X 9) 1oonl -x
Byja yor Ilop diag(a))k 01
(by Fact J.2)
(59)
& I| G Σ
Biya yosp
(by Lemma
isj
I C(n ((n-2(1-)) log k n) log 23 n. n
log
(by Lemma
On(n- 2 kog*+nn) n)
Therefore.
[({G))* - c2((8 Σ
1
Op(n
log n)
(60)
Proof of Lemma
F.4
Let A = HXW.) then
1
P(l SP
AAT
EAA T 1lop 2
-
-IIEAA T
[lop)
(by Lemma [.7
n
E
AAT
EAA 1lpp
(by Markov's inequality)
|EAAT
op
(IIEAAT Ilo 8)
W
E
||AAA
[lop]
8 max
(by Theorem 5.48 in Vershynin 2010
E [IA
[lop]
5 max
E [IIATT]]
lop], (by Jensen's 's inequality).
Let M E max; |aaes,|HHH (W)x,))' and C. M log
Moreover, we note that || 2
sub- proposition A.6 can
N
Weibull random variable, and the bound of Kuchibhotla and Chakrabortty (0022 be applied.
[. N
Use property
Wo and Hermite polynomials, we have
M < c, E max|m (W, 1|2 - e, E max < <cgg((
<
(61)
Therefore, C Colo Let 12 AA !lop the result derived above, we can continue expanding the expression as follows:
s.t. Qn is positive and increasing. Building on
t
max (V T [/E[IIATT
n [[AAT]
E
[lpp]
12
AA
[pp log n max(
E
AAT
lopp log n)
n
(62)
EI
11 (Qn 1)1 log n max
EIIAA
lops log n)
log n max( / /B|
AA Ilops log n) Q
E [AA
Choosing Qn = logg and using Lemma
E.5
conclude the proof.
35

---

For this proof, we apply the proof techniques from Lemma H.I for Gaussian random variables nub-Gaussian random variables and extend the non-centered technique from Theorem H.7
Proof of Lemmal F.5
For non-centered Sub-Gaussian random variable X with mean μa, the following inequalities hold with t B IR:
E(e(X-u)ey S se²" 122
(63)
Building on the inequalities above, we first prove the case where == with s. E For centered Sab-Gaussian vector let g¹ u, z' by Equation (63
v, p-correlated. s.t. ||u/]2 ||al| = 1, u then
k2 (2x + tz') S expl |u]|3 + +² est 2 11211/(2]
h.2
22 s exp
(s² 2pst + {2))
Dividing by exp( (5²² (2)) then
12 exp(t' 2 _ s explpst) s/y
12
E (22 -
2
By applying proof techniques analogous those in Lemma H.I one can derive the following:
(H H) 9) S ملا0
(64)
For the case, considering non-centered Sub- Sab-Gaussian Random vector with mean μ and centered Sub-Gaussian Random vector & s.t. g = { + μ.. We apply proof techniques analogous to those in Theorem H.7
Denote min(j, k). Considering u g, NT 9.
ET p + u (T)Hl(( p u+ vT
k
= E
(1) H (u 8)}
(v T 1)) Ha-aa E)
h
E
(n (n)--(() 1 ( (T)) {)] by Equation 64
(65)
Σ
(u u))-a(0) (w)T())na u)--a vlp
8=0
< C' kj)
G Proof of Clustering Criteria Analysis
To reduce redundancy and complexity, we first introduce the following notation.
36

---

Definition G.1. Given fixed N. d E R, s.t. the first standard basis of euclidean space, μ1, E R is normalized vector unit length and let
$ Ew--Unis((A- 1B/ [ S E R+
s s(2) d,k,k' = [w 141) (w) 12)
(1)
N.
Pl,k' d,k.tt 1 is even R.+
Pl.k (cos(11, N d.k, 1k4W even E R.
(2)
(66)
(3) &
(,)
16
(r 1)!!(k 1)!l E R+
N
20 5 11')
N ka (r' 1)11(K 1 R+
(r'
We note that PR.K P ((2)
> O. For 5(2) d, it depends on cos(1 P2). As cos(14 increases, d grows, while it decreases as cos( p2) decreases. e.g. when μn = 142, 5(2)
(3)
s(1) 2
and when μr
s(1) k
We introduce Propositions G.2 and G.3 which serve as extensions of the main results presented
in Propositions and 4.3 These extended versions incorporate the complete expansion of the
dominant feature representation.
In the following, provide detailed proof' of Propositions G.2 and G.3 Note that if exclude the
contribution from the component Fo,, in the computation, the resulting expressions reduce to those
in Propositions 4.4 and 4.5
Proposition G.2 (Cohesion of Ft, extension of Proposition 4.4 Following condition 4.2 the
Cohesion of Ft. for ea E [2] is given by:
Pk.k |u
e(Fr)
Cg
+2 +2 Σf Pkcke M M 3)K- ||8|| |a]l
(67)
+ rr= Er= PR.M .rar /u 3 -r-r ||
Proposition G.3 (Separability of FL, extension of Proposition Separability of for 02 is given by:
Following condition
4.2 the
Pk, (cos(u1, (2) (2)))l |p
(2)
Pk.k. I /11--11131 ||3|| ||
(3)
8(F)) Σ
(68)
L= Pakk Ilu 3 -r' || ||mll"
(3)
Er Ph. or (NT ~ "|)
We use the following Lemma prove Proposition G.2 and Proposition G.3 The role of each Lemma is as follows: Lemma G.4 is a foundational result in deriving the expected value associated with the initialized weights of the first layer and the corresponding input data distribution. Lemma G.5 forms the basis of the computation of the expected value of the spike component with the data distribution.
Lemma G.4 (Expectation of the Inner Product Between Uniform Sphere Sample and Given Vector).
Let Ca. A [w T e, *] s.t. w Unif(Sd-1) 1), then
Enl(wT p)*11 |||||
[10 (1) S.
(69)
evon
Proof G.4 The uniform distribution the sphere is symmmeri.mmmrrcc.. Therefore, when & is odd, the Expectation zero. In the other case, also use the isotropic property of a uniform sphere,
[w T [ ||a||* Ewl( ee)*1
I| (5()))
(70)
37

---

Lemma G.5 (Moment of the Product of Gaussian Random Vectors), Given vector a E RN B ER and Gaussian Random vector M1, I). Let b x 3 N( [1 then
(x Ba T)
Σ
(n B) 3)-"" 1)!!T, is even TkT
Ega
(k=
1)!! is evon
(71)
N
(k + k 1) k+k
a T aok'
N
Proof. This follows directly from Corollary J.11
In the proof below, we utilize the results of Corollary J.11
Corollary J.12 and Lemma G.4
Proof of Proposition G.2 Let the Cohesion of the initialized feature be
((Fo,)) Www, Beewer (x) Be'nes (o)
(72)
Let the Cohesion of the feature after training be
e(Fr.) Ewg. Banwyy Fi(x) Barney Fr(w'))
(73)
Calculate c. By Lemma G.4
L
C(Fo,)) = Ew, E
(Wo x)1 T))_EE Ev Σ
cw Hy (W x)]
k=1
===1
Ew,
(Wolll 1ott
(74)
CKCk (11ma K+* s(1) Sdktte) )1 1 1x-k'Jever Jeven
CuCH Pk.k 1
(1)
38

---

Calculate C(FL)
L
C(FL) EW [Eve
(W cxcf(x Ba) T Ex Σ
Hx Hy (Wr +)+ + 3a)
CaCH [Ez Ha x)) TE Hy (Wo
k=1
+ 2E, H. x)) cf (x Ba)ok + eith Ex (x Ba)okT ( Ba)
a Bw, By HA (Wo 2) Ear (x' Ba)ok =((oo.) +
C. cf &+K Fa(Ezz( Ba)ok Ex (x' Ba)okj
((FF.L) 2N Σ
chowe (1 (1
Σ (4) (M 1131|"
1131 (r'- 1)!((K 1)!!1 i''s
N
k
k
C
k'' ,"
(u) g)
|131| 1)!!(r' 1) k,,k'rr,,'''
k+t
00
Proof of Proposition
G.3
Let the Separability of the initialized feature be
S(Fo,t) = Fo. (Tx Exy Fo.
(x)
(75)
Let the Separability of the feature after training be
((FL) = -E Bawey FL(1) Fx
(76)
Calculate (Fo.L) By Lemma G.4
8(Fo.t)
C&e Ew
(Wold) (Wo(o] 12)))
qm
= =-N Σ
Ck CR BwoK (w #2)²1
N
cRck | |0 Ew (w 1
(77)
N
Ce || 1111 Sii lett iseven
ceew || 11 |1
(1)
Pk.k
39

---

Calculate (FF))
8(FL)
E Ex Ho(Wo x) "T,TE Be Hy (Wo 2¹)
+ Brwe, H& (W) x²) Ex'' H (x') Ba)ok'
C& Ewa
+ Br d (
Ba) Her (W x)
N- Cf BeNcs (x T Ba)ok T Ex'nez (2e² BB)ok
of (1lmll
5(1) (
(12 33 3) |||| - 1))!(K' - 1)!! k,k", even
N r'=0
+ +111212 & ((1)
k
k
(m] 1131| 1)!!(K - 1)!! kr,kk is
Sd,k''
N r=0
8(FoL)
C&
+ k+k
k
(K') p'
(M] T 3)m
(ug (u] 3) ||81| (r- 1)!!(r' 1)!!
r=0 r'=0
N b (k + - 1) kklk .e,r" even
Note G.6 (Discussion on the ||||| term). We note that when || decreases, it reduces magnitude of term inside the brackets of Eqs. 6 and (7). This aligns with the intuitive notion that noisier training data leads to less transferable fcatures. To illustrate, suppose the training data consists of two classes drawn from Cj ~ (p, I) and C [(R,, I). where R is a rotation matrix. Then the spike direction converges (under large n) to B + (p Rp) by law of large numbers
1 X n
y
1
X,y
2
1 11
Σ x,
Ex) (u RR). 2
n
In the extreme case where R = I. the classes are indistinguishable and B = O. eliminating both Cohesion and Separability. When R = I. μ., which yields maximal separation.
H Additional Results of Expectation of Hermite Polynomials
The non-standard Gaussian expectation of the product of two Hermite polynomials is computed as follows. It is generalization of the results of standard Gaussian distributions in O' Donnell [202]]. Moniri et al. (2024) into generalized multivariate Gaussian. These provide a useful analysis tool for Hermite polynomials and may offer a foundation for broader applications in future works involving nonlinear activations decomposable into Hermite polynomials under the assumption of a multivariate
Gaussian distribution. We start with previously known facts and derive our generalized results.
H.1
Expectation of product of two Hermite polynomials
Here the result of the expectation of the product of two Hermite polynomials, obtained by utilizing the orthogonality of Hermite polynomials with bivariate centered unit-variance correlated random variables.
Lemma H.1 (Gaussian Expectation of the Product of Two Hermite Polynomials from Lemma C.1 Monini et al. (2024)). See also derivation in Chapter 1 O''Dnnlllllll (2027y.
Let (Z,, 72) be jointly Gaussian with E. [ = [[ = 0, E[Z7] E E[23] 1, and B[Z p.
Then for any ki, k2 E (0.1,....) }}
E (Z) Hz, (Zz) = knlp 1k
(78)
In the other form, for E N, Z n(0, Ia), a. 1
E[H2, (Z 0) (Z T b)] ki T
1x 1
(79)
40

---

We extend Lemmal H.1 to vector form for application in multiple dimensions.
Fact H.2 (Vector Form of Lemma H.1 Let W E Raxa s.t. Vi W E s For Z 10,01).
Ez-no.n)|,,WW ( Z)H+(WW) 2) kM w)
(80)
E ) T H&(W T Z)] k! W 3 1j= kbN11==
(81)
Proof. We apply H2 element-wise. We can acquire the above result by Lemma H.1
The following remark presents a modified condition of Lemma H.1 for the case where a, b E S in Lemma H.1 In this case, the variances of Z and Z b are not equal to I, and the covariance may exceed the bounds [-1, Under this condition, we will compute the expectation of the product of two Hermite polynomials in Lemma H.T
Remark H.3 (the modified condition of Lemma H.1
For E N. u, RR ~ (7(00
Z1 (u, Z) ~ (0, [|_)(2), Z2 = (e, Z) ~ (0, (101)))
Then, Z1, 22 is
Tel
correlated
Ez (u, Z) (v. Z)
{u, v)
corr Z1, 72) =
(82)
uall ||-|||
| |el|
Additionally,
Zi Z22
(6))
( |all2 (1o u) ||ll
(u,
(83)
Now, generalize the unit variance distribution assumptions that Lemma H.T holds for arbitrary vectors as in Remark H.3 This could allow the networks' weights to become analyzable when they go beyond the assumption of lying on the unit spheres.
Theorem H.4 (Generalization of Lemma H.T for unit variance Gaussian distribution Remark
H.3
For d E N. u, Rd, (1, La). (u, ~ 77(0, [[ul1),, (0,9) ~ 70,0, ([01)))
(u H) g)]
|| jim [||1|2- 1)([u(1) - E. (T (TTr)]
jl(u, o/
||1|||11011
(84)
(1|1112 E (u (v T 9)*1+ + (I|ual2
| E, [H T g) g)'l
|0|2
Remark H.5 (arit-variance case of Theorem H.4 The same results can be derived as in Lemma H.] when the variance in Thm. H.4
Proof of Theorem H.4 (Generalize Chapter 1 11.2 Donnell 2021 's derivation to non-unit variance)
Ez=n(0,rry|e=* study
First, we study about E "0)) in order to analysis non-unit variance case.
1
/ e
Bg-n((00=))(**
z0 dg
V2ñO
1 et
/
/opp(-
(g 02t)
(85)
complete square
V2Ta
202
Ez,z (exp(s2 + 12') study
Studying Ez,z/ (sxp(sZ + tZ')], derive what need to show.
41

---

2z,z' exp(s2 + +12)
exp Ro exp (o(u,g)) + exp (1(,, (1)
E Eg-m(.)) exp ((su; + to:)).)
Use Equation (85)
II exp ((ou: + to:)²) = II exp 2
(5 1u11
1
(u, o)st
1 1at 11111)
2
(86)
Therefore,
exp (su') Tuug g { =2(1uul1²)
1 t² |0 12)
exp ((u, u)st)
Eg
exp (to
(87)
Fact H.6 (Facts for the Proof Lemma H.4). One can verify the propositions below with simple calculations.
Let (2) + = H2) Cu = ||ul| - 1, a >>>00 0.
Let f(s) = exp(s2 - 5²)) f(s) = exp(s2 } then
A. By Taylor expansion, exp((u, n)st) = Ei 1
إ! (u, v
B. By Taylor expansion, f(s) EE= 11 fim) (0)s)
C. ()) = (=) CCPP(3))
By using the fact that exp (fu, stst Eg exp(su eliminate the different orders of s & by a Taylor expansion and equating all monomials of the resulting polynomials.
g [|])) we can
jl(u, = (H:(( 9) + Pf(u (H) 9) + P2(0 P3(oT9)C.)
E, (11 T 9) + (Hz((²) 9) (u (H, (1 9) - (o 9))),.)
(88)
Hu H) (u 9)e + (1|ull - 1)000 (v (u 9)r
||]| 1) Hy T g) و)(e [|]]] - 1)B, ()
g)(u m)8l
Therefore,
Eg Hs H0 9)]
|| ||u||||uu|1 1i=k - 1)(111| - 1)
Mun w))
||u|||||u|12 Ey (1 () 9)
(89)
(110112-1 1) || Egg H1 (o) ox} [[lu|] 2 1) R, H, g)(u 9)
||u|l 2
Note that the result of Lemma I.9 can be applied for the concrete calculation of results in Theorem H.4 and conclude the proof.
H.2 Expectation of a product of two Hermite Genemmaiall-Geeeeaaaiaaitiiooo toward non-centered Gaussian
We will change Theorem and Lemma J.9 to adopt generalized Gaussian assumption with non-centered mean.
42

---

Theorem H.7 (Generalization of Thm. H.4 for any Gaussian distribution). For d E N. u E R Ç ~ n( 1). 9 ~ NH, I), Zi (u.g) [pppppppppp u, Eu). Z2 (v.g) 11 v, v Ev).
(2) (Z2))
ΣΣ
2)
k
(u (o
(
U a) Σn-a
(u Eu - 1)(2 T. Ev 1)
x
1 j-
Byl [0 ( u V (Vo E0
u Euv Ev
u STww Ee
Ev- - 1)
(Vu u EuE)(Vo -N + (u Eu 1)
v Eu My [ -Va
Ey [e,[(v V )EE)
[V H4 ( v EvEi]
u Eu
(90)
Proof of Theorem H.7
By reparametrization i.e. 21 = ouE + uuTT n, Zn = V + UTT.
and Lemma J.I
H2(V DVVu + u () TT)) Σ a=
(u T H
991)
a
Hy[H,(u] T Hx [o] = E (V EnE + uu 1)) (VD 0. Eut + v [x]
-Eg
(u T p
1 V/ / EuE) |>
k
(v T 1) Hn aal V TEE)]
(92)
k
(u) T (v) T w) 1 Eus) H (V EvE]]
a
8
Use the same proof technique as Theorem H.4 with (Vu Eug
~ n (e).
[u Eu u Ev
Evie
v Eu v Er
vT
Eg [1 (Vu
U EYEE)
(j - (u T En)a-a 1 1
(u T Eu - 1) T Ev - 1) 11 EyI(Ve u EnEy - (Vo EeE))=BB Dve))*
k8
u EwvT Ev
u1 Evv Ev
(u Lv - 1)
v Ev Egg [Hg (Vu (Vu (uE)(VVV EVTE)) 01 + (u 2u - 1) E Vu EuEr' H a( Vo T EvE)
u Ex
(93)
In summary,
EyHH, (u T H)) T9)
i
(u
(o) uys
3
0=0==
[j - au "oy-
(uTYu- Vu- -))( T De 1)
Euv Ev 1 j-a=k-0
Ee Bel(Ve Eug)i- (V Ee
T Euv T Ev
(u Ee - 1)
(u Tu 1)
Re [M| (V uu Eug)(V (VE) []
Bel(Ve u Engyi H 2( V(Vo'' v EnE)]
u Ev
n Eu
(94)
43

---

I Additional Lemmas of Sub-Gaussian Distribution
For more detailed explanation and well-known results of the Sub- Lub-Gaussian we used, please refer to Vershynin 2010 2018). We show below that the truncated Gaussian distribution utilized in our synthetic data experiments is Sub-Gaussian distribution.
The following lemmas serve distinct purposes in the overall paper: Lemma
employed to verify the Gau--Gaussanttty of Data 2 in Expr. I, II and III Lemma 1.2 establishes a key property that is instrumental throughout the entirety of our proof. Lemmas 1.3 and I.4 provide the necessary
extensions of the results by [a et al. [0022 and Moniri et al. [024]] to settings involving non-centered uub-Gaussian distributions.
Lemma L.1 (Sub-Gaussian Property of Truncated Gaussian). The truncated Gaussian distribution has support on (a,b) s.t. a,b E (-00, 0o) is Sub-Gaussian.
Proof. Denote N () (2, is Truncated Gaussian distribution which have support on (a, b) s.t. a, b E (-00, 2o). We utilize sufficient conditions for Sab-Gaussian distributions based on their tail structure. support (M), (0, o²)) C Rd, Therefore, P[IX] 2 1) s.t. X (O,, o²) has the same tail behavior as Gaussian, and Gaussian is Sub-Gaussian.
L11 Generalization of centered Sub-Gaussian results toward non-centered
We verify below that the results centered Gau--Gassiinn distributions from Vershynin [2018] can be extended to non-centered Sub-Gaussian distributions.
Lemma L.2 Sab-GGassiann Property of Sum of non-centered Sub- aub-Gaussian). The sum of non-centered Sub-Gause Gabusaassinn random variables is Sub-Gaussian,
Proof. If the Orlicz 2 norm is bounded || < co, then X is Sub-Gaussian. Also, | ll < C|| (ay. and the sum of the centered Sub-Gaussian random variable is Sub-Gaussian. We show ΣΣ Xillez 20, s.t. X is non-centered Sub- Gaussannn
Xillvz 1 [(x, Exx + =11 EX,
(Xi - (X: llve
+ Σ |
(95)
(X; - -E-EXX)) 1lve
C E1I
11 <0
Lemma I.3 (Operator norm bound for non-centered Sub-Gaussian matrix, generalization of 4,4.5 in
Vershynin (018 let E TRaxn, A[i][s] is independent, non-centered Sub-Gaussian. Vt > 0.
|| & CK + vn + () w.p. 1 1-e---eexppppp e2)
(96)
Alternativll, ||A|]] S CC w.p. 1 eexp(xxp
K = max:,] ||]
Lemma 1.4 (Expectation of operator norm for non-centered Sub-Gaussian matrix generalization of 4.4.6 in Vershynin 2018)..
|||A|| S CK(Vm + Vn)
(97)
Alternatively, || S CK(Vm and, |||A|| < C(m + n)
Proof of Lemmal 1.3 and Lemma I.4
Based on the result of Lemma 1.2 one can follow the same
proof process of Vershynin 2018
Supplementary Lemmas
Lemma J.1 (Taylor expansion of Hermite polynomials from Lemma C.2 Moniri et al. 2024 ). For any k1, k2 (........ and v, R.
(t)
(k)
H4 + v)
Σ
Hz)
(98)
1=0
44

---

J.1 Lemmas for Norm Bounds
The proofs of the lemmas that require justification are provided in Section J.2 Fact J.2 (Norm Bound for Hadamard Product from Ba al. (022 ) For m E RR/ E R M Rhixa,
mn
@ @M = diag(m)).
(99)
mm
3 M < ||diag(m]] [[M]] [[diag(n]]l ||ma] |o||
Using the sufficient conditions for Sub-Gau Saab-Gausiinity via the Orlicz norm and bound assumptions on activation, the following fact holds: Fact J.3 "Sub-Gaussian Property of Random Variables after Activation). Let Sub Gaussian random variable v s.t. ||llv I k, and bounded function a, then o (v) is Sub-Gaussian, i.c.
||l| cc.
(100)
Lemma J.4 (Norm Bounds of Data, Gradient and Parameters). The following facts will be used in subsequent proofs. Remark spike direction Bij - X.Jyy in Definition 3.2
A. [X,yl Oe(vm), Illll Or(vn), | = Or(1)
B. [KySSaag = || = Op(vn)
C. [Wol = (=))) |W] = 1W + G = |WWl + ||||| = Op(1)
D. [Xi = (v(n)
E. M. & ||ag xc = maxigaeN aas [c] -
C log
w.p - 2ne 2clgn
yn
M, |1 = maxisisn < X()), 3 > S C'log 1/2 n, w.p. -clo n
G. Mw. A sup I1 W.) < C w.p. 1 o(1)
H. || < ||
Fact J.5 (Bounds of Norms Vectors and Matrices). For any vector u, and any matrix A, B
A. Iuo llop = ||u||a|1e12z
B. ||||oo I |lull2 I /nllulla
C. | || < ||
D. | 1|2 S v llac S m (m(axx, (u;])* vnllulla
E. Schur Product Theorem
||A o Bloop
sup tr diag(x) Bdiag(x)) I ||Allop B |=]|=
(101)
Lemma J.6 (Probabilistic Bound the Inf Norm of Sub-Gaussian Random Vector), For Sub- Gaussian V. a,
P( St/VN) > 1 - 2N
(102)
Lemma J.7 (Operator Norm Bound of Random Matrices). Given random matrix A, Following siatement holds,
AAT EAA Hpp 2 22 -EAAA (lop)
P(I op
(103)
n
J.2 Proofs of Lemma J.4
J.6
and J.7
proof of Lemma J.4
is evident from Lemma
1.3 Equation (44) in the proportional regime that A,
B, C, and D hold. We employ proof techniques adapted from Moniri et al. (2024) to prove E, F, and
G. For E, by Lemma J.6 with t = log² à n. Ma < Clogh
w.p. 1 - o(1).
vn
45

---

For F.
P[C|= -) Ex Ex T 3 2t)
(104)
T B] S 2 eexp(-ct) -ct)). Then, P((22 2) - exp( -c(t Ex 3)) - 2 exp(
- P(C 2 1
Therefore, M, v C log w.p. 1 o(1 with t = log² n. For G, Moniri al.
2024]. For H, refer Bai and Silverstein 2010 Corollary A.21.
proof of Lemma J.6 We use the Hoeffding inequality such that
S (Uilal Z 1 s Σ
P([la]|oo
max lail 2
[ai
VN
Lid NP
|asl z
P(IvNa;) 20 S 2N exp( -ct²)
N
(105)
Proof of Lemma J.7
We use the properties of the norm.
8²
P ||
t)
1( 2 2
P(Il AAT 11 op
P(I L AA EAA + EEA 1lop
$2
-)
n
(106)
82
< F(1 AA - EAA 1Pp + ||EAA
1lpp
P(I -(I--AA -E(AA
12
--JAAA
op)
J.3 Expectation of the Polynomial form of Two Gaussian Random Variables
We first introduce Isserlis' theorem. This theorem allows the expectation of the product of centered Gaussian random variables to be expressed as a product of covariances, making the computation feasible.
Theorem J.8 (Isserlis' Theorem Isserlis 1918 Vignat 2011). X (X Xa)
Gaussian random vector s.t. E[X] O, and let A for,".. aN be set of integers s.t. 1 a₁ d, Vi. Denote XA- = IIo,e X and Xg = Let (A) denote partitions of into disjoint pairs, and
E (A) is a pair.
B[XA] Σ II E[X., Xo, even.
(107)
(A)((A) (i.j)en
Lemma J.9 (Moment of Centered Bivariate Gaussian Variables). For N, M.U.R. ~ (0,
Zi (u. g). Zz (v.g).
Z1
(())
(0-((11) |a/l2 (u, (v, u) ||011
(108)
Zz
Xay is defined at Thm.
J.8
13
(-1)m
Ez, 2g (H,(1)
j!
Σ
1itk II E[X., X
m![j )2m
m=0
TE[T((()) 2m 2m){[(2))xK)) (m-))e
E,, 22 12/2,11
Σ
II
B[X,, Xa, 1 is
-T12113/4HH (p:
(109)
46

---

Proof. By the explicit formula of Hermite polynomials
Σ (-1) m=U
E 2, (2,2
jز
ml(j 2m) 1 E2, 121 121--m
2m 2
(110)
Therefore, need to figure out Zy]. We know Z1, Zz is mean-zero Gaus-
sian. Thus, we can apply Theorem J.8 with A (121) p} U [122) x 9)}. B[Z EGEI((A) [[(p,)) M[ I is
Corollary J.10 (Corollary of Lemma Moment of Centered Univariate Gaussian Variables).
Remark Z1 17(0, [[u]l?) For the case k 0,
Ez, [2]]
|/ 1)!!1 even
(111)
Proof.
E2,,2,144 = E,, =
Σ
II
E,Xa, Xa. [a,]l]isoun jss
dE TT][(( x5)) (D.0)Ea
(112)
Σ
II
1 is
Σ
| jiseevn =i-
-1)!!/u/|/1 is even
( (M.O)E
The following Corollary. which calculates the Expectation of the Power of a Gaussian Random Variable, be derived using the binomial expansion with the reparametrization technique and Corollary 1.10 It corresponds to the case & = 0 in Lemma J.9
Corollary J.11 (Corollary of Lemma J.9 Moments of Gaussian Variables), Given w E Rd. let Gaussian Random Variable Z ~ M(y" wl))) then
Ez(Z)*
Σ
k
(pl "B2
(113)
(w wh)
'(t 1)!! ||||| 1 is
The following Corollary, which computes the Gaussian expectation of Hermite polynomials, is derived from the explicit form of Hermite polynomials and Corollary [.10) It corresponds to the case
k = O in Theorem H.7
Corollary J.12 (Corollary of Theorem H.7 Gaussian Expectation of Hermite Polynomials). Given w 8d-1 let Gaussian Random Variable Z N(p w, 1), then
Eg [H x)] (ewn((n))/Hale +
k)) i
(114)
Σ
(w (w) ())EEHH TH)=
47

---

K Empirical Insights into High-Dimensional Asymptotics
In asymptotic analysis, n, d, + oo crucial for observing the result. Please see Figure Figure I0 for the Cohesion and Separability in R22000 20000 R330000 respectively. As the dimension increases, the range where Cohesion and Separability align with our expectations.
For component analysis, please refer to Figure I1] Figure 12 and Figure 13 for Cohesion, which demonstrates progressively wider ranges, and Figure 14, Figure 15] and Figure [6] Separability, which exhibits a same ranges. We compare Cohesion and Separubility from the predominant 87. term (Equation (6) and Equation (7), with the suppressed contributions from other terms of Cohesion and
Separability of Fz (Equation 667 and Equation 68
L Additional Observation of Multi Classes Feature Analysis
We conduct experiments using new input data points æ1 and T2, and spike directions B2 and fs, which are not orthogonal to I and 22, well as 84, which is orthogonal to both in the main text. Furthermore, we extended the experiment by incorporating the midpoint direction B1 - in tra See Figure 17 Consistent with our analysis, we observed that B1 also does not contribute significantly to feature formation.
To isolate the source of this behavior, we decomposed the network function Fz into Fo. and s., and conducted additional experiments. See Figure 18 This decomposition confirms that the observed phenomenon primarily originates from s..
We also provide the original (unprocessed) data used before expressibility computation. See Figure 19 Figure 20 Figure 21 and Figure 22 Expressibility is the maximum feature distance achievable when two data points are rotated to maximize their angular separation. This corresponds to the slope the original plot before scaling.
summarize these slope values in Figure 23 to offer an alternative perspective. The results demonstrate that including either midpoint or orthogonal directions during training leads to negligible changes in slope.
These observations imply that training data aligned with the midpoint or orthogonal directions of new input data do not contribute meaningfully to feature extraction.
48

---

878
1000
500
100
200
502
166
160
40C
-50
6'1
6'P
30000
60000
10400
35000
10000
2352
-400
-202
9
200
405
460
-220
0
200
460
460
-200
0
200
400
TC403
50053
20000
20000
30000
-1000
-200
1.5
0.
00
0.0
$530
6'a
Figure 9: Cohesion 2200 2220000 R 320000 (left to right), with the computed range expanding from top bottom.
49

---

7.6e100
6.3e-00
63e100
1.7e+02
1.6c
3,0++00
3ic100
1.24+02
2.5ex00
1.te+02
1.3e+00
1.3e+00
1.5ex
00e+00
0.0+-00
2.5e 30
1..e+30
-23e+30
55.0
1.6e+
1.te+02
-7.5
-10
5.0e+00
-10
-3
1.5++0Z
100
50
75
0.0e+00
0.
100
50
75
5.0e-02
3.ee02
100
50
75
7.6e102
Be4
25
2.5e+32
3.8e1
1.3ex02
0.0e000
25
0.8e+02
2.
-10
-25
-7.6e1-22
-5.1e400
6.4e+02
-50
-55
2.5e..02
-.8e4432
-25
-50
2.5e1
0.0e+00
3.3em:2
000
-10
0.3c1 02
-100
3.8e+02
P
Ph.
2.5e+01
622+00
5.0e 04
200
0.0e100
200
936403
03
8.5e+03
11.2+005
033e+04
0.0e400
1.5e+05
147e404
3.2e00
1e+00
22 22.0++20
2.0e+04
48 Se443
2.2e+00
400
400
3e+e0
-400
-00
200
400
400
-20
0
200
400
8 PL.
3.8c-05
-1.1e-05
3.5404
-7.7c 05
500
2.2c+06
2.5e-01
1.5e-06
-.40--08 4550--8
0.0c 00
-1.90 06
5.5e-06
-1.ee+01
-300
2.7e-00
-7. Ith
3.8e 04
3.10-06 06
8.8c-06
-5. 5.0ex00
00000000
000
500
1100
-500
10000
0.0e+00
10000
10000
0.0e-00
.7.5x400
.280s14
6.6c+10
5000
-1.1e4-10
5000
4.3e+14
5000
-1
1.5e+10
7.1e114
-.Te+11
2.3c:
8.6014
2.0e=11
-5000
-000
5000
3.e++1
3.4e+10
1.3e115
-.0e+11
-10000
15000-5000
5000
10000
-10003
05000-5000
5000
30000
15000-5000
5000
Figure 10: Separability in 2000 20000 320000 (left to right), with the computed range expanding from top to bottom.
50

---

600
600
500
Cohasion
Catesion
400
400
300
100
100
01
-100
-50
100
-100
-50
0
50
100
-100
-50
100
P
17.4
17.3
417.5
Conesior
Coneston
617.0
16.9
16.0
118.0
16.7
100
-50
50
100
f 1'
8
(left right) in the range Cohesion of the dominant component ic. Equation other terms in Equation (67). which make only suppressed contributions.
Figure 11: Component analysis of Cohesion in 2200 22000 TR 320000
[-100, 100]. Top:
66 Bottom: sum of the
20000
15000
15000
12500
00000
Conestan
10000
40000
5000
20080
5000
-000
-200
200
400
-100
-00
0
200
000
-600
-200
200
400
$
0
425.0
17.0
422.5
Comesitn
D2heeion
16.5
470.0
417.5
412.5
410.3
400
14
PT
(left to right) in the range [-500, 500]. Top: Cohesion of the dominant component &, i.e. Equation 6 Bottom: sum of the other terms in Equation (67). which make only suppressed contributions.
Figure 12:
Component analysis of Cohesion in 2000 20000 320000
51

---

73000
1.0
30000
50000
Canveiion
Cihezton K0n
40000
0.6
50000
33000
0.4
20000
10000
6.0
-500
500
1000
1000
500
8
500
1000
1000
500
8 Pb.
300
1000
4.90
10
19.0
Cofrestonn
Colosson
7.5
415
5.0
405
2.5
-20
1000
-500
530
1000
-500
530
1000
1000
1000
(left to right) in the range [-000, 1000). Top: Cohesion of the dominant component sy i.c. Equation (6). Bottom: sum of the other terms in Equation (67). which make only suppressed contributions.
Figure 13: Component analysis of Cohesion in 2200 F 20000
RR 320000
2.6e02
1.36 02
50e=02
38e+02
0.0c
50
50
2.5e+02
50
1.30 O2
25
2..e 47
25
1.3e+02
25
0.0e=00
3.0e 42
5.
1.3er02
-50
G.4e+02
-50
-50
7.6e 02
-15
8.0e102
-75
-15
-302
-100
4.3e-22
100
100
75
1.0e00c
75
1.0e=20
1.6ev00
1.0e=00
2.50 422
1.6e00
1.0c+00
1.8040
1.0 1.0e400
1.8e4DC
1.0 10e=000
1.6e+02
-2b
1.5e4nc
-5
9.9 90
1.6e.02
1.5er0
99e-01
200
1.50
1.50402
996-01
1. 1.b440
1.5e402
9.0e00
-00
3. 1.0e407
-100
-50
1.5e+00
-100
400
-230
200
400
3. 1.ee002
Figure 14: Component analysis of Separability in 2000 IR 20000 320000 (left to right) in the range
[-500, 500). Top: Separability of the dominant component s, i.e. Equation Bottom: sum of the other terms in Equation (68). which make only suppressed contributions.
52

---

L0c400
3.0e+04
Se
42u105 05
Se5003
5,0e 104
0.0x-000
200
+4 4.2e-25
9.2e103
-3.0 42
4 0.5e=20
4.3a=-2
3.50 02
4.7e=20
-20
3.60 0z
-200
2.1e=24
-200
2.00 4z
2.8e=24
2.2e+02
400
3,0e-20
-400
200
200
400
100
P..
1.5c 100
100
1.2c
9.8e
400
1.6e
5.20 C1
2.50 4z
200
6.20 C1
200
20c
1.50 02
3.1-01
3.3e-07
0.0e-00
8.5e01
1.8e+07
-33e-51
8.60011
3.6e+02
-203
-62e-51
200
8.4e01
-00
1.6e102
-.2e-91
8.2001
3.6e 02
100
0.2e+00
-400
0.0e01
402
1.6c 02
-400
-200
200
400
-200
-200
1.
8 P1
(left to right) in the range [-500, 500]. Top: Separability of the dominant component sy i.e. Equation other terms in Equation (68), which make only suppressed contributions.
Figure 15: Component analysis of Separability in 2000
IR 20000
R
320000
7 Bottom: of the
150
3.3e400
750
300
300
500
250
2.3es06
150
250
2.7e+06
5.2e+06
5.0e04
-750
-3.5e00
-750
-750
4.3+o0t
1100
1500
550
1000
-100
1500
500
0
500
1000
-1000 10c0
500
503
1000
81
1000
5:2w+00
1000
1003
150
D.D
750
64c 100
750
1 1.0e02
3.2e-tu tu
5.3e+00
1 1.6e002
5.1e+00
1 1.0e-02
250
2244 tu
250
4.5e.00
1 1.0e402
-3.5c 00
4.7e100
3.9e+20
1 1.64402
3.5e+00
-200
3.3e=20
-250
1.60 02
2,6e+00
166402
7.3e+00
2.0e+00
16er02
-350
5.7e+00
-750
1.4e+00
-750
1. 1.Ge+02
4.te400
-1000
-1000
-900
500
1000
7.0e00
-1000
50c
-500
500
1000
1.0e=02
Figure 16: Component analysis of Separability in 2000 20000
320000 (left to right) in the range [-1000, 1000]. Top: Separability of the dominant component sy i.e. Equation (7). Bottom: sum of the other terms in Equation (68), which make only suppressed contributions.
53

---

B1 B2 B3 B4
B1 B2 B3
B2 B3 B4
B2 B3
B1 B3 B4
B-combination
B1 B3
B3 B4
B3
B1 B2 B4
B1 82
B2 B4
B2
B1 B4
B4
B1
0.00
0.25
0.50
0.75
1.00
Expressibility
(a) Theory
B2 83
B2
B3
B4
B1
B2
B3
B4
B1 B2 B3
B1 B3
B-combination
B1 B3 B4
B3 B4
B3
B2 B4
B2
B1 B2 B4
B1 82
B1 B4
B1
B4
0.00
0.25
0.50
0.75
1.00
Expressibility
(b) Two-layer Network
combinations of B,s. All cases are strongly influenced only by Bg and Bg directions. Thus when using only the 33 or 8g directions, the two features are always mapped to the nearly same position.
Figure 17: Expressibility measurement with F, of Σx (+)
54

---

B1
B2 B3 B4
B2 B3 84
B1
B3 B4
B1 B2 B4
B1 B2 B3
B-combination
B3 B4
B2 B4
B2 B3
B1 B4
B1 B3
B1 B2
B4
B3
B2
B1
0.0000 0.0005
0.0010 0.0015
Expressibility
(a) Fo.
B1 B2 B3 B4
B1 B2 B3
B2 B3 B4
B2 B3
B1 B3 B4
B.combination
B1 B3
B3 B4
B3
B1 B2 B4
B1 B2
B2 B4
B2
B1 B4
B1
B4
0.00
0.25
0.50 Expressibility
0.75
1.00
(b) 81.
Figure 18: Expressibility measurement with Fo.L, $ Σ ( combinations of B,s. Fo.L is not influenced by trainind data and generates random features in all cases. sy is influenced only by 8g and B3. Thus when using only the B1 or 84, the two features are always mapped to the same position.
55

---

slope
3: 0.36
2: 0.22
4: 0.02
0.02
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
1.0
0.8
0.6
0.4
0.2
0.0
cos X2)
Two-Layer Network with one 3 direction
slope
(2, 3): 0.45
(1, 0.36
(1, 2): 0.21
(3, 4): 0.36 --- (2, 4): 0.22
(1, 4): 0.03
0.4
f(x
0.3
0.2
[lf((1)
0.1
0.0
1.0
0.8
0.6
0.4
0.2
0.0
cos(x1, X2)
Two-Layer Network with two B directions
slope
(2, 4): 0.45
(1, 2, 3): 0.44
(1, 2, 4): 0.21
(1, 2, 3, 4): 0.44
--- (1, 3, 4): 0.36
0.4
(1x))l12
0.3
0.2
0.1
0.0
1.0
0.8
0.6
0.4
0.2
0.0
cos(x, X2)
Two-Layer Network with multi 3 directions
Figure 19: Experiments for 8r combinations. We vary angle between æi 22 and measure L2 distance of two-layer network feature F. Scaled y-axis by dividing 10-2/N.
56

---

slope
3: 3.84
2: 3.08
4: 0.02
1.0.
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
1.0
0.8
0.6
0.4
0.2
0.0
cos X2)
Theory estimation with one direction
slope
(2, 3): 7.41 --- (3, 4): 3.84
(2, 4): 3.08
(1, 3): 5.55 --- (1, 2): 3.64
(1, 4): 0.02
7.0
6.0
5.0
4.0
3.0
2.0
1.0
0.0
1.0
0.8
0.6
0.4
0.2
0.0
cos(x1, X2)
Theory estimation with two 3 directions
slope
(1, 3, 4): 8.72
(2, 3, 4): 7.41
(1, 2, 4): 3.64
(1, 2, 3): 8.72
--- (1, 3, 4): 5.55
8.0
2
6.0
4.0
2.0
0.0
1.0
0.8
0.6
0.4
0.2
0.0
cos(x, X2)
Theory estimation with multi / directions
Figure 20: Experiments for 8r combinations. We vary angle between æn z2 and measure L2 distance of approximated feature Fz. Scaled y-axis by dividing by 10-2 10-2/N. /N.
57

---

slope
4: 0.02
3: 0.02
0.02
0.02
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
1.0
0.8
0.6
0.4
0.2
0.0
cos X2)
Fo, with one 3 direction
slope
(3, 4): 0.02
3): 0.02
(1, 3): 0.02
(2, 4): 0.02
(1, 4): 0.02
(1, 2): 0.02
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
1.0
0.8
0.6
0.4
0.2
0.0
cos(x1, x2)
F with two B directions
slope
(1, 3, 4): 0.02
(1, 3, 4): 0.02
(1, 2, 3): 0.02
(2, 3, 4): 0.02
--- (1, 2, 4): 0.02
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
1.0
0.8
0.6
0.4
0.2
0.0
cos(x, X2)
Fo with multi 3 directions
Figure 21: Experiments for 8r combinations. We vary angle between æi 27 and measure L2 distance
of approximated initialized feature Fo. Scaled y-axis by dividing by 10- 10-2 //N2.N.
58

---

slope
3: 3.84
2: 3.08
4: 0.00
0.0000
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
1.0
0.8
0.6
0.4
0.2
0.0
cos x2)
81. with one direction
slope
(2, 3): 7.41
(3, 4): 3.84
(2, 4): 3.08
(1, 3): 5.55
(1, 2): 3.64
(1, 4): 0.00
7.0
6.0
5.0
4.0
3.0
2.0
1.0
0.0
1.0
0.8
0.6
0.4
0.2
0.0
cos(x1, X2)
8y with two 3 directions
slope
(1, 3, 4): 8.72
(2, 3, 4): 7.41
(1, 2, 4): 3.64
(1, 2, 3): 8.72
(1, 3, 4): 5.55
8.0
2
6.0
4.0
2.0
0.0
1.0
0.8
0.6
0.4
0.2
0.0
cos(x1, X2)
8L with multi / directions
Figure 22: Experiments for 8r combinations. We vary angle between æi 22 and measure L2 distance of approximated initialized feature sL. Scaled y-axis by dividing by 10-2, 10-2/N.
59

---

Inter
Extra
Inter
Extra
3.08
3.84
0.22
0.36
7.41
0.45
3.64
3.84
0.21
0.36
8.72
7.41
0.44
0.45
0.02
0.02
0.02
0.02
8.72
0.44
5.55
3.08
0.36
0.22
5.55
3.64
0.36
0.21
Mid
0.02
Orth
Mid
0.03
Orth
(a) FL
(b) F
Inter
Extra
Inter
Extra
0.02
0.02
3.08
3.84
0.02
7.41
0.02
0.02
3.64
3.84
0.02
0.02
8.72
7.41
0.02
0.02
0.00
0.00
8.72
0.02
0.02
5.55
3.08
0.02
0.02
5.55
3.64
Mid
0.02
Orth
Mid
0.00
Orth
(c) Fo.
(d) $L
Figure 23: Slope measurements summarized in the Venn diagram from Figure 19
Figure 20
Figure 21 and Figure 22, Each intersection represents training with the corresponding combination of B; directions. Fo. not influenced by trainind data and generates random features in all cases. F, Fz and & influenced strongly by 82 and 83, so when using only the 31 or Ba, the two features always mapped to the nearly same position.
60

---

cBt
Train Data 1, 2,
Eval 1,
Figure 24: Examples of training datasets (Data 1, 2, 3) and evaluation data Eval 1,
M
Additional Observations of two-classes Experiments
M11 Additional setup for Experiments I, II, III
We set n N 2 and use Shifted ReLU.
Training Datasets For illustration, see Figure 24 (Data 1) two uniform distributions radius- v ball, (Data 2) two
element- wise truncated Gaussian distributions, and (Data 3) two uniform distributions over a radius-vdd sphere, symmetric about the origin The two means of training class are denoted as u and -t, respectively. For Data 1, 3, we set A 2r u, with Unif (Sd-1). For Data 2, one class has support on [1, oo) across all dimensions, while the other class has support on (-o0,- -1].
Evaluation Datasets Eval 1 and 2 use the projected Gaussian distribution, which is projected onto the mean direction one training data v, as defined in Note M.I. For Eval we translate the mean of the projected Gaussian distribution with e; for Eval 2, rotate the mean of the projected Gaussian distribution with R and fixed e. Note that we generate distinct rotation matrices R using process in Appendix
Note M.1 (Sampling projected Gaussian distribution). For Eval 1, let ev, c = and for Eval 2, let v A Rev, c = 10-1 with = 0.01 for Data experiment and = 0.008 for Data and experiments. Note R (d)).. The projected Gaussian distribution is sampled follows,
Tvv
+
where
2~ (0,00,,T).
(115)
|
M.2 Comprehensive Results of All Experiments
The results of Expr. I are presented linear scale Figure 25 and logarithmic scale in Figure 26 Additionally, as demonstrated in Figure 4 experiments for Expr. II and III settings are shown in linear scale Figure 27 with results for Cohesion, Separability. and Recall@ (IP). Furthermore. @ecall@) (cos) results are presented in linear scale in Figure 28] Recall@ / in both cosine and inner product follows similar trend to Separability. All observed results align with the theoretical predictions.
M.3 Empirical Observations in Multi-Step and Multi-Layer Settings
To further examine the potential validity our theory in multi-step and multi-layer settings, we empirically observe that increasing 3 generally leads to improvements in both Cohesion and Separability, consistent with our theoretical predictions.
M.3.1
Setups
To avoid gradient explosion or vanishing issues, we the learning rate 0.001 for multi-step experiments, and to 0.1 for multi-layer experiments. For the multi-step experiments, we performed gradient descent on all parameters, including W and agj For the multi-layer experiments,
'The Sub- Sabb-Gassiinn property is proven for Data and in Vershynin 2018 and for Data in Lemmal L.I
61

---

constructed deep networks by repeatedly applying the feature structure defined in Section 2 For example, in a three-layer setting, the network defined as
F(x) - ( xW)
M.3.2 Results
and 32 for results. For Cohesion, we observe that training progresses or the network depth increases, Cohesion also increases with larger values of [ For Separability, although it generally tends to increase with larger 3 ju training proceeds or the network deepens, we note two exceptions: when training is extensive, Separability decrease at small 3 p.. and when the network deeper, Separability decline at high 3 μ. These behaviors are likely due to nonlinear learning dynamics and the nonlinearity of the model. This suggests an interesting direction for future work: identifying a potential critical threshold of alignment strength in deep models that balances directional consistency and proximity.
Please refer Figures 29
M.3.3 Insights
We discuss potential phenomena that may arise in multi-step step training and deeper models, and consider the possible extensions of our theoretical framework.
Multi-step training. Our findings suggest that extending the analysis beyond single-step update may provide additional insights. In particular, one possible hypothesis is that multi-step training progressively refines the spike component by amplifying directions that enhance Cohesion and Separability while suppressing others. Such process could also facilitate more efficient information compression for classification and improved transferability. The analytical framework of Dandi et al.) [2023), which characterizes the "staircase property" in multi-step dynamics, may offer promising starting point for developing such an extension.
Deeper models. Although our two-layer model captures key aspects of feature learning, extending the framework to deeper architectures could shed further light on how transferable features emerge. Previous studies Fan and Wang [202)]. Nichani al. (2023), Wang et al. [2023] have already generalized conjugate kernel analysis to multi-layer settings, suggesting that a similar extension of approach might be feasible. Interestingly, our preliminary multi-layer experiments indicate that when the depth increases, particularly in 3- and 4-layer models, improvements in Separability are not always sustained. Exploring this potential threshold effect represents intriguing direction for future investigation.
62

---

1.00
Cohetoo
Colvesion
9.50
0.25
0.00
-8000
-4000
4000
4600
4000
-8000
-0000
Cohesion: Data
Cohesion: Data 2
Cohesion: Data
60000
50000
100000
Separabiity
Separabiley
40000
30000
50000
20000
20000
10000
4000
4000
0000
-4000
4000
8000
8TP
Separability: Data 1
Separability: Data 2
Separability: Data 3
1.0
1.0
0.9
Recoii11
0.8
0.8
0.7
0.6
0.6
0.5
-8000
-4000
4000
8000
40000
0
4000
-8000
-4000
4000
8000
Recall @ I(IP): Data 1
Recall @ L(IP): Data
Recall I(IP): Data 3
1.0
1.0
1.0
0.9
0.8
0.7
0.6
0.5
-8000
-4000
4000
8000
-4000
0
4000
-8000
-4000
4000
8000
TTu
Recall @ (coo)): Data 1
Recall @ (coos): Data
Recall @ (ccos): Data 3
Figure 25: Expr. I: translation (e) variation case (linear scale). - is after one step training. - is from initialization. As the train-uod unseen similarity increases, both Cohesion and Separability become larger. Recall@/ follows in similar trend to Separability.
63

---

1.00
0.50
0.25
0.00
10
10
-10
10"
10
10a
Cohesion: Data
Recall @ 1(cos): Data 3 Cohesion: Data 2
Cohesion: Data
60000
50000
100000
Separabilny
40000
30000
50000
20000
20000
10000
20
100
-107
10a
B
8T
Separability: Data
Separability: Data 2
Separability: Data 3
0.9
RecHiBB
0.8
0.8
0.7
0.6
100
100
10'
107
10
10
1.07
8"
8T
Recall @ ((P)): Data 1
Recall @ (IP): Data 2
Recall @ (IP):: Data 3
1.0
1.0
0.9
Recollgg
0.8
0.8
0.7
0.6
0.5
107
10
10a
10.
107
10a
10a
1.0
10'
10
6
8
Recall @ ((cos): Data 1
Recall @ ((cos): Data
Figure 26: Expr. I: translation (e) variation (log scale). - is after one step training. - is from initialization. As the train-unseet similarity increases, both Cohesion and Separability become larger. Recall@/ follows in similar trend to Separability.
64

---

700
600
200
500
Separabiity
0.8
400
300
-200
200
-40
-20
-40
-20
-40
20
8
Cohesion: Data 2, Expr. II
Separability: Data 2, Expr.
Recall I(IP): Data Expr.
1.0
1000
Separbbily
0.8
-200
8T
Cohesion: Data 3, Expr. II
Separability: Data 3, Expr. II
Recall I(IP): Data 3, Expr.
200
600
Separabiley
0.7
4400
Recallg1
0.6
400
-600
0.5
200
AT
8
8
Cohesion: Data 2, Expr. III Separability: Data 2, Expr. III
Recall @ I(IP): Data 2, Expr. III
2500
-500
0.7
2000
Separabiing
11000
1500
1500
0.6
1000
-000
-2500
-60
-40
-20
-50
-50
-20
-50
-40
-20
8"
8TM
Cohesion: Data 3, Expr. III
Separability: Data 3, Expr. III
Recall@ I(IP): Data 3, Expr. III
Figure 27: Expr. II and Expr. III: rotation(H) (R) variation (linear scale). is from initialization. In Expr. II, the two unseen classes are assigned to different classes, leading to increased Cohesion and Separability as the train- unseen similarity increases, consistent with our analysis. In contrast, in Expr. III, the two unseen classes assigned to the same class, resulting in increased Cohesion and decreased Separability as the aain-unseen similarity increases, which is in line with our analysis. Recall@) @ / follows in similar trend to Separability.
after one step training.
65

---

0.98
0.96
0.96
0.94
0.92
0.92
0.90
0.90
8
Data 1, Expr. II
Data 2, Expr. II
Data 3, Expr. II
0.8
0.8
Recollg91
0.7
0.7
0.6
0.6
0.5
-50
-40
-20
-40
-20
-50
-40
B/u
Data Expr. IIII
Data 2, Expr. III
Data 3, Expr. III
Figure 28: @ with cosine similarity of Expr. II and Expr. III; rotation( R) variation (linear scale). after step training. is from initialization. Recall@) follows similar trend to corresponding Separability.
66

---

Data 1, Step 2
Data 2, Step 2
Data 3, Step
Contecnn
8T
Data 1, Stepp 3
Data 2, Step 3
Data 3, Step
Cohertio
Cobeston
18
Data Step 4
Data 2, Step
Data 3, Step
45"
1.50
Cones.n
-17"
+120
-050
+10
-290
-290
-10
6'
Data 1, Step 5
Data 2, Step 5
Data 3, Step
Figure 29: Evolution Cohesion two-layer network over multiple steps: as training progresses, Cohesion gradually strengthens.
67

---

18
by
Data Step 2
Data 2, Step
Data 3, Step
Sestnbi
Sp
Data 1, Stepp 3
Data 2, Step 3
Data 3, Step 3
121
by
08
0:1
-157
-10
-10
Data 1, Step 4
Data 2, Step
Data 3, Step
18
typarasity
-17
-19
-12"
-15
Data 1, Step 5
Data 2, Step 5
Data 3. Step
Figure 30: Evolution of Separability in two-layer network over multiple steps: as training progresses, Separability gradually strengthens. Exceptionally, when network is trained for many steps, there are cases where Separability decreases if the train-unseen similarity is very low.
68

---

Calvetr
130
Data 1, Layer
Data 2, 2 Layer
Data 3, 2 Layer
Coneeinn
Cahada
Data 1, Layer
Data 2, 3 Laaer
Data 3, 3 Layer
121
-50
Cohitton
Cohssto
-10
-10
Data 1, 4.4 Layer
Data 4 Layer
Data 3, 4 Layer
17
Data 1, Layer
Data 2, 5 Layer
Data 3, Layer
Figure 31: Evolution of Cohesion in multi-layer network over single steps: as training progresses, Cohesion gradually strengthens.
69

---

12
180
Saparst 1o
-27
Data 1, Layer
Data Layer
Data 3, 2 Layer
12
Spperssity
-15
-10
-10
-10
6T
sTu
Data 1, 3.3. Layer
Data 2, 3 Layer
Data 3, 3 Layer
12
18'
by
-12
-10
-10
-12
-20
6T
Data 1, Layer
Data 2, 4 Layer
Data 3, 4 Layer
12
127
6'u
Data 1, Layer
Data 2, 5 Layer
Data 3, Layer
Figure 32: Evolution of Separability in a multi-laye network over single steps: as training progresses, Separability gradually strengthens. Exceptionally. for deeper network, there are cases where Separability decreases if the train-unseen similarity is very high.
70

---

N
Additional Observations of Real-world dataset Experiments
N.1 Additional Results
presents experimental results using pretrained models with Domain datasets (CAR, CUB, SOP, ISC). Tigure[34] (4uure34showw shows results using the Domain+sub Inlk dataset across both initialized and pretrained models. In both cases, consistent with the results in Figure of the main text, models trained on specific domain exhibit superior performance on test data from the same domain.
Additional Results for Expr. IV is in Figure 33 and Figure 34 Figure 33
Additional Results for Expr. V is in Figure 35) Figure 35 includes experimental results for initialized 18 pretrained ResNet18, and pretrained ResNet50, which are not discussed in the main text. Across all settings, Domain+ D subsampled whole In/k does not outperform Domain or Domain+sub In/k, despite having access larger number of classes (as shown in Figure 9b.
Additional Results for Expr. VI is in F6gure We include high-resolution version of Figure 7b and additional results using pretrained models. In most cases, increasing the number of classes leads to improved performance. We note that Table 10 shows the number of classes per step for each dataset.
Additional Results for Expr. VII is in Appendix
N.2
Table 7 and Table g
Table 3 Table 4
Table
contain the original data prior to plotting.
N.1.1
Relationship between train-unseen similarity and semantic similarity
In real world validation of our feature transfer theory, surrogate the theoretical quantity (train- unseen similarity) using a semantic similarity of the dataset domain. This analogy is justified, as datapoints in each dataset (CUB, CAR, SOP, ISC) exhibit consistent visual characteristics within the dataset. Nevertheless, to strengthen our claim, we conducted the following additional experiment.
For each dataset (CUB, CAR, SOP, ISC) with three seeds, we trained model and extracted the class-wise mean embeddings from the training set as a surrogate for 3. Similarly, we computed class-wise mean embeddings from the test sets computed the maximum cosine similarity with any of the train class means as Listing !
surrogate for μ. Then, for each test class,
As shown in Figures 37
38 39 and 40 for estimated kernel density, and Table 2 for statistics, the
maximum cosine similarity between unseen and train class means tends to be highest when the train and test domains match. This supports our interpretation that semantic similarity across domains serves valid proxy for theoretical train- uninen alignment.
N.1.2 Relationship between clustering criteria and R
To further validate the use of Recall@ / as a surrogate for clustering criteria, we evaluated Separability as defined in Definition Definition 4.3 for Expr. IV. Following standard practice (Zhai and Wu, 2019). all models were trained with norm-normalized features, and thus we report only Separability. Please refer Table [5] The results indicate that models achieve the best Separability of test data when trained on the corresponding domain (e.g., CAR test dataset trained on CAR training dataset), consistent with the trends observed in Recall @ 1, thereby supporting the validity of Recall @ I as a surrogate.
N.2 Expr. VII: Removing Duplicately Assigned Eval Classes
In Expr. VII, suggested by the theoretical results on Separability, we validated whether eliminating duplicate in the assignments improves performance. To clarify, we will provide an example of duplicate assignment at Note N.I
Note N.1 (Example of duplicate assignment). For two train classes o (train) (train) and two test ,02
(teat) and o2 (test) are classified as o o (train) both test classes are assigned to #1 (train) resulting in duplication. Conversely, if o (teet)
classes of .22
(teat) (teat) if most instances of o
(teeet) classified as -a2
(train)
(train) they are assigned without duplication. and 02 (lest) as o
To validate, we introduce treatment and control groups. For the treatment group, we eliminate duplicate in the assignments for the train classes, i..., for each unseen class, the most frequently
71

---

classified training class is aggregated, and the classes are randomly removed to ensure that the selected training classes become unique (refer Algorithm Algorithm 2). For the control group, we randomly selected the same number of classes as the treatment group (refer to Algorithm Algorithm D These two groups are evaluated using Recall@) 1. This process is repeated five times, and the average reported. For the dataset, we used the following categories corresponding to each domain: Domain, sub Inlk, Domain+sub Inlk, and subsampled whole Inlk for Vchicle, Bird, Product, and Clothing. In the case of the subsampled whole Inlk dataset for this Experiment, we performed sampling by selecting 100 instances per class. The experimental results are presented in Table 7 and B A total of conducted, of which 51 demonstrated performance improvements: the estimated
64 experiments success rate is 79%. There is a 1.73% 2.87 average percentage point improvement in Recall@ with maximum improvement of 13.65 percentage point and minimum decrease of -3.28 percentage point. These results suggest that the duplicate reduction treatment group outperforms the randomly removed group with binomial test p-value of 9.40 10-7
Listing 1: Observing unseen vs. semantic similarity
same procedure for mean www.ddings.ettttteeeeee
for test_datases in [cub, car, sop, iscl:
embeddings, labels extract_ (model, test beat_dataset)
mean_ bai____eedddig__ttt testt = torch zeros (num_ wattauuuuseeenassseeeeeeeeeeeeee classes, dim)
for gear_enbediings_tsttttee
range (num (twp:uuuueencllasssss...
mubeddings(labell)
(dim-0)) normalize ()
# get statistics, sim: (num_unseen, sim mean_ eean_debedd
sue_seen)
@ mean mean_ bega__eeeediig_ttin train.
10
max max_ values sim. max (dim=1) # (num_unseen)
11
median (max, (max_ values), mean (max_ valxess std (max_ values)
Algorithm Random Sampling
Input:
Number if unseen classes u, number classes | Output: Sampled class set Saandom
Set Standom + random cattooo.ssmppe(( ((0, 1,...,,2. 13,
return Syandom
Algorithm 2 Duplicated assignment reduction sampling
Input: Model f, unseen data loader D. number of train classes Cuains number of unseen classes
Cunseen
Output: Sampled class set Snondup
Initialize counter matrix counter Creseen Cuxi for (img, label) in Ø do
pred +- f(mmg)
Predicted class indices
Update counter: counter[label, pred] += end for
sop1_index$55 + argson(counter, dim = 1, descending = True)].....0 uniqqe(top1_inder) Initialize Spondup + ()
unique_latie
for each label l unique_lase _abell do
1 - fi topp_indess (] f
Indices corresponding to label E
sample <- randon raando.ssapple( Te, 1)
Select one random index
Snandup < Smondup U {isaplle
end for
refurn Snondup
72

---

Test
SOP
ISC
CAR
38.64
CAR
CUB
44.47
60.41
49.24
26.32
CUB
43.88
64.06
48.45
25.21
Train
Train
SOP
40.37
19.64
49.02
31.73
SOP
49.89
28.67
63.84
36.63
ISC
22.36
11.14
43.69
71.69
ISC
24.55
14.80
54.54
86.72
R (pre)
ResNet50(pre)
Figure 33: Expr. IV pretrained models with Domain datasets (CAR, CUB, SOP, ISC)
Test
Test
CAR
CUB
SOP
CAR
CUB
SOP
ISC
CAR+I(V)
CAR+I(V)
31.66
CUB+I(B)
14.19
13.94
30.39
19.53
CUB+I(B)
7.41
6.63
20.43
10.87
Train
Train
SOP+1(P)
17.31
9.48
36.83
32.15
SOP+1(P)
17.41
9.78
38.29
30.63
ISC+I(C)
14.16
7.76
31.16
47.76
ISC+I(C)
12.26
7.31
30.40
49.92
ResNet18 (init)
ResNet50 (init)
Test
Test
CAR
CUB
SOP
15C
CAR
CUB
50P
ISC
CAR+I(V) 89.13
18.38
40.71
22.46
CAR+I(V) 92.50
22.99 43.90
22.54
CUB+I(B)
CUB+I((B
Train
Train
SOP+1(P)
40.63
19.63
48
31.57
SOP+1(P)
51.03
29.63
63.92
36.23
ISC+I(C) 22.50
11.33
42.68
70.47
ISC+I((C) 25.94 14.46
52.18
87.35
ResNet18 (pre)
ResNet50 (pre)
Figure 34: Expr. IV on ResNet18, ResNet50 with Domain + In(S) egg. CAR+I(V), CUB+I(B), SOP+I(P), ISC+I(C)
73

---

60
1
40
20
0
CAR
CUB
SOP
ISC
(a) ResNet50 (init)
75
Recall
50
25
0
CAR
CUB
SOP
ISC
(b) ResNet18 (pre)
75
Recall
50
25
0
CAR CUB
SOP
ISC
ResNe500 (pre)
Figure 35: Expr. V, additional results, it is represented as follows
Domain Domain + Related Subset of Inlk Domain Whole Inlk subsampled, Adding unrelated classes for training does not significantly affect the performance.
74

---

Init
50
40
R@1
30
20
0
Pretrained
90
80
R@1
70
50
Figure 36: Expr VI, it is represented as follows: ResNet18 --, ResNet50 Dataset cub, sop, isc, As the steps increased and related classes were added, performance generally improved consistently.
75

---

Baratsse
CAR Trained, Seed O
CAR Trained, Seed
CAR Trained, Seed 2
Cacra wonnee
CUB Trained, Seed 0
CUB Trained, Seed
CUB Trained, Seed 2
SOP Trained, Seed
SOP Trained, Seed 1
SOP Trained, Seed
MCU
ISC Trained, Seed 0 ISC Trained, Seed
ISC Trained, Seed 2
Figure 37: ResNet18 88 (Initialized): In most cases, the maximum cosine similarity is higher when the test dataset belongs to the same domain as the training dataset.
76

---

Berutise
CAR Trained, Seed O
CAR Trained, Seed
CAR Trained, Seed 2
alg
CUB Trained, Seed 0
CUB Trained, Seed
CUB Trained, Seed 2
SOP Trained, Seed
SOP Trained, Seed 1
SOP Trained, Seed
ISC Trained, Seed 0 ISC Trained, Seed
ISC Trained, Seed 2
Figure 38: ResNet50 (Initialized): In most cases, the maximum cosine similarity is higher when the test dataset belongs to the same domain as the training dataset.
77

---

LAE
30
Guare
Uinder clond
CAR Trained, Seed O
CAR Trained, Seed
CAR Trained, Seed 2
Max Caae
anl anseriann
CUB Trained, Seed 0
CUB Trained, Seed
CUB Trained, Seed 2
Code
SOP Trained, Seed
SOP Trained, Seed 1
SOP Trained, Seed
Mar To LEaAEe
ISC Trained, Seed 0
ISC Trained, Seed
ISC Trained, Seed
Figure 39: ResNet18 (Pretrained): In most cases, the maximum cosine similarity is higher when the test dataset belongs to the same domain as the training dataset.
78

---

CAR Trained, Seed 0
CAR Trained, Seed
CAR Trained, Seed 2
15
Max
CUB Trained, Seed 0
CUB Trained, Seed
CUB Trained, Seed 2
SOP Trained, Seed
SOP Trained, Seed 1
SOP Trained, Seed
ISC Trained, Seed 0 ISC Trained, Seed
ISC Trained, Seed 2
Figure 40: ResNet50 (Pretrained): In most cases, the maximum cosine similarity is higher when the test dataset belongs to the same domain as the training dataset.
79

---

Table Relationship between train unsinn similarity and semantic similarity, with the maximum value highlighted in each row. In model trained on single domain, the Train-imseen similarity is mostly highest for the test data that most semantically similar.
ResNet18 (Randomly Initialized)
Train,Tess Mest
CAR
CUB
SOP
ISC
median
mtd
median
mean
median
mean
median
meantsstd
CAR
0.80
0.78
0.58
0.58:00.09
0.53
0.53
0.59
0.59:10.06
CUB
0.68
0.68=10.07
0.85
0.8411007
0.61
0.60#10.13
0.43
0.44=10.88
SOP
0.64
0.64
0.67
0.67
0.68
0.6910.10
0.70
0.70=10.08
ISC
0.57
0.57+10.04
0.58
0.59:10.07
0.57
0.58
0.69
0.69:10.09
ResNet50 (Randomly Initialized)
Trast.Messt
CAR
CUB
SOP
ISC
median
meantstt
median
meanistd
median
mcan
median
mean
CAR
0.83
0.82:10.08
0.62
0.61#0.088
0.54
0.54+t0.11
0.55
0.54++0.07
CUB
0.85
0.83+#0.08
0.95
0.94::0.05
0.76
0.73:10.15
0.52
0.52=#0.07
SOP
0.59
0.59:10.07
0.62
0.61+0.00
0.62
0.63::0.11
0.70
0.69:10.08
ISC
0.57
0.57:10.03
0.67
0.67
0.59
0.60
0.72
0.72
ResNet18 (ImageNet IK Pretrained)
Train
CAR
CUB
SOP
ISC
median
mean-istd
median
mean
median
mean
median
mean
CAR
0.51
0.13
0.33
0.33
0.35
0.35::0.06
0.32
0.32
CUB
0.23
0.23
0.67
0.67:10.12
0.30
0.31
0.34
0.35
SOP
0.31
0.3210000
0.39
0.39110.05
0.36
0.3910.111
0.31
0.31
ISC
0.38
0.39++0.4
0.40
0.41+00.06
0.38
0.39++0.07
0.53
0.54
ResNet50 (ImageNet IK Pretrained)
Train,Vesst +
CAR
CUB
SOP
ISC
median
mstd
median
mean
median
mean
median
mean
CAR
0.48
0.11
0.29
0.29::0.04
0.33
0.34
0.27
0.28
CUB
0.28
0.28:10.04
0.68
0.69:10.12
0.35
0.35:10.07
0.39
0.39:10.06
SOP
0.30
0.31
0.35
0.36:10.06
0.39
0.42:10.14
0.35
0.36:10.08
ISC
0.29
0.29:10.04
0.35
0.36:10.05
0.32
0.34
0.53
0.54
80

---

Table 3: Table results for Expr. IV. In all cases, training with the corresponding domain data resulted in the best test performance.
ResNet18 (Randomly Initialized)
Train|Mestt
CAR
CUB
SOP
ISC
CAR
26.9709
8.9242=1.85
27.6459=0.18
19.7552-t0.17
CAR+I(V)
44.865315.00
11.1017=2.28
31.3428=0.09
20.9156=1.667
CUB
14.1270=0.17
12.5366-1144 1.45
29.9312=1.62
18.6831+1.18
CUB+I(B)
14.1885=0.62
13.9433+1.76
30.3902-+0.68
19.5329
SOP
1..5050=1.49
9.8132-#0.33
36.6247+1.22
30.2448+2.27
SOP+I(P)
17.3082+1.02
9.4812+1.73
36.8302+11.28
12.1444+1.799
ISC
14.1680=1.44
8.3446=11.89
31.3990-12.66
47.881713.009
ISC+I(C)
14.1557+1.20
1.75
31.1632++1.0
33.1550
ResNer50 (Randomly Initialized)
TTest
CAR
CUB
SOP
ISC
CAR
20.9609 14.26
6.7466:11.63
25.9485
14.8615-3.33
CAR+I(V)
31.6566-11.05
9.1201=1.11
30.9257=1.21
19.1651=1.04
CUB
1.3341=1.14
6.5946=2.31
19.2065=0.64
10.0050=0.93
CUB+I(B)
0.
6.6340=1.21
20.4285=2.26
10.8672=2.83
SOP
18.1732++.00
2.00
37.5448
29.7677
SOP+I(P)
17.4107+00.99
9.773811.38
38.2880
30.6237+1.23
ISC
11.7739+t1.36
6.7578=1.50
28.6343
47.3624+1.34
ISC+I(C)
1..2576+11.00
7.3093=1.61
30.3984=
49.9217=3.57
Table 4: Table results for Expr. IV. In the most cases, training with the corresponding domain data resulted the best test performance.
ResNet18 (ImageNet 1K Pretrained)
Trast
CAR
CUB
SOP
ISC
CAR 89.1280-2.62
19.542112.33
15.018014.31
38.6362-10.66
21.0411=1.55
CAR+I(V)
18.3772:66.2
40.7116:10.70
22.4612+1..4
CUB
44.4636=8.42
60.4096=5.84
49.2369-1.46
26.3188+1.09
CUB+I(B)
43.9511=8.04
60.122746.29
48.208910.66
24.4776=2255
SOP
40.3640-2.06
19.643344.03
49.020440.65
31.7294-0.78
SOP+I(P)
40.6346-0.82
19.626443.60
48.841411.09
31.5741
ISC
22.3589+0.99
11.1467+2.37
43.6862+1.69
71,6921 + 0.49
ISC+I(C)
22.5065+2.99
11.3324+1.42
42.681243.22
10.4721+0.75
ResNet50 (ImageNet 1K Pretrained)
Train,Vesst
CAR
CUB
SOP
ISC
CAR
92.7069 1.68
19,8458 8.36
41.9793-2.20
22.4363 4.995
CAR+I(V)
92.5019=1.46
22.9856-8.99
43.8988-11.74
22.5407+4.11
CUB
33.8814111.44
64.0558
48.4447-0.66
25.214312287
CUB+I(B)
40.0853+11..2 10.21
63.22874:5.81
45.648110..5
21.641242.57
SOP
49.895542..1
28.6743=5.80
63.8398+0.99
36.629410.78
SOP+I(P)
51.026943.89
29.6309-6.06
63.9213:0.23
36.2244-0.76
ISC
24.54
14.8042+1.82
54.5414=0.71
86.716440.63
ISC+ICC)
25.9419
14.4609
52.1762+1.38
87.3562 0.35
81

---

Table 5: Table of Expr. IV Results with Separability Measurements, In all cases, training with the corresponding domain data resulted the best Separability.
ResNet18 (Randomly Initialized)
Train/TTest
CAR
CUB
SOP
ISC
CAR -0.75:10.12
0.12
-0.63:10.12
-0.29:10.10
-0.82.10.07
CUB
0.11
-0.1441..11
-0.8110.066
SOP
-0.69:10.13 -0.42=#0.13
-0.01=
-0.47=10009
ISC
-0.86=0.08
-0.79=10.08
-0.28+1010
-0.01=0.01
ResNet50 (Randomly Initialized)
Train/VTest +
CAR
CUB
SOP
ISC
CAR
0.12
-0.72:10.13
-0.29:10.10
-0.8110.06
CUB
-0.71+0.15
-0.36=0..27
-0.10++0.11
+0.88++0066
SOP
-0.67++011
-0.37++0.1
-0.01+00011
-0.46++0.09
ISC
-0.90++0.05
-0.83++0.07
-0.30+0.10
-0.01=0.01
ResNet18 (ImageNet 1K Pretrained)
TTain/XTest +
CAR
CUB
SOP
ISC
CAR
10.09-0.00
-0.8510004
-0.36110.06
-0.7610.05
CUB
-0.8410.04
-0.2110.13
-0.37:10.05
-0.67::0.06
SOP
-0.64-10.08
-0.47110088
-0.01
-0.28:10.06
ISC
-0.82+10.09
-0.75+0.07
-0.22+10.08
-0.01=0.01
ResNet50 (ImageNet IK Pretrained)
Trait
CAR
CUB
SOP
ISC
CAR
-0.10=0.05
-0.83=0.04
-0.34=0.06
-0.68=0.05
CUB
-0.85+0.04
-0.28-0.12
-0.39110006
-0.733:0.05
SOP
-0.71=1009 -0.54=:0.08
-0.0110.00
-0.40:10.08
ISC
-0.86++0.08
-0.77++0.05
-0.25-+0.07
-0.01==0.00
Table 6: Table results of performance for Expr. V. The performance of D and D+I(Sub) is comparable that of D+I, despite D+I having a larger number of classes.
ResNet18 (Randomly Initialized)
Train Train Mest
CAR
CUB
SOP
ISC
D
26.97+2.89
12.54=1.45
36.62+t1.22
47.88-13.09
D+I(Sub)
44.87
13.94=1.76
36.83:11.28
47.76:3.113
D+I
31.61
14.11 0.10
36.62.10.88
44.82-00.55
ResNet50 (Randomly Initialized)
Trais/
+
CAR
CUB
SOP
ISC
D
6.59:12.31 20.96+4.26
37.54-10.33 38.2910.59
1..3
D+I(Sub)
31.66=11005
6.63
49.9243.57
D+1
32.55
9.70.10.26
37.32-10.69
55.92.111.32
ResNet18 (ImageNet 1K Pretrained)
Train Vest
+
CAR
CUB
SOP
ISC
D
89.54 2.33
60.41 5.84
49.02.50.65
7..69.10.8
D+I(Sub)
89.1342.62
60.12
48.84+1.09
70.4740.75
D+I
76.0210.42
53.69-0.52
47.35.10.93
68.3041.17
ResNet50 (ImageNet 1K Pretrained)
Train|VTest
CAR
CUB
SOP
ISC
D
92.71+1.66
6..06
63.84=0.97
86.72.10.63
D+I(Sub)
92.50
63.23-1.581
63.92 0.23
87.36.10.35
D+1
75.19
40.58
63.68
79.4749.09
82

---

Table 7: Expr. VII from (Randomly Initialized). Column Original indicates the Recall @ performance of the original trained model when neither treatment nor random removal is applied. Column A indicates the difference between Treatement and Random Recall score.
ResNet18 (Randomly Initialized)
Test
Train
Treatment
Random
Original
CAR
42.57 + 1.59
40.42 + 2.23
2.14 = 2.74
23.83
I(V)
32.96 = 1.23
27.93 + 3,98
5.02 = 4.17
11.17
CAR Test
CAR+I(V)
56.16 = 1.60
56.67 + 1,21
-0.51 2.00
39.22
In
41.43 + 0.88
38.65 + 2.42
2.77 = 2.57
25.70
CUB
21.46 24.56 1.22
3.10 = 2.85
10.89
I(B)
2 2.07
18.68 +
2.88 = 2.46
6,40
CUB Test CUB+I(B)
= -
18.93
1.04 2.48
12.05
In
37.66 + 0.87
+
4.74 + 1.76
21.49
SOP
41.63 + 0.22
0.21
0.15 + 0.30
48.87
(PP)
50.70 + 2.50
49.30
1.40 + 3.37
18.23
SOP Test
SOP+I(P)
4
44.57
0.02 0.50
48.70
In
52.24
5.771
-0.48 0.80
24.75
ISC
63.51 0.47
62.37
1.14 = 0.67
37.90
IC)
61.43 3.65
+ 1.84
8.89 = 4.09
31.29
ISC Test ISC+I(C)
56.81 = 0.59
56.10 0.38
0.71 = 0.70
37.20
In
45,25 + 1.59
45.43 + 1.56
-0.17 + 2.23
38.82
Average Improvement
2.05
Success Rate
0.8125
ResNet50 (Randomly Initialized)
Test
Train
Treatment
Random
A
Original
CAR
33.62 + 1.89
32.66 = 1.0T
0.97 = 2.15
20.67
I(V)
24.07 +: 1.29
1.30 + 0.97
2.77 + 1.61
10.48
CAR Test
CAR+I(V)
48.11 + 1.36
46.02 + 1.96
2.09 + 2.39
32.80
In
53.52 + 0.98
46.75 2.41
6.77 2.60
30.06
CUB
25.10 = 1.90
20.28 2.52
4.82
3.93
I(B)
26.96 + 1.40
22.31
4.64 2.61
3.58
CUB Test
CUB+I(B)
17.43 1.22
1
2.00
5.27
In
47.89 + 1.15
1.228
8.42 +
28.06
SOP
45.53 + 0.42
4.55
-0.17
45.81
I(P)
15.02
4..15
-0.53 #3.10
14.46
SOP Test
SOP+I(P)
44.33 0.36
4.550
0.61
53.18
59,42 - 0.85
0.76
1.31 1111111 1.14
22.85
ISC
64.90 0.46
0.59
1.98 - 0.73
37.50
IC)
55.21 + 1.52
50.42 4.59
4.78 4.84
27.16
ISC Test
ISC+I(C)
65.22 0.19
64.72
0.50 = 0.69
38.12
In
44.00 = 0.91
43.56 1.12
-0.44 1.44
42.93
Average Improvement
2.56
Success Rate
0.8125
83

---

Table 8: Expr. VII (ImageNet IK Pretrained). Column Original indicates the Recall performance of the original trained model when neither treatment nor random removal is applied. Column A indicates the difference between Treatement and Random Recall score.
ResNet18 8 (ImageNet 1K Pretrained)
Test
Train
Treatment
Random
Original
CAR
90.30 = 0.72
90.10 0.55
0.20 = 0.90
86,80
I(V)
64.40 = 3.23
65.51
-1.12 + 4.37
42.10
CAR Test
CAR+I(V)
91.37 + 0.52
90.38
1.00 0.82
86.10
In
74.96 + 5.14
66.56
8.40 = 6.52
26.00
CUB
62.86 + 1.53
2.56
-0.50 2.98
53.66
I(B)
6
54.89
6.32 = 2.74
34.00
CUB Test CUB+I(B)
71.03 = 0.86
67.02 1.08
4.01 = 1.38
52.89
In
46.29 +: 1.05
40.73
5.57 + 3.63
30.32
SOP
51.65 + 0.17
50.88 0.43
0.77 + 0.46
71.15
(PP)
69.91 + 0.79
+
1.74 + 1.22
24.57
SOP Test
SOP+I(P)
52.58 = 0.18
0.22
0.89 0.34
70.98
In
4 0.63
4.92
1,51 = 1.12
13.85
ISC
76.63 = 0.60
74.59
2.04 = 0.86
48.27
IC)
71.72 + 5.58
71.28 4.224
0.45 = 7.01
48.38
ISC Test ISC+I(C)
78.21 + 0.27
77.53 0.49
0.69 = 0.56
47.75
In
30.21 + 1.13
36.25 + 4.05
-6.04 4.20
30.66
Average Improvement
1.62
Success Rate
0.8125
ResNet50 (ImageNet 1K Pretrained)
Test
Train
Treatment
Random
A
Original
CAR
95.32 0.28
95.58 + 0.70
-0.26 + 0.76
90.77
I(V)
75.37 3.98
72.97 + 3.39
2.39 - 5.23
40.13
CAR Test
CAR+I(V)
94.85 + 0.93
94.82 + 0.59
0.03
90.81
In
40.45
68.57 5.35
11.88 5.80
32.51
CUB
76.05 1.96
3.96 2.44
57.78
I(B
6.753
58.16
2.37
33.37
CUB Test
CUB+I(B)
0.55
128331000
4.70
56.56
69.53
56.82
12.11 4.08
35.53
SOP
70.10 0.20
0.23
0.51 = 0.31
87.10
IP)
68.04
6.375
2.29 = 1.61
24.13
SOP Test
SOP+I(P)
68.61 + 0.09
6.37
0.55 = 0.38
87.18
51.94
3.87 + 1.56
8.68
ISC
90.70
89.85
0.85 + 0.43
62.75
IC)
66.42 4.51
67.06 +
-0.64 5.89
64.02
ISC Test
ISC+I(C)
90.85 0.35
89.98 0.47
0.87 = 0.59
63.66
40.84 3.553
2.31 + 7.12
28.87
Average Improvement
3.02
Success Rate
0.875
84

---

Additional Information of ImageNet subset used in Experiments
Table 9; Configuration of Expr. V
Table 10: Configuration
Expr. VI
Vehicle
Bird
Product
Clothing
Step 0
Step
Step 2
Step 3
D
98
100
I1316
3985
Vehicle
25
50
75
98
Bird
25
50
75
100
D+I(Sub)
138
159
11568
4031
Product
2829
5658
8487
11316
D+I
1098
1100
12316
4985
Clothing 996
1992
2989
3985
This section provides detailed information on the number of classes in the training datasets used in Expr. V and VI (9 TO), along with a list of manually selected classes that constitute the ImageNet subsets related to Vehicle, Bird, Product, and Clothing.
0.1 I(V): The Vehicle classes chosen in ImageNet
Total 40 classes.
ambulance, cab, convertible, fire engine, forklift, freight car, garbage truck, go-kart, golfcart, half track, harvester, horse cart, jeep. jinrikisha, limousine, minibus, minivan, Model T, moped, motor scooter, mountain bike, moving van, oxcart, passenger car, pickup, police van, racer, recreational vehicle, school bus, snowmobile, snowplow, sports car, streetcar, tank, tow truck, tractor, trailer truck, tricycle, trolleybus, unicycle
O.2
I(B): The Bird classes chosen in ImageNet
Total 59 classes.
cock, hen, ostrich, brambling, goldfinch, house finch, junco, indigo bunting, robin, bulbul, jay, magpie, chickadee, water ouzel, bald eagle, vulture, great grey owl, black grouse, ptarmigan, ruffed grouse, prairie chicken, peacock, quail, partridge, African grey, macaw, sulphur-crested cockatoo, lorikeet, coucal, bee eater, hornbill, hummingbird, jacamar, toucan, drake, red-breasted merganser, goose, black swan, tusker, white stork, black stork, spoonbill, flamingo, little blue heron, American egret, bittern, crane, limpkin, European gallinule, American coot, bustard, ruddy turnstone, red-backed sandpiper, redshank, dowitcher, oystercatcher, pelican, king penguin, albatross
0.3
I(P): The Product classes chosen in ImageNet
Total 353 classes.
abacus, accordion, acoustic guitar, altar, analog clock, apiary, ashean, assault rifle, backpack, balance beam, balloon, ballpoint, Band Aid, banjo, barbell, barber chair, barometer, barrel, barrow, baseball, basketball, bassinet, bassoon, bathing cap, bath towel, bathtub, beach wagon, beacon, beaker, bearskin, beer bottle, beer glass, bell cote, bib, bicycle-but builte--uut----twoo. binder, binoculars, bobsled, bolo tie, bonnet, bookcase, bottlecap, bow tie, brass, breakwater, broom, bucket, buckle, bulletproof vest, caldron, candle, cannon, canoe, can opener, car mirror, carousel, carpenter's 's kit, carton, car wheel, cash machine, cassette, cassette player, CD player, cello, cellular telephone, chain, chain saw, chest, chiffonier, chime, china cabinet, cleaver, clog, cocktail shaker, coffee mug, coffeepot, coil, combination lock, computer keyboard, confectionery, corkscrew, cornet, cradle, crash helmet, crate, crib, Crock Pot, croquet ball, crutch, dam, desk, desktop computer, dial telephone, digital clock, digital watch, dining table. dishrag, dishwasher, disk brake, dogsled, doormat, drum, drumstick, dumbbell, Dutch oven, electric fan, electric guitar, electric locomotive, envelope, espresso maker, face powder, feather boa, file, fire screen, flagpole, flute, folding chair, football helmet, fountain pen, four-poster, French horn, frying pan, gasmask, gas pump, goblet, golf ball, gondola, gong, grand piano, grille, guillotine, hair slide, hair spray, hammer, hamper, hand blower, hand-held computer, handkerchief, hard disc, harmonica, harp, hatchet, holster, honeycomb, hook, horizontal bar, hourglass, iPod, iron, jack-o' -lanten, jigsaw puzzle, joystick, knot, ladle, lampshade, laptop, lawn mower, lens cap, letter opener, lighter, lipstick, lotion, loudspeaker, loupe, magnetic compass, mailbox, maraca, marimba, matchstick, maypole, measuring cup, medicine chest, microphone, microwave, milk can, mixing bowl, modem, monitor, mountain tent, mousetrap. muzzle, nail, neck brace, necklace, nipple, notebook.
85

---

oboe, ocarina, odometer, oil filter, organ, oscilloscope, oxygen mask, packet, paddle, paddlewheel, padlock, paintbrush, paper towel, parachute, parallel bars, park bench, parking meter, pay-phone, pedestal, pencil box, pencil sharpener, perfume, Petri dish, photocopier, pick, picket fence, piggy bank, pill bottle, pillow, ping-pong ball, plastic bag, plate rack, plow, plunger, Polaroid camera, pole, pool table, pop bottle, pot, potter's wheel, power drill, prayer rug. printer, prison, projectile, projector, puck, punching bag, purse, quill, quilt, racket, radiator, radio, radio telescope, rain barrel, reel, reflex camera, refrigerator, remote control, revolver, rifle, rocking chair, rotisserie, rubber eraser, rugby ball, rule, safe, safety pin, saltshaker, sax, scabbard, scale, scoreboard, screen, screw, screwdriver, seat belt, sewing machine, shield, shopping basket, shopping cart, shovel, shower cap, shower curtain, ski, sleeping bag, sliding door, slot, snorkel, soap dispenser, soccer ball, sock, solar dish, soup bowl, space bar, space heater, spatula, spider web, spindle, spotlight, steel drum, stethoscope, stole, stopwatch, stove, strainer, stretcher, studio couch, sunscreen, swab, switch, syringe, table lamp, tape player, teapot, teddy, television, tennis ball, theater curtain, thimble, thresher, throne, tile roof, toaster, tobacco shop, toilet seat, torch, totem pole, tray, tripod, trombone, tub, turnstile, typewriter keyboard, umbrella, vacuum, vase, vault, velvet, vending machine, violin, volleyball, waffle iron, wall clock, wallet, wardrobe, washbasin, washer, water bottle, water jug, water tower, whiskey jug, whistle, window screen, window shade, wine bottle, wing, wok, wooden spoon, comic book, crossword puzzle, street sign, traffic light, book jacket, menu, plate
0.4
I(C: The Clothing classes chosen in ImageNet
Total 46 classes.
abaya, academic gown, apron, bikini, brassiere, breastplate, cardigan, chain mail, Christmas stocking, cloak, cowboy boot, cowboy hat, cuirass, diaper, fur coat, gown, hoopskirt, jean, jersey, kimono, knee pad, lab coat, Loafer, mailbag, mask, military uniform, miniskirt, mitten, overskirt, pajama, poncho, running shoe, sandal, sarong, ski mask, sombrero, suit, sunglass, sunglasses, sweatshirt, swimming trunks, trench coat, vestment, wig, Windsor tie, wool
P Rotation Matrix Generation Process of Setup
To generate of rotation matrices with diverse magnitudes of rotation, we constructed an algorithm that samples & = 300 random matrices, each formed by adding i.i.d. Gaussian noise matrix of varying variance to the identity matrix T. The process ensures the generation of rotation matrices with varying extents of rotation, from slight to more substantial deviations from the identity matrix.
The rotation matrices generated as follows:
1. A matrix is initialized as + e M, where M is a i.i.d. standard random Gaussian matrix.
2. Using the QR decomposition, we orthogonalize this matrix to ensure it forms a valid rotation matrix.
3. Finally, if the determinant of the resulting matrix is negative, we flip the sign of the first column maintain determinant of + ensuring is valid rotation.
In summary, this method provides a collection of matrices that progressively deviate from 7, allowing us to observe and sample rotations of increasing magnitude. Please refer Algorithm Algorithm
86

---

Algorithm
Gaussian-Sampler Random Rotation Matrix Generation
Input: Number of dimensions ni, number matrices k
Output: Stack of random rotation matrices
Initialize empty list
Set +- 0.5 for i + 0 to do
if mod ( 0 and i +f 0 then
E < e x 0.22360679775
end if
Generate random matrix M: M ~ 7700, 1)mxn
Compute perturbed matrix: A +- I, +cx M
Compute QR decomposition: Q, R + QR(A) if del(() then
Flip first column of Q::QQQ. Q]: < - Q|: end if
Add Q to Q end for return Q
87


Reviews:
[{"review_id": "r3nytUBbko", "reviewer": "Reviewer_MzGZ", "summary": "This paper theoretically analyzes how features learned by a classifier perform in clustering unseen classes. The study uses a simple model where a two-layer nonlinear network is trained with a single gradient descent step. As a result, it is shown that the learned features can be decomposed into a component derived from random initialization and a \"spike component\" that depends on the training data. Based on the properties of this spike component, the paper concludes that clustering Cohesion and Separability are determined by the similarity between the training data and unseen data distributions. Specifically, higher similarity improves Cohesion, while Separability changes depending on the assignment of unseen classes. It was also shown that feature expressiveness diminishes for inputs that are irrelevant to the training data. These theories are validated through experiments on both synthetic data and real-world benchmark datasets.", "questions": "- Your work focuses on a single gradient descent step. How do you anticipate the behavior of the spike component sL would change if multiple training steps or epochs were considered? Specifically, how might your conclusions (especially regarding Cohesion and Separability) be extended or modified as more complex features are learned?\n- In Expr. IV, you interpret the theoretical \"train-unseen similarity\" as practical \"semantic similarity.\" Would it be possible to perform a more quantitative analysis to bridge this gap? For instance, one could try to approximate the spike direction β from a model trained on one domain (e.g., CAR) and compute its inner product with the class means μ from another domain (e.g., CUB).\n- Your analysis of Separability shows that the \"assignment\" of unseen classes to training classes is critical. This assignment can be interpreted as being based on the learned decision boundary. Do you have any insights into the stability of this assignment itself (e.g., its robustness to initial conditions or minor perturbations in the data)? If the assignment is unstable, it seems that predicting Separability would also become difficult.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The framework of decomposing learned features via a \"spike component\" and explaining clustering performance (Cohesion, Separability) with \"train-unseen similarity\" is novel and provides deep insights into the mechanics of feature transfer. It moves beyond the empirical rule-of-thumb that simply \"training on large datasets is better.\"\n- The paper carefully validates its theoretical claims in two stages: (1) rigorous validation on synthetic data that meets the theoretical assumptions, and (2) practical validation in real-world scenarios with deep models (ResNet). This dual approach demonstrates both the theory's validity and its practical implications, making the paper highly persuasive.\n\n**Major Weaknesses**\n- The analysis is confined to \"a two-layer network trained with a single gradient descent step.\" Modern deep learning models are much deeper and are trained through complex dynamics over multiple epochs. How the insights from this simplified setting generalize to deeper, more complex models requires further discussion. While the experiments with ResNet on real data show consistent trends, a gap remains between the theory and practice.\n- The core theoretical metric, \"train-unseen similarity\" (β⊤μ), is clearly defined. However, in experiments on real data (e.g., Expr. IV), it cannot be directly computed and is replaced by the intuitive concept of \"semantic similarity.\" The relationship between these two metrics, while plausible, is not quantitatively analyzed, leaving the applicability of the theory somewhat ambiguous.\n\n**Minor Weaknesses**\n- While it is standard for major proofs to be in the appendix, some key theoretical steps (e.g., the transition from Theorem 3.3 to the definition of the Dominant Feature) feel a bit rushed in the main text. Including more intuition or a brief sketch of these steps in the main body would improve clarity for the reader.\n- As noted in Appendix A, the technique of approximating the activation function with polynomials has its limits. Specifically, it can require high-dimensional spaces for computation, and in finite dimensions, a discrepancy can arise between the approximation and the actual network's behavior. A more in-depth discussion in the main text on the validity of this approximation and its potential impact on the results would further strengthen the paper's credibility.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and accurate summary of our work. In this rebuttal, we address the concerns you raised regarding the **weaknesses and questions**.\n\n## Response to Question 1 and Major Weaknesses 1\n\nWe share **intuition on multi-step training and deeper models**. In addition, we provide **supporting experiments** showing that our theoretical predictions largely hold in both multi-step and multi-layer settings.\n\n### For multi-step training\n\nWe expect that multi-step training refines the spike component by amplifying directions that **enhance Separability** while suppressing others, causing class features to collapse more tightly and become better separated. This process may lead to effective information compression for classification and improved transferability. To this end, one may build on Dandi et al. (2023), who analyze the \"staircase property\" in multi-step training.\n\n### For the deeper model\n\nWhile our current two-layer model already offers insight into how feature learning enhances clustering, building on this with deeper architectures could deepen our understanding.  Existing techniques (e.g., Fan & Wang [2020]; Nichani et al. [2023]; Wang et al. [2023]) have already generalized conjugate kernel analysis to multi-layer settings, suggesting that our approach may extend similarly. Such extensions could help further explain how generally transferable features emerge. Notably, our multi-layer experiments showed that when $\\beta^\\top\\mu$ becomes too large, especially in 3- and 4-layer models, Separability may not improve. **Studying this critical threshold could be an interesting direction**.\n\n## Response to Question 2 and Major Weaknesses 2\n\nWe appreciate your concern regarding the applicability of our theoretical framework to real-world networks. In real world validation of our feature transfer theory, we surrogate the theoretical quantity $\\beta^\\top \\mu$ (train-unseen similarity) using a semantic similarity of  the dataset domain. This analogy is justified, as datapoints in each dataset (CUB, CAR, SOP, ISC) exhibit consistent visual characteristics within the dataset. Nevertheless, to address your request and strengthen our claim, **we conducted the following additional experiment**.\n\n### Additional Experiment:\n\nAs you suggested, for each dataset (CUB, CAR, SOP, ISC) with three seeds, we trained a model and *extracted the class-wise mean embeddings from the training set as a surrogate for $\\beta$*. Similarly, we computed *class-wise mean embeddings from the test sets as a surrogate for $\\mu$. Then, for each test class*, we computed the maximum cosine similarity with any of the train class means, as follows:\n\n```python\n# same procedure for mean_embeddings_train\nfor test_dataset in [cub, car, sop, isc]:\n\tembeddings, labels = extract_feature(model, test_dataset)\n\tmean_embeddings_test = torch.zeros(num_unseen_classes, dim)\n\tfor uc in range(num_unseen_classes):\n\t\tmean_embeddings_test[uc] = embeddings[labels == uc].mean(dim=0).normalize()\n\t\t# get statistics\n\t\tsim = mean_embeddings_test @ mean_embeddings_train.T # (num_unseen, num_seen) \n\t\tmax_values = sim.max(dim=1) # (num_unseen) \n\t\tmedian(max_values), mean(max_values), std(max_values)\n```\n\nFindings:\n\nAs shown in the table, **the maximum cosine similarity between unseen and train class means is consistently highest when the train and test domains match** (e.g., 0.84 for CUB → CUB, 0.78 for CAR → CAR). This supports our interpretation that semantic similarity across domains serves as a valid proxy for theoretical train-unseen alignment. We provide the results of ResNet18 trained initialization in the table below. We will include analyses of all setups and visualization in the camera-ready version.\n\n| Train data | median | mean | std |\n| --- | --- | --- | --- |\n| CAR test |  |  |  |\n| CAR | **0.80** | **0.78** | 0.09 |\n| CUB | 0.68 | 0.68 | 0.07 |\n| SOP | 0.64 | 0.64 | 0.06 |\n| ISC | 0.57 | 0.57 | 0.04 |\n| CUB test |  |  |  |\n| CAR | 0.58 | 0.58 | 0.09 |\n| CUB | **0.85** | **0.84** | 0.07 |\n| SOP | 0.67 | 0.67 | 0.06 |\n| ISC | 0.58 | 0.59 | 0.07 |\n| SOP test |  |  |  |\n| CAR | 0.53 | 0.53 | 0.10 |\n| CUB | 0.61 | 0.60 | 0.13 |\n| SOP | **0.68** | **0.69** | 0.10 |\n| ISC | 0.57 | 0.58 | 0.08 |\n| ISC test |  |  |  |\n| CAR | 0.59 | 0.59 | 0.06 |\n| CUB | 0.43 | 0.44 | 0.08 |\n| SOP | **0.70** | **0.70** | 0.08 |\n| ISC | *0.69* | *0.69* | 0.09 |\n\n## Response to Question 3\n\nWe believe there may have been a misunderstanding regarding the \"assignment\" in Note 4.6. The assignment we refer to is a theoretical construct defined **without training**, computed via the sign $\\beta^\\top \\mu = \\frac{1}{n} (X^\\top y)^\\top \\mu$ in data space. It reflects a linear decision boundary **in the data space**, not one learned through neural network training. We will clarify this in the revised manuscript to avoid potential misinterpretations.\n\nHowever, your question raises a valid point about **how stable this assignment** is to perturbations in the training data itself, such as *reduced class separability*. In such cases, **the magnitude of $\\beta$ decreases, reducing Cohesion and Separability scores in Eqs. 6 and 7 due to the $||\\beta||$ term**. This aligns with the intuitive notion that *noisier training data leads to less transferable features*.\n\nTo illustrate, suppose the training data consists of two classes drawn from $c_1 \\sim \\mathcal{N}(\\mu, I)$ and $c_2 \\sim \\mathcal{N}(R\\mu, I)$, where $R$ is a rotation matrix. Then the spike direction converges (under large $n$) to $\\beta \\rightarrow \\frac{1}{2} (\\mu - R\\mu)$ by law of large numbers. \n\n$\\beta = \\frac{1}{n} X^\\top y = \\frac{1}{n} X_i y_i = \\frac{1}{2} (\\frac{n}{2} \\sum_{c_1} X_i - \\frac{n}{2} \\sum_{c_2} X_j) \\rightarrow \\frac{1}{2} (\\mu - R \\mu) .$\n\nIn the extreme case where $R = I$, the classes are indistinguishable and $\\beta = 0$, eliminating both Cohesion and Separability. When $R = -I$, $\\beta = \\mu$, which yields maximal separation.\n\n## Response to Minor Weaknesses\n\n1. Specific key theoretical steps, especially the transition from Theorem 3.3 to Definition 3.4 (Dominant Feature), would benefit from a more intuitive explanation in the main text. We will *include a summary and interpretation to aid the reader's understanding*.\n2. We acknowledge limitations of this approach in low-dimensional regimes. In the camera-ready version, we will discuss the need for alternative, more interpretable and efficient approximations better suited for these settings. This will clarify the scope of our approximation.\n\n## Multi step and Multi-layer experiments\n\nTo test the validity of our theory in multi-layer and multi-step settings, we applied Expr 1 and observed that **increasing $\\beta^\\top\\mu$ generally improves both Cohesion and Separability**, consistent with the prediction in Section 4.1.\n\nIn deeper networks, however, excessively large $\\beta^\\top\\mu=5 \\cdot 10^3$ leads to reduced Separability in 3- and 4-layer cases, as unseen features become aligned yet too far in magnitude. This suggests an interesting direction for future work: identifying a potential critical threshold of alignment strength in deep models that balances directional consistency and proximity. \n\nAs we are unable to include figures, we provide tables instead. We report Cohesion and Separability in separate tables, where each column corresponds to a layer or step setting, and each row shows values for increasing $\\beta^\\top\\mu$.\n\n### Multi-layer One-step, lr 0.1, and Expr 1 of Training Data 2 (Truncated Gaussian)\n\nCohesion\n\n| $\\beta^\\top\\mu$ |  | 2 layer | 3 layer | 4 layer | 5 layer |\n| --- | --- | --- | --- | --- | --- |\n| 0 | $F_0$ | 0.61 | 58.02 | 144.89 | 182.08 |\n|  | $F$ | **0.61** | 53.67 | 144.89 | **185.57** |\n| $10^2$ | $F_0$ | 1.13 | 54.77 | 137.48 | 183.82 |\n|  | $F$ | **32.86** | 66.33 | 137.48 | **187.34** |\n| $10^3$ | $F_0$ | 49.02 | 65.66 | 136.58 | 181.23 |\n|  | $F$ | **5229.26** | 4260.94 | 360.86 | **200.20** |\n| $5 \\cdot 10^3$ | $F_0$ | 1279.74 | 428.04 | 163.06 | 171.20 |\n|  | $F$ | **119560.01** | 107193.12 | 13343.40 | **1868.45** |\n\nSeparability\n\n| $\\beta^\\top\\mu$ |  | 2 layer | 3 layer | 4 layer | 5 layer |\n| --- | --- | --- | --- | --- | --- |\n| 0 | $F_0$ | -0.04 | -59.73 | -144.00 | -198.57 |\n|  | $F$ | **-0.04** | -59.73 | -144.00 | **-200.27** |\n| $10^2$ | $F_0$ | 0.62 | -64.89 | -149.52 | -208.04 |\n|  | $F$ | **30.99** | -41.70 | -148.67 | **-200.3** |\n| $10^3$ | $F_0$ | 56.95 | -56.56 | -141.32 | -189.67 |\n|  | $F$ | **1027.47** | 390.64 | 175.64 | **-105.12** |\n| $5 \\cdot 10^3$ | $F_0$ | 582.36 | 132.975 | -23.26 | -142.38 |\n|  | $F$ | **5955.02** | ***-10772.3*** | ***148.312*** | **305.38** |\n\n### Multi-step Two-layer, lr 0.001, and Expr 1 of Training Data 2 (Truncated Gaussian)\n\nCohesion\n\n| $\\beta^\\top\\mu$ |  | 1 step | 2 step | 3 step | 4 step | 5 step |\n| --- | --- | --- | --- | --- | --- | --- |\n| 0 | $F_0$ | 0.66 | 0.67 | 0.66 | 0.65 | 0.59 |\n|  | $F$ | 0.66 | 0.67 | 0.66 | 0.65 | 0.59 |\n| $10^2$ | $F_0$ | 1.12 | 1.20 | 1.16 | 1.20 | 1.14 |\n|  | $F$ | 1.12 | 1.20 | **1.26** | **2.41** | **44.59** |\n| $10^3$ | $F_0$ | 48.48 | 51.70 | 48.80 | 50.22 | 46.25 |\n|  | $F$ | 48.43 | 51.70 | **59.28** | **217.51** | **4974.99** |\n| $5 \\cdot 10^3$ | $F_0$ | 1324.11 | 1311.85 | 1275.23 | 1253.07 | 1249.63 |\n|  | $F$ | 1324.11 | 1311.85 | **1617.79** | **6592.32** | **119587.06** |\n\nSeparability\n\n| $\\beta^\\top\\mu$ |  | 1 step | 2 step | 3 step | 4 step | 5 step |\n| --- | --- | --- | --- | --- | --- | --- |\n| 0 | $F_0$ | -0.05 | -0.05 | -0.05 | -0.05 | -0.05 |\n|  | $F$ | -0.05 | -0.05 | -0.05 | -0.05 | -0.05 |\n| $10^2$ | $F_0$ | 0.45 | 0.50 | 0.47 | 0.44 | 0.43 |\n|  | $F$ | 0.45 | 0.50 | **0.57** | **1.71** | **51.32** |\n| $10^3$ | $F_0$ | 42.67 | 43.02 | 45.39 | 46.56 | 43.34 |\n|  | $F$ | 42.67 | 43.02 | **51.59** | **132.29** | **1183.37** |\n| $5 \\cdot 10^3$ | $F_0$ | 602.20 | 608.54 | 581.89 | 599.86 | 590.46 |\n|  | $F$ | 602.20 | 608.54 | **703.97** | **1277.33** | **6420.12** |"}, {"review_id": "OjeBUDIaMB", "reviewer": "Reviewer_yqtE", "summary": "This paper presents a theoretical and empirical analysis of how features learned by a two-layer classifier generalize to unseen classes in clustering tasks. The authors use a simplified setting — one gradient descent step and Hermite polynomial expansions — to decompose features into components and study how similarity between training and unseen data affects clustering quality. Experiments on both synthetic and real-world data support their theoretical findings.", "questions": "- Is it possible to perform a similar analysis of deeper/multi-step models?\n- Did you find any unexpected behavior that could led to novel approeaches on the topic?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Quality:**\n- (+) The theoretical analysis is detailed and mathematically sound.\n- (+) The experiments are well-designed to support the theoretical claims.\n- (-) A two-layer network trained with just one step is far from how real models are used.\n- (-) The main result (features transfer better when unseen inputs are similar to training data) is well-known.\n\n**Clarity:**\n- (-) The introduction section is very hard to read. I recommend a profesional proofreading to make it more accessible to other researchers.\n- (-) The English writing is often overly complex. It is very hard to follow. \n- (-) I suggest the authors to separate simpler explanations from hard demonstrations.\n\n**Significance:**\n- (+) Understanding when and how features generalize to unseen classes is an important question in the machine learning community.\n- (+) The topic is relevant for researchers working on transfer learning, clustering, and metric learning.\n- (-) The contribution is **mostly theoretical formalization** of ideas that are already intuitively known (e.g., aligned inputs transfer better).\n- (-) The impact on practice is limited, as the analysis is far from current Machine Learning architectures. It would be interesting to know is the assumptions correlate to bigger networks.\n\n**Originality:**\n- (+) The authors' approach seems creative and novel.\n- (-) It is not a negative per se, but it is dissappointing that the theory confirms known behavior rather than discovering surprising or counterintuitive results", "was_revised": true, "final_justification": "The authors addessed my concerns, wo I will raise my score towards borderline accept.", "rebuttal": "We appreciate the recognition of the theoretical soundness and experimental rigor of our work. Your comments encouraged us to clarify key contributions more precisely.\n\nNotably, we would like to respectfully highlight an aspect that may have been overlooked in the summary. Beyond clustering analysis, a key contribution of our work lies in extending the analysis of two-layer neural networks from regression to classification. To address this, we consider inputs drawn from a non-centered sub-Gaussian distribution, broadening applicability to classification tasks. **To our knowledge, this is the first such tool to enable tractable, closed-form analysis of feature learning in classification.** Moreover, this framework not only supports clustering results but more generally **characterizes how feature expressivity for unseen classes depends on the number of semantically related training classes** (see Section 4.2, Corollary 4.7).\n\n## On the Gap Between Theoretical Setup and Practice\n\nWe acknowledge the simplified nature of our model (two-layer, one-step gradient descent). \n\nHowever, we support our claims with **experiments in Section 5 on deep networks** (e.g., ResNet) and observe that *the theoretical predictions remain consistent in practice.* We believe this combination of theory and empirics provides a compelling case for the relevance of our simplified setting.\n\nOn top of that, we conducted additional experiments for the rebuttal by extending our original 2-layer, 1-step setup. Specifically, we performed experiments under the Expr 1 setting with (1) 2-layer networks trained for up to 5 steps, and (2) **_Bigger networks_** with  3, 4, and 5 layers. In both cases, we observed that as the *train-unseen similarity $\\beta^\\top \\mu$ increases, Cohesion and Separability consistently improve.* These results are included in the final part of *our rebuttal to Reviewer MzGZ under Multi-Step and Multi-Layer Experiments*.\n\nLastly, this setting is consistent with a large body of works (see Appendix D) and is justified for the following reasons:\n\n1. Our assumptions enable analytical tractability and provide clear insight into the mechanisms of feature learning and transfer. In particular, our central research question—*”Can we capture the presence of feature learning in classification and identify the conditions where features cluster effectively in new distributions?”—can be addressed using a single training step of a two-layer neural network*. \n2. One-step update captures *early-phase dynamics that are known to be strongly influential in determining final model behavior* (Early Phase of NN Optimization Section of Ba et al., 2022).\n3. *Shallow network* has been successfully used in prior work to explain **nontrivial phenomena observed in bigger networks**, such as the superiority of neural networks over traditional kernel methods (Ba et al. [2022]), the mechanisms of feature learning (Dandi et al. [2023], Moniri et al. [2024]), the mechanisms of learning the internal structure of data (Ba et al. [2023], Mousavi-Hosseini et al. [2023]), the representational power of random features (Louart et al. [2017], Goldt et al. [2020], Cui et al. [2024]), lottery ticket hypothesis([1]), generalization, interpolation and double descent ([2, 3, 4]), overparameterization ([5]), Catastrophic Forgetting([6])\n\n[1] Malach, Eran, et al. \"Proving the lottery ticket hypothesis: Pruning is all you need.\" *International Conference on Machine Learning*. PMLR, 2020.\n\n[2] Suzuki, Keita, and Taiji Suzuki. \"Optimal criterion for feature learning of two-layer linear neural network in high dimensional interpolation regime.\" *The Twelfth International Conference on Learning Representations*. 2023.\n\n[3] Abeykoon, Chathurika S. *The Double Descent Behavior in a Two Layer Neural Network for Binary Classification*. Diss. The University of Mississippi, 2023.\n\n[4] Xu, Ruichen, and Kexin Chen. \"Rethinking benign overfitting in two-layer neural networks.\" *arXiv preprint arXiv:2502.11893*(2025).\n\n[5] Zhang, Yaoyu, et al. \"Local Linear Recovery Guarantee of Deep Neural Networks at Overparameterization.\" *Journal of Machine Learning Research* 26.69 (2025): 1-30.\n\n[6] Li, Boqi, Youjun Wang, and Weiwei Liu. \"Towards Understanding Catastrophic Forgetting in Two-layer Convolutional Neural Networks.\" *Forty-second International Conference on Machine Learning*.\n\nWe hope that our findings, *alignment in real-world networks, the empirical validity of our theoretical predictions in multi-step and multi-layer settings, and strong connections to prior literature linking simplified assumptions to deep models* help convey the broader relevance of our work and encourage further interest in our research.\n\n## Response to Question 1\n\n### Deeper Networks\n\nWhile our current two-layer model already offers insight into how feature learning enhances clustering, building on this with deeper architectures could deepen our understanding.  Existing techniques (e.g., Fan & Wang [2020]; Nichani et al. [2023]; Wang et al. [2023]) have  analysis for multi-layer settings, suggesting that our approach may extend similarly. Such extensions could provide insights into how neural networks learn to classify by selectively amplifying or suppressing directions in input space. Ultimately, this line of inquiry helps further explain how **generally transferable features** **emerge** across layers in deep models.\n\n### Multi-step training\n\n*Dandi et al. (2023)* proposed an analytical framework for studying multi-step training dynamics in two-layer neural networks. They analyzed the so-called *staircase property* in such networks and demonstrated that features are learned progressively from the data structure, leading to a monotonic decrease in generalization error. Given the similarity between their setting and ours, *their methods could potentially be adapted to study the evolution of classifier features across multiple training steps*, beyond the one-step clustering analysis we focus on.\n\n## Novelty & Question 2\n\nWe understand your concern that the core result—features cluster better when unseen inputs resemble training data—may seem unsurprising. However, we believe our work goes significantly beyond rephrasing known intuitions:\n\n1. Novel Theoretical Tools for Classifier Analysis – Extending to Non-Centered Sub-Gaussian Distributions:\n    \n    Our analysis extends prior work (see Table 1 in the Appendix) by generalizing feature learning theory from centered Gaussian settings to non-centered sub-Gaussian data, thus enabling application to classification tasks beyond regression. This requires new technical tools (see line 93, Appendices I–J) that allow analysis under arbitrary labels and finitely supported distributions.\n    \n    In Appendix I, we extend classical results on standard Gaussian expectations of Hermite polynomial products (O'Donnell [2021], Moniri et al. [2024]) **to non-centered, non-unit-variance Gaussians**. We then **derive computable bounds of the above expectations for general sub-Gaussian inputs.** In Appendix J, we generalize key results from Vershynin [2010, 2018]—originally for centered sub-Gaussians—to *the non-centered case*.\n    \n    These contributions are essential for analyzing models trained on non-Gaussian separable data and offer broadly applicable tools for future theoretical work.\n    \n2. A Novel Framework for Analyzing Classifier - Deterministic Spike-Based Closed-Form Decomposition:\n    \n    We establish a **foundational framework for analyzing the feature behavior of classifiers on new inputs**, rather than focusing on the commonly studied regression task (Ba et al. [2022] and Table 1) or training data (See line 685). Notably, our predominant feature component β admits a closed-form, deterministic expression, unlike the stochastic feature decompositions used in recent works such as Demir and Dogan [2025] (see line 650). As a result, *we are able to directly analyze the output of the learned feature map* in Section 4.\n    \n3. **(Question 2)** Counterintuitive Finding – Importance of Relevant Classes, not Many Classes:\n    \n    While it is widely believed that increasing the number of training samples improves feature quality (Brown et al., 2020; An et al., 2023; line 337), our results in Corollary 4.7 and Experiments V–VI suggest a more nuanced view—namely, that **only semantically aligned classes contribute meaningfully to expressivity**. This challenges the conventional emphasis on quantity and opens a promising direction aligned with recent work in coreset selection, where reducing the number of training classes rather than just the number of samples can enhance feature transfer.\n    \n4. Unifying empirical observations and prior theories - Theoretical Quantification of Feature Transfer Phenomena:\n    \n    Our work offers a theoretical contribution by unifying empirical observations and prior theories, clarifying when feature transfer succeeds or fails (Section 4.1). While feature transfer is known to work better when unseen inputs resemble training data, previous metric learning studies mainly focus on performance gains without consistent explanations (Chopra et al. 2005, Liu et al. 2018, El-Nouby et al., 2021, Caron et al., 2021, Ermolov et al. 2022). In contrast, our framework precisely identifies both when feature transfer works (Propositions 4.4 and 4.5) and when it breaks down, such as when multiple unseen classes map to the same spike β, causing Separability to fail despite high similarity. **This quantitative characterization of transfer success and failure is a key strength of our work** (See 671).\n\n## Response to Clarity\n\nFinally, we appreciate your suggestion regarding the introduction and presentation. In the revised version, we will:\n\n1. Simplify the introduction and text by breaking down longer sentences and providing intuitive context before diving into formalism.\n2. Separate intuition and theory throughout the paper to improve accessibility."}, {"review_id": "PwLEaj8Oqb", "reviewer": "Reviewer_yNMA", "summary": "This paper theoretically investigates the feature transfer abilities of two-linear neural network classifiers. \n\nThe authors consider the setting of a two-layer neural network with specific assumptions on the activation function and initialization of parameters and uncentered sub-gaussian data distribution. In a 1-vs-1 K-class classification setting with MSE loss, one gradient descent (GD) step is asymptotically shown to be approximated by a specific rank-1 structure. This yields a feature decomposition result that essentially decomposes the feature mapping function realized by the network after one GD step into two components: random initialization features and a so-called Spike Component, which is effectively responsible for feature learning from the training data.\n\nGiven this, the authors derive closed-form expressions for the two clustering performance criteria for unseen data, Cohesion and Separability, w.r.t. the Spike Component (Propositions 4.4 and 4.5) and relate them to the so-called Train-Unseen Similarity.\nNamely, *the better the alignment between training and testing data, the more effective the clustering of test data in the feature space*.\n\nExperimental results on synthetic and image data support theory.", "questions": "1. Can the current results obtained for the non-conventional setting of 1-vs-1 K-class classification be transferred to the more standard K-way multi-class classification (i.e., when last layer is an $N \\times K$ matrix)?\n2. What if the classes are not perfectly balanced?\n3. Can the requirement of $d \\to \\infty$ be avoided in the analysis? It seems very unnatural.\n4. Line 111: if $W_0[i]$ denotes the $i$-th row of matrix $W_0 \\in \\mathbb{R}^{d \\times N}$, how can it be initialized from the $(d-1)$-dimensional sphere? Is that a typo?\n5. Can the sphere initialization be relaxed with the Gaussian initialization?\n6. Line 114: $P \\to K$.\n7. Does Shifted ReLU actually follow Assumption 2.1? It seems that it is not bounded by some $\\lambda_\\sigma$ a.s.\n8. How exactly are $H_k(z)$ defined?\n9. Doesn't Theorem 3.1 imply the degenerate limit of $F_0(x) \\to 0$ as $n \\to \\infty$? \n10. There is a discrepancy between the definition of $\\beta$ (without indices) and the way it is used in the text $\\beta_{ij}$ (with indices).\n11. Line 162: Definition 4.3.\n12. Line 165: What is $\\mathcal{n}(\\mu_i, I_d)$?\n13. Line 166: What is $\\rho_{k,k',r,r'}$?\n14. What are $C(s_L)$ and $S(s_L)$ used in Propositions 4.4, 4.5?\n15. Recall@1 definition: is the data sampled from the test or train distribution?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\nThis paper proposes a profound theoretical analysis of feature transferring capabilities in two-layer neural networks.\nThe presented results and proof techniques are novel and extend the assumptions and scope of the prior work.\nTheoretical analysis is well supported by both illustrative examples (Fig. 1-3) and actual experiments on synthetic (Fig. 4-5) and real data (Fig. 6-7).\n\n# Weaknesses\n\nThere is a significant gap between the specific setup considered in this study for theoretical derivations and real practice.\nNot to mention the general questionability of applying the two-layer analysis to explain feature learning in actual deep neural networks, I hardly find the parallels between the common practice and the provided results for 1-vs-1 perfectly class-balanced K-classification in the limit of data dimension $d \\to \\infty$ and one gradient descent step with learning rate $\\eta = 1$ under additional assumptions.  \nTo me, the main contribution of this work &mdash; proving in a very limited context an intuitive thesis that *clustering performance in feature space depends on the degree of alignment between training and testing distributions* &mdash; is marginal and does not provide any significant novel insights.\n\nThe paper is not very clearly written, it contains a large amount of inaccuracies, undefined values, and notation abuse that hinders comprehension of the paper content.\n\nRelated Work: I am not an expert in this specific domain, but it seems that some related works on transfer learning theory (in two-layer models specifically) might be missing here.", "was_revised": true, "final_justification": "I have read the authors rebuttal and raised my score, since a large portion of my concerns were addressed. Still, I consider the results of this work very specific as it follows a very narrow setting, which is far from the conventional practice.", "rebuttal": "We appreciate the detailed reviews aimed at improving our paper. In particular, we are grateful for your recognition that our theoretical results provide novel extensions of prior work to non-centered sub-Gaussian distributions and are well supported by empirical evidence. However, you expressed concerns regarding the gap between our theoretical assumptions and practical neural networks, as well as the claim that our theoretical findings do not offer significant novel insights. We would like to first address these concerns before responding to your questions.\n\n## On the Gap Between Theoretical Setup and Practice\n\nWe acknowledge the simplified nature of our model (1. two-layer, 2. 1-vs-1 classification, 3. perfectly class-balanced, 4. proportional regime $d \\rightarrow \\infty$ 5. one-step gradient descent, 6. learning rate 1). However, **we support our claims with experiments on deep networks (e.g., ResNet) as in Section 5** and observe that the theoretical predictions remain consistent in practice. On top of that, we conducted **additional experiments for the rebuttal by extending our assumption**. Specifically, we performed experiments under the Expr 1 setting with **(1) 2-layer networks trained for up to 5 steps, and (2) networks with 2, 3, 4, and 5 layers.** In both cases, we observed that as the *train-unseen similarity $\\beta^\\top \\mu$ increases, Cohesion and Separability __consistently improve__.* These results are included in the final part of our rebuttal to Reviewer MzGZ under Multi-Step and Multi-Layer Experiments.\n\nLastly, **this setting parallels a large body of existing theoretical work** (see Table 1) and is justified for the following reasons:\n\n1. Two Layer networks\n    \n    We adopt this two layer network as it adequately enables analytical insight into feature learning and transfer, and has **proven effective and sufficient in prior work for explaining key phenomena in deep models**, such as feature learning and the superiority of neural networks (Goldt et al. [2020], Ba et al. [2022], Ba et al. [2023], Cui et al. [2024], Mousavi-Hosseini et al. [2023]).\n    \n2. 1-vs-1 classification: addressed in question 1\n3. Perfectly class-balanced: addressed in question 2\n4. Proportional regime: addressed in question 3\n5. One step update\n    \n    As supported by a growing body of theoretical work (Appendix Table 1), one-step updates have been successfully used to explain feature learning effects in real neural networks. This setting is not an arbitrary simplification but **a theoretically grounded framework** (Moniri et al. [2024],  Dandi et al. [2024], Demir and Dogan [2025]), especially given the *well-documented strong influence of early-phase dynamics on final model behavior (Ba et al., 2022)*. Within this framework, we show that **even a single gradient step reveals meaningful clustering** through the emergence of the $s_L$ term which sufficiently addresses our core research question. (see Appendix L).\n    \n6. Learning rate 1\n    \n    We would like to point out a detail that may have been overlooked respectfully — please refer to line 119. While **our theoretical results hold for learning rates $\\eta = \\Theta(1)$**, we represent the learning rate as 1 purely to avoid notational complexity. This setting is equivalent to using a small learning rate (e.g., 0.01) in standard neural networks.\n    \n\n## Novelty\n\nWe will explain below why our work goes far beyond merely rephrasing known intuitions.\n\n1. A Novel Framework for Analyzing Classifier - Deterministic Spike-Based Closed-Form Decomposition:\n    \n    We establish a foundational framework for analyzing the **feature behavior of classifiers on new inputs**, rather than focusing on the commonly studied regression task (Ba et al. [2022], Dandi et al. [2023], Cui et al. [2024], Dandi et al. [2024]). Notably, **our predominant feature component β admits a closed-form, deterministic expression**, unlike the stochastic feature decompositions used in recent works such as Demir and Dogan [2025] (see line 650). As a result, we are able to directly analyze the output of the learned feature map in Section 4.\n    \n2. Counterintuitive Finding – Importance of Relevant Classes, not Many Classes:\n    \n    Beyond clustering, Corollary 4.7 in Section 4.2 and Experiments V–VI show that, contrary to common belief, which emphasizes the importance of more training samples, **feature expressivity depends mainly on semantically related classes.** This insight challenges the focus on quantity (Brown et al., 2020; An et al., 2023; line 337), suggesting that reducing training classes, not just samples, may improve feature transfer performance and efficiency. Moreover, our work **suggests a potential new direction for coreset selection** —for instance, *reducing the number of training classes or training data not contributed to β direction to lower computational complexity while still enabling successful clustering and feature transfer.*\n    \n3. Unifying empirical observations and prior theories - Theoretical Quantification of Feature Transfer Phenomena:\n    \n    Our work offers a theoretical contribution by unifying empirical observations and prior theories, clarifying when feature transfer succeeds or fails (Section 4.1). While feature transfer is known to work better when unseen inputs resemble training data, previous metric learning studies mainly focus on performance gains without consistent explanations (See 671). In contrast, our framework precisely identifies both when feature transfer works (Propositions 4.4 and 4.5) and when it breaks down, such as when multiple unseen classes map to the same spike β, causing Separability to fail despite high similarity. **This quantitative characterization of transfer success and failure is a key strength of our work**.\n    \n\n## Response to Questions\n\n1. To ensure compatibility with existing regression-based frameworks and to allow for a more tractable derivation, we adopted a 1-vs-1 classification setting. However, by avoiding the matrix formulation in Equation 2 and instead representing the labels y using one-hot vectors, our analysis could potentially be extended to general k-class classification problems as well.\n2. Our proofs only require that $X$ is sampled from class-conditional distributions that are *non-centered sub-Gaussian*, so **the proof technique for $\\beta = \\frac{1}{n} X^\\top y$ remains unchanged even in imbalanced settings**.\n3. The assumption $d \\to \\infty$, commonly used in feature learning research (e.g., Ba et al. [2022]), facilitates theoretical analysis of neural networks. Ba et al. [2022] justify this regime as representative of modern neural networks with large datasets and architectures. We adopt it to isolate the key low-rank spike term (Theorems 3.1 and 3.3). **From Section 4 onward, our results assume a fixed, finite, and sufficiently large sample size $N$**, supported by **experiments demonstrating the theory’s applicability to large, finite neural networks.**\n4. The notation $\\mathcal{S}^{d-1}$ refers to the $(d-1)$-dimensional sphere embedded in $\\mathbb{R}^d$. $W_0[i]$ is a $d$-dimensional vector indexed by $i \\in [[N]]$. Sorry for the mistake.\n5. To ensure consistency in the theoretical objects we analyze, we adopted the assumption used in Moniri et al. [2024]. While Ba et al. [2022] initialize parameters with Gaussian distributions, which would lead to differences in the proof techniques, we expect similar qualitative results under that alternative setup.\n6. We apologize for the typographical error. The intended meaning is that the problem corresponds to *K* classification tasks, not *P*. Thank you for pointing this out.\n7. This is a typographical error. The assumptions are the same as in Moniri et al. [2024], where σ′, σ′′, and σ′′′ are bounded. **The boundedness of σ itself was not used in the proof.** We apologize for the mistake and will make the necessary correction.\n8. The Hermite polynomial $H_k$ is discussed in detail in Appendix E. For readability, we will include a reference to it in the main text.\n9. Theorem 3.1 concerns an approximation of the gradient and is not a direct result about $F_0$. Additionally, although the term $\\frac{1}{n}$ appears in $A_{ij} = \\frac{c_1}{n} X_{ij}^\\top y a_{ij}^\\top$, as summarized in Remark K.4 and Equation 24, as $n \\rightarrow \\infty$, almost surely $||\\beta||$ and $||a_{ij}||$ remain at constant scale, so $||A_{ij}||$ also stays at a constant scale and does not degenerate as $n$ grows.\n10. $\\beta$ refers to $\\beta_{ij}$, but the distinction of i and j is not essential, so we omit the subscripts. For clarity, we will explain this in the main text. We apologize for any confusion caused.\n11. Thank you for pointing out the typo.\n12. It is a Gaussian distribution. We will make this explicit.\n13. We simplified it as it does not play a significant role in our analysis. *It is a term unrelated to $\\beta$ and $\\mu$ that appears in the proof*, and it is defined in Appendix H. We will indicate where it is defined.\n14. We defined cohesion and separability in Definition 4.3 as **functionals** (functions from functions to scalars) that take **any** feature extractor F (which may be $s_L$, $F_0$, or $F_L$). It can be computed using $s_L$ instead of F in the definition. Using *F* seems to confuse, so that we will replace it with a different symbol, such as *G*.\n15. Sampling from the **test distribution**, which is standard practice in the metric learning field, will be clearly described in our work.\n\n## Response to Concerns on Related Work\n\nRelated work on transfer learning theory is discussed in *Appendix D* (line 661). These studies examine how changes in the teacher function structure affect sample complexity. **To our knowledge, theoretical analyses of feature transfer—especially in metric learning or open-set clustering without further training—remain unexplored** (see line 671). For clarity, we provide a more detailed discussion of these aspects in the main text."}, {"review_id": "HYM0CX4nBN", "reviewer": "Reviewer_fCzN", "summary": "The work establishes a theoretical framework to justify that the source representations referred to as spike direction can serve to indicate the cohesion and separability in the sense of alignment when applying downstream unseen data. The paper treats multi-classification as multiple two-class classification problems, leading to the Eq. (2) represented hidden-layer gradient in terms of MSE loss of a two-layer neural classifier as formulated in Eq. (1). Then, Eq. (4) formulates the feature undergoing update into two parts: The original one of random initialization, and the incremental part by applying a bounded proxy of gradient defined in Eq. (3), in the framework of Hermite decomposition, where the latter one is termed as Dominant Feature or Spike Component. Then, the train-unseen similarity is deifined as the alignment between the mean unseen pattern and the spike direction to allow computing the so-called Cohesion and Separability (definition 4.3), which leads to proposition 4.4 and proposition 4.5 in Eq. (6) and Eq. (7), respectively. Eq. (6) and Eq. (7) reveal that the cohesion increases with the alignment of unseen pattern to the spike direction, and the separability increases when the unseen patterns are aligned oppositely in reference to the spike direction. Figure 3(c) illustrates some cases in which the two patterns are distinguishable/indistinguishable. Numerical experiments are included.", "questions": "Q1: In fact, we concern manifolds more than the mean representations of two classes in measuring the separability. If Figure 3(c) illustrates the distribution of two classes, it is possible that only a few examples are indistinguishable when projecting to the spike direction, subject to the manifolds of distributions. Besides, the practical case is more complex in that the convolution or self-attention operations have modified the representations prior to feeding them to the classifier and the end-to-end learning can thus avoid the ad hoc cases shown in Figure 3(c).\n\nQ2: I cannot understand why recall@1 can manifest the alignment of unseen pattern to the spike direction in Eq. (6) and Eq. (7), which is subject to only the mean representation. \n\nQ3: A possible bug is: The recall@1 concerns the nearest neighbor only but omits the overall distribution of the test data.\n\nQ4: Does the bias term omitted in Eq. (1) affect the generality of the neural approximation?", "limitations": "The experiments are not solid.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength: Domain adaptation is studied from a theoretical perspective.\nWeakness: The conclusion is not surprising, and not informative to indicate the possible trend.", "was_revised": true, "final_justification": "It is the only paper I thought publishable in my reviewing list. My final rating is accpet.\n\nMy final justification is accept. I reviewed 5 papers but the quality of the other 4 papers is not plausible, very rare case in my experience. This paper seems fine althought not as fine as I expected. Please read my “Final Justification”, which is finished much earlier. My final recommendation is accept but I cannot improve the rating.", "rebuttal": "We appreciate your positive evaluation and constructive feedback, particularly regarding the clarity of the paper. First, we would like to take this opportunity to address some misunderstandings, especially those related to Q1 and the interpretation of Figure 3(c) and the usage of recall@1 (Q2, Q3) with additional experiments in the Expr 4 setup confirm that both **Separability and Recall@1 serve as indicators of clustering performance**, consistent with our findings in Expr 1, 2, and 3. Then, we address the concern regarding the conclusion and respond to Question 4.\n\n## interpretation of Figure 3(c) & Question 1\n\nWe believe the phrase ”*the two patterns”* in your summary refers to the Cohesion and Separability patterns, and we’d like to clarify this potential confusion. As shown in Eq. (6) and Eq. (7), the train-unseen similarity governs Cohesion and Separability, and their relationships are illustrated in Figures 1, 2, 4, and 5. In contrast, Figure 3(c) stems from Section 4.2 and aims to illustrate that, **depending on $\\beta$, inputs can become distinguishable or indistinguishable in the learned feature $s_L$**. This figure presents a pair of **individual points** (rather than class means or distributions), highlighting a key insight for the experimental section and the main conclusion: *only semantically relevant classes contribute meaningfully to feature extraction*.\n\n### Response to Question 1\n\nFrom a distributional viewpoint, when class **manifold distributions align with the spike direction $\\beta$, they are separable; if they lie in subspaces orthogonal to $\\beta$, they become indistinguishable**, regardless of manifold structure (Corollary 4.7). Your consideration of manifolds is valid, and our key contribution is identifying the precise condition—alignment with **$\\beta$ that governs distinguishabilit**y.\n\nWe expect that, despite the nonlinear transformations in deep architectures (e.g., convolution or attention), inputs orthogonal to the learned discriminative directions tend to persist after training, owing to the structure of matrix operations. Such inputs could be indistinguishable to the model, which underscores the importance of identifying spike directions—a central goal of our analysis in section 4.2. Moreover, discovering such directions in deeper architectures would also be crucial for understanding model behavior.\n\n## Response to Question 2\n\nYour question may stem from a few conceptual confusions. We first address your concern directly and then provide clarifications to ensure a precise understanding of the relationship between recall@1 and the alignment of unseen patterns to the spike direction.\n\nFirst, **recall@1 serves as a surrogate measure of the Cohesion and Separability** of learned features (as detailed in our response to Q3). According to the interpretation given in Eq. (6), Eq. (7), and Section 4.1, this **Cohesion and Separability is governed by the train-unseen similarity,** which you referred to as the *alignment of unseen patterns to the spike direction*.\n\nTherefore, our central claim is that **recall@1 is governed by the alignment of unseen patterns to the spike direction**, which is supported by our empirical results (Experiments 1–4).\n\nWe would also like to clarify the following technical points to avoid further misunderstanding:\n\n1. Eqs. (6) and (7) involve more than mean representations.\n    - As stated on Line 35, *Cohesion and Separability represent standard clustering performance criteria* [Clémençcon, 2011; Papa et al., 2015; Liu et al., 2017; Li and Liu, 2021]. (See also Line 705 for further background.)\n    - Eqs. (6) and (7) provide approximations of the Cohesion and Separability of feature representations learned after a single gradient step (see Theorem 3.3, Definition 3.4, Line 173, and Appendix L).\n    - Hence, these equations quantify the clustering performance of the trained features for a given unseen class.\n2. The alignment of unseen patterns to the spike direction is distinct from recall@1.\n    - *This alignment is captured by $\\beta^\\top \\mu$, as defined in Eq. (6) and Eq. (7), and reflects train-unseen similarity.*\n    - As in our analysis (Line 282), the alignment is captured through train-unseen similarity and semantic similarity, not recall@1.\n    - Instead, this alignment determines the clustering performance, and thus indirectly affects recall@1, but the two are conceptually distinct.\n\n## Response for Question 3: Recall@1 serves as a practical surrogate for clustering performance, cohesion, and separability.\n\n**When both Cohesion and Separability are high, the nearest neighbor is more likely to belong to the same class, resulting in a higher recall@1** measured in *inner product*. This is a standard practice in metric learning (Wang et al., [2018], Zhai and Wu [2019], An et al., [2023]) for measuring clustering performance. While recall@1 focuses on the top-1 neighbor, we claim that it is a practical and reliable surrogate for evaluating the combined effect of Cohesion and Separability. Moreover, in Experiments 1, 2, and 3, we observed that recall@1 exhibits a consistent increasing trend that aligns with the behavior of Separability, further supporting its relevance in evaluating performance (See appendix Figures 25, 26, 27, and 28).\n\n## Additional Experiment of Separability\n\nTo address concerns about Recall@1 as a surrogate for clustering criteria, we additionally evaluated **Separability in Definition 4.3 for Expr. 4**, alongside *Expr. 1–3*. Following standard practice (Zhai & Wu, 2019), we trained with unit-normalized features and, thus, report Separability only.\n\nWe observed that models **best separate test data when trained on the corresponding domain (e.g., CAR to CAR), consistent with the Recall@**1, reinforcing our main argument of Expr 4. The results are provided based on the *ResNet18 with the initialization* setting. Further experiments under different setups also confirmed this pattern and will be included in the camera-ready version.\n\nLastly, as discussed in our response to Reviewer MzGZ, we provide supporting evidence for a meaningful connection between **train-unseen similarity** and **semantic similarity**. We also test the validity of our theory in both **multi-layer** and **multi-step** settings in our response for MzGZ.\n\n| Train data | mean sep | std |\n| --- | --- | --- |\n| CAR test |  |  |\n| CAR | **-0.25** | 0.12 |\n| CUB | -0.75 | 0.12 |\n| SOP | -0.69 | 0.13 |\n| ISC | -0.86 | 0.08 |\n| CUB test |  |  |\n| CAR | -0.63 | 0.12 |\n| CUB | **-0.28** | 0.19 |\n| SOP | -0.42 | 0.13 |\n| ISC | -0.79 | 0.08 |\n| SOP test |  |  |\n| CAR | -0.29 | 0.10 |\n| CUB | -0.14 | 0.11 |\n| SOP | **-0.01** | 0.01 |\n| ISC | -0.28 | 0.10 |\n| ISC test |  |  |\n| CAR | -0.82 | 0.07 |\n| CUB | -0.81 | 0.06 |\n| SOP | -0.47 | 0.09 |\n| ISC | **-0.01** | 0.01 |\n\n## Addressing the Weakness in the Conclusion\n\nTo address the concerns raised regarding the strength of our conclusion, we propose the following improvements to articulate our contributions better and clarify the broader implications of our work:\n\n1. **Extension of prior theory**: Our analysis generalizes existing frameworks by incorporating *non-centered sub-Gaussian input distributions* (Refer Appendix I–J) and *deterministic feature approximations for classification tasks* (Refer line 650).\n2. **Versatile analytical framework**: Our approach opens up several promising avenues for extension, including applications to *multi-step training dynamics*, and *potential connections to Neural Collapse* under arbitrary input distributions.\n3. **Future extension to multi-class softmax**: A natural next step is to extend the notions of *Cohesion and Separability* to the multi-class softmax setting, incorporating techniques such as *normalization* and *temperature scaling* to more closely reflect practical neural network training.\n4. **First quantitative link between semantic alignment and clustering**: To our knowledge, this is the *first work to quantitatively connect train-unseen alignment with open-set clustering performance*, rather than relying solely on hypothesis-level arguments (line 671).\n5. **A representation-driven view of coreset construction**: Our work suggests a potential new direction for representation-aware coreset selection—for instance, *reducing the number of training classes or training data not contributed to $\\beta$ direction to lower computational complexity* while still enabling successful clustering and feature transfer.\n\nTaken together, these points clarify that our work offers not only strong theoretical insights but also **practical relevance and extensibility** across a range of future research directions.\n\n## Response to Question 4\n\nWe excluded the bias term in Eq. (1) for analytical simplicity. This follows the convention in several recent studies, such as Moniri et al. [2024] and Ba et al. [2022], where feature learning is analyzed without bias. As Ba et al. [2023] and Mousavi-Hosseini et al. [2023], incorporating the bias term is possible, but the simplified formulation in Eq. (1) is sufficient to address the research question."}]

Author rebuttal:

Meta-review: This paper studies transfer learning, in particular, a theoretical analysis of how features learned by a two-layer classifier generalize to unseen classes. The analysis is conducted in a simplified setting: one gradient descent step and an asymptotic regime where the sample size, data dimension, and feature dimension all grow to infinity. Within this framework, the authors decompose features into components and study how similarity between the training and unseen data affects clustering quality. Experiments on both synthetic and real-world data support the theoretical findings.

A common concern is the restricted setting of a two-layer model, a single gradient descent step, and the asymptotic case where all dimensions and sample size go to infinity. Nevertheless, the reviewers acknowledge the solid theoretical analysis and contributions, and overall recommend acceptance.
Decision: Accept (poster)