The Impact of Geometric Complexity on Neural Collapse in Transfer Learning
Authors: Michael Munn
Benoit Dherin
Javier Gonzalvo
Venue: NeurIPS 2024 poster (2024)
Keywords: transfer learning
geometric complexity
neural collapse
implicit bias
flatness
generalization bounds
Primary area: learning_theory
TL;DR: We show that pre-trained models with lower geometric complexity lead to lower neural collapse and better transfer learning performance, particularly in the few-shot setting.

Abstract: Many of the recent advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this empirical success is incomplete and remains an active area of research. Flatness of the loss surface and neural collapse have recently emerged as useful pre-training metrics which shed light on the implicit biases underlying pre-training. In this paper, we explore the geometric complexity of a model's learned representations as a fundamental mechanism that relates these two concepts. We show through experiments and theory that mechanisms which affect the geometric complexity of the pre-trained network also influence the neural collapse. Furthermore, we show how this effect of the geometric complexity generalizes to the neural collapse of new classes as well, thus encouraging better performance on downstream tasks, particularly in the few-shot setting.

The Impact of Geometric Complexity on Neural Collapse in Transfer Learning
Michael Munn" Google Research munn@google. com
Benoit Dherin" Google Research cherin@google. com
Javier Gonzalvo Google Research
xavigonzallo@google.coo
Abstract
Many of the recent remarkable advances in computer vision and language models be attributed to the success of transfer learning via the pre-training of large foundation models. However, theoretical framework which explains this empirical success is incomplete and remains active area of research. Flatness of the loss surface and neural collapse have recently emerged as useful pre-training metrics which shed light on the implicit biases underlying pre-training. In this paper, we explore the geometric complexity of model's learned representations as fundamental mechanism that relates these two concepts. We show through experiments and theory that mechanisms which affect the geometric complexity of the pre-trained network also influence the neural collapse. Furthermore, we show how this effect of the geometric complexity generalizes to the neural collapse of new classes as well, thus encouraging better performance on downstream tasks, particularly in the fho- setting.
1 Introduction
Many of the recent remarkable advances in modern machine learning their success in part to transfer learning [4] 2
49 50 (66). While there are different approaches to this technique, the standard one involves two stages. In first stage, called pre-training, ones trains deep neural network on a general, large- scale dataset the form a supervised or unsupervised source task; e.g., ImageNet CIFAR-100 [4 33 for image models the Common Crawl, C4 or LMIB datasets for language models. In the second stage, one then leverages portions of the pre-trained
[5] 28
1 49 network
as features map embeddings that can then be adapted, fine-tuned, on a more specific target task. Often these target tasks are unknown at the time of pre-training and labeled data may be very scarce, such in the context of few-shot learning [34 51 60). However, despite these limitations, this approach often results in a fine-tuned model that achieves quite impressive performance substantially better than training on the target task alone and requiring less computational resources [20). Despite this empirical success, a comprehensive theoretical understanding of the mechanisms underlying this effectiveness of transfer learning
fully understood and remains an
active area of research 63
One interesting line of research suggests that the effectiveness of transfer learning is due to the implicit biases encoded pre-trained models [2 36 48). These implicit biases effectively constrain the hypothesis space, guiding the model towards solutions that have preference for smoother functions
55 588 or reduced complexity of way that these implicit biases have been used to help explain the success of deep learning, recent work has shown that the notion of neural
12 38 40 (33] simpler geometry in the loss surface 3 [3
18
the internal learned representations 23 (2) In the
collapse 222 35 62
and flatness of the loss surface (36) can also inform the mechanisms behind
Equul contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

transfer learning. This suggests that these preferences during pre- trainini also act as a form of prior knowledge which is highly transferable to downstream tasks, even with limited task-specific data.
In this paper, we present a novel perspective that further sheds light on these mechanisms and implicit biases hidden within transfer learning. Our approach analyzes the geometric complexity [16 17 of the internal representations learned by the deep neural networks during pre-training and provides a complementary theoretical framework which unifies previous work examining the role of neural collapse and loss surface flatness in transfer learning 22 33)] In particular, we show through experiments and theory that the geometric complexity of the pre- rre-trained network directly controls the neural collapse of the pre-trained model and thus its efficacy in transfer learning, particularly in the few-shot setting. We argue that geometric complexity (similarly to flatness of the loss surface and neural collapse) can be used as hidden progress metrics, cf. [2]. for transfer learning, serving as an informative proxy toward the transfer power of pre-trained network.
Our primary contributions are the following:
We uncover relationships between learning-path flatness, neural network geometric- complexity, and embedding neural-collapse providing a framework to understand how these implicit biases interact (Section +
We show through theory that the geometric complexity (GC) of a neural network controls neural collapse and verify this empirically by showing how mechanisms which regularize the GC in turn put pressure on the neural collapse (Section 4.1 and Fig. D
We demonstrate both theoretically and empirically that pre praitedined networks with lower GC promote lower neural collapse on new unseen target classes, and thus enable improved transfer accuracy during fine-tuning (Section S and Fig. 4
We prove a new generalization bound in terms of geometric complexity (Section 4.3
We show that the empirical GC can be accurately and efficiently estimated on small number of samples, input coordinates, and output features making it computationally tractable compared other progress metrics in machine learning (Section 4.2 and Fig.
Notation.
Throughout, we denote by ||- || the 1.2 Euclidean norm and by ||-[[x the Frobenius norm.
2 Background and Related Work
The implicit biases introduced by an optimization algorithm play a crucial role in deep learning and in the generalization ability of the learned models [43 44 55 64). They help ensure that the model not only finds a solution with low error but also one with low complexity which generalizes well 126 677. Uncovering the mechanisms of these implicit regularizers is crucial for understanding how the model learns, both as a means to improve generalization and as valuable leverage for designing more efficient algorithms and decreasing costly experiment iteration cycles.
Here we focus on three seemingly different themes behind implicit regularization in deep learning and explain how they are related: the loss surface slope measuring the flatness of the learning path 3 29 45), the geometric complexity the learned model function measuring its variability with respect to dataset [16] 42), and the neural collapse B2] 46) measuring how neural network efficiently clusters its learned representations of the input class examples in embedding space.
Flatness in parameter space. In parameter space. the learning dynamics is fully characterized by the discrete optimization path 0, where t N. At each step, one can compute the flatness of the learning path as the slope of the tangent space at 0g to the loss surface C = {0, L(O)) 0 E R"). As shown in 3 Appendix A.2], this slope coincides with the loss- gaadderatentt square norm: (l) || ((,)) 2!? This quantity has been used to bound a generalization gap in [25] and as a beneficial
explicit regularizer in B 24 58 indicating that learning curves with lower slope values tend to also have better test performance.
Moreover, many standard optimizers and common training heuristics have been shown to put an implicit pressure on the learning-path flatness, making it an implicit bias of the training procedure 10
3 5
13
25
3 58 Related to the learning curve slope is the optima sharpness
1 trace H(.))
sharpness (8.)
n

---

where 0, is global minima toward which the learning dynamics converges to 8, + 8a and H is the Hessian of the loss. Since the sharpness of an optima corresponds the mean curvature of the loss surface at that point, flat minima (i.e., minima with low curvature in all directions) can be reached only through learning paths with shallower slopes.
Neural collapse in embedding space. Neural collapse [22] 27 46
refers to a phenomenon ob- served in deep learning where the embedding network (i.e., the subnetwork I before the logit layer g in neural network h(z)- = softmax(g(f (xx))) collapses the input points around their respective class means. Furthermore, these class means form a simplex creating an equiangular tight frame (ETF) centered around the global mean with roughly equal distance between its vertices. Intuitively, such a phenomenon is beneficial to generalization since the embeddings of different classes are optimally separated, making the job of the classifier head softmax(g(? easier. To measure neural collapse, the authors in [2 introduce the diistnd normalized variance (CDNV) as
Var, (Q.) + (Q,)
Vy(Q, Q2)
( - (,12))
where Qr - (x))x is the input distribution for classes E {i and where
12((2)) = Fe and Var,(Q,)= = 1000==1111(00000 ())((2)) -
(2)
Neural collapse between two classes happens when their class variance decreases while the distance between their class means increases, causing lower values of their CDNV. Following (22), given well-balanced input distribution Q = (Q1 +....... +
Qx) with & classes (Q, being the input distribution for class i), neural collapse (NC) during training is characterized by the following limit as the number of training steps goes to infinity:
lim NC(ft, Q) = where NC(f, Q) = Avg.ys (V,(Qi, Q,)), 1-000
(3)
and where fe is the learned embedding network at step f. Thus, more neural collapse (i.e., more clustering around better separated class-means) happens with lower NC values. Proposition from [22] proves that NC(J, Q) bounds generalization gap and lower values of NC(Q, are correlated with better test performance of the neural network f, and 337 shows that explicitly regularizing for neural collapse is beneficial.
Geometric complexity in function space: The geometric complexity (GC) of a function f : E IR* w.r.t. a data distribution Q = ((r)dr on Rdd is defined as the expectation of the gradient Frobenius square-norm w.r.t. to the data distribution 17 given by
GC(F () Fz>q [+)]](
Intuitively, the GC measures the function complexity or variability and is closely related to the Dirichlet energy in geometric analysis (21). Provided mild Lipschitz smoothness assumptions [22 Proposition 3.4], the GC can be estimated accurately on batches training data D by its empirical counterpart
GG
Σ (+)( RED
where D is an i.i.d. sample drawn from Q.
(4)
D
Previous work has explored the relationship between the GC and model generalization. When measured on the full neural network, lower GC values correlate experimentally with higher test accuracy [45] and it has also been used beneficial explicit regularizer (30 59]. In [T6] [7]. the
authors ignore the softmax activation and study the GC measured with respect to the logit network, exploring its connection with various implicit and explicit regularizers and training heuristics. The logit GC has also be used to prove a margin based multi-class generalization bound [42), assuming that the input distribution Q satisfies mild assumption known the Poincaré inequality [.
In fact, both this work and the proof of the generalization bound rely on the Poincaré inequality, which we argue is indeed natural and mild assumption for the types of data distributions typically encountered in machine learning; see Appendix A for further discussion. For completeness, we recall it here, cf. [19]. A distribution Q satisfies Poincaré inequality if, for all differentiable functions v defined on supp(Q),, there exists a constant e such that
Var, (Q) S ClEa [[[V = GC((v
(5)
3

---

Note that the same assumption is used in 7 to prove the law of robustness for overparameterizeed neural networks via isoperimetric inequalities. In this paper, we peel back yet another layer of the network and consider the embedding geometric complexity measured on feature map (Section
Relationship between flatness, neural collapse, and geometric complexity. The learning-patt flatness measured by its slope influences the geometric complexity of the learned solutions [16] Theorem 5. Namely, for dense layers a regularizing pressure on the slope of the learning path in parameter space transfers a regularizing pressure the geometric complexity of the learned solution in function space. similar result also been shown for special attention layers as well [52].
In this paper, will see that imposing regularization pressure the embedding geometric complexity encourages more neural collapse of the model. This results in the following chain of influences from regularization pressure:
learning path flateess
function geometric complexity Regularizing Pressure
embedding neural collapse Lower CDNV
Regularizing Pressure
3 Problem Formulation
We are interested in understanding the relationship between the geometric complexity (GC) and neural collapse (NC) in the transfer learning setting and explore this relationship in two stages. In the first stage, we examine the general impact of the GC on NC during model training/pre-traniing Secondly, we examine how this relationship provides insight into the mechanisms behind transfer learning and the advantageous implicit biases of the pre- treiiiaining stage. In short, lower GC during pre-training on source classes leads to lower NC for target classes and improved transfer accuracy.
For the first stage of our inquiry (Section 4). we are concerned with a k-classification task. Let D == {(3,, y4) m be a dataset drawn from distribution Q with E Rd and yi E (ez | 1,.... k} where ej E R are the canonical basis vectors representing a one-hot encoding of the labels. We aim to learn this task using neural network denoted by a function ha : Id + I parameterized by o (though simplify notation we subsequently drop this dependence on 0).
We can write ha) = softmax(g([(x)) where g TRP - & is a classifier head mapping from the feature space IRP to the prediction layer R$ and f R2d + RP is a feature mapping from the input space to the penultimate embedding layer of the network before the classification head. The standard way of training he is to find, via some stochastic optimization technique, an optimal parameter configuration 6_ such that 0_ argming E (2)) (2:4), y1) for given loss function
e: RR x R [0, 00).
To analyze the role of neural collapse and geometric complexity in this setting, we focus our attention on the feature map f Rd + RR. The NC as described in Section 2 is defined using this sub-network map/ f of and thus we also measure the geometric complexity with respect to this sub-network feature mapping as well. Throughout this paper, we use the term embedding GC to refer to the GC of such a feature map ʃ and unless otherwise specified, when we refer to the model GC we mean the embedding GC, not the GC of the logit network as in [6
Our second stage of inquiry (Section is focused on the role of geometric complexity in transfer learning. For this setting, we assume there is some 1-class classification target task we would like to solve and corresponding target distribution T. We aim to learn a classifier h²' Rdd - IR given some training data D' == {(</; mm which is potentially very limited size. In order to find a good solution, we can leverage a pre-trained feature mapping that has been trained some other source task and source distribution S where more data available. For example, by leveraging a feature map if pre-trained in the above setup, the target task classifier h'' can instead be trained on the outputs of our feature map; i.e., (f(x)), 2/) pmm In this way, we can write h =sftmaxxg' the parameters of f are fixed and we only need to learn the parameters of g Rp + R². Ideally, provided f is a rich enough feature map trained on a general enough source distribution dataset, then the task for learning g is much simpler.
where

---

4 Geometric complexity and neural collapse
In this section, we explore the general relationship between the embedding GC and neural collapse showing that the geometric complexity can be used to bound neural collapse. Next, see how the geometric complexity can be efficiently and accurately estimated, both as an approximation of the theoretical GC as well as through number of sampling techniques for the empirical GC via batch sampling, feature sampling and label sampling. Lastly, following [22], we derive a new generalization bound for neural networks expressed in terms of the embedding geometric complexity; cf. 42
4.1 Geometric complexity controls neural collapse
Previously, it has been shown [6 that the GC can be controlled implicitly through choice of learning rate and batch size, as well as through standard explicit regularization. Practically, this means that these same beneficial tuning strategies which ensure that models have low GC should also work to keep the NC low as well. The following proposition (which we prove in Appendix [A.I) bounds the neural collapse by the geometric complexity provided that the input distribution satisfies Poincaré inequality also assumed in [42 and 77.
Proposition 4.1. Suppose that we have a balanced multi-class cuput-distribution Q with classes satisfying the Poincaré inequality in (5) for constant c, then the geometric complexity of network embedding f bounds its neural collapse as measured by (3): namely, we have the following bound
NC(F,Q) C. QC
(6)
k-1
where k is the number of classes, and dis is the distance between the mean of class and class j.
We call the RHS of the bound in Eq. (6. excepting the Poincaré constant c, the geometric collapse:
GC(F.Q) k-1
Σ
ci
where k is the number of classes, and dış (Qp) - My ( denotes the Euclidean distance between the mean (Q,) and (Q,( of class and class j (resp.) in embedding space.
This quantity can be seen as another proxy metric measuring neural collapse. It is made of two main parts decoupling the variances and mean differences present in the neural collapse framework. The variances are consolidated into a single factor through the GC, while the mean differences averaged across classes a separate factor. This separation allows the overall intra-class variability to be influenced through the GC term, ensuring sufficient between-class separation through the mechanisms identified in neural collapse.
This bound provides powerful method to control the overall within-class variability via the L2-norm of the model gradient. As result, the geometric collapse provides simplified and more refined approach to managing class separability and variance in deep learning models.
We verify the relationship posed in Eq. (6) through experiments on VGG-13 trained on CIFAR-10 (see Figure []]. Through both implicit and explicit regularizers the pressure on the embedding GC translates to a direct pressure on the neural collapse of the model via the geometric collapse. As already observed in [1]. lower levels of GC coincide with higher test accuracy as show in Figure in Appendix A.5 containing the learning curves of that experiment.
In Appendix A.5 we see the same relationship holds across various datasets; e.g., MNIST, Fashion- MNIST, CIFAR- 100, and architectures; e.g., ResNet- 18, ResNet-50. To avoid possible confounding factors that could arise through batch size and learning rate manipulations, we also directly regularize with the geometric complexity in Appendix A.5.6 the same relations hold in this case too.
4.2
The geometric complexity is a reliable and robust measure
One of the key advantages of the GC as a complexity measure is its computational efficiency.
5

---

Leanting Tate
0,005
2234
0.03
0.0
890
Ioratioo
Iaraation
tanaaiion
06625
30
60020
Tek Gecmant Calapee
1.0
0.00
Denetioon
1000000
Tag ceritaationnn Rame
0.00
0,00000
3005
2005
Aumman
20630 40030 20000
30.000
10,000
0,00000
20090
40003
00300
50000
100300
00
20090
40000
00300
50009 500000
Figure 1: Controlling the neural collapse via the model geometric complexity for VGG-13 13 trained on CIFAR-10. Lower embedding GC produces lower geometric collapse (Fq. ( and more neural collapse (i.e., lower NC) for Top row: increased learning rates, Middle row: decreased batch sizes, and Bottom row: increased L2 regularization.
Under mild regularity constraints on the model function the theoretical geometric complexity of a map can be efficiently estimated by its empirical counterpart, as stated the proposition below, already proven in (22 for logit networks (see Appendíx [A.2 for proof). Additionally and crucially, we verify that the empirical GC is a robust and reliable measure with respect to that data sample.
Proposition 4.2. Let f Rd R2P be map with Lipschitz constant L and let Dx be a sample of elements drawn independently from input distribution Q. Then, for any 8 >> 0, we have with probability - 8/2 the following bound
log an
GC(S.)) & GC(F,Dx) + L
(8)
2m
For a function f and data sample Dx as Proposition 4.2 to measure GC(f, Dx) requires computing the Frobenius of potentially very large Jacobian matrix, particularly when p represents the embedding dimension of a feature map. Note that,
d GC(F.Dx) = ΣI
(
Ox,
Although the computational complexity of the GC is impervious to increasing complexity of the network architecture, e.g., in terms of parameter count or number layers, one may run into computation bottlenecks as the sample size m increases or when increasing the dimensionality d (resp. p) of the inputs (resp. outputs). these scenarios, is necessary to find efficient means to accurately approximate sample the Jacobian; ef. [61].

---

We explore the robustness of the GC when measured via samplings along these three axes; i.c., decreasing the number of examples m in the batch, randomly sampling the full Jacobian matrix which has order d x p. and randomly sampling the number of model outputs p. As shown Figure we find that the value of the sampled GC remains stable and consistent to its true value through these fairly simple and naive sampling tricks.
Randamly Ressed jacetian
(haahett3 trained on Ci6e2000
Fandamil Outpur dimeension (Raslecsa EI and Citaridi)
0.005
0.003
0.001
250
302
69200
0.1
0.55
8.74
SLSTECT SouTppie wwwppppci.ioo
Jeurchaless
(2812h abb
2031
Figure 2: The GC computation is robust and consistent to sampling via Left: number of examples in the batch, Middle: number of elements in the Jacobian, Right: by sampling the embedding dimension of the model. Here the GC and subnet GC have been computed over 20 trials, plotting the mean and standard deviations for ResNet- 18 model that has been trained to convergence CIFAR- 1-0 The true value of the GC for each setting is indicated by dotted line.
4.3 A generalization bound with GC through NC
In [42,, the authors derive a margin-based multi-class generalization bound for neural networks which depends on the geometric complexity GC(h, Q) of the full logit network. In this section, extend this result. With inspiration from [22] Proposition 5], we show that the geometric complexity measured the sub-network feature map GC(J, Q) bounds the classification error of the nearest- mean classifier defined by the feature map f.
As in 22 given balanced k-class classification problem, let S = Uceja) Se denote all class samples and define the nearest- mean classifier by hz, := argmin (x) - Hy (Sa)l given by the feature map f where (S.) denotes the sample mean of the set S. under the map f. Define the generalization eтror
Error := [x (L)),,s(*)) + yl]
where (())))(x)))))) # y) is the indicator function of the error set [hj,s(z) # y) and P is the full data distribution from which samples S are drawn.
Proposition 4.3. Suppose that we have a balanced sample S from a k-class imput- snput-distributiin (x. y) ~ P with m. samples per class. Assume further that the induced input distribution Q satisfies Poincaré inequality as in (5) some constant c. Then, for any ō >> O, with probability 1 - 6/2 we have the following bound for the generalization error
++++) (GOUSS)+
log 2mck
E Error] S 16c
+ L
(10)
where p the embedding dimension of the feature map and dej = (S.) - mS outer expectation the left hand side taken over the samples S used produce the classifier hs,s-
Note, the
Proof. This follows immediately from 222 Proposition 5] which gives the bound in terms of the neural collapse instead of the geometric complexity. By using Proposition A. we can replace the neural collapse with the geometric complexity and using Proposition 4.2 we then replace the theoretical GC with the empirical GC, yielding the additional term with the Poinarr constant.
In Figure 3, we plot the LHS and RHS of the generalization bound in 10 from Proposition 4.3 for a VGG-13 model trained on CIFAR-10. In this plot, we see that the bound is not vacuous, demonstrating a relatively tight fit. Note that on the RHS, we omitted the term involving the Lipschitz constant L,, assuming it is small due to its denominator. Additionally, we estimated lower bound for the Poincaré constant e by comparing the magnitudes of the RHS and LHS in the inequality (6) based on results in Figure This approximation yielded 1000 in setup.
7

---

Generalization Bound (c.f Prop'n 4.3)
Nearest Mean Cassifier
Train c A 0E E Σ
0.8
0.2
0.0
20000
40000
60000
80000
100000
lteration
Figure 3: VGG-13 trained on CIFAR-10 with 5 random seeds.
The impact of geometric complexity on transfer learning
Many implicit regularization mechanisms in gradient descent exert pressure on geometric complexity which in turn constrain the neural collapse. In this section, we discuss the affect of this implicit bias in transfer learning. Specifically, show with theory how this bias during pre pre-training explain the mechanisms behind transfer learning. Namely, that lower GC in a pre- pre-trained embedding target classes, simplifying the fine-tuning process.
help
network promotes neural collapse
However, our bound makes clear that certain compatibility conditions between the pre-training (source) distribution and the fine-tuning (target) distribution are needed, which we argue are satisfied in image foundational models and large language models. At last, we verify empirically that known methods to reduce geometric complexity for the pre-trained embedding result in better performance during fine-tuning on unseen tasks, in agreement with our neural collapse bound for transfer learning.
5.1 Lower pre-trained GC leads to improved fine-tuning
To analyze the impact of GC in the context of transfer learning, we consider the same formal setup as in [2 Proposition 2]. Assume class in set C of classes is represented by class-conditional distribution (z P(zly c). Both the pre-training pr-training/soouree classes Q Qs and the fine- tuning/target classes Q....., Q: are assumed to be drawn from a distribution over all classes in C. The induced input distribution on all the classes (i.e., the combined source and target input distribution) denoted Q. while the source-only input distribution denoted by Q. Let JF denote the set of pre-trained feature maps f Rd + I2 selected by the training procedure (e.g., the set of trained embeddings with different initialization seeds but the same training protocol) and consider
A(F*) inf inf (1p)(Q.)) - (((o))-(((e)))))
(11)
feF epc'
With this, we can state the transfer learning bound
Proposition 5.1. Suppose that the source and target input distributions satisfy a Poincaré inequality with constant Cg and eo (resp.). Then, with probability - 5, over the selection Qe and Qe of target class distributions, we have that the CDNV expectation for two target classes is bounded by the geometric complexity of the embedding network in the following way
EQ. [V((e.: x
2q GC(S.Q)
Σ
1
k-1
d
166g supeec GC(F* Qc))
.0c)
V 2T log 1o 2)
+ 8
A(F*)
(k-1)A(F))
4 4supresupp(Q) JEF
[[((x)]|
2 feq sup GC(F",
1+
A(F+)2
VkD(F.02
8

---

where k is the number of source classes, GC(F*, Qc) se.. GC(f.QU) d, (Qp)(Qe)) n uy(Qc,)l and (FF*, Q) is a complexity measure for F over Q (see Appendix A.3
Proof. This follows immediately from 122 Proposition 2] which proves this bound for the NC and distribution variances. Using our Proposition 4.1 we can replace the NC by the GC and swap the variances by the corresponding geometric complexities via the respective Poincaré inequality.
Interpretation. In the bound above, the first term depends on the geometric complexity measured over the source distribution Q and decreases as the number of source classes & increases. Thus, we can predict that lower GC(F. Q) encourages lower NC values (i.e. more neural collapse) on the new larget classes provided that the second and third term of the bound are also small.
However these other two terms involve both the source and target classes, making the full bound no longer dependent only the geometric complexity over Q.
Morcover these two terms can apriori explode since the infimum over the class-mean distances A ( no longer only involves source classes (where this distance is bounded away from zero due to neural collapse into an equidistant simplex) but also target classes (for which we no longer have theoretical guarantees).
However, if the source labels are granular enough in the sense the target labels can be represented as combinations of the source labels (as for instance target label of "dog" can be subsumed as all the breeds of dogs in source classes on ImageNet), then the issues above are mitigated. Formally, we can summarize this granularity compatibility condition between the source and target classes as follows: For each label e of the target class there exists labels c in the source classes such that Qe 1/k(Qu, +... + Qea). Because GC(f, -) is linear, we have
sup GC(F", Q+) S sup GC(F", ,Q))
cel
cE(A)
and thus recover dependence only on source classes.
To address the A(F*) term, observe that the compatibility condition above implies in terms of class- means that y (Qc) = 1 ( (Qc,) +... ( )). Because of neural collapse during pre-training, these source class class-means form the vertices of face in the neural-collapse class class-mean simplex making the target-class mean f (Qz) the barycenter of this face. Since the distance between any two points taken in the vertices of simplex plus its barycenters is bounded away from zero, is A(F"). Note, albeit intuitive and natural, this granularity condition is hard to verify in practice.
5.2 Improve fine-tuning by controlling pre-trained GC
When the compatibility conditions above are satisfied, Proposition 5.1 indicates that increasing the amount of neural collapse for the target classes can be achieved by lowering the embedding GC during pre-training. We verify this indeed occurs experimentally using the same regularization techniques exploited in Section 4 Furthermore, we verify that these implicit methods of controlling pre-training GC produce feature maps that perform better on fine-tuning tasks on CIFAR-FS with a ResNet- 18 in Figure and on mini- ImageNe with VGG-16 in Appendix A.5.5
6 Limitations and Conclusion
There is notable limitation when extending our findings to language modeling and, for example, large language models (LLMs). However, this limitation not specific our work per-se, but instead it is limitation of the application of neural collapse to language models in general, as described in the recent work (65). Namely, language modeling. as conducted via training by token prediction, creates classification task where the conditions for neural collapse are implausible. The main problem, addition to an imbalanced vocabulary, is that the embedding dimension for language models is typically far less than the number of classes (i.e., the total vocabulary size), making the neural collapse simplex impossible exist. Extending the notion of neural collapse to the language models is an open question and an active area of research and beyond the scope of this work.

---

00175
Learting Hall 0,001
0.0
0.150
Srgptt S-ghot
0.35
60100
00975
6.0
1.5
3.0
0.0000
20200
00000
20200
83000
20200
80000
Batch StDc
0.0
0032
16.0
32.0
Sarget S Accuray
S-ght
00006
6.0
0.022
20000
93300
00000
50000
190000
25000
43000
G0ECO
D0300
100000
25000
83000
60000
00050
100000
Ereatiio
Eanation
0.40
0.2001
0.2000
Torjet ghet ACCHRaCY
0.066
0.04
0.50
8.02
3.5
0.25
Exaration
Earation
Iteration
Figure 4: Controlling target neural collapse through source GC on CIFAR-FS with ResNet-18. Lower Source GC produces more neural collapse on target classes (i.e. lower target NC), and higher 5-shot transfer accuracy for Top row: increased learning rates, Middle row: decreased batch sizes, and Bottom row: increased L2 regularization.
Uncovering and understanding the implicit biases that enable successful transfer learning is critical area of research in modern machine learning. Here we provide a framework connecting three different themes behind implicit regularization: flatness of the loss surface, geometric complexity, and neural collapse. We show that the embedding geometric complexity directly controls the neural collapse during training and, moreover, plays role in the success of transfer learning.
Extensive experiments on different image classification and fine tuning tasks across different hy- perparameters verify our hypothesis. We believe this opens up an intriguing direction of research further exploring the role of geometric collapse and geometric complexity in deep learning and provides valuable insight for designing more efficient and effective techniques for model training and fine-tuning.
Acknowledgments and Disclosure of Funding
We would like to thank Patrick Cole for their support. We would also like thank Hanna Mazzawi and Peter Bartlett for helpful conversations.
10

---

References
[1] D. Bakry, Gentil, and M. Ledoux. A simple proof of the poincaré inequality for large class of probability measures. Electronic Communications in Probability, 13:391 1-101, 2008
[2] Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. Advances in Neural Information Processing Systems, 35:21750-21764 2022.
[3] David G.T. Barrett and Benoit Dherin. Implicit gradient regularization. In International Conference on Learning Representations, 2021.
[4] Yoshua Bengio. Deep learning representations for unsupervised and transfer learning. In Proceedings of ICML workshop on unsupervised and transfer learning, pages 17-36. JMLR Workshop and Conference Proceedings, 2012.
[5] Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neu- ral networks driven by an ornstein-uhlenbeck like process. In Annual Conference Computational Learning Theory, 2019.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19911, 2020.
[7] Sébastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. Journal of the ACM, 1012):1-88, 2023.
[8] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. 30th USENIX Security Symposium (USENIX Security 21), pages
2021.
[9] Rich Caruana. Learning many related tasks at the same time with backpropagation. Advances neural information processing systems, 7, 1994.
[10] Matias Cattaneo, Jason Klusowski, and Boris Shigida. On the implicit bias of adam.
WiXiv:230900079/ 2023.
[11] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv: 131.3305 2013.
[12] Anna Choromanska, Mikael Henaff, Michael Mathicu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pages 192-204. PMLR, 2015.
[13] Alex Damian, Tengyu Ma, and Jason D. Lee. Label noise SGD provably prefers flat global minimizers. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, NeurlPS 2021, 2021.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[15] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 180:001810 2018.
[16] Benoit Dherin, Micheal Munn, and David GT Barrett. The geometric occam's razor implicit in deep learning. NeurlPS, OPT2021, 2021.
[17] Benoit Dherin, Michael Munn, Mihaela Rosca, and David Barrett. Why neural networks find simple solutions: the many regularizers of geometric complexity. Advances in Neural Information Processing Systems, 35:2333-2349, 2022.
[18] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference Machine Learning, pages 1019-1028. PMLR. 2017.
[19] Lawrence C Evans. Partial differential equations, volume 19, American Mathematical Society, 2022.
11

---

[20]
Abolfazl Farahani, Behrouz Pourshojae, Khaled Rasheed, and Hamid R. Arabnia. A concise review of transfer learning. CoRR, abs/2104.02144, 2021. httpps://aaxiv..rggg /rg/
2104. 02144
[21] Herbert Federer. Geometric measure theory. Springer, 2014.
[22] Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. In ICLR, 2022.
[23] Peifeng Gao, Qianqian Xu, Yibo Yang, Peisong Wen, Huiyang Shao, Zhiyong Yang, Bernard Ghanem, and Qingming Huang. Towards demystifying the generalization behaviors when neural collapse emerges. arXiv preprint 1.Xiv:33100083355 2023.
[24] Jonas Geiping, Micah Goldblum, Phil Pope, Michael Moeller, and Tom Goldstein. Stochastic training is not necessary for generalization. In ICLR, 2022.
[25] Avrajit Ghosh, He Lyu, Xitong Zhang, and Rongrong Wang. Implicit regularization in heavy-ball momentum accelerated stochastic gradient descent. ICLR, 2023.
[26] Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems,
31, 2018.
[27] X. Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity and dynamics on the central path. In ICLR, 2022.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[29] Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 911::144, 1997.
[30] Judy Hoffman, Daniel A. Roberts, and Sho Yaida. Robust learning with jacobian regularization. arXiv, 2020. http::ttt:////tt //t:://aaiivvvv.. org/ at//a/ass////////////// 02729
[31] Valentin Khrulkov, Artem Babenko, and Ivan Oseledets. Functional space analysis of local gan convergence. In Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, 2021.
[32] Vignesh Kothapalli. Neural collapse: review on modelling principles and generalization. arXiv preprint arXiv:2206. 04041, 2022.
[33] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[34] K Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10657-10665, 2019.
[35] Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda. Zhihui Zhu, and Qing Qu Principled and efficient transfer learning of deep models via neural collapse. arXiv preprint arXiv:2212. 12206, 2022.
[36] Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. In ICML, 2023.
[37] Xuantong Liu, Jianfeng Zhang, Tianyang Hu, He Cao, Yuan Yao, and Lujia Pan. Inducing neural collapse in deep long-tailed learning. In AISTATS, 2023.
[38] Chao Ma and Lexing Ying. The sobolev regularization effect of stochastic gradient descent.
2021. URL BtRL/htpss// http://wwxw..///a/./a///
13462.
[39] Chao Ma and Lexing Ying. On linear stability of sgd and input-smoothness of neural networks.
In NeurlPS, 2021.
[40] Charles H Martin and Michael W Mahoney, Implicit regfareggulrizzation in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1-73, 2021.
[41] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018
[42] Michael Munn, Benoit Dherin, and Javier Gonzalvo. margin- multiclass generalization bound via geometric complexity. ICML, TAGML Workshop, 2023.
12

---

[43] Behnam Neyshabur. Implicit regularization in deep learning. arXiv preprint arXiv: 170.001933
2017.
[44] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614 2014.
[45] Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeftrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: an empirical study. In ICLR, 2018.
[46] Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117 (40), 2020.
[47] Loucas Fillaud-Viviens Francis Bach, Tony Lelièvre, Alessandro Rudi, and Gabriel Stoltz. Sta- tistical estimation of the poincaré constant and application to sampling multimodal distributions. In Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 2753-2763, 2020.
[48] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:200.002177, 2022.
[49] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine I Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[50] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero- shot text-to-image generation. In International conference on machine learming, pages 8821-8831. Pmlr, 2021.
[51] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In Interna- tional conference on learning representations, 2016.
[52] Liliang Ren, Yang Liu, Shuohang Wang, Yichong Xu, Chenguang Zhu, and ChengXiang Zhai Sparse modular activation for efficient sequence modeling. In NeurIPS, 2023.
[53] Mihacla Rosca, Theophane Weber, Arthur Gretton, and Shakir Mohamed. A case for new neural network smoothness constraints. In Proceedings on Can't 't Believe It's 's Not Better" at NeurIPS Workshops, volume 137 of Proceedings of Machine Learning Research, 2020.
[54]
André Schlichting. Poincaré and log-sobolev inequalities for mixtures. Entropy, 21(1):89, 2019.
[55]
Keskar Nitish Shirish, Mudigere Dheevatsa, Nocedal Jorge, Smelyanskiy Mikhail, and Tang Ping Tak Peter On Large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 04036, 2016.
[56] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv::4009 1556, 2014.
[57] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale
image recognition. In ICIR, 2015.
[58] Samuel Smith, Benoit Dherin, David G.T. Barrett, and Soham De. On the origin of im- plicit regularization in stochastic gradient descent. In International Conference on Learning Representations, 2021.
[59] Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel R. D. Rodrigues. Robust large margin deep neural networks. JEEE Transactions on Signal Processing, 65:4265-4280, 2017.
[60] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[61] Guanhua Wang and Jeffrey A Fessler. Efficient approximation of jacobian matrices involving non-uniform fast fourier transform (nufft). IEEE transactions on computational imaging, 9:
43-54, 2023.
[62] Zijian Wang, Yadan Luo, Liang Zheng, Zi Huang, and Mahsa Baktashmotlagh. How far pre- trained models are from neural collapse on the target dataset informs their transferability.. In Proceedings of the IEFF/CVE International Conference on Computer Vision, pages 5549-5558, 2023.
13

---

[63] Karl Weiss, Taghi Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big Data, 3. 05 2016. doi: 10.1 10.11866:00337--16-000333
[64] Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. Advances in neural informa- lion processing systems, 30, 2017.
[65] Robert Wu and Vardan Papyan. Linguistic collapse: Neural collapse in (large) language models, 2024.
[66] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
[67] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 107-115, 2021.
14

---

A Appendix
A.1 Proof of Proposition 3.2 (Neural collapse bound)
Proof. Suppose that we have an input probability distribution ((x) ((x)dr coming from data distribution P(X, Y) (i.e. ((=)dz is the marginal probability distribution of p(z, y)drdy) where Y represents the possible classes. Suppose further that P(X, Y) is balanced multi-class distribution. In terms of the input distribution (xx) this means that
1 (Q) + ... QK) K:
Q
where (z) (2) is the input distribution of class i. Suppose moreover that Q(x) satisfies the Poincaré inequality in (5. that is,
Vary S (I cGC(f.Q) Q)
for some constant Then the statement we want to prove is that the geometric complexity of f. that is,
GC(f.Q) / [V>](())
bounds its neural collapse as measured by
Var, (Qr)
NC(S.Q) #1 Σ
(12)
(1 - My (q12)
More precisely, we want to prove that we have the following inequality:
QC(S,)) k-1
Σ 1
NC(F,Q)
(13)
where d, - (1a)((2)) the distance between the mean of class and class j. Let us now prove this statement.
First of all, since the geometric complexity respects convex sums of data densities, have that for distribution Q = & (Q1 Qx) we can write
CC(SQ) = 1 (GC(F))) + +C(..)))))))++))000)))))))))
(14)
In particular, this means that GC(J < & GCJS Q). Furthermore, the Poincare inequality ensures that Varq, (f) < cGC(J Q,). Using these properties and the definition of the neural collapse, we obtain that:
Vary + (Qr) 21( ur(2,)]
NC(,
#li 1}
GC(F.Q) GCLF Qi)
# i $
2d2
ck GC(f.Q) k(k 1)
Σ 1
which simplifies to the desired result.
Remark A.I. Excepting the Poincaré constant, the quantity on the RHS above given by
GC(F.Q)
(15)
1-1
could be taken alternative measure of neural collapse. We call this the geometric collapse.
15

---

A.2 Proof of Proposition 4.2 Estimating the Theoretical GC by GC)
Proof. Let Q be an (input) probability distribution defined over R and let = {mr) of m 2 points drawn as i..d. samples from Q. Given the map f E RR, we denote the empirical geometric complexity of f over D by GC(F, D) which defined as
m
GÖ(J, D) m
[v,((=))
We begin by showing that
ED~Q"
GC(J, = GCUSG) Q)-
This follows by direct computation, keeping in mind that Q is probability distribution and that the points are independently sampled. We can write Q as (x)) q(r)drr and note that
Ep-Qm [OC(F.D]
E Ex
(z.) 1/
m
m
[ P. (21,,.., .m)
m gmxd
V (x:)(())))9(())) (.gen))))-mmmmdaa ((xm)dxx dxm
m Roxd
1/ [Vaf(())]]()/) ((x1)_.. g(())) ... ((#m)dx) .. dx: dæm
(.,(),) dd)()))
Σ Q)(f.Q)
GC(f.Q).
Now suppose we have two separate independent samples D and D' each of of size m 2 which differ by exactly one point, say F; in D and x in D'. By assumption, the map f L-Lipschitz for some Lipsehitz constnt L > Therefore, we have
GD(f,D) GC(ff m
(IV (V,f(+)) -IV < 1.2/m,
and similarly, GC(F, D') - GC(F, D) < L2 m. It follows that GC(f,D) - GÖ D') S 12/m and by applying McDiarmind's inequality (e.g.. 41). have that for any > 0,
P GC(f ED GC D) < 2 exp(-2me [22).
(16)
Since, from the computation above, En GC(J,)) GC Q) and setting 8/2 exp(-2mel LL2) and substituting for in 16 get that for any 0 with probability as least 1 - 5/2 the following holds:
log 2 2m
GC GC(J. D) + L
This completes the proof.
16

---

A.3 Definition of the complexity measure $1F, Q) in Proposition 4.1
Let us restate here for the sake of completeness the definition given in Galanti et al. 22 First consider the Rademacher complexity R(A) of a set A C IR" which is given as the expectation (X) of the random variable X A (e, a) where components uniformly distributed among the two values -1 and 1. Now consider the input distribution Q coming from multiclass distribution with labels in E C with label input distribution Qc. We define the complexity measure (F, Q) for the candidate functions in F over the source distribution Q as the Rademacher complexity of the following set:
(ei...., En) is a random vector with
(Ur(Qc)) Vary(Q.)) CEC,
A.4 Experiment details
A.4.1 Figure 1 Geometric Complexity Controls Neural Collapse on CIFAR-10.
We trained a VGG-13 neural network on the full CIFAR-10 10 dataset with the provided architecture [56] and using the standard train/test split. Throughout training, we reported the following metrics measured and averaged over multiple batches of the training dataset: 1) the geometric complexity of the model embedding layer; i.e., the layer on which the neural collapse measured, 2) the neural collapse as measured by (3) measured on the model embedding layer, and 3) the geometric collapse of the model also measured on the penultimate embedding layer of the model and given by
GC(S.Q k --1
Σ de (
(17)
where Q is the training distribution, k 10 this CIFAR-10, and (as in the paper) where d,- = (Q.) mQ(())) is the distance between the sample means of class i and class j.
These quantities were measured every 1000 steps of a very long pre-training of 100, 000 steps. The optimizer was plain SGD (note we obtained similar results with momentum) without any regularization nor schedule to avoid masking effects. We used random crop and random flip for data augmentation. The results were all averaged over random seeds for the neural network parameter default initialization. The plots have smoothing applied to the learning curves. Top row: We swept over a learning rate range of {0.001, 0.0025, 0.005, 0.01, 0.025, 0.1} with a constant batch size of 512. Middle row: We swept over batch size range of {8, 16, 32, 64, 128, 256} with a constant learning rate of 0.01. Bottom row: We swept over a 1.2 regularization rate range of (0.0, 0.00025, 0.0005, 0.001, 0.0025) with learning rate 0.01 and batch size 256. Each sweep took roughly 10h of training on a single Google Cloud TPU V3 accessed via a Google colab.
In Figure 5
we show the complete learning curves of that experiment.
A.4.2 Figure 3: Tightness of the generalization bound CIFAR-10
We trained a VGG-13 neural network [56] on the full CIFAR-I0 dataset using the provided train/test split. For this model architecture we use an embedding layer dimension p 1024. Note also that the number of examples per class is m. 5000. We used a constant learning rate of 0.005, a batch size of 512 and trained for 100000 steps reporting metrics every 1000 steps.
Throughout training, we reported the following metrics measured and averaged over multiple batches of the training dataset: 1) the empirical geometric complexity of the model embedding layer; i.e., the layer on which the neural collapse measured, denoted GC, and 2) the sum of the inverse squares of di which denotes the distance between the mean of class i and class j for i, E [k] for i * and, here, 10. These quantities were used to create the blue curve in Figure 3 We also computed the nearest mean classifier error on the test set. Recall, for the feature map and sample S the nearest
mean classifier is defined as h == argmin (x) - Jy (S.)) where, here l = 10, and / (Sc) denotes the sample mean of the set S. for class under the feature map f. To create the orange curve in Figure 3 we plot the average error hf # over the test set.
17

---

LOOPIIN R200
1.9%
0.11
0.06
0.40
6.13
0.00
0.12
1..00
0.5
3.3
22006
10030
50003
00300
0.60
20000
43055
50000
B0008
0.6
25200
10003
60306
30030
0.00
22365
10000
EX0
0900
Satth-5.0
0.95
120
6.12
32
0.80
01
3.3
22006
10230
50003
00:00
0.70
20090
4306
50000
B0004
6:4
20200
40003
62306
30030
23366
46030
Ang.Arisston Fate
0.10
0.09225
Tull DOMeTOR ChreAooe
2.00
0.24
20030
Eara5an
Ianaton
Banatan
Iaratoo
Figure 5: VGG-13 on CIFAR-10. Training has reached terminal phase of training (TPT) with training accuracy equal to (see first and second column). Lower GC in training correlates with higher test accuracy in all settings (see third and fourth column).
A.4.3 Figure Controlling target performance through source GC on CIFAR-FS.
We trained a RestNet-18 neural network with width 1 implemented in Flax hhttps://githti py CIFAR-FS with its initial convolution modified to stride and kernel size of 3 to adapt to CIFAR-FS instead of ImageNet. We used only 10 classes for the source datasets with 600 images per class, which further split in 10/90 split for the test and train splits. The metrics were computed on an average of 100 random samples of 5 classes in the remaining classes of CIFAR-FS. We reported 1) the geometric complexity of the embedding layer measured on the source dataset (Source GC), 2) the neural collapse as measured by (3) measured and averaged the 100 random samples of 5-label samples from the target dataset, 3) the test accuracy obtained by solving the normal equation directly for a ridge regression with only examples per class obtained from the target set. These three quantities were measured every 1000 steps of a very long pre-training of 100,00 steps. The optimizer was plain SGD (note we obtained similar results with momentum) without any regularization nor schedule avoid masking effects. We used random crop and random flip for data augmentation. The results were all averaged over random seeds for the neural network parameter default initialization. The plots have no smoothing applied to the learning curves. Top row: We swept over learning rate range of (0.001, 0.005, 0.01, 0.05, 0.1, 0.25, 0.5) with constant batch size of 256. Middle row: We swept over batch size range of {8, 16, 32, 64, 128} with a constant learning rate of 0.005. Bottom row: We swept over 1.2 regularization rate range of (0, 0.0001, 0.0005) Each sweep took roughly 10h of training a single Google Cloud TPU V3 accessed via Google colab.
18

---

A.5
Additional Experiments
A.5.1 Cifar-10 on ResNet-188
In Figure we trained ResNet-18 neural network on CIFAR-10. The experimental setup is the same described in Appendix A.4.1 The results were all averaged over 5 random seeds for the neural network parameter default initialization. The plots have no smoothing applied to the learning curves.
Top We swept over a learning rate range of (0.005, 0.01, 0.025, 0.05, 0.1, 0.2) with constant batch size of 256. Middle row: We swept over batch size range of {8, 16, 32, 64, 128, 256} with constant learning rate of 0.2. Bottom row: We swept {0.0, 0.0001, 0. 0.00025, 0.0005, 0.00075) with learning rate 0.02 and batch size 512.
1.2 regularization rate range of
Learing
0007
B.LL
00039
6.%
0003
00000
0003
0003
000000
0001
0.0000
40300
00000
10000
0.03
40030
B0030
80003
23003
40003
6000
08008
330000
0.00
0.0000
130.0
0.00
640M
Ranaton
01011
010122
C8305
0.000
0.000
61000
0188
20009 19:00 $0230
100000
0:000
10:00
35030
R3CC
200330
20090 43003 60XC
10303
Figure 6: Controlling the neural collapse via the model geometric complexity for R8 trained on CIFAR-10 10. Lower GC produces lower geometric collapse and more neural collapse (i.e., lower NC) for Top row: increased learning rates, Middle row: decreased batch sizes, and Bottom row: increased L2 regularization.
A.5.2 MNIST on VGG-11
In Figure we trained VGG-11 neural network (56) on MNIST. The experimental setup the same described in Appendix A.4.I The results were all averaged over 5 random seeds for the neural network parameter default initialization. The plots have no smoothing applied to the learning curves.
Top row: We swept over a learning rate range {0.0025, 0.005, 0.01, 0.01, 0.025, 0.05, 0.1} with a constant batch size of 512. Middle row: We swept over batch size range of {8, 16, 32, 64, 128, 256, 512, 1024] with a constant learning rate of 0.005. Bottom row: We swept over 1.2 regularization rate range of (0, 0.00001, 0.0001, 0.00025, 0.0005) with learning rate 0.02 and batch size 512.
19

---

Louniig 1itc
0.0095
05
Tasn Georanne 82200
0.em
Etaation
6.0090
0.5
0.04
Th GeTTee Callzie
254
1801
40000
100006
0.090
4000
00:00
100300
00
40005
00300
100000
0.0040
11943
0025
0.0002
0.00000
0.00255
003
6.0015
Taration
Figure 7: Controlling the neural collapse via model geometric complexity for VGG- trained on MNIST. Lower GC produces lower geometric collapse and more neural collapse (i.e., lower NC) for Top row: increased learning rates, Middle row: decreased batch sizes, and Bottom row: increased L2 regularization.
20

---

A.5.3 Fashion-MNIST on VGG-11
we trained a VGG-11 neural network [56] on Fashion-MNIST. The experimental setup is the same as described in Appendix [A.4.] The results were all averaged over 5 random seeds for the neural network parameter default initialization. The plots have no smoothing applied to the learning curves.
In Figure 8
Top row: We swept over a learning rate range of {0.005, 0.01, 0.025, 0.05, 0.2) with constant batch size of 256. Bottom row: We swept over L2 regularization rate range of {0, 0.00001, 0.0001, 0.00025, 0.0005} with learning rate 0.02 and batch size 512.
1ih Seawent Seeeenn C3lee
Rats
0005
Ievotion
Reuuton
01011
6.0000
Taa Geamette Calgsie
000
6C3003
40000 10000 1C0006
4002
10300
100306
000
2003
4395
40300
500500
Figure 8: Controlling the neural collapse via the model geometric complexity for VGG-11 11 trained on Fashion-MNIST. Lower GC produces lower geometric collapse and more neural collapse (i.e., lower NC) for Top row: increased learning rates, Bottom row: increased L2 regularization.
A.5.4 Cifar-100 on ResNet50
In Figure 9 trained ResNet-50 neural network CIFAR-100. The experimental setup is the same as described Appendix A...I The results were all averaged over 5 random seeds for the neural network parameter default initialization. The plots have no smoothing applied to the learning curves.
Top row: We swept over a learning rate range of f0.005, 0.01, 0.05, 0.1, with con- stant batch size of 256. Bottom row: We swept over L2 regularization rate range of (0.0, 0.0001, 0.00025, 0.0005, 0.00075) with learning rate 0.02 and batch size 256.
21

---

Leanting Tate
100000
60000
Ioraaiio
Isaratioo
mostion
0.05
2005
3.2
Figure 9: Controlling the neural collapse via the model geometric complexity for ResNet-50 trained on CIFAR-100 100. Lower GC produces lower geometric collapse and more neural collapse (i.e., lower NC) for Top row: increased learning rates, Bottom row: increased L2 regularization.
A.5.5
Lower Pre-trained GC Leads to Improved Fine-tuning: mini-ImageNe dataset VGG architecture
In Figure I0, we trained a VGG16 neural network (as described in [57]) implemented in Flax on Mini- Mini-lmaeeNet (as described in (60]). We used 10 classes with 600 examples per class for training, with 10/90 train/test split. To evaluate the downstream performance we used 100 downstream tasks consisting of 5 labels for few shot learning randomly chosen from pool of 20 separate from the training labels. The experimental setup was the same as described section (A.4.. Namely, we reported 1) the geometric complexity of the embedding layer measured on the source dataset (Source GC), 2) the neural collapse as measured by (3) measured and averaged on the 100 random samples of 5-label samples from the target dataset, 3) the test accuracy obtained by solving the normal equation directly for a ridge regression with only 5 examples per class obtained from the target set. These three quantities were measured every 1000 steps of very long pre-training of 100,000 steps. The optimizer was plain SGD (note we obtained similar results with momentum) without any regularization nor schedule avoid masking effects. We used random crop and random flip for data augmentation. The results were all averaged over 5 random seeds for the neural network parameter default initialization. We swept over a learning rate range of (0.001, 0.005, 0.01} with a constant batch size of 256. Each sweep took roughly 10h of training on single Google Cloud TPU V3 accessed via Google colab.
0.34
Leaming
400
0.12
0.003
0.775
2.200
borget NC
0.24
0.50
0.22
0.225
0.50
42000 0
20000
50000
20000
100000
20000
400C0
000CO
B0OOD
300000
83000 0
20200
40000
60000
100000
Baration
Figure 10: Controlling target NC through source GC on mini-imagenet with VGG-16: Increased learning rates produces lower GC and more neural collapse (i.e. lower NC) resulting in higher 5-shot transfer accuracy.
22

---

A.5.6 Direct GC regularization increases neural collapse
To mitigate possible confounding factors due to batch size and learning rate manipulations, we train a VGG-13 model [57] on CIFAR-10 10 with explicit GC regularization using fixed learning rate of 0.01 and batch size of 256. We explore increasing levels of explicit GC regularization with rates 10 6, 10 10-3 10- and plot the results Figure 11 We observe the same predictions as in our experiments which indirectly lower the GC through learning rates, batch sizes and L2 regularization. Namely, lower GC produces lower NC values (i.e., increased neural collapse). Note that in this experiment we regularized w.r.t. the GC computed at the logit layer rather than the embedding GC. This is because the high dimension of the embedding layer makes taking the gradient of the embedding GC prohibitively expensive. Already, regularization with GC taken at the logit layer was possible only for one seed. That being said, explicit regularization by GC computed at the logit layer provides a direct way to control GC computed at the embedding layer.
Segu-rrtaaann Fate
0.003
0.65
2932
40000
10000
100800
0.66
20300
6000
Donuttion
ROOD
30009
20:00
SCE30
1908
19065
Figure 11: VGG-13 on CIFAR-10 with explicit GC regularization (one seed).
A.6 Discussion on the Poincaré inequality
Recall that distribution Q satisfies a Poincaré inequality if, for all differentiable functions defined on the support of Q, there exists constant e > 0 such that
(ar,(Q) cEz~q [I] |
(18)
In this section, we argue that the Poincare Inequality (PI) is a mild and natural assumption to make. For instance, the Gaussian distribution, mixtures of Gaussians with equal (or different) variances, mixtures of Gaussians and -abuGausssnss, mixtures of uniform and Gaussian measures; and any log-concave probability measure all satisfy Pl; see, for example, 54). The same is true for most distributions of importance in probability and statistics; see [7].
The Poincaré constant itself can take very high values depending on the spread of the distribution. For example, for a standard normal distribution the Poincare constant is I while for a multivariate normal distribution the Poincare constant equals the largest eigenvalue of its covariance matrix which can be any positive real number. Intuitively speaking, the Poincaré constant will increase if the space is stretched in any direction and decrease if it is compressed in any direction.
The PI has also been assumed hold for real life image datasets, where it has been used help improve GAN convergence, as in [3 It is also a key assumption to understand the role of over- parameterization in generalization as happens for large neural networks (cf. which frames this as an equivalent isoperimetry condition).
On the contrary, non-PI distributions are considered pathological; they can be constructed for instance by artificially concatenating distributions with fully disjoint support. See [42] for a construction of pathological examples of distributions satisfying Poincaré inequality.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction accurately state the paper's contributions, as- sumptions, and limitations. The claims about how geometric complexity influences neural collapse align with the results, reflecting the paper's scope. The abstract clearly indicates the generalizability of the proposal based on the datasets used.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The experiments in this paper focus on image processing and involve large number of important hyperparameters and extensive runs. The paper clearly states that the generalization effect during transfer learning tested only on image processing datasets, such as CIFAR-10, 10, CIFAR- 100, MNIST, Fashion-MNIST and CIFAR-FS, mini mini-ImageNet in the transfer learning setting.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Yes, the paper provides the full set of assumptions and a complete, correct proof for each theoretical result. Each assumption is clearly stated, and the proofs meticulously detailed, ensuring the validity and rigor of the theoretical contributions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relics upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Yes, the paper fully discloses all the information needed to reproduce the main experimental results, ensuring that the main claims and conclusions can be independently verified. This includes detailed descriptions of the methodology, data sources and all relevant parameters and all hyperparameters.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
25

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: Yes, while the code is not open source, the paper open datasets, which are well-known benchmark datasets for image processing, such as CIFAR-1O, CIFAR- 100, MNIST, Fashion-MNIST and CIFAR-FS and mini-lmageNer for transfer learning. The supplemental material includes detailed instructions to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurlPS code and data submission guidelines (https / //nips.com lllipssccc/ubblccguiicee httppp/pll.l/g...eee////ebebbbb/
to
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, the paper specifics all the training and test details, including data splits, hyperparameters, their selection process, and the type of optimizer, ensuring comprehensive understanding of the results. The Appendix includes full experiment details for all main paper plots well as for all plots appearing in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
26

---

Answer: [Yes]
Justification: Yes, the paper reports eтrоr bars that are suitably and correctly defined, based on multiple runs for each training. It provides the standard deviation for each metric, such accuracy, geometric complexity, and neural collapse, ensuring clear understanding of the statistical significance of the experiments. All experiments are repeated over 5 random seeds and error bars are included in the Figures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Yes, the paper provides sufficient information on the computer resources required for each experiment, including the type of compute workers, and execution time, ensuring that the experiments can be accurately contextualized for performance.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httttp:://eurips.cooo http://pp.....bbii/cc/ccccceeeea
Answer: [Yes]
Justification: Yes, it does.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
27

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There are no societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeett of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
There are no risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
28

---

Answer: [NA]
Justification: No use of third-party code beyond machine learning frameworks like JAX and TensorFlow, and common Python libraries for experimental purposes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
No crowdsourcing research performed with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
29

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing research performed with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
30


Reviews:
[{"review_id": "tdQjOA465j", "reviewer": "Reviewer_5tN3", "summary": "The paper explores the relationship between neural collapse (NC) and geometric complexity (GC). It presents both theoretical and empirical evidence showing that geometric complexity is a robust metric across various variables. By substituting the NC metric with GC, the paper introduces a generalization bound based on geometric complexity. Furthermore, it highlights the significance of GC in transfer learning, demonstrating that by controlling the pre-training GC, downstream NC can also be controlled, leading to improved transfer learning results.", "questions": "* In equation (7), does the distance $d$ represent the norm distance? This should be stated clearly.\n* Both the paper and the prior work (Galanti et al, 2022) rely on the data assumption where the source and transfer datasets come from the same distribution $Q$. However, in practical settings, source and transfer datasets typically have domain gaps which can potentially violate the data assumption. Would the results in the paper still hold in this case? For example, pre-training on ImageNet/Cifar-100 while fine-tuning on DTD?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is overall well-written and easy to follow, the interpretation of each theoretical results is clear and intuitive.\n* The relationship between NC and GC is quite interesting.\n* Most of the empirical results align well with the theoretical part, demonstrating the validity of the results.", "weaknesses": "Some empirical results can be better aligned with the theoretical counterpart and additional experiments could be added to better support the claims. For example:\n* In proposition 4.1, NC is upper bounded by geometric collapse up to some constant scaling. What's the range of this constant $c$, does it purely depend on $Q$? If yes, it would be better to directly plot NC and RHS in (6) in the same plot to demonstrate the validity of the bound; and if no, it would be interesting to show how different settings examined in Figure 1 affect $c$.\n* The empirical evidence presented in Figure 2 clearly demonstrates the robustness of geometric complexity. However, I have some concerns regarding the Lipschitz notion. Neural networks are known to be not so Lipschitz, and making them more Lipschitz is an active area of research aimed primarily at improving robustness. Is Proposition 4.2 numerically verifiable by calculating the Lipschitz constant of the tested neural networks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. We are very happy that you found the work “well-written” and “easy to follow”, making the interpretation of the theoretical results “clear” and “intuitive”. We are also glad that you found the relation between “GC” and “NC” to be “quite interesting” and that the “empirical results align well with the theoretical part”.\n\nBelow you’ll find below our best effort to answer your questions. \n\nPlease consider raising your score if you feel your concerns have been resolved.\n\n\n>*\"In proposition 4.1, NC is upper bounded by geometric collapse up to some constant scaling. What's the range of this constant c, does it purely depend on Q.  If yes, it would be better to directly plot NC and RHS in (6) in the same plot to demonstrate the validity of the bound; and if no, it would be interesting to show how different settings examined in Figure 1 affect c.\"*\n\nYes, the Poincare constant depends exclusively on the distribution Q. Stacking the plots in Figure 1 as you are suggesting is a great idea, which we will add in our paper. On these plots, by comparing the magnitude RHS and LHS of (6), we can roughly see that the c ~ 1000 (lower bound) for CIFAR-10 in our setting. The Poincare constant can take very high values depending on the spread of the distribution. For instance, for a standard normal distribution the Poincare constant is 1 while for a multivariate normal distribution the Poincare constant equals the largest eigenvalue of its covariance matrix which can be any positive real number. Intuitively speaking, the Poincaré constant will increase if the space is stretched in any direction and decrease if it is compressed in any direction.\n\n>*\"The empirical evidence presented in Figure 2 clearly demonstrates the robustness of geometric complexity. However, I have some concerns regarding the Lipschitz notion. Neural networks are known to be not so Lipschitz, and making them more Lipschitz is an active area of research aimed primarily at improving robustness. Is Proposition 4.2 numerically verifiable by calculating the Lipschitz constant of the tested neural networks?\"*\n\nThe numerical experiment in Figure 2 (plot 1) shows that in practice the empirical GC estimate may be approaching the true value much faster than the bound specifies. Namely after 10 samples only, we already obtain a value very close to what we obtain with 50 times more samples, which is much better than a 1/sqrt(#sample) dependence, even ignoring the Lipschitz constant. This makes us believe that a much tighter bound here should be possible. We agree with you that this bound may be a looser bound than reality. \n\n>*\"In equation (7), does the distance d represent the norm distance? This should be stated clearly.\"*\n\nThank you for pointing this out. We will clarify that point in the paper. “d_ij” represents the euclidean distance between the mean for class i and the mean for class j. \n\n>*\"Both the paper and the prior work (Galanti et al, 2022) rely on the data assumption where the source and transfer datasets come from the same distribution Q. However, in practical settings, source and transfer datasets typically have domain gaps which can potentially violate the data assumption. Would the results in the paper still hold in this case? For example, pre-training on ImageNet/Cifar-100 while fine-tuning on DTD?\"*\n\nThis is a very good point. In theory any source and target distribution can always be concatenated together mathematically into a single larger distribution. However, as you are pointing out, issues may arise when “gaps” are created by this process. Namely, when the support of the source and target distribution do not overlap, this process results in an overall distribution with non-connected support. This particular configuration is a setting ripe for violations of the Poincare inequality even if the source and target distribution both satisfy it. \n\nIf this happens, then the range of validity of our bounds as well as our predictions concerning the impact of the learning rate, batchsize, regularization on the neural collapse and transfer performance may not hold.\n\nHowever, instead of seeing this as a limitation, we interpret it as a feature that may help us understand what compatibility conditions are necessary for a model pre-trained on a source distribution to transfer well to a target distribution. Namely, in line 286, we give compatibility conditions for the transfer bound in Prop'n 5.1 to be meaningful. This condition in particular implies that no such gaps are created by concatenating the source and target distributions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pp60Ezdgoy", "reviewer": "Reviewer_N73X", "summary": "This paper examines the relationship between geometric complexity (GC), neural collapse (NC), and transfer learning performance in deep neural networks. The key contributions are:\n\n* Proposing geometric complexity as a measure that connects the flatness of the loss surface and neural collapse\n* Deriving theoretical bounds showing GC controls NC under certain assumptions\n* Empirically demonstrating that mechanisms affecting GC (learning rate, batch size, regularization) also influence NC\n* Showing lower GC during pre-training leads to better transfer learning, especially for few-shot tasks\n* Proving a new generalization bound in terms of GC\n* Demonstrating GC can be efficiently estimated on small samples\n\nThe authors argue that GC can serve as an informative proxy for transfer learning potential and provide a unifying framework for understanding implicit biases in deep learning. By bridging concepts from optimization geometry, representation learning, and transfer learning, the paper aims to provide deeper insights into the fundamental mechanisms driving the success of modern deep learning techniques.", "questions": "1. How does geometric complexity compare to other measures of model/representation complexity like Fisher information, gradient norm, etc.? A more thorough comparison would strengthen the paper.\n2. The theoretical results rely on strong assumptions like the Poincaré inequality. Can you provide empirical evidence that these assumptions hold for real datasets/models? Or derive results under weaker assumptions?\n3. The paper shows correlations between GC, NC, and transfer performance but does not conclusively prove causation. Have you considered alternative explanations or confounding factors?\n4. How computationally expensive is it to measure GC during training? Is it feasible to use it as a regularization term or early stopping criterion in practice?\n5. The generalization bound in terms of GC is interesting, but how tight is it empirically? How does it compare to other generalization bounds in the literature?\n6. Have you explored using insights about GC to guide architecture design or improve training algorithms? Demonstrating practical benefits would strengthen the paper.", "limitations": "The authors acknowledge some limitations, like the focus on image classification tasks and relatively small datasets. However, they could go further in discussing potential shortcomings:\n\n* The strong assumptions required for the theoretical results may limit their applicability.\n* The computational cost of measuring GC is not thoroughly addressed.\n* Potential negative consequences of optimizing for low GC are not explored.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Originality**: The paper presents a novel perspective connecting several concepts in deep learning theory. Its original contribution is using geometric complexity as a bridge between loss surface geometry and neural collapse. This approach provides a fresh lens through which to view the implicit biases in deep learning, potentially unifying several strands of research. Using GC as a progress metric for transfer learning is innovative and could open up new avenues for improving pre-training techniques.\n\n**Quality**: The empirical evaluation thoroughly examines multiple datasets, architectures, and hyperparameter settings. The authors have made a commendable effort to validate their hypotheses across various conditions, including different network architectures (ResNet, VGG), datasets (CIFAR, MNIST, Fashion-MNIST), and transfer learning scenarios. The ablation studies exploring the impact of learning rate, batch size, and regularization on GC and NC provide valuable insights into the dynamics of deep network training. While limited in scope, the theoretical results give some formal grounding for the empirical observations and offer a starting point for further analytical work in this area.\n\n**Clarity**: The paper is well-structured and communicates the main ideas. The authors have presented a complex set of ideas in a logical flow, starting from the theoretical foundations and moving to empirical validation. The figures illustrate key results, particularly the relationships between GC, NC, and transfer performance. The use of color-coding and consistent formatting across figures aids in interpretation. The pseudo-code and detailed experimental setup descriptions in the appendix are beneficial for reproducibility.\n\n**Significance**: If the claims hold up, this work could provide valuable insights into the mechanisms behind transfer learning and self-supervised learning. The proposed geometric complexity measure may help analyze and improve deep learning models. The potential impact extends beyond just theoretical understanding. If GC can serve as a reliable proxy for transfer learning potential, it could guide the development of more effective pre-training strategies and architectures. Furthermore, the connection between GC and NC could help explain why certain training practices (like large batch training or specific learning rate schedules) are effective, potentially leading to more principled approaches to hyperparameter tuning.", "weaknesses": "**Limited theoretical foundation**: While the paper provides some theoretical results, the assumptions required (e.g., Poincaré inequality) are quite strong and may not hold in practice for real datasets and architectures. The connection between the linear model analysis and deep neural networks is not strongly justified. This is a significant limitation, as it's unclear how much of the theory translates to practical scenarios. The authors could strengthen this aspect by providing empirical evidence that these assumptions hold in realistic settings or deriving weaker results under more general conditions. Additionally, the paper could benefit from a more thorough discussion of the implications and limitations of these theoretical results.\n\n**Experimental limitations**: The empirical evaluation focuses on image classification tasks and relatively small datasets. The generalizability of the results to larger datasets, other domains (e.g., NLP), or more complex architectures is unclear. This narrow focus raises questions about the broader applicability of the findings. For instance, it's unclear whether the relationships between GC, NC, and transfer performance would hold for large-scale pre-training scenarios like those used in modern language models. The paper would be significantly strengthened by including experiments in a broader range of tasks and scales or at least by providing a more detailed discussion of the potential challenges in scaling up the approach.\n\n**Lack of comparison to related metrics**: The paper does not thoroughly compare geometric complexity to other related complexity measures or generalization bounds from the literature. This makes it difficult to assess the relative merits of GC. There's a rich body of work on complexity measures for neural networks, including various forms of capacity measures, stability-based bounds, and PAC-Bayesian approaches. A comprehensive comparison would help situate GC within this broader context and clarify its unique contributions. Such a comparison could also help identify scenarios where GC might be particularly advantageous or potentially suboptimal compared to existing approaches.\n\n**Causality vs correlation**: While the paper shows correlations between GC, NC, and transfer performance, it does not conclusively demonstrate a causal relationship. Alternative explanations for the observed trends have not been thoroughly explored. This is a critical limitation, as it leaves the possibility that GC is merely a proxy for some other underlying factor driving NC and transfer performance. The paper would be strengthened by a more rigorous causal analysis, perhaps through interventional studies or by controlling for potential confounding variables. Additionally, discussing potential mechanisms by which GC might causally influence NC and transfer learning would add depth to the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful review. We are very happy that you found our approach to be a “novel perspective” as well as an “original contribution” and that it “could open up new avenues for improving pre-training techniques” and “provide valuable insights into the mechanisms behind transfer learning and self-supervised learning”. We are also glad that you found the “empirical evaluation” to be “thoroughly” done and that the paper was “well-structured” with “ideas communicated clearly”.\n\nBelow are our answers to your concerns. They have been very helpful in further strengthening the paper. \n\nWe ask that you please consider raising your score if your concerns have been resolved.\n\n>*\"The connection between the linear models and DNNs is not justified\"*\n\nThere is no mention of linear models in our paper. Can you please reference the paragraph creating confusion so that we can clarify?\n\n>*\"Limited theoretical foundation: [..] the assumptions required (e.g., Poincaré inequality) are quite strong [..] the authors could strengthen this by providing evidence that these assumptions hold in realistic settings\"*\n\nThis is a good point but we respectfully disagree with the idea that the Poincare Inequality (PI)  is an overly strong or unrealistic assumption. We agree our paper would be strengthened by clarifying this point. Here, we include a short summary of key facts and can provide a more detailed exposition in the Appendix.\n\nNote that the Gaussian distribution; mixtures of Gaussians with equal (or different) variances; mixtures of Gaussians and sub-Gaussians; mixtures of uniform and Gaussian measures; and any log-concave probability measure all satisfy a PI ([1, 2] below). The same is true for most distributions of importance in probability and statistics ([3] below).\n\nThe PI has also been assumed to hold for real life image datasets, where it has been used to help improve GAN convergence ([4] below). It is also a key assumption to understand the role of over-parameterization in generalization as happens for large NNs ([5] below).\n\nOn the contrary, non-PI distributions are considered pathological; they can be constructed for instance by artificially concatenating distributions with fully disjoint support. The only restrictive assumption for the model is that it's differentiable wrt the input, which is a widely assumed property in the literature.\n\nFurthermore, we examine various real-life datasets like CIFAR10, CIFAR100, and ImageNet as well as common architectures like VGG and ResNet.\n\n[1] Bakry et al. A simple proof of the Poincaré inequality for a large class of probability measures, '08\n\n[2] Schlichting, Poincaré and log-Sobolev inequalities for mixtures, '19\n\n[3] Pillaud-Vivien et al, Statistical Estimation of the Poincaré constant and Applications..., AISTATS '20\n\n[4] Khrulkov et al, Functional Space Analysis of Local GAN Convergence, ICML '21\n\n[5] Bubeck et al, A universal law of robustness via isoperimetry, NeurIPS '21\n\n>*\"Experimental limitations: [..] The generalizability of the results to other domains (eg, LLMs) is unclear [..]\"*\n\nThis is good point. Thanks for raising it. We will definitely clarify this aspect in our paper. There is a limitation of our work when it comes to extending to LLMs. However, this limitation is not specific to our work per-se but it is a general limitation of the application of NC to LLMs (see Wu & Papyan, Linguistic Collapse: Neural Collapse in (Large) Language Models, ‘24 which outlines limitations).\n\nThe main problem is that for LLMs the embedding dimension is lower in general than the number of classes (vocabulary size), making the NC simplex impossible to exist. Extending the notion of NC to LLMs is an open question beyond our scope.\n\n>*\"How does GC compare to other measures [..]\"*\n\nThis is indeed interesting. However, this comparison has been done already; e.g. [17] introduces GC and thoroughly relates it to the gradient norm & [41] compares with other complexity measures. The relation with the Fisher information and the gradient norm has also been studied (Jastrzebski et al., Catastrophic Fisher Explosion, ICML '21). These works indicate that the GC as a measure in function space is a similar notion of complexity to the gradient norm or the Fisher information in parameter space.\n\nWe are happy to add an Appendix section on this.\n\n>*\"The paper shows correlations with GC [..] but not [..] causation\"*\n\nWe added an experiment (rebuttal:Fig. 3) that directly controls GC by explicitly regularizing it, mitigating confounding factors. We observe the same predictions: lower GC produces lower NC. Note, here we regularized w.r.t. the GC computed at the logit layer rather than the embedding GC. This is because the high dimensionality of the embedding layer makes taking the gradient of the embedding GC very prohibitively expensive.\n\n>*\"How computationally expensive is it to measure GC [..]? Is it feasible to use as a regularization term?\"*\n\nWe have not yet tried to use GC as an early stopping criterion but this could be a good idea. Measuring GC during training is relatively cheap, easy, and robust, especially if one samples it as explained in Section 4.2, see also Fig. 2. We have explored using GC as a regularization term (see above & rebuttal:Fig. 3) as have others, eg [17].\n\n>*\"The generalization bound in terms of GC is interesting, but how tight is it empirically?\"*\n\nWe added an experiment (rebuttal:Fig. 2) where we plotted the LHS and RHS (excepting the Lipschitz term) of this bound Eq. 10 in Prop 4.3, which we will add to our paper.\nOn that plot the bound is not vacuous and quite tight! A full comparison with other generalization bounds is beyond our scope. Our main goals are showing that 1) the mechanisms that control GC also control NC and 2) solutions with lower GC are more performant for transfer learning.\n\n>*\"Have you explored using GC insights to guide models or algorithms?\"*\n\nThis is a very interesting but beyond the scope of the current paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hc7wCGgoIu", "reviewer": "Reviewer_pQCH", "summary": "The paper studies the relationship between geometric complexity (based on penultimate layer features) and the neural collapse phenomenon (especially, the variability collapse property). The main theoretical result is a bound on the CDNV-based NC metric using geometric complexity (Proposition 4.1). This is then employed to extend generalization and transfer learning bounds (proposed by previous efforts) to rely on geometric complexity. Experimental results on classification tasks (along with transfer learning settings) are presented to validate the proposed relationship.", "questions": "1. Since the geometric complexity-based generalization/transfer learning bounds are the upper bounds of existing neural collapse-based bounds, what can be a potentially new implication of your results? since it is common practice to do a hyper-parameter search over learning rates, batch sizes, and regularization parameter values. \n\n2. The paper does not discuss any explicit regularization mechanism that aims to reduce the geometric complexity. What are the potential challenges in designing such a regularization mechanism?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The unified analysis of neural collapse and geometric complexity presents an interesting line of research for the community. Especially, in terms of studying seemingly disjoint phenomena under a common lens.", "weaknesses": "1. There seems to be a mismatch in the $\\log(2/\\delta)$ term in Proposition 4.2 vs $\\log(1/\\delta)$ term in Proposition 4.3, which makes the main results inconsistent (yet fixable). Additionally, why do we have an \"expectation\" over the \"error\" term in the LHS of Eq (10)? Isn't the \"error\" itself an expectation?\n\n2. Although the paper presents generalization bounds, a discussion on the generalization performance of the classifier (VGG-13 for Figure 1) seems to be missing. Only the neural collapse and geometric complexity plots are illustrated but a deeper relationship between the trends and the test performance is lacking. Is there an upper limit of \"good\" learning rates, and batch sizes above which the geometric complexity can reduce but the test performance can get worse? \n\n3. Similar to the above point, currently it seems like increasing the learning rate, increasing batch size, and increasing the regularization parameter value results in lower neural collapse values. Ideally, neural collapse is applicable during the terminal phase of training (TPT) where the training accuracy is $1$. Does this mean that all the experiments (at least for Figure 1) lead to TPT during training?\n\nnit: Please fix the notation for the norms as it is unclear if $||.||$ represents $||.||_2$ or $||.||_F$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We are very happy that you found that the “unified analysis of neural collapse and geometric complexity presents an interesting line of research for the community [...] in terms of studying seemingly disjoint phenomena under a common lens.”  We have addressed your comments to the best of our ability below. \n\nPlease consider raising your score if your concerns have been resolved.\n\nWe'd also like to respectfully note that your concerns seem to focus around only minor aspects of our work that are easily fixable (i.e., clarification, notation, typos, addition of plots). Could you please help us understand which of your concerns prompted you to a \"confident reject” so that we can address it?\n\n> *\"There seems to be a mismatch in the log⁡(2/𝛿) term in Proposition 4.2 vs log⁡(1/𝛿) term in Proposition 4.3, which makes the main results inconsistent (yet fixable). Additionally, why do we have an \"expectation\" over the \"error\" term in the LHS of Eq (10)? Isn't the \"error\"  itself an expectation?\"*\n\nThank you for noticing the typo: there should be $\\log(2/\\delta)$ instead of $\\log(1/\\delta)$ in Prop. 4.3. As for the second expectation in the LHS of Eq. (10) it is taken over the sample S used to produce the classifier $h_{f,S}$ on line 230  (similarly to  [22, Proposition 5]). We will correct the typo and make the second point clearer in the paper.\n\n> *\"Although the paper presents generalization bounds, a discussion on the generalization performance of the classifier (VGG-13 for Figure 1) seems to be missing. Only the neural collapse and geometric complexity plots are illustrated but a deeper relationship between the trends and the test performance is lacking.\"*\n\nWe included plots for the test accuracy in the rebuttal document (see Figure 1, 3rd column). As referenced in our paper, it has already been observed in [22] that lower levels of NC are indicative of higher test accuracy. Similarly, in [17] it has been shown that lower levels of GC also are characteristic of higher test accuracy. We will add a remark to this effect in the paper and include similar plots for our experiments in the appendix.\n\n> *\"Is there an upper limit of \"good\" learning rates, and batch sizes above which the geometric complexity can reduce but the test performance can get worse?\"*\n\nYes, you are correct. As for any form of regularization (implicit or explicit), test performance degrades beyond a certain level of the regularization rate, while the regularized quantity keeps decreasing at the expanse of the original objective.\n\n>*\"Similar to the above point, currently it seems like increasing the learning rate, increasing batch size, and increasing the regularization parameter value results in lower neural collapse values. Ideally, neural collapse is applicable during the terminal phase of training (TPT) where the training accuracy is 1. Does this mean that all the experiments (at least for Figure 1) lead to TPT during training?\"*\n\nYes, that is correct and a good clarification. All our experiments lead to training accuracy 1. We included these plots for our main experiments in the attached rebuttal document (see Figure 1, 1st and 2nd columns) and will add similar plots for our additional experiments to the appendix.\n\n> *\"Please fix the notation for the norms as it is unclear if $||\\cdot||$ represents $||\\cdot||_2$ or $||\\cdot||_𝐹$.\"*\n\nThank you. We will clarify the norm notation in the paper as you suggest.\n\n> *\"Since the geometric complexity-based generalization/transfer learning bounds are the upper bounds of existing neural collapse-based bounds, what can be a potentially new implication of your results? since it is common practice to do a hyper-parameter search over learning rates, batch sizes, and regularization parameter values.\"*\n\nTo our knowledge, there are no known theoretical mechanisms which control neural collapse. Our bound in Prop. 4.1 shows that the same mechanisms that control GC also control NC. This is significant because (see [17] from the paper references) there are a number of experimental mechanisms which control the GC and furthermore a strong theoretical justification as to why [17, Thm 5.1]. The current work provides a novel justification that these same mechanisms also control NC. This implication is theoretically motivated via our Prop 4.1 and experimentally validated. \n\nA second new implication that can be derived from our bound in Prop 5.1 is that pretrained networks with lower GC have better transfer learning performance on new tasks. These 2 new implications are indeed the key theoretical findings in our paper. Devising practical schemes based on these theoretical findings is beyond the scope of the current paper, yet opens a new avenue for future work.\n\n> *\"The paper does not discuss any explicit regularization mechanism that aims to reduce the geometric complexity. What are the potential challenges in designing such a regularization mechanism.\"*\n\nThis is a good point. While direct regularization by GC computed at the logit layer has been previously studied (for instance in [17]) and found beneficial, direct regularization by GC computed at the embedding layer becomes impractical because of the much higher dimension of the embedding layer, resulting in a very large computational cost. \n\nThat being said, explicit regularization by GC computed at the logit layer provides a direct way to control GC computed at the embedding layer. We added this experiment (see rebuttal document Fig. 3) and observed the predicted effect on the embedding GC (rebuttal Fig 3. left and middle) and on neural collapse (rebuttal Fig 3. right). We will add a paragraph to our paper about the challenge of directly regularizing for embedding GC.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for your thoughtful and thorough reviews. We are grateful that you found the paper "well-written" and "well-structured" and found our "novel perspective" to be an "original contribution" which you think “could open up new avenues for improving pre-training techniques” and “provide valuable insights into the mechanisms behind transfer learning and self-supervised learning”. This is great to hear and much appreciated! Furthermore, your thoughtful comments and review have helped us to substantially improve and strengthen the current work.

The overall goal of this paper is to connect two previously unconnected areas of research in machine learning and leverage that insight to better understand their role in successful transfer learning. We achieve this goal in two ways:

First, we clearly demonstrate the relationship between neural network geometric complexity and embedding neural-collapse (Section 4). We show through theory and verify empirically that mechanisms which regularize the GC in turn put measurable pressure on the neural collapse (Section 4.1: Prop 4.3 and Fig. 1). 

A second goal is to show that pre-trained networks with lower GC promote lower NC values on new unseen target classes which enable improved transfer accuracy during fine-tuning (Section 5: Prop 5.1 and Fig. 3).

We tried our very best below to satisfy your requests and answer your questions within the time frame imparted. 

**Please find attached a 1-page document containing figures referenced in our responses to individual reviewers comments.**

Please consider raising your scores in consequence if your requests have been met and questions answered.

Meta-review: This paper studies the relation between metrics of neural collapse (NC) and geometric complexity (GC).
They prove that NC can be bounded in terms of GC under certain assumptions, and they give a generalization bound depending on GC.
They also empirically demonstrate that GC and NC tend to be correlated under common training interventions (learning rate, etc).
Overall, this paper presents a promising case for using GC in understanding neural collapse.

The reviewer scores were borderline. Reviewers agreed that the relation between GC and NC was interesting and novel. The main concerns were about the lack of comprehensive experimental support, and lack of sufficient discussion of limitations.

I recommend acceptance, since reviewers found the ideas interesting and found no technical flaws.
I strongly suggest the authors incorporate reviewer feedback in their camera-ready. Specifically, please add explicit discussion of both the theoretical and the experimental limitations. Be especially careful in making causal claims without specifying your causal graph. Overall, adding such discussion will clarify the scope of your work, and thus make the paper stronger.
Decision: Accept (poster)