Title: The Impact of Geometric Complexity on Neural Collapse in Transfer Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: transfer learning

geometric complexity

neural collapse

implicit bias

flatness

generalization bounds
Abstract: Many of the recent advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this empirical success is incomplete and remains an active area of research. Flatness of the loss surface and neural collapse have recently emerged as useful pre-training metrics which shed light on the implicit biases underlying pre-training. In this paper, we explore the geometric complexity of a model's learned representations as a fundamental mechanism that relates these two concepts. We show through experiments and theory that mechanisms which affect the geometric complexity of the pre-trained network also influence the neural collapse. Furthermore, we show how this effect of the geometric complexity generalizes to the neural collapse of new classes as well, thus encouraging better performance on downstream tasks, particularly in the few-shot setting.

The Impact of Geometric Complexity on Neural Collapse in Transfer Learning

Abstract

Many of the recent remarkable advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this empirical success is incomplete and remains an active area of research. Flatness of the loss surface and neural collapse have recently emerged as useful pre-training metrics which shed light on the implicit biases underlying pre-training. In this paper, we explore the geometric complexity of a model's learned representations as a fundamental mechanism that relates these two concepts. We show through experiments and theory that mechanisms which affect the geometric complexity of the pre-trained network also influence the neural collapse. Furthermore, we show how this effect of the geometric complexity generalizes to the neural collapse of new classes as well, thus encouraging better performance on downstream tasks, particularly in the few-shot setting.

1 Introduction

Many of the recent remarkable advances in modern machine learning owe their success in part to transfer learning [4, 6, 9, 15, 28, 49, 50, 66]. While there are different approaches to this technique, the standard one involves two stages. In a first stage, called pre-training, ones trains a deep neural network on a general, large-scale dataset in the form of a supervised or unsupervised source task; e.g., ImageNet or CIFAR-100 [14, 33] for image models or the Common Crawl, C4 or LM1B datasets [8, 11, 49] for language models. In the second stage, one then leverages portions of the pre-trained network to use as features map or embeddings that can then be adapted, or fine-tuned, on a more specific target task. Often these target tasks are unknown at the time of pre-training and labeled data may be very scarce, such as in the context of few-shot learning [34, 51, 60]. However, despite these limitations, this approach often results in a fine-tuned model that achieves quite impressive performance substantially better than training on the target task alone and requiring less computational resources [20]. Despite this empirical success, a comprehensive theoretical understanding of the mechanisms underlying this effectiveness of transfer learning is not fully understood and remains an active area of research [63].

One interesting line of research suggests that the effectiveness of transfer learning is due to the implicit biases encoded in pre-trained models [22, 36, 48]. These implicit biases effectively constrain the hypothesis space, guiding the model towards solutions that have a preference for smoother functions [12, 17, 38, 40, 53], simpler geometry in the loss surface [3, 13, 18, 55, 58] or reduced complexity of the internal learned representations [23, 32]. In the same way that these implicit biases have been used to help explain the success of deep learning, recent work has shown that the notion of neural collapse [22, 35, 62] and flatness of the loss surface [36] can also inform the mechanisms behind

transfer learning. This suggests that these preferences during pre-training also act as a form of prior knowledge which is highly transferable to downstream tasks, even with limited task-specific data.

In this paper, we present a novel perspective that further sheds light on these mechanisms and implicit biases hidden within transfer learning. Our approach analyzes the geometric complexity [16, 17] of the internal representations learned by the deep neural networks during pre-training and provides a complementary theoretical framework which unifies previous work examining the role of neural collapse and loss surface flatness in transfer learning [22, 36]. In particular, we show through experiments and theory that the geometric complexity of the pre-trained network directly controls the neural collapse of the pre-trained model and thus its efficacy in transfer learning, particularly in the few-shot setting. We argue that geometric complexity (similarly to flatness of the loss surface and neural collapse) can be used as hidden progress metrics, cf. [2], for transfer learning, serving as an informative proxy toward the transfer power of a pre-trained network.

Our primary contributions are the following:

We uncover relationships between learning-path flatness, neural network geometriccomplexity, and embedding neural-collapse providing a framework to understand how these implicit biases interact (Section 4).

We show through theory that the geometric complexity (GC) of a neural network controls neural collapse and verify this empirically by showing how mechanisms which regularize the GC in turn put pressure on the neural collapse (Section 4.1 and Fig. 1).

We demonstrate both theoretically and empirically that pre-trained networks with lower GC promote lower neural collapse on new unseen target classes, and thus enable improved transfer accuracy during fine-tuning (Section 5 and Fig. 4).

We prove a new generalization bound in terms of geometric complexity (Section 4.3).

We show that the empirical GC can be accurately and efficiently estimated on a small number of samples, input coordinates, and output features making it computationally tractable compared to other progress metrics in machine learning (Section 4.2 and Fig. 2).

Notation. Throughout, we denote by ‖ · ‖ the L2 Euclidean norm and by ‖ · ‖ F the Frobenius norm.

2 Background and Related Work

The implicit biases introduced by an optimization algorithm play a crucial role in deep learning and in the generalization ability of the learned models [43, 44, 55, 64]. They help ensure that the model not only finds a solution with low error but also one with low complexity which generalizes well [26, 67]. Uncovering the mechanisms of these implicit regularizers is crucial for understanding how the model learns, both as a means to improve generalization and as valuable leverage for designing more efficient algorithms and decreasing costly experiment iteration cycles.

Here we focus on three seemingly different themes behind implicit regularization in deep learning and explain how they are related: the loss surface slope measuring the flatness of the learning path [3, 29, 45], the geometric complexity of the learned model function measuring its variability with respect to a dataset [16, 42], and the neural collapse [32, 46] measuring how a neural network efficiently clusters its learned representations of the input class examples in embedding space.

Flatness in parameter space. In parameter space, the learning dynamics is fully characterized by the discrete optimization path θ t where t ∈ N . At each step, one can compute the flatness of the learning path as the slope of the tangent space at θ t to the loss surface L = { ( θ, L ( θ )) : θ ∈ R n } . As shown in [3, Appendix A.2], this slope coincides with the loss-gradient square norm: slope( θ t ) = ‖∇ L ( θ t ) ‖ 2 . This quantity has been used to bound a generalization gap in [25] and as a beneficial explicit regularizer in [3, 24, 58] indicating that learning curves with lower slope values tend to also have better test performance.

Moreover, many standard optimizers and common training heuristics have been shown to put an implicit pressure on the learning-path flatness, making it an implicit bias of the training procedure [3, 5, 10, 13, 17, 25, 39, 58]. Related to the learning curve slope is the optima sharpness

where θ ∗ is a global minima toward which the learning dynamics converges to θ t → θ ∗ and H is the Hessian of the loss. Since the sharpness of an optima corresponds to the mean curvature of the loss surface at that point, flat minima (i.e., minima with low curvature in all directions) can be reached only through learning paths with shallower slopes.

Neural collapse in embedding space. Neural collapse [22, 27, 46] refers to a phenomenon observed in deep learning where the embedding network (i.e., the subnetwork f before the logit layer g in a neural network h ( x ) = softmax( g ( f ( x )) ) collapses the input points around their respective class means. Furthermore, these class means form a simplex creating an equiangular tight frame (ETF) centered around the global mean with roughly equal distance between its vertices. Intuitively, such a phenomenon is beneficial to generalization since the embeddings of different classes are optimally separated, making the job of the classifier head softmax( g ( z )) easier. To measure neural collapse, the authors in [22] introduce the class-distance normalized variance ( CDNV ) as

where Q r = q r ( x ) dx is the input distribution for classes r ∈ { i, j } and where

Neural collapse between two classes happens when their class variance decreases while the distance between their class means increases, causing lower values of their CDNV. Following [22], given a well-balanced input distribution Q = 1 k ( Q 1 + · · · + Q k ) with k classes ( Q i being the input distribution for class i ), neural collapse (NC) during training is characterized by the following limit as the number of training steps t goes to infinity:

glyph[negationslash]

and where f t is the learned embedding network at step t . Thus, more neural collapse (i.e., more clustering around better separated class-means) happens with lower NC values. Proposition 5 from [22] proves that NC( f, Q ) bounds a generalization gap and lower values of NC( Q,f ) are correlated with better test performance of the neural network f , and [37] shows that explicitly regularizing for neural collapse is beneficial.

Geometric complexity in function space: The geometric complexity (GC) of a function f : R d → R k w.r.t. a data distribution Q = q ( x ) dx on R d is defined as the expectation of the gradient Frobenius square-norm w.r.t. to the data distribution [17] given by

Intuitively, the GC measures the function complexity or variability and is closely related to the Dirichlet energy in geometric analysis [21]. Provided mild Lipschitz smoothness assumptions [42, Proposition 3.4], the GC can be estimated accurately on batches of training data D by its empirical counterpart

Previous work has explored the relationship between the GC and model generalization. When measured on the full neural network, lower GC values correlate experimentally with higher test accuracy [45] and it has also been used as a beneficial explicit regularizer [30, 59]. In [16, 17], the authors ignore the softmax activation and study the GC measured with respect to the logit network, exploring its connection with various implicit and explicit regularizers and training heuristics. The logit GC has also be used to prove a margin based multi-class generalization bound [42], assuming that the input distribution Q satisfies a mild assumption known as the Poincaré inequality [1].

In fact, both this work and the proof of the generalization bound rely on the Poincaré inequality, which we argue is indeed a natural and mild assumption for the types of data distributions typically encountered in machine learning; see Appendix A.6 for further discussion. For completeness, we recall it here, cf. [19]. A distribution Q satisfies a Poincaré inequality if, for all differentiable functions v defined on supp( Q ) , there exists a constant c such that

Note that the same assumption is used in [7] to prove the law of robustness for overparameterized neural networks via isoperimetric inequalities. In this paper, we peel back yet another layer of the network and consider the embedding geometric complexity measured on a feature map (Section 3).

Relationship between flatness, neural collapse, and geometric complexity. The learning-path flatness as measured by its slope influences the geometric complexity of the learned solutions [16, Theorem 5.1]. Namely, for dense layers a regularizing pressure on the slope of the learning path in parameter space transfers to a regularizing pressure on the geometric complexity of the learned solution in function space. A similar result also been shown for special attention layers as well [52].

In this paper, we will see that imposing a regularization pressure on the embedding geometric complexity encourages more neural collapse of the model. This results in the following chain of influences from regularization pressure:

learning path flatness glyph[squiggleright] function geometric complexity glyph[squiggleright] embedding neural collapse Regularizing Pressure Regularizing Pressure Lower CDNV

3 Problem Formulation

We are interested in understanding the relationship between the geometric complexity (GC) and neural collapse (NC) in the transfer learning setting and explore this relationship in two stages. In the first stage, we examine the general impact of the GC on NC during model training/pre-training. Secondly, we examine how this relationship provides insight into the mechanisms behind transfer learning and the advantageous implicit biases of the pre-training stage. In short, lower GC during pre-training on source classes leads to lower NC for target classes and improved transfer accuracy.

For the first stage of our inquiry (Section 4), we are concerned with a k -classification task. Let D := { ( x i , y i ) } i m =1 be a dataset drawn from a distribution Q with x i ∈ R d and y i ∈ { e j | j = 1 , . . . , k } where e j ∈ R k are the canonical basis vectors representing a one-hot encoding of the labels. We aim to learn this task using a neural network denoted by a function h θ : R d → R k parameterized by θ (though to simplify notation we subsequently drop this dependence on θ ).

We can write h θ ( x ) = softmax( g ( f ( x )) where g : R p → R k is a classifier head mapping from the feature space R p to the prediction layer R k and f : R d → R p is a feature mapping from the input space to the penultimate embedding layer of the network before the classification head. The standard way of training h θ is to find, via some stochastic optimization technique, an optimal parameter configuration θ ∗ such that θ ∗ = argmin θ ∑ m i =1 glyph[lscript] ( h θ ( x i ) , y i ) for a given loss function glyph[lscript] : R k × R k → [0 , ∞ ) .

To analyze the role of neural collapse and geometric complexity in this setting, we focus our attention on the feature map f : R d → R p . The NC as described in Section 2 is defined using this sub-network map f of h and thus we also measure the geometric complexity with respect to this sub-network feature mapping as well. Throughout this paper, we use the term embedding GC to refer to the GC of such a feature map f and unless otherwise specified, when we refer to the model GC we mean the embedding GC, not the GC of the logit network as in [16, 17].

Our second stage of inquiry (Section 5) is focused on the role of geometric complexity in transfer learning. For this setting, we assume there is some l -class classification target task we would like to solve and corresponding target distribution T . We aim to learn a classifier h ′ : R d → R l given some training data D ′ := { ( x ′ i , y ′ i ) } m ′ i =1 which is potentially very limited in size. In order to find a good solution, we can leverage a pre-trained feature mapping that has been trained on some other source task and source distribution S where more data is available. For example, by leveraging a feature map f pre-trained as in the above setup, the target task classifier h ′ can instead be trained on the outputs of our feature map; i.e., { f ( x ′ i ) , y ′ i ) } m ′ i =1 . In this way, we can write h ′ = softmax( g ′ ◦ f ) where the parameters of f are fixed and we only need to learn the parameters of g ′ : R p → R l . Ideally, provided f is a rich enough feature map trained on a general enough source distribution dataset, then the task for learning g ′ is much simpler.

4 Geometric complexity and neural collapse

In this section, we explore the general relationship between the embedding GC and neural collapse showing that the geometric complexity can be used to bound neural collapse. Next, we see how the geometric complexity can be efficiently and accurately estimated, both as an approximation of the theoretical GC as well as through a number of sampling techniques for the empirical GC via batch sampling, feature sampling and label sampling. Lastly, following [22], we derive a new generalization bound for neural networks expressed in terms of the embedding geometric complexity; cf. [42].

4.1 Geometric complexity controls neural collapse

Previously, it has been shown [16] that the GC can be controlled implicitly through choice of learning rate and batch size, as well as through standard explicit regularization. Practically, this means that these same beneficial tuning strategies which ensure that models have low GC should also work to keep the NC low as well. The following proposition (which we prove in Appendix A.1) bounds the neural collapse by the geometric complexity provided that the input distribution satisfies a Poincaré inequality also assumed in [42] and [7].

Proposition 4.1. Suppose that we have a balanced multi-class input-distribution Q with k classes satisfying the Poincaré inequality in (5) for some constant c , then the geometric complexity of a network embedding f bounds its neural collapse as measured by (3) ; namely, we have the following bound glyph[negationslash]

where k is the number of classes, and d ij is the distance between the mean of class i and class j .

We call the RHS of the bound in Eq. (6), excepting the Poincaré constant c , the geometric collapse :

glyph[negationslash]

where k is the number of classes, and d ij = ‖ µ f ( Q i ) -µ f ( Q j ) ‖ denotes the Euclidean distance between the mean µ f ( Q i ) and µ f ( Q j ) of class i and class j (resp.) in embedding space.

This quantity can be seen as another proxy metric measuring neural collapse. It is made of two main parts decoupling the variances and mean differences present in the neural collapse framework. The variances are consolidated into a single factor through the GC, while the mean differences are averaged across classes in a separate factor. This separation allows the overall intra-class variability to be influenced through the GC term, ensuring sufficient between-class separation through the mechanisms identified in neural collapse.

This bound provides a powerful method to control the overall within-class variability via the L2-norm of the model gradient. As a result, the geometric collapse provides a simplified and more refined approach to managing class separability and variance in deep learning models.

We verify the relationship posed in Eq. (6) through experiments on VGG-13 trained on CIFAR-10 (see Figure 1). Through both implicit and explicit regularizers the pressure on the embedding GC translates to a direct pressure on the neural collapse of the model via the geometric collapse. As already observed in [17], lower levels of GC coincide with higher test accuracy as show in Figure 5 in Appendix A.5 containing the learning curves of that experiment.

In Appendix A.5 we see the same relationship holds across various datasets; e.g., MNIST, FashionMNIST, CIFAR-100, and architectures; e.g., ResNet-18, ResNet-50. To avoid possible confounding factors that could arise through batch size and learning rate manipulations, we also directly regularize with the geometric complexity in Appendix A.5.6; the same relations hold in this case too.

4.2 The geometric complexity is a reliable and robust measure

One of the key advantages of the GC as a complexity measure is its computational efficiency.

0.006

Learning Rate

0.14

0.001

0.005

0.0025

0.005

0.01

0.025

0.02

0.001

1

0.00

0

20000

40000

Iteration

60000

80000

100000

0.000

0

20000

40000

Iteration

60000

80000

100000

0

0

20000

40000

Iteration

60000

80000

100000

0.10

0.0025

3.0

8

0.0020

18

2.5

Collapse

64

128

0.0015

256

uje

0.0005

0.5

0.00

0

20000

40000

60000

80000

100000

0.0000

0

20000

40000

60000

80000

100000

0.0

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

0.00200

3.0

Regularization Rate

0.08

0.00175

0.0

0.00025

2.5

0.0005

0.001

0.0025

0.00050

0.5

0.00025

0.00

0

20000

40000

60000

80000

100000

0.00000

0

20000

40000

60000

80000

100000

0.0

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

Under mild regularity constraints on the model function the theoretical geometric complexity of a map can be efficiently estimated by its empirical counterpart, as stated in the proposition below, already proven in [42] for logit networks (see Appendix A.2 for a proof). Additionally and crucially, we verify that the empirical GC is a robust and reliable measure with respect to that data sample.

Proposition 4.2. Let f : R d → R p be a map with Lipschitz constant L and let D X be a sample of m elements drawn independently from an input distribution Q . Then, for any δ > 0 , we have with probability 1 -δ/ 2 the following bound

For a function f and data sample D X as in Proposition 4.2, to measure ̂ GC( f, D X ) requires computing the Frobenius norm of a potentially very large Jacobian matrix, particularly when p represents the embedding dimension of a feature map. Note that,

Although the computational complexity of the GC is impervious to increasing complexity of the network architecture, e.g., in terms of parameter count or number of layers, one may run into computation bottlenecks as the sample size m increases or when increasing the dimensionality d (resp. p ) of the inputs (resp. outputs). In these scenarios, it is necessary to find efficient means to accurately approximate or sample the Jacobian; cf. [61].

We explore the robustness of the GC when measured via samplings along these three axes; i.e., decreasing the number of examples m in the batch, randomly sampling the full Jacobian matrix which has order d × p , and randomly sampling the number of model outputs p . As shown Figure 2, we find that the value of the sampled ̂ GC remains stable and consistent to its true value through these fairly simple and naive sampling tricks.

Randomly Masked Jacobian

Average Geometric Complexity (n=20)

(ResNet18 trained on Cifar100)

Average Geometric Complexity (n=20)

(ResNet18 trained on Cifar100)

(0=u)

Randomly Sampled Output Dimension

(ResNet18 trained on Cifar100)

5

Complexity(

0.005

4

4

0.004

m

Average Geometric

0.003

2

2

0.002

1

1

0.001

1

510

Number of examples in batch

25

50100250

500

0

0.9

0.99

Mask Probability

(batch size = 512)

0.999

0.9999

0.99999

1.0

0.000

0.1

Proportion of Embedding Dimension

0.05

(batch size = 250)

0.1

0.5

0.75

1.0

4.3 A new generalization bound with GC through NC

In [42], the authors derive a margin-based multi-class generalization bound for neural networks which depends on the geometric complexity GC( h, Q ) of the full logit network. In this section, we extend this result. With inspiration from [22, Proposition 5], we show that the geometric complexity measured on the sub-network feature map GC( f, Q ) bounds the classification error of the nearestmean classifier defined by the feature map f .

As in [22], given a balanced k -class classification problem, let S = ⋃ c ∈ [ k ] S c denote all class samples and define the nearest-mean classifier by h f,S ( x ) := argmin c ∈ [ k ] ‖ f ( x ) -µ f ( S c ) ‖ given by the feature map f where µ f ( S c ) denotes the sample mean of the set S c under the map f . Define the generalization error glyph[negationslash]

where I [ h f,S ( x ) = y ] is the indicator function of the error set [ h f,S ( x ) = y ] and P is the full data distribution from which samples S are drawn.

glyph[negationslash]

glyph[negationslash]

Proposition 4.3. Suppose that we have a balanced sample S from a k -class input-distribution ( x, y ) ∼ P with m c samples per class. Assume further that the induced input distribution x ∼ Q satisfies a Poincaré inequality as in (5) for some constant c . Then, for any δ > 0 , with probability 1 -δ/ 2 we have the following bound for the generalization error glyph[negationslash]

where p is the embedding dimension of the feature map f and d ij = ‖ µ f ( S i ) -µ f ( S j ) ‖ . Note, the outer expectation on the left hand side is taken over the samples S used to produce the classifier h f,S .

Proof. This follows immediately from [22, Proposition 5] which gives the bound in terms of the neural collapse instead of the geometric complexity. By using Proposition A.1 we can replace the neural collapse with the geometric complexity and using Proposition 4.2 we then replace the theoretical GC with the empirical GC, yielding the additional term with the Poincaré constant.

In Figure 3, we plot the LHS and RHS of the generalization bound in (10) from Proposition 4.3 for a VGG-13 model trained on CIFAR-10. In this plot, we see that the bound is not vacuous, demonstrating a relatively tight fit. Note that on the RHS, we omitted the term involving the Lipschitz constant L , assuming it is small due to its denominator. Additionally, we estimated a lower bound for the Poincaré constant c by comparing the magnitudes of the RHS and LHS in the inequality (6), based on results in Figure 1. This approximation yielded c ≈ 1000 in our setup.

Generalization Bound (c.f.Prop'n 4.3)

1.0

Test NearestMean Classifier Error

0.8

0.6

0.4

0.2

0.0

0

20000

40000

60000

80000

100000

Iteration

5 The impact of geometric complexity on transfer learning

Many implicit regularization mechanisms in gradient descent exert a pressure on geometric complexity which in turn constrain the neural collapse. In this section, we discuss the affect of this implicit bias in transfer learning. Specifically, we show with theory how this bias during pre-training can help explain the mechanisms behind transfer learning. Namely, that lower GC in a pre-trained embedding network promotes neural collapse on new target classes, simplifying the fine-tuning process.

However, our bound makes it clear that certain compatibility conditions between the pre-training (source) distribution and the fine-tuning (target) distribution are needed, which we argue are satisfied in image foundational models and large language models. At last, we verify empirically that known methods to reduce geometric complexity for the pre-trained embedding result in better performance during fine-tuning on unseen tasks, in agreement with our neural collapse bound for transfer learning.

5.1 Lower pre-trained GC leads to improved fine-tuning

To analyze the impact of GC in the context of transfer learning, we consider the same formal setup as in [22, Proposition 2]. Assume a class c in a set C of classes is represented by a class-conditional distribution Q c ( x ) = P ( x | y = c ) . Both the pre-training/source classes ˜ Q 1 , . . . , ˜ Q k and the finetuning/target classes Q 1 , . . . , Q l are assumed to be drawn from a distribution over all classes in C . The induced input distribution on all the classes (i.e., the combined source and target input distribution) is denoted Q , while the source-only input distribution is denoted by ˜ Q . Let F ∗ denote the set of pre-trained feature maps f : R d → R p selected by the training procedure (e.g., the set of trained embeddings with different initialization seeds but the same training protocol) and consider glyph[negationslash]

With this, we can state the transfer learning bound

Proposition 5.1. Suppose that the source and target input distributions satisfy a Poincaré inequality with constant c ˜ Q and c Q (resp.). Then, with probability 1 -δ , over the selection Q c and Q c ′ of target class distributions, we have that the CDNV expectation for two target classes is bounded by the geometric complexity of the embedding network f in the following way glyph[negationslash]

glyph[negationslash]

where k is the number of source classes, GC( F ∗ , Q c ) = sup f ∈F ∗ GC( f, Q c ) , d ij = ‖ µ f ( Q c i ) -µ f ( Q c j ) ‖ , and H ( F ∗ , ˜ Q ) is a complexity measure for F ∗ over ˜ Q (see Appendix A.3).

Proof. This follows immediately from [22, Proposition 2] which proves this bound for the NC and distribution variances. Using our Proposition 4.1, we can replace the NC by the GC and swap the variances by the corresponding geometric complexities via the respective Poincaré inequality.

Interpretation. In the bound above, the first term depends on the geometric complexity measured over the source distribution ˜ Q and decreases as the number of source classes k increases. Thus, we can predict that lower GC( f, ˜ Q ) encourages lower NC values (i.e. more neural collapse) on the new target classes provided that the second and third term of the bound are also small.

However these other two terms involve both the source and target classes, making the full bound no longer dependent only on the geometric complexity over ˜ Q .

Moreover these two terms can apriori explode since the infimum over the class-mean distances ∆( F ∗ ) no longer only involves source classes (where this distance is bounded away from zero due to neural collapse into an equidistant simplex) but also target classes (for which we no longer have theoretical guarantees).

However, if the source labels are granular enough in the sense the target labels can be represented as combinations of the source labels (as for instance a target label of 'dog' can be subsumed as all the breeds of dogs in source classes on ImageNet), then the issues above are mitigated. Formally, we can summarize this granularity compatibility condition between the source and target classes as follows: For each label c of the target class there exists labels c 1 , . . . , c k in the source classes such that Q c = 1 /k ( Q c 1 + · · · + Q c k ) . Because GC( f, · ) is linear, we have

and thus recover dependence only on source classes.

To address the ∆( F ∗ ) term, observe that the compatibility condition above implies in terms of classmeans that µ f ( Q c ) = 1 k ( µ f ( Q c 1 ) + · · · + µ f ( Q c k )) . Because of neural collapse during pre-training, these source class-means form the vertices of a face in the neural-collapse class-mean simplex making the target-class mean µ f ( Q c ) the barycenter of this face. Since the distance between any two points taken in the vertices of a simplex plus its barycenters is bounded away from zero, so is ∆( F ∗ ) . Note, albeit intuitive and natural, this granularity condition is hard to verify in practice.

5.2 Improve fine-tuning by controlling pre-trained GC

When the compatibility conditions above are satisfied, Proposition 5.1 indicates that increasing the amount of neural collapse for the target classes can be achieved by lowering the embedding GC during pre-training. We verify this indeed occurs experimentally using the same regularization techniques exploited in Section 4. Furthermore, we verify that these implicit methods of controlling pre-training GC produce feature maps that perform better on fine-tuning tasks on CIFAR-FS with a ResNet-18 in Figure 4 and on mini-ImageNet with VGG-16 in Appendix A.5.5.

6 Limitations and Conclusion

There is a notable limitation when extending our findings to language modeling and, for example, large language models (LLMs). However, this limitation is not specific to our work per-se, but instead it is a limitation of the application of neural collapse to language models in general, as described in the recent work [65]. Namely, language modeling, as conducted via training by token prediction, creates a classification task where the conditions for neural collapse are implausible. The main problem, in addition to an imbalanced vocabulary, is that the embedding dimension for language models is typically far less than the number of classes (i.e., the total vocabulary size), making the neural collapse simplex impossible to exist. Extending the notion of neural collapse to the language models is an open question and an active area of research and beyond the scope of this work.

0.0175

Learning Rate

8.0

0.001

0.40

0.0150

0.005

7.5

0.01

0.0125

0.05

0.1

0.35

Source

0.0100

NC

7.0

0.25

0.5

0.0075

6.0

0.0050

5.5

0.25

0.0025

5.0

0.0000

0.20

4.5

0

20000

4000060000

80000

100000

0

20000

40000

60000

80000

100000

0

20000

4000060000

80000

100000

Iteration

Iteration

Iteration

0.014

8.0

8.0

0.40

0.012

16.0

7.5

32.0

0.010

64.0

128.0

0.008

0.006

6.5

0.004

6.0

0.25

0.002

0.000

5.5

0.20

0

20000

40000

60000

80000

100000

0

20000

40000

60000

80000

100000

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

Regularization Rate

0.008

8.0

0.0001

0.0

0.40

7.5

0.0005

0.006

7.0

3

Source

0.004

6.0

0.002

5.5

0.25

5.0

0.000

0.20

0

20000

40000

60000

80000

100000

0

20000

40000

60000

80000

100000

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

Uncovering and understanding the implicit biases that enable successful transfer learning is a critical area of research in modern machine learning. Here we provide a framework connecting three different themes behind implicit regularization: flatness of the loss surface, geometric complexity, and neural collapse. We show that the embedding geometric complexity directly controls the neural collapse during training and, moreover, plays a role in the success of transfer learning.

Extensive experiments on different image classification and fine tuning tasks across different hyperparameters verify our hypothesis. We believe this opens up an intriguing direction of research further exploring the role of geometric collapse and geometric complexity in deep learning and provides valuable insight for designing more efficient and effective techniques for model training and fine-tuning.

Acknowledgments and Disclosure of Funding

We would like to thank Patrick Cole for their support. We would also like to thank Hanna Mazzawi and Peter Bartlett for helpful conversations.

References

D. Bakry, I. Gentil, and M. Ledoux. A simple proof of the poincaré inequality for a large class of probability measures. Electronic Communications in Probability , 13:391-401, 2008.

Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. Advances in Neural Information Processing Systems , 35:21750-21764, 2022.

David G.T. Barrett and Benoit Dherin. Implicit gradient regularization. In International Conference on Learning Representations , 2021.

Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Proceedings of ICML workshop on unsupervised and transfer learning , pages 17-36. JMLR Workshop and Conference Proceedings, 2012.

Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. In Annual Conference Computational Learning Theory , 2019.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Sébastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. Journal of the ACM , 70(2):1-18, 2023.

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pages 2633-2650, 2021.

Rich Caruana. Learning many related tasks at the same time with backpropagation. Advances in neural information processing systems , 7, 1994.

Matias Cattaneo, Jason Klusowski, and Boris Shigida. On the implicit bias of adam. arXiv:2309.00079 , 2023.

Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005 , 2013.

Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics , pages 192-204. PMLR, 2015.

Alex Damian, Tengyu Ma, and Jason D. Lee. Label noise SGD provably prefers flat global minimizers. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, NeurIPS 2021 , 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Benoit Dherin, Micheal Munn, and David GT Barrett. The geometric occam's razor implicit in deep learning. NeurIPS, OPT2021 , 2021.

Benoit Dherin, Michael Munn, Mihaela Rosca, and David Barrett. Why neural networks find simple solutions: the many regularizers of geometric complexity. Advances in Neural Information Processing Systems , 35:2333-2349, 2022.

Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning , pages 1019-1028. PMLR, 2017.

Lawrence C Evans. Partial differential equations , volume 19. American Mathematical Society, 2022.

Abolfazl Farahani, Behrouz Pourshojae, Khaled Rasheed, and Hamid R. Arabnia. A concise review of transfer learning. CoRR , abs/2104.02144, 2021. URL https://arxiv.org/abs/ 2104.02144 .

Herbert Federer. Geometric measure theory . Springer, 2014.

Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. In ICLR , 2022.

Peifeng Gao, Qianqian Xu, Yibo Yang, Peisong Wen, Huiyang Shao, Zhiyong Yang, Bernard Ghanem, and Qingming Huang. Towards demystifying the generalization behaviors when neural collapse emerges. arXiv preprint arXiv:2310.08358 , 2023.

Jonas Geiping, Micah Goldblum, Phil Pope, Michael Moeller, and Tom Goldstein. Stochastic training is not necessary for generalization. In ICLR , 2022.

Avrajit Ghosh, He Lyu, Xitong Zhang, and Rongrong Wang. Implicit regularization in heavy-ball momentum accelerated stochastic gradient descent. ICLR , 2023.

Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems , 31, 2018.

X.Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamics on the central path. In ICLR , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation , 9(1):1-42, 1997.

Judy Hoffman, Daniel A. Roberts, and Sho Yaida. Robust learning with jacobian regularization. arXiv , 2020. URL https://arxiv.org/abs/1908.02729 .

Valentin Khrulkov, Artem Babenko, and Ivan Oseledets. Functional space analysis of local gan convergence. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , 2021.

Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.04041 , 2022.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10657-10665, 2019.

Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, and Qing Qu. Principled and efficient transfer learning of deep models via neural collapse. arXiv preprint arXiv:2212.12206 , 2022.

Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. In ICML , 2023.

Xuantong Liu, Jianfeng Zhang, Tianyang Hu, He Cao, Yuan Yao, and Lujia Pan. Inducing neural collapse in deep long-tailed learning. In AISTATS , 2023.

Chao Ma and Lexing Ying. The sobolev regularization effect of stochastic gradient descent. 2021. URL https://arxiv.org/abs/2105.13462 .

Chao Ma and Lexing Ying. On linear stability of sgd and input-smoothness of neural networks. In NeurIPS , 2021.

Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research , 22(165):1-73, 2021.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Michael Munn, Benoit Dherin, and Javier Gonzalvo. A margin-based multiclass generalization bound via geometric complexity. ICML, TAGML Workshop , 2023.

Behnam Neyshabur. Implicit regularization in deep learning. arXiv preprint arXiv:1709.01953 , 2017.

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614 , 2014.

Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha SohlDickstein. Sensitivity and generalization in neural networks: an empirical study. In ICLR , 2018.

Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117 (40), 2020.

Loucas Pillaud-Vivien, Francis Bach, Tony Lelièvre, Alessandro Rudi, and Gabriel Stoltz. Statistical estimation of the poincaré constant and application to sampling multimodal distributions. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 2753-2763, 2020.

Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 , 2022.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning , pages 8821-8831. Pmlr, 2021.

Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International conference on learning representations , 2016.

Liliang Ren, Yang Liu, Shuohang Wang, Yichong Xu, Chenguang Zhu, and ChengXiang Zhai. Sparse modular activation for efficient sequence modeling. In NeurIPS , 2023.

Mihaela Rosca, Theophane Weber, Arthur Gretton, and Shakir Mohamed. A case for new neural network smoothness constraints. In Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops , volume 137 of Proceedings of Machine Learning Research , 2020.

André Schlichting. Poincaré and log-sobolev inequalities for mixtures. Entropy , 21(1):89, 2019.

Keskar Nitish Shirish, Mudigere Dheevatsa, Nocedal Jorge, Smelyanskiy Mikhail, and Tang Ping Tak Peter. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 , 2016.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR , 2015.

Samuel L Smith, Benoit Dherin, David G.T. Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. In International Conference on Learning Representations , 2021.

Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel R. D. Rodrigues. Robust large margin deep neural networks. IEEE Transactions on Signal Processing , 65:4265-4280, 2017.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems , 29, 2016.

Guanhua Wang and Jeffrey A Fessler. Efficient approximation of jacobian matrices involving a non-uniform fast fourier transform (nufft). IEEE transactions on computational imaging , 9: 43-54, 2023.

Zijian Wang, Yadan Luo, Liang Zheng, Zi Huang, and Mahsa Baktashmotlagh. How far pretrained models are from neural collapse on the target dataset informs their transferability. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5549-5558, 2023.

Karl Weiss, Taghi Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big Data , 3, 05 2016. doi: 10.1186/s40537-016-0043-6.

Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. Advances in neural information processing systems , 30, 2017.

Robert Wu and Vardan Papyan. Linguistic collapse: Neural collapse in (large) language models, 2024.

Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems , 27, 2014.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3): 107-115, 2021.

A Appendix

A.1 Proof of Proposition 3.2 (Neural collapse bound)

Proof. Suppose that we have an input probability distribution Q ( x ) = q ( x ) dx coming from a data distribution P ( X,Y ) (i.e. q ( x ) dx is the marginal probability distribution of p ( x, y ) dxdy ) where Y represents the k possible classes. Suppose further that P ( X,Y ) is a balanced multi-class distribution. In terms of the input distribution Q ( x ) this means that

where Q i ( x ) = q i ( x ) dx is the input distribution of class i . Suppose moreover that Q ( x ) satisfies the Poincaré inequality in (5), that is,

for some constant c . Then the statement we want to prove is that the geometric complexity of f , that is,

bounds its neural collapse as measured by glyph[negationslash]

glyph[negationslash]

More precisely, we want to prove that we have the following inequality:

glyph[negationslash]

where d ij = ‖ µ f ( Q i ) -µ f ( Q j ) ‖ is the distance between the mean of class i and class j . Let us now prove this statement.

First of all, since the geometric complexity respects convex sums of data densities, we have that for a distribution Q = 1 k ( Q 1 + · · · Q k ) we can write

In particular, this means that GC( f, Q i ) ≤ k GC( f, Q ) . Furthermore, the Poincare inequality ensures that Var Q i ( f ) ≤ c GC( f, Q i ) . Using these two properties and the definition of the neural collapse, we obtain that:

glyph[negationslash]

glyph[negationslash]

which simplifies to the desired result.

Remark A.1. Excepting the Poincaré constant, the quantity on the RHS above given by glyph[negationslash]

could be taken as an alternative measure of neural collapse. We call this the geometric collapse .

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

A.2 Proof of Proposition 4.2 (Estimating the Theoretical GC by ̂ GC )

Proof. Let Q be an (input) probability distribution defined over R d and let D = { x i } i m =1 of m ≥ 1 points drawn as i.i.d. samples from Q . Given the map f : R d → R p , we denote the empirical geometric complexity of f over D by ̂ GC( f, D ) which is defined as

This follows by direct computation, keeping in mind that Q is a probability distribution and that the points are independently sampled. We can write Q as Q ( x ) = q ( x ) dx and note that

Now suppose we have two separate independent samples D and D ′ each of of size m ≥ 1 which differ by exactly one point, say x i in D and x ′ i in D ′ . By assumption, the map f is L -Lipschitz for some Lipschitz constnt L > 0 . Therefore, we have

and similarly, ̂ GC( f, D ′ ) -̂ GC( f, D ) ≤ L 2 /m . It follows that | ̂ GC( f, D ) -̂ GC( f, D ′ ) | ≤ L 2 /m and by applying McDiarmind's inequality (e.g., [41]), we have that for any glyph[epsilon1] > 0 ,

Since, from the computation above, E D ∼ µ m [ ̂ GC( f, D )] = GC( f, Q ) and setting δ/ 2 = exp( -2 mglyph[epsilon1] 2 /L 2 ) and substituting for glyph[epsilon1] in (16), we get that for any δ > 0 with probability as least 1 -δ/ 2 the following holds:

We begin by showing that

This completes the proof.

A.3 Definition of the complexity measure H ( F , Q ) in Proposition 4.1

Let us restate here for the sake of completeness the definition given in Galanti et al. [22]. First consider the Rademacher complexity R ( A ) of a set A ⊂ R n which is given as the expectation E glyph[epsilon1] ( X ) of the random variable X = sup a ∈ A 〈 glyph[epsilon1],a 〉 where glyph[epsilon1] = ( glyph[epsilon1] 1 , . . . , glyph[epsilon1] n ) is a random vector with components uniformly distributed among the two values -1 and 1. Now consider the input distribution Q coming from a multiclass distribution with labels in c ∈ C with label input distribution Q c . We define the complexity measure H ( F , Q ) for the candidate functions in F over the source distribution Q as the Rademacher complexity of the following set:

A.4 Experiment details

A.4.1 Figure 1: Geometric Complexity Controls Neural Collapse on CIFAR-10.

We trained a VGG-13 neural network on the full CIFAR-10 dataset with the provided architecture [56] and using the standard train/test split. Throughout training, we reported the following metrics measured and averaged over multiple batches of the training dataset: 1) the geometric complexity of the model embedding layer; i.e., the layer on which the neural collapse is measured, 2) the neural collapse as measured by (3) measured on the model embedding layer, and 3) the geometric collapse of the model also measured on the penultimate embedding layer of the model and given by glyph[negationslash]

where Q is the training distribution, k = 10 as this was CIFAR-10, and (as in the paper) where d ij = ‖ µ f ( Q i ) -µ f ( Q j ) ‖ is the distance between the sample means of class i and class j .

These quantities were measured every 1000 steps of a very long pre-training of 100 , 000 steps. The optimizer was plain SGD (note we obtained similar results with momentum) without any regularization nor schedule to avoid masking effects. We used random crop and random flip for data augmentation. The results were all averaged over 5 random seeds for the neural network parameter default initialization. The plots have no smoothing applied to the learning curves. Top row: We swept over a learning rate range of { 0 . 001 , 0 . 0025 , 0 . 005 , 0 . 01 , 0 . 025 , 0 . 1 } with a constant batch size of 512. Middle row: We swept over a batch size range of { 8 , 16 , 32 , 64 , 128 , 256 } with a constant learning rate of 0.01. Bottom row: We swept over a L2 regularization rate range of { 0 . 0 , 0 . 00025 , 0 . 0005 , 0 . 001 , 0 . 0025 } with learning rate 0.01 and batch size 256. Each sweep took roughly 10h of training on a single Google Cloud TPU V3 accessed via a Google colab.

In Figure 5, we show the complete learning curves of that experiment.

A.4.2 Figure 3: Tightness of the generalization bound on CIFAR-10

We trained a VGG-13 neural network [56] on the full CIFAR-10 dataset using the provided train/test split. For this model architecture we use an embedding layer dimension p = 1024 . Note also that the number of examples per class is m c = 5000 . We used a constant learning rate of 0 . 005 , a batch size of 512 and trained for 100000 steps reporting metrics every 1000 steps.

glyph[negationslash]

Throughout training, we reported the following metrics measured and averaged over multiple batches of the training dataset: 1) the empirical geometric complexity of the model embedding layer; i.e., the layer on which the neural collapse is measured, denoted ̂ GC , and 2) the sum of the inverse squares of d ij which denotes the distance between the mean of class i and class j for i, j ∈ [ k ] for i = j and, here, k = 10 . These quantities were used to create the blue curve in Figure 3. We also computed the nearest mean classifier error on the test set. Recall, for the feature map f and a sample S the nearest mean classifier is defined as h f,S := argmin c ∈ [ k ] ‖ f ( x ) -µ f ( S c ) ‖ where, here k = 10 , and µ f ( S c ) denotes the sample mean of the set S c for class c under the feature map f . To create the orange curve in Figure 3 we plot the average error h f,S ( x ) = y over the test set.

glyph[negationslash]

0.5

Learning Rate

1.00

0.80

0.4

0.005

0.01

0.001

0.0025

0.95

0.90

0.78

0.75

0.12

0.025

0.09

0.2

0.07

0.70

0.65

0.1

0.65

0.63

0.05

0.0

0

20000

40000

60000

80000

0.60

0

20000

4000060000

80000

0.60

20000

4000060000

80000

0.04

20000

4000060000

80000

Iteration

Iteration

Iteration

Iteration

0.7

1.00

0.80

0.10

0.6

16 2 4

0.95

0.75

0.5

128

0.70

256

Train Geometric

0.04

0.2

0.80

0.60

0.02

0.1

0.75

0.55

0°0

0.70

0.50

0.00

0

20000

4000060000

Iteration

80000

0

20000

4000060000

Iteration

80000

0

20000

4000060000

Iteration

80000

0

20000

4000060000

Iteration

80000

0.7

Regularization Rate

1.00

0.80

0.10

0.6

0.5

0.0005

0.001

0.0

0.00025

0.95

0.78

0.75

0.0025

0.2

0.80

0.65

0.1

0.75

0.63

0.02

0.0

0

20000

4000060000

00000T00008

0.70

20000

4000060000

80000

100000

0.60

0

20000

4000060000

80000100000

0

20000

40000

60000

80000100000

Iteration

Iteration

Iteration

Iteration

A.4.3 Figure 4: Controlling target performance through source GC on CIFAR-FS.

We trained a RestNet-18 neural network with width 1 implemented in Flax https://github. com/google/flax/blob/main/examples/imagenet/models.py on CIFAR-FS with its initial convolution modified to stride 1 and kernel size of 3 to adapt to CIFAR-FS instead of ImageNet. We used only 10 classes for the source datasets with 600 images per class, which further we split in a 10/90 split for the test and train splits. The metrics were computed on an average of 100 random samples of 5 classes in the remaining classes of CIFAR-FS. We reported 1) the geometric complexity of the embedding layer measured on the source dataset (Source GC), 2) the neural collapse as measured by (3) measured and averaged on the 100 random samples of 5-label samples from the target dataset, 3) the test accuracy obtained by solving the normal equation directly for a ridge regression with only 5 examples per class obtained from the target set. These three quantities were measured every 1000 steps of a very long pre-training of 100 0 00 steps. The optimizer was plain SGD (note we obtained similar results with momentum) without any regularization nor schedule to avoid masking effects. We used random crop and random flip for data augmentation. The results were all averaged over 5 random seeds for the neural network parameter default initialization. The plots have no smoothing applied to the learning curves. Top row: We swept over a learning rate range of { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 25 , 0 . 5 } with a constant batch size of 256. Middle row: We swept over a batch size range of { 8 , 16 , 32 , , 64 , 128 } with a constant learning rate of 0.005. Bottom row: We swept over a L2 regularization rate range of { 0 , 0 . 0001 , 0 . 0005 } Each sweep took roughly 10h of training on a single Google Cloud TPU V3 accessed via a Google colab.

A.5 Additional Experiments

A.5.1 Cifar-10 on ResNet-18

In Figure 6, we trained a ResNet-18 neural network on CIFAR-10. The experimental setup is the same as described in Appendix A.4.1. The results were all averaged over 5 random seeds for the neural network parameter default initialization. The plots have no smoothing applied to the learning curves.

Top row: We swept over a learning rate range of { 0 . 005 , 0 . 01 , 0 . 025 , 0 . 05 , 0 . 1 , 0 . 2 } with a constant batch size of 256. Middle row: We swept over a batch size range of { 8 , 16 , 32 , 64 , 128 , 256 } with a constant learning rate of 0.2. Bottom row: We swept over a L2 regularization rate range of { 0 . 0 , 0 . 0001 , 0 . 00025 , 0 . 0005 , 0 . 00075 } with learning rate 0.02 and batch size 512.

0.008

0.0200

Learning Rate

0.0175

0.007

0.005

0.01

0.025

0.05

0.005

0.1

0.2

3

2

0.0025

0.001

0.0000

0

20000

40000

60000

80000

100000

0.000

0

20000

40000

60000

80000

100000

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

0.0010

0.010

8.0

1.2

16.0

Collapse

0.0008

32.0

64.0

1.0

128.0

0.0006

256.0

ometric

0.004

0.0002

0.4

0.000

0

20000

40000

Iteration

60000

80000

100000

0.0000

0

20000

40000

Iteration

60000

80000

100000

0.2

0

20000

40000

Iteration

60000

80000

100000

0.0014

1.4

0.020

0.0012

Regularization Rate

0.0001

0.0

1.2

0.00025

0.0010

0.0005

Collap

0.00075

0.0008

0.0002

0.2

0.000

0

20000

40000

60000

80000

100000

0.0000

0

20000

40000

60000

80000

100000

0.0

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

A.5.2 MNIST on VGG-11

In Figure 7, we trained a VGG-11 neural network [56] on MNIST. The experimental setup is the same as described in Appendix A.4.1. The results were all averaged over 5 random seeds for the neural network parameter default initialization. The plots have no smoothing applied to the learning curves.

Top row: We swept over a learning rate range of { 0 . 0025 , 0 . 005 , 0 . 01 , 0 . 025 , 0 . 05 , 0 . 1 } with a constant batch size of 512. Middle row: We swept over a batch size range of { 8 , 16 , 32 , 64 , 128 , 256 , 512 , 1024 } with a constant learning rate of 0.005. Bottom row: We swept over a L2 regularization rate range of { 0 , 0 . 00001 , 0 . 0001 , 0 . 00025 , 0 . 0005 } with learning rate 0.02 and batch size 512.

0.0006

0.6

Learning Rate

0.04

0.0005

0.0025

0.005

0.5

0.01

0.025

0.05

0.0003

0.0002

0.0001

0.1

0.00

0

20000

40000

60000

80000

100000

0.0000

0

20000

40000

60000

80000

100000

0.0

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

0.0005

0.5

0.04

0.0004

0.4

Collapse

128

0.0003

256

Train Geometric

512

1024

0.0002

0.0001

0.1

0.00

0.0000

0.0

0

20000

40000

Iteration

60000

80000

100000

0

20000

40000

Iteration

60000

80000

100000

0

20000

40000

Iteration

60000

80000

100000

0.00040

0.5

0.040

Regularization Rate

SE00

0.00035

0.0

1e-05

0.0001

0.4

0.00025

0.0005

0.1

0.005

0.00005

0.000

0

20000

40000

60000

80000

100000

0.00000

0

20000

40000

60000

80000

100000

0.0

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

A.5.3 Fashion-MNIST on VGG-11

In Figure 8, we trained a VGG-11 neural network [56] on Fashion-MNIST. The experimental setup is the same as described in Appendix A.4.1. The results were all averaged over 5 random seeds for the neural network parameter default initialization. The plots have no smoothing applied to the learning curves.

Top row: We swept over a learning rate range of { 0 . 005 , 0 . 01 , 0 . 025 , 0 . 05 , 0 . 1 , 0 . 2 } with a constant batch size of 256. Bottom row: We swept over a L2 regularization rate range of { 0 , 0 . 00001 , 0 . 0001 , 0 . 00025 , 0 . 0005 } with learning rate 0.02 and batch size 512.

0.0014

1.4

0.12

0.0012

1.2

Learning Rate

0.005

0.01

0.025

0.05

0.04

0.02

0.0002

0.2

0.00

0

20000

40000

60000

80000

100000

0.0000

0

20000

40000

60000

80000

100000

0.0

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

0.0012

0.6

0.12

0.0010

0.5

0.04

Regularization Rate

0.0

1e-05

0.02

0.0002

0.0001

0.1

0.00025

0.00

0.0005

0

20000

40000

60000

80000

100000

0.0000

0

20000

40000

60000

80000

100000

0.0

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

A.5.4 Cifar-100 on ResNet50

In Figure 9, we trained a ResNet-50 neural network on CIFAR-100. The experimental setup is the same as described in Appendix A.4.1. The results were all averaged over 5 random seeds for the neural network parameter default initialization. The plots have no smoothing applied to the learning curves.

Top row: We swept over a learning rate range of { 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 } with a constant batch size of 256. Bottom row: We swept over a L2 regularization rate range of { 0 . 0 , 0 . 0001 , 0 . 00025 , 0 . 0005 , 0 . 00075 } with learning rate 0.02 and batch size 256.

0.40

0.07

Learning Rate

0.35

0.06

0.01

0.005

0.05

0.2

0.04

2

0.05

0.01

0.00

0

20000

40000

60000

80000

100000

0.00

0

20000

40000

60000

80000

100000

0

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

0.16

0.06

Regularization Rate

0.14

0.0

0.05

0.0001

0.00025

0.0005

0.10

0.00075

0.01

0.02

0.00

0

20000

40000

60000

80000

100000

0.00

0

20000

40000

60000

80000

100000

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

A.5.5 Lower Pre-trained GC Leads to Improved Fine-tuning: mini-ImageNet dataset on VGG architecture

In Figure 10, we trained a VGG16 neural network (as described in [57]) implemented in Flax on Mini-ImageNet (as described in [60]). We used 10 classes with 600 examples per class for training, with a 10/90 train/test split. To evaluate the downstream performance we used 100 downstream tasks consisting of 5 labels for few shot learning randomly chosen from a pool of 20 separate from the training labels. The experimental setup was the same as described in section A.4.3. Namely, we reported 1) the geometric complexity of the embedding layer measured on the source dataset (Source GC), 2) the neural collapse as measured by (3) measured and averaged on the 100 random samples of 5-label samples from the target dataset, 3) the test accuracy obtained by solving the normal equation directly for a ridge regression with only 5 examples per class obtained from the target set. These three quantities were measured every 1000 steps of a very long pre-training of 100,000 steps. The optimizer was plain SGD (note we obtained similar results with momentum) without any regularization nor schedule to avoid masking effects. We used random crop and random flip for data augmentation. The results were all averaged over 5 random seeds for the neural network parameter default initialization. We swept over a learning rate range of { 0 . 001 , 0 . 005 , 0 . 01 } with a constant batch size of 256. Each sweep took roughly 10h of training on a single Google Cloud TPU V3 accessed via a Google colab.

Learning Rate

0.400

0.14

0.001

0.12

16

0.005

0.01

0.375

0.350

Source

GC

0.10

0.08

Target NC

14

0.325

W

0.300

0.06

12

333Sz

0.275

0.04

0.250

0.02

10

0.225

0.00

0

20000

40000

60000

80000

100000

0

20000

40000

60000

80000

100000

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

A.5.6 Direct GC regularization increases neural collapse

To mitigate possible confounding factors due to batch size and learning rate manipulations, we train a VGG-13 model [57] on CIFAR-10 with explicit GC regularization using a fixed learning rate of 0 . 01 and a batch size of 256 . We explore increasing levels of explicit GC regularization with rates 10 -6 , 10 -5 , 10 -4 and plot the results in Figure 11. We observe the same predictions as in our experiments which indirectly lower the GC through learning rates, batch sizes and L2 regularization. Namely, lower GC produces lower NC values (i.e., increased neural collapse). Note that in this experiment we regularized w.r.t. the GC computed at the logit layer rather than the embedding GC. This is because the high dimension of the embedding layer makes taking the gradient of the embedding GC prohibitively expensive. Already, regularization with GC taken at the logit layer was possible only for one seed. That being said, explicit regularization by GC computed at the logit layer provides a direct way to control GC computed at the embedding layer.

0.025

2.5

1.0

GC Regularization Rate

0.0

Train Geometric Complexity

0.8

Collapse

0.020

0.0001

1e-07

1e-06

Train Neural Collapse

2.0

0.6

Train Geometric

0.015

1.5

0.4

0.010

1.0

0.2

0.005

0.5

0.0

0

20000

40000

60000

80000

100000

0.000

0

20000

40000

60000

80000

100000

0.0

0

20000

40000

60000

80000

100000

Iteration

Iteration

Iteration

A.6 Discussion on the Poincaré inequality

Recall that a distribution Q satisfies a Poincaré inequality if, for all differentiable functions v defined on the support of Q , there exists a constant c > 0 such that

In this section, we argue that the Poincare Inequality (PI) is a mild and natural assumption to make. For instance, the Gaussian distribution, mixtures of Gaussians with equal (or different) variances, mixtures of Gaussians and sub-Gaussians, mixtures of uniform and Gaussian measures; and any log-concave probability measure all satisfy a PI; see, for example, [1, 54]. The same is true for most distributions of importance in probability and statistics; see [47].

The Poincaré constant c itself can take very high values depending on the spread of the distribution. For example, for a standard normal distribution the Poincare constant is 1 while for a multivariate normal distribution the Poincare constant equals the largest eigenvalue of its covariance matrix which can be any positive real number. Intuitively speaking, the Poincaré constant will increase if the space is stretched in any direction and decrease if it is compressed in any direction.

The PI has also been assumed to hold for real life image datasets, where it has been used to help improve GAN convergence, as in [31]. It is also a key assumption to understand the role of overparameterization in generalization as happens for large neural networks (cf. [7] which frames this as an equivalent isoperimetry condition).

On the contrary, non-PI distributions are considered pathological; they can be constructed for instance by artificially concatenating distributions with fully disjoint support. See [42] for a construction of pathological examples of distributions not satisfying a Poincaré inequality.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately state the paper's contributions, assumptions, and limitations. The claims about how geometric complexity influences neural collapse align with the results, reflecting the paper's scope. The abstract clearly indicates the generalizability of the proposal based on the datasets used.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The experiments in this paper focus on image processing and involve a large number of important hyperparameters and extensive runs. The paper clearly states that the generalization effect during transfer learning is tested only on image processing datasets, such as CIFAR-10, CIFAR-100, MNIST, Fashion-MNIST and CIFAR-FS, mini-ImageNet in the transfer learning setting.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, the paper provides the full set of assumptions and a complete, correct proof for each theoretical result. Each assumption is clearly stated, and the proofs are meticulously detailed, ensuring the validity and rigor of the theoretical contributions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, the paper fully discloses all the information needed to reproduce the main experimental results, ensuring that the main claims and conclusions can be independently verified. This includes detailed descriptions of the methodology, data sources and all relevant parameters and all hyperparameters.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Yes, while the code is not open source, the paper uses open datasets, which are well-known benchmark datasets for image processing, such as CIFAR-10, CIFAR100, MNIST, Fashion-MNIST and CIFAR-FS and mini-ImageNet for transfer learning. The supplemental material includes detailed instructions to faithfully reproduce the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, the paper specifies all the training and test details, including data splits, hyperparameters, their selection process, and the type of optimizer, ensuring a comprehensive understanding of the results. The Appendix includes full experiment details for all main paper plots as well as for all plots appearing in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Yes, the paper reports error bars that are suitably and correctly defined, based on multiple runs for each training. It provides the standard deviation for each metric, such as accuracy, geometric complexity, and neural collapse, ensuring a clear understanding of the statistical significance of the experiments. All experiments are repeated over 5 random seeds and error bars are included in the Figures.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, the paper provides sufficient information on the computer resources required for each experiment, including the type of compute workers, and execution time, ensuring that the experiments can be accurately contextualized for performance.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, it does.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There are no societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There are no risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: No use of third-party code beyond machine learning frameworks like JAX and TensorFlow, and common Python libraries for experimental purposes.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets released.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or research performed with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing or research performed with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the relationship between geometric complexity (based on penultimate layer features) and the neural collapse phenomenon (especially, the variability collapse property). The main theoretical result is a bound on the CDNV-based NC metric using geometric complexity (Proposition 4.1). This is then employed to extend generalization and transfer learning bounds (proposed by previous efforts) to rely on geometric complexity. Experimental results on classification tasks (along with transfer learning settings) are presented to validate the proposed relationship.\n\nSTRENGTHS:\nThe unified analysis of neural collapse and geometric complexity presents an interesting line of research for the community. Especially, in terms of studying seemingly disjoint phenomena under a common lens.\n\nWEAKNESSES:\n1. There seems to be a mismatch in the $\\log(2/\\delta)$ term in Proposition 4.2 vs $\\log(1/\\delta)$ term in Proposition 4.3, which makes the main results inconsistent (yet fixable). Additionally, why do we have an \"expectation\" over the \"error\" term in the LHS of Eq (10)? Isn't the \"error\" itself an expectation?\n\n2. Although the paper presents generalization bounds, a discussion on the generalization performance of the classifier (VGG-13 for Figure 1) seems to be missing. Only the neural collapse and geometric complexity plots are illustrated but a deeper relationship between the trends and the test performance is lacking. Is there an upper limit of \"good\" learning rates, and batch sizes above which the geometric complexity can reduce but the test performance can get worse? \n\n3. Similar to the above point, currently it seems like increasing the learning rate, increasing batch size, and increasing the regularization parameter value results in lower neural collapse values. Ideally, neural collapse is applicable during the terminal phase of training (TPT) where the training accuracy is $1$. Does this mean that all the experiments (at least for Figure 1) lead to TPT during training?\n\nnit: Please fix the notation for the norms as it is unclear if $||.||$ represents $||.||_2$ or $||.||_F$.\n\nQUESTIONS:\n1. Since the geometric complexity-based generalization/transfer learning bounds are the upper bounds of existing neural collapse-based bounds, what can be a potentially new implication of your results? since it is common practice to do a hyper-parameter search over learning rates, batch sizes, and regularization parameter values. \n\n2. The paper does not discuss any explicit regularization mechanism that aims to reduce the geometric complexity. What are the potential challenges in designing such a regularization mechanism?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the relationship between geometric complexity (GC), neural collapse (NC), and transfer learning performance in deep neural networks. The key contributions are:\n\n* Proposing geometric complexity as a measure that connects the flatness of the loss surface and neural collapse\n* Deriving theoretical bounds showing GC controls NC under certain assumptions\n* Empirically demonstrating that mechanisms affecting GC (learning rate, batch size, regularization) also influence NC\n* Showing lower GC during pre-training leads to better transfer learning, especially for few-shot tasks\n* Proving a new generalization bound in terms of GC\n* Demonstrating GC can be efficiently estimated on small samples\n\nThe authors argue that GC can serve as an informative proxy for transfer learning potential and provide a unifying framework for understanding implicit biases in deep learning. By bridging concepts from optimization geometry, representation learning, and transfer learning, the paper aims to provide deeper insights into the fundamental mechanisms driving the success of modern deep learning techniques.\n\nSTRENGTHS:\n**Originality**: The paper presents a novel perspective connecting several concepts in deep learning theory. Its original contribution is using geometric complexity as a bridge between loss surface geometry and neural collapse. This approach provides a fresh lens through which to view the implicit biases in deep learning, potentially unifying several strands of research. Using GC as a progress metric for transfer learning is innovative and could open up new avenues for improving pre-training techniques.\n\n**Quality**: The empirical evaluation thoroughly examines multiple datasets, architectures, and hyperparameter settings. The authors have made a commendable effort to validate their hypotheses across various conditions, including different network architectures (ResNet, VGG), datasets (CIFAR, MNIST, Fashion-MNIST), and transfer learning scenarios. The ablation studies exploring the impact of learning rate, batch size, and regularization on GC and NC provide valuable insights into the dynamics of deep network training. While limited in scope, the theoretical results give some formal grounding for the empirical observations and offer a starting point for further analytical work in this area.\n\n**Clarity**: The paper is well-structured and communicates the main ideas. The authors have presented a complex set of ideas in a logical flow, starting from the theoretical foundations and moving to empirical validation. The figures illustrate key results, particularly the relationships between GC, NC, and transfer performance. The use of color-coding and consistent formatting across figures aids in interpretation. The pseudo-code and detailed experimental setup descriptions in the appendix are beneficial for reproducibility.\n\n**Significance**: If the claims hold up, this work could provide valuable insights into the mechanisms behind transfer learning and self-supervised learning. The proposed geometric complexity measure may help analyze and improve deep learning models. The potential impact extends beyond just theoretical understanding. If GC can serve as a reliable proxy for transfer learning potential, it could guide the development of more effective pre-training strategies and architectures. Furthermore, the connection between GC and NC could help explain why certain training practices (like large batch training or specific learning rate schedules) are effective, potentially leading to more principled approaches to hyperparameter tuning.\n\nWEAKNESSES:\n**Limited theoretical foundation**: While the paper provides some theoretical results, the assumptions required (e.g., Poincaré inequality) are quite strong and may not hold in practice for real datasets and architectures. The connection between the linear model analysis and deep neural networks is not strongly justified. This is a significant limitation, as it's unclear how much of the theory translates to practical scenarios. The authors could strengthen this aspect by providing empirical evidence that these assumptions hold in realistic settings or deriving weaker results under more general conditions. Additionally, the paper could benefit from a more thorough discussion of the implications and limitations of these theoretical results.\n\n**Experimental limitations**: The empirical evaluation focuses on image classification tasks and relatively small datasets. The generalizability of the results to larger datasets, other domains (e.g., NLP), or more complex architectures is unclear. This narrow focus raises questions about the broader applicability of the findings. For instance, it's unclear whether the relationships between GC, NC, and transfer performance would hold for large-scale pre-training scenarios like those used in modern language models. The paper would be significantly strengthened by including experiments in a broader range of tasks and scales or at least by providing a more detailed discussion of the potential challenges in scaling up the approach.\n\n**Lack of comparison to related metrics**: The paper does not thoroughly compare geometric complexity to other related complexity measures or generalization bounds from the literature. This makes it difficult to assess the relative merits of GC. There's a rich body of work on complexity measures for neural networks, including various forms of capacity measures, stability-based bounds, and PAC-Bayesian approaches. A comprehensive comparison would help situate GC within this broader context and clarify its unique contributions. Such a comparison could also help identify scenarios where GC might be particularly advantageous or potentially suboptimal compared to existing approaches.\n\n**Causality vs correlation**: While the paper shows correlations between GC, NC, and transfer performance, it does not conclusively demonstrate a causal relationship. Alternative explanations for the observed trends have not been thoroughly explored. This is a critical limitation, as it leaves the possibility that GC is merely a proxy for some other underlying factor driving NC and transfer performance. The paper would be strengthened by a more rigorous causal analysis, perhaps through interventional studies or by controlling for potential confounding variables. Additionally, discussing potential mechanisms by which GC might causally influence NC and transfer learning would add depth to the analysis.\n\nQUESTIONS:\n1. How does geometric complexity compare to other measures of model/representation complexity like Fisher information, gradient norm, etc.? A more thorough comparison would strengthen the paper.\n2. The theoretical results rely on strong assumptions like the Poincaré inequality. Can you provide empirical evidence that these assumptions hold for real datasets/models? Or derive results under weaker assumptions?\n3. The paper shows correlations between GC, NC, and transfer performance but does not conclusively prove causation. Have you considered alternative explanations or confounding factors?\n4. How computationally expensive is it to measure GC during training? Is it feasible to use it as a regularization term or early stopping criterion in practice?\n5. The generalization bound in terms of GC is interesting, but how tight is it empirically? How does it compare to other generalization bounds in the literature?\n6. Have you explored using insights about GC to guide architecture design or improve training algorithms? Demonstrating practical benefits would strengthen the paper.\n\nLIMITATIONS:\nThe authors acknowledge some limitations, like the focus on image classification tasks and relatively small datasets. However, they could go further in discussing potential shortcomings:\n\n* The strong assumptions required for the theoretical results may limit their applicability.\n* The computational cost of measuring GC is not thoroughly addressed.\n* Potential negative consequences of optimizing for low GC are not explored.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the relationship between neural collapse (NC) and geometric complexity (GC). It presents both theoretical and empirical evidence showing that geometric complexity is a robust metric across various variables. By substituting the NC metric with GC, the paper introduces a generalization bound based on geometric complexity. Furthermore, it highlights the significance of GC in transfer learning, demonstrating that by controlling the pre-training GC, downstream NC can also be controlled, leading to improved transfer learning results.\n\nSTRENGTHS:\n* The paper is overall well-written and easy to follow, the interpretation of each theoretical results is clear and intuitive.\n* The relationship between NC and GC is quite interesting.\n* Most of the empirical results align well with the theoretical part, demonstrating the validity of the results.\n\nWEAKNESSES:\nSome empirical results can be better aligned with the theoretical counterpart and additional experiments could be added to better support the claims. For example:\n* In proposition 4.1, NC is upper bounded by geometric collapse up to some constant scaling. What's the range of this constant $c$, does it purely depend on $Q$? If yes, it would be better to directly plot NC and RHS in (6) in the same plot to demonstrate the validity of the bound; and if no, it would be interesting to show how different settings examined in Figure 1 affect $c$.\n* The empirical evidence presented in Figure 2 clearly demonstrates the robustness of geometric complexity. However, I have some concerns regarding the Lipschitz notion. Neural networks are known to be not so Lipschitz, and making them more Lipschitz is an active area of research aimed primarily at improving robustness. Is Proposition 4.2 numerically verifiable by calculating the Lipschitz constant of the tested neural networks?\n\nQUESTIONS:\n* In equation (7), does the distance $d$ represent the norm distance? This should be stated clearly.\n* Both the paper and the prior work (Galanti et al, 2022) rely on the data assumption where the source and transfer datasets come from the same distribution $Q$. However, in practical settings, source and transfer datasets typically have domain gaps which can potentially violate the data assumption. Would the results in the paper still hold in this case? For example, pre-training on ImageNet/Cifar-100 while fine-tuning on DTD?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}