The Impact of Geometric Complexity on Neural Collapse in Transfer Learning

Authors: michael  munn
benoit  dherin
javier  gonzalvo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 22:12:59

Abstract: Many of the recent advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this empirical success is incomplete and remains an active area of research. Flatness of the loss surface and neural collapse have recently emerged as useful pre-training metrics which shed light on the implicit biases underlying pre-training. In this paper, we explore the geometric complexity of a model's learned representations as a fundamental mechanism that relates these two concepts. We show through experiments and theory that mechanisms which affect the geometric complexity of the pre-trained network also influence the neural collapse. Furthermore, we show how this effect of the geometric complexity generalizes to the neural collapse of new classes as well, thus encouraging better performance on downstream tasks, particularly in the few-shot setting.

Main paper: 
 introduction: 
 Many of the recent remarkable advances in modern machine learning owe their success in part to transfer learning [4, 6, 9, 15, 28, 49, 50, 66] . While there are different approaches to this technique, the standard one involves two stages. In a first stage, called pre-training, ones trains a deep neural network on a general, large-scale dataset in the form of a supervised or unsupervised source task; e.g., ImageNet or CIFAR-100 [14, 33] for image models or the Common Crawl, C4 or LM1B datasets [8, 11, 49] for language models. In the second stage, one then leverages portions of the pre-trained network to use as features map or embeddings that can then be adapted, or fine-tuned, on a more specific target task. Often these target tasks are unknown at the time of pre-training and labeled data may be very scarce, such as in the context of few-shot learning [34, 51, 60] . However, despite these limitations, this approach often results in a fine-tuned model that achieves quite impressive performance substantially better than training on the target task alone and requiring less computational resources [20] . Despite this empirical success, a comprehensive theoretical understanding of the mechanisms underlying this effectiveness of transfer learning is not fully understood and remains an active area of research [63] .One interesting line of research suggests that the effectiveness of transfer learning is due to the implicit biases encoded in pre-trained models [22, 36, 48] . These implicit biases effectively constrain the hypothesis space, guiding the model towards solutions that have a preference for smoother functions [12, 17, 38, 40, 53] , simpler geometry in the loss surface [3, 13, 18, 55, 58] or reduced complexity of the internal learned representations [23, 32] . In the same way that these implicit biases have been used to help explain the success of deep learning, recent work has shown that the notion of neural collapse [22, 35, 62] and flatness of the loss surface [36] can also inform the mechanisms behind transfer learning. This suggests that these preferences during pre-training also act as a form of prior knowledge which is highly transferable to downstream tasks, even with limited task-specific data.In this paper, we present a novel perspective that further sheds light on these mechanisms and implicit biases hidden within transfer learning. Our approach analyzes the geometric complexity [16, 17] of the internal representations learned by the deep neural networks during pre-training and provides a complementary theoretical framework which unifies previous work examining the role of neural collapse and loss surface flatness in transfer learning [22, 36] . In particular, we show through experiments and theory that the geometric complexity of the pre-trained network directly controls the neural collapse of the pre-trained model and thus its efficacy in transfer learning, particularly in the few-shot setting. We argue that geometric complexity (similarly to flatness of the loss surface and neural collapse) can be used as hidden progress metrics, cf. [2] , for transfer learning, serving as an informative proxy toward the transfer power of a pre-trained network.Our primary contributions are the following:• We uncover relationships between learning-path flatness, neural network geometriccomplexity, and embedding neural-collapse providing a framework to understand how these implicit biases interact (Section 4). • We show through theory that the geometric complexity (GC) of a neural network controls neural collapse and verify this empirically by showing how mechanisms which regularize the GC in turn put pressure on the neural collapse (Section 4.1 and Fig. 1 ). • We demonstrate both theoretically and empirically that pre-trained networks with lower GC promote lower neural collapse on new unseen target classes, and thus enable improved transfer accuracy during fine-tuning (Section 5 and Fig. 4 ). • We prove a new generalization bound in terms of geometric complexity (Section 4.3).• We show that the empirical GC can be accurately and efficiently estimated on a small number of samples, input coordinates, and output features making it computationally tractable compared to other progress metrics in machine learning (Section 4.2 and Fig. 2 ).Notation. Throughout, we denote by • the L2 Euclidean norm and by • F the Frobenius norm. 
background and related work: 
 The implicit biases introduced by an optimization algorithm play a crucial role in deep learning and in the generalization ability of the learned models [43, 44, 55, 64] . They help ensure that the model not only finds a solution with low error but also one with low complexity which generalizes well [26, 67] . Uncovering the mechanisms of these implicit regularizers is crucial for understanding how the model learns, both as a means to improve generalization and as valuable leverage for designing more efficient algorithms and decreasing costly experiment iteration cycles.Here we focus on three seemingly different themes behind implicit regularization in deep learning and explain how they are related: the loss surface slope measuring the flatness of the learning path [3, 29, 45] , the geometric complexity of the learned model function measuring its variability with respect to a dataset [16, 42] , and the neural collapse [32, 46] measuring how a neural network efficiently clusters its learned representations of the input class examples in embedding space.Flatness in parameter space. In parameter space, the learning dynamics is fully characterized by the discrete optimization path θ t where t ∈ N. At each step, one can compute the flatness of the learning path as the slope of the tangent space at θ t to the loss surfaceL = {(θ, L(θ)) : θ ∈ R n }.As shown in [3, Appendix A.2] , this slope coincides with the loss-gradient square norm: slope(θ t ) = ∇L(θ t ) 2 . This quantity has been used to bound a generalization gap in [25] and as a beneficial explicit regularizer in [3, 24, 58] indicating that learning curves with lower slope values tend to also have better test performance.Moreover, many standard optimizers and common training heuristics have been shown to put an implicit pressure on the learning-path flatness, making it an implicit bias of the training procedure [3, 5, 10, 13, 17, 25, 39, 58] . Related to the learning curve slope is the optima sharpnesssharpness(θ * ) = 1 n trace H(θ * )where θ * is a global minima toward which the learning dynamics converges to θ t → θ * and H is the Hessian of the loss. Since the sharpness of an optima corresponds to the mean curvature of the loss surface at that point, flat minima (i.e., minima with low curvature in all directions) can be reached only through learning paths with shallower slopes.Neural collapse in embedding space. Neural collapse [22, 27, 46] refers to a phenomenon observed in deep learning where the embedding network (i.e., the subnetwork f before the logit layer g in a neural network h(x) = softmax(g(f (x))) collapses the input points around their respective class means. Furthermore, these class means form a simplex creating an equiangular tight frame (ETF) centered around the global mean with roughly equal distance between its vertices. Intuitively, such a phenomenon is beneficial to generalization since the embeddings of different classes are optimally separated, making the job of the classifier head softmax(g(z)) easier. To measure neural collapse, the authors in [22] introduce the class-distance normalized variance (CDNV) asEQUATIONwhere Q r = q r (x)dx is the input distribution for classes r ∈ {i, j} and whereEQUATIONNeural collapse between two classes happens when their class variance decreases while the distance between their class means increases, causing lower values of their CDNV. Following [22] , given a well-balanced input distributionQ = 1 k (Q 1 +• • •+Q k ) with k classes (Q i being the input distribution for class i), neural collapse (NC)during training is characterized by the following limit as the number of training steps t goes to infinity:EQUATIONand where f t is the learned embedding network at step t. Thus, more neural collapse (i.e., more clustering around better separated class-means) happens with lower NC values. Proposition 5 from [22] proves that NC(f, Q) bounds a generalization gap and lower values of NC(Q, f ) are correlated with better test performance of the neural network f , and [37] shows that explicitly regularizing for neural collapse is beneficial.The geometric complexity (GC) of a function f : R d → R k w.r.t. a data distribution Q = q(x)dx on R d is defined as the expectation of the gradient Frobenius square-norm w.r.t. to the data distribution [17] given byGC(f, Q) := E x∼Q ∇ x f (x) 2 F .Intuitively, the GC measures the function complexity or variability and is closely related to the Dirichlet energy in geometric analysis [21] . Provided mild Lipschitz smoothness assumptions [42, Proposition 3.4] , the GC can be estimated accurately on batches of training data D by its empirical counterpartEQUATION)Previous work has explored the relationship between the GC and model generalization. When measured on the full neural network, lower GC values correlate experimentally with higher test accuracy [45] and it has also been used as a beneficial explicit regularizer [30, 59] . In [16, 17] , the authors ignore the softmax activation and study the GC measured with respect to the logit network, exploring its connection with various implicit and explicit regularizers and training heuristics. The logit GC has also be used to prove a margin based multi-class generalization bound [42] , assuming that the input distribution Q satisfies a mild assumption known as the Poincaré inequality [1] .In fact, both this work and the proof of the generalization bound rely on the Poincaré inequality, which we argue is indeed a natural and mild assumption for the types of data distributions typically encountered in machine learning; see Appendix A.6 for further discussion. For completeness, we recall it here, cf. [19] . A distribution Q satisfies a Poincaré inequality if, for all differentiable functions v defined on supp(Q), there exists a constant c such thatEQUATIONNote that the same assumption is used in [7] to prove the law of robustness for overparameterized neural networks via isoperimetric inequalities. In this paper, we peel back yet another layer of the network and consider the embedding geometric complexity measured on a feature map (Section 3).Relationship between flatness, neural collapse, and geometric complexity. The learning-path flatness as measured by its slope influences the geometric complexity of the learned solutions [16, Theorem 5.1] . Namely, for dense layers a regularizing pressure on the slope of the learning path in parameter space transfers to a regularizing pressure on the geometric complexity of the learned solution in function space. A similar result also been shown for special attention layers as well [52] .In this paper, we will see that imposing a regularization pressure on the embedding geometric complexity encourages more neural collapse of the model. This results in the following chain of influences from regularization pressure: For the first stage of our inquiry (Section 4), we are concerned with a k-classification task. Let D := {(x i , y i )} m i=1 be a dataset drawn from a distribution Q with x i ∈ R d and y i ∈ {e j | j = 1, . . . , k} where e j ∈ R k are the canonical basis vectors representing a one-hot encoding of the labels. We aim to learn this task using a neural network denoted by a function h θ : R d → R k parameterized by θ (though to simplify notation we subsequently drop this dependence on θ).learningWe can write h θ (x) = softmax(g(f (x)) where g : R p → R k is a classifier head mapping from the feature space R p to the prediction layer R k and f : R d → R p is a feature mapping from the input space to the penultimate embedding layer of the network before the classification head. The standard way of training h θ is to find, via some stochastic optimization technique, an optimal parameter configuration θ * such that θ * = argmin θ m i=1 (h θ (x i ), y i ) for a given loss function :R k × R k → [0, ∞).To analyze the role of neural collapse and geometric complexity in this setting, we focus our attention on the feature map f : R d → R p . The NC as described in Section 2 is defined using this sub-network map f of h and thus we also measure the geometric complexity with respect to this sub-network feature mapping as well. Throughout this paper, we use the term embedding GC to refer to the GC of such a feature map f and unless otherwise specified, when we refer to the model GC we mean the embedding GC, not the GC of the logit network as in [16, 17] .Our second stage of inquiry (Section 5) is focused on the role of geometric complexity in transfer learning. For this setting, we assume there is some l-class classification target task we would like to solve and corresponding target distribution T . We aim to learn a classifier h : R d → R l given some training data D := {(x i , y i )} m i=1 which is potentially very limited in size. In order to find a good solution, we can leverage a pre-trained feature mapping that has been trained on some other source task and source distribution S where more data is available. For example, by leveraging a feature map f pre-trained as in the above setup, the target task classifier h can instead be trained on the outputs of our feature map; i.e., {f (x i ), y i )} m i=1 . In this way, we can write h = softmax(g • f ) where the parameters of f are fixed and we only need to learn the parameters of g : R p → R l . Ideally, provided f is a rich enough feature map trained on a general enough source distribution dataset, then the task for learning g is much simpler. 
geometric complexity and neural collapse: 
 In this section, we explore the general relationship between the embedding GC and neural collapse showing that the geometric complexity can be used to bound neural collapse. Next, we see how the geometric complexity can be efficiently and accurately estimated, both as an approximation of the theoretical GC as well as through a number of sampling techniques for the empirical GC via batch sampling, feature sampling and label sampling. Lastly, following [22] , we derive a new generalization bound for neural networks expressed in terms of the embedding geometric complexity; cf. [42] .Previously, it has been shown [16] that the GC can be controlled implicitly through choice of learning rate and batch size, as well as through standard explicit regularization. Practically, this means that these same beneficial tuning strategies which ensure that models have low GC should also work to keep the NC low as well. The following proposition (which we prove in Appendix A.1) bounds the neural collapse by the geometric complexity provided that the input distribution satisfies a Poincaré inequality also assumed in [42] and [7] .Proposition 4.1. Suppose that we have a balanced multi-class input-distribution Q with k classes satisfying the Poincaré inequality in (5) for some constant c, then the geometric complexity of a network embedding f bounds its neural collapse as measured by (3); namely, we have the following boundEQUATION)where k is the number of classes, and d ij is the distance between the mean of class i and class j.We call the RHS of the bound in Eq. ( 6), excepting the Poincaré constant c, the geometric collapse:EQUATION)where k is the number of classes, andd ij = µ f (Q i ) -µ f (Q j )denotes the Euclidean distance between the mean µ f (Q i ) and µ f (Q j ) of class i and class j (resp.) in embedding space.This quantity can be seen as another proxy metric measuring neural collapse. It is made of two main parts decoupling the variances and mean differences present in the neural collapse framework. The variances are consolidated into a single factor through the GC, while the mean differences are averaged across classes in a separate factor. This separation allows the overall intra-class variability to be influenced through the GC term, ensuring sufficient between-class separation through the mechanisms identified in neural collapse.This bound provides a powerful method to control the overall within-class variability via the L2-norm of the model gradient. As a result, the geometric collapse provides a simplified and more refined approach to managing class separability and variance in deep learning models.We verify the relationship posed in Eq. ( 6) through experiments on VGG-13 trained on CIFAR-10 (see Figure 1 ). Through both implicit and explicit regularizers the pressure on the embedding GC translates to a direct pressure on the neural collapse of the model via the geometric collapse. As already observed in [17] , lower levels of GC coincide with higher test accuracy as show in Figure 5 in Appendix A.5 containing the learning curves of that experiment.In Appendix A.5 we see the same relationship holds across various datasets; e.g., MNIST, Fashion-MNIST, CIFAR-100, and architectures; e.g., ResNet-18, ResNet-50. To avoid possible confounding factors that could arise through batch size and learning rate manipulations, we also directly regularize with the geometric complexity in Appendix A.5.6; the same relations hold in this case too.One of the key advantages of the GC as a complexity measure is its computational efficiency. Under mild regularity constraints on the model function the theoretical geometric complexity of a map can be efficiently estimated by its empirical counterpart, as stated in the proposition below, already proven in [42] for logit networks (see Appendix A.2 for a proof). Additionally and crucially, we verify that the empirical GC is a robust and reliable measure with respect to that data sample. Proposition 4.2. Let f : R d → R p be a map with Lipschitz constant L and let D X be a sample of m elements drawn independently from an input distribution Q. Then, for any δ > 0, we have with probability 1 -δ/2 the following boundEQUATION)For a function f and data sample D X as in Proposition 4.2, to measure GC(f, D X ) requires computing the Frobenius norm of a potentially very large Jacobian matrix, particularly when p represents the embedding dimension of a feature map. Note that,EQUATION)Although the computational complexity of the GC is impervious to increasing complexity of the network architecture, e.g., in terms of parameter count or number of layers, one may run into computation bottlenecks as the sample size m increases or when increasing the dimensionality d (resp. p) of the inputs (resp. outputs). In these scenarios, it is necessary to find efficient means to accurately approximate or sample the Jacobian; cf. [61] .We explore the robustness of the GC when measured via samplings along these three axes; i.e., decreasing the number of examples m in the batch, randomly sampling the full Jacobian matrix which has order d × p, and randomly sampling the number of model outputs p. As shown Figure 2 , we find that the value of the sampled GC remains stable and consistent to its true value through these fairly simple and naive sampling tricks. In [42] , the authors derive a margin-based multi-class generalization bound for neural networks which depends on the geometric complexity GC(h, Q) of the full logit network. In this section, we extend this result. With inspiration from [22, Proposition 5], we show that the geometric complexity measured on the sub-network feature map GC(f, Q) bounds the classification error of the nearestmean classifier defined by the feature map f .As in [22] , (5) for some constant c. Then, for any δ > 0, with probability 1 -δ/2 we have the following bound for the generalization errorEQUATION)where p is the embedding dimension of the feature map f andd ij = µ f (S i ) -µ f (S j ) .Note, the outer expectation on the left hand side is taken over the samples S used to produce the classifier h f,S .Proof. This follows immediately from [22, Proposition 5] which gives the bound in terms of the neural collapse instead of the geometric complexity. By using Proposition A.1 we can replace the neural collapse with the geometric complexity and using Proposition 4.2 we then replace the theoretical GC with the empirical GC, yielding the additional term with the Poincaré constant.In Figure 3 , we plot the LHS and RHS of the generalization bound in (10) from Proposition 4.3 for a VGG-13 model trained on CIFAR-10. In this plot, we see that the bound is not vacuous, demonstrating a relatively tight fit. Note that on the RHS, we omitted the term involving the Lipschitz constant L, assuming it is small due to its denominator. Additionally, we estimated a lower bound for the Poincaré constant c by comparing the magnitudes of the RHS and LHS in the inequality (6), based on results in Figure 1 . This approximation yielded c ≈ 1000 in our setup.  
the impact of geometric complexity on transfer learning: 
 Many implicit regularization mechanisms in gradient descent exert a pressure on geometric complexity which in turn constrain the neural collapse. In this section, we discuss the affect of this implicit bias in transfer learning. Specifically, we show with theory how this bias during pre-training can help explain the mechanisms behind transfer learning. Namely, that lower GC in a pre-trained embedding network promotes neural collapse on new target classes, simplifying the fine-tuning process.However, our bound makes it clear that certain compatibility conditions between the pre-training (source) distribution and the fine-tuning (target) distribution are needed, which we argue are satisfied in image foundational models and large language models. At last, we verify empirically that known methods to reduce geometric complexity for the pre-trained embedding result in better performance during fine-tuning on unseen tasks, in agreement with our neural collapse bound for transfer learning. EQUATIONWith this, we can state the transfer learning bound Proposition 5.1. Suppose that the source and target input distributions satisfy a Poincaré inequality with constant c Q and c Q (resp.). Then, with probability 1 -δ, over the selection Q c and Q c of target class distributions, we have that the CDNV expectation for two target classes is bounded by the geometric complexity of the embedding network f in the following wayE Qc =Q c [V f (Q c , Q c )] ≤ c Q GC(f, Q) k -1   i =j 1 d 2 ij   + 8 + 16c Q sup c∈C GC(F * , Q c ) ∆(F * ) 2π log(k)H(F * , Q) (k -1)∆(F * ) +   1 + 4 sup x∈supp(Q) f ∈F * f (x) ∆(F * ) 2      2 log 1 δ c Q sup c∈C GC(F * , Q c ) √ k∆(F * ) 2  where k is the number of source classes,GC(F * , Q c ) = sup f ∈F * GC(f, Q c ), d ij = µ f (Q ci ) - µ f (Q cj ) , and H(F * , Q) is a complexity measure for F * over Q (see Appendix A.3).Proof. This follows immediately from [22, Proposition 2] which proves this bound for the NC and distribution variances. Using our Proposition 4.1, we can replace the NC by the GC and swap the variances by the corresponding geometric complexities via the respective Poincaré inequality.Interpretation. In the bound above, the first term depends on the geometric complexity measured over the source distribution Q and decreases as the number of source classes k increases. Thus, we can predict that lower GC(f, Q) encourages lower NC values (i.e. more neural collapse) on the new target classes provided that the second and third term of the bound are also small.However these other two terms involve both the source and target classes, making the full bound no longer dependent only on the geometric complexity over Q.Moreover these two terms can apriori explode since the infimum over the class-mean distances ∆(F * ) no longer only involves source classes (where this distance is bounded away from zero due to neural collapse into an equidistant simplex) but also target classes (for which we no longer have theoretical guarantees).However, if the source labels are granular enough in the sense the target labels can be represented as combinations of the source labels (as for instance a target label of "dog" can be subsumed as all the breeds of dogs in source classes on ImageNet), then the issues above are mitigated. Formally, we can summarize this granularity compatibility condition between the source and target classes as follows: For each label c of the target class there exists labels c 1 , . . . , c k in the source classes such thatQ c = 1/k(Q c1 + • • • + Q c k ). Because GC(f, •) is linear, we have sup c∈C GC(F * , Q c ) ≤ sup c∈[k] GC(F * , Qc )and thus recover dependence only on source classes.To address the ∆(F * ) term, observe that the compatibility condition above implies in terms of classmeans thatµ f (Q c ) = 1 k (µ f (Q c1 ) + • • • + µ f (Q c k )).Because of neural collapse during pre-training, these source class-means form the vertices of a face in the neural-collapse class-mean simplex making the target-class mean µ f (Q c ) the barycenter of this face. Since the distance between any two points taken in the vertices of a simplex plus its barycenters is bounded away from zero, so is ∆(F * ). Note, albeit intuitive and natural, this granularity condition is hard to verify in practice.When the compatibility conditions above are satisfied, Proposition 5.1 indicates that increasing the amount of neural collapse for the target classes can be achieved by lowering the embedding GC during pre-training. We verify this indeed occurs experimentally using the same regularization techniques exploited in Section 4. Furthermore, we verify that these implicit methods of controlling pre-training GC produce feature maps that perform better on fine-tuning tasks on CIFAR-FS with a ResNet-18 in Figure 4 and on mini-ImageNet with VGG-16 in Appendix A.5.5. 
limitations and conclusion: 
 There is a notable limitation when extending our findings to language modeling and, for example, large language models (LLMs). However, this limitation is not specific to our work per-se, but instead it is a limitation of the application of neural collapse to language models in general, as described in the recent work [65] . Namely, language modeling, as conducted via training by token prediction, creates a classification task where the conditions for neural collapse are implausible. The main problem, in addition to an imbalanced vocabulary, is that the embedding dimension for language models is typically far less than the number of classes (i.e., the total vocabulary size), making the neural collapse simplex impossible to exist. Extending the notion of neural collapse to the language models is an open question and an active area of research and beyond the scope of this work. Uncovering and understanding the implicit biases that enable successful transfer learning is a critical area of research in modern machine learning. Here we provide a framework connecting three different themes behind implicit regularization: flatness of the loss surface, geometric complexity, and neural collapse. We show that the embedding geometric complexity directly controls the neural collapse during training and, moreover, plays a role in the success of transfer learning.Extensive experiments on different image classification and fine tuning tasks across different hyperparameters verify our hypothesis. We believe this opens up an intriguing direction of research further exploring the role of geometric collapse and geometric complexity in deep learning and provides valuable insight for designing more efficient and effective techniques for model training and fine-tuning. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Yes, it does.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: There are no societal impacts. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: There are no risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: No crowdsourcing or research performed with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Proof. Suppose that we have an input probability distribution Q(x) = q(x)dx coming from a data distribution P (X, Y ) (i.e. q(x)dx is the marginal probability distribution of p(x, y)dxdy) where Y represents the k possible classes. Suppose further that P (X, Y ) is a balanced multi-class distribution. In terms of the input distribution Q(x) this means thatQ = 1 k (Q 1 + • • • + Q k )where Q i (x) = q i (x)dx is the input distribution of class i. Suppose moreover that Q(x) satisfies the Poincaré inequality in (5) , that is,Var f (Q) ≤ cE Q ( ∇ x f 2 F ) = c GC(f, Q) for some constant c.Then the statement we want to prove is that the geometric complexity of f , that is,GC(f, Q) := ∇ x f (x) 2 q(x)dxbounds its neural collapse as measured byEQUATIONMore precisely, we want to prove that we have the following inequality:EQUATIONwhered ij = µ f (Q i ) -µ f (Q j )is the distance between the mean of class i and class j. Let us now prove this statement.First of all, since the geometric complexity respects convex sums of data densities, we have that for a distributionEQUATIONIn particular, this means that GC(f, Q i ) ≤ k GC(f, Q). Furthermore, the Poincare inequality ensures that Var Qi (f ) ≤ c GC(f, Q i ). Using these two properties and the definition of the neural collapse, we obtain that:NC(f, Q) = 1 #{i = j} i =j Var f (Q i ) + Var f (Q j ) 2 µ f (Q i ) -µ f (Q j ) 2 ≤ c #{i = j} i =j GC(f, Q i ) + GC(f, Q j ) 2d 2 ij ≤ ck GC(f, Q) k(k -1)   i =j 1 d 2 ij   ,which simplifies to the desired result.Remark A.1. Excepting the Poincaré constant, the quantity on the RHS above given byEQUATIONcould be taken as an alternative measure of neural collapse. We call this the geometric collapse.A.2 Proof of Proposition 4.2 (Estimating the Theoretical GC by GC)Proof. Let Q be an (input) probability distribution defined over R d and let D = {x i } m i=1 of m ≥ 1 points drawn as i.i.d. samples from Q. Given the map f : R d → R p , we denote the empirical geometric complexity of f over D by GC(f, D) which is defined asGC(f, D) = 1 m m i=1 ∇ x f (x i ) 2 F .We begin by showing thatE D∼Q m GC(f, D) = GC(f, Q).This follows by direct computation, keeping in mind that Q is a probability distribution and that the points are independently sampled. We can write Q as Q(x) = q(x)dx and note thatE D∼Q m GC(f, D) = E x1,...,xm∼Q m 1 m m i=1 ∇ x f (x i ) 2 F = 1 m R m×d m i=1 ∇ x f (x i ) 2 F dQ m (x 1 , . . . , x m ) = 1 m R m×d m i=1 ∇ x f (x i ) 2 F q(x 1 ) • • • q(x m )dx 1 • • • dx m = 1 m m i=1 R (m-1)×d R d ∇ x f (x i ) 2 F q(x i )dx i q(x 1 ) • • • q(x i ) • • • q(x m )dx 1 • • • dx i • • • dx m = 1 m m i=1 R d ∇ x f (x i ) 2 F q(x i )dx i = 1 m m i=1 R d ∇ x f (x i ) 2 F dQ(x i ) = 1 m m i=1 GC(f, Q) = GC(f, Q).Now suppose we have two separate independent samples D and D each of of size m ≥ 1 which differ by exactly one point, say x i in D and x i in D . By assumption, the map f is L-Lipschitz for some Lipschitz constnt L > 0. Therefore, we haveGC(f, D) -GC(f, D ) = 1 m ∇ x f (x i ) 2 F -∇ x f (x i ) 2 F ≤ L 2 /m, and similarly, GC(f, D ) -GC(f, D) ≤ L 2 /m. It follows that | GC(f, D) -GC(f, D )| ≤ L 2 /mand by applying McDiarmind's inequality (e.g., [41] ), we have that for any > 0,EQUATIONSince, from the computation above, E D∼µ m [ GC(f, D)] = GC(f, Q) and setting δ/2 = exp(-2m 2 /L 2 ) and substituting for in (16) , we get that for any δ > 0 with probability as least 1 -δ/2 the following holds:GC(f, Q) ≤ GC(f, D) + L log 2 δ 2m .This completes the proof.A.3 Definition of the complexity measure H(F, Q) in Proposition 4.1Let us restate here for the sake of completeness the definition given in Galanti et al. [22] . First consider the Rademacher complexity R(A) of a set A ⊂ R n which is given as the expectation E (X) of the random variable X = sup a∈A , a where = ( 1 , . . . , n ) is a random vector with components uniformly distributed among the two values -1 and 1. Now consider the input distribution Q coming from a multiclass distribution with labels in c ∈ C with label input distribution Q c . We define the complexity measure H(F, Q) for the candidate functions in F over the source distribution Q as the Rademacher complexity of the following set: We trained a VGG-13 neural network on the full CIFAR-10 dataset with the provided architecture [56] and using the standard train/test split. Throughout training, we reported the following metrics measured and averaged over multiple batches of the training dataset: 1) the geometric complexity of the model embedding layer; i.e., the layer on which the neural collapse is measured, 2) the neural collapse as measured by ( 3) measured on the model embedding layer, and 3) the geometric collapse of the model also measured on the penultimate embedding layer of the model and given by{(µ f (Q c ), Var f (Q c )) : c ∈ C, f ∈ F)} .EQUATION)where Q is the training distribution, k = 10 as this was CIFAR-10, and (as in the paper) whered ij = µ f (Q i ) -µ f (Q j )is the distance between the sample means of class i and class j.These quantities were measured every 1000 steps of a very long pre-training of 100, 000 steps. The optimizer was plain SGD (note we obtained similar results with momentum) without any regularization nor schedule to avoid masking effects. We used random crop and random flip for data augmentation. The results were all averaged over 5 random seeds for the neural network parameter default initialization. The plots have no smoothing applied to the learning curves. Top row: We swept over a learning rate range of {0.001, 0.0025, 0.005, 0.01, 0.025, 0.1} with a constant batch size of 512. Middle row: We swept over a batch size range of {8, 16, 32, 64, 128, 256} with a constant learning rate of 0.01. Bottom row: We swept over a L2 regularization rate range of {0.0, 0.00025, 0.0005, 0.001, 0.0025} with learning rate 0.01 and batch size 256. Each sweep took roughly 10h of training on a single Google Cloud TPU V3 accessed via a Google colab.In Figure 5 , we show the complete learning curves of that experiment.We trained a VGG-13 neural network [56] on the full CIFAR-10 dataset using the provided train/test split. For this model architecture we use an embedding layer dimension p = 1024. Note also that the number of examples per class is m c = 5000. We used a constant learning rate of 0.005, a batch size of 512 and trained for 100000 steps reporting metrics every 1000 steps.Throughout training, we reported the following metrics measured and averaged over multiple batches of the training dataset: 1) the empirical geometric complexity of the model embedding layer; i.e., the layer on which the neural collapse is measured, denoted GC, and 2) the sum of the inverse squares of d ij which denotes the distance between the mean of class i and class j for i, j ∈ [k] for i = j and, here, k = 10. These quantities were used to create the blue curve in Figure 3 . We also computed the nearest mean classifier error on the test set. Recall, for the feature map f and a sample S the nearest mean classifier is defined as h f,S := argmin c∈[k] f (x) -µ f (S c ) where, here k = 10, and µ f (S c ) denotes the sample mean of the set S c for class c under the feature map f . To create the orange curve in Figure 3 we plot the average error h f,S (x) = y over the test set. We trained a RestNet-18 neural network with width 1 implemented in Flax https://github. com/google/flax/blob/main/examples/imagenet/models.py on CIFAR-FS with its initial convolution modified to stride 1 and kernel size of 3 to adapt to CIFAR-FS instead of ImageNet. We used only 10 classes for the source datasets with 600 images per class, which further we split in a 10/90 split for the test and train splits. The metrics were computed on an average of 100 random samples of 5 classes in the remaining classes of CIFAR-FS. We reported 1) the geometric complexity of the embedding layer measured on the source dataset (Source GC), 2) the neural collapse as measured by ( 3) measured and averaged on the 100 random samples of 5-label samples from the target dataset, 3) the test accuracy obtained by solving the normal equation directly for a ridge regression with only 5 examples per class obtained from the target set. These three quantities were measured every 1000 steps of a very long pre-training of 100 0 00 steps. The optimizer was plain SGD (note we obtained similar results with momentum) without any regularization nor schedule to avoid masking effects. We used random crop and random flip for data augmentation. The results were all averaged over 5 random seeds for the neural network parameter default initialization. The plots have no smoothing applied to the learning curves. Top row: We swept over a learning rate range of {0.001, 0.005, 0.01, 0.05, 0.1, 0.25, 0.5} with a constant batch size of 256. Middle row: We swept over a batch size range of {8, 16, 32, , 64, 128} with a constant learning rate of 0.005. Bottom row:We swept over a L2 regularization rate range of {0, 0.0001, 0.0005} Each sweep took roughly 10h of training on a single Google Cloud TPU V3 accessed via a Google colab. In Figure 10 , we trained a VGG16 neural network (as described in [57] ) implemented in Flax on Mini-ImageNet (as described in [60] ). We used 10 classes with 600 examples per class for training, with a 10/90 train/test split. To evaluate the downstream performance we used 100 downstream tasks consisting of 5 labels for few shot learning randomly chosen from a pool of 20 separate from the training labels. The experimental setup was the same as described in section A.4.3. Namely, we reported 1) the geometric complexity of the embedding layer measured on the source dataset (Source GC), 2) the neural collapse as measured by (3) measured and averaged on the 100 random samples of 5-label samples from the target dataset, 3) the test accuracy obtained by solving the normal equation directly for a ridge regression with only 5 examples per class obtained from the target set. These three quantities were measured every 1000 steps of a very long pre-training of 100,000 steps. The optimizer was plain SGD (note we obtained similar results with momentum) without any regularization nor schedule to avoid masking effects. We used random crop and random flip for data augmentation. The results were all averaged over 5 random seeds for the neural network parameter default initialization.We swept over a learning rate range of {0.001, 0.005, 0.01} with a constant batch size of 256. Each sweep took roughly 10h of training on a single Google Cloud TPU V3 accessed via a Google colab.Figure 10 : Controlling target NC through source GC on mini-imagenet with VGG-16: Increased learning rates produces lower GC and more neural collapse (i.e. lower NC) resulting in higher 5-shot transfer accuracy.To mitigate possible confounding factors due to batch size and learning rate manipulations, we train a VGG-13 model [57] on CIFAR-10 with explicit GC regularization using a fixed learning rate of 0.01 and a batch size of 256. We explore increasing levels of explicit GC regularization with rates 10 -6 , 10 -5 , 10 -4 and plot the results in Figure 11 . We observe the same predictions as in our experiments which indirectly lower the GC through learning rates, batch sizes and L2 regularization. Namely, lower GC produces lower NC values (i.e., increased neural collapse). Note that in this experiment we regularized w.r.t. the GC computed at the logit layer rather than the embedding GC. This is because the high dimension of the embedding layer makes taking the gradient of the embedding GC prohibitively expensive. Already, regularization with GC taken at the logit layer was possible only for one seed. That being said, explicit regularization by GC computed at the logit layer provides a direct way to control GC computed at the embedding layer.Figure 11 : VGG-13 on CIFAR-10 with explicit GC regularization (one seed).Recall that a distribution Q satisfies a Poincaré inequality if, for all differentiable functions v defined on the support of Q, there exists a constant c > 0 such thatEQUATIONIn this section, we argue that the Poincare Inequality (PI) is a mild and natural assumption to make. For instance, the Gaussian distribution, mixtures of Gaussians with equal (or different) variances, mixtures of Gaussians and sub-Gaussians, mixtures of uniform and Gaussian measures; and any log-concave probability measure all satisfy a PI; see, for example, [1, 54] . The same is true for most distributions of importance in probability and statistics; see [47] .The Poincaré constant c itself can take very high values depending on the spread of the distribution. For example, for a standard normal distribution the Poincare constant is 1 while for a multivariate normal distribution the Poincare constant equals the largest eigenvalue of its covariance matrix which can be any positive real number. Intuitively speaking, the Poincaré constant will increase if the space is stretched in any direction and decrease if it is compressed in any direction.The PI has also been assumed to hold for real life image datasets, where it has been used to help improve GAN convergence, as in [31] . It is also a key assumption to understand the role of overparameterization in generalization as happens for large neural networks (cf. [7] which frames this as an equivalent isoperimetry condition).On the contrary, non-PI distributions are considered pathological; they can be constructed for instance by artificially concatenating distributions with fully disjoint support. See [42] for a construction of pathological examples of distributions not satisfying a Poincaré inequality.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Justification: Yes, the paper provides sufficient information on the computer resources required for each experiment, including the type of compute workers, and execution time, ensuring that the experiments can be accurately contextualized for performance.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 


Decision: This paper studies the relation between metrics of neural collapse (NC) and geometric complexity (GC).
They prove that NC can be bounded in terms of GC under certain assumptions, and they give a generalization bound depending on GC.
They also empirically demonstrate that GC and NC tend to be correlated under common training interventions (learning rate, etc).
Overall, this paper presents a promising case for using GC in understanding neural collapse.

The reviewer scores were borderline. Reviewers agreed that the relation between GC and NC was interesting and novel. The main concerns were about the lack of comprehensive experimental support, and lack of sufficient discussion of limitations.

I recommend acceptance, since reviewers found the ideas interesting and found no technical flaws.
I strongly suggest the authors incorporate reviewer feedback in their camera-ready. Specifically, please add explicit discussion of both the theoretical and the experimental limitations. Be especially careful in making causal claims without specifying your causal graph. Overall, adding such discussion will clarify the scope of your work, and thus make the paper stronger.

Reviews: review: paper_summary: paper_summary: The paper explores the relationship between neural collapse (NC) and geometric complexity (GC). It presents both theoretical and empirical evidence showing that geometric complexity is a robust metric across various variables. By substituting the NC metric with GC, the paper introduces a generalization bound based on geometric complexity. Furthermore, it highlights the significance of GC in transfer learning, demonstrating that by controlling the pre-training GC, downstream NC can also be controlled, leading to improved transfer learning results.
strength_weakness: strength_weakness: Some empirical results can be better aligned with the theoretical counterpart and additional experiments could be added to better support the claims. For example:
* In proposition 4.1, NC is upper bounded by geometric collapse up to some constant scaling. What's the range of this constant $c$, does it purely depend on $Q$? If yes, it would be better to directly plot NC and RHS in (6) in the same plot to demonstrate the validity of the bound; and if no, it would be interesting to show how different settings examined in Figure 1 affect $c$.
* The empirical evidence presented in Figure 2 clearly demonstrates the robustness of geometric complexity. However, I have some concerns regarding the Lipschitz notion. Neural networks are known to be not so Lipschitz, and making them more Lipschitz is an active area of research aimed primarily at improving robustness. Is Proposition 4.2 numerically verifiable by calculating the Lipschitz constant of the tested neural networks?
questions: questions: * In equation (7), does the distance $d$ represent the norm distance? This should be stated clearly.
* Both the paper and the prior work (Galanti et al, 2022) rely on the data assumption where the source and transfer datasets come from the same distribution $Q$. However, in practical settings, source and transfer datasets typically have domain gaps which can potentially violate the data assumption. Would the results in the paper still hold in this case? For example, pre-training on ImageNet/Cifar-100 while fine-tuning on DTD?
limitations: limitations: Yes
paper_summary: paper_summary: This paper examines the relationship between geometric complexity (GC), neural collapse (NC), and transfer learning performance in deep neural networks. The key contributions are:

* Proposing geometric complexity as a measure that connects the flatness of the loss surface and neural collapse
* Deriving theoretical bounds showing GC controls NC under certain assumptions
* Empirically demonstrating that mechanisms affecting GC (learning rate, batch size, regularization) also influence NC
* Showing lower GC during pre-training leads to better transfer learning, especially for few-shot tasks
* Proving a new generalization bound in terms of GC
* Demonstrating GC can be efficiently estimated on small samples

The authors argue that GC can serve as an informative proxy for transfer learning potential and provide a unifying framework for understanding implicit biases in deep learning. By bridging concepts from optimization geometry, representation learning, and transfer learning, the paper aims to provide deeper insights into the fundamental mechanisms driving the success of modern deep learning techniques.
strength_weakness: strength_weakness: **Limited theoretical foundation**: While the paper provides some theoretical results, the assumptions required (e.g., Poincaré inequality) are quite strong and may not hold in practice for real datasets and architectures. The connection between the linear model analysis and deep neural networks is not strongly justified. This is a significant limitation, as it's unclear how much of the theory translates to practical scenarios. The authors could strengthen this aspect by providing empirical evidence that these assumptions hold in realistic settings or deriving weaker results under more general conditions. Additionally, the paper could benefit from a more thorough discussion of the implications and limitations of these theoretical results.

**Experimental limitations**: The empirical evaluation focuses on image classification tasks and relatively small datasets. The generalizability of the results to larger datasets, other domains (e.g., NLP), or more complex architectures is unclear. This narrow focus raises questions about the broader applicability of the findings. For instance, it's unclear whether the relationships between GC, NC, and transfer performance would hold for large-scale pre-training scenarios like those used in modern language models. The paper would be significantly strengthened by including experiments in a broader range of tasks and scales or at least by providing a more detailed discussion of the potential challenges in scaling up the approach.

**Lack of comparison to related metrics**: The paper does not thoroughly compare geometric complexity to other related complexity measures or generalization bounds from the literature. This makes it difficult to assess the relative merits of GC. There's a rich body of work on complexity measures for neural networks, including various forms of capacity measures, stability-based bounds, and PAC-Bayesian approaches. A comprehensive comparison would help situate GC within this broader context and clarify its unique contributions. Such a comparison could also help identify scenarios where GC might be particularly advantageous or potentially suboptimal compared to existing approaches.

**Causality vs correlation**: While the paper shows correlations between GC, NC, and transfer performance, it does not conclusively demonstrate a causal relationship. Alternative explanations for the observed trends have not been thoroughly explored. This is a critical limitation, as it leaves the possibility that GC is merely a proxy for some other underlying factor driving NC and transfer performance. The paper would be strengthened by a more rigorous causal analysis, perhaps through interventional studies or by controlling for potential confounding variables. Additionally, discussing potential mechanisms by which GC might causally influence NC and transfer learning would add depth to the analysis.
questions: questions: 1. How does geometric complexity compare to other measures of model/representation complexity like Fisher information, gradient norm, etc.? A more thorough comparison would strengthen the paper.
2. The theoretical results rely on strong assumptions like the Poincaré inequality. Can you provide empirical evidence that these assumptions hold for real datasets/models? Or derive results under weaker assumptions?
3. The paper shows correlations between GC, NC, and transfer performance but does not conclusively prove causation. Have you considered alternative explanations or confounding factors?
4. How computationally expensive is it to measure GC during training? Is it feasible to use it as a regularization term or early stopping criterion in practice?
5. The generalization bound in terms of GC is interesting, but how tight is it empirically? How does it compare to other generalization bounds in the literature?
6. Have you explored using insights about GC to guide architecture design or improve training algorithms? Demonstrating practical benefits would strengthen the paper.
limitations: limitations: The authors acknowledge some limitations, like the focus on image classification tasks and relatively small datasets. However, they could go further in discussing potential shortcomings:

* The strong assumptions required for the theoretical results may limit their applicability.
* The computational cost of measuring GC is not thoroughly addressed.
* Potential negative consequences of optimizing for low GC are not explored.
paper_summary: paper_summary: The paper studies the relationship between geometric complexity (based on penultimate layer features) and the neural collapse phenomenon (especially, the variability collapse property). The main theoretical result is a bound on the CDNV-based NC metric using geometric complexity (Proposition 4.1). This is then employed to extend generalization and transfer learning bounds (proposed by previous efforts) to rely on geometric complexity. Experimental results on classification tasks (along with transfer learning settings) are presented to validate the proposed relationship.
strength_weakness: strength_weakness: 1. There seems to be a mismatch in the $\log(2/\delta)$ term in Proposition 4.2 vs $\log(1/\delta)$ term in Proposition 4.3, which makes the main results inconsistent (yet fixable). Additionally, why do we have an "expectation" over the "error" term in the LHS of Eq (10)? Isn't the "error" itself an expectation?

2. Although the paper presents generalization bounds, a discussion on the generalization performance of the classifier (VGG-13 for Figure 1) seems to be missing. Only the neural collapse and geometric complexity plots are illustrated but a deeper relationship between the trends and the test performance is lacking. Is there an upper limit of "good" learning rates, and batch sizes above which the geometric complexity can reduce but the test performance can get worse? 

3. Similar to the above point, currently it seems like increasing the learning rate, increasing batch size, and increasing the regularization parameter value results in lower neural collapse values. Ideally, neural collapse is applicable during the terminal phase of training (TPT) where the training accuracy is $1$. Does this mean that all the experiments (at least for Figure 1) lead to TPT during training?

nit: Please fix the notation for the norms as it is unclear if $||.||$ represents $||.||_2$ or $||.||_F$.
questions: questions: 1. Since the geometric complexity-based generalization/transfer learning bounds are the upper bounds of existing neural collapse-based bounds, what can be a potentially new implication of your results? since it is common practice to do a hyper-parameter search over learning rates, batch sizes, and regularization parameter values. 

2. The paper does not discuss any explicit regularization mechanism that aims to reduce the geometric complexity. What are the potential challenges in designing such a regularization mechanism?
limitations: limitations: N/A

Comments: title: Response to rebuttal
Thank you!
comment: Thanks again and thank you for adjusting your score. We are very glad that we were able to address your concern.
Thanks, that has addressed my concern.

Raising the score now.
This is a good question but indeed, taking the value c near 1000 does not make the bounds in Proposition 4.1 and Proposition 4.3 vacuous. Figure 1 in our paper plots the Geometric Collapse (i.e. the RHS of the inequality in Propn 4.1 excepting this constant c) in the middle column and the Neural Collapse (i.e. the LHS of the inequality of Propn 4.1) in the right column. From these plots, we see that by comparing the magnitude of the Neural Collapse and the Geometric Collapse that indeed c ~ 1000 for CIFAR-10 in our setting. 

We also explored directly the tightness of the bound derived in Proposition 4.3 during the rebuttal. Please see the 1-page pdf which we attached to our rebuttal response. Figure 2 of that rebuttal pdf shows empirically the tightness of the bound in Propn 4.3. For that experiment we trained a VGG-13 model on CIFAR-10 with 5 random seeds. The plot shows the LHS of the inequality in Propn 4.3 (i.e. the nearest mean classifier error on the test set ) in blue and the RHS of the inequality in Propn 4.3 (excepting the Lipschitz term which we expect to be negligible due to Fig 2 and (8) of our paper) in orange. 

Note that here we take $c = 1000$, $p = 1024$ (which denotes the embedding dimension of the feature map) and $m_c = 5,000$ (which denotes the number of samples per class). Indeed, the bound is not vacuous and in fact appears quite tight!

Thanks again for your careful and thoughtful read. I hope we've been able to successfully address your concerns.
Thanks to the authors for the detailed response.

Given that the authors said "c ~ 1000 (lower bound) for CIFAR-10 in our setting", would this make the bounds in Proposition 4.1 and 4.3 vacuous?
Thank you again for your review. We hope that the concerns in your original review were sufficiently addressed in our response, particularly, the application of  the Poincare inequality and the role of the constant c in our Propn 4.1. As the discussion period ends, we wanted to see if there is anything that needs additional clarification. Thank you.
Thank you again for your review. We hope that the concerns in your original review were sufficiently addressed in our response. In particular, the justification of  the Poincare inequality assumption for data distributions, the demonstrated empirical tightness of the Generalization bound (our Propn 4.3), and the explicit GC regularization experiments exploring causality of these phenomena. 

We also hope we addressed the concerns regarding experimental limitations particularly with respect to the applicability to modern language models and LLMs. As the discussion period ends, we wanted to see if there is anything that needs additional clarification. Thank you.
We are glad that we were able to address your concerns. And we agree, this in an important (and potentially useful) aspect, despite the inherent computational inefficiencies. Including these experiments has helped to improve the exposition. Thanks again for the suggestion!
Thank you for the responses and also the additional experimental results. 

- In my opinion, the authors discussion on explicit GC regularization is key to improve the contribution. Although it is computationally inefficient, it is always helpful to have experiments which validate the claim in this paper. Based on the rebuttal PDF, it seems like NC indeed reduces when explicit GC regularization is employed. 

- I would request the authors to focus more on this aspect (techniques, limitations, tuning etc of such a regularizer) to further strengthen their paper. Simply showing correlations by tuning common hyper-parameters might not fully justify your work.

I have increased the score. Good luck.