What Deep Representations Should We Learn? -- A Neural Collapse Perspective
Authors: Xiao Li
Sheng Liu
Jinxin Zhou
Carlos Fernandez-Granda
Zhihui Zhu
Qing Qu
Venue: Submitted to ICLR 2023 (2023)
Keywords: representation learning
neural collapse
transfer learning

Abstract: For classification problems, when sufficiently large networks are trained until convergence, an intriguing phenomenon has recently been discovered in the last-layer classifiers, and features termed neural collapse (NC): (i) the intra-class variability of the features collapses to zero, and (ii) the between-class feature means are maximally and equally separated. Despite of recent endeavors to understand why NC happens, a fundamental question remains: whether NC is a blessing or a curse for deep learning? In this work, we investigate the problem under the setting of transfer learning that we pretrain a model on a large dataset and transfer it to downstream tasks. Through various experiments, our findings on NC are two-fold: (i) when pretrain models, preventing intra-class variability collapse (to a certain extent) better preserve the structures of data, and leads to better model transferability; (ii) when fine-tuning models on downstream tasks, obtaining features with more NC on downstream data results in better test accuracy on the given task. Our findings based upon NC not only explain many widely used heuristics in model pretraining (e.g., data augmentation, projection head, self-supervised learning), but also leads to more efficient and principled transfer learning method on downstream tasks.

Under review as a conference paper at ICLR 2023
WHAT DEEP REPRESENTATIONS SHOULD WE LEARN? - A NEURAL COLLAPSE PERSPECTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
For classification tasks, when sufficiently large networks are trained until conver- gence, an intriguing phenomenon has recently been discovered in the last-layer classifiers, and features termed neural collapse (NC): (i) the within-class variabil- ity of the features collapses to zero, and (ii) the between-class feature means are maximally and equally separated. Despite of recent endeavors to understand why curse
NC happens, a fundamental question remains: whether NC is a blessing for deep learning? In this work, we investigate the problem under the setting of transfer learning that we pretrain model a large dataset and transfer it downstream tasks. Through various experiments, our findings on NC are two- fold: (i) when pre-training models, preventing intra-class variability collapse (to a certain extent) better preserve the structures of data, and leads to better model transferability; (ii) when fine-tuning models downstream tasks, obtaining fea- tures with more NC on downstream data results in better test accuracy on the given task. Our findings based upon NC not only explain many widely used heuristics model pretraining (e.g., data augmentation, projection head, self-supervised learn- ing), but also leads to more efficient and principled transfer learning method on downstream tasks.
INTRODUCTION
Recently, an intriguing phenomenon has been discovered in terms of learned deep representations, in which the last-layer features and classifiers collapse to simple but elegant mathematical structures on the training data: (i) for each class, the intra-class variability of last-layer features collapses to zero, and (ii) the between-class class means and the last-layer classifiers all collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling. This phenomenon, termed Neural Collapse (NC) (Papyan et al., 2020; Han et al., 2022), has been empirically demonstrated to persist across a variety of network architectures and datasets. Theoretically, more recent works (Fang et al., 2021; Zhu et al., 2021; Zhou et al., 2022; Tirer& & Bruna, 2022) justified the prevalence of NCC under simplified unconstrained feature models across a variety of training losses and problem formulations.
Despite of recent endeavors of demystifying such an interesting phenomenon, fundamental ques- tion lingers: NC a blessing or a curse for deep representation learning? ? Understanding such a question could address many important but mysterious aspects of deep representation learning. For example, quite a few recent works (Papyan et al., 2020; Galanti et al., 2022; Hui et al., 2022) studied the connection between ACC and generalization of overparameterized deep networks. In this work, we aim to understand transfer learning by studying the relationship between NC and the transfer- ability of pretrained deep models. Transfer learning has become an increasingly popular approach in computer vision, medical imaging, and natural language processing (Zhuang et al., 2020). With domain similarity, a pretrained large model on upstream datasets is reused as starting point for fine-tuning a new model model reuse with fine-tuning significantly reduces the computational cost, and achieves superior performances problems with limited training datasets.
much smaller downstream task (Zhuang et al., 2020). The pretrained
However, without principled guidance, the underlying mechanism transfer learning is not very well understood. First, when we are pretraining deep models on the upstream dataset, we lack good metrics for measuring the quality of the learned model or representation. In the past, people tended to rely empirically on controversial metries for predicting the transferred test performance, such as the validation accuracy on the pretrained data (e.g., validation accuracy on ImageNet (Kornblith et al., 2019)). For example, some popular approaches (e.g., label smoothing (Szegedy et al., 2016)

---

Under review as a conference paper at ICLR 2023
and dropout (Srivastava et al., 2014)) for boosting ImageNet validation accuracy turn out to hurt transfer performance on downstream tasks (Kornblith et al., 2021). Additionally, when pretraining deep models, many methods improve transferability, such as the design of loss functions, data aug- mentations, increased model size, and projection head layers (Chen et al., 2020; Khosla et al., 2020), designed largely based upon trial-and-error without much insight of the underlying mechanism. Second, given the pretrained models, how to efficiently fine-tune the model on downstream tasks remains an open question. Although fully fine-tuning all the parameters of the pretrained model achieve the best performance, it becomes increasingly expensive as the model size grows (e.g., GPT-3 and transformer (Brown et al., 2020; Vaswani al., 2017; Devlin al., 2019; Dosovitskiy et al., 2021)). All these challenges call for a deeper understanding of what makes pretrained deep models more transferable.
Contributions of this work. In this work, we provide a comprehensive investigation of the rela- tionship between the transferability of pretrained models and NC. As NCC implies that intra-class variability for each class collapses to zero, the representations learned via vanilla supervised learn- ing fails to capture the intrinsic dimensionality of the input data, and hence they often result in poor performance of transferability. Intuitively, to make the pretrained models transferable, for each class the learned features should be discriminative but diverse enough that they can preserve the intrinsic structures of the imput data. On the other hand, on the downstream task when we fine-tune pretrained models, we desire more collapse of the features on the downstream training data.
Based upon such intuitions, we adapt the metrics for evaluating NCC to measure the quality of learned representations in terms of both intra-class diversity and between-class discrimination. As such, not only can we demystify several heuristics that are widely used in transfer learning, but it also opens a door for designing methods to transfer large pretrained models more effectively. In words, our experimental findings based upon the NC metrics can be summarized as follows.
The transferability of pretrained models correlates with Icarned feature diversity on the source dataset. By evaluating the NC metrics on different loss functions (Hui & Belkin, 2020) and several widely used techniques in transfer learning (c.g.. the addition of projection head, different data augmentations (Chen et al., 2020; Chen & He, 2021; Khosla et al., 2020) and ad- versarial training (Salman et al., 2020; Deng et al., 2021)), we find that to a certain extent, the more diverse the features are, the better the transferability of the pretrained model. This helps to explain the underlying mechanism of many popular heuristics for transfer learning.
More collapse of fine-tuned models leads to better test performance on downstream tasks. In contrast, when we are evaluating different pretrained models on downstream tasks, we observe that more collapsed features on downstream data usually lead to better transfer accuracy. This phenomenon not only happens on the penultimate layer across different pretrained models, but also across different layers of the same pretrained model.
pretrained models can be more effectively transfered through NC. Efficient and effective transfer learning is of paramount importance for large models nowadays. Inspired by the above findings, with the aim to collapse the features of the penultimate layer, we improve the transfer ef- fectiveness while maintaining efficiency by only tuning one additional layer along with an add-on skip connection. We demonstrate that such a transfer learning strategy achieves better perfor- mances compared with the traditional fixed feature transfer learning setting and on-par or superior performances compared with full model fine-tuning setting.
Relationship to prior arts. The prevalence of NC phenomenon has caught significant attention both in practice and theory recently, and our work draws the connection between ACC and transfer learning. On the other hand, few recent works are investigating the properties of deep representa- tions for transfer learning, which is also related to ours. We'd d like to summarize and briefly discuss those results below.
Understandings of the NC phenomenon. There are a line of recent works deciphering training, generalization, and transferabilityy of deep networks in terms of NC, that are related to ours (see recent review work (Kothapalli et al., 2022)). For training, recent works showed that A/C happens for variety of loss function and formulations, such cross-entropy (CE) (Papyan et al., 2020;
We find that there is a certain threshold, that the transferability increases with the feature diversity below the threshold but decreases or become uncorrelated beyond it. Increasing the feature diversity will decrease the margin upon the threshold and hence the relationship with transferability becomes more involved with too large feature diversity.
2

---

Under review as a conference paper at ICLR 2023
lu2, 2022; Zhu et al., 2021; Fang at., 2021; et al., 2022; Yaras et al., 2022), mean- meean-suuared error (MSE) (Mixon et al., 2020; Han et al., 2022; Zhou et al., 2022; Tirer & Bruna, 2022; Rangamani & Banburski-Faben generalization, the work (Galanti et al., 2022) shows that NCC also happens on test data drawn from the same distribution asymptotically, but not for finite samples (Hui et al., 2022). Other works (Hui et al., 2022; Papyan, 2020) demonstrated that the variability collapse of features is actually happening progressively from shallow to deep layers, and Ben-Shaul & Dekel (2022)
2022), and supervised contrastive (SupCon) loss (Graf et al., 2021). For
showed that test performance can be improved when enforcing variability collapse on features of intermediate layers. The works Xie et al. (2022); Yang et al. (2022); Thrampoulidis et al. (2022) studied problems with imbalanced training data, showing that fixing the classifier simplex ETFs improves test performance on imbalanced training data and long-tailed classification problems. For transferability, the work Kornblith et al. (2021) implicitly showed that there is tradeoff between variability collapse and transfer accuracy by experiments on a varicty of loss functions. Representation learning and model pretraining. There are quite few recent works studying the factors that affect transferability of pretrained models, but the results are largely inconclusive. For example, the work Kornblith et al. (2019) argues that models pretrained on Imagenet with higher accuracy tend to perform better on other downstream tasks. However, such a conclusion has been challenged by more recent works (Kornblith al., 2021; Nayman et al., 2022). These results showed that the training loss and diversity of the features could be more important fac- tors of transferability than the pretrained accuracy. However, compared to our work, they only study few aspects (e.g., training loss) of deep network architectures that affect transferability, and they only focus on the diversity aspect on the source dataset. At the same time, the work Islam et al. (2021) showed that models learned using contrastive type of loss functions could have bet- ter transferability, and Wang & Isola (2020) showed that their representations are more uniform over hyperspheres. The architecture and depth of CNNs were also shown to impact transfer per- formance (Azizpour et al., 2015). Other work Zhang et al. (2019) exploited the importance of layers in overparameterized networks, suggesting the shallow and deep layers are more important in fine-tuning pretrained models for downstream tasks.
2 EVALUATING REPRESENTATIONS OF PRETRAINED MODELS VIA NC
In this section, let us first give brief overview of the NC phenomenon, upon which we introduce the metrics for evaluating the quality of learned representations for transfer learning in Section 3. Basics of deep neural networks. Let us first introduce some basic notations by considering a be the
multi-class (e.g., K class) classification problem with finite training samples. Let [m²) number of training samples in each class and we assume the training samples are balanced n = n1 f22 -.. Let Ta,i denote the ith input data in the kth class (1 S i S n, 1 S k S K), and we use y/ E R to denote a one-hot training label with only the keth entry equal to unity. Thus, given any input data Tk,i. deep network fits the corresponding (one-hot) training label ys via òe (Ik.))
yk R (zk,i)
WL linear classifier W feature he. (0(00).))
+ bL.
(1)
where W- = WW is the last-layer linear classifier and hE = h((x,,) = de (xx,)) denotes a
deep hicrarchical representation (or feature) of the input æR,i- Here, for a L-layer deep network (x)) each layer is composed of an affine transformation, followed by a nonlinear activation (-) and normalization functions (e.g., BatchNorm (loffe & Szegedy, 2015)). We use θ to denote all network parameters of (x) and 0 to denote the network parameters of (z)). Additionally, we use
hEn] E gdxn, 1 k: S K, to denote all the features in the matrix form. Additionally, we write the class mean for each class as
H = H
H2
Hg] E R
H₃ = [he,1
H hr
hg] E Ba and h
hk 1skS
Accordingly, we denote the global mean of H as hG - EiN Fug. A review of neural collapse. Over the training dataset {Z last-layer features H and classifiers W of trained network exhibit simple but elegant mathematical structures (Papyan et al., 2020; Papyan, 2020), that we highlight two key properties below?
is y4 ). has been widely observed that
2 Additionally, self-duality convergence has also been observed in the sense that wy c' c. for some O, hut this not the main focus of this work.
3

---

Under review as a conference paper at ICLR 2023
Intra-class variability collapse: for each class, the last-layer features collapse to their means,
h + h&s visisn n, 1 1 k: < K.
Maximum between-class separation: the class-means [1 ] k=) centered at their global mean hG are not only linearly separable, but are actually maximally distant and they form a Simplex Equiangular Tight Frame (ETF): for some e > 0, H = -h he hx hal satisfies
HH-
H
ck
IK IK- - 1
K -1
Recent work shows that NC persists across a range of canonical classification problems, on different loss functions (e.g., CE (Papyan et al., 2020; Fang et al., 2021), MSE (Mixon et al., 2020; Zhou et al., 2022: Han et al., 2022), SupCon (Fang et al., 2021; Graf et al., 2021)), on different neural network architectures (e.g., VGG (Simonyan & Zisserman, 2014), ResNet (He et al., 2016), and DenseNet (Huang al., 2017)), and on a variety of standard datasets (e.g.. MNIST (LeCun al., 2010), CIFAR (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009)). As we observe from above, although the maximum between-class separation suggests the learned features are discriminative in (3), the intra-class variability collapse memorizing the labels rather than preserving the intrinsic structures the data. As such, the loss of information of the input data could be detrimental for feature transferability.
single dimension in (2) implies that the network is
Measuring feature quality NCC metrics. Based upon above discussion, the study of NC offers us new metrics to evaluate the transferability of pre-trained models.
Variability and separation collapses (Papyan et al., 2020; Zhu al., 2021). We can measure the variability and separation collapse via
NC1 = K Irace (Iw
(4)
by measuring the magnitude of the within-class covariance Ew E Rddd of the learned features compared to the inter-class covariance ΣB E Raxd, where
K Σ hh hc) (Trs hc)
(ha.a h&)) (he.: - h&) h2) EB = K
Ew
nk
and Σ denotes the pseudo inverse of Eg. Here, ΣS serves as an normalization for Ew to capture the relativity between the two covariances. Intuitively, if the features their means, the smaller NC, is; on the other hand, with the same Ew. if the features have more separated class means, NC,, would also be smaller. Since the metric involves pseudo inverse of En computation of such a metric in (4) could be intractable when the feature dimension is large (for huge models).
collapse to
METHODS & EXPERIMENTS
In the following, we will utilize the above metrics to experimentally evaluate the quality of learned representations in terms of transferability, to corroborate our claims in Section 1. More specifically, in Section 3.1 we focus on pre-traine pre-training, that we demonstrate a positive correlation between NCC metrics and transfer accuracy on pre-trained dataset. In Section 3.2, we turn our attention to down- stream tasks, where we discover the ACC metrics and the associated transfer accuracy are negatively correlated. Finally, in Section 3.3. based on the above findings, we propose a simple and efficient fine-tuning method in a principled manner.
Experimental setups. We treitrain ResNet18 and ResNet50 (He et al., 2016) models Cifar-100 (Krizhevsky et al., 2009) and MinilmageNet (Vinyals et al., 2016) datasets. Unless otherwise spec- ified, we pre-train the models for 200 epochs using the SGDR learning rate scheduler (Loshchilov & Hutter, 2017) with initial learning rate 0.1 and minimum learning rate 0.0001. More details are postponed to Appendix A.2.
3.1 STUDY OF NC & TRANSFER ACCURACY ON MODEL. PRETRAINING
We begin our investigation by studying the relationship between NCC metrics on pre-training dataset and transfer accuracy. To certain extent, we find the two are positively correlated: larger NC metrics often leads better transfer accuracy. Our key intuition is that if the learned representations are less collapse on the pre-trae pre-trained data, they better preserve the intrinsic structures of the input data.
4

---

Under review as a conference paper at ICLR 2023
Training MSE (wlo proj.)
Cross-entropy (wlo proj.)
SupCon (w/ lincar proj.)
SSpCoon (w/ mlp proj.)
NC,, (Cifur-100)
0.001
0.771
0.792
2.991
Transfer Acc.
53.96
71.2
19.9
79.51
Table I: Transfer learning results model N comparison among different training settings. 8 models are pre-trained on the Cifar-100 dataset and transfered on Cifar-10. We use proj. to denote projection head.
CE
MSE
SupCon
Supcin my Invaar
Supian se anstayarMup
MBE two eyers VSE treee layers
Supiar wi onolaye
Trnsfer Acc
Trasser ACC
wy oyger
Transfer
prijed Read layer
CD sac layers
mime syere
14:3
ACI (Cffar110))
.VCG (Cifar-100)
NO (Cifar-100)
Figure 1: Trend of NCi during training and transfer learning accuracy of the pretrained mod- els. ResNer50 models are pretrained using Cifar- 100 dataset with CE loss (Left), MSE loss (Mid- dle) and SupCon loss (Right). Models are pretrained with different numbers of layers for projection heads and transfered on the Cifar-10 10 dataset.
Choices of training losses and architecture impact feature variability and hence transferability. First, we show that the choice of training losses and design of architecture substantially affects the collapse of the features on the penultimate layer, and hence the transfer accuracy. To show this, we pre-train ResNet18 models on the Cifar- 100 dataset with three different choices of loss functions (CE, MSE (Hui & Belkin, 2020), and supervised contrastive loss (SupCon) (Khosla et al., 2020)), and then test the transfer accuracy on the Cifar- 10 dataset. For the SupCon (Khosla et al., 2020), we follow the original setup that uses nonlinear multi-layer perception (MLP) module as projection head after the ResNet18 encoder. Once the model has been pre-trained, the projection head will be abandoned and only the encoder network will be utilized as the model for downstream tasks. In Table 1, we summarize the results of model NC₁ and transfer accuracy for different training scenarios. As we can see, the model trained with the MSE loss severely collapsed on the source dataset and has the worst transfer accuracy. On the other hand, the model trained using the SupCon with an MI. projection head is less collapse compared to the other models and has superior transfer performance. This observation is consistent with (Islam et al., 2021) that SupCon leads to better transferability than CE. Here, we further show that this is because SupCon learns relatively more diverse features (larger i) than CE. However, given both SupCon and CE learn the same NC representations in the so-called unconstrained features model (Fang et al., 2021; Zhu et al., 2021; Graf et al., 2021), it raises interesting question on why the features learned by SupCon are more diverse. We further address this question in the following.
Projection layers in pre-training increase feature diversity for better transferability. We con- jecture that the inclusion of MIP projection head serves as important role for the better per- formance of SupCon. This is based on the recent works (Papyan, 2020: Hui et al., 2022; He & Su, 2022) that show the within-class variability collapse actually happens progressively from shallow to deep layers: the closer to the final layer, the severer the variability collapse (i.e., the smaller the NC, of that layer). Therefore, the additional MLP projection layers prevent severe variability collapse of the encoder network. We empirically verify this conjecture by replacing the MLP by a linear layer for the projection head in SupCon and then train the model under the same setting. We observe from Table I that SupCon with linear projection head indeed achieves similar NCC and transfer accuracy as CE. Note that the layer-wise progressive collapse is universal across the choice of training losses, the usage of projection head is not limited to contrastive losses. Thus, can also improve the trans- ferability for supervised pre-training models with the CE/ / MSE loss by adding a MLP projection head. To verify this, we pre-train ResNet-50 models (we use it for better performance) the Cifar- 100 dataset. We report the NC and transfer accuracy for different choices of projection heads. In
For the choices of loss, the works (Islam et al., 2021; Kornblith et al., 2021) have similar observations, where different choices of training losses for pre-training lead to different transfer performance.
5

---

Under review as a conference paper at ICLR 2023
Figure 1, we can observe that the inclusion of projection heads substantially increase the diversity of the representations and the transfer accuracy compared with that of no projection head. We note that the usage of projection head is first introduced and popularized for self-supervised learning (Chen et al., 2020; Chen & He, 2021). But the reasons behind the success of such a method remain as mystery. Here, demystify the underlying mechanism of projection head based upon progressive variability collapse, where our experiments show that the usage of projection heads is not limited to pre-training by contrastive losses.
Pre-training NCC does not always explain transfer accu- racy. So far, we have shown that models that are very col- lapsed on the source dataset have inferior transferability, which aligned with the observation in (Hui et al., 2022). However, does this imply the truth of the opposite direction: models that are not collapsed on the source dataset will always transfer better? The answer is obviously no. A naive counter exam- ple would be an untrained model with randomly initialized weights, which will not collapse but also has poor transfer ac- curacy. Because the reason for the increase in NC1 could be either from the increase of feature diversity within each class or from the loss of discriminate power between classes. To more comprehensively characterize this relationship, we pre- train ResNet50 models on the Cifar- 100 dataset using different levels of data augmentations and different adversarial training (Madry et al., 2018; Salman et al., 2020; Deng et al., 2021) strength, and then test the transfer accuracy on the Cifar- 10 dataset, where the result is summarized in Figure 2. We ob- serve that positive relationship between the level of collapse on the source dataset and the transfer- ability only holds up to certain threshold based upon the NC1, metric. As shown in Figure 2, if the feature variability is beyond the threshold, the transfer accuracy decreases as the feature variability increases (i.e., NC, increases).
Transfer Acc
bel3
Lul4
tald
Lel5
bals
14
.C1
Figure 2: NC1 vs. Trans-
fer learning accuracy. Models are pretrained using the Cifar-100 dataset with different data augmen- tation levels and adversarial train- ing strength, transfer accuracy is evaluated on the Cifar- 10 dataset.
3.2 STUDY OF NCC & TRANSFER ACCURACY ON DOWNSTREAM TASKS
Second, given the pre-trained models, on downstream data we experimentally investigate the rela- tionship between NC metrics of their representations and transfer accuracy. Transferring pre-trained trained large models to smaller downstream tasks has become the dominant approach in both vision (Doso- vitskiy al., 2021) and language (Devlin et al., 2019) domains. Also, many cases the source data for pre-training is unavailable, 3 we can only evaluate the representation quality based upon the downstream data.
Here, to control the factors for affecting our study, the experimental setting is quite simple: for each downstream task, we freeze the whole pre-trained model with no fine-tuning, and for the model we only train the linear classifier on the downstream data. On the contrary to model pre-training, our discovery is that the transfer accuracy is negatively correlated with the ACC metrics on downstream data. In other words, the more collapsed the representations of the layer the downstream data or simpler dataset, the better the transfer accuracy. Moreover, as we show in the following, this phenomenon is quite universal: it not only happens across pre-trained models with different pre- training strategies, but also across different layers of the same pre-trained model.
Pre-trained models with more collapsed last-layer features result in better transferability. To validate our statement, we pre-train different ResNer50 models on the Cifar-100 dataset by using different levels of data augmentations and different levels of adversarial training strengh. Once a model is pre-trained, we test its transfer accuracy on 4 downstream datasets: 10 (Krizhevsky et al., 2009), FGVS-Aireraft (Maji et al., 2013), DTD (Cimpoi et al., 2014) and Oxford-IIIT-Pen (Parkhi et al., 2012) datasets. In Figure 3, we observe that the NC₁, on Cifar- 10 dataset has negative (almost linear) correlation with the transfer accuracy on different downstream tasks. The more collapse of feature the Cifar- 10 dataset (i.e. the smaller the NC1 metric is), the higher the
"We note the work (Kornblith al.. 2021) studied similar correlation by introducing notion called class separation. However, the work has not studied the aspects when the class separation approaches 0 which similar to entirely not collapsing with the ACC notions. ee.g. JFT dataset (Sun et al., 2017), which is used in the pretraining of Vision Transformer, is not publicly available.

---

Under review as a conference paper at ICLR 2023
Tansfer Acc. on 10
Pansfer Acc, on Alrcoot
Trarster Acc. on OTTO
Trander Act. on FET
Ac.
Acc.
Trane
Figure 3: Transfer accuracy on different downstream tasks and NC1. We pre-train ResNet50 models Cifar-100 using different levels of data augmentation or adversarial training. NC₂ is measured on the downstream Cifar-10 dataset.
transfer accuracy. As such, we observe that the NC1 metric on Cifar- 10 evaluated on pre-trained models can serve good performance indicator for the transfer accuracy on downstream tasks.
The opposite relationships between NC metrics and trans- fer accuracy on pre-traine trainer data and downstream data may seem contradictory at first glance. However, for pre- training,
Classter
New Classrlier
explained in Section 3.1, we desire less col- lapsed models on source data so that the learned features can capture the structure of the input data. On the contrary, on downstream data, the NC metrics act as a measure on model's ability of representing the input data to fit the out- put label. As such, the less collapse on pre-trained data, the better representation power of the pre-trained model. Therefore, the larger NC1 on pre-trained data translates to smaller ,C, on downstream data and better transfer accu- racy.
Layer
Layar
Source
Downstrear
Pre-training
Transfer
Figure Illustration of layer-wise transfer learning. We pre-train MLP model and use the outputs from each layer to do transfer learning.
Layers with more collapsed output features result in better transferability. More intriguingly, the phe- nomenon we observed above not only happens across dif- ferent pre-trained models, but also happens across (the out- puts of) different layers on exactly the same pre-trained model. More precisely, given the same pre-trained model, as shown in Figure 4, we use the output of each individual layer as a "feature extractor", and we test the transfer accuracy of the given layer by training a linear classifier on top of it. Surprisingly, if the outputs of the layer are more collapsed, using the corresponding features leads to better transfer accuracy, which happens regardless of the layer's depth.
To corroborate our claim, we pre-train a 12-layer MIP network with the same output dimension for every layer, 7 and we evaluate the NC,, metric on the layer's output feature upon the downstream data. To evaluate the transfer accuracy of the given layer, we train a linear classifier on top of the layer's output based upon downstream training data. As we observe in Figure 5, the smaller NC1 of the features of given layer, the better transfer accuracy we get. We can observe near linear relationship between NC,, and transfer accuracy from Figure 5 (Right). Thus, the transfer accuracy is more correlated with the variability collapse upon the layer rather than the depth of the layer. Furthermore, the phenomenon we illustrated above holds more universally beyond the vanilla MI. architecture. To demonstrate this, we conducted the same experiment on ViT-B (vision transformer base model) (Dosovitskiy et al., 2021) by using a pre-trained checkpoint released online 8. Similarly, we compute the NC1, of each layer and train a linear classifier for each of the encoder layers in the VTT-B. The results are reported in Figure 6, where we can observe the same trend and more linear relationship compared to the MLP experiment.
'When evaluating the relationship between A'C, and transfer accuracy measured on the same downstream dataset, the correlation is not as strong as we find on Cifar- 10 dataset, we leave the results and discussion in Appendix A.3.
We choose the MLP architecture instead of other architectures (e.g.. ResNet) is to get rid of the influence of feature dimensions for fair comparisons across layers. For MI.P, we can freely choose the the output dimension of each layer. However, the layer's output dimension in ResNet is usually fixed and varies significantly across different depth.
'The Vit-B model checkpoint we used could he found here.

---

Under review as a conference paper at ICLR 2023
2 -0.77
Transfer Accuracy Fitted Line
Thansfor Accurs Dy
Trandfer Accurscy
Transfer Acc
MEP Layers (shallow za deep)
.101
Figure 5: NC1 and transfer learning accuracy of different layers from pretrained MLP model (Left); Nearly linear relationship between transfer learning accuracy and NC, (Right). The 12 layer, 3072 hidden dimension MLP model is pretrained the Cifar-100 dataset and transfered on the Cifar- 10 dataset. NC1 is evaluated on the Cifar- dataset.
0.946
25
Transfer Accuracy Fitted Line
20
Transfer Accur accurrryy
Tranderr Accuracy
20-
10
NCC
50
VIt8 Layers (shallow >> deep)
Figure 6: NC,, and transfer learning accuracy of different layers from pretrained VIT-B model (Left); Nearly linear relationship between transfer learning accuracy and WC, (Right). We use pretrained ViT-B model and transfered on the Cifar- 10 dataset. NC1 is evaluated on the Cifar-10 dataset.
3.3 A SIMPLE & EFFICIENT FINE-TUNING STRATEGY FOR IMPROVING TRANSFERARILITY
Finally, we show that the phenomenon we discovered Section 3.2 can be very useful for designing simple yet more efficient fine-tuning strategies without sacrificing the performance. For vision tasks, model transfer learning (Kornblith et al., 2019; 2021; Deng et al., 2021: Salman al., 2020) typically adopts two strategies: (1) fixed feature training, use pre-trained model to the penultimate layer as a feature extractor and only train new linear classifier on top the features for a downstream the pre-trained model whole model to fit a downstream task. However, the full fine-tunen tne-ttnnng could be very expensive for large models, while fixed feature training often results in worse performance without adapting the model to the downstream data. Thus, the middle case, fine-tuning only a selected subset of layers, could be a promising approach for balancing the computation and performance, while it is seldom studied for vision problems (Utrera et al., 2021; Shen et al., 2021). Based upon the correlation between penultimate layer collapse and the transfer learning performances discovered in Section 3.2, our conjecture is that
task; (2) full model fine-tuning,
initialization and fine-tune the
The topmost transfer accuracy can be achieved by selectively fine-tuning the layers such that the features of penultimate layer are the most collapsed the downstream training data.
Towards this goal, in this work we propose simple strategy to increase the level of collapse of the penultimate layer- fine-tuning one additional layer besides the final linear classifier. We notice that by fine-tuning only one additional layer, the penultimate NC can be substantially decreased along with better transfer performance. More specifically, in table 2, we examine this simple strategy on various architectures and source / downstream datasets, where for each architecture, we pick several layers from the network, fine-tune each of them with the linear classifier and report the maximum transfer accuracy. As observe, the additional one layer fine-tuning leads to substantial performance gain compared with linear probing regardless of pre-training scenarios and downstream datasets.
8

---

Under review as a conference paper at ICLR 2023
Backbone
ResNet18
ResNet50
Vit-B
Dataset
Cifar-10
Cifar-100
Cifar- 10
Cifar- 100
Aircraft
DTD
PET
Linear Probe
43.12
23.84
49.70
29.71
43.65
73,88
92.23
Layer FT
82.00
54.03
85.03
59.39
65.83
77.13
93.02
SCL FT
84.83
58.51
86.14
64.90
65.80
77.34
93.19
Full Model FT
85.61
55.75
88.54
53.42
64.57
76.49
93.02
Table 2: Transfer learning results for Linear probing, layer fine-tuning, SCL fine-tuning and full model fine-tuning on various downstream datasets. We use released ResNet models pre- trained on ImageNet- Ik (Deng et al., 2009) and ViT-B model pre-trained on JFT (Sun et al., 2017) and ImageNet-2Ik (Ridnik et al., 2021) datasets.
Despite the success of the above simple approach, we note that fine-tuning a shallow layer could only implicitly impact the NCC dynamics of the penultimate layer because all the intermediate layers remain un un-trained trained.d.. To make the impact more direct, as shown in Figure 7, we add skip connection from the fine-tuned layer features the penultimate layer features and use the combined features to serve as the inputs for the final linear classifier. Such method enables the network to more effectively fine-tune the selected layer by explicitly passing the learned information to the classifier without suffering the information loss through the cascade of intermediate layers. On the other hand, since we are using a skip connection rather than discarding the top layers and using the fine-tuned layer features directly for classification, the network still benefits from being deep. We term method SCL. (Skip Connection Layer) fine tuning and compare its results with layer fine-tuning, linear probe and full model fine-tuning in Table 2. We observe that SCL fine-tuning nearly always outperforms layer fine-tuning and achieves comparable or even better results compared with full model fine- tuning.
4 DISCUSSION & CONCLUSION
Twofold relationship between NC and transferability. In this work, we examine the relationship between NCC and transferability through various aspects. Previous work (Hui et al., 2022) points out that NC is mainly opti- mization phenomenon which does not necessarily relate to generalization (or transferability). Our work, one hand, corroborates with the finding that pretraining NCC does not always suggest better transferbality, but also shows a pos- itive correlation between pretraining NC and transferabil- ity certain extent. On the other hand, our work also shows that downstream NCC on a dataset where NCC is well- defined correlates with the transfer performances across different datasets and thus could be a general indicator the transferability. This suggests that ACC may not be merely optimization phenomenon. An important future direction we will pursue is to theoretically understand the connection between transferability and ACC.
Layera Frozen
Trained Layers
Tublic Layer
Semertion
Tramed Layer
Theme Luwer
Dada
Data
Layer FT
FT
Figure 7: Illustration of SCL fine- tuning.
Boost model transferability by insights from AC. Our findings can be used improve model transferability through the following two perspective. First, the positive correlation between pre- training NC and transferability suggests that increase diversity of features certain extent can improve transferability. This can be achieved by popular techniques such as multi-layer projection heads and data augmentation. We believe other principled approaches could also be developed by explicitly working on the geometry of the representations. Second, by demonstrating the close cor- relation between downstream NCC and associated transfer accuracy, we are able to make simple yet effective strategies to do model transfer learning. However, simple approach is by means the optimal method to exploit such relationship. We believe there are more powerful approaches that could utilize this phenomenon better and thus achieves better transferability. We leave this as future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis, 59:224 241, 2022. ISSN 1063-5203. doi: https://doi.or//1// aah......cch...221........ Special Issue on Harmonic Analysis and Machine Learning.
Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson. Fac- tors of transferability for generic convnet representation. IEEE transactions pattern analysis and machine intelligence, 38(9): 7900-1802 2015.
Ido Ben-Shaul and Shai Dekel. Nearest class-center simplification through intermediate layers. arXiv preprint htt.:22011.89244, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. 2021 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15745-15753, 2021.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Zhun Deng, Linjun Zhang, Kailas Vodrahalli, Kenji Kawaguchi, and James Zou. Adversarial train- ing helps transfer learning via better representations. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL https: ppppeeveeeeeveeeeeeeeeeeeeeeeeee net. forum? i id= f= @o=f@ggg--W7
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04880 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x words: Transformers for image recognition at scale. ArXiv, 0bs/2010.11929. 2021.
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer- peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43) 2021.
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in trans- learning. International Conference on Learming Representations, 2022. URL https: / /poeneevieww net
Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised con- strastive learning. In International Conference Machine Learning, pp. 3821-3830 PMLR, 2021.
X.Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamics on the central path. International Conference on Learning Representations, 2022. URL https::///// /ttp:::////eeeeeeeeeeeeee net forum? www.foorottmttttttte/em
d= UbdvWH_R3.
10

---

Under review as a conference paper at ICLR 2023
Hangfeng He and Weijie Su. A law of data separation in deep learning. ArXiv, abs/2210.17020, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss cross- entropy in classification tasks. arXiv preprint arXiv:2006.07322, 2020.
Like Hui. Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint arXiv:2202.083884 2022.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ArXiv, abs/ 150..0300....
Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Richard Radke, and Rogerio Feris. A broad study on the transferabilityy of visual representations with contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8845-8855. 2021.
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer-peeled perspective on neural collapse. In International Conference on Learning Representutions, 2022.
URL https::t/////mm / openreview .nt/
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint
arXiv:2004. 11360. 2020.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2661- 2671, 2019.
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? In NeurIPS, 2021.
Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani. Neural collapse: review on modelling principles and generalization. arXiv preprint arXiv:2206.0404 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. &t&t labs, 2010.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In In- ternational Conference on Learning Representations, 2017. URL https::/
net/Eorum? dinnniidssskksxxxs
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learn- ing Representations, 2018. URL https: //pprreviieeeeeeee/////e wwt. wtt.://rmmmiieeee 2i
Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11679, 2020.
Niv Nayman, Avram Golbert, Asaf Noy, Tan Ping, and Lihi Zelnik-Manor, Diverse imagenet models transfer better. arXiv preprint arXiv:2204.09134 2022.

---

Under review as a conference paper at ICLR 2023
Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research, 21(252):1 2020.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):
24652-24663, 2020.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Farheen Ramzan, Muhammad Usman Ghani Khan, Asim Rehmat, Sajid Iqbal, Tanzila Saba, Amjad Rehman, and Zahid Mehmood. A deep learning approach for automated diagnosis and multi-class classification of alzheimer's disease stages using resting-state fmri and residual neural networks. Journal of Medical Systems, 44(2), 2019.
Akshay Rangamani and Andrzej BaanurskkiFaheey Neural collapse in deep homogeneous classifiers and the role of weight decay. In /CASSP 2022- 2022220022222 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1243-4244 IEEE, 2022.
T. Ridník, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. ArXiv, abs/2104. 10972, 2021.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? In ArXiv preprint arXiv:2007.08453 2020.
Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang- Ting Cheng. Partial is better than all: Revisiting fine-tuning strategy for few-shot learning. In AAAI, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):199-1158, 2014. URL http:// /t//mmmm...rrr//// wwwpoo/pgpp.////.//.. srivastaval .a..al4..tmmll..
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Kumar Gupta. Revisiting unreason- able effectiveness of data in deep learning era. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 843-852, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-22882 2818-2826, 2016.
Christos Thrampoulidis, Ganesh R Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revis- iting neural-collapse geometry. arXiv preprint arXiv:2208.05512 2022.
Nadav Timor, Gal Vardi, and Ohad Shamir. Implicit regularization towards rank minimization in relu networks. ArXiv, abs/2201. 2022.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural col- lapse. arXiv preprint arXiv:2202. 080000 2022.
Francisco Utrera, Evan Kravitz, N. Benjamin Erichson. Rajiv Khanna, and Michael W. Mahoney. Adversarially-trained deep nets transfer better: Illustration on image classification. Interna- tional Conference Learning Representations, 2021. URL https: http::s////eeeevvieeee .et.eeeeewwwwweeeeeeeee.e...e serumiiddijJbbooe
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
Oriol Vinyals, Charles Blundell, Timothy P. Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, 2016.
12

---

Under review as a conference paper at ICLR 2023
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9929-9939 PMLR, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. ArXiv, abs/1708.00747) 2017.
Liang Xie, Yibo Yang. Deng Cai, Dacheng Tao, and Xiaofei He. Neural collapse inspired attraction-
repulsion- repulsion-balanceds balanced loss for imbalanced learning. arXiv preprint arXiv:2204.087:: 08735, 2022.
Yibo Yang. Liang Xic, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do we really need learnable classifier at the end of deep neural network? arXiv preprint 2rxii:2203.09081, 2022.
Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu Neural collapse with normalized features: A geometric analysis over the riemannian manifold. arXiv preprint arXiv:2209.092) 2022.
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? arXiv preprint 2019.
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In International Conference on Machine Learning, 2022.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A ge- ometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34, 2021.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu Hengshu Zhu, Hui Xiong, and Qing He. comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1): 43-76, 2020.
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 OTHER METRICS FOR MEASURING NC
Class-distance normalized variance (CDNV) Galanti et al. (2022). To alleviate the computa- tional issue of 1C the class-distance normalized variance (CDNV) introduced in (Galanti et al., 2022) provides an alternative metric that inexpensive evaluate. Let &' denotes the space of the input data æ and let Qs be the distribution over X' conditioned on the class k. For two differ- ent classes with Q, and Q, + j), the CDNV metric can be described by the following equa-
Varep (Q.)+Varep (Q;)
tion: Ve QQ Q3) 2 Mop (Q:) Ko (Q,11l where Méo (Qx) Ex-Qu [de denotes the class-
conditional feature mean and Var (Qx) Ex Illloll (2) Pea [Qx) denotes the feature variance for the distribution Qx- Although the exact expectation is impossible to evaluate, we can approximate them via their empirical means and empirical variances on the given training samples, so that
Var (Qi) + Var (Qj) Po (Q:) Poo (2))[1²
Voo (Qi Qj)
(5)
1
Noo (Qk)
=
Φ (mrk.)), Var (Qx) n
( Mi (x))
(6)
nk
To characterize the overall degree of collapse for model, all pairwise classes (i.e., AvgiyylVa, (Q,, Qj)I). model achieves perfect NC, obviously have A [V complexity scales linearly with the feature dimension d, so that it serves as a good surrogate for NC1 when the feature dimension d is large.
use the average CDNV between
(i), O. Because the CDNV metric is purely norm-based, computation
Numerical rank of the features H. The NC1 does not directly reveal the dimensionality of the features spanned for each class. Measuring rank of the features (H₂) is more suitable. However, the calculations for both NC₁ and rank are expensive when feature dimension gets too large. Thus, we introduce numerical rank (Timor et al., 2022) as an approximation
K
rank H =
HA H
where 1|-1 represents the Frobenius norm and I1-112 represents the Spectral norm. rank( H (numerical rank) could be seen as an estimation of the true rank for any matrix. Note that for calculating the Spectral norm, we use the Power Method to find an approximation. The metric evaluated by averaging over all the classes. It is expected that the smaller rank H) is, the their class means.
more collapsed the features
A.2 TECHNICAL DETAILS FOR SECTION 3.
In the main paper, for the ease of presentation, some technical details in the experiments are not presented explicitly. We discuss the missing implementation details here.
Experimental set-up for Section 3.1 In Figure 1, we pretrain ResNet50 models with different number of projection layers using Cifar- 100 and MinilmageNet datasets for 200 epochs. Then we use the learned model to do transfer learning on Cifar- 10. The NC1 and CDNV are then evaluated on the source dataset.
Experimental set-up for Section 3.2 In Figure 2, we pretrain ResNet50 models using different levels of data augmentation and adversarial training. For data augmentation, we consider Random- ColorJitter and RandomGrayScale We add an additional augmenta-
Crop, RandomHooiizottlFFin tion for each level. Le., for augmentation level 1, we don't 't add any data augmentation, for level 2, add RandomCrop. for level 3, add RandomCrop + RandomHHrizontalFiin and etc. For ad- versarial training, we follow the lx norm bounded adversarial training framework in (Madry et al., 5
2018 8) with 5 levels of attack size: { 255 255
26% 253 255
14

---

Under review as a conference paper at ICLR 2023
MNIST
Gios-10
1Shaloo Coopl
Figure 9: Change of NC1 dynamic for frained MLP/ / ResNet18 models. The MLP model is trained on MNIST (LeCun et al., 2010), FashionMNIST (Xiao et al., 2017), Cifar- 10, respectively and the ResNet18 model is trained on Cifar-100.
With the same pretraining setup as in Figure we transfer the learned models on 4 different down- stream datasets: Cifar-10, FGVS-Aiscrale DTD and Oxford-IITT-ee., We note that there many benchmark datasets that we can potentilly use, we choose these 4 datasets because the number of samples for each class is balanced in these datasets, which is the same scenario where NC is first studied (Papyan et al., 2020).
In Figure 6, layer in Vit-B32, the outputs are size 145 (# of patches + an additional classification token) x 768 (hidden dimension). For the layer-wise transfer learning experiment, first do a average pooling on the 145 patches and then train a linear classifier with input dimension 768 on top of each encoder layer.
the Vit-B32 model with pretrained weights released online. For each encoder
Experimental set-up for Section 3.3
Restet Architecture
For Table 2, we use a wide variety of experimental setups, including dif- ferent model architectures, pretrain- ing datasets and downstream datasets. We then compare the performance between linear probing, layer fine- tuning, SCL fine-tuning and full model fine-tuning. For ResNet mod- els, we consider each block as a fine- tuning unit and fine-tune the first block of each layer (e.g., as shown in Fig- ure 8, ResNet18 has layers where each layer has 2 blocks, we then fine-tune on the first block of each layer). For Vit-B32 model, we treat each of the encoder layer as a fine-tuning unit. In terms of skip connection, for ViT- B32 model, since the feature dimension from each layer remain constant, the skip connection could be directly applied for the features of the fine-tuned layer and the penultimate layer. However, for ResNet models, the number of channels and the feature dimension change across layers. Therefore, to calculate the skip connection, we first do an adaptive average pooling on the fine-tuned layer fea- tures to make each channel has only entry; then we further do 1 convolution make the number of channels match with the penultimate layer features. Finally, we do BatchNorm (Ioffe & Szegedy, 2015) on the processed fine-tuned layer features and apply the skip connection.
20
Figure 8: Fine-tuning unit of ResNet. Image of ResNet18 18 from (Ramzan al., 2019)
A.3 ADDITIONAL RESULTS
A.3.1 ADDITIONAL RESULTS FOR SECTION 3.1
In Section 3.1, we conjecture that the success adding projection layers on transfer learning can be attributed to the progressively within-class variability collapse. Recently, work (He & Su, 2022) further found that log(NCCr) decays linearly across layers. We train MLP and ResNet18 models on different datasets and validate their observation in Figure 9, Such phenomenon accompanies results in Section 1 and explains why adding projection heads is useful the perspective feature collapse.
15

---

Under review as a conference paper at ICLR 2023
Tansfer Acc. on 10
Trorsto Ane DTD
Tonsfo Acc IET
Ac.
Tonsto
MC1 (AFroNf)
RCI
NCL (PET)
Figure 10: Transfer accuracy on different downstream tasks and NC Transfer accuracy and NC₁ measured the same downstream datasets.
FT method
Linear Prohe
Block
Block 3
Block 5
Block
Full Model
SCL FT
FT
SCI.FT
Layer
SCL FT
Layer
SCL.FT CT
Cifur-10
43.12
65.03
82.33
80.84
84.83
82.00
83.10 57.72
71.66 45.85
73.08 46,97
85.61
Cifar-100
23.84
35.98
53.52
46,26
58.51
54.03
55.75
Table 3: Transfer learning performance of ResNet18 (pretrained on Imagenet) on downstream datasets with different fine tuning methods.
A.3.2 ADDITIONAL RESULTS FOR SECTION 3.2
In Section 3.2, show that the NC1 Cifar- 10 dataset negative correlates with the transfer aceu- racy on different downstream tasks. Nevertheloss, in Figure 10, when we evaluate the relationship between the transfer accuracy and the associated NC1 on the same downstream dataset, we cannot find a strong correlation. We conjecture that the reason behind such mismatch is the similar with our analysis in Section 3.1: when the values of NC1 get large, the metric starts become less meaningful since the increase may come from either an expansion in the feature variance within each class, or loss of discriminative power between classes.
A.3.3 ADDITIONAL RESULTS FOR SECTION 3.3
In Table 2, for the clarity of presentation, report the maximum transfer performance gained fine- tuning different layers. Here, we report the results for all of the fine-tuned layers in Table 3, Table and Table for ResNet18, ResNet50 and Vit-B32 respectively. We can observe that fine-tune middle layers nearly always give the best transfer performance.
Moreover, confirm that SCL fine-tuning yields more collapsed penultimate layer features, we fine- tune a ResNet18 model trained with Cifar- 100 dataset on Cifar- 10 using the layer fine-tuning and SCL. fine-tuning methods. We then visualize the change the NC, dynamic across network layers before and after fine-tuning in Figure 11 (Left). Fine-tuning a block would make the features from the block and all following blocks become more collapse on the downstream task, and adding skip connections make the collapse more severe and thus lead more collapsed penultimate layers and potentially better transfer performance. and the transfer accuracy. We notice the negative correlation is well-aligned with observation in Figure 3. However, we note that although more collapsed penultimate features almost surely leads to better transferability for fixed feature transfer learning (linear probing). it is not always reliable in the case of model fine-tuning. This is because the more we change model parameters before the linear classifier, the model becomes more prune to remember the data-label relationship in the downstream datasets and hence make the fine-tuned model likely to overfit. Eventually when the full model being fine-tuned, we would get neural collapsed model on the downstream training data as well but the collapse in this case may not translate to better performances on the test set. Such phenomenon happens especially severe for the datasets with limited number size, as shown (Hui et al., 2022).
Figure II (Right), we plot the correlation between. NC,
16

---

Under review as a conference paper at ICLR 2023
FT method
Linear Prohe
Block
Block 4
Block 8
Block 14
Full Model
Layer
SCL FT
Layer FT
SCI.FT
Layer
SCL FT
Layer
SCL.FT
Cifur-10
49.70
38.65
78.58
49.76 25.02
79.24
85.03
14
75.78
76.02 53.56
88.34
Cifar-100
29.71
6.98
$3.94
59.36
59.39
64,90
52.29
53.42
Table 4: Transfer learning performance of ResNet50 (pretrained on Imagenet) on downstream datasets with different fine tuning methods.
FT method
Lincar Probe
Layer FT
Layer
SCL FT
Layer FT
Layer
SCI.FT
Layer FT
Layer 7
SCL FT
Layer FT
Layer
SCI.FT
Full Model
PET
92.23
92.42
92.23
92.67
93.19
92.94
93.13
93.02
93.13
93.01
Aircraft
43.68
57.64
56.50
64.93
62.35
65.83
62.80
62.32
64.57
Table 5: Transfer learning performance of Vit-B32 (pretrained on JFT and ImageNet-21k) on downstream datasets with different fine tuning methods.
17.5
Pre-train
29
Fine-tune Block
15.0
Fine-tune Block skip connection
Transfor ACC. (Cifar-10)
12.5
10.0
0.0
Block (Shallow ->> Deep)
0.
855
Figure 11: Change of NC, dynamic before and after fine-tuning (Left), Correlation between downstream NC1 and transfer accuracy (Right). Transfer accuracy and NC₁ are measured on the same downstream datasets.
17


Reviews:
[{"review_id": "hAUkM_iB8gQ", "reviewer": "Reviewer_24Wm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper empirically explored how the Neural Collapse (NC) phenomena may affect transfer learning. The authors proposed two metrics $\\mathcal{NC}_1$ and CDNV to measure the degree of intra-class variability in relation to between-class covariance. They showed that 1) higher $\\mathcal{NC}_1$ in the pre-trained features led to a higher transfer accuracy and 2) lower  $\\mathcal{NC}_1$ in the target-task features also led to a higher transfer accuracy.", "summary_of_the_review": "Overall, I believe that the paper suffers from a somewhat sloppy experiment design, incomplete experiment description, and potentially misleading conclusions. The main conclusions are either already pointed out by other papers, or the direct consequence of those findings. ", "strength_and_weaknesses": "Strenth:\n\n- The paper addressed a very important problem: How transfer learning can be performed in a principlined way, potentially through the lens of neural collapse.\n- The authors provided a review that covers many relevant prior studies.\n- The experiments contain some interesting (perhaps not very surprising) results.\n\nWeaknesses:\n\n- The paper was based on an interesting idea, but the execution is not convincing. Because the experiment setup is not very well described, I  can only fill in some details using my imagination. Overall, I have quite a few questions about the soundness of the experiment design.\n\nIn particular, the definition of the $\\mathcal{NC}_1$ metric (Eq 4)  is not sufficiently motivated. To me, it seems more useful to consider the impact of two factors, $\\Sigma_B$ and $\\Sigma_W$ separately. The original NC paper already defined some quantitative metrics for convergence to simplex ETF; Why bother reinventing the wheels?\n\n- The experiment results are mostly unsurprising, but the author's conclusions were sometimes not justified and may be misleading.\n\nIn particular, the discussion surrounding MLP projection layers does not look correct to me. NC is about the last-layer features in a terminal phase of training. The authors may have misunderstood the original NC paper.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cqoOj_2ssw6", "reviewer": "Reviewer_ny4r", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the neural collapse phenomenon (minimal intra-class variance and maximally separated class means in learned representations of neural networks) in the context of transfer learning. They find that this type of collapse in the representations of the pretrained model hurts transfer performance (though this is only true up to a threshold), but neural collapse in the representations obtained after training on the target task actually correlates with better transfer performance. Their intuition is that capturing all the intra-class variation ‘upstream’ leads to more diverse representations that are likely to transfer well, whereas collapsing in this way ‘downstream’ is a sign of fitting the target task well which increases performance. They also ran a set of interesting experiments investigating the collapse in intermediate layers of the deep network, and how that correlates with transfer performance obtained by training a classifier on top of the corresponding intermediate representation. Their insights led them to hypothesize that finetuning not only a linear classifier but also the penultimate layer during transfer is helpful and indeed increases neural collapse as well as downstream accuracy.", "summary_of_the_review": "Overall, I found the paper well-written, the topic explored interesting and relevant and the investigation novel, to the best of my knowledge. Some findings were interesting and not previously known. However, the empirical exploration is limited in some important ways and I’m wondering whether the findings would generalize to different settings (see above). It’s also hard to take-away a clear message from the paper due to the premature nature of the investigation in some cases (see above discussion about the relationship between NC in pretrained representations and transfer performance). I therefore recommend only a weak acceptance of the paper in its current form.\n", "strength_and_weaknesses": "Strengths\n========\n[+] The study of the relationship between neural collapse and transfer learning performance is novel to the best of my knowledge, interesting, and relevant, as it can offer an (additional) explanation for why increasing diversity (e.g. with data augmentation) and projection layers can boost transfer performance, for example.\n\n[+] The paper is well-written, clear and easy to understand for the most part.\n\n[+] The experiment design is technically sound.\n\nWeaknesses\n==========\n[-] The main weakness I see is the limited settings considered which, due to being an empirical paper, make me question the generality of the conclusions. Most prominently, the choice to learn a linear layer on top of a frozen network at transfer time, and the limited choice of datasets to transfer to (see detailed comments below).\n\n[-] In terms of the relationship between NC in pretrained representations and transfer accuracy, the story seems to be quite nuanced. I didn’t gain an understanding about when we should expect the two to be positively versus negatively correlated. What happens at the ‘turning point’ in Figure 2?  Is there some method that can be used to find the ‘sweet spot’ in terms of NC_1 that is just right for achieving best transfer performance? The investigation there feels a bit premature in the sense that there are still many open questions. Partially because of this, it’s hard to walk away with a solid take-home message from the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "R1PoazeiyI", "reviewer": "Reviewer_urV1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an empirical study on the relationships between the neural collapse (NC) and the transferability of DNNs from source datasets to downstream tasks. The class-distance normalized variance (CDNV) is used as the metric of evaluating NC. The empirical study is done using ResNet18/50 (from CIFAR100 or MiniImageNet to CIFAR10) and ViT-B (from JFT & ImageNet22k to CIFAR100). Several findings are obtained: (1) If the learned representations are less collapse on the pre-trained data (i.e. larger NC metrics), they often lead to better transfer accuracy with the intuitive explanations that they better preserve the intrinsic structures of the input data.  This positive relationship only holds up to certain threshold with the intuitive explanations that the increase of the NC metrics (i.e., larger feature variability) will significantly decrease the margin upon certain threshold. (2) On the downstream tasks themselves, the relationships are negatively preserved, that is the more collapsed the representations of the layer on the downstream data, the better the transfer accuracy. ", "summary_of_the_review": "Overall, the paper is a good paper, providing interesting insights on the relationships between neural collapse and the transferability.  If the empirical study is done with larger scale and the findings retain, the paper will be much more interesting to the community.  ", "strength_and_weaknesses": "Strength:\n\n1) The empirical study is comprehensive for classification tasks. \n2) The aforementioned two findings are interesting.\n \n\nWeaknesses: \n\n1) The presented empirical study mainly focuses on classification tasks mostly with rather small models (ResNet 18/50) and datasets which share similar nature between pretrain and downstream (e.g., CIFAR100/MiniImageNet  to CIFAR10).  The study on VIT-B is only test on CIFAR100. It will be interesting to see if the findings will retain between more diverse datasets such as the Visual Domain Decathlon dataset, or even from classification tasks to object detection and segmentation tasks. \n\n2) The performance on the pretrain data are not presented (e.g. in Table 1 and Fig. 2). Based on the findings, larger NC metrics on the pretrained data seem to indicate inferior performance on the pretrained data.  It may worth considering the performance on the source dataset, the NC, and the performance on the downstream datasets jointly.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9kFGI2B1EG", "reviewer": "Reviewer_ETik", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the relationship between transferability and neural collapse (NC). The authors find out that the NC can be a double-bladed sword in transfer learning. For pre-training, larger NC indicates the model learns richer features from pre-training, leading to better transfer performance. For downstream tasks, smaller NC indicates the classes are better separated, and the model potentially generalizes better on downstream tasks. The authors further propose tuning one additional layer to improve over linear probe.", "summary_of_the_review": "Although I like the topic of this paper, I don’t think it fulfill its goal adequately and meet the standard of ICLR as a top venue. Thus, I tend to reject this manuscript based on the above weaknesses.", "strength_and_weaknesses": "The paper is overall well-written and easy to follow. Relating NC with transferability is interesting. NC also provides a perspective to understand how various factors affect transferability. \n\nWeaknesses:\n\n1.There is no evidence showing that the relationship between NC and transferability is robust. As the authors already mentioned, a large NC leads might not lead to good transfer performance as well, e.g. the model is randomly initialized and not trained. This is a simple sanity check that the correlation between NC and transferability does not pass.\n\n2.It is unclear what it the causality between NC, transferability, and the diversity of features. To me, learning diverse features in pre-training is the cause, while NC and transferability are consequences. Therefore, using NC to understand transferability is misleading in this sense. The idea that learning less diverse features leads to bad transfer performance is not novel. [1] contains ideas like this. Self-supervised learning contains more transferable features than supervised learning, so that it is more transferable. \n\n3.It is also not surprising at all that NC on the downstream tasks correlated well with the performance on downstream tasks. Even without pre-training, this correlation should be true, because small NC means the margin in the classification problem is large. [5] already showed that this is the case for few-shot learning. \n\n4.Lacking comparison with other works on transferability. There exist a line of works on predicting the transferability of the models with various metric. See [2] and references therein. The authors should provide a comparison with them to give the readers a sense how NC perform as a metric of transferability.\n\n5.Lacking justifications on larger datasets. It’s better to provide the NC results on ImageNet apart from CIFAR100 and CIFAR-10. Evaluating the numbers based on ImageNet should not be difficult with the publicly available pre-trained models? This gives readers more confidence on this phenomenon.\n\n6.The proposed transfer algorithm is not novel. Training more than one layers should definitely perform better than linear probe. The authors should also provide the numbers of fine-tuning as a comparison. Besides, there are a bunch of works on efficient fine-tuning, such as fine-tuning only the bias [3], and adapters [4]. It would be better to compare with them as well.\n\nMinors: \n\nAbstract, “when pretrain models”, Should be pre-training\n\nBasics of NNs, Is the layer index notation used anywhere else? If not, including it here will only make it more cluttered. Besides, for resnets, it is not correct definition. The definition here only applies to MLPs.\n\n[1] Self-supervised Learning is More Robust to Dataset Imbalance.  \n[2] LogME: Practical Assessment of Pre-trained Models for Transfer Learning.   \n[3] BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models.   \n[4] VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks.   \n[5] Unraveling meta-learning: understanding feature representations for few-shot tasks.   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject