Understanding and Improving Transfer Learning of Deep Models via Neural Collapse
Authors: Xiao Li
Sheng Liu
Jinxin Zhou
Xinyu Lu
Carlos Fernandez-Granda
Zhihui Zhu
Qing Qu
Venue: Accepted by TMLR ()

Abstract: With the ever-increasing complexity of large-scale pre-trained models coupled with a shortage of labeled data for downstream training, transfer learning has become the primary approach in many fields, including natural language processing, computer vision, and multi-modal learning. Despite recent progress, the fine-tuning process for large-scale pre-trained models in vision still mostly relies on trial and error. This work investigates the relationship between neural collapse (NC) and transfer learning for classification problems. NC is an intriguing while prevalent phenomenon that has been recently discovered in terms of the final-layer features and linear classifiers of trained neural networks. Specifically, during the terminal phase of training, NC implies that the variability of the features within each class diminishes to zero, while the means of features between classes are maximally and equally distanced. In this work, we examine the NC attributes of pre-trained models on both downstream and training data for transfer learning, and we find strong correlation between feature collapse and downstream performance. In particular, we discovered a systematic pattern that emerges when linear probing pre-trained models on downstream training data: the more feature collapse of pre-trained models on downstream data, the higher the transfer accuracy. 
Additionally, we also studied the relationship between NC and transfer accuracy on the training data. Moreover, these findings allow us to develop a principled, parameter-efficient fine-tuning method that employs skip-connection to induce the last-layer feature collapse on downstream data. Our proposed fine-tuning methods deliver good performances while reducing fine-tuning parameters by at least 90\% and mitigating overfitting in situations especially when the downstream data is scarce.

Published in Transactions on Machine Learning Research (05/2024)
Understanding and Improving Transfer Learning of Deep Models via Neural Collapse
Xiao Li* Deportment of Electrical Engineering and Computer Science, University of Michigan
alrioo@umichh. edu
Sheng Liu
shenglässtaford..ct edu
Biomedical Data Scienoes, Stanford University
Jinxin Zhou
shou.38200hhuckeemmaloo.....
Department of Computer Science and Engineering, Ohio State University
Xinyu Lu
zinyulu01007@@mai..co
School of Computer Seience, Curnegie Mellon University
Carlos Fernandez-Granda
cfgranda.@cim..yy..co
Center for Data Science, New York University
Zhihui Zhu
zha.34400osu. edu
Department of Computer Science and Engineering, Ohio State University
Qing Qu Department of Electrical Engineering and Computer Science, University of Michigan
ginggu@umich.coo edu
Reviewed on OpenReview: http:////paaarrreee.eee//e/e/ees
Abstract
With the over-increasing complexity largo-scale pre-trained models coupled with a short- age of labeled data for downstream training, transfer learning has become the primary approach in many fields, including natural language processing, computer vision, and multi- modal lcarning. Despite recent progress, the fino-tuning process for largo-scale pro-trained models in vision still mostly relies on trial and error. This work investigates the relation- ship between neural collapse (NC) and transfer learning for classification problems. NC is an intriguing while prevalent phenomenon that has been recently discovered in terms of the final-layer features and lincar classifiers of trained neural networks. Specifically, dur- ing the terminal phase of training, NC implies that the variability of the features within each class diminishes to zero, while the means of features between classes are maximally and equally distanced. In this work, examine the NC attributes of pre-trained models both downstream and training data for transfer learning, and find strong correla- tion between feature collapse and downstream performance. In particular, we discovered a systematic pattern that emerges when linear probing pre-trained models on rownstream training data: the more feature collapse of pre-trained models on downstream data, the higher the transfer accuracy. Additionally, we also studied the relationship between NC and transfer accuracy on the training data. Morcover, these findings allow us to develop a prin- cipled, parameter-efficient fine-tuning method that employs skip-connection to induce the last-layer feature collapse downstream data. Our proposed fine- tinettunigg methods deliver
"The first. authors contributed equally work.

---

Published in Transactions on Machine Learning Research (05/2024)
good performances while reducing fine-tuning parameters by at least 90% and mitigating overfitting in situations especially when the downstream data is scarce.
1 Introduction
Transfer learning has gained widespread popularity in computer vision, medical imaging, and natural lan- guage procossing (Zhuang et al.] 2020 Devlin et al.) 2019 Cheplygina et al., 2019). By taking advantage of domain similarity, pre-trained, large model from source datasets is reused as starting point feature extractor for fine-tuning new model on smaller downstream task (Zhuang et al., (020). The reuse of the pre-trained model during fine-tuning reduces computational costs significantly and leads to superior performance on problems with limited training data. Despite of recent advances, the undorlying mechanism of transfer learning is still far from understood, and many of existing approaches lack principled guidance.
In the past, the bulk of research has focused on improving data representations during the phase of model pre- training, with various metrics developed to assess representation quality source data, such validation accuracy on ImageNet (Kornblith et al. 2019 and feature diversity {Kornblith et al. 2021; Nayman et al,, 2022). However, the evaluation of representation quality has become increasingly challenging due to the scale and limited accessibility of source data. Additionally, with the emergence of large-scale foundation models Bommassan et al.) 2021 Zhou et al. 2023 such as GPT-4 (OpenAl 2023), SAM (Kirillov et al. 2023). and CLIP (Radford et al. 2021), the research focus has shifted towards the evaluating and fine-tuning of these pro-trained foundation models (Kumar et al., 2022) on downstream tasks.
Therefore, in this work, we mainly focus the downstream data for evaluating and fine-tuning large-scale pre-trained models. We examine the principles of transfer learning by drawing connections to intriguing phenomenon prevalent across different network architectures and datasets, termed 'Neural Collapse" (NC) Papyan et al. 2020 Han et al. 2022), where the last-layer features and classifiers "collapse" to simple while elegant mathematical structures on the training data. Specifically, during the terminal phase of training, it has been observed that (i) the within-class variability of last-layer features collapses to zero for each class and (di) the between-class means and last-layer classifiers collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling.
While the initial discovery of NC was rooted in the analysis of the source training data, our study demon- strates a compelling relationship between NC and transferability on downstream datasets. We adapt the metrics for assessing NC to gauge the quality of learned representations concerning both within-closs diver- sity and between-class discrimination. Remarkably, when applying linear probing to pre-trained models on downstream training data, we uncover an intriguing trend: as the features of pre-trained models collapse more significantly on the downstream training data, the transfer accuracy improves. Capitalizing on this insight, we design more principled and parfmeter-officient fine-tuning approaches for large-scale pre-trained models, including foundation models like CLIP (Dosovilskiy el al. 2021 Radford et al. 2021
Finally, for a more comprehensive study, we also investigate the relationship between transferability and NC metrics on the source training data. We considered different types of factors that affect transfer accuracy of pre-trained models, such as training losses, projection heads, and netw architectures. Our work reveals the limitation of using source data for predicting transfer accuracy.
Contributions. In summary, we highlight our contributions below.
More collapsed features on downstream tasks lead to better transfer accuracy. Through an extensive examination of pre-trained models using linear probing across various scenarios, our work shows the following relationship: a higher degree of feature collapse on downstream data tends to yicld improved transfer accuracy. This phenomenon is supported by comprehensive experiments conducted on multiple
downstream datasets (Krizhevsky et al. 2009
Maji et al. 2013 Cimpoi et al.) 2014 Parkhi et al. 2012
diverse pre-trained models (He et al.] 2016 Dosovitskiy et al. 2021 Huang et al. 2017 Sandler et al.
2018 Radford et al. 2021). and even within the context of few-shot learning (Tian et al. 2020 Liu et al.
2023b). Notably, we find that this relationship holds across different layers when linear probing is applied
to features of distinct layers from the same pre- treitrd model, thereby offering valuable insights for the principled design of more effective linear probing techniques.
2

---

Published in Transactions on Machine Learning Research (05/2024)
More efficient fine-tuning of large pre-trained models via NC. Based the aforementioned insights, we propose a simple and memory-efficiet fine-tuning approach that achieves good results in vision classification tasks. Our method revolves around the utilization of skip connections to fine-tune a critical layer in the pre-trained network, thereby maximizing the collapse of the last-layer feature. To validate the effectiveness of our strategy, we conduct experiments using publicly available pre-trained models, including ResNet, ViT, and CLIP (He ct al., 2016 Dosovitskiy et al., 2021 Radford et al., 2021). Remarkably, our method substantially reduces the number of fine-tuning parameters by at least 90% compared to full model fine-tuning. Moreover, in the context of low-shot learning, our approach exhibits improved robustness to data scarcity, exhibiting reduced overfitting tendencies.
Limitations of studying transfer accuracy using source training data. In addition to investigating the connection between downstream NC and transferability, we also studied the relationship between source NC and transferability. More specifically, we investigated different key factors that affect the transfer accuracy based upon the NC metrics on the source data, such as loss functions (Hui & Belkin 2020), the projection head (Chen el al. ) data augmentations (Chen He] 2021; Khosla et al. 22020): and adversarial training (Salman et al. 2020 Deng al. 2021). Within a certain threshold, we found that the more diverse the features are, the better the transfcrability of the pro-trained model. However, randomly generated features that are diverse do not generalize well, the relationship does not always hold and using source data diversity metrics has limitations for predicting transfer accuracy.
The rest of the paper is structured follows. In Scction 2 briefty review the concept of Neural Collapse (NC) and introduce our method for measuring transferability through AC. In Section 3 we demonstrate how NC can be used to evaluate a pre-trained model's transferability on a downstream dataset prior to fine-tuning. Motivated by this observation, we propose parameter-effiiient fine-tuning methods in Section [ which show robust performance in both standard and limited data scenarios. In Section 5 we extend our study to the relationship between NC and transferability on the source dataset, revealing a more complex and non-monotonic relationship. Finally, we conclude the paper in Section 6 Discussions of related work, detailed experimental setups and additional experiments are postponed to the Appendix.
2 Evaluating Pre-trained Models via NC
In this section, we provide a brief overview of the NC phenomenon, followed by the introduction of metrics for evaluating the quality of learned representations for transfer learning.
Basics of Deep Neural Networks. Let us first introduce some basic notations by considering a multi- class (c.g., K class) classification problem with finite training samples. Let {n be the number of training samples in each class. Let Fb,i denote the ith input data in the kth class (i E [ma], h E [K]), and the corresponding one-hot training label is represented by yk E RR with only the kth entry equal to 1. Thus, given any input data Dyii, we learn a deep network to fit the corresponding (onc-hot) training label yv such that
W yk (x2.)) linear classifier W
(W (WL-I...0 (Wi#k,i + b,) by-1) bL: feature he, = (o(
(1)
where W- = WL represents the last-layer linear classifier and h = h(xx,) = (0(2)))) is a deep hicrar- chical representation (or feature) of the input Øx Here, for a L-layer deep network whe each layer is composed of an affine transformation, followed by a nonlinear activation o.)) and normalization functions (c.g., BatchNorm (Toffe & Szegedy (2015]) We use O to denote all the network paramcters of (a)()) and O
to denote the network parameters of (x). Additionally, we use
H₃ = [ to represent all the features in matrix form. Additionally, the class mean for cach class is written as
H H H2 Hx] E BdxN
hen] E Rddxa V E [K],
H = h hx] E k and he == 1 ha iVk E [K].
Accordingly, we denote the global mean of H he K Σk= h
3

---

Published in Transactions on Machine Learning Research (05/2024)
A Review of Neural Collapse. It has been widely observed that the last-layer features H and classifiers W of trained network on a balanced training dataset. (Kt..i ys} with nt = T2 = exhibit
simple but elegant mathematical structures (Papyan et al. 2020: Papyan 2020). Here, we highlight two key properties below
Within-class variability collapse. For each class, the last-layer features collapse to their means,
hRg - hgs AS n, 1 < h < K.
(2)
Maximum between-class separation. The mess- hg centered at their global mean ha are not only linearly separable but also maximally distant and form a Simplex Equiangular Tight Frame (ETF): for some 0, H h1 ha
hg hG] satisfies
ck H H K- -
1 R
(3)
Recent studies have shown that NC is prevalent across wide range of classification problems ( Papyan
ct al. 2020 Mixon et al., 2020 Zhou et al. 2022 Han et al. 2022 Fang et al. 2021 Graf et al. 2021 Zhou et al.) on the source training data, regardless of the loss function used, the neural network architecture and the dataset. Intuitively, the prevalence of NC phenomenon implics that the features in each class are maximally separated on the training data, and the network learns a max-margin linear classifier in the last-layer. Additionally, if NC would also occur on downstrcam data, it could serve as an indicator of the transferability of pre-trained models that we study below.
Mcasuring the Transfcrability of Pre-trained Models via NC Metrics. Based on the above discus- sion, we can assess the transferability of pre-trained models on the down by measuring the feature diversity and separation using metrics for evaluating NC (Papyan et al. 2020 Zhu et al. 2021 defined as follows:
1 K
NC1 = trace
(4)
More specifically, it measures the magnitude of the within-class covariance matrix Ew E gaxd of the learned features compared to the between-class covariance matrix EB E Rdxd, where
Ew
nk hRgs Tox) (hr, RT)) EB
1 K
Σ (h -ha) (hx ha)²
Here, Σ/B represents the pseudo-inverse of EB, which normalizes Ew to capture the relative relationship between the two covariances. Essentially, if the features of each class are more closely clustered around their corresponding class means, NC₁, will be smaller. Conversely, if the class means are more separated, NC₁ will also be smaller for the same Σw- However, computing this metric in (4] can be computationally expensive which is often the case for large models. To address this issue, alternative metrics
due to the pseudo-invers such as class-distance normalized variance (CDNV) (Galanti et al. (202bb) and numerical rank (Zhou et al., 2022 have been introduced recently; we refer interested readers to Appendix D for more details.
Based upon the above, in the following propose to evaluate the quality of learned representations for transfer learning using the metric in equation 4 on the downsbremmsooue dalu, providing new insights into model fine-tuning. Specifically, in Section 3 we focus on downstream tasks and find that the NC metric mcasured on the downstream dataset and transfer accuracy are negatively correlated: smaller AC1 on downstream data leads to better transfer accuracy. Based on the findings, we will propose a simple and parameter efficient fine-tuning method in Section [ that works well in transfer learning. Additionally, we also investigate the relationship between NCC on the source dataset and pre-training in Section 5 The experimental setup is detailed in Appendix B In order to comprehensively understand the relationship between NC and Transfer learning, measure NC at different layers of various models as different sections tend to have different focuses, see Table I for the setup of each section.
Additionally, self-duality convergence has also been observed in the sense that w. d' for some él We omit them here because they are the main focus of this work.
4

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1:
Genenral experimental setup for each section. We study the relationship between NC and
transfer learning with different sctups for each section.
Fuue-tuee/PPae Pretrained Models
Dataset for Evaluation of NC
Using Intermediate Layer NC
Using Penultimate Layer NC
Section 3
Pre-trained Models
Downstream training data
Fig.
Fig. 122.4; Table
Section 4
Fine-tuned Models
Downstream training, data
Fig. T; Table
Section 5
Pro-trained Models
Source training data
Fig. 10
Transfer Acc. on Cifac10
Transter on Aircraft
Transtr on DTD
Transte on PET
Air
Afr
Toantter ACC.
Tontter ACC.
Tonster
Taander
400
ARC
ARL
AR2
Figure 1: Transfer accuracy and NC1 of Cifar-100 pre-trained models on different downstream tasks. We pre-train ResNet50 models Cifar-100 using different levels of data augmentation adversarial training. Here, NC, is measured on the downstream Cifar-10 dataset.
3 Study of NCC & Transfer Accuracy without Model Fine-tuning
First, explore the relationship between the NN1, metric and transfer accuracy by evaluating pre-trained models on downstream data before fine-tuning. The practice of transferring pre-trained large models to smaller rownsttream tasks has become a common approach in NLP (Devlin et al.] 2019 Houlsby et al. 2019 Hu et al.] 2021), vision (Dosovitskiy et al. 2021 Evci et al. 2022 Adler et al., 2020), and multi- modal learning. In the meanwhile, evaluating NC metrics of pro-trained models on downstream data is more feasible, as the availability and size of source data are often limited and prohibitive To maintain control over the factors influencing our study, we focus on linear probing pre-trained models, wherein we freeze the entire model and train a linear classifier on top of it using the downstream data. Our findings reveal a negative correlation between transfer accuracy and the NC; metric on the downstream data. This phenomenon is universal, as it holds true across multiple downstream datasets (Krizhevsky et al.) 2009 Maji et al. 2013 Cimpoi ct al.; 2014 Parkhi et al. 2012). different pre-trained models (He et al. 2016 Dosovitskiy et al. 2021 Huang et al. 2017 Sandler et al. 2018 Radford el al., 2021), the few-shot learning regime, and even across different layers within individual models.
Pre-trained models with more collapsed last-layer features exhibit better transferability. To support our claim, we pre-train ResNet50 models on the Cifar-100 datasct using different levels of data aug-
mentations and adversarial training (Madry et al. 2018) Salman et al. 2020 Deng et al. 2021) strength |
Once a model is pre-trained, we evaluate its transfer accuracy on four downstream datasets: Cifar- 10
(Krizhevsky et al. 2009), FGVC-Aircraft (Maji et al., 2013), DTD (Cimpoi et al.) 2014) and Oxford- -1lT-
Pet (Parkhi et al. 2012). As shown in Figure 1 we find a negative (near lincar) correlation between NC1 on Cifar-10 and transfer accuracy on the downstream tasks, where lower NC1 on Cifar-10 corresponds higher transfer a Thus, the NC1 metric Cifar-10 can serve as reliable indicator of transfer accuracy on downstream tasks. To further reinforce our argument, conduct experiments on the same set of downstream tasks using publicly available pre-trained models on ImageNet-Ik (Deng et al. 2009), such as ResNet (He et al. 2016), DenseNet (Huang et al. 2017 and MobileNetV2 (Sndder et al. 2018). Tn Fig-
FFor instance, the JFT dataset (Sun et al. 2017 used in pretraining the Vision Transformer, is extensive and not publicly accessible.
We use levels of data augruettations, each level represents adding additional type of augmentation, e.g., Level means Normalization, level means Normalization + RandomCrop, etc. For adversarial training streugth, we follow the framework in (adry et 2018) and cousider 5 different attack sizcs Please refer to Appcudix D for more details.
When evaliating the correlation between NC1 and transfer accuracy on the same downstream dataset, the correlation is not as strong as find Cifar-10, which we discuss in Appendix
5

---

Published in Transactions on Machine Learning Research (05/2024)
Transfer Acc. on Cifar-10
Transfer Acc on Aircratt
Transfer Acc. on DTD
Transfer Acc.
TTandter ACc
Transter Acc
Trunder ACC.
Transfer
30
Figure Transfer accuracy and NC₁ of public ImageNet-1kk pre-trained models on different downstrcam tasks. We evaluate transfer accuracy and.CC ./1, on multiple downstream datascts using various ImageNet-Il 1k pre-trained models, such as ResNet (He al.] 2016). DenseNet (Huang et al. 2017) and MobileNetV2 (andler et al. 2018). The AC11 is measured on the corresponding downstream dataset.
Table 2: Average few-shot classification accuracy and NC; metrics on CIFAR-FS and Mini- ImageNet. The test few-shot accuracies and NC1 are evaluated on minilmageNet and tageeel meta-lest splits. ConvNet4 denotes 4-layer convolutional network with 64 filters each layer.
Fewshot Accuracy
ConvNet4
ResNet12 He al. 2016
SEResNe112 Hu et shots
2019
Architecture
shot
5 shots
shot
5 shots
shot
CIFAR-FS
61.59
77.45
68.61 59.52
82.81
69.99
83.34
Mini-ImageNeet
52.04
69.07
75.92
60.21
77.17
NC, of models pre-trained meta-train splits
CIFAR-FS
38.12
29.13
28,40
Mini-ImageNett
51.82
28.24
25.58
ure 2 observe the same negative correlation between NC1 the downstream data and transfer accuracy, demonstrating that this relationship is not limited to specific training scenario or network architecture.
Moreover, this negative correlation between downstream NC1 and transfer accuracy also applics to the few- shot (FS) learning settings, as shown in Table for minilmageNNt and CIFAR-FS datasets. Following (Tian) let al. 2020), we pre-train different models on the merged meta-training data, and then freeze the models, and learn linear classifier at meta-testing time. During meta-testing time, support images and query images transformed into embeddings using the fixed neural network. The lincar classifier is trained the support embeddings. On the query images, we compute the NC,, of the embeddings and record the few-shot classification accuracies using the linear model.
Layers with more collapsed output features result better transferability. Furthermore, we find the same correlation between the NC, metric and transfer accuracy also occurs across different layers of the same pre-trained model. Specifically, as depicted in Figure 5 we linear probe each individual layer of the same pre-trained network, where we use the output of each individual layer "feature extractor' and assess its transfer accuracy by training a linear classifier on top of it. Surprisingly, regardless of the layer's depth, if the layer's outputs are more collapsed (smaller NC1), the resulting features lead to better transfer accuracy, To support our claim, we carried out experiments using the ImageNet-Ik pro-trained ResNet34 model (Deng et al. 2009 He et al. 2016). We evaluated the NC, metric on each residual block's 's output feature for the downstream data, as shown in Figure 3 (a). The results indicate that the output features with a smaller NC,, lead to better transfer accuracy, and there near-linear relationship between NC1, and transfer accuracy, suggesting that transfer accuracy is more closely related to the degree of variability collapse in the layer than the layer's depth. This phenomenon is observed across different network architectures. Figure 3 (b) shows similar results with experiments on the ViT-B (vision transformer base model) (Dosovitskiy et al. 2021) using a pre-trained checkpoint available online As a result, our findings offer valuable insights into the linear probing of pre-trained models. The conventional practice of solely utilizing last-layer features for linear probing may not be optimal for transfer learning. Instead, employing the NC metric to identify the optimal layer for linear probing can result in improved performance.
The checkpoint used for VIT-B B can found here

---

Published in Transactions on Machine Learning Research (05/2024)
0.956
Transfer Accuracy
Transter Accuracy
Addu Doc
Townder Aorumayy
Ritted Line
Filted Line
Accu
-2
Alg
AROA
Transter ACC
Transter
16 Layers tho Seep)
8ayors Shallw
(a) ResNet34
(b) VIT-B
Figure 3: NC1 and transfer learning accuracy of different layers from pre-trained model (Left) and nearly linear relationship between transfer learning accuracy and NC₁ (Right). We use (a) an ImageNet-1k dataset pre-trained RosNet34 model and (b) a released pre-trained ViT-B model. We use the Cifar-10 dataset for transfer learning and measuring the corresponding NC1.
Negative correlation between NC1 and downstream accuracy in un-supervised models. Tn Figures [ and [ we present empirical evidence demonstrating negative correlation between downstream NC and downstream transfer accuracy among supervised pro-trained models. To examine claim in more broad settings, we conducted experiments utilizing CLIP (Dosovilskiy et al. 2021), model trained on matching image and caption pairs without reliance on labeling information. Specifically, we employed the image encoder of CLIP as a frozen feature extractor to extract features from diverse datasets including Cifar (Krizhessyy et al. 2009), DTD (Cimpoi et al. 2014], FGVC-Aireraft (Maji et al., 2013 Oxford- 102-Flower (Nilsback Zisserman, 2008) and SUN397 (Xiao et al.] 2010). Subsequently, we conducted linear probing and computed NC metrics using the extracted features. The transfer accuracy is then plotted against the corresponding NC1 values, as illustrated in Figure 1 (a. We can observe from the plot clear negative correlation between the calculated NC1 and linear probing accuracy, which extends our claims to un-supervised models.
Discussion between NC1 and direct classifier measurement. Despite the empirical evidence shown in this section, important aspect one may argue is that training classifier directly the downstream training data shares the same efficacy with NC for predicting transfer accuracy. To differentiate between measuring NC1 and direct classifier measurement, we note that from classical machine learning theory, good training accuracy on the downstream data does not translate to good generalization performance. We might overfit on limited downstrream training data but not generalize to the downstream test data. One must test the trained linear classifier on a held-out test set to know a model's actual performance. In contrast, NC, can provide valuable insights into model's transfer performance based solely the training features from downstream task. To further demonstrate this point, we conducted the image encoder of the CLIP model as feature extractor to extract training and testing features from multiple downstream datasets. We train linear classifiers evaluate ACC on training features and then calculate training/testing testing accuracy using the fitted linear classifier. The plots of NC1 vs transfer accuracy and training accuracy vs. transfer accuracy are shown in Figure !| We observe that the negative correlation between NCC and transfer accuracy persists even in the case of unsupervised models, while no obvious correlation can be observed for the training accuracy.
experiment for which we used
4 Parameter Efficient Fine-tuning via NC on Downstream Tasks
Furthermore, our observations, as detailed in the previous section, can provide valuable guidance in designing efficient fine-tuning strategies. In the context of adapting large-scale pre-trained models to downstream vision tasks, there are two common approaches:
(i) linear probing (Khosla et al. 2020 Kornblith et al,
2021 Dong ct al.) 2021), which uses the pre-trained model as a fcature extractor and only learns a lincar Blassifier for the downstream lask, and (ii) full model fine-tuning (Dosovitskiy et al. 2021 Kornblith
ett al.) 2019 Salman et al.) 2020), which adjusts the entire pre-trained model using downstream training data. Linear probing is a highly parameter-officien method, while full model fino-tuning, albeit more costly particularly for largo-scale foundation models, typically yields superior results. However, in scenarios with limited learning data, full model fine-tuning may lead to overfitting; see Figure To balance these trade-
7

---

Published in Transactions on Machine Learning Research (05/2024)
Transfer Acc. CLIP
Transfer Acc. CLIP
Transfer Acc.
Transfer Acc.
80
80
70
Cifar-10
Aircraft
Cifar-10
Aircraft
Cifar-100
Flowers
Cifar-100
Flowers
DTD
SUN397
DTD
SUN397
50
50
2
86
88
90
92
94
96
98
100
NC1
Training Acc.
(a) Downstream NC₁ / Trasfer Acc.
(b) Downstream Train Acc. TTrssfer Acc.
Figure 4: Negative correlation between transfer accuracy and NC₁ of downstream datasets hold on the CLIP model while downstream training accuracy doesn't have strong correlation with transfer accuracy. We use the image encoder of CLIP model as a feature extractor to extract training and testing features from multiple downstream datasets. We then train linear classifiers and evaluate NC1 on training features and evaluate transfer accuracy using the testing features.
offis, recent attention has been given parameter-efficiet fine-tuning in NLP, which selectively adjusts a small portion of network paramcters (Hu et al., 2021; Rebuffi et al. 2017 Houlsby et al., 2019 Given the
growing trend of foundation models beyond NLP (Dosovitskiy et al. 2021
Radford et al.
2021 Zhai el. al.
2021), our work focuses vision classification tasks, exploring parameter-efficient fine-tuning based on the correlation between last-layer feature variability on downstream data and transfer accuracy. Based upon our extensive experiments in Section we conjecture that
The optimal transfer accurocy can be achieved by selectively fine-tuning layers to make the last-layer features collapsed possible the downstream training data.
Based on this, we introduce simple fine-tuning strategy aimed at increasing feature collapse levels in the last layer, with our results presented in Table and Figure S These results show that our proposed method establishs a good balance between linear probing and full model fine-tuning saving over 90% of parameters (see Table 3) and reducing overfitting when training data is scarce (see Figure In the following, we provide a detailed description of our fine-tuning approach.
Proposed Method: Skip Connection Layer (SCL) Fine-tuning (FT). Inspired by the observa- tion in Section 3 we propose Skip Connection Layer (SCL) Fine-tuning (FT), which consists of two key
components:
Fine-tuning one key intermediate layer. To parameter efficient, we only fine-tune one of the intermediate layers while keeping the rest of the network frozen, which we called it layer fine-tuning (Layer FT). To find the optimal layer for fine-tuning, we conduct an ablation study compare the performances by fine-tuning different intermediate layers with the rest of the network frozen. As shown in Figure fine-tuning the layer closer to the final layer usually leads to more collapsed feature in the last layer and, potentially, the better transfer accuracy: This is because the information from the inputs has been better extracted and distilled as getting closer to final layers. On the other hand, in many deep network architectures, layers closer to the output typically have more paramcters Thus, to strike
We also compare with MLP probing, for which replace the fiual linear classifier by 2-layer MLP classifier with hidden dimension 1024.
We obscrve similar trend for ViT-base models, as shown Figure Forr example, ResNet18 model has 512 512 parameters the penultimate layer, while only parameters are the input layer.
8

---

Published in Transactions on Machine Learning Research (05/2024)
Layers Frozen
Classifier
Layers updats
New Classifier
Garafer
Classfeer
Layer 12
Trained Layer
Layer?
Mky Canoobe
Layer
Trained Layer?
Toinrer lyeer
Layer
Trained Layer
Trained
Tra
Data
Data
Pre-training
Transfer
Layer FT
SCL FT
Figure An illustration of layer-wise trans- for learning. We use pre-trained model up to the intermediate i-th layer as feature extractor for transfer learning on the downstream tasks.
Figure 6: Illustrations of Layer FL (left) and SCL FT (right). Layer FL simply fine-tune intermediate layer, while SCL FT adds skip con- nection to the final layer.
Bodk 1 (Armitt
Blok (treratt
Book Wertatt
Block TrArrrat!
6.-
Model
SCL
SCL IT
SCLFT
SCL IT
Bock Abee
Figure 7: Layer-wise NC₁ for different fine-tuning methods on ImageNet-Ik pre-trained ResNet18. We compare the layer-wise NC1 across different fine-tuning methods, including linear prob- ing, Layer FT, and SCL FT, using the ImageNet-Ik pre-trained ResNet18 backbone. We evaluate the models on the downstream dataset of FGVC-Aireraft. For the figures from left right, we plot the results of fine-tuning only Block 1, Block Block 5, and Block in both Layer FT and SCL FT.
balance between transfer performance and parameter efficiency, opt to fine-tune one of the middle to deep layers. More specifically, in all our experiments, we fine-tune Block 5 of ResNet18, Block 8 of ResNet50 and Layer 8 for Vit-B32.
Improving feature collapse via skip connections. As illustrated in Figure i building on the Layer FT, we introduce second key component to further the collapse of last-layer feature by adding skip connection from the key fine-tuned layer to the last layer. We then use the combined features (i.c., the sum of the two outputs) from these layers as the new feature for training the linear classifier and fine-tuning the selected layer. Tf the dimensions of the features differ (e.g., in CNN-based models), we zero-pad the lower-dimensional feature to match the dimension difference. Our proposed SCL FT method enables more effective fine-tuning of the selected layer by directly passing the data's information from the intermddiate layer to the classifier, without losing information through other intermediate layers. Moreover, this approach leverages the depth of deep models, ensuring that the more refined features from the penultinate layer are also passed to the linear classifier. As demonstrated on ResNet in Figure 7 SCL FT leads the most collapsed feature in the last-layer other than full model fine-tuning, and better performance compared with Layer FT. We also observe a similar phenomenon the case of ViT, for which we postpone to Appendix F
Advantages of Our Methods. Through comprehensive experiments, we demonstrate the superiority of our suggested SCL FT over conventional methods on vision classification problems, such as linear probing, full model fine-tuning, and zero-shot. learning. Full model fine-tuning requires retraining all parameters,
9

---

Published in Transactions on Machine Learning Research (05/2024)
90
80
80
Transfer Acc.
Transfer Acc.
70-
60
60
50
Linear Probing
40
Zero Shot
Layer FT
Layer FT
40
SCL
SCL
Full Model FT
Full Model
30
20
10a
'022
10a
102
Num. of Samples Per Class
Num. of Samples Per Class
(a) ResNet18, Cifar-10
(b) CLIP, Clfar-100
Figure 8: Transfer accuracy for different fine-tuning methods with varying size of downstream training dataset. We fine-tune ImageNet-1k pre-trained ResNet18 models and CLIP using subsets of the Cifar-10 and Cifar-100 downstream datasets, respectively with varying sizes. All results are averaged over 3 runs.)
which amounts to 23 million for ResNet50 or 88 million for ViT-B32. In contrast, our methods, SCL FT and Layer FT, are highly efficient, achieving comparable or better results by fine-tuning only around 8% of the model's parameters, shown in Table 3 Moreover, our meth exhibit improved resistance to overfitting while achieving equal superior performance compared to full model fine-tuning; see Figure g The detailed experimental setup and hyperparameter details can be found at Appendix Below, we discuss two main advantages of our method in detail.
Parameter efficiency. Our method is casily adaptable, working effectively with both CNN-based (ResNets) and transformer-based (VIT) network structures. As demonst in Table 3 with just 8% of parameters fine-tuned, SCL FT surpasses linear probing by a great margin and reaches comparable performances compared with full model FT in ViT-based experiments across the datasets FGVC-Aireres (Maji et al. 2013), DTD /Cimpoi plied with Nes-ebbbased backbones, our SCL FT technique still delivers significant performance boost compared to linear probing in all experiments and reaches full model FT's performance in some scenarios.
al.] 2014), and Oxford-IIIT-Pet (Parkhi al., 2012). When ap-
Mitigating overfitting in the presence of downstream data scarcity. When dealing with lim- ited downstream data, performing fine-tuning on the entire large-scale pre-trained model can lead overfitting, resulting in poor generalization performance. In contrast, our methods demonstrate better resilience data scarcity and decreased overfitting by specifically fine-tuning small subset of model parameters. To demonstrate this, conducted fino-tuning experiments on pre-trained ResNet18/CLIB models using varying sizes of subsets from the 10aa-10/Cifar-000 training samples. The outcomes are presented in Figure S Our findings reveal that full model fine-tuning is more vulnerable to data scarcity, underperforming linear wrobing/zcro-sie
approaches when the downstream training data size is limited. In comparison, our SCL FT and Layer FT methods maintain their robust and siguificantly surpass full model fine-tuning until a large amount of downstream training data becomes available.
Ablation Studies. There are several design choices to consider when conducting Layer FT. These include decisions on retaining intermediate layers between the fine-tuned layer and the classifier, as well as deter- mining the number of layers to fine-tune. Below, we discuss these two aspects in detail. Additional ablation studies can be found in Appendix
Effect of removing subsequent layers during Layer FT. An interesting question regarding Layer Fine-Tuning (Tayer FT) is the utility of intermediate layers between the fine-tuned layer and the classifier:
10

---

Published in Transactions on Machine Learning Research (05/2024)
Table 3: Transfer learning results for linear probing, layer FT, SCL FT and full model FT on downstream datascts. We use publicly available ResNet50 and ViT-B models. Results reported are averaged over 3 random seeds. We use bold font represent the best result for each dataset and underline the second-best result.
Backhone
ResNet50
VIT-B32
Dataset
Cifar-10
Cifar- 100
Airerat
DTD
PET
Aireraft
DTD
PET
Transfer accuracy
Linear Probe
0.01
65.38
0..4
68.87 0.29
0.009
50.74
74.92
0.15
92.67 0.17
MLP Probe
85,00 0.13
65.91 0.05
05.51
70.12 0.03
90.78
51.43
0.37
75.39 0.21
93.71 00.71=00 0.25
Layer FT
91.00 0.11
19.13
71.98 0.49
72.62 1 0.52
0.110
73.20
0.47
77.51 0.11
93.28 0.08
SCL
96.08 0.14
79.04 0.19
0.114
73.51
0.04
92.07 0.02
73.17 0.37
77.80 0.17
93 0.05
Full Model FT
0.10
77.41
NC; evaluated on the penultimate layer feature h
Linear Probe
1.81
18.40
19.84
3.54
1-49
17.90
1.99
0.66
Probe
1.81
19.84 3.44 0.14
Layer FT
0.29
2.71 + 0.03
1.14
0.06
0.56
4.75 +
1.02 +
0.33
SCL FT
0.19 1 0.02
2.40 1 0.15
1.23 1 0.19
0.65 1. 0.06
0.39 1. 0.01
4.97 1. 0.03
1.13 1. 0.06
0.37 1. 0.01
Full Model FT
0.04 + 0.00
0.06 + 0.01
0.25 + 0.02
0.25 + 0.02
0.12 + 0.00
3.33 +: 0.21
0.97 4: 0.05
0.14 4: 0.00
Percentage of parameters finestuned
Liuesr Probe
0.08%
0.86%
0.80%
0.41%
0.32%
0.09%
0.04%
0.033%
MLP Probe
8.23%
8.56%
8.56%
8.37%
8.33%
1.00%
0.94%
0,933
Layer FT
6.52%
T.24%
T.24%
6.82%
6.73%
8.18%
8.14%
8.13%
SCL FT
6.52%
7.24%
1.21%
6.82%
6.73%
8.18%
8.14%
8.13%
Full Model FT
100%
100%
100%
100%
100%
10096
100%
100%
100
100
1..
of Parameters to FT
0.8
Transfer Acc.
Transfer Acc.
0.66
0.4
Transfer Acc. (Layer FT)
-0.2
Layer FT
Parameter count
Layer LP
Transfer Acc. (SCL FT)
0.0
4
8
10
12
2
10
12
Vit-B Layers (shallow deep)
of Layers Fine-tuned
(a) Effeet of removing subsequeett layers
(b) Effect of fine-tuning layers
Figure 9:
We fine-tune the ViT model the Cifar-100 100 dataset. (a): Transfer accuracy drops when
moving subsequent layers during Layer FT. We conduct Layer FT and Layer LP (removing subsequent layers on top of the fine-tuned layer); (b): Fine-tuning more layers brings mediocre performance boost while using significantly more parameters. 0 in the x-axis denotes linear probing and 12 denotes full model fine-tuning.
these layers be discarded without negatively impacting performance? To investigate this, we con- ducted series of experiments with the ViT model on the CIFAR-100 datasct, employing both Layer FT and Layer LP approaches. During fine-tuning, Layer LP removes all layers subsequent to the fine-Luned layer and directly utilizes its features for classification. The accuracy results for both methods are detailed in Figure a). We can observe that the climination of subsoquent layers significantly compromises per- formance in comparison to Layer FT, with the degree of performance decline being positively correlated to the number of layers removed. This finding indicates that the layers following the fine-tuned one, even
11

---

Published in Transactions on Machine Learning Research (05/2024)
though not changed during fine-tuning, play a crucial role in deriving semantically rich information for classification.
Effect of fine-tuning more layers. We also explore the impact of unfreezing more layers during model fine-tuning. Specifically, we fine-tune the ViT--33 model on the CIFAR-100 dataset, varying the number of layers that are fine-tuned. As illustrated in Figure 9 b), the parameter increase is linear we unfreeze more layers, yet the improvement in performance is relatively marginal when compared to the gains observed from linear probing to Layer FT. Therefore, to strike a balance between parameter efficiency and transfer learning performance, we focus on only finc-tuning one layer of the entire model in this work.
5 Limitations of pre-training NC in predicting transfer accuracy
Numerous studies have explored the link between feature diversity and transfer accuracy from a variety of angles QIslam et al. 2021; Kornblith et al. 2021 Feng al.] 2021 Galanti et al., 2022b). Additionally, the rise of large-scale foundational models (OpenAl 2023; Radford al.) 2021; Kirillov et al., 2023), coupled with the inaccessibility of much of the pre- pre-trained data for these models, makes the evaluation of NC the source dataset impractical with these pre-trained models. Therefore, relationship between downstream NC and transferability in this work. To complement our study, we have done some small-scale experiments to study the relationship between the NCC metrics and transfer accuracy on the source training dataset (I.e., we measure NC on the source training dataset). These experiments revealed positive correlation between source NC and transferability, but only to a certain extent which prevents us from reaching a general conclusion regarding this relationship. Therefore, we leave these results and discussions to Appendix B and include only representative example in this section to illustrate that relying on pre-training NC to predict transfer accuracy may not be as reliable utilizing downstream NC, as demonstrated in earlier sections.
focus establishing the
To demonstrate the limitation of source NC, we pro-train ResNet50 models on the Cifar- datasel using different levels of data augmen- tations and adversarial training (Madry
al.) 2018 Salman al. 2020 Deng et al., 2021) strength [ and then report the transfer ac- curacy of the pre-trained models on the Cifar-10 dataset. As shown in Figure TO we observe that there exists positive correlation be- tween the NC1 on the source dataset and the transfer accuracy but the correlation only holds up to a certain threshold. We found it is generally hard to locate the turning point where the positive corre- lation diminishes across different experimental setups. In Figure T0 this turning point appears to be around the value of 1.0. However, in distinct experimental scenario, as depicted in Figure [1]] the positive correlation persists even when the pre-trained NCC exceeds 2.0.
Transfer Acc 12
Adv. Lxl
Lvl
Adv Lvi
Aug. LV13
Adv. Lvi
Aug, LV14
Adv. Lvi
Aug. LV15
64
22
25
NC1
Figure 10: NC, vs. transfer learning accuracy. Models are pre-trained on the Cifar-100 dataset with different levels of data aug- mentation and adversarial training strength, and then transfer accuracy of pre-trained models is reported on the Cifar-10 dataset.
We believe the reason behind the limitation of source NC is that the magnitude of NC, is affected by two factors of the learned features: (1) within class feature diversity and (ii) between class discrimina- tion. When the source NC, becomes too large, the features lose the between-class discrimination, which results in poor transfer accuracy. An extreme case would be untrained deep model with randomly initialized weights. Obviously, it possesses large NC,, with large feature diversity, but its features have poor between-class discrimination. Therefore, random features have poor transferability. To better predict the model transferability, we need more precise metrics for measuring both the within-class feature diversity and between-class discrimination, where the two could have a tradeoff between each other. We leave the investigation as future work.
We levels of data augmentations, each level represents adding additional type of augmentation, e.g., Level means Normalization, level means Normalization + RandomCrop, etc. For adversarial training strength, we follow the framework in Madry et 2018) and consider 5 different attack sizes. Please refer Appendix E for more details.
12

---

Published in Transactions on Machine Learning Research (05/2024)
6 Conclusion
In this work, we have explored the relationship between the degree of feature collapse, as measured by the NC, and transferability transfer learning. Our findings show that there is twofold relationship between NC and transferability: (i) more collapsed features on the downstream data leads better transfer performance; and (W) models that are less collapsed on the source data have better transferability up to certain threshold. This relationship holds both acToss and within models. Based these findings, we propose simple yet effective model fine-tuning method with significantly reduced number of fine-tuning parameters. Our experiments show that our proposed method can achieve comparable performance compared to full model FT across various tasks and setups. Further discussions of future directions are deferred to Appendix C
Acknowledgement
and QQ are grateful for the generous support from NSF CAREER CCF 2143904, NSF CCF 2212066, NSF CCF 2212326, NSF ISS 2312842, ONR N00014-22-1-2529, and an AWS AI Award. JZ and ZZ express their appreciation for the support received from NSF grants CCF-2240708 and CCF-2241298. SL acknowledges support from NSF NRT-HDR award 1922658 and Alzheimer's Association grant NARG-VTF-21-848800 while CFG acknowledges support from NSF OAC-2103936. We would also like to thank Dr. Chong You (Google Research) and Dr. Hongyuan Mei (TTIC) for their valuable discussions and support throughout this work.
References
Neural collapse under cross-entropy loss. Applied and Compulational Harmonic Analysis, 59:224 241, 2022. ISSN 1063-5203. doi: https://doi.org/ hat..org/1..111///11 a1t...///...caaaa22221.1 2021.12.11. Special Issue on Harmonic Analysis and Machine Learning.
Thomas Adler, Johannes Brandstetter, Michael Widrich, Andreas Mayr, David P. Kreil, Michael Kopp, Günter Klambauer, and Sepp Hochreiter. Cross-domain few-shot learning by representation fusion. ArXiv, abs/2000.06498, 2020.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/ 1ee//660700644000 2016.
Tina Behnia, Ganesh Ramachandra Kini, Vala Vakilian, and Christos Thrampoulidis. On the implicit geometry of cross-entropy parameterizations for label-imbalanced data. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022. URL http:://ppparreeeeeeeeeeeerrrrta
ictW
Tina Behnia, Ganesh Ramachandra Kini, Vala Vakilian, and Christos Thrampoulidis. On the implicit geometry of cross-entropy parameterizations for label-imbalanced data. In International Conference on Artificial Intelligence and Statistics, pp. 10815 10838. PMLR, 2023.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Fsin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Gross- man, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Ieard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Gooff Kecling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Ku- ditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Decpak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel J. Orr, Isabel Papadimitrion, Joon Sung Park, Chris Piech, Eva. Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren,
13

---

Published in Transactions on Machine Learning Research (05/2024)
Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matci A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiu, abs/ 222208007255 2021.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Yi Ma. Redunet: A white-box doep network from the principle of maximizing rate reduction. ArXie, abs/2105.10444 2021.
Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Generalized lora for parameter-efficient fine-tuning, arXiv preprint arxiv:2306. 07967, 2023.
Mayee Chen, Daniel Y Fu, Avanika Narayan, Michael Zhang, Zhao Song, Kayvon Fatahalian, and Christo- pher Ré. Perfectly balanced: Improving transfer and robustness of supervised contrastive learning. In International Conference on Machine Learning, pp.
PMLR, 2022a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and GeofTrey E. Hinton. A simple framework for con- trastive learning of visual representations. ArXiv, abs/2002.05709, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. 2021 IEEE/CVE Conference on Computer Vision and Poftern Recognition (CVPR), pp. 15745-15753, 2021.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Y. Qiao. Vision transformer adapter for dense predictions. ArXiv, 04//2220508534 2022b.
Veronika Cheplygina, Marleen de Bruijne, and Josien P.W. Pluim. Not-so-supervised: A survey of semi- supervised, aulli-instance and transfer learning in medical image analysis. pp. 280 296, 2019.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Hien Dang, Tan Nguyen, Tho Tran, Hung Tran, and Nhat Ho. Neural collapse deep linear network: From balanced to inbalanced data. arxiv preprint or Xiv:2301.00000 004077,,0077... 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenct: A largo-scale hicrarchical image database. 2009 IEEE conference on computer vision and pullern recognition, pp. 248 255. Ieee, 2009.
Zhun Deng, Linjun Zhang, Kailas Vodrahalli, Kenji Kawaguchi, and James Zou. Adversarial training helps transfer learning via better representations. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, 2021. URL https::/oppnrer//// htt/foormmmid=ffoohhorr
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, ArXiv, abs/1810.04805, 2019.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Noil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11999, 2021.
Utku Evci, Vincent Dumoulin, H. Larochelle, and Michael Curtis Mozer. Hcad2toc: Utilizing intermediate representations for better transfer learning. In International Conference on Machine Learning, 2022.
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-pecled model: Minorily collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43). 2021.
Yutong Feng, Jianwen Jiang, Mingqian Tang, Rong Jin, and Yue Gao. Rethinking supervised pre-training for better ream transferring. Xiu preprint arXio:2110.00 06014, 2021.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Tomer Galanti. A note on the implicit bias towards minimal depth of deep neural networks. arXiv preprint WxXi:2202.09028, 2022.
Tomer Galanti, András György, and Marcus Hutter. Generalization bounds for transfer learning with pre- trained classifiers. arXiv preprint arXiv:2212. 12532, 2022a.
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. International Conference on Learving Representations, 2022b. URL http://ppprrevie.etteeeeeeeeeeee
id=Sw1p410B6aQ
Peifeng Gao, Qianqian Xu, Peisong Wen, Huiyang Shao, Zhiyong Yang, and Qingming Huang. A study of neural collapse phenomenon: Grassmannian frame, symmetry, generalization. arXiv preprint SSK::233400891, 2023.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Jiao Qiao. Clip-adapter: Better vision-langnage models with feature adapters. ArXiv, abs/2110.04544,, 2021.
Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissocting supervised constrastive learning. In International Conference on Machine Learning, pp. 3821 181--830.220 PMLR, 2021.
X.Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamics the central path. In International Conference Learning Representations, 2022. URL
http:://pparrriiio..eeooouurrrre net/
Hangleng He and Weijie J Su. A law of data separation in deep learning. ar preprint or orXiv:2210.17000 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameter-efficiet model adaptation for vision transformers. 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-offiiien transfer learning for nlp. In International Conference Machine Learning, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXio, abs/2106.09685,, 2021.
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-excittttiin networks, 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708. 2017.
Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss cross-entropy in classification tasks. arXiv preprint arXic:2006.07322 2020.
Like Hui, Mikhail Belkin, and Prectum Nakkiran. Limitations of ncural collapse for understanding general- ization in deep learning. arXiv preprint arXiv:2202.0898, 2022.
Sergey Tofle and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ArXie, abs/1502.03166 2015.
Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Richard Radke, and Roge- rio Feris. broad study on the transferability of visual representations with contrastive learning. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 8845-8855, 2021.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer-peeled perspective on neural collapse. In International Conference on Learning Represenlations, 2022. URL
Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for a large number classes. arXiv preprial ar (xXx:2310000351,, 2023.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXi::2004.1 11362, 2020.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Yen Lo, et al. Segment anything. arXio:2304.02643 2023.
arXiv preprint
Simon Kornblith, Jonathon Shlens, and Quoc Le. Do better imagenet models transfer better? In Proceedings of the LEEE/CVF conference on computer vision and pattern recognition, pp. 2019.
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? In NeurIPS, 2021.
Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreviee.neet net.e/.w.eeeeeeeeeeeeeee.eeeeeee QTXocpAP9p
Alex Krizhevsky, GeofTrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Perey Liang, Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference Learning Representations, 2022. URL
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Mcta-learning with differen- tiable convex optimization. In Proceedings of the IEEE/CVI conference on compuler vision and pallern recognition, pp. 10657-10665, 2019.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.0844 2023a.
Sheng Liu, Jonathan Weed, Narges Razavian, and Carlos Fernandez-Grandee Early-learning regular- ization prevents memorization of noisy labels. Advances in newral information processing systems, 33: 20331-20342, 2020.
Sheng Liu, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:2811.06669 2023b.
Weiyang Liu, Longhui Yu, Adrian Weller, and Bernhard Schölkopf. Generalizing and decoupling neural collapse via hyperspherical uniformity gap. In The Eleventh International Conference on Learning Repre- sentations, 2023c. URL https::/openrevieer
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference Learning Representatiinss, 2017. URL http::///ppppaeeeeee...er/r/rees
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL http:p///oopeeereeeeeeeeeee///ra
S. Maji, Kannala, Rahtu, M. Blaschko, and A. Vedaldi, Fino-grained visual classification of aircraft. Technical report, 2013.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Picter Abbecl. A simple neural attentive meta-learner. arXiv preprint or Xiv: 1eXix1777.0000 707.03141, 2017.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint or 1ft:2011.1100 2020.
Niv Nayman, Avram Golbert, Asaf Noy, Tan Ping, and Lihi Zelnik-Manor. Diverse imagenet models transfer better. areie preprint arXiv:2204.0918. 2022.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
OpenAL. Gpt-4 technical report. ArXiu, abs/2303.08874, 2023.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems, 31, 2018
Vardan Papyan. Traces of class/cross- closs-clas structure pervade deep learning spectra. Journal of Machine Learning Research, 21(252):1-64, 2020.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652 24663, 2020.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2888 2830, 2011.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramosh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Farheen Ramzan, Muhammad Usman Ghani Khan, Asim Rehmat, Sajid Iqbal, Tanzila Saba, Amjad Rehman, and Zahid Mehmood. A deep learning approach for automated diagnosis and multi-class clas- sification of alzheimer's disease stages using resting-state fmri and residual neural networks. Journal of Medical Systems, 44(2), 2019.
Akshay Rangamani and Andrzej Banburski-Fahey, Neural collapse in deep homogeneous classifiers and the role of weight decay. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 4247- IEEE, 2022.
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso Poggio. Feature learning in deep classifiers through intermediate neural collapse. Technical report, Center for Brains, Minds and Machines (CBMM), 2023.
Sylvestre-Alvis Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In NIPS, 2017.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet modols transfer better? In ArXiu preprint XXXi:2007.08488, 2020.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chen. Mobilenctv2: Inverted residuals and linear bottlenecks. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Saurabh Sharma, Yongqin Xian, Ning Yu, and Ambuj K. Singh. Learning prototype classifiers for long- tailed recognition. ArXiv, abs/2302.0049) 2023.
17

---

Published in Transactions on Machine Learning Research (05/2024)
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Kumar Gupta. Revisiting unrcasonable effectiveness of data in deep learning era. 2017 TEEE International Conference on Computer Vision (ICCV), pp. 843 852, 2017.
Christos Thrampoulidis, Ganesh R Kini, Vala Vakilian, and Tina. Behnia. Imbalance trouble: Revisiting neural-collapse geometry. arXiv preprint arXiv:2208.0533 2022.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In European Conference on Computer Vision, pp. 266-282. Springer, 2020.
Nadav Timor, Gal Vardi, and Ohad Shamir. Implicit regularization towards rank minimization in relu networks. ArXio, abs/2201.1278 12760, 2022.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. arXiv preprint arXi::2202.0800 08087, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint aix:22200. 18971, 2023.
Peng Wang, Huikang Liu, Can Yaras, Laura Balzano, and Qing Qu. Linear convergence analysis of neural collapse with unconstrained features. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022. URL https: btattps:///penrevviiee http:/ne//ffoooo mett/forum2id-NC3iiMMM..
Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, and Qing Qu Understand- ing deep representation learning via layerwise feature compression and discrimination. orXie preprinl arXiv:2311.02900 2023a.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929 9939. PMLR, 2020.
Zijian Wang, Yadan Luo, Liang Zheng, Zi Huang, and Mahsa Baktashmotlagh. How far pre-trained models are from neural collapse on the target dataset informs their transferability, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5549 5558, October 2023b.
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2010. doi: 20100///VRR........5....
pp.
Liang Xie, Yibo Yang, Deng Cai, Dacheng Tao, and Xiaofei He. Neural collapse inspired attracttin--eppulsin balanced loss for imbalanced learning, arXiv preprint STT:::2004007333 2022.
Yibo Yang, Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do we really need a learnable classifier at the end of deep neural network? arXiv preprint arXiv:220s. 09081, 2022.
Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu Neural collapse with normalized features: A geometric analysis over the riemannian manifold. arXiv preprint (XXi::229900211. 2022.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of repre- sentation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.0448 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), pp. 1204-1213, 2021.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengeheng Hc, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fino-tuning. In The Eleventh International Conference on Learning Representations, 2023. URL. httph::p/://pttt http://s/..ggeeeeeeee/////////dd
18

---

Published in Transactions on Machine Learning Research (05/2024)
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Jiao Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. ArXiu, abs/2111 222211013933 2021.
Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi, Xiangyu Zhang, and Jiaya Jia. Understanding imbalanced semantic segmentation through neural collapse. 2023.
Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guan Wang, Kaichao Zhang, Cheng Ji, Qi Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, Lichao Sun Michigan State University, Beihang University, Lehigh University, Macquarie University, Nanyang Tech- nological University, University of California at San Diego, Duke University, University of Chicago, and Salesforce AI Research. A comprehensive survey pretrained foundation models: A history from bert to chatgpt. ArXiv, abs/2302.094419 2023.
Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. In Advances in Neurol Information Processing Systems.
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In International Conference on Machine Learning, 2022.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geomet- ric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34, 2021.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43-3) 76, 2020.
19

---

Published in Transactions on Machine Learning Research (05/2024)
Appendix
The appendiccs are organized as follows. In Appendix A
we discuss related recent advances in neural we provide more experiments regarding source NC and transferability. In Appendix C we discuss the future directions inspired by this work that we want to pursue, In Appendix D we review other metrics that measure feature diversity and discrimination. In Appendix
collapse, model pre-training and fine-tuning. In Appendix B
E we provide experimental details for cach figure and each table in the main body of the paper. Finally, in Appendix F we provide complimentary experimental results for Section and the proposed
fine-tuning methods in Section 4
A Related work
Our work can be related to many existing results on neural collapse, model pre-training, and parameter- efficient fine-tuning. Below, we provide a summary and brief discussion of these results.
Studies of the Neural Collapse Phenomenon. The occurrence of the ACC phenomennon has recently
gained significant attention in both practical and theoretical fields (Papyan et al. 2020 lu2. 2022 Zhu
let al. 2021 Fang et al. 2021 Han et al. 2022 Kothapalli 2023), with studies exploring its implications on training, generalization, and transferability of deep networks; see a recent review paper (Kothapalii
2023 Most of these existing studies focused training. For instance, various loss functions, such
cross enoss-yntropy (Papyan et al. 2020 lu2 2022 Zhu et al.
2021
Fang et al. 2021
Ji et al. 2022 Yaras
et. al. 2022: Liu et al. 2020). -qaan- error (Mixon et al. 2020 Han et al. 2022 Zhou et al. 2022
Tirer & Bruna
2022
Rangamani & Banburskk-
2022
Wang et al. 2022
Dang et al. 2023
and
supervised contrastive (Graf et al.
2021 losses, have been shown to exhibit NCC during training.
Other
lines of work have investigated the relationship between imbalanced training and NCC (Fang et al.
2021
Xie et al.] [022 Yang et al.] 2022 Thrampoulidis et al., 2022 [ehnii et al.) 2022 Zhong et al.
2023
Sharma et al. 2023 Behnia et al. 2023), generalization to the settings of large classes (Liu al.
2023c
Gao et al. 2023 Jiang ct al. 2023 Furthermore, a line of work found the phenomenon of progrossive
feature viriability collapse across network layers on the pre-training datasel (Hui et al. 2022 Papyan
2020 He & Su 2022
Rangamani et al. [0233 Wang et al. 2023a -.. Recent works have also investigated
the relationships between NC and both gencralization and transfcrability (Hui et al.) 2022 Galanti ct al.
2022b a Galanti 2022 Chen et al.) 2022a Wang et al. 2023b). In particular, the study in Galanti et al.
2022b a reveal that NC occurs on test data when drawn from the same distribution as training data, but the collapse is less severe for finite test samples (Hui et al., (002)). While Galanti al. (2022b a also investigated the correlation between NC and transfer learning, their study focused solely the influence of the number of source classes. Our research, on the other hand, is much more comprehensive and demonstrates universal correlation between NC and transfer accuracy on downstream data. ! We examine a broad range of settings, including various tasks, pre-trained model architectures, training setups, and even different layers within individual models. Furthermore, our investigation into NC has contributed to the principled development of more efficient fine-tuning mothods.
Model Pre-training & Fine-Tuning While various studies have explored factors influencing trans- ferability during model pre-training (Kornblith et al. 2019 2021 Nayman et al. 2022 their findings
largely remain ambiguous. Additionally, these studies primarily focus on characterizing transferability
based on the source dataset, which is often inaccessible in the era of large-scale pre- mre-trained foundation
models (Dosovitskiy et al. 2021 Radford et al. 2021 OpenAl 2023 Liu et al. 2023a Touvron et al.
PProgressive variability collapse happens for well-trained models when feature diversity is evaluated on the source dataset. Due to distribution shift, this pattern is not exact when evalusting feature diversity on the downstream datasets, see Figure 7 for example.
1he work (Wang al. 2023b also attempt to establish connections between NC on downstream data and transfer accuracy, However, their metric is more intricate than ours. Morcover, their study is exclusively concentrated NCC in downstream tasks of various pre- pre-trained models. In contrast, our research encompasses a more comprehensive scope, examining hoth inter-model and intra-model NC across diverse training configurations.
20

---

Published in Transactions on Machine Learning Research (05/2024)
In contrast, our work primarily concentrates on evaluating the transferability of pre-trained models using downstream data. This approach not only enables us to establish universal relationship wetween NC and transferability, but also aligns with the current landscape of large-scale pre-trained models that have limited access to source data. In the field of vision model fino-tuning, two commonly used ap- proaches are linear probing and full model fine-tuning (Kumar el. al., 2022). Linear probing focuses on training only the classifier, resulting in parameter-offiient models that often underperform compared to full model fine-tuning. Previous studies have attempted to strike a balance between parameter efficiency and performance by incorporating 'adaptors" as auxiliary subnetworks between pre-trained layers (Re- buffi et al. 2017; Houlsby et al., 2019). While effective for transformers in NLP and recently extended to multi-modality and vision domains (Gao et al.) 2021 ZZhang et al.) 2021; Chen et al.] 20226). this approach requires additional structures and is specific to certain net work architectures. In contrast, our fine-tuning method does not introduce extra components and can be easily applied to any network archi- tecture. Another line of work involves storing all intermediate layer features and lincar probing a subsct of them (Evei et al., 2022 Adler et al. 2020). While this approach shows promise when training data for downstream tasks is scarce, it fails to compete with full model fine-tuning when ample training data is available. In contrast, our approach is applicable regardless of data volume and is more memory-efficiet since it does not require storing all intermediate features. Furthermore, certain methods aim to adapt pre-trained models to downstream tasks by updating low-rank counterparts of weight matrices (Hu et al. 2021 He et al., 2022 Zhang et al., 2023; Chavan et al.) 2023). These methods often involve modifying multiple components pre-trained models at different layers, whereas our approach focuses on updating one single layer of the overall model.
2023
B Additional observations on source NC and transferability
In Section we have demonstrated the limitations of source NC in predicting transferability. In this section, we discuss this relationship in more detail by showing that the choice of training losses and the design of network architecture have a direct impact on the levels of feature collapse on the penultimate layer and the transfer accuracy. Similar observations were made in (Islam et al. 2021 Kornblith et al. 2021) regarding the impact of different training losses on transfer performance. To demonstrate this, we pre- trained ResNet18 models on the Cifar-100 dataset using three different loss functions: CE, MSF (Hui & Belkin 2020), and SupCon (Khosla et al. 2020). We then evaluated the test accuracy on the Cifar-10 dataset by training linear classifier
the frozen pre-trained models. For the SupCon loss, we followed the setup described in (Khosla et al.) 2020), which uses MLP as projection head after the ResNet18 encoder. After pre- training. we only used the encoder network as the pre-trained model for downstream tasks and abandoned the projection head. Based on the experiment results, we observe the following.
The choice of training loss impacts feature diversity, which in turn affects transfer accuracy. Larger feature diversity, as measured by a larger NC, value, generally leads to better transfer accuracy. As demonstrated in Appendix C where the last two columns reports the results for SupCon with different layers of projection heads, a model pre-trained with the MSE loss exhibits a severely collapsed representation on the source dataset, with the smallest NC1 value and the worst transfer accuracy. While SupCon with projection head results in higher NC1, and better transfer accuracy.
The MLP projection head is crucial for improved transferubility. The model pre-trained with the SupCon loss and a multi-layer MLP projection head shows the least feature collapse compared to the other models and demonstrates superior transfer accuracy["] [2] If we substitute the MLP with a linear projection layer, both the NC1 metric and transfer accuracy of SupCon decrease, resulting in performance comparable to the models pre-trained with the CE loss. This can also be demonstrated by our experiments in Figure [2] where we pre-train ResNet-50 models on the Cifar-100 dataset and report. NCC metrics and transfer ac- curacy for varying numbers of projection head layers (from one to three layers). Our results show that using projection heads significantly increasos represcntation diversity and transfer accuracy - adding more layers of MLP projection leads to higher NC,, and improved transfer accuracy, although the performance improvement quickly plateaus at three layers of MLP. This could suggest that the effectiveness of projec-
1hts observation aligns with recent work Islam et al. 2021
21

---

Published in Transactions on Machine Learning Research (05/2024)
Training NC₁ (Cifar-100)
MSE (w/o proj.)
Cross-entropy (w/o proj.)
SupCon (w/ linear proj.)
SupCon (w/ mlp proj.)
0.001
0.771
0.792
2.991
Transfer Acc.
53.96
71.2
69.89
79.51
tion heads in model pre-training is not limited to contrastive losses but applies universally across various training loss types (c.g. CE and MSE).
Although we found the positive correlation between source NC and transferability exists in certain scenarios, our discussion in Section D demonstrates that there exists certain threshold where the positive correlation vanishes. This phenomenon could be attributed to a tradeoff bet ween feature diversity and class separation, which will be a compelling topic to study in future work. Finally, We note that since our experiments in this section are conducted on small-scale datasets, the observations may not hold in general.
C Additional Discussions
Besides examining the relationship between NCC and transfer learning regarding both source and target data, our findings also open up new avenues for future research that we discuss in the following.
Neuron collapse is not a pure training phenomenon. Previous work HHui et al. 2022 points out that NC is mainly an optimization phenomenon that does not necessarily relate to generalize cinttalizaaion (or trans- ferability). Our work, on one hand, corroborates with the finding that pretraining ACC does not always suggest better transferability, but also shows a positive correlation between pretraining NC and transfer- ability to a certain extent. On the other hand, our work also shows that reanstream AC on a dataset where NC well-defined correlates with the transfer performances across different datasets and thus could be general indicator for the transferability. This suggests that NC may not be merely an optimization phenomenon. An important future direction will pursue is to theoretically understand the connection between transferability and NC.
Boost model transferability by insights from NC. Our findings can be leveraged to enhance model transferability from two perspectives. Finstly, the correlation between pretraining AC and transferability suggests that increasing NC,, to a certain extent can boost transferability, which can be accomplished through popular methods like multi-layer projection heads and data angmentation. We expect that other methods that focus the geometry of the representations could also be developed. Secondly, by showing the strong correlation between downstream NCC and transfer accuracy, efficient transfer learning. Although our simple approach may not be the optimal way to take advantage of this relationship, we believe that there are more potent methods that could exploit this relationship better and therefore load to better transforability. We leave this as a topic for future invostigation.
devise simple but effective strategies for
Transfer learning beyond NC. Our results indicate that ACC on the source datasets is related to trans- ferability to a certain extent. However, identifying the threshold and explaining the change in transferability beyond this threshold is challenging using the NCC framework alone. Moreover, solely relying on NC to study transfer learning may not always be appropriate. For instance, recent research (Wang & Isola 2020) demonstrated that representations learned through unsupervised contrastive methods are uniform across hyperspheres, and (Chan et al. 2021 | showed that representations learned using the principle of maximal coding rate reduction (MCR²) form subspaces rather than collapse to single points. Therefore, further dis- closing the mysteries shrouded around transfer learning would require new frameworks and new tools, which we leave for future investigation.
Other Metrics for Measuring NC
Numerical rank of the features H. The NC, does not directly reveal the dimensionality of the features spanned for each class. In this case, measuring the rank of the features (H²) is more suitable. However,
22

---

Published in Transactions on Machine Learning Research (05/2024)
CF
MSE
SupCon
Miz wie projecsion nead MLP tuo-layer HIP NUP
Supdon sg! lnner
SupCon sg onedayee RIP
SupCon up bec-layer MLP
Transfer Acc
Transfer Acc
Transfer Acc
SupCon uy three ayer MLP
05.4
60.3
CT sas pesjection head
CE one byer NUP
two hop-l NLP Usros MLP
13 NO (Co)
AC2 (Cifar-100)
013 011
5:5
02
.N., (Cifar-100)
04
Figure 11: Trend of NC1 during training and transfer learning accuracy the pretrained models. ResNet-50 models are pretrained using Cifar-100 dataset with CE loss (Teft), MSF loss (Middle) and SupCon loss (Right). Models are pretrained with different numbers of layers for projection heads and transfered on the Cifar-10 dataset.
Figure 12: Comparisons of NC1 and transfer accuracy with different training losses and settings. We pre-train R8sNet18 models on the Cifar-100 dataset, and then test the models on Cifar-10. We use proj. to denote projection head; w/o proj. means without projection head, w/ w/linear proj. means adding one linear layer projection layer, and w/ mlp proj. means adding a two-layer MLP projection.
the calculations for both NC1 and rank are expensive when feature dimension gets too large. Thus, emphasize introducing numerical rank Timor et al.
2022 as approximation.
K H /
1
rank H)
K
where I|-|I represents the Frobenius norm and |1-Il2 represents the Spectral norm. (nr (numerical rank) could be seen as an estimation of the true rank for any matrix. Note that we use the Power Method to approximate the Spectral norm. The metric is evaluated by averaging over all classes, and a smaller rank H) indicates more feature collapse towards their respective class means.
Class-distance normalized variance (CDNV) (Galanti et al., 2022b). To alleviate the computa- tional issue of NC1, the class-distance normalized variance (CDNV) introduced in (Galanti et al.) 2022b) provides alternative metric that is inexpensive to evaluate. Let X denote the space of the input data a, and let Qs be the distribution over X conditioned on class k. For two different classes with Q; and Qj (i + j), the CDNV metric can be described by the following equation:
Var (Q.) Var &a (Q,)
Ven (Q., = 211 (Q:) Moo (Q,)
where Poo Ex (z) Poo [I
Ex-qu (2)) denotes the class-conditionall feature mean and Vars, (Qx) denotes the fcature variance for the distribution Qs- Although the exact expectation is impossible evaluate, we can approximate it via its empirical mean and empirical variance the given training samples as
(Q QQ) Varso (Q:) Var do (Q₂)
Pl (Q,) Pop (Q1)
1 nh
Boo (Qx)
Se (x),)); Var oo (Qk) n Σ (T
1oo (xx)122.
To characterize the overall degree of collapse for a model, we can use the empirical CDNV between all pairwise classes (i.e., Argszy!Voo (Q., (Q. Q,)))- If a model achieves perfect collapse on the data, we have Avgaz (Vop (Q., Q,)) Because the CDNV metric purely norm-based, computational complexity scales linearly with the feature dimension d, making good surrogate for NC₁ when the feature dimension d is large.
23

---

Published in Transactions on Machine Learning Research (05/2024)
E Extra Experimental Details
In this section, we provide additional technical details for all the experiments in the main body of the paper to facilitate reproducibility. In particular, Appendix E. includes all the experimental details for the figures (from Figure to Figure and Appendix E.2 includes all the experimental details for the lables (Table 2
and Table 3
General experiment sctups. We perform all experiments using a single NVIDIA A40 GPU, most exper- iments could be finished in less than 4 hours. Unless otherwise specified, all pre-training and transfer learning are for 200 epochs using SGT) with a momentum of 0.9, weight decay of 10-1, and dynamically changing learning rate ranging from 1 x 10-1 to 1 x 10-4 controlled by CosineAnnealingg learning rate scheduler as described in (Loshchilov & Hutter 2017p. When using ImageNet pre-trained models, we resize each input image to 224 x 224 for training, testing, and evaluating AC.
E.1 Technical Detail for the Figures
Experimental details for Figure I and Figure 2 In Figure I we pre-train ResNet50 models us- ing different lcvels of data augmentation and adversarial training. For data augmentation, we consider RandomCrop, RandomHoorivonallFigg ColorJitter, and RandomGraySoale.. We increase the levels of data augmentation by adding one additional type of augmentation to the previous level. For Level 1, we only do the standardization for samples to make the values have mcan 0, variance 1; for level 2, we add Random- Crop to the Level 1 augmentation; for level 3, we add RandomHorizontalFFiy to the previous level, and so on, so forth. Previous research has demonstrated the effectiveness adversarial pre-training for enhancing transferability (Salman et al.) 2020 Deng et al.) 2021). To investigate the impact of adversarial training with more detail, we adopted the -nrm bounded adversarial training framework proposed in (Madry
3 5 and 255 Our experimental et al. 2018). employing five different levels of attack sizes: 255 255 255 255
1
results, illustrated in Figure [ confirm that using smaller attack sizes in adversarial training can enhance the transferability of pre-trained models, while larger step sizes can negatively impact the transferability. We transfer the pre-trained models to four different downstream datasets: Cifar-10, 10, FGVC-Aircraft, DTD, and Oxford-IIIT-Pet. Although there are many other benchmark datasets to use, we choose these four because the number of samples for each class is balanced for the case of study. This aligns with the scenario where NC is studied while in Figure 2 we conduct similar experiments using public avaiable pre-trained models on ImageNet- 1k including ResNet (He et al. 2016 DenseNet (Huang et al. 2017). and Mobilenet-v2
(Papyan et al.] 2020)). In Figure 1 we pre-train ResNet50 models on Cifar-100 dataset,
(Deng et al. 2009
Sandler et al.] 2018
Experimental details for Figure 3 In Figure (a), we used a pre-trained ResNet34 (He et al., 2016) model trained on ImageNet-1k (Dng et al., 2009). To simplify the computation of NC1, we extracted the output from each residual block as features (as demonstrated in Figure 13), and then conducted adaptive average pooling on these features, ensuring that each channel contained only one element. We then computed NC1 and performed transfer learning on the resulting features. In Figure 3 (b), we used a pre-trained Vit- B32 (Dosovitskiy et al.] 2021) model that is publicly available online. For each encoder layer in Vit-B32, the output features h are of dimension 145 (which includes the number of patches plus an additional classification token) x 768 (the hidden dimension), i.c., h E R15x768 To conduct our layer-wise transfer learning experiment, we first applied average pooling the 145 patches reduce the dimensionality of the feature vectors. Specifically, we used an average pooling operator ap : RR 1114x766 7678 to downsample hi resulting in hu ap(h:). Subsequently, we trained linear classifier using the downsampled features h; top of each encoder layer.
Experimental details for Figure 7 In Figure 7 we investigate the effects of different fine-tuning meth- ods on the layerwise AC,, when are fine-tuning 1e models pre-trained ImageNet-1k. To facilitate
24

---

Published in Transactions on Machine Learning Research (05/2024)
the computation of NC1, we adopt the same adaptive pooling strategy as in Figure 3 on the features from each residual block of the ResNet model 13
Experimental details for Figure We empirically demonstrate that our proposed methods are more robust to data scarcity compared with linear probing and full model FT. To this end, we transfer the ImageNet-1k pro-trained ResNet18 model on subsets of Cifar-10 with varying sizes. More specifically, we select the sizes from logarithmically spaced list of values: [3, 10, 32, 100, 316) for each class. We note that when fine-tuning the models using Layer FT and SCL FT, in particular, we only fine-tune Block 5 of the pre- trained ResNet18 model. We also conduct the same experiment using the transformer-basee architecture, for which we fine-tune pre-trained CLIP model using subsets of Cifar-100 dataset with sizes of cach class from: 17, 17, 38, 87, 200). When fine-tuning with Layer FT and SCL FT, we report the results obtained by fine-tuning Layer of the pre-trained CLIP model 14] We run each experiment reported in the figure 3 times using different random sceds and report the average results.
Experimental details for Figure 12 In Figure 12 we pre-train ResNet50 models with different number of projection layers and different loss functions (CE, MSE, SupCon) using Cifar-100 for 200 epochs. Then we use the learned model to do transfer learning on Cifar-10. The NC, is then evaluated on the source dataset. For projection head used for different losses, we use MLP with ReLU activation, and we vary the number of its layers.
With the same pre-training setup as in Figure 1 we transfer the pre-trained models to four different downstream datasets: Cifar-10, FGVC-Aireraft, DTD, and Oxford-IIIT- Pet. We investigate the relationship between data aatpnnntttiio/dvvevaatatt training and transfcrability and show the relationship is not universal for the source dataset and transferability.
Experimental details for Figure 10
E.2 Technical Detail for the Tables
Experimental details for Table 2 The negative correlation between downstream NC, and transfer accuracy extends to few-shot learning, as shown in Table 2] We observed that the NC1 of the penultimate layer negatively correlates with the few-shot classification accuracies on minilmageNet and CIFAR-FS meta- test splits. To obtain few-shot accuracy, we followed (Tian et al. 2020) and first learned a feature extractor through supervised learning on the meta-training set. We then trained linear classifier on top of the representations obtained by the extractor. We calculated the NC1, of the penultimate layer in the feature extractor network using different architectures and observed that they are negatively correlated with the 1-shot and 5-shot accuracies.
Following previous works (Mishra et al. 2017 Oreshkin et al. 2018 ResNet12 is adopted as our backbone: the network consists of residual blocks, where each has three convolutional layers with x kernel; 2 2 max-pooling layer is applied after each of the first three blocks; a global average-pooling layer is on top of the fourth block to generate the feature embedding.
On the optimization side we use SGD optimizer with a momentum of 0.9 and a weight decay of 5e 44 Each batch consists of 64 samples. The learning rate is initialized as 0.05 and decayed with a factor of 0.1 by three times for all datasets, except. for minilmageNet where we only decay twice as the third decay has no effect. We train 100 epochs for minilmageNet, and 90 epochs for both CIFAR-FS. When training the embedding network on transformed meta-training set, we adopt angmentations such as random crop, color jittering, and random horizontal flip as in (Lee al. 2019). For meta-testing time, we train softmax regression base classifier. We use the multi-class logistic regression implemented in scikitlearn [Pedregosa et al. 2011 for the base classifier.
Thi method significantly reduces the computational complexity, while it may also introduce certain degree of imprecision in the AC1, calculation. 1or the CLIP model, We sweep learning rates [le-2 1e-3] and weight decays [le-4 0.0] select the optimal setting for each method on the full Cifar- 100 dataset and use the found hyperparameters for all expriments.
25

---

Published in Transactions on Machine Learning Research (05/2024)
ResNet Architecture
3et cnn 12% 2
3u5 caan 128
3x3 com, 126
216
256
112,
3xl can $12
Se comn $12
Ang pool
Sofftoo
3e can
3x3 con
comy
3x3
3el
Part
Part
Part 3
Part 4
Block
Block
Block
Block
Block
Block
Block
Block
3
Figure 13: Fine-tuning unit of ResNet. Image of ResNet18 from
Ramzan et al. 2019).
Experimental details for Table 3 In Table 3 we compare the performance between linear probing, MLP probing, Layer FT, SCL FT and full model FT based upon a wide variety of experimental setups, such as different model architectures, different pre-training and downstream datasets. In particular, we consider the following.
Fine-tuning blocks for different network architectures. Each residual block in the ResNet models is regarded as fine-tuning unit, and only the first residual block of each channel dimension is fine-tuned. As illustrated in Figure 13 the feature extractor of ResNet18 is partitioned into four parts, with each part containing two residual blocks of the same channel dimension, and the first block cach part is fine-tuned. In the case of the Vit-B32 model, each of its 12 encoder layers is considered fine-tuning unit.
SCF FT for different models. For SCL FT with skip connections, the B3t (same architecture is used CLIP) model had a constant feature dimension across layers, making it easy to directly apply the skip connection between the fine-tuned layer and the penultimate layer features. However, ResNet models have varying numbers of channels and feature dimensions the skip connection differently. To address this for ResNet, we first perform adaptive average pooling the innetuned layer features to ensure that cach channel had only one entry, and then we use pero-padding to match the number of channels with the penultimate layer features. Finally, for both ResNet and ViT models, we apply the skip connection and used the combined features to train the classifier.
layers, making it necessary to calculate
Batch Layer normalization. For a fair comparison between different fine-tuning methods, we always let the normalization layers [Toffe & Szegedy) 2015 Ba et al. 2016) update the running means and variances on the downstream data for all fine-tuning methods (i.e., linear probing, layer FT, SCL. FT, and full model FT).
In terms of hyperparameter selection, consider 2 learning rates ([le 1,5e 2] for ResNet models and [ee - 2, le - 3] for ViT) and 2 weight decays ([le 4, 4,5e 4] for ResNet models and [le 4, 0.0) for ViT) to identify the optimal settings for each method. Due to limited computing resources, we conduct hyperparameter search using the Cifar-10 datasct and use the found optimal setting for all datascts. Wo report the final selected hyperparameters in Table | We run each experiment reported in the table times using different random seeds and report the mean and standard deviation.
Experimental details for Appendix C In Appendix C we pre-trained ResNet18 models on the Cifar- 100 dataset using three different loss functions: CE, MSE, and SupCon, each for 200 epochs. For transfer the Cifar-10 dataset, we froze the pre-trained models and trained only a linear classifier on top of them for 200 epochs. Both procedures employed the cosine learning rate scheduler with initial learning rate of 0.1.
26

---

Published in Transactions on Machine Learning Research (05/2024)
Table 4: Hyperparameter setup for Table 3
ResNet
VIT
Lineor probe
MLP probe
Layer FT
SCL FT
Full FT
Linear probe
MLP probe
Layer FT
SCL FT
Full FT
Learning
5e
le
le
weight decuy
le 5e
le
5e 0
0
Table 5:
Comparison between Layer FT and Layer LoRA on the VTAB-1k benchmark. We
report the average results from different runs.
Cifar 100
Caltech101
DTD
Flowers 102
Pets
SVIN
Sum307
EaroSAT
Resised5
Retinopathy
Clevr-Conee CountCCunt
Clovr-Dist
DMLab
dSpr-Luc
dSpe-Ori
SNORB-
&NORB-ER
Average
Layer FT
56.7
93.5
84.7
70.7
37.8
53.3
51.8
65.3
Layer LoRA
61.3
91.7
65.3
100.0
91.7
61.0
96.3
79.3
75.3
61.3
60.5
15.8
45.0
31.7
16.2
63.4
We computed NCC, on the source dataset (Cifar-100) and evaluated the downstream dataset's (Cifar-10) test accuracy.
F
Additional Experimental Results and Discussions
In this final part of the appendices, we present extra complementary experimental results to corroborate our claims in the main body of the paper.
F.1 Combination of LoRA and Layer FT
LoRA (Hu et al. 2021 a popular parameter-efficien fine-tuning strategy broadly used in different domains of deep learning. Its core strategy involves integrating an additional low-rank sub-module into all layers of the network during the fino-tuning phase. On the other hand, our method choose a key layer to fine-tune. As such, the two methods serve complementary purposes and can be combined. Therefore, we combine the two approaches by applying LoRA on single layer (with rank= 96) while freezing the rest, we term this combined approach Layer LoRA. We then use Layer FT and Layer LoRA on the same layer and compare their performances using the VTAB-1k benchmark (Zhai et al. 2019). The results are reported in Table 5 We can observe that although Layer LoRA has a much lower parameter budget, its performance is not significantly lower than Layer FT in this limited data regime. This outcome hints at potential improvement of Layer FT for better parameter efficiency, which we leave a future direction.
F.2 Layer selection on ViT models
For ResNet-based models, our experiments demonstrate that fine-tuning layers closer to the classifier consis- tently yields smaller penultimate NC, values and superior transfer performance after fine-tuning. However, our exploration of ViT models reveals distinct pattern. Specifically, conduct Layer-FT for various layers and evaluate downstream NCC and transfer accuracy after fine-tuning, According to Figure 14 we make the following obsorvations: (I) the negative correlation between NC,, and transfer accuracy persists in these experiments.; (ii) marginal disparity in transfer accuracy performance across layers, except for the initial and final layers; (iii) the most collapsed models are typically obtained by fine-tuning middle-to-late layers. Motivated by these observations and maintain consistency with our experimental methodology applied ResNet models, we opt to fine-tune Layer of ViT models throughout our analysis. Notably, this selection consistently yields near-optimal transfer performances in most scenarios.
27

---

Published in Transactions on Machine Learning Research (05/2024)
FOVC-Aircratt
DTD
Dxdord-ll Dxforc-IBT-Pet -iT-Pet
Ciaar100
hursker Acs
CAS
XP
Trapsee 8cC
90.3
Trancfer ACC.
Traester
633
123
VINB Layers (shallow
Vie0 Ishalow deep!
Vi-B8 (shallow
VR-B (shallow
Figure 14: Transfer accuracy and penultimate NC₁ for VIT models when fine-tuning different layers. We conduct Layer-FT on the ViT model for different layers and show the downstream transfer accuracy and penultimate NC1 after fine-tuning.
Transfer Acc, Cifan10
Trandter Acc. Alrcratt
Transter DTD
Transfor AcC PET
T13n2ter A22
(Aircraf)
(DTD)
(PET)
Figure 15: Transfer accuracy on different downstream tasks and NC1, Transfer accuracy and NC, are measured on the same downstream datasets.
F.3 Experiments for Section & Section
Extra experiments for Section 3 In the main paper's Figure we illustrate a correlation between NC,, Cifar-10 10 and transfer accuracy various downstream tasks for ResNet50 models pre-trained Cifar100. However, when evaluating this relationship directly on downstream datasets for ResNet50 models pre-trained on Cifar100, the negative correlation becomes less apparent, as depicted in Figure 15 Nonetheless, we believe that NC₁ somewhat less sensitive when the magnitude loo large, as shown in Figure when NC₁ has a smaller magnitude, the negative correlation on downstream data becomes clear again.
Extra experiments for Section 4 In Figure 7 we only visualized the layer-wise NC1, for ResNet18 models on downstream dataset FGVC-Aircraft with different fine-tuning methods. Here, we include the experiments but for DTD and Oxford-IIIT-Pet in Figure 16 Additionally, for more comprehensive study, we conducted similar experiments for the pre-trained ViT-B32 models, where use the classification token of each layer to calculate the associate NC1. In Figure 17 we observe similar trend that we have seen on ResNet18 models, that within-slass variability of the VIT model (measured by NC1)) decreases from shallow to deep layers and the minimum values often appear near the penultimate layers. Moreover, we can observe that Layer FT and SCT. FT always have smaller penultimate NC, compared with Linear Probing and also better transfer performance.
28

---

Published in Transactions on Machine Learning Research (05/2024)
(07D)
(DTD)
Blocks 5 10TD)
(DT3)
Lnee Photing
Lieer P1 Mode:
Linew Presing
Full Mooel
Mozel LayerFT
Linear Procing
FuE Hosst
Block Chrslin Loed!
Block Littl Dueo
Boocc Zeep
Eloo Shalow beep
51
510
Licar Prob rg F.l FT
Uieer Prozing Fill Mozil FT
Lrear Probing
Lincor Prosing
Full MadclFT
Full Moli
Layer
Layer
Layer
Figure 16: Layer-wise NC1 for different fine-tuning methods on ImageNet-Ilk pre-trained ResNet18. We compare the layer-wise NC₁ across different fine-tuning methods, including linear prob- ing, Layer FT, and SCL FT, using the ImageNet-Ik pre-trained ResNet18 backbone. We evaluate the models on the downstream datasets DTD (Top), and Oxford-IITT-Pet (bottom). For the figures from left to right, we plot the results of fine-tuning only Block 1, Block 3, Block 5, and Block 7 in both Layer FT and SCL FT.
(VB. Layver (tirewtt
MM:8321 Layen's (6vreratt
Layarr BiArewuti
McB321 Layer (ereratt
Linear
Prob
Cer
Lse (ML Layer
(V2 Layer (
Mi: Layerr (CTD)
(V1 Loyer (DTD)
Uneer Probing Medell SCLFT
Unee Frobing fut Model FT
Uneer Probing nul Model
Uiner Frobing
Model
8
Lie (ell Giep!
(hiaalie Deay (V:8521 Layer (PET)
Lage (iil Dieg!
(VE-522) ((EE))
(Vi: Lave 5 (IET)
(VI: 0321 Layer 21 (PET)
12-
.00
Ful Mood
Liees Prating
30-
Linerr Prozing
Mode ST
Proft
LayerFT FT
Doupl
Sta Desp!
Figure 17: Layer-wise NC1 for different fine-tuning methods. We compare the layer-wise NC1 across different fine-tuning methods, including linear probing, Layer FT, and SCL FT, using the pre-trained Vit- B32 model. We evaluate the models on the downstream datasets of FGVC-Aircraft (top), DTD (middle), and Oxford-IITT-Pet (bottom). For the figures from left to right, we plot the results of fine-tuning only Layer 2, Layer 5, Layer 8, and Layer 11 in both Layer FT and SCL FT.
29


Reviews:
[{"review_id": "uW35c2k3A7", "reviewer": "Reviewer_V8Ro", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: Experiments are done on a range of downstream tasks and source (pre-training) tasks. The proposed fine-tuning method also seems to satisfy the claims of similar performance to full model finetuning while being parameter efficient. The authors demonstration that NC metrics measured on downstream tasks are a better indicator of transferability helps clear up confusion regarding whether NC is useful for transfer learning\n\nConcerns: \n1. It is unclear whether the NC metrics are computed before or after finetuning on the downstream task. If the metrics are computed before finetuning this is a useful finding. If the NC metrics are computed after finetuning then this is a less surprising finding and not very useful.\n\n2. All experiments seem to involve pre-training classifiers on larger image classification tasks. Demonstrating the authors' observation for models that are trained in a self-supervised or unsupervised manner would be much more interesting since this would shed light on how foundation models that are trained without labels function.\n\n3. For the authors' proposed finetuning method, the authors do not compare how efficient their approach is when compared to LoRA finetuning. This would be an important comparison since LoRA is a popular finetuning method that is also parameter efficient.\n\n4. In the authors' proposed finetuning method that involves identifying the right layer to train, it seems as though you need to run experiments to determine which layer is best suited for finetuning. Is this not fairly expensive to determine?\n\n5. If smaller NC1 is better correlated with transfer accuracy, and progressive collapse indicates that the last layers will have more collapse than previous layers, does this not indicate that the best layer to finetune is the last layer?\n\n6. If the best layer to finetune is below the last layer, does one really need to retain the subsequent layers in the network? It would seem that those layers are redundant?\n\n7. Have you tried finetuning through a simplex ETF layer rather than through a skip connection? This approach would integrate more NC insights into finetuning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the correlation between neural collapse (NC) metrics and the transfer learning performance of pre-trained classification models. The paper suggests that NC metrics measured on the downstream task are correlated with better transfer learning performance, and that this is a better way to ascertain transfer learning accuracy than NC metrics measured on the source task data. The authors also propose an alternative fine-tuning technique to linear probe fine-tuning that is comparable to full model fine-tuning while being more parameter efficient.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to see responses to the questions I laid out in my \"Concerns\" section. If they have already been addressed in the text of the paper, please point me to where they are addressed.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dGUgUU7II8", "reviewer": "Reviewer_D8Kd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Unless I'm missing something, the study of \"neural collapse\" applied to the target task is uneventful.  The definition of NC measures a combination of covariances to incorporate within-class similarity and between-class difference.  In particular, as classes become more separate, NC is higher.  So the main thrust seems to boil down to comparing one measure of class separability (NC) with another (linear classification), and saying they relate.  This is not a very useful conclusion.\n\nThings get more interesting in the other two pieces of this paper, skip-connection tuning and the study of NC on feature diversity.  However, both appear underdeveloped.  SCL FT is an interesting construction I don't think I've seen in this exact form before, but it isn't compared with more vanilla constructions like varying the number of layers being trained, or a linear classifier or 2- or 3-layer MLP/convnet starting from different feature layers.  SCL FT is basically using both an MLP (original path) and linear classifier (skip-conn path), with MLP being frozen to the original model.  What are the particular advantages of this (if any) compared to smaller MLP or unfreezing some more weights?  In addition, while this paper cites Kumar &al 2022, I didn't see a comparison to their 2-stage method of linear followed by full fine-tuning, which would be another good point of comparison.\n\nThe study of NC measured on the source data starts to show some interesting behaviors, but doesn't go very far in exploring them.  The fact that there there is a peak value for target task accuracy vs source NC (and near 1.0, no less) is curious, but overall expected:  in the limiting cases, maximum collapse happens when every datapoint maps to one of K points (for K source classes), so target classes would be distinguishable only to the point where they correlate with single source classes.  At the other extreme, as the authors also describe, completely random features would have low collapse.  So best target classifier accuracy ought to happen at an intermediate value, as was found.  But the fact that this value is right at 1 is a little suggestive --- it might be a coincidence, but to me it seems it may have to do with a balanced data distribution and structure of downstream classifier.  Is there a reason it would peak at this particular value?  What about for other data distributions or classifier heads?  It's possible this route might go too far astray, but to me some additional investigation here looks it could be worthwhile.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the relationship of \"neural collapse\" (NC) with transfer learning accuracy on downstream tasks, comparing linear classifiers on different features, end-to-end finetuned models and an introduced technique, \"skip-connection layer\" tuning.  More collapse is correlated with better accuracy, with both measured on the target task.  However, when measured on the source data, the opposite is observed, with higher accuracy is reached at relatively low collapse, and there is a peak where best accuracy is observed.  Skip-connection tuning is an interesting construction, training only one intermediate layer with both direct and layered connections to the classifier, and achieves good performance on a variety of task metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "All critical:\n* Explanation of what, if any, insights are provided by measuring NC on the target data.  Or alternatively, deemphasizing these sections.\n* Comparisons between SCL FT and a wider selection of simple transfer learning constructions.  In particular, comparing to unfreezing more layers, and to small MLP (rather than linear) on top of intermediate layers (penultimate or same layer as chosen for SCL).\n* Clearer explanation of the relation between NC, and SCL FT motivations.  I don't understand how the observations in sec 3 for NC are linked to inspiration for the skip-connection, as described on p.8.  Is there a reason the skip connection in particular (as opposed to other alternatives) is motivated by NC measurements?\n\nLess critical but would strengthen work:\n* Additional investigation of the peak value NC on source data --- why this particular value?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VmP52p9y3N", "reviewer": "Reviewer_aafv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. **Promising heuristic**: I find using a neural collapse heuristic to select the best layer to fine-tune on a model and interesting idea worth exploring.\n2. **Interesting observations**: The trends of Figure 2, 3, and 4 seem consistent and interesting.\n\n## Weaknesses\n\n1. **Lack of thorough experimentation**: Overall, I believe that most claims in Sections 4 and 5 lack proper empirical validation. For example, the paper makes significant statements regarding the improvements of their new method that are not substantiated by the marginal and inconsistent improvements in Table 2. Moreover, the ablation studies of Section 5 have only been performed on a nowadays toyish setting (fine-tuning a network on CIFAR-10 after pre-training on CIFAR-100) without proper repetitions. Still the observed trends are  extrapolated in writing to arrive at ver generic conclusions by the authors without proper evidence. I highlight that I am not against performing experiments on simple an low-compute settings. My main concern is with claiming that those experiments can say anything about more realistic settings (like fine-tuning a CLIP model on some video retrieval task) without proper evidence.\n2. **Unclear how neural collapse is computed**: The paper is unclear in which features are used in Eq. (4) to estimate neural collapse. Are thy the pre-trained features of the model over the fine-tuned data, or the fine-tuned features over the fine-tuned data? If the former, how is it possible that different fine-tuning techniques yield different levels of neural collapse? And if the latter, how is it possible that methods yielding the lowest level of neural collapse in Table 2 do not lead to the best performances?\n3. **Hyperparameter tuning strategy is unclear**: The paper does not mention what strategy was used to tune the different experiments. In this regard, I am worried that most results and comparisons with other baselines may be obscured by a lack of proper tuning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the connections between transfer learning performance an degree of neural collapse of different layers of a neural network. Specifically, it argues that more collapsed features, from the point of view of the fine-tuning data, lead to better fine-tuned performance. The paper then presents a new fine-tuning technique consisting on running linear probing over a linear mixture of the final layer features and a hand-picked layer deeper in the network (chosen based on the previous neural collapse heuristic). The results show sometimes a marginal improvement over just linear probing over the deeper hand-picked layer. Finally, the paper discusses the results of some ablation studies fine-tuning CIFAR-10 starting from models pre-trained on CIFAR-100 with different settings.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper requires substantial changes in my opinion in order to be accepted, probably beyond what is possible on a minor revision to the manuscript. In particular, in order to support the generality claims of the authors, I would need to see a significant revamp of the experimental results including repeating most experiments over a consistent suite of diverse datasets, pre-trained checkpoints, and training conditions. The experiments would need to be better structured and replicated to isolate the effect of different factors in the results. Furthermore, all the unclear points that I mentioned in the Weaknesses should be addressed properly, as right now, these unclear points make the whole premise of the paper inconsistent.", "broader_impact_concerns": "I see no broader impact concerns stemming from this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission investigates the relationship between neural collapse of pretrained networks and accuracy when those networks are transferred using a linear classifier or through fine-tuning. The submission shows that greater collapse as measured on downstream tasks is associated with greater transfer accuracy in a variety of settings. It proposes a “Skip Connection Layer Fine-tuning” strategy that fine-tunes only a single intermediate layer (the layer that exhibits the greatest collapse) with a skip connection to the output. Finally, it shows a result where neural collapse on the source task be positively or negatively correlated with transfer accuracy. Specifically, stronger augmentation leads to less collapsed representations that perform better on downstream tasks, but stronger adversarial training leads to less collapsed representations that perform worse on downstream tasks.

Reviewer D8Kd raised a concern that measuring NC on a downstream task is similar enough to classifier training that a correlation between the two is not particularly surprising. I generally agree—the performance of a classifier depends not only upon linear separability but also upon the difficulty of finding a good decision boundary from a finite dataset, which is in turn related to the amount of neural collapse. Reviewer aafv raised concerns regarding the thoroughness of the experimental setup. The revised paper includes some experiments in additional settings (e.g. neural collapse for CLIP models), and, although more thorough experimentation is possible, I doubt that it would radically change the findings. All reviewers raised issues regarding the clarity of parts of the paper, which the authors addressed in their revision.

I believe the revised submission satisfies TMLR’s acceptance criteria, and thus I recommend acceptance. Although not all results in the paper are surprising, the claims made are well-supported and the findings are likely to be of interest to some in TMLR’s audience.

For the final version of the paper, I request the following changes:

1. Reviewer D8Kd suggested revising the organization of the appendix to make it less scattered and moving some of these results into the main paper. The authors promised to do so, but didn’t want to make these changes before the final version since they would alter figure numbers. Please make sure to make the requested revisions.
2. There are some in-text citations that should be parenthetical citations (likely `\cite`s that should be `\citep`s) in the last bullet point of the introduction, the caption of Figure 2, and potentially other places.
3. Please double-check that ResNet34 and ViT-B are not switched between panels (a) and (b) of Figure 3.
4. Please clarify whether NC1 in Section 5 is computed on the training or test set.
5. There is at least one typo (“the NC1 of is too large”) on p. 12. Please do a final read-through to catch as many typos as you can.
Decision: Accept (poster)