Fine-Tuning is Fine, if Calibrated
Authors: Zheda Mai
Arpita Chowdhury
Ping Zhang
Cheng-Hao Tu
Hong-You Chen
Vardaan Pahuja
Tanya Berger-Wolf
Song Gao
Charles Stewart
Yu Su
Wei-Lun Chao
Venue: NeurIPS 2024 poster (2024)
Keywords: Fine-Tuning
Pre-training
Domain Adaptation
Primary area: machine_vision
TL;DR: We study the problem of fine-tuning a pre-trained model capable of recognizing a large number of classes with only a subset of classes in the new domains.

Abstract: Fine-tuning is arguably the most straightforward way to tailor a pre-trained model (e.g., a foundation model) to downstream applications, but it also comes with the risk of losing valuable knowledge the model had learned in pre-training. For example, fine-tuning a pre-trained classifier capable of recognizing a large number of classes to master a subset of classes at hand is shown to drastically degrade the model's accuracy in the other classes it had previously learned. As such, it is hard to further use the fine-tuned model when it encounters classes beyond the fine-tuning data. In this paper, we systematically dissect the issue, aiming to answer the fundamental question, "What has been damaged in the fine-tuned model?" To our surprise, we find that the fine-tuned model neither forgets the relationship among the other classes nor degrades the features to recognize these classes. Instead, the fine-tuned model often produces more discriminative features for these other classes, even if they were missing during fine-tuning! What really hurts the accuracy is the discrepant logit scales between the fine-tuning classes and the other classes, implying that a simple post-processing calibration would bring back the pre-trained model's capability and at the same time unveil the feature improvement over all classes. We conduct an extensive empirical study to demonstrate the robustness of our findings and provide preliminary explanations underlying them, suggesting new directions for future theoretical analysis.

Fine-Tunins is Fine, if Calibrated
Zheda Mai'; ; Arpita Chowdhury's Ping Zhang'; Cheng-Hao Tu', Hong-You Chen', Vardaan Pahuja', Tanya Berger- Wolf', Song Gav?, Charles Stewart', Yu Su', Wei-Lun Chao' The Ohio State University, PUniversity of Wisconsin Madison, 3 Rensselaer Polytechnic Institute.
Abstract
Fine-tuning is arguably the most straightforward way to tailor pre-trained model (e.g.. foundation model) to downstream applications, but it also comes with the risk of losing valuable knowledge the model had learned in pre-training. For example, fine-tuning a pre-trained classifier capable of recognizing a large number of classes master a subset of classes at hand is shown drastically degrade the model's accuracy in the other classes it had previously learned. As such, it is hard to further use the fine-tuned model when it encounters classes beyond the fine-tuning data. In this paper, we systematically dissect the issue, aiming to answer the fundamental question, "What has been damaged in the fine-tuned model?" To our surprise, we find that the fine-tuned model neither forgets the relationship among the other classes nor degrades the features to recognize these classes. Instead, the fine-tuned model often produces more discriminative features for these other classes, even if they were missing during fine-tumi fine-tuning! What really hurts the accuracy is the discrepant logit scales between the fine-tuning classes and the other classes, implying that a simple post-process pros-procsssing calibration would bring back the pre-trained model's capability and at the same time unveil the feature improvement over all classes. We conduct an extensive empirical study to demonstrate the robustness of our findings and provide preliminary explanations underlying them, suggesting new directions for future theoretical analysis. Our code is available at attps://github COom/SU-MMLL Www.MMBBFFFeee Feiiee--ee-eeeeeeeeeeeeeeeeeeeee Tuning- Is Fene-Fine--- If- aff-Caibbrate
Introduction
Pre-trained models (e.g., foundation models) have become an indispensable component in modern AI development [2]. Building upon neural networks with millions if not billions parameters and trained with gigantic amounts of data, these models have led groundbreaking results in various
domains 30 32 39
and shown several emerging capabilities not observed priorly 21 27
Yet, to obtain superior downstream performance, fine-tuning is still often needed. Typically, fine- tuning optimizes the model's performance on the available downstream data. Taking image classifica- tion example, end-users typically fine-tune the pre- pre-trained classifier to maximize the accuracy of a certain set of classes at hand, no matter whether they know up front that it is the complete set of classes not. As result, it is hard further apply the model to other classes, even if the model had learned about those classes in pre-training. (Please see Figure for an illustration.)
A recent work by Tu et al. 49 systematically evidenced such problem. They fine-tuned a pre- trained classifier with a subset of classes had learned fine.fiin- classes) and found this led to drastic accuracy drop in the other classes (i.e., absent classes). Tu el al. [49] viewed this an instance of catastrophic forgetting 212 24 and suggested two ways to address it: (1) identifying the model updating direction that can improve both the fine-tuning and absent classes; (2) preserving class relationships. They presented a strong baseline, combining a new stochastic gradient descent (SGD)
"quual contributions.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Model Pre-trainling
Model Fine-tuning Downstream Domain (c.g. mageMee S]
Model Deployment
ImageNet ImageNet-S F Fine-tuning Classes)
Pre-traning Domain (+.g, ImageRle))
Downstream Domain Ica. MageNet 5)
8
Absent Closs Date Acturasy
Dog
Fina-tuning
Dog
Dog
80
Eagle
Eagle
Eagle
40
Truck
Absent
Truck
Truck
100
Cup
Cup
Cup
20
Fine-tuning +V
Pre trainininng
Pre-trsiningg
Fine-tuning
Deploy
10
Fine-turiing
Tu ol al. 2023
80
Finestuning Class Data Accuracy
FIGURE 1. An illustration of tine-tunin (FT) a pre-trained model. Typically, FT is performed with the available downstream data at hand, yet in deployment, the model may encounter some other classes, for example, the ones it had been pre-trained upon. Idc- ally, the FT model should recognize all classes well.
FIGURE
Fine-tuning (x) + post-
processing calibration with a bias factor y (
can outperform the SOTA solu-
tion (+) 49
strategy, feature rank regularization 6 46), distillation [66. and frozen classifiers 22], achieving decent results on preserving absent class accuracy while improving fine-tuning class accuracy.
We build upon [49 to further analyze the problem of fine-tuning a pre-trained model. Specifically, we want understand which part of the pre-trained classifier or what knowledge it had learned is damaged after fine-tuning with a subset of classes.
We first analyze the feature extractor of the fine-tuned model. the fine-tuned feature extractor forgets the absent classes, the extracted features would exhibit poor discrimination among these classes or confuse them with the fine-tuning classes. We apply the Nearest Class Mean (NCM) classifier to assess the feature quality after fine-tuning, using the hold-out data for calculating class means. Surprisingly, the fine-tuned feature extractor does not degrade but often improves for the absent classes! The resulting features could better separate absent-class data and raise the NCM classification accuracy over all the classes. This finding suggests that fine-tuning with merely a subset of classes can already holistically improve the feature extractor for the downstream domain.
5 14 555
To search for the root cause of accuracy drops, we analyze the fine-tuned classifier as whole but decompose its prediction rule into two parts: a binary classifier separating fine-tuning and absent classes, followed by the multi-class classifier dedicated to each set of classes. We find that the fine-tuned model can distinguish among the absent classes very well, implying that the absent-class relationship in the fully connected layer is preserved. The only component that is damaged but ends up failing the whole model is the binary classifier. Concretely, the biased logits towards fine-tuning classes make most absent-class examples misclassified as fine-tuning classes.
The above a findings are encouraging and have profound implications! They imply that a simple post- processing calibration can potentially address the fine-tuned model's s inferior accuracy on the absent classes, bringing back the pre-trained model's s capability while unveiling the improved feature quality over all classes. Indeed, by adding calibration bias factor to all the absent classes' logits 4 41], the fine-tuned model can successfully reclaim the absent class accuracy and obtain decent overall improvement in the downstream domain (Figure 2p. The resulting performance even beats the strong baseline [49] in many of the benchmarks, including ImageNet and its variants [8] [3] [4],
Office-Homa (53) and VTAB [64) without complicated training and hyperparameter setring.
The "unexpected benign behaviors" of fine-tuning with a subset of classes raise several interesting follow-up questions. For example, are there any specific setups in our experiment contributing to the benign behaviors? If not, what are the explanations underlying these benign behaviors?
We consider different splits of fine-tuning and absent classes in terms of their distributions and numbers. We find that the benign behaviors are robust to the number of fine-tuning classes. Even if the fine-tuning and absent classes are semantically distinct (e.g., animals as fine-tuning and vehicles as absent classes), the absent class accuracy after fine-tuning stays quite close that of the pre-traines trained model without suffering catastrophic forgetting. We further investigate different optimizers for fine-tuning. When the SGD optimizer is used, the benign behaviors are robust to hyperparameters such as learning rates and weight decay. When more advanced, adaptive optimizers like Adam 2 are applied, we observe noticeable degradation with improperly selected hyperparameters. Still, with smaller enough learning rates and weight decay, the benign behaviors show up and hold.

---

We conduct further analyses to understand the benign behaviors. Specifically, we investigate how fine-tuning with subset of classes impacts the other (i.e., absent) class features. Our derivation shows that after each SGD step, the feature update of an absent-class example is governed by the gradients w.r.t. its most similar fine-tuning class examples. Suppose the gradients w.r.t. fine-tuning class data could effectively capture the downstream-speeitn preliminary explanation of why fine-tuning with a subset of classes could improve the other, absent class features in the downstream domain. Please refer to secctiion 6 for details and other analyses regarding the class relationship, frozen classifiers, etc.
information, our derivation offers a
Remark. We systematically dissect the damage caused by fine-tuning a pre-trained classifier with a subset of classes at hand. Our insights suggest that a simple post-processing calibration is sufficient to mitigate the damage, reclaiming the pre-training model's ' capability while holistically improving the accuracy the downstream domain. Our study also opens up several interesting questions worth further exploration. For example, it is intriguing that end-to-end fine-tuning of the whole model without additional regularization and intermediate supervision only degrades a part of the model.
We note that NCM and post-processing calibration are well-known machine learning techniques and we certainly do not claim them as our novelties. Instead, we use them because we find their respective properties and application scenarios suitable for our problem. We use NCM to assess feature qualities because its performance solely depends on features, not the last fully connected layer. We identify post-pre post-processing calibration as a promising solution because we find that the only major damage in the fine-tuned model is the biased logits towards fine-tuning classes. In other words, if the fine-tuned model also suffers from feature degradation or class relationship forgetting (to our surprise, it does not), calibration alone is unlikely sufficient to address the problem.
2 Related Work
Fine-tuning. The basic methods are linear probing and full fine-tuning 226 Parameter-efficien fine- tuning (PEFT) [60 [35 50] has attracted increasing attention lately (mainly for Transformers (52), aiming to update only fraction of pre-trained parameters on top of linear probing. We focus on full fine-tuning because it is aggnossc and arguably still the most widely used method. That said, we expect the insights and implications from our study not to be limited to full fine-tuning and potentially transferable to other fine-tuning methods as well.
Risk in fine-tuning, When the downstream data is scarce, fine-tuning is prone to over wett-fttinn and needs certain forms of regularization (28. Even with sufficient data to represent the downstream task, fine-tuning may risk losing valuable knowledge the pre-trained model had learned. For example, [57] showed that fine-tuning with data from a specific domain (e.g., ImageNet- IK real images) degrades the pre-trained model's generalizability to other domains (e.g.,
On an orthogonal dimension, [49] showed that fine-tuning a pre-trained classifier with a subset of classes it had learned led to a huge accuracy drop in the other classes. Similar phenomenons have been observed in [66] [4]. Along with these findings come several proposed solutions. [57] showed that a weight interpolation between the pre-trained and fine-tuned models reclaims the pre-trained model's generalizability. 49 investigated many approaches, including weight interpolation, but found they insufficient to preserve the accuracy in the other classes. They presented novel SGD strategy that helps identify the gradient direction benefiting both fine-tuning and other classes [66] developed dedicated solutions to CLIP-based rision-languse models to preserve accuracy for absent classes.
We 1) consider the fine-tuning scenario studied in (49) and 2) use the standard neural network classifier architecture with fully connected layer on top. Our focus is not to propose a brand-new solution and compete with existing ones, but to understand the underlying cause of the accuracy drop.
Continual learning and catastrophic forgetting. Fine-tuning a pre-trained model with subset of classes it had learned is related continual learning 7 but has several differences. Class-incremental learning 33 36) aims to expand the model's label space. In contrast, we suppose the pre-trained model had learned a wide range of classes; fine-tuning is meant to tailor it to a downstream domain (e.g., a different image style), not to learn extra categories. Compared to domain- incremmentlll learning 33 31, we do not ask the fine-tuned model to retain its performance in the pre-training domain. That said, the accuracy drop observed in our study and continual learning can all be considered certain forms of catastrophic forgetting [7 12]. Indeed, a recent survey by Wang el al. (56) argued that forgetting is common issue in various research fields, including foundation models, meta- meta-learning.
3

---

test-time adaptation, and generative models, among others. We thus expect our findings to serve as valuable reference for addressing the forgetting issue resulting from fine-tuning.
Post-processing calibration. Training with class-imbalanced data is known to produce biased logits towards major classes (6]]
Such issue not only appears in long-tailed recognition [20
18 but also
few-shot learning 162
45 and continual learning with replay buffers [48] 47 34] Post- Post-processing
calibration 3 38 58 59
is a widely applicable method address this issue, which adjusts model
confidence during inference. Popular methods include normalizing classifier norms (20] 1 and adjusting logits according to class sizes 61
38 22 65]. Unlike the machine learning problems above, whether or not post-pon post-procsssing calibration can address the accuracy drop in our problem is not immediately clear. Without access the absent class data, fine-tuning may have ruined the features or linear classifiers corresponding to these classes. If so, simply performing post-processing calibration cannot reclaim the accuracy the absent classes. Our main contribution therefore not merely the solution, but the systematic study that identifies post-processing calibration as an effective solution.
Other paradigms. There are several other machine learning paradigms related to the fine-tuning setting we study. We refer the readers to 49 for an in-depth discussion and comparison.
3 Background
Problem definition. We study the problem of fine-tuning a pre-trained classifier capable of recogniz- ing classes, using data from a subset of C classes at hand. The goal is to tailor the classifier the downstream domain (e.g., different image style).
More formally, let us denote by Dr = { ys 5) the data set for fine-tuning, where S is strict subset of the pre-trained model's 's label space D, i.e, |S] = C* < C' = V Let us denote a neural network classifier by ý = arg max.ey w. fo(æ), where x is an input sample, fo is the feature extractor parameterized by 0, and W = (w.))_ is the set of linear classifiers, a.k.a., the fully-connected (FC) layer. We call {0, W} the model parameters. The value w fe(x) is often referred to as the decision value logit of class c.
Without loss of generality, we define y f1..... C} and S = (1.... C'*, That is, they share the first C* classes. We call S the fine-tuning classes and use U = {(C" + "),,,,,,,, to denote the absent classes during fine-tuning, where S nu = @ and S Ju y.
Fine-tuning and its issue. Full fine-tuning updates the pre-trained model (0o, Wo} by
(OT, Wr] arg min W) Σe (zi, ye 0, W), with {0, W) initialized by (0o, Wo},
where é denotes the cross-entropy loss; {0r, } denotes the fine-tuned model.
Since Dvr only covers a subset of classes S, the fine-tuned model (0T, } was observed to degrade drastically classifying data from the absent classes U [49] Figure 2 shows one example: the absent class accuracy in the y-axis drops from 23% (x) to only 3% (marked x) after fine-tuning, even though the fine-tuning class accuracy in the x-axis increases hugely by roughly 60%.
Terminology. Tu et al. named the above setting Holistic Transfer (HT) [4]] The core challenge is how to maintain and even improve the fine-tuned model's ability to recogmize the C' C* absent classes in the downstream domain. For ease of reference, we use the following terms interchangeably.
fine-tuning classes & seen classes; absent classes & unseen classes; downstream domain & target domain; pre-training domain & source domain;
fine-tuning & naive fine-tuning.
We emphasize that the "unseen" classes are indeed seen in pre-training but absent fine-tuning.
4 A Systematic Study of Fine-Tuning (FT)
Seeing the ability to classify the absent classes "disappears" after fine-tuning, we are curious about how each component of the fine-tuned model (0r, Wr} contributes to it;

---

Fine--uning Classes
Absent Classes
Fire tuning Classes
Absent Classes
Fine furing Classes
Absent Classss
Accunacy Gain
Gain
Accuracy Gain
20
-20
40
NN Accurasy
NCM Accuraty
NN Acouracy
Nocurryryyy
NN Accuracy
NCM Accurary
(A) ImageNet-(RR S}
(B) VTAB
(C) Office-Home
FIGURE 3. Accuracy gain after fine-tuning, We consider the neural network (NN) classifier with the FC layer section and the NCM classifier using only features (Equation 2}. We show the average accuracy gain on fine-tuning classes (Accs/y) and absent classes (Acou/s)). While the NN classifier suffers drops Accuyy. the NCM classifier enjoys a consistent gain, suggesting the holistic improvement of features after fine-tuning.
whether the ability is "forgotten" forever "buried" temporarily by some other damaging factors emerging during fine-tuning.
To this end, conduct systematic amalysis, aiming dissect the degradation caused by fine-tuning.
4.1 Experiment setup: datasets, models, and evaluation metries
We focus on two of the largest datasets used in 49 We also consider the ImageNet Distribution Shift benchmark widely used in out-of-distribution (OOD) generalization (57).
1. Office-Home [53]: a domain adaptation dataset with 65 classes and 4 domains, For each source- target pair, we pre-train ResNet-50 [4] using the source data and fine-tune it 30 randomly selected classes in the target domain. We evaluate the model on all 65 classes in the target domain.
2. VTAB [64] set of 19 visual recognition datasets. We follow 49 to use the 8 datasets provided with text labels and use CLIP (ViT-B/32) 42 as the pre-trained model. We extract the class name embedding to construct the FC layer and discard the CLIP text encoder afterward. We fine-tune the model on the randomly selected 50% of classes in each dataset and test on all the classes.
3. ImageNet-R [15] and ImageNet-S (54) datasets for OOD detection and generalization (57]. ImageNet-R 15 consists of 200 ImageNet classes [8 43 with renditions (paintings, cartoons, etc.). ImageNet-S [54] consists of IK ImageNet classes with sketching. We use ResNet-50 (ViT- B/32 (9) results in Appendix D pre-trained on ImageNet-IK, We fine-tune them on randomly sampled 50% of classes (100/500 for ImageNet-Ree respectively) and test them on all the classes.
By default, we use the cross-entropy loss and optimize the model using the SGD momentum optimizer.
Evaluation metric. Let Drr {(x:, V/ E } denote the downstream test set covering all the classes, we define the accuracy of classifying data belonging to classes into the label space by
A/B = E. Em E Al x 1ly = arg max w fo(::)
Σ, U1 E A]
For instance, Accs/y is the accuracy of classifying fine-tuning class data into all C classes; Accu/y the accuracy classifying absent class data into all classes. We note that these are the two accuracies reported in [49] and depicted in Fiiure 2] In this paper, we further consider Accs/s and Accy/u, corresponding to classifying each set of data into their respective label space.
4.2 Is the fine-tuned feature extractor damaged?
We first investigate the feature extractor fe, i.e., whether the fine-tuned extractor fe, forgets the discriminative ability to differentiate absent-class samples. We apply the NCM classifier 44 55). whose accuracy is solely governed by the feature quality. examine the feature extractor. Given test example æ, the NCM classification rule is
for (x) [| (m)l12
ÿ arg min
Me
(2)
which outputs the class 5 E B whose feature mean μ4 is the closest to (ar).. We hold out a subset of the downstream data to calculate the class mean, for both fine-tuning and absent classes.
5

---

86
62
mageltt
14 36m
Imaghleet
Aetureyy
50
Naart
12
20
Farch
FIGURE 4. Accu/er along with the fine-tuning epochs.
FIGURE 5. Average predicted probability that absent class ex- amples belong to absent classes.
FIGURE 6. % of absent class examples classified as absent classes.
We compute the NCM accuracy (with B ))) the fine-tuning class and absent class data using the fine-tuned (FT) feature extractor fe,, i.e.. (cos/>((or) and (ccu/y((err) We also apply the pre-trained feature extractor feo and obtain Accs/y and Acty/y((e>) We report the NCM accuracy gap between the two feature extractors in Figirr 3] As one may expect. the FT extractor for improves the fine-tuning class accuracy. However, surprisingly. also improves the absent class accuracy without catastrophic forgetting. This result sharply contrasts the accuracy obtained by the full neural network (NN) classifier (cf. |section 3). As shown in [Figure 3] the absent class accuracy by the NN classifier drops by 15 ~ 40% after fine-tuning. In short, we find that fine-tuning with subset of classes can adapt the feature extractor holistically to the downstream domain to improve both the fine-tuning and absent classes, capturing the gradient direction of the domain shift 49
Remark. We use NCM only to assess feature qualities, not as the final classifier. After all, we do not have absent class data to compute the class means. This contrasts the use of NCM in continual or few-shot learning, where the prior- or few-shot class data are accessible (e.g., through replay buffers).
We emphasize that the feature improvement here is not as trivial as observed in conventional transfer or few-shot learning. In these learning problems, the pre-trained model typically had not learned about the absent classes; thus, there is no risk of forgetting them. Besides, the fine-tuning and absent classes are often treated as two separate tasks in evaluation; thus, it is unclear whether the fine-tuned feature extractor improves the overall classification accuracy. In contrast, in our fine-tuning setting, the pre-trained model had already learned about the absent classes. Fine-tuning thus risks forgetting them. However, as shown in Figure 3] finettuning improves absent class accuracy, in the context where the fine-tuning and absent classes are evaluated together in a single task.
4.3 What is damaged in the fine-tuned neural network classifier?
The findings in subsection 4.2 climinate the feature extractor Oy from the candidate root causes of the degraded FT model (0r, W. The drastic drop of absent class accuracy thus must come from the FC layer Wi or its alignment with the feature extractor. To analyze what goes wrong, we decompose the softmax probability induced by the neural network classifier as follows,
exp (w. fe(x))
(x(
p(c(2)) Evey exp (w fo(x))
(3)
Ee E'(E())))) Fe (x))
Eveu Ze
2 (x)
(4)
Eces Zel (2) Eeeu Zel (x) Eveu Ze
where ==(x)) exp (m. fo())). Let be absent class, the 1st term stands for the predicted probability that æ belongs to the absent classes U, not the fine-tuning classes S; the 2nd term stands for the probability that within the absent classes U. x belongs to class Correctly classifying an absent class example (x, y E 21) i.e. y arg max,ey p(c|z)) thus requires obtaining a high probability in the 1st term and 2) correctly classifying the example among absent classes.
Building upon this insight, we analyze the accuracy of taking arg max of the 2nd term classify absent class examples. We note that this is exactly the Accy /u defined in subsection 4.1, As shown in Figure 4| Accy,u does not degrade but improves in the first few epochs of fine-tuning and stays stable afterward. This result implies two key messages, First, the FT model does not forget its ability to distinguish among absent classes. Second, the drastic accuracy drop of Accu/y results from the 1st term in Equation 4 the binary classifier separating fine-tuning and absent classes.
As shown in Figure 5] the predicted probability that absent class examples belong to absent classes (i.., the average of the 1st term over absent class examples) reduces notably along with the fine-tuning

---

ImageNat-(R. (R,
VTAB
Office-Home
(5)
A
Accs/y / Accu/y/ y Accy/y Accs/> / Accu/y y /Acy/y/ |Acssyy |Acc/ />
Pre-trained
23.6
23.5
23.8
58.3
59.3
57.4
63.8
61.8
65.5
Fine tuning Tu et al.
43.3
81.3
3.5
62.8
86.4
39.1
53.5
88.3
22.4
49
47.5
62.1
23.1
63.8
83.9
43.5
72.0
78.2
66.6
Fine uning + "ALG
55.9
80.3
30.5
66.8
85.3
48.2
65.0
87.7
44,9
Fine-tuning +eev
57.4
47.1
67.8
72.2
82.3
63.1
Finc-tuning T
60.8
73.6
47.6
69.3
75.6
62.8
72.7
79.1
66.9
Oraele
71.1
72.4
69.8
80.6
79.8
81.3
82.1
81.2
82.9
TABLE 1. Post-processing calibration can effectively bring back the pre-trained model's 's capability in recogniz- ing absent classes. Oracle is based on classifier fine-tuned with both fine-tuning and absent class data.
epochs, suggesting the tendency to misclassify an absent class sample as fine-tuning classes. Indeed, as shown in Figure 6) for most of the absent class samples, the FT model ends up producing a lower value of max.eu w [o( than maxces w fo(æ), misclassifying them into fine-tuning classes. In short, the main factor that damages the FT model's 's ability to correctly classify absent class examples is the biased logit values towards fine-tuning classes.
5 Post-Processing Calibration for the Rescue
The systematic study in section 4highlighs several key characteristics the FT model {θr, Wr}. First, the model retains and even improves the accuracy of classifying absent class samples when the label space is limited to absent classes (i.e., Accu/u). Second, the model tends to assign much higher decision values w. fe (z), a.k.a. logits, to the fine-tuning classes, ending up misclassifying most absent class samples into tinettuning classes and thus hurting Accury-
These characteristics suggest that a simple, post-procsssing calibration of the FT model's logits could potentially bring back the pre- pre-trained model's ability to classify absent classes correctly. To this end, we apply the calibration approach proposed in different context of zero-shot learning 4 which adds a factor y uniformly to the logits of all absent classes, leading new classification rule
g arg max
w. fo()) 71 E 21).
This calibration factor lifts the logits of absent classes level comparable with those of fine-tuning classes. Suppose an absent class example is correctly classified among absent classes but misclassified into fine-tuning classes, adding a sufficiently large " could reclaim the correct prediction.
We design two approaches to properly set 7 without accessing the absent class data. Average logit gap (ALG) estimates by the average logit gap between non--roound-rree truth fine-tuning
Abscent Class Data Accuracy
ImageNet ImageNel-R (50%, Fine-turing)
8 8 8 & 8 8
classes and absent classes in the fine-tuning data Dr. Pseudo cross-validation (PCV) partitions Dr into pscudo-finc-tuning and pseudo-absent classes and finds y that can balance the pseudo-fine fine--ninnttt-ggg and pseudo-absent class accuracy. We also investigate "cheating" ** based the test data estimate the upper-bound bound results. More details are in Appendix B
60
Fine-tuning Class Data Accuracy
Prestahhing
Preieiiing
Firetuiing
Performance. Table I shows the results on ImageNet- (R, S). VTAB, and Office-Home, averaged over datasets within each benchmark. Despite its simplicity, calibration effectively boosts the FT model's accuracy on the absent classes, achieving comparable accuracy to the SOTA method proposed in [49]. On ImageNet-(R. (R, SJ, both calibration approaches outperform the SOTA by a notable margin.
2008
ot 2023
FIGURE Fine-tuning vs. Absent Class Accuracy Curve (Acos/y at the x-axis; Accu/y at the y-axis) by vary- ing the factor y for ImageNet-R.
Compatibility of calibration. Post-processing calibration is potentially applicable to any model. In In/Figure 7 we extend the calibration approach to the pre-trained model and the SOTA model 49 with varying y. Calibration can effectively balance Accs/y and Accu/y and trade one for the other. Interestingly, we find the curve of the FT model, most cases, can well cover those of the other models. To characterize this, follow [4]
AUSUC [IN-[R, V
O-H
Pre-trained
0.083
0.444 (0.499
Tu et al. [49
0.317
0.553 006188
Fine Sine-tuning
0.439
0.586 (0.63
TABLE 2. Results in AUSUC.
7

---

ImagaNut-S5
0.5
ImaguNuttS
25.0
Office-Home
0.2
Office-Home Diff.]
5.0
itanrrcyy Random
Hemethy Rundom
Hundom
0.2
19:95
0.09
0.1
0.0
0.
0.3
00
0.12
0.18
-22
5:48
9.30
12.66
10.0
Doo
0.11
0.15
-.1
-740
2.64
1.06
-25
318 Num
210
348
110
218
20.0
15
25
Fine tuning Clsss
Num
Fine buning Class
Num of Fiie tuning class
Num of Fine buning Class
FIGURE 8. The performance gain on AUSUC and NCM Aceury (from the pre- pre-trained model to the FT model), under different data splits and fine- tuneettnngg class sizes. The hierarchical split of ImageNet-S contains dog (118 classes), mammal (218 classes), and animal (398 classes) as the fine-tuning classes.
calculate the Area Under the Scen-Unseen ((Fine-tuning Absent)) Curve (AUSUC) summarize the overall performance of each model across the entire spectrum of 7. Table 2 reports the AUSUC. The FT model notably outperforms the other methods, showing its robustness in learning from subset of classes.
Our study is not meant to compare them, but to show that calibration can effectively address the issue caused by fine-tuning. We also note that Tu et al. [49] has investigated many solutions, including weight interpolation 57). but found them less effective them their proposed SOTA method.
Remark. Many post post-processig calibration methods have been proposed (cf.. /section 2).
6
Ablation Study and Additional Analysis
Data split and fine-tuning class size. In the default setup of 49 fine-tuning classes and absent classes are uniformly randomly sampled and have each portion close to 50%. In practice, however, end-users may have smaller number of classes at hand or collect data from semantically or conceptually similar classes whose appearances are positively correlated. To explore such practical situations, we investigate how 1) biased sampling such that the fine-tuning classes are conceptually or distributionally similar to each other than the absent classes and 2) smaller size of fine-tuning classes would impact the performance of fine-tuning. Specifically, for Office-Home, classes that similar in the pre-trained model's feature space selected as fine-tuning classes, leaving the rest as absent ones. We also vary the number fine-tuning classes. For ImageNet-S, we leverage the WordNet [1I hierarchy to select coherent groups, such as dog (118 classes) and mammal (218 classes) as the fine-tuning classes. Please see Appendix A for details.
Figure 8 illustrates the performance gain on AUSUC and NCM Accu/y (from the pre-trained model to the FT model). Notably, the hierarchical split poses a greater challenge for fine-tuning in transferring domain knowledge from fine-tuning classes to absent classes, evidenced by its relatively minor AUSUC improvements compared to the random split. This difficulty is attributed to the inherent challenge of transferring features Iearned from dogs animals to distinctly different classes like TVs and trucks. Furthermore, smaller fine-tuning class sizes present additional difficulties, leading to less pronounced improvements, especially in Office-Homee In summary, suggests that the benign behaviors of fine-tuning are robust in more practical and difficult splits but its performance requires further improvement when the fine-tuning class size is exceptionally small. Detailed experimental setup and additional results are available in Apppendix D
analysis
Optimizer. Prior work 49 predominantly used the SGD optimizer. To investigate optimizers' influence in fine-tuning, we scrutinize six popular optimizers including SGD with Momentum, Adam (24] AdaBelief [67] Adadelta [63]. AdaGraa [10) and RMSprop. Our study includes 1) variation across learning rates (LR), adjusted as multipliers of the default ones, i.e., [10, 1, 0.1, 0.01] x default LR, and 2) three distinct weight decays: [0, 5e 4, 5e 3). The AUSUC, illustrated in Figure 9 reveal the robustness of the benign behaviors to different hyperparameter settings of the SGD optimizer. Conversely, more advanced, adaptive optimizers show higher sensitivity when hyperparameters are not properly chosen. Nevertheless, under small enough learning rates (and weight decay), they perform similarly to SGD and notably improve the pre-trained model (whose AUSUC is around 0.5 in -omee..
Why is absent class relationship preserved? To elucidate the preservation of the absent class relationship, we analyze how classifier weights W change within the fine-tunigg and absent classes. We commence by calculating the L2 normalized weight changes between fine- ine-tuning and the pre-
8

---

AdaBelief
Weight Decay
Se-4 Se:
0.57
0.62
0.58
0.6
Weight Decay
Se Se
0:62
0:11
0:02
0.6
Weight Decau
0:63
0.62
0.47
0:02
0.6
0.57
0.62
0.5
0.63
0.36
Sn-4
0.63
0.63
0.4
-0.4
0.4
0.57
0.63
0.61
0.04
0.63
0.63
0.01x
0.1x
10x
011
0.1x
1x
10x
0.01x
0.1x
1x
10x
Learring
Default Learring
Default Learning
Adadelta
AdaGrad
RMSprop
Weight Decay
Se-4 Se-
0.62
0.58
0.02
0.01
0.6
Weight Decay
3e4 34
0.62
0.40
0.05
0.02
00.6
Weight peray
0.02
0.02
0.01
0.01
0.6
0.62
0.62
0.55
0.02
0.67
0.61
0.15
0.01
0.5
0.02
0.02
0.01
0.01
0.5
0.4
0.4
0.4
0.62
0.62
0.55
0.03
0.62
0.63
0.46
0.01
0.59
0.27
0.03
0.02
0.01x
Default Learring
Learning
Default Learning
FIGURE 9. Different optimizers and hyperparameters. We use the Office-Home and report the AUSUC. The AUSUC of the pre- pre-trained model is ~ 0.5. Fine-tuning with SGD shows remarkable robustness to different learning rates and weight decay. Advanced optimizers necessitate more careful hyperparameter selection. Nevertheless, they perform similarly SGD under small learning rates and weight decay.
Fredem ey Civres Dan't ef Weihht Change Neleroratt
Jown Carees Chasiier We Chergeffeee eratts
24
244
42
FIGURE 10. Classifier update direction similarity within fine-tuning (left) and absent (right) classes for ImageNet-S. The update directions are highly similar within absent classes, thus preserving the inter-class relationships among absent classes.
WT-Wo Wo trained model AW W Wols Visualizing the cosine similarity of the change within fine-tuning
and absent classes, i.e. AW AWW() and (AW"" allows us to assess the directional similarities in classifier updates. As depicted in Figure 10 there exists a pronounced dissimilarity in the update directions among fine-tuning classes since the gradients aim to separate fine-tuning classes during fine-tuning. Conversely, notable similarity in the update directions is observed among absent classes. The absence of positive signals from absent class data results in nearly uniform updates of absent classifiers, thereby preserving the class relationships among absent classes throughout fine-tuning. More results for other datasets can be found in Appendix
Absent Features and linear classifier Alignment in fine-tuning, |section 4 has demonstrated that the alignment between absent features and the linear classifier (i.e., the FC layer) remains intact during fine-tuning, as evidenced by the stable or improved Accu/u, despite the absence of absent class data. This indicates that fine-tuning retains, and even enhances, its capacity to differentiate among absent classes. To delve deeper into this phenomenon, we scrutinize the behavior of ground -ruth class (GT) vs. the largest non-GT absent logits for absent test samples. As depicted in innFie 12, although both sets of logits exhibit decline throughout the training process- attributable to the lack of absent data during fine-tuning-te the relative difference between them persists, underlining a consistent and stable alignment between the features and the linear classifier. Future work may delve into the theoretical understanding of this observed alignment between features and classifiers for absent classes.
Why do absent class features improve after fine-tuning? During fine-tuning, the pre-trained model is updated solely by gradients derived from the fine-tuning class data. Surprisingly, the fine-tuned feature extractor does not forget but improves its ability to differentiate the absent class data. Here, we explain it by considering a two-layer linear neural network ý = arg max, w. (Ux)- = arg max, un_ 2. Let us denote by D = [(æ,, yi E N) a mini-batch during SGD, and denote by E the gradient w.r.t. extractor U is thus Voe updated feature extractor is U + U
Ux using the cross-entropy loss. The gradient w.r.t. the feature (V,,
Suppose we apply SGD with learning rate y,, the N Σ, V V2,lZz Building upon this formula, we can further

---

Target Data Movemmet tuning
bagee non-GT shoen
GT atsen lighs
Ire-rsined Fextures
8.50
Fine-tured Feoture
6.25
1.75
7.50
7.00
22
Spochs
FIGURE 12. We analyze the ground-truth (GT) FIGURE 11. The toy example demonstrates that the fea- logits compared with the largest non-GT absent log- its in ImageNet-S. While both logits decrease dur- tuning class training data, resulting in absent class features ing the training, the consistent relative difference moving in a similar direction as fine-tuning class features. between them signifies stable alignment between the features and the lincar classifier.
6.5 10.0 13.0 11.5 12.0
tures of absent class data are influenced by their similar fine-
derive how the feature z Uw of an absent class data æ changes after the model update
U
" N
x=
" E (.. N
(6)
Namely, the update of is governed by its similar training examples those o; with high inner products x. æ with - and their corresponding feature gradients V., f. Suppose the domain shift affects similar classes similarly and the gradients w.r.t. the fine-tuning class data and features could effectively overcome the domain shift, 6Joffers a preliminary explanation of why fine-tuning could improve the absent class features in the downstream domain.
To further illustrate this, we design a toy example with four classes, visually depicted by different colors in Figure 11. Blue and cyan denote the fine-tuning class data; red and magenta denote the absent class data. The dimensionality of æ and = are set to be 2; the size of U is thus x 2. We deliberately set to be non-negative to simulate the output of ReLU operation. We create the pre-training dataset and the fine-tuning dataset by performing local translations to the data, We then pre-train two-layer multi-layer perceptron (MLP) the pre-training data with four classes while keeping the first layer's weight (i.e., U) frozen as an identify matrix to ease visualization (i.e., 2). After pre- pre-trainin,, we then fine-tune the model the downstream fine-tuning data with only two classes without freezing U. (Please find more details about the data creation and the model architecture in Appendix A As shown in Figure 11 after fine-tuning, the update of the absent class features (from o to *) follows the update of their closest fine-tuning class features, even though the absent class data is not involved in fine-tuning Moreover, different classes stay quite distinguishable in the fine-tuned feature space, suggesting that fine-tuning with a subset of classes would not degrade but improve the feature quality.
More analysis. Due to the page limit, we leave more analyses including frozen classifiers and frozen backbones in fine-tuning, the investigation of biased logits toward fine-tuning classes, and absent class relationship analysis in AAppendxx C
7 Conclusion
"What happens if one fine-tunes pre-trained classifier with a subset of classes?" Prior work showed that while it improves the fine-tuning class accuracy in the downstream domain, it drastically degrades the model's ability recognize the other classes the model had previously learned. Our systematic study, however, provides a different opinion. We found that fine-tuning does not degrade but often improves the model's ability to recognize the other classes, if the classifiers' logits are well-calibrated. We expect our study to serve as a valuable reference for practical fine-tuning of pre-trained models.
"Suppose æ and æ;. Vi, are all non- negatiee vectors (for instance, coming from a ReLU operation on top of the prior neural network layers), their inner products will be non nog-negative well.
10

---

Acknowledgments
This research is supported by grants from the National Science Foundation (ICICLE: OAC-2112606).
We are grateful for the generous support of computational resources provided by the Ohio Supercom- puter Center.
References
[1] Sara Beery, Eljjah Cole, and Arvi Gjoka, The iwildcam 2020 competition dataset. CoRR, arXiv preprint arXiv:2004.10345
[2] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint
2021.
[3] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-259, 2018.
[4] Wei Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized seot learning for object recognition in the wild. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 52-68. Springer, 2016.
[5] BB Chaudhuri. new definition of neighborhood of point in multi-dimennional space. Pattern Recognition Letters, 17(1):1111,, 1996.
[6] Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. Advances in Neural Information Processing Systems, 32, 2019.
[7] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting in classification tasks. IEEE transactions pattern analysis and machine intelligence, 44(7):3366- 3385, 2021.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[10] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
[11] Christiane Fellbaum. WordNet: An electronic lexical database. MIT press, 1998.
[12] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, ((4):128-135) 1999.
[13] Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4367-4375, 2018.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[15] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis -ut-of-distribution generalization. ICCV, 2021.
11

---

[16] Geoffrey Hinton. Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.000 000000000.22220000 2015.
[17] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 831-839, 2019.
[18] Mobarakol Islam, Lalithkumar Scenivasan, Hongliang Ren, and Ben Glocker. Class-distribution- aware calibration for long-tailed visual recognition. arXiv preprint (oXiv:2009.02663. 2021.
[19] Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In Computer Vision-ECCY 2016: 14th European Conference, Amstenlam, The Netherlunds, October 11-14, 2016, Proceedings, Part VII 14, pages 67-84 Springer, 2016.
[20] Bingyi Kang, Saining Xie, Mareus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long- recognition. arXiv preprint
2019.
[21] Salman Khan, Muzammal Nascer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey, ACM computing surveys (CSUR), 54(10s):1-41, 2022.
[22] Byungju Kim and Junmo Kim. Adjusting decision boundary for class imbalanced learning. IEFE Access, 8:81674-81 81685, 2020.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
[24] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins. Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barbbkka--Briiska.. et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):3521-3326 2017.
[25] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519-3529. PMLR, 2019.
[26] Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2021.
[27] Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang. Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends@ in Computer Graphics and Vision, 16(1-2):1-22 2024.
[28] Dongyue Li and Hongyang Zhang. Improved regularization and robustness for fine-tuning in neural networks. Advances in Neural Information Processing Systems, 34:27249 272622, 2021.
[29] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pages 6028-6039. PMLR, 2020.
[30] Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey, arXiv preprint arXiv:2403. 14735 2024.
[31] Vincenzo Lomonaco, Lorenzo Pellegrini, Pau Rodriguez, Massimo Caccia, Qi She, Yu Chen, Quentin Jodelet, Ruiping Wang, Zheda Mai, David Vazquez, al. Cvpr 2020 continual learning in computer vision competition: Approaches, results, current challenges and future directions. Artificial Intelligence, 303:103635, 2022.
[32] Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Tianming Liu, Gao Cong, Yingjie Hu, et al. On the opportunities and challenges of foundation models for geospatial artificial intelligence. arXiv preprint arXiv:2304.06798, 2023.
12

---

[33] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing. 469:28-51, 2022.
[34] Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3589-3599, 2021.
[35] Zheda Mai, Ping Zhang, Cheng-Hao Tu, Hong- You Chen, Li Zhang, and Wei-Lun Chao. Lessons learned from a unifying empirical study of parameter-efficient transfer learning (petl) in visual recognition. arXiv preprini arXiv:2409. 16434, 2024.
[36] Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pullern Analysis and Machine Intelligence, 45(5):5513- 5533, 2022.
[37] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[38] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR, 2021.
[39] Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):259-266 265, 2023.
[40] Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puncet K Dokania. Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint 2rXiv: /3320, 2023.
[41] Tai- Yu Pan, Cheng Zhang, Yandong Li, Hexiang Hu, Dong Xuan, Soravit Changpinyo, Boqing Gong, and Wei-Lun Chao. On model calibration for long-tailed object detection and instance segmentation. Advances in Neural Information Processing Systems, 34:2529- 2542, 2021.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748 8763. PMLR, 2021.
[43] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 11:21112255 2015.
[44] José Salvador Sánchez, Filiberto Pla, and Francese J Ferri. On the use of neighbourhood non-parann non-parametric classifiers. Pattern Recognition Letters, 18(11-13): 179 1186, 1997.
[45] Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Gener- alized zero-and few-shot learning via aligned variational autoencoders. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8247-8255, 2019.
[46] Yujun Shi, Jian Liang, Wenqing Zhang, Vincent YF Tan, and Song Bai. Towards understanding and mitigating dimensional collapse in heterogeneous federated learning. ICLR, 2023.
[47] Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Online class-incremental continual learning with adversarial shapley value. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9630-9638, 2021.
[48] Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. Ger: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 99-108, 2022.
13

---

[49] Cheng-Hao Tu, Hong- You Chen, Zheda Mai, Jike Zhong, Vardaan Pahuja, Tanya Berger- Wolf, Song Gao, Charles Stewart, Yu Su, and Wei-Lun Chao. Holistic transfer: Towards non-disruptive fine-tuning with partial target data. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[50] Cheng-Hao Tu, Zheda Mai, and Wei-Lun Chao. Visual query tuning: Towards effective usage of intermediate representations for parameter and memory efficient transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7725-7735, 2023.
[51] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[53] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018-5027, 2017.
[54] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power. In Advances Neural Information Processing Systems, pages 10506-1055 10516,,11111111 2019.
[55] Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens Van Der Maaten. Simpleshot: Re- visiting nearest-neighbor classification for few-shot learning. arXiv preprint 1rX1v:191..04623, 2019.
[56] Zhenyi Wang, Enneng Yang, Li Shen, and Heng Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. arXiv preprint arXivv2307.092/8, 2023.
[57] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong. al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7959-7977, 2022.
[58] Tong Wu, Ziwei Liu, Qinggiu Huang, Yu Wang, and Dahua Lin. Adversarial robustness under long-taile distribution. In CVPR, 2021.
[59] Yue Wu, Yinpeng Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 374-382, 2019.
[60] Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du. Parameter-efficien fine-tuning for pre-trained vision models: A survey, arXiv preprint arXiv:2402.02242,, 2024.
[61] Han-Jia Ye, Hong- Chen, De-Chuan Zhan, and Wei-Lun Chao. Identifying and compen- sating for feature deviation in imbalanced deep learning. arXiv preprint arXiv:2001.00 00 2020.
[62] Han-Jia Ye, Hexiang Hu, and De-Chuan Zhan. Learning adaptive classifiers synthesis for generalized few-shot learning. International Journal of Computer Vision, 129(6):1930-13) 2021.
[63] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv: 1212.571, 2012.
[64] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint
2019.
14

---

[65] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 39208-13217, 2020.
[66] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[67] Juntang Zhuang. Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-18806 2020.
15

---

Appendix
We provide details omitted in the main paper.
Appendix
experiment and dataset details
Appendix
additional details for calibration
Appendix C
additional analysis of fine-tuning
Appendix D
detailed results of different architectures, datasets, and splits.
Tu et al. named the above the setting Holistic Transfer (HT). For ease of reference, we use the following terms interchangeably.
Our study is built upon 49
fine-tuning classes & seen classes; absent classes & unseen classes; downstream domain & target domain; pre-training domain source domain;
fine-tuning & naive fine-tuning.
emphasize that the "unseen" classes are indeed seen in pre-training but absent fine-tuning.
A
Experiment and Dataset Details
A.1 Main Investigation (ef. section 3.1 in the main paper)
A.1.1 Dataset Details
ImageNet- Variants includes ImageNett-(endiitin
15 and ImageNet-S(ketchh [44. ImageNet-
comprises 30,000 images across 200 ImageNet classes with various renditions (e.g.. paintings, embroidery, etc.) while ImageNet-S consists of 50,000 sketch-like images for 1K ImageNet classes. Each class randomly divided into training and testing sets following an 8:2 split. 50% of the classes are randomly selected as fine-tuning classes (100 for ImageNet-R and 500 for ImageNet-S), with the remainder as absent. The downstream test set encompasses all the 200 classes for ImageNet-R and IK classes for ImageNet-S to evaluate model performance across the full class spectrum.
Office-Home 53 is a popular domain adaptation dataset, comprising 65 classes from 4 domains (Art, Clipart, Real, and Product). Following the setup 49 Art and Real are used as pre-trained domains and each pre-trained model is then transferred to each of the three remaining downstream domains individually, resulting six (pre-training, downstream) pairs. Within each downstream domain, each class is randomly split into training and testing sets following 7:3 split. 30 classes are randomly selected fine-tuning classes; the remaining 35 classes are absent classes. The downstream test set contains all the 65 classes in each downstream domain.
encompasses a diversity of image classification tasks. To enable zero-shot predictions with CLIP, Tu al. 49 only uses the tasks that provide text names for classes: Caltech101. CIFAR100, DTD, EurOSAT, Flowers102, Pets, Resisc45, SVHN, and SUN397. Notably, the SVHN dataset was excluded from our experiments due CLIP's difficulty accurately predicting numerical labels as shown in [49. Adhering to the practice in 49,, we randomly sample half of the classes as tine-tuning and the remaining as absent. The downstream training set only includes images of fine-tuning classes; the test set contains images from all classes.
VTAB [64]
Table 3 summarizes the statistics of all the datasets used in this paper.
A.1.2 Training Details
For the ImageNet-Variants Variants benchmark, we use an ImageNet- 1K pre-trained ResNet-50 (results in the
main paper) and ViT-B/32 (results in the appendix) as pre-trained models. The pre-trained model is fine-tuned on downstream tasks for epochs using the SGD optimizer with a learning rate le-3, momentum 0.9, weight decay 1e-4, and batch size 64. For the compared method proposed in [4), we set the hyper-parameters Laisill 10 and Lank 100 for ImageNet-R and Caismll and Lrank for ImageNet-S.
16

---

TABLE 3. statistic summary the datasets used paper.
Pre-training domain
Downstream
HFins-tuning classes
#Downstream training
#Downstream
Dataset
#Classes
domain
Lest
ImageNet-
ImageNet- Variants
ImageNet-
Rendition
200
100
12567
5924
IK
ImageNet- Sketch
Clipart
1,471
1,330
Product
Office- Home
Real
1,41.
1,335
Art
857
750
Real
Clipart
1.493
1,330
Product
1,459
1,361
Caltech101
CTFAR100
100
50
22,513
10,000
DTD
1,880
EuroSAT
10
8,424
5,400
VTAB
CLIP
Flawers102
102
510
6,149
3,669
Resise45
45
22
9,159
6,300
SUN397
21,750
For the VTAB benchmark, extract the class name embedding to construct the FC layer and disregard the CLIP text encoder afterward. The pre-trained model is fine-tuned downstream tasks for 20 epochs using the SGD optimizer with learning rate le-5, momentum 0.9, weight decay 0.0, and batch size 64, For the compared method proposed in [49] we follow their paper and set the syper-parameters Edistill and Crank 5.
the ViT-B/32 CLIP models the backbone. Specifically,
For the Office-Home dataset, follow the training recipe of [49]. Firstly, the pre-train pre-trained classifier obtained by fine-tuning an ImageNet- pre-trained ResNet-50 with the source domain data for 20 epochs using the SGD optimizer with learning rate 1c-3, momentum 0.9, weight decay 5e-4, and batch size 64. After that, we fine-tuned the resulting model on each target domain for 20 epochs using the SGD optimizer with learning rate le-4, momentum 0.9, weight decay 5e-4, and batch size 64. For the compared method proposed 449 we follow their paper and set the hyper-parameters Edistill 10 and Lrank 100.
Computational resources. We use a combination of NVIDIA RTX A6000 and NVIDIA 2080Ti GPUs. Since we worked on fine-tuning the computation is quite manageable.
A.2 Ablation Study (ef. section in the main paper)
In section 5 of the main paper, we investigate how 1) a biased sampling of fine-tuning and absend classes and 2) smaller size of fine- mie-tuning classes would impact the performance of fine-tuning. Here, we provide details of how we split the data.
In the Office-Home dataset, we strategically select fine-tuning classes that are similar in the pre- trained model's feature space, aiming to create a meaningful distinction between fine-tuning and absent classes. To identify classes that closely related, apply the metric of total intra-group distance: lower value indicates higher similarity among classes within a group. This ensures finding fine-tuning classes that exhibit tight clustering in the feature space. We employ greedy strategy to grow the fine-tuning class set towards a pre-defined size. Figure 13 ad |Figure 14 present t-SNE visualizations that illustrate the distribution of chosen fine-tuning and absent classes for varying fine-tuning class sizes across two distinct pre-trained domains.
For the IetgeNet-S dataset, we leverage the WordNet [I] hierarchy to select coherent groups as fine-tuning classes. Specifically, we explore two different hierarchical splits. The details are shown in Table 4
17

---

Computer
Careda falclater
red
Red
Iele hone
Poet aetee
XART Cook
Lamp Shude Tuo
Lamp
Ruler
Part, Fn
Hammeeat
Sussers
Petmet
Mop
Tork
Spapn
Puy"
Nottle
Perc: Enner
Bittl
Howeo
Soda
tatteries
Speawors Piptipe
Speater
(A) Office- Home pre-trrined domain: Art
Fine-tuning Class Size: 5
Fine-tuning Class Size: 15
Fine-tuning Class Size: 25
15
-10
(B) Splits for different fine-tuning class sizes (red for fine-tuning classes and blue for absent classes)
FIGURE 13. (A) shows the t-SNE of the class mean features for 65 classes in the Art domain with their corresponding class names. (B) shows the fine-tuning and absent classes split for different fine-tuning class sizes.
Candls
S:
Mug Mit
rest
Mouse
Meea otor
Kiytwardd
Puli AY
Voosh Oiicow
Wiccumm
Speeker Rasin"
Testt
Duer
Marie
Tan
Drassr
Prn
File Cast
Buckpack
Stk
Denk Lamre
Shall
Lan Shuds
Pofl::
Folder
Calerddr
Sui Coarh
(A) Office-Home pre-trained domain: Real- World
Fine-tuning Class Size: 5
Fine-tuning Class Size: 15
Fine-tuning Class Size: 25
20
-10
(B) Splits for different fine-tuning class sizes (red for fine-tuning classes and blue for absent classes)
FIGURE 14. (A) shows the t-SNE of the class mean features for 65 classes in the Real-World domain with their corresponding class names. (B) shows fine-tuning and absent classes split for different fine-tuning class sizes.
18

---

TABLE 4. Two hierarchical splits for ImageNet-S.
Hierarchical Split One
Split Group Name
Dog
Mammal
Animal
Split Group Size (Classes)
118
218
398
Hierarchical Split Two
Split Group Name
Device
Instrumentality
Artifact
Split Group Size (Classes)
124
350
522
7-
6-
Feature 2
Pre-trained Feature
5
Fine-tuning Feature
4
3-
8.5
9.0
9.5
10.0
10.5
11.0
11.5
Feature 1
FIGURE 15. Visualization of pre-training and target domain data in the toy example, with distinct colors indicating different classes. The symbol O represents data from the pre-training phase, while O denotes those of the target domain (only blue and cyan are fine-tuning during fine-tuning).
A.3 Toy Example (ef. section in the main paper)
To elucidate the impact of similar fine-tuning training data on the feature representation of absent data, in section 6 and figure 12 of the main paper, we construct toy example featuring four classes of 2-dimensional data, each represented by distinct colors (blue, cyan, red, magenta). Figure 15 shows the same data. In this example, pre-training data (D) are generated from Gaussian distributions with a standard deviation of 0.2 and four different means: (10, 2), (10, 3), (10, 8), and (10, 7). To simulate domain shift, the target domain data (C) undergoes a horizontal shift, with the cyan and magenta classes moving to the right and the blue and red classes to the left by an identical distance. This setup, intentionally restricting data non-negative values, mirrors the effect of ReLU activation.
We employ a two-layer multi-layer perceptron (MLP) with a configuration of 2-2-4 (input dimension: 2, hidden layer dimension: 2, and output dimension: 4). The MLP is initially pre-trained on the four-class pre-training dataset, with the first layer weights set as an identity matrix to simplify visualization. Subsequent fine-tuning
target domain data incorporates only classes (blue and cyan), without the constraint freezing the first layer. Both the pre-training and fine-tuning phases utilize the SGD optimizer, applying learning rate of 0.01 for 100 epochs with cross-entropy loss as the objective.
B Additional Details for Calibration (cf. section 4 in the main paper)
We provide details for the calibration methods in section 4 of the main paper.
B.1 Average Logit Gap (ALG)
In a well-calibrated pre-trained model, the scale of average non-Ground-Trine Trruhd (non-GT) logits expected to be similar across different classes. Figure 16 demonstrates that using the pre-trained
19

---

model, the non-GT logits exhibit comparable magnitudes for both fine-tuning and absent class groups. However, the fine-tuning class group's 's non-GT logits become notably higher after fine-tuning, since fine-tuning tends to assign much higher logits see classes due to the absence of absent data. This disparity inspires an estimation of y based on the non-GT logit differences between fine-tuning and absent classes, as delineated in Eqquatin 7 Concretely, for each training example, we calculate 1) the average of non-GT fine-tuning class logits and 2) the average of non-GT absent class logits. We then average the difference between them over all the training examples calculation is also visually summarized in Figure 17] Since this approach is based on the average non-GT logit difference between fine-tuning and absent classes, call this method the Average Logit Gap (ALG).
estimate of 7, The
1
Σ Drr
Σ SE
w fo(())
1 U1
Σ
w fo(Ø,)
YALG
Darl
CElleey
AVG
AVG
cat
-1
dog
tiger
horse
-2
Free fre.ininnclaoe Aloant classo
zebra
-3
Pre-trained Model
fine-tuning Model
(pre-training)
(absent)
Office-Home absent test data's average non-GT logits within the fine-tuning and absent groups. In the pre-trained model, they are similar but the fine-tuning class non-GTT logits become notably higher after fine-tuning.
FIGURE 16.
FIGURE 17. Average Logit Gap (ALG) calibration
method: the calibration factor ry is calculated based on the difference non-GT logits between fine-tuning and absent groups.
B.2 Pseudo Cross-Validatie Validation (PCV)
To address the challenge of selecting y without access to target validation data, which ideally encompasses both fine-tuning and absent classes, we introduce a novel Pseudo Pross-Validdtion (PCV) method, demonstrated in Figure 18 Specifically, we partition the target training dataset Dr into pseudo training data D Psscudo-rr and pseudo validation data psscudo-val- Dpecudo-tr further divided into two subsets with disjoint label spaces, simulating
souud--fnnetuuinn data D Dsscud-ft and pseudo-absent data D preudo absent We then naively fine-tune the pre-trained model on Dpsemto absent and evaluate Acys/y and Accu/y using D paeudo-val- We select 7 by balancing these two accuracies. To enhance the robustness of 7 estimation, we employ bootstrapping, repeating the pseudo sepuud-spliitngg and fine-tuning process three times with varying partitions. The selected is applied to the fine-tuning model (OT, Wr). J. which fine-tuned from the pre-trained model (0o. Wo} on the entire target training data Drr.
C Additional Analysis of Fine-tuning (cf. section 3.4 and section 6 in the main paper)
In this section, we provide more analysis including an analysis of feature improvements, an investi- gation of the tendency for logits to be biased toward fine-tuning classes, and the effects of freezing the classifier and backbone during fine-tuning. Additionally, we quantitatively examine the class relationship change between the pre-trained and fine-tuning models. Finally, we extend our analysis to examine the impact of fine-tuning on the iWildCam dataset [
C.1 Feature Improvement by the Fine-Tuur Tuned Model
20

---

Spli
D puntuitftt-l
Eva
parudovvlll
P2
D tre
Splt
parsie-nhome-
(gtands Wi
Fireduni
poeudoo ft
(Ro, Wo]
Eval
D poeudo
Tay
D poeudo
porado
(grrth Wesning
D pacudo-faa
Evni >
pseudovvala
Fn
poccalosttsn
Spll
poeutio oboens-
Eglesd Wonddy
FIGURE 18. The illustration of the Pseudo Cross- Validation (PCV) method for estimating the calibration factor 7.
ImageNittRR
Fine-luniig
75
Abseet
50
25
-25
-50
-75
FIGURE 19. 1-SNE (ImageNet-R) The extractor for does not create an artificial margin between fine-tuning and absent classes.
21

---

Officetome-VVrrintss Classifer L2
ImageNet-Variants Classifer Norm
0.975
156
Feetra)
155
0.925
154
Magnitucee
0.900
153
0.975
152
1:11
0.825
0.800
149
0.775
148
12
10
20
30
Training epochs
Training epachs
(A) Office-Home
(B) ImageNet- Variants
FIGURE 20. L2 norm of classifier weights for fine-tuning and absent classes during fine-tuning
This illustrates significant increase in weight magnitude for fine-tuning classes compared to absent, potentially resulting in fine-tuning's s biased logits towards fine-tuning classes.
TABLE 6. The performance comparison between the neural network (NN) classifier with the FC layer and the cosine classifier.
Benchmarks
Classiter
AUSUC
Accy/y
Acesyy
Accu/y
NN
0.44
43.35
81.29
3.53
ImageNet Variants
Office Home
NN
0.63
53.25
87.84
22.37
Cosine
0.63
57.26
87.97
29.83
NCM Accy/ul
PT
FT
A
To further understand the improved NCM absent class accuracy NCM Accu/y(for), we plot the t-SNE embedding [51] in Fig- ure 19 We find that the FT feature extractor for does not super- ficially raise Acty/y by creating an artificial "margin" between
IN-(R, S) |40.3% 61.9% 1..6%
VTAB
883.7% 87.2%
3.5%
Office- |80.1% 81.5% 1.4%
the fine-tuning and absent class features F and destroying their semantic relationship. We further report the NCM accuracy of classifying absent class data into one of the absent classes, i.e., Accu/u with B u. As summarized in Table 5] FT indeed improves the feature quality to distinguish among absent classes.
TABLE 5. NCM Accu/u using the pre-trained (PT) and FT features. FT improves the fea- ture quality for absent classes.
C.2 Biased Logits Towards fine-tuning Classes
section of the main paper highlights how fine-tuning generates logits biased towards fine-tuning classes due to the lack of absent data during training. delve deeper into this, we analyze the L2 norm of the classifier weights WT for Office-Home and ImageNet- Variants. FFiure 20 reveals pronounced increase the weight magnitude for fine-tuning classes relative absent classes during fine-tuning, which could potentially result in fine-tuning's biased logits towards fine-tuning classes.
To determine whether the magnitude of classification weights is solely responsible for the biased logits, we apply the cosine classifier [13). which normalizes the weights to ensure uniform magnitude across all classes. According to the results shown in Table 6. employing a cosine classifier does yield slight improvement in absent class accuracy (Accu/y); however, the outcome remains suboptimal. This indicates that both the magnitude of the linear classifiers and their angles (i.e., cosine similarity) with the features contribute to the biased logits. These findings lay the groundwork for our proposed calibration approach, which seeks to directly adjust the logits, bypassing the limitations associated with modifying either the magnitude or the angle of the classification weights.
'Doing so would shrink the label space from y 44. We note that Accu/u 2 Accu/s! the former eliminates the emorr of classifying absent class samples into fine-tuning classes.
22

---

TABLE 7. Linear classifier CKA analysis results.
Dataset
Fine-tuning Classes CKA
Absent Classes CKA
ImageNet- Varaints
0.969
0..999
Office-Home
0.979
0.998
C.3 Should We Freeze the Linear Classifier or Feature Backbone?
Adhering to the practice in source-free DA [29]. Tu et al. [49] advocate for freezing the linear classifier Wo during fine-tuning (termed the frozen classifier approach) to preserve the absent class relationship and hence accuracy Accu/y. Surprisingly, our analysis reveals that while preserving class relationships within the FC layer, the frozen classifier leads to a worse AUSUC score and NCM absent accuracy compared to fine-tuning evidenced in Table 8 This suggests a deterioration in the
feature quality of the frozen classifier.
We hypothesize that the underlying issue arises from fine-tuning with only fine-tuning class data, which inadvertently biases the model towards classifying all samples as fine-tuning classes. While fine-tuning can adapt to this bias by adjusting the FC layer reduce absent logits, the frozen classifier must alter its features to achieve such goal, potentially compromising the quality of absent features.
We also investigate freezing the feature extractor backbone Oo and only fine-tuning the FC layer (a.k.a. linear probing). Since the feature representations remain unchanged, the NCM /ccy/y mirrors that of the pre-trare vre-trained model, indicating no improvement in feature quality, as evidenced in [Table g As result, it achieves lower NCM accuracy and lower AUSUC score than fine-tuning. In sum, neither the frozen classifier nor the frozen backbone are effective strategies in HT.
C.4 bbent Class Relationship Analysis
Inspired by the finding in [19 the similarity among vectors in the FC layer reflect the semantic relationships between their corresponding classes-we we investigate such class relationships are maintained during fine-tuning. To quantitatively capture the class relationships, cosine similarity between each pair of vectors in the FC layer. Specifically, the matrices ( and (WW)) capture the class relationships among fine-tuning and absent classes, respectively, within the pre-trained model. Similarly, S (WS)* and ( reflect these relationships in fine-tuning. Here, W represents the 122normalized linear classification weights, with superscripts indicating whether the weights are for fine-tuning or absent classes, and subscripts differentiating between the pre- pre-trained and the fine-tuning model.
compute the
To assess how fine-tuning affects class relationships, we employ the linear Centered Kernel Alignment (CKA) (25. to compare the class relationships before and after fine-tuning. Specifically, we compute CKA scores for fine-tuning and absent classes:
Fine-tuning Classes CKA = ( (W
Absent Classes CKA CCA(WW((WS))) W TW
Higher CKA scores signify from a pre-trained model to fine-tuning. As demonstrated Table 7 class relationships among absent classes are substantially more preserved than those among fine-tuning classes. This observation aligns the main paper. The distinction arises because fine-tur fine-tuning
robust preservation of class relationships through the transition
with insights discussed in section with fine-tuning class data prompts the classifier to differentiate between fine-tuning classes more distinctly. In contrast, without direct training signals for absent classes, updates to absent classifiers tend be more uniform, thus maintaining the original class relationships among absent classes throughout the fine-tuning process.
D
More Results
D.1 ImageNet-
ImageNet-Variants Variants with VIT
In the main paper, our findings for ImageNet- Variants were based on experiments with ResNet- 50 model pre-trained on ImageNet-IK. To broaden our analysis, this section reports on extended
23

---

TmageNett Varnints
Office-Home NCM Actuyyy
Acey/y
Pre: trained
0.083
32.5
0.499
73.7
Fine Tuning
0.439
50.5
0.632
75.8
Classifier
36.3
Linear Probing
0.218
32.5
0.595
73.7
TABLE 8. AUSUC and NCM Accu/y demonstrate that fine ine-tuning outperforms frozen classifier and linear probing.
TABLE 9, Performance comparison between ResNet-50 and ViT B/32 in ImageNet-RR R with 50% finc-tuning classes (100 classes).
Mithod
AUSUC
Actyyy
Accsry
Actury
Ac
NCM Accs/y
Acty/yy
Pre-trained
0.09
23.79
23.42
24.19
24.61
37.07
32.68
ResNer-50
line tuning
0.66
43.70
81.97
1.70
37.92
73.00
51.63
A
0.37
19.11
58.55
-22.49
13.31
35.93
18.95
Pre-trained
0.00
24.63
24.87
24.36
24.93
41.03
37,89
VII-B/32
fine-tuning
0.49
48.75
82.10
12.15
43.24
77.68
51.42
A
0.40
24.12
57.23
-2.21
18.31
36.65
13.53
experiments that employ the ImageNet-I pre-trained Vision Transformer (ViT-B/32) as an alternative pre-trained model. Our objective ascertain whether the observations regarding fine-tuning remain consistent when applied to a larger and more advanced model architecture.
Results, detailed in Table 9 and Table IO, indicate that both the ResNet-50 and ViT-B/32 architec- tures yield consistent improvements across AUSUC, Accu/u- and NCM metrics. This consistency underscores the inherent robustness of fine-tuning's benign behaviors, independent of the model architecture. Notably, the ViT-B/32 model exhibits a significantly milder decline in absent class accuracy Actu/y compared to ResNet-50. This finding suggests that the ViT-B/32 model is more robust in partial target data fine-tuning.
D.2 Detailed Results for Each Dataset
In the main paper, due to space constraints, we summarized the findings by presenting average performance metrics across all datasets within each benchmark. This section expands upon that summary by providing in-depth results for each pretraining-dwwnsee Home dataset and for individual datasets in VTAB, detailed in Table 1I and Table 12, respectively. Across the board, fine-tuning exhibits consistent enhancements in AUSUC scores, reinforcing its efficacy in diverse settings. In ImageNet- Variants, fine-tuning consistently improves the absent class feature (an increase of NCM absent accuracy Accu/y)) as mentioned in ssbbeectio D. the context of Office Office-Home, while the majority of domain pairs witnessed improvements NCM Accu />- exceptions such as Rw-Pr and Rr-Rw experienced a decline. Some exceptions are also observed within VTAB, such Flower1 02 and SUN397. These findings suggest that certain datasets may benefit from more sophisticated approaches to bolster absent class features.
domain pair within the Office-
D.3
The Fine-tuning / Absent Accuracy Curve For All Datasets
Owing to the space constraints within the main document, our presentation Accuracy Curve was limited to ImageNet-R and the Art-Product domain pair from Office-Home. This section expands our analysis by presenting the Fine-tuning/ Absent Accuracy Curves for all datasets within each benchmark. This comprchensive display aims to offer more detailed understanding of the performance dynamics between fine-tuning and absent classes across the diverse range datasets evaluated in our study.
Fine-tuning/ Absent
TABLE 10. Performance comparison between ResNet-50 50 ViT-B/32 in ImageNet-S with 50% fine-tuning classes (500 classes).
Architecture
Method
AUSUC
Accy
Accs/y
Accy
Acdgr/24
NCM ACMs/s
NCM A0t
Pre-trained
29.58
50
line tuning
0.11
42.97
80.60
5.34
55.22
64,50
49.40
25.64
33.1
17.02
Pre trained
0.06
21.33
20.78
88
27.32
30.98
30.46
VIT-B/32
fine-tuning
0.39
46.34
73.38
19.30
56.92
64.30
49.24
A
29.60

---

TABLE 11. Detailed results six pretraining- downstream domain pairs in Office-Homee benchmark.
Ofhee-Home
Method
AUSUC
Aocy/y
A2
A2
NCM A00g/2
Accy/
Pre-trained
0.30
47.07
50.29
57.43
64.04
Ar + CI
line tuning
0.45
44.06
81,58
8.63
59.94
72.45
66.67
A
0.16
37.93
.41.67
2.63
Pre-trained
0.53
67.52
61.32
71.88
76.5
80.39
80.25
Ar -> Pr
line tuning
0.99
51.87
93.05
23.00
76.88
87.17
83.13
A
-48.8
6.77
Pre-trained
0.62
72.73
70.07
74.54
80.87
77.88
77.78
Ar + Rw
line-tuning
0.70
59.33
89.74
32.63
81.15
82.21
77 77.50
A
0.09
13.41
19.07
41.91
0.228
4.35
-0.28
Pre-trained
0.51
65.60
63.06
68.19
73.32
69.39
72.24
+
lunang
29.11
72.30
74.12
0.09
-10.13
18.21
-39.08
2.96
2.50
1,89
Pre-trained
0.33
51.13
53.22
49.12
58.70
64.88
60.77
fine-tuning
0.53
46.47
87,88
6.64
64.16
78.83
64.60
A
0.20
-4.66
34.66
-42.48
5.46
13.96
3.83
Pre-trained
0.20
78.00
79.19
83.38
84.19
87.01
Rw
line luning
0.77
62.31
93.49
34.22
83.94
K8.066
86.45
A
0.07
-16.61
14.88
-44.77
0.56
3.88
-0.50
TABLE 12. Detailed results eight datasets
VTAB benchmark.
Dataset
Method
AUSUC
Accy/>/
A
Acteyy
Acc /14
NCM Acc
NCM Accg
Pre-truined
0.70
70.43
74.85
Caltech101
line tuning
0.82
82.27
97.19
69.10
85.22
90.18
88.31
T0.36
Pre Irained
0.51
64.17
65.94
62.40
72.16
65.68
66.50
CIFAR100
hine tuming
0.68
63.90
89.20
42.60
78.82
82.00
74.08
0.17
1.73
23.26
-19.80
6.66
16.32
7.58
Pre-trained
0.26
43.19
39.13
47.08
53.85
39.67
65.00
DTD
line-tuning
0.35
45.32
75.98
15.94
50.73
65.22
65.52
0.10
2.13
36.85
-31.15
-3.13
5.54
0.52
Pre-trained
0.14
32.22
43.10
20.34
32.86
76.02
77.88
EuroSAT
turing
47.46
0.33
20.00
55.84
-19.14
14.61
20.75
9.65
Pre-trained
0.28
46.71
44.32
49.08
55.50
69.12
72.00
SUN397
fine-tuning
0.36
35.69
57.33
71.07
70.75
A
0.08
3.09
19.72
-13.40
1.83
1.95
-1.24
Pro-trained
0.37
33.95
55.24
52.68
58.83
81.25
81.67
Resiso45
line-tuning
0.60
58.48
93.74
23.56
66.13
89.14
31.35
0.23
4.32
38.50
-29.02
7.35
7.89
-0.35
74.11
Flowers 102
hnc-tuning
0.55
64.16
78.29
49.21
71.40
77.82
88.53
0.05
0.44
18.96
-19.14
-2.71
-10.32
-3.61
Pre-trained
0.80
83.97
79.03
88.60
94.41
79.93
77.84 77.84
Pets
line tuning
0.88
84.44
93.80
75.67
93.93
85.34
88.34
Absent Class Data Accuracy
ImageNet -> ImageNet-R (50% Fine-tuning)
Absent Class Data Accuracy
ImageNet -> ImageNet-S (50% Fine-tuning)
60
50
50
40
30
30
20
20
10
20
40
60
80
20
40
60
80
Fine-tuning Class Data Accuracy
Fine-tuning Class Data Accuracy
Pre-training
Pre-training y Fine: funeitunin
Pre-training
Pre-training
Fine-tuning
Fine-tuning Tu al. 2023
Fine-tuning
Tu at al. 2023
Tu al. 2023
Tu et al. 2023
ImageNet-R
ImageNet-S
FIGURE 21. Accsyy at the x-axis and Accu/sy at the y-axis) by varying the calibration factor y for ImageNet- Variants.
25

---

Absent Class Dala Accuracy
OfficeHome Ar ~ CI (-50% Fine-tuning)
Absent Class Data Accuracy
OfficeHome Ar Pr (~50% Fine-tuning)
80
70
60
40
30
40
20
10
0
40
60
80
Fine-tuning Class Data Accuracy
Fine-tuning Class Data Accuracy
Pre-training
Pre-training
Pre-training
Pro-training Fine luning
Fine luri-luinng
Fine luning
Fine-luning
Tu et al, 2023
Tu al. 2023 +
Tu al. 2023
Tu et al. 2023
(A) Art-Clipart
(B) Art-Product
Absent Class Data Accuracy
OfficeHome Ar -> Rw (-50% Fine-tuning)
Absent Class Data Accuracy
OfficeHome Rw ~ Ar (-50% Fine-tuning)
80
80
70
60
50
50
40
40
30
20
10
10
40
80
20
40
60
80
Fine-tuning Class Data Acouracy
Fine-tuning Class Data Accuracy
Pre-training
Pre-training y Fine-luning Tu al. 2023
Pre-tmaining
Pre-training + Fine-tuning
Fine-lunke
Fine-tuning
Tu et al. 2023
Tu 2023
2023
(C) Art-Real World OfficeHome Rw - CI (-50% Fine-tuning)
(D) Real World-Art
Absent Class Data Accuracy
Absent Class Data Accuracy
90
OfficeHome Rw -> Pr (~50% Fine-tuning)
60
80
70
50
60
50
40
30
10
60 Fine-tuning Class Data Accuracy
20
60
Fine-tuning Class Data Accuracy
Pre-training
Pre-training Y Fine-tuning Tu at al. 2023 +
Pre-training
Pre-training
Fine-tuning Tu at al. 2023
Fine-tuning
Fine-tuning + Tu at al. 2023
Tu al. 2023
(E) Real World- World-Clipart
(F) Real Prolu-rroduct
FIGURE 22. Aces/y at the x-axis and Acquyg at the y-axis) by varying the calibration factor y for all pretraining-dowwsteeed domain pairs in Office-Home..
26

---

Absent Class Data Accuracy
80
CLIP VTAB-Caltech100 (50% Fine-tuning)
Absent Class Data Accuracy
CLIP >> VTlB-Cifar 100 (50% Fine-tuning)
80
80
70
70
60
60
50
50
40
40
30
20
20
10
10
40
60
80
100
20
40
Fine-tuning Class Data Accuracy
Fine-tuning Class Data Accuracy
Pre-training
Pre-Irannngg V Fine-tuning Tu et al. 2023
Pre-training
Pre-triiing Y
Fine-tuning
Fine-tuning
Fine-tuning
Tu al. 2023
Tu et 2023
Tu al. 2023
(A) Caltech 101 CLIP VTAB-DTD (50% Fine-tuning)
(B) CIFAR 100 CLIP NTAB-Eurosal (50% Fine-tuning)
Absent Class Data Accuracy
Absent Class Data Accuracy
50
50
30
20
10
20
60
80
20
40
100
Fine-tuning Class Data Accuracy
Fine-tuning Class Data Accuracy
Pre-traning
Pro-training Y
Pro-training Fine tuning
Pre-traiingg +
Fine-tuning
Fine auning +
Fine-tuning
al 2023
Tu at. 2023 Y
Tu et 2023
al. 2023
(C) DTD
(D) EuroSAT
Absent Class Data Accuracy
CLIP >> VTAB-Flowers 102 (50% Fine-tuning)
Absent Class Data Accuracy
CLIP > VTAB-Pats (50% Fine-tuning)
70
90
80
60
70
50
60
50
30
30
20
10
10
20
40
60
80
20
40
60
80
Fine-tuning Class Data Accuracy
Fine-tuning Class Data Accuracy
Pre-training
Pre-traning y Finc-tuning + Tu at 2023
Pre-training
Pre-training +
Fine-tuning et al. 2023
Fine-tuning
Fine-tuning +V
Tu et. 2023
Tu et al. 2023
(E) Flowers 102
(F) Pets
Absent Class Data Accuracy
CLIP >> RTsi-ces (50% Fine-tuning)
Absent Class Data Accuracy
CLIP VTAB-Sun397 (50% Fine-tuning)
60
60
50
50
40
40
30
20
10
10
20
80
60
Fine-tuning Class Data Accuracy
Fine-tuning Class Data Accuracy
Pre-training
Pre-training y Fine-tuning Tu at 2023
Pre-training
Pre-raining Y Fine-tuning
Fine-tuning et al, 2023
Fine-tuning
Tu at 2023
Tu al. 2023 +
(G) Resisc45
(H) Sun397
FIGURE 23. Accsyy at x-axis and Accu/y at the y-axis) by varying the calibration factor y datasets
VTAB.
27

---

ImageNet-R
ImageNet-R Acc,
Restiee 3U
Restie 5g
0.28
0:29
0:32
0:39
9.63
13.39
15.70
17.77
18.95
10.0
VIT-8922
VIT-8922
0.25
0.30
0.35
0:39
0.33
-0.2
3.23
8.86
12.41
15.
13.
-10.0
40 Num Fine- fuming Class
80
Num Fine- meeiiuunigg Class
FIGURE 24. ImageNet-R, RR the performance gain AUSUC and NCM Accu/yy pre- pre-trained model fine-tuning) under different data splits and fine-tuning class sizes using ImageNet- pre-trained models ResNet50 and VIT- B/32.
(AUSUC
21.0
Office (AUSUC
02
Office-Home
3.0
Mecadr Nroo
0.33
4.1
Hendyy
0.12
0.16
0.18
42
11:00
13:02
16.03
-1.00
0.07
0.11
0:16
9.1
-2.10
-0.0
0.85
-15
Nam of Piraeturing Class
Num of Finlunin Class
Nam of iineeannnggg Class
Nam Firwwluring Caoss
Animal hierarchical split for ImageNet-S: the performance gain on AUSUC and NCM Acoo:yy(frrm pre-trained model to finc-tuming) under different data splits and fine-tuning class sizes using ImageNet-IP IK pre-trained models ResNet50 and ViT-B/32. This split contains dog (118 classes), mammal (218 classes), and animal (398 classes) the fine-tuning classes.
FIGURE 25.
D.4 More Split Results
In the main manuseript, due to limitations on space, we restricted our presentation of data splits and fine-tuning class size-related ablation studies to ImageNet-S and Office-Home. This section extends our analysis to encompass ImageNet-R, as well as additional hierarchical splits for ImageNet-S, utilizing distinct ImageNet-IK pre-trained models: ResNet50 and ViT-B/32.
Figure delineates the outcomes ImageNet-R where different fine-tuning class sizes are randomly delve into hierarchical splits based WordNet within ImageNet-S.
chosen. Furthermore,
Figure 25 reveals the findings from the animal hierarchical split, which encompasses dogs (118 classes), mammals (218 classes), and broader animal categories (398 classes) as fine-tuning classes. Similarly, Figure 26) presents the results from the device hierarchical split, involving devices (124 classes), instrumentality (350 classes), and artifacts (522 classes) fine-tuning classes. Across
different datasets, splits, and model backbones, the enhancements attributed to fine-tuning manifest robust and consistent.
D.5 Classifier Update Direction Similarity
Due to page limitations in Section 6 of the main paper, our discussion on the similarity of classifier update directions was confined to ImageNet-S. To further substantiate the universality of our findings, this section extends the analysis to the effice-Home dataset, featuring the Art-Real World domain
ReiNeet AUSUC
05
ReaNet--D -D NCM Acc
20.0
VIT-B:32 (AUSUC
05
VIT-B122
20.0
Kandon
Randon
Random
60
Hesrerr
Heentry
0.2
15.23
10.0
0.24
0.24
022
16.63
10.0
350
602
124
330
Num oinietuiing Ciasa
Num Fine-luting Class
Num Fire-turing Ciasa
Num of Fine mee-uuinng Class
FIGURE 26. Device hierarchical split ImageNet-S: the performance gain on AUSUC NCM Accu/y (from pre-trained model to fine-tuning) under different data splits and fine-tuning class sizes using ImageNet-I1 pre-trained models ResNet50 and ViT-B/32. This split contains device (124 classes), instrumentality (350 classes), and artifact (522 classes) the fine-tuning classes.
(from
28

---

Rinetuniing Classifler Weight Change Relationship
Absent
Weight Change
1.0
1,0
0.8
0.8
12
0.5
18
22
25
28
12 14
16
10
FIGURE 27. Classifier update direction similarity within fine-tuning (left) and absent (right) classes for Office-Home. The update directions are highly similar within absent classes, thus preserving the inter-class relationships among absent classes.
TABLE 13. A noticeable performance gap exists between our proposed calibration methods, ALG and PCV, and the theoretical upper limit represented by fine-tuning- +y", which leverages target test data for y selection. This discrepancy highlights valuahle opportunity for further advancements in calibration techniques bridge this gap.
ImageNet- (R.
VTAB
Office-Home
Metries (%)
Accy/y Arts/y / Assu/y Arry
/Aes/y) Acst |
Aces Actpr/>?
Pre-trained fine tuning
23.5
43.3
81,3
3.5
62.8
86.4
39.1
53.5
88.3
22.4
line-tuning TALG
55.9
80.3
30.5
66.8
85.3
48.2
65.0
87.7
44.9
fine tuning +7rev
57.1
60.1
54.0
57.4
47.1
67.8
72.2
82.3
63.1
fine-runing 60.8
73.6
47.6
69.3
75.6
62.8
72.7
79.1
66.9
Oracle
71.1
This pattern of update direction similarity is similarly evident across other wetraining-doowssreed pairings within the Office-Honne dataset.
pair, detailed in Figure 27
D.6 Using Target Test Set For y Selection
Theoretically, utilizing the target test set for selection contradicts standard practices, it introduces data leakage that can considered as cheating. However, for comparison purposes, we leverage the upper bound for calibration methods. As indicated in Table [3] while our introduced calibration methods, ALG and PCV, demonstrate substantial improvements
target test set to select y, thereby denoting fine-tuning +
fine-tuning, they do not fully reach the performance of fine-tur fine-tuning + y². This gap underscores a direction for future research to focus on more sophisticated calibration strategies for fine-tuning.
D.7 iWildCam
Another realistic benchmark proposed in [49] iWildCam, which considers abundant camera trap images from various geographical locations pre-trained domains, with images from camera trap location serving as the target domain. The uniqueness of this benchmark lies the use of data collected within limited time frame (e.g.. the first month) at the new location the target training set. Given the unlikely presence of all animal species within this initial period, the target training data inherently exhibits class bias towards the species that are detected.
Contrary to other benchmarks, as depicted in Table 14 fine-tuning shows a decline in both AUSUC and Accuju- This performance drop is attributed to the natural data collection bias caused by time in iWildCam. Unlike conventional benchmarks that assume training and test sets come from the same distribution, iWildCam's 's unique structure-whee training and testing sets originate from different
29

---

TABLE 14. Performance of fine-tuning on iWildCam benchmark.
Method
AUSUC
Accyyy
Accg/y
Acczy
Ac
NCM Accs/y
NCM Acczr/2
0.156
line Lueettnnigg
0.13
36,5
53.52
0.595
27.256
44.075
31,778
-0026
0.71
17.173
-25.293
-6.623
0.11
-0.208
time periods introduces natural discrepancy in data distribution due to seasonal changes animal migration.
To investigate this further, we selected four target locations from the benchmark and identified the pre-trained and test data corresponding to the same target training classes for each location. By extracting features using pre-trained model, we aimed to determine whether significant changes occurred within the same classes across pre-trained data, target training data, and target testing data. The findings, illustrated in Figure 28] reveal notable differences in the data distributions among the same classes across the pre-trained domain, target training, and target testing sets. This discrepancy introduces further complexity in HT. Addressing this challenge, future research may focus on enhancing fine-tuning with strategies capable of bridging the data distribution gap.
Target Location 289
Target Location
80
Pre-training
60
Fine-tuning Fine-tuning
20
-20
-40
-40
Pre-training Fine-tuning train Fine tuning test
-80
-60
-60
40
20
20
40
60
80
-60
-40
-20
20
40
60
Target Location 287
Target Locati on 310
100
Pre-training
75
Fine-tuning train
Fne-tuning best
50
25
25
-50
-50
fre-training, rine-turing train
75
-75
Fine buning test
-100
-50
-50
-0
-20
60
-100
-75
25
100
FIGURE 28. iWildCam t-SNE visualization of the same group of classes (target training classes) in mreitrained domain data, target training data, and target testing data, demonstrating significant distribution differences. This variation presents an additional challenge in HT. The finding is consistent across 4 different target locations.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper's main contribution is systematic study of fine-tuning, Both the abstract and the introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide multiple remarks to describe the scope of the study, We also conduct further analysis (e.g.. section 6) to analyze the limitation.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer: [NA]
Justification:
Not theoretical paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provided detailed experimental setups in the main paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
32

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The data used in the paper is publicly accessible. Our code will be released in the near future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
oliicy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle pwwppppgpuuiiieeeee /etic///uuide//oeeuuuiiiiii
(CodeSubiisss wwtplubmmsssionii ssioiiolccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provided detailed experimental setups in the main paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification:
No error bar is provided. We study fine-tuning on already pre-trained models.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
33

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we discussed this in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httt:::://eeuris..coom http///rrrp..c.ooo./////cccctaaa
Question:
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper studied fundamental machine-learning problem of fine-tuning, If successful, will make fine-tuning much easier and more practical. To our knowledge, at the current stage, our paper does make any additional negative societal impacts compared to existing papers on fine-tuning.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
34

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The data we used are all publicly available and we cited the original papers.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
35

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "PtXXKOQjP3", "reviewer": "Reviewer_amJX", "summary": "The paper unveils the improved features of absent classes when a pre-trained model is fine-tuned on a subset of all classes. The paper presents an empirical study on three datasets to demonstrate this finding and proposes a calibration method to post-process the logits after fine-tuning to improve the classification result in absent classes. The reason why the absent classes are improved is analyzed and the effectiveness of the calibration method is supported by experimental results.", "questions": "NA", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The finding on the improved performance of absent classes is interesting. \n\n\n2. The proposed calibration method is simple and easy to use.\n\n\n3. The presentation of the paper is clear.\n\n4. The empirical results in the appendix are extensive.", "weaknesses": "1. The reason why the absent classes' features are improved is demonstrated to be that the fine-tuned classes have similar features as absent classes. That suggests that the improvement does not always hold when the fine-tuned classes have features that are not helpful or even harmful (e.g., spurious correlation) to absent classes. This should be discussed further in the submission.\n\n2. In some figures it looks like Tu et. al. achieves the best trade-off, while the proposed scaling method is only presented with a line. It is probably better to show the performance of the two $\\gamma$ selection methods in the figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment and valuable feedback on our work. \n\n**Q1: … the improvement does not always hold when the fine-tuned classes have features that are not helpful or even harmful ...**\n\nThank you for your insightful question, and we will include more discussions in our camera-ready version.  \n\nIn our study, while not explicitly mentioned, we suppose the pre-trained model has learned a decent discriminative ability and faithful similarity among classes (in the pre-training domain), and we aim to preserve and even improve these properties (in the downstream domain) after fine-tuning with a subset of classes. \n\nOur analysis in Section 6 (Lines 326 - 360) explains why the absent class features could improve after fine-tuning. *We note that we certainly did not claim that the improvement would always happen under any conditions.* For instance, in Lines 336 - 339, we pointed out one necessary condition. Essentially, if the domain shift affects similar classes *differently* — for example, huge domain shifts break the similarity learned in the pre-trained model — then the improvement would unlikely hold. We will extend this discussion to incorporate non-helpful or harmful features. For example, if the domain shift introduces spurious correlations so that dissimilar classes (e.g., some fine-tuning and absent classes in the pre-training domains) have similar features in the downstream domain, then the features after fine-tuning might become misleading. \n\nAfter all, just like domain adaptation techniques typically degrade when there is a huge discrepancy between the source and target domains, we think it is reasonable that our findings or proposed approach may not work in some (extreme) conditions. We will surely extend the discussion so that readers or future users can get a better understanding of when the improvement may or may not hold. \n\n\n**Q2: … it looks like Tu et. al. achieves the best trade-off, while the proposed scaling method is only presented with a line. …**\n\n\nThank you for the suggestion and we will include the two $\\gamma$ selection methods in the figures where appropriate. \n\n\nMeanwhile, we want to reiterate the *difference* and *compatibility* between Tu’s method and our calibration method. We note that Tu’s method is a fine-tuning approach that updates the pre-trained model, and in our paper, we compare it to no fine-tuning and conventional full fine-tuning. The three $\\star$ in Figure 2 and Figure 7 of the main paper correspond to them: black for Tu’s method; green for no fine-tuning; and red for conventional full fine-tuning. (All of them are without calibration yet.)\n\n\nIn contrast, the calibration method with $\\gamma$ is to adjust the strengths of the logits of the fine-tuning and absent classes, and it can be applied as post-processing to all the aforementioned models: it creates the three Seen-Unseen Accuracy Curves (AUSUCs) in Figure 7. \n\n\nIn our paper, our main finding is that with calibration, the conventionally fully fine-tuned model (red) can regain its Absent class accuracy and can potentially achieve a better balance of Fine-tuning and Absent class accuracies than Tu’s method (black), as evidenced by that the red curves surpass the black curves in many of the figures (c.f., Figure 7, Figure U, Figure V, and Figure W). \n\n\nWe appreciate your detailed observations in several figures in Figure V, where the black $\\star$ (Tu’s method, with more complex training than full fine-tuning) seems to surpass the red curves or achieve the best trade-off that one can obtain on the red curves. We will certainly add the selected $\\gamma$ on the red curves. Besides that, we note that the calibration method can be used to improve Tu’s method as well, as shown in Figure U and Figure W. With an appropriate $\\gamma$, we could obtain a better trade-off along the black curves than the black $\\star$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N34WScvFT8", "reviewer": "Reviewer_9Ei3", "summary": "It is commonly believed that fine-tuning zero-shot models on seen classes will lead to a decrease in performance on unseen classes.\nIn this paper, the authors systematically examine the issue that find that (1)  the fine-tuned feature extractor is not damaged: NCM improves the absent class accuracy without catastrophic forgettin (2) main factor that damages the FT model’s ability to correctly classify absent class examples is the biased logit values towards fine-tuning classes. (3) a simple post-processing calibration of logits, ie, offsets the seen classes could bring back the zero-shot performance of absent classes. Extensive experiments and analyses validate the claims of the authors.", "questions": "Please refer to the weaknesses.", "limitations": "No negative societal impact has been identified.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is well-organized and the presentation is clear. The analyses are comprehensive and convincing. This study provides insights that corrected my previous viewpoint that fine-tuning causes the forgetting of knowledge of absent classes. This paper is undoubtedly a valuable work and worth accepting.", "weaknesses": "This is a solid paper, and I did not find any major weaknesses.\n\nI suggest that the authors include more experiments on CLIP models, such as the base-to-new setting in CoCoOp, reporting AccY/Y, AccS/Y, and AccU/Y on 11 datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback. This project has been a challenging journey, yet rewarding. We respond to your valuable comment below. \n\n**W: “I suggest that the authors include more experiments on CLIP models, …”** \n\nThank you for the suggestion. We will certainly include more experiments and discussions on CLIP models. Compared to conventional classification models that *learn for each class a linear classifier* on top of the feature extractor, CLIP models *learn a shared “text encoder”* that can generate linear classifiers (i.e., text embeddings) given class names or descriptions. We surmise that such a shared encoder would facilitate fine-tuning with a subset of classes. Concretely, instead of directly updating the linear classifiers as in conventional classification models, fine-tuning a CLIP model would update the “text encoder” that generates the linear classifiers. Given only the fine-tuning data from a subset of classes, the CLIP model has the potential to learn their common properties (e.g., domain specifics) and transfer them to the absent classes through the shared encoder. As a result, the discrepant logit scales might be reduced. The prompting capability of CLIP models further enables new approaches like CoCoOp to adapt the model without adjusting its parameters. We will perform a detailed and systematic study on fine-tuning with CLIP models, considering both full fine-tuning and CoCoOp and reporting ACCy/y, ACCs/y, and ACCu/y. (We note that our fine-tuning classes correspond to CoCoOp’s base classes; our absent classes align with its new classes.) We will also investigate whether calibration is compatible with CLIP models to further boost the fine-tuning performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BAifuP9tDc", "reviewer": "Reviewer_5T9R", "summary": "The authors argue that fine-tuning doesn’t forget the features for classes not participating in it, but rather downscales their logits as a result of which the model ends up being overconfident for the fine-tuning classes. Counter-intuitively, the authors claim that fine-tuning also enhances the discriminative ability of the model for the classes not participating in fine-tuning. For this, the authors analyze that accuracy of the NCM classifier on the features of the model and show that it increases even for classes absent during fine-tuning. Then the authors show that the order of the absent classes is still in place amongst the set of absent classes, but the model becomes overconfident on the classes used for fine-tuning, thereby leading to drop in accuracy of absent classes. To fix this, the authors analyze simple post processing calibration methods and demonstrate recovery in the performance of absent classes, with some drop in performance of fine-tuning classes. The analysis is done on Imagenet-R, VTAB and Office-Home datasets.", "questions": "It would be great if the authors can present their results on domainnet dataset [1].\nI request the authors to kindly address the questions in the weaknesses section.\n\n[1] https://paperswithcode.com/dataset/domainnet", "limitations": "Yes, the authors have addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1) The gains observed on calibrating the model are impressive, and it is interesting to see that merely by calibrating the model, the model’s performance on absent classes can improve to this extent.\n2) The motivation behind the post hoc calibration method is interesting and it is unexpected that just the confidence of the absent would lower down while preserving their relative order on performing fine-tuning.", "weaknesses": "1) An increase in NCM classifier’s accuracy need not necessarily mean that the model has become better in discriminating features. It only means that the features become closer to the corresponding class mean in l2 distance metric space. I think this argument is not concrete enough and requires more evidence.\n2) It is not clear, why a drop in accuracy is seen in the fine-tuning classes on using PVC as the post calibration method. Even in case of ALG, where there isn’t a significant drop, the accuracy of classes absent during fine-tuning is still significantly lower (e.g. on office home it drops by over 20%) than the pre-trained model. This suggests that authors claim on feature enhancement of absent classes might not be true. \n3) The authors propose to use training data / validation data to find the right threshold. This would mean access to the model as well as data. In such a scenario someone could rather easily fine-tune the model on absent classes as well, which would not require a lot of compute. Therefore, on a practical standpoint, it is not completely obvious how the proposed method benefits more than mere fine-tuning. It would be great if authors could compare the budget required for fine-tuning on absent classes vs post-hoc calibration to address this point to achieve similar performance. Although I agree that the observation itself is interesting, but analyzing the efficiency aspect could help the authors in making the claim on using post calibration methods stronger.\n\n4) I think that the analysis shown in Fig-9 needs more rigor. Further, I think the authors need not discuss this in the main paper, as it doesn't adds much. The learning rate used for different optimizers are same, but the learning rates for adam, adagrad and adagrad need to be scaled down to make a fair comparison with SGD.\n5) The analysis on why absent class features improve on fine-tuning is not rigorous and would encourage authors to not discuss this in the main paper. Most of the claims in this analysis seem low hanging and not sound enough. This hinders the readily of the paper currently.\n6) Similarly, figure-10 doesn't adds much to the storyline authors have presented in this work and it requires more rigor. I would suggest them to restructure the paper a bit and remove this analysis from the main paper.\n\nMinor comments:\nIn figure-5, the y-axis should be between 0-1 since it represents a probability.\n\nI would be happy to increase my scores if my concerns are sufficiently addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments. We are pleased that the reviewer recognize several key strengths of our paper. We respond to your concerns (weaknesses and questions) as follows.\n\n**W1. NCM classifier’s accuracy and discriminating features.**\n\nThank you for the comment. Our paper focuses on classification problems. Thus, we consider improvement in *classification accuracies* as one critical metric to assess the *features’ discriminative abilities*. We note that increasing the NCM accuracy requires the features of each class not only to be close to the corresponding class but also to be far away from other classes, meaning that the class means need to be separated apart as well. These properties align with linear discriminative analysis and, in our humble opinion, are what we expect discriminative features to possess. \n\nBesides NCM, we note that in section 4.3 ( specifically Figure 4), the fine-tuned model’s accuracy among absent classes also increases. We view it as another evidence that the features’ discriminative ability improves. \n\nWith that being said, we are aware that NCM is not the only way to assess feature quality, and we will be happy to include further analyses. \n\n\n**W2: Accuracy by PCV and ALG.** \n\nWe respectfully think there might be a misunderstanding. Our main finding in the paper is that with proper calibration of the logits, the fine-tuned model (red $\\star$ in Figure 2) can regain and even increase its accuracy in the absent classes (the y-axis value along the red curve can surpass the green $\\star$), suggesting that its features on absent classes do not degrade but often improve. (Please also see our remark in Lines 94-98.) \n\nWe note that the post-processing calibration factor $\\gamma$ is applied after the model has been fine-tuned. In other words, the accuracy drop in either the fine-tuning or absent classes (as the reviewer found) may result from sub-optimal selections of $\\gamma$ and have nothing to do with feature qualities. Indeed, when $\\gamma$ is not properly set, either the fine-tuning class accuracy or the absent class accuracy can drop drastically, to zero, corresponding to the two endpoints of the red curves in Figure 2 and Figure 7. \nPCV and ALG are two approaches to selecting $\\gamma$, but we certainly do not claim they are optimal. In Table 1, we can see that “Fine-tuning + $\\gamma^\\star$” could achieve a much better balance of ACCu/y and ACCs/y than “Fine-tuning + $\\gamma$PCV” and  “Fine-tuning + $\\gamma$ALG.” We note that “Fine-tuning + $\\gamma^\\star$” outperforms the pre-trained model in ACCu/y and ACCs/y, indicating the fine-tuned models have enhanced accuracy in both categories, if calibrated properly. We leave a better, more robust approach to select $\\gamma$ in our future work.\n\n**W3: Training data/validation data to find the right threshold.** We respectfully think there might be a misunderstanding. We apologize if we did not describe the setting clearly and we will certainly improve it.\n\nAs mentioned in Section 5 of the main paper, “Pseudo cross-validation (PCV) partitions D_{tr} into **pseudo**-fine-tuning and **pseudo**-absent classes and finds $\\gamma$ that can balance the pseudo-fine-tuning and pseudo-absent class accuracy.” We note that D_{tr} is not the data used in pre-training (which contains absent class data), but the fine-tuning data, as defined in section 3 (Lines 150 - 155). The **pseudo**-fine-tuning and **pseudo**-absent classes are both from the fine-tuning classes; no absent class data are exposed to PCV. (More details can be found in section B.2 of the appendix.)\n\n**W4: Figure 9 and learning rates.** \nWe will improve Figure 9 to make it easier to read. We respectfully think that our study in section 6 (Lines 316 - 325) is valuable and essential, as we want to know whether our findings in sections 4 and 5 are general or optimizer-specific. Regarding the learning rate, we note that the values along the x-axis in Figure 9 are not the exact learning rates, but the multiplying factors to the **default learning rate (LR)** of each optimizer, respectively (please see Lines 319 - 320). We obtain the default LR from the `torch.optim` and `pytorch-optimizer` packages. We presented their values in Table R.3 in the rebuttal pdf, and we will include it in the camera-ready version. We apologize that we did not make this part clear in the main paper. After all, Figure 9 is not meant to argue that SGD is better than other optimizers, but to show that our findings in sections 4 and 5 hold for different optimizers when their learning rates are properly set. \n\n**W5 & 6: Analysis (Lines 326 - 360) and Figure 10.** \nThank you for your feedback. We apologize if these parts were not rigorous, and we will consider moving them to the Appendix. We acknowledge that the analysis is not a formal theory, and we will certainly clarify it. We note that we have conducted several other analyses to understand our findings (c.f. Line 361 - 363). Due to the space limit, we keep them in the Appendix (section C and section D.5). We will polish and condense some of them to replace the current Lines 326 - 360 in the main paper.\n\n**W7: Figure 5.** We will adjust the y-axis. Thank you for pointing it out.\n\n**Q1: DomainNet**\nAs per your suggestion, we conducted additional experiments on DomainNet. We followed setting 1 in section 4.1 to pre-trained a Resnet-50 model on the ‘Real’ domain with 345 classes and then fine-tuned it on the randomly selected 170 classes in other five domains(ClipArt, Sketch, Infograph, Painting, and Quickdraw). We include the results in the rebuttal PDF (Table R.1 and Table R.2).\n\nWe can see that average AUSUC (c.f. Lines 280 - 289 in the main paper) and ACCu/y both increase after fine-tuning, even though absent classes have not been involved in the fine-tuning process, which is consistent with our findings from other datasets. We will include the complete results and discussions in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N5EBjEHE38", "reviewer": "Reviewer_YzGv", "summary": "The paper proposes a simple post-training calibration technique for classifying missing classes after fine-tuning. For example, assuming the pre-trained model can classify 1000 classes and is fine-tuned on a subset of these classes from a different image domain, the proposed method improves the classification accuracy of the *absent classes* on this new domain. Specifically, the proposed method adds a calibration hyper-parameter to artificially boost the probability of predicting the absent classes.  To motivate the method, the paper investigates the quality of feature learning using the Nearest Class Mean classifier to isolate the cause of bad performance on the absent classes. While simple, the method shows good performance gain on multiple datasets.", "questions": "* While the paper investigated several factors that can affect calibration effectiveness, it is unclear how we should decide when the calibration is useful. Could the authors consider dependency on the model's **intrinsic** properties? The external factors all may lead to a common intrinsic property. For example, maybe the fine-tuned model's deviation (in the weight space) can be an indicator.  For example, RMSprop and Adam are known to converge faster and potentially lead to a larger deviation from the pre-trained model?", "limitations": "The paper does not have a potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed method is easy to implement and provides good performance gains. \n* The paper uses the NCM classifier to investigate the feature extractor's quality and isolate the linear classifier's influence after fine-tuning.  This methodology provides a clear motivation for the proposed method. \n*  The finding that fine-tuning does not completely destroy the features of absent classes is interesting. This insight can motivate further study, especially for improving the robustness of fine-tuning. \n* The paper provides a detailed ablation study showing the proposed method's strengths and limitations. \n  * The extent of performance gain hinges on the fine-tuning procedure. \n  * The distribution of absent classes also affects performance.", "weaknesses": "* **Statements are not precise**. In the abstract and introduction, the paper claims that a fine-tuned model does not forget the relationship among absent classes. However, this claim is not precise. As the paper points out, the extent of forgetting and degradation depends on the fine-tuning procedure. For example, an Adam optimizer with a larger learning rate can degrade the features of the absent class. This is consistent with the existing literature on the robustness of fine-tuning.  It's possible that the proposed technique only works well under moderate changes to the pre-trained model. \n\n* **The fine-tuning setting is limited**. The proposed method only works for a particular fine-tuning configuration under constrained assumptions. Specifically, the method assumes a classification task, and the pre-trained model can classify all fine-tuning classes, including absent ones. Therefore, the paper's claims on forgetting and feature learning are limited by its scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed review and positive assessment of the strengths of our paper. We address your concerns as follows.\n\nWeakness:\n**W1: Statements are not precise. … The extent of forgetting and degradation depends on the fine-tuning procedure. ….**\n\nWe apologize if our statement in the abstract and introduction were not precise or clear. In our humble opinion, learning rates are hyper-parameters of optimizers, and our statement was made assuming the hyper-parameters are properly selected. As mentioned in Lines 31-38, our study was motivated by the findings in [48]. We note that in [48], SGD is the main optimizer and *the learning rate has been carefully chosen.* However, the absent class accuracy still suffers a drastic drop, and [48] viewed it as an instance of forgetting. Our statement in the abstract, *“To our surprise, we find that the fine-tuned model neither forgets the relationship among the other classes nor degrades the features to recognize these classes,”* was made against the claim by [48]. We found that the accuracy drop is mainly due to the biased logits, not the forgetting of features and class relationships for the absent classes. \n\nWe hope the paragraph above clarifies your concern. \n\nIn our humble opinion, when the learning rate is not well chosen or the optimizer is not applied properly, any observations about the machine learning model may be misleading or doubtful. \nFor full disclosure, we indeed (re-)learned such a lesson when we extended our study beyond the SGD optimizer used in [48] (Lines 72 - 76 and Lines 316 - 325). When we first applied the Adam optimizer, we followed the practice of setting a small learning rate (e.g., 1e-3 and 1e-4) to ensure that the training could converge (on the fine-tuning classes). Under such a setting, we saw a notable gain in the fine-tuning class accuracy but a poor NCMu/y and ACCu/u (thus a poor AUSUC), contracting our findings (Lines 42 - 64) when using the SGD optimizer. At that time, *we almost drew the conclusion that our finding is optimizer-specific.* However, by further reducing the learning rate, we once again saw a decent NCMu/y and ACCu/u and a similar gain in AUSUC as using SGD (see Figure 9). This motivates us to conduct a comprehensive analysis, exploring a wide range of learning rates across six optimizers. \n\n**Q1: It's possible that the proposed technique only works well under moderate changes to the pre-trained model.** We appreciate your question. As our goal is to preserve (and even improve) the discriminative ability on the absent classes from the pre-training model to the fine-tuning models, it is reasonable that the change to the pre-trained model cannot go arbitrarily large. With that being said, the increase in AUSUC in Figure 9 (e.g., from 0.5 to 0.63) suggests that the fine-tuned model has undergone a sufficient change to improve its performance in the downstream domain.   \n\n**Q2. The fine-tuning setting is limited.** \nWe acknowledge that our scope is limited to classification tasks. However, we respectfully think it is a limitation but not necessarily a weakness. In our humble opinion, many advances in machine learning start from the investigation of classification problems (e.g., AlexNet and self-supervised learning) and gradually extend to other tasks. While we work on a particular fine-tuning configuration with a subset of classes, we respectfully think it is a practical setting (with access to large pre-trained models like CLIP but limited fine-tuning data) that deserves deeper exploration. With this in mind, we conducted a systematic, extensive analysis on a focused topic. While we agree that exploring other tasks could broaden the applicability of our findings, the focused scope of our current study allows us to thoroughly investigate and validate our claims.\n\n**Q1: Dependency on the model's intrinsic properties.** \nThank you for the insightful question. As mentioned in Lines 85 - 98 and Lines 253 - 257, the calibration method works when the drop in the absent class accuracy primarily comes from the biased logits, not the degradation of the feature extractor or the relationship among absent classes. \n\nWe follow your suggestion to explore the model's intrinsic properties. Since the parameter space of the model is of extremely high dimensionality, it is hard to use the L2 distance between the pre-trained and fine-tuned models to quantify the model deviation. We thus explore a different measurement. \n\n\nWe calculated the linear Centered Kernel Alignment (CKA) in Appendix C.5 (Lines 733-737) and found that the CKA of the absent classes’ linear classifiers (between the pre-trained and fine-tuned models) correlates well with performance after calibration. As shown in Figure R.1 of our rebuttal PDF, the Unseen CKA across different optimizers follows a similar pattern to their Area Under the Seen-Unseen Curve (AUSUC) in Figure 9. Specifically, when the Unseen CKA falls below a certain threshold (e.g., 0.98), there is a corresponding drastic drop in AUSUC. This suggests that the CKA can be a useful indicator of calibration effectiveness. We will complete and include this study in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable comments. We are glad that the reviewers found our findings and motivations “interesting” (amJX, 5T9R, YzGv), our study providing “insights” correcting prior belief (9Ei3), our solution “simple and easy to use” (amJX, YzGv) with “impressive” gains (5T9R), our empirical results “extensive” (amJX, 9Ei3, YzGv), our analyses “comprehensive and convincing” (9Ei3, YzGv), and our presentation “clear” and “well-organized” (amJX, 9Ei3). Reviewer 9Ei3 further said, “This paper is undoubtedly a valuable work and worth accepting.” Reviewer YzGv also said, “This insight can motivate further study, especially for improving the robustness of fine-tuning.” 

We have tried our best to address most if not all of the concerns, and we will modify our camera-ready version accordingly to incorporate all the feedback.

Meta-review: The paper presents a nice observation that "forgetting" of absent classes during finetuning might occur due to poor calibration rather than actually destroying information about classes. Based on this, the paper provides a method to finetune that does not result in this forgetting of absent classes. The experiments show good gains, and all the reviewers are generally positive. While this is overall a valuable contribution, the main weaknesses are around on the scope of the results: unclear how it applies to CLIP-style models with text encoders that are more popular currently, and only applies to cases where we know apriori the set of absent classes we care about (even if we don't need any explicit data access). This restricts the scalability/generality of this approach, but the findings and method are nevertheless interesting and hence I recommend acceptance.
Decision: Accept (poster)