Title: Fine-Tuning is Fine, if Calibrated
Venue: NeurIPS 2024
Accepted: yes
Keywords: Fine-Tuning

Pre-training

Domain Adaptation
Abstract: Fine-tuning is arguably the most straightforward way to tailor a pre-trained model (e.g., a foundation model) to downstream applications, but it also comes with the risk of losing valuable knowledge the model had learned in pre-training. For example, fine-tuning a pre-trained classifier capable of recognizing a large number of classes to master a subset of classes at hand is shown to drastically degrade the model's accuracy in the other classes it had previously learned. As such, it is hard to further use the fine-tuned model when it encounters classes beyond the fine-tuning data. In this paper, we systematically dissect the issue, aiming to answer the fundamental question, "What has been damaged in the fine-tuned model?" To our surprise, we find that the fine-tuned model neither forgets the relationship among the other classes nor degrades the features to recognize these classes. Instead, the fine-tuned model often produces more discriminative features for these other classes, even if they were missing during fine-tuning! What really hurts the accuracy is the discrepant logit scales between the fine-tuning classes and the other classes, implying that a simple post-processing calibration would bring back the pre-trained model's capability and at the same time unveil the feature improvement over all classes. We conduct an extensive empirical study to demonstrate the robustness of our findings and provide preliminary explanations underlying them, suggesting new directions for future theoretical analysis.

Fine-Tuning is Fine, if Calibrated

Abstract

Fine-tuning is arguably the most straightforward way to tailor a pre-trained model ( e.g ., a foundation model) to downstream applications, but it also comes with the risk of losing valuable knowledge the model had learned in pre-training. For example, fine-tuning a pre-trained classifier capable of recognizing a large number of classes to master a subset of classes at hand is shown to drastically degrade the model's accuracy in the other classes it had previously learned. As such, it is hard to further use the fine-tuned model when it encounters classes beyond the fine-tuning data. In this paper, we systematically dissect the issue, aiming to answer the fundamental question, 'What has been damaged in the fine-tuned model?' To our surprise, we find that the fine-tuned model neither forgets the relationship among the other classes nor degrades the features to recognize these classes. Instead, the fine-tuned model often produces more discriminative features for these other classes, even if they were missing during fine-tuning! What really hurts the accuracy is the discrepant logit scales between the fine-tuning classes and the other classes, implying that a simple post-processing calibration would bring back the pre-trained model's capability and at the same time unveil the feature improvement over all classes. We conduct an extensive empirical study to demonstrate the robustness of our findings and provide preliminary explanations underlying them, suggesting new directions for future theoretical analysis. Our code is available at https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated .

1 Introduction

Pre-trained models ( e.g ., foundation models) have become an indispensable component in modern AI development [2]. Building upon neural networks with millions if not billions of parameters and trained with gigantic amounts of data, these models have led to groundbreaking results in various domains [30, 32, 39] and shown several emerging capabilities not observed priorly [21, 27, 2].

Yet, to obtain superior downstream performance, fine-tuning is still often needed. Typically, finetuning optimizes the model's performance on the available downstream data. Taking image classification as an example, end-users typically fine-tune the pre-trained classifier to maximize the accuracy of a certain set of classes at hand, no matter whether they know up front that it is the complete set of classes or not. As a result, it is hard to further apply the model to some other classes, even if the model had learned about those classes in pre-training. (Please see Figure 1 for an illustration.)

A recent work by Tu et al . [49] systematically evidenced such a problem. They fine-tuned a pretrained classifier with a subset of classes it had learned ( i.e ., fine-tuning classes) and found this led to a drastic accuracy drop in the other classes ( i.e ., absent classes). Tu et al . [49] viewed this as an instance of catastrophic forgetting [12, 24, 7] and suggested two ways to address it: (1) identifying the model updating direction that can improve both the fine-tuning and absent classes; (2) preserving class relationships. They presented a strong baseline, combining a new stochastic gradient descent (SGD)

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

Absent Class Data Accuracy

ImageNet -> ImageNet-S (50% Fine-tuning Classes)

Fine-tuning +

Pre-training

Fine-tuning

ModelPre-training

ModelFine-tuning

ModelDeployment

Pre-trainingDomain

DownstreamDomain

DownstreamDomain

(e.g.,ImageNet)

(e.g.,ImageNet-S)

(e.g.,ImageNet-S)

Dog

Fine-tuning.

Dog

Dog

Eagle

Classes

Eagle

Eagle

Truck

Absent

Truck

Classes

Cup

LCup

Cup

Pre-training

Fine-tuning

Deploy

strategy, feature rank regularization [6, 46], distillation [16], and frozen classifiers [29], achieving decent results on preserving absent class accuracy while improving fine-tuning class accuracy.

We build upon [49] to further analyze the problem of fine-tuning a pre-trained model . Specifically, we want to understand which part of the pre-trained classifier or what knowledge it had learned is damaged after fine-tuning with a subset of classes.

Wefirst analyze the feature extractor of the fine-tuned model. If the fine-tuned feature extractor forgets the absent classes, the extracted features would exhibit poor discrimination among these classes or confuse them with the fine-tuning classes. We apply the Nearest Class Mean (NCM) classifier [5, 44, 55] to assess the feature quality after fine-tuning, using the hold-out data for calculating class means. Surprisingly, the fine-tuned feature extractor does not degrade but often improves for the absent classes! The resulting features could better separate absent-class data and raise the NCM classification accuracy over all the classes. This finding suggests that fine-tuning with merely a subset of classes can already holistically improve the feature extractor for the downstream domain.

To search for the root cause of accuracy drops, we analyze the fine-tuned classifier as a whole but decompose its prediction rule into two parts : a binary classifier separating fine-tuning and absent classes, followed by the multi-class classifier dedicated to each set of classes. We find that the fine-tuned model can distinguish among the absent classes very well, implying that the absent-class relationship in the fully connected layer is preserved. The only component that is damaged but ends up failing the whole model is the binary classifier. Concretely, the biased logits towards fine-tuning classes make most absent-class examples misclassified as fine-tuning classes.

The above findings are encouraging and have profound implications! They imply that a simple postprocessing calibration can potentially address the fine-tuned model's inferior accuracy on the absent classes, bringing back the pre-trained model's capability while unveiling the improved feature quality over all classes. Indeed, by adding a calibration bias factor to all the absent classes' logits [4, 41], the fine-tuned model can successfully reclaim the absent class accuracy and obtain decent overall improvement in the downstream domain (Figure 2). The resulting performance even beats the strong baseline [49] in many of the benchmarks, including ImageNet and its variants [8, 15, 54], Office-Home [53], and VTAB [64], without complicated training and hyperparameter setting .

The 'unexpected benign behaviors' of fine-tuning with a subset of classes raise several interesting follow-up questions. For example, are there any specific setups in our experiment contributing to the benign behaviors? If not, what are the explanations underlying these benign behaviors?

We consider different splits of fine-tuning and absent classes in terms of their distributions and numbers. We find that the benign behaviors are robust to the number of fine-tuning classes. Even if the fine-tuning and absent classes are semantically distinct ( e.g ., animals as fine-tuning and vehicles as absent classes), the absent class accuracy after fine-tuning stays quite close to that of the pre-trained model without suffering catastrophic forgetting. We further investigate different optimizers for fine-tuning. When the SGD optimizer is used, the benign behaviors are robust to hyperparameters such as learning rates and weight decay. When more advanced, adaptive optimizers like Adam [23] are applied, we observe noticeable degradation with improperly selected hyperparameters. Still, with smaller enough learning rates and weight decay, the benign behaviors show up and hold.

We conduct further analyses to understand the benign behaviors. Specifically, we investigate how fine-tuning with a subset of classes impacts the other ( i.e ., absent) class features. Our derivation shows that after each SGD step, the feature update of an absent-class example is governed by the gradients w.r.t. its most similar fine-tuning class examples. Suppose the gradients w.r.t. fine-tuning class data could effectively capture the downstream-specific information, our derivation offers a preliminary explanation of why fine-tuning with a subset of classes could improve the other, absent class features in the downstream domain. Please refer to section 6 for details and other analyses regarding the class relationship, frozen classifiers, etc.

Remark. We systematically dissect the damage caused by fine-tuning a pre-trained classifier with a subset of classes at hand. Our insights suggest that a simple post-processing calibration is sufficient to mitigate the damage, reclaiming the pre-training model's capability while holistically improving the accuracy in the downstream domain. Our study also opens up several interesting questions worth further exploration. For example, it is intriguing that end-to-end fine-tuning of the whole model without additional regularization and intermediate supervision only degrades a part of the model.

We note that NCM and post-processing calibration are well-known machine learning techniques and we certainly do not claim them as our novelties. Instead, we use them because we find their respective properties and application scenarios suitable for our problem. We use NCM to assess feature qualities because its performance solely depends on features, not the last fully connected layer. We identify post-processing calibration as a promising solution because we find that the only major damage in the fine-tuned model is the biased logits towards fine-tuning classes . In other words, if the fine-tuned model also suffers from feature degradation or class relationship forgetting (to our surprise, it does not), calibration alone is unlikely sufficient to address the problem.

2 Related Work

Fine-tuning. The basic methods are linear probing and full fine-tuning [26]. Parameter-efficient finetuning (PEFT) [60, 35, 50] has attracted increasing attention lately (mainly for Transformers [52]), aiming to update only a fraction of pre-trained parameters on top of linear probing. We focus on full fine-tuning because it is model-agnostic and arguably still the most widely used method. That said, we expect the insights and implications from our study not to be limited to full fine-tuning and potentially transferable to other fine-tuning methods as well.

Risk in fine-tuning. When the downstream data is scarce, fine-tuning is prone to over-fitting and needs certain forms of regularization [28]. Even with sufficient data to represent the downstream task, fine-tuning may risk losing valuable knowledge the pre-trained model had learned. For example, [57] showed that fine-tuning with data from a specific domain ( e.g ., ImageNet-1K real images) degrades the pre-trained model's generalizability to other domains ( e.g ., ImageNet-Sketch/Rendition). On an orthogonal dimension, [49] showed that fine-tuning a pre-trained classifier with a subset of classes it had learned led to a huge accuracy drop in the other classes. Similar phenomenons have been observed in [66, 40]. Along with these findings come several proposed solutions. [57] showed that a weight interpolation between the pre-trained and fine-tuned models reclaims the pre-trained model's generalizability. [49] investigated many approaches, including weight interpolation, but found they are insufficient to preserve the accuracy in the other classes. They presented a novel SGD strategy that helps identify the gradient direction benefiting both fine-tuning and other classes. [66] developed dedicated solutions to CLIP-based vision-language models to preserve accuracy for absent classes.

We 1) consider the fine-tuning scenario studied in [49] and 2) use the standard neural network classifier architecture with a fully connected layer on top. Our focus is not to propose a brand-new solution and compete with existing ones, but to understand the underlying cause of the accuracy drop.

Continual learning and catastrophic forgetting. Fine-tuning a pre-trained model with a subset of classes it had learned is related to continual learning [7] but has several differences. Class-incremental learning [33, 36] aims to expand the model's label space. In contrast, we suppose the pre-trained model had learned a wide range of classes; fine-tuning is meant to tailor it to a downstream domain ( e.g ., a different image style), not to learn extra categories. Compared to domain-incremental learning [33, 31], we do not ask the fine-tuned model to retain its performance in the pre-training domain. That said, the accuracy drop observed in our study and continual learning can all be considered certain forms of catastrophic forgetting [37, 12]. Indeed, a recent survey by Wang et al . [56] argued that forgetting is a common issue in various research fields, including foundation models, meta-learning,

test-time adaptation, and generative models, among others. We thus expect our findings to serve as a valuable reference for addressing the forgetting issue resulting from fine-tuning.

Post-processing calibration. Training with class-imbalanced data is known to produce biased logits towards major classes [61]. Such an issue not only appears in long-tailed recognition [20, 18] but also few-shot learning [62, 45] and continual learning with replay buffers [48, 47, 34]. Post-processing calibration [3, 38, 58, 59] is a widely applicable method to address this issue, which adjusts model confidence during inference. Popular methods include normalizing classifier norms [20, 17] and adjusting logits according to class sizes [61, 38, 22, 65]. Unlike the machine learning problems above, whether or not post-processing calibration can address the accuracy drop in our problem is not immediately clear. Without access to the absent class data, fine-tuning may have ruined the features or linear classifiers corresponding to these classes. If so, simply performing post-processing calibration cannot reclaim the accuracy of the absent classes. Our main contribution is therefore not merely the solution, but the systematic study that identifies post-processing calibration as an effective solution.

Other paradigms. There are several other machine learning paradigms related to the fine-tuning setting we study. We refer the readers to [49] for an in-depth discussion and comparison.

3 Background

Problem definition. We study the problem of fine-tuning a pre-trained classifier capable of recognizing C classes, using data from a subset of C ⋆ classes at hand. The goal is to tailor the classifier to the downstream domain ( e.g ., a different image style).

More formally, let us denote by D tr = { ( x i , y i ∈ S ) } N i =1 the data set for fine-tuning, where S is a strict subset of the pre-trained model's label space Y , i.e ., |S| = C ⋆ < C = |Y| . Let us denote a neural network classifier by ˆ y = arg max c ∈Y w ⊤ c f θ ( x ) , where x is an input sample, f θ is the feature extractor parameterized by θ , and W = { w c } C c =1 is the set of linear classifiers, a.k.a., the fully-connected (FC) layer. We call { θ , W } the model parameters. The value w ⊤ c f θ ( x ) is often referred to as the decision value or logit of class c .

Without loss of generality, we define Y = { 1 , · · · , C } and S = { 1 , · · · , C ⋆ } . That is, they share the first C ⋆ classes. We call S the fine-tuning classes and use U = { ( C ⋆ +1) , · · · , C } to denote the absent classes during fine-tuning, where S ∩ U = ∅ and S ∪ U = Y .

Fine-tuning and its issue. Full fine-tuning updates the pre-trained model { θ O , W O } by

where ℓ denotes the cross-entropy loss; { θ T , W T } denotes the fine-tuned model.

Since D tr only covers a subset of classes S , the fine-tuned model { θ T , W T } was observed to degrade drastically in classifying data from the absent classes U [49]. Figure 2 shows one example: the absent class accuracy in the y-axis drops from ∼ 23% ( ⋆ ) to only ∼ 3% (marked ⋆ ) after fine-tuning, even though the fine-tuning class accuracy in the x-axis increases hugely by roughly 60% .

Terminology. Tu et al . named the above setting Holistic Transfer (HT) [49]. The core challenge is how to maintain and even improve the fine-tuned model's ability to recognize the C -C ⋆ absent classes in the downstream domain. For ease of reference, we use the following terms interchangeably.

fine-tuning classes & seen classes; absent classes & unseen classes;

downstream domain & target domain; pre-training domain & source domain;

fine-tuning & naive fine-tuning.

We emphasize that the 'unseen' classes are indeed seen in pre-training but absent in fine-tuning.

4 A Systematic Study of Fine-Tuning (FT)

Seeing the ability to classify the absent classes 'disappears' after fine-tuning, we are curious about

how each component of the fine-tuned model { θ T , W T } contributes to it;

NN Accuracy

NCM Accuracy

20

0

20

40

60

Fine-tuning Classes

Absent Classes

(A) ImageNet-{R, S}

NN Accuracy

NCM Accuracy

20

0

20

Fine-tuning Classes

Absent Classes

(B) VTAB

NN Accuracy

NCM Accuracy

40

20

0

20

Fine-tuning Classes

Absent Classes

(C) Office-Home

whether the ability is 'forgotten' forever or 'buried' temporarily by some other damaging factors emerging during fine-tuning.

To this end, we conduct a systematic analysis, aiming to dissect the degradation caused by fine-tuning.

4.1 Experiment setup: datasets, models, and evaluation metrics

We focus on two of the largest datasets used in [49]. We also consider the ImageNet Distribution Shift benchmark widely used in out-of-distribution (OOD) generalization [57].

Office-Home [53]: a domain adaptation dataset with 65 classes and 4 domains. For each sourcetarget pair, we pre-train a ResNet-50 [14] using the source data and fine-tune it on 30 randomly selected classes in the target domain. We evaluate the model on all 65 classes in the target domain.

VTAB [64]: a set of 19 visual recognition datasets. We follow [49] to use the 8 datasets provided with text labels and use CLIP (ViT-B/32) [42] as the pre-trained model. We extract the class name embedding to construct the FC layer and discard the CLIP text encoder afterward. We fine-tune the model on the randomly selected 50% of classes in each dataset and test it on all the classes.

ImageNet-R [15] and ImageNet-S [54] : datasets for OOD detection and generalization [57]. ImageNet-R [15] consists of 200 ImageNet classes [8, 43] with renditions (paintings, cartoons, etc.). ImageNet-S [54] consists of 1K ImageNet classes with sketching. We use ResNet-50 (ViTB/32 [9] results in Appendix D) pre-trained on ImageNet-1K. We fine-tune them on randomly sampled 50% of classes (100/500 for ImageNet-R/S, respectively) and test them on all the classes.

By default, we use the cross-entropy loss and optimize the model using the SGD momentum optimizer.

Evaluation metric. Let D tr = { ( x i , y i ∈ Y ) } j M =1 denote the downstream test set covering all the classes, we define the accuracy of classifying data belonging to classes A into the label space B by

✶ For instance, Acc S / Y is the accuracy of classifying fine-tuning class data into all C classes; Acc U / Y is the accuracy of classifying absent class data into all C classes. We note that these are the two accuracies reported in [49] and depicted in Figure 2. In this paper, we further consider Acc S / S and Acc U / U , corresponding to classifying each set of data into their respective label space.

4.2 Is the fine-tuned feature extractor damaged?

We first investigate the feature extractor f θ , i.e ., whether the fine-tuned extractor f θ T forgets the discriminative ability to differentiate absent-class samples. We apply the NCM classifier [5, 44, 55], whose accuracy is solely governed by the feature quality, to examine the feature extractor. Given a test example x , the NCM classification rule is

which outputs the class ˆ y ∈ B whose feature mean µ ˆ y is the closest to f θ ( x ) . We hold out a subset of the downstream data to calculate the class mean, for both fine-tuning and absent classes.

0

5

10 15 20 25 30 35 40 45 50

Epoch

30

40

50

60

70

Absent Classes Acc

Office-Home

VTAB

ImageNet

0

5

10 15 20 25 30 35 40 45 50

Epoch

0

20

40

60

80

1st T erm

Office-Home

VTAB

ImageNet

0

5

10 15 20 25 30 35 40 45 50

Epoch

0

20

40

60

80

Office-Home

VTAB

ImageNet

We compute the NCM accuracy (with B = Y ) on the fine-tuning class and absent class data using the fine-tuned (FT) feature extractor f θ T , i.e ., Acc S / Y ( f θ T ) and Acc U / Y ( f θ T ) . We also apply the pre-trained feature extractor f θ O and obtain Acc S / Y ( f θ O ) and Acc U / Y ( f θ O ) . We report the NCM accuracy gap between the two feature extractors in Figure 3. As one may expect, the FT extractor f θ T improves the fine-tuning class accuracy. However, surprisingly, it also improves the absent class accuracy without catastrophic forgetting . This result sharply contrasts the accuracy obtained by the full neural network (NN) classifier (cf. section 3). As shown in Figure 3, the absent class accuracy by the NN classifier drops by 15 ∼ 40% after fine-tuning. In short, we find that fine-tuning with a subset of classes can adapt the feature extractor holistically to the downstream domain to improve both the fine-tuning and absent classes, capturing the gradient direction of the domain shift [49].

Remark. We use NCM only to assess feature qualities, not as the final classifier. After all, we do not have absent class data to compute the class means. This contrasts the use of NCM in continual or few-shot learning, where the prior- or few-shot class data are accessible ( e.g ., through replay buffers).

We emphasize that the feature improvement here is not as trivial as observed in conventional transfer or few-shot learning. In these learning problems , the pre-trained model typically had not learned about the absent classes; thus, there is no risk of forgetting them. Besides, the fine-tuning and absent classes are often treated as two separate tasks in evaluation; thus, it is unclear whether the fine-tuned feature extractor improves the overall classification accuracy. In contrast, in our fine-tuning setting , the pre-trained model had already learned about the absent classes. Fine-tuning thus risks forgetting them. However, as shown in Figure 3, fine-tuning improves absent class accuracy, in the context where the fine-tuning and absent classes are evaluated together in a single task.

4.3 What is damaged in the fine-tuned neural network classifier?

The findings in subsection 4.2 eliminate the feature extractor θ T from the candidate root causes of the degraded FT model { θ T , W T } . The drastic drop of absent class accuracy thus must come from the FC layer W T or its alignment with the feature extractor. To analyze what goes wrong, we decompose the softmax probability induced by the neural network classifier as follows,

where z c ( x ) = exp ( w ⊤ c f θ ( x ) ) . Let c be an absent class, the 1 st term stands for the predicted probability that x belongs to the absent classes U , not the fine-tuning classes S ; the 2 nd term stands for the probability that within the absent classes U , x belongs to class c . Correctly classifying an absent class example ( x , y ∈ U ) , i.e ., y = arg max c ∈Y p ( c | x ) , thus requires 1) obtaining a high probability in the 1 st term and 2) correctly classifying the example among absent classes.

Building upon this insight, we analyze the accuracy of taking arg max of the 2 nd term to classify absent class examples. We note that this is exactly the Acc U / U defined in subsection 4.1. As shown in Figure 4, Acc U / U does not degrade but improves in the first few epochs of fine-tuning and stays stable afterward. This result implies two key messages. First, the FT model does not forget its ability to distinguish among absent classes. Second, the drastic accuracy drop of Acc U / Y results from the 1 st term in Equation 4 - the binary classifier separating fine-tuning and absent classes.

As shown in Figure 5, the predicted probability that absent class examples belong to absent classes ( i.e ., the average of the 1 st term over absent class examples) reduces notably along with the fine-tuning

epochs, suggesting the tendency to misclassify an absent class sample as fine-tuning classes. Indeed, as shown in Figure 6, for most of the absent class samples, the FT model ends up producing a lower value of max c ∈U w ⊤ c f θ ( x ) than max c ∈S w ⊤ c f θ ( x ) , misclassifying them into fine-tuning classes. In short, the main factor that damages the FT model's ability to correctly classify absent class examples is the biased logit values towards fine-tuning classes.

5 Post-Processing Calibration for the Rescue

The systematic study in section 4 highlights several key characteristics of the FT model { θ T , W T } . First, the model retains and even improves the accuracy of classifying absent class samples when the label space is limited to absent classes ( i.e ., Acc U / U ). Second, the model tends to assign much higher decision values w ⊤ c f θ ( x ) , a.k.a. logits, to the fine-tuning classes, ending up misclassifying most absent class samples into fine-tuning classes and thus hurting Acc U / Y .

These characteristics suggest that a simple, post-processing calibration of the FT model's logits could potentially bring back the pre-trained model's ability to classify absent classes correctly. To this end, we apply the calibration approach proposed in a different context of zero-shot learning [4], which adds a factor γ uniformly to the logits of all absent classes, leading to a new classification rule

ˆ y = arg max c ∈Y w ⊤ c f θ ( x ) + γ ✶ [ c ∈ U ] . (5) This calibration factor lifts the logits of absent classes to a level comparable with those of fine-tuning classes. Suppose an absent class example is correctly classified among absent classes but misclassified into fine-tuning classes, adding a sufficiently large γ could reclaim the correct prediction.

We design two approaches to properly set γ without accessing the absent class data. Average logit gap (ALG) estimates γ by the average logit gap between non-ground-truth fine-tuning classes and absent classes in the fine-tuning data D tr . Pseudo cross-validation (PCV) partitions D tr into pseudo-fine-tuning and pseudo-absent classes and finds γ that can balance the pseudo-fine-tuning and pseudo-absent class accuracy. We also investigate a 'cheating' γ⋆ based on the test data to estimate the upper-bound results. More details are in Appendix B.

Performance. Table 1 shows the results on ImageNet-{R, S}, VTAB, and Office-Home, averaged over datasets within each benchmark. Despite its simplicity, calibration effectively boosts the FT model's accuracy on the absent classes, achieving comparable accuracy to the SOTA method proposed in [49].

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

Absent Class Data Accuracy

ImageNet -> ImageNet-R (50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

On ImageNet-{R, S}, both calibration approaches outperform the SOTA by a notable margin.

Compatibility of calibration. Post-processing calibration is potentially applicable to any model. In Figure 7, we extend the calibration approach to the pre-trained model and the SOTA model [49] with varying γ . Calibration can effectively balance Acc S / Y and Acc U / Y and trade one for the other. Interestingly, we find the curve of the FT model, in most cases, can well cover those of the other models. To characterize this, we follow [4]

118

218

398

Num of Fine-tuning Class

Random

Hierarchy

0.27

0.30

0.33

0.12

0.16

0.18

ImageNet-S [ AUSUC Diff.]

118

218

398

Num of Fine-tuning Class

Random

Hierarchy

15.59

19.95

23.85

5.48

9.30

12.66

ImageNet-S [ Acc

U

/

U

Diff.]

5

15

25

Num of Fine-tuning Class

Random

Close

0.02

0.09

0.13

0.07

0.11

0.15

Office-Home [ AUSUC Diff.]

5

15

25

Num of Fine-tuning Class

Random

Close

-2.61

1.67

3.07

-7.40

-2.64

1.06

Office-Home [ Acc

U

/

U

Diff.]

-0.5

-0.2

0.0

0.2

0.5

-20.0

-10.0

0.0

10.0

20.0

-0.2

-0.1

0.0

0.1

0.2

-5.0

-2.5

0.0

2.5

5.0

to calculate the Area Under the Seen-Unseen ((Fine-tuning / Absent)) Curve (AUSUC) to summarize the overall performance of each model across the entire spectrum of γ . Table 2 reports the AUSUC. The FT model notably outperforms the other methods, showing its robustness in learning from a subset of classes.

Remark. Many post-processing calibration methods have been proposed (cf. section 2). Our study is not meant to compare them, but to show that calibration can effectively address the issue caused by fine-tuning. We also note that Tu et al . [49] has investigated many solutions, including weight interpolation [57], but found them less effective them their proposed SOTA method.

6 Ablation Study and Additional Analysis

Data split and fine-tuning class size. In the default setup of [49], fine-tuning classes and absent classes are uniformly randomly sampled and have each portion close to 50% . In practice, however, end-users may have a smaller number of classes at hand or collect data from semantically or conceptually similar classes whose appearances are positively correlated. To explore such practical situations, we investigate how 1) a biased sampling such that the fine-tuning classes are conceptually or distributionally similar to each other than the absent classes and 2) a smaller size of fine-tuning classes would impact the performance of fine-tuning. Specifically, for Office-Home, classes that are similar in the pre-trained model's feature space are selected as fine-tuning classes, leaving the rest as absent ones. We also vary the number of fine-tuning classes. For ImageNet-S, we leverage the WordNet [11] hierarchy to select coherent groups, such as dog (118 classes) and mammal (218 classes) as the fine-tuning classes. Please see Appendix A for details.

Figure 8 illustrates the performance gain on AUSUC and NCM Acc U / Y (from the pre-trained model to the FT model). Notably, the hierarchical split poses a greater challenge for fine-tuning in transferring domain knowledge from fine-tuning classes to absent classes, as evidenced by its relatively minor AUSUC improvements compared to the random split. This difficulty is attributed to the inherent challenge of transferring features learned from dogs or animals to distinctly different classes like TVs and trucks. Furthermore, smaller fine-tuning class sizes present additional difficulties, leading to less pronounced improvements, especially in Office-Home. In summary, our analysis suggests that the benign behaviors of fine-tuning are robust in more practical and difficult splits but its performance requires further improvement when the fine-tuning class size is exceptionally small. Detailed experimental setup and additional results are available in Appendix D.

Optimizer. Prior work [49] predominantly used the SGD optimizer. To investigate optimizers' influence in fine-tuning, we scrutinize six popular optimizers including SGD with Momentum, Adam [23], AdaBelief [67], Adadelta [63], AdaGrad [10] and RMSprop. Our study includes 1) a variation across learning rates (LR), adjusted as multipliers of the default ones, i.e ., [10 , 1 , 0 . 1 , 0 . 01] × default LR, and 2) three distinct weight decays: [0 , 5 e -4 , 5 e -3] . The AUSUC, as illustrated in Figure 9, reveal the robustness of the benign behaviors to different hyperparameter settings of the SGD optimizer. Conversely, more advanced, adaptive optimizers show higher sensitivity when hyperparameters are not properly chosen. Nevertheless, under small enough learning rates (and weight decay), they perform similarly to SGD and notably improve the pre-trained model (whose AUSUC is around 0 . 5 in Office-Home).

Why is absent class relationship preserved? To elucidate the preservation of the absent class relationship, we analyze how classifier weights W change within the fine-tuning and absent classes. We commence by calculating the L2 normalized weight changes between fine-tuning and the pre-

0.01x

0.1x

1x

10x

Default Learning Rate

5e-3

5e-4

0

Weight Decay

0.57

0.62

0.63

0.58

0.57

0.62

0.63

0.61

0.57

0.62

0.63

0.61

SGD

0.01x

0.1x

1x

10x

Default Learning Rate

5e-3

5e-4

0

Weight Decay

0.62

0.11

0.04

0.02

0.63

0.36

0.04

0.02

0.63

0.61

0.24

0.04

Adam

0.01x

0.1x

1x

10x

Default Learning Rate

5e-3

5e-4

0

Weight Decay

0.63

0.62

0.47

0.02

0.63

0.63

0.59

0.09

0.63

0.63

0.58

0.07

AdaBelief

0.01x

0.1x

1x

10x

Default Learning Rate

5e-3

5e-4

0

Weight Decay

0.62

0.58

0.02

0.01

0.62

0.62

0.55

0.02

0.62

0.62

0.55

0.03

Adadelta

0.01x

0.1x

1x

10x

Default Learning Rate

5e-3

5e-4

0

Weight Decay

0.62

0.40

0.05

0.02

0.62

0.61

0.15

0.01

0.62

0.63

0.46

0.01

AdaGrad

0.01x

0.1x

1x

10x

Default Learning Rate

5e-3

5e-4

0

Weight Decay

0.02

0.02

0.01

0.01

0.02

0.02

0.01

0.01

0.59

0.27

0.03

0.02

RMSprop

0.3

0.4

0.5

0.6

0.7

0.3

0.4

0.5

0.6

0.7

0.3

0.4

0.5

0.6

0.7

0.3

0.4

0.5

0.6

0.7

0.3

0.4

0.5

0.6

0.7

0.3

0.4

0.5

0.6

0.7

0

17

34

51

68

85

102

119

136

153

170

187

204

221

238

255

272

289

306

323

340

357

374

391

408

425

442

459

476

493

0

16

32

48

64

80

96

112

128

144

160

176

192

208

224

240

256

272

288

304

320

336

352

368

384

400

416

432

448

464

480

496

Fine-tuning Classes Classifier Weight Change Relationship

0

17

34

51

68

85

102

119

136

153

170

187

204

221

238

255

272

289

306

323

340

357

374

391

408

425

442

459

476

493

0

16

32

48

64

80

96

112

128

144

160

176

192

208

224

240

256

272

288

304

320

336

352

368

384

400

416

432

448

464

480

496

Absent Classes Classifier Weight Change Relationship

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

trained model ∆ W = W T -W O ∥ W T -W O ∥ 2 . Visualizing the cosine similarity of the change within fine-tuning and absent classes, i.e . ∆ W Y (∆ W Y ) ⊤ and ∆ W U (∆ W U ) ⊤ , allows us to assess the directional similarities in classifier updates. As depicted in Figure 10, there exists a pronounced dissimilarity in the update directions among fine-tuning classes since the gradients aim to separate fine-tuning classes during fine-tuning. Conversely, a notable similarity in the update directions is observed among absent classes. The absence of positive signals from absent class data results in nearly uniform updates of absent classifiers, thereby preserving the class relationships among absent classes throughout fine-tuning. More results for other datasets can be found in Appendix D.

Absent Features and linear classifier Alignment in fine-tuning. section 4 has demonstrated that the alignment between absent features and the linear classifier ( i.e ., the FC layer) remains intact during fine-tuning, as evidenced by the stable or improved Acc U / U , despite the absence of absent class data. This indicates that fine-tuning retains, and even enhances, its capacity to differentiate among absent classes. To delve deeper into this phenomenon, we scrutinize the behavior of ground-truth class (GT) vs. the largest non-GT absent logits for absent test samples. As depicted in Figure 12, although both sets of logits exhibit a decline throughout the training process- attributable to the lack of absent data during fine-tuning-the relative difference between them persists, underlining a consistent and stable alignment between the features and the linear classifier. Future work may delve into the theoretical understanding of this observed alignment between features and classifiers for absent classes.

Why do absent class features improve after fine-tuning? During fine-tuning, the pre-trained model is updated solely by gradients derived from the fine-tuning class data. Surprisingly, the fine-tuned feature extractor does not forget but improves its ability to differentiate the absent class data. Here, we explain it by considering a two-layer linear neural network ˆ y = arg max c w ⊤ c ( Ux ) = arg max c w ⊤ c z . Let us denote by D = { ( x i , y i ∈ S ) } N i =1 a mini-batch during SGD, and denote by ∇ z i ℓ the gradient w.r.t. z i = Ux i using the cross-entropy loss. The gradient w.r.t. the feature extractor U is thus ∇ U ℓ = 1 N ∑ i ( ∇ z i ℓ ) x ⊤ i . Suppose we apply SGD with a learning rate η , the updated feature extractor is U ← U -η N ∑ i ( ∇ z i ℓ ) x ⊤ i . Building upon this formula, we can further

8.5

9.0

9.5

10.0

10.5

11.0

11.5

12.0

Feature 1

2

4

6

8

10

Feature 2

Target Data Movement in Fine-tuning Model

Fine-tuning Classes

Absent Classes

Pre-trained Features

Fine-tuned Features

2

6

10

14

18

22

26

30

34

38

42

46

50

Epochs

7.00

7.25

7.50

7.75

8.00

8.25

8.50

8.75

Average logits

Largest non-GT absent logits

GT absent logits

derive how the feature z = Ux of an absent class data x changes after the model update

Namely, the update of z is governed by its similar training examples - those x i with high inner products x ⊤ i x with x 2 -and their corresponding feature gradients ∇ z i ℓ . Suppose the domain shift affects similar classes similarly and the gradients w.r.t. the fine-tuning class data and features could effectively overcome the domain shift, Equation 6 offers a preliminary explanation of why fine-tuning could improve the absent class features in the downstream domain.

To further illustrate this, we design a toy example with four classes, visually depicted by different colors in Figure 11. Blue and cyan denote the fine-tuning class data; red and magenta denote the absent class data. The dimensionality of x and z are set to be 2 ; the size of U is thus 2 × 2 . We deliberately set x to be non-negative to simulate the output of a ReLU operation. We create the pre-training dataset and the fine-tuning dataset by performing local translations to the data. We then pre-train a two-layer multi-layer perceptron (MLP) on the pre-training data with four classes while keeping the first layer's weight ( i.e ., U ) frozen as an identify matrix to ease visualization ( i.e ., z = x ). After pre-training, we then fine-tune the model on the downstream fine-tuning data with only two classes without freezing U . (Please find more details about the data creation and the model architecture in Appendix A.) As shown in Figure 11, after fine-tuning, the update of the absent class features (from ◦ to ⋆ ) follows the update of their closest fine-tuning class features, even though the absent class data is not involved in fine-tuning. Moreover, different classes stay quite distinguishable in the fine-tuned feature space, suggesting that fine-tuning with a subset of classes would not degrade but improve the feature quality.

More analysis. Due to the page limit, we leave more analyses including frozen classifiers and frozen backbones in fine-tuning, the investigation of biased logits toward fine-tuning classes, and absent class relationship analysis in Appendix C.

7 Conclusion

'What happens if one fine-tunes a pre-trained classifier with a subset of classes?' Prior work showed that while it improves the fine-tuning class accuracy in the downstream domain, it drastically degrades the model's ability to recognize the other classes the model had previously learned. Our systematic study, however, provides a different opinion. We found that fine-tuning does not degrade but often improves the model's ability to recognize the other classes, if the classifiers' logits are well-calibrated. We expect our study to serve as a valuable reference for practical fine-tuning of pre-trained models.

Acknowledgments

This research is supported by grants from the National Science Foundation (ICICLE: OAC-2112606). We are grateful for the generous support of computational resources provided by the Ohio Supercomputer Center.

References

Sara Beery, Elijah Cole, and Arvi Gjoka. The iwildcam 2020 competition dataset. CoRR , arXiv preprint arXiv:2004.10340.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks , 106:249-259, 2018.

Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 52-68. Springer, 2016.

BB Chaudhuri. A new definition of neighborhood of a point in multi-dimensional space. Pattern Recognition Letters , 17(1):11-17, 1996.

Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. Advances in Neural Information Processing Systems , 32, 2019.

Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence , 44(7):33663385, 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research , 12(7), 2011.

Christiane Fellbaum. WordNet: An electronic lexical database . MIT press, 1998.

Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences , 3(4):128-135, 1999.

Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4367-4375, 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV , 2021.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 831-839, 2019.

Mobarakol Islam, Lalithkumar Seenivasan, Hongliang Ren, and Ben Glocker. Class-distributionaware calibration for long-tailed visual recognition. arXiv preprint arXiv:2109.05263 , 2021.

Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14 , pages 67-84. Springer, 2016.

Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09217 , 2019.

Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR) , 54(10s):1-41, 2022.

Byungju Kim and Junmo Kim. Adjusting decision boundary for class imbalanced learning. IEEE Access , 8:81674-81685, 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning , pages 3519-3529. PMLR, 2019.

Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Finetuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations , 2021.

Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends® in Computer Graphics and Vision , 16(1-2):1-214, 2024.

Dongyue Li and Hongyang Zhang. Improved regularization and robustness for fine-tuning in neural networks. Advances in Neural Information Processing Systems , 34:27249-27262, 2021.

Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning , pages 6028-6039. PMLR, 2020.

Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. arXiv preprint arXiv:2403.14735 , 2024.

Vincenzo Lomonaco, Lorenzo Pellegrini, Pau Rodriguez, Massimo Caccia, Qi She, Yu Chen, Quentin Jodelet, Ruiping Wang, Zheda Mai, David Vazquez, et al. Cvpr 2020 continual learning in computer vision competition: Approaches, results, current challenges and future directions. Artificial Intelligence , 303:103635, 2022.

Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Tianming Liu, Gao Cong, Yingjie Hu, et al. On the opportunities and challenges of foundation models for geospatial artificial intelligence. arXiv preprint arXiv:2304.06798 , 2023.

Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing , 469:28-51, 2022.

Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3589-3599, 2021.

Zheda Mai, Ping Zhang, Cheng-Hao Tu, Hong-You Chen, Li Zhang, and Wei-Lun Chao. Lessons learned from a unifying empirical study of parameter-efficient transfer learning (petl) in visual recognition. arXiv preprint arXiv:2409.16434 , 2024.

Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(5):55135533, 2022.

Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation , volume 24, pages 109-165. Elsevier, 1989.

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR , 2021.

Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature , 616(7956):259-265, 2023.

Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania. Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint arXiv:2308.13320 , 2023.

Tai-Yu Pan, Cheng Zhang, Yandong Li, Hexiang Hu, Dong Xuan, Soravit Changpinyo, Boqing Gong, and Wei-Lun Chao. On model calibration for long-tailed object detection and instance segmentation. Advances in Neural Information Processing Systems , 34:2529-2542, 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

José Salvador Sánchez, Filiberto Pla, and Francesc J Ferri. On the use of neighbourhood-based non-parametric classifiers. Pattern Recognition Letters , 18(11-13):1179-1186, 1997.

Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Generalized zero-and few-shot learning via aligned variational autoencoders. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8247-8255, 2019.

Yujun Shi, Jian Liang, Wenqing Zhang, Vincent YF Tan, and Song Bai. Towards understanding and mitigating dimensional collapse in heterogeneous federated learning. ICLR, 2023.

Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Online class-incremental continual learning with adversarial shapley value. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9630-9638, 2021.

Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. Gcr: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 99-108, 2022.

Cheng-Hao Tu, Hong-You Chen, Zheda Mai, Jike Zhong, Vardaan Pahuja, Tanya BergerWolf, Song Gao, Charles Stewart, Yu Su, and Wei-Lun Chao. Holistic transfer: Towards non-disruptive fine-tuning with partial target data. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Cheng-Hao Tu, Zheda Mai, and Wei-Lun Chao. Visual query tuning: Towards effective usage of intermediate representations for parameter and memory efficient transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7725-7735, 2023.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5018-5027, 2017.

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems , pages 10506-10518, 2019.

Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens Van Der Maaten. Simpleshot: Revisiting nearest-neighbor classification for few-shot learning. arXiv preprint arXiv:1911.04623 , 2019.

Zhenyi Wang, Enneng Yang, Li Shen, and Heng Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. arXiv preprint arXiv:2307.09218 , 2023.

Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7959-7971, 2022.

Tong Wu, Ziwei Liu, Qingqiu Huang, Yu Wang, and Dahua Lin. Adversarial robustness under long-tailed distribution. In CVPR , 2021.

Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 374-382, 2019.

Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du. Parameter-efficient fine-tuning for pre-trained vision models: A survey. arXiv preprint arXiv:2402.02242 , 2024.

Han-Jia Ye, Hong-You Chen, De-Chuan Zhan, and Wei-Lun Chao. Identifying and compensating for feature deviation in imbalanced deep learning. arXiv preprint arXiv:2001.01385 , 2020.

Han-Jia Ye, Hexiang Hu, and De-Chuan Zhan. Learning adaptive classifiers synthesis for generalized few-shot learning. International Journal of Computer Vision , 129(6):1930-1953, 2021.

Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 , 2012.

Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867 , 2019.

Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13208-13217, 2020.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16816-16825, 2022.

Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems , 33:18795-18806, 2020.

Appendix

We provide details omitted in the main paper.

Appendix A: experiment and dataset details

Appendix B: additional details for calibration

Appendix C: additional analysis of fine-tuning

Appendix D: detailed results of different architectures, datasets, and splits.

Our study is built upon [49]. Tu et al . [49] named the above the setting Holistic Transfer (HT). For ease of reference, we use the following terms interchangeably.

fine-tuning classes & seen classes; absent classes & unseen classes;

downstream domain & target domain; pre-training domain & source domain;

fine-tuning & naive fine-tuning.

We emphasize that the 'unseen' classes are indeed seen in pre-training but absent in fine-tuning.

A Experiment and Dataset Details

A.1 Main Investigation (cf. section 3.1 in the main paper)

A.1.1 Dataset Details

ImageNet-Variants includes ImageNet-R(endition) [15] and ImageNet-S(ketch) [54]. ImageNetR comprises 30,000 images across 200 ImageNet classes with various renditions ( e.g ., paintings, embroidery, etc .) while ImageNet-S consists of 50,000 sketch-like images for 1K ImageNet classes. Each class is randomly divided into training and testing sets following an 8:2 split. 50% of the classes are randomly selected as fine-tuning classes (100 for ImageNet-R and 500 for ImageNet-S), with the remainder as absent. The downstream test set encompasses all the 200 classes for ImageNet-R and 1K classes for ImageNet-S to evaluate model performance across the full class spectrum.

Office-Home [53] is a popular domain adaptation dataset, comprising 65 classes from 4 domains (Art, Clipart, Real, and Product). Following the setup in [49], Art and Real are used as pre-trained domains and each pre-trained model is then transferred to each of the three remaining downstream domains individually, resulting in six (pre-training, downstream) pairs. Within each downstream domain, each class is randomly split into training and testing sets following a 7:3 split. 30 classes are randomly selected as fine-tuning classes; the remaining 35 classes are absent classes. The downstream test set contains all the 65 classes in each downstream domain.

VTAB [64] encompasses a diversity of image classification tasks. To enable zero-shot predictions with CLIP, Tu et al . [49] only uses the tasks that provide text names for classes: Caltech101, CIFAR100, DTD, EuroSAT, Flowers102, Pets, Resisc45, SVHN, and SUN397. Notably, the SVHN dataset was excluded from our experiments due to CLIP's difficulty in accurately predicting numerical labels as shown in [49]. Adhering to the practice in [49], we randomly sample half of the classes as tine-tuning and the remaining as absent. The downstream training set only includes images of the fine-tuning classes; the test set contains images from all classes.

Table 3 summarizes the statistics of all the datasets used in this paper.

A.1.2 Training Details

For the ImageNet-Variants benchmark, we use an ImageNet-1K pre-trained ResNet-50 (results in the main paper) and ViT-B/32 (results in the appendix) as pre-trained models. The pre-trained model is fine-tuned on downstream tasks for 50 epochs using the SGD optimizer with a learning rate 1e-3, momentum 0.9, weight decay 1e-4, and batch size 64. For the compared method proposed in [49], we set the hyper-parameters L distill = 10 and L rank = 100 for ImageNet-R and L distill = 1 and L rank = 5 for ImageNet-S.

For the VTAB benchmark, we use the ViT-B/32 CLIP models as the backbone. Specifically, we extract the class name embedding to construct the FC layer and disregard the CLIP text encoder afterward. The pre-trained model is fine-tuned on downstream tasks for 20 epochs using the SGD optimizer with a learning rate 1e-5, momentum 0.9, weight decay 0.0, and batch size 64. For the compared method proposed in [49], we follow their paper and set the hyper-parameters L distill = 1 and L rank = 5 .

For the Office-Home dataset, we follow the training recipe of [49]. Firstly, the pre-trained classifier is obtained by fine-tuning an ImageNet-1K pre-trained ResNet-50 with the source domain data for 20 epochs using the SGD optimizer with a learning rate 1e-3, momentum 0.9, weight decay 5e-4, and batch size 64. After that, we fine-tuned the resulting model on each target domain for 20 epochs using the SGD optimizer with a learning rate 1e-4, momentum 0.9, weight decay 5e-4, and batch size 64. For the compared method proposed in [49], we follow their paper and set the hyper-parameters L distill = 10 and L rank = 100 .

Computational resources. We use a combination of NVIDIA RTX A6000 and NVIDIA 2080Ti GPUs. Since we worked on fine-tuning, the computation is quite manageable.

A.2 Ablation Study (cf. section 5 in the main paper)

In section 5 of the main paper, we investigate how 1) a biased sampling of fine-tuning and absend classes and 2) a smaller size of fine-tuning classes would impact the performance of fine-tuning. Here, we provide details of how we split the data.

In the Office-Home dataset, we strategically select fine-tuning classes that are similar in the pretrained model's feature space, aiming to create a meaningful distinction between fine-tuning and absent classes. To identify classes that are closely related, we apply the metric of total intra-group distance: a lower value indicates higher similarity among classes within a group. This ensures finding fine-tuning classes that exhibit tight clustering in the feature space. We employ a greedy strategy to grow the fine-tuning class set towards a pre-defined size. Figure 13 ad Figure 14 present t-SNE visualizations that illustrate the distribution of chosen fine-tuning and absent classes for varying fine-tuning class sizes across two distinct pre-trained domains.

For the ImageNet-S dataset, we leverage the WordNet [11] hierarchy to select coherent groups as fine-tuning classes. Specifically, we explore two different hierarchical splits. The details are shown in Table 4.

15

10

5

0

5

10

10

5

0

5

10

15

Exit_Sign

Bottle

Glasses

Computer

File_Cabinet

Shelf

Toys

Sink

Laptop

Kettle

Folder

Keyboard

Flipflops

Pencil

Bed

Hammer

ToothBrush

Couch

Bike

Postit_Notes

Webcam

Desk_Lamp

Telephone

Helmet

Pen

Monitor

Mop

Sneakers

Notebook

Backpack

Alarm_Clock

Push_Pin

Paper_Clip

Batteries

Radio

Fan

Ruler

Pan

Screwdriver

Trash_Can

Printer

Speaker

Eraser

Bucket

Chair

Calendar

Calculator

Flowers

Lamp_Shade

Spoon

Candles

Clipboards

Scissors

TV

Curtains

Fork

Soda

Table

Knives

Oven

Refrigerator

Marker

(A) Office-Home pre-trained domain: Art

15

10

5

0

5

10

10

5

0

5

10

15

Fine-tuning Class Size: 5

15

10

5

0

5

10

10

5

0

5

10

15

Fine-tuning Class Size: 15

15

10

5

0

5

10

10

5

0

5

10

15

Fine-tuning Class Size: 25

(B) Splits for different fine-tuning class sizes (red for fine-tuning classes and blue for absent classes)

10

5

0

5

10

15

10

5

0

5

10

Exit_Sign

Bottle

Glasses

Computer

File_Cabinet

Shelf

Toys

Sink

Laptop

Kettle

Folder

Keyboard

Flipflops

Pencil

Bed

Hammer

ToothBrush

Couch

Bike

Postit_Notes

Webcam

Desk_Lamp

Telephone

Helmet

Pen

Monitor

Mop

Sneakers

Notebook

Backpack

Alarm_Clock

Push_Pin

Paper_Clip

Batteries

Radio

Fan

Ruler

Pan

Screwdriver

Trash_Can

Printer

Speaker

Eraser

Bucket

Chair

Calendar

Calculator

Flowers

Lamp_Shade

Spoon

Candles

Clipboards

Scissors

TV

Curtains

Fork

Soda

Table

Knives

Oven

Refrigerator

Marker

(A) Office-Home pre-trained domain: Real-World

10

5

0

5

10

15

10

5

0

5

10

Fine-tuning Class Size: 5

10

5

0

5

10

15

10

5

0

5

10

Fine-tuning Class Size: 15

10

5

0

5

10

15

10

5

0

5

10

Fine-tuning Class Size: 25

(B) Splits for different fine-tuning class sizes (red for fine-tuning classes and blue for absent classes)

8.5

9.0

9.5

10.0

10.5

11.0

11.5

Feature 1

2

3

4

5

6

7

8

Feature 2

Pre-trained Feature

Fine-tuning Feature

A.3 Toy Example (cf. section 6 in the main paper)

To elucidate the impact of similar fine-tuning training data on the feature representation of absent data, in section 6 and figure 12 of the main paper, we construct a toy example featuring four classes of 2-dimensional data, each represented by distinct colors (blue, cyan, red, magenta). Figure 15 shows the same data. In this example, pre-training data ( □ ) are generated from Gaussian distributions with a standard deviation of 0.2 and four different means: (10, 2), (10, 3), (10, 8), and (10, 7). To simulate domain shift, the target domain data ( ⃝ ) undergoes a horizontal shift, with the cyan and magenta classes moving to the right and the blue and red classes to the left by an identical distance. This setup, intentionally restricting data to non-negative values, mirrors the effect of a ReLU activation.

We employ a two-layer multi-layer perceptron (MLP) with a configuration of 2-2-4 (input dimension: 2, hidden layer dimension: 2, and output dimension: 4). The MLP is initially pre-trained on the four-class pre-training dataset, with the first layer weights set as an identity matrix to simplify visualization. Subsequent fine-tuning on target domain data incorporates only two classes (blue and cyan), without the constraint of freezing the first layer. Both the pre-training and fine-tuning phases utilize the SGD optimizer, applying a learning rate of 0.01 for 100 epochs with cross-entropy loss as the objective.

B Additional Details for Calibration (cf. section 4 in the main paper)

We provide details for the calibration methods in section 4 of the main paper.

B.1 Average Logit Gap (ALG)

In a well-calibrated pre-trained model, the scale of average non-Ground-Truth (non-GT) logits is expected to be similar across different classes. Figure 16 demonstrates that using the pre-trained

model, the non-GT logits exhibit comparable magnitudes for both fine-tuning and absent class groups. However, the fine-tuning class group's non-GT logits become notably higher after fine-tuning, since fine-tuning tends to assign much higher logits to see classes due to the absence of absent data. This disparity inspires an estimation of γ based on the non-GT logit differences between fine-tuning and absent classes, as delineated in Equation 7. Concretely, for each training example, we calculate 1) the average of non-GT fine-tuning class logits and 2) the average of non-GT absent class logits. We then average the difference between them over all the training examples as an estimate of γ . The calculation is also visually summarized in Figure 17. Since this approach is based on the average non-GT logit difference between fine-tuning and absent classes, we call this method the Average Logit Gap (ALG).

̸

̸

Fi ne- t uni ng  cl as s es

Absent  cl asses

Pr e- t r ai ned  M odel

Fi ne- t uni ng  M odel

2

0

-2

B.2 Pseudo Cross-Validation (PCV)

To address the challenge of selecting γ without access to target validation data, which ideally encompasses both fine-tuning and absent classes, we introduce a novel Pseudo Cross-Validation (PCV) method, demonstrated in Figure 18. Specifically, we partition the target training dataset D tr into pseudo training data D pseudo-tr and pseudo validation data D pseudo-val . D pseudo-tr is further divided into two subsets with disjoint label spaces, simulating pseudo-fine-tuning data D pseudo-ft and pseudo-absent data D pseudo-absent. We then naively fine-tune the pre-trained model on D pseudo-absent and evaluate Acc S / Y and Acc U / Y using D pseudo-val. We select a γ by balancing these two accuracies. To enhance the robustness of γ estimation, we employ bootstrapping, repeating the pseudo-splitting and fine-tuning process three times with varying partitions. The selected γ is applied to the fine-tuning model { θ T , W T } , which is fine-tuned from the pre-trained model { θ O , W O } on the entire target training data D tr .

C Additional Analysis of Fine-tuning (cf. section 3.4 and section 6 in the main paper)

In this section, we provide more analysis including an analysis of feature improvements, an investigation of the tendency for logits to be biased toward fine-tuning classes, and the effects of freezing the classifier and backbone during fine-tuning. Additionally, we quantitatively examine the class relationship change between the pre-trained and fine-tuning models. Finally, we extend our analysis to examine the impact of fine-tuning on the iWildCam dataset [1].

C.1 Feature Improvement by the Fine-Tuned Model

tiger

zebra

cat

horse

dog

(absent)

(pre-training)

AVG

→

AVG

0

4

8

12

16

20

Training epochs

0.775

0.800

0.825

0.850

0.875

0.900

0.925

0.950

0.975

Magnitude

OfficeHome-Variants Classifer L2 Norm

Seen

Unseen

Power

.tcpdf.org)

Fine-tuning

Absent

Power

.tcpdf.org)

(A) Office-Home

0

10

20

30

40

50

Training epochs

1.48

1.49

1.50

1.51

1.52

1.53

1.54

1.55

1.56

Magnitude

ImageNet-Variants Classifer L2 Norm

Seen

Unseen

Power d by TCPDF (w

.tcpdf.org)

Fine-tuning

Absent

Power d by TCPDF (w

.tcpdf.org)

(B) ImageNet-Variants

To further understand the improved NCM absent class accuracy NCMAcc U / Y ( f θ T ) , we plot the t-SNE embedding [51] in Figure 19. We find that the FT feature extractor f θ T does not superficially raise Acc U / Y by creating an artificial 'margin' between the fine-tuning and absent class features 3 and destroying their semantic relationship. We further report the NCM accuracy of classifying absent class data into one of the absent classes, i.e ., Acc U / U with B = U . As summarized in Table 5, FT indeed improves the feature quality to distinguish among absent classes.

C.2 Biased Logits Towards fine-tuning Classes

section 4 of the main paper highlights how fine-tuning generates logits biased towards fine-tuning classes due to the lack of absent data during training. To delve deeper into this, we analyze the L2 norm of the classifier weights W T for Office-Home and ImageNet-Variants. Figure 20 reveals a pronounced increase in the weight magnitude for fine-tuning classes relative to absent classes during fine-tuning, which could potentially result in fine-tuning's biased logits towards fine-tuning classes.

To determine whether the magnitude of classification weights is solely responsible for the biased logits, we apply the cosine classifier [13], which normalizes the weights to ensure uniform magnitude across all classes. According to the results shown in Table 6, employing a cosine classifier does yield a slight improvement in absent class accuracy (Acc U / Y ); however, the outcome remains suboptimal. This indicates that both the magnitude of the linear classifiers and their angles ( i.e ., cosine similarity) with the features contribute to the biased logits. These findings lay the groundwork for our proposed calibration approach, which seeks to directly adjust the logits, bypassing the limitations associated with modifying either the magnitude or the angle of the classification weights.

C.3 Should We Freeze the Linear Classifier or Feature Backbone?

Adhering to the practice in source-free DA [29], Tu et al . [49] advocate for freezing the linear classifier W O during fine-tuning (termed the frozen classifier approach) to preserve the absent class relationship and hence accuracy Acc U / Y . Surprisingly, our analysis reveals that while preserving class relationships within the FC layer, the frozen classifier leads to a worse AUSUC score and NCM absent accuracy compared to fine-tuning, as evidenced in Table 8. This suggests a deterioration in the feature quality of the frozen classifier.

We hypothesize that the underlying issue arises from fine-tuning with only fine-tuning class data, which inadvertently biases the model towards classifying all samples as fine-tuning classes. While fine-tuning can adapt to this bias by adjusting the FC layer to reduce absent logits, the frozen classifier must alter its features to achieve such a goal, potentially compromising the quality of absent features.

We also investigate freezing the feature extractor backbone θ O and only fine-tuning the FC layer (a.k.a. linear probing). Since the feature representations remain unchanged, the NCM Acc U / Y mirrors that of the pre-trained model, indicating no improvement in feature quality, as evidenced in Table 8. As a result, it achieves a lower NCM accuracy and lower AUSUC score than fine-tuning. In sum, neither the frozen classifier nor the frozen backbone are effective strategies in HT.

C.4 Absent Class Relationship Analysis

Inspired by the finding in [19]-the similarity among vectors in the FC layer can reflect the semantic relationships between their corresponding classes-we investigate if such class relationships are maintained during fine-tuning. To quantitatively capture the class relationships, we compute the cosine similarity between each pair of vectors in the FC layer. Specifically, the matrices W S O ( W S O ) ⊤ and W U O ( W U O ) ⊤ capture the class relationships among fine-tuning and absent classes, respectively, within the pre-trained model. Similarly, W S T ( W S T ) ⊤ and W U T ( W U T ) ⊤ reflect these relationships in fine-tuning. Here, W represents the L2-normalized linear classification weights, with superscripts indicating whether the weights are for fine-tuning S or absent U classes, and subscripts differentiating between the pre-trained and the fine-tuning model.

To assess how fine-tuning affects class relationships, we employ the linear Centered Kernel Alignment (CKA) [25], to compare the class relationships before and after fine-tuning. Specifically, we compute CKA scores for fine-tuning and absent classes:

Higher CKA scores signify a more robust preservation of class relationships through the transition from a pre-trained model to fine-tuning. As demonstrated in Table 7, class relationships among absent classes are substantially more preserved than those among fine-tuning classes. This observation aligns with insights discussed in section 6 of the main paper. The distinction arises because fine-tuning with fine-tuning class data prompts the classifier to differentiate between fine-tuning classes more distinctly. In contrast, without direct training signals for absent classes, updates to absent classifiers tend to be more uniform, thus maintaining the original class relationships among absent classes throughout the fine-tuning process.

D More Results

D.1 ImageNet-Variants with ViT

In the main paper, our findings for ImageNet-Variants were based on experiments with a ResNet50 model pre-trained on ImageNet-1K. To broaden our analysis, this section reports on extended

experiments that employ the ImageNet-1K pre-trained Vision Transformer (ViT-B/32) as an alternative pre-trained model. Our objective is to ascertain whether the observations regarding fine-tuning remain consistent when applied to a larger and more advanced model architecture.

Results, as detailed in Table 9 and Table 10, indicate that both the ResNet-50 and ViT-B/32 architectures yield consistent improvements across AUSUC, Acc U / U , and NCM metrics. This consistency underscores the inherent robustness of fine-tuning's benign behaviors, independent of the model architecture. Notably, the ViT-B/32 model exhibits a significantly milder decline in absent class accuracy Acc U / Y compared to ResNet-50. This finding suggests that the ViT-B/32 model is more robust in partial target data fine-tuning.

D.2 Detailed Results for Each Dataset

In the main paper, due to space constraints, we summarized the findings by presenting average performance metrics across all datasets within each benchmark. This section expands upon that summary by providing in-depth results for each pretraining-downstream domain pair within the OfficeHome dataset and for individual datasets in VTAB, detailed in Table 11 and Table 12, respectively. Across the board, fine-tuning exhibits consistent enhancements in AUSUC scores, reinforcing its efficacy in diverse settings. In ImageNet-Variants, fine-tuning consistently improves the absent class feature (an increase of NCM absent accuracy Acc U / Y ) as mentioned in subsection D.1. In the context of Office-Home, while the majority of domain pairs witnessed improvements in NCM Acc U / Y , exceptions such as Rw-Pr and Ar-Rw experienced a decline. Some exceptions are also observed within VTAB, such as Flower102 and SUN397. These findings suggest that certain datasets may benefit from more sophisticated approaches to bolster absent class features.

D.3 The Fine-tuning / Absent Accuracy Curve For All Datasets

Owing to the space constraints within the main document, our presentation of the Fine-tuning/ Absent Accuracy Curve was limited to ImageNet-R and the Art-Product domain pair from Office-Home. This section expands our analysis by presenting the Fine-tuning/ Absent Accuracy Curves for all datasets within each benchmark. This comprehensive display aims to offer a more detailed understanding of the performance dynamics between fine-tuning and absent classes across the diverse range of datasets evaluated in our study.

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

Absent Class Data Accuracy

ImageNet -> ImageNet-R (50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(A) ImageNet-R

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

Absent Class Data Accuracy

ImageNet -> ImageNet-S (50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(B) ImageNet-S

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

Absent Class Data Accuracy

OfficeHome Ar -> Cl (~50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

70

80

Absent Class Data Accuracy

OfficeHome Ar -> Pr (~50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(B) Art-Product

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

70

80

Absent Class Data Accuracy

OfficeHome Ar -> Rw (~50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(C) Art-Real World

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

70

80

Absent Class Data Accuracy

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(D) Real World-Art

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

Absent Class Data Accuracy

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(E) Real World-Clipart

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

70

80

90

Absent Class Data Accuracy

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(F) Real World-Product

0

20

40

60

80

100

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

70

80

90

Absent Class Data Accuracy

CLIP -> VTAB-Caltech101 (50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(A) Caltech101

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

70

80

Absent Class Data Accuracy

CLIP -> VTAB-Cifar100 (50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(B) CIFAR100

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

Absent Class Data Accuracy

CLIP -> VTAB-DTD (50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(C) DTD

0

20

40

60

80

100

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

Absent Class Data Accuracy

CLIP -> VTAB-Eurosat (50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(D) EuroSAT

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

70

Absent Class Data Accuracy

CLIP -> VTAB-Flowers102 (50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(E) Flowers102

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

70

80

90

Absent Class Data Accuracy

CLIP -> VTAB-Pets (50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(F) Pets

0

20

40

60

80

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

70

Absent Class Data Accuracy

CLIP -> VTAB-Resisc45 (50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(G) Resisc45

0

20

40

60

Fine-tuning Class Data Accuracy

0

10

20

30

40

50

60

Absent Class Data Accuracy

CLIP -> VTAB-Sun397 (50% Fine-tuning)

Pre-training

Fine-tuning

Pre-training +

Fine-tuning +

(H) Sun397

10

20

40

80

100

Num of Fine-tuning Class

ResNet-50

ViT-B/32

0.26

0.29

0.32

0.39

0.37

0.25

0.30

0.35

0.39

0.39

ImageNet-R [ AUSUC

]

10

20

40

80

100

Num of Fine-tuning Class

ResNet-50

ViT-B/32

9.63

13.39

15.70

17.77

18.95

3.23

8.86

12.41

15.14

13.53

ImageNet-R [ NCM Acc

/

]

-0.5

-0.2

0.0

0.2

0.5

-20.0

-10.0

0.0

10.0

20.0

118

218

398

Num of Fine-tuning Class

Random

Hierarchy

0.27

0.30

0.33

0.12

0.16

0.18

ImageNet-S [ AUSUC

]

118

218

398

Num of Fine-tuning Class

Random

Hierarchy

15.10

17.39

18.99

11.03

13.32

14.63

ImageNet-S [ Acc

/

]

5

15

25

Num of Fine-tuning Class

Random

Closed

0.02

0.09

0.13

0.07

0.11

0.15

Office-Home [ AUSUC

]

5

15

25

Num of Fine-tuning Class

Random

Closed

-0.84

1.03

1.49

-2.10

-0.70

0.95

Office-Home [ Acc

/

]

-0.5

-0.2

0.0

0.2

0.5

-20.0

-10.0

0.0

10.0

20.0

-0.2

-0.1

0.0

0.1

0.2

-3.0

-1.5

0.0

1.5

3.0

D.4 More Split Results

In the main manuscript, due to limitations on space, we restricted our presentation of data splits and fine-tuning class size-related ablation studies to ImageNet-S and Office-Home. This section extends our analysis to encompass ImageNet-R, as well as additional hierarchical splits for ImageNet-S, utilizing two distinct ImageNet-1K pre-trained models: ResNet50 and ViT-B/32.

Figure 24 delineates the outcomes for ImageNet-R where different fine-tuning class sizes are randomly chosen. Furthermore, we delve into hierarchical splits based on WordNet within ImageNet-S. Figure 25 reveals the findings from the animal hierarchical split, which encompasses dogs (118 classes), mammals (218 classes), and broader animal categories (398 classes) as fine-tuning classes. Similarly, Figure 26 presents the results from the device hierarchical split, involving devices (124 classes), instrumentality (350 classes), and artifacts (522 classes) as fine-tuning classes. Across different datasets, splits, and model backbones, the enhancements attributed to fine-tuning manifest as robust and consistent.

D.5 Classifier Update Direction Similarity

Due to page limitations in Section 6 of the main paper, our discussion on the similarity of classifier update directions was confined to ImageNet-S. To further substantiate the universality of our findings, this section extends the analysis to the Office-Home dataset, featuring the Art-Real World domain

124

350

522

Num of Fine-tuning Class

Random

Hierarchy

0.27

0.32

0.34

0.21

0.22

0.22

ResNet-50 [ AUSUC

]

124

350

522

Num of Fine-tuning Class

Random

Hierarchy

16.26

18.95

18.64

13.07

13.23

14.08

ResNet-50 [ NCM Acc

/

]

124

350

522

Num of Fine-tuning Class

Random

Hierarchy

0.28

0.32

0.33

0.24

0.24

0.24

ViT-B/32 [ AUSUC

]

124

350

522

Num of Fine-tuning Class

Random

Hierarchy

15.18

18.31

18.33

16.44

16.83

17.36

ViT-B/32 [ NCM Acc

/

]

-0.5

-0.2

0.0

0.2

0.5

-20.0

-10.0

0.0

10.0

20.0

-0.5

-0.2

0.0

0.2

0.5

-20.0

-10.0

0.0

10.0

20.0

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

Fine-tuning Classes Classifier Weight Change Relationship

0

2

4

6

8

10 12 14 16 18 20 22 24 26 28 30 32 34

0

2

4

6

8

34 32 30 28 26 24 22 20 18 16 14 12 10

Absent Classes Classifier Weight Change Relationship

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

pair, as detailed in Figure 27. This pattern of update direction similarity is similarly evident across other pretraining-downstream pairings within the Office-Home dataset.

D.6 Using Target Test Set For γ Selection

Theoretically, utilizing the target test set for γ selection contradicts standard practices, as it introduces data leakage that can be considered as cheating. However, for comparison purposes, we leverage the target test set to select γ , thereby denoting fine-tuning + γ ⋆ as an upper bound for calibration methods. As indicated in Table 13, while our introduced calibration methods, ALG and PCV, demonstrate substantial improvements over fine-tuning, they do not fully reach the performance of fine-tuning + γ ⋆ . This gap underscores a direction for future research to focus on more sophisticated calibration strategies for fine-tuning.

D.7 iWildCam

Another realistic benchmark proposed in [49] is iWildCam, which considers abundant camera trap images from various geographical locations as pre-trained domains, with images from a new camera trap location serving as the target domain. The uniqueness of this benchmark lies in the use of data collected within a limited time frame (e.g., the first month) at the new location as the target training set. Given the unlikely presence of all animal species within this initial period, the target training data inherently exhibits a class bias towards the species that are detected.

Contrary to other benchmarks, as depicted in Table 14, fine-tuning shows a decline in both AUSUC and Acc U / U . This performance drop is attributed to the natural data collection bias caused by time in iWildCam. Unlike conventional benchmarks that assume training and test sets come from the same distribution, iWildCam's unique structure-where training and testing sets originate from different

time periods-introduces a natural discrepancy in data distribution due to seasonal changes or animal migration.

To investigate this further, we selected four target locations from the benchmark and identified the pre-trained and test data corresponding to the same target training classes for each location. By extracting features using a pre-trained model, we aimed to determine whether significant changes occurred within the same classes across pre-trained data, target training data, and target testing data. The findings, illustrated in Figure 28, reveal notable differences in the data distributions among the same classes across the pre-trained domain, target training, and target testing sets. This discrepancy introduces further complexity in HT. Addressing this challenge, future research may focus on enhancing fine-tuning with strategies capable of bridging the data distribution gap.

60

40

20

0

20

40

60

80

80

60

40

20

0

20

40

60

80

Target Location 289

Pre-training

Fine-tuning train

Fine-tuning test

60

40

20

0

20

40

60

60

40

20

0

20

40

60

Target Location 295

Pre-training

Fine-tuning train

Fine-tuning test

80

60

40

20

0

20

40

60

80

75

50

25

0

25

50

75

Target Location 287

Pre-training

Fine-tuning train

Fine-tuning test

100

75

50

25

0

25

50

75

100

100

75

50

25

0

25

50

75

100

Target Location 310

Pre-training

Fine-tuning train

Fine-tuning test

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper's main contribution is a systematic study of fine-tuning. Both the abstract and the introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide multiple remarks to describe the scope of the study. We also conduct further analysis ( e.g ., section 6) to analyze the limitation.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Not a theoretical paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provided detailed experimental setups in the main paper and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The data used in the paper is publicly accessible. Our code will be released in the near future.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provided detailed experimental setups in the main paper and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: No error bar is provided. We study fine-tuning on already pre-trained models.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, we discussed this in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper studied a fundamental machine-learning problem of fine-tuning. If successful, it will make fine-tuning much easier and more practical. To our knowledge, at the current stage, our paper does not make any additional negative societal impacts compared to existing papers on fine-tuning.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The data we used are all publicly available and we cited the original papers.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a simple post-training calibration technique for classifying missing classes after fine-tuning. For example, assuming the pre-trained model can classify 1000 classes and is fine-tuned on a subset of these classes from a different image domain, the proposed method improves the classification accuracy of the *absent classes* on this new domain. Specifically, the proposed method adds a calibration hyper-parameter to artificially boost the probability of predicting the absent classes.  To motivate the method, the paper investigates the quality of feature learning using the Nearest Class Mean classifier to isolate the cause of bad performance on the absent classes. While simple, the method shows good performance gain on multiple datasets.\n\nSTRENGTHS:\n* The proposed method is easy to implement and provides good performance gains. \n* The paper uses the NCM classifier to investigate the feature extractor's quality and isolate the linear classifier's influence after fine-tuning.  This methodology provides a clear motivation for the proposed method. \n*  The finding that fine-tuning does not completely destroy the features of absent classes is interesting. This insight can motivate further study, especially for improving the robustness of fine-tuning. \n* The paper provides a detailed ablation study showing the proposed method's strengths and limitations. \n  * The extent of performance gain hinges on the fine-tuning procedure. \n  * The distribution of absent classes also affects performance.\n\nWEAKNESSES:\n* **Statements are not precise**. In the abstract and introduction, the paper claims that a fine-tuned model does not forget the relationship among absent classes. However, this claim is not precise. As the paper points out, the extent of forgetting and degradation depends on the fine-tuning procedure. For example, an Adam optimizer with a larger learning rate can degrade the features of the absent class. This is consistent with the existing literature on the robustness of fine-tuning.  It's possible that the proposed technique only works well under moderate changes to the pre-trained model. \n\n* **The fine-tuning setting is limited**. The proposed method only works for a particular fine-tuning configuration under constrained assumptions. Specifically, the method assumes a classification task, and the pre-trained model can classify all fine-tuning classes, including absent ones. Therefore, the paper's claims on forgetting and feature learning are limited by its scope.\n\nQUESTIONS:\n* While the paper investigated several factors that can affect calibration effectiveness, it is unclear how we should decide when the calibration is useful. Could the authors consider dependency on the model's **intrinsic** properties? The external factors all may lead to a common intrinsic property. For example, maybe the fine-tuned model's deviation (in the weight space) can be an indicator.  For example, RMSprop and Adam are known to converge faster and potentially lead to a larger deviation from the pre-trained model?\n\nLIMITATIONS:\nThe paper does not have a potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors argue that fine-tuning doesn’t forget the features for classes not participating in it, but rather downscales their logits as a result of which the model ends up being overconfident for the fine-tuning classes. Counter-intuitively, the authors claim that fine-tuning also enhances the discriminative ability of the model for the classes not participating in fine-tuning. For this, the authors analyze that accuracy of the NCM classifier on the features of the model and show that it increases even for classes absent during fine-tuning. Then the authors show that the order of the absent classes is still in place amongst the set of absent classes, but the model becomes overconfident on the classes used for fine-tuning, thereby leading to drop in accuracy of absent classes. To fix this, the authors analyze simple post processing calibration methods and demonstrate recovery in the performance of absent classes, with some drop in performance of fine-tuning classes. The analysis is done on Imagenet-R, VTAB and Office-Home datasets.\n\nSTRENGTHS:\n1) The gains observed on calibrating the model are impressive, and it is interesting to see that merely by calibrating the model, the model’s performance on absent classes can improve to this extent.\n2) The motivation behind the post hoc calibration method is interesting and it is unexpected that just the confidence of the absent would lower down while preserving their relative order on performing fine-tuning.\n\nWEAKNESSES:\n1) An increase in NCM classifier’s accuracy need not necessarily mean that the model has become better in discriminating features. It only means that the features become closer to the corresponding class mean in l2 distance metric space. I think this argument is not concrete enough and requires more evidence.\n2) It is not clear, why a drop in accuracy is seen in the fine-tuning classes on using PVC as the post calibration method. Even in case of ALG, where there isn’t a significant drop, the accuracy of classes absent during fine-tuning is still significantly lower (e.g. on office home it drops by over 20%) than the pre-trained model. This suggests that authors claim on feature enhancement of absent classes might not be true. \n3) The authors propose to use training data / validation data to find the right threshold. This would mean access to the model as well as data. In such a scenario someone could rather easily fine-tune the model on absent classes as well, which would not require a lot of compute. Therefore, on a practical standpoint, it is not completely obvious how the proposed method benefits more than mere fine-tuning. It would be great if authors could compare the budget required for fine-tuning on absent classes vs post-hoc calibration to address this point to achieve similar performance. Although I agree that the observation itself is interesting, but analyzing the efficiency aspect could help the authors in making the claim on using post calibration methods stronger.\n\n4) I think that the analysis shown in Fig-9 needs more rigor. Further, I think the authors need not discuss this in the main paper, as it doesn't adds much. The learning rate used for different optimizers are same, but the learning rates for adam, adagrad and adagrad need to be scaled down to make a fair comparison with SGD.\n5) The analysis on why absent class features improve on fine-tuning is not rigorous and would encourage authors to not discuss this in the main paper. Most of the claims in this analysis seem low hanging and not sound enough. This hinders the readily of the paper currently.\n6) Similarly, figure-10 doesn't adds much to the storyline authors have presented in this work and it requires more rigor. I would suggest them to restructure the paper a bit and remove this analysis from the main paper.\n\nMinor comments:\nIn figure-5, the y-axis should be between 0-1 since it represents a probability.\n\nI would be happy to increase my scores if my concerns are sufficiently addressed.\n\nQUESTIONS:\nIt would be great if the authors can present their results on domainnet dataset [1].\nI request the authors to kindly address the questions in the weaknesses section.\n\n[1] https://paperswithcode.com/dataset/domainnet\n\nLIMITATIONS:\nYes, the authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

comments: SUMMARY:
It is commonly believed that fine-tuning zero-shot models on seen classes will lead to a decrease in performance on unseen classes.
In this paper, the authors systematically examine the issue that find that (1)  the fine-tuned feature extractor is not damaged: NCM improves the absent class accuracy without catastrophic forgettin (2) main factor that damages the FT model’s ability to correctly classify absent class examples is the biased logit values towards fine-tuning classes. (3) a simple post-processing calibration of logits, ie, offsets the seen classes could bring back the zero-shot performance of absent classes. Extensive experiments and analyses validate the claims of the authors.

STRENGTHS:
The paper is well-organized and the presentation is clear. The analyses are comprehensive and convincing. This study provides insights that corrected my previous viewpoint that fine-tuning causes the forgetting of knowledge of absent classes. This paper is undoubtedly a valuable work and worth accepting.

WEAKNESSES:
This is a solid paper, and I did not find any major weaknesses.

I suggest that the authors include more experiments on CLIP models, such as the base-to-new setting in CoCoOp, reporting AccY/Y, AccS/Y, and AccU/Y on 11 datasets.

QUESTIONS:
Please refer to the weaknesses.

LIMITATIONS:
No negative societal impact has been identified.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-09

comments: SUMMARY:
The paper unveils the improved features of absent classes when a pre-trained model is fine-tuned on a subset of all classes. The paper presents an empirical study on three datasets to demonstrate this finding and proposes a calibration method to post-process the logits after fine-tuning to improve the classification result in absent classes. The reason why the absent classes are improved is analyzed and the effectiveness of the calibration method is supported by experimental results.

STRENGTHS:
1. The finding on the improved performance of absent classes is interesting. 


2. The proposed calibration method is simple and easy to use.


3. The presentation of the paper is clear.

4. The empirical results in the appendix are extensive.

WEAKNESSES:
1. The reason why the absent classes' features are improved is demonstrated to be that the fine-tuned classes have similar features as absent classes. That suggests that the improvement does not always hold when the fine-tuned classes have features that are not helpful or even harmful (e.g., spurious correlation) to absent classes. This should be discussed further in the submission.

2. In some figures it looks like Tu et. al. achieves the best trade-off, while the proposed scaling method is only presented with a line. It is probably better to show the performance of the two $\gamma$ selection methods in the figure.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13