Holistic Transfer: Towards Non-Disruptive Fine-Tuning with Partial Target Data
Authors: Cheng-Hao Tu
Hong-You Chen
Zheda Mai
Jike Zhong
Vardaan Pahuja
Tanya Berger-Wolf
Song Gao
Charles Stewart
Yu Su
Wei-Lun Chao
Venue: NeurIPS 2023 poster (2023)
Keywords: Fine-tuning
Transfer learning
Domain adaptation
Continual learning
Robustness
Personalization

Abstract: We propose a learning problem involving adapting a pre-trained source model to the target domain for classifying all classes that appeared in the source data, using target data that covers only a partial label space. This problem is practical, as it is unrealistic for the target end-users to collect data for all classes prior to adaptation. However, it has received limited attention in the literature. To shed light on this issue, we construct benchmark datasets and conduct extensive experiments to uncover the inherent challenges. We found a dilemma --- on the one hand, adapting to the new target domain is important to claim better performance; on the other hand, we observe that preserving the classification accuracy of classes missing in the target adaptation data is highly challenging, let alone improving them. To tackle this, we identify two key directions: 1) disentangling domain gradients from classification gradients, and 2) preserving class relationships. We present several effective solutions that maintain the accuracy of the missing classes and enhance the overall performance, establishing solid baselines for holistic transfer of pre-trained models with partial target data.

Holistic Transfer: Towards Non-Disruptive Fine-Tuning with Partial Target Data
Cheng-Hao Tu"!, You Chen"l, Zheda Mai', Jike Zhong',
Vardaan Pahuja',
Tanya Berger-Wolf' Song Gao², Charles Stewart', Yu Su', Wei-Lun Chao
The Ohio State University, University of Wisconsin-Madison, Rensselaer Polytechnic Institute
Abstract
We propose a learning problem involving adapting a pre-trained source model to the target domain for classifying all classes that appeared in the source data, using target data that covers only a partial label space. This problem is practical, as it is unrealistic for the target end-usery collect data for all classes prior adaptation. However, has received limited attention in the literature. To shed light on this issue, construct benchmark datasets and conduct extensive experiments uncover the inherent challenges. We found a dilemma - on the one hand, adapting the new target domain is important to claim better performance; on the other hand, we observe that preserving the classification accuracy of classes missing in the target adaptation data is highly challenging, let alone improving them. To tackle this. we identify two key directions: 1) disentangling domain gradients from classification gradients, and 2) preserving class relationships. We present several effective solutions that maintain the accuracy of the missing classes and enhance the overall performance, establishing solid baselines for holistic transfer of pre-trained models with partial target data]
Introduction
We entering an in which can easily access pre-trained machine-learning models capable of classifying many objects. In practice, when end-users deploy these models diverse domains, adaptation is often necessary achieve high accuracy. To address this issue, considerable efforts have been devoted to domain adaptation (DA), aiming to transfer domain knowledge from the source to the target. This involves updating the model on an adaptation set collected from the target environment. Although various DA settings have been proposed to tackle different scenarios [75] 56 55], we argue that previous setups rely on a strong assumption that hinders their applicability in the real world: the adaptation set contains samples from all possible labels (c.g., classes) in the target environments. Taking classification as an example. adapt a pre-trained source model's recognition ability over C classes to novel domain, DA necessitates access to target dataset encompassing all C classes.
In reality, the adaptation set may only have samples of much fewer classes C' < C due to limited sample size, collections bias, etc. Many pertinent examples can be found realistic Al applications. Considering wildlife monitoring [38) an instance, where data collection often occurs passively, through smart camera traps, awaiting animal appearances. Consequently, when a smart camera trap deployed new location and requires adaptation, assembling comprehensive target dataset containing all the animal species becomes daunting task, especially if specific rare species does not appear within the data collection period due to seasonal variations or animal migrations.
"Equal contributions
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1: Motivating examples of Holistic Transfer (HT), We propose Holistic Transfer, a novel and realistic transfer learning problem. Unlike traditional paradigms, HT focuses scenarios where only subset of classes from the target test environment is present in the target training set. Naive fine-tuning (FT) here can be disruptive.
Dataset
Scenario
Source
Target Training Clipart (30 classes)
Target Test Clipart (65 classes)
Real (65 classes)
Office-Home 74
Domain shift
Many writers: 0-9,
Writer a,
Writer X b. c.
Personalization scarce data
FEMNIST
5
m
s 6
Many locations
Location N: before 2010 Location N: after 2010
iWildCam 58
Camera traps in the wild
CLIP's training data
Caltech 101 classes)
Caltech (101 classes)
FT zero-shot models
VTAB 91
CLIP's training data
Non-toxtie
Non-toxtie, toxic
iNaturalist (Fungi) 73
Confusion classes
To underscore the significance of the problem address in this paper, it is worth noting that the literature has made significant progress towards training a general recognition model that can predict almost any classes in the vocabulary space such as CLIP [60] and ALIGN [35]. However, that the current DA approaches applied to these pre-trained models face a practical learning dilemma in adapting the versatile classification capability. On the one hand, adaptation the target domain style is preferred. On the other hand, the models adapted on larget training data whose labels cover subset of the target test data can lead risky outcomes for the end users. Through experiments, we observed that fine-tuning the model with empirical risk minimization on such partial target data, which fails represent the overall target distribution, hampers the performance of classes that were already pre-trained but unavailable during target adaptation.
argue
This is frustrating for both parties lose-lose dilemma: on the pre-training side, it negates the extensive effort invested in preparing for such source model as the foundation for adaptation. On the side of downstream users, despite the collection of target dataset for adaptation purposes, the difficulty of covering all target classes results in an adapted model performing even worse than the source model in the actual target environment. In response to the dilemma, we ask: can we transfer more holistically, i.e., improving the overall performance without hurting missing class performance? We propose a novel and realistic transfer learning paradigm that aims transfer the source model's discriminative capabilities to the target domain specifically, the task of classifying all C' objects in the source domain while working with target training dataset comprising only a limited subset of these classes, i.e., C" < C. We refer to the problem as Holistic Transfer (HT).
Contributions. We formulate new learning paradigm called Holistic Transfer (HT).
1. We define new, practical transfer learning paradigm that complements existing ones (section 2}. We construct extensive benchmarks in ssbsecctionn 2.2 and and |section to support HT studies.
3. We systematically explore methods to approach HT in /section 3/and provide strong baselines. Our extensive experiments shed insights towards solving HT (section and section 5).
2 Holistic Transfer Learning
2.1 Problem Definition
We propose a novel and realistic transfer learning paradigm called Holistic Transfer (HT). Setup. We focus the task of multi-class classification. Let us consider source model parametrized by 0g. It is pre-trained on source datasef I = { yf ! where x is the input (e.g., an image) and y is the corresponding label (e.g., a class of the C classes in the label space).
As will be introduced in subsection 2.2 the concept of the "source" domain can be quite general: a dataset with different input style, data from a collection of different users, general pre-training dataset, etc.

---

Let the joint distribution of the source be Ps and the true distribution of the target environment be Pr.. The goal of HT, just like standard domain adaptation, to leverage the source model to learn a target model 0 that performs well on the true target data, measured by a loss L..
Remarks: Holistic Transfer. Typical transfer learning setups assume the target training data are sampled IID from Pr.. However, this may not hold in practical scenarios due to sampling bias or insufficient sample size, making the target training data 7 biased; i.c., PT # PT.. The target training data T = =(x] u? ) from the target domain are limited to a subset of classes C' C.
At first glance, our setting seems quite similar to partial DA 15) which also considers the situation of partial target classes (i.e., C" < C). However, its goal is to perform well only on those C' classes after adaptation. In contrast, our setting aims adapt the source model's holistic capability of recognizing all C' classes to the target domain. Below we first give the HT definition grounding on the literature formally. We will contrast the HT problem with other existing learning paradigms in suusscccttinn 2.3)
Assumptions of Ps vs. P7.. Following standard domain adaptation [75] 56 55 we consider Ps and PT. have covariate shifts; thus the target features need to be adapted from the source ones to tailor the target domain style, i.e., (a(wy) * Pr (æly)) Typically, the label space of the true target domain is already covered by the source counterpart, or DT.- S V/s.
Assumptions of PT PT.. We study the setting that T is (potentially biased) sampled dataset from Pr.. therefore, PT and Pr- should share the same domain styles and semantics. That is, Pr(z R= Pr- (xly), (y)(x)))))))))))))))()) se PT- (y(z)). However, the label space of Pr might be incomplete w.r.t. Py-; (y(y) + Pr-(y). In other words, the test set T* sampled from Py- might contain data belonging to classes
in S but not appear in the target training set T. The ultimate goal of the target model is to perform well on Py..
Objective. The objective of HT is to learn predictor f(: (T:)T) Xy. + V/- that minimizes the expected loss on the true target distribution PT. but given only the partial observations of a target training dataset T sampled from PT without accessing Pr.:
R(;; Pr.) = E(a.g)~Py.
RJ:T)- = - ((x)))
(1)
That is, we can only compute the empirical risk RJJ:T) on 'T but not the true risk R(J; Pr-). Challenges in a realistic HT scenario. We focus on a more realistic scenario of HT that comes with the following practical constraints and properties when solving R(J; PT-) in Equation 1
Missing classes in target training data. The main challenge is that some classes might only appear in Py- in testing but not yet seen in the target training set T. Minimizing R(f:T) naively might be risky since it is biased estimation of R(f: Py-).
Disparate impact [26]. Training on the partial target data can result in disproportionate effects on the performance of unseen target classes in testing. The impact may vary based on the semantic affinity between classes, potentially causing serious false negatives. For instance, if a biologist fine-tunes an ImageNet classifier on some non-toxic plants and uses it to recognize plants in the wild, will likely misclassify those visually similar toxic plants as non-toxic ones.
Covariate shifts from the source domain. If we only have T, the missing classes very difficult to be predicted as it is zero-shot classification. A more practical scenario is to leverage an external source classifier that has been pre-trained on S with many classes. This relaxes the problem to be special (supervised) domain adaptation task that leverages the source classifier's 's discriminative ability (as (s(y)z Re Pr (v)))) while adapting the input styles Ps(x)) F (r)()).
Source-data-fres transfer. Last but not least, the source model 0s is available but the source dataset S is not. Due to privacy concerns the dataset's large size, accessing the source dataset is often unrealistic, especially for foundational models pre-trained in large scales like CLIIP 60).
Therefore, we believe HT is an important and challenging problem to approach for improving towards safer transfer learning in the wild, given only a source classifier Og and a partial target dataset T.
2.2 Benchmarks Curation for Holistic Transfer: Motivating Examples
To support the study of the HT problem, we create a benchmark that covers extensive scenarios across both experimental and realistic public datasets. More details about dataset curation in the supplementary. As introduced in Table I] HT is applicable in various use cases, such as:
Office-Home [74] domain adaptation while some classes are missing in the target training set.
3

---

FEMNIST 4 personalized wundtwritten alphanumeric recognition with writers' styles. Training
samples of single writer are insufficient to cover all classes.
iWildCam 38 species recognition across camera traps of many geo-locations. Adaptation is
based on images collected so far, and is deployed for the future; i.e., samples are biased temporally.
VTAB [9]] fine-tuning zero-shot models for diverse vision tasks with partial classes each.
iNaturalist (2021 version, Fungi) [73]: classification of visually-similar poisonous fungi.
Each task pre-trains a source model on the source domain, transfers to the target domain with only partial classes, and tests on the full-class target test data. For the Office-Honn dataset, the source model is trained on one domain different from the target domain. For the FEMNIST/WildCan datasets, the source and target data are split by writers/locations. For the VTAB and iNaturalist datasets, there are no explicit domains. A general dataset can be used for pre- vre-training the source model. Here we will use CLIP (60) just as an example.
2.3 Comparison to Other Paradigms: To Adapt or Not Adapt, and How?
The most common way of transfer learning is probably fine-tuning, i.e., initializing the target model with the source model parameters and optimizing it with empir-
50
ical risk minimization (ERM) on the target data. In the HT setup, tne-ttning by minimizing R(f; 'T) in Equation 1 on the one hand, improves on the performance of those classes seen in the target training set T. On the other hand, can be disruptive in terms of the true risk PT.) due to catastrophic forgetting (37 of those classes already seen in the source domain but unseen during the target fine-tuning. This creates dilemma for the user about whether to adapt or not, As shown in the example in Fig. ure (details in sectioo 4), naive fine-tuning does improve those seen classes over the source model while unseen classes can degrade drastically. On the contrary, our proposed treatment (see sseesection 3) maintain the unseen performance and improve the seen ones, compared to the source model (the O epoch).
1est accuracy
40
Ours Overall
Ours Unseen
30
Naive Target Overall
Nalve Target Unseen
20
10:
0
2
4 Training opachs
10
Figure 1: Fine-tuning disrupts unseen tar- get class performance. An HT example of Ar + CI domains on Office-Home dataset.
We note the HT problem we proposed is unique machine learning paradigm that lies between and complementary to existing learning-wi berhinngwith-iiie tith-distribution-shiee problems [59]. From a high-level and simplified view, many of them involve three stages: 1) source training S to obtain source model, 2) target training on T, then 3) evaluation of the target goal distribution Ptrst.
We summarize the difference to some popular paradigms in Table 2, including general transfer learning and (supervised) domain adaptation (DA) 198 75 that assume the target distributions in
training and testing are matched. Continual learning (CL) 37 51 47) that typically disallows access to source data during target training, aiming to build a multi-tasking model that performs well on all the (possibly many) distributions that have been trained on so far. They all assume the training and test data are matched in distributions (Pr R Pfest). Some recent works about OOD domain generalization [4]] 3 [99 study fine-tu tne-tuning from versatile pre-trained model and preserving its robustness in testing to different unseen input styles of the seen classes but not for unseen classes
As illustrated in Figure 2 (d), HT requires a very different ability (the green arrow): "generalize" the style shifts learned on the target seen classes to the classes unseen during target training (the red dashed arrow). Compared to OOD generalization, our goal is much harder since it requires transferring the source model to the target styles, for classes both seen and unseen the target training data. Given only seen classes, using fine-tuning here is notoriously biased to them and disrupts the unseen performance. HT is also addressing quite different challenges from source-free DA 48, unsupervised DA (with partial classes (2). ). and test-time adaptation [69] which focus on recovering the true labels for the unlabeled target data. More related works are in the supplementary.
Our goals. To this end, raise the following research questions:
1. Can we resolve the adaptation dilemma and transfer holistically from the source model, improving the test performance of classes seen in the target training set while not hurting the unseen ones? Is possible to generalize the style shifts to unseen classes? Or how far are we?
3. When should HT work and not work in realistic cases?

---

Table 2: Comparison of different paradigms: Source Training-' Let Ps and Pr be the distributions of the source data S and the target training 7. respectively. p is the target test distribution as the linal goal. Pr the distribution
"raining-'Test on Target Goal.
may have certain discrepancy from Pr.
Settings
Target Training
Target Goal Pr
Ps vs. P
Py vs. P7
Standard
Ag andior
Pr
Ps Pf
Pr Pi
Domain adaptation
0g and/or S. T
Py
Psly m) 2 P7
Pr = Pf
Continual learning
0s.
Ps + Pr
Ps # P7
pr 7a Ps Pr
0OD generalization
0s.
Pyx
Ps # Pr
Pr(xly) # Pr. (z)
Prly) R PT
Holistic transfer
Pya
Psly œ) (yla)
Pr Pr. Pr(y) # PT.
Class ID
Class ID
Class ID
Class ID
Styles
Styles
Styles
Styles
(a) Domain Adaptation
(b) Continual Learning
(c) OOD Generalization
(d) Holistic Transfer
Figure 2: Different settings of learning with distribution shift. Consider a data distribution space of different
styles (x-axis) and classes (y-axis). each algorithm leverages the source data (the blue region to facilitate the knowledge transfer (the green arrow) the target training data (the yellow region The target test distribution pfrt is highlighted with the black box. The red arrow represents the generalization for unseen test data.
3 Towards Non-Disruptive Fine- Tine-Tuning
3.1 Solving the HT Objective
We now discuss approaching the HT objective of Equation The key difficulty is the fact that the true risk R(f: Pr.) ) and the empirical risk R( 7) are in different distributions (i.e., Pr-(w) * Py (v)), such that methods relying on ERM fine-tuning can be seriously biased seen classes, as discussed in inssubsection 2.3 Let Py has C labels y {yfeen ..... _2 and Py. has some extra unseen labels Unseen
DT. U [9C+1 .VC+2 ...., Unseeen Worth noting, we fairly assume the source data cover all target classes D'T- C Vss if
class was not seen in it becomes a zero-shot problem which might not be solved by transfer learning.
Oracle solutions. An oracle solution directly match the distributions, making Py. = PT by augmenting data to T. For instance, one can collect more data for those unseen classes in the target domain, or translate the source data sample an unseen class (x: UC Unsee) into (ax T _ with a style transfer model from S to T. They are obviously not available for HT- the former requires the cost to collect complete labeled data the target (essentially the upper bound in domain adaptation); the latter needs to access the source data which is unrealistic discussed in iilsubscction 2.1l
Suboptimal solutions. By leveraging techniques in other paradigms we discussed in insubbsctto 2.3 it might be possible improve over plain fine-tuning. However, probability of the target goal distribution Py (æ, y), but only Pr y) or (r)
explicitly recovers the joint
Next, we propose two directions that together could approach Py (æ, v) under HT constraints.
3.2 Disentangling Covariate Shifts from Disruptive Concept Shifts
Minimizing R(f:T) on T transfers the model from (s(æ, yı 0s) to (a 0T) which optimizes towards the target styles PT(xly) (which should be the same as P (xly)) but also collapses the label distribution to Pr (y; 07). ). The model will unlikely predict y for any inputs. we can only adapt the covariates from the source classifier, then we have
Py- (x, y. 0,) Py- (ulæ: 0, Pr 0r),
Pr. (y) 0s) (ax; 0s),
Sttle Adaptation P (x: 07) + Ps( 0s)] [P (u/z) = Pr(v(=) ~ Ps(y|=))
(3)
- Ps
(4)
5

---

Equation 3 will hold if (r-( = Pr(xly). However, how can we disentangle the covariate shifts from VR(f:T) and change the style to (z only without concept shift [40] ? We explore:
Updating batchnorm statistics only. The statistics in batchnorm layers (in feed forward networks) are believed to capture the domain information. Updating them is often considered as strong baseline in DA [46]. It infers on T and re-calculates the feature means and variances, which is less likely to be impacted by the missing classes issue.
Training extra instance normalization layers. We explore another way based on instance normalization (IN) 722 which normalizes within each input itself over its spatial dimension. IN is widely used in style transfer to capture the input styles without changing their semantics. We examine way for HT by inserting extra IN layers into the source model (which may not have IN layers originally) and fine-tuning them with other DNN parameters fixed to prevent concept shifts.
Unfortunately, we found updating normalization layers is not yet satisfying receiving limited improvements and still suffering from catastrophic forgetting. We thus hypothesize the key is to consider the optimization besides architectures.
Proposed Leave-Out Local SGD (LOLSGD). In our HT problem, fine-tuning is affected by the covariate shift (from the source to the target) and the disruptive concept shift (from classifying all the classes to classifying the seen classes). We aim to disentangle them or, more precisely, reduce the disruptive concept shift by subsampling multiple datasets that each contain a subset of the seen classes.
(
9 (
Partial Class Bias
gLOI
y()'r-) Ideal gradient
We propose a novel approach based on local SGD 68 in distributed learning. The local SGD gradient is accumulated over several consecutive SGD steps, possibly many runs in parallel. For notations, let the local SGD gradient computed on the samples that their labels are inside a label space Y be ())). sampling IID from T and the local step is 1, it reduces to standard SGD. Instead of using g))r), which will bias to the seen target classes Vr, our idea is to mitigate the change towards the label and encourage the change in styles.
9
Figure 3: LOLSGD illustration.
Inspired by recent theoretical analyses in meta- veeta-representation learning 21 20 12 13
we intention-
ally sample m non-lll) subsets of labels )op _ Y'r by leaving out some classes, compute g for each m. and average their local gradients g g)
():: = M E90
0 arg ming E (ieTTllee)) Clys, f(x;; 0)). (5)
During training, each update (with size 1)) becomes -nigh.r. or. instead of -v) (R(;)) Intuitively, g(TP) is biased towards the respective classes )am and drifts away from each other along more local steps, similar to the model shift problem decentralized learning 22 39 10] As a blessing here (rather than a curse!), our key insight is averaging them could "cancel out" the gradients biased to certain classes. The style should be adapted for Equation 3 without conflicts since all local gradients are directed towards the same target style, as illustrated in Figure 3] When updating the model separately in parallel with these subsampled datasets, each updated model is affected by a different concept shift but a shared covariate shift. Averaging these models can potentially cancel out the disruptive concept shifts and strengthen the shared covariate shift. We verify our intuition in Table 19 in the supplementary. Compared to VR(f: T), gLOL averages M (M = 10 in our experiments) accumulated local gradients thus more expensive to compute. As will be shown by experiments, LOLSGD can outperform standard SGD with similar computation budgets in fair comparisons.
LOLSGD is fundamentally different from meta-learning, whose goal is to learn a modal that can be easily applied to a future task (e.g.. few-shot classification task). While meta- learning also subsamples its meta- vetaa-taiinnn set, it is mainly to simulate multiple future tasks for learning the meta-model, not to cancel out unwanted gradients. LOLSGD is also an extension of local SGD. The goal of local SGD is mainly to reduce the communication overhead of large-scale distributed training. In contrast, in LOLSGD, the target training data set not decentralized initially, but we strategically subsample it to simulate different concept shifts.

---

3.3 Preserving Class Relationships
Noive Target Ours
The subsection 3.2 assumes we can change the styles while not changing the conditional probability of the source model Ps(y|ar (z)) in [qquatinn 4 We observe in practice it can still be sensitive and lean towards {yyS ). We, therefore, hypothesize that it is neces- sary to have some regularization to preserve the nice class relation- ships encoded in the source model already during target training such that Equation 4 could hold. Let us decompose the predictor g z w) ) h(æ; Φ) to be a feature extractor h and a linear classifier g. We explore the following strategies:
Lo Singuda
50
100
150
200
Index of singuarr values
Figure 4: Feature rank col- lapse. Top singular values of Office-Home adapted features.
fa;; 9)
Frozen linear classifier. Even the source features is already perfect for Ps(ulæ) = PT- (ylæz) it
can collapse quickly as the logits of y Unseen will be suppressed by VR( We found it crucial to freeze the linear classifier w, which corresponds to the common practice in source-free free DA 48 Selective distillation. Another way to utilize distillation that can prevent forgetting in CL 47
[2] forcing the soft labels of the target model and the source model to match. Given a sample (x, gy E T and let fUnssen (a: be the logits of yy Unseen dimensions. Here, only the unseen class logits need to be distilled since seen class logits are reliable as the target model is trained on ground truths, unlike source seen class logits (which are suboptimal due to domain shift). We add
a distillation loss Ldistll = (L.(((J [ (z: 0s)) (lor [J (a:; Q))) with the Kullback-Leibler
(KL) divergence, where a(-) is the softmax function.
Feature rank regularization. A key observation is in the feature dynamic of fine-tuning on the partial target data- - instead of shifting to another full-rank space, we found the features tend to collapse to low-dimensional space, shown in |Figure 4 implying the target model "shortcutting" the solution and hampering the generalization ability [76] to handle unseen target classes. We consider regularizer to avoid too many singular values of the features becoming zeros. Given the feature matrix of a mini-batch Z E R xd and its mean feature W E IRd we compute its covariance matrix C- N E ==) E)T and regularizer Lrank = || diag CT 2.
inspired by 36
66
We note that, these techniques only preserve the source class relationships, but they do not improve the transfer to the target. Thus they should be considered together with ERM LOLSGD for HT.
Ensemble with the source model. As pointed out by [79], a way to reclaim some ability of the source model is to average the parameters of the source and target models, assuming they lie in a linearly-connecte connected space. We consider it complementary post-processing for the target training techniques we considered above. We relax the assumption and further examine the ensemble of the source and target models in their predictions (after the softmax function a for logits).
4 Experiments
Table 4: Effects of Source Ensemble
(Avg. as in Table 3).
General setup. To answer the questions raised in subsseee [ion 2.3, we experiment on extensive HT scenarios proposed in|subsection 2.2 For each task, we pre-train source model in standard practice, discard the source data, and adapt the source parameters O.s obtain the target model. We split the target data for training and testing, based on different scenarios. The training and test sets share the same styles but cover different sets of classes, where some classes are unseen in training. We assume the source model covers all classes and the goal is to adapt to the target domain for both the seen and unseen classes. All methods use the cross-entropy loss for L and SGD momen- tum optimizer. All experiments fine-tune the source model for 20 epochs (10 for FEMNIST) by default. For the regularizers in section 3, we attach them as L + Ausainl (or rank) Casinll for rank),
Methods Acc.
Overall Unseen
Source
63,90
65.65
Naive Target (w/o SE)
53.50
22.36
Naive Target (WISE)
53.56
25.26
Naive Target (SF)
67.46
50.89
SGD (WISE) *
69.75
60.42
SGD (SE) $
71.01
59.19
SWA *
64.99
45.10
SWAD &
65.05
45.30
SGD C
$
69.78
62.94
SGD - Crank t5
71.05
65.48
LOLSGD
70.70
63.14
LOLSGD + Cassall
68.71
64.56
LOLSGD + Crank *
69.94
66.29
where the weights are quite stable thus we did not search for it exhaustedly for every method, but use the same ones per dataset. For the proposed LOLSGD, we set M 10 and randomly drop 3 classes when sampling ym in Equation 5
Best AACC. Tageet
17.55
43.12
Each subgradient in LOLSGD is by
7

---

Table 3: Domain udaptation 65 way test accuracy on Office- Hifice-Home with 30 seen and 35 unseen classes, Variances of random seeds are provided in the supplementary due to space limit. Blue: HT methods suggested by us in Insseciionn 3 Red: methods that significantly improve overall accuracy and successfully maintain unseen accuracy on the source model. * the linear classifier is frozen during training.
Domains: source sturec-tareet
Ar-+CI
Ar-PP Overall
Ar-Rw Overall
Rw Overall Unsoen
Rw-sCI
Rw-sPr
Avg.
Methods/ / Avc.
Overall
Unseen
Unseen
Unseca
Overall
Unsee
Overall
Unseen
Overall Unscen
Soures
47.07
67.45
72.00
72.73
74.68
51.13
19.26
79.28
Naive Target
44,96
9.06
52.39
23.75
59:48
32.91
54,93
28.03
46.92
6.34
62.31
34,08
53,50
22.36
BN only
45.94
14.33
54.90
90.25
63.22
41.91
61.47
42.32
49.10
13.42
71.57
$1.96
57,60
32.36
BN (stats) only
47.44
50.44
65,03
70.88
71.76
74.12
65.73
70.89
55.56
55.75
80.09
82.96
64.27
67.51
IN only
48.96
71.13
76.23
74.26
66.40
51.48
79.35
65.17
LP-PT
43.91
6.73
48:05
16.38
55.81
25.60
50.67
19.14
45.04
3.54
56.21
22.21
49,95
15.60
SGD (w/ frozen classifier
52.11
24.12
64.36
44.75
70.26
55.27
68.27
53.10
55.26
24.
75.75
59.22
64.34
43.44
SGD Last &
56.54
39.18
61.63
75.73
67.51
70.13
64.15
61.96
80.60
70.53
69.63
57.41
SGD f..a &
59.17
57.57
74:31
64.28
70,40
64.42
61.65
68.86
55.64
SWA 8
53,23
26.32
65,25
46.0 00
71.46
56.82
68.40
55.26
56.24
26.55
75.39
59.64
64,99
45.10
SWAD
53.38
26.46
65.25
46.00
71.61
57.24
68.53
55.53
55.87
26.40
75.68
60.220
65.05
45.30
LOLSGD
56.47
35.09
70.83
56.88
74.91
64.84
70.53
62.53
58.72
35.25
80.02
69.27
68.58
53.98
LOLSGD + Crek
58.57
43.86
72.59
64.53
75.06
68.92
70.13
66.04
61.58
44.10
80.82
74,02
69.79
60.26
LOLSGD + Cau *
57.44
75.24
67.38
76.33
70.75
69.07
60.68
45.55
81.56
75,28
61.85
LOLSGD Cauit Ceas 8
60.83
51.75
75.75
70.25
76.70
74.26
70.13
69.54
65.71
53,10
82.95
80.31
72.01
66.54
Best AAcc Kaw Tager
15.87
42.69
23.36
46.50
17.22
41.35
15.47
36.39
18.79
46. 76
20.64
46.23
18:51
44.1%
Orccle
79.32
82.46
90.96
91.50
84.57
84.95
78.13
80.59
79.85
75.52
91.92
92.04
84.13
84.51
local SGD 2 epoch) and we run the same total epochs, for a fair computation budget. Please see the supplementary for the details of setup, hyperparameters, and more analyses.
4.1 Domain Adaptation on Partial Target Data
We first use the Office-Home dataset [4] with ResNet-50 28 to study the effectiveness of the methods in section 3 Office-Home is a popular domain adaptation benchmark consisting of 65 object categories from four domains (Art, Clipart, Real, and Product).
Methods. We consider baselines including directly predicting with the Source model and naively fine- tuning it for Target model. As proposed in ssection 3] we explore methods that promote covariate shifts by learning the limited parameters in normalization layers with the backbone parameters frozen, like batchnorms (BN) [3 46 and an extra instance normalization (IN) [72] layer. We include another baseline LP-FT
proposed for the OOD generalization of fine-tuning but not for
missing classes in HT. To compare with our LOLSGD, we also include related techniques in domain generalization based on stochastic weight average (SWA) [33 and its extension SWAD
Comparison. We highlight the following observations in Table 3
Naive fine-tuning is disruptive: the unseen performance drops drastically, ultimately degrading overall accuracy and confirming the challenges of HT for all pairs of domains.
Normalization layers may help: maintaining unseen accuracy and improving the seen little. Freezing the classifier during training is necessary: we found that the frozen classifier is crucial when the feature extractor is adapted (Avg. +21% Unseen vs. Target), motivating our approach of preserving class (sssionnsips/uubsctttiii 3.3)
Eaisan and Crank regularize forgetting further: largely mitigating unseen performance drops. LOLSGD is effective: without any regularizers, the proposed local SGD optimization can outperform the strong generalized SGD baselines SWA and SWAD.
LOLSGD with regularizers performs best overall: as we suggested in section 3 LOLSGD needs to be carefully regularized while it adapts the domain styles.
Effects of Source Ensemble (SE). We further post-process the target models learned in Table 3 by ensembling with the source model, as introduced in (subsection 3.3) We also include WISE [79 that ensembles the model weights instead of predictions. Generally, found SE may reclaim some unseen performance, but not necessarily improve overall, as summarized in Table 4
Overall, we show that HT is promising, improving the overall performance without sacrificing the unseen accuracy (highlighted in red in Table 3). Notably, our proposed solutions are based on optimization and agnostic to architectures. We will mainly focus them later as they perform better.
8

---

Table 5: FEMNIST mean accuracy of 10 new writers. Table 6: iWildCam mean accuracy of 21 new locations.
Methods
Overall
Seen
Seen (Chopping)
Unseen
Mothods
Overall
Socn
(Chopping)
Unsoon
Source Naive Target
84.67 82.67
88.67 92.07
89.60 92.07
64.99 35.60
Source Naive Target
35.71
35.74
58.40 52.72
26.43
33.38
51.08
1.90
SGD
87,8 85
92.87
93.09
64.00
SGD
36.53
49.00
50.93
7.91
SGD + Cook *
88
93.29
93.51
64.00
SGD Cran #
36.28
40.98
56.13
19.76
SGD + Carm &
87.44
92.46
92.81
61.91
SGD fama H
41.59
50.40
54.23
17.90
LOLSGD
87.47
91.87
92.23
66.27
LOLSGD
38.12
48.91
52.49
13.30
L.0L,GD + Cra 8
87.16
91.61
91.97
66.27
LOLLSG Cent $
33.94
37.62
53.40
19.42
1.01.SGD + Cuin *
85.76
90.34
91.25
63.56
L.0LSGD Caaa
*
40.49
47.30
55.27
25.16
LOLSGD Cami +
85,10
89,52
90.58
63.56
LOLSGD Cama +
39.96
44,79
58.22
25,66
Table 7: Fine-tuning CLIP VIT-B/32 on VTAB. 50% of classes each task are missing during training.
Overall/Unae Unseen Ace.
Ca
CIFARICO
DTD
EunSAT
Flowers
Pots
Resise45
SVHN
SUN397
Avg.
-
ABl
Mothods
Source Naive Target
78.7
61.2
84.1
11.3
82.3
65.9
15.8
52.2
1.2
49.7
60.3
34.8
SGDSS LOLSSG
82.3
69.0
66.1
42.7
45.5
16.4
52.5
1.8
64.5
49.8
84.7
75.8
57.9
11.9
40.1
0.5
49.8
36.0
60.4
34,9
82.6
69.8
69.6
51.8
46.0
18.8
52.7
2.7
65.3
51.0
85.9
78.2
59.4
27.1
36.7
0.7
52.5
45.5
61.2
38.4
LOLSGD Erean &
82.0
67.1
49.1
22,9
52.7
3.8
66.3
52.9
52.6
47.6
60.8
38.8
LOLSGD + Craa 8
82.8
70.3
70,0
52.5
45.3
18.4
51.4
1.4
64.2
50.9
84.6
76.8
59.5
27.6
34.5
0.3
52.8
46.1
60.6
38.3
1.0LSGD + Crak S
82.5
70.8
72.3
58.4
52.6
37.3
54.5
8.4
69.2
62.5
87.3
83.2
64.8
39.7
95.8
3.8
53.4
50.0
63.6
46.0
88:0
83.1
60.0
61.3
197.7
97.1
75.5
190.4
90.3
994.2
95.6
58.8
60.0
$2.9
4.2 Realistic Holistic Transfer: FEMNIST and iWildCam Datasets
We simulate two realistic scenarios in which a new user collects a limited-size adaptation set and aims to customize source model. Each new user has its natural domain shift.
62-class hand-written characters dataset that consists of many writers [9] of different styles. We train LeNet 44] on 40 writers and adapt to each of the 10 new writers. Each writer only has a few samples per class, thus when the data are split into training and test sets (in ratio of 7 3) for personalization may suffer mismatched Py(y).
FEMNIST
iWildCam [38 2 animal recognition dataset of 181 classes of camera traps located worldwide. We train ImageNet pre-traines trained ResNet-50 on 53 locations and adapt to 21 new locations. The data of each location are split by a timestamp; we train on previous data and test on future data.
Encouragingly, the observations on both natural datasets are generally consistent with the Office- Home experimental dataset. As shown in Table 5) and Table 6 The seen and unseen accuracy can be improved upon the source model at the same time, even without might be two reasons if the seen class accuracy is
ensemble. Here, we note that
the seen class accuracy might be misleading. high: 1) it adapts well to the target style and/or 2) the model simply collapses an easier, fewer-way classifier. Just for a better understanding of the quality of the adapted features, we also report the seen classes' accuracy but "chopping out" the unseen classifiers in testing"] The seen accuracy with a chopped classifier is also competitive to naive fine-tuning, validating the feature quality is good; lower accuracy is mainly due to more classes making the problem harder.
4.3 Fine- Tinne-Tuinggae on Zero-Shot CLIP ViT for Vision Tasks
Next, we go beyond domain adaptation and consider distribution shifts at the task level using the
VTAB [92] benchmark that contains various vision tasks. We split each task by its labels; roughly half of the classes are considered unseen. We use the CLIP pre-trained ViT-B/32 [60]. Only those
tasks in which the class names are in natural languages are considered using their text embeddings for zero-shot predictions. The results are summarized in Table 7 We further compare to the CoCoOp [96 fine-tuning method designed for handling unseen classes the supplementary.
Across the tasks, we see HT methods remain effective in most cases. Interestingly, we see some exceptions that the degrees of improvement become diverged, which inspires discussions next.
In realistic cases, the observe environments not available for the oracle upper bounds.

---

Accuracy change
Accuracy change
Accuracy change
-10
Naive Target
-10
-20
Ours
-0
CH941100
Unseen
Resiboks
SUN307
Flowers.22
DTD
Cattecntos
EDroSAT
Pers
swww
-20
Powerstor
Unsees
DTo
Cottecnt01
CHAR100
Resisce5
SUN397
Pets
EooSAt
sWey
DTD
Uncron
Powerst02
Cattecn101
CIPARIDO
Resisee4
SUN397
Pets
Eurosar
swap
(a) CIFARI00
(b) Flowers102
(c) DTD
Figure 5: Disparate impact: will fine-tuning improve similar tasks? We fine-tune CLIP on the training set
of a task and evaluate on others. Naively fine-tuned target models actually degrade more on more similar tasks, while our HT method is much more robust. Tasks are sorted in the x-axis based on Affinity Score following (25]
4.4 Studies: When should HT Work and Not Work?
Semantic gaps of the source model. As discussed in subsection 2.3 HT relies on the DA assumption that the source and target tasks share similar semantics, 1.c., Ps(v x) = Pr (r(y(z). There should be little semantic shifts from the source to the target. In Table we found the unseen class performance of some tasks is very low, and applying HT methods cannot help much. We hypothesize that the images are likely to be predicted other meanings by the CLIP model rather than the goal of the target task. Looking at the Street View House Numbers (SVHN) dataset [54), we adversarially add class names like "number plate" to confuse with the digits (e.g., "9"). Interestingly, we found about 60% of the cases, CLIP fails to predict a number (Table 8). This validates that Ps (y) 2+) pre-trained by CLIP is quite different from the digit classification, therefore, becomes very hard if not impossible for it to recognize unseen digits correctly. Similarly, DTD (texture) and EuroSAT (land use) could suffer from the same concept shift issue. Resolving requires "task-level generalization" (e.g.. object recognition - digits classification), which is obviously out of the scope of HT.
Disparate impact. we fine-tune a task, should it benefit other similar tasks? We reveal that the answer might be negutive in HT settings. We fine-tune CLIPP on a task following Table 7 evaluate on others using the corresponding CL.IP text classifiers, and summarize the accuracy difference to the original CLIP in Figure 3) Intrigu- ingly, more similar tasks in fact drop more, likely because they are disrupted by the fine-tuned task most. This again highlights the importance of HT; we found our method makes it more robust.
Table 8: Examinizing HT of zero-shot CLIP on SVHN dataset. Classes
Predictions
"0" to "9"
39.4%
"number plate"
32.7%
"house number"
16.8%
"wallpaper"
11.1%
9
Case study: fine-tuning can lead to serious false negatives. So far, mainly evaluate HT from the lens of unseen class accuracy. Here, we discuss confusion in predictions and a case that transfers without considering HT could pose even larger risks. Considering training a classifier for recognizing mushrooms on some non-toxic ones and encountering visually similar but toxic one in testing, the classifier will likely label it as one of the non-toxic ones. Even worse, such bias exaggerated by the adaptation, due the partial training classes. We build an extreme case that selects 6 pairs of fungi (toxic/nonn- toxic each) from the iNaturalist dataset [73] and fine-tume men-tunes on only the non-toxic ones from CLIP. The results in TTble 9 show such concern exists and should be considered in future algorithm designs.
Missing rate
60.6%
Table 9: Fine-tuning can lead to serious false negatives.
Method
Scon
False Nogatives
Accuracy
Source Target
23.3%
46. 7%
63.3%
96.7%
Our HT
60.0%
83.3%
Toxic
Nox-Toxic (Train)
5 Conclusions (Related Works Limitations Supplementary)
We introduce novel and practical transfer learning problem that emphasizes generalization to unseen classes in the target domain but seen the source domain. Despite its challenges, we establish strong baselines and demonstrate the potential for improving both seen and unseen target classes simultaneously, paving the way for holistic transfer. To facilitate further research, we construct a comprehensive benchmark with diverse scenarios and conduct insightful experiments. In future work, we envision exploring various directions, including improved disentanglement of domain styles and classes, integrating our approach with other paradigms like test-time adaptation, and extending beyond classification tasks.
10

---

Acknowledgment
This research is supported by grants from the National Science Foundation (04C-2118240, OAC- 21 2112606). We are thankful for the generous support of the computational resources by the Ohio Supercomputer Center.
References
[1] Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty-biee based continual learning with adaptive regularization. Advances neural information processing systems, 32, 2019.
[2] Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. The iwildcam 2021 competition dataset. arXiv preprint arXiv:2105.0344 2021.
[3] Ruisi Cai, Zhenyu Zhang, and Zhangyang Wang. Robust weight signatures: Gaining robustness as easy as patching weights? arXiv preprint arXiv:2302.12480 2023.
[4] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný; H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: benchmark for federated settings. arXiv preprint 18X2v:8//22000000 2018.
[5] Zhangjie Cao. Lijia Ma, Mingsheng Long, and Jianmin Wang. Partial adversarial domain adaptation. In Proceedings of the European conference on computer vision (ECCV), pages 135-150, 2018
[6] Zhangjie Cao, Kaichao You, Mingsheng Long, Jianmin Wang, and Qiang Yang. Learning to transfer examples for partial domain adaptation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2985-2994, 2019.
[7] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Ches Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405-2241 2021.
[8] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and M Ranzato. Continual learning with tiny episodic memories. 2019.
[9] Hong- You Chen and Wei-Lun Chao. Gradual domain adaptation without indexed intermediate domains. Advances in Neural Information Processing Systems, 14:8201-8211 2021.
[10] Hong-You You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In ICLR, 2022.
[11] Hong-You You Chen, Yandong Li, Yin Cui, Mingda Zhang, Wei-Lun Chao, and Li Zhang. Train- once-for-all personalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 11818-1 1127,, 2023.
[12] Hong- You Chen, Cheng-Hat Hao Tu, Ziwei Li, Han- Kan-Wei Shen, and Wei-Lun Chao. On the impor- tance and applicability of pre- pre-training for federated learning. In ICLR, 2023.
[13] Hong-You You Chen, Jike Zhong, Mingda Zhang, Xuhui Jia, Hang Qi, Boqing Gong, Wei-Lun Chao, and Li Zhang. Federated learning of shareable bases for personalization-frienlly image classification. arXiv preprint arXiv:2304 0.08822, 2023.
[14] Lin Chen, Huaian Chen, Zhixiang Wei, Xin Jin, Xiao Tan, Yi Jin, and Enhong Chen. Reusing the task-specific classifier as a discriminator: Discriminator-fres adversarial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7181-7190, 2022.
[15] Shiming Chen, Ziming Hong, Guo-Sen Xie, Wenhan Yang, Qinmu Peng. Kai Wang, Jian Zhao, and Xinge You. Msdn: Mutually semantic distillation network for zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7612-7621, 2022.
[16] Shiming Chen, Wenjie Wang, Beihao Xia, Qinmu Peng, Xinge You, Feng Zheng, and Ling Shao. Free: Feature refinement for generalized zero-shot learning. In Proceedings of the IEEF/CVE international conference computer vision, pages 132-131, 2021.
11

---

[17] Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning, Advances in Neural Information Processing Systems, 32, 2019.
[18] Sachin Chhabra, Hemanth Venkateswara, and Baoxin Li. Generative alignment of posterior prob- abilities for source-free domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4125-4136 2023.
[19] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921-2926 IEEF, 2017.
[20] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Fedavg with fine tuning: Local updates lead to representation learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[21] Liam Collins, Aryan Mokhtari, Sewoong Oh, and Sanjay Shakkottai. MAML and ANIL prov- ably learn representations. In Kamalika Chaudhuri. Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 4238-4310. PMLR, 17-23 Jul 2022.
[22] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019.
[23] Ning Ding, Yixing Xu, Yehui Tang, Chao Xu, Yunhe Wang, and Dacheng Tao. Source-free domain adaptation via distribution estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7212-7222, 2022.
[24) Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Trans- formers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9285-9295, 2022.
[25] Utku Evei, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2Toe: Utilizing intermediate representations for better transfer learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 6009-6033. PMLR, 17-23 Jul 2022.
[26] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger. and Suresh Venkatasub- ramanian. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 259-268, 2015.
[27] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems. 35:29374-29385, 2022.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[29] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pages 4387-4398. PMLR, 2020.
[30] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Korblith, Ali Farhadi, and Ludwig Schmidt. Patching open- gpnnvooabulaayy models by interpolating weights. arXiv preprint arXiv:2208. 2022.
[31] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference muchine learning, pages pmir, 2015.
[32] Masato Ishii, Takashi Takenouchi, and Masashi Sugiyama. Partially zero-shot domain adaptation from incompleee target data with missing classes In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision pages 3052-3060, 2020.
[33] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint 17X 2018
12

---

[34] JoonHo Jang, Byeonghu Na, Dong Hyeok Shin, Mingi Ji, Kyungwoo Song, and I1-Chul Moon. Unknown- aware domain adversarial learning for open-set -et domain adaptation. Advances in Neural Information Processing Systems, 35: 15:16655-16667, 2022.
[35] Chao Jia, Yinfei Yang, Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904 49164 PMLR, 2021.
[36] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022.
[37] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agmieszka Grabska-Barwinsker Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, I 114(13)3521-3356 2017.
[38] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao. Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5637-5664. PMLR, 18-24 Jul 2021.
[39] Lingjing Kong, Tao Lin, Anastasia Koloskova, Martin Jaggi, and Sebastian Stich. Consensus control for decentralized deep learning. In International Conference on Machine Learning, pages 5686-560 5696. PMLR, 2021.
[40] Wouter M Kouw and Marco Loog, An introduction to domain adaptation and transfer learning. arXiv preprint 1tt.::8812.///0 2018
[41] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretraine features and underperform out-of-distribution. arXiv preprint 2200 2022.
[42] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform distrifdditibitionn. arXiv preprint 10054 2022.
[43] Jogendra Nath Kundu, Akshay R Kulkarni, Suvaansh Bhambri, Deepesh Mehta, Shreyas Anand Kulkarni, Varun Jampani, and Venkatesh Babu Radhakrishnan. Balancing discriminability and transferability for source-free domain adaptation. In International Conference on Machine Learning, pages
PMLR, 2022.
[44] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))2278-232) 1998.
[45] Xiangyu Li, Xu Yang, Kun Wei, Cheng Deng, and Muli Yang. Siamese contrastive embedding network for compositional zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9326-9335, 2022.
[46] Yanghao Li, Naiyan Wang, Jiamping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.04777 2016.
[47] Zhizhong Li and Derek Hoiem. Learning without forgetting, IEEE transactions pattern analysis and machine intelligence, 40(12):2935-294) 2017.
[48] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pages 6028-6039. PMLR, 2020.
[49] Hong Liu, Jianmin Wang, and Mingsheng Long, Cycle self-training for domain adaptation. Advances in Neural Information Processing Systems, 01-22968-22988 2021.
[50] Zilin Luo, Yaoyao Liu, Bernt Schiele, and Qianru Sun. Class-incremental exemplar compression for class- inss-incremental learning. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 11371-11380 2023.
13

---

[51] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing. 469:28-51, 2022.
[52] Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 3589-3599, 2021.
[53] Muhammad Ferjad Nacem, Yongqin Xian, Federico Tombari, and Zeynep Akata. Learning graph embeddings for compositional zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 953-962 2021.
[54] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits natural images with unsupervised feature learning. 2011.
[55] Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via
transfer component analysis. IEEE transactions on neural networks, 22(2):199-210, 2010.
[56] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-135, 2010.
[57] Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In Proceedings of the IEEE international conference on computer vision, pages 754-763, 2017.
[58) Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, and QM Jonathan Wu. A review of generalized zero-shot learning methods. IEEE transactions on pattern analysis and machine intelligence. 2022.
[59]
Joaquin Quinonero- Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008
[60] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748 8763. PMLR, 2021.
[61] Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Percy Liang. Un- derstanding and mitigating the tradeoff between robustness and accuracy. arXiv preprint 1rXiv:2002.10716, 2020.
[62] Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, Arihant Jain, and Venkatesh Babu Radhakrishnan. A closer look smoothness in domain adversarial training. In International Conference on Machine Learning, pages 18378-18399. PMLR, 2022.
[63] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Puttern Recognition, pages 2001-2010, 2017.
[64] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. Advances in neural information processing systems, 33: 13:12222216222, 2020.
[65] Kendrick Shen, Robbie M Jones, Ananya Kumar, Sang Michael Xie, Jeff Z HaoChen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In International Conference Machine Learning, pages 19847-19878.
PMLR, 2022.
[66] Yujun Shi, Jian Liang, Wenqing Zhang, Vincent YF Tan, and Song Bai. Towards understanding and mitigating dimensional collapse in heterogeneous federated learning. ICLR, 2023.
[67] Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Online class- caas-incrremenal continual learning with adversarial shapley value. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9630-9638, 2021.
[68] Sebastian U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations, 2019.
[69] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test- time training with supersupe for generalization under distribution shifts. In International conference machine learning, pages 9229-9248. PMLR, 2020.
14

---

[70] Junjiao Tian, Zecheng He, Xiaoliang Dai, Chih- Yao Ma, Yen-Cheng Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tuning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7845, 2023.
[71] Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. Ger: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 99-108, 2022.
[72] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: 2016.
[73] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam. Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769-8778, 2018
[74] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 5018-5027, 2017.
[75] Mei Wang and Weihong Deng. Deep visual domain adaptation: survey. Neurocomputing, 312:135-18: 153, 2018.
[76] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning. pages
PMLR, 2020.
[77] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[78] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 239998. PMLR, 2022.
[79] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes. Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong. et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959-7971, 1, 2022.
[80] Tz-Yin Ying Wu, Gurumurthy Swaminathan, Zhizhong Li, Avinash Ravichandran, Nuno Vascon- celos, Rahul Bhotika, and Stefano Soatto. Class-incremental learning with strong pre-trained models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 9601-9610. 2022.
[81] Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4582 4591, 2017.
[82] Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. In-n-out: Pre-training and self-training using auxiliary information for -iitt--distriuution robustness. arXiv preprint arXiv:2012 2,04550, 2020.
[83] Wenjia Xu, Yongqin Xian. Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Attribute prototype network for zero-shot learning. Advances in Neural Information Processing Systems, 33:21969- 21 21980, 2020.
[84] Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Vgse: Visually- grounded semantic embeddings for zero-shot learning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 9316 (1166922,,20 2022.
[85] Cuie Yang, Yiu-Ming Cheung, Jinliang Ding, Kay Chen Tan, Bing Xue, and Mengjie Zhang. Contrastive learning assisted-aligmment for partial domain adaptation. IEEE Transactions on Neural Networks and Learning Systems, 2022.
15

---

[86] Jinyu Yang, Jingjing Liu, Ning Xu, and Junzhou Huang. Tvt: Transferable vision transformer for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 520-530, 2023.
[87] Shiqi Yang, Yaxing Wang, Joost Van De Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8978-8987, 2021.
[88] Jachong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-basee continual learning. arXiv preprint wrXiv:2106.01085 2021.
[89] Jaehong Yoon, Funho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv: 1708.01547, 2017.
[90] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Universal domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2720-2729, 2019.
[91] Xiaohua Zhai, Joan Puigeerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: /tt.:::///000466.. 2019.
[92] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint atw.i::190000400000
2019.
[93] Junting Zhang. Jie Zhang. Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang. and C-C Jay Kuo.
learning via deep model consolidation. In Proceed- ings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 1131-1140, 2020.
[94] Jike Zhong, Hong- Chen, and Wei-Lun Chao. Making batch normalization great in federated deep learning. arXiv preprint 2rXivv23333066530 2023.
[95] Da-Wei Wei Zhou, Qi- Wang. Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep class-ncremmnta. learning: A survey. arXiv preprint arXiv:2302.036668 2023.
[96] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. Proceedingy of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 16816-16825, 2022.
[97] Lanyun Zhu, Tianrun Chen, Jianxiong Yin, Simon See, and Jun Liu. Continual semantic segmentation with automatic memory sample selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3082-3092, 2023.
[98] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey transfer learning. Proceedings of the IEEE, 109(1):43-76, 2020.
16

---

Supplementary Material
We provide details omitted in the main paper.
Appendix A
related work (cf. subseetion 2.3 and Section of the main paper).
Appendix
additional benchmark details (cf. |subsection 2.2 the main paper).
Appendix
additional training details (cf. section of the main paper).
Appendix D
additional results and analyses (cf. |section 4 of f the main paper).
Appendix
additional discussions (cf. sectionn 5 of the main paper).
A
Related Work
We review related work other transfer learning paradigms. We briefly describe their settings and distinguish their differences from our proposed holistic transfer (HTT) problem.
A.1 Domain Adaptation
Domain adaptation (DA) is the most iconical machine learning setting to tackle the domain-shil problem 86 27 62
65 49] With the common objective of transferring source-domain know Enowledge to target domains, various settings have been proposed to incorporate different constraints and assumptions. The assumption can the degrees of overlap between the source and the target label sets [90, 64 5
85 57 9]. To relax the constraint of accessing source data, source-free DA can solely rely on the target data for adaptation [87] 23 43 18). Despite the abundant variations, DA settings all share one common assumption: the target distributions in training and testing matched, making our HT fundamentally different from them. In our HT,, we can encounter target test classes that unseen in the target training set but seen in the source domain. Therefore, HT requires a distinct ability that can generalize the style shifts learned on the target seen classes to other unseen classes.
A.2 Out-of-domali Generalization
Although fine-tuning a pre-trained model often leads to impressive accuracy for downstream tasks, recent studies have revealed that it may compromise the out-of-domain (OOD) robustness of the model [4 77]. Several robust fine-tuning methods are thus proposed to balance the trade-off
between in-domain downstream accuracy and OOD generalization [70 61 [2) LP-FT 41 proposed to learn a classifier with frozen features before end-to-end fine-tuning to avoid feature distortion. Some other approaches relied ensembles with pre- trained models to increase the robustness [78, (0). However, the main focus of these studies remains on preserving the robustness to different input styles for classes in the target training set. This is significantly different from HT. Our HT problem aims to generalize the styles for classes unseen in the target training set.
A.3 Continal Learning
The goal of continual learning (CL) is sequentially adapt multiple tasks without catastrophically
forgetting the previously learned ones 337 proposed to exploit replay buffer for storing old data [63
51
47
To achieve this goal, existing studies have
52 95], to I 24 77). Unlike HT, CL still assumes all the aligned with their corresponding test distributions. Although reducing forgetting can be the first step for HT to maintain unseen class accuracy, we argue that this is insufficient in HT due the source-target domain mismatch. Adapting the features for unseen classes to the target domain remains a key challenge for HT. Moreover, HT can also be potentially compatible with CL to consider learning non-iid data stream.
8 80 71, 67 88 50 97
constrain the fine-tuning with old models 89 22
encountered training distributions, which could be many,
A.4 Zero-shot Learning
Zero-shot learning tackles the setting where training and test classes are completely disjoint 8| 84 58). As no training data are available for test classes, the main challenge resides in learning source
17

---

Table 10: A summary of the dataset statistics for our HT benchmark.
Dalasets
Source domains
Target domain
MClasses
#Seen classes
#Target training
#Target test
Clipart Product
1,471
1,330
Art
65
30
1,265
1,361
Real
1,413
1,335
Office-Home
Art
8557
750
Real
Clipart
65
30
1,493
1,330
Product
1,459
1,361
Vary by data collection bias
Vary by data collection bias
Vary by data collection bias
FEMNIST
40 writers
10 new writers
62
53 camera trap
21 new camera trap locations
Vary by data collection bias
Vary by collection
Vary by data collection bias
iWildCam
181
locations
Caltech CIFARIO0
102
51
1,371
6,084
100
50
22,513
10,000
DTD
47
23
920
1,880
EuroSAT
10
5
8,424
5,400
VTAB
CLIP
Flowers 102
102
510
6,149
Pets
18
1,445
3,669
Resisc45
45
22
9,159
6,300
SVHN
28,197
26,032
SUN397
397
198
37,542
21,750
iNaturalist (Fungi)
CLIP
Fungi
12 6
30
60
features that can generalize to unseen semantic meanings. To achieve this, auxiliary information (e.g.. texts attributes) is usually needed to describe the test classes and connect them back to the training classes 88 15 53 45). In HT, we assume the missing classes in target domains are already seen in the source domain. We make this assumption to simplify the problem that HT can focus on generalizing the domain shifts to unseen classes. However, we argue that HT is compatible with zero-shot learning to make the setting more flexible.
B Additional Benchmark Details
To support the study of the HT problem, create a benchmark that covers extensive scenarios across both experimental and realistic public datasets. We provide details about these datasets.
B.1 Office-Homme
Setup. We consider the standard domain adaptation setting but with some missing classes in the target training sets. We use the popular Office- Home dataset consisting of 65 categories from domains (Art, Clipart, Real, and Product). In our benchmark, source domain is then transferred to each of the three remaining target domains individually, resulting in six source-target pairs. For each source-target pair, a source model. Then, for each target domain, we randomly split the data of each class into training and test sets with ratio of 7:3. We randomly sample 30 seen classes and combine the training data of these seen classes to create the target training set. Finally, the target test set consists of the test images of all 65 classes in the target domain. A summary of the statisties
Art and Real as source domains: each
all the data the source domain train
be found in Table I0) Evaluation. We follow the standard evaluation metric in the Office-Home dataset to compute the overall accuracy for each source-target pair. Besides, explicitly compute the accuracy of the unseen-class data to evaluate the transferring performance of the unseen classes. The average accuracy over all the source-tan saureetarget pairs is also reported.
B.2 FEMNIST
Setup. The FEMNIST dataset contains 62-class hand-written characters from many writers with different writing styles. practical personalized Al applications, common to train on data from many users but needs to adapt many users after the (source) model is trained [T] As
18

---

can only collect limited- size data set for each writer, each writer's data only cover a subset of the 62-class characters, resulting in the need for HT. We randomly sample 40 writers whose data combined can cover all 62 classes and use their data to train source model. Then, we randomly sample 10 new writers. Each new writer's data is divided into training and test sets in a ratio of 7:3. Note that each client may not have enough images per class, which creates a realistic scenario of personalization with limited samples, which results in a mismatch of the class distributions between training and test sets. The dataset statistics are summarized in Table 10
Evaluation. We report the overall accuracy averaged over all the 10 new writers. To evaluate the trade-off between seen and unseen classes, we also report the averaged accuracy the seen and unseen classes, respectively, As this dataset has no oracle training set for each new writer, we report the seen accuracy computed by chopping out unseen classes in the classifier evaluate the quality of the adapted features.
B.3 iWildCam
Setup. We consider a realistic scenario of HT, where we initially have abundant camera traps installed across many geo-locations (source domains) and now need to transfer to a new camera trap location (target domain). In the new location, we can only use the data collected within fixed amount of time in the beginning (e.g., the first month) as our target training set. As it is impossible for all the animal species to appear in the first month, the target training data can be biased toward some classes that show up. This is a natural data collection bias caused by time.
We start from the iWildCam dataset in the WILDS 38) benchmark. As we mainly focus on animal species classification, we remove the "empty" class for simplicity and thus obtain a total of 181 classes. For each camera trap location, we sort the images by their timestamps and group images into sequences if the difference in their timestamps is smaller than 30 minutes, to avoid information leaks. We randomly sample 53 camera trap locations whose images cover all 181 classes and use all their data train a source model. For each of the remaining locations, we randomly sample training and test sets based on ratio of 7:3, We only keep locations with more than 500 images in both the training and test sets, thereby resulting in 21 new locations for adaptation. For each new location, we form the target training set by sorting the training images by time and only using the first 25% of them. A summary of the dataset statistics is given in Table T0
Evaluation. We report the overall accuracy averaged over all 21 new locations. To evaluate the trade-off between seen and unseen classes, we also report the averaged accuracy on the seen and unseen classes, respectively. As this dataset has oracle training set for each new location, report the seen accuracy computed by chopping out unseen classes in the classifier to evaluate the quality of the adapted features.
B.4 VTAB
Setup. We consider another practical use of HT by going beyond domain adaptation and fine-tuning the zero-shot CLIP [60] for distribution shifts at the task levels. We use the VTAB [92 benchmark that includes various image classification tasks. To enable zero-shot predictions, we only use the tasks that provide text names for classes, thereby resulting in 9 tasks: Caltech101, CIFARIOM 100, DTD, EuroSAT, Flowers 102, Pets, Resisc45, SVHN, and SUN397. We use the standard training and test sets provided by the VTAB benchmark. Then, randomly sample half of the classes as seen and the remaining as unseen. The target training set only includes the training images of the seen classes, while the target test set contains all the test images. A summary of the statistics of this dataset is shown in Table 10
Evaluation. Following the standard evaluation in VTAB, we report the overall accuracy for each of the 9 tasks. Besides, we also compute the accuracy on the unseen-class data to evaluate the transferring performance of unseen classes. Finally, the average accuracy across all 9 tasks is also reported.
B.5 iNaturalist (2021 Version, Fungi)
Setup. To demonstrate the impact of visually similar classes in HT, we carefully pick 6 pairs of fungi classes from the iNaturalist dataset, thus resulting in a total of 12 classes. Each pair of fungi classes corresponds to 2 species of visually similar fungi; one is non-toxic, while the other one is toxic. We
19

---

use the zero-shot CLIP model with the fungi names as our source model. Then, the training images from the 6 non-toxic fungi classes form the target training set. The target test set consists of all the test images from all 12 classes. A summary of the dataset statistics and some examples are shown Table 10
We report the seen accuracy on the target test set to evaluate the adaptation performance. As wrongly predicting toxic fungi as non-toxic ones can result in severe outcomes, we also report the false negative rate, which is computed as the percentage of the images of toxic fungi being predicted as non-toxic fungi classes.
Evaluation.
C Additional Training Details
We provide the training details for our results reported in section 4
For the Office- Home dataset, we initialize a ResNet-50 with ImageNet pre- pre-trained weights. Then, we train it on the source domain for 20 epochs using the SGD optimizer with a learning rate Ie-3, momentum 0.9, weight decay Se-4, and batch size 64. For all methods that adapt to the target domains, we fine-tune the source model for 20 epochs using the SGD optimizer with a learning rate le-4, momentum 0.9, weight decay 5e-4, and batch size 64. For our suggested HT methods, we set the hyper-parameters Caistill = 10 and Crank 100.
For the FEMNIST dataset, we train a LeNet from scratch on the data of the 40 source writers for 100 epochs using the SGD optimizer with a learning rate le-2, momentum 0.9, weight decay 5e-4, and batch size 32. To adapt to each new writer, we fine-tune the source model for 10 epochs using the SGD optimizer with a learning rate 1e-3, momentum 0.9, weight decay le-4, and batch size 32. We set the hyper- parameeerssssss Laisnll 0.1 and Lrank 10.
For the iWildCam dataset, train ResNet-50, which is initialized with ImageNet pre-trained weights, on the data of source camera trap locations for 50 epochs using the SGD optimizer with a learning rate 3e-5, momentum 0.9, weight decay 0.0, and batch size 16. When adapting each new location, we fine-tune the source model for 20 epochs using the SGD optimizer with learning rate 3e-6, momentum 0.9, weight decay 0.0, and batch size 16. We set the hyper-parameters Cdistll 50 and Cr.m 200.
For the VTAB benchmark, we the class names for each of the 9 tasks to form the zero-shot CLIP models, which are ViT-B/32. We fine-tune the source model on target tasks for 20 epochs using the SGD optimizer with learning le-5, momentum 0.9, weight decay 0.0, and batch size 64, We set the hyper-parameters Laisill = 1 and Lrank = 5.
For the iNaturalist Fungi dataset, we use the fungi species names to build zero-shot CLIP model with a ViT-B/32 architecture. We then fine-tune the source model on the target training set for epochs using the SGD optimizer with learning rate 5e-5, momentum 0.9, weight decay 0.0, and batch size 5. We set the dyper-parameter Ldistill and Lrank =
D Additinal Results and Analyses
D.1 Variances of the Results in sesettiinn 4
We provide variances of results reported in our main paper. We compute the variances across random seeds. [able IT shows the variances of the test accuracy on Office-Home, The variances of the mean accuracy on FEMNIST and iWildCam Finally, Table 14 gives the variances of the test accuracy for each of the tasks in VTAB. These results reveal that the reported accuracy is relatively robust across random seeds.
provided in Table 12 in Table 13] respectively.
D.2 More Discussions on Seen and Unseen Classes
Different numbers of images per seen class. In the real world, it is unrealistic for end-users to collect data for all classes before adaptation. To further consider lower data collection cost, we reduce the number of training images per seen class to study its effects. We conduct the experiment on the Office-Home dataset with "Art" as our source domain and "Clipart" as our target domain. Specifically, we randomly sample 10% and 50% of the training images for each seen class and
20

---

Table 11: Varainces of domain adaptation 65-way test accuracy on Home with 30 seen and 35 unseen classes (ef. Table 3). We compute the variances random seeds. Blue: HT methods suggested by us in Ieseeciion 3 Red: methods that significantly improve overall accuracy and successfully maintain unseen accuracy
on the source model. $ the linear classifier is frozen during training.
Domains: sourc--targe
Ar-+CI
Ar-PP Unseen
Ar-RR Overall
Rw
Rw-sCI
Rw-sPr
Avg. Overall Unsoen
Methods/ / Avc.
Overall Unseen
Overall
Unseca
Overall Unseen
Overall
Unsen
Overall Unseen
Soures
0.61
0.11
1.30
1.35
0.25
0.24
0.23
1.40
0.11
Naive Target
0.02
0.01
0.14
0.02
0.27
0.53
0.93
1.19
0.00
0.36
0.35
1.11
0.04
0.09
BN only
1.11
3.11
0.22
0.27
0.41
1.07
1.71
7.58
0168
2.44
1.43
4.47
0.03
0.17
BN (stals) only
0.63
0.06
1.00
1.63
0.02
0.32
0.08
0.17
0.16
0.31
0.15
0.05
0.14
0.16
(stats) only
0.57
0.58
0.25
0.18
0.21
0.02
0.03
LP-PT
0.14
0.14
0.05
0.04
0.08
0.18
0.16
0.61
0.01
0.31
0.35
0.87
0.02
0.04
SGD (w/ frozen classifie %)
0.02
0.52
0.63
1.27
0.75
0.98
0.07
0.31
0.57
0.85
0.09
0.60
0.07
0.18
SGD Last &
0.28
0.09
0.01
0.70
0.73
0.05
0.09
0.15
0.01
SGD f... &
1,02
0.48
0.88
1.13
0.01
0.07
0.67
0.05
SWA 8
0.23
0.73
0.62
1.44
0.33
0.60
0:45
0.31
1.39
3.39
0.13
0.65
0.02
0.15
SWAD
0.13
0.83
0.62
1.79
0.35
0.42
0.15
0.32
1.64
3.71
0.08
0.41
0.03
0.17
LOLSGD
0.36
0.20
0.62
1.00
0.20
1,25
0.73
0.61
1.20
1.59
0.17
0.65
0.00
0.11
LOLSGD + +re
0.02
0.05
0.24
0.95
0.06
0.05
0.34
0.22
0.63
1.09
0.16
0.96
0.04
0.14
LOLSGD + Caa *
0.14
0.54
0.25
0.09
0.28
1.88
0.07
0.01
LOLSGD Caaa Cuss 8
0.18
0.06
0.75
1.82
0.00
0.34
0.26
0.65
0.75
0.36
0.01
0.23
0.12
0.09
Omocle
0.05
0.05
0.16
0.01
0.16
0.17
0.22
0.05
0.35
0.20
0.02
0.00
Table 12: Varainces of FEMNIST mean accuracy of 10 new writers (cf. Table 5
Table 13: Variances of WWildCan mean accuracy of 21 new locations (cf. Table 6) We compute the variances over 3 random seeds.
We compute the variances
over 3 random seeds.
Methods
Overall Seen
Seen (Chopping)
Unseen
Methods
Overall Seen
Seen (Chopping)
Unseen
Source Naive Target
0.12
0.07
5.12
Source Naive Target
0.12 1.66
1.45
5,29
0.49
0.77
0.77
5.69
3.85
4.45
0.04
SGD
0.43
0.17
0.11
1,90
SGD
0.10
1.48
1.67
0.01
SGD + Luck AF
0.55
0.08
0.04
4.05
SGD + Lia 8
0.94
7.73
2.41
2.35
SGD + Caen $
0.37
0.19
0.13
1.72
SGD + Laon A
3.76
1.26
1.32
4.07
LOLSCD
0.48
0.10
0.17
4.73
LOLSGD
0.79
0.37
0.76
1.63
LOLSGD -rrm 8
0.37
0.10
0.20
2,04
LOLSGD +r A
1,70
7.85
5.03
3.98
LOLSGD Lamaa
0.16
0.37
4.90
LOLSGD +a $
1.33
0.67
1.86
3.36
LOLSGD - Laso +S
0.65
0.26
0.06
6.23
LOLSGD Ca +SE:3
1.17
1.07
2.58
0.15
fine-tune the source model for the same iterations for fair comparisons. Interestingly, Table 15 shows that naive fine-tuning can obtain higher unseen accuracy, compared to naive fine-tuning with more data. The reason might be that training with more data needs to update the model weights more, making the unseen classes easier to forget. contrast, applying our suggested HT methods, especially for LOLSGD with our regularization, the unseen classes can be better maintained across different training data sizes.
Unseen performance vs. overall performance. One might observe that sometimes the unseen performance could be better than the overall performance. We surmise this because the classes are not equally difficult for classification. In some cases, before we perform adaptation, we can already see that the unseen class accuracy of the source model is higher than its overall accuracy, meaning that those unseen classes are inherently easier than the seen classes. Some methods (e.g.. BN (stats) only) can better keep the unseen accuracy close to the source model with less forgetting. However, they cannot adapt the seen classes effectively to the target domain. Therefore, the results of these methods generally follow the trend in the source model with the unseen accuracy higher than the overall accuracy. In contrast, some methods (c.g.. LP-FT) can better adapt the seen classes to the target domain but suffer from serious forgetting of the unseen classes. These methods thus have lower unseen accuracy.
New classes in the target domain. We note that, in our current setup all the experiments assume the classes tested in the target domain appeared in source training. If some "seen" classes in the target domain are not the source domain, this will require expanding the label space of the model. A simple baseline would be first training the classification weights for those "seen" classes from scratch while keeping all other model components intact. Then, we can apply our holistic transfer approach to the expanded model. A more sophisticated solution would involve techniques from continual
21

---

Table 14: Variances of test accuracy for tuneitunng CLIP VIT- on VTAB (cf. Table 7
50% of classes in
each task missing during training. We compute the variances over random seeds.
Overall/Unseen Acc.
Caltech101
CITAR
DTD
TTTSATT
Flowers
Pets
Resisc45
SVHN
SUN399
Uns
All
Umm
AlL
Uns
All
line
All
Une
All.
1nm
AlL.
AlL
Uns
A1L
Une
Mothods
0.00
0.00
10.00
Target
0.00
1.32
0.02
0.30
00.05
0.90
SGD
0.00 0.01
0.02
0.17
0.38
0.59
0.16
0.03
0.15
0.00
0.02
0.02
LOLSGD
0.01 0.00
0.01
0.03
0.39
1.33
0.16
0.69
0.21
0.12
0.03
0.13
0.01
0.07
2.29
0.14
0.06
0.29
0.04 0.01
LOLSGG Casanl
0.00
0.01
0.04
0.00
0.13
0.20
0:01
0.01
1.80
0.00
0.00
0.00
LOLSGD C.ck 84
0.00
0.01
0.00
0.02
0.07
0.07
0.17
0.34
0.06
0.02
0.02
0.01
1.37
0.36
0.03
0.05
0.01 0.00
LOLSGD - Crok +
0.01
0.02
0.07
0.14
0.12
0.33
0.06
0.05
0.06
0.03
0.00
0.03
0.05
0.09
1991
0.82
0.01
0.05
0.00
0.02
10:29
(0.00
0:40
0.50
00.01
0.07
0.27
10.02
10.00
0.00
0.00
0.01
|0.02 0.03
Table 15: Different percentages of the target training data each seen class on Office Home: Ar - CL.
% of target training
10%
50%
100%
Methods/Acc.
Overall
Unseen
Overall
Unseen
Overall
Unseen
Source
47.07
50.29
47.07
50.29
47.07
50.29
Naive Target
41.73
19.74
43.46
9.94
44.96
9.06
SGD
51.28
41.96
52.33
28.36
52.11 11
24.12
SGD + Lonk
50.45
46.49
56.02
40.20
59.17
39.47
SGD + dasall
48.80
41.37
53.91
38.30
56.54
39.18
LOLSGD
52.63
46.20
54.21
34.94
56.47
35.09
LOLSGD + Lrink 2
51.13
48.83
55.86
44.74
58.57
43.86
LOLSGD + Edisall
51.28
45.91
55.19
44.88
57.44
46.35
LOLSGD + Ldisril + Lrnkk 8
51.05
50.88
58.65
52.05
60.83
51.75
learning, or more specifically, class-incremental learning. This machine learning paradigm aims to expand the label space of model. We leave a suitable combination of our approach and techniques from class-incremental learning as future work.
D.3 Effects of the Source Ensemble Coefficients
In sessecion 4 we apply Source Ensemble (with a mixing coefficient a = 0.5) to reclaim some ability of the source model to maintain the unseen accuracy (cf. TTale 4). To further understand the trade-off between the source and the fine-tuned target models, we study the effects of the mixing coefficient by varying it between [0, 1]. We conduct our study on Office-Home and report the overall and unseen accuracy averaged over all the torrce-target domain pairs. As shown in Figure applying either SE or WiSE cannot save the naively fine-tuned target model from being heavily biased seen classes. On SGD with frozen classifiers, our SE shows better trade-off than WISE [79]. Finally, fine-tuning target models with our suggested HT methods can clearly yield the best trade-off.
D.4 Comparison CoCoOp 96 CLIP
In our experiments, we use CLIP to construct the source models for the VTAB and iNaturalist experiments. We further compared to a recent CLIP fine-tuning method CoCoOp (96) in these experiments. CoCoOp fine-tunes CLIIP by training meta-net to condition the classification weights (i.c., the fully-connected layer) on the input images. In other words, CoCoOp freezes the visual features but changes the classifier weights by minimizing the standard cross-entropy loss. In contrast, our approach freezes the classifier weights but adapts the visual features by minimizing the loss designed specifically for HT.
We conduct two experiments: 1) CoCoOp alone for the HT problem, and 2) combining CoCoOp with our approach. For 2), we take the resulting model after CoCoOp as the improved source model and further adapt the feature. We report the result of the CIFAR- 100 task in VTAB (cf. Table 7). As shown in Table 16, CoCoOp alone performs well on unseen classes, but it improves the overall accuracy only marginally. Combining both approaches, we can obtain better accuracy in unseen and seen classes, leading to the best overall accuracy.
22

---

Naive larget (SE)
Target
60
SGD (SE)
Unseen accuracy
SGD (WiSE)
LOLSGD Casse (SE)
50
LOLSGD Crau
Source Model
40
30
55.0
57.5
60.0
62.5
65.0
67.5
70.0
Overall accuracy
Figure 6: Effects Source Ensembles. Ensemble the source (the star marker) and target models (the end point each from source model) with mixing enefficient a E [0, 1] on Office-Home.
D.5 More experiments with the updating BN baseline
Batchnorm layers are critical components in feed-forward neural networks especially when learning with distributional shift [46 94]. In cf.. Table 3 one may observe the baseline "BN (stats only)" of updating only the means and variances batchnorm layers performs surprisingly well on the unseen accuracy. We note that the ultimate goal of HT is to achieve high overall accuracy, not merely unseen accuracy. Although "BN (stats only)" maintains the unseen accuracy well on the Office-Homm dataset, it cannot effectively improve the seen accuracy, resulting in worse overall accuracy. We apply "BN (stats only)" iWildCAM (cf. TTall 6). As in Table cannot improve the seen accuracy. That said, we think that more sophisticated, dedicatedly designed version of BN might perform better, and we leave it as future work.
maintains the unseen accuracy well but
D.6 study for LOLSGD canceling out class biases
Our LOLSGD aims to disentangle them or, more precisely, reduce the disruptive concept shift by subsampling multiple datasets that each contain a subset the seen classes. When updating the model separately in parallel with these subsampled datasets, each updated model is affected by different concept shift but a shared covariate shift. By averaging these models, LOLSGD can potentially cancel out the disruptive concept shifts and strengthen the shared covariate shift. We provide more evidence that LOLSGD can cancel out the disruptive concept shifts. We compare the seen class accuracy among 1) naive fine-tuning with the partial target data, 2) LOLSGD, and 3) fine-tuning with full target data (i.e., the oracle model). As shown in Table 19 the seen class accuracy of naive fine-tuning (black dotted line) exceeds the oracle (green dotted line), indicating that naive fine-tuning learns undesired concept shifts towards seen classes, leading to an unreasonable accuracy. In contrast, the accuracy of LOLSGD (red dotted line) consistently stays below the oracle, indicating that undesired concept shifts are reduced. As a result, LOLSGD obtains much higher unseen accuracy than naive fine-tuning.
23

---

Table 16: Results of CoCoOp with VIT-B/32 CI- FARI00. 50 classes missing during training (50 seen classes), We treat trained CoCoOp a source model and apply our approaches.
Table 17: IWildCAM mean accuracy of 21 new locations. We extend cf. Table 6 to compare baseline "BN (stats) only", which update means and variances BN layers.
Methods
Overall Unseen
Methods
Overall Seen Unseen
Source (CLIP) Naive Target
64.18
62.42
65.94
42.66
Source
35.71
35.74
26.43
Naive Target
35.38
51.08
1.90
LOLSGD
69.59
51.78
LOLSGD Cdisill &
67.13
49.08
BN (stats) only
30.18
29.69
23.25
LOLSGD + Crank &
69.99
52.54
SGD
36.53
49.00
7.91
LOLSGD + Lrank +SB
72.31
58.38
SGD rank
#
36.28
40.98
19.76
CoCoOp
66.16
63.08
SGD Caium
A
41.59
50.40
17.90
CoCoOp LOLSGD
72.58
64.20
LOLSGD
38.12
48.91
13.30
CoCoOp + LOLSGD +Listil 42
69.34
61.90
LOLSGD +aan
33.94
37.62
19.42
CoCoOp + LOLSGD Link
72.77
64.36
LOLSGD +hdim A
40.49
47.30
25.16
CoCoOp LOO.SGD ma +S
72.34
65.52
Oracle
83.10
82.62
LOLSGD + +SE+
39.96
44.79
25.66
Table 18: Results Office-Home with 30 seen and 35 classes. We combine updating BN with partially tuning the last block ResNet-50. accuracy averaged over 6 source-target pairs used in our paper.
80
Test accuracy
Methods Acc.
Overall Unseen
40
Source
63.90
65.65
Naive Targei Unowen
Naive larqet Sean
Naive Target
53.50
22.36
20
BN (stats) only
64.27
67.51
BN (stats) only Partially tune
57.96
30.63
0
8
10
Training epochs
BN (stats)
57.60
32.36
BN (stats) + Partially tune
57.68
29.39
Table 19: An HT example showing that I.OLSG) cancel out undesired class biases learned by Naive Target. use - domains on Office-Home
Oracle
72.01
66.54
E
Additional Discussions
E.1
Limitations
In this paper, we introduce novel and practical transfer learning problem, holistic transfer, that emphasizes the generalization to domain shifts for classes unseen in the target domain but seen in the source domain. We establish strong baselines and demonstrate the potential for simultaneously improving both seen and unseen target classes. One potential limitation is that we mainly focus on vision classification tasks. We leave the studies image segmentation/ohject detection and natural language processing tasks as our future work. We also plan to explore better approaches for the disentanglement of domain styles and classes and integrate our approach with other learning paradigms, like test-time adaptation.
E.2 Potential Negative Societal Impact
The goal of our work is to introduce and study a practical transfer learning problem, holistic transfer. We provide strong baselines and analyze problem on publicly available datasets, which are adjusted and split to meet our problem setting. As far we know, our work does not introduce additional negative societal impacts compared the standard transfer learning topics, like domain adaptation and out-of-distribuute generalization.

---

E.3
Computation Resources
We conduct our experiments on PyTorch and on NVIDIA VI0O GPUs. On the Office-Homm dataset, fine-tuning for 1 target domain with all the compared methods and random seeds takes roughly 36 hours on GPU, Similar time consumption also applies to iWildCam and VTAB datasets. On the smaller FEMNIST dataset, it takes roughly 0.5 hours on GPU to get the required results for target domain. The whole experiment on iNaturalist Fungi takes roughly 0.5 on GPU. total, our experiments take roughly 1.3K GPU hours.
25


Reviews:
[{"review_id": "MG4LDrOOIr", "reviewer": "Reviewer_QPVq", "summary": "This paper proposes a problem of fine-tuning the source model with partial target data, where the source and target distributions have covariate shifts and the target test data contain classes unseen in the target training data (also called Holistic Transfer, HT, in this paper). It proposes Leave-Out Local SGD (LOLSGD) to disentangle domain gradients from classification gradients during training and several regularizations to preserve class relationships. In the experimental part, this paper builds some holistic transfer benchmarks based on existing domain adaptation and fine-tuning datasets and evaluates the proposed methods.", "questions": "Some questions are listed in ‘Weaknesses’, and some more questions here:\n\n* How would the performance change if the number of classes in the target training data is changed?\n \n* Would the increase in unseen classes sacrifice the performance on seen classes? \n \n* In Equation (5), how can we get argmin_theta?", "limitations": "The authors point out the limitation that this paper mainly focuses on vision classification tasks and leave the studies to image segmentation/object detection and natural language processing tasks as their future work.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* This paper generally writes well. The analysis of the dilemma in the problem makes sense, and the proposed methods are easy to understand.\n\n* This paper builds some benchmarks for the proposed holistic transfer problem and evaluates some methods on them. Some of the evaluations and results may help future works on related topics.\n", "weaknesses": "* The proposed problem does not seem very interesting or practical to me. In my opinion, a practical problem should be simple and realistic. For example, in the problem of fine-tuning, we only need to have a pre-trained model and some target training data. However, in the proposed holistic transfer problem, there exist too many constraints and assumptions (for example, it requires that the source domain covers all the labels in the target domain), which may not be satisfied in real applications. In Table 1, the authors give some examples of the HT problem, but they actually contain two different settings, making the definition of the problem a bit confusing. The first three examples are variants of source-free domain adaptation, with partial classes in target training data. The last two examples are fine-tuning of CLIP models, and CLIP models are actually self-supervised pre-trained, so it does not fit the definitions in Section 2.1. Besides, the problem of fine-tuning CLIP and testing on unseen classes has already been proposed by previous works [1].\n\n* The technical novelty of the proposed methods is limited. The LOLSGD seems similar to local SGD and meta-learning. It is unsure why its design can disentangle covariate shifts. What is its difference from using a larger batch size for SGD? The class relationship regularizations and ensemble methods also seem to be existing techniques.\n\n* This paper only discusses the problem itself and evaluates some proposed techniques. Some important related topics and methods are not discussed and compared in this paper. The source-free domain adaptation [2] and its following works are highly related but are not compared. Some fine-tuning methods may also solve the problem but are also missing, such as [3][4]. For the CLIP fine-tuning, some existing works already explore the problem of unseen classes [1], which are not mentioned.\n\n[1] Conditional Prompt Learning for Vision-Language Models.\n\n[2] Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation.\n\n[3] DELTA: Deep Learning Transfer using Feature Map with Attention for Convolutional Networks.\n\n[4] Catastrophic Forgetting Meets Negative Transfer: Batch Spectral Shrinkage for Safe Transfer Learning.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The proposed problem does not seem very interesting or practical … too many constraints and assumptions … Table 1 actually contains two different settings …** We want to reiterate that the proposed problem is fairly *practical* and *realistic*. It considers a *practical* scenario where *an end-user wants to adapt a pre-trained model’s capability of recognizing $N$ classes to the target domain.* In the literature, this typically requires preparing a target data set that covers all the $N$ classes, which can be challenging or even *unrealistic*, especially when $N$ is huge (L1-3, L16-21). The proposed problem releases this constraint, making the data preparation for fine-tuning *much simpler* for end-users. \n\nWe apologize if Sect. 2.1 gives the wrong impression that our proposed problem introduces many constraints and assumptions, and we will carefully refine it. Sect. 2.1 is meant to describe our setting and ground it in the literature formally. The assumptions we mentioned (L54-63) either follow existing works or help clarify our setting. For instance, *the source domain covers all the labels in the target domain* is a direct property of the aforementioned practical scenario; the assumptions in L58-63 actually *relax* the assumption made in L124-125 by existing works. We should not have used the word “constraints” at L69. Indeed, L70-85 simply describes the properties and challenges of the proposed problem. We consider a more realistic source-free setting (L83-85) as it removes the need to access the source data. \n\nWe also apologize for the confusion in Table 1. All five cases strictly follow the setting of holistic transfer — adapt a pre-trained model when the available target data only contain partial classes. We use CLIP mainly to extend our study to VTAB (Table 7) and iNaturalist (Table 9), for which we have no explicit source data to pre-train the model. We use CLIP’s zero-shot capability only to construct the source classifier, and we disregard its text encoder afterward. In other words, our problem definition has nothing to do with CLIP; our methods are not designed specifically for CLIP. We will clarify this in the final version. \n\n**Technical novelty.** We want to emphasize that our main contribution is studying the holistic transfer (HT) problem. The novelties are thus not merely technical but involve the investigation and understanding of the problem and the construction of benchmarks. For instance, Sect. 2.3 reveals the challenges of HT, which leads to the two directions of solutions (Sect. 3.2 and 3.3). Within each, instead of directly proposing new techniques, we deliberately seek existing techniques (initially developed for other problems) that can potentially address the challenges. While knowing that this approach might not result in many technical novelties, we believe it is crucial and valuable, as it helps establish the foundation of HT and connect it to the existing machine learning techniques. \n\nFor the novelty and detail of our proposed LOLSGD, please be referred to the global response.  \n\n**Related topics and methods.** Thank you for the comment and references. We will cite them and add more discussions on related topics. We did include the frozen linear classifier (L210-212) inspired by source-free adaptation (SFDA) in our experiments. Nevertheless, SFDA does not address adapting a pre-trained model using partially available target data. [3] and [4] aim to fine-tune pre-trained backbones for new downstream tasks. Their focus is to reduce over-fitting, for example, via distillation [3] or regularization [4]. Therefore, they are similar to the selective distillation and feature rank regularization (L213-226) in our experiments, which, however, could not fully solve the HT problem. \n\nFor CLIP fine-tuning, we want to reiterate that CLIP is not our main focus. We use CLIP only to construct the source models for the VTAB and iNaturalist experiments. That said, we follow your comment to compare to CoCoOp [1] in these experiments. CoCoOp [1] fine-tunes CLIP by training a meta-net to condition the classification weights (i.e., the fully-connected layer) on the input images. In other words, CoCoOp freezes the visual features but changes the classifier weights by minimizing the standard cross-entropy loss. In contrast, our approach freezes the classifier weights but adapts the visual features by minimizing the loss designed specifically for HT.\n\nWe conduct two experiments: 1) CoCoOp alone for the HT problem, and 2) combining CoCoOp with our approach. For 2), we take the resulting model after CoCoOp as the improved source model and further adapt the feature. We report the result on the CIFAR-100 task in VTAB (cf. Table 7). As shown in Table R1 (see the PDF in the global response), CoCoOp alone performs well on unseen classes, but it improves the overall accuracy only marginally. Combining both approaches, we can obtain better accuracy in unseen and seen classes, leading to the best overall accuracy. \n\n**Number of seen and unseen classes.** We conduct an experiment on OfficeHome (cf. Table 3). Starting from 10 seen classes, we gradually include more seen classes. As shown in Table R3, the overall accuracy consistently improves when the number of seen classes increases. \n\nTo answer whether the increase of unseen classes (equivalently, the decrease of seen classes) would sacrifice the performance of seen classes, we calculate the seen class accuracy on the 10 commonly seen classes — this makes the accuracy comparable across settings. As shown in Table R3, the accuracy on the 10 commonly seen classes actually improves when we decrease the number of seen classes. We surmise that this is because when fewer seen classes are available, the adapted model will focus more on the 10 commonly seen classes.\n\n**Equation (5).** The argmin_theta means training the model on a locally sampled class subset in LOLSGD. We minimize the loss using SGD for several steps. We will clarify this. "}, {"review_id": "Xg0ZIRRkDf", "reviewer": "Reviewer_tg8m", "summary": "This paper proposes \"Holistic Transfer\" as an important problem and also provides some solutions to it.  \"Holistic Transfer\" handles the situation when adapting a pre-trained source model (e.g. with 1000 classes) to a target domain (e.g. with 100 classes), but there are only data for part of the target domain (e.g. only with target domain data for 50 classes and no target domain data for the other 50 classes). Taking classification as an example, it assumes that all the target domain classes are in the source data, but may not be all in the target domain data.  Usual fine-tuning can improve the performance on seen classes (present in the target domain data), but may destroy the performance on unseen classes (not present in the target domain data). \n\nThe paper proposes some solutions in handling these situations, aiming to preserve the performance on unseen classes. In the proposed approach, the changes in the target domain are decomposed into two types of changes: \"style\" change and class change.  Style change can be learned via the seen classes of the target domain, assuming that the unsee classes are with the same style change. The information from the unseen classes are preserved via distillation and feature rank regularization.\n\nExperiments were conducted on various datasets to validate the approach.", "questions": "One question regarding the results in Table 1, from some rows, e.g. the row of \"BN (stats) only\", the \"Unseen\" performance is better than \"Overall\" performance, why is that?", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem proposed in the paper is valid and practically useful problem. As large foundation models gains popularity, how to adapt those models to specialized target domains effectively is of practical importance.  The paper handles the situation when the data for some classes of the target domain classes are missing. \n\nThe proposed approach seems to be sound. It decomposes the changes in the target domain into two types of changes: \"style\" change and class change.  Style change can be learned via the seen classes of the target domain, assuming that the unsee classes are with the same style change. The information from the unseen classes are preserved via distillation and feature rank regularization.\n\nThe effectiveness of the proposed approach is validated on various datasets.", "weaknesses": "The paper assumes that both the \"seen\" and \"unseen\" classes from the target domain are subsets of the classes in the source domain.  It may happen that some \"seen\" classes in the target domain are not in the classes of the source domain.  It would interesting to see how these cases are handled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive and constructive feedback on our paper.\n\n**The paper assumes that both the \"seen\" and \"unseen\" classes from the target domain are subsets of the classes in the source domain. It may happen that some \"seen\" classes in the target domain are not in the classes of the source domain. It would interesting to see how these cases are handled.** \n\nThank you for the comment. If some “seen” classes in the target domain are not in the source domain, this will require expanding the label space of the model. A simple baseline would be first training the classification weights for those “seen” classes from scratch while keeping all other model components intact. Then, we can apply our holistic transfer approach to the expanded model. A more sophisticated solution would involve techniques from continual learning, or more specifically, class-incremental learning. This machine learning paradigm aims to expand the label space of a model. We leave a suitable combination of our approach and techniques from class-incremental learning as future work.\n\n**One question regarding the results in Table 1, from some rows, e.g. the row of \"BN (stats) only\", the \"Unseen\" performance is better than \"Overall\" performance, why is that?**\n\nWe surmise this is because the classes are not equally difficult for classification. In some cases, before we perform adaptation, we can already see that the unseen class accuracy of the source model is higher than its overall accuracy, meaning that those unseen classes are inherently easier than the seen classes. Some methods (e.g., BN (stats) only) can better keep the unseen accuracy close to the source model with less forgetting. However, they cannot adapt the seen classes effectively to the target domain. Therefore, the results of these methods generally follow the trend in the source model with the unseen accuracy higher than the overall accuracy. In contrast, some methods (e.g., LP-FT) can better adapt the seen classes to the target domain but suffer from serious forgetting of the unseen classes. These methods thus have lower unseen accuracy.\n"}, {"review_id": "0xb9nRvXBV", "reviewer": "Reviewer_9DJT", "summary": "This paper studies a learning problem, called Holistic Transfer, which involves the adaptation of a pre-trained source model capable of classifying a wide range of objects, to a target domain using data that covers only a partial label space. To solve this problem,", "questions": "1. This paper is not well-structured. Considering that the learning setup is proposed in this paper. It is more important to convince me the new setup is valuable. Now the introduction is too short and evidence is not convincing.\n2. The learning setup is somewhat trivial. Researchers initially noticed distribution shift problem because the distribution shift is from sampling bias. It is okay to assume that the distribution of labels is different. In this case, sampling bias exists. It is confusing to assume that the marginal distribution of $T$ and $T^*$ is the same one.\n3.  The name \"Holistic Transfer\" is also confusing for me. What is \"Holistic\" means?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Distribution shift exists everywhere in real applications.\n2. Experiments show the effectiveness of the proposed method.", "weaknesses": "1. This paper is not well-structured. Considering that the learning setup is proposed in this paper. It is more important to convince me the new setup is valuable. Now the introduction is too short and evidence is not convincing.\n2. The learning setup is somewhat trivial. Researchers initially noticed distribution shift problem because the distribution shift is from sampling bias. It is okay to assume that the distribution of labels is different. In this case, sampling bias exists. It is confusing to assume that the marginal distribution of $T$ and $T^*$ is the same one.\n3.  The name \"Holistic Transfer\" is also confusing for me. What is \"Holistic\" means?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**It is more important to convince me the new setup is valuable. Now the introduction is too short and evidence is not convincing.** We apologize for not making the motivation clear. We provide a detailed motivation as follows. We will incorporate it in the final version to expand the introduction.\n\nThe proposed setting considers the practical scenario where *an end-user wants to adapt a pre-trained model’s capability of recognizing $N$ classes to the target domain* (L16-19). In the literature, this typically requires preparing a target data set that covers all the $N$ classes, which can be challenging or even *unrealistic*, especially when $N$ is huge (L20-21). The proposed setting aims to release this constraint, making the data preparation for fine-tuning *much simpler* for end-users. \n\nMore specifically, in most of the literature on domain adaptation, the target data set has been “pre-prepared” to cover all the $N$ classes of the pre-trained source model. However, in practice, this is not trivial and often infeasible for an end-user: collecting data can be quite laborious and costly. Take wildlife monitoring as an example. Data are often collected passively (e.g., via smart camera traps), waiting for animals to appear. As a result, when a smart camera trap is redeployed to a new location and requires adaptation, it is hard to prepare a complete target dataset that contains all the animal species of interest. This raises a dilemma: *should one wait for the data to be fully collected, even if it means sacrificing the model's performance in the meantime, or update the model right away with incomplete data, accepting the risk of compromising certain capabilities?* Our paper aims to address this dilemma by delving into an unexplored  Holistic Transfer (HT) setting: adapting a pre-trained classifier's *holistic* capability of recognizing $N$ classes to the target domain, using target data that covers only a subset of $N'<N$ classes. \n\nIt is worth noting that HT fundamentally differs from partial domain adaptation [A]. While partial domain adaptation also considers the situation where the target data set only contains $N'<N$ classes, its goal is only to perform well on those $N'$ classes after adaptation. In sharp contrast, HT aims to perform well on all the $N$ classes after adaptation. This is why our benchmarks in Table 1 explicitly separate the target data set into the one used for training ($N’<N$ classes) and the one used for testing ($N$ classes). Our study in Sect. 2.3 showcases the challenges of this new setting: standard fine-tuning would simply degrade the model’s capability on the $N - N’$ unseen target classes. \n\nWe hope the above paragraphs address your concern, and we will be happy to provide more information in the discussion period.\n\n[A] Cao, Zhangjie, et al. \"Partial adversarial domain adaptation.\" Proceedings of the European conference on computer vision (ECCV). 2018. \n\n**The learning setup is somewhat trivial … It is confusing to assume that the marginal distribution is the same one.** Sorry for the confusion. About the relationships between P_T and P_T* at L60, we should have written P_T(x|y) = P_T*(x|y), not P_T(x) = P_T*(x). That is, we assume that the data distributions in T and T* are the same per class, not marginally. We will correct this accordingly. \n\nWe hope this addresses your concern that *the learning setup is somewhat trivial.* If not, we kindly ask for more details about your comment, and we will be happy to address it in the discussion period.\n\n**The name \"Holistic Transfer\" is also confusing for me.** Sorry for the confusion. Following the motivations we provide above, the goal of our new setting is to adapt a pre-trained classifier’s *holistic* capability of recognizing $N$ classes to the target domain, even when the available target data does not cover all the $N$ classes. This is why we call our setting “holistic transfer.”\nWe will clarify this in the final version.\n\nThat said,  we remain open to discussion with you regarding the name and are willing to consider any adjustments based on your feedback.\n\n*In light of our clarifications, we would like to ask if you are willing to reconsider your score and if there are any new concerns or additional questions we can respond to!*"}, {"review_id": "Ac4MtWOqM2", "reviewer": "Reviewer_CeTn", "summary": "- This paper introduces a new setting: partial target data. A model is pretrained on a source domain with a set of classes. The model then has access to labels from a target domain, but only a subset of the classes. The goal is to do well on all classes (including the remaining, unseen classes) on the target domain\n- This paper repurposes datasets such as Office-Home, iWildCam, VTAB, iNaturalist (fungi), and FEMNIST, for this task.\n- They find that naive fine-tuning on the target does not do well. It does well on the seen classes, but poorly on unseen target classes\n- They consider a wide range of methods. For example, tuning only the batch norm parameters \n- They propose a method called LOLSGD, where they leave out a class and take gradient steps, and average this over the left-out class.\n- They combine these with ideas such as selective distillation, feature rank regularization \n- They show that their method does well on a wide range of datasets\n", "questions": "- For feature regularization, can you add some details for exactly what matrix you’re computing the singular values (is it the matrix C?). Also, if you regularize ||diag(C^T C)||_2^2, then doesn’t that push down the singular values? Are you subtracting this term instead of adding it? Sorry for I’m missing something basic.\n- For CLIP, do you initialize the head with the zero-shot initialized classifier?\n- I wonder if tuning just the bottom couple of layers would work well. Lee et al., 2023, found that for shifts in x it can often be better to tune the bottom few layers. This might have less overfitting to the seen classes as well. Maybe it can be combined with batchnorm fine-tuning.\n- LP-FT: I think it’s worth adding that it’s not designed for this scenario (it’s designed for the case where the classes from source / pretraining to target / fine-tuning are different, so you don’t have a “head”. Otherwise it looks like a bit of a strawman.\n- nit: Line 174 says “Unfortunately, we found updating normalization layers is yet satisfying” - do you mean it’s not yet satisfying? That is, updating normalization layers doesn’t quite solve the partial target data setting?\n", "limitations": "No concerns", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem seems very relevant, practical, and interesting. I’m not personally aware of other work in this space, but I might be missing parts of the related literature\n- I think their datasets generally seem to make sense, and could be useful for people working on this problem\n- Their explanation of why fine-tuning doesn't work very well makes sense\n- They try a lot of potential improvements, including some of their own, which are novel. These methods seem to work well\n- I like the selective distillation idea - one reason a model might do poorly in the partial target data setting, is that it might have much higher confidence on seen objects than unseen objects. Selective distillation might mitigate this issue, because the logits on unseen classes should be similar\n- The experiments generally look interesting and sound.\n\nMy overall verdict is an accept. While there are some weaknesses, I think the paper would be a solid addition to the conference.\n", "weaknesses": "- Batchnorm seems to do well, better than I expected from the text in line 174. BN (stats only) does the best on unseen data in Table 3.  Consider running these on the other datasets too? Maybe a slightly more sophisticated version might work better.\n- I don't understand the intuition for LOLSGD. It seems handwavy. I think it would be good to add a toy example where this method works. Or show a simple example where the method works. It's unclear why the gradients biased to certain classes will cancel out\n- Their solutions seem nice and plausible, but not especially convincing. This is fine, since they're introducing a problem, and try out some reasonable methods. Future work can focus on better or more principled solutions.\n- nit: I’m not a big fan of the name holistic transfer. There are many different transfer settings. What makes this more holistic than other types, for example OOD generalization? I think “partial target data setting”, or some other more specific naming choice would be better. \n- There should be a more comprehensive related work, and it should be in the main paper. Can you compare to universal domain adaptation, open set domain adaptation, etc? I wonder if there is some setting that is similar to yours, even if I'm not aware of the literature, which is why my confidence is low. Please select the closest related settings and compare with them.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive and constructive feedback.\n\n**Batchnorm seems to do well.** We note that our ultimate goal is to achieve high overall accuracy, not merely unseen accuracy. Although BN (stats only) *maintains* the unseen accuracy well on OfficeHome, it cannot effectively improve the seen accuracy, resulting in worse overall accuracy (cf. Table 3). Following your suggestion, we apply BN (stats only) to iWildCAM (cf. Table 6). As in Table R2 (in the PDF of the global response), BN (stats only) maintains the unseen accuracy well but cannot improve the seen accuracy. That said, we agree that a more sophisticated, dedicatedly designed version of BN might perform better, and we leave it as future work.\n\n**More details and demonstrations of LOLSGD.** Thank you for the comment. Please be referred to the global response.\n\n**The naming of holistic transfer.** We apologize for the confusion. The word “holistic” closely reflects our setting — adapt a pre-trained classifier’s *holistic* capability of recognizing $N$ classes to the target domain, even when the available target data does not cover all the $N$ classes. (Please be referred to the first and third responses to 9DJT for details.) We will clarify this in the final version. \n\nWe indeed have considered naming our setting with “partial target data,” but it may confuse our setting with a related problem named partial domain adaptation [A]. Partial domain adaptation also considers the situation where the target data set contains partial (i.e., $N'<N$) classes. However, after adaptation, it only aims to perform well on those $N'$ classes. In sharp contrast, our setting aims to perform well on all the $N$ classes after adaptation. We thus purposefully refrain from using the term \"partial\" to avoid confusion. That said, we remain open to discussion about the name and are willing to consider any adjustments based on your feedback.\n\n[A] Partial adversarial domain adaptation. ECCV 2018. \n\n**More related work.** Thank you for the comment. Our related work is in the Supplementary due to the page limit. We will strengthen it and include the most relevant part in the main paper. \n\nOur setting is highly related to domain adaptation (DA), whose objective is to adapt a model trained on a source domain (e.g., an image style) to perform well on a target domain (e.g., a different style). Depending on whether *labeled* target data is provided for adaptation, DA can be categorized into supervised, semi-supervised, and unsupervised settings, and we consider the supervised setting (L81). \n\nAnother way to categorize DA works is by the relationship between the label spaces of the source and target data. The standard closed-set setting assumes that the source and target data cover the same $N$ classes. That is, when one prepares the target data set for adaptation, all classes are expected to be present. However, in practice, this is not trivial and often infeasible for an end-user: collecting data can be quite laborious, especially when $N$ is huge. (Please be referred to the first response to 9DJT for details.) The proposed setting aims to release this constraint, making the data preparation for adaptation *much simpler* for end-users. \n\nAt first glance, our setting seems quite similar to partial DA [A], which also considers the situation \nwhere the target data set contains part (i.e., $N’<N$) of the source classes. However, the goal of partial DA is to perform well only on those $N'$ classes after adaptation. In contrast, our setting aims to adapt the source model’s holistic capability of recognizing $N$ classes to the target domain. This is why in Table 1 we explicitly separate the target data set into one for training ($N’<N$ classes) and one for testing ($N$ classes). Our study in Sect. 2.3 showcases the challenges of this new setting: standard fine-tuning would simply degrade the model’s capability on the $N - N’$ unseen target classes.\n\nOpen-set and universal DA consider a different scenario where the target data set contains additional “unknown” classes that do not appear in the source data. The goal is thus to equip the adapted model with the ability to predict “unknown” for those data. We argue that this setting is orthogonal but can be compatible with ours, and we leave the combination as future work.\n\nPlease also see the first response to HMzM.\n\n**Feature regularization.** Yes, Figure 4 shows the singular values of matrix C, which is the covariance matrix of feature vectors. Ours has more *uniform* singular values. That said, in training, we do not explicitly compute the singular values and regularize them. The regularization term $||diag(C^\\top C)||_2^2$ we use is inspired by [15] and [B], which is *added* to the loss function. While looking a bit counterintuitive, [B] proved that this regularizer has the effect of penalizing the variance among the singular values, thus discouraging the tail singular values from collapsing to 0, mitigating dimensional collapse. We apologize that we missed this detail in the manuscript, and we will clarify it in the final version.\n\n[B] Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning, ICLR 2023.  \n\n**CLIP initialization.** Yes, we initialize the classification head with class names’ embeddings extracted from CLIP’s text encoder. After that, we drop the text encoder. \n\n**Tuning the bottom couple of layers.** We experiment with fine-tuning the BN layers and the first (or last) block of the source ResNet-50 on OfficeHome. As shown in Table R4, fine-tuning parts of the layers drastically degrades the unseen accuracy. This showcases the challenge of holistic transfer. That is, even with just parts of the model being fine-tuned, the model can easily fit the seen classes and suffers from forgetting the unseen classes. \n\n**LP-FT.** We will do so.\n\n**Line 174.** You are right. We meant updating normalization layers does not solve holistic transfer. We will correct the typo."}, {"review_id": "byOgaXoghu", "reviewer": "Reviewer_HMzM", "summary": "In this paper, the authors tackled \"holistic transfer\" task in which imcomlete target data that only cover a part of class labels are given to fine-tune a pre-trained model. To gain generalization ability of the model to unseen-class data in the target domain, the proposed method adopts several techniques in the fine-tuning with the target data: leave-out local SGD, freezing a linear classifier, selective distillation, and feature rank regularization. The experimental results with several benchmark datasets demonstrate the effectiveness of the proposed method. ", "questions": "- Why did the authors not examine the existing methods such as test-time adaptation, online domain adaptation, or PZDA in the experiments?\n- Can we say that the model is also successfully adapted to unseen classes even if the accuracy in those classes does not change from the source model? If yes, why?", "limitations": "- LOLSGD essentially requires that the number of classes is large enough to provide a sufficient variety of label subsets.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The problem setting called holistic transfer is interesting and should be important when we consider practical ML applications. \n- The proposed method works well in terms of boosting overall accuracy across several datasets in the experiments.", "weaknesses": "- This paper fairly lacks discussion on the comparison of the proposed task/method with related work.\n    - The problem setting of holistic transfer is quite similar with partially zero-shot domain adaptation (PZDA) [R1]. Specifically, it can be seen as a supervised but source-free version of PZDA. Since a generative approach to source-free domain adaptation is somewhat common in the recent literature, using PZDA method with source-data generation seems to be a simple solution to holistic transfer.\n        - [R1] \"Partially Zero-shot Domain Adaptation from Incomplete Target Data with Missing Classes,\" WACV 2020.\n    - Additionally, it should be also straightforward to use test-time adaptation or online domain adaptation methods to solve the holistic transfer task.\n    - The performance of existing methods such as those raised above is not examined in the experiments, which makes the significance of the proposed method unclear. Comapring with domain generalization methods should be unfair, because they are essentially unable to utilize the information of the target domain.\n- In the experiments, the performance of the proposed method in unseen classes is often par or even worse than that of the source model, which raises questions about the effectiveness of the proposed method for generalizability to unseen classes.\n- The manuscript lacks several important topics such as related work and limitations, while the discription about the problem setting (section 2) is redundant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The manuscript lacks several important topics such as related work and limitations.** Due to the page limit, we leave the related work and limitations in the Supplementary (as mentioned in L343). We apologize if it was unclear, and we will clarify it in the final version.\n\n**Lacks discussion and comparison with PZDA, online domain adaptation, and test-time adaptation.**\nThank you for the valuable comments. Our holistic transfer (HT) setting shares conceptual similarities with several machine learning paradigms (such as continual learning and domain adaptation, as discussed in Sect. 2, Table 2, and related work). We tried to include comprehensive discussions, but it appears we missed some. We appreciate your references to partially zero-shot domain adaptation (PZDA), test-time adaptation, and online adaptation, and we will surely cite and discuss them in the final version. \n\nThe key difference between our HT and PZDA is that we consider a source-free setting, which is more practical, especially for models trained on large-scale datasets or scenarios with privacy concerns (L83-85). If the source data set is available during adaptation, the problem can indeed become much easier. We have discussed a solution at L148-153, which is to translate the source data into the target style. \n\nOur HT setting is also quite different from test-time adaptation and online adaptation. While both of them allow the source model to be dynamically adapted once new target data is collected, they are not meant to address our problem of adapting the source model’s holistic capability of recognizing $N$ classes to the target domain when the available target data only covers $N’<N$ classes. More specifically, test-time adaptation and online adaptation cannot directly tackle the situation where some of the target test classes are missing in the target training set (cf. Table 2). While both of them can *wait* till the model eventually see the missing target classes to adapt the model’s capability of those classes, the model may have already suffered a serious forgetting problem (please see our next response). In contrast, our HT setting aims to *proactively* adapt the model, even for some target test classes that are missing in the target training set.\n\nWe think the source-data generation idea you suggested is very interesting and itself deserves a future paper since we believe source-data generation remains a challenging and ongoing research problem. We would like to emphasize that our main contributions are proposing the HT problem and establishing some most relevant baselines (L35, L346). We certainly do not claim that we have solved the problem perfectly. That said, we have experimented with source-data generation for adaptation and found that the gap between using real source images and generated ones is still large. We consider the OfficeHome dataset and apply the popular DeepInversion [C] on the two source models trained on Ar and Rw domains, respectively. We conduct a simple baseline for our HT problem by combining the real/generated source data with the partial target data for fine-tuning. As shown in Table R5 (the PDF in the global response), the unseen accuracy gap between using real/generated source data remains significant. Examples of the generated source images can be seen in Figure R2 (the PDF in the global response).\n\n[C] Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion, CVPR 2020. \n\n**Comparing with domain generalization methods should be unfair, because they are essentially unable to utilize the information of the target domain.** We apologize for the confusion. While SWA and SWAD (L262-263) are initially proposed to improve model generalization, we use them to fine-tune the source model with the target data. We include them mainly because they share a conceptual similarity to our LOLSGD, in which they also take average over several models in training. We will clarify this.\n\n**... the performance of the proposed method in unseen classes is often par or even worse than that of the source model … Can we say that the model is also successfully adapted to unseen classes even if the accuracy in those classes does not change from the source model? If yes, why?** We appreciate your question. The ultimate goal of HT (as defined in the objective in Equation 1) is to make the adapted model’s performance close to the oracle model that is trained/fine-tuned on the full target data without missing classes. To claim we successfully resolve HT thus requires improving the accuracy for both the seen and unseen classes in the target domain. However, one should not treat maintaining the unseen class accuracy the same as the source model as a trivial task, as fine-tuning with only the seen class data degrades the unseen class accuracy drastically (Sect. 2.3). Therefore, even keeping the unseen accuracy while improving the seen accuracy is challenging, especially in the source-free setting. In our humble opinion, our methods have largely addressed this challenge (from a drastically degraded unseen accuracy to an unseen accuracy similar to the source model) and, in some cases, improved the unseen accuracy against the source model. After all, our main contributions are proposing the HT problem and investigating some most relevant baselines (L35, L346). We certainly do not claim that we have solved the problem perfectly, and we hope our contributions will establish the foundation for future research in HT. "}]

Author rebuttal:
We thank the reviewers for their valuable comments. We are glad that the reviewers found the proposed problem “interesting”, “important” (Reviewer HMzM), and “valid and practically useful” (Reviewer tg8m, CeTn); the proposed method “works well” (Reviewer HMzM, tg8m, 9DJT, CeTn, 9DJT); the experiments generally look “interesting and sound” (Reviewer CeTn). We address the comments about LOLSGD in this global response (raised by Reviewer QPVq and CeTn) and address other comments raised by individual reviewers separately. We also include a one-page PDF with requested experiments, tables, and figures. We will incorporate all the feedback in the final version.

**(Reviewer QPVq) The LOLSGD seems similar to local SGD and meta-learning. It is unsure why its design can disentangle covariate shifts. What is its difference from using a larger batch size for SGD?** Our LOLSGD is fundamentally different from meta-learning and SGD with a large batch size. In our holistic transfer (HT) problem (see Sect. 3), fine-tuning is affected by the covariate shift (from the source to the target) and the disruptive concept shift (from classifying all the classes to classifying the seen classes). Our LOLSGD aims to disentangle them or, more precisely, reduce the disruptive concept shift by subsampling multiple datasets that each contain a subset of the seen classes. When updating the model separately in parallel with these subsampled datasets, each updated model is affected by a different concept shift but a shared covariate shift. By averaging these models, LOLSGD can potentially cancel out the disruptive concept shifts and strengthen the shared covariate shift (L191-196). We provide more evidence that LOLSGD can cancel out the disruptive concept shifts in the following response to Reviewer CeTn.

We argue that this is fundamentally different from meta-learning, whose goal is to learn a meta-modal that can be easily applied to a future task (e.g., a few-shot classification task). While meta-learning also subsamples its meta-training set, it is mainly to simulate multiple future tasks for learning the meta-model, not to cancel out unwanted gradients. We also argue that LOLSGD is fundamentally different from SGD with a large batch size. Concretely, without the strategic subsampling in LOLSGD, SGD with a large batch size will not create multiple models from which we can potentially cancel out the disruptive concept shifts.

Our LOLSGD is inspired by local SGD, as mentioned in L177-179. Our key contribution is novel usage. Local SGD was initially proposed for distributed learning, in which the training data are decentralized by default. The goal is mainly for reducing the communication overhead of large-scale training. In contrast, in LOLSGD, the target training data set is not decentralized initially, but we strategically subsample it to simulate different concept shifts.

**(Reviewer CeTn) I don't understand the intuition for LOLSGD. I think it would be good to add a toy example where this method works. Or show a simple example where the method works. It's unclear why the gradients biased to certain classes will cancel out.** We apologize if we did not make the intuition clear. We have provided some more details in the above response to 
Reviewer QPVq. Please kindly be referred to it.

To give more evidence for the canceling-out effect, we compare the seen class accuracy among 1) naive fine-tuning with the partial target data, 2) LOLSGD, and 3) fine-tuning with the full target data (i.e., the oracle at L148-153). As shown in Figure R1 (the PDF in the global response), the seen class accuracy of naive fine-tuning (black dotted line) exceeds the oracle (green dotted line), indicating that naive fine-tuning learns undesired concept shifts towards seen classes, leading to an unreasonable accuracy. In contrast, the seen accuracy of LOLSGD (red dotted line) consistently stays below the oracle, indicating that undesired concept shifts are reduced. As a result, LOLSGD obtains much higher unseen accuracy than naive fine-tuning.


Meta-review: This submission received truly mixed ratings so there was a long discussion between the reviewers and authors. The rebuttal addressed some of the concerns. However, in the end, two reviewers still keep their rating as reject. The main weaknesses are practicality of the problem setting and lack of sufficient experiments. The AC took a closer look at the material and believe those weaknesses are partially addressed in the rebuttal. Then, given three positive reviews, the AC recommends accepting this paper.
Decision: Accept (poster)