SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Unseen performance vs. overall performance. One might observe that sometimes the unseen performance could be better than the overall performance. We surmise this is because the classes are not equally difficult for classification. In some cases, before we perform adaptation, we can already see that the unseen class accuracy of the source model is higher than its overall accuracy, meaning that those unseen classes are inherently easier than the seen classes. Some methods (e.g., BN (stats) only) can better keep the unseen accuracy close to the source model with less forgetting. However, they cannot adapt the seen classes effectively to the target domain. Therefore, the results of these methods generally follow the trend in the source model with the unseen accuracy higher than the overall accuracy. In contrast, some methods (e.g., LP-FT) can better adapt the seen classes to the target domain but suffer from serious forgetting of the unseen classes. These methods thus have lower unseen accuracy.
New classes in the target domain. We note that, in our current setup all the experiments assume the classes tested in the target domain appeared in source training. If some "seen" classes in the target domain are not in the source domain, this will require expanding the label space of the model. A simple baseline would be first training the classification weights for those "seen" classes from scratch while keeping all other model components intact. Then, we can apply our holistic transfer approach to the expanded model. A more sophisticated solution would involve techniques from continual
Table 14: Variances of test accuracy for fine-tuning CLIP ViT-B/32 on VTAB (cf. Table 7).  $50\%$  of classes in each task are missing during training. We compute the variances over 3 random seeds.
Overall/Unseen Acc.	Caltech101	CIFAR100	DTD	EuroSAT	Flowers102	Pets	Resisc45	SVHN	SUN397	Avg.
Methods	All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.
Source	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Naive Target	0.00	0.00	0.02	0.08	0.17	1.32	0.01	0.02	0.30	0.45	0.05	0.38	0.22	1.45	0.01	0.03	0.01	0.05	0.00	0.01
SGD*	0.00	0.01	0.02	0.17	0.31	0.33	0.21	0.59	0.16	0.07	0.03	0.15	0.13	1.00	0.03	0.02	0.00	0.01	0.02	0.02
LOLSGD*	0.01	0.00	0.01	0.03	0.39	1.33	0.16	0.69	0.21	0.12	0.03	0.13	0.01	0.07	2.29	0.14	0.06	0.29	0.04	0.01
LOLSGD + Ldistill*	0.00	0.00	0.01	0.04	0.04	0.00	0.13	0.20	0.01	0.02	0.01	0.00	0.01	0.02	1.80	0.01	0.00	0.00	0.01	0.00
LOLSGD + Lrank*	0.00	0.01	0.00	0.02	0.07	0.07	0.17	0.34	0.06	0.02	0.02	0.01	0.01	0.10	1.37	0.36	0.03	0.06	0.01	0.00
LOLSGD + Lrank + SE*	0.01	0.02	0.07	0.14	0.12	0.33	0.06	0.05	0.06	0.03	0.00	0.03	0.05	0.09	1.91	0.82	0.01	0.05	0.00	0.02
Oracle	0.29	0.19	0.00	0.01	0.40	0.50	0.01	0.07	0.27	0.35	0.02	0.14	0.02	0.12	0.00	0.00	0.00	0.01	0.02	0.03
Table 15: Different percentages of the target training data for each seen class on Office-Home:  $\mathrm{{Ar}} \rightarrow  \mathrm{{Cl}}$  .
% of target training
Methods/Acc.	10%	50%	100%
Overall	Unseen	Overall	Unseen	Overall	Unseen
Source	47.07	50.29	47.07	50.29	47.07	50.29
Naive Target	41.73	19.74	43.46	9.94	44.96	9.06
SGD*	51.28	41.96	52.33	28.36	52.11	24.12
SGD + Lrank*	50.45	46.49	56.02	40.20	59.17	39.47
SGD + Ldistill*	48.80	41.37	53.91	38.30	56.54	39.18
LOLSGD*	52.63	46.20	54.21	34.94	56.47	35.09
LOLSGD + Lrank*	51.13	48.83	55.86	44.74	58.57	43.86
LOLSGD + Ldistill*	51.28	45.91	55.19	44.88	57.44	46.35
LOLSGD + Ldistill + Lrank*	51.05	50.88	58.65	52.05	60.83	51.75
learning, or more specifically, class-incremental learning. This machine learning paradigm aims to expand the label space of a model. We leave a suitable combination of our approach and techniques from class-incremental learning as future work.
D.3 Effects of the Source Ensemble Coefficients
In section 4, we apply Source Ensemble (with a mixing coefficient  $\alpha = 0.5$ ) to reclaim some ability of the source model to maintain the unseen accuracy (cf. Table 4). To further understand the trade-off between the source and the fine-tuned target models, we study the effects of the mixing coefficient  $\alpha$  by varying it between [0, 1]. We conduct our study on Office-Home and report the overall and unseen accuracy averaged over all the source-target domain pairs. As shown in Figure 6, applying either SE or WiSE cannot save the naively fine-tuned target model from being heavily biased to seen classes. On SGD with frozen classifiers, our SE shows a better trade-off than WiSE [79]. Finally, fine-tuning target models with our suggested HT methods can clearly yield the best trade-off.
D.4 Comparison to CoCoOp [96] on CLIP
In our experiments, we use CLIP to construct the source models for the VTAB and iNaturalist experiments. We further compared to a recent CLIP fine-tuning method CoCoOp [96] in these experiments. CoCoOp fine-tunes CLIP by training a meta-net to condition the classification weights (i.e., the fully-connected layer) on the input images. In other words, CoCoOp freezes the visual features but changes the classifier weights by minimizing the standard cross-entropy loss. In contrast, our approach freezes the classifier weights but adapts the visual features by minimizing the loss designed specifically for HT.
We conduct two experiments: 1) CoCoOp alone for the HT problem, and 2) combining CoCoOp with our approach. For 2), we take the resulting model after CoCoOp as the improved source model and further adapt the feature. We report the result of the CIFAR-100 task in VTAB (cf. Table 7). As shown in Table 16, CoCoOp alone performs well on unseen classes, but it improves the overall accuracy only marginally. Combining both approaches, we can obtain better accuracy in unseen and seen classes, leading to the best overall accuracy.
Figure 6: Effects of Source Ensembles. Ensemble the source (the star marker) and the target models (the end point of each line from the source model) with a mixing coefficient  $\alpha \in [0,1]$  on Office-Home.
D.5 More experiments with the updating BN baseline
Batchnorm layers are critical components in feed-forward neural networks especially when learning with distributional shift [46, 94]. In cf. Table 3, one may observe the baseline "BN (stats only)" of updating only the means and variances in batchnorm layers performs surprisingly well on the unseen accuracy. We note that the ultimate goal of HT is to achieve high overall accuracy, not merely unseen accuracy. Although "BN (stats only)" maintains the unseen accuracy well on the Office-Home dataset, it cannot effectively improve the seen accuracy, resulting in worse overall accuracy. We apply "BN (stats only)" to iWildCAM (cf. Table 6). As in Table 17, it maintains the unseen accuracy well but cannot improve the seen accuracy. That said, we think that a more sophisticated, dedicatedly designed version of BN might perform better, and we leave it as future work.
D.6 A study for LOLSGD canceling out class biases
Our LOLSGD aims to disentangle them or, more precisely, reduce the disruptive concept shift by subsampling multiple datasets that each contain a subset of the seen classes. When updating the model separately in parallel with these subsampled datasets, each updated model is affected by a different concept shift but a shared covariate shift. By averaging these models, LOLSGD can potentially cancel out the disruptive concept shifts and strengthen the shared covariate shift. We provide more evidence that LOLSGD can cancel out the disruptive concept shifts. We compare the seen class accuracy among 1) naive fine-tuning with the partial target data, 2) LOLSGD, and 3) fine-tuning with the full target data (i.e., the oracle model). As shown in Table 19, the seen class accuracy of naive fine-tuning (black dotted line) exceeds the oracle (green dotted line), indicating that naive fine-tuning learns undesired concept shifts towards seen classes, leading to an unreasonable accuracy. In contrast, the seen accuracy of LOLSGD (red dotted line) consistently stays below the oracle, indicating that undesired concept shifts are reduced. As a result, LOLSGD obtains much higher unseen accuracy than naive fine-tuning.
Table 16: Results of CoCoOp with ViT-B/32 on CIFAR100. 50 classes are missing during training (50 seen classes). We treat the trained CoCoOp as a source model and apply our approaches.
Methods	Overall	Unseen
Source (CLIP)	64.18	62.42
Naive Target	65.94	42.66
LOLSGD*	69.59	51.78
LOLSGD + Ldistill	67.13	49.08
LOLSGD + Lrank	69.99	52.54
LOLSGD + Lrank + SE	72.31	58.38
CoCoOp	66.16	63.08
CoCoOp + LOLSGD*	72.58	64.20
CoCoOp + LOLSGD + Ldistill	69.34	61.90
CoCoOp + LOLSGD + Lrank	72.77	64.36
CoCoOp + LOLSGD + Lrank + SE	72.34	65.52
Oracle	83.10	82.62
Table 17: iWildCAM mean accuracy of 21 new locations. We extend cf. Table 6 to compare to the baseline "BN (stats) only", which update the means and variances in all BN layers.
Methods	Overall	Seen	Unseen
Source	35.71	35.74	26.43
Naive Target	35.38	51.08	1.90
BN (stats) only	30.18	29.69	23.25
SGD*	36.53	49.00	7.91
SGD + Lrank*	36.28	40.98	19.76
SGD + Ldistill*	41.59	50.40	17.90
LOLSGD*	38.12	48.91	13.30
LOLSGD + Lrank*	33.94	37.62	19.42
LOLSGD + Ldistill*	40.49	47.30	25.16
LOLSGD + Ldistill +SE*	39.96	44.79	25.66
Table 18: Results on Office-Home with 30 seen and 35 unseen classes. We combine updating BN with partially tuning the last block of ResNet-50. The accuracy is averaged over 6 source-target pairs used in our paper.
Methods / Acc.	Overall	Unseen
Source	63.90	65.65
Naive Target	53.50	22.36
BN (stats) only	64.27	67.51
BN (stats) only + Partially tune	57.96	30.63
BN (stats)	57.60	32.36
BN (stats) + Partially tune	57.68	29.39
Oracle	72.01	66.54
Table 19: An HT example showing that LOLSGD can cancel out undesired class biases learned by Naive Target. We use  $\mathrm{Ar} \rightarrow \mathrm{Cl}$  domains on Office-Home
E Additional Discussions
E.1 Limitations
In this paper, we introduce a novel and practical transfer learning problem, holistic transfer, that emphasizes the generalization to domain shifts for classes unseen in the target domain but seen in the source domain. We establish strong baselines and demonstrate the potential for simultaneously improving both seen and unseen target classes. One potential limitation is that we mainly focus on vision classification tasks. We leave the studies to image segmentation/object detection and natural language processing tasks as our future work. We also plan to explore better approaches for the disentanglement of domain styles and classes and to integrate our approach with other learning paradigms, like test-time adaptation.
E.2 Potential Negative Societal Impact
The goal of our work is to introduce and study a practical transfer learning problem, holistic transfer. We provide strong baselines and analyze the problem on publicly available datasets, which are adjusted and split to meet our problem setting. As far as we know, our work does not introduce additional negative societal impacts compared to the standard transfer learning topics, like domain adaptation and out-of-distribution generalization.
E.3 Computation Resources
We conduct our experiments on PyTorch and on NVIDIA V100 GPUs. On the Office-Home dataset, fine-tuning for 1 target domain with all the compared methods and random seeds takes roughly 36 hours on 1 GPU. Similar time consumption also applies to iWildCam and VTAB datasets. On the smaller FEMNIST dataset, it takes roughly 0.5 hours on 1 GPU to get the required results for 1 target domain. The whole experiment on iNaturalist Fungi takes roughly 0.5 on 1 GPU. In total, our experiments take roughly 1.3K GPU hours.