SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Case study: fine-tuning can lead to serious false negatives. So far, we mainly evaluate HT from the lens of unseen class accuracy. Here, we discuss confusion in predictions and a case that transfers without considering HT could pose even larger risks. Considering training a classifier for recognizing mushrooms on some non-toxic ones and encountering a visually similar but toxic one in testing, the classifier will likely label it as one of the non-toxic ones. Even worse, such bias is exaggerated by the adaptation, due to the partial training classes. We build an extreme case that selects 6 pairs of fungi (toxic/non-toxic each) from the iNaturalist dataset [73] and fine-tunes on only the non-toxic ones from CLIP. The results in Table 9 show such concern exists and should be considered in future algorithm designs.
5 Conclusions (Related Works & Limitations in Supplementary)
We introduce a novel and practical transfer learning problem that emphasizes generalization to unseen classes in the target domain but
seen in the source domain. Despite its challenges, we establish strong baselines and demonstrate the potential for improving both seen and unseen target classes simultaneously, paving the way for holistic transfer. To facilitate further research, we construct a comprehensive benchmark with diverse scenarios and conduct insightful experiments. In future work, we envision exploring various directions, including improved disentanglement of domain styles and classes, integrating our approach with other paradigms like test-time adaptation, and extending beyond classification tasks.
Table 8: Examining HT of zero-shot CLIP on SVHN dataset.
Classes	Predictions
“0” to “9”	39.4%
“number plate”	32.7%
“house number”	16.8%
“wallpaper”	11.1%
Missing rate	60.6%
Table 9: Fine-tuning can lead to serious false negatives.
Method	Seen Accuracy	False Negatives
Source	23.3%	46.7%
Target	63.3%	96.7%
Our HT	60.0%	83.3%
This research is supported by grants from the National Science Foundation (OAC-2118240, OAC-2112606). We are thankful for the generous support of the computational resources by the Ohio Supercomputer Center.
[86] Jinyu Yang, Jingjing Liu, Ning Xu, and Junzhou Huang. Tvt: Transferable vision transformer for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 520-530, 2023.
[87] Shiqi Yang, Yaxing Wang, Joost Van De Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8978-8987, 2021.
[88] Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. arXiv preprint arXiv:2106.01085, 2021.
[89] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547, 2017.
[90] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Universal domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2720-2729, 2019.
[91] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019.
[92] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019.
[93] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1131-1140, 2020.
[94] Jike Zhong, Hong-You Chen, and Wei-Lun Chao. Making batch normalization great in federated deep learning. arXiv preprint arXiv:2303.06530, 2023.
[95] Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep class-incremental learning: A survey. arXiv preprint arXiv:2302.03648, 2023.
[96] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[97] Lanyun Zhu, Tianrun Chen, Jianxiong Yin, Simon See, and Jun Liu. Continual semantic segmentation with automatic memory sample selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3082-3092, 2023.
[98] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43-76, 2020.
We provide details omitted in the main paper.
- Appendix A: related work (cf. subsection 2.3 and section 5 of the main paper).
- Appendix B: additional benchmark details (cf. subsection 2.2 of the main paper).
- Appendix C: additional training details (cf. section 4 of the main paper).
- Appendix D: additional results and analyses (cf. section 4 of the main paper).
- Appendix E: additional discussions (cf. section 5 of the main paper).
We review related work on other transfer learning paradigms. We briefly describe their settings and distinguish their differences from our proposed holistic transfer (HT) problem.
A.1 Domain Adaptation
Domain adaptation (DA) is the most iconical machine learning setting to tackle the domain-shift problem [86, 27, 62, 65, 14, 49]. With the common objective of transferring source-domain knowledge to target domains, various settings have been proposed to incorporate different constraints and assumptions. The assumption can be the degrees of overlap between the source and the target label sets [90, 64, 5, 6, 85, 57, 34, 9]. To relax the constraint of accessing source data, source-free DA can solely rely on the target data for adaptation [87, 23, 43, 18]. Despite the abundant variations, DA settings all share one common assumption: the target distributions in training and testing are matched, making our HT fundamentally different from them. In our HT, we can encounter target test classes that are unseen in the target training set but seen in the source domain. Therefore, HT requires a distinct ability that can generalize the style shifts learned on the target seen classes to other unseen classes.
A.2 Out-of-domain Generalization
Although fine-tuning a pre-trained model often leads to impressive accuracy for downstream tasks, recent studies have revealed that it may compromise the out-of-domain (OOD) robustness of the model [41, 3, 79]. Several robust fine-tuning methods are thus proposed to balance the trade-off between in-domain downstream accuracy and OOD generalization [70, 61, 82]. LP-FT [41] proposed to learn a classifier with frozen features before end-to-end fine-tuning to avoid feature distortion. Some other approaches relied on ensembles with pre-trained models to increase the robustness [78, 30]. However, the main focus of these studies remains on preserving the robustness to different input styles for classes seen in the target training set. This is significantly different from HT. Our HT problem aims to generalize the styles for classes unseen in the target training set.
A.3 Continal Learning
The goal of continual learning (CL) is to sequentially adapt to multiple tasks without catastrophically forgetting the previously learned ones [37, 51, 47]. To achieve this goal, existing studies have proposed to exploit a replay buffer for storing old data [63, 8, 80, 71, 67, 88, 50, 97, 52, 95], or to constrain the fine-tuning with old models [89, 22, 1, 24, 77]. Unlike HT, CL still assumes all the encountered training distributions, which could be many, are aligned with their corresponding test distributions. Although reducing forgetting can be the first step for HT to maintain unseen class accuracy, we argue that this is insufficient in HT due to the source-target domain mismatch. Adapting the features for unseen classes to the target domain remains a key challenge for HT. Moreover, HT can also be potentially compatible with CL to consider learning on a non-iid data stream.
A.4 Zero-shot Learning
Zero-shot learning tackles the setting where training and test classes are completely disjoint [81, 16, 84, 58]. As no training data are available for test classes, the main challenge resides in learning source
Table 10: A summary of the dataset statistics for our HT benchmark.
Datasets	Source domains	Target domain	#Classes	#Seen classes	#Target training	#Target test
Office-Home	Art	Clipart	65	30	1,471	1,330
Product	1,265	1,361
Real	1,413	1,335
Real	Art	65	30	857	750
Clipart	1,493	1,330
Product	1,459	1,361
FEMNIST	40 writers	10 new writers	62	Vary by data collection bias	Vary by data collection bias	Vary by data collection bias
iWildCam	53 camera trap locations	21 new camera trap locations	181	Vary by data collection bias	Vary by data collection bias	Vary by data collection bias
VTAB	CLIP	Caltech101	102	51	1,371	6,084
CIFAR100	100	50	22,513	10,000
DTD	47	23	920	1,880
EuroSAT	10	5	8,424	5,400
Flowers102	102	51	510	6,149
Pets	37	18	1,445	3,669
Resisc45	45	22	9,159	6,300
SVHN	10	5	28,197	26,032
SUN397	397	198	37,542	21,750
iNaturalist (Fungi)	CLIP	Fungi	12	6	30	60
features that can generalize to unseen semantic meanings. To achieve this, auxiliary information (e.g., texts or attributes) is usually needed to describe the test classes and connect them back to the training classes [83, 15, 53, 45]. In HT, we assume the missing classes in target domains are already seen in the source domain. We make this assumption to simplify the problem so that HT can focus on generalizing the domain shifts to unseen classes. However, we argue that HT is compatible with zero-shot learning to make the setting more flexible.
B Additional Benchmark Details
To support the study of the HT problem, we create a benchmark that covers extensive scenarios across both experimental and realistic public datasets. We provide details about these datasets.
B.1 Office-Home
Setup. We consider the standard domain adaptation setting but with some missing classes in the target training sets. We use the popular Office-Home dataset consisting of 65 categories from 4 domains (Art, Clipart, Real, and Product). In our benchmark, we use Art and Real as source domains; each source domain is then transferred to each of the three remaining target domains individually, resulting in six source-target pairs. For each source-target pair, we use all the data in the source domain to train a source model. Then, for each target domain, we randomly split the data of each class into training and test sets with a ratio of 7:3. We randomly sample 30 seen classes and combine the training data of these seen classes to create the target training set. Finally, the target test set consists of the test images of all 65 classes in the target domain. A summary of the statistics can be found in Table 10.
Evaluation. We follow the standard evaluation metric in the Office-Home dataset to compute the overall accuracy for each source-target pair. Besides, we explicitly compute the accuracy of the unseen-class data to evaluate the transferring performance of the unseen classes. The average accuracy over all the source-target pairs is also reported.
B.2 FEMNIST
Setup. The FEMNIST dataset contains 62-class hand-written characters from many writers with different writing styles. In practical personalized AI applications, it is common to train on data from many users but needs to adapt to many new users after the (source) model is trained [11]. As we
can only collect a limited-size data set for each writer, each writer's data only cover a subset of the 62-class characters, resulting in the need for HT. We randomly sample 40 writers whose data combined can cover all 62 classes and use their data to train a source model. Then, we randomly sample 10 new writers. Each new writer's data is divided into training and test sets in a ratio of 7:3. Note that each client may not have enough images per class, which creates a realistic scenario of personalization with limited samples, which results in a mismatch of the class distributions between training and test sets. The dataset statistics are summarized in Table 10.
Evaluation. We report the overall accuracy averaged over all the 10 new writers. To evaluate the trade-off between seen and unseen classes, we also report the averaged accuracy on the seen and unseen classes, respectively. As this dataset has no oracle training set for each new writer, we report the seen accuracy computed by chopping out unseen classes in the classifier to evaluate the quality of the adapted features.
B.3 iWildCam
Setup. We consider a realistic scenario of HT, where we initially have abundant camera traps installed across many geo-locations (source domains) and now need to transfer to a new camera trap location (target domain). In the new location, we can only use the data collected within a fixed amount of time in the beginning (e.g., the first month) as our target training set. As it is impossible for all the animal species to appear in the first month, the target training data can be biased toward some classes that show up. This is a natural data collection bias caused by time.
We start from the iWildCam dataset in the WILDS [38] benchmark. As we mainly focus on animal species classification, we remove the "empty" class for simplicity and thus obtain a total of 181 classes. For each camera trap location, we sort the images by their timestamps and group images into sequences if the difference in their timestamps is smaller than 30 minutes, to avoid information leaks. We randomly sample 53 camera trap locations whose images cover all 181 classes and use all their data to train a source model. For each of the remaining locations, we randomly sample training and test sets based on a ratio of 7:3. We only keep locations with more than 500 images in both the training and test sets, thereby resulting in 21 new locations for adaptation. For each new location, we form the target training set by sorting the training images by time and only using the first  $25\%$  of them. A summary of the dataset statistics is given in Table 10.
Evaluation. We report the overall accuracy averaged over all 21 new locations. To evaluate the trade-off between seen and unseen classes, we also report the averaged accuracy on the seen and unseen classes, respectively. As this dataset has no oracle training set for each new location, we report the seen accuracy computed by chopping out unseen classes in the classifier to evaluate the quality of the adapted features.
B.4 VTAB
Setup. We consider another practical use of HT by going beyond domain adaptation and fine-tuning the zero-shot CLIP [60] for distribution shifts at the task levels. We use the VTAB [92] benchmark that includes various image classification tasks. To enable zero-shot predictions, we only use the tasks that provide text names for classes, thereby resulting in 9 tasks: Caltech101, CIFAR100, DTD, EuroSAT, Flowers102, Pets, Resisc45, SVHN, and SUN397. We use the standard training and test sets provided by the VTAB benchmark. Then, we randomly sample half of the classes as seen and the remaining as unseen. The target training set only includes the training images of the seen classes, while the target test set contains all the test images. A summary of the statistics of this dataset is shown in Table 10.
Evaluation. Following the standard evaluation in VTAB, we report the overall accuracy for each of the 9 tasks. Besides, we also compute the accuracy on the unseen-class data to evaluate the transferring performance of unseen classes. Finally, the average accuracy across all 9 tasks is also reported.
B.5 iNaturalist (2021 Version, Fungi)
Setup. To demonstrate the impact of visually similar classes in HT, we carefully pick 6 pairs of fungi classes from the iNaturalist dataset, thus resulting in a total of 12 classes. Each pair of fungi classes corresponds to 2 species of visually similar fungi; one is non-toxic, while the other one is toxic. We
use the zero-shot CLIP model with the fungi names as our source model. Then, the training images from the 6 non-toxic fungi classes form the target training set. The target test set consists of all the test images from all 12 classes. A summary of the dataset statistics and some examples are shown in Table 10.
Evaluation. We report the seen accuracy on the target test set to evaluate the adaptation performance. As wrongly predicting toxic fungi as non-toxic ones can result in severe outcomes, we also report the false negative rate, which is computed as the percentage of the images of toxic fungi being predicted as non-toxic fungi classes.
We provide the training details for our results reported in section 4.
For the Office-Home dataset, we initialize a ResNet-50 with ImageNet pre-trained weights. Then, we train it on the source domain for 20 epochs using the SGD optimizer with a learning rate 1e-3, momentum 0.9, weight decay 5e-4, and batch size 64. For all methods that adapt to the target domains, we fine-tune the source model for 20 epochs using the SGD optimizer with a learning rate 1e-4, momentum 0.9, weight decay 5e-4, and batch size 64. For our suggested HT methods, we set the hyper-parameters  $\mathcal{L}_{\mathrm{distill}} = 10$  and  $\mathcal{L}_{\mathrm{rank}} = 100$ .
For the FEMNIST dataset, we train a LeNet from scratch on the data of the 40 source writers for 100 epochs using the SGD optimizer with a learning rate 1e-2, momentum 0.9, weight decay 5e-4, and batch size 32. To adapt to each new writer, we fine-tune the source model for 10 epochs using the SGD optimizer with a learning rate 1e-3, momentum 0.9, weight decay 1e-4, and batch size 32. We set the hyper-parameters  $\mathcal{L}_{\mathrm{distill}} = 0.1$  and  $\mathcal{L}_{\mathrm{rank}} = 10$ .
For the iWildCam dataset, we train a ResNet-50, which is initialized with ImageNet pre-trained weights, on the data of source camera trap locations for 50 epochs using the SGD optimizer with a learning rate 3e-5, momentum 0.9, weight decay 0.0, and batch size 16. When adapting to each new location, we fine-tune the source model for 20 epochs using the SGD optimizer with a learning rate 3e-6, momentum 0.9, weight decay 0.0, and batch size 16. We set the hyper-parameters  $\mathcal{L}_{\mathrm{distill}} = 50$  and  $\mathcal{L}_{\mathrm{rank}} = 200$ .
For the VTAB benchmark, we use the class names for each of the 9 tasks to form the zero-shot CLIP models, which are ViT-B/32. We fine-tune the source model on target tasks for 20 epochs using the SGD optimizer with a learning rate 1e-5, momentum 0.9, weight decay 0.0, and batch size 64. We set the hyper-parameters  $\mathcal{L}_{\mathrm{distill}} = 1$  and  $\mathcal{L}_{\mathrm{rank}} = 5$ .
For the iNaturalist Fungi dataset, we use the fungi species names to build a zero-shot CLIP model with a ViT-B/32 architecture. We then fine-tune the source model on the target training set for 5 epochs using the SGD optimizer with a learning rate 5e-5, momentum 0.9, weight decay 0.0, and batch size 5. We set the hyper-parameters  $\mathcal{L}_{\mathrm{distill}} = 1$  and  $\mathcal{L}_{\mathrm{rank}} = 1$ .
D Additinal Results and Analyses
D.1 Variances of the Results in section 4
We provide variances of our results reported in our main paper. We compute the variances across 3 random seeds. Table 11 shows the variances of the test accuracy on Office-Home. The variances of the mean accuracy on FEMNIST and iWildCam are provided in Table 12 and in Table 13, respectively. Finally, Table 14 gives the variances of the test accuracy for each of the 9 tasks in VTAB. These results reveal that the reported accuracy is relatively robust across random seeds.
D.2 More Discussions on Seen and Unseen Classes
Different numbers of images per seen class. In the real world, it is unrealistic for end-users to collect data for all classes before adaptation. To further consider a lower data collection cost, we reduce the number of training images per seen class to study its effects. We conduct the experiment on the Office-Home dataset with "Art" as our source domain and "Clipart" as our target domain. Specifically, we randomly sample  $10\%$  and  $50\%$  of the training images for each seen class and
Table 11: Variances of domain adaptation 65-way test accuracy on Office-Home with 30 seen and 35 unseen classes (cf. Table 3). We compute the variances over 3 random seeds. Blue: HT methods suggested by us in section 3. Red: methods that significantly improve overall accuracy and successfully maintain unseen accuracy on the source model.  $\text{※}$ : the linear classifier is frozen during training.
Domains: source→target Methods / Acc.	Ar→Cl	Ar→Pr	Ar→Rw	Rw→Ar	Rw→Cl	Rw→Pr	Avg.
Overall	Unseen	Overall	Unseen	Overall	Unseen	Overall	Unseen	Overall	Unseen	Overall	Unseen	Overall	Unseen
Source	0.64	0.11	1.30	1.35	0.25	0.24	0.82	0.90	0.23	1.40	0.30	0.49	0.05	0.11
Naive Target	0.02	0.01	0.14	0.02	0.27	0.53	0.93	1.19	0.00	0.36	0.35	1.11	0.04	0.09
BN only	1.11	3.11	0.22	0.27	0.41	1.07	1.71	7.58	0.68	2.44	1.43	4.47	0.03	0.17
BN (stats) only	0.63	0.06	1.00	1.63	0.02	0.32	0.08	0.17	0.16	0.31	0.15	0.05	0.14	0.16
BN (stats) only	0.57	0.05	0.87	0.58	0.25	0.18	0.05	0.02	0.21	0.02	0.30	0.37	0.12	0.03
LP-FT	0.14	0.14	0.05	0.04	0.08	0.18	0.16	0.61	0.01	0.31	0.35	0.87	0.02	0.04
SGD (w/ frozen classifier *)	0.02	0.52	0.63	1.27	0.75	0.98	0.07	0.31	0.57	0.85	0.09	0.60	0.07	0.18
SGD + Ldistill *)	0.28	0.09	0.03	0.01	0.70	0.73	0.23	0.94	0.05	0.09	0.15	1.02	0.02	0.01
SGD + Lrank *)	1.02	0.78	0.39	0.48	0.88	1.13	0.01	0.51	0.07	0.67	0.09	0.03	0.08	0.05
SWA *)	0.23	0.73	0.62	1.44	0.33	0.60	0.45	0.31	1.39	3.39	0.13	0.65	0.02	0.15
SWAD *)	0.43	0.83	0.62	1.79	0.35	0.42	0.45	0.32	1.64	3.71	0.08	0.44	0.03	0.17
LOLSGD *)	0.36	0.20	0.62	1.00	0.20	1.25	0.73	0.61	1.20	1.59	0.17	0.65	0.00	0.11
LOLSGD + Lrank *)	0.02	0.05	0.24	0.95	0.06	0.05	0.34	0.22	0.63	1.09	0.16	0.96	0.04	0.14
LOLSGD + Ldistill *)	0.14	0.56	0.79	0.54	0.25	0.09	0.22	0.17	0.28	1.88	0.07	0.08	0.06	0.01
LOLSGD + Ldistill + Lrank *)	0.18	0.06	0.75	1.82	0.00	0.34	0.26	0.65	0.75	0.36	0.01	0.23	0.12	0.09
Oracle	0.05	0.05	0.06	0.16	0.01	0.16	0.17	0.02	0.22	0.05	0.35	0.20	0.02	0.00
Table 12: Variances of FEMNIST mean accuracy of 10 new writers (cf. Table 5). We compute the variances over 3 random seeds.
Methods	Overall	Seen	Seen (Chopping)	Unseen
Source	0.44	0.12	0.07	5.12
Naive Target	0.49	0.77	0.77	5.69
SGD*	0.43	0.17	0.11	1.90
SGD + Lrank*	0.55	0.08	0.04	4.05
SGD + Ldistill*	0.37	0.19	0.13	1.72
LOLSGD*	0.48	0.10	0.17	4.73
LOLSGD + Lrank*	0.37	0.10	0.20	2.04
LOLSGD + Ldistill*	0.48	0.16	0.37	4.90
LOLSGD + Ldistill +SE*	0.65	0.26	0.06	6.23
Table 13: Variances of iWildCam mean accuracy of 21 new locations (cf. Table 6). We compute the variances over 3 random seeds.
Methods	Overall	Seen	Seen (Chopping)	Unseen
Source	0.12	1.45	5.29	0.86
Naive Target	1.66	3.85	4.45	0.04
SGD*	0.10	1.48	1.67	0.01
SGD + Lrank*	0.94	7.73	2.41	2.35
SGD + Ldistill*	3.76	1.26	1.32	4.07
LOLSGD*	0.79	0.37	0.76	1.63
LOLSGD + Lrank*	1.70	7.85	5.03	3.98
LOLSGD + Ldistill*	1.33	0.67	1.86	3.36
LOLSGD + Ldistill +SE*	1.17	1.07	2.58	0.15
fine-tune the source model for the same iterations for fair comparisons. Interestingly, Table 15 shows that naive fine-tuning can obtain higher unseen accuracy, compared to naive fine-tuning with more data. The reason might be that training with more data needs to update the model weights more, making the unseen classes easier to forget. In contrast, applying our suggested HT methods, especially for LOLSGD with our regularization, the unseen classes can be better maintained across different training data sizes.