SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cheng-Hao Tu $^{1}$ , Hong-You Chen $^{1}$ , Zheda Mai $^{1}$ , Jike Zhong $^{1}$ , Vardaan Pahuja $^{1}$ ,
Tanya Berger-Wolf<sup>1</sup>, Song Gao<sup>2</sup>, Charles Stewart<sup>3</sup>, Yu Su<sup>1</sup>, Wei-Lun Chao<sup>1</sup>
<sup>1</sup>The Ohio State University, <sup>2</sup>University of Wisconsin-Madison, <sup>3</sup>Rensselaer Polytechnic Institute
We propose a learning problem involving adapting a pre-trained source model to the target domain for classifying all classes that appeared in the source data, using target data that covers only a partial label space. This problem is practical, as it is unrealistic for the target end-users to collect data for all classes prior to adaptation. However, it has received limited attention in the literature. To shed light on this issue, we construct benchmark datasets and conduct extensive experiments to uncover the inherent challenges. We found a dilemma — on the one hand, adapting to the new target domain is important to claim better performance; on the other hand, we observe that preserving the classification accuracy of classes missing in the target adaptation data is highly challenging, let alone improving them. To tackle this, we identify two key directions: 1) disentangling domain gradients from classification gradients, and 2) preserving class relationships. We present several effective solutions that maintain the accuracy of the missing classes and enhance the overall performance, establishing solid baselines for holistic transfer of pre-trained models with partial target data.
We are entering an era in which we can easily access pre-trained machine-learning models capable of classifying many objects. In practice, when end-users deploy these models in diverse domains, adaptation is often necessary to achieve high accuracy. To address this issue, considerable efforts have been devoted to domain adaptation (DA), aiming to transfer domain knowledge from the source to the target. This involves updating the model on an adaptation set collected from the target environment. Although various DA settings have been proposed to tackle different scenarios [75, 56, 55], we argue that previous setups rely on a strong assumption that hinders their applicability in the real world: the adaptation set contains samples from all possible labels (e.g., classes) in the target environments. Taking classification as an example, to adapt a pre-trained source model's recognition ability over  $C$  classes to a novel domain, DA necessitates access to a target dataset encompassing all  $C$  classes.
In reality, the adaptation set may only have samples of much fewer classes  $C' < C$  due to limited sample size, collections bias, etc. Many pertinent examples can be found in realistic AI applications. Considering wildlife monitoring [38] as an instance, where data collection often occurs passively, through smart camera traps, awaiting animal appearances. Consequently, when a smart camera trap is deployed to a new location and requires adaptation, assembling a comprehensive target dataset containing all the animal species becomes a daunting task, especially if a specific rare species does not appear within the data collection period due to seasonal variations or animal migrations.
Table 1: Motivating examples of Holistic Transfer (HT). We propose Holistic Transfer, a novel and realistic transfer learning problem. Unlike traditional paradigms, HT focuses on scenarios where only a subset of classes from the target test environment is present in the target training set. Naive fine-tuning (FT) here can be disruptive.
Dataset	Scenario	Source
Real (65 classes)	Target Training
Clipart (30 classes)	Target Test
Clipart (65 classes)
Office-Home [74]	Domain shift	Many writers: 0-9, a-z	Writer X: a, 5	Writer X: b, 5, e, x
FEMNIST [4]	Personalization on scarce data	a	f	a	g	e
5	m	s	x
Many locations	Location N: before 2010	Location N: after 2010
iWildCam [38]	Camera traps in the wild
CLIP's training data	Caltech101 (50 classes)	Caltech101 (101 classes)
VTAB [91]	FT zero-shot models
CLIP's training data	Non-toxic	Non-toxic, toxic
iNaturalist (Fungi) [73]	Confusion classes
To underscore the significance of the problem we address in this paper, it is worth noting that the literature has made significant progress towards training a general recognition model that can predict almost any classes in the vocabulary space such as CLIP [60] and ALIGN [35]. However, we argue that the current DA approaches applied to these pre-trained models face a practical learning dilemma in adapting the versatile classification capability. On the one hand, adaptation to the target domain style is preferred. On the other hand, the models adapted on target training data whose labels cover a subset of the target test data can lead to risky outcomes for the end users. Through experiments, we observed that fine-tuning the model with empirical risk minimization on such partial target data, which fails to represent the overall target distribution, hampers the performance of classes that were already pre-trained but unavailable during target adaptation.
This is frustrating for both parties as a lose-lose dilemma: on the pre-training side, it negates the extensive effort invested in preparing for such a source model as the foundation for adaptation. On the side of downstream users, despite the collection of a target dataset for adaptation purposes, the difficulty of covering all target classes results in an adapted model performing even worse than the source model in the actual target environment. In response to the dilemma, we ask: can we transfer more holistically, i.e., improving the overall performance without hurting missing class performance? We propose a novel and realistic transfer learning paradigm that aims to transfer the source model's discriminative capabilities to the target domain — specifically, the task of classifying all  $C$  objects in the source domain — while working with a target training dataset comprising only a limited subset of these classes, i.e.,  $C' < C$ . We refer to the problem as Holistic Transfer (HT).
Contributions. We formulate a new learning paradigm called Holistic Transfer (HT).
1. We define a new, practical transfer learning paradigm that complements existing ones (section 2).
2. We construct extensive benchmarks in subsection 2.2 and section 4 to support HT studies.
3. We systematically explore methods to approach HT in section 3 and provide strong baselines.
4. Our extensive experiments shed insights towards solving HT (section 4 and section 5).
2 Holistic Transfer Learning
We propose a novel and realistic transfer learning paradigm called Holistic Transfer (HT).
Setup. We focus on the task of multi-class classification. Let us consider a source model parametrized by  $\theta_{S}$ . It is pre-trained on a source dataset  $S = \{(x_i^S, y_i^S)\}_{i=1}^{|\mathcal{S}|}$ , where  $x_i^S$  is the input (e.g., an image) and  $y_i^S$  is the corresponding label (e.g., a class of the  $C$  classes in the label space).
<sup>1</sup>As will be introduced in subsection 2.2, the concept of the "source" domain can be quite general: a dataset with different input style, data from a collection of different users, a general pre-training dataset, etc.
Let the joint distribution of the source be  $P_{S}$  and the true distribution of the target environment be  $P_{\mathcal{T}^{\star}}$ . The goal of HT, just like standard domain adaptation, is to leverage the source model to learn a target model  $\theta_{\mathcal{T}}$  that performs well on the true target data, measured by a loss  $\mathcal{L}$ .
Remarks: Holistic Transfer. Typical transfer learning setups assume the target training data  $\mathcal{T}$  are sampled IID from  $P_{\mathcal{T}^{\star}}$ . However, this may not hold in practical scenarios due to sampling bias or insufficient sample size, making the target training data  $\mathcal{T}$  biased; i.e.,  $P_{\mathcal{T}} \neq P_{\mathcal{T}^{\star}}$ . The target training data  $\mathcal{T} = \{(x_i^{\mathcal{T}}, y_i^{\mathcal{T}})\}_{i=1}^{|\mathcal{T}|}$  from the target domain are limited to a subset of classes  $C' < C$ .
At first glance, our setting seems quite similar to partial DA [5], which also considers the situation of partial target classes (i.e.,  $C' < C$ ). However, its goal is to perform well only on those  $C'$  classes after adaptation. In contrast, our setting aims to adapt the source model's holistic capability of recognizing all  $C$  classes to the target domain. Below we first give the HT definition grounding on the literature formally. We will contrast the HT problem with other existing learning paradigms in subsection 2.3.
Assumptions of  $P_{S}$  vs.  $P_{\mathcal{T}^{\star}}$ . Following standard domain adaptation [75, 56, 55], we consider  $P_{S}$  and  $P_{\mathcal{T}^{\star}}$  have covariate shifts; thus the target features need to be adapted from the source ones to tailor the target domain style, i.e.,  $P_{S}(x|y) \neq P_{\mathcal{T}^{\star}}(x|y)$ . Typically, the label space of the true target domain is already covered by the source counterpart, or  $\mathcal{V}_{\mathcal{T}^{\star}} \subseteq \mathcal{V}_{\mathcal{S}}$ .
Assumptions of  $P_{\mathcal{T}}$  vs.  $P_{\mathcal{T}^{\star}}$ . We study the setting that  $\mathcal{T}$  is a (potentially biased) sampled dataset from  $P_{\mathcal{T}^{\star}}$ , therefore,  $P_{\mathcal{T}}$  and  $P_{\mathcal{T}^{\star}}$  should share the same domain styles and semantics. That is,  $P_{\mathcal{T}}(\boldsymbol{x}|y) \approx P_{\mathcal{T}^{\star}}(\boldsymbol{x}|y), P_{\mathcal{T}}(y|\boldsymbol{x}) \approx P_{\mathcal{T}^{\star}}(y|\boldsymbol{x})$ . However, the label space of  $P_{\mathcal{T}}$  might be incomplete w.r.t.  $P_{\mathcal{T}^{\star}}; P_{\mathcal{T}}(y) \neq P_{\mathcal{T}^{\star}}(y)$ . In other words, the test set  $\mathcal{T}^{\star}$  sampled from  $P_{\mathcal{T}^{\star}}$  might contain data belonging to classes seen in  $S$  but not appear in the target training set  $\mathcal{T}$ . The ultimate goal of the target model is to perform well on  $P_{\mathcal{T}^{\star}}$ .
Objective. The objective of HT is to learn a predictor  $f(\cdot ;\pmb{\theta}_{\mathcal{T}}):\mathcal{X}_{\mathcal{T}^{\star}}\to \mathcal{Y}_{\mathcal{T}^{\star}}$  that minimizes the expected loss on the true target distribution  $P_{\mathcal{T}^{\star}}$  but given only the partial observations of a target training dataset  $\mathcal{T}$  sampled from  $P_{\mathcal{T}}$  without accessing to  $P_{\mathcal{T}^{\star}}$ :
$$
\mathcal {R} (f; P _ {\mathcal {T} ^ {\star}}) = \mathbb {E} _ {(\boldsymbol {x}, y) \sim P _ {\mathcal {T} ^ {\star}}} [ \mathcal {L} (y, f (\boldsymbol {x})) ]; \quad \hat {\mathcal {R}} (f; \mathcal {T}) = \sum_ {\left\{\left(\boldsymbol {x} _ {i}, y _ {i}\right) \right\} _ {i = 1} ^ {| \mathcal {T} |}} \mathcal {L} \left(y _ {i}, f \left(\boldsymbol {x} _ {i}\right)\right). \tag {1}
$$
That is, we can only compute the empirical risk  $\hat{\mathcal{R}}(f; \mathcal{T})$  on  $\mathcal{T}$  but not the true risk  $\mathcal{R}(f; P_{\mathcal{T}^*})$ .
Challenges in a realistic HT scenario. We focus on a more realistic scenario of HT that comes with the following practical constraints and properties when solving  $\mathcal{R}(f;P_{\mathcal{T}^{\star}})$  in Equation 1.
- Missing classes in target training data. The main challenge is that some classes might only appear in  $P_{\mathcal{T}^{\star}}$  in testing but not yet seen in the target training set  $\mathcal{T}$ . Minimizing  $\hat{\mathcal{R}}(f; \mathcal{T})$  naively might be risky since it is biased estimation of  $\mathcal{R}(f; P_{\mathcal{T}^{\star}})$ .
- Disparate impact [26]. Training on the partial target data  $\mathcal{T}$  can result in disproportionate effects on the performance of unseen target classes in testing. The impact may vary based on the semantic affinity between classes, potentially causing serious false negatives. For instance, if a biologist fine-tunes an ImageNet classifier on some non-toxic plants and uses it to recognize plants in the wild, it will likely misclassify those visually similar toxic plants as non-toxic ones.
- Covariate shifts from the source domain. If we only have  $\mathcal{T}$ , the missing classes are very difficult to be predicted as it is zero-shot classification. A more practical scenario is to leverage an external source classifier that has been pre-trained on  $\mathcal{S}$  with many classes. This relaxes the problem to be a special (supervised) domain adaptation task that leverages the source classifier's discriminative ability (as  $P_{\mathcal{S}}(y|\boldsymbol{x}) \approx P_{\mathcal{T}^{\star}}(y|\boldsymbol{x})$ ) while adapting the input styles  $P_{\mathcal{S}}(\boldsymbol{x}) \neq P_{\mathcal{T}^{\star}}(\boldsymbol{x})$ .
- Source-data-free transfer. Last but not least, the source model  $\theta_{S}$  is available but the source dataset  $S$  is not. Due to privacy concerns or the dataset's large size, accessing the source dataset is often unrealistic, especially for foundational models pre-trained in large scales like CLIP [60].
Therefore, we believe HT is an important and challenging problem to approach for improving towards safer transfer learning in the wild, given only a source classifier  $\theta_{\mathcal{S}}$  and a partial target dataset  $\mathcal{T}$ .
2.2 Benchmarks Curation for Holistic Transfer: Motivating Examples
To support the study of the HT problem, we create a benchmark that covers extensive scenarios across both experimental and realistic public datasets. More details about dataset curation are in the supplementary. As introduced in Table 1, HT is applicable in various use cases, such as:
- Office-Home [74]: domain adaptation while some classes are missing in the target training set.
- FEMNIST [4]: personalized hand-written alphanumeric recognition with writers' styles. Training samples of a single writer are insufficient to cover all classes.
- iWildCam [38]: species recognition across camera traps of many geo-locations. Adaptation is based on images collected so far, and is deployed for the future; i.e., samples are biased temporally.
- VTAB [91]: fine-tuning zero-shot models for diverse vision tasks with partial classes each.
- iNaturalist (2021 version, Fungi) [73]: classification of visually-similar poisonous fungi.
Each task pre-trains a source model on the source domain, transfers to the target domain with only partial classes, and tests on the full-class target test data. For the Office-Home dataset, the source model is trained on one domain different from the target domain. For the FEMNIST/iWildCam datasets, the source and target data are split by writers/locations. For the VTAB and iNaturalist datasets, there are no explicit domains. A general dataset can be used for pre-training the source model. Here we will use CLIP [60] just as an example.
2.3 Comparison to Other Paradigms: To Adapt or Not to Adapt, and How?
The most common way of transfer learning is probably fine-tuning, i.e., initializing the target model with the source model parameters and optimizing it with empirical risk minimization (ERM) on the target data. In the HT setup, fine-tuning by minimizing  $\hat{\mathcal{R}}(f; \mathcal{T})$  in Equation 1, on the one hand, improves on the performance of those classes seen in the target training set  $\mathcal{T}$ . On the other hand, it can be disruptive in terms of the true risk  $\mathcal{R}(f; P_{\mathcal{T}^*})$  due to catastrophic forgetting [37] of those classes already seen in the source domain but unseen during the target fine-tuning. This creates a dilemma for the user about whether to adapt or not. As shown in the example in Figure 1 (details in section 4), naive fine-tuning does improve
Figure 1: Fine-tuning disrupts unseen target class performance. An HT example of  $\mathrm{Ar} \rightarrow \mathrm{Cl}$  domains on Office-Home dataset.
those seen classes over the source model while unseen classes can degrade drastically. On the contrary, our proposed treatment (see section 3) can maintain the unseen performance and improve the seen ones, compared to the source model (the 0 epoch).
We note the HT problem we proposed is a unique machine learning paradigm that lies between and complementary to existing learning-with-distribution-shift problems [59]. From a high-level and simplified view, many of them involve three stages: 1) source training on  $S$  to obtain a source model, 2) target training on  $\mathcal{T}$ , then 3) evaluation of the target goal distribution  $P_{\mathcal{T}}^{test}$ .
We summarize the difference to some popular paradigms in Table 2, including general transfer learning and (supervised) domain adaptation (DA) [98, 75] that assume the target distributions in training and testing are matched. Continual learning (CL) [37, 51, 47] that typically disallows access to source data during target training, aiming to build a multi-tasking model that performs well on all the (possibly many) distributions that have been trained on so far. They all assume the training and test data are matched in distributions  $(P_{\mathcal{T}} \approx P_{\mathcal{T}}^{\text{test}})$ . Some recent works about OOD domain generalization [41, 3, 79] study fine-tuning from a versatile pre-trained model and preserving its robustness in testing to different unseen input styles of the seen classes but not for unseen classes.
As illustrated in Figure 2 (d), HT requires a very different ability (the green arrow): "generalize" the style shifts learned on the target seen classes to the classes unseen during target training (the red dashed arrow). Compared to OOD generalization, our goal is much harder since it requires transferring the source model to the target styles, for classes both seen and unseen in the target training data. Given only seen classes, using fine-tuning here is notoriously biased to them and disrupts the unseen performance. HT is also addressing quite different challenges from source-free DA [48], unsupervised DA (with partial classes [32]), and test-time adaptation [69] which focus on recovering the true labels for the unlabeled target data. More related works are in the supplementary.
Our goals. To this end, we raise the following research questions:
1. Can we resolve the adaptation dilemma and transfer holistically from the source model, improving the test performance of classes seen in the target training set while not hurting the unseen ones?
2. Is it possible to generalize the style shifts to unseen classes? Or how far are we?
3. When should HT work and not work in realistic cases?
Table 2: Comparison of different paradigms: Source Training  $\rightarrow$  Target Training  $\rightarrow$  Test on Target Goal. Let  $P_{S}$  and  $P_{\mathcal{T}}$  be the distributions of the source data  $S$  and the target training set  $\mathcal{T}$ , respectively.  $P_{\mathcal{T}}^{test}$  is the target test distribution as the final goal.  $P_{\mathcal{T}^{\star}}$  is the distribution that may have certain discrepancy from  $P_{\mathcal{T}}$ .
Settings	Target Training	Target Goal PTtest	PS vs. PTtest	PTvs. PTtest
Standard transfer	\( \theta_S \) and/or \( S, T \)	\( P_T \)	\( P_S \neq P_T^{test} \)	\( P_T \approx P_T^{test} \)
Domain adaptation	\( \theta_S \) and/or \( S, T \)	\( P_T \)	\( P_S(y|x) \approx P_T^{test}(y|x) \)	\( P_T \approx P_T^{test} \)
Continual learning	\( \theta_S, T \)	\( P_S + P_T \)	\( P_S \neq P_T^{test} \)	\( P_T^{test} \approx P_S + P_T \)
OD generalization	\( \theta_S, T \)	\( P_T^* \)	\( P_S \neq P_T^{test} \)	\( P_T(x|y) \neq P_{T^*}(x|y), \) \( P_T(y) \approx P_{T^*}(y) \)
Holistic transfer	\( \theta_S, T \)	\( P_T^* \)	\( P_S(y|x) \approx P_T^{test}(y|x) \)	\( P_T(x|y) \approx P_{T^*}(x|y), \) \( P_T(y) \neq P_{T^*}(y) \)
(a) Domain Adaptation
(b) Continual Learning
Figure 2: Different settings of learning with distribution shift. Consider a data distribution space of different styles (x-axis) and classes (y-axis), each algorithm leverages the source data (the blue region) to facilitate the knowledge transfer (the green arrow) to the target training data (the yellow region). The target test distribution  $P_T^{test}$  is highlighted with the black box. The red arrow represents the generalization for unseen test data.
(c) OOD Generalization
(d) Holistic Transfer
3 Towards Non-Disruptive Fine-Tuning
3.1 Solving the HT Objective
We now discuss approaching the HT objective of Equation 1. The key difficulty is the fact that the true risk  $\mathcal{R}(f;P_{\mathcal{T}^{\star}})$  and the empirical risk  $\hat{\mathcal{R}} (f;\mathcal{T})$  are in different distributions (i.e.,  $P_{\mathcal{T}^{\star}}(y)\neq P_{\mathcal{T}}(y))$  such that methods relying on ERM fine-tuning can be seriously biased to seen classes, as discussed in subsection 2.3. Let  $P_{\mathcal{T}}$  has  $C$  labels  $\mathcal{Y}_{\mathcal{T}} = \{y_1^{\mathrm{Seen}},\dots,y_C^{\mathrm{Seen}}\}$  and  $P_{\mathcal{T}^{\star}}$  has some extra unseen labels  $\mathcal{Y}_{\mathcal{T}^{\star}} = \mathcal{Y}_{\mathcal{T}}\cup \{y_{C + 1}^{\mathrm{Unseen}},y_{C + 2}^{\mathrm{Unseen}},\dots,y_{C^{\star}}^{\mathrm{Unseen}}\}$ . Worth noting, we fairly assume the source data cover all target classes  $\mathcal{Y}_{\mathcal{T}^{\star}}\subseteq \mathcal{Y}_S$ ; if a test class was not seen in  $\mathcal{S}$ , it becomes a zero-shot problem which might not be solved by transfer learning.
Oracle solutions. An oracle solution is to directly match the distributions, making  $P_{\mathcal{T}^*}(y) = P_{\mathcal{T}}(y)$  by augmenting data to  $\mathcal{T}$ . For instance, one can collect more data for those unseen classes in the target domain, or translate the source data sample of an unseen class  $(x^S, y_C^{\mathrm{Unseen}})$  into  $(x^T, y_C^{\mathrm{Unseen}})$  with a style transfer model from  $S$  to  $\mathcal{T}$ . They are obviously not available for HT—the former requires the cost to collect complete labeled data in the target (essentially the upper bound in domain adaptation); the latter needs to access the source data which is unrealistic as discussed in subsection 2.1.
Suboptimal solutions. By leveraging techniques in other paradigms we discussed in subsection 2.3, it might be possible to improve over plain fine-tuning. However, none explicitly recovers the joint probability of the target goal distribution  $P_{\mathcal{T}^{\star}}(\pmb{x}, \pmb{y})$ , but only  $P_{\mathcal{T}}(\pmb{x}, \pmb{y})$  or  $P_{\mathcal{T}}(\pmb{y})$ .
Next, we propose two directions that together could approach  $P_{\mathcal{T}^{\star}}(\pmb{x},y)$  under HT constraints.
3.2 Disentangling Covariate Shifts from Disruptive Concept Shifts
Minimizing  $\hat{\mathcal{R}}(f; \mathcal{T})$  on  $\mathcal{T}$  transfers the model from  $P_{S}(\boldsymbol{x}, y; \boldsymbol{\theta}_{S})$  to  $\hat{P}_{\mathcal{T}}(\boldsymbol{x}, y; \boldsymbol{\theta}_{\mathcal{T}})$  which optimizes towards the target styles  $P_{\mathcal{T}}(\boldsymbol{x}|y)$  (which should be the same as  $P_{\mathcal{T}^{\star}}(\boldsymbol{x}|y)$ ) but also collapses the label distribution to  $\hat{P}_{\mathcal{T}}(y; \boldsymbol{\theta}_{\mathcal{T}})$ . The model will unlikely predict a  $y_{C}^{\mathrm{Unseen}}$  for any inputs. If we can only adapt the covariates from the source classifier, then we have