SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \hat {P} _ {\mathcal {T} ^ {*}} (\boldsymbol {x}, y; \boldsymbol {\theta} _ {\mathcal {T}}) := \hat {P} _ {\mathcal {T} ^ {*}} (y | \boldsymbol {x}; \boldsymbol {\theta} _ {\mathcal {T}}) \hat {P} _ {\mathcal {T} ^ {*}} (\boldsymbol {x}; \boldsymbol {\theta} _ {\mathcal {T}}), (2) \\ \leftarrow \hat {P} _ {\mathcal {T}} \cdot (y | x; \boldsymbol {\theta} _ {\mathcal {S}}) P _ {\mathcal {S}} (\boldsymbol {x}; \boldsymbol {\theta} _ {\mathcal {S}}), [ \text {S t y l e A d a p t a t i o n} \hat {P} _ {\mathcal {T}} \cdot (\boldsymbol {x}; \boldsymbol {\theta} _ {\mathcal {T}}) \leftarrow P _ {\mathcal {S}} (\boldsymbol {x}; \boldsymbol {\theta} _ {\mathcal {S}}) ] (3) \\ \leftarrow P _ {\mathcal {S}} (y | \boldsymbol {x}; \boldsymbol {\theta} _ {\mathcal {S}}) P _ {\mathcal {S}} (\boldsymbol {x}; \boldsymbol {\theta} _ {\mathcal {S}}). \quad \left[ P _ {\mathcal {T} ^ {*}} (y | \boldsymbol {x}) \approx P _ {\mathcal {T}} (y | \boldsymbol {x}) \approx P _ {\mathcal {S}} (y | \boldsymbol {x}) \right] (4) \\ \end{array}
$$
Equation 3 will hold if  $P_{\mathcal{T}^*}(\boldsymbol{x}|y) \approx P_{\mathcal{T}}(\boldsymbol{x}|y)$ . However, how can we disentangle the covariate shifts from  $\nabla \hat{\mathcal{R}}(f; \mathcal{T})$  and change the style to  $P_{\mathcal{T}}(\boldsymbol{x})$  only without concept shift [40]? We explore:
- Updating batchnorm statistics only. The statistics in batchnorm layers (in feed-forward networks) are believed to capture the domain information. Updating them is often considered as a strong baseline in DA [46]. It infers on  $\mathcal{T}$  and re-calculates the feature means and variances, which is less likely to be impacted by the missing classes issue.
- Training extra instance normalization layers. We explore another way based on instance normalization (IN) [72], which normalizes within each input itself over its spatial dimension. IN is widely used in style transfer to capture the input styles without changing their semantics. We examine a way for HT by inserting extra IN layers into the source model (which may not have IN layers originally) and fine-tuning them with other DNN parameters fixed to prevent concept shifts.
Unfortunately, we found updating normalization layers is not yet satisfying — receiving limited improvements and still suffering from catastrophic forgetting. We thus hypothesize the key is to consider the optimization besides architectures.
Proposed Leave-Out Local SGD (LOLSGD). In our HT problem, fine-tuning is affected by the covariate shift (from the source to the target) and the disruptive concept shift (from classifying all the classes to classifying the seen classes). We aim to disentangle them or, more precisely, reduce the disruptive concept shift by subsampling multiple datasets that each contain a subset of the seen classes.
We propose a novel approach based on local SGD [68] in distributed learning. The local SGD gradient is accumulated over several consecutive SGD steps, possibly many runs in parallel. For notations, let the local SGD gradient computed on the samples that their labels are inside a label space  $\mathcal{V}$  be  $g(\mathcal{V})$ . If sampling IID from  $\mathcal{T}$  and the local step is 1, it
Figure 3: LOLSGD illustration.
reduces to standard SGD. Instead of using  $g(\mathcal{V}_{\mathcal{T}})$ , which will bias to the seen target classes  $\mathcal{V}_{\mathcal{T}}$ , our idea is to mitigate the change towards the label and encourage the change in styles.
Inspired by recent theoretical analyses in meta-representation learning [21, 20, 12, 13], we intentionally sample  $m$  non-IID subsets of labels  $\mathcal{V}_{\mathcal{T}}^{m} \subsetneq \mathcal{V}_{\mathcal{T}}$  by leaving out some classes, compute  $\pmb{g}(\mathcal{V}_{\mathcal{T}}^{m})$  for each  $m$ , and average their local gradients  $\pmb{g}(\mathcal{V}_{\mathcal{T}}^{m})$
$$
\mathbf {g} _ {\text {L O L}} \left(\mathcal {Y} _ {\mathcal {T}}; M\right) := \frac {1}{M} \sum_ {m} \mathbf {g} \left(\mathcal {Y} _ {\mathcal {T}} ^ {m}\right), \mathbf {g} \left(\mathcal {Y} _ {\mathcal {T}} ^ {m}\right) := \boldsymbol {\theta} - \arg \min  _ {\boldsymbol {\theta}} \sum_ {\left\{i \in | \mathcal {T} | | y _ {i} \in \mathcal {Y} _ {\mathcal {T}} ^ {m} \right\}} \mathcal {L} \left(y _ {i}, f \left(\boldsymbol {x} _ {i}; \boldsymbol {\theta}\right)\right). \tag {5}
$$
During training, each update (with size  $\eta$ ) becomes  $-\eta g_{\mathrm{LOL}}$  instead of  $-\eta \nabla \hat{\mathcal{R}}(f; \mathcal{T})$ . Intuitively,  $g(\mathcal{V}_{\mathcal{T}}^{m})$  is biased towards the respective classes  $\mathcal{V}_{\mathcal{T}}^{m}$  and drifts away from each other along more local steps, similar to the model shift problem in decentralized learning [29, 39, 10]. As a blessing here (rather than a curse!), our key insight is averaging them could "cancel out" the gradients biased to certain classes. The style should be adapted for Equation 3 without conflicts since all local gradients are directed towards the same target style, as illustrated in Figure 3. When updating the model separately in parallel with these subsampled datasets, each updated model is affected by a different concept shift but a shared covariate shift. Averaging these models can potentially cancel out the disruptive concept shifts and strengthen the shared covariate shift. We verify our intuition in Table 19 in the supplementary. Compared to  $\nabla \hat{\mathcal{R}}(f; \mathcal{T})$ ,  $g_{\mathrm{LOL}}$  averages  $M$  ( $M = 10$  in our experiments) accumulated local gradients thus more expensive to compute. As will be shown by experiments, LOLSGD can outperform standard SGD with similar computation budgets in fair comparisons.
LOLSGD is fundamentally different from meta-learning, whose goal is to learn a meta-modal that can be easily applied to a future task (e.g., a few-shot classification task). While meta-learning also subsamples its meta-training set, it is mainly to simulate multiple future tasks for learning the meta-model, not to cancel out unwanted gradients. LOLSGD is also an extension of local SGD. The goal of local SGD is mainly to reduce the communication overhead of large-scale distributed training. In contrast, in LOLSGD, the target training data set is not decentralized initially, but we strategically subsample it to simulate different concept shifts.
3.3 Preserving Class Relationships
The subsection 3.2 assumes we can change the styles while not changing the conditional probability of the source model  $P_{S}(y|\boldsymbol{x})$  in Equation 4. We observe in practice it can still be sensitive and lean towards  $\{y^{\mathrm{Seen}}\}$ . We, therefore, hypothesize that it is necessary to have some regularization to preserve the nice class relationships encoded in the source model already during target training such that Equation 4 could hold. Let us decompose the predictor  $f(\boldsymbol{x};\boldsymbol{\theta}) = g(\boldsymbol{z};\boldsymbol{w}), \boldsymbol{z} = h(\boldsymbol{x};\boldsymbol{\phi})$  to be a feature extractor  $h$  and a linear classifier  $g$ . We explore the following strategies:
Figure 4: Feature rank collapse. Top singular values of Office-Home adapted features.
- Frozen linear classifier. Even the source features is already perfect for  $P_{S}(y|\boldsymbol{x}) = P_{\mathcal{T}^{\star}}(y|\boldsymbol{x})$ , it can collapse quickly as the logits of  $y^{\mathrm{Unseen}}$  will be suppressed by  $\nabla \hat{\mathcal{R}}(f; \mathcal{T})$ . We found it crucial to freeze the linear classifier  $\boldsymbol{w}$ , which corresponds to the common practice in source-free DA [48].
- Selective distillation. Another way is to utilize distillation that can prevent forgetting in CL [47, 93, 22], forcing the soft labels of the target model and the source model to match. Given a sample  $(\pmb{x}, y^{\mathrm{Seen}}) \in \mathcal{T}$  and let  $f^{\mathrm{Unseen}}(\pmb{x})$  be the logits of  $y^{\mathrm{Unseen}}$  dimensions. Here, only the unseen class logits need to be distilled since seen class logits are reliable as the target model is trained on ground truths, unlike source seen class logits (which are suboptimal due to domain shift). We add a distillation loss  $\mathcal{L}_{\mathrm{distill}} = \mathrm{KL}(\sigma(f^{\mathrm{Unseen}}(\pmb{x}; \pmb{\theta}_S))||\sigma(f^{\mathrm{Unseen}}(\pmb{x}; \pmb{\theta}_T)))$  with the Kullback-Leibler (KL) divergence, where  $\sigma(\cdot)$  is the softmax function.
- Feature rank regularization. A key observation is in the feature dynamic of fine-tuning on the partial target data — instead of shifting to another full-rank space, we found the features tend to collapse to a low-dimensional space, as shown in Figure 4 — implying the target model "shortcutting" the solution and hampering the generalization ability [76] to handle unseen target classes. We consider a regularizer to avoid too many singular values of the features becoming zeros. Given the feature matrix of a mini-batch  $\mathbf{Z} \in \mathbb{R}^{N \times d}$  and its mean feature  $\bar{z} \in \mathbb{R}^d$ , we compute its covariance matrix  $C = \frac{1}{N} \sum_{n=1}^{N} (\mathbf{z}_n - \bar{z})(\mathbf{z}_n - \bar{z})^\top$  and a regularizer  $\mathcal{L}_{\mathrm{rank}} = \| \mathrm{diag}(C^\top C) \|_2^2$ , inspired by [36, 17, 66].
We note that, these techniques only preserve the source class relationships, but they do not improve the transfer to the target. Thus they should be considered together with ERM or LOLSGD for HT.
Ensemble with the source model. As pointed out by [79], a way to reclaim some ability of the source model is to average the parameters of the source and target models, assuming they lie in a linearly-connected space. We consider it complementary post-processing for the target training techniques we considered above. We relax the assumption and further examine the ensemble of the source and target models in their predictions (after the softmax function  $\sigma$  for logits).
General setup. To answer the questions we raised in subsection 2.3, we experiment on extensive HT scenarios proposed in subsection 2.2. For each task, we pre-train a source model in standard practice, discard the source data, and adapt the source parameters  $\theta_{S}$  to obtain the target model. We split the target data for training and testing, based on different scenarios. The training and test sets share the same styles but cover different sets of classes, where some classes are unseen in training. We assume the source model covers all classes and the goal is to adapt to the target domain for both the seen and unseen classes. All methods use the cross-entropy loss for  $\mathcal{L}$  and SGD momentum optimizer. All experiments fine-tune the source model for 20 epochs (10 for FEMNIST) by default. For the regularizers in section 3, we attach them as  $\mathcal{L} + \lambda_{\mathrm{dsitill}}$  (or rank)  $\mathcal{L}_{\mathrm{dsitill}}$  (or rank), where the weights  $\lambda$  are quite stable thus we did not search for it exhaustedly for every method, but use the same ones per dataset. For the proposed LOLSGD, we set  $M = 10$  and randomly drop 3 classes when sampling  $\mathcal{Y}_T^m$  in Equation 5.
Table 4: Effects of Source Ensemble (Avg. as in Table 3).
Methods / Acc.	Overall	Unseen
Source	63.90	65.65
Naive Target (w/o SE)	53.50	22.36
Naive Target (W1SE)	53.56	25.26
Naive Target (SE)	67.46	50.89
SGD (W1SE)	69.75	60.42
SGD (SE)	71.01	59.19
SWA	64.99	45.10
SWAD	65.05	45.30
SGD +Ldistill	69.78	62.94
SGD +Lrank	71.05	65.48
LOLSGD	70.70	63.14
LOLSGD +Ldistill	68.71	64.56
LOLSGD +Lrank	69.94	66.29
Best ΔAcc_Naive Target	17.55	43.12
Each subgradient in LOLSGD is by
Table 3: Domain adaptation 65-way test accuracy on Office-Home with 30 seen and 35 unseen classes. Variances of random seeds are provided in the supplementary due to space limit. Blue: HT methods suggested by us in section 3. Red: methods that significantly improve overall accuracy and successfully maintain unseen accuracy on the source model.  $\text{※}$ : the linear classifier is frozen during training.
Domains: source→target Methods / Acc.	Ar→Cl	Ar→Pr	Ar→Rw	Rw→Ar	Rw→Cl	Rw→Pr	Avg.
Overall	Unseen	Overall	Unseen	Overall	Unseen	Overall	Unseen	Overall	Unseen	Overall	Unseen	Overall	Unseen
Source	47.07	50.29	67.45	72.00	72.73	74.68	65.73	68.19	51.13	49.26	79.28	79.47	63.90	65.65
Naive Target	44.96	9.06	52.39	23.75	59.48	32.91	54.93	28.03	46.92	6.34	62.31	34.08	53.50	22.36
BN only	45.94	14.33	54.30	30.25	63.22	41.91	61.47	42.32	49.10	13.42	71.57	51.96	57.60	32.36
BN (stats) only	47.44	50.44	65.03	70.88	71.76	74.12	65.73	70.89	55.56	55.75	80.09	82.96	64.27	67.51
IN only	48.96	49.42	68.19	71.13	76.23	74.26	66.40	68.46	54.89	51.48	79.35	79.05	65.17	65.63
LP-FT	43.91	6.73	48.05	16.38	55.81	25.60	50.67	19.14	45.04	3.54	56.21	22.21	49.95	15.60
SGD (w/ frozen classifier *)	52.11	24.12	64.36	44.75	70.26	55.27	68.27	53.10	55.26	24.19	75.75	59.22	64.34	43.44
SGD + Ldistill *)	56.54	39.18	72.81	61.63	75.73	67.51	70.13	64.15	61.96	51.45	80.60	70.53	69.63	57.41
SGD + Lrank *)	59.17	39.47	70.68	57.57	74.31	64.28	70.40	64.42	61.65	39.09	79.57	68.86	69.30	55.64
SWA *)	53.23	26.32	65.25	46.00	71.46	56.82	68.40	55.26	56.24	26.55	75.39	59.64	64.99	45.10
SWAD *)	53.38	26.46	65.25	46.00	71.61	57.24	68.53	55.53	55.87	26.40	75.68	60.20	65.05	45.30
LOLSGD *)	56.47	35.09	70.83	56.88	74.91	64.84	70.53	62.53	58.72	35.25	80.02	69.27	68.58	53.98
LOLSGD + Lrank *)	58.57	43.86	72.59	64.53	75.06	68.92	70.13	66.04	61.58	44.10	80.82	74.02	69.79	60.26
LOLSGD + Ldistill *)	57.44	46.35	75.24	67.38	76.33	70.75	69.07	65.77	60.68	45.58	81.56	75.28	70.05	61.85
LOLSGD + Ldistill + Lrank *)	60.83	51.75	75.75	70.25	76.70	74.26	70.13	69.54	65.71	53.10	82.95	80.31	72.01	66.54
Best ΔAcc_Naive Target	15.87	42.69	23.36	46.50	17.22	41.35	15.47	36.39	18.79	46.76	20.64	46.23	18.51	44.18
Oracle	79.32	82.46	90.96	91.50	84.57	84.95	78.13	80.59	79.85	75.52	91.92	92.04	84.13	84.51
local SGD  $(\frac{1}{M}$  epoch) and we run the same total epochs, for a fair computation budget. Please see the supplementary for the details of setup, hyperparameters, and more analyses.
4.1 Domain Adaptation on Partial Target Data
We first use the Office-Home dataset [74] with ResNet-50 [28] to study the effectiveness of the methods in section 3. Office-Home is a popular domain adaptation benchmark consisting of 65 object categories from four domains (Art, Clipart, Real, and Product).
Methods. We consider baselines including directly predicting with the Source model and naively fine-tuning it for a Target model. As proposed in section 3, we explore methods that promote covariate shifts by learning the limited parameters in normalization layers with the backbone parameters frozen, like batchnorms (BN) [31, 46] and an extra instance normalization (IN) [72] layer. We include another baseline LP-FT [42] proposed for the OOD generalization of fine-tuning but not for missing classes in HT. To compare with our LOLSGD, we also include related techniques in domain generalization based on stochastic weight average (SWA) [33] and its extension SWAD [7].
Comparison. We highlight the following observations in Table 3:
- Naive fine-tuning is disruptive: the unseen performance drops drastically, ultimately degrading overall accuracy and confirming the challenges of HT for all pairs of domains.
- Normalization layers may help: maintaining unseen accuracy and improving the seen little.
- Freezing the classifier during training is necessary: we found that the frozen classifier is crucial when the feature extractor is adapted (Avg. +21% Unseen vs. Target), motivating our approach of preserving class relationships subsection 3.3.
-  $\mathcal{L}_{\mathrm{distill}}$  and  $\mathcal{L}_{\mathrm{rank}}$  regularize forgetting further: largely mitigating unseen performance drops.
- LOLSGD is effective: without any regularizers, the proposed local SGD optimization can outperform the strong generalized SGD baselines SWA and SWAD.
- LOLSGD with regularizers performs best overall: as we suggested in section 3, LOLSGD needs to be carefully regularized while it adapts the domain styles.
Effects of Source Ensemble (SE). We further post-process the target models learned in Table 3 by ensembling with the source model, as introduced in subsection 3.3. We also include WISE [79] that ensembles the model weights instead of predictions. Generally, we found SE may reclaim some unseen performance, but not necessarily improve overall, as summarized in Table 4.
Overall, we show that HT is promising, improving the overall performance without sacrificing the unseen accuracy (highlighted in red in Table 3). Notably, our proposed solutions are based on optimization and agnostic to architectures. We will mainly focus on them later as they perform better.
Table 5: FEMNIST mean accuracy of 10 new writers. Table 6: iWildCam mean accuracy of 21 new locations.
Methods	Overall	Seen	Seen (Chopping)	Unseen
Source	84.67	88.67	89.60	64.99
Naive Target	82.67	92.07	92.07	35.60
SGD*	87.85	92.87	93.09	64.00
SGD + Lrank*	88.18	93.29	93.51	64.00
SGD + Ldistill*	87.44	92.46	92.81	61.91
LOLSGD*	87.47	91.87	92.23	66.27
LOLSGD + Lrank*	87.16	91.61	91.97	66.27
LOLSGD + Ldistill*	85.76	90.34	91.25	63.56
LOLSGD + Ldistill +SE*	85.10	89.52	90.58	63.56
Methods	Overall	Seen	Seen (Chopping)	Unseen
Source	35.71	35.74	58.40	26.43
Naive Target	35.38	51.08	52.72	1.90
SGD*	36.53	49.00	50.93	7.91
SGD + Lrank*	36.28	40.98	56.13	19.76
SGD + Ldistill*	41.59	50.40	54.23	17.90
LOLSGD*	38.12	48.91	52.49	13.30
LOLSGD + Lrank*	33.94	37.62	53.40	19.42
LOLSGD + Ldistill*	40.49	47.30	55.27	25.16
LOLSGD + Ldistill +SE*	39.96	44.79	58.22	25.66
Table 7: Fine-tuning CLIP ViT-B/32 on VTAB.  $50\%$  of classes each task are missing during training.
Overall/Unseen Acc. Methods	Caltech101	CIFAR100	DTD	EuroSAT	Flowers102	Pets	Resisc45	SVHN	SUN397	Avg.
All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.	All.	Uns.
Source	78.7	70.5	64.2	62.4	43.1	47.0	32.2	20.3	63.8	68.4	84.1	88.7	54.0	52.7	8.8	11.3	46.7	49.1	52.8	52.3
Naive Target	82.3	69.1	65.9	42.7	45.3	15.8	52.2	1.2	64.2	49.2	84.5	75.8	58.5	23.8	39.8	0.2	49.7	35.4	60.3	34.8
SGD*	82.3	69.0	66.1	42.7	45.5	16.4	52.5	1.8	64.5	49.8	84.7	75.8	57.9	21.9	40.1	0.5	49.8	36.0	60.4	34.9
LOLSGD*	82.6	69.8	69.6	51.8	46.0	18.8	52.7	2.7	65.3	51.0	85.9	78.2	59.4	27.1	36.7	0.7	52.5	45.5	61.2	38.4
LOLSGD + Ldistill*	82.0	69.4	67.1	49.1	48.0	22.9	52.7	3.8	66.3	52.9	83.8	74.2	60.0	29.0	35.1	0.5	52.6	47.6	60.8	38.8
LOLSGD + Lrank*	82.8	70.3	70.0	52.5	45.3	18.4	51.4	1.4	64.2	50.9	84.6	76.8	59.5	27.6	34.5	0.3	52.8	46.1	60.6	38.3
LOLSGD + Lrank + SE*	82.5	70.8	72.3	58.4	52.6	37.3	54.5	8.4	69.2	62.5	87.3	83.2	64.8	39.7	35.8	3.8	53.4	50.0	63.6	46.0
Oracle	89.0	87.2	83.1	82.6	60.0	61.3	97.7	97.1	75.5	80.3	90.4	91.7	90.1	90.3	94.2	95.6	58.8	60.0	82.1	82.9
4.2 Realistic Holistic Transfer: FEMNIST and iWildCam Datasets
We simulate two realistic scenarios in which a new user collects a limited-size adaptation set and aims to customize a source model. Each new user has its natural domain shift.
FEMNIST [4]: a 62-class hand-written characters dataset that consists of many writers [19] of different styles. We train a LeNet [44] on 40 writers and adapt to each of the 10 new writers. Each writer only has a few samples per class, thus when the data are split into training and test sets (in ratio of  $7:3$ ) for personalization may suffer mismatched  $P_{T}(y)$ .
iWildCam [38, 2]: animal recognition dataset of 181 classes of camera traps located worldwide. We train an ImageNet pre-trained ResNet-50 on 53 locations and adapt to 21 new locations. The data of each location are split by a timestamp; we train on previous data and test on future data.
Encouragingly, the observations on both natural datasets are generally consistent with the Office-Home experimental dataset. As shown in Table 5 and Table 6. The seen and unseen accuracy can be improved upon the source model at the same time, even without an ensemble. Here, we note that the seen class accuracy might be misleading. It might be two reasons if the seen class accuracy is high: 1) it adapts well to the target style and/or 2) the model simply collapses to an easier, fewer-way classifier. Just for a better understanding of the quality of the adapted features, we also report the seen classes' accuracy but "chopping out" the unseen classifiers in testing<sup>2</sup>. The seen accuracy with a chopped classifier is also competitive to naive fine-tuning, validating the feature quality is good; lower accuracy is mainly due to more classes making the problem harder.
4.3 Fine-Tuning on Zero-Shot CLIP ViT for Vision Tasks
Next, we go beyond domain adaptation and consider distribution shifts at the task level using the VTAB [92] benchmark that contains various vision tasks. We split each task by its labels; roughly half of the classes are considered unseen. We use the CLIP pre-trained ViT-B/32 [60]. Only those tasks in which the class names are in natural languages are considered using their text embeddings for zero-shot predictions. The results are summarized in Table 7. We further compare to the CoCoOp [96] fine-tuning method designed for handling unseen classes in the supplementary.
Across the tasks, we see HT methods remain effective in most cases. Interestingly, we see some exceptions that the degrees of improvement become diverged, which inspires our discussions next.
In realistic cases, the fully-observed environments are not available for the oracle upper bounds.
(a) CIFAR100
(b) Flowers102
(c) DTD
Figure 5: Disparate impact: will fine-tuning improve similar tasks? We fine-tune CLIP on the training set of a task and evaluate on others. Naively fine-tuned target models actually degrade more on more similar tasks, while our HT method is much more robust. Tasks are sorted in the x-axis based on Affinity Score following [25].
4.4 Studies: When should HT Work and Not Work?
Semantic gaps of the source model. As discussed in subsection 2.3, HT relies on the DA assumption that the source and target tasks share similar semantics, i.e.,  $P_{S}(y|x) \approx P_{\mathcal{T}}(y|x)$ . There should be little semantic shifts from the source to the target. In Table 7, we found the unseen class performance of some tasks is very low, and applying HT methods cannot help much. We hypothesize that the images are likely to be predicted as other meanings by the CLIP model rather than the goal of the target task. Looking at the Street View House Numbers (SVHN) dataset [54], we adversarially add class names like "number plate" to confuse with the digits (e.g., "9"). Interestingly, we found about  $60\%$  of the cases, CLIP fails to predict a number (Table 8). This validates that  $P_{S}(y|x)$  pre-trained by CLIP is quite different from the digit classification, therefore, it becomes very hard if not impossible for it to recognize unseen digits correctly. Similarly, DTD (texture) and EuroSAT (land use) could suffer from the same concept shift issue. Resolving it requires a "task-level generalization" (e.g., object recognition → digits classification), which is obviously out of the scope of HT.
Disparate impact. If we fine-tune a task, should it benefit other similar tasks? We reveal that the answer might be negative in HT settings. We fine-tune CLIP on a task following Table 7, evaluate on others using the corresponding CLIP text classifiers, and summarize the accuracy difference to the original CLIP in Figure 5. Intriguingly, more similar tasks in fact drop more, likely because they are disrupted by the fine-tuned task most. This again highlights the importance of HT; we found our method makes it more robust.