SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Beier Zhu $^{1}$  Kaihua Tang $^{1}$  Qianru Sun $^{2}$  Hanwang Zhang $^{1}$
$^{1}$ Nanyang Technological University  $^{2}$ Singapore Management University
beier002@e.ntu.edu.sg, hanwangzhang@ntu.edu.sg
Foundation models like CLIP allow zero-shot transfer on various tasks without additional training data. Yet, the zero-shot performance is less competitive than a fully supervised one. Thus, to enhance the performance, fine-tuning and ensembling are also commonly adopted to better fit the downstream tasks. However, we argue that such prior work has overlooked the inherent biases in foundation models. Due to the highly imbalanced Web-scale training set, these foundation models are inevitably skewed toward frequent semantics, and thus the subsequent fine-tuning or ensembling is still biased. In this study, we systematically examine the biases in foundation models and demonstrate the efficacy of our proposed Generalized Logit Adjustment (GLA) method. Note that bias estimation in foundation models is challenging, as most pre-train data cannot be explicitly accessed like in traditional long-tailed classification tasks. To this end, GLA has an optimization-based bias estimation approach for debiasing foundation models. As our work resolves a fundamental flaw in the pre-training, the proposed GLA demonstrates significant improvements across a diverse range of tasks: it achieves 1.5 pp accuracy gains on ImageNet, a large average improvement (1.4-4.6 pp) on 11 few-shot datasets, 2.4 pp gains on long-tailed classification. Codes are in https://github.com/BeierZhu/GLA.
Thanks to the Web-scale data and self-supervised strategies, foundation models like CLIP [40] empower zero-shot transfer to a wide variety of domains [7, 1, 52]. However, the zero-shot performance is still weak on several domain-specific tasks such as differentiating models of cars, species of flowers, and variants of aircraft [40, 7]. Therefore, it is a common practice to improve the downstream performance via supervised fine-tuning on labeled data, e.g., linear probing, prompt tuning [54, 55], and end-to-end fine-tuning.
However, fine-tuned models are easily biased: they are adept in exploiting spurious correlations that only hold on the downstream distribution [40, 39, 48, 55]. To improve the robustness, several studies [48, 55, 56] propose to combine fine-tuned models with zero-shot models. For example, WiSE-FT [48] ensembles the fine-tuned and zero-shot models in weight space and ProGrad [55] uses zero-shot predictions to regularize the fine-tuning gradient. The underlying assumption lies in that the zero-shot models are robust to distribution shifts [40], and their predictions are complementary to those of fine-tuned models [48].
Despite these methods exhibiting performance gains on both in-distribution and out-of-distribution evaluations, they all overlook the inherent bias originating from the foundation models. Specifically, the Web-scale data for pre-training foundation models exhibit a highly skewed distribution due to Zipf's law of nature [42]. The resulting foundation models develop a biased decision boundary that leads to a poor zero-shot performance on rare classes. As evidenced in Figure 1(a) and (b), the purple
Figure 1: (a) Per class accuracy of CLIP-ViT/B16 on ImageNet. Class index are sorted using the estimated pre-training label prior. Curves are smoothed for better visualization. (b) Break-down performance of different models on ImageNet. We equally divide the ImageNet classes into three subgroups, according to the class index. Existing ensemble methods like WiSE-FT [48] exhibits a clear performance loss on tail classes, while our GLA stands out for all three subgroups.
line encounters a dramatic drop, and the zero-shot performance of tail classes is significantly lower than that of head classes (57.2% vs. 78.0%). Existing ensemble methods like WiSE-FT [48] overlook the label bias, resulting in an improvement in top-1 (+0.5%) and head accuracy (+1.7%) while a noticeable degradation on the tail performances (-0.9%) in Figure 1(b). Another evidence is that the orange line (WiSE-FT) is below the blue line (fine-tuned models) for rare classes in Figure 1(a).
We propose Generalized Logit Adjustment (GLA), a simple post-hoc method consisting of two steps: 1) removing the label bias of zero-shot model via estimating the label distribution in the pre-training dataset; 2) ensembling the fine-tuned and debiased zero-shot models. As illustrated in Figure 1 (b), our GLA achieves consistent improvement across all three subgroups, particularly showing a significant gain on tail classes  $(+1.5\%)$ . Despite its simplicity, our GLA has a firm statistical grounding: it is the Bayes optimal classifier given the fine-tuned and zero-shot models, thus consistent for minimizing the error on a class-balanced target distribution (Section 4.2). It is worth noting that removing the bias of foundation models is challenging since the label distribution is often inaccessible due to privacy or copyright concerns. In this work, we only use the downstream labeled data and the zero-shot model to estimate the foundation label bias. Specifically, we formulate the problem by adjusting the margin of the zero-shot models such that the lowest error is achieved on the downstream dataset. This grounding translates into strong empirical performance on real-world datasets, covering few-shot, many-shot, and long-tail learning (Section 5).
The contributions and novelties of this work are summarized as follows:
- We point out the overlooked label bias in foundation models, which originates from the skewness of pre-training distribution that affects the performance of downstream tasks.
- We formalize the estimation of the label bias as a constrained optimization problem (Section 3.2) with theoretical justification (Section 4.3). The entire process does not require access to the pre-training dataset, making it practical for fine-tuning scenarios. We present Generalized Logit Adjustment (GLA) method, which ensembles the debiased zero-shot and fine-tuned models, and demonstrate its superiority over conventional fine-tuning and ensembling by proving it's a Bayes optimal classifier (Section 4.2).
- We build a comprehensive benchmark for evaluation, which considers three real-world settings and three fine-tuning paradigms. The settings are: 1) many-shot learning with abundant data 2) few-shot learning; and 3) long-tail classification, representing a more challenging scenario that combines many-shot and few-shot data (Section 5). The three fine-tuning paradigms include: 1) end-to-end fine-tuning; 2) linear probing; and 3) prompt tuning (Section 3.1).
- We demonstrate the efficacy of our proposed method GLA by conducting extensive experiments across various settings and fine-tuning paradigms. We observe 1 to  $1.5\mathrm{pp}$  accuracy gains on ImageNet, large averaged improvement (1.4 to  $4.6~\mathrm{pp}$ ) on 11 few-shot datasets and  $2.4\mathrm{pp}$  averaged accuracy gains on long-tail datasets. (Section 5).
Image-text foundation models. Foundation models pre-trained by contrastive objectives have set impressive milestones for image and text representation learning, with CLIP [40], ALIGN [23], CoCa [52] and Flamingo [1] being the exemplars. Such models exhibit impressive prompt-based zero-shot performance on various image recognition downstream tasks. Our method aims to reduce foundation model biases to boost performance in downstream tasks. While [2] also addresses word frequency bias, we differ in two key areas: Firstly, we debias zero-shot models using fixed prompts, whereas [2] refines the prompting process. Secondly, our GLA doesn't require access to a subset of the pre-training data.
Ensembles. Ensemble methods aim to boost performances by combining multiple networks, which can be either implemented by aggregating model outputs [12, 4, 28, 14, 27, 51], weight-space ensembling [48, 22], or ensemble distillation [19, 29]. For the adaptation of foundation models, several work propose to ensemble the fine-tuned and zero-shot models for better performance: Wortsman et al. [48] ensembles them in weight space; ProGrad and ProReg [55, 56] propose to fuse them via knowledge distillation. Our GLA is orthogonal to these approaches, as it concentrates on mitigating the biases in foundation models that are detrimental to ensemble models.
Logit adjustment. Logit adjustment [35, 45, 24, 49, 20] is a post-hoc technique to adjust the biased output of classification networks. Kang et al [24] proposes an element-wise scaling adjustment for the classifier weight. Tang et al [45] removes the projection of features on a global biased direction. Menon [35] derives the theoretically optimal adjustment from the training distribution. Unlike the those approaches which rely on a transparent training data or class distribution, our GLA can eliminate the class bias without accessing to the pre-training statistics.
Task. Consider a classification problem with instances  $\mathbf{x} \in \mathcal{X}$  and labels  $y \in \mathcal{Y} = [K] = \{1, \dots, K\}$ . We have a zero-shot model  $f_{\mathrm{zs}}$  (given below), a downstream dataset  $\mathcal{D}_s = \{\mathbf{x}_i, y_i\}_{i=1}^{N_s}$  drawn from source distribution  $P_s$  and a fine-tuned model  $f_{\mathrm{ft}}$  (given below) trained on the dataset  $\mathcal{D}_s$ . Give a model  $f: \mathcal{X} \to \mathbb{R}^K$  that outputs prediction score, we define the risk of  $f$  on the target distribution  $P_t$  as the mis-classification rate:  $\mathcal{R}_t(f) = \mathbb{E}_{\mathbf{x}, y \sim P_t}[y \neq \operatorname{argmax}_i f(\mathbf{x})_i]$ . Our goal is to learn a model  $f_{\mathrm{gla}}$  that best leverages  $f_{\mathrm{zs}}$  and  $f_{\mathrm{ft}}$  that minimizes the risk  $\mathcal{R}_t$ .
Zero-shot models. We primarily explore CLIP [40] for zero-shot models. CLIP consists of a visual encoder  $\Phi_{\mathrm{v}}(\mathbf{x})$  and a text encoder  $\Phi_{\mathrm{t}}(\mathbf{t})$ , producing  $l_{2}$ -normalized features from an image  $\mathbf{x}$  and a text  $\mathbf{t}$  respectively. Zero-shot model  $f_{\mathrm{zs}}$  for  $K$  classes is enabled by matching image features  $\mathbf{v} = \Phi_{\mathrm{v}}(\mathbf{x})$  with classification weights  $\mathbf{w}_k = \Phi_{\mathrm{t}}(\mathbf{t}_k)$ , where  $\mathbf{t}_k$  is obtained by extending the class name  $\{c_k\}$  to a pre-defined prompt, e.g., "a photo of a  $\{c_k\}$ ". Additional details are provided in Appendix B.2. The probability of  $\mathbf{x}$  being classified as  $y$  is defined as:
$$
P (y | \mathbf {x}) = \operatorname {s o f t m a x} \left(f _ {\mathbf {z s}} (\mathbf {x})\right) _ {y} = \frac {\exp \left(\mathbf {v} ^ {T} \mathbf {w} _ {y}\right)}{\sum_ {k = 1} ^ {K} \exp \left(\mathbf {v} ^ {T} \mathbf {w} _ {k}\right)}. \tag {1}
$$
Fine-tuned models. Standard fine-tuning initializes the model  $f_{\mathrm{ft}}$  with the pre-trained parameters and then solve  $f_{\mathrm{ft}} = \operatorname{argmin}_f \mathcal{R}_s(f)$  to minimize the risk on downstream dataset. We consider three common variants of fine-tuning: (1) end-to-end, where all parameters of  $\Phi_{\mathrm{v}}$  and  $\mathbf{w}_k$  are updated; (2) linear probing, where only  $\mathbf{w}_k$  is modified while  $\Phi_{\mathrm{v}}$  is fixed; (3) prompt tuning, where the text input  $\mathbf{t}_k$  is learned, while keeping  $\Phi_{\mathrm{v}}$  and  $\Phi_{\mathrm{t}}$  freezed. See Appendix B.3 for details on fine-tuning methods.
Notation. Let  $P_{p}(y)$ ,  $P_{s}(y)$  and  $P_{t}(y)$  be the marginal probability of class  $y \in [K]$  for pre-training, source (training) and target (test) distribution, respectively. Let  $\pi_{p}$  and  $\pi_{s}$  denote the log probabilities of class for pre-training and training distribution, i.e.,  $\pi_{p}(y) = \log P_{p}(y)$  and  $\pi_{s}(y) = \log P_{s}(y)$ .
3.2 Generalized Logit Adjustment Framework
Fine-tuned models often yield significant gains compared to zero-shot models, and assembling them can further improve performance. This leads to a natural question: How should we best leverage
(a) Averaged outputs of zero-shot model
(b) Estimated label distribution  $\mathbf{q}$
Figure 2: Illustration of debiasing process on ImageNet validation set. (a) The original distribution of zero-shot outputs; (b) the estimated pre-train distribution  $\mathbf{q}$  based on our algorithm; (c) the distribution of debiased zero-shot outputs using estimated  $\mathbf{q}$ .
(c) Averaged outputs of debiased zero-shot model
the zero-shot and fine-tuned models for the prediction tasks? We attempt to answer by proposing generalized logit adjustment in Definition 1.
Definition 1. (GLA) The Generalized Logit Adjustment (GLA) model  $f_{\mathrm{gla}}$  is defined as follows:
$$
f _ {\mathrm {g l a}} (\mathbf {x}) = f _ {\mathrm {f t}} (\mathbf {x}) + f _ {\mathrm {z s}} (\mathbf {x}) - \pi_ {s} - \pi_ {p}. \tag {2}
$$
In Section 4.2, we prove that given the zero-shot and fine-tuned models, our GLA model is the Bayes optimal classifier and no other combination of the two models can outperform it. Here remains one important question: how could we obtain  $\pi_p$  as we have no access to the pre-training statistics? We provide the estimation process of  $\pi_p$  in Eq (4) and postpone the justification in Section 4.3. The entire GLA algorithm consists of two steps which is given as follows:
Step 1: Estimation of  $\pi_p$ . Let  $\mathbf{q}$  be an arbitrary probability simplex over  $K$  classes. Given the validation data from  $P_t$  or the balanced training data, we can estimate the  $\hat{\pi}_p = \log \mathbf{q}^*$  as the constrained optimization problem (proof in Section 4.3):
$$
\mathbf {q} ^ {*} = \underset {\mathbf {q}} {\operatorname {a r g m i n}} \mathcal {R} _ {t} (f _ {\mathbf {z s}} - \log \mathbf {q})
$$
$$
\begin{array}{l} \text {s . t .} \mathbf {q} _ {i} \geq 0, \text {f o r} i \in [ K ], \end{array}
$$
$$
\sum_ {i \in [ K ]} \mathbf {q} _ {i} = 1. \tag {3}
$$
We constrain the sum of  $\mathbf{q}$  to be 1 and ensure that each element is non-negative, guaranteeing that it forms a valid probability distribution. We solve the following Lagrangian problem to find optimal  $\mathbf{q}^*$ :
$$
\min  _ {\mathbf {q}} \max  _ {\lambda_ {i} \geq 0, v} \mathcal {R} _ {t} \left(f _ {\mathbf {z s}} - \log \mathbf {q}\right) - \sum_ {i} \lambda_ {i} \mathbf {q} _ {i} + v \left(1 - \sum_ {i \in [ K ]} \mathbf {q} _ {i}\right) \tag {4}
$$
Step 2: GLA ensembling. Given the estimated  $\hat{\pi}_p$  and the known downstream training  $\pi_s$ , we ensemble the zero-shot model  $f_{\mathrm{zs}}$  and the fine-tuned model  $f_{\mathrm{ft}}$  to get our GLA model  $f_{\mathrm{gla}}$  via Eq. (2). We can regard  $f_{\mathrm{zs}} - \hat{\pi}_p$  and  $f_{\mathrm{ft}} - \pi_s$  as the debiased zero-shot model (Figure 2(c)) and debiased fine-tuned models, respectively. Our GLA is actually ensembling two debiased models. Note that, different from [55, 48], we do not require a hyper-parameter to adjust the contribution of the two models, the optimal solution is to combine them equally (see Section 4.2 for justification).
In this section, we explain why our GLA model best ensembles the zero-shot and fine-tuned models (Section 4.2) and justify the estimation process of the pre-training label distribution  $\pi_p$  (Section 4.3). We start with some preliminaries on Bayes optimal classifier.
Suppose we have a pair  $(X,Y)\sim P$  takes values in  $\mathcal{X}\times \mathcal{Y}$ , where  $Y$  is the class label of input  $X$ .
Definition 2. The 0-1 error (risk) of a classifier  $\hat{y}:\mathcal{X}\to \mathcal{Y}$  on distribution  $P$  is given by:
$$
\mathcal {R} (\hat {y}) = P (Y \neq \hat {y} (X)) \tag {5}
$$
However, the 0-1 error is non-smooth, one typically minimizes a surrogate loss  $\ell$ , e.g., cross-entropy:  $\ell(f(\mathbf{x}), y) = \log \left[ \sum_{i \in [K]} \exp(f(\mathbf{x})_i - f(\mathbf{x})_y) \right]$ , where  $\hat{y}(\mathbf{x}) = \operatorname{argmax}_i f(\mathbf{x})_i$ . It is known that the cross-entropy loss is Bayes consistent [53], i.e., a nearly optimal minimizer of the cross-entropy loss  $(\mathbb{E}_{\mathbf{x}, y \sim P}[\ell(f(\mathbf{x}), y)])$  is also a nearly optimal optimizer of the mis-classification error  $(\mathbb{E}_{\mathbf{x}, y \sim P}[y \neq \hat{y}(\mathbf{x})])$ .
Definition 3. The Bayes optimal classifier  $y^{*}$  for  $P$  given input  $\mathbf{x}$  is defined as:
$$
y ^ {*} (\mathbf {x}) = \underset {y \in \mathcal {Y}} {\operatorname {a r g m a x}} P (y | \mathbf {x}) \tag {6}
$$
It is called Bayes optimal classifier because on the average no other classifier using the same hypothesis and prior knowledge can outperform it.
Lemma 1. The Bayes optimal classifier  $y^{*}$  for  $P$  has lower risk than all classifiers  $\hat{y} : \mathcal{X} \to \mathcal{Y}$ .
$$
\mathcal {R} \left(y ^ {*}\right) \leq \mathcal {R} (\hat {y}) \tag {7}
$$
4.2 Generalized Logit Adjustment Leads to Better Ensembling
Zero-shot and fine-tuned models are complementary. We revisit an empirical phenomena observed in [48] Section 5.1: After exploring a series of measures of diversity, covering predictions and features, they find that zero-shot and fine-tuned models have diverse predictions, despite sharing the same backbone. As the two data distribution  $P_{s}$  and  $P_{p}$  is known to be different, the resulting models leverage different cues to predict: fine-tuned models risk exploiting spurious correlations and indomain patterns which only hold for downstream dataset [46, 3]; On the other hand, zero-shot CLIP models capture stable correlations across diverse domains and exhibit much higher robustness [40]. For instance, zero-shot models rely on robust features for decisions that can achieve high performance on sketch and adversarial samples, while the fine-tuned models that trained on real images typically fail on these samples, as they rely on spurious correlations that only hold on real images. We formulate the phenomena in the following assumption.
Assumption 1. Zero-shot and fine-tuned models have diverse predictions:
$$
\left(f _ {\mathrm {f t}} (\mathbf {x}) \perp f _ {\mathrm {z s}} (\mathbf {x})\right) | y. \tag {8}
$$
We derive the conditional probability  $P_{t}(y|f_{\mathrm{ft}}(\mathbf{x}), f_{\mathrm{zs}}(\mathbf{x}))$  w.r.t. the outputs of  $f_{\mathrm{zs}}(\mathbf{x})$  and  $f_{\mathrm{ft}}(\mathbf{x})$ :
Lemma 2. For a balanced target distribution<sup>1</sup>, where  $P_{t}(y) = 1 / K$  for all  $y \in [K]$ , we have:
$$
P _ {t} (y | f _ {\mathrm {f t}} (\mathbf {x}), f _ {\mathrm {z s}} (\mathbf {x})) = \operatorname {s o f t m a x} \left(\underbrace {f _ {\mathrm {f t}} (\mathbf {x}) + f _ {\mathrm {z s}} (\mathbf {x}) - \pi_ {s} - \pi_ {p}} _ {f _ {\mathrm {g l a}} (\mathbf {x})}\right) (y) \tag {9}
$$
Intuitively, since the zero-shot and fine-tuned models provide diverse predictions, conditioned on the two predictions is equivalent to adding the logits in log space. Additionally, as the target distribution is class-balanced, we need to remove the class bias of two models by subtracting  $\pi_s$  and  $\pi_p$ . The formal proof is given in Appendix A.1. Note that the RHS of Eq. (9) is exactly the softmax output of our GLA model by Definition 1, which exhibits the following property:
Proposition 1. Let  $g: \mathbb{R}^K \times \mathbb{R}^K \to \mathbb{R}^K$  be an arbitrary function that ensemble the outputs of  $f_{\mathrm{zs}}$  and  $f_{\mathrm{ft}}$ . Our GLA classifier  $f_{\mathrm{gla}}$  has lower risk than any function  $f_g(\mathbf{x}) = g(f_{\mathrm{zs}}(\mathbf{x}), f_{\mathrm{ft}}(\mathbf{x}))$ , i.e.
$$
\mathcal {R} _ {t} \left(f _ {\mathrm {g l a}}\right) \leq \mathcal {R} _ {t} \left(f _ {g}\right). \tag {10}
$$
Proof. From Lemma 2 and Definition 1, we have:
$$
\underset {y \in \mathcal {Y}} {\operatorname {a r g m a x}} f _ {\mathrm {g l a}} (\mathbf {x}) _ {y} = \underset {y \in \mathcal {Y}} {\operatorname {a r g m a x}} \operatorname {s o f t m a x} \left(f _ {\mathrm {f t}} (\mathbf {x}) + f _ {\mathrm {z s}} (\mathbf {x}) - \pi_ {s} - \pi_ {p}\right) _ {y} = \underset {y \in \mathcal {Y}} {\operatorname {a r g m a x}} P _ {t} (y | f _ {\mathrm {f t}} (\mathbf {x}), f _ {\mathrm {z s}} (\mathbf {x})), \tag {11}
$$
which means  $f_{\mathrm{gla}}$  is the Bayes optimal classifier (see Definition 3) given  $f_{\mathrm{ft}}(\mathbf{x})$  and  $f_{\mathrm{zs}}(\mathbf{x})$ . According to Lemma 1, any other classifier  $g(f_{\mathrm{ft}}(\mathbf{x}),f_{\mathrm{zs}}(\mathbf{x}))$  must have higher risk, i.e.,  $\mathcal{R}_t(f_{\mathrm{gla}})\leq \mathcal{R}_t(f_g)$ .
This lemma can be easily extended to imbalanced target distributions (proof in Appendix A). Yet, as most test sets are class-balanced, we focus on the balanced case for brevity.
Proposition 1 demonstrates that our  $f_{\mathrm{gla}}$  model is the best model, as it has the lowest risk on target distribution. Proposition 1 further explains the superiority of  $f_{\mathrm{gla}}$  over the fine-tuned model  $f_{\mathrm{ft}}$  and the naive ensemble  $f_{\mathrm{ens}}(\mathbf{x}) = f_{\mathrm{ft}}(\mathbf{x}) + f_{\mathrm{zs}}(\mathbf{x})$ :
Corollary 1.  $f_{\mathrm{gla}}$  performs better than fine-tuned model  $f_{\mathrm{ft}}$  and naive emsembling  $f_{\mathrm{ens}}$ :
$$
\mathcal {R} _ {t} \left(f _ {\mathrm {g l a}}\right) \leq \mathcal {R} _ {t} \left(f _ {\mathrm {f t}}\right), \mathcal {R} _ {t} \left(f _ {\mathrm {g l a}}\right) \leq \mathcal {R} _ {t} \left(f _ {\mathrm {e n s}}\right) \tag {12}
$$
Discussion: when do the GLA models degenerate? Note that there are two equality signs in Eq. (12), indicating that the performance of the GLA model can degenerate to be equivalent to that of the fine-tuned model and naive ensembling in the following two cases.
Case 1: For the first equality, if zero-shot model  $f_{\mathrm{zs}}(\mathbf{x})$  provides no further information about  $y$  given  $f_{\mathrm{ft}}(\mathbf{x})$ , i.e.,  $(y \perp f_{\mathrm{zs}}(\mathbf{x}))|f_{\mathrm{ft}}(\mathbf{x})$ , then  $P_t(y|f_{\mathrm{ft}}(\mathbf{x}), f_{\mathrm{zs}}(\mathbf{x}))$  degenerates to  $P_t(y|f_{\mathrm{ft}}(\mathbf{x}))$  and the first equality applies. However, in practice, as downstream model and zero-shot model provides diverse predictions, we usually encounter strict inequality, i.e.,  $\mathcal{R}_t(f_{\mathrm{gla}}) < \mathcal{R}(f_{\mathrm{ft}})$ .
Case 2: The second equality applies when pre-training and downstream training distribution are both class-balanced. In fact, the pre-training dataset for foundation models are known to be highly skewed. Therefore, in most cases, we have  $\mathcal{R}_t(f_{\mathrm{gla}}) < \mathcal{R}_t(f_{\mathrm{ens}})$ .
In summary, the above two equalities are usually unattainable, which means that theoretically, our GLA model performs better than both the fine-tuned and the naive ensemble models.
4.3 Estimate the label bias of the pre-training dataset
However,  $\pi_p$  is usually unknown as we have no access to pre-training dataset. In this work, we seek to estimate  $\pi_p$  using the zero-shot models and the downstream data. Similar to Proposition 1, we have the following proposition says that  $f_{\mathrm{zs}} - \pi_p$  has lower error on target distribution than any other classifiers that use  $f_{\mathrm{zs}}$ , see Appendix A.2 for the full proof.
Proposition 2. Let  $h: \mathbb{R}^K \to \mathbb{R}^K$  be an arbitrary function that predicts labels using the outputs of the zero-shot model  $f_{\mathbf{zs}}(\mathbf{x})$ . Let the derived classifier be denoted as  $f_h(\mathbf{x}) = h(f_{\mathbf{zs}}(\mathbf{x}))$ . The classifier  $f_{\mathbf{zs}} - \pi_p$  is better than any  $f_h(\mathbf{x})$ :  $\mathcal{R}_t(f_{\mathbf{zs}} - \pi_p) \leq \mathcal{R}_t(f_h(\mathbf{x}))$ .