SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $\mathbf{q}$  be an arbitrary probability simplex over  $K$  classes, then we have  $\mathcal{R}_t(f_{\mathrm{zs}}(\mathbf{x}) - \pi_p)\leq \mathcal{R}_t(f_{\mathrm{zs}}(x) - \log \mathbf{q})$ . Therefore, we choose to optimize a probability simplex  $\mathbf{q}$  over  $K$  classes such that the model  $f_{\mathrm{zs}} - \log \mathbf{q}$  achieves the minimal empirical risk, as formulated in Eq. (3) (the Step 1 of GLA algorithm). Once we obtain the estimated class prior  $\hat{\pi}_p = \log \mathbf{q}$ , we can easily implement the GLA model by ensembling  $f_{\mathrm{gla}}(\mathbf{x}) = f_{\mathrm{ft}}(\mathbf{x}) + f_{\mathrm{zs}}(\mathbf{x}) - \pi_s - \hat{\pi}_p$  (the Step 2 of GLA algorithm).
Toy experiment. We conducted an experiment to show that the estimated label distribution closely approximates the true one. Specifically, we trained a model with a ResNet32 backbone on the imbalanced CIFAR-10-LT [10] dataset with an imbalanced ratio of 10. Subsequently, we used only the test set combined with our proposed method to estimate the label distribution. This procedure simulates scenarios where only downstream data is available and the pre-training data is inaccessible.
Figure 3 reveals a strong alignment between the estimated (orange line) and the actual distributions (blue line), which is further emphasized by a small KL-divergence value of 0.00062. The toy experiment validates the effectiveness of our debiasing method.
Figure 3: Estimating label bias of CIFAR-10-LT-IB-10.
Discussion. The log q we estimated is not the marginal log-probability of the entire pre-training distribution but the label bias matches the downstream distribution. In the above toy experiment, although training and test sets show different label distributions, their conditional distribution  $P(\mathbf{x} | y)$  remains invariant. In this case, our estimate will converge to the actual training label bias. For CLIP models, with diverse pre-training data, some might not align with the downstream domain, potentially compromising the accuracy of the estimation of the entire pre-training distribution.
However, we'd like to point out that removing the label bias of entire pre-training distribution may not optimal for downstream tasks. As a thought experiment, consider a pre-training dataset "sketch"
Method	B/32	B/16
Zero-shot	63.2	68.3
LP	75.8	79.9
E2E	76.3	81.3
WiSE-FT	76.9	81.7
GLA	77.9	82.8
(a) ImageNet
Method	B/32	B/16
Zero-shot	64.2	67.2
LP	80.5	83.1
E2E	89.4	91.0
WiSE-FT	90.0	91.3
GLA	90.7	91.9
(b) CIFAR100
Method	B/32	B/16
Zero-shot	59.7	64.4
LP	77.1	83.1
E2E	83.7	90.2
WiSE-FT	84.5	90.7
GLA	85.0	91.1
(c) Stanford Cars
Method	B/32	B/16
Zero-shot	62.2	64.8
LP	75.6	78.0
E2E	80.0	82.4
WiSE-FT	80.8	82.9
GLA	81.2	83.4
(d) SUN397
Table 1: Accuracy of various methods using CLIP ViT-B/32 and ViT-B/16. LP: linear probe; E2E: end-to-end fine-tuning. Results were obtained using the official implementation from WiSE-FT [48].
and "photo" styles for "dog" and "cat" samples. Suppose the sample size of "dog" and "cat" is equal but there are more "sketch dogs" than "sketch cats". This means that even if the overall distribution is balanced, each style isn't, resulting in biased zero-shot predictions. If we aim to deploy models for the "sketch dogs and cats" domain, adjusting the overall label bias is insufficient. Instead, the optimal label bias should be estimated on the "sketch" distribution. We also provide experiments using LAION-400M dataset in Appendix C.2, illustrating the situation when the downstream data diverges from the pre-training set.
We evaluate our GLA on three real-world scenarios: many-shot (Section 5.1), few-shot (Section 5.2) and long-tail learning (Section 5.3). We show that our GLA boosts performance on all three settings.
5.1 Many-shot learning
Datasets. We use ImageNet [11] and CIFAR100 [26] for generic object classification, StanfordCars [25] for fine-grained classification, and SUN397 [50] for scene recognition. See Appendix B.1 for details.
Baselines. We compare GLA against four methods: (1) Zero-shot model, (2) Linear Probing (LP), (3) End-to-End fine-tuning (E2E), and (4) weight ensembling method WiSE-FT[48].
Implementation details. We consider two models: CLIP ViT-B/32 and ViT-B/16. For learning-based models, we fine-tune with AdamW using a cosine annealing learning rate scheduler. We fine-tune for 10 epochs on ImageNet and 20 epochs on other datasets. See Appendix B.3 for further details.
Main results. Table 1 compares our GLA with various baselines. We observe that our GLA can increase the performance of end-to-end fine-tuned models: it achieves  $1.5\%$  gains on ImageNet. Compared to WiSE-FT, GLA gains  $1.1\%$  top-1 accuracy boost on ImageNet dataset.. Beyond generic object recognition, our method also improves accuracy on the fine-grained dataset (Stanford Cars) and the scene recognition dataset (SUN397), by  $0.4\%$  and  $0.5\%$ , respectively.
Breakdown performance analysis. To analyze the impact of pre-training label bias on fine-tuned and ensemble models, we present the breakdown results on ImageNet using CLIP ViT-B/16, as shown in Table 2. Specifically, we sort the class index using the estimated  $\pi_p$ , and assign the top third of the classes as the head classes, the last third as the tail classes, and the remaining classes as the medium classes. Due to the label bias, the zero-shot tail performance is significantly lower than the head one (57.2% vs. 78.0%). The resulting E2E models are also affected by the bias, with the tail performance being 6.3% lower than the head. Existing ensemble WiSE-FT overlooks the bias, exhibiting noticeable degradation on the tail performances  $(-0.9\%)$  compared to E2E model, while our GLA stands out for all three subgroups.
Estimated  $\pi_p$  is transferable across different zero-shot models. The estimated  $\pi_p$  should be transferable across different zero-shot models if they are trained on the same pre-training dataset. To verify this, we employed a CLIP ViT-B/32 based zero-shot model to estimate  $\pi_p$ , which is subsequently used to debias zero-shot models based on CLIP ViT-B/16 and ViT-L/14. As shown in Table 3, our debiased models outperform the original zero-shot versions by a clear margin.
Method	Head	Med.	Tail	All
Zero-shot	78.0	69.8	57.2	68.3
E2E	83.6	83.0	77.3	81.3
WiSE-FT	85.3	83.7	76.4	81.7
GLA	85.2	84.3	78.8	82.8
Table 2: Breakdown results on ImageNet.
Model	Source	Target
ViT-B/32	ViT-B/16	ViT-L/14
fzs(x)	63.4	68.8	75.6
fzs(x) - πp	65.4	69.3	76.3
Table 3: Estimated  $\hat{\pi}_p$  is transferable across different backbones.  $\hat{\pi}_p$  is estimated using CLIP ViT-B/32.
Figure 5: Accuracy  $(\%)$  of few-shot learning on 11 datasets.
Ensembling with mixing coefficient. In Section 5.1, we prove the optimal solution is to combine the debiased zero-shot and fine-tuned models equally. We now examine the claim by introducing a mixture coefficient  $\alpha \in [0,1]$ . The ensemble predictions are given by:  $f_{\mathrm{gla}}(\mathbf{x},\alpha) = (1 - \alpha)\cdot (f_{\mathrm{zs}}(\mathbf{x}) - \pi_p) + \alpha \cdot (f_{\mathrm{ft}}(\mathbf{x}) - \pi_s)$ . We compare the GLA and the naive ensembling with mixture  $\alpha$  in Figure 4, where GLA meets its optimal performance at  $\alpha = 0.5$ , which is in line with our theoretical analysis. We also observe that the debiased zero-shot model increases accuracy by  $2.3\%$  and our GLA consistently outperforms naive ensembling with various  $\alpha$ .
Figure 4: Accuracy with mixing coefficient  $\alpha$ .
5.2 Few-shot learning
For few-shot scenarios, we primarily choose prompt tuning for fine-tuning, since it is empirically more effective than end-to-end fine-tuning and linear probing [54, 55, 8].
Datasets. We follow CoOp[54] to use 15 datasets: ImageNet [11], Caltech101 [13], OxfordPets [37], StanfordCars [25], Flowers102 [36], Food101 [6], FGVCAircraft [34], EuroSAT [16], UCF101 [44],
Method	Source	Target
IN	V2	S	A	R
CLIP	59.8	52.8	35.5	22.8	60.6
CoOp	61.9	54.3	32.5	21.8	54.2
PLOT	63.0	55.1	33.0	21.9	55.6
ProGrad	63.5	55.4	33.1	21.3	55.2
GLA	65.6	57.1	36.4	23.0	62.1
Table 4: Evaluation on robustness to distribution shift at 16 training shots.
Figure 6: Comparison with naive ensembling.
Model	IN.	Caltech.	Pets.	Cars.	Flowers.	Food.	Aircraft.	SUN.	DTD	EuroSAT	UCF.	Avg.
fzs	59.8	87.1	85.8	55.6	65.3	77.9	17.1	60.3	42.3	37.5	61.5	59.1
fzs-πp	62.3	89.9	85.9	57.6	67.2	78.6	18.4	63.5	43.0	39.0	62.4	60.7
Δ	+2.5	+2.7	+0.1	+2.0	+1.9	+0.7	+1.3	+3.2	+0.7	+1.4	+0.9	+1.6
Table 5: Comparison between zero-shot models  $\left( {f}_{\mathrm{{zs}}}\right)$  and debiased zero-shot models  $\left( {{f}_{\mathrm{{zs}}} - {\widehat{\pi }}_{p}}\right)$  .
DTD [9], SUN397 [50], ImageNet-{V2 [41], Sketch [47], A [18], R [17]}. We randomly select  $\{1, 2, 4, 8, 16\}$  shots for training and use the original test set for evaluation. See Appendix B.1 for details.
Baselines. We compare with three prompt tuning methods: (1) CoOp [54] optimizes prompts via empirical risk minimization; (2) ProGrad [55] prevents forgetting the general knowledge using zero-shot predictions; (3) PLOT [8] applies optimal transport to match the vision and text modalities.
Implementation details. We implement our proposed GLA method using CLIP-ResNet-50 as the foundation model and adopt class-specific prompt tuning from CoOp. Results are averaged over three seeds, with training configurations aligned with CoOp. See Appendix B.4 for further information.
Main results. Figure 5 summarizes the few-shot results on 11 datasets. The detailed accuracy and standard deviation are in Appendix C.1. Overall, our GLA clearly outperforms the baselines on average performance by a large margin, e.g., our GLA gains  $4.6\%$ ,  $4.8\%$ ,  $3.1\%$ ,  $2.6\%$  and  $1.6\%$  performance boost over CoOp at 1, 2, 4, 8, 16 shots. In particular, on ImageNet dataset, we observe a large improvement, e.g.,  $3.9\%$ ,  $2.9\%$ ,  $1.9\%$ ,  $2.0\%$  and  $2.2\%$  over ProGrad at 1, 2, 4, 8, 16 shots. Furthermore, on OxfordPerts, Food101 and SUN397 datasets, our method's performance remains stable and consistently improves with increasing sample sizes, while the one of the baseline methods fluctuates significantly. In particular, on Food101, baseline models even underperform zero-shot models by a large margin, while our GLA shows clearly better performance than zero-shot models.
Robustness to distribution shifts. Following CoOp, we used the ImageNet at 16 training shots as the source domain and assess the robustness on ImageNet- $\{\mathrm{V2}, \mathrm{Sketch}, \mathrm{A}, \mathrm{R}\}$  datasets. Table 4 summarizes the results, where the prompt tuning baselines perform worse on distribution shifted datasets compared to the zero-shot model, as fine-tuning on limited data misleads the model to learn in-distribution correlations. In comparison, our GLA approach makes the best of both fine-tuned and zero-shot models, thus consistently outperforms other methods on both source and target domains.
GLA improves accuracy over naive ensemble. Figure 6 compares the results between our GLA and naive ensembling. We rank the absolute improvements over fine-tuning baseline at 16 training shots. In summary, our GLA demonstrates superior accuracy gains. It is worth noting that the naive ensembling does not always lead to improvements, e.g., on EuroSAT, SUN, Caltech, UCF and Aircraft, naive ensembling even underperforms fine-tuning baseline.
Debiased zero-shot models perform better. We estimate  $\pi_p$  at 16 shots and compare the original zero-shot models with the debiased zero-shot models in Table 5. It is clear that the debiasing leads to improvement on all 11 datasets, showcasing an average accuracy gain of  $1.6\%$ .
5.3 Long-tail learning
Datasets and metrics. We evaluate our method on two standard benchmarks: Places365-LT and ImageNet-LT [31]. In addition to top-1 accuracy, we report the accuracy on three test subsets
Method	Many	Med	Few	All
LP	ERM	64.4	39.8	20.1	46.6
LA	56.7	50.5	46.4	52.3
BS	57.7	50.5	43.9	52.4
GLA	65.5	60.8	57.7	62.2
PT	ERM	66.6	57.7	53.8	60.6
LA	67.9	60.0	58.7	62.9
BS	67.0	62.2	59.4	63.5
GLA	70.7	66.5	64.3	67.8
E2E	ERM	75.5	56.0	36.2	60.8
LWS [24]	70.4	62.6	56.2	64.7
LA [35]	70.4	62.8	56.9	64.9
BS [43]	71.2	62.5	56.8	65.1
WiSE-FT [48]	70.7	65.0	61.1	66.7
BALLAD [33]	71.0	66.3	59.5	67.2
GLA	72.1	66.4	63.4	68.2
(a) ImageNet-LT
Method	Many	Med	Few	All
LP	ERM	43.9	21.1	9.0	26.9
LA	39.2	33.7	29.5	34.9
BS	39.6	34.0	28.3	34.9
GLA	43.6	40.2	38.8	41.1
PT	ERM	47.7	32.6	24.9	36.5
LA	43.9	40.6	39.4	41.5
BS	43.4	42.5	42.3	42.8
GLA	47.0	47.1	47.8	47.2
E2E	ERM	46.3	27.4	12.5	31.3
LWS [24]	41.2	39.0	32.8	38.6
LA [35]	42.2	39.3	34.0	39.3
BS [43]	46.7	44.4	39.5	44.3
WiSE-FT [48]	46.0	44.0	43.9	44.7
BALLAD [33]	48.7	44.7	42.2	45.7
GLA	47.3	45.4	45.3	46.1
(b)Places365-LT
Table 6: The performances on ImageNet-LT and Places365-LT.
according to the number of samples per class: many-shot ( $>100$  samples), medium-shot ( $20\sim 100$  samples), and few-shot ( $<20$  samples). Detailed information is provided in Appendix B.1.
Fine-tuning and long-tail learning baselines. We compare our GLA with the combinations of fine-tuning protocols and long-tailed recognition methods. We consider three fine-tuning protocols: 1) Linear Probe (LP) 2) End-to-End (E2E) fine-tuning; 3) Prompt Tuning (PT). The three fine-tuning paradigms are introduced in Section 3.1 with details in Appendix B.3. We compare with 5 long-tail learning methods: 1) standard Empirical Risk Minimization (ERM); 2) Learnable Weight Scaling (LWS) [24]; 3) Logit Adjustment (LA) [35]; 4) Balanced Softmax (BS) [43], and 5) BALLAD [33], which is designed for VLMs. See Appendix B.6 for more details on long-tail baselines.
Implementation Details. For all combinations of the fine-tuning and long-tail learning baselines, visual backbones are initialized from CLIP-ResNet-50 and classifiers are initialized via zero-shot prompting. We use SGD for 50 epochs with batch size of 512. See Appendix B.6 for further details.
Results. Table 6 shows that our GLA method consistently surpasses baselines across all long-tailed datasets. Our approach outperforms PT-based models by  $3.5\%$  and  $4.4\%$  on ImageNet-LT and Places365-LT, respectively. Against E2E approaches, GLA exceeds not just the WiSE-FT but also the current SOTA method BALLAD, by a large margin, e.g., 1pp gains on ImageNet-LT.
In this paper, we identify the label bias in foundation models and underscore its adverse effects on downstream task performance. We propose the Generalized Logit Adjustment (GLA) framework for fine-tuning foundation models, which boosts the performance by effectively eliminating label bias and combining diverse predictions from zero-shot and fine-tuned models. We prove that when presented with zero-shot and fine-tuned models, our GLA is the Bayes optimal classifier for downstream task. Extensive experiments across a diverse range of tasks and fine-tuning framework demonstrate the effectiveness of our approach. We believe that the proposed GLA may partially improve the fairness and credibility of foundation models.
The first limitation is that we only focus on the label bias while other forms of model biases, e.g., representation bias [5], cannot be addressed by our algorithm yet. The second limitation is that we primarily focus on enhancing the fine-tuning performance for discriminative models. However, applying our GLA framework to generative models presents challenges. For instance, language generation operates as a Markov process, meaning each output depends on previous ones. This implies it's not straightforward to estimate the biasedness of a sequence with our GLA, as we only compute the bias in a pre-defined and independent label space.
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-01-002 and AI Singapore AISG2-RP-2021-022).
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022.
[2] James Urquhart Allingham, Jie Ren, Michael W Dusenberry, Xiuye Gu, Yin Cui, Dustin Tran, Jeremiah Zhe Liu, and Balaji Lakshminarayanan. A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models. In ICML, 2023.
[3] Martin Arjovsky. Out of distribution generalization in machine learning. PhD thesis, New York University, 2020.
[4] Eric Bauer and Ron Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine learning, 1999.
[5] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. Technical Report, 2021.
[6] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In ECCV, 2014.
[7] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
[8] Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Prompt learning with optimal transport for vision-language models. ICLR, 2023.
[9] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[10] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In CVPR, 2019.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[12] Thomas G Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems: First International Workshop, 2000.
[13] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPRW, 2004.
[14] Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 1997.
[15] Jose Gallego-Posada and Juan Ramirez. Cooper: a toolkit for Lagrangian-based constrained optimization. https://github.com/cooper-org/cooper, 2022.
[16] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
A.1 Proof of Lemma 2
Restated Lemma (Lemma 2). Let  $\pi_t$  denote the log probability of the target label distribution  $\pi_t(y) = \log P_t(y)$ , we have:
$$
P _ {t} (y \mid f _ {\mathrm {f t}} (\mathbf {x}), f _ {\mathrm {z s}} (\mathbf {x})) = \operatorname {s o f t m a x} \left(f _ {\mathrm {f t}} (\mathbf {x}) + f _ {\mathrm {z s}} (\mathbf {x}) - \pi_ {s} - \pi_ {p} + \pi_ {t}\right) (y). \tag {13}
$$
In class-balanced target distribution, Eq. (13) simplifies to:
$$
P _ {t} (y \mid f _ {\mathrm {f t}} (\mathbf {x}), f _ {\mathrm {z s}} (\mathbf {x})) = \operatorname {s o f t m a x} \left(f _ {\mathrm {f t}} (\mathbf {x}) + f _ {\mathrm {z s}} (\mathbf {x}) - \pi_ {s} - \pi_ {p}\right) (y). \tag {14}
$$
Proof. Denote the output  $\mathbf{e} = f_{\mathrm{ft}}(\mathbf{x})$  and  $\mathbf{z} = f_{\mathrm{zs}}(\mathbf{x})$ . We first use the Bayes Rule to decompose  $P_{t}(y|\mathbf{e},\mathbf{z})$  into  $P_{t}(\mathbf{e},\mathbf{z}|y)$ ,  $P_{t}(y)$  and  $P_{t}(\mathbf{e},\mathbf{z})$  in Eq. (15), then rewrite  $P_{t}(\mathbf{e},\mathbf{z}|y)$  in Eq. (16) according to Assumption 1. Focusing on label shift problem [35, 20, 30] where  $P(\mathbf{x}|y)$  does not change, we derive Eq. (17)
$$
\begin{array}{l} P _ {t} (y | \mathbf {e}, \mathbf {z}) = \frac {P _ {t} (\mathbf {e} , \mathbf {z} | y) P _ {t} (y)}{P _ {t} (\mathbf {e} , \mathbf {z})} (15) \\ = P _ {t} (\mathbf {e} | y) P _ {t} (\mathbf {z} | y) \frac {P _ {t} (y)}{P _ {t} (\mathbf {e} , \mathbf {z})} (16) \\ = P _ {s} (\mathbf {e} | y) P _ {p} (\mathbf {z} | y) \frac {P _ {t} (y)}{P _ {t} (\mathbf {e} , \mathbf {z})} (17) \\ = \frac {P _ {s} (y | \mathbf {e}) P _ {s} (\mathbf {e})}{P _ {s} (y)} \frac {P _ {p} (y | \mathbf {z}) P _ {p} (\mathbf {z})}{P _ {p} (y)} \frac {P _ {t} (y)}{P _ {t} (\mathbf {e} , \mathbf {z})} (18) \\ = \frac {P _ {s} (y | \mathbf {e})}{P _ {s} (y)} \frac {P _ {p} (y | \mathbf {z})}{P _ {p} (y)} \frac {P _ {s} (\mathbf {e}) P _ {p} (\mathbf {z}) P _ {t} (y)}{P _ {t} (\mathbf {e} , \mathbf {z})} (19) \\ \end{array}
$$
Since  $\mathbf{e},\mathbf{z}$  are fixed, we can replace the terms that not rely on  $y$  with a constant  $C_1$  in Eq. (21). We replace  $P_{s}(y) = \exp (\log P_{s}(y)) = \exp (\pi_{s}(y)),P_{p}(y) = \exp (\log P_{p}(y)) = \exp (\pi_{p}(y))$  and  $P_{t}(y) = \exp (\log P_{t}(y)) = \exp (\pi_{t}(y))$ . Suppose the underlying class-probabilities  $P_{s}(y|\mathbf{e})\propto \exp (\mathbf{e}_{y})$  and  $P_{p}(y|\mathbf{z})\propto \exp (\mathbf{z}_{y})$  for  $y\in [K]$ . Denote the constants  $C_s$  and  $C_p$  for normalizing  $\exp (\mathbf{e}_y)$  and  $\exp (\mathbf{z}_y)$  into probabilities, and merge all constants to  $C = \frac{C_1}{C_sC_p}$ , we get Eq. (23)
$$
\begin{array}{l} P _ {t} (y | \mathbf {e}, \mathbf {z}) = \frac {P _ {s} (y | \mathbf {e})}{P _ {s} (y)} \frac {P _ {p} (y | \mathbf {z})}{P _ {p} (y)} P _ {t} (y) C _ {1} (21) \\ = \exp (\mathbf {e} + \mathbf {z} - \pi_ {s} - \pi_ {p} + \pi_ {t}) (y) \frac {C _ {1}}{C _ {s} C _ {p}} (22) \\ = C \cdot \exp (\mathbf {e} + \mathbf {z} - \pi_ {s} - \pi_ {p} + \pi_ {t}) (y) (23) \\ \end{array}
$$
Because the summation of  $P_{t}(y|\mathbf{e},\mathbf{z})$  is 1,  $C = 1 / \sum_{i\in [K]}\exp (\mathbf{e} + \mathbf{z} - \pi_{s} - \pi_{p} + \pi_{t})(i)$ . Therefore, we have:
$$
\begin{array}{l} P _ {t} \left(y \mid f _ {\mathrm {f t}} (\mathbf {x}), f _ {\mathbf {z s}} (\mathbf {x})\right) = P _ {t} \left(y \mid \mathbf {e}, \mathbf {z}\right) (24) \\ = \frac {\exp (\mathbf {e} + \mathbf {z} - \pi_ {s} - \pi_ {p}) _ {y}}{\sum_ {i \in [ K ]} \exp (\mathbf {e} + \mathbf {z} - \pi_ {s} - \pi_ {p} + \pi_ {t}) _ {i}} (25) \\ = \operatorname {s o f t m a x} \left(f _ {\mathrm {f t}} (\mathbf {x}) + f _ {\mathbf {z s}} (\mathbf {x}) - \pi_ {s} - \pi_ {p} + \pi_ {t}\right) _ {y} (26) \\ \end{array}
$$
In class-balanced target distribution case,  $\pi_t = \log \frac{1}{K}$  is constant. Since the softmax function is invariant to constant offsets, Eq. (26) simplifies to:
$$
P _ {t} (y \mid f _ {\mathrm {f t}} (\mathbf {x}), f _ {\mathbf {z s}} (\mathbf {x})) = \operatorname {s o f t m a x} \left(f _ {\mathrm {f t}} (\mathbf {x}) + f _ {\mathbf {z s}} (\mathbf {x}) - \pi_ {s} - \pi_ {p}\right) _ {y} \tag {27}
$$
Dataset	Classes	Train size	Test size	Task
ImageNet	1,000	1.28M	50,000	Object-level
CIFAR100	100	50,000	10,000	Object-level
Caltech101	100	4,128	2,465	Object-level
DTD	47	2,820	1,692	Textures
EuroSAT	10	13,500	8,100	Satellite images
FGVCAircraft	100	3,334	3,333	Fine-grained aircraft
Flowers102	102	4,093	2,463	Fine-grained flowers
Food101	101	50,500	30,300	Fine-grained food
OxfordPets	37	2,944	3,669	Fine-grained pets
StanfordCars	196	6,509	8,041	Fine-grained car
SUN397	397	15,880	19,850	Scene-level
UCF101	101	7,639	3,783	Action
ImageNetV2	1,000	-	10,000	Robustness to collocation
ImageNet-Sketch	1000	-	50,889	Robustness to sketch domain
ImageNet-A	200	-	7,500	Robustness to adversarial attack
ImageNet-R	200	-	30,000	Robustness to multi-domains
Table 7: The detailed statistics of datasets for many-shot and few-shot learning.