SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Restated Proposition (Proposition 2). Suppose that the target distribution  $P_{p}$  is class-balanced. Let  $h: \mathbb{R}^{K} \to \mathbb{R}^{K}$  be an arbitrary function that predicts labels using the outputs of the zero-shot model  $f_{\mathbf{zs}}(\mathbf{x})$ . Let the derived classifier be denoted as  $f_{h}(\mathbf{x}) = h(f_{\mathbf{zs}}(\mathbf{x}))$ . The classifier  $f_{\mathbf{zs}} - \pi_{p}$  is better than any  $f_{h}(\mathbf{x})$ :  $\mathcal{R}_t(f_{\mathbf{zs}} - \pi_p) \leq \mathcal{R}_t(f_h(\mathbf{x}))$ .
Proof. Denote the output  $\mathbf{z} = f_{\mathrm{zs}}(\mathbf{x})$ . Similar to Eq. (15)-Eq. (26), we have
$$
\begin{array}{l} P _ {t} (y | \mathbf {z}) = \frac {P _ {t} (\mathbf {z} | y) P _ {t} (y)}{P _ {t} (\mathbf {z})} (28) \\ = \frac {P _ {p} (\mathbf {z} | y) P _ {t} (y)}{P _ {t} (\mathbf {z})} (29) \\ = \frac {P _ {p} (y | \mathbf {z})}{P _ {p} (y)} \frac {P _ {t} (y)}{P _ {t} (\mathbf {z})} (30) \\ = \exp (\mathbf {z} - \pi_ {p}) (y) / \sum_ {i \in [ K ]} \exp ((\mathbf {z} - \pi_ {p}) (i)) (31) \\ = \operatorname {s o f t m a x} (\mathbf {z} - \pi_ {p}) = \operatorname {s o f t m a x} \left(f _ {\mathbf {z s}} (\mathbf {x}) - \pi_ {p}\right) (32) \\ \end{array}
$$
$$
\underset {y \in \mathcal {Y}} {\operatorname {a r g m a x}} \left(f _ {\mathbf {z s}} (\mathbf {x}) - \pi_ {p}\right) _ {y} = \underset {y \in \mathcal {Y}} {\operatorname {a r g m a x}} \operatorname {s o f t m a x} \left(f _ {\mathbf {z s}} (\mathbf {x}) - \pi_ {p}\right) _ {y} = \underset {y \in \mathcal {Y}} {\operatorname {a r g m a x}} P _ {t} (y | f _ {\mathbf {z s}} (\mathbf {x})) \tag {33}
$$
Again, using Lemma 1, any other classifier  $f_{h}(\mathbf{x})$  has higher risk than  $f_{\mathrm{zs}}(\mathbf{x}) - \pi_p$ , i.e.,  $\mathcal{R}_t(f_{\mathrm{zs}} - \pi_p) \leq \mathcal{R}_t(f_h(\mathbf{x}))$ .
B.1 Dataset details
Many-shot and few-shot datasets. For many-shot learning, we use ImageNet, CIFAR100, StanfordCars and SUN397 datasets. For few-shot learning, we evaluate models on 15 datasets. The details of each dataset are presented in Table 7.
Long-tail datasets. We use two standard long-tail benchmarks: Places365-LT and ImageNet-LT [31]. The skewness of a long-tailed training set is typically represented by imbalanced ratio, which is defined as  $N_{\mathrm{max}} / N_{\mathrm{min}}$ .  $N_{\mathrm{max}}$  ( $N_{\mathrm{min}}$ ) denotes the largest (smallest) number of instances per class. A
Dataset	Size of all classes	Size of many classes	Size of medium classes	Size of few classes	Size of training samples	Imbalanced ratio
Places365-LT	365	131	163	71	62.5K	996
ImageNet-LT	1000	385	479	136	186K	256
Table 8: Details of long-tailed datasets.
larger imbalanced ratio means a more imbalanced training set. The test sets are divided into three splits: many-shot subset contains classes with  $>100$  images, medium-shot subset includes classes with  $\geq 20\& \leq 100$  images, and few-shot subset covers classes with  $< 20$  images. Details are listed in Table 8.
B.2 CLIP zero-shot
We use prompt ensembling of 80 prompts provided by CLIP [48] for ImageNet, CIFAR100, and Caltech101 to improve performance, i.e., averaging the text embedding of many captions, e.g., "a photo of a  $\{c_k\}$ " and "an image of a  $\{c_k\}$ ". For OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, EuroSAT, UCF101, DTD and SUN397, we use the pre-defined prompt from CoOp [54].
B.3 Fine-tuned models
End-to-end and linear probe fine-tuning. We follow WiSE-FT [48] to implement fine-tuning. We initialize the classifier with the zero-shot classifier and the output of the image encoder  $\Phi_{\nu}$  is normalized during fine-tuning. We fine-tune for a total of 10 epochs using AdamW [32] optimizer with default hyper-parameters  $\beta_{1} = 0.9, \beta_{2} = 0.999, \epsilon = 10^{-8}$  and weight decay 0.1. We choose a batch size of 512. We use the same data augmentation and cosine-annealing learning rate schedule as [48].
B.4 Prompt tuning.
Prompt tuning like CoOp [54] automates prompt engineering by learning the prompt given few samples from downstream tasks. CoOp provides two options of prompt design: unified prompt that is shared among all classes and class-specific prompt that is different for each class. In this paper, we adopt the class-specific prompt design as the fine-tuned model to implement GLA. In specific, given the word embedding  $\mathbf{t}_k^0$  initialized by zero-shot prompts, we aim to learn a collection of class-specific word embedding  $\mathbf{R} = \{\mathbf{r}_k\}_{k=1}^K$ , such that the text input  $\mathbf{t}_k = \mathbf{t}_k^0 + \mathbf{r}_k$  minimizes the empirical risk:  $\mathbf{R}^* = \operatorname*{argmin}_{\mathbf{R}} \mathbb{E}_{\mathbf{x},y}[y \neq \operatorname*{argmax}_i f(x; \mathbf{R})_i]$ .
We adhere CoOp to use CLIP ResNet-50 as image encoder for few-shot classification. The word embedding  $\mathbf{R}$  is initialized from zeros. For the  $m$  few-shot classification setting (where  $m \in \{1,2,4,8,16\}$ ), we randomly sample  $m$  training and  $m$  validation points from the respective full datasets. For all few-shot datasets except ImageNet, the training epoch is set to 200 for 16/8 shots, 100 for 4/2 shots, and 50 for 1 shot. For ImageNet, the epoch is set to 50 for all shots. We fine-tune the prompt with SGD optimizer decayed by the cosine annealing rule. The base initial learning rate and batch size are set to  $10^{-4}$  and 32. When given an  $m$ -shot sample setting, we increase the learning rate and batch size by  $m$  times simultaneously to accelerate the training speed.
B.5 Estimation of the class prior
To estimate the log-probability of the pre-training distribution  $\hat{\pi}_s = \log \mathbf{q}$ , we utilize the optimization toolkit Cooper [15] from https://github.com/cooper-org/cooper.  $\mathbf{q}$  is initialized as a uniform distributed,  $\mathbf{q}(y) = \frac{1}{K}$  for all  $y\in [K]$ . We use the standard SGD as the primal and dual optimizers for 2000 steps.
B.6 Long-tail learning baselines and training details
We compared with 5 long-tailed classification methods:
1. Standard ERM: We learn the model by standard empirical risk minimization on the long-tailed data.
2. Learnable Weight Scaling (LWS) [24]: We first learn the model by standard ERM, then fix the model and learn to re-scale the magnitude of the classifier using class-balanced sampling.
3. Logit Adjustment (LA) [35]: We first learn the model by standard ERM, then compensates the long-tailed distribution by subtracting a class-dependent offset to the model outputs.
4. Balanced Softmax (BS) [43] modifies the Softmax cross-entropy loss which explicitly accommodate the label distribution shift during optimization.
5. BALLAD [33] first fine-tunes the vision-language models via contrastive loss on long-tailed data, then freezes the backbone and finally employs an adapter to enhance the representations of tail classes with re-sampling strategies.
For all combinations of the fine-tuning baselines and long-tailed learning methods, visual backbones are initialized from CLIP-ResNet-50 and all classifiers are initialized by feeding prompt with class names to the text encoder. We use SGD for all experiments with a momentum of 0.9 for 50 epochs with batch size of 512. The initial learning rate is set to  $1.6 \times 10^{-3}$  which is decayed by the cosine annealing rule. To mitigate explosive gradients, we use the warmup learning rate equals to  $10^{-5}$  during the first epoch. For the sake of fairness in comparison, all hyper-parameters of baselines are carefully searched using grid search on the validation set.
C.1 Few-shot learning accuracy
We provide mean and standard deviation in Table 9 in for  $\{1,2,4,8,16\}$  shots on all 11 few-shot learning datasets.
Dataset	1 shot	2 shots	4 shots	8 shots	16 shots
ImageNet	61.65 ± 0.15	62.64 ± 0.01	63.32 ± 0.07	64.51 ± 0.09	65.61 ± 0.03
Caltech101	89.08 ± 0.09	90.25 ± 0.25	90.98 ± 0.43	91.90 ± 0.21	92.58 ± 0.42
OxfordPets	87.79 ± 0.15	87.86 ± 0.21	88.22 ± 0.21	88.09 ± 0.27	89.53 ± 0.16
StanfordCars	60.00 ± 0.14	63.10 ± 0.42	66.25 ± 0.19	69.87 ± 0.09	73.95 ± 0.11
Flowers102	73.45 ± 0.60	81.00 ± 0.46	88.31 ± 0.65	92.89 ± 0.46	95.41 ± 0.32
Food101	78.41 ± 0.07	78.62 ± 0.07	78.68 ± 0.06	78.85 ± 0.19	79.54 ± 0.47
FGVCAircraft	20.22 ± 0.59	22.09 ± 0.37	24.65 ± 0.85	28.23 ± 0.44	31.99 ± 0.50
SUN397	64.29 ± 0.19	66.32 ± 0.16	68.01 ± 0.08	69.99 ± 0.18	71.64 ± 0.21
DTD	47.38 ± 1.23	50.75 ± 1.46	56.90 ± 0.20	62.73 ± 0.80	65.78 ± 0.49
EuroSAT	56.50 ± 1.34	67.26 ± 3.58	72.40 ± 2.43	77.59 ± 1.84	84.93 ± 1.89
UCF101	65.32 ± 0.17	68.42 ± 0.81	70.88 ± 0.50	74.23 ± 0.24	76.07 ± 0.03
Table 9: GLA Accuracy (%) with standard deviation of few-shot learning on 11 datasets.
C.2 Experiments on LAION-400M
To support our thought experiment in the discussion of Section 4.3, we use the Open-CLIP ViT-B/16 [21], the first  $20\mathrm{k}$  image in LAION-400M datasets and the bias estimation method proposed by [2] to estimate the expected logits across 8 classes: "dog", "cat", "squirrel", "tiger", "elephant", "horse", "pig" and "bird". The bias estimation proposed by [2] provides a good estimation of  $\log P(\mathbf{x})$  over the labels under pre-training distribution. Our GLA estimates the label bias matches the downstream domain, we consider two downstream domain styles, i.e., "photo" and "sketch" from DomainNet [38] dataset. For each domain, we randomly sampled 50 images for each class.
Method	dog	cat	squirrel	tiger	elephant	horse	pig	bird
expected logits by [2]	0.059	0.039	0.043	0.053	0.043	0.062	0.061	0.028
πpby “photo”	0.059	0.041	0.047	0.055	0.048	0.062	0.060	0.033
πpby “sketch”	0.059	0.043	0.063	0.061	0.067	0.042	0.047	0.056
Table 10: Comparison among different bias estimation using Open-CLIP-ViT-B/16.
We present the expected logits estimated by [2], along with the one calculated from "photo" and "sketch" downstream domain data in Table 10. Since the softmax is invariant to constant offsets, i.e.,
Method	Sketch	Real Photo
Zero-shot Open-CLIP	92.25	97.00
Debiased by [2]	89.50	97.50
Debiased by GLA	93.00	97.75
Table 11: Performance on sketch and real photo domain.
$\mathrm{softmax}(\mathbf{x} + c) = \mathrm{softmax}(\mathbf{x})$ , we align the three label bias to yield the same logits on "dog" class by subtracting constant values. We observe that the label bias estimated by the "photo" domain align closely with [2] due to its close resemblance to the pre-trained domain. Conversely, the "sketch" image style, which significantly differs from the pre-training domain, results in a more pronounced deviation in the pre-trained label bias.
Additionally, we apply the three estimated label biases to debias the zero-shot model and evaluate the classification performance. The results are shown in Table 11, where the superiority of our method becomes evident on "sketch" domain (93.00% vs 92.25%). Applying the label bias from [2] on the "sketch" domain degrades the model's performance (from 92.25% to 89.50%). This is attributed to the overall pre-training label bias does not adequately reflect the bias specific to the "sketch" domain.