OTTER: Effortless Label Distribution Adaptation of Zero-shot Models
Authors: Changho Shin
Jitian Zhao
Sonia Cromp
Harit Vishwakarma
Frederic Sala
Venue: NeurIPS 2024 poster (2024)
Keywords: zero-shot classification
label distribution
adaptation
Primary area: other
TL;DR: We introduce a simple and extremely low-cost approach for label distribution adaptation in zero-shot models.

Abstract: Popular zero-shot models suffer due to artifacts inherited from pretraining. One particularly detrimental issue, caused by unbalanced web-scale pretraining data, is mismatched label distribution. Existing approaches that seek to repair the label distribution are not suitable in zero-shot settings, as they have mismatching  requirements, such as needing access to labeled downstream task data or knowledge of the true label balance in the pretraining distribution. We sidestep these challenges and introduce a simple and lightweight approach to adjust pretrained model predictions via optimal transport. Our technique requires only an estimate of the label distribution of a downstream task.  Theoretically, we characterize the improvement produced by our procedure under certain mild conditions and provide bounds on the error caused by misspecification. Empirically, we validate our method in a wide array of zero-shot image and text classification tasks, improving accuracy by 4.8% and 15.9% on average, and beating baselines like prior matching---often by significant margins---in 17 out of 21 datasets.

OTTER: Effortless Label Distribution Adaptation of Zero-shot Models
Changho Shin, Jitian Zhao, Sonia Cromp, Harit Vishwakarma, Frederic Sala
Department of Computer Sciences
University of Wisconsin-Madison {cshin23, jzhao326, cromp, hvishwakarma, fsala]@wisc, edu.
Abstract
Popular zero-shot models suffer due to artifacts inherited from pretraining, One particularly detrimental issue, caused by unbalanced web-scale pretraining data, is mismatched label distribution. Existing approaches that seek repair the label distribution are not suitable in zero-shot settings, as they have mismatching require- ments, such needing access labeled downstream task data knowledge of the true label balance the pretraining distribution. We sidestep these challenges and introduce a simple and light approach to adjust pretrained model predic- tions via optimal transport. Our technique requires only estimate of the label distribution of downstream task. Theoretically. we characterize the improvement produced by our procedure under certain mild conditions and provide bounds on the emror caused by misspecification. Empirically, validate method in a wide array of zero-shot image and text classification tasks, improving accuracy by 4.8% and 15,9 55.9% on average, and beating baselines like prior matching often by significant margins-in in 17 out of 21 datasets.
1 Introduction
Zero-shot models are popular but struggle with bi- inherited from their large pretraining datasets [21; 59; 2]. particular, zero-shot classification strongly biased by the label distribution of the pre- training task. When the label distribution of the down- stream task differs from pretraining. the performance of zero-shot classificrs suffers greatly. For example, Figure illustrates the effects of mismatched distri- butions on pet image classification task. Two CLIP models (RN50, and ViT-B/16) produce biased predic- tions the abyssinian and Persian classes. Furthermore, datasets with large number of classes, such as ImageNet, may contain both extremely com- mon and very rare classes, resulting in outsized probability that zero-shot model will predict some classes over others. As a result, even large models intended for use in zero-shot settings, such as CLIP [49], naturally have label distribution mismatch between pretraining data and downstream tasks.
RN50
0.5
VTT 9/16
Ground Truth
0.4
Progertion
0.3
0.2
0.1
Abyssinian
Persian
Pug
Figure 1: Label distribution mismatch exam- ple in zero-shot classification. In the Oxford- IITT-Pet dataset, the ground-truth labels are uniformly distributed, while zero-shot mod- els exhibit biased predictions toward certain classes. This bias is influenced by the distri- bution of labels in the pretraining task.
Existing methods that seek to address label distri- bution make strong assumptions have expensive requirements. For example, fine-tune model. we must obtain labeled fine-tuning dataset of adequate size, then obtain the time and compute to further train the model. To perform label shift adap-
Conference on Neural Information Processing Systems (NeurIPS 2024).

---

tation techniques, we must know the true label distribution of the pretraining distribution-difieee to impossible for real-world tasks.
Can we deal with label distribution mismatch without additional training or access to ground- truth downstream task information? While scemingly challenging, one cause for optimism is the observation that zero-shot models still give relatively high prediction probabilities for correct classes, though classes common in pretraining tend to have relatively inflated scores overall. Intuitively, the model has already learned identify examples of its downstream classes (and does not require further training) and is already impacted by the pretraining label distribution (and so does not need access to the ground- touth pretraining label distribution). Instead, the model's prediction probabilities must be adjusted based an estimated downstream label distribution specification.
To perform this label distribution adjustment, we view zero-shot learning through the lens optimal transport (OT) and develop a technique called OTTER (Optimal TransporT adaptER). This OT-based approach offers systematic way to rebalance predicted labels: data points transported optimal downstream classes, minimizing the overall cost in accordance with the estimated the downstream label distribution specifications.
Theoretically, we show that optimal transport given the true label distribution of the downstream can recover the Bayes-optimal classifier under mild conditions. Additionally, we provide error bounds on our adaptation method for misspecification. We provide synthetic experiments validating our theoretical claims. In real-world data settings, we validate our method on wide variety of image and text classification tasks, showing 4.8% and 15.5% accuracy improvement and text zero-shot classification tasks, respectively. Finally, we evaluate our method in few-shot adaptation scenarios. where OTTER provides further improvements when combined with linear probing. Our contributions include:
average in image
OTTER, an algorithm to deal with label distribution mismatch at inference time via optimal transport,
Theoretical results showing the effectiveness of our method, including the ability to recover the Bayes-optimal classifier and a sensitivity analysis with respect to the label distribution specification estimation,
Extensive empirical results on zero-shot classification for text and image datasets, showing accuracy improvements of up to 25%.
Experimental results demonstrating the applicability of OTTER to few-shot settings, showing accuracy improvements of up 15%, even with noisy label distribution specifications, Extensions of OTTER to leverage label hierarchy information or relax the batched prediction requirement and Application to LLM selection bias mitigation.
2
Background and Problem Formulation
Before presenting OTTER and providing our results, we introduce some crucial background and describe the problem setting formally.
2.1 Background
We briefly describe zero-shot models, the technical tool we use (optimal transport), along with other techniques that seek to address shifts. We have extended related work, including in-depth characterizations and comparisons with related methods, in Appendix B.
Zero-shot Models. Zero-shot classification, popularized by models such as CLIP [49], is pow- erful paradigm that enables prediction on downstream tasks without additional fine-tuning. Image, language, and multimodal models have been incrcasingly employed for zero-shot prediction [64; 36). These models undergo extensive pretraining on massive datasets with concept and label spaces that may be very different from those of downstream applications.
Optimal Transport. Optimal Transport (OT) is a framework for matching two probability distribu- tions [48; 53). We predominantly consider optimal transport between empirical discrete measures. Suppose that we are given points Z1, 22,..., In E X' and ......, E Y, a source measure defined by Σ wrdz,, and target measure given by v = Σy=1 PJoj, where Wy are positive values such that Σ" 0. Σ P/ 1. Suppose also that 6,, is a Dirac delta function

---

Algorithm 1 OTTER
I: Input: Input X (I1, Fn.. label distribution specification (pi, .P.....,KK .K)), cost matrix
E RT*k
2: Define input marginal A. prediction marginal P' = (P1....,PP'' PK)
3: Run optimal transport and obtain transport plan s.t. arg mig C).
4: Get modified classification outputs y/ = arg maxje øs.j.
Return (c/))
at i.e. (z'2²) if x' z, and (z²) = 0 otherwise. Given a cost matrix C e Ra** the Monge-Kantorovich Kantorovich formulation of optimal transport is to find minimal cost transport plan such
= arg min (7,, C)2, (
where 1) (r C h1 -. 1-
Distribution and Label Shifts. Distribution shift refers to the discrepancy between source distribution P. on which the model is trained, and target distribution on which the model deployed. Distribution shift often degrades trained model performance on target tasks. Label shift is a specific type of distribution shift such that PY(Y) ş P:(Y) and the data generation process is fixed other words, the conditional distributions of the inputs are the same: Px (Y) Techniques such as importance sampling [35; 6; 24), recalibration [3] and domain adaptation [55] are commonly used to mitigate the effects of label shift. Unfortunately, these methods assume access to source distribution data, whereas zero-shot models' pretraining data is inaccessible (often proprietary or blocked for privacy reasons). Thus, adapting zero-shot models to new label distributions poses challenges unmet by these pre-existing methods.
2.2 Problem Formulation
Let X {I1, 22,... ........ In } be an inference dataset with 2, X. Furthermore, let Y (V1 1/.................... ......... ....) be the true labels the K- K-elass classification dataset, such that ys E c = [K], are sampled according to the downstream label distribution v = (P1. .p. .P))
Let (z; == Pa = j|X = .)) be a pretrained classification model constrained to the downstream label space. During pretraining, has been biased to the source label distribution v², We wish to offset such label distribution bias with a label distribution specification & for the target distribution. á is expected to closer to the true label distribution of the downstream task. Given a label distribution specification, our goal is to rebalance predictions so that the predicted label distribution follows the label distribution specification.
3 Proposed Framework
We propose OTTER (Optimal TransporT adaptER), an optimal transport hansport-baased label distribution adaptation approach. Our goal is to have the input data points allocated to K classes match given label distribution è where Σ== 82 = 1, 22 2 0. Specifically, we want to classify ni, points as the first class, nıfy points as the second, and so on. However, there are many such allocations, and it is not a priori clear which one should be selected. We propose formulating an optimal transport problem that selects the allocation minimizing particular cost:
क = arg min (r,, C)2 7Ell(μ.0)
where I[ D) = (y E R 111= -0 1) n= 1 and C' is the cost (loss) matrix such that Cij represents loss when we classify x; as class j. This procedure is descrihed in A Algorithm I. Note that this procedure naturally matches the given label distribution specification &.
We wish to use Algorithm for zero-shot classification given the pretrained model 8p. To do so, we must select cost function and produce Ciy. An ideal choice of such a function is Ci5 llog PY = j|X i) such that optimal transport minimizes the negative log posterior under constraints. However, the target distribution P. is unknown. Instead, replace the posterior with the classifier scores 80 (), We highlight that this choice of cost mairix is natural extension of

---

zero-shot classification under the label distribution constraint. We prove this claim in the next section. First, we show a toy example that shows how OTTER improves zero-shot accuracy.
Example. To illustrate the benefits of OTTER, consider the following example for hinary classifica- tion. We have two data points, X {21, with Y {1, and true label distribution v ( 2). Suppose that the zero-shot model's 's prediction scores are $1 = (0.4, 0.6) and 82 = (0.1.
Traditional classification yields f1 2, 1/2 = 2. producing a 50% error rate. However, given the cost
matrix C' derived from the prediction score matrix
0.4
0.6
log 0.4 log 0.6 log 0.1
s
C -
0.1
0.9
log 0.9
along with = 0.55 and v = (0.5, 0.5). the optimal transport procedure discovers the transport
0.5 0.0
map
0.0
0.5 yielding 1, 9/2 2. This corrects the original zero-shot prediction error.
Extension to Online Predictions While OTTER offers efficient label distribution adaptation, requires a batched set of inference data points, making online predictions challenging. To address this, we introduce R-OTTER (Reweighting OTTER), which Icarns rewcighting factor-an an estimate of P.(Y)/P.(Y) using OTTER's 's predictions NOTTER on a validation set. Once learned, these reweighting parameters can be applied to online predictions by adjusting logit probability scores. We use reweighting formulation equivalent to logit adjustment in [71]. The reweighted probability scores of Pg. with reweighting vector y R are defined as:
TyPolY =x
(Y = -|=
The parameter is learned using cross-entropy loss VOTTER- We expect R-OTTER perform comparably to OTTER, with the additional advantage of not requiring OTTER to run over the entire datasct. In the following section, we provide a thcoretical result showing that r" (Y) is optimal reweighting parameter when learned from yoTTER as pseudolabels, effectively addressing label shift.
4 Theoretical Results
In practical scenarios, label distribution specifications are frequently subject to noise, and prediction probabilities may not be well-calibrated. To undersland the impact of these factors, examine how errors in label distribution specification and calibration influence the transport plan. Our theoretical analysis yields following findings: (a) OTTER can recover the Bayes-optimal classifier in the label shift setting, (b) for noisy cost matrix with the noisy label distribution specification setup, the suboptimality can be bounded by the deviation of cost matrix and label distribution, and (c) R-OTTER effectively addresses label shift when learned from BOTTER as pseudolabels.
Classification as optimal transport. Prior to discussing the main theoretical results, we demon- strate that standard interpreted as a (trivial) solution derived from optimal transport.
wetpfictttttooeppeeeeeeeee as û: arg max Po j|X x.)- be
Theorem 4.1. Let 7 1 Σ. 100 = 1}. where y. = arg max Po = j'( = x.).
Then, given C,5 log Po (
5|X xz).
= arg
min xs)
(7. C).
0 OT = arg max Wey.
jE(K)
Assuming there are ties in scores, Le.. Po jIX x.)) j'XX for all j Jr the OTTER predictions are equivalent shro-shot predictions, i.e. 5. = o/ for all i [n].
This theorem has the following implications. First, suggests that the predictions will remain unchanged we set ò = 25 Second, Bayes-optimal classifiers can be derived through optimal transport, using a (true) cost matrix defined as Cj₃y log P( j|X æz), coupled with the true label distribution D*

---

Our analysis begins with the label shift setup, which is a commonly-studiiet type of distribution shift- as well as prominent issue when applying zero-shot models. We demonstrate that when the label distribution is correctly specified, optimal transport preserves Bayes-oplimal classifier predictions under label shift. Next, consider general perturbations in label distribution and cost matrix as well as their impact on the resulting solutions.
4.1 Label Shift Invariance
this setting, we assume features follow the same conditional distribution across source and target distributions, i.c. P.(XYY (.))). Furthermore, accurately calibrated the training dataset, such that so (0(x,) .) PY j| æ). For zero-shol models, we often lack access to Ps- This is a typical scenario in zero-shot model applications: after training on large-scale corpora, we use the pretrained model without the source dataset.
suppose that the prediction scores
For a given downstream task with the target label distribution '" PY(Y), one standard approach achieve the Bayes-optimal classifier for the target distribution is to reweight the score function oulputs using the ratio P.Y s) This adjustment leads
P.YY =))
Pi(Y =) P.(X z|Y = j)- P = =) P,Y = -|X x).
(z = (t)
P. j)
This reweighted score function aligns with the target distribution, thus correcting label shift.
Although reweighting the score function a popular solution, it faces an important obstacle when applied to zero-shot models like CLIP, where the source distribution I. (Y) is typically unknown. We show that OTTER successfully induces Bayes classifier for the target distribution, represented as fx( = arg maxje((" jE[K] P.(Y = j|X = x), without requiring access to P. (Y).. This capability
particularly significant for zero-shot models, cnabling them to adapt to target distributions effectively, even in the absence of explicit knowledge of the source distribution.
Now, show that optimal transport be effective tool correct label shift.
Theorem 4.2. Suppose the pretrained model well-calibrated for the source distribution,
Po -| xx) PY j|X x;))
and there is no tie probability, for all j # jr, [n]]
Po j| x;) # Po = X =xxx)))..
Denote the Bayes optimal predictions the target distribution as V arg max log P.(Y i|X æz). Then OTTER predictions I OTTER(X, C) are the same as Bayes optimal predictions g"
That is, OTTER recovers Bayes classifier in the target distribution without access to the source distribution, given the target distribution and a well-calibrated model for the source dataset.
4.2 General Perturbation Sensitivity
practical applications, calibration error could extend beyond noise in the elements of the cost matrix. A key source of error is label distribution estimation error. Hence, we address more general setting, examining the impact of simultaneous perturbations in the label distribution and cost matrix the transport plan. Our result applies techniques from perturbation theory for linear programming
We rewrite our optimal transport problem min {T,, C') as a linear programming problem. Let π and C' be the transport plan and cost matrix respectively. Matrix G and vector are used to denote form feasible plan which transports distribution from y to v.
the row and column constraints
T U I
11 - H
H
1
G
:9
4
Then, we have the equivalent linear programming problem,
min
G vec(1) g. 2
(1)

---

We adapt a theorem from Robinson [52] with our optimal transport problem notation.
Theorem 4.3. Let the primal linear programming problem be defined equation (1), and its dual problem be maxfwr w g glw²"G G S vec(C)², 2 0}. Suppose perturbed cost matrix is C C Ac.
the perturbed class distribution & v + Dn, such that 9 = g + Ag where
0
i カー
A
=
-D v4
Assume that primal and dual problems are solvable. Denote the original solutions as æ, w and perturbed solutions as â and Then,
|| - S x²? (114,12 [I[ve((Ac))+11 [[vec(C)) vec(f) gf |/ w
where pSps S oc and K. is a Hoffman constant that only relates to the original problem [30].
Ignoring the constant and the subtraction part, the upper bound can be decomposed into three components,
Ap: noise (or the estimation error) of the target balance,
[vec(Ac)]+: noise (or the calibration error) of the cost matrix, vec(C)" vec(C))ecc(tt) g" ú: the suboptimality of perturbed solution ú.
Theorem 4.3 implies that the deviation from perturbed solution to true solution bounded by the magnitude of perturbations and suboptimality of the perturbed solution. From this result, we can expect prediction accuracy deteriorate with perturbations in the label distribution and calibration.
4.3 Effectiveness of R-OTTER
We provide theoretical result showing that R-OTTER can learn an optimal parameter by learning reweighting parameters from BOTTER as pseudolabels, and produce identical predictions to OTTER once the optimal parameter is obtained in the label shift setup.
Theorem 4.4. Under the same assumptions in Theorem 4.2, the parameter r" P((Y)/P,(Y) is optimal when learning with yoTTeR as pseudolabels.
The proof is provided in Appendix D.6.
5 Experiments
The primary objective of our experiments to (I) validate that OTTER improves zero-shot model performance when given accurate label distribution estimates and (2) investigate its sensitivity to perturbations. In experiments real datasets (Section 5.1), we confirm that OTTER improve zero-shot classification significantly in a variety of settings. In synthetic experiments (Section 5.2), we validate our theoretical claims-label shift invariance and sensitivity to perturbation in a fully controllable setting. Additionally, we show that OTTER can be combined with label distribution estimation methods in the few-shot learning setting (Section 5.3). Next, we show the empirical results for H-OTTER that leverages label hierarchy (Section 5.4) and R-OTTER that mitigates the batched prediction requirement (Section 5.5). Finally, (Section 5.6). Our code available at https//gilub.coom$ppkkker abbOTTER..
apply OTTER to mitigate I.I.M selection bias
5.1 Real Data Experiments
We hypothesize that the model performance can improve significantly when the label distribution specification is exact.
Setup and Procedure. We used 17 image classification datasets and 4 text classification datasets. We employed CLIP [49] for image vero-shol classification, and BERT [19]. A comprehensive list and details of experiments can be found in Appendix E.
Baseline. We adopt Prior Matching (PM) [37] a baseline. It optimizes score weighting parameters to align with the label distribution specification. A detailed explanation of Prior Matching is given in Appendix C. It is worth noting that the performance of Prior Matching is highly sensitive hyperparameters such as temperature and learning rate. Optimal hyperparameters may vary across

---

Zero-shot Prior Matching OTTER
Zero-shot Prior Matching OTTER
CIFARIO
88.3
91.3 (4.0.)
91.7
Oxford-IITT-Pet
83.8
82.0 (20.3)
88.8
CIFARIOS 100
63.8
64.1 (+2.7)
67.9
Stunford-Cars
55.7
39.8 (42.6)
59.7
Caltech10
79.8
59.3 (=11.4)
88.7
STL10
98.0
98.4 (#0.0)
98.6
255
79.8
9.5 (+:1.5)
87.0
SUN397
47.1
6.7 (+166)
54.1
21
19.8
19.0
21.1
CUB
46.0
40.4 (+0.00
50.4
DTD
39.0
42.1 (+0.1)
44.4
ImageNet
60.2
53.6 (+:.1)
62.9
EUROSAT
32,9
41.6 (+0.8)
42.0
ImageNet-r
68.9
16.7 (1:355
72.4
Flowers 102
64.0
54.0(444.1)
70.8
ImageNet-Sketchh
39.8
36.5 (1:0.4
44,5
Food101
85.6
86.8 (1:3.1)
89.9
Amazon review|
74.0
(1:46.4)
91.7
GenderBias HateXplain
84.1
41.4 (9.6))
91.9
CivilComments
48.4
57.2 (1:37.7)
81.4
30.9
31.3 (+:3.)
34.3
Table I: Accuracy (%) in zero-shot image classification (ViT-B/16) and text classification (BERT). We use the true label distribution as the label distribution specification. The numbers in parenthesis of Prior Matching represent the standard deviation of 10 different samplings of the validation sct. OTTER produces improvements nearly scross-the-boord with an average lift 4.9% in image classification and 15.5% in text classification, outperforming a powerful baselinc, prior matching in almost all cases.
different datasets. We selected hyperparameters through grid search, by evaluating their performance on validation set, consisting of 10 labeled examples per class. In contrast, we highlight that OTTER is funing-free.
Results. Table shows the image classification results with CLIP (ViT-B/16) and the text classifica- tion results with BERT. Notably, OTTER demonstrates 4.8% and 15.5% enhancement on average in image and text zero-shot classification, respectively. While Prior Matching shows competitive performance when accurately tuned, it often struggles. We found that hyperparameter tuning fails datascts such as Caltech256, SUN397, ImageNet-r (Appendix E, Table 7). This suggests that the hyperparameter selection process necessitates validation set label distribution unusable in zero-shot scenarios. More details and
in the dlass-imbalanee
similar to the target ceribbuttooneenddrrii additional experiment results including the sensitivity study on the label distribution specification error, computation time, and combination with other prompting methods - are provided in Appendix E.3.
5.2 Synthetic Experiments
We hypothesize OTTER is invariant to label shift under the conditions Theorem 4.2. We also investigate the sensitivity to perturbations of the cost matrix and the label distribution.
Setup and Procedure. We simulate label shift in logistic regression. Suppose X|Y ((-1,1) and X|Y =
(), Training data is sampled from mixture of Gaussians X, ~ 1f1((-1,1) + vfN such that P.(Y = 0) = vg, P.(Y = 1) = v4, vg + 1/f = 1. Similarly, sample the test data from X, 1/NN(-1,1) label distribution as v/ = 0.1, = 0.9 and vary test set label distribution sf simulate label shift. We train a logistic regression model with 10,000 samples from the source distribution, and test the model with 10,000 samples from the target distribution. A Bayes-optimal classifier in the target
(1, 1). We fix the training set
distribution is given by f (z) [ 2 (log ب + 1)]. The naive classifier defined as the maximizer of the predicted score. The OTTER predictions are produced with Algorithm 1, with the cost matrix Cjj = log PY j|X = Iz) and the label distribution specification v², where (X) represents the logistic regression model scores.
We separately investigate perturbed prediction score matrix and perturbed label distribution spec- ification's 's impact on the prediction accuracy. For perturbed prediction scores, we fix the label distribution to be the true one, and add noise 5 ~ (0, o²) of varying levels o to the predicted score PY 1[.). For the perturbed label distribution specification, we fix the prediction score be true scores and add noise e: e = D² + (e, -e). We use these perturbed variants to obtain perturbed solutions and compare with ground-truth solution.
Results. Figure 2 illustrates how accuracy changes with label shift when the predicted score is perturbed and when label distribution specification is perturbed. We observe that the naive classifier

---

80
80
Acc(%)
Acc(%%
70
70
Bayes
OTTER, [
GTTER, N-11 0.057)
OTTER,
60
CTTER M-MO 0.357
OTTER, =.1
OTTER 6- Mi8 0.2%
OTTER. 0.15
0.0
0.2
0.2
0.6
TVIPAY, Pa(Y))
TVIP-Y), P.(M)
(a) Prediction accuracy changes with perturbed confi- dence score.
(b) Prediction accuracy changes with perturbed label distribution.
Figure 2: Synthetic experiment results. X-axis represents total variation distance between the source and the target distribution, describing label shift severity. Y-axis represents prediction accuracy. Curves represent different methods and noise levels. Our approaches dramatically outperform the baseline at higher mismatch levels.
deteriorates as the total variation distance between source and target distributions increases. It indicates naive classifier sensitive to label shift. However, without perturbation, OTTER remains unaffected by the label distribution shift, which validates our invariance result in Section
In the case of confidence prediction perturbation, both the naive classifier and OTTER have accu- racy decreasing as perturbation level increases. For simplicity, omitted the naive classifier's performances under different levels of noise adding zero-mean noise does not alter accuracy significantly. We observe that OTTER has better performance than the naive method when sigmificant label shift exists. Similarly, for label distribution perturbation, observe the noise level increases, OTTER's accuracy downgrades but still yields better performance when label shift severe.
Our experimental results suggest simply using prediction scores for zero-shot classification leads to inaccurate predictions under label shift, while OTTER is robust to label shift when perturbations are present. Perturbations in both predicted score and label distribution specification downgrades the predicted accuracy, as expected, but OTTER still yields better results than the naive baseline.
5.3
Few-shot adaptation with label distribution estimation
We anticipate that OTTER can be used in few-shot learning when com- bined with label distribution estima- tion methods. expect OTTER can improve zero-shot classification if the label distribution estimation error is sufficiently small. Also, we expect OTTER can improve linear probing, which is one of standard approaches for few-shot learning.
Estimation Error
Accuracy
0.22
72.5
80.18
67.5
3.116
0.14
QTTER
0.12
60.0
0.10
25
50
75
100
25
50
75
100
Samples per class
5 Samples per cless
Setup and Procedure. We use the same datasets as the previous experi- ment. We consider 10-shot learning setting: 10 labeled samples per class are given. Note that labeled samples have uniform label distribution, while the label distribution in the target dis- tribution may not be uniform. This setting requires the use of label distri- bution estimation methods used in label shift adaptation [35; 6; 24]. We estimate the target label
(w) BBSE estimation error
(b) Accurucy
Figure 4: Ablation on the number of samples in few-shot learning. In (a). We observe that BBSE estimation get more precise the number of samples increases. Following this, OTTER gets better accuracy in (b). Additionally, OT- TER consistently improves linear probing when combined.

---

Dataset
ZS
BBSE+PM ZS BBSE+OT
BBSE4PM BBSE+OT
CIFARIO
88.3
72.7
87.5
90.2
89.8
90.0
CIFAR100
63.8
3.2
59.1
58.3
24.4
60.5
10
79.8
32.5
80.7
91.5
87.5
91.4
Caltech256
79.8
6.0
80.3
84.5
58.4
85,4
Country211
19.8
1.5
15.9
12.4
9.2
13.2
DTD
39.0
3.2
31.2
58.6
49,0
59.3
EUROSAT
32.9
19.2
34.0
74.6
71.6
75.9
Flowers 102
64.0
40.3
60.8
89.0
87.8
90.2
Food101
85.6
15.3
82.3
79.1
60.6
79.8
Oxford-IITTE -et-
83.8
43.3
71.4
75.7
72.0
75.6
Stanford- -arrs
55.7
2.3
51.7
64.5
65.4
66.3
STL10
98.0
97.4
96.9
97.7
97.5
97.6
SUN397
47.1
6.9
25.6
0.2
0.2
0.2
cub
46.0
3.3
45.5
72.2
63.3
75.6
ImageNet
60.2
0.8
57.7
56.8
53.6
59.8
IetgeNet-r
68.9
1.7
63.3
54.9
47.6
57.1
ImageNet-Sketch
39.8
0.8
40.4
43.4
37,9
48.3
Amazon
74.0
47.9
89.1
71.3
66.9
71.3
CivilComments
48.3
69.1
55.8
53.8
45.5
53.8
Gender
84.0
57.0
87.8
78.0
71.2
78.5
HateXplain
30.4
34.4
35.2
32.8
32,7
32.3
Table 2: Accuracy (%) with OTTER combined with class balance estimation. ZS BBSE denotes
BBSE label distribution estimation based zero-shot prediction scores, and LP BBSE denotes BBSE label distribution estimation based linear probing prediction scores, We report the mean of 10 different random samplings of the validation set. OTTER produces moderate improvements when comined with linear probing in image classification tasks. In text classification tasks, OTTER significantly improves accuracy, up to 15.1 ,, even with noisy label distribution estimation.
distribution with Black Box Shift Estimation (BBSE) [35]. BBSE estimates the target balance using confusion matrix, under the label shift assumption. For detailed explanation, refer to Appendix C.
Results. Table shows the image and text zero-shot classification results with the label distribution estimation via BBSE and linear probing. The image classification results show that OTTER can yield mild improvement over lincar probing, even with the label distribution estimation errors. Figure 4 shows that accuracy improvement consistent across the number of samples used for linear probing. found OTTER improves zero-shot text classifications where number of classes is small (K = 2 or 3). Whilc it shows relatively high variance due to the small sample size (20 30), the average accuracy improves significantly over zero-shot classification. More detailed analysis regarding label distribution estimation error and the number of samples is provided Appendix E.4.
text classification,
5.4 Zero-shot prediction improvement with class hierarchy
We hypothesize incorporating class hierarchy infor- mation can enhance few-shot label distribution esti- mation and thus improve zero-shot predictions.
OTTER
H-OTTER
RN50
38.5 (+:.9)
(E3.D
RN101
39.9 (+669)
(+5.1)
Setup and Procedure. We use subset of CIFAR 100 data with WordNet hierarchy. Specifically, we take 'fish' and 'tree' as superclasses and have subclasses in each of them. We suppose we can access 10 labeled samples per each subclass. We first apply OTTER with superlevel label distribution estimation and make pseudo-labels of superlevel class in the test set. Using them, we estimate the sublevel label distribu- tion and use OTTER.
VTT-B/32
39.0
59.3
VIT-B/16
54.6 (+..3)
58.2
(+3.6)
VIT-1/14
71.3 (+:39)
69.4 (+5.2)
Table Accuracy (%) with hierarchical OTTER (H-OTTER). (H-OTTER) yields additional improvements over OTTER, up 5.1%, using the hicrarchy information of labels.
Results. Table presents the results. As anticipated. we note enhancement in accuracy when compared to the naive implementation of OTTER. Specifically, observe significant improvement accuracy for RN50, RNI01, and ViT-B/16.

---

Zero-shot OTTER R-OTTER
Zero-shot OTTER R-OTTER
CIFARII IO
88.3
91.7
88.4
Oxford-IIT
83.8
88.8
85.7
CIF
63.8
67.9
65.3
Stanford-
55.7
59.7
51.0
Caltech101
79.8
88.7
88.2
STLI0
98.0
98.6
98.1
Caltech256
79.8
87.0
79.6
SUN397
47.1
54.1
46.6
Country211
19.8
21.1
19.5
CUB
46.0
50.4
44.4
DTD
39.0
44.4
44.0
ImageNet
60.2
62.9
59.8
EUROSAT
32,9
42.0
39.3
ImageNet-r
68.8
72.4
68.5
Flowers 102
64.0
70.8
69.7
ImageNet-
39.8
44.5
39.3
Food101
85.6
89.9
88.5
Table 4: Accuracy (%)
naive zero-shot, OTTER, and R-OTTER in zero-shol image classification
(ViT-B/16)
ARC-Challenge
CommonseesseQA (CSQA)
MMLU
Naive
OTTER
Naive
OTTER
Naive
OTTER
Model
Acc.(f)
(U)
Acc.
RSid
RSid
Acc.
RSid
Acc.
Acc.
RSid
Lisma-2-7b
36.0
45.5
28.4
22.5
Llama-2-13b
62.9
6.0
62.8
$7.0
10.2
58.1
2.0
51.1
6.9
51.8
1.3
Dlamf-2-7b-cfia
363
12.3
373
L3
56.5
13.5
60.4
335
7539"
"13.1"
46.6
"0.5
LJama-2-1 3b-chat
64.4
13.7
66.2
64.0
9.8
66.4
52.3
13.9
1.0
Vicuma-7h
31.5
3.6
34.1
2.3
56.9
8.3
57.6
1.0
46.6
5.8
46.9
0.3
vicuma-
Table 5: Mitigation of selection bias via OTTER.
which we attribute primarily to the reduction in label distribution estimation error. Further details are provided in Appendix E.5.
5.5 Effectiveness of R-OTTER
We show that R-OTTER provides a performance comparable to that of OTTER empirically,
Setup and Procedure. We use the identical setup for Section 5.1. R-OTTER learns reweighting parameters in validation set using JOTTER. Note that the validation set is not required to be labeled since MOTTER is used as pseudolabels in validation set.
Results. Although R-OTTER is suboptimal compared to OTTER due to generalization issues, it still provides label distribution correction, improving accuracy over zero-shol predictions. We also provide synthetic experiments for R-OTTER in Appendix E.
5.6 Mitigating selection bias in LLM multiple-choice questions
Selection bias is the tendency of L.LMs to favor certain prefix tokens multiple-choice questions [69; 17; 12; 65]. We demonstrate that OTTER can effectively mitigate selection bias by randomly shuffling the options and enforcing uniform class balance in OTTER.
Setup and Procedure. Our experimental setup follows Zheng et al. [69], utilizing the MMLU [28). ARC-Challenge [15], and CommonsenseQ4 (CSQA) [57] datasets. We test with Ilama-22 2(- chat)-7/13B [60] and vicuna-vl.3-7/133 [11] models, treating the probabilities of each option token (A/B/C/D) as prediction probabilities. OTTER is applied under the assumption uniform distribution. We use 0-shot predictions and evaluate performance using accuracy and the standard deviation of recalls (RStd) metrics.
Results. Table 5 presents experimental results. OTTER significantly reduces selection bias, enhancing accuracy by up to 10.6% and lowering RStd by as much as 25.7%.
6 Conclusion
While zero-shot models have been successful, pretraining using Internet-scale datasets yields artifacts that may harm downstream tasks. In this paper, we identify the bias in class balance, and provide a simple but powerful solution using optimal transport. Theoretically, we describe how OT can fix label distribution mismatch and its sensitivity to perturbations. Empirically, validated our approach's ability to improve zero-shot classification accuracy, mitigating label distribution mismatch zero-shot models. We believe our method can expedite the deployment of zero-shot classification. reducing the necessity of finetuning for downstream tasks.

---

Acknowledgments
We are grateful for the support the NSF under CCF2106707 (Program Synthesis for Weak Supervi- sion) and the Wisconsin Alumni Research Foundation (WARF). Jitian Zhao gratefully acknowledge support from the IFDS at UW- MW-aiiioon and NSF through TRIPODS grant 2023239 for their support. We thank Nick Roberts for his valuable discussions and insights.
References
[1] Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala. Zero-shot robustification of zero-shot models. In The Twelfth International Conference Learning Representations, 2023.
[2] Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. Evaluating clip: towards characterization of broader capabilities and downstream implications. arXiv preprint arXiv:2108.00 02818, 2021.
[3] Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias- corrected calibration is hard-to beat at label shift adaptation. In International Conference on Machine Learning, pages 222-232. PMLR, 2020.
[4] Bang An, Sicheng Zhu, Michael-Andrei Panaitescu- Chaithanya Kumar Mummadi, and Furong Huang. More context, less distraction: Visual classification by inferring and conditioning on contextual attributes. arXiv preprint arXiv:2308.0 01313, 2023.
[5] Dominique Azé and Jean-Noël Corvellec. On the sensitivity analysis of hoffman constants for systems linear inequalities. SIAM Journal on Optimization, 12(4):913-927, 2002.
[6] Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang. and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. In International Conference on Learning Representations, 2019.
[7] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings the 2019 world wide web conference, pages 491-500, 2019.
[8] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101-minin discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
[9] Wanxing Chang, Ye Shi, Hoang Tuan, and Jingya Wang. Unified optimal transport framework for universal domain adaptation. Advances in Neural Information Processing Systems, 35: 29512-29522 2022.
[10] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16: 321-357, 2002.
[11] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: / /wssys.sssss....m..ssm...s...... org htgg/bllg////0 2003 03- 00-30--20 3000viiuua.......
[12] Hycong Kyu Choi, Weijie Xu, Chi Xue, Stephanic Eckman, and Chandan K Reddy, Mitigating selection bias with node pruning and auxiliary options. arXiv preprint arXiv:2409.1888 2024.
[13] Ching Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debiasing languaeee models via biased prompts. arXiv preprint arXiv:2302.00070 2023.
[14] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.

---

[15] Peter Clark, Isaac Cowhey, Oren Etzioni. Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: / 1803.08435777 2018.
[16] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages
JMLR Workshop and Conference Proceedings, 2011. 1.
[17] Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. Discov- cring latent concepts learned in bert. In International Conference on Learning Representations.
[18] Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 245. Icee, 2009.
[19] Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
[20] Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. Multi- dimensional gender bias classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 314-331, Online, November 2020. Association for Computational Linguistics. doi: 10.18666 10.18653/1/2000. https: //t::///wwwwww. acl
URL
2020 0 emn 1-mmip.mmi......mmmm 23.
[21] Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Mcasuring and mitigating unintended bias in text classification. 2018.
[22] Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594-611, 2006.
[23] Alberto Fernández, Salvador Gareía, Mikel Galar, Ronaldo C Prati, Bartosz Krawezyk, and Francisco Herrera. Learning from imbalanced data sets, volume 10. Springer, 2018.
[24] Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. unified view of label shift estimation. Advances in Neural Information Processing Systems, 33:3290-3300, 2020.
[25] Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
[26] Dandan Guo, Zhuo Li, He Zhao, Mingyuan Zhou, Hongyuan Zha, et al. Learning to re-weight examples with optimal transport for imbalanced classification. Advances in Neural Information Processing Systems, 35:255 35:25517 17-25330, 2022.
[27] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2177-2226, 2019.
[28] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
[29] Dan Hendrycks. Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo. Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEFF//VF International Conference on Computer Vision, pages 3340-8349 2021.
[30] Alan J Hoffman. On approximate solutions of systems of linear inequalities. Journal of Research of the National Bureau of Standards, 49(4), 1952.
[31] Jonathan Kahana, Niv Cohen, and Yedid Hoshen. Improving zero-shot models with label distribution priors. arXiv preprint 2xxx:22112000788 2022.
[32] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEFE international conference computer vision workshops, pages 554-561, 2013.

---

[33] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
(34] Zachary Lipton, Yu- Xu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference machine learning, pages 3122-3130. PMLR, 2018.
[35] Zachary Lipton, Yu- Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. International conference on machine learning, pages 3122-3130. PMLR, 2018.
[36] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. arXiv preprint arXiv:2304.080
2023.
[37] Adian I Liusie, Potsawee Manakul, and Mark JF Gales. Mitigating word bias in zero-shot prompt- classifiers. arXiv preprint arXiv:2309.04992, 2023.
[38] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
[39] Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings the AAAI Conference on Artificial Intelligence, volume 35, pages 14867-14875, 2021.
[40] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07/88 2022.
[41] Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly- labeled reviews and fine-grained aspects. In Proceedings of the 2019 conference empirical methods in natural language processing and the 9th international joint conference on natural language processing DEMNLP-IJCNLL ), pages 188-197, 2019.
[42] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008
[43] Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg. Chils: Zero-shot image classification with hierarchical label sets. In International Conference on Machine Learning, pages 26342-26386 PMLR, 2023.
[44] Omkar M Parkhi, Andrea Vodaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pages 3498-3505 IEEE, 2012.
[45] Javier Pena, Juan C Vera, and Luis F Zuluaga. New characterizations of hoffmman constants for systems of linear constraints. Mathematical Programming, 187:79-109, 2021.
[46] Javicr F Peña. An casily computable upper bound on the hoffman constant for homogeneous incquality systems. Computational Optimization and Applications, 87(1):323-335, 2024.
[47] Hanyu Peng, Mingming Sun, and Ping Li. Optimal transport for long-tailed recognition with learnalle cost matrix. In International Conference on Learning Representations, 2021.
[48] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 11(5-6):355-677, 2019.
[49] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[50] Nils Reimers and Tryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: 1004, 2019.
13

---

[51] Nicholas Roberts. Xintong Li. Dyah Adila, Sonia Cromp. Tzu-Heng Huang, Jitian Zhao, and Frederic Sala. Geometry-aware adaptation for pretrained models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[52] Stephen M Robinson. Bounds for error in the solution set of a perturbed linear program. Linear Algebra and its applications, 6:69-81, 1973.
[53] Filippo Santambrogio. Optimal transport for applied mathematicians, Birkäuser, NY, 55(58-63): 94, 2015.
[54] Liangliang Shi, Haoyu Zhen, Gu Zhang, and Junchi Yan. Relative entropic optimal trans- port: a (prier-aware) matching perspective to (unbalanced) classification, Advances in Neural Information Processing Systems, 36, 2024.
[55] Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey Gordon. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Information Procexsing Systems, 19276-19289) 2020.
[56] Kai Sheng Tai, Peter D Bailis, and Gregory Valiant. Sinkhorn label allocation: Semi- superviie classification via annealed self-training. In International Conference on Machine Learning, pages 10065-10075. PMLR, 2021.
[57] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pages 4149-4111110 4158, 2019.
[58] Nguyen Thai-Nghe, Zeno Gantner, and Lars Schmidt-Thione Cost-sensitive learning methods for imbalanced data. In The 2010 International joint conference on neural networks (IJCNN), pages 1-8. IEEE, 2010.
[59] Antonio Torralba and Alexei A. Efros. Unbiased look dataset bias. In CVPR 2011, pages 1521-1528, 2011. doi: 10. 2000/CVP..221.59955444
[60] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 22233002.9977 2023.
[61] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 20rd2-200-2011 dataset. 2011.
[62] Haobo Wang, Mingxuan Xia, Yixuan Li, Yuren Mao, Lei Feng, Gang Chen, and Junbo Zhao. Solar: Sinkhorn label refinery for imbalanced partial-label learning. Advances in Neural Information Processing Systems, 35:8 (01-8117, 2022.
[63] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in Neural Information Processing Systems. 32, 2019.
[64] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01653 2021.
[65] Sheng-Lun Wei, Cheng- Kuang Wu, Hen-Hsen Huang. and Hsin-Hsi Chen. Unveiling selec- tion biases: Exploring order and token sensitivity in large language models. arXiv preprint urXiv:2406.03000) 2024.
[66] Ruicheng Xian, Lang Yin, and Han Zhao. Fair and optimal classification via post-processing. 2023.
[67] Jianxiong Xiao, James Hays, Krista A Ehinger. Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to 2oo. In 2010 JEFE computer society conference on computer vision and pattern recognition, pages 3485-3492, IEEE, 2010.

---

[68] Chuyu Zhang. Hui Ren, and Xuming He. Ptot: Progressive partial optimal transport for deep imbalanced clustering. arXiv preprint arXiv:2401.0922 2024.
[69] Chujie Zheng. Hao Zhou, Fandong Meng. Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2023.
[70] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 13009):3377-244,, 2022.
[71] Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. Advances in Neural Information Processing Systems, 36, 2024.

---

Appendix
The appendix contains glossary, algorithm details, proofs, and detailed experimental results. The glossary contains a convenient reminder of our terminology (Appendix A). Appendix B provides related works and discussion ahout the relationship between our work and related papers. Appendix describes the relevant algorithms used our experiments, including Prior Matching [37] and BBSE [35]. Appendix D provides the proofs of theorems that appeared in Section 4. Finally, we give more details and analysis of the experiments and provide additional experiment results in Appendix E.
A Glossary
The glossary given in Table 6 below.
Symbol
Definition
7 K
Number of points
Number of classes
K
The set of classes (1, K}
Input feature space
Y
Label space
X
Input features
True labels
P.
Source (training) distribution of data Target (lesting) distribution of data Prediction score function with parameter 0 Bayes optimal cost matrix for prediction
P.
Se
C
Č
Estimate of cost matrix for prediction Class balance for true labels
2s
Class balance predicted labels the zeroshot model Additive perturbations to cost matrix Additive perturbation to class balance
Ac
A.
Optimal transport plan
G.9
Constraint matrix and vector for linear programming s.t. feasible set is for C X Ga: 2g}
Dual solution for linear programming problem
K
Hoffman constant for the true optimal transport problem
Positive parts of i.e. [z]] = x1 [ 0)
Negative parts of xr i.e. [ar]
x [ 0
vec(A)
Vectorized A, vec(A) [An.... Am A12...,, An2. Alm... Amn) for A A Ruxn
Table 6: Glossary
B Extended Related Work
Improving Zero-shot Classification at Inference Time. As zero-shot classification has gained popularity. several works have been developed to improve zero-shot classification at inference time. Chuang et al. [13]; Adila et al. [1] vector projection methods remove spurious correlations inference time. Menon and Vondrick [40]; Novack al. [43]; with language models and combine their classification output to improve zero-shol performance. Roberts al. [51] uses additional information of label space geometry to extend model pre- pre-trained on the label subset broader use-cases. While these works try improve zero-shot classification inference time in common, the main difference is that our method lackles the inherent class prior of zero-shot models.
al. [4] augment prompts
Label Shift Adaptation. Label shift adaptation methods are designed to address the negative impacts arising from changes in the label distribution. These methods typically follow two-step process [35; 6; 24]. The first step involves estimating the label distribution within the target dataset using labeled data from the source distribution and unlabeled data from the target distribution. Next, the prediction
reweighted using the estimated target label distribution and the souree label

---

distribution. However, the standard approach requires access to the labeled source distribution data, which is usually not possible in zero-shot classification scenarios. OTTER provides a solution decoupled from the source data distribution, overcoming this limitation.
Several studies have explored leveraging prior information to enhance zero-shot classification, even the absence of access to source distribu- tions. In the context of prompt-based zero-shot models, prior matching [37] employs word prior distribution to alleviate word bias inherent in pretraining data. We adopted their adaptation method as baseline. Similarly, Kahana et al. [31] develop adaptation layers trained using priors, aiming to maintain proximity to the original scores. However, both approaches entail training additional layers and necessitate hyperparameter tuning, which may pose challenges in the context of zero-shot predictions. In contrast, OTTER presents straightforward and efficient adaptation method to new label distributions without the need for any hyperparameter tuning, backed by theoretical guarantees.
Improving Zero-shot Classification using Prior.
Leveraging Optimal Transport for Enhanced Sheud-llbbeling and Classification. A number of studies have explored the enhancement of pseudo-labeling and classification tasks through optimal transport, using label distribution specifications, in similar spirit to our work, but within different contexts. Tai et al. [56] uses optimal transport to allocate pseudo-labels to unlaheled datasets based on the label distribution specification in the semi- supervupe setup. Wang al. [62] deals with long-tail distribution in the partial-label learning setup based on optimal transport. Zhang et al. [68] uses partial optimal transport as a pseudo-labeling based approach for deep imbalanced clustering, progressively expanding the labeled sample proportion. Guo et al. [26] reweights the training dataset to match the label distribution specification using optimal transport. This work mainly deals with the class imbalance problem in the training step. Shi et al. [54] studies classification from a matching perspective, revealing the connection between softmax loss and inverse optimal transport and suggesting a new loss function address long-tail distributions. Their analysis provides useful insights for OTTER- - why cost matrix induced by pre-trained models can be useful in the inference step. Xian et al. [66] uses optimal transport as a postprocessing method to guarantee demographic parity. While sharing aspects of the approach, our work addresses class bias in zero-shol models. Peng et al. [47] used optimal transport to handle long-tail recognition with a learned cost matrix. Our study provides theoretical basis for understanding their empirical results. Chang et al. [9] employs optimal transport detect common and private classes between the source and the target domain, under the universal domain adaplation setting, where knowledge is transferred from source to target domain without any constraints on the label sets. In the context of zero-shot classification, there is need to manage label space disparities between the source and target domains. Instead, the main concern zero-shot classification is dealing with the distribution shift between the pretraining datasct and the downstrcam task. We tackle the label distribution shifts using optimal transport.
Class Imbalance. Class imbalance problems occur when the number of instances across different classes in dataset is disproportionately distributed. This imbalancc can severely bias the traininig process of a machine learning model, Icading to poor generalization performance, especially for the minority classes. It has been extensively studied in the context of traditional machine learning [23]. Oversampling [10] and cost-sensitive learning [58] well-known approaches to address class imbalance. Nonetheless, the inherent nature of class imbalance in pretraining datasets introduces a distinct set of challenges, especially when attempting to rectify such biases within the context of zero-shot classification scenarios.
C Algorithm details
Prior matching [37] proposed prior matching as a reweighting method for prompt-based classifiers to mitigate word bias the distribution shift between pretrained language models' word prior and the class priors in the downstream lask. We use it as a zero-shol model adaptation method given a class balance estimation.
Define reweighted probability scores of Pe with as PY = j|X
x:))
Ideally, we hope to estimate the weight vector y" e R such that T, Po j| = x.)
Σ/ Po ='
reweighted scores Pe.. (Y = j|x x;)) maximizes the accuracy in the target distribution. Since the labels are not given, this is impossible. Instead, prior matching matches the label distribution of
17

---

Algorithm 2 Black Box Shift Estimator (BBSE)
Input: Source input data X fai 2m Source labels Y fui...., Fm Target input data X x' {zi..... J. model prediction distribution Po
> PolY = = =
Estimate the source class balance v² such that vg
m
Σi=1 Po j|x x/)
Compute the naive target class balance i/ such that 8/3
Estimate confusion matrix V such that A₁k 1 - Po = k/X = xf)
Estimate the refined target class balance of AE
Return
predicted classes with the class balance estimate D, i.e.
arg min
Σ
PY = x.) 22,2-111 Vj
EyER
It can be solved using the standard optimization techniques - we used [38]. While this equivalently effective with OTTER when properly optimized, we found that the temperature parameter and learning rate crucially affect the final accuracy, making it less ideal for the zeroshot adaptation. We used the grid search with the small validation set (10 samples per class) in each task select hyperparameters. The hyperparameter ranges are as follows.
Temperature: [le-3, le-4, le-5, le-6, le-7]
Learning rate: [1c-3, Ic-4, 1c-5, 1c-6, lc-7]
Black Box Shift Estimation (BBSE) Label shift adaptation methods [35; 6: 24] aims estimate the class balance in the target distribution using the labeled source distribution data and the unlabeled target distribution data. We use Black Box Shift Estimation (BBSE) to estimate the class balance in the downstream task. Algorithm 2 deseribes the procedure. Note that the derivation of this algorithm depends on the label shift assumptions, thus the label distribution estimation with real-world data can be heavily biased.
Theory details
D.1 Proof of Theorem 4.1
Proof. Suppose j.T j 07 for some E [n]. It implies Σn - log Po(Y = 0,01xx X = n.) <
E - llog PY = 072 x = 2.). However, this is a contradiction since, for any i E [n], 1
arg max PY -|X xz). thus - log Po = "/ = x;)) S log Pa j|x = I:) for all j [K]. which results in j/TT 0.5.
D.2 Proof of Theorem 4.2
To prove Theorem 4.2, we show specific form of invariance property optimal transport first. Theorem D.I. Suppose x arg min (rs C) and is columnwise perturbation, i.e.,
E [c1 c21 ...cx]],
where 1 denotes n dimensional vectors and c....., CK are constants. Then the perturbed cost matrix δ C=C+ E, then x² also an optimal transport map with respect the cost matrix Č.
Proof. By the optimality condition, we have
Σ
Tico S
EETCO
for any E I(,,v) v). Then.
K n Ci + Σ
K Cij + Σ
Σ
Djej S Σ
Uyegs
1=1
igj
j=1

---

which
K n
K n
J - Σ
Fi Ci
TijEy-
j=l i=1
j=l i=1
Thus,
Σ 1
Cij $ Σ
Cij. Cij.
Kiy
10
This theorem is also valid for row-wise perturbations as well with a similar proof. Consequently, straightforward implication is that
Corollary D.2. Suppose = arg min (r' C)2 E a columnwise perturratiin and a row-wise perturbation, such that
E 8= le
E2
? F
1/1 T
where 1 denotes dimensional vectors with 1x, and E1,--., KR m..... VKK constants. Suppose
the perturbed cost matrix is defined as Ò = C' E 4 F, then x² is also optimal transport map with respect to the perturbed cost matrix C.
Now we provide the proof of Thocrem 4.2.
Proof. Given
C₁5 =
log Pa = j| = x4) log P. = j|x X = æ;),
the posteriors in the target distribution can be defined as Cū₁ = log P.(Y = j| = x:). From
P.(X = P,) =)
P.(Y = = 2;) = P. = j| = xi) Pi(X Po(Y =
we can see that
C₁ = - log P?Y = j|X = x;)
log P.(Y =]|X xx) P.(X P.)
P:(X P. =
log P.(Y j|X =x,) + P.Y
log P.(YY = -) log Pa(X = r;)) + log P:(X = xz)
Cij E.j F,.
where E.,- log PY 5) log Pt f),
log P.(X x,)) log P.(X xx). And, Σr [5 j]. where j is the Bayes classifier prediction such that
assuming V/j
d arg max P.Y i|x x.:) JE[K]
min log Po = -|,' = =::). ][[K]
Then, optimal transport solution
arg min (7, C*) (Ell())))
leads Bayes classifier predictions by Theorem 4.1. Finally, by Corollary D.2, we have
π arg
min
(T,, C*) arg min (%, C). ?ell(au))
DE ell((

---

D.3 Proof of Theorem 4.3
The proof of Theorem 4.3 relies on following result of [52].
Lemma D.3 ([52), Corollary 3.1.). Let the primal linear programming problem be
miGz =|G= g.=>0
and its dual be
max glw" G < > 0}.
Let 2, i be the primal, dual solution. And, let the perturbed primal linear programming problem be
min(p"2 GZ 2 9. > 0}
and its dual be
max(w² glwo" & p² 0}.
Let 2, u be the corresponding primal, dual solution.
Suppose that the primal and dual problems are solvable. Then,
GG - G) (g - g)
2 i -
S K
G)= à (p p)
(p - D) (g - 9 m
where 1 S p 00 and is the Hoffmann constant determined by ps G.g. [30].
This Lemma provides bound for error in the solution of the perturbed linear program. Since discrete optimal transport be translated standard linear program, we obtain Thcorem 4.3 by plugging the definitions.
Proof of Theorem 4.3 A discrete optimal transport problem
min
f = us Ti 2
can written as a linear program
min(p=2|Gi 2 0}.
11
1 Ix I, 1
11
where p= vec(C), = vec())
G
9
Note that the equality
P
constraints are converted to stacked incqualitics. We have noisy cost matrix and label distribution in our setting, which leads to the perturbation on cost matrix C' and v such that the perturbed cost matrix is Ć Ac, the perturbed label distribution & v Av, such that و = g + A, where
0
D
Since we don't have perturbation on the constraint matrix G, & = G. By plugging
g
- m
in these terms Lemma D.3.
20

---

[[G G) - (g 9)
[G - G) (p
(p - p) (g 9
9 9
p P
0 g)(0
0
18 11
Optimality of 6 in the perturbed problem.
47 0
p2 9m
10 V
8
D
Then, we have
[6 1/
22
P
pp gw
Then,
16 vl
= a|B v x²
[p - p
100 wl
p2. g
Plugging in vec(=), vec 4 = Dc C
C. and rearranging, we obtain
|| 1| S 2 (1 + [[vec( 11 122+ l|vec(C)) veo(fi) g" (1 | 3l2
We use definition of Hoffman constant & from [52]. Computing Hoffman constant or even bounding it has been Long-standing problem [5: 45; 46). However, it has been shown that the Hoffman constant is finite real number [52]. and specifically under problem setup, it is independent from perturbations and only related to original optimization problem. This suggests the possibility to regularize the parameters C, G, in the original problem such that & does not depend the dimensionality of cost matrix target distribution. We leave further exploration of in this context as future work.
D.4 Bounding label distribution estimation errors in few-shot learning
few-shot learning, we assume that few labeled samples per class are given. They can be used for estimating label distribution in the target distribution using label shift estimation methods [35; 6; 24]. They comes with the sample complexity analysis under the label shift assumptions, which can be used to obtain bound the label distribution estimation errors.
Lemma D.4. Let m and n denote the number of few-shot learning data and test datapoints, wi P//V and ú; Dz/vE. Also let aiin be the smallest eigenvalue of the covariance matrix Vi where Vo (f))())= = i. 1). For m > 80 log( m) in)oai mm and constant >> O, the perturbation Ap
may be bounded as
log
12 ||0 σ²
+ K
mio
m
n
with probability least 3/m
10 2Kn 10.
The proof of Lemma D.4 relies on the following result of [34].
Lemma D.5. Assume that
Vz X and Vy E Y,, (x(ly) (=
2. if P.(y)) then (v) Vy and
21

---

3. the expected confusion matrix C.(f) (.((())))) 8) E RK* for classifier f invertible.
Then, there exists a constant C O such that for all m > 80 log(m)o min with probability at least 3Km-1u 2K
2
(1w01 log m
log
w|l2 S
K
Proof of Lemma D.4. Where all norms are Euclidean unless otherwise denoted, we have that
||
||0 vil
1| (w w)ll
for element wise multiplication operation @. Further,
|| (w
w)ll < 10 (u - (w-=))
- 1" ||2| wil2
10ll c
[110|]22 log m
log
K
m
n
where the last line follows from Lemma D.5 with probahility least
3km 10 2Kn 10
D.5 Bounding the perturbation cost matrix
Further, we can bound [uec(Ac)) by the Total Variation Distance (TVD) between P. and Pg as follows.
Lemma D.6. Let = IP, Polli denote the Total Variation Distance between P. and Pa and define min(C) = min., Ciy. Then,
[|[ue((Ac))+11 /m(K 1) log min(C') + +1)
Proof of Lemma D.6. For each element A (E)) of Ac, have that
A log Po(Y = j| = x;)
log P.(Y = =]]X- = x;)
Po = i| x;)
log P.(Y = =i(X = xz))
For i.j such that Po j|X 2;) S P.(Y = =|XX- = r;). clearly A < 0 and so
[vec( A (())) |]+ =0.
Otherwise, for j such that Po(Y =||X x1) > P.(Y j| xx), it follows that (+) 0
and
PolY = =]|X- = x;)) P.(Y = == 2x) PAY = -| x;)
log
P.(Y = =j =xx=,,))))
log
P.(Y -|x== x.)) +1
min(C) +1).
< log
For each E [m]. there j|X 2.), because Σj(()) Po = j|X X = x.) = ΣScr P.(Y = j|X = z,). Therefore, there
K 1 possible such that Pa(Y = P.(Y
are al most m(K- - 1) pairs (i, & [m]] x [K] such that 0 < (1)
(9) log [ + 1). Thus,
[|[vec(Ac)++11 / mm(K - 1) min(C") + +1)
22

---

D.6 Proof of Theorem 4.4
We provide the proof of Theorem 4.4.
Proof. By assumption, we have Po X) = X.) By the result of Theorem 4.2, yOTTER samples are generated by arg ma x Pr(Y = = = z). Suppose r" = P((Y)/P,YY) Then,
P3Po = -|
Po.- (Y= = =||x = x)
> -3 =
-((x= PX
P, = =|| =x)
> P3..T... = =
P.YY=) P.X=:
P P.
P(Y= P
1 P((Y PXX
P,(Y =
y== P.(Y =))
K
P.(Y -))
y P.(Y =X)
K
X ((---
ATY=))
K
P.Y PY JI 2) 2]
y==
P()
P( -|XX =x)
K -/==1 Pi(Y j x)
Thus, we have
YR-OTTER = arg max Po. (Y = 5|XX x))
P((Y = = x)
= arg max Σ/= P.Y = --(x==)) =x) x)
arg mAx P.(Y = x)
BOTTER,
which implies r" is an optimal parameter.
23

---

CIFARIO
CIFARI00
Caltech101
Caltech256
Food101
STL10
10,000
10,000
7.162
22,897
25,250
8,000
K
10
100
101
256
101
10
Imbalance
1,00
1.00
49.06
15.94
1.00
1.00
SUN397
10owers102
FuroSAT
Oxford-IITT-Pt
STANFORD-Cars
Country211
87,004
6,149
22,000
3,669
8,041
211,00
K
397
102
10
37
196
211
Imbalance
25,43
11.90
1.67
1.14
2.83
1.00
DTD
CUB
ImageNet
ImageNet-r
ImageNet-Sketch
1,880
5,794
40,000
26,000
40,889
K
47
200
1,000
200
1,000
Imbalance
1,00
2.73
1.00
13.23
1.03
Amazon
Gender 21,750
CivilComments 131,782
HateXplain 1,621 3
89,078
K
2
Imhalance
19.45
6.03
8.26
1.52
Table 7:
Statistics of the test dataset in each task. maxze[K] P.Y j)
Class imbalance is measured by
minje[k] P:(Y = j)
Experiment details
E.1 Datasets
Zeroshot image classification datasets We use CIFARIO, CIFARI00 [33]. Caltech101 [22].
Caltech256 [25], Food101 [8], STL.0 [16], SUN397 [67], Flower102 [42], EuroSAT [27], Oxford-
IIIT Pet [44], Stanford Cars [32], DTD [14], CUB [61], ImageNet [18], ImageNet-r [29], and
ImageNet-Sketch [63].
Zeroshot text classification datasets We use Amazon [41], Gender [20], CivilComments [7], and
HateXplain [39].
24

---

CIRARIO
CIEARIO
Caltedhi))
Callech256
Country111
EUROSAT
Howwrt
DTD
Zero-shot
66.5
38.7
73.5
73.0
13.3
37.5
18.2
57.4
RN50
CLIPPR
69.2
20.1
89.4
67.1
8.2
13.0
19.3
30.7
Prior Matching 76.4(=:0)
41.1(+22.5)
44.3(=9.7)
9.8(+11.6) (3.11(#:0.6)
41.6(:0.2)
31.5 (#6.0)
50.0(4:2.8)
OTTER(Ours)
76.8
44.4
83.7
80.5
14.1
42.2
26.8
64.3
Zero-shot
79.1
46.3
81.0
76.6
14.8
37.3
33.8
61.3
CLIPPR
80.4
24.1
52.5
65.4
7.6
5.9
37.1
30.5
Prior Matching 80.3 (d:0.1) 1) 46.7
(3:1.3)
57.6
(4:14.2) 9.6
(d:2.4)
15.1
(4:0.7)
39.4
(d:0.1)
34.0 (d.5.9)
49. (4:20.9)
OTTER(Ours)
$1.1
50.9
89.4
83.7
16.0
40.7
33.0
67.7
Zero-shot
88.9
58.5
80.3
78.2
15.5
40.7
29.7
59.3
ViT-B/52
CLIPPR
89.9
37,8
55.5
69.3
8.5
8.4
31.8
29.8
Prior Matching 89.7 (#:0)
58. I (#1 (2) 54.7
(#:14.1) 9.5
(#:1 6)
14.8 (::0.) 1) 42,9 (#:0.1 13 41.2
(#:077
31.5 (4:32)
OTTER(Ours)
89.7
64.4
88.0
85.0
15.9
45.1
44.9
68.1
Zero-shot
88.3
63.8
79.8
79.8
19.8
39.0
32.9
64.0
ViT-R/16
CLIPPRR
90.3
41.6
60.3
75.4
12.0
9.1
36.4
36.2
Prior Matching 91.3 (d:0)
64.1
(1:2.7) 7) 59.3
(d:15.4) 9.5 (d:1.) (.)
19.0
(1:0.
1) 42.1
(4:0.1) 41.6
(d:0.8) 54.0
(d.14.1)
OTTER(Ours)
88.7
21.1
70.8
Zero-shot
95.0
72.3
78.2
83.4
28.2
50.8
25.8
72.3
VIT-L/14
CLIPPR
96.2
57.9
66.8
80.2
17.4
12.8
31.5
44.3
Matching 95.2 (4:0)
(4:2.5) 68.5
(#19.7) 9.4 (# 27.5
(4:03)
51.5 (=:)
(#:0.4) 66.5
(#17.9)
OTTER(Ours)
Table 8: CLIP Zero-shot image classification accuracy (%)
E.2 Zero-shot classification setup
For zero-shot image classification, we emply CLIP [49] models. We used "a photo of a [CLASS]" prompt. Scores are computed by so so(r;. Po j| .x)
Image zero-shot classification
exp (())s((()))
for image x, regarding the label j. given the image encoder f. ()T)
-y-=1 exp
the text encoder Cost matrix is constructed by C [Cy]een]]]]]]] where oy log se(x..)))
We run Algorithm 1 with the true class balance of the test dataset.
Text zero-shot classification We employ BERT and rext-embedding-adad ada-0002 sentence embeddings for text classification [50]. This process parallels the methodology used in image zero-shot classifi- cation we compute the prediction scores from the cosine similarity and then construct the cost matrix with the negative log probabilities.
E.3 Detailed experiment results of Section 5.1
Ablation on zero-shot models For the ablation study on zero-shol models, we provide experimental results with varying architectures, given the exact prior. Table 8, 9 show the image zero-shot classification results, and Table 10 shows the text zero-shot classification results. We also provide another baseline results with CLIPPR [31], which uses the label distribution for adapter training. CLIPPR is similar with Prior Matching in the point that it requires adapter training, but it has more adapter layers and additional loss function make the predictions stick to the original prediction scores. While the performance gain varies, we can observe that OTTER is effective for the most cuses.
25

---

Imagene Sketch
IanggeNotr
Food101
Stanfond
ImngoNet
SUNS
Pot
STL
CUB
Zero-shot
76.3
80.5
45.6
39.8
51.5
35.0
RN50
CLIPPR
69.5
61.9
31.1
23.8
Prior Matching (#1. 1) 77.4 (#0.3)
27.5 (#1.9)
94.7 (#:0)
12.3 (#3.9)
(#0.1) 44.6 (=0.1) (+1.88
5.0
OTT (R(Du)s)
81.1
82.9
49.2
95.5
42.7
54.0
37.6
Zero-shot
80.2
34,9
53.4
RN101
CLIPPR
72.3
57.7
97.2
25.7
Prior Matching 80.8 (1:3.7) 77.2
(#0044
34.8
(#:2.9)
96.7 (4:66
17.6 (1:5)
33.6 (#:0.1
46.7 (=0.1)
9.5 (4:3.1)
6.5 (2:01)
OTTER(Ours)
84,4
84.2
55.2
97.2
44.6
41.0
56.0
44.5
7,3
sero-shot
80.2
81.7
49.1
97.1
39.1
39,9
55.6
61.0
34.2
VIT-B/32
CLIPPR
73,9
58.0
30.6
97.5
41.6
11.9
32.6
51.7
19.9
Prior Matching 80.0 (#1. 77.5
(=0.6)
31.5
(#1.1)
97.4
(#:0.0)
11.3 (#3.1)
36.3
(+0.
48.3
(=0.1) 11.7
(=0.8) 30.4 (=0.3)
OTTER(Ours)
85.0
86.2
52.1
97.8
44.7
45.4
57.7
64.7
39.3
Zero-shot
85.6
83.8
55.7
98.0
47.1
46.0
60.2
68.9
39.8
CLIPPR
80.5
64.5
40.7
98.6
50.4
20.5
37.7
59.8
25.6
Prior Matching $6.8 (#:3.1 82.0
(=:0.)
39.8
(#:2.6)
98.4 (#:0)
6.7 (d:1.)
40.4 (::0)
53.6
(=0.1) 16.6
(=3.5) 36.5
(=0.4)
OTTER(Ours)
89.9
88.8
59.7
98.6
54.1
50.4
62.9
72.4
44,5
Zero-shot
89.8
87.9
64.1
99.2
64.9
47,3
67.5
80.9
$1.9
VTT-1114
CLIPPR
87.9
82.7
54.0
99.6
68.2
27.3
45.4
74.5
35.5
Prior Matching 90.4 (1338) 84.0
(.)) 53.0 (d: 15.6)
99.3 (d:0)
26.5
(.:1.1) 43.9 (1:0.3) 62.3
(=0.1) 17.6
(=: 1) 47.2
(2004)
OTTER(Ours)
93.6
91.0
70.0
99.4
71.5
55.4
70.2
83.7
55.2
Table CLIP Zero-shot image classification accuracy continued.
Amazon review
GenderBias
CivilComments
HateXplain
Zero-shot
74.0
84.1
48.4
30.9
CLIIPRR
74.9
84.2
49.4
31.0
BERT
Prior Matching
58.8 (#:: 46.4)
41.4 (+: 39.6)
$7.2 (+: 37.7)
31.3 (+:3.3
OTTER((Urs)
91.7
91.9
81.4
34.3
Zero-shot
72.3
50.9
56.2
27.9
CLIPPR
73.7
53.0
57.6
27.6
Ada
Prior Matching
58.8 (d:: 43.7)
50.0 (=41.3)
55.2 (#:5.55)
32.4 (1: 4.0)
OTTER(Ours)
97.0
73.7
82.0
32.0
Table 10: Text embedding zero-shot classification mean accuracy and standard deviation (%)
Ablation on the class balance specification We conducted a semi-synthetic experiment to in- vestigate the sensitivity to the label distribution specification error in real datasets. We generate the noisy label distribution specification and see how the accuracy changes. We control the noise the label distribution specification as follows. Given the true class balance v*, first we make adversarial class balance add such that uod" 1 for arg minje(k] Vý and rg = 0 for j # j To measure distance between class balance specification and true class balance,
the total
variance TV 0) = 2 1| u1l2. Next, we intepolate z/ and yad such that TV v²) = a, by 2a (1 TV parzt puady )v" + TV("", Jundey Tut"" vade, We set the interval of alpha as 0.01 and vary it up 0.2.
Figure shows the result. We observe the sensitivity to the label distribution specification error varies depending on the datasets, but generally we can observe that the accuracy degrades linearly propor- tionally to the class balance error. While the result may vary depending on the interaction between class balance error and calibration error in cost matrix, can expect performance improvement if the class balance specification good enough.
26

---

CIFAR10
CIFAR100
Caltech101
90
90
65
80
60
25
25
25
75
55
78
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.
0.00
0.05
0.10
0.15
0.20
Caltech256
Food101
STL10
80
70
Z5
25
25
OTTER
OTTER
OTTER
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
SUN397
Flowers102
EUROSAT
70
40.0
37.5
35.0
40
OTTER
50
OTTER
OTTER
0.15
0.20
0.15
0.20
32.0
0.05
0.10
0.15
OxfordiliTPet
Stanford-Cars
Country211
60
55
80
20
Acc.
50
19
75
OTTER
OTTER
OTTER
40
DTD
cub
ImageNet
60
40
45
55
Z5
35
50
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
ImageNet-r
ImageNet-Sketch
Amazon
70
Z5
42
65
00
Z5
30
25
36
0.00
0.05
0.10
0.15
0,20
0.00
0.05
0.10
0:15
0.20
0.00
0.05
0.10
0.15
0,20
Gender
CivilComments
HateXplain
70
80
---
25
60
25
32
25
75
50
OTTER
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10,
0.15
0.20
0.00
0.05
0.10
0.15
0.20
P:(M
Figure 5; Ablation experiment the class balance specification. X-axis represents the total variation distance between the class specification true class balance P. P.(Y)) and P.(Y). Y-axis represents accuracy. ViT-B/16 used as the image zero-shot classifier, and BERT is used the text zero-shot classifier.
Inference time comparison To show that the additional computation complexity induced by OTTER is not heavy, we measured the time consumption (in seconds) for the inference step in the experiments in Section 5.1, with the pre-computed embeddings. Table II presents the result. Time reduction column represents the time reduction rate OTTER compared to PM. Measurements were
27

---

Dataset
n
7$
PM
OTTER
Time reudction (%)
CIFARIO
10000
0.0381
3.7127
0.0733
98.03
CIFARIO 100
10000
0.0462
3.6296
0.1947
94.64
10
7162
0.0298
3.6445
0. 188
96.74
Caltech256
22897
0.2111
3.9597
0.8568
78.36
Food101
25250
0.1186
3.6968
0.4969
86.56
STI.10
8000
0.0304
3.4877
0.0546
98.43
SUN397
87004
1.1233
33.0386
10.5316
68.12
Flowers 102
6149
0.0280
3.7216
0.0959
97.42
EuroSAT
22000
0.0826
3,6655
0.3491
90.48
OXTORD-IIT--el
3669
0.0137
3.3901
0.0233
99.31
STANFORD-Cars
8041
0.0413
3.4910
O. 0914
94.37
Country211
21100
0.1285
3.7665
1.0537
72.02
DTD
1880
0.0070
3.4603
0.0156
99.55
CUB
5794
0.0306
3.5583
0.1410
96.04
ImageNet
40000
0.9954
37.6932
8.1003
78.51
ImageNet-r
26000
0.1921
3.8331
0,9834
74.35
ImageNNtt-Skttch
40889
1.0189
38.4853
9.0579
76.46
Table 11: Inference time comparison with pre-computed embeddings (in seconds).
7S
PM
OT
7S CD
PM CD
OT + CD
EuroSAT
32.90
11.36
42.03
53.62
11.37
57.15
Oxford-IITT-et
83.84
23.11
88.83
87.95
16.33
91.01
DTD
39.04
8.83
44.41
42.87
14.73
43.24
CUB 45.98
10.34
50.40
55.51
11.49
58.47
ImageNet
60.18
12.42
62.86
66.46
14.08
68.05
Table 12: Accuracy the prompl--chaneed zero-shot classification by Classification by Description
(CD) [40]. We can obscrve OTTER's 's capability to provide further enhancements upon refined the improvements achieved through refined prompts.
taken using machine equipped with Intel TM 17-11700K @ 3.60GHZ processor, 64GB RAM. and NVIDIA GPU RTX-4090. most cases (n < 30000), our method takes less than I second, while the prior matching baseline takes more than 3 seconds. It's worth noting that the time consumption for computing embeddings is more substantial; even 10 seconds is negligible compared the embedding time consumption (ranging from to minutes for each evaluation set), which common for all inference conditions.
Ablation on prompts Recent studies have demonstrated the efficacy of enhancing prompts means improve zero-shot models [70: 40]. In order to further illustrate the potential enhancements offered by OTTER beyond enhanced prompts, we reproduced Menon and Vondrick [40]'s approach (Classification by Description, CD), which employs multiple prompls generated by language models and takes scores of them for each class. We applied OTTER CD. The results of this experiment are summarized Table 12. As anticipated, OTTER exhibits enhancements zero-shot classification, even when prompt sets are refined using language models.
28

---

Dataset
TV(e", D.T)
TV(=". (Dn)
Dataset
TV(K". .0*))
TV(=", (P))
CIFARIO
0.071
0.038
STL10
0,021
0.011
CIFAR100
0.219
0.153
SUN397
0,503
0.458
Caltech 101
0.130
0.041
CUB
0.245
0.102
Caltech256
0.126
0.081
ImageNet
0.175
0.175
Country211
0.439
0.336
ImageNet-r
0.210
0.189
DTD
0,441
0.160
ImageNet-sketch
0.236
0.211
EUROSAT
0.404
0.084
Amazon
0.090
0.253
Flowers I02
0.202
0.067
CimilComments
0.369
0.383
Food101
0.112
0.090
Gender
0.083
0.155
Oxford- 0.219
0.114
HateXplain
0.253
0.203
Stanford-Cars 0.255
0.143
Table 13: Class balance estimation error with BBSE Section 5.3. We report the mean of different random samplings of the validation set. Lower is better.
E,4 Detailed experiment results of Section 5.3
Label distribution estimation errors We report the label distribution estimation errors Scction 5.3. As metric, we use the total variance TV (v, D)) = || 01|,- We use zeroshot prediction scores and linear probing prediction scores for BBSE. D=* denotes the estimated class balance based on zero-shot prediction scores, L represents the estimated class balance based linear probing prediction scores.
Table 13 shows the result. We can see that total variation decreases with linear probing in image classification tasks since they reduces the violation of label shift assumptions. However, total variation increases text classification tasks due to the small number of labeled sample size, following the size of label space (K = 2 or 3). Accordingly, we can expect OTTER will be more useful with linear probing, and just rebalancing zero-shot predictions with OTTER could be enough for text classification tasks.
Ablation experiments on linear probing We provide full results of Section 5.3. Specifically, we additionally report the results of combination with linear probing in text classification tasks and the results of zero-shot classification results image classification tasks.
The results are presented in Table While OTTER often provides additional improvement over LP, zero-shot classification was strong baseline in image classification tasks. Mcanwhile, class balance adaptation in text classification tasks is effective in all cases, giving significant improvement over zero-shot predictions.
29

---

Estimation Error
Estimation Error
Estimationn Error
Estimation Error
7wp.m. P499
0.20
E0.425
E0.00
0.3
0.15
E0.06
0.377
0.2
0.40
0.350
Accuracy
Accuracy
Accuracy
Accuracy
65.0
25
OTTER
300
62.5
10
80
60.0
OTTER
16
OTTER
OTTER
25. 50
75
100
25
50
75
300
25 50
100
25 50
75
100
SSmppics per class
Sampies per
Samples class
Samples class
(a) CIFARI00
(b) Country211
(c) Amazon
(d) CivilComments
Figure 6: Ablation experiment the number of samples. We report the of 10 different samplings in each setting. We use ViT-B/16 for image classification, and BERT for text classification.
Accuracy
Accuracy
Accuracy
Accuracy
24
70.0
22
90-
OTTER
70-
0 67.5
2.20
05
LP4OTTER
95
OTTER
$ 65.0
25
60
OPT
62.5
OTTER
CITEER
UP
14
LP4OTTER
LP4OTTER
50-
75
100
12
40
8b
100
Samples class
Samples per clsss
*5amples per class
SSmppees per clMs
(a) CIFARIO0
(b) Country211
(c) Amazon
(d) CivilComments
Figure 7: Comparison between OTTER, LP, and LP+OTTER with varying the number of samples. We report the mean of 10 different samplings in each setting. We use ViT-B/16 for image classification, and BERT for text classification.
Ablation experiments on the number of examples per class Fow-shot adaptation scenario assumes we have access to labeled data to estimate the target distribution. We hypothesize that an increase in the number of labeled samples enhances the accuracy of the class balance estimation. thereby improving the performance of OTTER. To test this hypothesis, we use few-shot adaptation in image and text classification tasks, without lincar probing. The experiment varies the number of samples per class from 10 to 100, anticipating reduction in class balance estimation error and an improvement in OTTER's ' accuracy with the increase in labeled samples.
The results, as depicted Figure 6, corroborate our hypothesis. It is evident that the error class balance estimation diminishes with an increasing number of samples, Icading to subsequent enhancement in the accuracy of OTTER.
Comparison between OTTER and Linear Probing with varying number of classes In the few-shot adaptation scenario. we explored three approaches: OTTER, linear probing (LP), and a combination of LP OTTER. We formulated two hypotheses. The first posits that OTTER might outperform LP, particularly situations with a limited number of samples. The second hypothesis suggests that OTTER could provide further enhancements to LP even when LP already surpasses the conducted using the same setup the previous
naive version of OTTER. This experiment
The results, displayed in Figure 7, reveal several insights regarding our hypotheses. To begin with, OTTER demonstrates performance on par with LP, especially in scenarios with fewer samples. Interestingly, OTTER achieves superior accuracy compared to LP in datasets like Amazon and CivilComments, characterized by a small number of classes (K- 2), resulting in relatively low total sample count. Furthermore, is observed that incorporating OTTER into LP lcads to an average increase in accuracy.
E.5 Detailed experiment setup and results of Section 5.4
Class hierarchy We used the following superclasses and subclasses classes for the proof of concept.
30

---

BBSE
H-BBSE
RNS0
0.335
0.246
RN101
0.378
0.294
VIT-B/32
0.156
0.167
VIT-B/16
0.287
0.246
VIT-1/14
0.131
0.152
Table 14: Class balance estimation error in the Section 5.4 experiment. Class balance estimation error is measured by total variation distance. We report the mean of 10 different samplings of the validation set. H-BBSE denotes the class balance estimation using hicrarchy upon BBSE.
90
Acc(%)
80
Naive
Bayes
70
OTTER
R-OTTER
0.0
0.1
0.2
0.3
0.4
0.5
0.6
(VP((())PssM))
Figure 8: Synthetic experiment result with R-OTTER. As expected, R-OTTER can successfully resolve the effects of label shift.
fish: aquarium fish, flatfish, ray, shark, trout
tree: maple tree, oak tree, palm tree, pine tree, willow tree
Class balance estimation error We report the class balance estimation error in Section 5.4. Table 14 shows the total variation between true class balance and estimated class balance. We can expect significant accuracy improvement for RN50, RN101, and ViT-B/16 based this table.
E.6 Synthetic experiments with R-OTTER
We validate our theoretical result (Theorem 4.4) by testing R-OTTER fully controllable setting.
Setup and Procedure. We use our synthetic experiment setup (Section 5.2) with perturbation noise 8 = and label distribution
0. Additionally, generated a validation set that follows the same distribution the test set. After learning the reweighting parameter
the validation set using Botter as pseudolabels, we evaluated R R-OTTER on the test set, comparing the results to those of zero-shot and OTTER. We expect that, if successful, R-OTTER will similarly gain improvement over zero-shot when the source and target distributions increasingly differ.
Results Figure presents the experimental result. As expected, R- R-OTTER presents the experimen- tal results. As expected, R-OTTER performs closely to the Bayes optimal solution, demonstrating its effectiveness, though exhibits slight suboptimality due generalization issues.
31

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency. research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how answer these questions. For each question
checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible the reviewers, area chairs, senior area chairs, and ethics reviewers. You will asked also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist While "[Yes] is generally preferable to "[No] ", it perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we
of the factors in their evaluation.
unable find the license for the dataset we used"). In general, answering "[No] or "[NA] not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, please just use your best judgment and write a justification elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided appendix. you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We substantiate every claim made in the introduction in the experimental result.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
32

---

Justification:
Yes. discuss the potentials and limitations across several sections.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-spccification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested runs. general, empirical results often depend on implicit assumptions, which should be articulated.
on a few datasets or with
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to- tottttootettttteee system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Yes, included the full of assumptions and complete (and correct) proof in Scction 4.2.
Guidelines:
The answer NA mcans that the paper does not include thcorctical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs either appear in the main paper the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information nerded reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
33

---

Justification:
provided detailed exhaustive experimental hyperparameters and setup.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not he perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access the model is limited in some way (e.g., to registered users), but it should possible for other rescarchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We submit zip folder of our code, and will release github link upon publication.
Guidelines:
The answer means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines guides/ Codeeuumi ssionPo seiionooicce oolicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers canmot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //tp///nps....////// ee/ public/ge weliic//uiideee wet.giiideeeeeeuueeeeeeeee ssi owwooonooioooooo ccy ) for more details.
The authors should provide instructions data access and preparation, including how to access the data, preprocessed data, intermediate data, and generated data, etc.
34

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Docs the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we provide all hyperparameters in the Appendix. We provide all data split and details in the main experimental result section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core of the paper to level of detail that necessary to appreciate the results and make sense them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, provide error bars much as possible, unless repeating experiment is expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error hars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma crror bar than state that they have a 96% CI, if the hypothesis of Normality emror is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
35

---

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: Provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full rescarch project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the rescarch conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://neuri.. http:///rrriisseeeppillleele/e/t
Answer: [Yes]
Justification:
adhere to the NeurIPS code ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [No]
Justification: In our knowledge, there is no additional societal impact beyond that of LLM's. This paper presents work whose goal is to advance the field of Machine Learning. There many potential socictal consequences of our work, none which we feel must be specifically highlighted here. As of now, we are not aware of additional potential socictal impacts beyond the typical implications associated with zero-shot models.
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used generate deepfakes for
36

---

disinformation. On the other hand, it not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the techmology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation stralegies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release new pre-trained model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards allow controlled use of the model, for example by requiring users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite all dataset and model original owners.
Guidelines:
The answer NA mcans that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper sww choode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged. both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
37

---

Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the puper does not release new ussets.
Researchers should communicate the details the datasct/codemmoodll as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sercenshots, if applicable, as well dctails about compensation (if any)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information supplemental material is fine, but the main contribution the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble). such the institution conducting the review.
38


Reviews:
[{"review_id": "TqL286Cdxg", "reviewer": "Reviewer_t3Bz", "summary": "Zero-shot models such as CLIP suffer from imbalanced predictions inherited from the uncurated pre-training data. This paper proposes adjusting the predictions of zero-shot models via optimal transport. This method requires only the downstream data and the label prior of the target distribution. Under the assumption that the conditional likelihood is unchanged between the pre-training data and the downstream data, the authors prove that the induced predictions are Bayes Optimal. The authors validate the effectiveness of the proposed method on several benchmark tests.", "questions": "Please refer to the weaknesses, and I am willing to raise the score if the major weaknesses are addressed.\n\nAdditionally, I have a question to discuss with the authors. Both [3] and OTTER use downstream data to generate Bayes optimal predictions based on the assumption that $P(x|y)$ remains unchanged between the pre-training distribution and the downstream distribution. In practice, this assumption might not hold. For instance, the pre-training data might mostly consist of real photos, while some test benchmarks might be sketches, e.g., ImageNet-Sketch. Under this situation, is your method theoretically sound in mitigating the pre-training bias?", "limitations": "Limitations have been discussed and no negative societal impact has been identified.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-organized and the presentation is clear. The idea of using optimal transport to adjust zero-shot predictions on downstream data is novel to me. Besides, the authors prove that the resulting predictions are Bayes optimal and justify the cost matrix using posteriors of zero-shot models. I have coarsely checked the proofs and find them convincing.", "weaknesses": "Major:\n\n(1) Related works are missing. Some papers have addresses the issue of removing the label bias in zero-shot models. For instance, [1][2] use a subset of the pre-training data to mitigate the bias. [3] focuses on only using the labeled downstream data to address imbalanced predictions and proposes two methods. The authors should consider including these prior works.\n\n[1] A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models.\n\n[2] The neglected tails of vision-language models.\n\n[3] Generalized Logit Adjustment: Improved Fine-tuning by Mitigating Label Bias in Foundation Models.\n\n(2) The proposed method is transductive and requires the entire test dataset to perform optimal transport. I am wondering if OTTER can estimate the label distribution of pre-training data $\\pi$ from unlabeled downstream training data and then the debiasing procedure can be performed by subtracting the logarithm $\\ln \\pi$ from the zero-shot predictions, as described in [3] (https://arxiv.org/pdf/2310.08106), without needing to operate on the entire test dataset\n\nMinor:\n\n(1) Duplicate citations: [29] and [30].\n\n(2) Line 120: should reference Algorithm 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their kind words, constructive feedback, and useful suggestions. We appreciate the reviewer recognizing the novelty of our work and its strong theoretical foundation. We plan to include the suggested related works and fix typos. \n\n* **On additional related works (Major 1)**: Thank you for providing more related works. We plan to include the suggested papers in our related works with the following discussion.\n[1, 2, 3] address the bias induced by the concept distribution in pretraining data, in a similar context as OTTER.\n[1] tackles word frequency bias in pretraining and spurious concept frequency bias in test data in prompt ensembles. This work normalizes/debiases logit scores with the logits from the pretraining data to reweight/select prompts.\n[2] addresses concept frequency bias in pretraining data as well via retrieval augmented prompting, i.e. retrieving the most frequent concept from synonyms of the label. First, they propose a new concept frequency estimation method with LLMs. By concept frequency estimation, they re-confirm long-tailed concept distribution, and they propose retrieval-based prompting/linear classifier training.\nSimilarly, [3] adjusts logits to debias pretraining / fine-tuning label distribution. They propose a pretraining label distribution estimation method with an optimization formulation and show that the proposed adjustment make the finetuning.\nThe main distinction of OTTER from these works is that it does not require access to the pretraining data distribution, as long as the label distribution specification is properly given. This is particularly advantageous because pretraining data is often inaccessible due to proprietary or privacy issues.\n\n* **On the comparison with [3] (Major 2)**: [3] tackles a similar problem in the sense they try to mitigate bias from pretraining, but **there are several key differences**: 1) [3] mainly considers fine-tuned models and uses ensembles of these fine-tuned models and zero-shot models, while our work **focuses exclusively on zero-shot models**. 2) [3] assumes a uniform label distribution in target distribution for their Bayes classifier result. If we make such an assumption, OTTER can achieve the Bayes classifier performance **without** access to the pretraining distribution. [3] presented an extension for label shifts in the target distribution in its appendix, which requires the target label distribution as well. Also, while our method often uses the entire test set, batch optimal transport can be applied in the case the large-scale inference is required (as we described in the common response). In practice, applying OTTER to inference set with $n<20000$ usually takes less than 1 second, as can be seen in Appendix E.3. Table 10. We added additional experimental results with the batched version of OTTER (randomly partition test data and apply OTTER with the global label distribution specification) in Figure 2 (attached pdf). The **results show that batched version still improves accuracy, enabling parallel processing**. We expect batched OTTER can be further improved with more recent works on batched optimal transport [4, 5].\n\n* **On the label shift assumption that P(X|Y) does not change**: If P(X|Y) changes, the invariance result does not hold (it is the same for [3]). Instead, our error bound in Section 4.3. provides an accuracy bound. Suppose the posterior ratio is given by $\\epsilon_{ij}=\\frac{P_t(X=x_i|Y=j)}{P_s(X=x_i|Y=j)}$ in the setup of Appendix D.2. Proof of Theorem 4.2. Then, the proof lines change as follows \n\n$C^*_{ij}$\n\n$= - \\log P_t(Y=j|X=x_i)$\n\n$= - \\log \\frac{P_t(X=x_i|Y=j)P_t(Y=j)}{P_t(X=x_i)}$\n\n$= - \\log \\frac{\\epsilon_{ij}P_s(X=x_i|Y=j)P_t(Y=j)}{P_t(X=x_i)}$\n\n$= - \\log \\frac{\\epsilon_{ij}P_s(Y_j|X=x_i)P_s(X=x_i)P_t(Y=j)}{P_s(Y=j)P_t(X=x_i)}$\n\n$= -\\log\\epsilon_{ij}-\\log P_s(Y=j|X=x_i)\\frac{P_s(X=x_i)P_t(Y=j)}{P_t(X=x_i)P_s(Y=j)} -\\log P_s(Y=j|X=x_i) + \\log P_s(Y=j)$\n\n$= -\\log\\epsilon_{ij}+C_{ij} + E_{\\cdot j} + F_{i \\cdot}$\n\nThis shows that when the label shift assumption is violated, OTTER with $C_{ij}=-\\log P_{\\theta}(Y=j|X=x_i)$ yields equivalent predictions with $C_{ij}^* + \\log{\\epsilon_{ij}}$ rather than $C^*_{ij}$, making it suboptimal. The similar deviation can be observed in the Proof of Proosition of [3] by plugging $\\epsilon_{yz}=\\frac{P_t(z|y)}{P_p(z|y)}$ into Equation (28), (29).\nHowever, in our analysis, Theorem 4.3. shows the additional error rate linearly depends on the log ratio of $P_t(X|Y)$ and $P_s(X|Y)$ by defining cost matrix gap as $\\Delta_{C_{ij}} = \\log \\epsilon_{ij}$. Thus, if the deviation is not significant, **we can expect near-invariant results. In practice, as we can see in ImageNet-Sketch results, OTTER works well even when P(X|Y) invariance is violated**.\n\nWe appreciate the reviewer's consideration and are more than willing to address any further concerns. If we have adequately resolved the issues, we would be grateful if the reviewer could consider raising their score.\n\n[1] Allingham, James Urquhart, et al. \"A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models.\" ICML'23.\n\n[2] Parashar, Shubham, et al. \"The Neglected Tails in Vision-Language Models.\" CVPR'24.\n\n[3] Zhu, Beier, et al. \"Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models.\" NeurIPS'23\n\n[4] Nguyen, Khai, et al. \"Improving mini-batch optimal transport via partial transportation.\" ICML'22.\n\n[5] Nguyen, Khai, et al. \"On Transportation of Mini-batches: A Hierarchical Approach.\" ICML'22.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LR8GdBroyI", "reviewer": "Reviewer_M3dD", "summary": "Zero shot classification suffers from label distribution mismatch. This paper suggest to adjust pretrained model prediction via optimal transport. By showing the optimal transport prediction is equivalent to the bayes optimal classifier's output theoretically and good model performances on various experimental settings empirically, the paper shows the validity of its proposed method.", "questions": "- Considering each cell of the cost matrix $C$, it is a loss (or the classifier score) calculated from the pretrained network. It means the cost matrix is biased to the source data distribution. In that sense, how the optimal transport can be considered as assigning true label?\n- How can the true label distribution of the target, $\\nu*$, be accessible?\n- Can the author show more details for the equation of section 4.1? I cannot understand how to remove $P_s(X)$ and $P_t(X)$\n- In table 1, what happened to Caltech256 Prior matching? Why the performance is far worse than zero-shot?\n- In figure 2, Why OTTER shows acc increase when the total variation distance between the source and the target increases from 0.4 to 0.8? With larger distribution shift, acc is supposed to deteriorate. If not, the synthetic experiment may be poorly modeled.\n- How long does it take to apply OTTER? I suppose it will take quadratically proportional to the dataset size.", "limitations": ".", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- The problem is well formulated\n- Various experiment", "weaknesses": "- Novelty is limited. This paper suggests applying OT for managing label distribution shift. The class imbalance for pretrained network prediction is already well known, and OT has been already applied for various distribution shift cases. Therefore, naively applying OT for the biased pretrained model prediction is imaginable.\n- The proposal saying that Bayes optimal classifier can be derived through optimal transport is inadequate and too strong since it requires a true cost matrix. If $P_t(Y|X)$ is accessible, we don't even need to use the optimal transport.\n- Please link theorems of the main paper and proofs in the appendices for readability.\n- I don't think figure 2 shows OTTER remains unaffected. The acc is deteriorated from 90 to 7X, meaning about 15%p decrease.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thoughtful comments and references. We will include links to proofs and will clarify the statements corresponding to the reviewer's questions in our updated draft.\n\n* **On the proposal saying that Bayes optimal classifier can be derived through optimal transport**: The statement in L151 means OTTER does not cause harm when the cost matrix has the true target posterior $P_t(Y|X)$; we do not need $P_t(Y|X)$ to make OTTER work well. This result is connected to Theorem 4.2 in Section 4.1. Since the assumption in Theorem 4.2 makes OTTER with $C_{ij}=P_{\\theta}(Y|X)$ and OTTER with $C_{ij}^*=P_{t}(Y|X)$ equivalent, OTTER can achieve the Bayes optimal classifier performance. We will make this connection clear.\n\n* **On the interpretation of Figure 2**: In Figure 2(a), $\\delta$ represents noise in prediction scores, which affects calibration. Such errors may affect the performance of OTTER. *Without perturbation ($\\delta=0$), OTTER remains unaffected by the label distribution shift* (L265-266), achieving a Bayes optimal classifier. \n\nAdditional questions:\n* *Considering each cell ... assigning true label?*: Our theorem shows that even when the cost matrix is biased to the source data distribution, optimal transport can fix the bias from the source data distribution using the label distribution specification. It does not necessarily assign true labels---it yields better predictions when the label distribution specification is better than the label distribution of biased zero-shot predictions, given proper calibration. We validate this finding by observing the performance gain in our experiments.\n* *How can the true label distribution of the target, $\\nu^*$, be accessible?*: **This is not necessary**---see our common response and additional experiments. We also note that there are many label distribution estimation algorithms (e.g. [1,2,3,4,5]). **Any available method can be plugged into OTTER**. For example, we used BBSE [1] as a label distribution estimation method in the linear probing setting. \n\n* *More details for the equation of section 4.1?*: On Equation in Section 4.1. Equation in Section 4.1. mainly proceeds with Bayes Rule and invariance assumption $P_s(X|Y)$=$P_t(X|Y)$.\n\n$\\tilde{s}_{\\theta}$\n\n$=s_{\\theta}\\frac{P_t(Y=j)}{P_s(Y=j)} \\qquad \\because$ by adjustments\n\n$=P_s(Y=j|X=x)\\frac{P_t(Y=j)}{P_s(Y=j)} \\qquad \\because$ by the assumption $s_{\\theta}=P_s(Y=j|X=x)$\n\n$=\\frac{P_s(X=x|Y=j)P_s(Y=j)}{P_s(X=x)}\\frac{P_t(Y=j)}{P_s(Y=j)} \\quad \\because$ by Bayes rule\n\n$=\\frac{P_s(X=x|Y=j)P_t(Y=j)}{P_s(X=x)}$\n\n$=\\frac{P_t(X=x|Y=j)P_t(Y=j)}{P_s(X=x)} \\quad \\because$ by the assumption $P_s(X=x|Y=j)=P_t(X=x|Y=j)$\n\n$\\propto P_t(X=x|Y=j)P_t(Y=j)$\n\n$=\\frac{P_t(Y=j|X=j)P_t(X=x)}{P_t(Y=j)}P_t(Y=j) \\quad \\because$ by Bayes rule\n\n$=P_t(Y=j|X=j)P_t(X=x)$\n\n$\\propto P_t(Y=j|X=j)$\n\nHere, $P_s(X=x)$ and $P_t(X=x)$ can be omitted using $\\propto$ as they do not affect the classification output. \n\n* *In table 1, what happened to Caltech256 Prior matching?*: We found that prior matching is sensitive to the learning rate and temperature hyperparameters---but hyperparameter tuning with plentiful data is typically not an option for zero-shot models. In the experiments for Table 1, we selected hyperparameters of prior matching via grid search, by evaluating their performance on small validation data (10 shots per class). Prior matching's accuracy in Caltech256 shows the case hyperparameter search fails, which leads to suboptimal solutions in the optimization of the weights $r$ (L557-558).\n\n* *In figure 2, Why OTTER shows acc increase...*: The Bayes error rate changes with the label shift. Thus accuracy of Bayes optimal classifier changes, and OTTER follows the accuracy of Bayes optimal classifier. Specifically, given $\\mathcal{N}(-1,1)$ and $X|Y=1 \\sim \\mathcal{N}(1,1)$, the error rate $\\int_x1-\\max_yP(Y=y|X=x)$ is maximized at $\\nu^s_0$=$\\nu^s_1$=0.5. Total variation 0.4 corresponds to this point, and the Bayes error rate is reduced after that. OTTER achieves the Bayes error regardless of label distribution shift, while naive classification deteriorates with the magnitude of the label distribution shift.\n\n* *How long does it take to apply OTTER? I suppose it will take quadratically proportional to the dataset size.*: We report computation time in Appendix E.3. Table 10. The computational complexity  depends on implementation. The linear programming version of the optimal transport algorithm can run in $\\tilde{O}(nk\\sqrt{n+k})$ time via minimum cost flow [6], where $n$ is the number of data points and $k$ is the number of classes. Thus, computation time **subquadratically** increases in the number of data points. In practice, we observed our method gives modified predictions **within 0.05 ms per sample**---a negligible overhead. Additionally, a batched version with parallel computing is an option for massive inference. We included an experiment where the performance of optimal transport changes depending on batch size in Figure 1 of the attached pdf. The result shows that reasonable accuracy improvements can be obtained even when randomly partitioning data and applying OTTER.\n\nWe appreciate the reviewer's consideration and are more than willing to address any further concerns. If we have adequately resolved the issues, we would be grateful if the reviewer could consider raising their score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0jTCrSIIf8", "reviewer": "Reviewer_t22L", "summary": "This paper introduces a novel approach to address the issue of label distribution mismatch in zero-shot models, which is a common problem due to the imbalance in the pretraining datasets. The proposed method uses optimal transport to adjust the predictions of pretrained models based on an estimated downstream label distribution, without requiring additional training or access to labeled downstream data.", "questions": "The paper presents a compelling and well-validated approach to address label distribution mismatch in zero-shot models using optimal transport. Its innovative method, strong theoretical foundation, and significant empirical improvements make it a valuable contribution to the field. However, further exploration of its limitations, more comparisons with recent methods, and practical implementation details would enhance its impact and utility.", "limitations": "The authors do not claim limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The use of optimal transport to handle label distribution mismatch in zero-shot models is a novel and elegant solution.\n2. The paper provides a solid theoretical foundation for the proposed method, including characterizations of the improvement under mild conditions and error bounds for misspecification. \n3. Extensive empirical validation across a wide range of image and text classification tasks demonstrates the effectiveness of OTTER.", "weaknesses": "1. dependence on estimated label distribution: the method relies on an accurate estimate of the downstream label distribution. In practice, obtaining a reliable estimate may be challenging, and errors in this estimate could impact the performance of OTTER.\n2. evaluation on diverse datasets: while the paper includes a variety of datasets for validation, a more detailed analysis of performance across different types of datasets (e.g., varying in size, complexity, and imbalance levels) would provide a deeper understanding of the method's strengths and limitations.\n3. comparison with recent methods: the paper compares OTTER with some existing baselines, but it could benefit from a more comprehensive comparison with the latest state-of-the-art methods in label distribution adaptation and zero-shot learning. There are many training-free or test-time adaption zero-shot baselines[a][b][c]. How does OTTER perform compared to these methods?\n4. scalability: although described as lightweight, the paper does not thoroughly discuss the scalability of OTTER for very large-scale datasets.\n\n\n[a] Mirza, Muhammad Jehanzeb, et al. \"Lafter: Label-free tuning of zero-shot classifier using language and unlabeled image collections.\" NIPS 2024.\n\n[b] Zhao, Shuai, et al. \"Test-time adaptation with clip reward for zero-shot generalization in vision-language models.\" arXiv preprint arXiv:2305.18010.\n\n[c] Abdul Samadh, Jameel, et al. \"Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization.\" NIPS 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for noting the strengths of our work and providing useful comments. The reviewer appreciated our work, recognizing the novelty of our method, its solid theoretical foundation, and extensive empirical validation.\n\n* **On dependence on estimated label distribution**: While our method requires estimated label distribution, the requirement for improvement is not heavy---the estimated distribution just needs to be better than zero-shot label distribubtion. Furthermore, label distribution estimation algorithms have been developed in other  works (e.g. [1,2,3,4,5]). Any availble method can be plugged into OTTER. For example, we used BBSE [1] as a label distribution estimation method in linear probing setting.\n\n* **On more detailed analysis of performance across different types of datasets**: We reported information about datasets in Appendix E.1 Table 6, which shows that experiments cover diverse datasets with respect to the number of data points, the number of classes and the imbalance level. Additionally, we provide further analysis illustrating the relationship between (n, K, Imbalance) and OTTER's performance gains in Figure 3 of the attached pdf. In summary, **accuracy tends to increase as dataset size increases, the number of classes decreases, and imbalance level increases**. \n\n* **On comparison with recent methods**: Thank you for suggesting useful references. The suggested papers share commonalities in seeking to perform test-time adapation without any labeled data. [6] fine-tunes zero-shot models using pseudo-labels generated by text classifiers trained on LLM-generated text descriptions and class names. [7] uses CLIP scores as reward signals in reinforcement learning, facilitating task-specific fine-tuning of VLMs. [8] uses test-time prompt tuning with alignment loss to effectively align representations with pretraining data. While OTTER is indeed also a test-time alignment method, **it offers a much more lightweight way to update predictions without any parameter updates, while [6, 7, 8] require additional training for some parts of models**.\nAdditionally, OTTER can be easily combined with [6, 7, 8], giving further improvement. We include a mini experiment with [6] in three datasets: Caltech101, DTD, OxfordFlowers.\n\n|                | Caltech101 | DTD   |  OxfordFlowers |\n| -------------- | ---------- | ----- |  ------------- |\n| Zeroshot       | 90.67      | 42.02 |  63.78         |\n| LaFTer         | 93.06      | 49.05 |  71.70         |\n| OTTER          | 94.73      | 45.98 |  68.01         |\n| LaFTer + OTTER | **95.90**      | **53.72** |  **76.29**         |\n\nOTTER is comparable to LaFTer (but, as we described, more lightweight) and  provides further improvements when combined with LaFTer.\n\n* **On scalability**: As mentioned in the common response, we reported computation time in Appendix E.3. Table 10. While it is true that our inference-time adaptation approach requires additional computation, **the computational overhead is not heavy** --- the linear programming version of OT can run with subqudratic computation comlexity. In practice, we observed our method gives modified predictions **within 0.05 ms per sample**---a negligible overhead\nAdditionally, for massive-scale inference, batch optimal transport with parallel computing can be used. Figure 1 in the attached file shows the accuracy and computation time (per batch) depending on the batch size. Note that this result can be further improved with more advanced batch optimal transport methods.\n\n[1] Saerens, Marco, Patrice Latinne, and Christine Decaestecker. \"Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure.\" Neural computation 14.1 (2002): 21-41.\n\n[2] Lipton, Zachary, Yu-Xiang Wang, and Alexander Smola. \"Detecting and correcting for label shift with black box predictors.\" ICML'18. \n\n[3] Azizzadenesheli, Kamyar, et al. \"Regularized Learning for Domain Adaptation under Label Shifts.\" ICLR'18.\n\n[4] Alexandari, Amr, Anshul Kundaje, and Avanti Shrikumar. \"Maximum likelihood with bias-corrected calibration is hard-to-beat at label shift adaptation.\" ICML'20.\n\n[5] Garg, Saurabh, et al. \"A unified view of label shift estimation.\" NeurIPS'20. \n\n[6] Mirza, Muhammad Jehanzeb, et al. \"Lafter: Label-free tuning of zero-shot classifier using language and unlabeled image collections.\" NeurIPS'24.\n\n[7] Zhao, Shuai, et al. \"Test-time adaptation with clip reward for zero-shot generalization in vision-language models.\" arXiv preprint arXiv:2305.18010.\n\n[8] Abdul Samadh, Jameel, et al. \"Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization.\" NeurIPS'24.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### Common Response
We thank all of the reviewers for their kind comments and feedback. Reviewers recognized the strengths of our paper:
* OTTER provides **a novel and elegant solution** to deal with label distribution using optimal transport. (Reviewers t3Bz, t22L)
* OTTER offers **theoretical results** showing that (1) OTTER can recover a Bayes optimal classifier under the label shift setup, and (2) the error bound can be derived with misspecification of label distribution estimation and miscalibration. (Reviewers t3Bz, t22L)
* **Empirical results** in a variety of experimental settings demonstrate the effectiveness of OTTER. (Reviewers M3dD, t22L)

We address two common questions before proceeding to individual responses:
* **On computation cost and scalability of OTTER**: We report computation time in Appendix E.3. Table 10. While it is true that our inference-time adaptation approach requires additional computation, **the computational overhead is not heavy**. The linear programming version of the optimal transport algorithm can run in $\tilde{O}(nk\sqrt{n+k})$ time via minimum cost flow [1], where $n$ is the number of data points and $k$ is the number of classes. Thus, computation time subquadratically increases in the number of data points. In practice, we observed our method gives modified predictions **within 0.05 ms per sample**---a negligible overhead.
Additionally, batched OTTER with parallel computing, instead of using the full inference dataset, can be used for massive-scale inference. Figure 1 in the attached file shows the accuracy and computation time (per batch) depending on the batch size. Note that this result can be further improved with more advanced batched optimal transport methods [2, 3]. 

* **On the dependency on estimated label distribution**: While the true label distribution enables the maximum improvement when using the proposed method, **it is not necessary**. Indeed, our algorithm can improve zero-shot classification *with just slightly better label distribution estimation than the one implicitly used in zero-shot prediction*---so that our requirements are extremely low. To illustrate this claim, we interpolate the label distribution of zero-shot prediction $\nu^{zs}$ and the true label distribution $\nu^{true}$ such that $$\hat{\nu}_\alpha=(1-\alpha)\nu^{zs} + \alpha\nu^{true},$$ where $0 \leq \alpha \leq 1$. We use $\hat{\nu}_\alpha$ as the label distribution specification for OTTER and provide a graph (Figure 2 in the attached pdf) that illustrates the resulting accuracy changes. As expected, **as long as the specification of the label distribution is closer to the true distribution, our technique shows performance improvement in all cases**.

[1] Lee, Yin Tat, and Aaron Sidford. "Path finding methods for linear programming: Solving linear programs in $\tilde{O}(\sqrt{rank})$ iterations and faster algorithms for maximum flow." 2014 IEEE 55th Annual Symposium on Foundations of Computer Science. IEEE, 2014.

[2] Nguyen, Khai, et al. "Improving mini-batch optimal transport via partial transportation." ICML'22.

[3] Nguyen, Khai, et al. "On Transportation of Mini-batches: A Hierarchical Approach." ICML'22.

Meta-review: The paper provides a simple method to handle changes in label proportions between a pretrained model and a downstream task, via optimal transport. The method is simple and straightforward, and the authors perform a comprehensive empirical evaluation backed by theory. The paper is generally well-written and easy to read. Thereby, I recommend acceptance as a poster.
Decision: Accept (poster)