Title: OTTER: Effortless Label Distribution Adaptation of Zero-shot Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: zero-shot classification

label distribution

adaptation
Abstract: Popular zero-shot models suffer due to artifacts inherited from pretraining. One particularly detrimental issue, caused by unbalanced web-scale pretraining data, is mismatched label distribution. Existing approaches that seek to repair the label distribution are not suitable in zero-shot settings, as they have mismatching  requirements, such as needing access to labeled downstream task data or knowledge of the true label balance in the pretraining distribution. We sidestep these challenges and introduce a simple and lightweight approach to adjust pretrained model predictions via optimal transport. Our technique requires only an estimate of the label distribution of a downstream task.  Theoretically, we characterize the improvement produced by our procedure under certain mild conditions and provide bounds on the error caused by misspecification. Empirically, we validate our method in a wide array of zero-shot image and text classification tasks, improving accuracy by 4.8% and 15.9% on average, and beating baselines like prior matching---often by significant margins---in 17 out of 21 datasets.

OTTER: Effortless Label Distribution Adaptation of Zero-shot Models

Abstract

Popular zero-shot models suffer due to artifacts inherited from pretraining. One particularly detrimental issue, caused by unbalanced web-scale pretraining data, is mismatched label distribution . Existing approaches that seek to repair the label distribution are not suitable in zero-shot settings, as they have mismatching requirements, such as needing access to labeled downstream task data or knowledge of the true label balance in the pretraining distribution. We sidestep these challenges and introduce a simple and lightweight approach to adjust pretrained model predictions via optimal transport. Our technique requires only an estimate of the label distribution of a downstream task. Theoretically, we characterize the improvement produced by our procedure under certain mild conditions and provide bounds on the error caused by misspecification. Empirically, we validate our method in a wide array of zero-shot image and text classification tasks, improving accuracy by 4.8% and 15.9% on average, and beating baselines like prior matching-often by significant margins-in 17 out of 21 datasets.

1 Introduction

Zero-shot models are popular but struggle with biases inherited from their large pretraining datasets [21; 59; 2]. In particular, zero-shot classification is strongly biased by the label distribution of the pretraining task. When the label distribution of the downstream task differs from pretraining, the performance of zero-shot classifiers suffers greatly. For example, Figure 1 illustrates the effects of mismatched distributions on a pet image classification task. Two CLIP models (RN50, and ViT-B/16) produce biased predictions on the Abyssinian and Persian classes. Furthermore, datasets with a large number of classes, such as ImageNet, may contain both extremely common and very rare classes, resulting in an outsized probability that a zero-shot model will predict some classes over others. As a result, even large models intended for use in zero-shot settings, such as CLIP [49], naturally have a label distribution mismatch between pretraining data and downstream tasks.

Existing methods that seek to address label distribution make strong assumptions or have expensive requirements . For example, to fine-tune a model, we must obtain a labeled fine-tuning dataset of adequate size, then obtain the time and compute to further train the model. To perform label shift adap-

38th Conference on Neural Information Processing Systems (NeurIPS 2024).

RN50

0.5

ViT-B/16

Proportion

0.4

0.3

0.2

0.1

0.0

Abyssinian

Persian

tation techniques, we must know the true label distribution of the pretraining distribution-difficult to impossible for real-world tasks.

Can we deal with label distribution mismatch without additional training or access to groundtruth downstream task information ? While seemingly challenging, one cause for optimism is the observation that zero-shot models still give relatively high prediction probabilities for correct classes, though classes common in pretraining tend to have relatively inflated scores overall. Intuitively, the model has already learned to identify examples of its downstream classes (and so does not require further training) and is already impacted by the pretraining label distribution (and so does not need access to the ground-truth pretraining label distribution). Instead, the model's prediction probabilities must be adjusted based on an estimated downstream label distribution specification.

To perform this label distribution adjustment, we view zero-shot learning through the lens of optimal transport (OT) and develop a technique called OTTER (Optimal TransporT adaptER) . This OT-based approach offers a systematic way to rebalance predicted labels: data points are transported to optimal downstream classes, minimizing the overall cost in accordance with the estimated the downstream label distribution specifications.

Theoretically, we show that optimal transport given the true label distribution of the downstream can recover the Bayes-optimal classifier under mild conditions. Additionally, we provide error bounds on our adaptation method for misspecification. We provide synthetic experiments validating our theoretical claims. In real-world data settings, we validate our method on a wide variety of image and text classification tasks, showing 4.8% and 15.5% accuracy improvement on average in image and text zero-shot classification tasks, respectively. Finally, we evaluate our method in few-shot adaptation scenarios, where OTTER provides further improvements when combined with linear probing. Our contributions include:

OTTER, an algorithm to deal with label distribution mismatch at inference time via optimal transport,

Theoretical results showing the effectiveness of our method, including the ability to recover the Bayes-optimal classifier and a sensitivity analysis with respect to the label distribution specification estimation,

Extensive empirical results on zero-shot classification for text and image datasets, showing accuracy improvements of up to 25%,

Experimental results demonstrating the applicability of OTTER to few-shot settings, showing accuracy improvements of up to 15%, even with noisy label distribution specifications,

Extensions of OTTER to leverage label hierarchy information or relax the batched prediction requirement and

Application to LLM selection bias mitigation.

2 Background and Problem Formulation

Before presenting OTTER and providing our results, we introduce some crucial background and describe the problem setting formally.

2.1 Background

We briefly describe zero-shot models, the technical tool we use (optimal transport), along with other techniques that seek to address shifts. We have extended related work, including in-depth characterizations and comparisons with related methods, in Appendix B.

Zero-shot Models. Zero-shot classification, popularized by models such as CLIP [49], is a powerful paradigm that enables prediction on downstream tasks without additional fine-tuning. Image, language, and multimodal models have been increasingly employed for zero-shot prediction [64; 36]. These models undergo extensive pretraining on massive datasets with concept and label spaces that may be very different from those of downstream applications.

Optimal Transport. Optimal Transport (OT) is a framework for matching two probability distributions [48; 53]. We predominantly consider optimal transport between empirical discrete measures. Suppose that we are given points x 1 , x 2 , . . . , x n ∈ X and y 1 , . . . , y K ∈ Y , a source measure µ defined by µ = ∑ n i =1 w i δ x i , and a target measure given by ν = ∑ K j =1 p j δ j , where w i , p j are positive values such that ∑ n i =1 w i = 1 , ∑ K j =1 p j = 1 . Suppose also that δ x is a Dirac delta function

Algorithm 1 OTTER

1: Input: Input X = { x 1 , . . . , x n } , label distribution specification ( p 1 , . . . , p K ) , cost matrix C ∈ R n × K

3: Run optimal transport and obtain transport plan π s.t. π = arg min γ ∈ Π( µ,ν ) 〈 γ, C 〉 .

2: Define input marginal µ = 1 1 n , prediction marginal ν = ( p 1 , . . . , p K )

4: Get modified classification outputs ˆ y i = arg max j ∈ [ K ] π i,j . Return { ˆ y i } i ∈ [ n ]

at x , i.e. δ x ( x ′ ) = 1 if x ′ = x , and δ x ( x ′ ) = 0 otherwise. Given a cost matrix C ∈ R n × K , the Monge-Kantorovich formulation of optimal transport is to find a minimal cost transport plan π such that

Distribution and Label Shifts. Distribution shift refers to the discrepancy between a source distribution P s on which the model is trained, and a target distribution P t on which the model is deployed. Distribution shift often degrades trained model performance on target tasks. Label shift is a specific type of distribution shift such that P s ( Y ) = P t ( Y ) and the data generation process is fixed -in other words, the conditional distributions of the inputs are the same: P s ( X | Y ) = P t ( X | Y ) . Techniques such as importance sampling [35; 6; 24], recalibration [3] and domain adaptation [55] are commonly used to mitigate the effects of label shift. Unfortunately, these methods assume access to source distribution data, whereas zero-shot models' pretraining data is inaccessible (often proprietary or blocked for privacy reasons). Thus, adapting zero-shot models to new label distributions poses challenges unmet by these pre-existing methods.

2.2 Problem Formulation

Let X = { x 1 , x 2 , . . . , x n } be an inference dataset with x i ∈ X . Furthermore, let Y = { y 1 , y 2 , . . . , y n } be the true labels of the K -class classification dataset, such that y i ∈ Y = [ K ] , are sampled according to the downstream label distribution ν = ( p 1 , p 2 , . . . , p K ) .

Let s θ ( x, j ) := P θ ( Y = j | X = x ) be a pretrained classification model constrained to the downstream label space. During pretraining, s θ has been biased to the source label distribution ν s . We wish to offset such label distribution bias with a label distribution specification ˆ ν for the target distribution. ˆ ν is expected to be closer to the true label distribution of the downstream task. Given a label distribution specification, our goal is to rebalance predictions so that the predicted label distribution follows the label distribution specification.

3 Proposed Framework

We propose OTTER (Optimal TransporT adaptER), an optimal transport-based label distribution adaptation approach. Our goal is to have the n input data points allocated to K classes match a given label distribution ˆ ν , where ∑ K j =1 ˆ ν j = 1 , ˆ ν j ≥ 0 . Specifically, we want to classify n ˆ ν 1 points as the first class, n ˆ ν 2 points as the second, and so on. However, there are many such allocations, and it is not a priori clear which one should be selected. We propose formulating an optimal transport problem that selects the allocation minimizing a particular cost:

where Π( µ, ˆ ν ) = { γ ∈ R n × K + | γ 1 = µ, γ T 1 = ˆ ν } , µ = 1 n 1 and C is the cost (loss) matrix such that C ij represents a loss when we classify x i as class j . This procedure is described in Algorithm 1. Note that this procedure naturally matches the given label distribution specification ˆ ν .

We wish to use Algorithm 1 for zero-shot classification given the pretrained model s θ . To do so, we must select a cost function and produce C ij . An ideal choice of such a function is C ij = -log P t ( Y = j | X = i ) such that optimal transport minimizes the negative log posterior under constraints. However, the target distribution P t is unknown. Instead, we replace the posterior with the classifier scores s θ ( x i , j ) . We highlight that this choice of cost matrix is an natural extension of

/negationslash

zero-shot classification under the label distribution constraint. We prove this claim in the next section. First, we show a toy example that shows how OTTER improves zero-shot accuracy.

Example. To illustrate the benefits of OTTER, consider the following example for binary classification. We have two data points, X = { x 1 , x 2 } with Y = { 1 , 2 } , and true label distribution ν = ( 1 2 , 1 2 ) . Suppose that the zero-shot model's prediction scores are s 1 = (0 . 4 , 0 . 6) and s 2 = (0 . 1 , 0 . 9) .

Traditional classification yields ˆ y 1 = 2 , ˆ y 2 = 2 , producing a 50% error rate. However, given the cost matrix C derived from the prediction score matrix

along with µ = (0 . 5 , 0 . 5) and ν = (0 . 5 , 0 . 5) , the optimal transport procedure discovers the transport map π = [ 0 . 5 0 . 0 0 . 0 0 . 5 ] , yielding ˆ y 1 = 1 , ˆ y 2 = 2 . This corrects the original zero-shot prediction error.

Extension to Online Predictions While OTTER offers efficient label distribution adaptation, it requires a batched set of inference data points, making online predictions challenging. To address this, we introduce R-OTTER (Reweighting OTTER), which learns a reweighting factor-an estimate of P t ( Y ) /P s ( Y ) -using OTTER's predictions y OTTER on a validation set. Once learned, these reweighting parameters can be applied to online predictions by adjusting logit probability scores. We use a reweighting formulation equivalent to logit adjustment in [71]. The reweighted probability scores of P θ , with a reweighting vector r ∈ R K , are defined as:

The parameter r is learned using cross-entropy loss on y OTTER. We expect R-OTTER to perform comparably to OTTER, with the additional advantage of not requiring OTTER to be run over the entire dataset. In the following section, we provide a theoretical result showing that r ∗ = P t ( Y ) /P s ( Y ) is an optimal reweighting parameter when learned from y OTTER as pseudolabels, effectively addressing label shift.

4 Theoretical Results

In practical scenarios, label distribution specifications are frequently subject to noise, and prediction probabilities may not be well-calibrated. To understand the impact of these factors, we examine how errors in label distribution specification and calibration influence the transport plan. Our theoretical analysis yields following findings: (a) OTTER can recover the Bayes-optimal classifier in the label shift setting, (b) for a noisy cost matrix with the noisy label distribution specification setup, the suboptimality can be bounded by the deviation of cost matrix and label distribution, and (c) R-OTTER effectively addresses label shift when learned from y OTTER as pseudolabels.

Classification as optimal transport. Prior to discussing the main theoretical results, we demonstrate that standard classification-expressed as ˆ y i = arg max j ∈ [ K ] P θ ( Y = j | X = x i ) -can be interpreted as a (trivial) solution derived from optimal transport.

/negationslash

Theorem 4.1. Let ν ZS j = 1 n ∑ n i =1 ✶ [ˆ y ZS i = j ] , where ˆ y ZS i = arg max j ′ ∈ [ K ] P θ ( Y = j ′ | X = x i ) . Then, given C ij = -log P θ ( Y = j | X = x i ) ,

Assuming there are no ties in scores, i.e. P θ ( Y = j | X = x i ) = P θ ( Y = j ′ | X = x i ) , for all j = j ′ , the OTTER predictions are equivalent zero-shot predictions, i.e. ˆ y OT i = ˆ y ZS i for all i ∈ [ n ] .

/negationslash

This theorem has the following implications. First, it suggests that the predictions will remain unchanged if we set ˆ ν = ν ZS . Second, Bayes-optimal classifiers can be derived through optimal transport, using a (true) cost matrix defined as C ∗ ij = -log P t ( Y = j | X = x i ) , coupled with the true label distribution ν ∗ .

Our analysis begins with the label shift setup, which is a commonly-studied type of distribution shift-as well as a prominent issue when applying zero-shot models. We demonstrate that when the label distribution is correctly specified, optimal transport preserves Bayes-optimal classifier predictions under label shift. Next, we consider general perturbations in label distribution and cost matrix as well as their impact on the resulting solutions.

4.1 Label Shift Invariance

In this setting, we assume features follow the same conditional distribution across source and target distributions, i.e. P s ( X | Y ) = P t ( X | Y ) . Furthermore, we suppose that the prediction scores are accurately calibrated in the training dataset, such that s θ ( x, j ) = P s ( Y = j | X = x ) . For zero-shot models, we often lack access to P s . This is a typical scenario in zero-shot model applications: after training on large-scale corpora, we use the pretrained model without the source dataset.

For a given downstream task with the target label distribution ν ∗ = P t ( Y ) , one standard approach to achieve the Bayes-optimal classifier for the target distribution is to reweight the score function outputs using the ratio P t ( Y = j ) P s ( Y = j ) . This adjustment leads to:

This reweighted score function aligns with the target distribution, thus correcting label shift.

Although reweighting the score function is a popular solution, it faces an important obstacle when applied to zero-shot models like CLIP, where the source distribution P s ( Y ) is typically unknown. We show that OTTER successfully induces a Bayes classifier for the target distribution, represented as f t ( x ) = arg max j ∈ [ K ] P t ( Y = j | X = x ) , without requiring access to P s ( Y ) . This capability is particularly significant for zero-shot models, enabling them to adapt to target distributions effectively, even in the absence of explicit knowledge of the source distribution.

Now, we show that optimal transport can be an effective tool to correct label shift.

Theorem 4.2. Suppose the pretrained model is well-calibrated for the source distribution,

/negationslash and there is no tie probability, for all j = j ′ , i ∈ [ n ]

/negationslash

Denote the Bayes optimal predictions in the target distribution as ˆ y ∗ i = arg max j ∈ [ K ] log P t ( Y = j | X = x i ) . Then OTTER predictions ˆ y = OTTER ( X , ν ∗ , C ) are the same as Bayes optimal predictions ˆ y ∗ .

That is, OTTER recovers a Bayes classifier in the target distribution without access to the source distribution, given the target distribution and a well-calibrated model for the source dataset.

4.2 General Perturbation Sensitivity

In practical applications, calibration error could extend beyond noise in the elements of the cost matrix. A key source of error is label distribution estimation error. Hence, we address a more general setting, examining the impact of simultaneous perturbations in the label distribution and cost matrix of the transport plan. Our result applies techniques from perturbation theory for linear programming .

We rewrite our optimal transport problem min π ∈ Π( µ,ν ) 〈 π, C 〉 as a linear programming problem. Let π and C be the transport plan and cost matrix respectively. Matrix G and vector g are used to denote the row and column constraints on π to form a feasible plan which transports distribution from µ to ν .

Then, we have the equivalent linear programming problem,

We adapt a theorem from Robinson [52] with our optimal transport problem notation.

Theorem 4.3. Let the primal linear programming problem be defined as in equation (1), and its dual problem be max { w T g | w T G ≤ vec ( C ) T , w ≥ 0 } . Suppose perturbed cost matrix is ˆ C = C +∆ C , the perturbed class distribution ˆ ν = ν +∆ ν , such that ˆ g = g +∆ g where

Assume that primal and dual problems are solvable. Denote the original solutions as π, w and perturbed solutions as ˆ π and ˆ w . Then,

, where 1 ≤ p ≤ ∞ and κ is a Hoffman constant that only relates to the original problem [30].

Ignoring the constant and the subtraction part, the upper bound can be decomposed into three components,

∆ ν : noise (or the estimation error) of the target balance,

[ vec (∆ C )] + : noise (or the calibration error) of the cost matrix,

vec ( C ) T vec (ˆ π ) -g T ˆ w : the suboptimality of perturbed solution ˆ w.

Theorem 4.3 implies that the deviation from perturbed solution to true solution is bounded by the magnitude of perturbations and suboptimality of the perturbed solution. From this result, we can expect prediction accuracy to deteriorate with perturbations in the label distribution and calibration.

4.3 Effectiveness of R-OTTER

We provide a theoretical result showing that R-OTTER can learn an optimal parameter by learning reweighting parameters from ˆ y OTTER as pseudolabels, and produce identical predictions to OTTER once the optimal parameter is obtained in the label shift setup.

Theorem 4.4. Under the same assumptions as in Theorem 4.2, the parameter r ∗ = P t ( Y ) /P s ( Y ) is optimal when learning with y OTTER as pseudolabels.

The proof is provided in Appendix D.6.

5 Experiments

The primary objective of our experiments is to (1) validate that OTTER improves zero-shot model performance when given accurate label distribution estimates and (2) investigate its sensitivity to perturbations. In experiments on real datasets (Section 5.1), we confirm that OTTER can improve zero-shot classification significantly in a variety of settings. In synthetic experiments (Section 5.2), we validate our theoretical claims-label shift invariance and sensitivity to perturbation in a fully controllable setting. Additionally, we show that OTTER can be combined with label distribution estimation methods in the few-shot learning setting (Section 5.3). Next, we show the empirical results for H-OTTER that leverages label hierarchy (Section 5.4) and R-OTTER that mitigates the batched prediction requirement (Section 5.5). Finally, we we apply OTTER to mitigate LLM selection bias (Section 5.6). Our code is available at https://github.com/SprocketLab/OTTER.

5.1 Real Data Experiments

We hypothesize that the model performance can improve significantly when the label distribution specification is exact.

Setup and Procedure. We used 17 image classification datasets and 4 text classification datasets. We employed CLIP [49] for image zero-shot classification, and BERT [19]. A comprehensive list and details of experiments can be found in Appendix E.

Baseline. We adopt Prior Matching (PM) [37] as a baseline. It optimizes score weighting parameters to align with the label distribution specification. A detailed explanation of Prior Matching is given in Appendix C. It is worth noting that the performance of Prior Matching is highly sensitive to hyperparameters such as temperature and learning rate . Optimal hyperparameters may vary across

different datasets. We selected hyperparameters through grid search, by evaluating their performance on a validation set, consisting of 10 labeled examples per class. In contrast, we highlight that OTTER is tuning-free.

Results. Table 1 shows the image classification results with CLIP (ViT-B/16) and the text classification results with BERT. Notably, OTTER demonstrates a 4.8% and 15.5% enhancement on average in image and text zero-shot classification, respectively . While Prior Matching shows competitive performance when accurately tuned, it often struggles. We found that hyperparameter tuning fails in the class-imbalanced datasets such as Caltech256, SUN397, ImageNet-r (Appendix E, Table 7). This suggests that the hyperparameter selection process necessitates a validation set label distribution similar to the target distribution-rendering it unusable in zero-shot scenarios. More details and additional experiment results - including the sensitivity study on the label distribution specification error, computation time, and combination with other prompting methods - are provided in Appendix E.3.

5.2 Synthetic Experiments

We hypothesize OTTER is invariant to label shift under the conditions in Theorem 4.2. We also investigate the sensitivity to perturbations of the cost matrix and the label distribution.

Setup and Procedure. We simulate label shift in logistic regression. Suppose X | Y = 0 ∼ N ( -1 , 1) and X | Y = 1 ∼ N (1 , 1) . Training data is sampled from a mixture of Gaussians X s ∼ ν s 0 N ( -1 , 1) + ν s 1 N (1 , 1) such that P s ( Y = 0) = ν s 0 , P s ( Y = 1) = ν s 1 , ν s 0 + ν s 1 = 1 . Similarly, we sample the test data from X t ∼ ν t 0 N ( -1 , 1) + ν t 1 N (1 , 1) . We fix the training set label distribution as ν s 0 = 0 . 1 , ν s 1 = 0 . 9 and vary test set label distribution ν t to simulate label shift. We train a logistic regression model with 10,000 samples from the source distribution, and test the model with 10,000 samples from the target distribution. A Bayes-optimal classifier in the target distribution is given by f Bayes ( x ) = ✶ [ x ≥ 1 2 (log ν t 0 ν t 1 +1) ]. The naive classifier is defined as the maximizer of the predicted score. The OTTER predictions are produced with Algorithm 1, with the cost matrix C ij = -log P θ ( Y = j | X = x i ) and the label distribution specification ν t , where represents the logistic regression model scores.

P θ ( Y | X )

We separately investigate perturbed prediction score matrix and perturbed label distribution specification's impact on the prediction accuracy. For perturbed prediction scores, we fix the label distribution to be the true one, and add noise δ ∼ N (0 , σ 2 ) of varying levels σ to the predicted score P θ ( Y = 1 | X ) . For the perturbed label distribution specification, we fix the prediction score to be true scores and add noise /epsilon1 : ˆ ν = ν t +( /epsilon1, -/epsilon1 ) . We use these perturbed variants to obtain perturbed solutions and compare with ground-truth solution.

Results. Figure 2 illustrates how accuracy changes with label shift when the predicted score is perturbed and when label distribution specification is perturbed. We observe that the naive classifier

90

80

Acc(%)

Naive

70

Bayes

OTTER,6=0

OTTER,6~N(0,0.052)

60

OTTER,6~N(0, 0.12)

OTTER, 6~N(0, 0.152)

OTTER,6~N(0,0.22)

0.0

0.2

0.4

0.6

TV(Pt(Y), Ps(Y))

90

80

Acc(%)

Naive

70

Bayes

OTTER,=0

60

OTTER,=0.05

OTTER, ε=0.1

OTTER, =0.15

0.0

0.2

0.4

0.6

TV(Pt(Y), Ps(Y))

deteriorates as the total variation distance between source and target distributions increases. It indicates that naive classifier is sensitive to label shift. However, without perturbation, OTTER remains unaffected by the label distribution shift , which validates our invariance result in Section 4.

In the case of confidence prediction perturbation, both the naive classifier and OTTER have accuracy decreasing as perturbation level increases. For simplicity, we omitted the naive classifier's performances under different levels of noise as adding zero-mean noise does not alter its accuracy significantly. We observe that OTTER has better performance than the naive method when significant label shift exists. Similarly, for label distribution perturbation, we observe as the noise level /epsilon1 increases, OTTER's accuracy downgrades-but still yields better performance when label shift is severe.

Our experimental results suggest simply using prediction scores for zero-shot classification leads to inaccurate predictions under label shift, while OTTER is robust to label shift when no perturbations are present. Perturbations in both predicted score and label distribution specification downgrades the predicted accuracy, as expected, but OTTER still yields better results than the naive baseline.

5.3 Few-shot adaptation with label distribution estimation

25

50

75

100

0.10

0.12

0.14

0.16

0.18

0.20

0.22

TV(

Pt

(

Y

),

Pt

(

Y

))

Estimation Error

(a) BBSE estimation error

25

50

75

100

60.0

62.5

65.0

67.5

70.0

72.5

Acc. (%)

ZS

OTTER

LP

LP+OTTER

(b) Accuracy

We anticipate that OTTER can be used in few-shot learning when combined with label distribution estimation methods. We expect OTTER can improve zero-shot classification if the label distribution estimation error is sufficiently small. Also, we expect OTTER can improve linear probing, which is one of standard approaches for few-shot learning.

Setup and Procedure. We use the same datasets as the previous experiment. We consider a 10-shot learning setting: 10 labeled samples per class are given. Note that labeled samples have uniform label distribution, while the label distribution in the target distribution may not be uniform . This setting requires the use of label distri- bution estimation methods used in label shift adaptation [35; 6; 24]. We estimate the target label

distribution with Black Box Shift Estimation (BBSE) [35]. BBSE estimates the target balance using confusion matrix, under the label shift assumption. For detailed explanation, refer to Appendix C.

Results. Table 2 shows the image and text zero-shot classification results with the label distribution estimation via BBSE and linear probing. The image classification results show that OTTER can yield mild improvement over linear probing, even with the label distribution estimation errors. Figure 4 shows that accuracy improvement is consistent across the number of samples used for linear probing. In text classification, we found OTTER improves zero-shot text classifications where the number of classes is small ( K = 2 or 3 ). While it shows a relatively high variance due to the small sample size (20 ∼ 30), the average accuracy improves significantly over zero-shot classification. More detailed analysis regarding label distribution estimation error and the number of samples is provided in Appendix E.4.

5.4 Zero-shot prediction improvement with class hierarchy

We hypothesize incorporating class hierarchy information can enhance few-shot label distribution estimation and thus improve zero-shot predictions.

Setup and Procedure. Weuse a subset of CIFAR100 data with WordNet hierarchy. Specifically, we take 'fish' and 'tree' as superclasses and have 5 subclasses in each of them. We suppose we can access 10 labeled samples per each subclass. We first apply OTTER with the superlevel label distribution estimation and make pseudo-labels of superlevel class in the test set. Using them, we estimate the sublevel label distribution and use OTTER.

Results. Table 3 presents the results. As anticipated, we note an enhancement in accuracy when compared to the naive implementation of OTTER. Specifically, we observe a significant improvement in accuracy for RN50, RN101, and ViT-B/16,

which we attribute primarily to the reduction in label distribution estimation error. Further details are provided in Appendix E.5.

5.5 Effectiveness of R-OTTER

We show that R-OTTER provides a performance comparable to that of OTTER empirically.

Setup and Procedure. We use the identical setup for Section 5.1. R-OTTER learns reweighting parameters in validation set using y OTTER. Note that the validation set is not required to be labeled since y OTTER is used as pseudolabels in the validation set.

Results. Although R-OTTER is suboptimal compared to OTTER due to generalization issues, it still provides label distribution correction, improving accuracy over zero-shot predictions. We also provide synthetic experiments for R-OTTER in Appendix E.6.

5.6 Mitigating selection bias in LLM multiple-choice questions

Selection bias is the tendency of LLMs to favor certain prefix tokens in multiple-choice questions [69; 17; 12; 65]. We demonstrate that OTTER can effectively mitigate selection bias by randomly shuffling the options and enforcing a uniform class balance in OTTER.

Setup and Procedure. Our experimental setup follows Zheng et al. [69], utilizing the MMLU [28], ARC-Challenge [15], and CommonsenseQA (CSQA) [57] datasets. We test with llama-2(chat)-7/13B [60] and vicuna-v1.3-7/13B [11] models, treating the probabilities of each option token (A/B/C/D) as prediction probabilities. OTTER is applied under the assumption of a uniform distribution. We use 0-shot predictions and evaluate performance using accuracy and the standard deviation of recalls (RStd) as metrics.

Results. Table 5 presents the experimental results. OTTER significantly reduces selection bias, enhancing accuracy by up to 10.6% and lowering RStd by as much as 25.7%.

6 Conclusion

While zero-shot models have been successful, pretraining using Internet-scale datasets yields artifacts that may harm downstream tasks. In this paper, we identify the bias in class balance, and provide a simple but powerful solution using optimal transport. Theoretically, we describe how OT can fix label distribution mismatch and its sensitivity to perturbations. Empirically, we validated our approach's ability to improve zero-shot classification accuracy, mitigating label distribution mismatch in zero-shot models. We believe our method can expedite the deployment of zero-shot classification, reducing the necessity of finetuning for downstream tasks.

Acknowledgments

We are grateful for the support of the NSF under CCF2106707 (Program Synthesis for Weak Supervision) and the Wisconsin Alumni Research Foundation (WARF). Jitian Zhao gratefully acknowledge support from the IFDS at UW-Madison and NSF through TRIPODS grant 2023239 for their support. We thank Nick Roberts for his valuable discussions and insights.

References

Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala. Zero-shot robustification of zero-shot models. In The Twelfth International Conference on Learning Representations , 2023.

Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. Evaluating clip: towards characterization of broader capabilities and downstream implications. arXiv preprint arXiv:2108.02818 , 2021.

Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with biascorrected calibration is hard-to-beat at label shift adaptation. In International Conference on Machine Learning , pages 222-232. PMLR, 2020.

Bang An, Sicheng Zhu, Michael-Andrei Panaitescu-Liess, Chaithanya Kumar Mummadi, and Furong Huang. More context, less distraction: Visual classification by inferring and conditioning on contextual attributes. arXiv preprint arXiv:2308.01313 , 2023.

Dominique Azé and Jean-Noël Corvellec. On the sensitivity analysis of hoffman constants for systems of linear inequalities. SIAM Journal on Optimization , 12(4):913-927, 2002.

Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. In International Conference on Learning Representations , 2019.

Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference , pages 491-500, 2019.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 , pages 446-461. Springer, 2014.

Wanxing Chang, Ye Shi, Hoang Tuan, and Jingya Wang. Unified optimal transport framework for universal domain adaptation. Advances in Neural Information Processing Systems , 35: 29512-29524, 2022.

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research , 16: 321-357, 2002.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/ .

Hyeong Kyu Choi, Weijie Xu, Chi Xue, Stephanie Eckman, and Chandan K Reddy. Mitigating selection bias with node pruning and auxiliary options. arXiv preprint arXiv:2409.18857 , 2024.

Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070 , 2023.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3606-3613, 2014.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. Discovering latent concepts learned in bert. In International Conference on Learning Representations .

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. Multidimensional gender bias classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 314-331, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.23. URL https://www.aclweb.org/anthology/2020.emnlp-main.23 .

Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. 2018.

Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence , 28(4):594-611, 2006.

Alberto Fernández, Salvador García, Mikel Galar, Ronaldo C Prati, Bartosz Krawczyk, and Francisco Herrera. Learning from imbalanced data sets , volume 10. Springer, 2018.

Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances in Neural Information Processing Systems , 33:3290-3300, 2020.

Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.

Dandan Guo, Zhuo Li, He Zhao, Mingyuan Zhou, Hongyuan Zha, et al. Learning to re-weight examples with optimal transport for imbalanced classification. Advances in Neural Information Processing Systems , 35:25517-25530, 2022.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7):2217-2226, 2019.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8340-8349, 2021.

Alan J Hoffman. On approximate solutions of systems of linear inequalities. Journal of Research of the National Bureau of Standards , 49(4), 1952.

Jonathan Kahana, Niv Cohen, and Yedid Hoshen. Improving zero-shot models with label distribution priors. arXiv preprint arXiv:2212.00784 , 2022.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for finegrained categorization. In Proceedings of the IEEE international conference on computer vision workshops , pages 554-561, 2013.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning , pages 3122-3130. PMLR, 2018.

Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning , pages 3122-3130. PMLR, 2018.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485 , 2023.

Adian Liusie, Potsawee Manakul, and Mark JF Gales. Mitigating word bias in zero-shot prompt-based classifiers. arXiv preprint arXiv:2309.04992 , 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.

Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 14867-14875, 2021.

Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183 , 2022.

Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantlylabeled reviews and fine-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages 188-197, 2019.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing , pages 722-729. IEEE, 2008.

Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg. Chils: Zero-shot image classification with hierarchical label sets. In International Conference on Machine Learning , pages 26342-26362. PMLR, 2023.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition , pages 3498-3505. IEEE, 2012.

Javier Pena, Juan C Vera, and Luis F Zuluaga. New characterizations of hoffman constants for systems of linear constraints. Mathematical Programming , 187:79-109, 2021.

Javier F Peña. An easily computable upper bound on the hoffman constant for homogeneous inequality systems. Computational Optimization and Applications , 87(1):323-335, 2024.

Hanyu Peng, Mingming Sun, and Ping Li. Optimal transport for long-tailed recognition with learnable cost matrix. In International Conference on Learning Representations , 2021.

Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning , 11(5-6):355-607, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bertnetworks. arXiv preprint arXiv:1908.10084 , 2019.

Nicholas Roberts, Xintong Li, Dyah Adila, Sonia Cromp, Tzu-Heng Huang, Jitian Zhao, and Frederic Sala. Geometry-aware adaptation for pretrained models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Stephen M Robinson. Bounds for error in the solution set of a perturbed linear program. Linear Algebra and its applications , 6:69-81, 1973.

Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY , 55(58-63): 94, 2015.

Liangliang Shi, Haoyu Zhen, Gu Zhang, and Junchi Yan. Relative entropic optimal transport: a (prior-aware) matching perspective to (unbalanced) classification. Advances in Neural Information Processing Systems , 36, 2024.

Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey J Gordon. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Information Processing Systems , 33:19276-19289, 2020.

Kai Sheng Tai, Peter D Bailis, and Gregory Valiant. Sinkhorn label allocation: Semi-supervised classification via annealed self-training. In International Conference on Machine Learning , pages 10065-10075. PMLR, 2021.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4149-4158, 2019.

Nguyen Thai-Nghe, Zeno Gantner, and Lars Schmidt-Thieme. Cost-sensitive learning methods for imbalanced data. In The 2010 International joint conference on neural networks (IJCNN) , pages 1-8. IEEE, 2010.

Antonio Torralba and Alexei A. Efros. Unbiased look at dataset bias. In CVPR 2011 , pages 1521-1528, 2011. doi: 10.1109/CVPR.2011.5995347.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

Haobo Wang, Mingxuan Xia, Yixuan Li, Yuren Mao, Lei Feng, Gang Chen, and Junbo Zhao. Solar: Sinkhorn label refinery for imbalanced partial-label learning. Advances in Neural Information Processing Systems , 35:8104-8117, 2022.

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems , 32, 2019.

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 , 2021.

Sheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. Unveiling selection biases: Exploring order and token sensitivity in large language models. arXiv preprint arXiv:2406.03009 , 2024.

Ruicheng Xian, Lang Yin, and Han Zhao. Fair and optimal classification via post-processing. 2023.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition , pages 3485-3492. IEEE, 2010.

Chuyu Zhang, Hui Ren, and Xuming He. P 2 ot: Progressive partial optimal transport for deep imbalanced clustering. arXiv preprint arXiv:2401.09266 , 2024.

Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations , 2023.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. Advances in Neural Information Processing Systems , 36, 2024.

Appendix

The appendix contains glossary, algorithm details, proofs, and detailed experimental results. The glossary contains a convenient reminder of our terminology (Appendix A). Appendix B provides more related works and discussion about the relationship between our work and related papers. Appendix C describes the relevant algorithms used in our experiments, including Prior Matching [37] and BBSE [35]. Appendix D provides the proofs of theorems that appeared in Section 4. Finally, we give more details and analysis of the experiments and provide additional experiment results in Appendix E.

A Glossary

The glossary is given in Table 6 below.

∈

B Extended Related Work

Improving Zero-shot Classification at Inference Time. As zero-shot classification has gained popularity, several works have been developed to improve zero-shot classification at inference time. Chuang et al. [13]; Adila et al. [1] use vector projection methods to remove spurious correlations at inference time. Menon and Vondrick [40]; Novack et al. [43]; An et al. [4] augment prompts with language models and combine their classification output to improve zero-shot performance. Roberts et al. [51] uses additional information of label space geometry to extend model pre-trained on the label subset to broader use-cases. While these works try to improve zero-shot classification at inference time in common, the main difference is that our method tackles the inherent class prior of zero-shot models.

Label Shift Adaptation. Label shift adaptation methods are designed to address the negative impacts arising from changes in the label distribution. These methods typically follow a two-step process [35; 6; 24]. The first step involves estimating the label distribution within the target dataset using labeled data from the source distribution and unlabeled data from the target distribution. Next, the prediction scores are reweighted using the estimated target label distribution and the source label

distribution. However, the standard approach requires access to the labeled source distribution data, which is usually not possible in zero-shot classification scenarios. OTTER provides a solution decoupled from the source data distribution, overcoming this limitation.

Improving Zero-shot Classification using Prior. Several studies have explored leveraging prior information to enhance zero-shot classification, even in the absence of access to source distributions. In the context of prompt-based zero-shot models, prior matching [37] employs word prior distribution to alleviate word bias inherent in pretraining data. We adopted their adaptation method as a baseline. Similarly, Kahana et al. [31] develop adaptation layers trained using priors, aiming to maintain proximity to the original scores. However, both approaches entail training additional layers and necessitate hyperparameter tuning, which may pose challenges in the context of zero-shot predictions. In contrast, OTTER presents a straightforward and efficient adaptation method to new label distributions without the need for any hyperparameter tuning, backed by theoretical guarantees.

Leveraging Optimal Transport for Enhanced Pseudo-labeling and Classification. A number of studies have explored the enhancement of pseudo-labeling and classification tasks through optimal transport, using label distribution specifications, in a similar spirit to our work, but within different contexts. Tai et al. [56] uses optimal transport to allocate pseudo-labels to unlabeled datasets based on the label distribution specification in the semi-supervised setup. Wang et al. [62] deals with long-tail distribution in the partial-label learning setup based on optimal transport. Zhang et al. [68] uses partial optimal transport as a pseudo-labeling based approach for deep imbalanced clustering, progressively expanding the labeled sample proportion. Guo et al. [26] reweights the training dataset to match the label distribution specification using optimal transport. This work mainly deals with the class imbalance problem in the training step. Shi et al. [54] studies classification from a matching perspective, revealing the connection between softmax loss and inverse optimal transport and suggesting a new loss function to address long-tail distributions. Their analysis provides useful insights for OTTER - why cost matrix induced by pre-trained models can be useful in the inference step. Xian et al. [66] uses optimal transport as a postprocessing method to guarantee demographic parity. While sharing aspects of the approach, our work addresses class bias in zero-shot models. Peng et al. [47] used optimal transport to handle long-tail recognition with a learned cost matrix. Our study provides a theoretical basis for understanding their empirical results. Chang et al. [9] employs optimal transport to detect common and private classes between the source and the target domain, under the universal domain adaptation setting, where knowledge is transferred from source to target domain without any constraints on the label sets. In the context of zero-shot classification, there is no need to manage label space disparities between the source and target domains. Instead, the main concern of zero-shot classification is dealing with the distribution shift between the pretraining dataset and the downstream task. We tackle the label distribution shifts using optimal transport.

Class Imbalance. Class imbalance problems occur when the number of instances across different classes in a dataset is disproportionately distributed. This imbalance can severely bias the traininig process of a machine learning model, leading to poor generalization performance, especially for the minority classes. It has been extensively studied in the context of traditional machine learning [23]. Oversampling [10] and cost-sensitive learning [58] are well-known approaches to address class imbalance. Nonetheless, the inherent nature of class imbalance in pretraining datasets introduces a distinct set of challenges, especially when attempting to rectify such biases within the context of zero-shot classification scenarios.

C Algorithm details

Prior matching [37] proposed prior matching as a reweighting method for prompt-based classifiers to mitigate word bias - the distribution shift between pretrained language models' word prior and the class priors in the downstream task. We use it as a zero-shot model adaptation method given a class balance estimation.

Define reweighted probability scores of P θ with r as P θ,r ( Y = j | X = x i ) = r j P θ ( Y = j | X = x i ) ∑ K j ′ =1 r j ′ P θ ( Y = j ′ | X = x i ) . Ideally, we hope to estimate the weight vector r ∗ ∈ R n such that reweighted scores P θ,r ∗ ( Y = j | X = x i ) maximizes the accuracy in the target distribution. Since the labels are not given, this is impossible. Instead, prior matching matches the label distribution of

Algorithm 2 Black Box Shift Estimator (BBSE)

Input: Source input data X s = { x s 1 , . . . , x s m } , Source labels Y s = { y s 1 , . . . , y s m } , Target input data X t = { x t 1 , . . . , x t n } , model prediction distribution P θ

Estimate confusion matrix V such that A jk = 1 m ∑ m i =1 P θ ( Y = k | X = x s i ) Estimate the refined target class balance ˆ ν t = A ˜ ν t

Estimate the source class balance ν s such that ν s j = ∑ n i =1 P θ ( Y = j | X = x s i ) m Compute the naive target class balance ˜ ν t such that ˜ ν t j = ∑ n i =1 P θ ( Y = j | X = x t i ) n

Return ˆ ν

predicted classes with the class balance estimate ˆ ν , i.e.

∣ ∣ It can be solved using the standard optimization techniques - we used [38]. While this is equivalently effective with OTTER when properly optimized, we found that the temperature parameter and learning rate crucially affect the final accuracy, making it less ideal for the zeroshot adaptation. We used the grid search with the small validation set (10 samples per class) in each task to select hyperparameters. The hyperparameter ranges are as follows.

Temperature: [1e-3, 1e-4, 1e-5, 1e-6, 1e-7]

Learning rate: [1e-3, 1e-4, 1e-5, 1e-6, 1e-7]

Black Box Shift Estimation (BBSE) Label shift adaptation methods [35; 6; 24] aims to estimate the class balance in the target distribution using the labeled source distribution data and the unlabeled target distribution data. We use Black Box Shift Estimation (BBSE) to estimate the class balance in the downstream task. Algorithm 2 describes the procedure. Note that the derivation of this algorithm depends on the label shift assumptions, thus the label distribution estimation with real-world data can be heavily biased.

D Theory details

D.1 Proof of Theorem 4.1

/negationslash

Proof. Suppose ˆ y OT i = ˆ y ZS i for some i ∈ [ n ] . It implies ∑ n i =1 -log P θ ( Y = ˆ y OT i | X = x i ) < ∑ n i =1 -log P θ ( Y = ˆ y ZS i | X = x i ) . However, this is a contradiction since, for any i ∈ [ n ] , ˆ y ZS i = arg max j ∈ [ K ] P θ ( Y = j | X = x i ) , thus -log P θ ( Y = ˆ y ZS i | X = x i ) ≤ -log P θ ( Y = j | X = x i ) for all j ∈ [ K ] , which results in ˆ y OT i = ˆ y ZS i .

D.2 Proof of Theorem 4.2

To prove Theorem 4.2, we show a specific form of invariance property of optimal transport first.

Theorem D.1. Suppose π ∗ = arg min γ ∈ Π( µ,ν ) 〈 γ, C 〉 and E is a columnwise perturbation, i.e.,

where 1 denotes n dimensional vectors and /epsilon1 1 , . . . , /epsilon1 K are constants. Then the perturbed cost matrix ˜ C = C + E , then π ∗ is also an optimal transport map with respect to the cost matrix ˜ C .

Proof. By the optimality condition, we have

for any π ∈ Π( µ, ν ) . Then,

which is

Thus,

This theorem is also valid for row-wise perturbations as well with a similar proof. Consequently, a straightforward implication is that

Corollary D.2. Suppose π ∗ = arg min γ ∈ Π( µ,ν ) 〈 γ, C 〉 , E is a columnwise perturbation and F is a row-wise perturbation, such that

where 1 denotes n dimensional vectors with 1s, and /epsilon1 1 , . . . , /epsilon1 K , η 1 , . . . , η K are constants. Suppose the perturbed cost matrix is defined as ˜ C = C + E + F , then π ∗ is also an optimal transport map with respect to the perturbed cost matrix ˜ C .

Now we provide the proof of Thoerem 4.2.

Proof. Given the posteriors in the target distribution can be defined as C ∗ ij = -log P t ( Y = j | X = x i ) . From

we can see that

where E · j = log P s ( Y = j ) -log P t ( Y = j ) , F i · = -log P s ( X = x i ) + log P t ( X = x i ) . And, assuming ν j = 1 n ∑ n i =1 ✶ [ˆ y ∗ i = j ] , where ˆ y ∗ is the Bayes classifier prediction such that

Then, optimal transport solution

leads to Bayes classifier predictions by Theorem 4.1.

D.3 Proof of Theorem 4.3

The proof of Theorem 4.3 relies on the following result of [52].

Lemma D.3 ([52], Corollary 3.1.) . Let the primal linear programming problem be

Let ¯ z, ¯ w be the primal, dual solution. And, let the perturbed primal linear programming problem be

Let ˆ z, ˆ w be the corresponding primal, dual solution.

Suppose that the primal and dual problems are solvable. Then,

where 1 ≤ p ≤ ∞ and κ is the Hoffmann constant determined by p, G, g . [30].

This Lemma provides a bound for error in the solution of the perturbed linear program. Since discrete optimal transport can be translated to standard linear program, we obtain Theorem 4.3 by plugging in the definitions.

Proof of Theorem 4.3 A discrete optimal transport problem

constraints are converted to stacked inequalities. We have noisy cost matrix and label distribution in our setting, which leads to the perturbation on cost matrix C and ν such that the perturbed cost matrix is ˆ C = C +∆ , the perturbed label distribution ˆ ν = ν +∆ , such that ˆ g = g +∆ where can be written as a linear program

C ν g     . Since we don't have perturbation on the constraint matrix G , ˆ G = G . By plugging

and its dual be and its dual be

Optimality of ˆ z, ˆ w in the perturbed problem.

Then, we have

Then,

Plugging in ¯ z = vec ( π ) , ˆ z = vec (ˆ π ) , ∆ ν = ˆ ν -ν, ∆ C = ˆ C -C , and rearranging, we obtain

We use the definition of the Hoffman constant κ from [52]. Computing Hoffman constant or even bounding it has been a long-standing problem [5; 45; 46]. However, it has been shown that the Hoffman constant is a finite real number [52], and specifically under our problem setup, it is independent from perturbations and only related to original optimization problem. This suggests the possibility to regularize the parameters C, G, g in the original problem such that κ does not depend on the dimensionality of cost matrix or target distribution. We leave further exploration of κ in this context as future work.

D.4 Bounding label distribution estimation errors in few-shot learning

In few-shot learning, we assume that a few labeled samples per class are given. They can be used for estimating label distribution in the target distribution using label shift estimation methods [35; 6; 24]. They comes with the sample complexity analysis under the label shift assumptions, which can be used to obtain bound the label distribution estimation errors.

Lemma D.4. Let m and n denote the number of few-shot learning data and test datapoints, w i = ν i /ν s i and ˆ w i = ˆ ν i /ν s i . Also let σ min be the smallest eigenvalue of the covariance matrix V ˆ y,y where [ V ˆ y,y ] i,j = P s ( f ( x ) = i, y = j ) . For m> 80 log( m ) σ -2 min and constant c > 0 , the perturbation ∆ ν may be bounded as with probability at least 1 -3 Km -10 -2 Kn -10 .

The proof of Lemma D.4 relies on the following result of [34].

Lemma D.5. Assume that

∀ x ∈ X and ∀ y ∈ Y , P s ( x | y ) = P t ( x | y ) ,

if P t ( y ) > 0 then P s ( y ) > 0 ∀ y ∈ Y , and

3. the expected confusion matrix C s ( f ) = P s ( f ( x ) , y ) ∈ R K × K for classifier f is invertible.

Then, there exists a constant c > 0 such that for all m > 80 log( m ) σ -2 min , with probability at least 1 -3 Km -10 -2 Kn -10 ,

Proof of Lemma D.4. Where all norms are Euclidean unless otherwise denoted, we have that

for element-wise multiplication operation ⊗ . Further, where the last line follows from Lemma D.5 with probability at least 1 -3 Km -10 -2 Kn -10 .

D.5 Bounding the perturbation on cost matrix

Further, we can bound [ vec (∆ C )] + by the Total Variation Distance (TVD) between P s and P θ as follows.

Lemma D.6. Let τ = 1 2 || P s -P θ || 1 denote the Total Variation Distance between P s and P θ and define min( C ) = min i,j C ij . Then,

Proof of Lemma D.6. For each element ∆ ( ij ) C of ∆ C , we have that

For i, j such that P θ ( Y = j | X = x i ) ≤ P s ( Y = j | X = x i ) , clearly ∆ ( ij ) C ≤ 0 and so [ vec (∆ ( ij ) C )] + = 0 .

Otherwise, for i, j such that P θ ( Y = j | X = x i ) > P s ( Y = j | X = x i ) , it follows that ∆ ( ij ) C > 0 and

For each i ∈ [ m ] , there are at most K -1 possible j such that P θ ( Y = j | X = x i ) > P s ( Y = j | X = x i ) , because ∑ j ∈ [ K ] P θ ( Y = j | X = x i ) = ∑ j ∈ K P s ( Y = j | X = x i ) . Therefore, there are at most m ( K -1) pairs ( i, j ) ∈ [ m ] × [ K ] such that 0 < ∆ ( ij ) C ≤ log ( τ min( C ) +1 ) . Thus,

D.6 Proof of Theorem 4.4

We provide the proof of Theorem 4.4.

Proof. By assumption, we have P θ ( Y | X ) = P s ( Y | X ) . By the result of Theorem 4.2, y OTTER samples are generated by arg max j ∈ [ K ] P t ( Y = j | X = x ) . Suppose r ∗ = P t ( Y ) /P s ( Y ) . Then,

Thus, we have

which implies r ∗ is an optimal parameter.

E Experiment details

E.1 Datasets

Zeroshot image classification datasets We use CIFAR10, CIFAR100 [33], Caltech101 [22], Caltech256 [25], Food101 [8], STL10 [16], SUN397 [67], Flower102 [42], EuroSAT [27], OxfordIIIT Pet [44], Stanford Cars [32], DTD [14], CUB [61], ImageNet [18], ImageNet-r [29], and ImageNet-Sketch [63].

Zeroshot text classification datasets We use Amazon [41], Gender [20], CivilComments [7], and HateXplain [39].

E.2 Zero-shot classification setup

Image zero-shot classification For zero-shot image classification, we emply CLIP [49] models. We used 'a photo of a [CLASS]" prompt. Scores are computed by s θ ( x i , j ) = P θ ( Y = j | X = x i ) =

exp(cos( f ( x i ) , g ( y j )) /τ ) ∑ K j ′ =1 exp(cos( f ( x i ) , g ( y j ′ )) /τ ) for image x i regarding the label j , given the image encoder f , the text encoder g . Cost matrix is constructed by C = [ C ij ] i ∈ [ n ] ,j ∈ [ K ] , where c ij = -log s θ ( x i , j ) . We run Algorithm 1 with the true class balance of the test dataset.

Text zero-shot classification We employ BERT and text-embedding-ada-002 sentence embeddings for text classification [50]. This process parallels the methodology used in image zero-shot classification - we compute the prediction scores from the cosine similarity and then construct the cost matrix with the negative log probabilities.

E.3 Detailed experiment results of Section 5.1

Ablation on zero-shot models For the ablation study on zero-shot models, we provide experimental results with varying architectures, given the exact prior. Table 8, 9 show the image zero-shot classification results, and Table 10 shows the text zero-shot classification results. We also provide another baseline results with CLIPPR [31], which uses the label distribution for adapter training. CLIPPR is similar with Prior Matching in the point that it requires adapter training, but it has more adapter layers and additional loss function to make the predictions stick to the original prediction scores. While the performance gain varies, we can observe that OTTER is effective for the most cases.

/negationslash

Ablation on the class balance specification We conducted a semi-synthetic experiment to investigate the sensitivity to the label distribution specification error in real datasets. We generate the noisy label distribution specification and see how the accuracy changes. We control the noise in the label distribution specification as follows. Given the true class balance ν ∗ , first we make adversarial class balance ν adv such that ν adv j ∗ = 1 for j ∗ = arg min j ∈ [ K ] ν ∗ j and ν adv j = 0 for j = j ∗ . To measure distance between class balance specification and true class balance, we use the total variance TV ( ν, ˆ ν ) = 1 2 || ν -ˆ ν || 1 . Next, we intepolate ν ∗ and ν adv such that TV ( ν ∗ , ν α ) = α , by ν α = (1 -α TV ( ν ∗ ,ν adv ) ) ν ∗ + α TV ( ν ∗ ,ν adv ) ν adv . We set the interval of alpha as 0.01 and vary it up to 0.2.

Figure 5 shows the result. We observe the sensitivity to the label distribution specification error varies depending on the datasets, but generally we can observe that the accuracy degrades linearly proportionally to the class balance error. While the result may vary depending on the interaction between class balance error and calibration error in cost matrix, we can expect performance improvement if the class balance specification is good enough.

Acc. (%)

0.00

0.05

0.10

0.15

0.20

70

75

80

85

90

CIFAR10

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

55

60

65

CIFAR100

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

70

80

90

Caltech101

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

70

80

Caltech256

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

70

80

90

Food101

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

80

90

STL10

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

40

45

50

SUN397

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

50

60

70

Flowers102

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

32.5

35.0

37.5

40.0

42.5

EUROSAT

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

70

80

90

OxfordIIITPet

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

40

45

50

55

60

Stanford-Cars

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

19

20

21

Country211

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

35

40

DTD

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

40

45

50

cub

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

50

55

60

ImageNet

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

60

65

70

ImageNet-r

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

36

38

40

42

44

ImageNet-Sketch

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

70

80

90

Amazon

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

75

80

85

90

Gender

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

50

60

70

80

CivilComments

ZS

OTTER

0.00

0.05

0.10

0.15

0.20

32

34

HateXplain

ZS

OTTER

TV(

(

Y

),

(

Y

))

Pt

Pt

Inference time comparison To show that the additional computation complexity induced by OTTER is not heavy, we measured the time consumption (in seconds) for the inference step in the experiments in Section 5.1, with the pre-computed embeddings. Table 11 presents the result. Time reduction column represents the time reduction rate of OTTER compared to PM. Measurements were

taken using a machine equipped with an Intel® Core™ i7-11700K @ 3.60GHz processor, 64GB RAM, and NVIDIA GPU RTX-4090. For most cases ( n < 30000 ), our method takes less than 1 second, while the prior matching baseline takes more than 3 seconds. It's worth noting that the time consumption for computing embeddings is more substantial; even 10 seconds is negligible compared to the embedding time consumption (ranging from 5 to 30 minutes for each evaluation set), which is common for all inference conditions.

Ablation on prompts Recent studies have demonstrated the efficacy of enhancing prompts as a means to improve zero-shot models [70; 40]. In order to further illustrate the potential enhancements offered by OTTER beyond enhanced prompts, we reproduced Menon and Vondrick [40]'s approach (Classification by Description, CD), which employs multiple prompts generated by language models and takes max scores of them for each class. We applied OTTER to CD. The results of this experiment are summarized in Table 12. As anticipated, OTTER exhibits enhancements in zero-shot classification, even when prompt sets are refined using language models.

E.4 Detailed experiment results of Section 5.3

Label distribution estimation errors We report the label distribution estimation errors in Section 5.3. As a metric, we use the total variance TV ( ν, ˆ ν ) = 1 2 || ν -ˆ ν || 1 . We use zeroshot prediction scores and linear probing prediction scores for BBSE. ˆ ν zs denotes the estimated class balance based on zero-shot prediction scores, and ˆ ν lp represents the estimated class balance based on linear probing prediction scores.

Table 13 shows the result. We can see that total variation decreases with linear probing in image classification tasks since they reduces the violation of label shift assumptions. However, total variation increases in text classification tasks due to the small number of labeled sample size, following the size of label space ( K = 2 or 3 ). Accordingly, we can expect OTTER will be more useful with linear probing, and just rebalancing zero-shot predictions with OTTER could be enough for text classification tasks.

Ablation experiments on linear probing We provide full results of Section 5.3. Specifically, we additionally report the results of combination with linear probing in text classification tasks and the results of zero-shot classification results in image classification tasks.

The results are presented in Table 2. While OTTER often provides additional improvement over LP, zero-shot classification was a strong baseline in image classification tasks. Meanwhile, class balance adaptation in text classification tasks is effective in all cases, giving a significant improvement over zero-shot predictions.

0.10

0.15

0.20

TV(

Pt

(

Y

),

Pt

(

Y

))

Estimation Error

25

50

75

100

60.0

62.5

65.0

Acc. (%)

ZS

OTTER

(a) CIFAR100

0.350

0.375

0.400

0.425

TV(

Pt

(

Y

),

Pt

(

Y

))

Estimation Error

25

50

75

100

16

18

20

Acc. (%)

ZS

OTTER

(b) Country211

0.04

0.06

0.08

TV(

Pt

(

Y

),

Pt

(

Y

))

Estimation Error

25

50

75

100

80

90

Acc. (%)

ZS

OTTER

(c) Amazon

0.2

0.3

TV(

Pt

(

Y

),

Pt

(

Y

))

Estimation Error

25

50

75

100

50

60

70

Acc. (%)

ZS

OTTER

(d) CivilComments

25

50

75

100

60.0

62.5

65.0

67.5

70.0

72.5

Acc. (%)

ZS

OTTER

LP

LP+OTTER

(a) CIFAR100

20

40

60

80

100

12

14

16

18

20

22

24

Acc. (%)

ZS

OTTER

LP

LP+OTTER

(b) Country211

20

40

60

80

100

75

80

85

90

95

Acc. (%)

ZS

OTTER

LP

LP+OTTER

(c) Amazon

20

40

60

80

100

50

55

60

65

70

75

Acc. (%)

ZS

OTTER

LP

LP+OTTER

(d) CivilComments

Ablation experiments on the number of examples per class Few-shot adaptation scenario assumes we have access to labeled data to estimate the target distribution. We hypothesize that an increase in the number of labeled samples enhances the accuracy of the class balance estimation, thereby improving the performance of OTTER. To test this hypothesis, we use few-shot adaptation in image and text classification tasks, without linear probing. The experiment varies the number of samples per class from 10 to 100, anticipating a reduction in class balance estimation error and an improvement in OTTER's accuracy with the increase in labeled samples.

The results, as depicted in Figure 6, corroborate our hypothesis. It is evident that the error in class balance estimation diminishes with an increasing number of samples, leading to a subsequent enhancement in the accuracy of OTTER.

Comparison between OTTER and Linear Probing with varying number of classes In the few-shot adaptation scenario, we explored three approaches: OTTER, linear probing (LP), and a combination of LP + OTTER. We formulated two hypotheses. The first posits that OTTER might outperform LP, particularly in situations with a limited number of samples. The second hypothesis suggests that OTTER could provide further enhancements to LP even when LP already surpasses the naive version of OTTER. This experiment was conducted using the same setup as the previous one.

The results, displayed in Figure 7, reveal several insights regarding our hypotheses. To begin with, OTTER demonstrates performance on par with LP, especially in scenarios with fewer samples. Interestingly, OTTER achieves superior accuracy compared to LP in datasets like Amazon and CivilComments, characterized by a small number of classes ( K = 2 ), resulting in a relatively low total sample count. Furthermore, it is observed that incorporating OTTER into LP leads to an average increase in accuracy.

E.5 Detailed experiment setup and results of Section 5.4

Class hierarchy We used the following superclasses and subclasses classes for the proof of concept.

0.0

0.1

0.2

0.3

0.4

0.5

0.6

TV(

Pt

(

Y

),

Ps

(

Y

))

70

80

90

Acc(%)

Naive

Bayes

OTTER

R-OTTER

fish: aquarium fish, flatfish, ray, shark , trout

tree: maple tree, oak tree, palm tree, pine tree, willow tree

Class balance estimation error We report the class balance estimation error in Section 5.4. Table 14 shows the total variation between true class balance and estimated class balance. We can expect a significant accuracy improvement for RN50, RN101, and ViT-B/16 based on this table.

E.6 Synthetic experiments with R-OTTER

We validate our theoretical result (Theorem 4.4) by testing R-OTTER in a fully controllable setting.

Setup and Procedure. We use our synthetic experiment setup (Section 5.2) with perturbation noise δ = 0 and label distribution α = 0 . Additionally, we generated a validation set that follows the same distribution as the test set. After learning the reweighting parameter r in the validation set using y otter as pseudolabels, we evaluated R-OTTER on the test set, comparing the results to those of zero-shot and OTTER. We expect that, if successful, R-OTTER will similarly gain improvement over zero-shot when the source and target distributions increasingly differ.

Results Figure 8 presents the experimental result. As expected, R-OTTER presents the experimental results. As expected, R-OTTER performs closely to the Bayes optimal solution, demonstrating its effectiveness, though it exhibits slight suboptimality due to generalization issues.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We substantiate every claim made in the introduction in the experimental result.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes. We discuss the potentials and limitations across several sections.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, we included the full set of assumptions and a complete (and correct) proof in Section 4.2.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provided detailed exhaustive experimental hyperparameters and setup.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We submit a zip folder of our code, and will release github link upon publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, we provide all hyperparameters in the Appendix. We provide all data split and details in the main experimental result section

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Yes, we provide error bars as much as possible, unless repeating an experiment is expensive.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Provided in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We adhere to the NeurIPS code of ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: In our knowledge, there is no additional societal impact beyond that of LLM's. This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. As of now, we are not aware of additional potential societal impacts beyond the typical implications associated with zero-shot models.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for

disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release a new pre-trained model.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all dataset and model original owners.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to address the issue of label distribution mismatch in zero-shot models, which is a common problem due to the imbalance in the pretraining datasets. The proposed method uses optimal transport to adjust the predictions of pretrained models based on an estimated downstream label distribution, without requiring additional training or access to labeled downstream data.\n\nSTRENGTHS:\n1. The use of optimal transport to handle label distribution mismatch in zero-shot models is a novel and elegant solution.\n2. The paper provides a solid theoretical foundation for the proposed method, including characterizations of the improvement under mild conditions and error bounds for misspecification. \n3. Extensive empirical validation across a wide range of image and text classification tasks demonstrates the effectiveness of OTTER.\n\nWEAKNESSES:\n1. dependence on estimated label distribution: the method relies on an accurate estimate of the downstream label distribution. In practice, obtaining a reliable estimate may be challenging, and errors in this estimate could impact the performance of OTTER.\n2. evaluation on diverse datasets: while the paper includes a variety of datasets for validation, a more detailed analysis of performance across different types of datasets (e.g., varying in size, complexity, and imbalance levels) would provide a deeper understanding of the method's strengths and limitations.\n3. comparison with recent methods: the paper compares OTTER with some existing baselines, but it could benefit from a more comprehensive comparison with the latest state-of-the-art methods in label distribution adaptation and zero-shot learning. There are many training-free or test-time adaption zero-shot baselines[a][b][c]. How does OTTER perform compared to these methods?\n4. scalability: although described as lightweight, the paper does not thoroughly discuss the scalability of OTTER for very large-scale datasets.\n\n\n[a] Mirza, Muhammad Jehanzeb, et al. \"Lafter: Label-free tuning of zero-shot classifier using language and unlabeled image collections.\" NIPS 2024.\n\n[b] Zhao, Shuai, et al. \"Test-time adaptation with clip reward for zero-shot generalization in vision-language models.\" arXiv preprint arXiv:2305.18010.\n\n[c] Abdul Samadh, Jameel, et al. \"Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization.\" NIPS 2024.\n\nQUESTIONS:\nThe paper presents a compelling and well-validated approach to address label distribution mismatch in zero-shot models using optimal transport. Its innovative method, strong theoretical foundation, and significant empirical improvements make it a valuable contribution to the field. However, further exploration of its limitations, more comparisons with recent methods, and practical implementation details would enhance its impact and utility.\n\nLIMITATIONS:\nThe authors do not claim limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nZero shot classification suffers from label distribution mismatch. This paper suggest to adjust pretrained model prediction via optimal transport. By showing the optimal transport prediction is equivalent to the bayes optimal classifier's output theoretically and good model performances on various experimental settings empirically, the paper shows the validity of its proposed method.\n\nSTRENGTHS:\n- The problem is well formulated\n- Various experiment\n\nWEAKNESSES:\n- Novelty is limited. This paper suggests applying OT for managing label distribution shift. The class imbalance for pretrained network prediction is already well known, and OT has been already applied for various distribution shift cases. Therefore, naively applying OT for the biased pretrained model prediction is imaginable.\n- The proposal saying that Bayes optimal classifier can be derived through optimal transport is inadequate and too strong since it requires a true cost matrix. If $P_t(Y|X)$ is accessible, we don't even need to use the optimal transport.\n- Please link theorems of the main paper and proofs in the appendices for readability.\n- I don't think figure 2 shows OTTER remains unaffected. The acc is deteriorated from 90 to 7X, meaning about 15%p decrease.\n\nQUESTIONS:\n- Considering each cell of the cost matrix $C$, it is a loss (or the classifier score) calculated from the pretrained network. It means the cost matrix is biased to the source data distribution. In that sense, how the optimal transport can be considered as assigning true label?\n- How can the true label distribution of the target, $\\nu*$, be accessible?\n- Can the author show more details for the equation of section 4.1? I cannot understand how to remove $P_s(X)$ and $P_t(X)$\n- In table 1, what happened to Caltech256 Prior matching? Why the performance is far worse than zero-shot?\n- In figure 2, Why OTTER shows acc increase when the total variation distance between the source and the target increases from 0.4 to 0.8? With larger distribution shift, acc is supposed to deteriorate. If not, the synthetic experiment may be poorly modeled.\n- How long does it take to apply OTTER? I suppose it will take quadratically proportional to the dataset size.\n\nLIMITATIONS:\n.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nZero-shot models such as CLIP suffer from imbalanced predictions inherited from the uncurated pre-training data. This paper proposes adjusting the predictions of zero-shot models via optimal transport. This method requires only the downstream data and the label prior of the target distribution. Under the assumption that the conditional likelihood is unchanged between the pre-training data and the downstream data, the authors prove that the induced predictions are Bayes Optimal. The authors validate the effectiveness of the proposed method on several benchmark tests.\n\nSTRENGTHS:\nThe paper is well-organized and the presentation is clear. The idea of using optimal transport to adjust zero-shot predictions on downstream data is novel to me. Besides, the authors prove that the resulting predictions are Bayes optimal and justify the cost matrix using posteriors of zero-shot models. I have coarsely checked the proofs and find them convincing.\n\nWEAKNESSES:\nMajor:\n\n(1) Related works are missing. Some papers have addresses the issue of removing the label bias in zero-shot models. For instance, [1][2] use a subset of the pre-training data to mitigate the bias. [3] focuses on only using the labeled downstream data to address imbalanced predictions and proposes two methods. The authors should consider including these prior works.\n\n[1] A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models.\n\n[2] The neglected tails of vision-language models.\n\n[3] Generalized Logit Adjustment: Improved Fine-tuning by Mitigating Label Bias in Foundation Models.\n\n(2) The proposed method is transductive and requires the entire test dataset to perform optimal transport. I am wondering if OTTER can estimate the label distribution of pre-training data $\\pi$ from unlabeled downstream training data and then the debiasing procedure can be performed by subtracting the logarithm $\\ln \\pi$ from the zero-shot predictions, as described in [3] (https://arxiv.org/pdf/2310.08106), without needing to operate on the entire test dataset\n\nMinor:\n\n(1) Duplicate citations: [29] and [30].\n\n(2) Line 120: should reference Algorithm 1.\n\nQUESTIONS:\nPlease refer to the weaknesses, and I am willing to raise the score if the major weaknesses are addressed.\n\nAdditionally, I have a question to discuss with the authors. Both [3] and OTTER use downstream data to generate Bayes optimal predictions based on the assumption that $P(x|y)$ remains unchanged between the pre-training distribution and the downstream distribution. In practice, this assumption might not hold. For instance, the pre-training data might mostly consist of real photos, while some test benchmarks might be sketches, e.g., ImageNet-Sketch. Under this situation, is your method theoretically sound in mitigating the pre-training bias?\n\nLIMITATIONS:\nLimitations have been discussed and no negative societal impact has been identified.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}