OTTER: Effortless Label Distribution Adaptation of Zero-shot Models

Authors: changho  shin
jitian  zhao
sonia  cromp
harit  vishwakarma
frederic  sala
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 23:45:53

Abstract: Popular zero-shot models suffer due to artifacts inherited from pretraining. One particularly detrimental issue, caused by unbalanced web-scale pretraining data, is mismatched label distribution. Existing approaches that seek to repair the label distribution are not suitable in zero-shot settings, as they have mismatching  requirements, such as needing access to labeled downstream task data or knowledge of the true label balance in the pretraining distribution. We sidestep these challenges and introduce a simple and lightweight approach to adjust pretrained model predictions via optimal transport. Our technique requires only an estimate of the label distribution of a downstream task.  Theoretically, we characterize the improvement produced by our procedure under certain mild conditions and provide bounds on the error caused by misspecification. Empirically, we validate our method in a wide array of zero-shot image and text classification tasks, improving accuracy by 4.8% and 15.9% on average, and beating baselines like prior matching---often by significant margins---in 17 out of 21 datasets.

Main paper: 
 introduction: 
 Figure 1 : Label distribution mismatch example in zero-shot classification. In the Oxford-IIIT-Pet dataset, the ground-truth labels are uniformly distributed, while zero-shot models exhibit biased predictions toward certain classes. This bias is influenced by the distribution of labels in the pretraining task.Zero-shot models are popular but struggle with biases inherited from their large pretraining datasets [21; 59; 2] . In particular, zero-shot classification is strongly biased by the label distribution of the pretraining task. When the label distribution of the downstream task differs from pretraining, the performance of zero-shot classifiers suffers greatly. For example, Figure 1 illustrates the effects of mismatched distributions on a pet image classification task. Two CLIP models (RN50, and ViT-B/16) produce biased predictions on the Abyssinian and Persian classes. Furthermore, datasets with a large number of classes, such as ImageNet, may contain both extremely common and very rare classes, resulting in an outsized probability that a zero-shot model will predict some classes over others. As a result, even large models intended for use in zero-shot settings, such as CLIP [49] , naturally have a label distribution mismatch between pretraining data and downstream tasks.Existing methods that seek to address label distribution make strong assumptions or have expensive requirements. For example, to fine-tune a model, we must obtain a labeled fine-tuning dataset of adequate size, then obtain the time and compute to further train the model. To perform label shift adap-tation techniques, we must know the true label distribution of the pretraining distribution-difficult to impossible for real-world tasks.Can we deal with label distribution mismatch without additional training or access to groundtruth downstream task information? While seemingly challenging, one cause for optimism is the observation that zero-shot models still give relatively high prediction probabilities for correct classes, though classes common in pretraining tend to have relatively inflated scores overall. Intuitively, the model has already learned to identify examples of its downstream classes (and so does not require further training) and is already impacted by the pretraining label distribution (and so does not need access to the ground-truth pretraining label distribution). Instead, the model's prediction probabilities must be adjusted based on an estimated downstream label distribution specification.To perform this label distribution adjustment, we view zero-shot learning through the lens of optimal transport (OT) and develop a technique called OTTER (Optimal TransporT adaptER). This OT-based approach offers a systematic way to rebalance predicted labels: data points are transported to optimal downstream classes, minimizing the overall cost in accordance with the estimated the downstream label distribution specifications.Theoretically, we show that optimal transport given the true label distribution of the downstream can recover the Bayes-optimal classifier under mild conditions. Additionally, we provide error bounds on our adaptation method for misspecification. We provide synthetic experiments validating our theoretical claims. In real-world data settings, we validate our method on a wide variety of image and text classification tasks, showing 4.8% and 15.5% accuracy improvement on average in image and text zero-shot classification tasks, respectively. Finally, we evaluate our method in few-shot adaptation scenarios, where OTTER provides further improvements when combined with linear probing. Our contributions include:• OTTER, an algorithm to deal with label distribution mismatch at inference time via optimal transport, • Theoretical results showing the effectiveness of our method, including the ability to recover the Bayes-optimal classifier and a sensitivity analysis with respect to the label distribution specification estimation, • Extensive empirical results on zero-shot classification for text and image datasets, showing accuracy improvements of up to 25%, • Experimental results demonstrating the applicability of OTTER to few-shot settings, showing accuracy improvements of up to 15%, even with noisy label distribution specifications, • Extensions of OTTER to leverage label hierarchy information or relax the batched prediction requirement and • Application to LLM selection bias mitigation. 
background and problem formulation: 
 Before presenting OTTER and providing our results, we introduce some crucial background and describe the problem setting formally.We briefly describe zero-shot models, the technical tool we use (optimal transport), along with other techniques that seek to address shifts. We have extended related work, including in-depth characterizations and comparisons with related methods, in Appendix B.Zero-shot Models. Zero-shot classification, popularized by models such as CLIP [49] , is a powerful paradigm that enables prediction on downstream tasks without additional fine-tuning. Image, language, and multimodal models have been increasingly employed for zero-shot prediction [64; 36] . These models undergo extensive pretraining on massive datasets with concept and label spaces that may be very different from those of downstream applications.Optimal Transport. Optimal Transport (OT) is a framework for matching two probability distributions [48; 53] . We predominantly consider optimal transport between empirical discrete measures. Suppose that we are given points x 1 , x 2 , . . . , x n ∈ X and y 1 , . . . , y K ∈ Y, a source measure µ defined by µ = n i=1 w i δ xi , and a target measure given by ν = K j=1 p j δ j , where w i , p j are positive values such that n i=1 w i = 1, K j=1 p j = 1. Suppose also that δ x is a Dirac delta function Algorithm 1 OTTER 1: Input: Input X = {x 1 , . . . , x n }, label distribution specification (p 1 , . . . , p K ), cost matrix C ∈ R n×K 2: Define input marginal µ = 1 1 n , prediction marginal ν = (p 1 , . . . , p K ) 3: Run optimal transport and obtain transport plan π s.t. π = arg min γ∈Π(µ,ν) ⟨γ, C⟩. 4: Get modified classification outputs ŷi = arg max j∈[K] π i,j .Return {ŷ i } i∈ [n] at x, i.e. δ x (x ′ ) = 1 if x ′ = x, and δ x (x ′ ) = 0 otherwise. Given a cost matrix C ∈ R n×K , the Monge-Kantorovich formulation of optimal transport is to find a minimal cost transport plan π such that π = arg min γ∈Π(µ,ν)⟨γ, C⟩,where Π(µ, ν) = {γ ∈ R n×K+ |γ1 = µ, γ T 1 = ν}.Distribution and Label Shifts. Distribution shift refers to the discrepancy between a source distribution P s on which the model is trained, and a target distribution P t on which the model is deployed. Distribution shift often degrades trained model performance on target tasks. Label shift is a specific type of distribution shift such that P s (Y ) ̸ = P t (Y ) and the data generation process is fixed -in other words, the conditional distributions of the inputs are the same: P s (X|Y ) = P t (X|Y ).Techniques such as importance sampling [35; 6; 24] , recalibration [3] and domain adaptation [55] are commonly used to mitigate the effects of label shift. Unfortunately, these methods assume access to source distribution data, whereas zero-shot models' pretraining data is inaccessible (often proprietary or blocked for privacy reasons). Thus, adapting zero-shot models to new label distributions poses challenges unmet by these pre-existing methods.Let X = {x 1 , x 2 , . . . , x n } be an inference dataset with x i ∈ X . Furthermore, let Y = {y 1 , y 2 , . . . , y n } be the true labels of the K-class classification dataset, such that y i ∈ Y = [K], are sampled according to the downstream label distribution ν = (p 1 , p 2 , . . . , p K ).Let s θ (x, j) := P θ (Y = j|X = x) be a pretrained classification model constrained to the downstream label space. During pretraining, s θ has been biased to the source label distribution ν s . We wish to offset such label distribution bias with a label distribution specification ν for the target distribution. ν is expected to be closer to the true label distribution of the downstream task. Given a label distribution specification, our goal is to rebalance predictions so that the predicted label distribution follows the label distribution specification. 
proposed framework: 
 We propose OTTER (Optimal TransporT adaptER), an optimal transport-based label distribution adaptation approach. Our goal is to have the n input data points allocated to K classes match a given label distribution ν, where K j=1 νj = 1, νj ≥ 0. Specifically, we want to classify nν 1 points as the first class, nν 2 points as the second, and so on. However, there are many such allocations, and it is not a priori clear which one should be selected. We propose formulating an optimal transport problem that selects the allocation minimizing a particular cost:π = arg min γ∈Π(µ,ν) ⟨γ, C⟩, where Π(µ, ν) = {γ ∈ R n×K + |γ1 = µ, γ T 1 = ν}, µ = 1n 1 and C is the cost (loss) matrix such that C ij represents a loss when we classify x i as class j. This procedure is described in Algorithm 1. Note that this procedure naturally matches the given label distribution specification ν.We wish to use Algorithm 1 for zero-shot classification given the pretrained model s θ . To do so, we must select a cost function and produce C ij . An ideal choice of such a function is C ij = -log P t (Y = j|X = i) such that optimal transport minimizes the negative log posterior under constraints. However, the target distribution P t is unknown. Instead, we replace the posterior with the classifier scores s θ (x i , j). We highlight that this choice of cost matrix is an natural extension of zero-shot classification under the label distribution constraint. We prove this claim in the next section. First, we show a toy example that shows how OTTER improves zero-shot accuracy.Example. To illustrate the benefits of OTTER, consider the following example for binary classification. We have two data points, X = {x 1 , x 2 } with Y = {1, 2}, and true label distribution ν = ( 1 2 , 1 2 ). Suppose that the zero-shot model's prediction scores are s 1 = (0.4, 0.6) and s 2 = (0.1, 0.9).Traditional classification yields ŷ1 = 2, ŷ2 = 2, producing a 50% error rate. However, given the cost matrix C derived from the prediction score matrix S = 0.4 0.6 0.1 0.9 , C = -log 0.4 -log 0.6 -log 0.1 -log 0.9 , along with µ = (0.5, 0.5) and ν = (0.5, 0.5), the optimal transport procedure discovers the transport map π = 0.5 0.0 0.0 0.5 , yielding ŷ1 = 1, ŷ2 = 2. This corrects the original zero-shot prediction error.Extension to Online Predictions While OTTER offers efficient label distribution adaptation, it requires a batched set of inference data points, making online predictions challenging. To address this, we introduce R-OTTER (Reweighting OTTER), which learns a reweighting factor-an estimate of P t (Y )/P s (Y )-using OTTER's predictions y OTTER on a validation set. Once learned, these reweighting parameters can be applied to online predictions by adjusting logit probability scores. We use a reweighting formulation equivalent to logit adjustment in [71] . The reweighted probability scores of P θ , with a reweighting vector r ∈ R K , are defined as:P θ,r (Y = j|X = x) = r j P θ (Y = j|X = x) K j ′ =1 r j ′ P θ (Y = j ′ |X = x).The parameter r is learned using cross-entropy loss on y OTTER . We expect R-OTTER to perform comparably to OTTER, with the additional advantage of not requiring OTTER to be run over the entire dataset. In the following section, we provide a theoretical result showing that r * = P t (Y )/P s (Y ) is an optimal reweighting parameter when learned from y OTTER as pseudolabels, effectively addressing label shift.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Justification: Yes, we included the full set of assumptions and a complete (and correct) proof in Section 4.2.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
theoretical results: 
 In practical scenarios, label distribution specifications are frequently subject to noise, and prediction probabilities may not be well-calibrated. To understand the impact of these factors, we examine how errors in label distribution specification and calibration influence the transport plan. Our theoretical analysis yields following findings: (a) OTTER can recover the Bayes-optimal classifier in the label shift setting, (b) for a noisy cost matrix with the noisy label distribution specification setup, the suboptimality can be bounded by the deviation of cost matrix and label distribution, and (c) R-OTTER effectively addresses label shift when learned from y OTTER as pseudolabels.Classification as optimal transport. Prior to discussing the main theoretical results, we demonstrate that standard classification-expressed as ŷi = arg max j∈[K] P θ (Y = j|X = x i )-can be interpreted as a (trivial) solution derived from optimal transport.Theorem 4.1. Let ν ZS j = 1 n n i=1 1[ŷ ZS i = j],whereŷZS i = arg max j ′ ∈[K] P θ (Y = j ′ |X = x i ). Then, given C ij = -log P θ (Y = j|X = x i ), π = arg min γ∈Π(µ,ν ZS ) ⟨γ, C⟩ , ŷOT i = arg max j∈[K] π ij .Assuming there are no ties in scores, i.e.P θ (Y = j|X = x i ) ̸ = P θ (Y = j ′ |X = x i ), for all j ̸ = j ′ , the OTTER predictions are equivalent zero-shot predictions, i.e.ŷOT i = ŷZS i for all i ∈ [n].This theorem has the following implications. First, it suggests that the predictions will remain unchanged if we set ν = ν ZS . Second, Bayes-optimal classifiers can be derived through optimal transport, using a (true) cost matrix defined as C * ij = -log P t (Y = j|X = x i ), coupled with the true label distribution ν * .Our analysis begins with the label shift setup, which is a commonly-studied type of distribution shift-as well as a prominent issue when applying zero-shot models. We demonstrate that when the label distribution is correctly specified, optimal transport preserves Bayes-optimal classifier predictions under label shift. Next, we consider general perturbations in label distribution and cost matrix as well as their impact on the resulting solutions.In this setting, we assume features follow the same conditional distribution across source and target distributions, i.e. P s (X|Y ) = P t (X|Y ). Furthermore, we suppose that the prediction scores are accurately calibrated in the training dataset, such that s θ (x, j) = P s (Y = j|X = x). For zero-shot models, we often lack access to P s . This is a typical scenario in zero-shot model applications: after training on large-scale corpora, we use the pretrained model without the source dataset.For a given downstream task with the target label distribution ν * = P t (Y ), one standard approach to achieve the Bayes-optimal classifier for the target distribution is to reweight the score function outputs using the ratio Pt(Y =j)Ps(Y =j) . This adjustment leads to:sθ (x, j) = s θ (x, j) • P t (Y = j) P s (Y = j) ∝ P t (X = x|Y = j) • P t (Y = j) ∝ P t (Y = j|X = x).This reweighted score function aligns with the target distribution, thus correcting label shift.Although reweighting the score function is a popular solution, it faces an important obstacle when applied to zero-shot models like CLIP, where the source distribution P s (Y ) is typically unknown. We show that OTTER successfully induces a Bayes classifier for the target distribution, represented as f t (x) = arg max j∈[K] P t (Y = j|X = x), without requiring access to P s (Y ). This capability is particularly significant for zero-shot models, enabling them to adapt to target distributions effectively, even in the absence of explicit knowledge of the source distribution. Now, we show that optimal transport can be an effective tool to correct label shift.Theorem 4.2. Suppose the pretrained model is well-calibrated for the source distribution,P θ (Y = j|X = x i ) = P s (Y = j|X = x i )and there is no tie probability, for all j ̸ = j ′ , i ∈ [n]P θ (Y = j|X = x i ) ̸ = P θ (Y = j ′ |X = x i ).Denote the Bayes optimal predictions in the target distribution as ŷ * i = arg max j∈[K] log P t (Y = j|X = x i ). Then OTTER predictions ŷ = OTTER(X, ν * , C) are the same as Bayes optimal predictions ŷ * .That is, OTTER recovers a Bayes classifier in the target distribution without access to the source distribution, given the target distribution and a well-calibrated model for the source dataset.In practical applications, calibration error could extend beyond noise in the elements of the cost matrix. A key source of error is label distribution estimation error. Hence, we address a more general setting, examining the impact of simultaneous perturbations in the label distribution and cost matrix of the transport plan. Our result applies techniques from perturbation theory for linear programming .We rewrite our optimal transport problem min π∈Π(µ,ν) ⟨π, C⟩ as a linear programming problem. Let π and C be the transport plan and cost matrix respectively. Matrix G and vector g are used to denote the row and column constraints on π to form a feasible plan which transports distribution from µ to ν.H := 1 T n ⊗ I K I n ⊗ 1 T K , G = H -H , g =    µ ν -µ -ν    .Then, we have the equivalent linear programming problem,EQUATIONWe adapt a theorem from Robinson [52] with our optimal transport problem notation.Theorem 4.3. Let the primal linear programming problem be defined as in equation ( 1), and its dual problem be max{w T g|w T G ≤ vec(C) T , w ≥ 0}. Suppose perturbed cost matrix is Ĉ = C + ∆ C , the perturbed class distribution ν = ν + ∆ ν , such that ĝ = g + ∆ g where∆ g =    0 ν -ν 0 -ν + ν    .Assume that primal and dual problems are solvable. Denote the original solutions as π, w and perturbed solutions as π and ŵ. Then,∥π -π∥ 2 F ≤ κ 2 ∥∆ ν ∥ 2 2 + ∥[vec(∆ C )] + ∥ 2 2 + ∥vec(C) T vec(π) -g T ŵ∥ 2 2 -∥w -ŵ∥ 2 2, where 1 ≤ p ≤ ∞ and κ is a Hoffman constant that only relates to the original problem [30] .Ignoring the constant and the subtraction part, the upper bound can be decomposed into three components,• ∆ ν : noise (or the estimation error) of the target balance,• [vec(∆ C )] + : noise (or the calibration error) of the cost matrix,• vec(C) T vec(π) -g T ŵ: the suboptimality of perturbed solution ŵ.Theorem 4.3 implies that the deviation from perturbed solution to true solution is bounded by the magnitude of perturbations and suboptimality of the perturbed solution. From this result, we can expect prediction accuracy to deteriorate with perturbations in the label distribution and calibration.We provide a theoretical result showing that R-OTTER can learn an optimal parameter by learning reweighting parameters from ŷOTTER as pseudolabels, and produce identical predictions to OTTER once the optimal parameter is obtained in the label shift setup.Theorem 4.4. Under the same assumptions as in Theorem 4.2, the parameter r * = P t (Y )/P s (Y ) is optimal when learning with y OTTER as pseudolabels.The proof is provided in Appendix D.6.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provided detailed exhaustive experimental hyperparameters and setup.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiments: 
 The primary objective of our experiments is to (1) validate that OTTER improves zero-shot model performance when given accurate label distribution estimates and (2) investigate its sensitivity to perturbations. In experiments on real datasets (Section 5.1), we confirm that OTTER can improve zero-shot classification significantly in a variety of settings. In synthetic experiments (Section 5.2), we validate our theoretical claims-label shift invariance and sensitivity to perturbation in a fully controllable setting. Additionally, we show that OTTER can be combined with label distribution estimation methods in the few-shot learning setting (Section 5.3). Next, we show the empirical results for H-OTTER that leverages label hierarchy (Section 5.4) and R-OTTER that mitigates the batched prediction requirement (Section 5.5). Finally, we we apply OTTER to mitigate LLM selection bias (Section 5.6). Our code is available at https://github.com/SprocketLab/OTTER.We hypothesize that the model performance can improve significantly when the label distribution specification is exact.Setup and Procedure. We used 17 image classification datasets and 4 text classification datasets. We employed CLIP [49] for image zero-shot classification, and BERT [19] . A comprehensive list and details of experiments can be found in Appendix E.Baseline. We adopt Prior Matching (PM) [37] different datasets. We selected hyperparameters through grid search, by evaluating their performance on a validation set, consisting of 10 labeled examples per class. In contrast, we highlight that OTTER is tuning-free.Results. Table 1 shows the image classification results with CLIP (ViT-B/16) and the text classification results with BERT. Notably, OTTER demonstrates a 4.8% and 15.5% enhancement on average in image and text zero-shot classification, respectively. While Prior Matching shows competitive performance when accurately tuned, it often struggles. We found that hyperparameter tuning fails in the class-imbalanced datasets such as Caltech256, SUN397, ImageNet-r (Appendix E, Table 7 ). This suggests that the hyperparameter selection process necessitates a validation set label distribution similar to the target distribution-rendering it unusable in zero-shot scenarios. More details and additional experiment results -including the sensitivity study on the label distribution specification error, computation time, and combination with other prompting methods -are provided in Appendix E.3.We hypothesize OTTER is invariant to label shift under the conditions in Theorem 4.2. We also investigate the sensitivity to perturbations of the cost matrix and the label distribution.Setup and Procedure. We simulate label shift in logistic regression. Suppose X|Y = 0 ∼ N (-1, 1) andX|Y = 1 ∼ N (1, 1). Training data is sampled from a mixture of Gaussians X s ∼ ν s 0 N (-1, 1) + ν s 1 N (1, 1) such that P s (Y = 0) = ν s 0 , P s (Y = 1) = ν s 1 , ν s 0 + ν s 1 = 1. Similarly, we sample the test data from X t ∼ ν t 0 N (-1, 1) + ν t 1 N (1, 1). We fix the training set label distribution as ν s 0 = 0.1, ν s 1 = 0.9 and vary test set label distribution ν t to simulate label shift. We train a logistic regression model with 10,000 samples from the source distribution, and test the model with 10,000 samples from the target distribution. A Bayes-optimal classifier in the target distribution is given by f Bayes(x) = 1[x ≥ 1 2 (log ν t 0 ν t 1 + 1)].The naive classifier is defined as the maximizer of the predicted score. The OTTER predictions are produced with Algorithm 1, with the cost matrix C ij = -log P θ (Y = j|X = x i ) and the label distribution specification ν t , where P θ (Y |X) represents the logistic regression model scores.We separately investigate perturbed prediction score matrix and perturbed label distribution specification's impact on the prediction accuracy. For perturbed prediction scores, we fix the label distribution to be the true one, and add noise δ ∼ N (0, σ 2 ) of varying levels σ to the predicted score P θ (Y = 1|X). For the perturbed label distribution specification, we fix the prediction score to be true scores and add noise ϵ: ν = ν t + (ϵ, -ϵ). We use these perturbed variants to obtain perturbed solutions and compare with ground-truth solution.Results. Figure 2 illustrates how accuracy changes with label shift when the predicted score is perturbed and when label distribution specification is perturbed. We observe that the naive classifier Curves represent different methods and noise levels. Our approaches dramatically outperform the baseline at higher mismatch levels.deteriorates as the total variation distance between source and target distributions increases. It indicates that naive classifier is sensitive to label shift. However, without perturbation, OTTER remains unaffected by the label distribution shift, which validates our invariance result in Section 4.In the case of confidence prediction perturbation, both the naive classifier and OTTER have accuracy decreasing as perturbation level increases. For simplicity, we omitted the naive classifier's performances under different levels of noise as adding zero-mean noise does not alter its accuracy significantly. We observe that OTTER has better performance than the naive method when significant label shift exists. Similarly, for label distribution perturbation, we observe as the noise level ϵ increases, OTTER's accuracy downgrades-but still yields better performance when label shift is severe.Our experimental results suggest simply using prediction scores for zero-shot classification leads to inaccurate predictions under label shift, while OTTER is robust to label shift when no perturbations are present. Perturbations in both predicted score and label distribution specification downgrades the predicted accuracy, as expected, but OTTER still yields better results than the naive baseline. We anticipate that OTTER can be used in few-shot learning when combined with label distribution estimation methods. We expect OTTER can improve zero-shot classification if the label distribution estimation error is sufficiently small. Also, we expect OTTER can improve linear probing, which is one of standard approaches for few-shot learning.Setup and Procedure. We use the same datasets as the previous experiment. We consider a 10-shot learning setting: 10 labeled samples per class are given. Note that labeled samples have uniform label distribution, while the label distribution in the target distribution may not be uniform. This setting requires the use of label distribution estimation methods used in label shift adaptation [35; 6; 24] . We estimate the target label distribution with Black Box Shift Estimation (BBSE) [35] . BBSE estimates the target balance using confusion matrix, under the label shift assumption. For detailed explanation, refer to Appendix C.. Table 2 shows the image and text zero-shot classification results with the label distribution estimation via BBSE and linear probing. The image classification results show that OTTER can yield mild improvement over linear probing, even with the label distribution estimation errors. Figure 4 shows that accuracy improvement is consistent across the number of samples used for linear probing.In text classification, we found OTTER improves zero-shot text classifications where the number of classes is small (K = 2 or 3). While it shows a relatively high variance due to the small sample size (20 ∼ 30), the average accuracy improves significantly over zero-shot classification. More detailed analysis regarding label distribution estimation error and the number of samples is provided in Appendix E.4. We hypothesize incorporating class hierarchy information can enhance few-shot label distribution estimation and thus improve zero-shot predictions.Setup and Procedure. We use a subset of CIFAR100 data with WordNet hierarchy. Specifically, we take 'fish' and 'tree' as superclasses and have 5 subclasses in each of them. We suppose we can access 10 labeled samples per each subclass. We first apply OTTER with the superlevel label distribution estimation and make pseudo-labels of superlevel class in the test set. Using them, we estimate the sublevel label distribution and use OTTER.Results. Table 3 presents the results. As anticipated, we note an enhancement in accuracy when compared to the naive implementation of OTTER. Specifically, we observe a significant improvement in accuracy for RN50, RN101, and ViT-B/ which we attribute primarily to the reduction in label distribution estimation error. Further details are provided in Appendix E.5.We show that R-OTTER provides a performance comparable to that of OTTER empirically.Setup and Procedure. We use the identical setup for Section 5.1. R-OTTER learns reweighting parameters in validation set using y OTTER . Note that the validation set is not required to be labeled since y OTTER is used as pseudolabels in the validation set.Results. Although R-OTTER is suboptimal compared to OTTER due to generalization issues, it still provides label distribution correction, improving accuracy over zero-shot predictions. We also provide synthetic experiments for R-OTTER in Appendix E.6.Selection bias is the tendency of LLMs to favor certain prefix tokens in multiple-choice questions [69; 17; 12; 65] . We demonstrate that OTTER can effectively mitigate selection bias by randomly shuffling the options and enforcing a uniform class balance in OTTER.Setup and Procedure. Our experimental setup follows Zheng et al. [69] , utilizing the MMLU [28] , ARC-Challenge [15] , and CommonsenseQA (CSQA) [57] datasets. We test with llama-2(chat)-7/13B [60] and vicuna-v1.3-7/13B [11] models, treating the probabilities of each option token (A/B/C/D) as prediction probabilities. OTTER is applied under the assumption of a uniform distribution. We use 0-shot predictions and evaluate performance using accuracy and the standard deviation of recalls (RStd) as metrics.Results. Table 5 presents the experimental results. OTTER significantly reduces selection bias, enhancing accuracy by up to 10.6% and lowering RStd by as much as 25.7%.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]Justification: We submit a zip folder of our code, and will release github link upon publication.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 While zero-shot models have been successful, pretraining using Internet-scale datasets yields artifacts that may harm downstream tasks. In this paper, we identify the bias in class balance, and provide a simple but powerful solution using optimal transport. Theoretically, we describe how OT can fix label distribution mismatch and its sensitivity to perturbations. Empirically, we validated our approach's ability to improve zero-shot classification accuracy, mitigating label distribution mismatch in zero-shot models. We believe our method can expedite the deployment of zero-shot classification, reducing the necessity of finetuning for downstream tasks.distribution. However, the standard approach requires access to the labeled source distribution data, which is usually not possible in zero-shot classification scenarios. OTTER provides a solution decoupled from the source data distribution, overcoming this limitation.Improving Zero-shot Classification using Prior. Several studies have explored leveraging prior information to enhance zero-shot classification, even in the absence of access to source distributions. In the context of prompt-based zero-shot models, prior matching [37] employs word prior distribution to alleviate word bias inherent in pretraining data. We adopted their adaptation method as a baseline. Similarly, Kahana et al. [31] develop adaptation layers trained using priors, aiming to maintain proximity to the original scores. However, both approaches entail training additional layers and necessitate hyperparameter tuning, which may pose challenges in the context of zero-shot predictions. In contrast, OTTER presents a straightforward and efficient adaptation method to new label distributions without the need for any hyperparameter tuning, backed by theoretical guarantees.Leveraging Optimal Transport for Enhanced Pseudo-labeling and Classification. A number of studies have explored the enhancement of pseudo-labeling and classification tasks through optimal transport, using label distribution specifications, in a similar spirit to our work, but within different contexts. Tai et al. [56] uses optimal transport to allocate pseudo-labels to unlabeled datasets based on the label distribution specification in the semi-supervised setup. Wang et al. [62] deals with long-tail distribution in the partial-label learning setup based on optimal transport. Zhang et al. [68] uses partial optimal transport as a pseudo-labeling based approach for deep imbalanced clustering, progressively expanding the labeled sample proportion. Guo et al. [26] reweights the training dataset to match the label distribution specification using optimal transport. This work mainly deals with the class imbalance problem in the training step. Shi et al. [54] studies classification from a matching perspective, revealing the connection between softmax loss and inverse optimal transport and suggesting a new loss function to address long-tail distributions. Their analysis provides useful insights for OTTER -why cost matrix induced by pre-trained models can be useful in the inference step. Xian et al. [66] uses optimal transport as a postprocessing method to guarantee demographic parity. While sharing aspects of the approach, our work addresses class bias in zero-shot models. Peng et al. [47] used optimal transport to handle long-tail recognition with a learned cost matrix. Our study provides a theoretical basis for understanding their empirical results. Chang et al. [9] employs optimal transport to detect common and private classes between the source and the target domain, under the universal domain adaptation setting, where knowledge is transferred from source to target domain without any constraints on the label sets. In the context of zero-shot classification, there is no need to manage label space disparities between the source and target domains. Instead, the main concern of zero-shot classification is dealing with the distribution shift between the pretraining dataset and the downstream task. We tackle the label distribution shifts using optimal transport.Class Imbalance. Class imbalance problems occur when the number of instances across different classes in a dataset is disproportionately distributed. This imbalance can severely bias the traininig process of a machine learning model, leading to poor generalization performance, especially for the minority classes. It has been extensively studied in the context of traditional machine learning [23] . Oversampling [10] and cost-sensitive learning [58] are well-known approaches to address class imbalance. Nonetheless, the inherent nature of class imbalance in pretraining datasets introduces a distinct set of challenges, especially when attempting to rectify such biases within the context of zero-shot classification scenarios.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Yes, we provide all hyperparameters in the Appendix. We provide all data split and details in the main experimental result section Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Yes, we provide error bars as much as possible, unless repeating an experiment is expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not release a new pre-trained model.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite all dataset and model original owners.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Prior matching [37] proposed prior matching as a reweighting method for prompt-based classifiers to mitigate word bias -the distribution shift between pretrained language models' word prior and the class priors in the downstream task. We use it as a zero-shot model adaptation method given a class balance estimation.Define reweighted probability scores of P θ with r asP θ,r (Y = j|X = x i ) = r j P θ (Y = j|X = x i ) K j ′ =1 r j ′ P θ (Y = j ′ |X = x i ). Ideally, we hope to estimate the weight vector r * ∈ R n such that reweighted scores P θ,r * (Y = j|X = x i ) maximizes the accuracy in the target distribution. Since the labels are not given, this is impossible. Instead, prior matching matches the label distribution of P θ,r (Y = j|X = x i ) -ν j .It can be solved using the standard optimization techniques -we used [38] . While this is equivalently effective with OTTER when properly optimized, we found that the temperature parameter and learning rate crucially affect the final accuracy, making it less ideal for the zeroshot adaptation. We used the grid search with the small validation set (10 samples per class) in each task to select hyperparameters. The hyperparameter ranges are as follows.• Temperature: [1e-3, 1e-4, 1e-5, 1e-6, 1e-7]• Learning rate: [1e-3, 1e-4, 1e-5, 1e-6, 1e-7]Black Box Shift Estimation (BBSE) Label shift adaptation methods [35; 6; 24] aims to estimate the class balance in the target distribution using the labeled source distribution data and the unlabeled target distribution data. We use Black Box Shift Estimation (BBSE) to estimate the class balance in the downstream task. Algorithm 2 describes the procedure. Note that the derivation of this algorithm depends on the label shift assumptions, thus the label distribution estimation with real-world data can be heavily biased. = arg max j∈[K] P θ (Y = j|X = x i ), thus -log P θ (Y = ŷZS i |X = x i ) ≤ -log P θ (Y = j|X = x i ) for all j ∈ [K], which results in ŷOT i = ŷZS i .To prove Theorem 4.2, we show a specific form of invariance property of optimal transport first.Theorem D.1. Suppose π * = arg min γ∈Π(µ,ν) ⟨γ, C⟩ and E is a columnwise perturbation, i.e.,E = [ϵ 1 1 ϵ 2 1 • • • ϵ K 1] ,where 1 denotes n dimensional vectors and ϵ 1 , . . . , ϵ K are constants. Then the perturbed cost matrix C = C + E, then π * is also an optimal transport map with respect to the cost matrix C.Proof. By the optimality condition, we havei,j π * ij C ij ≤ i,j π ij C ij for any π ∈ Π(µ, ν). Then, i,j π * ij C ij + K j=1 ν j ϵ j ≤ i,j π ij C ij + K j=1 ν j ϵ j , which is i,j π * ij C ij + K j=1 n i=1 π * ij ϵ j ≤ i,j π ij C ij + K j=1 n i=1 π ij ϵ j . Thus, i,j π * ij Cij ≤ i,j π ij Cij .This theorem is also valid for row-wise perturbations as well with a similar proof. Consequently, a straightforward implication is that Corollary D.2. Suppose π * = arg min γ∈Π(µ,ν) ⟨γ, C⟩, E is a columnwise perturbation and F is a row-wise perturbation, such thatE = [ϵ 1 1 ϵ 2 1 • • • ϵ K 1] , F =    η 1 1 T η 2 1 T • • • η K 1 T    ,where 1 denotes n dimensional vectors with 1s, and ϵ 1 , . . . , ϵ K , η 1 , . . . , η K are constants. Suppose the perturbed cost matrix is defined as C = C + E + F , then π * is also an optimal transport map with respect to the perturbed cost matrix C. Now we provide the proof of Thoerem 4.2.C ij = -log P θ (Y = j|X = x i ) = -log P s (Y = j|X = x i), the posteriors in the target distribution can be defined asC * ij = -log P t (Y = j|X = x i ). From P t (Y = j|X = x i ) = P s (Y = j|X = x i ) P s (X = x i )P t (Y = j) P t (X = x i )P s (Y = j) ,we can see thatC * ij = -log P t (Y = j|X = x i ) = -log P s (Y = j|X = x i ) P s (X = x i )P t (Y = j) P t (X = x i )P s (Y = j) = -log P s (Y = j|X = x i ) + log P s (Y = j) -log P t (Y = j) -log P s (X = x i ) + log P t (X = x i ) = C ij + E •j + F i• where E •j = log P s (Y = j) -log P t (Y = j), F i• = -log P s (X = x i ) + log P t (X = x i ). And, assuming ν j = 1 n n i=1 1[ŷ * i = j],where ŷ * is the Bayes classifier prediction such that ŷ * i = arg maxj∈[K] P t (Y = j|X = x i ) = arg min j∈[K] -log P t (Y = j|X = x i ).Then, optimal transport solution Let z, w be the primal, dual solution. And, let the perturbed primal linear programming problem bemin z {p T z| Ĝz ⩾ ĝ, z ⩾ 0}and its dual bemax w {w T ĝ|w T Ĝ ⩽ pT , w ⩾ 0}.Let ẑ, ŵ be the corresponding primal, dual solution.Suppose that the primal and dual problems are solvable. Then,z w - ẑ ŵ 2 ≤ κ [(G -Ĝ)ẑ -(g -ĝ)] - [(G -Ĝ) T ŵ -(p -p)] + (p -p)ẑ -(g -ĝ) ŵ p ,where 1 ≤ p ≤ ∞ and κ is the Hoffmann constant determined by p, G, g. [30] .This Lemma provides a bound for error in the solution of the perturbed linear program. Since discrete optimal transport can be translated to standard linear program, we obtain Theorem 4.3 by plugging in the definitions.Proof of Theorem 4.3 A discrete optimal transport problem min   i,j C i,j π i,j |π1 = µ, π T 1 = ν, π ij ≥ 0    can be written as a linear program min{p T z|Gz ⩾ g, z ⩾ 0}, where p = vec(C), z = vec(π), H = 1 T n ⊗ I K I n ⊗ 1 T K , G = H -H , g =    µ ν -µ -ν   .Note that the equality constraints are converted to stacked inequalities. We have noisy cost matrix and label distribution in our setting, which leads to the perturbation on cost matrix C and ν such that the perturbed cost matrix is Ĉ = C + ∆ C , the perturbed label distribution ν = ν + ∆ ν , such that ĝ = g + ∆ g where∆ g =    0 ν -ν 0 -ν + ν   .Since we don't have perturbation on the constraint matrix G, Ĝ = G. By plugging in these terms to Lemma D.3.z w - ẑ ŵ 2 ≤ κ [(G -Ĝ)ẑ -(g -ĝ)] - [(G -Ĝ) T ŵ -(p -p)] + (p -p)ẑ -(g -ĝ) ŵ 2 = κ [g -ĝ] - [p -p] + (p -p)ẑ -(g -ĝ) ŵ 2 = κ 0 [ν -ν] - 0 [ν -ν] - [p -p] + pẑ -g ŵ 2 ∵ Optimality of ẑ, ŵ in the perturbed problem. = κ [ν -ν] [p -p] + pẑ -g ŵ 2Then, we havez w - ẑ ŵ 2 2 ≤ κ 2 [ν -ν] [p -p] + pẑ -g ŵ 2 2.Then,∥z -ẑ∥ 2 2 ≤ κ 2 [ν -ν] [p -p] + pẑ -g ŵ 2 2 -∥ w -ŵ∥ 2 2Plugging in z = vec(π), ẑ = vec(π), ∆ ν = ν -ν, ∆ C = Ĉ -C, and rearranging, we obtain∥π -π∥ 2 F ≤ κ 2 ∥∆ ν ∥ 2 2 + ∥[vec(∆ C )] + ∥ 2 2 + ∥vec(C) T vec(π) -g T ŵ∥ 2 2 -∥w -ŵ∥ 2 2 .We use the definition of the Hoffman constant κ from [52] . Computing Hoffman constant or even bounding it has been a long-standing problem [5; 45; 46] . However, it has been shown that the Hoffman constant is a finite real number [52] , and specifically under our problem setup, it is independent from perturbations and only related to original optimization problem. This suggests the possibility to regularize the parameters C, G, g in the original problem such that κ does not depend on the dimensionality of cost matrix or target distribution. We leave further exploration of κ in this context as future work.In few-shot learning, we assume that a few labeled samples per class are given. They can be used for estimating label distribution in the target distribution using label shift estimation methods [35; 6; 24] . They comes with the sample complexity analysis under the label shift assumptions, which can be used to obtain bound the label distribution estimation errors.Lemma D.4. Let m and n denote the number of few-shot learning data and test datapoints, w i = ν i /ν s i and ŵi = νi /ν s i . Also let σ min be the smallest eigenvalue of the covariance matrix V ŷ,y where [V ŷ,y ] i,j = P s (f (x) = i, y = j). For m > 80 log(m)σ -2 min and constant c > 0, the perturbation ∆ ν may be bounded as||∆ ν || 2 ≤ ||ν s || 2 c σ 2 min ||w|| 2 log m m + K log n n ,with probability at least 1 -3Km -10 -2Kn -10 .The proof of Lemma D.4 relies on the following result of [34] .Lemma D.5. Assume that 1. ∀x ∈ X and ∀y ∈ Y, P s (x|y) = P t (x|y),2. if P t (y) > 0 then P s (y) > 0 ∀y ∈ Y, and 3. the expected confusion matrix C s (f ) = P s (f (x), y) ∈ R K×K for classifier f is invertible.Then, there exists a constant c > 0 such that for all m > 80 log(m)σ -2 min , with probability at least 1 -3Km -10 -2Kn -10 ,|| ŵ -w|| 2 ≤ c σ 2 min ||w|| 2 log m m + K log n n .Proof of Lemma D. 4 . Where all norms are Euclidean unless otherwise denoted, we have that||∆ ν || 2 = ||ν -ν|| 2 = ||ν s ⊗ ( ŵ -w)|| 2for element-wise multiplication operation ⊗. Further,||ν s ⊗ ( ŵ -w)|| 2 ≤ ||ν s ( ŵ -w)|| 2 F ≤ ||ν s || 2 2 || ŵ -w|| 2 ≤ ||ν s || 2 c σ 2 min ||w|| 2 log m m + K log n n ,where the last line follows from Lemma D.5 with probability at least 1 -3Km -10 -2Kn -10 .Further, we can bound [vec(∆ C )] + by the Total Variation Distance (TVD) between P s and P θ as follows.Lemma D.6. Let τ = 1 2 ||P s -P θ || 1 denote the Total Variation Distance between P s and P θ and define min(C) = min i,j C ij . Then,||[vec(∆ C )] + || ≤ m(K -1) log τ min(C) + 1 .Proof of Lemma D.6. For each element ∆(ij) C of ∆ C , we have that ∆ (ij) C = log P θ (Y = j|X = x i ) -log P s (Y = j|X = x i ) = log P θ (Y = j|X = x i ) P s (Y = j|X = x i ) .For i, j such thatP θ (Y = j|X = x i ) ≤ P s (Y = j|X = x i ), clearly ∆ (ij) C ≤ 0 and so [vec(∆ (ij) C )] + = 0. Otherwise, for i, j such that P θ (Y = j|X = x i ) > P s (Y = j|X = x i ), it follows that ∆ (ij) C > 0 and ∆ (ij) C = log P θ (Y = j|X = x i ) -P s (Y = j|X = x i ) + P s (Y = j|X = x i ) P s (Y = j|X = x i ) ≤ log τ P s (Y = j|X = x i ) + 1 ≤ log τ min(C) + 1 .For each i ∈ [m], there are at most K -1 possible j such thatP θ (Y = j|X = x i ) > P s (Y = j|X = x i ), because j∈[K] P θ (Y = j|X = x i ) = j∈K P s (Y = j|X = x i ). Therefore, there are at most m(K -1) pairs (i, j) ∈ [m] × [K] such that 0 < ∆ (ij) C ≤ log τ min(C) + 1 . Thus, ||[vec(∆ C )] + || ≤ m(K -1) log τ min(C) + 1 .We provide the proof of Theorem 4.4.Proof. By assumption, we have P θ (Y |X) = P s (Y |X). By the result of Theorem 4.2, y OTTER samples are generated by arg max j∈[K] P t (Y = j|X = x). Suppose r * = P t (Y )/P s (Y ). Then,P θ,r * (Y = j|X = x) = r * j P θ (Y = j|X = x) K j ′ =1 r * j ′ P θ (Y = j ′ |X = x) = r * j P s (Y = j|X = x) K j ′ =1 r * j ′ P s (Y = j ′ |X = x) = Pt(Y =j) Ps(Y =j) Ps(X=x|Y =j)Ps(Y =j) Ps(X=x) K j ′ =1 Pt(Y =j ′ ) Ps(Y =j ′ ) Ps(X=x|Y =j ′ )Ps(Y =j ′ ) Ps(X=x) = P t (Y = j)P s (X = x|Y = j) K j ′ =1 P t (Y = j ′ )P s (X = x|Y = j ′ ) = P t (Y = j)P t (X = x|Y = j) K j ′ =1 P t (Y = j ′ )P t (X = x|Y = j ′ ) = Pt(Y =j)Pt(Y =j|X=x)Pt(X=x) Pt(Y =j) K j ′ =1 Pt(Y =j ′ )Pt(Y =j ′ |X=x)Pt(X=x) Pt(Y =j ′ ) = P t (Y = j|X = x) K j ′ =1 P t (Y = j ′ |X = x) Thus, we have y R-OTTER = arg max j P θ,r * (Y = j|X = x) = arg max j P t (Y = j|X = x) K j ′ =1 P t (Y = j ′ |X = x) = arg max j P t (Y = j|X = x) = y OTTER ,which implies r * is an optimal parameter. Zeroshot image classification datasets We use CIFAR10, CIFAR100 [33] , Caltech101 [22] , Caltech256 [25] , Food101 [8] , STL10 [16] , SUN397 [67] , Flower102 [42] , EuroSAT [27] , Oxford-IIIT Pet [44] , Stanford Cars [32] , DTD [14] , CUB [61] , ImageNet [18] , ImageNet-r [29] , and ImageNet-Sketch [63] .We use Amazon [41] , Gender [20] , CivilComments [7] , and HateXplain [39] . (x i , j) = P θ (Y = j|X = x i ) = exp (cos(f (x i ), g(y j ))/τ ) K j ′ =1 exp (cos(f (x i ), g(y j ′ ))/τ )for image x i regarding the label j, given the image encoder f , the text encoder g. Cost matrix is constructed byC = [C ij ] i∈[n],j∈[K], where c ij = -log s θ (x i , j).We run Algorithm 1 with the true class balance of the test dataset.Text zero-shot classification We employ BERT and text-embedding-ada-002 sentence embeddings for text classification [50] . This process parallels the methodology used in image zero-shot classification -we compute the prediction scores from the cosine similarity and then construct the cost matrix with the negative log probabilities. Ablation on the class balance specification We conducted a semi-synthetic experiment to investigate the sensitivity to the label distribution specification error in real datasets. We generate the noisy label distribution specification and see how the accuracy changes. We control the noise in the label distribution specification as follows. Given the true class balance ν * , first we make adversarial class balance ν adv such that ν adv j * = 1 for j * = arg min j∈[K] ν * j and ν adv j = 0 for j ̸ = j * . To measure distance between class balance specification and true class balance, we use the total variance T V (ν, ν) = 1 2 ||ν -ν|| 1 . Next, we intepolate ν * and ν adv such that T V (ν * , ν α ) = α, byν α = (1 - α T V (ν * ,ν adv ) )ν * + α T V (ν * ,ν adv ) ν adv .We set the interval of alpha as 0.01 and vary it up to 0.2.Figure 5 shows the result. We observe the sensitivity to the label distribution specification error varies depending on the datasets, but generally we can observe that the accuracy degrades linearly proportionally to the class balance error. While the result may vary depending on the interaction between class balance error and calibration error in cost matrix, we can expect performance improvement if the class balance specification is good enough. Inference time comparison To show that the additional computation complexity induced by OTTER is not heavy, we measured the time consumption (in seconds) for the inference step in the experiments in Section 5.1, with the pre-computed embeddings. 12 : Accuracy in the prompt-enhanced zero-shot classification by Classification by Description (CD) [40] . We can observe OTTER's capability to provide further enhancements upon refined the improvements achieved through refined prompts.taken using a machine equipped with an Intel® Core™ i7-11700K @ 3.60GHz processor, 64GB RAM, and NVIDIA GPU RTX-4090. For most cases (n < 30000), our method takes less than 1 second, while the prior matching baseline takes more than 3 seconds. It's worth noting that the time consumption for computing embeddings is more substantial; even 10 seconds is negligible compared to the embedding time consumption (ranging from 5 to 30 minutes for each evaluation set), which is common for all inference conditions. Ablation on prompts Recent studies have demonstrated the efficacy of enhancing prompts as a means to improve zero-shot models [70; 40] . In order to further illustrate the potential enhancements offered by OTTER beyond enhanced prompts, we reproduced Menon and Vondrick [40] 's approach (Classification by Description, CD), which employs multiple prompts generated by language models and takes max scores of them for each class. We applied OTTER to CD. The results of this experiment are summarized in Table 12 . As anticipated, OTTER exhibits enhancements in zero-shot classification, even when prompt sets are refined using language models. Dataset TV(ν * , νzs ) TV(ν 13 : Class balance estimation error with BBSE in Section 5.3. We report the mean of 10 different random samplings of the validation set. Lower is better.We report the label distribution estimation errors in Section 5.3. As a metric, we use the total variance T V (ν, ν) = 1 2 ||ν -ν|| 1 . We use zeroshot prediction scores and linear probing prediction scores for BBSE. νzs denotes the estimated class balance based on zero-shot prediction scores, and νlp represents the estimated class balance based on linear probing prediction scores.Table 13 shows the result. We can see that total variation decreases with linear probing in image classification tasks since they reduces the violation of label shift assumptions. However, total variation increases in text classification tasks due to the small number of labeled sample size, following the size of label space (K = 2 or 3). Accordingly, we can expect OTTER will be more useful with linear probing, and just rebalancing zero-shot predictions with OTTER could be enough for text classification tasks.Ablation experiments on linear probing We provide full results of Section 5.3. Specifically, we additionally report the results of combination with linear probing in text classification tasks and the results of zero-shot classification results in image classification tasks.The results are presented in Table 2 . While OTTER often provides additional improvement over LP, zero-shot classification was a strong baseline in image classification tasks. Meanwhile, class balance adaptation in text classification tasks is effective in all cases, giving a significant improvement over zero-shot predictions. Ablation experiments on the number of examples per class Few-shot adaptation scenario assumes we have access to labeled data to estimate the target distribution. We hypothesize that an increase in the number of labeled samples enhances the accuracy of the class balance estimation, thereby improving the performance of OTTER. To test this hypothesis, we use few-shot adaptation in image and text classification tasks, without linear probing. The experiment varies the number of samples per class from 10 to 100, anticipating a reduction in class balance estimation error and an improvement in OTTER's accuracy with the increase in labeled samples.The results, as depicted in Figure 6 , corroborate our hypothesis. It is evident that the error in class balance estimation diminishes with an increasing number of samples, leading to a subsequent enhancement in the accuracy of OTTER.Comparison between OTTER and Linear Probing with varying number of classes In the few-shot adaptation scenario, we explored three approaches: OTTER, linear probing (LP), and a combination of LP + OTTER. We formulated two hypotheses. The first posits that OTTER might outperform LP, particularly in situations with a limited number of samples. The second hypothesis suggests that OTTER could provide further enhancements to LP even when LP already surpasses the naive version of OTTER. This experiment was conducted using the same setup as the previous one.The results, displayed in Figure 7 , reveal several insights regarding our hypotheses. To begin with, OTTER demonstrates performance on par with LP, especially in scenarios with fewer samples. Interestingly, OTTER achieves superior accuracy compared to LP in datasets like Amazon and CivilComments, characterized by a small number of classes (K = 2), resulting in a relatively low total sample count. Furthermore, it is observed that incorporating OTTER into LP leads to an average increase in accuracy.E.5 Detailed experiment setup and results of Section 5.4 Class hierarchy We used the following superclasses and subclasses classes for the proof of concept.Justification: Yes. We discuss the potentials and limitations across several sections.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.38th Conference on Neural Information Processing Systems (NeurIPS 2024).E.3 Detailed experiment results of Section 5.1 Ablation on zero-shot models For the ablation study on zero-shot models, we provide experimental results with varying architectures, given the exact prior. Table8, 9 show the image zero-shot classification results, and Table10shows the text zero-shot classification results. We also provide another baseline results with CLIPPR[31], which uses the label distribution for adapter training. CLIPPR is similar with Prior Matching in the point that it requires adapter training, but it has more adapter layers and additional loss function to make the predictions stick to the original prediction scores. While the performance gain varies, we can observe that OTTER is effective for the most cases. 


Decision: The paper provides a simple method to handle changes in label proportions between a pretrained model and a downstream task, via optimal transport. The method is simple and straightforward, and the authors perform a comprehensive empirical evaluation backed by theory. The paper is generally well-written and easy to read. Thereby, I recommend acceptance as a poster.

Reviews: review: paper_summary: paper_summary: Zero-shot models such as CLIP suffer from imbalanced predictions inherited from the uncurated pre-training data. This paper proposes adjusting the predictions of zero-shot models via optimal transport. This method requires only the downstream data and the label prior of the target distribution. Under the assumption that the conditional likelihood is unchanged between the pre-training data and the downstream data, the authors prove that the induced predictions are Bayes Optimal. The authors validate the effectiveness of the proposed method on several benchmark tests.
strength_weakness: strength_weakness: Major:

(1) Related works are missing. Some papers have addresses the issue of removing the label bias in zero-shot models. For instance, [1][2] use a subset of the pre-training data to mitigate the bias. [3] focuses on only using the labeled downstream data to address imbalanced predictions and proposes two methods. The authors should consider including these prior works.

[1] A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models.

[2] The neglected tails of vision-language models.

[3] Generalized Logit Adjustment: Improved Fine-tuning by Mitigating Label Bias in Foundation Models.

(2) The proposed method is transductive and requires the entire test dataset to perform optimal transport. I am wondering if OTTER can estimate the label distribution of pre-training data $\pi$ from unlabeled downstream training data and then the debiasing procedure can be performed by subtracting the logarithm $\ln \pi$ from the zero-shot predictions, as described in [3] (https://arxiv.org/pdf/2310.08106), without needing to operate on the entire test dataset

Minor:

(1) Duplicate citations: [29] and [30].

(2) Line 120: should reference Algorithm 1.
questions: questions: Please refer to the weaknesses, and I am willing to raise the score if the major weaknesses are addressed.

Additionally, I have a question to discuss with the authors. Both [3] and OTTER use downstream data to generate Bayes optimal predictions based on the assumption that $P(x|y)$ remains unchanged between the pre-training distribution and the downstream distribution. In practice, this assumption might not hold. For instance, the pre-training data might mostly consist of real photos, while some test benchmarks might be sketches, e.g., ImageNet-Sketch. Under this situation, is your method theoretically sound in mitigating the pre-training bias?
limitations: limitations: Limitations have been discussed and no negative societal impact has been identified.
paper_summary: paper_summary: Zero shot classification suffers from label distribution mismatch. This paper suggest to adjust pretrained model prediction via optimal transport. By showing the optimal transport prediction is equivalent to the bayes optimal classifier's output theoretically and good model performances on various experimental settings empirically, the paper shows the validity of its proposed method.
strength_weakness: strength_weakness: - Novelty is limited. This paper suggests applying OT for managing label distribution shift. The class imbalance for pretrained network prediction is already well known, and OT has been already applied for various distribution shift cases. Therefore, naively applying OT for the biased pretrained model prediction is imaginable.
- The proposal saying that Bayes optimal classifier can be derived through optimal transport is inadequate and too strong since it requires a true cost matrix. If $P_t(Y|X)$ is accessible, we don't even need to use the optimal transport.
- Please link theorems of the main paper and proofs in the appendices for readability.
- I don't think figure 2 shows OTTER remains unaffected. The acc is deteriorated from 90 to 7X, meaning about 15%p decrease.
questions: questions: - Considering each cell of the cost matrix $C$, it is a loss (or the classifier score) calculated from the pretrained network. It means the cost matrix is biased to the source data distribution. In that sense, how the optimal transport can be considered as assigning true label?
- How can the true label distribution of the target, $\nu*$, be accessible?
- Can the author show more details for the equation of section 4.1? I cannot understand how to remove $P_s(X)$ and $P_t(X)$
- In table 1, what happened to Caltech256 Prior matching? Why the performance is far worse than zero-shot?
- In figure 2, Why OTTER shows acc increase when the total variation distance between the source and the target increases from 0.4 to 0.8? With larger distribution shift, acc is supposed to deteriorate. If not, the synthetic experiment may be poorly modeled.
- How long does it take to apply OTTER? I suppose it will take quadratically proportional to the dataset size.
limitations: limitations: .
paper_summary: paper_summary: This paper introduces a novel approach to address the issue of label distribution mismatch in zero-shot models, which is a common problem due to the imbalance in the pretraining datasets. The proposed method uses optimal transport to adjust the predictions of pretrained models based on an estimated downstream label distribution, without requiring additional training or access to labeled downstream data.
strength_weakness: strength_weakness: 1. dependence on estimated label distribution: the method relies on an accurate estimate of the downstream label distribution. In practice, obtaining a reliable estimate may be challenging, and errors in this estimate could impact the performance of OTTER.
2. evaluation on diverse datasets: while the paper includes a variety of datasets for validation, a more detailed analysis of performance across different types of datasets (e.g., varying in size, complexity, and imbalance levels) would provide a deeper understanding of the method's strengths and limitations.
3. comparison with recent methods: the paper compares OTTER with some existing baselines, but it could benefit from a more comprehensive comparison with the latest state-of-the-art methods in label distribution adaptation and zero-shot learning. There are many training-free or test-time adaption zero-shot baselines[a][b][c]. How does OTTER perform compared to these methods?
4. scalability: although described as lightweight, the paper does not thoroughly discuss the scalability of OTTER for very large-scale datasets.


[a] Mirza, Muhammad Jehanzeb, et al. "Lafter: Label-free tuning of zero-shot classifier using language and unlabeled image collections." NIPS 2024.

[b] Zhao, Shuai, et al. "Test-time adaptation with clip reward for zero-shot generalization in vision-language models." arXiv preprint arXiv:2305.18010.

[c] Abdul Samadh, Jameel, et al. "Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization." NIPS 2024.
questions: questions: The paper presents a compelling and well-validated approach to address label distribution mismatch in zero-shot models using optimal transport. Its innovative method, strong theoretical foundation, and significant empirical improvements make it a valuable contribution to the field. However, further exploration of its limitations, more comparisons with recent methods, and practical implementation details would enhance its impact and utility.
limitations: limitations: The authors do not claim limitations.

Comments: title: Respond to author rebuttal
Reference
comment: I checked the authors' rebuttal, and appreciate their effort to relieve my concerns.

Since my questions for the novelty claim has not been solved, and the authors have thoroughly answered my questions, I increase my score to 5.
Dear reviewer, 

Thank you for your assistance in the review process. Could you please read the author rebuttal ASAP and let them / fellow reviewers know if it addressed your concerns?
The theory and experiments sound good to me. I am happy to increase my score to 8.
Thank you! We appreciate it. 

Following your suggestion, we have updated our paper to introduce a variant of our approach called R-OTTER (Reweighting OTTER). We added the R-OTTER formulation to Section 3, a set of theoretical results (which we sketch here) to Section 4, and empirical results (highlighted below, including both synthetic and real data settings) to Section 5. This approach helps reduce the reliance on large data subsets or the entire dataset for prediction.

R-OTTER involves learning a reweighting factor (an estimate of $P_t(Y)/P_s(Y)$) using OTTER's predictions $y_{otter}$ as pseudolabels in validation set. This reweighting can then be applied to adjust probability scores or logits without relying on the entire test dataset.

* R-OTTER formulation: For simplicity, we use a reweighting formulation equivalent to logit adjustment in [3]. The reweighted probability scores of $P_\theta$ with a reweighting parameter vector $r \in R^{K}$ defined as $P_{\theta, r}(Y=j|X=x) = \cfrac{r_j P_\theta(Y=j|X=x)}{\sum_{j'=1}^K r_{j'} P_\theta(Y=j'|X=x)}$. We learn $r$ with $y_{otter}$ using a cross entropy loss.

We expect R-OTTER to perform comparably to OTTER (but with the desired benefit of not requiring running OTTER over the entire dataset) by learning the reweighting factor accurately. We provide a theoretical result showing that $r^*=P_t(Y)/P_s(Y)$ is optimal in this formulation.

* **Theorem**. Under the same assumption of Theorem 4.2., $r^*=P_t(Y)/P_s(Y)$ is an optimal parameter when learning with $y_{otter}$.

**Proof (Sketch)**. By assumption $P_\theta(Y|X)=P_s(Y|X)$. Using the result of Theorem 4.2, $y_{otter}$ samples are generated by 
$\arg\max_{j}P_t(Y=j|X=x)$. Suppose $r^*=P_t(Y)/P_s(Y)$. Then,

$P_{\theta, r^*}(Y=j|X=x)$

$= \cfrac{r_j^* P_\theta(Y=j|X=x)}{\sum_{j'=1}^K r^*_{j'} P_\theta(Y=j'|X=x)}$

$=\cfrac{r_j^* P_s(Y=j|X=x)}{\sum_{j'=1}^K r_{j'}^* P_s(Y=j'|X=x)}$

$=\cfrac{\frac{P_t(Y=j)}{P_s(Y=j)}\frac{P_s(X=x|Y=j)P_s(Y=j)}{P_s(X=x)}}{\sum_{j'=1}^K\frac{P_t(Y=j')}{P_s(Y=j')}\frac{P_s(X=x|Y=j')P_s(Y=j')}{P_s(X=x)}}$

$=\cfrac{P_t(Y=j)P_s(X=x|Y=j)}{\sum_{j'=1}^KP_t(Y=j')P_s(X=x|Y=j')}$

$=\cfrac{P_t(Y=j)P_t(X=x|Y=j)}{\sum_{j'=1}^KP_t(Y=j')P_t(X=x|Y=j')}$

$=\cfrac{\frac{P_t(Y=j)P_t(Y=j|X=x)P_t(X=x)}{P_t(Y=j)}}{\sum_{j'=1}^K\frac{P_t(Y=j')P_t(Y=j'|X=x)P_t(X=x)}{P_t(Y=j')}}$

$=\cfrac{P_t(Y=j|X=x)}{\sum_{j'=1}^K P_t(Y=j'|X=x)}$

Thus, $y_{r\text{-}otter} = \arg\max_{j}P_{\theta, r^*}(Y=j|X=x)$

$=\arg\max_{j}\cfrac{P_t(Y=j|X=x)}{\sum_{j'=1}^K P_t(Y=j'|X=x)}$

$=\arg\max_{j}P_t(Y=j|X=x)$

$=y_{otter}$.

Our synthetic experiment validates the above theorem.
* **Synthetic experiment results**: We use our synthetic experiment setup with perturbation noise $\delta=0$ and label distribution $\alpha=0$. Additionally, we generated a validation set that follows the same distribution as the test set. After learning the reweighting parameter $r$ in the validation set using $y_{otter}$ as pseudolabels, we evaluated R-OTTER on the test set, comparing the results to those of zero-shot and OTTER. We expect that, if successful, R-OTTER will similarly gain improvement over zero-shot when the source and target distributions increasingly differ. We observe this:

| TV (P_t(Y), P_s(Y)) | ZS     | OTTER  | R-OTTER | Bayes Optimal |
|-----------------|--------|--------|---------|---------------|
| 0               | 0.9306 | 0.9238 | **0.9179**  | 0.9305        |
| 0.05            | 0.9041 | 0.9001 | **0.8944**  | 0.9065        |
| 0.1             | 0.8783 | 0.8830 | **0.8768**  | 0.8870        |
| 0.15            | 0.8516 | 0.8687 | **0.8628**  | 0.8732        |
| 0.2             | 0.8261 | 0.8598 | **0.8492**  | 0.8618        |
| 0.25            | 0.8003 | 0.8507 | **0.8407**  | 0.8516        |
| 0.3             | 0.7731 | 0.8443 | **0.8331**  | 0.8449        |
| 0.35            | 0.7483 | 0.8428 | **0.8334**  | 0.8430        |
| 0.4             | 0.7218 | 0.8392 | **0.8317**  | 0.8393        |
| 0.45            | 0.6978 | 0.8432 | **0.8355**  | 0.8434        |
| 0.5             | 0.6709 | 0.8433 | **0.8371**  | 0.8444        |

Overall, we observe comparable accuracy to OTTER, where the difference is controlled by the parameter estimation error.

We anticipate similar findings in our real-world experiments:

* **Real experiment results**: We conducted a mini real-world experiment by splitting the original dataset into validation and test sets, following the same procedure as in the synthetic experiment.

|            | ZS     | OTTER  | R-OTTER |
|------------|--------|--------|---------|
| CIFAR100   | 0.6381 | 0.6759 | **0.6532**  |
| Caltech101 | 0.7964 | 0.8541 | **0.8427**  |
| DTD        | 0.3893 | 0.4381 | **0.4156**  |
| EUROSAT    | 0.3286 | 0.4185 | **0.3453**  |
| Flowers102 | 0.6382 | 0.6894 | **0.6836**  |

As expected, R-OTTER, indeed performs better than zero-shot across all datasets and often comparably to OTTER. We will include the full results in the revised manuscript.
Thank you for your updates. In my opinion, combining OTTER with [3] to avoid transductive zero-shot predictions would make the proposed method more practical. As a result, I am strongly considering increasing my score for the paper if the revised version incorporates this suggested combination. If you agree with the suggestion, could you provide concrete details on what changes you plan to make and where they will be implemented?
Thank you for the interesting idea! Yes, combining two methods in the way you suggest could eliminate the need for the entire dataset or large data subsets in optimal transport. We appreciate the suggestion and will add the combination to our draft.
Thank you for your follow-up question. The reason OTTER shows greater accuracy gains with increased class imbalance is that there's more room for improvement in these scenarios. When the downstream class distribution is imbalanced, it often deviates more significantly from the concept distribution in the pretraining data, leading to a larger label distribution shift. This greater discrepancy allows OTTER to make more impactful corrections, resulting in the observed accuracy increase.
Thanks for the authors' answer. Regarding Q2, can you explain why the accuracy increases with the increase of imbalance? This is counterintuitive.
Thank you for your response. Regarding your reply to W2, if I understand correctly, the authors suggest that the proposed OTTER method at least requires a batch of test samples to produce an unbiased prediction. My question is, if we are given a set of i.i.d. validation data with zero-shot predictions $y_{zs}$ and debiased predictions $y_{otter}$, is it possible to use these terms to derive $(\log\frac{P_t(y)}{P_s(y)}$such that the requirement of using all test data or a batch of test samples can be avoided?
[1] Saerens, Marco, Patrice Latinne, and Christine Decaestecker. "Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure." Neural computation 14.1 (2002): 21-41.

[2] Lipton, Zachary, Yu-Xiang Wang, and Alexander Smola. "Detecting and correcting for label shift with black box predictors." ICML'18.

[3] Azizzadenesheli, Kamyar, et al. "Regularized Learning for Domain Adaptation under Label Shifts." ICLR'18.

[4] Alexandari, Amr, Anshul Kundaje, and Avanti Shrikumar. "Maximum likelihood with bias-corrected calibration is hard-to-beat at label shift adaptation." ICML'20.

[5] Garg, Saurabh, et al. "A unified view of label shift estimation." NeurIPS'20.

[6] Lee, Yin Tat, and Aaron Sidford. "Path finding methods for linear programming: Solving linear programs in $\tilde{O}(\sqrt{rank})$ iterations and faster algorithms for maximum flow." IEEE 55th Annual Symposium on Foundations of Computer Science. IEEE, 2014.