Title: Characterizing Out-of-Distribution Error via Optimal Transport
Venue: NeurIPS 2023
Accepted: yes
Keywords: Distribution Shift

OOD Error Prediction

Optimal Transport

Deep Learning
Abstract: Out-of-distribution (OOD) data poses serious challenges in deployed machine learning models,
so methods of predicting a model's performance on OOD data without labels are important for machine learning safety.
While a number of methods have been proposed by prior work, they often underestimate the actual error, sometimes by a large margin, which greatly impacts their applicability to real tasks. In this work, we identify *pseudo-label shift*, or the difference between the predicted and true OOD label distributions, as a key indicator of this underestimation. Based on this observation, we introduce a novel method for estimating model performance by leveraging optimal transport theory, Confidence Optimal Transport (COT), and show that it provably provides more robust error estimates in the presence of pseudo-label shift. Additionally, we introduce an empirically-motivated variant of COT, Confidence Optimal Transport with Thresholding (COTT), which applies thresholding to the individual transport costs and further improves the accuracy of COT's error estimates. We evaluate COT and COTT on a variety of standard benchmarks that induce various types of distribution shift -- synthetic, novel subpopulation, and natural -- and show that our approaches significantly outperform existing state-of-the-art methods with up to 3x lower prediction errors.

Characterizing Out-of-Distribution Error via Optimal Transport

Abstract

Out-of-distribution (OOD) data poses serious challenges in deployed machine learning models, so methods of predicting a model's performance on OOD data without labels are important for machine learning safety. While a number of methods have been proposed by prior work, they often underestimate the actual error, sometimes by a large margin, which greatly impacts their applicability to real tasks. In this work, we identify pseudo-label shift , or the difference between the predicted and true OOD label distributions, as a key indicator of this under-estimation. Based on this observation, we introduce a novel method for estimating model performance by leveraging optimal transport theory, Confidence Optimal Transport (COT), and show that it provably provides more robust error estimates in the presence of pseudo label shift. Additionally, we introduce an empirically-motivated variant of COT, Confidence Optimal Transport with Thresholding (COTT), which applies thresholding to the individual transport costs and further improves the accuracy of COT's error estimates. We evaluate COT and COTT on a variety of standard benchmarks that induce various types of distribution shift - synthetic, novel subpopulation, and natural - and show that our approaches significantly outperform existing state-of-the-art methods with up to 3x lower prediction errors. Our code can be found at https://github.com/luyuzhe111/COT.

1 Introduction

Machine Learning methods are largely based on the assumption that test samples are drawn from the same distribution as training samples, providing a basis for generalization. However, this i.i.d. assumption is often violated in real-world applications where test samples are found to be out-ofdistribution (OOD) - sampled from a different distribution than during training. This may result in a significant negative impact on model performance [15, 34, 25, 38]. A common practice for alleviating this issue is to regularly gauge the model's performance on a set of labeled data from the current target data distribution, and update the model if necessary. When labeled data is not available, however, one needs to predict the model's performance on the target distribution with unlabeled data, a task known as OOD performance prediction .

Performance prediction on unlabeled data has previously been shown to be impossible without imposing additional constraints over the unknown target distribution [7, 11, 5, 26], due to the fact that target samples may take any label. Thus, the feasibility of this task is dependent on what assumptions we make regarding the shift between the train and target distributions. Prior works often make the assumption that the conditional density P ( y | x ) remains fixed in the presence of covariate shift [37].

However, this tells us little when x falls outside the support of the train distribution. Despite the theoretical difficulty, prior works have proposed a number of heuristic methods to estimate the performance of a model based on unlabeled target samples. For instance, Average Confidence (AC) [20, 16] estimates error based on the average maximum softmax score for target samples, assuming the model has been calibrated for the train distribution. This method was further improved with the addition of a learned threshold [13], for which the error is predicted as the fraction of samples with confidence falling below it. Other approaches estimate model performance based on a disagreement score computed between the predictions of two models trained over the same dataset [22, 3]. Some works have found that applying a transformation over target samples and estimating the effect of the transformation leads to a reliable prediction of model performance in vision tasks [9, 10]. However, many of these prior methods have been shown to underestimate the model's error when it is miscalibrated in the target distribution [13, 22]; that is, the predicted softmax scores differ from the true class likelihoods. We empirically observe that this miscalibration is strongly positively correlated with pseudo-label shift , which is the difference between the predicted target label distribution P T (˜ y ) and the true label distribution P T ( y ) . Thus, we treat pseudo-label shift as a key indicator of error underestimation.

In this work, we propose an approach which provides robust error estimates in the presence of pseudolabel shift. Our approach, Confidence Optimal Transport (COT), leverages the optimal transport framework to predict the error of a model as the Wasserstein distance between the predicted target class probabilities and the true source label distribution. We theoretically derive lower bounds for COT's predicted error. This results in a more provably conservative error prediction than AC, which is crucial for safety in many real-world machine learning applications. In addition, we introduce a variant of COT, Confidence Optimal Transport with Thresholding (COTT), which introduces a learned threshold over the optimal transportation costs in line with prior work [13] and empirically improves upon COT's performance. In this work, we mainly target covariate shifts and compare our proposed methods to existing state-of-the-art approaches in extensive empirical experiments over eleven datasets exhibiting distribution shift from both vision and language domains. These distribution shifts include: visual corruptions (e.g., blurred image data), novel subpopulation shifts (e.g., novel appearances of a category), and natural shifts in the wild (e.g., different stain colors for pathology images), all of which are frequently experienced in the real world. We find that COT and COTT consistently avoid the significant error underestimations suffered by previous methods. In particular, COTT achieves significantly lower prediction errors than existing methods for most models and datasets (up to 3x better), establishing new state-of-the-art results.

2 Preliminaries and Motivation

In this section, we introduce the problem setup of OOD error prediction and show how a popular baseline, Average Confidence (AC), can be understood as the Wasserstein Distance (WD) between a reference label distribution and the softmax output distribution from an Optimal Transport (OT) perspective. We then demonstrate why the reference label distribution AC uses is problematic by explaining pseudo-label shift and its correlation with miscalibration. After establishing the relation and utility of OT to OOD error prediction, we formally introduce our proposed methods in Sec. 3.

2.1 OODPerformance Prediction

In this work, we address the OOD problem in the domain of classification tasks. Let X ⊆ R d be the input space of size d , and Y = { 1 , · · · , k } be the label space, where k is the number of classes. Let the source distribution over X × Y be P S ( x, y ) and the target distribution be P T ( x, y ) . A classifier f ⃗ : X → ∆ k -1 maps an input to a confidence vector (i.e. the output of the softmax layer), where ∆ k -1 = { ( z 1 , · · · , z k ) : z 1 + · · · + z k = 1 , z i ≥ 0 } is the k -dimensional unit simplex. Let the training samples be { ( x ( i ) train , y ( i ) train ) } ∼ P S ( x, y ) , and the validation samples be { ( x ( i ) val , y ( i ) val ) } ∼ P S ( x, y ) . The validation set is used in some previous methods and will also be used in COT and COTT to perform calibration [17].

The OOD performance prediction problem is formally stated as follows: Given an unlabeled test set { x ( i ) T } ∼ P T ( x ) , and a classifier f ⃗ trained over the training samples, predict the accuracy of

f ⃗ over the test set α = 1 n ∑ n i =1 ✶ [ y ( i ) T = arg max j f ⃗ j ( x ( i ) T )] , where y ( i ) T is the ground truth label. Equivalently, we can also predict the error ϵ = 1 -α with an estimate ˆ ϵ .

In this work, we are further investigating the distribution of confidence vectors . While a confidence vector f ⃗ ( x ) itself is a distribution of labels in the k -dimensional simplex ∆ k -1 , the distribution of confidence vectors f ⃗ # P ( ⃗ c ) is a distribution of distributions, where ⃗ c ∈ ∆ k -1 . f ⃗ # P ( ⃗ c ) is defined to be the pushforward of a covariate distribution P ( x ) using f ⃗ : f ⃗ # P ( ⃗ c ) = P ( f ⃗ -1 ( ⃗ c )) . Consider the following example: Suppose we have a uniform covariate distribution P ( x ) on x ∈ { A,B,C } and f ⃗ maps A,B to [0 . 5 , 0 . 5] ⊤ and C to [0 . 7 , 0 . 3] ⊤ . Then, f # P ( ⃗ c ) will assign 2 3 mass to [0 . 5 , 0 . 5] ⊤ and 1 3 mass to [0 . 7 , 0 . 3] ⊤ . To facilitate easy comparison between confidence vectors and labels, we denote ⃗ y ∈ { 0 , 1 } k ∩ ∆ k -1 to be the one-hot representation of y ∈ Y , where the only non-zero element in ⃗ y is the y -th element, i.e. ⃗ y j = ✶ [ y = j ] . We denote the distribution of one-hot labels as P ( ⃗ y ) . For a covariate distribution P ( x ) , we will refer to the distribution of predicted labels from classifier f ⃗ as the pseudo-label distribution P pseudo ( ⃗ y ) . We reuse the notation to denote the probability mass of ⃗ y also as which is given by the mass of the inputs that give this prediction, i.e. P pseudo ( ⃗ y ) = P ( { x ∈ X| arg max j f ⃗ j ( x ) = y } ) .

2.2 Wasserstein Distance and Optimal Transport

In recent years, optimal transport theory has found numerous applications in the field of machine learning [2, 4, 1, 27, 29]. Optimal transport aims to move one distribution of mass to another as efficiently as possible under a given cost function. In the Kantorovich formulation of optimal transport, we are given two distributions µ ( x ) over X and ν ( y ) over Y and a cost function c ( x, y ) that tells us the cost of transporting from location x to location y . Here, we aim to find a transport plan π ( x, y ) that minimizes the total transport cost. The transport plan is a joint distribution with marginal π ( · , Y ) = µ ( · ) and π ( X , · ) = ν ( · ) . The conditional distribution π ( x,y ) µ ( x ) of the transport plan informs us how much mass is moved from x to y . Let Π( µ, ν ) be the set of all transport plans. More formally, the Wasserstein Distance is defined as:

The Wasserstein distance satisfies the definition of a metric (non-negativity, symmetry, and subadditivity), inducing a metric space over a space of probability distributions. Unlike other metrics, such as total variation, the Wasserstein metric induces a weaker topology and provides a robust framework for comparing probability distributions that respect the underlying space geometry [2].

For discrete distributions µ, ν such as the empirical distributions, the Wasserstein distance simplifies to the following linear programming problem:

where C , P ∈ R m × n are the cost matrix and the plan matrix respectively and C ij is the transport cost from sample i to sample j . When n = m , the optimal transport problem reduces to the optimal matching problem (Proposition 2.1 in Peyré et al. [32]), i.e. the optimal transport plan P ∗ = arg min P ∈ Π( µ,ν ) ⟨ P , C ⟩ is a permutation matrix. Not only does this constraint enable efficient algorithms like the Hungarian algorithm, but it will also help draw the connection between pseudolabel shift and target error that is central to our Confidence Optimal Transport method.

2.3 Average Confidence as Wasserstein Distance

We use Average Confidence, a popular OOD performance prediction method, as the starting point of our analysis. Average Confidence with Max Confidence (AC-MC) estimates the target accuracy by taking the empirical mean of maximum confidence of the classifier over all target samples x ( i ) ∼ P T ( x ) , i.e. 1 n ∑ n i =1 max j f ⃗ j ( x ( i ) ) . Its corresponding target error estimate is therefore ˆ ϵ AC = 1 -1 n ∑ n i =1 max j f ⃗ j ( x ( i ) ) . By definition, | ϵ -ˆ ϵ AC | measures the miscalibration of the model, i.e. how far away the model's confidence is from its actual accuracy. In the following proposition, we connect the AC-MC estimates with distances in the Wasserstein Metric Space:

P

Ppseudo(y)

Ppseudo(y)

f#Pn(c

f#Pir(c)

f#Pr(c)

D

pseudo(y)

Pr(y)

PT(y)

iPr(y)

O

Proposition 1 ( ˆ ϵ ACW ∞ Equivalence) . Let ( P ( R k ) , W ∞ ) be the metric space of all distributions over R k , where W ∞ is the Wasserstein distance with c ( x, y ) = ∥ x -y ∥ ∞ . Then, the estimated error of AC-MC is given by ˆ ϵ AC = W ∞ ( f ⃗ # P ( ⃗ c ) , P pseudo ( ⃗ y )) .

We defer all proofs to the supplementary material. Here, we appeal to pictorial intuition in Figure 1, where δ 0 (Dirac delta over the zero vector) represents the origin. All one-hot label distributions, including P T ( ⃗ y ) and P pseudo ( ⃗ y ) , are represented as points on the (dashed) unit sphere around δ 0 . A distribution of confidence vectors f ⃗ # P ( ⃗ c ) is simply a point within the unit ball around δ 0 . The AC-MC accuracy estimate measures how far the point is to the origin δ 0 . Additionally, we can project the point to the unit sphere, resulting in the projection point P pseudo ( ⃗ y ) , which is the pseudo-label distribution. The AC-MC error estimate measures the length of the projection .

Corollary 1 ( P pseudo ( ⃗ y ) is closest to f ⃗ # P ( ⃗ c ) ) . Let P ′ ( ⃗ y ) ∈ P ( { 0 , 1 } k ∩ ∆ k -1 ) be a one-hot label distribution. Then W ∞ ( f ⃗ # P ( ⃗ c ) , P ′ ( ⃗ y )) ≥ W ∞ ( f ⃗ # P ( ⃗ c ) , P pseudo ( ⃗ y )) .

This suggests that AC-MC, among all possible reference one-hot label distributions, selects the closest one and reports the distance to be the predicted error. Thus, we can see that AC-MC is a very optimistic prediction strategy, so it is not surprising that AC-MC is widely reported to be over-estimating the performance on real tasks [13, 16, 22, 3].

2.4 Pseudo-Label Shift and its Correlation with Miscalibration

The pseudo-label shift is defined as W ∞ ( P pseudo ( ⃗ y ) , P T ( ⃗ y )) , i.e. the distance from the pseudo label distribution P pseudo ( ⃗ y ) to the ground truth target label distribution P T ( ⃗ y ) , which is also the length of the green line segment in Figure 1. An important property is W ∞ ( P pseudo ( ⃗ y ) , P T ( ⃗ y )) ≤ ϵ ≤ 1 , i.e. the pseudo-label shift is a lower bound of the true target error of the model (as true target error can be interpreted as the average transport cost of a suboptimal matching). We can clearly see how a large pseudo-label shift could potentially destroy the prediction of AC-MC from Figure 1 (right), where the red line segment is much shorter than the green one which should be a lower bound of the true error. This lower bound, however, can be loose when it is small, as shown in Figure 1 (middle).

Most existing prediction methods heavily rely on the model being well-calibrated, as pointed out by Jiang et al. [22], Garg et al. [13], yet the underlying difficulty is a lack of precise understanding of when and by how much neural networks become miscalibrated. This is evident in the large-scale empirical study conducted in [30], showing that the variance of the error becomes much larger among the different types of shift studied, despite some positive correlation between the expected calibration error and the strength of the distribution shift. In this section, we empirically show that there is a strong positive correlation between the pseudo-label shift and | ϵ -ˆ ϵ AC | , which we define as the model's miscalibration. Because of this correlation, we consider pseudo-label shift as the key signal to why existing methods have undesirable performance.

We evaluate the pseudo label shift and the prediction error of AC-MC | ϵ -ˆ ϵ AC | on CIFAR-10 and CIFAR-100 under different distribution shifts, and plot the results in Figure 2 (left). The plots demonstrate a strong positive correlation between these two quantities, which not only means that the performance of AC-MC worsens as the pseudo-label shift gets larger, but also implies that the performance of existing methods [16, 13, 22] that depend on model calibration will drop.

CIFAR-10

CIFAR-100

corruption

O shot_noise

CIFAR-10

CIFAR-100

Error of Average Confidence

0.1

0.3

R

中

△

00

4

。

品

Ogaussian_blur

Oelastic_transform

60j0

Obrightness

O defocus_blur

o9

o

frost

gaussian_noise

glass_blur

contrast

Ozoom_blur

severity

2

4

O spatter

3

Osnow

01

o

speckle_noise

Error

0.2

0.1

0.4

口

4

4

口

Ojpeg_compression

impulse_noise

5

Omotion_blur

0.00.10.20.30.40.50.6

0.00.2

0.4

0.6

0.8

Opixelate

0.00.10.20.30.40.50.6

0.0

0.2

0.4

0.6

0.8

saturate

PseudoLabelShift

PseudoLabel Shift

CIFAR-10

CIFAR-100

corruption

Oshot_noise

CIFAR-10

CIFAR-100

0.4

Obrightness

contrast

Osnow

Ospatter

0.4

Odefocus_blur

speckle_noise

0.3

ofog

elastic_transform

Ozoom_blur

severity

0.2

Ofrost

01

0.2

Ogaussian_blur

2

0.1

glass_blur

gaussian_noise

3

+4

0.1

0.0-

4

Ojpeg_compression

impulse_noise

5

0.0 0.4. 86

O1

motion_blur

0.00.10.20.30.40.50.6

0.0

0.2

0.4

0.6

0.8

Opixelate

0.00.10.20.30.40.50.6

0.0

0.2

0.4

0.60.8

PseudoLabelShift

PseudoLabelShift

saturate

PseudoLabel Shift

PseudoLabelShift

Our exploration of miscalibration and pseudo-label shift reveals a previously unexplored tradeoff: When the pseudo-label shift is small, prior work has given us some reassurance to trust the calibration of neural networks, which motivated methods such as AC [20] and ATC [13]. More critically, as miscalibration worsens, the pseudo-label shift becomes a tighter low bound of the error and thus a more trustworthy estimate of the error.

Motivated by this observation, we leverage the information of the pseudo-label shift to improve the performance of confidence-based prediction methods for miscalibrated models. The problem, as mentioned earlier, is that without any information about the target label distribution, it is theoretically impossible to estimate the pseudo-label shift when the test set contains data from unseen domains. Thus, the only option is to make assumptions on the target label distribution.

In our proposed method, we make a natural assumption: the target label distribution is close to the source label distribution. This assumption aligns with most natural shifts that can be observed in real-world datasets. This extra assumption allows us to develop COT and COTT, which perform much better than existing methods in most cases (See Table 1), especially when the pseudo-label shift is large. Given this assumption, Figure 3 (Left), shows the prediction error of COT versus the pseudo-label shift. We can see that equipped with the extra information, COT is able to maintain a low prediction error even under very large pseudo-label shift, and the correlation between COT performance and the pseudo-label shift is weak. Since the pseudo-label shift is strongly correlated with miscalibration, this implies that COT is much more robust to miscalibration than existing miscalibration-sensitive prediction methods.

3 Methods

In this section, we formally introduce our proposed method - Confidence Optimal Transport - and propose an additional variation, COTT, that utilizes a threshold over the optimal transportation costs between two distributions, instead of taking a simple average.

3.1 Confidence Optimal Transport

Let ˆ P S ( ⃗ y ) denote the empirical source label distribution. The predicted error of COT, which is the length of the blue line segment (assuming P T ( ⃗ y ) = P S ( ⃗ y ) ) in Figure 1, is given by

By Corollary 1, ˆ ϵ COT ≥ ˆ ϵ AC, which means that COT provably predicts a larger error than AC-MC, which empirically tends to produce more accurate predictions as we find overestimation of the error far less common than underestimation. As mentioned earlier, in the case where the pseudo-label shift is large, such as in Figure 1 (Right), AC-MC can have arbitrarily large prediction error, while COT always has the following guarantee:

Proposition 2 (Calibration independent lower bound of COT) . Under the assumption that P T ( ⃗ y ) = P S ( ⃗ y ) , we always have ˆ ϵ COT ≥ 0 . 5 W ∞ ( P pseudo ( ⃗ y ) , P T ( ⃗ y )) .

Thus, we can see that COT is by nature different from existing methods because it is a miscalibrationrobust method . Its success is dependent on the difference between P T ( ⃗ y ) and P S ( ⃗ y ) , which is relatively small in most real-world scenarios, while the dependency on calibration of existing methods can not always be controlled. The geometric explanation of this is that COT measures a fundamentally different length in the Wasserstein Space ( P ( R k ) , W ∞ ) , allowing for the above guarantee which does not exist in previous methods. Moreover, as we will empirically demonstrate in the next section, large pseudo-label shift is prevalent in real models and datasets. Consequently, COT performs much better than miscalibration-sensitive methods in most cases.

3.2 Thresholding as a Robust Transport Cost Stastistic

Computationally, COT (or Wasserstein Distance in general) is implemented as a two-step process: 1) calculating individual transport costs for all samples; 2) returning the mean across all transport costs as the error estimate. While we have seen that COT has some protection against miscalibration, it is not completely immune. Another outstanding issue lies in the second step - computing the mean of all transport costs. In statistics, it is well known that the mean is less robust to outliers than the median [21]. In the supplemental material, we show that the empirical transport cost distribution is also heavy-tailed and therefore COT is susceptible to outlier costs. A large fluctuation in the outliers impacts the mean much more than it does to the median. Therefore, a more robust statistic is desired to protect COT against these outliers.

To search for such a robust statistic that is suitable for our purpose, we turn to Average Thresholded Confidence (ATC) [13] for inspiration. ATC tremendously improves the performance of AC precisely by offering such protection against the overconfident outlier predictions (shown in supp material) by returning the percentile above the threshold rather than the mean. In a similar vein, we safeguard COT against outlier costs by introducing COT with Thresholding, COTT.

Specifically, to compute the threshold t ∈ [0 , 1] , we first sample a validation set { ( x ( i ) val , y ( i ) val ) } n i =1 ∼ P S ( x, y ) . Let ˆ P val denote the empirical validation distribution. We compute the optimal transport plan P ∗ val = arg min P ∈ Π ( f ⃗ # ˆ P val ( ⃗ c ) , ˆ P S ( ⃗ y ) ) ⟨ P , C ⟩ , where C is the cost matrix determined by the L-infinity distance. Note that P ∗ val ∈ { 0 , 1 } n × n is a permutation matrix due to the equal number of confidence vectors and one-hot labels (Proposition 2.1 in [32]). Then the threshold t is set such that the validation of error the classifier equals the fraction of samples with transport cost higher than t , i.e.

With the threshold t learned from the validation set, we can compute the COTT target error estimate. Let ˆ P S ( ⃗ y ) denote the empirical source label distribution and f ⃗ # ˆ P T ( ⃗ c ) denote the empirical distribution of confidence vectors of samples from the target distribution P T ( x ) . We compute the optimal transport plan P ∗ = arg min P ∈ Π ( f ⃗ # ˆ P T ( ⃗ c ) , ˆ P S ( ⃗ y ) ) ⟨ P , C ⟩ . Our COTT estimate is then defined as

Note on Implementation: While solving the linear program of optimal transport for COT and COTT is rather straightforward with existing solvers such as POT [12], its time complexity O ( n 3 ) prohibits large-scale inputs. We bypass this problem by breaking the test dataset into small batches with a maximum size of 10,000. Concretely, given n test samples, we sample with replacement for ⌈ n/ 10 , 000 ⌉ batches and compute an error estimate for each batch. We get the final error estimation of COT and COTT by taking an average of batch estimates.

4 Experiments

In this section, we empirically compare COT and COTT with existing methods on various benchmark datasets. For all experiments, we trained the model on in-distribution data and froze the model after convergance. To predict the model's performance on the target domain, we only used unlabeled data from the target domain. When we have a test set size greater than 10,000, we show the results of utilizing the batched version of COT and COTT detailed in Section 3.2. For context, solving the OT problem of size 10,000 only takes around 10s, thus adding only negligible computation overhead.

4.1 Datasets and Nature of Shift

In our comprehensive evaluation, we consider more than 10 benchmark datasets across multiple modalities, including vision and language, with a variety of distribution shifts. Each type of shift is introduced in the following paragraphs:

Synthetic Shift: First, we consider distribution shifts caused by common visual corruptions, such as brightness, defocusing, and blurriness, which are common in real-world settings. We used the corrupted versions of CIFAR10, CIFAR100, and ImageNet proposed in [19], which includes 19 types of common visual corruptions across 5 levels of severity.

Novel Subpopulation Shift: Next, we consider novel subpopulation shifts, where the subpopulations in the train and test sets differ. For example, models might have only observed golden retrievers for the dog class but not huskies. In our experiments, we used the BREEDS benchmark [36], which leveraged the ImageNet[8] class hierarchy to create 4 datasets, Living-17, Nonliving-26, Entity13, and Entity-30. With this benchmark, we aim to understand if error estimation methods could capture challenges in an increasingly diverse world.

Natural Shift: Finally, we consider non-simulated shifts in which the distribution shifts are induced through differences in the data collection process, such as ImageNet-V2 and CIFAR10-V2 proposed in [34]. We also include ImageNet-Sketch [40], which consists of sketched images of the original ImageNet classes. Additionally, we consider distribution shifts faced in the wild, such as ones curated in the WILDS benchmark [24]. We consider four WILDS datasets, two for language tasks (Amazon-WILDS, CivilComments-WILDS) and two for vision tasks (Camelyon17, RxRx1). These datasets reflect various discrepancies between data sources happening in the real world. For example, hospitals might use different stain colors for pathological images. Different groups might follow distinct standards assigning star ratings for their reviews.

4.2 Architectures and Evaluations

For vision tasks, We trained ResNet18 (CIFAR10, CIFAR100) and ResNet50 [18] (ImageNet, Living17, Nonliving26, Entity13, Entity30, Camelyon17-WILDS, RxRx1-WILDS); for language tasks (Amazon-WILDS, CivilComments-WILDS), we fine-tuned DistilBERT-base-uncased [35]. We followed training setups from previous works [13] and provided the full details in the supplemental material. After training, we calibrated models using Temperature Scaling (TS) [17] on the indistribution validation data, effectively adjusting the output probabilities of the neural network to match the actual correctness likelihood. This approach has previously demonstrated that TS consistently improves error estimation performance for all methods [13]. To evaluate different methods, we utilized the mean absolute difference between their predicted errors and the true errors, which are obtained using ground truth labels. We refer to this metric as Mean Absolute Error (MAE).

4.3 Baselines

We consider an array of baselines to compare against our methods: COT and COTT.

Average Confidence (AC) estimates target error by taking the average of one minus the maximum softmax confidence of target data. ˆ ϵ AC = E x ∼ ˆ P T [1 -max j ∈Y f ⃗ j ( x )] .

̸

Difference of Confidence (DoC a.k.a. DOC-Feat) [16] estimates target error through the difference between the confidence of source data and the confidence of target data. ˆ ϵ DoC = E x ∼ ˆ P S [ [arg max j ∈Y f ⃗ j ( x ) = y ]] + E x ∼ ˆ P T [1 -max j ∈Y f ⃗ j ( x )] -E x ∼ ˆ P S [1 -max j ∈Y f ⃗ j ( x )] .

✶ Importance Re-weighting (IM) estimates target error as a re-weighted source error. The weights are calculated as the ratio between the number of data points in each bin in target data and source data. This is equivalent to [6] using one slice based on the underlying classifier confidence.

̸

Generalized Disagreement Equality (GDE) [22] estimates target error as the disagreement ratio of predictions on the target data using two independently trained models f ⃗ ( x ) and f ⃗ ′ ( x ) . ˆ ϵ GDE = E x ∼ ˆ P T [ [arg max j ∈Y f ⃗ j ( x ) = arg max j ∈Y f ⃗ ′ j ( x )]] .

✶ Average Thresholded Confidence (ATC) [13] first identifies a threshold t such that the fraction of source data points that have scores below the threshold matches the source error on in-distribution validation data. Target error is estimated as the expected number of target data points that fall below the identified threshold. ˆ ϵ ATC ( s ) = E x ∼ ˆ P T [ ✶ [ s ( f ⃗ ( x )) < t ]] , where s is the score function mapping the softmax vector to a scalar. Two different score functions are used, Maximum Confidence (ATC-MC) and Negative Entropy (ATC-NE).

In addition, we also compare our method to ProjNorm [41]. ProjNorm cannot provide a direct estimate, instead, the authors demonstrated their metric has the strongest linear correlation to true target error compared to existing baselines. In this case, we followed their setup and performed a correlation analysis to draw a direct comparison. We included results in the supplemental material and showed that our method consistently outperforms ProjNorm.

CIFAR-100

Nonliving-26

ImageNet

1.0

1.0

1.0

Metric

AC

0.8

0.8-

0.8-

ATC-NE

Error

Error

Error

COT

0.6

COTT

0.6

0.6

True

True

0.4

0.4

0.4

0.2

0.2-

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.00.2

0.4

0.6

0.81.0

0.2

0.4

0.6

0.8

1.0

Predicted Error

Predicted Error

Predicted Error

4.4 Results

In Table 1, we report the MAE results grouped by datasets and nature of shifts. Across all benchmarks, we observe that COT always obtains lower estimation error than AC, supporting our theoretical analysis that COT additionally leverages pseudo-label shift to fight miscalibration. On synthetic shift benchmarks (CIFAR10-Synthetic, CIFAR100-Synthetic, ImageNet-Synthetic BREEDS-same), COT is 2 -4 × better than AC, drastically reducing the estimation error. On novel subpopulation shift (BREEDS-novel), COT cuts about half of the AC error. On natural shift, COT also improves upon AC. On ImageNet natural shift datasets, COT reduces the error from 8.5 to 3.88 compared to AC. COTT, presents the best overall results, surpassing the best current method (ATC) by a notable margin. On synthetic shift benchmarks, COTT is 2-3 × better than ATC-NE, the stronger version of ATC that uses a negative entropy score function. On novel subpopulation shift benchmarks, COTT is at least 4 absolute percent better than ATC-NE. On natural shift benchmarks, however, we observed mixed results. On ImageNet-Natural, COTT is better than ATC-NE while on CIFAR10-Natural, COTT is marginally worse. On WILDS benchmarks, no single method dominantly outperforms others. Note that the WILDS benchmark datasets contain label shift, meaning P S ( y ) = P T ( y ) . This leads COTT to overestimate the error on the CivilComments-WILDS. Nonetheless, COTT has the smallest worst-case error of 5.82 compared to ATC-NE's 7.73, demonstrating its robustness even on distribution shifts faced in the wild.

̸

In Figure 4, we use scatterplots to visualize estimations given by different methods, notably AC, ATC, COT, and COTT, where we plot the predicted error against the true error. Ideally, these scattered points should demonstrate a strong linear correlation and closely follow the y = x line. While this is true for COT and COTT, AC and ATC often underestimate, sometimes by a large margin. In Nonliving-26, we can see data points representing shifts whose ATC predicted errors are around 0.1 while true errors are close to 1. These observations corroborate the necessity of leveraging pseudo-label shifts to guard against such catastrophic failures of existing confidence-based methods. We include the scatterplots for all datasets in the supplemental material, where we show that COT and COTT successfully prevent severe underestimation seen in other methods.

5 Conclusion

In this work, we proposed COT and COTT, simple yet effective methods leveraging the optimal transport framework to estimate a classifier's performance on out-of-distribution data with only unlabeled samples. Our method is motivated by the observation that existing methods failed to take into account the fact that classifiers often demonstrate pseudo-label shifts when performance drops on an unseen target domain. We formalized this observation by connecting AC and COT under the Wasserstein metric and showing that COT provably provides more conservative estimates by assuming a target marginal distribution. Combining COT with a thresholding strategy that has shown extensive empirical success, we further introduced COTT. On all eleven benchmark tasks, COT and COTT manage to avoid severe underestimation of the model's error seen in current methods. In particular, COTT achieves significantly lower prediction errors than existing methods for most models and datasets (up to 3x better), establishing new state-of-the-art results. We believe our work makes

an important step forward in making performance estimation methods more reliable and theoretically grounded and thus more applicable in real-world applications.

Limitations: Despite the strong empirical performance, our method does suffer from several limitations. First, our method is currently only applicable to multi-class classification problems under the assumption that the marginal label distribution must remain constant between the train and test distributions, though as we will show in the supplemental material, our method is still quite robust when the difference between source and target marginal is small. In addition, while we have derived a lower bound for error estimates, we have not currently formulated an upper bound; however, we have not observed overestimating the error to be a common problem empirically. Lastly, the relationship between miscalibration and pseudo-label shift is currently only empirical in nature, not theoretical.

References

David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems , 33:21428-21439, 2020.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning , pages 214-223. PMLR, 2017.

Christina Baek, Yiding Jiang, Aditi Raghunathan, and Zico Kolter. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift, 2023.

Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Normalized wasserstein for mixture distributions with applications in adversarial learning and domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6500-6508, 2019.

Krishnakumar Balasubramanian, Pinar Donmez, and Guy Lebanon. Unsupervised supervised learning ii: Margin-based classification without labels. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 137-145. JMLR Workshop and Conference Proceedings, 2011.

Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Mandoline: Model evaluation under distribution shift. In International Conference on Machine Learning , pages 1617-1629. PMLR, 2021.

Shai Ben David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages 129-136. JMLR Workshop and Conference Proceedings, 2010.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15069-15078, 2021.

Weijian Deng, Stephen Gould, and Liang Zheng. What does rotation prediction tell us about classifier accuracy under varying testing environments? In International Conference on Machine Learning , pages 2579-2589. PMLR, 2021.

Pinar Donmez, Guy Lebanon, and Krishnakumar Balasubramanian. Unsupervised supervised learning i: Estimating classification and regression errors without labels. Journal of Machine Learning Research , 11(4), 2010.

Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research , 22(78):1-8, 2021. URL http://jmlr.org/papers/v22/20-451.html .

Saurabh Garg, Sivaraman Balakrishnan, Zachary C Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of-distribution performance. arXiv preprint arXiv:2201.04234 , 2022.

Saurabh Garg, Nick Erickson, James Sharpnack, Alex Smola, Sivaraman Balakrishnan, and Zachary Lipton. Rlsbench: Domain adaptation under relaxed label shift. In International Conference on Machine Learning (ICML) , 2023.

Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231 , 2018.

Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1134-1144, 2021.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations , 2018.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136 , 2016.

Peter J Huber. Robust statistics. In International encyclopedia of statistical science , pages 1248-1251. Springer, 2011.

Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and J Zico Kolter. Assessing generalization of sgd via disagreement. arXiv preprint arXiv:2106.13799 , 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML) , 2021.

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning , pages 5637-5664. PMLR, 2021.

Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning , pages 3122-3130. PMLR, 2018.

Xinran Liu, Yikun Bai, Yuzhe Lu, Andrea Soltoggio, and Soheil Kolouri. Wasserstein task embedding for measuring task similarities. arXiv preprint arXiv:2208.11726 , 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Yuzhe Lu, Xinran Liu, Andrea Soltoggio, and Soheil Kolouri. Slosh: Set locality sensitive hashing via sliced-wasserstein embeddings. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 2566-2576, January 2024.

Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems , 32, 2019.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning , 11(5-6):355-607, 2019.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.00451 , 2018.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning , pages 5389-5400. PMLR, 2019.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019.

Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprint arXiv:2008.04859 , 2020.

Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference , 90(2):227-244, 2000.

Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-conditioned imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems , 33:13139-13150, 2020.

Cédric Villani et al. Optimal transport: old and new , volume 338. Springer, 2009.

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems , pages 10506-10518, 2019.

Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting out-ofdistribution error with the projection norm. arXiv preprint arXiv:2202.05834 , 2022.

A Deferred Proofs

For readers' convenience, we review the statements of the propositions and corollaries and provide the full proofs below.

A.1 Proof of Proposition 1

Proposition 1 ( ˆ ϵ ACW ∞ Equivalence) . Let ( P ( R k ) , W ∞ ) be the metric space of all distributions over R k , where W ∞ is the Wasserstein distance with c ( x, y ) = ∥ x -y ∥ ∞ . Then, the estimated error of AC-MC is given by ˆ ϵ AC = W ∞ ( f ⃗ # P ( ⃗ c ) , P pseudo ( ⃗ y )) .

Proof. We first show the following equality. Let j ∗ = arg max j f ⃗ j ( x )

where ⃗ y j = ✶ [ j = j ∗ ] . Let f ⃗ -j ∗ ( x ) denote the vector f ⃗ ( x ) with j ∗ -th element removed. Since for a confidence vector ∥ f ⃗ ( x ) ∥ 1 = 1 ,

Therefore, we have obtained the desired Equality 1:

Next, we consider the optimal transport plan between f ⃗ # P ( ⃗ c ) and P pseudo ( ⃗ y ) . Namely, we show all confidence vectors f ⃗ ( x ( i ) ) are coupled with their one-hot pseudo-labels ⃗ y ( i ) . This can be observed by the fact that the one-hot pseudo-label is the one-hot label that achieves the lowest L-infinity cost, i.e.

Suppose there exist confidence vectors that are not coupled with their one-hot pseudo-labels, then all individual costs are suboptimal and the total cost is suboptimal as well, contradicting the assumption that the transport plan is optimal. Therefore,

Combining Equality 1 and Equality 2, we obtain the desired relationship between AC error estimate and W ∞ distance

A.2 Proof of Corollary 1

Corollary 1 ( P pseudo ( ⃗ y ) is closest to f ⃗ # P ( ⃗ c ) ) . Let P ′ ( ⃗ y ) ∈ P ( { 0 , 1 } k ∩ ∆ k -1 ) be a one-hot label distribution. Then W ∞ ( f ⃗ # P ( ⃗ c ) , P ′ ( ⃗ y )) ≥ W ∞ ( f ⃗ # P ( ⃗ c ) , P pseudo ( ⃗ y )) .

Proof. We first show the following equality, which establishes the relationship between AC accuracy estimate with W ∞ distance:

Since W ∞ ( f ⃗ # P ( ⃗ c ) , δ 0 ) transports f ⃗ # P ( ⃗ c ) to δ 0 , the optimal transport plan couples every element in f ⃗ # P ( ⃗ c ) to 0 . For each x ( i ) , its confidence vector f ⃗ # P ( ⃗ c ) has a transport cost ∥ f ⃗ ( x ( i ) ) -0 ∥ ∞ . Hence,

With this, our inequality is simply the Triangle Inequality in ( P ( R k ) , W ∞ ) ,

Combined with Equation 3, we obtain the desired inequality

A.3 Proof of Proposition 2

Notations: Let C ( ⃗ c ) = { ⃗ c ′ ∈ ∆ k -1 | arg max j ⃗ c ′ j = arg max j ⃗ c j } be the set of confidence vectors whose one-hot pseudo-labels that match with that of ⃗ c ∈ ∆ k -1 . Let P (∆ k -1 ) be the set of all distributions of confidence vectors and P c [ P pseudo ( ⃗ y )] = { P ′ ( ⃗ c ) ∈ P (∆ k -1 ) | P pseudo ( ⃗ y ) = P ′ (arg max j ⃗ c j = arg max j ⃗ y j ) } be the set of distributions of confidence vectors that share the same pseudo-label distribution P pseudo ( ⃗ y ) .

P c [ P pseudo ( ⃗ y )] defines an equivalence class for the space of distributions of confidence vectors ( P (∆ k -1 ) , W ∞ ) that share the same pseudo-label distribution P pseudo ( ⃗ y ) . Pictorially, in Figure 1, P c [ P pseudo ( ⃗ y )] represents the line between δ 0 and P pseudo ( ⃗ y ) . On this line, every distribution of confidence vectors shares the same pseudo-label distribution P pseudo ( ⃗ y ) .

To prove Proposition 2, we need the following lemma, which intuitively allows us to change the metric from measuring the distance between two points to the distance between an equivalence class and a point. *

Lemma 1 (Change-of-metric) . Let ⃗ y, ⃗ y ′ ∈ { 0 , 1 } k ∩ ∆ k -1 be two one-hot labels. Then the following holds

̸

Proof. If ⃗ y = ⃗ y ′ , then we know the optimal ⃗ c = ⃗ y

If ⃗ y = ⃗ y ′ , then we proceed by showing equality with two inequalities. First, observe { (0 . 5 + δ ) ⃗ y + (0 . 5 -δ ) ⃗ y ′ | δ ∈ (0 , 0 . 5] } ⊂ C ( ⃗ y ) .

̸

̸

We are now in a position to prove Proposition 2, which follows from the somewhat surprising fact that the left-hand side of the inequality is simply the distance between f ⃗ # P T ( ⃗ c ) and P T ( ⃗ y ) with a change-of-metric to the metric defined above.

Proposition 2 (Calibration independent lower bound of COT) . Under the assumption that P T ( ⃗ y ) = P S ( ⃗ y ) , we always have ˆ ϵ COT ≥ 0 . 5 W ∞ ( P pseudo ( ⃗ y ) , P T ( ⃗ y )) .

Proof. Since P ( ⃗ y ) = P ( ⃗ y ) ,

Equation 5 follows from the observation that C ( ⃗ c ) = C ( ⃗ y ′ ) for a confidence vector ⃗ c and its corresponding one-hot pseudo-label ⃗ y ′ . Furthermore, since our new metric inf ⃗ c ′ ∈C ( ⃗ y ′ ) ∥ ⃗ c ′ -⃗ y ∥ ∞ is only defined up to the equivalence class, replacing each ⃗ c ′ ∈ C ( ⃗ c ) with its pseudo-label ⃗ y ′ does not change the distance.

Plugging in Lemma 1,

A.4 Tightness of Proposition 2

While Inequality 4 seems loose, our bound is, in fact, tight if no further assumptions on the calibration status of the classifier f ⃗ are made. We need the following lemma that establishes the relationship between pseudo-label shift and the total variation distance between target label distribution and pseudo-label distribution. Note this equivalence only makes sense in the context of measuring W ∞ distance between two one-hot label distributions, but not under other contexts presented in the paper.

Lemma 2 (Pseudo-label shift is total variation) .

̸

Proof. For two ⃗ y, ⃗ y ′ ∈ { 0 , 1 } k ∩ ∆ k -1 , the transport cost c ( ⃗ y, ⃗ y ′ ) = ✶ [ ⃗ y = ⃗ y ′ ] . Then, the standard result on optimal transport [39] gives the desired equality. Corollary 2.

Proof.

Finally, we show Proposition 2 is tight by constructing a sequence of distributions of confidence vectors, the limit of which is exactly 0 . 5 W ∞ ( P pseudo ( ⃗ y ) , P T ( ⃗ y )) away from P T ( ⃗ y ) .

̸

Lemma 3 (Proposition 2 is tight) .

Proof. First, we construct the following family of distributions { P δ ( ⃗ c ) | δ ∈ (0 , 0 . 5] } , where P δ ( ⃗ c ) is the following mixture distribution

where P ∩ ( ⃗ y ) = γ -1 min { P pseudo ( ⃗ y ) , P T ( ⃗ y ) } , γ = ∑ ⃗ y ∈{ 0 , 1 } k ∩ ∆ k -1 min { P pseudo ( ⃗ y ) , P T ( ⃗ y ) } , P × ( ⃗ t ) is a distribution supported on ∆ k -1 ∩ { 0 . 5 + δ, 0 . 5 -δ, 0 } k (i.e. one element in ⃗ t is 0 . 5 + δ , another is 0 . 5 -δ , and the rest are 0 ). Additionally, P × ( ⃗ t i = 0 . 5 + δ ) = P pseudo ( ⃗ y i = 1) and P × ( ⃗ t i = 0 . 5 -δ ) = P T ( ⃗ y i = 1) . It is easy to check that P δ ( ⃗ c ) ∈ P c [ P pseudo ( ⃗ y )] .

Next, we construct an explicit transport plan π ( ⃗ c, ⃗ y ) ∈ Π( P δ ( ⃗ c ) , P T ( ⃗ y )) . We construct it via the factorization π ( ⃗ c, ⃗ y ) = P δ ( ⃗ c ) π ( ⃗ y | ⃗ c ) , where

The cost of this transport plan is therefore

where the last equality follows from Lemma 2. Taking infimum,

Combining everything so far, we obtain the desired result:

Inequality 6 follows from the fact that the optimal transport plan cannot have a greater cost than our explicit plan π . Inequality 7 is due to the fact that the family of distribution we are considering is a subset of P c [ P pseudo ( ⃗ y )] . Inequality 8 is an application of the lower bound 4 which holds for all P ( ⃗ c ) ∈ P c [ P pseudo ( ⃗ y )] .

B Extended Results

B.1 Results with Standard Deviation

We show the full experimental results with standard deviation in Table 2.

B.2 Qualitative Results

We show the qualitative results (scatter plots) in Fig 5

B.3 Correlation Analysis

ProjNorm [41] leverages pseudo labels on the target domain to retrain a copy of the reference model trained on the source domain. The authors show that the difference between the two models' parameters has a strong linear correlation to the true target error. Following the paper's experimental setup, we conducted the correlation analysis on CIFAR10 and CIFAR100 using three architectures, ResNet18, ResNet50, and VGG11. We note that ProjNorm in fact implicitly leverages the assumption that P T ( y ) = P S ( y ) as this condition holds for both CIFAR10 and CIFAR100. As Fig. 18 of their paper [41] shows, ProjNorm tends to overestimate when label shift exists.

B.4 Mild Label Shift

We motivate our methods under the assumption of no label shift. In Proposition 2, we showed that the worst-case underestimate of COT is half of the pseudo-label shift. Under mild label shifts, the guarantee for such worst-case underestimation becomes weaker. This can be observed from the following corollary of Proposition 2:

CIFAR-10

CIFAR-100

ImageNet

1.0-

1.0-

0.8-

Metric

0.8-

0.8-

AC

True Error

0.6-

TrueError

0.6-

0.6-

ATC-MC

0.4-

▲COT

0.4-

0.4-

+ COTT

0.2-

0.2-

0.2-

0.0

8090t0200°0

0.20.40.60.81.0

0.20.40.60.81.0

PredictedError

PredictedError

Predicted Error

Living-17

Nonliving-26

Entity-13

Entity-30

1.0-

1.0

1.0-

1.0

0.8-

0.8-

0.8-

0.8-

0.6-

True Error

0.6-

True Error

0.6-

Error

0.6

0.4-

0.4-

0.4

0.2-

0.2-

0.2-

0.2

0.2

0.4

0.6

0.81.0

0.0+

0.00.20.40.60.81.0

0.0+

0.00.20.40.60.81.0

0.2

0.4

0.6

0.81.0

PredictedError

Predicted Error

Predicted Error

PredictedError

Camelyon17

RxRx1

Amazon

CivilComments

0.30

0.40-

0.5-

0.9-

0.25

0.35

0.4-

True Error

0.20

True Error

0.30

0.3-

0.15-

0.10-

0.7-

0.25

0.2-

0.05

0.20

0.1-

0.6-

0.00

0.00

0.10

0.20

0.30

0.6

0.7

0.8

0.9

0.15

0.15

0.25

0.35

0.0

0.00.10.20.30.40.5

Predicted Error

Predicted Error

Predicted Error

Predicted Error

Corollary 3 (Calibration independent lower bound of COT under mild label shift) .

Proof. By Triangle Inequality in ( P ( R k ) , W ∞ ) ,

Combined with Proposition 2, we obtain the desired result.

As the label shift increases, we have a weaker guarantee of the worst-case underestimation error of COT as long as W ∞ ( P S ( ⃗ y ) , P T ( ⃗ y )) ≤ 0 . 5 W ∞ ( P pseudo ( ⃗ y ) , P T ( ⃗ y )) . However, we perform additional controlled experiments which suggest our methods remain to be the most performant despite the theoretical guarantee is not as strong as the case without label shift.

To simulate mild label shift for datasets with P S ( ⃗ y ) = P T ( ⃗ y ) , we first calculate the original target marginal and then sample the shifted target marginal from a Dirichlet distribution as in [14] with a parameter α = 50 . The parameter α controls the severity of the label shift, and a smaller α means a larger label shift. Concretely, let the shifted target marginal be P ˜ T ( ⃗ y ) . Then P ˜ T ( ⃗ y ) ∼ Dir( β ) where β ( ⃗ y ) = α · P T ( ⃗ y ) . Finally, based on P ˜ T ( ⃗ y ) , we sample a new set of test samples for which we estimate the performance. We conducted this mild label shift experiment for CIFAR10, CIFAR100, ImageNet, Living17, Nonliving26, Entity13, and Entity30 as these datasets have the same source and target marginal. We showed the results in Table 3. As we can see, our methods still dominate existing methods under this relaxed condition.

ATC > AC

ATC ≈ AC

COTT > COT

COTT ≈ COT

4

4

density

density

density

density

10

2

2

2

0

0

0

0

0.0

0.5

0.0

0.5

0.0

0.5

1.0

0.0

0.5

1.0

cost

cost

cost

cost

B.5 When does thresholding improve over averaging?

In this section, we provide some intuitions on when using a threshold provides better estimates than taking the average. From Fig. 6, we show that thresholding yields larger and more accurate error estimates when the cost distribution on the OOD data is more spread out and less concentrated around 0. By contrast, when the cost distribution is mostly near 0, thresholding leads to similar estimates as averaging. Interestingly, even on OOD data where the model has very low performance, there is still a decent amount of samples whose cost is near 0. Thus, when taking the average, we will end up with a smaller value which suggests a low error. In these cases, thresholding will give larger error estimates than averaging.

C Datasets

CIFAR10: The synthetic shifts included 19 common visual corruptions across 5 levels of severity from [19]. The natural shift is CIFAR10-V2 [33].

CIFAR100: The synthetic shifts included 19 common visual corruptions across 5 levels of severity from [19].

ImageNet: The synthetic shifts included 19 common visual corruptions across 5 levels of severity from [19]. The natural shifts include 4 datasets from ImageNet-V2 [34] and ImageNet-Sketch [40].

BREEDS: The BREEDS benchmark contains 4 datasets, Living-17, Nonliving26, Entity13, Entity30. For each of the datasets, the same subpopulation shifts include the corrupted versions of the test set with the same subpopulation; the novel subpopulation shifts include the clean as well as corrupted versions [19] of the test set with novel subpopulation.

WILDS: For all WILDS datasets, we used the official OOD datasets provided in their paper [25].

D Experiment Setup

We performed training in PyTorch [31], and we used RTX 6000 Ada GPUs.

For datasets without an official validation set, we randomly sampled a subset of the official training set as the validation set to perform calibration and learn thresholds for ATC and COTT. We trained 3 models for each dataset with random seeds { 0 , 1 , 10 } .

CIFAR10 and CIFAR100: We reserved 10000 images from the training set as the validation set. We trained ResNet18 from scratch, using SGD with momentum equal to 0.9 for 300 epochs. We set

weight decay to 5 × 10 -4 and batch size to 200. We set the initial learning rate to 0.1 and multiply it by 0.1 every 100 epochs.

ImageNet: Wereserved 50000 images from the training set as the validation set. We used ResNet50. While ImageNet pretrained weights are available in PyTorch, we needed multiple ones trained using different initializations. Due to limited computation resources, we reused the upper layer weights but reinitialized the last layer with different random seeds. We finetuned the whole model using Adam [23] with a batch size of 64 and a learning rate of 10 -4 , for 10 epochs.

BREEDS: We used the intersection set of images that are both in the ImageNet validation images we set aside and the BREEDS dataset as the validation set. For all BREEDS datasets (Living17, Nonliving26, Entity13, Entity30), we trained ResNet50 from scratch.

For Living17 and Nonliving26, we used SGD with weight decay of 10 -4 and batch size of 128. We trained for 450 epochs. We set the initial learning rate to 0.1 and multiplied it by 0.1 every 150 epochs.

For Entity13 and Entity30, we used SGD with weight decay of 10 -4 and batch size of 128. We trained for 300 epochs. We set the initial learning rate to 0.1 and multiplied it by 0.1 every 100 epochs.

Camelyon17-WILDS: We used the id_val group as the validation set. We fine-tuned ImageNet pretrained ResNet50 using SGD with momentum of 0.9, weight decay of 5 × 10 -4 , and batch size of 32, for 5 epochs.

RxRx1-WILDS: We used the id_text group as the validation set. We followed [25] to fine-tune an ImageNet pretrained ResNet50. We used Adam with weight decay of 10 -5 and batch size of 75, for 90 epochs. We increased the learning rate from 0 to 10 -4 linearly for the first 10 epochs and decayed it following a cosine learning rate schedule.

Amazon-WILDS: We used the id_val group as the validation set. We followed [25] to fine-tune a DistilBERT-base-uncased model [35]. We used AdamW [28] with weight decay of 10 -2 , learning rate of 10 -5 , and batch size of 8, for 3 epochs. We set the maximum number of tokens to 512.

CivilComments-WILDS: We used the val group as the validation set. We followed [25] to fine-tune a DistilBERT-base-uncased model [35]. We used AdamW [28] with weight decay of 10 -2 , learning rate of 10 -5 , and batch size of 16, for 5 epochs. We set the maximum number of tokens to 300.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors aim to predict model’s performance on out-of-distribution (OOD) data without access to labels during testing. Specifically, they identify pseudo-label shift, which gauges  the difference between the predicted and the true OOD label distributions, to indicate the under-estimation issue suffered by the existing methods. Based on the obervision, they propose Confidence Optimal Transport (COT) and an empirically-motivated variant of COT, Confidence Optimal Transport with Thresholding (COTT), for more robust OOD error estimation. Experiments are conducted on 10 datasets with various types of distribution shift to verify the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The motivation in this work is clear. The authors explain the under-estimation issue encountered by previous studies through the lens of pseudo-label drift, and propose a method leveraging optimal transport theory based on the observision.\n\n2. The code is available with the submission which improves the reproducibility index of the paper.\n\nWEAKNESSES:\n1. Since this paper contains many mathematical symbols, the meaning of some symbols is confused, which increases the difficulty of understanding. For example, in Section 2.1, you mention to predict the error $\\epsilon = 1-\\alpha$, but later the predicted error is denoted as $\\hat{\\epsilon}$. Comparing Figure 1 with Section 3.1, I find the assumption changes from $P_T(\\vec{y}) \\approx P_S(\\vec{y})$ to $P_T(\\vec{c}) \\approx P_S(\\vec{y})$. Is the meaning of $P_T(\\vec{y})$ equivalent to that of $P_T(\\vec{c})$? In addition, in Section 3.1, what is the meaning of $\\hat{P}_S(\\vec{y})$ and $\\hat{P}_T(\\vec{y})$? I suggest that the authors explain them after every equation and use harmonized symbols.\n\n2. The relationships between pseudo-label drift and the true target error is not clear. In Section 2.4, you briefly mention that the pseudo-label shift is a lower bound of the true target error of the model. Can you explain the property in details? Specifically, I prefer to know the relationships between your method (or miscalibration) and the true target error. Can you give some insights?\n\n3. This work is potentially vulnerable to two cases: 1) imbalanced test set.  2) The label space of the test set is only the subset of that of the training set.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nThe author provides the limitations of their work. It would be better if the authors discuss more cases such as the two cases mentioned in the main review.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors develop a methodology for estimating the accuracy of a classifier in the case where the distribution of the test set is different from the one used in the training data. Their methodology is based on computing a certain Wasserstein distance between suitable distributions on labels. Authors argue this definition is robust to pseudo label shift and validate their claims through extensive experimental validation.\n\nSTRENGTHS:\nStrengthsI think overall it is a good, strong paper. It addresses an important problem and it provides a highly innovative perspective on that problem. Experiments are compelling and make a strong point that the method is able to combat otherwise pervasive underestimation of error for of out of distribution samples.\n\nWEAKNESSES:\nThe main weakness is the lack of sufficient clarity. The entire argument is subtle and required a super clear explanation. Unfortunately, the way the manuscript is written does not help much. Another weakness is that while this is promoted as an OT based method, the theoretical results (while seemingly correct) don’t enlighten much about what the role of Optimal transport is.\n\nQUESTIONS:\n1)Proposition 2 is nice but it is unclear how such lower bound can be ever applied. In other words, the estimated error could be 1 always and the result is useless. Authors should elaborate more on the significance of this result.\n\n2)Similarly, Corollary 1 is helpful (line 189) but again, doesn't say the proposed method is always good. What AC got it just right while COT overestimates the error? The fact that there is a GAP between great experimental results and these somewhat weak theoretical results is a bit too bad and authors (although perhaps not here) should aim to clarify this.\n\n3)Authors may try to better explain the role of the Wasserstein infinity distance here. Is there anything special about this metric space that makes things work or it is just a mathematical convenience? The proofs are based on geometrical intuitions, but is there anything special about the Wasserstein geometry here?\n\n4) It would be great if the authors could elaborate more on why the Pseudo label shift is the right quantity to focus on.  By line 147 I am worried that the observed correlation showed in the experiments could be an artifact of the fact that the pseudo label shift is bounded by \\epsilon. How can this be ruled out?\n\n5) The definition of Pppseuudi in line 95 page 3 is awkward as it is not immediately clear whether it refers to domain X or Y.\n\n6) inline equation in line 188 page 5 seems wrong. Could you please correct?\n\nLIMITATIONS:\nn.a.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposed Confidence Optimal Transport (COT) and a variant, Confidence Optimal Transport with Thresholding (COTT) based on pseudo-label shift to characterize the error between predicted and true OOD label distribution.

STRENGTHS:
1. The idea of using pseudo-label shift to characterize error is natural and straight-forward.
2. Experiment results are reasonable.

WEAKNESSES:
1. This work does not have sufficient theoretical support for the method.
2. I am not fully convinced that the main assumption of same source label distribution and target label distribution is reasonable.

QUESTIONS:
1. I don't understand the assumption: "the target label distribution is close to the source label distribution" and think this assumption conflicts with the purpose of detecting OOD error --the test set should be assumed to behave differently from the training set and that's where the error come from. I wonder if authors could help clarify this assumption.
2. Figure 2 and 3 do not seem clear to me, could authors elaborate more on the purpose of including types of corruption in the figure?

LIMITATIONS:
Yes, the authors have addressed their limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to compute OOD error estimates using a Wassertein distance between the target label distribution and the distribution of confidence vectors. This is in contrast to AC-MC which compare the pseudo-label distribution and the confidence distribution. The paper motivates COT (confidence optimal transport) and COT - threshold to handle outliers and shows they are useful metrics in understanding OOD error. The paper does a variety of experiments to evaluate how the proposed methods do in fact estimate OOD error bettter than existing work.\n\nSTRENGTHS:\n1. The experiments in the paper show clear advantage of COT and COTT for predicting OOD error using unlabelled samples.\n2. The presentation in the paper is clear, barring some organization issues.\n3. The paper tackles the important problem of estimating OOD error without labels and the proposed method is not very complicated.\n\nWEAKNESSES:\n1. The paper does not specify the kind of distribution shift under which the method works; causal shift, anticausal shift, covariate etc.. For example, there are distribution shifts where p_S(X)  = p_T(X), which, given good prediction, would mean small COT and COTT (because confidences are concentrated around the correct label). Could the authors explain how this may not be an issue? Better, could the authors evaluate the methods on a dataset like ColoredMNIST with a flipped spurious correlation between train and test.\n\n\n2. The writing could be improved by introducing COT and COTT before showing plots with them.\n\n3. I'm a little fuzzy on the intuition of which AC-MC does not work as well as COT. Is it that the confidence distribution becomes more diffuse with distribution shift and AC-MC does not scale with that \"diffusion\" because the pseudo-label distribution also depends on the confidence distribution, but COT being that it looks at the $p_T(Y)$, breaks that dependennce?\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nIn one sense. The weaknesses section outlines the issue of understanding what kind of OOD the method is really testing.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript studies the challenges caused by out-of-distribution (OOD) data in machine learning models. They identify \"the difference between the predicted and true OOD label distributions\" (called pseudo-label shift) as the main reason many existing works underestimate OOD errors. They propose a new approach called Confidence Optimal Transport (COT), which uses optimal transport theory to provide more robust error estimates in the presence of pseudo-label shift. Another approach they propose is COT with Thresholding (COTT), which further applies thresholding to individual transport costs to improve the accuracy of COT's error estimates. The authors present impressive experiments to conclude that the COTT approach outperforms existing state-of-the-art OOD error-predicting methods. Predicting a classifier's performance on out-of-distribution (OOD) data is a significant challenge in machine learning, so this manuscript is timely and important.\n\nSTRENGTHS:\n- Due to the use of the Wassienstien distance, COT is robust to miscalibration, meaning it performs well even when the predicted probabilities of the model are not well-calibrated. This is a significant advantage as most other metrics are sensitive to miscalibration. \n\n- It looks like COT and COTT are better at handling pseudo-label shifts, i.e., when there are differences between the predicted and true OOD label distributions. Table 1 extensively compares COT and COTT against other metrics for measuring the OOD errors.\n\n- It is very welcomed that COT provides some performance guarantee. Under the assumption that the true and predicted label distributions are the same, COT predicts an error of less than half the Wasserstein distance between the pseudo-label distribution and the true label distribution\n\nWEAKNESSES:\n- The computation of COT and COTT involves solving a linear program of optimal transport. Yes, the authors propose a batching technique to make their methods scalable, but there is a concern that COTT is computationally infeasible for large-scale inputs. \n\n- In most practical setups, COT doesn't have any performance guarantees because the assumption that the true and predicted label distributions are the same is rarely valid.\n\nQUESTIONS:\n- The performance of COTT depends on the threshold. The choice of the threshold could impact the performance of COTT. How sensitive is COTT to the threshold value? Is thresholding effective in reducing the impact of outliers?\n\nLIMITATIONS:\nThe paper has a particular section addressing the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-22"}