Title: ADAPTING TO LABEL SHIFT WITH BIAS-CORRECTED CALIBRATION

Abstract: Label shift refers to the phenomenon where the marginal probability p(y) of observing a particular class changes between the training and test distributions, while the conditional probability p(x|y) stays fixed. This is relevant in settings such as medical diagnosis, where a classifier trained to predict disease based on observed symptoms may need to be adapted to a different distribution where the baseline frequency of the disease is higher. Given estimates of p(y|x) from a predictive model, Saerens et al. (2002) proposed an Expectation Maximization (EM) algorithm to efficiently correct for the difference in class proportions between training and test distributions. A limiting assumption of this EM algorithm is that the probabilities p(y|x) are calibrated, which is not true of modern neural networks. Recently, Black Box Shift Learning (BBSL) (Lipton et al., 2018) and Regularized Learning under Label Shifts (RLLS) (Azizzadenesheli et al., 2019) have emerged as state-of-the-art techniques to cope with label shift when a classifier does not output calibrated probabilities. In this work, we show that while EM in the absence of calibration achieves weak performance with modern neural networks, the combination of EM with appropriate calibration typically outperforms BBSL and RLLS. In particular, we show that the popular calibration strategy of Temperature Scaling (TS) is often insufficient to completely correct miscalibration; rather, EM achieve state-of-the-art results primarily when used with a calibration approach that includes class-specific bias parameters that TS lacks. We further introduce a theoretically principled strategy for estimating source-domain priors in EM-based domain adaptation that allows the EM algorithm to behave more robustly when the calibrated probabilities contain systematic bias. Colab notebooks reproducing experiments on CIFAR10, CIFAR100, MNIST and diabetic retinopathy detection are available at (anonymized link):

1. INTRODUCTION:

Imagine we train a classifier in country A to predict whether or not a person has a disease based on observed symptoms, and that we hope to deploy this classifier in country B, which has poorer access to healthcare. If the prevalence of the disease in country B is higher in than in country A, the classifier might systematically misdiagnose people as not having the disease. How can we adapt the classifier to cope with the difference in the baseline prevalence of the disease in the two countries? Formally, let y denote our labels (e.g. whether or not a person is diseased), and let x denote the observed symptoms. Let us denote the joint distribution (x, y) in country A (our "source" domain) as P, and let us denote the distribution in country B (our "target" domain, where we do not have labels) as Q. How can we adapt a classifier trained to estimate p(y|x) (the conditional probability in distribution P) so that it can instead estimate q(y|x) (the conditional probability in distribution Q)? Absent assumptions about the nature of the shift between P and Q, this problem is intractable. However, if the disease generates similar symptoms in both countries, we can assume that p(x|y) = q(x|y), and that the shift in the joint distribution q(x, y) is due to a shift in the label proportion q(y). Formally, we assume that q(x, y) = p(x|y)q(y). This is known as label shift or prior probability shift (Amos, 2008) , and it corresponds to anti-causal learning (i.e. predicting the cause y from its effects x) (Schoelkopf et al., 2012) . Anti-causal learning is appropriate for diagnosing diseases given observations of symptoms because diseases cause symptoms.

Given estimates of p(y) and p(y|x), Saerens et al. (2002) proposed a simple Expectation Maximization (EM) procedure to estimate q(y) without needing to estimate p(x|y). However, estimates of p(y|x) derived from modern neural networks are often poorly calibrated (Guo et al., 2017) , and the lack of calibration can decrease the effectiveness of EM. As an alternative, Lipton et al. (2018) developed a technique called Black Box Shift Learning (BBSL) that can work even when the predictions p(x|y) are not calibrated. Azizzadenesheli et al. (2019) further improved upon BBSL in a technique known as Regularized Learning under Label Shifts (RLLS). Both BBSL and RLLS leverage information in a confusion matrix calculated on a held-out portion of the training set. To our knowledge, neither BBSL and RLLS have been benchmarked against the EM approach.

Although the EM approach is limited by the assumption that the predictions p(y|x) are calibrated, a number of recent techniques have been proposed to correct for miscalibration of p(y|x) using a held-out portion of the training set (Guo et al., 2017) . The held-out set can be thought of as analogous to the held-out set used in BBSL and RLLS to calculate a confusion matrix. This suggests a simple yet novel hybrid algorithm for adapting to label shift: first, calibrate predictions using the held-out training set, then perform domain adaptation on the calibrated predictions using EM. In this work, we studied the effectiveness of this hybrid algorithm. More generally, we studied the impact of calibration on domain adaptation to label shift. Our primary contributions are as follows:

1.1 OUR CONTRIBUTIONS 1. In experiments on CIFAR10, CIFAR100, MNIST and Diabetic Retinopathy Detection, we found that EM achieves state-of-the-art results when used with an appropriate calibration approach. Although BBSL and RLLS both benefit from calibration, they did not tend to outperform EM when the probabilities were well-calibrated.

2. We observed that the popular calibration approach of Temperature Scaling (TS) (Guo et al., 2017) does not tend to achieve state-of-the art results in the context of adaptation to label shift, possibly owing to large systematic biases in the calibrated probabilities (Fig. 1 ). The best results are obtained using variants of TS that contain class-specific bias capable of correcting for systematic bias.

3. We identify a theoretically-grounded strategy for computing the source-domain priors in EM-based domain adaptation that can be critically important when calibrated probability estimates have systematic bias. Empirically, we find that computing source priors in this way substantially improves the robustness of EM to the presence of systematic biases in the class probabilities. We also prove that the likelihood function of EM is concave and unimodal; thus, EM converges to the maximum likelihood estimate.

2. BACKGROUND:



2.1. TEMPERATURE SCALING, VECTOR SCALING AND EXPECTED CALIBRATION ERROR:

Calibration has a long history in the machine learning literature (DeGroot and Fienberg, 1983; Platt, 1999; Zadrozny and Elkan; 2002; Niculescu-Mizil and Caruana, 2005; Kuleshov and Liang, 2015; Naeini et al., 2015; Kuleshov and Ermon, 2016) . In the context of modern neural networks, Guo et al. (2017) showed that Temperature Scaling, a single-parameter variant of Platt Scaling (Platt, 1999) , was effective at reducing miscalibration. Temperature scaling performs calibration by introducing a temperature parameter T to the logit vector of the softmax. Let z(x k ) represent a vector of the original softmax logits for example x k , and let y i be a random variable representing the label for class i. With temperature scaling, we have p(y i |x k ) = e z(x k ) i /T j e z(x k ) j /T , where T is optimized with respect to the Negative Log Likelihood (NLL) on a held-out portion of the training set, such as the validation set. Guo et al. (2017) compared TS to an approach defined as Vector Scaling (VS), where a different scaling parameter was used for each class along with class-specfic bias parameters.

Formally, in vector scaling, p(y i |x k ) = e (z(x k ) i W i )+b i j e (z(x k ) j W j )+b j . Guo et al. (2017) found that vector scaling had a tendency to perform slightly worse than TS as measured by a metric known as the Expected Calibration Error (Naeini et al., 2015) . To compute the ECE, the predicted probabilities for the output class are partitioned into M equally spaced bins, and the weighted average of the difference between In a seminal paper on label shift adaptation, Saerens et al. (2002) proposed an EM algorithm for estimating the shift in the class priors between the training and test distributions. Let q(s) (y = i) denote the estimate (from EM iteration s) of the prior probability q(y = i) of observing class i in the test set. The algorithm proceeds as follows: first, q(0) (y = i) is initialized to be equal to the class priors p(y = i) estimated from the training set. Then, the conditional probabilities in the E-step are computed

as q(s) (y = i|x k ) = q(s) (y=i) p(y=i) p(y=i|x k ) n j=1 q(s) (y=j) p(y=j) p(y=j|x k ) .

Finally, the prior estimates in the M-step are updated as q(s+1) (y

= i) = 1 N N k=1 q(s) (y = i|x k ),

where N is the number of examples in the testing set. The E and M steps are iterated until convergence. As there is no need to estimate p(x|y) in any step of the EM procedure, the algorithm can scale to high-dimensional datasets. Note this procedure assumes the conditional probability estimates p(y = i|x k ) are calibrated.

2.3. LABEL SHIFT ADAPTATION VIA BLACK BOX SHIFT LEARNING AND REGULARIZED LEARNING UNDER LABEL SHIFTS:

Following the EM approach of Saerens et al. (2002) , several additional approaches for labels shift adaptation have emerged (Chan and Ng; Storkey; Schoelkopf et al., 2012; Zhang et al., 2013; Lipton et al., 2018; Azizzadenesheli et al., 2019) . Many of these approaches build estimates p(x|y), which can scale poorly with dataset sizes and underperform on high-dimensional data (Lipton et al., 2018) . Lipton et al. (2018) proposed Black-Box Shift Learning (BBSL), which strives to efficiently estimate the weights [w] i = q(y=i) p(y=i) even in cases where the prediction model p(y = i|x k ) is poorly calibrated or biased. BBSL proceeds as follows: let f be a function that accepts an input and returns the model's predicted class, x k denote an example from a held-out portion of the training set, and x k denote an example from the testing set. The empirical estimate of w, denoted as ŵ, is computed as ŵ = Ĉ-1 ŷ,y ûŷ , where

[û ŷ ] i = k 1{f(x k )=i} m and [ Ĉŷ,y ] ij = 1 n k 1{f(x k ) = i and y k = j}.

Because the approach above is not guaranteed to produce positive values for all elements of ŵ, any negative elements of ŵ are set to 0 after they are estimated. Domain adaptation is then performed by retraining the model on the entire training set distribution with examples upweighted in accordance with ŵ. Lipton et al. (2018) denote the version of BBSL described above as BBSL-hard. They also compare to a variant that they call BBSL-soft, which they describe as the case where where f outputs probabilities rather than hard classes. We interpreted this to mean Azizzadenesheli et al. (2019) further improved upon BBSL by including regularization terms in a technique known as Regularized Learning under Label Shift (RLLS). In our experiments, we compare to BBSL-hard, BBSL-soft, RLLS-hard and RLLS-soft. Regularization parameters for RLLS were set in accordance with the hard-coded values present in the publicly available code provided by the authors at https://github.com/Angela0428/labelshift/blob/ 5bbe517938f4e3f5bd14c2c105de973dcc2e0917/label_shift.py#L453-L456.

[û ŷ ] i = k f (x k )i m and [ Ĉŷ,y ] ij = 1 n k f (x k ) i 1{y k = j}.

Note that BBSL and RLLS both require a portion of the training set to be held out during the initial training phase in order to accurately estimate the confusion matrix Ĉŷ,y ; in our experiments involving calibration, we use this same heldout set to calibrate the model.

3. METHODS:



3.1. NO-BIAS VECTOR SCALING AND BIAS-CORRECTED TEMPERATURE SCALING:

As shown in Fig. 1 , we often found that TS alone resulted in systematically biased estimates of p(y i |x k ), while VS, a generalization of TS that contains both class-specific bias terms and classspecific scaling terms, did not exhibit as much systematic bias. Intrigued by this observation, we investigated the performance of two intermediaries between Temperature Scaling and Vector Scaling. The first, which we refer to as No Bias Vector Scaling (NBVS), is equivalent to vector scaling but with all the class-specific bias parameters fixed at zero. The second, which we refer to as Bias-Corrected Temperature Scaling, is equivalent TS Scaling but with the addition of the class-specific bias terms from VS. As with TS and VS, the parameters are optimized to minimize the NLL on the validation set. Note that in the case of binary classification, the parameterization of BCTS reduces to Platt Scaling (Platt, 1999) . Thus, BCTS can be viewed as a multi-class generalization of Platt scaling.

3.2. DEFINING SOURCE-DOMAIN PRIORS IN THE EM ALGORITHM:

The EM algorithm of (Saerens et al., 2002) requires the user to provide estimates of the sourcedomain prior class probabilities p(y = i). Let us consider two possible approaches to estimating these probabilities. The first, and most obvious, is to set p(y = i) to the expected value of the binary label y = i over the source domain dataset. A second, slightly less obvious approach is to set it to the expected value of p(y = i|x) over the source domain dataset, formally denoted as

E x∼p(x) [p(y = i|x)]. If p(y = i|x)

were unbiased, we would anticipate that the two approaches agree. However, depending on the calibration of p(y = i|x), this may not be the case, bringing us to:

Lemma 1: In the absence of domain shift and in the limit of sufficient data, the EM algorithm will converge to the original priors p(y = i) if and only if p(y

= i) = E x∼p(x) [p(y = i|x)].

Proof: Note that the EM algorithm will converge when q(s+1) (y = i) = q(s) (y = i). From the M-step, we know that q(s+1) (y

= i) = 1 N N k=1 q(s) (y = i|x k ),

where the examples x k are drawn from the target distribution. Substituting the formula for q(s) (y = i|x k ) from the E-step, we have

q(s+1) (y = i) = 1 N N k=1 q(s) (y=i) p(y=i) p(y=i|x k ) n j=1 q(s) (y=j) p(y=j) p(y=j|x k )

. To prove our lemma, we consider the scenario where q(y = i) = p(y = i) and check whether convergence is attained. If the samples in the target distribution are drawn from the same distribution as the source, then in the limit of sufficient N , the value of q(s+1) (y = 1) will approach E x∼p(x)

1 1 p(y=i|x k ) n j=1 1 1 p(y=j|x k ) = E x∼p(x) p(y = i|x k ). Thus, convergence at p(y = i) will be attained if and only if p(y = i) = E x∼p(x) [p(y = i|x)]

We reason that, in the absence of domain shift, it is desirable that EM converge to the original priors p(y = i). In light of Lemma 1, we set p(y = i) to be the average value of p(y = i|x) over the source-domain validation set (we use the validation set to avoid the effects of overfitting on the training set; this is the same validation set used for calibration). If we instead compute p(y = i) as the average of the binary label in the validation set, we observe very poor (even detrimental) performance with EM when the calibrated probabilities do not have bias correction (Tab. C.1).

3.3. METRICS FOR EVALUATING ADAPTATION TO LABEL SHIFT:

The first metric we consider is the mean squared error in the true weights compared to the estimated weights (Azizzadenesheli et al., 2019) . Let us denote the true target-domain prior as q(y = i) and the true source domain prior as p(y = i). The true class weights are defined such that w i = q(y = i)/p(y = i). Both BBSL and RLLS directly output estimated weights ŵi . For EM, the weights can be obtained by dividing the estimated target-domain priors q(y = i) by the source-domain priors p(y = i) (where the source priors are computed as described in Sec. 3.2). The mean squared error of the weights is then simply (1/N ) i ( ŵiw i ) 2 , where N is the number of classes.

The second metric we consider is the improvement in accuracy of the domain-adapted model predictions relative to using the original model predictions. Given the ratio q(y = i)/p(y = i), the adapted model predictions can be computed as q(y = i|x

k ) = q(y=i) p(y=i) p(y=i|x k ) j q(y=j) p(y=j) p(y=j|x k )

, similar to the E-step of EM. For EM, we use these adapted predictions to assess accuracy. In the case of BBSL, Lipton et al. (2018) recommend retraining the model to obtain adapted predictions. Due to computational constraints, we did not perform model retraining, and thus we limit the comparisons of domain-adapted accuracy only to those calibration techniques that were used in conjunction with EM.

The third metric we consider is the Jenson-Shannon divergence between the true and estimated target-domain priors. This metric is discussed further in Sec. B.

4. RESULTS:



4.1. EXPERIMENTAL SETUP:

We evaluated the efficacy of BBSL and EM coupled to different calibration approaches on MNIST, CIFAR10, CIFAR100, and a diabetic retinopathy detection dataset. For our experiments on MNIST, we used the architecture from Azizzadenesheli et al. (2019) , and for our experiments on CIFAR10 and CIFAR100, we trained ten different models, each with a different random seed, using the code from Geifman and El-Yaniv (2017). For MNIST, CIFAR10 and CIFAR100, 10K examples of the training set were reserved as a held-out validation set. Dirichlet shift was simulated on the testing set by sampling with replacement in accordance with class proportions generated by a dirichlet distribution with uniform α values of 0.1, 1.0 and 10.0 (smaller values of α result in more extreme label shift). Samples from the validation set were used for calibration, EM initialization and BBSL confusion matrix estimation. Accuracy and JS Divergence were reported on the label-shifted testing set, while the calibration metrics of NLL and ECE (with 15 bins) were reported on the unshifted testing set. In addition to exploring different degrees of dirichlet shift, we also investigated how the algorithms behaved when the number of samples used in the validation and testing set were varied. For example, in experiments with n = 8000, only 8000 samples from the validation set and 8000 samples from the shifted testing set were presented to the domain adaptation and calibration algorithms. For each model, for a given α and n, 10 trials were performed, where each trial consisted of a different sampling (without replacement) of the validation set as well as a different sampling of the dirichlet prior and the label-shifted testing set. This resulted in a total of 100 experiments (10 for each of the 10 different models). Statistical significance was calculated using a signed Wilcoxon test with a one-sided p-value threshold of 0.01. For MNIST and CIFAR10, we also explored "tweak one" shift (Lipton et al., 2018) , where the prior of the fourth class was set to a parameter ρ and the remaining class priors were set to (1 -ρ)/9. We explored ρ = 0.01 and ρ = 0.9.

The Kaggle Diabetic Retinopathy dataset Kaggle ( 2015) is a collection of retinal fundus images and an associated "grade" from 0-4, where 0 indicates healthy and 1-4 indicate progressively more severe stages of retinopathy. For our experiments, we used the publicly-available pretrained model from De Fauw ( 2015), but it modified so as to make predictions on only one eye at a time (specifically, we supplied the mirror image of a given eye as the input for the second eye). Because test-set labels are unavailable, we separated the validation set used during the training of the model (consisting of 3514 examples) into "pseudo-validation" and "pseudo-test" sets. Specifically, for each of 100 trials, we sampled n examples from the original validation set without replacement to form a pseudo-validation set, and kept the remaining examples as the pseudo-test set. Calibration was performed on the pseudo-validation set, and calibration metrics of NLL and ECE were reported on the pseudo-test set. Domain shift was then simulated by sampling from the pseudo-test set in such a way that the proportion of "healthy" labels was set to a fraction ρ, and the relative proportions of diseased labels was kept the same as in the source distribution. In the source distribution, ρ = 0.73; for the simulated domain shift, we explored ρ = 0.5 and ρ = 0.9.

4.2. EM WITH APPROPRIATE CALIBRATION ACHIEVES STRONG PERFORMANCE AT ESTIMATING SHIFT WEIGHTS:

We compared the performance of EM, BBSL and RLLS in the presence of different types of calibration, using both MSE and JS Divergence as metrics as described in Sec. 2019)). However, as calibration improves, so does the performance of EM. In particular, the best overall performance is achieved when using the variants of temperature scaling that contain class-specific bias parameters -namely BCTS and VS -in combination with EM.

Shift Estimator:

Calibration Method We also computed the improvement in accuracy achieved by EM with different calibration methods compared to an unadapted baseline (Tables 5, 6 , 7, E.1). Across datasets, observed that either BCTS or VS tended to achieve the best accuracy. To reconcile this with the observation in Guo et al.

α = 0.1 α = 1.0 α =

(2017) that VS did not give the best ECE compared to TS, we calculated the Negative Log Likelihood (NLL) of different calibration methods on an unshifted test set and found that BCTS and VS tended to achieve the best NLL, even when they did not yield the best ECE (Sec. D). Empirically, we found that the NLL corresponds better with the improvement that a calibration method will give to domain adaptation (Sec. I). This is consistent with other reports stating that computing the ECE using only information about the most confidently predicted class, as was done in Guo et al. (2017) , is perhaps not the best metric (Vaicenavicius et al., 2019) . with ∆%accuracy as the metric. Unlike BBSL and RLLS, the EM algorithm does not rely on retraining to produce domain adapted probabilities. Value before the semicolon is the average change in %accuracy relative to a baseline of no adaptation. Value after the semicolon is the average rank compared to other methods in the same column. Bold values in a column are not significantly different from the best performing method in the column, as measured by a paired Wilcoxon test at p ≤ 0.01. Calibration techniques involving class-specific bias parameters (namely BCTS and VS) tend to achieve the best performance.

Calibration Method α = 0.1 α = 1.0 α = 10.0 n=7000 n=8500 n=10000 n=7000 n=8500 n=10000 n=7000 n=8500 n=10000 EM Table 6: CIFAR100: Comparison of calibration methods when using EM adaptation to dirichlet shift, with ∆%accuracy as the metric. Unlike BBSL and RLLS, the EM algorithm does not rely on retraining to produce domain adapted probabilities. Value before the semicolon is the average change in %accuracy relative to a baseline of no adaptation. Value after the semicolon is the average rank compared to other methods in the same column. Bold values in a column are not significantly different from the best performing method in the column, as measured by a paired Wilcoxon test at p ≤ 0.01. Calibration techniques involving class-specific bias parameters (namely BCTS and VS) tend to achieve the best performance.

Calibration Method ρ = 0.5 ρ = 0.9 n=500 n=1000 n=1500 n=500 n=1000 n=1500 

5. DISCUSSION:

In this work, we explored the effect of calibration on procedures designed to perform domain adaptation to label shift. In experiments on CIFAR10, MNIST, CIFAR100 and diabetic retinopathy detection, we found the combination of EM-based domain adaptation with an appropriate calibration approach tends to outperform BBSL and RLLS. In particular, we find that the best results are achieved when the calirbation approach incorporates class-specific bias parameters that can reduce systematic bias in the class probabilities -something that is not true of the popular Temperature Scaling approach recommended by Guo et al. (2017) .

In addition, we observed that when the calibrated probabilities retain systematic bias, domain adaptation to EM is sensitive to the strategy used to compute the source-domain priors. If the sourcedomain priors p(y = i) are not defined in a way that mirrors the systematic bias in the predicted probabilities p(y = i|x), then EM will estimate a label shift even if the target domain is identical to the source domain (Lemma 1) and can produce highly detrimental results (Tables C.1 & C.2). By contrast, if the source domain priors for EM are initialized as recommend in Sec. 3.2, EM becomes substantially more tolerant of systematic bias in the calibrated probabilities, although it does not tend to outperform BBSL or RLLS in the presence of poor calibration.

One concern when using EM is the possibility of getting trapped in local minima. To address this concern, we analyzed the optimization of the likelihood function of EM and determined that it is concave Sec. A. Thus, we can expect that EM will converge to the global maximum of the likelihood. Future work could extend this analysis to derive generalization guarantees for the domain adaptation.

In the meantime, our recommendation based on our empirical results is as follows: if there is insufficient data to calibrate the predicted probabilities, use RLLS for domain adaptation. Otherwise, use a calibration approach that incorporates class-specific bias parameters (either bias-corrected temperature scaling or vector scaling), and use Expectation Maximization for domain adaptation. 

A LIKELIHOOD FUNCTION OF EM OBJECTIVE:

In this section, we show that, in our label shift setting, the likelihood function is unimodal and concave. Thus, the EM approach converges to the maximum likelihood estimation.

Lemma A: the EM objective is concave.

Proof: Let q(w i ) and p(w i ) denote the target and source domain prior probabilities for class i. We wish to find target-domain priors q(w) that maximize the log-likelihood function given by l(X; q(w))

EQUATION

where (2) follows from the label shift assumptions and (3) follows from Bayes' rule. Now note that the maximization is independent of p(x k ) so the optimization problem is equivalently written as follows max q(w) k

log i p(w i |x k ) p(w i ) q(w i ) s.t. 1 T • q(w) = 1 q(w i ) ≥ 0 ∀i (6)

Note that in the objective, both p(w i |x k ) and p(w i ) are constants with respect to q(w). Hence, the objective function is the sum of logs of linear functions in our decision variable and the constraints are affine. Therefore, the maximization problem is concave.

B JENSON-SHANNON DIVERGENCE METRIC FOR EVALUATION ADAPTATION TO LABEL SHIFT:

In this section, we describe how we use the Jenson-Shannon divergence between the true targetdomain priors and the estimated target-domain priors to evaluate a given domain adaptation method.

Let us denote the true target-domain prior as q(y = i) and the estimate as q (y = i). In the case of BBSL and RLLS, q (y = i) can be calculated using the class weights ŵi , which are intended to estimate q(y = i)/p(y = i). Specifically, we use the formula q (y = i) = ŵip (y=i) j ŵj p (y=j) , where p (y = i) is defined as the average of the true class labels in the source domain.

In the case of EM, some nuance is required. In Sec. 3.2, we noted that when performing EM, the source domain priors p(y = i) should be defined as the average of the predictions p(y = i|x) over the examples from the source domain (rather than being defined as the average of the labels). Once the EM algorithm has reached convergence, it will output an estimate q(y = i) of the target domain priors. Naively, it would seem that we could use q(y = i) as our estimate of q(y = i) when computing the Jensen-Shannon divergence. However, if p(y = i) is systematically biased, the target domain priors output by EM would likely also be systematically biased. We found that we can obtain superior estimates of q(y = i) by leveraging the true labels in the source domain as follows: first, we average the labels in the source domain to obtain to obtain p (y = i), which is an unbiased estimate of p(y = i). Then, we use the ratio q(y = i)/p(y = i) from EM as a shift estimate. That is, we compute our estimate of the target domain priors using the formula q (y = i) = q(y=i) p(y=i) p (y=i) j q(y=j) p(y=j) p (y=j) . We find that target domain priors computed in this way can occasionally perform surprisingly well even when the predictions themselves are not very well calibrated. Table E .5: CIFAR10: Comparison of all calibration and domain adaptation methods, using JS Divergence (Sec. B) as the metric ("tweak-one" shift). Value before the semicolon is the average of the metric over all trials. Value after the semicolon is the average rank of the domain adaptation + calibration method combination relative to the other method combinations in the column. Bold values in a column are not significantly different from the best-performing method in the column as measured by a paired Wilcoxon test at p < 0.01. EM with BCTS or VS tends to achieve the best performance. See Sec. 4.1 for details on the experimental setup. 

I NLL CORRESPONDS BETTER TO BENEFITS IN LABEL SHIFT ADAPTATION:

To investigate whether NLL or ECE corresponded better to the benefits offered by a calibration method in the context of label shift adaptation, we adopted the following strategy: in a given experimental run, we identified the calibration method that provided the best NLL (or ECE) on the unshifted test set. We then looked at the performance of label shift adaptation using this calibration method. Note that the calibration method selected can differ from one run to the next. Across datasets, we observed that, by and large, selecting a calibration method according to the NLL produced better performance after domain adaptation as compared to selecting a calibration method according to ECE. Results are show in the tables below. 

:



Figures and Tables:

Figure 1: Temperature Scaling exhibits systematic bias. On CIFAR10 data, systematic bias was quantified by the Jensen-Shannon divergence between the true class label proportions and the average class predictions on a held-out test set drawn from the same distribution as the dataset used for calibration. TS: Temperature Scaling, NBVS: No-Bias Vector Scaling, BCTS: Bias-Corrected Temperature Scaling, VS: Vector Scaling. BCTS and VS had significantly lower systematic bias compared to TS and NBVS. Results are averaged over multiple models and dataset samples (Sec. 4.1).

3.3. Results for MSE are in Tables 1, 2, 3, 4, E.8 & F.3 , and results for JS Divergence are in Tables E.6, E.7 G.3 & H.3. Across all datasets, we observed the following general trends: first, in the absence of calibration, BBSL and RLLS tend to outperform EM, with RLLS tending to perform the best (consistent with the results in Azizzadenesheli et al. (

Table content: <table/>

CIFAR10: Comparison of EM, BBSL and RLLS (dirichlet shift). Value before the semicolon is the average MSE in the estimated shift weights (as defined in Sec. 3.3). Value after the semicolon is the average rank of a method relative to the others in the group that use the same calibration. α represents the dirichlet shift parameter (larger α corresponds to less extreme shift), n represents the sample size for both the validation set and the label-shifted test set. A bold value in a group is not significantly different from the best-performing method in the group, as measured by a paired Wilcoxon test at p < 0.01. See TableE.2 for an equivalent table but with statistical comparisons done across all calibration methods. EM tends to outperform BBSL and RLLS when calibration techniques involving class-specific bias parameters are used.

Table content: <table><tr><td>10</td></tr></table>

MNIST: Comparison of EM, BBSL and RLLS ("tweak-one" shift). Value before the semicolon is the average MSE in the estimated shift weights. Value after semicolon is the average rank of a method relative to others in the group that use the same calibration. A bold value in a group is not significantly different from the best-performing method in the group, as measured by a paired Wilcoxon test at p < 0.01. See TableF.2 for an equivalent table but with statistical comparisons done across all calibration methods. EM tends to outperform BBSL and RLLS when calibration techniques involving class-specific bias parameters are used.

Table content: <table/>

CIFAR100: Comparison of EM, BBSL and RLLS (dirichlet shift). Value before the semicolon is the avg. MSE in the estimated shift weights. Value after the semicolon is the avg. rank of a method relative to the others in the group that use the same calibration. A bold value in a group is not significantly different from the best-performing method in the group, as measured by a paired Wilcoxon test at p < 0.01. See TableG.1 for an equivalent table but with statistical comparisons done across all calibration methods. EM tends to outperform BBSL and RLLS when calibration techniques involving class-specific bias parameters are used.

Table content: <table/>

Kaggle Diabetic Retinopathy: Comparison of EM, BBSL and RLLS. ρ represents proportion of healthy examples in shifted domain; source domain has ρ = 0.73. Value before semicolon is the average MSE in the estimated shift weights. Value after the semicolon is the average rank of a method relative to others in the group that use the same calibration. A bold value in a group is not significantly different from the best-performing method in the group (paired Wilcoxon test at p < 0.01). See TableH.1 for an equivalent table but with statistical comparisons done across all calibration methods. EM tends to outperform BBSL and RLLS when calibration techniques involving class-specific bias parameters are used.

Table content: <table><tr><td>Shift</td><td>Calibration</td><td/><td>α = 0.1</td><td/><td/><td>α = 1.0</td><td/><td/><td>α = 10</td><td/></tr><tr><td>Estimator</td><td>Method</td><td>n=2000</td><td>n=4000</td><td>n=8000</td><td>n=2000</td><td>n=4000</td><td>n=8000</td><td>n=2000</td><td>n=4000</td><td>n=8000</td></tr><tr><td>EM</td><td>None</td><td colspan="6">6.986; 2.77 6.926; 3.17 6.938; 3.31 1.968; 3.36 2.016; 3.44 2.055; 3.69</td><td>0.25; 2.79</td><td colspan="2">0.217; 3.38 0.263; 3.42</td></tr><tr><td>EM</td><td>TS</td><td>7.251; 1.68</td><td>7.2; 2.13</td><td colspan="4">7.217; 2.21 2.127; 2.83 2.172; 2.92 2.204; 3.05</td><td>0.243; 3.1</td><td colspan="2">0.225; 3.34 0.276; 3.37</td></tr><tr><td>EM</td><td>NBVS</td><td colspan="3">7.324; 1.63 7.314; 1.59 7.314; 1.69</td><td>2.5; 1.46</td><td colspan="2">2.592; 1.47 2.631; 1.45</td><td>0.706; 1.4</td><td>0.788; 1.31</td><td>0.84; 1.3</td></tr><tr><td>EM</td><td>BCTS</td><td colspan="9">7.328; 1.69 7.337; 1.42 7.347; 1.4 2.593; 0.98 2.664; 1.0 2.688; 1.09 0.764; 1.24 0.839; 0.94 0.884; 0.93</td></tr><tr><td>EM</td><td>VS</td><td colspan="9">7.255; 2.23 7.331; 1.69 7.372; 1.39 2.548; 1.37 2.652; 1.17 2.724; 0.72 0.741; 1.47 0.838; 1.03 0.889; 0.98</td></tr></table>

CIFAR10: Comparison of calibration methods when using EM adaptation to dirichlet shift,

Table content: <table/>

Kaggle Diabetic Retinopathy: Comparison of calibration methods when using EM adaptation to domain shift, with ∆%accuracy as the metric. ρ represents proportion of healthy examples in shifted domain; source distribution has ρ = 0.73. Unlike BBSL and RLLS, the EM algorithm does not rely on retraining to produce domain adapted probabilities. Value before the semicolon is the average change in %accuracy relative to a baseline of no adaptation. Value after the semicolon is the average rank compared to other methods in the same column. Bold values in a column are not significantly different from the best performing method in the column, as measured by a paired Wilcoxon test at p ≤ 0.01. Calibration techniques involving class-specific bias parameters (namely BCTS and VS) tend to achieve the best performance.

Table content: <table><tr><td>EM</td><td>None</td><td colspan="4">1.926; 3.09 2.076; 3.49 2.196; 3.64 1.296; 3.42 1.375; 3.81</td><td>1.477; 3.8</td></tr><tr><td>EM</td><td>TS</td><td colspan="4">1.902; 2.96 2.225; 3.17 2.495; 3.13 1.626; 3.01 1.923; 2.88 1.973; 2.97</td></tr><tr><td>EM</td><td>NBVS</td><td>3.23; 1.69</td><td colspan="3">3.789; 1.49 4.062; 1.54 2.074; 2.44 2.266; 2.24 2.405; 2.17</td></tr><tr><td>EM</td><td>BCTS</td><td colspan="4">3.766; 0.88 4.356; 0.74 4.58; 0.82 3.548; 0.35 3.567; 0.36 3.722; 0.44</td></tr><tr><td>EM</td><td>VS</td><td colspan="2">3.67; 1.38 4.278; 1.11 4.545; 0.87</td><td>3.5; 0.78</td><td>3.57; 0.71 3.746; 0.62</td></tr></table>

Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699. ACM, 2002. Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 819-827, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL http://proceedings.mlr.press/v28/zhang13d.html.

Table content: <table/>

.16 0.054; 1.14 0.046; 1.18 0.026; 1.36 0.018; 1.46 0.014; 1.32 0.016; 0.94 0.011; 1.11 0.008; 1.14 Table E.6: CIFAR10: Comparison of EM, BBSL and RLLS (dirichlet shift) using JS Divergence as the metric. Analogous to Table 1, but with JS Divergence as the metric rather than MSE. Table H.3: Kaggle Diabetic Retinopathy Detection: Comparison of EM, BBSL and RLLS. Analogous to Table 4, but with JS Divergence (Sec. B) as the metric rather than MSE.

Table content: <table><tr><td>Shift</td><td>Calibration</td><td/><td>α = 0.1</td><td/><td/><td>α = 1.0</td><td/><td/><td>α = 10</td><td/></tr><tr><td>Estimator</td><td>Method</td><td>n=2000</td><td>n=4000</td><td>n=8000</td><td>n=2000</td><td>n=4000</td><td>n=8000</td><td>n=2000</td><td>n=4000</td><td>n=8000</td></tr><tr><td>EM</td><td>None</td><td colspan="8">0.035; 0.28 0.033; 0.42 0.033; 0.8 0.021; 1.28 0.018; 1.76 0.016; 2.28 0.016; 1.44 0.011; 1.8</td><td>0.009; 2.2</td></tr><tr><td>BBSL-hard</td><td>None</td><td colspan="3">0.076; 3.36 0.064; 3.32 0.055; 3.13</td><td>0.03; 2.65</td><td>0.023; 2.61</td><td>0.016; 2.6</td><td colspan="3">0.019; 2.86 0.013; 2.52 0.009; 2.27</td></tr><tr><td>BBSL-soft</td><td>None</td><td colspan="9">0.068; 2.26 0.056; 2.23 0.049; 2.22 0.027; 1.67 0.019; 1.46 0.014; 1.2 0.016; 1.62 0.011; 1.58 0.008; 1.42</td></tr><tr><td>RLLS-hard</td><td>None</td><td>0.074; 2.6</td><td colspan="2">0.063; 2.49 0.054; 2.37</td><td>0.03; 2.69</td><td colspan="5">0.023; 2.59 0.016; 2.62 0.019; 2.66 0.013; 2.58 0.009; 2.47</td></tr><tr><td>RLLS-soft</td><td>None</td><td>0.066; 1.5</td><td colspan="8">0.055; 1.54 0.048; 1.48 0.027; 1.71 0.019; 1.58 0.014; 1.3 0.016; 1.42 0.011; 1.52 0.008; 1.64</td></tr><tr><td>EM</td><td>TS</td><td colspan="9">0.024; 0.04 0.022; 0.09 0.021; 0.14 0.019; 0.58 0.017; 0.94 0.014; 1.18 0.016; 1.18 0.012; 1.22 0.009; 1.26</td></tr><tr><td>BBSL-soft</td><td>TS</td><td colspan="6">0.067; 1.81 0.056; 1.78 0.048; 1.71 0.026; 1.21 0.019; 0.94 0.013; 0.84</td><td>0.016; 1.0</td><td colspan="2">0.011; 0.93 0.008; 0.77</td></tr><tr><td>RLLS-soft</td><td>TS</td><td colspan="9">0.066; 1.15 0.055; 1.13 0.048; 1.15 0.026; 1.21 0.019; 1.12 0.013; 0.98 0.016; 0.82 0.011; 0.85 0.008; 0.97</td></tr><tr><td>EM</td><td>NBVS</td><td colspan="9">0.021; 0.02 0.017; 0.04 0.017; 0.05 0.017; 0.3 0.013; 0.38 0.01; 0.46 0.015; 0.88 0.01; 0.82 0.008; 0.96</td></tr><tr><td>BBSL-soft</td><td>NBVS</td><td>0.067; 1.84</td><td>0.055; 1.8</td><td>0.048; 1.8</td><td colspan="6">0.026; 1.32 0.018; 1.21 0.013; 1.22 0.016; 1.15 0.011; 1.14 0.008; 0.97</td></tr><tr><td>RLLS-soft</td><td>NBVS</td><td colspan="9">0.066; 1.14 0.054; 1.16 0.047; 1.15 0.026; 1.38 0.018; 1.41 0.013; 1.32 0.016; 0.97 0.011; 1.04 0.008; 1.07</td></tr><tr><td>EM</td><td>BCTS</td><td colspan="9">0.02; 0.02 0.016; 0.02 0.015; 0.08 0.016; 0.22 0.012; 0.34 0.009; 0.36 0.015; 0.84 0.01; 0.76 0.008; 0.86</td></tr><tr><td>BBSL-soft</td><td>BCTS</td><td colspan="3">0.066; 1.82 0.055; 1.81 0.047; 1.75</td><td>0.025; 1.4</td><td colspan="5">0.018; 1.22 0.014; 1.29 0.016; 1.18 0.011; 1.14 0.008; 1.01</td></tr><tr><td>RLLS-soft</td><td>BCTS</td><td colspan="8">0.065; 1.16 0.054; 1.17 0.046; 1.17 0.025; 1.38 0.018; 1.44 0.014; 1.35 0.016; 0.98 0.011; 1.1</td><td>0.008; 1.13</td></tr><tr><td>EM</td><td>VS</td><td colspan="9">0.023; 0.02 0.017; 0.0 0.015; 0.06 0.016; 0.3 0.013; 0.28 0.009; 0.44 0.015; 0.88 0.01; 0.72 0.008; 0.86</td></tr><tr><td>BBSL-soft</td><td>VS</td><td colspan="8">0.066; 1.82 0.055; 1.86 0.047; 1.76 0.025; 1.34 0.018; 1.26 0.014; 1.24 0.016; 1.18 0.011; 1.17</td><td>0.008; 1.0</td></tr><tr><td>RLLS-soft</td><td>VS</td><td>0.065; 1</td><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

1: CIFAR10: NLL vs ECE, ∆%Accuracy, dirichlet shift. Entry in "calibration method" column indicates how the calibration method for any given run was selected: either according to whether it produced the best NLL or whether it produced the best ECE, where NLL and ECE were calculated on the unshifted test set. Value before the semicolon is the average change in %accuracy relative to unadapted predictions. Value after the semicolon is the average rank of the given metric relative to the other metric in the pair. A bold value is significantly better than the non-bold value in the pair using a paired Wilcoxon test at p ≤ 0.01. See Sec. 4.1 for details on the experimental setup. CIFAR10: NLL vs. ECE, metric: ∆%accuracy, "tweak-one" shift. Analogous to TableI.1. The "tweak-one" shift strategy is explained in Sec. 4.1.

Table content: <table><tr><td>Shift</td><td>Calibration</td><td/><td>α = 0.1</td><td/><td/><td/><td>α = 1.0</td><td/><td/><td>α = 10</td></tr><tr><td>Estimator</td><td>Method</td><td>n=2000</td><td>n=4000</td><td>n=8000</td><td colspan="2">n=2000</td><td>n=4000</td><td>n=8000</td><td>n=2000</td><td>n=4000</td><td>n=8000</td></tr><tr><td>EM</td><td>Best NLL</td><td colspan="10">7.332; 0.3 7.326; 0.32 7.37; 0.28 2.593; 0.36 2.664; 0.09 2.688; 0.06 0.764; 0.42 0.839; 0.04 0.884; 0.03</td></tr><tr><td>EM</td><td>Best ECE</td><td colspan="10">7.298; 0.7 7.302; 0.68 7.318; 0.72 2.548; 0.64 2.172; 0.91 2.204; 0.94 0.741; 0.58 0.225; 0.96 0.276; 0.97</td></tr><tr><td>Shift</td><td colspan="2">Calibration</td><td/><td>ρ = 0.01</td><td/><td/><td/><td/><td colspan="2">ρ = 0.9</td></tr><tr><td>Estimator</td><td colspan="2">Method</td><td>n=2000</td><td>n=4000</td><td/><td colspan="2">n=8000</td><td>n=2000</td><td colspan="2">n=4000</td><td>n=8000</td></tr><tr><td>EM</td><td colspan="2">Best NLL</td><td colspan="9">1.192; 0.17 1.253; 0.21 1.301; 0.15 17.724; 0.47 17.779; 0.08 17.84; 0.07</td></tr><tr><td>EM</td><td colspan="2">Best ECE</td><td colspan="3">1.053; 0.83 1.149; 0.79</td><td colspan="6">1.16; 0.85 17.727; 0.53 17.26; 0.92 17.288; 0.93</td></tr><tr><td>Table I.2:</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
This work conducted exhaustive experiments for a label shift problem with EM and BBSE over CIFAR10/100 and retinopathy detection datasets. In addition, as for model calibration, they also considered temperature and vector scaling and introduced intermediaries between those, namely, NBVS and TBVS. Overall, the paper is well organized and provides rigorous experiments with conclusions based on the empirical observations. As stated in the paper, the main contribution of this work is to explore the impact of calibration. However, it is still doubtful whether those empirical results can be generalized as there is no analytical and thoughtful discussions. Further, the label shift was simulated by means of dirichlet shift on both datasets. It would be great to apply the method on real cases.