Adapting to Label Shift with Bias-Corrected Calibration
Authors: Avanti Shrikumar
Amr M. Alexandari
Anshul Kundaje
Venue:  (2020)
Keywords: calibration
label shift
domain adaptation
temperature scaling
em
bbse

Abstract: Label shift refers to the phenomenon where the marginal probability p(y) of observing a particular class changes between the training and test distributions, while the conditional probability p(x|y) stays fixed. This is relevant in settings such as medical diagnosis, where a classifier trained to predict disease based on observed symptoms may need to be adapted to a different distribution where the baseline frequency of the disease is higher. Given estimates of p(y|x) from a predictive model, one can apply domain adaptation procedures including Expectation Maximization (EM) and Black-Box Shift Estimation (BBSE) to efficiently correct for the difference in class proportions between the training and test distributions. Unfortunately, modern neural networks typically fail to produce well-calibrated estimates of p(y|x), reducing the effectiveness of these approaches. In recent years, Temperature Scaling has emerged as an efficient approach to combat miscalibration. However, the effectiveness of Temperature Scaling in the context of adaptation to label shift has not been explored. In this work, we study the impact of various calibration approaches on shift estimates produced by EM or BBSE. In experiments with image classification and diabetic retinopathy detection, we find that calibration consistently tends to improve shift estimation. In particular, calibration approaches that include class-specific bias parameters are significantly better than approaches that lack class-specific bias parameters, suggesting that reducing systematic bias in the calibrated probabilities is especially important for domain adaptation.

Under review as a conference paper at ICLR 2020
ADAPTING TO LABEL SHIFT WITH BIAS-CORRECTED CALIBRATION
Anonymous authors Paper under double-blind review
ABSTRACT
Label shift refers the phenomenon where the marginal probability p(y) of ob- serving a particular class changes between the training and test distributions, while the conditional probability p(æ y)) stays fixed. This is relevant settings such as medical diagnosis, where classifier trained to predict disease based on observed symptoms may need to be adapted
different distribution where the baseline fre- quency of the dísease is higher. Given estimates of p(y (z) from a predictive model, Saerens et al. (2002) proposed an Expectation Maximization (EM) algorithm to efficiently correct for the difference in class proportions between training and test distributions. A limiting assumption of this EM algorithm is that the probabilities p(y (z) are calibrated, which is not true of modern neural networks. Recently, Black Box Shift Learning (BBSL) (Lipton et al. 2018) and Regularized Learning under Label Shifts (RLLS) (Azizzadenesheli et al.] (2019) have emerged as state-of-the-art techniques to cope with label shift when a classifier does not output calibrated probabilities. In this work, we show that while EM in the absence of calibration achieves weak performance with modern neural networks, the combination of EM with appropriate calibration typically outperforms BBSL and RLLS. In particular, we show that the popular calibration strategy of Temperature Scaling (TS) is often insufficient to completely correct miscalibration; rather, EM achieve state-of-the-arr results primarily when used with a calibration approach that includes class-specific bias parameters that TS lacks. We further introduce a theoretically principled strategy for estimating source-domain priors in EM-based domain adaptation that allows the EM algorithm to behave more robustly when the calibrated probabilities contain systematic bias. Colab notebooks reproducing experiments CIFAR100, MNIST and diabetic retinopathy detection are available at (anonymized link): https: ttt.b//ttttuugcooo com/bl
CIFARIO. 10,
abel shi wetexeerieente
INTRODUCTION
Imagine we train a classifier in country A to predict whether or not a person has a disease based on observed symptoms, and that we hope deploy this classifier in country B, which has poorer access to healthcare. If the prevalence of the disease in country B is higher in than in country A, the classifier might systematically misdiagnose people not having the disease. How can we adapt the classifier to cope with the difference in the baseline prevalence of the disease in the two countries?
Formally, let y denote our labels (e.g. whether not a person is diseased), and let denote the observed symptoms. Let us denote the joint distribution (æ, y) in country A (our "source" domain) as I. and let us denote the distribution in country B (our "target" domain, where we do not have labels) as Q. How can we adapt a classifier trained to estimate p(y(æ)) ) (the conditional probability in distribution P) so that it can instead estimate q(y) a=) (the conditional probability in distribution Q)? Absent assumptions about the nature of the shift between IP and Q, this problem is intractable. However, the disease generates similar symptoms in both countries, we can assume that p(z(y) = q(xly), and that the shift in the joint distribution g( (z, y) is due to a shift in the label proportion q(y). Formally, we assume that ((x, y) = p( (y) This is known as label shift or prior probability shift (Amos) (008), and it corresponds to anti-causal learning (i.e. predicting the cause from its effects x) (Schoelkopt et al, 2012). Anti-causal learning is appropriate for diagnosing diseases given observations of symptoms because diseases cause symptoms.

---

Under review as a conference paper at ICLR 2020
Given estimates of p(y) and p(ylaz), Saerens al. (2002) proposed a simple Expectation Maximiza- tion (EM) procedure to estimate ( without needing to estimate p(zly). However, estimates of p(y|z)) (z) derived from modern neural networks
often poorly calibrated (Guo et al. 2017 and the lack of calibration can decrease the effectiveness of EMM As an alternative, Lipton et al. (2018) developed technique called Black Box Shift Learning (BBSL) that can work even when the pre- dictions p(æly) are not calibrated. Azizzadeneshelii et al. (2019) further improved upon BBSL in technique known as Regularized Learning under Label Shifts (RLLS). Both BBSL and RLLS leverage information in confusion matrix calculated on held-out portion of the training set. To our knowledge, neither BBSL and RLLS have been benchmarked against the EM approach.
Although the EM approach is limited by the assumption that the predictions p(y x)) are calibrated, a number of recent techniques have been proposed to correct for miscalibration of p(y a:) using a held-out portion of the training set (Guo et al. (017). The held-out set can be thought of as analogous to the held-out set used BBSL and RLLS to calculate a confusion matrix. This suggests simple yet novel hybrid algorithm for adapting to label shift: first, calibrate predictions using the held-out training set, then perform domain adaptation on the calibrated predictions using EM. this work, studied the effectiveness of this hybrid algorithm. More generally, we studied the impact of calibration on domain adaptation to label shift. Our primary contributions are as follows:
1.1 OUR CONTRIBUTIONS
1. In experiments on CIFARIO, CIFARIO0, MNIST and Diabetic Retinopathy Detection, we found that EM achieves state-of-the -he---rrtt---tttte results when used with an appropriate calibration approach. Although BBSL and RLLS both benefit from calibration, they did not tend to outperform EM when the probabilities
well-calibrated.
2. We observed that the popular calibration approach of Temperature Scaling (TS) (Guo et al. 2017) does not tend to achieve state-of-the art results in the context of adaptation to label shift, possibly owing to large systematic biases in the calibrated probabilities (Fig. [ The best results are obtained using variants of TS that contain class-specific bias capable of correcting for systematic bias.
3. We identify a theoretically-groumeee EM-based domain adaptation that can be critically important when calibrated probability estimates have systematic bias. Empirically, we find that computing source priors this way substantially improves the robustness of EM to the presence of systematic biases in the class probabilities. We also prove that the likelihood function of EM is concave and unimodal; thus, EM converges the maximum likelihood estimate.
strategy for computing the source-domain priors in
2 BACKGROUND
2.1 TEMPERATURE SCALING, VECTOR SCALING AND EXPECTED CALIBRATION ERROR
Calibration has long history in the machine learning literature (DeGroot and Fienberg 1983 Platt
1999 Zadrozny and Elkan; 2002, Niculescu- Mizil and Caruana 2005 Kuleshov and Liang 2015
Naeini et al. [2133] Kuleshov and ( In the context of modern neural networks, Guo et al. (2017) showed that Temperature Scaling, single- parammeerrreterr variant of Platt Scaling (Platt 1999). was effective at reducing miscalibration. Temperature scaling performs calibration by introducing temperature parameter T' to the logit vector of the softmax. Let =(x*) represent a vector of the original softmax logits for example x*, and let yı be random variable representing the label for
class i. With temperature scaling, we have p(u)|=** Σ, e where T is optimized with
respect to the Negative Log Likelihood (NL.)) on held-out portion of the training set, such as the validation set. Guo et al. (2017) compared TS to an approach defined as Vector Scaling (VS), where a different scaling parameter was used for each class along with class-specfic bias parameters.
Formally, in vector scaling, p(y (x*)= = c(st= ()3))
Σ, E,e 1,544) Guo et al. 2017) found that vector scaling
had tendency to perform slightly worse than TS measured by metric known the Expected Calibration Error (Nacini et al. 2015). To compute the ECE, the predicted probabilities for the output class partitioned into M equally spaced bins, and the weighted average of the difference between
2

---

Under review as a conference paper at ICLR 2020
the bin's 's accuracy and the bin's confidence is computed, where the weights are determined by the n
proportion of examples falling in the bin. Formally, ECE Em=1 [Bml |ac((mm) - conf(Bm)). where n is the number of samples.
2.2 LABEL SHIFT ADAPTATION VIA EXPECTATION MAXIMIZATION
In seminal paper on label shift adaptation, Saerens et al. (2002 proposed an EM algorithm for estimating the shift the class priors be- tween the training and test distributions. Let
JS Divergence
0.03
(v") = 2) denote the estimate (from EM it- eration s) of the prior probability q(y = i) of observing class i in the test set. The algorithm
0.02
proceeds as follows: first, g())
a) is ini- tialized to be equal the class priors p(y = i)
0.01
estimated from the training set. Then, the con-
0.00
ditional probabilities the E-step are computed
TS NBVS BCTS VS
((m)))
PINEL p(y=ilmx)
as g( = i|<x))
_=
(v-()=))) (y=j)mm)
Figure 1: Temperature Scaling exhibits system- atic bias. On CIFAR 10 data, systematic bias was quantified by the Sansen-Shannon divergence be- tween the true class label proportions and the aver- age class predictions on a held-out test set drawn from the same distribution as the dataset used for calibration. TS: Temperature Scaling, NBVS: No- Bias Vector Scaling, BCTS: Bias-Corrected Tem- perature Scaling, VS: Vector Scaling, BCTS and VS had significantly lower systematic bias com- pared to TS and NBVS. Results are averaged over multiple models and dataset samples (Sec. 4.1
Finally, the prior estimates in the M-step are updated as g(=+1)(y i) N Ex ( =
i f/æx)) where N is the number of examples in the testing set. The E and M steps are iterated until convergence. As there is no need to esti- mate p(æ]y) in any step of the EM procedure, the algorithm can scale to high-dimensional datasets. Note this procedure assumes the con- ditional probability estimates p(y = i|ær)) are calibrated.
2.3 LABEL SHIFT ADAPTATION VIA BLACK Box SHIFT LEARNING AND REGULARIZED LEARNING UNDER LABEL SHIFTS
Following the EM approach of Saerens et al. 2002).
several additional approaches for labels shift Schoelkopf et al., 2012) Zhang et al., 2013) al. 2019 Many of these approaches build estimates
adaptation have emerged (Chan and Ng. Storkey
Lipton al. 2018 Azizzadenesheli
p(zly). which can scale poorly with dataset sizes and underperform on high-dimensional data (Lipton al. 2018). Lipton et al.] (2018) proposed Black-Box Shift Learning (BBSL), which strives to efficiently estimate the weights [w], p(ymi)
g(v=() even in cases where the prediction model
ply = illxx) is poorly calibrated or biased. BBSL. proceeds as follows: let f be function that accepts an input and returns the model's predicted class, Wy denote an example from a held-out portion of the training set, and æ% denote example from the testing set. The empirical estimate of w, denoted as よ. is computed as v = "gg where [ūg]] = Σ. (f(m)))) and
(Cg.yly 1 E. ((f(x)) = i and yı = j}. Because the approach above is not guaranteed to produce positive values for all elements of ú, any negative elements of v are set to 0 after they are estimated. Domain adaptation then performed by retraining the model on the entire training distribution with examples upweighted accordance with t. Lipton et al. (2018) denote the version of BBSL described above as BBSL-hard. They also compare to a variant that they call BBSL-soft, which they describe as the case where where ff outputs probabilities rather than hard
classes. We interpreted this to mean [ûg]: Σ. f(m(); and [Cg. = 1 Σx f(xx) {2k j)- Azizzadeneshelii et al./ (2019) further improved upon BBSL by including regularization terms in technique known as Regularized Learning under Label Shift (RLLS). In experiments, compare to BBSL-hard, BBSL-soft, RLLS-har and RLLS-son soft. Regularization parameters for RLLS were set in accordance with the hard-coded values present in the publicly available code provided by the authors at httpp/////gttaa...co////aaaggaga htt.:///222//aaaeeseeee.s////e/e
com

---

Under review as a conference paper at ICLR 2020
Note that BBSL and RLLS both require portion of the training set to be held out during the initial training phase in order to accurately estimate the confusion matrix Čg.yi in experiments involving calibration, we use this same heldout set to calibrate the model.
3 METHODS
3.1 NO-BIAS VECTOR SCALING AND HLAS-CORRECTER TEMPERATURE SCALING
As shown in Fig. I we often found that TS alone resulted in systematically biased estimates of p(us|="), while VS, generalization of TS that contains both class-specific bias terms and class- specific scaling terms, did not exhibit as much systematic bias. Intrigued by this observation, we investigated the performance of two intermediaries between Temperature Scaling and Vector Scaling, The first, which we refer to as No Bias Vector Scaling (NBVS), is equivalent to vector scaling but with all the class-specific bias parameters fixed at zero. The second, which we refer to as Bias-Corrected Temperature Scaling, is equivalent TS Scaling but with the addition of the stass-specific bias terms from VS. As with TS and VS. the parameters are optimized to minimize the NLL on the validation set. Note that in the case of binary classification, the parameterization of BCTS reduces to Platt Scaling (Plat 1999). Thus, BCTS can be viewed as a multi-class generalization of Platt scaling.
3.2 DEFINING SOURCE-DOMAIN PRIORS IN THE EM ALGORITHM
The EM algorithm of (Saerens et al. 2002) requires the user to provide estimates of the source- domain prior class probabilities ply = i). Let us consider two possible approaches to estimating these probabilities. The first, and most obvious, is to set p(y = i) to the expected value of the binary label y = i over the source domain dataset. A second, slightly less obvious approach is to set it to the expected value of ply a|ır) over the source domain dataset, formally denoted as Ex=p((z)) (p = )] If p(y = (|r) were unbiased, we would anticipate that the two approaches agree. However, depending on the calibration of ply = ilaz), this may not be the case, bringing us to:
Lemma 1:
In the absence of domain shift and in the limit of sufficient data, the EM algorithm will converge to the original priors p1y = i) if and only if ply = i) = E E = i|æ)]]-
Proof: Note that the EM algorithm will converge when g(=+1) = i) = ql [y = i). From the ilæx), where the examples Hy drawn
M-step, we know that ((++((y i) y/4 from the target distribution. Substituting the formula for g(*) i| from the E-step, we have g(+)(y=1) p(y=i|æx)
g(s+t) (y = 1) = = E
To prove our lemma, we consider the scenario
Σÿ=1 p(u-y) p(y=)læz)
where qyy -) ply i) and check whether convergence is attained. If the samples in the target distribution are drawn from the same distribution as the source, then in the limit of sufficient N, +f(y=i(mu)
the value of g(a+1)(y (y 1) will approach Ez<p(=) Σラ= +R(== Ex-p(e)d(y = ilæx). Thus, convergence f(y i) will be attained if and only if p(y
Em-p(a) jiv i|())
We reason that, in the absence of domain shift, it desirable that EM converge to the original priors p(y i). In light of Lemma 1, we set p(y i) to be the average value of ş(y i|æ) over the source-domain validation set (we use the validation set to avoid the effects of overfitting on the training set; this is the same validation set used for calibration). If we instead compute p = i) the average of the binary label in the validation set, we observe very poor (even detrimental) performance with EM when the calibrated probabilities do not have bias correction (Tab. C.I
3.3 METRICS FOR EVALUATING ADAPTATION TO LABEL SHIFT
The first metric we consider is the mean squared error in the true weights compared to the estimated weights (Azizzadenesheli et al. 2019). Let us denote the true target-domain prior as g = i) and the true source domain prior ply i). The true class weights are defined such that w, q(v = f)/p(y = i). Both BBSL and RLLS directly output estimated weights wū For EMM the weights can be obtained by dividing the estimated target- domai- priors g(y i) by the source-domain priors ply = i) (where the source priors are computed as described in Sec. 3.2). The mean squared error of the weights is then simply (1/N) E. w where N is the number of classes.
4

---

Under review as a conference paper at ICLR 2020
The second metric we consider is the improvement in accuracy of the domain-adapted model predictions relative to using the original model predictions. Given the ratio q(y = i)/p(y = i).
(1y=d) p(y=i(z))
similar to the E-step of EM. For EM, we use these adapted predictions to assess accuracy. In the case of BBSL,Lipton et al. (2018) recommend retraining the model to obtain adapted predictions. Due to computational constraints, we did not perform model retraining, and thus we limit the comparisons of comain-adapted accuracy only to those calibration techniques that were used in conjunction with EM.
the adapted model predictions can be computed as á(y x = ΣS ددحاا (y=)|(=))
The third metric we consider is the
divergence between the true and estimated target-domain priors. This metric is discussed further in Sec. B
4 RESULTS
4.1 EXPERIMENTAL SETUP
We evaluated the efficacy of BBSL and EM coupled to different calibration approaches on MNIST, CIFARIO, CIFAR 100, and diabetic retinopathy detection dataset. For our experiments on MNIST, we used the architecture from Azizzadenesheli et al. (2019), and for our experiments on CIFARIO and CIFAR 100, we trained ten different models, each with a different random seed, using the code from Geifman and HI- Yaniv (2017). For MNIST, CIFAR10 and CIFAR 100, 10K examples of the training set were reserved
held-out validation set. Dirichlet shift was simulated set by sampling with replacement in accordance with class proportions generated by a dirichlet distribution with uniform a values of 0.1, 1.0 and 10.0 (smaller values of a result in more extreme label shift). Samples from the validation set were used for calibration, EM initialization and BBSL confusion matrix estimation. Accuracy and IS Divergence were reported on the label-shifted testing set, while the calibration metrics of NLL and ECE (with 15 bins) were reported on the unshifted testing set. In addition to exploring different degrees of dirichlet shift, we also investigated how the algorithms behaved when the number of samples used in the validation and testing set were varied. For example, in experiments with n 8000, only 8000 samples from the validation set and 8000 samples from the shifted testing set were presented to the domain adaptation and calibration algorithms. For each model, for a given a and n, 10 trials were performed, where each trial consisted of different sampling (without replacement) of the validation set as well as a different sampling of the dirichlet prior and the label-shifted testing set. This resulted total of 100 experiments (10 for each of the 10 different models). Statistical significance was calculated using signed Wilcoxon test with a one-sided p-value threshold of 0.01. For MNIST and CIFARIO, 10, we also explored "tweak one" shift (Lipton et al. 2018), where the prior of the fourth class set to a parameter p and the remaining class priors were set to (1 p)/9. We explored p = 0.01 and p = 0.9.
the testing
The Kaggle Diabetic Retinopathy dataset dataset|Kaggle (015) is a collection of retinal fundus images and an associated "grade" from 0-4, where 0 indicates healthy and 1-4 indicate progressively more severe stages of retinopathy. For our experiments, we used the publicly-availabl De Fauw (2015). but it modified so as to make predictions on only one eye at a time (specifically, we supplied the mirror image of given eye the input for the second eye). Because test-set labels unavailable, we separated the validation set used during the training of the model (consisting of 3514 examples) into "pseudo- validation" " and "pseudo-test" sets. Specifically, for each of 100 trials, we sampled examples from the original validation set without replacement to form ppeudo-validatio set, and kept the remaining examples as the pseudo-test set. Calibration was performed on the valudo-validation set, and calibration metrics of NLL and ECE were reported on the pseudo-test set. Domain shift was then simulated by sampling from the pseudo-test set in such a way that the proportion of "healthy" labels was set to a fraction p. and the relative proportions of diseased labels was kept the same as in the source distribution. In the source distribution, = 0.73; for the simulated domain shift, we explored p 0.5 and p = 0.9.
pretrained model from
"pseudo-validation"
4.2
EM WITH APPROPRIATE CALIBRATION ACHIEVES STRONG PERFORMANCE AT ESTIMATING SHIFT WEIGHTS
We compared the performance of EM, BBSL and RLLS in the presence of different types of calibration, using both MSE and JS Divergence metries as described in Sec. 3.3 Results for MSE
5

---

Under review as a conference paper at ICLR 2020
in Tables
E.8 & and results for JS Divergence are in Tables E.6 E.7
H.3 Across all datasets, we observed the following general trends: first, in the absence of calibrttion, BBSL and RLLS tend to outperform EM, with RLLS tending to perform the best (consistent with the results in Azizzadenesheli et al. (2019). However, as calibration improves, so does the performance of F.M. In particular, the best overall performance is achieved when using the variants of temperature scaling that contain class-specific bias parameters namely BCTS and VS in combination with EM.
Shift Batinndor
Calthration Mechod
1.0
To
km2000
500000
n=2000
1=40000
n=5000
n=2000
n=4000
n=6000
EXI
None
0,02795 2.9
00004800 3.35
3:4
00005 22; 2.22
0U80092
0100155
3.25
0000222 1.7
CU0CT12: 1.55
0500006 23
BR81--Red
Noes:
0:0961: 2.46
0.0066::22 2.2
B100222
2.0
0.00S5:
0.00200;
2.36
0L00105: 2.09
0.00285;
2-0
0L0R164;
2.66
BL0GOW7:
2.31
BBSL sott
Nore
0 1,31
0JB386G122 1.25
0.80193:
1,38
0,NDSN 1.41
0,00159: 1,05
0,00075: 0:97
0.00212:
1.48
0.00104)
1.54
BL00R54; 1,39
RLL.S-loed
0,0895:
0.006: 1.99
0:00221:
1.85
0.003522233
0.00200; 3.25
0.00105;
2.36
0.00285
2.7
0.00144:2.5
0.00067: 2.35
IUR738:
0ANGSTI
0001159 1,32
0.60078:
BJM0212:
1:22
0JIMNS4; 1.85
EM
15
0.0306; 1.44
0,2825 1.62
0,2303:
1.65
0500733
0,00483; 1.35
CC0057.11 1.7
0,00239; 1.42
0:0012 1.35
0.00006; 1.42
BRSI
JUN9S2;
0J06399; (LA7
0.80197: BAR
0,0291:0.83
00001 0.61
0.00079:0000
0100311::00
0,8066;;88
BLIMKSS; RL09
KLLS soft
0JR735:
0.725770073
0,00196:000
0,66
0,00158;000
0.80
0.00079:
0.55
0,00211: 0.69
0.00305: 0.72
0,0085:: 0,89
EM
NBVS
KURE32 ILS3
8.36
0,R01065
8,88
8.80RNZ;
8.54
CU00 0036
0,80091:8.
CU00SS (L32
RBSI sott vof
NBVS
0X0002: 1.27
0:00592 1.17
010019: 1.15
U
1.26
0000E:::
1.06
0.10077:-000 0:95
0U0E200:: 1.19
BLOGOIA; 1.35
0.80091: 1
0,00719: 1.2
0 1.14
0,00199;110 1,03
000268: 1.35
0.00143;
1.26
0.00277:
0.00207: 0:95
0.00095: 1,07
0,00051; 1.08
EM
ECTS
108138 1LIS
00.875500266
0,00054
1.43
0.006370.36
0,800199
0,48
VUHNIES 0.6
0,80091:00
0,00AR; 8.3
BRSI-
RCTS
0:0816: 1.53
0:0099:
0:0019211
1.34
0600779;
1.34
0000145;
1.14
0U00077:
0:002::
0.00099:
0.0000:: 1.05
R1I.S-uffi
RCTS
0:0017: 1.39
0.0023:
0:0019:
1.24
0.00744; 1.4
0.00145; 1.36
0.00077:
1:36
0,0021-0.06
0.00099: 1.13
0.00052: 1.15
EM
009182: 1094
0.0077.0.21
0,80053; 0.27
0.00661:0228
IL00097.044 0.4
0,80154;
0.52
BJ002: BUR
8,00091; 8.66
0,0005 0L5
BRSI -eett RLLS soft
01001: 1.51
0180N:1 1.39
0.001 80: 1:43
0000779;
0:00146:11
1-2
0L100775:
1/09
0:0021: 1.21
6 0LOGTOA:: 1:31
0 1.00
0,00221:
0.00292
0.00157
0002211 1.43
0,00145 1.4
0,00077;
0,0021: 0.99
0.00095: 1.13
0.00052,
1.14
Table 1: CIFARIO: Comparison of EM, BBSL and RLLS (dirichlet shift). Value before the semicolon is the average MSE in the estimated shift weights (as defined in Sec. (.3) Value after the semicolon is the average rank of a method relative to the others in the group that use the same calibration. represents the dirichlet shift parameter (larger corresponds to less extreme shift), represents the sample size for both the validation set and the label-shifted test set, A bold value in group not significantly different from the best pest-performing method in the group, as measured by a paired Wilcoxon test at p < 0.01. See Table E.2 for an equivalent table but with statistical comparisons done across all calibration methods. EM tends to outperform BBSL and RLLS when calibration techniques involving class-specific bias parameters are used.
Shift Estimator
Calibration Method
= 0.01
0.9
7==0000
7==000
800000
12=2000
R=4000
n==0000
EM
None
0.00219; 2.28
0.001 2.2
0.00072: 2.04
0.00998: 1.72
0.00648: 1.47
0.00528; 1.7
BRSL-hard
None
0.00235; 2.89
0,00123;
0,00083:
0,0118333005
0.00796;
3.12
0.00652: 3.48
BBSL-soft
None
0,00186; 1.67
0.00099; 1.63
0,00063; 1.53
0.00926: 1.69
0.00488;
1.44
0.00336; 0.84
RLLS-hard
None
0.00235, 2.15
0,00123;
2.19
0,00083;
2.39
0.01099;
2.21
0.0076;
0.00633:
2.78
RLLS soft
None
0,00186; 1.01
0.000099; 0.95
0.00063; 0.67
0,00875;
1.35
0.00478:
1.43
0.04:335; 1.2
EM
TS
0.00183; 1.04
0,000991;
0.92
0,00058;
0.78
0,0062;
0.65
0,00325;
0,52
0,00199; 0,42
BBSL-soft
TS
0.00178; 1.35
0,00093;
1.36
0.0006,
1.55
0.00914;
1.35
0.00515:
1.25
0.00384: 1.17
RLLS soft
TS
0,00178; 0.61
0.00093;; 0.72
0,0006; 0.67
0. 0.00863; 1.0
0.00505:
1,23
0.00333: 1.41
EM
NBVS
0,00177; 0.7
0,000885;
0.62
0.00056;
0,44
0.00181:
0.08
0,00088;
0.1
0,000444;
0.0
BBSL-soft
NBVS
0.00184; 1.46
0.00096;
1.5
0.00062;
1.72
0. 0.00887;
1.63
0.00509:
1.43
0.0038: 1.38
RL.SS soft
NBVS
0.00184; 0.84
0,00096; 0.88
0,00062;
0.0084; 1,29
0.00499: 1.47
0.00379: 1.62
EM
BCTS
0,00173; 0.82
0.00087;
0.72
0,00056;
0.48
0.0007: 0.0
0,000433;
0.02
0.00003; 0.0
BBSL-soft
BCTS
0.0018: 1.42
0,00094;
1.46
0.00061:
1.7
0.00879;
1.65
0.00506:
1.45
0.00373: 1.31
RLISS soft
BCTS
0,0018: 0.76
0.00094;
0.82
0,00061:
0.82
0.00832: 1.35
0.00497:
1.53
0 1.69
FM
VS
0.IIT177 0.76
0L0MO077 0.56
0.000566
0.3
0,00083; 0.0
0.00N199;
0.02
0.IN0A3;; 0.0
BBSL-soft
VS
0.00184; 1.4
0.00096;
1.53
0,00063; 1.77
0.00894;
1.67
0.00526;
1.51
0.00415; 1.36
RLLS-soft
VS
0.00184; 0.84
0.00096; 0.91
0.00063:0.99
0.00843; 1.33
0.00515: 1.47
0.00413: 1.64
Table 2: MNIST: Comparison of EM, BBSL and RLLS (nt shift). Value before the semicolon is the average MSE in the estimated shift weights. Value after semicolon is the average rank of method relative to others in the group that use the same calibration. A bold value in a group is not significantly different from the best-performing method in the group, as measured by a paired Wilcoxon test at < 0.01. See Table [22 for an equivalent table but with statistical comparisons done across all calibration methods. EM tends to outperform BBSL and RLLS when calibration techniques involving class-specific bias parameters are used.
We also computed the improvement in accuracy achieved by EM with different calibration methods compared to an unadapted baselinn (Tables [
Across datasets, observed that either BCTS or VS tended to achieve the best accuracy. To reconcile this with the observation in Guo et al.) (2017) that VS did not give the best ECE compared to TS, we calculated the Negative Log Likelihood (NLL) of different calibration methods on an unshifted test set and found that BCTS and VS tended to achieve the best NLL, even when they did not yield the best ECE (Sec. D). Empirically, we found that the NLL corresponds better with the improvement that a calibration method will give to domain adaptation (Sec. D}. This is consistent with other reports stating that computing the ECE using only information about the most confidently predicted class, as was done in Guo et al. (2017). is perhaps not the best metric (Vaicenavicius et al. 2019)
6
7 E.I.

---

Under review as a conference paper at ICLR 2020
Shift Extimndor
Calibratiin Method
1.0
50-7300
n-10000
EM
Navs:
2.26415 3.01
2.13137:315
200006 321
075130:155
3:55
D.RRE: 1.52
0.6619; 1:59
375
B
0.38455: 3:36
0.36SSR: 3:54
MISL- hord
None
1.7199
1.27283:222
505
295
0
2/96
OZWT 3:06
0.24165:
3.03
0.2161, 2.11
BBSL-sc0
1.3248:
0982211117
033389-111
6:32000
028689
1286831144
1/49
1:64
0.10P96 1.6
0.14857; 1.59
0.73652
061146:044
8-27
0.106S;
083752 1.3
1/11
CO0O 144
01550:
0.14 TAS
BBSL sott
10811: 1.16
1.00
01199030095
0,58352:088 0.84
0-52299.88
0.0866:: R:43
0.39904
0.27776:044
0220440055 0.55
0,154880.5
0.1428
0.72
0,34304
6.1132 675
0.10635:
0L10156; 1.3
0-46347
IL19225, 01
0.15747;
8.1294:
8.5
BCTS
8.245%
0.25185:0.8
1125023:14 1L4
0110106: 0.65
0.13716; 0.54
6,09503; B.W
CUDOESS; 1-2
sott RLLS-uoft
0 001000
0,6399,
0,51168:
044275;
0,09027, 0.92
0.15529;03
0.118-9:
0,09528;
0.08269; 0.56
8.19% 0.34
0.31
0.304%6
0,13988; 0.44
ILEM0: 0.56
8,127 8.76
0L10468;
0.85869;
BURA6:: 1:38
BBSL soft
054791:
0.57666111
0.2366:115
0 1,54
0.151:
0.116:
0.52134;
0,1382;
6.063%6
Table 3: CIFAR100: Comparison of E.M, BBSL and RLLS (dirichlet shift). Value hefore the semicolon is the avg. MSF in the estimated shift weights. Value after the semicolon is the avg. rank of method relative to the others in the group that use the same calibration. A bold value in group is not significantly different from the best- perfforrrniigg method in the group, as measured by a paired Wilcoxon test at p < 0.01. See Table G.1 for an equivalent table but with statistical comparisons done across all calibration methods. EM tends to outperform BBSL and RLLS when calibration techniques involving class-specific bias parameters are used.
Shift Estimator
Calibration Method
P 0.5
p P= 0.9
^=500
==1000
N=1500
n=5000
/= n=1000
1=1500
EM
None
1,258; 1,03
0.53; 0.83
0.389; 0.92
0.112; 1.96
0.079:
0.081:
BBSL-hard
Nome
695.531; 2.96
1087.163; 3.14
1.746; 3.1
370,245; 3.25
284,462, 3.18
0,743; 2.77
BBSL- soft
None
12,221; 2.17
1.407; 1.94
0.815; 1.89
1,171; 2.24
0.098; 1.86
0.088; 1.78
hard
None
2.2144; 2.06
1,398; 2.56
1.064; 2.6
0.102; 1.6
0.049; 1.48
0.054; 1.61
RLLS soft
None
1,953; 1.78
0,927: 1.53
0.67: 1.49
0.067; 0.95
0.041; 1.07
0.039; 1.09
EM
TS
1.14;
0.465; 0.57
0.334; 0.52
0.11; 1.18
0.08; 1.31
0.079; 1.54
BBSL-soft
TS
10.72: 1.44
1,286; 1.39
0.782: 1.4
0.536; 1.29
0.089: 1.14
0.071: 0.88
RLLS- soft
TS
1.866; 1.06
0,905; 1.04
0.646; 1.08
0.069; 0.53
0.046; 0.55
0.04; 0.58
EM
NBVS
1.18;0.61 0.61
0.549; 0.65
0.396; 0.63
0.168; 1.25
0.123: 1.35
0.125: 1.39
BBSL-soft
NBVS
18.236; 1.53
2,241: 1.47
1.021: 1.39
2.678: 1.13
0.109: 0.88
0.067: 0.77
RLLS soft
NBVS
1.852; 0.86
0.879; 0.88
0,751;
0.072; 0.62
0.054; 0.77
0.046; 0.64
EM
BCTS
L082; 0.44
0.426; 0.49
0.304; 0.46
0.069; 0.65
0.038; 0.57
0.036; 0.69
BBSL-soft
BCTS
61.304; 1.57
1.439; 1.45
0,887: 1.49
0.747; 1.29
0.049; 1.2
0.043: 1.11
RLLS-soft
BCTS
2,412; 0.99
0.867; 1.06
0.736; 1.05
0,066; 1.06
0,043; 1.23
0.036; 1.2
EM
VS
148;; 0.6
0.503; 0.55
0,347: 0.5
0.066; 0.7
0.032; 0.56
0.029; 0.67
BBSL-soft
VS
14.874; 1.47
1,359; 1.44
0,866, 1.43
0.33; 1.3
0.049; 1.15
0.042: 1.15
RLLS-soft
VS
2.243; 0.93
0.89; 1.01
0.7: 1.07
0.065; 1.0
0.042; 1.29
0.035; 1.18
Table 4: Kaggle Diabetic Retinopathy: Comparison of EM, BBSL and RLLS. represents proportion of healthy examples in shifted domain; source domain has م = 0.73. Value before semicolon is the average MSE in the estimated shift weights. Value after the semicolon is the average rank of a method relative to others in the group that use the same calibration. A bold value in a group is not significantly different from the best performing method in the group (paired Wilcoxon test at p < 0.01). See Table H.I for an equivalent table but with statistical comparisons done across all calibration methods. EM tends to outperform BBSL and RLLS when calibration techniques involving class specifs bias parameters are used.
Shift Estimator
Callbrationn Method
11.1
1.0
10
--2000
w-40X0
--0000
X-00
1--2000
--9000
w-4000
--4000
EM
None
6.986, 2.73
6.126:3.17
6.938; 3:31
1,968, 3.36
2016: 3.44
2405;
3.69
0.25; 2.79
0.217: 3.58
0.263; 3-42
FM
TS
7,251: 1.68
2.13
2.21
2,127, 2.83
2.172:
2.92
2,204;
3.05
0.243, 3.1
0.225;
3.54
0.276;
3.37
EM
NBVS
7.324: 163
7.314: 1.59
7,318: 1.89
2.5: 1.46
2.592:
1.47
2,631:
1.45
0.706:
1.4
0.788:
1.31
0.84: 1.3
EM
BCTS
7.32% 1.99
7.337: 1.42
7,347: 1.4
2.593: 0.98
2.664; 1.0
2.688: 1.09
0.764;
1.24
0.33%)
0.94
0,884; 0.93
VS
7.255, 2:23
7,331:
1,372;
2.548: 1.37
2.652:
1.17
2.724;
0.72
0.741: 1.47
0.838; 1.05
0.88%; 0.98
Table 5: CIFAR10: Comparison of calibration methods when using EM adaptation to dirichlet shift, with A &accuracy as the metric. Unlike BBSL and RLLS, the EM algorithm does not rely on retraining to produce domain adapted probabilities. Value before the semicolon is the average change in &accuracy relative to a baseline of no adaptation. Value after the semicolon is the average rank compared to other methods in the same column. Bold values in a column are not significantly different from the best performing method in the column, as measured by a paired Wilcoxon test at p < 0.01. Calibration techniques involving class-specific bias parameters (namely BCTS and VS) tend to achieve the hest performance.
7

---

Under review as a conference paper at ICLR 2020
Shift Retimator
Calibration Method
o 0.11
o 1.0
10.0
7=0000
n=8500
==10000
n=7000
n:8500
n=10000
n=2000
R=5500
n=10000
EM
Nore
1.,41:40
44,45544.0
14.4c3:4 4.0
12,55:40
12.202: 4.0
12.109 4.0
1E:11:40
118195.4 4.0
11,83, 4.0
EM
T5
36.112, 1.4
16,101; 1.64
26.08:
1.68
21 21,455 1.82
21,638: 19
21,62.
1.9
20.721: 195
20.575. 2.2
20.89; 2.05
EM
NBVS
36.382, 1.6
26,33;
1.75
26.464: 1.7
21.588: 1.86
21.711:
1911
21,708
2.04
20.9: 1.9
21,059;
1.85
21,032. 1.93
EM
BCTT
36,485; 1.7
26,02;
1.47
26.73::
1.44
31,977 1.17
22,04:
1.33
22,015: 1.34
21,131; L1
21.31%
1,77
21,297: 1,99
EM
VS
26,889 1.1
26,901;
136
26:54:
138
21.9; 115
22,097:
22.183; 9.82
21.166;
145
21.40; 0.88
21.36; 0.93
Table 6: CIFAR100: Comparison of calibration methods when using EM adaptation to dirichlet shift, with A aaceuracy as the metric. Unlike BBSL and RLLS, the EM algorithm does not rely retraining produce domain adapted probabilities. Value before the semicolon is the average change &accuracy relative to a baseline of no adaptation. Value after the semicolon is the average rank compared to other methods in the same column. Bold values in a column are not significantly different from the best performing method in the column, measured by paired Wilcoxon test at I 0.01. Calibration techniques involving class-specific bias parameters (namely BCTS and VS) tend to achieve the best performance.
Shift Estimator
Calibration Method
P -= 0.5
= 0.9
n==00
n=1000
72= 1==1000
n=0
7== n==000
1==1500
EM
None
1.926; 3.09
2.076: 3.49
2.196: 3.64
1.296: 3,42
1.375;
1.477; 3.8
EM
TS
1.902; 2.96
2.225;
2.495:
1.626; 3.01
1.923; 2.88
1.973; 2.97
EM
NBVS
3,23; 1.69
3.789; 1.49
4.062; 1.54
2.074; 2.44
2.266; 2.24
2.405; 2.17
RM
BCTS
3.766; 0.88
4.356; 0.74
4.58; 0.82
3.548; 0.35
3.567; 0,36
3.722; 0.44
EM
VS
3.67; 1.38
4.278; 1.11
4.545; 0.87
3.5; 0.78
3.57; 0.71
3.746; 0.62
Table 7: Kaggle Diabetic Retinopathy: Comparison of calibration methods when using EM adaptation to domain shift, with A % accuracy as the metric. P represents proportion healthy examples in shifted domain; source distribution has = 0.73. Unlike BBSL and RLLS, the EM algorithm does not rely on retraining to produce domain adapted probabilities. Value before the semicolon is the average change in acccrray relative a bascline of no adaptation. Value after the semicolon is the average rank compared to other methods in the same column. Bold values a column are not significantly different from the best performing method in the column, as measured by a paired Wilcoxon test at S 0.01. Calibration techniques involving class- sass-speciific bias parameters (namely BCTS and VS) tend to achieve the hest performance.
5 DISCUSSION
In this work, we explored the effect of calibration on procedures designed to perform domain adaptation to label shift. In experiments CIFAR1O, MNIST, CIFARIO0 and diabetic retinopathy detection, we found the combination of EM-based domain adaptation with an appropriate calibration approach tends to outperform BBSL and RLLS. In particular, we find that the best results are achieved when the calirbation approach incorporates class-specific bias parameters that can reduce systematic bias in the class probabilities- something that not true of the popular Temperature Scaling approach recommended by Guo et al. (2017).
In addition. we observed that when the calibrated probabilities retain systematic bias, domain adaptation to EM is sensitive to the strategy used to compute the source-domain priors. If the source- domain priors p(y = i) are not defined probabilities p(y ;lrr), then EM will estimate a label shift even if the target domain is identical to the source domain (Lemma 1) and can produce highly detrimental results (Tables C.I & C.2). By contrast, if the source domain priors for EM are initialized as recommend in Sec. 3.2 EM becomes
way that mirrors the systematic bias in the predicted
substantially more tolerant of systematic bias in the calibrated probabilities, although it does not tend to outperform BBSL or RLLS in the presence of poor calibration.
One concern when using EM is the possibility of getting trapped in local minima. To address this concern, we analyzed the optimization of the likelihood function of EM and determined that it is concave Sec. A Thus, we can expect that EM will converge to the global maximum of the likelihood. Future work could extend this analysis derive generalization guarantees for the domain adaptation. In the meantime, our recommendation based on our empirical results is as follows: if there is insufficient data to calibrate the predicted probabilities, use RLLS for domain adaptation. Otherwise, use calibration approach that incorporates class-specific bias parameters (either bias-correcte temperature scaling or vector scaling), and use Expectation Maximization for domain adaptation.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Storkey Amos. When training and test sets are different: Characterizing learning transfer. In Dataset Shift Machine Learning. The MIT Press, 2008.
Kamyar Azizzadenesheli. Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. In International Conference on Learning Representations, 2019. /tttps::/oppnrseieew...
Yee Seng Chan and Hwee Tou Ng. Word sense disambiguation with distribution estimation.
Jeffrey De Fauw. Jeffeeed//hggglelllbbiiiiieeeeed www.aaaaiaceeiinnoaaaaiiteeeee Fifth place solution of the kaggle diabetic
retinopathy competition. htttps://gitini com Coom//effrey///aaggls diabetic retinopthyy Oct 2015. (Accessed on 01/22/2019).
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2):12-22, 1983.
Y. Geifman and R. Yaniv... Selective Classification for Deep Neural Networks. ArXiv e-prints, 2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. June 2017.
Kaggle. Kaggle competition diabetic retinopathy detection. http::/ww..ttts:://.w....ooooo https://wwwkaagle..coom -labbeti--reiinnpattyyyyooo -etpeppttttyyyeettttteee #eserriptiooooooooooooooooo 2015.
Volodymyr Kuleshov and Stefano Ermon. Reliable confidence estimation via online learning. arXiv preprint arXiv: pages 2586-2594, 2016.
Volodymyr Kuleshov and Percy S Liang. Calibrated structured prediction. In Advances in Neural Information Processing Systems, pages 3474-3482, 2015.
Zachary Lipton, Yu-Xiang Wang. and Alexander Smola. Detecting and correcting for label shift with black box predictors. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3122-3130, Stockholmsmässann Stockholm Sweden, 10-15 Jul 2018. PMLR. html
URL http://proceecinng/ mwr...mm.r...cm wrwppppessssee. wttp.pppees////////ttooooo.oo
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pages 625-632, ACM, 2005.
John C Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In ADVANCES IN LARGE MARGIN CLASSIFIERS, 1999.
Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new priori probabilities: simple procedure. Neural Comput., 14(1):21-41, January 2002.
Bernhard Schoelkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. June 2012.
Amos Storkey. When training and test sets are different: characterizing learning transfer.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B
Schön. Evaluating model calibration in classification. arXiv preprint /902: 06970, 2019.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. Citeseer.

---

Under review as a conference paper at ICLR 2020
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probabil- ity estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699 ACM, 2002.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th Interatioonal Conference on Machine Learning. volume 28 of Proceedings of Machine Learning Research, pages 819-827, Atlanta, Georgia. USA, 17-19 Jun 2013. PMLR. URL r. press/ /v28/222////// zhang 13d html
10

---

Under review as a conference paper at ICLR 2020
A LIKELIHOOD FUNCTION OF EM OBJECTIVE
In this section, we show that, in our label shift setting, the likelihood function is unimodal and concave. Thus, the EM approach converges the maximum likelihood estimation.
Lemma A: the EM objective is concave.
Proof: Let g(m))) and p(w, denote the target and source domain prior probabilities for class i. We wish to find target-domain priors ((w) that maximize the log-likelihood function given by
[(X: ((::((w))
Σ log
g( wx) (w:,)
(1)
Σ log
E p(ara w (
p(u; ( ( g(ws) p(ws)
Σ log
Σ
p(
Σ log
p(aak)
Σ
g(woz)
(4)
p(w;)
pla, xxx) g(us))
Σ log(p(<*)) + log
(5)
p(ws)
where 2 follows from the label shift assumptions and (3) follows from Bayes' rule. Now note that the maximization is independent of p(æxs) so the optimization problem is equivalently written as follows
plw (' p(w;) (wws)
Σ log
max ((w)
b
(6)
s.t.
1
q( =
q ((w.)
20 Vi
Note that the objective, both p(u; (xx) and p(w;) are constants with respect to ((w). Hence, the objective function is the sum of logs of linear functions in our decision variable and the constraints are affine. Therefore, the maximization problem concave.
JENSON- COSON-SHANNON DIVERGENCE METRIC FOR EVALUATION ADAPTATION B To LABEL SHIFT
In this section, we describe how we use the Jenson-Shannow divergence between the true target- domain priors and the estimated target-domain priors to evaluate given domain adaptation method. Let us denote the true target-domain prior as g(y i) and the estimate as q'(y = i). In the case of BBSL and RLLS, g'(x = i) can be calculated using the class weights w; which are intended to estimate q i)/p(y i). Specifically,
(yme) the formula d'(y i) Σ, [(= p' (y = i) is defined as the average of the true class labels in the source domain.
where
In the case of EM, some nuance is required. In Sec. 3.2 we noted that when performing EM, the source domain priors p(u = i) should be defined as the average of the predictions p(y = i|æ=) over the examples from the source domain (rather than being defined as the average the labels). Once the EM algorithm has reached convergence, it will output an estimate ğ(y i) of the target domain priors. Naively, it would seem that we could use q(y = i) as our estimate of q(y = i) when computing the Lensen-Shanne divergence. However, if p(y domain priors output by EM would likely also be systematically biased. We found that we can obtain superior estimates of q by leveraging the true labels in the source domain follows: first,
is systematically biased, the target
11

---

Under review as a conference paper at ICLR 2020
average the labels in the source domain to obtain to obtain p'y = i), which is an unbiased estimate of ply = z). Then, we use the ratio g(y = 1)/p(y = 1) from EM as a shift estimate. That is, we dis= Plast p' (y=i)
compute our estimate of the target domain priors using the formula q' (' = ΣS الwسي of (y=j)
We find that target domain priors computed in this way can occasionally perform surprisingly well even when the predictions themselves are not very well calibrated.
C COMPARISON OF STRATEGIES FOR INITIALIZING EM SOURCE PROBABILITIES
Shift
Calibration
p
0.9
Estimator
Method
w=500
1=1000
n=1300
N=500
1=10000
R=1300
EM: source priors from preds
Nome
1,266; 0.0
2,076; 0.0
2.19;; 0.0
1.2%;; 0.26
1.375: 0.17
1.477: 0.14
EM: source priors from labels
None
-3,488; 1.0
-3,541; 1.0
-3,382: 1.0
0.782; 0.74
0,937: 0.83
1.043: 0.86
EM: source priors from prods
TS
1,902; 0.0
2,225; 0,0
2,495: 0.0
1666;; 0.0
1.923; 0.0
1,973; 0.0
EM: source priors from labels
TS
-56.162 1.0
-62.552; 1.0
-64.195; 1.0
469.146 1.0
-76.619; 1.0
-83.083; 1.0
EM: source priors from preds
NBVS
3.23; 0.0
3,789;
4,062; 0.0
2.074; 0.02
2.266; 0.01
2.405; 0.02
EM: source priors from labels
NBVS
-9.48;; 1.0
-5.134; 1.0
4,772; 1.0
-2,616 0.98
0,431; 0.99
0.631;
EM: source priors from prods
BCTS
3,766; 0.0
4,356; 0.03
4,58; 0.01
3,548, 0.0
3,567; 0.01
3.722; 0.01
source priors from
RCTS
3,764; 1.0
4.387;
4.58;
3.548; 1.0
3.568; 0.99
3.723; 0.99
EM: source priors from preds
VS
3.675 0.08
4.278: 0.08
4.345; 0.08
3.5% 0.03
3.57:
3,746;
EM: source priors from labels
VS
3.662; 0.92
4.278; 0.92
4.559; 0.92
3.506; 0.97
3.572; 0.97
3.746; 0.97
Table C.I: The strategy for computing EM source priors heavily affects domain adaptation if probabilities retain systematic bias. Value before the semicolon is the average improvement in &accuracy (across 100 trials) caused by applying domain adaptation to the predictions on a diabetic retinopathy prediction task. Value after the semicolon is the average rank of a particular method relative to the other method in the pair. Domain shift is induced by varying the proportion of "healthy" examples p: in the source distribution, P = 0.73. We see that calibration methods that lack class- specific bias parameters (i.e. no calibration, TS and NBVS) can hurt domain adaptation if source priors are initialized by averaging true labels rather than the predicted probabilities. A bold value in pair significantly better than the non-bold value according to a paired Wilcoxon test at p S 0.01. See Table C.2 for analogous results using JS Div. See Sec. 4.1 for details on the experimental setup.
Shift Estimator
Calibration Method
P 0.5
p= 0.9
5=000
#=1000
n=1500
5=5000
>=1000
==0500
EM: source priors from preds
None
0,077; 0.0
0,059; 0.0
0,084; 0.0
0.111: 0.4
0.1; 0.37
0.102; 0.32
FM: source prioes from lahels
None
0.253; 1.0
0.256; 1.0
0,256; 1.0
0.125; 0.6
0.116; 0.63
0.114; 0.68
EM: source priors from preds
TS
0.09; 0.0
0.068: 0.0
0.061: 0.0
0.104: 0.0
0.094: 0.0
0.094; 0.0
source priors from lahels
TS
0.61; 1.0
0.628; 1.0
0.647: 1.0
0.629;
0.639;
0.643; 1.0
EM: source priors from preds
NBVS
6.107:
0.089; 0.0
0.079; 0.0
0.11; 0.0
0.1;
0.102: 0.0
EM: source prices from labels
NBVS
0.348; 1.0
0,327: 1.0
0.32; 1.0
0.214: 1.0
0.192: 1.0
0,188; 1.0
EM: source priors from preds
BCTS
0.111:0.4
0.095; 0.44
0.082;
0.41
0.078; 0.49
0.065: 0.54
0.063: 0.52
EM: source prioes from lahels
BCTS
0.111:0.6
0.095; 0.56
0.082; 0.59
0.078: 0.51
0.065; 0.46
0.063; 0.48
EM: souree priors from prods source priors from lahels
VS
0.108; 0.43
0.088: 0.3
0.076;; 0.48
0.077:
0.38
0.062,
0.059, 0.34
VS
0.108;
0.088; 0.5
0.076; 0.52
0.077: 0.42
0.062;
0.059; 0.46
Table C.2: Similar to Table quality of domain adaptation. See Sec. metric is calculated.
C.1
but using
Divergence as a metric to assess the B for description of how the Shasnonn Divergence
D CALIBRATION QUALITY COMPARISON
We find that biar-cord versions of Temperature Scaling (namely Bias-Corrected Temperature Scaling and Vector Scaling) tend to yield the best Negative Log Likelihood on an unshifted test set, even if they do not always yield the best ECE. Results are shown in the tables below.
12

---

Under review as a conference paper at ICLR 2020
Calibration
NLL
ECE
Method
n=2000)
n=4000
n=8000)
n=2000
n==000
n=
None
0.299; 4.0
0.299; 4.0
0.299; 4.0
2.726; 4.0
2.726; 4.0
2.726; 4.0
TS
2.99
0.291:
0.2
1.069; 1.23
1.06; 1.83
1.027;
NBVS
0.277: 1.67
0.275: 1.92
0.274; 1.99
1.109; 1.51
1 1.023; 1.51
0.952; 1.35
BCTS
0.274; 0.34
0.272; 0.54
0.271; 0.71
1,06; 1.02
0.987; 1.02
0.937; 1.06
VS
0.275; 1.0
0.272; 0.54
0.271; 0.3
1.161; 2.24
1.035; 1.64
0.976; 1.47
Table D.I: CIFAR10: NLL and ECE for different calibration methods. Metrics were computed on test set that had the same distribution as the validation set. Value before the semicolon the average of the metric over all the runs. Value after the semicolon is the average rank of the method relative to other methods in the column. n indicates the number of examples used for calibratin. Bold values in column are not significantly different from the best performing method in the column, as measured by paired Wilcoxon test at S 0.01. See Sec. 4.1 for details on the experimental setup.
Calibration
NLL
ECE
Method
n=7000
==8500
n= 1000)
n=7000
n=8500
==10000
None
1.735:
1.735; 4.0
1.735; 4.0
20.041; 4.0
20.041: 4.0
20.041;
4.0
TS
1,286;
1,286; 3.0
1,286;
3.134;
3.151;
3.135; 2.9
NBVS
2.0
1.2
1.239;
2.263; 0.09
2.281; 0.1
2.324; 0.1
BCTS
1.234; 0.71
1,233; 0.9
1.232; 1.0
2.879; 2.11
2.9:
2.881; 2.1
VS
1,234; 0.29
1,231; 0.1
1.229; 0.0
2,458; 0.93
2.48; 0.91
2.456; 0.9
Table D.2: CIFAR100; NLL and ECE for different calibration methods. Analogous to Table
D.I
Calibration
NLL
ECE
Method
n=500
1=100
n=1500
n=500
==1000
n=1300
None
0.64:4.0
0.639; 4.0
0.639; 4.0
8.734;
8.737; 4.0
8.767;
TS
3.0
0.57;
0.569; 3.0
3.75
3.729; 2.92
3.853; 2.76
NBVS
0.5
0.54; 2.0
0.539; 2.0
2.13;
2.028; 0.97
2.129; 1.01
BCTS
0.514; 0.21
0.511; 0.57
0.511: 0.63
2.255; 1.21
2.097; 1.17
2.171; 1.14
VS
0.518: 0.79
0.512; 0.43
0.51; 0.37
2.323; 1.35
2.065; 0.94
2.153; 1.09
Table D.3: Kaggle Diabetic Retinopathy Detection: NLL and ECE for different calibration methods. Analogous to Table D.I
13

---

Under review as a conference paper at ICLR 2020
E CIFARIO SUPPLEMENTARY TABLES
Shift Estimator
Calibration Method
P= 0.01
p= 0.9
n=20000
N=4000
n=80000
==20000
n=40000
==
EM
None
0.784; 2.91
0.798; 3.04
0.761:3.31
16.30; 3.79
16.356: 3.84
16,369; 3.92
EM
TS
0.807: 3.14 2.90
0.775.
17.193:
$7.26:2.67
17.288; 2.75
EM
NBVS
1.149; 1.31
1.172: 1.56
1.199; 1.39
17.588; 1.52
17.674: 1.51
17.738: 1.68
EM
BCTS
1.175; 1.38
1.224; 1.27
1.262; 1.15
17.724; 1.09
17.779: 1.17
17.84; 1.24
EM
VS
1.182; 1.48
1.258; 0.99
1.301; 0.75
17.727; 1.12
17.874; 0.81
17,988; 0.41
Table E.I: CIFARIO:
Comparison of calibration methods when using EM adaptation to "tweak-one" shift, with A &aaceurayy as the metric. Analogous to Table 5
Stiff Eotimator
Callbration
G
a 1.0
Nore
B100592 DN
QOPTINA: 1XTh7
0110119:
01001 100TEE5-0
0.00155:
0L00593;
0.0099::77
0.0005:
EM
BCTS
0.0115: 1.37
0 000075:222 3.35
0.0664:
3,04
0.00163:2 2.69
0
0.00099: 3.29
0.0062: 4.36
0,022 6.19
0.8009:: 5.53
0.0884: 5.99
EM
VS
0.00082 1.55
0.80077: 1.45
8006522 2.42
0J0066::2 2.73
0 0.80097 3,66
8,0R54:
0JM02 6.35
8.80011:32
0L000E 6.69
BOSL-
0.0866:
10.57
0.00222
000633:
0.03009
0.00005
S00028; 11.07
0.00144: 11:23
000007
B0SL-on
0,00822 8.29
B 000300 3:23
0.00190 7-9
0.00291 8.55
0.00159:7.00
0.00674: 2.54
0.021117444 1.44
0.20105:8
0
GUDDIS:: K.01
SBSL-art
NUYS
CU00IIC K.5
0.00092:88 8.2
01001:: 8.1
2.02
0.00143:700
0:00677: 2.21
IL0O2TT 7.36
0.00074: 6:3
BJIRST 4.69
SUSL-ac
BCTS
0/00616 8-49
0L20192:
2.72
0.01R6: 8:00
6:00-45: 7.69
0.00077--655
0.001:2.35
0.20090::5 2:1
QUODEN2 8.1
0:00075:
7.50 DU00RRS
RLIShad
0,0095: 10.55
0.00C21:
0,0055
0.00000 10.63
0.00440 11.15
0.0065:
Nooe
0,0073::60 8.33
B 010555 5/13
0.00992: 7.90
0.0055: 8.51
0.0019::8 8.7
0.00078: 7.55
0.00112 7.59
0,0014: 8
0,0054 7.86
0,0075:: 7.35
010257: 7:39
0.00995 7.63
0,00789: 8.57
11001 58: 7.09
0:00009: 7.95
0100111 1:24
0.0005: 811
KLLS-of
0,00119 8:33
000254::2 762
0.00199:
0.00288
0:00-43: 7.16
0U20071
2.46
0.0029:
0.20099: 6.27
0,005:: 6.77
KLLSoof
VS
0,00221,758 7.55
0.025222 7.96
0.00197; 7.54
0.00171 7.52
0. 0.00143;733
0J00077 2.63
0/021 7.3
0,20098
0.00622 7.51
Comparison of all calibration and domain adaptation methods, using MSE (Sec. 3.3) as the metric (dirichlet shift). Value before the semicolon is the average of the metric over all trials. Value after the semicolon is the average rank of the domain adaptation + calibration method combination relative to the other method combinations in the column. Bold values in column are not significantly different from the best best-performing method in the column as measured by a paired Wilcoxon test at p < 0.01. EM with BCTS or VS tends to achieve the best performance. See Sec. 4.1 for details the experimental setup.
Table E.2: CIFAR10:
Shift Estimator
Calibration Method
P 0.01
p= 0.9
n=20000
#==0000
N=8000
Nim 2000
#==000
#=80000
EM
None
0.00200; 9.59
0,001004; 11.31
0.00069; 13.25
0.08233;
11.91
0.07773;
13.38
0,0756; 14,63
TS
0.00212;
0,001
0.00074:
0.11247:
0.10919; 12.63
0.10942; 13.26
EM
NBVS
0,0015: 5.05
0.00066; 5.01
0.00041: 6.2
0.0041: 2.84
0.00263: 3.75
0.00206; 4.8
BCTS
0.0015:
0,00065;
0,0004; 4.89
0,00221:
0
0,00104: 1.72
EM
VS
0.00151; 5.09
0,00064; 4.14
0,0004; 5.04
0.00266; 1.08
0,00144; 1.19
0.00104; 1.25
BRSL-hard
None
0.00258; 12.4
0.001 19; 12.01
0.00059; 10.53
0.02157: 10:62
198; 10.15
0,00599; 8.98
BBSL.sof
Nome
0.00183: 8.21
0.00081: 8.03
0.00045; 6.85
0.01905; 9.79
0.01100: 9.49
0.00612: 9.55
BBSL-sof
TS
0,00181: 7.56
0.0008: 7.34
0.00044; 6.67
0.0201: 9.89
0.01171: 10.23
0.00677: 10.92
BBSL-soft
NBVS
0.00175; 8.07
0.0008: 7.62
0.00047: 7.67
0.01791: 7.96
0.00921: 7.03
0.00498: 6.99
BBfL-soft
BCTS
0.00174; 7.39
0. 0.0008 7.74
0.00046; 7.07
0.01808: 7.96
0,00937: 7.28
0,00501: 6.17
BBSI
VS
0.001
0.00081: 7.91
0.00047:
0.01 798;
0.00912;
0.00486;
RLLS hard
Nome
0,00256: 12.13
0.00119; 11.7
0.00059; 10.11
0.02055; 10.47
0.01172: 10.43
0.00596; 9.35
RLLS-soft
None
0.00182;
0.00081: 7.77
0.00045;
0.01872:
0.011:
0.0061: 10.24
RLLS-soft
TS
0.00179; 19: 7.24
0.0008: 7.06
0.00044; 6.37
0,0198; 9.8
0.0117: 10.82
0.00675: 11.68
RLIS-sofft
NBVS
0.00174; 7.55
0.0008: 7.24
0.00047: 7.29
0.01757: 7.92
0.00916; 7.54
0,00496; 7.73
RLLS soft
BCTS
0.00173; 6.9
0.0008 7.38
0.00046; 6.71
0.01769; 7.76
0.0093: 7.58
0.00499; 6.96
RLLS-soft
VS
0.00175; 7.71
0.00081 7.53
0.00047: 8.31
0.01759; 7.64
0.00906: 6.61
0.00484; 6.26
Table E.3: CIFAR10: Comparison of all calibration and domain adaptation methods, using MSE (Sec. 3.3) as the metric ("tweak-one" shift). Value before the semicolon is the average of the metric over all trials. Value after the semicolon the average rank of the domain adaptation + calibration method combination relative to the other method combinations in the column. Bold values column are not significantly different from the best- persfrrrrornng method in the column as measured by a paired Wilcoxon test at p < 0.01. EM with BCTS or VS tends to achieve the best performance, See See. 4.1 for details on the experimental setup.
14

---

Under review as a conference paper at ICLR 2020
Shif Eatimator
Calibration Method
0.1
1.0
10
u=4000
e=0000
n=10000
e=4000
==
==8000
EM
Nune
00035,
0.013; 5.8
6,035,6.45
0:25,9.25
0016,
0006: 7:44
9:3
EM
TS
2.34
0,022,28
3,32,33311
0.017, 8.67
Q014: 9.55
0006:537
1 9.73
OUME: 10:19
EM
NBVS
0.01: 1.56
0.017: 1.72
0017 1-
0013; 3
3,99
W985; 6
0,008
FM
BCTTS
0 1.38
0.016: 1.25
B015 1.38
6.00 2-01
0012 300
0.0%: 2.87
0945; 6-24
6:1:5574 5:7
6,00% 18
EM
VS
2,12
0.017: 1,56
0015:15
0.13 2.95
0.00% 3:74
0815: 6.59
6:11: 6.0
6 6,000 6.96
B8SL.
None
0.76, 13.38
0.164,
0005: 13:16
0.00
11:00
6:03 12.03
8016, 11.11
0019; 12:06
0,033 11:07
0002 10.1
BBSL-seft
None
0088: 11
11.37
0:00) 11:38
0,019, 9:02
0014, 8.36
0006:
0.08:
BRSI -00
TS
0067: 10 96
0.0% 58
0/H8: 10:29
9:000.900
0.019. 36
001% 7.73
0:016:
0011:845
0.00:
BRSI -00
NRVS
0.67: 10.70
0,055:
10.51
0/48 10.47
0.018: 7.80
0.13: 7.50
0,116 7.27
0011: 2,12
0.008:
BCTTS
0,066: 10.09
0,055; 10.03
9:55
8-87
0.018, 8.07
0.14, 8.02
0.016: 7.44
0011: 7.47
0.008: 6.91
BBSL-ssoft
VS
0:06;: 10:23
0,055: 10
9,67
0005:4-51
0.01::84
0.014::2
0,016: 7.53
0.11::
0.008: 7:58
RLI.S-hand
None
0.07;; 12.48
0,063: 24
0:056: 12
0.0 11 15.05
0:023:
0016: 11:91
0:019: 11.86
0.015; 11 13
0.009: 10.3
RIISS-of
None
0.065: 10.01
0.055: 9.94
B.HR; 9:05
9:007
0.019,9.12 14
0014: 8.46
0.016: 8.36..... 8.33
3011:8.46
0.008: 8.03
RLLS soft
TS
0.065: 8,94
0.055: 9.15
0.048; 9.13
0.00%
0.019; 8.54
0.03: 7.87
0.016:7:5 7.76
8.37
0.008: 8:48
RLLS soft
NDVS
0.066: 9.3
0.054; 8.93
0:047: 8.99
0.00% 9:01
0.018; 8.09
0.013: 7.56
0.166 7.09
0.011: 7,00
0,008: 6.93
soff-
BCTTS
0.065: 8.46
0.054; 8:39
0:06;: 9.34
0.025; 8.48
0.01:883
0.014: 8:07
0.016: 7.24
0.01::744 7:43
0.008: 7.83
Soft-soft
VS
0.065, 8.73
0,054: 8.65
0.046, 5.49
0,026, 8.45
0.015. 8.6
0.014, 8:24
0.066, 7.29
0.013,
0.008 7.72
Table E.4: CIFARIO: Comparison of all calibration and domain adaptation methods, using JS Divergence (Sec. [D as the metric (dirichlet shift). Value before the semicolon is the average of the metric over all trials. Value after the semicolon is the average rank of the domain adaptation + calibration method combination relative the other method combinations in the column. Bold values in a column are not significantly different from the best- tesfffrrrrming method in the column as measured by a paired Wilcoxon test at p < 0.01. EM with BCTS or VS tends to achieve the best performance. See Sec. 4.1 for details on the experimental setup.
Shift Estimator
Calibration Method
0.01
p 0.9
n 2000000
n==4000
12: R0=0000
2000
4==000
8000000
EM
None
0,018; 7.31
8.113,8.12
0.012;
TT.66 66
0.047:
4.59
0,045; 6.84
0.044; 10.62
EM
TS
0.02: 8.76
0.015, 9.26
0.014: 11.54
0.036; 2.51
0.034; 3.4
0.032;
EM
NBVS
0.016; 4.61
0.01: 4.73
0.008; 5.66
0.033;
0.028; 2.65
0.026, 3.77
EM
BCTS
0.016; 5.29
0.01;
0.009:
0.028; 1.0
0.023; 1.35
0.021; 1.79
EM
VS
0.016; 5.07
0,01:
0,008; 4.25
0,029; 1.32
0.023; 1.14
0.02; 0.88
BBSL-hard
None
0.026; 11.55
0.018; 11.66
0.013; 11.01
0.096; 14.24
0.078; 13.7
0.063; 13,76
BBSL-soft
None
0.021; 9.03
0.21
7.93
0.08; 11,66
0.064; 11.64
0.045; 11.17
BBSL-soft
TS
0.021;
0.014;7.45
0.39
0.078; 10.31
0.062; 10.2
0.043; 10.08
BBSL-soft
NBVS
0.02; 7.85
0.41
0.01; 7.56
0.075; 8.99
0.058;
0.04; 7.49
BBSI-soft
BCTS
0.02; 7.85
0.71
0.01
0.074; 8.61
0.057; 7.98
0.038; 6.36
BBSL-soft
VS
0.02; 8.27
8.66
0.01;
0.074; 8.32
0.057:
0.038; 5.8
-Lad--hard
None
0.026; 11.45
0.018; 11.4
0.013; 10.67
0.096; 14.22
0.078; 14.09
0.063; 14.1
RLLS-soft
Nore
0.021: 8.9
0,014;
7.77
0.08; 11.51
0.064; 11.95
0.045: 11.71
RLLS-soft
TS
0.021;
0.014;722
0.11
0.078; 10.33
0.062: 10.54
0.043: 10.46
RLLS-soft
NBVS
0.02;
0.014;8.25
7.41
0.075; 9.11
0,058; 8.91
0.04; 8.33
BCTS
0.02; 7.64
0.014;8.54
0.01; 7.01
0.074; 8.77
0.057; 8.55
0.038; 7.26
RLLS-soft
VS
0.02; 8.07
0.014;8.44
7.91
0.074; 8.41
0.057; 7.66
0.038; 6.72
Table E.5: CIFARIO: Comparison of all calibration and domain adaptation methods, using JS Divergence (Sec. B) as the metric ("tweak-one" shiff... Value before the semicolon is the average of the metric over all trials. Value after the semicolon is the average rank of the domain adaptation + calibration method combination relative to the other method combinations in the column. Bold values in column are not significantly different from the berf-perfo method in the column as measured by a paired Wilcoxon test at p < 0.01. EM with BCTS or VS tends to achieve the best performance. See Sec.
4.1 for details on the experimental setup.
Shift
Calibration Method
1.0
10
Estimator
1=00
4000000
0=0000
EM
None
0,035; 6.28
6.033: 6.42
0,033; 0.8
0,011 1,28
0.018; 1.76
0.016:
0.066 L.4
0.011; 1.8
0.009 2.2
HBSL-hard
None
0,176, 3.36
0.064:
3.32
D.05; 3.13
0,05: 2.65
(L023;
2.61
0.16.24
0.019;
0.013:
2.52
0.809) 2.27
BBSL
None
0.068:
0.056: 2.25
0.049: 2.22
0,027; 1.67
0.01%; 1.46
0,014; 1.2
0016 1.62
0.011: 1.58
0 1.42
RLLS hard
None
0,074:266
0.063:
0.054;
2.69
0.023: 2.59
0,016:
0.019 2.66
0.013: 2.58
0,009;
soft
None
0,066:
0.055:
0.018: 1.48
0.027;
0.019; 1.58
0,014: 1.3
0.016:
0,011:
0
EM
IS
0,024: 0.04
0.022:
0.09
0,021: 0.14
0.019;
0.58
0.017: 0.94
0.014; 118
0.016: L.88
0.012: 1.22
0.00%; 1.26
BBSL-
TS
0.067, 1.81
0.066:
1.78
0.048:
1.31
0.056
1:21
0.01%
0.94
0.013:0344
0,016 1.0
0011: 0:95
0.008 0.77
RLLS-so Ssfft
TS
0,066: 1.15
0.055:
1.15
0,048: 1.15
0.028 1.21
0.019: 1.12
0.013:095
0.016:
0.01:0085
0,068; 0.97
FM
NAVS
IL221; RJ02
8.017
6:14
0.0
0.38
0.01
D.015 0.88
0.08 0.01
DBSL soft
NBVS
0.067: 1.34
0.055:
1.8
0.048: L$
0.026
1.32
0.018:
1.21
0.013: 1.22
0.016: L15
0.01::
0.97
RLLS-toft
NBVS
0,866, 1.14
0.054:
1.16
0.047; 1.15
0:026
1.58
(LOEK: 1.41
0:013; 1332
0.016
0.97
0011;
1:04
0.038 1.07
BCTS
R002 1002
0031
0.22
0.009 1.36
0001 884
0.0
BBSL-sfft
BCTS
0.066, 1.82
0,055: 1.81
0.047; 1.35
0,025 1.4
0L08: 1.22
0.014; 1.29
0016 1.18
0011.
0.008; 1.01
RLLS--ooft
BCTS
0.065; 1.16
0.054: L.7
0.046: 1.17
0.025:
1.38
0.018: 1.44
0.014: 1:35
0.16; 0.98
0011: 1.1
0.008: 1.13
0.023;
0.017:0.0
0.015;
0.016;
0.013;
0 0.44
0,015:
0.01:
0.008;036
BBSL-soft
VS
01866,
0.055:
0.047: 1.36
0.025 1.34
0.018: 1.26
0.014; 1.24
0.01G: 1.18
0.011:
0:008,
RLf.Ssoftt
VS
0.063; 1.16
0.054: 1.14
0.046; 1.18
0.026. 1.36
0.018: 1.46
0.014; 1.32
0.016: 0.94
0.011: 1.11
0.008; 1.14
Table E.6: CIFARIO: Comparison of EM, BBSL and RLLS (dirichlet shift) using JS Diver- gence as the metric. Analogous to Table MSE.
[ but with JS Divergence as the metric rather than
15

---

Under review as a conference paper at ICLR 2020
Shift
Calibration
p 0.01
0.9
Estimator
Method
2000
4==0000
==8000
n=20000
w=4000
==80000
EM
None
0.018; 1.32
0.013; 1,66
0.012; 2.6
0.047; 0.14
0.045; 0.56
0.044; 1.42
BBSL-hard
None
0.026; 2.81
0.018;
0.51
0.096; 3.2
0.078; 2.87
0.063: 2.82
BBSL-soft
None
0.021; 1.61
1.61
0,014; 1.62
0.01;
0,08; 1.8
0.064; 1.5
0.045; 1.03
RLLS-hard
None
0,026; 2.73
0.018; 2.52
0.013;2.21
0,096; 3.18
0.078; 3.25
0.063: 3.16
RLLS-soft
None
0.021; 1.53
0.014;
0.01; 1.24
0.08; 1.68
0.064;
0.045;
EM
TS
0.02; 0.98
0.015; 1.1
0.014; 1.46
0.036; 0.04
0.034; 0.14
0.032; 0.48
BBSL-soft
TS
0.021; 1.05
0.014; 1.06
0.01: 0.88
0.078: 1.46
0.062; 1.25
0.043; 1.07
RLLS-soft
TS
0.021; 0.97
0.014; 0.84
0.01; 0.66
0.078; 1.5
0,062; 1.61
0.043; 1.45
EM
NBVS
0.016; 0,56
0.01; 0.6
0.008; 0.76
0.033; 0.04
0.028; 0.18
0.026; 0.42
BBSL-soft
NBVS
0.02: 1.28
0.014: 1.29
0.01: 1.19
0.075: 1,45
0.058; 1.12
0.04; 0,86
RLLS-soft
NBVS
0.02; 1.16
0.014; 1.11
0.01; 1,05
0.075; 1,51
0.058; 1.7
0.04; 1.72
EM
BCTS
0.016; 0.68
0.01; 0.54
0.009; 0.78
0.028;
0.023;
0.021; 0.24
BBSL-soft
BCTS
0.02; 1.24
0.014; 1.32
0.01: 1.21
0.074: 1.44
0.057; 1.17
0.038: 0.92
RLLS-soft
BCTS
0,02; 1,08
0.014: 1.14
0.01: 1.01
0.074; 1.56
0.057; 1.73
0.038; 1.84
EM
VS
0.016; 0,58
0.01; 0.42
0,008; 0.54
0.029; 0.02
0.023; 0.08
0.02; 0.1
BBSL-soft
VS
0.02; 1.29
0.014: 1.4
0.01: 1.31
0.074: 1.46
0.057: 1.15
0.038;
RLLS-soft
VS
0.02;
0.014; 1.18
0.01; 1.15
0.074;
0.057; 1.77
0.038; 1.92
Table E.7: CIFAR10: Comparison of EM, BBSL and RLLS ("tweak-one" shift) using JS Diver- gence as the metric, Analogous Table T but with tweak-one shift instead of dirichlet shift and JS Divergence as the metric rather than MSF.
Shift
Calibration Method
p
Estimator
n=2000
===0000
x=8000
==2000
1=40000
R=0000
EM
None
0,00202; 1.8
0.00104;; 2.38
0,00069;
3.04
0,08233;
2.83
0.07713:
3.23
0.0756:
3.74
BBSL-hard
None
0.00258; 2.95
0.00119;22.
2.77
0.00059;
2.53
0.02157:
2.04
0.01198: 1.83
0.00599; 1.37
BBSL-soft
None
0.00183; 1.45
0,00081; 1.31
0,00045; 1.33
0.01905; 1.47
0.01102:
1.19
0.00612; 1.2
RLI.S-hmm hard
None
0.00256; 2.68
0.00119;2.44
2.47
0.00059; 2.11
0.02055: 2.03
0.01172:
2.09
0.00596; 1.74
RLLS-soft
None
0,00182; 1.12
0,00081; 1.07
0,00045; 0.99
0.01872: 1.63
0.011: 1.66
0.0061; 1.95
EM
IS
0,00212; 1.35
0.00111:
0.00074:
0.11247:
0.10919: 1.48
0.10942: 1.6
BBSL-
TS
0,00181; 0.96
0.0008:
0.89
0.00044: 0.74
0.0201: 0.6
0.01171: 0.46
0.00677; 0.3
RLLS-soft
TS
0,00179; 0.69
0.00M8; 0.61
0.00044; 0.44
0.0198: 0.98
0.0117: 1.06
0.00675; 1.1
EM
NBVS
0.00115;
0,000066;
0.66
0,00041;;
0.0041: 0.18
0.00263;
0.42
6,00206; 0.64
BBSL-soft
NBVS
0.00175: 1.42
0.0008:
1.36
0.00047; 1.28
0.01791: 1.25
0.00921:
0.98
0.00498: 0.75
RLLS-sol >oft
NBVS
0.00174; 0.96
0.0008: 0.98
0.00047: 0.9
0.01757: 1.57
0.00916; 1.6
0.00496; 1.61
EM
BCTS
0,0015; 0.6
0.04065:
0.58
0.0004; 0.6
0.002211: 0.14
0.00137: 0.16
0.00164; 0.18
BBSL-soft
BCTS
0.00174; 1.41
0.00081:
1.39
0,00046;
0.01808;
0.00937; 1.13
0.00501: 0.96
RLLS soft
BCTS
0.00173; 0.99
0.00081;
1.03
0.00046; 1.02
0.01769: 1.58
0.0093: 1.71
0.00499: 1.86
EM
VS
0,000151; 0.56
0.000064;
0.52
0,0004; 0.56
0,00266;
0.04
0,00144;
0,12
0.00164; 0.22
BBSL-soft
VS
0.00176; 1.46
0,00081:
1.43
0.00047: 1.43
0.01796.
1.29
0.00912:
1.13
0.00486: 0.96
RLLS sofl
VS
0.00175; 0.98
0.00081; 1.05
0.00047: 1.01
0.01759; 1.67
0.00906; 1.75
0.00484; 1.82
Table E.8: CIFAR10: Comparison of EM, BBSL and RLLS ("tweak-one" shift) using MSE as the metric. Analogous to Table II but with tweak-one shift instead of dirichlet shift.
16

---

Under review as a conference paper at ICLR 2020
F MNIST TABLES
Shin Calhration
0IARS 12:54
BOOERE 11M
BUOISE 13:45
17:55
SCOIDIE 1735
IIM
NBVS
010043222 3.51
000153: 3.4
000048 4.3
0 0JM0202 4.6
6.6100:4115
(L00070 5.4
000/90
0.0066:
8.74
EM
BCTS
0.00128 1.53
0.0004: 1.97
0.00680
2.73
0000991 4.97
4455
090066
5:85
0
CURD6 SM2
0JM854 5.73
EM
VS
0.00333 3.65
0.00013 1.58
2.23
0000195 4.46
60800:440 444
PO070: 5.5
000444 2:70
0MN0K 980
000066 7.23
0.0407:
10.04
0.002344 3.95
17:34
600199
80001
0,00566
BBSL-
15
0.00573: 9:07
0.0035: 9.26
0.06302: K.35
0.00301 7.2
0300144663
6.61
0.0005:
6.70
0.001 5.5
GL0SSOK 6.28
0L0954: 5.58
58SL- -00
NBVS
0.00530 9:78
000397 9,69
0.00290 9.42
0U0005: 8:5
D00045: 7.87
000095:
7.65
000182;
000187:7790 7:56
00008: 7.90
000355: 7.44
BBSI-om
BCTN
010817:33 9:13
010792 0:41
8 B2MR0R-22 250
na0005: 7.24
110TIS: 2-4
BJBANN 6.4
0,80055:
3ard
9.69
0.0034
10.30
10.86
0.00144 11.03
RLLSaa
Noss
0.00513: 8.94
060419906
0,00550 9.13
0,00CTA 8.94
0.00551 900
0.00001: 8.20
000158: 8,39
0800022 875
0.0356: 7.54
15
0.00558: 5.12
0L00333 8.28
0.02580 7.51
0,00361 7.1
000044, 6.67
0,00P55 6.79
8,008: 5.7
0,000 6.14
0.0954: 5.46
NRVS
010817:
00009: 8.65
0.0005
0,0005:: 8-43
0000466300 8.0
GU00EAA: 2.18
000085: 7.0
010069:
010055: 1:16
0.0053:
0.0034:: 9.43
0.00266 8.8
0L00085: 8.61
0L001-9: 9.85
0U00099 8:43
0.00189 7.02
0 7:42
0.00066 8.5
Table F.1: MNIST: Comparison of all calibration and domain adaptation methods, using MSE
(Sec. 3.3) as the metric (dirichlet shift). Value before the semicolon is the average of the metric over all trials. Value after the semicolon is the average rank of the domain adaptation calibration method combination relative to the other method combinations in the column. Bold values in a column are not significantly different from the best- pest-performing method in the column measured by a paired Wilcoxon test at p < 0.01. EM with BCTS or VS tends to achieve the best performance, particularly for larger amounts of shift (corresponding to smaller a). See Sec. 4.1 for details on the experimental setup.
Shift Estimator
Calibration Method
= 0.01
0.9
n=2000
>==0000
N=X000
R=20000
==00000
n=5000
None
0.00219: 11.93
0,00112: 11.76
0.00072:
0.00998;
0.00648:
0.00528:
EM
TS
0.00183: 7.28
0,00091: 6.61
0.00058: 5.55
0.0062; 6.44
0.00325; 5.72
0,00199: 4.86
FM
NBVS
0,00177: 6.01
0,00088: 5.53
0.00056: 4.3
0.00181; 2.45
0.00088; 2.47
0.00044; 1.8
EM
BCTS
0.00173; 5.1
0.00087: 5.07
0. 0000056 3.74
0.0007: 0.75
0.00043; 0.69
0.0003: 0.61
EM
VS
0.00177: 6.18
0,00087: 5.01
0. 0,00056; 3.71
0.00183: 1.0
0,00049: 1.29
0,00033: 1.33
BBSL-hard
None
0.00235. 11.89
0.00123; 12.7
0,00033 14.18
0.01183;
12.81
0.00796: 13:47
0.00652: 14.83
BSSL-
None
0,00186; 9.08
0. 0.00099; 9.18
O 00063; 9.44
0. 0.00926; 10.52
0,00488; 8.26
0,00336; 5.59
BBSL
TS
0.00178;
0.00093: 6.6
0,0006: 6.73
0.00914; 10:29
0,00515;
0.00184;
BBSL soft
NBVS
0,00184; 8.63
0.00096: 8.7
O. 0.00062; 8.93
0.00887: 9.19
0.00509; 8.99
0.00388 8.79
BBSL-soft
BCTS
0.0018:
0,00094:
0.00061:
0.00879:
0.00506:
0.00373:
BBSL-soft
VS
0,00184; 8.57
0.00096; 8.55
0. 0.00063: 10.07
0.00894; 9.96
0.00526; 10.98
0.00415; 12.5
horlS-hard
None
0.00235; 11.15
0,00123; 11.86
000083 13.2
0.01099: 11.75
0.0076: 12.84
0.00633: 14.05
RLfS
Nome
0.001 86; 8.42
0.00099; 8.5
0.00063; 8.58
0.00875: 9.3
0.00478: 7.95
0.00335; 5.92
RLLS-soft
TS
0,00178; 5.89
0.08093: 5.96
0,0006: 5.85
0.00863: 9.0
0.00505: 8.94
0,00383; 9.41
RLLS-soft
NBVS
0.00184; 8.01
0.00096; 8.08
0.00062; 8.05
0.0084; 7.74
0.00499; 8.59
0.00379; 8.94
RLL.--soft
BCTS
0.0018; 6.28
0.00094; 6.66
0. 0.00061:
0.00832: 7,83
0.00497; 8.54
0.00372; 8.0
VS
0.00184; 8.01
0.00096; 7.93
0.00063; 9.23
0.00843;
D.00515; 10.46
0.00413: 12.72
Table F.2: MNIST:
Comparison of all calibration and domain adaptation methods, using MSE (Sec. 3.3 as the metric ("tweak-one" shift). Value before the semicolon is the average of the
metric over all trials. Value after the semicolon is the average rank of the domain adaptation + calibration method combination relative to the other method combinations in the column. Bold values in a column are not significantly different from the best- rest-performing method in the column as measured by a paired Wilcoxon test at p < 0.01. EM with BCTS or VS tends to achieve the best performance, See Sec. 4.1 for details on the experimental setup.
Shift Kottmattr
Calibeation Method
m01
1.0
CA 10
RMZ000
E33
Nane
21B5
05001150 1.55
CABEE, 200
DUOINN 2.74
2.77
WISI- Jued
Nov::
0800A8;
2.54
0.00887:211
0.00116;: 2.15
0U007T0: 2.91
0400E4: 3.14
3.57
BUR114:
3.12
BLDROO2: 2.44
BBSL sont
Nove
0300344; 1.97
0.0043
2.25
0.0099;
0UNI244;
1.54
0.80L51:
L.15
0,00101:
1.25
0.00185;
1.33
8.
8.00892)
1.44
0.00856; 1.17
RLLS-bord
0:0056: 2.04
0.0055,1
1191
0,0013:; 1.93
000015;
000020:
3.19
0400154:
2.34
0L00128;
2.15
000014:
0L0000:: 3.5%
KLLS sift
RUNGE3: 1.59
BJM9T: 1,83
0,ND53:169 1,69
0,0274)
1.48
0,00151;
1,31
0,00100:
0,00188 1,21
0,0092)
0J00556;
RL9S
EMI
0X0045 1UP
000055
0000070 1.0
000055
8.72
0000177 1.34
0000100;
1.16
1,02
BUINOOT 1,0
BRSI
010NTS: 1.18
0:00555 1.21
0.0076:;
1.16
1.07
0000164;
1.02
0.94
DODIE 1.00
010080 1.00
000084: 1.00
RLLS
0.01333-0922
0,8025; 0.84
000251:
1.01
0.00144;
1,05
0,00096:
0.93
0,0015:
0.00086
0.92
0,00254:
NBVS
8.56 0L000090
8,00079
CO00SS0 10
BUK08S,050 0032
0,0026, 0.9
BRSL-
0:0029: 1-6
0.00997
0,00H99
1.44
000085; 1.25
0.00146;
1.18
0.00187:
1.12
0.0088:
0,0005;; 1.00
KLIS
02001T: 1
0,00R5:
1.15
060066
112
0001160: 1.33
0UI0ENT: 0.80
0,00085;
0U00ES: 1.01
EM
BCTS
1J9128; 0.12
0,00058;
0.28
0,01911,048
0,80167;
0.54
0,005:: 0.52
0.001877 0.98
EL00064, 0.94
0,00254, 0.9
BRSI
BCTS
010612:11 1.5
0:0099:111
1:55
0,0009
1.51
000067;
0000149 1.1%
0L000099
0U00187: 1.11
00005:: 1.12
RLLS sift
0,0099. 1.28
0,0088;
1,27
0,0265:
1.2)
0002622
0,0014:: 1.33
0,00095:
0,00133: 0.91
0,00086;
0L00044; 0.98
ESI
10913A 1L14
0L17
0L00 000N
BL00RRS 0.34
LLDMOSS 2L55
BBSI soft RLL
0106350 1.55
1.55
01007:
1.50
000016:
0000L99;
1.17
1/20
1.11
0L00085: 1:31
0.00006 1.1
0.00023
0.00354:
1.27
0,00266
1.25
000265; 1.25
0,00149; 1.32
0.00098; 1.13
6.00189;
0.93
0.00085;
0.00005; 1,02
Table F.3: MNIST: Comparison of EMM BBSL and RLLS (dirichlet shift). Analogous to Table 2. but with dirichlet shift rather than tweak-one shift.
17

---

Under review as a conference paper at ICLR 2020
G CIFARIO0 SUPPLEMENTARY TABLES
Stift
Cullbration
Neat
RUSTES
0.85752. 8.06
8.15
0.5005)
9:55
0.9059G,
0.26756,
0.15609 10.56
IM
NIVS
0.24944 4.03
0 2222816
4007
0L29994: 4.68
0.15649: 6.75
18833 5.34
0.14394: 6.21
TUSS55: 5-5
0. 1110675: 2
0.16256 K.
EM
BCTS
3-189:3116
B 252555: 1-53
0L25629: 3.68
0.14522: 3.81
8 160000- 4.37
0.03 1955 3.1
I.OOURS A34
0U00811 3.44
QUWSSIC 8.54
1L1994: 2.41
6.2011: 2,67
0.386:6
3
0.13788:
3.06
0.1387: 3.6
0.E27M6: 4.34
0,2968: 3.2
0U2066:: 8.68
BBSL.wo
1.3338
0:94231
0.5585
94
0.37300
0.21547
12.63
0.1699
14857: 12:35
BASI-wrt
TS
18011:907
D 73366 5:25
0 9.4
0.25363:50 5.51
0 0221188 8:50
0
15
0 0119557 7:69
0. 0111961:22
2:15
0,10114 69
BBSI-o0
NRVS
LO0GAN 8.98
D 0A643 5.76
0.60997: 8.05
0126200 8:3
81 19391: 5.27
0.00877 8.11
15885 1-46
0122011:22 7:56
0,10221: 7.45
19399:
0.11855:7 7.11
0.1
E0SL-ad
0,23565
7.44
0J86574
7.52
0,09966
KLLShta
0,89144 10.2
0,24954; 10.58
0,71544, 10.44
0.3129: 10.82
0,26379. 10.74
0.23164: 10.5
0.18167 10.78
0.15771: 18.1
0,14035 10:62
RLLS-oo
Noar
6.72650 8,08
D 066146 7.78
0.51115; 8.52
0.22919: 7.92
0. 1948 7.85
0.17306: 5-2
(01425 8.06
0.12055: 8.13
0 1055: 7.67
6.58
5835322 541
0.52499
0.20168 593
0L8663
0J14395
011200
0.39967:
008666
BCTS
0 1111666550 5.17
0.4722% A.
1919557 5.05
B 15339 4:51
IL154I: 3
0:31545:41
6,8953:: 3.36
0J18289 3.01
R11Sach
VS
6.66076664
0 021132:562
0/47947 635
0. $.5
1S999 5003
IL13822 4.14
0.11992: 4.65
CUSSM) 4.2
010179: 3-5
Table G.I: CIFAR100: Comparison of all calibration and domain adaptation methods, using
MSE (See. 3.3) as the metric (dirichlet shift). Value before the semicolon is the average of the
metric over all trials. Value after the semicolon is the average rank of the domain adaptation + calibration method combination relative to the other method combinations in the column. Bold values in a column are not significantly different from the best- gest-performing method in the column as measured by a paired Wilcoxon test at p < 0.01. EM with VS tends to achieve the best performance, particularly for larger amounts of shift (corresponding to smaller a)). See Sec. 4.1 for details on the experimental setup.
Shift FM
Callbrationn Methed
Nonr
11283:11:7
0.233 12.11
0.232 12:55
0.292 15
0.2311 15
0.23; 15.58
0.205; I8
OC254: 15.89
0.25% 15.94
LM
TS
0,113; 1.27
0.113: 1.37
0 1.45
0.109; 1.07
0,107: 1.05
0.106; 1.14
OL107: 0.96
0.105: 0.87
0,100; 0.92
LM
NBVS
2.39
0.119: 2:45
0.119:2-4
2.48
2.6
0.117: 2.85
0.1166 3.15
0.117: 2.92
0,115: 3.04
0.113: 3:21
DM
BCTS
0.118: 1.99
0.117: 2.12
0.119:2.0 2.12
0.112: 1.55
0.1 1.78
0.1.: 1.96
0.112: 1.63
0.11: 1.6
0.108: 1.8
FM
V5
0.113: 1.83
0.112: 1.46
0.112: 1.59
0:011: 1:21
0.108: 1.06
6.167: 1.27
0.108: 0.9
0.106; 0.86
0.101;
BRSI -hand
Nonr
0.278: 15.00
0.27: 15.8%
0.267: 15.88
0.209: 15.57
0.230 15:47
0.227:
15 36
0:221: 15.19
0:215: 15.1
0.209: 15.06
DBSL soft
None
0,348: 14.35
0.241: 14.31
0.257: 14.34
0.21: 14.01
0.204: 13.95
0.198: 13.91
0.196; 15.51
0.19: 13.8
0.183: 13.82
DOSL soft
TS
0.224: 10.6
0,218: 10.83
0.214: 10.85
0.157: 10.56
0.151: 10.72
0.178:
10.83
0.176: 10.43
0.17: 10.54
0.165 10.67
DOSL 1oft
NBVS
0.226: 12.0
0 11.95
0.216: 11.76
0,159: 11.09
0.183: 11
0.178: 11.59
0.178: 11:23
0.172 11.86
0.166: 11.84
BBSI -off
BCTS
0,224; 10.99
0.218: 10.97
0.214: 10.75
0.187: 10.31
0.181: 10:05
0.1 10.11
0. 0.176: 10.05
0.17: 10.07
0.164: 10.05
HRSI -uff
V5
0.226; 11.7
0.22: 11:59
0.215: 11:35
0. 18888; 10.
1.122 10:82
0.17% 10.76
6. 0.177: 11.21
0.171: 11.0
0.165 11:07
RLLS hard
None
0,226: 11.48
0 11.13
0.218: 0.218:11.35
0,195: 11.91
0.189: 11,83
0.183: 11.64
0,182: 11.67
0.176: 11.65
0.1T: 11.55
RLLS soft
None
0,302: 8.09
0.197:
0.193:
0.111:3.04
0,166: 8.13
0.15:33.9
0.161: 7:92
0.156:
0.149; K.0
RLLS soft
TS
0,182: 4.96
0.176: 4.92
0.172: 4.72
0.152 499
0. 0.146 4.82
0.54:4455
0. 4.94
0.145 4.81
0.134:4555 4.59
RILS-soff
NBVS
0. 1868: 6.01
0.17% 6.05
0.176:555 5.82
0.185 6.05
0. 149: 6.05
0.144: 5.92
0. 149: 6.04
0. 6.142:6117 17
0.137: 6.15
RIIS-
RCTS
0.183 5.0
0.177: 5.01
0.174: 4.88
0.183: 4.74
O. 146: 466
0.141: 4.44
146: 4.53
0.14:-4 4-46
0 4:32
RLLS soft
V5
0,186, 6.07
0.17%; 5.89
5.176.5.78
0.955,5.92
0,149; 5.85
0,144::592
0.349; 6.27
0,143:6.29
0.137; 6.12
Table G.2: CIFARI00: Comparison of all calibration and domain adaptation methods, using IS Divergence (Sec. B) as the metric (dirichlet shift). Value before the semicolon the average of the metric over all trials. Value after the semicolon is the average rank of the domain adaptation + calibration method combination relative to the other method combinations in the column. Bold values in column are not significantly different from the best-performing method in the column as measured by a paired Wilcoxon test at p < 0.01. Although TS attains good performance according to the JS Divergence metric, this is not the case with the MSE metric (Table G.I.. See Sec. 4.1 for details on the experimental setup.
Shift Estimator
Calibration Method
Q-
10.0
7=70000
10m 10000
7-70000
1500
TH 10000
7000
15-5500
None
1.77
0.232;
0.232.
01311 3.44
0,25,338
0,215,
3.81
0:239;
0.233;
#BSL-hard
None
0.278,3.92
0.27,
0:267; 39
0.259,3.55
01331 3,47
0:221:3330 3.36
0228-3319
3.19
0.215:3.1 3.1
0.209;
BBSL-sofi
None
0.248.255
0.44
0.237,24 24
021:200
0.206 1.95
0398:
191
11186115
1.11
0.19:11
0.183; 1.82
RLLS hard
None
0.225, 1.52
0.22: 1.4
0.218: 1.44
0.195 1.19
0119%
112
0188
113
111
9182:
1.18
01176: 1.19
0.17;
R11.5-offt
None
0.202; 0.28
0.197; 0.28
0.193: 0.22
0,1711
0.9
0.166;
0
0.16:
0:02
0.161:
0.01
0.156;
0.02
0.149; 0.01
0.113;
0.113:
0113: 0.07
0.10%
0,00
0.107: 900
0.10%0
0,0
0.107:000
0.02
0.105;
0.103
TS
0.224,30 2.0
0.318:
0211::20
0187,
2.0
0181:20
01176 02
01176:
2.0
017:2.0
0.165;
RLLS soft
TS
0.182:097
0.176: 0.95
0.92
0.152
0.99
0.146,
0.146.
0.14;:
0.134:088
EM
NBVS
6.119
0.119
0.18
0.117
0.115:
0.
soft
NBVS
0.226:
0.22: 1.99
0.216:
0.189.20
0.183: 2.0
0.18::20
0.178.200
0.172;
RLLS-1oft
NBVS
0.186, 0.92
0.179.
0.176:09
0.155,
0.14% 0.93
0.14:09
0.14% 0.96
0.142;
0.157,091
FM
RCTS
I.TR% RJ07
8.11
0.117:
0.T1T:
0.11:
L 8083
0.TT:
0,166;
BBSL soft
BCTS
0,224: 1.99
0.218; 1.99
0.214; 1.99
0.187:2.0
0.1
0.176:2.0
0.176, 2.0
0.17:
0.164:200
RLLS-soft
BCTS
0.163:094
0.977:0..2
0.94
0.153:
0,141:091
0,146 0.97
0.14:
0,134:093
EM
VS
0.113:
0.112: 6.07
0.112: 0.1
0.111: 0.02
0,108: 0.03
0,107; 0.06
0.108: 0,01
0,1066; 0.01
0,364; 0.03
BBSL-
VS
0.226, 1.99
0.22;
0.215: 1.99
0.188 2.0
0.1822 2.0
0.177;20
0.177, 2.0
0,171:
0.165: 2.0
RLLS--oftt
VS
0.186:0.95
0.179:0.93
0,176; 0.91
0.155. 0.98
0.149; 0.97
0,144:054
0. 0.149, 0.99
0.143: 0.99
0,137:0.97
Table G.3: CIFAR100: Comparison of EM, BBSL and RLLS (dirichlet shift). Analogous to
Table 3 but with IS Divergence (Sec.
B as the metric rather than Mean Squared Error.
18

---

Under review as a conference paper at ICLR 2020
H DIABETIC RETINOPATHY SUPPLEMENTARY TABLES
Shift Estimator
Callbration Method
P- 0.5
0.9
TUDO
EM
Nonc
1,25445; 5.32
0.52957. 5,33
0,38875, 2,,5,,,,. 26
0,T12AS;
0107876: 11 25
0.0007 12.07
EM
0.33443;
0.11
0,07959:
EM
NBVS
1,18016: 6.23
0,54915: 5.98
0,39599; 6.26
0.16834; 10.2
0.1248: 11.07
0.155:: 12.53
EM
BCTS
1,8208; 4.54
0.42625; 4.02
0. 4.55
0.06894; 4.33
0J83; 3.91
0.03581: 4.15
1,47999,
0,50291; 4,78
0,34718;
0.06652;
0,3383:
595.53053: 11.48
1087. 16344;
1,74885; 12
244774
46239;
0.7437X
BBSL-sof
Nose
12. 22.250 9.3
A40652 9.88
0.81453, 8.88
1.17077; 10.75
0,09828: 9.95
0.08817: 9.31
BRSL-
TS
10:71961: 9.3
2.5777, 8.81
0,783133, 6.63
0,53555 9887
0,08911 9:45
D.0714 8.81
BBSL-so0
1:
1,2200:
2.67819:
0.06728:
BBSL-saft
BCTS
61.20409: 10.7
1.3944; 9.08
0.88667; 9.61
0.74657; 8.1
0.04882: 6.26
0.04445; 6.08
BBSL-sof
VS
14,57357: 9.91
1.35907: 8.96
0,8657: 9.08
0.3301: 7.25
0.04923; 6.34
0.01 6.0
hard
Nome
2,20358;
1.39785; 11.65
1006933; 11.24
0,10203 8.01
0.04938: 7.69
0.05432: 8.04
RLLS-soft
None
1,95288;
092663, 8.85
0.66962,
0.06693; 6.08
0,04131; 6.27
DA0922:
REI S-soff
TS
L.86IS:: 7.66
0.90546, 7.71
0.64625; 7.48
0.0992: 6.07
0,04575: 6.87
0.0IER6: 7.09
RLLS-soft
NBVS
188199; 7.38
0,87914, 7.36
0,7507 8.55
0 6.8
0,05395: 8.1
0.04557: 7.5
RLLS-soft
0.86736: 7.31
0,7362;
0.03614:
RLLS-soft
2.2435: 7.5
0,88968, 7.39
0.70032; 8.07
0.06544; 6.1
0.01173; 6.74
0.03524: 6.4
Table H.1: Kaggle Diabetic Retinopathy Detection: Comparison of all calibration and domain adaptation methods, using MSE (Sec. 3.3) as the metric. P represents the porportion of healthy examples in the sfhited domain; the source distribution has p = 0.73. Value before the semicolon is the average of the metric over all trials. Value after the semicolon the average rank of the domain adaptation + calibration method combination relative to the other method combinations in the column. Bold values in a column are not significantly different from the best-performing method the column as measured by paired Wilcoxon test < 0.01. See Sec. 4.1 for details on the experimental setup.
Shift Estimator
Calibration Method
p 01.5
p
==500
n=1000
==1500
n=300
7o=0000
1=1300
EM
None
0.077; 1,96
0.059; 2.03
0.054; 2.26
0.111; 10.66
0.1: 10.84
0.102: 11.62
BM
TS
0.09: 3.27
0.068; 2.57
0.061; 2.74
0.104; 8.84
0.094; 8.91
0.094; 9.55
EM
NBVS
0.107; 5.56
0.089; 5.25
0.079;
0.11; 9.69
0.1:9.95
0.102; 10.77
EM
BCTS
0.111:
6.04
0.082;
0.078; 3.65
0.065;
0.063; 3.67
HM
VS
0.108; 5.53
01088; 4.82
0.076; 4.98
0.377:3..66
0,062; 3.02
0.059; 3.03
BBSL-hard
None
0,227: 12.49
0.19; 13.67
0.154; 13.18
0,196; 13.12
0.157; 13.15
0.128: 12.18
BB So-fftt
None
0.17: 10.69
0.127: 10.67
0.105:9.7
0,131: 11.43
0.096: 10.27
0.091:
SBS--foft
TS
0.16; 10,23
0.55
0.1:9.11
0.122: 10.84
0.092: 9.42
0.089: 9.29
BBSL-
NBVS
0.176; 11.21
0.121:9.36
0.106; 10.22
0.125; 9.81
0.093;
0,086; 8.41
BBSL-soft
BCTS
0.167: 10.34
0.116; 8.9
0.096;
0.107: 8.18
0.079:
0.077: 6.54
BBSL-soft
0.159; 10.13
0.118; 9.3
0.093; 8.25
0.101; 7.29
0.079; 6.59
0.077;
RLLd-hurd
None
0,148; 9.41
0,161; 12.31
0.143; 12.16
0.104; 8.47
0.889;888
0.087;
S-sft
None
0,134; 8.22
0.123; 9.88
0.102;9.05
0.092; 6.6
0.082;
0,081; 7.03
RLLS-soft
0.133; 8.12
8.64
0.098;
0.092; 6.6
0.083,
0.083; 7.83
RLLS-soft
NEVS
0.124; 7.69
0.305;
9.48
0.086;
0.085; 8.36
RLLS-soft
BCTS
0,127; 7.69
0.71
0.094; 8.14
0.089: 5.47
0.078: 6.28
0,078; 6.56
RLLS-soft
VS
0.124:
7.1
0.091; 7.28
0.089; 5.77
0.079; 6.37
0.077, 6.52
Table H.2: Kaggle Diabetic Retinopathy Detection: comparison of all calibration and domain
adaptation methods, using MSE (Sec.
(3.3 as the metric. Like Table H.1 but with JS Divergence
(Sec. B) as the metric rather than MSE
19

---

Under review as a conference paper at ICLR 2020
Shift Estimator
Calibration Method
0.5
0.9
Y==00
T=00000
N= x=1500
N=000
V== 1000
YU= 1500
1=0000
EM
None
0.077; 0.36
0.059; 0.15
0.054; 0.26
0.111; 1.99
0.1:
0. 0.102; 2.44
BBSL-hard
None
0.227;3.2
0.19; 3.39
0.154: 3.25
0.196; 3.14
0.157: 3.16
0.128: 2.86
BBSI-soft
None
0.17:2.54
0.127; 1.98
0.105: 1.97
0.131; 2.35
0.096;
1.84
0.091: 1.79
RLLS-hard
None
0.148; 2.16
0,161: 2.73
0.74::2.7
0.104: 1.61
0.089; 1.63
0,087; 1.6
RLLS-soft
None
0.134; 1.74
0.123; 1,75
0.102; 1.75
0.092; 0.91
0.082; 1.22
0.081; 1.31
EM
TS
0.09; 0.26
0.068; 0.2
0,061; 0.15
0.104; 1.04
0.094; 1.05
0.094; 1.16
BBSL-soft
TS
0.16; 1.65
0.119; 1.59
0.1; 1.54
0.122:
0.092; 1.17
0.089; 1.06
RI.L--soft
TS
0.133; 1.09
0.1 1,21
0.098: 1.31
0.092; 0.56
0,083; 0.78
0.083; 0.78
HM
NBVS
0.107; 0.42
0,089; 0.52
0.079; 0.42
0.11; 1.2
0.1: 1.21
0.102; 1.31
BBSL-soft
NBVS
0.176; 1.6
0.121: 1.48
0.106; 1.42
0,125; 1.2
0.093;
0.086: 0.89
RLLS-soft
NBVS
0.124; 0.98
0.105; 1.0
0.101; 1.16
0.092; 0.6
0.086; 0.79
0.085;
EM
BCTS
0,111; 0.6
0.095; 0.62
0.082; 0.72
0.078; 0.61
0.065; 0.65
0.063; 0.54
BBSL-soft
BCTS
0.167: 1.49
0.116; 1.37
0.096; 1.31
0,107: 1.42
0.079; 1.18
0.077; 1.21
RLLS-soft
BCTS
0.127: 0.91
0.11; 1.01
0.094; 0.97
0.089; 0.97
0.078; 1.17
0.078; 1.25
EM
VS
0.108; 0.55
0,088; 0.5
0,076; 0.61
0.51
0.062; 0.47
0,059; 0.55
BBSL-soft
VS
0.159; 1.6
0,118: 1.43
0.093: 1.4
0.101: 1.46
0.079; 1.31
0,077: 1.24
RLLS-soft
VS
0.124; 0.85
0.11; 1.07
0.091; 0.99
0.089; 1.03
0.079; 1.22
0.077; 1.21
Table H.3: Kaggle Diabetic Retinopathy Detection: but with JS Divergence (Sec.
Comparison of EM, BBSL and RLLS. as the metric rather than MSE.
Analogous to Table
B
I NLL CORRESPONDS BETTER TO BENEFITS IN LABEL SHIFT ADAPTATION
To investigate whether NLL or ECE corresponded better to the benefits offered by calibration method in the context label shift adaptation, we adopted the following strategy: in a given experimental run, we identified the calibration method that provided the best NI. (or ECE) on the unshifted test set. We then looked at the performance of label shift adaptation using this calibration method. Note that the calibration method selected can differ from that, by and large, selecting a calibration method according to the NLL produced better performance after domain adaptation compared to selecting calibration method according to ECE. Results show in the tables below.
to the next. Across datasets, observed
Calibration
Estimator EM
Method
8=0000
X-20000
n=20000
on R000
6=200K1
6-4000
w-0000
Best NCL
7,332;
7,326 0.32
7.37;
2.393: 0.36
2.664; 0.19
2,885 0.96
0.161:0422
0.83% 0.04
0,884; 0/3
Best ECE
7,298:
0.68
0.72
1777
0.54
0. 0.276,
Table L.I: CIFARIO: NLL vs ECE, A% Accuracy, dirichlet shift. Entry in "calibration method" column indicates how the calibration method for any given run was selected: either according to whether produced the best NLL or whether it produced the best ECE, where NI.L and ECE were calculated on the unshifted test set. Value before the semicolon is the average change in Saacurrac relative to unadapted predictions. Value after the semicolon is the average rank of the given metric relative to the other metric in the pair. A bold value is significantly better than the non-bold value in the pair using a paired Wilcoxon test at & 0.01. See Sec. 4.1 for details on the experimental setup.
Shift Estimator
Calibration Method
P 0.01
0.9
2000000
40=0000
800000
12=
41=0000
8000000
Best NLL
1.192; 0.17
1,253; 0.21
1,301; 0.15
17.724;
0.47
17.779; 0.08
17.84; 0.07
EM
Best ECE
LO05;; 0.83
1.149; 0.79
1.16; 0.85
17.727: 0.53
17.26: 0.92
17.288; 0.93
Table I.2: CIEAR1O: NLL vs. ECE, metric: A Aaccuracy, "tweak-one" shift. Analogous to
Table [.1
The "tweak-one" shift strategy explained in Sec. 4.1
20

---

Under review as a conference paper at ICLR 2020
Calibration
Estimator
Method
6=2300
R=:000
n-5000
7=2000
1120000
N=3000
R=2000
7510000
R-5000
HM
Bist NIT.
1.03
TA35 0.4
1488 01.36
0..008
0.14
0391 0.12
1 0.51
T:41; 1.31
0,775 0.3
EM
Best ECE
2.144: 0.67
1,65;
1,642:
1.69:0662
1.714; 0.56
1,398; 0.88
1,315: 0.69
1,156
0,882,0.7
BBSL-soft
Best NLL
6.636, 0,41
0.35
4.089;
0.49
2,555, 0.49
L.00 047
1,356 0.55
1.359,03 0.5
1,0%;; 0.41
0.789; 0.38
BBSL
Bevl ECE
6.53
5.506: 0.65
4.703: 0.51
2.55,0.1
1.87: 0.53
1.3500045
1.59: 0.5
0.59
0.818;
RLLS-soft
Best NEL
6.33,0036
5.39%
0.36
4.625,0.49
2,545,040
1.559 0.5
1,0%6: 0.41
0.78%;
R1..S-ooft
Best ECE
6.55.084 0.64
0.64
4,638: 0.51
2,550; 0.51
1.87:
1.339; 0.45
1.35%; 0.5
0.134
0.818, 0.62
Table L3: CIFARIO: NLL vs. ECE, metric: JS Divergence, dirichlet shift. Analogous to Table
I.1
but using JS Divergence (Sec.
B as the metric rather than change in aaccuracy.
Shift Estimator
Calibration
= 0.01
= 0.9
Method
11 2000
rc=4000
n=8000
n=20000
n==0000
8==0000
EM
Best NLL
1.566; 0.29
10;; 0.33
0.793; 0.29
2.787: 0.39
2.339; 0.13
2.132; 0.04
FM
Best FCE
1.695; 0.71
1.099; 0.67
0.906; 0.71
2.908; 0.61
3.388; 0.87
3.223; 0.96
BBSL-soft
Best NLL
2,012; 0.4
1.409; 0.51
0.985; 0.55
0.59
5.737; 0.25
3.84; 0.16
BBSL-soft
Best ECE
2,005; 0.6
1.389; 0.49
0.959; 0.45
7.393; 0.41
6.214; 0.75
4.328: 0.84
RLLS-soft
Best NLL
2,011; 0.4
1.409: 0.51
0,985: 0.55
7.433; 0,58
5.738;
0.25
3.841; 0.16
RLLS-soft
Best ECE
2,004; 0.6
1.389; 0.49
0.959; 0.45
7.395; 0.42
6.214; 0.75
4.328; 0.84
Table L.4; CIFAR10: NLL vs. ECE, metric: JS Divergence, "tweak-one" shift. Analogous to
Table L.
Estianttr
Method
0.17093: 0.17
0,8711: 8:17
686336: 8.27
OU6TE 0044
8.9884:
8:88
6,85126; 8:06
011989:03
6,80144;
0.49999 0.22
1,5595
0.4624 0.55
0.56
0.92
0.
0L11909 0.76
Best NLL
030018 1.33
0,26300 TLA
0.14399 0.3
0:40
0.08544 8.41
0A81666 8.4
WASI
Bow RD8
033099 0.57
10146 066
617883-0
0.54
0.009
6 0.445
0.011
Bii NLT
0.71655 1::1
0:
0.14
02191250.31
0.00854; 8.41
0,8166;; 8.4
KLLS soft
Best BUB
0,703322 1.59
0.289965 0.62
0.18421:0311
0.27112:0443
0.15% 0.5
0,0785 US4
0,2095 0.49
0. 104755 0.59
0
Table I.5: CIFAR10: NLL vs. ECE, metric: MSE, dirichlet shift. Analogous to Table
I.1 but
using MSE (Sec. 3.3
as the metric rather than change in %accuracy.
Shift Estimator
Calibration
p =: 0.01
p 0.9
Method
2000
7==0000
N000
2000000
4==0000
R000000
EM
Best NLL
0.14964; 0.27
0,06447; 0.3
0,03926; 0.35
0.2213, 0.43
0.13654; 0.02
0.1036; 0.02
EM
Best ECE
0.17892: 0.73
0.07276; 0.7
0.04478: 0.65
0,26599: 0.57
10.91926; 0.98
10.94244; 0.98
BBSL-soft
Best NLL
0.17484: 0.44
0.08129:
0.52 0.0799;
0.04739: 0.57
180777: 0.54
0.93671: 0.24
0.50136; 0.14
BBSL
ECE
0,17423;
0,04615;
1.79819; 0.46
1.1714: 0.76
0.67683;
soft
NLL.
0.17377; 0.44
129; 0.52
1,76868; 0.54
0.92956;
0.23
0.49941;
RLLS-soft
Best ECE
0.17305; 0.56
0.0799; 0.48
0.04615; 0.43
1.75943; 0.46
1.16983; 0.77
0.6753; 0.87
Table 1.6: CIFARIO: NLL vs ECE, metric: MSE, "tweak-one" shift. Analogous to Table I.1
Shift Estimaator
Calibration Method
7000
8500
TOOKI
150500
BM
Bos NLL
30,88%; 0.
16301 0.3T
0.
XLMPE CCB
21,001 6.22
21.36 0.2"
FM
Bost ECF
26 26.33200. 0.7
26 323;
26 0.69
21.58: 0.72
21
21 0.8
0.78
21. 103, 0.79
21.00 03
Table L.7: CIFAR100: NLL vs ECE, metrie: A AcAccrrcy dirichlet shift. Analogous to Table
Calibration
Estiaattor
Method
X=-0000
655000
1-- 1000
10-000
1- 100000
70KK
NeRR00
1-100MI
FM
Best N.T.
1.113, R.31
6.112::227
0.112
I.TTI, 8.17
O.TOR: 0.19
0.107 IL
O.TING RL11
ILT64 EL8
EM
Best BCB
0,119.066
0.71
0.119;0.11
0.118: 0.83
0.117:0.9)
0.116:09
0.116: 0.97
0.114: 0.97
0.112:
BBSL-soft
Best NLL
0.226, 0.41
0.32:0.35
0,215; 0.39
0.18% 0.3
0.182; 0.31
0.177: 0.24
0.177.0.2
0,171:
0.165; 0.29
Rest PCR
0.236,0.5 0.59
0.22, 0.65
0.216:044
0.189. 0.7
0.833 0.69
0.178:0.6
0.178, 0.8
0.172: 0.8
0.166:
RLLS-soft
Best NLL
0.186,03
0.179,0:22
0.176:0.48
0.155,0.0
0.149
0:48
0144:051
0.148; 0.34
0142;
0.46
0.137,0.99
R13.S-4oft
Rest ECB
0.186,
0.179:0.55 0.58
0.176: 0.52
0.155, 0.51
0.149; 0.52
0.144:0.49
0.14%, 0.66
0.147 0.54
0.137:0.11
Table I8: CIFAR100: NLL vs ECE, metric: JS Divergence, dirichlet shift, Analogous to Table
Shift Extimater
Calibrattion
10.
Method
R-10000
057030
EMI
BESTNEL
8.93
8.0011:8366
I.35
0.13758:82222
I.1307 8.33
8.1276 8,26
0.18309:022
0,00664; 8.2
EXPAAAT RL17
EM
Best BCT
0280002 1063
0.27676 0644
0,66940
0.1822:0077
0114306;0077
201246000
0,10512,
UL10INE: 0L53
BBSL-Pot
BLLNLL
1.3
0,199773335
0.16374:02
0.EIN 6.24
6.11807 8.23
8,89960; 8.1
MINS) Heef
Best NCR
1O1066 0.64
0AMAN306 64
(6)
96
09999100.77
81 11
0.17
O. 10TIN ON
BGTNEL
SE
1L55
G.TSST 0A4
011955 0.39
TC45
B.0336 EL27
RLLS soft
Best BCB
0,50477 0.5
0,52422
0.83477
0.5225
0.15747:045
0.13843:0.5
0.12089: 0.61
0LU9T32;0055 0.55
0L087666 0.73
Table 1.9: CIFAR100: NLL vs ECE, metric: MSE, dirichlet shift. Analogous to Table
21

---

Under review as a conference paper at ICLR 2020
Shift
Calibration
0.5
e 0.9
Estimator
Method
n=500
n== 1000
==1500
2=500
== 1=100
== 1500
EM
Best NLL
0.21
4.315: 0.26
4.543; 0.19
3.548; 0.02
3.57:
3.746; 0.02
EM
Best ECE
0.79
4.099: 0.74
4.179: 0.81
2.074: 0.98
3.57: 1.0
2.405: 0.98
Table 1.10: KaggleDR: NLL vs ECE, metric: A A%Accurccy Shift strategy modifies the proportion of healthy examples. Analogous to Table L.1
Shift
Calibration Method
0,5
0.9
Estimator
==300
== 1000
== ==1300
n=500
n=T000
nc 1500
EM
Best NLLI
0.11; 0.42
0.093; 0.31
0.079; 0.33
0.078; 0.08
0.062; 0.0
0.059; 0.07
EM
Best ECE
0.104: 0.58
0.092: 0.69
0.079; 0.67
0.11: 0.92
0.062: 1.0
0.102; 0.93
BBSI-
Best NI.I.
0.166; 0.37
0.12; 0.32
0.096; 0.31
0.107; 0.24
0.079: 0.0
0.077; 0.32
BBSL-soft
Best ECE
0.158: 0.63
0.123: 0.68
0.101: 0.69
0.125: 0.76
0.079: 1.0
0.086; 0.68
RLLS-soft
Best NLL
0.128: 0.44
0.112: 0.38
0.093; 0.31
0.089; 0.38
0.079: 0.0
0.077: 0.38
RLLS-sof
Best ECE
0.123; 0.56
0.109; 0.62
0.098; 0.69
0.092; 0.62
0.079; 1.0
0.085; 0.62
Table 1.11:
KaggleDR: NLL vs ECE, metric: JS Divergence. Shift strategy modifies the proportion
of healthy examples. Analogous to Table
1.1
Shift
Calibration Method
= 0.5
p #= 0.9
Estimator
n=500
n= 1000
1==1500
n=
1=1000
n= 1==1500
EM
Best NLL
L.76,
0.46; 0.24
0.319: 0.32
0.069; 0.07
0.032: 0.0
0.029: 0.01
EM
Best BCE
1.028; 0.7
0.549; 0.76
0.354; 0.68
0.168; 0.93
0.032; 1.0
0.125; 0.99
soft
Best NLL
61.132; 0.43
1.439; 0.27
0.875; 0.29
0.747: 0.27
0.049; 0.0
0.042; 0.34
BBSL-soft
ECE
8.74; 0.57
2.181; 0.73
0,932: 0.71
2.678;
0.049: 1.0
0.067:
RLLS-soft
Best NLL
2.445; 0.44
0.859; 0.33
0.726: 0.31
0.066; 0.38
0.042: 0.0
0.035; 0.43
RLLS-soft
Best BCE
2.089; 0.56
0.867; 0.67
0.742: 0.69
0.072: 0.62
0.042; 1.0
0.046: 0.57
Table 1.12: KaggleDR: NLL vs ECE, metric: MSE. Shift strategy modifies the proportion of
healthy examples. Analogous to Table
[.1
22


Reviews:
[{"review_id": "rklbC4Gatr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the label shift problem and investigates the role of the calibration in improving the results of two different domain adaptation solutions including EM and BBSE. They show with having better estimation of conditional distribution in the source domain the final label distribution obtained by EM and BBSE in the target domain is more accurate. They conduct the experiments on three different datasets i.e. CIFAR10, CIFAR100 and Kaggel Diabetic Retinopathy as the proof of concept.\n\nOverall, I think the paper should be rejected as it suffers from not high enough level of novelty in proposed method. The paper used different variants of famous Platt Scaling family approaches as the calibration methods to improve the estimation of conditional distribution of the training data $p(y|x)$ and showed the positive influence of that on label shift approaches like EM and BBSE, which is not enough contribution. \n\nThe paper is well-written and point the interesting problem. But the way of reporting the results are not clear enough. It would be better that the accuracy is reported for the baseline, EM,  BBSE-Soft and BBSE-hard at the same table with mean and std values. The tables some how should show the impact of using the calibration to improve significantly the final label shift accuracy. But in this way of reporting the results it is not clear how big calibration can improve the final accuracy. Comparing to other baselines like RLLS [1] is also missing in this table.\n\nPointing out that ECE is not a good metric to measure the calibration is also reported before in the related research works [2]. \n\n\nReferences\n[1] Azizzadenesheli, Kamyar, et al. \"Regularized learning for domain adaptation under label shifts.\" arXiv preprint arXiv:1903.09734 (2019).\n[2] Vaicenavicius, Juozas, et al. \"Evaluating model calibration in classification.\" arXiv preprint arXiv:1902.06977 (2019).\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "S1xUndTTYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work conducted exhaustive experiments for a label shift problem with EM and BBSE over CIFAR10/100 and retinopathy detection datasets. In addition, as for model calibration, they also considered temperature and vector scaling and introduced intermediaries between those, namely, NBVS and TBVS.\n\nOverall, the paper is well organized and provides rigorous experiments with conclusions based on the empirical observations. As stated in the paper, the main contribution of this work is to explore the impact of calibration. \n\nHowever, it is still doubtful whether those empirical results can be generalized as there is no analytical and thoughtful discussions. Further, the label shift was simulated by means of dirichlet shift on both datasets. It would be great to apply the method on real cases. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "ByeZrqUZ5S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds upon recent work on detecting and correcting for label shift.\nThey explore both the BBSE algorithm analyzed in Detecting and Correcting for Label Shift (2018)\nand another approach based on EM where the predictive posteriors and test set label distributions\nare iteratively computed, each an update based on the estimate of the other.\n\nCrucially, while the former method requires only that the confusion matrix be invertible,\nthe  latter method only appears valid under strong assumptions including the calibration fo the classifier.\nThus the authors propose an approach for “bias-corrected calibration”\nand shows that bias-corrected calibration can improve the performance of BBSE and EM.\nThe method is crucial for EM and with it, the results seem to show that EM,\nin the large sample (8000 examples) regime and with good initial classifiers\n (on the relatively easy CIFAR10 task with a strong baseline)\nthat EM outperforms BBSE.\n\nThe paper is easy to follow an the authors should also be credited for releasing \ncode anonymously with which we could reproduce their results. \n\nI have as few specific concerns/questions about the paper that I would like the authors to address: \n\n * They consider JS divergence as a metric for evaluation. But they don’t consider other metrics\n  like the error in weight estimates which is considered in most of the prior work \n * They don’t compare their results with regularizations suggested on top of BBSE, particularly Azizzadenesheli et. al. https://arxiv.org/abs/1903.09734. \n * They compare methods for particularly limited ranges of Dirichlet shift (\\alpha=0.1,1.0). \n  * What happens when the \\alpha increases to have less severe shifts? \nOptimizing ELBO with EM can lead to local convergence to the likelihood function when the likelihood is not unimodal. \n *  Is this likelihood function unimodal?  Does the EM approach converges to MLE under some appropriate initialization and assumptions? \n  \nA small presentation note: many of the papers are reporting the same metric and ought to be grouped as a large table, not as many tables. Also every table should state clearly what it is reporting in the caption, not just referring to earlier tables.\n\n=======Update\nI have read the rebuttal and appreciate that the authors took the time to establish the concavity of the likelihood function for EM. Overall this paper makes an interesting contribution in establishing the usefulness of the likelihood formulation (here optimized by EM) of label shift estimation and its apparent benefits over BBSE in some settings. I am happy to keep my score despite apparent disagreement from the other reviewers. I must say that some other reviews were disappointingly lacking in thoroughness.  \n \nThe paper still leaves open some serious questions, e.g. --- why is this bias correction heuristic so effective vis-a-vis EM and is this explained by its performance at the calibration task itself? The original temperature scaling paper reported a similar heuristic yet didn't see such a benefit wrt their metrics. Why is it so useful here? \n\nStill, while this paper can be improved in some key ways, it does make an interesting contribution.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This was a borderline paper, but in the end two of the reviewers remain unconvinced by this paper in its current form, and the last reviewer is not willing to argue for acceptance. The first reviewer's comments were taken seriously in making a decision on this paper. As such, it is my suggestion that the authors revise the paper in its current form, and resubmit, addressing some of the first reviewers comments, such as discussion of utility of the methodology, and to improve the exposition such that less knowledgable reviewers understand the material presented better. The comments that the first reviewer makes about lack of motivation for parts of the presented methodology is reflected in the other reviewers comments, and I'm convinced that the authors can address this issue and make this a really awesome submission at a future conference.

On a different note, I think the authors should be congratulated on making their results reproducible. That is definitely something the field needs to see more of.
Decision: Reject