Adapting to Label Shift with Bias-Corrected Calibration

1 INTRODUCTION . Imagine we train a classifier in country A to predict whether or not a person has a disease based on observed symptoms , and that we hope to deploy this classifier in country B , which has poorer access to healthcare . If the prevalence of the disease in country B is higher in than in country A , the classifier might systematically misdiagnose people as not having the disease . How can we adapt the classifier to cope with the difference in the baseline prevalence of the disease in the two countries ? Formally , let y denote our labels ( e.g . whether or not a person is diseased ) , and let x denote the observed symptoms . Let us denote the joint distribution ( x , y ) in country A ( our “ source ” domain ) as P , and let us denote the distribution in country B ( our “ target ” domain , where we do not have labels ) as Q . How can we adapt a classifier trained to estimate p ( y|x ) ( the conditional probability in distribution P ) so that it can instead estimate q ( y|x ) ( the conditional probability in distribution Q ) ? Absent assumptions about the nature of the shift between P and Q , this problem is intractable . However , if the disease generates similar symptoms in both countries , we can assume that p ( x|y ) = q ( x|y ) , and that the shift in the joint distribution q ( x , y ) is due to a shift in the label proportion q ( y ) . Formally , we assume that q ( x , y ) = p ( x|y ) q ( y ) . This is known as label shift or prior probability shift ( Amos , 2008 ) , and it corresponds to anti-causal learning ( i.e . predicting the cause y from its effects x ) ( Schoelkopf et al. , 2012 ) . Anti-causal learning is appropriate for diagnosing diseases given observations of symptoms because diseases cause symptoms . Given estimates of p ( y ) and p ( y|x ) , Saerens et al . ( 2002 ) proposed a simple Expectation Maximization ( EM ) procedure to estimate q ( y ) without needing to estimate p ( x|y ) . However , estimates of p ( y|x ) derived from modern neural networks are often poorly calibrated ( Guo et al. , 2017 ) , and the lack of calibration can decrease the effectiveness of EM . As an alternative , Lipton et al . ( 2018 ) developed a technique called Black Box Shift Learning ( BBSL ) that can work even when the predictions p ( x|y ) are not calibrated . Azizzadenesheli et al . ( 2019 ) further improved upon BBSL in a technique known as Regularized Learning under Label Shifts ( RLLS ) . Both BBSL and RLLS leverage information in a confusion matrix calculated on a held-out portion of the training set . To our knowledge , neither BBSL and RLLS have been benchmarked against the EM approach . Although the EM approach is limited by the assumption that the predictions p ( y|x ) are calibrated , a number of recent techniques have been proposed to correct for miscalibration of p ( y|x ) using a held-out portion of the training set ( Guo et al. , 2017 ) . The held-out set can be thought of as analogous to the held-out set used in BBSL and RLLS to calculate a confusion matrix . This suggests a simple yet novel hybrid algorithm for adapting to label shift : first , calibrate predictions using the held-out training set , then perform domain adaptation on the calibrated predictions using EM . In this work , we studied the effectiveness of this hybrid algorithm . More generally , we studied the impact of calibration on domain adaptation to label shift . Our primary contributions are as follows : 1.1 OUR CONTRIBUTIONS . 1 . In experiments on CIFAR10 , CIFAR100 , MNIST and Diabetic Retinopathy Detection , we found that EM achieves state-of-the-art results when used with an appropriate calibration approach . Although BBSL and RLLS both benefit from calibration , they did not tend to outperform EM when the probabilities were well-calibrated . 2 . We observed that the popular calibration approach of Temperature Scaling ( TS ) ( Guo et al. , 2017 ) does not tend to achieve state-of-the art results in the context of adaptation to label shift , possibly owing to large systematic biases in the calibrated probabilities ( Fig . 1 ) . The best results are obtained using variants of TS that contain class-specific bias capable of correcting for systematic bias . 3 . We identify a theoretically-grounded strategy for computing the source-domain priors in EM-based domain adaptation that can be critically important when calibrated probability estimates have systematic bias . Empirically , we find that computing source priors in this way substantially improves the robustness of EM to the presence of systematic biases in the class probabilities . We also prove that the likelihood function of EM is concave and unimodal ; thus , EM converges to the maximum likelihood estimate . 2 BACKGROUND . 2.1 TEMPERATURE SCALING , VECTOR SCALING AND EXPECTED CALIBRATION ERROR . Calibration has a long history in the machine learning literature ( DeGroot and Fienberg , 1983 ; Platt , 1999 ; Zadrozny and Elkan ; 2002 ; Niculescu-Mizil and Caruana , 2005 ; Kuleshov and Liang , 2015 ; Naeini et al. , 2015 ; Kuleshov and Ermon , 2016 ) . In the context of modern neural networks , Guo et al . ( 2017 ) showed that Temperature Scaling , a single-parameter variant of Platt Scaling ( Platt , 1999 ) , was effective at reducing miscalibration . Temperature scaling performs calibration by introducing a temperature parameter T to the logit vector of the softmax . Let z ( xk ) represent a vector of the original softmax logits for example xk , and let yi be a random variable representing the label for class i . With temperature scaling , we have p ( yi|xk ) = e z ( xk ) i/T∑ j e z ( xk ) j/T , where T is optimized with respect to the Negative Log Likelihood ( NLL ) on a held-out portion of the training set , such as the validation set . Guo et al . ( 2017 ) compared TS to an approach defined as Vector Scaling ( VS ) , where a different scaling parameter was used for each class along with class-specfic bias parameters . Formally , in vector scaling , p ( yi|xk ) = e ( z ( xk ) iWi ) +bi∑ j e ( z ( xk ) jWj ) +bj . Guo et al . ( 2017 ) found that vector scaling had a tendency to perform slightly worse than TS as measured by a metric known as the Expected Calibration Error ( Naeini et al. , 2015 ) . To compute the ECE , the predicted probabilities for the output class are partitioned into M equally spaced bins , and the weighted average of the difference between the bin ’ s accuracy and the bin ’ s confidence is computed , where the weights are determined by the proportion of examples falling in the bin . Formally , ECE = ∑M m=1 |Bm| n |acc ( Bm ) − conf ( Bm ) | , where n is the number of samples . 2.2 LABEL SHIFT ADAPTATION VIA EXPECTATION MAXIMIZATION . In a seminal paper on label shift adaptation , Saerens et al . ( 2002 ) proposed an EM algorithm for estimating the shift in the class priors between the training and test distributions . Let q̂ ( s ) ( y = i ) denote the estimate ( from EM iteration s ) of the prior probability q ( y = i ) of observing class i in the test set . The algorithm proceeds as follows : first , q̂ ( 0 ) ( y = i ) is initialized to be equal to the class priors p̂ ( y = i ) estimated from the training set . Then , the conditional probabilities in the E-step are computed as q̂ ( s ) ( y = i|xk ) = q̂ ( s ) ( y=i ) p̂ ( y=i ) p̂ ( y=i|xk ) ∑n j=1 q̂ ( s ) ( y=j ) p̂ ( y=j ) p̂ ( y=j|xk ) . Finally , the prior estimates in the M-step are updated as q̂ ( s+1 ) ( y = i ) = 1N ∑N k=1 q̂ ( s ) ( y = i|xk ) , where N is the number of examples in the testing set . The E and M steps are iterated until convergence . As there is no need to estimate p ( x|y ) in any step of the EM procedure , the algorithm can scale to high-dimensional datasets . Note this procedure assumes the conditional probability estimates p̂ ( y = i|xk ) are calibrated . 2.3 LABEL SHIFT ADAPTATION VIA BLACK BOX SHIFT LEARNING AND REGULARIZED LEARNING UNDER LABEL SHIFTS . Following the EM approach of Saerens et al . ( 2002 ) , several additional approaches for labels shift adaptation have emerged ( Chan and Ng ; Storkey ; Schoelkopf et al. , 2012 ; Zhang et al. , 2013 ; Lipton et al. , 2018 ; Azizzadenesheli et al. , 2019 ) . Many of these approaches build estimates p ( x|y ) , which can scale poorly with dataset sizes and underperform on high-dimensional data ( Lipton et al. , 2018 ) . Lipton et al . ( 2018 ) proposed Black-Box Shift Learning ( BBSL ) , which strives to efficiently estimate the weights [ w ] i = q ( y=i ) p ( y=i ) even in cases where the prediction model p̂ ( y = i|xk ) is poorly calibrated or biased . BBSL proceeds as follows : let f be a function that accepts an input and returns the model ’ s predicted class , xk denote an example from a held-out portion of the training set , and x′k denote an example from the testing set . The empirical estimate of w , denoted as ŵ , is computed as ŵ = Ĉ−1ŷ , yûŷ , where [ ûŷ ] i = ∑ k 1 { f ( x ′ k ) =i } m and [ Ĉŷ , y ] ij = 1 n ∑ k 1 { f ( xk ) = i and yk = j } . Because the approach above is not guaranteed to produce positive values for all elements of ŵ , any negative elements of ŵ are set to 0 after they are estimated . Domain adaptation is then performed by retraining the model on the entire training set distribution with examples upweighted in accordance with ŵ . Lipton et al . ( 2018 ) denote the version of BBSL described above as BBSL-hard . They also compare to a variant that they call BBSL-soft , which they describe as the case where where f outputs probabilities rather than hard classes . We interpreted this to mean [ ûŷ ] i = ∑ k f ( x ′ k ) i m and [ Ĉŷ , y ] ij = 1 n ∑ k f ( xk ) i1 { yk = j } . Azizzadenesheli et al . ( 2019 ) further improved upon BBSL by including regularization terms in a technique known as Regularized Learning under Label Shift ( RLLS ) . In our experiments , we compare to BBSL-hard , BBSL-soft , RLLS-hard and RLLS-soft . Regularization parameters for RLLS were set in accordance with the hard-coded values present in the publicly available code provided by the authors at https : //github.com/Angela0428/labelshift/blob/ 5bbe517938f4e3f5bd14c2c105de973dcc2e0917/label_shift.py # L453-L456 . Note that BBSL and RLLS both require a portion of the training set to be held out during the initial training phase in order to accurately estimate the confusion matrix Ĉŷ , y ; in our experiments involving calibration , we use this same heldout set to calibrate the model .

Summary: The paper considers the label shift problem and investigates the role of the calibration in improving the results of two different domain adaptation solutions including EM and BBSE. They show with having better estimation of conditional distribution in the source domain the final label distribution obtained by EM and BBSE in the target domain is more accurate. They conduct the experiments on three different datasets i.e. CIFAR10, CIFAR100 and Kaggel Diabetic Retinopathy as the proof of concept.