Privacy Preserving Recalibration under Domain Shift

1 INTRODUCTION . Machine learning classifiers are currently deployed in high stakes applications where ( 1 ) the cost of failure is high , so prediction uncertainty must be accurately calibrated ( 2 ) the test distribution does not match the training distribution , and ( 3 ) data is subject to privacy constraints . All three of these challenges must be addressed in applications such as medical diagnosis ( Khan et al. , 2001 ; Chen et al. , 2018 ; Kortum et al. , 2018 ) , financial decision making ( Berestycki et al. , 2002 ; Rasekhschaffe & Jones , 2019 ; He & Antón , 2003 ) , security and surveillance systems ( Sun et al. , 2015 ; Patel et al. , 2015 ; Agre , 1994 ) , criminal justice ( Berk , 2012 ; 2019 ; Rudin & Ustun , 2018 ) , and mass market autonomous driving ( Kendall & Gal , 2017 ; Yang et al. , 2018 ; Glancy , 2012 ) . While much prior work has addressed these challenges individually , they have not been considered simultaneously . The goal of this paper is to propose a framework that formalizes challenges ( 1 ) - ( 3 ) jointly , introduce benchmark problems , and design and compare new algorithms under the framework . A standard approach for addressing challenge ( 1 ) is uncertainty quantification , where the classifier outputs its confidence in every prediction to indicate how likely it is that the prediction is correct . These confidence scores must be meaningful and trustworthy . A widely used criterion for good confidence scores is calibration ( Brier , 1950 ; Cesa-Bianchi & Lugosi , 2006 ; Guo et al. , 2017 ) — i.e . among the data samples for which the classifier outputs confidence p ∈ ( 0 , 1 ) , exactly p fraction of the samples should be classified correctly . Several methods ( Guo et al. , 2017 ) learn calibrated classifiers when the training distribution matches the test distribution . However , this classical assumption is always violated in real world applications , and calibration performance can significantly degrade under even small domain shifts ( Snoek et al. , 2019 ) . To address this challenge , several methods have been proposed to re-calibrate a classifier on data from the test distribution ( Platt et al. , 1999 ; Guo et al. , 2017 ; Kuleshov et al. , 2018 ; Snoek et al. , 2019 ) . These methods make small adjustments to the classifier to minimize calibration error on a validation dataset drawn from the test distribution , but they are typically only applicable when they have ( unrestricted ) access to data from this validation set . Additionally , high stakes applications often require privacy . For example , it is difficult for hospitals to share patient data with machine learning providers due to legal privacy protections ( Centers for Medicare & Medicaid Services , 1996 ) . When the data is particularly sensitive , provable differential privacy becomes necessary . Differential privacy ( Dwork et al. , 2014 ) provides a mathematically rigorous definition of privacy along with algorithms that meet the requirements of this definition . For instance , the hospital may share only certain statistics of their data , where the shared statistics must have bounded mutual information with respect to individual patients . The machine learning provider can then use these shared statistics — possibly combining statistics from many different hospitals — to recalibrate the classifier and provide better confidence estimates . In this paper , we present a framework to address all three challenges – calibration , domain shift , and differential privacy – and introduce a benchmark to standardize performance and compare algorithms . We show how to modify modern recalibration techniques ( e.g . ( Zadrozny & Elkan , 2001 ; Guo et al. , 2017 ) ) to satisfy differential privacy using this framework , and compare their empirical performance . This framework can be viewed as performing federated learning for recalibration , with the constraint that each party ’ s data must be kept differentially private . We also present a novel recalibration technique , accuracy temperature scaling , that is particularly effective in this framework . This new technique requires private data sources to share only two statistics : the overall accuracy and the average confidence score for a classifier . We adjust the classifier until the average confidence equals the overall accuracy . Because only two numbers are revealed by each private data source , it is much easier to satisfy differential privacy . In our experiments , we find that without privacy requirements the new recalibration algorithm performs on par with algorithms that use the entire validation dataset , such as ( Guo et al. , 2017 ) ; with privacy requirements the new algorithm performs 2x better than the second best baseline . In summary , the contributions of our paper are as follows . ( 1 ) We introduce the problem of “ privacy preserving calibration under domain shift ” and design a framework for adapting existing recalibration techniques to this setting . ( 2 ) We introduce accuracy temperature scaling , a novel recalibration method designed with privacy concerns in mind , that requires only the overall accuracy and average confidence of the model on the validation set . ( 3 ) Using our framework , we empirically evaluate our method on a large set of benchmarks against state-of-the-art techniques and show that it performs well across a wide range of situations under differential privacy . 2 BACKGROUND AND RELATED WORK . 2.1 CALIBRATION . Description of Calibration Consider a classification task from input domain ( e.g . images ) X to a finite set of labels Y = { 1 , · · · , m } . We assume that there is some joint distribution P ∗ on X × Y . This could be the training distribution , or the distribution from which we draw test data . A classifier is a pair ( φ , p̂ ) where φ : X → Y maps each input x ∈ X to a label y ∈ Y and p̂ : X → [ 0 , 1 ] maps each input x to a confidence value c. We say that the classifier ( φ , p̂ ) is perfectly calibrated ( Brier , 1950 ; Gneiting et al. , 2007 ) with respect to the distribution P ∗ if ∀c ∈ [ 0 , 1 ] Pr P∗ ( x , y ) [ φ ( x ) = y | p̂ ( x ) = c ] = c. ( 1 ) Note that calibration is a property not only of the classifier ( φ , p̂ ) , but also of the distribution P ∗ . A classifier ( φ , p̂ ) can be calibrated with respect to one distribution ( e.g . the training distribution ) but not another ( e.g . the test distribution ) . To simplify notation we drop the dependency on P ∗ . To numerically measure how well a classifier is calibrated , the commonly used metric is Expected Calibration Error ( ECE ) ( Naeini et al. , 2015 ) defined by ECE ( φ , p̂ ) : = ∫ c∈ [ 0,1 ] Pr [ p̂ ( x ) = c ] · |Pr [ φ ( x ) = y | p̂ ( x ) = c ] − c| . ( 2 ) In other words , ECE measures average deviation from Eq . 1 . In practice , the ECE is approximated by binning — partitioning the predicted confidences into bins , and then taking a weighted average of the difference between the accuracy and average confidence for each bin ( see Appendix A.1 for details . ) Recalibration Methods Several methods apply a post-training adjustment to a classifier ( φ , p̂ ) to achieve calibration ( Platt et al. , 1999 ; Niculescu-Mizil & Caruana , 2005 ) . The one most relevant to our paper is temperature scaling ( Guo et al. , 2017 ) . On each input x ∈ X , a neural network typically first computes a logit score l1 ( x ) , l2 ( x ) , · · · , ln ( x ) for each of the n labels , then computes a confidence score or probability estimate p̂ ( x ) with a softmax function . Temperature scaling adds a temperature parameter T ∈ R+ to the softmax function p̂ ( x ; T ) = max i eli ( x ) /T∑ j e lj ( x ) /T . ( 3 ) A higher temperature reduces the confidence , and vice versa . T is trained to minimize the standard cross entropy objective on the validation dataset , which is equivalent to maximizing log likelihood . Despite its simplicity , temperature scaling performs well empirically in classification calibration for deep neural networks . Alternative methods for classification calibration have also been proposed . Histogram binning ( Zadrozny & Elkan , 2001 ) partitions confidence scores ∈ [ 0 , 1 ] into bins { [ 0 , ) , [ , 2 ) , · · · , [ 1− , 1 ] } and sorts each validation sample into a bin based on its confidence p̂ ( x ) . The algorithm then resets the confidence level of each bin to match the average classification accuracy of data points in that bin . Isotonic regression methods ( Kuleshov et al. , 2018 ) learn an additional layer on top of the softmax output layer . This additional layer is trained on a validation dataset to fit the output confidence scores to the empirical probabilities in each bin . Other methods include Platt scaling ( Platt et al. , 1999 ) and Gaussian process calibration ( Wenger et al. , 2019 ) . 2.2 ROBUSTNESS TO DOMAIN SHIFT . Preventing massive performance degradation of machine learning models under domain shift has been a long-standing problem . There are several approaches developed in the literature . Unsupervised domain adaptation ( Ganin & Lempitsky , 2014 ; Shu et al. , 2018 ) learns a joint representation between the source domain ( original data ) and target domain ( domain shifted data ) . Invariance based methods ( Cissé et al. , 2017 ; Miyato et al. , 2018 ; Madry et al. , 2017 ; Lakshminarayanan et al. , 2017 ; Cohen et al. , 2019 ) prevent the classifier output from changing significantly given small perturbations to the input . Transfer learning methods ( Pan & Yang , 2009 ; Bengio , 2012 ; Dai et al. , 2007 ) fine-tune the classifier on labeled data in the target domain . We classify our method in this category because we also fine-tune on the target domain , but with minimal data requirements ( we only need the overall classifier accuracy ) . 2.3 DIFFERENTIAL PRIVACY . Differential privacy ( Dwork et al. , 2014 ) is a procedure for sharing information about a dataset to the public while withholding critical information about individuals in the dataset . Informally , it guarantees that an attacker can only learn a limited amount of new information about an individual . Differentially private approaches are critical in privacy sensitive applications . For example , a hospital may wish to gain medical insight or calibrate its prediction models by releasing diagnostic information to outside experts , but it can not release information about any particular patient . One common notion of differential privacy is -differential privacy ( Dwork et al. , 2014 ) . Let us define a database D as a collection of data points in a universe X , and represent it by its histogram : D ∈ N|X | , where each entry Dx represents the number of elements in the database that takes the value x ∈ X . A randomized algorithmM is one that takes in input D ∈ N|X | and ( stochastically ) outputs some valueM ( D ) = b for b ∈ Range ( M ) . Definition 1 . Let M be a randomized function M : N|X | → Range ( M ) . We say that M is - differentially private if for all S ⊆ Range ( M ) and for any two databases D , D′ ∈ N|X | that differ by only one element , i.e . ‖D −D′‖1 ≤ 1 , we have Pr [ M ( D ) ∈ S ] Pr [ M ( D′ ) ∈ S ] ≤ e Intuitively , the output ofM should not change much if a single data point is added or removed . An attacker that learns the output ofM gains only limited information about any particular data point . Given a deterministic real valued function f : N|X | → Rz , we would like to design a functionM that remains as close as possible to f but satisfies Definition 1 . This can be achieved by the Laplace mechanism ( McSherry & Talwar , 2007 ; Dwork , 2008 ) . Let us define the L1 sensitivity of f : ∆f = max D , D′∈N|X| ‖D−D′‖1=1 ‖f ( D ) − f ( D′ ) ‖1 Then the Laplace mechanism adds Laplacian random noise as in ( 4 ) : ML ( D ; f , ) = f ( D ) + ( Y1 , . . . , Yz ) ( 4 ) where Yi are i.i.d . random variables drawn from the Laplace ( loc = 0 , scale = ∆f/ ) distribution . The functionML satisfies -differential privacy , and we reproduce the proof in Appendix A.2 .

Summary: The paper studies the problem of classifier recalibration under differential privacy constraints. They propose a framework with a calibrator and several private data sources, and it works as follows. At each iteration, the calibrator queries each source, and the data source sends back the private answer, which will be used to optimize the calibration. They also provide a recalibration technique, accuracy temperature scaling, which is effective under the privacy constraint for the reason of low sensitivity. Rigorous experimental results are provided.