Privacy Preserving Recalibration under Domain Shift
Authors: Rachel Luo
Shengjia Zhao
Jiaming Song
Jonathan Kuck
Stefano Ermon
Silvio Savarese
Venue:  (2021)
Keywords: uncertainty calibration
differential privacy

Abstract: Classifiers deployed in high-stakes applications must output calibrated confidence scores, i.e. their predicted probabilities should reflect empirical frequencies. Typically this is achieved with recalibration algorithms that adjust probability estimates based on the real-world data; however, existing algorithms are not applicable in real-world situations where the test data follows a different distribution from the training data, and privacy preservation is paramount (e.g. protecting patient records). We introduce a framework that provides abstractions for performing recalibration under differential privacy constraints. This framework allows us to adapt existing recalibration algorithms to satisfy differential privacy while remaining effective for domain-shift situations. Guided by our framework, we also design a novel recalibration algorithm, accuracy temperature scaling, that is tailored to the requirements of differential privacy. In an extensive empirical study, we find that our algorithm improves calibration on domain-shift benchmarks under the constraints of differential privacy. On the 15 highest severity perturbations of the ImageNet-C dataset, our method achieves a median ECE of 0.029, over 2x better than the next best recalibration method and almost 5x better than without recalibration.

Under review as a conference paper at ICLR 2021
PRIVACY PRESERVING RECALIBRATION UNDER DOMAIN SHIFT
Anonymous authors Paper under double-blind review
ABSTRACT
Classifiers deployed in high-stakes applications must output calibrated confidence scores, i.e. their predicted probabilities should reflect empirical frequencies. Typi- cally this is achieved with recalibration algorithms that adjust probability estimates based on real-world data; however, existing algorithms are not applicable in real- world situations where the test data follows different distribution from the training data, and privacy preservation is paramount (e.g. protecting patient records). We introduce a framework that provides abstractions for performing recalibration un- der differential privacy constraints. This framework allows us to adapt existing recalibration algorithms to satisfy differential privacy while remaining effective for domain-shift situations. Guided by our framework, we also design novel recalibration algorithm, accuracy temperature scaling, that is tailored to the re- quirements of differential privacy. In an extensive empirical study, we find that our algorithm improves calibration on domain-shift benchmarks under the constraints of differential privacy. On the 15 highest severity perturbations of the ImageNet-C dataset, our method achieves a median ECE of 0.029, over 2x better than the next best recalibration method and almost 5x better than without recalibration.
INTRODUCTION
Machine learning classifiers are currently deployed in high stakes applications where (1) the cost of failure is high, so prediction uncertainty must be accurately calibrated (2) the test distribution does not match the training distribution, and (3) data is subject to privacy constraints. All three of these challenges must be addressed in applications such as medical diagnosis (Khan et al., 2001; Chen al., 2018; Kortum et al., 2018), financial decision making (Berestycki et al., 2002; Rasekhschaffe & Jones, 2019; He & Antón, 2003), security and surveillance systems (Sun et al., 2015; Patel et al., 2015; Agre, 1994), criminal justice (Berk, 2012; 2019; Rudin & Ustun, 2018), and mass market autonomous driving (Kendall & Gal, 2017; Yang et al., 2018; Glancy, 2012). While much prior work has addressed these challenges individually, they have not been considered simultaneously. The goal of this paper is to propose framework that formalizes challenges (1)-(3) jointly, introduce benchmark problems, and design and compare new algorithms under the framework.
standard approach for addressing challenge I) is uncertainty quantification, where the classifier outputs its confidence in every prediction to indicate how likely it is that the prediction is correct. These confidence scores must be meaningful and trustworthy. widely used criterion for good confidence scores is calibration (Brier, 1950; Aasa-Bianchi & Lugosi, 2006; Guo et al., 2017) - i.c. among the data samples for which the classifier outputs confidence p E (0, 1), exactly p fraction of the samples should be classified correctly.
Several methods (Guo et al., 2017) learn calibrated classifiers when the training distribution matches the test distribution. However, this classical assumption is always violated in real world applications, and calibration performance can significantly degrade under even small domain shifts (Snoek et al., 2019). To address this challenge, several methods have been proposed to re-calibrate classifier on data from the test distribution (Platt et al., 1999; Guo et al., 2017; Kuleshov et al., 2018; Snoek et al., 2019). These methods make small adjustments the classifier to minimize calibration error validation dataset drawn from the test distribution, but they are typically only applicable when they have (unrestricted) access to data from this validation set.
Additionally, high stakes applications often require privacy. For example, it is difficult for hospitals to share patient data with machine learning providers due to legal privacy protections (Centers for

---

Under review as a conference paper at ICLR 2021
Medicare & Medicaid Services, 1996). When the data is particularly sensitive, provable differential privacy becomes necessary. Differential privacy (Dwork et al., 2014) provides a mathematically rigorous definition of privacy along with algorithms that meet the requirements of this definition. For instance, the hospital may share only certain statistics of their data, where the shared statistics must have bounded mutual information with respect to individual patients. The machine learning provider can then use these shared statistics - possibly combining statistics from many different hospitals recalibrate the classifier and provide better confidence estimates.
In this paper, we present a framework to address all three challenges calibration, domain shift, and differential privacy and introduce benchmark to standardize performance and compare algorithms. We show how to modify modern recalibration techniques (e.g. (Zadrozny & Elkan, 2001; Guo et al., 2017)) to satisfy differential privacy using this framework, and compare their empirical performance. This framework can be viewed as performing federated learning for recalibration, with the constraint that each party's s data must be kept differentially private.
We also present novel recalibration technique, accuracy temperature scaling, that is particularly effective in this framework. This new technique requires private data sources to share only two statistics: the overall accuracy and the average confidence score for classifier. We adjust the classifier until the average confidence equals the overall accuracy. Because only two numbers are revealed by each private data source, it is much casier to satisfy differential privacy. In our experiments, we find that without privacy requirements the new recalibration algorithm performs on par with algorithms that use the entire validation dataset, such as (Guo al., 2017); with privacy requirements the new algorithm performs 2x better than the second best baseline.
In summary, the contributions of our paper are as follows. (1) We introduce the problem of "privacy preserving calibration under domain shift" and design a framework for adapting existing recalibration techniques to this setting.
(2) We introduce accuracy temperature scaling, a novel recalibration method designed with privacy concerns in mind, that requires only the overall accuracy and average confidence of the model on the validation set. (3) Using our framework, we empirically evaluate our method on a large set of benchmarks against tlate-of-the--ee art techniques and show that it performs well across a wide range of situations under differential privacy.
2 BACKGROUND AND RELATED WORK
2.1 CALIBRATION
Description of Calibration Consider a classification task from input domain (c.g. images) X' to a finite set of labels y = {1.... m.. We assume that there is some joint distribution Ps on X x V. This could be the training distribution, or the distribution from which we draw test data. A classifier is a pair (ø. P) where á X' y' maps each input & E X to label y y and ő X - [0, 1] maps each input x to a confidence value c. We say that the classifier (Ø. p) is perfectly calibrated (Brier, 1950; Gneiting al., 2007) with respect the distribution Pa if Vc [0, 1]
Pr (Ó(=) y ((x) c) =c (=(20)
Note that calibration is property not only of the classifier (ø. P), but also of the distribution P*, classifier (ő. p) can be calibrated with respect to one distribution (e.g. the training distribution) but not another (e.g. the test distribution). To simplify notation we drop the dependency on P*
To numerically measure how well classifier is calibrated, the commonly used metric is Expected Calibration Error (ECE) (Naeini et al., 2015) defined by
ECE(
Pr[p(>) = c) |Pr((()) y p(x)) c - cl-
In other words, ECE measures average deviation from Eq. 1. In practice, the ECE is approximated by binning partitioning the predicted confidences into bins, and then taking a weighted average of the difference between the accuracy and average confidence for each bin (see Appendix A.1 for details.)
Recalibration Methods Several methods apply post-training adjustment to classifier (ø, p) to achieve calibration (Platt et al., 1999; Niculescu-Mizz & Caruana, 2005). The one most relevant to paper is temperature scaling (Guo et al., 2017). On each input E X, neural network
2

---

Under review as a conference paper at ICLR 2021
typically first computes logit score (1x2), (2)).... (x) for each of the labels, then computes a confidence score or probability estimate p(z) with softmax function. Temperature scaling adds a temperature parameter T IR t to the softmax function
dda)IT
p(:; T) max Σ, e
A higher temperature reduces the confidence, and vice versa. 7 is trained minimize the standard cross entropy objective on the validation dataset, which is equivalent to maximizing log likelihood. Despite its simplicity, temperature scaling performs well empirically in classification calibration for deep neural networks.
Alternative methods for classification calibration have also been proposed. Histogram bin- ning (Zadrozny & Elkan, 2001) partitions confidence scores E [0, 1] into bins {00, e), [e. 2c).... [1- e, {]1] and sorts each validation sample into bin based its confidence p(x). The algorithm then resets the confidence level of each bin to match the average classification accuracy of data points in that bin. Isotonic regression methods (Kuleshov al., 2018) learn an additional layer on top of the softmax output layer. This additional layer is trained on a validation dataset to fit the output confidence scores the empirical probabilities in each bin. Other methods include Platt scaling (Platt et al., 1999) and Gaussian process calibration (Wenger et al., 2019).
2.2 ROBUSTNESS TO DOMAIN SHIFT
Preventing massive performance degradation of machine learning models under domain shift has been a long- -oag-standing problem. There are several approaches developed in the literature. Unsuper- vised domain adaptation (Ganin & Lempitsky, 2014; Shu et al., 2018) learns joint representation between the source domain (original data) and target domain (domain shifted data). Invariance based methods (Cissé et al., 2017; Miyato et al., 2018; Madry et al., 2017; Lakshminarayanan et al., 2017; Cohen et al., 2019) prevent the classifier output from changing significantly given small perturbations to the input. Transfer learning methods (Pan & Yang, 2009; Bengio, 2012; Dai et al., 2007) fine-tund the classifier on labeled data in the target domain. We classify our method this category because also fine-tune the target domain, but with minimal data requirements (we only need the overall classifier accuracy).
2.3 DIFFERENTIAL PRIVACY
Differential privacy (Dwork et al., 2014) is procedure for sharing information about a dataset to the public while withholding critical information about individuals in the dataset. Informally, it guarantees that an attacker can only learn limited amount of new information about an individual. Differentially private approaches are critical in privacy sensitive applications. For example, hospital may wish to gain medical insight or calibrate its prediction models by releasing diagnostic information outside experts, but cannot release information about any particular patient.
One common notion of differential privacy is e-differential privacy (Dwork et al., 2014). Let us define a database D as a collection of data points in a universe X, and represent by its histogram: D NI*I, where each entry D, represents the number of elements in the database that takes the value E X. A randomized algorithm M is one that takes in input D E N and (stochastically) outputs some value M(D) = for b E Range(M).
Definition 1. Let M be randomized function M N - Range(M).. We say that M is differentially private for all S C Range(M)) and for any two databases D, D' E NMx that differ by only one element, i.e. [|D - D'|l1 1, we have
Pr[M(D) E S]
Se
Pr[M(D') E S]
Intuitively, the output of M should not change much single data point added or removed. An attacker that learns the output of M gains only limited information about any particular data point.
Given a deterministic real valued function f that remains close possible to f but satisfies Definition 1. This be achieved by the Laplace
x + R we would like to design a function M
3

---

Under review as a conference paper at ICLR 2021
mechanism (McSherry & Talwar, 2007; Dwork, 2008). Let us define the L1 sensitivity of
Af- max D. D'D' |D-
IIf(D)
1(D))lz
Then the Laplace mechanism adds Laplacian random noise as in (4):
ML(D:f,c) f(D) + (YI....,Y.))
(4)
where Y are i.idd. random variables drawn from the Laplace(loc 0, scale Af/e) distribution. The function Mr satisfies e-differential privacy, and we reproduce the proof in Appendix A.2.
RECALIBRATION UNDER DIFFERENTIAL PRIVACY
In this section we propose framework for performing recalibration that allows independent parties to pool their data for improved calibration, while maintaining differential privacy. This setup be framed as differentially private federated learning for recalibration. Multiple parties experience the same domain shift (e.g. because they live in the same changing world). Each party would benefit from access to additional data, but each party also wants to keep their own data private. Our framework allows all parties to react domain shifts more quickly by pooling their data (so each individual party needs less labeled data from the new distribution), while maintaining the privacy of each party.
3.1 EXAMPLE APPLICATIONS
We begin with example scenarios that illustrate the main desiderata and challenges of this problem.
Suppose you have a classifier for diagnosing a medical condition and deploy your classifier across many hospitals. The hospitals need calibrated confidences for a similar but more unusual condition (e.g. the original model may have been trained on an already existing virus strain but need to be recalibrated for novel strain of the virus). There are two options: 1. Each hospital uses only their own private data to calibrate the classifier; 2. Each hospital sends some (differentially private) information you, and you aggregate the information and calibrate the classifier. Option is preferable if each hospital has only a handful patients for the particular condition.
Example 1:
In this case, the hospitals are the parties that wish to keep their data (patient info) private. The novel strain of the virus represents a domain shift. If the hospitals each have only a few data points, they want to aggregate their data in order to improve their classifier's 's calibration while still respecting patient privacy.
Example 2: Suppose that there is third- parttpaattyyyy advertising company that runs ads for websites. This advertising company has worked with news websites before, but recently acquired new clients from different category of websites. The individual websites have user information, but they cannot provide the third-party advertising company with this user information due to privacy constraints. The third-party advertising company wants calibrated models for whether user will click on an ad.
Example 3: Another category of scenarios in which our framework can be used is for individual privacy. An individual may have labeled data that he wishes keep private; however, he would still like calibrated confidences from prediction models (e.g. financial software for individuals). With differential privacy, individuals can provide summary statistics with added noise to an aggregator. In this setup, differential privacy is guaranteed on the individual level. Aggregators can then improve their confidence estimation using noisy summary statistics from many individuals.
3.2 GENERAL FRAMEWORK
We propose a standard framework to handle the general situation represented by the examples above. This two-party framework involves (1) calibrator and (2) private data sources, and it allows to adapt recalibration algorithms for differential privacy. private data source may be e.g. a hospital (as in Example 1 above), website (as in Example 2), individual (as in Example 3).
1. [Calibrator:] Input an uncalibrated classifier (o. p).
2. [Private Data Sources:] Each data source i 1,. .. d inputs private dataset D,,
4

---

Under review as a conference paper at ICLR 2021
3. At iteration A: 1,
(a) [Calibrator:] The calibrator designs function f* N + I where E N. For each i = 1, d, the calibrator sends function f to private data source i.
(b) [Private Data Sources:] For each = 1, 1. d. the i-th private data source uses the Laplace mechanism Eq. 4 to convert f to M² that satisfies c/ K-ddifferential privacy, and sends .M (D;) back to the calibrator.
4. [Calibrator:] Output a new classifier (²) based on kD),, =
K 1, .d. )
Under this framework, differential privacy is automatically satisfied: if for each A: = 1,... K, M is -'K-diffrrentially private, then the combined function (M' ( is e-differentially private (Dwork al., 2014). The differential privacy guarantees for each private data source i are independent of the policy of the calibrator or other private data sources; i.e. even the calibrator and all other private data sources collude steal information from the i-th data source i-th private data source follows the protocol, its data will be protected by differential privacy.
long the
This framework simplifies the problem into design choices: select the query function f for k == 1,... K, and select the mapping from observations (D)),, (Da) at k th=1, 1,..., K to the calibrated confidence function f/. We will discuss the most reasonable choices for several existing recalibration algorithms, Note that in general, the calibration quality degrades as the privacy level increases (i.e. decreases).
3.3 ADAPTING EXISTING ALGORIHHMM
In this section, we explain how we adapt algorithms introduced Section to our framework. Note that many existing recalibration algorithms involve parametric optimization, and in these cases multiple iterations K are needed to search the parameter space. However, using additional iterations hurts the calibration since larger K increases the added Laplace noise for e/ -ddifferential privacy; ie. for any fixed Laplace noise, more queries means less privacy. Thus, we propose the use of the better alternative to grid search for parametric optimization,
golden section search algorithm since it is more efficient at finding the extremum a unimodal function within a specified interval, and requires fewer queries. See Appendix C.1 for additional details about the golden section search.
Temperature Scaling Temperature scaling finds the temperature T in Eq. 3 that maximizes log likelihood. At each iteration k 1, 1,... K, the function f* queries D, for the log likelihood at some temperature, and we average the log likelihood over all the private datasets. We observe that log likelihood unimodal function of the temperature in Proposition (see Appendix B for proof). Therefore, the golden section search algorithm can find the maximum of the unimodal function with the fewest queries. We may refer to temperature scaling as NLL-T for brevity.
Proposition 1. For any distribution p' on X x y where y (1, ,,], and for any set of functions 1,... 4m x R, Ex,p
e//e// log E, 651777
is unimodal function T.
ECE Minimization (ECE-T) Instead of finding temperature that maximizes log likelihood, we find that empirically is often better to directly minimize the discretized ECE in Eq. 2. Adapting ECE minimization to our framework is similar to log likelihood maximization, except that we query for the necessary quantities to compute the ECE score instead the log likelihood. In Appendix C.2.2, show how to compute the ECE score with as few queried quantities possible.
Histogram Binning Histogram binning can be adapted to the above protocol with only one iteration (K 1). The function f queries D, for the number of correct predictions in each bin and the total number of samples each bin. We average the query results from different datasets. To compute the new confidence for bin, divide the average number of correct predictions by the average total number of samples in each bin.
4 ACCURACY TEMPERATURE SCALING
When add Laplace noise according to Eq. 4, the added noise increases with the number of iterations K and the L, sensitivity of the query functions f². In other words, when we adapt a calibration algorithm our framework, need to add more noise if the original algorithm gains
5

---

Under review as a conference paper at ICLR 2021
lot of information about the private datasets D1:... Da. The relative amount of noise also increases as the amount of data available decreases, as is the case when binning is used. Larger noise will degrade calibration performance. To improve performance, propose new recalibration algorithm called accuracy temperature scaling that acquires much less information than previous algorithms.
Our method is form of temperature scaling that is based weaker notion than calibration. Let classification accuracy and average confidence be denoted as
Acc(Ø) Pr(o(z) y). and Conf [[](x))
Ace and Conf are expectations of [0. -oouuddd random variables, so they can be accurately es- timated even from relatively small quantity of data. We say that classifier is consistent if (o)( = (p). We tune the temperature parameter in Eq. 3 until the average confidence Conf is identical the average accuracy Acc, i.e. until consistency is achieved. We will refer to our method as Acc-T for brevity.
Consistency is strictly weaker condition than calibration. Surprisingly, even when there is lot of data and no privacy requirements, optimizing for consistency achieves similar performance as directly optimizing for ECE in our experiments, shown in Appendix E.2.
4.1 ACCURACY TEMPERATURE SCALING UNDER DIFFERENTIAL PRIVACY
Adapting Acc-T to our differential privacy framework is similar to doing for temperature scaling in Section 3.3. As we show in Proposition 2 (see Appendix B for proof), the Acc-T objective is also unimodal function of T. so objective function. Algorithm provides the complete algorithm for Acc-T under differential privacy. On Line 2, select initial temperature values. Line specifies query function that the hospitals use to pool their data while respecting differential privacy. Lines 4-12 implement differentially private golden section search over the recalibration temperature parameter. The algorithm outputs temperature value that improves the classifier's 's calibration on the new domain.
golden section search to find the T that minimizes the
Proposition For any distribution p" on X Y where y {1,... .m), and for any set of AAIAT
functions 4, -., Im X + I,, let Pr + max; Σ, E, and & + arg max, L(xx).
[Prz.gpp' (d(x) = = ((r()]) is unimodal funcction T.
Algorithm Acc-T with differential privacy
1: Input Private datasets D,. Da- Logit functions I,... mm : X + R. Initial temperature
range [T_, T2). Number of iterations K. Define á and pr in Proposition 2.
2: Set To T (T T 0.618, Tp T2 (T) TT) ) 0.618
For TØ M : Di + EX.ED, (1(0(z.) = y:) - pry (z:) Lap (K-1) and sample i== (D.)). Similarly set M for T and sample o9.
eg = Ed
4:
for = 0.... K-
5:
if 1o01 > uf then
6:
Set Th+ T 7 =
= Tbr = = T =T + (T+ T-) 0.618
Set u 6+1 ot. Sample vf .+1 for T as in line 3.
7:
8:
else
9:
Set kkt Th T T Tk Tk- Th. T 1
T (T+ - TT) 0.618
10:
Set u k+1 = v/. Sample op k+1 for T as in line 3.
end if
12:
end for
Return (T T/2 as the optimal temperature.
4.2 COMPARISON
We will briefly discuss how our method, Acc-T, compares to others such as histogram binning, temperature scaling, or ECE-T in terms of its theoretical bias (calibration error given infinite data), worst case variance (calibration error degradation when less data available). and adaptability to differential privacy (based on the relative amount of noise that must be added to satisfy differential

---

Under review as a conference paper at ICLR 2021
privacy). Ace-T has a higher theoretical bias than the other methods, since its objective function does not directly minimize the calibration error. However, in our experiments on deep neural networks, the bias of Ace-T is only slightly worse comparable to that of ECE-T temperature scaling in practice. Acc-T also has a lower worst case variance than other methods because it does not use binning (so there are more data points per bin) and its objective function has smaller range than that of temperature scaling. Overall, Acc-T has the highest adaptability to differential privacy; it has smaller L1 sensitivity than the other methods, less noise is necessary for differential privacy, Additional factors that affect the calibration quality and the level of privacy are discussed in Appendix D.I.
EXPERIMENTS
In this section, we run an extensive series of large, controllable experiments on three datasets to compare our proposed method Acc-T against five different baseline methods, three of which are designed with privacy concerns in mind, using the general procedure in Section 3. These benchmarks include various domain shifts and privacy settings, and our proposed Acc-T method consistently outperforms the other baseline methods. We also extensively validate the relationship between calibration error and several relevant factors for domain shift and privacy. Additional experimental details are included in Apppendix E.
5.1 EXPERIMENTAL SETUP
Methods We evaluate the differentially private versions of temperature scaling, ECE-T, histogram binning, and Acc-T over an extensive range of settings that considers calibration under various domain shifts and privacy concern.. We also include two baseline methods, (I) no calibration and (2) recalibration with only one private dataset from the target domain (so data from other sources not used; in this case privacy constraints need not be taken into account but less data is available).
Datasets To simulate various domain shifts, we use the ImageNet- C, CIFAR- 100-R- and CIFAR-1O-C datasets (Hendrycks & Dietterich, 2019), which are perturbed versions of the ImageNet (Deng et al., 2009), CIFAR- 100 (Krizhevsky Hinton, 2009), and CIFAR-10 10 (Krizhevsky & Hinton, 2009) test sets. Each- -C dataset includes 15 perturbed versions of the original test set, with perturbations such as Gaussian noise, motion blur, jpeg compression, and fog. We divide each perturbed test set into a validation split containing different "private data sources" with the same number of samples, and test split containing all of the remaining images. We then apply the recalibration algorithms over the validation split and evaluate the ECE on the test split. Note that only the unperturbed training sets were used to train the models.
Relevant factors We evaluate the ECE for all of the methods while controlling the following three factors: (1) the number of private data sources, (2) the number of samples per data source, and (3) the privacy level When we vary
factor, we keep the other two factors constant.
Additional details We use K = 5 iterations for all experiments, and report the average ECE achieved over 500 trials with randomly divided splits for each experiment. We report other experimental setup details including the type of network used in Appendix E.
5.2 RESULTS AND ANALYSIS
In Fig. 1, we plot the ECE vs. (la) the number of private data sources, (lb) the number of samples per data source, and (1c) the e value, for the ImageNet "fog" perturbation. Fig. 2 shows a similar plot for the CIFAR- "jpeg compression" perturbation, and Fig. 3 shows similar plot for the CIFAR-10 10 "motion blur" perturbation. Our proposed method, Acc-T, is shown in red, and clearly outperforms other methods under the constraints of differential privacy for these ranges of values. Full plots for all perturbations and datasets are included in Appendix E.3. Table I shows the overall median and mean ECE achieved by each recalibration method on ImageNet, CIFAR- 100, and CIFAR-10 10. These averages computed over all perturbations, numbers of private data sources, numbers of samples per source, and settings from the suite of experiments in E.3. Our method, Ace-T, far outperforms other methods in the domain-shifter differential privacy setting.
The performance of all recalibration algorithms degrades when subjected to the constraints of differential privacy, but some affected more than others for given situation. Selecting
7

---

Under review as a conference paper at ICLR 2021
differentially private recalibration algorithm for particular situation thus requires some consideration. To this end, we provide some analysis over these methods under the three relevant factors.
Number of Private Data Sources As the number of sources increases, Acc-T tends do well, even when the number of samples per source is small. Because Ace-T does not involve binning and the sensitivity of its objective function is small, there is relatively less noise for this method than for others. Therefore, it can effectively combine data from multiple sources even under the constraints of differential privacy, and is the best method in general.
Number of Samples Per Source As the number of samples per source increases, Acc-T tends to do well given enough data sources. As the number of samples per source grows towards infinity, recalibration with only one source works very well since we do not need to query other sources or apply privacy constraints. Histogram binning and ECE-T may also perform quite well with many bins when the number of samples is very large.
Privacy concern e When very low (i.e. the privacy requirements are very high), recalibrating with only one data source works well; this method remains unaffected by the strong privacy constraints, while all other methods worsen drastically due to the increased noise. For mid-range e values, Acc-T works well. When is very high, ECE-T can work well, since privacy is not much of a concern.
Number Sources
ECE Number of Samples
ECE
0.25-
0.30
0:29
ere, sturce
0.20
Scurce
0.18
ACP-T
ACCT
0:18
HB
0.15
0.20
0 0.12-
0:12
0.10-
6.10
0.08
0.05
0.03
250 500 750 1000 1250 1500 1750 2000
20
40
60
80
100
025 0.50 0.75
1.50 1:25 1.50 1.75 2.00
Private Data Sources
Samples per Data Source
(a) Samples 10, e 1.0
(b) Sources 100, 1.0
(c) Samples = 50, Sources 100
Figure 1: Recalibration results for ImageNet under the "fog" perturbation, with varying (la) number private data sources, (lb) number of samples per data source, and (Ic) privacy level e. Acc-T does best in these settings.
vs. Number Sources
ECE vs. Number Samples
ECE VS.
0.30
0.30
0.25
AGDT
HB
0.25
HR
ACCT
0.25
ACC-I
0
602-1
0.20
0.20
015
0.15
0.15
810-
6.10
50
100
150
200
250
10
20
30
40
025 0.50 0:75 1.50 1,25 1.50 1:75 2.00
Private Data Sources
Samples per Data Source
(a) Samples = 10, 1.0
(b) Sources = 50, 1.0
(c) Samples 30, Sources 50
Figure 2: Recalibration results for CIFAR- 100 under the "jpeg compression" perturbation, with varying (2a) number of private data sources, (2b) number of samples per source, and (2c) privacy level e. Acc-T does best.
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.22
0.29
H1
0.18
HS ACCT
0116-
ACDIT
EC2-T
BCLFT
BEE-T
ACCTT
11
a14
012-
010
0.08
an8-
0.05
805
50
100
150
200
250
10
20
40
025
0.75
1.50 1.25 1.50 1.75 2:00
Private Data Sources
Samples per Data Source
(a) Samples 10, e 1.0
(b) Sources 50, 1.0
(c) Samples 30, Sources 50
Figure 3: Recalibration results for CIFAR-IO under the "motion blur" perturbation, with varying (3a) number of private data sources, (3b) number of samples per source, and (3c) privacy level Acc-T does best.
8

---

Under review as a conference paper at ICLR 2021
Expected Calibration Error (median mean)
Recalibration method
ImageNet
CIFAR- 100
CIFAR-10
No recalibration
0.1343 / 0.1334
0.2718 0.3124
0.2187 0.2992
One source
0.0657 0.0700
0.1204 0.1209
0.1241 0.1355
Histogram binning
0,0656 0.0787
0.1867 0.1850
0.1168 0.1181
ECE-T
0.0684 / 0.0079
0,1655 / 0.1121
0.1160 / 0.1668
NLL-T
0.0597 0.0624
0.1583 0.1607
0.1157 0.1653
Acc-T
0.0289 0.0325
0.0890 0.0973
0.0836 0.1
Table Median and expected calibration enoor (ECE) achieved for domain-shifted data under differential privacy. Columns from left to right show the median/mean ECE achieved over perturbations, number of private data sources, number of samples per source, and e for ImageNet, CIFAR-I00 and CIFAR-10. Best calibration is shown in bold.
CONCLUSION
Simultaneously addressing the challenges of calibration, domain shift, and privacy is extremely important in many environments. In this paper, we introduced framework for recalibration on domain-shifted data under the constraints of differential privacy. Within this framework, we designed novel algorithm to handle all three challenges. Our method demonstrated impressive performance across a wide range of settings on a large suite of benchmarks. In future work, we are interested in investigating recalibration under different types of privacy mechanisms.
REFERENCES
Philip E. Agre. Surveillance and capture: Two models of privacy. Inf. Soc., 10:101-127, 1994.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Proceed- ings of ICML workshop on unsupervised and transfer learning, pp. 17-36, 2012.
Henri Berestycki, Jérôme Busca, and Igor Florent. Asymptotics and calibration of local volatility models. Quantitative finance, 2(1):61 2002.
Richard Berk. Criminal justice forecasts of risk: machine learning approach. Springer Science & Business Media, 2012.
Richard Berk. Machine learning risk assessments in criminal justice settings. In Springer International Publishing, 2019.
Glenn W Brier. Verification of forecasts expressed terms of probability. Monthly weather review, 78(1):1-3, 1950.
Centers for Medicare Medicaid Services. The Health Insurance Portahility and Accountability Act of 1996 (HIPAA). Online at http://ww.ccms.hhs gow/hipaa/, www..mmm.hhs.ggo/hiaaaaa.. 1996.
Nicolo Cesa-Bianchi and Gabor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
Weijie Chen, Berkman Sahiner, Frank W. Samuelson, Aria Pezeshk, and Nicholas A. Petrick. Calibration of medical diagnostic classifier scores to the probability of disease. Statistical methods in medical research, 27:1394 1409, 2018
Moustapha Cissé, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier, Parseval networks: Improving robustness adversarial examples. ICML, 2017.
Jeremy M Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint arXiv: 000v:/990200000 190..99022.911 18 2019.
Wenyuan Dai, Qiang Yang, Gui-Rong Xue, and Yong Yu. Boosting for transfer learning. In Proceedings the 24th international conference Machine learning, pp. 193-200. ACM, 2007.

---

Under review as a conference paper at ICLR 2021
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009.
Cynthia Dwork. Differential privacy: A survey results. In International conference on theory and applications of models of compuiation, pp. 1-19. Springer, 2008
Cynthia Dwork, Aaron Roth, al. The algorithmic foundations differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):21 1-70, 2014.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409. 7495, September 2014.
Dorothy J. Glancy. Privacy in autonomous vehicles. Santa Clara law review, 52:1 52:1171, 2012.
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society; Series B (Statistical Methodology), 69(2): 243-268, 2007.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1321-1330. JMLR. org, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv: December 2015.
Qingfeng He and Annie Antón. A framework for modeling privacy requirements in role engineering. Proceedings of the 9th International Workshop on Requirements Engineering: Foundation for Software Quality (REFSQ 03), 06 2003.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. /CLR, 2019.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? In Advances in neural information processing systems, pp. 5574-5584, 2017.
Javed Khan, Jun S Wei, Markus Ringner, Lao H Saal, Mare Ladanyi, Frank Westermann, Frank Berthold, Manfred Schwab, Cristina R Antonescu, Carsten Peterson, et al. Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks. Nature medicine, 7(6):673, 2001.
Xiaowei Kortum, Lorenz Grigull, Urs Muecke, Werner Lechner, and Frank Klawonn. Improving the decision support in diagnostic systems using classifier probability calibration. In IDEAL, 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning, pp. 2801-2809, 2018.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances in Neural Information Processing Systems, pp. 3787-3798. 2019.
Balaji Lakshminarayanann Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402 6413, 2017.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, arXiv preprint http::/608.099833. August 2016.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2017.
10

---

Under review as a conference paper at ICLR 2021
Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science O7), pp. 94-103. IEEE, 2007.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ICLR, 2018.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference Machine learning, pp. 625-622. ACM, 2005.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 13450):1345,135, 2009.
Vishal M. Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Visual domain adaptation: A survey of recent advances. IEEE Signal Processing Magazine, 32:53-69, 2015.
John Platt al. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Keywan Christian Rasekhschaffe and Robert Jones. Machine learning for stock selection. Financial Analysts Journal, 75:70- 88, 2019.
Cynthia Rudin and Berk Ustun. Optimized scoring systems: Toward trust in machine learning for healthcare and criminal justice. Interfaces, 48:449 2018.
Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. arXiv preprint
2018.
Alexander J Smola, Peter Bartlett, Dale Schuurmans, Bernhard Schölkopf, Michael I Jordan, et al. Advances in large margin classifiers. MIT press, 2000.
Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan. Sebastian Nowozin, D Sculley, Joshua Dillon, Jie Ren, and Zachary Nado. Can you trust your model's 's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances Neural Information Processing Systems, pp. 13969-13980, 2019.
Yi Sun, Ding Liang, Xiaogang Wang. and Xiaoou Tang. Deepid3: Face recognition with very deep neural networks. arXiv preprint arXiv: 008733 2015.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. International conference on machine learning, pp. 1139-1147, 2013.
Jonathan Wenger, Hedvig Kjellström, and Rudolph Triebel. Non-parametric calibration for classifica- tion. ArXiv, abs/1906.04933 2019.
Luona Yang, Xiaodan Liang. Tairui Wang, and Eric P. Xing. Real-to-virtual domain unification for end-to-end autonomous driving. In ECCV, 2018.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, pp. 609-616. Citeseer, 2001.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 1605v: May 2016.
11

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL BACKGROUND INFORMATION
A.1 COMPUTATION OF ECE
To compute the ECE, discretization is necessary. We first divide [0, 1] into bins (c1, c&.)
such that 0 < C1 < .... < C2 = 1, and then we compute the average accuracy Acc and average confidence Conf in each bin (for convenience, denote Co 0)
A, Pr (f(xa) = vlū(x) E [c)1, Co, Elp(>))((()) E [c)-1,
Then the ECE defined in Eq. 2 can be approximated by discretized version ECE(S,1) x ECE(S, cc)
Pr ((()) [c-11, ci)l |Aco(f, Conf(S,
Given empirical data D (m ) we can estimate ECE(S, as OCE(f.p:c) ECE(f.p:c,
Σ
I(f(x,)) yy) -((x;)
Note that there are two approximations: we first discretize the ECE, and then use finite data to approximate the discretized expression
(C = ECE(f,P;C)
BČE(f.p;c,)
In practice, if the first approximation is better (more bins are used), then the second approximation must be worse (there will be less data in each bin) (Kumar et al., 2019). In other words, with finite data, there is tradeoff between calibration error and estimation error. Note that newer estimators. e.g. (Kumar et al., 2019), can measure the ECE even more accurately, particularly when there are more bins.
A.2 LAPLACE MECHANISM PROOF
Theorem 1. The Laplace mechanism (Dwork et al., 2014) preserves e-differential privacy.
Proof. Let D E N222 and E N be two databases that differ by up to one element, i.e. |D - D'Il, < Let function f Na + R=, and let pD and p/D denote the probability density functions of Mr(D; and (D' f.e), respectively. Then we can take the ratio of pn to p'p at an arbitrary point x E R
exp( - f(())
PD(
(x)
cxp(
t=1
Af
<(I(D"); xz IS(D); xil)
exp
AS
1=1
d - S(D'dl)
< exp
Af
i=1
[IF((D) - F(D'Ili
exp
Af
exp(()
where the first inequality follows from the triangle inequality, and the second inequality follows from the definition of sensitivity (Dwork et al., 2014).
12

---

Under review as a conference paper at ICLR 2021
B PROOFS
Proof of Proposition I.
0
E yrup'
log
elu(a/T
yrup
(x)//T
log Σ eb(0)/T
0T
0T
0T
E (
-l,(1//T2
Eeebley/
E, L1b0etobl Ej
-b((0)
T Rayypp
Let us set the derivative equal to 0. Suppose there are multiple solutions Ti T2; this implies that
E, b( Σ, diler/
Σ b(0)eb000 E, cb(l))//
E Eggyrpp
Ex gyrup"
(5)
((x)ebloo Eg 25
Therefore, if there are 0 or so- is monotonically non-increasing.
lutions to Eq. 5, the original function must be unimodal. there are at least 2 solutions must be constant function VT E T2), which im- Σn E,t0
Σ ((
Ti T2, then Enyp
plies that Exyynn
constant function of T T2). This further implies that esGar
Eng
is constant function for all T' R, which is also unimodal.
Proof of Proposition 2. Because pr is a monotonically decreasing function of T, Ep- [p) is
also a monotonically decreasing function of T. This means that Pra.y<p- [f(x) = yl - Ep- (f()))
a monotonically decreasing function of T. The absolute value of a monotonic function must be monotonic or unimodal.
C ADDITIONAL DETAILS FOR SECTION
C.I GOLDEN SECTION SEARCH
The golden section search is algorithm for finding the extremum of unimodal function within specified interval. It is an iterative method that reduces the search interval with each iteration. The algorithm described below. Note that we describe the algorithm for minimization problem, but also works for maximization problems.
1. Specify the function to be minimized, g(.), and specify an interval over which to minimize g. [Tminy Tmaz]-
2. Select two interior points Ti and T2, with Ti < T2, such that Ti = Tmax
V5-1 (Tmas maz
2
T T and T2 TTmin -5-1 (Tmaz- Tmin)). Evaluate g(Ti) and 9(T2).
3. If g(T)) (T2), then determine new Tiins Tı, T2, T Tmax as follows:
T Tmin = T1
Tmax Tmaz
Ti = T2
V5 V5-1-
T2 Tomn +
(Tmas Tmin)
2
13

---

Under review as a conference paper at ICLR 2021
If ((T)) (T2)), determine new min: T1, T2, max as follows:
min Tinn
Taa T2
T2 Ti
V5-1 (Tmax T min))
Ti T maa
Note that in either case, only one new calculation is performed.
4. If the interval is sufficiently small, i.e. Tmaz Twin < 6, then the maximum occurs at (Tmin T 1/2. Otherwise, repeat Step
C.2 ADAPTING EXISTING RECALIBRATION METHODS FOR DIFFERENTIAL PRIVACY
In this section, we go into more detail about how to adapt several existing recalibration algorithms for the differential privacy setting with our framework.
C.2.1 TEMPERATURE SCALING
Temperature scaling optimizes over the temperature parameter T using the negative log likelihood loss, and thus requires multiple iterations to query the databases D, at different temperature values using golden section search. In this case, the objective function 9(-) the negative oiglliieldoood (NLL) loss over all samples. In the standard NLL formulation, the overall loss is the average of the samples' NLL losses, but summing these losses for each database rather than taking the average equivalent except for constant scale factor (the total number of samples in the database). Thus, the function j queries each D, for its summed NLL loss. The sensitivity f is technically infinite, since the range of the NLLI function is infinite, but in practice can choose some sufficiently large value (we chose A = 10, since that was approximately the largest NLL value we saw among the images that we checked). We chose Trin 0.5 and Tmaz 3.0, since empirically the optimal temperature always seems to fall within this range, and used K = 5 iterations. To aggregate information from different D,, simply average the (D)))),- ... .M (Da). The new classifier (ø, f¹) outputs probabilities that are recalibrated with the (noisy) optimal temperature.
C.2.2 TEMPERATURE SCALING BY ECE MINIMIZATION
The standard recalibration objective when applying temperature scaling is to maximize the log likelihood of validation dataset. This objective given in both recent papers (Guo et al., 2017) and established texthooks (Smola et al., 2000). An alternative, but surprisingly overlooked, objective is to minimize the discretized ECE directly. To adapt this method differential privacy, must again use multiple iterations to query the databases D; at different temperature values using golden section search. Here we want find the temperature that minimizes the discretized ECE:
min Σ |Acc Conf' pr min Σ béna
Dcorrect
Σ.
Tlhin
Thba
Abbn
Miotal
where pr is the proportion of samples in the bin, Roorrect is the number of correct predictions in the bin, Aain is the total number of samples in the bin, E: G is the sum of the confidence scores for all samples in the bin, and nestal is the total number of samples across all bins.
Simplifying Eq. and ignoring Nootal as a constant, our objective function g()) becomes
0 T)
Σ Meorrect Σ cal
The function f*r queries each D, for the quantity (Ncorrect Σ. ci) in each bin. The sensitivity Aʃ = 1, since this quantity could change by up to with the addition removal of one sample to database. We use Tman 0.5, Tmaz = 3.0, and K = 5 iterations. We use 15 bins (since we also evaluate the discretized ECE with 15 bins), so the M are vectors E R 15 To aggregate information from different D.. we average the (D,))),.-. and then this absolute value vector all bins. In the absence of noise, this aggregation process
(M((Da)). take the absolute value of this average.
14

---

Under review as a conference paper at ICLR 2021
will yield the correct overall g(-) exactly, using all samples from all sources. The new classifier (ó. (') outputs probabilities that are recalibrated with the (noisy) optimal temperature. Unsurprisingly, ECE-T performs very well without the constraints of differential privacy, so this method may be good choice when is high.
C.2.3 HISTOGRAM BINNING
Histogram binning is a relatively simple, non-parametric recalibration method that can be adapted to differential privacy with single iteration (i.e. K 1). The function f queries D; for the number of correct predictions in each bin and the total number of samples in each bin. A 2 because if exactly one entry is added removed from database, the number of correct predictions can change by at most for exactly one of the bins, and the total number of samples can change by most 1 for exactly one of the bins. We use 15 bins in our experiments. so the M are vectors 30 To aggregate information from different D; we average the (D,),, ... .M*(Da). The new confidence for each bin is the average number of correct predictions divided by the average total number of samples for that bin.
D ADDITIONAL DETAILS FOR SECTION
D.1 FACTORS THAT AFFECT CALIBRATION QUALITY AND PRIVACY
Table 2: The impact of various factors recalibration quality and privacy preservation.
t
Data
Iterations
Bins
Sensitivity
te
Calibration quality
Privacy preservation
Table 2 shows several factors and hyperparameter choices that affect the calibration quality and the level of privacy for all recalibration methods. More data improves both calibration and privacy. More iterations improves calibration when privacy is not required (e.g. running more iterations of gradient descent), but hurts privacy (making multiple queries in a parametric optimization setting with the same amount of added noise increases e). Using more bins for methods that involve binning improves calibration when enough data is available, but may hurt privacy. Higher sensitivity of the f* function hurts privacy, and higher represents less privacy. We discuss each of these more detail below.
Data Differentially private recalibration algorithms require sufficient data in order to work well. We cannot trivially combine data from different private datasets because each dataset holder must honor its agreement with the individuals whose information is in that dataset. Our framework describes method for pooling data from different private datasets while allowing each one to respect differential privacy for its users, which is necessary for improved calibration while preserving privacy.
Number of iterations For parametric optimization recalibration methods, multiple iterations are generally needed to search the parameter space. Using additional iterations improves the calibration without differential privacy (e.g. rumning more iterations of gradient descent), but hurts the calibration when differential privacy is required. With multiple iterations, a worst-case bound on the overall L sensitivity of f is K times the sensitivity of single query sisigle since single database entry may change the response to each query by up to Afsingie. Thus, the amount of noise added to the true query responses must follow L(D, K A single/e) distribution maintain e-differential privacy. Because using more iterations increases the amount of noise added, it is best to search through the parameter space while minimizing the number of iterations needed for the desired granularity. We use golden section search to do this. Each iteration of the golden section search narrows the range of possible values of the extremum, but increases the amount of noise added the data; general, we select K such that the granularity and the noise are balanced.
Binning Several the recalibration methods discussed use binning, where all of the confidence estimates divided into mutually-exclusive bins. Without differential privacy, using more bins
15

---

Under review as a conference paper at ICLR 2021
generally improves calibration when a lot of data is available (i.e. above "data threshold"). but hurts calibration below this data threshold. When not enough data is available, using more bins increases the estimation error since there are too few samples in each bin. In the differential privacy setting, using more bins may degrade the calibration. In this setting, one query may request a summary statistic from each bin. Because single database entry can be in exactly one bin, the remaining bins are unaffected and the sensitivity does not increase with more bins. However, although the number of bins does not affect the absolute amount of noise, it can affect the relative amount of noise. When more bins are used, there are fewer elements in each bin on average. Thus, the summary statistics involved tend to be lower, and the noise is relatively higher.
Note that when multiple equal-width bins are involved, as in temperature scaling by ECE minimization (see Section C.2.2), the optimization problem may not be strictly unimodal since samples can change bins as the temperature changes. Using bins with equal numbers of samples, rather than equal widths, ensures unimodality in temperature scaling but makes it difficult to combine information from different private data sources (since different sources will have different bin endpoints). Thus, we elected to use equal-width bins in our experiments. Although the function to be minimized not necessarily unimodal, it is generally close enough approximation that golden section search returns reasonably good results with few queries, and empirically performs better than grid search.
Sensitivity of fh An f function with a large range has a detrimental effect on the amount of noise added. For instance, the range of the negative log-likelihood technically infinite (although in practice we used some sufficiently large value). Thus, the sensitivity of a method with the negative log-likelihood in the objective function is quite high, and the amount of noise needed to preserve differential privacy is large.
value Calibration is worse when is smaller, i.e. when there is a higher privacy level with stronger differential privacy constraints.
E ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS
E. EXPERIMENTAL SETUP
We simulated the problem of recalibration with multiple private datasets on domain-shifted data using the ImageNet-C, CIFAR-100-C. and CIFAR-10-C datasets (Hendrycks Dietterich, 2019), which are perturbed versions of the ImageNet (Deng et al., 2009), CIFAR-100 (Krizhevsky & Hinton, 2009), and CIFAR-IO (Krizhevsky & Hinton, 2009) test sets respectively. We randomly divided each perturbed test set into Resources validation sets of size Rsamples and test set comprising the remaining images, where nsources represents the number of private data sources and Rsamples represents the number of samples per source. We computed each ECE value by binning with 15 equal-widtt bins.
For ImageNet, we varied the number of private data sources from 100 to 2000 in step sizes of 100, with 10 samples per data source and e 1. We varied the number of samples per data source from 5 to 100 in step sizes of 5, with 100 private data sources and c = 1. We varied e from 0.2 to 2.0 in step sizes of 0.2, with 50 samples per data source and 100 private data sources. For CIFAR- and CIFAR-10 10, we varied the number of private data sources from 10 to 250 in step sizes of 10, with 10 samples per data source and 1. We varied the number of samples per data source from 5 to 50 in step sizes of 5, with 50 private data sources and c = 1. We varied e from 0.2 to 2.0 in step sizes of 0.2, with 30 samples per data source and 50 private data sources. We used K = 5 iterations for all experiments. We reported the average ECE achieved over 500 randomly divided trials for each experiment.
All models were trained on only the unperturbed training sets. For ImageNet, we trained ResNet50 network (He et al., 2015) for 90 epochs with an SGD optimizer (Sutskever et al., 2013) with an initial learning rate of 0.1, and decayed the learning rate according to cosine annealing sched- ule (Loshchilov & Hutter, 2016). For CIFAR- 100 and CIFAR- 10, we trained Wide ResNet-28- 10 networks (Zagoruyko & Komodakis, 2016) for 200 epochs with an SGD optimizer with initial learning rate of 0.1, and again decayed the learning rate with a cosine annealing schedule. For each dataset, tested both the unperturbed accuracy and the perturbed accuracy each of 15
16

---

Under review as a conference paper at ICLR 2021
perturbation types in (Hendrycks & Dietterich, 2019) multiple severity levels ensure sharpness. These accuracy tables can be found in E.2.
E.2 EXPERIMENTS WITHOUT DIFFERENTIAL PRIVACY CONSTRAINTS
Classification Accuracy
Perturbation Type
CIFAR-10
CIFAR-100
ImageNet
Brightness
0.9290
0.7107
0.5570
Contrast
0.4656
0.2967
0.0422
Defocus Blur
0.6402
0,4008
0.1506
Elastic Transform
0.7616
0.5214
0.1477
Fog
0.7639
0,4808
0.2270
Frost
0.6907
0,4196
0.2064
Gaussian Noise
0,2889
0.1046
0.0447
Glass Blur
0.5313
0.2212
0.0834
Impulse Noise
0.2940
0.0642
0.0463
Jpeg Compression
0.7056
0.4190
0.3318
Motion Blur
0.7062
0.4997
0.1337
Pixelate
0.5137
0.2994
0.2260
Shot Noise
0.3581
0.1190
0.0507
Snow
0.7975
0.5268
0.1594
Zoom Blur
0.7163
0.4708
0.2287
Unperturbed
0.9613
0.8050
0.7613
Table 3: Classification accuracies for CIFAR- 10, CIFAR-100 and ImageNet under the highest severity perturba- tions of the CIFAR-10-4 CIFAR-100-C. and ImageNet-C C test sets. The classification models used achieve the expected statc-of-the-art results for accuracy on the unperturbed test
CIFAR-10
Perturbation Severity
Base
NLL-T
Acc-T
ECE-T
Brightness
0.0456
0.0194
0.0278
0.0182
Contrast
0,4202
0,0503
0.0348
0.0368
Defocus Blur
0.2431
0.0381
0.0372
0.0366
Elastic Transform
0.1555
0.0287
0.0264
0,0319
Fog
0.1813
0.0463
0.0433
0.0435
Frost
0,2207
0.0636
0.0570
0.0581
Gaussian Noise
0.6052
0.0624
0.0364
0.0530
Glass Blur
0.3434
0.0426
0.0393
0.0389
Impulse Noise
0.4963
0.0570
0.0499
0.0566
Jpeg Compression
0,2000
0.0423
0.0344
0.0338
Motion Blur
0.2153
0.0430
0.0395
0.0427
Pixelate
0.3840
0.0676
0.0620
0.0649
Shot Noise
0.5282
0.0503
0.0464
0.0484
Snow
0,1412
0.0390
0.0321
0.0391
Zoom Blur
0.1931
0.0382
0.0343
0.0363
Unperturbed
0.0251
0.0078
0.0089
0.0075
Table Expected calibration error (ECE) on CIFAR-10 without privacy constraints shown. Columns from left to right show ECE for the baseline without calibration, recalibration with temperature scaling by minimizing the negative log likelihood, recalibration with temperature scaling by matching predictive confidence to accuracy (our method), and recalibration by minimizing the ECE directly. Rows indicate the type of perturbation applied. These results correspond perturbation severity of S, Best calibration each perturbation shown bold.
17

---

Under review as a conference paper at ICLR 2021
CIFAR-100
Perturbation Severity
Base
NLL-T
Acc-T
ECE-T
Brightness
0.1087
0.0602
0.0460
0.0449
Contrast
0.3817
0.0839
0.0574
0.0518
Defocus Blur
0.2707
0.0847
0.0785
0.0780
Elastic Transform
0,1776
0,0626
0.0589
0.0639
Fog
0.2217
0.0656
0.0560
0.0574
Frost
0.2929
0.0872
0.0791
0.0771
Gaussian Noise
0.6313
0.0518
0.0423
0.0316
Glass Blur
0.4438
0.0777
0.0658
0.0700
Impulse Noise
0.3574
0.0160
0.0061
0.0081
Jpeg Compression
0,2042
0.0667
0.0512
0.0492
Motion Blur
0,2040
0.0707
0.0548
0.0546
Pixelate
0.3639
0.0599
0.0322
0.0327
Shot Noise
0.6200
0.0654
0.0468
0.0391
Snow
0.1912
0.0674
0.0575
0.0585
Zoom Blur
0.2189
0.0805
0.0726
0.0724
Unperturbed
0.0793
0.0456
0.0319
0.0305
Table Expected calibration emor (ECE) on CIFAR-100 without privacy constraints is shown. Columns from left to right show ECE for the baseline without calibration, recalibration with temperature scaling by minimizing the negative log likelihood, recalibration with temperature scaling by matching predictive confidence to accuracy (our method), and recalibration by minimizing the ECE directly. Rows indicate the type of perturbation applied. These results correspond perturbation severity of 5. Best calibration for each perturbation shown in bold.
ImageNet
Perturbation Severity 5
Base
NLL-T
Acc-T
ECE-T
Brightness
0.0413
0.0325
0.0298
0.0307
Contrast
0.0651
0.0083
0.0083
0.0116
Defocus Blur
0.0618
0.0235
0.0230
0.0239
Elastic Transform
0.2426
0.0287
0.0308
0.0308
Fog
0.1572
0.0255
0.0231
0.0232
Frost
0,1430
0.0254
0.0253
0.0247
Gaussian Noise
0.1501
0.0070
0.0080
0.0092
Glass Blur
0.1340
0.0160
0.0164
0.0168
Impulse Noise
0.1555
0.0084
0.0069
0.0066
Jpeg Compression
0.0855
0.0188
0.0228
0.0189
Motion Blur
0.1254
0.0180
0.0183
0.0194
Pixelate
0.1306
0.0175
0.0172
0.0170
Shot Noise
0.1820
0.0085
0.0081
0.0109
Snow
0,1895
0.0327
0,0323
0.0321
Zoom Blur
0.1343
0.0200
0.0191
0.0193
Unperturbed
0.0390
0.0240
0.0239
0.0261
Table 6: Expected calibration error (ECF) on ImageNet without privacy constraints is shown. Columns from left to right show ECE for the baseline without calibration, recalibration with temperature scaling by minimizing the negative log likelihood, recalibration with temperature scaling by matching predictive confidence accuracy (our method), and recalibration by minimizing the ECE directly. Rows indicate the type of perturbation applied. These results correspond to a perturbation severity of S. Best calibration for each perturbation is shown in bold.
Table shows the classification accuracy achieved by our models on each of the 15 perturbations of the CIFAR-10-C. CIFAR- 100-C, and ImageNet-C test sets, well as the unperturbed test set. Note that the models are trained only on unperturbed training data. The accuracies achieved are in line with reported sate-of-the-ar numbers.
18

---

Under review as a conference paper at ICLR 2021
Tables 4, 5, and 6 summarize calibration results without differential privacy constraints for CIFAR-10, 10, CIFAR-I00 100, and ImageNet, respectively. Our Acc-T algorithm generally improves the model's calibration compared to the standard temperature scaling method NLL-T. Despite its simplicity, Ace-T also performs on par with ECE-T, generally achieving similar ECEs, even when privacy is not required.
E.3 EXPERIMENTS WITH DIFFERENTIAL PRIVACY CONSTRAINTS
The figures in this section show recalibration results for ImageNet, CIFAR-100, and CIFAR-10 under the highest severity perturbations. In the left panel of each figure, we vary the number of private data sources. In the middle panel, we vary the number of samples per data source. In the right panel, we vary the privacy level c. Our method, Acc-T, generally does best in these settings.
IMAGENET RESULTS
ImageNet, unperturbed
ECE Number of Data Sources
ECE vs. Number of samples
ECE VS. E
no, maol
0.25
0.20
0.15-
0nt SAMXC
0.20
0.18
0.35
0.12
18 0.15
0.12
SCET
14-1
0.10
0.08
0.05
0.04
0.05
250 500 150
1000 1250
6.
100
9.25
0.00
1.00 1.25 1.75 2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, 1.0
(b) Sources 100,
1.0
(c) Samples = 50, Sources 100
ImageNet, brightness perturbation
ECE Number of Data Sources
ECE Number of Samples
ECE VS.
0.25
no meal enuuaturse
one, source
0.30
0.20
0.25
0.20
1 0.15
N
0.20
MLLT
ML-1
0.10
0.10
0.30
0.05
0.05
0.55
250 500 750 1000 1250 1500 1750 2000
20
60
100
0.25 0.50
0.35 1.60 1.25
1.50 1.75 2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, 1.0
(b) Sources 100, 1.0
(c) Samples = 50, Sources 100
ImageNet, contrast perturbation
vs. Number Data Sources
ECE Number Samples
ECE
0.08-
0.12
peri
0.08
0.07-
0.10
H5
HD
8
0.05-
0.08-
ACDT
0.08
0.06
MLL-T
ACCT
0.05
0.04
0.36
0.64-
0.04
0.05-
0.02-
0.02
0.02
0.00
250 500 750 1000 1250 1500 1750 2000
20
40
60
80
100
0.25
0.50
0.75
1.00
1.25
1.50
1.55 2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, c 1.0
(b) Sources 100, 1.0
(c) Samples = 50, Sources
19

---

Under review as a conference paper at ICLR 2021
ImageNet, defocus blur perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.14
source
0.20
0.18
no. peueeal
0.34
0.16
0.12
0.35
ACCTT
ACCTT
SEET
0 0.10-
N
0.12
0.05
0.08
0.05
0.00
0.56
0.04-
0:03
0.04
0.82
250 500 150
1000
60
100
0.25
0.00 1.00
Private Sources
Samples per Data Source
(a) Samples 10, 1.0
(b) Sources 100,
1.0
(c) Samples = 50, Sources 100
ImageNet, elastic transform perturbation
Number
ECE Number Samples
ECE
0.25
0.25
ene_seurte
(91, Sture
0.20
HD
0.20
0.20
ACCT
ADD-T
8 0.15
0.35
0.10
0.30
0.10
0.05
0.05
250 500 750 1000 1250 1500 1750 2000
20
60
100
0.25 0.50 0.55 1.00 1.25 1.00 1.75 250
Private Data Sources
Samples Data Source
(a) Samples 10, c
(b) Sources 100, 1.0
(c) Samples = 50, Sources
ImageNet, fog perturbation
ECE Number of Data Sources
ECE vs. Number er of Samples
ECE VS.
no, Jeae onn Souree
real
0.23
0.20
0.20
0.18
0.25
ACCT
0.38
0.15-
EGST
0.15
RCET
0.12
NLLT
0.15
MULT
M.-T
0.10
0.30
0.08
0.10
0.05
0.65
0.03
100
5.25
1.00
250
Private Sources
Samples per Data
(a) Samples 10, c 1.0
(b) Sources 100, = 1.0
(c) Samples = 50, Sources 100
ImageNet, frost perturbation
vs. Number Sources
ECE Number Samples
ECE
0.30
peci
0.20
0.18
0.25
H5
0.38
HD
0.15-
ACDT
0.20
0.15
0 0.12
0.10
0.08
0.36
0.05
0.05
0.00
250 500 750 1000 1250 1500 1750 2000
20
40
60
80
100
0.25
0.50
0.35
1.00
1.25
1.50
1.55
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, c
(b) Sources 100,
1.0
(c) Samples = 50, Sources
20

---

Under review as a conference paper at ICLR 2021
ImageNet, Gaussian noise perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.14
Prii source
no. Plueaat
0.34
0.12
ACCTT
ACCTT SEET
0.10-
600
0 0.08
N
0.06-
0.08
0.04
0.05
0.34
0.02
0.03
0.52
250
500
150
1000 1250
100
0.25
0.00 1.00
Private Data Sources
Samples per Data Source
(a) Samples 10, 1.0
(b) Sources 100,
1.0
(c) Samples = 50, Sources 100
ImageNet, glass blur perturbation
Number
ECE Number Samples
ECE
0.14
0:30
HD
ene_seurte
0.32
0.14
Sture
0.15
ACCT
ADD-T
0.30
0.08-
1 0.30
0.08
0.05-
0.08
0.06
0.04-
0.85
0.04
0.03
250 500 750 1000 1250 1500 1750 2000
20
60
100
0.25 0.50 0.55 1.00 1.25 1500 1.75 250
Private Data Sources
Samples Data Source
(a) Samples 10, c
Sources 100,
1.0
(c) Samples = 50, Sources
ImageNet, impulse noise perturbation
ECE Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.20
no, Jeal
opn souree
0.18
recl
0.15
ACCT
0.12
0.32
EGS1T
0.10
RCET
0.08-
NLLT
0.08
M.-T
0.05
0.08
0.04-
0.05
0.04
0.02
0.03
0.02
L:so Private Sources
100
0.25
1.00
250
Samples per Data
(a) Samples 100 c 1.0
(b) Sources 100, = 1.0
(c) Samples = 50, Sources 100
ImageNet, jpeg compression perturbation
vs. Number Data Sources
ECE Number Samples
ECE
peci
0.25
H5
0.20
HD
ACDTT
0.15
0.20
0.15
0.15
0.10
0.05
0.65
250 500 750 1000 1250 1500 1750 2000
20
40
60
80
100
0.25
0.50
0.35
1.00
1.25
1.50
1.55
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, c
(b) Sources 100,
1.0
(c) Samples = 50, Sources
21

---

Under review as a conference paper at ICLR 2021
ImageNet, motion blur perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.25
no. peueaal
0.15
source
eng.seure
0.14
0.20
0.14
0 0.19
0.12
N
0.35
ACCTT
0.12
ACCTT
BEET
0.08
0.30
0.08
0.06
0.06
0.05
0.02
250
150
1000
100
0.25 0.00 1.00
Private Sources
Samples per Data Source
(a) Samples 10, 1.0
(b) Sources 100,
1.0
(c) Samples = 50, Sources 100
ImageNet, pixelate perturbation
Number
ECE Number Samples
ECE
0.23
0.30
0.23
0.18
0.25
HD
ene_seurte
0.38
Sturc
0.15-
ACCT
ADD-
0.12
0.15
0.12
0.19-
0.30
0.06
0.05
0.05
0.65
0.03
0.58
250 500 750 1900 1250 1500 1750 2000
20
60
100
0.25 0.50 0.55
1.00
1.25 1.00 1.75 250
Private Data Sources
Samples Data Source
(a) Samples 10, c
Sources 100,
1.0
(c) Samples = 50, Sources
ImageNet, shot noise perturbation
ECE Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.18-
no, 2eal
0.20
no. real
0.18
0.15
opn Souree
0.38
0.35
0.12
0.10-
NLLT
w 0.12
0.30
0.35
ACCT
EGS1T
MLLT
0.10
0.12
REET
M.-T
0.08
0.08
0.56
0.05
0.05
0.03
0.85
Liso
100
0.25
1.00
250
Sources
Samples per Data
(a) Samples 10, c 1.0
(b) Sources 100, = 1.0
(c) Samples = 50, Sources 100
ImageNet, snow perturbation
vs. Number Sources
ECE Number Samples
ECE
0.20
peal
0.20
0.18
H5
0.28
HD
0.15 1
ACDT
0.15
0.12
0.10
0.30
0.65
0.05
0.85
0.30
250 500 750 1000 1250 1500 1750 2000
20
40
60
80
100
0.25
0.50
0.35
1.00
1.25
1.50
1.55 2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, c
(b) Sources 100,
1.0
(c) Samples = 50, Sources
22

---

Under review as a conference paper at ICLR 2021
ImageNet, zoom blur perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS. E
0.23-
0.20-
pource
0.30
no. plueaa
eng.teute
0.20
0.18
0.15
ACD
0.25
0.20
ACCTT
0.15
ACCTT BEET
G 0.12
0.19-
N
0.15
0.30
0.08-
0.05
0.05
0.45
0.03
250
500
150
1000 1250
60
100
9.25
0.20
1.00 1.25
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, 1.0
(b) Sources 100, c 1.0
(c) Samples = 50, Sources 100
CIFAR-100 RESULTS
CIFAR-100. 100, unperturbed
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS. E
0.88-
re.ncal
JAMLE
0.36
0.18
0.38
0.35
PLJ
0.16-
ACCTT
0.14
ACC-T
4 0.14
EG2-T
NULT
600-1
0.12
A12-
0.08
0.50
0.06
0.36
0.8
0.04
0.25
0:00
1.00 1.25
255
Private Data Sources
Samples per Data Source
(a) Samples 10, 1.0
(b) Sources = 50, 6 1.0
(c) Samples = 30, Sources = 50
CIFAR-100, brightness perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.22
0.24
ne, meal
tre Joure
no mesl
0.20
0.23
0.20
0.18
ECE-T
0.15
ECS-T
MLL1
0.18
0.35
BCETT
A14-
0.12
0.32
0.30
0.10
015-
0.08
0.28
0.8
0.05
0.55
50
100
150
200
250
10
20
30
46
50
0.25
0.50
0.75
1.00
1.25
1.50
1.75 2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, c 1.0
(b) Sources 50, 1.0
(c) Samples 30, Sources 50
CIFAR-100, contrast perturbation
ECE Number Data Sources
ECE Number Samples
ECE
0.35
0.35
0.35
0.30-
HB
ACCIT
0.30
H5
ACCT
0.30
HD
0.25
N.L.T
0.25
0.25
a29
0.20
0.20
A15-
0.35
A13-
50
100
150
200
250
0.05
10
20
30
40
50
0.25
0.50
0.35
1.00
1.25
1.50
1.75
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, c 1.0
(b) Sources 50, 6 1.0
(c) Samples -30, Sources
23

---

Under review as a conference paper at ICLR 2021
CIFAR-I00 100, defocus blur perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.30
FR, ru.r
JAACEE
eng.seurte
no. Plueeat
0.30
0.25
ACCTT
0.25
ACC-T
0.25
ACCTT BEET
600-1
9.2
M.LTT
0.20
0.15
0.10
0.30
0.30
30
1.0
200
230
20
30
0.25
0.00
1.00 1.25
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, 1.0
(b) Sources = 50, 6 1.0
(c) Samples 30, Sources = 50
CIFAR-100 elastic transform perturbation
ECE Number Sources
ECE Number Samples
ECE VS.
0.30
0.30
0.25
ACC-T
Source
0.25
HD
ACCTT
ene_seurte
0.25
ADOT
Sture
020-
0.20
0.20
0.35
0.30
813-
50
100
150
200
230
0.05
20
30
40
50
9.25
0.50
0.55
1.00
1.25
1.50 1.75 250
Private Data Sources
Samples Data Source
(a) Samples 10, c
(b) Sources 50.
(c) Samples 30, Sources
CIFAR- 100, fog perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS. E
0.30
n0. /ecll
0.55
AGDT ECETT
0.25
ACCT
0.25
EGSTT
0.20
RCET
M.L-T
MULT
0.35
0.35
A15-
0.30
0.30
0.05
40
0.05
0.25
1.00
2.00
Private Data Sources
Samples per Data Source
(a) Samples -00 c 1.0
(b) Sources = 50. 6 1.0
(c) Samples = =30, Sources = 50
CIFAR-100, frost perturbation
ECE Number Data Sources
ECE Number Samples
ECE V5.
0.30
0.30
0.30
AGCIT
HB
0.25
H5
ACCT
0.25
HD
0 022-
N.LTT
0.20
0.15
815
0.30
010-
50
100
150
200
250
10
40 Samples per Data Source
20
30
50
0.25
0.50
0.35
1.00
1.25
1.50
1.55
2.50
Private Data Sources
(a) Samples 10, c 1.0
(b) Sources 50, 6 1.0
(c) Samples -30, Sources
24

---

Under review as a conference paper at ICLR 2021
CIFAR-100, 100, Gaussian noise perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.60
FR, ru.r
0.60
no. prueeal
JAACEE
eng.seutee
0.50-
ACCTT
0.50
ACCTT
0.50
ACC-T SEET
0.43
M.L-T
600
0.30
0.30
0.30
0.29
0.20
0.30
0.10
8.1
30
1.50
230
30
0.25
0.00
1.00 1.25
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10,
1.0
(b) Sources = 50, 6 1.0
(c) Samples 30, Sources = 50
CIFAR- 100, glass blur perturbation
ECE Number Sources
ECE Number Samples
ECE VS.
0.63
Source
0.40
ene_seurte
0.40
(91, Stuce
0.35
ACC-T
0.35
HD
ACCTT
0.35
ADD-T
0.30
1
0.25
0.25
0.2
0.20
0.20
0.15
0.10
0.10
50
100
150
200
230
20
30
40
50
0.25
0.50
0.55
1.00
1.25 1.00 1.75 250
Private Data Sources
Samples Data Source
(a) Samples 10, c
(b) Sources 50.
(c) Samples 30, Sources
CIFAR- 100, impulse noise perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS. E
n6, moal
recll
ACD-T ECE-T N.L-T
ACCT
ECS-T
RCET
MLTT
M.-T
0.15
0.05
0.00
40 Samples per Data Source
0.25 1.00
250
Private Data Sources
(a) Samples 100 c 1.0
(b) Sources 50. 1.0
(c) Samples = =30, Sources = 50
CIFAR-100, jpeg compression perturbation
ECE Number Data Sources
ECE Number Samples
ECE
0.30
0.30
0.5-
AGCTT
H2
H5
ACCT
0.25
HD
a20-
N.LTT
0.20
015
0.15
810
0.10
0.65
0.65
50
100
150
200
250
10
40 Samples Data Source
20
30
50
0.25
0.50
0.35
1.00
1.25
1.50
1.55
2.50
Private Data Sources
(a) Samples 10, c
(b) Sources 50, 6 1.0
(c) Samples -30, Sources
25

---

Under review as a conference paper at ICLR 2021
CIFAR-100 100, motion blur perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.30-
FR, re.roal
0.30
no. Plueaal eng.seute
JAACEE
0.25
ACCTT
ACC-T
ACDTT BEET
020
M.L-T
600-1
0.20
0.15
0.15
0.1-
0.30
0.10
0.05
0.05
30
1.50
200
230
30
0.25
0.00
1.00 1.25
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10,
1.0
(b) Sources = 50, 1.0
(c) Samples 30, Sources = 50
CIFAR-106 100, pixelate perturbation
ECE Number Sources
ECE Number Samples
ECE VS.
Source
ene_seuree
(91, Sture
0.10
HD
ACC-T
0.25
ACCTT
0.25
ADD-T
1 828-
0.20
0.20
A15-
0.15
0.15
0.30
0.10
0.65
0.05
8.05
50
100
150
200
230
20
30
40
50
0.25
0.50
0.55
1.00
1.25 1.00 1.75 250
Private Data Sources
Samples Data Source
(a) Samples 10, c
(b) Sources 50,
(c) Samples 30, Sources
CIFAR-100 100, shot noise perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS. E
0.62
n6, moa
0.60
no. /ecal
0.50
0.50
AGD-T
ACCT
0.60
ECE-T
M.L-T
0.40
ECS-T
MLLT
0.40
RCET
M.-T
0.30
0.30
010-
0.10
0.30
0.25
1.00
250
Private Data Sources
Samples per Data Source
(a) Samples 100 c 1.0
(b) Sources = 50. 6 1.0
(c) Samples = =30, Sources = 50
CIFAR-100, snow perturbation
ECE Number Data Sources
ECE Number Samples
ECE
0.3-
peal
0.25
H2
0.25
H5
0.25
HD
ACCIT
ACCT
0.20
N.L.T
0.20
0.20
015
0.15
0.10
810-
50
100
150
200
250
0.25
10
20
30
40
50
0.25
0.50
0.35
1.00
1.25
1.50
1.55
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, c
(b) Sources 50, 6 1.0
(c) Samples =30, Sources
26

---

Under review as a conference paper at ICLR 2021
CIFAR- 100, zoom blur perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.30-
FR, re.roal
0.30
no. Plueaa
0.30
JAACEE
eng.seute
ACCTT
0.25
ACCTT
ACCTT BEET
0 0.0-
M.U-T
600-1
0.20
815
0.15
012-
0.30
0.30
30
1.50
200
230
30
9.25
0.00
1.00 1.25
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, c 1.0
(b) Sources = 50, 1.0
(c) Samples 30, Sources = 50
CIFAR-10 RESULTS
CIFAR-10, unperturbed
ECE vs. Number of Data Sources
ECE vs. Number of samples
ECE VS.
0.07
ro.ncal
PLJ
0.07
JAMLE
0.06
0.06
ACCTT
E02-1
ACC-T
600
0.05
ACC-T SCET
9.05
NULT
0.04
MLLTT
0.04
0.83
0.00
0.25
0.00
1.00 1.25
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, 1.0
(b) Sources = 50, 6 1.0
(c) Samples 30, Sources = 50
CIFAR-10, 10, brightness perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.09
0.68
ne, meal
HE
tre Joane
ECE-T
0.09
0.08
0.07
no meal
ECS-T
0.08
0.57
BCETT
0.07
0.08
MULT
ML-11
A.8-
0.05
0.5
0.64
0.04
0.03
0.88
50
100
150
200
250
20
30
40
50
0.25
0.50
0.75
1.00
1.25 1.50 1.75 2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, c 1.0
(b) Sources 50, 1.0
(c) Samples 30, Sources 50
CIFAR-10, 10, contrast perturbation
ECE Number Data Sources
ECE Number Samples
ECE
0.40
0.40
peri
0.40
HB
0.35
H5
0.35
HD
AGCIT
ACCTT
033-
0.25
1 025
0.20
0.20
0.15
0.35
815-
0.10
50
100
150
200
250
10
20
30
40
50
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, c
(b) Sources 50, 6 1.0
(c) Samples 30, Sources
27

---

Under review as a conference paper at ICLR 2021
CIFAR-10, 10, defocus blur perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.25
0.25
0.25
FR, roul
no. peueeal
JAACEE
0.23
eng.seute
0.17-
0.23
ACC-T
0.20
0.18
ACCTT
0.38
0.20
ACC-T SEET
8.5
M.LT
0.35
0.15
0.32
0.12
0.30
0.3
0.00
0.88
0.08-
0.05
30
1.0
220
30
0.25
0.00
1.00 1.25
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, 1.0
(b) Sources = 50, 1.0
(c) Samples 30, Sources = 50
CIFAR-10 10, elastic transform perturbation
ECE Number Sources
ECE Number Samples
ECE
Soarce
0.16
ene_seurte
0.35
Sture
0.14
816
ACD-T
H3
0:32
0.34
HD
ACCTT
0.14
0.32
ADD-
012
0.30
0.10
0.1
0.88
0.88-
0.06
0.06
0.04
50
100
150
200
230
20
30
40
50
0.25
0.50
0.55
1.00
1.25
1500 1.75 250
Private Data Sources
Samples Data Source
(a) Samples 10, c
(b) Sources 50.
(c) Samples 30, Sources
CIFAR 10, fog perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.18
real
016
0.36
0.36
0.14-
AGD-T
ECE-T
0.14
ACCT
EGS-T
0.34
RCET
N.L-T
0.12
0.32
0.30
0.10
0.10
0.28
0.66
8.6
40
0.00
0.25
1.00
2500
Private Data Sources
Samples per Data
(a) Samples 100 c 1.0
(b) Sources = 50. 1.0
(c) Samples = =30, Sources = 50
CIFAR-10, frost perturbation
ECE Number Data Sources
ECE Number Samples
ECE
peai
0.20
0.20
0.18-
AGCIT
HB
0.18
H5
HD
016-
0.36
0.36
0.14
A12-
0.32
0.32
0.30
010-
0.00
0.30
8.08-
6.06
50
100
150
200
250
10
40 Samples Data Source
20
30
50
0.25
0.50
0.35
1.00
1.25
1.50
1.75
2.50
Private Data Sources
(a) Samples 10, c
(b) Sources 50, 6 1.0
(c) Samples 30, Sources
28

---

Under review as a conference paper at ICLR 2021
CIFAR- 10, Gaussian noise perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.80
no. peceeal
PL
JAACEE
eng.seuree
0.50
0.50
0.50
ACCTT
ACC-T
ACCT SEET
8.43
M.L-T
0.40
600
0.40
9.30
0.30
0.20
0.20
828
0.30
0.10
8.10
30
1.0
230
0.25
0.00
1.00 1.25
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, 1.0
(b) Sources = 50, 1.0
(c) Samples 30, Sources = 50
CIFAR- 10, glass blur perturbation
ECE Number Sources
ECE Number Samples
ECE VS.
0.30
Source
0.30
HD
ene_seurte
0.30
Stare
025-
ACD-T
0.25
ACCT
0.25
ADD-T
0.15
0.15
0.30
0.30
011-
0.05
50
100
150
200
230
20
30
40
50
0.25
0.50
0.55
1.00
1.25
1.00 1.75 250
Private Data Sources
Samples Data Source
(a) Samples 10, c
(b) Sources 50.
(c) Samples 30, Sources
CIFAR 10, impulsc noise perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.50
ne, moa
0.30
recal
0.50
0.45
0.40
AGD-T ECE-T N.L.T
ACCT
0.35-
EGS-T
RCET
A.3-
MULT7
025-
0.20
0.20
015
0.30
0.25
1.00
2.00
Private Sources
Samples per Data Source
(a) Samples -00 c 1.0
(b) Sources = 50. 1.0
(c) Samples = =30, Sources = 50
CIFAR-10, jpeg compression perturbation
ECE Number Data Sources
ECE Number Samples
ECE
0.3-
0.20
peci
0.10
0.18
AGCT
M2
0.36
H5
0.16
HD
0.14
0.14
0.12
0.10
010-
0.08
0.58
a.6-
0.36
005
0.04
0.34
50
100
150
200
250
10
20
30
40
50
0.25
0.50
0.35
1.00
1.25
1.50
1.75
2.50
Private Data Sources
Samples Data Source
(a) Samples 10, c
(b) Sources 50, 6 1.0
(c) Samples 330, Sources
29

---

Under review as a conference paper at ICLR 2021
CIFAR-10, 10, motion blur perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.23
FR, re.e
JAMCEE
0.20
eng.seute
no. Plueeal
0.20
PLJEGE
0.15-
0.18
0.15
ACC-T
ACCTT
ACCTT SEET
11
0.14-
M.L-T
0.12
0.15
600
0.32
0.35
0.12
0.00
0.8-
0.05
0.00
30
1.50
230
30
0.25
0.00
1.00 1.25
2.50
Private Data Sources
Samples per Data Source
(a) Samples 10, 1.0
(b) Sources = 50, 1.0
(c) Samples 30, Sources = 50
CIFAR- 10, pixelate perturbation
ECE Number Sources
ECE Number Samples
ECE VS.
0.40
0.40
0.35
Source
0.35
ene_seurte
0.35
Stare
0.30
ACD-T
0.30
HD
ACC-T
0.30
ADD-T
0.25
0.25
0 0.5
0.20
0.20
0.2
0.35
0.15
0.30
0.88
50
100
150
200
230
20
30
40
50
0.25
0.50
0.55
1.00
1.25
1.00 1.75 250
Private Data Sources
Samples Data Source
(a) Samples 10, c
(b) Sources 50.
(c) Samples 30, Sources
CIFAR- 10, shot noise perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.3
n6, moa
reral
ACD-T ECE-T
0.40
ACO-T
0.40
ECS-T
RCET
M.L.T
0.30
MULT
0.30
023-
0.30
0.30
0.10
0.25
1.00
250
Private Sources
Samples per Data Source
(a) Samples -00 c 1.0
(b) Sources 50. 6 1.0
(c) Samples = =30, Sources = 50
CIFAR-10, snow perturbation
ECE Number Data Sources
ECE Number Samples
ECE
816-
0.36
peal
0.34
011
M2
0.14
H5
HD
AGOIT
0.12
ACCT
012
0.30
815
0.08
0.08
8.8
0.26
8.05
0.04
0.36
50
100
150
200
250
10
20
30
40
50
0.25
0.50
0.35
1.00
1.25
1.50
1.55
2.50
Private Data Sources
Samples Data Source
(a) Samples 10, c
(b) Sources 50, 6 1.0
(c) Samples =30, Sources
30

---

Under review as a conference paper at ICLR 2021
CIFAR 10, zoom blur perturbation
ECE vs. Number of Data Sources
ECE vs. Number of Samples
ECE VS.
0.20
0.18
FR, roul
JAMCEE
0.30
eng
no peceeal
0.38
0
0116-
0.14
ACCTT
N.LT
0.32
0.14
ACCT
0.12
0.14
ACCT
SEET
0.68
0.06
0.06
0.04
30
230
0.25
Private Data Sources
Samples per Data Source
(a) Samples 10,
1.0
(b) Sources = 50. 1.0
(c) Samples 30, Sources = 50
We note that using different clipping thresholds for NLL-T (where the clipped NLL loss is min(clipping threshold, NLL)) can affect its performance slightly. In practice, selecting the op- timal clipping threshold would violate differential privacy, because doing would require access to the labeled test data. However, even under the most favorable threshold, Acc-T significantly outperforms NLL-T. Fig. 52, we show an example NLL-T performance different clipping thresholds for CIFAR-I0 under the "snow" perturbation with a perturbation severity of 1. In this case, using the optimal clipping threshold would improve performance by 0.7% over using clipping threshold of 10, and this improvement comes at cost of privacy violations.
ECE at Different Clipping Thresholds
0.065
Acc-T
0.060
0.050
0.040
10
12
14
Clipping Threshold
Figure 52: Recalibration results for CIFAR-10 10 under the "snow" perturbation with perturbation severity of 1. with different clipping thresholds. Number of sources 50, number of samples per source 30, and 1.0. Acc-T significantly outperfoms NLL.-T regardless of the clipping threshold.
Finally, Table shows the overall median and mean ECE achieved by each recalibration method on CIFAR- 100 with perturbation severity of (the lowest perturbation level). These averages computed over all perturbations, numbers private data sources, numbers of samples per source, and settings from the suite of experiments. Comparing these results to those shown in Table 1, which used a perturbation severity 5 (the highest level), we see that the overall calibration improves for all methods when the degree of domain shift lower, but our proposed algorithm, Acc-T, still outperforms other methods.
31

---

Under review as a conference paper at ICLR 2021
Expected Calibration Error (median / mean) Recalibration method
CIFAR-
No recalibration
0.1036 0.1342
One source
0.1067 / 0.1082
Histogram binning
0.1536 / 0.1570
ECE-T
0.1656 / 1610
NLL-T
0.1419 / 0. 1576
Acc-T
0.0766 0.0841
Table 7: Median and mean expected calibration error (ECE) achieved for domain-shifted data with a perturbation
severity of 1 under differential privacy.
32


Reviews:
[{"review_id": "dkbicLSuEuq", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles the problem of privacy preserving calibration under domain shift, which is an interesting combination of 3 separate problems that may often occur together. The main contribution is the use of accuracy temperature scaling for calibration, which is a good match for differential privacy. Extensive experimental validation is given.\n\nStrengths:\n- The use of accuracy temperature scaling is neat in combination with DP. Whilst not a startling novelty, it is a nice observation that their method Accuracy Temperature scaling (Acc-T) combines so well with differential privacy, and yet does not lose out in terms of utility (and in fact does better under more stringent privacy settings, since fewer DP noise iterations are required.\n- Strong experimental section. Although only on image data, the experiments cover a multitude of shift types, and the baselines are apt for the task.\n\nWeaknesses:\n- In terms of methodology, the paper is only tangentially related to domain shift, since there is a form of fine-tuning on the target dataset (using the overall accuracy) (see §2.2). There are extensive experiments on recalibration under domain shift (using perturbed image datasets). However, there must also be an interplay between the degree of shift, the effectiveness of transfer learning (e.g. fine-tuning) and calibration. This is not really covered\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TgGh-KYSNi7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper studies the problem of recalibrating a classifier under the presence of domain shift and the constraints of differential privacy.  They show how to adapt several algorithms for dealing with domain shift to the paradigm of differential privacy, giving mechanisms that achieve both goals.\n\nStrong Points:\n\n1. Addresses a new, interesting, and practically relevant problem.\n2. Technically strong, good insights and clearly bridges the gap between two distinct areas.  \n3. Well organized and written, very clear for the most part. \n\nWeak Points:\n\n1. The sensitivity of f cannot be bounded.  What the authors propose (Section C.2.1) to address this --- i.e., using a sufficiently large value based on the empirical values --- is not generally accepted.\n\nOther Notes:\n\nI think the ideas are cool.  The reduction to a 1-dimensional minimization problem over T makes sense.  The technical insight to use golden search to reduce queries/noise is clever.  Experiments show that the proposed approach actually works as it is expected to.  My one weak point is a bit concerning however.  Hopefully authors can address it adequately in the rebuttal.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MBsIQ8ykjII", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of privacy-preserving calibration under the domain shift. The authors propose ''accuracy temperature scaling'' with privacy guarantees.  \n \nThe empirical results seem complete. I still have several concerns about the technical part.\n\n \n1. The proposed algorithm is not described very clearly in section 3 and section 4. After spending considerable time reading sections 3 and 4, I still feel hard to follow how they address the domain-shift issues.\n \n2. The privacy part seems like a plug-and-play of the Laplace mechanism. Hence, the technical novelty might be limited. Note that the privacy computation in section 3 based on a naive composition --- ` each $M_i$ satisfies $\\epsilon/k$, the total privacy cost follows $\\epsilon$. I would suggest the authors use recently advanced composition [1,2] for better privacy and utility tradeoffs. Moreover, the calculation of sensitivity seems to be wrong. As the authors claim in Section C.2.1 in the appendix, the sensitivity is technically infinite. They set $\\triangle_f=10$ based on empirical observation, which violates the privacy definition.\n\n[1] The Composition Theorem for Differential Privacy.\n[2] Renyi Differential Privacy.", "extra_scores": {}, "extra_text": {}}, {"review_id": "211VMxAg_6v", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\n \nThe paper studies the problem of classifier recalibration under differential privacy constraints. They propose a framework with a calibrator and several private data sources, and it works as follows. At each iteration, the calibrator queries each source, and the data source sends back the private answer, which will be used to optimize the calibration. They also provide a recalibration technique, accuracy temperature scaling, which is effective under the privacy constraint for the reason of low sensitivity. Rigorous experimental results are provided.\n\n\nReasons for score: \n\n Overall, I am positive about this paper, but I have a few concerns.  I've listed the strengths and weaknesses below. Hopefully, the authors can address my concern in the rebuttal period. I'd be happy to raise my score if I am wrong.\n\nStrengths: \n\n\n1. The problem is well-motivated, giving the rising privacy concern and the importance of recalibration.\n\n2. The choice of the query function is novel for privacy constraint, as it has lower sensitivity compared to the log-likelihood function.\n\n3. They provide extensive experimental results to demonstrate the effectiveness of the proposed method.\n\n \n Weaknesses:\n\n1. I don't see how the algorithm addresses the domain shift problem. But they claim that ''We also fine-tune on the target domain'' in section 2.2.\n\n2. According to Tables 3 and 4 in the appendix, the Acc-T works well compared to others without privacy constraints. As it is expected to have higher biases, I would appreciate it if the authors could provide more details on how they evaluated the error and explanations.\n\n3. The framework seems like federated learning with differential privacy, where a central server only gets private local updates from users and takes the average to optimize the parameters. The framework doesn't seem to be novel, but it can be a novel use of this setting for recalibration. It would be good to add a few sentences discussing the connections to federated learning.\n\n4.. Truncating the log likelihood function can also lower the sensitivity. It would be interesting to see the comparison.\n\nMinor comments:\n\n1. In algorithm 1, the query functions for all d data sources are the same. But the general framework states that they can be different. I just wonder how to design a customized query function for each data source. Or maybe it would be more clear to remove the subscript.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work considers the problem of calibrating a multi-class classifier while preserving differential privacy. It proposes a method Accuracy Temperature Scaling, that aims to achieve consistency rather than calibration. The method is particularly easy to implement under the constraint of DP. The paper then evaluates  the calibration algorithm in the context of domain perturbation/shift and, as the authors demonstrate it outperforms adaptations of other technques to DP.

The strong sides of this work are 
* the first work to study calibration in this setting (albeit that is also a result of the setting being of a relatively narrow interest)
* proposes a new algorithm
* evaluation on multiple benchmarks

The weaknesses
* The method is not justified either by theoretical analysis or clear intuition
* Evaluation of performance in the context of domain shift makes the the presentation somewhat confusing and experiments much more involved but is largely orthogonal to the problem of calibration

Overall the work has merits but also significant issues.
Decision: Reject