A Closer Look at the Calibration of Differentially Private Learners
Authors: Hanlin Zhang
Xuechen Li
Prithviraj Sen
Salim Roukos
Tatsunori Hashimoto
Venue: Submitted to ICLR 2023 (2023)
Keywords: Calibration
Differential Privacy

Abstract: We systematically study the calibration of classifiers trained with differentially private stochastic gradient descent (DP-SGD) and observe miscalibration across a wide range of vision and language tasks. Our analysis identifies per-example gradient clipping in DP-SGD as a major cause of miscalibration, and we show that existing approaches for improving calibration with differential privacy only provide marginal improvements in calibration error while occasionally causing large degradations in accuracy. As a solution, we show that differentially private variants of post-processing calibration methods such as temperature scaling and Platt scaling are surprisingly effective and have negligible utility cost to the overall model. Across 7 tasks, temperature scaling and Platt scaling with DP-SGD result in an average 3.1-fold reduction in the in-domain expected calibration error and only incur at most a minor percent drop in accuracy.

Under review as a conference paper at ICLR 2023
A CLOSER LOOK AT THE CALIBRATION OF DIFFERENTIALLY PRIVATE LEARNERS
Anonymous authors Paper under double-blind review
ABSTRACT
We systematically study the calibration of classifiers trained with differentially private stochastic gradient descent (DP-SGD) and observe miscalibration across a wide range of vision and language tasks. Our analysis identifies per-example gradient clipping in DP-SGD as major cause of miscalibration, and we show that existing approaches for improving calibration with differential privacy only provide marginal improvements in calibration error while occasionally causing large degradations in accuracy. As solution, we show that differentially private variants of post-processing calibration methods such as temperature scaling and Platt scaling are surprisingly effective and have negligible utility cost to the overall model. Across 7 tasks, temperature scaling and Platt scaling with DP-SGD result in an average 3. Ifold reduction in the in-domain expected calibration error and only incur at most a minor percent drop in accuracy.
INTRODUCTION
Modern deep learning models tend memorize their training data in order to generalize better (Zhang et al., 2021; Feldman, 2020), posing great privacy challenges in the form of training data leakage or membership inference attacks (Shokri al., 2017; Hayes et al., 2017; Carlini et al., 2021). To address these concerns, differential privacy (DP) has become popular paradigm for providing rigorous privacy guarantees when performing data analysis and statistical modeling based on private data. In practice, commonly used DP algorithm to train machine learning (ML) models is DP-SGI (Abadi et al., 2016). The algorithm involves clipping per-example gradients and injecting noises into parameter updates during the optimization process.
Despite that DP-SGD can give strong privacy guarantees, prior works have identified that this privacy comes at a cost of other aspects of trustworthy ML, such as degrading accuracy and causing disparate impact (Bagdasaryan et al., 2019; Feldman, 2020; Sanyal et al., 2022). These tradeoff's pose a challenge for preseeriiin ML, as it forces practitioners make difficult decisions how to weigh privacy against other key aspects of trustworthiness. In this work, we expand the study of privacy-related tradeoffs by characterizing and proposing mitigations for the ppriacy--alibrattin tradeoff. The tradeoff is significant as accessing model uncertainty is important for deploying models in safety-critical scenarios like healthcare and law where explainability (Cosmides & Tooby, 1996) and risk control (Van Calster et al., 2019) are needed in addition to privacy (Knolle et al., 2021).
The existence of such tradeoff may be surprising, we might expect differentially private training to improve calibration by preventing models from memorizing training examples and promoting generalization (Dwork et al., 2015; Bassily et al., 2016; Kulynych et al., 2022). Moreover, training with modern pre-trained architectures show a strong positive correlation between calibration and classification error (Minderer et al., 2021) and using differentially private training based on pre-trained models are increasingly performant (Tramer & Boneh, 2021; ; Li et al., 2022b; De et al., 2022). However, we find that DP training has the surprising effect of consistently producing over-confident prediction scores in practice (Bu et al., 2021). We show an example of this phenomenon in simple 2D logistic regression problem (Fig. 1). We find a polarization phenomenon, where the DP-trained model achieves similar accuracy to its non-private counterpart, but its confidences are clustered around either 0 or 1. As we will see later, the polarization insight conveyed by this motivating example transfers to more realistic settings.

---

Under review as a conference paper at ICLR 2023
Our first contribution quantifies existing privacy-calibration tradeoffs for state-of-the-art models that leverage DP training and pre-trained backbones such as RoBERTa (Liu et al., 2019b) and vision transformers (ViT) (Dosovitskiy et al., 2020). Although there have been some studies of miscalibration for differentially private learning (Bu et al., 2021; Knolle et al., 2021), 1), they focus on simple tasks (e.g., MNIST, SNLJ) with relatively small neural networks trained from scratch. Our work shows that miscalibration problems persist even for state-of-the-art private models with accuracies approaching or matching their non-private counterparts. Through controlled experiments, we show that these calibration errors are unlikely solely due to the regularization effects of DP-SGD, and are more likely caused by the per-example gradient clipping operation in DP-SGD.
Our second contribution shows that the canaacycalibbatiion tradeoff can be easily addressed through differentially private variants of temperature scaling (DP-TS) and Platt scaling (DP-PS). To enable these modifications, we provide a simple privacy accounting analysis, proving that DP-SGI based re- calibration on a held-out split does not incur additional privacy costs. Through extensive experiments, we show that DP-TS and DP-PS effectively prevent Dr-trained models from being overconfident and give a 3.ld reduction in in-domain calibration error on average, substantially outperforming more complex interventions that have been claimed to improve calibration (Bu et al., 2021; Knolle et al., 2021).
DP
Noc-private
1.0
Text Accuracy
Test Accurayy
2.5
-25
0.2
0.4
0.6
0.8
1.0
0.2
0.1
0.8
1.0
Model ConEddenc
Model Confidence
2
2
(a) Non-separable Gaussian Data
(b) Calibration comparison of logistic regression w and w/o DP
Figure 1: DP-SGD gives rise to miscalibration for logistic regression. (a) Logistic Regression model (blue line) with 8 on Gaussian data ((I3, 34) )))> where (æ, y) E Rº' x (1, -), (z-b)ly (0, 12x2). b = (1.5, 0) if y = else b- = (0, 1.5), and y is Rademacher distributed. (b) Reliability diagram and confidence histogram. DP-SGD trained classifier, which shows poor calibration with a large concentration of extreme confidence values (Left); the baseline is a standard, non-private logistic regression model trained by SGD, which is much better calibrated (Right).
RELATED WORK
Differentially Private Deep Learning. DP-SGD (Song et al., 2013; Abadi et al., 2016) is a popular algorithm for training deep learning models with DP. Recent works have shown that fine-tuning high-quality pre-trained models with DP-SGD results in good downstream performance (Tramer & Boneh, 2021: Li et al., 2022b; De et al., 2022: Li et al., 2022a). Existing works have studied how ensuring differential privacy through mechanisms such as DP-SGD leads to tradeoffs with other properties, such as accuracy (Feldman, 2020) and fairness (Bagdasaryan et al., 2019; Tran al., 2021; Sanyal et al., 2022; Esipova et al., 2022) (measured by the disparity in accuracies across groups). Our miscalibration findings are closely related to the above privacy-fairnes tradcoff that has already received substantial attention. For example, per-example gradient clipping is shown to exacerbate accuracy disparity (Tran et al., 2021; Esipova et al., 2022). Some fairness notions also require calibrated predictions such calibration over demographic groups (Pleiss et al., 2017; Liu et al., 2019a) or a rich class of structured "identifiable" subpopulations (Hébert-Johnsee et al., 2018; Kim et al., 2019). Our work expands the understanding of tradeoffs between privacy and other aspects of trustworthiness by characterizing privacy civacc-calibration tradeoffs.
Calibration. Calibrated probability estimates match the true empirical frequencies of an outcome, and calibration is often used to evaluate the quality of uncertainty estimates provided by ML models. Recent works have observed that highly-accuranl models that leverage pre-training are often well- calibrated (Hendrycks et al., 2019; Desai & Durrett, 2020; Minderer et al., 2021; Kadavath et al., 2022). However, we find that even pre-trained models are poorly calibrated when they are fine-tuned using DP-SGD. Our work is not the first to study calibration under learning with DP, but we provide
2

---

Under review as a conference paper at ICLR 2023
more comprehensive characterization of privacy- callbbccaiiiaiio tradeoffs and solutions that improve this tradeoff which are both simpler and more effective. Luo et al. (2020) studied private calibration for out-of-domain settings, but did not study whether DP-SGD causes miscalibration in-domain. Angelopoulos et al. (2021) modified split conformal prediction to be only studied vision models and their private models have substantial performance decrease compared to non non-private ones. They also did not study the miscalibration of private models and the causes of the privacy-calibration tradeoff. Knolle et (2021) studied miscalibration, but only on MNIST and small pneumonia dataset. Our work provides a more comprehensive characterization across more realistic datasets, and our comparisons show that our recalibration approach is consistently more effective. Closer to our work, the work by Bu et al. (2021) identified that DP-SGD produces miscalibrated models on CIFAR-10, 10, SNLI, and MNIST. As a solution, they suggested an alternative clipping scheme that empirically reduces the expected calibration error (ECE). Our work differs in three ways: our experimental results cover harder tasks and control for confounders such as model accuracy and regularization;
but they
study transfer learning settings that are closer the slate-ofthheaa art setup in differentially private learning and find substantially worse ECE gaps (e.g. they identify a 43% relative increase in ECE on CIFAR- 10, while find nearly 400% on Food101); compare our simple recalibration procedure to their method and find that DP-TS is substantially more effective reducing ECE.
PROBLEM STATEMENT AND METHODS
Our main goal is to build classifiers that are both accurate and calibrated under differential privacy. We begin by defining core preliminary concepts.
3.1 DIFFERENTIAL PRIVACY
Differential privacy is formal privacy guarantee for randomized algorithm which intuitively ensures that no adversary has high probability of identifying whether a record was included in dataset based on the output of the algorithm. Throughout our work, we will study models trained with approximate-DP) (c, -P algorithms.
Definition 3.1. (Approximate-D DP (Dwork et al., 2006)). The randomized algorithm M: X - y is (c, 6D if for all neighboring datasets X, X' E X' that differ on a single element and all measurable Y P(M()) Y) exp(c)P (M (X') E Y) + 8.
3.2 DIFFERENTIALIS PRIVATE STOCHASTIC GRADIENT DESCENT
The standard approach to train neural networks with DP is using the differentially private stochastic gradient descent (DP-SGD) (Abadi et al., 2016) algorithm. The algorithm operates by privatizing each gradient update via combining per-example gradient clipping and Gaussian noise injection.
Formally, one step of DP-SGD to update with batch of samples B, is defined as
g(++1) = 0(() - -]e { Σ
clipe (VC, (D(.)) +
B
IEB.
where T7 is the learning rate at step t, L (B(t)) is the learning objective, clipo (VL, (0(1)) clips the gradient using clipe (VC, (0(9) = VC, (o()). min (1,C/IVV,, (0'0) l1z) and & is Gaussian noise defined as { ~ N (0 C B= I,) with the standard deviation as the noise multiplier returned by accounting and the expected batch size B. Each step of DP-SGD is approximate-DP. and the final model satisfies approximate-DD DP with privacy leakage parameters that can be computed with privacy loss composition theorems (Abadi al., 2016; Mironov, 2017; Wang et al., 2019b; Dong et al., 2019; Gopi et al., 2021).
3.3 CALIBRATION
A probabilistic forecast is said to be calibrated if the forecast has accuracy p on the set of all examples with confidence p. Specifically, given multi-class classification problem where want to predict a categorical variable based on the observation X, we say that probabilistic classifier hg parameterized by 0 over C classes satisfies canonical calibration if for each p in the simplex A C-1
3

---

Under review as a conference paper at ICLR 2023
and every label y, P(Y y hg (X) p) Py holds.' Intuitively, calibrated model should give predictions that can truthfully reflect the predictive uncertainty, e.g., among the samples to which a calibrated classifier gives confidence 0.1 for class k. 10% of the samples actually belong to class k.
The canonical calibration property can be difficult to verify in practice when the number of classes is large (Guo et al., 2017). Because of this, we will consider simpler top-label calibration criterion in this work. In this relaxation, we consider calibration over only the highest probability class. More formally, we say that a classifier he is calibrated if
Ap [0, 1), PY E arg maxp max ho (X) p²) p";
where p" is the true predictive uncertainty. With the same definition of p", we will quantify the degree to which a classifier is calibrated through the expected calibration error (ECE), defined by
B ll p' E E E arg max ho(X) (X) max hx(x)= p"]]]-
In practice, we estimate ECE by first partitioning the confidence scores into M bins Bi...., BM before calculating the empirical estimate of ECE
Bml aace (Bm) - conf (Bm)l,
ECE Σ
where acc (Bm) = [mm] Dien.,. 1 (2: arg max ha (x:)), conf (Bm) BB Eiep... (x())) and {(x;, yi)} are set of i.i.d. samples that follow distribution P(X,Y* Y). When appropriate, we will also study fine-grane -ne-grai miscalibration errors through the histogram of conf( B..) (the confidence histogram) and plot acc (..) against conf( (mm) (the reliability diagram).
3.4 RECALIBRATION
When models are miscalibrated, post-hoc recalibration methods are often used to reduce calibration errors. These methods alleviate miscalibration by adjusting the log-probability scores generated by the probabilistic prediction model. More formally, consider a score-based classifier that produces probabilistic forecasts via softmax (hg (x)) We can adjust the calibration of this classifier by learning a Is that adjusts the log probabilities and produces a better calibrated forecast softmax( o he (x)).
Typically, the re-calibration function 96 is learned by minimizing proper scoring rule separate validattio/reealiivation set Xreeal with the following optimization problem
min E [(softmaxx (x)))))))]
(4)
Specific examples of this type of post hoc recalibration technique include temperature scaling (Guo et al., 2017), Platt scaling (Platt et al., 1999), and isotonic regression (Zadrozny & Elkan, 2002). These methods differ in their choice of 9a- In this work, we will consider the temperature scaling (x) = x/T) and the Platt scaling (go (x) = Wx + b) with the choice of log loss for e.
3.5 DIFFERENTIALLY PRIVATE RECALIBRATION
As observed in our motivating example (Fig. 1) and later experimental results (Fig. 2), DP training (with DP-SGD and Dd-Adam) tends to produce miscalibrated models. Motivated by the success of recalibration methods such Platt scaling and temperature scaling in the non-private setting (Guo et al., 2017), we study how these methods can be adapted to build well-calibrated classifiers with DP guarantees.
Our proposed approach is very simple and consists of three steps, shown in Algorithm 1. We first split the training set into model training part (Xuain) and validation/recalibrattoo part (X We then train the model using SP-SGD on Xurain- followed by recalibration step using DP-SGD on X Depending the choice of fa. will refer this algorithm either DP temperature scaling (DP-TS) or DP Platt scaling (DP-PS).
We slightly abuse the notation of X and Y.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Differentially Private Recalibration
Input: X = ((1, V/) ..... (Xn: bn validation ratio a.
Initial: Parameters of models hg, recalibrator Iø.
1. Xuain; Xeeaal = RandomSplit(X, a:)
2. Train he (x(x) using DP-SGD to optimize ming IE [e (softmax (ho(x)) , with Xuain
3. Train Is using DP-SGD to optimize ming E [e (softmax (ge o hx(x)) y)] with X
Output: Is o h-)-)
The use of sample splitting for recalibration makes privacy accounting simple. To achieve target (e, 6)-DP guarantee after recalibration, we can simply run both stages with DP-SGD parameters that achieve (c, 5)-DP (Prop. A. While sample splitting does reduce the number of samples available for model training, using 90% of the dataset for Xaain results in minor utility cost for the model training step in practice.
4 EXPERIMENTAL RESULTS
We study three different experimental settings. We first consider in-domain evaluations, where we evaluate calibration errors on the same domain that they are trained on. Results show that using pre-trained models does not address miscalibration issues in-domain. We then evaluate the same models above in out-of-domain settings, showing that both miscalibration and effectiveness of our recalibration methods carry over to the out-of -o-mann----nn setting. Finally, we perform careful ablations to isolate and understand the causes of in-domain miscalibration. In each case, we will show that DP-SGD leads to high miscalibration, and DP recalibration substantially reduces calibration errors.
Models. Our goal is to evaluate calibration errors for state-of of-the-art private models. Because of this, our models are based on transfer learning from a pre-trained model. For the text datasets, we fine-tune RoBERTa-base using the procedure in Li et al. (2022b), and for vision datasets, we perform linear probe of ViT and ResNet-50 features, following Tramer Boneh (2021).
Datasets. Following prior work (Li et al., 2022b), we train on MNLI, QNLI, QQP, SST-2 (Wang et al., 2019a) for the text classification tasks, and perform OOD evaluations on common transfer targets such as Scitail (Khot et al., 2018), HANS (McCoy et al., 2019), RTE, WNLI, and MRPC (Wang et al., 2019a).² For the vision tasks, we focus on the in-domain setting and evaluate on a subset of the transfer tasks in Korbliih et al. (2019) with at least 50k examples.
Methods. As basclines, we train the above models using non-private SGD (NON-PRIVATE), stan- dard DP-SGD (DP), global clipping (Bu et al., 2021) (GLOBAL CLIPPING), and differentially private stochastic gradient Langevin dynamics (Knolle et al., 2021) (DP-SGLD). The last two methods are included to evaluate our simple recalibration approaches against existing methods which are reported to improve calibration.
For our recalibration methods, we run the private recalibration method over the in-domain recali- bration set X in Sec. 3.5 using private temperature scaling (DP-TS) (Guo et al., 2017) and Platt scaling (DP-PS) (Platt et al., 1999; Guo et al., 2017). We also include non-private baseline that com- bines differentially private model training with non-private temperature scaling (DP+NON-PRIVAN TS) as a way to quantify privacy costs in the post-hoc recalibration step. Further implementation details and default aaper-parameters for DP training are in Tab. in Appendix B.
4.1 IN-DOMAIN CALIBRATION
We now conduct in-depth experiments across multiple datasets and domains to study miscalibration (Tab. 1, 2). We train differentially private models using pre-trained backbones, and find that their accuracies match previously reported high performance (Tramer & Boneh, 2021; Li et al., 2022b; De et al., 2022).
However. we find that these same models have substantially higher calibration errors. For example, the linear probe for 10od101 in Fig. 2 has private accuracy within 7% of the non-private counterpart,
TTo match the label space between MNLI and the OOD tasks, we merge "contradiction" and "neutral" labels into single "not-contradiition" label.
5

---

Under review as a conference paper at ICLR 2023
CIFAR-I0
SUN397
Food101
Category
Model
Accuracy
ECE
Accuracy
ECE
Accuracy
ECE
DP
0.7951
0.0903
0.6844
0.1302
0,7582
0.154
Buseline
DP-SGLD
0.7122
0.1331
0.6062
0.1952
0.6476
0.2416
Global Clipping
0.7712
0,0804
0.6215
0.1125
0.7451
0.1017
DP-PS
0.789
0,012
0.674
0.104
0,7543
0.0554
Recalibration
DP-TS
0.789
0.674
0.7543
0.0540
Non-private
DP-Non-private-13
0.789
IX0222
0.674
0X0764
0.7543
010539
Non-private
0.83
0.0794
0.7044
0.1062
0.8245
0.0349
Table 1: The image classification performance tion. Results for e are in Appendix B.3
8) of different models before and after recalibra-
MNLI
QNLI
QQP
SST-2
Category
Model
Accuracy
ECE
Accuracy
ECE
Accuracy
ECE
Accuracy
ECE
DP
0,8281
0.166
0,8503
0.149
U.8685
0.13
0,8922
0.105
Baseline
DP-SGLD
0.7188
0,2625
0.7787
0,2138
0.7917
0,2009
0.82
0,1742
Global Clipping
0,8236
0.1667
0,8502
0.1491
0,8685
0.1296
0,8922
0.1047
Recalibration
DP-PS
0.826
0,0487
0.3464
0,0305
U.8639
0.0672
U.3842
0.0201
DP-IS
0.826
0.0849
0,8464
0.0915
0,8659
0.0635
0,8842
0.0665
Non-private
DP+Non
TS
0.826
0.0849
0,8464
0.0915
0.8659
0.06335
0,8842
010665
prinpprivate
0.8642
0.0699
0.914
0.028
0,9042
0.0891
0,9323
0.0425
Table The text classification performance (e 8) before and after recalibration.
but the ECE is more than 4x that of the non-private counterpart, In the language case, we see similar results on QNLI with a -6% decrease in accuracy but a -4.3x increase in ECE. The overall trend of miscalibration is clear across datasets and modalities (Fig. 2).
DP recalibration. We now turn our attention to recalibration algorithms and see whether DP- TS and DP-PS address in-domain miscalibration. We find that DP-TS and DP-PS perform well consistently over all datasets and on both modalities with marginal accuracy drops (Tab. I and Tab.2). many cases, the differentially private variants of recalibration work nearly well as their non-private counterparts. The ECE values for the private DP-TS and non-private baseline of Or+Noo-private-- are generally close across all the datasets.
We note that both DP-TS and DP-PS perform consistently well, with an average relative (in-domain) ECE reduction of 0.58. Despite being simple, the two methods never underperform Global Clipping and DP-SGL.D in terms of ECE, and can have very close or even higher accuracies despite the added cost of sample splitting.
Qualitative analysis. Examining the reliability diagram before and after DP-TS, we see two clear phenomena. First, the model confidence distribution under DP-SGD highly polarized (Fig. first two panels) with nearly all examples receiving confidences of 1.0. Next, we see that after DP-TS, this confidence distribution is adjusted to cover much broader range of confidence values. In the case of SUN397, after recalibration, we see almost perfect agreement between the model confidences and actual accuracies.
4.2 OUT-OF-DOMAIN CALIBRATION
We complement our in-domain experiments with oft-of-domain evaluations. To do this, evaluate the zero-shot transfer performance of models trained over MNLI, QNLI (Tab. 3) and QQP (Tab. 4).
Non- private
0.05
0.00
MNUI
QNLI
QQP
SST2 Bataset
Food101 SUN397 CIFAR-10
Figure 2: DP trained models display consistently higher ECE than their non-private counterparts.

---

Under review as a conference paper at ICLR 2023
Toxk Arcumiy
16
Text Arcumic
1.
Tex Arcumic
1.3
Teat Test
D4
06
B6
Rd Model Confiilance
04
04
Meddl Cunsfalence
Modil Cunffilance
Model Confiilance
Figure 3: Reliability diagram and confidence histogram before (Left) and after (Right) recalibration using DP-TS. Recalibration parameters are learned on the validation set X of MNLI and SUN397.
Dutaset
Category
Model
Accuracy
Hans
ECE
Accuracy
Scitail
ECE
Accuracy
RTE
ECE
Accuracy
WNLI
ECE
DP
0.5195
0.4786
0.7761
0.2172
0.7437
0.2541
0.4507
0,5492
Baseline
DP-SGLD
0.4996
0.4995
0.7515
0.233
0.6498
0.3169
0.4077
0.5491
Global Clipping
0.5221
0.4747
0.7845
0.2051
0.7076
0.2737
0.4366
0.5632
MNLI
Recalibration
DP-PS
0.3237
0.348
0,7707
0.1089
0.7220
0.1316
0.4366
0.4416
0.5237
0.7707
0.1168
0.7220
0.1593
Non-private
DP private-TS
0.5237
0.3544
0.7707
0.1168
0.7220
0.1393
0.4366
0.4495
Non-private
0.668
0.2687
0,7853
0.1348
0.7906
0.1518
0,4677
DP
0.5046
0.4932
0.729
0.2666
0.5637
0.4407
0,4724
0.5215
Baseline
DP-SGLD Global Clipping
0.5
0.4986
0.7209
0.2723
0.5668
0.4266
0.4225
0.5738
0.5025
0.4971
0.7293
0.2684
0.5199
0.4761
0,4789
0.52
QNII
Recalibration
DP-PS
0.5002
0.1244
0,7377
0.0832
0.5632
0.2578
0146648
0.3464
0.1353
0.5632
Non-private
DP DP Non-privvte--TS
0.5002
0.385
0.7377
0.1353
0.5632
0.3121
0.4648
0.404
Non private
0.538
0.1959
0,7454
0.0690
0.5199
0.3036
0.5493
0,2438
Table 3: The zero-shot transfer NLI performance (e 8) across multiple OOD test datasets.
Our findings are consistent with the in-domain evaluations. Differentially private training generally results in high ECE, while DP-TS and DP-PS generally improve calibration. The gaps out-of-domain are substantially smaller than the in-domain case, as all methods are of low accuracy and miscalibrated out of domain. However, the general ranking of miscalibration methods, and the observation that DP-TS and DP-PS lead to private models with calibration errors on-par to non-private models is unchanged.
4.3 ANALYSES AND ABLATION STUDIES
Finally, we carefully study two questions better understand the miscalibration of private learners: What component of DP-SGD leads to miscalibration? What are other confounders such as accuracy or regularization effects that lead to miscalibration?
Model (Ciloping Nomm]
Non
DP (10)
11.0)
Model (Clippingg
Model
DP (2)
w/o clip
DP (1)
DP
DP
0.84
0.85
0.02
0.04
0.82
0.83
0.14
0.16
0.83 0.85 0.88
0.30 Acc
0.00
Aoc
ECE
Acc
ECE
ECE
(a) 2D synthetic data
(b) MNLI
(c) Gradient clipping ablation
Figure 4: Per-example gradient clipping (e 8) causes large ECE errors in (a) logistic regression on non-separable 2D synthetic data, and (b) fine-tuning RoBERTa on MNLI. (c) Performing only gradient noising leads to high accuracy and low ECE.
Ablation per-example gradient clipping and noise injection. DP-SGD involves per-example gradient clipping and noise injection. To better understand which component contributes more to miscalibration, perform experiments to isolate the effect of each individual component.
On 2D synthetic data (example given in Fig. 1), Fig. 4(a) shows that fixing the overall privacy guarantee (c) and increasing the clipping threshold from DP (0.1) to DP (1) and further to DP (10)

---

Under review as a conference paper at ICLR 2023
MRPC
Dataset
Category
Model
Accuracy
ECE
DP
0.7475
0.252
Baseline
DP-SGLD
0.6936
0.2979
Global Clipping
0.7475
0.252
QQP
DP-PS
0.7426
0.1252
Recalibration
DP-TS
0.7426
0.1796
DP+Non OP+Non-private--S TS
0,7426
0,1796
Non-private
Non-private
0.7255
0.2635
Table 4: The zero-shot transfer paraphrase performance (e 8) from QQP to MRPC.
affect the accuracy only marginally but substantially improve calibration. Repeating this ablation with RoBERTa fine-tuning MNLI (Fig. 4(b)) confirms that increasing the clipping threshold (slightly) decreases ECE but does not substantially impact model accuracy. Finally, Fig. 4(c) shows that completely removing clipping and training with only noisy gradient descent dramatically reduces ECE (and increases accuracy). These results suggest that intensive clipping exacerbates miscalibration (even under a fixed privacy guarantee).
MNLI
0.16
2 0.9974
0.15
Non-private
0.14
ECE
0.12
ECE
0.10
11
0.1
25
0.08
Non-private
0.00
0.84
MNLI
QNU
QQP
SST-2
Food101 SUN397 CIFAR-10
Acc
Dataset
(a) Privacy budget ablation
(b) Comparison with non- privati models with matched accuracies
Figure 5: (a) MNLI performance under varying privacy budgets e. (b) Controlling for accuracy by early stopping non- non-private models to match the DP models does not substantially affect differences in ECE. The accuracy differences are within 1%.
MNLI
Scitail
QNLI
Method
Accuracy
ECE
Accuracy
ECE
Accuracy
ECE
DP
0.8281
0.166
0.7761
0.2122
0.3058
0.4942
Nroateivate
0.8642
0.099
0,7853
0.1348
0.5050
0.4426
La (1e-4)
0.8664
0.0347
0.0822
0.4874
fg (le-3)
0.8672
0.0349
0.7891
0.0816
0.5056
0.4862
fg (1e-2)
0.8620
0.0326
0.7845
0.0845
0.5059
0.4870
fy (1e-1)
0.8684
0.1874
0.786
0.0835
0.5059
0.4872
dropout (0.1)
0.8684
0.1874
0.786
0.08335
0.5059
0.4872
dropout (0.2)
0.8601
0.046
0.7722
0.1076
0.5058
0.487
dropout (0.3)
0,8380
0.0629
0.7423
0.1523
0,5050
0.486
early stopping (2)
0.8423
0.0288
0.7806
0.0662
0.5050
0.4818
early stopping
0.8572
0.0299
0.78
0.094
0.5058
0.486
early stopping (6)
0.8623
0.0355
0.7837
0.0811
0.5056
0.486
early stopping (8)
0.8684
0.1874
0.786
0.0835
0.5059
0.4872
Table 5: Comparison with non non-private models trained using common regularizers, i.e. & (weight decay factor), dropout (probability) and early stopping (total training epochs). Models are trained on MNLI and evaluated over MNLI, Scitail and QNLI.
Controlling for accuracy and regularization. Accuracy and calibration are generally positively correlated (Minderer et al., 2021; Carrell et al., 2022). This poses question: Does the miscalibration of DP models arise due to their suboptimal accuracy? We find evidence against this in two different experiments.
In the first experiment, we vary é for fine-tuning RoBERTa with DP on MNLI. This results in several models situated linear ECF-accuracy tradeoff curve (Fig. 5(a)). Intuitively, extrapolating this
8

---

Under review as a conference paper at ICLR 2023
curve helps identify the anticipated ECE for DP trained model with given accuracy. Fig. 5(a) shows that when compared to these private models, the non-privv con-private model has substantially lower ECE than would be expected by extrapolating this tradcoff alone. This suggests that private learning experiences a qualitatively different ECE-accuracy tradeoff than standard learning.
In the second experiment, controlled the in-domain accuracy of non-private models to match their private counterparts by early-stopping the non-private models to be within 1% of the DP model accuracy. Fig. 5(b) shows that the ECE gap between the private and non-private models persists even when controlling for accuracy.
More generally, we find that regularization methods such as early stopping impact the ECE-accuracy tradeoff qualitatively differently than DP-SGD. Our results in Tab. 5 show that most other regularizers such as carly-stopping lead to an accuracy-ECE tradeoff, which highly regularized models are less accurate but better calibrated. This is not the case for DP training, where the resulting models are
1.75
0.20
1.50
0.15
1.25
Eval ECE
Eval
Train ECE
1.00
Train Loss
0.10
0.75
DP
Non-private
Non-private
0.50
0.05
0.00
0.00
5000
10000
15000
20000
25000 30000
5000
10000
15000
20000 25000 30000
(a) ECE
(b) Loss
Figure 6: DP-SGD training (c = 8) makes train and eval ECE close but both of them are large. The training dynamics of (a) ECE and (b) Loss both QNLI training and evaluation sets.
both of lower accuracy and less calibrated relative their non-private counterparts. These findings suggest that calibration errors in private and non-private settings may be caused by different reasons the miscalibration of private models may not be due the regularization effects of DP-SGD.
DP training leads to similarly high train and test ECE. Learning algorithms which satisfy tight DP guarantees known to generalize well, meaning that the train (empirical) and test (population) losses of a DP trained model should be similar (Dwork et al., 2015; Bassily et al., 2016). In QNLI with DP-SGD
controlled experiment,
fine-tune RoBERTa train-test gaps for both ECE and loss are smaller for DP models than the non-private ones (Fig. 6). Yet, for DP trained models, both the train and test ECEs are high compared to the non-private model. Interestingly, these observations with DP trained models are very different from what's 's seen in miscalibration analyses of non-private models. For instance, Carrell al. (2022) showed that non-private models tend to be calibrated on the training set but can be miscalibrated on the test set due to overfitting (large calibration generalization gap). Our results show that DP trained models have small calibration generalization gap, but are miscalibrated on both the training and test sets.
8) and observe that the
CONCLUDING REMARKS
In this work, we study the calibration of ML models trained with DP-SGD. We quantify the miscali- bration of DP-SGD trained models and verify that they exist even using state-of-the-art pre-trained backbones. While the calibration errors are substantial and consistent, we show that adapting existing post-hoe calibration methods is highly effective for DP-SGD models. We believe it is open question whether it is possible to leverage the generalization guarantees of SP-SGD to naturally obtain similarly well-calibrated models without the use of sample-splittig and recalibration.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In SIGSAC, 2016. 1,
Anastasios Angelopoulos, Stephen Bates, Tijana Zrnic, and Michael Jordan. Private prediction sets. arXiv preprini arXiv:2/02.06620, 2021. 3
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. NeurlPS, 2019. 1,2
Raef Bassily, Kobbi Nissim, Adam Smith, Thomas Steinke, Uri Stemmer, and Jonathan Ullman. Algorithmic stability for adaptive data analysis. In STOC, 2016. 1,9
Zhiqi Bu, Hua Wang, Qi Long, and Weijie Su. On the convergence and calibration of deep learning with differential privacy. arXiv preprint arXiv:2106.07830 2021. 1, 3, 14
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagiclski, Ariel Herbert- Morbert-Voss,, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security, 2021.
Annabelle Carrell, Neil Mallinar, James Lucas, and Preetum Nakkiran, The calibration generalization gap. arXiv preprint arXiv:2210.0196/ 2022. 8, 9, 15
Leda Cosmides and John Tooby. Are humans good intuitive statisticians after all? rethinking some conclusions from the literature on judgment under uncertainty. cognition, 1996.
Soham De, Leonard Berrada, Jamie Hayes, Samuel Smith, and Borja Balle. Unlocking high- accuracy differentially private image classification through scale. arXiv preprint 221::22204116500 2022. 1, 5
Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In EMNLP, 2020. 2, 15
Jinshuo Dong, Aaron Roth, and Weijie J Su. Gaussian differential privacy. RSS, 2019. 3
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2020. 2, 15
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In TCC, 2006. 3
Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Leon
Roth. Preserving statistical validity in adaptive data analysis. In STOC, 2015. 1,
Maria S Esipova, Atiyeh Ashari Ghomi, Yaqiao Luo, and Jesse Cresswell. Disparate impact in differential privacy from gradient misalignment. arXiv preprint arXiv:2206.07737, 2022. 2
Vitaly Feldman. Does learning require memorization? a short tale about long tail. In STOC, 2020. 1.2
Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. NeurlPS, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017. 5, 14, 15
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership
inference attacks against generative models. PoPETs, 2017.
Ursula Jehero-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Cali- bration for the (compuaationall-ddenfffabbb masses. ICML, 2018.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness
and uncertainty. In ICMI, 2019. 2, 15
10

---

Under review as a conference paper at ICLR 2023
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly)
know what they know. arXiv preprint ar arXiv:2207.05221/ 2022. 2, 15
Tushar Khot, Ashish Sabharwal, and Peter Clark. SciTail: textual entailment dataset from science question answering. In AAAI, 2018. 5
Michael P Kim, Amirata Ghorbani, and James Zou. Multiaccuraay: Black-box post-processing for fairness in classification, In A/ES, 2019.
Moritz Knolle, Alexander Ziller, Dmitrii Usynin, Rickmer Braren, Marcus R Makowski, Daniel Rueckert, and Georgios Kaissis. Differentially private training of neural networks with langevin dynamics for calibrated predictive uncertainty. arXiv preprint arXiv:2107.04296 2021. 1, 3, 14
Simon Korbblith, Jonathon Shlens, and Quoc Le. Do better imagenet models transfer better? In CVPR, 2019.
Bogdan Kulynych, Yao-Yus Yua- Yang. Yaodong Yu, Jarostaw Blasiok, and Preetum Nakkiran. What you see is what you get: Distributional generalization for algorithm design in deep learning. NeurlPS, 2022. 1, 15
Xuechen Li, Daogao Liu, Tatsunori Hashimoto, Huseyin A Inan, Janardhan Kulkarni, Yin Tat Lee, and Abhradeep Guha Thakurta. When does differentially private learning not suffer in high dimensions? NeurlPS, 2022a.
Xuechen Li. Florian Tramer, Percy Liang. and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In ICLR, 2022b. 1, 5, 14
Lydia T Liu, Max Simchowitz, and Moritz Hardt. The implicit fairness criterion of unconstrained learning. In ICMI., 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / 1692, 2019b.
Rachel Luo, Shengjia Zhao, Jiaming Song, Jonathan Kuck, Stefano Ermon, and Silvio Savarese. Privacy preserving recalibration under domain shift. arXiv preprint arXiv:2008.0960 2020. 3
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In ACL, 2019.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration modern neural networks. NeurlPS, 2021. 1, 2, 8, 15
Ilya Mironov. Rényi differential privacy. In CSF, 2017.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 1999. 4.
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On faimess and calibration. NeurIPS, 2017. 2
Amartya Sanyal, Yaxi Hu, and Fanny Yang. How unfair is private learning ? In AISTATS, 2022. 1, 2
Reza Shokri, Mareo Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In SP, 2017.
Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differen- tially private updates. In GlobalSIP, 2013. 2
Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). ICI.R, 2021. 1, 2,
11

---

Under review as a conference paper at ICLR 2023
Cuong Tran, My Dinh, and Ferdinando Fioretto. Differentially private empirical risk minimization under the faimess lens. NeurIPS, 2021.
Ben Van Calster, David J McLernon, Maarten Van Smeden, Laure Wynants, and Ewout W Steyerberg. Calibration: the achilles heel of predictive analytics. BMC medicine, 2019. 1
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE; A multi-task benchmark and analysis platform for natural language understanding. 2019a. ICLR.
Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differential privacy and analytical moments accountant. In AISTATS, 2019b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers:
State-of-the-ar natural language processing. EMNLP, 2020. 14
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek,Jhh Nguyen, Sayan Ghosh, Akash Bharadwaj. Jessica Zhao, et al. Opacus: User-friendly differential privacy library pytorch. arXiv preprint arXiv: 109. /2298, 2021. 14
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probabil- ity estimates. In KDD, 2002. 4
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 2021.
12

---

Under review as a conference paper at ICLR 2023
A PRIVACY ANALYSIS FOR INDEPENDENT RELEASES WITH A PARTITION OF DATA
Our post-processing calibration setup requires splitting the original (private) training data into two disjoint splits where one of which is used solely for training and the other solely for post hoc recalibration. Given that both the training and post hoc recalibration algorithms to ask what is the overall privacy spending of the joint release. While one can essentially resort to any "off-the-shelf" privacy composition theorem, we note that our setup the splits of data used in the two algorithms are disjoint, and thus a tighter characterization of privacy leakage is possible. The following is common knowledge, and we only include the proof for completeness.
DP, it is natural
Proposition A.1. Let M1 : X1 - Y and M2 X2 x y + 2 be (e, -eDP algorithms consuming independent random bits operating on disjoint splits of the dataset. Then, the algorithm M Y x Z defined by
M(X) = (y,2), == (X1), == ==M2(X2
where (X1, X2) is partition of X determined through some procedure independent on X, is also (D..)-DP.
Proof. Let X and X' be neighboring datasets. Suppose that the first component in both partitions is the same, i.e., X = (X1, X2). and X" (2), where X₂ and X2 are neighhoring. Then, M is (e, 6)- directly follows from that M2 is (e, 8)-D..
The more subtle case is when the second component in both partitions is the same. Specifically, suppose that X (X1, X2), and X' (x( where X₁ and Xi are neighboring. Let R denote the random variable that controls only the randomness of M2, i.e., conditioned a draw of R r, M2 a deterministic function. With slight abuse of notation, we denote this deterministic function by M2(r). Let O = eo {01} } O2(01) C y x Z be a subset of the codomain. Define the following shorthand for the preimage M2 conditioned on
M2(r) (22)) = fu M2(r)(X2.v) E).
(X2,
Then, we have
Pr (M((() R=] Σ Pr (M = 01) Pr (M22(X2)) O2(01) R
01E01
Σ Pr(M = 01) o)) (M) 02(01))
a0E
Σ Pr (M((X)) o Mi(X)) M2(r) (X2,
O0
Pr (MI(X)) Uoi (o()) (x2,00))))
se Pr (xi( UoIEO ((0))n (x2. 02(01))) +
=e =c'PP(M(X)) O
Since the above holds for all draws of R, we conclude that Pr (M(X) E c)) Se Pr 8 for all neighboring X and X' which differ only in their first components. This concludes the proof.
e Pr(((X)) E 0)+
B EXTENDED EXPERIMENTAL DETAILS AND RESULTS
B. SETTINGS FOR SYNTHETIC EXPERIMENTS
For synthetic experiments, we generate two-dimensional mixture Gaussian data of size 10k. The distance between the centers of two class data shifts by a constant, which is set to be 2 * 1.3. We use logistic regression to do the binary classification. We set the amount of data points from each class as 5k and batch size as 4k. We include the results with different maximum gradient norm C. E (0.1, 0.5, 1) of DP training.
13

---

Under review as a conference paper at ICLR 2023
Dataset
CIFAR-10
SUN397
Food101
MNLI
QNLI
QQP
SST-2
Learning rate
2c-3
le-2
le-4
5e-4
lc-3
5c-4
le-3
Batch size
32
32
32
6,000
2,000
6,000
1,000
LR decay
False
False
False
True
True
True
True
Epochs
10
10
10
18
6
18
3
Weight decay
le-4
le-4
le-4
0
0
Clipping norm
1.0
1.0
1.0
0.1
0.1
0.1
0.1
Privacy budget
3,
3,8
3,8
8
8
8
Validation ratio
0.1
Noise scale
calculated numerically so that a DP budget of (c. 5) is spent after E epochs
Table Default hyperparameter of DP finetuning over different datasets for reproducibility. Batch size is based on unit batch size 20 with different amount of gradient accumulation steps. We use the validation ratio, the proportion of validation set, split the training set for tuning recalibration methods.
B.2 IMPLEMENTATION DETAILS
We use pre-trained checkpoints and trainers from Huggingface library (Wolf al., 2020) NL.P experiments. We do linear probe for CV experiments using ResNet50 for CIFAR-10, VIT for SUN397 and Food101. We use the modified Opacus privacy engine (Yousefpour et al., 2021) from (Li ct al., 2022b), which computes per-example gradients for transformers. We compare DP training with popular regularizers used for finetuning like lz, dropout and early stopping over NLP datasets. lz is the weight decay rate {le
le 2, 3, le - 4] during optimization. We apply dropout both hidden and attention layers of transformers, which takes the value in {0.1, 0.2, 0.3, 0.4). We do early stopping by setting the maximum amount of training epochs to be smaller, i.e. values in {2, 4, 6, 8). The default hyper-parameters for bg, dropout, early stopping are le 1. 0.1, 8 respectively so some of the results in Tab.5 are reused.
For recalibration training, we use fixed amount of epochs without hyper- parameter tuning to avoid privacy leakage of validation sets. We initialize the temperature parameter in DP-TS as 1.0 and train 100 epochs for all the tasks except Food1001 (which uses 30 epochs) using DP-SGD with a 0.1 learning rate, 10 maximum gradient clipping norm, and linearly decayed learning rate scheduler. We adapt multiclass extensions for Platt scaling by considering higher-dimensional parameters (Guo et al., 2017).
For baselines, grid search the maximum norm bound Z {100, 500, 1000} and epochs over {6, 8, 18} for global clipping (Bu et al., 2021); ; we use pre-noise scale 0.046, temperature 7 = 6.08, exponential learning rate decay with learning rate 0.005 and decay factor 0.028 as suggested by Knolle et al. (2021).
B.3 ADDITIONAL IMAGE CLASSIFICATION RESULTS
Tab. 7, we give additional results when we have a smaller privacy budget 3. We see consistent results that DP fine-tuning gives poor calibration performance while DP-TS and/or DP-PS can recalibrate the classifiers effectively.
CIFAR-10
SUN397
Food101
Category
Model DP
Accuracy
ECE
Accuracy
ECE
Accuracy
ECE
0.7912
0.0916
0,6751
0.2806
0.7097
0.2464
Baseline
DP-SGLD
0.6953
0.1595
0.562
0.3295
0,6217
0.2834
Global Clipping
0.7659
0.0782
0.6345
0.285
0,6853
0.2276
DP-PS
0.7823
0.0109
0.6694
0.2826
0.7084
0.0626
Recalibration
DP-TS
0.7823
0.0217
0.6694
0.0183
0.7084
0.0601
OP+Non-private-- TS
0.7823
0.02T8
0.6694
0.019
0.7084
0.0598
Non-private
Non-prite Non-private
0.83
0.0794
0.7044
0.1062
0.8245
0.0349
Table 7: The image classification performance (c = 3) of different models before and after recalibra-
tion across datasets.
14

---

Under review as a conference paper at ICLR 2023
B.4 ADDITIONAL ABLATION STUDIES
Label noise injection. All of the datasets we consider have labels that are designed to be unam- biguous, and the Bayes optimal predictor would produce a confidence histogram that is concentrated at 1.0. In this case, we might wonder whether the polarized confidence histograms observed in Fig. are an artifact for datasets with unambiguous labels.
To understand this, we intentionally inject label noise into MNLI and study how this changes the behavior of DP-SGD and non-private learning algorithms. Specifically, we uniformly corrupt training labels by selecting a uniform random class with probability E (0.6, 0.8}. We compare DP-SGD trained models and non-private models with 0.2 dropout regularization. The confidence histograms in Fig. 7 clearly demonstrate that differentially private models result in 100% confidence, even when the Bayes optimal classifier can be al most 60% confident. This shows that DP-SGD trained model's 's miscalibration behavior that results in near 100% confidence is not driven by dataset's 's label distribution and this behavior is likely to be even worse on tasks with inherent label uncertainty.
DP (9.4)
Non private (9.4)
DP (0.5)
Noo private (0.5)
Teat Assenny
Tax Asceni
Taa Ascuncy
Taa Ascuncy
0.5
0.5
0.5
0.5
Costideace
Contideace
Coufiduace
Confidnace
Figure 7: Reliability diagram and confidence histogram for label noise settings with different models (corruption rates) trained on MNLI. For comparison, non-private models are included.
C REMARKS ON CORRELATIONS BETWEEN ACCURACY AND CALIBRATION
In general, the correlations between accuracy and calibration are not clearly understood even for non-private learners many factors impact calibration such architecture, regularization, optimization, data distribution, overparameteriiztion,, etc. Below we include some notable empirical findings. Convolutional networks like ResNets and DenseNets 2017). However, Minderer et al. (2021) show that modern models like ViT (Dosovitskiy et al., 2020) are better calibrated compared to past models; modern neural networks tend to have strong positive correlation between calibration and classification error; model architectures matter greatly in calibration properties. Using pre-traine pre-training can improve model uncertainty and calibration (Hendrycks et al., 2019; Desai & Durrett, 2020; Minderer et al., 2021; Kadavath et al,, 2022). Regularizations like gradient noise injection can promote stability and distributional generalization so good calibration over the training set can transfer to the test set (Kulynych et al., 2022). Carrell et al. (2022) empirically shows that popular models with small generalization gaps will have small test calibration errors.
be miscalibrated (Guo al.,
Realizing the above observations, it is possible that the per-example gradient clipping and gradient noise injection in DP-SGD can contribute to both accuracy and calibration in different ways. There- fore, carefully control the accuracy and regularization when conducting analyses and drawing conclusions (Tab. 5, Fig. 5(a) and 5(b), Fig. 7). However, even with the confounding controls above, DP-SGD trained models are still miscalibrated. In other words, the reason for the finding that private learners are much more miscalibrated than non-private counterparts is less likely to be the unambiguous labels in datasets, accuracy discrepancy or regularization effects of DP-SGD but more likely to be the per-example gradient clipping operation.
15


Reviews:
[{"review_id": "_3YXQtK2D0m", "reviewer": "Reviewer_ckRn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors  identify the per-example gradient clipping in differentially private stochastic gradient descent (DP-SGD) as a major cause of miscalibration, and argue that existing baselines for improving private calibration only provide small improvements in calibration error while occasionally causing large degradation in accuracy. Authors propose using post-processing calibration. \n\n", "summary_of_the_review": "The main emphasis of this work is empirical evidence. More detailed analysis is needed. \n", "strength_and_weaknesses": "Strengths:\n\n\nAuthors have a good grasp of some of the challenges of differentially private computations in the context of the paper. The paper is generally well-written. The empirical context is well motivated. \n\n\nWeakness:\n\n\nThis paper is primarily on empirical evidence for a recalibration idea. It is not clear to me that the recalibration preserves differential privacy, although there is claim in the paper about this. Perhaps additional discussion on this in an eventually revised manuscript is in order. There does not seem to be adequate theoretical justification about when, why and how the standard DP-SGD method fails, or when, why and how the proposed recalibration will work, or when it will fail. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Cl2Qk34HSaX", "reviewer": "Reviewer_zkmx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the calibration of state-of-the-art classifiers trained using differentially private stochastic gradient descent (DP-SGD). It shows that there are miscalibration issues in these DP classifiers even when their accuracy matches that of their non-private counterparts. The likely cause of that is conjectured to be the repeated per-example gradient clipping in DP-SGD. The other contribution is addressing these miscalibrations using DP versions of temperature scaling (TS) and Platt scaling (PS), which are almost as good as their non-private counterparts. This work is mostly empirical in nature, and uses multiple datasets and models, with applications in text-processing and computer vision, and has experiments to evaluate both in-domain and out-of-domain calibrations.", "summary_of_the_review": "I am not convinced about the overall quality of this paper. The contributions don't seem that significant to me, unfortunately, especially given the lack of motivation of the problem in this draft and the not-so-novel technical ideas. The writing quality wasn't that great either, but that doesn't matter much to me as far as the quality of the work is concerned -- my feedback there is just to help the authors improve their manuscript. I do acknowledge that the authors seem to have achieved the claims of this paper. All this said, I will be open to changes in my final score based on the authors' responses though.\n\nUpdate: Based on the responses, I have bumped the score.", "strength_and_weaknesses": "Strengths:\n1. The paper successfully depicts the issue of miscalibrations in DP classifiers (trained via DP-SGD). All the experiments indicate the DP counterparts having more miscalibration than their respective non-private counterparts.\n2. This paper also establishes its second goal of providing a DP method to lower the miscalibration errors significantly. These methods involve the use of DP-SGD again. The experimental results indicate very low miscalibration errors that are similar to those from the non-private methods.\n\nWeaknesses:\n1. I have some concerns about the writing quality. Please, see the section regarding clarity of the submission.\n2. I'm also concerned about the significance of the problem being solved here. I'm curious as to why this problem under DP has not been studied much before this. In other words, why is reducing miscalibrations that important a problem to solve, especially under DP?\n3. The algorithmic contributions don't seem to be that interesting, unfortunately -- just optimising a calibration function using DP-SGD. The techniques aren't novel, and simply involve blackbox uses of DP-SGD. Is the goal of this paper to bring the issue of miscalibrations in DP classifiers to people's attention?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7E5ovuago-p", "reviewer": "Reviewer_s6au", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the calibration of differentially private learners based on stochastic gradient descent. The paper first observes the miscalibration  is due to the per-example gradient clipping. Then, the paper provides differentially private recalibration to reduce  calibration errors. The basic idea is to divide the training dataset into two parts: one part is used to train a classifier, while the other is used to train a recalibration function. Extensive experimental results are reported to show the effectiveness of the differentially recalibration method. ", "summary_of_the_review": "The paper presents a simple and general method for recalibration in a differentially private setting. The paper conducts many experiments to show the effectiveness of the method. The proposed algorithm is not presented in detail and there is no theoretical analysis.", "strength_and_weaknesses": "**strength**\n\nThe proposed method is simple and general. It provides a framework which includes several post-hoc recalibration techniques. There are many experimental results, which show the proposed method works well in practice.\n\n**weakness**\n\nThere is no theoretical analysis of the proposed method. There is no theoretical guarantee and therefore it is not clear how the method would behave in general.\n\nThe idea of extending recalibration to differentially private recalibration seems to be natural. It is not clear whether the extension is novel enough.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4OrxUVnsev", "reviewer": "Reviewer_9fSP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the trade-off between differential privacy and classifier calibration. In particular, the main observation is that classifiers trained with the standard DP-SGD algorithm can be highly miscalibrated. To this end, the authors propose a natural and intuitive solution to fix this issue using differentially private recalibration. First, the algorithm runs DP-SGD in order to obtain the standard classification model with a subset of the data and then runs DP-SGD again in order to find a post-hoc recalibration function using the remaining data. Finally, it outputs the composition of the two functions.\n", "summary_of_the_review": "In short, even if there is no theoretical contribution, I believe that the paper is beyond the acceptance threshold, since its message can potentially raise some theoretical questions. The experimental observations are interesting.", "strength_and_weaknesses": "The paper provides extensive experimental evidence about the impact of differential privacy to the calibration property of classifiers. I find that the provided empirical observations are interesting and may raise some nice theoretical questions too. More to that, the paper is nicely written. The only issue is that the paper does not have any theoretical contribution.\n\nA potential question is whether variants of DP-SGD or other training methods would also cause such miscalibration phenomena. Is gradient clipping the only obstacle towards private training that is well-calibrated?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject