Title: Training Private Models That Know What They Don’t Know
Venue: NeurIPS 2023
Accepted: yes
Keywords: differential privacy

selective classification

selective prediction

abstain option

reject option

uncertainty quantification

misclassification detection
Abstract: Training reliable deep learning models which avoid making overconfident but incorrect predictions is a longstanding challenge. This challenge is further exacerbated when learning has to be differentially private: protection provided to sensitive data comes at the price of injecting additional randomness into the learning process. In this work, we conduct a thorough empirical investigation of selective classifiers---that can abstain under uncertainty---under a differential privacy constraint. We find that some popular selective prediction approaches are ineffective in a differentially private setting because they increase the risk of privacy leakage. At the same time, we identify that a recent approach that only uses checkpoints produced by an off-the-shelf private learning algorithm stands out as particularly suitable under DP. Further, we show that differential privacy does not just harm utility but also degrades selective classification performance. To analyze this effect across privacy levels, we propose a novel evaluation mechanism which isolates selective prediction performance across model utility levels at full coverage. Our experimental results show that recovering the performance level attainable by non-private models is possible but comes at a considerable coverage cost as the privacy budget decreases.

Training Private Models That Know What They Don't Know

Abstract

Training reliable deep learning models which avoid making overconfident but incorrect predictions is a longstanding challenge. This challenge is further exacerbated when learning has to be differentially private: protection provided to sensitive data comes at the price of injecting additional randomness into the learning process. In this work, we conduct a thorough empirical investigation of selective classifiersthat can abstain under uncertainty-under a differential privacy constraint. We find that some popular selective prediction approaches are ineffective in a differentially private setting because they increase the risk of privacy leakage. At the same time, we identify that a recent approach that only uses checkpoints produced by an off-the-shelf private learning algorithm stands out as particularly suitable under DP. Further, we show that differential privacy does not just harm utility but also degrades selective classification performance. To analyze this effect across privacy levels, we propose a novel evaluation mechanism which isolates selective prediction performance across model utility levels at full coverage. Our experimental results show that recovering the performance level attainable by non-private models is possible but comes at a considerable coverage cost as the privacy budget decreases.

1 Introduction

State of the art machine learning (ML) models are gaining rapid adoption in high-stakes application scenarios such as healthcare [Challen et al., 2019, Mozannar and Sontag, 2020], finance [Vijh et al., 2020], self-driving [Ghodsi et al., 2021], and law [Vieira et al., 2021]. However, major challenges remain to be addressed to ensure trustworthy usage of these models. One major concern in sensitive applications is to protect the privacy of the individuals whose data an ML model was trained on. To prevent privacy attacks on ML models, ( ε, δ ) differential privacy (DP) [Dwork et al., 2014] has emerged as the de-facto standard with widespread usage in both academic and industrial applications.

While the introduction of DP successfully protects against privacy attacks, it also limits model utility in practice. For instance, the canonical algorithm for training models with DP, DP-SGD [Abadi et al., 2016], clips the per-sample gradient norm and adds carefully calibrated noise. These training-time adaptations frequently lead to degradation in predictive performance in practice. This is especially a problem for datasets containing underrepresented subgroups whose accuracy has been shown to degrade with stronger DP guarantees [Bagdasaryan et al., 2019]. Faced with this challenge, it is therefore of vital importance to detect samples on which a DP model would predict incorrectly on.

One popular technique used to detect inputs that the model would misclassify with high probability is given by the selective classification (SC) framework [Geifman and El-Yaniv, 2017]: by relying on a model's uncertainty in the correct prediction for an incoming data point, the point is either accepted with the underlying model's prediction, or rejected and potentially flagged for additional

downstream evaluation. Hence, SC introduces a trade-off between coverage over the test set (i.e., the goal of accepting as many data points as possible) and predictive performance on the accepted points (i.e., the goal of making as few mistakes as possible). Although identification of misclassified points seems of particular importance under differential privacy, the application of selective prediction to private models is under-explored to date. While lots of past works have studied privacy or selective prediction in isolation, best practices for selective prediction under differential privacy have not been established yet. In this work, we are the first to answer the question of whether selective classification can be used to recover the accuracy lost by applying a DP algorithm (at the expense of data coverage).

To analyze the interplay between differential privacy and selective classification, we first show that not all approaches towards selective classification are easily applicable under a differential privacy constraint. In particular, approaches relying on multiple passes over the entire dataset to obtain the full SP performance profile [Geifman and El-Yaniv, 2019, Lakshminarayanan et al., 2017] suffer significantly under differential privacy. This is due to the fact that the worst-case privacy leakage increases with each analysis made of the dataset which means that each training run forces more privacy budget to be expended. On the other hand, our analysis shows that an SC approach based on harnessing intermediate model checkpoints [Rabanser et al., 2022] yields the most competitive results. Notably, this approach performs especially well under stringent privacy constraints ( ε = 1 ).

Next, we find that differential privacy has a more direct negative impact on selective classification that goes beyond the characteristic drop in utility. Based on a simple synthetic experiment, we observe a clear correlation between differential privacy strength and wrongful overconfidence. In particular, even when multiple private models trained with different ε levels offer the same utility on the underlying prediction task, we show that stronger levels of DP lead to significantly decreased confidence in the correct class. This reduction effectively prevents performant selective classification.

Motivated by this observation, we realize a need to disentangle selective classification from model utility; SC ability and accuracy should be considered independently. To that end, we show that the evaluation metric typically used for non-private selective classification is inappropriate for comparing SC approaches across multiple privacy levels. The in-applicability of the current method stems from the fact that full-coverage accuracy alignment is required across experiments. In particular, accuracyaligning multiple DP models via early-stopping frequently leads to a more stringent than targeted privacy level, rendering the intended comparison impossible. Due to this limitation of previous evaluation schemes, we propose a novel metric which isolates selective classification performance from losses that come from accuracy degradation of the classifier overall (as frequently caused by DP). The performance metric we propose to tackle this problem computes an upper bound on the accuracy/coverage trade-off in a model-dependent fashion and measures the discrepancy between the actual achieved trade-off to this bound. Using this score, we determine that, based on a comprehensive experimental study, differential privacy does indeed harm selective prediction beyond a loss in utility.

We summarize our key contributions below:

We provide a first analysis on the interplay between selective classification and differential privacy. As a result, we identify an existing SC method as particularly suitable under DP and present an illustrative example showcasing an inherent tension between privacy and selective prediction.

We unearth a critical failure mode of the canonical SC performance metric preventing its off-theshelf usage under DP experimentation. We remedy this issue by introducing a novel accuracydependent selective classification score which enables us to compare selective classification performance across DP levels without explicit accuracy-alignment.

We conduct a thorough empirical evaluation across multiple selective classification techniques and privacy levels. As part of this study, we confirm that selective classification performance degrades with stronger privacy and further find that recovering utility can come at a considerable coverage cost under strong privacy requirements.

2 Background

2.1 Differential Privacy with DP-SGD

Differential Privacy (DP) [Dwork et al., 2014] is a popular technique which allows us to reason about the amount of privacy leakage from individual data points. Training a model with differential privacy

ensures that the information content that can be acquired from individual data points is bounded while patterns present across multiple data points can still be extracted. Formally, a randomized algorithm M satisfies ( ε, δ ) differential privacy, if for any two datasets D,D ′ ⊆ D that differ in any one record and any set of outputs S the following inequality holds:

The above DP bound is governed by two parameters: ε ∈ R + which specifies the privacy level, and δ ∈ [0 , 1] which allows for a small violation of the bound.

DP-SGD. Acanonical way of incorporating differential privacy in deep learning is via Differentially Private Stochastic Gradient Descent (DP-SGD) [Bassily et al., 2014, Abadi et al., 2016]. We describe the DP-SGD algorithm in detail in Algorithm 2 in the Appendix. The main adaptations needed to incorporate DP into SGD are: (i) per-sample gradient computation , which allows us to limit the per-point privacy leakage in the next two steps; (ii) gradient clipping , which bounds the sensitivity (i.e., the maximal degree of change in the outputs of the DP mechanism); and (iii) noise addition , which introduces Gaussian noise proportional to the intensity of gradient clipping.

2.2 Selective Classification

Supervised classification. Our work considers the supervised classification setup: We assume access to a dataset D = { ( x n , y n ) } N n =1 consisting of data points ( x , y ) with x ∈ X ⊆ R d and y ∈ Y = { 1 , . . . , C } . All data points ( x , y ) are sampled independently and identically distributed from the underlying distribution p defined over X × Y . The goal is to learn a prediction function f : X → Y which minimizes the classification risk with respect to the underlying data distribution p as measured by a loss function ℓ : Y × Y → R .

Selective classification. Selective classification augments the supervised classification setup by introducing a rejection class ⊥ via a gating mechanism [El-Yaniv and Wiener, 2010]. This mechanism produces a class label from the underlying classifier if the mechanism is confident that the prediction is correct and abstains otherwise. Note that this mechanism is often directly informed by the underlying classifier f and we make this dependence explicit in our notation. More formally, the gating mechanism introduces a selection function g : X × ( X → Y ) → R which determines if a model should predict on a data point x . If the output of g ( x , f ) undercuts a given threshold τ , we return f ( x ) , otherwise we abstain with decision ⊥ . The joint predictive model is therefore given by:

Evaluating SC performance. The performance of a selective classifier ( f, g ) is based on two metrics: the coverage of ( f, g ) (corresponding to the fraction of points to predict on) and the accuracy of ( f, g ) on accepted points. Note that there exists a tension between these two performance quantities: naively increasing coverage will lead to lower accuracy while an increase in accuracy will lead to lower coverage. Successful SC methods try to jointly maximize both metrics.

To characterize the full performance profile of a selective classifier ( f, g ) , we consider the selective accuracy at coverage level c , formally acc c ( f, g ) , over the full coverage spectrum by computing an area-under-the-curve (AUC) score s AUC as follows:

Each existing selective classification algorithm proposes a g that tries to maximize this metric. We describe popular approaches below and include additional details in Appendix A.2.

Prediction confidence. The traditional baseline methods for selective prediction is the Softmax Response ( SR ) method [Hendrycks and Gimpel, 2016, Geifman and El-Yaniv, 2017]. This method uses the confidence of the final prediction model f as the selection score. To reduce overconfident predictions yielded by this method, confidence calibration [Guo et al., 2017] has been proposed.

Ensembling. To further improve calibration and to reduce variance, ensemble methods have been proposed which combine the information content of M models into a single final model. The canonical instance of this approach for deep learning based models, Deep Ensembles ( DE ) [Lakshminarayanan et al., 2017], trains multiple models from scratch with varying initializations using a proper scoring rule and adversarial training. Then, after averaging the predictions made by all models, the softmax response ( SR ) mechanism is applied. To overcome the computational cost of estimating multiple models from scratch, Monte-Carlo Dropout ( MCDO ) [Gal and Ghahramani, 2016] allows for bootstrapping of model uncertainty of a dropout-equipped model at test time. Another ensembling approach that has recently demonstrated state-of-the-art selective classification performance is based on monitoring model evolution during the training process. To that end, Selective Classification Training Dynamics ( SCTD ) [Rabanser et al., 2022] records intermediate models produced during training and computes a disagreement score of intermediate predictions with the final prediction.

Architecture & loss function modifications. A variety of SC methods have been proposed that leverage explicit architecture and loss function adaptations. For example, SelectiveNet ( SN ) [Geifman and El-Yaniv, 2019] modifies the model architecture to jointly optimize ( f, g ) while targeting the model at a desired coverage level c target . Alternatively, prior works like Deep Gamblers [Liu et al., 2019] and Self-Adaptive Training [Huang et al., 2020] have considered explicitly modeling the abstention class ⊥ and adapting the optimization process to provide a learning signal for this class. For instance, Self-Adaptive Training ( SAT ) incorporates information obtained during the training process into the optimization itself by computing and monitoring an exponential moving average of training point predictions over the training process. Introduced to yield better uncertainty estimates, Liu et al. [2020] employs weight normalization and replaces the output layer of a neural network with a Spectral-Normalized Neural Gaussian Process to improve data manifold characterization.

Uncertainty for DP models. An initial connection between differential privacy and uncertainty quantification is explored in Williams and McSherry [2010], which shows that probabilistic inference can improve accuracy and measure uncertainty on top of differentially private models. By relying on intermediate model predictions, Shejwalkar et al. [2022] has proposed a mechanism to quantify the uncertainty that DP noise adds to the outputs of ML algorithms (without any additional privacy cost).

3 The Interplay Between Selective Classification and Differential Privacy

We now examine the interplay between these approaches to selective classification and training algorithms constrained to learn with differential privacy guarantees. We first introduce a candidate selective classification approach which leverages intermediate predictions from models obtained during training. Then, we study how current selective classification approaches impact DP guarantees, as each access they make to training data increases the associated privacy budget. Next, we investigate how in turn DP affects selective classification performance. Indeed, DP alters how optimizers converge to a solution and as such DP can impact the performance of SC techniques. Last, we discuss how selective classification performance can be fairly compared across a range of privacy levels.

Algorithm 1: SCTD [Rabanser et al., 2022]

Require: Checkpointed model sequence { f 1 , . . . , f T } , query point x , weighting

1: Compute prediction of last model: L ← f T ( x )

2: Compute disagreement and weighting of intermediate predictions:

3: for t ∈ [ T ] do

4: if f t ( x ) = L then a t ← 0 else a t ← 1 5: v ( t ) k

t ← T

6: end for

7: Compute sum score: s sum ← ∑ t v t a t

8: if s sum ≤ τ then accept f ( x ) = L else reject with f ( x ) = ⊥

3.1 Performative Private Selective Classification via Training Dynamics Ensembles

As a classical technique from statistics, ensembling methods are often employed for confidence interval estimation [Karwa and Vadhan, 2017, Ferrando et al., 2022], uncertainty quantification [Lakshminarayanan et al., 2017], and selective prediction [Zaoui et al., 2020]. Under a differential privacy constraint, although for simpler tasks like mean estimation ensembling methods have been proven to be effective [Brawner and Honaker, 2018, Covington et al., 2021, Evans et al., 2019], in this paper

we demonstrate that they are fairly ineffective for selective classification. This is primarily due to the increased privacy cost due to (advanced sequential) composition [Dwork et al., 2006].

In light of this challenge, we expect one recent method in particular to perform well in a private learning setting: selective classification via training dynamics ( SCTD ) [Rabanser et al., 2022] (details described in Algorithm 1). For a given test-time point, SCTD analyzes the disagreement with the final predicted label over intermediate models obtained during training. Data points with high disagreement across this training-time ensemble are deemed anomalous and rejected. Most importantly, while SCTD also constructs an ensemble, only a single training run is used to obtain this ensemble. As a result of the post-processing property of DP, the original ( ε, δ ) -DP guarantee can be maintained. To further motivate the effectiveness of relying on intermediate checkpoints, Shejwalkar et al. [2022] has shown in adjacent work that intermediate predictions can improve the predictive accuracy of DP training, yielding new state-of-the-art results under DP.

3.2 How Do Other Selective Classification Approaches Affect Privacy Guarantees?

We can extend the previous analysis based on post-processing and composition to other SC techniques. This allows us to group selective classification techniques into three classes: (i) directly optimized; (ii) post-processing; and (iii) advanced sequential composition. For directly optimized and postprocessing approaches, we can obtain selective classification for free. On the other hand, compositionbased approaches either require an increased privacy budget or suffer from decreased utility.

Direct optimization. Many selective classification approaches directly modify the loss function and optimization is performed w.r.t. this adapted loss function. As DP-SGD is loss function and architecture agnostic ( ε, δ ) guarantees hold automatically for SC methods that only change the loss function. Applicable to: Softmax Response ( SR ), Deep Gamblers ( DG ), Self-Adaptive Training ( SAT ).

Post-processing. If a function ϕ ( x ) satisfies ( ε, δ ) -DP, then for any deterministic or random function ψ ( · ) , the application of ψ on ϕ , formally ψ ◦ ϕ ( x ) , still satisfies ( ε, δ ) -DP [Dwork et al., 2006]. Applicable to: Monte-Carlo Dropout ( MCDO ), Selective Classification Training Dynamics ( SCTD ).

Advanced sequential composition. If in a set of aggregated functions { ϕ 1 ( x ) , . . . , ϕ M ( x ) } each ϕ i ( x ) satisfies ( ε, δ ) -DP, then releasing the set of all outputs ψ ( x ) = ( ϕ 1 ( x ) , . . . , ϕ M ( x )) satisfies ≈ ( √ Mε,Mδ ) -DP [Dwork et al., 2006]. To maintain ( ε, δ ) -DP over the aggregated output, each function needs to satisfy ≈ ( ε √ M , δ M ) -DP. Applicable to : Deep Ensembles ( DE ), SelectiveNet ( SN ).

3.3 How Does Differential Privacy Affect Selective Classification Performance?

After having examined how current selective classification techniques influence differential privacy guarantees, this subsection considers the opposite effect: what is the impact of differential privacy on selective classification? As we will see by means of an intuitive example, we expect that differential privacy impacts selective classification beyond a loss in utility.

To showcase this, we present a synthetic logistic regression example with the softmax response SC mechanism. We generate data from a mixture of two two-dimensional Gaussians with heavy class-imbalance. Concretely, we generate samples for a majority class { ( x i , 1) } 1000 i =1 with each x i ∼ N ( 0 , I ) and a single outlier point form a minority class ( x ∗ , -1) with x ∗ ∼ N ([10 0] ⊤ , I ) . We then train multiple differentially private mechanisms with ε ∈ {∞ , 7 , 3 , 1 } on this dataset and evaluate all models on a test set produced using the same data-generating process.

We show the outcome of this set of experiments in Figure 1 where we plot the test set and the decision boundary of each model across ε levels. For ε = ∞ , the model successfully discriminates the majority and the minority class. However, the decision boundary is heavily influenced by a single data point, namely the outlier ( x ∗ , -1) . Note that, even though all models with ε ∈ { 7 , 3 , 1 } misclassify the outlier point (i.e., their utility is aligned), the changing decision boundary also increases the model's confidence in predicting the incorrect class for the outlier. Hence, even at an aligned utility level, the softmax response approach for SC performs increasingly worse on the outlier point under stronger DP constraints. We provide additional class-imbalanced results on realistic datasets in Appendix B.2.

5

0

5

10

-

2

.

5

0

.

0

2

.

5

e

=

∞

5

0

5

10

-

2

.

5

0

.

0

2

.

5

e

=

7

.

0

5

0

5

10

-

2

.

5

0

.

0

2

.

5

e

=

3

.

0

5

0

5

10

-

2

.

5

0

.

0

2

.

5

e

=

1

.

0

-

-

-

-

3.4 How Should We Evaluate Selective Classification Under Differential Privacy?

As we have now established, we expect differential privacy to harm selective classification performance. In light of this insight, we are now investigating whether the standard evaluation scheme for SC is directly applicable to the differentially private models.

The default approach of using the SC performance metric introduced in Equation 4 free from its bias towards accuracy is to align different SC approaches/models at the same accuracy. However, accuracy-aligning can have unintended consequences on SC performance, especially under DP. If we are considering SC performance across models trained for multiple ε levels, accuracy-alignment would require lowering the performance of the less private models. One seemingly suitable approach to do this is by early-stopping model training at a desired accuracy level. However, early-stopping a DP training algorithm in fact gives a model with greater privacy than the privacy level targeted by the optimization process. This is a direct consequence of privacy loss accumulation via privacy accounting during training, and so training for less leads to expending less privacy budget. Hence, instead of comparing models with different privacy guarantees at the same baseline utility, early-stopping yields models with potentially very similar privacy guarantees.

To address this limitation, we propose to measure SC performance by how close the observed trade-off is to an upper bound on the achievable SC performance at a given baseline utility level (at 100% coverage). To compute this metric, we first derive an upper bound on selective classification performance (as a function of coverage) for a given baseline accuracy. We obtain this characterization by identifying the optimal acceptance ordering under a particular full-coverage accuracy constraint.

Definition 3.1. The upper bound on the selective classification performance for a fixed full-coverage accuracy a full ∈ [0 , 1] and a variable coverage level c ∈ [0 , 1] is given by

Measuring the area enclosed between the bound acc ( a full , c ) and an SC method's achieved accuracy/coverage trade-off acc c ( f, g ) yields our accuracy-normalized selective classification score.

Definition 3.2. The accuracy-normalized selective classification score s a full ( f, g ) for a selective classifier ( f, g ) with full-coverage accuracy a full is given by

We provide additional intuition for both the proposed upper bound and our score as part of Figure 2.

Bound justification. To understand the upper bound given in Equation 5, note that a full-coverage accuracy level of a full means that a fraction a full of points are correctly classified. An ideal selective classification method for this model would, by increasing the coverage c from 0 to 1, first accept all points that the model classifies correctly: this gives the optimal selective classification accuracy for coverage c ≤ a full of 100% . For any coverage level c > a full , note that the accuracy at the coverage is

0

0

.

2

0

.

4

0

.

6

0

.

8

1

0

0

.

5

1

a

full

c

=

a

full

correct points

incorrect points

Coverage (

c

)

acc

(

a

full

, c

)

0

0

.

2

0

.

4

0

.

6

0

.

8

1

0

.

4

0

.

6

0

.

8

1

Coverage (

c

)

acc

(

a

full

, c

)

acc

c

(

f, g

)

s

a

full

(

f, g

)

the number of correct points accepted divided by the total number of points accepted. The largest value this can take is by maximizing the numerator, and the maximum value for this (as a fraction of all points) is a full . Plugging this into the previous statement for accuracy at coverage c , we have the best selective accuracy at a coverage c is a full c . This derives the upper bound stated above as a function of coverage. We remark that this bound is in fact achievable if a selective classifier separates correct and incorrect points perfectly, i.e., it accepts all correct points first and then accepts and increasing amount of incorrect points as we increase the coverage of the selective classification method. See Appendix B.3 for an experiment that matches the bound exactly.

4 Experiments

We now present a detailed experimental evaluation of the interplay between differential privacy and selective classification. All of our experiments are implemented using PyTorch [Paszke et al., 2019] and Opacus [Yousefpour et al., 2021]. We publish our full code-base at the following URL: https://github.com/cleverhans-lab/selective-classification .

Setup. Our evaluation is based on the following experimental panel. In particular, we conduct experiments for each dataset/SC-method/ ε combination:

Datasets : FashionMNIST [Xiao et al., 2017], CIFAR-10 [Krizhevsky et al., 2009], SVHN [Netzer et al., 2011], GTSRB [Houben et al., 2013].

Selective Classification Methods : Softmax Response ( SR ) [Geifman and El-Yaniv, 2017], SelectiveNet ( SN ) [Geifman and El-Yaniv, 2019], Self-Adaptive Training ( SAT ) [Huang et al., 2020], Monte-Carlo Dropout ( MCDO ) [Gal and Ghahramani, 2016], Deep Ensembles ( DE ) [Lakshminarayanan et al., 2017], Selective Classification Training Dynamics ( SCTD ) [Rabanser et al., 2022].

Privacy levels : ε ∈ {∞ , 7 , 3 , 1 } .

Based on recent results from Feng et al. [2023], we (i) apply additional entropy regularization to all methods; and (ii) derive the selective classification scores for SN and SAT by applying Softmax Response ( SR ) to the underlying classifier (instead of relying on model-specific abstention mechanisms). Across all private experiments, we set δ = 10 -( N +1) for datasets consisting of training samples on the order of 10 N . For each combination of SC method, dataset, and privacy level, we train a model following the ResNet-18 [He et al., 2016] model architecture using either the SGD optimizer (for ε = ∞ ) or the DP-SGD optimizer (for all ε < ∞ ). All models are trained for 200 epochs using a

∞

e

0

.

95

1

.

00

=

FashionMNIST

0

.

95

1

.

00

CIFAR-10

0

.

96

0

.

98

1

.

00

SVHN

0

.

950

0

.

975

1

.

000

GTSRB

0

.

9

1

.

0

e

=

7

.

0

0

.

6

0

.

8

1

.

0

0

.

9

1

.

0

0

.

9

1

.

0

0

.

9

1

.

0

e

=

3

.

0

0

.

6

0

.

8

1

.

0

0

.

8

0

.

9

1

.

0

0

.

8

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

8

0

.

9

1

.

0

e

=

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

8

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

5

1

.

0

MSP

SAT

MCDO

DE

SCTD

SN

mini-batch size of 128. For all DP training runs, we set the clipping norm to c = 10 and choose the noise multiplier adaptively to reach the overall privacy budget at the end of training. Note that, since we want to consider a consistent value of ε across all SC methods, we restrict the overall ε in both SN and DE . As explained in Section 3.2, this effectively enforces a reduction in ε for each individual training run due to sequential composition. Concretely, we train DE with 5 ensemble members and SN with target coverage levels c target ∈ { 0 . 1 , 0 . 25 , 0 . 5 , 0 . 75 , 1 . 0 } . This leads to a reduction of ≈ ε √ 5 for each trained sub-model in both cases. We account for the precise reduction in the privacy parameter by increasing by the number steps of DP-SGD as measured by a Rényi DP accountant [Mironov, 2017, Yousefpour et al., 2021]. All experiments are repeated over 5 random seeds to determine the statistical significance of our findings. We document additional hyper-parameters in Appendix B.1.

Evaluation of selective classification techniques across privacy levels. We report our main results in Figure 3 where we display the accuracy-coverage trade-off for our full experimental panel. Overall, we observe that Selective Classification Training Dynamics ( SCTD ) delivers the best accuracycoverage trade-offs. While this is consistent with non-private [Rabanser et al., 2022], we find that SCTD delivers especially strong performance at lower ε levels. This suggests that leveraging training dynamics information is useful for performative selective classification under DP. As expected, based off of the discussion in Section 3.2, Figure 3 also shows that the performance of both Deep Ensembles ( DE ) and SelectiveNet ( SN ) suffers under DP, showing increasing utility degradation as ε decreases.

∞

e

0

.

95

1

.

00

=

FashionMNIST

0.0072

0

.

95

1

.

00

CIFAR-10

0.0086

0

.

975

1

.

000

SVHN

0.0028

0

.

975

1

.

000

GTSRB

0.0109

0

.

9

1

.

0

e

=

7

.

0

0.0211

0

.

75

1

.

00

0.0982

0

.

9

1

.

0

0.0156

0

.

95

1

.

00

0.0050

0

.

9

1

.

0

e

=

3

.

0

0.0227

0

.

75

1

.

00

0.1074

0

.

9

1

.

0

0.0184

0

.

9

1

.

0

0.0092

0

.

0

0

.

5

1

.

0

Coverage

0

.

8

1

.

0

e

=

1

.

0

0.0317

0

.

0

0

.

5

1

.

0

Coverage

0

.

5

1

.

0

0.1521

0

.

0

0

.

5

1

.

0

Coverage

0

.

8

1

.

0

0.0266

0

.

0

0

.

5

1

.

0

Coverage

0

.

75

1

.

00

0.0621

Recovering non-private utility by reducing coverage. Recall that one key motivation for applying selective classification to a DP model is to recover non-private model utility at the expense of coverage. We investigate this coverage cost in detail by examining how many samples a DP model can produce decisions for while maintaining the utility level of the respective non-private model (i.e., a model trained on the same dataset without DP). The results are outlined in Table 1. For high (i.e., ε = 7 ) and moderate (i.e., ε = 3 ) privacy budgets, we observe that a reduction of 20% -30% in data coverage recovers non-private utility. However, at low (i.e., ε = 1 ) privacy budget we observe that dataset coverage degrades strongly on most datasets. In particular, we find that for CIFAR-10 (across all ε values) and GTSRB (at ε = 1 ), the coverage reduces to below 30% . This result showcases that, depending on the particular choice of ε , recovering non-private model performance from a DP model can come at a considerable coverage cost. Moreover, this coverage cost varies widely across SC methods with SCTD leading to the lowest incurred loss in coverage by far.

0

.

9

1

.

0

e

=

∞

FashionMNIST

0.0097

0.0106

0

.

9

1

.

0

CIFAR-10

0.0070

0.0237

0

.

975

1

.

000

SVHN

0.0041

0.0045

0

.

95

1

.

00

GTSRB

0.0009

0.0048

0

.

9

1

.

0

e

=

7

.

0

0.0267

0.0269

0

.

5

1

.

0

0.0993

0.1491

0

.

9

1

.

0

0.0180

0.0229

0

.

75

1

.

00

0.0033

0.0569

0

.

0

0

.

5

1

.

0

Coverage

0

.

8

1

.

0

e

=

1

.

0

0.0392

0.0513

0

.

0

0

.

5

1

.

0

Coverage

0

.

5

1

.

0

0.2221

0.2369

0

.

0

0

.

5

1

.

0

Coverage

0

.

75

1

.

00

0.0667

0.1067

0

.

0

0

.

5

1

.

0

Coverage

0

.

5

1

.

0

0.1273

0.1657

Disentangling selective prediction performance from base accuracy. Recall from Section 3.4 that the standard evaluation pipeline for selective classification is not suitable for comparing DP models across varying privacy guarantees. To overcome this issue, we compute the accuracy-dependent upper bound (Equation 5) for each experiment and measure how closely the achieved accuracy/coverage trade-off aligns with this bound as per Equation 6. We document these results computed over all datasets, SC methods, and ε levels in Table 2. We find that, as ε decreases, all methods perform progressively worse. That is, for stronger privacy, all methods increasingly struggle to identify points they predict correctly on. Recall that this behavior is expected based on our discussion in Section 3.3. Again, we observe SCTD offering the strongest results, leading to the smallest bound deviation. To graphically showcase closeness to the upper bound, we further plot the accuracy-coverage trade-off and the corresponding upper bound for each experiment with the SCTD method in Figure 4.

Parallel composition with partitioned ensembles. As previously stated, Deep Ensembles underperform under DP due to composition. One potential mitigation strategy is to partition the data and train isolated models with no data overlap. This circumvents composition but also limits the utility of individual ensemble members as less data is available for training. We experiment with both setups and report results in Figure 5. Overall, these experimental results indicate that (i) partitioned deep ensembles lead to lower full-coverage accuracy when compared to non-partitioned deep ensembles while (ii) at the same time being less performant in terms of selective classification performance.

5 Conclusion

In this work we have studied methods for performing selective classification under a differential privacy constraint. To that end, we have highlighted the fundamental difficulty of performing selective prediction under differential privacy, both via a synthetic example and empirical studies. To enable this analysis, we introduced a novel score that disentangles selective classification performance from baseline utility. While we establish that SC under DP is indeed challenging, our study finds that a specific method ( SCTD ) achieves the best trade-offs between SC performance and privacy budget.

Limitations. This work presents insights drawn from empirical results and we acknowledge that a more thorough theoretical analysis is needed for a deeper understanding of the interplay between SC and DP. Also, we did not carefully investigate fairness implications beyond class imbalance. This connection to fairness requires special attention with past works showing that both DP and SC can negatively affect sensitive subgroups [Jones et al., 2020, Bagdasaryan et al., 2019].

Future work. We believe that this work initiates a fruitful line of research. In particular, future work can expand on the intuition provided here to obtain fundamental bounds on selective classification performance in a DP setting. Although we did not focus on this here, we believe that the ideas developed in this paper could help mitigate the trade-offs between privacy and subgroup fairness.

References

Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.

Bagdasaryan, O. Poursaeed, and V. Shmatikov. Differential privacy has disparate impact on model accuracy. Advances in neural information processing systems , 32, 2019.

Bassily, A. Smith, and A. Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science , pages 464-473. IEEE, 2014.

Brawner and J. Honaker. Bootstrap inference and differential privacy: Standard errors for free. Unpublished Manuscript , 2018.

Challen, J. Denny, M. Pitt, L. Gompels, T. Edwards, and K. Tsaneva-Atanasova. Artificial intelligence, bias and clinical safety. BMJ Quality & Safety , 28(3):231-237, 2019.

Covington, X. He, J. Honaker, and G. Kamath. Unbiased statistical estimation and valid confidence intervals under differential privacy. arXiv preprint arXiv:2110.14465 , 2021.

Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 , pages 265-284. Springer, 2006.

Dwork, A. Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

El-Yaniv and Y. Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research , 11(53):1605-1641, 2010. URL http://jmlr.org/papers/v11/ el-yaniv10a.html .

Evans, G. King, M. Schwenzfeier, and A. Thakurta. Statistically valid inferences from privacy protected data. American Political Science Review , 2019.

Feng, M. O. Ahmed, H. Hajimirsadeghi, and A. H. Abdi. Towards better selective classification. In The Eleventh International Conference on Learning Representations , 2023.

Ferrando, S. Wang, and D. Sheldon. Parametric bootstrap for differentially private confidence intervals. In International Conference on Artificial Intelligence and Statistics , pages 1598-1618. PMLR, 2022.

Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050-1059. PMLR, 2016.

Geifman and R. El-Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems , 30, 2017.

Geifman and R. El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International conference on machine learning , pages 2151-2159. PMLR, 2019.

Ghodsi, S. K. S. Hari, I. Frosio, T. Tsai, A. Troccoli, S. W. Keckler, S. Garg, and A. Anandkumar. Generating and characterizing scenarios for safety testing of autonomous vehicles. arXiv preprint arXiv:2103.07403 , 2021.

Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning , pages 1321-1330. PMLR, 2017.

He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136 , 2016.

Houben, J. Stallkamp, J. Salmen, M. Schlipsing, and C. Igel. Detection of traffic signs in real-world images: The German Traffic Sign Detection Benchmark. In International Joint Conference on Neural Networks , number 1288, 2013.

Huang, C. Zhang, and H. Zhang. Self-adaptive training: beyond empirical risk minimization. Advances in neural information processing systems , 33:19365-19376, 2020.

Jones, S. Sagawa, P. W. Koh, A. Kumar, and P. Liang. Selective classification can magnify disparities across groups. arXiv preprint arXiv:2010.14134 , 2020.

Karwa and S. Vadhan. Finite sample differentially private confidence intervals. arXiv preprint arXiv:1711.03908 , 2017.

Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017.

Liu, Z. Lin, S. Padhy, D. Tran, T. Bedrax Weiss, and B. Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems , 33:7498-7512, 2020.

Liu, Z. Wang, P. P. Liang, R. R. Salakhutdinov, L.-P. Morency, and M. Ueda. Deep gamblers: Learning to abstain with portfolio theory. Advances in Neural Information Processing Systems , 32, 2019.

Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF) , pages 263-275. IEEE, 2017.

Mozannar and D. Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning , pages 7076-7087. PMLR, 2020.

Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf .

Rabanser, A. Thudi, K. Hamidieh, A. Dziedzic, and N. Papernot. Selective classification via neural network training dynamics. arXiv preprint arXiv:2205.13532 , 2022.

Shejwalkar, A. Ganesh, R. Mathews, O. Thakkar, and A. Thakurta. Recycling scraps: Improving private learning by leveraging intermediate checkpoints. arXiv preprint arXiv:2210.01864 , 2022.

N. Vieira, M. O'Hagan, and C. O'Sullivan. Understanding the societal impacts of machine translation: a critical review of the literature on medical and legal use cases. Information, Communication &Society , 24(11):1515-1532, 2021.

Vijh, D. Chandola, V. A. Tikkiwal, and A. Kumar. Stock closing price prediction using machine learning techniques. Procedia computer science , 167:599-606, 2020.

Williams and F. McSherry. Probabilistic inference and differential privacy. Advances in neural information processing systems , 23, 2010.

Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Yousefpour, I. Shilov, A. Sablayrolles, D. Testuggine, K. Prasad, M. Malek, J. Nguyen, S. Ghosh, A. Bharadwaj, J. Zhao, G. Cormode, and I. Mironov. Opacus: User-friendly differential privacy library in PyTorch. arXiv preprint arXiv:2109.12298 , 2021.

Zaoui, C. Denis, and M. Hebiri. Regression with reject option and application to knn. Advances in Neural Information Processing Systems , 33:20073-20082, 2020.

A Additional Method Details

A.1 DP-SGD Algorithm

We provide a detailed definition of DP-SGD in Algorithm 2.

Algorithm 2: DP-SGD [Abadi et al., 2016]

Require: Training dataset D , loss function ℓ , learning rate η , noise multiplier σ , sampling rate q , clipping norm c , iterations T .

2: for t ∈ [ T ] do

1: Initialize θ 0

3: 1. Per-Sample Gradient Computation

5: for i ∈ B t do

4: Sample B t with per-point prob. q from D

6: g t ( x i ) ←∇ θ t ℓ ( θ t , x i )

7: end for

8: 2. Gradient Clipping

9: ¯ g t ( x i ) ← g t ( x i ) / max ( 1 , ∥ g t ( x i ) ∥ 2 c )

11: ˜ g t ← 1 | B t | (∑ i ¯ g t ( x i ) + N (0 , ( σc ) 2 I ) )

10: 3. Noise Addition

12: θ t +1 ← θ t -η ˜ g t

14: Output θ T , privacy cost ( ε, δ ) computed via a privacy accounting procedure

13: end for

A.2 Selective Classification Method Details

Softmax Response ( SR ) The traditional baseline methods for selective prediction is the Softmax Response ( SR ) method [Hendrycks and Gimpel, 2016, Geifman and El-Yaniv, 2017]. This method uses the confidence of the final prediction model f as the selection score:

While this method is easy to implement and does not incur any additional cost, SR has been found to be overconfident on ambiguous, hard-to-classify, or unrecognizable inputs.

SelectiveNet ( SN ) A variety of SC methods have been proposed that leverage explicit architecture and loss function adaptations. For example, SelectiveNet ( SN ) [Geifman and El-Yaniv, 2019] modifies the model architecture to jointly optimize ( f, g ) while targeting the model at a desired coverage level c target . The augmented model consists of a representation function r : X → R L mapping inputs to latent codes and three additional functions: (i) the prediction function f : R L → R C for the classification task targeted at c target ; (ii) the selection function g : R L → [0 , 1] representing a continuous approximation of the accept/reject decision for x ; and (iii) an additional auxiliary function h : R L → R C trained for the unconstrained classification tasks. This yields the following losses:

The selection score for a particular point x is then given by:

Self-Adaptive Training ( SAT ) Alternatively, prior works like Deep Gamblers [Liu et al., 2019] and Self-Adaptive Training [Huang et al., 2020] have also considered explicitly modeling the abstention

class ⊥ and adapting the optimization process to provide a learning signal for this class. For instance, Self-Adaptive Training ( SAT ) incorporates information obtained during the training process into the optimization itself by computing and monitoring an exponential moving average of training point predictions over the training process. Samples with high prediction uncertainty are then used for training the abstention class. To ensure that the exponential moving average captures the true prediction uncertainty, an initial burn-in phase is added to the training procedure. This delay allows the model to first optimize the non-augmented, i.e., original C -class prediction task and optimize for selective classification during the remainder of the training process. The updated loss is defined as:

The rejection/abstention decision is then determined by the degree of confidence in the rejection class:

Deep Ensembles ( DE ) Finally, ensemble methods combine the information content of M models into a single final model. Since these models approximate the variance of the underlying prediction problem, they are often used for the purpose of uncertainty quantification and, by extension, selective prediction. The canonical instance of this approach for deep learning based models, Deep Ensembles ( DE ) [Lakshminarayanan et al., 2017], trains multiple models from scratch with varying initializations using a proper scoring rule and adversarial training. Then, after averaging the predictions made by the model, the softmax response ( SR ) mechanism is applied:

Monte-Carlo Dropout ( MCDO ) To overcome the computational cost of estimating multiple models from scratch, Monte-Carlo Dropout ( MCDO ) [Gal and Ghahramani, 2016] allows for bootstrapping of model uncertainty of a dropout-equipped model at test time. While dropout is predominantly used during training to enable regularization of deep neural nets, it can also be used at test time to yield a random sub-network of the full neural network. Concretely, given a model f with dropout-probability o , we can generate M random sub-networks at test-time by deactivating a fraction o of nodes. For a given test input x we can then average the outputs over all models and apply softmax response ( SR ):

Selective Classification Training Dynamics ( SCTD ) Another ensembling approach that has recently demonstrated state-of-the-art selective classification performance is based on monitoring the model evolution during the training process. Selective Classification Training Dynamics ( SCTD ) [Rabanser et al., 2022] records intermediate models produced during training and computes a disagreement score of the intermediate predictions with the final prediction for any test-time input x :

̸

SCTD only observes the training process and does not modify the model architecture/training objective.

B Additional Experimental Details

B.1 Hyperparameters

In this section, we document additional hyper-parameter choices. For Self-Adaptive Training ( SAT ), we set the pre-training epochs to 100 and momentum parameter 0 . 9 . For Selective Classification Training Dynamics ( SCTD ), we set the weighting parameter k = 3 and consider checkpoints at a 50 batch resolution. For Monte-Carlo Dropout, we set the dropout probability to 0 . 1 . Entropy regularization as suggested in Feng et al. [2023] is employed with β = 0 . 01 .

e

0

.

0

0

.

5

1

.

0

0

.

9

1

.

0

e

=

∞

p

0

= 0.5

0

.

0

0

.

5

1

.

0

0

.

8

0

.

9

1

.

0

p

0

= 0.25

0

.

0

0

.

5

1

.

0

0

.

8

1

.

0

p

0

= 0.1

0

.

0

0

.

5

1

.

0

0

.

5

1

.

0

p

0

= 0.01

0

.

0

0

.

5

1

.

0

0

.

50

0

.

75

1

.

00

=

7

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

e

=

3

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

0

0

.

5

1

.

0

e

=

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

0

0

.

5

1

.

0

B.2 Class Imbalance Experiments

We provide additional experiments on the effect of class imbalance to extend our intuition from Section 3.3. To that end, we take two data sets from our main results, namely CIFAR-10 and FashionMNIST, and produce four alternate datasets from each dataset. These datasets feature various degrees of class imbalance with p 0 ∈ { 0 . 5 , 0 . 25 , 0 . 1 , 0 . 01 } specifying the sampling probability for class 0 . All other classes maintain a sampling probability of 1 . We then train the same model as described in Section 4 and apply the softmax response SC algorithm.

We document these results in Figures 6 and 7. For ε = ∞ , we observe the expected gains from SC: minority points are accepted towards the end of the coverage spectrum [Jones et al., 2020] and correct points are accepted first. This effect is independent of the sampling probability p 0 . As we decrease the ε budget we observe that (i) the acceptance of minority groups starts to spread over the full coverage spectrum; (ii) the accuracy on the subgroup increasingly deteriorates with smaller ε , and (iii) wrongful overconfidence on the minority reverses the acceptance order at low sampling probabilities p 0 (i.e., incorrect points are often accepted first). These results indicate that employing selective classification on private data can have unwanted negative effects in the presence of subgroups. Future work should investigate this connection more thoroughly.

B.3 Upper Bound Reachability

In Equation 5, we have introduced an upper bound on the selective classification performance on a model with full-coverage accuracy a full . We now present a simple experimental panel across varying

0

.

0

0

.

5

1

.

0

0

.

8

0

.

9

1

.

0

e

=

∞

p

0

= 0.5

0

.

0

0

.

5

1

.

0

0

.

8

1

.

0

p

0

= 0.25

0

.

0

0

.

5

1

.

0

0

.

8

1

.

0

p

0

= 0.1

0

.

0

0

.

5

1

.

0

0

.

5

1

.

0

p

0

= 0.01

0

.

0

0

.

5

1

.

0

0

.

8

1

.

0

e

=

7

.

0

0

.

0

0

.

5

1

.

0

0

.

6

0

.

8

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

8

1

.

0

e

=

3

.

0

0

.

0

0

.

5

1

.

0

0

.

6

0

.

8

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

8

1

.

0

e

=

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

6

0

.

8

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

Coverage

0

.

0

0

.

5

1

.

0

Coverage

0

.

0

0

.

5

1

.

0

Coverage

0

.

0

0

.

5

1

.

0

Coverage

0

.

0

0

.

5

1

.

0

Coverage

Achieved Accuracy-Coverage Tradeoff

full-coverage accuracy levels showing that this bound is in fact reachable by a perfect selective classifier.

We assume a binary classification setting for which we generate a true label vector y ∈ { 0 , 1 } n 0 + n 1 with balanced classes, i.e., n 0 = n 1 where n 0 corresponds to the number points labeled as 0 and n 1 corresponds to the number points labeled as 1 . Then, based on a desired accuracy level a full , we generate a prediction vector p which overlaps with y for a fraction of a full . Finally, we sample a scoring vector s where each correct prediction is assigned a score s i ∼ U 0 , 0 . 5 and each incorrect prediction is assigned a score s i ∼ U 0 . 5 , 1 . Here, U a,b corresponds to the uniform distribution on the interval [ a, b ) . This score is clearly optimal since thresholding the scoring vector s at 0 . 5 perfectly captures correct/incorrect predictions: all s i < 0 . 5 correspond to a correct prediction, while all s i ≥ 0 . 5 correspond to an incorrect prediction. Computing the accuracy/coverage trade-off of this experiment, across various utility levels, matches the bound exactly.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the combination of differential privacy (DP) and selective classification. It examines the performance variations of existing selective classification methods when equipped with DP-SGD. Additionally, the paper demonstrates the complex correlation between selective accuracy and differential privacy. To address the challenge of disentangling selective classification performance from baseline utility, a new metric is proposed to evaluate different selective classification algorithms under differential privacy. The paper concludes by highlighting the superior performance of the recent algorithm SCTD in DP settings.\n\nSTRENGTHS:\nThe paper explores an intriguing intersection between differential privacy and selective classification. The experiments conducted are comprehensive.\n\nWEAKNESSES:\nOne potential weakness of this work lies in the exclusive use of DP-SGD for all the differential privacy implementations, which could be a subject of questioning. There exist various methods to enforce differential privacy, such as perturbing output scores ($g(x,f)$) or perturbing training data ($x,y$). In the case of ensembling, applying the DP mechanism in the softmax step, as shown in [1], could be considered. Consequently, ensembling methods do not necessarily suffer from DP composition. Therefore, it might be an overstatement to claim that SCTD performs the best in a private learning setting; instead, it can be stated that it performs well with a straightforward adaptation of DP-SGD.\n\n[1] Phan, NhatHai, et al. \"Differential privacy preservation for deep auto-encoders: an application of human behavior prediction.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 30. No. 1. 2016.\n\nQUESTIONS:\n- The term \"full-coverage accuracy\" is repeatedly referenced without a formal definition. It is unclear whether it represents an arbitrary number within the range of [0,1].\n- The advantage of Definition 3.2 is not adequately explained. Since integration is additive, Equation (6) seems to yield a negative AUC score plus a constant determined only by $a_{full}$. This may not provide more informative insights than the original AUC score. Consequently, the benefits of this new metric remain unclear. Moreover, Definition 3.2 is rather confusing. Is $a_{full}$ a variable or a fixed value determined by $f,g$? If it is a variable, how is it chosen among different algorithms? If it is algorithm-dependent, then this is just the (negative) original AUC score penalized by a function of the full-coverage accuracy. It would be helpful to elaborate further on the motivation behind this metric.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the relationship between selective classification (SC) and differential privacy (DP).\nFirst, the authors consider various SC approaches and if they incur additional privacy/utility costs under the constraint of DP.\nThey then investigate the effect of DP on SC algorithms and find that DP can drastically reduce the efficacy of SC as models become overconfident in the wrong class under DP.\nFinally, the authors introduce a new metric to evaluate SC under DP that takes into account the baseline accuracy of the model.\nAn extensive experimental evaluation is given, showing which SC techniques perform best, the coverage needed to provide non-private accuracy, and an evaluation under the new metric.\nOverall, SCTD (an SC approach that abstains based on the model's disagreement at various checkpoints) is shown to perform the best.\n\nSTRENGTHS:\n- This is a very interesting problem and a clever way to give private learning a boost in accuracy. The coverage required for the non-private accuracy table, in particular, was quite interesting.\n- The experimental evaluation contains many different SC techniques. This seems like a large implementation effort and a nice contribution to the community.\n- The example with class imbalance gave a nice intuition for the problem with SC under DP.\n- The SCTD approach is a nice way to make use of intermediate checkpoints. It is often argued these intermediate model checkpoints are a waste of privacy budget or assume too strong of an adversary. It was nice to see them actually utilized here to improve classification accuracy.\n\nWEAKNESSES:\n## Groupings of SC techniques\nThe discussion of why the SC techniques were grouped as post-processing or composition was non-existent. There was discussion of why the DE technique affects privacy/utility, but none of the other approaches received any discussion as to why they were ``free\" or had an effect.\n\nI believe the SN approach was grouped with composition as it would also affect utility by changing the loss function (but not privacy, as DP-SGD is applied to the gradients, not the loss). But, the authors do not make this argument. Furthermore, by my previous logic, I expect DG or SAT to affect utility as they also modify the loss function. In general, this section on grouping the techniques had the least justification. I would like to see more justification here. It would also be beneficial to explain why specific approaches are free, as I think post-processing is an oversimplification (as some require modifying the training algorithm).\n\n## DE evaluation\nWhile I agree the DE approach affects privacy (needing multiple models), I wasn't convinced that composition was the only way to fix this problem. I was curious what would happen if one instead portioned the dataset between the models in the ensemble and used the parallel composition theorem. I realize this would decrease utility, but how would that compare to the composition?\n\n## Delta too high\n$1/n$ is too large for a delta parameter. It technically allows a trivial mechanism that publishes a single record of the database. The general rule of thumb is that $\\delta << 1/n$ [Dwork & Roth](https://www.nowpublishers.com/article/Details/TCS-042). I don't believe changing the delta would affect the overall conclusions of the paper, so I am willing to let it go, but I wanted to bring attention to it.\n\n## Typos/Clarity\nOverall this paper was incredibly well written. However, below I give a couple of trouble spots that could be made more clear.\n- Line 64: \"more stringent than targeted privacy level\" is clunky/hard to parse\n- Line 124: Overloading the notation of $acc_c$ was rather confusing. I am not sure if this is normal in the SC literature, but having $c$ represent both a threshold and coverage with no clear distinction is rather confusing.\n- Line 203: Incorrect acronym for deep ensembles.\n- Line 215: mechanisms? or models? A mechanism is used to train a model, so I think the model makes more sense.\n- Line 229: This first sentence is somewhat confusing. Specifically, \"free from its bias towards accuracy is to align...\" is a lot to parse.\n- Line 246: It should be stated somewhere that this metric lower is better (as opposed to the previous metric that higher is better). To save the reader having to figure that out (although it is easy to figure out).\n- Figure 2: not sure if error bars could be added here, although it might make it too hard to read\n- Table 1: it should be made clear what the $\\pm$ represents. I assume a confidence interval but at what $\\alpha$?\n- In Tables 1 & 2, the labels seem quite off. First, there are six techniques, but only five appear in Table 1. Second, MSP and NNTD are not defined. I assume NNTD = SCTD and MSP=SR?\n\nQUESTIONS:\n- Can the authors comment on the DE approach using the dataset partitioning idea? Perhaps they have tried it.\n- Although SC does not affect the privacy guarantee, does it increase privacy leakage? For example, could a membership inference attack use the absence of a prediction to rule out non-member samples and increase overall attack performance?\n\nLIMITATIONS:\nThe authors do a good job of acknowledging limitations. I agree the initial experimental evaluation is a significant contribution, but theoretical evaluations are important for future work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the interaction between differential privacy and selective classification. Several methods for training selective classifiers are considered and evaluated for their compatibility with DP-SGD. Experimental results are given that show decreasing privacy parameters leads to selective classifiers becoming increasingly confident in incorrect predictions. An alternative performance metric for selective classification is proposed for the purposes of evaluating DP selective classifiers.\n\nSTRENGTHS:\nThe paper, to the best of my knowledge, is the first to empirically investigate the degradation of performance of selective classifiers under the constraint of differential privacy. It addresses interesting questions regarding the training and evaluation of DP selective classifiers, and highlights important distinctions between SC training methods that become particularly relevant in the setting of differential privacy.\n\nWEAKNESSES:\nI think additional attention to presentation is necessary to clarify the main contributions of the paper. The punchline of the experiment in the body of the paper is that the observed overconfidence in an incorrect classification prevents selective classification, but this is an underspecified claim. I take it to mean that highly accurate selective classification based on prediction confidence is in tension with strong privacy guarantees, but this is consistent with known privacy/accuracy tradeoffs and doesn't seem terribly surprising. The necessity for new evaluation metrics is also still unclear to me. The metric given in equation 4 is an area under the curve of accuracy as a function of coverage, as coverage ranges from 0 to 1, but then the stated motivation for the new evaluation metric is that we want to avoid aligning selective classifiers based on accuracy. As stated, it doesn't seem like equation 4 is aligning on accuracy, and that training two selective classifiers with different privacy parameters would allow for a meaningful comparison by (an approximation) of equation 4.\n\nQUESTIONS:\nMy main questions are reiterations of earlier critique. Why do existing evaluation metrics fail in the case of private selective classification? Is there additional significance to the empirical results beyond the extension of privacy/utility tradeoffs to the case of selective classification?\n\nIn Figure 1, it would be nice to have the meaning of the various colors used explicitly stated, ideally in the caption. It's inferable from context, but additional description would probably save the reader time. I also found the results in figure 4 and 5 were challenging to parse, mostly the significance of the blue and orange lines. Additional explanatory text to help the reader interpret these lines would be greatly appreciated.\n\nLIMITATIONS:\nYes, the authors have sufficiently addressed the impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies selective classification with differential privacy (DP). In particular, the authors conduct a comprehensive empirical study of different selective classifiers under DP. The key findings and contributions are:\n\n* An existing SC method that uses checkpoints is suitable under DP.\n* A novel accuracy-dependent selective classification score proposed by the authors helps compare selective classification performance across DP levels without explicit accuracy alignment.\n* Selective classification performance degrades with stronger privacy.\n* Recovering utility can come at a considerable coverage cost under strong privacy requirements.\n\nSTRENGTHS:\n* The paper initializes the study of the interplay between DP and SC. This is the first study of its kind. The proposed metric is effective in evaluating the performance of SC under different coverage. Thus, the significance of the proposed metric is beyond the study of the paper. \n* The paper might have an impact on the community in the long run since it bridges previously disconnected fields (SC and DP). Thus, The paper pushes the community in a new, interesting/important direction.\n* The paper is well-written and clearly organized. The evaluation is also comprehensive.\n\nWEAKNESSES:\nI do not find any obvious weaknesses in the paper. One minor point that could be improved is the explanation of the proposed metrics (Definition 3.1) in lines 255-260. It takes me some time to digest why the authors define the upper bound in this way.\n\nGoing beyond this, I think a more rigorous theoretical analysis and characterization between the trade-offs of SC and DP could be an interesting direction.\n\nQUESTIONS:\nPlease improve the explanation of the proposed metrics (Definition 3.1).\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}