SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Stephan Rabanser $^{1,2*}$ , Anvith Thudi $^{1,2}$ , Abhradeep Thakurta $^{3}$ , Krishnamurthy (Dj) Dvijotham $^{3}$ , Nicolas Papernot $^{1,2}$
<sup>1</sup>University of Toronto, <sup>2</sup>Vector Institute, <sup>3</sup>Google DeepMind
Training reliable deep learning models which avoid making overconfident but incorrect predictions is a longstanding challenge. This challenge is further exacerbated when learning has to be differentially private: protection provided to sensitive data comes at the price of injecting additional randomness into the learning process. In this work, we conduct a thorough empirical investigation of selective classifiers—that can abstain under uncertainty—under a differential privacy constraint. We find that some popular selective prediction approaches are ineffective in a differentially private setting because they increase the risk of privacy leakage. At the same time, we identify that a recent approach that only uses checkpoints produced by an off-the-shelf private learning algorithm stands out as particularly suitable under DP. Further, we show that differential privacy does not just harm utility but also degrades selective classification performance. To analyze this effect across privacy levels, we propose a novel evaluation mechanism which isolates selective prediction performance across model utility levels at full coverage. Our experimental results show that recovering the performance level attainable by non-private models is possible but comes at a considerable coverage cost as the privacy budget decreases.
State of the art machine learning (ML) models are gaining rapid adoption in high-stakes application scenarios such as healthcare [Challen et al., 2019, Mozannar and Sontag, 2020], finance [Vijh et al., 2020], self-driving [Ghodsi et al., 2021], and law [Vieira et al., 2021]. However, major challenges remain to be addressed to ensure trustworthy usage of these models. One major concern in sensitive applications is to protect the privacy of the individuals whose data an ML model was trained on. To prevent privacy attacks on ML models,  $(\varepsilon, \delta)$  differential privacy (DP) [Dwork et al., 2014] has emerged as the de-facto standard with widespread usage in both academic and industrial applications.
While the introduction of DP successfully protects against privacy attacks, it also limits model utility in practice. For instance, the canonical algorithm for training models with DP, DP-SGD [Abadi et al., 2016], clips the per-sample gradient norm and adds carefully calibrated noise. These training-time adaptations frequently lead to degradation in predictive performance in practice. This is especially a problem for datasets containing underrepresented subgroups whose accuracy has been shown to degrade with stronger DP guarantees [Bagdasaryan et al., 2019]. Faced with this challenge, it is therefore of vital importance to detect samples on which a DP model would predict incorrectly on.
One popular technique used to detect inputs that the model would misclassify with high probability is given by the selective classification (SC) framework [Geifman and El-Yaniv, 2017]: by relying on a model's uncertainty in the correct prediction for an incoming data point, the point is either accepted with the underlying model's prediction, or rejected and potentially flagged for additional
*Correspondence to: stephan@cs.toronto.edu
downstream evaluation. Hence, SC introduces a trade-off between coverage over the test set (i.e., the goal of accepting as many data points as possible) and predictive performance on the accepted points (i.e., the goal of making as few mistakes as possible). Although identification of misclassified points seems of particular importance under differential privacy, the application of selective prediction to private models is under-explored to date. While lots of past works have studied privacy or selective prediction in isolation, best practices for selective prediction under differential privacy have not been established yet. In this work, we are the first to answer the question of whether selective classification can be used to recover the accuracy lost by applying a DP algorithm (at the expense of data coverage).
To analyze the interplay between differential privacy and selective classification, we first show that not all approaches towards selective classification are easily applicable under a differential privacy constraint. In particular, approaches relying on multiple passes over the entire dataset to obtain the full SP performance profile [Geifman and El-Yaniv, 2019, Lakshminarayanan et al., 2017] suffer significantly under differential privacy. This is due to the fact that the worst-case privacy leakage increases with each analysis made of the dataset which means that each training run forces more privacy budget to be expended. On the other hand, our analysis shows that an SC approach based on harnessing intermediate model checkpoints [Rabanser et al., 2022] yields the most competitive results. Notably, this approach performs especially well under stringent privacy constraints  $(\varepsilon = 1)$ .
Next, we find that differential privacy has a more direct negative impact on selective classification that goes beyond the characteristic drop in utility. Based on a simple synthetic experiment, we observe a clear correlation between differential privacy strength and wrongful overconfidence. In particular, even when multiple private models trained with different  $\varepsilon$  levels offer the same utility on the underlying prediction task, we show that stronger levels of DP lead to significantly decreased confidence in the correct class. This reduction effectively prevents performant selective classification.
Motivated by this observation, we realize a need to disentangle selective classification from model utility; SC ability and accuracy should be considered independently. To that end, we show that the evaluation metric typically used for non-private selective classification is inappropriate for comparing SC approaches across multiple privacy levels. The in-applicability of the current method stems from the fact that full-coverage accuracy alignment is required across experiments. In particular, accuracy-aligning multiple DP models via early-stopping frequently leads to a more stringent than targeted privacy level, rendering the intended comparison impossible. Due to this limitation of previous evaluation schemes, we propose a novel metric which isolates selective classification performance from losses that come from accuracy degradation of the classifier overall (as frequently caused by DP). The performance metric we propose to tackle this problem computes an upper bound on the accuracy/coverage trade-off in a model-dependent fashion and measures the discrepancy between the actual achieved trade-off to this bound. Using this score, we determine that, based on a comprehensive experimental study, differential privacy does indeed harm selective prediction beyond a loss in utility.
We summarize our key contributions below:
1. We provide a first analysis on the interplay between selective classification and differential privacy. As a result, we identify an existing SC method as particularly suitable under DP and present an illustrative example showcasing an inherent tension between privacy and selective prediction.
2. We unearth a critical failure mode of the canonical SC performance metric preventing its off-the-shelf usage under DP experimentation. We remedy this issue by introducing a novel accuracy-dependent selective classification score which enables us to compare selective classification performance across DP levels without explicit accuracy-alignment.
3. We conduct a thorough empirical evaluation across multiple selective classification techniques and privacy levels. As part of this study, we confirm that selective classification performance degrades with stronger privacy and further find that recovering utility can come at a considerable coverage cost under strong privacy requirements.
2.1 Differential Privacy with DP-SGD
Differential Privacy (DP) [Dwork et al., 2014] is a popular technique which allows us to reason about the amount of privacy leakage from individual data points. Training a model with differential privacy
ensures that the information content that can be acquired from individual data points is bounded while patterns present across multiple data points can still be extracted. Formally, a randomized algorithm  $\mathcal{M}$  satisfies  $(\varepsilon, \delta)$  differential privacy, if for any two datasets  $D, D' \subseteq \mathcal{D}$  that differ in any one record and any set of outputs  $S$  the following inequality holds:
$$
\mathbb {P} [ \mathcal {M} (D) \in S ] \leq e ^ {\varepsilon} \mathbb {P} [ \mathcal {M} \left(D ^ {\prime}\right) \in S ] + \delta \tag {1}
$$
The above DP bound is governed by two parameters:  $\varepsilon \in \mathbb{R}_+$  which specifies the privacy level, and  $\delta \in [0,1]$  which allows for a small violation of the bound.
DP-SGD. A canonical way of incorporating differential privacy in deep learning is via Differentially Private Stochastic Gradient Descent (DP-SGD) [Bassily et al., 2014, Abadi et al., 2016]. We describe the DP-SGD algorithm in detail in Algorithm 2 in the Appendix. The main adaptations needed to incorporate DP into SGD are: (i) per-sample gradient computation, which allows us to limit the per-point privacy leakage in the next two steps; (ii) gradient clipping, which bounds the sensitivity (i.e., the maximal degree of change in the outputs of the DP mechanism); and (iii) noise addition, which introduces Gaussian noise proportional to the intensity of gradient clipping.
2.2 Selective Classification
Supervised classification. Our work considers the supervised classification setup: We assume access to a dataset  $D = \{(x_{n},y_{n})\}_{n = 1}^{N}$  consisting of data points  $(x,y)$  with  $x\in \mathcal{X}\subseteq \mathbb{R}^d$  and  $y\in \mathcal{V} = \{1,\ldots ,C\}$ . All data points  $(x,y)$  are sampled independently and identically distributed from the underlying distribution  $p$  defined over  $\mathcal{X}\times \mathcal{Y}$ . The goal is to learn a prediction function  $f:\mathcal{X}\to \mathcal{Y}$  which minimizes the classification risk with respect to the underlying data distribution  $p$  as measured by a loss function  $\ell :\mathcal{Y}\times \mathcal{Y}\rightarrow \mathbb{R}$ .
Selective classification. Selective classification augments the supervised classification setup by introducing a rejection class  $\bot$  via a gating mechanism [El-Yaniv and Wiener, 2010]. This mechanism produces a class label from the underlying classifier if the mechanism is confident that the prediction is correct and abstains otherwise. Note that this mechanism is often directly informed by the underlying classifier  $f$  and we make this dependence explicit in our notation. More formally, the gating mechanism introduces a selection function  $g: \mathcal{X} \times (\mathcal{X} \to \mathcal{Y}) \to \mathbb{R}$  which determines if a model should predict on a data point  $\pmb{x}$ . If the output of  $g(\pmb{x}, f)$  undercuts a given threshold  $\tau$ , we return  $f(\pmb{x})$ , otherwise we abstain with decision  $\bot$ . The joint predictive model is therefore given by:
$$
(f, g) (\boldsymbol {x}) = \left\{ \begin{array}{l l} f (\boldsymbol {x}) & g (\boldsymbol {x}, f) \leq \tau \\ \perp & \text {o t h e r w i s e .} \end{array} \right. \tag {2}
$$
Evaluating SC performance. The performance of a selective classifier  $(f, g)$  is based on two metrics: the coverage of  $(f, g)$  (corresponding to the fraction of points to predict on) and the accuracy of  $(f, g)$  on accepted points. Note that there exists a tension between these two performance quantities: naively increasing coverage will lead to lower accuracy while an increase in accuracy will lead to lower coverage. Successful SC methods try to jointly maximize both metrics.
$$
\operatorname {c o v} _ {\tau} (f, g) = \frac {| \{\boldsymbol {x}: g (\boldsymbol {x} , f) \leq \tau \} |}{| D |} \quad \operatorname {a c c} _ {\tau} (f, g) = \frac {| \{\boldsymbol {x}: f (\boldsymbol {x}) = y , g (\boldsymbol {x} , f) \leq \tau \} |}{| \{\boldsymbol {x}: g (\boldsymbol {x} , f) \leq \tau \} |} \tag {3}
$$
To characterize the full performance profile of a selective classifier  $(f,g)$ , we consider the selective accuracy at coverage level  $c$ , formally  $\mathrm{acc}_c(f,g)$ , over the full coverage spectrum by computing an area-under-the-curve (AUC) score  $s_{\mathrm{AUC}}$  as follows:
$$
s _ {\mathrm {A U C}} (f, g) = \int_ {0} ^ {1} \operatorname {a c c} _ {c} (f, g) d c \quad \operatorname {a c c} _ {c} (f, g) = \operatorname {a c c} _ {\tau} (f, g) \quad \text {f o r} \tau \text {s . t .} \operatorname {c o v} _ {\tau} (f, g) = c \tag {4}
$$
Each existing selective classification algorithm proposes a  $g$  that tries to maximize this metric. We describe popular approaches below and include additional details in Appendix A.2.
Prediction confidence. The traditional baseline methods for selective prediction is the Softmax Response (SR) method [Hendrycks and Gimpel, 2016, Geifman and El-Yaniv, 2017]. This method uses the confidence of the final prediction model  $f$  as the selection score. To reduce overconfident predictions yielded by this method, confidence calibration [Guo et al., 2017] has been proposed.
Ensembling. To further improve calibration and to reduce variance, ensemble methods have been proposed which combine the information content of  $M$  models into a single final model. The canonical instance of this approach for deep learning based models, Deep Ensembles (DE) [Lakshminarayanan et al., 2017], trains multiple models from scratch with varying initializations using a proper scoring rule and adversarial training. Then, after averaging the predictions made by all models, the softmax response (SR) mechanism is applied. To overcome the computational cost of estimating multiple models from scratch, Monte-Carlo Dropout (MCD0) [Gal and Ghahramani, 2016] allows for bootstrapping of model uncertainty of a dropout-equipped model at test time. Another ensembling approach that has recently demonstrated state-of-the-art selective classification performance is based on monitoring model evolution during the training process. To that end, Selective Classification Training Dynamics (SCTD) [Rabanser et al., 2022] records intermediate models produced during training and computes a disagreement score of intermediate predictions with the final prediction.
Architecture & loss function modifications. A variety of SC methods have been proposed that leverage explicit architecture and loss function adaptations. For example, SelectiveNet (SN) [Geifman and El-Yaniv, 2019] modifies the model architecture to jointly optimize  $(f,g)$  while targeting the model at a desired coverage level  $c_{\mathrm{target}}$ . Alternatively, prior works like Deep Gamblers [Liu et al., 2019] and Self-Adaptive Training [Huang et al., 2020] have considered explicitly modeling the abstention class  $\perp$  and adapting the optimization process to provide a learning signal for this class. For instance, Self-Adaptive Training (SAT) incorporates information obtained during the training process into the optimization itself by computing and monitoring an exponential moving average of training point predictions over the training process. Introduced to yield better uncertainty estimates, Liu et al. [2020] employs weight normalization and replaces the output layer of a neural network with a Spectral-Normalized Neural Gaussian Process to improve data manifold characterization.
Uncertainty for DP models. An initial connection between differential privacy and uncertainty quantification is explored in Williams and McSherry [2010], which shows that probabilistic inference can improve accuracy and measure uncertainty on top of differentially private models. By relying on intermediate model predictions, Shejwalkar et al. [2022] has proposed a mechanism to quantify the uncertainty that DP noise adds to the outputs of ML algorithms (without any additional privacy cost).
3 The Interplay Between Selective Classification and Differential Privacy
We now examine the interplay between these approaches to selective classification and training algorithms constrained to learn with differential privacy guarantees. We first introduce a candidate selective classification approach which leverages intermediate predictions from models obtained during training. Then, we study how current selective classification approaches impact DP guarantees, as each access they make to training data increases the associated privacy budget. Next, we investigate how in turn DP affects selective classification performance. Indeed, DP alters how optimizers converge to a solution and as such DP can impact the performance of SC techniques. Last, we discuss how selective classification performance can be fairly compared across a range of privacy levels.
Algorithm 1: SCTD [Rabanser et al., 2022]
Require: Checkpointed model sequence  $\{f_1,\dots ,f_T\}$  query point  $\pmb{x}$  weighting parameter  $k\in [0,\infty)$
1: Compute prediction of last model:  $L\gets f_{T}(\pmb {x})$
2: Compute disagreement and weighting of intermediate predictions:
3: for  $t\in [T]$  do
4: if  $f_{t}(\pmb{x}) = L$  then  $a_{t} \gets 0$  else  $a_{t} \gets 1$
5:  $v_{t} \gets \left(\frac{t}{T}\right)^{k}$
6: end for
7: Compute sum score:  $s_{\mathrm{sum}} \gets \sum_t v_t a_t$
8: if  $s_{\mathrm{sum}} \leq \tau$  then accept  $f(\pmb{x}) = L$  else reject with  $f(\pmb{x}) = \bot$
3.1 Performative Private Selective Classification via Training Dynamics Ensembles
As a classical technique from statistics, ensembling methods are often employed for confidence interval estimation [Karwa and Vadhan, 2017, Ferrando et al., 2022], uncertainty quantification [Lakshminarayanan et al., 2017], and selective prediction [Zaoui et al., 2020]. Under a differential privacy constraint, although for simpler tasks like mean estimation ensembling methods have been proven to be effective [Brawner and Honaker, 2018, Covington et al., 2021, Evans et al., 2019], in this paper
we demonstrate that they are fairly ineffective for selective classification. This is primarily due to the increased privacy cost due to (advanced sequential) composition [Dwork et al., 2006].
In light of this challenge, we expect one recent method in particular to perform well in a private learning setting: selective classification via training dynamics (SCTD) [Rabanser et al., 2022] (details described in Algorithm 1). For a given test-time point, SCTD analyzes the disagreement with the final predicted label over intermediate models obtained during training. Data points with high disagreement across this training-time ensemble are deemed anomalous and rejected. Most importantly, while SCTD also constructs an ensemble, only a single training run is used to obtain this ensemble. As a result of the post-processing property of DP, the original  $(\varepsilon, \delta)$ -DP guarantee can be maintained. To further motivate the effectiveness of relying on intermediate checkpoints, Shejwalkar et al. [2022] has shown in adjacent work that intermediate predictions can improve the predictive accuracy of DP training, yielding new state-of-the-art results under DP.
3.2 How Do Other Selective Classification Approaches Affect Privacy Guarantees?
We can extend the previous analysis based on post-processing and composition to other SC techniques. This allows us to group selective classification techniques into three classes: (i) directly optimized; (ii) post-processing; and (iii) advanced sequential composition. For directly optimized and post-processing approaches, we can obtain selective classification for free. On the other hand, composition-based approaches either require an increased privacy budget or suffer from decreased utility.
Direct optimization. Many selective classification approaches directly modify the loss function and optimization is performed w.r.t. this adapted loss function. As DP-SGD is loss function and architecture agnostic  $(\varepsilon, \delta)$  guarantees hold automatically for SC methods that only change the loss function. Applicable to: Softmax Response (SR), Deep Gamblers (DG), Self-Adaptive Training (SAT).
Post-processing. If a function  $\phi(x)$  satisfies  $(\varepsilon, \delta)$ -DP, then for any deterministic or random function  $\psi(\cdot)$ , the application of  $\psi$  on  $\phi$ , formally  $\psi \circ \phi(x)$ , still satisfies  $(\varepsilon, \delta)$ -DP [Dwork et al., 2006]. Applicable to: Monte-Carlo Dropout (MCD0), Selective Classification Training Dynamics (SCTD).
Advanced sequential composition. If in a set of aggregated functions  $\{\phi_1(x),\ldots ,\phi_M(x)\}$  each  $\phi_i(x)$  satisfies  $(\varepsilon ,\delta)$ -DP, then releasing the set of all outputs  $\psi (x) = (\phi_1(x),\dots ,\phi_M(x))$  satisfies  $\approx (\sqrt{M}\varepsilon ,M\delta)$ -DP [Dwork et al., 2006]. To maintain  $(\varepsilon ,\delta)$ -DP over the aggregated output, each function needs to satisfy  $\approx (\frac{\varepsilon}{\sqrt{M}},\frac{\delta}{M})$ -DP. Applicable to: Deep Ensembles (DE), SelectiveNet (SN).
3.3 How Does Differential Privacy Affect Selective Classification Performance?
After having examined how current selective classification techniques influence differential privacy guarantees, this subsection considers the opposite effect: what is the impact of differential privacy on selective classification? As we will see by means of an intuitive example, we expect that differential privacy impacts selective classification beyond a loss in utility.
To showcase this, we present a synthetic logistic regression example with the softmax response SC mechanism. We generate data from a mixture of two two-dimensional Gaussians with heavy class-imbalance. Concretely, we generate samples for a majority class  $\{(x_i,1)\}_{i = 1}^{1000}$  with each  $x_{i}\sim \mathcal{N}(0,I)$  and a single outlier point form a minority class  $(\pmb {x}^*, - 1)$  with  $\pmb{x}^{*}\sim \mathcal{N}([10\quad 0]^{\top},\pmb {I})$  We then train multiple differentially private mechanisms with  $\varepsilon \in \{\infty ,7,3,1\}$  on this dataset and evaluate all models on a test set produced using the same data-generating process.
We show the outcome of this set of experiments in Figure 1 where we plot the test set and the decision boundary of each model across  $\varepsilon$  levels. For  $\varepsilon = \infty$ , the model successfully discriminates the majority and the minority class. However, the decision boundary is heavily influenced by a single data point, namely the outlier  $(x^{*}, -1)$ . Note that, even though all models with  $\varepsilon \in \{7,3,1\}$  misclassify the outlier point (i.e., their utility is aligned), the changing decision boundary also increases the model's confidence in predicting the incorrect class for the outlier. Hence, even at an aligned utility level, the softmax response approach for SC performs increasingly worse on the outlier point under stronger DP constraints. We provide additional class-imbalanced results on realistic datasets in Appendix B.2.
Figure 1: Synthetic example of privacy impacts on selective classification. Shaded regions show softmax response confidence in the red and blue class, respectively. As  $\varepsilon$  decreases, the blue outlier point is increasingly protected by DP. Importantly, not only does DP induce a misclassification on the outlier point, but stronger levels of DP further increase confidence in the wrong prediction. This overconfidence in the incorrect class prevents selective classification.
3.4 How Should We Evaluate Selective Classification Under Differential Privacy?
As we have now established, we expect differential privacy to harm selective classification performance. In light of this insight, we are now investigating whether the standard evaluation scheme for SC is directly applicable to the differentially private models.