Training Uncertainty-Aware Classifiers with Conformalized Deep Learning
Authors: Bat-Sheva Einbinder
Yaniv Romano
Matteo Sesia
Yanfei Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep learning
Uncertainty
Conformal inference
Multi-class classification
Overfitting
Confidence.

Abstract: Deep neural networks are powerful tools to detect hidden patterns in data and leverage them to make predictions, but they are not designed to understand uncertainty and estimate reliable probabilities. In particular, they tend to be overconfident. We begin to address this problem in the context of multi-class classification by developing a novel training algorithm producing models with more dependable uncertainty estimates, without sacrificing predictive power. The idea is to mitigate overconfidence by minimizing a loss function, inspired by advances in conformal inference, that quantifies model uncertainty by carefully leveraging hold-out data. Experiments with synthetic and real data demonstrate this method can lead to smaller conformal prediction sets with higher conditional coverage, after exact calibration with hold-out data, compared to state-of-the-art alternatives.

Training Uncertainty-Aware Classifiers with Conformalized Deep Learning
Bat-Sheva Einbinder"
Yaniv Romano
Faculty of Electrical & Computer Engineering (ECE) Technion, Israel
Faculty of ECE and of Computer Science
Technion, Israel
bat bat-shevab&campus technion. ac ill
yromano@technion.co ac.
Matteo Sesia
Yanfei Zhou
Department of Data Sciences and Operations
Department Data Sciences and Operations
University of Southern California Los Angeles, California, USA sesia@marshall usw.allllsss..eooooooooom..o...l edu
University of Southern California
Los Angeles, California, USA yanfei whou@marshall. usc edu
Abstract
Deep neural networks are powerful tools to detect hidden patterns in data and lever- age them make predictions, but they are not designed understand uncertainty and estimate reliable probabilities. In particular, they tend to be overconfident. We begin address this problem in the context of multi-class classification by developing a novel training algorithm producing models with more dependable uncertainty estimates, without sacrificing predictive power. The idea is mitigate overconfidence by minimizing a loss function, inspired by advances in conformal inference, that quantifies model uncertainty by carefully leveraging hold-out data. Experiments with synthetic and real data demonstrate this method can lead to smaller conformal prediction sets with higher conditional coverage, after exact calibration with hold-out data, compared to state-of-the-an alternatives.
1 Introduction
The predictions of deep neural networks and other complex machine learning (ML) models affect important decisions in many applications (1-6). including autonomous driving, medical diagnostics, security monitoring. Prediction errors in those contexts can be costly or even dangerous, which makes dependable and explainable uncertainty estimation essential. Unfortunately, deep neural networks are not designed to understand uncertainty and they are easily prone to overfitting; consequently, they may lead overconfidence 77 Overeonfidence is especially problematic in the face of aleatory uncertainty [10], which refers to situations in which the outcome to be predicted is intrinsically noisy. Unlike the complementary epistemic uncertainty, aleatory uncertainty cannot be eliminated by training more flexible model by increasing the sample size. For example, think of prognosticating COVID-19 survival [2], assessing genetic disease predisposition [5] or anticipating credit card defaults [6). In those applications, the outcome of interest is potentially complicated and likely depends on many unmeasured variables. Therefore, no practical model may achieve perfect accuracy, but models that can offer practitioners principled measures of confidence for each individual-level prediction naturally tend to be more useful and trustworthy. Among many existing techniques for estimating uncertainty in ML predictions 77)), conformal inference [17]. stands out for its ability to provide finite-sample guarantees without unrealistic algorithmic simplifications and without strong assumptions about the data generating process or the underlying sources of uncertainty.
Authors listed in alphabetical order.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Conformal inference is designed to convert the output of any ML model into a prediction set of likely outcomes whose size is automatically tuned using hold-on out data, in such a way that the same procedure applied to future test data will yield prediction sets that are well calibrated in frequentist sense. particular, these sets have provable marginal coverage; i.e., at the 90% level this means the outcome for a new random test point is contained in the output set 90% of the time. The hold-out data are utilized to evaluate conformity scores, or goodness-of-Be scores, whose ordering determines how the sets are to be expanded so that the desired fraction of test points is guaranteed to be covered.
limitation of conformal inference is that it involves of two distinct phases, training and calibration, that are generally not designed to work together efficiently. The calibration algorithm takes as input pre- rre-trained models that may already be overconfident, and this is sub-optimal because bad habits are harder to correct after they become entrenched. As a result of this uncoordinated two-step approach, conformal predictions may be either unnecessarily conservative or overconfident for certain types of test cases, which can make them unreliable [8 [9] and unfair [20]. We address this challenge by developing a new ML training method that synergetically combines the learning and calibration phases of conformal inference, reducing the overconfidence of the predictions calculated by the output model. The idea is to minimize a new loss function designed to measure the discrepancy in distribution between the conformity scores computed by the current model estimate and those of an imaginary oracle that can leverage perfect knowledge of the data generating process to construct the most informative and reliable possible predictions. As we will demonstrate empirically, applying standard conformal inference (based on independent calibration data) to models trained with our novel algorithm leads to smaller prediction sets with more accurate coverage for all test points. In MI. models trained with the proposed algorithm tend to work
other words, the Luncertainty-wward more efficiently within a standard conformal inference framework compared black-box models, which perhaps should not be surprising. Further, we will also show that the models obtained with our method produce relatively reliable prediction sets even if the post-training conformal calibration step is omitted, although the latter remains necessary in theory to guarantee valid coverage.
Related work
Many methods have been developed to mitigate overconfidence in ML [21:34], for example by
allowing an agnostic output, by suitable post-processing 7 35 40), or through early stopping [41]. Additional relevant research includes that of [2 46 52 and these will provide us with informa- tive benchmarks. However, unlike conformal inference, these methods have no frequentist guarantees predictions,
in finite samples, and largely rely on loss functions targeting the accuracy of best- without explicitly addressing uncertainty during training. We build upon conformal inference [53]-
(58) pioneered by [17), which typically deals with off-the-shelf models 17 19 59 60) Although other very recent works have proposed leveraging ideas from this field to improve training (61+66), this paper is novel as it combines the adaptive conformity scores of [99 with completely new uncertainty-aware loss function. This departs from [62 63), which sought to minimize the cardinality of the prediction sets, and from [6]] 644666, which utilized conformal inference ideas for tuning low-dimensional typer-parameters as opposed to fully guiding the training of all model parameters. Although we focus classification [19], our method could be repurposed for regression [58 67) and other supervised tasks [60 (68 69). Conformal inference can also be utilized to test hypotheses and calibrate probabilities (70), and our work could be extended to those problems.
2
Relevant background on conformal inference
2.1 Uncertainty quantification via conformal prediction sets
Consider a data set of i.i.d. (or sometimes simply exchangeable) observations (X,, Yo sampled from an arbitrary unknown distribution Pxy. Here, X; IR" contains p features for the ith sample. and Y; E {I,...,,' K} = [K] denotes its label, which we assume to be one of K possible categories. The goal is to train model on n data points, (X,, Y. i. and construct a reasonably small prediction set Cn.a C [K] for Ya+1 given Xn+1 such that, for some fixed level a E (0.
IP [Yn E (Xn-1) 2 a.
This property called marginal coverage because it treats (X,, Yoy= as all random. If = 0.1, ensures Yn+1 is contained in the prediction sets 90% of the time. Marginal coverage is practically

---

feasible but not fully satisfactory, as it is not as reliable and informative as conditional coverage:
IP Yn+1 E Cx, Xa+1 = x - a, E
Conditional coverage would give one confidence that C.. (z) contains the true Y for any individual data point, which is stronger than (T). For example, imagine a population in which color is a feature and 90% of samples are blue while the others are red. Then, 90% marginal coverage is attained by any prediction sets that contain the true Y for all blue samples but never do for the red ones. Valid coverage can be obtained conditional on a given protected category [20], but it is impossible to guarantee (2) more generally without unrealistically strong assumptions [2).. Thus, a typical compromise is to construct prediction sets with marginal coverage and hope they are also reasonably valid conditional on X. For multi-class classification, solution is offered by the conformity scores developed in 19), which are reviewed below as the starting point of our contribution.
2.2 Review of adaptive conformity scores for classification
Imagine an oracle knowing the conditional distribution of Y given X, namely Pyxx, and think of how it would construct the smallest possible prediction sets Coracke (I:) with exact 1 - a conditional coverage. For any x E X and y E [ define (x) PlY = y X =x=a x.. Then, the oracle would output the smallest subset S S [K] such that 2 yyes Ty (z) > 1 - a. In truth, this set may have coverage strictly larger than 1 - a due to the discreteness of Y; however, exact coverage can be achieved by introducing a little extra randomness
(20]. For any nominal coverage level 7 E (0, 1) and any random noise variable u E (0, 1) ), let S be the function with input z, u, and that computes the set of most likely labels up to (but possibly excluding) the one identified by the above deterministic oracle: Szz, T) 7) S [K]. Note that the explicit expression of S is written in Appendix [1 If u = is a uniform random variable independent of everything else, it is easy to verify that the following oracle prediction sets have exact conditional coverage at level r = 1 a:
(3) For example, if Ti (x) 0.3, T2(x) 0.6, and m3 <3(x) 0.1. then Cyrile (x) {1,2) with probability 2/3, and Ogrzile (2) = [2] with probability 1/3. Of course, this oracle is only a thought experiment because Py|x is generally unknown. Therefore, to construct practical prediction intervals, must be replaced by an ML model न. Then, conformal inference is needed to ensure the output sets based on the possibly inaccurate ML model at least satisfy marginal coverage (
Croracle (z) = S(x, U; T. 1 ax).
Conformal inference 19 begins by training any classifier on part of the data, indexed by Dj C [m]. to fit an approximation of the unknown æ. After substituting á into the oracle rule S, the hold-out data indexed by D2 = [n] \ DD, are leveraged to adjust the prediction sets as to empirically achieve the desired coverage. More precisely, define the following conformity scores W. for all observations in D2, and for the test point n + 1. Intuitively, W; is the smallest r E [o, 1] such that S(X,, Us T) contains the true Y, while U₁ is a uniform random variable independent of everything else:
E D2 U (n + 1). (4) These statistics are observed for all i E D2. but not for n + 1. Define also in. as the [(1
W, = (X,, Y,, 4) = min (r E [o, 1]: Y, E S(X,, U2; r)},
a) + [D21)] largest element of { (W. Intuitively, in.o is the smallest E [0, 1] such that S(X;; contains a fraction 1 - a of the hold-out data in D2. Then, the output prediction set for new X,+1 is Cr. (X,+1)
S(X,-1,, Un41; R, an.a). This has 1 a marginal coverage due to the exchangeabilityy of the calibration and test data [19]. In fact, Yz+1 & C implies Wn+1 fr.a. and by exchangeability the probability of this event is smaller than a see (58). Unlike alternative approaches based on different scores [7 [8, 53). this solution would yield prediction sets equivalent to those of the oracle 19 if A = æ. Although generally A A x, the above prediction sets often achieve relatively high conditional coverage [9 in practice. Our goal is to further improve their empirical performance by training the ML model be more deliberately aware of uncertainty.
3 Methods
3.1 The distribution of the adaptive conformity scores
The conformity scores defined in (4) are uniformly distributed conditional on X = Jr, for any if a = क. This property was hinted without proof in [19] and it serves as the starting point of our contribution. Note that all mathematical proofs can be found in Appendix A2
3

---

Proposition 1. The distribution of the conformity scores W, in (4) is uniform conditional on X; if =
n. That is, P[W (X, Y, =; | f all B E (0. 1). where (X,Y) is random sample from Px.., and U ~ Uniform(0, 1] independent of everything else. Further, W; X; is uniform if and only S( क; a) has conditional coverage at level for all a E [0, 1].
As we seek accurate conditional coverage, this result suggests training र as to produce scores that are approximately uniform on hold-out data, at least marginally. Therefore, we will evaluate (4) on hold-out data while training म, encouraging the conformity scores to follow uniform distribution.
3.2 An uncertainty-aware conformal loss function
We develop loss function that approximately measures the deviation from uniformity of the confor- mity scores defined in (4) by combining classical non parametric tests for equality in distribution with fast algorithms for smooth sorting and ranking [73 74.. This loss is combined with the traditional cross entropy as to also promote accurate predictions, and it can be approximately optimized by stochastic gradient descent (it will generally be non-convex). The novel uncertainty-aware component of this loss only sees the hold-out samples through the lens of a non non-parametric test applied to the empirical score distribution within a subset of the data. Therefore, it provides little incentive to overfit compared to a traditional loss targeting point-wise predictive accuracy, such as the cross entropy. By contrast, it discourages overconfident predictions which would yield non-uniform scores, as we shall see below. This solution is outlined in Figure A1 Appendix A1.2 and detailed below.
First, the training samples partitioned into two subsets, Iι and Ig such that In UT2 = [m] = D1- Here we assume the training data are indexed by D1 = [n1;; this notation is slightly different from Section 2 but it is simple and does not introduce ambiguity because the additional calibration data in D2 remain untouched during training. The training algorithm approximately minimizes loss function e consisting of two additive components, each evaluated on one subset of the data:
i = (1 - X) (a(T)) + A. (Z
(5)
Above, the hyper- apper-paramtter la component is evaluated on the data in In, and its purpose is to seek high predictive accuracy, as customary. For example, this could be the cross entropy:
[o. 1] controls the relative weights of the two components. The
ΣΣ
la
1 = (X.),
(6)
I
where क is the output of the final softmax layer. The novel uncertainty-aware component e, is evaluated on Iz, and its role is to mitigate overconfidence. Concretely, conformity scores W, are evaluated according to (4) for all e Iz, and their empirical distribution is compared to the ideal uniformity expected if न = т. Ideally, we would like to quantify this discrepancy by directly applying powerful non- parameametric test, for example by computing the Cramér-von Mises [75 76 or Kolmogorov- -mminoov 77 78 test statistics. Concretely, in the latter case,
Eu = sup w
[iz))(w) w
where (-)z(() is the empirical cumulative distribution function (CDF) of W, for i E Iz: (w) = (1/1I2)) 1 [W; w]. Unfortunately, (z) is not differentiable, which makes the overall loss intractable to minimize. This requires introducing some approximations in lu, as explained next.
3.3
Differentiable approximations
The empirical CDF of the conformity scores is not differentiable because it involves sorting, which non-smooth operation. Further, these scores themselves are not differentiable in the model parameters 0 because they involve ranking and sorting the estimated class probabilities â. In fact, the score can be computed in a closed form,
(8) where U, is a uniform random variable independent of everything else; see Appendix A for details, Fortunately, there exist fast approximate algorithms for differentiable sorting and ranking that
Wi * (X.) + (X.) +... ((()) - U, (X.))

---

work well in combination with standard back-pon propagation [73] 74]. Note that evaluating W, in (
requires accessing elements of (X))......... [(K (()))(())) (X,)) through a d-dependent index, r(Y, (Xx,) )).
which is also non-differentiable. Therefore, indexing by r(Y,, # (X,)) must be approximated with smooth linear interpolation: see Appendix [A1.2 for further details. In conclusion, the &n loss in (7) approximated by evaluating differentiable version of the scores in then by replacing their empirical CDF with differentiable approximation obtained with the same techniques from [74]. This procedure, combined with stochastic gradient descent for fitting the model parameters 0, is summarized in Algorithm [ Although here we assume Z I2 for simplicity, this algorithm can easily accommodate Iι + 12. A more technically detailed version of Algorithm I is provided in Appendix A1.2, and an open-source software implementation of this method is available online httph::t/ssg wtw.b.to.bb.bbbb..eeecoeo wtwp//aattseaaaaooooooeeeoeeeeee
described above, and
learning
Algorithm 1: Conformalized uncertainty-wware training of deep multi-class classifiers
Input: Data (X,, Y hyper- paper--rraetter E [0, 1], learning rate Y > 0, batch size MM
Randomly initialize the model parameters g());
Randomly split the data into two disjoint subsets, I1, Iz, such that I U I2 [n]; Set the number of batches to B = (n/2)/M (assuming for simplicity that [Z [Tg);
for / ,,,
T do
Randomly divide I and I2 into B batches;
for 1 = ,,..., B do
Evaluate (softmaxx) conditional probabilities At (X;)) for all i in batch 6 of I₁ U
Generate a uniform independent random variable U, for all in batch b of I2; Evaluate We for all i in batch b of Iz, using Ug and differentiable approximation of 4
Evaluate the gradient (la(((() of la in (6) using the data in batch b of Iı;
Evaluate the gradient Vi, (0(9)) of differentiable approximation é, of e, using the differentiable scores W, in batch of Iz;
Define vε(((!)) = (1 A)- (l(()) + x (0(+)) based on 5 Update the model parameters: g(*) ((+ V)(
end end
Output: The model á corresponding to the fitted parameters 0(2),
3.4 Theoretical analysis
The uncertainty-awaat loss function in Algorithm can be justified theoretically by noting that it is approximately minimized (although possibly non-uniquely) by the imaginary oracle model क, which yields the smallest possible prediction sets with exact conditional coverage. This analysis focuses on the original version of the loss function defined in (5) 7 ignoring for simplicity the additional subtleties introduced by the differentiable approximations described in Section 3.3
Proposition 2. The loss function 2 O in (5) is bound from above by g + 88, where e 2 0 attains
value zero if ã = n, and öt O1 M//MMMM) as I2 + cc
Of course, Algorithm does not minimize (5) exactly because it involves solving a high-dimensional non-convex optimization problem that is difficult to study theoretically. Yet, it is possible to prove at least weak form of convergence for its stochastic gradient descent, whose solution may not however necessarily approach a global minimum. This analysis is in Appendix A2 for lack of space.
Numerical experiments
4.1 Experiments with synthetic data
The performance of Algorithm
investigated here on synthetic data that mimic a multi-class classification problem in which most samples are relatively easy to classify but a few are unpredictable. Specifically, data are simulated with 100 independent and uniformly distributed features X =
(X....., (1o0) [0, 1000 and label Y [K], for K = 6. The first feature controls whether the
5

---

sample is intrinsically difficult to classify, while the next two features determine the most likely labels; all other features are useless. On average. 20% of the samples are impossible to classify with absolute confidence. This conditional distribution is written explicitly in Appendix A3.1
The conditional class probabilities á are estimated as the output of final softmax layer in fully connected neural network implemented with PyTorch [79]; see Appendix f3r more information about network architecture and training details. This model fitted separately with Algorithm | and three benchmark techniques including traditional cross entropy minimization and focal loss
minimization [80]. Unfortunately, we cannot directly compare the recent methods of 62 63 as
originally implemented by those authors for lack of openly available computer code. Instead, we consider hybrid benchmark that combines elements of Algorithm []] with the main idea of [63]. essentially seeking model that yields small conformal prediction sets, irrespective of conditional coverage; see Appendix A1.3 for details about this benchmark. early stopping can help mitigate overfitting [41] it is informative to also investigate its effect on each of the aforementioned learning algorithms. For this purpose, we generate an additional validation set of 2000 independent data preview the out-of-sample accuracy and loss value at each epoch. Then, the best
points and use versions of each model according to these two early stopping criteria are saved during training. After training each model, 10,000 additional independent samples are utilized to calibrate split-conformal prediction sets with 90% marginal coverage, as explained in Section 2.2 All models, including those trained with our method, undergo conformal calibration with these independent data points prior to constructing the prediction sets, as this step is necessary to theoretically guarantee marginal coverage. Further, ensure the comparisons between different models always fair by giving each training algorithm access to exactly the same data set, regardless of whether its inner workings involve sample splitting to evaluate separately distinct components of the loss function.
Figure D compares the performance of conformal prediction sets obtained with each model for 2000 test points as a function of the number of training samples, averaging over 50 independent experiments. The prediction sets are evaluated terms of their average size and coverage conditional on X S ö; i.e, separately for the "hard" samples. For each learning algorithm, the results corresponding to the model achieving the highest conditional coverage among the fully trained and two early stopped alternatives reported. This allows us to focus the overall behaviour of different losses while accounting for possible differences the optimal choices of carly-stopping strategies.
(a)
(b)
0.90
3.5
Conditional Coverage
0.85
3.0
0.80
Crucke
Conformll
0.75
Hybrdd
Oress
0.70
0.65
1.5
0.60
1200
2400
3000
3600
4800
4800
Number of training samples
Number of training samples
Figure 1: Performance of conformal prediction sets with 90% marginal coverage based on the ideal oracle model and a deep neural network trained with four alternative algorithms. (a): Conditional coverage for intrinsically hard samples. (b): Average size of prediction sets. The results are averaged over independent experiments and standard errors are below 0.1 (not shown explicitly).
Figure I does not visualize marginal coverage because that is guaranteed to be above 90%. The results show our algorithm yields the prediction sets with highest conditional coverage and smallest size, especially when the number of training samples large. By contrast, the conditional coverage obtained with the cross entropy does not visibly improve as the training data set grows, suggesting systematic cer-confieence with hard-to-classify samples. The focal loss improves upon the cross entropy by reducing the average size of the prediction sets, but it does not lead to higher conditional coverage, Finally, the hybrid algorithm increases conditional coverage slightly compared to the cross entropy, but it does not lead to smaller prediction sets. This is likely because the hybrid loss does not introduce very different incentives compared the cross entropy, as the latter already attempts to maximize the estimated probability of the observed labels, thus effectively seeking small conformal

---

prediction sets without necessarily high conditional coverage. Note that the focal loss is applied here with syper-parameter y = 1, as we have found larger values to yield lower accuracy; see Figure [A2] In this section, our method is applied with the hyper-parameter X = 0.2 in (5): additional results
obtained with different values of À are in Figures A3 A4
Appendix A3.2
The improved performance of our method can be understood by looking at the distribution of the corresponding conformity scores for test data, as shown in Figure A for the model trained on 2400 samples. Our method leads to scores that are more uniformly distributed than those obtained with the cross entropy, which is indicative of more reliable uncertainty quantification. In fact, our estimated conditional probabilities i are more similar to the true oracle π compared to those obtained by minimizing the cross entropy: see Figure A7 All figures here refer to fully trained models, without early stopping; analogous results with early stopping are presented later. It is worth emphasizing that post-training conformal calibration is always necessary to guarantee marginal coverage, regardless of how the model is trained. At the same time, it is interesting to observe that models trained with our method tend to estimate more reliable probabilities compared to the benchmarks. Therefore, should not be surprising that our models lead to larger prediction sets with relatively high marginal coverage if we skip the post-training conformal calibration step; see Figure AS
Figure A8 compares the performance of the prediction sets obtained with each method when covariate shift occurs in the test data. In particular, here we imagine that at test time the eeeeraintt-controlline feature X1 is sampled uniformly from [0, a) with a < 1, so that lower values of correspond to higher proportions of intrinsically hard-to-classify samples. Of course, in this case marginal coverage is no longer guaranteed for the same reason why conditional coverage in Figure [ is not always controlled, As expected, all models produce smaller set sizes with higher marginal coverage for closer to 1, consistently with Figure but our method outperforms the benchmarks.
Several additional results are in Appendix A3.2 Figure A9 reports on experiments in which the number K of labels is varied, ranging from 4 to 12. These results show our methods leads to prediction sets with consistently smaller size and typically higher conditional coverage compared to all benchmarks. Figure reorrrtsss on experiments in which the proportion of hard-to-classify samples varied, ranging from 0.1 to 0.5. All models lead to higher conditional coverage for larger 6, as implied by the fixed marginal coverage. but our method consistently achieves it with smaller prediction sets. Figures A I A reportt on experiments with models trained using early stopping based on maximum prediction accuracy on the validation data. Again, our method achieves higher conditional coverage and smaller prediction sets relative to the benchmarks. Figures 51 A18 report analogous results obtained with early stopping based on minimum validation loss. Figures A (A20 (resp. (A2)) show the distribution of conformity scores and the estimated class probabilities, as
in Figures A64A7 from models trained with early stopping based on validation predictive accuracy (resp. loss). Finally, Figures A23 A24 (resp. (A2)) show the distribution of conformity scores
and the estimated class probabilities from the focal loss (res. hybrid) models.
4.2 Experiments with CIFAR-10 data
Convolutional neural networks guided by the conformal loss are trained on the publicly available CIFAR-10 image classification data set (81) (10 classes), and the models thus obtained are compared to those targeting the three benchmark losses considered before. As these data are not too hard to
classify, we make the problem more interesting by randomly applying RandomErasing [82] some images-a a form of corruption that makes images very hard to recognize. The number of training samples is varied from 3000 to 45000. See Appendix A3.3 for details. To measure the performance of conformal prediction sets based on each model, we set aside 5000 calibration and test observations prior to training. All models are calibrated after training. as in the previous section, in order to guarantee valid marginal coverage. The proportions of corrupt images in the calibration and test sets are 0.2, while that in the training set varied. All models are calibrated as in [1)) seeking 90% marginal coverage. Their performance is measured on the test data in terms of the size of the output prediction sets and their coverage conditional on the indicator of corruption. Further, the test accuracy of each model is evaluated based on the misclassification rate of its best label. Figure 2 showcases two example test images, respectively intact and corrupted by RandomErasing. along with their corresponding conditional class probabilities calculated by different models fully trained on 45000 data points. This shows the model trained with our conformal loss not as overconfident when dealing with the corrupted images as that minimizing the cross entropy.
7

---

Conformal
Conformal
(Ship: 1.00]
[Dog: 0.34, Cat: 0.32)
Hybrid
Hybrid
(Ship: 1.00]
(Dog: 0.99, Horse: 0.01)
Cross entropy
Cross entropy
(Ship: 0.96, Car: 0.04)
[Deer: 0.99, Dog: 0.01)
Focal
Focal
(Car: 0.95, Ship: 0.03)
(Deer: 0.84, Dog: 0.08)
Figure 2: Two example test images from the CIFAR-10 data set, with their top two estimated class probabilities computed by the output softmax layer of convolutional neural networks trained to minimize different loss functions. Left: intact image of a ship. Right: corrupted image of a dog.
Figure A27 summarizes the performance of the prediction sets obtained with each fully trained model, as function of the number of training samples, Median performance measures are reported over 10 experiments with random data subsets. The conformal loss leads higher coverage for intrinsically hard images, and smaller prediction sets for the easy ones. As shown in Figure A288 this improvement associated with higher test accuracy for the models targeting our loss, consistently with their increased robustness to overfitting. As overfitting can also be mitigated by early stopping, we report in Figure A29 the corresponding results obtained with early stopping based on maximum accuracy on validation data set of size 2000 (or 5000, training with 45000 samples). In this case, all models achieve similar test accuracy, but those trained with our method lead to conformal prediction sets with (slightly) higher conditional coverage and smaller size, especially if trained many samples and compared to the focal loss. These conclusions are summarized in Table iin the case of 45000 training samples, which includes also the results corresponding to models trained with early stopping based on validation accuracy. See Table AT for results with early stopping based on validation loss. Overall, the models targeting the conformal loss perform best when trained fully or with early stopping based on accuracy, and they tend to achieve higher coverage for the hard images while enabling smaller prediction sets with valid coverage for the easy cases. A similar summary for the models trained with 3000 samples is in Table A2 there, the advantage of the conformal loss compared to the hybrid method is not marked in the large-sample experiments. Note that the models in Table 1 have accuracy below 83% due to the corrupted images in the training data; by comparison, 89% accuracy could be achieved by applying the cross-entropy method on the clean data set [83]
and In Appendix A3.4 Figures A30 and A31 |nd report on results similar to those in Figures A27 and A29
respectively, but using prediction sets directly constructed based the probability estimates computed by each model, without conformal calibration. Consistently with the experiments described in the previous section, these results show our models lead to prediction sets with relatively high marginal coverage, especially if the sample size is large. Of course, post-training conformal calibration remains necessary guarantee the nominal coverage level in finite samples. Figures A32 and A33 visualize some concrete examples of prediction sets obtained with and without post-training conformal calibration, respectively for intact and corrupt images. These results confirm the prediction sets obtained with our method tend to be less overconfident compared to the benchmarks, whether or not post-training conformal calibration is applied to guarantee marginal coverage.
Coverage
Size intact/corrupted
Accuracy
intact/corrupted
intact/corrupted
Full
ES (acc)
Full
ES (acc)
Full
ES (acc)
Conformal Hybrid
0.90/0.90
0.90/0.87
1.41/5.95
1.41/6.09
0.81/0.35
0.81/0.35
0.92/0.81
0.91/0.86
5,54/5.75
1.38/5.30
0.65/0.40
0.83/0.37
Cross Entropy
0.92/0.84
0.92/0.81
3.30/4.43
1.50/5.05
0.68/0.43
0.82/0.36
Focal 0.91/0.83
0.93/0.77
2.57/3.99
1.91/4.25
0.68/0.42
0.77/0.34
Table Conditional coverage and size of conformal prediction sets with 90% marginal coverage on CIFAR-10 10 data, based on models trained on 45000 data points. The models are trained either fully for many epochs, or with early stopping (ES) based on classification accuracy (acc). The last two columns report the best-guess classification accuracy of the underlying models applied to test data.
8

---

Figure A34 reports performance measures as in Figure A27 after fixing the number of training samples to 45000 and varying the corruption proportion. The model trained with the conformal loss leads to smaller prediction sets with higher conditional coverage compared to the benchmarks, and its test accuracy does not decrease as the proportion of corrupted training images increases. Figure A35 demonstrates early stopping mitigates overfitting and allows the benchmarks to maintain relatively high accuracy as the training corruption proportion increases. However, the conformal loss outperforms even if all models are trained with early stopping based on validation accuracy. Figure 3 reports on experiments similar to those in Table but performed under covariate shift. Here, the proportions of corrupt images in the training and calibration data sets are still fixed to 0.2, while the proportion of corrupt images in the test set is varied from 0.2 (no covariate shift) to 1.0 (extreme covariate shift). In these experiments, no model is theoretically guaranteed to achieve valid marginal coverage due to the covariate shift, but the model trained with our method remains approximately valid because it has practically high conditional coverage. In conclusion, these results confirm the model trained with our method can provide reliable uncertainty estimates for both intact and the corrupt images, while the benchmark models tend to be overconfident in the hard-to-cll toclassyyl cases.
5.50
Conformal
Coverage
Hybrid
5.00
Cross entropy
Size
Focal
0.66
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
02
0.5
0.4
0.5
0.6
0.7
0.8
0.0
1.5
Proportion corrupt images (test)
Proportion of corrupt images (test)
Figure 3: Marginal coverage and average size of conformal prediction sets obtained with models trained by different methods, the CIFAR-10 data under covariate shift. The results are shown function of the proportion of corrupt images in the test set. Other details are as in Table
The distributions of conformity scores obtained with different methods on test data are compared in Figures A339 These results confirm the scores obtained with the conformal loss tend to be more uniform compared to the benchmarks, especially if the training sample size is large. If few training samples are available, the focal loss yields scores that are slightly closer being uniform, but the cost of lower accuracy and conditional coverage (Table A2 In Figures A440AA4 the performances of all models are compared separately for each of the 10 possible test labels. These results suggest corrupt images with true labels 4.5, or 6 are most difficult to classify, and their prediction sets have the lowest coverage. However, this issue is mitigated by models fully trained to minimize the proposed conformal loss. Further, the results confirm that models trained with our loss yield more informative prediction sets for the easier unperturbed images, while achieving the desired coverage rate.
4.3 Experiments with credit card data
In this section, we analyze publicly available credit card default data set [84] containing 30000 observations of 23 features and a binary label. Approximately 22% of the labels are equal to 1. The data are randomly divided into 16800 training samples, 4500 calibration samples, and 4500 test samples, while the remaining 4200 samples are utilized to determine early stopping rules. All experiments are repeated 20 times with independent data subsets. The conformal and hybrid losses are implemented using 70% of the training samples for evaluating the cross entropy component. The model architecture is in Section 4.1 see Appendix A3.5 for further implementation details.
The performances of conformal prediction sets for models trained with different losses are measured in terms of their respective sizes and coverage conditional on the true label being equal to 1. As the samples with label are minority, constructing prediction sets with valid coverage for their group an interesting problem that may be relevant in the context of algorithmic fairness [20]. Note that it is possible to calibrate conformal prediction sets in such a way as to achieve perfect label-conditional coverage [1 19 at the cost of higher data usage, but here we focus training uncertainty-awer models as to approximately achieve this goal without explicit label conditionl calibration. Therefore, we apply a slightly modified version of Algorithm in which the empirical distribution of the hold-out

---

conformity scores is evaluated separately on the samples from each class, and then the sum of these two instances of (7) is utilized as e.- The same strategy is also incorporated into the hybrid method.
Table the average performances of each model and shows our method achieves the highest conditional coverage. Here, early stopping is applied based on minimum validation loss, as the alternative accuracy criterion achieved lower conditional coverage for all models. Our conformal models have relatively high classification errors on these data, but this can be explained by the high class imbalance. Thus, the error rate in Table 2 may be less informative as a performance metric than the corresponding F-score reported in Figure A44 which higher (better) for the models trained with our method. Finally, Figure|A and Table A3 in Appendix A3.6 demonstratee our model leads to conformity scores that are closer to being uniformly distributed on test data, as expected.
Coverage
Size all labels/0/1
Classification error
Marginal
Conditional
Full
ES
Full
ES
Full
ES
Full
ES
Conformal
0.83
0.83
0.60
0.52
1.34/1.1//145
1.29/1.27/1.39
33.05
28.52
Hybrid
0.83
0.83
0.51
0.53
1.28//.25/1.38
27.00
27.52
Cross Entropy 0.82
0.84
0.51
0.42
1.25/1.23/1
1,24//.21/1.35
26.16
24.36
Focal 0.81
0.82
0.48
0.50
1.25/1.23/1.32
26.56
26.30
Table 2:
Performance of conformal prediction sets with 80% marginal coverage on credit card default data, based on convolutional neural networks targeting different losses. Other details are as in Table T
5 Discussion
The conformal loss function presented in this paper mitigates overconfidence in deep neural networks and can lead to smaller prediction sets with higher conditional coverage compared to standard benchmarks. This contribution is practically relevant for many applications in which overfitting may occur, but it is especially useful when dealing with noisy data that do not allow very accurate out-of-sample predictions. One limitation of the proposed method is that it is more computationally expensive than its benchmarks. For example, training conformal loss model on 45000 images in the CIFAR-10 data set took us approximately 20 hours on an Nvidia P100 GPU, while training models with the same architecture to minimize the cross entropy or focal loss only took about 11 hours. Another limitation of the conformal loss is that a relatively large number of training samples appears to be required for significant performance improvements. Nonetheless, the ability of an ML model to more openly admit ignorance when asked to provide an unknown answer is a valuable achievement for which may sometimes be worth investing additional training resources.
Future research may explore extensions of our method for uncertainty-aware learning to problems beyond multi-class classification or to ML models other than deep neural networks, as well as further applications to real-world data sets. Focusing more closely on image classification, it would be interesting to combine the method proposed in this paper with data augmentation techniques that can further increase predictive accuracy [83], especially if the available training set is limited (85). Further, it could be useful combine our method with different random image corruption techniques as well as data augmentation in order to improve robustness to covariate shift and adversarial cases. Such extensions are not straightforward because data augmentation may violate the sample exchangeability assumptions, but recent theoretical advances in conformal inference may open the door to principled solution 86 877. We plan to explore these directions in future work.
Software implementing the proposed method is available online at lattps://githut: com/ bat ti--shevacconformaaee learning with all code needed to reproduce the numerical experiments.
Acknowledgements
M. S. and Y. Z. are supported by NSF grant DMS 2210637. M. S. is also supported by an Amazon Research Award. B. F. and Y. R. were supported by the Israel Science Foundation (grant No. 729/21). R. thanks the Career Advancement Fellowship, Technion, for providing research support. The authors also thank four anonymous reviewers for helpful comments and an insightful discussion.
10

---

References
[1] Jonathan Schmidt, Mário RG Marques, Silvana Botti, and Miguel AL Marques. Recent advances and applications of machine learning in solid-state materials science. npj Comput. Mater., 5(1): 1-36, 2019.
[2] Alexandre Tkatchenko, Machine learning for chemical discovery. Nat. Commun., 11(1):1-4, 2020.
[3] Gisbert Schneider. Mind and machine in drug design. Nat. Mach. Intell., 1(3):128-130) 2019.
[4] Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 3763)336 2020.
[5] Daniel S Hoadley and Nathan J Lucas. Artificial intelligence and national security, 2018.
[6] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell., 165):206-215, 2019.
[7] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1321-1330. JMLR. org, 2017.
[8] Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Adv. Neural. Inf. Process. Syst., pages 13888-13899, 2019.
[9] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Adv. Neural. Inf. Process. Syst., pages 13991-14002, 2019.
[10] Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3):477-066, 2021.
[11] Chansik An, Hyunsun Lim, Dong-Wood Wook Kim, Jung Hyun Chang, Yoon Jung Choi, and Scong Woo Kim. Machine learning prediction for mortality of patients diagnosed with covid-19: a nationwide korean cohort study. Scientific Reports, 10(1):18716, 2020. ISSN 2045-2322.
[12] Yue Gao, Guang- Yao Cai, Wei Fang, Hua- Yi Li, Si- Yuan Wang. Lingxi Chen, Yang Yu, Dan Liu, Sen Xu, Peng-Fei Cui, Shao-Qing Zeng, Xin-Xia Feng. Rui-Di Yu, Ya Wang, Yuan Yuan, Xiao-Fei Jiao, Jian-Hua Chi, Jia-Hao Liu, Ru-Yuan Li, Xu Zheng, Chun-Yan Song, Ning Jin, Wen-Jian Gong, Xing-Ye Yu Liu, Lei Huang, Xun Tian, Lin Li, Hui Xing, Ding Ma, Chun-Rui Li, Fei Ye, and Qing-Lei Gao. Machine learning based early warning system enables accurate mortality risk prediction for covid-19, 19. Nut. Commun., 11(1):0033, 2020. ISSN 2041-1723.
[13] Michela Carlotta Massi, Francesca Gasperoni, Francesca leva, Anna Maria Paganoni, Paolo Zunino, Andrea Manzoni, Nicola Rares Franco, Liv Veldeman, Piet Ost, Valérie Fonteyne, et al. A deep learning approach validates genetic risk factors for late toxicity after prostate cancer radiotherapy requite multi-national cohort. Frontiers in oncology, 10, 2020.
[14] Adrien Badré, Li Zhang, Wellington Muchero, Justin C Reynolds, and Chongle Pan. Deep neural network improves the estimation of polygenic risk scores for breast cancer. Journal of Human Genetics, 66(4):359-369 2021.
[15] Amo van Hilten, Steven A Kushner, Manfred Kayser, M Arfan Ikram, Hieab HH Adams, Caroline CW Klaver, Wiro Niessen, and Gennady V Roshchupkin. Gennet framework: interpretable deep learning for predicting phenotypes from genetic data. Communications biology, 40 (1):1199 2021.
[16] Ting Sun and Miklos A Vasarhalyi. Predicting credit card delinquencies: An application of deep neural networks. Handbook of Financial Econometries, Mathematics, Statistics, and Machine Learning, pages 4349-4381. World Scientific, 2021.
11

---

[17] Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic learning in random world. Springer, 2005,
[18] Maxime Cauchois, Suyash Gupta, and John Duchi. Knowing what you know: valid confidence sets in multiclass and multilabel prediction. 20Xiv:0004.0188/1 2020.
[19]
Yaniv Romano, Mateo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. In Adv. Neural. Inf. Process. Syst., 2020.
[20]
Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, and Emmanuel Candès. With malice toward none: Assessing uncertainty via equalized coverage. Harvard Data Science Review, 2020.
[21] Marten Wegkamp. Lasso type classifiers with reject option. Electron. J. Statist., 1: 1:155-168, 2007.
[22] Yves Grandvalet, Alain Rakotomamoniy, Joseph Keshet, and Stéphane Canu. Support vector machines with reject option. In Adv. Neural. Inf. Process. Syst., pages 537-544, 2009.
[23] Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Boosting with abstention. In Adv. Neural. Inf. Process. Syst., pages 1660-1668, 2016.
[24] Cláudio Rebelo de Sá, Carlos Soares, Arno Knobbe, and Paulo Cortez. Label ranking forests. Expert systems, 34( (1 ): 12666, 2017.
[25] Leying Guan and Rob Tibshirani. Prediction and outlier detection in classification problems. preprint 190.v: 2019.
[26] Peter L Bartlett and Marten H Wegkamp. Classification with a reject option using a hinge loss.
J. Mach. Learn. Res., 9:1823-1840, 2008
[27] Juan José Del Coz, Jorge Diez, and Antonio Bahamonde. Learning nondeterministic classifiers.
J. Mach. Learn. Res., 10(10), 2009.
[28] Yair Wiener and Ran EI- Yaniv. Agnostic selective classification. In Adv. Neural. Inf. Process. Syst., pages 1665-1673, 2011.
[29] Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust a classifier. In Adv. Neural. Inf. Process. Syst., pages 5541-5552, 2018.
[30] Yotam Hechtlinger, Barnabás Póczos, and Larry Wasserman. 1rXiv:1805.09466 2018
Cautious deep learning.
[31] Ziyin Liu, Zhikang Wang, Paul Pu Liang, Russ R Salakhutdinov, Louis-Philippe Morency, and Masahito Ueda. Deep gamblers: Learning to abstain with portfolio theory. In Adv. Neural. Inf. Process. Syst., pages 10633-10633, 2019.
[32] Ron Slossberg, Oron Anschel, Amir Markovitz, Ron Litman, Aviad Aberdam, Shahar Tsiper, Shai Mazor, Jon Wu, and R Manmatha. On calibration of scene-text recognition models. arXiv preprint arXiv:2012. 12433, 2020.
[33] Lijing Wang, Dipanjan Ghosh, Maria Gonzalez Diaz, Ahmed Farahat, Mahbubul Alam, Chetan Gupta, Jiangzhuo Chen, and Madhav Marathe. Wisdom of the ensemble: Improving consistency of deep learning models. Advances in Neural Information Processing Systems, 13:19750-19761, 2020.
[34] Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, and Stefano Soatto. Positive-congruent training: Towards regression-free model updates. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14299-14308. 2021.
[35] John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances large margin classifiers, 10(3):61-74, 1999.
[36] Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In lemi, volume 1, pages 609-616. Citeseer, 2001.
12

---

[37] Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699, 2002.
[38] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Francis Bach and David Blei, editors, Proceedings of the 32nd Inter- national Conference on Machine Leurning, volume 37 of Proceedings of Machine Learning Research, pages 1613-1622, Lille, France, 2015. PMLR.
[39] Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of Machine Learning Research, volume 89 Proceed- ings of Machine Learning Reseurch, pages 3459-3467. PMLR, 2019.
[40] Lukas Neumann, Andrew Zisserman, and Andrea Vedaldi. Relaxed softmax: Efficient confi- dence auto-calibration for safe pedestrian detection, 2018.
[41] Lutz Prechelt. Early stopping-but when? In Neural Networks: Tricks of the trade, pages 55-69. Springer, 1998.
[42] Antonio Bella, Cèsar Ferri, José Hernández- and María José Bamírez-Quintana Calibra- tion of machine learning models. In Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques, pages 128-146. IGI Global, 2010.
[43] Yarin Gal. Uncertainty in deep learning. University of Cambridge, 1(3), 2016.
[44] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Adv. Neural. Inf. Process. Syst., pages 6402-6413, 2017.
2017.
[45] Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. In Adv. Neural. Inf. Process. Syst., pages 3787-3798, 2019.
[46] Natasa Tagasovska and David Lopez-Paz. Single-model uncertainties for deep learning. In Adv. Neural. Inf. Process. Syst. pages 6417-6428, 2019.
[47] Yonatan Geifman and Ran El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International Conference on Machine Learning, pages 2151-2159, 2019.
[48] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. In Ady. Neural. Inf. Process. Syst. pages 13153-13164, 2019.
[49] José Mena, Oriol Pujol, and Jordi Vitrià. Uncertainty-based rejection wrappers for black-box classifiers. IEEE Access, 2020.
[50] Yukun Ding, Jinglan Liu, Xiaowei Xu, Meiping Huang, Jian Zhuang. Jinjun Xiong, and Yiyu Shi. Uncertainty-awat awaar training of neural networks for selective medical image segmentation. In Medical Imaging with Deep Learning, 2020.
[51] Biraja Ghoshal and Allan Tucker. Estimating uncertainty and interpretability in deep learning for coronavirus (covid-19) detection. arXiv:2003. 10766,, 2020.
[52] Eli Simhayev, Gilad Katz, and Lior Rokach. PIVEN: A deep neural network for prediction intervals with specific value prediction. 200.:::0006.51..30 2020.
[53] Jing Lei, James Robins, and Larry Wasserman. Distribution-free prediction sets. J. Am. Stat. Assoc., 108(501):28--87,, 2013.
[54] Jing Lei and Larry Wasserman. Distribution-fiet prediction bands for non-parametric regression.
J. Royal Stat. Soc. B, 76(1):71-96, 2014.
[55] Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Am. Stat. Assoc., 1 113(523) 2018.
13

---

[56] Rina Foygel Barber, Emmanuel J Candès, Aaditya Ramdas, and Ryan J Tibshirani. Predictive inference with the jackknife+. The Annals of Statistics, 49(1)) 507, 2021.
[57] Ryan Tibshirani, Rina Foygel Barber, Emmanuel Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019.
[58] Yaniv Romano, Evan Patterson, and Emmanuel Candès. Conformalized quantile regression. In Adv. Neural. Inf. Process. Syst., pages 3538-3380 -3488, 2019.
[59] Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009. 14/93, 2020.
[60] Stephen Bates, Anastasios Angelopoulos. Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-frre, risk-controlling prediction sets. Journal of the ACM (JACM), 68(6):1334, 2021.
[61] Nicolo Colombo and Vladimir Vovk. Training conformal predictors. In Conformal and Probabilistic Prediction and Applications, pages 55-64, PMLR, 2020.
[62] Anthony Bellotti. Optimized conformal classification using gradient descent approximation. arXiv preprint arXiv:2105.11255, 2021.
[63] David Stutz, Ali Taylan Cemgil, Arnaud Doucet, et al. Learning optimal conformal classifiers. arXiv preprint arXh::21 2021.
[64] Haoxian Chen, Ziyi Huang, Henry Lam, Huajie Qian, and Haofeng Zhang. Learning prediction intervals for regression: Generalization and calibration. In International Conference on Artificial Intelligence and Statistics, pages 820-828. PMLR, 2021.
[65] Yachong Yang and Arun Kumar Kuchibhotla. Finite-sample efficient conformal prediction. arXiv preprint 100::22114.3811,,
[66] Yu Bai, Song Mei, Huan Wang, Yingbo Zhou, and Caiming Xiong, Efficient and differentiable conformal prediction with general function classes. arXiv preprint arXiv:2202. 1009,, 2022.
[67] Matteo Sesia and Yaniv Romano. Conformal prediction using conditional histograms. In Adv. Neural. Inf. Process. Syst., volume 34, 2021.
[68] Anastasios N Angelopoulos, Amit P Kohli, Stephen Bates, Michael Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Image-to-image regression with distribution-free uncertainty quantification and applications in imaging. arXiv preprint arXiv:2202.05225. 2022.
[69] Matteo Sesia and Stefano Favaro. Conformalized frequency estimation from sketched data. arXiv preprint arXi::200..04270, 2022.
[70] Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values. arXiv preprint arXivv2104.088 082799 2021.
[71] Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian conference machine learning, pages 475-490, 2012.
[72] Rina Foygel Barber, Emmanuel Candès, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution-fiee conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2):455-482, 2021.
[73] Marco Cuturi, Olivier Teboul, and Jean-Philippe Vert. Differentiable ranking and sorting using optimal transport. In Adv. Neural. Inf. Process. Syst., pages 6861-6871, 2019.
[74] Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sorting and ranking, 2020.
[75] Harald Cramér. On the composition of elementary errors: First paper: Mathematical deductions. Scandinavian Actuarial Journal, 1928(1):13-74, 1928.
14

---

[76] Richard Von Mises. Statistik und wahrheit. Julius Springer, 20, 1928.
[77] Nikolai V Smirnov. Estimate of deviation between empirical distribution functions in two independent samples. Bulletin Moscow University, 2(2):3-16, 1939.
[78] Frank J. Massey Jr. The kolmogorov-smingg test for goodness fit. J. Am. Stat. Assoc., 46 (253):68-78, 1951.
[79] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[80] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 13:11288-15555 2020.
[81] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 data set, 2009. URL http
// //ww.cc cs. wwwottorooooo.ooo waw.e/oooddddkrriiii/iei html
[82] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001-13008, 2020.
[83] Connor Shorten and Taghi Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1 2019.
[84] I-Cheng Yeh and Che-hui Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert systems with applications, 36(2): 2473-2480, 2009.
[85] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 18:12204-1214 2020.
[86] Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. arXiv preprint arXiv:2202. /3415, 2022.
[87] Robin Dunn, Larry Wasserman, and Aaditya Ramdas. Distribution-free prediction sets for two-layer hierarchical models. Journal of the American Statistical Association, pages 1-12, 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
15

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasng new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "iAgcSKyV1GN", "reviewer": "Reviewer_Xxp1", "summary": "In this paper, a new training strategy which integrates conformal prediction in training stage is proposed. The proposed training algorithm uses an additional regularization term that encourages the conformity scores to follow a uniform distribution to improve the performance of final conformal prediction.", "questions": "See weaknesses.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Previous training stage doesn't consider the performance of conformal prediction, and hence results in sub-optimal problem. The proposed method addresses this issue by encouraging the conformity scores to follow a uniform distribution.\n\n2. The proposed differentiable approximation for the empirical CDF of the conformity score could be efficiently implemented. The validation dataset could be used in both training (for calculating $l_u$) and post-hoc calibration.\n\n3. The experiments with synthetic data show some positive points of the proposed method, and also give some intuitive analyses.\n\nWeaknesses:\n\n1. The authors mentioned that \"The idea is to mitigate overconfidence\" in Abstract. Does the proposed method (i.e., adding conformal loss term $l_u$ in loss function) directly mitigate the over-confidence issue? \n\n2. The proposed method could be a regularization term on the loss function. What about the relationship between it and other regularization methods (which have been used to address the calibration problem) like label smoothing?\n\n3. Other real-world datasets should be used in experiment part. This paper only uses CIFAR-10 for evaluating the effectiveness of the proposed method. As it is not too hard to classify, image augmentation processes are used to generate harder samples. Why not use datasets like CIFAR-100 or (Tiny-)imagenet for better evaluation?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fDcaB4Bn8MC", "reviewer": "Reviewer_9Dxm", "summary": "The authors propose a method of combining the learning and calibration phases\nof conformal inference, reducing overconfidence in the ML model, and giving\nsmaller prediction sets. They minimize a new loss with minimizes the\ndiscrepancy in conformity scores between the current model and an unknown\noracle.\n", "questions": "- I am not sure there is enough information in the current version of the paper in order to implement the algorithm. Section 3.3 is easy to intuitively understand, but it leaves me confused as to where to go in order to actually implement what is written in the paper. Could a more detailed explanation of the exact equation needed to reproduce $\\ell_u$ be included?\n\n# Summary\n\nOverall, I think the contribution is solid, but there can be some improvements made as reflected in my comments and questions. ", "limitations": "The authors have highlighted the computational complexity as a limitation. I am not aware of any adverse societal impacts of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- The method id well justified and comes equipped with in depth analysis\n- The experiments verify the claims made within the text and also provide ablations on the proposed components.\n- The experiments are thorough.\n\n# Weaknesses\n\n- Equation three seems to give incomplete infromation. $u$ and $\\tau$ are\ndefined but there is no mention of how they contribute to the function\n$\\mathcal{S}$. It is then defined in the appendix, but A2 in the appendix takes\n$t$ (not $\\tau$ but I guess they are the same?) as an argument and equation 3 takes $1 - \\alpha$ leaving $\\tau$ unused.\nThis needs to be cleared up.\n\n- The covariate shift experiments occur on the synthetic data. It would be nice\nto see if these results hold for larger datasets as well. For example, what\nwould happen if a model were trained with regular CIFAR10 data with no\ncorruptions in either the training or calibration set, and then tested with the\nrandom erase corruptions or CIFAR10-C [1] test set.\n\n- L237: states that figure A6 shows typically higher conditional coverage as\ncompared to all benchmarks. It appears there is actually a clear trend which\nshows that the proposed model performs worse on larger numbers of classes, yet\nthere is no mention of this. Can the authors provide any explanation for this?\n \n# References\n\n[1] https://github.com/hendrycks/robustness\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZAbPk_ZEsu", "reviewer": "Reviewer_3Bsi", "summary": "This paper proposes a differentiable loss function for training conformal predictors. In conformal prediction, a popular scoring function for classification sets is the APS method of Romano et. al. [1], as if the learned conditional distribution $\\hat \\pi_y(x) \\approx  \\pi_y(x) := \\mathbb{P}(Y = y \\mid X = x)$ is exact, then the constructed prediction sets are the smallest randomized prediction sets with the desired conditional coverage, i.e., satisfying $\\mathbb{P}(Y_{n+1} \\in C(X_{n+1}) \\mid X_{n+1} = x) \\geq 1 - \\alpha$. Typically, the way to go about solving this problem is to first learn $\\hat \\pi_y(x)$, and then plug it into the APS method (which is then calibrated using normal conformal techniques). Unfortunately, as the authors point out, predictions based on poorly pre-trained $\\hat \\pi_y(x)$ are hard to correct once $\\hat \\pi_y(x)$ is fixed. This paper links the two steps of conformal prediction into one joint loss, with a focus on improving conditional coverage via better recovering the oracle behavior of the true $\\pi_y(x)$ combined with APS. This joint objective is composed of the normal cross-entropy classification loss, together with a regularizer that encourages conformal scores on held-out data to be uniformly distributed (as they would be if $\\hat \\pi_y(x) = \\pi_y(x)$). The method is empirically validated on both synthetic and real datasets, though the gains are fairly minor depending on the setting.\n\n[1] Classification with Valid and Adaptive Coverage. https://arxiv.org/abs/2006.02544.", "questions": "- Low 80% top-1 accuracy seems pretty low for a ResNet-18 on CIFAR-10 (see, e.g., https://github.com/kuangliu/pytorch-cifar). This makes me question if the cross-entropy baseline is indeed well trained? Can you explain this difference?", "limitations": "The authors have adequately addressed the limitations and potential societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=== Strengths ===\n\n- The paper is well-written and well-evaluated. The method is also well-motivated. In particular, I appreciate the focus on conditional coverage rather than set size, as set size can be gamed when only subject to marginal coverage constraints.\n\n- The proposed algorithm is simple (at least when relying on pre-existing differentiable sorting algorithms). Proposition 1 yields good intuition (and also interestingly appears somewhat related to the fact that the distribution of $Z = F_{Y|X}(Y)$ is uniform for continuous r.v.s $Y$ with CDF $F_{Y|X}$, but here by construction for discrete $y$).\n\n- Though I have some questions about the empirical effectiveness on real data (see below), at least under certain settings the method can lead to smaller prediction sets with better conditional coverage---two important and impactful qualities for real-world deployment of conformal algorithms.\n\n=== Weaknesses ===\n\n- When factoring in early stopping, the empirical gains on real datasets vs. baselines (i.e., Hybrid) appear minor at best (e.g., the best seems to be a +4% gain in corrupted coverage on CIFAR-10 when comparing fully-trained conformal with early-stopped hybrid?). The raw (top-1) accuracy also seems to be negatively on the credit card default task (whereas standard conformal methods that don't modify the base conformal score don't affect the top-1 accuracy).\n\n- This is addressed in the paper, but the proposed method is quite expensive to train (2x compared to cross-entropy loss, as reported in Section 5). As a result, I'm not sure how well this would scale to larger models than a ResNet18. Given the somewhat small gains in efficiency/conditional coverage, the impact of this approach seems likely to be somewhat limited. (That said, I imagine that the \"Hybrid method\" of [1] is similarly slow, as it also involves a differentiable sort.)\n\n- One thing which bothers me is the data inefficiency of the proposed approach. It seems rather wasteful to only use labeled data in $\\mathcal I_2$ for regularization. It seems if the main goal is to simply get a better estimate of $\\pi_y(x)$ by reproducing its behaviour when plugged into conformal APS (e.g., uniform score distr.), what if we just use that data to train a bigger, modern model (e.g., see models in [2]), optionally with better regularization (e.g., calibration objective in [3])? Or, since the method also requires validation of $\\lambda$, early stopping on the same validation set (as done in the experiments, which improves baselines substantially) also seems to be a fair data-wise comparison. \n\n- Likewise, from the paper experiments, benefits become more pronounced with more data, however, one could argue that uncertainty estimation is particularly important for tasks in which we don't have much data (and hence base models are poorly trained). Note that this data inefficiency is also shared by [1], but the objective in [1] can dependent on properties of the set $C$ that may be orthogonal to conditional coverage (and recovering $\\pi_y(x)$ may not yield the oracle).\n\n=== Minor ===\n\n- Notation: In several places $|\\mathcal I_*|$ should be used in place of $\\mathcal{I}_*$ (e.g., L165). \n\n[1] Learning Optimal Conformal Classifiers. https://arxiv.org/abs/2110.09192.\n\n[2] Revisiting the Calibration of Modern Neural Networks. https://arxiv.org/abs/2106.07998.\n\n[3] Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings. https://proceedings.mlr.press/v80/kumar18a.html.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ef1wFM7Mp98", "reviewer": "Reviewer_m7U1", "summary": "The paper combines ideas from conformal prediction and differentiable ranking/sorting to develop a loss function that improves the predictive uncertainty of a classifier by training it to make predictions with a desired conformity score. The paper reports competitive coverage and conformal prediction set sizes on synthetic data, CIFAR10 and a credit card default dataset.", "questions": "I'm overall not too confident in my review due to being unfamiliar with the related literature and commonly used experimental setups and results, but I'd appreciate the following questions being clarified:\n\n* Are you indeed using a two-level procedure even with your conformal prediction loss? If yes, is it possible to use the loss as a one-level approach and if so, how does it perform? I had a look at the appendix and did not find anything there, but might have missed it.\n* Do you have an explanation for where the need for early stopping arises? I can imagine that with an added regularizer this can happen, but the bad test accuracy for the cross-entropy baseline on CIFAR10 seems extremely strange to me, I don't think I have ever seen the test accuracy of a properly trained Resnet go down over time on that dataset. Is it due to the data corruption? That being said, it would be nice to have results trained on clean data as a baseline to see that the implementation is reasonable. \n* How do different choices of $\\lambda$ affect performance?\n\nOther (minor) notes:\n* I'm a bit uncomfortable with some statements in the introduction. In particular, neural nets trained with proper scoring rules should be calibrated in the infinite data limit. Intrinsically noisy data is not problematic, as this would be reflected in the training set and lead to less confident predictions. The problem in benchmarking on datasets such as CIFAR10 is the opposite, namely that they are highly curated and there is hardly any noise in the data, but once the underlying data distribution shifts the predictions become overconfident. I'm probably making some statements here myself that are not perfectly accurate either, but I'd encourage have another pass over the first paragraph of the introduction and ensure that all statements are technically precise and correct.\n* Exchangeability of the data seems to be a core assumption, it would be nice to have an evaluation of the method under some shift of the test distribution on the real data, e.g. using different degrees of corruption as in (Ovadia et al., 2019. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In NeurIPS).\n* Table captions should go above the tables (unless the style instructions have been changed).", "limitations": "The computational overhead (roughly doubled training time in the experiments) is discussed explicitly. I did not see discussion around the reliance of the proposed loss on a post-hoc conformalization step.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The method is novel as far as I can tell (although I am not familiar at all with the related literature on frequentist approaches for uncertainty estimation) and appears to be technically sound.\n* The technical setup is well-structured and clear for the most part (although see exceptions).\n* The proposed method seems to perform the best on the synthetic data and is competitive on real data.\n\nWeaknesses:\n* I found section 3.3 quite vague and think the paper would benefit from recapping the specific techniques it uses from the referenced papers. As it stands, the text is not self-contained and I don't think I'd be able to implement the method without working through other works. The algorithm box does not help in this regard as it is quite wordy rather than technical and concise.\n* I was fairly surprised that the proposed method appears to require a post-hoc calibration step (lines 199-201, 259, 305), even though the introduction criticizes this two-step approach as a limitation of conformal learning.\n* The empirical results on the real data seem to be somewhat of a mixed bag with many close scores. Reporting results without early stopping seems unnecessary to me since the classification performance is significantly better with early stopping (although I am surprised by the need for it; see questions).\n* There is no ablation study on the hyperparameter for the weight of the conformal loss. I'd imagine that this parameter allows for trading off more accurate vs better calibrated predictions, but it would be helpful to see this confirmed experimentally.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept