Confidence Estimation Using Unlabeled Data
Authors: Chen Li
Xiaoling Hu
Chao Chen
Venue: ICLR 2023 poster (2023)

Abstract: Overconfidence is a common issue for deep neural networks, limiting their deployment in real-world applications. To better estimate confidence, existing methods mostly focus on fully-supervised scenarios and rely on training labels. In this paper, we propose the first confidence estimation method for a semi-supervised setting, when most training labels are unavailable. We stipulate that even with limited training labels, we can still reasonably approximate the confidence of model on unlabeled samples by inspecting the prediction consistency through the training process.  We use training consistency as a surrogate function and propose a consistency ranking loss for confidence estimation. On both image classification and segmentation tasks, our method achieves state-of-the-art performances in confidence estimation. Furthermore, we show the benefit of the proposed method through a downstream active learning task. 

Published as a conference paper at ICLR 2023
CONFIDENCE ESTIMATION USING UNLABELED DATA
Chen Li
Xiaoling Hu
Chao Chen Stony Brook University
Stony Brook University
Stony Brook University
ABSTRACT
Overconfidence is common issue for deep neural networks, limiting their deploy- ment in real-world applications. To better estimate confidence, existing methods mostly focus on fully-supervised scenarios and rely on training labels. In this paper, we propose the first confidence estimation method for a semi-supervised setting, when most training labels are unavailable. We stipulate that even with lim- ited training labels, we can still reasonably approximate the confidence of model on unlabeled samples by inspecting the prediction consistency through the train- ing process. We use training consistency as surrogate function and propose a consistency ranking loss for confidence estimation. On both image classifica- tion and segmentation tasks, our method achieves state-of-the-arr performances in confidence estimation. Furthermore, we show the benefit of the proposed method through a downstream active learning task.
INTRODUCTION
Besides accuracy, the confidence, measuring how certain a model is of its prediction, is also criti- cal in real world applications such as autonomous driving (Ding et al. 202T) and computer-aided diagnosis (Laves et al., (211)). Despite the strong prediction power of deep networks, their overcon- fidence is very common issue (Guo et al.) 2017 Nguyen al.) 2015 Szegedy al. 2014). The output of a standard model, e.g., the softmax output, does not correctly reflect the connidence. The reason is that the training is only optimized regarding to the training set (Naeini et al.) 2015), not the underlying distribution. Accurate confidence estimation is important in practice. In autonomous driving and computer-aided diagnosis, analyzing low confidence samples can help identify subpopu- lations of events patients that deserve extra consideration. Meanwhile, reweighting hard samples, ie.. samples on which the model has low confidence, can help improve the model performance. Highly uncertain samples can also be used to promote model performance in active learning (Sid-)
diqui et al. 2020 Moon et al. 2020
Different ideas have been proposed for confidence estimation. Bayesian approaches (MacKay 1992
Neal 1996) Graves 2011) rely on probabilistic interpretations of a model's output, while the high computation demand restricts their applications. Monte Carlo dropout (Gal & Ghahramani 2016
introduced to mitigate the computation inefficiency. But dropout requires sampling multiple model predictions at the inference stage. which is consumin.... Another idea is to use an ensemble of neural networks (Lakshminarayanan et al.] (017)). which can still be expensive in both inference time and storage. To overcome the inefficiency issue, some recent works focus on the whole training process rather than the final model. Moon et al. (2020) use the frequency of correct predictions through the training process to approximate the confidence of a model on each training sample. Geifman al. (2018) collect model snapshots over the training process to compensate for overfitting and estimate confidence.
However, most existing methods purely rely on labeled data, and thus are not well suited for semi- supervised setting. Indeed, confidence estimation critically needed in the semi- cemi-suuer setting, where we have limited labels and large amount of unlabeled data. A model trained with limited labels is sub-optimal. Confidence will help efficiently improve the quality of the model, and help annotate the vast majority of unlabeled data in scalable manner (Wang et al. 2022 Sohn al.
2020 Xu et al. 2021).
In this paper, we propose the first confidence estimation method specifically designed for the semi- supervised setting. The first challenge is to leverage the vast majority of unlabeled data for confi-
"Email: Chen Li www.chenn@tttonhrroo..

---

Published as a conference paper at ICLR 2023
dence learning. For data without labels, our idea is to use the consistency of the predictions through the training process. An initial investigation suggests that consistency of predictions tends to be correlated with sample confidence on both labeled and unlabeled data.
Having established training consistency as an approximation of confidence, the next challenge is that the consistency can only be evaluated on data available during training. To this end, we propose to re-calibrate model's 's prediction by aligning it with the consistency. In particular, we propose a novel Consistency Ranking Loss regulate the model's output after the softmax layer so it has a similar ranking of model confidence output as the ranking of the consistency. After the re-calibration, we expect the model's 's output correctly accounts for its confidence on test samples. We both theoretically and empirically validate the effectiveness of the proposed Consistency Ranking Loss. Specifically, we show the superiority of our method on real applications, such as image classification and medical image segmentation, under semi-supervised setrings. We also demonstrate the benefit of our method through active learning tasks.
Related work. There are two mainstream approaches for confidence (uncertainty) estimation: confi- dence calibration and ordinal ranking. Confidence calibration treats confidence as the true probabil- ity of making correct prediction and tries to directly estimate it (Plati 2000, Guo et al. 2017, Jungo & Reyes| 2019 Zadrozny & Elkan 2002 Nacini et al., 2015). For any sample, the confidence estimate generated by a well-calibrate classifier should be the likelihood of predicting correctly. Directly estimating the confidence may be challenging. Instead, many works focus on the ordinal ranking aspect (Geifman et al., 2018 Geifman & Yl-vaniv] (2077) Mandelbaum & Weinshall 2017] Moon et al., [0200] Lakshminarayanal et al., 2017). In spite of the actual estimated confidence val- ues, the ranking of samples with regard to the confidence level should be consistent with the chance of correct prediction. A model with well-ranked confidence estimate can be widely used in the field of selective classification, active learning and supervipe learning (Siddiqui al. 2020: Yoo & Kweon [ & Savarese 2018: Tarvainen & Valpola| 2017, Zhai et al.. [2199] Miyato et al.,
2018 Xie et al.| 2020 Chen et al. 2021 & Yin 2020). Semi-supervised active learning meth-
ods (Gao et al. 2020 Huang et al. 2021 only focus on finding high-uncertainty samples through
training actions, thus are unable to be applied to estimate uncertainty for unseen samples.
2 CONSISTENCY A NEW SURROGATE OF CONFIDENCE
Our main idea is to use the training consistency. i.e., the frequency of a training datum getting the same prediction in sequential training epochs, as a surrogate function of the model's confidence. In this section, we first formalize the definition of training consistency. Next, we use qualitative and quantitative evidences to show that consistency can be used as a surrogate function of confidence. This justifies the usage of training consistency as supervision for model confidence estimation, which we will introduce in the next section.
training consistency. Assume given dataset with labeled and p unlabeled data, D = =X, YU, Here X = [ai,
Definition:
), x; E X' is the set of labeled data with corresponding labels Y {v 3, 34 E y = {1..... K}. The set of unlabeled data U {#n+11.... kn+p): 2,
X' cannot be directly used to train the model, but will be used to help estimate confidence. We assume simple training setting where we the labeled set (X, to train a model f(x, y; W): : X' x y > [0, 1]. The method can naturally generalize to more sophisticated semi- supervised learning methods, where unlabeled data can also be used. For any data, either labeled or unlabeled, I; E X U U, we have the classification D; = arg yay (xt, yi W). Note that tradi- tionally the model output regarding the classification label, [1i, 9z; W) = max [( yr W), is used the confidence.
Our definition involves the training process. Denote by W the model weights at the 1-th training epoch, 1,... T. We use ýf arg max (x,, w/ to denote the model classification for
sample x; at the t-th epoch. For a sample T;, we define its training consistency as the frequency of getting consistent predictions in sequential training epochs during the whole training process (T epochs in total):
1 T
1
(1)
Qualitative analysis shows consistency is a good surrogate of confidence. We provide a quali- tative example in Fig. [ with the feature representations. We observe that the data further from the
2

---

Published as a conference paper at ICLR 2023
1.0
0.8
0.6
0.4
0.0
(a) Label
(b) Softmax
(c) Conrectness
(d) Consistency
Figure 1: Feature representations of 6000 unlabeled CIFAR1O data. The model is trained on 20% labeled data. We use t-SNE for dimension reduction and only show data of three classes. (a) different class labels: O-blue, 1-yellow, 9-brown. (b) the softmax output is clearly overconfident. (c) and (d) are correctness and consistency maps respectively. They are similar to each other and seem to be well correlated with confidence.
decision boundary have higher consistency, and the ones closer decision boundary have lower consistency. This is consistent with our intuition and seems to correlate with model confidence. Although we do not have the ground truth of confidence, is reasonable to believe that data further from the decision boundary will have higher confidence, whereas data near the decision boundary will have lower confidence.
The data representation is acquired by training model Ressel 10) on only 20% of CI- FARTO training set. We extract the penultimate fully connected (FC) layer output as the feature representation. Here we use t-SNE to visualize the feature representations of 80% unlabeled train- ing images. For ease of exposition, we only focus on class O, class and class As shown in Fig. aa), since the model is trained with limited labeled data, the three classes are not completely separated in the representation space. We expect lower confidence data to be closer to the decision boundary. However, this is not well reflected by the model confidence output, i.e., the maximum softmax output, [ 0;; W). As shown in Fig. (b). softmax output tends to be overconfident.
Meanwhile, calculate consistency unlabeled data and visualize them in Fig. dd. The sistency map seems much more reasonable and better correlates with model confidence. The con- sistency is high within each class, but slowly transits low towards the decision boundary. Since we do not have ground truth of confidence, as a reference, we also calculate and visualize the cor- rectness of data, a known good proxy of confidence in fully-supervised setting (Moon et al. 2020). The correctness of datum is the frequency at which the model makes correct classification on x across different epochs, i.c., E 1 0 As shown in Fig. c). the correctness map is very similar to consistency. This further justifies our proposal of using consistency.
Please note that this analysis does not justify correctness good confidence surrogate when train- ing with limited labels. When labeled data limited, we can only calculate and use correctness on limited amount of samples. They cannot provide sufficient supervision for confidence learning. This will be evident in our experiment section.
Quantitative analysis. We further evaluate the quality of consistency confidence surrogate using a popular confidence evaluation metric, the area under the risk-coverage curve (AURC) and the AURC (E-AURC). Based on the ordinary ranking principle, AURC and E-AURC measure how well an estimated confidence is correlated with the chance of correct prediction; a sample with high confidence should have high chance to be correctly predicted. A low AURC or B-AURC indicates the estimation is a good approximation of the true confidence. A formal definition of AURC and E-AURC is provided Sec. 3.2
We evaluate AURC and E-AURC on the training data for consistency, correctness and softmax out- put of the model at different training epochs with regard to a model trained on limited labeled data. The setting is the same as in the previous qualitative analysis. The results are shown in Fig. 2 We observe that overall all three confidence proxies improve their quality as the training continues (i.e., their AURCs/E-AURG decrease as the training progresses). Consistency is better than softmax output throughout the training (i.e., having a lower AURC or E-AURC). Correctness has the best performance among the three (i.e., the lowest AURC or E-AURC). However, correctness is unavail- able for unlabeled data, thus does not help learning confidence in our problem. In Append. A.8 we compare training consistency with other label-free uncertainty surrogates.
3

---

Published as a conference paper at ICLR 2023
3 LEARNING TO ESTIMATE CONFIDENCE: CONSISTENCY RANKING LOSS
We have established that consistency good approximation of the con- fidence. However, consistency is only available for training data. At the inference stage, we do not have such measure for test data. To this end, we propose a novel Consistency Ranking Loss during training to en- force the model's maximum softmax output approximates the consistency well. The hope is the inference stage, the maximum softmax out- put will well approximate the consis- tency, and thus the confidence.
0.00
Corracnness
0.95
0.30
0.40
0.00
0.30
epochs
(a) AURC
(b) E-AURC
Figure 2: Performance of consistency as confidence esti- mator. We evaluate the performance of softmax output, con- sistency and correctness terms of popular metrics, AURC and E-AURC. We train model with only 20% of CIFARIO labeled data, and evaluate on the remaining unlabeled data.
The consistency ranking loss is pairwise ordinal ranking function to enforce the maximum softmax output to be consistent with the training consistency on all labeled and unlabeled training data X UU. Re- call for the i-th training datum, x; X U we denote by o; its training consistency (Eq. ()). For convenience, we denote by K; the model's maximum softmax output for r; The loss function en- forces the two be consistent in terms of ordinary ranking, i.e. if e < Cs. then K; < Kg. Formally,
Loms (1)
Σ
ma (c. -G) - (Ni k:)).
The loss sums over all pairs of training data and penalizes pair (i,s) when v; has lower confidence than and the difference C. C, is bigger than the difference K, Rg. Through training, the confidence estimator и learned to have similar gap as the consistency c's, i... K₂ K₁ 2 C2 - Ci. Ve, > G.- The loss seems expensive as it is quadratic to the sample size. In practice, however, computation is not a major issue. We evaluate the loss on samples within each minibatch (both labeled and unlabeled). The loss is then only quadratic to the minibatch size. As shown in the experiment section, this suffices practice. The details of the pairing strategy are provided in Append. A.2
To fully exploit labeled training samples for confidence estimation, we also incorporate correctness into training, using ranking loss Lcorr. Note Loorr is only evaluated labeled samples. Our overall loss C is
L Lee + >i X2Ccons
(3)
Here Loe is cross entropy loss on labeled samples. A and X2 are the weights of losses. The pseudo code of our method is shown in Append. A.6
3.1 EXPERIMENT VERIFICATION
In practice, it is intractable to compare all possible sample pairs even for small batch data. We use experiment to show the empirical performance of consistency ranking loss on CIFAR-IO dataset. Here we adopt the same experimental setting as Sec. 2 All the results are collected from unlabeled training set (40000).
shown in Tab. [ our consistency ranking loss makes the AURC (E-AURC) of model confidence output (maximum softmax output) approximate the AURC (E-AURC) of train- ing consistency very well. The experimental results indicate that our consistency ranking loss can achieve very good performance even optimized on minibatch data with limited pairs.
3.2 THEORETICAL GUARANTEE
In this section, we provide theoretical analysis of the proposed consistency ranking loss (Eq. (2). We show that the loss upperbounds the quality of the confidence estimator k. Informally, our main
In general, K2 can he any confidence estimator (e.g., one separate head output ki).
4

---

Published as a conference paper at ICLR 2023
Table 1: The quality of optimization on consistency ranking loss. Softmax (ours): the maximum softmax output trained with consistency ranking loss. TC (ours): training consistency. Diff. (ours): the evaluation metric difference between our maximum softmax output and training consistency.
Softmax, TC, Diff. are corresponding results without consistency ranking loss. All values x x10².
Dataset (labeled size)
Method
E-AURCI
AURCL
Method
E-AURCU
AURCL
Softmax
29.07
40,57
Softmax (ours)
19.35
32.57
CI
TC
17.63
29.13
TC (ours)
17.17
30.39
(10000)
Diff.
11.44
11.44
Diff. (ours)
2.17
2.17
theorem (Thm. T states that the loss upperbounds the difference between the confidence estimation performance of and the consistency c, in terms of the popular metrics AURC and E-AURC. In other words, as we minimize the loss, K will have a similar performance as the consistency c in terms of confidence estimation.
Definition of AURC. To evaluate confidence estimator correlated with the chance of a sample being correctly predicted by classifier f, the area under the rivk-coverage curve (AURC) is proposed. Given dataset with m samples pm (Xm: Ym)) and a confidence estimator k, the em-
E1 1(0i. ((002))))(2)) where ÿ arg maxyey [I y PDm) m =m go (u.)
pirical selective risk f(J, go (Dm)
We define indicator function ((i y1) 1(ū + y: Here, go(x;) = 1k,, > 0} is selec- tion function thresholding on confidence scores K. The AURC is defined as AURC(f, k/ (m)) =
1 Eece f(f. (P.).. where θ is the set of all distinct values of confidence scores Xmss i.e., Im).
m ②
E-AURC was proposed as a normalized version of AURC. N., is an optimal confidence score function of f on Dm. yielding the correctly predicted samples having higher confidence scores than misclassified ones. K. minimizes AURC(S, - (Dm). Excess-AURC (E-AURC) is defined as, (URC(f,x|DD,,
E-AURC(f, x| Dm) f is not an optimal classifier, then к assigns higher confidence score to correct predictions than incorrect ones. In such way, can filter out correct predictions by
AURC(F, [D]))). When E-AURC(f, xD)... 0 and
Next, we formally state and prove our main theorem. Let T()lDm) be the set constituted by all the correctly predicted samples, T())Dm)= = {ar ÿ y. arg myx f()),, (I,, y) D Here we define R,, a ranking function based on confidence score function ks on Xm, (x) E 21.2.... m,),, Vr E Xm. If n(x;)) < sαz,) than R2 (x;)) > (x,), Vz,, Ty E Xm. Here we assume both &. and Rs are one-to-one functions. Given Rx, we define ranking set Hg = [Rx(x)
ÿ = y, 9 = arg m f(r). (z, v) E Dm). Here we define c(x,) = Gi I; E Xm. Note that training consistency is a kind of confidence estimator and in the context below, we will use c to denote training consistency. Here we define R. certain way such that it one-to-one function (More details are included in Append. A.3).
Definition 3.1. Given ranking function Re and Rx, we introduce set De flc(R, (i) of (R)* = 7m RR, (x)). = 7m (Ry (n.( (x)) + sign( R, (x) 7m (R (x)))), E T, R. (0) & T())D,)) where 7/m = arg min .H.) Ever (x) - > (R,(())) [I) sign is sign function, is the collection of the bijective function between the elements of H₂ and He
Theorem 1. Given dataset Dm = (Xm: Ym)) and training consistency Cm
(AURC(,,(D..) - AU m mimin De Leons (J, Xm,; Cma)
(4)
We notice that [E-AURC[, c(D... (EAURC(,*((..))) (AURC(J,((D.) AURC(S, (Dmm)) thus the theorem above works for E-AURC too.
Proof. Let (i) be the set constituted by all the correct predicted samples with confidence less or equal than R (i), Ti)(i) (a : (x) S i, ÿ = y = arg max, [(x); (x. y) E Dm T.(i) is the corresponding set introduced by training consistency c. We define Holf. k(mm) Σ 1I#Ix(1) # #Tc(i) and (f, k()m))) E 1(#Tx(1) + #Tc(i)). It can be proved:
AURC(F.CD..) - (URC(F,)DD..)) He(
(5)
m
5

---

Published as a conference paper at ICLR 2023
There is fact that Vi (1. if (1)
#Tc())), than
Rff.gn (i) (m)) = f(f.9n; (8) (Pm), so |AURC(F, c((m) AURC(f, k) Dm)] S
M. Σ HT. (F(19R= [ m) F(f, (2 (Pm)) We also have cf(
E II - (z)) Si) 1 = Uj), where új = arg max,ey (zz)) and (æz, Ub) E 2m Here
we define (ff.clmm = #T(
(f which is the number of correct predicted
samples with lower training consistency than at least one incorrect sample. Then we prove that:
Ccons(f, Xm; Cm)
(6)
M min De
m
A detailed proof version is provided in Append.
A.3
EXPERIMENT
To verify the effectiveness of our method in confidence estimation with limited labeled data, we start with evaluating on image classification and medical image segmentation tasks. Active learning is usually used as a follow-up task of confidence estimation. Here we implement active learning on image classification tasks to show that the highly uncertain samples picked up by our method will promote the learning process in a better manner. We also evaluate our method on 10/100 through anomaly detection in Append. A.7
4.1 IMAGE CLASSIFICATION
In this section, we show that our method can generate reliable confidence estimates with a small portion of labeled samples and majority of unlableled samples through classification tasks. We evaluate our method on benchmark datasets, CIFAR-10 and CIFAR- (Krizhevsky et al. 2009). Besides, we also provide the results on cancer survival dataset in Append. A.9
Implementation details. To show the superiority of our method in making use of unlabeled training samples, We compare our method with other baselines under different portions of labeled training data: 5% (2500). 10% (5000), 20% (10000), 100% (50000). As for network architecture, we adopt PreAct-ResNet1 10 (He et al. 2016) for CIFAR- 10 and DenseNetBC (k- = 12, = 100) (Huang al., 2017) for CIFAR-100 100. All methods are trained by SGD with a momentum of 0.9 and a weight decay of 0.0001. We train our method for 300 epochs with the mini-batch size of 192, in which 64 are labeled, and use initial learning rate of 0.1 with reduction by factor of 10 at 150 and 250 epochs. A standard data augmentation scheme for image classification is used, including random horizontal flip, 4 pixels padding and 32 x 32 random crop. More details are included in Append. A.
For our method, we set the loss weights λ1, A2 in Eq, 3 as 0.5, 0.5 respectively, and use maximum softmax as the model confidence output for both training and evaluation. We compare method with cross entropy loss only (Softmax). CRL (Moon et al.) 2020). AES with 30 snapshot mod- els (Geifman et al. 2018), Mcdrop with 50 stochastic ensembles Gal & Ghahramani (016) and Aleatorie=MCdrrpour (Kendall & (017). As for evaluation, besides the ordinal ranking evalu- ation metrics AURC and E-AURC, we also adopt three widely used calibration evaluation metrics, the expected calibration error (ECE) (Naeini et al. 2015), negative log likelihood (NLL) and the Brier score (Brier et al. 1950). The false positive rate at 95% true positive rate (FPR-95%-TPR) is also used to evaluate confidence estimation performances.
The experimental results are shown in Tab. (The results of 'full' setting are included in Ap- pend. A.5). All the experiments are repeated for five times, and we report the means and standard deviations. From the results, we can see that the proposed method achieves better performances in both confidence estimation and classification accuracy. The improvement of our consistency rank- ing loss is even more salient under the scenarios of less labeled training samples. This indicates that our method utilizes unlabeled samples to encourage neural network classifier to generate believable confidence estimates effectively.
It is also obvious that the improvement of our method is more significant CIFAR-100. This is because for CIFAR- 100, the number of training samples in each class is much less than CIFAR-10, which worsens 'the lack of samples' effect in confidence estimation and classification. This further demonstrates the necessity of utilizing the confidence information unlabeled samples.

---

Published as a conference paper at ICLR 2023
Table 2: Comparison of confidence estimates on CIFAR-10/100 with various labeled training data sizes. The best results are shown in bold. We multiply the values of AURC & E-AURC by 10", FPR by 10a and NLL by 10 for clarity.
Dataset (labeled size)
Method
Acc1
AURCI
EAURRH
FPR-95.
ECE.
NLL
Brier.
CIFARIO
Soflmax
0.717:10.08
109.78:16.69
65.32.13.22
75.64:1.559
00.23
14.83.0600
47.17:1.553
AES
0.713::0.022
108.94110.87
63.34.1 1.02
74.99
16.82.1047
14.46:10.33
44.36.10.32
CIFARIO
Medrop
0.700:20.066
122.45:6.37
72.42-389
76.43=11.99
16.76=0.59
16.4940.53
16.18:1..19
(2500)
MC
0.7022
15.076115.06
70.66.1..9
75.29.1..35
1.55
16.30.1.23
45.6213.49
CRL
0.718=0.003
105.92:42.27
105
61 1.2277
74.17:40..0
03.15
10.63:0.119
42.20:10.555
Ours
0.755=0.004
81.50:+2.70
48.72=1.75
71.42:+1008
0.77
0.56
36.24+0..53
Softmax
0.705
60.54:1:.44
18.02-+1.14
68.64:
14.73-10.20
11.26::0115
34.28
AES
0.799=00.002
11.68+1.04
29.99=0.80 80
60.44+:0.90
0.16
9.46=00099
28.51+0.21
CIFARIO
Medrop
0.808=00003
52.94=-1007
33.25=00.47
67.70=
8.80=00244
10.41=0..0
29.01=0.43
(5000)
Aleatoric-MC
0,810=00.001
10.96=1..47
31.73 36
67.46
8.67
10
28.09
CRL
0.807=00002
51.76+1911
31
65.94=11.31
9.08
7.16=0.17
29.21+0044
Ours
0.808
4..4
26,88
64.64:10.93
5,26-10.43
6.42
26.88 10.43
Softmax
0.849=00000
1,,4
23.44=0.93
63.34 1.88
01:03=0.13
0.49=0.09
26.59
AES
0.005
30.75:10.88
07.0 0..7
6.61
0,23+00221
21,68
CIFARIO
Medrop
0,865 0.004
28.58.11.08
19.0910.58
61.39 1
5.68 1..34
1.22
2..37
(10000)
MC
0.005
28.64:10070
19.22:10.65
2.
5.75
7.47
20.47.10.23
CRL
0.856.10.001
31.28 0.0.32
20.51
61.63.11.36
5.7110.211
1.00
11.71
0.860:10.022
1..59
18.19-0..6
59.23
3.99-10.17
0.03
20.7740.18
CIFARI0O
Softmax
0.292=0.000
506, 7.5
139.23:56.38
19.13=1.554
31.89
25.
97.1
AES
0.289:10.009
509.38:12207
157.50-1:300
1.393:11.38
00.1710833
08.63
95.77110.29
CIFARIO0 (2500)
Medrop
0,269=0.005
542,10+7.75
165.06-+2.000
79.95
42.43=0.84
19.71
108.54=1.07
Aleatoric-MC
0.269=00.005
542.86::10.72
165.56:+3.95
80.96:+1.63
42.48:11.70
49.61=22.02
108.81=1778
CRL
0.287 100004
1.5.43
152.8211.288
79.12.1
29.01
37.33
1,,40.1.34
Ours
0.005
399.49-16.23
133.1543.88
75.26:10.82
0..5
03.5410.39
0..24
1.00550.005
14.30911.31
132.9510.95
77.1011.18
10.48
86.17.10.73
AES
0.419-000005
335.02-14.86
118.75:10.87
73.30.10.73
22.89-10.69
34.04410.63
77.93-10.74
CIFARI00 100
Mcdrop 0.394=20.004
379.60
4.24
141.06+22.55
77.04+0.30
39.32=0044
14.4341.19
94.73+0.72
(5000)
MC
0.394:10.002
378.72.13.47
140.5411148
07.48.10.80
19.12.1.49
44.19.10.95
94.77
CRL
0.43::0.0033
322.94-33.17
122.50
75.28
29.51
31 50
82.09:10.54
0,482=00005
266.08+5887
100.22+22400
1.850=
08.61=0.54
23.99:0.47
70.33-10.91
Softmax
0.546:00004
214.1742.81
90.78:10:99
72.72:10.52
24.31
24.19:10.44
67.41
AES 0.546=0.002
209.77+3.62
86.38+2.24
71.55=
19:63=0.20
24.29=00.46
63.62+0042
CIFARIO0 (10000)
Medrop
0.523=20.044
238.32+:5.79
100,70=+3.53
0,799+00,72
00.5
31.73+1.005
15.05=1.022
MCC
0.521=00.006
240.02::6.93
100.94
73.15 1.9
30.5420.55
11
75.27
CRL
0.563=00.05
196.11=:2.35
82.976:10.95
1.18
20.71-0.31
21.20=0.19
63.02+0.58
Ours
0.590
168.57-1:2.31
70,26 0,89
68.30
14.34 0.32
17.25:10.22
56.23 0.49
4.2 MEDICAL IMAGE SEGMENTATION
Even though deep neural network has achieved very impressive performance in medical image seg- mentation (Litjens et al. 2017). it is still important to estimate the model confidence samples, so the expert can get involved in time to avoid misjudgement. This emphasizes the importance of detecting and reacting to the failure of deep learning models. Thus, confidence estimation is promising way to manage this reliability concern (Jungo & Reyes 2019). On the other hand, im- age segmentation (dense label prediction) task is a good scenario to show the effectiveness of our consistency ranking loss, as the convergence of pairing loss is much more challenging. Here we use publicly available dataset conduct the experiment: the international skin imaging collaboration (ISIC) lesion segmentation dataset 2017 (Codella et al. 2018), which consists 2000 training, 150 validation and 600 testing annotated images.
Implementation details. We conduct experiments with various portions of labeled training samples: 1/16 (125), 1/8 (250), 1/4 (500) and full (2000). We use UNet (Ronneberger al. 2013) with ResNet34 backbone as the model architecture for all methods. The loss weights λ1. X2 are set to 0.5, 0.15 for segmentation task. All models are trained by Adam with a learning rate 0.0001. We train our method for 200 epochs with a mini-batch size of 96 (32 labeled and 64 unlabeled). As for augmentation, a scheme with resizing (192 x 256), random scale, random crop (192 x 256), random horizontal and vertical flip is used. Here we use mean the segmentation quality. More details are provided in Append. A.I
Sereeccttoo-vvere:e. Union (IOU) to evaluate
The results are shown in Tab. 4 (The results of 'full' setting are included in Append. A.4) We ob- serve that our method can produce more reliable confidence estimates than other baseline.. The lack of labeled training samples has little impact on the segmentation accuracy (mIOU), but deteriorates the confidence estimation quality severely. Our method shows robustness to the shortage of labeled samples in terms of confidence estimation, because of the effective utilization of unlabeled samples.
7

---

Published as a conference paper at ICLR 2023
Table 3: The correctness supervision ablation study results on CIFARIO
(labeled
Method
Acct
AURCJ
E-AURCI FPR-95,
ECE,
Brier
CIFARI
(I.I)
76.73:.5.79
55.8814.87
7.84 28.51:00.71
Unified normalization
0.812400.001 48.03=2.31
29.07+2.26 64,83+0.799
9.3440.36
7.33+00.06 29.87=00.02
5000
w/o corr
0.816:1:00044
29.16:141 47.12:+1.87 66.37:20.86
5.39::0336
6,7440.16
27.38+0,32
0.81810.000
1.04 26.68-0.666 6..6441..33
5.26:10.43 6.42100.88 26.88-10443
Table 4: Comparison of confidence estimates on ISIC2017. The value setting is the same as Tab.
Dataset (labeled size)
Method
miOUt
AURCJ
E-URRC
FPR-95,
ECEL
Brier]
ISIC2017
Softmax
0.801
34.36.7.776
31.327922
60.042.11
6.43
3.220046
13:96:0211
AEES
0.802
21.12:11.10
18.07
57.26.14.65
5.4610.37
4,1710022
12.54.00.53
Medrop
0.801+0.005
30,2343..81
27.1943.75
61.45+1.25
6.36+t0.30
4,05+0333
13.88:00449
125
Alealoric+MC
0.802110.001
27.66.1:187
24.624::93
59.55:10.84
6.21
3.86::029
13.68:0.13
CRI.
0.810110.044
22.14.12.82
19.33:12.81
62.15.1..75
4.11
2.86.10.44
12.25:10422
Ours
0.812:10.007
1..1
11.23:10.66
56.81-12.79
3.05
2.31
11.464:0.58
Softmax
0.819
28.91
26.45:14.72
58.51
5.62.10.20
3.35
12.45.10.18
0.8194:0.002
18.08:
15.6041.433
24.09:12.26
4.57.10.21
11.38:00.16
Medrop
0.819:100008
25.38:14.76
22.89:14.57
58.3111..4 1.111.1.499
5.45100.00
3.39
12.27.1067
MM
0.817:1..005
25.9611..6
21.37:1..80
57,80:1:1
5.49:10.33
3.43:0.29
12.43:10.00
CRL
0.818+00002
16.43:+0.94
13.90=00.00
62.56-+9.111
3.93+0091
2.60::.25
11.64+0882
Ours
0.817:00007
12.79:10.79
10.21=:0.22
54.51+-2.79
0.56
2.23
10.71
Softmax
0.82240.003
23.45:00.33
21:00:0000
56,00:11 49
5.22:5022
2.980.17
11:85:0.33
AES
0.823++0000
15.74
13.39=11.13
54,29+3.51
4.39
3.10
11.10=00.24
Medrop
0.821::0.003
21,21
18.83-22.55
56.69:1:.04
4.61
2.68:0.17
11.49:1027
500
Alcatoric+MC
0.821=00003
21.3242.18
18.9642.12
46.41=1442
1.09
3.20
11.73+00.6
CRL
0.822:10.006
14.04:1005
11.44:1.00
$5.46:1-2.34
2.33 2
10.64:10.116
Ours
0.82310.004
11.85 9.42
54.14.1.97
2.48 1..24
2.08 10.44.10.00
When it comes to confidence calibration, our method has superior performances under various sizes of labeled samples.
In Fig. the qualitative results ISIC2017 showed, and we can observe that: 1) our method performs better for segmentation accuracy, 2) the maximum softmax out- put optimized on cross-entropy only is overconfident near
16-
10
the boundary of lesion region, 3) the confidence map gen- erated by our method is more reasonable and consistent with training consistency.
AURC E-AURC
Ablation study for X2. We also discuss the effect of weight changes on consistency ranking loss. As shown
0.
0.15
02
0.5
0.5
in Fig. 3 the perturbation of weight X2 has limited in- fluence on confidence estimation performances, and our choice (A2 0.15) achieves the best outcome.
A
Figure Ablation study for dz-
With/without correctness supervision. full use of limited training labels. Here conduct ablation study to the importance of the correctness ranking loss. According to Tab. 3 w/o corr, removing correctness ranking loss hurts the performance both classification and segmentation tasks.
used correctness ranking loss in our method to make
Normalization strategy. In this paper, we do min-max normalization for training consistency on labeled samples and unlabeled samples in mini-batch separately. Here try concatenate training consistency on labeled and unlabeled samples first then normalize mini-batch consistency together alternative solution. The results are shown in Tab. 3 Unified normalization. We can see all evaluation metrics deteriorate significantly with the alternative normalization strategy.
Point-wise loss. Here use ranking loss optimize confidence output, an alternative solution is to use a point-wise loss like LI loss. However, due to training consistency is not of the same scale as the probability-basee uncertainty, LI loss cannot achieve good performance (Tab.
4.3 ACTIVE LEARNING
In practice, there scenarios where unlabeled samples cheap acquire but manual annotations are costly. To solve this problem, a promising direction is active learning, which requires learning algorithm actively queries experts for annotations. The critical issue here is how algorithm can
8

---

Published as a conference paper at ICLR 2023
Ori.
(b) GT
(c) Consi.
(d) Soft.
Sconfi.
(f) Ours
(g) Confi
Figure 4: Qualitative results on 250 labeled training samples compared to baselines. From left to
right: (a) Original image, (b) GT label, (c) GT consistency map, (d) Segmentation map of cross en-
tropy, (e) Maximum softmax of cross entropy (Softmax), (f) Segmentation map of proposed method, (g) Confidence map of the proposed method.
decide which unlabeled samples are valuable for the next stage learning. A common opinion that the confidence levels of unlabeled samples denote the value for learning process. Therefore, in this section, we adopt the strategy, which queries the samples with least model certainty.
Implementation details. We evaluate active learning performance CIFAR-10 10 and CIFAR-100. ResNet18 architecture is used for all models. For each stage, we train our model by SGD for 200 epochs with initial learning rate of 0.1, which shrinks factor of 10 120 and 160 epochs. The rest hyper-parameters for our method are the same as Sec, 4.
Here we compare our method with
CRL, MC-dropout, core-set selection (Sener Savarese 2018 as confidence output) and random sampling.
cross-entropy loss (using entropy
In the first training stage, 1000 labeled training samples and 49000 unlabeled samples are fed into training process. After
07
training. 1000 samples from the unlabeled set with least model confidence are selected and an- notated. We iteratively train and annotate for the next 9 stages. All 10 stages construct com- plete training procedure. To make sure the experimental re- sults are repeatable, we repeat each experiment for times. The results are shown in Fig. [ We can see our method performs better in most stages for both CIFAR-10 and CIFAR-100. The reason is that our method can utilize the confidence information contained unlabeled training samples by inspecting training consis- tency. This advantage is more apparent CIFAR-100 because the shortage of training samples for each class is more severe for CIFAR-100. Through active learning experiment, we can conclude that the samples selected by confidence estimates generated by method are effective for the following active learning stages.
Accurscy
Ours
Accurcyy
3.5
MC-dropaut
Coreset
6.5
Entropy
Random
6.5
Query Stage
Query Stage
(a) CIFAR-10
(b) CIFAR-100
Figure 5: Active learning results comparison on CIFAR-10 and CIFAR-100. Curves are the mean accuracy of five iterations, shaded regions are standard deviations.
5 CONCLUSION
In this paper, we study the validation of training consistency in estimating the model confidence of unlabeled samples. Both t-SNE visualization and AURC (E-AURC) evaluation suggest that generates reliable confidence estimates for both labeled and unlabeled samples with minor labeled samples. We propose consistency ranking loss make the model confidence output be consistent with training consistency on labeled and unlabeled samples. We demonstrate the efficacy of con- sistency ranking loss by mathematical proof and empirical results. The superiority of our method on image classification and medical image segmentation tasks suggest wide application prospect of our method. The confidence estimates generated by our method also work for querying high quality training images and achieve better performance than other basclines in active learning.

---

Published as a conference paper at ICLR 2023
Reproducibility Statement: We provide the necessary experimental details in Sec. More specif- ically, the implementation details of image classification tasks are provided in Sec. Sec. 4.2
contains the implementation details for image segmentation task. The implementation details of active learning tasks are provided in Sec. 4.3 The details of baseline methods are described in Ap- pend. [A.]] The details of the pairing strategy provided in Append. A.2] The details of evaluation metrics AURC and E-AURC are included in Sec. 3.2
Acknowledgement The authors thank anonymous reviewers for their constructive feedback. This research was partially supported by NSF CCF-2144901.
REFERENCES
Glenn W Brier et al. Verification of forecasts expressed in terms of probability. Monthly weather review, 1950.
Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmen- tation with cross pseudo supervision. In CVPR, 2021.
Noel CF Codella, David Gutman, M Emre Celebi. Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion anal- ysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In ISBI, 2018.
Liuhui Ding, Dachuan Li, Bowen Liu, Wenxing I Lan, Bing Bai, Qi Hao, Weipeng Cao, and Ke Pei. Capture uncertainties in deep neural networks for safe operation of autonomous driving vehicles. In CSSA////CloodSSoccaCCeeeeee Con.essaaooomSttstiinnCooooo 2021.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In ICMI., 2016.
Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan Ö Ark, Larry S Davis, and Tomas Pfister. Consistency- based semi-supervised active learning: Towards minimizing labeling cost. In ECCV, 2020.
Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks, NeurIPS, 2017.
Yonatan Geifman, Guy Uziel, and Ran EI- Yani..... Bias-reduced uncertainty estimation for deep neural classifiers. In ICLR, 2018.
Alex Graves. Practical variational inference for neural networks. NeurIPS, 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Siyu Huang, Tianyang Wang, Haoyi Xiong, Jun Huan, and Dejing Dou. Semi-supervised active learning with temporal output discrepancy. In ICCV, 2021.
Alain Jungo and Mauricio Reyes. Assessing reliability and challenges of uncertainty estimations for medical image segmentation. In MICCAI. 2019.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? NeurIPS, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Aviral Kumar, Sunita Sarawagi. and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In ICML., 2018.
10

---

Published as a conference paper at ICLR 2023
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NewrIPS, 2017.
Max-Heinrich Laves, Sontje Ihler, Tobias Ortmaier, and Lüder Kahrs. Quantifying the uncer- tainty of deep learning-based computer-aided diagnosis for patient safety. Current Directions in Biomedical Engineering, 2019.
Haohan Li and Zhaozheng Yin. Attention, suggestion and annotation: a deep active learning frame- work for biomedical image segmentation. In MICCAI, 2020.
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. survey on deep learning in medical image analysis. Medical image analysis, 2017.
Sheng Liu, Aakash Kaku, Weicheng Zhu, Matan Leibovich, Sreyas Mohan, Boyang Yu, Laure Zanna, Narges Razavian, and Carlos Fernandez-Granda. Deep probability estimation. In ICML, 2022.
David JC MacKay, A practical bayesian framework for backpropagation networks. Neural compu- fation, 1992.
Amit Mandelbaum and Daphna Weinshall. Distance-based confidence score for neural network classifiers. arXiv preprint 17Xiv:1700.09844, 2017.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. TPAMI, 2018.
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural networks. In ICML, 2020.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In AAAI, 2015.
Radford M Neal. Bayesian leaning for neural networks, 1996.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. CVPR, 2015.
John Platt. 5 probabilities for machines. Advances in Large Margin Classifiers, 2000.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI. Springer, 2015.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set
approach. In ICLR, 2018
Yawar Siddiqui, Julien Valentin, and Matthias Nießner. Viewal: Active learning with viewpoint entropy for semantic segmentation. In CVPR, 2020.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurIPS, 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- consis- tency targets improve semi-supervised deep learning results. NeurIPS, 2017.
Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Lc. Semi-supervised semantic segmentation using unreliable pseudo-labels. In CVPR, 2022.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. NeurIPS, 2020.
11

---

Published as a conference paper at ICLR 2023
Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang. Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-end sumi-vueee object detection with soft teacher. In ICCV, 2021.
Donggeun Yoo and In So Kweon. Learning loss for active learning, In CVPR, 2019.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass proba- bility estimates. KDD, 2002.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S4I: Self-supervised semi- supervised learning. In ICCV, 2019.
Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In ICML, 2020.
12

---

Published as a conference paper at ICLR 2023
A
APPENDIX
Append.
A.
shows the implementation details of confidence estimation baselines.
Append.
shows the details of mini-batch data pairing strategy.
Append.
shows a detailed proof of Thm.
Append.
A.4
shows the full confidence estimation results ISIC2017
Append.
A.5
shows the full confidence estimation results on CIFAR-IO and CIFAR-100
Append.
A.6
shows the pseudo code of training model using our method.
Append.
A.7
7
shows the anomaly detection results.
Append.
A.8
shows the other possible ways for extracting consistency information.
Append.
A.9
shows Cancer Survival dataset results.
A.1
IMPLEMENTATION DETAILS OF CONFIDENCE ESTIMATION BASELINES
For image classification, to make sure all baselines are fully converged, We train baseline models for 2700, 1200, 600, 300 epochs on 1/20 (2500), I/10 (5000), 1/5 (10000), (50000) labeled training images. The mini-batch size is 128. The initial learning rate is 0.1 and shrinks factor of 10 at 1/2 x number of epochs and 5/6 x number of epochs.
train all baseline models for 1500, 700, 300, 200 epochs on 1/16 (125). 1/8 (250), 1/4 (500). (2000) labeled training images. The learning rate 0.0001 all the time. The mini-batch size of all baselines is 64.
As for medical image segmentation (ISIC2017),
A.2 DETAILS OF MINI-BATCH DATA PAIRING STRATEGY
The mini-batch data are constructed by labeled samples D. = {1 W) and unlabeled samples Du = ((x,, v/) )-= We define TM = To, F[1] = Ti+1, i = 1,..., b - and æ" = z. ( =
14 = =1 1,..., d. In practice, we use empirical consistency ranking loss, which is defined by
Lrcn = Σ sign(c, (e) - - -(is -mp)) Σ sign(c. - c(2 - o%) (N. ki(a))) 1=1
where Cs = c(x.); (ts = c(a)))); 1ox m(xss, mgp) = m(mp))) oy = cdx")) c/j] d(olg))
k(x%), Mi(]] = k(x))))
A.3 DETAILED PROOF
Given Rx. we define ranking set H. = (xx v. arg max [(x), (ar, y) E Dm To make sure the duplicate value in C., won't it affect proof result, we stipulate that beside the rule above training consistency ranking function R. also need to minimize
Σ [R,((=) - (Rxx
min HE.
H.) TET
where r is the collection of the bijective function between the elements of H3 and H Satisfying two conditions above, we can define R. as an one-to-one function. Given Re, we introduce a set De
[e(R (i) c(R- R-R11 i = Tm [R,(2) [z),) = 7m (RR (z)) ))+ + sign (z) 7' - (x)))) x e
T. R. () & T(f(D,) mn)m }} where 7m arg min Eper (R, (x)( sign is
sign function and c(a;)) = Cs T3 E Xm.
To answer this question, we prove that consistency training loss is an upper bound of the absolute difference between the F-AURC of training consistency and confidence estimator on given dataset Dm, (Xm, Ym) and training consistency C.n
|AURC[, ((D.) AD
Hz (Dm) m min D. Leons (f, Xm, Cm)
13

---

Published as a conference paper at ICLR 2023
Proof. Let (1) be the set constituted by all the correct predicted samples with confidence less or
equal than R, (t), Tx = {2 : (x)x) S 9 = y, ÿ = arg my f()). (2, E Dm Ti( is the corresponding set introduced by training consistency e and T.(i) = 1z : Ra(x) .ig
U. = arg my [(r); (x. E Dm We define Ho(S, ki ())m) = >i= (#Tx(()) # #T.(1))
and (D)m) = E (()Tx(i)) + #T.())). It can be proved that
AAURC (Pm) - (URCC,,((DM.) - (D(
(8)
There is a fact that Vi E (1,2,...,,m) if (1#Tx(1) = #To(i)), than F(f.9R.'() (m)lm)) = P(f.RR: (1) (Dm), so (AURC(F,DDm) ( 1
Σ
|ff.9n=' (Dm) - f(f, 9 (Dm)
m #T H((s,
m
We also have ()(
II (g) S i) mj), where új
arg ey f(zz); and (z,, Ub) Dm- Here we define ((f, c(pm) #T(ffDm) clmm which is the number of correct predicted samples with lower training consistency than at least one incorrect sample.
In the next part, we will prove that:
1 m. min De Leons (f, Xm; Cm) Ho(f.n(Dm) We first discuss the case when K ((S,clmmi ples should have higher training consistency than the misclassified. Let C. be the set con- structed by the consistency event count of correct classified samples, C, = {c(x) 5 = y, 9 = arg max f(x);, (x,y) E } and Cy be the set constructed by the consistency event count of false classified samples, C5 = {c(x) : á + y. = arg myx,yy f(x),, (x, y) E D. It is not difficult to tell that min De min C. max Cyl under the case [ff. (Dm) O. Because De = {min C. - max Cy} when K(f,c(,,,, = O.
m
O. This means that the correct classified sam-
Here have:
1
(1c-ccl (Ki - k2))+
min C, max Cj1 feons XmX.mICm)
min C. max Cjl
1=
m- 1
= 110 vi) = 00 Kg no < (x)) 2 He(s, k(D...)
The details of above formula please take look at Theorem. We should notice that the conclusion above does not rely on the data size of Dm- Here we define the ranking function RT(. k ((,.)) on T((lDm). Rr(x, xi Dm) E ?1,2,.... Rr(2, (m)m)) < Rr(z,, K (D)), VI,, E (m))
................................ 4T.....4f1..... ( }), Vx E T()(mm) and if Rx((x)) < (x(x,) than
We assume the result stands when (pm) discussion with the situation " (Dm) (Dm) RTT (#T(flPm) c(m) = I.. Accord- ing to assumption here, we can get that
Than when K(f,c =n+ 1, start the
Σ iELECE
Σ
1(,, v. # x.) (Ics cil - (Ks ki))+ He (x.1) min D.
There is fact that H. Hiin ( (s.) (xx)) (f, k\Dm// (xx)) }) H minl (e.. (D.)) Since
Ho xl m//m///))))))))))))))(()) > H.min (n.). (m.). (m-))) (f. x/| m//Dm// {x+}) We have
m-
(Ico c:l (Ne kz))+ > (..in(B.(s..)...(....)) (f, (fx (m))) min D.
Σ
I 4 xz]
14

---

Published as a conference paper at ICLR 2023
We begin below part with the scenarios (xx)) 2 Re(xx), than it is straight that there must
ist (x,) - Re(x)) samples from Fmf constructing a set V = {o E Xm Rx (z) <
(x, (2) > (x,x...
F(f(Pm)) }. It is easy see that Ym (Rn, (z.)) = Rx( and
sign( R.(TT.) - 7mmm Rxx (x., })) = 1. So Vor E V, [c(x.) - c(>) 2 lc(x.) - cl R (Re(F.) + 1))l E
De. Therefore, can get that
(c. 0/ - (1i. ka))+
(lc, - cal - (n. kz))+ 2 (xx - Re(x..)
min De
min De
B.V
As for Rx( < Re( the proof is in the same manner. Summary the result above we get:
min De Xcons(f:XmlCC
1
Σ
Σ
1 I. # x.} (1os x Gi| - (Kis - kii)+
min De
s=1
(Iex- - cal -16a (15x-Ks))e ki))+
Σ
min Do
La
2 Tymiu(Ra(as)).ee (I, (f, k: Dm)) + (R.( - Re(x))) Hc(
If (FF(((( = Iı, (#T()))))) = 22 and # * x2, we can have one new confidence estimator by switching the confidence value between 21 and 22:
(k(21)
x x= 32
(x) =
x= I1 k(()) otherwise
16(22)
(10)
is not difficult to see that consistency ranking loss introduced by K4: feons xm (Dm) (Dm) satisfying
Lcons (J, X...ml/a 2 (
To see this fact, we pay attention to single sample. First, we discuss sample x, satisfying c(r,) < c(x2). we can define a =
- k(xx)) by = - - (M(x1) - x(xx)),
a = c(12))
c(x.)= - (x2 - kx.))) c(22) - c(a.) (n(x)) k(xx)). bt = c(21)
c(x.) - (Na(11) - kx(x))) = c(x)) - c(x.) (M(x2) - x(x.)). Apparently, by > ar bi > be de > ai;be > at and at + b (0,+ = 0. Here we define function g(x) = (0, x). It is straight that g(a:) + +g(b)) 2 g(a,) + g(be). We discuss this by different situations: 1) if by, we have
< 0, then g(a)) + +g(b))
((b)) = 0: 2) if a I O, b O, since
g(ai) + g(bb)) = =g(b)) = bi > b. =((b)) =g(a,) + (bb,); 3) if b,, a,, b, > 0, a: < O, then
g(ar) + (bi) 2 ai + b 2 a + b, = -g(a))+(()))) As for other cases, they are easy to get.
Here we have ((x.) = g(ax) + g(bi) and 4(x.) = g(as) + g(be), so ((x.) (xx)) 2 0.
As for c(z1) 2 c(xx) 2 c(x2), we have a; = -(a) -<(=:)=((()))))))))))))))))))))) = c(x.) c(z2) (N((..) x(I2)), a. = c(x1) - c(x.)
((z(()))((=z)))))))))))))))))))) = c(z.) c(z2) (k(z,) - r((x))
Apparently a; 2 a, and by 2 by, so we have g(a)) + g(h)) 2 g(ar) + (bb)). Also ((x.) 2 le(x)).
As for c(xx)= > c(z1), the proof is the same as the situation c(z.)) < c(x2).
If x. = 21, than 1(x,)= = c(x1) c(r2) - (m((2)) - n(x2)) and 4e (x.) = c(x1) - c(x2)) (M((12) k(z1)). is straight that 1(x.) 2 (xx).
Summary the result above, we can know that Ceons Xm (C)) Crons (f, Xm (m)) Exex 1(x) (xx) 2 0
So the proof stands for case ((),)Dm)) n + 1. under the assumption it is true while K(f,cDmm)= = n.
Let T, be the set constituted by all the correct predicted samples, Te = fi 1(0. v) = 0, 0 =
arg max f()), (x. (x.y) E Dm Trank (R)(t) t E T,} is the corresponding ranking
set. We also define F, to be the wrong predicted set, which is F, = fo 1(). v) = 1. =
arg max (xx), (x. E D and Frank (Rx(2) : d E F.).
15

---

Published as a conference paper at ICLR 2023
Theorem 2. Given dataset with samples Dm. training consistency Cm. confidence estimator and classifier f, we have
Holf.n|Dm) m
1
ml min C.- max Cjl froms(J, Xmi Cm)
Proof. In this proof, we don't consider the situation when f is an optimal classifier or k is a perfect confidence estimator (correctly predicted samples have higher confidence that the wrong ones). Because for those two cases, The E-AURC difference will always be O. Here we de- fine (. = min Frank. (T x((mm) = max T rank- It is straight that Ho x: (m,) = O.
[, x - (x(5, when K (f,, c
First we show that I, Xm. if K, > Ks. G4 < Ca. U0ss Uo) = 1 and Uir, y1) 0. than Ic,- cil 2 (min C, max Cÿ). is apparent that C, E Cj and ai E Ct, so we have 2 min Cj and C, S max C.. This inequation thus stands. So far we have
min Ci max Cy Loons (S, (Lf..m/CCn Xm)Cm)
Ex Σn Eizlaxse
min Ci max Cal
Ex Li=Lase, 1((().: Ua)
1n 1(0,, v)) 0n n4
Kg) [lea (le--ail (15- +Ri)l+
M
min max Csl
Σ El=1 1((().. Uo) = 1 n l0t y4) 0n > Kg neg ce) = I(f, n. Xm (Pm)
Next, we prove that for any confidence score function K. there is a fact that
I(f, K. Xmlmmmm = I(f, (m/mm) 2 (((m, (((
= 0}. If #Km 0, than Here we define Km fx E Xm (x) mD))
Va,, Ty E Xm. 1(36, v.) = L, 1, B1) O, we have RR(I,) < Rx(x)). This property makes under this condition an optimal confidence estimator and T, K (mm) < ()))) It is straight that Im, (m))m))
) 2 0. Therefore, we have I(f,Xm (Dm) 2 Tr(f,n((m)) (((m)
If #Km = 1, we have Km
{x.), (I,, E Dm and Gm fo Xm : (x(x) = 1}. Since 1( y-) = 0, it is clear that (xx))) I x.(f.n(Dm)) But if (x..).) < (T (D)),, we can get It = RT (T,(( D).)).. Apparently 3t E T, and (xx > Rx(x.)) > (=( so v, E Km. which conflicts the fact that # Km = 1. We have (xx)) =Tr(f, m.)) According to our assumption about training consistency, is apparent that VI& E (0m:(#44 (e)) E Dm, 1(0e, U-) = O, (i# V#) = =1, x(r_) < <(*#) and C(x,) > C(I ) where U. arg m f(x.); and ني = arg max f(x Here we also have #Gm, m T,(f. mm) F, k. ( To prove this, we assume #G < [f.(fx|Dmm) - Fr(f. (m))
Tr ki (m).mm..ii....
We have if x E Gm. than R (x) 2 (((m) because F, x(Dm is the minimum value of Frank. Therefore, if #Gm < T, (J.m))() FF(J,s(Dmm) x(( < Ry (xx)= < (J, k(Dm) and xx E Ts, which makes Fy E Km. Apparently Iy and I, are two different elements of Km, which conflicts the fact that # Km 1. This is simple fact that (Uf.XDDrn Pm) 2 ExS EG.. 1(x, E T a E F P K(x,) < x(xx)) n C(x.) > C(*+)) 2 Exwea,.. 1 TT
Fr(S. k(D..), there must exist a E Xm such that
In the context below, we show that for every & > 1, if #Km k this inequation holds, than when #Km = k + 1, this inequation also holds. Otherwise, we assume this inequation holds for any confidence estimator satisfying #Km k. but there exists confidence estimator Ky such that I(f, X. Xm (Pm) < T,(fRR (D (y K fi Xm Rx > F,(f,xy|Pm) (00 v) = 0} and #Km + Here we can have a confidence estimator
k (F(F,
x = arg min SeK"! Rx, (x) x RT (F(, otherwise
<x(x)
(arg min mek"! Ra, (x))
(z)
It is apparent that we get 1s, by swapping the confidence score of Ky at arg min zEK"! Rug (x) and
and I[f, K4, Xm (Dm) 2 Hy F, K2 ). It is also straight to see that (f (mm) = (, K6 (Dm), F, (yl, =
so #K%
[ff,
Fr Fr (e (Dm) 1. We also get that T(J, Kg, Xm (D..) T(J, K2, X D..)) 2 1, because
16

---

Published as a conference paper at ICLR 2023
Table 5: Comparison of confidence estimates on ISIC2017.
The value setting is the same as Tab.
Dataset (labeled size)
Method
mIOut
AURCL
E-AURCL
FPR-95L
ECEL
NLL.
Brier
ISIC2017
Sottmax
0.801
34.36:27.76
31.33:7992
60:90:2.111
6,55:0.30
3.82:0..6
13.90:00221
AES
0.80240.006
21.12=11.10
1800=1001
57.26:1-4.55
5.46:0.37
4.1==0222
12.84-0.53
Medrup
0.801
30.23:1.3.81
27.19:1.755
61.4::1:25
6,3610.30
4.05:0.33
13.88:10.49
125
Aleatoric+MC
0.802-4:.001
27.66:1:1.87
24.62.1:.93
59.55-10.84
6,21
1.6610299
13.68:40.13
CRL
0.810=00.004
22.14::2..2
19.33=2281
62.15-22.75
4,11
2.86=0334
12.25++0:42
Ours
0.812:40.007
14.11=:1.06
11.23::0.86
56.81=22.79
3.05=0049
2.31=0.23
11.46440.58
Softmax
0.81920.0012 28.91=:4700
26.45::772
58.5
5.62::0.0
3.3520.21
12.45:10.18
AES
0.819110.002
18.08.1...88
15.60.1
54.09.12.26
4.57.00.11
1.41 1:0.9
11.38.100.16
Medrop
0.819400.008
25.38:444776 22.89:44.57
58.31:11499
5.540040
3.39
12.27:40..7
250
Aleatoric+MC
0.817=0.0055
25,9 3.776 23.37=3.80
57,80=1.77
5,49=00333
3.43=0.29
12.43++0.0
CRL
0.818:0.002
16.43-4:0994 13:90=:0.90
62.56:19.11
3.93+#0.11
2.60=0.25
11.64:+0.82
Ours
0.007.1.0007 12.79 0.779 10.21:10.62
$4.51
2.56
0.26
2.23
10.71:10.31
Sottmax
0.822+0.003 23:45::0:93
21.09:0.90
$6.90:11.49
5.22::.222
2.98=00.17
11.8820.33
AES
0.823-40.066 15.74=t1.118
13.39- 1.13
54.29:43.51
4.39-+0.15
3.10=0.14
11.10++0:24
Modrop
0.801=0.000
21.21
18.83:22.55
56.69=1104
4.61=
2.68=0.17
11.49+00.77
Aleatoric+MC
0.0003
21.32:42.18
18.96:12.12
56,41=1142
5.07::0.19
3.20==0:18
11.73:::0.66
CRL
0.822.10.006 14.04.11.03
11.64:11000
55.46.12.30
2.82.10.25
2.33
10.64:10.16
Ours
0.823:10.004 11.85:11177
9.42=11133
14.14:11.97
2.48-10.24
2.08=00118
10.44:10.20
Softmax
1.000 16.19110.61
14.29.10.61
53.69
4.67.0..1
10.94.10:25
AES
0.829=00.004
12.54=:0.61
10,40=:0.60
51.02+23.22
3.98:10.13
3.09=0.07
10.40:1:014
full:2000
Aleatoric+MC
Medrop
0.834110.005 15.41
0.828:10.010 14.81:1:.05
13.40.10.89
126661100
54.17110.55
54.92.10.49
4.57-10.22
4.51
2.86.10.88
1.04-0.21
10.94:10.58
10.69.10.33
CRL
0.83420.001
10.17=00.66
8.08-0.46
518833318
2.61
2.02-0.077
9.76=0021
Ours
0.838=0.007
11.18:11.99
9.17
52.27+3.14 2.23:0433
189
9.49=0.43
Xm (im) Ilf, Pm)) EEFURNIDD.)RRR.CCC (x) min sEW"! Re, 1(x
F,21 (F(, E T,K(z) n(z1), C(zi)) 2 1. So far according to our assumption we have
T(,K3, Xm (m)) < ((,, FA(S, x (Dm)
I(S Kr, Xm (m) )m)++ + < TT (m)) - FF(S, Kz (Dm)
T(J, K4, X (mm) +1 < T, [2 Fr +1
T(J, Kit, Xm (mm) < (T (Dm) Fr(S, Kg (m))
Which leads to a conflict.
A.4
ISIC2017 CONFIDENCE ESTIMATION RESULTS IN TAB.
A.5
CIFAR-10 AND CIFAR-100 100 CONFIDENCE ESTIMATION RESULTS IN TAB.
A.6
PSEUDO CODE
Here provide the pseudo code for the training process of consistency ranking loss. Here we need to notice that we normalize the training consistency for labeled samples and unlabeled samples mini-batch independently. The reason that training consistency has bias on labeled training samples, so the labeled samples have higher training consistency than unlabeled ones. There is mismatch between the training consistency distribution of labeled samples and unlabeled samples. A simple way to overcome this issue is Min-max normalize the consistency of labeled and unlabeled samples separately. practice, this will achieve better performance.
A.7 ANOMALY DETECTION
We evaluate the anomaly detection ability our method on CIFARIO and use the samples of CI- FAR 100 oui-of-distribution samples. In this part, use the 10% (5000) labeled samples setting. The results are shown in the table below, and our method achieves best performance. This indicates that the deep classifier trained by our method can detect duitt--distribution samples efficiently. For the training details please refer to Sec. 4. We compare our method with Cross-entropy loss and detection error (the minimum error among all thresh-
the correctness ranking loss. Here
17

---

Published as a conference paper at ICLR 2023
Table 6: Comparison of confidence estimates on CIFAR-10/100 with various labeled training data size. The best results of experiments are shown in bold. The values of AURC & E-AURC are multiplied by 10 FPR are multiplied by 102 and NLL is multiplied by 10 for clarity
Dataset (labeled size)
Method
Acct
AURCL
E-UURC
FPR-95
ECE
NLL
Brier]
CIFRRID
Soltmax
0.717=00008
109,7816.69
65.32-33.22
75.6440.59
20.23-0.766
14.8340.00
47,1741.555
AES
0.713:10.002
108.9410.87
63.34.1..0 1.02
74.31
0..82.0.47
14.46.1..3
44.36.10.32
CIFAR (2500)
Mcdrop
0.700=000088
122.45:6.37
72.42.1.89
1.44
16.76:0.59
16.4910.53
16.18:1.19
Aleatoric+MC
0.704:10.022
11.076:115.06
70.66:17.69
75.29:1..355
1.55
16.30:11.23
45.62.1.449
0.718-10.003
105.92:4:.27
61.95:+1.27
74.17110.80
0..1540062
10.63:10.19
42.20:1...55
Ours
0.755=0.004
81.50=22.70
4..72=1.755
71.42:+
5.77
8.56=+0.24
38.24+0.53
Softmax
0.795
60.54:11.45
18.02.1
68.64.1 1.08
04.73...20
11.26.10.15
34.28.10.7
AES
0.799=00.002
51 1.00
29.99=0.880
60.44::0.90
8.16::0.19
9.46200.09
28.51+0.21
CIFARIO (5000)
Medrop
0.808=00.003
52.94=1.077
33.25=00:47
63.70=
8.80
10.41=0020
29.01=0043
Aleatoric+Mo
0.810:20.001
10.96=+1.47
11.73+1..6
67.46:1
8.67:10.24
10.22:40.16
28.69-10.30
CRL
0.807=0,002
51.76+1991
11.82
13.
0,08
7.16=0.17
29.21+0.44
Ours
0.818=0.002
44,43=1003
26.88=0.66
64.64-10.93
5.26-+0.43
6.42
04.88--443
Softmax
0.849=0.001
T..46+1.099
03.44=00.3
63.44 88
0100300133
0.09
26
AES
0.855:00.004
30.75=+0.89
19.71=0.55
0.70
6.61
0.23
2..6
CIFARIO (1000D)
Medrop
0.865
1.
19.09-10.58
61.39.1..66
5.68
1.22
20.37.10.46
Alestoric-MC
0.000
28.64:1.0.70
19.22:10.65
23
5.75
0.4
20.47
CRL
0.856:20.011
31.28:10.32
20.51::0.16
61.63-+
5.71-:0.21
0.08
21.71110.16
Ours
0.860
28.39.10.54
0..19
59.23.1 1.96
3,991.0.17
1
20.77.10.18
Sotlmax
0.901=00002
9.11=0..4
0.34
2..42+2..30
4.46:20.16
0.34
10.19:10.32
0.942-10.002
0.0.5
36.37
1:020
1.02
8.6910..9
CIFARIO
Medrop
0.942=20.000
5.18
0.50
16.74
1.45
0.88
0.
(Full:
Alcatorio-MC
0.943=00000
6.02-0.33
0.38
38.72 1.82
1.25+0.07
180=0.03
8.36--0.12
50000)
CRI.
0.940:10.001
6.02-0.26
1.21
18.81
1.23:10.18
1.11
0.85::0:20
Ours
0.942:20.001
5.83=0025
4.16:20.16
10.09:11..37
0.86:20.07
1.76
8.60
CIFARTOO
Softmax
0.292=00.006
706.70:14352
139.23:56.38
79.13
31.8911.3
38.25:00999
97.18
0.289:10.009
509.38:12.07
157.50:1:300
19.9311.338
00.
03.79
05.77-10.39
CIFARIO0 (2500)
Medrop
0.269110.005
$42.10.1..75
165.0661...00
79.95
0..4
49.71
108.54
1.07
Aleatoric+MC
0.269:10.005
542.86+10.72
165.564:3.95
80.96:
12.48
10881
CRL
0.287=0.004
507.43++5596
152.82+2222
79.12+
29.01=1.74
37.33+1.000
15,40=1..34
Ours
0.365=0.004
692.49-1..23
133.154378
0..36:10.82
07.55=0.35
33.5410.39
0.624
0.425
15.30915.31
132.9510.95
77.001..18
10.48
86.17.10.33
AES
0.419:000005
335.02.1486
118.75:00.87
03.30.10.33
22.89-10.69
34.04-10.63
77.93
CIFARI00 100
Mcdrop
0.394=00.044
4.24
251.06
77.04+
39.32-0.44
14.4341.19
94.73+0072
(5000)
Alcateric-MC
0.394=0.002
378
140,54+1448
07.48:0.00
39.12=0.49
44.19=00.95
94.67=00.73
CRL
0.407:20.003
322.94:13.17
122.50.10.77
75.28:11.05
29.51
50.10:14
82.09.1..54
Ours
0.482:10.005
266.08 15887
100.22:1..40
71.50 1.88
1..6
23.99.10.47
0.99
Softmax
0.546:0.004
214.1742.81
70.78:0:59
72.72:10.52
24:31
24.19:10.44
67.41
AES
0.546=0.002
309.77+3.62
86. 86.38+2.24
71.55=
19.63=0.20
24.29:00.46
63.62+0042
CIFARIOO (10D0D)
Medrop
0.523=00.044
238.32+55.79
100,70::3.33
02799+00772
0.75
31.73+11005
1.005
AC
0.501
240.
100.9411.005
13.15.1
00.54.0.55
31.79.11116
75.27
CRI.
0.563-10.005
196.1112.35
82.976:1095
70.37
20.31
29.20:10.19
63.02-10.58
Ours
0.590=0..003
168.57:42.99
70.26=0.89
68.30-+1.37
14.34=0.32
17.58:00.2
56.23-10.49
Sommax
0.753=0.002
71.75=0.89
38.63=0.72
63,30=:1.95
02.67=0.25
11.4420.08
37.26:00.31
AES
0.760:00.011
65.22:10.73
33.95:10.68
0..1
0,322
9.04
33,96:10.16
CIFARIO0
Medrop
0.758
66.92.11.45
34.97.10.46
63.27.1
5.59
9.49
34.02.1..88
50000)
(Pull:
MC
0.005
67 1.587
35.05-00.65
61.69
6.02
9.45=0.13
34.25+0477
CRL
0.768=20.002
6..77+1077
32.57=0881
61.79++2.200
8.59:0.17
9.11=00.99
33,39:10.28
0.780:00.02
55,40:10.90
29.36:1057
60067:1103
7.8710.18
8.45:0007
31.52.10.35
olds) and the area under the receiver operating characteristic curve (AUROC) evaluate the out-of-
distribution performances. All experiments are iterated 3 times and shown in Tab. 7 The mean and standard deviation are reported.
A.8 ADDITIONAL QUALITATIVE RESULTS
In this paper, we use consistency events to quantify the sensitive levels of samples vibrating around the decision boundary in the training process. It's worth mentioning that this is not the only way to capture such information. For example, we define label frequency r.¹ = ΣT 1 (ot = y} as a variable to count the number of sample z; being predicted as label y by classifier during training and it is also able to capture training consistency. We can use the maximum label frequency (L-count) max r. to quantify the training consistency sample 2;. Apparently, the higher maximum label frequency introduces less vacillation the training process, leading to higher confidence. Be- sides, can also use the label frequency entropy (L-entropy) and margin (L-margin) summarize
18

---

Published as a conference paper at ICLR 2023
Algorithm 1: Consistency ranking loss training
Input: Dataloader for labeled and labeled training samples Output: Trained deep model
Definition: wLaaeer and sLoader denote the dataloader for unlabeled and labeled samples; Deory is the dictionary, storing the count of correctness for each labeled sample. Doon is the dictionary, storing the count of consistency for each sample.
1: for epoch in range(number of epochs) do
2: siterator dierator(sloader
3: for (uninputs, umlabel-index) in uLooder do
4: if siterator iteration end then
4:
siterator = (lerator(sLoaders
4:
update Doorr, Dvon
4:
inputs, targets, label-index = nexter)
5: else
5:
inputs, targets, label-index = next(sLoader)
6:
end if
7:
corTiatel
// We use Min-max normalization here contobet Normalize (emmaliiz(DDooollallliiiiii // Extracting training consistency for batch
8:
labeled samples
9:
Conmataaee (DomaliizeDeeomuuniiii (Dom(uunlabell-indee)) // Extracting training consistency for batch
unlabeled samples
10:
conboteh = Concat Concat(coniasee (contabel, con
labeloutput model(inputs)
12:
unlabeloutput = mwwde (uninnuu)s
outputbatch (obocca((labllupppll unlabeloutput)
14:
LCE Lcx (labeloutpt, targets)
Loorr Loor poottmax(labeloutpui corrtabel) Leon Leon (softma
16:
combatch)
17:
Losstotal = LCE + λ Ccorr + XzLcon
18:
Update model weights
19:
end for
20:
end for
Table 7: Anomaly detection results on 10% (5000) labeled CIFARI0
Method
Accuracy!
AURO AUROCT
Detection
CI
Cross entrupy 0.797 00.002 0.75110..02
0.301:10.000
CRI. 0.807-10.003
0.762-10.002 0.818:10.003
0.293-10.001
0.786:t:0001
0.276:10.001
consistency information. We show that our consistency event has superior performance in estimating confidence in terms the criterion of ordinal ranking. As shown in Fig. compared with maximum softmax output, training consistency estimate has much better performance in distinguishing correct predictions from wrong ones on unlabeled samples. This indicates training consistency has the abil- ity to extract valid confidence information without the necessity of ground truth labels. Furthermore, we also find our consistency event captures the essential of training consistency a better manner, resulting in better performances across all shown epochs. Though Correctness also achieves good results, the demand of annotations restricts its application in scenarios with limited labeled data. For the of this paper, training consistency is especially referring our confidence estimate method.
A.9 RESULTS ON CANCER SURVIVAL DATASET
In Cancer Survival dataset Liu et al. (0022) histopathological features are used to predict the 5-year survival of lung cancer patients, which can be taken as classification task. This dataset has 1512 whole slide images (1203 for training, 151 for validation, 158 for testing), 352 of which died in
19

---

Published as a conference paper at ICLR 2023
0.075
Consistency
0.050
Correctness
Saftmax
0.065
L-count
0.040
L-margin
AURC
Ventropy
EAURC
0.035
0.055
Consistency Correctness Softmax
0.025
0.045
L-count
0.020
L-margin
Lentropy
0.035
120
220
120
140
160
220
epochs
epochs
Figure 6: Performance of consistency as a confidence estimator. We evaluate the performance of consistency, correctness, softmax, L-count, L-margin, L-entropy in terms of popular metries, AURC and E-AURC. We train a model with only 20% of CIFARIO labeled data, and evaluate on the remaining unlabeled data. Table 8: Results on Cancer Survival dataset (120 labeled sample)
Method
Acct AURCJ
E-AURC FPR-95 ECE,
NLL Brier]
Cancer Survival
0.616
418.50
7.35
MMCE (Kumar el al.
2018
0.625
436.54
355.43
89.88
8.86
6.74
48.01
CaPE(bin) Lau et
0.614
428.10
341.29
92.82
9.60
7.10
50.57
CaP'/keruel
2012
0.614
414.27
327.46
88.96
10.11
7.12
50.56
Deep ensemble Lakchmmaray anaann al 2017)
0.620
386.71
303.07
90.41
10.33
7.05
49.54
ETS Zhang
0.627
384,90
304.29
90.89
15.13
7.69
52.63
Mcdrop
0.628
351.36
271.38
91.51
10.66
6.80
48 18.40
CRL
Ours
0.627
343.73
263.20
91.52
6.43
6.68
47.42
years. For our experiment, we randomly choose 120 training samples as labeled and the rest training samples are unlabeled.
Results are shown in the Tab, Our method achieves very good performance on this Cancer Survival dataset. Most evaluation metrics of our method are best among compared methods. This is because our method can make use of the confidence information on unlabeled samples effectively.
20


Reviews:
[{"review_id": "VwHzV02z6Cl", "reviewer": "Reviewer_qWap", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach to estimate the confidence of the predictions using unlabeled data in the semi-supervised setup. The main idea is that training consistency, i.e., the frequency of training data getting the same prediction in sequential training epochs, can serve as a surrogate for the model’s prediction. \n", "summary_of_the_review": "The novelty and contribution of the paper are limited. In addition, assumptions made based on the exploratory evaluations are not accurate and require further justification. \n", "strength_and_weaknesses": "Strength: \n\n-The problem of overconfident predictions is important and requires major attention from the community. \n-The paper is well-written and easy to understand.  \n-Extensive experiments are conducted to evaluate the performance of the model. \n\n\n\nWeaknesses:\nMAJOR:\n-In Figure 1, the paper attempts to show that the training consistency correlates with the distance to the decision boundary and then claims that the distance to the decision boundary can be interpreted as confidence. This claim does not seem to be correct. Since the output space is not calibrated, the distance to the decision boundary in the output is also overconfident and cannot be used to represent confidence. We can see this more clearly by noting that the linear classifier that performs the classification on top of the representations is also estimating a relative distance to the decision boundary. Therefore, the overconfidence problem would not exist in the first place if the distances in the representation space were correlated with confidence. \n-Another point is that the distance in the output may correlate with confidence only on the samples that are close to the decision boundary as depicted in Figure 1. This is obvious because we also observe in conventional networks that low predicted probability shows low confidence. The challenge is actually on samples that are distant from the decision boundary. Therefore, the assumptions made based on the observation in Figure 1 are not accurate and only consider the samples close to the boundary while there is no clue or intuition for samples not close to the boundary.\n-The novelty is limited. The main idea is the simple adaptation of the correctness measure proposed by (Moon et al., 2020) to the semi-supervised training setup. Other choices for the adaptation could be considered to augment the contribution. \n-L_{corr} in Equation 3 is not defined.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TtbLzxph8i7", "reviewer": "Reviewer_ZHpU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a consistency ranking loss that enables the model's softmax output to be a close approximation of consistency. Consistency in turn correlates well with confidence estimation. In other words, if the model's prediction for a sample is consistent as the training progresses, it is likely to have high confidence in its prediction for that sample. Their method can leverage unlabeled data and thereby works well in limited label settings. They show that their method of estimating confidence outperforms other methods for image classification and image segmentation tasks. ", "summary_of_the_review": "The method proposed in the paper tackle many important issues like confidence estimation in semi-supervised settings which was not adequately addressed in the previous work. Additionally, the proposed method is sound, novel, and elegant. Therefore, my recommendation would be marginally above the acceptance threshold (6).", "strength_and_weaknesses": "Strength(s):\n1. Using consistency as a surrogate for the correctness and the model's confidence is an interesting and novel idea. The experiment demonstrating a high correlation between consistency and confidence was quite convincing.\n2.  The authors nicely transferred their learning about consistency being a decent surrogate to confidence into a loss function that enables softmax outputs to be as close to consistency as possible.\n3. Their method can leverage unlabeled data and thereby works well in limited label settings. This is again novel as prior works have mainly focussed on the supervised training setting.  \n4. Their experiment with active learning was again a clever way to demonstrate their method's superiority.\n\nWeakness(es)/Suggestion(s):\n1. It seems that results on CIFAR10 and CIFAR 100 should be supported by a real-world dataset(s) where we actually see some uncertainty in the outcomes like predicting the weather or predicting a future event from the current data.  Recent work in deep probability estimation actually addresses this issue and also proposes some datasets [1]. I feel the proposed method is an interesting one and therefore, should be tried on some real-world uncertainty prediction datasets. It would only strengthen the paper. \n2. Additionally, some existing methods like [1], [2], [3], and [4] for the model calibration should be added to the comparisons. This again would strengthen the paper. If you feel that the above methods would obviously not outperform the proposed method, the reasons for the same should be included in the relevant work section.\n3. For figure 3, some details like which dataset was used, how many training samples were labeled, etc. are missing. \n\n[1] Deep probability estimation: https://arxiv.org/pdf/2111.10734.pdf\n[2] Deep ensembles: https://arxiv.org/pdf/1612.01474.pdf\n[3] Mix and Match: https://arxiv.org/pdf/2003.07329.pdf\n[4] MMCE: https://proceedings.mlr.press/v80/kumar18a/kumar18a.pdf\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ARcN6_32LWO", "reviewer": "Reviewer_L49h", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose a new method for uncertainty estimation in image classification and segmentation considering a semisupervised setting. Their approach takes advantage of also unlabelled data, which is a novel aspect in uncertainty estimation. They propose so called consistency loss which forces softmax output to consistence (defined as consistency of predictions between training epochs). The proposed approach is validated numerically using CIFAR10&100, image segmentation (ISIC2017) and in a active learning experiment. ", "summary_of_the_review": "The approach seems novel and relatively simple, and numerical experiments also looks very promising. There are a few points that I would have liked to be see in their study (see \"Minor weaknesses / comments / questions\" above), but on the hand, I agree that not all can be included into this paper and those could also be part of future study etc. Therefore I would recommend acceptance. ", "strength_and_weaknesses": "Strengths:\n- They take advantage of unlabelled sampled. Having unlabelled samples is a common situation in practical problems and especially very natural situation in active learning problems. \n- Numerical results looks good: they reach state-of-art in most of experiments, often with a large margin\n- They also provide theoretical considerations supporting their approach\n\nWeaknesses:\n- It seems that the approach is only applicable with networks with softmax output (i.e. classification or segmentation), not clear if can be extended to regression etc?\n- It is not clear to me that what are actually the metrics used in active learning methods. I guess \"entropy\" results is computed from softmax output? \"MC-dropout\" is also based on entropy? Your method is based on softmax or entropy?\n\nMinor weaknesses / comments :\n- Baselines are based on quite old studies (the latest is from 2018). There are also several more recent approach. But on the hand, MCdropout is still one of best performing solutions and none are considering semisupervised setup, so this is also just a minor comment. \n- Furthermore, numerical results considers only a single set such that training and evolution sets are quite similar (i.e. I would say those are \"same distributions\"). Some confidence studies considers out-of-distribution detection, meaning that they study if method can detect a sample which is from a significantly different distribution (e.g. Dirty MNIST introduced by Mukhoti & Gal, https://paperswithcode.com/dataset/dirty-mnist). I would like to see also this aspects of their method. But can also be future study. \n- Another similar study could be if method can separate aleotoric (uncertainty due to noise etc) and epistemic uncertainty (uncertainty due to lack of training data etc). This is meaningful in active learning if different levels of aleatoric noise are present as it is pointless to pick samples with high aleotoric noise. See Yarin Gal's work. But can also be future study.\n\nQuestions:\n- Double sum consistency loss is expensive to compute as the authors mention. But can it be avoided? Does the loss sense make any sense without taking only difference of consistency and softmax output  (i.e. computing $\\sum_s\\max\\{0,c_s-\\kappa_s\\}$?\n- In training, does minibatch include fixed ratio of labelled and unlabelled samples or is this ratio random? If latter, is it a problem if no labelled samples end up to the mini batch?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8zEnXT_6gk6", "reviewer": "Reviewer_pST4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In a nutshell, authors extend confidence estimation in DNNs to semi-supervised learning. To my understanding, the way they have approached the problem is that the more consistent the prediction for a given data sample x_i across different stages of training (including weights) is, the more confident the model will be; and that is independent of y_i, i.e. consistency is the proxy. To some extent it reminds me of this work (https://link.springer.com/article/10.1007/s00521-019-04332-4) where the authors annotated unlabelled data based on uncertainty estimation. The results presented (and visualised, e.g. figure 1) are promising, but half of the paper (especially the first half) tries to mathematically formulate the problem unnecessarily. Even the theorem might be redundant, with the most relevant being eq. 2.", "summary_of_the_review": "Good paper, with some useful intuitions around expanding confidence in semi-supervised settings, but the paper lack proper organisation, useful ablations/experiments, and a more compelling provision of the importance of this work. It would require some moderate improvements to make it publishable, but it is nevertheless an important piece of work.", "strength_and_weaknesses": "Strengths:\n\na)\tA simple but effective method, that works competitively with respect to other methods.\nb)\tThe authors also show some results in medical image segmentation\nc)\tIn theory, it should generalise well in other datasets\n\nWeaknesses:\n\na)\tI think it is a weak formulation and some quantification of the bias of the estimator would be useful – to be honest, though appendix covers me but it is also not an excuse to use the appendix to write a longer paper\nb)\tI appreciate the use of medical images but results on more datasets would demonstrate the generalisation of the method, as CIFAR and ISIC2017 might be not the best ones for this task\nc)\tI think most of the formulas and maths in the first half of the paper, mainly pages 2 and 3 are redundant in my opinion. You want to explain something very simple and the maths just makes it more complicated than it is.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster