Failure Detection in Medical Image Classification: A Reality Check and Benchmarking Testbed
Authors: Mélanie Bernhardt
Fabio De Sousa Ribeiro
Ben Glocker
Venue: Accepted by TMLR ()

Abstract: Failure detection in automated image classification is a critical safeguard for clinical deployment. Detected failure cases can be referred to human assessment, ensuring patient safety in computer-aided clinical decision making. Despite its paramount importance, there is insufficient evidence about the ability of state-of-the-art confidence scoring methods to detect test-time failures of classification models in the context of medical imaging. This paper provides a reality check, establishing the performance of in-domain misclassification detection methods, benchmarking 9 widely used confidence scores on 6 medical imaging datasets with different imaging modalities, in multiclass and binary classification settings. Our experiments show that the problem of failure detection is far from being solved. We found that none of the benchmarked advanced methods proposed in the computer vision and machine learning literature can consistently outperform a simple softmax baseline,  demonstrating that improved out-of-distribution detection or model calibration do not necessarily translate to improved in-domain misclassification detection. Our developed testbed facilitates future work in this important area.

Published in Transactions on Machine Learning Research (10/2022)
Failure Detection in Medical Image Classification: A Reality Check and Benchmarking Testbed
Mélanie Bernhardt. Imperial College London, UK
bernhhhhrd:2 @kmperial. uk
Fabio De Sousa Ribciro Imperial College London, UK
f. Aidesoonssrrtbcccrooiimppprr...o
Ben Glocker
b. D.glocker@kippriill.ccc uk
Imperial College London, UK
Reviewed OpenReview: https: httpe://openrevieeu openreview. net/ forum? id= fnOu
Abstract
Failure detection in automated image classification is critical safeguard for clinical deploy- ment. Detected failure cases can be referred to human assessment, ensuring patient safety in computer-aided clinical decision making. Despite its paramount importance, there in- sufficient evidence about the ability state-of-the- test-time failures of classification models in the context. of medical imaging. This paper pro- vides rcality check, establishing the performance of in-domain misclassification detection methods, benchmarking widely used confidence scores on 6 medical imaging datasets with different imaging modalities, in multiclass and binary classification settings. Our experi- ments show that the problem of failure detection is far from being solved. We found that none of the benchmarked advanced methods proposed in the computer vision and machine learning literature can consistently outperform a simple softmax baseline, demonstrating that improved out-of-distribution detection or model calibration do not necessarily trans- late improved in-domain misclassification detection. Our developed testbed facilitates future work in this important area
confidence scoring methods to detect
1 Introduction
In recent years, several studies have shown the potential of AI systems for improving clinical workflows
[Esteva et al. 2017 Oktay et al. 2020 Faust et al.) 2022 Calisto et al.) 2022 Zhong et al. 2021
Calisto
et al. 2021) both in terms of workload reduction and improved patient outcomes. However, none these
Al-systems are perfect and errors will always be present (Shamshirband al. 2021 Challen et al. 2019
As such, it is of utmost importance to put in place safeguards prevent potential adverse consequences arising from those failure cases. To this end, some studies have focused how to best. optimize Al-human collaboration for maximizing patients outcomes via the introduction of interpretability tools, or by using the AI as sccond reader as opposed to having a stand-alone computational model (Calisto et al. 2022: 2021 Shamshirband et al. 2021 Calisto et al. 2017)- However, it has also been shown that such "human safeguards" are not necessarily fail proof since incorrect model predictions can mislead human readers, even experts (Tschandl et al.] 2020). On the other hand, some models are designed to be used as a stand- alone system for example automated triaging (Kim et al. 2022 Annarumma et al. 2019 Van De Leur al.] 2021 Leibig et al. 2022). In these cases, there is clear need for additional safeguards such automated failure detection to prevent the use of incorrect predictions and mitigate clinical risk. For clinical
"ode available http:////iiiii..ccaaaaaaaaaaaaaa benchhark

---

Published in Transactions on Machine Learning Research (10/2022)
Model prediction
Final decision
Test input
Softmax output
Classification model
"STANDARD" CLASSIFICATION SYSTEM
Confidence score
Final decision
JECT
Confidence scoring
If rejected: Human referral
Rejectioo threshold
Ground truth label
Assigned label
CLASSIFICATION SYSTEM WITH AUTOMATED FAILURE DETECTION
Figure 1: Standard classification system versus system with automated failure detection. In the latter, confidence score is computed for each lest sample in addition to the model prediction to determine whether the prediction should be accepted or referred for human annotation, based on a chosen rejection threshold.
deployment, reliable failure detection is critical for patient safety, enabling automatic referral to human experts for suspicious predictions (Kompa et al. 2021 Van De Leur et al. 2021 Leibig et al. 2022). For example, some triaging systems include uncertainty-basse automatic reject options (Van De Leur et al.
2021
Leibig et al. 2022). i.e. only cases with high prediction confidence are triaged by the AT system.
Despite their increasing use in clinical applications, there is insufficient evidence about the quality auto- mated in-domain failure detection in medical image classification models. This is an important and non- trivial problem, in most. deployment settings the majority of inputs are expected to be in-domain, and model is error-free, even in the absence of any input perturbation data corruptions. As depicted in fig. [ failure detection frameworks are typically divided in two stages: (i) confidence scoring (to quantify the likelihood of the prediction being correct); (ii) a thresholding-step (to reject/refer samples with a low confidence score) (Corbière et al. 2019 Jiang et al., 2018 Band et al. 2021). such a system, if the confidence scoring scheme improved, the misclassifiiction detection performance improves.
In the machine learning literature, numerous attempts have been made to obtain better confidence estimates for machine lcarning classifiers. However, most of the literature in this space has concentrated bench- marking and improving confidence estimates in terms of robustness to oul-of-distribution (OOD) inputs (Hendryoks & Gimpel 2016: Takshminarayanan et al. 2016 Gal & Chahramani 2016 Daxberger et al. 2021] Maddox et al.] 2019), model calibration (Lakssmminarayaaan et al.) 2016;; Maddox et al. 2019), or
datasot shift (Ovadia et al., 2019 Only few works have studied how useful these statc-of-the-art uncer- tainty estimates and other confidence scores are the context of in-domain failure detection. Recently, Band al. (2021), extending the work of Leibig et al. (2017), evaluated Bayesian uncertainty estimates the context of failure detection for in-distribution as well as in data-shifted settings. However, this study has Bayesian methods. Whereas the Bayesian community has indeed devoted lot of effort to finding better uncertainty
been limited to: (i) the binary diabetic retinopathy detection setting; (ii) focuses exclusively
estimates (Lakshminarayanan) et al. 2016 Gal & Ghahramani 2016 Laplace 1774 Maddox et al. 2019 others have also proposed confidence scores designed explicitly for failure detection, e.g. by analysing the internal representations from the network instead of focusing solely on the output layer dJiang et al. 2018 Corbière et al. 2019 Van Amersfoort et al.] (2020). There
need for reality check with a comprehensive comparison of confidence scores for failure detection in terms of methods (Bayesian and non- non-Bayesian) and, importantly, for diverse set of medical datasets. This is the purpose of this study.
Summary of contributions:
Our study presents novel extensive multi-dataset evaluation of in-domain failure detection methods in the context of medical imaging classification models. We benchmark different commonly-usee -usd uncertainty scores and misclassification detection scores diverse set of medical datasets, covering
2

---

Published in Transactions on Machine Learning Research (10/2022)
6 imaging modalities and resolutions ranging from 28x28 to 512x512 pixels, in both multi-class and binary classification settings;
Surprisingly, find that none of the benchmarked confidence scores were able to consistently outperform a simple softmax baseline for detecting failures across multiple tasks;
Our findings demonstrate that reliability against out-of-distribution inputs or improved model cali- bration does not necessarily translate to improved failure detection for in-domain inputs;
The developed failure detection testbed (which is made publicly available) aims to facilitate future work, building an essential component for rigorous, comprehensive testing and comparative evalua- tion of new approaches in the important area of misclassification detection in medical imaging.
2 Background
2.1 Failure Detection Methods
In this study, we focus on widely used uncertainty scoring schemes, these would be most likely to be employed in practice at. this time. Additionally, we cover diverse set of methods that believe are representative of the field, and they can be divided into 5 categories: (i) confidence scores based softmax outputs (softmax baseline Hendrycks & Gimpel (2016), DOCTOR Grancse et al. (2021 )): (ii) Bayesian
uncertainties (MC-dropout Gal & Ghahramani (2016), Laplace Laplace (1774), SWAG Maddox et al. (2019));
(iii) ensembles [Takshminarayanar et al., 2016); ; (iv) non-softmax based models (DUQ, Van Amersfoort et al.
(2020)); ); (v) confidence scores based on feature representations (TrustScore Jiang et al. (2018); ConfidNet
Corbière et al. (2019)). A more in depth discussion on the choices of basclines evaluate can be found in the discussion.
2.1.1 Baseline Confidence Score.
Hendrycks & Gimpel 2016 have shown that the softmax output of neural networks constitutes a good bascline confidence score for failure detection. Precisely, using the softmax output associated the predicted class as the confidence score i.e. c(x: Pÿ: where p E RR is the softmax output and is the predicted class index. For multiclass settings (and binary task where the prediction threshold is 0.5) this is equivalent to c(r)= max P...
2.1.2 Uncertainty Estimation.
Recently, several methods have been proposed to obtain better umcertainty/confideeee estimates, notably:
MC-dropoul (MC): Gal & Ghahramani 2016 showed that training neural network with dropout regularization Srivastava et al. 2014) produces a Bayesian approximation of the posterior, where the approximation is obtained by Monte-Cari sampling of the network's parameters i.e. by applying dropout at test-time and averaging the outputs over several inference passes. The confidence in the prediction can then be approximated by the negative entropy of the outputs; by taking the softmax confidence score the averaged outputs.
Laplace approzimation (L) [Laplace 1774): where the posterior locally approximated with Gaussian distribution centered at local maximum (in practice around the maximum-a a-ott---rootttoori estimate), with covariance matrix corresponding to the local curvature (obtained by an approxima- lion of the Hessian). Recent work from Daxberger et al. (2021 proposed a simple-to-use, lightweight, Python implementation of this approximation, enabling benchmarking the approach easily on pre- trained neural networks.
SWAG: Maddox et al. (2019) define a Gaussian distribution whose mean is parameterized by the stochastic weight averaging (Izmailov et al. 2018) solution, and whose covariance matrix is a low rank matrix plus diagonal covariance derived from the stochastic gradient descent iterates. They then sample several times from this distribution to form the approximate posterior solution.
3

---

Published in Transactions on Machine Learning Research (10/2022)
Table 1: Comparing the technical requirements of each failure detection method considered this study. 'Base' stands for softmax confidence score baseline (Hendrycks & Gimpel 2016). 'MC' for Monte- Carlo
dropout (Gal & Ghahramani] 2016), 'Da" for Doctor (Granese 2019 : DUQ (Van Amersfoort al. 2020
20210. 'TS' for TrustScore (Jiang al.
2018), 'I.' for Laplace (Laplace 1774]. : 'SWAG' (Maddox al.
"CN" for ConfidNet (Corbière et al. 2019
EEns' for Ensembles Lakshmminaaaaaan et al. 2016).
Base
MC
D,
TS
L
SWAG
DUQ
CN
Ens
Special classification model training
x
X
X
X
d
Architecture constraints
/
X
X
X
X
x
Increased model training cost
x
X
X
Post hoc uncertainty score training/fitting
x
x
X
X
/
Increased inference time
x
Deep Ensembles (Ens): Lakshminarayanan et al. 2016 have shown that ensembling predictions from several trained models can yield better calibrated uncertainty estimates than the ones obtain from single deterministic models and even from Bayesian neural networks, in particular due to their increased capacity to capture multi-modal solutions.
Deterministic Uncertainty Quantification (DUQ): Van Amersfoort et al. (2020) estimate predic- tive confidence based on distances between points and class centroids the embedding space and demonstrated improved OOD performance.
DOCTOR: Granese et al. (222) proposed to use Da(x) = I-g(a) g(z) where g(x) E. (z) as a score quantifying the likelihood of being misclassified (i.e. negative confidence score) as an alternative to the classic predicted softmax confidence score.
These uncertainty estimates have been shown be state-of-the-an in terms of model calibration and OOD detection (Laplace) 1774 Gal & Ghahramani 2016 Lakshminarayanan et al. 2016 Maddox et al. 2019:
Van Amersloort al., 2020). However, only subset of them has been benchmarked for their failure detection performance and only for a limited number of tasks e.g. Band et al. 2021
2.1.3 Utilizing intermediate representations for confidence scoring.
Another line of rescarch focuses on computing confidence scores using an auxiliary model or intermedi- ate representation, instead of merely looking the model's final output. Jiang al. (2018) construct a neighbour-graph in the embedding space from the penultimate layer (on the training set) and use distances in this space to derive TrustScore (TS). In practice, this confidence score is defined the ratio between: (i) the distance between the test point and the closest point that does not belong to the predicted class; (ii) the distance between the test point and closest, point that belongs to the predicted class. Corbière et al. (2019)
proposed ConfidNet (CN) regression network that is placod on top of the classification model to predict the "true class probability" Le. the probability predicted by the main model for the true class (as opposed the probability of the predicted class). . In other words, the image is fed through the trained classification model to extract its embedding (penultimate layer), which in turn is fed into ConfidNet which predicts the
softmax output for the true class, as originally predicted by the main model. In table [ summarize the requirements of each confidence scoring method.
2.2 Failure Detection Evaluation
In the recent failure detection literature, a variety of metrics have been used, with seemingly little consensus which metric most appropriate. These metrics can be broadly divided in two groups.
4

---

Published in Transactions on Machine Learning Research (10/2022)
2.2.1 Classification metrics
The first approach consists of treating the failure detection problem as a classification problem where the goal is to predict whether a. given sample has been correctly classified or not, using the confidence score as predictive score. The quality of a given confidence score can then be evaluated with any standard binary classification metric such as ROC-AUC, or the false positive rate (FPR) at given true positive rate (TPR) (Hendrycks & Gimpel
2016
Corbière et al. 2019
2.2.2 Selective classification metrics
Another set of metrics derives from the selective classification literature, notably the use of risk-coverage curves Yaniv et al.) 2010). This metric consists of plotting the risk (typically percentage of errors) as a
function of the dataset coverage (the percentage of dataset that has not been rejected) (Corbière et al. 2019:
El- Yaniv et al.] 2010). Others have used alternative versions of this metric such the ROC-AUC-coverago
curve [Band ct al., 2021 or the risk-score percentiles curve [Lakshminarayanan et al., 2016 Jiang et al. 2018). It is imporlant to note that risk-coverage curves are by definition sensitive to the initial classification performance of the model. Therefore, using this metric to compare various confidence scoring schemes may introduce confounding between model performance and confidence estimation quality. For example, the initial classification accuracy of an ensemble is lypically superior to that of single model, thus we may observe a lower risk-coverage curve for the ensemble even if both models are equally competent at failure detection. This may falsely indicate that one confidence scoring scheme is better than the other. This is the reason why in this work we focus on metrics like ROC-AUC for misclassification detection and FPR at 80% TPR.
2.2.3 Note on misclassification detection and model calibration metrics.
In the model uncertainty literature, one common approach to evaluate the ability to capture uncortainty is by measuring model calibration, often quantified via the Expected Calibration Error (ECE) (Nacini et al. 2015 Guo et al. 2017). ECE is obtained by partitioning the prediction into M bins and measuring the difference between the observed accuracy Ac)(Bm)) and the average model confidence Conf( Bm.) in each bin Bm. that Here, we would like to emphasize that ECE is not necessarily
is ECE = Σm [Bml [Ac)(Bm)) Conf(Bm)) a good metric for measuring how well a model will perform terms of misclassification detection. Indeod, the ability of a given confidence score to discriminate between correctly and incorrectly classified samples is determined by its ability to correctly attribute lower confidence to misclassified samples than to correctly classified samples. Only the relative ranking of confidences scores affects their ability to find failures, not their absolute value which merely changes the choice of rejection threshold.
To further illustrate why ECE. is not necessarily a good metric for failure detection, we can look at the following toy example. Let us suppose that we have two classification models that predict the exact same labels for any given point. "Model 1" is perfectly calibrated model, that is Confi (z;) = p/ = P(4 where y/ is the true class and di the predicted class. "Model 2", on the other hand, while keeping the same ranking of its predictions, is overconfident: Conf2(x;) = pg. (2) = 0.9 + 0.1 P(1y=&)) We further assume for both models p(1 l2;)) ~ Bernoulli(z;), where x; ~ Uniform(0, 1]. We then simulate the models' responses by sampling 10,000 pairs (z,, 1y.=))) according to the above distribution, compute the corresponding confidence scores distribution for Model and Model and compute the ECE as well as the ROC-AUC for misclassification detection. Results are rated fig. 2 After simulation, we obtained a near-perfect ECE of 0.02 for Model but a substantially worse score of 0.45 for Model 2. However, in terms of misclassification performance both models yield the exact same ROC-AUC for misclassification detection, the predictions of Model 2 are simply shifted version of the ones of Model 1. That is I b = -82.
ROC-AUC(Confi (I:)): w= ) ROC-AUC(Conff(
This simple example provides an intuition of the reason why even though predicted softmax outputs tend to yield ever-confident probability estimates [Hendrycks & Gimpel, 2016 Guo et al., 2017). they were still shown to provide good baseline for misclassification detection (Hendrycks & Gimpel 2016 Granese et al. 2021). Given the fact that misclassification detection metrics are only sensitive to the relative ranking of predicted probabilities, methods like temperature scaling (Guo et al. 2017) aiming to calibrate outputs by
5

---

Published in Transactions on Machine Learning Research (10/2022)
Model 1
Model
ECE 0.02
ECE 0.45
ROC-AUC misclassification detection 0.82
ROC-AUC misclassification detection 0.82
Correct
2000
Correct
Misclassified
Misclassified
200
1500
Count
Count
1000
100
500
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Confidence Score
Confidence Score
Figure Distribution of observed confidence scores distributions for 'Model 1' and 'Model 2 in the toy example described in section 2.2.3 We can see that Expected Calibration Error (ECE) not necessarily a good metric for measuring the misclassification detection performance of a given confidence score.
rescaling them, do not impact the failure detection abilities of given model (but merely change the value of the rejection threshold).
3 Experimental Setup
3.1 Datasets
First, evaluate confidence scores on 3 tasks from MedMNIST-v2 Yang et al. 2021 (all with test set size above 5,000 images). PathMNIST (Kather et al. 2019 consists of von-overlapping patches from histology slides annotated with 9 colon diseases classes (with train and test splits from different clinical centers). TisSucMNIST (Ljosa et al.) 2012) is comprised of kidncy cortex cells microscope images, classified into 8 classes cell subtypes. Organ images in axial view, classified by organ type (11 classes). All images have resolution of 28x28 pixels and we use the original train-val-test splits. We have included these small resolution datasets as: (i) this resolution is very common in the machine learning literature it allows for fast prototyping and benchmarking: (ii) these datasets contain tens of thousands of labelled images, including test sets with more than 5,000 images; (ili) despite their low resolutions, even simple models can achieve good performance on their respective tasks (see table 2 indicating that sufficient signal is still present in the downsampled images.
(Xu al. 2019 comprised center slices from abdominal CT
Secondly, we evaluate on three more challenging medical imaging tasks with higher resolution images, using data from the RSNA Pneumonia Detection Challenge (Shih et al. 2019), the Breast Ultrasound Image Dalaset (Al-Dhabyani
al. 2020) (BUSI) and the ByePACs Diabetic Retinopathy Detection Challenge dataset. The RSNA Pneumonia Detection Challenge consists of detecting the presence of pneumonia-like opacities on chest X-rays (26,684 scans). In the BUSI dataset, brcast ultrasound scans are classified normal, benign tumors, and malignant tumors on smaller dataset (780 images in total). For both datasets, we randomly split the data in 70%-10%--20% train-val-test splits. The EyePACS dataset comprised of high- resolution retina images depicting various stages of diabetic retinopathy. The original labels consisted of a 5-class classification task, here we follow the approach of [Band et al. ): Leibig al. (2017) and binarisc the task distinguish sight-threatening diabetic relinopathy' (original classes [2, 3, 4}) and 'non-sight- threatening diabetic retinopathy' (original classes {0, 1}). This dataset consists of 32,126 training, 10,906 validation and 42,670 test images. These three datascts add another perspective in terms of resolution (images resized to 224 x 224 for RNSA and BUSI, 512 x 512 for EyePACS), task difficulty, imaging modality and dataset size compared to the datasets from MedMNIST-v2. Visual examples for each dataset. can be found in fig.

---

Published in Transactions on Machine Learning Research (10/2022)
BUSI
RSNA
Eyc
OrganAMNIST
PathMNIST
TissueMNIST
Figure 3: Examples images of each dataset used in the failure detection testbed. For the EyePACS dataset depict the preprocessed images.
3.2
Implementation Details
3.2.1
Classification Models.
For our main benchmark we use ResNet-18 (He et al. 2016 architecture for all the MedMNIST and BUSI tasks and a ResNet-50 model for the RSNA Pneumonia and EyePACS tasks (note that the goal of this study is to assess the suitability various confidence scoring schemes, not to improve the performance of disease detection models themselves). All models are trained with an additional dropout layer after each weights layer to be able to run the MC-dropout comparison (with dropout probability p=0.1 for all experiments, based validation performance). For all models the learning rate is divided by 10 after 10 epochs with decrease in validation loss. We stop training after 15 consecutive epochs with decrease in validation loss and chose the model with the lowest validation loss for testing. For BUSI, all images were resized to 224 x x244, we applied brightness, contrast, horizontal flips, random rolations and random crop augmentations at training time. For the pneumonia detection task, we used the same augmentations as in Bernhardt et al. (2022). We processed the EyePACS dataset using following Band et al. (2021) which follows the preprocessing from the winning solution of the EyePacs Diabetic Retinopathy Detection Challenge. After preprocessing images resized to 512x512 prior to input the network. For both binary tasks (RSNA and EyePACS) the classification threshold was chosen such that the FPR was 20% on the validation set.
3.2.2 Failure Detection Methods.
For MC-dropout and SWAG, we set the number of inference passes to 10 (we did not find any notable improvement when increasing the number of inference passes). For SWAG and Ensemble we used the average softmax output of the predicted class as confidence score. For the training of ConfidNet we parameterisod network and optimizer following the code provided by Corbière et al. (2019), and also used the checkpoint with the lowest validation AUPR for error detection. We have not further tuned the main model during
7

---

Published in Transactions on Machine Learning Research (10/2022)
OrganAMNIST
RCC AVCAVC Ener Detacton
0.18
0.93
0.92
6.14
0.12
0.91
0.90
0.06
Dase
0,
MC-S
MG-E
CN
SWAG
DUQ
Em
Bam
MCS
NC-E
CN
SWAG
DUG
Ens
PathMNIST
0.85
0.43
BOCAUC Ererr Detection
0.84
0.40
0.83
0.82
0.35
0.81
0.39
0.79
0.28
0,78
0.25
Sase
O,
15
MC-S
MCEE
CN
SWAG
DEQ
Ems
Bas
0g
MCS
MC-E
CN
SIAAG
DUQ
Ens
TissueMNIST
0.70
RDC PC-UUC Enerr Ererrerreeeee Ditection
0.75
0.74
0.23
0.44
Base 0,
MC-S MCE
CN SWAG DUO
Base
MCS
NC-E
SWAG DUC
Ens
Figure 4: Failure detection benchmark for smaller resolutation datasets: OrganAMNIST, PathMNIST, TiSSueMIIST. Comparison of Baseline (Base), DOCTOR Da, TrustScore (TS), MC-dropout with softmax score (MC-S), MC-dropout with entropy score (MC-E), Laplace (L), ConfidNet (CN), SWAG, DUQ and Ensemble (Ens). Except for Ensemble, boxplots are constructed with results of repeated training over seeds, whiskers denote minimum and maximum value observed. For Ensemble, we formed 5 different ensembles by taking 5 different combinations of out of the trained models.
the training of ConfidNet. For the Laplace method, we apply the Laplace approximation on the last layer weights using Kronecker approximation of the Hessian, as per the recommended parameters in Daxberger let al. (2021). For SWAG (Maddox al.] 2019), we tuned the learning rale schedule the validation set, and for DUQ, we followed Van Amersfoort et al. 2020)) for tuning of hyperparameters. Note that when using DUQ for BUSI and EyePACS, we had difficulties finding set of parameters that provided competitive accuracy with our baseline, nonetheless we show the results obtained with the best hyperparameters found. All the code and hyperparameter configurations to reproduce our results can be found at htttps://githn com/ htt////maannefafaaaaeaeeaeaeeeee detection benchmark.
4 Results
In figs. and compare the failure detection performance of confidence scores: Baseline (softmax score), TrustScore, Laplace Approximation, MC-dropout, ConfidNet, SWAG, DUQ and Ensemble for 6 different datasots. We evaluate the failure detection performance using ROC Error Detection, the ROC- AUC score for classifying correct against misclassified cases (where the positive class is "correctly classified"") and FPRØ80TPR: the FPR at 80% TPR for the error detection task (where the positive class is "correctly classified") L.c. percentage of missed error cases at 20% false alarm rate. We also report classification performance table To avoid confounding effects between confidence estimation and the model, use the same architecture and regularization scheme for all confidence scores. In particular, all models were
2
8

---

Published in Transactions on Machine Learning Research (10/2022)
BUSI
RCC AVA Erer Detacioon
0.25
0.70
0.80
0.60
0.77
0.70
0.65
0.30
0.20
Dase
0,
15
MC-S
MG-E
CN
SWAG
DUQ
Em
Baw
MCS
MC-E
CN
SWAG
DUC
Ens
RSNAPneumonia
8DC AUUC Erer Detoctio
0.86
0.40
0.85
0.35
0.83
0.82
0.30
0.81
0.28
0.25
5aM
P.
15
MC-S
CN
SWAG
DUQ
Ens
Babe
0,
15
MC-S
CN
SWAG
DUQ
Ens
EyePACS
0.90
BOC XUUC Endr Difecion
0.80
0.75
0.70
0.65
0.55
0,20
0.50
Bsse 0.
MC-5
CN
SWAG
DUO
Ens
Base
G.
T5
MC-5
SWAG
DUO
Ens
Figure 5: Failure detection benchmark for higher resolution datasets: BUSI, RNSA Pneumonia Detection and EyePACS datasets. Note that we excluded MC-E on the binary tasks as the chosen classification threshold was different from 0.5.
trained with dropout, but we only apply dropout at test-time for the MC-dropout confidence. This differs from other works (Maddox et al. 2019 Band el al. 2021] where numbers reported for the softmax baseline were obtained from model trained without dropout. We argue that changing the regularisation scheme between different methods detection performance, and hence decided to use the same regularization techniques for all models.
confound the effects of regularisation and uncertainty estimation failure
The results show that softmax confidence is a strong baseline for misclassification detection. None of the benchmarked methods are able to significantly outperform this baseline consistently across datasets, not even computationally expensive methods such as ConfidNet, MC-dropout, SWAG or Ensembling, and this across all our misclassification detection metrics. particular, we observe that ensembles increase model performance over a single deterministic model across datasets, however, when evaluating their misclassifica- tion detection ability with metrics controlling for classification accuracy, they do not improve consistently simple baseline. To test this hypothesis further, reran all experiments using two additional model architectures (DenscNet-121 and WideResnet-50) and observe the same phenomenon: none of the methods oulperform the baseline softmax score, see fig. 6 where we report failure detection ROC-AUC for each model, dataset combination. Moreover, we observed that TrustScore is significantly underperforming for TissueM- NIST and RSNA Pneumonia, whercas it performing well on other datascts. To understand why, fig. 7 we analysed the t-SNE (Van der Maaten Hinton (0088) representation of the embedding spaces and we observed that this correlates with less well class-separated embedding space.

---

Published in Transactions on Machine Learning Research (10/2022)
ResNet-28
WideResNet550
0.94
Dctaciio
ROC Eree Detedtion
RCC OrganAminsT
0.93
0.92
0.92
Eneer
0.91
0.90
0.30
0.90
2.89
0.99
0.89
0.80
Realvet-
WideResNet500
0.85
0.88
AOC Enter Detedion
0.30
0.84
0.85
0.88
0.86
0.83
0.84
0.82
0.83
0.82
0.81
ROC
0.81
0.80
0.80
0.78
0.79
Reaivet-18
WideReNNe:300
ROC Detection
0.77
0.78
ROC-AMC Eiver Deteetton
0.79
ROC-UU Envor Detectin
0.78
0.77
0.76
0.75
0.77
0.75
0.73
0.75
0.73
WideResNet300
0.85
ROC.AUC Detection
0.84
6.84
ROCAUC Enerr Detection
0.80
0.82
0.82
0:75
0.80
0.78
0.70
0.78
9.78
0.65
0.74
0.60
0.74
0.35
Densefvet-121
WideResNet:30
0.66
Bec AUc-AUC Eioor Detection
0.85
ROCAUC Entor Ditertion
0.26
0.85
RSNAPneurool
0.84
0.84
0.83
0.82
0.02
0.81
0.80
ROC.AUC Einor Detectioo
WDC AUC Ereor Detesston
ROC AUC Entor Detection
0.89
0.88
0.88
0.87
0.87
0.86
0.26
3.86
0.85
0.84
Figure 6: Model architecture effect ablation study. SWAG, ConfidNet and DUQ not included in this ablation
because of computational cost (as they all require separate training).
10

---

Published in Transactions on Machine Learning Research (10/2022)
Table 2: Classification performance for ResNet models: accuracy for multiclass tasks and ROC-AUC for binary tasks. For all methods except Ensemble, we report the average over seeds, standard deviation brackets. For Ensemble, we formed different ensembles by taking different combinations of 3 of the trained models, report the average over of the 5 ensembles, standard deviation in brackets.
Dataset
Bascline
MC
Laplace
SWAG
DUQ
Ensemble
OrganAMNIST
.902 (.005)
.902 (.005)
.901 (.006)
.910 (.004)
.917 (.004)
.921 (.002)
PathMNIST
.888 (.009)
.322 (.010)
.835 (.010)
.830 (.010)
.828 (.025)
.853 (.002)
TisSueMNIST
.663 (.004)
.664 (_003)
.663 (.004)
.670 (.001)
.655 (.007)
.682 (.001)
BUSI
740 (.020)
.740 (.021)
.740 (.020)
.792 (.017)
.561 (.000)
.742 (.017)
RSNA
871 (.007)
.873 (.006)
.871 (.006)
.873 (.003)
.865 (.003)
.877 (.003)
EyePACS
899 (.006)
.899 (.007)
.899 (.007)
.913 (.004)
.730 (.007)
.918(.02))
aa) OrganAMNIST
(b) PathMNNST
(c) TisSueMNIST
S
(d) BUSI
(e) RSNA Pneumonia
(1) EyePACS
Figure t-SNE representation of the embedding space on the training set for each dataset for ResNet models. Note that for better visibility, maximum of 5,000 random samples plotted (but we used the entire training set to construct the 1-SNE representation).
11

---

Published in Transactions on Machine Learning Research (10/2022)
5 Discussion
Key take-aways. Our investigation showed that the softmax confidence score baseline is difficult to beat: none of the benchmarked advanced confidence scoring methods were able to consistently outperform the baseline across datasets. Crucially, our experiments show that previously demonstrated improved robustness for out-of-distribution detection or model calibration does not necessarily translate to improvements in error detection for in-domain inputs. This finding indicates that the in-domain failure detection an orthogona,, complementary task to these related tasks and should as such be evaluated separately. This task has been vastly understudied in comparison to the attention that OOD detection has received and we hope that these findings will foster more research in this key arca. Morcover, our study distinguishes itself by the particular care taken to avoid confounding effects between model performance and improvements in confidence score quality, both by: (i) our choice of evaluation metrics; (ii) whenever possible, keeping the same classification model to evaluate various confidence scores, and ensuring that regularization schemes such as dropout were kepl constant for all models evaluated.
Practical implications. Our study presents the first extensive failure detection benchmark covering a wide range of medical datasets with various image resolutions (from 28x28 to 512x512 pixels), a wide-range of imaging modalities and with tasks of varying difficulty. In particular, our work shows that care should be taken prior to deploying an automated failure detection system to a real-world scenario by carefully evaluating the performance of such a system on the practical task at hand. Our results indicate that current uncertainty scoring, including simple softmax baseline, are able to detect some misclassified cases at a substantially better than chance level (ROCAUC .75 for all benchmarked datasets), demonstrating the potential to improve practical performance of Al systems by deferring suspicious cases to humans. It is important to note that regardless of the dataset or the confidence score, there is still clear margin for improvement as FPR@SOTPR (the proportion of missed errors at reasonable false alarm rate of 20%) remain very high for most datascts. We highlight that our work is not meant as a one-time assessment, but as a starting point towards a more systematic and objective evaluation of failure delection methods. We envision this testbed to dynamically grow, inviting more results to be added over time through contributions from the community.
Baseline choices and limitations. As mentioned in the background section, in this study we focused on most widely used confidence scoring schemes, as these are most likely to be deployed in practical scenarios and are thus at a higher risk of being misused. Nevertheless, despite our best efforts to choose representative methods of the field - covering a diverse set of approaches to confidence scoring (Bayesian uncertainties, softmax-based scores, scores based on intermediate representations) this benchmark is naturally not an exhaustive evaluation of all uncertainty and confidence scores ever presented in the literature. As such, we make no claims as to whether other uncertainty or confidence score methods proposed in literature are (or will be) able to beat the softmax baseline consistently. Nonetheless, our study clearly demonstrates that failure detection ought to be carcfully studied in isolation, as other tasks such as OOD or calibration do not provide a reliable proxy for the misclassification detection performance of a given confidence score. Moreover, given the variability of the results from one dataset to another, this experimental setup demonstrates the need for more broad evaluation of new confidence scores in the future.
6 Conclusion
In this study, we conducted thorough evaluation of 9 widely used confidence scores for failure detection medical datasets. We found that among all the benchmarked confidence scores, none were able to outperform the softmax baseline consistently across datasets demonstrating that improved model calibra- tion or performance on OOD does not necossarily translate to improvement for in-domain misclassification detection. Overall, our study strongly suggests that failure delection requires further research towards find- ing more appropriate confidence scoring schemes, as this topic is a key concern for the medical imaging community where safcty concerns are paramount for deployment of AI models. This work facilitates further investigations by providing complete testbed for evaluating future progress in this space in a rigorous and comprehensive manner.
12

---

Published in Transactions on Machine Learning Research (10/2022)
Broader Impact Statement
This work evaluates failure detection methods in the context of medical imaging and is therefore closely related to AI safety concerns for healtheare-related deployment. This work provides a testbed to facilitate further work in the key area of automated failure detection and calls upon more research in this direction. Nevertheless, this work does not arguc for the use of any particular method in practice and this testbed has been designed for research purposes; as such any conclusions obtained using this testbed should always be validated locally using representative data for the considered practical deployment scenario.
Acknowledgments
M.B. is funded by an Imperial College London President's 's scholarship. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant Agreement No. 757173, Project MIRA).
References
Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmyy Datasct of breast ultrasound 000100000010/..0.01000044462
images. Data in Brief, 28:104863, 2020. ISSN 2352-3409. doi:
Mauro Annarumma, Samuel Withey, Robert Bakewell, Emanuele Pesce, Vicky Goh, and Giovanni Montana. Automated triaging of adult chest radiographs with deep artificial neural networks. Radiology, (11(1) 2019.
Neil Band, Tim GJ Rudner, Qixuan Feng, Angelos Filos, Zachary Nado, Michael W Dusenberry, Ghassen Jerfel, Dustin Tran, and Yarin Gal. Benchmarking bayesian deep lcarning on diabetic retinopathy detection tasks. In NewrIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
Mélanie Bernhardt, Daniel C Castro, Ryutaro Tanno, Anton Schwaighofer, Kerem C Tezcan, Miguel Mon- teiro, Shruthi Bannur, Matthow Lungren, Aditya Nori, Ben Glocker, et al. Active label clcaning for improved dataset quality under resource constraints. Nature communications, 13(1):1 11, 2022.
Francisco M. Calisto, Alfredo Ferreira, Jacinto C. Nascimento, and Daniel Goncalves. Towards touch- medical image diagnosis annotation. New York, NY, USA, 2017. Association for Computing Machin- ery. ISBN 9781450346917. doi: 10.1145/3132222.333111 URL https://doi.org: 11155/32114//332227
3134111
Francisco Maria Calisto, Carlos Santiago, Nuno Nunes, and Jacinto C. Nascimento. Introduction of human- centric ai assistant aid radiologists for multimodal breast image classification. International Journal of
Human-Commpin Studies, 150:102607, 2021. ISSN 1071-5819. doi: https://ddo..org/10000115
102607. URL http:///wwwcccienneeeeeet....... htt.://ccccccccccc...occcceeeeaa
Francisco Maria Calisto, Carlos Santiago, Nuno Nunes, and Jacinto C. Nascimento. Breastscreening-ll Evaluating medical intelligent agents for human-ai interact ions. Artificial Intelligence in Medicine, 127: 102285, 2022. ISSN 0933-3657. doi: https://doi.org// 10. ottp://////bboooo///ooo/oooo/ota sciencedirect.co httpapacww....ooo....a./a/aaaaaa
URL https://wwa
Robert Challen, Joshua Denny, Martin Pitt, Luke Gompels, Tom Edwards, and Krasimira Tsaneva- Atanasova. Artificial intelligence, bias and clinical safety. BMJ Quality & Safety, 28(3):231-22) 2019.
Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. arXiv preprint arXiw:1910.04851, 2019.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-offrrlless bayesian deep learning. Aduances in Neural Information Processing Systems, 34, 2021.
13

---

Published in Transactions on Machine Learning Research (10/2022)
Ran El-Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, (5). 2010.
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. nature, 542(7639); 115 118, 2017.
Oliver Faust, Wanrong Hong, Hui Wen Loh, Shuting Xu, Ru-San Tan, Subrata Chakraborty, Prabal Datta Barua, Filippo Molinari, and U. Rajendra Acharya. Hcart rate variability for medical decision support systems: review. Compulers Biology and Medicine, 145:105407, 2022. ISSN 0010-4825, doi: https:// /oi.oor/10.1111// comph/o/d00..ooooooooooo/dd02220 2022..0407... pi1/$00004425222019998
URL wtt.:ttsss/ww..siienndiiiiee http::/www..ccecceee...a.caaeeea hotp:///ccccciieee//aiilloo
Yarin Gal and Zoubin Ghahramani. Dropout a bayesian approximation: Representing model uncertainty in deep learning. In international conference machine learning, pp. 1050-1059. PMLR, 2016.
Federica Granese, Marco Romanelli, Daniele Gorla, Catuscia Palamidessi, and Pablo Piantanida. Doctor: A simple method for detecting misclassification errors. Advonces in Newral Information Processing Systems, 34, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330 PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, 770 778, 2016.
Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and out-of-disttriuttio examples in neural networks. arXiv preprint arXiv:1610.0214 2016.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. aiviv preprinl urXiv:1803.05.. 2018.
Heinrich Jiang, Been Kim, Melody Y Guan, and Maya Gupta, To trust or not to trust a classifier. arXiv preprint 11X5 11783, 2018.
Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo- Clon Weis, Timo Gaiser, Alexander Marx, Nektarios A Valous, Dyke Ferber, al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS medicine, 16(1)=e1002730, 2019.
Chris K Kim, Ji Whae Choi, Zhicheng Jiao, Dongeui Wang, Jing Wu, Thomas Y Yi, Kasey C Halsey, Feyisope Eweje, Thi My Linh Tran, Chang Liu, et al. An automated covid-19 triage pipeline using artificial intelligence based on chest radiographs and clinical data. NPJ digital medicine, 5(1):1-9 9, 2022.
Benjamin Kompa, Jasper Snock, and Andrew Beam. Second opinion needed: communicating uncertainty in medical machine learning, upj Digital Medicine, 4, 12 2021. doi: 10.1038//411444-000-0033--
Balaji Lalshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty cstimation using deep ensembles. arXiv preprint 16Xi::662..0147, 2016.
Pierre Simon Laplace. Momoir on the probability of the causes events. tome sixième. Mémoires de Mathématique et de Physique (English translation by SM Stigler 1986. Statist. Sci., (19): 364-378),
1774.
Christian Leibig, Vanceda Allken, Murat Soçkin Ayhan, Philipp Berens, and Siegfried Wahl. Leveraging uncertainty information from deep neural networks for discase detection. Scientific reports, 7(1):1-14, 2017.
14

---

Published in Transactions on Machine Learning Research (10/2022)
Christian Leibig, Moritz Brehmer, Stefan Bunk, Danalyn Byng, Katja Pinker, and Lale Umutlu. Combining the strengths of radiologists and ai for breast cancer screening: retrospective analysis. The Lancet Digital Health, 4(7)=e507 e519, 2022. ISSN 2589-7500. doi: http:////w///oooo////0//0///0000
URL
hatpps:/wwit www. www.snedieeeeetecc...... oag.acach.....ooo.eee.../ttaaaap
Vebjorn Ljosa, Katherine Sokolnicki, and Anne Carpenter. Annotated high- high-throughput microscopy image sets for validation. Nature methods, 9:637, 06 2012. doi: 10.1038/mmeth.200 mmt///mmmtttt......
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32:13153-13164, 2019.
Mahdi Pakdaman Nacini, Gregory F. Cooper, and Milos Hauskrecht, Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAT Conference Artificial Intelligence, AAAI11,, pp. 2901 2907. AAAI Press, 2015. ISBN 0262511290.
Ozan Oktay, Jay Nanavati, Anton Schwaighofer, David Carter, Mclissa Bristow, Ryutaro Tanno, Rajesh Jena, Gill Barnett, David Noble, Yvonne Rimmer, et al. Evaluation of deep learning to augment image-guided radiotherapy for head and neck and prostate cancers. JAMA network open, 8(11)::2027426-22227777 2020.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V Dillon, Bal- aji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? evaluating predictive uncertainly under dataset shift. arviv preprint Xrx: 02530, 2019.
Shahab Shanshirband, Mahdis Fathi, Abdollah Dehzangi, Anthony Thcodore Chronopoulos, and Hamid Alinejad-Rokny. A review on deep learning approaches in healthcare systems: Taxonomies, chal- lenges, and open issues. Journal of Biomedical Informatics, 113:103627, 2021. ISSN 1532-0464. doi: 2020..0077..... URL
pJ1//$15320466200025500
George Shih, Carol C Wu, Safwan S Halabi, Marc D Kohli, Luciano M Prevedello, Tessa S Cook, Arjun Sharma, Judith K Amorosa, Veronica Arleaga, Maya Galperin-Aizenbery Aizenberg,, et al. Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiology: Artificial Intelligence, 1(1):e180041, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
Philipp Tschandl, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel Codella, Allan Halpern, Monika Janda, Aimilios Lallas, Caterina Longo, Josep Malvehy, et al. Human cancer recognition. Nature Medicine, 26(8))1229-122) 2020.
Human-computer computer collaboration for skin
Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a sin- gle deep deterministic neural network. In International conference on machine learning, pp. 9690-9700. PMLR, 2020.
R R Van De Leur, B Hompot, R J Hassink, P A Doevendans, and R Van Es. Interpretable uncertainty estimation for automated triage of 12-lead electrocardiogram using deep convolutional neural networks. European Heart Journal, 42(Supplement 1), 10 2021. doi: 10. 10.1093 10093//uhhaartt//aaa24...1111 URL https:
//doi.org/10) 10930099/euue 1ww.////eeeaaaatehaaaa4......1
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Xuanang Xu, Fugen Zhou, Bo Liu, Dongshan Fu, and Xiangzhi Bai. Efficient multiple organ localization in ct image using 3d region proposal network. IEEE transactions on medical imaging, 38(8):1885- 18988, 2019.
15

---

Published in Transactions on Machine Learning Research (10/2022)
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2: A large-scale wiihtt benchmark for 2d and 3d biomedical image classification. arXiv preprint Nri::22110.1470 2021.
Hai Zhong, Jiaqi Wu, Wangyuan Zhao, Xiaowei Xu, Rumping Hou, Lu Zhao, Ziheng Deng, Min Zhang, and Jun Zhao. A sclf-supervieed learning based framework for automatic heart failure classification cine cardiac magnetic resonance image. In 2021 43rd Anmual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 1887-28901 2021. doi: 10.1109/EMBC441IC EMBC46164 202.9630288.
16


Reviews:
[{"review_id": "w9WSoWM5c8", "reviewer": "Reviewer_XowJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and well-executed, and has a clear and important take-away message. Just because confidence scores are useful for OOD detection, we should not assume they will be useful for ID failure detection. That none of the methods consistently outperformed the softmax baseline is concerning, and points to a need to develop better methods for this important use case in medical AI. The experiments included a large variety of confidence scoring methods, and the choice of model architectures was sound.\n\nThe main weakness of the paper is that out of six datasets, three are trivial with 28x28 image resolution, which does not reflect a realistic medical AI setting. Additionally, of the three remaining datasets, the breast ultrasound dataset only has 780 images. Since there are only two datasets with tens of thousands of high-resolution images, I am not certain whether the paper's conclusions would transfer to a state-of-the-art medical AI setting with hundreds of thousands or millions of high-resolution images.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present an empirical study comparing confidence scores for in-distribution (ID) failure detection across Bayesian and Non-Bayesian methods, several medical imaging datasets, and multiple deep learning architectures. Their main contribution is to show that no confidence score outperforms a simple softmax baseline consistently across datasets. While these methods may be useful for out-of-distribution (OOD) detection, this does not necessarily mean they are useful for ID failure detection. This is problematic, as ID is the more commonly occurring and thus more practical use case in the clinical setting.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The paper would be stronger with the incorporation of at least one more publicly available large-scale medical imaging dataset such as CheXpert.", "broader_impact_concerns": "There is no broader impact statement present, and one does not seem necessary. The paper addresses ID failure detection in the medical setting, where improvements are likely to benefit society as a whole without harming any particular demographic.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vxup2evrkv", "reviewer": "Reviewer_vyW2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper starts with a focus on out-of-distribution detection methods and then focuses on a few scattered methods related to uncertainty estimates. There is a significant number of methods focused on just out of distribution methods. Such as a comprehensive OOD benchmark here: https://arxiv.org/abs/1809.04729 and then one specifically for Medical OOD here: https://arxiv.org/abs/2007.04250 Both of these works explore many OOD methods not used in this submission. It is impossible to evaluate all methods but there needs to be some argument for the methods selected which relate to the point that is being made.\n\nBecause of this I think the scope of the paper needs to be focused on uncertainty estimates and with that I do not think it is comprehensive. So some argument should be given for why the methods chosen are representative. There are 24 papers with uncertainty in the title in NeurIPS 2021 alone.\n\nI do not see this work providing actionable insights. There is no focus on modality, types of features, or types of tasks. What insights can I take away from this work to better develop a method for uncertainty estimation?\n\nThe author states \"our study strongly suggests that failure detection requires further research towards finding more appropriate confidence scoring schemes\" which I don't believe is surprising and I don't think this work is comprehensive enough to prove it either.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "\nThis paper argues that \"previously demonstrated improved robustness for out-of-distribution detection or model calibration does not necessarily translate to improvements in error detection for in- domain inputs\".\n\nThey look at 6 different methods across a variety of small and large resolution datasets.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Clarify the scope. Narrow it to the specific point you are trying to make. Then justify the selection of methods and datasets to address this specific point. If you want to make a broad sweeping statement of all failure detection methods then you need to have a good argument that me evaluating some random method I find won't change the conclusion of your paper.\n\nEnumerate the insights that will help move the field forward.\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IE5CoxULFu", "reviewer": "Reviewer_K7ez", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strengths\n\n**1.1.** The manuscript addresses key concerns of failure detection in healthcare;\n\n**1.2.** The manuscript brings forward novel perspectives for ML developments in healthcare;\n\n#### Weaknesses\n\n**2.1.** Contributions and impact for this scientific community are not clear;\n\n**2.2.** The novelty of the work is also dubious;\n\n**2.3.** Findings and implications of the work are merely discussed;", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this manuscript, the authors report a study for the evaluation of nine different confidence scores for failure detection on six medical datasets. Specifically, their study presents the first extensive failure detection benchmark covering a wide range of medical datasets with various image resolutions, a wide-range of imaging modalities, and with tasks of varying difficulty. In the end, the authors are suggesting that failure detection requires further research towards finding more appropriate confidence scoreing schemes. Which is not a fair conclusion after all. Although this work is facilitating further investigations, the work findings for the community and final contribution are not clear. In fact, providing a complete testbed for evaluating future progress in this space seems a small contribution for this venue. However, the authors are following a well represented method and metrics to denote their achievements. Overall, this is an exciting line of work, but the current draft of this manuscript has some minor concerns that must be addressed first before publication. Contributions, if so, must be described clearly. Additionally, the findings are also important attributes of such important research work. As follows, further comments are detailing improvements for the next iterations of the manuscript.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "There is much to appreciate in this manuscript, as it investigates that the softmax confidence score baseline is difficult to beat. However, the provided findings and work implications are not enough, and the discussion section would need to be considerably strengthened by a more robust engagement with the cited literature. I would encourage the authors to think about how their findings extend or add to the ML and medical literature in this space in order to refine their contribution to these fields. For instance, it would be interesting to understand the application of these failure detection techniques to a real scenario [1, 4]. At least, a small discussion on how could these techniques help real problems, such as different medical domains [2, 3, 5].\n\nThe paper needs a brief proofreading as some English phrases seem to be off. Be aware of tenses also. It would be wise to proofread the paper and correct these, as well as other spelling errors.\n\nThe related work seems scarce for a work around the healthcare domain. The paper must discuss recent work in a methodological sense and base their work on the proposed approach. A lot of work is already available into this topic [1, 3, 5]. Specifically, the scientific community has numerous works on this topic [1, 5], but authors could also map some literature around other domains [3, 5], where these level of approaches are essential.\n\nTo conclude, the paper is almost ready for publication and the authors did a splendid work. However, I strongly recommend the authors improve the manuscript for the next iterations of the work, as this is a chief importance domain.\n\n# Major Comments\n\n1. Contributions and impact for this scientific community are not clear;\n\n2. The novelty of the work is also dubious;\n\n3. Findings and implications of the work are merely reported;\n\n# Minor Comments\n\n4. The literature must be further addressed;\n\n5. The paper hardly fails on missing citations;\n\n6. Lack of necessary context about the domain and problem;\n\n7. Discussion section could be improved;\n\n8. It would be wise to proofread the paper, as well as correct grammar and other spelling errors;\n\n#### Missing References\n\n[1] https://doi.org/10.1016/j.compbiomed.2022.105407\n\n[2] https://doi.org/10.1016/j.artmed.2022.102285\n\n[3] https://doi.org/10.1109/EMBC46164.2021.9630228\n\n[4] https://doi.org/10.1016/j.ijhcs.2021.102607\n\n[5] https://doi.org/10.1016/j.jbi.2020.103627\n\n[6] https://doi.org/10.1145/3399715.3399744\n\n[7] https://doi.org/10.1016/j.cmpb.2019.03.008\n\n[8] https://doi.org/10.1145/3132272.3134111", "broader_impact_concerns": "Although this is an interesting read and its topic is timely, as is, the paper has some minor concerns, for the impact and its implications, that must be addressed first. That said, the authors need to improve the current status of the manuscript.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: First of all, I would like to apologize for the delay in making a decision.

After carefully reading the paper, the reviews and the responses, I think that although the idea in this paper is not particularly exciting (could an empirical analysis of existing methods be particularly exciting?), it is a well executed empirical work on a relevant research question with practical implications.

The reviews for this paper were of high quality and the reviewers brought a number of valid concerns. Especially, some points brought up by the reviewer vyW2 might not have been sufficiently addressed (e.g. the rationale for the choice of methods, I recommend to acknowledge this in the final version of this paper). Overall, with the vote 2:1 for rejection after rebuttal indicates that the authors did not manage to fully convince the reviewers. Having said that, I think the reviewers of this paper might have been on the harsher side in terms of their final recommendations. 

I think this is a valid contribution, with correctly executed experiments, which will be of interested to the community. Given that "TMLR emphasizes technical correctness over subjective significance", I recommend that this paper to be accepted with minor revisions. The main effort should be to incorporate the discussion points brought up by the reviewer yvW2 to the best of your ability. Finally, I think it would be beneficial to carefully proofread this paper by some of the experienced co-authors to make the paper flow a little better (e.g. in the background section).
Decision: Accept (poster)