Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection
Authors: Chunjong Park
Anas Awadalla
Tadayoshi Kohno
Shwetak Patel
Venue: NeurIPS 2021 Poster (2021)
Keywords: out-of-distribution
dataset shift
trustworthy AI
health

Abstract: Unpredictable ML model behavior on unseen data, especially in the health domain, raises serious concerns about its safety as repercussions for mistakes can be fatal. In this paper, we explore the feasibility of using state-of-the-art out-of-distribution detectors for reliable and trustworthy diagnostic predictions. We select publicly available deep learning models relating to various health conditions (e.g., skin cancer, lung sound, and Parkinson's disease) using various input data types (e.g., image, audio, and motion data). We demonstrate that these models show unreasonable predictions on out-of-distribution datasets. We show that Mahalanobis distance- and Gram matrices-based out-of-distribution detection methods are able to detect out-of-distribution data with high accuracy for the health models that operate on different modalities. We then translate the out-of-distribution score into a human interpretable \textsc{confidence score} to investigate its effect on the users' interaction with health ML applications. Our user study shows that the \textsc{confidence score} helped the participants only trust the results with a high score to make a medical decision and disregard results with a low score. Through this work, we demonstrate that dataset shift is a critical piece of information for high-stake ML applications, such as medical diagnosis and healthcare, to provide reliable and trustworthy predictions to the users. 

Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection
Chunjong Park, Anas Awadalla, Tadayoshi Kohno, Shwetak Patel Paul G. Allen School of Computer Science & Engineering University of Washington {cjparkuw, anasa2, yoshi, shwetak) atwetakk/os..wwhhhnto edu
Abstract
Unpredictable MI. model behavior on unseen data, especially in the health domain, raises serious concerns about its safety repercussions for mistakes be fatal. In this paper, we explore the feasibility of using state-of-the-ar out-of-distribution detectors for reliable and trustworthy diagnostic predictions. We select publicly available deep learning models relating to various health conditions (e.g., skin cancer, lung sound, and Parkinson's 's disease) using various input data types (e.g.. image, audio, and motion data). We demonstrate that these models show unrea- sonable predictions on -iitt--diittiittoonn datasets. We show that Mahalanobis distance- and Gram matrices based out-of- of-dissribuution detection methods are able detect -ititt--dittiittionn data with high aceuracy for the health models that operate on different modalities. We then translate the cnttof-distribution score into a human interpretable CONFIDENCE SCORE to investigate its effect on the users' interaction with health ML applications. Our user study shows that the CONFIDENCE SCORE helped the participants only trust the results with a high score to make medical decision and disregard results with a low score. Through this work, we demonstrate that dataset shift is a critical piece information for high-stake ML applications, such as medical diagnosis and healthcare, to provide reliable and trustworthy predictions to the users.
Introduction
Advances in artificial intelligence and machine learning have made medical diagnostic and screening tools more accurate and accessible. Al-powered diagnostic tools 5 14] are intended to assist medical personnel by making unbiased decision based on thousands of examples. In recent years, these models [5
32 are even becoming available to consumers through the growth of mobile health with the intention of expediting diagnoses through increasingly frequent testing. Moreover, mobile health ! 50 aims to improve access to medical expertise for those who are uninsured or live far away from hospitals.
Despite the potential benefits of health AI systems, there are concerns about their performance in real-world settings. Data-driven models learn from examples, making them heavily reliant the data upon which they have been trained. However, datasets often fail to get complete coverage domain, particularly for emerging datasets; when new pulmonary diseases (e.g., MERS and COVID-19) emerge, pulmonary classifier trained on the existing lung sounds would not be able to interpret sound of the new diseases. Previous work 42 78 has found that machine learning models
behave unpredictably on the unseen data. This problem 4 69 is especially critical for medical diagnostic and screening tools since there are significant repercussions for mistakes.
Researchers have proposed methods to estimate the uncertainty of a machine learning models' predictions based the input 29
detection methods can distinguish whether the input lies within the distribution of the training dataset, with out-if-disttibuttin
40
4
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

data leading to less reliable prediction results. However, such important information has not been widely explored in the context of health upplications. When health applications are put into the hands of consumers with limited understanding of the underlying algorithms, they may upload poor quality data that lies outside the distribution of the data that was collected by experts. For example, consumers who are using health application that involves image processing may take photographs in poor lighting conditions or framing of the target object. Even when the data is high-quality, it may be captured with smartphone that has different hardware specifications than the devices that were used to collect the training dataset. Unless the models are explicitly designed or trained to detect invalid data, the models will incorrectly produce diagnostically meaningless result.
In this work, we explore the utility of -fittiiibitiitoinnnnn detection for improving model performance and pereeieed trustworthiness of health-related models. We first benchmark our approach using publicly available deep learning models relating to various medical challenges and sensing domains images for skin lesion classification, motion data for Parkinson's 's disease severity, and audio for lung sound classification. After demonstrating that these models are susceptible dataset shift, we demonstrate that the state-of-the-art out-of -iittiibiitiiiiioinnnn detectors can effectively exclude such data with over 95% detection accuracy in most cases. We then explore the implications of this detector on user-perceived trustworthiness of the health models. After translating the out-of-distribution score into a human- iimman-itterrrttbble metric, CONFIDENCE SCORE, we found that showing this information to end-users improved the user- rer-peeceiiee trustworthiness of the models. Furthermore, participants stated that they were more willing to make medical decisions based on models when they were shown the certainty metric. Our contributions in this work are as follows:
We identify and quantify the limitations of current health deep learning models when encountered with unseen data,
We evaluate the utility of audio, motion) for medical screening and diagnosis, and
detection on various data types (e.g., image,
We evaluate the impact that dataset shift information has on user-perceived trustworthiness of health diagnostic results.
2 Related Work
Machine Learning- Health Screening and Diagnosis In recent years, machine learning has been widely used for medical diagnosis and screening tool to help doctors diagnosis patients easier, faster, and more accurately. Machine learning models that learn from targe-scale medical datasets are able to detect various symptoms and conditions, including mental health [26] [8 retinal disease [14], lung cancer [5). With the increasing ubiquity of smartphone and advances in its computing power, machine tearning-based health screening can be done mobile devices. Various machine learning- based mobile health applications have been proposed to detect health conditions (e.g., traumatic brain injury 49 pancreatic cancer 488) jaundice (15)) and vital signals (e.g.. heart rate [d , respiratory rate heart rate variability 32 blood pressure 655 SpO2 BI). Such mobile health applications can benefit nurses, health workers, and the general population for easier medical screening.
While the health machine learning models show high accuracy on their own test datasets, their performance is questionable in real-world settings where the input data can vary drastically, resulting Researchers have investigated the dataset shift problem for
in unreliable prediction results 77 6X]
medical imaging (e.g., x-ray [1I J0,, fundus eye images [II] CT scans 74] dermatology [63] (66). focusing on developing and evaluating outtof-distribution detection methods for specific domains. However, as more consumer- facingg health applications are available in the market, this issue can lead the users to make medical decision based on incorrect results. In this work, we aim to explore ways to leverage dataset shift information to make the health machine learning models more reliable and trustworthy to the users.
Dataset Shift Detection Recently researchers have proposed various methods to estimate the models' uncertainty due to dataset shift. The proposed methods leverage the output of the models to effectively
detect out-of- distiriibbttion input that are different from the known distribution, in-distribution. Softmax confidence (29 has been the baseline for the out-s--disti detection. Several work has been proposed for -itto--distribution detection using deep ensemble
38 Mahalanobis distance 40).
Gram matrices [e]] energy score 43), temperature scaling 41
input perturbation
mean and variance of channels activations [58] Alternate training strategies
30
39 have

---

been proposed to enable model to detect -ittof-dittribution. Generative models [54 60 53
are examples. However, many approaches require re- re-training and
proposed to detect aut-of-distribure re-designing of the models and prior knowledge of out-of-distribution datasets; it is not realistic to apply these methods to the existing models. this work, we explore Mahalanobis distance- [40],
Gram matrices- [64], and energy-based [+ dut-of-distribution detection methods for reliable and trustworthy machine learning for health since these methods show reasonable out-of-distriutiion
detection performance, do not require retraining or prior knowledge of out-of iitoo-ddistributon datasets, and work on pre-trained discrimitive classifiers.
Trustworthy AI Machine learning systems are deployed in real-world settings to billions of users, making significant impacts on high-stake decision making such as healthcare, policy, economy, and transportation. Failures in machine learning systems can cause fatal consequences and building trustworthy AT is one of the most important problems in machine learning community. In recent years, there are active and ongoing efforts aimed at making machine learning systems causal ( (55 explainable
72,, and This work contributes to trustworthy AI by improving reliability and user-perceived
36
K5] fair
robust [8 21
[
57
51 F
trustworthiness of machine learning for health using estimated uncertainty. Bhatt et al. [8] proposed to leverage uncertainty for users making decision and placing trust in machine learning models. This work explores similar approach where we adopt out-of- -fdiiiiiiibtttinn detection as a method to measure uncertainty. We took step further to investigate and quantify its effect in improving reliability and trustworthiness in the context of machine learning for health.
3 Background: Dataset Shift Detection Methods
We aim leverage state-of-the- out-of-distriuution detection methods 40 64 1
the health
domain for users to safely use health deep learning models. We selected three out- -iittiibbitonnonn methods that show high accuracy on different out-of-distribution datasets, do not require re-training and prior knowledge of out-of--istribur These characteristics are important to help developers or other stakeholders (c.g.. regulators, auditors, platforms) easily adopt out-of- -ii---i-iitiitttionn detectors to any pre-trained models. In this section, we detection methods.
datasets, and work on pre- pre-trained discrimitive classifiers.
provide background on each
3.1 Mahalanobis Distance-Base Based Diittr-DDittiittionn Detection
Mahalanobis distance used to measure the proximity of point to a certain Gaussian distribution. In the Mahalanobis distance- based -utttf-diitributtin detector 40), this property is used to represent each class's 's samples at each layer of a network as a class conditional Gaussian distribution with mean Ad and co-variance Edt, where e indicates the class and indicates the layer in the model. Given a sample input to neural network, for each layer, it computes the minimum layer-wise class conditional Mahalanobis distances for x. That is, for each layer, it finds the Mahalanobis distance associated with the closest class x. In other words, this is equivalent to M (xx) max.- - (f(x)- p= E-1 po). The authors have demonstrated that adding small noise to the input can help better distinguish between in-distribution and -iitr---iitiitttion data. As the authors suggested, for the real-world setting where the out-of- disttiibutiion datasets are generally not available, we obtain the input noise magnitude by generating adversarial samples generated by FGSM 27
3.2 Gram Matrices-Based Out-of- Out-of-Distributin Distribution Detection
Gram matrices are used to compute pairwise feature correlations and encode stylistic attributes. For out-of-distribution detection [44, higher order Gram matrices are used to compute class-conditional bounds of feature correlations at all hidden layers of the network as higher order shows better out-of- distribution detection performance. Higher order Gram matrices is expressed as G.¹² (F.' pf"") } where Fi is feature map at (-th layer and P is order. All elements of Gram matrices of an input at each layer are compared against the prepossessed minimum and maximum Gram matrices element values from nn-distribution dataset to obtain deviation. If the input data is predicted as a certain class, the minimum and maximum values of the corresponding class will be used for comparison. The comparison is done for each layer to obtain layerwise deviations. Then, the deviations are used to get a total deviation, which is defined by the normalized sum of layerwise deviations. Whether the input data from -istr--distiiuttion is determined with a threshold which is defined as 95% percentile of the total deviations of in-distribution energy score distribution.
3

---

3.3 Energy-Basee Out-of-Distribution Defection
The energy-based based -ittt--diitiiuution detector 4 seeks to provide an alternative scoring function to the softmax function that is less susceptible to over- ver-confiee and therefore can better distinguish between and out-o--distribution inputs. takes a discriminative classifier [() that maps input E IR_ to logits, which are traditionally used to derive a categorical confidence score using a softmax function. It defines the energy function the classifier E(x; f) - log Σx ef(())/T, where K' is the number of classes in the model's output space and T is a temperature parameter that can be used to alter the shape of the energy score distribution. Energy score threshold that distinguishes between in- and out-of off-dsfriibbbttin samples is defined at the 95% percentile of the in-distribution samples.
4 Performance Evaluation
In this section, we demonstrate the performance degradation of the existing health models when encountered with out-of-distribution datasets highlighting that the existing models are vulnerable to dataset shift. We then evaluate the performance of state-of-the-an distinguishing between in- and out-of-distribttion examples. We have selected out-of-distributiin datasets that consist of both near- and frot-distributiion samples that represent realistic use cases in the real-world settings. For mobile health applications that use mobile sensors for health screening, non-expert users are expected to input data collected by themselves. Unlike clinicians, who may either receive training how to operate these mobile apps or may already understand what must be done to generate high-quality, non non-expert consumers may collect relevant but low-quality data due to environmental factors or totally irrelevant data by mistake scenarios, we include out-of of--isiiiiittiionn datasets caused by covariate shift, label shift, and open-set recognition. The covariate and label shifts aim to evaluate model's performance when tested on data pertaining to the same classification task but from different data sources and environment. Open-set recognition evaluates the model's performance Table 2 we indicate dataset shift type for each out-of-distribution dataset.
out-t--distribution detectors for
lack of understanding. To reflect these
new classes included in the training set. In
4.1 Models and Datasets
used in this work. The model aims to classify an image into seven different skin lesions: actinic keratoses, basal cell carcinoma, benign keratosis, dermatofibroma, melanoma, melanocytic nevi and vascular lesions. The following datasets are used for training and evaluation:
Skin lesion A DenseNet-122 121 based skin lesion classifier 56
(In-distribution) HAM10000 12::
(CC BY-NC 4.0) dataset containing 10,000 samples of dermatoscopic skin tumor images taken using different devices and from different populations. These tumors are part of 7 classes: actinic keratoses, basal cell carcinoma, benign keratosis-like lesions, dermatofibroma, melanoma, melanocytic nevi, and vascular lesions.
(CC BY-NC 4.0) previous version the HAM100000 dataset which contains 2000 dermatoscopic skin tumor images labelled for binary classification. tumor is labelled malignant corresponds to melanoma to benign if it corresponds to nevus or seborrheic keratosis.
(Out-of-distribution) ISIC 2017 13:
Face 16 (CC BY 4.0) A dataset containing frontal view face images of 102 adults without making neutral facial expression. Face images are personally identifiable information. But, all individuals gave signed consent for their images to be "used in lab-based and web-based studies in their original or altered forms and to illustrate research (e.g., in scientific journals, news media or presentations)."
-intr---distibbution) CIFAR-I0 37 (MIT License) A common image classification benchmark with 10 non-medical classes (airplane, car, cat, dog, horse, bird, deer, ship, frog, truck) which contains 6,000 images per class.
Lung Sound A lung sound classification model crackle from an audio sample. This model is based on ResNet-34 and uses spectograms of audio samples as inputs and outputs 4 lung sound classes (normal, wheezing, crackle, and wheezing + crackle).
23 classifies normal lung sound, wheeze, and

---

(In-disributtion) ICBHI 2017 Respiratory Challenge [11| A dataset collected using mul-
tille microphones and stethoscopes containing 6898 samples normal lung sound, wheeze, and crackle audio
OOut-of-distribution) Stethoscope [22] (CC BY 4.0) A dataset containing stethoscope
respitory sounds with 336 samples of normal breathing, wheeze, and crackle audio sounds. The dataset was collected using a 3M Littmann Electronic Stethoscope.
(CC BY 4.0) A large dataset of millions of sound labelled YouTube audio of which portion of the dataset contains breathing, cough, and wheezing samples which we use to create suitable out-of-distribution dataset for this model.
AudioSet (24
Parkinson's 's Disease This is a binary classification model [76 that showed highest performance in Parkinson's disease digital biomarker DREAM challenge [66]. The model uses accelerometer signals to detect tremors in a person's 's movement and outputs whether a participant has Parkinson's ss This model consists of 5 ID-convolutional layers and single output.
(In-distribution) mPower 9 (CC BY 4.0) A dataset contains 30-second accelerometer readings from 3,100 participants rest for both healthy and Parkinson's patients. The dataset was used in Parkinson's 's disease digital biomarker DREAM challenge [66].
(Out-of-distribution) Kaggle Parkinson's 's disease 25 (CCO 1.0) A dataset with accelerom-
eter readings from healthy participants simulate movements of Parkinson's 's patients.
(Out-of-distribution) MotionSense :4); (MIT License) A dataset contains accelerometer
readings from 24 participants performing various activities (e.g., walking, jogging, sitting, standing, etc).
(Out-oo-distributtio)) MHEALTH 7 An activity classification dataset which contains
accelerometer readings from 10 participants executing various activities (e.g., standing, sitting, walking, cycling, etc).
4.2 Performance Impact by Dataset Shift
In evaluating the model's performance the futtof-diitributton dataset, used pre- tre-trained models
from the previous work [233 when the authors make it available. Otherwise, we trained the model in
the same way specified in their work (56 [6]. We trained skin lesion model [56] for 150 epochs
using Adam optimizer with a learning rate of 0.0001 and weight decay of 0.2. For Parkinson's 's disease
model 76.. we trained for 50 epochs using Adam optimizer with a learning rate of 0.0005. The pre-trained lung sound model [X] is trained for 200 epochs using SGD optimizer with a learning rate of 0.001 and momentum of 0.9. For all of these models, we used an 80/20 split and applied the same preprocessing for train and test sets. All training and testing is done in a server (Intel Xeon 2. 1GHz, 64GB, GeForce RTX 2080 Ti) from an internal cluster. We then ran inference on each dataset and calculated the classification accuracy for the datasets that have corresponding labels. For the datasets that do not have the same labels from the in-distribution, the accuracy could not be computed. Table ! summarizes the classification accuracy for the models on in- and out-of-distribution datasets.
We generally observed significant performance drop for all health machine learning models that tested with out-of-distributie datasets. The models output unreasonable and arbitrary predictions on datasets that are not related health conditions. For example, skin lesion classifier predicts all face images as vascular lesions and CIFARIO images as various types skin lesions. Similarly, Parkinson's disease classifier predicts significant portion of physical activities by health participants as tremor caused Parkinson's 's disease. For lung sound classification, ordinary sound events (e.g.. speech, walking, laughing) are classified as certain type of lung sounds (e.g., crackle, wheezing). When the models are evaluated on out-of out-of-distrinn didstribbutoon datasets that have similar data characteristics the nn-iitribution data(i.e., mear-distti datasets), all health models exhibit a performance decrease that ranges from 18% to 76%. This implies that the models are also sensitive to small dataset shift, such as datasets collected with different devices and in different environments. All of these failure scenarios can occur in real-world deployment of health machine learning applications. Users
5

---

Health ML Models
In-Distribution
Out-of-Distriben
Skin Lesion (DenseNet-12))
HAM10000
ISIC 2017
Face
CIFAR
92.05%
74.00%
N/A
N/A
Lung Sound (ResNet-34)
ICBHI 2017
Stethoscope
AudioSet
78.50%
2.10%
Parkinson's Disease
mPPowe
Kaggle Parkinson's
MotionSense
MHEALTH 10.00%
(5x (3xIDoConv)
82.01%
26.67%
45.83%
Table 1: Accuracy of health deep learning models on in-distribution and out-of-distribution dataset.
Accuracy is not available (N/A) for out-of-distribution datasets that do not have corresponding labels.
can imput face image to skin lesion classifier, improperly record lung sound and input ambient sound to the lung sound classifier, or imput motion data when they are not at rest to the Parkinson's s disease classifier. Furthermore, diverse sensors and devices used in real-world deployment can cause significant performance drop. This evaluation demonstrates that users are exposed to the health machine learning applications that can provide unreliable diagnostic results.
4.3
Out-of-Distribution Detection Performance
The previous evaluation implies that it is crucial to determine whether the input data belongs to in- or out-of-distriuution avoid failures caused by dataset shift. this section, we investigate the feasibiliity of using state-of-the-art out-of-distributio detection methods in the context of machine learning for health. We evaluate Mahalanobis distance- 400. Gram matrices- (64, and energy- based [33 methods, which work any pre-trained discrimitive classifiers and do not need re-training
and prior knowledge of -f--itiiiitttnionn datasets, in detecting -itor--distribution data for different health models.
4.3.1 Experimental Setup
For Mahalanobis distance-base based method we extracted Mahalanobis distance-basee based scores from the output dense and residual block found in DenseNet and ResNet respectively. For the Parkinson's model which does not contain dense and residual blocks, we extracted the scores at the end of each convolutional layer. Then, optimized the input noise magnitude using in-distribution samples and corresponding adversarial samples generated by FGSM (77] The noise magnitude obtained 0.0 for skin lesion classifier, 0.0005 for Jung sound classifier, and 0.0 for Parkinson's 's disease classifier. For Gram matrices- matrices-based method we extracted class-specific minimum and maximum correlation values for all orders of Gram matrices for all feature pairs. Total deviation values, which are used for out-of-distribun samples from in-disrribution datasets. For energy- based method we use their method that does not require fine-tuning to avoid re- re-training of the network. We use the default temperature scaling (T 1) as suggested in 4 All evaluations are repeated for trials and we report the mean (Table 2 and standard deviation (Table 4 of all metrics.
detection threshold, are computed with multiple sets of random
4.3.2 Evaluation Metrics
For out-of-distribution detection, it common to use true negative rate (TNR) at 95% true positive rate (TPR), AUROC, and detection accuracy to evaluate the performance of detector. Particularly, as the out-of-distribution problem a binary classification problem, we consider out-of-distribution samples negative and in-distribution samples positive. TNR TPR 95% is defined as the percentage of correctly detected ofttt--diitributtion samples, when 95% of in-distribution samples are correctly detected. The AUROC metric measures the area under the TPR vs FPR curve. The detection accuracy measures the maximum possible classification accuracy over all possible thresholds in distinguishing between in-distribution and out-of-distribution examples. Detailed explanations the metrics are available in Appendix
htttp://ggthab..cc//pppppppppppp Mahalaaoiiaaaa detector
bttpa///ggtaab..cc///rtrrttttttt gereee/ggam--eeee gram-oodd-eeectiinn

---

Validation on OOD Samples (TNR e TPRSS/AU CDreeeeerieeeoeoooooooooooo Accuracy)
Hoalth ML Models In-Distibuttin
OuttuffDDurriaaiion
Distributiio Shif
Mahalanohis
Gram
Energy
Skin Leiion
HAMEOO0
ISIC 2017
Covariatstabel shilt
10.13/56.21 99.25
25.90/11.14/7 14/114//74 774.88
14,28/76.20 70.76
(DenseNe-111)
Face
Open-set recognition
100,00999.0 98 /90 96
95.01/9K
34
0.00/ / 80:45 88.81
CIFARIO
Open-sct recogntion
99,33/99.00 90/ 99.61
95,14 98.66//9////// 96.90
5,06 / 58.33 / $7.89
Lung Sound (ResNo-S4)
ICBHI
AudioSet
Upen-stt recogaiton
97.96/99.47 97.34
96.55 18/ 99597
8.12/56.79 6.12/56.9//5711
Stathoscope
Covristooibbel shift
45,40/36.2 86.27 90:87
48.77/88.755 /8.75/ 76.05
7.29 / (0.95 $8.94
Pukiimon's Disease (SxID-Com)
inPower
Motionensic
Open resegaition
10IL00/ 86, 99,89
100,00 (09.94 199,60
0,000 / 58.71 /66496
Opensect recagnition
100.00 100.00 100.00
160.0/ 99 99 1099999990000
0.00 /41.41 50 44
Keggle Parkinwnn's
Conristooibbel shift
18.00/99,800 89 /99.47
98.95/ 99.96 99.67
70,00/ /95.91 /9334
Table 2: Out-of-distribution detection performance across different networks and datasets.
4.3.3 Results
shows out-of-distribution detection performance for different methods across different health machine learning models and datasets. Overall, Mahalanobis distance- and Gram matrices-based out-of-distribution detection methods consistently show outstanding performance across different neural networks and different out-of-distribute These methods show lower performance in distinguishing near- distriiitiittion datasets (e.g.. ISIC 2017, Stethoscope, Kaggle Parkinson's s),, which aligns with the results from previous out-of-distribution work [64]. On the other hand, the energy-based method did not show reasonable performance in detecting out-of-distribution samples. We found that the energy scores of were not able to effectively discriminated from in-distribution samples shown in Appendix C Note that we used the energy scores without fine-tuning; however, the authors of energy score-based method [ have demonstrated that classifier that is fine-tuned using the energy score in place of the softmax score shows significant improvement in out-of-distribution detection performance. This -iitr--distiitttonn detectors can be applied to health machine learning applications to provide reliable diagnostic results to the users.
Table
datasets, showing TNR @ TPR95 of 95% or above.
samples
evaluation implies that state-of-the-ar
User Study
According to the trustworthy Al literature 220, providing users with interpretable information can enhance the trustworthiness of the result and potentially impact users' decisions. We therefore conducted an online survey-based user study to validate this effect and the impact that our approach has on model trustworthiness. We first defined CONFIDENCE SCORE as how confident the model is in interpreting an input. In other words, in-distribution input would have a high CONFIDENCE SCORE, whereas out-of-distribution input would have low CONFIDENCE SCORE. We compute CONFIDENCE SCORE by scaling raw out-of-distribun 100, where 0 is most likely to be an nut-of-distribution example and 100 is most likely to be an in-distribution example. Scaling is done in a piecewise manner. When out-of outtof-distribution scores are within an n--iisributtin threshold, which set to include 95% of in in-distribution examples, we compute min-max scaling that ranges from 90 to 100. In this way, we ensure that most of the in-distribution examples have confidence scores of 90 or above. When out-of-distribution scores outside of an in-distribution threshold, we compute min-max scaling from O to 90, where the same denominator is used as above since -iitt--iisttituttion examples might not be available in practice and any negative values are clipped to O. We then investigate the effect of CONFIDENCE SCORE on user-perceived trustworthiness and its impact on medical decisions. Additionally, we also quantify potential learning that can be gained when it comes to distinguishing between in- and out-of-diie distriiibtt input samples. Specifically, we aim to answer the following research questions:
scores from out-of-distribution detectors 40 6
to 0-
RQ. How does the CONFIDENCE SCORE affect the perceived trustworthiness of diagnostic results?
RQ. How does the CONFIDENCE SCORE affect medical decisions based on diagnostic results?
RQ. 3 Is there potential learning effect from CONfIDENCE SCORE when it comes to distinguishing between input data with high and low CONFIDENCE SCORE?
5.1 Study Procedure and Participants
The overview of the online user study is illustrated in Figure I and list of the user study interfaces detailed Appendix A In short, the interface displays simulated results from the health screening models used in Section 4 (.c., models for skin cancer, lung sound, and Parkinson's 's discase). We made
7

---

Repeat for nach health ML models
Repeat for each input sample
Consent
Model Information
Bineline
Confidence Score
Learning Effect
Demographics Instruction
pepurt
breat Genbdeees Geee
He'r wate ennner dingrestle Al spottn thst sineiu benign (ast concerual. Tho modil show d9% ocnanooy obbrrsin stuciss
medigrnet
Figure 1: User study overview. The participants are first asked to give consent, read instruction, and provide demographies. Then, they report perceived trustworthiness and willingness to make medical decision after seeing input samples that consist of different data types, diagnostic results, and CONFIDENCE SCORE for baseline and CONFIDENCE SCORE condition. Screenshots of the user study interface are demonstrated in Appendix A
User-perceived trustworthiness
Impact on making medical decisions Effect Size
Wilcoxon Test (WV
P
Effect Size
Wilcoxon Test (IV
P
529,950.0
0.001 ***
0.393
223,227.0
0.001 ***
0.178
In-Distribution
138,778.5
< 0.001 ***
0.475
52,056.0
< 0.001 ***
0.200
distri-iiitiiiation
126,814.0
< 0.001"**
0.317
59,790.5
0.001 **.
0.158
Negative result
131,910.0
2.0001"' "*
0.393
51,197.5
0.026'
0.100
Positive result
133,301.0
< 2.0011* ***
0.394
60,751.5
< 0.001 ***
0.258
Image
55,890.0
< 0.000
0.436
20,884.0
< 0.000
0,225
Audio
64,440.0
2.0001 ...
0.384
25,848.0
0.002
0.173
Motion
56,767.5
< 000011**
0.361
28,084.5
0.019*
0.133
Table 3: Results of Wilcoxon test in comparing baseline and CONFIDENCE SCORE conditions for
the perceived trustworthiness and impact on decision making. All comparisons show statistically significant results. ($*#; < 0.001, #*; < 0.01, *: p < 0.05).
the input data as human-readable as possible to maximize interpretability. Images were shown as is, while audio included in audio player that participants could play, pause, and stop the track. We presented the motion data as a time-series plot of accelerometer signals from x-, y-, and z-axis. Since time-series data can be particularly challenging for non- non-experts, we explain that high-amplitude signals are associated with fast motion while while low-amplitude signals are associated with slow motion. The interface explained the model's purpose and accuracy, which was fixed 90% to remove potential bias. For each model, the interface presents prediction results in two different conditions: (baseline) input and result, and (confidence score) input, result, and CONFIDENCE SCORE. For each result, we asked participants how much they trust the model's prediction and whether they would be willing to make medical decision based on that result. Participants saw total of 24 scenarios (3 data types (image, audio, motion) x 2 conditions (baseline vs. CONFIDENCE SCORE X 2 CONFIDENCE SCORE (high vs. low) x 2 results (positive vs. negative). To provide realistic experience, we provide different skin tone images for skin lesion samples based on the reported skin tone. With the exception of the data type, the scenarios were shuffled across all other factors avoid any ordering effects. After participants saw all of the scenarios for given data type, we presented them with five data examples and asked them to pick the ones that the model would confident processing according to CONFIDENCE SCORE. We added these questions to assess whether people were able to learn about how the CONFIDENCE SCORE was being generated after sceing a series of examples.
We intend to target ordinary, non-expert consumers at random rather than expert clinicians for our study. Our research is primarily directed toward the boom in consumer-facing mobile health applications, where non-expert users are expected to collect input data themselves. We believe this where models are most susceptible to out-of ----isiiiibttiinnn inputs, providing unreliable predictions to the users. For the safe use of Al Al-powered health applications, the users would need support via automated uncertainty measures. To this end, We recruited participants from Amazon Mechanical Turk and compensated with S3 USD for 15-min online study. In total, 192 participants (1555 male, 67 female) completed the online study with an average age of 42.7 9.1 years. The study was approved by Institutional Review Boards at the University of Washington.
8

---

5.2 Results
We analyzed the responses using the Wilcoxon signed-rank test 75 to compute a pairwise comparison of the categorical responses between the baseline and CONFIDENCE SCORE conditions. Table 3 summarizes these statistical results along with the Rosenthal correlation coefficient 62] (r) for effect
size.
RQ. 1: User-Perceived Trustworthiness In general, we found that user-perceived trustworthiness (p < 0.001) was higher in the CONFIDENCE SCORE condition with medium effect size (r = 0.393). In other words, the dataset shift information helped the participants decide when to trust or not to trust the oulput of the models. Higher CONFIDENCE SCORES led to increasing trustworthiness: high scores had a large effect size (r 0.475), while low scores had medium effect size (r 0.317). The impact on trustworthiness was similar for positive and negative diagnostic results. The effect sizes varied for the different input data types, with the images having the largest effect size and motion having the smallest. We suspect that the effect size was correlated with the intuitiveness of the data types, with images being more intuitive than motion data.
When we examined the impact of CONFIDENCE SCORE making medical decisions, we found that there was statistically significant difference (p < 0.001) between the baseline and CONFIDENCE SCORE conditions. In other words, participants were more willing make medical decisions when positive results were presented with high CONFIDENCE SCORE and vice versa. Similar to the results for user-perceived trustworthiness, the effect of CONFIDENCE SCORE was larger on input data with high scores than low scores and highest for images compared to audio and motion data.
RQ. 2: Impact on Making Medical Decisions
Learning Effect on Distinguishing In- and that the participants were able to learn from their interaction with CONFIDENCE SCORE. The average Jaccard index when it came to selecting high CONFIDENCE SCORE input data was 0.75, 0.66, and 0.64 for image, audio, and motion data, respectively, which is moderately high similarity. As with our other results, the Jaccard index was highest on images and lowest on the motion data, implying that ability to understand the input data also has impact on learning effect. This implies that the dataset shift information can make users better understand input data that the machine learning models can interpret for the future interaction.
RQ. 3:
Input Data We found
6 Discussion
Dataset Shift Information for Health Application Users Based on the results from our performance evaluation and user study, we can imagine two potential use cases of the dataset shift information to improve safety and trust in mHealth applications. First, mHealth applications with machine learning models can exclude -ut-o--disti samples avoid making inferences and suggestions that are likely to be inaccurate and unreasonable. Second, our user study shows that the dataset shift information can enhance their interaction with the health machine learning applications. It was found to be particularly effective in improving trustworthiness for in-distribution data and leading the users to make the right medical decision. As the users interact with the health applications longer, they would have better understanding of importance of data quality for future interactions.
Dataset Shift Information for Health Application Developers Our dataset shift information not only improves the user experience, but also yields potential benefits for model developers. If a user correctly captures data but the model rejects it as being out-of-distribution, then there likely exists intrinsic problems or biases with the model. For example, if a skin lesion classifier is only trained on data from people with pale skin and a user with darker skin submits image of their own, the out-of-distributtion detector be triggered due to the incompleteness of the training dataset. The same issues may occur when training dataset only collected from a specific set of sensors (c.g., camera, microphone, IMU) with particular specifications.
Limitations and Future Work Detecting near-distribution samples (e.g., ISIC 2017. Stethoscope, Kaggle Parkinson's 's was a difficult problem for all the out-of-distributio detectors we evaluated. For the distriiitiibtion datasets, we evaluated model's accuracy on the data that are distinguished as in-distribution. This issue is actively investigated by researchers and the improved near diatriiitrinttion detection method would benefit this work.
Our user study was limited the fact that it dealt with hypothetical scenarios. There were no repercussions for users decisions, so they may not have spent as much time making their decisions as

---

they would in real life. There are also many other factors that impact people's health-related decision making, such as the perceived severity of the medical condition and the perceived benefits of taking action 33 [7 We tried to make some of the data more realistic by aligning data with the user's demographic information (e.g., we displayed skin lesion images based on their reported skin tone); nevertheless, participants were aware that the data was not their own. Additionally, increased trust might be affected by participants' own understanding and interpretation of the input data. Although observed increased and decreased trust in examples with high and low CONFIDENCE SCORE, respectively, randomizing the CONFIDENCE SCORE of input data could further quantify impact of CONFIDENCE SCORE user trust of health predictions.
In future work, we would like to (1) investigate the best way to present this information for the users, (2) leverage the dataset shift information for finding potential biases in the train dataset and inherent problems with the model, (3) investigate an out-of-distribution method for better near-distribution detection performance.
7 Conclusion
In this work, we investigated the utility of dataset shift information for improving reliability and trustworthiness of machine learning- based health applications. Using publicly available health deep learning models and datasets, we first demonstrated that the models fail when encountered with unseen data. We then evaluated the -utto--distribution detection performance of state-of-the-ard methods, showing high accuracy in distinguishing between in- and out-of citt-f--dittrbuttinn datasets for different input data types (e.g., image, audio, motion data). We conducted an online user study to investigate the effect of dataset shift information on potential users. We found that the participants trusted prediction results with high CONFIDENCE SCORE and are more willing to make a right medical decision, while they considered prediction results with low CONFIDENCE SCORE less trustworthy and are less willing to make medical decision. This work shows that the dataset shift is meaningful piece of information for building consumer- facing trustworthy Al applications for high-stake decision making.
8 Acknowledgments and Disclosure of Funding
We would like to thank Alex Mariakakis for insightful feedback and discussions. This work was supported by NSF CN5-1565252. Shwetak Patel is on partial leave at Google.
References
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308-318, 2016.
[2] Julius Adebayo, Justin Gilmer, Michacl Christoph Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances Neural Information Processing Systems, volume 31, pages 9505-9515, 2018.
[3] Alekh Agarwal, Miroslav Dudik, and Zhiwei Steven Wu. Fair regression: Quantitative definitions and reduction-basse algorithms. In 36th International Conference on Machine Learning, ICML 2019, pages 120-129. 2019.
[4] Saba Akbar, Enrico Coiera, and Farah Magrabi. Safety concerns with consumer-facing mobile health applications and their consequences: a scoping review. Journal of the American Medical Informaics Association, 233)):33--340, 2020.
[5] Diego Ardila, Atilla Kiraly, Sujeeth Bharadwaj, Bokyung Choi, Joshua J Reicher, Lily Peng, Daniel Tse, Mozziyar Etemadi, Wenxing Ye, Greg Corrado, et al. End-to-end lung cancer screening with three- dimensional deep learning on low-dose chest computed tomography. Nature medicine, 25(6):954-9611 2019.
[6] Susan Athey. Beyond prediction: Using big data for policy problems. Science, 355(6324):483-4855 2017.
[7] Oresti Banos, Rafael Garcia, Juan A Holgado-Tenine Miguel Damas, Hector Pomares, Ignacio Rojas, Alejandro Saez, and Claudia Villalonga. mhealthdroid: a novel framework for agile development of mobile health applications. In International workshop on ambient assisted living, pages 91 91-98. Springer, 2014.
10

---

[8] Umang Bhatt, Javier Antorán, Yunfeng Zhang. Q Vera Liao, Prasanna Sattigeri, Riccardo Fogliato. Gabrielle Gauthier Melançon. Ranganath Krishnan, Jason Stanley. Omesh Tickoo, et al. Uncertainty form of transparency: Measuring, communicating, and using uncertainty. arXiv preprimt arXi::201.005866 2020.
[9] Brian M Bot, Christine Suver, Elias Chaibub Neto, Michael Kellen, Amo Klein, Christopher Bare, Megan Doerr, Abhishek Pratap, John Wilbanks, E Ray Dorsey, et al. The mpower study, parkinson disease mobile data collected using rescarchkit. Scientific data, 3(1):1-9 -9, 2016.
[10] Erdi Çalli, Keelin Murphy, Ecem Sogancioglu. and Bram Van Ginneken. Frodo: Free rejection of dustr--disttiuution samples: application to chest x-ray analysis. arXiv preprint 10X.00:00000053. 2019.
[11] Tianshi Cao, Chinwei Huang, David Yu- Tung Hui, and Joseph Paul Cohen. A benchmark of medical out of distribution detection. arXiv preprint arXiv:2007.04250 2020.
[12] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo. Konstantinos Liopyris. Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the interaationl skin imaging collaboration (isic). arXiv preprint arXiv:1902.000 03368. 2019.
[13] Noel CF Codella, David Gutman. M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: A challenge the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 168-172. IEEE, 2018.
[14] Jeffrey De Fauw, Joseph R Ledsam, Bernardino Romera- Romera-Paredes, Stanislav Nikolov, Nenad Tomasev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan O' Donoghue, Daniel Visentin, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature medicine, 24(9):1322-1350 2018.
[15] Lilian De Greef, Mayank Goel, Min Joon Sco, Eric C Larson, James W Stout, James A Taylor, and Shwetak N Patel. Bilicam: using mobile phones to monitor newborn jaundice. In Proceedings of the 2014 ACM International Joint Conference on Pervaxive and Ubiquitous Computing, pages 331-342, 2014.
[16] Lisa DeBruine and Benedict Jones. Face Research Lab London Set. 2021.
[17] Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mchrnoosh Sameki, Mario Schlener, Jason Tuo, and Hanna Wallach, Assessing and mitigating unfaimess in credit models with the fairlearn toolkit. 2020.
[18] Ivan Evtimov, Weidong Cui, Eee Kamar, Emre Kiciman, Tadayoshi Kohno, and Jerry Li. Security and machine learning in the real world. arXiv preprint arXiv:2007.07205, 2020.
[19] Kevin Eykholt, Ivan Eytimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-wond attacks deep learning visual classification. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1625-1634, 2018
[20] Heike Felzmann, Eduard Fosch Villaronga, Christoph Lutz, and Aurelia Tamô-Larricux. Transparency you can trust: Transparency requirements for artificial intelligence between legal norms and contextual concerns:. Big Data & Society, 6(1):1-14, 2019.
[21] Nic Ford, Justin Gilmer, Nicolas Carlini, and Dogus Cuhuk. Adversarial examples are a natural consequence of test error in noise. arXiv preprint 1901. 1901. 2019.
[22] Mohammad Fraiwan, Luay Fraiwan, Basheer Khassawneh, and Ali Ibnian. A dataset of lung sounds recorded from the chest wall using an electronic stethoscope. Data in Brief, 35:106913, 2021.
[23] Siddhartha Gairola, Francis Tom, Nipun Kwatra, and Mohit Jain. Respirenet: A deep neural network for accurately detecting abnormal lung sounds in limited data setting, 2020.
[24] Jort F. Gemmeke, Daniel P. W. Bllis, Dylan Freedman, Aren Jansen, Wade Lawrenee, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In Proc. IEEE ICASSP 2017, New Orleans, LA, 2017.
[25] Giorgia. Simulation of parkinson movement disorders kaggle, May 2018.
[26] George Gkotsis, Anika Oellrich, Sumithra Velupillai, Maria Liakata, Tim JP Hubbard, Richard JB Dobson, and Rina Dutta. Characterisation of mental health conditions in social media using informed deep learning. Scientific reports, 7(1::-11,, 2017. 2017.

---

[27] lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
[28] Dan Hendrycks and Thomas G. Dietterich, Benchmarking neural network robustmess to common corrup- tions and perturbations. In International Conference on Learning Representations, 2019.
[29] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and df-stiiibittt examples in neural networks. In ICLR (Poster), 2016.
[30] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich, Deep anomaly detection with outlier exposure. arXiv preprint arXiv: 1812.0666, 2018.
[31] Jason S Hoffmman, Varun Viswanath, Xinyi Ding, Matthew J Thompson. Eric C Larson, Shwetak N Patel, and Edward Wang. Smartphone camera oximetry an induced hypoxemia study, arXiv preprint wwXiv:2104.0003: 2021.
[32] Sinh Huynh, Rajesh Krishna Balan, JeongGil Ko, and Youngki Lee. Vitamon: Mcasuring heart rate variability using smartphone front camera. In Proceedings of the 17th Conference on Embedded Networked Sensor Systems, pages 1-14, 2019.
[33] Nancy Janz and Marshall H Becker. The health belief model: decade later. Health education quarterly, 111):1147,, 1984.
[34] Heinrich Jiang, Been Kim, Melody Y. Guan, and Maya R. Gupta. To trust not to trust classifier. In Advances in Neural Information Processing Systems, volume 31, pages 5541 -5552, 2018.
[35] Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wortman Vaughan. Interpreting interpretability: Understanding data scientists' of interpretability tools for machine learning. In Proceedings of the 2020 CHI Conference Human Factors Compuring Systems, pages 1414, 2020.
[36] Been Kim, Martin Wattenberg. Justin Gilmer, Carrie Cai, James Wexler. Fernanda Viegas, al. In- cerpretability beyond feature attribution: Quantitative testing with concept activation vectors (teav). In International conference on machine learning, pages 2668-2677. PMLR, 2018.
[37] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[38] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, pages 6402-6413, 2017.
[39] Kimin Lec, Honglak Lec, Kibok Lec, and Jinwoo Shin. Training confidence-calibrated classificrs for detecting out-oo-disttrbution samples. arXiv preprint arxiv: 2017.
[40] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa- DesaaBBancchi, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[41] Shiyu Liang. Yixuan Li, and Rayadurgam Srikant, Enhancing the reliability of disttriubtniinn image detection in neural networks. In 6th International Conference on Learning Representations, ICLR 2018 2018.
[42] Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pages 3122-31 -130. PMLR, 2018
[43] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribttion detection. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21464-21475. Curran Associates, Inc., 2020.
[44] Xin Liu, Josh Fromm, Shwetak Patel, and Daniel McDuff. Multi-task temporal shift attention networks for on-device contactless vitals measurement. arXiv preprint arXiv:2006.028 03790, 2020.
[45] Scott M. Lundberg and Su-Ia Lee. A unified approach to interpreting model predictions. In NIPS'17 Proceedings of the 31st International Conference on Neural Information Processing Systems, volume 30, pages 4768 4777, 2017.
[46] Mohammad Malekzadeh, Richard G. Clegg, Andrea Cavallaro, and Hamed Haddadi. Mobile sensor data anonymization. In Proceedings of the International Conference on Internet of Things Design and Implementation, IoTDI 19, pages 39-58, New York, NY, USA, ACM.
12

---

[47] Andrey Malinin and Mark Gales, Predictive uncertainty estimation via prior networks. arXiv preprint S00i::1802.100011 2018.
[48] Alex Mariakakis, Megan A Banks, Lauren Phillipi, Lei Yu, James Taylor, and Shwetak N Patel. Biliscreen: smartphone-based scleral jaundice monitoring for liver and pancreatic disorders. Proceedings of the ACM Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(2):1-26, 2017.
[49] Alex Mariakakis, Jacob Baudin, Eric Whitmire, Vardhman Mehta, Megan A Banks, Anthony Law, Lynn Megrath, and Shwetak N Patel. Pupilscreen: using smartphones to assess traumatic brain injury. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3):1-27, 2017.
[50] Alex Mariakakis, Edward Wang, Shwetak Patel, and Mayank Goel. Challenges in realizing smartphone- based health sensing, IEEE Pervasive Computing, 18(2):76 84, apr 2019.
[51] Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE Symposium on Security and Privacy (SP), pages 691-706, 2019.
[52] Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for gener- alizable out-of of-dissributtin detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5216-5223, 2020.
[53] Warren Morningstar, Cusuh Ham, Andrew Gallagher, Balaji Lakkshmnarayaaaan Alex Alemi, and Joshua Dillon. Density of states estimation for out of distribution detection. International Conference on Artificial Intelligence and Statistics, pages 3232-3240. PMLR, 2021.
[54] Eric Nalismick, Akihiro Matsukawa, Yee whye Tch, and Balaji Lakshminarayanan. Detecting out-of- distribution inputs to deep generative models using a test for typicality. arXiv preprint arXiv: 10Kx::90660222 1906,9299, 5:5, 2019.
[55] Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogencous treatment effects. Biometrika, 2020.
[56] Andre GC Pacheco, Chandramouli S Sastry. Thomas Trappenberg, Sageev Oore, and Renato A Krohling. On out-of-distriuttion detection algorithms with deep neural skin cancer classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 732-733, 2020.
[57] Nicolas Papemot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable private learning with pate. In International Conference on Learning Representations, 2018.
[58] Igor M. Quintanilha, Roberto de M. F. Filho, José Lezama, Mauricio Delbracio, and I Leonardo O. Nunes. Detecting out-of-distribution samples using low-order deep features statistics. 2018
[59] Manish Raghavan, Solon Barocas, Jon Kleinberg, and Karen Levy. Mitigating bias in algorithmic hiring: evaluating claims and practices. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 469-481, 2020.
[60] Jie Ren, Peter Liu, Emily Fertig, Jasper Snock, Ryan Poplin, Mark A DePristo, Joshua V Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distriuution detection. arXiv preprint arXiv: I 1906.02845. 2019.
[61] Bruno M Rocha, Dimitris Filos, Luis Mendes, Gorkem Serbes, Sezer Ulukaya, Yasemin P Kahya, Nikša Jakovljevie, Tatjana L Turukalo, Ioannis M Vogiatzis, Eleni Perantoni, et al. An open access database for the evaluation of respiratory sound classification algorithms. Physiological measurement, 40(30:03500,, 2019.
[62] R. Rosenthal, H. Rosenthal, and inc Sage Publications. Meta-Analytic Procedures for Social Research. Applied Social Research Methods. SAGE Publications, 1991.
[63] Abbijit Guha Roy, Jie Ren, Shekoofeh Azizi, Aaron Loh, Vivek Natarajan, Basil Mustafa, Nick Pawlowski, Jan Freyberg, Yuan Liu, Zach Beaver, et al. Does your dermatology classifier know what it doesn't 't know? detecting the doig of unseen conditions. arXiv preprini 1rXi::2104.03829, 2021.
[64]
Chandramouli Shama Sastry and Sageev Oore. Detecting In ICML 2020: 37th International Conference on Machine Learning, volume 1, pages 8491-8500 2020.
examples with gram matrices.
[65] Patrick Schoettker, Jean Degott, Gregory Hofmann, Martin Proença, Guillaume Bonnier, Alia Lemkaddem. Mathieu Lemay. Raoul Schorer, Urvan Christen, Jean- Françora Knebel, et al. Blood pressure measurements with the optibp smartphone app validated against reference auscultatory measurements. Scientific Reports. 10(1):1-12, 2020.
13

---

[66] Solveig K Sieberts, Jennifer Schaff. Marlena Duda, Bálint Ármin Pataki, Ming Sun, Phil Snyder, Jean- Francois Daneault, Federico Parisi, Gianluca Costante, Udi Rubin, et al. Crowdsourcing digital health measures to predict parkinson's 's disease severity: the parkinson's 's disease digital biomarker dream challenge. NPJ digital medicine, 4(1):1-12, 2021.
[67] Victor Strecher and Irwin Rosenstock. The health belief model. Cambridge handbook of psychology. health and medicine, 113:117, 1997.
[68] Chang Su, Zhenxing Xu, Jyotishman Pathak, and Fei Wang. Deep learning in mental health outcome research: a scoping review, Translational Psychiatry, 10(1):1-26 2020.
[69] Adarsh Subbaswamy and Suchi Saria. From development to deployment: dataset shift. causality, and shift-stable models in health ai. Biostatistics, 21(2):345-352. 2020.
[70] Jayaraman Thiagarajan, Prasanna Sattigeri, Deepta Rajan, and Bindya Venkatesh. Calibrating healthcare ai: Towards reliable and interpretable deep predictive models. arXiv preprint arXiv:2004. 11200004444400 14480, 2020.
[71] Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). In ICLR 2021: The Ninth International Conference on Learning Reprexentations, 2021.
[72] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian J. Goodfellow, Dan Boneh, and Patrick D. McDaniel. Ensemble adversarial training: Attacks and defenses. In 6th International Conference on Learning Representations, ICLR 2018, 2018.
[73] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 9, 2018.
[74] Abinav Ravi Venkatakrishnan, Seong Tae Kim, Rami Eisawy, Franz Pfister, and Nassir Navab. Self- supervised out-of-distribttion detection in brain et scans. arXiv preprint arXiv:2011.05423 2020.
[75] Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics, pages 196-202. Springer, 1992.
[76] Hanrui Zhang, Kaiwen Deng, Hongyang Li, Roger L Albin, and Yuanfang Guan. Deep learning identifies digital biomarkers for self-reported parkinson's 's disease. Patterns, 1()): 2020.
[77] Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In European Conference on Computer Vision, pages 102-117. Springer, 2020.
[78] Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International Conference on Machine Learning, pages 819-827. PMLR, 2013.
14


Reviews:
[{"review_id": "wFixd0GWqZ", "reviewer": "Reviewer_hRAn", "summary": "The authors provide a small scale study of out of distribution detection methods on three medical datasets and perform a survey of Turkers to see how these OODs impact trustworthiness and medical decision making. They find that Gram and Manhabolis based methods detect OOD data well and that these scores translate into increased trustworthiness and medical decision making in their survey.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Main comments:\nI would like to commend the author's for taking the time to conduct a survey to see how their experimental results translate into user perception and behavior. A very rare thing for computational papers! \n\nIn addition to my detail comments below, my primary concern about this paper stems from the scope of its evaluations. 3 datasets and 3 methods is good, but still modest by most standards. Furthermore, the \"difficulty\" of the OOD scenarios tested is unlikely to be reflective of real clinical scenarios. Realistic OOD in healthcare is partially captured by some of the examples, but a practitioner in this space will likely still have many questions about which if any OOD detection methods will work in practice. Much more subtle forms of distributional shift occur in practice and inclusion of these would strengthen the conclusions.\n\nI also think that the author's should more formally define what they mean by OOD, i.e. are they testing covariate shift, label shift, or something else? Providing this additional specificity will help users know if the authors' results are applicable to their wrork.\n\n### Detailed comments:\n\n__Originality__: The work is original and includes both a small computational evaluation and user survey.\n\n__Quality__:  The experimental sections are sound and the survey is as well done, if a bit small and artificial. It would be great to include the distribution of softmax probabilities on the OOD data. I am curious to know if the models are extremely confident on data that has nothing to do with their training task. \n\n- I do not believe _CONFIDENCE SCORE_ is ever formally define. Please include this definition.\n\n__Clarity__: The work is clearly presented and I found text and experiments easy to follow.\n\n__Significance__: The work looks at a very important topic that is understudied. Further experiments would increase the significance of this work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. This research involved human subjects and the authors state they received appropriate IRB approval for it.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "M1zSdG37Q0e", "reviewer": "Reviewer_hXoK", "summary": "This paper touches on a very interesting and important problem in the applied machine learning field, the trustworthiness of ML models, how to measure it, and its effect on user experience. The paper presents a very nice introduction to the topic and opens the problem in the context of out-of-distribution detection. Three different approaches for out-of-distribution detection are benchmarked on several datasets and ML methods. Furthermore, a confidence score is proposed to translate the out-of-distribution scores into more interpretable scores for the users. A user study is further conducted to evaluate the effect of using confidence scores on the trust level of human participants.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper discusses a significant problem in the field and the text is very clear. I have few concerns regarding the experimental setups and results:\n1) It seems from the text that the user study was performed on random participants. In order to truly measure the effect of using the confidence score on the perceived trustworthiness, the experiments must be performed on expert participants (as true users of the ML model) with domain knowledge about the experimental data. \n2) It is not clear from the text, how the confidence score is computed.\n3) The authors may also discuss the advantage of their proposed confidence score with commonly used metrics in the field such as uncertainty measures.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "APqNdnoxu5E", "reviewer": "Reviewer_QZBK", "summary": "This work investigates the effect of out-of-distribution data on machine learning models on healthcare applications. The authors first display a decrease in model performance on near and far out-of-distribution samples for 3 health applications. They then assess the performance of 3 techniques for out-of-distribution detection and build a confidence score from this binary classifier. The confidence score is assessed in a user-study as a complementary piece of information to be presented to the user.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n---------------\nOut-of-distribution detection is not a novel problem, and technically the methods proposed are not new. The authors mention that their application to healthcare data makes this piece different, however I found no evidence that the considered healthcare applications affected the techniques or their use. Similarly, a decrease of model performance on ood data has been highlighted in several works (see works referring to 'robustness' to e.g. new diseases, time shifts or local perturbations). On the other hand, user studies are rarely conducted when claiming better ood detection methods. I would therefore encourage the authors to highlight the user-study more.\n\nQuality\n----------\nThe work is overall well executed and convicing. I have a couple of questions for the authors that would help me to better assess the impact, especially of the user study:\n- In the Appendix, the questions refer to the user's trust in the system, and whether they would see a doctor if given this diagnosis. Is the assumption that there is a deterministic relationship between the model's prediction of e.g. malignancy with whether they would consult a doctor? Can a decision be factorized into 2 components: how the user/patient combines the model's prediction (+ confidence score) with their own priors to make a diagnosis themselves, and whether they decide to see a doctor based on this diagnosis? Or is it assumed that the model is only informational and that the user/patient task is to decide whether they see a doctor or not? Was there any investigation into which mental model of the classifier the user were referring to? If the former, it could have been interesting to add a question about what their diagnosis was. I understand this is probably a broader issue about the perception of a classifier as a diagnosis vs referral vs informational tool, but I was wondering whether the authors had any thoughts?\n- On the increased user trust: Is the additional information increasing trust, independently of its quality? One way to test this would be to provide randomized confidence scores to images, to understand whether the added trust comes from the quality of the score and the correct understanding of the user, or whether it comes from a perception of increased information.\n\nClarity\n---------\nThe paper is well written and easy to follow.\n\nSignificance\n-----------------\nI believe this is a nice application of existing techniques, well executed. I think the most interesting part is the user study, and I think it could be improved to have even more impact (see questions above).\n\n\nReferences:\n- Does Your Dermatology Classifier Know What It Doesn’t Know? Detecting the Long-Tail of Unseen Conditions https://arxiv.org/pdf/2104.03829.pdf\n- Feature Robustness in Non-stationary Health Records, http://proceedings.mlr.press/v106/nestor19a/nestor19a.pdf", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors acknowledge the 'theoretical' setup of the healthcare applications. This remains a limitation, as it is unclear whether predictions in themselves are useful in clinical settings. However, I think that the proposed framework has merits in terms of ood detection and transparency with the user, that can lead to more realistic studies.\n\nI would encourage the authors to highlight a bit more the user study and discuss its limitations.", "review_text": "", "extra_scores": {"ethical_concerns": "Out of curiosity: is $3 for 15 minutes a normal pay? 12$/ hour seems to be below the minimum pay in the US (at least in some states), and is low for European standards as well.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "6NAHsyO_BB", "reviewer": "Reviewer_Ku9s", "summary": "This work applies known out-of-distribution (OOD) detection techniques on various medical datasets and confirms that the OOD detection techniques indeed works. Specifically, the medical prediction models tend to make more incorrect predictions for samples that OOD detection algorithms deem OOD.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n- This paper focuses on an important problem; the reliability of prediction models in healthcare. They specifically employ OOD detection methods to test if they can help increase the trustworthyness of prediction models.\n- The authors conduct extensive experiments using medical datasets of multiple modalities (e.g. images, audio) and multiple conditions (e.g. skin cancer, lung sound) to provide reliable conclusion\n\nCons\n- The focus of this work is to study whether the existing OOD detection methods can work in various medical datasets. Therefore there is limited methodological novelty, which is the main interest of the NeurIPS community.\n\nSummary\n- The work itself is valuable, but probably not in NeurIPS. This work would be much more appreciated in relevant venues just such as MLHC, CHIL or JAMIA, JMIR.\n\nPost Rebuttal\n- I appreciate the authors taking the time to describe their thoughts in detail. However, I do not belive that simply applying well-known OOD detection methods on publicly available datasets justifies an acceptance to this venue, as that is not the main interest of this community. Therefore I'll keep my initial rating. (The user study is an interesting aspect, and will be of huge relevance to the healthcare community)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks to the authors for an interesting and thorough study of OOD methods in the healthcare context. 

The reviewers all agreed that this work was well done and interesting, but the main concern is relevance for the NeurIPS community.  While I agree with reviewer Ku9s that the main interest of the this community is in ML methodology and theory, I do believe it is important to investigate real use cases of methods --- two of which were published recently in NeurIPS, and one in ICML.  MLHC, CHIL, JAMIA, and JMIR may also be appropriate venues, but the NeurIPS/ICML community developed these methods, and are an important audience for this kind of evaluation.  I tend to agree with the authors that their work falls squarely within the "Social Aspects of Machine Learning" (and "Applications"), so I am not as concerned about the relevance of this work for the ML community. 

Beyond the relevance concern, all reviewers were quite positive about the work itself.  Reviewer hRAn detailed some concerns about definitions (OOD, confidence score), which should be clarified in the main text.  Reviewer QZBK also describes an improvement to the user study, which should be incorporated into the discussion. 



Decision: Accept (Poster)