Analyzing the Role of Model Uncertainty for Electronic Health Records
Authors: Michael W. Dusenberry
Dustin Tran
Edward Choi
Jonas Kemp
Jeremy Nixon
Ghassen Jerfel
Katherine Heller
Andrew M. Dai
Venue:  (2020)
Keywords: medicine
uncertainty
neural networks
Bayesian
electronic health records

Abstract: In medicine, both ethical and monetary costs of incorrect predictions can be significant, and the complexity of the problems often necessitates increasingly complex models. Recent work has shown that changing just the random seed is enough for otherwise well-tuned deep neural networks to vary in their individual predicted probabilities. In light of this, we investigate the role of model uncertainty methods in the medical domain. Using RNN ensembles and various Bayesian RNNs, we show that population-level metrics, such as AUC-PR, AUC-ROC, log-likelihood, and calibration error, do not capture model uncertainty. Meanwhile, the presence of significant variability in patient-specific predictions and optimal decisions motivates the need for capturing model uncertainty. Understanding the uncertainty for individual patients is an area with clear clinical impact, such as determining when a model decision is likely to be brittle. We further show that RNNs with only Bayesian embeddings can be a more efficient way to capture model uncertainty compared to ensembles, and we analyze how model uncertainty is impacted across individual input features and patient subgroups.

medicine specifically, individualized Under review as a conference paper at ICLR 2020
ANALYZING THE ROLE OF MODEL UNCERTAINTY FOR ELECTRONIC HEALTH RECORDS
Anonymous authors Paper under double-blind review
ABSTRACT
In medicine, both ethical and monetary costs of incorrect predictions can be signifi- cant, and the complexity of the problems often necessitates increasingly complex models. Recent work has shown that changing just the random seed is enough for otherwise well-tuned deep neural networks to vary in their individual predicted probabilities. In light of this, we investigate the role of model uncertainty methods in the medical domain. Using recurrent neural network (RNN) ensembles and various Bayesian RNNs, we show that population-level metrics, such as AUC-PR, AUC-ROC, log-iikelihood and calibration error, do not capture model uncertainty. Meanwhile, the presence of significant variability in patient-specific predictions and optimal decisions motivates the need for capturing model uncertainty. Under- standing the uncertainty for individual patients is an area with clear clinical impact, such as determining when a model decision likely to be brittle. We further show that RNNs with only Bayesian embeddings can be a more efficient way to capture model uncertainty compared to ensembles, and we analyze how model uncertainty is impacted across individual input features and patient subgroups.
INTRODUCTION
Machine learning has found great and increasing levels of success in the last several years on many well-known benchmark datasets. This has led to mounting interest in non-traditions problems and domains, each of which bring their own requirements. predictions are of great importance to the field (Council et al., 2011). and there can be severe costs for incorrect predictions and decisions due to the risk human life and the associated ethical concerns Gillon 1994).
Existing state-of-the-art approaches using deep neural networks in medicine often make use of either single model or an average over small ensemble of models, focusing on improving the accuracy of probabilistic predictions (Harutyunyan et al. 2017 Rajkomar et al., 2018b Xu et al. 2018 Choi al. 2018). These works, while focusing capturing the data uncertainty, do not address the model uncertainty that is inherent in fitting deep neural networks. For example, when predicting patient mortality in an ICU setting, existing approaches might be able to achieve high AUC-ROC, but will be unable to differentiate between patients for whom the model is certain about its probabilistic prediction, and those for whom the model is fairly uncertain.
In this paper, we examine the use of model uncertainty specifically in the context of predictive medicine. Model uncertainty has made many methodological advances in recent years including
iiaraamteerzzatio--aasde variational Bayesian neural networks (Blundell et al. 2015 Kucukelbir
al. 2017 Louizos & Welling (017). Monte Carlo dropout (Gal & Ghahramani 2016)) ensembles
[Lakshminarayanan et al. 3).7), and function priors (Hafner et al. [0188 8 Garnelo et al.) 2018
Malinin Gales| (218)) Deep neural networks combined with advanced model uncertainty methods
can directly impact clinical care by answering several questions that naturally occur in predictive medicine:
How do the realized functions in any of the approaches, such as individual models in the ensemble approach, compare in terms of population-level metric performance such as AUC-PR, AUC-ROC, or log-likelihood?
If and how does model uncertainty assist in calibrating predictions?

---

Under review as a conference paper at ICLR 2020
How does model uncertainty change across different patient subgroups, in terms of ethnicity, gender, age, or length of stay?
How do various feature values contribute towards model uncertainty?
How does model uncertainty affect optimal decisions made under given clinically-relevant cost function?
Using sequence models on the MIMIC-III clinical dataset (Johnson et al. 2016). make several important findings. For the ensembling approach of quantifying model uncertainty, we find that the models within the ensemble can collectively exhibit a wide variability in predicted probabilities for some patients, despite being well-calibrated and having nearly identical dataset-level metric performance. We find that this even extends into the space of optimal decisions. That is, models with nearly equivalent metric performance can disagree significantly on the final decision, thus transforming an "optimal" decision into a random variable. Significant variability in patient-specific predictions and decisions can be an indicator of when model decision is likely to be brittle, and we show that using single model or average over models can mask this information. This motivates the importance of model uncertainty for clinical decision systems. Given this, we proceed with an analysis over different clinical tasks and datasets, looking how model uncertainty is impacted across individual input features and patient subgroups. We then show that models with Bayesian embeddings can be more efficient way to capture model uncertainty compared to deep ensembles.
Contributions
2 BACKGROUND
Data uncertainty Data uncertainty can be viewed as uncertainty regarding a given outcome due to incomplete information, and is also known "output uncertainty" or "risk" (Knight (957). For binary tasks, this equates to single probability value. More specifically, this can be described as Bernoulli(A), A f(x, w),
where the model f, as a function of the inputs x and parameters w, outputs the parameter for Bernoulli distribution representing the conditional distribution p(y|x, w) for the outcome y.
Model uncertainty Model uncertainty can be viewed as uncertainty the correct values of the parameters for the predictive outcome distribution due to lack of knowledge of the true function. For binary tasks, this equates a distribution of plausible probability values for a Bernoulli distribution, corresponding to set of plausible functions. More specifically, this can be described as
(2) where there is an induced distribution over the Bernoulli parameter À for a given example that represents uncertainty in the true outcome distribution due to uncertainty in function space. For the remainder of the paper, we will use the phrase predictive uncertainty distribution to refer to this distribution over the parameter of the outcome distribution.
Bernoull(A)): À ~ >>>p(A],,, w),
Calibration A model is said to be perfectly calibrated if, for all examples for which the model produces the same prediction p for some outcome, the percentage of those examples truly associated with the outcome is equal to p. across all values of p. If a model is systematically over- or under- confident, can be difficult to reliably use its predicted probabilities for decision making. The expected calibration error (ECE) metric (Nacini et al. 2015) is one tractable way to approximate the calibration of model given a finite dataset.
Deep Ensembles Deep ensembles (Lakshminarayanan et al. 2017) is a method for quantifying model uncertainty. In this approach, an ensemble of M deterministic neural networks (NNs) trained by varying only the random seed of an otherwise well-tuned set of hyperparameters. Given this ensemble, predictions can be made with each model m for a given example where (for a binary
task) each prediction is the probability parameter (i) for the Bernoulli distribution over the outcome.
The set of M probabilistic predictions (A(2) >x(²) "(.)...,,, (4) for the same example then be viewed the distribution p(A (X||x, x, w) over (6), where this distribution represents model uncertainty.
We use the term "deterministie" to refer to the usual setup in which optimize the parameter values of our function directly, yielding a trained model with fixed parameter values time.
2

---

Under review as a conference paper at ICLR 2020
Regairre Peelive
12
doviatton
0.33
6-
0.64
0.0
2-
52
0. 0.2
mcan
0.5
0.5
1.0
Bernoull probability parameter
Figure 2: A plot of the versus standard deviation of the predictive uncertainty distribu- tions of the deterministic ensemble for positive and negative patients in the validation set. We find that the standard deviations do not form simple linear relationship with the mean. For reference, we note that the variances of the distributions are generally lower than that of a Bernoull distribution's variance curve.
Figure 1: A histogram of predictions from M deterministic RNN models trained with differ- ent random seeds for given ICU patient's mor- tality probability. As shown here, model un- certainty cause a high-vass sigh-variance predictive distribution across individual models in an en- semble. This is not captured when using sin- gle model or an average over an ensemble.
Bayesian recurrent neural networks (RNNs) (Fortunato et al. 2017) are RNNs with prior distribution placed over the parameters. This allows to express model uncertainty as uncertainty over the true values for the parameters in the model, i.c., "weight uncertainty" (Blundell al. 2015). By introducing distribution over all, subset, of the weights in the model, we can induce different functions, and thus different outcomes, through realizations of different weight values via draws from the posterior distributions. This allows us to empirically capture model uncertainty in the predictive uncertainty distribution p( p()|x, w) by drawing M samples from single Bayesian RNN for given example.
Bayesian RNNs
3 MEDICAL UNCERTAINTY
3.1 CLINICAL TASKS
We first show results using Medical Information Mart for Intensive Care (MIMIC-III) (Johnson et al. 2016), publicly available EHR dataset, in terms of two tasks: in in-patient mortality prediction, and multiclass diagnosis prediction discharge, where diagnosis codes represented by the single-level Clinical Classifications Software (CCS) codes. Similar to RRajkomar et al.) (201 (2018)), we train deep RNNs on MIMIC-III, which is collected from 46,520 patients admitted intensive care units (ICUs) at Beth Israel Deaconess Medical Center, where 9,974 expired during the encounter (i.e., 1:4 ratio between positive and negative samples). In order demonstrate that our findings generalize, we additionally experiment with eICU Collaborative Research Database (eICU) (Pollard et al. 2018). another publicly available EHR dataset, in terms of in- in-patient mortality prediction. Our model embeds and aggregates patient's time-series features (e.g. medications, lab measures) and global features (e.g. gender, age), feeds them to one or more long short-term memory (LSTM) layers (Schmidhuber & Hochreiter 1997). followed by hidden and output affine layers. See the Appendix for more details.
Existing deep learning approaches in predictive medicine focus on capturing data uncertainty, namely accurately predicting the risk X f(x, w) of a patient's 's mortality (i.e., how likely is the patient to expire?). This work, on the other hand, also focuses on addressing the model uncertainty aspect of deep learning, namely the distribution À p() x. w) over the risk of mortality for a patient (i.e., there alternative risk predictions, and if so, how diverse are they?).
3.2 CHOICE OF UNCERTAINTY METHONS
To quantify model uncertainty for clinical tasks, explore the of deep RNN ensembles and various Bayesian RNNs. For the deep ensembles approach, we optimize for the ideal hyperparameter values for RNN model via black-box Bayesian optimization (Golovin al. 2017), and then
3

---

Under review as a conference paper at ICLR 2020
train M replicas of the best model. Only the random seed differs between the replicas. At prediction time, we make predictions with all M models for each patient. For the Bayesian RNNs, we train single model, and then draw M samples from it at prediction time. To train the Bayesian RNN, we take a variational inference approach by adapting our RNNs to use factorized weight posteriors q( w distributions with learnable mean and diagonal covariance parameters represented as 0,. Normal distributions with zero mean and tunable standard deviation used as weight priors. We train our models by minimizing the Kullback-Leibe (KL) divergence
II: 7( w:)O:). where weight tensors W; in the models represented by normal
((
KL[g (w
p w X)]
KL w 0) pl [w]] - Bg( [np(y|XX w)]
between the approximate weight posterior and the true, but unknown posterior, which overall equates to minimizing an expectation over the usual negative log likelihood term plus KL. regularization term. To easily shift between the deterministic and Bayesian models, we make use of the Bayesian Layers (Tran et al. 2018) abstractions.
3.3 OPTIMAL DECISIONS VIA SENSITIVITY REQUIREMENTS
The key desire in clinical practice is to make a decision based on the model's predicted probability and its associated uncertainty. Given set of potential outcomes yk. a set of conditional probabilities p()]|x) for the given outcomes, and the associated costs Laj for either correctly or incorrectly predicting the outcome, an optimal decision can be determined by minimizing the expected decision loss
J B[L] -リ=ΣΣ!
(xsp(yx(x)p(x) dx,
(4)
where R, is the decision region for assigning example x to class j. and p(x) is the density of x (Bishopl 2006).
Designing elaborate decision cost functions for clinical applications is not an easy task as it requires expert knowledge the prediction target, cost-benefit analysis, and medical resource allocation. Fortunately we can use clinically relevant alternative, which is the sensitivity requirement. Often in clinical research, certain sensitivity (i.e., recall) levels must be met when making predictions in order for model to be clinically relevant. The goal in such cases is to maximize the precision while still maintaining the required sensitivity level. Viewed as a decision cost function, the cost is infinite if the recall is below the target level, and is otherwise minimized as the precision is increased, where the optimized parameter is the global probability threshold.
For each of the M models in our ensemble, we can optimize the sensitivity-based decision cost function and make optimal decisions for examples. Thus, for each example, there will be a set of M optimal decisions, forming a distribution. The optimal decision d then becomes random variable
d ~ Bernoulli())
Σ AEA
1(1 2 t)),
(5)
M
where @ is the percentage of the set = (A1,A2,.... dg....... Av) of M probability values for given example that greater than equal to the optimized decision threshold t.
4 EXPERIMENTS
We perform four sets of experiments. First, in order to demonstrate the importance of quantifying uncertainty in predictive medicine, we examine individual models in the RNN ensemble terms of prediction metrics, calibration, uncertainty distributions, and decision-making. Second, we examine multiple variants of Bayesian RNNs to understand where uncertainty the model matters most, comparing them with the deterministic ensemble counterpart. Third, use the deterministic RNN ensemble to examine uncertainty across different patient subgroups. Finally, we analyze the Bayesian RNN with embedding distributions to examine uncertainty across individual features.
4

---

Under review as a conference paper at ICLR 2020
Table 1: Dataset-level metrics for the MIMIC-III binary mortality and multiclass CCS prediction tasks across M 200 models in the deterministic RNN ensemble. Individual models are nearly identical in terms of dataset-level performance across both tasks, but selecting a single model would remove the model uncertainty information such as that visualized in Figure
TASK
METRIC
VALIDATION
TEST
MORTALITY
AUC-PR AUC-ROC
0.4496 (0.0025)
0.3886 (0.0059)
0.8753 (0.0019)
0.8623 (0,0031)
ECE
0.0176 (0.0040)
0.0162 (0.0043)
ACE
0.0210 (0.0042)
0.0233 (0.0057)
CCS
TOP-5 RECALL
0.7126 (0.0071)
0.7090 (0.0088)
TOP-5 5 PRECISION
0.1425 (0.0014)
0.1418 (0.0018)
TOP-5
0.2375 (0.0024)
0,2363 (0.0029)
LOG-LIKELIHCOD
-5.1040 (0. (0.0075)
-5.1081 (0.0083)
ECE
(0.0072)
0.0499 (0.0082)
ACE
4.21896-3 (7.31 IE-8)
4.21916-3 (7.61368-8)
COIEE
court
10
50-
ede
Optimal
deditin
Figure 3: Left two: The deterministic ensemble produces different mortality prediction distributions (i.e. model uncertainty) for two different patients from the validation set. Right two: The corre- sponding optimal decision distributions, "True" being the positive prediction for mortality and "False" being the opposite. For some patients the ensemble is relatively certain about the optimal decision, while for other patients there a large amount of uncertainty.
4.1
WHEN Do WE OBSERVE UNCERTAINTY?
Clinical Metries For our clinical tasks, we first measure the dataset-level metrics: area under the precision-recall curve (AUC-PR), area under the receiver operating characteristic curve (AUC-ROC), top-5 recall, top-5 precision, top-5 F1, held-out likelihodooood, ECE (Naeiní al. 2015), and adaptive calibration eтror (ACE) (Nixon et al.) 2019). Table 1 shows the performance averaged over individual models in our deterministic ensemble, with standard deviation the parenthesis. Interestingly, individual models are overall well-calibrated and nearly equivalent in terms of performance. If we were to choose only one model in practice based on the dataset-level metrics, it is highly likely any one could be selected good model, and we would lose the model uncertainty information.
Predictive Uncertainty Distributions & Statistics Knowing that the models in ensemble well-calibrated and effectively equivalent in terms of performance, we turn to making predictions for individual examples. FFigure visuulizes the predictive uncertainty distribution for single patient. We find that there is a wide variability in predicted Bernoulli probabilities for some patients (with spreads high as 57.5%). Ignoring this variance through the use of either a single deterministic model or an average over an ensemble is likely harmful since it prevents this uncertainty from being conveyed physician. Figure visualizes the means versus standard deviations of the predictive uncertainty distributions for the deterministic ensemble all validation set examples, In contrast to the variance of Bernoulli distribution, which is simple function the mean, we find that the standard deviations are patient-specific, and thus cannot be determined a priori.
Optimal Decision Distributions & Statistics In practice, model uncertainty is important as it can affect one's 's decisions. To test this, we optimize the sensitivity (i.e. recall) based decision cost function with respect to the probability threshold for each model in RNN ensemble separately
5

---

Under review as a conference paper at ICLR 2020
Table 2:
Metrics for marginalized predictions on the MIMIC-III and eICU mortality tasks given M 200 models in the deterministic RNN ensemble, and M = 200 samples from each of the Bayesian RNN models. Confidence intervals are computed via validation and test set bootstrapping with 1000 bootstrap sets.
MODEL
VAL. AUC- PR
VAL. AUC- RCC-ROC
VAL. NLL
TEST AUC-PE PR
TEST AUC-ROC
TEST
NLL
DETERMINISTIC
0.4564
0.8774
0.7113
0.3921
0,8643
0.7148
(11c--)
(155e
(17e-5)
(11e-3)
(1
(d:7e-5)
BAYESIAN
0,4580
0.8776
0.7182
0.3977
0.8612
0.7186
EMBEDDINGS
(+1e-3)
(#4e-4)
(#7e-5)
(#2e-3)
(#5e-4)
(#7e-
MIMIC-III
BAYESIAN
0,4382
0.8714
0.7058
0.3702
0.8572
0.708T
OUTPUT
(122==3
(d:5e-4 4)
(#7e-5)
(d:le=3)
(e
(d7e-
BAYESIAN
0.41122
0.8751
0.7118
0.3893
0.8607
0.7149
HIDDEN-OUTRES
(#1e-3)
(#5ee4)
(#7e-5)
(#1e--3
(#5e-4)
(d:7-55)
BAYESIAN
0.4396
0.8673
0.7097
0.3860
(42e-3) 3)
(#5e-4) 4)
(#7e-5)
(#2e-3)
(#5e-4)
(#7e-5)
FULLY
0.4354
0.8092
0,7100
0.3829
0,8552
0.7133
BAYESIAN
(426-3)
(#3e-4)
(#
(#1e-3)
(#5e-4)
(#6e-5)
DETERMINISTIC
0.1951
0.7882
0.1435
0.2196
0. 0.7868
0.2435
ENSEMBLE
(#1e-3)
(#7e-4)
(#3e-4] 4)
(#1e-3)
(#6e-4)
(#5e-4) 4)
BAYESIAN
0.1996
0.7807
0.1455
0.2244
0.7733
0.1620
EMBEDDINGS
(+1e-3)
(#1e-4)
(# (=))))
(#1e-3)
(=Te-4)
(#4e-4)
BAYESIAN
0.1738
0.7677
0.1664
0.1942
0.7580
0.1810
RICU
OUTPUT
(Lle-3)
(#7e-4)
(4)
(dlle-3)
(#7e-4)
(#4m-44
BAYESIAN
0.1712
0.7801
0.1619
0.2140
0.7817
0.1713
HIDDEN-OUTPUT
(#1e-3)
(#7e-4)
(=3e-4)
(#e-)
(#6e-4)
(#3e-4)
BAYESIAN
0.1675
0.7791
0.1477
0.2147
0.7809
0.1583
RNN-HIDDE+OOUTOU
(+1e-3)
(#7e-1)
(#3e-1)
(#1e-3)
(#7e-4)
(#3e-1)
FULLY
0.2004
0.7910
0.1377
0,2280
0.7818
0.1541
BAYESIAN
(11e-3)
(17e-4)
(13c-4)
(11e-3)
(17e-4)
(14c-4)
to achieve a recall of 70%, and then make optimal decisions for each example with each of the M models. Figuure 3] visualizes how model uncertainty in probability space is realized in optimal decision space for two patients. We see that the model uncertainty does indeed extend into the optimal decision space, converting the optimal decision into random variable. Furthermore, the decision distribution's 's variance can be quite high, and knowing when this is the case is important in order to avoid the cost of any incorrect decisions made by the system due to lack of precise knowledge about the correct level of data uncertainty.
4.2 COMPARISON: VARIANTS OF BAYESIAN RNNS AND DETERMINISTIC RNN ENSEMBLES
natural question in practice when employing the Bayesian approach is which part of the model should capture model uncertainty. To do so, study Bayesian RNNs under variety of priors:
Bayesian Embeddings A Bayesian RNN in which the embedding parameters are stochastic, and all other parameters are deterministic.
Bayesian Output A Bayesian RNN in which the output layer parameters are stochastic, and all other parameters are deterministic.
Bayesian Hidden+Output A Bayesian RNN in which the hidden and output layer parame- ters stochastic, and all other parameters deterministic.
Bayesian RNN+Fiiden++ A Bayesian RNN in which the LSTM, hidden, and output layer parameters are stochastic, and all other parameters are deterministic.
Fully Bayesian A Bayesian RNN which all parameters are stochastic.
Table 2 displays the metrics over marginalized predictions for each of the Bayesian RNN models and the deterministic RNN ensemble on the MIMIC-III and eICU mortality tasks. We find that the Bayesian Embeddings RNN model outperforms all other Bayesian variants and slightly outperforms the deterministic ensemble in terms AUC-PR for MIMIC-III, and that the fully-Bayesian RNN outperforms the other models on the eICU dataset. Additionally. all of the Bayesian variants either comparable or outperform the deterministic ensemble in terms of held-out log- likelihoodd on both datasets. Since our Bayesian models achieve strong performance while only requiring training

---

Under review as a conference paper at ICLR 2020
Deterministic
Determinisüi Bayeslan
Deterministic Bayesian
Deterministi
Bayesian
30-
25
25
25
20
20
20
15
15
15
15-
10
10
10.
5
5
5
5
0
0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
10
Figure 4: Predictive uncertainty distributions of both the RNN with Bayesian embeddings and the deterministic RNN ensemble for individual patients. We find that the Bayesian model is qualitatively able to capture model uncertainty that is quite similar to that of the ensemble with considerably smaller number of parameters
of single model (7.22 million parameters in the MIMIC-III Bayesian Embeddings RNN), versus M models in the deterministic ensemble (200 x 6.16 million parameters), as well as only requiring the single model at prediction time, using Bayesian RNNs can be a more efficient approach.
Figure 4] visualizes the predictive distributions of both the Bayesian RNN and the deterministic RNN ensemble for four individual patients. We find that the Bayesian model qualitatively able to capture model uncertainty that is quite similar to that of the deterministic ensemble. Overall, the Bayesian Embeddings RNN, compared the deterministic ensemble, demonstrated slightly better predictive performance and qualitatively similar uncertainty performance, with considerably less computational resources, making an attractive choice in clinical practice.
4.3 PATIENT SUBGROUP ANALYSIS
We next turn to an exploration of the effects of model uncertainty across patient subgroups. We split validation encounters into subgroups by demographic characteristics, namely patient gender (3089 male vs. 2548 female) or age (adults divided into quartiles of 1216, with a separate fifth group of 773 neonates). For this analysis, focus the deterministic RNN ensemble described in Section 4.1, as the Bayesian models sample M = 200 weights separately for each prediction rather than globally for the complete validation set. For each model in the ensemble, we compute validation set performance metrics separately over each subgroup and then compute the correlation between these metrics over all models in the ensemble, specialize to one or more subgroups at the cost of performance on others. We find some evidence of this phenomenon: for example, AUC-PR for male patients negatively correlated with AUC-PR for female patients (Pearson's 's -0.442, see Figure 5). and AUC-PR for the oldest quartile of adult patients is somewhat negatively correlated with AUC-PR for other adults or for neonates (Pearson's 's between 18.18 and -0.37).
evaluate whether the ensemble models tend to
We also compare uncertainty metrics across subgroups, including standard deviation and range the predictive uncertainty distributions and variance of the optimal decision distributions for patients in each subgroup. For this analysis, we examine both the deterministic ensemble and the best Bayesian model, the RNN using Bayesian embeddings. In both cases, find that all metrics correlated with subgroup label prevalence: both uncertainty and mortality rate increase monotonically across age groups (Figure 5). and both are slightly higher in women than in men. These findings imply that random model variation during training may actually cause unintentional harm to certain patient populations, which may not be reflected in aggregate performance.
4.4 EMBEDDING UNCERTAINTY ANALYSIS
Another motivation for model uncertainty lies in understanding which feature values are most responsible for the variance of the predictive uncertainty distribution. Our RNN with Bayesian embeddings model is particularly well suited for this task in that the uncertainty in embedding space directly corresponds to the predictive uncertainty distribution and represents uncertainty associated with the discrete feature values. Understanding model uncertainty associated with features can allow recognize particularly difficult examples and understand which feature values leading to
7

---

Under review as a conference paper at ICLR 2020
0.49
0.25
Age group
Infant
0.48
Adult
Adult Q4
0.15
AUC PR. female
0.08
0.43
0.46 0.00
0.64 0.45
Dacision variance
Prediction SD Uncartainty
Prodiction
AUC-PR,
Figure Left: Model performance comparison on male vs. female patients. Each point represents stratified AUC-PR for single model from the deterministic ensemble. Correlation coefficient r = -0.422. Right: Summary of uncertainty measures within each age subgroup, using the Bayesian embeddings RNN. On all measures, uncertainty increases monotonically with age. This corresponds to an increase in mortality rate with age, as positive cases are more uncertain on average.
Table 3: Top and bottom 10 words in free-text clinical notes based their associated Bayesian embeddings distribution's s entropy, along with their frequency in the training dataset.
LOWEST ENTROPY ENTROPY
HIGHEST ENTROPY
WORD
FREQUENCY
WORD
ENTROPY
FREQUENCY
THE
-82.5445
41803
24PM
-16.0790
336
-80.6055
42812
LABWORK
-16.0750
272
-80.2735
43191
COLONIAL
-16.0690
198
NO
-79.8994
43420
ZOYSN
166.6001
269
TRACING
-78,5988
32181
HT
-16.0523
515
IS
-78.5553
42560
TXCE
15.9982
112
-77.6408
42365
ANGEMEENS
-15.9795
407
FOR
-76.8005
42972
PARVUS
-15.9773
132
WITH -75.3513
42819
NAS
-15.9164
251
IN
-72.8006
42144
ANESTHESIOLOGIST
-15.8796
220
the difficulties. Additionally, it provides a means of determining the types of examples that could be beneficial to add to the training dataset for future updates to the model.
For this analysis, we focus on the free-text clinical notes found in the EHR. For each word in the notes vocabulary, have associated embeddings distribution formulated
multivariate Normal. We rank each word by its level of model uncertainty (measured by its embedding distribution' 's entropy). Table 3 lists the top and bottom 10 words, along with their frequency in the training dataset. We find that common words, both subjectively and based on prevalence counts, have low entropy and thus limited model uncertainty, while rarer words have higher levels of model uncertainty.
DISCUSSION
In this work, demonstrated the need for capturing model uncertainty in medicine and examined methods to do so. Our experiments showed multiple findings. For example, an ensemble of deterministic RNNs captures individualized uncertainty conditioned on each patient, while the models each maintained nearly equivalent rinieally-relevant dataset-level metrics. As another example, we found that models need only be uncertain around the embeddings for competitive performance, with the benefit of also enabling the ability to determine the level of model uncertainty associated with individual feature values. Furthermore, using model uncertainty methods, we examined patterns in uncertainty across patient subgroups, showing that models can exhibit higher levels of uncertainty for certain groups
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 265-283, 2016.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, New York, corrected 8th printing 2009 edition, 2006. ISBN 978-0-387-31073- 2.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight Uncertainty in Neural Networks, arXiv.or, May 2015.
Edward Choi, Cao Xiao, Walter Stewart, and Jimeng Sun. Mime: Multilevel medical embedding of electronic health records for predictive healtheare. In Advances in Neural Information Processing Systems, pp. 4547-4557, 2018.
National Research Council et al. Toward precision medicine: building a knowledge network for biomedical research and a new taxonomy of disease. National Academies Press, 2011.
Meire Fortunato, Charles Blundell, and Oriol Vinyals. Bayesian Recurrent Neural Networks. arXiv.org, April 2017. URL httt::t/p//xi...../ wwtp://///p////...ooo//ooo 02700//990027
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv: 107.00180. 2018.
Raanan Gillon. Medical ethics: four principles plus attention to scope. Bmj, 309(6948):884, 1994.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley, Google Vizier: Service for Black-Box Optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining KDD '17. pp. 1487-1495, Halifax, NS, Canada, 2017. ACM Press. ISBN 978 48874-4887--4 3098043. URL https: ai wtwp:////eesaaae///////////ii/ia
doi: 14/1149/3097983 pdE
Danijar Hafner, Dustin Tran, Alex Irpan, Timothy Lillicrap, and James Davidson. Reliable uncertainty estimates in deep neural networks using noise contrastive priors. arXiv preprint 107.v:180,,09289, 2018.
Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data, arXiv preprint arXiv: 077777770..7. 2017.
Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng. Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. MIMIC-III, a freely accessible critical care database. Scientific Data, 3:160035, May 2016. ISSN 2052-4463, doi: 10. 20.1088/sdata.211635.. RaLhhtp:/// www. nature
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1eXiv:1412.660 2014.
Frank H. Knight Risk, Uncertainty and Profit. New York, Kelley & Millman, 1957. URL hattps://mises htt.s//s..orssstteeeeeeaaa/////a #20anc$20Profit__.pdd
$20Uncertainty. hagp://aeeeaiii....222eoeeaaaaa,
Alp Kucukelbir, Dustin Tran, Rajesh Ranganath, Andrew Gelman, and David M Blei. Automatic differentiation variational inference. The Journal of Machine Learning Research, 18(1):430-474 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In Advances in Neural Information Processing Sys- fems, volume MI. December 2017. URL _ hwt.l:ttt///ee w.w.lvorrrggaa.... hbtp:/ab//155 61.. 01474v3

---

Under review as a conference paper at ICLR 2020
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2218-2227, JMLR. org, 2017.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks, In Advances in Neural Information Processing Systems, pp. 7047-7058, 2018.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining Well Calibrated Prob- abilities Using Bayesian Binning. In AAAI Conference on Artificial Intelligence, volume 2015, pp.
2901-2907, January 2015. URL https:t/// www.ww www..ccbi.oo nlm. nih.
DBC4441030//dtf/// himms 679964. pdf
Jeremy Nixon, Mike Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring Calibration in Deep Learning. arXiv: [cs, stat), April 2019. URL http://arxiv
org/aba 904. 01685
Tom J Pollard. Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. The eicu collaborative research database, a freely available multi-center database for critical care research. Scientific data, 5, 2018.
Alvin Rajkomar, Michaela Hardt, Michael D. Howell, Greg Corrado, and Marshall H. Chin. Ensuring Fairness in Machine Learning to Advance Health Equity, Annals of Internal Medicine, 169(12): 866, December 2018a. ISSN 0003-4819. doi: N0.7326/M18-1990 URL URLhttp::
org/article.co asp. 2doi=10. 7326 3233222M122 M18-1990)
Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M. Dai, Nissan Hajaj, Michaela Hardt, Peter J. Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, Patrik Sundberg, Hector Yee, Kun Zhang, Yi Zhang, Gerardo Flores, Gavin E. Duggan, Jamie Irvine, Quoc Le, Kurt Litsch, Alexander Mossin, Justin Tansuwan, De Wang, James Wexler, Jimbo Wilson, Dana Ludwig, Samuel L. Volchenboum, Katherine Chou, Michael Pearson, Srinivasan Madabushi, Nigam H. Shah, Atul J. Butte, Michael D. Howell, Claire Cui, Greg S. Corrado, and Jeffrey Dean. Scalable and accurate deep learning with electronic health records. Nature Puriner Journals: Digital Medicine, 1(1):18, May 2018 8b. ISSN 2398-6352. doi: 10.1033/641746-018-0000
Juergen Schmidhuber and Sepp Hochreiter. Long short-term memory. Neural computation, 9(8):
1735-1780, November 1997. doi: doi.org/10. 162/neco. 1997.9..999999 1997.9.8.1 197...335.
Dustin Tran, Michael W. Dusenberry, Mark van der Wilk, and Danijar Hafner. Bayesian Layers: A Module for Neural Network Uncertainty. arXiv: [cs, stat], December 2018. URL http://arxiv. wttp///g///bb/oo..eeee 03973
Yanbo Xu, Siddharth Biswal, Shriprasad R Deshpande, Kevin O Maher, and Jimeng Sun. Raim: Recurrent attentive and intensive model of multimodal patient monitoring data. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- 2565 2573. ACM, 2018.
10

---

Under review as a conference paper at ICLR 2020
Table A.1: Hyperparameters and their associated search sets or ranges.
HYPERPARAMETER
RANGE/SET
BATCH SIZE
[32, 64, 128, 256, 512)
LEARNING RATE
[0.00001, 0.1]
KL OR REGULARIZATION ANNEALING STEPS
[1, 1E6]
PRIOR STANDARD DEVIATION (BAYESIAN ONLY)
[0.135, 1.0]
DENSE EMBEDDING DIMENSION
(16, 32, 64, 100, 128, 256, 512)
EMBEDDING DIMENSION MULTIPLIER
[0.5, 1.5]
RNN DIMENSION
(16,32, 64, 128, 256, 512, 1024)
NUMBER OF RNN LAYERS
[1,2,3)
HIDDEN AFFINE LAYER DIMENSION
(0, 16, 32, 64, 128, 256, 512)
BIAS UNCERTAINTY (BAYESIAN ONLY)
[TRUE, FALSE]
Table A.2: Model-specific hyperparameter values.
ВATCH SIZE:
PRIOR STD. DEV.
DENSE DIM
EMBRD, DIM. MULT.
RNN DIM
NUM LAYERS
HIDDEN DIM.
RATE
STEPS
UNCERT.
DETERMINISTIC
256
3.035E-4
32
0.858
1024
512
ENSEMBLE
RAYESIAN
256
1.238B-3
9.722E+5
0.292
32
0.858
1024
512
FALSE
EMBEDDINGS
RAYESIAN
256
1.6476-4
8.7826+5
0.149
32
0.858
1024
512
FALSE
OUTPUT
BAYESIAN
256
2.7108-4
9.912E+5
0.149
32
0.858
1024
512
FALSE
HIDDEN+OUTPUT
HAYESIAN
RNN HIDDEN
512
1.488F-3
6.3426+5
0.252
32
1.291
16
TRUE
+OUTPUT
DAYESIAN
128
1.265k-3
9.9838+5
0.162
256
1.061
16
TRUE
A APPENDIX
A.1 ADDITIONAL TRAINING DETAILS
Our RNN model uses the embedding logic as used in Rajkomar et al. (2018a) sequential and contextual features. Sequential embeddings are bagged into 1-day blocks, and fed into more L.STM layers. The final time-step output of the LSTM layers is concatenated with the contextual embeddings and fed into hidden dense layer, and the output of that layer is then fed into an output dense layer yielding a single probability value. ReLU non-linearity is used between the hidden and output dense layers, and default initializers in layees. * are used for all layers. More details on the training setup can be found in the code
embed
In terms of hyperparameter optimization, we searched over the hyperparameters listed in Table A. for the original deterministic RNN (all others the ensemble differ only by the random seed) and each of the Bayesian models. Table A.2 lists the final hyperparameters associated with each of the models presented in the paper.
Models were implemented using TensorFlow 1.13 Abadi et al. (2016), and trained on machines equipped with Nvidia's P100 using the Adam optimizer Kingma Ba (2014). MIMIC-III data were split into train, validation, and test set in 8:1:1 ratio.
A.2 ADDITIONAL METRICS AND STATISTICS
Figure A. examines the distribution of maximum predicted probabilities over the CCS classes, along with the distribution of predicted classes associated with the maximum probabilities. Similar to the
"Code will he open-sourced.
11

---

Under review as a conference paper at ICLR 2020
30-
20
20
10.
6943 cass asis dec d07 691 2.609 Nax probabiiiy parameter
0.0
0.
2.6
520
56 300
Max probobilkyy paramitter
Max prodiction
Mox claSs profction
Figure A.1: Left two: A set of distributions for the maximum predicted probability from our deterministic ensemble for two patients in the validation dataset on the mortality task. Note the difference in x-axis scales. Right two: The corresponding distributions of classes associated with the max probabilities. Similar to the mortality task, for some patients, such as the one on the left, the ensemble is relatively certain about the predicted class (completely certain in this case), while for other patients, such as the one on the right, there a larger amount of model uncertainty.
-0.695
-0.700
-0.705
Log- log-llkellhood
-0.710
-0.715
-0.720
-0.725
0.440
0.445
0.450
0.455
0.460
AUC-PR
Figure A.2: Validation AUC-PR versus held-out log-likelihood values for the deterministic RNN ensemble the mortality task. We find that there is apparent correlation between the two metrics, likely due to the limited differences between the models.
binary mortality task, this demonstrates the presence of model uncertainty in the multiclass clinical setting.
In Figure A.2
examine the correlation between held-out log- likelihood and AUC-PR values for models in the deterministic RNN ensemble on the mortality task.
In Figure A.3 we plot the differences between the maximum and minimum predicted probability values for each patient's 's predictive uncertainty distribution. We find that there is wide variability in predicted probabilities for some patients, and that negative patients have less variability average.
In Table A.3 we measure the calibration of marginalized predictions our deterministic RNN ensemble and the Bayesian RNNs
12

---

Under review as a conference paper at ICLR 2020
Negative
Postive
15
10
5
0.0
0.2
0.3
0.4
0.6
max-min
Figure A.3: histogram of differences between the maximum and minimum predicted probability values for each patient's predictive uncertainty distribution. This shows that there is wide variability predicted probabilities for some patients, and that negative patients have less variability on average.
Table A.3: Calibration error for marginalized predictions on the mortality task for an average over .M 200 models in the deterministic RNN ensemble, and M 200 samples from each of the Bayesian RNN models. We find that marginalization slightly increases the calibration of the deterministic ensemble, and that the Bayesian models are comparably well-calibrated.
VAL.
VAL.
TEST ECE
TEST
MODEL
ECE
ACE
ACE
DETERMINISTIC ENSEMBLE
0.0157
0.0191
0.0157
0.0191
BAYESIAN EMBEDDINGS
0.0167
0.0194
0,0163
0.0221
BAYESIAN OUTPUT
0.0263
0.0217
0.0241
0.0279
BAYESIAN HUDDEN+OUTPUT
0.0194
0.0212
0.0173
0.0240
BAYESIAN RNN+HIDDEN-OUTPUT
0.0240
0.0228
0.0182
0.0247
FULLY BAYESIAN
0.0226
0.0192
0.0178
0.0197
We additionally measure the correlation between entropy and word frequency as visualized in Figure A.4 We find further confirmation that rarer words are associated with higher model uncertainty, although there is level of variance at a given frequency.
we plot the precision-recall (PR) curves of all M models in our deterministic RNN ensemble across the full dataset along with error bars. We find that the PR curves are nearly identical for all models, and thus it again seems highly likely that any one of the models could have been selected if we were focused our recall-based decision cost function.
In Figure A.5
13

---

Under review as a conference paper at ICLR 2020
-10
-20
40
entropy
-50
-70
-80
-90
-10000
10000
20000
30000
40000
50000
trequency
Figure A.4: Correlation between the entropy of the Bayesian embedding distributions for free-text clinical notes and the associated word frequency. We find that rarer words are associated with higher model uncertainty, with some level of variance given frequency.
1.0
0.8
Precision
0.4
0.2
7
0.2
0.6
Recall
Figure A.5: Precision-recall curves for all 200 models the deterministic ensemble, All of the curves are nearly identical, which is in line with the AUC-PR results.
14


Reviews:
[{"review_id": "Skea4CjLcB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am not an expert in the field of model uncertainty\n\nSummary / contributions:\nThis paper discusses the important problem of model uncertainty in the output of ML models developed for clinical applications. The authors illustrate the underlying concepts using RNNs which are popular in the medical ML literature by applying these to two datasets. They argue that Bayesian RNNs with Bayesian embeddings should be the models of choice in such settings as they explicitly allow the expression of uncertainty whereby obtaining confidence intervals etc is easy. The other advantage is the fewer number of parameters that need to be stored to get such statistics.\n\nNovelty:\n-- Some of the ideas presented are standard or well-known properties to most ML practitioners. For instance, the relationship between mean and variance in Fig 2 or the uncertainty in predictions / optimal decisions in Fig 3. Is the point of the paper to make it more obvious?\n-- It is certainly the case that medicine practioners are not as aware of these issues, but to reach that audience this paper would do better in a venue that caters to that community. However, the paper needs to address the concerns first so as to not confuse that community\n-- The Bayesian RNN models being discussed are not novel either and their properties have been discussed in the corresponding papers (probably not in such detail and with examples).\n-- What is the value of the Bernoulli distribution? Isn't the single output from a well calibrated model is enough to give the same information.\n\nWriting:\nThe paper is very well written and has good figures and examples to explain the ideas. The one area that can be improved is the contributions section.\n\n\nResults:\n-- The authors do not discuss the related issue of model calibration in much detail. It is unclear what additional information we are gaining from the author's perspective of model uncertainty. A well calibrated model as well as other ways of obtaining confidence intervals (via hypothesis tests) would serve just as well.\n-- Are the conclusions derived on the specific datasets general?\n-- The results showing group-level biases are not very helpful and come across as anecdotal. These can be derived from most other models too.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BklmTfihKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thank you for an interesting read.\n\nThis paper is an experimental paper which argues the importance of epistemic/model uncertainty in applications for electronic health records (EHR). The main arguments are the following:\n1. current metrics on dataset level cannot reveal uncertainty in prediction on personal level;\n2. when evaluated on personal level, deterministic NNs with different random initialisations can produce very different predictions (thus require consideration of model uncertainty)\n\nI am not exactly sure if ICLR is the best venue for this submission, as there is quite little innovation in modelling methodology, and the empirical analysis is domain specific. I feel this paper is more suitable to e.g. MLHC or MICCAI which focus on data analysis/machine learning methods applied to healthcare science.\n\nStill I think the set of experiments in the paper is overall supportive to the main argument that the authors is trying to make. Possible improvements:\n1. The histograms in Figure 3 & 4 clearly show that, deterministic NNs trained with different initialisations produces diverse predictions on individual patients. I commend the authors for presenting these visualisations, and I think it would be more useful to quantify this phenomenon on dataset level, e.g. compute the mean and variance of this variation of individual predictions.\n2. I would expect to see an improvement of ECE for the Bayesian/deep ensemble models. The Table A.3 so marginal improvements, and I wonder how would this result support the author's claim? Also how do the ECE/ACE metrics look like when computed on sub-groups?\n\nApart from section 3.3, in general I think the paper writing is clear to me. The loss sensitive optimal decision method is interesting, but a lot of details are missing:\n1. The presentation in section 3.3 is unclear, e.g. minimising eq. (4) w.r.t. what? What's the definition of decision region? Also what exactly is the mathematical form of the associated cost used in the experiments?\n2. If I understand it correctly, in experiments the optimal thresholding method has only been applied to individual networks in the ensemble. If so what is the intention of discussing eq. (5) in the first place? Also it is unclear to me how this method performs in the deep ensemble/Bayesian RNN case. See e.g. https://arxiv.org/pdf/1805.03901.pdf for a relevant approach.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers an important problem in medical applications of deep learning, such as variability/stability of  model's predictions in face of various perturbations in the model (e.g., random seed), and evaluates different approaches to capturing model uncertainty. However, it appears to be little innovation in terms of machine-learning methodology, so ICLR might not be the best venue for this work, while perhaps other venues focused more on medical applications might be a better fit. 
 
Decision: Reject