Title: ANALYZING THE ROLE OF MODEL UNCERTAINTY FOR ELECTRONIC HEALTH RECORDS

Abstract: In medicine, both ethical and monetary costs of incorrect predictions can be significant, and the complexity of the problems often necessitates increasingly complex models. Recent work has shown that changing just the random seed is enough for otherwise well-tuned deep neural networks to vary in their individual predicted probabilities. In light of this, we investigate the role of model uncertainty methods in the medical domain. Using recurrent neural network (RNN) ensembles and various Bayesian RNNs, we show that population-level metrics, such as AUC-PR, AUC-ROC, log-likelihood, and calibration error, do not capture model uncertainty. Meanwhile, the presence of significant variability in patient-specific predictions and optimal decisions motivates the need for capturing model uncertainty. Understanding the uncertainty for individual patients is an area with clear clinical impact, such as determining when a model decision is likely to be brittle. We further show that RNNs with only Bayesian embeddings can be a more efficient way to capture model uncertainty compared to ensembles, and we analyze how model uncertainty is impacted across individual input features and patient subgroups.

1. INTRODUCTION:

Machine learning has found great and increasing levels of success in the last several years on many well-known benchmark datasets. This has led to a mounting interest in non-traditional problems and domains, each of which bring their own requirements. In medicine specifically, individualized predictions are of great importance to the field (Council et al., 2011) , and there can be severe costs for incorrect predictions and decisions due to the risk to human life and the associated ethical concerns (Gillon, 1994) .

Existing state-of-the-art approaches using deep neural networks in medicine often make use of either a single model or an average over a small ensemble of models, focusing on improving the accuracy of probabilistic predictions (Harutyunyan et al., 2017; Rajkomar et al., 2018b; Xu et al., 2018; Choi et al., 2018) . These works, while focusing on capturing the data uncertainty, do not address the model uncertainty that is inherent in fitting deep neural networks. For example, when predicting patient mortality in an ICU setting, existing approaches might be able to achieve high AUC-ROC, but will be unable to differentiate between patients for whom the model is certain about its probabilistic prediction, and those for whom the model is fairly uncertain.

In this paper, we examine the use of model uncertainty specifically in the context of predictive medicine. Model uncertainty has made many methodological advances in recent years-including reparameterization-based variational Bayesian neural networks (Blundell et al., 2015; Kucukelbir et al., 2017; Louizos & Welling, 2017) , Monte Carlo dropout (Gal & Ghahramani, 2016) , ensembles (Lakshminarayanan et al., 2017) , and function priors (Hafner et al., 2018; Garnelo et al., 2018; Malinin & Gales, 2018) . Deep neural networks combined with advanced model uncertainty methods can directly impact clinical care by answering several questions that naturally occur in predictive medicine:

• How do the realized functions in any of the approaches, such as individual models in the ensemble approach, compare in terms of population-level metric performance such as AUC-PR, AUC-ROC, or log-likelihood? • If and how does model uncertainty assist in calibrating predictions?

• How does model uncertainty change across different patient subgroups, in terms of ethnicity, gender, age, or length of stay? • How do various feature values contribute towards model uncertainty?

• How does model uncertainty affect optimal decisions made under a given clinically-relevant cost function?

Contributions Using sequence models on the MIMIC-III clinical dataset (Johnson et al., 2016) , we make several important findings. For the ensembling approach of quantifying model uncertainty, we find that the models within the ensemble can collectively exhibit a wide variability in predicted probabilities for some patients, despite being well-calibrated and having nearly identical dataset-level metric performance. We find that this even extends into the space of optimal decisions. That is, models with nearly equivalent metric performance can disagree significantly on the final decision, thus transforming an "optimal" decision into a random variable. Significant variability in patient-specific predictions and decisions can be an indicator of when a model decision is likely to be brittle, and we show that using a single model or an average over models can mask this information. This motivates the importance of model uncertainty for clinical decision systems. Given this, we proceed with an analysis over different clinical tasks and datasets, looking at how model uncertainty is impacted across individual input features and patient subgroups. We then show that models with Bayesian embeddings can be a more efficient way to capture model uncertainty compared to deep ensembles.

2. BACKGROUND:

Data uncertainty Data uncertainty can be viewed as uncertainty regarding a given outcome due to incomplete information, and is also known as "output uncertainty" or "risk" (Knight, 1957) . For binary tasks, this equates to a single probability value. More specifically, this can be described as

EQUATION

where the model f , as a function of the inputs x and parameters w, outputs the parameter λ for a Bernoulli distribution representing the conditional distribution p(y|x, w) for the outcome y.

Model uncertainty Model uncertainty can be viewed as uncertainty in the correct values of the parameters for the predictive outcome distribution due to lack of knowledge of the true function. For binary tasks, this equates to a distribution of plausible probability values for a Bernoulli distribution, corresponding to a set of plausible functions. More specifically, this can be described as

EQUATION

where there is an induced distribution over the Bernoulli parameter λ for a given example that represents uncertainty in the true outcome distribution due to uncertainty in function space. For the remainder of the paper, we will use the phrase predictive uncertainty distribution to refer to this distribution over the parameter of the outcome distribution. 

(i) 1 , λ (i) 2 , . . . , λ (i)

M } for the same example can then be viewed as the distribution p(λ|x, w) over λ (i) , where this distribution represents model uncertainty. Bayesian RNNs Bayesian recurrent neural networks (RNNs) (Fortunato et al., 2017) are RNNs with a prior distribution placed over the parameters. This allows us to express model uncertainty as uncertainty over the true values for the parameters in the model, i.e., "weight uncertainty" (Blundell et al., 2015) . By introducing a distribution over all, or a subset, of the weights in the model, we can induce different functions, and thus different outcomes, through realizations of different weight values via draws from the posterior distributions. This allows us to empirically capture model uncertainty in the predictive uncertainty distribution p(λ|x, w) by drawing M samples from a single Bayesian RNN for a given example.

3 MEDICAL UNCERTAINTY (Schmidhuber & Hochreiter, 1997) , followed by hidden and output affine layers. See the Appendix for more details.

Existing deep learning approaches in predictive medicine focus on capturing data uncertainty, namely accurately predicting the risk λ = f (x, w) of a patient's mortality (i.e., how likely is the patient to expire?). This work, on the other hand, also focuses on addressing the model uncertainty aspect of deep learning, namely the distribution λ ∼ p(λ|x, w) over the risk of mortality for a patient (i.e., are there alternative risk predictions, and if so, how diverse are they?).

3.2. CHOICE OF UNCERTAINTY METHODS:

To quantify model uncertainty for clinical tasks, we explore the use of deep RNN ensembles and various Bayesian RNNs. For the deep ensembles approach, we optimize for the ideal hyperparameter values for our RNN model via black-box Bayesian optimization (Golovin et al., 2017) , and then train M replicas of the best model. Only the random seed differs between the replicas. At prediction time, we make predictions with all M models for each patient. For the Bayesian RNNs, we train a single model, and then draw M samples from it at prediction time. To train the Bayesian RNN, we take a variational inference approach by adapting our RNNs to use factorized weight posteriors q(w|θ) = i q(w i |θ i ), where weight tensors w i in the models are represented by normal distributions with learnable mean and diagonal covariance parameters represented as θ i . Normal distributions with zero mean and tunable standard deviation are used as weight priors. We train our models by minimizing the Kullback-Leibler (KL) divergence

EQUATION

between the approximate weight posterior and the true, but unknown posterior, which overall equates to minimizing an expectation over the usual negative log likelihood term plus a KL regularization term. To easily shift between the deterministic and Bayesian models, we make use of the Bayesian Layers (Tran et al., 2018) abstractions.

3.3. OPTIMAL DECISIONS VIA SENSITIVITY REQUIREMENTS:

The key desire in clinical practice is to make a decision based on the model's predicted probability and its associated uncertainty. Given a set of potential outcomes y k , a set of conditional probabilities p(y k |x) for the given outcomes, and the associated costs L kj for either correctly or incorrectly predicting the outcome, an optimal decision can be determined by minimizing the expected decision loss

EQUATION

where R j is the decision region for assigning example x to class j, and p(x) is the density of x (Bishop, 2006) .

Designing elaborate decision cost functions for clinical applications is not an easy task as it requires expert knowledge on the prediction target, cost-benefit analysis, and medical resource allocation.

Fortunately we can use a clinically relevant alternative, which is the sensitivity requirement. Often in clinical research, certain sensitivity (i.e., recall) levels must be met when making predictions in order for a model to be clinically relevant. The goal in such cases is to maximize the precision while still maintaining the required sensitivity level. Viewed as a decision cost function, the cost is infinite if the recall is below the target level, and is otherwise minimized as the precision is increased, where the optimized parameter is the global probability threshold.

For each of the M models in our ensemble, we can optimize the sensitivity-based decision cost function and make optimal decisions for all examples. Thus, for each example, there will be a set of M optimal decisions, forming a distribution. The optimal decision d then becomes a random variable

EQUATION

)

where φ is the percentage of the set Λ = {λ 1 , λ 2 , . . . , λ M } of M probability values for a given example that are greater than or equal to the optimized decision threshold t.

4. EXPERIMENTS:

We perform four sets of experiments. First, in order to demonstrate the importance of quantifying uncertainty in predictive medicine, we examine individual models in the RNN ensemble in terms of prediction metrics, calibration, uncertainty distributions, and decision-making. Second, we examine multiple variants of Bayesian RNNs to understand where uncertainty in the model matters most, comparing them with the deterministic ensemble counterpart. Third, we use the deterministic RNN ensemble to examine uncertainty across different patient subgroups. Finally, we analyze the Bayesian RNN with embedding distributions to examine uncertainty across individual features. The corresponding optimal decision distributions, "True" being the positive prediction for mortality and "False" being the opposite. For some patients the ensemble is relatively certain about the optimal decision, while for other patients there is a large amount of uncertainty.

4.1. WHEN DO WE OBSERVE UNCERTAINTY?:

Clinical Metrics For our clinical tasks, we first measure the dataset-level metrics: area under the precision-recall curve (AUC-PR), area under the receiver operating characteristic curve (AUC-ROC), top-5 recall, top-5 precision, top-5 F1, held-out log-likelihood, ECE (Naeini et al., 2015) , and adaptive calibration error (ACE) (Nixon et al., 2019) . Table 1 shows the performance averaged over individual models in our deterministic ensemble, with standard deviation in the parenthesis. Interestingly, individual models are overall well-calibrated and nearly equivalent in terms of performance. If we were to choose only one model in practice based on the dataset-level metrics, it is highly likely any one could be selected as a good model, and we would lose the model uncertainty information.

Predictive Uncertainty Distributions & Statistics Knowing that the models in our ensemble are well-calibrated and effectively equivalent in terms of performance, we turn to making predictions for individual examples. Figure 1 visualizes the predictive uncertainty distribution for a single patient. We find that there is a wide variability in predicted Bernoulli probabilities for some patients (with spreads as high as 57.5%). Ignoring this variance through the use of either a single deterministic model or an average over an ensemble is likely harmful since it prevents this uncertainty from being conveyed to a physician. Figure 2 visualizes the means versus standard deviations of the predictive uncertainty distributions for the deterministic ensemble on all validation set examples. In contrast to the variance of a Bernoulli distribution, which is a simple function of the mean, we find that the standard deviations are patient-specific, and thus cannot be determined a priori.

Optimal Decision Distributions & Statistics:

In practice, model uncertainty is important as it can affect one's decisions. To test this, we optimize the sensitivity (i.e. recall) based decision cost function with respect to the probability threshold for each model in our RNN ensemble separately to achieve a recall of 70%, and then make optimal decisions for each example with each of the M models. Figure 3 visualizes how model uncertainty in probability space is realized in optimal decision space for two patients. We see that the model uncertainty does indeed extend into the optimal decision space, converting the optimal decision into a random variable. Furthermore, the decision distribution's variance can be quite high, and knowing when this is the case is important in order to avoid the cost of any incorrect decisions made by the system due to lack of precise knowledge about the correct level of data uncertainty.

4.2. COMPARISON: VARIANTS OF BAYESIAN RNNS AND DETERMINISTIC RNN ENSEMBLES:

A natural question in practice when employing the Bayesian approach is which part of the model should capture model uncertainty. To do so, we study Bayesian RNNs under a variety of priors:

• Bayesian Embeddings A Bayesian RNN in which the embedding parameters are stochastic, and all other parameters are deterministic. • Bayesian Output A Bayesian RNN in which the output layer parameters are stochastic, and all other parameters are deterministic. • Bayesian Hidden+Output A Bayesian RNN in which the hidden and output layer parameters are stochastic, and all other parameters are deterministic. • Bayesian RNN+Hidden+Output A Bayesian RNN in which the LSTM, hidden, and output layer parameters are stochastic, and all other parameters are deterministic. • Fully Bayesian A Bayesian RNN in which all parameters are stochastic.

Table 2 displays the metrics over marginalized predictions for each of the Bayesian RNN models and the deterministic RNN ensemble on the MIMIC-III and eICU mortality tasks. We find that the Bayesian Embeddings RNN model outperforms all other Bayesian variants and slightly outperforms the deterministic ensemble in terms of AUC-PR for MIMIC-III, and that the fully-Bayesian RNN outperforms the other models on the eICU dataset. Additionally, all of the Bayesian variants are either comparable or outperform the deterministic ensemble in terms of held-out log-likelihood on both datasets. Since our Bayesian models achieve strong performance while only requiring training of a single model (7.22 million parameters in the MIMIC-III Bayesian Embeddings RNN), versus M models in the deterministic ensemble (200 × 6.16 million parameters), as well as only requiring the single model at prediction time, using Bayesian RNNs can be a more efficient approach.

Figure 4 visualizes the predictive distributions of both the Bayesian RNN and the deterministic RNN ensemble for four individual patients. We find that the Bayesian model is qualitatively able to capture model uncertainty that is quite similar to that of the deterministic ensemble. Overall, the Bayesian Embeddings RNN, compared to the deterministic ensemble, demonstrated slightly better predictive performance and qualitatively similar uncertainty performance, with considerably less computational resources, making it an attractive choice in clinical practice.

4.3. PATIENT SUBGROUP ANALYSIS:

We next turn to an exploration of the effects of model uncertainty across patient subgroups. We split validation set encounters into subgroups by demographic characteristics, namely patient gender (3089 male vs. 2548 female) or age (adults divided into quartiles of 1216, with a separate fifth group of 773 neonates). For this analysis, we focus on the deterministic RNN ensemble described in Section 4.1, as the Bayesian models sample M = 200 weights separately for each prediction rather than globally for the complete validation set. For each model in the ensemble, we compute validation set performance metrics separately over each subgroup and then compute the correlation between these metrics over all models in the ensemble, to evaluate whether the ensemble models tend to specialize to one or more subgroups at the cost of performance on others. We find some evidence of this phenomenon: for example, AUC-PR for male patients is negatively correlated with AUC-PR for female patients (Pearson's r = -0.442, see Figure 5 ), and AUC-PR for the oldest quartile of adult patients is somewhat negatively correlated with AUC-PR for other adults or for neonates (Pearson's r between -0.18 and -0.37).

We also compare uncertainty metrics across subgroups, including standard deviation and range of the predictive uncertainty distributions and variance of the optimal decision distributions for patients in each subgroup. For this analysis, we examine both the deterministic ensemble and the best Bayesian model, the RNN using Bayesian embeddings. In both cases, we find that all metrics are correlated with subgroup label prevalence: both uncertainty and mortality rate increase monotonically across age groups (Figure 5 ), and both are slightly higher in women than in men. These findings imply that random model variation during training may actually cause unintentional harm to certain patient populations, which may not be reflected in aggregate performance. For this analysis, we focus on the free-text clinical notes found in the EHR. For each word in the notes vocabulary, we have an associated embeddings distribution formulated as a multivariate Normal. We rank each word by its level of model uncertainty (measured by its embedding distribution's entropy). Table 3 lists the top and bottom 10 words, along with their frequency in the training dataset. We find that common words, both subjectively and based on prevalence counts, have low entropy and thus limited model uncertainty, while rarer words have higher levels of model uncertainty.

5. DISCUSSION:

In this work, we demonstrated the need for capturing model uncertainty in medicine and examined methods to do so. Our experiments showed multiple findings. For example, an ensemble of deterministic RNNs captures individualized uncertainty conditioned on each patient, while the models each maintained nearly equivalent clinically-relevant dataset-level metrics. As another example, we found that models need only be uncertain around the embeddings for competitive performance, with the benefit of also enabling the ability to determine the level of model uncertainty associated with individual feature values. Furthermore, using model uncertainty methods, we examined patterns in uncertainty across patient subgroups, showing that models can exhibit higher levels of uncertainty for certain groups. We find that there is no apparent correlation between the two metrics, likely due to the limited differences between the models.

binary mortality task, this demonstrates the presence of model uncertainty in the multiclass clinical setting.

In Figure A.2, we examine the correlation between held-out log-likelihood and AUC-PR values for models in the deterministic RNN ensemble on the mortality task.

In Figure A.3, we plot the differences between the maximum and minimum predicted probability values for each patient's predictive uncertainty distribution. We find that there is wide variability in predicted probabilities for some patients, and that negative patients have less variability on average.

In Table A .3, we measure the calibration of marginalized predictions of our deterministic RNN ensemble and the Bayesian RNNs. 

:

We use the term "deterministic" to refer to the usual setup in which we optimize the parameter values of our function directly, yielding a trained model with fixed parameter values at test time.

Code will be open-sourced.

Figures and Tables:

Figure 3: Left two: The deterministic ensemble produces different mortality prediction distributions (i.e. model uncertainty) for two different patients from the validation set. Right two:The corresponding optimal decision distributions, "True" being the positive prediction for mortality and "False" being the opposite. For some patients the ensemble is relatively certain about the optimal decision, while for other patients there is a large amount of uncertainty.

Figure4: Predictive uncertainty distributions of both the RNN with Bayesian embeddings and the deterministic RNN ensemble for individual patients. We find that the Bayesian model is qualitatively able to capture model uncertainty that is quite similar to that of the ensemble with considerably smaller number of parameters .

EMBEDDING UNCERTAINTY ANALYSIS Another motivation for model uncertainty lies in understanding which feature values are most responsible for the variance of the predictive uncertainty distribution. Our RNN with Bayesian embeddings model is particularly well suited for this task in that the uncertainty in embedding space directly corresponds to the predictive uncertainty distribution and represents uncertainty associated with the discrete feature values. Understanding model uncertainty associated with features can allow us to recognize particularly difficult examples and understand which feature values are leading to

Figure 5: Left: Model performance comparison on male vs. female patients. Each point represents stratified AUC-PR for a single model from the deterministic ensemble. Correlation coefficient r = -0.442. Right: Summary of uncertainty measures within each age subgroup, using the Bayesian embeddings RNN. On all measures, uncertainty increases monotonically with age. This corresponds to an increase in mortality rate with age, as positive cases are more uncertain on average. Table 3: Top and bottom 10 words in free-text clinical notes based on their associated Bayesian embeddings distribution's entropy, along with their frequency in the training dataset.

Figure A.1 examines the distribution of maximum predicted probabilities over the CCS classes, along with the distribution of predicted classes associated with the maximum probabilities. Similar to the

Figure A.2:Validation AUC-PR versus held-out log-likelihood values for the deterministic RNN ensemble on the mortality task. We find that there is no apparent correlation between the two metrics, likely due to the limited differences between the models.

Figure A.4: Correlation between the entropy of the Bayesian embedding distributions for free-text clinical notes and the associated word frequency. We find that rarer words are associated with higher model uncertainty, with some level of variance at a given frequency.

Calibration A model is said to be perfectly calibrated if, for all examples for which the model produces the same prediction p for some outcome, the percentage of those examples truly associated with the outcome is equal to p, across all values of p. If a model is systematically over-or underconfident, it can be difficult to reliably use its predicted probabilities for decision making. The expected calibration error (ECE) metric(Naeini et al., 2015) is one tractable way to approximate the calibration of a model given a finite dataset.

Table content: <table><tr><td colspan="2">Deep Ensembles Deep ensembles (Lakshminarayanan et al., 2017) is a method for quantifying</td></tr><tr><td colspan="2">model uncertainty. In this approach, an ensemble of M deterministic 1 neural networks (NNs) is</td></tr><tr><td colspan="2">trained by varying only the random seed of an otherwise well-tuned set of hyperparameters. Given</td></tr><tr><td colspan="2">this ensemble, predictions can be made with each model m for a given example i, where (for a binary</td></tr><tr><td>task) each prediction is the probability parameter λ</td><td>(i) m for the Bernoulli distribution over the outcome.</td></tr><tr><td>The set of M probabilistic predictions {λ</td><td/></tr></table>

Dataset-level metrics for the MIMIC-III binary mortality and multiclass CCS prediction tasks across M = 200 models in the deterministic RNN ensemble. Individual models are nearly identical in terms of dataset-level performance across both tasks, but selecting a single model would remove the model uncertainty information such as that visualized in Figure1.

Table content: <table><tr><td>TASK</td><td>METRIC</td><td>VALIDATION</td><td>TEST</td></tr><tr><td>MORTALITY</td><td>AUC-PR</td><td>0.4496 (0.0025)</td><td>0.3886 (0.0059)</td></tr><tr><td/><td>AUC-ROC</td><td>0.8753 (0.0019)</td><td>0.8623 (0.0031)</td></tr><tr><td/><td>ECE</td><td>0.0176 (0.0040)</td><td>0.0162 (0.0043)</td></tr><tr><td/><td>ACE</td><td>0.0210 (0.0042)</td><td>0.0233 (0.0057)</td></tr><tr><td>CCS</td><td>TOP-5 RECALL</td><td>0.7126 (0.0071)</td><td>0.7090 (0.0088)</td></tr><tr><td/><td>TOP-5 PRECISION</td><td>0.1425 (0.0014)</td><td>0.1418 (0.0018)</td></tr><tr><td/><td>TOP-5 F1</td><td>0.2375 (0.0024)</td><td>0.2363 (0.0029)</td></tr><tr><td/><td>LOG-LIKELIHOOD</td><td>-5.1040 (0.0075)</td><td>-5.1081 (0.0083)</td></tr><tr><td/><td>ECE</td><td>0.0446 (0.0072)</td><td>0.0499 (0.0082)</td></tr><tr><td/><td>ACE</td><td>4.2189E-3 (7.3111E-8)</td><td>4.2191E-3 (7.6136E-8)</td></tr></table>

Metrics for marginalized predictions on the MIMIC-III and eICU mortality tasks given M = 200 models in the deterministic RNN ensemble, and M = 200 samples from each of the Bayesian RNN models. Confidence intervals are computed via validation and test set bootstrapping with 1000 bootstrap sets.

Table content: <table><tr><td>DATA</td><td>MODEL</td><td>VAL. AUC-PR</td><td>VAL. AUC-ROC</td><td>VAL. NLL</td><td>TEST AUC-PR</td><td>TEST AUC-ROC</td><td>TEST NLL</td></tr><tr><td/><td>DETERMINISTIC</td><td>0.4564</td><td>0.8774</td><td>0.7113</td><td>0.3921</td><td>0.8643</td><td>0.7148</td></tr><tr><td/><td>ENSEMBLE</td><td>(±1e-3)</td><td>(±5e-4)</td><td>(±7e-5)</td><td>(±1e-3)</td><td>(±5e-4)</td><td>(±7e-5)</td></tr><tr><td/><td>BAYESIAN</td><td>0.4580</td><td>0.8776</td><td>0.7152</td><td>0.3977</td><td>0.8612</td><td>0.7186</td></tr><tr><td/><td>EMBEDDINGS</td><td>(±1e-3)</td><td>(±4e-4)</td><td>(±7e-5)</td><td>(±2e-3)</td><td>(±5e-4)</td><td>(±7e-5)</td></tr><tr><td>MIMIC-III</td><td>BAYESIAN OUTPUT BAYESIAN HIDDEN+OUTPUT BAYESIAN</td><td>0.4382 (±2e-3) 0.4492 (±1e-3) 0.4396</td><td>0.8714 (±5e-4) 0.8751 (±5e-4) 0.8673</td><td>0.7058 (±7e-5) 0.7118 (±7e-5) 0.7097</td><td>0.3702 (±1e-3) 0.3893 (±1e-3) 0.3860</td><td>0.8572 (±5e-4) 0.8607 (±5e-4) 0.8542</td><td>0.7087 (±7e-5) 0.7149 (±7e-5) 0.7125</td></tr><tr><td/><td>RNN+HIDDEN+OUTPUT</td><td>(±2e-3)</td><td>(±5e-4)</td><td>(±7e-5)</td><td>(±2e-3)</td><td>(±5e-4)</td><td>(±7e-5)</td></tr><tr><td/><td>FULLY</td><td>0.4354</td><td>0.8692</td><td>0.7100</td><td>0.3829</td><td>0.8552</td><td>0.7133</td></tr><tr><td/><td>BAYESIAN</td><td>(±2e-3)</td><td>(±5e-4)</td><td>(±6e-5)</td><td>(±1e-3)</td><td>(±5e-4)</td><td>(±6e-5)</td></tr><tr><td/><td>DETERMINISTIC</td><td>0.1951</td><td>0.7882</td><td>0.1435</td><td>0.2196</td><td>0.7868</td><td>0.2435</td></tr><tr><td/><td>ENSEMBLE</td><td>(±1e-3)</td><td>(±7e-4)</td><td>(±3e-4)</td><td>(±1e-3)</td><td>(±6e-4)</td><td>(±5e-4)</td></tr><tr><td/><td>BAYESIAN</td><td>0.1996</td><td>0.7807</td><td>0.1455</td><td>0.2244</td><td>0.7733</td><td>0.1620</td></tr><tr><td/><td>EMBEDDINGS</td><td>(±1e-3)</td><td>(±1e-4)</td><td>(±4e-4)</td><td>(±1e-3)</td><td>(±7e-4)</td><td>(±4e-4)</td></tr><tr><td/><td>BAYESIAN</td><td>0.1738</td><td>0.7677</td><td>0.1664</td><td>0.1942</td><td>0.7580</td><td>0.1810</td></tr><tr><td>EICU</td><td>OUTPUT BAYESIAN HIDDEN+OUTPUT</td><td>(±1e-3) 0.1712 (±1e-3)</td><td>(±7e-4) 0.7801 (±7e-4)</td><td>(±3e-4) 0.1619 (±3e-4)</td><td>(±1e-3) 0.2140 (±e-)</td><td>(±7e-4) 0.7817 (±6e-4)</td><td>(±4e-4) 0.1713 (±3e-4)</td></tr><tr><td/><td>BAYESIAN</td><td>0.1675</td><td>0.7791</td><td>0.1477</td><td>0.2147</td><td>0.7809</td><td>0.1583</td></tr><tr><td/><td>RNN+HIDDEN+OUTPUT</td><td>(±1e-3)</td><td>(±7e-4)</td><td>(±3e-4)</td><td>(±1e-3)</td><td>(±7e-4)</td><td>(±3e-4)</td></tr><tr><td/><td>FULLY</td><td>0.2004</td><td>0.7910</td><td>0.1377</td><td>0.2280</td><td>0.7818</td><td>0.1541</td></tr><tr><td/><td>BAYESIAN</td><td>(±1e-3)</td><td>(±7e-4)</td><td>(±3e-4)</td><td>(±1e-3)</td><td>(±7e-4)</td><td>(±4e-4)</td></tr></table>

1: Hyperparameters and their associated search sets or ranges. uses the same embedding logic as used inRajkomar et al. (2018a)  to embed sequential and contextual features. Sequential embeddings are bagged into 1-day blocks, and fed into one or more LSTM layers. The final time-step output of the LSTM layers is concatenated with the contextual embeddings and fed into a hidden dense layer, and the output of that layer is then fed into an output dense layer yielding a single probability value. A ReLU non-linearity is used between the hidden and output dense layers, and default initializers in tf.keras.layers.* are used for all layers. More details on the training setup can be found in the code 2 .In terms of hyperparameter optimization, we searched over the hyperparameters listed in TableA.1 for the original deterministic RNN (all others in the ensemble differ only by the random seed) and each of the Bayesian models. TableA.2 lists the final hyperparameters associated with each of the models presented in the paper.

Table content: <table><tr><td colspan="2">HYPERPARAMETER</td><td/><td/><td/><td/><td colspan="2">RANGE/SET</td><td/><td/><td/></tr><tr><td>BATCH SIZE</td><td/><td/><td/><td/><td/><td colspan="4">{32, 64, 128, 256, 512}</td><td/></tr><tr><td colspan="2">LEARNING RATE</td><td/><td/><td/><td/><td colspan="2">[0.00001, 0.1]</td><td/><td/><td/></tr><tr><td colspan="5">KL OR REGULARIZATION ANNEALING STEPS</td><td/><td>[1, 1E6]</td><td/><td/><td/><td/></tr><tr><td colspan="8">PRIOR STANDARD DEVIATION (BAYESIAN ONLY) [0.135, 1.0]</td><td/><td/><td/></tr><tr><td colspan="4">DENSE EMBEDDING DIMENSION</td><td/><td/><td colspan="5">{16, 32, 64, 100, 128, 256, 512}</td></tr><tr><td colspan="4">EMBEDDING DIMENSION MULTIPLIER</td><td/><td/><td colspan="2">[0.5, 1.5]</td><td/><td/><td/></tr><tr><td colspan="2">RNN DIMENSION</td><td/><td/><td/><td/><td colspan="5">{16, 32, 64, 128, 256, 512, 1024}</td></tr><tr><td colspan="3">NUMBER OF RNN LAYERS</td><td/><td/><td/><td>{1, 2, 3}</td><td/><td/><td/><td/></tr><tr><td colspan="4">HIDDEN AFFINE LAYER DIMENSION</td><td/><td/><td colspan="4">{0, 16, 32, 64, 128, 256, 512}</td><td/></tr><tr><td colspan="5">BIAS UNCERTAINTY (BAYESIAN ONLY)</td><td/><td colspan="3">{TRUE, FALSE}</td><td/><td/></tr><tr><td/><td colspan="7">Table A.2: Model-specific hyperparameter values.</td><td/><td/><td/></tr><tr><td>MODEL</td><td>BATCH SIZE</td><td>LEARN. RATE</td><td>ANNEAL. STEPS</td><td>PRIOR STD. DEV.</td><td>DENSE EMBED. DIM.</td><td>EMBED. DIM. MULT.</td><td>RNN DIM.</td><td>NUM. RNN LAYERS</td><td>HIDDEN LAYER DIM.</td><td>BIAS UNCERT.</td></tr><tr><td>DETERMINISTIC ENSEMBLE</td><td>256</td><td>3.035E-4</td><td>1</td><td>-</td><td>32</td><td>0.858</td><td>1024</td><td>1</td><td>512</td><td>-</td></tr><tr><td>BAYESIAN EMBEDDINGS</td><td>256</td><td>1.238E-3</td><td>9.722E+5</td><td>0.292</td><td>32</td><td>0.858</td><td>1024</td><td>1</td><td>512</td><td>FALSE</td></tr><tr><td>BAYESIAN OUTPUT</td><td>256</td><td>1.647E-4</td><td>8.782E+5</td><td>0.149</td><td>32</td><td>0.858</td><td>1024</td><td>1</td><td>512</td><td>FALSE</td></tr><tr><td>BAYESIAN HIDDEN+OUTPUT</td><td>256</td><td>2.710E-4</td><td>9.912E+5</td><td>0.149</td><td>32</td><td>0.858</td><td>1024</td><td>1</td><td>512</td><td>FALSE</td></tr><tr><td>BAYESIAN</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>RNN+HIDDEN</td><td>512</td><td>1.488E-3</td><td>6.342E+5</td><td>0.252</td><td>32</td><td>1.291</td><td>16</td><td>1</td><td>0</td><td>TRUE</td></tr><tr><td>+OUTPUT</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>FULLY BAYESIAN</td><td>128</td><td>1.265E-3</td><td>9.983E+5</td><td>0.162</td><td>256</td><td>1.061</td><td>16</td><td>1</td><td>0</td><td>TRUE</td></tr><tr><td colspan="2">A APPENDIX</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="4">A.1 ADDITIONAL TRAINING DETAILS</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="11">Our RNN model Models were implemented using TensorFlow 1.13 Abadi et al. (2016), and trained on machines</td></tr><tr><td colspan="11">equipped with Nvidia's P100 using the Adam optimizer Kingma &amp; Ba (2014). MIMIC-III data were</td></tr><tr><td colspan="6">split into train, validation, and test set in 8:1:1 ratio.</td><td/><td/><td/><td/><td/></tr><tr><td colspan="5">A.2 ADDITIONAL METRICS AND STATISTICS</td><td/><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
Thank you for an interesting read. This paper is an experimental paper which argues the importance of epistemic/model uncertainty in applications for electronic health records (EHR). The main arguments are the following: 1. current metrics on dataset level cannot reveal uncertainty in prediction on personal level; 2. when evaluated on personal level, deterministic NNs with different random initialisations can produce very different predictions (thus require consideration of model uncertainty) I am not exactly sure if ICLR is the best venue for this submission, as there is quite little innovation in modelling methodology, and the empirical analysis is domain specific. I feel this paper is more suitable to e.g. MLHC or MICCAI which focus on data analysis/machine learning methods applied to healthcare science. Still I think the set of experiments in the paper is overall supportive to the main argument that the authors is trying to make. Possible improvements: 1. The histograms in Figure 3 & 4 clearly show that, deterministic NNs trained with different initialisations produces diverse predictions on individual patients. I commend the authors for presenting these visualisations, and I think it would be more useful to quantify this phenomenon on dataset level, e.g. compute the mean and variance of this variation of individual predictions. 2. I would expect to see an improvement of ECE for the Bayesian/deep ensemble models. The Table A.3 so marginal improvements, and I wonder how would this result support the author*s claim? Also how do the ECE/ACE metrics look like when computed on sub-groups? Apart from section 3.3, in general I think the paper writing is clear to me. The loss sensitive optimal decision method is interesting, but a lot of details are missing: 1. The presentation in section 3.3 is unclear, e.g. minimising eq. (4) w.r.t. what? What*s the definition of decision region? Also what exactly is the mathematical form of the associated cost used in the experiments? 2. If I understand it correctly, in experiments the optimal thresholding method has only been applied to individual networks in the ensemble. If so what is the intention of discussing eq. (5) in the first place? Also it is unclear to me how this method performs in the deep ensemble/Bayesian RNN case. See e.g. https://arxiv.org/pdf/1805.03901.pdf for a relevant approach.