SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Positive PE: This label is critical as it signifies the presence of a PE, a potentially life-threatening condition where one or more of the pulmonary arteries in the patient's lungs is blocked by a blood clot. Accurate identification of PE in radiology reports is a crucial step towards timely treatment and patient recovery.
- Subsegmental PE: This label indicates a PE that affects the subsegmental branches of the pulmonary arteries, the smaller vessels within the lung. This label is essential in tailoring patient treatment as subsegmental PE sometimes have different treatment protocols compared to PE located in the larger pulmonary arteries. The classification of PE down to the subsegmental level is vital for precision medicine.
- Acute PE: This label marks a sudden onset of PE. Acute PE is particularly significant due to its immediate risk to the patient. Rapid identification and treatment of acute PE can mean the difference between life and death. As such, the Acute PE label serves as an urgent signal in the patient's radiology report, prompting immediate medical intervention.
Using these reports and labels, we train a text-based labeling model that can automatically label the impression sections of radiology reports with "Positive PE", "Subsegmental PE", and "Acute PE" labels. We utilize a pretrained version of the Clinical Longformer model [45], as the backbone of our NLP labeler. This model is then finetuned, validated, and tested using our hand-labeled cases. After finetuning, this model is applied to generate labels for every case in our dataset, with the "Positive PE" label in particular used for all analyses. Each label was deemed positive if the model's prediction probability exceeded 0.5; otherwise, the label was classified as negative. The labeling process is shown in Figure 7.
Task Label Validation
We validate our PE NLP labels using the test set of the manual labels. The performance of the model can be found in Table 10. The precision and recall are quite high, especially for the main "Positive PE" label, indicating that our NLP-generated labels are high quality.
Figure 7: A flowchart of our NLP labeling process.
Figure 8: Confusion matrix between NLP labeler and human experts on notes under test set of our training data for NLP labeler
Even with satisfactory performance, we are interested to know the error modes of our NLP labeler so we manually examine the predictions of it against the ground truth. The confusion matrices are shown in Figure 8. When comparing against human annotation on notes, for false positive cases, we observed that the NLP might have mistakenly used the wording ‘...thrombus within the superior vena cava...’ as an indicator of positive pulmonary embolism when it is not. For false negative cases, the NLP labeler might have incorrectly used the wordings ‘...possible pulmonary emboli are incompletely evaluated. Consider ct pe protocol if clinically indicated...’ as positive PE. Overall the misclassifications when compared to experts' annotated notes are relatively low (12 out of 682).
D.2 Pulmonary Hypertension
We construct a set of prognostic pulmonary hypertension labels that attempt to define whether or not a patient has a future incidence of pulmonary hypertension in the next year.
Positive PE	Subsegmental PE	Acute PE
AUROC	0.99	0.99	0.99
F1	0.97	0.95	0.96
Precision	0.97	0.98	0.98
Recall	0.98	0.93	0.94
Accuracy	0.98	0.99	0.98
Class counts (pos)	228	43	201
Class counts (neg)	454	639	481
Total support	682	682	682
Table 10: NLP PE labeler performance. Each label was deemed positive if the model's prediction probability exceeded 0.5; otherwise, the label was classified as negative.
# Patients
Positive PH	97
Negative / Unknown PH	23
Table 11: Statistics for our ground truth hand-labeled pulmonary hypertension labels.
We start by having a board-certified clinician create a manually annotated label set for this task on 120 patients in our cohort. We define ground truth for this task based on the review of a subset of notes for those patients. Each of these notes is either labeled "Positive" or "Negative", where "Positive" is that the patient has pulmonary hypertension and "Negative" is that the patient either doesn't have pulmonary hypertension or it is unknown. We then aggregate these labels at the patient level, labeling a patient with any "Positive" label as "Positive" and "Negative" otherwise. The statistics for these labels are in table 11.
Hand-labeling all of the cases in our dataset is not viable so we use this seed set of manual labels to develop a structured data-based phenotyping algorithm that can then be applied to all of the patients in our dataset. From manual review and expert assistance, we derive Table 12 which contains a comprehensive list of ICD and internal Stanford codes that can be used to identify pulmonary hypertension. This phenotyping algorithm is applied to obtain the pulmonary hypertension labels that we use for our primary analysis.
We validate our pulmonary hypertension phenotyping algorithm by testing it using the hand-labeled set. Our hand labels don't incorporate time, so we can't directly compare the 12-month PH task used in our analysis to them. Instead, we compare a slightly modified algorithm that uses the same ICD/Stanford code list to identify patients who have ever had PH and compare that set of patients to the set of "Positive" patients in our hand-labeled set. The precision, recall, F1 and accuracy are in Table 13. Our phenotyping algorithm has a very high recall, of 0.91, with slightly worse precision. The reduced precision is likely due to how we only hand labeled a subset of notes, so our ground truth here has poor recall. Regardless, this demonstrates that the structured data phenotyping algorithm we are using is effective.
E Example of CTPA scan
For the readers who are unfamiliar with CTPA, we also attached an example in Figure 9. This scan demonstrates an MPR (multi-planar reconstruction) format rendering of the 3D volumetric CTPA scan from our INSPECT cohort.
Table 12: Concepts of pulmonary hypertension and their ICD9/10 codes
Concept Name	Vocabulary ID	Code
Pulmonary hypertension	STANFORD_CONDITION	1029634
Secondary pulmonary arterial hypertension	ICD10CM	I27.21
Pulmonary hypertension due to left heart disease	ICD10CM	I27.22
Chronic pulmonary heart disease	ICD9CM	416
Kyphoscoliotic heart disease	ICD9CM	416.1
Chronic pulmonary embolism	ICD9CM	416.2
Other secondary pulmonary hypertension	ICD10	I27.2
Other secondary pulmonary hypertension	ICD10CM	I27.29
Eisenmenger's syndrome	ICD10CM	I27.83
Primary pulmonary hypertension	ICD10	I27.0
Primary pulmonary hypertension	ICD9CM	416.0
Other chronic pulmonary heart diseases	ICD9CM	416.8
Other specified pulmonary heart diseases	ICD10CM	I27.89
Chronic pulmonary embolism	ICD10CM	I27.82
Pulmonary hypertension due to alveolar hypoventilation disorder	STANFORD_CONDITION	1170535
Kyphoscoliotic heart disease	ICD10CM	I27.1
Pulmonary hypertension, unspecified	ICD10CM	I27.20
Pulmonary hypertension due to lung diseases and hypoxia	ICD10CM	I27.23
Chronic pulmonary heart disease, unspecified	ICD9CM	416.9
Cor pulmonale (chronic)	ICD10CM	I27.81
Pulmonary heart disease, unspecified	ICD10CM	I27.9
Pulmonary heart disease, unspecified	ICD10	I27.9
Primary pulmonary hypertension	ICD10CM	I27.0
Kyphoscoliotic heart disease	ICD10	I27.1
Secondary pulmonary hypertension	STANFORD_CONDITION	67294
Chronic pulmonary heart disease (CMS-HCC)	STANFORD_CONDITION	142308
Chronic thromboembolic pulmonary hypertension	ICD10CM	I27.24
Other secondary pulmonary hypertension	STANFORD_CONDITION	2065632
Other secondary pulmonary hypertension	ICD10CM	I27.2
Positive PH
F1	0.88
Precision	0.85
Recall	0.91
Accuracy	0.80
Table 13: Structured data-based PH labeler performance.
Figure 9: An example of CTPA examination scan in multi-planar reconstruction
Hyperparameters	Values
LightGBM
max_depth	3,6,-1
learning_rate	0.02,0.1,0.5
num_leaves	10,25,100
software_version	LightGBM 3.3.5
MOTOR
linearprobe_l2_strength	automatic between 10 and 0
dropout	0
learning_rate	10-5
num_time_bins	8
survival_dim	512
inner_dim	768
layers	12
max_sequence_length	16,384
vocabulary_size	65,536
software_version	femr 0.1.8
CTPA Slice Encoder
learning_rate	0.0005
optimizer	AdamW
loss	BCEWithLogitsLoss
architecture	resnext101_32x8d
pretrain data	BigTransfer
software_version	timm 0.9.2
CTPA Sequence Encoder
learning_rate	0.001,0.0005,0.0001,0.00005
n_epochs	50
slice_aggregation	max, mean, attention, attention+max
sequenceEncoder_type	LSTM, GRU, Transformer
hidden size	128,256,512
bidirectional	True, False
num_layers	1,3,5
dropoutprob	0.0,0.25,0.5,0.75
weighted_sampleing	True, False
pretrain data	RSNA RESPECT
input_sie	256
PE NLP Labeler
max_sequence_length	1536
learning_rate	2e-5
n_epochs	15
architecture	Longformer
pretrain type	Clinical-Longformer
software_version	hugging face 4.30.1
Table 14: Hyperparameter search grids of methods under comparison in our experiments. The software version for implementing each method is also shown.
F Additional Model Details And Hyperparameters
Hyperparameters are selected through grid search on the validation set. Table 14 contains the hyperparameter grids, and the software versions used for each model.
F.1 CTPA model
Windowing We here begin to describe the viewing window for our CTPA imaging model. (window center  $= -600$ , window width  $= 1500$ ), pulmonary embolism (window center  $= 400$ , window width  $= 1000$ ), and the mediastinum (window center  $= 40$ , window width  $= 400$ ). Specifically, for each viewing window, we clipped the Hounsfield Unit (HU) pixel values to fall within the range [windowcenter - windowwidth/2, windowcenter + windowwidth/2]
Slice Encoder Augmentations After the windowing operation, every CT scan is resized to dimensions of  $256 \times 256$  followed by a random cropping operation to yield a  $224 \times 224$  size. Before inputting into the model, each slice is normalized using the mean and standard deviation values from ImageNet. Once the slice encoder training phase is concluded, each slice is inputted into the trained model for the extraction of a latent representation. In this phase, center cropping is applied as opposed to random cropping for retrieving slice representations.
Sequence Encoder Augmentations Before the slice representations are input into a sequence encoder, we ensure each series is standardized to the same input size through either random sampling or padding. Specifically, if a series has a higher slice count than num_slices, a random sampling of the slices is conducted to equalize with num_slices. Alternatively, if a series possesses fewer slices, padding is executed with zero vectors to complete the series.
F.2 Structured Electronic Health Records Models
Gradient Boosted Tree Model
For our featurization, we use count featurization augmented by ontologies. For count featurization, we count each occurrence before the prediction time of every medical code (diagnoses, procedures, lab orders, and medications) and have a column containing the count for each code. Normally, this is a very sparse matrix as each code individually is relatively rare, so we take advantage of the standard ontology expansion technique, where we count higher level concepts in addition to the raw codes themselves. For instance, we will have a column both for the number of very specific ICD/I27.29 codes as well as a column for the more generic ICD/IXX (and I class ICD code) concept.
These features are then fed into a hyperparameter tuned LightGBM model [35].
MOTOR Model
MOTOR [63] is a self-supervised transformer model designed for long-term medical prediction. For our experiments, we use a version that was already pretrained on de-identified Stanford data. We explicitly construct our training, validation, and test cohorts in sync with that pretrained model such that there is no overlap between its pretraining data and our test and validation data.
We use the linear probe method for adapting MOTOR to our tasks. Aka, we extract the final patient representation from the last transformer layer and then train a logistic regression model with L2 regularization on those representations.
F.3 Model Fusion
For model fusion, we apply a simple late fusion strategy of taking a weighted average of the outputs of each source model. We implement this by first converting all output probabilities to logits, and then fitting a logistic regression model on those logits using the validation set. We do not use any regularization for that logistic regression model as it only has at most 3 features in our setup.
Furthermore, we examine the agreement and disagreement between the three source models by computing the Spearman correlations between their output probabilities on the 8 tasks in our dataset. Figure 10 contains the corresponding heatmaps. As expected, the two EHR based models, MOTOR and GBM, are much more correlated with each other than the CT based model.
Figure 10: Spearman correlation matrices for each pair of models' output probabilities. CT is the CTPA based LRCN model,  $\mathbf{M}$  is the structured EHR based MOTOR model, and  $\mathbf{G}$  is the structured EHR based gradient-boosted trees model.
Input Modality	Diagnostic	Prognostic
Image	EHR	PE	In-Hospital Mortality	Re-admission	PH
CT	M	G	(+)	1 m	6 m	12 m	1 m	6 m	12 m	12 m
✓	(0.69, 0.75)	(0.76, 0.83)	(0.73, 0.78)	(0.72, 0.77)	(0.50, 0.60)	(0.48, 0.55)	(0.49, 0.56)	(0.63, 0.69)
✓	(0.66, 0.70)	(0.91, 0.94)	(0.89, 0.92)	(0.88, 0.91)	(0.73, 0.81)	(0.75, 0.81)	(0.74, 0.79)	(0.80, 0.85)
✓	(0.66, 0.71)	(0.82, 0.87)	(0.85, 0.88)	(0.84, 0.87)	(0.69, 0.78)	(0.71, 0.77)	(0.70, 0.75)	(0.80, 0.85)
✓	✓	(0.74, 0.78)	(0.91, 0.94)	(0.89, 0.92)	(0.88, 0.91)	(0.73, 0.82)	(0.75, 0.80)	(0.74, 0.79)	(0.80, 0.84)
✓	✓	(0.74, 0.79)	(0.84, 0.89)	(0.86, 0.89)	(0.85, 0.88)	(0.69, 0.78)	(0.71, 0.76)	(0.70, 0.75)	(0.81, 0.85)
✓	✓	(0.68, 0.72)	(0.91, 0.94)	(0.89, 0.92)	(0.88, 0.91)	(0.74, 0.82)	(0.76, 0.81)	(0.75, 0.80)	(0.82, 0.87)
✓	✓	✓	(0.75, 0.79)	(0.91, 0.94)	(0.89, 0.92)	(0.88, 0.91)	(0.74, 0.82)	(0.76, 0.81)	(0.75, 0.79)	(0.82, 0.86)
Table 15:  $95\%$  confidence intervals as a function of the test set for model performance in AUROC. CT is the CTPA based LRCN model,  $\mathbf{M}$  is the structured EHR based MOTOR model, and  $\mathbf{G}$  is the structured EHR based gradient-boosted trees model.
G Experiment Compute Environment
EHR experiments are performed in a local on-prem university compute environment using 24 Intel Xeon 2.70GHz CPU cores and 1 Nvidia V100 GPU.
Image experiments are performed on a HIPAA-compliant Google virtual machine using 4 x Nvidia V100 GPU with 96 Intel Skylake vCPU with 624GB of RAM.
All compute environments supported HIPAA-compliant data protocols.
H Confidence Intervals
We obtain some uncertainty estimates for our results by bootstrapping with respect to the test set.
First, we estimate the uncertainty of the AUROC of each model. For each task, we create 1,000 bootstrap samples and compute the AUROC for each model on each bootstrap sample. We then extract the  $2.5\%$  and  $97.5\%$  percentiles of the 1,000 samples to obtain  $95\%$  confidence intervals.
Table 15 presents the results of this analysis. The widths of the intervals are often around 0.04, indicating that we are able to estimate model performance with reasonable precision.
Second, we estimate the uncertainty of the relative AUROC of each model. We use the same bootstrap samples as in the first analysis, but compute the relative performance between each model and our chosen baseline, which we arbitrarily choose as MOTOR.
Input Modality	Diagnostic	Prognostic
Image	EHR	PE	In-Hospital Mortality	Re-admission	PH
CT	M	G	(+)	1 m	6 m	12 m	1 m	6 m	12 m	12 m
✓	(0.01, 0.07)	(-0.16, -0.10)	(-0.17, -0.12)	(-0.17, -0.12)	(-0.29, -0.16)	(-0.30, -0.22)	(-0.28, -0.20)	(-0.19, -0.13)
✓	(0.00, 0.00)	(0.00, 0.00)	(0.00, 0.00)	(0.00, 0.00)	(0.00, 0.00)	(0.00, 0.00)	(0.00, 0.00)	(0.00, 0.00)
✓	(-0.02, 0.03)	(-0.10, -0.05)	(-0.05, -0.02)	(-0.05, -0.02)	(-0.08, 0.01)	(-0.06, -0.02)	(-0.06, -0.02)	(-0.02, 0.03)
✓	✓	(0.06, 0.10)	(-0.00, 0.01)	(-0.00, 0.01)	(-0.00, 0.01)	(-0.00, 0.00)	(-0.00, 0.00)	(-0.01, 0.00)	(-0.01, 0.00)
✓	✓	(0.06, 0.11)	(-0.08, -0.03)	(-0.04, -0.01)	(-0.04, -0.01)	(-0.08, 0.01)	(-0.07, -0.02)	(-0.07, -0.02)	(-0.01, 0.03)
✓	(0.01, 0.03)	(-0.00, 0.00)	(-0.00, 0.00)	(-0.00, 0.00)	(-0.00, 0.02)	(0.00, 0.01)	(0.00, 0.01)	(0.01, 0.04)
✓	✓	✓	(0.07, 0.11)	(-0.00, 0.01)	(-0.00, 0.01)	(-0.00, 0.01)	(-0.00, 0.02)	(0.00, 0.01)	(-0.00, 0.01)	(0.00, 0.04)
Table 16:  $95\%$  confidence intervals for the difference in AUROC performance between a particular model and the structured EHR based MOTOR model. CT is the CTPA based LRCN model,  $\mathbf{M}$  is the structured EHR based MOTOR model, and  $\mathbf{G}$  is the structured EHR based gradient-boosted trees model. Statistically significant differences at  $\mathrm{p} = 0.05$  are bolded.
Input Modality	Diagnostic	Prognostic
Image	EHR	PE	In-Hospital Mortality	Re-admission	PH
CT	M G	(+)	1 m	6 m	12 m	1 m	6 m	12 m	12 m
✓	0.715, (0.003)	0.741, (0.007)	0.753, (0.001)	0.750, (0.003)	0.549, (0.008)	0.547, (0.014)	0.551, (0.012)	0.658, (0.005)
Table 17: The mean and standard deviation in AUROC for the various tasks when the random seed is changed. We use 5 random seeds to estimate both the mean and standard deviation.
Table 16 contains these relative confidence intervals. Most (12/14) of the intervals for individual models exclude zero, indicating that we have enough precision to accurately tell the difference in performance between CT, MOTOR, and gradient-boosted tree models. The fused models have a less clear separation, with about half of the differences being statistically insignificant.
In order to conduct variation study, we have rerun our image-based modality for 5 times for different seeds. Note that our EHR baselines, MOTOR and LightGBM, are deterministic with the hyperparameters we used in our study, so we do not perform reseeding experiments. The results of this analysis are in Table 17.
I Model Performance as a Function of PE Status
In our results section, we present performance statistics on the entire cohort. However, it is sometimes useful to look at performance within patients who test positive for PE (+) vs patients who test negative for PE (-). Table 18 contains the performance on the seven prognostic tasks by PE status.
J Comparison of Models vs. Simplified PESI
Clinical risk scores are heuristics commonly used in medicine to inform treatment decisions. We compare our machine learning-based models against a common PE rule-based risk calculator, the simplified PESI (sPESI) score [55]. sPESI is a 0-6 scoring rule comprised of the following additive criteria (each rule contributes +1 to the overall score) in Table 19.
To ensure that the features used to calculate sPESI reflect the patient's condition at the time of the imaging, we only use data between the most recent 10 days prior to the CT exam and the 2 days after the CT scan for the numeric sPESI features. Patients that are missing data required for sPESI are dropped. In addition, as sPESI is only designed for use with patients that have PE, we further restrict this analysis to patients that have a positive diagnosis for PE. This results in a total of 1,719 cases derived from 1,609 unique patients with the required sPESI features and PE. The amount of patients with each total score is listed in Table 20.
We evaluate the sPESI score by measuring the performance in terms of AUROC of using the score to rank patients for our seven prognostic tasks. Table 21 contains the results of this comparison. Note that sPESI is designed for short-term mortality prediction, and might not be meaningful in the context of other prognostic tasks. We observe relatively low performance for the sPESI. One potential cause of that low performance is that our retrospective data has a much higher degree of missingness
Has PE	Input Modality	Prognostic
Image	EHR	In-Hospital Mortality	Re-admission	PH
CT	M	G	1 m	6 m	12 m	1 m	6 m	12 m	12 m
(+)	✓	0.761	0.738	0.726	0.609	0.586	0.629	0.596
✓	0.914	0.897	0.869	0.782	0.770	0.755	0.761
✓	0.853	0.850	0.835	0.773	0.763	0.729	0.762
✓	✓	0.879	0.902	0.870	0.752	0.766	0.760	0.740
✓	✓	0.817	0.858	0.844	0.712	0.748	0.732	0.740
✓	✓	0.914	0.897	0.871	0.788	0.789	0.757	0.772
✓	✓	✓	0.879	0.899	0.870	0.762	0.776	0.758	0.752
(-)	✓	0.806	0.758	0.754	0.534	0.504	0.507	0.677
✓	0.925	0.902	0.897	0.771	0.782	0.770	0.852
✓	0.847	0.869	0.861	0.728	0.737	0.728	0.852
✓	✓	0.927	0.903	0.900	0.771	0.778	0.764	0.850
✓	✓	0.872	0.879	0.873	0.729	0.728	0.716	0.859
✓	✓	0.924	0.905	0.898	0.775	0.787	0.775	0.871
✓	✓	✓	0.926	0.905	0.901	0.775	0.783	0.768	0.868
Table 18: The performance in AUROC for our different baseline modeling strategies, split by PE status. CT is the CTPA based LRCN model,  $\mathbf{M}$  is the structured EHR based MOTOR model, and  $\mathbf{G}$  is the structured EHR based gradient-boosted trees model. The best overall models are bolded and the best individual models are underlined.
#	PESI Score criteria
1	Age > 80
2	History of Cancer
3	History of Chronic Cardiopulmonary Disease
4	Heart Rate (bpm) ≥ 110
5	Systolic BP (mmHg) < 100
6	O2Saturation < 90%
Table 19: Criteria for PESI score
sPESI Score	# Cases With Score
0	169
1	361
2	529
3	450
4	184
5	30
6	1
Table 20: The statistics for the sPESI scores on the 1,719 cases in our cohort that it can be calculated on.
Input Modality	Prognostic
Image	EHR	In-Hospital Mortality	Re-admission	PH
CT	M	G	P	1 m	6 m	12 m	1 m	6 m	12 m	12 m
✓	0.676	0.634	0.663	0.643	0.478	0.631	0.579
✓	0.808	0.813	0.787	0.745	0.684	0.678	0.725
✓	0.749	0.749	0.741	0.679	0.620	0.619	0.688
✓	0.569	0.571	0.571	0.701	0.679	0.614	0.567
Table 21: The performance in AUROC for our different baseline modeling strategies given patients with PE. Note that this set of evaluations is only done on the subset of cases that have both PE and enough data for the simplified PESI risk score. CT is the CTPA based LRCN model,  $\mathbf{M}$  is the structured EHR based MOTOR model,  $\mathbf{G}$  is the structured EHR based gradient-boosted trees model, and  $\mathbf{P}$  is the simplified PESI risk score. The best models are bolded.
than the prospective studies used to generate and validate sPESI. For example, our ability to extract features like the history of Chronic Cardiopulmonary Disease is relatively limited as we are restricted to structured data already within the health record.
K Additional Metrics
For our main analysis, we compare models in terms of AUROC as it is a low variance and widely used metric. However, additional metrics, especially in the clinical space, can also be important when evaluating the utility of models.
Input Modality	Diagnostic	Prognostic
Image	EHR	PE	In-Hospital Mortality	Re-admission	PH
CT	M	G	(+)	1 m	6 m	12 m	1 m	6 m	12 m	12 m
✓	0.463	0.189	0.288	0.324	0.056	0.124	0.169	0.230
✓	0.327	0.396	0.537	0.588	0.160	0.342	0.402	0.485
✓	0.335	0.234	0.426	0.497	0.145	0.276	0.334	0.582
✓	✓	0.510	0.426	0.545	0.599	0.164	0.337	0.393	0.481
✓	✓	0.515	0.295	0.447	0.521	0.145	0.271	0.330	0.573
✓	✓	0.354	0.399	0.542	0.587	0.179	0.346	0.407	0.597
✓	✓	✓	0.523	0.428	0.542	0.598	0.180	0.343	0.399	0.589
Table 22: The performance in AUPRC for our different baseline modeling strategies, including late fusion. CT is the CTPA based LRCN model,  $\mathbf{M}$  is the structured EHR based MOTOR model, and  $\mathbf{G}$  is the structured EHR based gradient-boosted trees model. The best overall models are bolded and the best individual models are underlined.
We thus perform additional analysis to compare our models in terms of both AUPRC (area under the precision-recall curve) (see Table 22 and ECE (expected calibration error) (see Table 23). AUPRC provides an estimate of the precision of a model under various recall thresholds and ECE provides an estimate of the calibration of a model. We use 10 bins for our ECE estimate.
The relative model performance rankings for both of these additional metrics are very similar to the rankings seen with AUROC, with the EHR models doing better at prognostic tasks while the image models do better at the diagnostic task.