HypUC: Hyperfine Uncertainty Calibration with Gradient- boosted Corrections for Reliable Regression on Imbalanced Electrocardiograms
Authors: Uddeshya Upadhyay
Sairam Bade
Arjun Puranik
Shahir Asfahan
Melwin Babu
Francisco Lopez-Jimenez
Samuel Asirvatham
Ashim Prasad
Ajit Rajasekharan
Samir Awasthi
Rakesh Barve
Venue: Accepted by TMLR ()

Abstract: The automated analysis of medical time series, such as the electrocardiogram (ECG), electroencephalogram (EEG), pulse oximetry, etc, has the potential to serve as a valuable tool for diagnostic decisions, allowing for remote monitoring of patients and more efficient use of expensive and time-consuming medical procedures. Deep neural networks (DNNs) have been demonstrated to process such signals effectively. However, previous research has primarily focused on classifying medical time series rather than attempting to regress the continuous-valued physiological parameters central to diagnosis. One significant challenge in this regard is the imbalanced nature of the dataset, as a low prevalence of abnormal conditions can lead to heavily skewed data that results in inaccurate predictions and a lack of certainty in such predictions when deployed. To address these challenges, we propose HypUC, a framework for imbalanced probabilistic regression in medical time series, making several contributions. (i) We introduce a simple kernel density-based technique to tackle the imbalanced regression problem with medical time series. (ii) Moreover, we employ a probabilistic regression framework that allows uncertainty estimation for the predicted continuous values. (iii) We also present a new approach to calibrate the predicted uncertainty further. (iv) Finally, we demonstrate a technique to use calibrated uncertainty estimates to improve the predicted continuous value and show the efficacy of the calibrated uncertainty estimates to flag unreliable predictions. HypUC is evaluated on a large, diverse, real-world dataset of ECGs collected from millions of patients, outperforming several conventional baselines on various diagnostic tasks, suggesting potential use-case for the reliable clinical deployment of deep learning models and a prospective clinical trial. Consequently, a hyperkalemia diagnosis algorithm based on HypUC is going to be the subject of a real-world clinical prospective study.

Published in Transactions on Machine Learning Research (08/2023)
HypUC:
Hyperfine Uncertainty Calibration with Gradient- boosted Corrections for Reliable Regression on Imbalanced Electrocardiograms
Uddeshya Upadhyay!
Sairam Bade
Arjun Puranik
Shahir Asfahan'
Melwin Babu* 1
Francisco Lopez-Jimener'
Samuel J. Asirvatham
Ashim 2rasad
Ajit Rajasekharan'
Samir Awasthi22
Rakesh Barvel.2
Nference Inc.
Anumana Inc.
Mayo Clinic, USA
Reviewed OpenReview: https: openreview.
formm? id= EXo9g5EZZN
Abstract
The automated analysis of medical time series, such as the electrocardiogram (ECG), elec- troencephalogram (EEG), pulse oximetry, etc, has the potential to serve as valuable tool for diagnostic decisions, allowing for remote monitoring of patients and more efficient use of expensive and time-consuning medical procedures. Deep neural networks (DNNs) have been demonstrated to process such signals effectively. However, previous research has primarily focused on classifying medical time series rather than attempting to regress the continuous- valued physiological parameters contral to diagnosis. One significant challenge in this regard the imbalanced nature of the dataset, as a low prevalence abnormal conditions can lead to heavily skewed data that results in inaccurate predictions and a lack of certainty in such predictions when deployed. To address these challenges, we propose HypUC, framework for imbalanced probabilistic regression in medical time scries, making several contributions. (i) We introduce a simple kernel density-based technique to tackle the imbalanced regres- sion problem with medical time series. (ii) Moreover, we employ a probabilistic regression framework that allows uncertainty estimation for the predicted continuous values. (iii) We also present a new approach to calibrate the predicted uncertainty further. (iv) Finally, we demonstrate a technique to use calibrated uncertainty estimates improve the predicted continuous value and show the efficacy of the calibrated uncertainty estimates to flag un- reliable predictions. HypUC is evaluated collected from millions of patients, outperforming several conventional baselines various diagnostic tasks, suggesting potential use-case for the reliable clinical deployment of deep learning models and prospective clinical trial. Consequently, a hyperkalemia diagnosis al- gorithm based on HypUC is going to be the subject of a real-world clinical prospective study.
large, diverse, real-world dataset of ECGs
1 Introduction
Electrocardiogram (ECG) signals widely used in the diagnosis and management of cardiovascular dis- cases. However, widely popular manual analysis of ECG signals can be time-consuming and subject to human error. Automated analysis of ECG signals using machine learning (ML) techniques has the po- tential to improve the accuracy and efficiency ECG signal analysis. Moreover, will open up the avenue for remote pationt monitoring for managing the cardiovascular health of the patients scale. Recent advances in ML have led to the development of various algorithms for the automated analy- sis of FCG signals. These include methods for FCG signal classification and diagnosis of specific

---

Published in Transactions on Machine Learning Research (08/2023)
diovascular discases (Hanmun et al., 2019). Convolutional neural networks (CNNs) and recurrent neu- ral networks (RNNs) are commonly used for these tasks (Siontis et al., 2021; Şentürk et al., 2018).
Existing literature often focuses on classifying the medical time series (in one of the disease- related classes) (Siontis et al., 2021; Hannun et al., 2019) instead of regressing the continuous- valued physiological parameter which the di- agnosis is based (Von Bachmann et al., 2022). But often, solving the regression problem is more valuable than classification; for instance, con- sider the problem of measuring electrolytes like potassium in the blood. Excess of potassium (known as Hyperkalemia) can be fatal. Detecting if the patient has Hyperkalemia non-invasively by analyzing the ECG is desirable and has been studied previously (Galloway et al., 2019). However, a binary classifier does not distinguish between the different severity of Hyperkalemia as shown in Figure that may decide the final diagnosis for the condition, i.., severe/mild hyperkalemia requires different treatments.
Normal
Mild DC mEill
Moderate mEgti
Severe #EghL)
(K" 5.5 mEgl))
nusweiing severity of Hypenkoderiis
Figure Severity of the Hyperkalemia often decides diagnosis. Hence, the binary classification not sufficient.
One of the significant challenges in the real world for regressing the reriinuuos-vvlluee physiological parameter from medical time series is the imbalanced nature of the dataset, i..., such datasets are often highly skewed if the prevalence of the abnormal condition is low. A heavily skewed dataset leads to a trained network that is inaccurate in the prediction. Moreover, a deterministic deep-learning model cannot quantify uncertainty in predictions at deployment. To address these limitations, we propose a new framework called HypUC, shown in Figure 2. It leverages idcas from probabilistic regression and density cstimation to tackle uncortainty-awaro imbalanced regression with medical lime series. particular, our method makes several contributions:
We introduce simple kernel density-based technique to tackle the imbalanced regression problem with medical time series, discussed in detail in Section 3.2.2, 3.3.
Our method builds on probabilistic deep regression that allows uncertainty estimation for the pre- dicted continuous values in the novel context of medical time series, presented in Section 3.2.3, 3.3.
We propose new approach for calibrating the predieted uncertainty in Section 3.3.2.
Finally, we demonstrate the efficacy of the calibrated uncertainty estimates to improve the predicted continuous value and also flag unreliable predictions, as discussed in Section 3.3.3.
We study the efficacy of our methods on large real-world dataset of ECGs (collected using millions of patients with diverse medical conditions) to prodict various physiological parameters such Left Ventricular Ejection Fraction, Serun Polassium Level, Age, Survical, etc.
2 Related Work
Automated analysis of medical time series. Deep learning techniques have been extensively used to an- alyze medical time series signals. For instance, the work by Schaekermann et al. (2020); Calisto et al. (2017; 2022; 2023); Enevoldsen et al. (2023) compare the performance of machine learning models medical time series classification in the presence of ambiguity and improve the workflow of medical practitioners. There are also works that synthesize the personalized medical time scries data, like glucose monitoring signal using generalive models (Zhu et al., 2023). Similar lo other time-series in helathcare, in the constext of rocar- diograms, one of the main areas of focus has been the use of convolutional neural networks (CNNs) for ECG signal classification. For instance, the studics by Rajpurkar et al. (2017); Hannun et al. (2019); Murugesan et al. (2018) used CNNs to classify ECG signals into normal and abnormal beats accurately. Another set of studies by Ebrahimi et al. (2020); Li et al. (2016) proposed deep learning-based framework for arrhythmia
2

---

Published in Transactions on Machine Learning Research (08/2023)
Inpurt Waveform
Probabiliatic Deep Neural Network Regressor
Output Probability Distribution
Output Calibrated Probability Distribution
Proeaaii Dosaly Foeeyaa 9080)
Pobaasity Daseap Pandha doo
0.025
0.020
Global Uncertainty
Predicted Distr.
Calibration
0.015
0.015
Woll Voll-callbrated
Predicted
0.010
Hypertine Uncertainty
0.003
Calibration
0.005
0.000
0.000
Prodicted
Precictad
Figure 2: HypUC: The proposed framework processes the ECG waveform input to predict the parameters of the distribution (i.e., entire parametric distribution) representing the continuous value. The predicted distribution is subjected to global and hyperfine uncertainty calibration techniques to get well-calibrated predictive distribution and uncertainty estimate as explained in Section 3.
detection using ECG signals. Moreover, some studies use ECGs predict QRS complex location (Han al., 2022; Camps et al., 2018), heart-rate (Xu et al., 2019; Sentürk et. al., 2018), and cardiovascular diseases like myocardial infarction (Baloglu et al., 2019; Mahendran et al., 2021), hypertension (Martinez-Ric et al., 2021), atrial fibrillation (Siontis ct al., 2021; Feeny et al., 2020), arrhythmia (Feeny et al., 2020; Pant et al., 2022), cardiac death (Ko et al., 2020), heart failure (Bos et al., 2021), etc.
However, crucial to note that many cardiovascular health-related diagnoses are based on continuous value predictions, for instance, hyper/hype syper/bypo-kalemia is basod on scrum potassium concentration (Webster et al,, 2002), low ventrieular-ejectinn fraction is based on ejection fraction (Yao el al., 2020), etc. This underscores the need for newer methods that effectively tackle the regression problem. Moreover, in critical applications, like healthcare, it is desirable to quantify the uncertainty in the predictions made it can prevent fatal decision- making and improve the overall outcomes (Chen et al., 2021; Tanno et al., 2021; Rangnekar et al., 2023; U Upadhyay et al., 2021b). But, the above intersection has not been studied in the context of medical time series like ECGs. We explore this gap in the literature and provide effective solutions for the same.
3 Methodology
We first formulate the problem in Section 3.1. Preliminaries necessary to motivate the design choices for our framework (HypUC) are presented in Section 3.2.1 (on Imbalanced Regression), Section 3.2.2 (on Kernel Density Estimation), and Section 3.2.3 (on Uncertainty Estimation). In Section 3.3, we construct HypUC, that performs uncertainty-aware regression followed by a new calibration technique that significantly improves the quality of uncertainty estimates and utilizes simple technique based on gradient-boosted decision trees to enhance decision-making using the predicted point estimate and well-calibrated uncertainty estimate.
3.1 Problem formulation
Let D y(),, be the training set with pairs from domain X and Y (i.e., x; E X,y, E Y,W), where X, Y lies in Rm and I., respectively. While our proposed solution is valid for data of arbitrary input dimension, we present the formulation for medical time series (like ECGs) with applications that predict a desired physiological parameter helpful in making a diagnosis using ECG. Therefore, (x, represents pair of ECG and physiological ECG signal, and y; is the serum potassium level (indicating the amount of potassium in the blood, which may be obtained from the blood test that serves as groundtruth labels for training).
For instance, to aid in diagnosing Hyperkalemia, x, is
We aim learn mapping, say ((): X + P(Y), that learns to map a given input ECG (x) to a probability density function P(y), indicating the probable values. We use deep neural network (DNN) to parameterize V with 0 and solve optimization problem to obtain optimal parameters 0*, described in Section 3.3.
3

---

Published in Transactions on Machine Learning Research (08/2023)
3.2 Preliminaries
This section presents the preliminary concepts highlighting the current challenges, existing literature, and some techniques used to construct our proposed framework, HypUc. In particular, Section 3.2.1 discusses the problem of imbalanced regression that is very typical in real-world clinical scenarios and some solutions to tackle the same using machine learning. Section 3.2.3 presents the existing framework often used to estimate the uncertainty in the predictions made by the deep neural network and the shortcomings associated with such methods. The culmination of ideas presented in this section helps us construct our method.
3.2.1 Imbalanced Regresison
Imbalanced regression refers to the problem of training machine learning model for regression a dataset where the target variable is imbalanced, meaning that specific values appear more often than others. This is lypical for clinical datasets where different medical conditions appear with different prevalences. For instance, Figure 3 shows the density and the distribution of Scrum Potassium Levels (i.c., the concentration of potassium in the blood) and left ventricular ejection fraction (i.e., the volune of blood pumped out of the left ventricle of the heart relative its total vol- ume) for real-world dataset. We notice that most of the patients have serum potassium level in the range of 3.5 to 5.5 (which is of- ten accepted as the range for normal potassium level). However, it crucial for machine learning models to work well in the high potassium levels (defined above 5.5) that occur rarely, i.c., the pop- ulation density in the region very low. A similar trend is observed for IVEF, where most of the data is concentrated in the healthy range (between 40-60%), but low-LVEI regions (defined below 40%), which is essential for the model to identify, are sparsely present in the dataset. Imbalanced datasets can present challenges for training regression modcls, as the model may be biased towards the densely populated samples. Works by Yang el. al. (2021); Steininger et al. (2021) proposes distribution smoothing for both labels and features, which explicitly acknowledges the effects of nearby targets and cal- ibrates both labels and learned feature distributions lead to better point estimates in sparsely populated regions. However, the hod does not explore the application to healthcare, where uncertainty quantification is critical, and the extension to incorporate uncer- fainty estimation a similar framework is non-trivial. Our proposed HypUC proposes a method to tackle imbalanced regression using deep neural networks that also incorporates a novel technique for calibrated uncertainty quantification.
179000
Nunbeer tampls
100000
25009
10
12
Potastium Lavd mmmbl.)
14800
Nanber of sumples
2010
Ejection de
Figure 3: Imbalanced distribution of target values (top) Serum Potassium Levels and (bottom) Ejection Fraction in real-world medical datasets.
3.2.2 Kernel Density Estimation
Kernel density estimation (KDE) (Terrell & Scott, 1992; Chen, 2017) is the application of kernel smoothing for probability density estimation. Given set of i.i.d. samples y2.. drawn from some unknown den- sity function f. The kernel density estimate provides estimate of the shape of f. This can be leveraged to train deep regression mod- els with imbalanced dataset by estimating the density for each target value in the training set and use this density to create a weight- ing scheme be applied with the loss for that sample. This approach can help to give more
0000
Numbor of Sanelos
Inverso Densty Welgh
120
1000
100
4000
86
8
2000
8
10.0 12.5
50
10.0 12.5
15.0
Data Values
Data Values
Figure 4: KDE and related weights for imbalanced targets. (left) Example of imbalanced target datascts with KDE. (right) Inverse of KDE with exponent used as weights for the loss term. Higher exponent leads to sharper weights.
4

---

Published in Transactions on Machine Learning Research (08/2023)
importance to samples with rare target values, which are often more difficult to predict. This is achieved by estimating the density as:
Ph (y) 1
Cy yz) n Σ
K
y Yi
(1)
nhh
where K is the kernel -ornel-aa a non-negative function and h > O is a smoothing parameter called the bandwidth. Scaled kernel Ks. is defined as, (y) x x and K. is often set to standard normal density function. The weighting scheme can be designed by leveraging the fact that lover-density samples must receive higher weights and vice-versa, a simple technique to achieve the same to design the weight for target value y be w(y) (y)* Figure 4-(left) shows an example of imbalanced target distribution and Figure 4-(rihht) shows the designed weighting scheme, we note that hyperparameter X2 controls the sharpness of the weights.
3.2.3 Uncertainty Estimation
The works in (Kendall & Gal, 2017; Laves el al., 2020b; Upadhyay et al., 2022; 2023; Sudarshan et al., 2021) discuss the key techniques to estimate the irreducible (i.e., aleatoric) uncertainty, and reducible (i.e., epistemic) uncertainty, using deep neural networks and also highlight the importance of capturing alcatoric uncertainty the presence of large datasets. For an arbitrary deep neural network (DNN), say (:; parameterized by learnable parameters ñ, that learns a probabilistic mapping from input domain X and output domain Y (as defined in Section 3), i.e, p(::)) X To estimate the aleatoric uncertainty, the model must estimate the parameters of the conditional probability distribution, Pyxx, which is then used to maximize the likelihood function. This distribution is assumed to be parametrized by set of parameters {fr, P/1, P...... Dxx, } which we need to estimate (e.g., for Laplace distribution the set of parameters is given by location and scale, {1, 6) (Ilg et al., 2018), for Gaussian distribution the set of parameters is given by mean and
ie, {u, of (Kendall & Gal, 2017), for generalized Ganssian the set of parameters is given by location, scale, and shape parameters {μe, a, 0 (Upadhyay et al., 2021a)). That is, for an input x;, the model produces a set of parameters representing the output given by, (9:, 221,,,,,.... .. D.a (xx; Q) that characterizes the distribution Pyix (y; (&r, 02,......... ba, such that y, ~ Prxx (y: 15 Dri, P:2,... Dis }). The likelihood D): == II Py (y:; {Pi, Dri, "i,,0.....De DaH)) is then maximized to estimate the optimal parameters of the network. Moreover, the distribution Prix is often chosen such that uncertainty can be computed using a closed form solution I of estimated parameters, i.e.,
{yr, D2... ... Du) (xx;:)) with Ç* : argmax L((;D) -argmax II Prix (yn 24, D.x))
(2)
Uncortainty($.) = (Dii, Diz,... Dsk))
(3)
It common to use a heteroscedastic Gaussian distribution for PYΙx (Kendall & Gal, 2017: Laves et al., 2020b), in which case (::{) is designed predict the mean and variance of the Gaussian distribution, i.e., (8,,6])= (xx; {), and the predicted variance itself can be treated uncertainty in the prediction. The optimization problem becomes,
19 2912
argmin 18: - y
N
log(d?)
5 argmax <
II
26
(4)
2To²
5
26²
2
1=1
Uncertainty($5) 0².
(5)
Work by Laves et al. (2020a) discusses that uncertainty estimated using above technique is not well cali- brated and provides a remedy for the same. They demonstrate that learning a scaling factor for the estimated uncertainty a post-hoe optimization step on validation data (i.e., after training) help improve the cali- bration. However, this technique learns a single scaling factor, that not optimal for imbalanced datasets (e.g., clinical datasets as discussed in Section 3.2.1). We propose to alleviate this problem by proposing a hyperfine uncertainty calibration scheme discussed in Section 3.3.2.
5

---

Published in Transactions on Machine Learning Research (08/2023)
3.3 Building HypUC
In this section, we leverage the preliminary concepts discussed above to construct our proposed HypUc. We explain the building blocks for our framework, which consists of three major steps: (i) Optimization for improved contimuous-value point and uncortainty estimation, (ii) Post-training hyperfine uncertainty calibration, (iii) Improved decision-making with an ensemble of gradient- learners.
3.3.1 Optimization for Improved Continuous-value Point and Uncertainty Estimation
As described in Section 3.1, we aim to learn the estimation function (:; Here we design the network (=: to model the output as a Gaussian distribution as thoroughly discussed in previous works (Kendall & Gal, 2017; Laves et al., 2020b; Upadhyay et al., 2021b). That is, the DNN v takes the medical time series as the imput and predicts the probability density function parameters for the output, iee, L (x; 0) = {V,0) = {[ [ (x,; 9). [ (x;: }. Where (x;; 0)lg is the y; estimate given by the network and [ (x;; A)ls is the og estimate given by the network. We propose to learn the optimal parameters of the DNN, 0*, by:
8" argmin N A1
1
Σ
{
( - + Xs * (1 (1(1V(X 0) -2 + log (X: L (xx;; 0)13
)} (6)
Where (y) is the Gaussian kernel density estimate (fit on the training labels) for the target value y. d2; As h are the hyperparametens, controlling the strength of imbalance correction and its relative con- tribution with respect to uncertainty-based regression terms as detailed below.
In the above optimization, the term (x,:))))))))) computes the discrepancy between the continuous value prediction and the groundtruth, where the discrepancy is weighed down if the groundtruth label is present in abundance, i.c., density (y) is high. The strength of the weighting term is controlled by the hyper- parameters h, A2. The discrepancy is weighed up if the label density is low, addressing the imbalanced nature of the datasot. The term, (K(*((x))))
Ph (y:))²²
+ log (V(x;;)))) likelihood of heteroscedastic Gaussian distribution assumed for each prediction. The hyperparameters Å1, As control the relative contribution of the density-basod discrepancy term and the heteroscedastic negativo-log- likelihood term that learns to estimate the uncertainty.
in the above optimization, is the nogative-log-
(xx,00)1
3.3.2 Post-training Hyperfine Uncertainty Calibration
Once the DNN is trained, ie., the optimal parameters 0 have been found as described in the above op- timization step. The trained DNN, (:8"), is capable of estimating both the target continuous value and the uncertainty in the prediction. However, the uncertainty estimates obtained are biased and are not well calibrated (Laves et al., 2020a; Bishop & Nasrabadi, 2006; Hastie et al., 2009). Therefore, we propose a new tochnique to calibrato the uncertainty estimates. We first find a global optimal scaling factor s" by solving the following optimization problem on the validation set Deattd = {(x, ((x,y)))
where the parameters of the DNN are fixed to the optimal obtained above (i.c., 8*) as described in (Laves et al., 2020a),
= argmin M (Mlg(()) +
1 M
([,) ] - y:) 4 [x,;; 13
25
We then run the model in inference mode and genorate the estimates for all the samples in the validation set, i.e, we get a new augmented validation set, Dealid-aug = {x, y,, Rr, 0.,6?), )- We then create hyperfine bins of uniform length spanning the entire range of predicted continuous- valuee, i.e., we create int ( (Ymar= bins, where 5 << Ymar - Ymin leading to hyperfine bins, where Ymar mar((y)):=1.) min(tyyhenn
and Ymii The thth bin being B, spanning [y man + (n 1)6, ymi +no]. Each sample in the validation set then belongs to one of the bins. We then find bin-wise scaling factor for each of the above hyperfine bins by solving an optimization problem. The bin-wise scaling factor for bin BR, mB., is given by,
(en(11): yıl < nss delyiEn.), >E)
BB = argmin
(8)
len(Ba)
7
6

---

Published in Transactions on Machine Learning Research (08/2023)
Learning Gradient-boosted Forests
Denisd-ang Yx., Sr. 0f, Pim
Apphy Y (.)
Enoor
Diagneatic (assistiv) Decision
Calbcatian
g calib
Brpectee Tarenakey Calteation
AR
{8, 0) (x; calih ()Bin())
Borations
based ensemble of decision trees incorporating calibrated uncertainty (via both global and hyperfine calibration) and point estimate to help in diagnostic decision making.
Figure 5:
Where, len(-) is the function that gives the number of elements in the imput set and hyperparameter é represents the minimum acceptable fraction in cach bin where the discrepancy between the prediction and the groundtruth (i.., yl) is captured by the scaled uncertainty estimates (i.e., 118" āz)) The above scaling factors allow us to compute the well-calibrated uncertainty estimates for each prediction. Given input x, the continuous value prediction (y) and the calibrated uncertainty (ōcals)) are given by,
{5. = (x; 0*) and éealit "]Binly)s'
(9)
Where, Byn(y) is the hyperfine bin to which belongs ()ii(( is the bin-wise scaling factor of the bin.
3.3.3 Improved Decision- Decision-making with an Ensemble of
Learners
Medical professionals often diagnose the condition based on continuous value lying a cortain standardizod range (Galloway el al., 2019; Bachtiger et al., 2022). We propose technique that utilizes both the predicted continuous value and calibrated uncertainty estimate, obtained in the above step, to infer the standardized range (that is better than relying just the continuous-wer value point estimate). The proposed technique uses predictions (both ÿ and &colib) from the trained DNN (I) as input features to an ensemble of decision trees that uses a gradient-boosting technique to learn a random forest (represented by Y with Ntrees trees and deree as the maximum possible depth for each tree) on dataset Dtrain U Deolid, that infers the standardized- range/class class in which the predicted contimuousvaalte belongs, accounting for the uncertainty in the predictions, range/
Class(x) Y (( )1 Biin(( (xx; (x))))))
(10)
In the above, note that the regression model also allows classification. This can be achieved by imposing interpretation rules on the predicted continuous values (either explicitly by thresholding the continuous value predictions for different classes implicitly by learning a classifier on top of predicted continuous values and other features, practitioners to not only know about the severity of the condition (indicated by the continuous value), but also use it to infer the broad category (by deriving the class disease class using continuous value as described).
described in this section). Therefore, this approach could potentially allow medical
4 Experiments and Results
4.1 Tasks and Datasets
Our dataset is curated at Mayo Clinic, USA and consists of over 2.5 million patients, with over 8.9 million 12-lead ECGs coming from different types of hardware, covering a wide range medical conditions, age, and demographic variables. The data. collection is from the years 1980 2020. The retrospective ECG- AI research described herein has been approved and determined to be minimal risk by the Mayo Clinic Institutional Review Board (IRB 18-008992).

---

Published in Transactions on Machine Learning Research (08/2023)
4.1.1 Estimating Age from ECG
Estimating age from an electrocardiogram (ECG) has gained increasing attention in recent years due to the potential for non-invasive age estimation to aid in diagnosing and treating age-related conditions. Moreover, ECGs have been shown to indicate cardiovascular age, which is helpful in determining the overall cardiovas- cular health (van der Wall et al., 2022; Chang et al., 2022; Baek et al., 2023). Previous works like (Attia et al., 2019) have shown that using deep learning, it is possible to analyze the ECG signals such that the neural network can identify patterns in the ECG waveform that change with age, which can be used to estimate an individual's cardiovascular age. For our experiments, we curated a diverse datasct of 8.8 million ECGs (along with the patient's 's age). We used 6.7/0.75/1 75.35 million for train/ven /aa//val//eeeee datasets.
4.1.2 Estimating Survival (time-to-death) from ECG
Estimating survival, or time to death, from an electrocardiogram (ECG) is a relatively new field of study that has gained momentum in recent years as it may aid the treatment and management of life-threatening conditions (Diller al., 2019; Tripoliti et al., 2017). We curate the data to consist only of patients for which the death has been recorded. The time interval from ECG recording to death for such patients provides groundtruth for survival (time-to-death) In total, there were 3 million ECGs with the corresponding time- to-death, we used 25/.266/0.44 million for train/val/ten sets.
4.1.3 Estimating Level of Potassium in Blood from ECG: Detecting Hyperkalemia
Serum potassium level measures the amount of potassium in the blood. Potassium is an electrolyte essential for properly functioning the body's 's cells, tissues, and organs. It helps regulate the balance of fluids in the body and maintain normal blood pressure, among other functions. The normal range for serum potassium levels typically between 3.5-5.0 millimoles per liter (mmol/L.). Levels below 3.5 mmol/T. are considered to be low (hypokalemia), and levels above 5.0 mmol/L are considered to be high (hyperkalemia). Both low and high potassium levels can have serious health consequences. While a blood Lest is used measure the serum potassium level, recent works have evaluated machine learning techniques to infer the potassium level using ECGs. This can play critical role in remote patient monitoring (Galloway et al., 2019; 2018; Feeny et al., 2020; Alm et al., 2022). In total, there are over 4 million ECGs with the corresponding serum polassium level (derived using blood test). We used 2.1/0.4/1.7 million for vrainval/ttess sets.
4.1.4 Estimating Left Ventricular Ejection Fraction from ECG: Detecting low LVEF
Low left ventricular ejection fraction (LVEF) is a condition which the left. ventricle of the heart is unable to pump a sufficient amount of blood out of the heart and into the body. The left ventricle is the main pumping chamber of the heart and is responsible for pumping oxygenated blood to the rest of the body, LVEF is typically measured
percentage, with a normal range being between 55 - 70%. A LVEF below 40% is generally considered to be low and may indicate heart failure. LVEF can be evaluated through various non-invasive tests, including an cchocardiogram, a nuclear stress test. But ECG can still provide valuable information about heart. function and can be used in conjunction with other tests to assess LVEF and diagnose heart conditions (Noseworthy et al., 2020; Vaid et al., 2022). In our experiments, we leverage DNNs to infer the LVEF- this can help roduce the costs associated with overall diagnosis and treatment by reducing the need for more expensive Echocardiogram. In total, there are over 0.6 million ECGs with the corresponding IVEE (derived using Echo). We used 13/0.1/0.2 million for train/val hal//all//tete teettstt sets.
4.2 Compared Methods and Training details
To perform a comparative study with 12-lcad ECG time scries, we create a backbone DNN architecture adapting ResNets into a 1D version (with 1D convolutiona layers) as discussed in previous works (Attia et al., 2019). The DNN backbone architecture is used to create a class of (i) deterministic regression models, (ii) class of probabilistic regression models, and (iii) class of classification models for some of the tasks. For the probabilistic regression, we append the backbone architecture with two smaller feed-forward heads estimate the parameters of the predictive distribution as described in (Kendall & Gal, 2017; Upadhyay et al.,
8

---

Published in Transactions on Machine Learning Research (08/2023)
2022). The deterministic regression model is trained with (i) L2 loss, termed Regres. (ti) L1 loss termed as Regres -L1, (iii) We incorporate the density-based weights to modulate the loss terms, we call this SLgress-L11KDDA We train two variants of probabilistic regression models (i) the standard heteroscedastic Gaussian model that estimates the mean and variance for the prediction and trained to maximize the likelihood as described in (Von Bachmann et al.; 2022), called Regres. Regres.-w.-U and (ii) Our proposed model HypUC, that incorporates a novel calibration technique along with gradient-boosted corrections. For serum potassium level estimation (and LVEF estimation), we also compare with a binary classifier that predicts whether the given ECG has Hyperkalemia (and low LVEF for LVEF estimation). All the regression models are trained on standardized target values, i.e., (target Mtarget) ( We also present the regression models trained with standardized log targets, iee, (log(target) Mlog(target)) (drost)))))) (ooo(target)). We train all the models using Adam optimizer (Kingma & Ba, 2014), with (31, (22) set to (0.9,0.999) The learning rate used to train models was 1e 4. For HypUc, the
{Az, h, X2, )3} are set to (i) {1, 0.8, 1e - 3} for Age estimation. (ii) {1, 0.5, 0.5, 1e-4] 4} for Survival estimation. (iii) {1, 1, 0.2, 43 for Scrum potassium estimation. (iv) {1, 1.5, 0.1, le 4} for Left Ventricular Ejection Fraction (LVEF) estimation.
Tusks
Methods
MSE 1. MAE
Spear. Corr.
Pears. Corr.
UCE
NLL 1 Iumo 1 <leniZuns] 2
AUC
Sensiei. t Specifi, f PPV f NPV f
Regres, +L.2
100.12
8.49
0:30
0:30
NA
NA
NA
NA
NA
NA
NA
NA
NA
Regrea.-Li1
73.21
7,78
0.38
NA
NA
NA
NA
NA
NA
NA
NA
Survical
Listinationn
Blggree-. (log)
61.33
NA
NA
-L1-ea.-L1--0E4 -L1-KDE4
67.97
6.30
0.51
NA
NA
Bogres.-x
81.62
8.13
0.35
0.38
2.37
5.89
0.89
20.5
NA
NA
NA
NA
NA
-2.- (log)
82.23
8.06
0.36
0.38
2.14
5.22
0.90
19:2
NA
NA
NA
NA
NA
Hyplic
54.62
5.88
0.56
0.61
0.58
8.45
0.93
13.4
NA
NA
NA
NA
NA
ByptC (log)
55.77
5.52
0.52
0.59
0.00
3.62
0.91
15.6
NA
NA
NA
NA
NA
Regres.-L2
151.09
9.47
0.58
NA
XA
NA
NA
NA
NA
NA
NA
Regres, -L1
116.52
0.73
NA
NA
NA
NA
Age Extimation
Blggree.. (log)
101.58
NA
NA
NA
NA
Regres. -L1-KDE4
90.77
0.78
NA.
NA
NA
NA
NA
Bogres.-x.-D
136.27
8.22
0.68
0.72
13,32
7.84
0.88
17.6
NA
NA
NA
NA
NA
-2.-1 (log)
138.23
8.56
0.68
0.71
12.96
7:23
0.91
17.1
VA
NA
NA
NA
NA
BypuC
74.9
6.70
0.84
0.88
1.06
3.70
0.92
11.3
NA
NA
NA
NA
NA
HyptIC (log)
76.5
7.18
0.81
0.84
1.37
3.93
0.91
14.8
NA
NA
NA
NA
NA
Elsary-Claasirier
NA
0.39
NA
NA
0.87
0.74
0.85
0.99
Serui Protacatam
Regres, LL2
0.26
NA
0,70
Esstioation tom
Ragrea.-L11
0.19
0.52
NA.
NA
0.70
0.87
0.99
Begree. -11 (log)
0.21
0.39
0.48
0.50
NA
NA
NA
NA
0.85
0.71
0.84
0.10
0.99
Begree -L1-KDE>
0.19
0.33
0.52
0.55
NA
NA
NA
NA
0.86
0.72
0.85
0.11
0.99
0.23
0.38
0.51
0.55
1.83
8.76
0.80
5.1
0.86
0.71
0.86
0.10
0.00
-2.-U (log)
0.23
0.37
0.51
0.55
1.97
8.82
0.89
5.0
0.56
0.71
0.88
0.10
0.99
RypuC
0.29
0.32
0.52
0.55
0.41
3.12
0.94
3.1
0.89
0.77
0.86
0.10
0.99
Byptic (log)
0.21
0.36
0.52
3.8
0.88
0.75
0.84
0.99
Btnary-Clanett Claee--Cl
NA
0:28
NA
0.77
Rivetion Froction Essimation
Ragrea.-112
171.37
13.56
0.37
NA.
NA
0.50
0.75
0.84
Regres.-Li -L1
153.66
11.78
0.11
0.42
NA
NA
NA
NA
0.91
0.76
0.88
0.10
0.98
Regrea.-1h (log)
138.32
10.23
0.44
0.49
NA
NA
NA
NA
0.91
0.75
0.86
0.10
0.98
Begres. -L1-HDE>
141.62
10.71
0.11
0:43
NA
NA
NA
NA
0.91
0.75
0.87
0.09
0.99
165.81
12.24
0.41
0.44
10.56
9.83
0.89
36.1
0.50
0.74
0.85
0.10
0.99
Begres, -1 (log)
150.33
11.25
0.40
0.43
8.11
8.36
0.90
30.3
0.90
0.73
0.85
0.10
0.90
Rypuc
133.26
0.43
0.91
26.2
0.77
0.87
0.99
IlypUIC (log)
139.32
0.12
29.6
0.92
0.75
Quantitative results on 4 different tasks (Survival, Age, Scrum Potassium, and Left Ventricular Ejection Fraction Estimation) showing the performance of various deterministic and probabilistis regression and classification models in terms of Mean Squared Error (MSE), Mcan Absolute Error (MAE), Spearman Correlation (Spear. Corr.), Pearson Correlation (Pear. Corr.), Uncertainty Calibration Error (UCE), Nega- tive liglliielihood (NLL), Io.95, len(Io.g3) > AUC, Sensitivity, Specificity, Positive /eeetivveeee Predictive Values PPV/NPV. 1f, for task/method certain metric is not applicable, we show NA.
Table 1:
9

---

Published in Transactions on Machine Learning Research (08/2023)
10-
Screm Potasiimm Level (Cronnd truth)
Least uncertain
Moderata uncertain.
Most uncertain
-2
0.0 0.1
0.2
0.5
0.4
0.5
2
3
4
8
9
10
Predicted seores from binary
Predicted scores from binory classifiers (post-sigmoid)
Prodicted values from probubilistic regresser (lyyUUC)
classifiers (pre-sigmoid)
Figure 6: Scrum Potassium Estimation model score corrolation with groundtruth Serum Potassium levels. (left) Model scores (pre-activation) for the binary classifier. (middle) Model scores (post-activation) for the binary classifier. (right) Model scores for the proposed HypUC. The proposed HypUC not only produces better-correlated outputs but can also quantify the uncertainty estimates.
4.3 Results
Table shows the performance of various methods on different tasks and datasots. For all the regression tasks, we compute MSE and MAE to measure the discrepancy between the predictions and the groundtruth. Moreover, Spearman and Pearson correlation coefficients indicate the positive association between the pre- dicted point estimates from the network and the groundtruth target variable. A high correlation indicates that the prediction will be able to indicate the severity of the medical condition (which may help prescribe the treatment as explained in Section 1). For probabilistic models, the quality of uncertainty estimates is measured by UCE and NLL (Laves ct al., 2020a; Upadhyay et al., 2022). In addition, we also quantify the calibration of the estimated uncertainty using the interval that covers the true value with a probability of a (under the assumption that the predicted distribution is correct) is given by,
(J 8 (a) ++ Ca) 0 2
(11)
Where, (-)(() refers the quantile function, i.e., inverse of the cumulative distribution function, discussed thoroughly in (Kuleshov et al., 2018; Gawlikowski et al., 2021). In particular, set the a 0.95, which makes (0.95) = 1.645 for Gaussian distribution, and the length of the interval (8, becomes, len (f, (8,0) 0) C0.90))) = 1.6450. For good calibration, fraction of groundtruth samples should lie in the interval Io (y; 5) and the length of the interval, (a)(a),, should be small. We report both metrics in our evaluation. For the interval length, we report the mean interval length of the test set, represented by < len(Ta (y, à))= >. Finally, as discussed in Section 3.3.3, regression model also allows classification, which is the approach taken by some of the prior works. To compare them, for the binary classification tasks, we also report the AUC, Sensitivity, Specificity, and Dusitive/Negating Predictive values (PPV/NPV)
4.3.1 Task Specific Performance
As indicated Table 1, for Survival Estimation, the regression model trained with L,2 loss (Regres. -L2) has MSE of 100.12 (MAE of 8.49, Spearman/Peattee formance of the model trained with L1 loss (Regres. (MSE of 73.21, MAE of 7.78, Spearman/Peanee PPearson correlation of 0.38/0.41), this is in line with previous works. Moreover, we could further improve the performance of the regression model trained with L1 loss by incorporating kernel density-based weights (Regres. (Regres.-L11KDEx, -KD- as described in Section 3.3, that accounts for imbalance in the dataset and leads to a performance with MSE of 67.97 (MAE of 6.30, Spearman/Pearson correlation of 0.51/0.53). Additionally, we notice that heteroscodastic Gaussian probabilistic regression model (Regres. -U) from (Von Bach- mann al., 2022) performs better than homoscodastic Gaussian model (Regres. -L2) but not as good (Legres.-L1). However, Regres. estimates the uncertainty in the prediction (with UCE/NI.I
correlation of 0.30/0.30), which are worse than the per-
10

---

Published in Transactions on Machine Learning Research (08/2023)
Peak of Groundtruth dastr:
Peak of Callbrated Predicted distr. (HyppUC)
Spread
Spread Calibrated Predicted (HyptC)
30
Age Estimation
Survival Estimation
80
25
20
Values
Values
60
15
40
10
5
20
20
40
60
80
15 Predicted Values
20
30
Predicted Values
Serum Potassium Estimation
Ejection Fraction Estimation
80
10.0
60
Values
Values
7.5
40
5.0
20
2.5
2
3
4
5
6
%
20
40
60
80
Predieted Values
Predicted Values
Figure Groundtruth and predicted distributions for different tasks AAge/Survival Potas- sium/LVEP estimation). Predicted distribution from HypUC tightly encompasses groundtruth distribution (compared to predicted distribution from Regres -U. -U) indicating predictions are well-calibrated.
of 2.37/5.89), but our proposed HypUC not only improves regression performance (with MSE/MAE of 54.62/5.38) but the quality of uncertainty as well (with UCE/NLL of 0.58/3.45). We observe a similar trend for the Age Estimation.
For Serum Potassium Estimation, we compared with binary classifier baseline (Binary-Classifier in Table 1), approach found in (Galloway et al., 2019), which predicts if given ECG corresponds to Hy- perkalemia not, instead of estimating the serum potassium level. While such binary classifier achieves good classification performance as indicated by an AUC of 0.87, it does not gauge the severity of the condition that may decide the diagnosis. Figure 6 shows the correlation between the ground truth serum potassium level compared to the scores derived from binary classifier (Figure 6-left and middle), the lack of high positive correlation (Spearman correlation of 0.39) indicates that scores from the model can not be used as an indication for the severity of the medical condition (within Hyperkalemia), but can be used to dis- criminate Hyperkalemia from non-Hyperkalemia cases. We also train different kinds of regression models to regress the serum potassium level. As indicated in Table outputs that are better correlated with the groundtruth (e.g., Regres L2 has Spcarman correlation cocf- ficient of 0.51 vs. 0.39 for Binary Classifier). Moreover, we observe that among deterministic regression models Regres.-Li -L1-KDEw performs the best with MSE 0.19 (MAE of 0.33). The probabilistic regressor Regres. -w. performs better than Regres -L2 and provides uncortainty estimates as well (UCE/NLL of 1.83/8.76). But, HypUC provides the best regression performance along with better uncertainty estimates. Additionally, the outputs from the regression models be used to classify the ECG into Hyperkalemia
that regression models generally produce
11

---

Published in Transactions on Machine Learning Research (08/2023)
not (similar to binary classifier), notice that all the regression models perform comparable classification to classifier. In partieular, HypUC has AUC of 0.89 which is higher than the binary classifier (AUC of 0.87).
A similar trend is observed for Left Ventriculer Ejection Fraction (LVEF) Estimation where the proposed HypUC performs well in regression along with better-calibratee uncertainty estimates. Moreover, when used as classifier, it also yields a better performance than the binary classifier trained from scratch, emphasizing the benefits of tackling this problem with the proposed probabilistic regression framework, HypUC.
4.3.2 Calibration of Uncertainty Estimates on the Test-set
Figure 7 shows the groundtruth and the predicted distribution for gge/Survival/Seuu Potassium/IIVB estimation tasks the test dataset. The x-axis in the plot indicates the predicted point estimates from the HypUC. For given point on x-axis: (i) Red bold curve shows the peak of the groundtruth distribution (i.e., the distribution of groundtruth values that correspond to predicted value given by the x-axis). (ii) Shaded rod region indicate the spread of the groundtruth distribution (i.c., 2 and 98 percentile of the groundtruth distribution). (iii) Blue bold curve shows the peak of the HypUC-predicted distribution. (iv) Blue shaded region indicates the spread of the groundtruth distribution (i.e., and 98 percentile of the groundtruth distribution). We notice that predicted distribution from HypUC tightly encompasses groundtruth distribution
(compared to predicted distribution from Regres -U) indicating predictions well-calibrated. We calibrate HypUC using small validation sets but it generalizes well to much larger test sets.
4.3.3 Entropy- based Filtering Trigger Human Expert Intervention
As the proposed HypUC produces well-calibrated predictive distribution, we propose technique to use the predicted distribution to flag the unreliable predictions. For given input sample x, the trained DNN produces () = (xx 0*). We then apply the post-hoc calibration techniques to obtain {y, Gealib). We then compute the entropy of the predicted distribution, i.e. N(y, 6 daaaa) given by H(z), i.e,
H(z)= = / p(()) log p(z)dz where a ~ N(y seat
(12)
H(z) - -Eoog N( o2aat))) = log v 2Tee calib
(13)
At the test time, if for the sample x the entropy log /2meō cala > Tq.val then the prediction flagged as unreliable. The threshold Tq is computed us- ing the validation set as the qtttile value from the distribution of entropy values as shown in Figure for LVEF estimation using HypUC. Figure 9 evalu- ates the regression performance of HypUC (in terms of MAE) on the test sets for various tasks by re- moving the unreliable predictions based on different
1750
Number of Samples
1000
Prediction with high-enteopy, predictions reliable
thresholds derived from the validation set. We ob-
serve that as we remove the higher fraction of unreli-
able samples (i.e., decreasing the threshold Tq,eal by lowering the value of 9) the performance improves. This phenomenon demonstrates that could en- hance the predictions made by the model by only showing reliable predictions and flagging the unre- liable predictions (based on entropy measure) to be evaluated by human experts which may be critical for deploying machine learning models for health- care applications in the real world.
250
2.0
3.5 Entropy
5.0
Figure 8: Distribution of entropy for LVEF valida- tion set with a threshold of 10%-tile.
12

---

Published in Transactions on Machine Learning Research (08/2023)
Survival Estimation
Age Estimation
Mcan Absolute Error (MAE)
Mean Absolute Error (MAE)
50
80
100
50
60
80
90
100
Entropy Threshold Quantile
Entropy Threshold Quantile
Serum Potassium Estimation
Left Ventricular Ejection Estimation
Absolute (MAE)
Mean Absolute Error (MAE)
10.5
0.30
10.0
0.25
9.0
0.20
8.5
0.15
0.10
7.5
90
Entropy Threshold Quantile
Entropy Threshold
Figure 9: Evaluating HypUC on test-set by removing the unreliable (i.e., high entropy) predictions. We see consistent improvement as we decrease the entropy threshold based on quantile as described in Scction 4.3.3
5 Discussion, Conclusion, and Future Work
Automated analysis of medical time series (e.g., ECG) using ML techniques opens several avenues in large- scale remote healthcare assistance while making efficient use of existing procedures. This work proposes HypUC to solve the imbalanced regression problems with medical time series along with uncertainly estimation. We also introduced a new hyperfine uncertainty calibration technique that provides reliable uncertainty estimates for diverse range of test sets. Additionally, we present a method for using these calibrated uncertaintics to improve decision-making through an ensemble gradicnt- boossttt learners. In our studios, we use the well-calibrated predictive distribution derived from HypUC in an entropy-based technique flag the unreliable predictions the inference time for human expert evaluation and improve the performance of large, real-world dataset of ECGs from millions of patients
the MI model. Our approach is demonstrated with various medical conditions, and is shown to outperform several baselines while also providing calibrated uncertainty estimates. Note that, while HypUC presents technical solution to the above- problems, it is crucial to explore the clinical impact of our work using real-workd validation studies. Additionally, it is important to extend our method to predict multiple target values in a single forward pass (i.e., multi-task learning) to avoid training models for different disoases and allow cross-task learning. Another important aspect for clinical deployment to design protocols that will help physicians make docisions the presence of quantified uncertainty, which much needed. These challenges are going to be explored in near future clinical study for hyperkalemia detection algorithm based
proposed method.
Broader Impact. The largo-scale rcal-world datascts from millions of patients were obtained from Mayo Clinic, USA following the guidelines for retrospective study with deidenlified dala approved by the institu- tional review board of Mayo Clinic, USA following the state-of-the-an identification protocol as described in Murugadoss al. (2021). While this study done on a large real-world anonymized dataset, the ML mod- els trained in this study must be evaluated for bias. Typically, formal clinical deployment of this technique will require validation and testing on demographically diverse datasets.
13

---

Published in Transactions on Machine Learning Research (08/2023)
References
Joseph C Ahn, Zachi 1 Attia, Puru Rattan, Aidan F Mullan, Seth Buryska, Alina M Allen, Patrick S Kamath, Paul A Friedman, Vijay H Shah, Peter A Noseworthy, et al. Development of the ai-cirrhosis-ocg score: deep learning model in cirrhosis. The American Journal of Gastroenlerology
An eleetrocarddoggra-bbee 117(3):424 432, 2022.
Zachi I Attia, Paul A Friedman, Peter A Noseworthy, Francisco Lopez-Jimenezz Dorothy J Ladewig, Gaurav Satam, Patricia A Pollikka, Thomas M Munger, Samuel J Asirvatham, Christopher G Scott, et al. Age and sex estimation using artificial intelligence from standard 12-lead eogs. Circulation: Arrhythmia and Electrophysioloyy, 12(9))e007284, 2019.
Patrik Bachtiger, Camille F Petri, Francesca E Scout Se Ri Park, Mihir A Kelshiker, Harpreet K Sahemey, Bianca Dumea, Regine Alquero, Pritpal S Padam, Isobel R Hatrick, et al. Point-of-care screening for heart failure with reduced ejection fraction using artificial intelligence during eeg-cnabled stethoscope examination in london, uk: a prospective, observational, multicentre study. The Lancel Digital Health, 4 (2) e125, 2022.
Yong-Soo Baek, Dong- Ho Lee, Yoonsu Jo, Sang-Chul Chul Lee, Wonik Choi, and Dae-Hyeok Kim. Artificial intellieenc-eestimated biological heart age using a 12-lead electrocardiogram predicts mortality and car- diovascular outcomes. Frontiers in cardiovascular medicine, 10, 2023.
Clas Baran Baloglu, Muhammed Talo, Ozal Yildirim, Ru San Tan, and U Rajendra Acharya. Classification of myocardial infarction with multi-lead eog signals and deep cnn. Pallern Recognilion Lellers, 122:23 30, 2019.
Christopher M Bishop and Nasser M Nasrabadi. Pallern recognalion and machine learning, volume Springer, 2006.
J Martijn Bos, Zachi I Attia, David E Albert, Peter A Noseworthy, Paul A Friedman, and Michael J Ackerman. Use of artificial intelligence and deep neural networks in evaluation of patients with electrocar- diographically concealed long qt syndrome from the surface 12-lead electrocardiogram. JAMA cardiology. 6(5):532-55) 538, 2021.
Francisco M Calisto, Alfredo Ferreira, Jacinto C Nascimento, and Daniel Gonçalves. Towards touch-based medical image diagnosis annotation. In Proceedings of the 2017 ACM International Conference on Inter- active Surfaces and Spaces, pp. 390-395, 2017.
Francisco Maria Calisto, Nuno Nuncs, and Jacinto C Nascimento. Modeling adoption of intelligent agents in medical imaging. International Journal of Fleman-Comppee Studies, 168:102922, 2022.
Francisco Maria Calisto, João Fernandes, Margarida Morais, Carlos Santiago, João Maria Abrantes, Nuno Nunes, and Jacinto C Nascimento. Assertiveness based agent communication for personalized medicine medical imaging diagnosis. In Proceedings of the 2023 CHT Conference Human Factors in Computing Systems, pp. 1-20, 2023.
Julià Camps, Blanca Rodríguez, and Ana Mincholé. Deep learning based qrs multilead delineator in elec- trocardiogram signals. 2018 Compuling in Cardiology Conference (CinC), volume 45, pp. 1-4. IEEE, 2018.
Chiao- Haso-Hsiang Chang, Chin-Sheng Lin, Yu-Sheng Luo, Yung- Tsai Lee, and Chin Lin. Electrocardiorran- based heart age estimation by a deep learning model provides more information on the incidence of cardiovascular disorders. Frontiers in Cantiovascular Medicine, 9, 2022.
Irene Y Chen, Shalmali Joshi, Marzyeh Ghassemi, and Rajesh Ranganath. Probabilistic machine learning for healthcare. Annual Review of Biomedical Data Science, 4:393-415, 2021.
Yen- Ven-Chi Chen. tutorial on kernel density estimation and recent advances. Biostatistics & Epidemiology, 111:1611187, 2017.
14

---

Published in Transactions on Machine Learning Research (08/2023)
Gerhard-Paul Diller, Aleksander Kempny, Sonya V Babu-Narayan, Marthe Henrichs, Margarita Brida, Anselm Uebing, Astrid E. Lammers, Helmut Baumgartner, Wei Li, Stephen J Wort, et al. Machine learning algorithms estimating prognosis and guiding therapy in adult congenital heart disease: data from a single tertiary centre including 10 019 patients. European heart journal, 40(13):1069-1077, 2019.
Zahra Ebrahimi, Mohammad Loni, Masoud Daneshtalab, and Arash Gharchbaghi. A review on deep learning methods for ecg arrhythmii classification. Expert Systems with Applications: X, 7:100033, 2020.
Johannes Enevoldsen, Gavin L Simpson, and Simon T Vistisen. Using generalized additive models decompose time series and waveforms, and dissect heart- lung interaction physiology. Journal of Clinical Monitoring and Computing, 37(1):16--177, 2023.
Albert K Feeny, Mina K Chung, Anant Madabhushi, Zachi Attia, Maja Cikes, Marjan Firouznia, Paul A Friedman, Matthew M Kalscheur, Suraj Kapa, Sanjiv M Narayan, et al. Artificial intelligence and machine learning in arrhythmias and cardiac electrophysiology. Circulation: Arrhythmia and Electrophysiology, 13 (8):0007952, 2020.
Conner D Galloway, Alexander V Valys, Frank L Petterson, Vivek P Gundotra, Daniel L Treiman, David E Albert, John J Dillon, Zachi Attia, and Paul Friedman. Non-invasive detection of hyperkalemia with a smartphone elettrocardiogram and artificial intelligence. Journal of the American College of Cardiology, 71(11S):A272 A272, 2018.
Conner D Galloway, Alexander V Valys, Jacqueline Shreibati, Daniel L Treiman, Frank L Petterson, Vivek P Gundotra, David E. Albert, Zachi T Attia, Rickey Carter, Samuel J Asirvatham, et al. Devel- opment and validation of a deop-learning model to screen for hyperkalcmia from the electrocardiogram. JAMA cardiology, 45):422 436, 2019.
Jakob Gawlikowski, Cedrique Rovile Njicutcheu Tassi, Mohsin Ali, Jongscok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty in deep neural networks. Xixi preprint arXiw:2107.03342 2021.
Chuang Han, Wenge Que, Songwei Wang, Jinying Zhang, Jie Zhao, and Li Shi. Qrs complexes and t waves localization in multi-lead eeg signals based on deep learning and electrophysiology knowledge. Expert Systems with Applications, 199:117187, 2022.
Awni Y Hannun, Pranav Rajpurkar, Masoumeh Haghpanshi, Geoffrey H Tison, Codie Bourn, Mintu P Turakhia, and Andrew Y Ng. Cardiologist-Ievel arrhythmia detection and classification in ambulatory electrocardiograns using a deep neural network. Nature medicine, 25(1) (5(1):65-6 ):65-6666 69, 2019.
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
Fddy Tlg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, and Thomas Brox Uncertainty estimates and multi-hypothesos networks for optical flow. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 652 667, 2018.
Alex Kendall and Yarin Gal. What uncertaintics do we need in bayesian deep learning for computer vision? NIPS, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic oplimization. ar Xív preprinl arXiv:1412.696 2014.
Wei-Yin Ko, Konstantinos C Siontis, Zachi Attia, Rickey Carter, Suraj Kapa, Steve R Ommen, Steven J Demuth, Michael J Ackerman, Bernard J Gersh, Adelaide M Arruda-Olson et al. Detection of hypertrophic cardiomyopathy using convolutional neural network-enabled electrocardiogram. Journal of the American College of Cardiology, 75(7):722-733 2020.
Volodymyy Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning, pp. PML.R, 2018.
15

---

Published in Transactions on Machine Learning Research (08/2023)
Max-Heinrich Laves, Sontje Ihler, Jacob F Fast, Lüder A Kahrs, and Tobias Ortmaier. Well-calibrated regression uncertainly in medical imaging with deep learning. In Medical Imaging with Deep Learning, pp. 393 412. PMLR, 2020a.
Max-Heinrich Laves, Sontje Ihler, Karl-Philipp Kortmann, and Tobias Ortmaier. Calibration of model uncertainty for dropout variational inference. arXiv preprint arXiv:2006. 11584, 2020b.
Hongqiang Li, Danyang Yuan, Youxi Wang, Dianyin Cui, and Lu Cao. Arrhythmia classification based on multi-domain feature extraction for an eog recogmition system. Sensors, 16(10):1744, 2016.
Rakesh Kumar Mahendran, V Prabhu, Parthasarathy, and A Mary Judith. Deep learning based adaptive recurrent neural network for detection of myocardial infarction. Journal of Medical Imaging and Health Informatics, 11 (112))30044-3063 2021.
Erick Rioss, Luis Montesinos, Mariel Alfaro-Ponce, and Leandro Pecchia. A review of machine learning in hypertension detection and blood pressure estimation based clinical and physiological data. Biomedical Signal Processing and Control, 68: 68:102813, 2021.
Karthik Murugadoss, Ajit Rajasekharan, Bradley Malin, Vineet Agarwal, Sairam Bade, Jeff R Anderson, Jason L Ross, William A Faubion, John D Halamka, Venky Soundararajan, et al. Building a best-in-class aulomated de-identiiication tool for electronic health records through ensemble learning. Pallerns, 2(6),
2021.
Balamurali Murugesan, Vignesh Ravichandran, Keerthi Ram, SP Preejith, Jayaraj Joseph, Sharath Shankaranarayana, and Mohanasankar Sivaprakasam. Ecgnet: Doep network for arrhythmia classifi- cation. In 2018 IEEE International Symposium on Medical Measuremends and Applications (MeMeA), pp. 1-6. IEEE, 2018.
Peter A Noscworthy, Zachi Attia, LaPrincess C Brewer, Sharonne N Hayes, Xiaoxi Yao, Suraj Kapa, Paul A Friedman, and Francisco Lopez-Jimenez. Assessing and mitigating bias in medical artificial intelligence: the effects of race and ethnicity on deep learning model for eog analysis. Circulation: Arrhythmia and Electrophagsiology, (300000 2020.
Himanshu Pant, Hitesh Kumar Dhanda, and Sachin Taran. Sleep apnea detection using electrocardiogram signal input to fawt and optimize ensemble classifier. Measurement, 189:110485, 2022.
Pranav Rajpurkar, Awni Y Hannun, Masoumeh Haghpanahi, Codie Bourn, and Andrew Y Ng. Cardiologist- level arrhythmia detection with convolutional neural networks. arXie preprint ar arxiv: 1707. fl.:1707.0136, 2017.
Vikrant Rangnekar, Uddeshya Upadhyay, Zeynep Akata, and Biplab Banerjce. Usim-dal: Uncortainty-awaro statistical image modeling-basee based dense active learning for super-resolution The Conference on Uncertainty in Artificial Intelligence (UAT), 2023.
Mike Schackermann, Graeme Beaton, Elahch Sanoubari, Andrew Lim, Kate Larson, and Edith Law. Ambiguity-aware ai assistants for medical data analysis. In Proceedings of the 2020 CHI conference human factors in computing systems, pp. 1-14, 2020.
Ümit Şentürk, ibrahim Yücedağ, and Kemal Polat. Repetitive neural network (rnn) based blood pressure estimation using ppg and ecg signals. In 2018 2Nd international symposium on multidisciplinary studies and innovative technologies (ISMSIT), pp. 1-4. Icce, 2018
Konstantinos C Siontis, Peter A Noseworthy, Zachi I Attia, and Paul A Friedman. Artificial intelligence- enhanced electrocardiograpyy cardiovascular discase management. Nature Reviews Cardiology, 18(7): 465-478, 2021.
Michael Steininger, Konstantin Kobs, Padraig Davidson, Anna Krause, and Andreas Hotho. Density-based weighting for imbalanced regression. Machine Learning, 110(8):2187-2211 2021.
16

---

Published in Transactions on Machine Learning Research (08/2023)
Viswanath P Sudarshan, Uddeshya Upadhyay, Gary F Egan, Zhaolin Chen, and Suyash P Awate. To- wards lower-dose pet using physics-based uncertainty-aware multimodal learning with robustness to out- of-distribution data. Medical Image Analysis, 2021.
Ryutaro Tanno, Daniel E Worrall, Enrico Kaden, Aurobrata Ghosh, Francesco Grussu, Alberto Bizzi, Stama- tios N Sotiropoulos, Antonio Criminisi, and Daniel C Alexander. Uncertainty modelling in deep learning for safer neuroimage enhancement:
Demonstration in diffusion mri. Newrolmage, 225:117366, 2021.
George R Terrell and David W Scott. Variable kernel density estimation. The Annals of Statistics, pp. 1236- 1265, 1992.
Evanthia E Tripoliti, Theofilos G Papadopoulos, Georgia S Karanasiou, Katerina K Naka, and Dimitrios Fotiadis. Heart failure: diagnosis, severity estimation and prediction of adverse events through machine learning techniques. Computational and structural biotechnology journal, 15:26-47, 2017.
Uddeshya Upadhyay, Yanbei Chen, and Zeynep Akata. Robustness via uncertainty-await cycle consistency, NewrIPS, 2021a.
Uddeshya Upadhyay, Yanbci Chen, Tobias Hepp, Sergios Gatidis, and Zeynep Akata. Uncertainty-guide progressive gans for medical image translation. In MICCAI, 2021b.
Uddeshya Upadhyay, Shyamgopal Karthik, Yanbei Chen, Massimiliano Mancini, and Zeynep Akata. Bayescap: Bayesian identity cap for calibrated uncertainty in frozen neural networks. In European Con- ference on Computer Vision, pp. 299-317. Springer, 2022.
Uddeshya Upadhyay, Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata. Probvlm: Probabilis- tic adapter for frozen vison-language models. IEEE ICCV, 2023.
Akhil Vaid, Kipp W Johnson, Marcus A Badgeley, Sulaiman S Somani, Mesude Bicak, Isotta Landi, Adam Russak, Shan Zhao, Matthew A Levin, Robert S Freeman, et al. Using deep-learning algorithms to simultaneously identify right and left ventricular dysfunction from the electrocardiogram. Cardiovascular 2022.
Imaging,
Hein F.C van der Wall, Gert-Jan Hassing, Robert-Jan Doll, Gerard JP van Westen, Adam F Cohen, Jasper I, Selder, Michiel Kemme, Jacobus Burggraaf, and Pim Gal. Cardiac age detected by machine learning applied to the surface ecg of healthy subjects: Creation of a benchmark. Journal of Electrocartiology. 72: 49-55, 2022.
Philipp Von Bachmann, Daniel Gedon, Fredrik K Gustafsson, Antônio H Ribeiro, Erik Lampa, Stefan Gustafsson, Johan Sundström, and Thomas B Schön. Ecg-based electrolyte prediction: Evaluating regres- sion and probabilistic methods. arXiv preprint arXiv:2212. 13890, 2022.
A Webster, W Brady, and F Morris. Recognising signs of danger: Eeg changes resulting from an abnormal serum potassium concentration. Emergency medicine journal, 19(1):74 77, 2002.
Ke Xu, Xinyu Jiang, Haoran Ren, Xiangyu Liu, and Wei Chen. Deep recurrent neural network for extract- ing pulse rate variability from photoplethysmography during strenuous physical exercise. In 2019 IEEE Biomedical Circuits and Systems Conference (BioCAS), pp. 1-4. TRFE, 2019.
Yuzhe Yang, Kaiwen Zha, Ying Cong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In International Conference on Machine Learning (ICMT), 2021.
Xiaoxi Yao, Rozalina G McCoy, Paul A Friedman, Nilay D Shah, Barbara A Barry, Emma M Behnken, Jonathan W Inselman, Zachi I Attia, and Peter A Noseworthy. Ecg ai-guided screening for low ejection fraction (eagle): Rationale and design of a pragmatic cluster randomized trial. American heart journal, 219:31-36 -36, 2020.
Taiyu Zhu, Kezhi Li, Pau Herrero, and Pantelis Georgion. Glugan: Generating personalized glucose time series using generative adversarial networks. IEEE Journal of Biomedical and Health Informatics, 2023.
17


Reviews:
[{"review_id": "4y06h39cjh", "reviewer": "Reviewer_hek2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- While targeting a very specific yet important task of regression with uncertainty, the proposed method is reasonably derived.\n- The paper does not stop at simply proposing an objective function for regression, but addresses the practical aspect by proposing a calibration strategy as well as ensemble-based decision making\n- The experiments are extensive in that HypUC was put to regression, calibration, and decision-making tasks, all showing its effectiveness.\n\nWeaknesses:\n- Although the title indicates HypUC is developed for general medical time-series, the entire experiment was conducted only on the ECG dataset.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes HypUC, a framework for performing a regression task specifically on medical time-series data (e.g. ECG signal) while quantifying the prediction uncertainty. The paper proposes a modified objective function involving kernel density estimation to better cope with imbalanced regression, and introduces a post-hoc calibration method, namely Hyperfine Uncertainty Calibration. On a large-scale ECG dataset, the paper conducts four different regression tasks, showing that it outperforms standard regression methods. The paper also shows HypUC's efficacy regarding its calibrated-ness and effectively removing samples with uncertain predictions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "After many revisions to reflect other reviewers' comments, I believe there is only one noticeable problem.\n- The title says \"medical time series\" but the entire paper focuses on only one application, which is ECG-based prediction. Therefore, if the title is changed to \"HypUC: Hyperfine Uncertainty Calibration with Gradient- boosted Corrections for Reliable Regression on Imbalanced Electrocardiogram\", then there would be no problem. Or the paper could include more experiments with other medical time-series such as EEG, or vital signs, but I assume that would be too much to ask.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JLoil94MyQ", "reviewer": "Reviewer_oWh4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- It is quite interesting to deal with the imbalanced data in the regression problem by using KDE to reweight the loss as analogous to using class weights to reweight the loss in the classification problem.\n \n- The method has significant improvements on the baseline methods in multiple regression tasks on the ECG data.\n\n\nWeakness: \n\n- Need to compare with some simple baselines that deal with long-tail regression problems. For instance, taking logarithms or standardizing the label during training.\n\n- The evaluation on calibration is insufficient. If the authors would like to claim that the calibration is improved, some key metrics used in uncertainty estimation should be shown. For instance, the probability of the ground truth falls in the 1 - $\\alpha$ predicted range. \n\n- The entropy-based filter is not quite novel. Since the entropy here is in a monotonic relationship with $\\hat{\\sigma}_{calib}$. This finding just shows a common belief in the well-calibrated model that the more uncertain the model is, the greater error the model will make.\n\n- The paper is quite mixed. It covers too many arguments which makes the outline of the paper unclear. Does the method focus on improving regression performance or uncertainty estimation? Some sections also include classification tasks, which makes the main idea of  the paper more vague. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper address the imbalanced label problem in medical time series problem. The model first uses a weighted loss function based on KDE to learn more on the data with minority labels. Then it recalibrates the prediction with a post-processing step. The proposed method outperforms the vanilla regression methods. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify what the main purpose of this paper is.  Modifying and restructuring some sections that have very diverse arguments can help the reader understand the paper better. \n\n- Conduct the suggested experiment. Choose the correct calibration metric if it is one of the contributions in the paper.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WbLWqVbt4l", "reviewer": "Reviewer_3msx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Unfortunately I found this submission quite difficult to follow. The mathematical notation are not clearly introduced and it appears unclear to me how the \"hyperfine uncertainty calibration\" step and the \"ensemble of gradient-boosted learners\" step are central to the proposed approach. \n- I'm not sure if TMLR is a suitable venue for this submission. Based on the [editorial policies](https://jmlr.org/tmlr/editorial-policies.html), it doesn't seem to fit under any of the items under \"Scope\". Perhaps the closest one is \"accounts of applications of existing techniques that shed light on the strengths and weaknesses of the methods\"; however, I believe that this submission is still lacking in these aspects as it does not seem to \"shed light on the strengths and weaknesses of the methods\". ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies regression tasks on physiological signals, specifically focusing on using ECG to diagnose cardiovascular diseases. The proposed approach combines kernel density estimation and uncertainty estimation techniques. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- Please define the notation before using them. For example, on page 4, Sec 3.2.3 Uncertainty Estimation, the symbols $\\zeta$ and $\\hat{\\nu}_i$, $\\hat{\\rho}_i$ were used without being defined. I'm guessing $\\zeta$ is the parameters of the DNN, but I still have no idea what $\\\\{\\hat{\\boldsymbol{y}}_i, \\hat{\\nu}_i, \\dots, \\hat{\\rho}_i\\\\}$ refers to. This has greatly hindered my understanding of the paper. \n- Please update the related work section to explain how each past work is related and how the present work differs. Right now, the realted work section simply lists prior work in the format of \"A study.... Another study...There have been studies\" without explicitly saying what previous work hasn't achieved and what this work adds. Readers not too familiar with the field (such as myself) will find it tempting to skip this section as it reads like an enumeration of papers about deep learning on ECG, but there's no discussion about why or how they are related specifically to this work. \n- Please update in-text citations to distinguish between \\citet vs \\citep", "broader_impact_concerns": "One sentence in the abstract (also conclusion) says that the proposed approach \"is suitable for clinical use and real-world deployment\". This is a very strong claim that necessitates further discussion and justification. For example, how are medical professionals performing the task that the ML model is solving, but without ML? How well do they perform currently? How important or useful are tasks such as \"estimating age from ECG\"? Without addressing these, I disagree with the statement that the proposed approach is ready for real-world clinical use. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V4AZEl4zFw", "reviewer": "Reviewer_DNgP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper showcases significant strengths in addressing an important problem, proposing a novel framework, and demonstrating its effectiveness. Attention to the weaker elements mentioned would further improve the clarity, comprehensiveness, and impact of the manuscript.\n\n1. Strengths:\n\n1.1. The paper addresses a significant and practical concern in the field of medical time series analysis, specifically the regression of continuous-valued physiological parameters.\n\n1.2. The suggested HypUC framework incorporates probabilistic machine learning techniques and a hyperfine uncertainty calibration method, providing reliable uncertainty estimates in imbalanced regression tasks.\nThe ensemble approach, utilizing calibrated uncertainties, improves decision-making and enhances the performance of the machine learning model.\n\n1.3. The paper demonstrates the effectiveness of the proposed approach on a large, real-world dataset of ECGs, showcasing its superior performance compared to several baselines.\nThe entropy-based technique for flagging unreliable predictions adds an critical layer of interpretability and trustworthiness to the proposed approach.\n\n1.4. The paper provides a comprehensive overview of the proposed framework, its implementation, and evaluation, allowing for reproducibility and further research.\n\n2. Weaknesses:\n\n2.1. The related work section could be more comprehensive, discussing recent work in the healthcare domain and other related domains. This would provide a stronger basis for the proposed approach and demonstrate the novelty of the contribution.\n\n2.2. The paper would benefit from a more detailed explanation of the methodology and implementation details. Providing a step-by-step overview of the HypUC framework and the ensemble approach would help readers understand the proposed methods more clearly.\n\n2.3. While the evaluation on a large real-world dataset is commendable, more detailed analysis and discussion of the results could be included. Specifically, providing insights into the performance of the proposed approach on different medical conditions or subgroups of patients would further strengthen the paper.\n\n2.4. The paper could consider discussing potential limitations and challenges associated with the proposed approach. Addressing these limitations and outlining future directions would enhance the impact and applicability of the research.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents several contributions and new knowledge in the field of medical time series analysis. Firstly, it addresses the challenge of imbalanced regression in medical time series by proposing the HypUC framework. This framework incorporates probabilistic machine learning techniques and a hyperfine uncertainty calibration method to accurately predict continuous-valued physiological parameters. By focusing on regression rather than classification, the paper expands the scope of previous research in the field.\n\nSecondly, the paper introduces a novel approach to uncertainty estimation in medical time series analysis. The calibrated uncertainties derived from the HypUC framework provide reliable estimates that generalize well to diverse test sets. This is crucial for clinical decision-making and enables clinicians to assess the confidence and reliability of the predictions.\n\nFurthermore, the paper presents an ensemble approach that utilizes the calibrated uncertainties to improve decision-making. By combining an ensemble of gradient-boosted learners, the paper demonstrates how the calibrated uncertainties can enhance the performance of the machine learning model. This ensemble approach contributes to more accurate and reliable predictions, leading to improved clinical outcomes.\n\nAdditionally, the paper introduces an entropy-based technique to flag unreliable predictions. This technique helps identify cases where the model may not be providing trustworthy predictions, enabling human expert evaluation and intervention when necessary. This contributes to the overall reliability and interpretability of the proposed approach.\n\nThe evaluation of the HypUC framework on a large, real-world dataset of ECGs collected from millions of patients with various medical conditions demonstrates its superior performance compared to several baselines. The framework achieves accurate predictions and provides calibrated uncertainty estimates, making it suitable for clinical use and real-world deployment.\n\nIn summary, the contributions of this paper include the development of the HypUC framework for imbalanced probabilistic regression in medical time series. The introduction of reliable uncertainty estimation, the utilization of calibrated uncertainties to improve decision-making, and the application of an entropy-based technique for identifying unreliable predictions are important contributions to the field. These contributions advance the field of medical time series analysis and have the potential to enhance diagnostic decision-making and remote healthcare assistance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Proposed Changes:\n\n- Strengthen the Related Work section by including recent work in the healthcare domain and other related domains (doi.org/10.1145/3313831.3376506, doi.org/10.1145/3132272.3134111). This would provide a stronger basis for the proposed approach and demonstrate the novelty of the contribution.\n\n- Provide a more detailed explanation of the methodology and implementation details (doi.org/10.1109/JBHI.2023.3271615, doi.org/10.1016/j.ijhcs.2022.102922). Specifically, provide a step-by-step overview of the HypUC framework and the ensemble approach. This would help readers understand the proposed methods more clearly.\n\n- Include a more in-depth analysis and discussion of the results, particularly insights into the performance of the proposed approach on different medical conditions or subgroups of patients (doi.org/10.1007/s10877-022-00873-7, doi.org/10.1145/3544548.3580682). This would enhance the paper's impact and applicability.\n\n- Discuss potential limitations and challenges associated with the proposed approach. Addressing these limitations and outlining future directions would strengthen the paper.\n\n- Improve the organization and structure of the paper to ensure a clear flow of ideas and logical progression.\n\n- Address proofreading and formatting issues to enhance the overall quality and readability of the manuscript.\n\nThese proposed changes, while not critical to securing a recommendation for acceptance, would significantly strengthen the work by addressing important aspects and improving clarity, comprehensiveness, and impact.", "broader_impact_concerns": "While the paper provides valuable contributions to the field of automated analysis of medical time series, there are a few ethical implications that should be addressed in the Broader Impact Statement. Specifically:\n\n- Privacy and Data Security: Given the use of large-scale datasets collected from millions of patients, it is crucial to address privacy concerns and ensure appropriate data protection measures are in place. The paper should discuss how patient privacy was protected during data collection, storage, and analysis, as well as any steps taken to anonymize or de-identify the data.\n\n- Bias and Fairness: The use of machine learning algorithms in healthcare raises concerns about bias and fairness. The authors should acknowledge the potential for bias in the dataset used and discuss steps taken to mitigate and monitor bias in the algorithmic predictions. Additionally, the paper should address the fairness considerations in terms of the impact on different demographic groups or subpopulations.\n\n- Clinical Validation and Human-in-the-Loop: While the proposed approach shows promising results, it is important to emphasize the need for clinical validation and the importance of human experts in the decision-making process. The paper should discuss how the proposed framework can be integrated into clinical workflows and emphasize the collaborative role of healthcare professionals in interpreting and utilizing the algorithmic predictions.\n\n- Deployment Challenges and Societal Impact: The authors should address potential challenges and barriers to the real-world deployment of the proposed framework. This includes considerations of cost, infrastructure requirements, acceptance by healthcare providers, and the potential impact on healthcare disparities. Discussing these aspects would provide a more comprehensive understanding of the broader impact of the work.\n\nBy addressing these ethical concerns in the Broader Impact Statement, the authors can ensure a more comprehensive assessment of the potential implications and contribute to a responsible and ethically sound implementation of the proposed framework.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following the discussion, there was no controversy that this paper should be accepted. It is not on a topic that is popular at TMLR, however, it is well executed.
Decision: Accept (poster)