Temporal Label Smoothing for Early Prediction of Adverse Events
Authors: Hugo Yèche
Alizée Pace
Gunnar Ratsch
Rita Kuznetsova
Venue: Submitted to ICLR 2023 (2023)
Keywords: Healthcare
Time-Series
Label Smoothing
Deep Learning
Application

Abstract: Models that can predict adverse events ahead of time with low false-alarm rates are critical to the acceptance of decision support systems in the medical community. This challenging machine learning task remains typically treated as a simple binary classification, with few bespoke methods proposed to leverage temporal dependency across samples. We propose Temporal Label Smoothing (TLS), a novel learning strategy that modulates smoothing strength as a function of proximity to the event of interest. This regularization technique reduces model confidence at the class boundary, where the signal is often noisy or uninformative, thus allowing training to focus on clinically informative data points away from this boundary region. From a theoretical perspective, we also show that our method can be framed as an extension of multi-horizon prediction, a learning heuristic proposed in other early prediction work. TLS empirically matches or outperforms all competitor methods across all evaluation measures on various early prediction benchmark tasks. In particular, our approach significantly improves performance on clinically-relevant metrics such as event recall under low false-alarm rates.

Under review as a conference paper at ICLR 2023
TEMPORAL LABEL SMOOTHING FOR EARLY PREDICTION OF ADVERSE EVENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Models that predict adverse events ahead of time with low false-alarm rates critical to the acceptance of decision support systems in the medical community. This challenging machine learning task remains typically treated as simple binary classification, with few bespoke methods proposed to leverage temporal dependency across samples. We propose Temporal Label Smoothing (TLS), novel learning strategy that modulates smoothing strength as a function of proximity to the event of interest. This regularization technique reduces model confidence at the class boundary. where the signal is often noisy or uninformative, thus allowing training focus on clinically informative data points away from this boundary region. From theoretical perspective, we also show that method can be framed an extension of multi-horizon prediction, a learning heuristic proposed in other early prediction work. TLS empirically matches outperforms all competitor methods on various early prediction benchmark tasks. In particular, our approach significantly improves performance clinically-relevantt metrics such event recall at low false-alarm rates.
INTRODUCTION
Raise alarm q0
Early prediction of adverse events is key to safety- critical operations such as clinical care [1] or en- vironmental monitoring [2] In particular, adverse event prediction is highly relevant to clinical decision- making, as the deployment of in-patient risk strat- ification models significantly improve patient outcomes and facilitate resource planning [. For instance, the National Early Warning Score (NEWS), a simple rule-based model predicting acute deterio- ration in critical care units, has been demonstrated reduce in in-patient mortality 3
Laboratory
teets
signs
Medication
4
Deteriorating patient signals often identified by mining large quantities of existing medical data and associated patient outcomes, which has sparked a growing interest in machine learning and medical lit- erature. Applications of such adverse event prediction models include alarm systems for delirium 5 septic shock [6], as well as circulatory or kidney failure in the intensive care unit (ICU) 7 8
Adverse event
Figure 1: Early prediction task.
Adverse event prediction remains a challenging modeling task requiring specific technical solutions. Recent years have seen the development of deep learning architectures for electronic health records (EHR), which help tackle the high dimensionality, irregular sampling, and informative missingness patterns in patient covariates [6 9 10 (8. Still, adverse clinical events often noisy, infrequent, and, as illustrated Figure | must be predicted with enough anticipation to allow for appropriate physician response yet early prediction remains largely considered simple binary classification task
As result, current decision support models often suffer from high false positive prediction rates, with associated risks of alarm fatigue and thus limited physician engagement [12 [3] M As highlighted
in Figure 2al the traditional cross-entropy objective results in the highest emror rates near the class

---

Under review as a conference paper at ICLR 2023
boundary, corresponding the prediction horizon before the event. Data in this boundary region dominates the loss but may not be clinically discriminative of patient deterioration patterns. Motivated by this observation, we propose Temporal Label Smoothing (TLS), novel regularization strategy making label smoothing [14] time-dependent to better match prediction uncertainty patterns over time. As visualized in Figure 2b our method is designed to reduce model confidence with stronger smoothing at the class boundary, allowing training to focus on more clinically informative data points away from this noisily labeled region.
Contributions. The contributions of work are threefold: (i) In Section 3.2
we introduce
novel label smoothing method" which leverages the temporal structure of early prediction tasks to focus training and model confidence on areas with a stronger predictive signal. (ii) In Section [ we show that our approach improves prediction performance over previously proposed objectives, particularly for clinically relevant criteria. (iii) In Section [.3] we bridge the gap between prior work
on multi-horizon prediction (MHP) []] and label smoothing [14] by showing the former is equivalent
to special case TLS under reasonable assumptions that we verify empirically.
1.0
1.0
0.5
FPR
y- 0.05
TPR
y-
V-5
0.6
0.4
0.2
0.2
5-
(Hous)
(a) Timestep performance of regular cross-entropy training for decompensation on MIMIC-III.
(b) Comparison of temporally and ground-truth labels.
Figure 2: Illustration of temporal label smoothing for early prediction adverse events. Predictions carried out over horizon h and te is the time of the next event, shaded in grey. True labels in black. (a) Model confusion is highest near the label boundary be h (maximum false positive, FPR, and minimum
positive rates, TPR), while performance is best close to occurrence (f.)) and away from it (t. 2h). This motivates greater smoothing near t. h. Metrics are computed over four-hour bins based on 50% precision threshold. (b) y controls the smoothing strength of surrogate labels q² LS
RELATED WORK
Recent years have seen the development of custom machine learning methods to predict expected patient evolution and support clinical of adverse clinical events is a particularly complex task due to their typically rare occurrence and noisy label definition, which induces challenging, highly imbalanced datasets for model training [8]. As result, prediction systems often suffer from high false-alarm rates with limited usefulness in the clinical context [1. Prior works early event prediction have adopted various approaches tackle this issue, which we compare in Table [ and formalize in Appendix A.4 We also discuss similarities and distinctions between our task and the frameworks of early time-series classification and survival
[15] 16 17. Amongst these, early prediction
analysis 18 in Appendix A.3
Learning objectives for imbalanced datasets. Class imbalance is often addressed through loss reweighting techniques. Static class reweighting was used for sepsis or circulatory failure predic- tion [17 7 through a balanced cross-entropy, which assigns higher weight to samples from the minority class [19]. Still, performance improvements with this objective remain limited on highly imbalanced prediction tasks (20). In contrast, dynamic reweighting methods such focal loss and extensions (21 22 induce learning bias towards samples with high model uncertainty, typically harder to classify. This approach can improve the prediction of discase progression from imbalanced datasets 23 but does not consider patterns of sample informativeness over time.
'lll code is made publicly available httts:://aoonmooooiii 4oper sci ence/r/tss/
2

---

Under review as a conference paper at ICLR 2023
Table 1: Related work. Comparison with different training objectives for binary early prediction
tasks, E {0, } corresponds to sample's s true label at time t and ÿ E [o, 1| to the model's prediction.
Additional details and respective advantages of each work are further discussed in Appendix A.4
Temporal Computationally scalable
Impacts sample optimum
Loss
Related work
inductive bias
for class {0, [0.1)
Cross-entropy loss
Sy log(ū)
Balanced cross-entropy loss
19
x
4 log(
Focal loss [2]
X
w 8) log(ū)
Label smoothing
14
X
4. (cly) log(ū)
Multi-horizon prediction
Σ n 10 log(t")
Temporal label smoothing
/
q (cly. (og(())
Multi-horizon prediction. In contrast, other early prediction models learn to leverage temporal
trends in the data by outputting event predictions over several horizons [8 24 25]. This training
heuristic improves prediction performance on the horizon of interest but scales poorly with the number of output horizons. Section 3.3] we highlight that TLS can induce a similar temporal bias in learning while overcoming scalability limitations.
Label smoothing. For greater generalization of models applied to heterogeneous real-world data, another well-known training strategy is to avoid model overconfidence through label smoothing (14] This regularization technique improves both the calibration of deep learning models [26) and their performance under noisy labeling [27 266. Still, despite extensions including novel prior distributions over classes (28] modifications to the objective itself (29] B0), label smoothing remains designed for classification problems with i.i.d. samples, ill-adepted to the uinedeppnnent nature our data.
To the best of our knowledge, we the first work explore adding temporal dependence to label smoothing and empirically demonstrate the added value of this approach.
Whereas reweighted loss functions only bias learning towards minority uncertain data points, multi-horizon prediction and label smoothing approaches alter the individual sample optimum. As consequence, these approaches avoid model overconfidence and are thus more robust to noisy labeling [27]. In this work, we propose to combine the respective advantages of these established methods a novel way improve the early prediction of adverse events.
Glossary
3 METHOD
h. Xix eat ti,tt)
Prediction horizon
Covariates
Binary event label
We first formalize the problem of early adverse event prediction and introduce temporal label smoothing. We then highlight how MHP can be framed spe- cial case of TLS.
Next event time
Wiet
Binary early prediction label
q T1.8 (wll..
Smoothed early prediction label
a(i,
Smoothing function
3.1 PROBLEM FORMALISM
We assume access to a dataset of N patient stays. These consist of irregular time series of high-dimensional patient covariates X1, [x.o..... X,,) and binary event labels Ci encoding whether a patient of index i is undergoing an adverse event of interest time t. For each patient, we thus have sequence (i.I), ... (x,,T,, ei,;;) of length T.
Our early prediction task consists of modeling a binary target variable Vt. positive if the event occurs within a given prediction horizon h. For labelling purposes, we define the next event time for each time point, (i, t) arg min Ci,r 1}. If patient never undergoes any event, we set (i, t) +oc Thus, we have: V4.t 1 It < te]. As our task focuses specifically early modeling for clinical relevance, no prediction is carried out if the patient is currently undergoing the event. Then, for any binary learning problem, we define model f parameterized by 0 with Dr., = fo (x++) = Pa (Vi, = 1). We denote the optimal set of parameters minimizing the objective function 0" giving vī., fo- (X..).
3

---

Under review as a conference paper at ICLR 2023
Temporal structure. An important distinction must be made with the classification tasks typically addressed with label smoothing. In adverse event prediction, data is not independent and identically distributed (i..d.) each sample 2i, depends on timestep and a patient stay indexed i. Contiguous samples within a common stay are thus timely dependent:
(I) Treating data as i.i.d, as is commonly done early prediction works I does not account for the increase in signal strength as the prediction time is approaching the event. Our goal is to leverage this structure in our data to focus training on relevant timesteps and help address issues of noisy label boundaries and class imbalance, which are inherent to our choice of real-world medical datasets.
p(yi,+d = 1) 2 p(y.. = 1) :d:05 (e(tt)
3.2 TEMPORAL LABEL SMOOTHING
As introduced by Szegedy et al. [14, label smoothing consists of substituting the original label distribution, Sy. for class with a smooth version g (cly) in the cross-entropy objective L; L (ys, ūt). For binary tasks, label smoothing becomes linear interpolation:
و (1lys) (1 a)y + a(1 ys)
where parameter a controls the smoothing strength.
By shifting the minimum of the objective function away from v ys towards v qاS label smoothing prevents models from becoming overconfident during training. This approach should therefore help improve the robustness of carly prediction models against the inherently noisy nature of the task [27] but does not account for the time dependency between samples of given stay. For this purpose, we propose temporal label smoothing, an approach to modulate smoothing based on time t infuse this prior knowledge into the training objective. We define the corresponding surrogate distribution similarly to label smoothing:
L --a(tt
t1)i,
For early prediction of events, to enforce the temporal inductive bias in Equation D we parametrize a(i, 1) as monotonously decreasing function E [0, (1, Figure Bal this increases smoothing strength around the label boundary t te h, reducing prediction certainty in this region prone to high error rates, as shown in Figure 2a
practice, as illustrated in
Smoothing parametrizations. We propose various temporal smoothing parametrizations for a in Appendix A.2 Experimental results suggest that an exponential parametrization, defined as follows, performs best considered tasks. Corresponding smoothed labels gr'H (1)(,, t) can be visualized in Figure 2b
e ((t.000))) A if hmmn < t (i, t) - t < hmaz
(
(e, 1) 1 < hmmin
(4)
(i, 2 hmar
Parameters Rmmn and Kmar define the time range over which we apply smoothing, namely [fe hmaan- te - A Under this constraint, parameters {d, A) defined enforce a(ii,t) t) to be continuous at boundary points (see Appendix A.2). Finally, y controls the smoothing strength at a given time.
100
- 02
H= 24
0.375
0.0
0.290
0.125
0.00
5.000
26
4
G-
(a) Parametrization (Equation 4
Parametrization a step (Equation
Figure 3: Label smoothing strength over time under different parametrizations, with (hymire Amaz) (0, 2h). Note that ly - g" TLS corresponds the difference in optimum y be- tween the TLS objective and cross-entropy. Smoothing function a step is equivalent to multi-horizon prediction with unique output.
4

---

Under review as a conference paper at ICLR 2023
3.3 LINK WITH MULTI-HORIZON PREDICTION
As motivated above, temporal label smoothing adapts the contribution of each sample to reflect prior knowledge about the temporal structure of event prediction labels. In this section, find that MHP leverages the same information in Equation to teach the model to predict events over multiple horizons/ [8] Under simplifying assumptions justified empirically in Section 5.2 we show that this approach can be seen as special case of temporal label smoothing with a "staircase' parametrization.
In this framework, the unique label Vi.t associated with patient covariates X for an horizon of interest h, is replaced by a vector Yi.c = ........................ ............. Vi,c' corresponding to H distinct horizons. The prediction model is thus adapted to output Vi., lo....... ......... -... For temporal consistency between samples, Tomašev et al. [8] enforce predictions to be monotonically increasing over time, such that hu S by = 0 2 ho training objective for patient becomes LMMP = -- shi log(t! + +1 v/;
With these additional components, the log(1 10s(1-8:)) (..)). Proposition 1. Under the assumption that model outputs (ol: }a are equal for all {ha}s (rather than monotonically increasing), MHP is equivalent to temporal label smoothing parameterized with a (itep(i, t). This function, illustrated in Figure 3b is defined as the following sequence of step functions in time:
k
if
h2 (t,ttt - -t h S
H
step (i,t)
if
teli,t) (tttt--5- hn
(5)
1
if teli,t) - hи
Proof. See Appendix A.I
Proposition fframes MHP as a special case of TLS with step-function parametrization. We empiri- cally justify the equal-output assumption through an ablation study in Section 5.2
EXPERIMENTAL SETUP
4.1 EARLY PREDICTION TASKS
We demonstrate the effectiveness of our method on three clinical early prediction tasks with different characteristics, understand its added value in each case. All tasks, established in existing literature and published benchmarks, deal with electronic health records from the ICU, where early prediction of organ failure or acute deterioration critical patient management [1]
Our work is first benchmarked on the prediction of acute circulatory failure and mild respiratory failure within the next 12 hours. These tasks are part of BERID-ICCU-ennhmaae (HiB) (20]. built on the publicly available HiRID dataset 77. The dataset contains high-resolution observations of over 33,000 ICU admissions. Our third evaluation task consists early prediction of patient mortality, or decompensation, within horizon of h 24 hours. Although less clinically relevant, this task has been widely studied in the machine learning literature BI] Defined in the MIMIC-II Benchmark (M3B) 332, this task originates from the widely used MIMIC-III dataset approximately 40,000 patient stays.
(33, counting
1.0
Recall at 50% Precision
Circllttrty Fore
All three clinical events
labeled fol-
Rerpimttrry
Decompennation
lowing internationally accepted criteria as
in Harutyunyan et al. (32) and Yèche al. (20).
0.6
Positive label prevalence is 4.3%, 3%, 38.6%, and 2. 1%
of timepoints for cireulatory, respiratory failure,
and decompensation prediction respectively with rarer events associated with more severe states, this instance. Further details on task definition and data pre-processing are provided Appendix [
0.0 4-
Time to event
Figure 4: Comparison of naive performance function of time on all tasks, using cross- entropy objective. Events should be predicted a horizon from an event time te. Model performance is reported at increments à T2
Signal deterioration over time. As visualized in Figure 4 all tasks show a reduction in recall between event time l. and prediction horizon l.
5

---

Under review as a conference paper at ICLR 2023
h, suggesting a weakening in the discriminative signal associated with events and an increase in noise close to the label boundary. Whereas this performance decay is strong for the circulatory failure and decompensation tasks, respiratory failure shows observation, we expect that temporal label smoothing should improve performance on circulatory failure and decompensation prediction to greater extent than respiratory failure.
consistent recall over time. From this
4.2 BENCHMARKING STRATEGY
Baselines. We quantify the added value of our method by comparing its performance to alternative learning approaches used for early event prediction, discussed in Section 2 Our first baselines consist of balanced cross-entropy [9] and focal loss [21], popular sample reweighting methods for imbalanced tasks. We also implement multi-horizon prediction as multi-output model trained to predict event occurrence over different horizons between 0 and 2h. Note that for a fair comparison, we set (hman )mex) (0, 2h) in TLS. As in Tomašev et al. [8]. cumulative distribution function layer on logits enforces the monotonicity of predictions (Eq. D. Finally, we also compare our method to conventional label smoothing [4 to confirm that temporal dependency does improve performance.
Hyperparameter tuning. Hyperparameters introduced by our method, such strength term y in smoothing parametrization a a (Equation D. are optimized through grid searches on the validation set. The same approach is adopted for hyperparameters specific to each baseline, shown in Figure
Architecture choice. As method and baselines are model-agnostic and only vary in terms of optimization objective, unique model architecture is used for each task, selected through a random search on cross-entropy validation performance. Following a published benchmark on the HiRID dataset [20]. we use GRU [4 and transformer (35] architecture for the circulatory and respiratory failure tasks respectively. For decompensation prediction, transformers outperform the LSTM-based models [36] originally proposed in the M3B benchmark (32), and are thus used in our work. As recommended by Tomašev et al. (8) we apply I-rreulariizatin to input embedding layers, which improves performance on all tasks. Further implementation details are provided in Appendix C
4.3 EVALUATION METRICS
To account for the imbalanced nature of clinical early prediction tasks, model performance is often reported through the area under the receiver operating characteristic curve (AUROC). Although this widely-used metric can be informative for moderate imbalances, the area under the precision-recat curve (AUPRC) provides more insight for our tasks: under a low prevalence of positive samples, precision is more sensitive false alarms than specificity 37 Still, "area under the curve" metrics can be poorly representative of clinical usefulness, improvements in low precision regions can dominate such global metrics but remain incompatible with the low false alarm rates required for clinical deployment. Thus, to better assess model performance in this context, we also measure performance at a clinically motivated operating point through recall at 50% precision (24.
In addition to timestep-level metrics, which measure prediction performance at each data point, we also evaluate models in an event-based approach. Following Tomašev et al. [8's definition, an event prediction is positive if the model outputs positive prediction at any time over the / hours before the event. The threshold defining a positive prediction is chosen based on a precision lower-bound: in 50% stepwise precision criterion. This allows to measure the event recall of our approach in comparison to published baselines. Unless stated otherwise, we always report mean performance with 95% confidence intervals computed over ten training runs.
practice,
RESULTS
5.1 PREDICTION PERFORMANCE
Overall, our results highlight that TLS improves performance over other approaches proposed to address the challenges of early clinical prediction. In Table 2] we find TLS to outperform other baselines across all metrics for both circulatory failure and decompensation. Despite over- lapping confidence intervals between multi-horizon and TLS decompensation due to indi-

---

Under review as a conference paper at ICLR 2023
Table Timestep-level performance on different early prediction tasks. Recall is reported at 50% precision. Circulatory and respiratory failure are predicted on the HiB dataset, decompensation on M3B. In bold, we highlight best-perforning methods with statistically significant p-values (< 0.05) under paired Student's t-tests [38]. As expected, performance gains respiratory failure prediction are not significant on these metrics.
Task
Circulatory Failure
Decompensation
Respiratory Failure
Method
AUPRC
Recall
AUPRC
Recall
AUPRC
Recall
Cross-entropy
39.1 + 0.4
29.3 1 0,9
34,5 + 0.4
28.2 1 0.5
60.5 1 0.2
77.3 + 0,5
Label Smoothing 14
39.3 + 0.4
2.8
33,9 4 0.3
27.7 2.5
60.1 + 0.2
76.6 + 0.5
Multi-horizon g
39.6 + 0.5
30.3 1.0
34.9 +: 0.3
28.6 0:0::000 0.5
60.3 + 0.1
76.6 0.5
Temporal Label Smoothing
40.6 + 0.3
32.3 0.7
3..
29.3 = 0.4
60.4 + 0.2
77.0 0.3
p-value (Ho : MHP 2 TLS)
0.00
0.00²
0.00
0.02
0.15
0.14
vidual training run variability, we can reject the null hypothesis that MHP has a higher perfor- mance to our method (p-values < 0.05), which supports the alternative hypothesis that TLS achieves superior results. Full precision-recal curves are given in Figures 6a and 14 This validates the experimental hypothesis proposed in Section [4.]] with, expected, more lim- ited improvements on respiratory failure a result analyzed in greater detail in Section 5.3
In contrast, illustrated in Figure loss reweight- ing methods designed to tackle class imbalance were found to reduce performance on all tasks over tradi- tional cross-entropy. For weighted cross-entropy. we attribute to the increase in false alarms resulting from the drive to improve recall. It further reduces the low precision of all models, thus negatively af-
20
35.6
35.6
35.7
35.7
35.4
37
Positive weight
10-
36.3
36.4
36.3
36.3
35.1
36
5- 37.2
37.2
34.8
fecting the AUPRC (as visualized in Appendix D.5). On the other hand, focal loss down-weighs confi- dent samples in training, constraining the model to focus on samples with uncertain predictions. In the context of noisy labeling, as is the case close class boundary, data points with ambiguous signals cannot be correctly predicted and thus dominate the loss, impeding improvements in other regions of in- put space. We analyze model performance over time in Section 5.2 to further support this hypothesis.
37.8
37.9
37.9
37.8
34.2
1
38.1
38.1
38.0
D
37.8
33.4
34
0.1
5
Focal weight C
Figure 5:
Performance loss with class reweighting methods, on the validation set for circulatory failure prediction. Weighted cross-entropy corresponds to = 0.
Clinically-relevant performance. We also compare the full precision-recall curve of models trained with these different objectives in Figure 6aa note that we obtain comparable results for decompensation prediction in Appendix D.2] In addition to visually confirming the numerical results in Table we find that our training objective affords particular performance improvements in the clinically-relevant region corresponding with low false-alarm rates (precision greater than 50% M.
Finally, highlighted in Figure 6b TL.S improves performance in terms of predicting overall adverse event episodes throughout a stay on all prediction tasks. This suggests that performance improvements at the timestep level affect large number of events and translate better event detection. Indeed, we demonstrate in Section 5.2 that TLS affords larger performance gains close to the event time, thus leading for both other tasks (see Appendix D.Ip. For circulatory failure, temporal label smoothing is able to predict 7.4% more events than the closest baseline muulti-horizon prediction): this corresponds to reducing the number of missed events in the test set by a factor of 2, from 303 to 152 out of 2045 events on average. Within these events not captured by MHP, TLS predicts them on average 104 minutes before their occurrence, giving clinicians sufficient time to take action and avoid patient degradation.
Event-based analysis.
better recall imminent events. We obtain similar conclusions
Whiie some methods have overlapping confidence intervals on circulatory failure and decompensation prediction, TI.S remains superior on each training run, giving p-values of O.
7

---

Under review as a conference paper at ICLR 2023
1.0
Lor
Lave
0.8
0.6
Procision
0.6
0.4
Jypp
Precision
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
2h
12t
6h 32h
to 12h
10h 22h
Recall
Time
Event
(a) Precision-recall curve. Inset shows the
(b) Event-level performance for a 50% timestep-level precision threshold.
dimically-aaplicabll region with precision > 50%.
Figure 6: Clinically-orientedd performance analysis of different training objectives on eirculatory failure prediction. See Appendix for results on other tasks.
5.2 ILLUSTRATIVE INSIGHTS
We propose ablations and analyses to build intuition around our proposed method. In particular, aim to highlight how temporal smoothing works and why it outperforms other training approaches for early prediction tasks.
Performance time. In Figure compare the performance difference between method, TLS, and the regular cross-entropy objective over time -previously studied Figure 2a We perform the same analysis in Appendix [ for other tasks. As expected, the prediction model trained with TLS is less competitive where label smoothing is strongest, near 4, - h. but this performance loss remains minor even with significant smoothing. This result validates our hypothesis that the signal is too noisy in the boundary region for any model to recover the original label distribution. In contrast, away from the label boundary, TL.S results a significant increase in true positive and negative rates. From a clinical perspective, errors made in the boundary region are less critical, as they result in the latest false positives or earliest false negatives. Consequently, TLS not only improves global event prediction performance but allows these gains to occur at more critical times for clinicians.
Empirical comparison to multi-horizon prediction. In our theoretical discussion in Section 3.3 we demonstrated how MHP is a restriction label smoothing with step function a atepp (i, t). This claim relies the constraint to produce unique prediction across all considered horizons, reflecting the design of our method. We verify the impact of this assumption by measuring performance gains afforded by learning distinct predictions per horizon. As shown in Table 3, with full precision-recall curves in Figure 19, we find no statistical evidence for performance gain over using a step on all tasks and studied metrics. Thus, models do appear leverage this additional flexibility offered by MHP. With superior results on all timestep- and event-based experiments, and greater scalability thanks to the single prediction horizon modeled, we find temporal label smoothing be superior training objective to MHP in early prediction tasks.
Table 3: Do MHP's multiple outputs improve performance over TLS with slep? We provide p-values for the paired Student-t test (38) on the null hypothesis Ho Patep 2 ИМиP- With no statistically significant improvements (p 0.05), we justify our assumption in Proposition
Task
Circulatory Failure
Decompensation
Respiratory Failure
Method
AUPRC
Recall
AUPRC
Recall
AUPRC
Recall
MHP
39.6 + 0.5 39.3 0.2
30.3 + 1.0
34.9 = 0.3
28.6 - 0.5
60.3 + 0.1
76.6 + 0.5 77.4 0.5
TLS (a step)
29.4 + 0.8
35.2 + 0.3
29.2 0.4
60.5 + 0.1
v-luuue (Ho)
0.11
0.10
0.95
0.97
0.99
0.98
8

---

Under review as a conference paper at ICLR 2023
TPRms TPRcE
TNiR
4-
6-h
(a) True negative rate (TNR).
True positive rate (TPR).
Figure 7: Performance improvement over time for TLS over traditional gross-entropy on circulatory failure prediction. Timestep-level metrics computed for precision of 0.5 over two-hour bins.
5.3 TRADE-OFFS AND LIMITATIONS
Despite the demonstrated advantage of our training paradigm for two distinct early prediction tasks, observed more limited performance gain over traditional cross-entropy when predicting respiratory failure in Table 2 as with other baselines. This observation motivated an analysis of the specific problem settings which objective helps. Respiratory failure events much more frequent than circulatory failure or decompensation, with the majority of ICU patients undergoing approximately two such events during their stay, as quantified in Appendix B We hypothesize that this reduced class imbalance leads to sufficient discriminative information within the label boundary region. This belief supported by the lower performance drop from event to prediction time Figure 4 in comparison to other tasks, and results in more significant performance loss close to te with TLS, with a % drop in true positive rate (TPR) in Figure [
However, expected by design, our method im- proves recall (+1% 1% TPR) over cross-entropy close to the event. This also leads to non-negligible 0.4% (p-value < 0.05) improvement in event re- call, visualized in Appendix D.2 Overall, this analysis reveals that whereas TLS has little im- pact on global metrics for tasks with limited performance reduction over time (e.g., close-to- balanced, of often limited usefulness clinical decision support efforts (8), it still results in clinically meaningful performance improvements along per-horizon and event-based metrics.
TPR.s TPR. (8)
Figure 8: Performance improvement over time for TLS over traditional cross-entropy, for res- piratory failure. True positive rates (TPR) are computed for precision of 0.5 over 2-hour bins.
CONCLUSION
Early prediction of adverse events is paramount to the development of clinical decision support systems, with a demonstrated potential to improve patient outcomes [3] Still, this task remains poorly studied in the machine learning literature, with few training solutions tailored to address its challenges. Based on typically rare and noisy labels, models must learn to discriminate a predictive signal in anticipation of events to allow adequate medical response.
After highlighting the limitations traditional classification objectives and methods designed to address the class imbalance, we propose novel training framework that leverages trends event signals over time. We show that multi-horizon prediction, a heuristic used to improve early prediction, can be formalized empirically matches or outperforms all considered baselines various tasks and datasets, with significant improvements on clinically-relevan evaluation metrics. Performance gains are limited, as with other baselines, for respiratory failure prediction in which higher event prevalence provides sufficient informative data points for the model to learn through a conventional cross-entropy objective. In further work, aim explicitly adapt the temporal inductive bias to the task at hand and to combine temporal label smoothing with recent objectives designed to directly optimize AUPRC, such minimum precision constraint 39 or dice-based loss functions [40].
restriction of framework. Simple but effective, temporal label smoothing
Looking ahead, we expect that temporal label smoothing will be leveraged to develop more clinically reliable systems for risk prediction of rare adverse events. Further research tailored machine learning solutions to improve real-world decision support holds promise for better clinical care and operations management.

---

Under review as a conference paper at ICLR 2023
REFERENCES
[1] Reed T Sutton, David Pincock, Daniel C Baumgart, Daniel C Sadowski, Richard N Fedorak, and Karen Kroeker. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ digital medicine, 3(1):1-10, 2020.
[2] Francesca Di Giuseppe, Florian Pappenberger, Fredrik Wetterhall, Blazej Krzeminski, Andrea Camia, Giorgio Libertá, and Jesus San Miguel. The potential predictability of fire danger provided by numerical weather prediction. Journal of Applied Meteorology and Climatology. 55(1 55(11):2469 2491, 2016. doi: 10.1175/AAMC-D1155
URL Utthttps: //jornna
amet weergvvveeeee wttprr//vwwwooooorarroo http://wwwwoorroaaaeeeaee/////// jaclll/jjmccaar
0297. xx1
[3] Gary B Smith, David R Prytherch, Paul Meredith, Paul E Schmidt, and Peter I Featherstone. The ability of the national early warning score (news) discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation, 84(4): 465-470 2013.
[4] Anne Pullyblank, Alison Tavaré, Hannah Little, Emma Redfern Hein le Roux, Matthew Inada- Kim, Kate Cheema, and Adam Cook. Implementation of the national early warning score in patients with suspicion of sepsis: evaluation of sydtem-wide quality improvement project. British Journal of General Practice, 70(695):e381-e35)) 2020.
[5] Andrew Wong, Albert T Young, April S Liang, Ralph Gonzales, Vanja C Douglas, and Dexter Hadley. Development and validation of an electronic health record-basee machine learn- ing model to estimate delirium risk in newly hospitalized patients without known cognitive impairment. JAMA network open, 1000ee11101811111111 2018.
[6] Josef Fagerström, Magnus Bång, Daniel Wilhelms, and Michelle S Chew. Lisep Istm: machine learning algorithm for early detection of septic shock. Scientific reports, 9(1):1-8. 2019.
[7] Stephanie L. Hyland, Martin Faltys, Matthias Hüser, Xinrui Lyu, Thomas Gumbsch, Cristóbal Esteban, Christian Bock, Max Horn, Michael Moor, Bastian Rieck, al. Early prediction of circulatory failure in the intensive care unit using machine learning. Nature medicine, 26(3): 364-373, 2020.
[8] Nenad Tomašev, Xavier Glorot, Jack W Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Anne Mottram, Clemens Meyer, Suman Ravuri, Ivan Protsyuk, al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature, 172(7767):166-119 2019.
[9] Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten M. Borgwardt. Set functions for time series. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4353-4363. PMLR, 2020. URL http::ttpp//oooe http://rroceeeiggs mlr
btess/vl19/hoznee
html
[10] Satya Narayan Shukla and Benjamin M. Marlin. Multi-time attention networks for irregularly sampled time series. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. PRLLssttta://operreeee net/ met/forung wtr.m//rummmmmm 2atprrummmd=440003300 600.c00666w000000
[11] Simon Meyer Lauritsen, Mads Ellersgaard Kalør, Emil Lund Kongsgaard, Katrine Meyer Lauritsen, Marianne Johansson Jørgensen, Jeppe Lange, and Bo Thiesson. Early detection of sepsis utilizing deep learning on electronic health record event sequences. Artif. Intell. Medicine, 104:101820, 2020. doi: 10.1016/j.ariin 200mm
URL hhtps::/ddo
org/10 1016/0 art.jeddare.......... tmed. 2020 10.1822
[12] Maria Cvach. Monitor alarm fatigue: an integrative review. Biomedical instrumentation & technology, 46(4):268-277) 2012.
[13] Sue Sendelbach and Marjorie Funk. Alarm fatigue: a patient safety concern. AACN advanced critical care, 24(4):378-38) 2013.
10

---

Under review as a conference paper at ICLR 2023
[14] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA. June 27-30, doi: 20.1109/CYPR.2016.303 URL 1109/VVRRR Ttt.:/CVRR2206..... 300
2016, pages 2818 IEEE Computer Society, 2016.
https:///oi..on
[15] Konstantina Kourou, Themis P. Exarchos, Konstantinos P. Exarchos, Michalis V. Karamouzis, and Dimitrios Fotiadis. Machine learning applications in cancer prognosis and prediction. Computational and Structural Biotechnology Journal, 13:8-17, 2015, ISSN 2001-0370. doi: https///bbi.org/100000//// scceene/aarriceee////20000111111
1.05. URL https: www www.sscieencic... receedirrett rect com/
[16] Jing Xiao, Ruifeng Ding, Xiulin Xu, Haochen Guan, Xinhui Feng, Tao Sun, Sibo Zhu, and Zhibin Ye. Comparison and development of machine learning tools in the prediction of chronic kidney disease progression. Journal of translational medicine, 17(1):1-13, 2019.
[17] Joseph Futoma, Sanjay Hariharan, and Katherine A. Heller. Learning to detect sepsis with mul- titask gaussian process RNN classifier. In Doina Precup and Yee Whye Tch, editors, Proceedings of the 34th International Conference on Machine Learning. ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1174 1182. PMLR, 2017.
htttt:::/procceed.nng mlr prwss/ / 700 out html
[18] D. R. Cox. Regression models and life-tables. Journal of the Royal Statistical Sociery. Series (Methodological), 34(2):187-22,, 1972. ISSN 00359246. URL http: www. jstor. org/ stable/298518
[19] Gary King and Langche Zeng. Logistic regression in rare events data. Political analysis, 92): 137-163, 2001.
[20] Hugo Yčche, Rita Kuznetsova, Mare Zimmermann, Matthias Hüser, Xinrui Lyu. Martin Faltys, and Gunnar Rätsch. high-resolution icu data. arXiv preprint arXiv:2111.08552 2021.
comprehensive machine learning benchmark on
[21] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pages
2017.
[22] Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Xiaojie Shi, Shuyang Cheng, and Dragomir Anguelov. Polyloss: A polynomial expansion perspective of classification loss functions. CoRR, abs/2204. 12511, 2022. doi: 10448550/axXiv.2244.222 12511. URL https
//doi.org/10. 48550/ arw.s//aaxivvvv... 2204. 1204.12511
[23] Subhrajit Roy, Diana Mincu, Lev Proleev, Negar Rostamzadeh, Chintan Ghate, Natalic Harris, Christina Chen, Jessica Schrouff, Nenad Tomašev, Fletcher Lee Hartsell, et al. Disability prediction in multiple sclerosis using performance outcome measures and demographic data. In Conference on Health, Inference, and Learning, pages 375-396. PMLR, 2022.
[24] Nenad Tomašev, Natalie Harris, Sebastien Baur, Anne Mottram, Xavier Glorot, Jack Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Valerio Magliulo, et al. Use of deep learning develop continuous-risk models for adverse event prediction from electronic health records. 2021.
Nature Protocols, 16(6):2765-22)
[25] Subhrajit Roy, Diana Mincu, Eric Loreaux, Anne Mottram, Ivan Protsyuk, Natalie Harris, Yuan Xue, Jessica Schrouff, Hugh Montgomery, Alistair Connell, et al. Multitask prediction of organ dysfunction in the intensive care unit using sequential subnetwork routing. Journal of the American Medical Informatics Association, 28(9): 1936-1946, 2021.
[26] Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smooth- ing help? In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Gamett, editors, Advances in Neural Infor- mation Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 4696 4705, 2019. URL oetoo /ttp://uuuuuuaaaaapa///////////a

---

Under review as a conference paper at ICLR 2023
[27] Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of ingi
Machine Learning Research, pages 6448 6458. PMLR, 2020. URL m1 wwwess/vl9//1sse
k20a html.
[28]
Weizhi Li, Gautam Dasarathy, and Visar Berisha. Regularization via structural label smoothing. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy.. volume 108 of Proceedings of Machine Learning Research, pages 1453-1463, PMLR, 2020.
URL http::/ppooeeeiinggggg//oooo//et
m wtt....p......e///.e
html
[29] Clara Meister, Elizabeth Salesky, and Ryan Cotterell. Generalized entropy regularization or: There's 's nothing special about label smoothing. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 6870-6886. Association for Computational Linguistics, 2020. doi: 10. (01118655vv1// /22/2/2222001iin...mm //doi.org//2
URL (ttpps
acl main. 615
[30] Julian Lienen and Eyke Hüllermeier. From label smoothing to label relaxation. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021. Thirty-Thinn Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposiun Educational
Advances in Artificial Intelligence. EAAI 2021, Virtual Event, February 2-9, 2021, pages
8583 8591. AAAI Press, 2021. URL https: /ttpps://oss.oo www.aaaa........a... wwtph/deexppppp.aa.eeeeae phwpppAI/////ppp.ppp.. article/view/1700
[31] David Bellamy, Leo Celi, and Andrew I. Beam. Evaluating progress on machine learning for longitudinal electronic healthcare data. arXiv preprint arXiv:20/0.01149 2020.
[32] Hrayr Harutyumyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):1-18, 2019.
[33] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[34] Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling, CoRR, abs/1412.3555, 2014. URL
wwwp:riwww..o.//...b.b/aeab/.aae owg./rgg//......... 3555.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information P Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 199886008 2017. URL Btt./tttpp::/prrrcediiiai neur neur pss cc/paper/20117 hash/3f5ee2 244E5e2233544 4e0e44444444ee11000000000eeo1e.0
Abst .ttll
[36] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
[37] Takaya Saito and Mare Rehmsmeier. The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3):001184) 2015.
[38] Student. The probable error of mean. Biometrika, pages 1-25, 1908
[39] Preetish Rath and Michael Hughes. Optimizing early warning classifiers to control false alarms via a minimum precision constraint. In Gustau Camps- Valls, Francisco R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 4895 4914. PMLR, html
28-30 Mar 2022. URL https:: pttttps://proocee/e/ ing www.pres//eeee 51 htrp:/eeeeennnnmmm..mr.rs//////s
12

---

Under review as a conference paper at ICLR 2023
[40] Michael Yeung, Evis Sala, Carola Carola-Bibiane Schönlieb, and Leonardo Rundo. Unified focal loss: Generalising dice and cross entropy-basd based losses to handle class imbalanced medical image segmentation. Computerized Medical Imaging and Graphics, 95:102026, 2022.
[41] FJ Richards. A flexible growth function for empirical use. Journal of experimental Botany, 10 (2):290-01,, 1959.
[42] David Collett. Modelling survival data in medical research. CRC press, 2015.
[43] Daniel Jarrett, Jinsung Yoon, and Mihacla der Schaar. Dynamic prediction in clinical survival analysis using temporal convolutional networks. HEEEjounin of biomedical and health informatics, 24(2):424-43 2019.
[44] Zhengzheng Xing, Jian Pei, and Philip S. Yu. Early prediction time series: A nearest neighbor approach. In Craig Boutilier, editor, IJCAI 2009, Proceedings of the 21 International Joint Conference on Artificial Intelligence, Pasadena, California. USA, July 11-17, 2009, pages 1297-1302, 2009, URL
org.i.orr//... /ttp://acoa.....eeeeoeeo//////// Ptwp://annnn///aaae///seseeeeeee /211 218 pdf
[45] Guoliang He, Yong Duan, Tieyun Qian, and Xu Chen. Early prediction imbalanced multi- variate time series. In Proceedings of the 22nd ACM International Conference on Information Knowledge Management, CIKM 13, page 1892, New York, NY. USA. 2013. Association for Computing Machinery. ISBN 978 1403222688. doi: 10.1145/200555555077888 URL https: wtt.:/...ogg/...o 11.11.1.144.00 1141114554 11e11114555555 25071...550880 2507888
[46] Yuzhe Yang, Kaiwen Zha, Ying-Cong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 11842- -1851. PMLR, 2021. html
URL patttp://rroceddee wttppp/cecnnnao.....eeeeee/e////
mir wwtpppeesss.eeeeeeeeee
[47] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Des- maison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chin- tala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Newral Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL
neurips cc/paper/ 9015-pytorch-ar an imperative style wechhh--erfrrmmmmc deep Lew.ee---eaannnn-ibbaaae pdf
[48] Pedregosa, G. Varoquaux, A. Gramfort, Michel, B. Thirion, Grisel, Blondel, P. Prettenhofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Scikit-learn: Machine lcarning in Python. Journal of Machine Learning Research,
2011.
[49] NVIDIA, Péter Vingelmann, and Frank H.P. Fitzek. Cuda, release: 10.2.89, 2020. URL /ttps://develiee
nvidia. com/
[50] Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv: 140.009,, 2014.
[51] The gin-config Team. gin-config python packaged. https::/gtthub.commm/// gin- cinn-config 2019.
13

---

Under review as a conference paper at ICLR 2023
A THEORETICAL DETAILS
A.1 MULTI-HORIZON PREDICTION: PROOF OF PROPOSITION 1
Equivalency between MHP and TLS objectives. Recalling the formalism of multi-horizon prediction outlined in Section 3.3 true labels and model predictions can be rewritten as Yi,
[vi...... vh .......... Vi.
and Yi.t [8 1i......... th ......... "'. where H is the number of horizons
considered. The training objective for patient becomes:
LHP IM (y...8s.))
H 1
Σ
vli log (1 (1-
v/tt) log(1 +
The assumption that fola be is equal for all allows to rewrite the objective as follows:
MHH (tee) log(Or.)) (v.)))
Σ Mit + log(1 (i,e) 1
Σ viitt
with de,t being the common prediction shared across all horizons. This equation can now be viewed as temporal label smoothing objective with smoothed labels " (1]0, t) 11 Σ h.. Vi.t
LHPP Sy.). = - -Pog((0.)) . (1 t) + log(1 (.,1) (1 step [1]i,))]
Smoothing parametrization. Next, we aim to recover the explicit form of ( i, t). Without loss of generality, we assume that horizons {h are in ascending order. The temporal dependency between samples, formalized in Equation]] D results in the following relationship between predictions horizons h,, and h,:
and
yi 1
(6)
u and
by V = 0
=
Vi by
= 0
Thanks the above property. we can determine geter t) by studying three cases of multi-horizon labels, illustrated in Figure For notational simplicity, we define de (i, = te (i, t.
d= te -t h2
hH
ha+1
Whaneraa Whine-1
(2
vh: =
Vh y1=1
0
t hy
to he+1
hk
to h1
to
Figure 9: Label values for multi-horizon prediction, and conversion to smoothed labels tee (1)t).
Case 1: di, S hn-
From label definition. we have that vitt = (f, < h1 As hy is the smallest horizon, following Equation we have wh: Vc [1, H1. We can rewrite the objective as:
L MHP St.e) log(Ar,))
-10 log(úne) (1 9 (1)(())))) log(1 sr.e)
14

---

Under review as a conference paper at ICLR 2023
where q ( 1. Case 2: de (i, > ft
Similarly, de (i,t) > his then Var which implies yhA = 0, Ve E [1, H] from Equation 7 The objective can be rewritten as:
I.MHP (ya,, log(1 (i.e)) = - log(Ds,)) + (1 step d (1)i,t)) log(1 - Dr,c)l
where getep (1)2,) O.
Case 3: 36 E [L, 1] s.t hg (t) h
Following the same reasoning as in the first two cases, we now have specific index & which separates
positive and negative labels. We have Vi 0, ve E [1, and yht = 1, vc [ + 1, H[] This
he
allows to rewrite the objective as follows:
H k
h log(1 - úr,e)l
L MHP S1.),
-1 -1
log
H
-1 (1/1, log(ūr.)) (1 - q step (1li, log(1 [1,e31
where
(1úit) H-k
Defining a new smoothing parametrisation step such that 9 (0tep (1]i, = 1 - a strep (i, t), we obtain:
if h& S di, hkt1 Ak H -1
step step (i,t)
if de S if
de t) > hn
Thus, (l) > O. find that T.MUP ITLS when smoothed labels defined a (1(i,)
1 a step (i. 1). This concludes our proof.
A.2 TEMPORAL LABEL SMOOTHING FUNCTIONS
hare 12
--- yl1.c)
0.6
0.6
0.0
6 20
4-
6 2n
2 (Houusi
C (Hous))
(a) q chift
(b) g step and & Finear
1.0
y-
0.2
v
0.6
0.4
y(3,f
0.05
-==
0.2
0.0
4-20 20
(Hours)
C (toours)
(c) o
concave
(d) q sigmoid
Figure 10: Illustration of temporal label smoothing with alternative smoothing parametrizations.
15

---

Under review as a conference paper at ICLR 2023
Motivated by prior work [8] 18 compare the performance of various smoothing functions a(i, t). All proposed parametrizations are continuous and monotonous decreasing functions that satisfy boundary conditions a(i, (i, 2h) 1 and a(i, te(, find exponential label smoothing to perform best or as well as others across all tasks and metrics.
0. As evidenced in Table 4
Performance function of hyperparameter setting can be visualized in Figure [1] All model and hyperparameter selection were carried out on the validation set, including the final choice of parametrization function.
Circulatory Failure (Validation AUPRC)
Circulatory Failure (Validation AUPRC)
0.40
0.40
0.38
0.38
0.36
0.36
0.34
0.34
0.32
Qoeo
0.32
0.30
Ssigmond
Gesp
HP
0.28
0.28
0.01
0.05
0.1
0.2
0.5
1.0
10
12
14
16
18
20
22
24
havn (Hours)
(a) Circulatory failure.
Decompensation (Validation AUPRC)
Decompensation sation (Validation AUPRC)
0.40
0.40
0.36
0.36
0.34
0.34
0.32
Qaen
0.32
9rnncoue
0.30
Quspenni
best HP
0.28
0.28
0.01
0.05
0.1
0.2
0.5
1.0
8
12
16
20
24
28 (Hours)
32
36
40
44
48
hunh
(b) Decompensation.
Respiratory Failure (Validation AUPRC)
Respiratory Fallure (Validation AUPRC)
0.63
0.63
0.61
0.61
0.60
0.60
0.59
0.59
0.57
0.57
grancwe
0.56
Qsipmokd
0.56
deep
best HP
0.55
0.55
0.01
0.05
0.1
0.2
0.5
1.0
huve (Hours)
(c) Respiratory failure.
Figure Validation AUPRC performance of temporal label smoothing as function of smoothing hyperparameters, with different smoothing parameterizations. (Left) Performance for different smoothing strengths y with "rp, concave. Algmoid (Right) Performance for different prediction horizons hahift with a shift smoothing.
Shifted boundary labels. Shifting the prediction horizon or label boundary in training can be viewed form of temporal label smoothing. in which class labels are inverted within prediction
16

---

Under review as a conference paper at ICLR 2023
Table 4: Performance of different smoothing functions on early prediction tasks. Recall is reported at a 50% precision.
Task
Circulatory Failure
Decompensation
Respiratory Failure
Method
AUPRC
Recall
AUPRC
Recall
AUPRC
Recall
asep
39.3 1 0.2
29.4 + 0.8
35.2 + 0.3
29.2 + 0.4
60.5 + 0.1
77.4 0,5
shift
40.1 + 0.3
31.80..0
34.5 +: 0.4
28.2 0.5
60.5 4 0.2
77.3
linear
39.4 + 0.3
35.1 0.8
29.2
60.3
7.6
sigmoid
39.4 0.3
2.87
3.
28.8
60.6
7..
a concove
39.4 1: 0.3
3.41 0.8
29.2
60.3
77.0
cxp
40.6 0.3
3.7
35.5
29.3
60.4 0.2
77.0 77.03....
window of interest. This defines the following smoothing parametrization o si
shift t = (t, - 2 hanifal
(8)
where hshift is hyperparameter controlling the horizon of the smoothed labels (hahift corresponds to cross-entropy training). The strength of this smoothing function is illustrated in
Figure T0a
Figure outlines the performance of this alternative smoothing parametrization as a function of hahift- For both decompensation and respiratory failure, shifting the label boundary closer the event time decreases performance. On circulatory failure, performance does improve over traditional cross-entropy training the label horizon is brought closer to the event of interest, which can be interpreted an inductive bias similar to that induced by the exponential smoothing function.
Linear label smoothing. The most straightforward extension to the step function step described in Section 3.3 is linear label smoothing corresponding to the case H + +00. Our parametrization a linear (i, is thus defined follows:
(a.A)) 2h
if te(i,t) 2h
linear (i,
if (e(i,) - 2 2h
We illustrate the impact of the number of steps H in Figure 10b
Sigmoidal label smoothing. Another natural direction to explore is to smooth labels starting from the true distribution, a unique step function at f = (t) - h. This can be achieved by defining (x() generalized logistic function 41
1
K-A to
A (f, < 2h
a sigmoid
(i,t)
14c
(10)
if tf, - - 2h
where K, and d are three constants fixed by imposing the boundary conditions at la(i, 1) and = (i(i, 1), as well as (i, 2h) = This yields:
2h-2
K Ae
7 +
A
2h-4
d h
As shown in Figure 10d y controls the smoothing strength, interpolating between the true distribution 5y,=1 =1 7 and mear when 7 +00.
17

---

Under review as a conference paper at ICLR 2023
Exponential label smoothing. The smoothing function we find to perform best is the exponential decay one. This idea is motivated by survival analysis, where patient survival probability can be modeled the exponential decay of a cumulative hazard function [8 42 In practice, defined in our exponential smoothing function a a (i, () is defined as follows:
Section 3.2
7(2.66)))--)) - (f, - < 2h
(i,
(11)
ifte(i,t) 2 2h
where parameters (d, A} are set to satisfy boundary conditions:
A - ((2--d)
1
d = In - 222
Here, y also controls the smoothing strength between glineer when y - 0 and q(t) = 0 Vt < te when 7 +00.
Overall, despite a sigmoid and shift achieving good results on respiratory and circulatory failure respectively, = statistically outperforms these smoothing parameterizations across all tasks on validation metrics. An interesting avenue for further work would be to combine exponential smoothing with the boundary shift approach, effectively change (Amin Amaz), which fixed to (0, 2h) in our work for a fair comparison to multi-horizon prediction.
Concave exponential label smoothing. Finally, to mirror the behavior of the exponential smoothing function away from linear interpolation and investigate its effect on performance, we designed the following concave smoothing function a concave;
(e (c-7dd- A
(,, - < 2h
at concare (i,t)
(12)
(f,, 2 2h
Parameters {d, A} are identical to the convex smoothing function parameters, set to satisfy boundary conditions. The strength of this concave smoothing function is illustrated Figure T0c
No performance gains were obtained through temporal label smoothing with a concave function, as shown in Figure [I This smoothing function effectively penalizes false positives harder than false negatives, which is less adapted to our tasks of interest (in contrast to the convex "P"), As result, the berf-perf concave parametrization is consistently obtained with the lowest value of y, closer to linear function choice.
A.3 RELATED TIME-SERIES TASKS
Comparison to survival analysis. Survival analysis consists statistical methods concerned with predicting the probability of certain event taking place over time 42 In our formalism outlined in Section 3.1 the corresponding task is to regress the time the next event, t (t, i), based on patient information accumulated up time 1. To recover early event prediction, threshold the hazard model can thus be applied to determine whether an event will happen within our horizon of interest h. Modeling constraints imposed survival analysis improve time-to-event prediction performance over traditional regression methods, which supports our approach to leverage the temporal structure of our comparable task. Interestingly, recent developments survival modeling to deal with dynamic predictions have been addressed with multi- multi-horizon prediction
43
Still, distinctions must be highlighted between our adverse event prediction problem and the typical experimental setup for survival analysis: in our case, multiple events can occur over the course of a patient's s stay, with unknown patient states during and immediately after event occurrence. This results in complex, informative censoring patterns and challenges common assumptions in survival analysis, which can therefore not be directly applied to our task.
Comparison to early time-series classification. A distinction must be drawn between our task of early prediction of adverse events and that of early time-series classification. The latter has been more extensively explored in the literature
44
46
but addresses distinct problem.
18

---

Under review as a conference paper at ICLR 2023
Considering time series up timestep t, early event prediction is concerned with classifying whether a particular event will occur between and + h, for fixed horizon h. Predictions are made each timepoint over the entire time series: as multiple samples arise from the same time series and therefore depend on one another over time, these should not be considered as i.i.d.
In contrast, early classification of time series aims regress the first timepoint at which label for the entire time series can be predicted with a desired accuracy 4 A single prediction is made, as soon as possible, for the entire series which can be considered
independent sample from the dataset of time series. This latter task can be framed as early prediction of the event "prediction is possible", where oo, given a separate time-series classifier. As result, an interesting avenue of further work would be to apply temporal label smoothing to the latter task.
On the other hand, early event prediction cannot be translated into a simple early classification problem. As result, methods designed for early time-series classification are therefore not applicable this problem setting.
A.4 BASELINE OBJECTIVE FUNCTIONS
In this section, we clarify the mathematical formalism behind baselines facilitate comparison to temporal label smoothing. All baselines explored effectively propose a modification of the cross-entropy objective often used for binary classification tasks, L; L CE (y:, Dx).
Balanced cross-entropy. To facilitate learning from highly imbalanced datasets, balanced cross- entropy relies on reweighting samples based on their class prevalence, as follows:
N
OF
Wy. L(Di, 34)
(13)
N
where C' is the number of classes, Wy. ( and b(c) defines the prevalence of class such that
E. b(c) = 1. Regular cross-entropy corresponds to the case where b(c) δ for all classes. In the binary setting, b(1) can be treated as a hyperparameter determining the contribution of the minority class to the loss.
Focal loss.
Denoting our output prediction as ÿJ = po 1), the focal loss objective for binary classification of target y4 is variant the balanced cross-entropy loss:
I focal (y,, 9() = -wi (1 in)5 yi log(ūs) wofff (1 - ys) log(1 út)
where wy, is a balancing weight for class y; and is the focal loss weight.
Multi-horizon prediction. as the following objective:
As highlighted in Section 3.3
multi-horizon training can be formalized
H log (og()))) (1))) + (1 - yit log(1 (l:e)
MHP (y... = H
where true labels and model predictions are given by Yıą = [ N........ will and Ve, W ..... 1wn for H distinct horizons.
Label smoothing. As introduced by Szegedy et al. 14 label smoothing consists of substituting
the original label distribution oy.=c in the cross-cntropy objective L, = L _C Ey,, ūi) by a smoothed version g (clys). This surrogate distribution over classes e is defined as follows:
1. (clp)) = 5y (1 a) u(c)a
(14)
In the original approach, u is uniform and a E [0, 1] controls the smoothing strength. By shifting the minimum of the objective function away from becoming overconfident during training. Alternative designs for u have been proposed (28 29 30 but are incompatible with the binary nature of adverse event prediction. In binary tasks, labeling is
1, labels smoothing prevents the model from
19

---

Under review as a conference paper at ICLR 2023
defined according to the positive class such that yi 10,1) and Ør = p 1). Label smoothing therefore becomes a linear interpolation with parameter a such that //.0 (1(3)) = p(y: = 1):
L (1lys) (1 a)y + a(1 ys)
(15)
As suggested by Lukasik et al. 277. label smoothing can be used regularize early prediction models due to the inherently noisy nature the task. does not, however, account for the time dependency between samples of given stay - highlighted in our problem formalism (Section (.1)). In contrast, temporal label smoothing modulates smoothing based on time to infuse this prior knowledge into the training objective.
B DATASET DETAILS
B.I TASK DEFINITION
In this section, we provide more details on the definition of our early prediction tasks for circulatory failure and respiratory failure from HiB (20] and decompensation from M3B [32 A breakdown of event prevalence for each clinical endpoint given in Table
Table 5: Event prevalence analysis, highlighting class imbalance. Positive timesteps are counted for 12-hour and 24-hour horizons for HiRID tasks and decompensation respectively. Statisties computed on the training set.
Patients undergoing event (%)
Number of events per positive patient
Task
Positive timesteps (%)
Circulatory Failure (HiRID) Respiratory Failure (HiRID)
4.3
25.6
1.9
38.6
83.0
1.8
Decompensation (MIMIC)
2.1
8.3
1.0
Circulatory failure is a failure of the cardiovascular system, detected in practice through elevated arterial lactate (> mmol/l) and either low mean arterial pressure (< 65 mmHg) or administration of a vasopressor drug. Yèche et al. (20) defines a patient to be experiencing a circulatory failure event at a given time those conditions are met for 2/3 of time points in a surrounding two-hour window. Early prediction labels are then derived from these event labels as outlined in Section 3.]
Respiratory failure is defined by Yèche et al. 20 as a P/F ratio (arterial pO2 over FIO2) below 300 mmHg. This definition includes mild respiratory failure, which explains higher event prevalence in Table As above, Yèche et al. (00 consider a patient to be experiencing respiratory failure 2/3 of timepoints are positive within surrounding 2h window.
Decompensation refers to the death of a patient. Event labels are directly extracted from the MIMIC- III 33 metadata about the time of death of patient. Early prediction labels are also extracted following Section 3.1 Note that decompensation can occur outside of the ICU stay if a patient is sent to palliative unit, for instance, which can result in patient stays with fewer than 24 positive samples.
B.2 PRE-PROCESSING
We describe the pre-processing steps applied to both datasets, HiRID and MIMIC-III
Imputation. Diverse imputation methods exist for ICU time series. For simplicity, we follow the approach of original benchmarks [32 20 by using forward imputation when a previous measure existed. The remaining missing values zero-imputed after scaling, corresponding to mean imputation.
Scaling. Whereas prior work explored clipping the data to remove potential outliers 8 we do not adopt this approach as we found it to reduce performance early prediction tasks. A possible explanation is that, due to the rareness of events, clipping extreme quantiles may remove parts of the signal rather than noise. Instead, we simply standard-scale data based the training sets statistics.
20

---

Under review as a conference paper at ICLR 2023
C IMPLEMENTATION DETAILS
Training details. For all models, we set the batch size according to the available hardware capacity, Because transformers are semory-consumingg we train the models for respiratory failure and decom- pensation with batch size of 8 stays. On the other hand, we train the GRU model for circulatory failure with a batch size of 64. We early stopped each model training according their validation loss when no improvement was made after 10 epochs.
Libraries. A full list of libraries and the version we used is provided in the envi ronment yml file. The main libraries on which we build our experiments the following: pytorch 1.11.0 47] scikit-leam 0.24. [48] ignite 0.4.4, CUDA 10.2.89[49) cudNN 7.6.5 [.) gin-config 0.5.0 [51]
Infrastructure. We follow all guidelines provided by pytorch documentation to ensure the reproducibility of our results. However, reproducibility across devices is not ensured. Thus provide here the characteristics of our infrastructure. We trained all models on single NVIDIA RTX2080Ti with a Xeon E5-2630v4 core. Training took between 3 and 10 hours for a single run.
Uncertainty estimation. We compute uncertainty estimates over a population of 10 training instances with different seeds. This widely-used approach has the advantage to account for the stochasticity of the training procedure, which we found to be predominant in early prediction tasks. which computes uncertainty estimate by bootstrapping the test population. We compare both approaches in Appendix D.4 to demonstrate that using a pivot bootstrap estimator decreases confidence intervals by effectively increasing the population size. To be conservative with our results, we retained the former approach to compute statisties across 10 training instances. We report the 95% confidence interval over the population means in all experiments.
This approach differs from other work 25
23
g
24
Architecture choices We used the same architecture and hyperparameters reported giving the best performance on respiratory and circulatory failure in Yèche et al. [20]. For these tasks, we only optimized embedding regularization parameters [8] Exact parameters are reported in Table 6 and
Table 7 For decompensation, as we found transformer architecture to perform better than originally proposed models [32], we carried out our own random search on validation AUPRC performance. Exact parameters for this task are reported in Table
Table 6: Hyperparameter search range for circulatory failure with GRU parameters selected by random search.
34 backbone. In bold
Hyperparameter
Values
Learning Rate
(1e-5, 3e-5, 1e-4, 3e-4)
Drop-out
(0.0, 0.1, 0.2, 0.3, 0.4)
Depth
(1, 2, 3) (32, 64, 128, 256)
Hidden Dimension
LI Regularization
(Ic-2, Ic-1, 1, 10)
C.I BASELINE IMPLEMENTATION
Balanced cross-entropy. In the binary setting, the only hyperparameter of balanced cross-entropy the relative contribution of the minority class to the loss, wi. As discussed in Section 5.2 no value of wi was found improve validation performance over the non-balanced case W1
Focal loss. A grid search over focal loss hyperparameters cross-entropy, on all tasks, no values of focal loss weight Ç or balancing weight W/ were found to outperform regular cross-entropy corresponding Ç 0 and w1 = 1.
also carried out. Similarly to balanced
21

---

Under review as a conference paper at ICLR 2023
Table Hyperparameter search range for respiratory failure with Transformer bold are parameters selected by random search.
35 backbone. In
Hyperparameter
Values
Learning Rate
(1e-5, 3e-5, le-4, 3e-4)
Drop-out
(0.0, 0.1, 0.2, 0.3, 0.4)
Attention Drop-out
(0.0, 0.1, 0.2, 0.3, 0.4)
Depth
(1, 2, 3)
Heads
(1, 2, 4)
Hidden Dimension (32, 64, 128, 256)
L1 Regularization
(le-2, le-1, 10)
Table 8: Hyperparameter search range for decompensation with Transformer 35 backbone. In bold are parameters selected by random search.
Hyperparameter
Values
Learning Rate
(le-5, 3e-5, 1e-4, 3e-4)
Drop-out
(0.0, 0.1, 0.2, 0.3, 0.4)
Attention Drop-out
(0.0, 0.1, 0.2, 0.3, 0.4)
Depth
(1, 2, 3)
Heads
(1, 2, 4)
Hidden Dimension
(32, 64, 128, 256)
LI Regularization
(le-2, 1e-1, 1, 10)
Multi-horizon prediction. Following Tomašev et al. [], we consider // horizons on both side of the true horizon h between and 2h. As didn't find Hl - +00, increase performance, selected H = 11 (including true horizon h) compared to H = 8 in Tomašev et al. (8), which we found to perform slightly worse. This means we made a prediction every 2 hours for HiB tasks and every 4 hours for decompensation.
Label Smoothing. Label smoothing [1]. as defined in Section 3.2 setting. We still compared our method to it for two reasons. First, to explore can help when dealing with noisy signal we claim is the impact of adding a temporal dependency to the method. Again, we select the hyperparameter a through a grid search. Interestingly. we found label smoothing to slightly improve performance over the validation set for all tasks as opposed to the results reported for the test set in Table We found 0.05 to perform best for circulatory failure and decompensation. For respiratory failure, we found = 0.1 to have the best validation performance.
normally used in multi-class
for early event detection. Second, ablate the
C.2 TLSS IMPLEMENTATION
TLS depends on two components, the temporal range over which we smooth labels, defined by hmin and Amaz. and the smoothing function a(i, t). Concerning the temporal range, for fair comparison, we fix it match MHP, thus for all experiments we set flmi 0 and ftmaz = 2h. For the smoothing function, we perform a grid search over the type function discussed in Appendix A.2 and the smoothing strength parameter For all experiments, we found "#p to outperform other considered functions. Given validation performance, we used 7 = 0.2 for circulatory failure and y = 0.05 for respiratory failure and decompensation.
22

---

Under review as a conference paper at ICLR 2023
def get_amocthed_ labellllaaeesssee h_max, **kwargs):
smoothing_fn, b_true, h_nin,
Fing when event lasel changes
di ffs. - up. woccatenate ([np. zerca
sattt_labe__patieeee [1:] hatpr__laae_paieettt==t--- axis--1) pos_ ees_evet__chagee np. wherh (diffs 1) & (event_label _aattlabe__paenttt_e [0]
Handle patients with events (pes_ aass__e_ee__a_eee____ee 0: pos_event_change np. np.array ( (Inp.inf)
Compute distance closest event for each time point time_array np. (event__labell (ere (-1, 1) time_array di dist_to_closest np. (dist_al > 0,
dist_allevente np.inf) .iin (axis=0)
return smoothing fn (dist toist_to__loseeeeeeeee_ ALtrue-h_true h min=h min, h_max=h_max,
**wargg)
Figure 12: Temporal label smoothing algorithm. Python-style code to obtain smooth early prediction labels from event labels.
As discussed in Section contrary to MHP, TLS does not require any change to the architecture leading to computational overhead. The smoothing of the labels can be easily integrated into the data loader, as shown in Figure 12
D ADDITIONAL EXPERIMENTS AND ABLATION STUDIES
This section provides additional results and experiments to complete our findings from the main manuscript. Unless otherwise stated, mean results are shown with 95% confidence interval on the mean shaded or in eтror bars.
D.1 EVENT-BASER METRICS FOR OTHER TASKS
Ler
Lie
3.975
Even Lnus
0.8
Lys
3.90
Evert Recell
0.925
Event Recall
0.6
0.075
0.800
3.855
0o 12h
27 to 12P
4h to 12h
Gh 12h Time
Bh 12h
10 12h
to 24h
4h 24h
8h 24h
12th 24h
16h to 24h:
20h 24h
to Event
Time Event
(a) Respiratory Failure
(b) Decompensation
Figure 13: Event recall at 50% timestep-level precision, for two additional tasks.
Event-level performance trends for decompensation and respiratory failure prediction were similar to those obtained for circulatory failure in Figure 6bl As discussed in Section 5.1 temporal label
23

---

Under review as a conference paper at ICLR 2023
smoothing improves recall of adverse event episodes eots-py and MHP. If the improvement observed over the baselines in terms of event-recall between 0 and h are smaller than for circulatory failure, in both tasks TLS improvements over both baselines are statistically significant with paired Student's p values below 0.05 for the hypothesis JTLS > (baseline-
D.2 TIMESTEP-BASE METRICS FOR OTHER TASKS
0.8
0.8
Prcctiion
0.6
Prctsion
0.6
0.4
0.4
Fcz
0.2
Laye
0.2
Lnss
Lnss
50% Precision
50% Precision
0.4
0.6
Recall
(a) Respiratory Failure
(b) Decompensation
Figure 14: Precision-recall curves, for two additional tasks. Inset shows the annically-applicablle
region with precision greater than 0.5.
ThRm TNRRrr (%)
TPRrus TPRCE (%)
4-
(a) True negative rate (TNR).
(b) True positive rate (TPR).
Figure 15: Performance improvement over time for TLS over traditional cross-entropy on de-
compensation prediction. Timestep-level metries computed for precision of 0.5 over two-hour bins.
Decompensation. Precision-recal curves obtained for timestep-level event prediction respiratory failure and decompensation tasks are given in Figure [4] As for circulatory failure prediction, decompensation recall gains concentrated in regions of low false-alarm rates (>50% precision) which are most clinically relevant. Likewise, whereas recall near the label boundary f, is slightly negatively affected by temporal label smoothing in Figure [5] true positive rates significantly improved leading up to the event time t.. This mirrors the temporal smoothing pattern which favors higher model confidence away from the label boundary. As discussed in Section 5.2, this is aligned with clinical priorities in terms of model performance, as it ensures imminent events are better predicted.
Respiratory Failure. As discussed in Section on respiratory failure, there is no clear advantage of using temporal label smoothing (or any baseline) over cross-entropy timestep level metrics as in Figure 14 This can be attributed to the more balanced nature this task. Still, we find that performance time in Figure reflects the design of temporal label smoothing, true positive rates are negatively affected near the highly smoothed label boundary but improve when approaching event time.
24

---

Under review as a conference paper at ICLR 2023
TNRRa -NNRer (%)
TPR TPRce (%)
22
4-
4-b
True negative rate (TNR)
(b) True positive rate (TPR)
Figure 16: Performance improvement over time for TLS over traditional cross-entropy res- piratory failure prediction. Timestep-level metrics computed for precision of 0.5 over two-hour bins.
D.3 SUB-GROUP ANALYSIS
Populations in the intensive care unit are often heterogeneous. This has motivated recent works to focus the fairness of deep learning across these sub-populations. In this analysis, ensure that temporal label smoothing does not negatively affect performance in specific subgroups, compared to the objectives commonly used in the literature 8
To achieve this, we measured event prediction performance across genders and age groups (below 50, between 50 and 70, and over 70 years old). As shown in Table TLS matches or outperforms bascline performance across all studied subgroups. suggesting that the overall population-wide improvements are not achieved by disproportionally favouring specific cohorts. While some algorithmic bias can be observed across all methods, for instance in poorer decompensation performance amongst female patients, TLS does not appear to be amplifying this issue. In further work, we look forward to extending this analysis to more specific subgroups and to study the fairness of early event prediction methods for clinical applications.
Table 9: Sub-group performance analysis. We color in green improvement above the 95% confi- dence interval and in orange differences within it, often within the confidence interval of cross-entropy (CH).
Circulatory Failue
Age S. 50
50 < Age $770
Age 70
Fomale
Male
Mothod
AUPRC
Recall
AUPRC
Recall
AUPRC
Recall
AUPRC
Recall
AUPRC
Recall
CE
40.4 + 0.5
29.4 = 0.6
38.8 0.6 0.6
29.6 1.1
39. 0.3
29.0 100
39.3 + 0.6
30,0 -0 0.7
30.1 + 0.4
29.0 1..0 32.0 1.88
TLS
0.5
32.7 2 1.0
1.110. 0.4
32.6 1.7
40.0 1 0.3
31.7 007
41.2 1
32.8 1.66
40.4 0.3
0.0
+23
+330
Respiraocy Failure
Age 50
50 < Age 70
Age 70
Fomale
Male
Method
AUPRC
Recall
AUPRC
Recall
AUPRC
Recall
AUPRC
Recall
AUPRC
Recall
20.3 +
62.5
78.0
63.2
81.7 +
72.9
0.2
+
70.5 0.7
62.4 0.2
7788
63.2 2
81.1 11:100
19.8..04
CETLS-CE)
+.3
0.3
00.
Decompensation
Age 30
50
70
Age 70
Female
Male
Mehhol
AUPPC:
Recall
AUPRC
Rocal
AUPRC
Rocall
AUPPC
Recall
AUPRC
Recall
CF
29.2
25.3 1.2
0.9
27.
35.8
29.4 +
0.4
0.6
38.3
314-
TLS
30.5 30.5 0.5
26.2 1 L1
36.7 0.5
29.1 0.5
36.3
30.3 0.4
31.6 0.3
25.7 0.5
39.8 1288100 32.8 0288000
ATTLS-CE)
+1.3
+1.
+18
+17
+.5
+0.9
007
+0.9
+13
+14
D.4 UNCERTAINTY ESTIMATION WITH PIVOT BOOTSTRAP
As mentioned in Appendix our uncertainty estimation approach was based on measuring standard eтror across 10 training runs. With the uncertainty evaluation framework from Tomašev et al. ( bootstrapping patients from our test set 200 times for each training instance, we obtained similar means to Table 2 but with confidence intervals all smaller than or equal to 0.1%. 1.. Variance within bootstrap samples from the same training instance is therefore much smaller than across instances. Our alternative uncertainty estimation approach, measuring variability between training runs, returns more conservative estimates, and was thus chosen for all results reported in this work.
D.5 LOss REWEIGHTING METHODS
Hyperparameter grid search results for different loss reweighting methods are shown in Figures 5 and 17 For all three tasks, both weighted cross-entropy and focal loss were found negatively affect
25

---

Under review as a conference paper at ICLR 2023
performance in comparison to traditional cross-entropy. Likely explanations for these results provided in Section [522] focal loss focuses training on noisily labeled samples, and weighted cross- entropy largely reduces precision. We validate the latter hypothesis by visualizing precision-recall curves of models trained with this objective in Figure Tg
60
38
20
59.6
59.5
59.4
59.4
55.9
20-
35.6
35.6
35.7
35.7
35.4
Positive class weight ())
Positive class weight
10-
59.6
36.4
56
AUPRC
59.9
59.8
59.8
59.5
55.3
37.2
37.2
37.2
37.0
34.8
2
60.3
60.3
60.0
60.1
52.2
37.8
37.9
37.9
37.8
34.2
52
60.5
60.5
60.3
60.3
50.2
38.1
38.1
38.0
37.8
33.4
0.1
0.5
1
0.1
0.5
5
Focal weight
Focal weight
(a) Respiratory Failure
(b) Decompensation
Figure 17: Performance loss with class reweighting methods, on validation set. Balanced cross- entropy corresponds to
O.
Impact of weighted cross-entropy on precision. With a relative weight for the positive class 0.5
Wi 6(am1) 1, weighted cross-entropy encourages a greater number of true positives improve recall. Doing so also increases the of false positives, impairing precision. In Figure [8] as the starting precision of all cross- rross-entropy models is poor, no discernible improvements in the recall can be observed as class weights are increased, whereas precision is markedly reduced in low-recall regions. This explains the overall reduction in AUPRC with this method across all tasks.
1.0
0.8
0.5
20
0.8
20%
50%
50 % 0lvl
50
0.0
0.2
0.2
0.0
0.0
0.0
Rocall
Rocall
Rocall
(a) Circulatory Failure
(b) Respiratory Failure
(c) Decompensation
Figure 18: Class reweighting impact AUPRC. Class reweighting does not improve AUPRC because significantly reduces precision. Balance weights correspond to b(c).
D.6 VISUAL COMPARISON OF TLS WITH Qstep AND MHP PERFORMANCE
we compare the precision recaal curve of multi titihoooozon prediction and temporal label smoothing with qstep smoothing. ensuring that there is no area where MHP is superior. In complement Ta 3nd and the analysis Section 5.2 this confirms that predicting single horizon with step function smoothing is sufficient to match the performance of multi-horizon prediction.
In Figure 19
D.7 COMBINING TLS WITH OTHER METHODS
Finally, we investigated whether temporal label smoothing could be combined with other objective functions leverage their respective added value and further improve prediction performance. The
26

---

Under review as a conference paper at ICLR 2023
1.0
3.0
3.0
52% Focciinn
52% Pocciion
52% PoNcito
85031
(a) Circulatory Failure
(b) Respiratory Failure
(c) Decompensation
Figure 19: Precision-recall curves of multi-horizon prediction and temporal label smoothing with Gstep. Both curves overlap, suggested by metrics in Table further demonstrating that the multiple outputs of multi-horizon prediction do not lead to superior performance, and supporting assumptions Proposition
performance of temporal label smoothing combined with weighted cross-entoe cross-entropy objective is given in Figure 20) Balanced reweighting per class results in performance drop, as observed when applied traditional cross-entropy (see Section 5.1 Figure 5). Another possible approach to combine these methods would be to leverage temporal information sample re-weighting, and we reserve this investigation for further work.
Similarly, no additional performance gains were obtained from combining multi-horizon prediction focal loss with temporal label smoothing over using TLS with cross-entropy loss.
27

---

Under review as a conference paper at ICLR 2023
0.42
Model
0.42
Cross- Cross-Entropy
0.40
TLS
0.40
Test AUPRC
0.38
AUPRC
0.38
0.36
0.36
0.34
0.32
0.30
0.30
Balanced
Identical
Balanced
Identical
Class Weight
Class Weight
(a) Circulatory failure.
0.64
Model
0.64
Cross-Entropy
0.62
0.62
Test AUPRC
AUPRC
0.60
0.60
0.58
Balanced
Identical
Balanced
Identical
Class Weight
Class Weight
(b) Respiratory failure.
0.42
Model
0.42
Tross-Entropp
Test AUPRC
0.38
AUPRC
0.38
0.36
0.36
0.34
0.34
0.30
0.30
Balanced
Identical
Balanced
Identical
Class Weight
Class Weight
(c) Decompensation failure.
Figure 20: AUPRC performance of temporal label smoothing combined with weighted cross- entropy. (Left) Test set performance. (Right) Validation set performance.
28


Reviews:
[{"review_id": "UN9teXyxFXe", "reviewer": "Reviewer_JLK5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends label smoothing [Szegedy et al., 2016] to temporal data, and relates the proposed temporal label smoothing approach to multi-horizon prediction. Experiments on real data show that the proposed temporal label smoothing approach outperforms several baselines.", "summary_of_the_review": "Overall, I find this paper to be incremental and it's unclear to me how practically significant the improvements are of the proposed method vs the baselines evaluated. Furthermore, I think that there simply aren't enough baselines considered.", "strength_and_weaknesses": "Strengths:\n1. The problem being addressed is well-motivated.\n2. The proposed TLS method is straightforward to understand.\n\nWeaknesses:\n1. The proposed TLS method appears to be a rather incremental advance, extending label smoothing to the temporal setting in a straightforward manner, and then the choice of how the temporal component is parameterized comes off as ad hoc.\n2. It took me a bit of time to figure out how to interpret Figure 2. If at all possible, I would suggest providing some additional explanation to make parsing this figure easier.\n3. In terms of discussing related work, I think it would be helpful to compare the proposed method with deep imbalanced regression [Yang et al., 2021], which also involves imbalanced data and label smoothing (I realize that the proposed approaches are different and the latter isn't tailored for early adverse event prediction; I think these authors also did look at time series data as one of their examples).\n4. In Table 2, I would suggest providing some more significant figures or at least stating something about the p-values that are \"0.00\" as I would guess they aren't actually exactly equal to 0.\n5. Statistical significance aside, I think it would be helpful to provide more commentary on how much of a *practical* difference there is between the proposed temporal label smoothing method and the different baselines. Basically even if there's a statistically significant difference in an evaluation metric, the difference might not be practically significant (some of the numbers from TLS look quite close to those of the multi-horizon baseline, for instance). If it's at all possible to try to quantify the difference in terms of how much earlier TLS can predict an outcome compared to different baselines, that would be helpful (in this case, just as an example, it could be that TLS can in some sense predict as accurately as the multi-horizon baseline 5 seconds faster, and that this 5-second difference is statistically significant but one could argue that 5 seconds is not going to make a difference practically in many clinical applications).\n6. I find the number of baselines to be rather small, and it is unclear to me why this is the case. There are *many* machine learning and data mining methods proposed for early prediction/detection/classification of various critical events. I would suggest doing a more thorough literature search to find additional baselines to try. As just a few examples of existing work (this listing is very much non-exhaustive), see the work by Xing et al. [2009], He et al. [2013], Chen et al. [2013], and Lauritsen et al. [2020], or the recent software package by Tavenard et al [2020]; note that some of these would require some sort of conversion of irregularly sampled data into regularly sampled numerical time series but there are straightforward ways to do this (e.g., discretizing time and using some imputation strategy like forward filling to fill in missing values). Moreover, for the specific applications considered, I think it's also worthwhile comparing to any sort of early detection method that is actually currently in clinical use, if any (if not, then it would be helpful indicating that this is the case).\n7. There are small typos here and there. Please proofread carefully. As a few examples (not exhaustive), equation (2) ends in \"[\" which looks like a typo, the third line of the first full paragraph of Section 4.2 has the misspelled word \"reweigthing\", and the last sentence of the first full paragraph of Section 4.3 has a missing space after a period: \".Thus\".\n\nReferences:\n- George H. Chen, Stanislav Nikolov, Devavrat Shah. A Latent Source Model for Nonparametric Time Series Classification. NeurIPS 2013.\n- Guoliang He, Yong Duan, Tieyun Qian, Xu Chen. Early prediction on imbalanced multivariate time series. CIKM 2013.\n- Simon Meyer Lauritsen, Mads Ellersgaard Kalør, Emil Lund Kongsgaard, Katrine Meyer Lauritsen, Marianne Johansson Jørgensen, Jeppe Lange, Bo Thiesson. Early detection of sepsis utilizing deep learning on electronic health record event sequences. AIIM 2020.\n- Romain Tavenard, Johann Faouzi, Gilles Vandewiele, Felix Divo, Guillaume Androz, Chester Holtz, Marie Payne, Roman Yurchak, Marc Rußwurm, Kushal Kolar, Eli Woods. Tslearn, A Machine Learning Toolkit for Time Series Data. JMLR 2020.\n- Zhengzheng Xing, Jian Pei, Philip S. Yu. Early Prediction on Time Series: A Nearest Neighbor Approach. IJCAI 2009.\n- Yuzhe Yang, Kaiwen Zha, Ying-Cong Chen, Hao Wang, Dina Katabi. Delving into Deep Imbalanced Regression. ICML 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SHEXiXd513", "reviewer": "Reviewer_kP8w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors studied the problem of improving model predictions of adverse clinical events by using a novel regularization scheme on the target labels. The main novelty lies in adapting the classical Label Smoothing technique to be temporally aware and empirical results have been presented to justify the claim of effectiveness of the method. \n", "summary_of_the_review": "Overall, the authors have proposed an interesting and simple solution to handle multi-horizon forecasts, especially for EHR data. While the presented results are interesting, the clinical meaningfulness of the experiments is not well justified. Some other aspects that may need to be addressed are as below\n\n- Section 4.3, the reported method of providing uncertainty around model predictions is arguably measuring the training stability. There are other and arguably important forms of uncertainty that typically plagues EHR data (such as data uncertainty due to selection bias and model uncertainty due to over-specification). It may be useful for the authors to justify what form of uncertainty are they reporting and how to interpret the results for model usage\n- Section 5.1, below Table 2, the authors claim that their method is 'statistically superior' - this is a vague term. They may consider reporting on the 'statistical significance' of the results\n", "strength_and_weaknesses": "Some of the key strengths of the paper are as follows;\n\n- The authors study a clinically important problem. The proposed mechanism is simple and a natural extension to classical techniques. Furthermore, being a model agnostic regularization scheme, the proposed solution can have a significant impact on medical AI problems.\n- The results at a high level is promising across 2 tasks. The authors have also provided some deliberation on when the method may not be satiable\n- Figure 5 and the associated analysis is particularly interesting. The insights about the negative weighting of re-weighting samples has the potential to impact future research\n\nThe paper may need to address the following aspects\n\n- First, the presentation of the paper makes it less comprehensible and hard to follow. Figures have been presented without properly marking the x axis (e.g Figure 4 - what does each of the ticks around x-axis represent?). The method also lacks clarity. The authors may want to improve the method presentation and use a glossary to help the readers follow the proposed notations. \n- A somewhat related criticism around the presentation can be identified in understanding the effect of the label smoothing technique (for instance Figure 2 is missing x-tick annotations as well). It seems from the illustrations that the regularization technique rewards the model to learn the near horizon data points better. If this understanding is correct, the clinical significance of the model can be debatable. For instance, predictions for certain tasks such as decompensation a few time points from actual event is clinical irrelevant as the actionability of such predictions is low. \n- This connects to the third criticism around clinical impact of the work - while the authors have claimed that clinically relevant metrics (e.g in Figure 7) is in scope, there is no sub-group level analysis to make sure the model is not unduly decreasing performance across certain sub-groups. The cutoff criteria for AU-PRC evaluation also seems arbitrary - if this was selected under the guidance of clinicians/practicioners, it should be reported.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2UtYXu3Jl-w", "reviewer": "Reviewer_zja8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Motivated to reduce model confidence with stronger smoothing at the class boundary, this paper proposed to Temporal Label Smoothing, for early prediciton of adverse event. Simple form comparison with existing methods and experiments using three ICU datasets are presented. It is also proved that stepwise temporal label smoothing is equivalent to Multi-horizon prediciton. ", "summary_of_the_review": "This paper proposes a smoothed version of regulaization called temporal label smoothing to improve boundary performace. The idea is nice for clinic practise and shown to be with better performace with certain clinic datasets with AEs repeatation rate not so high. \n\nThe presentation and idea is nice, though the final performance can be improved. ", "strength_and_weaknesses": "This paper focuses on the existing practical issue of adverse event (AE) prediciton to propose temporal label smoothing. Detailed backgroud review and ample exploration of performance under multiple existing prediction methods and smoothing methods. \n\nIt's nice that the author compared not only AUPRC but also Recall. \n\nThis reviewer is wondering how the repeated AEs were presetned in the HiRID dataset and did the author pre-process the data or train the data to add a specific feature indicating first/repeated/worsen/relieved/... AE, and if such effort can help improve the performace, especially for the with frequent repeated AEs scenario, under which the proposed method isn't significant compared to existing methods.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject