SMART: Towards Pre-trained Missing-Aware Model for Patient Health Status Prediction
Authors: Zhihao Yu
Xu Chu
Yujie Jin
Yasha Wang
Junfeng Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Multivariate Time Series
Healthcare Prediction
Electronic Medical Records
Primary area: machine_learning_for_healthcare

Abstract: Electronic health record (EHR) data has emerged as a valuable resource for analyzing patient health status. However, the prevalence of missing data in EHR poses significant challenges to existing methods, leading to spurious correlations and suboptimal predictions. While various imputation techniques have been developed to address this issue, they often obsess difficult-to-interpolate details and may introduce additional noise when making clinical predictions. To tackle this problem, we propose SMART, a Self-Supervised Missing-Aware RepresenTation Learning approach for patient health status prediction, which encodes missing information via missing-aware temporal and variable attentions and learns to impute missing values through a novel self-supervised pre-training approach which reconstructs missing data representations in the latent space rather than in input space as usual. By adopting elaborated attentions and focusing on learning higher-order representations, SMART promotes better generalization and robustness to missing data. We validate the effectiveness of SMART through extensive experiments on six EHR tasks, demonstrating its superiority over state-of-the-art methods.

SMART: Towards Pre-trained Missing-Aware Model for Patient Health Status Prediction
Zhihao Yu!.3
Xu Chu 12,4,55 Yujie Jin Yasha Wang Junfeng Zhao 13,4,66
School of Computer Science, Peking University
2 Center on Frontiers of Computing Studies, Peking University, Beijing, China
National Research and Engineering Center of Software Engineering, Peking University Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China Peeking University Information Technology Institute (Tianjin Binhai)
Nanhu Laboratory, Jiaxing, China yuzhihaodstu, pku. edu. en, chu thu_xu@pku edu. cn, wangyasha@pku.edd edu. cn
Abstract
Electronic health record (EHR) data has emerged as a valuable resource for analyz- ing patient health status. However, the prevalence of missing data in EHR poses significant challenges to existing methods, leading to spurious correlations and suboptimal predictions. While various imputation techniques have been developed to address this issue, they often obsess difficultto-interpolaaa details and may in- troduce additional noise when making clinical predictions. To tackle this problem, we propose SMART, a Self-Supervised Missing-Awan RepresenTatte Tation Learning approach for patient health status prediction. which encodes missing information via missing-aware temporal and variable attentions and learns to impute missing values through novel self-supervised pre-training approach which reconstructs missing data representations in the latent space rather than in input space usual. By adopting elaborated attentions and focusing on learning higher-order represen- tations, SMART promotes better generalization and robustmess to missing data. We validate the effectiveness of SMART through extensive experiments on six EHR tasks, demonstrating its superiority over state-of-the-arr methods. Our code is available httph:t//////t http://g.gtttt..//// thub. com com/yyzhoowrrd/SMM....
Introduction
The rapid accumulation of electronic health record (EHR) data, driven by the widespread adoption of health information systems, has opened up new avenues for analyzing patient health status. EHR data in Intensive Care Units (ICUs) primarily captures patients' laboratory tests and vital signs, providing a rich resource for describing and analyzing their health conditions. These time series data have been leveraged to predict patient prognosis and physical status, attracting significant attention from both computer scientists and medical researchers. Various deep learning methods have been developed to exploit the patterns from EHR data 22 5 7 9. aiming to assist doctors in making informed decisions, improving work efficiency, and ultimately enhancing patient outcomes.
Recent advancements in EHR analysis be divided into two categories. The first category focuses on improving the learning ability of time series by capturing feature correlations and exploring the collaborative relationships between variables with various techniques, such as convolutions [IO]. attentions [2 13 [5]. and graph neural networks [1]. These methods have achieved promising results patient health status prediction tasks, providing valuable insights into modeling the underlying patterns. However, critical challenge is the prevalence of missing data in EHRs.
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Due to the fact that patients do not undergo all tests during each visit, EHR data is often highly sparse. These missing values can compromise the integrity of learned representations and potentially mislead models to capture spurious correlations, leading to erroneous predictions of patient health status. To address this issue, the other works [[8] 19 20 attempt to interpolate missing values in the input space by capturing the dynamics and regularizing the time series. Nevertheless, these methods may struggle with the complexity of the EHR data and concentrate on difficult-to-ntten tointerrpolate details instead of capturing the implicit semantic information. Furthermore, while these studies have demonstrated the potential of imputation methods to fill in missing values and enhance performance on clinical tasks in their settings, these approaches face limitations in maximizing the predictive power of the available data. By masking certain observations to serve as targets for imputation. a portion of the data is withheld from the model and cannot be fully utilized for predicting patient outcomes. Imperfect imputations can introduce additional noise into the data, causing unintended shifts in the data distribution and hindering the performance of subsequent tasks (21].
To tackle these challenges, we propose SMART, Self-Supervised Bissing-AAwre Represen' Tation Learning approach for patient health status prediction. The primary objective of our approach is to enable the model to encode missing information effectively. It performs aariabl--independent encoding on multivariate time series inputs and utilizes Missing- blocks, namely MART blocks, to capture temporal and variable interactions, with both modules incorporating missing information. Instead of deploying the vanilla self-attention mechanism [22), we develop novel missing- aware attentions inside the MART block to cope with sparse data. Through stacking multiple blocks, the model can adequately learn correlations while perceiving missingness.
Represen learning
Another key innovation of our method lies in its two-stage training strategy. Inspired by previous work I8 23 (4 25 on imputing missing values, we adopt a self-supervised pre-training stage to enhance the ability to cope with sparse data, after which we fine-tune the model to accomplish the EHR tasks, such mortality prediction. In contrast to previous works that perform reconstruction in the input space, our model conducts missing data reconstruction in the latent space. By randomly removing a portion of observations and training the model to reconstruct its representations, we enable the model to focus on learning higher-order representations which contain more semantic features rather than struggling with unnecessary details. This approach promotes better generalization and robustness to missing data. After pre- trainiiinng, the embedding decoder for reconstruction is replaced with a task-specific decoder for patient health status prediction during fine-tuning. To further bridge the gap between these two tasks, we introduce a learnable vector that serves as the query for the proposed attention mechanism and as the basis for patient health status prediction.
We validate the effectiveness of SMART through extensive experiments on six different EHR tasks, including in-hospital mortality, sepsis, decompensation, phenotyping, and length of stay. Our results demonstrate that SMART, with its specific designs tailored to the missing characteristic of EHR data, significantly outperforms existing methods on all the metrics, setting new state-of-the-arr tasks. Furthermore, we showcase the robustness of our model under settings with higher missing rates, highlighting its potential for real- world applications in healthcare. Comparisons on the model efficiency illustrate that the proposed SMART is highly efficient, with lightweight parameters and fast training time among existing methods. Our work underscores the importance of integrating missing data awareness into representation learning for enhancing patient health status prediction and paves the way for more accurate and reliable clinical decision support systems.
these
2 Related Work
Clinical Predictive Models for EHR: Analyzing EHR data has become an increasingly popular
research topic in the medical domain 115 26 27 Numerous deep learning models have been developed to mine and leverage information from EHR data (28) 29 30
3
32 Early methods used For learning better
recurrent networks and attentions to capture temporal information I
33
34
representation and achieving higher performance, some works attempt to learn feature correlations via sophisticated network architectures [36] 37 38 For example, Baytas al. [22 and Gao et al. [9] refine the design of recurrent networks by incorporating sampling intervals and disease progression in EHR data, thereby learning more comprehensive associations. Zhang et al. [39] design hierarchical recurrent encoders to capture both fragmented and global temporal variance. Ma et al. [K] and Ma al. connections with outcomes. Other works try to incorporate medical knowledge from EHR data or
[55] adopt attention mechanisms to enhance biomarkers that have strong
2

---

human expertise 40 patients in the dataset and utilize their information to enhance learned representations and provide interpretations. Xu et al. medical code data to improve predictions. Ye et al.
40 41 42)
For example, Zhang et al. [6] and Yu et al. 43 discover similar
4
and Lu et al. [44] combine information from knowledge graphs for
445] integrate medical knowledge graphs with
patient disease progression paths to obtain better health representations. However, as discussed in the Introduction, all these methods face challenges in handling missing values. Most of them merely populate missingness with mean or front values during EHR data pre-processing, although it may be implausible and mislead the model to make wrong decisions. To address this challenge, we introduce missing-awareness awareness mechanisms multiple positions in SMART, enabling it to encode missingness and avoid the negative impact of missing values on representation learning.
Imputation Models for EHR Analysis: Imputation models are widely used in EHR analysis to handle missing values. Some works make efforts to interpolate missingness for learning better
representations via clinical predictive tasks [21 46 47 Specifically, Neil et al. [48], Che et al. 49]
and Tan et al. 47 incorporate missing information into recurrent methods. Horn et al. 50 use
set-based approach and transform time series into sets of observations modeled by set functions insensitive to misalignment. Zhang et al. [51 consider sampling frequency and unify irregular time series in multiple scales. Nevertheless, these methods do not apply reconstruction losses and do not perform real imputations. The other works impute irregularly observed values to aligned reference points. notably recurrent methods [24], variational uuto-encoders [23 [8], generative adversarial networks [24] 52), ordinary differential equations [1] [53], and probabilistic interpolation methods (0)]. However, this paradigm randomly masks existing observations and conducts both clinical tasks and interpolation simultaneously, making already sparse data even sparser when predicting health status, leading unsatisfactory performance compared to methods that do not mask observations. Recently, a pre-training approach [25) has been proposed for empowering models to impute missing values through self-supervised pre-training and then perform clinical tasks, allowing the model to execute EHR analyses on intact data while possessing imputation capabilities. Nonetheless, interpolating in the input space can lead to getting bogged down in optimizing some details rather than capturing the implicit information in the entire sequence, as well as skewing the underlying data distribution (2I). In contrast, SMART conducts missing data reconstruction in the latent space, enabling the model to focus on learning more semantic representations.
3 Methodology
In this section, we present the proposed SMART, self-supervised missing-aware representation learning approach for predicting patients' health status. In Figure I we give an overview of the information flow and highlight our designs.
3.1 Variable Independent Encoder
Notation: Let (x, m. y) denotes the EHR data of a specific patient, where the visit sequence x E RT'N contains T visits with variables. x², records the value of the n-th indicator for the patient at visit t, accompanied by binary mask m'; indicating whether the value is observed. For different patients, the visit length T can vary. To avoid the possible negative effects of varying intervals between visits, the observation intervals are aligned hours following 13 16 17 43
Every patient corresponds to a label y. indicating the diagnosis or the outcome. Our goal is to improve the model's representation learning capability achieve better performance y.
Given a patient's EHR data, we apply a ceriable-independen encoding strategy to map (æ, m)
into a latent representation E xxxxx inspired by the success of previous works [5]
51 [4].
where d is the dimension of latent space. Although variable-dependent approaches (9 10 capture
variable interactions and compress dimensions when embedding. such methods cannot learn further associations, i.e., the attention weight of each variable [33. Unlike variable-inddependent recurrent models 13] 17 49 we adopt linear projections to encode data, allowing the entire encoding process to be parallel and without accumulating noise from past visits. When handling variable values æ and masks m, we combine them directly and let the model capture the interactions between them.
CLS Vector: When encoding the EHR data, we introduce a learnable vector which is concatenated before the time series. The role of this vector is to learn the information of the whole sequence in subsequent interactions and act as the pooled hidden state used for prediction (similar to the
3

---

Embedding Decoder
Label Decoder
Feed Forward
+
Layer Normalization
MART Block
MART Block
MART Block
Variable Attention
X
Update
Layer Normalization
Input Encoder
Input Encoder
Input Encoder
Temporal Attention
Layer Normalization
X
Variable independent Encoder
192
272
CLS Vector
CLS Mask
Pre-training
Fine-tuning
Figure I: Overview of SMART. Left: Given EHR data with missingness, randomly mask them on the existing observations and conduct reconstruction the latent space. The reconstruction targets generated by EMA updated parameters. Right: We illustrate the detailed architecture of the input encoder and the MART block. The input encoder embeds each variable (which can also be referred as biomarker) and missing mask into separate hidden space. The MART block employs various techniques to capture feature interactions in both the temporal and variable dimensions while further encoding missing information.
[CLS] token in the language model such as BERT (55 ). This design also bridges the gap between pre-training and fine-tuning tasks, the information in the vector used for both reconstruction and prediction. Specifically, we concatenate the CLS vector v E IR N xd at the temporal dimension before the hidden representation h. which can be formulated as h, [u, ht). where h, E R ()*+1)x Xxd This process can be interpreted as concatenating learnable parameter before the time series æ and True vector before the mask m, making the consultation sequence one step longer (as illustrated in Figure []). Here, we concatenate on the embedding h instead of adding vectors the input data since this procedure provides more flexibility in the learned representation space. At the end of this module, we introduce positional information to the representation h, for subsequent interactions in the MART blocks using sinusoidal positional encoding [22
3.2 MART Block
The MART block is the core module elaborated learn the patient's health representation. It is mainly composed of two attention mechanisms operating on the temporal and variable dimensions. To further mitigate the impact of missing data on representation learning, we introduce masks into the attention mechanism and strengthen the attention weights of existing observations.
Temporal Attention: The variable-independenn encoder described above embeds patient information into representation h. E [[T+1)xxNx In this section, introduce how calculate temporal weights leveraging mask information. Following the convention for self-attention mechanisms, we compute the query, key, and value via linear transformations of h for the temporal attention. denoted as Q temp: K Ktemps and Viemp. To incorporate mask information, we construct the temporal attention bias B E IR. (gT+1)x((+1)xx The bias Bid between visit i and for the n-th variable can be computed by
2, m" True and m" True, m" m n True,
(I)
m = False and m / = False,
where m', [True, m.) and [ denotes concatenation of matrices. Then obtain the temporal attention weights through softmax Qremp KJ temp
+ B), where both matrix multiplication and softmax are imposed on the temporal dimension. In this way, strengthen information from
Vd

---

observed visits and suppress the others. At the same time, we do not completely block the missing visits, providing an opportunity for the model to interpolate them and utilize their information.
Variable Attention: We capture correlations between variables via the proposed variable attention. In contrast previous approaches which calculate variable interactions separately for each visit [5T 54), we capture variable relationships from global perspective of the patient. Given representation htemp E (T+ Xxd for the variable attention mechanism as follows:
xd from the temporal attention, we get query Quar. key varr, and value Vear
Qvar Linear (nnear(hempp))
K KPar var Linear
Σ
hiomp where m = -True)
var = Linear Linear(htemp)
The query Qear is only obtained using the vector at the first step (hiemp), which is also the location of learnable parameter inserted in the input encoder. This operation motivates these vectors to learn the overall health state of the patient. The key K Kear are acquired using the averaged embedding of all observed visits to minimize the effect of missingness in visits. Then we calculate the invariat
Q par K
correlations between the variables and get attention output by softmax
V vear
Between these attentions, we utilize layer normalizations [6) and skip connections 57 to avoid overfitting and accelerate convergence. Besides, feed-forward constituted by linear projections and activation functions is used following the vanilla design of Transformer [22]. The MART block can be stacked in multiple layers to allow for sufficient interactions. The final health status of patients embedded by the MART blocks is denoted as [TT+1)xXxd
3.3 Two-Stage Training Strategy
Pre-training Stage: To empower the model with missing imputation capabilities to enhance the learned representation, we propose self-supervised pre-training method that occludes some of the observations and reconstructs their representation in the hidden space. Different from previous methods (25), we do not seek complex manual data augmentation, but simply remove some of the observations as the targets whose representations will be reconstructed. When generating the targets, we generate them randomly with a probability interval rather than with a fixed probability. This encourages the model to achieve better generalization across sequences with different sampling rates, rather than overfitting on missing probabilities. For the purpose of strengthening the reconstructing capability additionally and avoiding the model being trapped in the local minimum, we randomly sample the targets each epoch of pre-trae pre-training instead of using fixed data.
To emphasize a nontrivial pre-training task, we apply a self-motivated paradigm with an asymmetric architecture inspired by [58), illustrated in Figure Specifically, given the EHR data (ær, m), we randomly generate a mask in to remove the existing observations partially and obtain augmented data (x" m²) Defining the modules being trained (input encoder, MART blocks, and embedding decoder) as f and the modules (input encoder and MART blocks) used to generate labels as f. the reconstructions Syre-train are generated by f using augmented data, while the reconstruction targets & are acquired by embedding the original data via f whose parameters are updated by exponential moving average (EMA) (59) of the parameters from f. The embedding decoder consists of an MLP with activation functions that uses the patient health representation s" E [TT++)xxxxd from augmented data as input and outputs the reconstructed health state Spre-train E (T+ This paradigm provides a smooth label update curve that avoids model oscillations and underfitted embedding decoder. The pre-training loss is computed by L1 distance considering only the features of the removed data (i.e., the position where in True) the latent space:
Cyre-train Σ m ppe-trasn 8||1.
Since the model is trained to reconstruct the missing data, it is forced to learn the underlying structure of the data and the temporal relationships between variables, which can be beneficial for subsequent tasks. In particular, the CLS vector is not aligned in Cprestrain (i.c., the position of the CLS vector is
5

---

False in the mask r)), which allows the model to store the information of the whole sequence in the CLS vector and exploit it in the fine-tuning stage.
Fine-tuning Stage: After pre-training, we replace the embedding decoder with a task-specific decoder for patient health status prediction, such as classification task. While making predictions, only use the representation at the first step (s"), which is the position of the CLS vector. The label decoder an MLP with layer normalizations and activations which can be simplified as a projection function R Nxd R el In the fine-tuning stage, the parameters (e.g. cross-entropy for classification). In the first few rounds of training, we freeze the parameters of the other modules and only update the label decoder to make the pre-trained parameters to be reserved. The proposed two-stage training procedure can be formulated Algorithm
updated by the task-specific loss
Algorithm Algorithm SMART
1: Input: EHR data æ and its mask m.
2: Output: Patient health status prediction ý-
3: for all E in pre-training epochs do
4: Sample a mask rn with probability interval p and generate augmented data (æ*,, mm*)
5: Generate reconstructions 8 Spretrrii <- f(xx,, mm) ) and reconstruction targets $ < f(x m))
6: Update f by minimizing Eprestrain
7: Update f- Jf)
8: end for
9: Freeze the parameters of the input encoder and the MART blocks
10: for all E' in fine-tuning epochs do
11: if E unfreeze epoch then
12: Unfreeze the parameters of the input encoder and the MART blocks
13: end
Update parameters by minimizing the task-specific loss with prediction
15:
end for
4 Experiments
4.1 Experimental Settings
Datasets: We follow previous works 110
13 17
51
compare models three EHR datasets,
Cardiology [60] 61.. Sepsis (62). and MIMIC-III 663 The Cardiology dataset consists of 37 vital signs and biomarkers describing patients admitted cardiac, medical, surgical, and trauma ICUs. Each record contains sparse measurements from the first 48 hours after admission. We follow the
preprocessing procedures of previous works [13] 21 where the observation times are aligned to hours. After preprocessing, there are 11,988 patients and the observed rate is 24.7%. The prediction target to determine in-hospital mortality. 13.8% of examples are in the positive class. The Sepsis dataset contains 34 vital signs and laboratory values relevant to sepsis. The EHR data are recorded once an hour including 40,335 patients from ICUs. The prediction task is to identify the patient's risk of developing sepsis, with a positive rate of 7.3%. For convenience, we used only the records of the first 60 visits for each patient. The observed rate only 19.8%. The MIMIC-III dataset is multivariate time series dataset consisting of 17 physiological signals after pre-processing. It contains many clinical scenarios calling for accurate predictions over diversified clinical signals. We conduct four clinical tasks on the MIMIC-III including in-hospital mortality, decompensation, phenotyping, and length of stay. Additional details on these datasets can be found in Appendix A
We assess the performance on the binary classification tasks (including Cardiology. Sepsis, in-hospital mortality, and decompensation) using the area under the precision- recall curve (AUPRC) and FI Score. AUPRC is the most informative and primary evaluation metric when dealing with a highly imbalanced and skewed dataset [64] such as healtheare data. We calculate FI Score focusing on positive patients, which is more relevant in clinical scenarios. Phenotyping multi-label classification, each label indicating the diagnosis of a phenotype. Therefore, we examine phenotyping the macro and micro area under the receiver operating characteristic curve (AUROC), abbreviated as ma-ROC and mi-ROC, respectively. For length-of-stay prediction, we follow Harutyunyan et al. 665 to separate labels into multiple bins and evaluate the ma-ROC and
Evaluation Protocols:

---

Table 1: Performance comparison with standard deviation on the six clinical tasks. The best results are marked in bold. The second best results are underlined.
Cardiology
Sepsis
In-hospital Mortality
Model
AUPRC(%)
Score(
AUPRC(%)
FI Score( %6)
AUPRC(%)
FI Score (%)
AdaCare
10.54
27. 27.90
41.13
36.87 0.96
1.62
27.47.11.8
StageNet
42.01
38.27=2.74
59.11
54.89
51.77=3.62
40,90=11.0
ConCare
47.77
30.22.1..79
14.83
63.27
51.64
35.76
GRASP
15.73
31.9012.60
74.47
66.7513441
51.84
38.00 1.75
SAFARI
45.58+3.59
11.06:22.09
72.06+1174
69.06=1.14
48.96=3.01
33.96=3.83
PPN
49.98-197
37.64-13.19
72.70:14.99
68.08
51.24
40.60 1.83
RainDrop
37.23=1.20
23.85:+3.39
72.82++3.68
67.58=2.54
46,21
36.29:+3.20
Warpformer
48.10=33.95
40.26:13.68
73.03++2.55
6..93-+1.79
11.49
36,83+3,12
PrimeNet
49.25
37. 2771077
33.38:22299
12.46=287
51.41
38.7 38.71
SMART
23.84-2.24
47.53
81.67
75.37=2.62
53.30=0.12
44.23
Decompensation Score(%)
Phenotyping
Length of Stay
Model
AUPRC(%)
ma- ma-ROC(%)
mi mi-ROC(%)
ma- ma-ROC(%)
mi mi-ROC((%)
AdaCare
12.41
51.62.10.84
63.28 10.62
73.7510.22
66.06.00.7
82.1 .10.40
StageNet
64.97=0.60
62.51
73.33=0.30
79.53=00.11
69.54=0.39
83 13.26=00.11
ConCare
1..50
59.73
71.05
78.03
69.27 10.50
83. 83.19 190.2
GRASP
64.29=1064
58.48=0013
69, 69.08
76.97=0025
69.24=0.47
83.20=#0.00
SAFARI
1..73
59.71
66.91
75.69
68.51
02.23
PPN
64.91-10.3
62.80
68.77-00.4
76.55=0.41
68.5220048
83.13
RainDrop
61.16:1144
58.2112.61
71.59
78.3
6..23
82.83
Warpformer
1..21
63.70:+0.88
74.65+0.23
80.36=0.23
69.53=0:45
83.34
PrimeNet
1..86=1.32
51.59
68.90=0.17
76.66=0.19
67.88=0.40
82.89
SMART
71.26:20.81
67.38:10.67
76.13=20.44
81.41
70.29=0.44
83.92=10.44
mI-ROC. We randomly divide the data set into training set containing 80% of the patients, a validation set of 10% patients, and test set containing the remaining 10% instances. The model achieving the best AUPRC (or ma-ROC) on the validation set evaluated on the test set. To eliminate the randomness, conduct each experiment with three random seeds and report both the mean and standard deviation of the results. Results on more metrics are provided in Table Hyperparameters and implementation details are introduced in Appendix A.2
Baselines: We organize existing methods of handling EHR data into three paradigms. The first type utilizes various techniques to enhance representation learning of EHR data. Such as AdaCare 10] multi-scale model focusing on extracting information from multiple levels, StageNet (}] which refines the design of LSTM by incorporating personalized disease stage development, and ConCare [3], which consists ariaabe-inddepndent encoders and self-attention mechanism to learn feature correlations. Works that employ global information other patient information to aid in modeling also belong to this category, including SAFARI [77. which learns feature correlations from group-wise perspective and integrates correlations by graph neural network (GNN), PPN [43], model that extracts typical patients and uses them help predicting and interpreting, and GRASP 16), which exploits similar patients and GNN to improve learned representation. The second paradigm is the state-of-the-art that delves into interpolating missingness and achieving better performance on clinical tasks, such as RainDrop (21). which utilizes GNN to capture the dependency among variables, Warpformer [5I]. which estimates the sampling frequency and interpolates missingness with attentions. The others are supervieed pre- pre-training methods, including PrimeNet [25), model that is aware of missingness and adopts input reconstruction with contrastive learning to pre-train the model.
4.2 Experimental Results
4.2.1 Main Results
We include the overall comparison results on the six tasks in Table 1 SMART consistently out- performs existing baselines across all prediction metrics. Specifically, on the binary classification tasks, SMART achieves the results with average improvement of 3.80% and 5.1% absolutely
7

---

Table 2: Ablation study of SMART on the Cardiology, Sepsis, and MIMIC-III in nn-hospital mortality.
Cardiology
Sepsis
In-hospital Mortality
SMART
AUPRC(%)
Fl Score(%)
AUPRC(S)
F1 Score(%)
AUPRC(%))
Fl Score(
Full
23.84=2.24
47.53
81.67
75.37=1262
53.30:0.12
44.23:22.03
w/ Imputation
22.77=2007
22.40:2.00
81.04
74.71
52.91
43.69
w/o Pre-training
52.26=3.14
46.60+2.07
3:.24
74.26:42.65
51.84+0.95
41,69
w/o Mask
49.03=1193
41 41.98:1.71
76,35
69.51+2.11
50.43+22
39, 1,6
w/o Temporal Attention
49.28=2294
40.5342.31
65.22:1144
58.47:22.6
46.641166
33.37=382
w/o Variable Attention
52.27=2.53
44.75:2..2
80,47
74.65+22.9
50.98:10.62
41.70=22.2
w/o CLS Vector
52.96=0.34
46.4221.77
77.56=2.89
71.53:+2.56
50.78:1.64
43.8622285
on AUPRC and F1 Score compared to the best baseline, respectively. The results demonstrate the effectiveness of SMART in learning representations and predicting patient health status. AdaCare performs the worst in the baseline, which may be due to the fact that its convolutional structure only averages the proximity visits and does not perceive the missing ones. Interestingly, although StageNet and PrimeNet show competitive performance on some tasks such as Cardiology and in-hospital mortality, they perform surprisingly poorly on the Sepsis dataset. This is because the Sepsis dataset has a lower observed rate, which makes it more challenging for models to learn effective representa- tions. Some recurrent models, including ConCare, GRASP, and PPN, exhibit robust performance across all datasets among the baselines. Nevertheless, with the capability of encoding missingness, SMART achieves remarkable performance improvements. As observed, the datasets with higher missing rate (Cardiology and Sepsis), SMART outperforms the other methods by a larger margin, indicating its robustness to missing values. On the phenotyping and length-of-stay prediction, our proposed model achieves the best results as well, showing its generalization ability to different clinical scenarios. The results also show that the performance of SMART more stable than other methods, as indicated by the smaller standard deviation.
4.2.2 Ablation Study
To evaluate the effectiveness of each component in SMART, we conduct ablation study the Cardiology, Sepsis, and in-hospital mortality. Firstly, we investigate the effects of different pre-training strategies on model performance. We introduce two variants of the self-supervised pre-training strategy: (1) imputing the missing values in the input space like previous methods (w/ Imputation), and (2) directly training the model for patient health status prediction without the proposed self supervised pre- tr-ininiing (w/o Pre-training). As the results shown in Table 2 we observe that although combining the pre-training strategy in the input space can improve the performance compared to models without pre-training it is still inferior to the proposed strategy that reconstructs in the representation space. These findings highlight the necessity of the pre-training strategy that integrates the missing imputation ability by reconstructing latent representations.
[18 24 25
Additionally, we explore the importance of different components and designs in SMART, and the results are also presented in Table 2 In particular, we compare with the following reduced variants: (1) w/o Mask, which removes the mask information totally in SMART, including mask in the input encoder and the attentions in the MART blocks; (2) w/o Temporal Attention, which removes the temporal attention mechanism in SMART: (3) w/o Variable Attention, which removes the variable attention mechanism in SMART; and (4) w/o CLS Vector, which removes the CLS vector in the input encoder and using the representation last observation as query in the variable attention and prediction. We observe that all components significantly contribute to the improvement. Notably, incorporating missing information is important for the model to learn high-quality representations. The results show that the temporal and variable attention mechanisms are crucial and fundamental in capturing temporal and variable dependencies, respectively. The CLS vector also plays a critical role in improving the effect of pre-training since it narrows the difference between the two training stages. By considering these components together, SMART is able capture the intricate temporal relationships and characteristics inner sparse EHR data.
8

---

AdeCare
StageNNet
ConCare
GRASP
SAFARI
Rainrrop
Warpforme
PrimeNet
SMART
13.0
80.0
300
50.0
AUPRC199)
45.0
AUPRC159)
43.0
40.
ALPRICIO
00.0
15.0
10.
33.0
10.0
30.-
5ON
Observed Rate
Observed
(a) Cardiolagy
(b) Sepsis
(c) In-hospital Mortality
Figure 2: Performance on different observed ratio of EHR.
80.0
52.5
ConCar
$2.0
CorCa Wepfomee
Stupofict
70.0
Worpformer
SAFARI
PPN
50.0
Primehet
AUPRCTK)
47.5-
45.0
BRASP
SAPAR
AUTRCIK)
60.0
Stapdhet
AUPRCIK)
48.0
SAAAN
42.5
50.0
46.0
40.0
AgcCoroN23K
46.0
Ra
37.5
40.0
Frmeetor
42.0
35.0
30.0
20
30
40
50
70
25
50
75
100
125
175
20
60
20
100
120
GPU Minutes
GPU Minutes
GPU Minutes
(n) Cardiology
(b) Sepsis
(6) In-hospital Mortality
Figure 3: Training time, parameters, and AUPRC(%) of all models on the three datasets. The size of the circle represents the number of parameters. The GPU runtime is counted at the start of training.
4.2.3 Effect of Missingness
To further investigate the impact of missingness in the data on performance, we conduct a com- prehensive experiment on the Cardiology, Sepsis, and in-hospital mortality tasks by varying the observed rate from 10% to 100%. The results of AUPRC are shown in Figure AdaCare shows the most significant performance degradation the observed rate decreases, which confirms that the convolutional architecture inside AdaCare is sensitive to missing values. Though Warpformer is stable on the Sepsis, cannot handle the missingness well the Cardiology and in-hospital mortality, Among the baselines, StageNet and PPN exhibit robustness to missing values. However, there is still large margin compared to SMART. Especially, on the Sepsis, we observe only very small decline on SMART despite only 10% of the available data. SMART performs outstandingly under different missing rate scenarios, demonstrating its fruitful results in missingness perception.
4.3 Model Efficiency
We conduct runtime and parameter comparisons between our method and baselines on the Cardiology, Sepsis, and in-hospital mortality prediction tasks. The number of parameters, the averaged runtime (minutes) on GPU (counted at the start of training), and the AUPRC of different methods are reported in Figure 3 for comparisons. Since the Sepsis dataset contains a larger number of patients, the model trained on it takes longer on average. As illustrated. RainDrop has the largest number of parameters and its training time is several times that of some of the models, such as AdaCare, ConCare, GRASP, Warpformer, and SMART, yet its prediction performance not satisfactory and unstable on different datasets. StageNet, PPN, and PrimeNet achieve competitive performance on some of the datasets, however their training times are very long. Although AdaCare has the fewest parameters and the shortest training time, it also has one of the worst performances. In nutshell, SMART is lightweight, fast, and reliable among all the models, reaching the best performance with only fewer parameters and shorter training time, which confirms its efficiency and effectiveness. Nevertheless, it is worth

---

mentioning that due to the temporal attention mechanism in SMART performs in the visit dimension, its training time may grow quadratically as the number of visits mereases.
5 Conclusions and Limitations
In this work, we propose SMART. a self-supervised pre-trained model that is designed to handle the challenges of missingness and irregularity in EHR data. We introduce two-stage training strategy that integrates imputing missingness in the representation space to enhance performance on clinical tasks. We elaborate on novel MART block that captures temporal and variable interactions by introducing masks into the attention mechanism. Extensive experiments on six EHR tasks demonstrate that SMART outperforms existing baselines. The ablation study shows that all components in SMART contribute to the improvement. We further display that SMART is lightweight, efficient, and robust to missing values and achieves stable performance across different missing rates. In the future, we plan to provide more insights into the decision- process and make more explainable.
Despite the promising results obtained in our work, it is important to acknowledge its limitations. Like all models with temporal attention, the time and space complexity of our model is quadratic. which means that the model may face slow speed or insufficient memory when the input sequence is too long. Besides, we evaluated the experimental results with three datasets in reference to previous work, while there remain other datasets that could be taken into account.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (No.82241052).
References
[1] Huan Song, Deepta Rajan, Jayaraman Thiagarajan, and Andreas Spanias. Attend and diagnose: Clinical time series analysis using attention models. Proceddings of the AAAI conference on artificial intelligence, volume 32, 2018.
[2] Junyu Luo, Muchao Yc, Cao Xiao, and Fenglong Ma. Hitanet: Hicrarchical time-aware attention networks for risk prediction on electronic health records. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 647-656, 2020.
[3] Liantao Ma, Xinyu Ma. Junyi Gao, Xianfeng Jiao, Zhihao Yu, Chaohe Zhang, Wenjie Ruan, Yasha Wang. Wen Tang, and Jiangtao Wang. Distilling knowledge from publicly available online emr data to emerging epidemic for prognosis. In Proceedings of the Web Conference 2021, pages 3558-3568, 2021.
[4] Yongxin Xu, Kai Yang, Chaohe Zhang, Peinie Zou, Zhiyuan Wang. Hongxin Ding, Junfeng Zhao, Yasha Wang, and Bing Xie. Vecocare: visit sequenecs-elinical notes joint learning for diagnosis prediction in healthcare data. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 4921-499 4929, 2023.
[5] Yongxin Xu, Xu Chu, Kai Yang. Zhiyuan Wang. Peinie Zou, Hongxin Ding, Junfeng Zhao, Yasha Wang. and Bing Xie. Seqcare: Sequential training with external medical knowledge graph for diagnosis prediction in healtheare data. In Proceedings the ACM Web Conference 2023, pages 2819-28300 2023.
[6] Pengcheng Jiang, Cao Xiao, Adam Richard Cross, and Jimeng Sun. Grapheare: Enhancing healthcare predictions with personalized knowledge graphs. In The Twelsth International Conference on Learning Representations, 2024.
[7] Isotta Landi, Benjamin S Glicksberg, Hao-Chih Lee, Sarah Cherng, Giulia Landi, Mattco Danieletto, Joel T Dudley, Cesare Furlanello, and Riccardo Miotto. Deep representation learning of electronic health records to unlock patient stratification scale. NPJ digital medicine, 3(1):96, 2020.
[8] Riccardo Miotto, Li Li, Brian Kidd, and Joel T Dudlcy. Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. Scientific reports, 6(1):1-10, 2016.
[9] Junyi Gao, Cao Xiao, Yasha Wang. Wen Tang, Lucas M Glass, and Jimeng Sun. Stagenet: Stage-aware neural networks for health risk prediction. In Proceedings of The Web Conference 2020, pages 530-540, 2020.
10

---

[10] Liantao Ma, Junyi Gao, Yasha Wang, Chaohe Zhang, Jiangtao Wang. Wenjie Ruan, Wen Tang, Xin Gao. and Xinyu Ma. Adacare: Explainable clinical health status representation learning via scale-adaptive feature extraction and recalibration. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 825-832. 2020.
[11] Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances in neural information processing systems, 29, 2016.
[12] Inci Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K Jain, and Jiyy Zhou. Patient subtyping via time- aware Istm networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 65-74, 2017.
[13] Liantao Ma, Chaohe Zhang, Yasha Wang, Wenjie Ruan, Jiangtao Wang. Wen Tang, Xinyu Ma, Xin Gao, and Junyi Gao. Concare: Personalized clinical feature embedding via capturing the healthcare context. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages
2020.
[14] Tian Bai, Shanshan Zhang, Brian L Egleston, and Slobodan Vucetic. Interpretable representation learning for healthcare via capturing disease progression through time. In Proceedings of the 24th ACM SIGKDD international conference knowledge discovery & data mining, pages 43-51, 2018
[15] Liantao Ma, Chaohe Zhang, Junyi Gao, Xianfeng Jiao, Zhihao Yu, Yinghao Zhu, Tianlong Wang. Xinyu Ma, Yasha Wang, Wen Tang, et al. Mortality prediction with adaptive feature importance recalibration for peritoneal dialysis patients. Putterns, 4(12), 2023.
[16] Chaohe Zhang, Xin Gao, Liantao Ma, Yasha Wang, Jiangtao Wang, and Wen Tang. Grasp: generic framework for health status representation learning based on incorporating knowledge from similar patients. In Proceedings of the AAAI conference on artificial intelligence. volume 35, pages 715-723, 2021.
[17] Xinyu Ma. Yasha Wang, Xu Chu, Liantao Ma, Wen Tang, Junfeng Zhao, Ye Yuan, and Guoren Wang- Patient health representation learning via correlational sparse prior medical features. JEEE Transactions on Knowledge and Data Engineering. 2022.
[18] Satya Narayan Shukla and Benjamin Marlin. Multi time attention networks for irregularly sampled time series. In International Conference Learning Representations, 2021.
[19] Yulia Ruhanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for iregularly-sampled time series. Advances in neural information processing systems, 32, 2019.
[20] ScungHyun Kim, Hyunsu Kim, Eunggu Yun, Hwangrae Lec, Jachun Lec, and Juho Lec. Probabilistic imputation for time-series classification with missing data. In International Conference Machine Learning, pages 16654-16667. PMLR, 2023.
[21] Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In International Conference on Learning Representations, 2022.
[22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[23] Satya Narayan Shukla and Benjamin Marlin. Interpolation-predittionn networks for irregularly sampled time series. In International Conference on Learning Representations, 2019.
[24] Xiaoye Miao, Yangyang Wu, Jun Wang, Yunjun Gao, Xudong Mao, and Jianwei Yin. Generative semi- supervised learning for multivariate time series imputation. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 8983-8999 8991, 2021.
[25] Ranak Roy Chowdhury, Jiacheng Li. Xiyuan Zhang, Dezhi Hong, Rajesh K Gupta, and Jingbo Shang Primenet: Pre-training for irregular multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7182-7192, 2023.
[26] Xiongcai Cai, Oscar Concha, Enrico Coiera, Fernando Martin Sartin-Sanchez, Richard Day, David Roffe, and Blanca Gallego. Real-time prediction of mortality, readmission, and length of stay using electronic health record data. Journal of the American Medical Informatics Association, 23(3):553-55) 16153561,220 2016.
11

---

[27] Xinlu Zhang, Shiyang Li. Zhiyu Chen, Xifeng Yan, and Linda Ruth Petzold. Improving medical predictions by irregular multimodal electronic health records modeling. In International Conference on Machine Learning, pages 41300-41313, PMLR, 2023.
[28] Tengfei Ma, Cao Xiao, and Fei Wang. Health-atm: A deep architecture for multifaceted patient health record representation and risk prediction. In Proceedings of the 2018 SIAM International Conference on Data Mining, pages 261-269. SIAM, 2018.
[29] Yuqi Si, Jingcheng Du, Zhao Li, Xiaoqian Jiang. Timothy Miller, Fei Wang. W Jim Zheng, and Kirk Roberts. Deep representation learnnng of patient data from electronic health records (ehr): A systematic review. Journal of biomedical informatics, 115:103671, 2021.
[30] Chaohe Zhang, Xu Chu, Liantao Ma, Yinghao Zhu, Yasha Wang, Jiangtao Wang, and Junfeng Zhao. M3care: Learning with missing modalities in multimodal healtheare data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2418-2428, 2022.
[31] Jingyue Gao, Xiting Wang, Yasha Wang, Zhao Yang, Junyi Gao, Jiangtao Wang. Wen Tang, and Xing Xie. Camp: Co-attention memory networks for diagnosis prediction in healthcare. In 2019 IEEE international conference on data mining (ICDM), pages 1036-1041. IEEE, 2019.
[32] Tian Bai and Slobodan Vucetic. Improving medical code prediction from clinical text via incorporating online knowledge sources. In The World Wide Web Conference, pages 72-82, 2019.
[33] Xiaohan Li, Shu Wu, and Liang Wang. Blood pressure prediction via recurrent models with contextual layer. In Proceedings of the 26th International Conference on World Wide Web, pages 685-693, 2017.
[34] Yanbo Xu, Siddharth Biswal, Shriprasad R Deshpande, Kevin O Maher, and Jimeng Sun. Raim: Recurrent attentive and intensive model of multimodal patient monitoring data. In Proceedings of the 24th ACM SIGKDD international conference on Knowledge Discovery & Data Mining, pages 2565-2573, 2018.
[35] Fenglong Ma, Radha Chitta, Jing Zhou, Quanzeng You, Tong Sun, and Jing Gao. Dipole: Diagnosis prediction in hcalthcare via attention based bidirectional recurrent neural networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1911-1911, 2017.
[36] Chengxi Zang and Fei Wang. Scehr: Supervised contrastive learning for clinical risk prediction using electronic health records. In Proceedings. IEFF International Conference Data Mining, volume 2021, page 857. NIH Public Access, 2021.
[37] Yujie Feng, Jiangtao Wang, Yasha Wang, and Sumi Helal. Completing missing prevalence rates for multiple chronic diseases by jointly leveraging both intra-and inter-disease population health data correlations. In Proceedings of the Web Conference 2021. pages 183-193, 2021.
[38] Chang Lu, Tian Han, and Yue Ning. Context-awarr health event prediction via transition functions on dynamic discase graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4567-44 4574, 2022.
[39] Jinghe Zhang, Kamran Kowsari, James H Harrison, Jennifer M Lobo, and Laura E. Barnes. Patient2vec: personalized interpretable deep representation of the longitudinal electronic health record. IEEE Access. 2665333-55346. 2018.
[40] Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F Stewart, and Jimeng Sun. Gram: graph- based attention model for healthcare representation learning. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 787-795, 2017.
[41] Fenglong Ma, Quanzeng You, Houping Xiao, Radha Chitta, Jing Zhou, and Jing Gao. Kame: Knowledge- based attention model for diagnosis prediction in healthcare. In Proceedings of the 27th ACM international conference on information and knowledge management, pages 143-752, 2018
[42] Junyi Gao, Chaoqi Yang, Joerg Heintz, Scott Barrows, Elise Albers, Mary Stapel, Sara Warfield, Adam Cross, and Jimeng Sun. Medml: fusing medical knowledge and machine learning models for carly pediatric covid- 19 hospitalization and severity prediction. Iscience, 25(9), 2022.
[43] Zhihao Yu, Chaohe Zhang, Yasha Wang, Wen Tang, Jiangtao Wang, and Liantao Ma. Predict and interpret health risk using ehr through typical patients. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1510. IEEE, 2024.
[44] Chang Lu, Chandan K Reddy, Prithwish Chakraborty, Samantha Kleinberg, and Yue Ning. Collaborative graph learning with auxiliary text for temporal event prediction in healthcare. 2021.
12

---

[45] Muchao Ye. Suhan Cui, Yaqing Wang, Junyu Luo, Cao Xiao, and Fenglong Ma. Medpath: Augmenting health risk prediction via medical knowledge paths. In Proceedings of the Web Conference 2021. pages 1397-1409, 2021.
[46] Qianli Ma, Sen Li, and Garrison W Cottrell. Adversarial joint-learing recurrent neural network for incomplete time series classification. JEEE Transactions on Pattern Analysis and Machine Intelligence, (K(4):765--1766 2020.
[47] Qingxiong Tan, Mang Ye, Baoyao Yang. Siqi Liu, Andy Jinhua Ma. Tery Cheuk-Furg Yip, Grace Lai- Hung Wong, and PongChi Yuen. Data-gru: Dual-attention time-aware gated recurrent unit for irregular multivariate time series. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 930 (00-9937,22 2020.
[48] Daniel Neil, Michael Pfeiffer, and Shih Liu. Phased Istm: Accelerating recurrent network training for long or event bese- sequences. Advances in neural information processing systems, 29, 2016.
[49] Zhengping Che. Sanjay Purushotham, Kyunghyun Cho. David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085, 2018.
[50] Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. International Conference on Machine Learning, pages 4353-4363. PMLR, 2020.
[51] Jiawen Zhang, Shun Zheng, Wei Cao, Jiang Bian, and Jia Li. Warpformer: A multi-scale modeling approach for irregular clinical time series. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3273-3285, 2023.
[52] Ying Zhang, Baohang Zhou, Xiangrui Cai, Wenya Guo, Xiaoke Ding, and Xiaojie Yuan. Missing value imputation in multivariate time series with end-to-end generative adversarial networks. Information Sciences, 551:67-82, 2021.
[53] Yuqi Chen, Kan Ren, Yansen Wang, Yuchen Fang, Weiwei Sun, and Dongsheng Li. Contiformer: Continuous time transformer for irregular time series modeling, Advances in Neural Information Processing Systems, 36, 2024.
[54] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference Learning Representations, 2023.
[55] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.
[56] Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv: 1607.06450, 2016.
arXiv preprint
[57] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[58] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self supervised learning from images with a joint-embeddee embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1.519-15629, 2023.
[59] Stuart Hunter. The exponentially weighted moving average. Journal of quality technology. 18(4):203-210, 1986.
[60] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung- Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation, 101 101 2000.
[61] Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark. Predicting in-hospital mortality of icu patients: The physionet/comppriig in cardiology challenge 2012. 20/2 Computing Cardiology, pages 245 248. IEEE, 2012.
[62] Matthew A Reyna, Christopher S Josef, Russell Jeter. Supreeth P Shashikumar, M Brandon Westover, Shamim Nemati, Gari D Clifford, and Ashish Sharma. Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019. Critical care medicine, 48(2):210-21 2020.
13

---

[63] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi. Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iti, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[64] Jesse Davis and Mark Goadrich. The relationship between precision-recall and curves. In Proceedings of 23rd international conference on Machine learning, pages 233-240. 2006.
[65] Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(1):96, 2019.
[66] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint vrXiv:1412.6988 2014.
14

---

A Additional Experimental Results
A.1 Details of Datasets
When preprocessing all the datasets, conduct z-score normalizations (subtract the mean and divide the deviation of the observed values) on the continuous features and one-hot encoding on the categorical features. We align the observation times to hours and pad the missing values with zeros. We use the first 48 hours of data for the Cardiology and the MIMIC-III mortality prediction dataset and the first 60 visits for the Sepsis dataset. We refer to the procedures from Harutyunyan et al. [65] when generating samples from the MIMIC-III dataset. We split the data into training, validation, and test sets with a ratio of 8:1:1. The Cardiology dataset can be obtained at https //physionet http:///ootee/ccalleeee------
The Sepsis dataset can be obtained at The MIMIC-III dataset can be obtained at https: httttts::pphysinneee org/ content wttt:mmmmiiiiiii http://nnn///miiiiii//........./ ci1i/144 Please follow the instructions on the PhysioNet website to access the data. We provide more details about the tasks on the MIMIC-III
https::/ppyysionet. org http///tteeentaalennneeeeeeeeeee 2t1.0aa00011n2221110000000.00
dataset as follows:
The goal of this task is to predict whether patient will die at the end of hospitalization using the first 48 hours of data. We extract 21,139 records each containing 17 physiological variables with 43.3% observed rate. There are 13.2% of patients who have positive labels. In this task, the model can only observe the first 48 hours, which is advantageous from a practical point of view, as the earlier the clinician identifies the risk, the more timely the intervention can be implemented.
In-hospital Mortality:
Phenoty ping: Phenotyping can be applied in cohort construction for clinical studies, comorbidity de- tection and risk adjustment, quality improvement and surveillance, diagnosis and treatment planning. There are 25 different phenotypes in the dataset. We extract 41,903 patients on this task with 41.8% observed rate.
The objective is to determine whether patient will decease in the next 24 hours based on the data within a 24-hour time window. This can be utilized as an early warning in ICUs. We extract 41,744 patients The observed rate is 43.2% and the positive rate is 3.5%.
Decompensation:
Length of Stay: The target of length-of ofyyth-ff-stay prediction is to predict the length of stay of a patient in the ICU. This task provides more Lin-grained view of the physiological states of the patients, assisting clinicians in monitoring the progress of the disease. We obtain 33,360 patients with 43.8% observed rate. The length of stay is discretized into 10 bins, where bins 1-8 correspond to 1-8 days of stay, respectively, bin for more than eight days but less than two weeks of stay, and bin 9 for living over two weeks.
A.2 Implementation Details
All experiments of this model are carried out on Linux server equipped with RTX 2080Ti GPUs. PyTorch 2.1.2 and CUDA 12.1 deep learning libraries are applied to build and train our neural network. While training models, we apply Adam optimizer [66] with learning rate 1e-3. We train all the models (including baselines) with the same optimizer, learning rate, and a total batch size of 256. The training epochs and hyperparameters (such as dropout and hidden dimensions) of baselines are tuned for better performance. To evaluate precisely, we repeat every experiment 3 times with different random seeds (1, 42, and 3407). The standard deviation of the experimental results appears to be slightly larger because the division of the training. validation, and test sets is related to the seed rather than being fixed. We cannot provide evaluations about imputations since we reconstruct in latent space and the absence of ground truth.
The hidden size d of all modules in SMART 32. The heads of the multi-head attention computation in temporal attention and variable attention are set to 4. The layer number L of the MART blocks is 2. The model is pre-trained for 25 epochs and fine-tuned for 25 epochs. The unfreeze epoch is set to 5. We employ the same hyperparameter configurations across the datasets in our experiments. The probability interval p for generating masks is set to (0, 0.75), and the dropout rate is set to 1. The exponential moving average (EMA) decay rate is set to 0.996.
#yperparameters:
15

---

Table 3: Performance comparison of AUROC and min(Se, P+) with standard deviation on the four clinical tasks. The best results are marked in bold. The second best results are underlined.
Cardiology
Sepsis
Model
AUROC
min(Se, P+)
AUROC
min(Se, P+)
AdaCare
18.01
41.57.13600
84.70.10.3
42.88
StageNet
82.55
49.91
91.25
56.21 1.308
ConCare
82.74.1447
46.651220
94.3910.74
69.79..307
GRASP
82.9810.6
49.23
94.43
69.7512.10
SAFARI
81.84
47.5442.32
94.644188
10.03
PPN
83,42+1.23
48.45+1151
94.25+1007
68. 68.96.42.25
RainDrop 76.86+1167
38.31
92.90+079
68.741242
Warpformer
83.21
48.73:3.22
93.86+0077
65.93
PrimeNet
84.19
49.67:2.58
83.60
18.2.92
SMART
85.80+10.5
52.63++137
95.92+0084
75.61+2276
In-hospital Mortality
Decompensation min(Se, P+)
Model
AUROC
min(Se, P+)
AUROC
AdaCare
18
42.68+00.1
87.06+1.23
49.96+1195
StageNet
85.32+0022
49.84+3.12
92.49+0.47
60.4440.93
ConCare
84.97
49.61
92.43+0046
61.54=1101
GRASP
84.7941.26
49.68+2.19
92.25+004
60.76+1154
SAFARI
84,48+1116
48.81
90.69+0.16
58.644173
PPN
84.90+0.96
50.42++1.12
91.50+0.70
60.65
RainDrop
83.21
46.30+1151
90,48+1160
56.55+224
Warpformer
84.1111.10
48.35.1202
93.24
64.42.10.97
PrimeNet
84.82.1105
50.96.11.1
91.15
54.9911266
SMART
85.61:1062
51.29
94.20
65.52
Table Ablation study of mask information in SMART on the Cardiology, Sepsis, and MIMIC-II in-hospital mortality.
Cardiology
Sepsis
In-hospital Mortality
SMART
AUPRC(S}
Fl Score((5)
AUPRC((5)
Fl Score((S)
AUPRC((5)
Fl Score( (5))
Full
23.8412:24
47.53
81.67
75.37
53.30
44.23
49.03=198
17.9821711
76.35
69.51=2.11
50.43=2.44
39.70:1144
in Encoder
49.74.1248
42.70.1..11
77.37
71.96.1226
52.29 1.06
42.65
w/n Mask in Temporal Attention
53.71
46 2.
81.08-2.97
75.62=281
53.29=1.55
42.83:3300
w/o Musk in Variable Attention
52.37=3.17
44,53
76.87:1220
72.06:2266
52.97-00.36
41.21:22.9
A.3 Experimental Results on More Metrics
In addition AUPRC and F1 Score, also provide the results of other metrics the binary classification task, such as the area under the receiver operating characteristic curve (AUROC), which widely used metric in the healthcare field. We also provide the results of the minimum of the precision and sensitivity (min(Se, P+)), which is often used in the healthcare field [16]. The results are shown in Table 3 The results illustrate that SMART outperforms existing baselines across all metrics on the four climical tasks. AUROC can be used to examine the model's ability to distinguish between positive and negative samples, while min(Se, P+) can be used to evaluate the model's 's ability to balance sensitivity and precision. By observing the results, we can find that the results on these Score. Specifically, AdaCare performs
metrics are consistent with the results on AUPRC the worst in the baseline, and StageNet, RainDrop. and PrimeNet show unstable performance on some datasets. In summary, SMART achieves the best performance on all metrics, demonstrating its effectiveness in learning representations and predicting patient health status.
16

---

Table 5: Study on the layer numbers of the MART block on the Cardiology, Sepsis, and in-hospital mortality. The best results are marked in bold.
SMART
Cardiology
Sepsis
In-hospital Mortality
AUPRC(%)
FI (c)
AUPRC( %6)
Fl Scorc(%)
AUPRC(%)
Fl Score (%)
52.66=22.7
44.07
81.26+2268
74,68++2.92
13.40=1.66
42.55+2.83
2
23.84
47.53++2.3
81.67
75.37=22.22
33.30=0.12
44.23+2.03
52.18
16.54:1188
81.35
74.30:12.74
155
43.28
50.56:13.00
16.65
79.98
73.68:12.02
51.89
44.97:12.2
Table 6: Study on the mask ratios of SMART in the pre-training stage on the Cardiology, Sepsis, and in-hospital mortality. The best results are marked in bold.
SMART
Cardiology
Sepsis
In-hospital Mortality
AUPRC(%)
Fl Score(%)
AUPRC(%)
FI Score(%)
AUPRC(%)
F1 Score (%)
(0, 0.25)
22.81
44.05
80.78.12.46
74.46:1295
52.55 10.39
14.19113.1
(0, 0.5)
53.77=2.45
46.92=0.61
80,89+2.23
74.04=2.75
53,41=0.78
44.90+3.56
0.75)
53,84=2.24
47.53+2.33
81.67=0084
75.37=2.62
53.30=0.12
44.23++203
(0.25, 0.75)
53.66=2.64
45.06+3.09
81.82++2.04
74.71+3.03
53,48=0.66
43.03-+3.00
(0.5, 0.75)
53.63=2.38
46.00+11.64
81.69++..0
74.87+2.29
53.37=1.00
42.45
A.4 Ablation Study on the Mask Information
We conduct an ablation study on the mask information in SMART. This ablation study can be regard extension of the reduced version w/o Mask in Table Sableeandd n it is deeper exploration of how the mask information affects the performance of SMART. Specifically, we remove the mask information the input encoder and the attentions the MART blocks separately, namely w/o Mask in Encoder, w/o Mask in Temporal Attention, and w/o Mask in Variable Attention. The results are shown in Table We observe that the performance of SMART drops significantly when the mask information is removed in the input encoder, which indicates that the mask information in the input encoder is critical learn complete representations. When the mask information is removed in the attentions, the performance of SMART also decreases, which demonstrates that the mask information in the attentions is essential resist noise when capturing the temporal and variable interactions. These results of the ablation study on the mask information further confirm each design in SMART is indispensable and contributes the improvement.
A.5 Study on the Hyperparameters
To explore the affect of the hyperparameters the performance of SMART, conduct study the layer numbers of the MART block and the mask ratios in the pre-training stage. The results are shown in Table and Table 6 On the layer numbers of the MART block, the results show that the performance of SMART is the best when the layer numbers are set to 2. Besides, we find that too much layer numbers may lead decreased performance.
When it comes to the mask ratios in the pre-traininn gre-training stage, when the mask ratios are set to (0, 0.75), SMART achieves the best performance the Cardiology and Sepsis, whereas the mask ratio (0.25, 0.75) obtains the best AUPRC and (0, 0.5) achieves the best F1 Score on the in-hospital
Table 7: Performance comparison with standard deviation on Cardiology, Sepsis, and in in-hospital mortality with basic baselines.
Cardiology
Sepsis
In-hospital Mortality AUPRC(%)
Model
AUPRC(%)
Fl Score(%)
AUPRC(S)
Score(%)
Score (%)
GRU
47.1013.56
37.84
49.73
47.66.1211
50.68.12.07
41.70.1..11
Transformer
42,92+3.70
42.82+3.73
22.56=4441
24.75+33.00
43,0743.68
42.9443.36
SMART
53.84
47.53
81.67
75.37
53.30.10.12
44.23
17

---

mortality. These experiments shed light on the importance of hyperparameters in the performance of SMART, and provide guidance for the selection of hyperparameters practice.
A.6 Comparisons with Basic Baselines
We compare SMART with basic baselines on the Cardiology, Sepsis, and in-hospital mortality tasks, including gated recurrent units (GRU) and Transformer [22] The results shown in Table 7 We observe that SMART outperforms the basic baselines on all datasets, which shows the effectiveness of SMART in learning representations and predicting patient health status. The results also show that the basic baselines have poor performance especially on the Sepsis datasets, which may be due to the complexity and higher missing rate of the dataset.
B Broader Impact
The proposed model is designed to improve the representation learning capability of EHR data in ICU scenarios. The model can used to predict patients' health status, such as in-hospital mortality, sepsis, and other diseases. The model can be used to assist physicians in making clinical decisions, such as early warning of patients' health status, which may help reduce the mortality rate of patients. The model can also be used to assist in the allocation of medical resources, such the allocation of ICU beds, which may help reduce the burden on the healthcare system. However, the model may also have some negative impacts. For example, the model's predictions may be biased, which may result in patients being treated unfairly. Additionally, there may be ethical issues with this model, as it may lead to decisions that are not in the best interest of the patient. Therefore, is important carefully evaluate the model before actually using it. It should be noted that the model cannot replace doctors in making decisions. The model only a tool to assist doctors in making decisions, The model should be used in conjunction with doctors to make the best decisions for patients.
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We focus analyzing EHR data from ICU and propose novel model using missing-aware and self-supervised pre-training to enhance representation learning capability.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the proposed model in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed information on hyperparameters in Appendix A.2 Besides, we provide the code for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
We provide the code link in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee 7/nips.com htc/://bbb//gudddd///eeeeeueisss ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide detailed information on hyperparameters Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We repeat each experiment with three random seeds and report both the mean and standard deviation of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute in Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:::://oorrps.ccppbbbbibb wwtpcc/ppppcccceeeeeeeeeceeeeece htt.ai///hhhcccidiennnees.e
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics. We conducted our study using publicly available EHR data, which had passed ethical review.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential positive societal impacts and negative societal impacts of the proposed model in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pre-trained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper does not release data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original papers that produced the datasets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release code and provide instructions how to reproduce the results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "9FdgFBIFis", "reviewer": "Reviewer_PYgK", "summary": "The paper presents a strategy to account for missing data in EHR called SMART. This is broken down in 2 stages: pretraining and fine tuning. Pretraining learns a hidden state representation which is done by randomly making the input and predicting the label, while fine tuning uses this hidden state representation and is tuned for specific down stream tasks. The authors demonstrate the efficacy on multiple datasets and on high impact areas: cardiology, sepsis and in-hospital mortality. The methodology is also quite light weight as demonstrated in Fig3 and the authors also capture some of the limitations of the modified attention mechanism.", "questions": "Can the authors show: projecting the hidden state representation back to the data level what the model is learning by using the masks? To present us an idea of what the model deems important to learn when the data is missing", "limitations": "Yes they have addressed it.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Enough experiments to convince of initial efficacy \nNew attention mechanism is quite light weight\nLatent representations are being learnt effectively to handle missing data", "weaknesses": "Can show more areas of impact by looking at conditions where the missing data can cause more problems to look at the limit of SMART \nQuadratic attention mechanism is also a problem (although approximations have shown to reduce that cost) \nCan the authors show: projecting the hidden state representation back to the data level what the model is learning by using the masks? To present us an idea of what the model deems important to learn when the data is missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the strengths of SMART, especially its effectiveness and novel designs. We address your concerns and answer your questions below.\n\n---\n\n**W1: Can you discuss the limitations of SMART by showing more areas of impact by looking at conditions where missing data could cause more problems?**\n\nThank you for your question about the limitations of SMART. We have discussed the limitations in clinical scenarios in the *Conclusions and Limitations* section and the *Broader Impact* in the Appendix. Missing vital signs or lab results can lead to incorrect prediction or delayed diagnoses. SMART can be aware of the missingness in the time series and enhance its prediction performance, avoiding potential delayed diagnoses due to lack of observations. However, we recognize the necessity for more precise methods in future research.\n\nFor non-clinical fields, such as finance, electricity, meteorology, etc., where missingness are common in time series, SMART has the potential to play a role in these fields. However, for ECG/EEG or other high-frequency data, the quadratic computational complexity may limit the usability of SMART.\n\n**W2: The quadratic attention mechanism is also a problem (although approximations have been shown to reduce that cost).**\n\nThank you for your concern about the computational complexity. Our contribution lies in proposing a missing-aware EHR model to better accomplish clinical prediction tasks. Although the quadratic computational complexity may limit the use of SMART on very long time series, it is applicable to EHR data, given the limited length of time series.  In addition, there are some methods compatible with SMART, such as model compression or linear attention [1,2], which can reduce overhead and improve scalability, but this is beyond the scope of this paper and can be further explored in future work.\n\n[1] Linformer: Self-attention with linear complexity. arXiv 2020.\n\n[2] RWKV: Reinventing RNNs for the Transformer Era. EMNLP 2023.\n\n**W3&Q1: Can you project the hidden state representation back to the data level and show what the model is learning by using the masks?**\n\nThank you for your question about how the model imputes the missingness. We focus on reconstructing missing data in the latent space, so the imputed data cannot be mapped back to the input space. In particular, because our goal is to improve the accuracy of clinical tasks when designing the method, we encourage the model to learn as much information as possible about the entire sequence in the pre-training stage and learn task-related embeddings in fine-tuning, rather than simply imputing.\n\nFor better understanding, we give a detailed explanation and we will add the explanation in our future submission. The observed value in the input space can be viewed as a composed signal. The learned encoder can be viewed as a signal filter that decomposes the observed composed signal over a learned \"dictionary\". The \"dictionary\" is the affine transformation(s) shared by all the time series that transforms the input-composed signal to learned decomposed embeddings. The learned embeddings can be regarded as the decomposition of the \"dictionary\". Our reconstruction in the latent space is essentially a reconstruction of the decomposed filtered signals of different entries. On the one hand, this reduces the noise in the original input-composed signal. On the other hand, this pursues the consensus of different time series that converge to the underlying expectation. Thus, reconstruction in the latent space achieves better performance than reconstruction in the input space.\n\nWe hope these explanations adequately address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WYh0PWFKGk", "reviewer": "Reviewer_MRHA", "summary": "The authors propose a novel approach to handling missing data in an attention-based module in a method that is geared for predicting downstream health-related outcomes given multivariate time series patient data in EHR settings. Specifically, the proposed module, termed as the MART block, biases the attention across the temporal dimension using a heuristic which favors time steps with observations. In quantitative benchmarks involving six different disease-like outcomes, the method obtains the highest accuracy among several other methods that are also designed to learn patient representations given time series data in an EHR setting.", "questions": "Is the CLS token involved in the pre-training stage at all similarly BERT? If so, what does it predict in the pre-training stage?\n\nI just want to confirm that it would be true for a reader to interpret that even when the model is ablated in any which way (no mask, no temporal or variable attention, no cls), it would still be a top 1 or 2 ranking method in the benchmarks? Was there any ablation which degraded the model beyond this ballpark?", "limitations": "The work discusses some limitations in the final section. I did not assess that this work could have a potentially negative societal impact.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written. The functionality of the proposed module was described in a straightforward way. I believe that the MIMC pipeline adhered to is quite standard and if the quantitative results are reliable then the amount of improvement attained by the approach deserves recognition.", "weaknesses": "I felt there was not enough motivation in the introduction in terms of why handling imputation in the latent space is expected to be a better approach than prior approaches, other than that it is possibly a novel approach. Is the intuition supposed to be similar to how latent diffusion works well? Some background or citations would further in convince the reader.\n\nDoes the work have any considerations for MAR, MCAR, MNAR, and structured missingness cases (all of which are present in EHR data); are MART blocks supposed to work well for all such types of missing data patterns? Due to some of these questions, I was not convinced upon a first read that biasing pairwise missing value statuses in a monotonically increasing scale (Eq 1) should intuitively be beneficial; although the quantitative results speak for themselves.\n\nWere there any results on measuring how well the imputation methods impute missing observations in the actual time series data? eg. This could be done through a simulation. It seemed that all benchmarks were based on downstream tasks. I assume the proposed method also does well in this case, or there was an assessment that it does not matter in the perspective of improving downstream performance. It would be nice to understand if the contribution of the method is mainly in improving prediction of downstream tasks, or if it also imputes the data well.\n\nThe quantitative result obtained by SMART was encouraging, but I did not feel that the benchmark fully encompassed the breadth of methods which are available. Firstly, it seemed lacking in terms of basic baselines using logistic regression and gradient boosting methods to provide a good intuition on how easy or difficult the downstream tasks are; I believe these are important for studies that employ their own benchmarking pipeline on MIMIC. \n\nI also personally wished to see more baselines which are commonly explored in the imputation literature (or at least mention them at all). To list some, there is softImpute, MissForest, MICE, and a Joint low-rank model from Sportisse et al. (2020).\nAmong more recent works, there is HI-VAE, GP-VAE, and notMIWAE. It is true that many of these works do not handle time series data or may not have been designed with EHR in mind, but there are straightforward ways to process the data such that they can be applied.\n\nFurthermore, there are a few recent works that also use an attention model and claimed sota at the time of their release in terms of time series imputation. Some words on how they are related or why or why not they are a good fit for these tasks would add to the comprehensiveness of the work.\n\n[1] Zhang et al 2023 “Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modeling“\n[2] SAITS from Du et al 2023 “SAITS: Self-Attention-based Imputation for Time Series”\n[3] GRIN from Cini et al 2022 “Filling the G_AP_S: Multivariate Time Series Imputation by Graph Neural Networks”\n[4] Marisca et al 2022 “Learning to Reconstruct Missing Data from Spatiotemporal Graphs with Sparse Observations”\n\nSince the representation learning part of the approach is emphasized in a few parts of the work, I was surprised not to see any visualizations of the embeddings learned (per patient or per variable), and how if any it differs significantly from those obtained from prior works.\n\nA minor point, but for first time readers it would be nice to have citations in lines 257 to 259 for SAFARI, PPN, and GRASP despite they might have been mentioned & cited elsewhere.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Why handling imputation in the latent space is expected to be a better approach than prior approaches?**\n\nThank you for your insightful questions about why our approach works. We have shown in lines 59-60 that reconstruction in the latent space can help it better learn higher-order data patterns instead of focusing on trivial details. For more explanation, please refer to our global response.\n\n**W2: Are MART blocks supposed to work well for all types of missing patterns? Why biasing in a monotonically increasing scale in Eq 1 is beneficial?**\n\nMissingness in EHR is usually considered to be MNAR [5]. Whether to check an indicator is determined by physicians, and the reason is not reflected in the data. Whether a particular indicator is assessed is informative, as the absence could suggest that the condition with it is not severe. The performance gains with mask also verify that MART is well suited for MNAR data. However, since it is impossible to model the cause of missingness, we randomly sample missing data in pre-training.\n\nFor the question in Eq 1, we intended to endow the perception of the degree of missingness (both, only one, or none). It did not matter whether it was manually set or learnable since the weight matrices in the attention will adapt to them. We also conducted a simple experiment by replacing 1 and 2 in Eq 1 with two learnable parameters (w/ learnable bias), as shown in the rebuttal PDF.\n\nThe experimental results show that the results of the manual value are better, so we finally used the manual setting of the monotonically increasing values.\n\n[5] A Bayesian latent class approach for EHR-based phenotyping. Statistics in Medicine 2019.\n\n**W3: Were there any results on measuring how the imputation methods impute missingness?**\n\nSince the goal of pre-training is to improve performance on clinical tasks, SMART is not designed to provide an imputed series. For further exploration, we added a decoder at the same level as the embedding decoder in the pre-training, which is used to impute in the input space, and evaluated the performance after fine-tuning (w/ both imputation), as shown in the rebuttal PDF.\n\nWe found that with this decoder, the performance declined and was even worse than w/o Pre-training, which indicates that there may be a trade-off between imputation in the latent and input space, and the pursuit of accurate imputation may lead to poor performance in downstream tasks. (We did not add special designs on this ablation, so the conclusion may be inaccurate.)\n\n**W4: It seemed lacking in terms of basic baselines.**\n\nWe have shown the results of the basic methods GRU and Transformer in Appendix A.6, and they are worse than most of the baselines, showing the difficulty of tasks. Since LR and tree models (XGBoost, etc.) are not recursive, they do not apply to our datasets composed of variable-length series.\n\n**W5: I wished to see more baselines commonly explored in the imputation literature.**\n\nThank you for providing a lot of literature. We are delighted to read them and will discuss them to enrich our paper. However, most methods, including [3,4], are not designed for completing clinical tasks. There are also some imputation models combined with tasks, such as [1,2] you mentioned, but [1] does not essentially use any supervision related to imputation, similar to Warpformer[6]. \n[2] does not apply to time series with varying lengths, so it cannot be evaluated on our dataset. In addition, although it is not designed to complete downstream tasks, their authors compare the results of these tasks by using GRU to model the imputed series. Nevertheless, training [2] for imputation will remove some of the sampled data as imputation supervision, which potentially reduces the performance on downstream tasks, as we mentioned in lines 107-109.\n\n[6] Warpformer: A multi-scale modeling approach for irregular clinical time series. KDD 2023.\n\n**W6: Were there any visualizations of the embeddings learned?**\n\nThanks for your reminder. We are sorry that we overlooked it. We have uploaded the embedding visualization comparison by t-SNE of the patients in the test set from the Cardiology dataset in the rebuttal PDF. We can find that the embedding learned by SMART is more discriminative, which qualitatively verifies its effectiveness.\n\n**W7: It would be nice to have more citations of baselines.**\n\nThanks for your suggestion. We will add them in future versions, which will improve the quality and readability.\n\n**Q1: Is the CLS vector in the pre-training similar to BERT? What does it predict?**\n\nThe CLS vector is somewhat similar to [CLS] in BERT, but they are not the same. The mention of BERT is just for ease of understanding. In the Next-Sentence-Prediction task of BERT, [CLS] is used for prediction to explicitly encourage it to express the information of the entire sequence, while there is no such supervision in the pre-training of SMART. In the pre-training, the encoding of the CLS vector position is not calculated in the loss (because its $\\hat{m}$ is False), because our original intention is to calculate the hidden representation of the removed sample points, and including it in the loss may bring slight performance changes.\n\nBut similar to [CLS], the CLS vector does not represent any time step but learns the overall representation, which is a bridge between the pre-training and fine-tuning, as mentioned in line 308. It provides a location to store the overall information (because its mask is always True), and using it as a query in Variable Attention encourages this.\n\n**Q2: Would SMART be a top-ranking method without novel designs?**\n\nThank you for your question about the performance. However, this idea may be incorrect. Without all the unique designs, the model will degenerate into a Transformer. As shown in Appendix A.6, it is significantly weaker than most baselines, and the gains brought by pre-training obviously cannot make up for such a large gap.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o3u2nzI352", "reviewer": "Reviewer_Hfjo", "summary": "This paper presents SMART, a novel model designed to tackle the challenges of missing and irregular data in electronic health records (EHRs). Utilizing a two-stage training strategy, SMART first pre-trains to handle missing data in the latent space and then fine-tunes for specific clinical tasks. The model's innovative masked attention recurrent transformer (MART) block captures temporal and variable interactions, significantly improving prediction accuracy across various clinical tasks. Evaluated on three EHR datasets, SMART outperformed existing models, demonstrating robust performance and versatility. Despite its strengths, including handling missing data and achieving superior prediction accuracy, the model's complexity and limited dataset variety highlight areas for further exploration and improvement.", "questions": "I'd like for the authors to respond to below questions if they can: -\n(1) Could you elaborate on why you chose to reconstruct latent representations during pre-training rather than imputing missing values in the input space?\n(2) Which component of SMART do you believe contributes the most to its superior performance, and why?\n(3) Given the quadratic complexity of temporal attention in SMART, how scalable is the model when applied to EHR datasets with varying lengths of patient records? Have you explored strategies to mitigate computational costs without compromising performance?\n(4) The ablation study provides insights into the importance of different components within SMART. What specific findings surprised you the most during these experiments, and how did they influence the design or interpretation of the final model?\n(5) Could you discuss any challenges or limitations encountered during the implementation of SMART in real-world clinical settings?", "limitations": "There is a notable focus on performance across various datasets, but the generalizability of the model to different healthcare systems or diverse patient populations remains unclear. It would be beneficial for the authors to discuss potential ethical concerns or unintended consequences their approach might introduce, such as biases in predictions or challenges in interpretability that could affect clinical decision-making.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "One of the notable strengths of this paper is its originality in addressing the pervasive issue of missing and irregular data in electronic health records (EHRs). By introducing the SMART model with its innovative masked attention recurrent transformer (MART) block, the authors present a novel approach that captures temporal and variable interactions more effectively than traditional methods. The paper excels in explaining the complex mechanisms of the SMART model and its components, making it accessible even to those not deeply familiar with the intricacies of machine learning models.", "weaknesses": "(1) the lack of comparative analysis with state-of-the-art models beyond the specific baseline models mentioned. While SMART outperformed these baselines, a broader comparison with the latest advancements in EHR prediction models would provide a clearer benchmark of its superiority. This omission could leave readers questioning how SMART fares against the most cutting-edge approaches in the field.\n(2) the paper could benefit from a more in-depth discussion on the computational efficiency of the SMART model, particularly regarding inference time and resource requirements. Given the increasing emphasis on real-time decision support in clinical settings, understanding the model's computational demands is crucial for practical implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the strengths of SMART, especially its effectiveness and novel designs. We address your concerns and answer your questions below.\n\n---\n\n**W1: Lack of comparative analysis with state-of-the-art models beyond the specific baseline models mentioned.**\n\nThank you for your question about the latest baseline methods. We would like to clarify that we have used the latest methods, including SAFARI and RainDrop in 2022, Warpformer and Primenet in 2023, and PPN in 2024. If we missed any latest methods, please remind us and we will include them in our comparison. \n\n**W2: The paper could benefit from a more in-depth discussion on the computational efficiency of the SMART model, particularly regarding inference time and resource requirements.**\n\nThank you for your concern about the computational efficiency. We would like to clarify that we have compared the computational efficiency of the models in 4.3, and SMART has achieved the best balance among all methods in the comparison of training time performance. As for inference time, SMART can complete the inference of patients in a very short time (within 1 second), which can meet the needs of clinical scenarios.\n\n**Q1: Could you elaborate on why you chose to reconstruct latent representations during pre-training?**\n\nThank you for your question about why we reconstruct in the input space. As we explained in lines 59-60 in the introduction, imputation in the input space may cause the model to get stuck in unnecessary details instead of learning more high-level semantic features that are beneficial to downstream tasks. More specifically, the imputation models pursue more accurate interpolation at each missing sample point, while this work pursues performance on clinical tasks, that is, completing tasks based on the overall information of the sequence. The optimization directions of these two goals are different. Therefore, we did not impute missingness in the input space during pre-training. For more explanation, please refer to our global response.\n\n**Q2: Which component do you believe contributes the most to its superior performance, and why?**\n\nThank you for your curiosity about the effectiveness of SMART. We conducted ablation experiments on each design of the model in Section 4.2.2, and the results show that the information from the missing mask is the most important. Without it, the temporal and spatial attention we proposed cannot work properly, and the model cannot perceive the missing data. In addition, as described in lines 287-310, the ablation experiment verifies that each innovative design (including pre-training, temporal attention, variable attention, and CLS Vector) is very important and can bring performance improvement.\n\n**Q3: Given the quadratic complexity of temporal attention in SMART, how scalable is the model when applied to EHR datasets with varying lengths of patient records? Have you explored strategies to mitigate computational costs without compromising performance?**\n\nThank you for your question about scalability. The length of patients in our dataset is variable, and SMART is scalable for them. However, due to the lack of corresponding datasets, we did not conduct experiments on longer series. The focus of our work is to combine missing awareness to improve performance on clinical predictions, so we have not explored strategies to reduce computational costs without affecting performance. There are some existing methods, such as Linformer [1], RWKV [2], etc., which reduce the computational complexity of the attention mechanism. They can be utilized to accelerate this method, which can be a future work.\n\n[1] Linformer: Self-attention with linear complexity. arXiv 2020.\n\n[2] RWKV: Reinventing RNNs for the Transformer Era. EMNLP 2023.\n\n**Q4: What specific findings surprised you the most during ablation studies, and how did they influence the design or interpretation of the final model?**\n\nThank you for your question about the ablation experiment. In the ablation experiment, we found that compared with the method of imputing in the input space, the improvement of SMART is obvious, so we are very glad to see that the proposed method is verified in the experiment. In addition, the effectiveness of the CLS vector also surprised us since it brought a very large improvement. The introduction of the CLS vector also provides direction and insights for the future development of models for time series.\n\n**Q5: Could you discuss any challenges or limitations during the implementation in real-world clinical settings?**\n\nThank you for your interest in the application of SMART in real clinical scenarios. We have mentioned the challenges of SMART in real-world application scenarios in Appendix B *Broader Impact*, including the possibility of making unfair predictions for patients and potential ethical issues. In particular, we would like to emphasize that the model is only a tool to assist physicians in making decisions. The model should be used together with physicians to make the best decision for patients.\n\n**Limitations: The generalizability of the model to different healthcare systems or diverse patient populations remains unclear. It would be beneficial for the authors to discuss potential ethical concerns or unintended consequences their approach might introduce, such as biases in predictions or challenges in interpretability that could affect clinical decision-making.**\n\nThank you for your concern about ethical issues. We have mentioned the possible ethical issues of SMART in *Broader Impact*. We acknowledge that SMART may make unfair predictions for patients, leading to potential ethical issues. However, the model cannot make any decisions on behalf of physicians. The model is only a tool to help physicians understand the patient's condition. How to improve the fairness of the model is one of the possible research directions in the future.\n\nWe hope these explanations adequately address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fjOMdYQEt5", "reviewer": "Reviewer_CSeo", "summary": "The paper presents SMART, a self-supervised representation learning approach that tries to tackle the problem of missingness in EHR data. It proposes a novel self-supervised pre-training approach which is able to reconstruct missing data representations in the input space and makes use of both temporal and variable attention mechanisms to achieve that. The pre-trained encoder can be further fine-tuned with a label-specific decoder for different downstream classification tasks. Through multiple datasets, comparisons with baselines, and comprehensive ablation studies, the authors show the effectiveness of their method at generalization and robustness to missing data.", "questions": "1. Are there any evaluations done just for the pre-training task? How good is the model at reconstruction and imputing missing values? \n2. Were there any constraints placed on the imputed values during training? It might happen that the model imputes missing variables with unrealistic values.\n3. Can this model be extended to incorporate multiple modalities like images or text (clinical notes), perhaps by learning separate missing-aware encoders for each of them? Something like this is done in [1].  \n4. How does this model compare against a fully-supervised method where all of the data is available? \n\n[1] Wu, Zhenbang, et al. \"Multimodal patient representation learning with missing modalities and labels.\" The Twelfth International Conference on Learning Representations. 2024.", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Missingness is an important issue in medical domain, especially in EHR data, so having a ML model that is missing-aware and can still create meaningful representations is impactful \n2. Even though the two-stage training process may not be new, the creation of the MART blocks and the pre-training paradigm seems novel\n3. The proposed method is able to comprehensively beat the previous baselines across all datasets that were tested, both in performance and in training times\n4. Multiple ablations showcase the effectiveness of different components of the model architecture \n5. The paper is well-written and code is provided", "weaknesses": "1. During the pre-training stage, since it is based on reconstruction, access to the full dataset with all observations is required. The method would not work if the training data had missing values as well. \n2. During fine-tuning, only the label decoder is updated during the first few epochs of training but it is unclear as to why this is needed. It is mentioned that the pre-trained parameters are reserved, but a more quantitative explanation or an ablation would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the strengths of SMART, especially its novelty and effectiveness. We address your concerns and answer your questions below.\n\n---\n\n**W1: During the pre-training stage, since it is based on reconstruction, access to the full dataset with all observations is required. The method would not work if the training data had missing values as well.**\n\nThank you for your concern about data integrity. It appears there may be a misunderstanding regarding the task at hand.  We have mentioned in lines 122-128 that the EHR data we use contains a lot of missingness. The goal of this work is to endow the model with the ability to perceive missing data. In the pre-training stage, we sample missingness based on probability on the basis of existing missing data as our learning target, rather than on fully observed data. The missing-aware method we proposed effectively enhances the model's ability to learn missing data and improves the predictive performance in clinical tasks.\n\n**W2: During fine-tuning, only the label decoder is updated during the first few epochs of training but it is unclear as to why this is needed. It is mentioned that the pre-trained parameters are reserved, but a more quantitative explanation or an ablation would help.**\n\nThank you for your interest in the fine-tuning experiment settings. We hope that the model can retain the pre-trained parameters in the first several epochs instead of updating all parameters, so that the initial optimization goal of the model is to improve the classifier's adaptation to the embedding. Assigning different learning rates or schedulers to various parameters is a common strategy in the fine-tuning process. To further resolve your doubts, we provide the experimental results of not freezing parameters in fine-tuning as follows.\n\n|Model|Cardiology AUPRC(\\%)|F1(\\%)|Sepsis AUPRC(\\%)|F1(\\%)|In-hospital Mortality AUPRC(\\%)|F1(\\%)|\n| - | - | - | - | - | - | - | \n|w/o freeze|51.46$\\pm$2.38|46.89$\\pm$3.19|79.81$\\pm$3.15|74.29$\\pm$3.11|50.81$\\pm$1.47|42.85$\\pm$2.63|\n|SMART|53.84$\\pm$2.24|47.53$\\pm$2.33|81.67$\\pm$0.84|75.37$\\pm$2.62| 53.30$\\pm$0.12|44.23$\\pm$2.03|\n\nThe ablation results show that if the parameters are not frozen, the model performance will degrade, which may be caused by the large initial learning rate, resulting in the pre-trained parameters not being fully retained.\n\n**Q1: Are there any evaluations done just for the pre-training task? How good is the model at reconstruction and imputing missing values?**\n\nThank you for your interest in pre-training evaluation. Our model is not specifically designed to solve the imputation problem, but to improve performance on clinical prediction tasks. Thus, it can only complete the reconstruction in the latent space. \n\nFor better understanding, we give a detailed explanation and we will add the explanation in our future submission. The observed value in the input space can be viewed as a composed signal. The learned encoder can be viewed as a signal filter that decomposes the observed composed signal over a learned \"dictionary\". The \"dictionary\" is the affine transformation(s) shared by all the time series that transforms the input-composed signal to learned decomposed embeddings. The learned embeddings can be regarded as the decomposition of the \"dictionary\". Our reconstruction in the latent space is essentially a reconstruction of the decomposed filtered signals of different entries. On the one hand, this reduces the noise in the original input-composed signal. On the other hand, this pursues the consensus of different time series that converge to the underlying expectation. Thus, reconstruction in the latent space achieves better performance than reconstruction in the input space. When it comes to evaluation, although the loss can be calculated, this loss may be meaningless and cannot be compared with existing imputation methods. \n\n**Q2: Were there any constraints placed on the imputed values during training? It might happen that the model imputes missing variables with unrealistic values.**\n\nThank you for your concern about the imputed values. Because our learning rate is relatively small and the pre-training process is data-driven, there will be no imputation outside the data distribution. Thus, no explicit constraints are required. Although the model may provide incorrect imputed representations, they still reflect some objective reality learned by the model. On the other hand, since our model reconstructs in the missing in the latent space, we cannot really confirm whether it will provide unrealistic imputed values.\n\n**Q3: Can this model be extended to incorporate multiple modalities like images or text (clinical notes)?**\n\nThank you for your curiosity about the potential of the model to handle multimodal data. The work you provided is a very good reference for handling multimodal data, and we will cite it in future versions. The method we proposed has the potential to be extended to more modalities. It can support other modalities by masking part of the image or text and restoring them in pre-training, so that the model can better learn high-order representations and complete prediction tasks. However, since multimodal data is more complex, multi-task learning brought by pre-training may be one of the challenges, which can be used as a future exploration direction.\n\n**Q4: How does this model compare against a fully-supervised method where all of the data is available?**\n\nThank you for your curiosity about the results of full data training. The fine-tuning phase of our method and all baseline methods are fully supervised and trained on the full data, although they contain a large amount of missing data. The results show that SMART is significantly ahead of these methods, verifying the effectiveness of our work on the full data.\n\nWe hope these explanations adequately address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their high-quality comments and for recognizing the strengths of SMART. We have addressed all the concerns and answered questions in the rebuttal.

Here, for the commonly asked question of why reconstruction in latent space is more effective than imputation in input space, we give a detailed explanation and we will add the explanation in our future submission. The observed value in the input space can be viewed as a composed signal. The learned encoder can be viewed as a signal filter that decomposes the observed composed signal over a learned "dictionary". The "dictionary" is the affine transformation(s) shared by all the time series that transforms the input-composed signal to learned decomposed embeddings. The learned embeddings can be regarded as the decomposition of the "dictionary". Our reconstruction in the latent space is essentially a reconstruction of the decomposed filtered signals of different entries. On the one hand, this reduces the noise in the original input-composed signal. On the other hand, this pursues the consensus of different time series that converge to the underlying expectation. Thus, reconstruction in the latent space achieves better performance than reconstruction in the input space.


---

In the rebuttal PDF, we uploaded more ablation experiments and the embedding visualization comparison by t-SNE of SMART and baseline methods on the Cardiology dataset mentioned by Reviewer MRHA. By observation, we can find that the embedding learned by SMART is more discriminative, which qualitatively verifies the effectiveness of our method.

Meta-review: The paper presents SMART, a novel approach for handling missing data in electronic health records (EHR) using an attention-based module. The method aims to improve predictions for downstream health-related outcomes given multivariate time series patient data.

The reviewers generally agreed on the paper's strengths, noting its clear presentation, novel approach to handling missing data in the latent space, and impressive performance on multiple datasets and high-impact clinical areas. The lightweight nature of the modified attention mechanism was also appreciated.

However, some concerns were raised. Reviewer MRHA questioned the motivation for handling imputation in the latent space and requested more background on why this approach is expected to be superior. The authors addressed this in their rebuttal, explaining that reconstruction in the latent space helps learn higher-order data patterns and reduces noise in the original input signal.

Another concern was about the method's effectiveness for different types of missing data patterns (MAR, MCAR, MNAR). The authors clarified that missingness in EHR is usually considered MNAR and provided evidence that SMART is well-suited for this type of data.

Reviewer MRHA also requested more comprehensive baselines, including basic methods like logistic regression and gradient boosting, as well as more recent imputation techniques. The authors explained that some basic methods were included in the appendix and that certain suggested baselines were not directly applicable to their variable-length time series data.

Both reviewers raised questions about the imputation quality in the input space. The authors conducted additional experiments showing that pursuing accurate imputation in the input space may lead to poor performance in downstream tasks, supporting their focus on latent space reconstruction.

Reviewer PYgK suggested exploring more areas of impact and discussing limitations in scenarios where missing data could cause more problems. The authors acknowledged this and provided some discussion on potential applications and limitations in non-clinical fields.

After considering the reviews, rebuttal, and subsequent discussions, I believe the paper presents a valuable contribution to the field of EHR data analysis and clinical prediction tasks. The authors have adequately addressed most of the reviewers' concerns and demonstrated the effectiveness of their approach.

As an Area Chair, my recommendation is to accept this paper for publication at NeurIPS. The novel approach, strong empirical results, and potential impact in clinical applications make it a worthy addition to the conference. However, I encourage the authors to incorporate the additional explanations and discussions from their rebuttal into the final version of the paper to enhance its clarity and comprehensiveness.
Decision: Accept (poster)