Effective Self-Supervised Transformers For Sparse Time Series Data
Authors: Alex Labach
Aslesha Pokhrel
Seung Eun Yi
Saba Zuberi
Maksims Volkovs
Rahul G Krishnan
Venue: Submitted to ICLR 2023 (2023)
Keywords: Representation learning
Transformers
Sparse Time Series

Abstract: Electronic health records (EHRs) typically contain a wide range of time series data that is characterized by high sparsity and irregular observations. Self-supervised Transformer architectures have shown outstanding performance in a variety of structured tasks in natural language processing and computer vision. However, their use in modelling sparse irregular time series with tabular data has not been widely explored. One of the major challenges is the quadratic scaling of self-attention layers that can significantly limit the input sequence length. In this work, we introduce TESS, Transformers for EHR data with Self Supervised learning, a self-supervised Transformer-based architecture designed to extract robust representations from EHR data. We propose an input binning scheme that aggregates the time series inputs and sparsity information into a regular sequence with fixed length, enabling the training of larger and deeper Transformers. We demonstrate that significant compression of EHR input data is possible without sacrificing useful information, likely due to the highly correlated nature of observations in small time bins. We then introduce self-supervised prediction tasks that provide rich and informative signals for model pre-training. TESS outperforms state-of-the-art deep learning models on multiple downstream tasks from the MIMIC-IV and PhysioNet-2012 EHR datasets.

Under review conference paper at ICLR 2023
EFFECTIVE SELF-SUPERVISED SPARSE TIME SERIES DATA
TRANSFORMERS
FOR
Anonymous authors Paper under double-bling review
ABSTRACT
Electronic health records (EHRs) recorded in hospital settings such intensive units (ICUs) typically contain a wide range of numeric time series data that is characterized by high sparsity and irregular observations. Self-super Transformer architectures have shown outstanding performance in a variety of structured tasks in natural language pro- cessing and computer vision. However, the sparse irregular time series nature of ICU EHR data poses challenges for the application of transformers that have not been widely explored. One of the major challenges is the quadratic scaling of self-attention layers that can significantly limit the input sequence length. In this work, we introduce TESS, Transformers for EHR data with Self Supervised learning, a self-supervised Transformer- based architecture designed to extract robust representations from EHR data. We propose the application of input binning to aggregate the time series inputs and sparsity informa- tion into regular sequence with fixed length, enabling the training of larger and deeper Transformers. We demonstrate that significant compression of ICU EHR data is possible without sacrificing useful information, likely due to the highly correlated nature of obser- vations in small time bins. We then introduce self-supervised prediction tasks that provide rich and informative signals for model pre-training. TESS outperforms -tate-of-the-art deep learning models on multiple downstream tasks from the MIMIC-IV and PhysioNet- 2012 ICU EHR datasets.
INTRODUCTION
Electronic health record (EHR) data collected in the hospital contains immense amount of informa- tion about patients. This data typically comes in the form of vital sign measurements, lab results, and diagnoses/rreamments. Patients in an Intensive Care Unit (ICU) are particularly heavily monitored, with frequent vital sign observations and diagnostic tests. The resulting multivariate numeric time series is high- dimensional, sparse, and irregularly distributed across time, making it challenging to apply standard time series analysis methods that are primarily designed for densely sampled data. These challenges are not unique to health care, and data with such characteristics commonly arises in fields such as finance, banking, and e-commerce (Cao et al. 2021 Gosez-Losada & Duch- Duch-Brown. 2019,/Zang et al. (015).
Good models of clinical outcomes need extract predictive signal from the values, frequencies and missing- ness patterns from such data. Hand-crafting such features is non-trivial and time-consuming task, which has led to the exploration of deep learning for problems arising healthcare. However, when the labels are noisy and scarce, such methods too susceptible to overfitting. Self-supervised learning (SSL) (Chopra et al.] 20055] Caron et al.) 20. has risen in popularity as a tool to reduce the dependence of representation learning on large amounts of labelled data. SSL relies the premise that domain experts have prior knowl- edge about the patterns in high- high-dimensional data; by translating this domain knowledge into pseudo-tasks, practitioners ensure that this knowledge is transferred to representation learning models prior to fine-

---

Under review conference paper at ICLR 2023
tuning them with labelled data. The premise of SSL attractive for EHR data, where few positive samples might be observed for a desired outcome and privacy limitations can prevent the collection of larger labelled datasets (Krishnan et al. 2022 Bak et al. 2022).
Methods for SSL are often applied to Transformers (Vaswani et al. 2017 which have proven to be an effective neural architecture for finding useful patterns across a variety of different domains. Self-supervised Transformer models currently produce state-of-the-art results in natural language processing (NLP) (Brown
el 2020), computational histopathology (Chen Krishnan 2021 | et al. 2022), computer vision (He ct al. 2022), and cross-modee learning (Radford et al. [021] On numeric EHR data (Li ct al. 202 Ren
et al. 2021 Tipireni & Reddy 2022) however, there remain many open questions regarding good self-
supervised tasks and whether SSL and Transformers can achieve the same level of success as in other fields.
In this work, we present TESS, an approach for self-supervised training of Transformers on ICU EHR data that produces representations which generalise well to different downstream tasks of interest. Prior work along this vein has used embedded input sequences that have one sequence element for every patient event (Li et al., [021: Tipirenn & Reddy (022). This is limited in scalability as patients can have hundreds of events in a relatively short period of time, while memory and runtime complexity of self-attention lay- ers scales quadratically with input length. More efficient attention layers have been proposed (Wang al., 2020), but state-of-the-art Transformer models still generally use quadratic attention. Training large mod- els with this input representation consequently requires significant hardware resources or aggressive input truncation, which can negatively impact accuracy.
To address the limitations discussed above, we first propose an application of time binning to compress the input. We observe that increasing time resolution beyond certain point does not improve model performance, likely due to consecutive measurements of the same event within small time window being highly correlated. By aggregating events within time bins and including auxiliary data describing the input sparsity structure, we can significantly compress the input without substantial loss useful infor- mation. Each bin is projected using multi-layer perceptron (MLP) to a given embedding dimension, and combined with an embedded representation of the time period the bin represents. This shorter and denser input allows us to train larger and deeper Transformer models, which in turn leads to better representations.
Contributions:
Next, we propose an SSL approach to train TESS. Measurements that clinicians choose to take reflect their understanding of patient's condition and related treatment strategies. Consequently, the missingness patterns of different events contain predictive signals of interest for a variety of different tasks (Lipton et al.) 2016) since they represent a clinician's unobserved lack of intent to treat or measure the value in question. A good representation of a patient's 's state should be aware of both the past state that the patient transitioned from, as well as the future states they might evolve into. We construct SSL tasks by predicting a combination of both missingness masks and event values. Since adjacent measurements of the same event are likely to be highly correlated, we introduce masked event type dropout scheme that encourages the model to design representations that pull information from other time-bins and events rather than using simple interpolation.
We evaluate TESS on the MIMIC-IV (Johnson et al.) 2022) and PhysioNet- (Silva et al.) 2012) datasets, showing that it outperforms state-of-the-art baselines multiple downstream tasks such mortality pre- diction and phenotype classification. We also evaluate the efficacy of self-supervised learning with TESS, showing that it learns to produce good representation of patients that can be fine-tuned effectively with only a small amount of labelled data.
2 RELATED WORK
Deep learning for sparse irregular time series. A variety of neural network models have been pro- posed for supervised learning on sparse irregular time series data. Most are based on recurrent neural networks (Hochreiter & Schmidhuber 1997: Cho al. 2014) that expect regularly sampled inputs with-
2

---

Under review conference paper at ICLR 2023
out missingness, so modifications are required to account for sparse data. One simple approach impute missing values based on previous values or overall statistics. mTANN (Shukla & Marlin 2021 use more advanced attention-bann based interpolation approach to produce a regular input for an RNN model. Architec- tural modifications can also be added to allow RNNs to adapt their hidden state appropriately when inputs are missing, as in CT-GRU (Mozer et al. 2017) and GRU-D (Che et al., 2018). Another line of research uses differential equations to model underlying continuous processes that are related to irregularly sampled inputs (Rubanova et al.) 2019, Lechner& & Hasami 2020 Kidger et al.) 2020), but these approaches require the use of differential equation solvers during traming and inference. usually making them slower than or- dinary neural networks (Shukla & Marlin, 2021). More recently, Raindrop (Zhang et al. 2022) has applied graph neural networks to aggregate observation embeddings, achieving ofte-of-theeart results.
The success of Transformer models in NLP makes them an attractive candidate for other tasks involving sequential data. Many Transformer-based models have been proposed for regular time series data (Wen et al. 2022), but there are fewer models that extend them to sparse irregular time series. RAPT (Ren et al. 2021) ) introduces a modified time-aware attention mechanism deal with irregular inputs. STraTS (Tipimenni & Reddy (2022) instead embeds every individual observation as triplets of time, variable and value using MLPs, then passes this sequence of observations a Transformer. This approach suffers from the limitation that Transformer memory usage is quadratic in sequence length, meaning that the amount of observed data is practically limited and the method must use very small Transformers (the proposed model only uses two Transformer layers and embeddings of length 50). A different approach, Hi-BEHRT (Li et al. 2021), uses a hierarchical Transformer architecture to be able to process longer input sequences of individual observations.
In contrast, our approach addresses this limitation by creating binned time series input representation. This approach is loosely inspired by recent work in computer vision, which showed that processing patches of images before passing them to a Transformer achieves excellent results (Dosovitskiy et al. 2021). suggesting that patching can be used to adapt Transformers to new data modalities.
Previous works have applied transformers to model longitudinal EHR data by using the sets of diagnostic codes applied at hospital or doctor visits as their inputs (Rasmy et al., 2021 Li et al. 2020 Zhang et al., 2020). However, these models not applicable to the kinds of data we investigate here, since their inputs do not contain numeric measurements and their temporal resolution is limited to one observation per visit.
Self-supervised learning. SSL has become important framework to enable learning useful represen- tations from data without relying on labels. SSL with Transformers has driven recent advances in NLP and computer vision (e.g. Devlin et al. (2019): Dosovitskiy et al.) (202I)). and has clear potential to ad- vance other fields. Different approaches for SSL with numeric EHR data are explored in McDermott et al. (02TT) but these are applied to a relatively basic GRU model and do not claim achieve ofae-of-thh--a results. mTANs use similar approach to SSL based on reconstructing inputs, incorporating VAE loss into their training procedure, but without a distinct pre-training stage. SSL is used with Transformers in Hi-BEHRT, which applies BOO (Grill et al. (020) to augmentations of EHR time series data. STra'Ts uses masked value prediction for SSL. with Transformers, while RAPT additionally uses a reasonability check to identify corrupted sequences and contrastive patient similarity task. Our SSL approach instead adds presence/absence prediction task, which captures meaningful priors of clinicians regarding a patient's 's state.
3 TRANSFORMER FOR EHR DATA WITH SELF SUPERVISED LEARNING
We consider a sparse irregular time series dataset of the form D= ((s WP. } where each patient stay p is associated with a set of outcomes y", a vector of static inputs sP "sai and a sequence of events WP = (wf, wo, wh of variable length, Np. Each event w" contains a triplet (13:1% wf) representing
event-type, time since start of stay and value (if applicable); for example, [heart.rate, 5.32 days, 4 41bpm] Static variables do not change during the stay and contain data such as age, gender, and so
3

---

Under review conference paper at ICLR 2023
[REP]
Prop
Classifiction boad
inputs
%2
AG
%
MLPazi:
Kusai
Time-series inputs and
F1,1
F1.4
Lince
MI
Vi
CVR(
e1
Predict value head
Fea
Multiayerr Transformer cocoder
MLP,
CVE((s)
21,3
23.4
MLP.
//
[MAS]]
mask head
Lam
MLPs
Va
+ CVB(2))
Ca
X
Figure I: Architecture diagram for a single patient stay. Binned time series events a and corresponding masks are passed through the MLP encoder to produce d-dimensional representations These represen- tations are combined with the time embedding CVE(t) for each bin. Similarly, static patient data is passed through the MLP static encoder and concatenated with bin representations form the inputs the Trans- former. During pre-training, bins are masked with a learned [MASK] embedding and then reconstructed via mask and value heads. For downstream fine-tuning, patient representation [REP] added the input sequence and the corresponding Transformer output Zrep is used to attach the downstream task head.
In the following section, we present the construction of the input representation for TESS from the full event sequence for each patient stay and omit the corresponding index p for notational simplicity. Then, we outline the architecture, SSL pre-training, and fine-tuning approach.
3.1 MODEL ARCHITECTURE
The first step of our input processing splits the full sequence of patient events, W, into L time bins of equal length. For each patient stay, we define binned input matrix æ E IR 4xm where the element æ7, contains single value representing an aggregation of all observed values of event-type in a given time bin I, and ris is the number of unique event-types selected inputs the model. We can express 21,4 as
F1,6 (S(x)) where St,, {wy If, i 1)6 t, S bl}
(1)
and where b /,,,///// / is the bin size for the patient stay, and g is aggregation function. Choices for the aggregation function include in the time bin. Elements of with observations in the corresponding time bin set to 0. We also build a mask tensor m E I Lx where my, is set to if there are observations for event-type and 0 otherwise. Passing the presence/absence of events to the model provides useful information types of analyses and treatments that clinicians have selected for the patient, as well as allowing the model distinguish between measured zero value in and missing value. By adapting the number of bins
mean value of events, maximum or minimum, or the last value observed
time bin I, the
4

---

Under review conference paper at ICLR 2023
L, this input representation allows us to effectively control the trade-off between computational complexity and granularity of event information. In comparison with the input representation used by STraTS, this representation reduces the computational complexity from (n to O(L2) in terms of time an memory, where generally. L & Np. We also aim to develop an SSL approach that can be effectively applied to raw input data with minimal processing, as feature engineering can highly time consuming and require extensive domain expertise, and so we only consider basic aggregation functions.
time series encoder MI.P&s used to convert Ti, and my, input representations for a given time bin I into a fixed-size embedding of dimension d, which we denote as Vi E I24, The encoder a fully connected feed-forward neural network with ReLU activation and batch normalization between layers and is shared for all time bins. The static data input 8 is similarly passed through a separate encoder MLP static to produce an encoding with the same dimensionality sstatic R.
Since the overall length of time represented in the imput bins can vary from patient to patient, only using the positional encodings would discard potentially useful information about the time scale. To incorporate this information, our model learns embeddings calculated from the continuous time values representing each time bin. We use the continuous value embedding (CVE) approach proposed in Tipimeni & Reddy (2022), which passes each time value through fully connected feed-forward neural network with one hidden layer of size vd and a tanh activation, followed by an output layer that produces a time embedding in Rdd In addition to incorporating continuous time information, the neural network is able to learn an embedding function that is well adapted to the data. These embeddings are then added to the output of the time series encoder to produce a final embedding for each time bin. The time value for time bin / is t bl, the difference between the bin end time and start of the patient's ' stay, and represents the (fractional) number of days that have passed since the start of the stay.
In addition to time bin embeddings, we also prepend the static data embedding to the input sequence as well as the learned patient representation token [REP]. Incorporating the static variables into the input rather than via late fusion as in recent models TTiirneni & Reddy 2022 /Zhang et al.) 2022) enables the Transformer to fully leverage this information in every self-attention block. Static variables such as age and sex provide crit- ical prior information so early fusion is beneficial here. The patient representation token queries a summary of relevant patient in- formation, and the corresponding Transformer output embedding Zeep is used for downstream classification tasks. To summarize, the input embedding tensor e E (2(+2))d is constructed according to:
the patient that can considerably influence both treatment strategies and outcomes,
[[REP], MLP static (s), MLP ((=1,; m..)) + CVE((t)) MLP, ([LL.) m2..)) + CVE(t))) (2)
Our approach of adding heterogeneous data by extending the input sequence makes it easy to incorporate other data sources and modalities. For instance, patches or other representations of medical images could be appended to the sequence along with learned embeddings representing their position, time, and modality. This is an especially promising possibility for health data, where medical images and doctors' notes are often available in addition to numeric measurements (Soenksen et al.) (022). This is not a straightforward task for most other models; for example, mANN and Raindrop are defined explicitly for time series input and XGBoost for tabular inputs which limits their ability to incorporate other modalities such as image and text data. Similarly, STraTS proposes late fusion of the tabular data which limits its ability to extract useful information through self-attention layers.
The input embeddings tensor is passed through a multi-layer Transformer encoder to produce a representa-
tion sequence x We use the standard Transformer blocks in Vaswani et al. (0117) with two modifications: we move normalization inside the residual connections as described in Xiong et al. (2020) and use ScaleNorm instead of LayerNorm as in Nguyen & Salazar (2019) to make training more stable. Dropout is used on feed-forward and attention connections. A full diagram of our model architecture is shown Figure I
5

---

Under review conference paper at ICLR 2023
3.2 TRAINING
The model is trained in two phases: self-supervised pre-training, then fine-tuning on downstream tasks.
SSL pre-training: During pre- treitrainnng, we adopt the mask mask-predict approach that has been highly successful in the NLP and computer vision domains (Devlin et al., [209 He et al. 2022). and repeatedly mask time bin input embeddings e with a learned [MASK] embedding. Bins are chosen uniformly at random for masking, and the number of masked bins each training step is set hyperparameter. We design self-supervised task focused on predicting both the presence/absence of an event and its value. For every masked bin 1, the corresponding Transformer output ž is passed mask and value prediction heads to produce the predictions 9ituss E " and D E "R The mask head predicts which events occurred in the time bin, and the value head predicts the value of observed events. These predictions are then compared with the actual values using the cross entropy and squared eтror losses. The pre-training loss for a single time bin / is:
L
Σ This
1
no
mi,s (Di.1
=1,1² [ms,, log(U), (giis + (1 m4,5) log(1 (
(3)
where a is a hyperparameter that controls the contribution of each task. We also introduce a novel masked event dropout scheme. For each event-type i having m4, 1 in at least one masked bin, all values of this event-type are dropped from the inputs by setting their values to across all time bins with some fixed probability. Patient observations such as labs and vitals are typically highly correlated within short time frame, so this dropout scheme prevents the model from learning to reconstruct events by simply interpolating from nearby time bins. The loss function and other inputs are unaffected.
During fine-tuning, we use the patient representation Zeep produced by the Transformer from the [REP] input and attach heads tailored the downstream tasks. Note that the [REP] embedding value is learned at this stage, since its output is not used during pre-training. For the tasks explored in this paper we use MLP classification heads with sigmoid output and binary cross entropy loss.
Fine-tuning:
4 EXPERIMENTS
We evaluate our proposed model on two widely used EHR datasets: MIMIC-IV (Johnson et al. 2022) and the PhysioNeu/CinC Challenge 2012 (Silva et al. 2012). In this section, we present our data preprocessing steps, experimental design and the results of evaluating our model performance compared to the leading baselines. We also present experiments to demonstrate the quality of the learned representation and conduct an ablation study to evaluate the impact of the components of our approach. We consider the following tasks to evaluate our models:
MIMIC-IV (Johnson et al. 2022 is a publicly available dataset that contains retrospective, deidentified data of patients admitted to the ICU or the emergency department at the Beth Israel Deaconess Medical Center between 2008 and 2019. This dataset contains data of various modalities such as time series data, static tabular data, and medical images from 53, 150 patients with 69, 211 admissions. Our dataset consists of ICU stays from MIMIC-IV along with related targets. Following Harutyunyan et al. (019). we define mortality prediction and phenotype classification targets. We exclude patients below 18 years of age and patients with no chart or lab events recorded during the stay. Unlike Harutyunyan et al. (2019), we do not exclude patients with multiple ICU stays or transfers between ICU units during their stay. This results in larger dataset that more closely mimics the practical use of machine learning system in a hospital setting. The mortality prediction task uses the first 48 hours of the patient stay as the input time window, predicting whether death occurs later during the hospital stay and has 13% positive instances. Patients with stays of less than 48 hours and patients with no recorded events before 48 hours are excluded from this task. The phenotype classification task uses the entire ICU stay as the input time window and uses a multi-label classification target, predicting 25 common hospital diagnoses. Details are provided in Appendix D We include all input
6

---

Under review conference paper at ICLR 2023
Table 1: Performance on MIMIC-IV mortality and phenotyping, and PhysioNet-2012 mortality tasks. Phe- notyping metrics are macro- averro-de Multiple top scores are bolded when the difference between them is not statistically significant according to the Wilcoxon signed rank test.
MIMIC-IV
PhysioNet-2012
Model
Mortality
Phenotyping
Mortality
ROC-AUC
PR-AUC
ROC-AUC
PR-AUC
ROC-AUC
PR-AUC
XGBoost
0,886 + 0.003
0.593 = 0.004
0.8287 + 0.0003
0.5891 + 0.0007
0.865 + 0.001
0,531 : 0.01
LSTM
0.881 + 0.001
0.533 = 0.006
0.7564 + 0.0002
0.4467 + 0.0004
0.848 + 0.002
0.494 = 0.002
m'TANDD
0.864 + 0.002
0.540 0.007
0.812 + 0.001
0.553 + 0.003
0.857 + 0.001
0.515 + 0.007
STraTS
0.882 + 0.004
0.552 0,013
0.8196 + 0.0008
0.565 0.002
0.852 + 0.008
0.527 = 0.02
Raindrop
0.878 + 0.001
0.546 0.002
0.824 0.001
0.577 0.003
0.838 + 0.01
0.479 + 0.02
TE(SS-)
0.878 + 0.004
0.552 = 0.007
0.818 + 0.0008
0.562 + 0.002
0.854 + 0.01
0.522 + 0.02
TESS
0.895 + 0.002
0.607 0.003
0.829 + 0.001
0.588 + 0.002
0.871 + 0.004
0.561 + 0.007
variables used in Harutyunyan al. events that are observed in more than 50% of ICU stays. This substantially increases the set of variables, and provides a rich input signal the model. The variables are listed in Appendix [ We use a patient-level 70%: 19%:11 split between the training, validation, and test sets.
2019 as well as a number of static variables and all chart and lab
PhysioNet-2012 (Silva et al., 2012) a standardized dataset with the task of predicting in- in-hospital mortality after the first 48 hours of patient stays in the ICU, where 14% of mortality labels are positive. The dataset consists of 12, 000 ICU stays with 42 different variables including 37 time series event-types. The details of the dataset, including data statistics, are provided in (Silva et al. (2012). We use the torchtime (Darke et al.) (022) data library to preprocess the data in a standard way and to split the dataset into training, validation and test sets (70%: 15% 15%).
Implementation: We apply zero-mean and unit-standard deviation normalization to all inputs in æ. We also clip outliers using threshold of three median absolute deviations from the median. These steps allow for stable training without clinical knowledge of normal variable ranges. For all models, when doing supervised training, we use binary cross entropy loss. We weight positive and negative instances according to the target positive fraction so that they receive equal weight in the loss. TESS is trained with L = 32 time steps, a 0.2, batch size of 512, and embedding dimension d = 768. Events in each time bin are aggregated by taking the last observed value. We perform self-supervised pre-training for 200 epochs using Adam W (Loshchilov & Hutter, 2017) with learning rate 10-4 and weight decay 10 Linear warmup is applied for the first 1000 steps and the learning rate is reduced by factor of 10 after 100 epochs. During pre-training, we apply dropout to the Transformer attention and feedforward layers with probability 0.2. We mask one time step per iteration, as masking more steps did not improve performance, and we apply our proposed masked event dropout with probability 0.5. Appendix A provides more details on hyperparameters. To show that our model generalises well without extensive tuning, we use the same hyperparameters for all tasks and datasets.
During fine-tuning, we find that additional regularization needed given the sparsity of positive labels. We increase the Transformer dropout probability to 0.5, mask input time bins with probability 0.2, increase the weight decay to 0.01, and increase the warmup steps to 5000. Note that we continue to use input bin masking here as it provides effective regularization analogous to input dropout. We stop fine-tuning when the PR-AUC does not increase for 40 epochs on the validation set and take the model from the epoch with highest PR-AUC on the validation set.
Baselines: We show that TESS is competitive across a range of baseline models, including the well estab- lished XGBoost and LSTT baselines well as state-of-the-arr deep learning models:
XGBoost CCen & Guestrin 2016): A scalable tree-based gradient boosting model that has been shown to outperform deep learning models tabular data (Shwartz-Ziv & Armon 2022).
7

---

Under review conference paper at ICLR 2023
0.9
0.6
PR AUC
ROC-AUC
Model
ROC-AUC
PR-AUC
0.5
01.66
mTAND
0.847 = 0.006
0.473 = 0.016
STaTS
0.875 = 0.001
0.525 = 0.001
0.84
TESS
0.8818 0.0004
0.5441 0.002
0.4
0.82
10
30
50
70
90
10
30
50
70
90
Table 2: Performance on the MIMIC- IV mortality prediction task with pre- trained encoders where the encoder weights are frozen during fine-tuning.
%6 of labelled training data
% of labelled training data
TESS
STaATS
in TAND
XGBoost
Raindrop
Figure 2: Performance on MIMIC-IV mortality prediction task using different percentages of labelled data for selected models.
LSTM (Graves 2012): A standard time series RNN. We use the same binned input format as for TESS. m'TAND Shukla & Marlin (202T): An encoder-decoder based model that uses an attention module to
interpolate irregular and sparse multivariate time series. Uses an unsupervised training task.
STraTS (Tipirenni & Reddy 2022): A Transformer-based model where every observation is embedded separately to produce the Transformer input sequence. Uses self-supervised pre-training. Raindrop Zhang et al. (2022): A graph-based neural network model that uses message passing between time series variables to learn relevant relationships.
We also compare to TESS without SSL, labelled TE(SS-). Results are shown in Table I and the details on baseline implementations are given in Appendix B We report the mean and standard deviation of ROC-AUC and PR-AUC over five supervised training runs using different random seeds. We note that while ROC-AUC and PR-AUC demonstrate similar trends, PR-AUC provides more discrimination between methods.
From Table we see that TESS outperforms baseline models on all datasets, except for XGBoost on MIMIC-IV pheotyping, where the performance of both models is par. Notably, our tuned XGBoost model is the strongest baseline across both datasets. This indicates that XGBoost with appropriate feature engineering and hyperparameter tuning is still competitive with neural network models for sparse irregular time series, and should be included evaluations of future methods. Compared to the Transformer- baseline model, STraTS, TESS shows large consistent gains across tasks, with 10% increase in PR-AUC on MIMIC-IV mortality. These results suggest that the increased input granularity of STraTS relative to TESS is not required for these tasks. The performance of TE(SS-) shows that our SSL approach drives gains on both datasets. We investigate the impact of input resolution and SSL further in our ablation study.
Results:
Representation Quality: To evaluate the quality of the representations learned by our model compared other baselines, we first carry out self-supervised pre-training, then freeze the encoder weights and fine-tune the model with linear classifier attached to the encoder. Among our baselines, mTAND and STraTS can also be trained in this way. The original mTAND model augments supervised training with an unsupervised component, so for fair comparison, we pretrain the mTAND ancoder-decode architecture using only the unsupervised loss. As shown in Table 2 TESS outperforms both of these baselines. This demonstrates the ability of TESS to learn useful patient representations from our self-supprvised pre-training approach, without relying labelled data. We also see that results for all models are lower than SSL combined with indicating that is preferable to fine-tune all weights.
end-to-end fine-tuning in Table D
We compare the performance of TESS with the baseline models when only a fraction of labelled data is used for supervised finetuning. Figure | shows that TESS outperforms the baselines consistently across all fractions of labelled data. The performance gap relative to the self-supervised Transformer baseline, STra'S widens as more labelled data is available, while the gain of TESS over the XGBoost increases as the percentage of labelled data decreases, demonstrating the effectiveness of SSL in the sparse setting.
3l visualizes 768 dimensional representations learned by TESS on MIMIC-IV using t-SNE (Van der Maaten & Hinton! 2008). We first reduce the embedding dimension to 50 using PCA and classify each point
Figure
8

---

Under review conference paper at ICLR 2023
Experiment
PR-AUC
Comorbidity
Baseline
0.607 + 0.003
No SSL
0.552 + 0.007
SSL without value prediction
0.554 + 0.004
SSL without mask prediction
0.597 + 0.004
SSL without masked event dropout
0.599 1 0.005
Omitting mask m from inputs
0.580 1 0.006
Using positional encoding for time
0.589 0.008
(a) With SSL pre-training
(b) With SSL pre- pre-training and fine-tuning Phenotyping
No static input
0.575 + 0.005
Binning with max aggregation Binning with mean aggregation
0,607 + 0.003
0.604 + 0.007
Figure 3: t-SNE visualization of the learned represen- tations for the MIMIC-IV dataset. Each dot represents patient categorized by the number of comorbities in their phenotyping target.
Table 3: Ablation study results measured on the MIMIC-IV mortality task.
based on the number of comorbidities, defined as the sum of diagnoses in the phenotyping target. The figure demonstrates that TESS distinguishes cases with high comorbidities from those with low comorbidities after SSL pre-training and the separation improves with fine-tuning.
To evaluate the importance of key components of our approach, including SSL and the input representation, we conduct an ablation study with results in Table As noted from
Ablation Study:
PR-AUC
0.61
0.6
G
the results with TE(SS-) in Table [ there is a large decrease performance without sule-supe pre- to-irraining. We see from T that removing each component of our SSL. approach, in- cluding value prediction, mask prediction and masked event drop out, leads to a considerable drop in performance. For the input representation, removing missingness information contained in the mask m results in a significant performance decrease. We see that using continuous value time embeddings also useful, given the performance drop from switch- ing to learned positional embeddings, which do not take into account the actual time values represented by each bin. We see that using different aggregation functions for binning such as taking max or mean value makes no notable difference in the performance. Finally, omitting static variables from the Transformer input results in a significant drop in the performance as expected.
0.59
8
16
24
32
40
48
Number of time steps
Figure 4: TESS PR-AUC on MIMIC-IV mortality using different numbers of time bins T..
Figure 4 shows the effect performance of varying L, the number of bins in the time series input. While the information loss from aggregating events in very few time bins is harmful, performance saturates above 40 time bins and starts to decrease. This shows that for EHR data, time series without negatively impacting performance, likely due to nearby measurements being highly correlated. We hypothesize that the decrease in performance above I, 40 is caused by useful inputs being more sparse, which can be harmful for neural network models handling tabular data Grinsztajn et al. (002
significantly compress the input
5 CONCLUSION
We introduce TESS, a Transformer architecture for deep learning on sparse irregular time series data in health care. Our experiments show that this architecture is especially effective in learning useful informa- tion during self-supervised pre-training, allowing it to outperform state-of-the-art models. We believe that advancing the state of the art in self-supervised learning will help drive substantial improvements in future health care modelling. It is straightforward to extend TESS to modtl inputs, so future work can take advantage of wider range of input data. We also expect that its approach is general enough to be applied to non-ICU EHR data and to sparse irregular time series data in other domains beyond health care.
9

---

Under review conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
We provide the details of our model architecture and hyperparameters necessary replicate our model in Section and Appendix A Further, to allow better reproducibility, we run the experiments independently in Tables to 3 and Figure by presetting the random seeds to 5 different values from 2020 2025. All other experiments were performed with random seed value set to 2020. We also provide the details of baseline implementations in Appendix [ We will be publishing implementation code for model as well the IDs for the patient-level splits on our lab's Github page along with the camera-ready version of the paper.
REFERENCES
Marieke Bak, Vince Istvan Madai, Marie-Christine Fritzsche, Michaela Th. Mayrhofer, and Stuart McLen- nan. You can't have AI both ways: Balancing health data privacy and access fairly. Front. Genet., 0, 2022. ISSN 1664-8021. doi: 20.388
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 2020.
Longbing Cao, Qiang Yang, and Philip Yu. Data science and AI in FinTech: An overview. International Journal of Data Science and Analytics, 12(2):81-99 2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv, April 2021. doi: 10.48550/ arXiv.210..12294.
Zhengping Che. Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural net- works for multivariate time series with missing values. Scientific reports, 8(1), 2018.
Richard Chen and Rahul G Krishnan. Self-supervised vision transformers learn visual concepts in histopathology. Learning Meaningful Representations of Life, NeurlPS 2021, 2021.
Richard J. Chen, Chengkuan Chen, Yicong Li, Tiffany Y. Chen, Andrew D. Trister, Rahul G. Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 16144-16155, June 2022.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the ACM Conference Knowledge Discovery and Data Mining, pp. 785-794, 2016.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decodder approaches. arXiv preprini arXiv::4099 1259, 2014.
S. Chopra, R. Hadsell, and Y. LeCun. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 1, pp. 539-546 vol. 1, 2005. doi: 10. 109/CVPR.2005.22
Philip Darke, Paolo Missier, and Jaume Bacardit. Benehmark time series data sets for PyTorch the torchtime package. arXiv preprint arXiv:2207. 12503, 2022.
Jacob Devlin, Ming-Wet Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
10

---

Under review conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Interna- tional Conference on Learning Representations, 2021.
Álvaro Gómez- Losada and Néstor Duch-Brown. Time series forecasting by recommendation: an empirical analysis on amazon marketplace. In International Conference on Business Information Systems, pp. 45- 54. Springer, 2019.
Alex Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks, 37 37-45, 2012.
Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernaro Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self- supervised learning. In Neural Information Processing Systems, 2020.
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do breeb models still outperform deep learning typical tabular data? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(1), 2019. doi: 10.1038/ $41597-019-0103--
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVE Conference on Computer Vision and Puttern Recognition, 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1(8):1735-1788 1997.
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV, 2022. URL / 1ttp://...1122/verrrre 2vcreeee//
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 2020.
Rayan Krishnan, Pranav Rajpurkar, and Eric J. Topol. Self-supervised learning in medicine and healthcare. Nat. Biomed. Eng., pp. 1-7, August 2022. ISSN 2157-846X. doi: 10. 0008//41551-022-000111
Mathias Lechner and Ramin Hasani. Learning long-term dependencies in irreullaly-ssaplle time series. arXiv preprint arXiv:2006. 04418, 2020.
Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Khorshiid. Behrt: transformer for electronic health records. Scientific reports, 10(1):1-12, 2020.
Yikuan Li, Mohammad Mamouei, Gholamreza Salimi- Khorshiii Shishir Rao, Abdelaali Hassaine, Dexter Canoy, Thomas Lukasiewicz, and Kazem Rahimi. Hi-BEHRT: Hierarchical transformer-basee model for accurate prediction of clinical events using multimodal longitudinal electronic health records. arXiv preprint arXiv:2106. 1360, 2021.
Zachary C. Lipton, David C. Kale, and Randall Wetzel. Modeling missing data clinical time series with RNNs. arXiv, June 2016. doi: 10.48550/arXiv.166 160..00..100000000..
11

---

Under review conference paper at ICLR 2023
Ila Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 2017.
Matthew McDermott, Bret Nestor, Evan Kim, Wancong Zhang, Anna Goldenberg, Peter Szolovits, and Marzyeh Ghassemi. A comprehensive EHR timeseries pre-training benchmark. In Proceedings of the ACM Conference on Health, Inference, and Learning, 2021.
Michael C Mozer, Denis Kazakov, and Robert V Lindsey. Discrete event, continuous time RNNs. arXiv preprint 17X.:1110000411,, 2017.
Toan Q Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self- attention. arXiv preprint arXiv:1910.05880 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Laila Rasmy, Yang Xiang. Ziqian Xie, Cui Tao, and Degui Zhi. Med-bert: pretrained contextualized em- beddings large-scale structured electronic health records for disease prediction. NPJ digital medicine, 4(1):1-13, 2021.
Houxing Ren, Jingyuan Wang, Wayne Xin Zhao, and Ning Wu. RAPT: Pre-training of time-aware trans- former for learning robust healthcare representation. In ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining, 2021.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for sangle time series. Advances in Neural Information Processing Systems, 32, 2019.
Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021.
Ravid Shwartz-Ziv Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
Ikaro Silva, George Moody, Roger Mark, and Leo Anthony Celi. Predicting mortality of ICU patients: The PhysioNer/Commputing Cardiology challenge 2012, Jan 2012. URL htttss://phsiinnete org/
content/challengee 2011 0.07
Luis R Soenksen, Yu Ma, Cynthia Zeng, Leonard Boussioux, Kimberly Villalobos Carballo, I Liangyuan Na, Holly M Wiberg, Michael Li, Ignacio Fuentes, and Dimitris Bertsimas. Integrated multimodal artificial intelligence framework for healthcare applications. NPJ Digital Medicine, 5(1):1-10, 2022.
Sindhu Tipimeni and Chandan K. Reddy. Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series. ACM Transactions on Knowledge Discovery from Duta 1(1), 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE, Journal of machine learning research, 9(1 1), 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances Neural Information Processing Systems, 2017.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprini arXiv:2006.04766 2020.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. arXiv preprint a7xi::2202.077/25, 2022.
12

---

Under review conference paper at ICLR 2023
Ruibin Xiong, Yunchang Yang. Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference Machine Learning, 2020.
Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for ir- regularly sampled multivariate time series. In International Conference Learning Representations, 2022.
Xianli Zhang, Buyue Qian, Shilei Cao, Yang Li, Hang Chen, Yefeng Zheng, and lan Davidson. Inprem: an interpretable and trustworthy predictive model for healtheare. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 450-460, 2020.
Yongfeng Zhang, Min Zhang, Yi Zhang, Guokun Lai, Yiqun Liu, Honghui Zhang, and Shaoping Ma. Daily- aware personalized recommendation based on feature-level time series analysis. In Proceedings of the 24th International Conference on World Wide Web, pp. 1373-1383, 2015.
13

---

Under review conference paper at ICLR 2023
Table 4: XGBoost Hyperparameter Tuning Distributions
Hyperparameter
Distribution
Number of rounds
Uniform on {50, 51,
250}
max.depth.
Uniform on {2.3..... 16)
eta
Log-uniform on [0.001,
lambda a alpha
Log-uniform on [0.001, 1
Log-uniform on 00.001,
subsample
Uniform on [0.2, 1]
min.chill.weightt
Log-uniform on [0.01, 100]
A ADDITIONAL TESS HYPERPARAMETERS
our experiments, we use the following dimensions for the subnetworks within TESS. The static encoder a single linear layer. The time series encoder has two hidden layers of size 256. The Transformer encoder has 4 layers, uses heads in its multi-head attention blocks, and uses hidden layers of size 4096 in its feedforward networks. The self- sepe-supe prediction heads have two hidden layers of size 256. The classification head has three hidden layers of size 128.
For the t-SNE plot, we used the implementation provided by scikit-learri with the following hyper- parameters: perplexity = 20, n_ieer 1000, learning-rate 900, random = 2021, init = "pca".
B BASELINE DETAILS
For XGBoost, we use the same aggregated input representation as for TESS, with all æ. m, and 8 values concatenated into feature vector. However, we find that XGBoost does not handle the sparsity of inputs well, and so we impute missing x values using the last previously observed value when available. We perform random tuning with 100 tests using the hyperparameter distributions given in Table 4 and use the configuration with best PR-AUC on the validation set.
For mTAND, lished code repository, using their PhysioNet hyperparameters for our PhysioNet tests and their MIMIC-III hyperparameters for MIMIC-IV tests. Unlike the datasets evaluated in their paper, our MIMIC-IV phe- notyping task uses arbitrarily long patient stays as input data, making it infeasible to train with the provided configurations. To mitigate this issue, for phenotyping only, increase the quantization windows from 30 minutes and we limit the length of input data to the first two weeks of the patient stay. We also find that our pero-mean unit-variance normalization massively increases the mTAND reconstruction loss and reduces performance. For all tasks, we instead scale all variables to range from 0 to 1, matching their provided code. Further, we encode the static variables as time series with one sample as an input, which matches the mTAN code repository.
the configuration/lyperpeee perparameters given in Shukla & Marlin 2021 and their pub-
For Raindrop, we use the confguraiionhyyerpraamtess given Zhang et al. (002)) and their published code repository for PhysioNet-2012. For our PhysioNet tests, we use the raw time steps given in the dataset and do not discretize time. Unlike PhysioNet, the set of time steps at which observations can be made in MIMIC-IV is not limited, making it infeasible to use raw inputs. To provide fair comparison on MIMIC- IV, we use the same discretized time bins as for TESS. The static data is passed directly into the Raindrop model their implementation also handles the static data along with the time series data.
http:///dchhtaaaaaaaaallmmmmlmmd manifold. www.iifo...SN....... html
14

---

Under review conference paper at ICLR 2023
For STraTS, the configuatioohhyyeraaranteses are set according Tipimeni Reddy (2022) and their pub- lished code repository. As suggested in the paper, we set the maximum number of observations to the 99th percentile the observations the 48h observation window. This results in 1832 and 1898 maximum sequence length for MIMIC-IV and PhysioNet respectively. The static data for STraTS is passed through a feed-forward neural network to obtain the embedding before concatenating with the time series embedding and passing through the final dense layer as described in the original paper.
C ADDITIONAL RESULTS
The extended version of Figure 2 along with the performance of Raindrop and all the models evaluated at additional data points is given in Figure 3 We see that the same trend noted in Section 4 holds for this extensive evaluation.
0.9
0.6
0.85
0.5
ROC-AUC
PR-AUC
0.8
0.4
0.75
0.3
0.7
0.2
60 70 80
90
100
20
30
40
50
60
70
100
labelled training data
% labelled training data
TESS
LSTM
STaaTS
mTTAND
XGBoost
Raindrop
Figure 5: Performance on MIMIC-IV mortality prediction task using different percentages of labelled data.
To analyze the performance of TESS on different patient sub- sub-groups, we evaluated our model on different groups of patients in MIMIC-IV dataset stratified according various demographics such age (Table 5). race (Table and gender (Table We see from Table 5 that the model performance is higher for the lower age group. This could be explained by the fact that the mortality risk varies across different age groups and it might be easier to predict mortality for some age groups compared to others. As shown in Table 6 TESS performs well across different gender groups. We also are fairly evenly distributed in both the training and test dataset. Table shows the performance of T'ESSn different race groups. We low performance the American Indian sub-group which can be explained by the very low number of samples in both the training and testing data. As the test dataset for American Indian race only consists of 13 samples, the performance metries reliable.
that the number of samples for each sub-groups
15

---

Under review conference paper at ICLR 2023
Age
ROC-AUC
PR-AUC
Test Samples
Train Samples
18-29
0.963
0.725
202 (3.6%)
715 (3.63%)
30-49
0.92
0.658
655 (11.8%)
2608 (13.24%)
50-59
0.911
0.661
951 (17.2%)
3279 (16.65%)
60-69
0.9
0.599
1346 (24.3%)
4583 (23.27%)
70 above
0.864
0.587
2385 (43.1%)
8514 (43 (3.22%)
Table 5: Performance of TESS the MIMIC-IV mortality task different sub- groups stratified by age along with the number samples in train and test dataset.
Gender ROC-AUC
PR-AUC
Test Samples
Train Samples
M
0.882
0.608
3140 (56.7%)
10964 (55.66%)
F
0.902
0.597
2399 (43.3%)
8735 (44.34%)
Table 6: Performance of TESS on the MIMIC-IV mortality task on different sub-grouph stratified by gender along with the number samples in train and dataset.
Race
ROC-AUC
PR-AUC
Test Samples
Train Samples
Asian
0.844
0.55
144 (2.6%)
594 (3.02%)
White
0.887
0.574
3729 (67.3%)
13068 (66.34%)
American Indian
()*
0.125*
13 (0.23%)
39 (0.20%)
Black/Africe American
0.907
0.545
578 (10.4%)
2096 (10.64%)
Hispanic
0.891
0.601
216 (3.9%)
699 (3.55%
Other
0.834
0.627
236 (4.26%)
809 (4.11%
Missing
0.8997
0.575
68 (1.23%)
345 (1.75%)
Unknown
0.898
0.723
555 (10.01%)
2049 (10.40%)
Table 7: Performance of TESS on the MIMIC-IV mortality task on different sub- sub-groups stratified by race along with the number of samples in train and test dataset. *Metries showing unrealistic results due few test samples.
16

---

Under review conference paper at ICLR 2023
-10
6
A PR-AUC
0.2
4
0.1
2
18-29
30-49
50-59
60-69
above 70
2-4
4-6
6-8
above 8
Age
LOS in days
Figure 6: Performance gain with SSL pre-training task on the MIMIC-IV mortality prediction task. The patients are stratified into different sub-categories based age and the length of the ICU stay (LOS).
Ground troth
Predictious
Predictions with nearby inputs
Ground troth
Predictious
Predictions with nearbhy inputs
Figure 7: Sample ground truths and model predictions from masked value pre-training. Only variables that are present in the masked time step are shown. The bottom plot shows the model output in the context of nearby input time steps to show when the model can and cannot interpolate from nearby data.
6ssows the performance gain across different sub-groups with SSL pre-training. We stratify MIMIC- IV mortality test dataset based on different metadata such as age of the patient, and patient's 's length of stay (LOS) in the ICU and evaluate TESS and TE(SS-) on each sub-category. We see a gain from SSL pre- training across all the age and LOS strata. We can also see that the gain in the performance increases with the LOS which indicates that the pre-training is able learn good representation across time.
Figure 7 visualizes some sample outputs from the value prediction head after pre-training, comparing them ground truth values and showing them in the context nearby time steps that are not masked. The effects of our proposed event type dropout can be seen here, as many events that the model could otherwise use to predict inputs using interpolation have been removed entirely. Despite this added difficulty, the model can still make good predictions for the dropped event types.
17

---

Under review conference paper at ICLR 2023
D MIMIC-IV BENCHMARK DETAILS
We use all variables from Harutyunyan et al. (019)) except GCS total, which does not have corresponding item in MIMIC-IV, plus chart and lab variables observed in 50% or more of patient stays. This amounts 85 chart event variables and 29 lab event variables. Similarly, we include 9 different static variables. All variables are given in Table The categorical variables are encoded using one-hot encoding giving us a 47-dimensional vector. Our traiiing set consists of total of 19, 699 instances with a positive mortality rate of 12.95%, our validation set contains 4, 257 instances with a mortality rate of 13.55%, and our test set contains 4, 245 instances with mortality rate of 12.39%.
Variable
Type
Source
Item IDs
Variables from Harutyunyan et al. (2019)
Capillary refill rate Diastolic blood pressure
Time series Time series
223951, 224308
220051, 2201 80,
224643, 225310,
227242
Fraction inspired oxygen Glasgow coma scale eye opening Glasgow scale verbal response Glasgow coma scale motor response Glucose
Time series
223835
Time series
220739
Time series
223900
Time series
223901
Time series
220621, 225664,
226537, 228388
Heart rate Height
Time series
220045
Time series
226707, 226730
Mean blond pressure
Time series
220052, 220181
Oxygen saturation
Time series
220227, 220277
Respiratory rate
Time series
220210. 10. 223851,
224689, 224690
Systolic blood pressure
Time series
220050, 220179,
224167, 225309,
227243
Temperature
Time series
223761, 223762,
224027
Weight
Time series
224639, 226512,
226531
pH
Time series
220274, 220734,
223830. 228243
Additional time series variables
Heart Rate
Time series
ICU Chartevents
220045
02 saturation pulseoxymetry
Time series
ICU Chartevents
220277
Respiratory Rate
Time series
ICU Chartevents
220210
GCS- Eye Opening
Time series
ICU Chartevents
220739
GCS- Verbal Response
Time series
ICU Chartevents
223900
GCS Motor Response
Time series
ICU Chartevents
223901
Alarms On
Time series
ICU Chartevents
224641
Parameters Checked
Time series
ICU Chartevents
224168
Heart Rate Alarm Low
Time series
ICU Chartevents
220047
Heart rate Alarm High
Time series
ICU Chartevents
220046
Non Invasive Blood Pressure mean
Time series
ICU Chartevents
220181
Non Invasive Blood Pressure systolic
Time series
ICU Chartevents
220179
Non Invasive Blood Pressure diastolic
Time series
ICU Chartevents
220180
02 Saturation Pulseoxymetry Alarm Low
Time series
ICU Chartevents
223770
18

---

Under review conference paper at ICLR 2023
02 Saturation Pulseoxymetry Alarm High
series
ICU Chartevents
223769
Resp Alarm High
Time series
ICU Chartevents
224161
Resp Alarm Low
Time series
ICU Chartevents
224162
Braden Sensory Perception
Time series
ICU Chartevents
224054
Braden Mobility
Time series
ICU Chartevents
224057
Braden Moisture
Time series
ICU Chartevents
224055
Braden Activity
Time series
ICU Chartevents
224056
Braden Nutrition
Time series
ICU Chartevents
224058
Braden Friction/Shear
Time series
ICU Chartevents
224059
SpO2 Desat Limit
Time series
ICU Chartevents
226253
Temperature Fahrenheit
Time series
ICU Chartevents
223761
IV/Saline lock
Time series
ICU Chartevents
227344
Gait/ Tail/Transeering
Time series
ICU Chartevents
227345
Ambulatory aid
Time series
ICU Chartevents
227343
Mental status
Time series
Chartevents
227346
Secondary diagnosis
Time series
ICU Chartevents
227342
History of falling (within 3 mnths)
Time series
ICU Chartevents
227341
Potassium (serum)
Time series
ICU Chartevents
227442
Sodium (serum)
Time series
ICU Chartevents
220645
Chloride (serum)
Time series
ICU Chartevents
220602
Creatinine (serum)
Time series
ICU Chartovents
220615
BUN
Time series
ICU Chartevents
225624
HC03 (serum)
Time series
ICU Chartevents
227443
Anion gap
Time series
ICU Chartevents
227073
Hematocrit (scrum)
Time series
ICU Chartevents
220545
Glucose (serum)
Time series
ICU Chartevents
220621
Hemoglobin
Time series
ICU Chartevents
220228
Platelet Count
Time series
ICU Chartevents
227457
WBC
Time series
ICU Chartevents
220546
Magnesium
Time series
ICU Chartevents
220635
Non- Ton-Invasive Blood Pressure Alarm Low
Time series
ICU Chartevents
223752
Non-Invasive Blood Pressure Alarm High
Time series
ICU Chartevents
223751
Phosphorous
Time series
ICU Chartevents
225677
Calcium non-ionized
Time series
ICU Chartevents
225625
Pain Level
Time series
ICU Chartevents
223791
Richmond-Re RAS Scale
Time series
ICU Chartevents
228096
Prothrombin time
Time series
ICU Chartevents
227465
INR
Time series
ICU Chartevents
227467
PTT
Time series
ICU Chartevents
227466
Capillary Refill R
Time series
ICU Chartevents
223951
Capillary Refill L
Time series
ICU Chartevents
224308
Admission Weight (lbs.)
Time series
ICU Chartevents
226531
Goal Richmond-RAs Scale
Time series
Chartevents
228299
ST Segment Monitoring On 02 Flow
Time series
ICU Chartovents
228305
Time series
ICU Chartevents
223834
Glucose finger stick (range 70-100)
Time series
ICU Chartevents
225664
Pain Level Response
Time series
ICU Chartevents
224409
Intravenous/ IV access prior to admission
Time series
ICU Chartevents
225103
20 Gauge Dressing Occlusive
Time series
ICU Chartevents
227368
Strength Arm
Time series
ICU Chartevents
228412
Strength Arm
Time series
ICU Chartevents
228409
Strength Leg
Time series
ICU Chartevents
228411
Strength Leg
Time series
ICU Chartevents
228410
20 Gauge placed in outside facility
Time series
ICU Chartevents
226138
Insulin pump
Time series
ICU Chartevents
228236
19

---

Under review conference paper at ICLR 2023
Self
series
ICU Chartevents
225092
20 Gauge placed in the field
Time series
ICU Chartevents
228100
History of slips / falls
Time series
ICU Chartevents
225094
High risk (251) interventions
Time series
ICU Chartevents
227349
Lactic Acid
Time series
ICU Chartevents
225668
Home
Time series
ICU Chartevents
228648
ETOH
Time series
ICU Chartevents
225106
Pressure Uleer Present
Time series
ICU Chartevents
228649
Difficulty swallowing
Time series
ICU Chartevents
225118
Gauge Dressing Occlusive
Time series
ICU Chartevents
227367
18 Gauge placed in outside facility Eye Care
Time series
ICU Chartevents
226137
Time series
ICU Chartevents
225184
Visual hearing deficit
Time series
ICU Chartevents
225087
Currently experiencing pain
Time series
ICU Chartevents
225113
Dialysis patient
Time series
Chartevents
225126
Daily Weight
Time series
ICU Chartevents
224639
Potassium
Time series
ICU Labevents
50971
Chloride
Time series
ICU Labevents
50902
Sodium
Time series
ICU Labevents
50983
Creatinine
Time series
ICU Labevents
50912
Urea Nitrogen
Time series
ICU Labevents
51006
Bicarbonate
Time series
ICU Labevents
50882
Anion Gap
Time series
ICU Labevents
50868
Glucose
Time series
ICU Labevents
50931
Hematocrit
Time series
ICU Labevents
51221
Platelet Count
Time series
ICU Labevents
51265
White Blood Cells
Time series
ICU Labevents
51301
Hemoglobin
Time series
ICU Labevents
51222
Red Blood Cells
Time series
ICU Labevents
51279
MCV
Time series
ICU Labevents
51250
MCH
Time series
ICU Labevents
51248
MCHC
Time series
ICU Labevents
51249
RDW
Time series
ICU Labevents
51277
Magnesium
Time series
ICU Labevents
50960
Phosphate
Time series
ICU Labevents
50970
Caleium, Total
Time series
ICU Labevents
50893
PT
Time series
ICU Labevents
51274
INR(PT)
Time series
ICU Labevents
51237
PTT
Time series
ICU Labevents
51275
pH
Time series
ICU Labevents
50820
Lactate
Time series
ICU Labevents
50813
Base Excess
series
ICU Labevents
50802
pO2
Time series
Labevents
50821
pCO2
Time series
ICU Labevents
50818
Caleulated Total CO2
Time series
ICU Labevents
50804
Static tabular variables
Age
Binary Variable
Admission Table
Gender
Binary Variable
Admission Table
English Language
Binary Variable
Admission Table
Marital Status
Categorical Variable
Admission Table
Insurance
Categorical Variable
Admission Table
Admission Location
Categorical Variable
Admission Table
Admission Type
Categorical Variable
Admission Table
Race
Categorical Variable
Admission Table
20

---

Under review conference paper at ICLR 2023
First Care Unit
Categorical Variable
ICU Admission Table
Observation Window Length
Continuous Variable
Derived
Table 8: Time series and static variables used from MIMIC-IV dataset.
Following Harutyunyan et al. (2019), the phenotyping task has 25 binary target variables, corresponding to whether the following conditions were billed during the stay:
Acute and unspecified renal failure
Acute cerebrovascular disease
Acute myocardial infarction
Cardiac dysrhythmias
Chronic kidney disease
Chronic obstructive pulmonary disease and bronchiectasis
Complications of surgical procedures or medical care
Conduction disorders
Congestive heart failure; nonhypertensive
Coronary atherosclerosis and other heart disease
Diabetes mellitus with complications
Diabetes mellitus without complication
Disorders of lipid metabolism
Essential hypertension
Fluid and electrolyte disorders
Gastrointestinal hemorrhage
Hypertension with complications and secondary hypertension
Other liver diseases
Other lower respiratory disease
Other upper respiratory disease
Pleurisy; pueumothorax; pulmonary collapse
Pneumonia (except that caused by tuberculosis or sexually transmitted disease)
Respiratory failure; insufficiency; arrest (adult) Septicemia (except in labor)
Shock
This task has total of 54, 024 training instances, 11, 401 validation instances and 11, 509 testing instances.
21


Reviews:
[{"review_id": "gkeuMS9L81c", "reviewer": "Reviewer_fWW7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on the robust representation of sparse time series EHR data. The authors propose a new model TESS, Transformers for EHR data with Self Supervised learning, a self-supervised Transformer-based architecture, to represent the patients’ data as fixed-size vectors with the consideration of irregular observations and missing values. The experiments on real-world datasets show that the proposed model outperforms the baselines.", "summary_of_the_review": "The authors present a transformer model to represent the patients’ static variables and dynamic observations as fixed-size vectors. The paper is neither novel from machine learning nor from clinical informatics. There are also some major concerns in the experiments.\n", "strength_and_weaknesses": "Strength:\n1. The authors present a transformer model to represent the patients’ static variables and dynamic observations as fixed-size vectors. \n\nWeakness:\n1. The novelty is somewhat incremental. \n    a) Model training: SSL pre-training and fine-tuning are also common tricks for deep learning models [1,2,8]. The work is an application of Transformer to the healthcare domain.\n    b) Model framework: Lots of existing studies (e.g., [5,6,7]) have combined static variables and dynamic variables when learning patient representations and considering the time intervals and missing value problems. \n\n2. The aggregation methods (e.g., mean, maximal, minimal, last observed variable) used in the paper might not be enough for real-world clinical settings. The frequency of lab tests could imply the patients’ disease severity. A simple aggregation might cause information loss.\n\n3. The experiments could be improved:\n    a) It would be better if the authors conducted experiments to show which aggregation methods are better for clinical settings (e.g., mean, maximal, minimal, last observed variable)?\n    b) The lengths of different patients’ EHR data vary a lot (e.g., from 1~3 days to several weeks in ICUs). It is tricky to select the value of L for different patients. It would be better if the authors conducted experiments to show how the model is sensitive to L for different groups of patients, especially for patients with extremely short or long observation periods. Moreover, in Figure 4, when L is set 48, the performance becomes worse, which is unusual and worth explaining why.\n    c) The statistics of dataset (e.g., age, gender) are missing.\n    d) It is unclear how to learn patient representation [REP].\n\n4. Experiment results show that the proposed model performs just as comparable to baselines on some tasks (e.g., phenotyping).\n\n5. The paper writing could be improved. For example, the authors claim that “the use of Transformer in modelling sparse irregular time series with tabular data has not been widely explored.” However, Transformer has been widely used in modeling EHR data (e.g., [1-4]).\n\n6. The implementation code is not available.\n\nReferences:\n\n[1] Rasmy L, Xiang Y, Xie Z, et al. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ digital medicine, 2021, 4(1): 1-13.\n\n[2] Li Y, Rao S, Solares J R A, et al. BEHRT: transformer for electronic health records. Scientific reports, 2020, 10(1): 1-12.\n\n[3] Li F, Jin Y, Liu W, et al. Fine-tuning bidirectional encoder representations from transformers (BERT)–based models on large-scale electronic health record notes: an empirical study. JMIR medical informatics, 2019, 7(3): e14830.\n\n[4] Zhang X, Qian B, Cao S, et al. INPREM: an interpretable and trustworthy predictive model for healthcare//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020: 450-460.\n\n[5] Baytas I M, Xiao C, Zhang X, et al. Patient subtyping via time-aware LSTM networks//Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 2017: 65-74.\n\n[6] Tipirneni S, Reddy C K. Self-supervised transformer for multivariate clinical time-series with missing values. arXiv preprint arXiv:2107.14293, 2021.\n\n[7] Yuan Luo, Peter Szolovits, Anand Dighe, and Jason Baron. 2018. 3D-MICE: integration of cross-sectional and longitudinal imputation for multi-analyte longitudinal clinical data. JAMIA 25, 6 (2018), 645–653.\n\n[8] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Q5dYvI09If", "reviewer": "Reviewer_HAwr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel solution to effectively handle the sparsity of data in EHR using a combination of techniques including self-supervision and aggregation based normalization of sparse inputs. The authors compared their model on two real world datasets for three different tasks. They claim that the scheme effectively improves modeling efficacy over state of the art methods\n", "summary_of_the_review": "The authors have analyzed an important and well-known problem in clinical AI. \nWhile the experiments on the chosen baseline is promising, the claims, especially considering the clinical perspective, is not well substantiated. Of note, the SSL method by itself is novel and from the results seem to contribute the most to the model performance. The method can be arguably applied to other architectures as well. It will be useful for the authors to deliberate on the claims e.g. either narrow the claims to ICU (i.e more regular) settings or provide experimental validations on longer term EHR data. Apart from the aforementioned aspects the authors may also want to consider the following aspects:\n\n- Provide intuitions in the paper (rather than form citation) about the CVE approach to capture temporal information over simpler methods such as cosine embedding\n- From Table 1, it seems TE(SS-) doesn't perform better than STraTS. This may indicate that SSL is the main driver of the performance. With this in mind, the authors may consider a simple extension of the SSL to base models such as LSTM or more classical DL method for clinical AI such as RETAIN and SAnD.\n- In the implementation section, the authors note that \n> We apply zero-mean and unit-standard deviation normalization of all inputs in x. We also clip outliers using a threshold of three median absolute deviations from the median. These steps allow for stable training without requiring clinical knowledge of normal variable ranges.\n    * While this is indeed beneficial from a ML modeling standpoint, it is not always clinically reasonable to neglect domain information such as typical values of clinical variables. EHR data is often noisy and such steps are often crucial to ensure that the model learns from reasonable manifestations of patient data\n- The authors claim that TESS handle the temporal granularity in a simpler manner than StraTS. While some justification is provided, it may be important to quantify these using some measures such as perplexity and/or computational requirement.\n- The discussions are model interpretations are not well supported. While t-SNE plots have been reported in Figure 3 there are two important problems with the analysis; (a) t-SNE can provide widely different manifolds based on the hyper-parameters. These should be reported. Comparison with other stable methods such as PCA could be considered. (b) the plots in Figure 3b can be argued to have some overlap - it is not evident from the naked eye whether TESS produces well separated representations. \n- On a similar note, for the model to be of real-world usage, the authors may want to investigate on the feature importance/attributions driving the improved performance\n", "strength_and_weaknesses": "Some of the key strengths of the paper are as follows:\n\n- The paper studies a very important problem that is prevalent in longitudinal EHR data. The results against the chosen baselines show strong performance and may have significant impact in using AI for clinical settings. Of special note the method performs better than XGBoost (see more below).\n- The proposed method of self-supervision is novel and simple. While the method is proposed for transformer based architecture, it could be adopted to most architectures such as LSTM as well other forms of attention based models.\n- The paper is well presented and the illustrations makes the paper easy to follow. The variables used in the model have been communicated. The experiments have also been conducted on two well known benchmarks and the results should be reproducible with some level of effort.\n\nThe paper can be improved upon by addressing some of the concerns below:\n\n- While the identified problem is well motivated, the clinical significance seems to lack sufficient depth. Compared to other forms of EHR, ICU data is well populated with less randomly missing data points. Also, typical EHR records, especially for chronic diseases, can go back for decades with large periods of data that are irrelevant to prediction for the time point of interest. The claim of the paper as a general solution for EHR data is thus not well supported. \n- Regarding novelty, the first aspect of the proposed method considers aggregation of data into bins, while novel in a model integrated manner, is a well known and often used pre-processing technique to handle EHR data. In fact, other forms of aggregation such as those based on higher level EHR concepts have also been used to reduce data sparsity. \n- In order to interpret the usefulness of the proposed method, it is critical to understand the performance over different sub-groups. At the very minimum, a typical \"Table one\" describing the data distribution should be presented. It is also important to understand whether the self-supervision can amplify biases in the data and discuss any proposed solutions to that\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3tLgjlguja", "reviewer": "Reviewer_6eHs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on the problem of learning from irregularly sampled time series data. Similar to the NLP and vision community, it employs a self-supervised pretraining approach using Transformers to improve the modeling of irregularly sampled time series data. It first applies a binning technique to covert the irregularly sampled time series to regularly sampled data to apply Transformers models. The paper also proposes self-supervised learning tasks for such data to learn better representation. Experiments show that the proposed approach outperforms several recent approaches on multiple downstream tasks across two datasets.  ", "summary_of_the_review": "My recommendation for this paper is weak reject. Although the paper brings some interesting ideas from vision and NLP community, it fails to deliver in terms of the results.", "strength_and_weaknesses": "### Strengths\n1. The paper proposes a simple approach to improve the learning of irregularly sampled time series data using self-supervised pretraining.\n2. Experimental results show the effectiveness of the approach when compared to baselines and other recent approaches.\n3. The paper focuses on the task of learning from irregularly sampled data which is important in many domains.\n\n### Weaknesses\n1. The paper claims to propose the idea of using a binning technique to transform irregularly sampled time series into a regularly sampled sequence data with missing values. This has been widely studied in literature as discussed in this survey ( Section 4.1, [1]). \n2. The novelty is marginal as the self-supervised training is similar to mTAN and some other approaches mentioned in the paper and the input representation is widely known.\n3. The experimental results seem to be just marginally better than a standard baseline of XGBoost even when the proposed approache uses self-supervised pretraining. XGBoost clearly outperforms the TE(SS-) approach on all datasets. Could this be addressed by pretraining on a large scale time series data? As shown in NLP and vision community, scale of the data really matters. Its something the authors could try to further improve the performance. Since this seems like mostly an empirical work, I would also expect to see some study towards the transfer learning capabilities of the pretrained models. \n4. How do the authors deal when different features in the input have different sampling rate which is quite common in healthcare data? How does the proposed approach differentiates between missing data (from binning) and masked data for pretraining? \n5. It is not clear if the baseline approaches are trained with input representation mentioned in the current paper or their corresponding papers. Some of the baseline approaches (like mTAND) do not utilize the static inputs. Were these baseline approaches trained with static inputs? mTAND results on PhysioNet seem to be the same as mentioned in the original paper where no static inputs were used.\n\n\n#### References\n1. Shukla, Satya Narayan, and Benjamin M. Marlin. \"A survey on principles, models and methods for learning from irregularly sampled time series.\" arXiv preprint arXiv:2012.00168 (2020).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "0-3uEfjWiq", "reviewer": "Reviewer_EBR4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presented a new method for self-supervised learning on sparse time series data. Specifically, Transformers for EHR data with self-supervised learning -- TESS was proposed with an input binning scheme and a combination of both missingness masks and event values. Experimental analysis on two public EHR datasets shows the effectiveness of the proposed method. The main contributions of this paper are the proposed bin-based data pre-processing scheme and the joint processing model for multiple data types in EHR applications.", "summary_of_the_review": "This paper is generally okay with clear motivation and a new approach to process EHR data with shown better performance on two widely used public datasets. There are a few concerns regarding the novelty and contributions, experiments and some statements/claims, but they could possibly be addressed within the rebuttal phase. As a result, I recommend a positive rating at the current stage and may change my recommendation after the rebuttal.", "strength_and_weaknesses": "**Strengths**\n\n\\+ This paper proposed a feasible and effective approach for self-supervised learning on EHR data.\n\n\\+ Each of the designed/used components within the proposed architecture was well-motivated.\n\n\\+ The proposed method was shown to perform better than the compared previous methods.\n\n\\+ An ablation study was presented to show the effectiveness of the components within the proposed method.\n\n\\+ The paper is generally well-written and easy to follow.\n\n\n**Weaknesses**\n\n\\- The technical novelty and contributions are a bit limited. The overall idea of using a transformer to process time series data is not new, as also acknowledged by the authors. The masked prediction was also used in prior works e.g. MAE (He et al., 2022). The main contribution, in this case, is the data pre-processing approach that was based on the bins. The continuous value embedding (CVE) was also from a prior work (Tipirneni & Reddy 2022), and also the early fusion instead of late fusion (Tipirneni & Reddy, 2022; Zhang et al., 2022). It would be better to clearly clarify the key novelty compared to previous works, especially the contribution (or performance gain) from the data pre-processing scheme.\n\n\\- It is unclear if there are masks applied to all the bins, or only to one bin as shown in Fig. 1.\n\n\\- It is unclear how the static data (age, gender etc.) were encoded to input to the MLP. The time-series data was also not clearly presented.\n\n\\- It is unclear what is the \"learned [MASK] embedding\" mean in the SSL pre-training stage of the proposed method.\n\n\\- The proposed \"masked event dropout scheme\" was not clearly presented. Was this dropout applied to the ground truth or the prediction? If it was applied to the prediction or the training input data, will this be considered for the loss function?\n\n\\- The proposed method was only evaluated on EHR data but claimed to be a method designed for \"time series data\" as in both the title and throughout the paper. Suggest either tone-down the claim or providing justification on more other time series data.\n\n\\- The experimental comparison with other methods seems to be a bit unfair. As the proposed method was pre-trained before the fine-tuning stage, it is unclear if the compared methods were also initialised with the same (or similar scale) pre-trained model. If not, as shown in Table 1, the proposed method without SSL performs inferior to most of the compared methods.\n\n\\- Missing reference to the two used EHR datasets at the beginning of Sec. 4.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject