CLUSTER-BASED FEATURE IMPORTANCE LEARNING
Year: 2021

Abstract:
The recent availability of Electronic Health Records (EHR) has allowed for the development of algorithms predicting inpatient risk of deterioration and trajectory evolution. However, prediction of disease progression with EHR is challenging since these data are sparse, heterogeneous, multi-dimensional, and multi-modal time-series. As such, clustering is used to identify similar groups within the patient cohort to improve prediction. Current models have shown some success in obtaining cluster representation of patient trajectories, however, they i) fail to obtain clinical interpretability for each cluster, and ii) struggle to learn meaningful cluster numbers in the context of the imbalanced distribution of disease outcomes. We propose a supervised deep learning model to cluster EHR data based on the identification of clinically understandable phenotypes with regard to both outcome prediction and patient trajectory. We introduce novel loss functions to address the problems of class imbalance and cluster collapse, and furthermore propose a feature-time attention mechanism to identify cluster-based phenotype importance across time and feature dimensions. We tested our model in over 100,000 unique trajectories from hospitalised patients with Type-II respiratory failure to predict four different outcomes. Our model yielded added interpretability to cluster formation and outperformed benchmarks by at least 5% in mean AUROC.

heading: 1 INTRODUCTION
text: Chronic conditions such as Chronic Obstructive Pulmonary Disease (COPD) and Cardiovascular Disease (CVD) describe a broad spectrum of medical ailments, and affect a significant percentage of the overall population (Adeloye et al., 2015). Such diseases are characterized by the existence of multiple distinct patient subgroups, largely distinguished by differences in pathology and in the response to different treatments and medical interventions Turner et al. (2015); Vogelmeier et al. (2018). Exacerbation of COPD, a condition of respiratory failure, can result in emergency hospital admission and mortality if it is not well treated and managed. Early identification of COPD patients’ subgroups is therefore of high medical importance and relevance. EHR time-series data are typically used to determine clinically relevant COPD inpatient subgroups, and have been applied to detect risk of deterioration (Pikoula et al., 2019).
However, modelling disease progression and risk prediction is challenging due to the extreme data heterogeneity nature of EHRs. Firstly, EHR data contains a mixture of demographic or static variables (i.e. time independent such as age and sex), and multi-dimensional time-series (e.g Heart Rate, HR, and laboratory measurements, such as blood tests). Secondly, EHR time-series are multi-modal as different features are collected from different devices, representing distinct clinical properties of relevance. Similarly, time-series features are sampled at different times and have low and distinct sampling rates, as well as different missing value properties. Furthermore, each feature is associated with different noise and evolution patterns.
Recent advances in deep learning (DL) approaches have shown promising results in EHR modelling due to their capacity to handle complex data (Rajkomar et al., 2018). Nonetheless, DL approaches lack relevant interpretability frameworks to be scaled and applied in hospital settings. Several such models have since been proposed to tackle this issue (Mayhew et al., 2018), however, most of them focus on a subset of EHR features (usually vital signs only) and fail to provide a clinically-focused phenotypic analysis of learnt patient sub-groups (via clustering).
This work builds on previous research in literature to introduce a cluster-based feature-time attention mechanism to predict patient outcomes based on EHR data. Our method also leverages phenotypic information to aid in clinical interpretability, not only making use of demographics and vital-signs information but also of relevant laboratory measurements (all present in the EHR) to provide a more complete patient physiological status. Our contributions include the following:
• An end-to-end DL supervised model to cluster EHR patient data based on the identification of clinically understandable cluster phenotypes with regard to both outcome prediction and patient trajectory in a multi-class setting;
• A weighted loss to address data imbalance for both tasks of clustering and prediction, a common issue in the medical domain;
• The incorporation of a novel loss mechanism in the model, to address the issue of cluster collapse and promote sample assignment to all available clusters;
• Finally, the inclusion of a novel interpretability framework, derived from a cluster-based feature-time attention layer, aiming to identify relevant timestamps and feature variables to represent the patient physiology, cluster assignment and, ultimately, outcome prediction.
This paper is structured as follows. In Section 2, we describe previous research in EHR time-series modelling, clustering and attention methods. Section 3 introduces the dataset used for analysis and provides description of the proposed model. The experimental setup and results of our analysis are presented in Section 4 and discussion takes place in Section 5. Finally, concluding remarks and future work are available in Section 6.

heading: 2 RELATED WORK
text: EHR data comprise complex time-series data, being high-dimensional, multi-modal and heterogeneous, and thus presenting challenges when used in machine learning models (Keogh & Kasetty (2003); Rani & Sikka (2012)). An important goal in a medical setting is to identify phenotypically separable clusters with distinct phenotypic profiles (which we denote as phenotypic clustering hereafter). For the purpose of this work, cluster phenotypes result from the combination of two distinct components: a) the evolution profile of patient trajectories’ within the cluster, and b) the characterisation of the cluster with regards to clinical variables of interest. The latter may include features not used for clustering and may provide information about the underlying or future health status. Traditional clustering models such as K-Means or hierarchical clustering have been shown to fail to capture the existing time-dependent feature relationships. As such, variants have been proposed to mitigate this problem. A temporal version of the K-Means algorithm, Time-Series K-Means (TSKM, Tavenard et al. (2020)), models the distance between time-series of different datapoints, using the Euclidean distance (which is equivalent to considering all temporal observations as an independent feature value for the corresponding patient admission), or time-series alignment strategies such as Dynamic-Time Warping (DTW, Berndt & Clifford (1994)) and soft-DTW (Cuturi & Blondel (2017)).
Recent DL architectures, ranging from Auto-Encoders (AE, Ma et al. (2019)), Convolutional Neural Networks (CNN, Munir et al. (2018)) and others, have shown great promise when applied to timeseries data across a variety of domains. Fortuin et al. (2019) proposed a Self-Organising Map - Variational Auto-Encoder (SOM-VAE), is a state-of-the-art, unsupervised, DL clustering algorithm which extends a variational auto-encoder architecture (Kingma & Welling, 2013) for observation learning and representation, through the addition of a Markov model (Gagniuc, 2017), to infer temporal evolution within the latent space. Clustering is performed in the low dimensional latent space through the use of self-organising maps (Kohonen, 1982) to obtain a discrete, topologically-interpretable latent representation of the learnt clusters. In a supervised setting, AC-TPC (Lee & Van Der Schaar, 2020) serves as the current state-of-the-art for identifying phenotypically separable clusters in patient trajectories in EHR data. AC-TPC maps EHR data into a latent space via an encoder, and uses an actor-critic network (Konda & Tsitsiklis, 2000) which leverages clinical outcomes to aid in cluster formation and obtaining cluster phenotypes. Neither SOM-VAE and AC-TPC provide clinically meaningful interpretation of feature-time importance or outcome of interest.
Attention mechanisms have recently been proposed to provide greater interpretability to Recurrent Neural Networks (RNN) and to aid in dealing with long-term dependencies (Vaswani et al., 2017; Xu et al., 2015), and have also been used in modelling EHR time-series (Schwab et al., 2017; Shashikumar et al., 2018). RETAIN (Choi et al., 2016) proposes a two-level reverse attention mechanism to mimic physician’s decision process and predict a future diagnosis. In other recent works, attention mechanisms based on bi-directional RNN and CNN outperformed standard classification models in predicting high risk vascular diseases with the addition of medication information as input data (Kim et al., 2017). A drawback of such attention mechanisms is the focus on temporal interpretability only, and inability to look at individual features, which is key in a medical setting. To solve this issue, Shamout et al. (2019) considered independent RNN per feature, with a concatenation of the resulting latent vectors. However, the latter does not allow the joint modelling across both feature and time dimensions. Alternatively, Kaji et al. (2019); Gandin et al. (2021) proposed learning attention weights directly on the original inputs, prior to being transformed by a RNN, which does not allow modelling of the resulting latent representations. To the best of our knowledge, no existing models have been proposed that jointly leverage both feature and time dimensions (feature-time) to determine clinical observation relevance on clustered EHR data.

heading: 3 METHODS

heading: 3.1 DATASET AND PRE-PROCESSING
text: Our dataset was retrieved from a retrospective database of routinely collected observations from concluded hospital admissions between March 2014 and March 2018 (the HAVEN project, REC reference: 16/SC/0264 and Confidential Advisory Group reference 08/02/1394). The database includes EHR measurements of adult patients admitted to four hospitals from the Oxford University Hospitals NHS Foundation Trust. Note that the HAVEN dataset does not include data from Intensive Care Units (ICU), and we have excluded observations taken in the Emergency Department. Key characteristics of HAVEN cohort data include a) heterogeneity, b) multi-modality, and difference in: c) noise distributions, d) sampling rates, e) missing values, etc. Such properties are common across EHR settings, and are challenging with respect to learning useful representations and predictions.
We used the protocol defined in Pimentel et al. (2019) to subset the cohort to those patients at risk of developing Type-II Respiratory Failure (T2RF) in hospital (a diagram of the data selection steps can be found in Figure A.1 in the Appendix). Four patient outcomes were considered in our analysis: i) no event during hospital stay, leading to successful discharge from the hospital, or the first instance of one of three possible events, ii) unplanned entry to ICU, iii) cardiac arrest (also named ”Cardiac” hereafter) and iv) death. Outcome groups are not clearly separable (see Tables A.2, A.3 in the Appendix), so patient clusters will naturally contain a mix of different admission outcomes. In this setting, the clinically relevant component of a cluster phenotype (henceforth denoted as cluster outcome propensity or cluster outcome) is represented as a categorical distribution indicating the corresponding propensity for cluster-assigned patients to each corresponding outcome.
For each admission, observations were grouped according to mean window observation value into 4 hour blocks based on the time to outcome (discharge in the case of no event during stay) - only observations within 24 and 72 hours before the outcome were considered. This time window was selected based on those traditionally used for validating Early Warning Score (EWS) systems (baseline models used by UK NHS staff to track inpatient physiology, (Royal College of Physicians, 2017) and clinical input, such that the target phenotype represents the patient status in the subsequent 24 hours. Features were transformed according to min-max normalization due to skewness and heterogeneity in their distributions. Patient admissions were randomly split into train, validation and test sets. Missing values were imputed based on the previously observed time block - all remaining missing observations were imputed according to the feature median from the aggregated validation and test data (see Section 4 for description of train-test data split). Imputed values were flagged in a three-dimensional mask matrix.
After processing, input data contained over 100,000 unique patient trajectories corresponding to 4,266 unique patient admissions (only last patients admissions were considered in our analysis). Original trajectories for the patient cohort are shown in the Appendix in Figures A.4, A.5, A.6 for different variables/features. A lack of clear outcome group separability can be observed across
temporal and static variables. Furthermore, we note the high degree of imbalance in the data - admissions with no event account for over 86.8% of the total number of admissions, while event classes correspond to 10.3% Death, 1.8% ICU and 1.1% Cardiac.

heading: 3.2 PROPOSED MODEL
text: We propose a novel model, which we denote by Cluster-bAsed iMportancE Learning fOr Timeseries (CAMELOT). Our proposed methodology is displayed in Figure 11. Our model builds on previous literature on 3 key items: a) a modified loss function to target the multi-class imbalance, b) a novel loss function to ensure cluster assignment and phenotype are representative, and c) a novel feature-time attention-level framework to boost representation and introduce feature-time interpretability for cluster assignment.
Let N denote the number of patients and Df the number of input features. Input data consists of a set of patient trajectories X = {{xn,t}Tnt=1}Nn=1, where Tn is the maximum number of temporal observations for patient n, and a set of patient outcomes Y = {yn}Nn=1. Input trajectory data for the n-th patient is represented as Xn = [xn,1, ...,xn,Tn ], where each xn,t ∈ RDf is referred to as an observation (vector), with a maximum of observed Df feature values. The corresponding patient outcome is a one-hot encoded vector yn ∈ R4 (more generally, the dimension of yn equals to the number of possible outcomes).
Our DL model can be decomposed into 3 neural network blocks: an Encoder, Identifier and Predictor. We refer the action of each network respectively as E, I and P (for example, I(x) denotes the output of the Identifier given some input vector x). Both the Identifier and Predictor are Multilayer Perceptrons (MLP), networks of stacked feed-forward dense layers. On the other hand, the Encoder block can be further sub-divided into a) a stack of RNN layers and b) our proposed custom attention layer (see Section 3.3 for further details). Separately, we also consider a set of trainable cluster representation vectors, C = {c1, ..., cK}. We assign the outcome for cluster i as P (ci). A model call is as follows: Given the n-th patient input trajectory data Xn, the Encoder network returns a latent representation zn := E(Xn) ∈ Rl. Consequently, the Identifier network computes cluster assignment probabilities, πn := I(zn) ∈ RK . Each element of πn, πin, represents the probability assignment of zn to cluster i, given a total of K clusters. A cluster, knsamp is selected according to categorical sampling (knsamp ∼ Cat(πn)), and the corresponding cluster representation, cnsamp := cknsamp is then selected from C. The output of the model is ypred := P (c n samp) ∈ R4. We note that knsamp is only sampled during a training phase; at prediction stage, cluster selection follows the equation knpred = argmax
i=1,...,K πin.

heading: 3.3 ENCODER NETWORK AND A CUSTOM ATTENTION LAYER
text: The diagram of our proposed Encoder network is presented in Figure 2. The Encoder contains (i) a Recurrent Neural Network (RNN) block of stacked Long Short-Term Memory (LSTM) layers, and (ii) a customised attention layer, which computes a latent representation by comparing input data with the sequence of output states from the RNN block. We use the same notation as above,
1code will be shared online after review
and write the sequence of output states of the final LSTM layer as on,1, ...,on,Tn , with on,i ∈ Rl. Theoretically, each on,t corresponds to a representative summary of input patient information up until time t. We propose to approximate on,t as a linear combination of latent representations of each individual feature, thereby allowing the separation of output states into contributions from each feature. Note that it is important feature transformations be time-independent, to avoid overparametrising the model, over-fitting and ensure feature representation maps are similar across time.
Our attention layer behaves as a set of Df feed-forward neural network layers, U1, ...,UDf , jointly represented by: (i) a matrix of learnable kernel weights D ∈ Rl×Df . We write D = [D1, ...,DDf ]; (ii) a matrix of learnable bias vectors B ∈ Rl×Df . Similarly, we can write B = [B1, ...,BDf ]; and (iii) an activation function, σ which matches the output activation of the RNN block.
Input data for patient n, Xn is fed as input to the RNN block, which outputs a sequence of latent output states (on,t)Tnt=1. For t = 1, ..., Tn, we compute Df feature representations in latent space as:
Rn,t := σ(D ⊙ xn,t +B) (1)
where Rn,t = [R1n,t, ...,R Df n,t ] is our collection of feature representations, σ is applied element-wise and A := D ⊙ xn,t is a matrix satisfying Ai,j = Di,j(xn,t)j . Equivalently, Rin,t is the output of a dense layer, Ui with kernel Di, bias Bi, activation σ and input (xn,t)i. We approximate on,t ≈∑Df
i=1 α i tR i n,t = Rn,tαt. This approximation is minimised following a least squares criterion,
which has a well-known solution, α̂t, and corresponding optimal approximation ôn,t. Given, ôn,t, a similar procedure is used to compute a context vector as z := ∑
t βtôn,t, where weights β are learned to provide a more representative context vector.

heading: 3.4 ATTENTION MAP VISUALISATION
text: Given cluster representation vectors, ck, we can compute a cluster-wise feature-time attention visualisation map as follows. First, we normalise feature-weights α̂t according to a softmax function, st = σ(α̂t) ∈ RDf , where σ is the softmax function: σ(x) = exp |x|∥ exp |x|∥1 . Secondly, we compute cluster-wise weights, γkt according to a least-square approximation of ck ≈ ∑Tn t=1 ôn,tγ k n,t, and solved as before. We similarly normalise γkn,t to obtain cluster temporal scores, e k n,t = σ(γ k n,t). Finally, we can compute K scoring matrices, M1n, ...,M K n ∈ RTn×Df : ( Mkn ) t,f = ekn,ts f t . Note
that: ∥Mkn∥1 = ∑ t e k n,t ∑ f s f t = ∑ t e k n,t = 1. Given that Matrices M k n are normalised, they may be consequently, visualised as a normalised feature-time map for cluster assignment relevance and provide further model interpretability.

heading: 3.5 LOSS OPTIMISATION
text: The model is optimised through consideration of three distinct loss functions. We introduce a weighted cross-entropy loss function:
Lpred(ytrue, ypred) = − C∑
c=1
wcy c true log ( ycpred ) = −wc′ log(ypred)c′ (2)
where c′ is the true outcome for a particular patient. We propose inversely proportional normalised weights: ∑C c=1 wc = 1 and wc is inversely proportional to the class distribution, i.e., wc ∝ N Nc
with N being the number of patients and Nc being the number of patients with outcome label c. Class weighting penalises misclassification more heavily on less sampled classes.
We also propose a novel distribution loss function, Ldist(π). We define the average cluster probability of assignment as πC := 1N ∑ n πn. Then we introduce Ldist(π) = −H(πC), where H denotes entropy. Note that Ldist is minimised when πC is uniform, ensuring all clusters are ’explored’ and have comparable number of samples. While clusters should not necessarily be assigned the same number of samples, this loss helps to overcome a concern of cluster collapse, where clusters do not separate and samples are assigned to a very small number of non-representative clusters. Finally, to separate cluster representation vectors, we define the cluster separation loss as Lclus(C) = − 1K(K−1) ∑ i,j ∥ci − cj∥2
To optimise our model, iterative gradients are applied according to weighted combinations of the above loss functions with hyper-parameter weights α, β:
1. Outcome Predictor is updated according to Lpred;
2. Encoder and Identifier are trained according to Lpred + αLdist;
3. Cluster representation vectors are updated with regards to Lpred + βLclus.

heading: 3.6 INITIALISATION
text: Our proposed model also follows a set of initialisation pre-training procedures. Firstly, the Encoder and Outcome Predictor are pre-trained according to a classification task (ỹ = P (E(x))), with corresponding loss Lpred. Latent state representations E(x) are clustered through a K-means algorithm with K clusters across the whole training set. Cluster representation vectors are initialised as given by the resulting cluster centroids, and finally the Cluster Identifier network is pre-trained to identify clusters as predicted by the K-Means algorithm with categorical cross-entropy loss. Implementation was completed in Python, with TensorFlow 2, scikit-learn and NumPy. All experiments were run with 1 Tesla v100 GPU, and 8 CPUs Intel(R) Xeon(R) Gold 6246 @ 3.30GHz.

heading: 4 RESULTS
text: For Benchmark purposes, we considered TSKM as a classic clustering benchmark, and SOM-VAE and AC-TPC as state-of-the-art phenotypic clustering methods. AC-TPC considers temporal subsequences of a complete patient set of observations - for comparison purposes, we consider only the model output for the complete patient sequence. For simplicity, we present results with all input features considered.
All models were trained on the same training set (60% of the complete input data) and evaluated against the same test set (remaining 40%). For DL models, we further split the training set into a purely training and validation sets. All experiments with varying hyper-parameters were repeated 10 times with a fixed set of 10 distinct seeds, and results are reported according to average metric and standard deviation. A complete list of the hyper-parameters considered for each model is included in Table A7 in the Appendix. In bold, top-performing hyper-parameters are indicated. Optimal integer hyper-parameters (K, l) were selected according to an “Occam’s Razor” approach - for each parameter, we assign it the highest value such that increasing this amount does not lead to a significant increase in performance according to mean AUROC and a Friedman’s hypothesis test. Neural network size parameters were kept consistent across all DL models where applicable. All
other optimal hyper-parameters were selected according to highest AUROC performance conditional on the model predicting each class (e.g. not making predictions solely for no event or Death Events).
On top of performance evaluation with regards to clustering separability and outcome prediction, we display a comparison between the learnt cluster phenotypes of the proposed model and that of phenotypic clustering benchmark AC-TPC. For each cluster, the corresponding outcome propensity P (c) is shown as a bar plot over the 4 possible outcomes with corresponding probability value. We also display cluster outcome propensity plots for both TSKM and SOM-VAE in the Appendix (Figures A.8 and A.9). We note that the cluster outcome propensity distributions learnt by CAMELOT also align with the empirical outcome relevance in the learnt clusters (Table A.12). Lastly, we also display feature-time cluster relevance attention maps in Figure 6. For each cluster, a patient was randomly selected from the set of patients in the corresponding cluster, and a corresponding featuretime attention matrix and visualised as a heatmap.

heading: 5 DISCUSSION
text: Our proposed model shows an improvement in clustering performance ( see Table 3) when compared to the current phenotypic clustering benchmark (AC-TPC), and outperforms SOM-VAE according
to VRI. Although the cluster separability metrics are superior in the case of TSKM, this is expected given a metric bias towards convex clusters, and the convexity of the K-Means based algorithm. In particular, DL clustering occurs in a latent space, which, unfortunately, is not easily comparable with an algorithm targetting the input space (such as K-Means). Furthermore, we argue clusters learnt by TSKM are less relevant that our model’s as a) TSKM clusters are extremely hard to distinguish with regards to outcome propensity (as evidenced by very low performance on a prediction task (Table 4)), and b) TSKM clusters are less separable with regards to trajectory evolution, as there is less separation of mean HR trajectories, and less cluster separation when data is projected to a two-dimensional domain with t-stochastic neighbour embedding (tSNE) - both figures are in the Appendix, Figures A10 and A11.
With regards to predictive power, it can be seen in Table 4 that our model outperforms both standard classifiers (at least 8%) and benchmarking clustering methods (at least 5%) according to mean AUROC. A similar increase can be seen in other classification metrics, with the exception of F1-score, where model performance is slightly below to that of AC-TPC. Our model is able to more accurately determine patterns in the data than the previously proposed models, as EHR data is extremely complex and heterogeneous. It is particularly promising that the model obtains good predictive task results despite a clustering bottleneck (i.e. sample predicted outcomes are done through the assigned cluster, as opposed to tailored to the precise input data). While it is possible that other models could show better performance on the direct task of outcome prediction given EHR input, such models can potentially be associated with lack of robustness or input sensitivity difficulties. Furthermore, it
is likely they would struggle with identifying relevant trends and properties of clinical interest. As such, for new admissions, such models could provide a prediction for the overall outcome, but no robust understanding of how this outcome will occur, and how to prevent potential risks of deterioration, let alone the ability to pool data from other similar patients.

heading: 6 CONCLUSION AND FUTURE WORK
text: In this work, we propose a novel deep learning model for the task of identification of phenotypically separable clusters applied to EHR data for. As part of our proposed model, we propose 2 distinct loss functions and introduce a novel feature-time attention layer to better represent patient data and to introduce a feature-time relevance map for each cluster. Our experiments show promising results with the addition of both methodological tools above, on both cluster separability and outcome prediction performance. The addition of the feature-time layer has the added benefit of introducing key interpretability tools for researchers to understand relevant regions for good patient physiology representation as well as an indication of what can lead to patient deterioration.
There are multiple interesting avenues of investigation building on this work. On the other hand, the current attention layer mechanisms could potentially be improved with the addition of temporal weight smoothness, or, alternatively, weight regularization to encourage exploration of the complete feature-time space. Alternatively, cluster selection through a neural network mechanism introduces high capacity at the potential cost of robustness and cluster collapse. Potentially, more traditional methods incorporated into a similarly complex pipeline can achieve better performing through a clearer identification of cluster regions in latent space. Furthermore, methodological improvements will also benefit from a more extensive testing across other diverse datasets and other potential areas of application.

heading: A APPENDIX
text: A.1 DATA
A description of the complete pipeline of data re-processing, following the protocol defined in Pimentel et al. (2019), is shown in Figure A.1.
A.1: HAVEN processing
A total of 26 input features were considered. Firstly, 4-hourly vital-sign sets which included 8 features: Heart Rate (HR), Respiratory Rate (RR), Systolic Blood Pressure (SBP), Diastolic Blood Pressure (DBP), peripheral Oxygen Saturation (SpO2), Temperature (TEMP), level of consciousness via the AVPU scale - Alert, Verbal, Pain, Unresponsive - and estimated Fraction of Inspired Oxygen (FiO2, available when an oxygen mask is applied to the patient). Each set consisted of a timestamp and the vital-sign numerical values. Secondly, 4 demographic variables were selected (modelled as static variables): age, sex, and admission type (elective or surgical). Thirdly, we included 6 features resulting from biochemistry blood tests, denoted as ’Serum’: Serum levels of urea, albumin, creatinine, sodium, potassium and C-reactive protein. Finally, 8 haematological blood test features were also included: white and haemaglobin cell counts, concentration of eosinophils, basophils, neutrophils, and lymphocytes, as well as eosinophil-to-basophil and neutrophil-to-lymphocyte ratios. These features were selected based on domain knowledge of features related to severity in the prognosis and outcome of T2RF inpatients.
Descriptive statistics for all input variables is described in Table A.2. Median and inter-quartile range (IQR) is displayed for continuous and categorical variables, while binary variables are shown according to number of counts in the dataset and corresponding cohort proportion. Statistics are displayed for the complete data (”All”), but also for each sub-cohort defined by the overall outcome. We can observe that these sub-cohorts are not clearly separable and are hard to identify solely from this information.
A summary of the patient cohort in relation to outcomes and target phenotypes can be seen in
Table A.3. Challenges with regards to obtaining phenotypically separable clusters can similarly be observed - there is no clear significant difference between the target outcome sub-cohorts with regards to demographic input variables. With regards to outcome distribution, we also note the high degree of imbalance in the dataset - the large majority of the patients in our dataset suffered from no adverse events (over 86%), while only 48 had a Cardiac event, and 76 were re-directed to the ICU.
The lack of outcome sub-cohort separability can further observed in a temporal domain. Figures A.4, A.5, A.6 plot the mean trajectories of different temporal variables sets for each outcome subcohort, respectively, according to vital signs, haematological and serum features. Mean is calculated based on the time to outcome, and missing observations are disregarded and ignored.
A.2: Descriptive statistics and information of all input data features. Variables are displayed with type, description, units and average statistics. We separate all features according to medical literature, including vital-sign, static, serum and haematological variables, and we also display statistics per outcome sub-cohort, defined as a cohort with those patients assigned to a given outcome.
No Event Death ICU Cardiac
N 3701 441 76 48 Age (IQR) 71 (61 - 80) 81 (74 - 88) 69 (61 - 74) 76 (69 - 82) Gender, M 1810 (48.9%) 247 (56.0%) 38 (50.0%) 28 (58.33%) CCI (IQR) 4 (3 - 13) 14 (4 - 21) 7 (4 - 17) 15 (4 - 23)
Elective 1126 (30.4%) 3 (0.7%) 8 (10.5%) 2 (4.2%) Surgical 1054 (28.5%) 48 (10.1%) 22 (29.0%) 6 (12.5%)
A.3: Descriptive demographic variable information for each outcome sub-cohort.
A.4: Plot of mean vital-sign trajectories (median with respect to SpO2) in solid line as given by the 4 outcome groups: admissions with a) Cardiac-, b) Death- , or c) ICU-, and d) No-events. The respective standard errors are represented by the dashed lines. We visualised trajectories from up to 7 days prior to an outcome event or discharge - the black lines represent the time window (72 - 24 hours prior to an event or discharge) considered for input to all models.
A.2 MODEL TRAINING
A list indicating the grid-search range of hyper-parameters considered in our experiments are indicated in Table A.7. For simplicity, we define P := {0.001, 0.01, 0.1, 1, 10}, L := {32, 64, 128, 256} and K := {3, ..., 20}. In bold, top-performing hyper-parameters according to target metrics defined in Section 4 are highlighted.
A.3 RESULTS COMPARISON
In Figures A.8 and Figures A.9 we display cluster outcome propensity distributions for some of our experiments with benchmark clustering models SOM-VAE and TSKM, respectively. Both models do not naturally associate clusters with a distribution - we estimate the cluster outcome as the empirical outcome distribution for the patient cohort assigned to the corresponding cluster.
A.5: Plot of mean haematological trajectories in solid line as given by the 4 outcome groups: admissions with a) Cardiac-, b) Death- , or c) ICU-, and d) No-events. The respective standard errors are represented by the dashed lines. We visualised trajectories from up to 7 days prior to an outcome event or discharge - the black lines represent the time window (72 - 24 hours prior to an event or discharge) considered for input to all models.
A.6: Plot of mean serum trajectories in solid line as given by the 4 outcome groups: admissions with a) Cardiac, b) Death- , or c) ICU-, and d) No-events. The respective standard errors are represented by the dashed lines. We visualised trajectories from up to 7 days prior to an outcome event or discharge - the black lines represent the time window (72 - 24 hours prior to an event or discharge) considered for input to all models.
We note that clusters learnt by both clustering benchmark models have identical outcomes, which provides no useful clinical interpretability to the cluster-defined populations, as well as likely not assisting models to learn relevant cluster representations.
A.8: Bar plots of learnt cluster phenotypes for SOM-VAE with optimal hyper-parameters. Each plot represents a cluster - its phenotype is the corresponding empirical outcome distribution in its cluster-assigned patient cohort.
We go further in comparing clusters learnt by TSKM and by our proposed model. We argue clusters learnt by CAMELOT are much more relevant towards our goal. We show this through two distinct plots. Firstly, in Figure A10, we display a scatter plot of patients in each cluster (CAMELOT on the right and TSKM clusters on the left) after projection to two dimensions. Projection was completed through a principal component analysis reduction to 50 dimensions, followed by t-stochastic neighbour embedding dimensionality projection to two.
Furthermore, we also demonstrated that TSKM does not learn as separable cluster trajectory evolution profiles as CAMELOT. This is shown in Figure A11, where Heart-Rate mean trajectories for each cluster (i.e., average HR observations aligned to the same time until end of observations for patients in the clusters) are displayed. It is clear that CAMELOT cluster trajectories are easier to separate.
A complete description of the number of patient admissions with a given outcome per learnt cluster in our proposed model can be seen in Table A.12.
Parameter TSKM SOM-VAE AC-TPC CAMELOT SVM XGB
seeds {1001, 1012, 1134, 2475, 6138, 7415, 1663, 7205, 9253, 1782} α - P (0.1) P (0.01) P (0.01) - - β - P (0.1) P (0.01) P (0.001) - - γ - - - - - {0.1, 0.2, 0.6}
latent dim - L (64) L (128) L (128) - - SOM dim - L2(4, 4) - - - -
K K(7) - K (6) K (6) - - kernel {’DTW’, ’euclidean’} - - - {’polynomial’, ’rbf’} -
C - - - - P (10) - n-estimators - - - - - {100, 200, 300}
depth - - - - - {1, 3, 5, 10} min-child-weight - - - - - {1, 2, 3, 5}
A.7: Parameter range used for Grid-search hyper-parameter optimisation. For each model, the list of parameter values tested is indicated. In bold, the optimum set of hyper-parameters is indicated for each model.
A.9: Bar plots of learnt cluster phenotypes for TSKM with K = 6. Each plot represents a cluster - its phenotype is the corresponding empirical outcome distribution in its cluster-assigned patient cohort.
A.10: Scatter plot of cluster patient data after projection to 2 dimensions.
Outcome Healthy Death ICU Cardiac Cluster 0 149 44 10 7 Cluster 1 739 28 5 5 Cluster 2 579 15 6 2 Cluster 3 93 92 12 5 Cluster 4 373 2 2 0 Cluster 5 288 84 10 10
A.12: Table with empirical number of outcome admissions observed for each cluster learnt by the proposed model.
A.11: Plot of mean Heart-Rate (HR) trajectory in solid line as given by the TSKM learnt clusters (top) and CAMELOT (bottom). The respective standard errors are represented by the dashed lines. We visualised trajectories from up to 7 days prior to an outcome event or discharge - the black lines represent the time window (72 - 24 hours prior to an event or discharge) considered for input to all models.
We also computed summary statistics for the learnt CAMELOT clusters. For each of the resulting clusters, median, and quartile values were computed and plotted, except on the case of binary variables, where only the number of positive occurrences (and the corresponding proportion in the cluster) are shown.
Lastly, we plot the mean cluster trajectory evolution for SBP and FIO2 to present supportive evidence for the personalised attention maps in Figure 6. These two features were selected from attention map analysis.
A.13: Descriptive statistics and information of all input data features. Variables are displayed with type, description, units and average statistics. We separate all features according to medical literature, including vitalsign, static, serum and haematological variables. Statistics are shown for each cohort as learnt by our model.
A.14: Plot of mean Systolic Blood Pressure (SBP) trajectories in solid line as given by the clusters learnt by our model (top). In the bottom, mean FIO2 trajectories are displayed. The respective standard errors are represented by the dashed lines. We visualised trajectories from up to 7 days prior to an outcome event or discharge - the black lines represent the time window (72 - 24 hours prior to an event or discharge) considered for input to all models.

Review summaries:
The paper presents a way to predict outcomes of patients using EHR records. The authors propose a phenotyping model which clusters similar patients based on their outcome distribution. They also introduce feature-time relevance map which helps explain for a particular patient the feature-time combinations which are important for their prediction.