Cluster-based Feature Importance Learning for Electronic Health Record Time-series

1 INTRODUCTION . Chronic conditions such as Chronic Obstructive Pulmonary Disease ( COPD ) and Cardiovascular Disease ( CVD ) describe a broad spectrum of medical ailments , and affect a significant percentage of the overall population ( Adeloye et al. , 2015 ) . Such diseases are characterized by the existence of multiple distinct patient subgroups , largely distinguished by differences in pathology and in the response to different treatments and medical interventions Turner et al . ( 2015 ) ; Vogelmeier et al . ( 2018 ) . Exacerbation of COPD , a condition of respiratory failure , can result in emergency hospital admission and mortality if it is not well treated and managed . Early identification of COPD patients ’ subgroups is therefore of high medical importance and relevance . EHR time-series data are typically used to determine clinically relevant COPD inpatient subgroups , and have been applied to detect risk of deterioration ( Pikoula et al. , 2019 ) . However , modelling disease progression and risk prediction is challenging due to the extreme data heterogeneity nature of EHRs . Firstly , EHR data contains a mixture of demographic or static variables ( i.e . time independent such as age and sex ) , and multi-dimensional time-series ( e.g Heart Rate , HR , and laboratory measurements , such as blood tests ) . Secondly , EHR time-series are multi-modal as different features are collected from different devices , representing distinct clinical properties of relevance . Similarly , time-series features are sampled at different times and have low and distinct sampling rates , as well as different missing value properties . Furthermore , each feature is associated with different noise and evolution patterns . Recent advances in deep learning ( DL ) approaches have shown promising results in EHR modelling due to their capacity to handle complex data ( Rajkomar et al. , 2018 ) . Nonetheless , DL approaches lack relevant interpretability frameworks to be scaled and applied in hospital settings . Several such models have since been proposed to tackle this issue ( Mayhew et al. , 2018 ) , however , most of them focus on a subset of EHR features ( usually vital signs only ) and fail to provide a clinically-focused phenotypic analysis of learnt patient sub-groups ( via clustering ) . This work builds on previous research in literature to introduce a cluster-based feature-time attention mechanism to predict patient outcomes based on EHR data . Our method also leverages phenotypic information to aid in clinical interpretability , not only making use of demographics and vital-signs information but also of relevant laboratory measurements ( all present in the EHR ) to provide a more complete patient physiological status . Our contributions include the following : • An end-to-end DL supervised model to cluster EHR patient data based on the identification of clinically understandable cluster phenotypes with regard to both outcome prediction and patient trajectory in a multi-class setting ; • A weighted loss to address data imbalance for both tasks of clustering and prediction , a common issue in the medical domain ; • The incorporation of a novel loss mechanism in the model , to address the issue of cluster collapse and promote sample assignment to all available clusters ; • Finally , the inclusion of a novel interpretability framework , derived from a cluster-based feature-time attention layer , aiming to identify relevant timestamps and feature variables to represent the patient physiology , cluster assignment and , ultimately , outcome prediction . This paper is structured as follows . In Section 2 , we describe previous research in EHR time-series modelling , clustering and attention methods . Section 3 introduces the dataset used for analysis and provides description of the proposed model . The experimental setup and results of our analysis are presented in Section 4 and discussion takes place in Section 5 . Finally , concluding remarks and future work are available in Section 6 . 2 RELATED WORK . EHR data comprise complex time-series data , being high-dimensional , multi-modal and heterogeneous , and thus presenting challenges when used in machine learning models ( Keogh & Kasetty ( 2003 ) ; Rani & Sikka ( 2012 ) ) . An important goal in a medical setting is to identify phenotypically separable clusters with distinct phenotypic profiles ( which we denote as phenotypic clustering hereafter ) . For the purpose of this work , cluster phenotypes result from the combination of two distinct components : a ) the evolution profile of patient trajectories ’ within the cluster , and b ) the characterisation of the cluster with regards to clinical variables of interest . The latter may include features not used for clustering and may provide information about the underlying or future health status . Traditional clustering models such as K-Means or hierarchical clustering have been shown to fail to capture the existing time-dependent feature relationships . As such , variants have been proposed to mitigate this problem . A temporal version of the K-Means algorithm , Time-Series K-Means ( TSKM , Tavenard et al . ( 2020 ) ) , models the distance between time-series of different datapoints , using the Euclidean distance ( which is equivalent to considering all temporal observations as an independent feature value for the corresponding patient admission ) , or time-series alignment strategies such as Dynamic-Time Warping ( DTW , Berndt & Clifford ( 1994 ) ) and soft-DTW ( Cuturi & Blondel ( 2017 ) ) . Recent DL architectures , ranging from Auto-Encoders ( AE , Ma et al . ( 2019 ) ) , Convolutional Neural Networks ( CNN , Munir et al . ( 2018 ) ) and others , have shown great promise when applied to timeseries data across a variety of domains . Fortuin et al . ( 2019 ) proposed a Self-Organising Map - Variational Auto-Encoder ( SOM-VAE ) , is a state-of-the-art , unsupervised , DL clustering algorithm which extends a variational auto-encoder architecture ( Kingma & Welling , 2013 ) for observation learning and representation , through the addition of a Markov model ( Gagniuc , 2017 ) , to infer temporal evolution within the latent space . Clustering is performed in the low dimensional latent space through the use of self-organising maps ( Kohonen , 1982 ) to obtain a discrete , topologically-interpretable latent representation of the learnt clusters . In a supervised setting , AC-TPC ( Lee & Van Der Schaar , 2020 ) serves as the current state-of-the-art for identifying phenotypically separable clusters in patient trajectories in EHR data . AC-TPC maps EHR data into a latent space via an encoder , and uses an actor-critic network ( Konda & Tsitsiklis , 2000 ) which leverages clinical outcomes to aid in cluster formation and obtaining cluster phenotypes . Neither SOM-VAE and AC-TPC provide clinically meaningful interpretation of feature-time importance or outcome of interest . Attention mechanisms have recently been proposed to provide greater interpretability to Recurrent Neural Networks ( RNN ) and to aid in dealing with long-term dependencies ( Vaswani et al. , 2017 ; Xu et al. , 2015 ) , and have also been used in modelling EHR time-series ( Schwab et al. , 2017 ; Shashikumar et al. , 2018 ) . RETAIN ( Choi et al. , 2016 ) proposes a two-level reverse attention mechanism to mimic physician ’ s decision process and predict a future diagnosis . In other recent works , attention mechanisms based on bi-directional RNN and CNN outperformed standard classification models in predicting high risk vascular diseases with the addition of medication information as input data ( Kim et al. , 2017 ) . A drawback of such attention mechanisms is the focus on temporal interpretability only , and inability to look at individual features , which is key in a medical setting . To solve this issue , Shamout et al . ( 2019 ) considered independent RNN per feature , with a concatenation of the resulting latent vectors . However , the latter does not allow the joint modelling across both feature and time dimensions . Alternatively , Kaji et al . ( 2019 ) ; Gandin et al . ( 2021 ) proposed learning attention weights directly on the original inputs , prior to being transformed by a RNN , which does not allow modelling of the resulting latent representations . To the best of our knowledge , no existing models have been proposed that jointly leverage both feature and time dimensions ( feature-time ) to determine clinical observation relevance on clustered EHR data . 3 METHODS . 3.1 DATASET AND PRE-PROCESSING . Our dataset was retrieved from a retrospective database of routinely collected observations from concluded hospital admissions between March 2014 and March 2018 ( the HAVEN project , REC reference : 16/SC/0264 and Confidential Advisory Group reference 08/02/1394 ) . The database includes EHR measurements of adult patients admitted to four hospitals from the Oxford University Hospitals NHS Foundation Trust . Note that the HAVEN dataset does not include data from Intensive Care Units ( ICU ) , and we have excluded observations taken in the Emergency Department . Key characteristics of HAVEN cohort data include a ) heterogeneity , b ) multi-modality , and difference in : c ) noise distributions , d ) sampling rates , e ) missing values , etc . Such properties are common across EHR settings , and are challenging with respect to learning useful representations and predictions . We used the protocol defined in Pimentel et al . ( 2019 ) to subset the cohort to those patients at risk of developing Type-II Respiratory Failure ( T2RF ) in hospital ( a diagram of the data selection steps can be found in Figure A.1 in the Appendix ) . Four patient outcomes were considered in our analysis : i ) no event during hospital stay , leading to successful discharge from the hospital , or the first instance of one of three possible events , ii ) unplanned entry to ICU , iii ) cardiac arrest ( also named ” Cardiac ” hereafter ) and iv ) death . Outcome groups are not clearly separable ( see Tables A.2 , A.3 in the Appendix ) , so patient clusters will naturally contain a mix of different admission outcomes . In this setting , the clinically relevant component of a cluster phenotype ( henceforth denoted as cluster outcome propensity or cluster outcome ) is represented as a categorical distribution indicating the corresponding propensity for cluster-assigned patients to each corresponding outcome . For each admission , observations were grouped according to mean window observation value into 4 hour blocks based on the time to outcome ( discharge in the case of no event during stay ) - only observations within 24 and 72 hours before the outcome were considered . This time window was selected based on those traditionally used for validating Early Warning Score ( EWS ) systems ( baseline models used by UK NHS staff to track inpatient physiology , ( Royal College of Physicians , 2017 ) and clinical input , such that the target phenotype represents the patient status in the subsequent 24 hours . Features were transformed according to min-max normalization due to skewness and heterogeneity in their distributions . Patient admissions were randomly split into train , validation and test sets . Missing values were imputed based on the previously observed time block - all remaining missing observations were imputed according to the feature median from the aggregated validation and test data ( see Section 4 for description of train-test data split ) . Imputed values were flagged in a three-dimensional mask matrix . After processing , input data contained over 100,000 unique patient trajectories corresponding to 4,266 unique patient admissions ( only last patients admissions were considered in our analysis ) . Original trajectories for the patient cohort are shown in the Appendix in Figures A.4 , A.5 , A.6 for different variables/features . A lack of clear outcome group separability can be observed across temporal and static variables . Furthermore , we note the high degree of imbalance in the data - admissions with no event account for over 86.8 % of the total number of admissions , while event classes correspond to 10.3 % Death , 1.8 % ICU and 1.1 % Cardiac .

Summary: The paper presents a way to predict outcomes of patients using EHR records. The authors propose a phenotyping model which clusters similar patients based on their outcome distribution. They also introduce feature-time relevance map which helps explain for a particular patient the feature-time combinations which are important for their prediction.