Actor-Critic Approach for Temporal Predictive Clustering

Due to the wider availability of modern electronic health records ( EHR ) , patient care data is often being stored in the form of time-series . Clustering such timeseries data is crucial for patient phenotyping , anticipating patients ’ prognoses by identifying “ similar ” patients , and designing treatment guidelines that are tailored to homogeneous patient subgroups . In this paper , we develop a deep learning approach for clustering time-series data , where each cluster comprises patients who share similar future outcomes of interest ( e.g. , adverse events , the onset of comorbidities , etc. ) . The clustering is carried out by using our novel loss functions that encourage each cluster to have homogeneous future outcomes . We adopt actorcritic models to allow “ back-propagation ” through the sampling process that is required for assigning clusters to time-series inputs . Experiments on two realworld datasets show that our model achieves superior clustering performance over state-of-the-art benchmarks and identifies meaningful clusters that can be translated into actionable information for clinical decision-making . 1 INTRODUCTION . Chronic diseases – such as cystic fibrosis , dementia , and diabetes – are heterogeneous in nature , with widely differing outcomes even in narrow patient subgroups . Disease progression manifests through a broad spectrum of clinical factors , collected as a sequence of measurements over time in electronic health records ( EHR ) , which gives a rise to complex progression patterns among patients ( Samal et al. , 2011 ) . For example , cystic fibrosis evolves slowly , allowing for the development of related comorbidities and bacterial infections , and creating distinct behaviors/responses to therapeutic interventions , which in turn makes the survival and quality of life substantially different ( Ramos et al. , 2017 ) . Identifying patient subgroups with similar progression patterns can be advantageous for understanding such heterogeneous underlying diseases . This allows clinicians to anticipate patients ’ prognoses by comparing “ similar ” patients in order to design treatment guidelines that are tailored to homogeneous patient subgroups ( Zhang et al. , 2019 ) . Temporal clustering has been recently used as a data-driven framework to partition patients with time-series observations into a set of clusters ( i.e. , into subgroups of patients ) . Recent research has typically focused on either finding fixed-length and low-dimensional representations ( Zhang et al. , 2019 ; Rusanov et al. , 2016 ) or on modifying the similarity measure ( Giannoula et al. , 2018 ; Luong and Chandola , 2017 ) both in an attempt to apply the conventional clustering algorithms ( e.g. , K-means ( Lloyd , 1982 ) ) to time-series observations . However , clusters identified from these approaches these approaches are purely unsupervised – they do not account for each patient ’ s observed outcome ( e.g. , adverse events , the onset of comorbidities , etc . ) – which leads to heterogeneous clusters if the clinical presentation of the disease differs even for similar patients . Thus , a common prognosis in each cluster remains unknown which can mystify the understanding of the underlying disease progression ( Boudier et al. , 2019 ) . For instance , patients who appear to have similar timeseries observations may develop different sets of comorbidities in the future which , in turn , require different treatment guidelines to reduce such risks ( Wami et al. , 2013 ) . To overcome this limitation , we focus on predictive clustering ( Blockeel et al. , 2017 ) which combines prediction with clustering . Therefore , the cluster assignments are optimized such that patients in a cluster share similar future outcomes to provide a prognostic value . In this paper , we propose an actor-critic approach for temporal predictive clustering , which we call AC-TPC.1 Our model consists of three neural networks – an encoder , a selector , and a predictor – and a set of centroid candidates . More specifically , the encoder maps an input time-series into a latent encoding ; the selector utilizes the encoding and assigns a cluster to which the time-series belongs to via a sampling process ; and the predictor estimates the future outcome distribution conditioned on either the encoding or the centroid of the selected cluster . The following three contributions render our model to identify the predictive clusters . First , to encourage each cluster to have homogeneous future outcomes , we define a clustering objective based on the Kullback-Leibler ( KL ) divergence between the predictor ’ s output given the input time series , and the predictor ’ s output given estimated cluster assignments . Second , we transform solving a combinatorial problem of identifying cluster into iteratively solving two sub-problems : optimization of the cluster assignments and optimization of the cluster centroids . Finally , we allow “ back-propagation ” through the sampling process of the selector by adopting the training of actor-critic models ( Konda and Tsitsiklis , 2000 ) . Throughout the experiments , we show significant performance improvements over the state-of-theart clustering methods on two real-world medical datasets . Then , to demonstrate the practical significance of our model , we consider a more realistic scenario where the future outcomes of interest are high-dimensional – such as , development of multiple comorbidities in the next year – and interpreting all possible combinations is intractable . Our experiments show that our model can identify meaningful clusters that can be translated into actionable information for clinical decision-making . 2 PROBLEM FORMULATION . Let X ∈ X and Y ∈ Y be random variables for an input feature and an output label ( i.e. , one or a combination of future outcome ( s ) of interest ) with a joint distribution pXY ( and marginal distributions are pX and pY , respectively ) where X is the feature space and Y is the label space . Here , we focus our description on C-class classification tasks , i.e. , Y = { 1 , · · · , C } .2 We are given a time-series dataset D = { ( xnt , ynt ) T n t=1 } Nn=1 comprising sequences of realizations ( i.e. , observations ) of the pair ( X , Y ) for N patients . Here , ( xnt , y n t ) Tn t=1 is a sequence of T n observation pairs that correspond to patient n and t ∈ T n , { 1 , · · · , Tn } denotes the time stamp at which the observations are made.From this point forward , we omit the dependency on n when it is clear in the context and denote x1 : t = ( x1 , · · · , xt ) for ease of notation . Our aim is to identify a set of K predictive clusters , C = { C ( 1 ) , · · · , C ( K ) } , for time-series data . Each cluster consists of homogeneous data samples , that can be represented by its centroid , based on a certain similarity measure . There are two main distinctions from the conventional notion of clustering . First , we treat subsequences of each times-series as data samples and focus on partitioning { { xn1 : t } T n t=1 } Nn=1 into C. Hence , we define a cluster as C ( k ) = { xn1 : t|t ∈ T n , snt = k } for k ∈ K , { 1 , · · · , K } where snt ∈ K is the cluster assignment for a given xn1 : t. This is to flexibly update the cluster assignment ( in real-time ) to which a patient belongs as new observations are being accrued over time . Second , we define the similarity measure with respect to the label distribution and associate it with clusters to provide a prognostic value . More specifically , we want the distribution of output label for subsequences in each cluster to be homogeneous and , thus , can be well-represented by the centroid of that cluster . Let S be a random variable for the cluster assignment – that depends on a given subsequence x1 : t – and Y |S = k be a random variable for the output given cluster k. Then , such property of predictive clustering can be achieved by minimizing the following Kullback-Leibler ( KL ) divergence : KL ( Yt|X1 : t = x1 : t ∥∥Yt|St = k ) for x1 : t ∈ C ( k ) ( 1 ) where KL ( Yt|X1 : t = x1 : t ∥∥Yt|St = k ) = ∫y∈Y p ( y|x1 : t ) ( log p ( y|x1 : t ) − log p ( y|st ) ) dy . Here , p ( y|x1 : t ) and p ( y|st ) are the label distributions conditioned on a subsequence x1 : t and a cluster assignment st , respectively . Note that ( 1 ) achieves its minimum when the two distributions are equivalent . 1Source code available at https : //github.com/ICLR2020-ACTPC/ACTPC_submission.git 2In this paper , we focus our description on C-class classification task , i.e. , Y = { 1 , · · · , C } ; in Appendix A , we discuss simple modifications of our model for regression and M -dimensional binary classification tasks , i.e. , Y = R and Y = { 0 , 1 } M , respectively . Finally , we establish our goal as identifying a set of predictive clusters C that optimizes the following objective : minimize C ∑ k∈K ∑ x1 : t∈C ( k ) KL ( Yt|X1 : t = x1 : t ∥∥Yt|St = k ) . ( 2 ) Unfortunately , the optimization problem in ( 2 ) is highly non-trivial . We need to estimate the objective function in ( 2 ) while solving a non-convex combinatorial problem of finding the optimal cluster assignments and cluster centroids . 3 ACTOR-CRITIC APPROACH FOR TEMPORAL PREDICTIVE CLUSTERING . To effectively estimate the objective function in ( 2 ) , we introduce three networks – an encoder , a selector , and a predictor – and an embedding dictionary as illustrated in Figure 1 . These components together provide the cluster assignment and the corresponding centroid based on a given sequence of observations and enable us to estimate the probability density p ( y|st ) . More specifically , we define each component as follows : • The encoder , fθ : ∏t i=1 X → Z , is a RNN ( parameterized by θ ) that maps a ( sub ) sequence of a time-series x1 : t to a latent representation ( i.e. , encoding ) zt ∈ Z where Z is the latent space . • The selector , hψ : Z → ∆K−1 , is a fully-connected network ( parameterized by ψ ) that pro- vides a probabilistic mapping to a categorical distribution from which the cluster assignment st ∈ K is being sampled . • The predictor , gφ : Z → ∆C−1 , is a fully-connected network ( parameterized by φ ) that esti- mates the label distribution given the encoding of a time-series or the centroid of a cluster . • The embedding dictionary , E = { e ( 1 ) , · · · , e ( K ) } where e ( k ) ∈ Z for k ∈ K , is a set of cluster centroids lying in the latent space which represents the corresponding cluster . Here , ∆D−1 = { q ∈ [ 0 , 1 ] K : q1 + · · ·+ qD = 1 } is a ( D− 1 ) -simplex that denotes the probability distribution for a D-dimensional categorical ( class ) variable . At each time stamp t , the encoder maps a input ( sub ) sequence x1 : t into a latent encoding zt , fθ ( x1 : t ) . Then , based on the encoding zt , the cluster assignment st is drawn from a categorical distribution that is defined by the selector output , i.e. , st ∼ Cat ( πt ) where πt = [ πt ( 1 ) , · · · , πt ( K ) ] , hψ ( zt ) . Once the assignment st is chosen , we allocate the latent encoding zt to an embedding e ( st ) in the embedding dictionary E . Since the allocated embedding e ( st ) corresponds to the centroid of the cluster to which x1 : t belongs , we can , finally , estimate the density p ( y|st ) in ( 2 ) as the output of the predictor given the embedding e ( st ) , i.e. , ȳt , gφ ( e ( st ) ) .

Summary: This work proposes a method to perform clustering on a time-series data for prediction purposes, unlike the classical clustering where it is done in an unsupervised manner. The authors use an encoder (RNN) to process the time-series medical records, a selector to sample the cluster label for each encoding, and a predictor to predict the labels based on the selected cluster centroids. Since the sampling process prevents the authors from using back-prop, they employ an actor-critic method.