Title: ACTOR-CRITIC APPROACH FOR TEMPORAL PREDIC-TIVE CLUSTERING

Abstract: Due to the wider availability of modern electronic health records (EHR), patient care data is often being stored in the form of time-series. Clustering such timeseries data is crucial for patient phenotyping, anticipating patients' prognoses by identifying "similar" patients, and designing treatment guidelines that are tailored to homogeneous patient subgroups. In this paper, we develop a deep learning approach for clustering time-series data, where each cluster comprises patients who share similar future outcomes of interest (e.g., adverse events, the onset of comorbidities, etc.). The clustering is carried out by using our novel loss functions that encourage each cluster to have homogeneous future outcomes. We adopt actorcritic models to allow "back-propagation" through the sampling process that is required for assigning clusters to time-series inputs. Experiments on two realworld datasets show that our model achieves superior clustering performance over state-of-the-art benchmarks and identifies meaningful clusters that can be translated into actionable information for clinical decision-making.

1. INTRODUCTION:

Chronic diseases -such as cystic fibrosis, dementia, and diabetes -are heterogeneous in nature, with widely differing outcomes even in narrow patient subgroups. Disease progression manifests through a broad spectrum of clinical factors, collected as a sequence of measurements over time in electronic health records (EHR), which gives a rise to complex progression patterns among patients (Samal et al., 2011) . For example, cystic fibrosis evolves slowly, allowing for the development of related comorbidities and bacterial infections, and creating distinct behaviors/responses to therapeutic interventions, which in turn makes the survival and quality of life substantially different (Ramos et al., 2017) . Identifying patient subgroups with similar progression patterns can be advantageous for understanding such heterogeneous underlying diseases. This allows clinicians to anticipate patients' prognoses by comparing "similar" patients in order to design treatment guidelines that are tailored to homogeneous patient subgroups (Zhang et al., 2019) .

Temporal clustering has been recently used as a data-driven framework to partition patients with time-series observations into a set of clusters (i.e., into subgroups of patients). Recent research has typically focused on either finding fixed-length and low-dimensional representations (Zhang et al., 2019; Rusanov et al., 2016) or on modifying the similarity measure (Giannoula et al., 2018; Luong and Chandola, 2017) both in an attempt to apply the conventional clustering algorithms (e.g., K-means (Lloyd, 1982) ) to time-series observations. However, clusters identified from these approaches these approaches are purely unsupervised -they do not account for each patient's observed outcome (e.g., adverse events, the onset of comorbidities, etc.) -which leads to heterogeneous clusters if the clinical presentation of the disease differs even for similar patients. Thus, a common prognosis in each cluster remains unknown which can mystify the understanding of the underlying disease progression (Boudier et al., 2019) . For instance, patients who appear to have similar timeseries observations may develop different sets of comorbidities in the future which, in turn, require different treatment guidelines to reduce such risks (Wami et al., 2013) . To overcome this limitation, we focus on predictive clustering (Blockeel et al., 2017) which combines prediction with clustering. Therefore, the cluster assignments are optimized such that patients in a cluster share similar future outcomes to provide a prognostic value.

In this paper, we propose an actor-critic approach for temporal predictive clustering, which we call AC-TPC. 1 Our model consists of three neural networks -an encoder, a selector, and a predictorand a set of centroid candidates. More specifically, the encoder maps an input time-series into a latent encoding; the selector utilizes the encoding and assigns a cluster to which the time-series belongs to via a sampling process; and the predictor estimates the future outcome distribution conditioned on either the encoding or the centroid of the selected cluster. The following three contributions render our model to identify the predictive clusters. First, to encourage each cluster to have homogeneous future outcomes, we define a clustering objective based on the Kullback-Leibler (KL) divergence between the predictor's output given the input time series, and the predictor's output given estimated cluster assignments. Second, we transform solving a combinatorial problem of identifying cluster into iteratively solving two sub-problems: optimization of the cluster assignments and optimization of the cluster centroids. Finally, we allow "back-propagation" through the sampling process of the selector by adopting the training of actor-critic models (Konda and Tsitsiklis, 2000) .

Throughout the experiments, we show significant performance improvements over the state-of-theart clustering methods on two real-world medical datasets. Then, to demonstrate the practical significance of our model, we consider a more realistic scenario where the future outcomes of interest are high-dimensional -such as, development of multiple comorbidities in the next year -and interpreting all possible combinations is intractable. Our experiments show that our model can identify meaningful clusters that can be translated into actionable information for clinical decision-making.

2. PROBLEM FORMULATION:

Let X ∈ X and Y ∈ Y be random variables for an input feature and an output label (i.e., one or a combination of future outcome(s) of interest) with a joint distribution p XY (and marginal distributions are p X and p Y , respectively) where X is the feature space and Y is the label space. Here, we focus our description on C-class classification tasks, i.e., Y = {1, • • • , C}. 2 We are given a time-series dataset D = {(x n t , y n t ) T n t=1 } N n=1 comprising sequences of realizations (i.e., observations) of the pair (X, Y ) for N patients. Here, (x n t , y n t ) T n t=1 is a sequence of T n observation pairs that correspond to patient n and t ∈ T n {1, • • • , T n } denotes the time stamp at which the observations are made.From this point forward, we omit the dependency on n when it is clear in the context and denote x 1:t = (x 1 , • • • , x t ) for ease of notation.

Our aim is to identify a set of K predictive clusters, C = {C(1), • • • , C(K)}, for time-series data. Each cluster consists of homogeneous data samples, that can be represented by its centroid, based on a certain similarity measure. There are two main distinctions from the conventional notion of clustering. First, we treat subsequences of each times-series as data samples and focus on partitioning {{x n 1:t } T n t=1 } N n=1 into C. Hence, we define a cluster as

C(k) = {x n 1:t |t ∈ T n , s n t = k} for k ∈ K {1, • • • , K}

where s n t ∈ K is the cluster assignment for a given x n 1:t . This is to flexibly update the cluster assignment (in real-time) to which a patient belongs as new observations are being accrued over time. Second, we define the similarity measure with respect to the label distribution and associate it with clusters to provide a prognostic value. More specifically, we want the distribution of output label for subsequences in each cluster to be homogeneous and, thus, can be well-represented by the centroid of that cluster. Let S be a random variable for the cluster assignment -that depends on a given subsequence x 1:t -and Y |S = k be a random variable for the output given cluster k. Then, such property of predictive clustering can be achieved by minimizing the following Kullback-Leibler (KL) divergence:

KL Y t |X 1:t = x 1:t Y t |S t = k for x 1:t ∈ C(k) (1)

where

KL Y t |X 1:t = x 1:t Y t |S t = k = y∈Y p(y|x 1:t ) log p(y|x 1:t ) -log p(y|s t ) dy.

Here, p(y|x 1:t ) and p(y|s t ) are the label distributions conditioned on a subsequence x 1:t and a cluster assignment s t , respectively. Note that (1) achieves its minimum when the two distributions are equivalent.

Figure 1 : The block diagram of AC-TPC. The red line implies the procedure of estimating p(y|S t = s t ) which includes a sampling process and the blue line implies that of estimating p(y|X 1:t = x 1:t ).

Finally, we establish our goal as identifying a set of predictive clusters C that optimizes the following objective: minimize

EQUATION

Unfortunately, the optimization problem in ( 2) is highly non-trivial. We need to estimate the objective function in (2) while solving a non-convex combinatorial problem of finding the optimal cluster assignments and cluster centroids.

3. ACTOR-CRITIC APPROACH FOR TEMPORAL PREDICTIVE CLUSTERING:

To effectively estimate the objective function in (2), we introduce three networks -an encoder, a selector, and a predictor -and an embedding dictionary as illustrated in Figure 1 . These components together provide the cluster assignment and the corresponding centroid based on a given sequence of observations and enable us to estimate the probability density p(y|s t ). More specifically, we define each component as follows:

• The encoder, f θ : t i=1 X → Z, is a RNN (parameterized by θ) that maps a (sub)sequence of a time-series x 1:t to a latent representation (i.e., encoding) z t ∈ Z where Z is the latent space.

• The selector, h ψ : Z → ∆ K-1 , is a fully-connected network (parameterized by ψ) that provides a probabilistic mapping to a categorical distribution from which the cluster assignment s t ∈ K is being sampled.

• The predictor, g φ : Z → ∆ C-1 , is a fully-connected network (parameterized by φ) that estimates the label distribution given the encoding of a time-series or the centroid of a cluster.

• The embedding dictionary, E = {e(1), • • • , e(K)} where e(k) ∈ Z for k ∈ K, is a set of cluster centroids lying in the latent space which represents the corresponding cluster.

Here,

∆ D-1 = {q ∈ [0, 1] K : q 1 + • • • + q D = 1} is a (D -1)-simplex that denotes the probability distribution for a D-dimensional categorical (class) variable.

At each time stamp t, the encoder maps a input (sub)sequence x 1:t into a latent encoding z t f θ (x 1:t ). Then, based on the encoding z t , the cluster assignment s t is drawn from a categorical distribution that is defined by the selector output, i.e., s t ∼ Cat(π t ) where

π t = [π t (1), • • • , π t (K)] h ψ (z t ).

Once the assignment s t is chosen, we allocate the latent encoding z t to an embedding e(s t ) in the embedding dictionary E. Since the allocated embedding e(s t ) corresponds to the centroid of the cluster to which x 1:t belongs, we can, finally, estimate the density p(y|s t ) in (2) as the output of the predictor given the embedding e(s t ), i.e., ȳt g φ (e(s t )).

3.1. LOSS FUNCTIONS:

In this subsection, we define loss functions to achieve our objective in (2); the details of how we train our model will be discussed in the following subsection.

Predictive Clustering Loss: Since finding the cluster assignment of a given sequence is a probabilistic problem due to the sampling process, the objective function in (2) must be defined as an expectation over the cluster assignment. Thus, we can estimate solving the objective problem in (2) as minimizing the following loss function:

EQUATION

where 1 (y t , ȳt ) = -C c=1 y c t log ȳc t . Here, we slightly abuse the notation and denote y = [y 1 • • • y C ] as the one-hot encoding of y, and y c and ȳc indicates the c-th component of y and ȳ, respectively. It is worth to highlight that minimizing 1 is equivalent to minimizing the KL divergence in (2) since the former term of the KL divergence is independent of our optimization procedure.

One critical question that may arise is how to avoid trivial solutions in this unsupervised setting of identifying the cluster assignments and the centroids (Yang et al., 2017) . For example, all the embeddings in E may collapse into a single point or the selector simply assigns equal probability to all the clusters regardless of the input sequence. In both cases, our model will fail to correctly estimate p(y|s t ) and, thus, end up finding a trivial solution. To address this issue, we introduce two auxiliary loss functions that are tailored to address this concern. It is worth to highlight that these loss functions are not subject to the sampling process and their gradients can be simply back-propagated.

Sample-Wise Entropy of Cluster Assignment: To motivate sparse cluster assignment such that the selector ultimately selects one dominant cluster for each sequence, we introduce sample-wise entropy of cluster assignment which is given as

EQUATION

where

π t = [π t (1) • • • π t (K)] = h ψ (f θ (x 1:t )).

The sample-wise entropy achieves its minimum when π t becomes an one-hot vector.

Embedding Separation Loss: To prevent the embeddings in E from collapsing into a single point, we define a loss function that encourages the embeddings to represent different label distributions, i.e., g φ (e(k)) for k ∈ K, from each other:

EQUATION

where 1 is reused to quantify the distance between label distributions conditioned on each cluster. We minimize (5) when updating the embedding vectors e(1), • • • , e(K).

3.2. OPTIMIZATION:

The optimization problem in ( 2) is a non-convex combinatorial problem because it comprises not only minimizing the KL divergence but also finding the optimal cluster assignments and centroids. Hence, we propose an optimization procedure that iteratively solves two subproblems: i) optimizing the three networks -the encoder, selector, and predictor -while fixing the embedding dictionary and ii) optimizing the embedding dictionary while fixing the three networks. Pseudo-code of AC-TPC can be found in Appendix F.

3.2.1 OPTIMIZING THE THREE NETWORKS -f θ , h ψ , AND g φ Finding predictive clusters incorporates the sampling process which is non-differentiable. Thus, to render "back-propagation", we utilize the training of actor-critic models (Konda and Tsitsiklis, 2000) . More specifically, we view the combination of the encoder (f θ ) and the selector (h ψ ) as the "actor" parameterized by ω A = [θ, ψ], and the predictor (g φ ) as the "critic". The critic takes as input the the output of the actor (i.e., the cluster assignment) and estimates its value based on the samplewise predictive clustering loss (i.e., 1 (y t , ȳt )) given the chosen cluster. This, in turn, renders the actor to change the distribution of selecting a cluster to minimize such loss. Thus, it is important for the critic to perform well on the updated output of the actor while it is important for the actor to perform well on the updated loss estimation. As such, the parameters for the actor and the critic need to be updated iteratively.

Given the embedding dictionary E fixed (thus, we will omit the dependency on E), we train the actor, i.e., the encoder and the selector, by minimizing a combination of the predictive clustering loss L 1 and the entropy of cluster assignments L 2 , which is given by

EQUATION

where α ≥ 0 is a coefficient chosen to balance between the two losses. To derive the gradient of this loss with respect ω A = [θ, ψ], we utilize the ideas from actor-critic models (Konda and Tsitsiklis, 2000) as follows; please refer to Appendix B for the detailed derivation:

EQUATION

Note that since no sampling process is considered in L 2 (θ, ψ), we can simply derive

∇ ω A L 2 (θ, ψ).

Iteratively with training the actor, we train the critic, i.e., the predictor, by minimizing the predictive clustering loss L 1 as the following:

EQUATION

whose gradient with respect to φ can be givens as

∇ φ L C (φ) = ∇ φ L 1 (θ, ψ, φ).

Note that since the critic is independent of the sampling process, the gradient can be simply back-propagated.

3.2.2. OPTIMIZING THE CLUSTER CENTROIDS:

Now, once the parameters for the three networks (θ, ψ, φ) are fixed (thus, we omit the dependency on θ, ψ, and φ), we updated the embeddings in E by minimizing a combination of the predictive clustering loss L 1 and the embedding separation loss L 3 , which is given by

EQUATION

where β ≥ 0 is a coefficient chosen to balance between the two losses.

3.2.3. INITIALIZING AC-TPC VIA PRE-TRAINING:

Since we transform the non-trivial combinatorial optimization problem in (2) into iteratively solving two sub-problems, initialization is crucial to achieve better optimization as a similar concern has been addressed in (Yang et al., 2017) .

Therefore, we initialize our model based on the following procedure. First, we pre-train the encoder and the predictor by minimizing the following loss function based on the predicted label distribution given the latent encodings of input sequences, i.e., ŷt g φ (z t ) = g φ (f θ (x 1:t )), as the following:

EQUATION

)

Minimizing (10) encourages the latent encoding to be enriched with information for accurately predicting the label distribution. Then, we perform K-means (other clustering method can be also applied) based on the learned representations to initialize the embeddings E and the cluster assignments {{s n t } T n t=1 } N n=1 . Finally, we pre-train the selector h ψ by minimizing the cross entropy treating the initialized cluster assignments as the true clusters.

4. RELATED WORK:

Temporal clustering, also known as time-series clustering, is a process of unsupervised partitioning of the time-series data into clusters in such a way that homogeneous time-series are grouped together based on a certain similarity measure. Temporal clustering is challenging because i) the data is often high-dimensional -it consists of sequences not only with high-dimensional features but also with many time points -and ii) defining a proper similarity measure for time-series is not straightforward since it is often highly sensitive to distortions (Ratanamahatana et al., 2005) . To address these challenges, there have been various attempts to find a good representation with reduced dimensionality or to define a proper similarity measure for times-series (Aghabozorgi et al., 2015) .

Recently, Baytas et al. (2017) and Madiraju et al. (2018) proposed temporal clustering methods that utilize low-dimensional representations learned by RNNs. These works are motivated by the success of applying deep neural networks to find "clustering friendly" latent representations for clustering static data (Xie et al., 2017; Yang et al., 2017) . In particular, Baytas et al. (2017) utilized a modified LSTM auto-encoder to find the latent representations that are effective to summarize the input time-series and conducted K-means on top of the learned representations as an ad-hoc process. Similarly, Madiraju et al. (2018) proposed a bidirectional-LSTM auto-encoder that jointly optimizes the reconstruction loss for dimensionality reduction and the clustering objective. However, these methods do not associate a target property with clusters and, thus, provide little prognostic value in understanding the underlying disease progression.

Our work is most closely related to SOM-VAE (Fortuin et al., 2019) . This method jointly optimizes a static variational auto-encoder (VAE), that finds latent representations of input features, and a self-organizing map (SOM), that allows to map the latent representations into a more interpretable discrete representations, i.e., the embeddings. However, there are three key differences between our work and SOM-VAE. First, SOM-VAE aims at minimizing the reconstruction loss that is specified as the mean squared error between the original input and the reconstructed input based on the corresponding embedding. Thus, similar to the aforementioned methods, SOM-VAE neither associates future outcomes of interest with clusters. In contrast, we focus on minimizing the KL divergence between the outcome distribution given the original input sequence and that given the corresponding embedding to build association between future outcomes of interest and clusters. Second, to overcome non-differentiability caused by the sampling process (that is, mapping the latent representation to the embeddings), Fortuin et al. (2019) applies the gradient copying technique proposed by (van den Oord et al., 2017), while we utilize the training of actor-critic model (Konda and Tsitsiklis, 2000) . Finally, while we flexibly model time-series using LSTM, SOM-VAE handles time-series by integrating a Markov model in the latent representations. This can be a strict assumption especially in clinical settings where a patient's medical history is informative for predicting his/her future clinical outcomes (Ranganath et al., 2016) .

5. EXPERIMENTS:

In this section, we provide a set of experiments using two real-world time-series datasets. We iteratively update the three networks -the encoder, selector, and predictor -and the embedding dictionary as described in Section 3.2. For the network architecture, we constructed the encoder utilizing a single-layer LSTM (Hochreiter and Schmidhuber, 1997) with 50 nodes and constructed the selector and predictor utilizing two-layer fully-connected network with 50 nodes in each layer, respectively. The parameters (θ, ψ, φ) are initialized by Xavier initialization (Glorot and Bengio, 2010) and optimized via Adam optimizer (Kingma and Ba, 2014) with learning rate of 0.001 and keep probability 0.7. We chose the balancing coefficients α, β ∈ {0.1, 1.0, 3.0} utilizing grid search that achieves the minimum validation loss in (3); the effect of different loss functions are further investigated in the experiments. Here, all the results are reported using 5 random 64/16/20 train/validation/test splits.

5.1. REAL-WORLD DATASETS:

We conducted experiments to investigate the performance of AC-TPC on two real-world medical datasets; detailed statistics of each dataset can be found in Appendix C:

• UK Cystic Fibrosis registry (UKCF) 3 : This dataset records annual follow-ups for 5,171 adult patients (aged 18 years or older) enrolled in the UK CF registry over the period from 2008 and 2015, with a total of 25,012 hospital visits. Each patient is associated with 89 variables (i.e., 11 static and 78 time-varying features), including information on demographics and genetic mutations, bacterial infections, lung function scores, therapeutic managements, and diagnosis on comorbidities. We set the development of different comorbidities in the next year as the label of interest at each time stamp. • Alzheimer's Disease Neuroimaging Initiative (ADNI)4 : This dataset consists of 1,346 patients in the Alzheimer's disease study with a total of 11,651 hospital visits, which tracks the disease progression via follow-up observations at 6 months interval. Each patient is associated with 21 variables (i.e., 5 static and 16 time-varying features), including information on demographics, biomarkers on brain functions, and cognitive test results. We set predictions on the three diagnostic groups -normal brain functioning, mild cognitive impairment, and Alzheimer's diseaseas the label of interest at each time stamp.

5.2. BENCHMARKS:

We compare AC-TPC with clustering methods ranging from conventional approaches based on Kmeans to the state-of-the-art approaches based on deep neural networks. All the benchmarks compared in the experiments are tailored to incorporate time-series data as described below:

• Dynamic time warping followed by K-means: Dynamic time warping (DTW) is utilized to quantify pairwise distance between two variable-length sequences and, then, K-means is applied (denoted as KM-DTW). • K-means with deep neural networks: To handle variable-length time-series data, we utilize our encoder and predictor that are trained based on (10) for dimensionality reduction; this is to provide fixed-length and low-dimensional representations for time-series. Then, we apply K-means on the latent encodings z (denoted as KM-E2P (Z)) and on the predicted label distributions ŷ (denoted as KM-E2P (Y)), respectively. • Extensions of DCN (Yang et al., 2017) : Since the DCN is designed for static data, we replace their static auto-encoder with a sequence-to-sequence network to incorporate time-series data (denoted as DCN-S2S). 5 In addition, to associated with the label distribution, we compare a DCN whose static auto-encoder is replaced with our encoder and predictor (denoted as DCN-E2P) to focus dimensionality reduction while preserving information for predicting the label.

• SOM-VAE (Fortuin et al., 2019) : We compare with SOM-VAE -though, this method is oriented towards visualizing input data via SOM -since it naturally clusters time-series data (denoted as SOM-VAE). In addition, we compare with a variation of SOM-VAE by replacing the decoder with our predictor in order to find embeddings that capture information for predicting the label (denoted as SOM-VAE-P). For both cases, we set the dimension of SOM to K.

It is worth highlighting that the label information is provided for training DCN-E2P, KM-E2P, and SOM-VAE-P while the label information is not provided for training KM-DTW, DCN-S2S, and SOM-VAE. Please refer to Appendix D for the summary of major components of the tested benchmarks and the implementation details.

5.3. PERFORMANCE METRICS:

To assess the clustering performance, we applied the following three standard metrics for evaluating clustering performances when the ground-truth cluster label is available: purity score, normalized mutual information (NMI) (Vinh et al., 2010) , and adjusted Rand index (ARI) (Hubert and Arabie, 1985) . More specifically, the purity score assesses how homogeneous each cluster is (ranges from 0 to 1 where 1 being a cluster consists of a single class), the NMI is an information theoretic measure of how much information is shared between the clusters and the labels that is adjusted for the number of clusters (ranges from 0 to 1 where 1 being a perfect clustering), and ARI is a corrected-for-chance version of the Rand index which is a measure of the percentage of correct cluster assignments (ranges from -1 to 1 where 1 being a perfect clustering and 0 being a random clustering). In a nutshell, all the three performance metrics are commonly used but all have its pros and cons; for instance, the purity score easily converges to 1 when there are as many clusters as data samples. Thus, using them together suffices to demonstrate the effectiveness of the clustering methods.

To assess the prediction performance of the identified predictive clusters, we utilized both area under receiver operator characteristic curve (AUROC) and area under precision-recall curve (AUPRC) based on the label predictions of each cluster and the ground-truth binary labels on the future outcomes of interest. Note that the prediction performance is available only for the benchmarks that incorporate the label information during training.

5.4. CLUSTERING PERFORMANCE:

We start with a simple scenario where the true class (i.e., the ground-truth cluster label) is available and the number of classes is tractable. In particular, we set C = 2 3 = 8 based on the binary labels for the development of three common comorbidities of cystic fibrosis -diabetes, ABPA, and intestinal obstruction -in the next year for the UKCF dataet and C = 3 based on the mutually exclusive three diagnostic groups for the ADNI dataset. We compare AC-TPC against the aforementioned benchmarks with respect to the clustering and prediction performance in Table 1 .

As shown in Table 1 , AC-TPC achieved performance gain over all the tested benchmarks in terms of both clustering and prediction performance -where most of the improvements were statistically significant with p-value < 0.01 or p-value < 0.05 -for both datasets. Importantly, clustering methods -i.e., KM-DTW, DCN-S2S, and SOM-VAE -that do not associate with the future outcomes of interest identified clusters that provide little prognostic value on the future outcomes (note that the true class is derived from the future outcome of interest). This is clearly shown by the ARI value near 0 which indicates that the identified clusters have no difference with random assignments. Therefore, similar sequences with respect to the latent representations tailored for reconstruction or with respect to the shape-based measurement using DTW can have very different future outcomes.

In Figure 2 , we further investigate the purity score, NMI, and ARI by varying the number of clusters K from 4 to 16 on the UKCF dataset in the same setting with that stated above (i.e., C = 8).

Here, the three methods -i.e., KM-DTW, DCN-S2S, and SOM-VAE -are excluded for better visualization. As we can see in Figure 2 , our model rarely incur performance loss in both NMI and ARI while the benchmarks (except for SOM-VAE-P) showed significant decrease in the performance as K increased (higher than C). This is because the number of clusters identified by AC-TPC (i.e., the number of activated clusters where we define cluster k is activated if |C(k)| > 0) was the same with C most of the times, while the DCN-based methods identified exactly K clusters (due to the K-means). Since the NMI and ARI are adjusted for the number of clusters, a smaller number of identified clusters yields, if everything being equal, a higher performance. In contrast, while our model achieved the same purity score for K ≥ 8, the benchmark showed improved performance as K increased since the purity score does not penalize having many clusters. This is an important property of AC-TPC that we do not need to know a priori what the number of cluster is which is a common practical challenge of applying the conventional clustering methods (e.g., K-means).

The performance gain of our model over SOM-VAE-P (and, our analysis is the same for SOM-VAE) comes from two possible sources: i) SOM-VAE-P mainly focuses on visualizing the input with SOM which makes both the encoder and embeddings less flexible -this is why it performed better with higher K -and ii) the Markov property can be a too strict assumption for time-series data especially in clinical settings where a patient's medical history is informative for predicting his/her future clinical outcomes (Ranganath et al., 2016) .

5.5. CONTRIBUTIONS OF THE AUXILIARY LOSS FUNCTIONS:

As described in Section 3.1, we introduced two auxiliary loss functions -the sample-wise entropy of cluster assignment (4) and the embedding separation loss (5) -to avoid trivial solution that may arise in identifying the predictive clusters. To analyze the contribution of each auxiliary loss function, we report the average number of activated clusters, clustering performance, and prediction performance on the UKCF dataset with 3 comorbidities as described in Section 5.4. Throughout the experiment, we set K = 16 -which is larger than C -to find the contribution of these loss functions to the number of activated clusters. As we can see in Table 2 , both auxiliary loss functions make important contributions in improving the quality of predictive clustering. More specifically, the sample-wise entropy encourages the selector to choose one dominant cluster. Thus, as we can see results with α = 0, without the sample-wise entropy, our selector assigns an equal probability to all 16 clusters which results in a trivial solution.

We observed that, by augmenting the embedding separation loss (5), AC-TPC identifies a smaller number of clusters owing to the well-separated embeddings; in Appendix E, we further investigate the usefulness of ( 5) in identifying the number of clusters in a data-driven fashion.

5.6. TARGETING MULTIPLE FUTURE OUTCOMES -A PRACTICAL SCENARIO:

In this experiment, we focus on a more practical scenario where the future outcome of interest is high-dimensional and the number of classes based on all the possible combinations of future outcomes becomes intractable. For example, suppose that we are interested in the development of M comorbidities in the next year whose possible combinations grow exponentially C = 2 M . Interpreting such a large number of patient subgroups will be a daunting task which hinders the understanding of underlying disease progression. Since different comorbidities may share common driving factors (Ronan et al., 2017) , we hope our model to identify much smaller underlying (latent) clusters that govern the development of comorbidities. Here, to incorporate with M comorbidities (i.e., M binary labels), we redefine the output space as Y = {0, 1} M and modify the predictor and loss functions, accordingly, as described in Appendix A. Figure 3 : Clusters with high-risk of developing diabetes. We reported the cluster-specific frequencies of developing comorbidities -liver disease, asthma, ABPA, and osteopenia that are co-occurred with diabetes -in the next year.

Throughout this experiment, we aim at identifying subgroups of patients that are associated with the next-year development of 22 different comorbidities in the UKCF dataset. In Table 3 , we reported 12 identified clusters -on average, the number of activated clusters were 13.6 -and the top three frequent comorbidities developed in the next year since the latest observation and the corresponding frequency; please refer to Appendix E for a full list. Here, the frequency is calculated in a clusterspecific fashion based on the true label. As we can see in Table 3 , the identified clusters displayed very different label distributions; that is, the combination of comorbidities as well as their frequency were very different across the clusters. For example, patients in Cluster 1 experienced low-risk of developing any comorbities in the next year while 85% of patients in Cluster 0 experienced diabetes in the next year.

In Figure 3 , we further investigated subgroups of patients -Cluster 0, 5, 7, 8, and 10 -who had high risk of developing diabetes in the next year. Although all these clusters displayed high risk of diabetes, the frequency of other co-occurred comorbidities was significantly different across the clusters. In particular, around 89% of the patients in Cluster 5 experienced asthma in the next year while it was less than 3% of the patients in the other cluster. Interestingly, "leukotriene" -a medicine commonly used to manage asthma -and "FEV 1 % predicted" -a measure of lung function -were the two most different input features between patients in Cluster 5 and those in the other clusters. We observed similar findings in Cluster 7 with ABPA, Cluster 8 with liver disease, and Cluster 10 with osteopenia. Therefore, by grouping patients who are likely to develop a similar set of comorbidities, our method identified clusters that can be translated into actionable information for clinical decision-making.

6. CONCLUSION:

In this paper, we introduced AC-TPC, a novel deep learning approach for predictive clustering of time-series data. We carefully defined novel loss functions to encourage each cluster to have homogeneous future outcomes (e.g., adverse events, the onset of comorbidities, etc.) and designed optimization procedures to address challenges and to avoid trivial solution in identifying such cluster assignments and the centroids. Throughout the experiments on two real-world datasets, we showed that our model achieves superior clustering performance over state-of-the-art and identifies meaningful clusters that can be translated into actionable information for clinical decision-making.

A VARIATIONS FOR REGRESSION AND BINARY CLASSIFICATION TASKS:

As the task changes, estimating the label distribution and calculating the KL divergence in (2) must be redefined accordingly:

• For regression task, i.e., Y = R, we modify the predictor as g φ : Z → R and replace 1 by

1 (y t , ȳt ) = y t -ȳt 2 2

. Minimizing 1 (y t , ȳt ) is equivalent to minimizing the KL divergence between p(y t |x 1:t ) and p(y t |s t ) when we assume these probability densities follow Gaussian distribution with the same variance.

• For the M -dimensional binary classification task, i.e., Y = {0, 1} M , we modify the predictor as

g φ : Z → [0, 1] M and replace 1 by 1 (y t , ȳt ) = - M m=1 y m t log ȳm t + (1 -y m t ) log(1 -ȳm t )

which is required to minimize the KL divergence. Here, y m t and ȳm t indicate the m-th element of y t and ȳt , respectively. The basic assumption here is that the distribution of each binary label is independent given the input sequence.

B DETAILED DERIVATION OF (7):

To derive the gradient of the predictive clustering loss in (7) with respect ω A = [θ, ψ], we utilized the ideas from actor-critic models (Konda and Tsitsiklis, 2000) . The detailed derivation of the former term in ( 7) is described below (for notational simplicity, we omit the expectation on E x,y∼p XY ):

EQUATION

C DETAILS ON THE DATASETS C.1 UKCF DATASET UK Cystic Fibrosis registry (UKCF)6 records annual follow-ups for 5,171 adult patients (aged 18 years or older) over the period from 2008 and 2015, with a total of 25,012 hospital visits. Each patient is associated with 89 variables (i.e., 11 static and 78 time-varying features), including information on demographics and genetic mutations, bacterial infections, lung function scores, therapeutic managements, and diagnosis on comorbidities. The detailed statistics are given in Table 4 .

C.2 ADNI DATASET:

Alzheimer's Disease Neuroimaging Initiative (ADNI)7 study consists of 1,346 patients with a total of 11,651 hospital visits, which tracks the disease progression via follow-up observations at 6 months interval. Each patient is associated with 21 variables (i.e., 5 static and 16 time-varying features), including information on demographics, biomarkers on brain functions, and cognitive test results. The three diagnostic groups were normal brain functioning (0.55), mild cognitive impairment (0.43), and Alzheimer's disease (0.02). The detailed statistics are given in Table 5 .

D DETAILS ON THE BENCHMARKS:

We compared AC-TPC in the experiments with clustering methods ranging from conventional approaches based on K-means to the state-of-the-art approaches based on deep neural networks. The details of how we implemented the benchmarks are described as the following:

• Dynamic time warping followed by K-means8 : Dynamic time warping (DTW) is utilized to quantify pairwise distance between two variable-length sequences and, then, K-means is applied (denoted as KM-DTW). • K-means with deep neural networks: To handle variable-length time-series data, we utilized an encoder-predictor network as depicted in Figure 4b and trained the network based on (10) for dimensionality reduction; this is to provide fixed-length and low-dimensional representations for time-series. Then, we applied K-means on the latent encodings z (denoted as KM-E2P (Z)) and on the predicted label distributions ŷ (denoted as KM-E2P (Y)), respectively. We implemented the encoder and predictor of KM-E2P with the same network architectures with those of our model: the encoder is a single-layer LSTM with 50 nodes and the decoder is a two-layered fully-connected network with 50 nodes in each layer. • Extensions of DCN9 (Yang et al., 2017) : Since the DCN is designed for static data, we utilized a sequence-to-sequence model in Figure 4a for the encoder-decoder network as an extension to incorporate time-series data (denoted as DCN-S2S) and trained the network based on the reconstruction loss (using the reconstructed input sequence x1:t ). For implementing DCN-S2S, we used a single-layer LSTM with 50 nodes for both the encoder and the decoder. And, we augmented a fully-connected layer with 50 nodes is used to reconstruct the original input sequence from the latent representation of the decoder.

In addition, since predictive clustering is associated with the label distribution, we compared a DCN whose encoder-decoder structure is replaced with our encoder-predictor network in Figure 4b (denoted as DCN-E2P) to focus the dimensionality reduction -and, thus, finding latent encodings where clustering is performed -on the information for predicting the label distribution. We implemented the encoder and predictor of DCN-E2P with the same network architectures with those of our model as described in Section 5.

• SOM-VAE10 (Fortuin et al., 2019) : We compare with SOM-VAE -though, this method is oriented towards visualization of input data via SOM -since it naturally clusters time-series data assuming Markov property (denoted as SOM-VAE). We replace the original CNN architecture of the encoder and the decoder with three-layered fully-connected network with 50 nodes in each layer, respectively. The network architecture is depicted in Figure 4c where xt and xt indicate the reconstructed inputs based on the encoding z t and the embedding e t at time t, respectively.

In addition, we compare with a variation of SOM-VAE by replacing the decoder with the predictor to encourage the latent encoding to capture information for predicting the label distribution (denoted as SOM-VAE-P). For the implementation, we replaced the decoder of SOM-VAE with our predictor which is a two-layered fully-connected layer with 50 nodes in each layer to predict the label distribution as illustrated in Figure 4d . Here, ŷt and ȳt indicate the predicted labels based on the encoding z t and the embedding e t at time t, respectively.

For both cases, we used the default values for balancing coefficients of SOM-VAE and the dimension of SOM to be equal to K.

We compared and summarized major components of the benchmarks in Table 6 .

E ADDITIONAL EXPERIMENTS E.1 ADDITIONAL RESULTS ON TARGETING MULTIPLE FUTURE OUTCOMES:

Throughout the experiment in Section 5.6, we identified 12 subgroups of patients that are associated with the next-year development of 22 different comorbidities in the UKCF dataset. In Table 7 , we reported 12 identified clusters and the full list of comorbidities developed in the next year since the latest observation and the corresponding frequency. Here, the frequency is calculated in a clusterspecific fashion based on the true label.

E.2 TRADE-OFF BETWEEN CLUSTERING AND PREDICTION PERFORMANCE:

In predictive clustering, the trade-off between the clustering performance (for better interpretability) -which quantifies how the data samples are homogeneous within each cluster and heterogeneous across clusters with respect to the future outcomes of interest -and the prediction performance is a common issue. The most important parameter that governs this trade-off is the number of clusters. More specifically, increasing the number of clusters will make the predictive clusters have higher diversity to represent the output distribution and, thus, will increase the prediction performance while decreasing the clustering performance. One extreme example is that there are as many clusters as data samples which will make the identified clusters fully individualized; as a consequence, each cluster will lose the interpretability as it no longer groups similar data samples.

To highlight this trade-off, we conducted experiments under the same experimental setup with that of Section 5.6 where our aim is to identify underlying (unknown) clusters when the future outcome of interest is high-dimensional. For the performance measures, we utilized the AUROC and AUPRC to assess the prediction performance, and utilized the average Silhouette index (SI) (Rousseeuw, 1987) -a widely used measure of how similar a member is to its own cluster (homogeneity within a cluster) compared to other clusters (heterogeneity across clusters ) when the ground-truth cluster labels are not available -to assess the identified clusters. Formally, the SI for a subsequence x n 1:t ∈ C k can be given as follows:

SI(n) = b(n) -a(n) max a(n), b(n) (12)

where a

(n) = 1 |C k |-1 m =n y n t -y m t 1 and b(n) = min k =k 1 |C k | m∈C k y n t -y m t 1 .

Here, we used the L1-distance between the ground-truth labels of the future outcomes of interest since our goal is to group input subsequences with similar future outcomes. To control the number of identified clusters (i.e., the activated clusters) of our method, we set β = 0 (since the embedding separation loss in (5) controls the activation of clusters) and reported the performance by increasing the number of possible clusters K. As can be seen in Figure 5 , the prediction performance increased with an increased number of identified clusters due to the higher diversity to represent the label distribution while making the identified clusters less interpretable (i.e., the cohesion and separation among clusters become ambiguous as seen in the low average SI). On the other hand, when we set β = 1.0 (which is selected based on the validation loss in 3), our method consistently identified a similar number of clusters for K > 20, i.e., 13.8 on average, in a data-driven fashion and provided slightly reduced prediction performance with significantly better interpretability, i.e., the average SI 0.120 on average. This highlights the usefulness of (5) which helps to identify clusters with different label distributions. 

:

Source code available at https://github.com/ICLR2020-ACTPC/ACTPC_submission.git

In this paper, we focus our description on C-class classification task, i.e., Y = {1, • • • , C}; in Appendix A, we discuss simple modifications of our model for regression and M -dimensional binary classification tasks, i.e., Y = R and Y = {0, 1} M , respectively.

https://www.cysticfibrosis.org.uk/the-work-we-do/uk-cf-registry

https://adni.loni.usc.edu

This extension is a representative of recently proposed deep learning approaches for clustering of both static data(Xie et al., 2017;Yang et al., 2017) and time-series data(Baytas et al., 2017;Madiraju et al., 2018) since these methods are built upon the same concept -that is, applying deep networks for dimensionality reduction to conduct conventional clustering methods, e.g., K-means.

https://www.cysticfibrosis.org.uk/the-work-we-do/uk-cf-registry

https://adni.loni.usc.edu

https://github.com/rtavenar/tslearn

https://github.com/boyangumn/DCN

https://github.com/ratschlab/SOM-VAE

Figures and Tables:

(a) The averaged purity score. (b) The averaged NMI. (c) The averaged ARI.

Figure 2: The purity score, NMI, and ARI (mean and 95% confidence interval) for the UKCF dataset (C = 8) with various K.

Figure 4: The block diagrams of the tested benchmarks.

Figure 5: AUROC, AUPRC, and average SI (mean and 95% confidence interval) and the number of activated clusters with various K.

Performance Comparison on the UKCF and ADNI datasets. indicates p-value < 0.01, † indicates p-value < 0.05

Table content: <table><tr><td>Dataset</td><td>Method</td><td>Purity</td><td>NMI</td><td>ARI</td><td>AUROC</td><td>AUPRC</td></tr><tr><td/><td>KM-DTW</td><td>0.573±0.01  *</td><td>0.010±0.01  *</td><td>0.014±0.01  *</td><td>N/A</td><td>N/A</td></tr><tr><td>UKCF</td><td colspan="2">KM-E2P (Z) 0.719±0.01  DCN-S2S 0.607±0.06  *  DCN-E2P 0.751±0.02  *</td><td>0.059±0.08  *  0.275±0.02  *</td><td>0.063±0.09  *  0.184±0.01  *</td><td>N/A 0.772±0.03  *</td><td>N/A 0.487±0.03  *</td></tr><tr><td/><td>SOM-VAE</td><td>0.573±0.01  *</td><td>0.006±0.00  *</td><td>0.006±0.01  *</td><td>N/A</td><td>N/A</td></tr><tr><td/><td>SOM-VAE-P</td><td>0.638±0.04  *</td><td>0.201±0.05  *</td><td>0.283±0.17  †</td><td>0.754±0.05  *</td><td>0.331±0.07  *</td></tr><tr><td/><td>Proposed</td><td>0.807±0.01</td><td>0.463±0.01</td><td>0.602±0.01</td><td>0.843±0.01</td><td>0.605±0.01</td></tr><tr><td/><td>KM-DTW</td><td>0.566±0.02  *</td><td>0.019±0.02  *</td><td>0.006±0.02  *</td><td>N/A</td><td>N/A</td></tr><tr><td/><td>KM-E2P (Z)</td><td>0.736±0.03  †</td><td>0.249±0.02</td><td>0.230±0.03  †</td><td>0.707±0.01  *</td><td>0.509±0.01</td></tr><tr><td/><td>KM-E2P (Y)</td><td>0.776±0.05</td><td>0.264±0.07</td><td>0.317±0.11</td><td>0.756±0.04</td><td>0.503±0.04</td></tr><tr><td>ADNI</td><td>DCN-S2S DCN-E2P</td><td>0.567±0.02  *  0.749±0.06</td><td>0.005±0.00  *  0.261±0.05</td><td>0.000±0.01  *  0.215±0.06  †</td><td>N/A 0.721±0.03  †</td><td>N/A 0.509±0.03</td></tr><tr><td/><td>SOM-VAE</td><td>0.566±0.02  *</td><td>0.040±0.06  *</td><td>0.011±0.02  *</td><td>N/A</td><td>N/A</td></tr><tr><td/><td>SOM-VAE-P</td><td>0.586±0.06  *</td><td>0.085±0.08  *</td><td>0.038±0.06  *</td><td>0.597±0.10  †</td><td>0.376±0.05  *</td></tr><tr><td/><td>Proposed</td><td>0.786±0.03</td><td>0.285±0.04</td><td>0.330±0.06</td><td>0.768±0.02</td><td>0.515±0.02</td></tr><tr><td/><td/><td/><td/><td/><td/><td/></tr></table>

Performance comparison with varying the balancing coefficients α, β for the UKCF dataset.

Table content: <table><tr><td colspan="2">Coefficients</td><td/><td colspan="2">Clustering Performance</td><td/><td colspan="2">Prognostic Value</td></tr><tr><td>α</td><td>β</td><td>Activated No.</td><td>Purity</td><td>NMI</td><td>ARI</td><td>AUROC</td><td>AUPRC</td></tr><tr><td>0.0</td><td>0.0</td><td>16</td><td>0.573±0.01</td><td>0.006±0.00</td><td>0.000±0.00</td><td>0.500±0.00</td><td>0.169±0.00</td></tr><tr><td>0.0</td><td>1.0</td><td>16</td><td>0.573±0.01</td><td>0.006±0.00</td><td>0.000±0.00</td><td>0.500±0.00</td><td>0.169±0.00</td></tr><tr><td>3.0</td><td>0.0</td><td>8.4</td><td>0.795±0.01</td><td>0.431±0.01</td><td>0.569±0.01</td><td>0.840±0.01</td><td>0.583±0.02</td></tr><tr><td>3.0</td><td>1.0</td><td>8</td><td>0.808±0.01</td><td>0.468±0.01</td><td>0.606±0.01</td><td>0.852±0.00</td><td>0.608±0.01</td></tr></table>

The top-3 frequent comorbidities developed in the next year for the 12 identified clusters. The values in parentheses indicate the corresponding frequency.

Table content: <table><tr><td>Clusters</td><td/><td colspan="2">Top-3 Frequent Comorbidities</td></tr><tr><td>0</td><td>Diabetes (0.85)</td><td>Liver Enzymes (0.21)</td><td>Arthropathy (0.14)</td></tr><tr><td>1</td><td>Liver Enzymes (0.09)</td><td>Arthropathy (0.08)</td><td>Depression (0.07)</td></tr><tr><td>2</td><td>ABPA (0.77)</td><td>Osteopenia (0.21)</td><td>Intestinal Obstruction (0.11)</td></tr><tr><td>3</td><td>Asthma (0.89)</td><td>Liver Disease (0.87)</td><td>Diabetes (0.29)</td></tr><tr><td>4</td><td>Osteoporosis (0.76)</td><td>Diabetes (0.43)</td><td>Arthropathy (0.20)</td></tr><tr><td>5</td><td>Asthma (0.88)</td><td>Diabetes (0.81)</td><td>Osteopenia (0.28)</td></tr><tr><td>6</td><td>Liver Disease (0.85)</td><td>Asthma (0.03)</td><td>ABPA (0.09)</td></tr><tr><td>7</td><td>ABPA (0.83)</td><td>Diabetes (0.78)</td><td>Osteopenia (0.25)</td></tr><tr><td>8</td><td>Diabetes (0.94)</td><td>Liver Disease (0.83)</td><td>Liver Enzymes (0.43)</td></tr><tr><td>9</td><td>Asthma (0.89)</td><td>Osteopenia (0.26)</td><td>ABPA (0.19)</td></tr><tr><td>10</td><td>Osteopenia (0.82)</td><td>Diabetes (0.81)</td><td>Arthropathy (0.23)</td></tr><tr><td>11</td><td>Osteopenia (0.77)</td><td>Liver Enzymes (0.18)</td><td>Arthropathy (0.12)</td></tr></table>

Comparison table of benchmarks.

Table content: <table><tr><td>Methods</td><td>Handling Time-Series</td><td>Clustering Method</td><td>Similarity Measure</td><td>Label Provided</td><td>Label Associated</td></tr><tr><td>KM-DTW</td><td>DTW</td><td>K-means</td><td>DTW</td><td>N</td><td>N</td></tr><tr><td>KM-E2P (Z)</td><td>RNN</td><td>K-means</td><td>Euclidean in Z</td><td>Y</td><td>Y (indirect)</td></tr><tr><td>KM-E2P (Y)</td><td>RNN</td><td>K-means</td><td>Euclidean in Y</td><td>Y</td><td>Y (direct)</td></tr><tr><td>DCN-S2S</td><td>RNN</td><td>K-means</td><td>Euclidean in Z</td><td>N</td><td>N</td></tr><tr><td>DCN-E2P</td><td>RNN</td><td>K-means</td><td>Euclidean in Z</td><td>Y</td><td>Y (indirect)</td></tr><tr><td>SOM-VAE</td><td colspan="3">Markov model embedding mapping reconstruction loss</td><td>N</td><td>N</td></tr><tr><td>SOM-VAE-P</td><td colspan="2">Markov model embedding mapping</td><td>prediction loss</td><td>Y</td><td>Y (direct)</td></tr><tr><td>Proposed</td><td>RNN</td><td>embedding mapping</td><td>KL divergence</td><td>Y</td><td>Y (direct)</td></tr></table>

The comorbidities developed in the next year for the 12 identified clusters. The values in parentheses indicate the corresponding frequency.

Table content: <table><tr><td>Clusters</td><td/><td colspan="2">Comorbidities and Frequencies</td><td/></tr><tr><td/><td>Diabetes (0.85)</td><td>Liver Enzymes (0.21)</td><td>Arthropathy (0.14)</td><td>Depression (0.10)</td></tr><tr><td/><td>Hypertens (0.08)</td><td>Osteopenia (0.07)</td><td>Intest. Obstruction (0.07)</td><td>Cirrhosis (0.04)</td></tr><tr><td>Cluster</td><td>ABPA (0.04)</td><td>Liver Disease (0.04)</td><td>Osteoporosis (0.03)</td><td>Hearing Loss (0.03)</td></tr><tr><td>0</td><td>Asthma (0.02)</td><td>Kidney Stones (0.01)</td><td>Bone fracture (0.01)</td><td>Hemoptysis (0.01)</td></tr><tr><td/><td>Pancreatitis (0.01)</td><td>Cancer (0.00)</td><td>Gall bladder (0.00)</td><td>Colonic stricture (0.00)</td></tr><tr><td/><td>GI bleed -no var. (0.00)</td><td>GI bleed -var. (0.00)</td><td/><td/></tr><tr><td/><td>Liver Enzymes (0.09)</td><td>Arthropathy (0.08)</td><td>Depression (0.07)</td><td>Intest. Obstruction (0.06)</td></tr><tr><td/><td>Diabetes (0.06)</td><td>Osteopenia (0.05)</td><td>ABPA (0.04)</td><td>Asthma (0.03)</td></tr><tr><td>Cluster</td><td>Liver Disease (0.03)</td><td>Hearing Loss (0.03)</td><td>Osteoporosis (0.02)</td><td>Pancreatitis (0.02)</td></tr><tr><td>1</td><td>Kidney Stones (0.02)</td><td>Hypertension (0.01)</td><td>Cirrhosis (0.01)</td><td>Gall bladder (0.01)</td></tr><tr><td/><td>Cancer (0.01)</td><td>Hemoptysis (0.00)</td><td>Bone fracture (0.00)</td><td>Colonic stricture (0.00)</td></tr><tr><td/><td>GI bleed -no var. (0.00)</td><td>GI bleed -var. (0.00)</td><td/><td/></tr><tr><td/><td>ABPA (0.77)</td><td>Osteopenia (0.21)</td><td>Intest. Obstruction (0.11)</td><td>Hearing Loss (0.10)</td></tr><tr><td/><td>Liver Enzymes (0.07)</td><td>Diabetes (0.06)</td><td>Depression (0.05)</td><td>Pancreatitis (0.05)</td></tr><tr><td>Cluster</td><td>Liver Disease (0.04)</td><td>Arthropathy (0.04)</td><td>Asthma (0.03)</td><td>Bone fracture (0.02)</td></tr><tr><td>2</td><td>Osteoporosis (0.02)</td><td>Hypertension (0.01)</td><td>Cancer (0.01)</td><td>Cirrhosis (0.01)</td></tr><tr><td/><td>Kidney Stones (0.01)</td><td>Gall bladder (0.01)</td><td>Hemoptysis (0.00)</td><td>Colonic stricture (0.00)</td></tr><tr><td/><td>GI bleed -no var. (0.00)</td><td>GI bleed -var. (0.00)</td><td/><td/></tr><tr><td/><td>Asthma (0.89)</td><td>Liver Disease (0.87)</td><td>Diabetes (0.29)</td><td>Osteopenia (0.28)</td></tr><tr><td/><td>Liver Enzymes (0.24)</td><td>ABPA (0.15)</td><td>Osteoporosis (0.11)</td><td>Hearing Loss (0.06)</td></tr><tr><td>Cluster</td><td>Arthropathy (0.05)</td><td>Intest. Obstruction (0.05)</td><td>Depression (0.04)</td><td>Hypertension (0.03)</td></tr><tr><td>3</td><td>Cirrhosis (0.02)</td><td>Kidney Stones (0.02)</td><td>Pancreatitis (0.02)</td><td>Gall bladder (0.02)</td></tr><tr><td/><td>Cancer (0.01)</td><td>Bone fracture (0.00)</td><td>Hemoptysis (0.00)</td><td>Colonic stricture (0.00)</td></tr><tr><td/><td>GI bleed -no var. (0.00)</td><td>GI bleed -var. (0.00)</td><td/><td/></tr><tr><td/><td>Osteoporosis (0.76)</td><td>Diabetes (0.43)</td><td>Arthropathy (0.20)</td><td>Liver Enzymes (0.18)</td></tr><tr><td/><td>Osteopenia (0.15)</td><td>Depression (0.13)</td><td>Intest. Obstruction (0.11)</td><td>ABPA (0.11)</td></tr><tr><td>Cluster</td><td>Hearing Loss (0.09)</td><td>Liver Disease (0.08)</td><td>Hypertension (0.07)</td><td>Cirrhosis (0.07)</td></tr><tr><td>4</td><td>Kidney Stones (0.03)</td><td>Asthma (0.02)</td><td>Hemoptysis (0.02)</td><td>Bone fracture (0.02)</td></tr><tr><td/><td>Gall bladder (0.01)</td><td>Pancreatitis (0.01)</td><td>Cancer (0.00)</td><td>Colonic stricture (0.00)</td></tr><tr><td/><td>GI bleed -no var. (0.00)</td><td>GI bleed -var. (0.00)</td><td/><td/></tr><tr><td/><td>Asthma (0.88)</td><td>Diabetes (0.81)</td><td>Osteopenia (0.28)</td><td>ABPA (0.24)</td></tr><tr><td/><td>Liver Enzymes (0.22)</td><td>Depression (0.15)</td><td>Osteoporosis (0.14)</td><td>Intest. Obstruction (0.12)</td></tr><tr><td>Cluster</td><td>Hypertension (0.10)</td><td>Cirrhosis (0.10)</td><td>Liver Disease (0.09)</td><td>Arthropathy (0.08)</td></tr><tr><td>5</td><td>Bone fracture (0.01)</td><td>Hemoptysis (0.01)</td><td>Pancreatitis (0.01)</td><td>Hearing Loss (0.01)</td></tr><tr><td/><td>Cancer (0.01)</td><td>Kidney Stones (0.01)</td><td>GI bleed -var. (0.01)</td><td>Gall bladder (0.00)</td></tr><tr><td/><td>Colonic stricture (0.00)</td><td>GI bleed -no var. (0.00)</td><td/><td/></tr><tr><td/><td>Liver Disease (0.85)</td><td>Liver Enzymes (0.37)</td><td>Osteopenia (0.27)</td><td>ABPA (0.09)</td></tr><tr><td/><td>Arthropathy (0.07)</td><td>Diabetes (0.06)</td><td>Intest. Obstruction (0.06)</td><td>Osteoporosis (0.05)</td></tr><tr><td>Cluster</td><td>Depression (0.03)</td><td>Asthma (0.03)</td><td>Hearing Loss (0.03)</td><td>Cirrhosis (0.02)</td></tr><tr><td>6</td><td>Hemoptysis (0.02)</td><td>Hypertension (0.01)</td><td>Kidney Stones (0.01)</td><td>Pancreatitis (0.00)</td></tr><tr><td/><td>Gall bladder (0.00)</td><td>Bone fracture (0.00)</td><td>Cancer (0.00)</td><td>Colonic stricture (0.00)</td></tr><tr><td/><td>GI bleed -no var. (0.00)</td><td>GI bleed -var. (0.00)</td><td/><td/></tr><tr><td/><td>ABPA (0.83)</td><td>Diabetes (0.78)</td><td>Osteopenia (0.25)</td><td>Osteoporosis (0.24)</td></tr><tr><td/><td>Liver Enzymes (0.15)</td><td>Intest. Obstruction (0.12)</td><td>Liver Disease (0.09)</td><td>Hypertension (0.07)</td></tr><tr><td>Cluster</td><td>Hearing Loss (0.07)</td><td>Arthropathy (0.06)</td><td>Depression (0.04)</td><td>Cirrhosis (0.02)</td></tr><tr><td>7</td><td>Asthma (0.01)</td><td>Bone fracture (0.01)</td><td>Kidney Stones (0.01)</td><td>Hemoptysis (0.01)</td></tr><tr><td/><td>Cancer (0.00)</td><td>Pancreatitis (0.00)</td><td>Gall bladder (0.00)</td><td>Colonic stricture (0.00)</td></tr><tr><td/><td>GI bleed -no var. (0.00)</td><td>GI bleed -var. (0.00)</td><td/><td/></tr><tr><td/><td>Diabetes (0.94)</td><td>Liver Disease (0.83)</td><td>Liver Enzymes (0.43)</td><td>Osteopenia (0.30)</td></tr><tr><td/><td>Hearing Loss (0.11)</td><td>Osteoporosis (0.10)</td><td>Intest. Obstruction (0.09)</td><td>Cirrhosis (0.08)</td></tr><tr><td>Cluster</td><td>Depression (0.08)</td><td>ABPA (0.07)</td><td>Arthropathy (0.06)</td><td>Hypertension (0.05)</td></tr><tr><td>8</td><td>Kidney Stones (0.05)</td><td>Asthma (0.02)</td><td>Hemoptysis (0.01)</td><td>Bone fracture (0.01)</td></tr><tr><td/><td>Cancer (0.00)</td><td>Pancreatitis (0.00)</td><td>Gall bladder (0.00)</td><td>Colonic stricture (0.00)</td></tr><tr><td/><td>GI bleed -no var. (0.00)</td><td>GI bleed -var. (0.00)</td><td/><td/></tr><tr><td/><td>Asthma (0.89)</td><td>Osteopenia (0.26)</td><td>ABPA (0.19)</td><td>Arthropathy (0.14)</td></tr><tr><td/><td>Intest. Obstruction (0.11)</td><td>Depression (0.08)</td><td>Osteoporosis (0.08)</td><td>Diabetes (0.06)</td></tr><tr><td>Cluster</td><td>Liver Enzymes (0.06)</td><td>Hemoptysis (0.03)</td><td>Hypertension (0.02)</td><td>Liver Disease (0.02)</td></tr><tr><td>9</td><td>Pancreatitis (0.02)</td><td>Bone fracture (0.01)</td><td>Cancer (0.01)</td><td>Cirrhosis (0.01)</td></tr><tr><td/><td>Gall bladder (0.01)</td><td>Hearing Loss (0.01)</td><td>Kidney Stones (0.00)</td><td>Colonic stricture (0.00)</td></tr><tr><td/><td>GI bleed -no var. (0.00)</td><td>GI bleed -var. (0.00)</td><td/><td/></tr><tr><td/><td>Osteopenia (0.82)</td><td>Diabetes (0.81)</td><td>Arthropathy (0.23)</td><td>Depression (0.19)</td></tr><tr><td/><td>Liver Enzymes (0.18)</td><td>Hypertension (0.16)</td><td>Hearing Loss (0.10)</td><td>Liver Disease (0.10)</td></tr><tr><td>Cluster</td><td>Osteoporosis (0.10)</td><td>Intest. Obstruction (0.09)</td><td>ABPA (0.09)</td><td>Kidney Stones (0.07)</td></tr><tr><td>10</td><td>Cirrhosis (0.05)</td><td>Asthma (0.01)</td><td>Cancer (0.00)</td><td>GI bleed -var. (0.00)</td></tr><tr><td/><td>Bone fracture (0.00)</td><td>Hemoptysis (0.00)</td><td>Pancreatitis (0.00)</td><td>Gall bladder (0.00)</td></tr><tr><td/><td>Colonic stricture (0.00)</td><td>GI bleed -no var. (0.00)</td><td/><td/></tr><tr><td/><td>Osteopenia (0.77)</td><td>Liver Enzymes (0.18)</td><td>Arthropathy (0.12)</td><td>Depression (0.09)</td></tr><tr><td/><td>Hypertension (0.06)</td><td>Diabetes (0.06)</td><td>Hearing Loss (0.06)</td><td>ABPA (0.05)</td></tr><tr><td>Cluster</td><td>Liver Disease (0.05)</td><td>Osteoporosis (0.04)</td><td>Intest. Obstruction (0.04)</td><td>Cirrhosis (0.02)</td></tr><tr><td>11</td><td>Asthma (0.02)</td><td>Pancreatitis (0.02)</td><td>Bone fracture (0.01)</td><td>Cancer (0.01)</td></tr><tr><td/><td>Kidney Stones (0.00)</td><td>Gall bladder (0.00)</td><td>Colonic stricture (0.00)</td><td>Hemoptysis (0.00)</td></tr><tr><td/><td>GI bleed -no var. (0.00)</td><td>GI bleed -var. (0.00)</td><td/><td/></tr></table>

Peer review (iclr2020):
Summary: This work proposes a method to perform clustering on a time-series data for prediction purposes, unlike the classical clustering where it is done in an unsupervised manner. The authors use an encoder (RNN) to process the time-series medical records, a selector to sample the cluster label for each encoding, and a predictor to predict the labels based on the selected cluster centroids. Since the sampling process prevents the authors from using back-prop, they employ an actor-critic method. Strengths: - Although some would argue otherwise, patient similarity has some promise to be useful in clinical practice. - The proposed method clearly outperformed various baselines in terms of clustering (Table 1). - Table 3 and Figure 3 show that the proposed method can capture heterogeneous subgroups of the dataset. Concerns: - I*m not a clustering expert, but I*m skeptical this is the first work to combine clustering and supervised prediction using an RL technique. - It is unclear what it means to train the embedding dictionary. Are there trainable parameters in the embedding dictionary? It seems that all it does is calculate the mean of the z_t*s (i.e. centroid) in each cluster. Or do you take the centroid embeddings and put that through a feed-forward network of some sort? - The effect of Embedding Separation Loss (Eq.5) seems quite limited. According to Table 2, it doesn*t seem to help much. And contrary to the authors* claim, using _x0008_eta increase the number of activated clusters from 8 to 8.4. - Most importantly, the central theme of this work is combining clustering with prediction labels for the downstream prediction task. But the authors do not compare the prediction performance of the proposed method with other clustering method or *patient similarity* methods, or even simple supervised models. The only prediction performance metric to be found is Table 2 from the ablation study.