Actor-Critic Approach for Temporal Predictive Clustering
Authors: Changhee Lee
Mihaela van der Schaar
Venue:  (2020)
Keywords: Temporal Clustering
Predictive Clustering
Actor-Critic

Abstract: Due to the wider availability of modern electronic health records (EHR), patient care data is often being stored in the form of time-series. Clustering such time-series data is crucial for patient phenotyping, anticipating patients’ prognoses by identifying “similar” patients, and designing treatment guidelines that are tailored to homogeneous patient subgroups. In this paper, we develop a deep learning approach for clustering time-series data, where each cluster comprises patients who share similar future outcomes of interest (e.g., adverse events, the onset of comorbidities, etc.). The clustering is carried out by using our novel loss functions that encourage each cluster to have homogeneous future outcomes. We adopt actor-critic models to allow “back-propagation” through the sampling process that is required for assigning clusters to time-series inputs. Experiments on two real-world datasets show that our model achieves superior clustering performance over state-of-the-art benchmarks and identifies meaningful clusters that can be translated into actionable information for clinical decision-making.

Under review as a conference paper at ICLR 2020
ACTOR-CRITIC APPROACH FOR TEMPORAL PREDIC- TIVE CLUSTERING
Anonymous authors Paper under double-blind review
ABSTRACT
Due the wider availability of modern electronic health records (EHR), patient care data is often being stored in the form of time-series. Clustering such time- series data is crucial for patient phenotyping, anticipating patients' prognoses by identifying "similar" patients, and designing treatment guidelines that are tailored homogeneous patient subgroups. In this paper, develop deep learning ap- proach for clustering time-series data, where each cluster comprises patients who share similar future outcomes of interest (e.g., adverse events, the onset of comor- bidities, etc.). The clustering is carried out by using our novel loss functions that encourage each cluster to have homogeneous future outcomes. We adopt actor- critic models to allow "back-propagation" through the sampling process that is required for assigning clusters to time-series inputs. Experiments on two real- world datasets show that our model achieves superior clustering performance over state-of of-the-an art benchmarks and identifies meaningful clusters that can be trans- lated into actionable information for clinical decision decision-making.
INTRODUCTION
Chronic diseases such cystic fibrosis, dementia, and diabetes are heterogeneous in nature, with widely differing outcomes even in narrow patient subgroups. Disease progression manifests through a broad spectrum of clinical factors, collected as a sequence of measurements over time in electronic health records (EHR), which gives a rise to complex progression patterns among patients (Samal et al., 1). For example, cystic fibrosis evolves slowly, allowing for the development of related comorbidities and bacterial infections, and creating distinct behaviors/responses to therapeu- tic interventions, which in turn makes the survival and quality of life substantially different (Ramos et al., 2017). Identifying patient subgroups with similar progression patterns can be advantageous for understanding such heterogeneous underlying diseases. This allows clinicians to anticipate pa- tients' prognoses by comparing "similar" patients in order to design treatment guidelines that are tailored to homogeneous patient subgroups (Zhang et al., 2019).
Temporal clustering has been recently used as data-driven framework to partition patients with time-series observations into a set of clusters (i.e., into subgroups of patients). Recent research has typically focused on either finding fixed-length and low-dimensional representations (Zhang al., 2019; Rusanov et al., 2016) or on modifying the similarity measure (Giannoula et al., 2018; Lu- ong and Chandola, 2017) both in attempt apply the conventional clustering algorithms (e.g.. K-means (Lloyd, 1982)) to time-series observations. However, clusters identified from these ap- proaches these approaches are purely unsupervised they do not account for each patient's 's observed outcome (c.g., adverse events, the onset of comorbidities, etc.) - which leads to heterogeneous clus- ters if the clinical presentation of the disease differs even for similar patients. Thus, common prognosis in each cluster remains unknown which can mystify the understanding of the underlying disease progression (Boudier et al., 2019). For instance, patients who appear to have similar time- series observations may develop different sets of comorbidities in the future which, in turn, require different treatment guidelines to reduce such risks (Wami et al., 2013). To overcome this limitation, focus predictive clustering (Blockeel et al., 2017) which combines prediction with clustering. Therefore, the cluster assignments are optimized such that patients in a cluster share similar future outcomes to provide prognostic value.

---

Under review as a conference paper at ICLR 2020
In this paper, we propose actor-critic approach for temporal predictive clustering, which we call AC-TPC. Our model consists of three neural networks - an encoder, a selector, and a predictor and set of centroid candidates. More specifically, the encoder maps imput time-series into a latent encoding; the selector utilizes the encoding and assigns a cluster to which the time-series belongs to via a sampling process; and the predictor estimates the future outcome distribution conditioned on either the encoding or the centroid of the selected cluster. The following three contributions render our model to identify the predictive clusters. First, to encourage each cluster have homogencous future outcomes, we define a clustering objective based on the Kullback- between the predictor's output given the input time series, and the predictor's ' output given estimated cluster assignments. Second, we transform solving a combinatorial problem of identifying cluster into iteratively solving two sub-problems: optimization of the cluster assignments and optimization of the cluster centroids. Finally, we allow ppocc--poppaaation" through the sampling process of the selector by adopting the training of actor-critic models (Konda and Tsitsiklis, 2000).
(KL) divergence
Throughout the experiments, we show significant performance improvements over the state-of-the- art clustering methods on two real-world medical datasets. Then, to demonstrate the practical sig- nificance of our model, we consider more realistic scenario where the future outcomes of interest are high-dimensional such as, development of multiple comorbidities in the next year and inter- preting all possible combinations is intractable. Our experiments show that our model can identify meaningful clusters that can be translated into actionable information for clinical decision-making
PROBLEM FORMULATION
Let X E X' and Y y be random variables for an input feature and an output label (i.e., one or a combination of future outcome(s) of interest) with joint distribution pxy (and marginal distri- butions are Px and py, respectively) where X' is the feature space and y is the label space. Here, we focus our description on C'-class classification tasks, i.e., y = {1..... ... C). ? We are given time-series dataset DD = {(x], )T J )/))))) comprising sequences of realizations (i.e., observations) of the pair (X, Y) for N patients. Here, (x", )("" is sequence of T observation pairs that cor- respond to patient and L E T _ (1.... , denotes the time stamp at which the observations are Fade.From this point forward, we omit the dependency on n when clear the context and denote X1:t = (x1.... xx)) for ease of notation.
Our aim is to identify set of K predictive clusters, = C(K)), ). for time-series data. Each cluster consists of homogeneous data samples, that can be represented by its centroid, based on a certain similarity measure. There are two main distinctions from the conventional notion of clustering. First, we treat subsequences of each times- as data samples and focus parti- tioning
J into C. Hence, we define a cluster as C(A: = (xT:el4 E T", s7 = k) for k E IC 2 {1,-... K} where sy E I. is the cluster assignment for a given x This is to flexibly update the cluster assignment (in real-time) to which patient belongs as new observations are being accrued over time. Second, we define the similarity measure with respect to the label distribution and associate it with clusters to provide a prognostic value. More specifically, we want the distribution of output label for subsequences in each cluster to be homogeneous and, thus, can be well-represented by the centroid of that cluster. Let be random variable for the cluster as- signment- that depends on given subsequence x and YIS k be random variable for the output given cluster k. Then, such property of predictive clustering can be achieved by minimizing the following Kullback-Leibler (KL) divergence:
KT x1 V. = for x1a E C(K)
1.. X1.0 V. k)= f p(yl x1x) logg(v)x1a) - log poggp(slll (s)))dy dy. Here, p(y)x1:) and p(y)s:) are the label distributions conditioned on a subsequence x1 and a cluster assignment $t, respectively. Note that (1) achieves its minimum when the two distributions are equivalent.
where (T
www.ummmmssio.... In this paper, we focus our description on C-class classification task, i..., y = [1,... C): in Appendix A, we discuss simple modifications of our model for regression and M-dimensional hinary classification tasks, i.e., IR and ) {0, 1) respectively.
Source code available https: /tt::/ggtttbb//b com.ttcooooooocccccooooooooooott CCIR2000
2

---

Under review as a conference paper at ICLR 2020
Actor
Critic
Encoder (fa)
Z:
Predictor (gg)
LSTM Cell
9c
e(s))
Selector (hgg)
Sampling
Embeddings (E)
Figure I: The block diagram of AC-TPC. The red line implies the procedure of estimating p(y]S; 8.) which includes a sampling process and the blue line implies that of estimating p(y Xi:t x1a))
Finally, establish our goal as identifying of predictive clusters C' that optimizes the following objective:
minimize
Σ
KI X1 YiIS: k:).
XIIrE((K)
Unfortunately, the optimization problem (2) is highly non-trivial. We need to estimate the objec- tive function in (2) while solving non-convex combinatorial problem of finding the optimal cluster assignments and cluster centroids.
3 ACTOR-CRITHE APPROACH FOR TEMPORAL PREDICTIVE CLUSTERING
To effectively estimate the objective function in (2), introduce three networks encoder, selector, and a predictor and an embedding dictionary as illustrated in Figure These components together provide the cluster assignment and the corresponding centroid based given sequence of observations and enable us to estimate the probability density p(yl (t). More specifically, we define each component follows:
The encoder, Se I X - 2, RNN (parameterized by 0) that maps a (sub)sequence of time-series X1:t to a latent representation (i.c., encoding) z, Z where Z is the latent space.
The selector, h3 2 - K-1, fully-connected network (parameterized by v)) that pro- vides probabilistic mapping to categorical distribution from which the cluster assignment 8; is being sampled.
The predictor, 9o Z + C-1, is fully-connected network (parameterized by Φ) that esti- mates the label distribution given the encoding of a time-series the centroid of a cluster.
The embedding dictionary, E (e(1), e(K)) where e(k) E Z for k K,, is set of cluster centroids lying in the latent space which represents the corresponding cluster.
Here, D-1 = {q [o, 1]K +... dp = 1} is (D - 1)-simplex that denotes the probability distribution for D-dimensional categorical (class) variable.
At each time stamp the encoder maps a input (sub)sequence X into latent encoding z, A fo(x):). Then, based on the encoding 2,, the cluster assignment 8, is drawn from a categorical dis- tribution that is defined by the selector output, i.c., S; Cat(TT) where т. = [me[L),-
.T. [((K] 2 (2.). Once the assignment 8; is chosen, we allocate the latent encoding z; to an embedding e(st) in the embedding dictionary E. Since the allocated embedding o(st) corresponds to the centroid of the cluster to which 1 belongs, we can, finally, estimate the density p(yl 8t) in (2) as the output of predictor given the embedding e(s)), i.e., T/ go (e(+)))
3.1 LOss FUNCTIONS
In this subsection, we define loss functions to achieve our objective in (2); the details of how we train model will be discussed in the following subsection.

---

Under review as a conference paper at ICLR 2020
Predictive Clustering Loss: Since finding the cluster assignment of given sequence is prob- abilistic problem due to the sampling process, the objective function in (2) must be defined as an expectation over the cluster assignment. Thus, can estimate solving the objective problem in (2) as minimizing the following loss function:
T E Can Dt]]
(1, 1100,000 Φ. = Expexpxr
(3)
where (y), Dt) = - Vff lo. Here, we slightly abuse the notation and denote y = l.'.... as the one-hot encoding of y, and yf and T indicates the e-th component of and y, re- spectively. is worth to highlight that minimizing & is equivalent to minimizing the KL divergence in (2) since the former term of the KI. divergence is independent of our optimization procedure.
One critical question that may arise is how to avoid trivial solutions this unsupervised setting of identifying the cluster assignments and the centroids (Yang et al., 2017). For example, all the embeddings in may collapse into single point or the selector simply assigns equal probability to all the clusters regardless of the input sequence. In both cases, our model will fail to correctly estimate p(y st) and, thus, end up finding trivial solution. auxiliary loss functions that are tailored to address this concern. It is worth highlight that these loss functions are not subject to the sampling process and their gradients can be simply back mack-propagated
address this issue, we introduce two
Sample- Wise Entropy of Cluster Assignment: To motivate sparse cluster assignment such that the selector ultimately selects one dominant cluster for each sequence, we introduce sample-wise entropy of cluster assignment which is given as
C2(0, Expxux
Σ (e(k)log कe (k)
KEK
where Te [1) ( (K)) = hy (fo( when Ne becomes an one-hot vector.
The sample-wise entropy achieves its minimum
To prevent the embeddings in & from collapsing into a single point, we define loss function that encourages the embeddings represent different label distributions, i.e, 3o (c(k)) for k E K., from each other:
Embedding Separation Loss:
Ca(E) = Σ ((gg(e(k))
where 4 is reused quantify the distance between label distributions conditioned on each cluster. We minimize (5) when updating the embedding vectors e(1),..,,e(
3.2 OPTIMIZATION
The optimization problem in (2) is a non-convex combinatorial problem because it comprises not only minimizing the KL divergence but also finding the optimal cluster assignments and centroids. Hence, we propose an optimization procedure that iteratively solves two subproblems: i) optimizing the three networks the encoder, selector, and predictor while fixing the embedding dictionary and ii) optimizing the embedding dictionary while fixing the three networks. Pseudo-code of AC- TPC can be found in Appendix
3.2.1 OPTIMIZING THE THREE NETWORKS -fo- hig AND J
Finding predictive clusters incorporates the sampling process which is non-differentiabl. Thus, to render pook-ppropagation.. we utilize the training of actor-critic models (Konda and Tsitsiklis, 2000). More specifically, we view the combination of the encoder (fo) and the selector (hw) as the "actor" parameterized by W [0, v²), and the predictor (g²) as the "critic". The critic takes as input the the output of the actor (i.e., the cluster assignment) and estimates its value based on the sample- wise predictive clustering loss (i.e., (1 Øc)) given the chosen cluster. This, in turn, renders the actor to change the distribution of selecting a cluster to minimize such loss. Thus, it is important for the critic to perform well the updated output of the actor while it is important for the actor
4

---

Under review as a conference paper at ICLR 2020
to perform well on the updated loss estimation. As such, the parameters for the actor and the critic need to be updated iteratively.
Given the embedding dictionary & fixed (thus, will omit the dependency on E), train the actor, ie., the encoder and the selector, by minimizing a combination the predictive clustering loss C and the entropy of cluster assignments L2, which is given by
LA v5, の) - L (0, v. Φ) + (0,
(6)
where a 2 is coefficient chosen to balance between the two losses. To derive the gradient of this loss with respect WA [0, v².. we utilize the ideas from actor-critic models (Konda and Tsitsiklis, 2000) as follows; please refer to Appendix B for the detailed derivation:
)]
Ca Exyydxx
VWA
Ee Cal { + Vev
Σ
Ex.
En (Ye, Dc) De Vw log bg
+ -2V0,
Note that since no sampling process is considered in L2(0, v),, we can simply derive V_, C2(0, 1))
Iteratively with training the actor, we train the critic, i.e., the predictor, by minimizing the predictive clustering loss L1 as the following:
Cc(0)= (1,
whose gradient with respect to Φ can be givens as (d) = V (0, v. Note that since the critic is independent of the sampling process, the gradient can be simply back mnok-ppro.gated.
3.2.2 OPTIMIZING THE CLUSTER CENTROIDS
Now, once the parameters for the three networks (0, 4/, Φ) are fixed (thus, we omit the dependency on 0, v), and Φ), we updated the embeddings in & by minimizing a combination of the predictive clustering loss L, and the embedding separation loss L3. which given by
(E) (E) + (E)
where 8 2 0 is coefficient chosen to balance between the two losses.
3.2.3 INITIALIZING AC-TPC VIA PRE- TRE-TRAINING
Since we transform the non-trivial combinatorial optimization problem in (2) into iteratively solving two sub-probe orobproblems, initialization is crucial achieve better optimization a similar concern has been addressed in (Yang et al., 2017).
Therefore, initialize model based on the following procedure. First, we pre-train the encoder and the predictor by minimizing the following loss function based on the predicted label distribution given the latent encodings of input sequences, i.e., ÿr - (z( = (x):))) as the following:
Σ ( Dc)
Ci(0,0) = Expxxxy
(10)
Minimizing (10) encourages the latent encoding to be enriched with information for accurately pre- dicting the label distribution. Then, perform K-means (other clustering method be also applied) based on the learned representations to initialize the embeddings and the cluster assign- ments { {( 17
{ " J. Finally, we pre- pre-train the selector hy by minimizing the cross entropy treating the initialized cluster assignments as the true clusters.
4 RELATED WORK
Temporal clustering, also known as clustering, is a process of unsupervised partitioning of the time-series data into clusters in such way that homogeneous time-series are grouped together
5

---

Under review as a conference paper at ICLR 2020
based on certain similarity measure. Temporal clustering is challenging because i) the data is often high-dimensional - consists of sequences not only with high-ding high-dimensional features but also with many time points and ii) defining proper similarity measure for time-series is not straightforward since it is often highly sensitive to distortions (Ratanamahatana et al., 2005). To address these chal- lenges, there have been various attempts to find good representation with reduced dimensionality or to define a proper similarity measure for times-series (Aghabozorgi et al., 2015).
Recently, Baytas et al. (2017) and Madiraju al. (2018) proposed temporal clustering methods that utilize low-dimensional representations learned by RNNs. These works are motivated by the success of applying deep neural networks find "clustering friendly" latent representations for clustering static data (Xie et al., 2017; Yang et al., 2017). In particular, Baytas et al. (2017) utilized a modified L.STM auto-encoder to find the latent representations that are effective to summarize the input time-series and conducted K-means on top of the learned representations as an ad-hoc process. Similarly, Madiraju al. (2018) proposed a bidirectional--SST the reconstruction loss for dimensionality reduction and the clustering objective. However, these methods do not associate a target property with clusters and, thus, provide little prognostic value in understanding the underlying disease progression.
auto-encoder that that jointly optimizes
Our work is most closely related to SOM-VAE (Fortuin et al., 2019). This method jointly opti- mizes static variational auto-encoder (VAE), that finds latent representations of imput features, and a self-organizing map (SOM), that allows to map the latent representations into a more interpretable discrete representations, i.e., the embeddings. However, there are three key differences between our work and SOM-VAE, First, SOM-VAE aims at minimizing the reconstruction loss that is specified as the mean squared error between the original input and the reconstructed input based on the cor- responding embedding. Thus, similar to the aforementioned methods, SOM-VAE neither associates future outcomes of interest with clusters. In contrast, we focus on minimizing the KL divergence between the outcome distribution given the original input sequence and that given the correspond- ing embedding to build association between future outcomes of interest and clusters. Second, to overcome von-differentiabbiiyy caused by the sampling process (that is, mapping the latent represen- tation to the embeddings), Fortuin et al. (2019) applies the gradient copying technique proposed by (van den Oord et al., 2017), while we utilize the training of actor-critic model (Konda and Tsitsiklis, 2000). Finally, while we flexibly model time-series using LSTM, SOM-VAB handles time-series by integrating Markov model in the latent representations. This can be a strict assumption espe- cially in clinical settings where a patient's 's medical history is informative for predicting his/her future clinical outcomes (Ranganath al., 2016).
EXPERIMENTS
In this section, we provide a set of experiments using two real-world time-series datasets. We itera- tively update the three networks- the encoder, selector, and predictor and the embedding dictio- nary as described in Section 3.2. For the network architecture, we constructed the encoder utilizing a single-layer LSTM (Hochreiter and Schmidhuber, 1997) with 50 nodes and constructed the selector and predictor utilizing two-layer fully-connected network with 50 nodes in each layer, respectively. The parameters (0, vi, Φ) are initialized by Xavier initialization (Glorot and Bengio, 2010) and opti- mized via Adam optimizer (Kingma and Ba, 2014) with learning rate of 0.001 and keep probability 0.7. We chose the balancing coefficients a, B E {0.1, 1.0, 3.0} utilizing grid search that achieves the minimum validation loss in (3); the effect of different loss functions are further investigated in the experiments. Here, all the results are reported using 5 random 64/16/20 crain/aliiation/test splits.
5.1 REAL-WORLDD DATASETS
We conducted experiments to investigate the performance of AC-TPC on two real-world medical datasets; detailed statistics of each dataset can be found in Appendix C:
UK Cystic Fibrosis registry (UKCF)): This dataset records annual follow-ups for 5,171 adult patients (aged 18 years or older) enrolled in the UK CF registry over the period from 2008 and 2015, with a total of 25,012 hospital visits. Each patient associated with 89 variables (i.e., 11 static and 78 varying features), including information on demographics and genetic
hhttps://ww.cystiiee. of .rg. ww/the worckeewww----ee we-do/uk- wtw-ddokukf--rgiiiiee cf-registryy

---

Under review as a conference paper at ICLR 2020
mutations, bacterial infections, lung function scores, therapeutic managements, and diagnosis on comorbidities. We set the development of different comorbidities in the next year as the label of interest at each time stamp.
Alzheimer's Disease Neuroimaging Initiative (ADNI)*: This dataset consists of 1,346 patients the Alzheimer's disease study with total of 11,651 hospital visits, which tracks the discase progression via follow-up observations at 6 months interval. Each patient is associated with 21 variables (i.e., static and 16 time-varying features), including information on demographics, biomarkers brain functions, and cognitive test results. We set predictions on the three diag- nostic groups - normal brain functioning, mild cognitive impairment, and Alzheimer's ' disease- the label of interest each time stamp.
5.2 BENCHMARKS
We compare AC-TPC with clustering methods ranging from conventional approaches based on K- means to the state-of-the approaches based on deep neural networks. All the benchmarks com- pared in the experiments are tailored incorporate time-series data described below:
Dynamic time warping followed by K-means: Dynamic time warping (DTW) is utilized to quantify pairwise distance between two variable-length sequences and, then, K-means is applied (denoted as KM-DTW).
K K-means with deep neural networks: To handle variable-length time-series data, utilize our encoder and predictor that are trained based on (10) for dimensionality reduction; this is provide fixed-length and low-dimensional representations for time-series. Then, apply K-means on the latent encodings z (denoted as KM-E2P (Z)) and on the predicted label distri- butions & (denoted as KM-E2P ())), respectively.
Extensions of DCN (Yang al., 2017): Since the DCN is designed for static data, replace their static auto-encoder with sequence-to-sequence network to incorporate time-series data (denoted as DCN-S2S). In addition, to associated with the label distribution, we compare DCN whose static auto-encoder is replaced with our encoder and predictor (denoted as DCN- E2P) to focus dimensionality reduction while preserving information for predicting the label.
SOM-VAE (Fortuin et al., 2019): We compare with SOM-VAE- though, this method is oriented towards visualizing input data via SOM since it naturally clusters time-series data (denoted as SOM-VAE). In addition, we compare with a variation of SOM- by replacing the decoder with our predictor in order to find embeddings that capture information for predicting the label (denoted as SOM-VAF-P). For both cases, we set the dimension of SOM to K.
worth highlighting that the label information is provided for training DCN-E2P, KM-F2P, and SOM-VAE-P while the label information is not provided for training KM-DTW, DCN-S2S, and SOM-VAE, Please refer to Appendix for the summary of major components of the tested bench- marks and the implementation details.
5.3 PERFORMANCE METRICS
To assess the clustering performance, we applied the following three standard metrics for evaluating clustering performances when the ground-truth cluster label is available: purity score, normalized mutual information (NMI) (Vinh et al., 2010), and adjusted Rand index (ARI) (Hubert and Arabie, 1985). More specifically, the purity score assesses how homogeneous each cluster is (ranges from to where being a cluster consists of a single class), the NMI is an information theoretic measure of how much information is shared between the clusters and the labels that is adjusted for the number of clusters (ranges from 0 to 1 where being a perfect clustering). and ARI corrected-for -ohhcceeeee version of the Rand index which is measure of the percentage of correct cluster assignments (ranges from -1 where 1 being perfect clustering and 0 being random clustering). In nutshell, all the three performance metrics are commonly used but all have its pros and cons; for instance, the
'https: adni 'This extension is a representative of recently proposed deep learning approaches for clustering of both static data (Xie et al., 2017; Yang et al., 2017) and time-series data (Baytas et al., 2017; Madiraju et al., 2018) since these methods are built upon the same concept that is, applying deep networks for dimensionality reduction conduct conventional clustering methods, e.g., K-means.
7

---

Under review as a conference paper at ICLR 2020
Table 1: Performance Comparison on the UKCF and ADNI datasets.
Dataset
Method
Purity
NMI
ARI
AUROC
AUPRC
KM:DTW
0.37
0.01000011
0.14=0001
N/A
NIA
KM-E2P (2)
0.719=0.01
0.211=0.01
0.107=0.01
0.726=00.11
0.425=0.002
KM-E2P
0.751
0.325
0.40=0002
0.807:10.00"
0.514:10.01
DCN $28
0.059 0.008
0.063
NIA
N/A
UKCF
DCN-E2P
0.751=0002
0.275 0.002
0.184=0011
0.772:+0.03"
0.487:20003
SOM-VAE
0.573
0.000
0.00610.011
NIA
NIA
SOM VAE-P
0.638
0,201 0.055
0.283-10.17
0.754-10.05
0.007
Proposed
0.80
0.463=00.01
0.602-0.01
0.843-0001
0.605=10.01
KM-DTW
0.506=0.02
0.019=0.02
0.00600.02
NIA
KM-E2P (2)
0.736=0.03
0.24
0.230=0.03
0.707
0.509=0.01
KM-E2P (3))
0.056
0.264
0.11
0.756:10.04
0.503-10.04
ADNI
DCN-S28
0.567=0.02
0,005 =0.00
0.000=0.01
NIA
DCN-E2P
0.749=0.06
0,261
0,215=0.06
0.721=20.031
0.509=20.03
SOM VAE
0.566
0.04
0.011
NIA
N/A
SOM- VAE-P
0,586=0.06*
0.085 0.008
0.038=00066
0.597-+00110
0.376=2005
Proposed
0.786 4.786:0.03
0,285 0.04
0.330 0.06
0.768.10000
0.015:10.02
indicates p-value < 0.01.
indicates p-valuc < 0.05
6.00
0.43
072
0.23
616
0:5
640
0117
80
P
humber
(a) The averaged purity score.
(h) The averaged NMI.
(c) The averaged ARI.
Figure 2: The purity score, NMI, and ARI (mean and 95% confidence interval) for the UKCF dataset
(C = 8) with various K.
purity score easily converges to when there are as many clusters as data samples. Thus, using them together suffices to demonstrate the effectiveness of the clustering methods.
To assess the prediction performance of the identified predictive clusters, we utilized both area un- der receiver operator characteristic curve (AUROC) and area under precision-recall curve (AUPRC) based on the label predictions of each cluster and the ground-truth binary labels on the future out- comes of interest. Note that the prediction performance is available only for the benchmarks that incorporate the label information during training.
5.4 CLUSTERING PERFORMANCE
We start with a simple scenario where the true class (i.e., the ground-truth cluster label) is available and the number of classes is tractable. In particular, the development of three common comorbidities of cystic fibrosis diabetes, ABPA, and intestinal obstruction in the next year for the UKCF dataet and three diagnostic groups for the ADNI dataset. We compare AC-TPC against the aforementioned benchmarks with respect the clustering and prediction performance in Table
C 2 8 based the binary labels for
3 based on the mutually exclusive
As shown in Table 1, AC-TPC achieved performance gain over all the tested benchmarks in terms of both clustering and prediction performance where most of the improvements were statistically sig- nificant with y-value < 0.01 p-value < 0.05 for both datasets. Importantly, clustering methods -.e.. KM-DTW, DCN-S2S, and SOM-VAE that not associate with the future outcomes of terest identified clusters that provide little prognostic value the future outcomes (note that the true class is derived from the future outcome interest). This is clearly shown by the ARI value near O which indicates that the identified clusters have difference with random assignments. There- fore, similar sequences with respect to the latent representations tailored for reconstruction or with respect the shape-based measurement using DTW can have very different future outcomes.
In Figure 2, we further investigate the purity score, NMI, and ARI by varying the number clusters K from 4 to 16 the UKCF dataset in the same setting with that stated above (i.e., C' = 8).
8

---

Under review as a conference paper at ICLR 2020
Here, the three methods i.., KM-DTW, DCN-S2S, and SOM-VAE are excluded for better visualization. As we can see in Figure 2, our model rarely incur performance loss in both NMI and ARI while the benchmarks (except for SOM-VAE-P) showed significant decrease in the performance as K increased (higher than C). This is because the number of clusters identified by AC-TPC (i.e., the number of activated clusters where define cluster l is activated if [C(K) > 0) was the same with C' most of the times, while the DCN-based methods identified exactly K clusters (due to the K K-means). Since the NMI and ARI are adjusted for the number of clusters, smaller number of identified clusters yields, if everything being equal, a higher performance. In contrast, while our model achieved the same purity score for K 8, the benchmark showed improved performance as K increased since the purity score does not penalize having many clusters. This is an important property of AC-TPC that we do not need to know a priori what the number of cluster is which is a common practical challenge of applying the conventional clustering methods (e.g., K K-means).
The performance gain of our model over VAE) comes from two possible sources: i) SOM-VAE-P mainly focuses on visualizing the input with SOM which makes both the encoder and embeddings less flexible - this is why it performed better with higher K and ii) the Markov property can be too strict assumption for time-series data especially in clinical settings where a patient's 's medical history is informative for predicting his/her future clinical outcomes (Ranganath et al., 2016).
(and, our analysis is the same for SOM-
5.5 CONTRIBUTIONS OF THE AUXILIARY LOSS FUNCTIONS
As described in Section 3.1, I, introduced two auxiliary loss functions - the sample-wise entropy of cluster assignment (4) and the embedding separation loss (5) to avoid trivial solution that may arise identifying the predictive clusters. To analyze the contribution of each auxiliary loss function, we report the average number of activated clusters, clustering performance, and prediction performance on the UKCF dataset with 3 comorbidities as described in Section 5.4. Throughout the experiment, we set K 16 which is larger than C - to find the contribution of these loss functions to the number of activated clusters.
Table 2: Performance comparison with varying the balancing coefficients a, 3 for the UKCF dataset.
Coefflcients
Clustering Performance
Prognostic Value
Activated No.
Purity
NMI
ARI
AUROC
AUPRC
0.0
0.0
16
0.073
0.00610.000
0.000 0.000
0.500
0.169
0.0
1.0
0.573110.01
0.006:10.00
0.000-10.00
0.500-10.00
0.169:10.00
3.0
0.0
8.4
0.795=00001
0.431+0.01
0.569=10.01
0.840=10001
0.583=+0.02
3.0
0.808:10001
0.468:10001
0.606:10.01
0.852:10.00
0.608::0.01
As we can see in Table 2, both auxiliary loss functions make important contributions in improving the quality of predictive clustering. More specifically, the sample- wise entropy encourages the selec- can results with 0, without the sample-wise
tor to choose one dominant cluster. Thus, entropy, our selector assigns an equal probability to all 16 clusters which results in a trivial solution. We observed that, by augmenting the embedding separation loss (5), AC-TPC identifies smaller number of clusters owing to the well-separated embeddings; in Appendix E, we further investigate the usefulness of (5) in identifying the number of clusters data-driven fashion.
5.6 TARGETING MULTIPLE FUTURE OUTCOMES A PRACTICAL SCENARIO
In this experiment, we focus on more practical scenario where the future outcome of interest is high-dimensional and the number of classes based on all the possible combinations of future outcomes becomes intractable. For example, suppose that we are interested in the development of M comorbidities in the next year whose possible combinations grow exponentially C = M Interpreting such large number of patient subgroups will be a daunting task which hinders the understanding of underlying disease progression. Since different comorbidities may share common driving factors (Ronan et al., 2017), we hope our model to identify much smaller underlying (latent) clusters that govern the development of comorbidities. Here, to incorporate with M comorbidities (i.e.. M binary labels), we redefine the output space as y = {0, 1) MM and modify the predictor and loss functions, accordingly, as described in Appendix A.

---

Under review as a conference paper at ICLR 2020
Table 3: The top-3 frequent comorbidities developed in the next year for the 12 identified clusters. The values in parentheses indicate the corresponding frequency.
Clusters
Top-3 Frequent Comorbidities
0
Diabetes (0.85)
Liver Enzymes (0.21)
Arthrupahhy (0.14)
Liver Enzymes (0.09)
Anthropathy (0.08)
Depression (0.07)
2
ABPA (0.77)
Osteopenia (0.21)
Intestinal Obstruction (0.11)
(0.89)
Liver
(0.87)
(0.29)
4
Osteoporosis (0.76)
Diabetes (0.43)
Arthropathy (0.20)
Asthma (0.83)
Diabetes (0.81)
Ostcopeaia (0.28)
Liver Discuse (0.85)
Asthma (0.03)
ABPA (0.09)
7
ABPA (0.83)
Diabetes (0.78)
Osteupenia (0.25)
Diabetes (0.94)
Liver Discase (0.83)
Liver Enzymes (0.43)
9
Asthma (0.89)
Ostcopenia (0.26)
ABPA (0.19)
Osteapenia (0.82)
Diabetes (0.81)
Anhropathy (0.23)
11
Ostcopenia (0.77)
Liver Enzymes (0. (0.18)
Arthropathy (0.12)
3.0
20ima
AdPA
3.6
Ohteoperts
Clustor
Cluster
Clusse
Cluster 9
Cluste 10
Figure 3: Clusters with high-risk of developing diabetes. We reported the cluster-specific frequen- cies of developing comorbidities - liver disease, asthma, ABPA, and osteopenia that are co-occurred with diabetes in the year.
Throughout this experiment, aim at identifying subgroups of patients that associated with the next-year development of 22 different comorbidities the UKCF dataset. In Table 3, we reported 12 identified clusters average, the number of activated clusters were 13.6 and the top three frequent comorbidities developed in the next year since the latest observation and the corresponding frequency; please refer Appendix E for full list. Here, the frequency is calculated in cluster- specific fashion based on the true label. As we can see in Table 3, the identified clusters displayed very different label distributions; that is, the combination of comorbidities as well as their frequency were very different across the clusters. For example, patients in Cluster 1 experienced low-risk of developing any comorbities in the next year while 85% of patients in Cluster 0 experienced diabetes in the next year.
In Figure 3, we further investigated subgroups of patients Cluster 5, 8, and 10 who had high risk of developing diabetes in the next year. Although all these clusters displayed high risk of diabetes, the frequency of other co-occurred comorbidities was significantly different across the clusters. In particular, around 89% of the patients in Cluster 5 experienced asthma in the next year while it was less than 3% of the patients the other cluster. Interestingly, "leukotriene" medicine commonly used manage asthma and "FEV, % predicted" measure of lung function were the two most different input features between patients in Cluster and those the other clusters. We observed similar findings Cluster 7 with ABPA, Cluster 8 with liver disease, and Cluster 10 with osteopenia. Therefore, by grouping patients who are likely to develop similar set of comorbidities, our method identified clusters that can be translated into actionable information for clinical decision decision-making.
6 CONCLUSION
In this paper, we introduced AC-TPC, a novel deep learning approach for predictive clustering of time-series data. We carefully defined novel loss functions encourage each cluster have ho- mogeneous future outcomes (e.g., adverse events, the onset of comorbidities, etc.) and designed optimization procedures to address challenges and avoid trivial solution identifying such clus- ter assignments and the centroids. Throughout the experiments showed that our model achieves superior clustering performance over state-of-the-arr and identifies meaningful clusters that be translated into actionable information for clinical decision decision-making.
two real-world datasets,
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
L. Samal, A. Wright, B. Wong, J. Linder, and D. Bates. Leveraging electronic health records to support chronic disease management: the need for temporal data views. Informatics Primary Care, 19(2):65-44, 2011.
J. Ramos, B. S. Quon, S. L.. Heltshe, N. Mayer-Hamblett, E. D. Lease, M. L. Aitken, N. S. Weiss, and C. H. Goss. Heterogeneity in survival in adult patients with cystic fibrosis with FEV1 < 30% June 2017.
of predicted in the united states. Chest, 13206))
Xi Zhang, Jingyuan Chou, Jian Liang, Cao Xiao, Yize Zhao, Harini Sarva, Claire Hencheliffe, and Fei Wang. Data-driven subtyping of parkinson's 's disease using longitudinal clinical records: A cohort study. Scientific Reports, 9(797):1-12, January 2019.
A. Rusanov, P. V. Prado, and C. Weng. Unsupervised time-series clustering over lab data for au- tomatic identification of uncontrolled diabetes. In Proceedings of the 4th IEEE International Conference Healthcare Informatics (ICHI), 2016.
A. Giannoula, A. Sutierrez-Saeristán,, Á. Bravo, F. Sanz, and L. I. Furlong. Identifying temporal patterns in patient disease trajectories using dynamic ping: A population-basee study, Scientific Reports, 8(4216): 1-14, March 2018.
D. T. A. Luong and V. Chandola. A k-means approach clustering disease progressions. In Pro- ceedings of the Sth IEEE International Conference on Healthcare Informatics (ICHI), 2017.
S. Lloyd. Least squares quantization in pem. IEEE Transaction on Information Theory, 28(2): 129-137, March 1982.
A. Boudier, S. Chanoine, S. Accordini, M. Anto, X. Basaga na, J. Bousquet, P. Demoly, J, Garcia- Aymerich, F. Gormand, J. Heinrich, C. Janson, N. Künzli, R. Matran, C. Pison, C. Raherison, J. Sunyer, R. Varraso, D. Jarvis, B. Leynaert, I. Pin, and V. Siroux. Data-driven adult asthma phenotypes based on clinical characteristies are associated with asthma outcomes twenty years later. Allegy, 74(5):953-963, May 2019.
W. M. Wami, Buntinx, S. Bartholomceusen, G. Goderis, C. Mathieu, and M. Aerts. Influence of chronic comorbidity and medication on the efficacy of treatment in patients with diabetes in general practice. The British Journal of General Practice, 63(609):267-2777 March 2013.
H. Blockeel, S. Dzeroski, J. Struyf, and B. Zenko. Predictive Clustering. Springer New York, 2017.
V. R. Konda and J. N. Tsitsiklis. Actor-critic algorithms. In Proceedings of the 13th Conference on Neural Information Processing Systems (NIPS 2000), 2000.
B. Yang. X. Fu, N. D. Sidiropoulos, and M. Hong. Towards k-means-friendly spaces: Simultancous deep learning and clustering. In Proceedings of the 34th International Conference on Machine Learning (ICML 20/7), 2017.
C. A. Ratanamahatana, F. Keogh, A. J. Bagnall, and S. Lonardi. A novel bit level time series representation with implications for similarity search and clustering. In Proceedings of the 9th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD 2005), 2005.
Aghabozorgi. A. S. Shirkhorshidi, and T. Y. Wah. Time-scries clustering a decade review. Information Systems, 53:16-38, May 2015.
I. M. Baytas, C. Xiao, X. Zhang, F. Wang, Istm networks. In Proceedings of the 23rd ACM SIGKDD Conference Knowledge Discovery and Data Mining (KDD 2017), 2017.
Jain, and Zhou. Patient subtyping via time-aware
N. S. Madiraju, S. M. Sadat, D. Fisher, and H. Karimabadi. Deep temporal clustering: Fully unsu- pervised learning of time-domain features. arXiv preprint arXiv:1802.01059, 2018.
Xie, R. Girshick, and A. Farhadi. Unsupervised deep embedding for clustering analysis. In Proceedings of the 33rd International Conference on Machine Learning (ICMI.2016), 2017.

---

Under review as a conference paper at ICLR 2020
V. Fortuin, M. Hüser, F. Locatello, H. Strathmann, and G. Rätsch. SOM-VAE. Interpretable discrete representation learning on time series. In Proceedings of the 7th International Conference on Learning Representations (ICLR 2019),
A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning, Pro- ceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), 2017.
R. Ranganath, A. Perotte, N. Elhadad, and D. Blei. Deep survival analysis. In Proceedings of the 1st Machine Learning for Healthcare Conference (MLHC 2016), 2016.
Hochreiter and Schmidhuber. Long short-term memory. Neural Computation, 935):1335-1780, 1997.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AIS- TATS 2010), 2010.
D. Kingma and J. Ba. arXiv: 1412,6900, 2014.
Adam: method for stochastic optimization. arXiv preprint
N. X. Vinh, J. Epps, and J. Bailey. Information theoretic measures for clusterings comparison: Vari- ants, properties, normalization and correction for chance. Journal of Machine Learning Research, 11(1):2837-2854, October 2010.
L. Hubert and Arabie. Comparing partitions. Journal of Classification, 2(1):193-218, December 1985.
N.J. Ronan, J. Elborn, and B. J. Plant. Current and emerging comorbidities in cystic fibrosis. Presse Med., 1656):1 June 2017.
Peter J. Roussecuw. Silhouettes: a graphical aid to the interpretation and validation of cluster anal- ysis. Computational and Applied Mathematics, 20:53-65, 1987.
12

---

Under review as a conference paper at ICLR 2020
A VARIATIONS FOR REGRESSION AND BINARY CLASSIFICATION TASKS
As the task changes, estimating the label distribution and calculating the KL divergence in (2) must be redefined accordingly:
R and replace by (y,, 9t) l[ye 3el.3 Minimizing (1e,, Be) is equivalent minimizing the KL divergence between p(yr x1:z) and p(ye (se) when we assume these probability densities follow Gaussian distribution with the same variance.
For regression task, i.e., V = R. we modify the predictor as 9o
For the M-dimensional binary classification task, i.e., 3/ {0, IJM we modify the predictor as De : - [0, 1] 1] and replace & by & (3t, Dr) = >m= yt" log yi + (1 (11-y/ y2" "0o"""""oo"" log(1 - "t") which is required to minimize the KL. divergence. Here, y!"" and yf" indicate the m--th element of y and Yt. respectively. The basic assumption here is that the distribution of each binary label is independent given the input sequence.
B DETAILED DERIVATION OF (7)
To derive the gradient of the predictive clustering loss in (7) with respect WA = [0, vb) we utilized the ideas from actor-critic models (Konda and Tsitsiklis, 2000). The detailed derivation of the former term (7) is described below (for notational simplicity, we omit the expectation on Ex,y-pxy)
V
E. BarCCal(se (4 V.
Σ
Σ
(e(Se)l1 Ite
1=1 s
V (WTT(st)(
KEEK
(WIT(St) me (1) De)
(11)
T((st)
t=
Te (s:) De) V. log (s:))
E,, E,, (16, V. log m((:)
C DETAILS ON THE DATASETS
C.I UKCF DATASET
UK Cystic Fibrosis registry (UKCF)* records annual follow-ups for 5,171 adult patients (aged 18 years or older) over the period from 2008 and 2015, with total of 25,012 hospital visits. Each pa- tient is associated with 89 variables (i.e., 11 static and 78 time-varying features), including informa- tion on demographics and genetic mutations, bacterial infections, lung function scores, therapeutic managements, and diagnosis on comorbidities. The detailed statistics are given in Table 4.
C.2 ADNI DATASET
Alzheimer's Disease Neuroimaging Initiative (ADNI)² study consists of 1,346 patients with a total of 11,651 hospital visits, which tracks the disease progression via follow-up observations at 6 months interval. Each patient is associated with 21 variables (i.c., 5 static and 16 time-varying features), including information demographics, biomarkers on brain functions, and cognitive test results.
"https://ww.ccye http:/wwwsss...oooooo.../.. 'https://adni Toni usc.edu
13

---

Under review as a conference paper at ICLR 2020
Table 4: Summary and description of the UKCF dataset.
STATIC COVARIATES
Type
Freq.
Type
Freq-
Demographic
Gender
Bin.
0.35
Genetic
Class Mutation
Bin.
0.05
Class VI Mutation
Bin.
0.36
Class II Mutation
Bin.
0.87
DF508 Mutation
Bin.
0.87
Mutation
Bin.
GS5LD Mutation
Class IV Muration
Bin.
0.05
Homorygons
Bin.
0.58
Class V Mulstion
Bin.
0.04
Heterozygous
Bin
0.42
TIEEINARYING COVARIATES
Mean
Type
(Freq.)
Min Max
Type
(Freqq)
Min / Max
Demographic
Age
Cont.
30.4
18.07660 N6.0
Height BMI
Cons.
168.0
129.07 198.6
Weight
Cont.
30.0
Smoking Starus
Bin.
0.1
Lung Func. Scores
FEV.
Cont.
233
027633
Best FEV,
Cont.
2,5
0.378.0
HEV, %
Cont.
9.07
Rest FEV, %
Cons.
71.2
7.5/ 164.3
Hespitalizatton
IV ABX Days Hosp.
Cont.
1233
07431
Non.IV Hosp. Adm.
Cons.
1.2
07 2005
ABX Days Home
Cont.
11.9
0/441
Lung Infections
B. Cepacia
Bin.
0.05
P. Actuginosa
Bin.
0.59
H. Influenza
Bin.
0.05
K. Pneumonine
Bin.
0.00
Coli
Bin.
0.01
ALCA
Bin.
Aspergillus
Bin.
0.14
NTM Xamhomonas
Bin.
0.03
Gran-Negative
Bin.
S. Aurcus
Bin.
0.30
Comorbidities
Liver Diccsse
Bin.
0.16
Depression
Bin.
0.07
Hemoptysis
ABPA
Bin.
0.12
Pancreatilus
Bin.
0.01
Hypertension
Bin.
0,04
Hearing Loss
Bia.
0.03
bladder
Arthropally
Bin.
0.09
Colonic structure
Bin.
0.00
Bone fracture
Bin.
0.01
Intest. Obstruction
Bin.
0.08
Osteoporosis
Bin.
0.09
Gl bloed no var.
Bin.
0.00
Osteopenia
Bin.
0.21
Gl bleed var.
Bin.
0.00
Cancer
Bin.
0.00
Liver Enzymes
Bin.
0.16
Cirrhsis
Bin.
0.05
Kidney Stones
Bin.
0.02
Treutments
Domase Alpha
Bin.
0.56
Inhaled B. BAAC.
Bin.
0.03
Anti-fungals
Ishaled LAAC
HyperSaline
Bin.
0.23
Inhaled SAAC
Bin.
0:05
HypertonicSaline
Bin.
0.01
Inhaled LABA
Bin.
0.11
Tobi Solution
Inhaled
Cortico Comho
Bin.
0.41
Cortica Inhaled
Bin.
0.15
Non-IV Ventilation
Bin.
0.05
Oral B. Theoph.
Bin.
0.04
Acetylcysteine
BA
Amisoglyeoside
Bin.
0.03
Oral Hypo. Agonts
Bin.
0.01
iBuprofen
Bin.
0.00
Chronic Oral ABX
Bia.
0.526
Drug Dornase
Bin.
0.02
Cortico Oral
Bin.
0.14
HDI Buprofen
Bin.
0.00
Oxygen Therapy
Bin.
0.11
Tobrnmycin
Bin.
0.03
Og Exacerbation
Bin.
0.03
Loukolriene
Bin.
0.07
O2 Nocturnal
Bin.
0.03
Colistin
Bin.
0.03
O2 Censinuous
Bin.
0.03
Dialstes Insulin
O₂
Macrolida ABX
Bin.
0.02
ABX: antibioties
Table 5: Summary and description of the ADNI dataset.
STATIC COVARIATES
Mean (Freq.)
MINMMa (Mode)
Mcan (Freq.)
Min/Mux (Mode)
Type
Type
Demoggappic
Race
CaL.
0.93
White
Ethnicity Marital Status
Ca..
0.97
No Hisp/Latino
Education
0.23
C16
Car.
0.75
Married
Genetic
APOE,
Cont.
0.44
0/2
VARYNN COVARIATES
Type
Mean
Min/ Max/
Type
Mean
Min / Max
Demographic Bomurker
Age
Cont.
73.6
5582
Entorhinal
Cont.
3.55+3
L0E4376.7E43
Mid Temp
Cont.
2.0E+4
8.E+37 3.21+4
Fusiforr
Cont.
188445
2.9445
Ventriales
Cont.
4,1F
5.7R43 / 16615
Hippocampus
Cont.
6.9E+
288+33
1.14.4
Whole
Cont.
6.5E+5 1,5
Intracrnnial
Cont.
1.58466
2,9942 2.1466
Cognitive
ADAS-II
Cont.
8.58
(710
ADAS-13
Cont.
[3.61
6935
CRD Sum
Boxes
Cont.
1.21
0/17
Mini Mental State
Cont.
27.84
2/50
RAVLT Forgetting
Cont.
4.19
-12/15
RAVLT Immediate
Cont.
38.25
0775
Leamiing
Cont.
4.65
RAVLE Pereent
Cont.
-5001000
14

---

Under review as a conference paper at ICLR 2020
1
X
Encoder (S))
X
Encoder (fe)
Encader (fa)
LSTM Cell
Encoder (fa)
SOM
sOM
LSTM Cell
Decoder (gx)
Zt
et
Zt
er
Zt
LSTM
Predictor (g&)
Decoder (gip)
Predictor (g+)
Fully Connected
81:t
9t
X &
P1
9t
(a) DCN-S2S
(b) DCN-E2P, KM-E2P
(c) VAM
(d) SOM-VAF-P
Figure 4: The block diagrams of the tested benchmarks.
Table 6: Comparison table of benchmarks.
Handling
Clustering
Similarity Measure
Label
Label Associated
Methods
Time-Series
Method
Provided
KM-DTW
DTW
K-means
DTW
N
KM-E2P (2)
RNN
K-means
Euclidean 2
(indirect) (direct)
KM-E2P (P)
RNN
K-means
Euclidean
DCN-S2S
RNN
K-means
Euclidean
DCN-E2P
RNN
K-means
Euclidean
(indirect)
SOM-VAL VAE
Markov model
embedding mapping
reconstruction loss
DOM-VAE-P
Markov model
embedding mapping
prediction loss
(direct)
Proposed
RNN
embedding mapping
KL divergence
(direct)
The three diagnostic groups normal brain functioning (0.55), mild cognitive impairment (0.43), and Alzheimer's disease (0.02). The detailed statistics are given in Table 5.
D DETAILS ON THE BENCHMARKS
We compared AC-TPC in the experiments with clustering methods ranging from conventional ap- proaches based on K K-means to the state-of-the-ar approaches based on deep neural networks. The details of how we implemented the benchmarks are described as the following:
Dynamic time warping followed by K-means Dynamic time warping (DTW) is utilized to quantify pairwise distance between two variable-length sequences and, then, --mans is applied (denoted KM-DTW).
To handle variable-length time-series data, we utilized an encoder-pret prredicctr network as depicted in Figure 4b and trained the network based on (10) for dimensionality reduction; this is to provide fixed-length and low-dimensional representations for time-series. Then, we applied K K-means on the latent encodings (denoted as KM-F22P (2)) and the predicted label distributions ý (denoted as KM-E2P ())), respectively. We implemented the encoder and predictor KM-E2P with the same network architectures with those of our model: the encoder is single-layer LSTM with 50 nodes and the decoder is two-layered fully-connected network with 50 nodes each layer.
K K-means with deep neural networks:
Extensions of DCN" (Yang et al., 2017): Since the DCN is designed for static data, we utilized a sequence-to-sequeest model in Figure 4a for the encoder-decodes network as an extension to
15

---

Under review as a conference paper at ICLR 2020
incorporate time-series data (denoted as DCN-S2S) and trained the network based on the recon- struction loss (using the reconstructed input sequence X1.1). For implementing DCN-S2S, we used a single-layer LSTM with 50 nodes for both the encoder and the decoder. And, aug- mented a fully-connected layer with 50 nodes is used to reconstruct the original input sequence from the latent representation of the decoder.
In addition, since predictive clustering is associated with the label distribution, we compared a DCN whose encoder-decode structure is replaced with our encoder-predictor network in Figure 4b (denoted as DCN-E2P) to focus the dimensionality reduction and, thus, finding latent en- codings where clustering is performed
the information for predicting the label distribution. We implemented the encoder and predictor of DCN-E2P with the same network architectures with those of our model as described in Section 5.
SOM-VAE 10 (Fortuin et al., 2019): We compare with SOM-VAE though, this method is ori- ented towards visualization of input data via SOM since it naturally clusters time-series data assuming Markov property (denoted as SOM-VAE). We replace the original CNN architecture of the encoder and the decoder with three-layered fully-connected network with 50 nodes in each layer, respectively. The network architecture is depicted in Figure 4c where X and x, indicate the reconstructed inputs based on the encoding z, and the embedding e; at time t, respectively.
addition, we compare with a variation of SOM-VAB by replacing the decoder with the predic- tor to encourage the latent encoding to capture information for predicting the label distribution (denoted as SOM-VAR V-E-P). For the implementation, we replaced the decoder of SOM- VAE with our predictor which is two-layered fonneccon layer with 50 nodes in each layer predict the label distribution as illustrated in Figure 4d. Here, de and D) indicate the predicted labels based on the encoding 2, and the embedding e, time t, respectively.
For both cases, we used the default values for balancing coefficients of SOM-VAE and the di- mension of SOM to be equal to K.
We compared and summarized major components of the benchmarks in Table 6.
E ADDITIONAL EXPERIMENTS
E. ADDITIONAL RESULTS ON TARGETING MULTIPLE FUTURE OUTCOMES
Throughout the experiment Section 5.6, we identified 12 subgroups of patients that are associated with the next-year development of 22 different comorbidities in the UKCF dataset. In Table 7, we reported 12 identified clusters and the full list of comorbidities developed in the next year since the latest observation and the corresponding frequency. Here, the frequency is calculated in a cluster- specific fashion based on the true label.
E.2 TRADE-OFH BETWEEN CLUSTERING AND PREDICTION PERFORMANCE
In predictive clustering, the trade-off between the clustering performance (for better interpretability) which quantifies how the data samples are homogeneous within each cluster and heterogeneous across clusters with respect to the future outcomes of interest and the prediction performance is common issue. The most important parameter that governs this trade-off is the number clusters. More specifically, increasing the number of clusters will make the predictive clusters have higher diversity to represent the output distribution and, thus, will increase the prediction performance while decreasing the clustering performance. One extreme example is that there as data samples which will make the identified clusters fully individualized; as a consequence, each cluster will lose the interpretability it no longer groups similar data samples.
many clusters
To highlight this trade-off, we conducted experiments under the same experimental setup with that of Section 5.6 where our aim is identify underlying (unknown) clusters when the future outcome of interest is high-dimensional. For the performance measures, we utilized the AUROC and AUPRC to assess the prediction performance, and utilized the average Silhouette index (SI) (Rousseeuw, 1987) aa widely used measure of how similar a member is to its own cluster (homogeneity within cluster) compared to other clusters (heterogeneity across clusters when the ground tround-truth cluster labels are
hhtpps:/ thg.. com www/rattcci ratsch aab /0M0MM
16

---

Under review as a conference paper at ICLR 2020
not available assess the identified clusters. Formally, the SI for subsequence xi C* can be given as follows:
b(n) a(n)
ST(n) max (((n).(m))
(12)
where a(n) |e- E men 11 vi" Il and b(n) min je-1 Emcck' lly! yt'll 111. Here, used the Ll-distance between the ground-truth labels of the future outcomes of interest since our goal is to group input subsequences with similar future outcomes. To control the number of identified clusters (i.e., the activated clusters) of our method, we set separation loss in (5) controls the activation of clusters) and reported the performance by increasing the number of possible clusters K.
(since the embedding
2.25
032
2375
$.28
(a) AUROC
(b) AUPRC
(c) Average SI
Figure 5: AUROC, AUPRC, and average SI (mean and 95% confidence interval) and the number of activated clusters with various K.
As can be seen in Figure 5, the prediction performance increased with an increased number of identi- fied clusters due to the higher diversity to represent the label distribution while making the identified clusters less interpretable (i.e., the cohesion and separation among clusters become ambiguous as seen in the low average SI). On the other hand, when we set = 1.0 (which selected based on the validation loss in 3), method consistently identified similar number of clusters for K > 20, i.e, 13.8 on average, in a data-driven fashion and provided slightly reduced prediction performance with significantly better interpretability, i.e., the average SI 0.120 average. This highlights the usefulness of (5) which helps to identify clusters with different label distributions.
17

---

Under review as a conference paper at ICLR 2020
Table The comorbidities developed in the next year for the 12 identified clusters. The values in parentheses indicate the corresponding frequency.
Chusters
Comorbidities and Frequencies
Diabetes (0.85)
Liver Enzymes (0.21)
Arthrupatyy (0.14)
Depression (0. (0.10)
Hypertens (0.08)
Osteopenia (0.07)
Intess, Obstmuction (0.07)
Cimhosss (0.04)
Cluster
ABPA (0.04)
Liver Discase (0.04)
Ostcoporosis (0.03)
Hearing Loss (0.03)
(0.02)
Kidney Stones (0.01)
fracture (0.01)
Hemaptysis (0.01)
Pancreatitis (0.01)
Cancer (0.00)
Gall bladder (0.00)
Colonic stricture (0.00)
GI bloed no var. (0.00)
GI bloed var. (0.00)
Liver Enzymes (0.09)
Arthropathy (0.08)
Depression (0.07)
Intest, Obstruction (0.00)
Diabctes (0.06)
Ostcopenia (0.05)
ABPA (0.04)
Asthma (0.03)
(0.03)
Hearing
Osteoporosis
Pancreatits (0.02)
Kidacy Stones (0.02)
Hypertension (0.01)
Cirrhosis (0.01)
Gall bladder (0.01)
Cancer (0.01)
Hemoppysis (0.00)
Bone fracture (0.00)
Colonic stricture (0.00)
Gl bleed
(0.00)
GI bleed (0.00)
ABPA (0.77)
Osteopenia (0.21)
Intest. Obstruction (0.11)
Hearing Loss (0.10)
Enzymes (0.07)
Diabetes
Depression (0.05)
Pancresttis (0.05)
Cluster
Liver Discase (0,04)
Arthopathy (0.04)
Asthma (0.03)
Bone fracture (0.02)
Osteoporosis (0.02)
Hypertension (0.01)
Cancer (0.01)
Cirhossis (0.01)
Kidney Stones (0.01)
Gall bladdlr (0.01)
Hemoptysis (0.00)
Colonic stricture
Gl bleed no var. (0.00)
Gl bleed var. (0.00)
(0.89)
(0.87)
(0.29)
Osteupenia (0.28)
Liver Enzymes (0.24)
ARPA (0.15)
Ostesporosis (0.1
Hearing Loss (0.06)
Cluster
Arthropathy (0.05)
Intest. Obstruction (0.05)
Depression (0.04)
Hypertension (0.03)
Cirtosis (0.02)
Kidacy Stones (0.02)
Pancreatiti (0.02)
Gall bladder (0.02)
Cancer (0.01)
Bone fracture (0.00)
Hemoprysis (0.00)
Colonic stricture (0.00)
G1 bloed
bloed (0.00)
Ostcoporosis (0.76)
Diabetes (0.43)
Arthropathy (0.20)
Liver Enzymes (0.18)
Ostcopenia (0.15)
Depression (0.13)
Intest. Obstruction (0.11)
ABPA (0.11)
Cluster
Hearing Loss (0.09)
Liver Disease (0.08)
Hypertennion (0.07)
Cirrhosis (0.07)
Kidney Stomes (0.03)
Asthma (0.02)
Hemoopyyss (0.02)
Bone fracture (0.02)
Gall
Pancreatiti
Cancer (0.00)
Colonic
(0.00)
Gl bleed no var. (0.00)
Gl bleed var. (0.00)
Asthma (0.88)
Diabetes (0.81)
Osteopenia (0.28)
ABPA (0.24)
Liver Enzymes (0,22)
Depression (0.15)
Osteoporosis (0.14)
Intest, Obstruction (0.12)
Cluster
Hypertension (0. 10)
Cinhosis (0.10)
Liver Discase (0.09)
Arthropathy (0.08)
Bone fracture (0.01)
Hemoptysis (0.01)
Pancreatitis (0.01)
Hearing Loss (0.01)
Cancer (0.01)
Kidacy Stones (0.01)
GI blood var. (0.01)
Gall bindder (0.00)
stricture (0.00)
bleed var. (0.00)
Liver Discase (0.85)
Liver Enzymes (0.37)
Osteopenia (0.27)
ABPA (0.09)
Arthmpathy (0.07)
Diabetes (0.06)
Intest, Obstruction (0.06)
Osteoparosis (0.05)
Cluster
Depression
Asthma (0.03)
Hcaring Loss (0.03)
Cirrhosis (0.02)
Hemoptysis (0.02)
Hypertension (0.01)
Kidney Stones (0.01)
Pancreatitis (0.00)
Gall bladder (0.00)
Bone fracture (0.00)
Cancer (0.00)
Colonic (0.00)
G1 bloed no var. (0.00)
G1 bloed (0.00)
ABPA (0.83)
Diabetes (0.78)
Ostcopenia (0.25)
Ostcoporosis (0.24)
Liver Enzymes (0.15)
Obstruction
Liver Discase (0.09)
Hyperteusion
Cluster
Hearing Loss (0.07)
Arthoopatyy (0.06)
Depression (0.04)
Cirrusis (0.02)
Asthina (0.01)
Bone fracture (0.01)
Kidney (0.01)
Hemoptyssis (0.01)
Cancer (0.00)
Pancreatitis (0.00)
Gall bladder (0.00)
Colonic strieture (0.00)
Gl bloed no var. (0.00)
bloed var, (0.00)
Diabetes (0.94)
(0.83)
Enzymes (0.43)
Osteopenia (0.30)
Hearing Loss (0.11)
Osteoporesis (0.10)
Intest. Obstruction (0.09)
Cirhusis (0.08)
Depression (0.08)
ARPA (0.07)
Arthoopahyy (0.06)
Hypertension (0.05)
8
Kidncy Stones (0.05)
Asthma (0.02)
Hemoptysis (0.01)
Bone fracture (0.01)
Cancer (0.00)
Pancreatitis (0.00)
Gall biadder (0.00)
Colonic strictrre (0.00)
Gl bleed
(0.00)
GI bleed (0.00)
Asthia (0.89)
Osteopenia (0,26)
ABPA (0.19)
Arthropathy (0.14)
(0.
Depression (0.08)
Osteoporosis (0.08)
Diabetes (0.06)
Cluster
Liver Enzymes (0.06)
Hemoptysis (0.05)
Hypertension (0.02)
Liver Discase (0.02)
Pancreatiti (0.02)
Bone fracture (0.01)
Cancer (0.01)
Cirrhosis (0.01)
Gall bladder (0.01)
Hearing Lass (0.01)
Kidney Stones (0.00)
Colonic stricture (0.00)
Gl hleed no var. (0.00)
Gl bleed var. (0.00)
Ostcopenia (0.82)
Diabetes (0.81)
Arthroparhy (0.23)
Depression (0.19)
Liver Enzymes (0.18)
Hypertension (0.16)
Hearing Loss (0.10)
Liver Discase (0.10)
Cluster 10
Osteoporesis (0.10)
Intest. Obstruction (0.09)
ABPA (0.09)
Kidney Stones (0.07)
Cirhosis (0.05)
Asthma (0.01)
Cancer (0.00)
GI bleed (0.00)
Bone fracture (0.00)
Hemoptysis (0.00)
Pancreatitis (0.00)
Gall bladder (0.00)
Colonic
(0.00)
bloed var. (0.00)
Osteopenia (0.77)
Liver Enzymes (0.18)
Arthropathy (0.12)
Depression (0.09)
Hypertension (0.06)
Diabetes (0.06)
Hearing Loss (0.06)
ABPA (0.05)
Cluster II
Liver Discase (0.05)
Osteoporosis (0.04)
Intest. Obstmuction (0.04)
Cimrhssi (0.02)
Asthma (0.02)
Pancreatitis (0.02)
Bone fracture (0.01)
Cancer (0.01)
Kidacy Stones
bladder (0.00)
Colonic stricture (0.00)
Hemoptyais (0.00)
G1 bleed no var. (0.00)
GI bleed var. (0.00)
18

---

Under review as a conference paper at ICLR 2020
F PSEUDO-CODE OF AC-TPC
As illustrated in Section 3.2, AC-TPC is trained an iterative fashion. We provide the pseudo-code for optimizing our model in Algorithm 1 and that for initializing the parameters in Algorithm 2.
Algorithm Pseudo-code for Optimizing AC-TPC
Input: Dataset P {(x,, Me y J number clusters K, coefficients (a, B). learning rate (7A qJ; M,)) mini-batch size Nmb. and update step M
Output: AC-TPC parameters (0, V/ x) and the emhedding dictionary E
Initialize parameters (0, 1/, (6) and the embedding dictionary & via A1 Algori thm
repeat
Optimize the Encoder, Selector, and Predictor
for = I.... .M do
Sample mini-batch of nemb data samples: {(x" Vi Yi Jams ~ for 1,-.............. 7mmb do
Calculate the assignment probability: =²= [ -....--- (K) + ho
Draw the cluster assignment: * Cat((T")))
Calculate the label distributions: De' (e(s)))) and Di + go(fo(xi.))
end for
Update the encoder So and selector hep:
0
li V. (o) a VVo
m (k) log x +A())
Timb
n=l
T
V 7A
(vi 0f V log (f a
mt (k) log m()
90msh
mml Cml
Am1
Update the predictor Is:
① x-0-
V Vahi
Romb
end for
Optimize the Cluster Centroids
for m = 1,... M do
Sample mini batch of data samples: {(x"" 34 ) )ami D for I.... 7mmb do
Calculate the assignment probability: x² [ + (f(((()))))
Draw the cluster assignment: sf Cat(m])
Calculate the label distributions: Df <- (e("))
end for
for = 1.... K
Update the embeddings e(k):
T
e(k) e(k) T/F
V Ve(ay w.)))
V Va(ay (eal((()))()))
Tymb
end for Update the embedding dictionary: end for until convergence
+ (e(1)....((k))
19

---

Under review as a conference paper at ICLR 2020
Algorithm Pseudo-code for pre-training AC-TPC
Input: Dataset D [(x"" yi I } m=1, number of clusters K, learning rate mini- batch size Temb Output: AC-TPC parameters (0, v. and the embedding dictionary &
Initialize parameters (0, 1b, Φ) via Xavier Initializer
Pre-train the Encoder and Predictor repeat
Sample mini-batch of Resb data samples: {(x+; v(") ( De=13 D
for n = 1, ..... Rmss do Calculate the label distributions: Di + (o((((x().)
end for
Pumb
+ δ Volilui, 97))
V Vali (vi .E)
T0mb
1mmb
until convergence
Initiatize the Cluster Centroids
Calculate the embedding dictionary and initial cluster assignments ch
E, ((c (( + K-means (1 (1z) / 1 K)
Pre-train the Selector repeat
Sample a mini- batch of nhnh data samples: {(x?' "xxf,v/)) ( ) ~ D
for n 1, Rmb do
Calculate the assignment probability: .. = (=] of (K{) - (o(f((x(,)) end for Update the selector hp:
2 K
1/ x- 1)
(k) log (k)
Tmmb
me tm1 k=
until convergence
20


Reviews:
[{"review_id": "rkxb9oaZqB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a clustering approach for time series that encourages instances with similar time profiles to be clustered together. The approach consists of three modules: an encoder, a cluster assigner and a (future outcome) predictor, all specified as neural networks.\n\nThe objective of the model is to produce cluster embeddings that are as informative of the outcomes as possible, while not being a direct function of covariates. Note that (2) may look misleading because it indicates that the outcome is a function of the cluster assignment, however, it does not show that the assignment is indeed a function of the covariates.\n\nIt is not entirely clear how a patient is assigned to a cluster provided that cluster assignments are a function of time.\n\nIt is desirable that performance metrics do not seem affected by the unknown number of clusters, however, this makes for difficult to interpret clusters. More so in practice when the number of identified clusters is a function of the model architecture and hyperparameters (\\alpha and \\beta). Is the number of clusters selected by cross-validation and if so, what performance metric is used to select the best choice?\n\nIn Table 3 for UKCF with 3 comorbidities, how are AUROC and AUPRC evaluated provided these are binary predictions?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Skxh_3ORFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis work proposes a method to perform clustering on a time-series data for prediction purposes, unlike the classical clustering where it is done in an unsupervised manner. The authors use an encoder (RNN) to process the time-series medical records, a selector to sample the cluster label for each encoding, and a predictor to predict the labels based on the selected cluster centroids. Since the sampling process prevents the authors from using back-prop, they employ an actor-critic method.\n\nStrengths:\n- Although some would argue otherwise, patient similarity has some promise to be useful in clinical practice.\n- The proposed method clearly outperformed various baselines in terms of clustering (Table 1).\n- Table 3 and Figure 3 show that the proposed method can capture heterogeneous subgroups of the dataset.\n\nConcerns:\n- I'm not a clustering expert, but I'm skeptical this is the first work to combine clustering and supervised prediction using an RL technique.\n- It is unclear what it means to train the embedding dictionary. Are there trainable parameters in the embedding dictionary? It seems that all it does is calculate the mean of the z_t's (i.e. centroid) in each cluster. Or do you take the centroid embeddings and put that through a feed-forward network of some sort?\n- The effect of Embedding Separation Loss (Eq.5) seems quite limited. According to Table 2, it doesn't seem to help much. And contrary to the authors' claim, using \\beta increase the number of activated clusters from 8 to 8.4.\n- Most importantly, the central theme of this work is combining clustering with prediction labels for the downstream prediction task. But the authors do not compare the prediction performance of the proposed method with other clustering method or \"patient similarity\" methods, or even simple supervised models. The only prediction performance metric to be found is Table 2 from the ablation study.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1latqSJ5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a temporal clustering algorithm for the medical domain. The main advantage of the proposed method is that it uses supervised information for temporal clustering. The proposed method is evaluated on two real-world datasets and showed improvements against a few other temporal clustering methods.\n\nDetailed Comments: \n\nMethodology:\nThe actor-critic part of the loss is really just a type of policy gradient. There is no estimation of a value function in Eqn. 7 or anywhere to be found in Appendix B. This is very misleading on the part of the authors because policy grad and AC are very different algorithms. AC type of algorithms provide some variance reduction mechanisms for the classical policy gradient (not done here), but they require additional work to estimate a value function for future trajectories that incur bias. If the authors claim a real AC algorithm for the predictive clustering loss, then some justification for bias-variance tradeoffs should be mentioned instead of attributing it to tuning the hyperparameters between the losses.  \n\nPerhaps a bigger issue is that there is a general tradeoff between reconstructing the time-series (the unsupervised learning portion) vs. predictive performance that is common to predictive clustering problems -- it is not addressed here. One can increase the performance in one (in this case prediction accuracy) while sacrificing the other. \n\nIn the extreme case, one can just tune the hyperparameter of the loss function to turn this into a pure supervised learning problem while sacrificing the capacity of the embedding representations to have any meaning (e.g., to reconstruct time-series [SOM-VAE] or predict future ones). The authors did not really propose a systematic way to control this tradeoff, nor did they provide some experiments to show how well the embeddings can be used to recover the temporal patterns. \n\nExperiments:\nThe baseline experiments are rather weak. For example, 3 out of the 4 models (DTW, DCN extensions and SOM-VAE) were all unsupervised learning techniques that are not designed to take into account label information. This is especially true for comparison against SOM-VAE (which the authors called state of the art for the proposed problem). SOM-VAE is actually less intended for prognostication than the likes of Baytas et al. (2017) and Madiraju et al. (2018) and provides less informative baselines. The authors should also provide more details regarding how these ``extensions'' are done for things like DCN and K-means clustering on deep neural networks. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a reinforcement learning approach to clustering time-series data. The reviewers had several questions related to clarity and concerns related to the novelty of the method, the connection to RL, and experimental results. While the authors were able to address some of these questions and concerns in the rebuttal, the reviewers believe that the paper is not quite ready for publication.
Decision: Reject