Variational pSOM: Deep Probabilistic Clustering with Self-Organizing Maps
Authors: Laura Manduchi
Matthias Hüser
Gunnar Rätsch
Vincent Fortuin
Venue:  (2020)
Keywords: Self-organizing maps
Generative models
Unsupervised representation learning

Abstract: Generating visualizations and interpretations from high-dimensional data is a
common problem in many fields. Two key approaches for tackling this problem 
are clustering and representation learning. There are very performant deep
clustering models on the one hand and interpretable representation learning techniques, 
often relying on latent topological structures such as self-organizing maps,
on the other hand. However, current methods do not yet successfully combine
these two approaches. We present a new deep architecture for probabilistic clustering, 
VarPSOM, and its extension to time series data, VarTPSOM, composed of VarPSOM 
modules connected by LSTM cells. We show that they achieve superior 
clustering performance compared to current deep clustering methods on static 
MNIST/Fashion-MNIST data as well as medical time series, while inducing an
interpretable representation. Moreover, on the medical time series, VarTPSOM
successfully predicts future trajectories in the original data space.

Under review as a conference paper at ICLR 2019
VARIATIONAL PSOM: DEEP PROBABILISTIC CLUS- TERING WITH SELF-ORGANIZING MAPS
Anonymous authors Paper under double-blind review
ABSTRACT
Generating visualizations and interpretations from high-dimensional data is a common problem in many fields. Two key approaches for tackling this problem are clustering and representation learning. There very performant deep cluster- ing models on the one hand and interpretable representation learning techniques, often relying on latent topological structures such as self-organd self-organizing maps, the other hand. However, current methods do not yet successfully combine these two approaches. We present a new deep architecture for probabilistic clustering, VarP- SOM, and its extension to time series data, Var We show that they achieve superior clustering performance compared to current deep clustering methods on static MNIST/Fashion F MNIST data as well medical time series, while inducing interpretable representation. Moreover, on the medical time series, VarTPSOM successfully predicts future trajectories in the original data space.
2
4
6
11
INTRODUCTION
14 Information visualization techniques are essential in areas where humans have to make decisions
15 based on large amounts of complex data. Their goal is to find an interpretable representation of
16 the data that allows the integration of humans into the data exploration process. This encourages
17 visual discoveries of relationships in the data and provides guidance to downstream tasks. In this way, much higher degree of confidence in the findings of the exploration is attained (Keim) 2002). 19 An interpretable representation of the data, in which the underlying factors are easily visualized.
particularly important in domains where the reason for obtaining certain prediction is valuable
21 as the prediction itself. However, finding a meaningful and interpretable representation of complex data can be challenging.
23 Clustering is one of the most natural ways for retrieving interpretable information from raw data. Long-estad long-established methods such k-means (MacQueen 1967) and Gaussian Mixture Models (Bishop 2006) represent the cornerstone of cluster analysis. Their applicability, however, is often constrained to simple data and their performance is limited in high-dimensional,, complex, real-world data sets, which do not exhibit a clustering feritrring-freenllyy structure.
25
27
28 Deep generative models have recently achieved tremendous success in representation learning. Some of the most commonly used and efficient approaches are Autoencoders (AEs), Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) (Kingma Welling, 2013] Goodfellow et al. 2014). The compressed latent representation generated by these models has been proven ease the clustering process (Aljalbout al., 2018). As result, the combination of deep generative models for feature extraction and clustering results in dramatic increase of the clustering performance (Xie et al. 2015). Although very successful, most of these methods do not investigate the relationship among clusters and the clustered feature points live in high-dimensional latent space that cannot be easily visualized or interpreted by humans.
29
30
31
32
33
34
36
The Seff-Organizing Map (SOM) (Kohonen 1990) is clustering method that provides such an interpretable representation. produces a dimdenmnnioonl (typically 2-dimensional), discretized representation of the input space by inducing flexible neighbourhood structure over the clusters. 40 Alas, its applicability is often constrained to simple data sets similar to other classical clustering methods.
38

---

Under review as a conference paper at ICLR 2019
To resolve the above issues, propose novel deep architecture, the Variational Probabilistic SOM (VarPSOM), that jointly trains a VAE and SOM to achieve an interpretable discrete representation while exhibiting state-of-the-art clustering performance. Instead of hard assignment of data points to clusters, our model uses a centroid-based probability distribution. It minimizes its Kullback- Leibler divergence against auxiliary target distribution, while enforcing SOM-friendly space. To highlight the importance of an interpretable representation for different purposes, we extended this model to deal with temporal data, yielding VarTPSOM. We discuss related work in Section Extensive evidence of the superior clustering performance of both models, on MNIST/Fashion- MNIST images as well as real-world medical time series presented in Section
43
45
47
48
49
2
50
51
Our main contributions are:
52
A novel architecture for deep clustering, yielding an interpretable discrete representation through the use of a probabilistic self-organizing map.
53
54
An extension of this architecture to time series, improving clustering performance on this data type and enabling temporal predictions.
55
58
A thorough empirical assessment of our proposed models, showing superior performance on benchmark tasks and challenging medical time series from the intensive care unit.
57
RELATED WORK
58
59
Self-Organe Orgaaniggg Maps have been widely used as a means to visualize information from large amounts of data (Tirunagari al. (014) and as a form of clustering in which the centroids are con- nected by a topological neighborhood structure (Flexer 1999). Since their early inception, several variants have been proposed to enhance their performance and scope. The adaptive subspace SOM, ASSOM (Kohonen 1995), for example, proposed to combine PCA and SOMs to map data into a reduced feature space. TTkunaga & Furukawa 2009) combine SOMs with multi-layer perceptrons to obtain a modular network. (Liu et al. 2013) proposed Deep SOM (DSOM), an architecture com- posed of multiple layers similar to Deep Neural Networks. There exist several methods tailored to representation learning on time series, among them (Franceschi et al. 2019 Fortuin & Rätsch 2019 Fortuin et al., 2019), which are however not based on SOMs. Extensions of SOM optimized
60
61
63
65
87
68
69
for temporal data include the Temporal Kohonen map (Chappell & Taylor 1993) and its improved
70
version Recurrent SOM (MeQueen et al.) 2004) as well as Recursive SOM (Voegtlin 2002). While
71
SOM and its variants are particularly effective for data visualization (Liu et al.) 2015 it was rarely
72 attempted to combine their merits in this respect with modern state-of-the-are clustering methods, which often use deep generative models in combination with probabilistic clustering.
73
74 In particular, recent works on clustering analysis have shown that combining clustering algorithms with the latent space of AES greatly increases the clustering performance (Aljalbout et al. 2018). 76 Xie et al. 2015) proposed DEC, a method that sequentially applies embedding learning using Stacked Autoencoders (SAE), and the Clustering Assignment Hardening method the obtained representation. An improvement of this architecture, IDEC (Guo et al.) 2017). includes the decoder network of the SAE in the learning process, that training is affected by both the clustering loss 80 and the reconstruction loss. Similarly, DCN (Yang et al. (016)) combines a k-means clustering loss
78
81 with the reconstruction loss of SAE to obtain an end-to-end architecture that jointly trains repre-
82 sentations and clustering. These models achieve state-of-the-arr clustering performance but they do
83 not investigate the relationship among clusters. An exception is the work by (Li et al., 2018), in
84 which they present an unsupervised method that learns latent embeddings and discovers fulti-facet clustering structure. Relationships among clusters were discovered, however, they do not provide a latent space that can be easily interpreted and which eases the process of analytical reasoning.
a5
86
87
While there exist previous efforts to endow VAEs with a hierarchical latent space (Vikram et al., 2018 Goyal et al. 2017). to the best of our knowledge, only two models used deep generative models in combination with a SOM structure in the latent space. The SOM-VAE model (Fortuin 89
88
et al. 2018), inspired by the VQ-VAE architecture (yan den Oord et al.) 2017) (which itself was later extended in (Razavi et al. 2019)), uses an AE to embed the input data points into a latent space and then applies SOM-based clustering loss on top of this latent representation. It features hard
91
93 assignments of points to centroids, as well as the use of a Markov model for temporal data, both
94 of which yield inferior expressivity compared our method. The Deep Embedded SOM, DESOM
2

---

Under review as a conference paper at ICLR 2019
(Forest al. 2019), improved the previous model by using Gaussian neighborhood window with exponential radius decay and by learning the SOM structure in a continuous setting. Both methods feature topologically interpretable neighborhood structure and yield promising results in visual- izing state spaces. However, those works did not feature empirical comparisons to state-of-the-arr deep clustering techniques and did not make of many of the design principles that have recently proven to be successful in this space.
96
98
100
3
PROBABILISTIC CLUSTERING WITH VARIATIONAL PSOM
Given set of data samples (a;))==1.... where E Rd the goal is to partition the data into set of clusters {S.]i=i,..... while retaining a topological structure over the cluster centroids.
102
103
The proposed architecture for static data is presented in Figure Ta The input vector x; is embedded into latent representation 2; using a VAE. This latent vector is then clustered using PSOM, new SOM clustering strategy that extends the Clustering Assignment Hardening method (Xie et al.) 2015). The VAE and PSOM are trained jointly to learn a latent representation with the aim to boost the clustering performance. To prevent the network from outputting trivial solution, the decoder network reconstructs the input from the latent embedding, encouraging it to be as similar as possible to the original input. The obtained loss function is linear combination of the clustering loss and the reconstruction loss. To deal with temporal data, we propose another model variant, which is
105
107
109
111
112
depicted in Figure lb
PSOM
VAE
ENCODER
Inputs
4
qdee (t)
encoder
PSOM
RNN
DECCOER
prior
decoder
L PSOM
L VAE
(a) VarPSOM architecture for cluster- ing static data. Data points z; are mapped a continuous embedding z; using a VAE (parameterized by 4)). The loss function is the sum of SOM- based clustering loss and the ELBO.
(b) VarTPSOM architecture, composed of VarPSOM modules nected by LSTMs across the time axis, which predict the continu- ous embedding 2t+1 of the next time step. This architecture allows to unroll future trajectories in the latent space as well as the origi- nal data space by reconstructing the s. using the VAF.
Figure Model architectures (a) VarPSOM and (b) VarTPSOM.
113 3.1 BACKGROUND
114 A Self-Organizing Map is comprised of K nodes connected to form grid M C N where the 115 node my,j, at position (i,j) of the grid, corresponds to centroid vector, (4, the input space. The centroids are tied by neighborhood relation N (pi.s) { M (45-11 p. Given the centroids, the SOM algorithm randomly selects an input 2y and updates both its closest centroid ji,j and its neighbors N (Mi,) move them closer to 24. For a complete description of the SOM algorithm, we refer to the appendix (A.
117 random initialization
119
The Clustering Assignment Hardening method has been recently introduced by the DEC model (Xe ct al. 2015) and was shown to perform well in the latent space of AEs (Aljalbout et al. 2018). Given embedding function f(xx), it uses Student's t-distribution (S) as a kernel to measure the
3

---

Under review as a conference paper at ICLR 2019
similarity between embedded data point 2,, and a centroid My:
1 + || 11511 /(x)
Eg (1 - Mar /a) Sij
improves the cluster purity by enforcing the distribution S to approach target distribution, T: //// Ex 80'
Lij Eg 8iy. Ezy
By taking the original distribution to the power of y and normalizing it, the target distribution puts more emphasis on data points that are assigned high confidence. We follow (Xie al. (0155) in choosing 7=22, which leads to larger gradient contributions of points close to cluster centers, as they show empirically. The resulting clustering loss defined as:
=ΣΣ Les log 4j
L KL(TIIS) -L(T)))=
8ij
J
120
3.2 PROBABILISTIC SOM (PSOM) CLUSTERING
Our proposed clustering method, called PSOM, expands Clustering Assignment Hardening in- clude a SOM neighborhood structure over the centroids. We add an additional loss to (T) to achieve an interpretable representation. This loss term maximizes the similarity between each data point and the neighbors of the closest centroids. For each embedded data point 2; and each centroid Hj the loss defined as the negative sum of all the neighbors of μj. fe pe E (N, (a+))), of the probability that is assigned to defined as This sum is weighted by the similarity Sij between 2; and the centroid jty-
CsOM = N +ΣΣs, Sij
Sie
The complete PSOM clustering loss is then:
LesoM = KT((T)))+ + 3L
121 We note that for 3 0 it becomes equivalent Clustering Assignment Hardening.
3.3 VARPSOM: VAE FOR FEATURE EXTRACTION
In our method, the nonlinear mapping between the input z; and embedding z; is realized by a VAE. Instead of directly embedding the input z; into latent embedding 2,, the VAE learns a probability distribution go(2 (2 x;)) parametrized as a multivariate normal distribution whose mean and variance (Jaz Ex) f (3rg). Similarly, it also learns the probability distribution of the reconstructed
output given a sampled latent embedding, Po (a(x; =)) where (μa. En) = fo(=). Both Sa and so are neural networks, respectively called encoder and decoder. The ELBO loss is:
CELBO EI-E. pa(;; 2)) (x x4) || p(z))]
where p(z) is an isotropic Gaussian prior over the latent embeddings. The second term can be interpreted form of regularization, which encourages the latent space to be compact. For each data point v; the latent embedding is sampled from yo(z æz). Adding the ELBO loss to the PSOM loss from the previous subsection, we get the overall loss function of VarPSOM:
LVaPSOM = LPSOM + LELOO
(3)
123
To the best of our knowledge, no previous SOM methods attempted to use a VAE to embed the inputs into latent space. There are many advantages of a VAE over an AE for realizing our goals. Its prior on the latent space encourages structured and disentangled factors (Higgins et al.] 2016)
124
125
126
which could help clustering. The suitability of VAES for anomaly detection (An & Cho. 2015
means that points with a higher variance in the latent space could be treated less accurate and
128
trustworthy. The regularization term of the VAE can be used to prevent the network from scattering
the embedded points discontinuously in the latent space, which naturally facilitates the fitting of the
130
SOM. To test if the use of CNNs can boost clustering performance on image data, we introduce another model variant called VarCPSOM, which uses convolutional filters part of the VAE.
4

---

Under review as a conference paper at ICLR 2019
3.4 VARTPSOM: EXTENSION TO TIME SERIES DATA
To extend our proposed model to time series data, we add a temporal component to the architecture. Given a of N time series of length T. trajectories on the SOM grid. To do so, the VarPSOM could be used directly but would treat each time step of the time series independently, which is undesirable. To exploit temporal information and enforce smoothness in the trajectories, we add an additional loss to
the goal is learn interpretable
ΣΣ Vis,is+1 NT
1
smoothh
(4)
133 where Uigd044 g( 2i++1) 1) is the similarity between Fi, and 2i,+1 using a Student's distribution and Fi,e refers to the embedding of time series 2; at time index t. It maximizes the similarity between latent embeddings of adjacent time steps, such that large jumps in the latent state between time points are discouraged.
134
135
One of the main goals time series modeling is to predict future data points, or alternatively, future embeddings. This can be achieved by adding long short-term memory network (LSTM) across the latent embeddings of the time series, as shown in Fig lb] Each cell of the LSTM takes as input the latent embedding 2; at time step I, and predicts a probability distribution over the next latent embedding, Pw Pw, (2 21). We parametrize this distribution as a Multivariate Normal Distribution whose mean and variance are learnt by the LSTM. The prediction loss is the log-likelihood between the learned distribution and a sample of the next embedding Z4+1:
Cyred = -E log Pw (2 22x)
(5)
The final loss of VarTPSOM, which is trainable in fully end-to-end fashion, is
LvarTPSOM LVarPSOM
Ssmooth + r)
(6)
137 4 EXPERIMENTS
First, we evaluate VarPSOM and VarCPSOM and compare them with state-of-the-art non- interpretable as well as SOM-based clustering methods on MNIST (Lecun et al. 1998) and Fashion- MNIST (Xiao al. 2017) data. Here, particular focus is laid on the comparison of VarPSOM and the clustering models DEC and IDEC, to investigate the role of the VAE and the SOM loss. We then present visualizations of the obtained 2D representations, ease visual reasoning about the data. Finally, we present extensive evidence of the performance of real-world complex time series from the eICU data set (Pollard et al. [0188) and illustrate how it allows visualization of patient health state trajectories in an casily understandable 2D domain. For details on the data sets, we refer to the appendix B.I
139
141
142
illustrate how method could
143
144
VarTPSOM
145
148
Baselines We used two different types of baselines. The first category contains clustering methods that not provide any interpretable discrete latent representation. Those include k-means, the DEC model. well as its improved version IDEC, whose clustering methods are related to ours. We also include a modified version of IDEC that we call VarIDEC, in which we substitute the AE with a VAE, to investigate the role of the VAE. For all these methods we use 64 clusters. In the second category, we include state-of-the-are clustering methods based on SOMs. Here, we used a standard SOM (minisom), AE+SOM, an architecture composed of AE and SOM applied on top of the latent representation (trained sequentially). SOM-VAE and DESOM. Finally, we create a modified version of our model, called AEPSOM, in which we substitute the VAE with an AE (similarly to VarIDEC), For all SOM-based methods we set the SOM grid size to (8 x 8). For different grid configurations we refer the appendix. [B.
148
150
152
153
154
155
156
157
Implementation In implementing our models we focused on retaining fair comparison with the baselines. Hence we decided to use a standard network structure, with fully connected layers of dimensions dl 500 - 500 2000 - to implement both the VAE of our models and the AE of the baselines. The latent dimension, I, is set to 100 for the VAE, and to 10 for the AEs. Since the prior in the VAE enforces the latent embeddings be compact, it also requires more dimensions to learn
159
181
162
5

---

Under review as a conference paper at ICLR 2019
Table 1: Clustering performance of VarPSOM using 64 clusters arranged in 8 x 8 SOM map, compared with baselines. The methods are grouped into approaches with no topological structure in the discrete latent space and interpretable methods using a SOM-based structure in the latent space, as well as extension of our method using convolutional filters. Means and standard errors across 10 runs with different random model initializations are displayed.
MNIST
IMNIST
pur
nmd
pur
nmi
Kmeans
0.845 = 0.001
0.581 + 0.001
0.716 0.001
0.514 + 0.000
DEC
0.944 = 0.002
0.682 + 0.001
0.758 + 0.002
0.562 + 0.001
IDEC
0.950 = 0.001
0.681 = 0.001
VarIDEC (ours)
0.961 = 0.002
0.698 = 0.001
0.765 + 0.003
0.569 + 0.002
SOM
0.701 = 0.005
0.539 = 0.002
0.667 = 0.003
0.525 = 0.001
AE+SOM
0.874 = 0.004
0.646 = 0.001
0.706 0.002
0.543 + 0.001
SOM-VAB
0.868 = 0.004
0.595 + 0.004
0.739 + 0.005
0.520 + 0.003
DESOM
0.939
0.657 0.555 + 0.001
0.752 0.700 + 0.008
0.538 0.493 + 0.008
AEPSOM (ours)
0.816 = 0.003
VarPSOM (ours)
0.964 + 0.001
0.705 +: 0.001
0.764 + 0.003
0.571 + 0.001
VarCPSOM (ours)
0.980 = 0.001 0.726 + 0.001
0.783 + 0.003
0.574 + 0.001
163
meaningful latent space. On the other hand, providing the AE models with higher-dimennsional
184
latent space, needed for the VAE, resulted in a dramatic decrease of performance (see appendix(B.2) VarCPSOM composed of convolutional layers of feature maps [32, 64, 128, 256] and kernel size
185
166
3 x 3 for all layers. For all architectures, no greedy layer-wise pretraining was used to tune the
167
VAE. Instead we simply run the VAE without the clustering loss for a few epochs for initialization.
168
A standard SOM was then used to produce an initial configuration of the centroids/neighborrooodd relation. Finally, the entire architecture is trained for 100, 000 iterations. To avoid fine-tuning hy- perparameters, given the unsupervised setting, is set to 10 for all experiments while the other hyperparameters are modified accordingly to maintain the same order magnitude of the different loss components.
169
171
173
Clustering Evaluation Table shows the clustering quality results of VarPSOM and VarCPSOM on MNIST and Fashion-MNIST data, compared with the baselines. Purity and Normalized Mutual Information are used as evaluation metrics. We observe that our proposed models outperform the baselines of both categories and achieve state-of-the-and clustering performance.
174
175
176
VarPSOM vs. IDEC VarPSOM is inspired by IDEC but it has two major differences. It uses VAE instead of an AE and it improves interpretability in the latent space by adding a new loss that en- forces a SOM structure. Since both VarIDEC and VarPSOM show superior clustering performance compared to IDEC and AEPSOM respectively (Table [ we conclude that the VAE indeed suc- ceeds in capturing more meaningful latent representation compared the second difference, the SOM structure was expected to slightly decrease the clustering perfor- mance, due to a trade-off between interpretability and raw clustering performance. However, we do not observe this in our results. Adding the SOM loss rather leads to an increase of the clustering performance. We suspect this due the regularization effect of the SOM's topological structure. Overall, VarPSOM outperforms both DEC and IDEC.
178
180
standard AE. Regarding
182
183
184
185
186
Improvement over Training After obtaining the initial configuration of the SOM structure, both clustering and feature extraction using the VAE are trained jointly. To illustrate that our architecture improves clustering performance over the initial configuration, we plotted NMI and Purity against the number of training iterations Figure 2 We observe that the performance is stable when increasing the number of epochs and overhitiigg is visible.
188
190

---

Under review as a conference paper at ICLR 2019
020
6.0
0.0
030
18 <0003
Figure 2: NMI (left) and Purity (right) performance of VarPSOM over the number of epochs on the MNIST test set.
192
Role of the SOM loss To investigate the influence of the SOM loss component, we plot the clus- tering performance of VarPSOM against the weight (5) of LsOM in Fig. [ using the MNIST dataset. With 8 30, the K L term (responsible for improving clustering purity) and the LsOM term (respon- sible for enforcing SOM structure over the centroids) are almost equal. It is interesting to observe the different trends in NMI and purity. The NMI performance increases for increasing values of B while purity slightly decreases. Overall, enforcing a more interpretable latent space results more
193
195
196
197
robust clustering model with higher NMI clustering performance.
0.74
0.2
0.7
0.2
0.76
11
0.9
0.7
09
0.59
0.0
Figure 3: NMI (left) and Purity (right) performance of VarPSOM, with standard error, over values on MNIST test set.
199
Time Series Evaluation We evaluate the clustering performance of our proposed models on eICU dataset, comprised of complex medical time series. We compare them against SOM- VAE, as this is the only method among the baselines that is suited for temporal data. Table shows the cluster cell enrichment in terms of NMT for three different labels, the current (APACHE-O) and worst future (APACHE-6/12 hours) physiology scores. VarTPSOM clearly achieves superior clustering performance compared SOM-VAE. This, we hypothesize, is due the better feature extraction
201
203
204
205
using a VAE as well as the improved treatment of uncertainty using PSOM, which features soft assignments, whereas SOM-VAE contains a deterministic AE and hard assignments. Moreover,
206
207
both the smoothness loss and the prediction loss seem to increase the clustering performance. More results on ICU time series are reported in the appendix (B.4).
208
209
To quantify the performance of VarTPSOM in unrolling future trajectories, we predict the final latent embeddings of each time series. For each predicted embedding we reconstruct the input using the decoder of the VAE. Finally, we measure the MSE between the original input and the reconstructed inputs for the last hours the ICU admission. As baselines, we used an LSTM that takes input the first 66 hours of the time series and then predicts the next hours. Since most of the trajectories tend to stay in the same state over long periods of time, another strong baseline obtained by duplicating the last embedding over the final 6 hours. The results (Table 3) indicate that the joint training of clustering and prediction used by VarTPSOM clearly outperforms
210
212
214
216
the 2 baselines.
7

---

Under review as a conference paper at ICLR 2019
Table Mcan NMI and standard error of cluster enrichment vs. current/future APACHE physiology scores, using 2D (8 x 8) SOM map, across 10 runs with different random model initializations.
Model
APACHE-12
APACHE-6
APACHE-0
SOM-VAL VarPSOM
0.0444 + 0.0006
0.0474 0.0005
0.0510 = 0.0005
0.0631 + 0,0008
0.0639 = 0.0008
0.0730 + 0.0009
Var (1) - 0)
0.0710 + 0.0005
0.0719 = 0.0006
0.0818 + 0.0006
VarTPSM
0.0719 + 0.0004
0.0733 = 0.0004
0.0841 + 0.0005
Table 3: MSE for predicting the time series of the last 6 hours before ICU dispatch, given the prior time series.
Model
LSTM
SameState
VarTPSOM
MSE
0.0386 + 0.0049
0.0576 + 0.0012
0.0297 + 0.0009
7
7
3
3
0
7
7
3
3
3
5
0
0
7
7
/
3
5
5
8
5
5
/
9
8
6
6
/
/
9
8
6
6
2
4
4
9
9
8
6
2
2
4
4
9
9
8
2
2
2
4
8
8
(a) MNIST
(b) Fashion MNIST
Figure 4: Reconstructions of MNIST Fashion MNIST data from SOM cells in the 8x8 grid learned by VarPSOM, illustrating the topological neighbourhood structure induced by our method, which aids interpretability.
218
Interpretability To illustrate the topological structure in the latent space, we present reconstruc- x 8) grid, on static MNNISTFFashion-MNIIS data
219
tions of the VarPSOM centroids, arranged in Figure 4 On the ICU time series data, we show example trajectories for one patient dying at the end of the ICU stay, as well two control patients which are dispatched healthily from the ICU. We observe that the trajectories are located in different parts of the SOM grid, and form smooth and interpretable representation (Fig- 5). For further results, including more quantitative evaluation using randomly sampled trajectories, enrichment for future mortality well as an illustration of how the uncertainty generated by the soft assignments can help data visualization, we refer the
220
221
222
223
225
appendix [.5
227
5
CONCLUSION
We presented two novel methods for interpretable unsupervised clustering, VarPSOM and VarTP- SOM. Both models make use of VAE and a novel clustering method, PSOM, that extends classical SOM algorithm to include centroid-based probability distribution. Our models achieve
229
231
superior clustering performance compared to state-of state-of-the-arr deep clustering baselines on bench- medical time series. The use of VAE for feature extraction, instead of an AE, used in previous methods, and the use of soft assignments of data points to clusters result in interpretable model that quantify uncertainty in the data.
mark data and
233
8

---

Under review as a conference paper at ICLR 2019
Start
Scart Erd
X
X
(a) Patient dispatched expired
(b) Patient dispatched healthy 1
(c) Patient dispatched healthy 2
Figure 5: Illustration of 3 example patient trajectories between the beginning of the time series and ICU dispatch, the 2D SOM grid of VarTPSOM. The heatmap shows the enrichment cells for the current APACHE physiology score. We observe qualitative differences in the trajectories the dying and the healthy patients.
235
REFERENCES
236
Elie Aljalbout, Vladimir Golkov, Yawar Siddiqui, and Daniel Cremers. Clustering with deep learn ing: Taxonomy and new methods. CoRR, abs/ 1ess18000.7648, 2018. URL httt:://aaxxii......m org/
237
238
abs/ abs/1801. 07648
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruc- tion probability. Special Lecture on IE, 2(1), 2015.
240
Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statis- tics). Springer- Verlag, Berlin, Heidelberg. 2006. ISBN 0387310738.
242
Geoffrey Chappell and John G. Taylor. The temporal kohonen map. Neural Netw,, 6(3):441-455 March 1993. ISSN 0893-6080. doi: 10. (01001600995 1-K. URL htt.tttp://dx.oom
244
245
otg/10.1016..0
6080 (93) (93)90011- 90011-K
248
Arthur Flexer. On the use of self orga-iigg maps for clustering and visualization. In Jan M. Zytkow and Jan Rauch (eds.), Principles Data Mining and Knowledge Discovery, pp. 80-88, Berlin, Heidelberg, 1999. Springer Berlin Heidelberg. ISBN 978-3-540-48247-55
249
Florent Forest, Mustapha Lebbah, Hanene Azzag, and Jérôme Lacaille. Deep embedded som: Joint representation learning and self-organization. 04 2019.
251
Vincent Fortuin and Gunnar Rätsch. Deep mean functions for meta-learning in gaussian processes. arXiv preprint arXiv: 2019.
253
Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. Som-vae: Interpretable discrete representation learning time series. arXiv preprint 1rXiv:1806.02 2018.
255
Vincent Fortuin, Gunnar Rätsch, and Stephan Mandt. Multivariate time series imputation with variational autoencoders. arXiv preprint 1907.0490 2019.
257
258
Jean- Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. arXiv preprint 1rXIv:1901.107333 2019.
259
260 Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. arXiv e-prints, art. 416.::140..2661. Jun 2014.
282
Prasoon Goyal, Zhiting Hu, Xiaodan Liang, Chenyu Wang, and Eric P Xing. Nonparametric vari- ational auto-encoders for hicrarchical representation learning. In Proceedings of the IEEE Inter- national Conference Computer Vision, pp. 5094-5102, 2017.
284

---

Under review as a conference paper at ICLR 2019
Xifeng Guo, Long Gao, Xinwang Liu, and Jianping Yin. Improved deep embedded clustering with local structure preservation. In Proceedings of the Twenty- International Joint Conference on Artificial Intelligence. IJCAI-17, pp. 1753-1759 2017. doi: 10.249663jjcci.221177444 URL
267
269
https::/doii
24963/ cai 2017/243
270
Irina Higgins, Loic Matthey, Xavier Glorot, Arka Pal, Benigno Uria, Charles Blundell, Shakir Mo- hamed, and Alexander Lerchner. Early visual concept learning with unsupervised deep learning. arXiv preprint arXiv:1606.05577 2016.
272
273
D. A. Keim. Information visualization and visual data mining. IEEE Transactions on Visualization and Computer Graphics, 8(1):188, Jan 2002. ISSN 1077-2626. doi: 10.1109/2945.98
274
275
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv e-prints, art.
276
arXiv:1312.6114 Dec 2013.
T. Kohonen. The self-organizing map. Proceedings of the IEEE, 78(9))1464-1444 Sep. 1990. ISSN 0018-9219. doi: 10.1 10955.355533...
278
279
Teuvo Kohonen. The adaptive-subspace som (assom) and its use for the implementation of invariant feature detection. 1995.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-233) Nov 1998. ISSN 0018-9219. doi: 10.1109/5.726791.
282
Xiaopeng Li, Zhourong Chen, and Nevin L. Zhang. Latent tree variational autoencoder for joint representation learning and multidimensional clustering. CoRR, abs/1803.00506, 2018. URL
285
286
http://arxis> wtt/
05206
287
N. Liu, Wang, and Y. Gong, Deep self-organizing map for visual classification. In 2015 Interna- tional Joint Conference on Neural Networks (IJCNN) pp. 1-6, July 2015, doi: 10.1109/UICN. 2015.7280357.
288
289
290
Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probubility, Vol- ume 1: Statistics, pp. 281-297, Berkeley, Calif., 1967. University of California Press. URL
MacQueen.
292
T. A. McQueen, A. A. Hopgood, J. A. Tepper, and T. J. Allen. A recurrent self-organizing map for temporal sequence processing. In Ahamad Lotfi and Jonathan M. Garibaldi (eds.), Applications and Science in Soft Computing, pp. 3-8, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg. ISBN 978-3-540-45240-99
295
297
298
Tom Pollard, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. The eicu collaborative research database, freely available multi-center database for critical care research. Scientific data, 5, 2018.
289
300
301
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with 2.. arXiv preprint arXiv: 1900100446664.... 2019.
302
303
Tirunagari, N. Poh, K. Aliabadi, D. Windridge, and D. Cooke. Patient level analytics using self-organising maps: A case study on type-1 diabetes self-care survey responses. In 2014 IEEE Symposium on Computational Intelligence and Data Mining (CIDM), pp. 304-309, Dec 2014. doi: D0.109//CDDM.211..7088
304
305
Kazuhiro Tokunaga and Tetsuo Furukawa. Modular network som. Neural Networks, 22(1):82
308
90, 2009. ISSN 0893-6080. doi: http:://ooiiooy///...lo/eneeeeee 10.006. URL http: www. www.sciee.cedrrect.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete represetiinnnnnnttettattionn learn- ing. CORR, 1.000777, 2017. http:///....ceeea......////////.
311
312 Sharad Vikram, Matthew Hoffman, and Matthew Johnson. The loracs prior for vaes: Letting the trees speak for the data, arXiv preprint arXiv: 1 1810068811 2018.
10

---

Under review as a conference paper at ICLR 2019
Thomas Voegtlin. Recursive self-organizing maps. Newral Networks, 15(8):979 991, 2002.
ISSN 0893-6080. doi: http:://ooaoooo/l/ooooooooooooo0 URL httpp://ww.
315
sciencedirect. htt.a///w...........c....o.oooaa
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. CoRR,
318
2017. URL httt:://aaxxiiv org/
319
abs/1708. 07747
320
Junyuan Xie, Ross B. Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. CoRR, abs/1511.066 2015. URL http::/:/aara...rrr//////......0
Bo Yang. Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. Towards E-means--riende spaces: Simultaneous deep learning and clustering. CoRR, abs/1610.04794, 2016. URL
323
http:://rke wwwplli..rg//bb.e
abs wttpg//ab//61....oo 04794

---

Under review as a conference paper at ICLR 2019
APPENDIX
326
A
SELF-ORGANIZING MAPS
Among various existing interpretable unsupervised learning algorithms, Kohonen's self-organizing map (SOM) (Kohonen 1990) is one of the most popular models. It is comprised of К neurons connected form discrete topological structure. The data are projected onto this topographic map which locally approximates the data manifold. Usually it is finite two-dimensional region where regular hexagonal or rectangular grid. Here we use grid, M S N because of its simplicity and its visualization properties. Each neuron maj, at position (i,j) of the centroid vector, Mi.j in the input space. The centroids
328
329
331
neurons are arranged
333
grid, for i I,..., VK, corresponds are tied by neighborhood relation, here defined as N ((4.) = {Pi-1.j j.j Mi+ II
334
Given random initialization of the centroids, the SOM algorithm randomly selects input s; and updates both its closest centroid μi,j and its neighbors N (His) to move them closer to The algorithm then iterates these steps until convergence.
338
337
Algorithm Self-Organizing Maps
Require: 0 < a(t) < 1;
lim Ea(t) + oo; lim E oc;
repeat
each time i, present an input (t) and select the winner,
v(l)
arg min kefl
[x(() w
Update the weights of the winner and its neighbours,
Aw a(t) k, [x(t) wt)
until the map converges
The range of SOM applications includes high dimensional data visualization, clustering, image 339 and video processing, density or spectrum profile modeling, text/document mining, management systems and gene expression data analysis.
12

---

Under review as a conference paper at ICLR 2019
B EXPERIMENTAL AND IMPLEMENTATION DETAILS
342
B.1
DATASETS
343
MNIST: It consists of 70000 handwritten digits of 28-by-28 pixel size. Digits range from 0 9, yielding 10 patterns in total. The digits have been size-normalized and centered in fixed-size image Lecun et al. (1998).
344
345
348
Fashion MNIST: A dataset of Zalando's article images consisting of training set of 60,000 examples and test set of 10,000 examples Xiao et al. (2017). Each example is x8 grayscale image, associated with label from 10 classes.
347
348
vital sign/lab measurements of intensive care unit (ICU) patients resampled to a I-hour based grid using forward filling and filling with population statistics from the training set if measurements were available. From all ICU stays, we excluded ICU stays, which were shorter than 1 day, longer than 30 days or which had at least one gap the continuous vital sign monitoring, which we define by a interval between 2 HR measurements of at least 1 hour. This yielded N = 10559 ICU stays from the eICU database. dviiass 14 vital sign variables and d 84 lab measurement variables were included, giving an overall data dimension of = 98. The last 72 hours of these multivariate time series were used for the experiments. As labels we use a variant of the current dynamic APACHE physiology score (APACHE-O) score in the next and 12 hours (APACHE-6// 6/12), and the mortality in the next 24 hours. Only those variables from the APACHE score definition which recorded in the eICU database were taken into account.
eICU: For temporal data
350
352
353
354
355
356
357
well the worst APACHE
359
381
362
Each dataset divided into training, validation and test sets for both our models and the baselines.
383
B.2
LATENT SPACE DIMENSION
364
We evaluate the DEC model for different latent space dimensions. Table ST S shows that the AE, used in the DEC model, performs better when lower dimensional latent space used.
Table S1: Mcan/Standard error of NMI and purity of DEC model MNIST test set, across 10 runs with different random model initializations. We use 64 clusters and different latent space dimen- sions.
Latent dimension
Purity
NMI
= 10
0.950 + 0.001
0.681 + 0.001
= 100
0.750 + 0.001
0.573 +'t 0.001
B.3 NUMBER OF CLUSTERS
387
We evaluate the NMI and purity clustering performance of our model, VarPSOM, with a varying number of clusters on the MNIST dataset. Since IDEC represents the main competitor we include it in this analysis. Figure [I shows that VarPSOM outperforms IDEC for all the different configu- rations. In particular, it is interesting observe that NMI decreases with an increasing number of
368
389
370
clusters in both models. This is because the entropy of the clustering increases with the number of clusters.
372
13

---

Under review as a conference paper at ICLR 2019
BBCC WePMCM
D0C WWrPSMM
Mamber ChetteS
Number diuvaios
Figure S1: NMI (left) and purity (right) clustering performance of VarPSOM and IDEC with varying number of clusters on the MNIST test set.
373
B.4
LEARNING HEALTH STATE REPRESENTATIONS IN THE ICU
By enforeing a SOM structure, VarPSOM, well as VarTPSOM, project the cluster centroids a discrete 2D grid. Such grid is particularly suited for visualization purposes and relations between centroids become immediately intuitive. Fig. S2 a heat-map (colored according enrichment in the current APACHE score, well as mortality risk in the next 24 hours) shows compact enrichment structures. VarTPSOM succeeds in creating a meaningful and smooth neighbourhood structure.
375
378
377
378
distinguishes risk profiles with practically zero mortality risk from high mortality risk, reaching up
380
to 15 %, different regions of the map, even though is learned a purely unsupervised fashion. Remarkably, the two heat-maps S2b and S2a) show different enrichment patterns. Clusters which are enriched in health states with higher APACHE scores often do not correspond exactly to clusters with higher mortality risk. This suggests that traditional representations of physiologic values, such as the APACHE score, fail to fully use all complex multivariate relationships present in the
382
384
385
ICU recordings, and associated with dynamic mortality in simple way.
(a) Current APACHE score
(b) Mortality risk next 24 hours
Figure S2: Heat-maps of enrichment in mortality risk in the next 24 hours as well as the current dynamic APACHE score, superimposed the discrete 2D grid learned by VarTPSOM.
14

---

Under review as a conference paper at ICLR 2019
B.5
VISUALIZING HEALTH STATE TRAJECTORIES THE ICU
387
To analyze the trend of the patient pathology, VarTPSOM induces trajectories on the 2D SOM grid which be easily visualized. Fig. S3 shows 20 randomly sampled patient trajectories obtained
388
389 by our model. Trajectories ending in the death of the patient are shown red, healthily dispatched
patients are shown in green.
1
X
Figure S3: Randomly sampled VarTPSOM trajectories, from patients expired at the end of the ICU stay, as well as healthily dispatched patients. Superimposed is a heatmap which displays the cluster enrichment in the current APACHE score, from this model run. We observe that trajectories of dying patients are often in different locations the map as healthy patients, in particular in those regions enriched for high APACHE scores, which corresponds with clinical intuition.
15

---

Under review as a conference paper at ICLR 2019
One of the main advantage of VarTPSOM the traditional SOM algorithm is the of soft assigmments data points to clusters which results in a better ability to quantify uncertainty in the data. For visualizing health states in the ICU, this property is very important. In Fig [S4 we plot an example patient trajectory, where 6 different time-steps (in temporal order) of the trajectory were chosen. Our model yields soft centroid-hased probability distribution which evolves with time and which allows estimation of likely discrete health states at a given point in time, For each time-step the distribution of probabilities is plotted using heat-map, whereas the overall trajectory plotted using a black line. The circle and cross indicate ICU admission and dispatch, respectively.
392
394
396
387
stop
stop
stop
Time-step 42
Time--teg 56
Time-stap 70
Start End
Start End
Start
X
x
x
X
X
Figure S4: Probabilities over discrete patient health states for 6 different time-steps of the selected time series.
398
16


Reviews:
[{"review_id": "SyxaZiL6Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to build a deep clustering algorithm with interpretable latent topology. The authors combine VAE, clustering assignment hardening and SOM. The authors further extend the model to deal with sequence input, by adopting temporal smoothness regularization and language model. \nGood performance in clustering is achieved, experimental results also show that the topological latent structure is captured and the full (sequence) model captures more temporal information than the basic model.\nHowever, the effect of SOM seems to be complicated and not fully explored. \nConsidering that VAE+SOM has been introduced before, the novelty of this paper is to combine a few techniques/tricks to strengthen the clustering performance.\n\nSome concerns:\n1.\tRepresentation\nGood clustering does not guarantee to capture a generally good representation. For example, clustering algorithms lead to a good interpretation of “color” of images that do not need to be good at clustering the semantics of images. If we consider all the factors that generate the data, it is not easy to design metrics to cluster all these factors. \n2.\tVector Quantization/K-means\nCan you review, compare and contrast your work with some recent discrete representation learning works? There are some works (e.g. VQ-VAE (-2)) pursing discrete representations via vector quantization or the Gumbel-softmax trick. The learned representations are also highly structured, can be used for visualization and can be used for further clustering (though might not be as good as clustering-driven approaches).\nHow’s the qualitative difference in the topology of centroids between your method and vector-quantization-based approaches? \n3.\tThe role of SOM\nThe clustering performance mostly comes from clustering assignment hardening, while SOM does not clearly benefit the performance of clustering according to table one. \nIn figure three, you show that SOM’s row in affecting the clustering performance. As tuning SOM would lead to different trends in NMI and purity, what is your strategy in determining your beta?\nHave you ever checked what happened to the centroid topology when varying beta? \n4.\tSome suggestions on experiments\n.\tTo fully understand the benefit of each component of your loss, you can try a variant of VarTPSOM without enforcing “smoothness” \n.\tHow is the beta affect VarTPSOM in sequence evaluation?\n.\tIn your table one, it might be good to compare with some recent discrete latent variable models (as mentioned in bullet point one). \n.       It might be good to try the number of clusters other than 64 in your experiments, towards understanding how it affects the performance.\n\nMinor:\nDenote N as the number of samples, and K as the number of centroids. At the bottom of page 3, you need O(K) to calculate each s_{i,j}. At the beginning of page 4, based on {s_{i,j}}, you either need to store intermediate computational results for acceleration, or you might need O(NK) to calculate each t_{i,j}. The whole process would be much slower than k-means/vector quantization, which seems to be not good for large enough data set.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1l5BDKiYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes VarPSOM, a method which utilizes variational autoencoders (VAEs) and clustering techniques based on self-organizing maps (SOMs) to learn clustering of image data (MNIST and Fashion MNIST in particular). An LSTM-based extension termed VarTPSOM is also evaluated on medical time series data. For the most part, the experimental results are promising, and the visualizations are particularly nice.\n\nOne of my main points of confusion is with the exposition of the method. To start, the objective presented in Eq. 3 is simply a sum of a variational lower bound and the PSOM clustering loss. Does this have a probabilistic interpretation, e.g., is it a lower bound for a particular generative model? If so, this would be useful to discuss prominently in the paper. If not, it is not clear to me what the authors are gaining from the variational framework. The paragraph at the bottom of page 4 that discusses the \"advantages of a VAE over an AE\" is not convincing to me. The authors claim that \"points with a higher variance in the latent space could be identified as potential outliers and therefore treated as less precise and trustworthy\". This isn't demonstrated in the experiments, and to the best of my knowledge, this has not been shown in prior work. If I am mistaken, a citation would be appreciated and should be included in the paper. Additionally, the claim that \"the regularization term of the VAE prevents the network from scattering the embedded points discontinuously in the latent space\" can also be accomplished with AEs with simple regularization, a standard technique for a wide range of AEs.\n\nSimilar comments can be made for the VarTPSOM objective in Eq. 6. Prior work in variational inference for time series, e.g., [1, 2] define a probabilistic time series generative model, from which variational inference naturally prescribes a learning objective. In my opinion, this stands in stark contrast to this work, which takes the VarPSOM objective and simply adds a time series loss on top. This is also a viable approach to building models, but why emphasize variational so much if the method is hardly motivated by anything variational?\n\nI believe that the authors need more thorough experimental comparisons if they wish to demonstrate that their method actually benefits from the variational pieces. Most obviously, I do not believe that any of the comparisons represent the proposed method but with the VAE swapped out for some type of AE? It is my understanding that AE+SOM, SOM-VAE, and DESOM do not represent this exact ablation. VarIDEC performing better than IDEC is a data point in support of this hypothesis, however, this is a comparison of a prior method and not the proposed method.\n\nThe related work section mentions that SOM-VAE and DESOM are \"likely limited by the absence of techniques used in state-of-the-art clustering methods\". Is it possible to address this limitation of prior work? If so, how would this approach compare to the proposed method in terms of implementation and performance? I am not necessarily interested in an actual empirical evaluation, but including this in the related work section would likely be interesting for the reader.\n\nThe authors claim in the implementation details that \"[s]ince the prior in the VAE enforces the latent embeddings to be compact, it also requires more dimensions to learn a meaningful latent space\". Is there a citation for this? My understanding is that posterior collapse leads to VAEs not using additional dimensions even when they are provided, which seems to contradict this claim.\n\nTable 2 seems to have very low NMI numbers across the board, am I reading this incorrectly? Are there prior SOTA numbers that can be included?\n\nFinally, it seems that some of the ideas and motivation in the paper are related to learning discrete structures with variational approaches, e.g., [3, 4]. If the authors agree, it may be appropriate to include some discussion in related work.\n\n[1] Johnson et al, \"Composing graphical models with neural networks for structured representations and fast inference\". NIPS 2016.\n[2] Fraccaro et al, \"Sequential neural models with stochastic layers\". NIPS 2016.\n[3] Tomczak and Welling, \"VAE with a VampPrior\". AISTATS 2018.\n[4] Vikram et al, \"The LORACs prior for VAEs: Letting the trees speak for the data\". AISTATS 2019.\n\n------\n\nTo elaborate on my \"Experience Assessment\" of \"I have read many papers in this area\": \"this area\" in my case refers to amortized variational inference and VAEs, not clustering techniques and SOM.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1e-sprCtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes combining the latent space of a variational autoencoder with two losses that regularize the latent space. The first loss is the cluster hardening loss in Aljalbout et. al [https://arxiv.org/pdf/1801.07648.pdf]. This loss attempts to convert from a  soft-assignments of points (in latent space) to cluster centers (where the assignments are based on similarities computed via a Student t kernel) to a hard assignments of points in latent space to cluster centers. The transformation is posed as the minimization of a KL divergence.\nIn the model under consideration there is assumed to be a grid of \"cluster centers\" or centroids that all points must cluster along.\n\nThe second loss (new in this paper), penalizes the similarities between a point and a centroid from being far away from the similarities of the points to the neighbors of the centroid (on the grid). i.e. this loss tries to ensure that neighboring centroids on the grid correspond to similar points in latent space.\n\nFinally, an extension to temporal data is proposed. The temporal model is as follows: an encoder used at each time step to obtain the latent representation and a third loss to encourage that the latent representations across consecutive time steps stays close to each other is incorporated into the learning algorithm. The latent representation is presented as input to an RNN which is used to reconstruct the data.\n\nOn NMI and cluster purity (evaluated on MNIST, fashionMNISt), the model outperforms two closely related models (the SOM-VAE and DESOM). Similarly on clustering time series data from physionet, the proposed method outperforms the SOM-VAE. The model also compares results on mean squared error (in predicting the time series 6 hours before ICU dispatch) but their baseline is an LSTM model *without* a latent variable -- a fairer baseline would be against their own model with only the variational bound used for learning. The paper also visualizes what is encoded in the centroids.\n\nOverall this paper's contribution is the use of a VAE (rather than an autoencoder as in related work) that contains a latent space regularized to favour learning cluster structure. The paper provides good empirical evidence to suggest that the combination of the proposed losses alongside the VAE does yield better clustering performance. However, I find the addition of the two losses somewhat ad-hoc and little in the way of explanation is provided for when we should expect such a model to work and when it may not. There is not much of a discussion regarding the complexity of learning with the proposed losses but it looks like a simple algorithm for learning with Equation (1) would have to use the entire dataset to compute it. Could you comment on the scalability of the learning algorithm?\n\nMinor comments, there are several places that need editing for grammar and context:\n * The equation at the top of page 4 needs editing within the subscripts of the summation since i is overloaded\n * line 29 talks about \"the observed centroids\", but centroids are not mentioned until much later in the paper\n * expand AE the first time it is used as an acronym\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a deep model for probabilistic clustering and extend it to handle time series data.   The proposed method beats existing deep models on two datasets and  the representations learned in the process are also interpretable.

Unfortunately, despite detailed responses by the authors, the reviewers felt that some of their main concerns were not addressed. For example, the authors and the reviewers are still not converging on whether SOM-VAE uses a VAE or an autoencoder. Further, the discussion about the advantages of VAE over AE is still not very convincing. Currently the work is positioned as a variational clustering method but the reviewers feel that it is a clustering method which uses a VAE (yes, I understand that this difference is subtle but needs to be clarified). 

The reviewers read the responses of the author and during discussions with the AC suggested that there were still not convinced about some of their initial questions. Given this, at this point I would prefer going by the consensus of the reviewers and recommend that this paper cannot be accepted.
Decision: Reject