Variational pSOM: Deep Probabilistic Clustering with Self-Organizing Maps

Generating visualizations and interpretations from high-dimensional data is a1 common problem in many fields . Two key approaches for tackling this problem2 are clustering and representation learning . There are very performant deep cluster-3 ing models on the one hand and interpretable representation learning techniques,4 often relying on latent topological structures such as self-organizing maps , on the5 other hand . However , current methods do not yet successfully combine these two6 approaches . We present a new deep architecture for probabilistic clustering , VarP-7 SOM , and its extension to time series data , VarTPSOM . We show that they achieve8 superior clustering performance compared to current deep clustering methods on9 static MNIST/Fashion-MNIST data as well as medical time series , while inducing10 an interpretable representation . Moreover , on the medical time series , VarTPSOM11 successfully predicts future trajectories in the original data space.12 1 INTRODUCTION13 Information visualization techniques are essential in areas where humans have to make decisions14 based on large amounts of complex data . Their goal is to find an interpretable representation of15 the data that allows the integration of humans into the data exploration process . This encourages16 visual discoveries of relationships in the data and provides guidance to downstream tasks . In this17 way , a much higher degree of confidence in the findings of the exploration is attained ( Keim , 2002 ) .18 An interpretable representation of the data , in which the underlying factors are easily visualized , is19 particularly important in domains where the reason for obtaining a certain prediction is as valuable20 as the prediction itself . However , finding a meaningful and interpretable representation of complex21 data can be challenging.22 Clustering is one of the most natural ways for retrieving interpretable information from raw data.23 Long-established methods such as k-means ( MacQueen , 1967 ) and Gaussian Mixture Models24 ( Bishop , 2006 ) represent the cornerstone of cluster analysis . Their applicability , however , is often25 constrained to simple data and their performance is limited in high-dimensional , complex , real-world26 data sets , which do not exhibit a clustering-friendly structure.27 Deep generative models have recently achieved tremendous success in representation learning.28 Some of the most commonly used and efficient approaches are Autoencoders ( AEs ) , Variational29 Autoencoders ( VAEs ) and Generative Adversarial Networks ( GANs ) ( Kingma & Welling , 2013 ; 30 Goodfellow et al. , 2014 ) . The compressed latent representation generated by these models has been31 proven to ease the clustering process ( Aljalbout et al. , 2018 ) . As a result , the combination of deep32 generative models for feature extraction and clustering results in a dramatic increase of the clustering33 performance ( Xie et al. , 2015 ) . Although very successful , most of these methods do not investigate34 the relationship among clusters and the clustered feature points live in a high-dimensional latent35 space that can not be easily visualized or interpreted by humans.36 The Self-Organizing Map ( SOM ) ( Kohonen , 1990 ) is a clustering method that provides such an37 interpretable representation . It produces a low-dimensional ( typically 2-dimensional ) , discretized38 representation of the input space by inducing a flexible neighbourhood structure over the clusters.39 Alas , its applicability is often constrained to simple data sets similar to other classical clustering40 methods.41 To resolve the above issues , we propose a novel deep architecture , the Variational Probabilistic SOM42 ( VarPSOM ) , that jointly trains a VAE and a SOM to achieve an interpretable discrete representation43 while exhibiting state-of-the-art clustering performance . Instead of hard assignment of data points44 to clusters , our model uses a centroid-based probability distribution . It minimizes its Kullback-45 Leibler divergence against an auxiliary target distribution , while enforcing a SOM-friendly space.46 To highlight the importance of an interpretable representation for different purposes , we extended47 this model to deal with temporal data , yielding VarTPSOM . We discuss related work in Section48 2 . Extensive evidence of the superior clustering performance of both models , on MNIST/Fashion-49 MNIST images as well as real-world medical time series is presented in Section 4.50 Our main contributions are:51 • A novel architecture for deep clustering , yielding an interpretable discrete representation52 through the use of a probabilistic self-organizing map.53 • An extension of this architecture to time series , improving clustering performance on this54 data type and enabling temporal predictions.55 • A thorough empirical assessment of our proposed models , showing superior performance56 on benchmark tasks and challenging medical time series from the intensive care unit.57 2 RELATED WORK58 Self-Organizing Maps have been widely used as a means to visualize information from large59 amounts of data ( Tirunagari et al. , 2014 ) and as a form of clustering in which the centroids are con-60 nected by a topological neighborhood structure ( Flexer , 1999 ) . Since their early inception , several61 variants have been proposed to enhance their performance and scope . The adaptive subspace SOM,62 ASSOM ( Kohonen , 1995 ) , for example , proposed to combine PCA and SOMs to map data into a63 reduced feature space . ( Tokunaga & Furukawa , 2009 ) combine SOMs with multi-layer perceptrons64 to obtain a modular network . ( Liu et al. , 2015 ) proposed Deep SOM ( DSOM ) , an architecture com-65 posed of multiple layers similar to Deep Neural Networks . There exist several methods tailored66 to representation learning on time series , among them ( Franceschi et al. , 2019 ; Fortuin & Rätsch,67 2019 ; Fortuin et al. , 2019 ) , which are however not based on SOMs . Extensions of SOM optimized68 for temporal data include the Temporal Kohonen map ( Chappell & Taylor , 1993 ) and its improved69 version Recurrent SOM ( McQueen et al. , 2004 ) as well as Recursive SOM ( Voegtlin , 2002 ) . While70 SOM and its variants are particularly effective for data visualization ( Liu et al. , 2015 ) , it was rarely71 attempted to combine their merits in this respect with modern state-of-the-art clustering methods,72 which often use deep generative models in combination with probabilistic clustering.73 In particular , recent works on clustering analysis have shown that combining clustering algorithms74 with the latent space of AEs greatly increases the clustering performance ( Aljalbout et al. , 2018 ) .75 ( Xie et al. , 2015 ) proposed DEC , a method that sequentially applies embedding learning using76 Stacked Autoencoders ( SAE ) , and the Clustering Assignment Hardening method on the obtained77 representation . An improvement of this architecture , IDEC ( Guo et al. , 2017 ) , includes the decoder78 network of the SAE in the learning process , so that training is affected by both the clustering loss79 and the reconstruction loss . Similarly , DCN ( Yang et al. , 2016 ) combines a k-means clustering loss80 with the reconstruction loss of SAE to obtain an end-to-end architecture that jointly trains repre-81 sentations and clustering . These models achieve state-of-the-art clustering performance but they do82 not investigate the relationship among clusters . An exception is the work by ( Li et al. , 2018 ) , in83 which they present an unsupervised method that learns latent embeddings and discovers multi-facet84 clustering structure . Relationships among clusters were discovered , however , they do not provide a85 latent space that can be easily interpreted and which eases the process of analytical reasoning.86 While there exist previous efforts to endow VAEs with a hierarchical latent space ( Vikram et al.,87 2018 ; Goyal et al. , 2017 ) , to the best of our knowledge , only two models used deep generative88 models in combination with a SOM structure in the latent space . The SOM-VAE model ( Fortuin89 et al. , 2018 ) , inspired by the VQ-VAE architecture ( van den Oord et al. , 2017 ) ( which itself was90 later extended in ( Razavi et al. , 2019 ) ) , uses an AE to embed the input data points into a latent space91 and then applies a SOM-based clustering loss on top of this latent representation . It features hard92 assignments of points to centroids , as well as the use of a Markov model for temporal data , both93 of which yield inferior expressivity compared to our method . The Deep Embedded SOM , DESOM94 ( Forest et al. , 2019 ) , improved the previous model by using a Gaussian neighborhood window with95 exponential radius decay and by learning the SOM structure in a continuous setting . Both methods96 feature a topologically interpretable neighborhood structure and yield promising results in visual-97 izing state spaces . However , those works did not feature empirical comparisons to state-of-the-art98 deep clustering techniques and did not make use of many of the design principles that have recently99 proven to be successful in this space.100 3 PROBABILISTIC CLUSTERING WITH VARIATIONAL PSOM101 Given a set of data samples { xi } i=1 , ... , n , where xi ∈ Rd , the goal is to partition the data into a set102 of clusters { Si } i=1 , ... , K , while retaining a topological structure over the cluster centroids.103 The proposed architecture for static data is presented in Figure 1a . The input vector xi is embedded104 into a latent representation zi using a VAE . This latent vector is then clustered using PSOM , a105 new SOM clustering strategy that extends the Clustering Assignment Hardening method ( Xie et al.,106 2015 ) . The VAE and PSOM are trained jointly to learn a latent representation with the aim to boost107 the clustering performance . To prevent the network from outputting a trivial solution , the decoder108 network reconstructs the input from the latent embedding , encouraging it to be as similar as possible109 to the original input . The obtained loss function is a linear combination of the clustering loss and110 the reconstruction loss . To deal with temporal data , we propose another model variant , which is111 depicted in Figure 1b.112 3.1 BACKGROUND113 A Self-Organizing Map is comprised of K nodes connected to form a grid M ⊆ N2 , where the114 node mi , j , at position ( i , j ) of the grid , corresponds to a centroid vector , µi , j in the input space . The115 centroids are tied by a neighborhood relation N ( µi , j ) = { µi−1 , j , µi+1 , j , µi , j−1 , µi , j+1 } . Given a116 random initialization of the centroids , the SOM algorithm randomly selects an input xi and updates117 both its closest centroid µi , j and its neighbors N ( µi , j ) to move them closer to xi . For a complete118 description of the SOM algorithm , we refer to the appendix ( A ) .119 The Clustering Assignment Hardening method has been recently introduced by the DEC model ( Xie et al. , 2015 ) and was shown to perform well in the latent space of AEs ( Aljalbout et al. , 2018 ) . Given an embedding function zi = f ( xi ) , it uses a Student ’ s t-distribution ( S ) as a kernel to measure the similarity between an embedded data point zi , and a centroid µj : sij = ( 1 + ‖zi − µj‖2 /α ) −α+12 ∑ j′ ( 1 + ‖zi − µj′‖2 /α ) −α+12 . It improves the cluster purity by enforcing the distribution S to approach a target distribution , T : tij = sγij/ ∑ i′ si′j∑ j′ s γ ij′/ ∑ i′ si′j′ . By taking the original distribution to the power of γ and normalizing it , the target distribution puts more emphasis on data points that are assigned a high confidence . We follow ( Xie et al. , 2015 ) in choosing γ=2 , which leads to larger gradient contributions of points close to cluster centers , as they show empirically . The resulting clustering loss is defined as : L = KL ( T‖S ) = ∑ i ∑ j tij log tij sij . ( 1 ) 3.2 PROBABILISTIC SOM ( PSOM ) CLUSTERING120 Our proposed clustering method , called PSOM , expands Clustering Assignment Hardening to include a SOM neighborhood structure over the centroids . We add an additional loss to ( 1 ) to achieve an interpretable representation . This loss term maximizes the similarity between each data point and the neighbors of the closest centroids . For each embedded data point zi and each centroid µj the loss is defined as the negative sum of all the neighbors of µj , { e : µe ∈ N ( µj ( xi ) ) } , of the probability that zi is assigned to e , defined as sie . This sum is weighted by the similarity sij between zi and the centroid µj : LSOM = − 1 N ∑ i ∑ j sij ∑ e : µe∈N ( µj ( xi ) ) sie . The complete PSOM clustering loss is then : LPSOM = KL ( T‖S ) + βLSOM . We note that for β = 0 it becomes equivalent to Clustering Assignment Hardening.121 3.3 VARPSOM : VAE FOR FEATURE EXTRACTION122 In our method , the nonlinear mapping between the input xi and embedding zi is realized by a VAE . Instead of directly embedding the input xi into a latent embedding zi , the VAE learns a probability distribution qφ ( z | xi ) parametrized as a multivariate normal distribution whose mean and variance are ( µφ , Σφ ) = fφ ( xi ) . Similarly , it also learns the probability distribution of the reconstructed output given a sampled latent embedding , pθ ( xi | z ) where ( µθ , Σθ ) = fθ ( zi ) . Both fφ and fθ are neural networks , respectively called encoder and decoder . The ELBO loss is : LELBO = ∑ i [ −Ez ( log pθ ( xi | z ) ) +DKL ( qφ ( z | xi ) ‖ p ( z ) ) ] , ( 2 ) where p ( z ) is an isotropic Gaussian prior over the latent embeddings . The second term can be interpreted as a form of regularization , which encourages the latent space to be compact . For each data point xi the latent embedding zi is sampled from qφ ( z | xi ) . Adding the ELBO loss to the PSOM loss from the previous subsection , we get the overall loss function of VarPSOM : LVarPSOM = LPSOM + LELBO . ( 3 ) To the best of our knowledge , no previous SOM methods attempted to use a VAE to embed the123 inputs into a latent space . There are many advantages of a VAE over an AE for realizing our goals.124 Its prior on the latent space encourages structured and disentangled factors ( Higgins et al. , 2016 ) ,125 which could help clustering . The suitability of VAEs for anomaly detection ( An & Cho , 2015 ) 126 means that points with a higher variance in the latent space could be treated as less accurate and127 trustworthy . The regularization term of the VAE can be used to prevent the network from scattering128 the embedded points discontinuously in the latent space , which naturally facilitates the fitting of the129 SOM . To test if the use of CNNs can boost clustering performance on image data , we introduce130 another model variant called VarCPSOM , which uses convolutional filters as part of the VAE.131 3.4 VARTPSOM : EXTENSION TO TIME SERIES DATA132 To extend our proposed model to time series data , we add a temporal component to the architecture . Given a set of N time series of length T , { xt , i } t=1 , ... , T ; i=1 , ... , N , the goal is to learn interpretable trajectories on the SOM grid . To do so , the VarPSOM could be used directly but it would treat each time step t of the time series independently , which is undesirable . To exploit temporal information and enforce smoothness in the trajectories , we add an additional loss to ( 3 ) : Lsmooth = − 1 NT ∑ i ∑ t uit , it+1 , ( 4 ) where uit , it+1 = g ( zi , t , zi , t+1 ) is the similarity between zi , t and zi , t+1 using a Student ’ s t-133 distribution and zi , t refers to the embedding of time series xi at time index t. It maximizes the134 similarity between latent embeddings of adjacent time steps , such that large jumps in the latent state135 between time points are discouraged.136 One of the main goals in time series modeling is to predict future data points , or alternatively , future embeddings . This can be achieved by adding a long short-term memory network ( LSTM ) across the latent embeddings of the time series , as shown in Fig 1b . Each cell of the LSTM takes as input the latent embedding zt at time step t , and predicts a probability distribution over the next latent embedding , pω ( zt+1 | zt ) . We parametrize this distribution as a Multivariate Normal Distribution whose mean and variance are learnt by the LSTM . The prediction loss is the log-likelihood between the learned distribution and a sample of the next embedding zt+1 : Lpred = − ∑ i ∑ t log pω ( zt+1 | zt ) . ( 5 ) The final loss of VarTPSOM , which is trainable in a fully end-to-end fashion , is LVarTPSOM = LVarPSOM + Lsmooth + ηLpred . ( 6 ) 4 EXPERIMENTS137 First , we evaluate VarPSOM and VarCPSOM and compare them with state-of-the-art non-138 interpretable as well as SOM-based clustering methods on MNIST ( Lecun et al. , 1998 ) and Fashion-139 MNIST ( Xiao et al. , 2017 ) data . Here , particular focus is laid on the comparison of VarPSOM and140 the clustering models DEC and IDEC , to investigate the role of the VAE and the SOM loss . We141 then present visualizations of the obtained 2D representations , to illustrate how our method could142 ease visual reasoning about the data . Finally , we present extensive evidence of the performance of143 VarTPSOM on real-world complex time series from the eICU data set ( Pollard et al. , 2018 ) , and144 illustrate how it allows visualization of patient health state trajectories in an easily understandable145 2D domain . For details on the data sets , we refer to the appendix ( B.1 ) .146 Baselines We used two different types of baselines . The first category contains clustering methods147 that do not provide any interpretable discrete latent representation . Those include k-means , the DEC148 model , as well as its improved version IDEC , whose clustering methods are related to ours . We also149 include a modified version of IDEC that we call VarIDEC , in which we substitute the AE with a150 VAE , to investigate the role of the VAE . For all these methods we use 64 clusters . In the second151 category , we include state-of-the-art clustering methods based on SOMs . Here , we used a standard152 SOM ( minisom ) , AE+SOM , an architecture composed of an AE and a SOM applied on top of the153 latent representation ( trained sequentially ) , SOM-VAE and DESOM . Finally , we create a modified154 version of our model , called AEPSOM , in which we substitute the VAE with an AE ( similarly to155 VarIDEC ) . For all SOM-based methods we set the SOM grid size to ( 8 × 8 ) . For different grid156 configurations we refer to the appendix , ( B.3 ) .157 Implementation In implementing our models we focused on retaining a fair comparison with the158 baselines . Hence we decided to use a standard network structure , with fully connected layers of159 dimensions d− 500− 500− 2000− l , to implement both the VAE of our models and the AE of the160 baselines . The latent dimension , l , is set to 100 for the VAE , and to 10 for the AEs . Since the prior161 in the VAE enforces the latent embeddings to be compact , it also requires more dimensions to learn162 a meaningful latent space . On the other hand , providing the AE models with a higher-dimensional163 latent space , needed for the VAE , resulted in a dramatic decrease of performance ( see appendix B.2 ) .164 VarCPSOM is composed of 4 convolutional layers of feature maps [ 32 , 64 , 128 , 256 ] and kernel size165 3 × 3 for all layers . For all architectures , no greedy layer-wise pretraining was used to tune the166 VAE . Instead we simply run the VAE without the clustering loss for a few epochs for initialization.167 A standard SOM was then used to produce an initial configuration of the centroids/neighbourhood168 relation . Finally , the entire architecture is trained for 100 , 000 iterations . To avoid fine-tuning hy-169 perparameters , given the unsupervised setting , α is set to 10 for all experiments while the other170 hyperparameters are modified accordingly to maintain the same order of magnitude of the different171 loss components.172 Clustering Evaluation Table 1 shows the clustering quality results of VarPSOM and VarCPSOM173 on MNIST and Fashion-MNIST data , compared with the baselines . Purity and Normalized Mutual174 Information are used as evaluation metrics . We observe that our proposed models outperform the175 baselines of both categories and achieve state-of-the-art clustering performance.176 VarPSOM vs. IDEC VarPSOM is inspired by IDEC but it has two major differences . It uses a177 VAE instead of an AE and it improves interpretability in the latent space by adding a new loss that en-178 forces a SOM structure . Since both VarIDEC and VarPSOM show superior clustering performance179 compared to IDEC and AEPSOM respectively ( Table 1 ) , we conclude that the VAE indeed suc-180 ceeds in capturing a more meaningful latent representation compared to a standard AE . Regarding181 the second difference , the SOM structure was expected to slightly decrease the clustering perfor-182 mance , due to a trade-off between interpretability and raw clustering performance . However , we do183 not observe this in our results . Adding the SOM loss rather leads to an increase of the clustering184 performance . We suspect this is due to the regularization effect of the SOM ’ s topological structure.185 Overall , VarPSOM outperforms both DEC and IDEC.186 Improvement over Training After obtaining the initial configuration of the SOM structure , both187 clustering and feature extraction using the VAE are trained jointly . To illustrate that our architecture188 improves clustering performance over the initial configuration , we plotted NMI and Purity against189 the number of training iterations in Figure 2 . We observe that the performance is stable when190 increasing the number of epochs and no overfitting is visible.191 Role of the SOM loss To investigate the influence of the SOM loss component , we plot the clus-192 tering performance of VarPSOM against the weight ( β ) of LSOM in Fig . 3 , using the MNIST dataset.193 With β = 30 , theKL term ( responsible for improving clustering purity ) and the LSOM term ( respon-194 sible for enforcing a SOM structure over the centroids ) are almost equal . It is interesting to observe195 the different trends in NMI and purity . The NMI performance increases for increasing values of β196 while purity slightly decreases . Overall , enforcing a more interpretable latent space results in a more197 robust clustering model with higher NMI clustering performance.198 Time Series Evaluation We evaluate the clustering performance of our proposed models on the199 eICU dataset , comprised of complex medical time series . We compare them against SOM-VAE,200 as this is the only method among the baselines that is suited for temporal data . Table 2 shows the201 cluster cell enrichment in terms of NMI for three different labels , the current ( APACHE-0 ) and worst202 future ( APACHE-6/12 hours ) physiology scores . VarTPSOM clearly achieves superior clustering203 performance compared to SOM-VAE . This , we hypothesize , is due to the better feature extraction204 using a VAE as well as the improved treatment of uncertainty using PSOM , which features soft205 assignments , whereas SOM-VAE contains a deterministic AE and hard assignments . Moreover,206 both the smoothness loss and the prediction loss seem to increase the clustering performance . More207 results on ICU time series are reported in the appendix ( B.4 ) .208 To quantify the performance of VarTPSOM in unrolling future trajectories , we predict the final209 6 latent embeddings of each time series . For each predicted embedding we reconstruct the input210 using the decoder of the VAE . Finally , we measure the MSE between the original input and the211 reconstructed inputs for the last 6 hours of the ICU admission . As baselines , we used an LSTM that212 takes as input the first 66 hours of the time series and then predicts the next 6 hours . Since most213 of the trajectories tend to stay in the same state over long periods of time , another strong baseline214 is obtained by duplicating the last seen embedding over the final 6 hours . The results ( Table 3 ) 215 indicate that the joint training of clustering and prediction used by VarTPSOM clearly outperforms216 the 2 baselines.217 Model LSTM SameState VarTPSOM MSE 0.0386± 0.0049 0.0576± 0.0012 0.0297± 0.0009 Interpretability To illustrate the topological structure in the latent space , we present reconstruc-218 tions of the VarPSOM centroids , arranged in a ( 8× 8 ) grid , on static MNIST/Fashion-MNIST data219 in Figure 4 . On the ICU time series data , we show example trajectories for one patient dying at the220 end of the ICU stay , as well as two control patients which are dispatched healthily from the ICU . We221 observe that the trajectories are located in different parts of the SOM grid , and form a smooth and222 interpretable representation ( Fig . 5 ) . For further results , including a more quantitative evaluation223 using randomly sampled trajectories , enrichment for future mortality as well as an illustration of224 how the uncertainty generated by the soft assignments can help in data visualization , we refer to the225 appendix ( B.5 ) .226 5 CONCLUSION227 We presented two novel methods for interpretable unsupervised clustering , VarPSOM and VarTP-228 SOM . Both models make use of a VAE and a novel clustering method , PSOM , that extends the229 classical SOM algorithm to include a centroid-based probability distribution . Our models achieve230 superior clustering performance compared to state-of-the-art deep clustering baselines on bench-231 mark data sets and real-world medical time series . The use of a VAE for feature extraction , instead232 of an AE , used in previous methods , and the use of soft assignments of data points to clusters result233 in an interpretable model that can quantify uncertainty in the data.234 REFERENCES235 Elie Aljalbout , Vladimir Golkov , Yawar Siddiqui , and Daniel Cremers . Clustering with deep learn-236 ing : Taxonomy and new methods . CoRR , abs/1801.07648 , 2018 . URL http : //arxiv.org/237 abs/1801.07648.238 Jinwon An and Sungzoon Cho . Variational autoencoder based anomaly detection using reconstruc-239 tion probability . Special Lecture on IE , 2 ( 1 ) , 2015.240 Christopher M. Bishop . Pattern Recognition and Machine Learning ( Information Science and Statis-241 tics ) . Springer-Verlag , Berlin , Heidelberg , 2006 . ISBN 0387310738.242 Geoffrey J. Chappell and John G. Taylor . The temporal kohonen map . Neural Netw. , 6 ( 3 ) :441–445,243 March 1993 . ISSN 0893-6080. doi : 10.1016/0893-6080 ( 93 ) 90011-K. URL http : //dx.doi.244 org/10.1016/0893-6080 ( 93 ) 90011-K.245 Arthur Flexer . On the use of self-organizing maps for clustering and visualization . In Jan M. Żytkow246 and Jan Rauch ( eds . ) , Principles of Data Mining and Knowledge Discovery , pp . 80–88 , Berlin,247 Heidelberg , 1999 . Springer Berlin Heidelberg . ISBN 978-3-540-48247-5.248 Florent Forest , Mustapha Lebbah , Hanene Azzag , and Jérôme Lacaille . Deep embedded som : Joint249 representation learning and self-organization . 04 2019.250 Vincent Fortuin and Gunnar Rätsch . Deep mean functions for meta-learning in gaussian processes.251 arXiv preprint arXiv:1901.08098 , 2019.252 Vincent Fortuin , Matthias Hüser , Francesco Locatello , Heiko Strathmann , and Gunnar Rätsch.253 Som-vae : Interpretable discrete representation learning on time series . arXiv preprint254 arXiv:1806.02199 , 2018.255 Vincent Fortuin , Gunnar Rätsch , and Stephan Mandt . Multivariate time series imputation with256 variational autoencoders . arXiv preprint arXiv:1907.04155 , 2019.257 Jean-Yves Franceschi , Aymeric Dieuleveut , and Martin Jaggi . Unsupervised scalable representation258 learning for multivariate time series . arXiv preprint arXiv:1901.10738 , 2019.259 Ian J. Goodfellow , Jean Pouget-Abadie , Mehdi Mirza , Bing Xu , David Warde-Farley , Sherjil Ozair,260 Aaron Courville , and Yoshua Bengio . Generative Adversarial Networks . arXiv e-prints , art.261 arXiv:1406.2661 , Jun 2014.262 Prasoon Goyal , Zhiting Hu , Xiaodan Liang , Chenyu Wang , and Eric P Xing . Nonparametric vari-263 ational auto-encoders for hierarchical representation learning . In Proceedings of the IEEE Inter-264 national Conference on Computer Vision , pp . 5094–5102 , 2017.265 Xifeng Guo , Long Gao , Xinwang Liu , and Jianping Yin . Improved deep embedded clustering with266 local structure preservation . In Proceedings of the Twenty-Sixth International Joint Conference267 on Artificial Intelligence , IJCAI-17 , pp . 1753–1759 , 2017. doi : 10.24963/ijcai.2017/243 . URL268 https : //doi.org/10.24963/ijcai.2017/243.269 Irina Higgins , Loic Matthey , Xavier Glorot , Arka Pal , Benigno Uria , Charles Blundell , Shakir Mo-270 hamed , and Alexander Lerchner . Early visual concept learning with unsupervised deep learning.271 arXiv preprint arXiv:1606.05579 , 2016.272 D. A. Keim . Information visualization and visual data mining . IEEE Transactions on Visualization273 and Computer Graphics , 8 ( 1 ) :1–8 , Jan 2002 . ISSN 1077-2626. doi : 10.1109/2945.981847.274 Diederik P Kingma and Max Welling . Auto-Encoding Variational Bayes . arXiv e-prints , art.275 arXiv:1312.6114 , Dec 2013.276 T. Kohonen . The self-organizing map . Proceedings of the IEEE , 78 ( 9 ) :1464–1480 , Sep. 1990 . ISSN277 0018-9219. doi : 10.1109/5.58325.278 Teuvo Kohonen . The adaptive-subspace som ( assom ) and its use for the implementation of invariant279 feature detection . 1995.280 Y. Lecun , L. Bottou , Y. Bengio , and P. Haffner . Gradient-based learning applied to document281 recognition . Proceedings of the IEEE , 86 ( 11 ) :2278–2324 , Nov 1998 . ISSN 0018-9219. doi:282 10.1109/5.726791.283 Xiaopeng Li , Zhourong Chen , and Nevin L. Zhang . Latent tree variational autoencoder for joint284 representation learning and multidimensional clustering . CoRR , abs/1803.05206 , 2018 . URL285 http : //arxiv.org/abs/1803.05206.286 N. Liu , J. Wang , and Y. Gong . Deep self-organizing map for visual classification . In 2015 Interna-287 tional Joint Conference on Neural Networks ( IJCNN ) , pp . 1–6 , July 2015. doi : 10.1109/IJCNN.288 2015.7280357.289 J. MacQueen . Some methods for classification and analysis of multivariate observations . In290 Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability , Vol-291 ume 1 : Statistics , pp . 281–297 , Berkeley , Calif. , 1967 . University of California Press . URL292 https : //projecteuclid.org/euclid.bsmsp/1200512992.293 T. A. McQueen , A . A. Hopgood , J . A. Tepper , and T. J. Allen . A recurrent self-organizing map for294 temporal sequence processing . In Ahamad Lotfi and Jonathan M. Garibaldi ( eds . ) , Applications295 and Science in Soft Computing , pp . 3–8 , Berlin , Heidelberg , 2004 . Springer Berlin Heidelberg.296 ISBN 978-3-540-45240-9.297 Tom J Pollard , Alistair EW Johnson , Jesse D Raffa , Leo A Celi , Roger G Mark , and Omar Badawi.298 The eicu collaborative research database , a freely available multi-center database for critical care299 research . Scientific data , 5 , 2018.300 Ali Razavi , Aaron van den Oord , and Oriol Vinyals . Generating diverse high-fidelity images with301 vq-vae-2 . arXiv preprint arXiv:1906.00446 , 2019.302 S. Tirunagari , N. Poh , K. Aliabadi , D. Windridge , and D. Cooke . Patient level analytics using303 self-organising maps : A case study on type-1 diabetes self-care survey responses . In 2014 IEEE304 Symposium on Computational Intelligence and Data Mining ( CIDM ) , pp . 304–309 , Dec 2014.305 doi : 10.1109/CIDM.2014.7008682.306 Kazuhiro Tokunaga and Tetsuo Furukawa . Modular network som . Neural Networks , 22 ( 1 ) :82 –307 90 , 2009 . ISSN 0893-6080. doi : https : //doi.org/10.1016/j.neunet.2008.10.006 . URL http:308 //www.sciencedirect.com/science/article/pii/S0893608008002335.309 Aäron van den Oord , Oriol Vinyals , and Koray Kavukcuoglu . Neural discrete representation learn-310 ing . CoRR , abs/1711.00937 , 2017 . URL http : //arxiv.org/abs/1711.00937.311 Sharad Vikram , Matthew D Hoffman , and Matthew J Johnson . The loracs prior for vaes : Letting the312 trees speak for the data . arXiv preprint arXiv:1810.06891 , 2018.313 Thomas Voegtlin . Recursive self-organizing maps . Neural Networks , 15 ( 8 ) :979 – 991 , 2002.314 ISSN 0893-6080. doi : https : //doi.org/10.1016/S0893-6080 ( 02 ) 00072-2 . URL http : //www.315 sciencedirect.com/science/article/pii/S0893608002000722.316 Han Xiao , Kashif Rasul , and Roland Vollgraf . Fashion-mnist : a novel image dataset for benchmark-317 ing machine learning algorithms . CoRR , abs/1708.07747 , 2017 . URL http : //arxiv.org/318 abs/1708.07747.319 Junyuan Xie , Ross B. Girshick , and Ali Farhadi . Unsupervised deep embedding for clustering320 analysis . CoRR , abs/1511.06335 , 2015 . URL http : //arxiv.org/abs/1511.06335.321 Bo Yang , Xiao Fu , Nicholas D. Sidiropoulos , and Mingyi Hong . Towards k-means-friendly322 spaces : Simultaneous deep learning and clustering . CoRR , abs/1610.04794 , 2016 . URL323 http : //arxiv.org/abs/1610.04794.324 APPENDIX325 A SELF-ORGANIZING MAPS326 Among various existing interpretable unsupervised learning algorithms , Kohonen ’ s self-organizing327 map ( SOM ) ( Kohonen , 1990 ) is one of the most popular models . It is comprised of K neurons328 connected to form a discrete topological structure . The data are projected onto this topographic map329 which locally approximates the data manifold . Usually it is a finite two-dimensional region where330 neurons are arranged in a regular hexagonal or rectangular grid . Here we use a grid , M ⊆ N2,331 because of its simplicity and its visualization properties . Each neuron mij , at position ( i , j ) of the332 grid , for i , j = 1 , . . . , √ K , corresponds to a centroid vector , µi , j in the input space . The centroids333 are tied by a neighborhood relation , here defined as N ( µi , j ) = { µi−1 , j , µi+1 , j , µi , j−1 , µi , j+1 } .334 Given a random initialization of the centroids , the SOM algorithm randomly selects an input xi and335 updates both its closest centroid µi , j and its neighbors N ( µi , j ) to move them closer to xi . The336 algorithm ( 1 ) then iterates these steps until convergence.337 Algorithm 1 Self-Organizing Maps Require : 0 < α ( t ) < 1 ; limt→∞ ∑ α ( t ) →∞ ; limt→∞ ∑ α2 ( t ) < ∞ ; repeat At each time t , present an input x ( t ) and select the winner , ν ( t ) = arg min k∈Ω ‖x ( t ) −wk ( t ) ‖ Update the weights of the winner and its neighbours , ∆wk ( t ) = α ( t ) η ( ν , k , t ) [ x ( t ) −wν ( t ) ] until the map converges The range of SOM applications includes high dimensional data visualization , clustering , image338 and video processing , density or spectrum profile modeling , text/document mining , management339 systems and gene expression data analysis.340 B EXPERIMENTAL AND IMPLEMENTATION DETAILS341 B.1 DATASETS342 • MNIST : It consists of 70000 handwritten digits of 28-by-28 pixel size . Digits range from343 0 to 9 , yielding 10 patterns in total . The digits have been size-normalized and centered in a344 fixed-size image Lecun et al . ( 1998 ) .345 • Fashion MNIST : A dataset of Zalando ’ s article images consisting of a training set of346 60,000 examples and a test set of 10,000 examples Xiao et al . ( 2017 ) . Each example is347 a 28×28 grayscale image , associated with a label from 10 classes.348 • eICU : For temporal data we use vital sign/lab measurements of intensive care unit ( ICU ) 349 patients resampled to a 1-hour based grid using forward filling and filling with population350 statistics from the training set if no measurements were available . From all ICU stays , we351 excluded ICU stays , which were shorter than 1 day , longer than 30 days or which had at352 least one gap in the continuous vital sign monitoring , which we define by a interval between353 2 HR measurements of at least 1 hour . This yielded N = 10559 ICU stays from the354 eICU database . dvitals = 14 vital sign variables and dlab = 84 lab measurement variables355 were included , giving an overall data dimension of d = 98 . The last 72 hours of these356 multivariate time series were used for the experiments . As labels we use a variant of the357 current dynamic APACHE physiology score ( APACHE-0 ) as well as the worst APACHE358 score in the next 6 and 12 hours ( APACHE-6/12 ) , and the mortality in the next 24 hours.359 Only those variables from the APACHE score definition which are recorded in the eICU360 database were taken into account.361 Each dataset is divided into training , validation and test sets for both our models and the baselines.362 B.2 LATENT SPACE DIMENSION363 We evaluate the DEC model for different latent space dimensions . Table S1 shows that the AE , used364 in the DEC model , performs better when a lower dimensional latent space is used.365 Table S1 : Mean/Standard error of NMI and purity of DEC model on MNIST test set , across 10 runs with different random model initializations . We use 64 clusters and different latent space dimensions . Latent dimension Purity NMI l = 10 0.950± 0.001 0.681± 0.001 l = 100 0.750± 0.001 0.573± 0.001 B.3 NUMBER OF CLUSTERS366 We evaluate the NMI and purity clustering performance of our model , VarPSOM , with a varying367 number of clusters on the MNIST dataset . Since IDEC represents the main competitor we include368 it in this analysis . Figure S1 shows that VarPSOM outperforms IDEC for all the different configu-369 rations . In particular , it is interesting to observe that NMI decreases with an increasing number of370 clusters in both models . This is because the entropy of the clustering increases with the number of371 clusters.372 Figure S1 : NMI ( left ) and purity ( right ) clustering performance of VarPSOM and IDEC with varying number of clusters on the MNIST test set . B.4 LEARNING HEALTH STATE REPRESENTATIONS IN THE ICU373 By enforcing a SOM structure , VarPSOM , as well as VarTPSOM , project the cluster centroids onto374 a discrete 2D grid . Such a grid is particularly suited for visualization purposes and relations between375 centroids become immediately intuitive . In Fig . S2 a heat-map ( colored according to enrichment in376 the current APACHE score , as well as mortality risk in the next 24 hours ) shows compact enrichment377 structures . VarTPSOM succeeds in creating a meaningful and smooth neighbourhood structure . It378 distinguishes risk profiles with practically zero mortality risk from high mortality risk , reaching up379 to≈15 % , in different regions of the map , even though it is learned in a purely unsupervised fashion.380 Remarkably , the two heat-maps ( S2b and S2a ) show different enrichment patterns . Clusters which381 are enriched in health states with higher APACHE scores often do not correspond exactly to clusters382 with a higher mortality risk . This suggests that traditional representations of physiologic values,383 such as the APACHE score , fail to fully use all complex multivariate relationships present in the384 ICU recordings , and are not associated with dynamic mortality in a simple way.385 ( a ) Current APACHE score ( b ) Mortality risk in the next 24 hours Figure S2 : Heat-maps of enrichment in mortality risk in the next 24 hours as well as the current dynamic APACHE score , superimposed on the discrete 2D grid learned by VarTPSOM . B.5 VISUALIZING HEALTH STATE TRAJECTORIES IN THE ICU386 To analyze the trend of the patient pathology , VarTPSOM induces trajectories on the 2D SOM grid387 which can be easily visualized . Fig . S3 shows 20 randomly sampled patient trajectories obtained388 by our model . Trajectories ending in the death of the patient are shown in red , healthily dispatched389 patients are shown in green . Figure S3 : Randomly sampled VarTPSOM trajectories , from patients expired at the end of the ICU stay , as well as healthily dispatched patients . Superimposed is a heatmap which displays the cluster enrichment in the current APACHE score , from this model run . We observe that trajectories of dying patients are often in different locations of the map as healthy patients , in particular in those regions enriched for high APACHE scores , which corresponds with clinical intuition . 390 One of the main advantage of VarTPSOM over the traditional SOM algorithm is the use of soft391 assignments of data points to clusters which results in a better ability to quantify uncertainty in the392 data . For visualizing health states in the ICU , this property is very important . In Fig S4 we plot an393 example patient trajectory , where 6 different time-steps ( in temporal order ) of the trajectory were394 chosen . Our model yields a soft centroid-based probability distribution which evolves with time and395 which allows estimation of likely discrete health states at a given point in time . For each time-step396 the distribution of probabilities is plotted using a heat-map , whereas the overall trajectory is plotted397 using a black line . The circle and cross indicate ICU admission and dispatch , respectively . Figure S4 : Probabilities over discrete patient health states for 6 different time-steps of the selected time series . 398

Summary: The paper proposes combining the latent space of a variational autoencoder with two losses that regularize the latent space. The first loss is the cluster hardening loss in Aljalbout et. al [https://arxiv.org/pdf/1801.07648.pdf]. This loss attempts to convert from a  soft-assignments of points (in latent space) to cluster centers (where the assignments are based on similarities computed via a Student t kernel) to a hard assignments of points in latent space to cluster centers. The transformation is posed as the minimization of a KL divergence.