Mixture-of-Experts Variational Autoencoder for clustering and generating from similarity-based representations
Authors: Andreas Kopf
Vincent Fortuin
Vignesh Ram Somnath
Manfred Claassen
Venue:  (2020)
Keywords: Variational Autoencoder
Clustering
Generative model

Abstract: Clustering high-dimensional data, such as images or biological measurements, is a long-standing problem and has been studied extensively. Recently, Deep Clustering gained popularity due to the non-linearity of neural networks, which allows for flexibility in fitting the specific peculiarities of complex data. Here we introduce the Mixture-of-Experts Similarity Variational Autoencoder (MoE-Sim-VAE), a novel generative clustering model. The model can learn multi-modal distributions of high-dimensional data and use these to generate realistic data with high efficacy and efficiency. MoE-Sim-VAE is based on a Variational Autoencoder (VAE), where the decoder consists of a Mixture-of-Experts (MoE) architecture. This specific architecture allows for various modes of the data to be automatically learned by means of the experts. Additionally, we encourage the latent representation of our model to follow a Gaussian mixture distribution and to accurately represent the similarities between the data points. We assess the performance of our model on synthetic data, the MNIST benchmark data set, and a challenging real-world task of defining cell subpopulations from mass cytometry (CyTOF) measurements on hundreds of different datasets. MoE-Sim-VAE exhibits superior clustering performance on all these tasks in comparison to the baselines and we show that the MoE architecture in the decoder reduces the computational cost of sampling specific data modes with high fidelity.

Under review as a conference paper at ICLR 2020
MIXTURE-OF-EXPERTS VARIATIONAL AUTOEN- CODER FOR CLUSTERING AND GENERATING FROM SIMILARITY-BASED REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Clustering high-da sigh-dimensionl data, such as images or biological measurements, is long-standing problem and has been studied extensively. Recently, Deep Cluster- ing gained popularity due to the non-linearity of neural networks, which allows for flexibility in fitting the specific peculiarities of complex data. Here we introduce the Mixture-of-Expert Similarity Variational Autoencoder novel generative clustering model. The model can learn multi-modal distribu- tions of high-dimensional data and use these generate realistic data with high efficacy and efficiency. MoE-Sim- VAB is based on a Variational Autoencoder (MoE) architecture.
SEEE-Sim-VABB
(VAB), where the decoder consists of Mixture-of-Expere This specific architecture allows for various modes of the data to be automatically learned by means of the experts. Additionally, we encourage the lower dimen- sional latent representation of our model to follow Gaussian mixture distribution and accurately represent the similarities between the data points. We assess the performance of our model on synthetic data, the MNIST benchmark data set, and a challenging real-world task of defining cell subpopulations from mass cytom- etry (CyTOF) measurements exhibits superior clustering performance on all these tasks in comparison to the baselines and we show that the MoE architecture in the decoder reduces the com- putational cost of sampling specific data modes with high fidelity.
hundreds of different datasets. MoB-Sim-VAL VAE
INTRODUCTION
Clustering has been studied extensively (Aljalbout al. 2018 at. 2018) in machine learning- Recently, many Deep Clustering approaches were proposed, which modified (Variational) Autoen- coder ((V)AE) architectures (Min et al., 2018 ZZhang et al. 2017] or with varying regularization
of the latent representation (Dizaji al. 2019)
2017 Jiang al. 2017 Yang al. 2017 Fortuin al.,
Reconstruction error usually drives the definition of the latent representation learned from an AE or VAE. The representation for AE models unconstrained and typically places data objects close to each other according to an implicit similarity measure that also yields favorable reconstruction error. In contrast, VAE models regularize the latent representation such that the represented inputs follow certain variational distribution. This construction enables sampling from the latent representation and data generation via the decoder of a VAE. Typically, the variational distribution is assumed standard Gaussian. but for example, Jiang et al. (2017) introduced mixture of Gaussian variational distribution for clustering purposes.
key component of clustering approaches the choice of similarity metric for the considered data objects which we try to group (Irani et al. 2016). Such similarity metrics are either defined priori or learned from the data to specifically solve classification tasks via a Siamese network architecture (Chopra et al. 2005). Dimensionality reduction approaches, such UMAP (MeInnes et al. 2018) or t-SNE (van der Maaten Hinton 2008), allow to specify a similarity metric for projection and thereby define the data separation in the inferred latent representation.
In this work, introduce the offture-of-Experte Similarity Variational Autoencoder (MoE-Sim- VAE), a new deep architecture that performs similarity- based representation learning, clustering of the data and generation of data from each specific data mode. Due to combined loss function,

---

Under review as a conference paper at ICLR 2020
Data
Encoder
Latent representation
D) MoE Decoder
Data generation
C2
G Grd(pp))
t Gi + ox. 1nyC
F) Data similarity
G) Clustering
=S PPT
PIA PPanPraPPPal
Figure I: Overview of the proposed model MoE-Sim-VALe Data (in panel A) gets encoded via encoder network (B) into latent representation (C) which is trained Gaussians. Via clustering network (G), which is trained to reconstruct user-defined similarity matrix (F), the encoded samples get assigned to the data mode-specific decoder subnetworks (which we call experts) in the MoE Decoder (D). The experts reconstruct the original input data and can be used for data generation when sampling from the variational distribution (E).
mixture of standard
be jointly optimized. We assess the scope of the model on synthetic data and we present superior clustering performance on MNIST. Moreover, in an ablation study, we show the efficiency and precision of MoE-Sim- VAE for data generation purposes comparison the most related state-of-the-ats method (Jiang et al. 2017). Finally, we show an application of MoE-Sim VAE on real-world clustering problem in biology on multiple datasets.
Our main contributions are
Develop novel autoencoder architecture for
similarity- based representation learning
unsupervised clustering
accurate and efficient data generation
Embed the Mixture-of-ENae -f--eeert architecture into Variational Autoencoder setup to train separate generator for each data mode
Show superior clustering performance of the model benchmark dataset and real-world biological data
MIXTURE-OF-EXPERR SIMILARITY VARIATIONAL AUTOENCODER
Here we introduce the Mixture-of of-Exxerts Similarity Variational Autoeneoder (MoE-Sim LAE, Fig-
ure Dp. The model is based on the Variational Autocncoder (Kingma & Welling 2014). While the encoder network is shared across all data points, the decoder of the MoE-Sim VAE consists of number of K different subnetworks, forming Mfxture-of-Expee -x-exs architecture (Shazeer et al.) 2017]. Each subnetworks constitutes generator for a specific data mode and learned from the data.
The variational distribution over the latent representation is defined to be a mixture of multivari- ate Gaussians, first introduced by Diang et al. (2017). our model, we aim learn the mixture components in the latent representation to be standard Gaussians
K
(1)
k=
2

---

Under review as a conference paper at ICLR 2020
where wy mixture coefficients, μ4d the means for each mixture component, I is the identity matrix and K is the number mixture components. The dimension of the latent representation needs to be defined to suit the demands of Gaussian mixtures which have limitations in higher dimensions (Bishop] 1995). Similar to optimizing an Evidence Lower Bound (ELBO), we penalize the latent representation via the reconstruction loss of the data Lreconst and by using the Kullback- Leibler (KL) divergence for multivariate Gaussians (Jiang et 2017) on the latent representation
LKL = DDx(No = - (1r((i Eo) + (M1 - -14)" ( (m - 14) + In
(2)
Σnl
where k is a constant, No ~ N(Ho Eg = I). and I is the identity matrix. Further, W1 ~ ((1, Σ diag(o,)), where Jj for j I...., D. for number of dimensions D, is estimated from the samples of the latent representation. Finally, we assume po μr resulting in the following simplified objective
E
LKL = DRI(No.NN) -
(tr k In Eg
(3)
penalize exclusively the covariance of each cluster. remains define the reconstruction loss Freconst, where we choose a Binary Cross-Entropy
N
D
Creconat Σ
Σ
aidd log(x;,d (r
(4)
between the original data x (scaled between and 1) and the reconstructed data reconstt where iterates the batch size N and dl the dimensions of the data D. Finally the loss for the VAE part is defined by
AV Creconst TILKL
(5)
with weighting coefficient Ti which can be optimized as a hyperparameter.
SIMILARITY CLUSTERING AND GATING OF LATENT REPRESENTATION
Training of data mode-specific generator expert requires samples from the same data mode. This necessitates to solve a clustering problem, that is, mapping the data via the latent representation into K clusters, each corresponding to of the K generator experts. We solve this clustering problem via a clustering network. also referred to as gating network for MoE models. takes as input the latent representation z; of sample i and outputs probabilities Pit [0, for clustering sample into cluster k. According to this cluster assignment, sample i is then gated to expert h = argmax, Pix for each sample We further define the cluster centers μ4 for (l..... .. similar in the Expectation Maximization (EM) algorithm for Gaussian Mixture models (Bishop 2006) as
14
(6)
NO
where N₂ is the absolute number of data points assigned to cluster & based on highest probability pa for each sample 1,...., N. The Gaussian mixture distributed latent representation (via KL loss in Equation B) is motivation for the empirical computation of the cluster means and further, similar as in the EM algorithm, it allows iterative optimization the means of the Gaussians. We train the clustering network reconstruct data-driven similarity matrix S, using the Binary Cross-Entropy
L Similarity = -> St (PPP
to minimize the error in PPT RS S. with p:= (put () where N is the number of samples (e.g., batch size). Intuitively, PPT approximates the similarity matrix since values in PP" only close to when similar data objects assigned to the same cluster, similar to the entries in the adjacency similarity matrix S. This similarity matrix is derived an unsupervised way in our experiments (e.g. UMAP projection of the data and n-neabost-neighhorss distance
3

---

Under review as a conference paper at ICLR 2020
thresholding to define the adjacency matrix for the batch), but can also be used include weakly- supervised information (e.g., knowledge about diseased vs. non-diseased patients). If labels are available, the model could even be used derive latent representation with supervision. The similarity feature MoF-SSim-AAN thus allows to include prior knowledge about the best similarity measure the data.
Moreover, we apply the DEPICT loss from Dizaji et al. (017), to improve the robustness of the clustering. For the DEPICT loss, we additionally propagate a noisy probability fix through the clustering network using dropout after each layer. The goal is to predict the same cluster for both, the noisy Pix and the clean probability Pix (without applying dropout). Dizaji et al. (0017) derived as objective function standard cross-entropy loss
DDEPICT
9sk log(Pit)
(8)
whereby que is computed via the auxiliary function
Pik/ Pit/(E. PvK)
Pik: = EN Pik / / )
(9)
where we refer to Dizaji et al. (2017) for exact derivation. The DEPICT loss encourages the model to learn invariant features from the latent representation for clustering with respect to noise (Dizaji et al. 2017). Looking at it from different perspective, the loss helps to define a latent representation which has those invariant features to be able to reconstruct the similarity and therefore the clustering correctly. The complete clustering loss function LChustering is then defined by
CClustering = Simillarity + TELDEPICT
(10)
with a mixture coefficient T2 which can be optimized as a hyperparameter.
MoE-SIMI VAE LOSS FUNCTION
Finally, the MoE-Sim- -AE model loss is defined by
L Sim -VA
LVAE
LCtustering
(11)
Creconse tT EXL
Cyimiturity t LoCDEIITT
which consists of the two main loss functions AVAB acting as a regularization for the latent repre- sentation, and LClustering. which helps to learn the mixture components based on an a priori defined data similarity. The model objective function LM Sim V VAE can then be optimized end-to-end to train all parts of the model.
3 RELATED WORK
(V)AEs have been extensively used for clustering (ie al. 2016 Dizaji al. 2017 Li et al.
2017
Yang et al.) 2017 Saito & Tan 2017: Chen et al. 2017 Aljalbout et al. 2018 Fortuin et al. 2019)
The most re ratated approaches
MoE-Sim VAE are Jiang et al. (2017) and [Zhang et al. (2017).
Jiang et al. (2077) introduced the VaDE model, comprising mixture of Gaussians as underlying dis- tribution in the latent representation of a Variational Autoencoder. Optimizing the Evidence Lower Bound (EI.O) of the log-likelihood of the data can rewritten to optimize the reconstruction loss of the data and KI. divergence between the variational posterior and the mixture of Gaussians prior. Jiang al. (2017) motivate the use of to two separate networks for reconstruction and the gener- ation process of the model. Further, to effectively generate images from specific data mode and to increase image quality, the sampled points have to surpass a certain posterior threshold and are otherwise rejected. This leads an increased computational effort. The MoE Decoder of our model, which is used for both reconstruction and generation, does not need such a threshold, as we discuss in more detail in Section 4.2.1
Zhang et al. (207)) have introduced mixture of autoencoders (MIXAE) model. The latent rep- resentation of the MIXAE defined as the concatenation of the latent representation vectors of each single autoencoder in the model. Based this concatenated latent representation, Mixture
4

---

Under review as a conference paper at ICLR 2020
Table 1: Performance comparison of our method MoE-Sim- SiEE-ii--VAE with several published methods. The Table is mainly extracted from aojjaboooou et al. (2018) and complemented with results of interest. (" metric not reported)
METHOD
NMI
ACC
JULE,
Yang et al.
(016bb)
0.915
10
CCNN.
Hsu
2017
0.876
DEC.
Xie et al.
2016
0.8
0.843
DBC,
al.
0.917
0.964
DEPICT.
Dizaji et al. (20177
0.916
0.965
DCN. Yang al. (2017)
0.81
0.83
Neural Clustering, Saito & Tan
2017
0.966
UMMC, Chen al. (201)
0.864
VaDE, Jiang et al.
2017
0.945
TAGnet, Wang et al. (2016
0.651
0.692
IMSAT, Hu et al. (2017)
0.984
Aljalbout et al. 2018
0.923
0.961
MIXAE, /Zhang al. (2017)
0.945
Spectral clustering, |Shaham et al. (2018)
0.754
0.717
SpectralNet (input space, Euclidean dist.), Shaham et al.
2018
0.607
0.622:10.008
SpectralNet (input space, Siamese dist.). Shaham et al.
2018)
0.884+0.02
0.826+0003
SpectralNet (code space, Euclidean dist.), Shaham et al.
(0188)
0.814+1.008
0.800#00000
SpectralNet (code space, Siamese dist.), Shaham al. MoE-Sim-Val (proposed)
2018
0.924
0.971
0.935
0.975
Assignment Network predicts probabilities which are used in the Mixture Aggregation to form the output of the generator network. Each AE model learns the manifold of a specific cluster, similarly to our MoE Decoder. However, MIXAE does not optimize variational distribution, such that gen- eration of data from distribution over the latent representation is not possible, in contrast to the MoE-Sim- VAE (Figure 2).
4 EXPERIMENTS
We evaluate the MoB-Sim: SAm using synthetic data and the MNIST data set of handwritten digits (LeCun et al. 1998) for clustering and data generation. Furthermone, we performed an ablation study demonstrate the importance of the MoE Decoder. Finally, we present experiments
real- worldworld application of defining cellular subpopulations from mass cytometry measurements (Bandura et al. 2009) of multiple publicly available datasets (Weber & Robinson 2016 Bodenmiller al. 2012) Model implementation details are reported in the appendix in section A.I
We found that our model achieves superior clustering performance compared to other models on synthetic, MNIST and real-world datasets. Moreover, we show that MoE-Sim-VAL more effec- tively and efficiently generate data from specific modes in comparison to other methods.
4.1 EVALUATION MOE-SIM-VAE ON SYNTHETIC DATA
We evaluated our model using data sampled from 100-dimensional multivariate Gaussian with equal mixture weights for each component. We tested two aspects of our model: Firstly, we evalu- ated up to how many clusters our model can fit well. Therefore, we sampled data from distributions with up hundred mixture components. For this experiment, number of clusters in the data for both methods, MoE-Sim-VAE and GMMs. Secondly, we tested if our model is able to identify the true number of clusters in the data. The similarity matrix S was defined as an adjacency matrix over the data items. Adjacency indicators were based on projecting the data via dimensionality reduction with UMAP (McInnes al., 2018) and selecting neighbors according to a distance threshold. Details on model parameters can be found in Section A.I.I
assume knowledge of the true
MoE-Sim- VAE performs better or comparable to the baseline for the number of clusters of up to 40 (Figure Aaa. The model predicts with close perfect F-measure until reaching true number
5

---

Under review as a conference paper at ICLR 2020
D) Clustering / Gating probabilities
A] Variational distribution
Σ an
Probabiitty
BJ Clustering network
A2
Po
C) MoE Decoder
8 8
7
5 S 5
3
6 6
9 9
9
0
O
3
6
6
9
9
Figure 2: Generation of MNIST digit images. Data points from the latent representation were sampled from the variational distribution (A) which is learned to be a mixture of standard Gaussians and then clustered and gated (B) to the tata-mode-specifff experts the MoE Decoder (C). (D) All samples from the variational distribution were correctly classified and therefore also correctly gated.
of clusters of 30. Within the range of true number of clusters from 30 to 40, the model performs comparable GMMs. Further, MoE-Sim- VAE learns the true number of clusters on its own (Figure Alb.. For up to 23 components in the data, MoE-Sim-VAF learns the true number of clusters even when defining model with K = 40 experts in the MoE Decoder. This suggests that the model is robust to misspecification regarding the number of experts.
4.2
UNSUPERVISED CLUSTERING, EMBEDDING AND DATA GENERATION OF MNIST
We trained a MoE- SLE-SSi--aee VAE model on images from MNIST. We compared our model against mul- tiple models which were recently reviewed in Aljaabout et al. 2018)), and specifically against VaDE (Jian et al. 2017) which shares similar properties with MoE VAE (see Sec
We compare the models with the Normalized Mutual Information (NMI) criterion but also classifi- cation accuracy (ACC) [Table The MoE-Sim- VAE outperforms the other methods w.r.t. cluster- ing performance when comparing NMI and achieves the second-best -estt result when comparing ACC. Note that used the number of experts k 10 in our model fit the existing number of digits in MNIST. Regarding the similarity measure, we decided to use as similarity a UMAP projection batch.
(Mcinnes et al. 2018) MNIST and then apply neighrstt of each sample More details on the model are reported in Section A.1.2 In an ablation study we show the im- portance of the similarity matrix to create clear separation of the different digits the latent representation (Figure A4
In addition to the clustering network, we can make use of the latent representation for image genera- tion purposes. The latent representation is trained as a mixture standard Gaussians. The means of these Gaussians are the centers of the clusters trained via the clustering network. Therefore, the vari- ational distribution can be sampled from and gated to the cluster-specific expert in the MoE-decoder. The expert then generates new data points for the specific data mode. Results and the schematic are displayed in Figure 2 and in more detail and with greater sample size in the Appendix in Figure
4.2.1 WHY DOES A MOE DECODER ACTUALLY MATTER?
In an ablation study, we compare the two models MoE-Sim- VAE and VaDE (Jing al. 2017) on generating MNIST images with the request for specific digit. The goal is to show that MoE decoder, as proposed in our model, beneficial. We focus our comparison to VaDE since this model, the MoE-Sim- VAE, resorts images by means of a single decoder network instead of a Mixture-of -ffturre--f-Exeetttt decoder network. The rationale for our design choice is to ensure that smaller mub-networks learn reproduce and generate specific modes of the data, in this case of specific MNIST digits.
mixture of Gaussian latent representation but differs in generating
To show that both models' latent representations are separating the different clusters well, we com- puted the Maximum Mean Discrepancy (MMD), defined in Section A.1.2 The MMD be inter-

---

Under review as a conference paper at ICLR 2020
preted as a distance between distributions computed based samples drawn from these distribu- tions. The heatmaps of the MMDs for VaDE and MoE- Moo--Sim-ee VAE as well as an UMAP projection of the latent representation colored with the mixture component confirm visually the separation of the clusters in the latent representations of both models (Fig. A3). As a result, we can conclude that both latent representations can separate the clusters of respective digits well, such that the decoder gets well-defined samples to generate the requested digit. Therefore, the main difference of generating specific digits arises in the decoder/generatoor networks.
We evaluated the importance of the MoB-Decoder to (1) accurately generate requested digits and (2) be efficient in generating requested digits. Specifically, we sampled 10, 000 points from each mixture component in the latent representation, generated images, and used the model's internal clustering to assign a probability to which digits were generated. To generate correct and high- quality images with VaDE, the posterior of the latent representation needs be evaluated for each sample. This was done for the different thresholds & E [0.0, 0.1, 0.2, 0.9,, 0.999]. The default threshold Jiang et al. (2017) used was & 0.999. Instead of thresholding the latent representation, ran the generation process for MoE-Sim-VAE for each threshold with the same settings. To gen- crate images from VaDE we used the Python implementatior and model weights publicly available from Jiang et al. (2017).
As result of this analysis report confusion matrix for MoE-Sim-VAL in Figure A6 the confu- sion matrices for each threshold for VaDE in Figure A7 the accuracy of generating a requested digit and the number of runs required in Figure AS In summary, one can that the MoE-Sime Sim--Ai generates digits more accurately with fewer resources required. This can especially be seen when comparing the number of iterations required to fulfill the default posterior threshold of 0.999. VaDE needs nearly 2 million iterations to find samples that fulfill the aforementioned threshold criterion whereas the MoE-Sim- Sim-VVE only requires 10, 000 for a comparable sample accuracy. In comparison the mean accuracy over all thresholds for MoE-Sim- VAE is 0.970, whereas VaDE reaches on aver- age 0.944. VaDE reaches a maximum accuracy of 0.995, which costs the aforementioned million iterations for generating 100, 000 images, whereas MoE-Sim- VAE reaches a maximum accuracy of 0.971 with 100, 000 runs, without accounting for a systematic generating/custering error (confusing 5 and 8) of MoE-Sim-VAL VAE which can be seen in the confusion matrix in Figure A6
4.3 LEARNING CELL TYPE COMPOSITION IN PERIPHERAL BLOOD MONONUCLEAR CELLS USING CYTOF MEASUREMENTS
In the following, we want to show representation learning performance on a real-world problem in biology. Specifically, we focus on cell type definition from single-cell measurements. Cytometry by time-of- of-fightt mass spectrometry (CyTOF) (Bandura et al. 2009) is a state-of-the-er of-the-art technique allowing measurement of up to 1, 000 cells per second and in parallel over 40 protein markers of the cells (Kay et al. 2013). Defining biologically relevant cell subpopulations by clustering this data is a common learning task (Aghaeepour et al., 2013) Weber & Robinson (016)).
Many methods have been developed to tackle the problem introduced above and were compared on four publicly available datasets in Weber & Robinson (2016). The best out of 18 methods were FlowSOM (Gassen et al., 2015), PhenoGiraph (Levine et al. ] (015] and X-shift (Samusik et al. 2016). These are based on k-nearest-neighbors heuristics, either defined from spanning graph or from estimating the data density. In contrast to these methods, MoB-Sim- VAB can map new cells into the latent representation, assign probabilities for cell types and infer an interpretable latent representation allowing intuitive downstream analysis by domain experts.
We applied MoE-Sim- VAE to the same datasets as in Weber & Robinson (2016) and achieve superior results in classification using the F-measure (Equation 12) in three out of four datasets. Similarly as in Weber & Robinson| (2016) we trained MoE-Sim- VAË 30 times and report in Table (adoop(dddeeed
from Weher & Robinson| (2016) the means across all runs. The reproducibility of model for each dataset can be seen in Figure A8
Further, we trained a MoE-Sim-VAE model on 268 datasets from Bodenmiller et al. (012) (more details on the data in A.1.)) and achieve superior classification results of cell subpopulations in the data when comparing to state-of-the-art methods in this field (PhenoGraph, X-Shift, FlowSOM).
01D/V
7

---

Under review as a conference paper at ICLR 2020
Table 2: Comparison of MoB-Sim-Van performance competitor methods in defining cell type composition in CyTOF measurements. The results in the table are extracted from the review paper of & Robinson (2016), where 18 methods are compared on four different datasets. Our model outperforms the baselines on four out of five data sets.
Method
Levine 32dim
Levine 13dim Samusik
Samusik all
ACCENSE
0.494
0.358
0.517
0.502
ClusterX
0.682
0.474
0.571
0.603
DensVM
0.66
0,448
0.239
0,496
FLOCK
0.727
0.379
0.608
0.631
flowClust
NA
0.416
0.612
0.61
flow Means
0.769
0.518
0.625
0.653
flowMerge
NA
0.247
0,452
0.341
flowPeaks
0.237
0.215
0.058
0.323
FlowSOM
0.78
0.495
0.707
0.702
FlowSOM prr
0,502
0,422
0.583
0.528
immunoClust
0,413
0.308
0.552
0.523
kmeans
0.42
0.435
0.65
0.59
PhenoGraph
0.563
0.468
0.671
0.653
Relusterpp
0.605
0.465
0.637
0.613
SamSPECTRAL.
0.512
0,253
0.263
0.138
SPADE
NA
0.127
0.169
0.13
SWIFT
0.177
0.179
0.202
0,208
Xshift
0.691
0.47
0.679
0.657
MoE-Sim- VAË (proposed)
0.70
0.68
0.76
0.74
Exact results can be seen in Table A or visualized in Figure More details on the MoE-Sim-VAD setting used for all results on CyTOF data are reported the appendix (Section A.1.3)
CONCLUSION
Our MoE-Sim- Sim-VVE model can infer similarity-basee representations, perform clustering tasks, and efficiently as well as accurately generate high-dimensional data. The training of the model is per- formed by optimizing a joint objective function consisting of data reconstruction, clustering, and KL loss, where the latter regularizes the latent representation. On synthetic data, we have shown the strengths and limitations of the model. On the benchmark dataset of MNIST, we presented superior clustering performance and the efficiency and accuracy of MoE-Sim-VAs -ABE in generating high-dimensional data. On the biological real-world task of defining cell subpopulations complex single-cell data, show superior clustering performances compared state-of of-t----thhheeeeee-- art methods over 270 datasets and therefore demonstrate MoE-Sim-- VAE's real-world usefulness.
Future work might include to add adversarial training to the MoE decoder, which could improve image generation create even more realistic images. Also, specific applications might benefit from replacing the Gaussian with a different mixture model. So far the MoE-Sim-Vas. VAE's similarity measure has be defined by the user. Relaxing this requirement and allowing for learning useful similarity measure automatically for inferring latent representations will be an interesting extension explore. This could be useful in weakly-supervised setting, which often occurs for example in clinical data consisting of healthy and diseased patients. Minor details between a healthy and diseased patient might make huge difference and could be learned from the data using neural networks.
REFERENCES
N. Aghaeepour, G. Finak, FlowCAP Consortium, DREAM Consortium, H. Hoos, TR. Mosmann, R. Brinkman, R. Gottardo, and Rh. Scheuermann. Critical assessment automated flow cytom- etry data analysis techniques. Nature Methods, 2013.
8

---

Under review as a conference paper at ICLR 2020
1.0
0.8
F-measure
0.6
PhenoG aaph
0.2
FlowSOM
MoE-Sim-VAE
AKTI
Crassin
Dasabinib
GDC-0941
Go89
H-99
IKK]
Imatinbb
Jakli
Jak2i
Jakai
PP2
Rapernycn
50202
SP6
Soratentb
Sesurose rine
Strepponight
sunitinib
Syki
Uo120
vx680
Inhibitor
Figure 3: Comparison of MoE-Sim-VAD the most popular competitor methods on defining cell types in peripheral blood mononuelear cell data via CyTOF measurements, On the x-axis differ- inhibitor treatments are listed whereas the y-axis reports the respective F-measure, defined in Equation [2] performance measure of the methods. Each violin plot represents a run a dif- ferent inhibitor with multiple wells, whereas the line connects the means of the performance on the specific inhibitor.
Elie Aljalbout, Vladimir Golkov, Yawar Siddiqui, Maximilian Strobel, and Daniel Cremers. Clus- tering with deep learning: Taxonomy and new methods. arXiv, 2018.
DR. Bandura, VI. Baranov, OI. Ornatsky, A. Antonov, R. Kinach, X. Lou, S. Pavlov, S. Vorobiev, JE. Dick, and SD. Tanner. Mass eytometry: Technique for real time single cell multitarget im- munoassay based on inductively coupled plasma time-of-flight mass spectrometry. Analytical Chemistry, 2009.
Christopher M. Bishop. Neural Networks for Pattern Recognition. CLARENDON PRESS, 1995.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
Bernd Bodenmiller, Eli R. Zunder, Rachel Finck, Tiffany J. Chen, Erica S. Savig, Robert V. Brug- gner, Erin Simonds, Sean C. Bendall, Peter O. Krutzik Karen Sachs, and Garry P. Nolan. Multiplexed mass cytometry profiling of cellular states perturbed by small-molecule regulators. Nature Biotechnology, 2012.
D. Chen, J. Lv, and Z. Yi. Unsupervised multi-manifold clustering by learning deep representation. Workshops at the AAAI Conference on Artificial Intelligence, 2017.
S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity metric discriminatively, with application to face verification. IEEE, 2005.
Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. arXiv, 2017.
Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. Som- vae: Interpretable discrete representation learning on time series. Conference paper ICLR 2019, 2019.
Sofie Van Gassen, Britt Callebaut, Mary Van Helden, Bart Lambrecht, Piet Demeester, Tom Dhaene, and Yvan Sacys. Flowsom: Using self organizgnn maps for visualization and interpreta- tion of cytometry data. Cytometry Part A, 2015.
Arthur Gretton, Karsten Borgwardt, Malte J. Rasch, Bernhard Scholkopf, and Alexander Smola. A kernel method for the two-sample problem. arXiv, 2008.

---

Under review as a conference paper at ICLR 2020
C.-C. Hsu and C.-W. Lin. Cnn-based joint clustering and representation learning with feature drift compensation for large-scale image data. arXiv, 2017.
W. Hu, T. Miyato, S. Tokui, E. Matsumoto, and M. Sugiyama. Learning discrete representations via information maximizing self augmented training. arXiv, 2017.
Jasmine Irani, Nitin Pise, and Madhura Phatak. Clustering techniques and the similarity measures used in clustering: survey. International Journal of Computer Applications, 2016.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach clustering. arXiv, 2017.
Alexander W. Kay, Dara M. Strauss-Albee, and Catherine A. Blish. Application of mass cytome- try (cytof) for functional and phenotypic analysis of natural killer cells. Methods in Molecular Biology, 2013.
D. P. Kingma and M. Welling. Auto-encoding variational bayes. International Conference Learning Representations (ICLR), 2014.
Yann LeCun, Leon Botto, Yoshua Bengi, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings ofthe IEEE, 1998.
Jacob H. Levine. Erin F. Simonds, Sean C. Bendall, Kara L. Davis, El ad D. Amir, Michelle D. Tadmor, Oren Litvin, Harris G. Fienberg, Astraca Jager, Eli R. Zunder, Rachel Finck, Amanda L. Gedman, Ina Radtke, James R. Downing, Dana Peer, and Garry P. Nolan. Data-driven phenotypic dissection of aml reveals progenitor- cells that correlate with prognosis. Cell, 2015.
F. Li, H. Qiao, B. Zhang, and X. Xi. Discriminatively boosted image clustering with fully convolu- tional autoencoders. arXiv, 2017.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv, 2018
Erxue Min, Xifeng Guo, Qiang Liu, Gen Zhang, Jianjing Cui, and Jun Long. A survey of clustering with deep learning: From the perspective of network architecture. IEEE, 2018.
Peng Qiu, Erin Simonds, Sean C. Bendall, Kenneth D. Gibbs Jr., Robert V. Bruggner, Michael D. Linderman, Karen Sachs, Garry P. Nolan, and Sylvia K. Plevritis. Extracting cellular hierarchy from nigh-dimensional cytometry data with spade. Nature Biotechnology. 2011.
S. Saito and R. T. Tan. Neural clustering: Concatenating layers for better projections. Workshop track at ICLR 2017, 2017.
Nikolay Samusik, Zinaida Good, Matthew H. Spitzer, Kara Davis, and Garry P. Nolan. Automated mapping of phenotype space with single-cell data. Nature Methods, 2016.
Uri Shaham, Kelly Stanton, Henry Li, Boaz Nadler, Ronen Basri, and Yuval Kluger. Spectralnet: Spectral clustering using deep neural networks. Published conference paper at ICLR 2018, 2018.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le1, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gal gaged mixture-of-expprts layers. arXiv, 2017.
Dougal J. Sutherland, Hsiao-Yu Tung. Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum mean discrepancy. arXiv, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2008
Z. Wang, S. Chang, J. Zhou, M. Wang, and T. S. Huang. Learning a task specifcc deep architecture for clustering. Proceedings the SIAM International Conference Data Mining (ICDM), 2016.
10

---

Under review as a conference paper at ICLR 2020
Lukas M. Weber and Mark D. Robinson. Comparison of clustering methods for highdimensional singlecell flow and mass cytometry data. Cytometry Part A, 2016.
J. Xie, R. Girshick, and A. Farhadi, Unsupervised deep embedding for clustering analysis. Interna- tional Conference on Machine Learning (ICML), 2016.
Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. Towards k-means-friendly spaces: Simultaneous deep learning and clustering. arXiv, 2017.
J. Yang, D. Parikh, and D.Batra. Joint unsupervised learning of deep representations and image clus- ters. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016b.
Dejiao Zhang, Yifan Sun, Brian Briksson, and Laura Balzano. Deep unsupervised clustering using mixture of autoencoders. arXiv, 2017.
A APPENDIX
A.I
EXPERIMENTAL DETAILS
In the following sections we provide more details on model implementations, metrics used and additional result figures for the experiments described in the main text.
A.1.1 EVALUATION OF MOE-SIM-VAE ON SYNTHETIC DATA
Model and training details:
number of experts: .2.
batch size: 512
code size:
Number of iterations: 5000
activation function; elu
loss coefficient data reconstruction: 0.487
loss coefficient clustering: 0.487
loss coefficient mixture of Gaussian: 0.024
learning rate: 0.001
batch normalization
dropout rate: 0.5
distance threshold (perplexity parameter): 2
depth clustering network: 5
internal size clustering network: 100
trainable parameters: depending on number experts
We compare results based on F-measure (Aghaeepour et al. 2013), which is defined as follows
lei
F(C,
Σ
max KyEK
( F(c, k,)
(12)
N
GEC
where N is the number of samples C(er, C2 ..., Cn) and K (ki, k2,..., km are the cluster result and the reference cluster, respectively. Further F(c,,k,) is the harmonic mean of precision and recall according to
2Pr(c,, Re(c,, ky) F(c, k2) Pr(c,, kú) Re(ci, kg)
(13)
whereby Pr(c;; kj) is the precision and Re(ce, kj) is the recall. Results shown in Table 2
11

---

Under review as a conference paper at ICLR 2020
F-measure
MoE-Sim-AE
GMM
Number of GMM components/experts
Number of GMM components
F-measure
92
SAR222 9A 2a..a22maaa8a.a 11122321133003333 Number of Experts
(b)
Figure A1: Testing MoE-Sim- VAE on data sampled from Gaussian mixture model with random sampled parameters. Figure Ala Testing with exact numbers of experts. When comparing to classification results with GMMs one can see that our model achievs better results until around 30 mixture components and is still compatitive until 40 mixture components. With more then 40 mixture components ourr MoE-Sim- VAE is not able anymore to compete with a GMM. Figure Ab lb Testing for specific number of synthetic mixture components and iterating number of experts. Until a number of GMM components 23 MoE-Sim- VAE very precise in learning the real number of clusters even when allowing the model have 40 experts.
A.1.2 UNSUPERVISED CLUSTERING, EMBEDDING AND DATA GENERATION OF MNIST
Model and training details:
number of experts: 10
batch size: 128
code size: 68
Number of iterations: 20000
activation function; elu
loss coefficient data reconstruction: 0.487
12

---

Under review as a conference paper at ICLR 2020
loss coefficient clustering: 0.487
loss coefficient mixture of Gaussian: 0.024
learning rate: 0.0001
batch normalization
dropout rate: 0.5
k from kNN (perplexity parameter): 10
depth clustering network: 3
internal size clustering network: 200
trainable parameters: 1619446
One estimator of the Maximum Mean Discrepancy (MMD)
(Gretton et al. 2008) is defined as
MMD (X,Y)
kXx, Xx) +
k(Y, yg) 2
kX,, Y2)
(14)
J
where X = { 3m) P.Y (91, 3m Q are samples from two distributions (e.g. samples from two different clusters the latent representation, for MNIST two different digits) and k is kernel function, where use the popular RBF kernel. Based on that estimator Sutherland et al. (2019) introduced the hypothesis test
HO: T0:P=C P H1:
(15)
(16)
using the statistic MMMD (X,Y), The distribution for P and Q is not required to be known. Sutherland et al. 2019) used MMD and this test to train Generative Adversarial Network (GAN) 2 and also evaluate the generative performance of the model. In this work we use MMD (X, Y) to test samples of different clusters of the latent representation are similar, or in other words the distance of the distributions. We used the Python poplemmmnatiio"ffrrm Sutherland al. 2019).
A.1.3 LEARNING CELL. TYPE COMPOSITION PERIPHERAL. BLOOD MONONUCLEAR CELLS USING CYTOF MFASUREMENTS
Model and training details for all experiments on CyTOF data:
number of experts: 25 WWber & Robinson 2016). (Bodenmiller et al.
2012
batch size: 128
code size: 9
Number of iterations: 30000 (Weber & Robinson
2016 20000 (Bodenmiller et al. 2012)
activation function: relu
loss coefficient data reconstruction:
loss coefficient clustering
loss coefficient mixture of Gaussian: 0
learning rate: 0.001 (Weber & Robinson 2016), 0.005 (Bodenmiller et al. 2012
batch normalization
dropout rate: 0.5
distance threshold (perplexity parameter):
distance metric: correlation
depth clustering network: 5
internal size clustering network: 9
https:://g thub. sutheri htt.://m.............../.e/ettte mmd__ mnd_test..p
13

---

Under review as a conference paper at ICLR 2020
MoE Decoder
Expert
expert
Expert
Expert 2
Eapert
10.0
Expert
7.5
Expert
Expert
7
225
Expert 5
1102 10
-5.0
Export
-7.5
Expert
55
10
2xport
Expert 7
O
O
Expert 8
O
O
O
O
Expert
Expert
Capert
Figure A2: More detailed overview of results and generated samples of MNIST images. The plot on the left side shows the latent representation where the red crosses are the cluster centers. Those can be used as a mean sample from a standard Gaussian for data generation via the MoE Decoder. The boxplots on the right show the clustering and gating result of each sample from the variational distribution.
trainable parameters: 37563 /Weber& Robinson 2016), 22228 (Bodenmiller et al. 2012)
Results are computed setting the loss coefficient for the KL loss B equal to zero, since we do not in- tend to generate any data, but rather give the chance the AE pick up the correct subpouplations.
14

---

Under review as a conference paper at ICLR 2020
10-
UNAPZ
-10
-15
-10
-5
UMAP1
(a)
240
UMAPZ
180
-10
-15
-35
UMAP1
(b)
Figure A3: Comparison of two sample MMD test (Sutherland al. 2019) on the distributions from the different mixture components in the latent representation. The heatmaps on the left side show the estimation the MMD which can be seen as the distance between pairs of distributions. The figures on the right side show the separation of the cluster in the latent representation based on dimensionality reduction via UMAP (McInnes et al.) 2018). Figure A3a shows the results for the clusters of VaDE at posterior threshold of 0.8 which is the first threshold which shows total separation of clusters. Figure A36 shows the separation of the clusters in latent space learned from MoB-Sim-VAR For both methods, all distributions belonging to clusters of different respective digits show a larger distance compared to the diagonal of matching distributions, such that we generate images from a well-separated latent representation for both methods and therefore the main difference comes from the decoders.
metric evaluate the models. For the data compared in Weber & Robinson (2016) we ran each model 30 times and report reproducability of our results in A8 The model trained on all data and validated the on with labels.
Also here we use the F-measure defined in Equation 12
each model on time on the each of the 268 datasets. Hereby we focused on the following surface markers: (D3(110:114)0 CD45(InI 15) CD4(Nd145)Dd, CD20(Sm147)Dd,, CD33(Nd148)dd, 1DI23(Eul51)Dd, 6D14(Gd160)Dd, IgM(Ybi71)Dd, HLA-DR(YbI74DDd, CD7(Yb17))dd. The subpopulations were originally de- fined via the SPADE algorithm QQu et al. 2011) which is visualization tool using Agglomerative
For the data from Bodenmiller et al. (2012)
15

---

Under review as a conference paper at ICLR 2020
15
UlAPP
60
-10
-5
0
10
UMAP1
(a)
UMAP2
120
600
UMAP1
(b)
Figure A4: Ablation study on the similarity matrix Both figures show the MMD statistic and UMAP (Mcinnes et al., 2018) projection of reconstructed MNIST digits computed on the latent representation. Figure A4a shows the results on MoE-Sim- VAE trained with the similarity matrix. The different digits separate well which can also be seen in the heatmap showing the MMD statistics between all digits. comparison, Figure A4b shows results of the MoE-Sima VAE model ignoring the similarity matrix setting the loss coefficient to zero. One observe that the MMD statistic, which can be seen as a similarity measure two distributions, is way lower compared to the model including the similarity matrix. Further, also the UMAP projection confirms less separation in the latent representation between the different digits.
hierarchical clustering and minimum spanning trees. The gating of the cells is done manually via coloring of the tree leaves. With MoE-Sim- VAE we try to reconstruct the defined manually defined subpopulations. [Bdeemiller et al. (2012) performed experiments on multiple well plates were dif- ferent inhibitors and their effect was tested. We selected for each well plate row A to test model on. decided for all methods to discard subpopulations which are smaller then 30 cells. As similarity measure for MoB-Sim-Vas we reduced the dimension of the data using UMAP (McInnes et al. 2018) using the Canberra distance
Pp
d(p, q)
(17)
Ppl
+ 19
16

---

Under review as a conference paper at ICLR 2020
1.0
Accuracy
0.6
0.1
0.2
MoE-Sim-VAE
VaDE
0.0
0.0
0.2
0.4
0.6
1.0
Posterior Threshold VaDE
2000000
MoE-Sim-Van -AE
VaDE
1750000
Number of runs
1500000
1250000
Number of Turns
1000000
00000
75C000
.000
20000
500000
62
344
66
Posterior Threshold VaDE
25C000
0.0
0.2
0.4
0.6
0.8
1.0
Posterior Threshold VaDE
Figure A5: Comparison of data generation process between Moe-Sim-VAE and VaDE (Jiang et al. 2017). Figure A5a shows the accuracy how certain a specific digit can be generated from the respective cluster in the latent representation whereas Figure A5b compares the number of runs until a sample from the latent representation satisfied the posterior criterion from VaDE. It needs to be mentioned that MoE-Sim-VAE does not require any thresholding such that we ran the data generation process multiple times with the same settings to compare with VaDE. In total 10000 samples generated for each digit.
where p = (p1, P.. and q = (91, T2..... 9. Cells were defined to similar in MoE-Sim- smaller then threshold. We trained and tested MoE-Sim- VAE on a splitted dataset with rations 0.8/0.2 and evaluated the performance the unseen test dataset. In comparison the compatitor methods trained and
VAE when the distance between the cells in the UMAP UMAA-projection
17

---

Under review as a conference paper at ICLR 2020
1000C
9
0
0
10
9989
0000
8000
7
3974
18
0
1
3999
0
0
6000
True label
7476
30
2395
10
4000
3
9A12
0
186
2
2
9968
0
32
0
9998
9999
7
S Predicted label
Figure A6: Confusion map for data generation using MoB-Sim-VAR Besides the systematic error of confusing digit and 8, which can also depend on the clustering network, the digit generation of our model performs very precise with high accuracy of generating the digit asked for. In comparison to VaDE (Jiang et al. 2017) our model does not need any threshold on samples from the latent representation which reduces the computational costs by far.
tested on all the data, which is an advantage in comparison to our model, but still MoE-Sim- VAE outpreforms the compatitors.
18

---

Under review as a conference paper at ICLR 2020
13
104
9299
7
454
B22
10
8:10
91
8000
9347
276
107
6
107
3
419
9331
0
6000
443
7326
61
8281
77
10
1537
-4000
R736
385
22
25
9327
131
54
23
141
206
9950
-2000
9516
0
32
14
224
195
0
17
0
7
Predicted label
(a)
9355
10
83
350
688
16
8796
58
8800
83
3
121
3
10
298
9369
0
13
0
6000
3
54
1
B426
60
14
47
1390
4400
9888
22
2500
116
33
310
171
2000
96112
49
12
166
136
0
Predictod label
(b)
14
312
604
11
8922
72
8000
25
5
9634
224
1
112
1
407
9377
6000
580
0
326
7923
54
9506
50
20
1294
4000
142
9040
299
506
92
45
80
9947
0
0
12
2000
128
Pradicted labal
(c)
Figure A7: Confusion maps for data generation using VaDE.
Figure A7a
Posterior threshold 0.0.
Figure 7b
Posterior threshold 0.1.
Figure A7c
Posterior threshold 0.2.
19

---

Under review as a conference paper at ICLR 2020
64
9512
9
248
541
15
9009
54
27
9646
224
85
103
3
381
8408
0
6000
279
81400
44
65
8663
68
1147
9700
254
18
20
0588
81
12
37
72
160
9936
9724
0
32
6
111
105
0
19
0
7
Predicted label
(d)
13
9595
11
50
215
412
12
9240
54
8000
68
3
131
1
12
352
9410
0
15
0
6000
2
57
0
9820
56
10
45
997
4400
0254
26
17
2596
72
32
74
164
2000
9736
43
3
97
103
16
0
Predictoo label
(e)
6
64
132
367
10
9376
40
8000
14
5
0
9708
204
102
1
343
6482
6000
476
238
8468
29
9897
42
12
948
44000
94111
181
279
31
60
9959
0
0
15
5
2000
9822
Pradicted labal
(f)
Figure A7: Confusion maps for data generation using VaDE. Posterior threshold 0.3.
Figure A7d
Figure A7e
Posterior threshold 0.4.
Figure A71
Posterior threshold 0.5.
20

---

Under review as a conference paper at ICLR 2020
59
9731
7
110
300
12
9491
35
8000
9733
192
52
2
89
0
258
B478
0
3
6000
225
8669
29
3
R009
51
837
-4000
153
19
14
9633
71
39
54
151
9948
905
38
6
36
51
0
14
0
%
7
Predicted label
9781
3
60
104
189
9
9395
34
8000
69
1
90
1
337
94880
13
6000
2
48
0
9175
50
10
4
29
679
4400
9642
12
24
9678
61
24
54
139
2000
9885
6
23
46
0
label
(h)
9
48
64
130
15
9705
28
8000
14
1
4
9787
146
52
0
309
9526
6000
300
132
9039
31
49
2
546
44000
9010
46
0
23
25
9958
0
0
2000
9912
10
Pradicted labal
Figure A7:
Confusion maps for data generation using VaDE. Posterior threshold 0.6.
Figure A7g
Figure 77h
Posterior threshold 0.7.
Figure A7i
Posterior threshold 0.8.
21

---

Under review as a conference paper at ICLR 2020
9
30
40
47
9846
20
8000
33
0
88
0
284
B587
3
6000
102
9243
2
9526
30
4
15
374
4000
11
9765
37
23
23
109
9948
0
2700
9902
2
8
Predicted label
0
D
14
9984
8.00
2
0
3
9071
23
6000
22
0
20
9751
0
205
2
0
3993
2
0
0
9989
0
0
0
2
2000
Predicted label
(k)
Figure A7: Confusion maps for data generation using VaDE.
Figure A7 Posterior threshold 0.9.
Figure A7k Posterior threshold 0.999. (default for Jiang et al. (0117)
22

---

Under review as a conference paper at ICLR 2020
1.0
0.8
F-measure
0.6
0.4
0.2
0.0
Levine_32dim
Levine_13din
Samusik 01
Samusik_al all
Dataset
Figure A8; The boxplots show similar as in Weber & Robinson (016) the reproducibility of MoB-
Sim-VAE the four datasets when running MoE-Sim- VAE 30 times. The variance defining the correct subpopulations of MoE-Sim-We VAE is quite small and therefore also an improvment to many methods compared in Weber & Robinson (2066).
23

---

Under review as a conference paper at ICLR 2020
Table A1: Results of MoE-Sim- Sim- data published in Bodenmiller al. (2012). CyTOF mea- surements from peripheral blood mononuclear cells (PBMCs) were taken and the goal is to define the different cell types present in the data. The ground truth was definied using the SPADE algo- rithm (Qiu et al. 2011), which can visualize the high dimensional data such a way to be able to manual gate the cells. We compare other fully unsupervised methods FlowSOM, X-shift and PhenoGraph and achieve in most cases the best F-measure, which is defined as in Equation 12
Inhibitor
Well
MoE-Sim- VAE
FlowSOM
X-shift
PhenoGraph
AKTI
A02
0.7666
0.5147
0.5704
0.6588
AKTI
A03
0.7541
0.4793
0.546
0.6026
AKTI
A04
0.6815
0.6405
0.5298
0.5974
AKTi
A05
0.7127
0.7108
0.6089
0.6104
AKTi
A06
0.6711
0.7383
0.572
0.6611
AKTI
A07
0.7233
0.7034
0.5583
0.6981
AKTi
A08
0.7901
0.7024
0.4541
0.5287
AKTI
A09
0.7604
0.4292
0.5014
0.6414
AKTI
A10
0.7275
0.4952
0.4144
0.677
AKTI
A11
0.7540
0.6456
0.6673
0.6302
BTKi
A02
0.7261
0.698
0.7136
0.7478
BTKi
A03
0.7982
0.6643
0.6012
0.7141
BTKi
A04
0.7835
0.6864
0.6983
0.7103
BTKi
A05
0.7484
0.6397
0.7454
0.7474
BTKi
A06
0.8196
0.703
0.7625
0.7949
BTKi
A07
0.7976
0.6729
0.6841
0,7102
BTKi
A08
0.8108
0.6715
0.5887
0.6884
BTKi
A09
0.7789
0.5299
0.6426
0.7236
BTKi
A10
0.7726
0.6319
0.6775
0.7148
BTKi
A11
0.7857
0.6078
0.5939
0.6786
BTKi
A12
0.6600
0.5503
0.6028
0.6308
Crassin
A01
0.6727
0.6488
0.6315
0.6237
Crassin
A02
0.8225
0.557
0.6435
0.7165
Crassin
A03
0.8346
0.5736
0.6628
0.7085
Crassin
A04
0.8446
0.5348
0.7146
0.7045
Crassin
A05
0.8462
0.7444
0.6227
0.7202
Crassin
A06
0.8569
0.7448
0.7078
0.6972
Crassin
A07
0.8170
0.5164
0.6546
0.6309
Crassin
A08
0.8431
0.8283
0.5504
0.6546
Crassin
A09
0.8412
0.5814
0.6027
0.6684
Crassin
A10
0.8527
0.7537
0.6586
0.6338
Crassin
A11
0.8453
0.7174
0.6437
0.7358
Crassin
A12
0.7320
0.6161
0.6436
0.6949
Dasatinib
A0I
0.7235
0.4466
0.554
0.6725
Dasatinib
A02
0.8019
0.516
0.6238
0.701
Dasatinib
A03
0.7864
0.5108
0.5366
0.6566
Dasatinib
A04
0.6661
0.4796
0.5527
0.647
Dasatiníb
A05
0.7910
0.5014
0.5804
0.6904
Dasatinib
A06
0.7979
0.5167
0.6258
0.6707
Dasatinib
A07
0.8105
0.5215
0.6016
0.6809
Dasatinib
A08
0.8047
0.6928
0.5802
0.633
Dasatinib
A09
0.7485
0.5203
0.5958
0.6861
Dasatinib
A10
0.8062
0.5158
0.5742
0.6503
Dasatinib
A11
0.7837
0.5066
0.6331
0.6813
GDC-0941
A01
0,5632
0.6434
0.5987
0.6279
GDC-0941
A02
0.8257
0.7291
0.7349
0.7507
GDC-0941
A03
0.8268
0.7321
0.6822
0.7853
GDC-0941
A04
0.8389
0.7115
0.7569
0.7421
GDC-0941
A05
0.8382
0.7946
0.7171
0.7735
GDC-0941
A06
0.8463
0.6125
0.6858
0.764
GDC-0941
A07
0,8382
0.6061
0.7776
0.7612
24

---

Under review as a conference paper at ICLR 2020
GDC-0941
A08
0.8249
0.5493
0.6058
0.7796
GDC-0941
A09
0.8606
0.7689
0.8043
0.7206
GDC-0941
A10
0.8412
0.7227
0,653
0.6465
GDC-0941
A11
0.7859
0.5703
0.7297
0.7891
GDC-0941
A12
0.7803
0.6326
0.69
0.6727
Go69
AOI
0.6520
0.6571
0.718
0.5822
Go69
A02
0.7835
0.7693
0.6075
0.7322
Go69
A03
0,7305
0.7334
0.757
0,6414
Go69
A04
0.7640
0.7456
0.8013
0.7425
Go69
A05
0.7812
0.7555
0.7294
0.7727
Go69
A06
0.7816
0.7404
0.7437
0.6443
Go69
A07
0.7407
0.8513
0.7527
0.6811
Go69
A08
0.7293
0.7338
0.6984
0.6525
Go69
A09
0.8228
0.6955
0.6985
0.7317
Go69
A10
0,7560
0.7512
0.7689
0.7071
Go69
A11
0.7565
0.7373
0.7213
0.7315
Go69
A12
0.7426
0.7086
0.7846
0.6442
H89
AOI
0.6734
0.6952
0.6003
0.6105
H89
A02
0.7288
0.5391
0.5918
0.678
H89
A03
0.8051
0.5414
0.6856
0.6759
H89
A04
0.8144
0.7314
0.662
0.7287
H89
A05
0.7821
0.5468
0.6485
0.6672
H89
A06
0.7647
0.5636
0.8281
0.7165
H89
A07
0.7762
0.6983
0.7284
0.6442
H89
A09
0.8131
0.5442
0.5906
0.6707
H89
A10
0.7517
0.5549
0,6028
0.682
H89
A11
0.7417
0.7414
0.6863
0.7257
H89
A12
0.7939
0.6934
0,5831
0.6401
IKI
A02
0.7945
0.6619
0.7371
0.6475
IKKi
A03
0.6873
0.6568
0.5661
0.6895
IKKi
A04
0.7942
0.6754
0.6386
0.7052
IKKi
AOS
0.6977
0.6569
0.6157
0.6899
IKKi
A06
0.7442
0.6931
0.7024
0.7077
IKKi
A07
0.7352
0.5303
0.669
0.7001
IKKi
A08
0.7470
0.7006
0.5358
0.6869
IKKi
A09
0.8097
0.5175
0.6299
0.6969
IKKi
A10
0.7647
0.6308
0.657
0.7334
IKKi
A11
0.7878
0.6365
0.6757
0.6613
IKKi
A12
0.6673
0.5629
0.497
0.6043
Imatinib
A02
0.7935
0.7571
0.6721
0.7677
Imatinib
A03
0.7763
0.7429
0.7041
0.7499
Imatinib
A04
0.8058
0.7564
0.6921
0.7229
Imatinib
A05
0.7714
0.7559
0.6689
0.7609
Imatinib
A06
0.7756
0.746
0.6956
0.7296
Imatinib
A07
0.7468
0.7515
0.6974
0.7137
Imatinib
A08
0.7631
0.7534
0.5189
0.7096
Imatinib
A09
0.8082
0.5605
0.5819
0.7447
Imatinib
A10
0.7964
0.5645
0.5637
0.78
Imatinib
A11
0.7289
0.7664
0.7576
0.7395
Imatinib
A12
0.7012
0.8451
0.6369
0.7259
Jakli
A02
0.8210
0.5167
0.5771
0.616
Jakli
A03
0.7343
0.7139
0.6526
0.7133
Jakli
A04
0.7321
0.7066
0,6346
0,7189
Jakli
A05
0.7413
0.5163
0.6551
0.7089
Jakli
A06
0.7244
0.5525
0.6804
0.6905
Jakli
A07
0.7779
0.5499
0.5605
0.7099
Jakli
A08
0.7281
0.6995
0.6021
0.6605
Jakli
A09
0.8043
0.5064
0.6054
0.6717
Jakli
A10
0.7801
0.5295
0.5538
0.7015
25

---

Under review as a conference paper at ICLR 2020
Jakli
A11
0.7128
0.7307
0.7386
0.6812
Jakli
A12
0.7204
0.6229
0.6321
0.6905
Jak2i
A01
0.6944
0.6379
0.6014
0.6207
Jak2i
A02
0.7961
0.664
0.6656
0.7083
Jak2i
A03
0.7629
0.6742
0.7138
0.7024
Jak2i
A04
0.7890
0.6716
0,6227
0.7072
Jak2i
A05
0.6666
0.4689
0.5314
0.6459
Jak2i
A06
0.8110
0.6474
0.6651
0.6833
Jak2i
A07
0.7595
0.6818
0.7593
0.6982
Jak2i
A08
0.8050
0.6601
0.6152
0.686
Jak2i
A09
0.8028
0.5253
0.6414
0.6501
Jak2i
A10
0.8030
0.6762
0.6067
0.6364
Jak2i
A11
0.8228
0.5398
0.694
0.7473
Jak2i
A12
0.6831
0.6214
0.5825
0.5687
Jak3i
A02
0.7986
0.7108
0.5666
0.6912
Jak3i
A03
0.7170
0.7116
0.6991
0.7001
Jak3i
A04
0.7983
0.5243
0.6654
0.691
Jak3i
A05
0.7087
0.6498
0.6884
0.7073
Jak3i
A06
0.7272
0.7244
0.654
0.7059
Jak3i
A07
0.7768
0.5167
0.696
0.735
Jak3i
A08
0.7196
0.6797
0.5946
0.7287
Jak3i
A09
0.7988
0.6918
0.6013
0.6826
Jak3i
A10
0.8026
0.7103
0.7104
0.7219
Jak3i
A11
0.7281
0.5107
0.6854
0.6614
Jak3i
A12
0.7511
0.6135
0.4861
0.61
Leki
AOT
0.7359
0.7582
0.6106
0.7201
Lcki
A02
0.7605
0.7453
0.6391
0.7696
Leki
A03
0.8032
0.5608
0.6814
0.721
Leki
A04
0,7608
0.5764
0.6788
0.7904
Leki
A05
0.8210
0.5435
0.7204
0.7442
Leki
A06
0.7564
0.7662
0.728
0.7556
Lcki
A07
0.8304
0.579
0.6992
0.696
Leki
A08
0.7854
0.7457
0.5904
0.6972
Leki
A09
0.8452
0.5859
0.6018
0.7569
Lcki
A10
0.7387
0.744
0.6598
0.6627
Lcki
A11
0.7835
0.7639
0.6836
0.7558
Leki
A12
0.7467
0.8271
0.6888
0.6878
PP2
A02
0.7687
0.759
0.7717
0.7605
PP2
A03
0.8395
0.7644
0.7304
0.7953
PP2
A04
0,8442
0.7703
0.7116
0.7162
PP2
AO5
0.8248
0.5777
0.7205
0.7547
PP2
A06
0.7866
0.7612
0.7461
0.7431
PP2
A07
0.8595
0.7616
0.724
0.7213
PP2
A08
0.8505
0.7489
0.7109
0.7195
PP2
A09
0.7902
0.5755
0.6511
0.7738
PP2
A10
0.8089
0.743
0.6635
0.7389
PP2
A11
0.7977
0.5852
0.6564
0.7846
PP2
A12
0.7667
0.6012
0.6524
0.6636
Rapamycin
A01
0.7028
0.675
0.5882
0,5677
Rapamyein
A02
0.7215
0.6831
0.6124
0.6697
Rapamycin
A03
0.7322
0.6707
0.6296
0.6861
Rapamycin
A04
0.6787
0.6696
0.6887
0.7267
Rapamycin
A05
0.7231
0.653
0.7134
0,6466
Rapamyein
A06
0.7310
0.6473
0.7009
0.6386
Rapamycin
A07
0.7595
0.6642
0.748
0.5882
Rapamycin
A08
0.7773
0.836
0.6371
0.571
Rapamycin
A09
0.7732
0.6573
0.6826
0.6615
Rapamycin
A10
0.7586
0.6702
0.7136
0.6344
26

---

Under review as a conference paper at ICLR 2020
Rapamyein
A12
0.6955
0.6361
0.6561
0.5472
SB202
A0I
0,6884
0.6713
0.941
0.7101
SB202
A03
0.7869
0.7549
0.6686
0.7633
SB202
A05
0.7856
0.5564
0.7387
0,6999
SB202
A06
0.7707
0.755
0.7913
0.7869
SB202
A10
0.7559
0.7554
0.7749
SP6
A01
0.7033
0.6882
0.4191
0.532
SP6
A02
0.7536
0.5035
0.5104
0.657
SP6
A03
0.7387
0.6973
0.534
0.3858
SP6
A04
0.6910
0.503
0.5065
0.5975
SP6
A05
0.7210
0.5068
0.5643
0.6869
SP6
A06
0.7052
0.719
0.5063
0.6384
SP6
A07
0.7281
0.7074
0.5382
0,6501
SP6
A08
0.7301
0.6832
0.4665
0.6133
SP6
A09
0.7743
0.5001
0.4618
0.6208
SP6
A10
0.7198
0.5111
0.524
0.6773
SP6
A11
0.7494
0.493
0.5407
0.5935
SP6
A12
0.7311
0.6131
0.4488
0.6198
Sorafenib
AOT
0.7185
0.7217
0.5884
0.6574
Sorafenib
A02
0.8250
0.7659
0.6658
0.7664
Sorafenib
A03
0.7689
0.7732
0.7078
0.6869
Sorafenib
A04
0.8360
0.7094
0.7114
0.7218
Sorafenib
A05
0.8304
0.5571
0.7672
0.7153
Sorafenib
A06
0.8021
0.5783
0.6991
0.7506
Sorafenib
A07
0.8461
0.7051
0.7267
0.6701
Sorafenib
A09
0.8226
0.7275
0.7522
0.7587
Sorafenib
A10
0.8103
0.7561
0.7457
0.7214
Sorafenib
A11
0.8465
0.5777
0.7192
0.7503
Sorafenib
A12
0.7715
0.6533
0.6084
0.6129
Staurosporine
AOI
0.7985
0.8464
0.6057
0.5945
Staurosporine
A02
0.8347
0.8312
0.5999
0.6626
Staurosporine
A03
0.8079
0.7072
0.6704
0.6787
Staurosporine
A04
0,8418
0.8666
0,6452
0.6776
Staurosporine
A05
0.8657
0.7305
0.7071
0.7515
Staurosporine
A06
0.8694
0.516
0.6453
0.6619
Staurosporine
A07
0.8277
0.7052
0.6349
0.6657
Staurosporine
A08
0.8310
0.8316
0.6213
0.678
Staurosporine
A09
0.8319
0.5117
0.6747
0.6726
Staurosporine
A10
0.8417
0.5108
0.6211
0.7126
Staurosporine
A11
0.8246
0.8711
0.6547
0.7445
Streptonigrin
A01
0.7128
0.5689
0.6571
0.6599
Streptonigrin
A02
0.7836
0.5095
0.549
0.6155
Streptonigrin
A03
0.7776
0.547
0.6497
0.6527
Streptonigrin
A04
0.8466
0.7521
0.5762
0.7061
Streptonigrin
A05
0.8130
0.5406
0.6459
0.6928
Streptonigrin
A06
0.8031
0.7409
0.6446
0.6343
Streptonigrin
A07
0.7987
0.5353
0.5882
0.6657
Streptonigrin
A08
0.7470
0.7458
0.5864
0.6443
Streptonigrin
A09
0.7586
0.7034
0.5928
0.6196
Streptonigrin
A10
0.7159
0.6974
0.5174
0.6809
Streptonigrin
A11
0.8178
0.5649
0.593
0.6814
Streptonigrin
A12
0.7410
0.6034
0.5896
0.6286
Sunitinib
A0I
0.7152
0.6622
0.5653
0.6522
Sunitinib
A02
0.8056
0.498
0.6138
0.6521
Sunitinib
A03
0.8095
0.6873
0.6889
0.6913
Sunitinib
A04
0.8142
0.6925
0.6467
0.7121
Sunitinib
A05
0.8157
0,6959
0.673
0.7073
Sunitinib
A06
0.7968
0.5061
0.6654
0.7025
Sunitinib
A07
0.8110
0.7
0.6333
0.6572
27

---

Under review as a conference paper at ICLR 2020
Sunitinib
A08
0.8186
0.6894
0.5999
0.674
Sunitimib
A09
0.8029
0,4886
0.6699
0.6621
Sunitinib
A10
0,8126
0.848
0.6087
0.6713
Sunitinib
A11
0.8241
0.824
0.6408
0.6811
Sunitinib
A12
0.7747
0.7898
0.5942
0.5867
Syki
A02
0.7682
0.7073
0.6636
0.685
Syki
A03
0.7224
0.7042
0.6424
0.7116
Syki
A04
0.7461
0.7069
0.7908
0.7256
Syki
A05
0.7468
0.7182
0.6263
0.6804
Syki
A06
0.7381
0.7134
0.7718
0.7154
Syki
A07
0.7891
0.7
0.7434
0.6479
Syki
A08
0.7509
0.7154
0.6903
0.6542
Syki
A09
0.7712
0.73
0.7357
0.6918
Syki
A10
0.7695
0.7531
0.7197
0.7242
Syki
AI1
0,7360
0,7311
0.7577
0.78
Syki
A12
0.6717
0.6793
0.7426
0.7123
U0126
AOI
0.6844
0.6178
0.6486
U0126
A02
0.8440
0.5545
0.5362
0.7043
U0126
A03
0.8340
0.5346
0.616
0.6881
U0126
A04
0.8263
0.7079
0.6166
0.7059
U0126
A05
0.8535
0.5468
0.7091
0.7031
U0126
A06
0.8199
0.5285
0.6018
0.6874
U0126
A07
0.8079
0.5304
0.5671
0.7249
U0126
A08
0.8278
0.6864
0.5359
0.6577
U0126
A09
0.8331
0.5394
0.5678
0.6967
U0126
A10
0.8436
0.5593
0,6092
0.6867
UO126
A11
0.7654
0.5072
0.6374
0.6767
U0126
A12
0.7227
0.6496
0.6253
0.6281
VX680
AOI
0.6930
0.4818
0.6028
0.6452
VX680
A02
0.7340
0.711
0.5587
0.633
VX680
A03
0.7525
0.6976
0.5663
0.7292
VX680
A04
0.8127
0.6435
0.6722
0.5954
VX680
A05
0.6937
0.6742
0.7374
0.6454
VX680
A06
0.7168
0.7101
0.5769
0.6202
VX680
A07
0.7663
0.4944
0.5382
0.718
VX680
A08
0.7315
0.7082
0.4753
0.6482
VX680
A09
0.7703
0.7054
0.5859
0.6722
VX680
A10
0.7143
0.7137
0.6648
0.6167
VX680
AIl
0.7050
0,6773
0.7269
0.6947
VX680
A12
0.7852
0.7922
0.5583
0.6808
28


Reviews:
[{"review_id": "S1gYbn7fqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\n\nThe paper proposes to expand the VAE architecture with a\nmixture-of-experts latent representation, with a\nmixture-component-specific decoder that can specialize in a specific\ncluster.  Importantly, the method can take advantage of a similarity\nmatrix to help with the clustering.\n\nOverall, I recommend a weak accept.  The method seems reasonable, and\nthe paper is well-written, but the results are only marginally better\nthan other methods, and there are several weaknesses with the proposed\narchitecture and experimental setup.\n\nPositives:\n\n* The idea of a more expressive variational distribution seems good,\n  although it is not novel.\n\n* The ability to have multiple decoder networks seems reasonable.\n\n* The ability to incorporate domain knowledge (in the form of a\n  similarity matrix S) is a plus.\n\n* The experiments are thorough, although the method is generally only\n  slightly better than competing methods.\n\nNegatives:\n\n* It's not clear if the similarity matrix S is already solving the\n  clustering problem - in which case, why do we need the rest of the\n  model?  For example, in your experiments you often used UMAP to\n  cluster data.  How does using UMAP by itself work?  (Along these\n  lines, it was not clear if your GMM experiments clustered data in\n  the original space, or in the UMAP'd space - please clarify this).\n  A good ablation would be to somehow remove the S matrix, to see if\n  the model can accurately cluster samples.\n\n* There is little variance in the generated samples.  \n\n* There is not a one-to-one mapping of clusters to labels, so it is\n  hard to use this method to generate a specific type of data (for\n  example, it is hard to generate a specific digit).  This is a big\n  difference from, say, a conditional sampler as learned by a GAN.\n  This also arises in Fig. 3, where it is clear that latent cluster\n  assignments do not match human-interpretable cluster assignments.  I\n  suppose this is to be expected, but taken with the previous point\n  (little variance in generated samples) I think it seriously weakens\n  the paper's claim that this is an \"accurate an efficient data\n  generation method.\"\n\n* The method does not do well when the number of clusters is large.\n  Regular GMMs seem to outperform it.\n\n* I felt that this paper made excessive use of the appendix.  The\n  paper is not self-contained enough, effectively violating the length\n  restrictions.  Please make an effort to move key results back in to\n  the main body of the paper.\n\n\nExperiments to run:\n\nAn ablation regarding the similarity matrix S.\n\nClarification of whether GMM experiments are run in data-space, or\nUMAP'd space.\n\nMIXAE features prominently in your related works, but is not compared\nto in your experiments.  It sounds like a natural comparison.  Please\nrun this experiment, or explain why it is not a comparable method.\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gqaqHe5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present an extension of variational autoencoders (VAEs), where Gaussian distribution of the latent variable is replaced by a mixture of Gaussians. The approach can be used for clustering and generation. The authors carry out experiments to evaluate the performance of the method in these tasks and compare it to competing methods.\n\nThe paper is well written and easy to read and understand. Specialized related work is discussed. I find the extension of VAEs to GMMs interesting for the ICLR community, although it is somewhat straight forward in terms of its technical difficulty. However, the technical novelty together with the fine empirical evaluation are just good enough for ICLR, in my opinion.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "H1gaakiptr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed method of mixture-of-experts variational autoencoders\nis valuable and insightful.\nOn the other hand the work could be improved and clarified at some points:\n\n- in the abstract it is claimed that the method works for high-dimensional data.However, it should be better explained why this is the case. The method is largely based on density estimation with a mixture of Gaussians which is known to have limitations in higher dimensions (see e.g. classical textbooks like Bishop 1995)\n\n- the similarity matrix and the similarity values should be carefully defined. Is there also an underlying similarity function assumed?\n\n- a main shortcoming is that there is no discussion or experimental comparison with methods like spectral clustering and kernel spectral clustering. Given that the paper and the proposed method relates to similarity-based representations it would be important to know how it compares to such methods. Though e.g. in Table 1 the authors compare with about 10 other methods it would be more relevant that among some of these would have been spectral clustering and kernel spectral clustering, because of the similarity-based representations.\n\n- in section 4.1 the MNIST data are taken with k=10. Though it is nicely explained and illustrated on this data set, it is possibly somewhat misleading as an example. The reason is that this is a classification problem with 10 classes, therefore the choice k=10 is obvious. It would be more important to consider benchmark problems for clustering, instead of classification, for which the choice of k is also an important model selection issue and for which k is unknown (how should k be selected then?).\n\n- is each cluster always be assumed to be a Gaussian (which seems to be a strong assumption in general, and possibly not always realistic)? Could other components be used in the mixture?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a VAE with a mixture-of-experts decoder for clustering and generation of high-dimensional data. Overall, the reviewers found the paper well-written and structured , but in post rebuttal discussion questioned the overall importance and interest of the work to the community.  This is genuinely a borderline submission. However, the calibrated average score currently falls below the acceptance threshold, so I’m recommending rejection, but strongly encouraging the authors to continue the work, better motivating the importance of the work, and resubmitting.
Decision: Reject