A Mixture of Variational Autoencoders for Deep Clustering
Authors: Avi Caciularu
Jacob Goldberger
Venue:  (2021)
Keywords: deep clustering
variational auto encoder
VAE

Abstract: In this study, we propose a deep clustering algorithm that utilizes a variational autoencoder (VAE) framework with a multi encoder-decoder neural architecture. This setup enforces a complementary structure that guides the learned latent representations towards a more meaningful space arrangement. It differs from previous VAE-based clustering algorithms by employing a new generative model that uses multiple encoder-decoders.
We show that this modeling results in both better clustering capabilities and improved data generation. The proposed method is evaluated on standard datasets and is shown to outperform state-of-the-art deep clustering methods significantly.

Under review as a conference paper at ICLR 2021
A MIXTURE OF VARIATIONAL AUTOENCODERS FOR DEEP CLUSTERING
Anonymous authors Paper under double-blind review
ABSTRACT
In this study, we propose deep clustering algorithm that utilizes a variational autoencoder (VAE) framework with multi encoder-decoder neural architecture. This setup enforces a complementary structure that guides the learned latent rep- resentations towards a more meaningful space arrangement. It differs from previ- ous VAE-based clustering algorithms by employing a new generative model that uses multiple encoder-decoders. We show that this modeling results in both bet- ter clustering capabilities and improved data generation. The proposed method is evaluated on standard datasets and is shown to outperform state-of-the-an deep clustering methods significantly.
INTRODUCTION
Clustering is one of the most fundamental techniques used in unsupervised machine learning. It is the process of classifying data into several classes without using any label information. In the past decades, plethora of clustering methods have been developed and successfully employed in various fields, including computer vision (Jolion et al. 1991). natural language processing (Ngomo) & Schumacher] (009), social networks (Handcock et al. (007) and medical informatics (Gotz al., 2011). The most well-known clustering approaches include the traditional k-means algorithm and the generative model, which assumes that the data points are generated from cexture-of-Gaussian (MoG), and the model parameters are learned via the Expectation Expecation-MMxximizztoon (EM) algorithm. However, using these methods over datasets that include high- dimenssiini data is problematic since, in these vector spaces, the inter-point distances become less informative. As result, the respective methods have provided new opportunities for clustering (Min et al., 2018). These methods incorpo- rate the ability to learn a (non-linear) mapping of the raw features in low-dimensional vector space that hopefully allow a more feasible application of clustering methods. Deep learning methods are expected to automatically discover the most suitable non-linear representations for specified task. However, a straightforward implementation of "deep" k-means algorithm by jointly learning the embedding space and applying clustering to the embedded data, leads to trivial solution, where the data feature vectors are collapsed into a single point in the embedded space, and thus, the centroids are collapsed into single spurious entity. For this reason, the objective function of many deep clustering methods is composed of both clustering term computed in the embedded space and a regularization term in the form of reconstruction error avoid data collapsing.
One broad family of successful deep clustering algorithms, which was shown to yield state-of- the-art results, is the generative model-based methods. Most of these methods are based on the Variational Autoencoder framework (Kingma & Welling, 2014), e.g., Gaussian Mixture Variational Autoencoders (GMVAE) (Dilokthanakul et al.j 2016) and Variational Deep Embedding (VaDE). Instead of using an arbitrary prior to the latent variable, these algorithms proposed using specific distributions that will allow clustering at the bottleneck, such as MoG distributions. This design results in a VAE based training objective function that is composed of significant reconstruction term and a second parameter regularization term, as discussed above. However, this objective seems to miss the clustering target since the reconstruction term is not related to the clustering, and actual clustering is only associated with the regularization term optimization. This might result in inferior clustering performance, degenerated generative model, and stability issues during training.
We propose a solution to alleviate the issues introduced by previous deep clustering generative models. To that end, propose the k-Deep Variational Auto Encoders (dubbed k-DVAE). Our

---

Under review as a conference paper at ICLR 2021
k-DVAE improves upon the current sfate-of-the-att clustering methods in several facets: (1) A novel model that outperforms the current methods in terms of clustering accuracy. (2) A novel Variational Bayesian framework to balance the data reconstruction and actual clustering that differs from the previous methods. (3) A network architecture that allows better generative modeling and thus more accurate data generation. Importantly, this architecture uses lower amount of parameters compared to previous models. We implemented the k-DVAE algorithm on various standard document and image corpora and obtained improved results for all the datasets we experimented with compared to flate-of-the-as clustering methods.
RELATED WORK
Deep clustering has been studied extensively the literature. The most common deep clustering methods aim to project the data into non-linear, low-dimensional feature space, where the task of clustering appears be feasible. Then, traditional clustering methods are further applied to perform the actual clustering. Previous works have employed autoencoders (Yang et al. 2016 Ghasedi Dizaji et al. 2017, Yang et al. 2017: Fogel et al. 2019 Opochinsky et al., 2020). Variational Autoencoders (VAEs) (Jiang et al. 2016 Dilokthanakul et al. [2016; Yang et al. [2019 Li et al. 2019) and Gen- erative Adversarial Networks (GANs) (Springenberg/ 2015 Chen et al., 2016). IMSAT (Hu et al.)
2017), is another recent method that augmented the training data. Our method does not make any use of augmented data during training and therefore, we do not consider IMSAT to be an appropri- ate fair baseline for comparison. Additionally, the GMVAE method has shown to yield inferior performance results compared to the rest of VAE-based deep clustering, hence we do not present it in our evaluations.
Among the aforementioned work, VaDE (Jiang et al. 2016) and k-DAE (Opochinsky et al. 2020) are most relevant our work. Both VaDE and our work utilize the Varitional Bayes framework, and use probabilistic generative process to determine the data generation model. Yet, the difference lies in both the generative process and the use of several autoencoders: our network consists of set of k autoencoders, where each specializes on encoding and reconstructing a different cluster. The k-DAE architecture consists of a set of k autoencoders, but does not consider generative modelling, which we show, proved to be more powerful and yields significant clustering performance results recent years.
The recent, offte-of-the-att DGG method (Yang et al.) 2019) built on the foundations of VaDE, and integrates graph embeddings that serves as a regularization over the VaDE objective. Using the DGG revised objective, each pair of samples that are connected on the learned graph, will have similar posterior distributions, using the Jenson-Shannon (JS) divergence similarity metric. The other baselines used in this study described in Section 4.2
3 THE h &-DVAE CLUSTERING ALGORITHM
In this section, we describe our k-Deep Variational Auto Encoders (dubbed k-DVAE). First, we formulate the generative model that our algorithm is based on. Next, we derive the optimization objective score. Then we discuss the differences between our model and previous VAE based algo- rithms such VaDE (Jiang et al. 2016) and illustrate the advantages of our approach.
3.1 GENERATIVE MODEL
In our generative modeling, we assume that the data are drawn from a mixture of VAEs, each with standard Gaussian latent r.V., follows:
i = I,..., k. 2. Sample latent from the unit normal distribution, IO.
1. Draw a cluster by sampling from ply = = ai,
3. Sample an observed r.v. x:
(a) If is real-valued vector: sample data vector using the conditional distribution, x| (z, = i) ~ ((ue, (z), Σe, (z)).
(b) x is binary vector: sample a data vector using the conditional distribution, x| (,, y i) Ber(uo, (z)).
2

---

Under review as a conference paper at ICLR 2021
0g is the stacked vector of parameters of the i-th neural network (NN). It formulates de- coder NN that corresponds to the i-th cluster, S Σ k. assuming that the total number of clusters is k. Mo, (z), Eo, (z) are computed by decoder NN with an input and parameters 0, We denote the parameter set of all the decoders by {01.-.,0k} 0z)-
Note that the latent data representation is drawn independently of the selected class y, and the class only affects when selecting the sample x.
3.2 LEARNING THE MODEL PARAMETERS BY OPTIMIZING A VARIATIONAL. LOWER BOUND
Direct optimization of the likelihood function:
p(x:0) = E p(z))(6))(x)) v:0)dz
is intractable. Instead, we can use variational approximation methods and learn the model parame- ters by maximizing the Evidence Lower BOund (ELBO) lower bound. The ELBO(O, A) expression is given by:
ELBO A) = Σ g(y z|x;A) log p(x|y, 2;0)dz - DKI(aly, 2////// z;))),
(1)
where DKL. is the Kullback Leibler (KI.) divergence between two density functions, and g(v 2 xx;A) a conditional density function parametrized by X.
We use an approximate conditional density q(y, z z|x) that mirrors the structure of the generative
model. For each cluster we define an encoder that transforms the input x into the latent space of that cluster:
(3 = i, i,z(x:A) gly llx)g(z)xx = i;A;),
such that q(z)x,y y encoder NN with input x and parameter-set X and we use the notation = {AI. A
i; (Z.HA, (x), Σx, (x)) where nsx (x), Σx, (x) are computed by an
The first term of the ELBO expression can be written as:
Σ y
(v,, 2 log 2;0) Σ a( = (lx) log (x:uaa (z), Eo. (z)).
We next use Monte-Carlo sampling to approximate the expectation in Eq. (2):
E Bg(x)x.y=it..) (og.((x;uuu (z), Es, (z)) xe logNN (z), Es, (z)),
(3)
such that z|(x, y = i) is sampled from (Hx, (x), Σx, (x)).
Applying the chain rule for KI. divergence to the second term of the ELBO expression we get: DK A|xxA))lp(y 2:0))
DKL (xl
Σ = (lx) (HAx E., Ex. (x)) 1)).
(4)
We next replace the soft clustering in Eq. 33 and Eq. 4 by a hard clustering:
alv (Vog (z:), Ee, (z:)) - DKL DKL(WWWA (N Ex, (x))
(5)
R max(log.N (z.), Eo, (z.)) DKr, (x), Ex, (O) I))).
Finally, by neglecting the term DEL (a( ( (p(,;0) (or equivalently setting ((1xx) p(y:0)),
we obtain the following objective for optimization:
ELBO(O, A) ss max (log (logN((xxs (x:
Eg, DRI.(N((M,, (x), Ex, (x)
s.t.
2, ~ ~ (x), Ex,(x)).
3

---

Under review as a conference paper at ICLR 2021
DVAE #1
DVAE #
Ax
81 MP,, (x). Ex, (x))
Me ~ Maya (x), (x))
log N(x; M4g (zi), Eo. (2:)) DKL (N (x), Ex, (W I)
Figure 1: A block diagram of the autoencoder that computes the ELBO of the k-DVAE clustering method, during training phase.
Algorithm ELBO score computation
Algorithm 2 Hard clustering
Input: Data sample Output: Estimated score.
Input: Data sample
Output: Estimated cluster (xx) of
for to & do
for 1 do Compute 2; +- 14n, (x) using the encoder.
Compute μxx (x) and Ex, (x) using the i-th en-
coder.
Compute μeo (2:) and Do, (z.) using the i-th decoder. end for
Draw MHs,,, (x), Σx, (x)).
Compute peo, (2:) and De, (2:) using the i-th
decoder. end for
Compute the ELBO score using Eq. 6
Compute the cluster ((x) using Eq.
When optimizing the ELBO expression, sample the Gaussian r.v. 2; 2;|(x, y i) using the repa- rameterization trick. Note that the ELBO objective function (6) consists of a reconstruction term and regularization term and both involved in the clustering decision. In the derivation of the objective function above we assumed that x real-valued vector. The derivation of the FLBBO objective function for the discrete case is similar. The computation procedure is depicted in Algorithm and the overall architecture of the autoencoder used in the training is depicted in Fig.
3.3 HARD CLUSTERING OF DATA POINTS
After the model parameters have been learned we can extract the data clustering. We chose deter- ministic version of the clustering procedure (6) that avoids sampling of z and was empirically shown to yield more stable results in our simulations. We used the expectation vector z, = μxx (x) instead of sampled z,. The hard clustering is thus defined as:
p(z,(x:Ai))
())) arg max(log N (Z:41o, (z:), Ea, (z.)) log
p(z,00,)
According to our generative model p(2;00:) N(2;; I), and
d
d
log (ogppZ/(xxx, = (z,;Z, Σx, (x)) = Σ
log N (ZisiZiis Ex, =
Σ
log dis
4

---

Under review as a conference paper at ICLR 2021
where d is the dimensionality of the latent r.v. and Σx, (x) Var(z|x, i) diag ...... Rg)-
This finally implies:
f(x) arg max(log.W(xxxx Eg, (2,)) 1
Σ
log dis)
(8)
The hard clustering procedure is depicted in Algorithm 2
3.4 COMPARISON TO THE VADE METHOD
Our method and the VaDE algorithm (Jang et al. 2016) are both based on generative models learned by variational autoencoders. We will now briefly describe VaDE and focus the differences from our model. The VaDE generative process is based on a MoG model combined with a non-linear function (decoder) and given by:
1. Draw cluster y by sampling from p(y =) City i= 1=1,...
2. Sample latent r.v. using the conditional distribution, zly = ~ N (z) E,(2))
3. If x is real valued, sample it using the conditional distribution, x/2 ~ N (No (z), (z)).
If x is binary valued, sample using the conditional distribution, x/z Ber ue (z)).
( Ez are computed by decoder NN with an input z and parameters O.
Note that unlike our method, this modeling uses the same decoder (parametrized by 0) to construct the observed data for all the different clusters. Hence the decoder is likely be very complex. In comparing the performance of two methods in the next section, we show that much less number of parameters are needed in our model than in VaDE, and the reconstruction quality of our model is much better.
The VaDE ELBO( A) term can approximated follows:
ELBO( A) z log N N( (x:41((z) (z)
poly = Σ
( (Diz Ex(x)) E,)) + +log
Pa =1(z)
(9)
ai
where = is sampled from ((xx (z), Σ> (x)). After the VaDE parameters are learned, the soft clus- tering of is 2))))2 where is sampled from (xx (x)))) For the full derivation, we refer the reader to Jiang et al. (2016).
Note that the term C Eq. (9) refers the actual MoG-based soft clustering performed by VaDE during the learning phase. The clustering is thus performed here only within the ELBO regulariza- tion term. our method, both the reconstruction and regularization parts of the ELBO term are involved in the clustering decision.
Another variant of our algorithm term, and it only minimizes the reconstruction error (Opochinsky et al. (020)) We show in the next section that this results in significant degradation of the clustering performance. Hence, it is required that both the reconstruction term and the regularization term of the ELBO should be involved in the clustering process.
non-generative approach that not have regularization
EXPERIMENTS AND RESULTS
In this section, we present the datasets, hyperparameters, and experiments conducted to evaluate our approach's 's clustering results and compare it to other clustering methods.
4.1 DATASETS
We used the following datasets in experiments:
5

---

Under review as a conference paper at ICLR 2021
MNIST:
The MNIST dataset consists of 70, 000 handwritten (ten) digits images, of size 28 x 28 pixels. Prepossessing includes centering the pixel values and flattening each image to a 784-dimensional vector.
STL-10:
The STL-10 dataset consists of RGB colored images of size 96 96 pixels. This dataset contains total number of 10 classes. Since clustering directly from raw pixels of high- resolution images is rather difficult, Prepossessing includes features extraction by passing the images to pre-trained ResNet-50 (He et al. (016) and then applying an average pool- ing operation reduce the dimensionality 2048.
REUTERS:
The REUTERS dataset consists of 10, 000 English news stories that relate to a total number of categories. Prepossessing includes computing of 2000-dimensional TF-IDF feature vectors for the most frequent words the articles.
HHAR: The Heterogeneity Human Activity Recognition (HHAR) dataset consists of 10, 200 sam- ple records, where each sample relates to one of 6 different categories. Each sample in this dataset is dimensiona vector.
Note that set be the actual number of classes of the given datasets during our simulations. The overall datasets statistics are summarized in Table
4.2 EVALUATED MODELS
We compared our method to the following state-of-the-art deep clustering algorithms:
Autoencoder followed by Gaussian Mixture Model (AE+GMM): This method trains single AE using the reconstruction objective, and then applies GMM-based clustering on the em- bedding space.
Variational Deep Embedding (VaDE): Introduces VAE based generative model that assumes the latent variables follows mixture Gaussians, where the means and variances of the Gaussian components are trainable (Jiang et al. (0166).
Latent Tree Variational Autoencoder (LTVAE): A VAE based model that assumes tree struc-
ture of the latent variables (Li et al. 2019).
Deep clustering via a Gaussian mixture VAE with Graph embedding (DGG): recent VAE based model that assumes tree structure of the latent variables (Yang al. 2019).
A d-Deep-AutoEncooder (k-DAE): This algorithm uses k-AEs for deep clustering, where k is
sumed to the number of clusters (Opochinsky et al. 2020).. This method serves as the ablation study for method, since it induces the same (reconstruction) objective without the KI. term (which stands for regularization).
k D--Dee--aariaionss AutoEncoder (k-DVAE): Our clustering method.
The encoder-decoder structure used for the first four methods is the same (for fair comparison) and composed as follows. Each encoder network uses dense layers of sizes D 500 -500020000-00000 10, and each decoder network uses dense layers of sizes 10 2000 500 - 500 D. All these methods use additional mid-layers (to perform clustering). This setting and the remaining hyperparameters were taken from Jiang et al. (2016). and Yang et al. (2019). For both method and the k-DAE method, the autoencoders used dense layers of sizes 10 100 500- D for the decoder. Note that although we needed to allocate one encoder-decodes network to each cluster, the number of parameters was still drastically lower than the compared
500 100 10 for the encoder, and
Table Datasets statisties
Dataset
# Samples
Input Dimension
Clusters
MNIST
70, 000
784
10
STL-10
13, 000
2048
10
REUTERS
10, 000
2000
4
HHAR
10, 200
561
6

---

Under review as a conference paper at ICLR 2021
Table 2: Clustering accuracy results for clustering benchmarks, Best performance is bolded.
MNIST
STL-10
Reuters
HHAR
AE+GMM
42.20)00) 0.2
79.40 + 0.2
71.12 1.1
77.74 + 0.1
VaDE
94.43 0.1
85.45 + 0.1
79.84 + 1.5
84.49 + 0.1
LTVAE
06.32+0..
90.05 + 0.1
80.96 + 1.8
85.10 + 0.2
DGG
0.1
90.59 0.2
$2.30
89.04 + 0.1
k-DAE
9.1
87.30 - 0.1
79.92
+ 1.1
87.26 = 0.1
k-DVAE
97.87 0.1
91.52 + 0.1
82.74 1.0
90.74 0.1
methods. We tried increasing the number of parameters for each method, but it did not result in any performance gains. Each encoder network outputs mean and variance vectors that form the multivariate normal distribution. The output of the decoder is single mean vector if the input is discrete; otherwise, it also outputs a variance vector to form the normal distribution.
In our implementation of k-DVAE, Similar to the DGG method (Yang et al. 2019), we first pre- train the VaDE network initialization. Then, we set the initial clusters by applying the k-means clustering over the VaDE embedded space. In our case, this architecture was used only in this initialization step.
4.3 CLUSTERING RESULTS
Clustering performance of all the compared methods was evaluated with respect to the unsupervised clustering accuracy (ACC) measure, given by
ACC
max mESt
Σ m(j(x)
N
where is the total number of data samples, is the ground-truth label that corresponds to that x; sample, (x;,) is the cluster assignment obtained by the model, and ranges over the S₃ of all possible one-to-one mappings between cluster assignments and labels. This measures the proportion of data points for which the obtained clusters can be correctly mapped to ground-truth classes, where the matching is based on the Hungarian algorithm (Kuhn 1955). It lies in the range of 0 to I where one is perfect clustering result and zero is worst.
In Table 2 we depict the quantitative clustering results over the tested benchmarks compared to clus- tering methods. We show the mean and average ACC clustering results over ten training sessions with different random parameter initializations. The table shows that our method outperformed the other methods terms of accuracy. In addition, using the non-variational k-DAE variant yields infe- rior results compared to our method, which emphasizes the superiority of the variational generative framework this setup.
4.4 QUALITATIVE ANALYSIS
A key modeling difference between Í-DVAE and recent wate-of-the-ar models is that model allocates a different decoder to each cluster. We saw that this yields improved clustering results. We show below that we also gain improved generation capability. In classification tasks, that discriminative methods are better than generative ones since the classes are known, and we only need to find the discriminative features. However, in clustering tasks where we need learn the clusters, there is a tight relationship between a model's 's generation capabilities and its clustering performance. To gain insight into our model's ''' data generation capabilities, we present examples of images generated by the model's generator network.
known
To generate an example from the i-th clusters, we first sample a random vector z from the unit nor- mal distribution and then feed it to the i-th decoder network, parametrized by 0: The VaDE/DGG algorithm, in contrast, uses a single decoder for all the clusters, Fig. iillustrates the generated sam- ples for digits 0 to 9 of MNIST by method compared to D Note that unlike the results shown
VaDE has a similar generative model as DGG. Thus we choose to depict the results of DGG, which is statc-of-the-art.
7

---

Under review as a conference paper at ICLR 2021
O
0
O
O
0
0
0
0
0
O 000000 0
0
0
0
0
1 1
1
/
1
/
/
/
/
2
2 2
222
2 2
7
3 9
3
3 3 333333 333
3 3 3 3 3
35
4
4 4 4 4 44
44 4 4 4 4 9 4
55
5 5 5555555550
55505580 535
6 6 b 6.6666 66
6
6
6 4 6 6
6 6 66
7 2 7 7
02777777
7
77
7 7 1
7
7
7
7 77
8 8
8
8
8
8
8
8
8
8
8 8 8
8
8
8
8
8
9 8
9 4
9
9
9
9
9
9
9
9
9
9
9
9
9
4
4
9
4 9
(a) DGG
(b) k-DVAE
Figure 2: Examples of generated digits. The digits were generated by the decoder networks of DGG and k-DVAE, digits in the same row come from the same cluster.
in Jiang et al. (2016), we performed the digits generation process without restricting the posterior's high values. note passing that in Jiang et al. (2016), the authors presented generation re- sults only for good cases where the posterior probability of the correct clustering While both &-DVAE and DGG were able to generate smooth and diverse digits, the images gener- ated by the DGG are prone to errors. In contrast, each decoder network of the k-DVAE successfully reconstructed its corresponding digit by only using random normal noise as an input.
at least 0.999.
CONCLUSION
In this work, proposed k- k-Deep Variational AutoEncoder (k-DVAE), neural generative model for deep clustering. This framework facilitates & encoder-decoder models designed to learn insight- ful low-dimensional representations for better clustering. The model is optimized by maximizing the evidence lower bound (ELBO) of the data log-likelihood. Using a distinct set of & parametrized models combined with the variational probabilistic framework results in a much richer representa- tion of each cluster than previous methods. Extensive experimental results on four different datasets demonstrate our method's s effectiveness over different ofate-ofthhe baselines, which require more parameters for training than our proposed architecture. Our qualitative analysis showcases the high quality of the generative model induced by our k-DVAE. Future research can extend our work by utilizing graph embeddings similarity objective or adding a discriminator network to further regu- larize the posterior.
REFERENCES
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the International Conference Neural Information Processing Systems (NIPS), 2016.
Nat Dilokthanakul, Pedro AM Mediano, Marta Garelo,, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders, arXiv preprint 161 1.02648, 2016.
Sharon Fogel, Hadar Averbuch-Elor, Daniel Cohen-Or. and Jacob Goldberger. Clustering-driver deep embedding with pairwise constraints. IEEE Computer Graphics and Applications, 39(4): 16-27, 2019.
8

---

Under review as a conference paper at ICLR 2021
Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
David Gotz, Jimeng Sun, Nan Cao, and Shahram Ebadollahi. Visual cluster analysis in support of clinical decision intelligence. In AMIA Proceedings of the Annual Symposium on American Medical Informatics Association (AMIA), 2011.
Mark S Handcock, Adrian E Raftery, and Jeremy M Tantrum. Model-based clustering for social networks. Journal of the Royal Statistical Society: Series A (Statistics in Society), 170(2):301- 354, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition (ICCV), 2016.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing ouggeneeeete training. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
Zhuxi Jiang, Yin Zheng. Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. arXiv preprint arXiv: 1.05148, 2016.
Jean-Michel Jolion, Peter Meer, and Samira Bataouche. Robust clustering with applications in computer vision. IEEE Transactions on Putlern Analysis and Machine Intelligence, 13(8):791- 802, 1991.
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes. In International Conference on Learning Representations (ICLR), 2014.
Harold W Kuhn. The hungarian method for the assignment problem. Naval Research Logistics
Quarterly, 2:83-97, 1955.
Xiaopeng Li, Zhourong Chen, Leonard KM Poon, and Nevin I. Zhang. Learning latent superstruc- tures in variational autoencoders for deep multidimensional clustering. In International Confer- ence on Learning Representations (ICLR), 2019.
Erxue Min, Xifeng Guo, Qiang Liu, Gen Zhang, Jianjing Cui, and Jun Long. A survey of clustering with deep learning: From the perspective of network architecture. IEEE Access, 6:39501-39514, 2018.
Axel-Cyrill Ngonga Ngomo and Frank Schumacher. Borderflow: A local graph clustering algo- rithm for natural language processing. In International Conference on Intelligent Text Processing and Computational Linguistics, 2009.
Yaniv Opochinsky, Shlomo E Chazan, Sharon Gannot, and Jacob Goldberger. K-autoencoders deep clustering. In IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP), 2020.
Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint arXiv: 1511.06 2015.
Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. Towards K-means-friennll spaces: Simultaneous deep learning and clustering. In International Conference on Machine Learning (ICML), 2017.
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Linxiao Yang, Ngai-Man Cheung, Jiaying Li, and Jun Fang. Deep clustering by gaussian mix- ture variational autoencoders with graph embedding. In Proceedings of the IEEE International Conference Computer Vision (ICCV), 2019.


Reviews:
[{"review_id": "TPtLdLIeoB6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1) Quality of the writing.\n\nThe paper is rather difficult to read because of many typos and confusing statements. The authors should spend more time in refining the clarity of the presentation. Some suggestions: run a spell checker, avoid sloppy statements, be more meticulous in the description of related works and improve Figure 1.\nTo give some examples, here are two sentences from the Introduction:\n\n“As a result, the respective methods have provided new opportunities for clustering. These methods incorporate the ability to learn a (non-linear) mapping of the raw features in a low-dimensional vector space ...”. Here I struggle to understand which are the respective methods and which are these methods. The only models described before are the K-means and the Gaussian Mixture Model. They surely do not incorporate the ability to learn a mapping of the raw features into a latent representation.\n\n“However, this objective seems to miss the clustering target, since the reconstruction term of is not related to the clustering and actual clustering is only associated with the regularization term optimization.”  Besides some typos, this seems a rather strong statement as VaDE and GMMVAE indeed are quite good at clustering and the fact that the clustering objective is enforced by the prior distribution in the latent space does not result in “missing the clustering target”. A reference or a more systematic description could help.\n\n\n2) Hard clustering and K-DAE.\nThe title and the first chapters (till Chapter 3.2) are unfortunately quite misleading. The authors state that they propose a novel Variational Bayesian framework with K encoder-decoder architecture and they derive the corresponding ELBO in Chapter 3.2. However, they then choose to retain a deterministic approach. The latter is nothing else than a mixture of K Autoencoders with an L2 regularisation in the latent space. Hence, the proposed approach does not differ much from the k-DAE approach and I believe it cannot be described as a “novel Variational Bayesian framework\".\n\n3) Latent space.\nOne of the advantages of the VaDE method is that the latent space is nicely clustered following a Mixture of Gaussians distribution. The proposed method, on the other hand, chooses a unit normal distribution instead. As a result, the latent space cannot be used to visually investigate the data and there is no principled method to measure the uncertainty of the clustering prediction.\n\n3) Comparison with related work.\nThis paper reminded me of [1] which seems quite similar. I would find the paper more persuasive if it stated what the authors do over and above.\n[1] Kopf, A. et al. “Mixture-of-Experts Variational Autoencoder for clustering and generating from similarity-based representations.” ArXiv abs/1910.07763 (2019): n. pag.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "T-D076YJk0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to cluster a data-set into k groups using k-VAE.\nThe model is at the intersection of k-DAE (Opochinsky et al., 2020), and VaDE (Jiang et al., 2016).\n\nThe paper is straightforward and goes directly to the point: VAEs improve AEs.\nThe reason is however not discussed.\n\nSome sentences are a bit clumsy, eg.:\n\"Our k(DVAE improves upon the current state-of-the-art clustering methods in several facets: (1) A novel model ...(2) A novel, \"\nNovelty is not per se an improvement. Besides, the third point is not novel since it is what does k-DAE.\n\nRelated Works: Why talking about augmented method?\nThe original paper of GMVAE suggests that it does yield SoA performances.\nTYPO: 'GMVAE method has shown to yield[ed] inferior\"\n\nFigure 1 is unnecessarily large.\n\nSection 3.4 typo: \"in VaDE and the [R]econstruction quality\"\nTo some extent, an AE is always a generative model.\n\nExperiments: GMVAE is missing\nSection 4.3 is unnecessary.\n\nSection 4.4: \nk-DVAE is also prone to error:  lines for 3, 4 and 5 contain a 9\n\"This obviously affects the clustering accuracy of VaDe/DGG given in Section 4.3.\"\nDGG reports very similar ACC than k-DVAE.\n\n\"In contrast, each decoder network of the k-DVAE was able to successfully reconstruct its corresponding digit by only using y random normal noise as an input.\"\nThis is a bit far stretched: the decoder of VaDE is also able to reconstruct a number from Gaussian noise.\n\nTypo: \"by only using [y] random normal\"\n\n---\n\nThe fact that the paper is straightforward is a quality. However, some analysis are missing. For example, at first sight having k AE to  train seems a bulky situation. However the authors claim that the SoA performances can be reached with smaller architecture. How does this evolves with k? I other the words, what are the limit of VaDE and k-DVAE.\nOverall, the contributions are incremental and mildly novel.\n\n---\n\nIndeed, the GMVAE was already omitted in DGG's paper.\nRegarding section 4.3, I meant that you don't need to recall the definition of ACC. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-K7y2GEO8YE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a deep clustering method based on variational autoencoders.  The proposed method adopts a k-decoders architecture with a separate decoder for each cluster.  The proposed method was shown to achieve better clustering accuracy over several other recent deep clustering methods on four real-world data sets.\n\nThe paper is good that it provides relatively detailed explanation of the proposed method.  On the other hand, the proposed method basically combines the ideas of two previous works.  It uses a variational Bayesian estimation to estimate the autoencoders in k-DAE method.  The derivation looks straightforward., and the novelty and contribution look incremental.  The paper may perhaps emphasise more on the non-trivial parts.\n\nThe proposed method has been compared with a reasonable collection of baseline methods and data sets.  Although the proposed method produced the best results on all the selected data sets, the improved clustering accuracy is relatively small (less than 1) over the best methods. Besides, the empirical results are not very interesting.  It would be better if more results could be demonstrated.  For example, could any difference between the autoencoders for the different clusters be identified in data sets other than the MNIST digit data set?\n\nThe paper does not seem to provide the source code.  So there is concern on the reproducibility of the results and whether the proposed method can be easily used by practitioners.\n\nOverall, the proposed method has been shown have state of the art performance.  However, the paper is not very exciting as the proposed method appears to have limited novelty and the improvement in clustering accuracy is relatively small.  \n\nMinor comment:\n\n- Figure 1 looks too large proportionally.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-BA_FUH2INA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed to cluster the data using k different VAEs’. The method is different from the existing VAE-based deep clustering method (VaDE), which uses only one VAE but employs a Gaussian mixture prior to achieve the clustering goal. The difficulties of the proposed model lie at how to train the model efficiently. To this end, some approximations are made to the ELBO by using the MAP value to replace the expectation as well as dropping some KL term. The approximations are the key to the training, but not justified well. Experiments are conducted on several image and text datasets, and show superior performance comparing to existing deep clustering methods.\n\nStrength:\n\n1. The idea of using K different VAE’s to perform clustering is interesting, and is a good complementary to existing deep clustering methods.\n\n2. The experimental results demonstrate the superiority of the proposed method over existing ones.\n\nWeakness:\n\n1. The approximation to the ELBO by dropping one KL terms is somewhat unreasonable.\n\n2. Using MAP value to approximate the expectation is acceptable. But here, what you need is not the expectation value, but the gradient of the expectation w.r.t. model parameters \\lambda. So, when you replace the MAP value with the expectation in (5), the gradient computed from the MAP expression will be much different from the exact gradient. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submitted paper proposes a novel model/approach for deep clustering which shows good empirical performance on a set of standard benchmark datasets as compared to state of the art baseline algorithms. While I believe that this paper can be turned into a good ICLR paper, it doesn’t meet the standard of ICLR in its current form.

More specifically:
1) The quality of the writeup is poor, containing many typos but more problematically many unclear/confusing statements which are either vague/unclear, not supported by citations and/or substantiated in other parts of the paper (Examples: „This might result in inferior clustering performance, degenerated generative model, and stability issues during training.“ Or “However, this objective seems to miss the clustering target, since the reconstruction term of is not related to the clustering and actual clustering is only associated with the regularization term optimization.“). An important contribution of the paper could be to substantiate these statements and I argue that achieving better performance alone is not sufficient therefore.

2) From a theoretical perspective, it would be interesting whether there is any justification for statements like the ones references above. Furthermore, the authors’ approach involves several approximations whose implications are neither studied nor explained. The authors responded only partially to questions in that regard by reviewers leaving certain concerns unanswered. 

3) From an empirical perspective, an extended study of the proposed approach would help t better understand its benefits over existing approaches. Commonly considered settings like mismatch in the number of specified clusters are not studied at all. The proposed approach also seems to be initialized by VaDe (mentioned in Section 4) and it would be interesting to understand to which extend this is necessary and why (and how does performance change/degrade if this is not done). It also makes statements regarding stability unclear as training VaDe itself can be quite challenging. Furthermore, the overall algorithm for training the proposed model should be presented in a compact form in the paper. The paper should also be self-contained in the sense of containing information on the important hyper-parameters needed for training the proposed model.

In summary, the proposed approach is potentially interesting but the paper should not be accepted in its current form.

Decision: Reject